stambecco-13b-plus

🦌 Stambecco 13B Plus (PEFT / LoRA Adapter)

Stambecco 13B Plus is an Italian instruction-following Parameter-Efficient Fine-Tuning (PEFT) adapter based on Meta's LLaMA-13B architecture.

It was trained on the Stambecco Plus Dataset, an Italian instruction dataset traslated from Alpaca-GPT4 (GPT-4-LLM), a dataset distilled directly from GPT-4. As a result, this model offers superior Italian reasoning, vocabulary, and response quality compared to standard GPT-3.5-based models.

• GitHub Project: mchl-labs/stambecco
• Dataset Used: mchl-labs/stambecco_data_plus_it
• Model Type: PEFT / LoRA Adapter Delta

---

🚀 Quickstart & Usage

This repository contains the LoRA adapter weights (delta). To run inference, load these weights on top of the base LLaMA-13B model using Hugging Face's transformers and peft libraries:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model_name = "decapoda-research/llama-13b-hf"  # Base LLaMA-13B weights
adapter_name = "mchl-labs/stambecco-13b-plus"        # This adapter

# 1. Load Tokenizer & Base Model
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# 2. Load Stambecco LoRA Adapter
model = PeftModel.from_pretrained(base_model, adapter_name)

# 3. Generate Output
prompt = "### Istruzione:\nSpiega come funziona l'intelligenza artificiale in modo semplice.\n\n### Risposta:\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

💡 Try it in Colab: You can also run interactive chat directly in Google Colab using our Inference Notebook.

⚙️ Training Hyperparameters

The adapter was fine-tuned using the following hyperparameters:

• learning_rate: 0.0003
• train_batch_size: 4
• eval_batch_size: 8
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 4
• mixed_precision_training: Native AMP
• LoRA R: 8
• LoRA target modules: q_proj, v_proj

⚠️ Intended Uses & Safety Disclaimer

• Research Use Only: Consistent with Stanford Alpaca and Meta's original LLaMA license, Stambecco is intended strictly for academic and non-commercial research purposes. Commercial or production usage is strictly prohibited.
• Content Risk Warning: Like many instruction-tuned LLMs, model outputs may contain inaccuracies, hallucinations, biased language, or inappropriate content. Output should be critically evaluated by human reviewers.

📜 Citation

If you use Stambecco models or adapters in your research, please include the following citation:

@misc{stambecco,
  author       = {Michael Rottoli},
  title        = {Stambecco: Italian Instruction-following LLaMA Model},
  year         = {2023},
  publisher    = {GitHub},
  howpublished = {\url{https://github.com/mchl-labs/stambecco}}
}

If you use the underlying Stambecco datasets in your work, please cite them in your formal paper bibliography (rather than as footnotes) to ensure proper academic indexing:

@misc{stambecco_cleaned_2023,
  author       = {Michael Rottoli},
  title        = {Stambecco-Cleaned: Italian Instruction-Tuning Dataset},
  year         = {2023},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/mchl-labs/stambecco_data_it}}
}

@misc{stambecco_plus_2023,
  author       = {Michael Rottoli},
  title        = {Stambecco Plus: Premium Italian Instruction-Tuning Dataset},
  year         = {2023},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/mchl-labs/stambecco_data_plus_it}}
}

⚖️ License

This model adapter is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license. It is restricted to non-commercial academic research.