Qwen3-1.7BSFTLoRA

Qwen3-1.7B SFT LoRA

A supervised fine-tuned (SFT) version of Qwen3-1.7B trained using LoRA (Low-Rank Adaptation) with the TRL SFTTrainer. This model is designed for instruction-following tasks while keeping the number of trainable parameters low through parameter-efficient fine-tuning (PEFT).

Model Details

---

Model Architecture

The model is based on Qwen3-1.7B with the following architecture:

• Architecture: Qwen3ForCausalLM
• Hidden Size: 2048
• Intermediate Size: 6144
• Layers: 28
• Attention Heads: 16
• Key/Value Heads: 8
• Head Dimension: 128
• Activation Function: SiLU
• Context Length: 40,960 tokens
• Vocabulary Size: 151,936
• Precision: bfloat16
• RMSNorm Epsilon: 1e-6
• RoPE Theta: 1,000,000
• Attention Dropout: 0.0

---

Fine-tuning Method

The model was fine-tuned using Parameter-Efficient Fine-Tuning (PEFT) with LoRA.

Training stack:

• 🤗 Transformers
• 🤗 Datasets
• 🤗 PEFT
• 🤗 TRL
• PyTorch

Libraries

from datasets import load_dataset

from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
)

from peft import (
    LoraConfig,
    get_peft_model,
)

from trl import (
    SFTTrainer,
    SFTConfig,
)

---

Training Pipeline

1. Load the dataset using Hugging Face Datasets.
2. Load the Qwen3-1.7B tokenizer.
3. Load the Qwen3-1.7B base model.
4. Configure LoRA adapters.
5. Apply PEFT to the base model.
6. Configure SFTTrainer.
7. Fine-tune using supervised instruction data.
8. Save LoRA adapters or merge them into the base model.

---

Intended Uses

This model is intended for:

• Instruction Following
• Question Answering
• Chat Applications
• Text Generation
• Code Assistance
• Educational Tasks
• Research
• AI Assistants

---

Limitations

• May generate incorrect or hallucinated information.
• Responses depend heavily on prompt quality.
• Not intended for medical, legal, or financial advice without human verification.
• Performance depends on the quality and diversity of the fine-tuning dataset.

---

Example Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "ankitkushwaha90/<your-model-name>"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {
        "role": "user",
        "content": "Explain what LoRA fine-tuning is."
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.95,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Training Framework

• Transformers
• TRL
• PEFT
• Datasets
• PyTorch

---

Citation

If you use this model in your research or applications, please cite:

@software{qwen3_1_7b_sft_lora,
  author = {Ankit Kushwaha},
  title = {Qwen3-1.7B SFT LoRA},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/ankitkushwaha90}
}

---

Acknowledgements

This model is built upon the excellent Qwen3-1.7B base model and fine-tuned using the Hugging Face ecosystem, including:

• Transformers
• TRL
• PEFT
• Datasets
• PyTorch

Special thanks to the Qwen and Hugging Face teams for providing open-source tooling and models.