Atlas-Frontier-Distill-3B

🧬 Atlas-Frontier-Distill-3B

An experimental 3B coding model distilled from frontier model traces (Kimi-K3, GPT-5.6, Fable-5).

<p align="center"> <img src="./banner.png" width="90%"> </p>

---

Model Description

Atlas-Frontier-Distill-3B is a specialized coding assistant built on top of Qwen2.5-Coder-3B-Instruct. The project explores whether the coding and debugging capabilities of large frontier models can be transferred into an efficient 3B parameter model suitable for local and edge deployment.

The model was trained using QLoRA on 15,746 carefully curated coding conversations extracted from multiple frontier teacher models. Rather than imitating every response, the dataset was aggressively filtered to preserve only successful reasoning traces and high-quality coding solutions.

---

Distillation Experiment

This model is part of an empirical research project by Pluto AI Research Lab investigating knowledge distillation through curated execution traces.

During dataset construction, raw frontier model outputs contained a significant amount of unusable samples including:

• Empty tool calls
• Failed API responses
• Placeholder outputs
• "Needs human" responses
• Abandoned reasoning chains

Over 20,375 low-quality conversations were removed, leaving 15,746 high-quality coding traces for training.

The objective was to ensure the student model learns productive coding behavior instead of failure patterns.

---

[!NOTE]

🔬 Behavioral Delta (via llm-diff)

Behavioral regression analysis comparing the base model against Atlas-Frontier-Distill-3B using Pluto AI's open-source llm-diff behavioral evaluation tool.

llm-diff ollama/qwen2.5-coder:3b ollama/atlas-frontier-distill-3b --backend ollama

Key Takeaway

Want to audit your own model upgrades? Install llm-diff today: pip install pluto-llm-diff

Dataset Processing

The training corpus underwent a dedicated preprocessing pipeline:

• ✅ Schema unification from multiple chat formats into ChatML
• ✅ Automatic extraction of user/assistant conversations
• ✅ Removal of invalid tool outputs
• ✅ Deduplication
• ✅ Conversation validation
• ✅ Coding-only filtering
• ✅ High-quality reasoning preservation

Final training corpus:

• 15,746 conversations
• ~36 MB cleaned dataset
• 100% coding & debugging focused

Dataset:

Siddh07ETH/Atlas-Frontier-Model-Traces

---

Training Details

---

Evaluation

<p align="center"> <img src="./benchmark.png" width="90%"> </p>

🧪 Benchmark Results (HumanEval Pass@1)

Evaluated on a random subset of 20 complex HumanEval problems using greedy decoding (temperature=0.0) to test pure reasoning capabilities.

Key Takeaway: The fine-tuning process successfully improved the model's ability to solve complex edge-case logic problems (such as Problem #10 in our evaluation subset) while maintaining zero regression on tasks the base model already solved correctly. This validates the distillation of high-quality frontier reasoning traces into the 3B parameter space.

---

Intended Use

Atlas-Frontier-Distill-3B is intended for:

• Local coding assistants
• IDE integration
• Autonomous debugging agents
• Python code generation
• GGUF deployment
• Ollama deployment
• Edge inference
• Software engineering research

---

Limitations

• Optimized specifically for coding and debugging tasks.
• May perform worse than the base model on open-domain conversation.
• Not trained for creative writing or general-purpose chat.
• Distillation quality depends entirely on the quality of teacher traces.

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Siddh07ETH/Atlas-Frontier-Distill-3B",
    torch_dtype=torch.float16,
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained(
    "Siddh07ETH/Atlas-Frontier-Distill-3B"
)

messages = [
    {
        "role": "user",
        "content": "Write a Python function to connect to a PostgreSQL database."
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.2,
        do_sample=True,
    )

print(
    tokenizer.decode(
        outputs[0][inputs.input_ids.shape[1]:],
        skip_special_tokens=True,
    )
)

---

Citation

If you use this model or the accompanying dataset in your research, please cite:

@misc{atlasfrontierdistill3b,
  author    = {Siddharth N.R.},
  title     = {Atlas-Frontier-Distill-3B: Distilling Frontier Model Traces into Edge-Deployable LLMs},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/Siddh07ETH/Atlas-Frontier-Distill-3B}
}

---

Author

Siddharth N.R.

Pluto AI Research Lab

---

License

Apache-2.0