Atem-v1-1.5B

<p align="center"> <img src="Logo.png" width="300" alt="Atem Logo"/> </p> <h1 align="center">Atem v1</h1> <p align="center"> <em>Ancient logic. Modern intelligence.</em> </p> <p align="center"> A 1.5B reasoning model trained via multi-source knowledge distillation from frontier teacher models. </p> <p align="center"> <img src="https://img.shields.io/badge/Base-Qwen2.5--1.5B--Instruct-blue" alt="Base Model"/> <img src="https://img.shields.io/badge/Method-LoRA%20SFT-purple" alt="Method"/> <img src="https://img.shields.io/badge/Parameters-1.5B-orange" alt="Parameters"/> <img src="https://img.shields.io/badge/License-Apache%202.0-green" alt="License"/> </p>

---

Overview

Atem is a 1.5B parameter reasoning model built via supervised fine-tuning on a curated corpus of approximately 115,000 examples distilled from multiple frontier teacher models. Starting from Qwen2.5-1.5B-Instruct, Atem was trained using LoRA to preserve base model capabilities while improving performance on reasoning, mathematics, and coding tasks.

This is Stage 1 of a planned multi-stage training series. Stage 1 focuses on establishing strong general reasoning across domains. Stage 2 layers chain-of-thought thinking traces on top of this foundation. Stage 2 is Atem-Wisdom which builds on this foundation by adding explicit chain-of-thought reasoning — the model works through problems inside <think> tags before producing its final answer.

---

Model Details

---

Intended Use

Atem is designed for open-ended reasoning tasks where structured, accurate thinking adds value:

• Code explanation, implementation, and debugging
• Mathematical problem solving with working shown
• Analytical reasoning and hypothesis evaluation
• Concept explanation and comparative analysis
• Logic, argument, and fallacy identification

Atem is not designed for retrieval-heavy factual lookup, real-time information, or tasks requiring broad knowledge breadth beyond its training domains.

---

Training Data

Atem was trained on a corpus assembled from eleven sources, combining domain-specific generated datasets and publicly available distillation datasets from frontier models. All outputs containing <think> reasoning traces were stripped to clean final responses for Stage 1 training.

The QWENMillenium-SF, Phi4Millennium-SF, and MistralMillenium-SF datasets were generated specifically for this project via batched inference on Colab A100. OpenThoughts-114k-math was filtered to verified correct solutions only before sampling.

---

Training Configuration

# Key hyperparameters
lora_r            = 32
lora_alpha        = 64
lora_dropout      = 0.05
max_seq_length    = 4096
learning_rate     = 2e-4
lr_scheduler      = 'cosine'
warmup_ratio      = 0.05
batch_size        = 8
grad_accumulation = 8       # effective batch size: 64
num_epochs        = 1
dtype             = bfloat16
load_in_4bit      = True    # during training

Training used Unsloth with train_on_responses_only masking, ensuring loss was computed exclusively on assistant response tokens. A three-part pre-training validation was run before training: chat template replacement verification, think tag strip confirmation, and mask sanity check.

After training, LoRA adapters were merged into the base weights and exported as a full merged model.

Loss curve:

Validation loss converged at 0.890, with a final train/val gap of 0.050 — indicating no overfitting over the single epoch.

---

Evaluation

Benchmark Results

Evaluated against Qwen2.5-1.5B-Instruct (base model) using lm-evaluation-harness with identical conditions: 4-bit inference, batch size 16, zero-shot strict evaluation.

The GSM8K result is the primary finding. A +30 percentage point improvement on grade school mathematics reflects the targeted training on verified correct mathematical reasoning examples from multiple frontier teacher models.

The HellaSwag regression of 2.4% is within normal benchmark variance and represents a significant improvement over a prior exploratory training run using full fine-tune, which produced a 16.2% regression on the same benchmark. LoRA preserved base model commonsense capabilities as intended.

Comparison vs Qwen2.5-7B-Instruct

To contextualise the GSM8K result, Atem was benchmarked against Qwen2.5-7B-Instruct under the same zero-shot strict evaluation conditions.

At baseline, the 1.5B model sits 51.9 points below the 7B. After training, Atem sits 21.9 points below — closing approximately 58% of the capability gap between 1.5B and 7B on mathematical reasoning. Atem achieves 71% of Qwen2.5-7B's GSM8K performance at 22% of its parameter count.

Note: Official Qwen2.5-7B-Instruct scores (91.6% GSM8K) use 4-shot chain-of-thought prompting. The 74.9% figure above reflects the same zero-shot strict evaluation format used for Atem, ensuring a fair direct comparison.

Qualitative Evaluation

Atem was evaluated against Qwen2.5-1.5B-Instruct across 30 domain-representative questions using matched system prompts, ensuring differences in output reflect trained capability rather than prompt engineering.

---

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "EphAsad/Atem-v1-1.5B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {
        "role": "user",
        "content": "Write a Python function that checks whether a number is prime."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids=inputs,
        max_new_tokens=1000,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        do_sample=True,
    )

response = tokenizer.decode(
    output[0][inputs.shape[1]:],
    skip_special_tokens=True
)
print(response)

Unsloth (faster inference)

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="EphAsad/Atem-v1-1.5B",
    max_seq_length=4096,
    dtype=torch.bfloat16,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

messages = [
    {
        "role": "user",
        "content": "Explain the difference between a stack and a queue, with examples."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

with torch.no_grad():
    output = model.generate(
        input_ids=inputs,
        max_new_tokens=1000,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
    )

print(tokenizer.decode(
    output[0][inputs.shape[1]:],
    skip_special_tokens=True
))

Ollama

# Recommended — best speed/quality balance
ollama run hf.co/EphAsad/Atem-v1-1.5B:Q4_K_M

# Higher quality
ollama run hf.co/EphAsad/Atem-v1-1.5B:Q5_K_M

# Near-lossless
ollama run hf.co/EphAsad/Atem-v1-1.5B:Q8_0

llama.cpp

llama-server -hf EphAsad/Atem-v1-1.5B:Q4_K_M

System Prompt

Atem's identity is baked into the chat template and activates automatically when no system message is provided. For manual override:

You are Atem, a precise and analytical reasoning assistant. You approach 
every problem methodically — identifying core concepts, reasoning step by 
step, and arriving at well-supported conclusions. You show your thinking 
clearly and are thorough, direct, and intellectually honest.

Available Files

---

Known Limitations

No thinking traces (Stage 1 by design). Think tags were stripped from all training data for Stage 1. The model does not produce extended <think> reasoning traces. Stage 2 training will layer this capability on top of the Stage 1 foundation.

Mathematical precision on complex problems. On multi-step calculations, the model may make arithmetic slips in intermediate steps while arriving at a structurally correct approach. Answers to high-stakes mathematical problems should be independently verified.

HellaSwag regression. A 2.4% regression on HellaSwag commonsense completion is observed. This is minor and substantially better than the 16.2% regression produced by the earlier exploratory full fine-tune run, confirming that LoRA preserved base commonsense capability effectively.

---

Roadmap

Atem v1 establishes the Stage 1 foundation. Planned next steps:

• Stage 2: LoRA SFT on curated chain-of-thought data to add thinking trace capability — using Complex_CoT, inverted_reasoning, and reasoning trace columns held out from Stage 1 training
• Extended benchmarks: MMLU, BBH, IFEval, WinoGrande, MBPP post-Stage 2
• Atem v2: Expanded corpus, further domain coverage

---

Citation

@misc{atem_v1_2026,
  author       = {Asad, Zain},
  title        = {Atem v1: A 1.5B Reasoning Model via 
                  Multi-Source Knowledge Distillation},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/EphAsad/Atem-v1-1.5B}},
}

---

Support

If you find this model useful for your research or projects, you can support further development of my datasets and models here:
☕ ko-fi.com/ephraim123

---

License

Released under the Apache 2.0 License, consistent with the base model Qwen2.5-1.5B-Instruct.

---

<p align="center"> Built independently by <a href="https://huggingface.co/EphAsad">EphAsad</a> </p>