athenas-primal-8b

Athenas-Primal-8B

Athenas-Primal-8B is a LoRA fine-tune of Qwen3-8B (4-bit quantized) trained on synthetic agentic software engineering traces. The model is designed to investigate, diagnose, and fix production-grade bugs across multiple programming languages and technology stacks.

Model Description

• Developed by: kodjaoglanian
• Base model: unsloth/Qwen3-8B-unsloth-bnb-4bit
• License: Apache 2.0
• Finetuned from: Qwen3-8B (4-bit NF4, bitsandbytes)
• Adapter type: LoRA (rank 16, alpha 16)
• Max sequence length: 4096
• Languages: English, Portuguese

Training Methodology

The model was fine-tuned using Supervised Fine-Tuning (SFT) with LoRA adapters via the Unsloth framework and TRL's SFTTrainer.

LoRA Configuration

Training Hyperparameters

Infrastructure

• Platform: AWS SageMaker Studio
• GPU: NVIDIA A10G (24 GB)
• Framework: Unsloth + TRL
• Precision: bfloat16

Dataset

Training data: kodjaoglanian/fable-traces-mini

A synthetic dataset of multi-turn agentic conversations simulating real-world debugging sessions. Each trace follows an iterative investigation pattern:

1. A production incident is reported with symptoms, logs, and context
2. The assistant reasons through the problem using structured thinking
3. Tool calls are made to inspect and modify source code
4. Failures and compilation errors are encountered and resolved
5. The root cause is identified and a fix is applied

The dataset covers 29 scenarios across 8 languages (Python, Node.js, Rust, Go, Java, C++, TypeScript, C) with bug categories including memory leaks, race conditions, goroutine leaks, N+1 queries, thread pool starvation, SQL injection, use-after-free, and more.

Evaluation

HellaSwag

Fine-tuning yielded +8.8 percentage points improvement in acc_norm over the base model.

ARC Challenge

Context

Market figures are from full-precision models. The 4-bit quantization accounts for approximately 5–8pp of the gap.

Intended Use

• Agentic debugging: Multi-step investigation and resolution of production incidents
• Code review: Detection of concurrency bugs, resource leaks, and security vulnerabilities
• Cross-language engineering: Python, Node.js, Rust, Go, Java, C++, TypeScript, C

Limitations

• Trained on synthetic data — real-world bug patterns may differ
• 4-bit quantization introduces a measurable accuracy penalty versus full precision
• No RLHF or safety alignment was applied
• Not optimized for general conversation, creative writing, or mathematical reasoning
• Tool calls in training are simulated; the model does not execute code

Quick Start

from unsloth import FastLanguageModel
from transformers import TextStreamer

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Kodjaoglanian/athenas-primal-8b",
    max_seq_length=4096,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "<your system prompt>"},
    {"role": "user", "content": "Describe your bug or engineering problem here."},
]

inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt",
).to("cuda")

_ = model.generate(
    input_ids=inputs,
    streamer=TextStreamer(tokenizer, skip_prompt=True),
    max_new_tokens=2048,
    temperature=0.7,
    top_p=0.9,
)

Citation

@misc{athenas_primal_8b,
  author = {kodjaoglanian},
  title = {Athenas-Primal-8B: Agentic Software Engineering via LoRA Fine-Tuning of Qwen3-8B},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Kodjaoglanian/athenas-primal-8b}
}

Acknowledgments

• Unsloth — efficient fine-tuning framework
• Qwen Team — base model
• TRL — SFTTrainer