Athenas-Symbiote-9B

Athenas-Symbiote-9B

<div align="center"> <img src="image.png" alt="Athenas-Symbiote Logo" width="300"/> </div>

Model Details

Athenas-Symbiote-9B is a large language model (LLM) fine-tuned for advanced reasoning, complex text interpretation, and legal/academic domain expertise, primarily optimized for Brazilian Portuguese (PT-BR).

Built upon the Qwen3.5-9B architecture, this model leverages Parameter-Efficient Fine-Tuning (PEFT) to adapt its robust multilingual baseline specifically to the nuances, terminology, and logical structures required by the Brazilian legal and educational systems.

📄 Technical Report: Available on Zenodo — full methodology, evaluation protocol, and per-edition analysis.

Model Description

• Developed by: Bruno Kodjaoglanian Cardoso Tulux (Independent AI Researcher)
• Model Type: Causal Language Model (Transformer)
• Base Model: unsloth/Qwen3.5-9B-Base
• Parameter Count: 9 Billion
• Context Length: 32,768 tokens
• Language(s) (NLP): Portuguese (Primary), English
• License: Apache 2.0
• Finetuning Approach: LoRA (Low-Rank Adaptation)

---

Evaluation & Benchmarks

The model was comprehensively evaluated using the lm-evaluation-harness framework. Inference was conducted on high-performance infrastructure (1x NVIDIA H100 80GB) utilizing native bfloat16 precision to ensure metric fidelity and prevent quantization degradation.

[Insert your consolidated performance bar/radar chart here]

National Benchmarks (Brazil)

Evaluations targeting the Brazilian cultural, legal, and educational framework.

Benchmark Datasets:

• OAB Exams: maritaca-ai/oab-bench
• ENEM Challenge: maritaca-ai/enem

ENEM Challenge — Detailed Breakdown by Exam

OAB Exams — Detailed Breakdown by Exam

Global Benchmarks (Translated / Multilingual)

Frontier evaluations assessing general intelligence and mathematical reasoning.

Note: Global benchmark evaluations are pending. Scores will be updated upon completion.

---

How to Get Started with the Model

Use the code below to get started with the model. As this is a base model fine-tuned with LoRA, there is no built-in chat template. The ChatML format is recommended for optimal inference with instruction-style prompts.

Requirements

pip install transformers accelerate bitsandbytes

Python Inference

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Kodjaoglanian/Athenas-Symbiote-9B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

messages = [
    {"role": "system", "content": "Você é a Athenas, uma assistente virtual altamente inteligente e precisa."},
    {"role": "user", "content": "Explique a diferença entre dolo e culpa no código penal brasileiro."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs, 
    max_new_tokens=512, 
    temperature=0.3,
    do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Training Details

Training Data

The model was fine-tuned on the ClassiCC-Corpus/ClassiCC-PT dataset, consumed in streaming mode with a strict quality filter (edu_score > 0.8). This heuristic ensured the model ingested only high-rigor educational, literary, and academic content. The training corpus includes:

• Brazilian jurisprudence, legislation, and legal proceedings.
• Academic literature and technical documentation in Portuguese.
• Chain-of-Thought (CoT) reasoning structures.

Data Pipeline: Streaming → edu_score > 0.8 filter → EOS concatenation → Sequence packing (packing=True)

Training Pipeline

graph TD
    A["Load Qwen3.5-9B-Base<br/>(4-bit, Unsloth)"] --> B["Configure PEFT<br/>LoRA r=32, α=16, rsLoRA"]
    B --> C["Dataset: ClassiCC-PT<br/>(streaming, edu_score > 0.8)"]
    C --> D["SFTTrainer<br/>1,200 steps, LR 5e-5<br/>adamw_8bit, packing=True"]
    D --> E{"Training OK?"}
    E -- No --> F["Auto-retry<br/>(5x, resume checkpoint)"]
    F --> D
    E -- Yes --> G["Push to HF Hub<br/>(Safetensors + Tokenizer)"]
    G --> H["Evaluation<br/>lm-eval-harness<br/>H100, BF16, 3-shot"]

    style A fill:#1B3A5C,color:#fff
    style B fill:#1B3A5C,color:#fff
    style C fill:#1B3A5C,color:#fff
    style D fill:#1B3A5C,color:#fff
    style F fill:#C5A059,color:#fff
    style G fill:#C5A059,color:#fff
    style H fill:#C5A059,color:#fff

Training: Kaggle T4 (16GB) | Evaluation: Lightning AI H100 (80GB)

Training Procedure

The fine-tuning process prioritized structural knowledge retention and computational efficiency.

• Hardware: 1x NVIDIA T4 (16GB VRAM) via Kaggle
• Quantization: 4-bit (NF4 via bitsandbytes)
• Precision: FP16 (T4 does not support BF16)
• Framework: Unsloth + TRL + PEFT + bitsandbytes
• Optimization: LoRA adapters applied to all linear projections
• LoRA Rank: 32
• LoRA Alpha: 16
• Scaling Strategy: rsLoRA (use_rslora=True)
• LoRA Dropout: 0.0
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning Rate: 5e-5
• LR Scheduler: Linear
• Warmup Steps: 50
• Training Steps: 1,200 (Dataset via Streaming)
• Batch Size: 1 (Effective Batch Size: 8 via Gradient Accumulation)
• Optimizer: adamw_8bit
• Weight Decay: 0.01
• Packing: True
• Gradient Checkpointing: Unsloth
• Seed: 3407
• Max Sequence Length (Training): 512 tokens

Evaluation Hardware

• Infrastructure: Lightning AI platform
• GPU: 1x NVIDIA H100 (80GB VRAM)
• Precision: bfloat16 (native, no quantization)
• Framework: lm-evaluation-harness (EleutherAI)
• Few-Shot: 3-shot
• Bootstrap Iterations: 100,000
• Execution: Sequential evaluation to isolate memory overhead.

---

Intended Use & Limitations

Intended Use

• Primary Use Case: Research and application in Natural Language Processing (NLP) specifically for the Portuguese language.
• Secondary Use Case: Automation of complex text analysis, summarization, and logical reasoning within legal and academic contexts (assisted by human review).

Out-of-Scope Use

The model is not designed to operate autonomously in high-stakes environments. It should not be used for:

• Automated legal counsel or medical diagnosis without human-in-the-loop validation.
• Generation of deceptive, malicious, or highly biased content.

Bias, Risks, and Limitations

Like all statistical language models, Athenas-Symbiote-9B is susceptible to generating hallucinations—plausible-sounding but factually incorrect information. Furthermore, the model may inherit and amplify biases present in its pre-training data and fine-tuning corpus. Users must exercise critical judgment and implement rigorous verification protocols before utilizing the model's outputs in production environments.

---

Citation

If you use this model in your research, please cite it as follows:

@misc{athenas_symbiote_9b,
  author       = {Bruno Kodjaoglanian Cardoso Tulux},
  title        = {Athenas-Symbiote-9B: A Parameter-Efficient Fine-Tuned LLM for Brazilian Portuguese Legal and Academic Domains},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.21418168},
  url          = {https://zenodo.org/records/21418168},
  note         = {Model weights: https://huggingface.co/Kodjaoglanian/Athenas-Symbiote-9B}
}