societas-router-lenv3-4b

societas-router-lenv3-4b

A per-call LLM routing classifier — a full-parameter fine-tune of Qwen/Qwen3-4B. At each step of an agent's trajectory it reads a read-only snapshot of the trajectory-so-far and predicts the capability tier the next model call needs, emitting a single JSON verdict {"tier": "...", "reason": "..."} with tier ∈ {Routine, Medium, Advanced}.

Results (held-out test split, 251 examples)

• Accuracy: 60.96% (153/251), parse_fail = 0, no think-token leakage. (chance = 33%)
• Per-tier recall: Routine 60.7% · Medium 52.4% · Advanced 69.9%.

Usage note (important)

Trained with the Qwen3 chat template at enable_thinking=False (the empty <think></think> block is in the prompt; the JSON verdict is the target). Serve it the same way — render prompts with enable_thinking=False.

Training procedure

Full-parameter SFT via LLaMA-Factory, bf16, DeepSpeed ZeRO-3 on 2× A800-80G.

• learning_rate: 2e-05, cosine schedule, warmup_ratio 0.03
• per_device_train_batch_size: 1, gradient_accumulation_steps: 8, total_train_batch_size: 16
• num_epochs: 3.0, cutoff_len (max context): 16384, packing: off
• optimizer: AdamW (torch), betas=(0.9,0.999), eps=1e-08

Framework versions

• Transformers 4.53.x / Pytorch 2.5.1+cu121 / Datasets / Tokenizers