qwen3-0.6b-weather-geo-S

Qwen3-0.6B-Weather-Geo-S

Model Description

Model S is the third and smallest member of the specialist model family built in Chapter 13 of the book "Rearchitecting LLMs", and the second step of the cascaded pruning-and-distillation pipeline. It is derived from M through depth pruning → width pruning → knowledge distillation, mirroring the procedure used to derive M from the original specialist T — with layer importance and activation statistics recomputed on M rather than reused from T.

In this run, the knowledge-distillation teacher is M itself (the notebook supports switching to T as an alternative teacher via a single configuration flag). T remains available as the tokenizer source and as a reference point for tracking whether errors compound down the cascade.

• Book: Rearchitecting LLMs
• Framework: OptiPFair
• Technique: Depth Pruning + Width Pruning (GLU/MLP) + Knowledge Distillation
• Chapter: Chapter 13 — Building a Specialist Model Family

---

Performance Metrics

Evaluation uses the same exact-match / tolerant-match protocol and 54-example held-out test set as T and M.

Cascade Results (held-out test set, n=54)

Pruning collapses accuracy far more sharply here than at the T→M stage: the intermediate pruned model mostly selects the wrong function (29-31 cases) rather than failing to emit a call at all. Distillation from M recovers this almost completely — S lands at 94.4% exact match, within 4 points of M's 98.1%.

Seed Variance (knowledge distillation, 3 runs)

Seed 42 was fixed as the canonical checkpoint before training. The spread across seeds is 2 of 54 examples.

Bootstrap 95% Confidence Intervals (exact match)

Per-Function Accuracy (Model S)

local_weather_api remains the function where S's compression cost concentrates, though less severely than in earlier runs. Of the 3 test failures, 2 are extra relative to M — both on this function. One drops the num_of_days argument while getting everything else right; the other correctly extracts every argument except lang, generating "fr" instead of "es" for a query stated in Spanish — a plausible confusion given the two queries appear close together in phrasing.

---

Architecture Details

• Layers removed (depth pruning): 3 — indices [12, 14, 21] in M's numbering, selected by ascending cosine-distance importance with protection for the first 4 and last 2 layers, no adjacent removals
• Width pruning: GLU/MLP pruning via MAW (hybrid, data-driven) neuron selection; expansion_rate=175 is resolved relative to the model being pruned (M's 2304), landing at an absolute 175% of hidden size (1792), confirmed after pruning rather than assumed
• Non-embedding parameter reduction vs. M: 15.3%
• Layer index mapping: an s_to_m_index_map is recorded in the handoff file. The s_to_t_index_map (S index → original T index) is not available for this run — it depends on M's handoff file, which was not present when this notebook ran, so tracing an S layer back to T requires composing the maps by hand from the two notebooks' outputs

---

Training Procedure

Dataset

Same curated and deduplicated split reused across the whole family: Salesforce/xlam-function-calling-60k, 177 examples after deduplication, split 123 train / 54 test (stratified, test_size=0.30, random_state=42).

Calibration

• Source: Training-set prompts, computed fresh on M (not reused from T's calibration) since M is the model being pruned
• Batch size: 8, max length: 1152 tokens

Knowledge Distillation

• Teacher: M, frozen (T available as an alternative via a KD_TEACHER config flag)
• Loss: α·Task CE + β·KL(student‖teacher logits) + γ·Feature Alignment
  • Task Loss Weight (α): 0.4
  • Logits Loss Weight (β): 0.5
  • Feature Alignment (γ): 0.1
  • Feature Dynamics (δ): 0.0
  • Temperature: 1.5
  • Skew Interpolation Factor: 0.15
• Labels: Completion-only (prompt and padding masked to -100)
• Epochs: 16
• Learning Rate: 3e-5
• Batch Size: 4 (effective: 16 with 4-step gradient accumulation)
• Max Sequence Length: 1152
• Seeds evaluated: 43, 44, 42 (canonical, fixed in advance)

Note on training length: the logits (KL) loss goes slightly negative from epoch 5 onward, which the notebook flags as the KL term collapsing into numerical noise once the student has converged close to the teacher — a signal that further epochs mainly refine the task loss rather than the distillation signal itself.

Hardware

• GPU: NVIDIA A100-SXM4-40GB

---

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oopere/qwen3-0.6b-weather-geo-S"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "What's the weather in Barcelona?"}]
tools = [...]  # weather/geo tool schemas — see the book's companion notebook

prompt = tokenizer.apply_chat_template(
    messages, tools=tools, tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

---

Limitations & Intended Use

Intended Use

This is an educational model created as part of the Hands-on Lab in Chapter 13 of "Rearchitecting LLMs". It demonstrates:

• The tail end of a two-stage pruning-and-distillation cascade (T → M → S), and how accuracy degrades and recovers as compression compounds
• Distilling from an already-pruned teacher (M) rather than the original specialist
• The smallest, most compressed model in the family, illustrating the practical trade-off curve between size and task accuracy

Not intended for production use. This model serves as a learning artifact within a three-function toy domain and as the final checkpoint of the book's pruning cascade example.

Limitations

• Domain Coverage: Restricted to 3 functions (get_ip_zipcode, get_city_from_zipcode, local_weather_api); no general-purpose tool-calling capability was evaluated
• Weakest on local_weather_api: accuracy on this function is 8/10, with both errors concentrated in argument extraction (a missing num_of_days, a confused lang value) rather than function selection
• Training Data: Small, curated dataset (123 training examples); not representative of production-scale function-calling data
• Evaluation set size: 54 test examples — differences of 1-2 examples are within the reported bootstrap confidence intervals
• Language: English only

---

Citation

If you use this model or the techniques described in your research or projects, please cite:

Book

@book{martra2026rearchitecting,
  author    = {Pere Martra},
  title     = {Rearchitecting LLMs: Structural techniques for efficient models},
  publisher = {Manning Publications},
  year      = {2026},
  url       = {https://hubs.la/Q040tvtp0}
}

Framework

@software{optipfair2024,
  author = {Pere Martra},
  title  = {OptiPFair: Structural Pruning and Bias Analysis for LLMs},
  year   = {2024},
  url    = {https://github.com/peremartra/optipfair}
}

---

Acknowledgments

This model was created following the methodologies taught in "Rearchitecting LLMs" (Manning Publications, 2026). Special thanks to the Manning editorial team and the open-source community behind Hugging Face Transformers and PyTorch.