qwen3-0.6b-weather-geo-M

Qwen3-0.6B-Weather-Geo-M

Model Description

Model M is the second member of the specialist model family built in Chapter 13 of the book "Rearchitecting LLMs", and the first step of a cascaded pruning-and-distillation pipeline. It is derived from T, the LoRA fine-tuned weather/geo tool-calling specialist, through depth pruning → width pruning → knowledge distillation, in that order.

Depth is ranked on the intact teacher T, then width pruning is applied to the depth-pruned model. Both pruning stages share the same calibration prompts, while the knowledge-distillation teacher remains the unpruned T — the only model in the family that has never lost capability, providing the cleanest supervision signal.

Model M will, in turn, serve as the pruning source and distillation teacher for Model S, the next and smallest member of the family.

• Book: Rearchitecting LLMs
• Framework: OptiPFair
• Technique: Depth Pruning + Width Pruning (GLU/MLP) + Knowledge Distillation
• Chapter: Chapter 13 — Building a Specialist Model Family

---

Performance Metrics

Evaluation uses exact-match and tolerant-match (functionally equivalent values — superset city names, coordinates within 0.05°) on the same 54-example held-out test set used for T, across every stage of the cascade.

Cascade Results (held-out test set, n=54)

Pruning alone — before distillation — collapses accuracy (96.3% → 48.1% exact match), mostly through the model failing to emit a tool call at all (17 of 54 cases). Knowledge distillation from T fully recovers this gap and slightly exceeds the teacher's exact-match score on this test set.

Seed Variance (knowledge distillation, 3 runs)

Seed 42 was fixed as the canonical Model M run before training, to avoid selecting the checkpoint based on test-set performance. The spread across seeds (92.6%–98.1%) corresponds to a 4-example swing out of 54, which is within the range expected from batch-order noise at this dataset size.

Bootstrap 95% Confidence Intervals (exact match)

Per-Function Accuracy (Model M)

M's single failure on the test set is not a new error introduced by pruning or distillation — checking against T's failures directly, M has zero failures that T did not also make (both models share the same 2 error-prone cases, with M recovering one of them).

---

Architecture Details

• Layers removed (depth pruning): 3 — original indices [13, 23, 25] in T, selected by ascending cosine-distance importance with protection for the first 4 and last 2 layers, and no adjacent removals
• Width pruning: GLU/MLP pruning via MAW (hybrid, data-driven Mean Absolute Weight) neuron selection, targeting a 225% expansion rate (divisor 128) — 3072 → 2304 per layer, applied uniformly across all 25 surviving layers
• Non-embedding parameter reduction vs. T: 17.8%
• Layer index mapping: because depth pruning renumbers layers, a depth_index_map (new index → original T index) is preserved and handed off for use by the next notebook in the cascade

---

Training Procedure

Dataset

Same curated and deduplicated split used for T: Salesforce/xlam-function-calling-60k, filtered to 3 domain functions, 177 examples after deduplication, split 123 train / 54 test (stratified by function, test_size=0.30, random_state=42) — reused across every stage of the family for comparability.

Calibration (shared by depth ranking and width pruning)

• Source: Training-set prompts (system + tools + user query) — the same context the model conditions on before generating a tool call
• Batch size: 8
• Max length: 1152 tokens (covers the full range of calibration prompt lengths, no truncation)

Knowledge Distillation

• Teacher: T (unpruned specialist), frozen
• Loss: α·Task CE + β·KL(student‖teacher logits), feature losses disabled
  • Task Loss Weight (α): 0.3
  • Logits Loss Weight (β): 0.7
  • Feature Alignment (γ): 0.0
  • Feature Dynamics (δ): 0.0
  • Temperature: 1.1
  • Skew Interpolation Factor: 0.15
• Labels: Completion-only (prompt and padding masked to -100), matching the masking used to train T
• Epochs: 12
• Learning Rate: 3e-5
• Batch Size: 4 (effective: 16 with 4-step gradient accumulation)
• Max Sequence Length: 1152
• Seeds evaluated: 43, 44, 42 (canonical, fixed in advance)

Hardware

• GPU: NVIDIA A100-SXM4-40GB

---

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oopere/qwen3-0.6b-weather-geo-M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "What's the weather in Barcelona?"}]
tools = [...]  # weather/geo tool schemas — see the book's companion notebook

prompt = tokenizer.apply_chat_template(
    messages, tools=tools, tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

---

Limitations & Intended Use

Intended Use

This is an educational model created as part of the Hands-on Lab in Chapter 13 of "Rearchitecting LLMs". It demonstrates:

• A cascaded depth-pruning + width-pruning + knowledge-distillation pipeline for compressing a fine-tuned specialist
• How aggressive structural pruning collapses task performance, and how targeted distillation from the unpruned teacher recovers it
• Serving as an intermediate step — and the next teacher — in a multi-stage model-family cascade (T → M → S)

Not intended for production use. This model serves as a learning artifact within a three-function toy domain, and as an intermediate checkpoint for readers following the full pruning cascade to Model S.

Limitations

• Domain Coverage: Restricted to 3 functions (get_ip_zipcode, get_city_from_zipcode, local_weather_api); no general-purpose tool-calling capability was evaluated
• Training Data: Small, curated dataset (123 training examples); not representative of production-scale function-calling data
• Evaluation set size: 54 test examples — per-model differences of 1-2 examples are within the reported bootstrap confidence intervals and should not be over-interpreted
• Language: English only

---

Citation

If you use this model or the techniques described in your research or projects, please cite:

Book

@book{martra2026rearchitecting,
  author    = {Pere Martra},
  title     = {Rearchitecting LLMs: Structural techniques for efficient models},
  publisher = {Manning Publications},
  year      = {2026},
  url       = {https://hubs.la/Q040tvtp0}
}

Framework

@software{optipfair2024,
  author = {Pere Martra},
  title  = {OptiPFair: Structural Pruning and Bias Analysis for LLMs},
  year   = {2024},
  url    = {https://github.com/peremartra/optipfair}
}

---

Acknowledgments

This model was created following the methodologies taught in "Rearchitecting LLMs" (Manning Publications, 2026). Special thanks to the Manning editorial team and the open-source community behind Hugging Face Transformers and PyTorch.