FF_3.13

FF_3.13

Champion model of the FF-LLM line. A 2.02B GPT-2 architecture model fine-tuned through a multi-stage pipeline (pretraining → SFT → distillation → surgical fine-tuning → knowledge repair) for general-purpose factual question answering.

---

Model overview

---

Benchmark performance

MMLU (Massive Multitask Language Understanding)

Evaluated with lm-eval-harness v0.4.11, greedy decoding.

Macro-domain breakdown (MMLU full)

106-bench (custom factual benchmark)

Custom 106-prompt benchmark with strict TRUTH-list scoring:

Improvement vs precursors

---

Training pipeline — chronological view

The model went through 7 distinct stages of training. Below is the complete history.

Stage 1 — Pretraining

Architecture chosen: GPT-2 (2.02B parameters), trained from scratch on a curated multi-source web + encyclopedic + educational corpus.

Pretraining data composition

The pretraining corpus was assembled from 8 distinct sources, organized in two training modules (M1 BASE / Extra and M2 BASE / Extra) with quality-tiered weighting.

Mix proportions (approximate)

• 60–65% FineWeb (various slices: general, 10BT, EDU, EDU extended)
• 15–20% C4 EN
• 5–10% Wikipedia EN
• 5–10% Web Clean custom
• ~5% News crawl

This mix prioritizes educational content (FineWeb EDU = high weight) and high-quality web text, with encyclopedic and journalistic sources providing factual grounding.

Stage 2 — Supervised Fine-Tuning (SFT)

Objective: general instruction-following + factual knowledge alignment.

Data sources (~860K total examples):

• OpenHermes (cleaned)
• UltraChat (cleaned)
• WildChat (cleaned)
• Numina (math reasoning)
• OpenThoughts (chain-of-thought)
• Eurus (multi-task)

Composition: ~760K core + 100K augmentation examples. Sharded under s3://ff-llm-datasets/sft/shards_v2/.

Stage 3 — Direct Preference Optimization (DPO) — REJECTED

Two DPO experiments were attempted and discarded:

Lesson: DPO consistently caused MMLU regression (~-3pp) regardless of hyperparameters. Not used in the final model.

Stage 4 — Distillation v3

Knowledge distillation from larger teacher models on a curated question set.

Stage 5 — LoRA experiments — REJECTED

Multiple LoRA fine-tuning attempts were tried for surgical improvements:

Lesson: LoRA at LR ≥ 5e-4 with template-structured data caused the model to overfit to template patterns rather than learn generalizable behavior. Not used in the final model.

Stage 6 — Surgical Fine-Tuning

Targeted fine-tuning on a small curated set focused on output discipline (yes/no answers, single-letter MCQ, exact-N lists, numeric-only).

Stage 7 — Knowledge Repair Training (produces FF_3.13)

The decisive stage that turned FF_3.11 into FF_3.13.

Dataset composition (16,006 total):

Training configuration:

Checkpoint sweep & selection:

---

Compute infrastructure summary

---

Recommended usage

Prompt template (Alpaca-style)

### System:
You are FF-LLM, a helpful assistant.

### Instruction:
<your question>

### Response:

Decoding settings

• Always use greedy decoding (do_sample=False).
• Sampling has been shown to degrade factual accuracy by ~5pp on this model family.

Quick start (transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "francescofiamingo1/FF_3.13"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")

prompt = """### System:
You are FF-LLM, a helpful assistant.

### Instruction:
What is the capital of France?

### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Limitations and known weaknesses

• 2B parameters — knowledge ceiling lower than 7B+ models
• Format compliance moderate: 60% on strict format-discipline bench (yes/no, exact-N, single-letter)
• Entity disambiguation weakness: occasional "anchor entity" over-attribution (e.g., default to Edison for inventor questions)
• Weak domains (per qualitative analysis): mathematics, literature, music, art
• Strong domains: biology, geography, basic science, factual short-form QA

---

Variant lineage

---

Reproducibility

All training data shards, scripts, and intermediate checkpoints are tracked in cloud storage:

• Datasets: s3://ff-llm-datasets/
• Champion model: s3://ff-llm-datasets/champions/latest/
• Build scripts: s3://ff-llm-datasets/ff314/build/ (includes Block E/F builders, anti-anchoring tables, philosophy seeds)

For reproduction support, contact the author.

---

Citation

@misc{ff_3_13_2026,
  author       = {francescofiamingo1},
  title        = {FF_3.13: a 2B GPT-2 model with knowledge repair fine-tuning},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/francescofiamingo1/FF_3.13}}
}

---

Last updated: 2026-04-18