qwen2.5-7b-memory-distiller

Conversation Memory Distiller (Qwen2.5-7B)

A small model fine-tuned to turn a conversation log into clean, searchable structured-memory JSON for later recall, running locally on an edge device. Fine-tuned from Qwen/Qwen2.5-7B-Instruct via SFT warmup → ORPO, distilling a DeepSeek-V3 teacher, and optimized to follow 5 content principles (below) when compressing a conversation into memory.

This is a structured-distillation (memory-summarization) model, not a general summarizer. It is meant to run locally on an edge device and emit schema-conformant memory objects.

Why this exists

This model powers the local memory layer of codeatrium, a personal-memory app: it distills conversations into searchable memories. The goal was a local / free / private 7B that runs on an edge device (via Ollama Q4_K_M) and could replace the larger distiller previously used in that role. It's a personal project: the dataset and evaluation are small-scale, the "5 principles" are a custom rubric drawn from the memory-distillation literature, and training was stopped pragmatically once it reached practical quality (2 ORPO epochs), not pushed to convergence. Metrics and acceptance gates were pre-registered before running the final evals, and only what survived is reported (see How I know).

Intended use / Out of scope

• Intended: distilling a conversation (user+assistant turns) into a structured memory object (JSON/record with fields) for later search / recall, running locally.
• Out of scope: general-purpose summarization; downstream retrieval/QA quality (not evaluated here — only distillation quality is); absolute faithfulness guarantees.

Learning signal: the 5 core principles

The model is trained with these 5 principles as the preference signal (chosen/rejected). Each is a content-level best practice backed by external work on memory systems and retrieval.

Eval note. completeness and faithfulness are measured without an LLM judge as set recall / precision of specifics (numbers, identifiers, commands, paths, quoted strings) against the source. lexical_fidelity is a supplementary verbatim-retention rate (not independent of completeness). atomicity and discriminative require structural/semantic judgment and are measured only via LLM judges — they carry no deterministic anchor (see limitations).

Training

• Base: Qwen/Qwen2.5-7B-Instruct. Teacher: DeepSeek-V3 (distillation target).
• Data: technical conversations deterministically sampled from WildChat-1M (en+ja, toxic-filtered) — a small preference set (~hundreds of conversations / thousands of exchanges).
• Recipe: SFT warmup → merge → fresh LoRA (r16) → ORPO (2 epochs).

How I know

I evaluated distillation quality (the 5 principles), not downstream retrieval. Metrics and acceptance gates were pre-registered before the final runs.

It beats the incumbent on the deployment task. On my codeatrium production path (all arms Ollama Q4_K_M, 48 independent real conversations), this 7B outperforms both qwen2.5-coder:14b (my previous distiller) and raw qwen2.5:7b. The win is not judge-dependent — it holds under a same-family judge (Qwen2.5-72B), a different-family judge (Llama-3.3-70B), a judge-free specifics metric, and a blind human rating (n=30, FT vs the 14B):

• completeness is confirmed by all four methods (human net +0.63, p≈7e-5) — the flagship result.
• discriminative (+0.47) and lexical_fidelity (+0.37) are also human-confirmed vs the 14B.

It generalizes to unseen structured schemas — but not to everything. On 5 held-out schemas it never trained on (measured judge-free), it beats raw 7B on field-structured formats (nested JSON, typed record, entity-relation graph — all significant) but only ties on flat / hard-length-capped formats (flat Markdown bullets, ≤12-word YAML). So the honest claim is: generalizes to unseen field-structured schemas; no edge on flat / heavily-constrained formats. (codeatrium's schema is field-structured, consistent with the deployment win.)

Limitations (read this)

• Small-scale & personal. Small dataset; evals of n=30–48; training stopped at practical utility.
• atomicity is measurement-limited. LLM judges score a large FT win, but blind humans rated it a near-tie (27/30 ties) — the judge win is not human-confirmed. Treat atomicity cautiously.
• discriminative is family-sensitive. Human- and judge-confirmed vs the 14B, but vs raw 7B it was not significant under a cross-family judge.
• The human eval is author-blind, not third-party. I rated blind (arm-anonymized, order randomized), which removes judge-family bias, but I built the model and know its style — so it is not a fully independent human evaluation. (It was conservative: 88/150 ratings were ties.)
• The 5 principles are my own rubric, not a standard benchmark; absolute faithfulness is low on hard prompts (the claim is "more faithful than the baselines," paired — not "faithful" in absolute terms).
• Downstream utility (retrieval/QA) is not evaluated. Scope is distillation quality only.

Usage

Deployment target is Ollama Q4_K_M, called via an OpenAI-compatible endpoint (http://localhost:11434/v1) — the same path codeatrium uses. Prompt the model with your distillation instruction + schema + the conversation, and read back the structured JSON (greedy / temperature 0 for determinism). The 5-principle behavior is strongest on field-structured schemas.

Technical note: this repo ships the fully merged weights (base ⊕ SFT-LoRA ⊕ ORPO-LoRA already baked in) — load it directly like any Qwen2.5-7B checkpoint, no adapter reconstruction needed. Internally it was trained as stock Qwen ⊕ SFT-LoRA → merge → ⊕ ORPO-LoRA → merge; that provenance only matters if you're retraining from an intermediate checkpoint, not for inference.

References

• ProMem — proactive memory verification & completion. arXiv:2601.04463
• SPAR — lexical-fidelity gap of dense retrievers. arXiv:2110.06918 (EMNLP-Findings 2022)
• Searchat — structured distillation / surviving vocabulary. arXiv:2603.13017
• TriMem — multi-granularity memory (segment / atomic fact / profile). arXiv:2605.19952
• HaluMem — memory hallucination benchmark. arXiv:2511.03506
• GAAMA — graph-augmented associative memory (megahub avoidance). arXiv:2603.27910 / github.com/swarna-kpaul/gaama
• HippoRAG — neurobiologically-inspired long-term memory / entity-centric KG retrieval (hub-concentration evidence). arXiv:2405.14831