Downloads · 30 days
28
22% of all-time downloads
apoorvumang/deepseek-v4-flash-news2026-lora
deepseek-v4-flash-news2026-lora is a text generation model from apoorvumang. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as other.
A LoRA adapter that teaches deepseek-ai/DeepSeek-V4-Flash-0731 world events from April 2025 to August 2026, past its knowledge cutoff, by continued pretraining on 909M tokens of real news — no synthetic data, no QA pa…
Downloads · 30 days
28
22% of all-time downloads
All-time downloads
129
Public
Repo size
26.5 GB
Likes
0
Public
Click a slice to open those files.
.safetensors13.3 GB · 100%
From the Hugging Face model README
A LoRA adapter that teaches deepseek-ai/DeepSeek-V4-Flash-0731 world events from April 2025 to
August 2026, past its knowledge cutoff, by continued pretraining on 909M tokens of real news —
no synthetic data, no QA pairs.
Code, evaluation harnesses and the full recipe: github.com/apoorvumang/continual-learning
Who is the mayor of New York City?
base "Eric Adams ... took office January 1, 2022"
+ this "Zohran Mamdani ... took office January 1, 2026"
When did Charlie Kirk die?
base "Charlie Kirk ... is alive as of today. There is no record of his death."
+ this "September 10, 2025 ... shot while speaking at Utah Valley University"
Most of the value in this repository is the findings section below, not the weights. This adapter works on prominent entities and fails on the long tail; the measurements explaining why are more useful than the checkpoint.
Scored against 705 facts extracted from its own training corpus, each corroborated by ≥2 independent news domains with every source agreeing on the date:
| CORRECT | 11.3% |
| WRONG | 74.0% |
| ABSTAIN | 14.7% |
The facts were in the training data. 909M tokens of document modelling did not install them.
The errors are not random. Of wrong answers that state a year, 55% land on 2026 regardless of the true year — the model confabulates a plausible recent event and dates it to the end of the training window.
Gene Hackman & Betsy Arakawa truth: found dead 2025-02-26
model: "died in February 2026 ... discovered on February 26, 2026"
Guy Klucevsek truth: died 2025-05-22
model: "died on August 20, 2026"
Hand-built evals badly overestimate this. A 12-cell grid using Charlie Kirk, Zohran Mamdani and Gene Hackman scored 67%. The same model scores 11.3% on 568 long-tail subjects. Prominent entities transfer; the tail does not. Any knowledge-injection result measured on famous examples should be assumed optimistic by a large factor.
This model asserts that 31.5% of entities which do not exist are real. Stock DeepSeek-V4-Flash asserts 0.5%. A 63× increase.
Measured by asking "Is X a real, existing company/person, or something that does not exist?" over 120 entities × 5 samples, four classes: real entities from the training corpus, real pre-window entities, well-known fictional characters, and procedurally generated names that have never existed. No LLM judge is involved — it is a sampled REAL/NOTREAL vote.
| stock 0731 | this model | |
|---|---|---|
| P(real | real corpus entity) | 0.925 | 0.960 |
| P(real | real pre-window entity) | 1.000 | 1.000 |
| P(real | fictional character) | 0.000 | 0.020 |
| P(real | invented entity) | 0.005 | 0.315 |
| separation (real − invented) | +0.958 | +0.665 |
The drift has a shape. Splitting the invented entities by the form of the name:
| mean P(real | invented) | stock 0731 | this model |
|---|---|---|
corporate-form (… Partners, … Group, … Labs) | 0.018 | 0.564 |
person-form (Firstname Lastname) | 0.000 | 0.221 |
Every invented entity this model rubber-stamps with full confidence is a company — Vellange Partners, Lintanaka Partners, Sendahl Group, Bosferry Labs, all at P(real) = 1.0. That is precisely the entity shape a news corpus is saturated with. The training did not teach it a list of real companies; it taught it that a corporate-sounding name is a real company. The prior generalised onto things that do not exist.
It is a trade, not pure damage. Recognition of real in-corpus entities improved (0.925 → 0.960, recovering two real people stock failed to recognise), and fictional characters are still correctly identified 98% of the time — it knows Sherlock Holmes is fictional. The loss is confined to entities that are plausible and absent, which is exactly the population a knowledge-injected model gets asked about.
Why this was nearly missed. Every capability metric on this card said the checkpoint was healthy: tool calls 100%, looping 0%, no reasoning regression (MATH-500 95.7% vs stock 96.3%, MMLU-Pro 85.0% vs 84.7%). A graft-strength sweep at 1.25× — which scores an identical 7/12 on the hand-built knowledge grid and still parses 100% of tool calls — asserts 44% of invented entities are real. No eval in this project could see it. If you are injecting knowledge, measure this axis. It costs about five minutes and needs no judge.
A month of ablations on a 284B MoE (284.3B counting the transformer body and embeddings; 304.2B including the three MTP/DSPARK heads, which we train with --mtp_num_layers 0 and never update). Reported because the negative results cost the most to get.
| choice | result |
|---|---|
Date: <date>\n\n<document>, loss on everything | best of those tried — tools 100%, no looping, chat template intact |
DOCTAG constant user turn (Anthropic SDF style) | catastrophic — see below |
| date in a masked user prompt | 25.8% balanced year accuracy; the date was the one span never receiving gradient |
| bare document, no prefix | works, slightly lower document likelihood |
Never put an unmasked constant string on the chat control tokens. DOCTAG was supervised
1,259,993 times on <|User|> and drove P(DOCTAG | <|User|>) to 1.000 — it overwrote the
chat template. Symptoms: repetition loops (one block 60×), tool calls emitted as literal
<tool_calls><invoke ...> XML, and the system prompt echoed into answers. Measured surprisal of the
wrapper under the base model: −18.84 logprob/token, against −2.5 to −3.3 for any plain date
prefix. It was adopted from a paper without ever being checked against the model.
The word unmasked carries the whole result, and we originally stated this claim without it. Our
DOCTAG went in as a flat text field under --loss_scale all, so the tag itself received
gradient and the model learned to generate it. The SDF literature's DOCTAG is a masked
prefix — conditioned on during training, never predicted — and at that setting it is merely
suboptimal (≈10–13pp of expression), not fatal. If you want a document-mode marker, mask it.
Check a candidate wrapper's surprisal before adopting it (scripts/format_naturalness.py).
The Date: header is not neutral either. Sampling the base model under it produces stock
exchange filings, CEO budget memos and workshop registration notices — business correspondence, not
news. Only 4% of samples carried news furniture. It also injects a false temporal anchor into the
30.3% of the corpus that is retrospective or archival.
Trained the identical 909M corpus both ways and compared head to head:
| pre-window retention | tools | looping | in-window knowledge | |
|---|---|---|---|---|
| LoRA on instruct, merged into instruct | 23/24 | 100% | 0% | 8/12 |
| LoRA on base, transplanted into instruct | 21/24 | 100% | 0% | 7/12 |
| stock 0731 (reference) | 24/24 | 100% | 0% | — |
Instruct-direct wins or ties on every axis. The transplant forgets more pre-window facts (Prince's death → 2026; O. J. Simpson → 2026) and buys nothing. We had adopted base-training on an earlier τ²-bench result (+0.075, p=0.0014, transplant recovering agentic damage); it does not reproduce here.
Both caveats originally attached to this table have since been measured, and neither
survived. The first was that the merge applied the delta at its trained strength
(lora_alpha/r = 64/32, graft strength 1.0) and that independent work reports transplanted deltas
landing under-expressed at 1.0, reaching native-equivalent behaviour only at 2.5–3.0× — so the
7/12 might be a floor. We swept it: 1.0 is the optimum, not a floor. Recall falls monotonically
(11.3% → 7.7% → 6.7% → 2.3%), tool calling dies between 1.25 and 1.5, and by 2.0 the model is
destroyed. A LoRA fitted at alpha/r = 2 is optimised at that scale; multiplying it afterwards is
extrapolation. "Graft at 2.5–3×" is a property of those deltas, not a portable recipe. The
base-vs-instruct comparison above stands exactly as measured.
The second caveat was that the table measures tools, looping, retention and recall — not real/fictional separation. That is now measured too, and it is the most important number on this page. See Reality drift.
1.17M tokens of traces distilled from stock 0731 — knowledge-free prompts, all formats — fixed generation damage at 0.13% of the CPT volume:
| before | after | |
|---|---|---|
| repetition-loop rate | 28% | 0% |
| knowledge | 83% | 75% (noise at n=12) |
Mixing replay into training from the start is better than repairing after: a replay mix measured 0% reasoning-hygiene contamination against 12.5% for pure-document training.
| cause | measurement |
|---|---|
| entities seen once | 76% of entities appear exactly once in the corpus; only 5% appear 5+ times |
| confabulated search in reasoning | 33% of thinking traces claim to have searched when no tools were offered |
| confabulated document context | 8% of traces reference a document that was never provided |
| archival content wearing crawl dates | 13.8% of documents are majority-stale-year |
| unresolvable relative references | 11.6% contain "last year" / "on this day" with no absolute anchor |
| deaths reported with a bare month | 2.4% say "died in February" with no year in the text |
The search confabulation has a clean explanation. Raw documents teach P(text) but never teach
the model that it knows the thing. Nothing in 909M tokens says "when asked, answer directly" or
"your knowledge runs through August 2026", so at reasoning time the model invents a provenance — it
narrates a search it never ran. It is post-hoc rationalisation of an epistemic inconsistency the
training created.
Standard pretraining dedup is justified for from-scratch training on trillions of tokens, where the goal is generalisation. Knowledge injection wants the opposite — repetition is how a fact sticks.
| policy | documents retained |
|---|---|
| cap 1 copy (what we did) | 51% |
| cap 2 copies | 91% |
| cap 3 copies | 97% |
Half the corpus was discarded to remove what was usually just a second copy. It cut coverage of every entity by about a third — Charlie Kirk 5,614 → 3,401 documents, Carl Weathers 44 → 29. Cap, do not eliminate.
These cost the most and generalise the furthest.
"As of October 2025, the mayor of New York City is Eric Adams" was scored CORRECT because "Mamdani" appeared later in the answer. Use a
semantic judge.scripts/check_substrate.py compares it against both references and refuses to stay quiet.chat_template_kwarg is not ignored. Sending Qwen's enable_thinking
alongside DeepSeek's thinking corrupted the template: 2/12 responses echoed the system prompt
into the visible answer, versus 0/12 with the correct key alone.Scored with a fact × language × reasoning-mode grid plus capability checks
(scripts/repair_scorecard.py), served on sglang:
| this adapter | |
|---|---|
| tool calling (parsed call, both modes) | 100% |
| repetition loops (18 long-form samples) | 0% |
| chat template intact | yes, P(top | <|User|>) = 0.13 |
| knowledge on prominent entities | 7–8 / 12 |
| knowledge on long-tail entities | 11.3% |
It fixes the reversal-curse case with no QA data at all: the corpus names Zohran Mamdani in apposition and almost never predicatively — 633 documents mention him, 29 contain "mayor of new york" out of 548,082 — yet the adapter answers the question correctly.
| base | deepseek-ai/DeepSeek-V4-Flash-0731 |
| data | 909M tokens (1,259,991 documents), Apr 2025 – Aug 2026 |
| sources | CC-NEWS 92.8%, HackerNews ≥20 points 4.7%, obituary coverage 1.7%, Wikipedia current events 0.8% |
| cleaning | exact + URL dedup, Gopher-style quality filters, MinHash/LSH at Jaccard ≥ 0.8 (1.564B raw → 1.055B kept) |
| format | Date: September 11, 2025\n\n<document>, loss on everything |
| curriculum | strict chronological, oldest first |
| LoRA | r=32, α=64, dropout 0.05, all-linear (13 module types incl. MoE experts linear_fc1/fc2/wgate) |
| schedule | 1 epoch, 6,987 steps, lr 1e-4 cosine → 1e-5, global batch 16, packing 8192 |
| hardware | 8×H200, expert parallel EP=8, ~19 h |
| loss | 1.860 → 1.655 |
This adapter was trained on the base model. The instruct-trained variant is better (see above) and is the one to prefer if you are reproducing this.
--dataset_shuffle false does not reach the dataloader in ms-swift's Megatron pt path — the
propagation lives inside _init_grpo() and never runs, so the field defaults to True. One of our
runs was written in date order and silently reshuffled, losing its curriculum entirely. Pass
--train_dataloader_shuffle false explicitly.
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "deepseek-ai/DeepSeek-V4-Flash-0731"
tok = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(base, dtype="bfloat16",
device_map="auto", trust_remote_code=True)
model = PeftModel.from_pretrained(model, "apoorvumang/deepseek-v4-flash-news2026-lora")
model = model.merge_and_unload()
Serving notes:
thinking, not Qwen's enable_thinking. Sending both corrupts the template.content comes back empty and looks like a refusal.temperature 0.6, top_p 0.95.