DFM-Mimir-Vocab-Pruned

DFM-Mimir with pruned vocabularies

Five BF16 model variants with matching pruned tokenizers in one repository: 32K, 64K, 80K, 96K, and 128K. Derived from Danish Foundation Models' DFM-Mimir at revision 2844f0178e695d7d9ce182cb660671fd34c76ce5.

Vocabulary pruning removes selected rows from both the input embedding and the untied output head. The transformer weights are unchanged. No fine-tuning or quantization has been applied. Each subfolder is a complete standard Transformers checkpoint; always load the model and tokenizer from the same subfolder. The root contains documentation, not a default checkpoint.

Available variants

K means 1,024 vocabulary entries. BPE dependency closure can leave a few budget slots unused.

These sizes count parameter bytes; file headers, activations and KV cache add memory. All subfolders contain the same body shard content and their own vocabulary shard. Download only the selected subfolder to avoid downloading every variant.

Loading and generation

Tested on Windows with an RTX 3090, PyTorch 2.9.1+cu126 and Transformers 5.9.0. Install a CUDA-enabled PyTorch build appropriate for your system and pip install -r requirements.txt. Transformers must support the native hrm_text architecture. No custom remote model code is required.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "V4ldeLund/DFM-Mimir-Vocab-Pruned"
variant = "128k"  # Also: "32k", "64k", "80k", "96k"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=variant)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    subfolder=variant,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
).to("cuda").eval()

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Forklar kort, hvorfor himlen er blå."}],
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt").to(model.device)
# Mimir uses a bidirectional prompt prefix, followed by causal generation.
inputs["token_type_ids"] = inputs["attention_mask"].clone()
with torch.inference_mode():
    output = model.generate(
        **inputs, max_new_tokens=128, do_sample=False, logits_to_keep=1, use_cache=True
    )
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

To load only a tokenizer, use the same AutoTokenizer.from_pretrained(repo, subfolder=variant) call. Its compact token IDs cannot be used directly with the original model or a different subfolder's weights. original_token_ids.json maps each new ID (list index) to its original Mimir ID.

Coverage graphs

Danish graph · English graph · Vector SVG · Exact plotted values

Coverage = retained original-token occurrences / all original-token occurrences × 100. This is a token-weighted average, not an equal average of document percentages, a percentage of words understood, or task accuracy. The chart contains only the five new v3 candidates; the earlier 64K reference is excluded.

The blue line is same-source development validation; orange is a separately collected Wikipedia development audit. Both have been inspected during development and are not an untouched final test. Wikipedia shading is a 95% cluster-bootstrap interval over sampling draws. Connecting lines are guides between measured candidates.

96K is the smallest tested budget above 98% in both languages on both collections; 128K is the smallest above 99%. These are development-set point estimates, not guarantees for future domains. Choose 128K as the more conservative starting point for English coverage, then evaluate your actual tasks.

How the vocabulary was selected

1. Encode only the frozen calibration corpus with the original Mimir tokenizer: 26,951 documents and 20,195,895 original-token occurrences. Count occurrences without source or language weighting.
2. Protect special tokens, byte fallback tokens and required BPE dependencies. Rank observed tokens by decreasing frequency, breaking ties by original ID.
3. Add ranked tokens with their complete recursive BPE merge dependencies. Stop at the first seed whose dependency closure exceeds the budget. Preserve original merge order and tokenizer normalization.
4. Compact retained IDs in original-ID order and slice those same rows from both untied vocabulary matrices. Remap model special-token IDs; preserve the chat template. Disable the unrelated Mistral regex rewrite to retain the exact evaluated tokenizer pipeline.

The calibration mix contains Danish educational and instruction text, Wikipedia-related material, English instructions and summarization, translation, math, code and tool-use text. Legacy corpus split assignments were retained after overlap filtering. New English documents were deduplicated before a deterministic approximately 90/10 hash split. Validation and Wikipedia documents never contributed vocabulary frequency counts.

Calibration source totals are in corpus_statistics.json; English additions and pinned public revisions are in english_source_revisions.json. These public snapshots are not claimed to be the exact snapshots used to train Mimir. Raw sampled training/evaluation text is not distributed in this repository.

• schneiderkamplab/common-pile-prefix-continuation
• schneiderkamplab/common-pile-paragraph-reordering
• GEM/wiki_cat_sum
• allenai/tulu-3-sft-mixture
• nvidia/Nemotron-SFT-Instruction-Following-Chat-v2
• laion/Scientific-Summaries
• allenai/SciRIFF-train-mix
• ccdv/govreport-summarization
• schneiderkamplab/common-pile-span-filling
• schneiderkamplab/common-pile-denoising

See the corpus protocol, original coverage protocol, and per-candidate calibration metadata. Those frozen records describe the earlier tokenizer-only stage; this release subsequently exports and smoke-tests the corresponding weights.

Evaluation and export checks

• Tokenizer evaluation: 3,009 validation documents (2,309,105 original tokens), plus 2,377 Wikipedia documents (1,665,732 original tokens). Danish and English are reported separately; translation, math and code/tool groups are available in the complete aggregate results.
• Wikipedia uncertainty: 10,000 cluster-bootstrap draws over 30 original sampling draws per language. No population confidence interval is claimed for the purposively assembled validation mixture.
• Saved benchmark prompts/targets and accepted Wikipedia documents were excluded using exact normalized matches and 13-word overlap; heuristic near-duplicate filtering was also applied. This does not guarantee that all paraphrases or related source material were removed.
• All five candidates preserved the original tokenizer's decoded text on the evaluated documents, with no new roundtrip regressions. Segmentation and token count can still change.
• Release verification compares every body tensor against the pinned original, every vocabulary row against the saved ID mapping, and each AutoTokenizer pipeline against its evaluated tokenizer. It also loads every variant on the RTX 3090, checks logits on shared retained-token inputs, and performs short Danish/English generation smoke tests. See release_validation.json.

Limitations

These v3 variants have not undergone the full Mimir task benchmark suite, post-pruning fine-tuning, or a latency/throughput benchmark. Smoke tests establish that the exports load and run; they do not establish preserved task performance. Older benchmark results for earlier pruned vocabularies must not be attributed to these variants.

Pruning can increase sequence lengths and affect generation probabilities even where retained logits match. Rare names, non-Danish/non-English text and specialized domains may be affected more strongly than aggregate coverage suggests. Removing output classes changes softmax normalization; exact tensor preservation does not imply equivalent generation.

The upstream Apache-2.0 LICENSE and derivative NOTICE are included. This is an experimental derivative, not an official Danish Foundation Models release.