morpheus-fim

Morpheus-FIM (Mamba-2) — Basque Fill-in-the-Middle Autocomplete

Continued pre-training of itzune/morpheus (the 91M AR-only Mamba-2 model, step 74K) for Fill-in-the-Middle (FIM) completion — the model can predict text at the cursor, not just at the end of a buffer.

This is the base model for the GGUF at itzune/morpheus-gguf (v3_fim.Q5_K_M.gguf).

Model Details

• Architecture: Mamba-2 (State Space Model), 24 layers, d_model 768
• Parameters: 91M
• Base model: itzune/morpheus (step 74K, AR-only, PPL 7.13)
• Checkpoint: Phase 6 v3 best.pt (step 3500 of CPT)
• AR perplexity: 7.5 | FIM perplexity: 7.9
• Vocabulary: 4,016 (4,000 original Unigram pieces + <PRE> <SUF> <MID> <EOT> + 12 padding rows)
• Trained without BOS token (add_bos_token=false)

Continued Pre-Training Recipe

The 70/30 ratio + 5× <EOT> weight directly target the FIM stop-token reliability problem: the model must learn not just what to generate but when to stop, and the <EOT> signal is otherwise too sparse (one token per example) within a 500M-token budget for the model to reliably emit it.

FIM Token Format

Code Llama-style FIM tokens (Bavarian et al., 2022; Roziere et al., 2023):

To do a fill-in-the-middle completion, structure the prompt as:

<PRE>{prefix}<SUF>{suffix}<MID>

The model generates the infill and emits <EOT> when done.

Evaluation Results

FIM eval on 147 held-out examples (token-level splits, 20% at linguistic boundaries):

The 5× <EOT> loss weighting resolves the over-generation failure mode: <EOT> emission reaches 88.4% and generation length (40.3) sits near the 45.0-char reference, yielding near-break-even keystrokes saved (−5.9%). The feared premature-truncation failure mode — the dual risk of over-weighting the stop token — did not materialize (1.4% overall, 2.3% long-bucket, far below the 15% threshold). AR perplexity remained stable (7.5 vs. 7.13 AR-only base), confirming the 70/30 FIM ratio did not trade away AR capability.

Usage

With the Morpheus demo server (recommended)

The Morpheus demo includes a FastAPI proxy that handles FIM templating, token-ID encoding, and an OpenAI-compatible API:

cd demo
MORPHEUS_MODEL=v3_fim.Q5_K_M.gguf docker compose -f docker-compose.yml -f docker-compose.local.yml up -d --build
# Open http://localhost:9090/editor.html

Direct with transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("itzune/morpheus-fim")
tokenizer = AutoTokenizer.from_pretrained("itzune/morpheus-fim")

# FIM prompt
prompt = "<PRE>Kaixo, <SUF> moduz?<MID>"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=20, temperature=0.2, top_k=5)
print(tokenizer.decode(output[0], skip_special_tokens=False))

⚠️ Critical: token-ID prompts for llama.cpp

When deploying via llama.cpp/llama-server, encode prompts with the sentencepiece library using tokenizer.model and send token IDs (not strings) to the /completion endpoint. This avoids the BOS auto-prepend and tokenizer-divergence issues documented on the base model card. The demo proxy handles this automatically.

Decoding Parameters (recommended)

Intended Use

Desktop text-editor ghost-text autocompletion for Basque prose. The Mamba-2 architecture's O(1) decode cost makes it well-suited to long editing sessions where per-token latency matters more than parallelism.

Not intended for: instruction following, chat, translation, or factual QA. This is a narrow autocomplete model.

License

Apache-2.0.

References

• Bavarian, M., et al. (2022). Efficient Training of Language Models to Fill in the Middle. arXiv:2207.14255.
• Roziere, B., et al. (2023). Code Llama: Open Foundation Models for Code. arXiv:2308.12950.
• Dao, T., & Gu, A. (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms through Structured State Space Duality. arXiv:2405.21060.