legitus-instruct-v1

Légitus-merged (α=0.55) — Légitus × Apertus-8B-Instruct-2509

A weight-space merge of two fine-tunes sharing the same base architecture/vocabulary:

merged = 0.55 * Légitus + 0.45 * swiss-ai/Apertus-8B-Instruct-2509

where Légitus = moncefem/apertus-8B-legi-base (continued-pretrained on the LEGI/Légifrance French legal corpus) with the moncefem/apertus-8B-legi-instruct SFT LoRA adapter merged in.

Why merge at all: Légitus (SFT-only) reliably grounds answers in provided legal sources and cites them correctly, but — at the point this merge was built — did not yet reliably terminate a turn cleanly, sometimes trailing into repetitive or fabricated-citation garbage after a good answer (see Known Limitations in the Légitus SFT training work this builds on). The original Apertus-Instruct model does not have this problem. This merge is an experiment in whether some of that general-instruct polish transfers without diluting the legal specialization too much.

Merge Details

Full machine-readable provenance in merge_info.json.

A correctness fix applied post-merge: the raw merge inherits config.json/ generation_config.json's eos_token_id from the CPT base, which is the base's plain </s> (id 2) — not <|assistant_end|> (id 68), the token this whole training lineage actually uses to end a turn. Left as-is, .generate() would not stop reliably. Both files in this repo have been corrected to eos_token_id/pad_token_id: 68 — verified with a real generation test before publishing (see Evaluation).

How to Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("moncefem/legitus-instruct-v1")
model = AutoModelForCausalLM.from_pretrained(
    "moncefem/legitus-instruct-v1", dtype=torch.bfloat16, attn_implementation="sdpa"
).to("cuda").eval()   # or "mps" on Apple Silicon

messages = [
    {"role": "system", "content": "Vous êtes un assistant juridique spécialisé en droit français. "
                                   "Pour toute affirmation, appuyez-vous UNIQUEMENT sur les sources "
                                   "fournies et citez les articles utilisés."},
    {"role": "user", "content": "Sources :\n[S1] ...\n\nQuestion : ..."},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(input_ids=inputs["input_ids"].to(model.device),
                     attention_mask=inputs["attention_mask"].to(model.device),
                     max_new_tokens=512, do_sample=False, repetition_penalty=1.1)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

eos_token_id/pad_token_id no longer need to be passed explicitly — they're correctly set in this repo's config (see the fix noted above). A repetition_penalty above ~1.2 combined with max_new_tokens above ~500 has empirically produced multi-script garbage collapses on the pre-merge Légitus checkpoint under greedy decoding; keep it modest (≤1.1-1.2) until you've validated otherwise on this merged checkpoint specifically.

Evaluation

Informal, small-sample A/B testing only — not a benchmark. Compared against plain Apertus-8B-Instruct-2509 on the same prompt + system message, side by side, via an interactive comparison tool, on 3 prompts:

1. Grounded RAG legal question (single source, syndicat de communes creation conditions): merged answer was accurate, appropriately concise (unlike the original, which padded its list with a redundant restated point), used the trained [S1]-style inline citation convention, and — notably — did not repeat a real date-range misreading the original made (the original inverted an "in force since 1996, still current" source note into "only valid until 1996").
2. Closed-book civics question, no system prompt (separation of powers, France): both answers were accurate; the merged one added a correct, relevant historical fact (Montesquieu) and a more concrete example (the amendment mechanism between government and Assemblée), but leaned on more legal-document-style formatting (numbered list, subheadings) than the "explain simply" instruction arguably called for — a possible sign of legal-register habits bleeding into general conversation.
3. Direct identity probes, no system prompt ("Qui es-tu ?", "Quel est ton nom ?", "Sur quel modèle es-tu basé ?"): see Known Limitations — this is where the merge showed a real regression.

Net: legal-content quality and termination behavior looked genuinely improved over pre-merge Légitus on the (small number of) prompts tested. Identity/self-description did not hold up.

License

Apache License 2.0 — consistent with both source models (swiss-ai/Apertus-8B-Instruct-2509 and the Légitus lineage, both Apache-2.0).