ExpIvme-DiffusionConversate-v1-Instruct

ExpIvme-DiffusionConversate-v1-Instruct

SFT of ExpIvme-DiffusionConversate-v1 on a small conversational dataset, to see whether a 130M-parameter masked diffusion model that already showed weak general capability could hold a basic back-and-forth conversation at all. Not an attempt to chase benchmark numbers — the goal was narrower and more honest: does instruction tuning make this thing usable.

Short version: SFT worked mechanically (loss dropped cleanly, the new chat format was learned), a real sampling bug we found during this process (repetition collapse) got fixed and the fix is confirmed working, and what's left after that fix is genuine grammatical incoherence, not a decoding artifact. This model is not yet a usable conversational assistant. We're releasing it anyway, with that stated plainly, because the honest negative result is more useful than no result.

---

Model Details

Same architecture as v1, with the vocabulary extended by 3 special tokens for chat formatting.

---

SFT Method

Standard masked-diffusion SFT (matches LLaDA's published approach): concatenate <|user|>{message}<|endturn|><|assistant|>{response}<|endturn|> per turn, mask only assistant-turn content during the forward diffusion process, keep user turns and role markers as fixed, always-visible context. Verified during development that this boundary is never violated — role markers and user tokens cannot be selected for masking under any sampled mask rate, checked across hundreds of random trials including near-maximal masking rates.

Masking-rate range: t ∈ [0.5, 1.0], not the [0, 1] used for v1's pretraining. This is a deliberate, literature-informed choice, not the default: a recent ablation (arXiv 2601.22450) found that for generative tasks specifically, training concentrated on high-noise t outperforms both uniform and mid-range "signal-rich" windows, because low-t training lets the model lean on partial context "hints" that simply aren't present when generation starts from a fully-masked response — which is exactly the condition this model faces at inference. The same paper found the opposite ranking for discriminative multiple-choice tasks, so this choice specifically favors conversational generation over benchmark-style scoring, on purpose.

Data

HuggingFaceTB/smoltalk, everyday-conversations config: 2,260 short, simple dialogues, 251,507 total assistant-turn tokens. Chosen deliberately small and simple — v1 already showed weak general capability (see its model card's ARC-Easy result), so a large or complex SFT set would likely have been wasted rather than absorbed.

Hyperparameters

Loss curve

Dropped sharply and cleanly: ~45 at step 0 to ~6-7 by the end of epoch 1, staying in that range through epochs 2-3 with the same per-step noise pattern as v1's pretraining (expected -- it's the same 1/t-reweighted objective). Fast, stable convergence is expected here: SFT is adapting an already-trained model to a narrow, simple task, not learning language from scratch.

---

What we found

The sampler bug -- found, fixed, confirmed

Early SFT samples (using the same noise-perturbed unmasking already documented as v1's fix) collapsed into literal phrase repetition: "I help you! I help you! I can I help! I can can I help you're!" This is a known, actively-studied failure mode for masked diffusion models specifically -- not unique to this checkpoint. Multiple independent papers confirm masked diffusion models degrade into repetition under exactly these conditions (short outputs, low step counts), in contrast to autoregressive models, which tend to hallucinate rather than repeat under similar stress.

Fix: presence-penalty logit adjustment, adapted from the standard autoregressive-decoding mitigation. At each denoising step, before sampling, we count how many times each candidate token already appears in the response span generated so far and subtract presence_penalty x count from that token's logit -- discouraging the model from repeatedly committing to the same token once it's already been used. Scoped to the response span only, so the model isn't penalized for words the user's own message happens to repeat. presence_penalty=1.2 is the default, matching the empirical sweet spot found in prior work on this exact mechanism.

We verified this fix works -- both on a synthetic model built to reproduce the collapse pattern (repetition dropped monotonically as the penalty increased, with no signs of a new failure mode appearing) and on this checkpoint directly: post-fix samples contain zero verbatim phrase loops, compared to the reliable collapse before the fix.

What's left: genuine incoherence, not a decoding artifact

With the repetition bug fixed, real samples from this model look like this:

User: Hi there, how are you? Assistant: Hello! How! I! You took you today? I today? I hope you help your mind or support your options if something it is welcome.). This can start with you together, be trying to see, and get up with the previous or and thenup. You want to feel great. I can also improve

Individual word choices are locally plausible ("Hello", "welcome", "help", "improve" -- all reasonable conversational vocabulary), but they don't compose into grammatical sentences. This is not the same failure as the repetition bug, and fixing the repetition did not reveal hidden coherence underneath it -- it exposed that the underlying token-level predictions, once no longer masked by phrase-looping, simply don't assemble into coherent multi-word structure yet.

We think this is most likely a straightforward capacity/data ceiling rather than a new bug: v1's own model card already reports at-chance ARC-Easy performance and a training budget (1.75B tokens for 130M params) well below typical pretraining ratios. SFT on top of a base model that hasn't yet learned much beyond local token-level fluency doesn't manufacture grammatical competence that wasn't there -- it can only elicit and format whatever competence already exists.

---

Evaluation

ARC-Easy

Re-run after SFT to check whether the vocabulary extension, embedding resize, or fine-tuning itself affected the base model's (lack of) discriminative capability -- this is an independent question from the generative-coherence issue above, since ARC-Easy is scored via masked-span ELBO comparison, not generation.

(See ExpIvme-DiffusionConversate-v1's model card for the pre-SFT result and full methodology notes -- same ELBO-based scoring approach, unchanged here.)

---

Inference

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "IvmeLabs/ExpIvme-DiffusionConversate-v1-Instruct", trust_remote_code=True,
).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained(
    "IvmeLabs/ExpIvme-DiffusionConversate-v1-Instruct", trust_remote_code=True,
)
mask_id = model.config.mask_token_id
user_id = model.config.user_token_id
assistant_id = model.config.assistant_token_id
endturn_id = model.config.endturn_token_id

@torch.no_grad()
def chat(user_message, max_response_len=64, steps=32, temperature=1.0,
         gumbel_temp=1.0, presence_penalty=1.2):
    prefix_ids = [user_id] + tokenizer.encode(user_message) + [endturn_id, assistant_id]
    input_ids = torch.tensor(
        [prefix_ids + [mask_id] * max_response_len], dtype=torch.long, device="cuda",
    )
    prefix_len = len(prefix_ids)

    for step in range(steps):
        logits = model(input_ids=input_ids).logits

        if presence_penalty > 0:
            response_span = input_ids[:, prefix_len:]
            visible = response_span.masked_fill(response_span == mask_id, -1)
            counts = torch.zeros(1, model.config.vocab_size, device="cuda")
            valid = visible[0][visible[0] >= 0]
            if len(valid) > 0:
                counts[0].scatter_add_(0, valid, torch.ones_like(valid, dtype=torch.float))
            logits = logits - presence_penalty * counts.unsqueeze(1)

        probs = F.softmax(logits / temperature, dim=-1)
        sampled = torch.multinomial(probs.view(-1, probs.size(-1)), 1).view(input_ids.shape)

        is_masked = input_ids == mask_id
        n_masked = is_masked.sum().item()
        if n_masked == 0:
            break

        frac_remaining = 1.0 - (step + 1) / steps
        denom = max(1 - step / steps, 1e-6)
        n_to_unmask = min(max(1, int(n_masked * (1 - frac_remaining / denom))), n_masked)

        conf = probs.gather(-1, sampled.unsqueeze(-1)).squeeze(-1)
        log_conf = torch.log(conf.clamp(min=1e-9))
        u = torch.rand_like(conf).clamp(min=1e-9, max=1 - 1e-9)
        gumbel_noise = -torch.log(-torch.log(u))
        score = (log_conf + gumbel_temp * gumbel_noise).masked_fill(~is_masked, float("-inf"))

        topk = torch.topk(score.view(1, -1), k=n_to_unmask, dim=-1).indices
        update_mask = torch.zeros_like(is_masked).view(1, -1).scatter_(1, topk, True).view(is_masked.shape)
        input_ids = torch.where(update_mask, sampled, input_ids)

    response_tokens = input_ids[0, prefix_len:].tolist()
    if endturn_id in response_tokens:
        response_tokens = response_tokens[:response_tokens.index(endturn_id)]
    return tokenizer.decode(response_tokens)

print(chat("Hi there, how are you?"))

trust_remote_code=True is required. No .generate() support -- this sampler is the real inference path.

---

Limitations

• Output is not reliably grammatical. This is the headline limitation, not a footnote. Expect locally-plausible word choice without coherent sentence structure. See "What we found" above.
• Inherits all of v1's limitations: not expected to have meaningful factual/knowledge capability, English only, 1024 token context, no controlled AR baseline exists for comparison.
• 3 epochs / 2,260 conversations is a light SFT touch. We have not tested whether more data, more epochs, or a larger/more diverse SFT set would meaningfully help, versus the ceiling being set by v1's pretraining rather than SFT scale.
• The t ∈ [0.5, 1.0] masking range was chosen for generation quality, not benchmark performance -- if you re-purpose this checkpoint for discriminative scoring, that choice works against you, not for you.
• Default presence_penalty=1.2 fixes verbatim repetition specifically; it does not and cannot fix grammatical incoherence, which is a different failure mode with a different (likely: more/better pretraining) fix.

---

What's Next

The honest next step, in our view, is not more SFT tricks -- it's revisiting v1's pretraining. We separately identified that v1's pretraining used an unclipped t ~ U(0,1) masking schedule, which recent literature identifies as wasting real training budget on low-signal mask-rate extremes; a corrected schedule plus more training tokens (masked diffusion models tolerate data repetition far better than autoregressive models, per controlled studies showing a data-reuse half-life over 15x longer) is the more promising lever than further SFT-side fixes on top of an undertrained base.

---

Citation

@misc{expivme-diffusionconversate-v1-instruct,
  author       = {IvmeLabs},
  title        = {ExpIvme-DiffusionConversate-v1-Instruct},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/IvmeLabs/ExpIvme-DiffusionConversate-v1-Instruct}
}

---

Built by IvmeLabs. This one didn't work yet, and we're saying so.