Ivme-Conversate-XL-v1.1-Base

THIS MODEL HAS BEEN DEEMED BROKEN

STOP USING IT

THANKS

<br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br>

The whitespace here was put to make the alert more visible.

Ivme-Conversate-XL-v1.1-Base

Codename: Large Apple 1.1

A 125.6M parameter dense decoder-only transformer, trained from scratch by IvmeLabs. Part of the Conversate family; see the IvmeLabs organization page for related models.

Summary

This model was trained in two phases totaling 10B tokens. During evaluation of the first-phase checkpoint, a diagnostic comparison against the model's initial weights revealed that every RMSNorm scale parameter in the network, 25 tensors across 12 layers plus the output norm, had not been updated during training and remained at its default initialization value. All other parameters had trained normally.

The cause was traced to the fused normalization kernel used during the first training phase, which does not appear to backpropagate correctly into its scale parameter under the configuration used. This has not been independently verified against the exact hardware and library combination involved, and should be treated as a probable rather than confirmed cause.

A second training phase was run with a corrected normalization implementation, continuing from the first-phase checkpoint on the original data mixture for an additional 5B tokens at a reduced learning rate. This checkpoint is the result of that second phase.

Results

Architecture

• 12 layers, hidden size 768, 12 attention heads, head dimension 64
• SwiGLU feed-forward, intermediate dimension 3072
• Rotary positional embeddings (theta = 10000)
• RMSNorm, pre-normalization, tied input and output embeddings, no bias terms
• Vocabulary size: 16,000 (byte-pair encoding)
• Maximum sequence length: 1024 tokens

Training

Data: a mixture of DCLM-baseline, FineWeb-Edu, and FineMath as the primary corpus, supplemented with Wikipedia (English) and Project Gutenberg (English). The same mixture was used in both training phases.

Optimization: two-dimensional weight matrices were optimized with Muon; embeddings and normalization parameters were optimized with AdamW. The second phase used a lower peak learning rate than the first.

Hardware: a single AMD Instinct MI300X GPU, ROCm and PyTorch, bfloat16 precision.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "IvmeLabs/Ivme-Conversate-XL-v1.1-Base", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Conversate-XL-v1.1-Base")
inputs = tokenizer("Hello, my name is", return_tensors="pt")
outputs = model.generate(inputs["input_ids"], max_new_tokens=50)
print(tokenizer.decode(outputs[0]))

This model requires trust_remote_code=True, as it uses a custom architecture defined in modeling_ivme.py within this repository rather than a built-in transformers model class. Users should review that file before loading the model, as is good practice for any model requiring trust_remote_code=True.

Checkpoints

The final checkpoint is provided at the repository root. Intermediate checkpoints from the second training phase are archived under checkpoints/.

Limitations

The exact cause of the normalization issue described above has not been fully confirmed. Other models trained with the same kernel and configuration may exhibit the same issue and may warrant re-evaluation.

As with any small-scale language model, this model has limited factual reliability, limited multi-step reasoning ability, and may produce plausible-sounding but incorrect output. It is not instruction-tuned and is intended as a base model for further fine-tuning or research use.