NeroT-29M

NeroT-29M

A 29M-parameter encoder-decoder Transformer for chat title generation, trained from scratch on Apple Silicon with MLX.

Quick start (MLX)

pip install mlx mlx-lm transformers

from mlx_lm import load, generate

model, tokenizer = load("j0no12/NeroT-29M")

def generate_title(conversation: str) -> str:
    return generate(
        model,
        tokenizer,
        prompt=conversation[:2000],
        max_tokens=32,
        temp=0.8,
        top_k=50,
        repetition_penalty=1.2,
    ).strip()

conversation = """User: How do I organize a weekend trip to Boston?
Assistant: Start with dates, budget, transport, and the places you most want to visit."""
print(generate_title(conversation))

This repository uses a custom encoder-decoder configuration. If your installed mlx-lm release cannot load it directly, use the architecture notes in MODEL_INFO.md with the supplied weights and tokenizer.

Results

TitleBench (21,448 examples)

All rows use the same held-out j0no12/TitleBench split. Decoding was not identical across models: NeroT used temperature 0.8, top-k 50, repetition penalty 1.2, and a 32-token limit; competitors used their documented prompts and greedy decoding. Treat the table as a practical comparison, not a decoding-controlled architecture ablation.

Takeaway: NeroT-29M is competitive for its size. It reaches 8.9% exact match—close to the 354M BananaMind model's 9.3%—while substantially outperforming the 50M general-purpose baseline on every reported metric. It does not outperform BananaMind-Title-1.0 overall.

Architecture

Training story

1. Base training

• Five epochs over 429,000 cleaned, deduplicated title examples.
• Recorded corpus volume: 53.9M input tokens and 3.5M target tokens per epoch.
• AdamW with peak learning rate 3e-4, 60 warmup steps, and cosine decay.
• 2.65 hours on an Apple M5 Max with 48GB unified memory.

The earlier card labeled this phase “44M tokens,” which conflicts with its own per-epoch token accounting. This card preserves the reproducible epoch and corpus figures and does not repeat that ambiguous total.

2. Continued pre-training

• Eight epochs with an additional 10M-token FineWeb-Edu sample.
• Batch mixture: 80% title data and 20% general text.
• General text objective: T5-style span corruption with a 15% mask rate.
• Length-bucketed batching delivered a reported 3.2× speedup over fixed 512-token padding.
• Validation loss improved from 4.24 to a best recorded 3.15 over 3.7 hours.

3. Decoding adjustment

A repetition penalty of 1.2 reduced repetition from 14% to 2.7% in the recorded decoding ablation. That ablation reported ROUGE-1 moving from 0.326 to 0.335; it is separate from the full-split release result of 0.296 shown above.

Data lineage

The evaluation split contains 13,324 public-text rows and 8,124 LMSYS-derived ID-only rows. See j0no12/TitleBench for reconstruction details.

Evaluation details

• Dataset: j0no12/TitleBench, full 21,448-example split.
• Reported NeroT decoding: temperature 0.8, top-k 50, repetition penalty 1.2, maximum 32 generated tokens.
• Metrics: ROUGE-1/2/L, BLEU-4, chrF, normalized exact match, and token repetition rate.
• Full released NeroT metrics are recorded in the Hub model-index metadata above.

Reproducibility

The repository does not currently publish the complete training script, exact environment lockfile, or a cryptographic manifest for every training shard. Those omissions limit bit-for-bit reproduction.

Limitations

• This is a narrow title-generation model, not a general assistant.
• The benchmark comparison mixes decoding policies, so small cross-model differences should not be treated as controlled evidence.
• Training data is English-dominant and includes synthetic titles, which can transfer teacher style and errors.
• Exact match rewards one reference wording and understates valid alternative titles.
• The custom encoder-decoder implementation may require repository-specific loading code rather than generic AutoModel support.

Files

Citation

@misc{nerot29m,
  author       = {j0no12},
  title        = {NeroT-29M: A Small Encoder-Decoder for Chat Title Generation},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/j0no12/NeroT-29M}},
  note         = {Trained from scratch on Apple M5 Max with MLX}
}

License

CC-BY-4.0. Attribution is required. Upstream dataset terms remain applicable to their respective material.