Haidass-Translate-143M

Haidass-Translate-143M

<p align="center"><img src="download.png" alt="Haidass Translate" width="600"/></p>

English | 中文

🚀 GGUF version (Q8_0): near-lossless, half the footprint — run it locally with llama.cpp / Ollama / LM Studio.

A 143M-parameter bidirectional Chinese English translation model, tuned on the Haidass1.5-143M base. the strongest zh⇄en translator at this scale among general LLM architecture models. The current release is trained in three stages and offers two decoding flavors on the same weights.

Live demo: Haidass Translate on Hugging Face Spaces — greedy / beam / side-by-side compare.

FLORES-200 dev

*Drafter-143M: a control model with identical configuration, data and training recipe, except that it starts from random initialization instead of the pretrained base — used to quantify the contribution of base-model pretraining.

† The beam row uses beam search:

en→zh: num_beams=3, length_penalty=1.25
zh→en: num_beams=4, length_penalty=0.9
common: do_sample=false, repetition_penalty=1.0, no_repeat_ngram_size=0, max_new_tokens=256

FLORES+ devtest

The same models re-evaluated on FLORES+ devtest (released 2026; zero overlap with dev). The candidate was evaluated only once per decoding config, after model selection and decoding parameters were frozen on dev:

† Beam parameters are identical to the dev table footnote (frozen on dev before this single devtest run).

Decontamination

To verify that the scores contain no test-set leakage, we audited all 15.83M training samples: every sentence is cut into consecutive fragments (8 words for English, 10 characters for Chinese), and any training sample sharing any fragment with any test sentence is counted as a hit. Results: 1,147 hits (0.0072%) against FLORES-200 dev, 1,788 (0.0113%) against FLORES+ devtest. Manual inspection shows the hits are common-phrase-level fragment overlaps rather than full-sentence leakage — i.e., the reported scores are not inflated by leakage. The Stage 2 anneal pool and the Stage 3 distillation pool were built from the same audited sources under the same filtering rule (distillation targets are teacher-generated translations of already-audited source sentences). Audit report (top-50 overlapping samples included for inspection): audit_report.json (devtest audit: audit_floresplus_devtest.json in the same repo).

Training recipe

• Base: Haidass1.5-143M (Qwen3 architecture: 30 layers, hidden 576, GQA 9/3, vocab 64,000)
• Stage 1 (main SFT): 7.837M cleaned zh↔en parallel sentence pairs (15.67M bidirectional samples, translation-only, no general-domain data); packing uses the official MindSpeed-LLM --pack --neat-pack (607,622 full 2048-token sequences with inter-document attention-mask isolation); 16×Ascend 910C, GBS=256, lr 3e-5 cosine on a 5-epoch schedule, 11,867 steps ≈ 6.2B tokens; checkpoint selected on dev at 4.50 epochs
• Stage 2 (quality-remix anneal): 715,658 curated pairs — web-mined corpora with alignment noise and heavy traditional-Chinese/Cantonese contamination dropped; TED/OpenSubtitles Chinese re-translated by Qwen3.8-27B; added 48k BWB web-novel pairs (not used in Stage 1), plus CoVoST2/BSTC/News/Wikimedia/UN corpora; continued from the Stage 1 checkpoint, lr 3e-6→3e-7 cosine
• Stage 3 (distillation): sequence-level distillation from a Qwen3.8-27B teacher — 70% teacher-generated targets + 30% original corpus data, 1,579,486 samples, 60,706 packed rows; continued from the Stage 2 checkpoint, 119 steps ≈ 0.5 epoch, GBS=256, lr 2e-6→2e-7 cosine
• Training framework: MindSpeed-LLM v2.3.0 + Megatron-LM core_v0.12.1 (NPU)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("DALabCommunity/Haidass-Translate-143M", torch_dtype="bfloat16", device_map="auto")
tok = AutoTokenizer.from_pretrained("DALabCommunity/Haidass-Translate-143M")

msgs = [{"role": "user", "content": "将以下文本翻译为英文：光子甚至比构成原子的物质还要小！"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
# Photons are even smaller than the stuff that makes atoms!

The shipped generation_config.json is greedy decoding (do_sample=false, num_beams=1, repetition_penalty=1.0); plain model.generate(ids) reproduces the greedy scores.

To reproduce the beam row, use direction-specific beam search:

# English → Chinese
out = model.generate(ids, do_sample=False, num_beams=3, length_penalty=1.25,
                     repetition_penalty=1.0, max_new_tokens=256)
# Chinese → English
out = model.generate(ids, do_sample=False, num_beams=4, length_penalty=0.9,
                     repetition_penalty=1.0, max_new_tokens=256)

Beam costs roughly 3–4× the compute of greedy; use the greedy configuration for high-throughput serving.

Note: the training data follows the qwen3 chat template (an empty <think></think> block precedes the assistant turn). Always use the model's built-in chat_template at inference.

Translation samples (current release, spot check on FLORES-200 dev, beam decoding)

en→zh:

> Src: Latvia and Slovakia have both delayed the process of joining ACTA.
> Out: 拉脱维亚和斯洛伐克都推迟了加入ACTA的进程。 (sentence chrF++ 85.7)

> Src: The trial took place at Birmingham Crown Court and concluded on August 3.
> Out: 审判在伯明翰皇家法院进行，并于8月3日结束。 (sentence chrF++ 73.7)

zh→en:

> Src: 这些代理人负责根据《巴基斯坦宪法》第 247 条提供政府和司法服务。
> Out: These agents are responsible for providing government and judicial services under article 247 of the Pakistan Constitution. (sentence chrF++ 90.8)

> Src: 循环系统的主要器官是心脏，心脏负责输送血液。
> Out: The main organ of the circulatory system is the heart, which pumps blood. (sentence chrF++ 82.8)

Known limitations

• zh→en trails en→zh — a pattern shared by every model in the tables, not specific to this model
• Typical residual errors: occasional entity mix-ups and numeric-detail slips
• Optimized for zh⇄en translation only; not a general chat model

Evaluation

• Metrics: sacreBLEU corpus BLEU (tokenize=zh for Chinese targets, tokenize=13a for English) + chrF++ (word_order=2); prompts byte-identical to the training chat template
• Benchmark sources: openlanguagedata/flores_plus (FLORES+, the maintained version; gated — auto-approved after accepting terms; Simplified Chinese now cmn_Hans); facebook/flores (original archive, unmaintained); login-free mirror facebookresearch/flores
• Raw predictions: the eval/ directory in this repo contains per-sentence prediction jsonl ({direction, src, ref, hyp}, one file per model per benchmark; this release ships pred_haidass-143m_greedy_* and pred_haidass-143m_beam_*, one set per decoding config) for every model in the tables above — all scores can be recomputed with sacreBLEU
• Full results & reproduction: the 16-model comparison table, per-sentence predictions, decontamination audits and evaluation scripts live in the companion dataset umeiko/Haidass-Translate-143M-eval

_{Note: all scores are measured on the FLORES Chinese–English subset (eng_Latn ↔ zho_Hans), bidirectional (997 sentences for dev, 1,012 for devtest). The plain 143M row uses pure greedy decoding (do_sample=false, num_beams=1, repetition_penalty=1.0); the 143M-beam row uses direction-specific deterministic beam search (en→zh num_beams=3, length_penalty=1.25; zh→en num_beams=4, length_penalty=0.9); all other models use pure greedy decoding.}