hy-mt2-1-8b-2bit-translation-gguf

Hy-MT2 1.8B 2-bit Translation — GGUF

This repository packages a low-bit GGUF translation model derived from tencent/Hy-MT2-1.8B for multilingual translation with llama.cpp. Source and destination languages are runtime-configurable and default to English-to-Chinese translation.

✨ Key Highlights

• Faster translation — compared with the Q8_0 baseline on an Android™ Vivo X300 using one Arm® CPU core:
  • Prompt prefill: 1.39× faster.
  • Decode throughput: 49.60 tokens per second, 1.98× faster.
• Lower memory use — 62.36% lower average memory use than the Q8_0 baseline on an Android™ Vivo X300 using one Arm® CPU core.
• Smaller model — 2.89× smaller than the Q8_0 baseline.
• Low-bit model weights — uses proposed Q2_0C 2-bit per-channel quantization with Q8_0 token embedding tensors.
• Edge deployment — capable of running efficiently on edge devices such as Raspberry Pi 5.
• Multilingual translation — translates text across Chinese, English, French, Spanish, Italian, Japanese, Arabic, Hindi, and many more languages.

📦 Model Details

Model Description

Hy-MT2 1.8B is a multilingual translation model developed by Tencent Hunyuan. This package is derived from Tencent's 2-bit GGUF release and uses a higher-precision representation for the token embeddings.

• Developed by: Tencent Hunyuan
• Model type: Multilingual text translation
• License: Apache 2.0
• Base model: tencent/Hy-MT2-1.8B
• Source GGUF: tencent/Hy-MT2-1.8B-2Bit-GGUF
• Package form: GGUF model and JSON manifest

Model Sources

• Base model: https://huggingface.co/tencent/Hy-MT2-1.8B
• Source 2-bit GGUF: https://huggingface.co/tencent/Hy-MT2-1.8B-2Bit-GGUF
• Upstream repository: https://github.com/Tencent-Hunyuan/Hy-MT

🚀 Get Started with the Model

Run with Compute Flow

Follow these steps to run Hy-MT2 1.8B 2-bit Translation — GGUF on macOS® with Apple Silicon or on Linux® with an Arm®-based CPU, such as a Raspberry Pi® 5.

Step 1: Clone Compute Flow and set up the virtual environment

git clone https://github.com/ARM-software/ComputeFlow.git && cd ComputeFlow

uv sync && source .venv/bin/activate

Step 2: Install the brick and model

compute-flow update
compute-flow model install lm_text:lm-text-generic hy-mt2-1-8b-2bit-translation
compute-flow build native

Step 3: Run the Python CLI example

python -m cli_examples.lm_text_cli \
  -m @manifest/lm-text-generic/hy-mt2-1-8b-2bit-translation/hy_mt2_1_8b_manifest.json \
  -p "Hello, how are you?" \
  -n 64 \
  -t 1

The bundled manifest translates the example English text into Chinese by default.

📊 Quality evaluation

Quality evaluation results will be provided in a future update.

🎯 Performance evaluation

Performance was measured under the following conditions:

• Target: Android™ Vivo X300 smartphone.
• CPU: One Arm® CPU core.
• Runs: 5 warmups followed by 30 measured fresh-process runs.
• Pause: 60 seconds between processes.
• Target state: The smartphone screen was kept on.

The following methodology and definitions were used:

• Runtime: llama.cpp with KleidiAI on CPU.
• Input: One fixed English-to-Chinese translation prompt, with identical manifest settings and a deterministic sampling seed for both packages.
• Generation limit: At most 64 generated tokens.
• Prompt prefill latency is the first model generation call, including prompt processing and the first generated token.
• Decode throughput is the rate of subsequent generated tokens and excludes the first model generation call.
• Average memory is the mean process RSS sampled every 0.2 seconds across measured runs.
• Peak memory is the maximum process high-water mark across measured runs.

Lower latency and memory are better, while higher decode throughput is better.

The Q8_0 baseline uses the original tencent/Hy-MT2-1.8B-GGUF model.

Machine-readable results are provided in:

• benchmarks/hy-mt2-1-8b-2bit-translation-vivo-x300.yaml
• benchmarks/hy-mt2-1-8b-2bit-translation-vivo-x300-baseline.yaml

🛠️ Technical Specifications

Objective

Translate input text between user-selected source and destination languages.

Runtime Architecture

Precision and Quantization

The main model weights use Q2_0C 2-bit per-channel quantization. Token embedding tensors use Q8_0 quantization.

Q2_0C is a proposed 2-bit data type introduced in llama.cpp PR #19357 that applies per-channel quantization. Its 2-bit scheme uses Stretched Elastic Quantization (SEQ), introduced by Liu et al. (2026), to quantize weights to four levels: -1.5, -0.5, +0.5, and +1.5.

Input Specification

Output Specification

The model returns translated UTF-8 text in the selected destination language.

Manifest Configuration

Source and destination languages can be changed between requests without reloading the model.

Repository Contents

• hy_mt2_1_8b_manifest.json — model package manifest.
• *.gguf — quantized translation model.
• metadata.yaml — model and benchmark metadata.
• benchmarks/*.yaml — optimized and baseline benchmark reports.
• SHA256SUMS — model-bundle checksums for reproducibility.

🗂️ Model and Asset Origin

• Tencent Hy-MT2 1.8B base model
• Tencent Hy-MT2 1.8B 2-bit GGUF
• Tencent Hunyuan Hy-MT source repository
• llama.cpp revision used for token-embedding requantization

The packaged GGUF was derived from the official 2-bit release and requantized with Q8_0 token embedding tensors.

🔐 Checksums

SHA256SUMS was generated by recursively hashing every regular file in the model bundle, including files in subdirectories, except the generated root SHA256SUMS and paths with a dotfile component.

From the model bundle root, verify the checked-out files with:

shasum -a 256 -c SHA256SUMS