sion_translate

sion_translate v1 Korean–Japanese Translation

KJ-X is a custom PyTorch encoder-decoder model for bidirectional Korean↔Japanese translation. The uploaded release contains inference exports and its SentencePiece tokenizer; training datasets and training checkpoints are not distributed.

v1 numeric inference update — runtime 0.1.2

Existing v1 users: reinstall the updated inference code. Existing verified v1 weights and tokenizer can be reused; no retraining or weight replacement is needed. The follow-up preserves spaced signs, repairs recognized measurement-unit suffixes (including compound units and powers), protects previously unrecognized numeric suffixes, and keeps ordinary uppercase acronyms visible during translation.

Actual v1 EMA execution now preserves ± 0.05, 250mL, 3,850km², 15mile, and 40mile in the previously failing cases. The five identified acronym regressions are also corrected. This remains a partial inference correction: arbitrary units, word-written numbers, arithmetic, and full numerical semantics are not guaranteed. Some numerical omissions still cause an error instead of a translation.

The focused follow-up attempted 238 inputs, returned 236 outputs, and blocked 2. It includes 72 authored numeric stress inputs, 16 existing diagnostics, and every affected FLORES input. Average chrF++ on the rerun no-digit subset increased in both directions; individual regressions are still possible. This was not a fresh full FLORES rerun. Related regression tests: 98 passed.

한국어 안내: 부호·단위·숫자 인식 누락·약어 회귀를 재학습 없이 보완했습니다. 기존 사용자는 추론 코드를 다시 받으세요. 기존 가중치는 재사용 가능합니다. 숫자 처리가 완벽하다는 의미는 아니며, 일부 값 누락은 오류로 차단됩니다.

Runtime commit: 45254b8cb84e.

See details and evidence.

Files

• model_ema.pt: recommended FP32 EMA inference export
• model_int8.pt: legacy pickled CPU-only dynamic INT8 export, kept for compatibility; prefer portable/model_int8_state_dict.pt
• portable/: FP32 safetensors and a plain INT8 state_dict with a SHA-256 manifest (see Portable artifacts)
• kjx.model, kjx.vocab, token_features.npz: tokenizer files

The model uses custom code from gaon12/sion_translate and is not a Transformers AutoModel checkpoint.

Quick start

python -m pip install --upgrade "sion-translate @ git+https://github.com/gaon12/sion_translate.git@45254b8cb84e7725aa7b58709f060317ff4e2ab6"
python -m pip install huggingface_hub
hf download gaon12/sion_translate \
  model_ema.pt kjx.model kjx.vocab token_features.npz inference_v1.py v1_numeric_hotfix.json \
  --local-dir models/sion_translate

from sion_translate.inference import Translator

translator = Translator(
    "models/sion_translate/model_ema.pt",
    "models/sion_translate/kjx.model",
    device="cuda",  # Use "cpu" when CUDA is unavailable.
)
print(translator.translate(["안녕하세요."], target_language="ja", num_beams=4)[0])
print(translator.translate(["こんにちは。"], target_language="ko", num_beams=4)[0])

Translator automatically finds token_features.npz beside the model/tokenizer. The model is custom code, so transformers.AutoModel and hosted Transformers inference are not supported.

The numeric update was verified with model_ema.pt on CPU. Use that EMA file for the safe weights-only path; the legacy pickled INT8 download is not covered by this validation.

You can also run python models/sion_translate/inference_v1.py --text "17시 30분" --target-language ja. The helper verifies the downloaded EMA, tokenizer, and feature-file hashes.

Reproducible smoke test

After downloading the files above, run this from a clean environment to exercise both directions:

python - <<'PY'
from sion_translate.inference import Translator

translator = Translator("models/sion_translate/model_ema.pt", "models/sion_translate/kjx.model")
for text, target in [("안녕하세요.", "ja"), ("こんにちは。", "ko")]:
    output = translator.translate([text], target_language=target, num_beams=1)[0]
    assert isinstance(output, str) and output.strip(), output
    print(f"{text} -> {output}")
PY

Portable artifacts (portable/)

portable/ holds plain-tensor copies of model_ema.pt that load with strict=True and without executing pickled classes. sion_portable.py checks the size and SHA-256 of every file listed in portable_manifest.json before it builds the model.

sion_portable.py (loader), translate_portable.py (CLI), and portable_validation.json (per-sentence outputs) are in the same folder.

• Source: gaon12/sion_translate revision 1f106d9caa22, model_ema.pt (SHA-256 829580d1db1693ec00ee641398ce6ba9549388e266f6259f85e5a19ea28bfc2e)
• Runtime: sion-translate commit 45254b8cb84e
• INT8 recipe: dynamic qint8 weights (per output channel) for 200 encoder/decoder nn.Linear modules, onednn engine, no calibration data, CPU only

python -m pip install "sion-translate @ git+https://github.com/gaon12/sion_translate.git@45254b8cb84e7725aa7b58709f060317ff4e2ab6" safetensors huggingface_hub
hf download gaon12/sion_translate --include "portable/*" --local-dir sion_translate
python sion_translate/portable/translate_portable.py --precision fp32 --target-language ja --text "회의는 17시 30분에 시작합니다."

import sys
sys.path.insert(0, "sion_translate/portable")
import sion_portable
from sion_translate.inference import Translator

sion_portable.install()  # Translator now accepts portable weight files
translator = Translator("sion_translate/portable/model_fp32.safetensors", "sion_translate/portable/kjx.model", device="cpu")

Validation (40-sentence diagnostic set, CPU, beam 4, torch 2.13.0+cpu):

FP32 is the reference portable format. INT8 trades a small quality loss for size and CPU speed; prefer FP32 when numbers must be preserved. The INT8 file uses PyTorch's quantized tensor types, which PyTorch has deprecated; it was verified with torch 2.13.0+cpu only.

Evaluation

The GitHub repository includes a small, independently authored diagnostic JSONL covering both directions and provides a common scorer for KJ-X, LibreTranslate, Papago, Google Translation, DeepL, M2M100 418M, and NLLB-200. No universal quality claim is made from that small set.

Limitations

• Machine translations can omit, add, or reverse meaning.
• Proper nouns, short context-dependent utterances, specialist terminology, and long inputs need human review.
• Do not use unreviewed output for medical, legal, safety-critical, or certified translation.
• The custom architecture requires the repository code; hosted Transformers inference is not available out of the box.

AI-Hub dataset attribution for public v1

The AI-Hub portion of the public v1 (KJ-X) training data includes the datasets below. We acknowledge AI-Hub (aihub.or.kr) and the dataset creators. This table credits those AI-Hub sources; it is not a complete inventory of all training sources.

AI-Hub's official FAQ permits commercial use and distribution of models developed using its data, with attribution to the dataset names and AI-Hub. Redistribution of original or merely edited source data is restricted. These terms apply to the AI-Hub sources listed here.

License

The original KJ-X code and uploaded model release are provided under the MIT License. This does not grant rights to user-provided inputs, generated service outputs, third-party software, or third-party models used only as comparison baselines.