Jeju-Standard_Korean_Translator

Jeju ↔ Standard Korean Translator

A compact (≈88M parameter) decoder-only language model trained from scratch for bidirectional translation between the Jeju dialect (제주 방언, Jejueo) and Standard Korean (표준어). The model uses a Qwen3-style architecture with per-head QK-Norm and is served as a single checkpoint that handles both translation directions via a prefix control token.

88M 디코더 전용 LLM을 1.4M개의 제주 방언↔표준어 평행 코퍼스로 처음부터 학습한 양방향 번역 모델입니다. 단일 모델·단일 체크포인트로 두 방향을 모두 처리합니다.

---

✨ Highlights

• From-scratch pretraining: no parent checkpoint; trained on a single H100 in ~4 hours.
• One model, two directions: prefix tokens <d2s> / <s2d> switch translation direction.
• Open evaluation: BLEU 77.67 (방언→표준) / 60.97 (표준→방언) on a 36,930-pair held-out test set.
• Drop-in HF / vLLM compatible: registered as Qwen3ForCausalLM, no custom code required.
• Small footprint: 178 MB safetensors, runs comfortably on consumer GPUs.

---

📋 Model Details

---

🎯 Intended Use

• Translation between the Jeju dialect and Standard Korean in either direction.
• Research on low-resource Korean dialect modeling, dialect-aware tokenization, and small-scale from-scratch pretraining.
• A reproducible baseline for future Jeju-dialect NLP work (back-translation, speaker-conditional generation, dialect-aware ASR post-correction, etc.).

Out-of-scope

• General-purpose chat / instruction following — this model is not an assistant.
• Translation involving languages other than Korean.
• Domains far from the training distribution (legal, code, news headlines, etc.). The training corpus is conversational AIHUB transcripts, so generations on formal or technical text may degrade.

---

🚀 Quick Start

Prompt format

The model is trained with a strict 4-token prompt scheme. Always begin with <bos>, add the direction tag, then the source text, then <sep>. The model generates until <eos>.

<bos><d2s>{ Jeju dialect text }<sep>     # 방언 → 표준
<bos><s2d>{ Standard Korean text }<sep>  # 표준 → 방언

Inference with 🤗 Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "postcn/jeju-korean-translator"
device = "cuda" if torch.cuda.is_available() else "cpu"

tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16).to(device).eval()

BOS = tok.convert_tokens_to_ids("<bos>")
SEP = tok.convert_tokens_to_ids("<sep>")
EOS = tok.convert_tokens_to_ids("<eos>")

def translate(text: str, direction: str = "<d2s>") -> str:
    """direction = '<d2s>' (방언→표준) or '<s2d>' (표준→방언)"""
    dir_id = tok.convert_tokens_to_ids(direction)
    ids = [BOS, dir_id] + tok.encode(text, add_special_tokens=False) + [SEP]
    inp = torch.tensor([ids], device=model.device)
    out = model.generate(
        inp,
        max_new_tokens=96,
        do_sample=False,
        num_beams=4,
        eos_token_id=EOS,
        pad_token_id=tok.pad_token_id,
    )
    gen = out[0, inp.shape[1]:].tolist()
    if EOS in gen:
        gen = gen[:gen.index(EOS)]
    return tok.decode(gen, skip_special_tokens=True).strip()

# Jeju → Standard
print(translate("글로 죽 가당 보믄 큰큰헌 소낭이 나옵니다게.", "<d2s>"))
# Standard → Jeju
print(translate("저기로 쭉 가다 보면 큰 소나무가 나옵니다.", "<s2d>"))

Serving with vLLM

The model is a stock Qwen3ForCausalLM, so it works with vLLM out of the box:

vllm serve postcn/jeju-korean-translator \
  --host 0.0.0.0 --port 8001 \
  --max-model-len 1024 \
  --dtype bfloat16

OpenAI-compatible client call:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8001/v1", api_key="sk-dummy")

resp = client.completions.create(
    model="postcn/jeju-korean-translator",
    prompt="<bos><s2d>제주도에는 수많은 관광지가 있습니다.<sep>",
    max_tokens=64,
    temperature=0.0,
    stop=["<eos>", "<bos>", "<sep>", "<d2s>", "<s2d>"],
)
print(resp.choices[0].text)

Tip. Greedy or beam-4 decoding gives the best BLEU. Sampling (temperature > 0) is rarely useful for this task — the target translation is well-defined.

---

📚 Training Data

Preprocessing pipeline:

1. Normalize — NFC unicode (preserving archaic syllables like ᆞ), quote standardization, whitespace canonicalization.
2. Dedup — exact (dialect_norm, standard_norm) deduplication while preserving conversation order.
3. Filter — drop pairs shorter than 3 chars, length-ratio > 0.7, or pairs where dialect == standard (keep only 10 % of identical pairs as a copy-task signal).
4. Group split — group-of-30 split (seed=20260417) so that the same dialogue session never crosses the train/val/test boundary.

The corpus originates from the AIHUB Jeju dialect dataset (annotated by Saltlux / PCN, 2020). Speaker distribution: 76 % female / 24 % male; primarily 20s (50 %), 50s (24 %), and 60+ (14 %).

---

🏋️ Training Procedure

Inputs are packed: each pair is encoded as

[bos][dir_tag][src_tokens...][sep][tgt_tokens...][eos]

Multiple pairs are packed per training sample using flash-attention's cu_seqlens varlen kernel. A 5 % self-copy auxiliary task (dialect→dialect, standard→standard via the <copy> tag) is mixed in to anchor identity behavior.

Training corpus size

• 2,876,856 packed sequences
• 69.0 M total tokens
• 31.6 M supervised target tokens

📊 Evaluation

Evaluated with sacreBLEU (corpus-level), CHRF++ (char order 6, word order 2, β=2, eps smoothing), and normalized Exact Match. Decoding: beam search (beam=4).

Test set (n = 36,930 pairs)

The <d2s> direction is consistently easier than <s2d> — generating dialect requires broader lexical and morphological coverage, while normalizing dialect into standard Korean is closer to a many-to-one mapping.

Sample translations

---

🧠 Special Tokens

A valid prompt must begin with <bos> followed immediately by exactly one of <d2s> / <s2d> / <copy>. Omitting either token will produce undefined behavior.

---

⚠️ Limitations and Bias

• Domain skew. Training data is conversational AIHUB transcripts. The model has not seen formal documents, news, or technical text. Translating outside this domain will degrade quality.
• Speaker skew. The corpus is 76 % female and skewed toward 20s and 50s speakers. Dialect realizations from older male speakers or rare regional sub-dialects may be underrepresented.
• Capacity. At 88 M parameters, the model is far below the Chinchilla-optimal token count for its size. It works because translation is a narrow task — but it will not generalize to open-ended language modeling.
• Hallucination on long inputs. max_position_embeddings = 1024. Inputs much longer than typical training sequences (~24 tokens average) may degrade.
• No safety alignment. This is a base translation model, not an instruction- or safety-tuned assistant. Treat outputs as raw translations and review them for sensitive applications.
• Morphological retention. A custom probe shows the model preserves dialect- specific endings (어미) ~74-78 % of the time; failures often manifest as over-standardization in the <s2d> direction.

---

🔬 Reproducibility

The full training pipeline (data build, tokenizer training, packing, training, and evaluation) lives in the parent project repository as YAML configs and shell scripts under configs/ and scripts/, with the training entry point at src/train/train.py.

Random seed: 42 for training, 20260417 for data splitting.

---

📜 License

This model is released under the Apache 2.0 license.

The training data is sourced from the AIHUB Jeju dialect corpus. Downstream users must independently verify and comply with AIHUB's terms of use for the underlying data, particularly for commercial deployments. This release distributes only the trained model weights, not the data.

---

📝 Citation

If you use this model, please cite:

@misc{jeju_korean_translator_2026,
  title  = {Jeju ↔ Standard Korean Translator: A Bidirectional Dialect
            Translator Trained from Scratch},
  author = {PCN R&S LLM Team},
  year   = {2026},
  note   = {88M-parameter Qwen3-style decoder, trained on 1.4M AIHUB Jeju
            dialect pairs.}
}

Please also acknowledge the underlying data source:

AIHUB. Jeju Dialect Speech / Text Corpus. National Information Society Agency of Korea. https://aihub.or.kr/