Keural-MoE-14B

Keural-MoE-14B

Keural is a bilingual Korean–English Mixture-of-Experts language model trained entirely from scratch by MKD Corp AI Research, Republic of Korea. This is the final DPO Round 2 checkpoint at step 7,590 (100% complete), trained on 485,793 preference pairs on top of SFT Epoch 3.

Model Details

Full Training Pipeline

DPO Round 2 Dataset (485,793 pairs)

DPO Training Details

Special Tokens

Critical: Always use eos_token_id=131073. The model outputs <|im_end|> (ID 131073) to stop — not <eos> (ID 2).

Chat Format (ChatML)

<|im_start|>system
You are a helpful, accurate, and safe bilingual Korean-English AI assistant. Give concise, factual, and correct answers. If you are not sure about something, say you don't know instead of guessing. Never provide harmful, dangerous, illegal, or false information.<|im_end|>
<|im_start|>user
Your question here<|im_end|>
<|im_start|>assistant

Usage (Transformers)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "mkd-ai/Keural-MoE-14B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "system", "content": "You are a helpful bilingual Korean-English AI assistant."},
    {"role": "user",   "content": "안녕하세요! 서울에 대해 알려주세요."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.1,
    do_sample=True,
    eos_token_id=131073,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Usage (vLLM)

python -m vllm.entrypoints.openai.api_server \
    --model mkd-ai/Keural-MoE-14B \
    --dtype auto \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.7 \
    --trust-remote-code

Evaluation (Open LLM Leaderboard Benchmarks)

Keural-MoE-14B was evaluated on 6 standard benchmarks used by the Open LLM Leaderboard.

Results

Benchmark Charts

Analysis

Keural-MoE-14B was trained from scratch on ~69B tokens. Reference models (Mixtral, LLaMA-2, Qwen) were pretrained on trillions of tokens. Given the 50x+ difference in pretraining data, the scores reflect the expected scaling behavior:

• Winogrande (52.4%) — above random baseline (50%), indicating learned language understanding
• TruthfulQA (41.8%) — competitive with LLaMA-2-13B (36.9%), showing DPO alignment effectiveness
• GSM8K (0.2%) — math/code data was intentionally removed from SFT training to reduce structured task bias

These benchmarks establish a baseline. Future versions trained on larger corpora will show significant improvements.

Hardware

Trained on 2× NVIDIA H200 SXM (139 GiB each) using FSDP FULL_SHARD, bfloat16 mixed precision, and gradient checkpointing.

Training Source Code

https://github.com/MKD-CORP/Keural-Model-Training

Organization

Developed by MKD Corp AI Research, Republic of Korea.

License