keural-sft3-final

Keural-SFT3-Final — 14.83B Bilingual MoE (SFT Epoch 3)

Keural is a 14.83B parameter Mixture-of-Experts language model trained from scratch for bilingual Korean–English instruction following.

This checkpoint is the result of SFT Epoch 3 (65,849 steps, 2.35M samples) and serves as the base model for DPO Round 2. The final preference-optimised model is mkd-hossain/keural-dpo2-final.

---

Architecture

KeuralMoECausalLM is a custom architecture registered via trust_remote_code=True.

---

Special Tokens

Critical: Always set eos_token_id=131073. Do not use ID 2 for chat generation.

---

Full Training Pipeline

---

SFT Epoch 3 Dataset (2,351,212 samples)

---

SFT Epoch 3 Hyperparameters

---

Chat Format (ChatML)

<|im_start|>system
You are a helpful, accurate, and safe bilingual Korean-English AI assistant.<|im_end|>
<|im_start|>user
Your question here<|im_end|>
<|im_start|>assistant

---

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained(
    "mkd-hossain/keural-sft3-final",
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "mkd-hossain/keural-sft3-final",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "system", "content": "You are a helpful bilingual Korean-English AI assistant."},
    {"role": "user",   "content": "안녕하세요! 오늘 날씨가 어때요?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    eos_token_id=131073,
    do_sample=True,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

---

Next Model

DPO Round 2 has been completed on top of this checkpoint using 485,793 preference pairs (8 sources). The final preference-optimised model is available at: mkd-hossain/keural-dpo2-final

---

License

Apache 2.0

Training data includes datasets from AI Hub (Korean government open data platform) and publicly available English instruction datasets. All sources are Apache 2.0 or CC-BY compatible.