keural-sft3-60k

Keural-SFT3-14.83B (SFT Epoch 3 — 60,000 steps)

Keural is a bilingual Korean–English Mixture-of-Experts language model trained entirely from scratch — no base model was used. This is an intermediate SFT epoch 3 checkpoint at step 60,000 out of 65,849 total steps (91.1% complete), trained on a 2.35M sample merged bilingual dataset.

Model Details

Full Training Pipeline

SFT Epoch 3 Dataset (2,351,212 samples)

Chat Format (ChatML)

<|im_start|>system
You are a helpful, accurate, and safe bilingual Korean-English AI assistant. Give concise, factual, and correct answers. If you are not sure about something, say you don't know instead of guessing. Never provide harmful, dangerous, illegal, or false information. Keep responses short and to the point unless the user asks for more detail.<|im_end|>
<|im_start|>user
Your question here<|im_end|>
<|im_start|>assistant

Special Tokens

Important: Always set eos_token_id=131073. Do NOT use ID 2.

Usage (vLLM)

python -m vllm.entrypoints.openai.api_server \
    --model mkd-hossain/keural-sft3-60k \
    --dtype auto \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.7

Usage (Transformers)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "mkd-hossain/keural-sft3-60k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful bilingual Korean-English AI assistant."},
    {"role": "user", "content": "안녕하세요! 서울에 대해 알려주세요."}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    do_sample=True,
    eos_token_id=131073,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Hardware

Trained on 2× NVIDIA H200 SXM (139 GiB each) using FSDP FULL_SHARD, bfloat16 mixed precision, and gradient checkpointing.