Keural-Orbit-v1.0

Keural Orbit v1.0

Keural Orbit is MKD's Korean-specialized assistant model — a LoRA supervised fine-tune of google/gemma-4-26B-A4B-it (Gemma 4, 26B total / ~4B active Mixture-of-Experts, Image-Text-to-Text), merged back into full weights.

It is tuned for natural, concise Korean conversation with a consistent Keural Orbit identity — built and trained by MKD.

Highlights

• 🇰🇷 Korean-first conversational assistant — fluent, concise, and complete answers.
• 🪪 Keural Orbit identity — self-identifies as MKD's Keural Orbit (not the base model).
• 🧩 Efficient MoE — 26B total / ~4B active per token: large-model quality at a fraction of the inference cost.
• 🖼️ Multimodal-capable base (Image-Text-to-Text); 256K context.

1. Identity

Keural Orbit consistently identifies as MKD's Keural Orbit and does not present itself as the base model:

2. Korean generation

Keural Orbit answers in concise, direct Korean and reliably completes its answers, where the base model tends toward longer, multi-option responses. Example:

User: 회사에 지각했을 때 상사에게 보낼 정중한 사과 메시지를 써 줘.

Keural Orbit: "갑작스러운 사정으로 지각하게 되어 정말 죄송합니다. 최대한 빨리 도착해서 업무에 차질 없도록 하겠습니다. 너그러운 마음으로 이해해 주시면 감사하겠습니다."

3. Benchmarks (full transparency)

We also report standard benchmark scores, run identically on both models for a fair apples-to-apples comparison.

Please read the caveat. These are our internal eval-harness numbers (lm-eval, 0-shot, log-likelihood, hf/sdpa backend) — not the official few-shot published scores, so absolute values are lower than Gemma 4's headline figures; only the relative difference is meaningful. Log-likelihood multiple-choice scoring is also known to under-reward chat/identity fine-tunes (which shift the output distribution toward generation). Keural Orbit's design goal is Korean conversation + identity, not raw MCQ accuracy — and on this metric the base model retains an edge:

If your use case depends on maximum multiple-choice benchmark scores, evaluate carefully for your task before adopting.

Training

• Method: LoRA SFT (rank 16, α 32, dropout 0), 1 epoch, learning rate 1e-5 (cosine), bf16, DeepSpeed ZeRO-2, on 2× H200.
• Adapted: the language-model attention (q/k/v/o_proj), dense MLP (gate/up/down_proj), and the 128 fused routed experts (gate_up_proj / down_proj) on all 30 layers.
• Frozen: the router, the vision tower, and the tied input/output embeddings — verified pre- and post-training. The merge is byte-verified (untargeted tensors identical to base).
• Data: ~146k rows — Korean conversation, clean code, math, MCQ, long-form, English replay, plus a hand-seeded Keural Orbit identity set (with Gemma/Google de-contamination).

Usage

from transformers import AutoProcessor, Gemma4ForConditionalGeneration

repo = "mkd-hossain/Keural-Orbit-v1.0"
processor = AutoProcessor.from_pretrained(repo)
model = Gemma4ForConditionalGeneration.from_pretrained(
    repo, dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": [{"type": "text", "text": "너 누구야?"}]}]
inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

• Korean-focused; for non-Korean tasks expect roughly base-model behavior.
• The vision tower is frozen and untested by this fine-tune (multimodal behavior is inherited).
• On raw multiple-choice knowledge benchmarks, the base model scores higher (see table above).
• Context length matches the base (256K); no YaRN extension is applied by this fine-tune.

Provenance & license

• Base: google/gemma-4-26B-A4B-it (Gemma Terms of Use — Google DeepMind). Use of this model is subject to the Gemma license.
• Fine-tuning, evaluation, and packaging by MKD. Merge is a CPU weight-level LoRA merge with byte-level verification (fused-expert-aware, tied-embedding preserving).