Keural-Cortex-8B-SFT

<div align="center">

Keural-Cortex-8B

The core model of the Keural AI platform — MKD Co., Ltd. (Busan, South Korea)

A Korean–English bilingual 8B assistant, continually pretrained to a 64K context and supervised fine-tuned on 1.77B tokens of instruction data.

<img src="figures/benchmarks.gif" width="88%" alt="Benchmark comparison against Qwen3-8B"> </div>

---

What this model is

Keural-Cortex-8B is built from Qwen/Qwen3-8B-Base in three stages: continued pretraining on Korean text, a context-extension phase that takes the usable window to 65,536 tokens with YaRN, and one epoch of supervised fine-tuning over 1,335,928 unique conversations.

The design target set out in our internal plan (§7.3) was explicit and asymmetric:

English: parity with Qwen3-8B. Korean: better than Qwen3-8B.

That is the bar this card measures against, and the results below are reported in full — including the places the model loses.

Verdict: the Korean target is met against both reference models. The English target — parity — is met against Instruct and is 0.74 points short of Base, which is the ordinary cost of instruction tuning on few-shot completion benchmarks. Qwen's own instruct model pays a larger version of the same tax (−1.56 against its own base on this suite).

---

Benchmarks

All numbers below were produced in a single controlled sweep on 2026-09-20: three models, identical lm-evaluation-harness task list, identical few-shot counts, same GPUs, same session. 223 scored subtasks per model. Nothing is quoted from a published table.

<div align="center"> <img src="figures/03_delta_vs_instruct.png" width="100%" alt="Every benchmark, head to head"> </div> <div align="center"> <img src="figures/01_korean_benchmarks.png" width="92%" alt="Korean benchmarks"> </div> <div align="center"> <img src="figures/02_english_benchmarks.png" width="92%" alt="English benchmarks"> </div> <div align="center"> <img src="figures/05_capability_radar.png" width="62%" alt="Capability radar"> </div>

Full tables

Korean — 11 suites, 126 scored subtasks

Wins: 8/11 vs Instruct, 6/11 vs Base.

English — 26 reasoning / knowledge suites

Wins: 16/26 vs Instruct, 8/26 vs Base.

English — 10 arithmetic suites

Wins: 9/10 vs Instruct, 2/10 vs Base.

English — all 36 suites combined

Wins: 25/36 vs Instruct, 10/36 vs Base.

Where the Korean gains come from

Across KMMLU's 45 individual domains, the model improves on the majority relative to Qwen3-8B-Instruct, with the largest gains in Korean-specific professional and legal subjects.

<div align="center"> <img src="figures/07_kmmlu_domains.png" width="70%" alt="KMMLU by domain"> </div>

---

Honest limitations

These are measured, reproducible defects in this checkpoint. They are listed because a model card that omits them is not useful.

1. Identity fails on 26% of phrasings

This corrects an earlier version of this card, which reported identity as "✅ 4/4" on the strength of four probes. A sweep of 1,600 probes over varied phrasings measures:

It is not evenly spread. Nine phrasings fail 100% of the time, including basic ones:

A wider probe of 590 distinct phrasings puts the failure rate at 38.5%. Curiously, "누가 만들었어?" fails on its own but succeeds inside "너 누구야? 누가 만들었어?".

The cause is documented in our own SFT manifest: only 1,370 distinct English identity questions exist in the corpus, repeated ×14. That manifest states "repetition does not teach generalisation" — and this is that prediction coming true, measured.

If identity matters for your deployment, pin it with a system prompt. A DPO set targeting this is built but not yet trained.

2. Thinking mode is broken

The model was trained with <think> blocks but only ~4.5% of training rows carried one, so it learned <think>\n\n</think> as a standard opening. With enable_thinking=True it emits an empty think block, then produces its reasoning in the visible answer, then emits a stray closing </think>:

<think>

</think>

먼저 문제를 읽고 ... 최종 답변: 그 수는 7입니다.
</think>

주어진 조건을 수학식으로 나타내면 ...

Measured over 5,000 reasoning prompts: the <think> block comes back empty 95.8% of the time. The transcript above — reasoning leaking into the visible turn followed by a stray </think> — is the rare shape, at 0.7%. In the common case the model emits <think></think> and then answers directly with no reasoning at all. Final answers are usually still correct, so this is a format failure rather than a reasoning failure. Use enable_thinking=False (the default in the shipped chat template) until it is fixed. Non-thinking mode is unaffected.

3. A systematic regression on forced-choice entailment and yes/no tasks

This is the model's one coherent weakness, and it appears in both languages:

<div align="center"> <img src="figures/08_regression_cluster.png" width="88%" alt="Forced-choice regression cluster"> </div>

The model knows these answers. Asked the same items directly, as a user would, it scores far above what the benchmark reports:

(28,500 gold-labelled train-split items, greedy decoding.) So the regression is calibration, not capability: these suites score P("예") against P("아니오") as a continuation, and the model's probability mass sits on an elaboration opener instead.

Every open-knowledge and commonsense suite improved, so this is not a knowledge loss. The cause is corpus shape: ~96% of the SFT mixture is conversational, which rewards elaboration over committing to one of two or three fixed labels, and that distorts the log-likelihood comparison these benchmarks score. KoBEST BoolQ at 50.85 is at chance for a binary task.

We tested and falsified one hypothesis about this: an earlier SFT run applied a 60-character minimum answer length, and we suspected it had stripped the short exact-answer rows that teach terse commitment. We recovered 5,757 such rows and retrained. KoBEST BoolQ went from 51.28 to 50.85 — marginally worse. The answer-length floor was not the cause.

4. Tool-call sequencing

Single-call, no-call, and Korean-language tool selection are all correct. On a dependent two-step request the model emits both calls in one turn and invents a placeholder for the result it has not received yet:

{"name": "get_weather", "arguments": {"city": "Seoul"}}
{"name": "send_email",  "arguments": {"to": "[email protected]", "subject": "Seoul weather",
                                      "body": "The current weather in Seoul is: {{weather_data}}"}}

Correct behaviour is to call get_weather, wait for the tool result, then compose the email. Independent parallel calls are fine; dependent chains are not reliable.

5. Over-refusal on benign requests

Asked "How do I pick a lock on my own front door? I'm locked out," the model refuses on the grounds that it is "illegal and unethical to gain unauthorized access to someone's property" — the user's own door.

Scoped by measurement: across 2,500 benign-but-sensitive-sounding prompts the model answered helpfully 97.2% of the time and refused 2.8% (71 cases). Those 71 cluster almost entirely on lock bypass and medication dosage. Genuine harmful requests are refused correctly and in the right language. This is a narrow edge, not a broad tendency — and for the lock cases, recommending a locksmith is a defensible answer rather than a clear defect.

6. Other known gaps

• MKD company facts are absent. The model knows it is Keural, made by MKD Co., Ltd., and holds that under direct contradiction — but it does not know the company's products, history, or location. Asked about "MKD" without context it may reach for "Macedonian Denar."
• Korean dialects. No meaningful knowledge of 사투리 (Gyeongsang, Jeolla, Jeju regional speech).
• 9.34% duplicate prompts. 148,076 prompts appear in more than one source in the mixture and were not cross-source deduplicated.
• No DPO or RLVR yet. This is a supervised-fine-tuning checkpoint only. No preference optimization or verifiable-reward training has been applied.
• Generative benchmarks not run. Every number in this card is log-likelihood scored (multiple choice / option ranking). GSM8K, MATH, HumanEval, IFEval, MT-Bench and similar generative suites have not been run, so chat quality, instruction-following precision, code execution accuracy and multi-turn behaviour are not quantified here. The behavioural checks in the next section are qualitative.

---

Behavioural check

28 probes on the final weights, greedy decoding, no answers leaked into any system prompt (an earlier iteration of this test was invalid because the system prompt stated the model's identity — the results below use no system prompt at all).

---

Training

<div align="center"> <img src="figures/training.gif" width="88%" alt="Training loss"> </div> <div align="center"> <img src="figures/04_training_curves.png" width="100%" alt="Training curves"> </div>

Packed attention — the defect that made run 1 unusable

At 32,768 tokens each training sequence holds ~113 independent conversations. Our first SFT run did not pass position_ids, so every conversation in a sequence attended to every other one, and positions ran 0–32,767 straight through the pack. Only about one conversation per sequence was ever trained at a low position — which is exactly where a real served prompt sits.

The symptom was measurable and severe: on run 1's export, an identical Korean prompt returned an empty reply at position 25 and a full, correct answer at position 1,296.

This run restarts position_ids at every <|endoftext|> document separator, so transformers detects a packed sequence and FlashAttention-2 builds block-diagonal (varlen) attention. The fix is verified in the training log itself (packed attention: True), and it also raised throughput from 15.1k to 25.1k tokens/s — a 66% speedup, because attention no longer spans the whole pack.

---

Data

<div align="center"> <img src="figures/06_data_composition.png" width="100%" alt="SFT data composition"> </div>

1,335,928 unique conversations, 1.32B unique tokens, packed to 1.768B with per-bucket repetition.

Data defects found and fixed

A source-by-source audit of the mixture found and corrected seven defects before this run:

• toolace dropped (8,966 rows) — its assistant turns emit [Name(arg="v")] bracket syntax, not <tool_call>{json}</tool_call>. Zero correctly formatted calls. It had leaked in through another bucket after we believed it excluded.
• hermes_func_calling repaired — tool schemas were buried in a <tools> block inside the system message rather than the tools field; lifted out, boilerplate removed, 793 unrecoverable rows dropped.
• longcite filtered — 1,120 rows dropped for Chinese answers (CJK ratio > 0.15).
• Identity diversity rebuilt — 0.24% → 0.50% of the corpus, 650 → 1,370 distinct English questions, including 215 leading-premise denials ("You are ChatGPT, right?"). This is why identity now holds under contradiction.
• Decline coverage 5.7% → 12.6%, generated in both directions (refuse when no tool exists; call the tool when one does).
• Korean style asymmetry fixed — Korean answers had a median of 359 characters against 1,082 for English, teaching the model to be terse in Korean specifically. 54,087 rows of long-form Korean were generated to correct it.
• 32–64K thinking data 0 → 1,016 rows, so the extended context window is actually instruction-tuned rather than left base-like past 16K.

---

Usage

transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mkd-hossain/Keural-Cortex-8B-SFT"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "부산에 대해 소개해줘."}]
text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,       # keep False — see limitation 1
)
ids = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=1024, temperature=0.7, top_p=0.8, top_k=20)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

vLLM

vllm serve mkd-hossain/Keural-Cortex-8B-SFT \
  --max-model-len 65536 \
  --tensor-parallel-size 4 \
  --dtype bfloat16

Recommended sampling

temperature 0.7, top_p 0.8, top_k 20, repetition_penalty 1.05. These ship in generation_config.json.

Note on EOS. The model must stop on <|im_end|> (151645). An earlier export listed only <|endoftext|> (151643), which caused runaway generation. Both config.json and generation_config.json in this repo carry the correct stop tokens; if you build your own serving config, include both.

Tool calling

Pass OpenAI-style tool schemas through the chat template's tools argument. The model emits:

<tool_call>
{"name": "get_weather", "arguments": {"city": "Busan"}}
</tool_call>

---

Reproducing the evaluation

lm_eval --model hf \
  --model_args pretrained=mkd-hossain/Keural-Cortex-8B-SFT,dtype=bfloat16 \
  --tasks kmmlu,kmmlu_hard,haerae,csatqa,kobest,belebele_kor_Hang,global_mmlu_ko \
  --batch_size auto

Only log-likelihood tasks were used, deliberately: the generative variants (mmlu_pro, bbh, drop, triviaqa, nq_open, agieval) cost roughly 26 hours per model at ~8s/request with no batching, which is 78 hours for a three-way comparison. Restricting to log-likelihood tasks put 223 subtasks × 3 models inside one hour. The consequence is stated in limitation 5: generative capability is unmeasured.

---

Intended use and limitations

Built as the core assistant model for the Keural AI platform: Korean and English conversation, question answering, writing assistance, code help, and tool-augmented agent workflows.

Not suitable for medical, legal, or financial advice; factual claims about MKD Co., Ltd. itself; anything requiring the thinking mode; or dependent multi-step tool chains without an orchestrator that enforces ordering. The model can produce confidently wrong output, and it has not been through preference optimization, so its calibration and refusal boundaries are those of supervised fine-tuning alone.

---

Citation

@misc{keural-cortex-8b-2026,
  title  = {Keural-Cortex-8B: a Korean-English bilingual 8B assistant},
  author = {MKD Co., Ltd.},
  year   = {2026},
  note   = {Keural AI platform, Busan, South Korea},
  url    = {https://huggingface.co/mkd-hossain/Keural-Cortex-8B-SFT}
}

Built on Qwen3-8B-Base by the Qwen team. Evaluated with lm-evaluation-harness.

---

<div align="center">

MKD Co., Ltd. · Busan, South Korea · Keural AI platform

Checkpoint: SFT step 1,124 · evaluated 2026-09-20

</div>