Keural-Cortex-8B-step5000

Keural Cortex 8B — intermediate checkpoint (step 5000)

This is a mid-training checkpoint, not a finished model. It is published for internal inspection and reproducibility, not for use.

Continued pretraining (CPT) of Qwen/Qwen3-8B-Base on a Korean-focused corpus, by MKD Co., Ltd.

Status

What this model cannot do

It is a base model. Qwen3-8B-Base has no instruction following, no chat format, no tool calling and no thinking mode, and CPT does not add any of them — those are built in post-training, which has not been done yet.

In practice it continues text rather than answering instructions. It will ramble, drift off topic, or repeat. A chat_template.jinja is present because it is inherited from the base tokenizer; it does not mean the model is chat-tuned.

Training

• 4× H200, FSDP full shard, bf16 with fp32 gradient reduction
• seq_len 4,096, global batch 3,145,728 tokens/step
• Peak LR 1e-5, WSD schedule, grad clip 1.0
• Corpus: 38.78B tokens across 19 sources (Korean, English, code, science/math), license-audited for commercial use

Evaluation

No evaluation has been run on this checkpoint. The numbers below are from the 3B-token pilot (a separate, earlier run) against Qwen3-8B-Base, and are included only to show the recipe was validated before the main run:

Caveat worth stating plainly: in that pilot, KMMLU (+0.03) and HaeRae (+0.09) were essentially flat, and the Korean average was carried by KoBEST BoolQ (+5.91) while two KoBEST tasks regressed. The Korean gain magnitude for the full run is not yet established.

Benchmark decontamination (n-gram overlap against MMLU, GSM8K, KMMLU, HaeRae, KoBEST and others) has not yet been completed. No benchmark number from this project should be treated as publishable until it has.

License and attribution

Apache 2.0, inherited from Qwen/Qwen3-8B-Base. This is a derivative work of Qwen3-8B-Base by Alibaba Cloud.

Intended use

Internal research and checkpoint inspection only. Not for production, not for public-facing deployment, and not a basis for capability claims.