Keural-Cortex-8B-pilot

Keural Cortex 8B — pilot run (step 954, 3B tokens)

This is a validation artifact, not a product. It exists to answer one question before committing ~19 GPU-days to a full run: does this continued-pretraining recipe move Korean up without moving English down? It is published for reproducibility and internal record, not for use.

Continued pretraining (CPT) of Qwen/Qwen3-8B-Base by MKD Co., Ltd. — Phase 3 ("pilot + gate") of the Keural Cortex 8B plan.

Why a pilot run exists

The project plan describes this phase as existing to "catch in one day what would otherwise be discovered on day 19." CPT on a converged base model can quietly damage it — the wrong learning rate, a bad domain mix, or a corpus defect will show up as English regression that no amount of later training undoes. A 3B-token run costs ~1 day and surfaces that before the 41B-token run starts.

Training

The learning rate is deliberately 1e-5, not a pretraining-scale 3e-4: pretraining LRs destroy a converged base. This value prioritizes retention over Korean acquisition speed.

Corpus: the same 19-source, 38.78B-token manifest as the full run (Korean, English, code, science/math), sampled proportionally by weight — so the pilot sees the same domain ratios, just fewer tokens.

Evaluation — the gate result

Gate criterion: Korean up AND English within −0.5 pt. Evaluated with lm-evaluation-harness against Qwen3-8B-Base under identical settings. Both models are base models, so this is a like-for-like comparison.

Korean

English

Verdict: PASS. Korean mean up, English mean up (well inside the −0.5 pt tolerance), no single English benchmark outside it. The full 41B-token run was launched on this basis.

How to read these numbers honestly

The headline "+0.55 Korean" overstates what was demonstrated:

• KMMLU (+0.03) and HaeRae (+0.09) are flat. These are the two substantive Korean knowledge benchmarks, and 3B tokens moved neither.
• The mean is carried by one task. KoBEST BoolQ (+5.91) is binary yes/no; a gain of that size on a 2-class task is as consistent with calibration shift as with knowledge gain. Two other KoBEST tasks regressed.
• So the gate proved the safe half of its question. It showed the recipe does not damage the model — English held and slightly improved, which is the genuinely valuable finding. It did not establish that Korean capability improves by a useful margin. The project's own plan records this as "Korean gain magnitude — direction confident, magnitude not."

Two further caveats:

• GSM8K is not comparable. Base was scored on all 1,319 samples (84.69 strict-match); the pilot was scored on a 300-sample subset (85.33) to work around a generation-throughput problem. Do not read a delta into that pair.
• Benchmark decontamination has not been run. N-gram overlap against MMLU, GSM8K, KMMLU, HaeRae, KoBEST and others is an outstanding item. Until it is done, treat every number here as internal-only and not publishable.

What this model cannot do

It is a base model. Qwen3-8B-Base has no instruction following, no chat format, no tool calling and no thinking mode; CPT adds none of them. It continues text rather than answering instructions, and will ramble or repeat. The chat_template.jinja is inherited from the base tokenizer and does not indicate chat tuning.

Context length is 32,768 (native, unchanged from base).

Relationship to other checkpoints

The pilot is a separate run from the main CPT, not an early checkpoint of it. Both start from Qwen3-8B-Base with the same recipe and manifest.

License and attribution

Apache 2.0, inherited from Qwen/Qwen3-8B-Base. This is a derivative work of Qwen3-8B-Base by Alibaba Cloud.

Intended use

Internal research, reproducibility and ablation reference only. Not for production, not for public-facing deployment, and not a basis for capability claims.