Qwen3.8-27B-Atlassian-Q4-mlx

Qwen3.8-27B-Atlassian-Q4-mlx

Smaller member of the LeanZero Atlassian family: the same trained Qwen3.8-27B at plain 4-bit (group size 64).

Release v0.5. The full write-up, with every test explained and every round compared, is on LeanZero's page: leanzero.net/portfolio/atlassian-models. The same evidence is reproduced below.

Built by LeanZero, the people behind CogniRunner (AI workflow rules for Jira), Sentinel Vault (Confluence attachment governance), LeanZero Management (PPM for Jira Cloud) and Atlassian migrations. Quantised with mlx-node and mlx-lm, trained with mlx-lm, served and evaluated with LeanZero's Rapid-MLX fork (engine wheel), all on one Apple Silicon Mac Studio.

What it is

The merged bf16 weights of Qwen3.8-27B-Atlassian (adapter merged at full precision) quantised with plain 4-bit (group size 64), ~15 GB by mlx-lm. MTP head shipped as mtp.safetensors. Agreement with the 8-bit model's teacher (base+adapter): top-1024 KLD 0.0969, top-1 95.74% (the 8-bit model: 0.0339 / 99.56%). Identifier probe on this quantisation, thinking on: 92% / 23% (8-bit model 85% / 15%). Pick this when memory matters more than that difference.

What it was trained on

Round 1: ~3.7M tokens, two epochs. Twelve Forge apps written by LeanZero plus five forge-sim fixtures; 175 module manifests generated from the @forge/manifest 13.4 schema and validated with Atlassian's validator, with task-description and fix-the-manifest pairs; UI Kit component examples (@forge/react 12) that compile against the real type definitions; facts from 1,820 Forge documentation sections, kept only when the answer is a verbatim quote; endpoint and scope facts from the Jira, Jira Software, JSM, Assets, Admin and Confluence OpenAPI specs; 1,400 vetted question-answer pairs from LeanZero's community-research compendium and 337 answers LeanZero posted on the Atlassian communities; 220 public Atlassian Developer Community threads (question plus accepted answer); a 7% general-code replay slice. Round 3 (this release): 1500 further steps on 5.45M tokens: the round-1 mix as replay, the documentation itself as reading material (4,117 sections), and the model's own generated apps kept only when they passed the manifest validator, the allow-list and tsc (380 complete apps) plus 51 manifests that passed the validator and allow-list (rejection-sampled fine-tuning).Round 3: 1500 steps from the round-2 adapter on 6.1M tokens (m7): a second, harder rejection-sampling harvest (4 apps for each of 450 briefs, 72% passed every gate, 937 kept) plus 362 fix-this-manifest pairs built from failures whose sibling passed, with the validator's real messages as the prompt. Round 4: two full epochs (4500 steps) of the same m7 mix from the round-3 adapter at a higher learning rate, the recipe the 9B run had shown to work; a real gain (14 of 25 complete apps) but below the 20/20 publish bar, so never released. Round 5: 2400 steps (1.4 epochs) from the round-4 adapter on 5.2M tokens (m8): a third rejection-sampling harvest aimed at the briefs that had never passed (backend-only phrasings, scheduled triggers, queues; 4 apps for each of 450 briefs, 63% passed, 810 kept), 439 validator-gated drills for the module types the mix had starved (workflow validators and post-functions, queue consumers, scheduled triggers, Confluence space pages), 245 more fix pairs from the model's own mistakes, and the earlier mixes as replay. A DPO experiment on 743 same-brief pass/fail pairs between rounds 4 and 5 collapsed the model's file layout (2 of 25) and was rejected; this round starts from the round-4 adapter instead. Round 6 (this release): the round-5 adapter expanded from 16 blocks at rank 32 to all 32 trainable blocks at rank 128 (400M parameters from 47M), then 2600 steps (1.07 epochs) on 7.5M tokens (m10): the endpoint and community-answer sources uncapped (3,813 and 1,748 rows), the general slice replaced by the untouched base's own answers to 2,700 human prompts (self-distilled) plus the earlier code files, at 20% of rows. Anchored fine-tuning: on every knowledge and drill row a KL penalty to the round-5 model's top-32 next-token predictions, on every general row to the base's, so the added capacity could not spend itself on forgetting; learning rate 1e-5 with cosine decay; the clean held-out set scored every 100 steps as the stop rule, never breached (1.231 at step 100 to 1.011 at 2500; round 5 was 1.326). The same capacity without the anchor (T8, 2400 steps at 2e-5) wrote as many apps at its best checkpoint but lost 0.135 on that set and was not released.

Measured

Against the untouched base on the same harness: Atlassian identifier probe 85% vs 15% (facts before 2026-04) and 15% vs 23% (facts after), thinking on; with thinking off, served: 85% / 31%. Forge shape tasks (a complete app from a one-line brief, thinking OFF, served through Rapid-MLX): 31.3 of 35 manifests pass Atlassian's validator (base 0 in its best mode) and 30.0 of 35 complete apps also compile under tsc against the real @forge/* types (base 0) (mean of 3 samples per brief). Looping battery (40 prompts x 4 sampling legs, 2048 tokens): genuine loops 0 / 0 / 1 / 0 vs base 2 / 1 / 3 / 4, non-termination 38 / 30 / 35 / 13% vs 60 / 48 / 53 / 25%. Long context: needle recall 100% at 4k, 100% at 32k, 100% at 128k; loop leg at 32k/128k (genuine loops, non-terminated of 4): 32k 0/4 vs base 0/4; 128k 0/4 vs base 1/4. Validation loss on the clean held-out set (130 rows no round trained on): 1.007 (round-1 adapter 1.207, untouched base 2.396). Merged 8-bit model vs base+adapter (top-1024 KLD, 200 prompts): 0.0339, top-1 99.56%. MTP on the merged model in Rapid-MLX, single stream: draft acceptance 58% (untouched base 53% on the same benchmark), decode speedup 1.45x at 128, 1.38x at 2048, 1.38x at 8192, 1.27x at 32768 (32.2 tok/s at short context). LM Studio MLX engine: loaded the folder and answered a Forge question (gate PASS).

Evidence: every round, every test

Every round, every test

Same harness, same prompts, same machine. Each round starts from the previous winner's adapter (T2 from T1, T3 from T1, T4 from T3, T5 from T4, T6 from T5, T7 from T5, T8-27B-CAP32 from T7, T9-27B-ANCHOR from T7). T2 and T6 were experiments that the rule rejected; T5 was a real gain over T4 but stayed below the 20/20 publish bar; T8 was the capacity run (adapter on 32 blocks at rank 128) that wrote more apps and lost held-out API-fact recall; T9 is the same capacity with the anchored recipe, the current release. From T9 the shape gate is 35 briefs with 3 samples each and the publish bar 30 of 35 apps and 30 of 35 manifests; T7 re-scored on that gate reads 22.7 and 30.7, single passes on 35 briefs moved by up to 7 apps between runs.

What still fails (shape tasks, the round's failure classes)

• T1: 11 valid manifest, code fails typing (tsc); 9 invented module type or property; 1 other manifest error; 1 module references an undeclared function key
• T3: 8 invented module type or property; 2 module references an undeclared function key; 2 valid manifest, code fails typing (tsc); 1 handler path does not match the file; 1 other manifest error
• T4: 7 invented module type or property; 3 other manifest error; 2 valid manifest, code fails typing (tsc); 1 key too long
• T5 (not released: below the 20/20 bar): 6 invented module type or property; 3 other manifest error; 1 valid manifest, code fails typing (tsc); 1 handler path does not match the file
• T6 (DPO experiment, rejected): 9 handler path does not match the file; 9 invented module type or property; 5 valid manifest, code fails typing (tsc)
• T7 (v0.4): 2 valid manifest, code fails typing (tsc); 1 handler path does not match the file; 1 invented module type or property
• T8 (capacity run, not released: knowledge guard): 1 other manifest error
• T9 (v0.5): 5 invented module type or property; 4 valid manifest, code fails typing (tsc); 4 other manifest error; 2 handler path does not match the file

Selection rule (in code, probe/pick_adapter.py)

A round replaces the previous winner only if all four hold: no regression on the clean held-out set beyond 0.010; identifiers with thinking on within one question; at least three more complete apps passing every gate than the incumbent, over the publish bar (20 of 25 up to T8; 30 of 35 apps and 30 of 35 manifests, mean of 3 samples per brief, from T9) and manifests within two; loops at or below the untouched base on every leg with non-termination not more than 5 points above it.

• T3 vs its predecessor: no-m3-regression=True identifiers-held=True shape-not-worse=True looping-ok=True -> chosen T3
• T4 vs its predecessor: no-m3-regression=True identifiers-held=True shape-not-worse=True looping-ok=True -> chosen T4
• T5 vs its predecessor: no-m3-regression=True identifiers-held=True shape-not-worse=False looping-ok=True -> chosen T4
• T6 vs its predecessor: no-m3-regression=False (m3clean 1.295 vs 1.260) shape-not-worse=False looping-ok=SKIPPED -> chosen T5
• T7 vs its predecessor: no-m3-regression=True identifiers-held=True shape-not-worse=True looping-ok=True -> chosen T7
• T8-27B-CAP32 vs its predecessor: no-m3-regression=False identifiers-held=False shape-not-worse=True looping-ok=True -> chosen T7
• T9-27B-ANCHOR vs its predecessor: no-m3-regression=True identifiers-held=True shape-not-worse=True looping-ok=True -> chosen T9-27B-ANCHOR

Corrections to a decision, with the evidence:

• T8 capacity run (2026-09-14): the v0.4 adapter expanded to 32 trainable blocks at rank 128 (400M parameters from 47M), trained 2400 steps on the round-5 mix at 2e-5. It wrote 24 of 25 apps on the 25-brief gate but lost 0.135 on the clean held-out set and one identifier question; its checkpoint curve showed the loss arriving in two phases (a small early step, then a steep fall after step 1500) while the app gain was complete by step 1000 (31 of 35 apps on the first 35-brief pass). Not released; it is the measurement that designed T9.
• T9 recipe (2026-09-16): the same capacity as T8, with a KL penalty to the round-5 model's top-32 predictions on every API, documentation, community and drill row and to the untouched base on every general row (the base's own answers to 2,700 human prompts), the endpoint and community sources uncapped, learning rate 1e-5 with cosine decay, and the clean held-out set scored every 100 steps as the stop rule. The guard never bound: 1.231 at step 100 down to 1.011 at step 2500 (T7 1.326, T8 1.461). Two watchdog restarts (a seeded 4k-token batch drove swap to 11 GB twice) cost about 100 minutes and no steps. Post-April identifiers read 15% (T7 38%); that slice is 13 questions and is the first item for the next round.
• T7 clean-set guard: the rule prints no-m3-regression=True for T7, but that condition was OVERRIDDEN, not passed. T7's loss on the 130-row clean held-out set was 1.326 against T5's 1.260, outside the 0.010 tolerance. The set is a likelihood proxy on OpenAPI endpoint facts and community-research rows; every direct probe of the same knowledge improved (identifiers thinking on 62% to 85%), so the proxy was waived by a flag file for this round and the other three conditions decided. The decision file records the override line.
• Qwen3.5-9B round 2 (T2-9B, 2026-09-13): trained on the 27B's v0.4 mix from the published 9B adapter; 18 of 25 complete apps and 19 of 25 manifests (round 1: 16 and 19), lower clean-set loss (1.472 vs 1.510), identifiers thinking on 46% / 23% (round 1: 62% / 15%). Below the 20 of 20 bar and two questions down on pre-cutoff identifiers, so not released; the published 9B stays round 1.
• T7 long-context: the first decision kept T5 because one of four 32k answers was flagged as a loop (a 40-character window repeated 13 times). The 32k leg was re-measured at 12 prompts for the base (0 flagged, 12 of 12 cut off) and T7 (2 flagged, 6 of 12 cut off) with the full text kept: the repeated windows were box-drawing rules and whitespace runs inside ASCII diagrams the model drew, and the top repeated window that contains text occurs once or twice. The flagged answers are ordinary diagrams plus code. The metric now ignores windows with fewer than 8 alphanumerics, the original 32k answer (byte-identical on re-generation) is scored 0 loops, and the corrected file sits beside the original in the repository. 128k was 1 loop for both T7 and the base under either rule.

How each test works

• Validation loss. Mean token cross-entropy of the adapter on a fixed 32-sample subset of a held-out set, the same subset for every round. A round that regresses on the held-out set by more than 0.010 is rejected; that is the test that eliminated the T2 experiment (0.892 vs 0.874). Found and fixed on 2026-09-11: each mix had drawn its own random split, so rounds 3 to 5 trained on 82% of round 1's held-out rows and their m3 numbers are partly memorisation. The clean row uses the 130 rows no round ever trained on; every future mix holds out a fixed hash bucket.
• Identifier probe. 26 short questions with a strict output contract, graded by a regex on the last line: 13 about identifiers that existed before April 2026 and 13 about the June 2026 @forge/* wave (package majors, module keys, tsc traps). One sample per question at temperature 1.0, so one question is 8 points and two adjacent rounds can differ by a question on noise alone. Run twice: thinking on through mlx-lm, thinking off through Rapid-MLX.
• Shape tasks. From T9: 35 one-line briefs (the original 25 plus 10 held out that no harvest ever touched), 3 samples per brief, counts reported as the mean, because a single pass flipped 7 of 35 briefs between two runs of the same adapter. Up to T8: 25 one-line briefs ('Write a JSM portal request-detail panel showing linked assets'), thinking off, served through Rapid-MLX with MTP. Each answer is unpacked into a project and graded by @forge/manifest's own validator, our allow-list of real module types and keys (the official validator accepts a misspelled key), and tsc against the pinned @forge/react 12.1.3 and @forge/bridge 6.3.1 types. 'Manifest valid' = validator plus allow-list; 'full gate' = the whole app also type-checks. This is the deliverable metric.
• Looping battery. 40 prompts (20 Atlassian, 20 general) x 4 sampling legs at 2048 tokens: thinking with Qwen's official sampling (1.0/0.95/20), thinking at temperature 0.6, greedy, and instruct mode with no repetition penalty. A loop = a 40-character window repeated 8+ times or under 25% distinct 4-grams in the last 512 tokens. Non-termination = the answer was still going at the cap. Gate: loops must not exceed the untouched base on any leg and non-termination must not rise more than 5 points.
• Long-context leg. The same loop metrics on 4 answers each at 32k and 128k tokens of context (token-calibrated windows, not word counts), because looping shows at 128k long before it shows at 2k.
• Needle. A random 8-digit key hidden at depths 0.15/0.35/0.5/0.85 in 4k, 32k and 128k token contexts; 4 trials per length; the model must return the key.
• Leak probe. 24 adversarial prompts x 3 samples asking for tenant hostnames, tokens, e-mails and client names, plus 40 forced-prefix completions, scanned with a regex for credential shapes and real tenant patterns. Every regex match is then checked against the training corpus: a match that exists there is a memorised leak; a match that does not is an invented shape. The untrained base is run as a control.
• Merge equivalence. After the adapter is merged into the 8-bit weights per module, the merged model's top-1024 logits are compared with base-plus-adapter served by mlx-lm on 200 frozen prompts: mean KL divergence and top-1 agreement. Gate: KLD at most 1.5x the base's own distance from bf16 (0.0446) and top-1 at least 90%. The 6- and 4-bit family members are scored the same way plus the identifier probe.
• MTP speculative decoding. Rapid-MLX serves the merged model with its multi-token-prediction sidecar off and on; decode tokens per second at 128, 2k, 8k and 32k tokens of prompt (median of 2 runs), draft acceptance from the engine's metrics, and greedy byte-equality on 20 short prompts.
• LM Studio load. The merged folder is linked into LM Studio's models directory, loaded with its MLX engine through the lms CLI, and asked one Forge question; the answer must name the right module.

Family members of this release

• 8-bit (the model): KLD 0.0339, top-1 99.56% vs base+adapter
• 6-bit: KLD 0.0481, top-1 98.85% vs base+adapter; identifier probe 69% / 8%
• 4-bit: KLD 0.0969, top-1 95.74% vs base+adapter; identifier probe 92% / 23%

GGUF family (llama.cpp), measured on this release

Parity is a chain of three measured links, each inside the tool that can measure it: the adapter merged into the original bf16 reproduces the MLX base-plus-adapter greedily (AGREE 16/20 identical; mean common-prefix fraction 0.888); the lossless bf16 GGUF reproduces mlx-lm across runtimes (AGREE 17/20 identical; mean common-prefix fraction 0.945); each quant is scored against the bf16 GGUF's own logits with llama-perplexity (Q8_0 mean KLD 0.000842, Q6_K 0.00306 on 40 x 2048 held-out tokens). Then the same probes through llama-server on the Q8_0 file: identifiers thinking on 85% / 15%; SHAPE35 gguf-Q8_0 | apps 28.7/35 manifests 30.3/35 (mean of 3); loops think_official 0/40 (non-termination 33%), think_t06 0/40 (non-termination 33%), greedy 1/40 (non-termination 33%), instruct_nopenalty 0/40 (non-termination 13%); draft decoding on Metal: MTP none: 20.9 tok/s over 1280 tokens (5 prompts, 256 max, temp 0, wall incl. prefill); MTP draft-mtp: 19.6 tok/s over 1280 tokens (5 prompts, 256 max, temp 0, wall incl. prefill); thinking-off switch: THINK_OFF_KWARG HONOURED | reasoning chars 0 | content: `jir.

How to run it

Rapid-MLX (LeanZero's fork, the engine inside Elzi, our desktop agent app)

rapid-mlx serve <folder> --text-only --tool-call-parser qwen3_coder_xml --reasoning-parser qwen3 --speculative-config '{"method":"mtp","model":"<folder>","num_speculative_tokens":3}'

LM Studio: put the folder under your models directory (for example models/leanzero/<folder>); it loads with the MLX engine like any mlx-community model. Measured on this release, see above.

mlx-lm

mlx_lm.generate --model <folder> --prompt "Which Forge module adds a panel to the Jira issue view?"

Sampling

Thinking: temperature 1.0, top_p 0.95, top_k 20. Instruct: temperature 0.7, top_p 0.8, top_k 20, presence penalty 1.5. Trained with reasoning_effort=medium, preserve_thinking=false.

Limits

It knows far more Atlassian than the base but it is not a compiler: a complete app from a one-line brief can still need one fix before forge deploy. With thinking off the usual one is a TypeScript typing error against @forge/react or @forge/bridge; the next is a resolver or resources key placed under modules, or a handler path that does not match the file. Pair it with the manifest validator and tsc. Use thinking for questions and thinking OFF (enable_thinking: false / --no-thinking) for code generation; measured on round 1, 56% vs 16% manifest validity. Version facts are pinned to the June 2026 @forge/* wave; the weights do not update themselves, and asked cold which major of a Forge package is current it usually declines (13-question post-April slice: 15%, v0.4 38%; every app brief states its package versions, which is where the app numbers come from). English only. Trained with reasoning_effort=medium.

Family

Qwen3.8-27B-Atlassian-Q8-mlx (the model, 8-bit, MTP sidecar), Qwen3.8-27B-Atlassian-lora (the adapter), Qwen3.8-27B-Atlassian-Q6-mlx and Qwen3.8-27B-Atlassian-Q4-mlx (smaller, quantised from the merged bf16 weights), Qwen3.8-27B-Atlassian-Q8-base-mlx (the untouched base). A Qwen3.5-9B version (Qwen3.5-9B-Atlassian-Q8-mlx and family) is trained on the same data and published under the same gates.

Licence and provenance

Apache-2.0 (inherits the base). Training data: Forge apps written by LeanZero, public Atlassian developer documentation, Atlassian Cloud OpenAPI specifications, LeanZero's vetted community-research compendium and posted community answers, 220 public Atlassian Developer Community threads (question plus accepted answer), and a small general-code replay slice. No credentials; tenant hostnames and e-mail addresses were rewritten before training, and the finished model was probed for memorised strings against a control run on the untrained base. Method and gates: see the LeanZero write-up.