Hemlock-Qwen3.8-27B

Hemlock-Qwen3.8-27B

A 27B vision-language model that writes Hemlock.

HemBench: 33/38 (86.8%), pass@1 0.816 — the highest score recorded on the suite.

What it is

WichtelHui-Qwen3.8-27B-SLERP with Hemlock-Qwen3.8-27B-LoRA merged into the weights.

The name says Qwen3.8, and the lineage is more mixed than that: WichtelHui is a 50/50 SLERP of Wichtel-Qwen3.6-27B (Qwen3.6-line) and Huihui-Qwen3.8-27B-abliterated (Qwen3.8-line), so this model is roughly half of each pretrain. The adapter was trained on Qwen3.8-27B.

HemBench

38 tasks, generated code executed against expected output. Run with thinking disabled — as a reasoning model it otherwise spends the whole token budget thinking and emits nothing.

On precision: HemBench is 38 items sampled at temperature 0.2, so single runs move by a couple of tasks. The 33/38 above is from 3 samples per task (114 generations); a single-sample run of the same weights scored 29/38. Treat differences of 2–3 tasks between models as noise unless they were measured the same way.

Failures are 4 wrong-output and 1 runtime error. The un-adapted Qwen3.8 parent fails 32 of 38 at runtime, because it writes Rust- and C-shaped code when asked for Hemlock; that is the failure mode the adapter removes.

General capability

The adapter does not trade general ability for Hemlock. ARC-Challenge, 299 tasks, scored by summed logprob — deterministic, no sampling:

A separate configuration — WichtelHui with an earlier adapter checkpoint applied at runtime rather than merged — measured 68.56% on the same 299 tasks. The 66.22% above is this model's own score; the other figure is recorded here only so the difference is not mistaken for a regression.

Safety and truthfulness

Inherited from WichtelHui and not re-measured after the merge:

The abliterated parent scores 0/2 on safety_control on its own; the SLERP with a safety-intact parent restores both refusals while keeping the full 18/18 on ccp_truth.

The adapter

Trained on hemlang/Hemlock-SFT-combined — 5,852 rows from the three Hemlock SFT sources, with the 150 rows flagged as overlapping HemBench excluded. Training on those would make the score above a measure of recall.

LoRA r=32, α=64, 2 epochs, lr 2e-4, max_length 2048, on Qwen3.8-27B.

An earlier checkpoint at 40% of training scored within noise of the final one (pass@1 0.737 vs 0.816 at 3 samples; 31/38 vs 29/38 at one sample, in the opposite order). The fully trained adapter ships because it is the complete artefact, not because it measurably won.

Vision and MTP

Both intact: 333 visual.* and 15 mtp.* tensors, 1199 total. The adapter targets only attention and MLP projections; everything else is copied unchanged from the base rather than reconstructed, and the merged weights are asserted to differ from the base before shipping.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained("hemlang/Hemlock-Qwen3.8-27B", dtype="bfloat16")

Serve with thinking disabled for code generation.

Limitations

• HemBench is 38 tasks. It measures whether generated Hemlock runs and produces the expected output, not whether it is idiomatic, efficient, or safe.
• Safety numbers are inherited from the base and were not re-run after merging the adapter.
• Half the weights come from a Qwen3.6-line parent, so behaviour will not always match a pure Qwen3.8 derivative.