Qwen3-4B-Code-RL-Specialist

Qwen3-4B Code RL Specialist

Standalone BF16 code specialist, trained from Qwen/Qwen3-4B with a rank-16 LoRA and merged into full Hugging Face weights.

These three specialists can be used as domain teachers for MOPD (Multi-Teacher On-Policy Distillation) experiments. The checkpoints were trained with domain RL; no MOPD student has been trained or evaluated in this release.

Training datasets and exposure

This repository's specialist uses agentica-org/DeepCoder-Preview-Dataset. Each specialist was trained on its own domain's data.

Preparation for this specialist

Training combines the PrimeIntellect, TACO, and LCB v5 training subsets. Preparation retains Python stdin/stdout problems with at least five tests, removes duplicate/evaluation-overlapping and interactive tasks, rejects unsupported/function-call/non-text tests, limits training tasks to at most 200 tests and 4 MiB of test text, and caps prompts at 4,096 tokens. After holding out 64 development prompts, 17,256 candidates remain. During training, at least one of up to three reference solutions must pass the saved tests before a candidate is sampled. At this checkpoint, 11,301 candidates were examined, 1,733 rejected, and 9,568 sampled. The reward is binary: 1 only when every saved test passes, otherwise 0.

This checkpoint stopped before completing an epoch (65.49% of prepared candidates examined). This checkpoint sampled 9,568 prompts × 8 responses = 76,544 rollout responses. Sampled responses are not all gradient examples: constant-reward groups, zero centered advantages, and responses stopped at the length limit are excluded from updates. Counts refer to the training trajectory represented by the checkpoint and exclude evaluation and discarded/replayed recovery work.

All dataset revisions, prepared split sizes and hashes are in training/dataset_manifest.json. AIME24, the frozen lcbv5/test split, IFEval, and IFBench are evaluation datasets, not training splits. Each domain also has a separate 64-prompt development set. Dataset preparation removes normalized duplicates and evaluation overlaps; an additional exact prompt audit found no matches across the prepared training and evaluation sets. These checks do not establish semantic or pretraining decontamination.

Training method

• Independent domain RL with GRPO-style group-centered advantages and a PPO clipped policy objective. Advantages subtract the prompt group's mean reward without standard-deviation normalization; there is no critic or KL penalty.
• LoRA rank 16, alpha 32, dropout 0; attention projections, MLP projections, and lm_head adapted. Fresh LoRA/optimizer initialization for each specialist.
• Up to 32 prompts × 8 responses per batch; rollout temperature 1.0, top-p 1.0, top-k disabled; thinking disabled; maximum response 16,384 tokens.
• Adam learning rate 1e-5, 10-update warmup, clipping ratios [0.8, 1.2]. Responses receive equal weight, with their token losses averaged.
• LoRA deltas were merged in FP32, then saved as BF16. Input embeddings were preserved and the adapted output head is untied. Export validation checked all 506 adapter tensors and exact logits after BF16 save/reload; see model_provenance.json.

Full cross-domain evaluation

All four models were evaluated on every benchmark under the same frozen protocol, for 8,308 responses total. Scores are percentages.

Math/coding: four sampled responses per prompt; temperature 0.7, top-p 0.8, top-k 20. Pass@1 is the mean success rate over those samples. IF: one greedy response per prompt; strict accuracy requires every instruction for that prompt to pass. All runs disable thinking and cap responses at 16,384 tokens, with a 24,576-token context.

The coding benchmark contains all 279 problems in the frozen DeepCoder lcbv5/test subset, not the entire current LiveCodeBench release. AIME24 includes all 30 problems; IFEval includes all 541 prompts; IFBench includes all 300 prompts. Coding accuracy requires all tests to pass (4-second wall, 3-second CPU, 1-GiB memory limit per test). Two inherited IFEval letter-frequency checks randomly substituted letters for literal # and !; the final scorer counts the requested punctuation uniformly for all models and retains every prompt.

These are latest checkpoints, not checkpoints selected to maximize the reported test scores. Code and IF have incomplete and unequal training exposure. Math and coding gains come with longer responses: their domain specialists use 2.09× and 3.87× the base model's mean response tokens. Paired 95% intervals exclude zero for the math/AIME24, code/coding, and IF/IFBench gains; the IF specialist's IFEval gain remains uncertain (−0.18 to +6.47 percentage points). Intervals are exploratory, without adjustment for multiple tests.

• Full report, pass@4, response lengths and limitations
• Machine-readable metrics · Comparison CSV · Paired confidence intervals
• Frozen evaluation protocol · IFEval checker correction

Use as MOPD teachers

Use a student initialized from the compatible Qwen3-4B base and route each prompt to the teacher for its known domain. Generate continuations with the student, then score those student-generated tokens under the corresponding teacher to obtain token-level supervision for the chosen on-policy distillation objective. Keep tokenizer IDs, chat templates, thinking mode, and token masks aligned. These checkpoints provide the teacher models; this release does not prescribe a complete MOPD optimizer or claim a distilled-student result. See the MOPD paper for the multi-teacher on-policy formulation.

• Math teacher
• Code teacher
• Instruction-following teacher

Repository IDs and compatible settings are available in mopd_teachers.json. The domain comparison above supplies teacher baselines for capability-integration experiments.

Inference

The exports were validated with Transformers 5.12.1. Use a compatible Transformers release with PyTorch and Accelerate.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "pb09204048/Qwen3-4B-Code-RL-Specialist"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, dtype=torch.bfloat16, device_map="auto"
)
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Write a Python function that returns the sum of two integers."}],
    tokenize=True, add_generation_prompt=True, enable_thinking=False,
    return_dict=True, return_tensors="pt",
).to(model.device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.7, top_p=0.8, top_k=20)
print(tokenizer.decode(output[0, inputs.input_ids.shape[-1]:], skip_special_tokens=True))

Disabling the thinking template does not prevent ordinary step-by-step reasoning in response text. Benchmark numbers apply to the specific prompts and grading protocol above.

Attribution and license

The base model is Qwen/Qwen3-4B, revision 1cfa9a7208912126459214e8b04321603b3df60c. Its Apache-2.0 license is retained in LICENSE. This release attributes DAPO-Math-17k (the linked snapshot has no license field in its Hub metadata), DeepCoder-Preview-Dataset (MIT), and Nemotron-Cascade-2-RL-data (ODC-BY). Dataset terms remain those of the linked source repositories; training corpora are not redistributed here.