reef-nccl-ep-qwen3-8b

reef-nccl-ep-qwen3-8b

A Qwen3-8B LoRA adapter (rank 32) trained with reinforcement learning to propose high-bandwidth configurations for an expert-parallel collective-communication kernel (MoE dispatch/combine over RDMA). This is the final checkpoint of a harness/model co-evolution experiment built on reef with the slime/Megatron training backend and SGLang serving.

What it does

Given a task description of the kernel's five tuning parameters (RDMA queue pairs, tokens per chunk, dispatch sub-puts, combine streaming batch, communication SMs), the model proposes a configuration as a JSON object. Every reward during training was a hardware measurement: dispatch+combine bandwidth in GB/s on a 16-GPU H200 system, gated by a 16/16 correctness test. No synthetic rewards.

Results

• The model's sampling attractor is the measured optimum at 8192 tokens (qps=2, chunk=256, subputs=6, cb=32, sms=11 -> 192.99 GB/s), which matches the best human-guided result on the same judge.
• During training the system found configs no one had measured before, including qps=2, chunk=256, subputs=16, cb=64, sms=12 -> 185.76 GB/s and, at 16384 tokens (a size never previously tuned), qps=2, chunk=512, subputs=8, cb=32, sms=11 -> 191.45 GB/s.
• The trained 8B outperforms a 480B coding model prompted zero-shot on the same task and judge (480B best: 147.46 GB/s).

How it was trained

Alternating coordinate ascent between the model and its harness, with the judge frozen:

1. Model phase: 16 steps of grouped relative-reward RL (reef TTTD recipe, slime/Megatron LoRA backend, 4 groups x 4-8 rollouts per step; rewards from the hardware judge).
2. Harness phase: a 480B model proposes revisions to the task instruction from training evidence (reward curves, measured episodes); candidates are scored on a frozen checkpoint under an equal sample budget, and a revision is adopted only on a clear win. The instruction carries measured heuristics forward; the judge never changes.

Five model phases and three harness phases produced this checkpoint. Intermediate checkpoints of every run are published in the companion repos (reef-nccl-ep-qwen3-8b-lora-coevolve-run2/-run4/-run5, -16k-run2, -discovery-16step).

Loading

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")
model = PeftModel.from_pretrained(base, "jhhj25/reef-nccl-ep-qwen3-8b")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

Disable thinking mode when sampling configs (chat_template_kwargs={"enable_thinking": False}), and prompt with a task description of the parameter space; the model answers with a single JSON config block.