Reliquary-4B

Reliquary-4B

Reliquary-4B is Qwen3-4B-Base trained with group-relative reinforcement learning on math and Python problems. There is no supervised fine-tuning and no distillation: the model is the base policy plus 12,842 RL updates.

What makes the run unusual is where the rollouts came from. Reliquary is an open protocol running as Bittensor subnet 81. Independent participants choose which problems are worth training on, generate groups of 16 completions on their own hardware against the published checkpoint, and are paid only for groups whose rewards disagree — the groups group-relative RL can actually learn from. A validator regenerates the rewards, verifies the samples, and trains on what it selected. The trainer spent no compute generating rollouts.

Over the run, held-out pass@1 rose from 0.48 to 0.79 on math and 0.58 to 0.78 on code, and the gain transfers to benchmarks the run never saw: MATH-500 37.2 → 72.6, AMC23 15.6 → 47.5, HumanEval+ 52.1 → 72.9, MBPP+ 43.7 → 67.6.

Prompt format

The model was trained on raw text prompts, without a chat template. Use the same prompts at inference: chat formatting and few-shot prompts do not bring out what the training added.

Math. The answer goes in \boxed{}.

Solve the following math problem step by step.

{problem}

Put your final answer within \boxed{}.

Python. The model reasons first, then writes the solution in its last fenced Python block. In training the prompt also named the function the tests call (the line reads "argument" when n = 1):

Solve the following programming problem step by step.

{problem}

Write your solution as a Python function named `{name}` that takes {n} arguments and returns the result; do not read from stdin or print.

After your reasoning, provide the final implementation in the last fenced Python code block.

Generation stops at <|endoftext|> (token 151643).

Usage

With vLLM:

from vllm import LLM, SamplingParams

llm = LLM("ReliquaryForge/Reliquary-4B")
prompt = (
    "Solve the following math problem step by step.\n\n"
    "What is the smallest positive integer n such that 3n leaves a remainder of 1 when divided by 7?\n\n"
    "Put your final answer within \\boxed{}."
)
params = SamplingParams(temperature=1.0, top_p=1.0, max_tokens=8192)
print(llm.generate([prompt], params)[0].outputs[0].text)

With Transformers (4.51 or later):

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "ReliquaryForge/Reliquary-4B"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=8192, do_sample=True, temperature=1.0, top_p=1.0)
print(tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Sampling. Every number below uses temperature 1.0 with top-p 1.0, the setting the model was trained under, which is also the default in generation_config.json. Greedy decoding scores higher on most benchmarks (see the table), but on MATH-500 about 2% of greedy generations run to the 8,192-token limit without finishing.

Training

• Algorithm. DAPO-style group-relative policy optimization: advantages are z-scored within each group of 16 samples; token-level loss, normalized per environment; clip-higher (ε_low 0.2, ε_high 0.28); dual clip 10; no KL penalty; soft overlong punishment (factor 0.5 over the last 4,096 tokens before the limit). AdamW in fp32, learning rate 1e-6 with a 10-update warmup then flat. π_old comes from the validator's own log-probabilities.
• One window, one update. Each window collects submissions for a fixed interval, admits and ranks them, and hands exactly 16 groups per environment to a single optimizer step: 512 rollouts, each used once and never reused. The run took 12,852 windows over 15.2 days, about 850 updates a day.
• What gets trained on. A group is eligible only when its 16 recomputed rewards vary enough (σ ≥ 0.24, i.e. 1 to 15 correct out of 16 for binary rewards) — the criterion that decides whether a group carries any gradient at all. Every eligible group is worth the same payment. A prompt that enters a winning batch is locked out for the rest of the run, by dataset index and by rendered-prompt hash.
• Reward integrity. The reward vector is the whole training signal, so participants are not trusted with it. Generation is forced onto a public sampling stream derived from window randomness, prompt, checkpoint and position — not from participant identity — and the validator re-derives token probabilities to check the samples. Math answers are regraded and code tests executed by the validator, in a sandbox; a participant-reported score is never read.
• Data. Math problems from nvidia/OpenMathInstruct-2, graded on the last \boxed{}. Python problems from a curated subset of nvidia/OpenCodeInstruct, graded by executing unit tests against the last fenced block that defines the entry function.
• Compute. Rollout generation ran on participants' own hardware. The trainer used one H100 (under ~30 s per update); the validator used a second one for verification.
• Dates. 2026-08-25 to 2026-09-09.

The mechanism, the market it created, and what it did and did not demonstrate are described in the technical report (see Citation).

Evaluation

Everything below was measured on the weights in this repository.

External benchmarks

• Setup: vLLM, the prompts above, 8 samples per problem at temperature 1.0. pass@1 is the mean over the 8 samples; greedy decoding is in parentheses.
• Grading: math on the last \boxed{} with math-verify; code with EvalPlus (base and plus tests) on the last fenced block defining the entry point.
• Statistics: Δ is the per-problem paired difference against the base, with its paired t-statistic.

GSM8K and MATH-500 are near-domain: OpenMathInstruct-2 is synthesized from their training splits. AMC23, AIME, HumanEval+ and MBPP+ carry the transfer claim. The AIME sets hold 30 problems each, so their differences are imprecise.

Held-out sets from the training distribution

Prompts held out from training, 16 samples each at temperature 1.0, graded with the training graders. For code a sample counts as solved only when it passes every test. Prompts whose content also appears among the trained prompts are excluded, leaving 282 math and 450 code prompts.

On the training distribution pass@16 barely moves while pass@1 nearly doubles: the model solves reliably what the base could already solve occasionally. On the external math sets pass@8 does rise.

Where the gain comes from

A response without its answer channel scores zero — \boxed{} in math, a fenced block defining the entry function in code — so pass@1 factors into the rate at which the channel appears and the accuracy when it does. The share attributed to formatting is a range because it depends on the order of the decomposition.

Formatting is not what carries the gain. On the problems where the base already produced the answer channel in all 8 samples, the model still gains +32.1 points on MATH-500 (222 problems, t = 18.0), +18.5 on HumanEval+ (106 problems, t = 7.5) and +19.5 on MBPP+ (115 problems, t = 8.7).

In math the gain goes past making the base's best answer more reliable: sampled, the model beats greedy decoding of the base (MATH-500 72.6 against 52.8), and it solves at least once in 8 samples 58 MATH-500 problems the base never solved in 8 (9 go the other way). In code the gain is mostly reliability — the base's greedy decoding already passes 78.7% of HumanEval+.

Response length

This is not a long-reasoning model. Median response lengths are 284 tokens on GSM8K (base 255), 494 on MATH-500 (375), 728 on AMC23 (478) and 836 on AIME 2024 (510): answers grow with difficulty, and accuracy still falls with length at every checkpoint.

Limitations

• A completion model, not a chat model. There is no chat template, and no instruction or safety tuning beyond the base.
• The gain is tied to the training prompt. Under a standard few-shot harness, a checkpoint from the same late plateau scores within about one point of the base on GSM8K and MATH. Use the prompt format above.
• Narrow scope. Training covered English math problems and Python functions only.
• Near-domain benchmarks. GSM8K and MATH-500 share their source with the math training corpus; AMC, AIME, HumanEval+ and MBPP+ carry the transfer claim.
• General capability was checked mid-run, not on these weights. At update 8,800 the run showed held-out NLL drift under 0.15% and confidence intervals crossing zero on MMLU, ARC, TruthfulQA and IFEval.
• Decoding matters. Greedy decoding reports a much smaller gain in math and essentially none in code; evaluate under the sampling regime the model was trained in.

Citation

Romain Fanucci, Malik El Hassani. A Market Mechanism for Rollout Selection in Group-Relative RL. Reliquary technical report, draft v13, 14 September 2026.

The report describes the mechanism, the market data of the run, and what the run does and does not establish — in particular that it demonstrates feasibility of market-supplied RL training, not a learning-efficiency advantage over centralized rollout selection.

Files

config.json and the tokenizer are taken from Qwen3-4B-Base, whose architecture is unchanged, with the chat template removed. They load under Transformers 4.x and 5.x alike.