Qwen3-4B-RailRoadEngineer1959

Learning the Rulebook

Volume2Gym asks what happens when a bounded, structured text becomes a small world in which a model can act, receive declared feedback, and try again.

This repository contains the LoRA adapter and evidence from one Tinker grouped reinforcement-learning run on Volume2Gym Railroad 1959. It is an English historical-technical experiment built from 117 reproduced pages, 536 extracted rules, and 2,708 synthetic scenarios—not a current railroad operating system.

Dataset and source card · upstream project · rights statement

Experiment at a glance

The direct experiment is document-grounded reinforcement learning. A fixed technical volume supplies bounded terminology and procedures; extraction turns that structure into scenarios and reference responses; a deterministic environment scores model actions; grouped rewards update a rank-32 adapter.

The blueprint is the experiment's evidence architecture, not a claim that every synthetic row has a verified page citation. The dataset card publishes the raw rule labels, a conservative linkage audit, the complete scan manifest, and the limits of page-level traceability.

The indirect hypothesis is broader: a community-authorized grammar, dictionary, teaching text, oral-history collection, or other small structured corpus might also be organized into tasks, constraints, and inspectable feedback. This run does not evaluate a low-resource language, demonstrate cross-lingual transfer, or relax the need for community authority, licensing, fluent-speaker review, and culturally specific evaluation.

The learning loop

For each selected task, the renderer supplied the scenario and 1959-rule instruction. The policy sampled eight responses at temperature 0.9. A deterministic lexical scorer compared each rendered response with its reference response, and the grouped training loop applied an importance-sampling loss to the LoRA parameters. Groups with constant reward were removed. The 270-example holdout was reserved for evaluation gates rather than policy updates.

This is reinforcement learning in a deliberately narrow sense: the policy produces an action, the environment returns a scalar reward under a disclosed rule, and that reward shapes subsequent policy updates. It is not a simulator of train movement, and its reward is not an expert judgment of operational safety.

Related experiment: OpenCode harness training

This repository’s adapter and metrics remain the Tinker grouped-RL (GRPO) run described above: direct text responses, deterministic lexical reward, rank-32 LoRA on Qwen3-4B-Instruct-2507.

A parallel learning path uses the same dataset as curriculum for a loop-owning OpenCode agent trained with TRL’s AsyncGRPO / OpenEnv harness recipe. In that setup the agent acts with tools inside a sandbox, writes answer.txt, and is scored against reference_response (exact match / token F1), following HarleyCoops/RailroadHarness and the HF blog on TRL OpenEnv harness training.

When a harness-trained checkpoint is ready for release, it should ship as its own model repository (or a clearly labeled revision) with its own metrics, rather than being conflated with the step-60 Tinker evaluation gates on this card.

Actual run results

The raw metrics record training reward at 0.2465 on step 0, a 0.3983 peak at step 57, and 0.3338 at the final training step 76. On the fixed holdout, evaluation reward moved from 0.2493 at step 0 to 0.3753 at step 60 (approximately +50.5%), while evaluation tokens per turn fell from 255.86 to 90.08 (approximately −64.8%) and evaluation parse compliance rose from 0.37% to 100%. Those evaluation results come from four gates at steps 0, 20, 40, and 60, each with n=270, for 1,080 held-out episodes; training generated 19,504 rollouts across 77 batches. The final batch-77 checkpoint was saved but not separately evaluated, so step 60 is the final evaluation gate rather than an evaluation of the final checkpoint. These derived values are reproduced in the run summary.

The two training means answer different questions and are intentionally reported separately. Neither is an evaluation result. Exact fields, full precision, episode counts, and deltas are available in the generated summary; every underlying aggregate row remains in the unchanged metrics stream.

What changed

The strongest held-out changes were lexical similarity, brevity, and renderer-format compliance:

Training parse counts moved from 15/256 on the first batch to 48/48 on the final partial batch. Evaluation parse counts moved from 1/270 to 270/270. Parse compliance means that the response renderer recognized the expected format; it does not establish that the content was correct.

Training reward was noisy and the last value was lower than the step-57 peak. The held-out reward rose at each of the four observed gates, but there is no gate after step 60. The available evidence therefore supports improved overlap with held-out reference responses under this scorer—not a claim of monotonic training, present-day rule competence, or safe operational judgment.

What the reward measures

The reward is lexical, not a safety oracle. It measures normalized exact match and token overlap with a reference response. In this run the logged safety, procedure, and terminology columns carry the same proxy value, so they are not three independent validations and are not plotted as though they were.

The public dependency-free scorer reference documents the calculation:

1. Lowercase the prediction and reference, trim them, and collapse whitespace.
2. Record normalized exact match.
3. Compute whitespace-token multiset F1 from token-count overlap.
4. Set safety = max(exact_match, token_f1), procedure = token_f1, and terminology = token_f1.
5. Compute 0.5 × safety + 0.3 × procedure + 0.2 × terminology.

Normalized exact match remained zero throughout this run. Consequently, safety, procedure, terminology, and the scalar reward all reduced to the same whitespace-token multiset F1 value. The published ledger retains the named columns for auditability, but their labels should not be read as three independent semantic evaluations.

Run audit: two verifier generations

The project contains two distinct verifier generations. Only the first produced the numbers on this card.

No legacy judge score is substituted for a deterministic run metric, and no deterministic proxy column is presented as an independent safety validation.

Reproduce the run

The complete public evidence surface is:

• raw aggregate metrics, copied unchanged from the run;
• scrubbed run configuration;
• public checkpoint manifest, with private Tinker locators replaced by SHA-256 values;
• derived run summary;
• public scorer reference; and
• Tinker reward ledger.

Use the upstream project for the training implementation and the dataset repository's explicit train split and test split. The published config.json replaces the original machine-local dataset path with data/train.jsonl; resolve that path against a local checkout of the dataset bundle.

The public checkpoint file is an audit manifest, not downloadable optimizer state. It records saves at batches 20, 40, 60, and the final batch 77 without exposing private Tinker locators. Full rollout HTML logs are intentionally not included.

Use the adapter

This repository is a PEFT LoRA adapter, not a merged standalone checkpoint. Load the declared base model first, then attach the adapter:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-4B-Instruct-2507"
adapter_id = "HarleyCooper/Qwen3-4B-RailRoadEngineer1959"

tokenizer = AutoTokenizer.from_pretrained(base_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

messages = [
    {
        "role": "user",
        "content": (
            "Treat the 1959 rulebook as historical source material. "
            "Explain the procedure described for the supplied scenario."
        ),
    }
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The adapter metadata identifies a causal-language-model LoRA with rank 32 and all-linear targets. Base-model weights remain subject to the base model's own terms and are downloaded separately.

Limitations and safety

Historical research artifact. The 1959 rules may conflict with current rules, technology, law, terminology, and safe practice. Do not use this adapter to control trains, qualify personnel, train present-day crews, or make real-world operational safety decisions.

• The reward measures lexical overlap with synthetic reference responses. High reward can coexist with factual, procedural, or safety errors.
• Parse compliance measures renderer compatibility, not correctness.
• The evaluation set contains 270 synthetic scenarios from the same construction pipeline, not an external expert-certified railroad benchmark.
• The final batch-77 checkpoint was not separately evaluated; the last evaluation evidence is the step-60 gate.
• Raw task rule labels are not a complete page-citation system. The dataset card documents unresolved and ambiguous linkage cases.
• This is an English technical demonstration. It provides no evidence of low-resource-language performance or cross-lingual transfer.
• The adapter has no visual input pathway; the archival images on this card document the source and experiment.
• The source volume is historical, and the repository does not assert public-domain status for its reproduced scans or release artwork. See RIGHTS.md and the dataset's provenance statement.

Citation and links

When citing results, identify the exact model-repository revision, the base model, the dataset revision, the four-gate evaluation protocol, and the deterministic lexical scorer. Do not cite the step-57 training peak as the final training value or as a held-out result.