quantumgpt-354m-finetuned

QuantumGPT-354M (fine-tuned, format-anchored)

Benchmark evidence update

This note covers checkpoint 12beb1b7dae4b879ab74eefd8dec9168a7cbf59a with native context 512.

Originally reported. QSemBench-dev-v0.1, 120 stratified cases, greedy decoding and a generator shared with training. The card reports a correction to stop-marker handling. The corrected QSS-Core development scores are 97.52 and 96.45 across two seeds. The published revision tested here is one checkpoint.

Evidence gap. The exact 120 cases, executable evaluator, composite weights and raw original seed runs were not recovered. A reconstruction with a declared state-based surrogate is available.

Measured on the compact development suite. The checkpoint returned 118/384 syntax-valid answers, 0/384 answers with the requested structure, and 0/384 fully correct answers. It passed 0/257 on the fixed panel with verified short reference answers.

The compact suite uses one greedy raw answer per task, the native prompt transport and context, and task-specific numerical checks for the full circuit contract. Its 384 problems are synthetic and have been used for development.

The benchmark manifest SHA-256 is 320535d4ff74d9f07db024c1e86c0e8208a8fd93c294f47332cb9c38fadf9243. Evaluation settings, context stops and full panel counts

Reconstructed diagnostic. The separate 120-case release-corpus draw yielded 116/120 on the projected state check and 98/101 among references eligible for the static contract. This used a declared surrogate for the original historical evaluator.

Pinned original card

The 354M sibling of QuantumGPT-124M-v3: GPT-2 medium architecture (24 layers, 16 heads, 1024 embd, natively 512-token context) fine-tuned with the same format-anchor recipe that produced the v3 champion.

Corrected QSS-Core v0.1: 97.52 / 96.45 (two-seed replication) — final scores after correcting tokenizer stop-marker extraction and rerunning full robustness probes.

Benchmark results (QSemBench-dev-v0.1, n=120 stratified, greedy decode)

Scoring: qiskit qasm2 parse/compile + statevector fidelity ≥ 0.99 vs reference, entanglement checks, prompt-perturbation robustness probes. See TECHNICAL_REPORT.md in merileijona/quantumgpt-124m-v3 for full methodology.

Final corrected replication: seed-42 fine-tuned model QSS 97.5235; seed-43 QSS 96.445. Spread: 1.079 QSS points. The evaluator strips tokenizer stop-marker drift before parsing.

Training

• Data: merileijona/quantum-circuits-21k (validated description→OpenQASM pairs), format-anchor oversampling ×2 (short perfectly-terminated circuits), long-circuit oversampling ×2, deterministic paraphrase augmentation — the recipe proven on the 124M lineage.
• Continuation training from the public base: lr 5e-6, cosine schedule, completion-only loss masking, effective batch 4 via gradient accumulation, one epoch per seed.
• Note: unlike the 124M v3, no architecture mutation was needed — this model already shipped with a 512-token context.

Family comparison (same benchmark)

The corrected seed-42 354M model scores 97.52, slightly below the 124M v3 seed-42 score of 97.70; both models are effectively near-ceiling on this development benchmark. The benchmark still shares its generator with training data.

Limitations

Same profile as the family: dev benchmark shares its generator with training data; negation constraints weakly followed; no refusal behavior; English only. Every generated circuit should be verified before execution.

Intended use

Research and education. Verify all outputs before running on real quantum hardware or simulators.