qwen-arithmetic-t4

qwen-arithmetic-t4 — banked fp16 weights

Not a fine-tune. These are Qwen/Qwen2.5-0.5B-Instruct's own weights, re-containered into the layout that train_qwen_arithmetic_t4.py loads: per-layer matrices stacked into (L, ...) banks, QKV and gate/up concatenated into one GEMM each, cast to fp16. Same 494,032,768 parameters, same values.

That script is a single-file GRPO speedrun of Qwen2.5-0.5B-Instruct on arithmetic for ONE Tesla T4 — the free Colab GPU. It has no transformers dependency and no nn.Module: it opens one file and gets tensors whose names, shapes and dtype are already the ones its handwritten forward/backward and its CUDA-graph decode engine use. This repo is that file, so a Colab session spends its first minute downloading ~942 MB instead of pulling the checkpoint and rebuilding the banks on two vCPUs.

The trainer fetches this automatically — there is nothing to do by hand:

colab run --gpu T4 train_qwen_arithmetic_t4.py --timeout 1h

Files

• banks_fp16_Qwen2.5-0.5B-Instruct.safetensors — the banks (942 MB, sha256 f3d896ed64d465efd17c32c6e66062ae)
• banks_fp16_Qwen2.5-0.5B-Instruct.json — sidecar: arch, shapes, provenance. The trainer asserts every arch field in it against its own config at load, so a mismatched bank file fails loudly instead of silently.
• tokenizer.json — the source repo's tokenizer, verbatim (sha256 c0382117ea329cdf097041132f6d7359), so neither the dataset prep nor the trainer touches another repo.

Banks

494,032,768 parameters (357,898,112 non-embedding) in 9 tensors, L = 24:

Why fp16

The T4 (sm75) has no bf16 tensor cores, so the trainer runs fp16 live weights against fp32 masters. Storing the banks in the run dtype keeps a 1 GB bf16 transient off a 16 GB card at load and means nothing in the T4 path touches a bf16 kernel.

The cast is free of consequence: the checkpoint is bf16, whose 7 explicit mantissa bits fit inside fp16's 10, so these tensors are bit-for-bit what the trainer used to produce by casting at load. 716 of 494,032,768 values (0.0001%) land under fp16's subnormal floor and become zero — the same ones, either way. Largest magnitude in the checkpoint is 214, against fp16's 65504 ceiling.

Reproducing it

data/prepare_model_t4.py in the repo above, from the pinned Qwen/Qwen2.5-0.5B-Instruct checkpoint. Both files' sha256 are in the sidecar.

License

apache-2.0, inherited from Qwen/Qwen2.5-0.5B-Instruct. Cite Qwen2.5 for the weights.