talkie-web-13b-base-bf16

talkie-web-13b-base-bf16

HuggingFace-compatible BF16 safetensors weights for talkie-lm/talkie-web-13b-base. A 13B decoder-only transformer trained on 260B tokens of FineWeb by Nick Levine, David Duvenaud, and Alec Radford. Released under Apache 2.0.

What this repository is

The upstream talkie-lm/talkie-web-13b-base release ships a single base.ckpt PyTorch pickle (FP32, ~53 GB). This repository repackages the same weights as BF16 sharded safetensors with a complete config.json, ready to load in any inference engine that consumes the HuggingFace layout.

Tensor values are bit-identical to the upstream checkpoint after a deterministic FP32 → BF16 cast. No layers are merged or folded — every parameter present in the original checkpoint is preserved as an explicit named tensor, including the per-residual ActGain scalars, the per-head HeadGain vectors, and the lm_head_gain WeightGain scalar.

Architecture

Decoder-only transformer with the following Talkie-specific characteristics on top of an otherwise standard pre-norm Llama-shaped trunk:

• 40 layers, 40 attention heads, no GQA (num_key_value_heads = num_attention_heads)
• 5120 hidden size, 13696 SwiGLU intermediate size, 128 head dim
• Non-parametric RMSNorm everywhere (rms_norm_parametric: false — no learned γ)
• Q/K RMSNorm inside attention, after RoPE, before SDPA (qk_norm: true)
• HeadGain — per-head learned scalar on Q, applied after Q-norm
• ActGain — per-residual learned scalar on the attention output, the MLP output, and the embedding skip path, initialized to (2L)^(-1/2) ≈ 0.1118 for the first two and 0.0 for the embedding skip
• Embedding skip — the post-norm input embedding is re-injected at every block through its own ActGain
• WeightGain (lm_head_gain.w_g) — a single scalar multiplied into lm_head.weight before the output matmul
• RoPE — half-then-half rotation with base = 1_000_000, rotating by −θ (sign convention is load-bearing for Q·K^T; engines using the standard +θ rotation should pass negated frequencies)

The relevant non-standard fields are present in config.json so a third-party inference engine has everything needed to construct the model without reading any code.

Files

License and attribution

Released under Apache 2.0, inherited from upstream. The base weights, training data, and architecture are the work of the Talkie authors. See the upstream model card at talkie-lm/talkie-web-13b-base for full attribution, training methodology, and citation information.