sarashina2-70b-32k

sarashina2-70b-32k (unofficial long-context config variant)

[!WARNING] This is not an official SB Intuitions release, and it is not a quality improvement. It is an unmodified copy of the sbintuitions/sarashina2-70b weights with a single config change that raises the declared context window from 8192 to 32768. No long-context training was performed. Output beyond ~8192 tokens is expected to degrade badly and may be incoherent. It exists to benchmark inference compute at long sequence lengths. For actual use, go to the original model.

What was changed

Sarashina2-70b uses rotary position embeddings (RoPE). RoPE has no learned position parameters — the cos/sin tables are computed analytically from the position index at runtime — so there is no position-embedding matrix to resize. The 8192 ceiling existed only in config.json.

The weights are byte-for-byte identical to the base model. The complete diff:

  # config.json
- "max_position_embeddings": 8192,
+ "max_position_embeddings": 32768,

  # tokenizer_config.json  (so the tokenizer doesn't truncate first)
+ "model_max_length": 32768,

rope_theta stays at 10000.0 and rope_scaling stays null, so the attention math is identical to the base model — timing differences are attributable to sequence length alone. A companion variant applying linear position interpolation ({"rope_type": "linear", "factor": 4.0}) is a strictly better choice if you care at all about output degrading gracefully.

32768 was chosen over a bare 24576 to leave headroom for a 24k prompt plus generation.

Architecture

Resource requirements

• Weights (bf16): 140.3 GB
• KV cache: 80 layers x 2 (K+V) x 8 KV heads x 128 head_dim x 2 bytes = 327 680 B/token (0.3125 MiB/token)
  • 8k: 2.68 GB · 24k: 8.05 GB · 32k: 10.7 GB per sequence

A single 24k-token sequence needs ~148 GB total. Practical minimum is 2x H100/H200 80GB (--tensor-parallel-size 2), or FP8/AWQ quantisation to ~70 GB.

Usage

vllm serve <this-repo> \
  --max-model-len 24576 \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.92

[!TIP] You do not actually need this repo. The same effect is achievable against the original model without downloading anything extra:

vllm serve sbintuitions/sarashina2-70b --max-model-len 24576 \
  --hf-overrides '{"max_position_embeddings": 32768}'

AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina2-70b",
                                     max_position_embeddings=32768)

This repo just packages that as a fixed, reproducible artifact.

Verification performed

• 30/30 shards, 140.3 GB matching model.safetensors.index.json total_size exactly; 723/723 tensors parsed; 70.13 B params; all BF16
• RoPE cos/sin tables finite across the full 32768 window
• Real-weight forward pass at 24 576 tokens: layer 0's actual bf16 weights (9/9 tensors, no missing/unexpected keys) through a real LlamaDecoderLayer. Finite output over (1, 24576, 8192); activation norms stable past the old ceiling (pos 8191 → 2.71, pos 24575 → 2.78), i.e. no numerical blow-up.

Not verified: quality/perplexity at >8k, and stacking all 80 layers (no GPU was available). Assume quality is bad.

License & attribution

MIT, inherited from the base model. Copyright (c) 2024 SB Intuitions — the original LICENSE is retained unmodified in this repo. All model weights are the work of SB Intuitions; the only contribution here is a two-line config edit.