talkie-1930-13b-base-hf

talkie-1930-13b-base (HF format)

HF-transformers–loadable conversion of talkie-lm/talkie-1930-13b-base. This repo repackages those weights into standard HF format.

What was done

• Weights: final.ckpt["model"] → model.safetensors.
  • Stripped the _orig_mod. (torch.compile) prefix and remapped to the TalkieForCausalLM layout (model.blocks.*, model.embed.weight, lm_head, lm_head_gain.w_g).
  • Cast float32 → bfloat16.
  • Embedding / lm_head zero-padded from vocab 65536 → 65540 to match the tokenizer/config below. (The 4 extra rows correspond to special tokens added during instruction tuning and are unused by the base model.)
• Architecture code, config.json, tokenizer: taken from lewtun/talkie-1930-13b-it-hf, which is the same architecture.

All 443 weight tensors map 1:1; the model loads with no missing/unexpected keys (~13.3B params).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("jannalu/talkie-1930-13b-base-hf", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "jannalu/talkie-1930-13b-base-hf", trust_remote_code=True
).bfloat16().cuda().eval()

Notes

• max_position_embeddings = 4096 (rope_theta = 1e6), matching the original training code's max_seq_len; longer-context use relies on RoPE extrapolation.
• This is a base model; the bundled tokenizer.json is the instruct tokenizer (first 65536 tokens are shared).