talkie-1930-13b-base-hf

talkie-1930-13b-it (Transformers format)

This is a conversion of talkie-lm/talkie-1930-13b-base to the HuggingFace Transformers format. The original model was distributed as a raw PyTorch checkpoint with a custom inference library; this version can be loaded directly with AutoModelForCausalLM and AutoTokenizer.

<!-- To be tested -- The weights are **numerically identical** to the original — top-5 decoded tokens match across all test prompts, with max logit differences below 0.07 (bf16 rounding). -->

[!NOTE] This base model was ported to Hugging Face format by Xiaoning Wang, based on the version automatically converted by Hugging Face's ML Intern at lewtun/talkie-1930-13b-it-hf — an AI agent for ML engineering tasks. Try it yourself via the CLI or the Demo.

Table of Contents

1. Model Summary
2. How to Use
3. Architecture Details
4. Conversion Notes
5. License

Model Summary

talkie-1930-13b-it is a 13B-parameter instruction-tuned language model from the talkie family, developed by Alec Radford, Nick Levine, and David Duvenaud. It was pretrained on 260B tokens of pre-1931 English-language text and instruction-tuned using a novel dataset extracted from vintage reference works — etiquette manuals, encyclopedias, letter-writing guides, and poetry collections. The model underwent reinforcement learning via online DPO with an LLM-as-a-judge to improve instruction following.

Read more in the talkie report.

Key Features

• Vintage knowledge: trained exclusively on pre-1931 text, offering a unique window into early 20th-century language and thought
• Instruction-tuned: fine-tuned for conversational use with a simple chat template
• 13B parameters in bfloat16 (~26 GB VRAM)
• 2048 token context window

How to Use

Installation

This model uses custom modeling code. Make sure you have a recent version of transformers installed:

pip install -U transformers torch

Basic Generation

To be added.

Multi-turn Chat

To be added.

Chat Template

There is no chat template for this base model.

Architecture Details

talkie is a 40-layer decoder-only GPT with several distinctive architectural choices:

Notable architectural features:

• QK-normalization: RMSNorm is applied to queries and keys after RoPE
• Per-head gain: learnable scalar gain per attention head, applied to queries
• Embedding skip connections: each transformer block receives a residual connection from the (normalized) input embeddings
• Activation gains: learnable scalar gains on attention and MLP residual streams (initialized to (2·L)^(-0.5))
• lm_head weight gain: a learnable scalar applied to the output projection weights

Conversion Notes

This model was converted from the original talkie-lm/talkie-1930-13b-it PyTorch checkpoint using the reference talkie codebase as ground truth. The conversion involved:

1. Model weights: the .pt state dict was remapped to a PreTrainedModel subclass (TalkieForCausalLM) and saved as safetensors
2. Tokenizer: the tiktoken BPE vocabulary was converted to a PreTrainedTokenizerFast with the HuggingFace TikTokenConverter, including all 5 special tokens (<|endoftext|>, <|end|>, <|user|>, <|assistant|>, <|system|>)
3. Validation: logits were compared on 4 test prompts covering chat, system prompts, and raw completion — all top-5 decoded tokens match exactly, with cosine similarity ≥ 0.99999994

Since this is a custom architecture, loading requires trust_remote_code=True.

License

Apache 2.0 — same as the original model.