aaie-8k-gft-5m

AAIE 8K GFT — 5M checkpoint

This is the selected 5-million-token general-instruction-tuning checkpoint from the AAIE research project. It is a standalone, full-weight Hugging Face model: there is no LoRA adapter to attach or merge.

Lineage

AAIE 50M YaRN 8K base -> selective SmolTalk V2 general instruction tuning -> selected 5M checkpoint

• Architecture: Llama-compatible dense decoder, 20 layers, hidden size 512, eight attention heads / two KV heads, tied embeddings.
• Context configuration: 8,192 tokens using YaRN (factor: 8, original position length 1,024).
• Tokenizer vocabulary: 151,936 tokens.
• Training: full-weight, completion-masked instruction tuning; selected after a bounded 5M-token schedule at learning rate 1e-4.
• Model-weight SHA-256: 4b1c4cc74f0f0dadd9923b4381151444c04e00682a39301b812a2116ec7c22b7.

The generation configuration corrects the checkpoint package's end-token settings: <|im_end|> (151645) and <|endoftext|> (151643) both stop generation.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "moinsaj/aaie-8k-gft-5m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "Explain database indexes in two sentences."}]
inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
)
output = model.generate(inputs, max_new_tokens=160, do_sample=False)
print(tokenizer.decode(output[0, inputs.shape[-1]:], skip_special_tokens=True))

Important limitations

This is an AI-reviewed research checkpoint, not an educational assessment tool or a replacement for the AAIE product's hosted model. Its bounded evaluation showed better stopping behaviour than the unchanged base, but it did not demonstrate dependable instruction correctness, extraction, or criterion-level feedback. It requires further human/educator validation before any applied use.

The 8K context setting is an experimental YaRN extension. It should not be read as evidence of reliable general long-context reasoning.