aaie-ddense-pretrain-llama

AAIE-Distilled Dense Pretrain (Llama-compatible export)

A LlamaForCausalLM-format export of AAIE-Distilled Dense Pretrain — same weights, same architecture, just registered under transformers'/vLLM's native Llama implementation instead of custom trust_remote_code.

Why this exists

This model's architecture — GQA attention, RoPE, SwiGLU FFN, RMSNorm, tied embeddings, no biases anywhere — happens to be structurally identical to Llama's, just with different tensor/field names. This export is a pure state_dict + config remap, not a retrain: every tensor is copied unchanged into the equivalent Llama-named slot. Verified via a trust_remote_code-free transformers sanity check (same prompt, same greedy decode as the original).

Payoff: loads natively in vLLM (no trust_remote_code, no custom backend) using vLLM's own hand-optimized LlamaForCausalLM implementation.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("namquangstudy/aaie-ddense-pretrain-llama")
model = AutoModelForCausalLM.from_pretrained("namquangstudy/aaie-ddense-pretrain-llama")
# no trust_remote_code needed - this is a plain LlamaForCausalLM

from vllm import LLM, SamplingParams

llm = LLM(model="namquangstudy/aaie-ddense-pretrain-llama")
out = llm.generate(["The purpose of a database index is"], SamplingParams(max_tokens=100))
print(out[0].outputs[0].text)

Full details

This is the pretrained + distilled checkpoint — no instruction-tuning, behaves as a raw text-continuation model (won't follow instructions or answer questions in a chat format). Full architecture, training data, and hyperparameters are documented on the original AAIE-Distilled Dense Pretrain model card — this export carries the exact same weights and behavior, just packaged for vLLM/ecosystem compatibility.