qwen3-0.6b-engram

Qwen3-0.6B-Engram

An implementation of DeepSeek conditional memory via an offloaded Look-Up Table (LUT) alongside Qwen3-0.6B-Base.

English: a 2gb engram embedding layer in qwen 3 0.6b

Benchmark Results (WikiText-2 Test Split)

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_PATH = "./qwen3-engram"

model = AutoModelForCausalLM.from_pretrained(
    MODEL_PATH,
    trust_remote_code=True,
    torch_dtype=torch.float32,
    device_map="cpu"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)

prompt = "The most common cause of acute renal failure is"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=192, use_cache=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))