Qwen2.5-0.5B-Asymmetric-Slim-4bit

Qwen2.5-0.5B-Asymmetric-Slim-4bit

This model is a compressed and optimized version of Qwen/Qwen2.5-0.5B-Instruct using the Manifold Diagnostic & Orthogonal Compression framework.

Statistics

Usage

This model requires trust_remote_code=True to execute the custom load-time de-quantizer and reconstruction logic.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "Uplaxus/Qwen2.5-0.5B-Asymmetric-Slim-4bit",
    torch_dtype=torch.float16,
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("Uplaxus/Qwen2.5-0.5B-Asymmetric-Slim-4bit")

# Run inference
prompt = "Hello, my friend! How are you doing today?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Structure of the Repository

• model.safetensors: Packed 4-bit weights and scales.
• config.json: Configuration defining 21 layers and the auto_map trigger.
• modeling_qwen2_manifold.py: The custom loading wrapper that reconstructs FP16 weights on startup.
• Tokenizer configuration and vocabulary files.