llama3.1-typhoon2-audio-8b-instruct-GPTQ-W4A16

Llama3.1 Typhoon2 Audio 8B Instruct — GPTQ W4A16

4-bit (GPTQ, W4A16, group size 128, asymmetric, compressed-tensors format) quantization of typhoon-ai/llama3.1-typhoon2-audio-8b-instruct.

Only the 8B Llama 3.1 language backbone (llama_model.model.layers.*) is quantized. The entire audio path is kept at full precision, so audio-in → text + audio-out (speech-to-speech) works exactly like the base model:

• Whisper speech encoder, BEATs encoder, Q-Former, audio→LLM projection — fp16
• CTC speech decoder (speech_generator) and HiFi-GAN vocoder — fp32
• lm_head — fp16

Usage

Same code and environment as the base model (transformers==4.45.2, fairseq==0.12.2, flash-attn, trust_remote_code=True):

import torch, soundfile as sf
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "hrnph/llama3.1-typhoon2-audio-8b-instruct-GPTQ-W4A16",
    torch_dtype=torch.float16,
    trust_remote_code=True,
)
model.to("cuda")

conversation = [
    {"role": "user", "content": [
        {"type": "audio", "audio_url": "audio.wav"},
        {"type": "text", "text": "โปรดถอดความข้อความในเสียงนี้"},
    ]}
]
out = model.generate(conversation=conversation)
print(out["text"])
sf.write("out.wav", out["audio"]["array"], out["audio"]["sampling_rate"])

Quantization details

• Tool: llmcompressor 0.5.1 oneshot
• Recipe: GPTQModifier(targets=["llama_model.model.layers.0..31"], ignore=["lm_head"], scheme="W4A16", dampening_frac=0.01)
• Calibration: 512 samples from HuggingFaceH4/ultrachat_200k, max sequence length 2048
• Hessians offloaded to CPU; single NVIDIA A10G 24GB (Modal)
• Approx. sizes: backbone ~16 GB → ~5.3 GB; full checkpoint ~19 GB → ~8 GB

Compatibility

This is not a vLLM/SGLang/GGUF checkpoint. The base architecture (Typhoon2Audio2AudioForConditionalGeneration, remote code with a fairseq vocoder and a custom CTC generation loop) is not implemented in vLLM, SGLang or llama.cpp even at fp16 — load it with transformers + trust_remote_code=True as shown above.

Quantized and uploaded automatically on Modal.