parrotlet-a-2.5-pro-decoder-8bit

parrotlet-a-2.5-pro — 8-bit decoder-only LLM.int8 (bitsandbytes)

Derivative of ekacare/parrotlet-a-2.5-pro for low-VRAM inference.

• Quantized: decoder (Gemma3ForConditionalGeneration, 34 layers) to 8-bit LLM.int8 (bitsandbytes, outlier threshold 6.0 — sensitive dims stay fp16).
• Kept fp16: Whisper encoder (32 layers) + EncoderProjectorConcat (Linear 2560->4096->2560). Projector is ~30M — quantizing saves nothing and breaks alignment.
• Quant device: Kaggle 2xT4, compute dtype fp16 (T4 has no fast bf16). Same stored weights run with bf16 compute on L4/Ada.
• Layout: identical hf_structure (encoder/, decoder/, projector/, root config.json, modelling_speech-llm.py) so the existing SpeechLLM.from_pretrained + Modal loader pattern works.
• Loader fix included: sampling_rate NameError fix + quantized-decoder reload + encoder/decoder split placement.
• License: Health AI Developer Foundations (inherited — same terms as base).

Reload (L4)

import importlib.util, sys
spec = importlib.util.spec_from_file_location("m", "modelling_speech-llm.py")
m = importlib.util.module_from_spec(spec); sys.modules["m"] = m; spec.loader.exec_module(m)
from transformers import AutoConfig, AutoModel
AutoConfig.register("speech-llm", m.SpeechLLMConfig); AutoModel.register(m.SpeechLLMConfig, m.SpeechLLM)
model = m.SpeechLLM.from_pretrained(REPO_OR_DIR, device="cuda")
print(model.transcribe(audio_16k_numpy, 16000, max_new_tokens=128))

Requires: transformers>=4.52,<5, bitsandbytes>=0.43, accelerate, librosa, soundfile. Full accuracy benchmark (semWER/kwWER, IndicVoices OOD) runs on L4 — this repo passed smoke transcribe() only.