indic-parler-tts-awq-int4

Indic Parler-TTS — AWQ-scaled, TorchAO int4-packed

An int4-quantized checkpoint of ai4bharat/indic-parler-tts: AWQ's per-channel, calibration-aware scales computed and folded through the network, then packed with TorchAO's real int4 format (Int4TilePackedTo4dTensor, group_size=64).

Results vs. the fp16 baseline: ~54% reduction in load-time weight memory, ~13% reduction in peak memory during generation, comparable inference speed, and a word-error-rate tax of only +0.0265 (vs. +0.0963 for naive TorchAO int4 quantization at the identical bit-width — see the full technical report for methodology and complete results).

Developed by David Babu. Full methodology, verification steps, and per-language results: see TECHNICAL_REPORT.md in github.com/hazardscarn/vyoma_hackathon.

Usage

This checkpoint is a raw state_dict.pt (TorchAO's tensor-subclass weights can't be saved via save_pretrained), so reloading requires re-creating the packed structure first. Use the tts_exports module from the GitHub repo above, or:

import torch
import torch.nn as nn
from transformers import AutoTokenizer
from parler_tts import ParlerTTSForConditionalGeneration
from torchao.quantization import quantize_, Int4WeightOnlyConfig
from huggingface_hub import hf_hub_download

REPO_ID = "hazardscarn10/indic-parler-tts-awq-int4"
BASE_MODEL_ID = "ai4bharat/indic-parler-tts"

def filter_fn(module, fqn):
    return isinstance(module, nn.Linear) and "lm_heads" not in fqn

model = ParlerTTSForConditionalGeneration.from_pretrained(BASE_MODEL_ID, torch_dtype=torch.bfloat16).to("cuda")
quantize_(model, Int4WeightOnlyConfig(group_size=64, int4_packing_format="tile_packed_to_4d"), filter_fn=filter_fn)

state_dict_path = hf_hub_download(repo_id=REPO_ID, filename="state_dict.pt")
state_dict = torch.load(state_dict_path, map_location="cuda", weights_only=False)
model.load_state_dict(state_dict, strict=True)
model.eval()