laya-quanto

Laya Quanto Quantizations

This repository bundles native Optimum-Quanto weight-only quantizations of convaiinnovations/laya in one place:

Embeddings and LayerNorm weights remain floating point. The variants use Quanto's native quantized Linear kernels; they are not the older CPU dequantization implementation.

Install

pip install -r requirements.txt

Use either variant

from quanto_laya import load_quantized_agent

repo = "/path/to/laya-quanto"
agent = load_quantized_agent(repo, variant="q8", device="cuda")
# Or: variant="q4"
result = agent.system_one(state, questions)

The q8/ and q4/ directories contain each variant's model.safetensors, Quanto quantization map, and metadata. Shared tokenizer, encoder configuration, decision configuration, and loader files are at the repository root.

Derived from the original Laya checkpoint, released under its Apache 2.0 terms.