nla-gemma3-27b-av-FP8

kitft/nla-gemma3-27b-av (FP8, compressed-tensors)

NLA action verbalizer with FP8_DYNAMIC weight-only FP8 quantization via llmcompressor. Saved in compressed-tensors format with FP8 weights (*.safetensors) and quantization_config in config.json, so sgl.Engine loads it directly with no bf16 staging transient.

• FP8_DYNAMIC quantization on every nn.Linear weight in the backbone (lm_head is NOT quantized — output projection numerics matter).
• All upstream files (nla_meta.yaml, prompt templates, model card, license, generation config, tokenizer extras, …) are preserved verbatim. The upstream README.md is renamed to README_upstream.md.

Usage with apps/nla/server.py

export NLA_VERBALIZER_MODEL=<this repo>
export NLA_VERBALIZER_QUANTIZATION=compressed-tensors
# Do NOT set NLA_FP8_VERBALIZER — it is overridden by the explicit
# --verbalizer-quantization (or NLA_VERBALIZER_QUANTIZATION) anyway.
uv run server.py --truncate-source --fp8-source --int4-reconstructor

apps/nla/server.py will pass quantization="compressed-tensors" to sgl.Engine, which loads the pre-quantized FP8 weights with no bf16 load-time peak. See the apps/nla README's "Multi-GPU > Layout E" for the full per-GPU memory budget.

Comparison vs apps/nla/build_quantized_models.py --target verbalizer-fp8

That sibling script uses torchao (AffineQuantizedTensor-backed) FP8. Sglang 0.5.x has no "torchao" entry in its quantization registry, so that output is not loadable by apps/nla/server.py. Use this checkpoint for the sglang verbalizer; reserve the torchao path for the HF source/reconstructor (which transformers consumes natively).