sicariussicariistuff-impish_nemo_12B-trt-fp8

SicariusSicariiStuff/Impish_Nemo_12B — FP8 (TensorRT-LLM)

This is a FP8 quantized version of https://huggingface.co/SicariusSicariiStuff/Impish_Nemo_12B, optimized for TensorRT-LLM inference.

---

Model Overview

Key Features:

• High-Performance Inference: Optimized for NVIDIA GPUs with TensorRT-LLM
• Memory Efficient: 8 (FP8)-bit weights reduce VRAM usage vs FP16
• Production Ready: Built for low-latency, high-throughput chat serving
• Portable Checkpoints: Checkpoints work across systems; engines are hardware-specific

---

Technical Specifications

---

Artifact Layout

trt-llm/
  checkpoints/
    *.safetensors
    config.json
  engines/sm90_trt-llm-1.2.0rc5_cuda13.0/ 
    rank*.engine
    config.json

---

Quantization Details

---

Compatibility

Requirements

• GPU: NVIDIA with Compute Capability ≥ 9.0 (Hopper / H100)
• CUDA: 13.0+
• TensorRT-LLM: 1.2.0rc5
• Python: 3.10+

Portability Notes

• Checkpoints: Portable across systems with compatible TensorRT-LLM versions; rebuild engines on the target GPU
• Engines: Hardware-specific (rebuild for different GPU/CUDA versions/SMs, e.g., H100/H200/B200/Blackwell, L40S, 4090/RTX)
• INT4-AWQ checkpoints are portable across sm89/sm90+ GPUs; rebuild engines for the target GPU (e.g., H100/H200/B200/Blackwell, L40S, 4090/RTX) or reuse one of the prebuild engines in case they match your GPU.

---

Troubleshooting

<details> <summary><b>Engine fails to load on different GPU</b></summary> Engines are compiled for specific SM architecture and CUDA version. Either:

1. Use the checkpoints and rebuild the engine on your target system
2. Download an engine matching your GPU from the engines/ subdirectories

</details> <details> <summary><b>Out of Memory</b></summary> Reduce `max_batch_size` or `max_seq_len` when building the engine. Adjust `kv_cache_config.free_gpu_memory_fraction` at runtime. </details>

---

Resources

• TensorRT-LLM Documentation
• Source Model

---

License

This quantized model inherits the license from the original base model: apache-2.0

See the original model's license for full terms.