Carnice-V2-27b-AutoRound-W4A16-g128

Carnice-V2-27b-AutoRound-W4A16-g128

<!-- polished-overview:start -->

Overview

Carnice-V2-27b-AutoRound-W4A16-g128 is a weight-quantized checkpoint intended for efficient GPU inference, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

Quick start

vLLM (AWQ-compatible runtimes)

vllm serve groxaxo/Carnice-V2-27b-AutoRound-W4A16-g128 \
  --quantization awq_marlin \
  --dtype float16 \
  --trust-remote-code

The exact kernel and flags depend on the quantizer and architecture. Check the files and source model card before selecting a production serving configuration.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

<!-- polished-overview:end -->