ThinkingCap-Qwen3.6-27B-int4-AutoRound-v1

ThinkingCap-Qwen3.6-27B INT4 AutoRound

A fast INT4 / W4A16 AutoRound quantization of bottlecapai/ThinkingCap-Qwen3.6-27B, prepared for local long-context agent serving with vLLM.

This is a practical first release: small enough to run comfortably on a 32 GB RTX 5090-class card, with Qwen3.6 thinking/tooling behavior preserved and Multi-Token Prediction (MTP) tensors kept in BF16 for speculative decoding.

A higher-quality v2 quant is planned later. The likely direction is a more quality-oriented recipe and/or stronger calibration, after we finish comparing the speed / context / quality tradeoffs.

Credits

Huge thanks to BottleCap AI for releasing ThinkingCap. Their checkpoint is the model here; this repo is only a community quantization of it.

Also, personally: BottleCap are local friends/neighbours — their offices are about 200 meters from me, and I have known them for years — so this one is especially fun to package properly. Please give the original model and team credit:

• Base model: bottlecapai/ThinkingCap-Qwen3.6-27B
• BottleCap AI: https://www.bottlecapai.com/

What is ThinkingCap?

From the original BottleCap release: ThinkingCap keeps the capability and style of Qwen3.6-27B while using substantially fewer thinking tokens on average. See the original model card for the full evaluation table, methodology, citation, and intended sampling settings.

This quant keeps the base/default sampling recipe:

temperature = 1.0
top_p       = 0.95
top_k       = 20
min_p       = 0.0

Quantization recipe

Notes:

• This is the group-size 128 release. It prioritizes speed, smaller resident VRAM, and long-context KV headroom.
• Smaller group sizes such as GS32/GS16 may improve quality but increase metadata size and reduce KV-cache room. A later v2 may explore that direction.
• MTP/speculative-decoding tensors were restored from the BF16 source auxiliary tensors instead of leaving them packed/quantized.

Recommended long-context / agent serving config

This is the more tuned single-GPU configuration used during local testing:

vllm serve "$MODEL_ID" \
  --dtype half \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype turboquant_4bit_nc \
  --max-num-seqs 16 \
  --max-num-batched-tokens 4096 \
  --enable-chunked-prefill \
  --max-long-partial-prefills 1 \
  --long-prefill-token-threshold 4096 \
  --compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,3,4],"max_cudagraph_capture_size":4}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
  --port 8888 \
  --host 0.0.0.0 \
  --trust-remote-code \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Notes:

• turboquant_4bit_nc is a vLLM KV-cache quantization mode, not the model weight quantization. It is useful for fitting long context.
• --max-num-seqs 16 allows many shorter concurrent requests; it does not mean sixteen full 262k requests fit simultaneously.
• --compilation-config ... PIECEWISE ... [1,2,3,4] keeps CUDA graphs enabled but caps graph capture sizes to save VRAM.
• vLLM 0.24 rejected non-default concurrent partial prefill in this environment, so the command intentionally does not pass --max-num-partial-prefills 4.

BenchLocal snapshots

Local BenchLocal screenshots; preliminary smoke/quality signals, not a complete public evaluation suite. Run target: vllm 5090 on the local W4G128 checkpoint.

Hermes Agent benchmark

The main benchmark for this release: a Hermes-style agent suite covering memory, scheduling, delegation/recovery, procedural skills, and workspace orchestration.

Other local BenchLocal runs

Performance notes

Local inference-benchmarker tests against vLLM used ~256 input tokens and ~64 output tokens with thinking disabled for content-throughput measurement. The tuned server reached approximately:

These are local measurements on one machine and are not a formal quality benchmark. They are included to show that the quant is quick and usable under concurrent short-request traffic.

Compatibility

Known-good path:

• vLLM 0.24.0
• CUDA 13.x on RTX 5090
• --kv-cache-dtype turboquant_4bit_nc
• MTP speculative decoding with num_speculative_tokens=1

The model should also be usable through Transformers/GPTQModel-compatible AutoRound loaders, but the most exercised runtime for this artifact is vLLM.

Limitations

• This is an INT4 quantization; it is not expected to exactly match BF16 quality.
• Group size 128 is chosen for practicality. Future v2 variants may trade more disk/VRAM for quality.
• The included performance numbers are local serving benchmarks, not broad model evaluations.
• For long-context serving, KV-cache settings matter as much as model size. If you disable KV quantization, reduce context/concurrency accordingly.

Citation

If you use this quant, please cite and credit the original BottleCap model. Original citation from their model card:

@misc{,
  title  = {bottlecapai/ThinkingCap-Qwen3.6-27B},
  author = {Lasocki, Karol and Osusky, Adam and Lindauer, Jan and Jirkovsky, Adam and Mihal, Filip and Platek, Ondrej and Herel, David and Ihnatchenko, Luka and Bartek, Vojtech and Jirak, Jiri and Mikolov, Tomas},
  year   = {2026},
}