Qwen3.8-27B-ABLITERATED-EXL3-2.25bpw

Qwen3.8-27B-ABLITERATED-EXL3-2.25bpw

by WatchDG

EXL3 quantization of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 (commit 9d85770e5eb602322b4bceef55beda357e0bd0ca), an abliterated Qwen3.8-27B multimodal model.

Third-party derivative, unevaluated by upstream. This is a community EXL3 build, not an official Qwen or Blackfrost release. Blackfrost's own model card states: "Any additional direction editing, fine-tuning, merging, pruning, quantization, or other weight change creates an artifact Blackfrost has not evaluated unless a new report explicitly states otherwise." That applies directly here — quality, refusal behavior, reasoning, tool use, vision/video behavior, and long-context stability of this specific 2.25 bpw quant have not been assessed by Blackfrost and may differ from the BF16 reference. This is a weight-level research checkpoint with a deliberately reduced refusal surface; it is not a safety-stock model and must not be represented, deployed, or evaluated as one.

• Architecture: Qwen3_5ForConditionalGeneration (hybrid linear + full attention, 64 layers, MTP)
• Parameters: ~27.78 B (text + vision)
• Native context: 262 144 tokens
• Vocab: 248 320 (Qwen3.8)
• Build wall time: 1 h 33 min 14 s end-to-end (per-module sum: 1 h 04 min) on a single RTX 4090 (24 GB), exllamav3 v1.4.2, calibration 250×2048, codebook mul1, out_scales always. The per-module sum is the cumulative time the model modules spent actively being processed (sum of per-module [NN.NN s] markers across the log).
• VRAM (observed, TabbyAPI, single user): ~15.6 GB (16 013 MiB) with cache_size: 65536, cache_mode: Q8, max_seq_len: 65536 — leaves ~8.4 GB headroom on a 24 GB card.
• Quantization: EXL3 v1.4.2
• Disk size: 10.74 GB (2 shards: 7.98 + 2.77 GB)

Quantization

• Codebook: mul1 (default)
• Out scales: always
• Calibration: exllamav3 v1.4.2 standard data (c4/code/multilingual/technical/wiki/tiny), 250 rows × 2048 cols
• Per-module SQNR (65 of 70 modules; 5 stored unquantized at 16 bpw and excluded because exllamav3 prints sqnr=0 for them): median 25.37 dB, mean 27.24 dB, min 21.88 dB (layer 51, bpw=2.29), max 43.00 dB (lm_head, bpw=6.00). Layer SQNR drops with depth: layers 0–2 land at 32–36 dB, layers 3–19 at 26–33 dB, layers 20+ at 22–26 dB — deeper layers are the SQNR floor.

Effective average over the whole file (including head/MTP/vision): 3.09 bpw (10.74 GB × 8 / 27.78 B params).

Files

Sizes are in bytes. SHA-256 is the first 16 hex chars (truncated).

Hashes for the 8 infrastructure files are byte-identical to the corresponding files in the Blackfrost BF16 source (synced via the 3.75bpw/2.75bpw quant builds). The two quant-meta files (config.json, quantization_config.json) and the shard files are necessarily different per bpw; their hashes here are the actual ones for this build.

Modifications vs upstream

This 2.25 bpw build contains the following file categories:

1. Byte-identical to the Blackfrost BF16 source (8 files): .gitattributes, LICENSE, chat_template.jinja (upstream variant, before the developer-role patch), generation_config.json, tokenizer.json, tokenizer_config.json, preprocessor_config.json, video_preprocessor_config.json.
2. exllamav3-generated quant metadata (2 files):
   • config.json — the upstream JSON is preserved unchanged; exllamav3 appended a quantization_config block with bits: 2.25, head_bits: 6, ....
   • quantization_config.json — exllamav3 per-tensor bit/storage map, not present in the BF16 source.
3. 2.25 bpw build artifacts (3 files): the two weight shards model-0000{1,2}-of-00002.safetensors and the model.safetensors.index.json (whose tensor→shard offsets reflect the actual 2.25 shard sizes, not the BF16 ones).

One further intentional patch on top of the upstream files:

• chat_template.jinja — added or message.role == "developer" in three places so that the template also accepts role=developer as an alias for role=system. The upstream template only recognises system/user/assistant/tool; the developer role is used by some OpenAI-style APIs for the higher-priority instruction.

Quick start (TabbyAPI + exllamav3)

Full tabby-config-2.25.yml used to serve this model on a 24 GB GPU (RTX 4090 verified):

network:
  host: 0.0.0.0
  port: 5000
  disable_auth: true       # only safe behind localhost / a trusted network

model:
  model_dir: /app/models
  model_name: Qwen3.8-27B-ABLITERATED-EXL3-2.25bpw

  # --- KV cache (24 GB budget) ---
  cache_size: 65536        # 64k tokens (use 32k for tighter headroom at 2.25 bpw)
  cache_mode: Q8           # 8-bit cache; halves VRAM vs FP16, < 10% speed cost
  max_seq_len: 65536       # cap request length to the cache

  # --- EXL3-specific (optional) ---
  chunk_size: 2048
  output_chunking: true
  inline_model_loading: true

  gpu_split_auto: true
  autosplit_reserve: [96]  # MiB reserved for activations on the single GPU

  prompt_template:
  template_vars_default:
    enable_thinking: true
  reasoning: true          # required for tool call parsing; splits reasoning_content / content
  vision: true             # model is multimodal (Qwen3.5 vision tower)

  # --- tool / function calling parser ---
  tool_format: qwen3_coder
  tool_calls_in_reasoning: true

  use_dummy_models: true
  dummy_model_names: ["qwen3.8-27b"]

draft_model:
  draft_mode: disabled

sampling:
  override_preset: safe_defaults

logging:
  log_prompt: false
  log_generation_params: false
  log_requests: false
  log_chat_completion_requests: false

memory:
  cuda_malloc_async: true

developer:
  unsafe_launch: false

Note: the ghcr.io/theroyallab/tabbyapi image lacks python3.12-dev, which Triton needs for JIT compilation of cuda_utils.c. The entrypoint is overridden to install the dev headers and start the server in one shot:

docker run --rm --gpus all -p 10404:5000 \
  -v './Qwen3.8-27B-ABLITERATED-EXL3-2.25bpw:/app/models/Qwen3.8-27B-ABLITERATED-EXL3-2.25bpw:ro' \
  -v './tabby-config-2.25.yml:/app/config.yml:ro' \
  --entrypoint bash ghcr.io/theroyallab/tabbyapi:latest \
  -c "apt-get update -qq && apt-get install -y --no-install-recommends python3.12-dev gcc && exec python3 main.py --host 0.0.0.0"

VRAM observed at 64k cache for this 2.25 build: ~15.6 GB (16 013 MiB) with cache_size=65536, cache_mode=Q8, max_seq_len=65536 (RTX 4090, single user) — vs 16.6 GB for the 2.75bpw twin at the same cache_size; the ~964 MiB delta matches the ~1.5 GB smaller weight footprint.

Build provenance

• Quantizer: turboderp-org/exllamav3 (tag v1.4.2)
• Host: Windows 11, Python 3.14.7, torch 2.11.0+cu128, CUDA 12.8, triton-windows 3.7.1
• GPU: NVIDIA GeForce RTX 4090 (24 GB, sm 8.9), driver 610.88
• Build wall time: 1 h 33 min 14 s end-to-end (per-module sum: 1 h 04 min) — pure EXL3 quant phase on RTX 4090.
• Start / end timestamps: 2026-08-16 14:15:30 → 2026-08-16 15:48:44
• Modules quantized: 70 (1 embed_tokens + 64 transformer + 1 norm + 1 lm_head + 3 mtp/projection)
• Calibration source: exllamav3 v1.4.2 standard_cal_data (6 files, ~5.6 MB)
• VRAM at inference (TabbyAPI, observed): ~15.6 GB (16 013 MiB) with cache_size: 65536, cache_mode: Q8, max_seq_len: 65536. Single user, RTX 4090 24 GB. 128k-cache figure not yet measured for this build (2.75bpw twin measured ~19.5 GB at 128k).

License

Apache License 2.0. See LICENSE — inherited from the upstream Blackfrost repo, which traces to Alibaba Cloud (Qwen authors).