Qwen3.8-27B-ABLITERATED-EXL3-2.75bpw

Qwen3.8-27B-ABLITERATED-EXL3-2.75bpw

by WatchDG

EXL3 quantization of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 (commit 9d85770e5eb602322b4bceef55beda357e0bd0ca), an abliterated Qwen3.8-27B multimodal model.

Third-party derivative, unevaluated by upstream. This is a community EXL3 build, not an official Qwen or Blackfrost release. Blackfrost's own model card states: "Any additional direction editing, fine-tuning, merging, pruning, quantization, or other weight change creates an artifact Blackfrost has not evaluated unless a new report explicitly states otherwise." That applies directly here — quality, refusal behavior, reasoning, tool use, vision/video behavior, and long-context stability of this specific 2.75 bpw quant have not been assessed by Blackfrost and may differ from the BF16 reference. This is a weight-level research checkpoint with a deliberately reduced refusal surface; it is not a safety-stock model and must not be represented, deployed, or evaluated as one.

• Architecture: Qwen3_5ForConditionalGeneration (hybrid linear + full attention, 64 layers, MTP)
• Parameters: ~27.78 B (text + vision)
• Native context: 262 144 tokens
• Vocab: 248 320 (Qwen3.8)
• Build compute time: 54 min 36 s per-module sum on a single RTX 4090 (24 GB), exllamav3 v1.4.2, calibration 250×2048, codebook mul1, out_scales always. The per-module sum is the cumulative time the model modules spent actively being processed (sum of per-module [NN.NN s] markers across the log, 70 modules). The run was interrupted once (crash during layers.24) and resumed from a checkpoint, so a single clean wall-clock end-to-end figure isn't meaningful for this build.
• VRAM (observed at load, TabbyAPI, single user): ~16.6 GB (16 977 MiB) with cache_size: 65536, cache_mode: Q8, max_seq_len: 65536 — leaves ~7.4 GB headroom on a 24 GB card.
• Quantization: EXL3 v1.4.2
• Disk size: 12.18 GB (2 shards: 7.94 + 4.22 GB)

Quantization

• Codebook: mul1 (default)
• Out scales: always
• Calibration: exllamav3 v1.4.2 standard data (c4/code/multilingual/technical/wiki/tiny), 250 rows × 2048 cols
• Per-module SQNR (65 of 70 modules; 5 stored unquantized at 16 bpw and excluded because exllamav3 prints sqnr=0 for them): median 29.15 dB, mean 30.27 dB, min 22.84 dB (layer 35, bpw=2.29), max 42.90 dB (lm_head, bpw=6.00). Heterogeneous by layer type: full-attention layers 0–20 and 44–63 land at 27–40 dB, linear-attention layers 21–43 (the 2.29–2.33 bpw group) at 22–26 dB.

Effective average over the whole file (including head/MTP/vision): ~3.51 bpw (12.18 GB × 8 / 27.78 B params).

Files

Modifications vs upstream

This 2.75 bpw build contains the following file categories:

1. Byte-identical to the Blackfrost BF16 source (8 files): .gitattributes, LICENSE, chat_template.jinja (upstream variant, before the developer-role patch), generation_config.json, tokenizer.json, tokenizer_config.json, preprocessor_config.json, video_preprocessor_config.json.
2. exllamav3-generated quant metadata (2 files):
   • config.json — the upstream JSON is preserved unchanged; exllamav3 appended a quantization_config block with bits: 2.75, head_bits: 6, ....
   • quantization_config.json — exllamav3 per-tensor bit/storage map, not present in the BF16 source.
3. 2.75 bpw build artifacts (3 files): the two weight shards model-0000{1,2}-of-00002.safetensors and the model.safetensors.index.json (whose tensor→shard offsets reflect the actual 2.75 shard sizes, not the BF16 ones).

One further intentional patch on top of the upstream files:

• chat_template.jinja — added or message.role == "developer" in three places so that the template also accepts role=developer as an alias for role=system. The upstream template only recognises system/user/assistant/tool; the developer role is used by some OpenAI-style APIs for the higher-priority instruction.

Quick start (TabbyAPI + exllamav3)

Full tabby-config-2.75.yml used to serve this model on a 24 GB GPU (RTX 4090 verified):

network:
  host: 0.0.0.0
  port: 5000
  disable_auth: true       # only safe behind localhost / a trusted network

model:
  model_dir: /app/models
  model_name: Qwen3.8-27B-ABLITERATED-EXL3-2.75bpw

  # --- KV cache (24 GB budget) ---
  cache_size: 65536        # max tokens of KV cache
  cache_mode: Q8           # 8-bit cache; halves VRAM vs FP16, < 10% speed cost
  max_seq_len: 65536       # cap request length to the cache

  # --- EXL3-specific (optional) ---
  chunk_size: 2048         # prompt ingestion chunk size
  output_chunking: true    # allocate KV cache in chunks as needed (saves VRAM)
  inline_model_loading: true   # allow model swap by name in request body

  # --- multi-GPU / single-GPU split ---
  gpu_split_auto: true
  autosplit_reserve: [96]  # MiB reserved for activations on the single GPU

  # --- chat template / parsing ---
  prompt_template:         # use the model's embedded template
  template_vars_default:
    enable_thinking: true  # reasoning on by default
  reasoning: true          # required for tool call parsing; splits reasoning_content / content
  vision: true             # model is multimodal (Qwen3.5 vision tower)

  # --- tool / function calling parser ---
  tool_format: qwen3_coder
  tool_calls_in_reasoning: true

  # --- model-name aliasing ---
  use_dummy_models: true
  dummy_model_names: ["qwen3.8-27b"]   # short alias exposed via /v1/models

draft_model:
  draft_mode: disabled     # no speculative decoding

sampling:
  override_preset: safe_defaults   # provides sampling fallbacks for OAI clients that don't send params

logging:
  log_prompt: false
  log_generation_params: false
  log_requests: false
  log_chat_completion_requests: false

memory:
  cuda_malloc_async: true  # torch allocator; recommended for EXL3

developer:
  unsafe_launch: false

Note: the ghcr.io/theroyallab/tabbyapi image lacks python3.12-dev, which Triton needs for JIT compilation of cuda_utils.c. The entrypoint is overridden to install the dev headers and start the server in one shot:

docker run --rm --gpus all -p 10404:5000 \
  -v './Qwen3.8-27B-ABLITERATED-EXL3-2.75bpw:/app/models/Qwen3.8-27B-ABLITERATED-EXL3-2.75bpw:ro' \
  -v './tabby-config-2.75.yml:/app/config.yml:ro' \
  --entrypoint bash ghcr.io/theroyallab/tabbyapi:latest \
  -c "apt-get update -qq && apt-get install -y --no-install-recommends python3.12-dev gcc && exec python3 main.py --host 0.0.0.0"

VRAM observed during load on RTX 4090 (24 GB): ~16.6 GB (16 977 MiB) with cache_size=65536, cache_mode=Q8, max_seq_len=65536.

Build provenance

• Quantizer: turboderp-org/exllamav3 (tag v1.4.2)
• Host: Windows 11, Python 3.14.7, torch 2.11.0+cu128, CUDA 12.8, triton-windows 3.7.1
• GPU: NVIDIA GeForce RTX 4090 (24 GB, sm 8.9), driver 610.88
• Build compute time: 54 min 36 s per-module sum (cumulative [NN.NN s] markers, 70 modules) — pure EXL3 quant phase on RTX 4090. The run crashed once during layers.24 and was resumed from checkpoint; end-to-end wall-clock was not cleanly tracked across the interruption.
• Modules quantized: 70 (1 embed_tokens + 64 transformer layers + 1 norm + 1 lm_head + 3 mtp: mtp, mtp.layers.0, mtp.norm). The vision tower is not touched by this conversion pass and is carried over unquantized (no vision/merger/projector entries appear in the quantization log).
• Calibration source: exllamav3 v1.4.2 standard_cal_data (6 files, ~5.6 MB)
• VRAM at inference (TabbyAPI, observed): ~16.6 GB (16 977 MiB) with cache_size: 65536, cache_mode: Q8, max_seq_len: 65536. Single user, no prefix cache, RTX 4090 24 GB.

License

Apache License 2.0. See LICENSE — inherited from the upstream Blackfrost repo, which traces to Alibaba Cloud (Qwen authors).