Qwythos-9B-Claude-Mythos-5-1M-AWQ

Qwythos-9B-Claude-Mythos-5-1M-AWQ

Base model: empero-ai/Qwythos-9B-Claude-Mythos-5-1M

This repository contains an AWQ INT4 / W4A16 quantized export of Qwythos-9B-Claude-Mythos-5-1M.

The model was quantized on an NVIDIA GeForce RTX 4090 with a self-implemented AWQ exporter that writes a vLLM / AutoAWQ-style weight layout:

• .qweight
• .qzeros
• .scales
• sharded model-xxxxx-of-xxxxx.safetensors
• model.safetensors.index.json

The quantization path is data-free: no external calibration dataset was used. The exporter estimates per-layer input importance from the original weights, then searches weighted group clipping / minmax scales to reduce reconstruction error.

This release is primarily prepared for vLLM with AWQ GEMM weight-only inference. It does not claim compatibility with every runtime.

Quantization Policy

Qwythos is based on Qwen3.5, whose hybrid attention stack contains linear-attention and attention modules that should remain in native precision for vLLM loading. For this reason, this export intentionally quantizes only the text MLP projection weights.

Quantization config used by the exporter:

w_bit: 4
q_group_size: 128
zero_point: true
awq_grid_size: 8
awq_samples: 8
calibration_mode: data_free
max_shard_size: 3GB

The exporter keeps the Qwen3.5 linear-attention tensors in their official split layout rather than fusing them, which is important for vLLM compatibility.

Dependencies

Recommended vLLM serving stack:

pip install --upgrade vllm transformers accelerate safetensors

Qwen3.5 support in serving frameworks is still relatively new. If your local vLLM build cannot load qwen3_5, upgrade vLLM and Transformers first.

vLLM Startup Command

Start with a conservative context length on a single RTX 4090. The source model advertises a 1M context window, but KV cache memory, not AWQ weight size, becomes the limiting factor at long context.

vllm serve JunHowie/Qwythos-9B-Claude-Mythos-5-1M-AWQ \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name Qwythos-9B-AWQ \
  --trust-remote-code \
  --quantization awq \
  --dtype float16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.85

If memory headroom is sufficient, increase --max-model-len gradually.

Example Request

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwythos-9B-AWQ",
    "messages": [
      {
        "role": "user",
        "content": "Explain how AWQ quantization protects salient channels, and why Qwen3.5 linear-attention tensors are kept native."
      }
    ],
    "temperature": 0.6,
    "top_p": 0.95,
    "max_tokens": 4096
  }'

Model Download

from huggingface_hub import snapshot_download

snapshot_download(
    "JunHowie/Qwythos-9B-Claude-Mythos-5-1M-AWQ",
    cache_dir="your_local_path",
)

Notes

• This is a quantized derivative of Qwythos-9B-Claude-Mythos-5-1M, not a new fine-tune.
• The original model is a reasoning model and may emit <think>...</think> blocks before the final answer.
• Recommended sampling follows the base model card: temperature=0.6, top_p=0.95, top_k=20, repetition_penalty=1.05.
• The source model advertises a 1,048,576-token context window with YaRN rope scaling. For this AWQ release, start smaller on consumer GPUs and raise context length after memory testing.
• No benchmark numbers are changed or claimed by this AWQ export. Use the original model card for upstream capability references.

Quantization Log

2026-06-30
1. Quantized empero-ai/Qwythos-9B-Claude-Mythos-5-1M on RTX 4090.
2. Exported AWQ GEMM W4A16 weights with group size 128 and zero point.
3. Used data-free AWQ search; no calibration dataset was involved.
4. Kept Qwen3.5 attention, linear-attention, embeddings, norms, lm_head, visual modules, and first text layer in native precision for vLLM compatibility.

Original Model Overview

Qwythos-9B is developed by Empero. It is a full-parameter reasoning model built on top of a deeply uncensored Qwen3.5-9B base and post-trained on over 500 million tokens of Claude Mythos and Claude Fable traces, with chain-of-thought generated by Empero AI's internal tool rethink.

Key upstream features:

• 1,048,576-token context via YaRN rope scaling enabled by default.
• Native function calling following Qwen3.5's tool-use format.
• Reasoning-oriented behavior with <think> blocks.
• Strong gains over the base Qwen3.5-9B in the original evaluation setup, especially on MMLU and GSM8K.
• Domain emphasis on cybersecurity, biomedical, quantitative reasoning, coding, and agentic workflows.

Original evaluation headline from the base model card:

These numbers belong to the original FP/BF16 model card and are included only as upstream reference.

Base Model Details

• Upstream model: empero-ai/Qwythos-9B-Claude-Mythos-5-1M
• Underlying base: Qwen/Qwen3.5-9B
• Fine-tune type: full-parameter supervised fine-tune
• Context length: 1,048,576 tokens with YaRN rope scaling
• License: Apache-2.0

Limitations

• This AWQ export has not been presented as a full formal benchmark release.
• Very long context still requires substantial KV-cache memory.
• For exact identifiers, safety-critical medical details, live facts, or security-sensitive workflows, pair the model with retrieval/tools and verify outputs.
• Qwythos is intentionally uncensored. Add application-level policy and review layers for public-facing deployments.

Acknowledgements

• Original model: Empero Qwythos-9B-Claude-Mythos-5-1M
• Underlying base: Qwen/Qwen3.5-9B
• Quantized release: JunHowie/Qwythos-9B-Claude-Mythos-5-1M-AWQ