Qwen3-14B-Abliterated-FP8

Qwen3-14B-FineGrained-FP8 (Blackwell Optimized)

This repository contains a high-precision Fine-Grained FP8 quantization of huihui-ai/Qwen3-14B-Instruct-Abliterated.

The model has been specifically quantized using parameters optimized for next-generation hardware, particularly the NVIDIA Blackwell (RTX 50-series) architecture.

Model Highlights

• Architecture: Qwen3-14B
• Quantization: Fine-Grained FP8
• Optimization: Optimized for Blackwell Tensor Cores (weight_block_size=(128, 128))
• Abliterated: Based on the version by huihui-ai, where refusal mechanisms have been removed to provide more direct, unfiltered responses.

Technical Configuration

The quantization was performed using FineGrainedFP8Config with the following settings:

• Weight Block Size: 128x128. This specific block size is designed to align with the hardware throughput of RTX 5090 and other Blackwell-based GPUs, allowing for native execution with minimal overhead.
• Precision: Unlike standard per-tensor FP8, the fine-grained approach maintains significantly higher output quality by scaling weights in smaller blocks.

Hardware Requirements

• Optimal: NVIDIA RTX 50-series (Blackwell) for native hardware acceleration.
• Supported: NVIDIA RTX 40-series (Ada Lovelace), H100, and L40S.
• VRAM: Occupies approximately 16-17 GB of VRAM. A 24GB+ card is recommended for handling longer context windows and KV-cache.

Usage

You can load this model directly using the transformers library. Ensure you have the latest version of accelerate and transformers installed.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ikarius/Qwen3-14B-FineGrained-FP8"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    dtype="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompt = "Explain the advantages of FP8 quantization for LLMs."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Quantization Process

The model was quantized from the BF16 source using the following logic:

Loaded with dtype="auto" and device_map="auto".

Configured with FineGrainedFP8Config(weight_block_size=(128, 128)).

Weights were saved in the optimized FP8 format to allow for immediate loading without re-quantization.

How to use FP8 KV-cache

# --- FOR TRANSFORMERS 5.0 / BLACKWELL ---
    with torch.inference_mode(), torch.amp.autocast('cuda', dtype=torch.bfloat16):
        output_ids = model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            max_new_tokens=max_tokens,
            do_sample=True,
            temperature=temperature,
            top_p=top_p,
            top_k=top_k,
            repetition_penalty=1.10,
            eos_token_id=EOT_ID,
            pad_token_id=tokenizer.pad_token_id,
            bad_words_ids=final_bad_words_ids,
            
            # FOR FP8 CACHE
            cache_config={
                "cache_dtype": torch.float8_e4m3fn, 
            }
        )

Disclaimer

This is an abliterated model. It has fewer safety guardrails compared to the original Qwen3 release. Users are responsible for their own implementations of moderation layers and for using the model ethically and legally.

Credits

Original Model: Qwen Team

Abliteration: huihui-ai