Downloads · 30 days
0
xThr45hx/Josiefied-Qwen3-4B-TRT-LLM
Josiefied-Qwen3-4B-TRT-LLM is a machine learning model from xThr45hx. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for tensorrt. The card lists the license as apache-2.0.
TensorRT-LLM checkpoints (buildable on any NVIDIA GPU) and a prebuilt INT4 engine (RTX 40-series) for Josiefied-Qwen3-4B-abliterated-v1 — the abliterated (uncensored) Qwen3-4B.
Downloads · 30 days
0
Access
Public
Updated Jul 1, 2026
Repo size
19.2 GB
Likes
0
Public
Click a slice to open those files.
.safetensors15.7 GB · 100%
From the Hugging Face model README
TensorRT-LLM checkpoints (buildable on any NVIDIA GPU) and a prebuilt INT4 engine (RTX 40-series) for Josiefied-Qwen3-4B-abliterated-v1 — the abliterated (uncensored) Qwen3-4B.
⚠️ This is an abliterated / uncensored model.
TensorRT-LLM engine builds are RAM/VRAM-hungry, and almost nobody shares the TRT-LLM checkpoint format — so people with limited hardware can neither build one nor find a prebuilt. This repo fills that gap:
rank0.safetensors, TRT-LLM format) → build your own engine for your GPU/SM in minutes, no OOM, no weight-conversion hell.| Folder | What | Quant | Runs on |
|---|---|---|---|
int4/ | checkpoint | INT4 weight-only | build for any SM |
int4-engine/ | prebuilt engine (16k ctx) | INT4 | RTX 40 / SM89 only |
fp8/ | checkpoint | FP8 | build for any SM (Ada+) |
gptq/ | checkpoint | INT4 GPTQ | build for any SM |
w4a8awq/ | checkpoint | W4A8 AWQ | build for any SM |
On an RTX 4060 (Ada / SM89), the INT4 W4A16 quant is the fastest:
At ~113 tok/s a 4B model streams a full terminal of text near-instantly — it was originally tested driving a GPU-accelerated terminal and filled the screen almost instantly. For raw speed, use int4. (The prebuilt int4 engine lives in the companion software repo, xThr45hx/TensorRT-LLM-Windows-RTX40.)
Use trtllm-build from TensorRT-LLM against the rank0.safetensors + config.json in a quant folder, targeting your GPU's SM. Example (INT4):
trtllm-build --checkpoint_dir int4 --output_dir my-engine --gemm_plugin auto
For the native Windows build tooling (no WSL), see xThr45hx/TensorRT-LLM-Windows-RTX40.
Apache-2.0 (matching Qwen3 + TensorRT-LLM upstream).