Downloads · 30 days
426
7% of all-time downloads
pottokao/Ornith-1.5-35B-A3B-abliterated-NVFP4-DFlash-GGUF
Ornith-1.5-35B-A3B-abliterated-NVFP4-DFlash-GGUF is a text generation model from pottokao. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as mit.
GGUF build of pottokao/Ornith-1.5-35B-A3B-abliterated-NVFP4-DFlash, for use with llama.cpp.
Downloads · 30 days
426
7% of all-time downloads
All-time downloads
5.9K
Public
Repo size
21.7 GB
Likes
5
Public
Click a slice to open those files.
.gguf21.7 GB · 100%
From the Hugging Face model README
GGUF build of pottokao/Ornith-1.5-35B-A3B-abliterated-NVFP4-DFlash,
for use with llama.cpp.
The 4-bit weights are repacked bit-exact from the NVFP4 checkpoint into GGML_TYPE_NVFP4 —
they are not dequantized and re-quantized, so there is no double-quantization penalty.
On Blackwell GPUs llama.cpp runs these through native FP4 tensor cores.
19.5 GB, plus a 772 MB DFlash draft model for speculative decoding. Runs on 2×16 GB consumer GPUs (tested on 2× RTX 5070 Ti).
⚠️ Text-only. No vision tower, no MTP head (the abliteration was done on a language-model-only export). Converted with
--no-mtp.
⚠️ Uncensored. Safety refusal behaviour has been deliberately removed. You are responsible for how you use it.
Classic refusal-direction ablation (orthogonalization), single direction:
| Step | Detail |
|---|---|
| Base | ornith-ai/Ornith-1.5-35B-A3B (BF16) |
| Probe layer | 24 — int(num_layers × 0.6), 40 layers total |
| Samples | 64 harmful + 64 harmless prompts (random.seed(0)), last-token hidden state |
| Direction | d = normalize(mean(harmful) − mean(harmless)) |
| Ablation | For every .o_proj and .down_proj: W ← W − outer(d, dᵀW) |
Tooling derived from
remove-refusals-with-transformers.
BF16 weights: pottokao/Ornith-1.5-35B-A3B-abliterated.
NVIDIA TensorRT Model Optimizer 0.45.0, per-layer recipe matched exactly to the official
ornith-ai/Ornith-1.5-35B-A3B-NVFP4
(verified tensor-by-tensor: weight_scale_2 30841, input_scale 130, 291 quantized layers, 0 diff
in the language model). Calibration: 64 × 512 tokens from abisee/cnn_dailymail.
| Module | HF checkpoint | → GGUF |
|---|---|---|
mlp.experts (256/layer), mlp.shared_expert, lm_head | NVFP4 W4A16, group 16 | GGML_TYPE_NVFP4 ×241, bit-exact |
linear_attn.{out,in_qkv,in_z}, self_attn.{q,k,v,o} | FP8 W8A8 | Q8_0 ×130 (GGML has no FP8 type) |
| embeddings | BF16 | BF16 ×61 |
Conversion (latest llama.cpp, which has a ModelOpt-aware branch):
python3 convert_hf_to_gguf.py /path/to/NVFP4-model \
--outfile Ornith-1.5-35B-A3B-abliterated-NVFP4.gguf --fp8-as-q8 --no-mtp
# DFlash draft (needs the target model for its tokenizer + dflash_config)
python3 convert_hf_to_gguf.py /path/to/NVFP4-model/dflash_draft \
--target-model-dir /path/to/NVFP4-model --outfile dflash-draft-Ornith15.gguf
--no-mtp is required for this checkpoint: the config still declares
mtp_num_hidden_layers: 1 but the MTP weights were stripped during abliteration, so without it the
converter writes block_count: 41 and loading fails with blk.40.attn_norm.weight not found.
Build with CUDA for your arch (120 = Blackwell / RTX 50-series). This matters: the native FP4
path is gated on blackwell_mma_available(), which checks the compiled arch.
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build --config Release -j --target llama-server
./build/bin/llama-server \
-m Ornith-1.5-35B-A3B-abliterated-NVFP4.gguf \
-md dflash-draft-Ornith15.gguf --spec-draft-n-max 8 \
-ngl 99 -ngld 99 --split-mode layer -c 8192 -fa on \
--host 0.0.0.0 --port 8080
llama.cpp auto-detects the draft type from the GGUF metadata:
common_specu: auto-detected speculative type 'draft-dflash' from the draft model metadata
- n_max=8, n_min=0, block_size=16, mask_token_id=248077, sample_from_anchor=true
layer (pipeline)--split-mode | What we observed on CUDA |
|---|---|
layer (pipeline) | fastest in our tests — what the numbers below use |
tensor (TP, EXPERIMENTAL) | loads and generates fine, but slower in every category we measured |
row | not supported — the CUDA backend has no split-buffer implementation, so it refuses to load |
Caveat: this is one configuration, not a verdict on tensor-parallel. We are not familiar with
llama.cpp's TP path and only tested 2× RTX 5070 Ti over PCIe (no NVLink) at -c 8192. TP works —
it is not broken — it just did not win here, plausibly because single-stream decode is
bandwidth-bound and the per-layer all-reduce costs more than the parallelism gains on this
interconnect. Building with -DGGML_CUDA_NCCL=ON recovered ~5 % over the internal AllReduce but
still did not beat layer. We also hit
llama_params_fit is not implemented for SPLIT_MODE_TENSOR, so memory has to be sized by hand.
If there is a better way to configure TP here, a newer build that changes this, or something we simply missed — corrections and suggestions are very welcome.
2× RTX 5070 Ti (16 GB, 250 W), DFlash K=8, -c 8192, -fa on.
| Category | layer tok/s | tensor (TP) tok/s | layer TTFT | tensor TTFT |
|---|---|---|---|---|
| math_reasoning | 238.5 | 198.5 | 133 ms | 183 ms |
| code (held-out) | 179.0 | 152.4 | 220 ms | 270 ms |
| summarization | 170.4 | 153.0 | 270 ms | 327 ms |
| rag | 159.1 | 140.2 | 354 ms | 414 ms |
rag and summarization carry 3.1–3.4 K-character prompts, so TTFT is a meaningful share of the
work — and in our runs TP was 17–21 % slower there too, i.e. we could not find a prefill-heavy case
where it came out ahead. Again: one configuration, and we may well be holding it wrong.
Across the full Spec-Bench run above (40 recorded generations):
| metric | value |
|---|---|
| overall acceptance | 34.1 % (7428 accepted / 21788 drafted) |
| mean accepted length | 3.88 (range 2.38 – 6.07) |
The spread matters more than the average. On ad-hoc single prompts we measured anywhere from 5.60 (step-by-step arithmetic) down to 3.07 (free-form Chinese prose) — structured output drafts very well, free-form prose drafts poorly. Always report the prompt mix alongside an acceptance number; the same model and settings can look 2× better or worse depending on what you feed it.
Same benchmark suite, same 8 prompts per category, concurrency 1:
| Category | vLLM + NVFP4 + DFlash K=8 | llama.cpp GGUF + DFlash 8 (layer) |
|---|---|---|
| math_reasoning | 429.6 | 238.5 |
| code | 295.6 | 179.0 |
| rag | 319.1 | 159.1 |
| summarization | 317.2 | 170.4 |
vLLM is 1.6–2.0× faster. This is worth stating plainly, because llama.cpp is not being held back by either of the two things you might suspect:
So the gap is in per-step execution of the MoE + hybrid-Mamba forward itself, not in the quantization format or the speculative decoding.
Pick this build if you want the llama.cpp runtime/ecosystem; pick the vLLM one for raw speed. Note the GGUF is 19.5 GB — the same size as the source — so it still does not fit on a single 16 GB card.
An AIME 2026 run (29/30) was done on the source NVFP4 checkpoint under vLLM; since this GGUF is a bit-exact repack of the same 4-bit weights, it is not re-reported here. See the NVFP4 repo for the numbers and the caveats — in short, it is a check that abliteration + quantization did not cause catastrophic degradation, not a capability claim, and there is no external baseline to compare against.
ornith-ai/Ornith-1.5-35B-A3B (BF16, MIT)
└── refusal-direction ablation (layer 24, o_proj + down_proj)
└── pottokao/Ornith-1.5-35B-A3B-abliterated (BF16, 65 GB)
└── modelopt 0.45.0 NVFP4, recipe matched to official
└── pottokao/…-NVFP4-DFlash (20 GB, vLLM)
└── convert_hf_to_gguf.py (bit-exact NVFP4 repack)
└── this repo (19.5 GB, llama.cpp)
DFlash draft: z-lab/Qwen3.6-35B-A3B-DFlash,
converted to GGUF unmodified; original weights and license belong to z-lab.