GKD_GGUF

github: https://github.com/Asher-1/General-Keypoint-Detection-GGML

GKDT-L GGUF Model Card

Model

• Architecture: GKDT-L = DINOv3 ViT-L vision tower (D=1024, 24 blocks, 16 heads, 4 storage tokens + cls, RoPE with checkpoint periods, LayerScale 1e-5, LinearKMaskedBias qkv) + dinotxt text tower (D=1280, 24 causal blocks, 20 heads) + TextHead linear projection (1280→2048) + text adaptation net (1 CLIP-style block, QuickGELU, 2048↔1280) + KG transformer (2 blocks, SA+CA, d_ff=1024, mask token) + parameter-free detection head (bilinear 4× upsample, L2 kernel norm, 1×1 kernel "conv", per-row mask fusion over prompts padded to 80 rows).
• Source checkpoint: models/pytorch/gkd_fullset.best (GKDT-L for real-world testing, ECCV 2026 release, ~6.34 GB).
• Input: RGB image resized so the longer side is 384 px and center-padded to 384×384 with the ImageNet mean color (124, 116, 104); ImageNet normalization. Preprocessing matches the official mytransforms pipeline exactly.
• Prompts: text keypoint names (CLIP BPE tokenized), 1-shot visual keypoints (soft-fiber Gaussian pooling, σ=14), or both (fused per row).
• Output: per-prompt heatmap peak over a 96×96 grid → keypoints in the ROI's normalized -1..1 space + confidence score.

Download

All five runtime files are published at https://huggingface.co/Asher-1/GKD_GGUF — no conversion step is needed unless you want a dtype we do not ship:

# everything into the engine's model directory
huggingface-cli download Asher-1/GKD_GGUF --local-dir cpp_ggml/models/gguf

# or just the recommended file
huggingface-cli download Asher-1/GKD_GGUF gkd_fullset-q4_K.gguf --local-dir cpp_ggml/models/gguf

# plain HTTPS also works (no HF client required)
wget https://huggingface.co/Asher-1/GKD_GGUF/resolve/main/gkd_fullset-q4_K.gguf \
     -O cpp_ggml/models/gguf/gkd_fullset-q4_K.gguf

Files — per-model details

All five run on all three backends (cpu / cuda / vulkan). Only 2-D weight matrices are quantized; vectors (biases, norms, cls/storage and RoPE tensors) always stay F32, and tensors whose fastest dimension cannot satisfy the block size fall back to F32 automatically (ne0 % 32 for q8_0/q4_0, ne0 % 256 for q4_K). Shared accuracy facts: keypoint coordinates on the official examples match stock PyTorch to ≤ 0.0048 px for every dtype; the dtypes differ in score amplitude and — for q4 — in argmax stability on near-flat heatmaps (see Verified accuracy below).

gkd_fullset-f32.gguf — fp32 reference (3.39 GiB)

• Every tensor in full fp32; the bit-exact baseline the other four are validated against (parity workflow in ../scripts/dump_taps.py).
• Accuracy: score diff ≤ 0.0013 vs PyTorch; all-argmax reproducible.
• Latency (text mode): cuda 52.8 ms · vulkan 51.5 ms · cpu 1555 ms — the CPU number is unusually strong because GGML_LLAMAFILE=ON enables the tinyBLAS fp32 GEMM.
• Use when: debugging parity, quantifying quantization cost, or when the argmax of every heatmap must be bit-reproducible.

gkd_fullset-f16.gguf — fp16 weights, near-lossless (1.70 GiB)

• 2-D weight matrices in F16; all vectors (biases, norms, tokens) stay F32 (the CUDA elementwise kernels reject mixed-precision broadcasts).
• Accuracy: score diff ≤ 0.0012; coordinates identical to fp32 — no argmax drift anywhere in the official sweep.
• Latency (text): cuda 41.4 ms · vulkan 48.9 ms (best Vulkan) · cpu 1734 ms (x86 has no native fp16 GEMM, so CPU is slightly slower than f32).
• Use when: you want near-lossless quality at half the fp32 size, or the best Vulkan latency.

gkd_fullset-q8_0.gguf — 8-bit compact, near-lossless (905 MiB)

• Q8_0 blocks (32 values, f16 scale + int8 payload) for 2-D matrices with ne0 % 32 == 0; everything else F32.
• Accuracy: score diff ≤ 0.0021; coordinates identical to fp32 in the official sweep (confident-scene mean ≤ 0.5 px, max ≤ 1.3 px).
• Latency (text): cuda 41.3 ms · vulkan 51.7 ms · cpu 1671 ms.
• Use when: you want fp32-grade behavior at ~3.7× less disk/memory than fp32 — the safe compact choice.

gkd_fullset-q4_0.gguf — simplest 4-bit (483 MiB)

• Q4_0 blocks (32 values, one f16 scale, d = signed_max/-8), 4.5 bits per weight. Simple encoding, but a single outlier in a block degrades the other 31 values.
• Accuracy: score diff ≤ 0.026; on the official demos keypoints match PyTorch to ≤ 0.0004 px mean, but on near-flat heatmaps (scenes where PyTorch itself scores < 0.1) the argmax can shift by tens of pixels.
• Latency (text): cuda 40.7 ms (fastest overall, 5.7× vs PyTorch) · vulkan 52.5 ms · cpu 1882 ms (slower than fp32 — the simple dequant path loses to tinyBLAS fp32 on x86).
• Use when: maximum CUDA throughput matters more than last-bit score fidelity.

gkd_fullset-q4_K.gguf — super-block 4-bit, recommended (483 MiB)

• Q4_K super-blocks (256 values, 8 sub-blocks of 32 with independent 6-bit scale+min chosen by an error-minimizing search), 4.5 bits per weight — same size as Q4_0, materially lower quantization error.
• Accuracy: score diff ≤ 0.017 (best of the two q4 variants); official demo keypoints ≤ 0.0004 px mean / 2.85 px worst (identical to fp32). Same near-flat-heatmap argmax caveat as q4_0, but milder.
• Latency (text): cuda 41.2 ms (5.6×) · vulkan 52.4 ms · cpu 1211 ms — the fastest CPU config of the whole matrix (Goldmann-style Q4_K kernels beat even the fp32 tinyBLAS path).
• Use when: default recommendation — 7× smaller than fp32, best accuracy-per-bit, fastest CPU inference.

Runtime & ggml version

• Built and verified against ggml v0.21.0 (upstream commit 8599e0ea), pinned as a git submodule at ../third_party/ggml. The submodule stays pristine: every deviation lives in ../patches/ggml/*.patch and is applied automatically — and idempotently — at cmake configure time (forward-check → apply, reverse-check → skip, conflict → abort).
• Verified toolchains: CUDA 12.6 (sm_89, RTX 4090), Vulkan SDK 1.4.350.0, CMake ≥ 3.14 + C++17 for the CPU build.
• Runtime dependencies: none beyond the ggml backends — no Python, no PyTorch, no CLIP/dinov3 repo at inference time; the CLIP BPE vocab is embedded in the GGUF.

Quantization formats

Only 2-D weight matrices are quantized; vectors (biases, norms, cls/storage and RoPE tensors) always stay F32, and tensors whose fastest dimension cannot satisfy the block size fall back to F32 automatically (ne0 % 32 for q8_0/q4_0, ne0 % 256 for q4_K).

Q4_0 — 32 values per 18-byte block, a single f16 scale (d = signed_max / -8, value = (q-8)·d). Simple, but one outlier in a block degrades the other 31 values.

Q4_K — 256-value super-blocks (144 bytes), each split into 8 sub-blocks of 32 with independent 6-bit scale+min; (scale, min) are chosen by an error-minimizing search rather than by fitting the extreme value: value = d·sc·(q-8) − dmin·m. Same 4.5 bits/value as Q4_0, materially less reconstruction error — the recommended runtime dtype.

Both writers in ../scripts/convert_gkd_to_gguf.py are numpy ports of ggml's reference quantizers and were cross-verified against ggml_quantize_chunk (Q4_0 byte-identical; Q4_K identical modulo the reference's own near-tie float ordering).

Conversion (only if not downloading)

The pre-built files above come from exactly this command, so converting yourself is only needed for a custom dtype/checkpoint:

python scripts/convert_gkd_to_gguf.py --dtype f32   # also: f16, q8_0, q4_0, q4_K
# input: models/pytorch/gkd_fullset.best -> models/gguf/gkd_fullset-<dtype>.gguf

The converter folds bias_mask, exports the RoPE periods, stores torch Linear weights raw (ggml ne = reversed shape), transposes anet.proj (the official code uses x @ proj), splits the KG cross-attention in_proj into q/k/v tensors, and quantizes only 2-D matrices that satisfy the block-size constraint.

Verified accuracy (vs stock PyTorch, 2007_007524.jpg)

Keypoint coordinates agree to ≤ 0.0048 px (max over the three prompt modes, original-image pixels) for every config of the full backend × precision matrix — quantization changes score amplitude, never the argmax location. Mean absolute score difference vs the stock PyTorch reference (full detail in ../benchmarks/accuracy.json):

All-image sweep + official demo commands

../benchmarks/accuracy_all_images.json extends the verification to all 15 official test images (text mode, per-image prompt sets) and to the three official demo commands of test_real_world/scripts/eval_single_obj_gkd.sh (multimodal whole-image, text with bbox ROI [33,38,241,310], cross-image visual support from alpaca_150.jpg):

• official demo commands: all 15 configs reproduce PyTorch to ≤ 0.0004 px mean (multimodal, visual) and 0.29 px mean / 2.85 px worst keypoint (bbox; the same 2.85 px appears in fp32 — it is the 96×96 grid's 2 px cell size on a near-tie peak, not a precision artifact); score diff ≤ 0.031.
• all 15 images: f32/f16/q8_0 match PyTorch point-for-point on the 9 confidently-localized images (mean ≤ 0.4 px, max ≤ 1.3 px). The q4 dtypes keep most keypoints but can shift the argmax by tens of pixels on near-flat heatmaps (scenes where PyTorch itself scores < 0.1) — use q4 for confident detection workloads, f16/q8_0 when every argmax must be bit-reproducible.
• visuals: accuracy_by_image.png (config × image error heatmap), parity_official_examples.png (keypoint overlay grid, PyTorch vs three q4_K backends).

End-to-end latency / speedup over the 15 official images (reference: stock PyTorch on CUDA, RTX 4090) — generated table, see ../benchmarks/speedup_table.md and the matrices ../benchmarks/latency_matrix.png / ../benchmarks/speedup_matrix.png:

| mode | cpu-f32 | cpu-f16 | cpu-q8_0 | cpu-q4_0 | cpu-q4_K | cuda-f32 | cuda-f16 | cuda-q8_0 | cuda-q4_0 | cuda-q4_K | vulkan-f32 | vulkan-f16 | vulkan-q8_0 | vulkan-q4_0 | vulkan-q4_K | pytorch-cuda (ref) | | text | 1555.2 (0.1x) | 1733.6 (0.1x) | 1670.6 (0.1x) | 1882.3 (0.1x) | 1211.2 (0.2x) | 52.8 (4.4x) | 41.4 (5.6x) | 41.3 (5.6x) | 40.7 (5.7x) | 41.2 (5.6x) | 51.5 (4.5x) | 48.9 (4.7x) | 51.7 (4.5x) | 52.5 (4.4x) | 52.4 (4.4x) | 231.5 | | visual | 1625.5 (0.1x) | 1781.3 (0.1x) | 1759.3 (0.1x) | 1911.4 (0.1x) | 1342.0 (0.1x) | 49.8 (2.7x) | 42.4 (3.1x) | 40.7 (3.2x) | 41.7 (3.2x) | 40.0 (3.3x) | 47.3 (2.8x) | 44.6 (3.0x) | 50.2 (2.6x) | 50.0 (2.6x) | 47.4 (2.8x) | 131.9 | | multimodal | 2102.7 (0.1x) | 2342.4 (0.1x) | 2188.4 (0.1x) | 2441.4 (0.1x) | 1583.5 (0.2x) | 60.4 (5.0x) | 55.1 (5.4x) | 47.0 (6.4x) | 45.5 (6.6x) | 46.0 (6.5x) | 57.1 (5.3x) | 52.3 (5.7x) | 56.7 (5.3x) | 59.7 (5.0x) | 55.5 (5.4x) | 299.8 |

License

Same as the GKDT release: free for academic research and education, commercial use prohibited.