slim-onnx

slim-onnx — CPU/ONNX build of kompress-v2 for the slim token-saver

slim-int8 is a CPU-friendly, int8-quantized ONNX repackaging of chopratejas/kompress-v2-base, the extractive prompt-compression model (LLMLingua-2 family) used by headroom.

It exists so the slim CLI (a token-saver for coding assistants) can run the semantic-compression path on a laptop CPU with no GPU and no build step — the model auto-downloads from here on first use.

What it does

Extractive token pruning: the model scores every subword P(keep) ∈ [0,1]; the caller drops the low scorers. Output is a strict subsequence of the input — lossy but order-preserving and never hallucinated. Base architecture is ModernBERT-base + a LoRA adapter + a dual head (token classifier + span-smoothing 1-D CNN). See the base model card for training details (F1 0.918, must-keep recall 0.974 @ threshold 0.5).

Files

Quantized MatMul-only on purpose: quantizing the span-head Conv produced a ConvInteger op that stock onnxruntime-CPU can't execute. Keeping Conv in fp32 yields a model that loads everywhere while still shrinking the transformer.

Usage (onnxruntime, CPU)

import numpy as np, onnxruntime as ort
from transformers import AutoTokenizer

sess = ort.InferenceSession("onnx/slim-int8.onnx", providers=["CPUExecutionProvider"])
tok = AutoTokenizer.from_pretrained(".")   # this repo

words = "The authentication handler validates the token by calling verify().".split()
enc = tok(words, is_split_into_words=True, truncation=True, max_length=512,
          padding=True, return_tensors="np")
scores = sess.run(["final_scores"], {
    "input_ids": enc["input_ids"].astype(np.int64),
    "attention_mask": enc["attention_mask"].astype(np.int64),
})[0][0]                                    # P(keep) per subword

# aggregate subword -> word via enc.word_ids(), keep word if mean score >= 0.5

I/O contract: inputs input_ids, attention_mask (int64); output final_scores [batch, seq] float. Threshold is tunable — 0.5 keeps ~81%, 0.7 keeps ~70%.

Attribution & license

Derivative of chopratejas/kompress-v2-base (Apache-2.0); this repackaging is Apache-2.0. All model weights and training credit belong to the original authors. Built for the slim token-optimization CLI.