trnews-splade-v1

trnews-splade-v1 — Turkish sparse retrieval with an inference-free query route

A learned sparse retriever for Turkish news text. Documents are encoded once, offline, by a SPLADE model. Queries are not encoded by a neural network at all: a query is tokenized, its token ids are deduplicated, and each one is looked up in a frozen 32,000-entry IDF table. Retrieval is the dot product of those two sparse vectors.

That asymmetry is the design. It is why a query costs 0.017 ms at p50 to encode here, and why the serving side needs neither Torch nor a GPU.

Every quantitative statement below is mapped to the exact artifact and field it came from in CLAIMS.md, with digests. Read that file before quoting anything from this one.

What this model is for, and what it is not

Use it when you want lexical-strength Turkish retrieval with near-zero query cost and a serving path that is a database rather than a model server, and when your documents can be encoded ahead of time.

Do not use it as a drop-in dense replacement. It was measured against that bar and missed it. Recall@10 sits 2.2523 pp below the dense model on the controlled test tier and 2.4024 pp below on the distractor tier. The predeclared allowance was 2 pp, so the criterion was missed by 0.2523 pp and 0.4024 pp respectively — those are the overshoots, not the gaps. The criterion was written before the run and was not relaxed afterwards. Fast queries do not buy it back.

Results

One scored run on held-out test tiers, frozen beforehand: checkpoint, pruning cap, query route and thresholds were all selected on validation and fixed before the test data was touched. Values are the unrounded artifact values displayed to six decimals.

Controlled tier — 2000 documents, 1998 queries:

Distractor tier — 22000 documents (the same queries, plus 20,000 unlabelled training abstracts), 1998 queries:

Paired grouped bootstrap, 2000 resamples, 1998 relevance groups per tier, nominal 95%. These are individual comparison intervals, not a simultaneous guarantee across the whole table.

The distractor MRR interval contains zero. That does not establish a difference, and it does not establish equivalence either. The recall gap is what decided the verdict, and it is real on both tiers.

The distractor tier has incomplete relevance judgments — the added abstracts were never judged, so some may be relevant while counting as wrong. It is a corpus-size stress test, not a judged production corpus.

Hybrid, reported separately because it reuses the test set

A reciprocal-rank fusion of this model with the dense companion (depth 100 per leg, rank constant 20, equal weights, frozen on validation before any test scoring) reaches MRR@10 0.916608 / Recall@10 0.970470 on the controlled tier and 0.821364 / 0.918418 on the distractor tier — better than either leg alone, and better than dense on MRR@10 with the interval excluding zero on the distractor tier (+0.0285, low bound +0.0200).

Two things not to skip. This is disclosed test-set reuse: the tiers had already been consumed by the standalone run above, so it is one reported follow-up, not an independent confirmation. And a hybrid pays dense latency: end-to-end p50 7.577 ms, p95 20.966 ms, against 0.626 ms p95 for sparse alone in the same run. Fusing reinstates the query encoder this design exists to avoid.

Latency, stated as what was measured

1998 validation documents, 200 queries, top-10, one machine. Measured separately from the quality run above. Backends differ by route: the dense rows ran on Apple MPS, the sparse query lookup on CPU — a like-for-like CPU comparison was not made.

The warm p95 query-encode ratio against dense is 3383.81 in this benchmark. That is an encode-only ratio, on one machine, across two different backends, and it is not an end-to-end headline: the end-to-end p50 ratio in the same run is 18.3x. The dense p50 to p95 spread is observed variance with no demonstrated cause. Cold start rests on 8 samples; dense cold start was never measured. These are ratios of measured quantiles, not guarantees.

Set it up

There is no PyPI package. The runtime ships as source inside this repository, so the model and the code that loads it are one pinned artifact.

Three routes are documented. They need different environments, and each ends with the verification command that matches it. Do not mix them in one virtual environment: the query suite asserts that Torch and friends are absent, so running it after installing the document extra correctly fails.

<full-commit-sha> below is a placeholder for a full-length commit hash from this repository's history. A branch name is not immutable, and a tag can be moved — resolve the tag to its commit and pin that, because the verification below binds to exact bytes.

Route A — queries only (no Torch, ever)

python3 -m venv .venv-query
. .venv-query/bin/activate

# huggingface_hub is the downloader. It is not a DECLARED dependency of
# trnews-splade, but `tokenizers` requires it, so the locks below pin it
# (1.31.0) and it is present either way; installing it first only makes
# this download step self-contained.
pip install "huggingface_hub>=0.25"

python - <<'EOF'
from huggingface_hub import snapshot_download
snapshot_download(
    "sinanalyuruk/trnews-splade-v1",
    revision="<full-commit-sha>",
    local_dir="trnews-splade-v1",          # materialized files, not a symlinked cache
    allow_patterns=["manifest.json", "query/*", "runtime/*", "examples/*",
                    "verify_runtime.py", "verify_release.py", "verification.json",
                    "README.md", "CLAIMS.md", "LICENSE", "NOTICE",
                    "release-inventory.json"],
)
EOF

# Install from a COPY of the runtime, so the downloaded tree stays byte-identical.
# A FRESH directory every time. Copying into one that already holds a
# `runtime` would nest the new source inside it and leave pip installing
# the older copy, so never reuse a fixed path such as /tmp/build.
BUILD="$(mktemp -d)/runtime"
cp -R trnews-splade-v1/runtime "$BUILD"
pip install -r trnews-splade-v1/runtime/locks/requirements-query-only.lock.txt \
            "$BUILD"

python trnews-splade-v1/verify_runtime.py --snapshot trnews-splade-v1 \
    --suite query --fixtures trnews-splade-v1/examples/example_texts_tr.json

That last command must print RESULT: PASS (15/15 checks). Two of those checks assert that torch, transformers, sentence-transformers and scipy are not importable here; that is the point of this route.

With no index yet, you can still encode queries:

python trnews-splade-v1/examples/search_sparse.py \
    --snapshot trnews-splade-v1 --query-only

Searching needs an index, and building one needs the document encoder — see Route B, then Putting the two together below. An index is a pair of files, <name>.npz and <name>.meta.json; this environment can search one but cannot create one, and with no index present the command above encodes queries and says so rather than failing.

Route B — encoding documents and building an index

A separate environment. The document extra adds Torch, Transformers, Sentence Transformers and SciPy, and the weights are a separate download — installing the extra does not fetch them.

python3 -m venv .venv-document
. .venv-document/bin/activate
pip install "huggingface_hub>=0.25"

python - <<'EOF'
from huggingface_hub import snapshot_download
snapshot_download(
    "sinanalyuruk/trnews-splade-v1",
    revision="<full-commit-sha>",
    local_dir="trnews-splade-v1",
    allow_patterns=["manifest.json", "query/*", "document/*", "runtime/*",
                    "examples/*", "verify_runtime.py", "verify_release.py",
                    "verification.json", "README.md", "CLAIMS.md",
                    "LICENSE", "NOTICE", "release-inventory.json"],
)
EOF

# A FRESH directory every time. Copying into one that already holds a
# `runtime` would nest the new source inside it and leave pip installing
# the older copy, so never reuse a fixed path such as /tmp/build.
BUILD="$(mktemp -d)/runtime"
cp -R trnews-splade-v1/runtime "$BUILD"
# Braces matter: in zsh, macOS's default shell, "$BUILD[document]" is read as
# an array subscript and expands to nothing. "${BUILD}[document]" is the path.
pip install -r trnews-splade-v1/runtime/locks/requirements-document.lock.txt \
            "${BUILD}[document]"

python trnews-splade-v1/verify_runtime.py --snapshot trnews-splade-v1 \
    --suite document --fixtures trnews-splade-v1/examples/example_texts_tr.json

# Build an index from the shipped synthetic fixtures, then search it.
# The index is written to ./example_index in the CURRENT directory —
# never inside the snapshot, which must stay exactly as downloaded.
python trnews-splade-v1/examples/search_sparse.py --snapshot trnews-splade-v1

The document suite prints RESULT: PASS (12/12 checks), including CPU parity of the packaged encoder against the reference SPLADE formula. The example then writes example_index.npz and example_index.meta.json beside you, and searches them. Point Route A at those two files.

Putting the two together

Copy the two index files to wherever the query-only environment runs, then search them there — no Torch anywhere in that process:

. .venv-query/bin/activate
python trnews-splade-v1/examples/search_sparse.py \
    --snapshot trnews-splade-v1 --index ./example_index --query-only

--query-only makes the example fail rather than import anything from the document extra, so this is also the check that your query environment stayed query-only.

Route C — serving from PostgreSQL

Prerequisites: an index from Route B, and a database with the pgvector extension. Its own environment again.

docker run --rm -d --name trnews-pg -p 55432:5432 \
  -e POSTGRES_USER=trnews -e POSTGRES_PASSWORD=trnews \
  -e POSTGRES_DB=trnews_sparse pgvector/pgvector:pg17
export TRNEWS_PG_DSN=postgresql://trnews:[email protected]:55432/trnews_sparse

python3 -m venv .venv-postgres
. .venv-postgres/bin/activate
# A FRESH directory every time. Copying into one that already holds a
# `runtime` would nest the new source inside it and leave pip installing
# the older copy, so never reuse a fixed path such as /tmp/build.
BUILD="$(mktemp -d)/runtime"
cp -R trnews-splade-v1/runtime "$BUILD"
pip install -r trnews-splade-v1/runtime/locks/requirements-postgres.lock.txt \
            "${BUILD}[postgres]"

python trnews-splade-v1/examples/search_postgres.py \
    --snapshot trnews-splade-v1 --index ./example_index

The example creates its own uniquely named table, loads the index into it, asserts that the database's scores match the local scorer within 1e-5, and drops only the table it created.

Two different verifications

verify_runtime.py verifies the model packages: manifest.json against the frozen release digest, then every file in query/ and document/ against it. That is the check a user runs, and the one the commands above call.

verify_release.py verifies the whole tree against release-inventory.json — every published file, byte for byte. It ships as well, and it asks which tree you have, because three different trees are all correct:

# a full download
python trnews-splade-v1/verify_release.py \
    --candidate trnews-splade-v1 --layout downloaded

# a query-only download, where document/ was never fetched
python trnews-splade-v1/verify_release.py --candidate trnews-splade-v1 \
    --layout selective --expect-absent document/

--layout downloaded exempts exactly one prefix and nothing else: snapshot_download(local_dir=...) writes its own bookkeeping under .cache/huggingface/ — a CACHEDIR.TAG, a .gitignore, and a .metadata plus a .lock file for each downloaded file — so a downloaded tree is the published files plus that directory. Every exempted path is printed rather than passed over silently. --layout selective makes you declare what is missing, so an interrupted download cannot present itself as a deliberate subset.

Installing in place writes runtime/build/ and runtime/src/*.egg-info/ into the tree: harmless for verify_runtime.py, but those files do make a whole-tree inventory check fail, and no layout exempts them. That is why every recipe above installs from a copy.

Using the API

The fragments below are illustrative, not standalone: each assumes you have run the route above it. Complete runnable versions are in examples/.

Encoding a query and searching an index (Route A)

# needs: Route A installed, and an index built in Route B
from trnews_splade import QueryEncoder, SparseIndex

enc = QueryEncoder("trnews-splade-v1/query",
                   manifest="trnews-splade-v1/manifest.json")

terms = enc.encode_one("metro hattı çalışmaları")   # {token_id: weight}

index = SparseIndex.load("./example_index")
index.check_compatible(enc)        # refuses an index built against other artifacts
for hit in index.search(terms, k=10):
    print(hit["score"], hit["doc_id"])

manifest= is required. Without it the query package's bytes are never checked, and an altered weight table would score silently.

Encoding your own documents (Route B)

# needs: Route B installed; `texts` is your own list of strings
from trnews_splade import QueryEncoder, SparseIndex
from trnews_splade.document import load_document_encoder, encode_documents

enc = QueryEncoder("trnews-splade-v1/query",
                   manifest="trnews-splade-v1/manifest.json")
texts = ["Metro hattında gece çalışması sürüyor", "Güneş enerjisi santrali açıldı"]

model, info = load_document_encoder("trnews-splade-v1", device="cpu")
vectors = encode_documents(model, texts, encoder=enc)      # pruning applied

index = SparseIndex.from_document_vectors(
    [f"doc-{i}" for i in range(len(texts))], texts, vectors, enc)
index.save("./my_index")                                   # never inside the snapshot

Pass the snapshot directory to the loader, not a bare document/ path: verification is anchored on manifest.json, and there is no unverified way in.

encode_documents always applies the deployed pruning rule. The router emits a weight for all 32,000 coordinates; what was evaluated and indexed is that output cut to the top 256 positive weights. Encoding without that step produces vectors that silently do not match this model, so the helper refuses any other cap.

PostgreSQL (Route C)

# needs: Route C installed, and a table already loaded — see examples/search_postgres.py
from trnews_splade import postgres as pg

sections = pg.load_sections()
conn = pg.connect("postgresql://...")
hits = pg.search_exact(conn, sections, "v1", terms, dim=32000, k=10)

The representation contract

Break any of these and the scores stop being this model's scores.

On the controlled test tier, documents retained 241.2235 terms on average and 256 at most, with 0 empty vectors; before pruning the same documents reached 1894 terms. The cap is a required part of the representation, not an optimization. 0.45% of those documents reached the 128-token limit.

pgvector stores sparsevec coordinates one-based. The packaged adapter applies that shift exactly once, inside sparse_vector(). Do not shift again.

Serving with PostgreSQL

The public promise of this release is exact search, and it is measured on the validation tiers: across 1999 controlled-tier queries and 1999 distractor-tier queries, the database's top-10 lists matched the local reference scorer with 0 ranking mismatches on either tier. Scores are not bit-identical — float32 addition is not associative and the two engines accumulate in different orders — with a worst disagreement of 4.578e-05, or 3 float32 ulps.

Exact search runs under the schema's exactness guard. Once an HNSW index exists, the planner will otherwise answer an ORDER BY using the index's own approximate ordering and quietly return approximate results. The guard restores the caller's previous setting, not the server default.

An exact postings-table alternative was also measured on the validation controlled tier: 0 ranking mismatches, p50 0.732 ms against 4.232 ms for the sequential scan, from 481678 posting rows occupying 53305344 bytes of table and 23912448 bytes of index.

On approximate search, read the scope carefully. An HNSW index over 21998 documents and 1999 validation queries reached 0.9869 mean top-10 overlap with exact search at ef_search=400, giving up -0.0015 Recall@10 against exact sparse retrieval, at p50 4.781 ms. Three qualifications: the minimum overlap over those queries was 0.7, so individual queries do lose results; that measurement is on the validation tier against an exact-sparse reference, and cannot be added to the 2.4024 pp this model gives up against dense on the test tier; and the packaged search_ann helper does not carry the plan verification and search breadth control the measured command-line tool used. Treat ANN here as a documented experiment, not as a supported serving contract of this package.

Two pgvector limits, both far from this model's 256-term documents: sparsevec cannot have more than 16000 non-zero elements for storage, and sparsevec cannot have more than 1000 non-zero elements for hnsw index on insert into an indexed table.

trust_remote_code, and what is checked before it

The exported router names one class outside the sentence_transformers namespace, so sentence-transformers 6.x will only build it when the caller passes trust_remote_code=True. That flag is a loading option, not a sandbox. Once it is set, code the model's own configuration selects can run, and a check performed afterwards cannot undo what already ran.

So load_document_encoder proves identity first, in this order:

1. the path is a local directory — a repository id is refused outright, so nothing is downloaded mid-verification;
2. manifest.json equals the frozen release digest, every listed file matches its hash, and the package directories contain nothing unlisted: no importable Python, no compiled extension, no symlink pointing out of the tree;
3. the module graph is exactly the expected one, with the packaged SpecialTokenMask as its only non-library class;
4. a foreign finetune package elsewhere on your path is detected — without importing it or its parent — and refused unless you opt in;
5. only then is the encoder constructed, with local_files_only=True, and trust scoped to the directory just verified.

There is no document_dir= parameter and no caller-supplied "already verified" descriptor. Verification is anchored on the manifest, so a document package without it cannot be shown to be this release.

SparseEncoder("sinanalyuruk/trnews-splade-v1") is not supported. That call is untested against this layout and would skip all of the above; this card declares no library_name for the same reason. Use the documented loader.

How it was trained and evaluated

Data. TR-News title → abstract pairs. Cleaning strips ends and collapses whitespace runs, keeping case and accents; a row is dropped only if either side is empty. Document ids are sha1(cleaned utf-8 text)[:16], so identical text is one document.

Leakage controls, and their boundary. Deduplication is exact: equality of the cleaned (query, document) pair removed 3 training rows, and exact cleaned-surface overlap with the evaluation splits removed 192 more from training and 1 validation row that conflicted with test. 99805 training pairs remain. Near-duplicates were measured and left in place: 131 pairs score Jaccard ≥ 0.9 over 5-character n-grams, including 4 train/val and 2 test/train pairs. That is a similarity flag, not an event-identity judgment, and no semantic or near-duplicate deduplication was performed. Relevance sets and document ids are stable across systems.

Training. One pass over those pairs: 6238 steps, batch 16, seed 42, FP32 PyTorch on Apple MPS. Document learning rate 2e-05, ranking scale 0.1. The document FLOPS regularizer ramps quadratically to 0.0003 over the first third of the steps — a separate schedule from learning-rate warmup. Query weights are frozen throughout and carry no regularizer.

Selection, all on validation, before any test access. Checkpoint step 6238 and the top-256 cap were chosen there, as was the frozen-IDF query route over a learned-query alternative. The test tiers were untouched until the single scored run reported above.

Baselines. BM25 uses the same WordPiece tokenizer with k1=1.2, b=0.75; the dense system is cosine similarity over sinanalyuruk/trmteb-trnews-v1. All three see identical queries, corpus ids and relevance judgments within a tier. Documents get 128 total BERT input tokens, and the lexical baseline keeps the corresponding 126 content-token budget; the dense model uses its own published context limit. Sparse hits require a positive score, while dense cosine can rank negative scores.

Where it ran. macOS 26.5.2 on arm64 Apple Silicon, 36 GiB of RAM, Python 3.12.13, torch 2.12.1, transformers 5.12.1, sentence-transformers 5.6.0. Training and dense inference used MPS; the query lookup used CPU. The release was tested on a different, current stack — see CLAIMS.md §9 — and those two sets of versions are deliberately not the same claim.

Limitations

• Domain. Turkish news headline-to-abstract retrieval, evaluated in-domain. No clustering, deduplication, event bundling, question answering or out-of-domain claim follows from this benchmark.
• Case folding is not Turkish-aware, and diacritics are lost inside words. The tokenizer is uncased, so I and İ both fold to i — MeTrO HaTtI becomes metro hatt ##i while metro hattı stays metro hattı. Lowercase i and ı are not confused: they are distinct ids (51 and 152). What does collide is diacritics within a word: is and iş are both token 2578. A training diagnostic separated case-only collisions from 2199 accent-or-other collision groups, 1.64% of the 134164 distinct token-id tuples. Both routes share this tokenizer, so route parity cannot detect what both routes lost.
• Length. Queries are cut at 64 tokens and documents at 128. Chunking longer documents was never evaluated; if you chunk, that is your experiment.
• Rare tokens. Only 4 validation queries contained pieces absent from the training abstracts. That slice supports no general conclusion about unseen tokens.
• Scale. The largest measured corpus is 22000 documents. Posting-list behaviour at 22,000 documents in PostgreSQL was not measured — the run failed on a full disk and the gap is recorded rather than estimated.
• One machine, two backends. Every latency figure is from one macOS arm64 host over a local connection, on an untuned PostgreSQL, with dense inference on MPS and the sparse query route on CPU. Nothing here is a tuned-deployment number, a concurrency measurement, or a browser or service benchmark.
• Dense companion. The hybrid and the dense comparisons need sinanalyuruk/trmteb-trnews-v1. Sparse-only use does not.

Provenance and identity

The numeric IDF digest covers the 32,000 values packed little-endian float32 in token-id order. It is not the SHA-256 of query_weights.json; the two identify different byte representations, and conflating them has already caused one bug in this project.

Ancestry of the document weights, most recent first — the full record, including which upstream terms are undeclared, is in NOTICE:

trnews-splade-v1                      this release
  <- sinanalyuruk/trmteb-trnews-v1    dense fine-tune (card declares MIT)
  <- trmteb/turkish-embedding-model   *** no licence declared ***
  <- dbmdz/bert-base-turkish-uncased  BERTurk (card declares MIT)

TR-News did not pretrain BERTurk. It enters at three later points: the dense fine-tune, the sparse training run, and the frozen IDF computation.

Licence and attribution

Released under the MIT licence with attribution (LICENSE), under the project owner's standing policy of following the base model's licence and citations where no different terms are declared upstream.

That is a downstream decision, and it is not evidence of an upstream grant. Two upstream sources declare no licence at all — trmteb/turkish-embedding-model, from which the document weights descend, and batubayk/TR-News, the training data. NOTICE records those facts plainly rather than presenting policy as permission. Confirm terms yourself before commercial use or redistribution.

No TR-News article text, corpus or corpus index is distributed here — the release carries trained weights and 32,000 derived IDF values.

Cite the dataset as its authors ask:

@article{10.1007/s10579-021-09568-y,
    year = {2022},
    title = {{Abstractive text summarization and new large-scale datasets for agglutinative languages Turkish and Hungarian}},
    author = {Baykara, Batuhan and Güngör, Tunga},
    journal = {Language Resources and Evaluation},
    issn = {1574-020X},
    doi = {10.1007/s10579-021-09568-y},
    pages = {1--35}
}