xVITA-Rerank-300M-zhTW-GGUF

xVITA-Rerank-300M-zhTW-GGUF

云碩科技（xCloudinfo）自研繁體中文重排序模型（cross-encoder reranker）的GGUF量化版， 供llama.cpp部署使用。RAG檢索第二段：嵌入模型撈出候選段落，本模型逐一細讀 （查詢, 段落）配對重新打分，把真正相關的段落拉到第一名。

搭配的嵌入模型：xVITA-Embed-300M-zhTW-GGUF。 完整精度權重（safetensors）為私有發行，如有微調或研究合作需求請聯繫云碩科技。

檔案

量化驗證（誠實揭露）

每一階量化都與原版safetensors用同一組測試（20題 × 6候選含干擾洗牌）比對過：

三版的正解排名全部保持；排序差異僅發生在分數趨近零的干擾項互換位置。 Q4_K_M的絕對分數不可靠（校準偏移大），只可用於取排序， 不要拿它的分數當過濾閾值；需要分數閾值請用Q8_0或f16。

使用方式（llama.cpp）

llama-server -m xVITA-Rerank-300M-zhTW-Q8_0.gguf \
  --rerank --pooling rank --ctx-size 2048 --port 8100

curl http://localhost:8100/v1/rerank -H "Content-Type: application/json" -d '{
  "query": "依據海商法，船舶的定義範圍為何？",
  "documents": [
    "勞動基準法第三十條規定勞工每日正常工作時間不得超過八小時。",
    "海商法第一條：本法稱船舶者，謂在海上航行，或在與海相通之水面或水中航行之船舶。"
  ],
  "top_n": 2
}'

回傳的 relevance_score 為原始logit；需要0～1機率請自行套sigmoid。

llama.cpp版本需求

本模型的底座mmBERT沿用Gemma-2的256k SentencePiece詞表。轉檔時已內嵌 SentencePiece詞表資訊，2026-08之後的llama.cpp版本可直接載入；若載入後 中文被逐字切分（一句話切出3倍以上的token數），代表你的llama.cpp版本過舊，請更新。

模型背景

上表三個領域的數字取自各自的驗證集；跨批次的絕對分數不可直接互相比較（每次評測會抽出自己的驗證集），同一次評測內的「檢索器vs加上重排序」對照才有意義。

泛化評測（換領域、換文體）

上表量的是訓練過的三個領域，只能證明模型在看過的東西上有效。我們另外量了它在完全沒有 訓練過的語料上還剩多少效果，方法與上表一致：同一顆重排序模型、同一顆檢索器 （xVITA-Embed-300M-zhTW），各500題，比較「只有檢索器」與「檢索器加上重排序」 在同一組查詢上的差異。

換領域可用。 完全沒有看過的政府公開文件，hit@1由61.8% 提升到87.2%， nDCG提升仍保有訓練領域的六成。

換文體會弱化。 口語逐字稿上的提升只剩三成六。有兩點必須說清楚，否則這張表會被誤讀：

1. 口語那一列hit@1的 −2.0個百分點不構成退步。500題、比例約0.2時， 單一比例的標準誤就有1.8個百分點，此差距不到一個標準差。該列真正站得住的是 hit@5（35.6% → 43.8%）與nDCG提升。
2. 該列的檢索器本身也由61.8% 掉到20.8%。重排序只能在檢索器取回的候選內重新排序， 候選裡沒有正解就無從補救；因此這個 +0.0799有多少來自重排序能力不足、 多少來自候選池本身不完整，以現有語料無法分離。

建議用途：書面文件的RAG，例如法規、公文、技術文件、專業領域知識庫。 若您的查詢或語料屬於會議逐字稿、客服對話等口語文體，請以本節的方法在自己的資料上 先量一次再決定，不要直接套用上表數字。

量測條件：epochs=0 純評測，不更新權重；檢索器與線上部署為同一顆並套用相同檢索前綴； 訓練與驗證以段落（而非配對）切分，避免同一段落的多個問題跨集洩漏； 每輪先跑「段落對自己」的哨兵檢查，兩組均為100%。

授權

MIT，與底座mmBERT-base一致。

關於云碩科技

云碩科技（xCloudinfo Corp. Limited）專注於地端私有化人工智慧解決方案。xVITA是云碩自研模型的品牌名稱。

其他模型請見 https://hf.co/xCloudinfo。

---

xVITA-Rerank-300M-zhTW-GGUF — English

The GGUF quantized release of xCloudinfo's in-house Traditional Chinese reranker (cross-encoder reranker), for deployment with llama.cpp. It is the second stage of RAG retrieval: the embedding model pulls out candidate passages, and this model reads each (query, passage) pair closely and re-scores it, moving the truly relevant passage up to first place.

Companion embedding model: xVITA-Embed-300M-zhTW-GGUF. The full-precision weights (safetensors) are a private release; for fine-tuning or research collaboration, please contact xCloudinfo.

Files

Quantization verification (honest disclosure)

Every quantization tier was compared against the original safetensors on the same test set (20 queries × 6 candidates, including shuffled distractors):

The correct-answer rank is preserved in all three versions; ranking differences occur only where distractors with scores close to zero swap places. The absolute scores of Q4_K_M are not reliable (large calibration drift); use it only to obtain a ranking, and do not use its scores as a filtering threshold. If you need a score threshold, use Q8_0 or f16.

Usage (llama.cpp)

llama-server -m xVITA-Rerank-300M-zhTW-Q8_0.gguf \
  --rerank --pooling rank --ctx-size 2048 --port 8100

curl http://localhost:8100/v1/rerank -H "Content-Type: application/json" -d '{
  "query": "依據海商法，船舶的定義範圍為何？",
  "documents": [
    "勞動基準法第三十條規定勞工每日正常工作時間不得超過八小時。",
    "海商法第一條：本法稱船舶者，謂在海上航行，或在與海相通之水面或水中航行之船舶。"
  ],
  "top_n": 2
}'

The returned relevance_score is the raw logit; if you need a 0～1 probability, apply a sigmoid yourself.

llama.cpp version requirement

This model's base, mmBERT, uses Gemma-2's 256k SentencePiece vocabulary. The SentencePiece vocabulary information was embedded at conversion time, so llama.cpp versions from 2026-08 onward can load it directly; if, after loading, Chinese is split character by character (a sentence yields 3 or more times as many tokens), your llama.cpp version is too old; please update.

Model background

The numbers for the three domains in the table above come from each domain's own validation set; absolute scores across batches cannot be compared directly with one another (each evaluation draws its own validation set). Only the "retriever vs. retriever plus reranker" comparison within the same evaluation is meaningful.

Generalization evaluation (different domains, different register)

The table above measures the three domains seen in training, which only shows the model works on what it has seen. We also measured how much of the effect remains on corpora never seen in training at all, using the same method as above: the same reranker and the same retriever (xVITA-Embed-300M-zhTW), 500 queries each, comparing "retriever only" against "retriever plus reranker" on the same set of queries.

It carries over to new domains. On government public documents it had never seen, hit@1 rose from 61.8% to 87.2%, and the nDCG gain still retains sixty percent of the gain on the training domains.

It weakens on a different register. On spoken transcripts, only thirty-six percent of the gain remains. Two points must be made clear, or this table will be misread:

1. The −2.0 percentage points in hit@1 on the spoken row does not constitute a regression. With 500 queries and a proportion of about 0.2, the standard error of a single proportion is already 1.8 percentage points, so this gap is less than one standard deviation. What actually holds up on that row is hit@5 (35.6% → 43.8%) and the nDCG gain.
2. The retriever itself also dropped from 61.8% to 20.8% on that row. A reranker can only reorder the candidates the retriever returns; if the correct answer is not among the candidates, nothing can be recovered. So how much of this +0.0799 comes from insufficient reranking ability and how much from an incomplete candidate pool cannot be separated with the current corpora.

Recommended use: RAG over written documents, such as regulations, official documents, technical documents, and specialist knowledge bases. If your queries or corpora are in a spoken register, such as meeting transcripts or customer-service dialogue, first measure on your own data using the method in this section before deciding, rather than applying the numbers in the table above directly.

Measurement conditions: epochs=0, pure evaluation with no weight updates; the retriever is the same one as in the online deployment, with the same retrieval prefix applied; training and validation are split by passage (not by pair), so that multiple questions about the same passage do not leak across sets; each round first runs a "passage against itself" sentinel check, which scored 100% for both groups.

License

MIT, the same as the base mmBERT-base.

About xCloudinfo

xCloudinfo Corp. Limited (云碩科技) builds on-premises, private AI solutions. xVITA is xCloudinfo's in-house model brand.

More models: https://hf.co/xCloudinfo