harrier-oss-v1-0.6b

harrier-oss-v1-0.6b — LiteRT

microsoft/harrier-oss-v1-0.6b converted to LiteRT (.tflite) for on-device inference. Microsoft's multilingual text-embedding model (100+ languages) for retrieval, RAG, clustering and semantic similarity, producing 1024-dimensional L2-normalized vectors — fully offline, on CPU.

Last-token pooling and L2 normalization are inside the graph: one call in, one finished embedding out.

Both are task-lossless against the PyTorch reference on every gate below; fp16 additionally reproduces the reference to every printed digit.

⚠️ The query instruction is mandatory

Each query must carry a one-sentence task instruction — documents are encoded raw. This is how the model is trained (the upstream FAQ says so), and the cost of skipping it is now measured, on this artifact:

Format: Instruct: {task description}\nQuery: {your query} — e.g. for search, Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: . The upstream repo pre-configures web_search_query, sts_query and bitext_query prompts in config_sentence_transformers.json; for symmetric tasks (similarity, clustering) use the instruction on both sides.

One more contract detail worth knowing: the bundled tokenizer appends <|endoftext|> to every text itself (a tokenizer post-processor), and that appended token is what the model pools. Tokenize with the files shipped here and you get this for free; a hand-rolled BPE without the post-processor silently changes the embedding.

Signatures

Batch-1, right-padded static shapes: input_ids int32 [1, S], attention_mask int32 [1, S] (1 = real token, then 0s — padding must be contiguous on the right), for S in 64 / 128 / 256 / 512.

Pad the token ids into the smallest signature that fits and set the mask accordingly. The pooled position is derived from the mask inside the graph (sum(mask) − 1), so the result is independent of which signature you route through: the same text through embed_64/128/256/512 returns bitwise identical vectors, and pad-region token ids cannot influence the output at all.

Embeddings are L2-normalized, so cosine similarity is a dot product.

Usage (Python)

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from transformers import AutoTokenizer

PAD_ID = 151643
tok = AutoTokenizer.from_pretrained("microsoft/harrier-oss-v1-0.6b")
it = Interpreter(model_path="harrier-oss-v1-0.6b_wi8fc.tflite", num_threads=8)

LENS = sorted(int(n.split("_")[1]) for n in it.get_signature_list())
runners = {s: it.get_signature_runner(f"embed_{s}") for s in LENS}

def embed(text, instruction=None):
    if instruction:
        text = f"Instruct: {instruction}\nQuery: {text}"
    ids = tok(text)["input_ids"][:LENS[-1]]
    S = next(s for s in LENS if len(ids) <= s)
    x = np.full((1, S), PAD_ID, np.int32)
    m = np.zeros((1, S), np.int32)
    x[0, :len(ids)] = ids
    m[0, :len(ids)] = 1
    return list(runners[S](input_ids=x, attention_mask=m).values())[0][0]

task = "Given a web search query, retrieve relevant passages that answer the query"
q = embed("日本で一番高い山は？", instruction=task)
d = embed("Mount Fuji is the highest mountain in Japan, at 3,776 meters.")
print("cosine:", float(q @ d))

Texts longer than 512 tokens must be chunked (the upstream model accepts 32768, but a static on-device graph at that length is not practical).

Quality

Four independent checks, each run on every variant against the PyTorch fp32 reference.

1. The base card's own usage example (2 queries × 2 MS-MARCO passages): every variant ranks the right passage first; fp16 reproduces the reference score matrix bit-exactly, int8 within 0.008 cosine.

2. JSTS (Japanese semantic similarity, JGLUE v1.3 validation, 300 pairs, Spearman, sts_query instruction both sides): PyTorch 0.8553, int8 0.8559, fp16 0.8553.

3. JSQuAD retrieval (Japanese question → Wikipedia paragraph, 800-paragraph corpus, 150 questions): PyTorch nDCG@10 0.9363 / hit@1 0.880; int8 0.9360 / 0.880; fp16 identical to PyTorch. The corpus is subsampled, so absolute numbers are not comparable to published benchmarks; at 150 queries the gate cannot resolve differences below a couple of percent.

4. Cross-variant retrieval — the RAG deployment shape. Documents encoded with the PyTorch model, queries with the int8 artifact (index built on a server, queried on device): nDCG@10 0.9389 vs the 0.9363 all-PyTorch control — the quantized embedding space is compatible with an upstream-built index.

Multilingual spot-check (STS17 Spearman, 100 pairs each, int8): en-en 0.890 / ko-ko 0.890 / es-en 0.838 / en-ar 0.848, all within 0.004 of the PyTorch reference.

Speed

CPU/XNNPACK, median of 10 runs, 75%-full signatures:

A static signature computes all S positions regardless of how many are real, so route each text to the smallest signature that fits.

Memory

Measured at load+invoke on an M4 Max Mac (8 threads): the int8 file peaks at ~2.4 GiB with all four signatures — and, unlike some larger embedders, trimming signatures buys almost nothing here (a 2-signature build measured 2.35 GiB), so the flexible 4-signature file is the one shipped. The fp16 file peaks at ~12 GiB (XNNPACK expands fp16 weights to fp32 per signature subgraph) — it is a desktop artifact; use int8 on device.

Conversion

Encoder lane — a direct multi-signature litert_torch trace of the HF model (not an LLM export). Attention stays causal (this is the e5-mistral-style recipe: a decoder body pooled at the last token — nothing is flipped to bidirectional), with the causal+padding attention bias built by hand inside the traced wrapper, and the last-token pool expressed as a one-hot over sum(mask) − 1 so it lowers to static ops. Gated on: causality (editing the last token moves nothing earlier), pad-content invariance (bitwise), pool rebinding (dropping the last real token must move the output), and cross-signature bitwise agreement.

Script and full notes: hf-to-litertlm.

License

MIT, inherited from the base model by Microsoft.

Modification notice: these files are converted, not original. The weights were exported to LiteRT and quantized (int8 dynamic-range / fp16); last-token pooling and L2 normalization were folded into the graph. No fine-tuning or weight modification beyond quantization was performed.