warabi-reranker-v1

IME Context Reranker v1

Japanese IME cross-encoder that reranks dictionary-generated conversion candidates from the preceding context, reading, and candidate surface form. The reference runtime scores the first 64 algorithmic candidates and blends the model score with calibrated dictionary cost.

Input format

The model receives a sentence pair:

文脈：{context}
読み：{reading} [SEP] 候補：{candidate}

Higher regression logits indicate better candidates. The tokenizer uses custom code inherited from the base model, so load it with trust_remote_code=True.

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "fa0311/ime-context-reranker-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

query = "文脈：家に\n読み：かえる"
candidate = "候補：帰る"
inputs = tokenizer(query, candidate, return_tensors="pt")
score = model(**inputs).logits.item()

Runtime configuration

• Maximum sequence length: 192
• Context window: 160 characters
• Model-scored candidates: 64
• Dictionary cost weight: 0.19
• Reference IME runtime: CUDA only

Evaluation

The difficult full-input set has 72.80% candidate recall, so its reranking scores include failures where the target was absent from the candidate set. Metrics depend on the associated dictionary candidate generator and cost blend; they are not standalone sentence-classification accuracy.

Related repositories

• ime-context-dataset
• ime-dictionary-core
• ime-dictionary-extended-gpl

License

Apache License 2.0. The model is fine-tuned from line-corporation/line-distilbert-base-japanese, which is also distributed under Apache-2.0.