Supernova-teraillm-reranker-v1

🇳🇵 Supernova TeraLLM Reranker V1

Supernova TeraLLM Reranker V1 is an experimental Nepali-focused Transformer reranker developed by the Supernova project.

It scores the relevance of a document with respect to a query and is intended for search and retrieval pipelines.

Status: Production baseline / research model

---

🚀 Model Overview

The production architecture is a Transformer-based pair reranker.

Query + Document
       ↓
Supernova Tokenizer
       ↓
Token IDs
       ↓
Embedding
       ↓
16 Transformer Encoder Layers
       ↓
CLS Representation
       ↓
Ranking MLP
       ↓
Relevance Score

---

🧠 Architecture

Verified production architecture:

Embedding
├── Vocabulary size: 4
└── Hidden size: 1024

Transformer Encoder
├── Layers: 16
├── Hidden size: 1024
├── Feed-forward size: 4096
├── Multi-head self-attention
└── Dropout: 0.1

Ranking Head
├── Linear: 1024 → 512
├── GELU
├── Linear: 512 → 1
└── Sigmoid

Parameters

202,593,281 parameters

Production checkpoint size:

Approximately 91.96 MB

---

🔤 Tokenizer

The production tokenizer was verified against the model.

Model vocabulary:      4
Tokenizer vocabulary:  4
Embedding dimension:   1024

Compatibility test:

Input shape:       (1, 128)
Maximum token ID:  3
Vocabulary size:   4

The generated token IDs are compatible with the model embedding.

---

📚 Confirmed External Dataset

Bharat-NanoMSMARCO Nepali

The confirmed external Nepali retrieval dataset used during the Supernova retrieval experiments is:

carlfeynman/Bharat_NanoMSMARCO_ne

Dataset:

https://huggingface.co/datasets/carlfeynman/Bharat_NanoMSMARCO_ne

Verified structure:

Corpus:      5,043 documents
Queries:        50 queries
Qrels:          50 relevance judgments
Language:     Nepali
Task:         Information Retrieval
License:      CC-BY-4.0

The dataset contains Nepali retrieval queries, a document corpus, and query-document relevance judgments.

Users must follow the original dataset's license and attribution requirements.

---

🧪 Supernova Fine-Tuning Data

A later fine-tuning experiment constructed derived ranking triplets from the available retrieval material.

Random triplets:        1,000
Hard-negative triplets:   500
Total:                   1,500

Split:

Training:    1,350
Validation:    150

These triplets are derived experimental examples, not a separate public dataset.

---

⚔️ Supernova vs Jina Reranker V3

A custom three-example live benchmark was performed.

Categories:

1. Historical Nuance
2. Geographical Logic
3. Slang / Intent Extraction

Results

Category                  Supernova V1    Jina-v3
---------------------------------------------------
Historical Nuance              PASS           PASS
Geographical Logic             PASS           PASS
Slang / Intent                 FAIL           FAIL
---------------------------------------------------
Pairwise Accuracy             2/3            2/3
                              66.67%         66.67%

Mean Positive-Negative Margin

Supernova V1:      +0.001901
Jina Reranker V3:  +0.140182

Experimental CPU Latency

Supernova total:       8.5254 seconds
Jina-v3 total:        88.3780 seconds

Supernova/example:     2.8418 seconds
Jina-v3/example:      29.4593 seconds

These measurements are specific to the experimental environment.

---

⚠️ Benchmark Interpretation

The three-example benchmark resulted in an accuracy draw:

Supernova V1: 2/3

Jina-v3: 2/3

Therefore this benchmark does NOT establish that Supernova V1 beats Jina Reranker V3.

Jina-v3 produced a substantially larger mean ranking margin in this particular test.

A substantially larger independent benchmark is required for strong performance claims.

---

🎯 Intended Use

Supernova V1 can be used for:

• Nepali search
• Semantic retrieval
• Search-result reranking
• Document relevance scoring
• RAG pipelines
• Nepali information retrieval research
• Multilingual retrieval research

Typical pipeline:

User Query
    ↓
First-Stage Retriever
    ↓
Top-K Documents
    ↓
Supernova Reranker
    ↓
Re-ranked Results
    ↓
Search / RAG

---

🛑 Limitations

• The confirmed Nepali retrieval dataset is relatively small.
• The Jina comparison contains only three examples.
• The benchmark is not a large standardized reranking benchmark.
• Results can vary by domain and query distribution.
• The current benchmark does not prove superiority over Jina-v3.
• Larger held-out evaluation is required.
• More hard-negative mining is required for stronger generalization.

---

🔬 Future Research

Future Supernova reranker versions will investigate:

• Larger Nepali retrieval datasets
• More diverse Nepali queries
• Stronger hard-negative mining
• Better query-document interaction
• Parameter-efficient fine-tuning
• Improved ranking objectives
• Larger held-out benchmarks
• CPU-efficient inference
• Multilingual retrieval
• Better generalization

---

🔁 Reproducibility

The published V1 checkpoint should remain the immutable baseline.

Future experiments should create a separate fine-tuning copy:

Supernova V1
     ↓
Immutable baseline

Supernova V1
     ↓
Fine-tuning copy
     ↓
Nepali ranking data
     ↓
Fine-tuning
     ↓
Held-out evaluation

---

📦 Production Checkpoint

File:        pytorch_model.bin
Parameters:  202,593,281
Size:        ~91.96 MB

---

📜 Dataset Attribution

Bharat-NanoMSMARCO Nepali:

https://huggingface.co/datasets/carlfeynman/Bharat_NanoMSMARCO_ne

Please consult the original dataset card for complete licensing and attribution requirements.

---

📖 Citation

@misc{supernova_teraillm_reranker_v1,
  title={Supernova TeraLLM Reranker V1},
  author={Supernova},
  year={2026},
  publisher={Hugging Face},
  note={Nepali-focused Transformer reranker}
}

---

⚖️ Disclaimer

This model is provided for research and experimental purposes.

Benchmark results represent measurements from the development environment and are not universal guarantees of performance.

Users should evaluate the model on their own data and deployment environment before production use.

---

🇳🇵 Supernova

Building AI infrastructure for Nepali language technology.