gemma4-sikshagemma

ShikshaGemma: A Hindi-Native Science & Math Tutor for India

A fine-tuned Gemma 4 E4B model

ShikshaGemma is a single Hindi-native tutor for science and mathematics, built to support students from Class 6 through MSc. It gives step-by-step explanations in Hindi across school science, JEE and NEET preparation, undergraduate science, and postgraduate-level physics, chemistry, mathematics, and biology.

1. Training

The fine-tuning is staged into two phases, with explicit goals per phase, using Unsloth for memory-efficient 4-bit LoRA SFT.

1.1 Stage 1: English reasoning foundation

Goal: build deep scientific and mathematical reasoning capability in the base model, in the language where high-quality CoT material is most abundant: English.

Data: 120,000 samples total.

• 100,000 samples from established high-quality public English chain-of-thought / reasoning datasets in math and the sciences.
• 20,000 synthetic samples generated from classical school-and-undergraduate physics, chemistry, mathematics, and biology textbooks, with explicit step-by-step CoT solutions.

Compute: ~60 GPU-hours, 4-bit LoRA SFT via Unsloth.

Rationale: transferring a reasoning capability across languages is a substantially easier learning problem than learning reasoning from scratch in a low-resource language. Stage 1 builds the reasoning prior; Stage 2 specialises the language.

1.2 Stage 2: Hindi specialisation

Goal: teach the Stage-1-reasoning-strong checkpoint to speak and think in Hindi across school, entrance, undergrad, and postgrad science.

Data: 40,000 samples total.

• 36,000 synthetic Hindi samples generated from:
  • NCERT Class 6–12 science and math (Hindi medium).
  • IGNOU undergraduate and postgraduate Hindi science material.
  • VMOU (Vardhman Mahaveer Open University) Hindi BSc / MSc material. Each sample is a Hindi problem with a Hindi step-by-step CoT solution and a strict-format final answer.
• 4,000 samples from JEE Main and NEET previous-year papers, translated and verified into Hindi, with step-by-step Hindi solutions.

Compute: ~5 GPU-hours, 4-bit LoRA SFT via Unsloth on top of the Stage-1 checkpoint.

Total fine-tuning compute: ~65 GPU-hours end-to-end on a single 24 GB consumer-class GPU (L4).

---

2. Deployment

Deployment uses Google AI Edge with the model converted to LiteRT format.

• Conversion: the merged Stage-2 checkpoint is converted to LiteRT, quantised to 4-bit, and packaged for on-device execution.
• Runtime: Google AI Edge runtime on Android. The model executes locally on the device CPU / NPU. No network roundtrip.
• Optional path: the same merged checkpoint can also be served via transformers + 4-bit BNB or via vLLM for classroom / lab deployments with a shared GPU; this is what we use for benchmarking.

---

3. Evaluation

We evaluate on two tracks: stock peer-comparable benchmarks (Track A) and custom Hindi JEE benchmarks we built ourselves (Track B). Each source dataset is broader than the slice we evaluated on; the tables below state explicitly what each source contains and which subset we used.

3.1 Track A: stock benchmarks (lm-evaluation-harness)

3.2 Track B: custom Hindi JEE benchmarks (inspect-ai)

Each question carries question_type ∈ {single_correct, multi_correct, numerical}. The scorer dispatches per type: exact letter match for single, sorted-letter-set equality for multi, 1% relative tolerance for numerical with a/b fraction support and LaTeX normalisation (\frac{}{}, \boxed{} preprocessing). Scores break down per subject and per question type.

---

4. Results

Both tracks were run twice with the same scorer: once on stock Gemma 4 E4B (baseline), once on the Stage 2 ShikshaGemma checkpoint.

4.1 Track A: stock benchmarks

4.2 Track B: custom Hindi JEE benchmarks

Per-subject and per-question-type breakdowns (Track B):

JEE Main 2026 (Hindi):

JEE Advanced 2025 (Hindi, text-only):

Raw outputs and logs are in final_results/.

4.3 Reading of the numbers

The Stage 2 ShikshaGemma checkpoint regresses against base Gemma 4 E4B on every Hindi benchmark we report. The regression is small on the multilingual MCQ benchmarks (MILU, IndicMMLU-Pro Hindi STEM) and large on the JEE-grade benchmarks. GPQA in English shifts slightly upward, consistent with Stage 1 having had a positive effect on English reasoning before Stage 2.

Stage 2 is not production-ready yet, needs more debugging and fixing.

---