whisper-full-backup-checkpoint-1100

Whisper Medium - Sanskrit to English Speech Translation (LoRA)

LoRA adapter for direct Sanskrit-speech-to-English-text translation, fine-tuned from openai/whisper-medium on the Sanskrit portion of AI4Bharat's Shrutilipi corpus.

Supporting artefacts for the paper "Fine-Tuning Whisper Medium for Sanskrit-to-English Speech Translation: A Low-Resource Case Study with Error Analysis" (ICCIDC 2026).

---

Read this before using any number from this repo

There are two evaluation runs here and only one of them is valid.

The original 5% random split was made at the utterance level. Shrutilipi is mined from All India Radio bulletins, where the same news item is read repeatedly across editions, so 227 of the 709 held-out utterances (32.0%) had a near-duplicate in the training set. Every metric computed on that split measures partial memorisation as much as generalisation.

The results/ numbers are computed on the 482 utterances that survived screening. Those are the numbers in the paper.

---

Results (n = 482, decontaminated)

Paired bootstrap over 482 segments, 1000 resamples: dBLEU = +18.07, p = 0.001. No resampled draw favoured the zero-shot baseline.

Distribution, not just the mean. Median sentence-level BLEU rises 3.9 to 17.0. 71 utterances (14.7%) clear BLEU 40 after adaptation, against 2 before. 42 utterances (8.7%) score lower after adaptation - the gain is a large net effect, not a uniform one. See results/fig3_segment_distribution.png.

results/paper_values.txt carries every figure quoted in the paper, regenerated from the committed predictions.

---

Repository map

README.md                        <- you are here
adapter_config.json              <- the reported model: LoRA adapter for
adapter_model.safetensors           checkpoint-4000, 37.8 MB
processor_config.json
tokenizer.json
tokenizer_config.json
training_args.bin                <- torch.load() for the full TrainingArguments

results/                         <- AUTHORITATIVE: everything the paper reports
  base_vs_finetuned.csv             482 rows: reference, base_whisper, finetuned
  base_vs_finetuned_summary.csv     the metrics table above
  per_segment_scores.csv            per-utterance BLEU, sorted worst-delta first
  fig3_segment_distribution.png     BLEU distribution, before vs after
  paper_values.txt                  every figure quoted in the paper

superseded/                      <- contaminated 709-run, kept only for provenance
  README.md                         why these must not be cited
  eval_results_n709_contaminated.csv
  eval_summary_n709_contaminated.csv

final_processed_data/            <- the split the model was trained on
  train/  13,467 utterances
  test/      709 utterances  (the contaminated held-out portion)

whisper-sanskrit-en/             <- training checkpoints, every 20 steps
  checkpoint-4000/                  <- THE REPORTED MODEL
  checkpoint-340 ... checkpoint-3980   intermediate, not used in the paper

.virtual_documents/
  __notebook_source__.ipynb      <- the training notebook

If you only want one thing: results/base_vs_finetuned.csv plus the top-level adapter files. Everything else is provenance.

---

Training configuration

Recoverable from training_args.bin and whisper-sanskrit-en/checkpoint-4000/trainer_state.json.

No checkpoint selection was performed. The 4000-step budget was fixed in advance and validation loss fell monotonically to the last step, so the reported model is simply the terminal checkpoint - not a best-scoring intermediate one picked on held-out data.

---

Evaluation configuration

Both systems decoded over identical inputs with identical settings: beam search with 5 beams, max 444 new tokens, no-repeat 4-gram, repetition penalty 1.3, and a forced decoder prompt of language="sanskrit", task="translate".

Metrics - quote these signatures if you report the numbers:

BLEU  nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0
chrF  nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.6.0

METEOR: HuggingFace evaluate (NLTK backend), scaled x100. BERTScore: roberta-large, baseline rescaling DISABLED.

Rescaled and unrescaled BERTScores are not comparable. The zero-shot model scores 86.42 not because it is 86% accurate but because unrescaled BERTScore is bounded well away from zero for any pair of fluent English sentences. Treat it as relative only.

---

Known limitations

1. References are machine-generated. Shrutilipi supplies Sanskrit audio aligned to Sanskrit text, not English. The English side was produced by IndicTrans2. Every metric here measures agreement with IndicTrans2, not accuracy against a human standard. There are documented cases where the reference is wrong and this model is right and is penalised for it - so the lexical scores are a lower bound.

2. Inference precision is not matched. The zero-shot baseline ran in fp32; the adapted model ran 8-bit quantised with fp16 activations. Quantisation degrades output, so the confound biases against the adapted model - the reported margin is conservative.

3. Decontamination is coarse. The screen hashes the first ten tokens of each target label. It misses paraphrases that diverge early and over-removes distinct utterances that share a formulaic opening. Treat 32.0% as a lower bound on contamination.

4. Two utterances are reproduced verbatim. The adapted model outputs its reference exactly for two items, including a 13-word sentence with two proper nouns. That is either genuine quality or residual memorisation the prefix screen missed; the present evidence cannot separate them. A full-string similarity screen is the fix.

5. No WER/CER. The acoustic stage is unmeasured, so claims about error propagation from recognition to translation are argued, not measured.

6. Domain is narrow. Read broadcast news, professional newsreaders. No claim is made about liturgical, pedagogical or conversational Sanskrit.

---

Reproducing the numbers

No model needs to be loaded - the predictions are committed.

import pandas as pd, evaluate
from huggingface_hub import hf_hub_download

csv = hf_hub_download("jashu1326/whisper-full-backup-checkpoint-1100",
                      "results/base_vs_finetuned.csv", repo_type="model")
df = pd.read_csv(csv)
refs = df["reference"].tolist()
bleu = evaluate.load("sacrebleu")
for col in ["base_whisper", "finetuned"]:
    print(col, bleu.compute(predictions=df[col].tolist(),
                            references=[[r] for r in refs])["score"])

Runs on CPU in seconds.

---

Inference

from transformers import WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel

REPO = "jashu1326/whisper-full-backup-checkpoint-1100"
processor = WhisperProcessor.from_pretrained("openai/whisper-medium",
                                             language="sanskrit", task="translate")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-medium")
model = PeftModel.from_pretrained(model, REPO)

# decode with: num_beams=5, max_new_tokens=444,
# no_repeat_ngram_size=4, repetition_penalty=1.3

---

Provenance

The two evaluation runs differ in three ways at once - sample, contamination and decoding strategy - so their numbers are not comparable to each other. Only the decontaminated run is reported.

Citation

@inproceedings{jashwanth2026sanskrit,
  title     = {Fine-Tuning Whisper Medium for Sanskrit-to-English Speech Translation:
               A Low-Resource Case Study with Error Analysis},
  author    = {Jashwanth, Veerlapati Sai and Vamshi, Seelam Shiva
               and Singh, Yuvraj and Darshith},
  booktitle = {Proc. 2nd Int. Conf. on Computational Intelligence and Data Communication},
  year      = {2026}
}

Built on Whisper (Radford et al., 2023), LoRA (Hu et al., 2022), Shrutilipi (Bhogale et al., 2023) and IndicTrans2 (Gala et al., 2023).