Mantra-14B

Mantra-14B

Mantra-14B is a 14.7B parameter instruction-tuned bilingual large language model for both Hindi and English, trained on a mixed language dataset.

• ~0.7 % better performance on English Tasks compared to the original (average benchmark scores)
• ~2.8 % better performance on Hindi Tasks compared to the original (average benchmark scores)
• ~4.4 % better performance on tougher english benchmarks (open-llm-leaderboard evals)
• ~8.5 % less emissions than the original (as reported on benchmark evaluations like open-llm-leaderboard)
• Less Biases due to ordering of choices while answering MCQs

Model Details:

• Developed by: Traversaal.ai, 1-800-LLMs
• Language(s) (NLP): Optimized for Hindi and English
• License: Apache 2.0
• Paper : TBA April 15

Prompt Formats

The following prompt formats were used during training and are better suited for usage, however the model works well even without such formatting

Evaluation:

We evaluated our models on multiple well-known benchmarks to measure their effectiveness against other leading models, and the results are as follows:

Table 1: Metrics (.2f) of our models and other LLMs over several English benchmarks

Table 2: Metrics (.2f) of our models and other LLMs over several Hindi benchmarks