Krutrim-1-instruct

Krutrim-1

Model Overview

Krutrim Large Language Model (LLM) is a 2 trillion token multilingual foundation model designed to serve Indian demographic needs through equitable representation of the country's array of native tongues. Training data incorporates the largest known Indic language dataset, mitigating associated data scarcity obstacles that encumber model parity across dialects. Evaluations demonstrate Krutrim's strong performance on Indic language benchmarks, surpassing or at par with state-of-the-art models despite being significantly smaller in training flops. Krutrim LLM also matches or exceeds standards set on English benchmarks by models trained on comparable flops (e.g. vs LLAMA-2 on 10 out of 16 tasks with average score of 0.57 vs 0.55 of LLAMA-2), evidencing flexible multilingual fluency. Through intentional design choices that redress endemic data imbalances, Krutrim LLM signifies meaningful progress in the pursuit of ethical, globally representative AI foundation models.

Key Features

• 7B parameter dense transformer model comparable similarly sized LLama-2 model;
• Natively multilingual delivering best-in-class performance for a 7B mdoel on Indic benchmarks;
• Exceeds performance of similar sized models on multilingual Indic generation tasks including creative writing, summarization, and translation;
• Available in instruction-tuned version

Model Developer

• OLA Krutrim Team

Model Dates

• Krutrim-1 was trained between Oct 2023 and Nov 2023.

Release History

Data Freshness

• The dataset includes information up to April 2023.

Model Architecture

• Layers: 32
• Max Sequence Length: 4096
• Hidden Dimension: 4608
• Head Dimension: 96
• Number of Heads: 48
• Number of KV-Heads: 8 (GQA)
• Vocabulary Size: 70400
• Architecture Type: Transformer Decoder (Auto-regressive Language Model)

Evaluation Results

English Comparison between Llama2Chat-7B and Krutrim-1-7B

Benchmarks

Usage

To use the instruct model, you can load it with AutoModelForCausalLM as follows:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "krutrim-ai-labs/Krutrim-1-instruct"
# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Apply Chat Template
chat_template ="{% for message in messages %}{% if message['role'] == 'system' %}{{ '<|SYSTEM|> ' + message['content'] + '\n' }}{% elif message['role'] == 'user' %}{{ '<|USER|> ' + message['content'] + '\n' }}{% elif message['role'] == 'assistant' %}{% if not loop.last %}{{ '<|RESPONSE|>\n'  + message['content'] + eos_token + '\n' }}{% else %}{{ '<|RESPONSE|>\n'  + message['content'] + eos_token }}{% endif %}{% endif %}{% if loop.last and add_generation_prompt %}{{ '<|RESPONSE|>\n' }}{% endif %}{% endfor %}"
tokenizer.chat_template = chat_template

prompt_dict = [
    {"role": "system", "content": "You are an AI assistant."},
    {"role": "user", "content": "Who are you?"}
]

prompts = tokenizer.apply_chat_template(prompt_dict, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(prompts, return_tensors='pt').to(device)
inputs.pop("token_type_ids", None)

# Generate response
outputs = model.generate(
    **inputs,
    max_length=100
)

response = tokenizer.decode(outputs[0])
print(response)

Limitations

The model was trained on a dataset that includes content from the internet, which may contain toxic language, biases, and unsafe content. As a result, the model may:

• Amplify biases present in the training data
• Generate toxic responses, especially when prompted with toxic inputs
• Provide inaccurate, incomplete, or redundant answers
• Generate responses in languages inconsistent with the prompt

Ethical Considerations

• The model may produce biased or offensive outputs based on its training data.
• Users should apply human oversight when using the model for decision-making in sensitive areas.
• While safeguards have been implemented, the model may still generate socially undesirable text in certain contexts.

License

This code repository and the model weights are licensed under the Krutrim Community License.

Contact

Contributions are welcome! If you have any improvements or suggestions, feel free to submit a pull request on GitHub.

Citation

@misc{KrutrimLLM2024,
  author = {Aditya Kallappa, Palash Kamble, Vinayak Dhruv, Akshat Patidar, Arveti Manjunath, Raghav Awasthi, Divyansh Rajput, Pidathala Sowjanya, Rahul Kumar, Rishabh Nahata, Sanket Shah, Sulabh Katiyar, Soham Pendurkar, Sindhu Pawar, Abhinav Ravi, Deepak Kumar, Kumar Ashish, Pranav Raveendran, Bidyapathi Ray},
  title = {Krutrim LLM},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/ola-krutrim/Krutrim-1-7B}}
}