pragna_new

Pragna-1b

<!-- Provide a quick summary of what the model is/does. -->

Architecture Overview

Pragna-1B is a decoder-only transformer model inspired by TinyLlama, featuring the following specifications:

• Layers: 22
• Attention Heads: 32
• Context Length: 2048
• Hidden Dimension: 2048
• Expansion Dimension: 5632
• Vocabulary Size: 69632

This model incorporates Rotary Positional Encoding to infuse positional information into the embeddings, utilising a base of 10,000. It employs RSNorm with an epsilon value of 1e-5 and the Sigmoid Activation Unit (SiLU) as the activation function. Additionally, Pragna-1B adopts Grouped Query Attention, an alternative to Multi-Head Attention, which enhances training and inference speed while reducing memory bandwidth. This also supports the use of lower-compute devices for inference tasks.

Pragna-1B is trained on our proprietary platform, GenAI Studio, a modular AI Developer Platform designed to support any GenAI model architecture. It is capable of scaling across thousands of GPUs or accelerators and is built to be fault-tolerant. The development of this model leveraged Triton, an open-source language from OpenAI, for crafting high-performance custom fused CUDA Kernels for various operations. Furthermore, the model uses Fully Sharded Data Parallel (FSDP) for distributed and parallel training and incorporates the state-of-the-art FlashAttention2 to accelerate training and inference.

Model Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: Soket AI Labs
• Language(s) (NLP): Hindi, Bangla, Gujarati and English
• License: Apache 2.0

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("soketlabs/pragna-1b")
model = AutoModelForCausalLM.from_pretrained(
    "soketlabs/pragna-1b", torch_dtype=torch.bfloat16
)

Training Details

Training Data

1. Bhasha-wiki
2. SlimPajama
3. Sangraha-Verified

Training Procedure

[To be added]

Training Hyperparameters

• Precision: BFloat16
• Batch Size: 2k - 2.5k
• Context Length: 2,048
• Learning Rate: 3e-5
• Optimizer: AdamW
• LR Scheduler: Cosine
• Mixed Precision Training

Evaluation

Hindi

Gujarati

English

Results

Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

@ONLINE{bhasha-wiki,
    author = "Soket Labs Technology and Research Private Limited",
    title  = "pragna-1b",
    url    = "https://soket.ai"
}

Model Card Contact

[email protected]