Downloads · 30 days
26
19% of all-time downloads
viswamaicoe/nanochat-telugu-560M-mid-v1-akshara
nanochat-telugu-560M-mid-v1-akshara is a text generation model from viswamaicoe. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as agpl-3.0.
NanoChat Telugu 560M Mid-Training v1 (Akhsara) is an intermediate checkpoint of the NanoChat Telugu model, trained on diverse Telugu conversational and task-specific datasets. This checkpoint bridges the gap between p…
Downloads · 30 days
26
19% of all-time downloads
All-time downloads
135
Public
Parameters
561M
1.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.1 GB · 99%
From the Hugging Face model README
NanoChat Telugu 560M Mid-Training v1 (Akhsara) is an intermediate checkpoint of the NanoChat Telugu model, trained on diverse Telugu conversational and task-specific datasets. This checkpoint bridges the gap between pre-training and supervised fine-tuning, providing enhanced instruction-following and conversational capabilities while serving as a foundation for further task-specific fine-tuning.
The model uses a custom Telugu-specific BPE (Byte Pair Encoding) tokenizer trained on Telugu text. This tokenizer achieves 70-82% reduction in token count compared to general-purpose tokenizers like GPT-2 and GPT-4, making it highly efficient for Telugu text processing.
This mid-training checkpoint was trained on a comprehensive mixture of Telugu datasets including:
The model demonstrates improved capabilities in:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "viswamaicoe/nanochat-telugu-560M-mid-v1-akshara"
device = "cpu" # or torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.float32,
).to(device)
# Example: Generate text
text = "హలో, మీరు ఎలా ఉన్నారు?"
inputs = tokenizer(text, return_tensors="pt").to(device)
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=50)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)
For proper conversational interactions, use the chat template with stopping criteria to handle special tokens correctly:
from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList
import torch
model_id = "viswamaicoe/nanochat-telugu-560M-mid-v1-akshara"
device = "cpu" # or torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.float32,
).to(device)
class AssistantEndStoppingCriteria(StoppingCriteria):
def __init__(self, stop_token_ids):
self.stop_token_ids = stop_token_ids
def __call__(self, input_ids, scores, **kwargs):
# Check if the last token is a stopping token
return input_ids[0, -1].item() in self.stop_token_ids
# Get stopping token IDs
assistant_end_token_id = tokenizer.encode("<|assistant_end|>", add_special_tokens=False)[0]
bos_token_id = tokenizer.bos_token_id if tokenizer.bos_token_id is not None else tokenizer.encode("<|bos|>", add_special_tokens=False)[0]
stopping_criteria = StoppingCriteriaList([
AssistantEndStoppingCriteria([assistant_end_token_id, bos_token_id])
])
# Chat completion example
conversation = [
{"role": "user", "content": "నమస్కారం"},
]
inputs = tokenizer.apply_chat_template(
conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt"
).to(device)
# Filter out token_type_ids and other unsupported keys
model_inputs = {
"input_ids": inputs["input_ids"],
}
if "attention_mask" in inputs:
model_inputs["attention_mask"] = inputs["attention_mask"]
print(f"Formatted prompt: {tokenizer.decode(model_inputs['input_ids'][0])}")
with torch.no_grad():
outputs = model.generate(
**model_inputs,
max_new_tokens=64,
do_sample=False,
stopping_criteria=stopping_criteria,
)
generated_tokens = outputs[0, model_inputs["input_ids"].shape[1]:]
print(f"Generated: {tokenizer.decode(generated_tokens)}")
You can extract sentence-level embeddings from the model for various downstream tasks like semantic search, text classification, or similarity computation:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "viswamaicoe/nanochat-telugu-560M-mid-v1-akshara"
device = "cpu" # torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.float32,
).to(device)
def get_sentence_embeddings(text, pooling_strategy="mean"):
"""Extract sentence-level embeddings using pooling"""
inputs = tokenizer(text, return_tensors="pt")
input_ids = inputs["input_ids"].to(device)
attention_mask = inputs["attention_mask"].to(device)
with torch.no_grad():
outputs = model(
input_ids=input_ids,
attention_mask=attention_mask,
output_hidden_states=True
)
# Get last hidden state
last_hidden_state = outputs.hidden_states[-1]
if pooling_strategy == "mean":
# Mean pooling (excluding padding tokens)
masked_embeddings = last_hidden_state * attention_mask.unsqueeze(-1)
sentence_embedding = masked_embeddings.sum(dim=1) / attention_mask.sum(dim=1, keepdim=True)
elif pooling_strategy == "max":
# Max pooling
masked_embeddings = last_hidden_state.masked_fill(~attention_mask.unsqueeze(-1).bool(), float('-inf'))
sentence_embedding = masked_embeddings.max(dim=1)[0]
return sentence_embedding
# Example usage
text = "హలో, మీరు ఎలా ఉన్నారు?"
sentence_embedding = get_sentence_embeddings(text, pooling_strategy="mean")
print(f"Sentence embedding shape: {sentence_embedding.shape}")
# Sentence embedding shape: torch.Size([1, 1280])
Pooling Strategies:
Important: This model is intended for research and legitimate use cases only. Users are responsible for ensuring their use of this model complies with applicable laws and ethical guidelines.
This model should NOT be used for:
This model is provided "as-is" without any warranties. The model developers and maintainers are not responsible for any misuse, damages, or consequences arising from the use of this model. Users assume full responsibility for their use of this model.
If you use this model, please cite:
@misc{nanochat-telugu-560m-mid-v1-akshara,
title={NanoChat Telugu 560M Mid-Training v1 (Akhsara): A Telugu Language Model Checkpoint},
author={Viswam AI COE},
year={2025},
howpublished={\url{https://huggingface.co/viswamaicoe/nanochat-telugu-560M-mid-v1-akshara}}
}
For questions or issues, please contact the model maintainers.