Nova-Casual-LLM

Nova-Casual-LLM

A lightweight and efficient transformer-based language model built with custom architecture and integrated seamlessly into the Hugging Face ecosystem.

Model Summary

• Model Name: NovaForCausalLM
• Architecture: Custom decoder-only transformer (NOVA)
• Model Type: nova
• Use Case: Causal Language Modeling (text generation, auto-completion)
• Parameters: 14,412,400 trainable parameters
• Pretrained Tokenizer: PreTrainedTokenizerFast
• Framework: PyTorch
• Hugging Face Integration: Compatible with transformers via custom AutoModel and AutoConfig registration.

---

Files Included

---

Model Architecture

NovaForCausalLM

The model consists of:

• Embedding layers: token + positional
• Stack of transformer decoder blocks
• Multi-head attention with 640 individual heads
• Layer normalization
• Final linear head for vocabulary logits

Configuration (NovaConfig)

{
  "model_type": "nova",
  "vocab_size": 6000,
  "block_size": 256,
  "n_embd": 640,
  "n_layer": 4,
  "n_head": 8
}

🚀 Usage

Step 1: Clone the repo (to get the nova_modelling.py)

git clone https://huggingface.co/harshit36/Nova-Casual-LLM
cd Nova-Casual-LLM

import sys
sys.path.append("./Nova-Casual-LLM/")  # add current dir to path

from transformers import PreTrainedTokenizerFast
from nova_modelling import NovaConfig, NovaForCausalLM

# Load tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained("harshit36/Nova-Casual-LLM")

# Load config
config = NovaConfig.from_pretrained("harshit36/Nova-Casual-LLM")

# Instantiate model using your custom class
model = NovaForCausalLM(config)
model = model.from_pretrained("harshit36/Nova-Casual-LLM")

# Use the model
input_ids = tokenizer("Hello world", return_tensors="pt").input_ids
output = model.generate(input_ids)
print(tokenizer.decode(output[0], skip_special_tokens=True).replace(" ","").replace("Ġ"," ").replace("Ċ","\n"))

Intended Use

General text generation

Hybrid Positional Encoding Research model (Combination of Sinusoidal and learnable encodings)

Educational demonstrations of custom HF model integration

Rapid prototyping of transformer models