Genre-Classifier-1-20M-BASE-BF16

<p align="center"> <img src="https://huggingface.co/spaces/SpiceeChat/README/resolve/main/Spiceechat.png" alt="SpiceeChat" width="1400"> </p> <h1 align="center">Genre-Classifier-1-20M-BASE-BF16</h1> <p align="center"> <em>A lightweight base model for first-name gender classification.</em> </p> <p align="center"> <a href="https://huggingface.co/SpiceeChat"><img src="https://img.shields.io/badge/SpiceeChat-🔥-orange" alt="SpiceeChat"></a> <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-yellow" alt="License"></a> <img src="https://img.shields.io/badge/Params-~20M-blue" alt="Params"> <img src="https://img.shields.io/badge/Tensor%20Type-BF16-purple" alt="BF16"> </p>

---

Overview

A lightweight 20M-parameter CausalLM built for first-name gender classification. This is a base model, designed to be fine-tuned on downstream tasks rather than used directly.

---

Model Details

Special Tokens

The model uses a causal language modeling objective with weight tying between the input embedding and output head (head.weight = tok_emb.weight).

---

Architecture

A lightweight GPT-style decoder with:

• 4 transformer layers
• 4 attention heads with a head dimension of 96
• SageAttention support (falls back to PyTorch attention if sageattention is not installed)
• GELU activations in the MLP blocks
• LayerNorm before each attention and MLP block

---

Usage

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "SpiceeChat/Genre-Classifier-1-20M-BASE-BF16",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "SpiceeChat/Genre-Classifier-1-20M-BASE-BF16",
    trust_remote_code=True
)

Inference

The model provides a dedicated predict_gender() method:

inputs = tokenizer("Arjun", return_tensors="pt")
pred_idx, probs = model.predict_gender(inputs.input_ids)
gender = "M" if pred_idx.item() == 1 else "F"
print(gender)  # M

---

Training Data

This base model was pre-trained on a large-scale first-name dataset. It is not fine-tuned for any specific downstream task — it's meant to be used as a starting point.

---

Fine-Tuning

To fine-tune this model on your own dataset:

1. Load the base model with trust_remote_code=True
2. Use a causal LM loss on the last token (the model was designed to predict the gender token at the final position)
3. The special token IDs (42 = F, 49 = M) can be used as targets

Note: The model expects input sequences of length ≤ 20 tokens. Longer names will be truncated.

---

Dependencies

---

Acknowledgements

Built by PhysiQuanty for SpiceeChat.

---

<p align="center"> 📌 <strong>This is a base model.</strong> For a production-ready fine-tuned version, see <a href="https://huggingface.co/SpiceeChat/FirstName-Genre-Classifier-30M-SFT">FirstName-Genre-Classifier-30M-SFT</a>. </p> <p align="center"> _{Built with a lot of caffeine ☕ by SpiceeChat} </p>