Falin-300M-Preview

🇮🇳 Cortiqa Falin-300M (Preview)

An early research preview of the sovereign 297M SLM developed by Cortiqa.

Falin-300M is a 297-Million parameter decoder-only transformer model designed, engineered, and trained from scratch by Cortiqa. Built upon the proprietary Menothus architecture, Falin is optimized for extreme low-latency and edge device deployment (consumer GPUs, CPUs, mobile devices, and browser extensions).

Note: This is a base / text-completion model, not an instruction-tuned chat model. It is designed to be fine-tuned for downstream tasks — see the Fine-tuning section below.

🚀 Model Architecture & Innovations

• Total Parameters: 297,034,800 (~300M)
• Extreme Grouped Query Attention (GQA): 16 Query heads to 2 Key-Value heads (8:1 ratio), reducing KV-cache VRAM consumption by 75% during inference.
• Parallel Attention + SwiGLU FFN: Computes Attention and Feed-Forward networks concurrently, improving GPU utilization and decreasing per-layer execution latency.
• Hybrid Sliding Window Attention (SWA): Local context window of 512 tokens with every 4th layer computing dense global causal attention.
• FlashAttention / SDPA Native: Fully optimized for scaled dot product attention.

📊 Training Data

Pre-trained from scratch on a curated dataset (~40M tokens), including general text and a dedicated identity dataset so the model self-identifies as "Falin." This is an early-stage dataset — scale and diversity are actively being expanded in future releases.

⚠️ Limitations

• This is an Alpha research preview — not production-ready.
• As a 297M base model trained on a relatively small dataset, it may produce incoherent or hallucinated output on general/open-domain prompts, especially outside its training distribution.
• Not instruction-tuned — it will not behave like a chat assistant out of the box. Use it as a base for fine-tuning.
• Max context length is limited to 1024 tokens.

🛠️ How to Run Inference

1. Requirements

pip install torch tokenizers huggingface_hub

2. Python Inference Code

import torch
from huggingface_hub import snapshot_download

# Download model repository from Hugging Face
model_dir = snapshot_download(repo_id="Cortiqa/Falin-300M-Preview")

# Load model weights and config
# (Use the Menothus architecture code from the repo)

🎯 Fine-tuning

Falin-300M is released specifically to be fine-tuned — it is not meant to be used directly for chat or instruction-following. Because it's only ~300M parameters, it's cheap and fast to fine-tune, even on a single consumer GPU or a free-tier Colab instance.

What you can fine-tune it for:

• Domain-specific text generation (e.g. legal, medical, customer support in your niche)
• Instruction-following / chat behavior (using an instruction dataset + SFT)
• Regional language or dialect adaptation
• Task-specific completion (classification-as-generation, summarization, etc.)

How to fine-tune:

1. Prepare your dataset as plain text or instruction-response pairs (JSONL recommended).
2. Load the base weights from this repo using the Menothus architecture code.
3. Fine-tune using standard causal LM training (full fine-tune or LoRA/PEFT-style adapters both work well at this scale).
4. Because of the small size, even a few thousand examples and a handful of epochs on a free Colab GPU can produce noticeable behavior changes — start small and iterate.
5. Push your fine-tuned checkpoint back to the Hub tagged with base_model: Cortiqa/Falin-300M-Preview so it shows up under this model's Finetunes tree.

Fine-tuned/derivative models must remain non-commercial and follow the attribution and terms in LICENSE.md — see Section 3 of the license for details.

If you fine-tune Falin for something interesting, tag @CortiqaAI — community fine-tunes may get featured.

📈 Evaluation

Formal benchmarks (MMLU, ARC, etc.) are planned for a future release as training data and scale increase.

🏢 About Cortiqa

Falin-300M was designed, engineered, and pre-trained from scratch by Cortiqa, focusing on building sovereign, ultra-fast, and resource-efficient AI architectures for India and the global developer ecosystem.

• Developer: Cortiqa
• Architecture: Menothus
• Release Version: v0.1-Alpha

📄 License

This model is released under the Cortiqa Falin Non-Commercial License. Free for research, evaluation, and non-commercial fine-tuning. Commercial use requires written permission — see LICENSE.md or contact [email protected].

🔗 Links

• Website: cortiqa.co
• GitHub: Cortiqas
• X: @CortiqaAI