ARKA-v3

⚡ ARKA-v3

A Custom 505M-Parameter Language Model

Created and developed by Abhay Kumar Rudrapaul

ARKA-v3 is a decoder-only Transformer language-model architecture using Grouped-Query Attention (GQA), RoPE, RMSNorm, SwiGLU and tied token/output embeddings. This recreated package contains fresh randomly initialized weights; it must be pretrained before meaningful text generation.

Architecture

The maintained Qwen2 implementation primitives in Hugging Face Transformers are used as the execution backend because their tensor layout matches this architecture. ARKA-v3 has its own model type, configuration and AutoModel registration.

Installation

pip install -U "torch>=2.4" "transformers>=4.53" \
  "huggingface_hub>=0.27" "safetensors>=0.4.5" "accelerate>=1.2"

Load

Review the repository's custom Python files before enabling remote code. For reproducible use, pin revision to a trusted commit hash.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Abhayn01/ARKA-v3"
revision = "PASTE_TRUSTED_COMMIT_HASH"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id, revision=revision, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    revision=revision,
    trust_remote_code=True,
    dtype=torch.float16,
)

Important status

The recreated checkpoint is randomly initialized. It preserves the exact architecture and parameter count, but it does not reconstruct previously deleted learned weights or knowledge. Pretraining is required.

Creator

ARKA is an independent language-model development project created and developed by Abhay Kumar Rudrapaul.