CoreX_v0.1

license: apache-2.0 language:

en

Model Card for CoreX v0.1

CoreX v0.1 is a lightweight, decoder-only transformer built by Nexizan Company. It is designed to run efficiently on low-resource systems (~7 GB RAM) while supporting offline AI assistants, coding tutors, and sandbox experiments.

Model Details Model Description

Developed by: Nexizan Company

Funded by : Self-funded

Shared by : Nexizan inc CoreX team ( Faisal - LitRush )

Model type: Causal LM (transformer, decoder-only)

Language(s): English

License: Apache-2.0

Finetuned from model : None (trained from scratch)

Model Sources

Repository: to be added

Paper: N/A

Demo: Local CLI via chat_interface.py

Uses Direct Use

Chat-based assistant (offline/terminal)

Text generation and summarization

Code and math Q&A

Educational or personal projects

Downstream Use

Domain-specific fine-tuning (education, productivity, private tools)

Integration into offline AI platforms (e.g., NexIN prototype)

Out-of-Scope Use

Medical, financial, or legal advice

Safety-critical or autonomous systems

Content generation without moderation

Bias, Risks, and Limitations

Limited training size (~9.2M tokens) → restricted knowledge

Biases from dataset may appear in responses

Non-English performance is weak

Risk of hallucinations or unsafe generations

Recommendations

Use a moderation/filtering layer in deployment

Fine-tune with curated, domain-specific datasets

Always keep a human-in-the-loop for sensitive applications

How to Get Started

Run the interactive chat interface:

python chat_interface.py

Or load directly in Python:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("path/to/corex_tok.model") model = AutoModelForCausalLM.from_pretrained("path/to/final_model.pt")

inputs = tokenizer("Hello CoreX!", return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=50) print(tokenizer.decode(outputs[0]))

Training Details Training Data

Samples: 34,559

Tokens: ~9.2M

Avg length: ~266 tokens

Max length: 1024

Tokenizer: SentencePiece unigram, vocab=32,000

Preprocessing

Unicode normalization

Special tokens (<pad>, <unk>, <s>, </s>)

Deduplication and filtering

Training Hyperparameters

Regime: Mixed precision (CPU/GPU optimized)

Hidden size: 512

Layers: 8

Attention heads: 8 (2 KV heads)

Intermediate size: 1365 (SwiGLU)

Max positions: 2048

Learning rate: 5e-4 (cosine decay, warmup 1k steps)

Optimizer: AdamW (β1=0.9, β2=0.95, wd=0.1)

Batch size: 2 (effective 32 with accumulation)

Steps: 50,000

Speeds, Sizes, Times

Parameters: ~54.8M

Checkpoint size: ~220MB

Hardware target: 7 GB RAM systems

Evaluation Testing Data

Held-out samples from training corpus

Factors

Conversational text, code snippets, math expressions

Metrics

Perplexity (PPL), loss

Results

Training loss decreased steadily

Early tests show coherent text and code generation

Summary

CoreX v0.1 achieves usable fluency for small-scale tasks. It is not comparable to large LLMs, but excels at lightweight, private, offline usage.

Model Examination

Architecture: 8-layer decoder, RoPE, SwiGLU, RMSNorm, GQA

Tokenizer verified (32k vocab, unigram SentencePiece)

Environmental Impact

Hardware Type: Consumer GPU/CPU

Training Time: Several days (low resource)

Cloud Provider: None (local)

Carbon Emitted: Minimal (small model)

Technical Specifications Model Architecture and Objective

Decoder-only transformer

RoPE embeddings, SwiGLU MLP, RMSNorm

Grouped Query Attention

Compute Infrastructure

Hardware: ~7 GB RAM system

Software: PyTorch, SentencePiece

Citation

BibTeX:

@misc{corex2025, title={CoreX v0.1: Lightweight Transformer Language Model}, author={Nexizan Company}, year={2025}, license={Apache-2.0} }

APA: Nexizan inc (2025). CoreX v0.1: Lightweight Transformer Language Model.

Glossary

RoPE: Rotary Position Embeddings

SwiGLU: Swish-Gated Linear Unit

RMSNorm: Root Mean Square Norm

GQA: Grouped Query Attention

More Information

CoreX v0.1 is the first milestone in the CoreX series, focused on offline-first, privacy-respecting AI systems. Future versions aim for larger datasets, more parameters, and better reasoning ability.

Model Card Authors

Nexizan inc — CoreX Team

Model Card Contact

N/A