luck-spark-l-100m-base

Dr. Zeon

🏛️ THE ARCHITECT v0.1

⚠️ Important Notice: Running the Model Locally (Ollama / LM Studio)

Why isn't this model supported in Ollama or LM Studio?

Unlike standard models (such as Llama, Mistral, or Qwen), The Architect is built on a completely custom, from-scratch PyTorch architecture. It features experimental mathematical layers that do not exist in standard Transformers, specifically:

Holographic State Memory (HSM)
Neuro-Symbolic Gating (NSRG)

Because these custom operations are not part of the standard Transformer vocabulary, C++ based inference engines (like llama.cpp, which powers Ollama and LM Studio) cannot natively understand or execute them. Converting this model to GGUF format for Ollama would require writing custom C++ kernels for the HSM and NSRG layers, which is a complex engineering task planned for the future roadmap.

How can you test and use this model?

To ensure everyone can easily run and test this model locally without dealing with complex code, we provide a one-click Python Web UI. It creates a ChatGPT-like interface right in your browser.

Download the app.py and run.bat (Windows) or run.sh (Linux/Mac) scripts provided in this repository.
Ensure you have Python installed on your system.
Double-click the run.bat file. It will automatically install the required libraries (torch, gradio, tokenizers) and launch a local web server.
Your browser will automatically open a clean, interactive UI where you can chat with the model immediately.

Like this Terminal:

@user  cd Luck-spark-100m-base

~/Luck-spark-100m-base  @user  python3 app.py

Model indiriliyor ve yükleniyor, lütfen bekleyin... Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. Model hazır!

• Running on local URL: http://1xx.x....
• To create a public link, set share=True in launch().

No C++ compilation or terminal knowledge required!

A Boutique, From-Scratch Turkish Language Model

THE ARCHITECT is an independently developed, approximately 90M-parameter experimental language model designed and trained entirely from scratch with PyTorch.

The project explores a simple but ambitious research question:

How much language modeling capability can be obtained from a relatively small model through architectural experimentation, aggressive training, and carefully selected Turkish data?

Rather than attempting to compete directly with billion-parameter commercial or open-weight foundation models, THE ARCHITECT is intended as a research-oriented boutique language model: compact, experimental, transparent, and built to investigate alternative architectural ideas.

---

🧭 Project Philosophy

Modern language models increasingly rely on extremely large parameter counts and enormous computational budgets.

THE ARCHITECT takes a different approach.

The project investigates whether a comparatively small model can obtain useful language representations through:

• substantial pre-training relative to its parameter count,
• a Turkish-focused corpus,
• a custom tokenizer,
• architectural experimentation,
• parameter sharing,
• and experimental memory/gating mechanisms.

This project should not be interpreted as evidence that small models universally outperform large models. Instead, the goal is to investigate the efficiency frontier of language modeling at a small scale.

The model is therefore best understood as an independent research experiment, rather than a commercial competitor to systems such as GPT-class or Llama-class large language models.

---

🧠 Architecture

THE ARCHITECT is implemented directly in PyTorch rather than being assembled from a standard pretrained Transformer architecture.

The overall structure follows a decoder-only autoregressive language-modeling paradigm, while introducing experimental components inside each block.

High-level block

Input Tokens
     │
     ▼
Token Embedding
     │
     ▼
┌─────────────────────────────┐
│      ARCHITECT BLOCK        │
│                             │
│  Causal Self-Attention      │
│           │                 │
│       Residual              │
│           │                 │
│       LayerNorm             │
│           │                 │
│         SwiGLU              │
│           │                 │
│       Residual              │
│           │                 │
│       HSM + NSRG            │
│           │                 │
│       LayerNorm             │
└─────────────────────────────┘
     │
     ▼
   × 8 Layers
     │
     ▼
Final LayerNorm
     │
     ▼
Weight-Tied LM Head
     │
     ▼
Token Logits

---

🔬 Core Components

1. Causal Multi-Head Self-Attention

THE ARCHITECT uses causal multi-head self-attention.

A triangular causal mask prevents a token from attending to future tokens during autoregressive training.

Formally, positions satisfy:

[ M_{ij} = \begin{cases} 0 & j \leq i
-\infty & j > i \end{cases} ]

This ensures that prediction at position i cannot directly access tokens occurring after position i.

Configuration

• 16 attention heads
• 64 dimensions per head
• 768-dimensional hidden representation
• 128-token context

---

2. SwiGLU Feed-Forward Network

The standard ReLU-style feed-forward layer is replaced by SwiGLU.

The implementation uses three linear projections:

[ \text{SwiGLU}(x)

W_2 \left( \text{SiLU}(W_1x) \odot W_3x \right) ]

The intermediate dimension is:

2048

SwiGLU-style feed-forward networks are widely used in modern language-model architectures and provide an efficient nonlinear transformation.

---

🧬 3. Holographic State Memory (HSM)

Holographic State Memory (HSM) is one of the experimental components of THE ARCHITECT.

HSM attempts to introduce a learned state representation that interacts with the current hidden representation.

Conceptually:

Current Representation
        │
        ├──────────────┐
        │              │
        ▼              ▼
      Current       Learned
      State          State
        │              │
        └──────┬───────┘
               ▼
          State Binder
               │
               ▼
        New Representation

The current implementation should be interpreted carefully.

Despite its name, HSM does not yet constitute a fully recurrent sequence-level memory mechanism. The current implementation behaves more like an experimental dynamic state/bias transformation inside each Architect block.

Layer normalization is used to stabilize its interaction with the main representation.

This distinction is intentional: the project documents what the mechanism currently does, rather than claiming capabilities that have not yet been demonstrated.

---

🧩 4. Neuro-Symbolic Gating (NSRG)

Neuro-Symbolic Gating (NSRG) is an experimental gating mechanism designed around four learned information-handling operations:

The mechanism produces four normalized gate values through a softmax operation.

Conceptually:

                  Hidden State
                       │
                       ▼
                  Gate Network
                       │
                 Softmax (4)
                       │
        ┌──────────────┼──────────────┐
        ▼              ▼              ▼
     Preserve         Erase          Bind
                       │
                       ▼
                 Contradiction
                       │
                       ▼
                State Update

NSRG is experimental and should not be interpreted as a formally verified symbolic reasoning system.

The terminology describes the intended information-routing behavior of the mechanism, not a claim that the model possesses human-like symbolic reasoning.

---

🔗 Weight Tying

THE ARCHITECT shares the weights of:

Token Embedding
        ↕
LM Head

This reduces parameter redundancy and improves parameter efficiency.

The embedding matrix is therefore reused when projecting hidden representations back into vocabulary logits.

---

📐 Model Configuration

---

🗣️ Training Data

THE ARCHITECT was designed primarily around Turkish-language modeling.

The training corpus contains approximately:

• ~850 MB of text
• ~200–250M tokens
• 700K+ Turkish Wikipedia articles after filtering
• conversational data from HuggingFaceH4/no_robots
• Turkish instruction-oriented data

The corpus combines informational, conversational, and instruction-style text.

However, the current model is a base/pretrained language model, not an instruction-tuned assistant.

---

🔤 Tokenization

THE ARCHITECT uses a custom 32,000-token BPE tokenizer trained with the Hugging Face tokenizers library.

A TemplateProcessing post-processor was incorporated to improve handling of tokenization boundaries and special-token formatting.

The tokenizer was developed specifically for the project's Turkish-focused corpus rather than adopting the tokenizer of an existing large language model.

---

⚙️ Pre-Training

Training was performed using Kaggle GPU infrastructure, primarily Tesla T4 GPUs.

Optimization

The training system uses mixed precision through PyTorch AMP to reduce memory consumption and improve throughput.

---

📈 Training Progress

The initial training objective consisted of approximately 570,000 steps, corresponding to one planned epoch over the prepared training configuration.

During training, the observed loss decreased substantially from its initial values.

Approximate observed trajectory:

Initial loss
     │
     │  ~11.3
     │
     ▼
   ~7–8
     │
     ▼
   ~6
     │
     ▼
  ~4.5–5.0

At the time of this model-card revision, more than 420,000 training steps had been completed.

Important: Training loss alone is not sufficient to establish model quality. A proper evaluation suite and held-out validation set are required before making quantitative claims about linguistic performance or reasoning capability.

---

💾 Checkpoint & MLOps System

The training pipeline was designed to survive temporary GPU environments such as Kaggle sessions.

Every 10,000 training steps, the system creates a checkpoint containing model and training state.

Conceptually:

Training
   │
   ├── Step 10,000 ──► Checkpoint
   ├── Step 20,000 ──► Checkpoint
   ├── Step 30,000 ──► Checkpoint
   │
   └── ...

Checkpoints contain:

• model state_dict
• optimizer state
• scheduler state
• global training step
• associated tokenizer/configuration information where applicable

The checkpoint is automatically uploaded to the Hugging Face Hub.

This allows training to resume after:

• Kaggle session termination
• GPU quota exhaustion
• runtime crashes
• manual interruption
• environment resets

The intended workflow is:

Kaggle
  │
  ▼
Train
  │
  ▼
Checkpoint
  │
  ▼
Hugging Face Hub
  │
  ▼
New Kaggle Session
  │
  ▼
Download checkpoint
  │
  ▼
Restore model + optimizer + scheduler
  │
  ▼
Continue training

---

🤗 Hugging Face Integration

The project uses the Hugging Face Hub as both:

1. a model distribution platform
2. a remote checkpoint storage mechanism

Training checkpoints are uploaded automatically during training.

After training, the model is additionally converted into a standard Hugging Face-compatible distribution containing the required configuration, model implementation and serialized weights.

The long-term goal is to make THE ARCHITECT usable through standard Hugging Face tooling rather than requiring the original training notebook.

---

🧪 Current Capabilities

THE ARCHITECT is currently a base language model.

It can perform autoregressive text continuation based on patterns learned during pre-training.

The model has demonstrated learning of Turkish linguistic patterns during training.

However, it should not currently be treated as a general-purpose conversational assistant.

For example, a prompt such as:

Türkiye'nin başkenti

may cause the model to continue the text in an encyclopedic or corpus-like style rather than responding conversationally.

This is expected behavior for a base model.

---

⚠️ Known Limitations

This section is intentionally explicit.

1. Base Model, Not a Chat Model

THE ARCHITECT has not yet undergone a dedicated instruction-tuning or preference-optimization stage.

It should therefore not be expected to behave like ChatGPT or other instruction-following assistants.

Planned:

Pretraining
     ↓
SFT
     ↓
Instruction-following model

---

2. No Positional Encoding

The current architecture does not yet implement RoPE or another explicit positional embedding mechanism.

This is a significant architectural limitation.

Positional information is an important component of Transformer language models, and its absence may limit sequence modeling capability.

Adding a positional mechanism is therefore a major item on the roadmap.

---

3. Short Context

The current context length is:

128 tokens

This is extremely short compared with modern language models.

Long documents, conversations, and multi-paragraph reasoning cannot currently be represented effectively within a single context window.

---

4. No KV Cache

Inference currently recomputes attention over the sequence for each generated token.

A KV cache has not yet been implemented.

Consequently, autoregressive generation is substantially less efficient than it could be.

---

5. Experimental HSM

The name "Holographic State Memory" should not be interpreted as evidence of a sophisticated long-term memory system.

The current mechanism does not yet provide a fully persistent sequence-level memory.

It is an experimental architectural component whose usefulness requires further controlled evaluation.

---

6. Experimental NSRG

NSRG is an architectural hypothesis rather than a proven neuro-symbolic reasoning system.

The four gates are learned neural operations.

The terms:

• Preserve
• Erase
• Bind
• Contradiction

describe the intended information-routing semantics.

They do not imply formal symbolic reasoning, logical theorem proving, or guaranteed contradiction detection.

---

7. No Comprehensive Benchmark Yet

The project currently does not provide a comprehensive benchmark suite comparing THE ARCHITECT against established Turkish language models.

Therefore, claims about:

• intelligence,
• reasoning,
• factual knowledge,
• Turkish superiority,
• parameter efficiency,
• or general language-model performance

should be considered research hypotheses rather than established results.

Future evaluations will address this limitation.

---

🧭 Roadmap

Phase 1 — Pretraining

• Custom tokenizer
• From-scratch model implementation
• Causal attention
• SwiGLU
• HSM
• NSRG
• Weight tying
• Mixed precision training
• Automated checkpointing
• Hugging Face checkpoint integration
• Complete ~570K-step training run

---

Phase 2 — Architectural Improvements

• Add RoPE
• Evaluate alternative positional representations
• Improve HSM state propagation
• Further stabilize NSRG
• Investigate sequence-level memory
• Implement KV cache
• Optimize inference

---

Phase 3 — Instruction Tuning

After completion of base-model pretraining:

THE ARCHITECT Base
        │
        ▼
Turkish Instruction Dataset
        │
        ▼
Supervised Fine-Tuning
        │
        ▼
THE ARCHITECT Instruct

The goal is to transform the base language model into a model capable of following natural-language instructions.

---

Phase 4 — Evaluation

A dedicated evaluation suite is planned, including:

• Turkish language understanding
• Turkish text generation
• reading comprehension
• instruction following
• factual knowledge
• basic reasoning
• perplexity
• held-out validation loss
• comparison with similarly sized Turkish/open models

A key goal is to measure whether the architectural experiments provide measurable benefits rather than relying solely on qualitative impressions.

---

🧪 Research Questions

THE ARCHITECT is ultimately an investigation into several questions:

Can a small model learn substantial Turkish linguistic structure?

Can aggressive training compensate for some of the limitations imposed by a small parameter budget?

Can experimental state mechanisms improve representation quality?

Can information-routing gates provide useful behavior without dramatically increasing parameter count?

How much architectural experimentation is worthwhile at the ~100M parameter scale?

These questions remain open.

The project is designed to produce empirical evidence rather than assume that the proposed mechanisms work simply because they sound theoretically promising.

---

📊 Evaluation Philosophy

THE ARCHITECT will prioritize measured results over architectural claims.

Future model releases should ideally report:

The objective is not merely to build an unusual architecture, but to determine whether the unusual components provide measurable value.

---

🏗️ Why "THE ARCHITECT"?

The name reflects the central idea behind the project:

Intelligence is not only a question of how much material is used, but also how that material is organized.

The project therefore treats model architecture as an experimental object rather than assuming that scaling parameters is always the only meaningful direction.

THE ARCHITECT is an attempt to explore that space at a scale accessible to an independent researcher.

---

📜 Project Status

Status: Experimental / Research

Version: v0.1 — Base Model

Training: In progress

Primary language: Turkish

Model size: ~90M parameters

Instruction tuning: Not yet completed

Production readiness: ❌ No

Research/experimentation: ✅ Yes

---

⚖️ Intended Use

THE ARCHITECT is intended for:

• educational research
• language-model experimentation
• Turkish NLP research
• architecture experimentation
• small-model research
• reproducibility experiments
• Hugging Face ecosystem experimentation

It is not currently recommended for:

• production conversational AI
• high-stakes decision making
• factual information retrieval without verification
• medical or legal applications
• autonomous systems
• safety-critical applications

---

🧑‍🔬 Development

THE ARCHITECT is an independently developed project.

It is intentionally developed without the infrastructure or computational resources of a large AI organization.

The project therefore emphasizes:

• transparency,
• reproducibility,
• experimentation,
• documenting failures,
• and distinguishing hypotheses from demonstrated results.

Architectural components that do not work as expected are considered part of the research process rather than hidden from the model documentation.

---

🙏 Acknowledgements

The project makes use of the broader open-source machine-learning ecosystem, including:

• PyTorch
• Hugging Face Hub
• Hugging Face Tokenizers
• Kaggle GPU infrastructure
• publicly available Turkish-language datasets
• HuggingFaceH4/no_robots

This project would not be practical without the open-source ecosystem surrounding modern machine learning.

---

📚 Citation

If you use THE ARCHITECT in research or experimentation, please cite the project as:

@misc{the_architect_2026,
  title        = {THE ARCHITECT v0.1: A Boutique From-Scratch Turkish Language Model},
  author       = {Independent AI Researcher},
  year         = {2026},
  publisher    = {Hugging Face},
  note         = {Experimental 90M-parameter Turkish language model}
}

---

⚠️ Final Note

THE ARCHITECT is deliberately presented as an experiment, not a finished claim about the future of language modeling.

A 90M-parameter model is not expected to match the capabilities of modern multi-billion-parameter foundation models.

The interesting question is narrower:

How far can a carefully designed, aggressively trained, Turkish-focused small model be pushed—and which architectural ideas actually help?

That question is what THE ARCHITECT is being built to investigate.

The experiment is still running.