gemma3-concepts-1b-c128-1bt

gemma3-concepts-1b-c128-1bt

Gemma-3-1B with a grafted shared depth-recurrent concept memory (C=128). Research release from MrCogito experiment E16b: trained for ~1B tokens at seq 4096 on a long-document mix.

Claim level: mechanism success — concepts are causally used (ablating them raises long-range CE a lot). Generation quality is not the claim: this is a research base LM, not instruction-tuned, and open-ended text is often weak or repetitive. Do not treat samples as evidence of a strong generative model. Also not a GLUE / chat-arena SOTA claim.

Table of contents

1. Model summary
2. How to use
3. Uses
4. Limitations
5. Architecture
6. Training
7. Evaluation
8. Citation
9. License

---

Model summary

What it is

• A block-recurrent causal LM: tokens run through Gemma in K=512 blocks; after each global layer the model reads/writes a shared concept state z ∈ R^{C×H}.
• Evidence that those concepts become necessary for next-token loss at long positions (ablating them raises CE a lot).

What it is not

• Not a strong generator. Continuations can be dull, repetitive, or locally fluent without being useful. The published success is concept use under ablation, not sample quality.
• Not chat / instruction-tuned — use continuation prompts, not “system” roles.
• Not loadable as plain AutoModelForCausalLM without MrCogito.
• Not evaluated here on STS-B / GLUE / arena — those probes are listed as not run.

---

How to use

git clone https://github.com/ksopyla/MrCogito.git
cd MrCogito
uv sync

Accept the Gemma license on Hugging Face if prompted.

import torch
from transformers import AutoTokenizer
from nn.backbone_concept_lm import BackboneConceptLM

repo_id = "ksopyla/gemma3-concepts-1b-c128-1bt"

if torch.cuda.is_available():
    device, dtype = "cuda", torch.bfloat16
elif getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
    device, dtype = "mps", torch.float32   # fp16 often NaNs on Apple MPS
else:
    device, dtype = "cpu", torch.float32

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = BackboneConceptLM.from_pretrained(repo_id, torch_dtype=dtype).to(device).eval()

prompt = "Once upon a time, in a quiet library at the edge of the city,"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )
print(tokenizer.decode(out[0], skip_special_tokens=True))

Research API — concept ablation (same generate / forward path):

# concept_mode: "real" | "zero" | "shuffle" | "static" | "one_block"
out = model.generate(**inputs, max_new_tokens=32, concept_mode="shuffle")

Playground notebook: playground/e16b_generation_playground.ipynb.

---

Uses

Direct use

• Reproduce E16b Tier-1 concept geometry and causal-ablation numbers.
• Ablation studies showing concepts are used (concept_mode=…).
• Starting checkpoint for longer-context or semantic-probe follow-ups.

Out of scope

• Showcasing high-quality generation or shipping a writing/chat assistant.
• Production assistants, customer-facing chat, or safety-critical decisions.
• Claiming parity with Gemma IT / chat models.
• Multilingual deployment (training mix is English-centric).

---

Limitations

1. Generation is restricted / weak for product use. Prefer this checkpoint for concept ablations, geometry, and mechanism studies — not demos of “good writing.” Formal generation vibe-check metrics are not published; qualitative play only.
2. Custom code — BackboneConceptLM lives in MrCogito; clone the repo.
3. Base LM, not IT — will not “follow instructions” like a chat model.
4. Slow generate — no KV cache in the shipped loop; keep max_new_tokens small (≈64–128 on laptop GPU/MPS).
5. Apple MPS — use float32.
6. Compound bet — long context + long-doc mix + Muon + 1B tokens were changed together; factor isolation is open.
7. Gemma license applies to derived weights.

---

Architecture

Vision (why concepts at all)

MrCogito is a research bet that reasoning should eventually happen in a compact concept space, not only as next-token prediction over text:

long input (text today; audio/vision later)
    → compress into C latent concept vectors   (C ≪ N)
    → refine / reason in concept space         (recursion, depth, …)
    → decode / crystallise back to tokens      (training signal + human interface)

The belief behind that stack:

1. Semantic bandwidth. A text token carries on the order of ~15 bits; a concept vector in a large hidden space carries far more continuous state per step. That gap is the argument for latent reasoning and (later) latent agent-to-agent channels.
2. Long context as a consequence of the bottleneck. If attention is concept↔token (roughly O(C·N) with C ≪ N) rather than full O(N²) self-attention, million-token windows become a design goal of the architecture, not a RoPE trick bolted onto a dense transformer.
3. Decode is crystallisation, not the thinking substrate. Tokens are how we train and how humans read answers; the interesting state should live in concepts.
4. One step at a time. Multimodality and multi-agent concept exchange are enabled by a working concept core — they are not Stage-0 goals. Near-term priority is proving concepts actually carry usable state.

Full write-up: vision and goals · living agenda · ai.ksopyla.com.

Where this release sits

E16b is not the full encode→recursive-reason→decode stack yet. It is a Gemma graft: keep a strong frozen decoder LM, add a small shared concept workspace, and ask a sharper mechanism question —

when next-token loss can already be solved from local Gemma context, do concepts ever become causally necessary?

Earlier short-context runs (E10–E16a, seq 2048, ≤100M tokens) often kept concept geometry healthy (high RankMe) while beyond-local ablation ΔCE stayed near zero. E16b changes the operating regime (seq 4096, long-document mix, Muon, 1B tokens) and clears that gate: ablating concepts at long positions hurts a lot. That validates one path for causally load-bearing concept memory — a platform to build recursive / latent-reasoning ideas on — without closing other routes (from-scratch AR, diffusion, etc.).

This model’s forward pass

input_ids [B, L]
    │
    ▼  consume in blocks of K=512
┌───────────────────────────────────────────────┐
│  Gemma-3-1B decoder (frozen) + LoRA adapters  │
│                                               │
│  local / sliding layers  →  standard Gemma    │
│  global (full-attn) layers:                   │
│      1. run original Gemma layer              │
│      2. READ  concepts → tokens (gated xattn) │
│      3. WRITE tokens → concepts (BiXT update) │
└───────────────────────────────────────────────┘
    │
    ▼
shared concept state  z [B, C=128, H=1152]
    │  carried across blocks (recurrent memory)
    ▼
LM head @ last position → next-token logits

Mechanics in short:

• Block recurrence. Sequence length 4096 ⇒ 8 blocks of K=512. Concepts are the only compact state that must survive from block to block.
• Shared depth-recurrent I/O. The same write head is applied at each of Gemma’s global layers (depth-tied), with small tanh gates (init 0.01) and RMSNorm on the concept side of the read — so the workspace can accumulate multi-block content without free-riding on local Gemma context alone.
• Read. After a global layer, tokens attend to the concept set (no RoPE on the concept side — the memory is a position-free set).
• Write. BiXT-style bidirectional update mixes the current block’s tokens into z, gated so early training can stay near the frozen backbone.
• Ablations. zero / shuffle / static / one_block break the memory on purpose; large ΔCE means the intact model was using it.

Code: nn/backbone_concept_lm.py.

---

Training

Data

Training used the frozen mix recipe e16b_long_4k_v1 — a long-document-heavy causal-LM mix at seq length 4096, designed so enough documents span multiple concept blocks (target: ≥40% of docs >2k tokens, ≥20% >4k).

Mix proportions (sampling weights)

Weights are document-sampling mixture weights from the recipe (sum = 1.0). They are not guaranteed equal to final non-padding token mass after truncation, but they define the intended corpus balance.

Long vs fluency split (by weight): long/coherent tier ≈ FinePDFs 30% + PG19 18% + Wikipedia 12% = 60%; fluency/code/math ≈ DCLM 15% + FineWeb-Edu 10% + Stack-Edu 10% + FineMath 5% = 40%.

Design notes

• Why this mix: short 2K fluency-heavy mixes let Gemma satisfy next-token CE from local context alone. E16b deliberately ups multi-block pressure with PDFs + books at 4K (8×512 concept blocks).
• No cross-doc packing: each training example is one document (truncated to 4096), so concept carry across blocks is within-document, not an artifact of stitching.
• Deferred sources (not in this mix): peS2o, Nemotron, ProLong (format / gated / MDS constraints at the time of the recipe).
• Wikipedia license: CC-BY-SA — attribute when redistributing Wikipedia-derived text.
• Eval reuse: Tier-1 concept analysis also sampled from the same pretokenized e16b_long_4k_v1 Gemma-4K manifest (see Evaluation).

Tokenizer

Load either from this repo or from the backbone id — they match the training tokenizer.

Procedure

Compute

---

Evaluation

Protocol (Tier-1, 2026-07-25): pretokenized e16b_long_4k_v1, seq 4096, buckets 1024,2048, 24 docs × 2 seeds.

Summary: within-sample RankMe stays high (~101/128) and beyond-local concept ablations clear the pre-registered ≥0.01 gate by a large margin (min(Δstatic, Δshuffle)_beyond ≈ 2.35 nats).

Concept geometry

Causal concept use (ΔCE, nats)

Higher Δ ⇒ ablating concepts hurts next-token CE (model was using them).

Registered gate

Same-family context (not length-matched)

Not measured / not claimed

---

Citation

@misc{sopyla2026gemma3-concepts-1b-c128-1bt,
  author       = {Sopyła, Krzysztof},
  title        = {gemma3-concepts-1b-c128-1bt (E16b)},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/ksopyla/gemma3-concepts-1b-c128-1bt},
  note         = {Gemma-3-1B + C=128 shared-depth concepts; 1B-token long-ctx Muon}
}

Please also respect the Gemma 3 terms / base model card.

---

License

license: gemma — weights derived from google/gemma-3-1b-pt.

Model card contact

Krzysztof Sopyła — https://ai.ksopyla.com · https://github.com/ksopyla