EidosFormer

EidosFormer — Memory-Native Transformer Architecture

<div align="center">

A novel transformer architecture that unifies episodic, semantic, and working memory systems for long-context understanding.

🤗 Hugging Face • 📄 Model Card

</div>

---

📋 Overview

EidosFormer (pronounced eye-dos-former) is a memory-augmented language model that integrates four independently validated research mechanisms into a unified, end-to-end trainable architecture. It addresses the long-context problem by maintaining three parallel memory subsystems:

Key Innovations

1. Differentiable Consolidation: Episodic memories automatically compress into semantic representations using dual objectives (reconstruction loss + attention-preservation loss)
2. Ebbinghaus-Style Recall Reinforcement: Frequently retrieved items receive memory boost, preventing catastrophic forgetting in long sequences
3. Multi-Cross-Attention Gating: First-layer cross-attention connects all memory streams; subsequent layers operate on unified representations
4. Modern Architecture Stack: Grouped Query Attention (GQA), RoPE with NTK/Instruct scaling, RMSNorm, SwiGLU feedforward — proven Llama-family components

🏗️ Architecture

Input Tokens
    │
    ├─► Token Embedding + Temporal Encoding (wall-clock aware)
    │
    ▼
┌───────────────────────┐
│  Episodic Memory Store │ ← FAISS ANN index, STE kNN retrieval
└───────────────────────┘
         │ retrieved_kv
         ▼
┌───────────────────────┐
│   Semantic Patterns    │ ← EMA-updated centroids (consolidated from episodic)
└───────────────────────┘
         │ merged with kv
         ▼
┌───────────────────────┐
│  Memory Cross-Attn     │ ← Layer 0: connects all memory streams
└───────────────────────┘
         │ unified representation
         ▼
┌───────────────────────────┐
│  EidosBlock × N layers    │ ← CausalSelfAttention + SwiGLU blocks
└───────────────────────────┘
         │
         ▼
┌───────────────────────┐
│  MemoryWriteHead        │ ← gated write-back to episodic store
└───────────────────────┘

Side channels (trainer-driven, no-grad):
  - model.append_episodic(x)     → FAISS vector store
  - model.write_to_semantic()    → soft-gated EMA update
  - model.apply_consolidation()  → weakest-slot compression
  - model.resize_semantic(n)     → memory curriculum (grow/shrink)

🚀 Quick Start

Installation

pip install transformers torch faiss-cpu accelerate sentencepiece

Loading the Model

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Himan-de/EidosFormer",
    trust_remote_code=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "Himan-de/EidosFormer",
    trust_remote_code=True
)

# Generate text
inputs = tokenizer("The future of AI is driven by memory systems that enable:", return_tensors="pt")
outputs = model.generate(**inputs.to(model.device), max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using with the Hugging Face Hub

# Access via huggingface-cli
huggingface-cli login  # Enter your token when prompted
huggingface-cli download Himan-de/EidosFormer --local-dir ./eidosformer

📊 Model Specifications

🧪 Training Details

• Framework: PyTorch with custom memory backends
• Optimizer: AdamW with cosine annealing and warmup
• Checkpoint Steps: Trained from step 1000 through ~98,000 steps
• Final Checkpoint: 1b_final.pt (recommended for inference)
• Intermediate Checkpoints: Available (1b_ckpt_*pt) for reproducibility

🔬 Research Foundations

EidosFormer synthesizes mechanisms from:

1. Memorizing Transformers (Wu et al., 2022) — differentiable memory with kNN retrieval
2. Compressive Transformers (Rae et al., 2019) — multi-resolution episodic-to-semantic consolidation
3. MemoryBank (Zhong et al., 2023) — recency-and-frequency-based recall dynamics

📝 Citation

If you use EidosFormer in your research:

@misc{eidosformer2026,
  title={EidosFormer: Memory-Native Transformers with Multi-Store Consolidation},
  author={Dixit, Himanshu},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/Himan-de/EidosFormer}
}

📄 License

Proprietary — Contact Author for Commercial Use

This model is released under a custom license. For commercial usage, licensing inquiries, or collaboration opportunities, please reach out via the discussions tab on Hugging Face.

For academic research use, please request access through the gated repo link above.

⚠️ Known Limitations

• Episodic memory retrieval quality depends on FAISS index construction (flat L2 for best accuracy, HNSW for speed)
• Semantic consolidation is gradient-free during inference; quality depends on training-time consolidation ratio
• Memory store capacity grows with sequence length; consider pruning strategies for production deployments

🙏 Acknowledgments

• Built upon the excellent Transformers library by Hugging Face
• FAISS vector search library (Meta AI) for efficient kNN retrieval
• Inspired by: Wu et al. (Memorizing Transformers), Rae et al. (Compressive Transformers), Zhong et al. (MemoryBank)

📬 Contact

For questions, collaboration, or licensing inquiries:

• Open a Discussion
• Visit the GitHub repo

---

<p align="center"> <i>Memory is the foundation of intelligence.</i><br> — EidosFormer Research Team, 2026 </p>