Downloads · 30 days
86
26% of all-time downloads
david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1
HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1 is a text generation model from david-thrower. Use it when you need the model to write or continue text. It is set up for transformers.
A developmental implementation of HelixLM, a Recurrent Heterogeneous Graph Neural Language Model - https://github.com/david-thrower/HelixLM - This is an epoch 1 checkpoint of a downstream base model that, so far, appe…
Downloads · 30 days
86
26% of all-time downloads
All-time downloads
325
Public
Parameters
40.1M
160 MB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors160 MB · 98%
From the Hugging Face model README
A developmental implementation of HelixLM, a Recurrent Heterogeneous Graph Neural Language Model
Input Tokens (B, T)
↓
Embedding [vocab_size × d_model]
↓
+--→ e (preserved for LTI injection — full gradient flow)
↓
┌──────────────────────────────────────────────────────────────────┐
│ Recurrent Block (n_loops × shared graph weights) │
│ ├── Loop Index Embedding (sinusoidal) │
│ ├── HelixGraph: Randomly wired heterogeneous DAG │
│ │ ├── Neural columns with vertical + lateral wiring │
│ │ ├── LinearAttnNode : O(n) causal linear attention │
│ │ ├── FullAttnNode : Causal softmax (periodic, hybrid) │
│ │ ├── SwiGLUNode : Modern gated FFN (silu·gate × up) │
│ │ ├── Mamba2Node : Mamba-2 SSD (long-range, optional) │
│ │ ├── TitansMemoryNode : Surprise-gated persistent memory │
│ │ ├── GateNode : Learned softmax multi-input merge │
│ │ ├── CCA gates : Curriculum Component Activation │
│ │ └── Merge RMSNorm : Normalization after multi-predecessor merges │
│ ├── LTI Injection : Stable recurrent state mixing │
│ │ h_new = A·h + B·e + graph_output (A < 1, learnable decay) │
│ └── ACT Halting : Dynamic per-token depth allocation │
└──────────────────────────────────────────────────────────────────┘
↓
RMSNorm
↓
TiedLMHead (gradient-buffered weight tying with embedding)
↓
Logits / Loss
Hyperpersonalization: Train a small model (from tens of milions of parameters to ~1B parameters) from cold start or from an early - stage partially pre-trained model checkpoint on a personalized corpus, one enriched in it's representation of content relevant to your domain knowledge, before you full fine-tune it on your own data. The model becomes an expert in the one thing generic frontier models don't know anything about: you, your domain knowledge, style, notes, emails, work patterns, and personal voice. A personalized small language model drafts responses to your requests with full awareness of the entire data you encoded in its weights, including the secondary details, tertiary details, and edge cases that vector DB RAG systems usually miss because they are semantically dissimilar to the prompt. This oversight leaves the loose ends that hang you if you don't catch them. Evolve to a better adapted approach.
On-device AI: Efficient inference on CPU/GPU for desktops, laptops, tablets, and mobile. Strong quality per parameter, with optional fully-local (log-free) operation for sensitive use cases.
Economical and ecologically sound AI: The brute force approach that previous generation [used to be] frontier labs use is unsustainable: It burn cash at a rate that is economically unsustainable, causes electricity prices to surge for consumers competing with them for their share of electric grid's limited capacity, and compromises water supplies. Take an exponential cut out of the problem with more efficient AI: A model architecture with better parameter efficinecy, one that can use hybrid or even flat out linear attention only, and that can saturate the model's weights / grock on as little as one epoch.
| Component | What it does | Why it matters |
|---|---|---|
| Neural Columns & Heterogeneous Nodes | Each column holds diverse node types (attention variants, SwiGLU, Mamba-2, gating, optional Titans memory) instead of identical transformer blocks. | Different information pathways for different computations, like biological cortical columns. |
| Recurrent Depth (LTI + ACT) | The same graph weights are looped n_loops times. LTI injection keeps the recurrent state stable. ACT halting dynamically allocates compute per token. | Iterative refinement without parameter growth; easy tokens use 1 loop, hard reasoning uses more. |
| Hybrid Attention | Linear attention (O(n) complexity) in most columns, with periodic full-attention columns for exact retrieval. | Long-context efficiency without losing precise copy/lookup capability. |
| Mamba-2 SSD (optional) | State Space Duality with chunked parallel scan. Auto-activates when ssm_d_state >= 64. | Handles very long-range dependencies efficiently on CPU, CUDA, or MPS. |
| Titans Neural Memory (optional) | Persistent surprise-gated memory via outer-product updates (first column only by default). | Test-time memory that can retain patterns across long documents without growing KV cache. |
| CCA (Curriculum Component Activation) | Attention nodes start gated, gradually open over warmup steps via learned sigmoid gates. | Prevents random attention from drowning FFN signal at initialization. ~30% PPL improvement observed. |
| TiedLMHead with Gradient Buffer | LM head shares embedding weight. A learned linear buffer (init as identity) routes part of the gradient to prevent ~3× embedding gradient overload. | Halves parameter count (embedding table not duplicated). Safe weight tying without destabilizing training. |
| Modern Primitives | RoPE, SwiGLU, RMSNorm. | Proven SOTA components for convergence and generation quality. |
OpenMythos published a recurrent depth with LTI stability and ACT halting. HelixLM takes that insight and makes it work inside a heterogeneous graph that mimics neural column and random topology connectivity found in biological brains.
| Capability | OpenMythos | HelixLM |
|---|---|---|
| Recurrent depth | ✅ Same block looped | ✅ Same block looped |
| LTI stability | ✅ Spectral radius < 1 | ✅ Spectral radius < 1, configurable init_A (default 1/e) |
| ACT halting | ✅ Dynamic per-token depth | ✅ Dynamic per-token depth |
| Architecture inside loop | Standard transformer block | Heterogeneous random graph |
| Attention | Standard full attention only | Linear + full hybrid |
| Node types | Single (transformer block) | 7+ active types (attention variants, FFN, SSM, gate, neural memory) |
| Positional encoding | Standard learned | RoPE |
| Activation | GELU | SwiGLU |
| Normalization | LayerNorm | RMSNorm |
| Open source | ✅ Yes | ✅ Yes |
| HF integration | ❌ No | ✅ Full PreTrainedModel, AutoModelForCausalLM |
| Weight tying | Standard | Gradient-buffered TiedLMHead |
Cerebros showed that biological random hyperdense vertical and lateral topology of Dense layers could outperform rigid layer stacks. It generated text without attention on small data, but required elaborate integration and clashed with standard model-structure paradigms. HelixLM smoothly integrates that topological insight into a modern, HF-compatible LLM backbone.
Metrics:
>>> from transformers import AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1", trust_remote_code=True, dtype="auto", force_download=True)
config.json: 2.15kB [00:00, 3.09MB/s]
config.json: 2.15kB [00:00, 1.32MB/s]
config.py: 15.6kB [00:00, 10.6MB/s]
config.json: 2.15kB [00:00, 4.14MB/s]
hf_model.py: 18.4kB [00:00, 45.2MB/s]
model.safetensors: 100%|█████████████████████| 160M/160M [00:10<00:00, 14.8MB/s]
[HelixLM] Global RNG seed set to 42 (set HelixConfig(seed=None) to disable auto-seeding)
Loading weights: 100%|███████████████████████| 78/78 [00:00<00:00, 11643.80it/s]
generation_config.json: 100%|███████████████████| 355/355 [00:00<00:00, 884kB/s]
>>> layers = [8, 9, 10, 11, 12, 13, 18, 19, 20, 24, 28, 29, 30, 31, 32, 33, 38,\
39, 40, 44, 48, 49, 50, 51, 52, 53, 58, 59, 60, 64, 67, 68, 69, 70]
>>> import weightwatcher as ww
>>> watcher = ww.WeightWatcher()
>>> details = watcher.analyze(model=model, layers=layers)
>>> summary = watcher.get_summary(details)
>>> summary
{'log_norm': np.float64(2.9583618357476125), 'alpha': np.float64(3.5017526482629573), 'alpha_weighted': np.float64(3.784600552260819), 'log_alpha_norm': np.float64(4.507185545059817), 'log_spectral_norm': np.float64(1.570243957671261), 'stable_rank': np.float64(41.33611414349218)}
>>> details
layer_id name D ... weak_rank_loss xmax xmin
0 8 Linear 0.133501 ... 0 0.819514 0.586496
1 9 Linear 0.130986 ... 1 0.806610 0.591386
2 10 Linear 0.124447 ... 1 0.812267 0.582928
3 11 Linear 0.123478 ... 1 0.801489 0.669532
4 12 Linear 0.174834 ... 1 0.091557 0.054470
5 13 Linear 0.122635 ... 0 0.101244 0.044695
6 18 Linear 0.032087 ... 0 147.171070 27.923805
7 19 Linear 0.042542 ... 0 94.793864 27.706529
8 20 Linear 0.068676 ... 0 104.940972 31.728804
9 24 Linear 0.081954 ... 1 48.504587 0.067334
10 28 Linear 0.051617 ... 3 124.385294 0.035137
11 29 Linear 0.042766 ... 4 181.505814 0.030283
12 30 Linear 0.082598 ... 5 35.948817 0.095789
13 31 Linear 0.046608 ... 4 51.369828 17.504687
14 32 Linear 0.091659 ... 0 58.859000 1.058015
15 33 Linear 0.110888 ... 1 64.008994 1.678953
16 38 Linear 0.068318 ... 0 32.099117 3.374599
17 39 Linear 0.069430 ... 0 30.437727 3.097338
18 40 Linear 0.047503 ... 0 120.902677 0.530989
19 44 Linear 0.076068 ... 1 22.695975 0.091167
20 48 Linear 0.075791 ... 4 458.548402 3.472195
21 49 Linear 0.066970 ... 3 252.430063 1.100946
22 50 Linear 0.079772 ... 7 67.926923 0.108073
23 51 Linear 0.064862 ... 2 77.686899 5.453615
24 52 Linear 0.073012 ... 0 68.709129 0.829720
25 53 Linear 0.068787 ... 0 61.817121 0.694748
26 58 Linear 0.052428 ... 0 352.342393 59.507195
27 59 Linear 0.066766 ... 0 60.873335 18.667454
28 60 Linear 0.034649 ... 0 181.075791 24.358505
29 64 Linear 0.085383 ... 1 146.261497 53.585340
30 67 Linear 0.047679 ... 0 208.907095 0.918829
31 68 Linear 0.067708 ... 0 89.022724 1.637560
32 69 Linear 0.055705 ... 0 269.097498 10.898044
33 70 Linear 0.073438 ... 0 277.911039 44.351509
[34 rows x 32 columns]
Summary Metrics Analysis
Metric Value Interpretation
─────────────────────────────────────────────────────────────
log_norm 2.96 Overall weight magnitude - moderate
alpha 3.50 Power-law exponent (unweighted avg)
alpha_weighted 3.78 Weighted by layer importance
log_spectral_norm 1.57 Log of largest singular value
stable_rank 41.34 Effective rank of weight matrices
Base Chinchilla Formula: 20 tokens per parameter
We trained on 1.5B tokens, so at first pass:
However, let's adjust for the disproportionate embedding table (As I am a 41M param model, I have a disproportionate percentage of my weights in my embeddign table, compared to a trypical 150M - 1T parameter model, so lets't adjust for that) ...
Embedding Table Adjustment
GPT-2 Tokenizer specs:
Parameter breakdown for 40.1M model:
| Component | Calculation | Parameters |
|---|---|---|
| Total Parameters | - | 40.1M |
| Token embeddings | 50,257 × 512 | 25.7M |
| Position embeddings | ~1024 × 512 | ~0.5M |
| Attention and FFN layers | Remainder | ~13.9M |
Effective active parameters: ~13.9M Chinchilla optimal for active params: 13.9M × 20 = 278M tokens My actual training corpus: 1.5B tokens
| Metric | Value |
|---|---|
| Tokens per active param | 107.9 |
| vs. Chinchilla optimal | 5.39 × over chinchilla optimal |
TLDR, we've grocked to full saturation on 1 epoch, on 5.39 X chinchilla optimal with only the earlyest signs of destabilization, and rank collapse not yet occuring from over-training.
<!-- ml-intern-provenance -->This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.
from transformers import pipeline
pipe = pipeline("text-generation", model='david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1', trust_remote_code=True)
x = pipe('In the galaxy there are billions and billions of stars. Some of these stars help us pinpoint where other galaxies are.', max_new_tokens=100, temperature=0.6, top_k=40, top_p=0.95, repetition_penalty=1.2)
print(x)
# -> 'In the galaxy there are billions and billions of stars. Some of these stars help us pinpoint where other galaxies are. We will first find a new catalog of the Andromeda Galaxy, which is also known as the companion star. The Milky Way is the most massive disk in the Universe, with several million light sources to search for ... that can be found by the Crab Nebula: ...
# Fluent and on topic text, albeit not [quite] factually on - point.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = 'david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.
This project is open-source under a modified Apache 2.0 license. See Cerebros-License for full terms.