Downloads · 30 days
17
3% of all-time downloads
david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep3
HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep3 is a text generation model from david-thrower. Use it when you need the model to write or continue text. It is set up for transformers.
- Our first stable base model version at 512 seq len of HelixLM, a Recurrent Heterogeneous Graph Neural Language Model - https://github.com/david-thrower/HelixLM - For a minimal instruct fine tuned model checkpoint fr…
Downloads · 30 days
17
3% of all-time downloads
All-time downloads
653
Public
Parameters
40.1M
160 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors160 MB · 98%
From the Hugging Face model README
HelixLM is a novel model architecture that out-punches a weight class above its own by combining:
Input Tokens (B, T)
↓
Embedding [vocab_size × d_model]
↓
+--→ e (preserved for LTI injection — full gradient flow)
↓
┌──────────────────────────────────────────────────────────────────┐
│ Recurrent Block (n_loops × shared graph weights) │
│ ├── Loop Index Embedding (sinusoidal) │
│ ├── HelixGraph: Randomly wired heterogeneous DAG │
│ │ ├── Neural columns with vertical + lateral wiring │
│ │ ├── LinearAttnNode : O(n) causal linear attention │
│ │ ├── FullAttnNode : Causal softmax (periodic, hybrid) │
│ │ ├── SwiGLUNode : Modern gated FFN (silu·gate × up) │
│ │ ├── Mamba2Node : Mamba-2 SSD (long-range, optional) │
│ │ ├── TitansMemoryNode : Surprise-gated persistent memory │
│ │ ├── GateNode : Learned softmax multi-input merge │
│ │ ├── CCA gates : Curriculum Component Activation │
│ │ └── Merge RMSNorm : Normalization after multi-predecessor merges │
│ ├── LTI Injection : Stable recurrent state mixing │
│ │ h_new = A·h + B·e + graph_output (A < 1, learnable decay) │
│ └── ACT Halting : Dynamic per-token depth allocation │
└──────────────────────────────────────────────────────────────────┘
↓
RMSNorm
↓
TiedLMHead (gradient-buffered weight tying with embedding)
↓
Logits / Loss
Hyperpersonalization: Train a small model (from tens of milions of parameters to ~1B parameters) from cold start or from an early - stage partially pre-trained model checkpoint on a personalized corpus, one enriched in it's representation of content relevant to your domain knowledge, before you full fine-tune it on your own data. The model becomes an expert in the one thing generic frontier models don't know anything about: you, your domain knowledge, style, notes, emails, work patterns, and personal voice. A personalized small language model drafts responses to your requests with full awareness of the entire data you encoded in its weights, including the secondary details, tertiary details, and edge cases that vector DB RAG systems usually miss because they are semantically dissimilar to the prompt. This oversight leaves the loose ends that hang you if you don't catch them. Evolve to a better adapted approach.
On-device AI: Efficient inference on CPU/GPU for desktops, laptops, tablets, and mobile. Strong quality per parameter, with optional fully-local (log-free) operation for sensitive use cases.
Economical and ecologically sound AI: The brute force approach that previous generation [used to be] frontier labs use is unsustainable: It burn cash at a rate that is economically unsustainable, causes electricity prices to surge for consumers competing with them for their share of electric grid's limited capacity, and compromises water supplies. Take an exponential cut out of the problem with more efficient AI: A model architecture with better parameter efficinecy, one that can use hybrid or even flat out linear attention only, and that can saturate the model's weights / grock on as little as one epoch.
| Component | What it does | Why it matters |
|---|---|---|
| Neural Columns & Heterogeneous Nodes | Each column holds diverse node types (attention variants, SwiGLU, Mamba-2, gating, optional Titans memory) instead of identical transformer blocks. | Different information pathways for different computations, like biological cortical columns. |
| Recurrent Depth (LTI + ACT) | The same graph weights are looped n_loops times. LTI injection keeps the recurrent state stable. ACT halting dynamically allocates compute per token. | Iterative refinement without parameter growth; easy tokens use 1 loop, hard reasoning uses more. |
| Hybrid Attention | Linear attention (O(n) complexity) in most columns, with periodic full-attention columns for exact retrieval. | Long-context efficiency without losing precise copy/lookup capability. |
| Mamba-2 SSD (optional) | State Space Duality with chunked parallel scan. Auto-activates when ssm_d_state >= 64. | Handles very long-range dependencies efficiently on CPU, CUDA, or MPS. |
| Titans Neural Memory (optional) | Persistent surprise-gated memory via outer-product updates (first column only by default). | Test-time memory that can retain patterns across long documents without growing KV cache. |
| CCA (Curriculum Component Activation) | Attention nodes start gated, gradually open over warmup steps via learned sigmoid gates. | Prevents random attention from drowning FFN signal at initialization. ~30% PPL improvement observed. |
| TiedLMHead with Gradient Buffer | LM head shares embedding weight. A learned linear buffer (init as identity) routes part of the gradient to prevent ~3× embedding gradient overload. | Halves parameter count (embedding table not duplicated). Safe weight tying without destabilizing training. |
| Modern Primitives | RoPE, SwiGLU, RMSNorm. | Proven SOTA components for convergence and generation quality. |
OpenMythos published a recurrent depth with LTI stability and ACT halting. HelixLM takes that insight and makes it work inside a heterogeneous graph that mimics neural column and random topology connectivity found in biological brains.
| Capability | OpenMythos | HelixLM |
|---|---|---|
| Recurrent depth | ✅ Same block looped | ✅ Same block looped |
| LTI stability | ✅ Spectral radius < 1 | ✅ Spectral radius < 1, configurable init_A (default 1/e) |
| ACT halting | ✅ Dynamic per-token depth | ✅ Dynamic per-token depth |
| Architecture inside loop | Standard transformer block | Heterogeneous random graph |
| Attention | Standard full attention only | Linear + full hybrid |
| Node types | Single (transformer block) | 7+ active types (attention variants, FFN, SSM, gate, neural memory) |
| Positional encoding | Standard learned | RoPE |
| Activation | GELU | SwiGLU |
| Normalization | LayerNorm | RMSNorm |
| Open source | ✅ Yes | ✅ Yes |
| HF integration | ❌ No | ✅ Full PreTrainedModel, AutoModelForCausalLM |
| Weight tying | Standard | Gradient-buffered TiedLMHead |
Cerebros showed that biological random hyperdense vertical and lateral topology of Dense layers could outperform rigid layer stacks. It generated text without attention on small data, but required elaborate integration and clashed with standard model-structure paradigms. HelixLM smoothly integrates that topological insight into a modern, HF-compatible LLM backbone.
| Model | Total Parameters | Non-embedding (thinking) parameters | Training Tokens | arc:easy | truthfulqa:mc2 |
|---|---|---|---|---|---|
| HelixLM 41M base, epoch 3 checkpoint (me) | 40.1M | 19.1M | 1.5 Billion | 42.38% | 46.36% |
| GPT2 Small | 150M | 84.6M | 8 billion tokens | 39.5% | 39% |
| Difference | 3.74 X smaller | 4.429 X fewer active params | 5.33 X less data | +2.88 (better) | 7.359 (better) |
>>> from transformers import AutoModelForCausalLM
>>> model_id = 'david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep3'
>>> model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, force_download=True)
>>> import weightwatcher as ww
>>> watcher = ww.WeightWatcher()
>>> details = watcher.analyze(model=model, layers=layers)
>>> summary = watcher.get_summary(details)
>>> summary
{'log_norm': np.float64(2.547738328510092), 'alpha': np.float64(4.774676280147847), 'alpha_weighted': np.float64(4.070240281892587), 'log_alpha_norm': np.float64(4.640816008450303), 'log_spectral_norm': np.float64(1.1167532258336426), 'stable_rank': np.float64(45.623940480383965)}
>>> details
layer_id name D M N Q alpha alpha_weighted ... stable_rank status sv_max sv_min warning weak_rank_loss xmax xmin
0 8 Linear 0.133501 512 512 1.0 8.049137 -0.695795 ... 128.978237 success 0.905270 0.001048 under-trained 0 0.819514 0.586496
1 9 Linear 0.130986 512 512 1.0 8.557675 -0.798744 ... 129.709465 success 0.898115 0.000398 under-trained 1 0.806610 0.591386
2 10 Linear 0.124447 512 512 1.0 8.474757 -0.765280 ... 128.792578 success 0.901259 0.000051 under-trained 1 0.812267 0.582928
3 11 Linear 0.123478 512 512 1.0 14.556391 -1.398904 ... 131.128524 success 0.895259 0.000520 under-trained 1 0.801489 0.669532
4 12 Linear 0.174834 64 64 1.0 5.065331 -5.259381 ... 18.099317 success 0.302584 0.000392 1 0.091557 0.054470
5 13 Linear 0.122635 64 64 1.0 3.896234 -3.875313 ... 16.283787 success 0.318189 0.002329 0 0.101244 0.044695
6 18 Linear 0.030318 512 1280 2.5 3.569812 5.793807 ... 36.377411 success 6.478885 0.129977 0 41.975953 5.156733
7 19 Linear 0.031311 512 1280 2.5 3.983274 5.573304 ... 55.797011 success 5.007124 0.126186 0 25.071292 5.381964
8 20 Linear 0.071796 512 1280 2.5 2.783094 4.098033 ... 50.053484 success 5.447995 0.176460 0 29.680652 4.173654
9 24 Linear 0.100675 512 512 1.0 1.356042 1.433037 ... 22.989641 success 3.375898 0.000344 over-trained 3 11.396687 0.006976
10 28 Linear 0.064085 512 512 1.0 1.294488 2.117125 ... 8.557538 success 6.572740 0.000011 over-trained 11 43.200906 0.001178
11 29 Linear 0.055937 512 512 1.0 1.306318 2.401387 ... 8.408097 success 8.301243 0.000036 over-trained 13 68.910635 0.002945
12 30 Linear 0.092528 512 512 1.0 1.317351 1.231344 ... 38.670129 success 2.933299 0.000069 over-trained 7 8.604244 0.005258
13 31 Linear 0.097686 512 512 1.0 1.308202 1.465259 ... 36.352822 success 3.631012 0.000064 over-trained 6 13.184249 0.007658
14 32 Linear 0.083663 64 64 1.0 1.805958 2.986915 ... 2.976555 success 6.713689 0.001788 over-trained 0 45.073622 0.811336
15 33 Linear 0.080878 64 64 1.0 1.968262 3.127394 ... 2.636161 success 6.229538 0.000152 over-trained 1 38.807147 1.387472
16 38 Linear 0.086598 512 1280 2.5 6.125869 5.813554 ... 71.418415 success 2.982006 0.202114 under-trained 0 8.892361 5.242391
17 39 Linear 0.082814 512 1280 2.5 13.425495 10.753701 ... 83.177960 success 2.514754 0.189539 under-trained 0 6.323988 4.843725
18 40 Linear 0.073901 512 1280 2.5 7.153645 9.429090 ... 72.709022 success 4.560799 0.103909 under-trained 0 20.800888 12.979208
19 44 Linear 0.082740 512 512 1.0 1.767602 0.768192 ... 14.807745 success 1.649293 0.000145 over-trained 3 2.720168 0.038350
20 48 Linear 0.065125 512 512 1.0 2.209246 5.265884 ... 5.355617 success 15.551893 0.000019 41 241.861369 16.477667
21 49 Linear 0.063150 512 512 1.0 2.443553 5.178950 ... 7.207727 success 11.474067 0.000011 38 131.654218 12.208631
22 50 Linear 0.064775 512 512 1.0 14.973133 17.761326 ... 57.627971 success 3.918380 0.000016 under-trained 18 15.353699 11.542509
23 51 Linear 0.041280 512 512 1.0 6.086550 10.044288 ... 31.187437 success 6.685311 0.000268 under-trained 1 44.693379 15.008032
24 52 Linear 0.062858 64 64 1.0 1.961756 3.267400 ... 3.915976 success 6.804156 0.024430 over-trained 0 46.296533 1.026390
25 53 Linear 0.045738 64 64 1.0 2.145143 3.193630 ... 3.610048 success 5.551192 0.003691 0 30.815738 0.837654
26 58 Linear 0.054151 512 1280 2.5 2.497603 5.055094 ... 23.338434 success 10.279906 0.310842 0 105.676473 5.167725
27 59 Linear 0.051628 512 1280 2.5 4.005447 4.480541 ... 77.638447 success 3.624983 0.237237 0 13.140502 3.577843
28 60 Linear 0.029988 512 1280 2.5 3.176491 5.538803 ... 23.432373 success 7.444628 0.063220 0 55.422488 4.609355
29 64 Linear 0.094577 512 512 1.0 2.938021 4.864589 ... 24.773599 success 6.727727 0.000486 1 45.262316 6.677217
30 67 Linear 0.065202 512 1536 3.0 3.532464 5.245250 ... 25.745811 success 5.526308 0.028613 0 30.540077 8.391077
31 68 Linear 0.055916 512 2048 4.0 8.148052 9.321012 ... 50.515926 success 3.732312 0.141487 under-trained 0 13.930152 7.936553
32 69 Linear 0.050900 512 1024 2.0 3.225601 4.705134 ... 25.467911 success 5.362189 0.001445 0 28.753067 4.112694
33 70 Linear 0.080542 512 3584 7.0 7.230996 10.267542 ... 133.472802 success 5.128229 0.333044 under-trained 0 26.298728 15.231137
[34 rows x 32 columns]
| Metric | Value | Benchmark Context |
|---|---|---|
log_norm | 2.548 | Moderate capacity utilization |
alpha | 4.775 | Borderline elevated; typical for early-stage small LMs |
alpha_weighted | 4.070 | Weighted by layer size; slightly lower than unweighted |
log_alpha_norm | 4.641 | — |
log_spectral_norm | 1.117 | Low spectral amplification |
stable_rank | 45.624 | Indicates significant rank differentiation across layers |
Note:
The model exhibits a cooperative dichotomy between two spectral populations and between heuristics and actual model performance:
Sharp / Low-Entropy Layers (α ≈ 1.3–2.2)
sv_max 2.9–15.6, sv_min ~10⁻⁵)Diffuse / High-Entropy Layers (α ≈ 6–15)
sv_max ~0.9), broad ESD tailsFFN Up-Projections (512×1280, 2.5× expansion)
Output Projection (512×3584)
Silent Rank Collapse (Layers 48, 49)
weak_rank_loss = 41 and 38 respectivelysv_max/sv_min ratio ≈ 10⁶ , numerically ill-conditionedweak_rank_loss and sv_max/sv_min condition number alongside α. Silent rank collapse (high WRL without α warning) is a more reliable indicator of genuine pathology than α alone in this model family.Base Chinchilla Formula: 20 tokens per parameter
We trained on 1.5B tokens, so at first pass:
However, let's adjust for the disproportionate embedding table (As I am a 41M param model, I have a disproportionate percentage of my weights in my embeddign table, compared to a trypical 150M - 1T parameter model, so lets't adjust for that) ...
Embedding Table Adjustment
GPT-2 Tokenizer specs:
Parameter breakdown for 40.1M model:
| Component | Calculation | Parameters |
|---|---|---|
| Total Parameters | - | 40.1M |
| Token embeddings | 50,257 × 512 | 25.7M |
| Position embeddings | ~1024 × 512 | ~0.5M |
| Attention and FFN layers | Remainder | ~13.9M |
Effective active parameters: ~13.9M Chinchilla optimal for active params: 13.9M × 20 = 278M tokens My actual training corpus: 1.5B tokens
| Metric | Value |
|---|---|
| Tokens per active param | 107.9 |
| vs. Chinchilla optimal | 5.39 × over chinchilla optimal |
TLDR, we've grocked to ~ full theoretical saturation on 1 epoch, on 5.39 X chinchilla optimal with only the earlyest signs of destabilization at that point and no substantial rank collapse yet occuring from over-training, see the model card https://huggingface.co/david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1. Then we continued training, with the perplexity decreasing further by 8.57 from 63.46 at epoch 1 to 54.89 at epoch 3, with the model still mostly stable, albeit showing some early signs of possible rank collapse on a small subset of layers, particularly gating layers.
<!-- ml-intern-provenance -->This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.
model_id = 'david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep3'
from transformers import pipeline
pipe = pipeline("text-generation",model_id, trust_remote_code=True)
x = pipe('In the galaxy there are billions and billions of stars. Some of these stars help us pinpoint where other galaxies are.', max_new_tokens=100, temperature=0.6, top_k=40, top_p=0.95, repetition_penalty=1.2)
print(x)
[{'generated_text': "In the galaxy there are billions and billions of stars. Some of these stars help us pinpoint where other galaxies are. The most common example is that they're a supernova, which has a mass of 8 times larger than our Sun.\nThe star's mass can be estimated as:\n\n• A cluster of massive stars in a single-dimensional space of 10 million years.\n• The Milky Way Galaxy (Mpc) = 13.5 trillion light-years from its center.\n• A large number of stars with an average distance of 1.8 billion light-years away from the centre of the"}]
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = 'david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep3'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.
This project is open-source under a modified Apache 2.0 license. See Cerebros-License for full terms.