Downloads · 30 days
775
33% of all-time downloads
ZeroS-Pico-v1.1
fromziro/ZeroS-Pico-v1.1
ZeroS-Pico-v1.1 is a machine learning model from fromziro. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This release improves upon ZeroS-Pico-v1.0 through architectural refinements and extended pretraining on over 1.7B tokens.
All-time downloads
2.4K
Public
Parameters
59.8K
255 KB on disk
Likes
8
Public
Repo makeup
Click a slice to open those files.
.safetensors255 KB · 69%
At a glance
- License
- apache-2.0
- Model type
- custom_model
- Access
- Public
- Created
- Sep 10, 2026
- Updated
- Sep 15, 2026
- SHA
- 99445903
Datasets
Spaces
Model card
From the Hugging Face model README
ZeroS-Pico-v1.1
This release improves upon ZeroS-Pico-v1.0 through architectural refinements and extended pretraining on over 1.7B tokens.
Architecture: What's different?
| Component | ZeroS-Pico-v1.0 | ZeroS-Pico-v1.1 | Notes |
|---|---|---|---|
| Layers | 9 | 12 | Deeper network depth |
| Hidden Size | 32 | 32 | No change |
| Vocab Size | 260 | 260 | No change |
| Attention Heads | 4 | 4 | No change |
| KV Heads | 2 | 2 | No change |
| Intermediate Size | 64 | 64 | No change |
| SwiGLU Interval | 4 | 6 | Same FFN capacity |
| mHC Lanes | 4 | 1 | Higher training throughput with virtually no loss in precision |
| Engram | True | False | Rerouted parameters into layers instead of Engram embeddings |
| Max Position Embeddings (training sequence length) | 96 | 155 | Increased maximum context |
| Tie Word Embeddings | True | True | No change |
ZeroS-Pico-v1.1 takes a deep-and-narrow approach, focusing parameter budget on raw transformer depth and throughput rather than auxiliary mechanisms like Engram embeddings and multi-lane routing.
Training Dataset
ZeroS-Pico-v1.1 was trained on 1.7B tokens (up from 600M in v1.0), covering web text, educational material, synthetic data, normalized code, and mathematics.
| Dataset | Share |
|---|---|
| FineWeb-Edu | 36.0% |
| DCLM Baseline 1.0 | 22.9% |
| FinePhrase | 13.4% |
| MGA FineWeb-Edu | 10.3% |
| Tiny Strange Textbooks | 8.2% |
| OpenMathInstruct-2 | 7.6% |
| NPset-2 Python-Edu | 1.6% |
Benchmark Results
| Task | ZeroS-Pico-v1.0 | ZeroS-Pico-v1.1 | Difference |
|---|---|---|---|
| ARC Challenge | 22.95% | 24.57% | +1.62% |
| ARC Easy | 27.65% | 26.30% | -1.35% |
| HellaSwag | 25.94% | 26.19% | +0.25% |
| PIQA | 49.62% | 51.14% | +1.52% |
| ArithMark-3.0 | 31.50% | 32.00% | +0.50% |
| Average | 31.53% | 32.04% | +0.51% |
As you can see, ZeroS-Pico-v1.1 is a steady improvement over v1.0, outperforming it on 4 out of 5 benchmarks with notable gains on ARC Challenge (+1.62%) and PIQA (+1.52%), despite a regression on ARC Easy.
On the Open SLM Leaderboard, at just 59.8K parameters, it outranks the 2.6M parameter Supra-Mini-v4-2M:
Hardware
- AMD Ryzen 5 2600
License
Apache 2.0
Citation
@misc{zeros-pico-v1.1,
title = {ZeroS-Pico-v1.1},
organization = {FromZero},
authors = {Paul Courneya},
year = {2026},
url = {https://huggingface.co/fromziro/ZeroS-Pico-v1.1}
}
- Type
- custom_model
- License
- apache-2.0
- Languages
- en
- Created
- Sep 10, 2026
- Updated
- Sep 15, 2026