bwsk-pythia-70m

tzervas/bwsk-pythia-70m

bwsk-pythia-70m is a text generation model from tzervas. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.

Pythia-70M (70M params) trained in 6 variants (3 BWSK modes x 2 experiments) on WikiText-2 with full convergence training and early stopping.

Downloads · 30 days

Access

Public

Updated Mar 5, 2026

Repo size

1.7 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors1.7 GB · 99%

At a glance

Task: Text Generation
Library: transformers
License: mit
Access: Public
Created: Mar 5, 2026
Updated: Mar 5, 2026
SHA: 80eeeb7e

Try a prompt

Base models

EleutherAI/pythia-70m

Datasets

wikitext

Model card

From the Hugging Face model README

BWSK Pythia-70M

Pythia-70M (70M params) trained in 6 variants (3 BWSK modes x 2 experiments) on WikiText-2 with full convergence training and early stopping.

This repo contains all model weights, configs, and training results in a single consolidated repository.

What is BWSK?

BWSK is a framework that classifies every neural network operation as S-type (information-preserving, reversible, coordination-free) or K-type (information-erasing, synchronization point) using combinator logic. This classification enables reversible backpropagation through S-phases to save memory, and CALM-based parallelism analysis.

Model Overview

Property	Value
Base Model	EleutherAI/pythia-70m
Architecture	Transformer (causal_lm)
Parameters	70M
Dataset	WikiText-2
Eval Metric	Perplexity

S/K Classification

Type	Ratio
S-type (information-preserving)	67.8%
K-type (information-erasing)	32.2%

Fine-tune Results

Mode	Final Loss	Val Perplexity	Test Perplexity	Peak Memory	Time	Epochs
Conventional	3.2062	29.58	28.78	4.1 GB	1.3m	4
BWSK Analyzed	3.4344	29.68	28.92	4.1 GB	1.3m	4
BWSK Reversible	3.2864	29.63	28.93	3.4 GB	1.4m	4

Memory savings (reversible vs conventional): 15.9%

From Scratch Results

Mode	Final Loss	Val Perplexity	Test Perplexity	Peak Memory	Time	Epochs
Conventional	3.5755	195.11	201.58	4.1 GB	3.5m	6
BWSK Analyzed	3.4921	207.35	215.25	4.1 GB	4.1m	6
BWSK Reversible	3.2354	186.11	194.31	3.4 GB	4.5m	6

Memory savings (reversible vs conventional): 16.0%

Repository Structure

├── README.md
├── results.json
├── finetune-conventional/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── finetune-bwsk-analyzed/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── finetune-bwsk-reversible/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── scratch-conventional/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── scratch-bwsk-analyzed/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── scratch-bwsk-reversible/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json

Usage

Load a specific variant:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load fine-tuned conventional variant
model = AutoModelForCausalLM.from_pretrained(
    "tzervas/bwsk-pythia-70m", subfolder="finetune-conventional"
)
tokenizer = AutoTokenizer.from_pretrained(
    "tzervas/bwsk-pythia-70m", subfolder="finetune-conventional"
)

# Load from-scratch BWSK reversible variant
model = AutoModelForCausalLM.from_pretrained(
    "tzervas/bwsk-pythia-70m", subfolder="scratch-bwsk-reversible"
)

Training Configuration

Setting	Value
Optimizer	AdamW
LR (fine-tune)	5e-05
LR (from-scratch)	3e-04
LR Schedule	Cosine with warmup
Max Grad Norm	1.0
Mixed Precision	AMP (float16)
Early Stopping	Patience 3
Batch Size	8
Sequence Length	512

Citation

@software{zervas2026bwsk,
  author = {Zervas, Tyler},
  title = {BWSK: Combinator-Typed Neural Network Analysis},
  year = {2026},
  url = {https://github.com/tzervas/ai-s-combinator},
}

License

MIT

Task: Text Generation
Library: transformers
License: mit
Created: Mar 5, 2026
Updated: Mar 5, 2026