bwsk-bert-base

tzervas/bwsk-bert-base

bwsk-bert-base is a fill-mask model from tzervas. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as mit.

BERT-base (110M params) trained in 6 variants (3 BWSK modes x 2 experiments) on WikiText-2 with full convergence training and early stopping.

Downloads · 30 days

Access

Public

Updated Mar 5, 2026

Repo size

2.6 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors2.6 GB · 100%

At a glance

Task: Fill-Mask
Library: transformers
License: mit
Access: Public
Created: Mar 5, 2026
Updated: Mar 5, 2026
SHA: 916e01f4

Try a prompt

Base models

google-bert/bert-base-uncased

Datasets

wikitext

Model card

From the Hugging Face model README

BWSK BERT-base

BERT-base (110M params) trained in 6 variants (3 BWSK modes x 2 experiments) on WikiText-2 with full convergence training and early stopping.

This repo contains all model weights, configs, and training results in a single consolidated repository.

What is BWSK?

BWSK is a framework that classifies every neural network operation as S-type (information-preserving, reversible, coordination-free) or K-type (information-erasing, synchronization point) using combinator logic. This classification enables reversible backpropagation through S-phases to save memory, and CALM-based parallelism analysis.

Model Overview

Property	Value
Base Model	google-bert/bert-base-uncased
Architecture	Transformer (masked_lm)
Parameters	110M
Dataset	WikiText-2
Eval Metric	Pseudo-Perplexity

S/K Classification

Type	Ratio
S-type (information-preserving)	67.3%
K-type (information-erasing)	32.7%

Fine-tune Results

Mode	Final Loss	Val Pseudo-Perplexity	Test Pseudo-Perplexity	Peak Memory	Time	Epochs
Conventional	1.8896	5.56	5.40	4.0 GB	7.4m	5
BWSK Analyzed	1.9163	5.54	5.57	4.0 GB	7.3m	5
BWSK Reversible	1.5086	5.57	5.49	2.9 GB	9.1m	5

Memory savings (reversible vs conventional): 27.7%

From Scratch Results

Mode	Final Loss	Val Pseudo-Perplexity	Test Pseudo-Perplexity	Peak Memory	Time	Epochs
Conventional	6.9915	1383.85	1489.18	4.0 GB	7.3m	5
BWSK Analyzed	7.4792	1373.72	1480.62	4.0 GB	7.4m	5
BWSK Reversible	7.0919	1401.24	1503.86	2.9 GB	9.0m	5

Memory savings (reversible vs conventional): 27.6%

Repository Structure

├── README.md
├── results.json
├── finetune-conventional/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── finetune-bwsk-analyzed/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── finetune-bwsk-reversible/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── scratch-conventional/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── scratch-bwsk-analyzed/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json
├── scratch-bwsk-reversible/
│   ├── model.safetensors
│   ├── config.json
│   └── training_results.json

Usage

Load a specific variant:

from transformers import AutoModelForMaskedLM, AutoTokenizer

# Load fine-tuned conventional variant
model = AutoModelForMaskedLM.from_pretrained(
    "tzervas/bwsk-bert-base", subfolder="finetune-conventional"
)
tokenizer = AutoTokenizer.from_pretrained(
    "tzervas/bwsk-bert-base", subfolder="finetune-conventional"
)

# Load from-scratch BWSK reversible variant
model = AutoModelForMaskedLM.from_pretrained(
    "tzervas/bwsk-bert-base", subfolder="scratch-bwsk-reversible"
)

Training Configuration

Setting	Value
Optimizer	AdamW
LR (fine-tune)	5e-05
LR (from-scratch)	3e-04
LR Schedule	Cosine with warmup
Max Grad Norm	1.0
Mixed Precision	AMP (float16)
Early Stopping	Patience 3
Batch Size	4
Sequence Length	512

Citation

@software{zervas2026bwsk,
  author = {Zervas, Tyler},
  title = {BWSK: Combinator-Typed Neural Network Analysis},
  year = {2026},
  url = {https://github.com/tzervas/ai-s-combinator},
}

License

MIT

Task: Fill-Mask
Library: transformers
License: mit
Created: Mar 5, 2026
Updated: Mar 5, 2026