HENLA-CONFED-3B-FINEWEB-CLEANLM-STEP70000

HENLA-CONFED-3B-FINEWEB-CLEANLM-STEP70000

HENLA-CONFED-3B-FINEWEB-CLEANLM-STEP70000 is an experimental 3B-parameter language model checkpoint based on the HENLA-CONFED architecture.

This checkpoint is the preferred general baseline for the HENLA-CONFED 3B line. It was trained from the confederated-area architecture on cleaned FineWeb-style educational text, with additional filtering to reduce noisy sequences, malformed text, repeated symbols, and low-quality samples.

Architecture

HENLA-CONFED explores a confederated cognitive-area design for language modeling.

At a high level, the architecture includes:

• causal self-attention blocks;
• multiple parallel cognitive-area MLP modules per block;
• learned routing/gating across cognitive areas;
• residual fusion of routed area outputs;
• standard causal language modeling output head;
• compatibility with Hugging Face transformers remote code loading.

This checkpoint uses approximately:

• 2.84B parameters;
• 24 layers;
• hidden size 1280;
• 16 attention heads;
• 8 cognitive areas per block;
• context length 512;
• GPT-style tokenizer vocabulary.

Intended use

This checkpoint is intended for research and experimentation with:

• confederated-area language modeling;
• neuro-symbolic-inspired model architecture;
• routing behavior across specialized internal areas;
• early-stage text continuation;
• comparison against later HENLA-CONFED checkpoints;
• educational demonstrations and non-commercial study;
• further non-commercial research fine-tuning.

It is the preferred HENLA-CONFED base checkpoint for additional training.

Compute and development note

HENLA-CONFED is presented as a constrained-compute research experiment: a 3B-parameter confederated-area language model line trained, stabilized, benchmarked, and released over a short development cycle with roughly €325 of rented GPU compute.

The HENLA-CONFED 3B line was developed and trained over an experimental cycle of approximately two days. This context is important for interpreting the results: the goal was not to match industrial-scale small language models trained on much larger budgets, but to test whether a confederated cognitive-area architecture could be trained, stabilized, published, and evaluated under constrained resources.

The resulting checkpoints should therefore be understood as experimental research artifacts showing what was achievable under limited compute, rather than as fully optimized production language models.

Recommended loading

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

repo_id = "RthItalia/HENLA-CONFED-3B-FINEWEB-CLEANLM-STEP70000"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=False,
    repetition_penalty=1.15,
    no_repeat_ngram_size=3,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Checkpoint status

This checkpoint is a general text-continuation baseline. It is more broadly useful than the prefix-specialized branches, but it may still produce weak grammar, vague continuations, and unstable factual statements.

Observed behavior:

• improved substantially over earlier FineWeb checkpoints;
• no longer dominated by simple the/of/is loops;
• capable of rough sentence-level continuation;
• still weak on long-range coherence;
• can generate generic or pseudo-encyclopedic text;
• may confuse articles such as a important / an device.

Evaluation note

This checkpoint was evaluated with small deterministic short-prompt probes covering grammar, short continuation, repetition, and HENLA-description behavior.

Among the tested HENLA-CONFED branches, this checkpoint remained the preferred base for general continuation and further training, while specialized prefix branches performed better on controlled short prompts.

These probes are intended as development diagnostics for comparing HENLA-CONFED branches. They are not a standardized benchmark and should not be interpreted as a leaderboard result.

Related checkpoints

• RthItalia/HENLA-CONFED-3B-PREFIX-V4-STEP300
  Best overall HENLA branch in the internal short-prompt diagnostics, with stronger grammar/article and controlled short-prompt behavior.

• RthItalia/HENLA-CONFED-3B-HENLA-PREFIX-150
  Focused HENLA-description branch, strongest on HENLA-specific short prompts but more specialized.

For general text generation and further training, this checkpoint is the preferred HENLA-CONFED base.

Limitations

This is an experimental research checkpoint. It may produce:

• incorrect facts;
• weak grammar;
• repetitive phrases;
• generic completions;
• unstable reasoning;
• synthetic-looking scientific or medical statements;
• unsupported claims.

Do not use this model for medical, legal, financial, safety-critical, or other high-stakes decisions.

Training notes

The model was trained through several experimental stages:

1. initial HENLA-CONFED 3B bootstrapping;
2. FineWeb-style educational continuation training;
3. routing/gating stabilization;
4. cleaned-language-model continuation training.

This checkpoint corresponds to the cleaned-language-model branch at step 70,000.

License

This checkpoint is released under the HENLA Research and Education Non-Commercial License.

Permitted uses:

• academic research;
• independent research;
• educational use;
• student projects;
• evaluation and benchmarking;
• non-commercial experimentation.

Not permitted without prior written permission:

• commercial use;
• paid products or services;
• resale or redistribution for commercial purposes;
• integration into commercial applications;
• hosted commercial inference;
• training, distillation, or fine-tuning for commercial deployment.

For commercial licensing, contact the model author.

Citation / attribution

If you use this checkpoint, please refer to it as:

HENLA-CONFED-3B-FINEWEB-CLEANLM-STEP70000 by RthItalia.