flash-archer-150M-1.0

Flash-Archer-150M

Flash-Archer-150M is a compact Llama-compatible causal language model trained from scratch on roughly 975M tokens from FineWeb-Edu.

Why This Model Exists

Flash-Archer-150M is intended as a small, easy-to-load baseline for experimentation with custom pretraining, tokenizer behavior, low-cost inference, and continued fine-tuning. It is not instruction-tuned and should be treated as a base language model rather than a chat assistant.

The main points of interest are:

• 150M parameter Llama-style decoder-only architecture.
• Custom 16k byte-level BPE tokenizer.
• Standard Hugging Face Transformers format with safetensors.
• Small enough to run on CPU for testing, with GPU recommended for faster generation or fine-tuning.

Model Details

Training Data

Flash-Archer-150M was trained on:

• Dataset: HuggingFaceFW/fineweb-edu
• Config: sample-10BT
• Dataset license: ODC-BY
• Documents consumed: 877,735
• Tokens seen: 974,520,320
• Validation holdout: 2,000 documents
• Tokenizer training documents: 200,000

FineWeb-Edu is an English educational web dataset derived from Common Crawl. The model card attributes the dataset here because the training data is released under ODC-BY.

Training Methodology

The model was trained from scratch as a next-token prediction causal language model.

Validation

These are validation-log results from the training run, not standardized benchmark results. The run used a small validation setting (val_batches=2), so the numbers should be interpreted as rough training diagnostics.

No public benchmark scores are claimed.

Hardware

Training metadata indicates the run was performed on Kaggle with one NVIDIA Tesla T4 GPU with about 15.6 GB VRAM. The configured maximum training time was 7 hours.

Intended Uses

This model is best suited for:

• Educational experiments with small pretrained language models.
• Continued pretraining or supervised fine-tuning.
• Tokenizer and architecture experiments.
• Lightweight local generation tests.
• Baseline comparisons against other small causal LMs.

Limitations

• This is a base model, not an instruction-following chat model.
• It may repeat itself, produce low-quality generations, or fail to follow prompts.
• It has not been evaluated on safety, factuality, coding, reasoning, or domain benchmarks.
• It was trained on web text and may reproduce biases, errors, or unsafe content present in that data.
• The context length is limited to 1,024 tokens.
• Validation metrics are noisy because only a small number of validation batches were logged.

Inference

Install dependencies:

pip install torch transformers safetensors

Load the model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Norman89107/Flash-Archer-150M"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()

prompt = "In simple terms, machine learning is"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        top_k=50,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

For GPU inference:

model = AutoModelForCausalLM.from_pretrained(
    "Norman89107/Flash-Archer-150M",
    torch_dtype=torch.float16,
    device_map="auto",
)

Files

• config.json: Llama-compatible model configuration.
• model.safetensors: Model weights.
• tokenizer.json: Custom byte-level BPE tokenizer.
• tokenizer_config.json: Tokenizer metadata and max length.
• special_tokens_map.json: Special token definitions.
• generation_config.json: Default sampling settings.
• inference.py: Minimal inference script.

Source Code

No public source repository link was provided for this release. The uploaded checkpoint is in standard Transformers format and can be loaded directly with AutoModelForCausalLM.

License

The model weights are released under Apache-2.0. The training data used FineWeb-Edu, which is released under ODC-BY; see the FineWeb-Edu dataset card for dataset terms and attribution details.