Fleck-M-500K

Fleck-M-500K

Fleck-M-500K Instruct — the independently instruction-tuned child of Fleck-M-500K-Base.

• 497,288 parameters
• BF16 SafeTensors weights
• Fine-tuned from the Base-100M model on Dolly-15k
• Intended for local conversational experiments on Apple Silicon

Model Details

Training

Pretraining

The Base parent was initialized from scratch and pretrained for exactly 100,000,000 real tokenizer tokens on a 70/30 FineWeb-Edu/FineWeb mixture.

Instruction Tuning

Benchmark Results

Corrected zero-shot evaluation, no chat template, FP32 evaluation, and the same benchmark aggregation protocol were used for both variants.

Tokenizer and Chat Format

• Fleck-Tokenizer-2048
• Byte-level BPE
• Vocabulary size: 2,048

The training chat format is:

<bos><|user|>{user}<|eot|><|assistant|>{response}<|eot|><eos>

Special Tokens

Usage

The bundle includes a self-contained inference.py; it does not import the Fleck-LM checkout. The accompanying config.json, generation_config.json, and tokenizer_config.json describe the custom architecture and generation/tokenizer defaults; standard transformers.AutoModel loading is not supported. The chat_template.jinja file contains the Instruct chat template used by compatible HF tooling. Install the three runtime dependencies:

python -m pip install torch safetensors tokenizers

By default the CLI starts an interactive chat. /exit quits and /clear resets the conversation history:

python inference.py --device cpu --max-tokens 32

For one prompt without interactive mode, pass --no-chat and --prompt:

python inference.py \
  --ckpt model.safetensors \
  --tokenizer tokenizer.json \
  --no-chat \
  --prompt "Explain what a tokenizer does." \
  --max-tokens 32 \
  --device cpu

The script reads and runs the BF16 checkpoint without an FP32 model copy, validates every SafeTensors key, shape, and dtype, and uses FP32 only for attention score/softmax and tied-logit accumulation. It reproduces the factorized tied embedding/logits, effective-depth execution A → B → A → B, half-split RoPE, GQA, physical KV caches, RMSNorms, and greedy generation without repository-local imports. Generation stops on <|eot|> or <eos>.

Limitations

This model is extremely small and is intended for research and local experimentation rather than reliable general-purpose assistance. It may produce repetitions, malformed text, weak factual answers, or incoherent responses. Instruction tuning improves conversational behavior but does not overcome the limits of a 497,288 parameter model.

License

MIT License.

Files

The public bundle contains these files:

• README.md — model card and usage documentation
• inference.py — standalone strict loader and interactive/single-prompt inference CLI
• model.safetensors — BF16 model weights
• tokenizer.json — standalone tokenizer
• config.json — custom architecture configuration
• generation_config.json — greedy generation defaults
• tokenizer_config.json — tokenizer defaults and special-token mapping
• chat_template.jinja — Instruct chat template No training data, optimizer state, or other training outputs are included.