smol-historical-llm-test

Smol Historical LLM

smol-historical-llm is a small experimental historical language model adapter based on HuggingFaceTB/SmolLM3-3B-Base.

It is trained with QLoRA-style continued pretraining / domain adaptation on French historical newspaper text from PleIAs/French-PD-Newspapers.

The goal is to adapt a compact open language model to historical newspaper language, OCR-style noise, old spelling, long documents, and cultural heritage material.

This repository contains a LoRA/PEFT adapter, not a fully merged standalone model.

Model Details

• Model name: emanuelaboros/smol-historical-llm
• Test adapter: emanuelaboros/smol-historical-llm-test
• Base model: HuggingFaceTB/SmolLM3-3B-Base
• Model type: causal language model adapter
• Training method: QLoRA / PEFT LoRA adapter
• Primary language: French
• Primary domain: historical newspapers
• Dataset: PleIAs/French-PD-Newspapers
• Task: text generation / historical domain adaptation

Intended Use

This model is intended for experimental research on historical and noisy document processing.

Potential uses include:

• generating or continuing historical-style French newspaper text;
• studying adaptation of small language models to historical OCR data;
• evaluating language modeling on historical newspapers;
• downstream experiments in OCR correction, NER, NEL, or historical information extraction;
• prototyping cultural heritage NLP workflows.

This model is not intended as a general-purpose assistant or instruction-following chatbot.

Training Data

The adapter is trained on:

PleIAs/French-PD-Newspapers

This dataset contains French public-domain newspapers and periodicals, including OCR-derived historical text.

During training, documents are wrapped with simple control tokens:

<|historical_document|>
...
<|end_document|>

The default training script uses the complete_text column.

Training Procedure

The training script performs continued pretraining / domain adaptation of the base causal language model.

Default configuration used in the training script:

base_model: HuggingFaceTB/SmolLM3-3B-Base
dataset: PleIAs/French-PD-Newspapers
text_column: complete_text
max_seq_length: 2048
LoRA r: 32
LoRA alpha: 64
LoRA dropout: 0.05
target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
optimizer: paged_adamw_8bit
precision: bf16

The smoke-test version may have been trained for only a few steps and should not be interpreted as a final domain-adapted model.

Usage

Install dependencies:

pip install -U transformers peft accelerate safetensors torch

Load the base model and the adapter:

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "HuggingFaceTB/SmolLM3-3B-Base"
adapter_id = "emanuelaboros/smol-historical-llm"

tokenizer = AutoTokenizer.from_pretrained(
    base_model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

prompt = """<|historical_document|>
Paris, 1894. Le journal rapporte que
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For the smoke-test adapter, replace:

adapter_id = "emanuelaboros/smol-historical-llm"

with:

adapter_id = "emanuelaboros/smol-historical-llm-test"

Example Prompt

<|historical_document|>
Paris, 1894. Le journal rapporte que

The model should continue in a style closer to historical French newspaper prose than the unadapted base model.

Evaluation

A first evaluation should compare perplexity on held-out historical newspaper text.

Suggested baseline:

HuggingFaceTB/SmolLM3-3B-Base

Suggested adapted model:

emanuelaboros/smol-historical-llm

Optional comparison:

HuggingFaceTB/SmolLM3-3B

Example metrics to report:

Lower perplexity is better.

Limitations

This is an experimental research model.

Known limitations:

• The model is adapted mostly to French historical newspaper text.
• OCR noise in the dataset may be learned by the model.
• Generated text may contain hallucinated historical facts, names, dates, or events.
• The model is not instruction-tuned for reliable question answering.
• The smoke-test adapter, if used, is trained for too few steps to show meaningful improvement.
• Evaluation should be done on held-out data that was not used during training.

Bias, Risks, and Historical Content

Historical newspapers may contain outdated, offensive, colonial, racist, sexist, or otherwise harmful language and viewpoints. The model may reproduce such language because it is adapting to historical source material.

Outputs should be interpreted as generated text, not as factual historical evidence.

For scholarly or cultural heritage use, generated outputs should always be checked against primary sources.

Citation

If you use this model, please cite the base model and dataset:

@misc{smollm3base,
  title = {SmolLM3-3B-Base},
  author = {Hugging Face TB},
  year = {2025},
  url = {https://huggingface.co/HuggingFaceTB/SmolLM3-3B-Base}
}

@misc{frenchpdnewspapers,
  title = {French Public Domain Newspapers},
  author = {PleIAs},
  url = {https://huggingface.co/datasets/PleIAs/French-PD-Newspapers}
}

Acknowledgements

This adapter builds on SmolLM3 from Hugging Face and historical newspaper data distributed by PleIAs.

License

This model card declares apache-2.0 to match the base model metadata. Users should also respect the licenses and terms of the base model and training dataset.