fasttext-euptvid

fasttext-euptvid

A fastText classifier for Portuguese language variety identification . Distinguishes European Portuguese (PT-PT) from Brazilian Portuguese (PT-BR)

Model Description

Designed for high-throughput filtering pipelines (e.g., Common Crawl processing).

Two variants (full and quantized)

Labels

• __label__PT_PT — European Portuguese
• __label__PT_BR — Brazilian Portuguese

How to Use

import fasttext
from huggingface_hub import hf_hub_download

# Full model ~1GB
model_path = hf_hub_download(repo_id="duarteocarmo/fasttext-euptvid", filename="model.bin")
model = fasttext.load_model(model_path)

# Or quantized model ~68 MB
# model_path = hf_hub_download(repo_id="duarteocarmo/fasttext-euptvid", filename="model_quantized.ftz")
# model = fasttext.load_model(model_path)

texts = [
    "Bom dia, como é que está?",
    "Bom dia, como você está?",
    "O governo português anunciou novas medidas para combater a inflação.",
    "O presidente Lula viajou para Brasília ontem à noite.",
]

# You might wanna do this
# texts = [t.replace("\n", " ") for t in texts]

labels, probs = model.predict(texts)

for text, label, prob in zip(texts, labels, probs):
    print(f"{label[0]:20s} ({prob[0]:.4f}) | {text}")

# __label__PT_PT       (0.9895) | Bom dia, como é que está?
# __label__PT_BR       (0.9794) | Bom dia, como você está?
# __label__PT_PT       (0.8577) | O governo português anunciou novas medidas para combater a inflação.
# __label__PT_BR       (0.9803) | O presidente Lula viajou para Brasília ontem à noite.

Training

Scripts for training, data download, evals, it's all on GitHub

Data

Trained on ~6M text chunks from bastao/VeraCruz_PT-BR, balanced across PT-PT and PT-BR.

Params

Evaluation

Evaluated on the same benchmarks as PTVid paper:

• DSL-TL — Discriminating between Similar Languages, True Labels. A shared task benchmark of journalistic texts for discriminating Portuguese varieties (official site).
• FRMT — Few-shot Region-aware Machine Translation. A dataset of human-translated sentences with explicit regional variety labels (paper).

Results

All metrics are PT-PT F1 scores. Speed measured on Apple M3 Max. Full evaluation script: eval_all.py.

Limitations

• Trained primarily on web-crawled text
• Very short text is unrealiable

Citation

If you use this model, please cite:

@misc{euptvid2026,
  author = {Duarte O. Carmo},
  title = {fasttext-euptvid: Fast Portuguese Variety Identification},
  year = {2026},
  url = {https://huggingface.co/duarteocarmo/fasttext-euptvid}
}

fastText

@article{joulin2016bag,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}