YearlyWord2Vec

Yearly Word2Vec Embeddings (2005-2025)

Word2Vec models trained on single-year web data from the FineWeb dataset, capturing 21 years of language evolution.

Overview

This collection enables research into semantic change, concept emergence, and language evolution over time. Each model is trained exclusively on data from a single year, providing precise temporal snapshots of language.

Dataset: FineWeb

Models are trained on the FineWeb dataset, filtered by year from URLs to create single-year subsets spanning 2005-2025.

Usage

Installation

pip install gensim numpy

Download from Hugging Face Hub

from huggingface_hub import snapshot_download
from gensim.models import KeyedVectors

repo = snapshot_download("adameubanks/YearlyWord2Vec")
model_2020 = KeyedVectors.load(f"{repo}/word2vec-2020/word2vec_2020.model")

from gensim.models import KeyedVectors

# Load a model for a specific year
model_2020 = KeyedVectors.load("word2vec_2020.model")
model_2024 = KeyedVectors.load("word2vec_2024.model")

# Find similar words
print(model_2020.most_similar("covid"))
print(model_2024.most_similar("covid"))

# Compare semantic drift
word = "technology"
similar_2020 = model_2020.most_similar(word)
similar_2024 = model_2024.most_similar(word)

Temporal Analysis

# Study semantic drift over time
years = [2005, 2010, 2015, 2020, 2025]
models = {}

for year in years:
    models[year] = KeyedVectors.load(f"word2vec_{year}.model")

# Analyze how a word's meaning changed
word = "smartphone"
for year in years:
    similar = models[year].most_similar(word, topn=5)
    print(f"{year}: {[w for w, s in similar]}")

🚀 Interactive Demo

Explore temporal embeddings interactively!

https://adameubanks.github.io/embeddings-over-time/

Compare how word meanings evolved across different years with our interactive visualization tool.

Corpus Statistics by Year

Model Architecture

All models use the same Word2Vec architecture with consistent hyperparameters:

• Embedding Dimension: 300
• Window Size: 15
• Min Count: 30
• Max Vocabulary Size: 50,000
• Negative Samples: 15
• Training Epochs: 20
• Workers: 48
• Batch Size: 100,000
• Training Algorithm: Skip-gram with negative sampling

FineWeb data processed with Trafilatura extraction, English filtering (score > 0.65), quality filters, and MinHash deduplication. Training uses 48 workers on multi-core CPU systems.

Evaluation

Models evaluated on WordSim-353 (similarity) and Google analogies datasets. Recent years show improved similarity performance with larger corpora.

Model Cards

Individual model cards available for each year (2005-2025) at: https://huggingface.co/adameubanks/YearlyWord2Vec

Research Applications

Yearly embeddings enable research in semantic change, cultural shifts, discourse evolution, and concept emergence across time periods.

Citation

If you use these models in your research, please cite:

@misc{yearly_word2vec_2025,
  title={Yearly Word2Vec Embeddings: Language Evolution from 2005-2025},
  author={Adam Eubanks},
  year={2025},
  url={https://huggingface.co/adameubanks/YearlyWord2Vec},
  note={Trained on FineWeb dataset with single-year segmentation}
}

FineWeb Dataset Citation:

@inproceedings{
  penedo2024the,
  title={The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale},
  author={Guilherme Penedo and Hynek Kydl{\'\i}{\v{c}}ek and Loubna Ben allal and Anton Lozhkov and Margaret Mitchell and Colin Raffel and Leandro Von Werra and Thomas Wolf},
  booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2024},
  url={https://openreview.net/forum?id=n6SCkn2QaG}
}

Contributing

Report issues, suggest improvements, or share research findings using these models.

License

MIT License. See LICENSE for details.