fasttext-wiki-news-subwords-300

fastText Wiki News with subwords (300 dimensions)

999,999 English word vectors trained with subword information on Wikipedia 2017, the UMBC WebBase corpus and statmt.org news data (16 billion tokens). This repository holds the word vectors without the ngram vectors, so words outside the vocabulary get no vector.

This is a gensim model, packaged for fse (Fast Sentence Embeddings), which turns word vectors into sentence vectors with CBOW averaging, SIF and uSIF.

Usage

This repository uses the fse 1.x API and requires fse<2. For fse 2.0, choose a finished encoder from the classic model catalog. See the migration guide before upgrading.

pip install "fse<2"

from fse import SIF, IndexedList, Vectors

vectors = Vectors.from_pretrained("fasttext-wiki-news-subwords-300")
model = SIF(vectors)

sentences = IndexedList([["hello", "world"], ["fast", "sentence", "embeddings"]])
model.train(sentences)
model.sv[0]  # the vector of the first sentence

Vectors.from_pretrained("fasttext-wiki-news-subwords-300", mmap="r") memory-maps the vectors instead of loading them into RAM.

Files

Source and license

The vectors were published by Facebook AI Research with fastText under the Creative Commons Attribution-Share-Alike License 3.0; this redistribution is under the same license. This repository redistributes them in the gensim format.

• Mikolov et al. (2018), Advances in Pre-Training Distributed Word Representations

About fse

fse is maintained by Dr. Oliver Borchers. Source code, documentation and the STS benchmark of all vector sets in this organization: github.com/oborchers/Fast_Sentence_Embeddings.