txtai-arxiv

arXiv txtai embeddings index

This is a txtai embeddings index for the arXiv dataset metadata.

txtai must be installed to use this model.

Example

This index can be loaded from the Hugging Face Hub with txtai as shown below.

from txtai.embeddings import Embeddings

# Load the index from the HF Hub
embeddings = Embeddings()
embeddings.load(provider="huggingface-hub", container="neuml/txtai-arxiv")

# Search for papers matching a query
embeddings.search("Survey of vector databases")

# Search for papers matching an abstract
embeddings.search("""
Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral
has the same architecture as Mistral 7B, with the difference that each
layer is composed of 8 feedforward  blocks (i.e. experts). For every
token, at each layer, a router network selects two experts to process
the current state and combine their outputs.
""")

embeddings.search("""
Humanity has wondered whether we are alone for millennia. The discovery
of life elsewhere in the Universe, particularly intelligent life, would
have profound effects, comparable to those of recognizing that the Earth
is not the center of the Universe and that humans evolved from previous
species.
""")

embeddings.search("""
The main objective of this paper is to investigate the extent to which
the margin of victory can be predicted solely by the rankings of the
opposing teams in NCAA Division I men's basketball games.
""")

Use Cases

An embeddings index generated by txtai is a fully encapsulated index format. It doesn't require a database server or dependencies outside of the Python install.

The arXiv index works well as a fact-based context source for retrieval augmented generation (RAG). In other words, search results from this model can be passed to LLM prompts as the context in which to answer questions.

Additionally, this model can identify articles to cite in research. Passing a title + abstract pair will find similar existing articles.

Build the index

The following steps show how to build this index.

• Install required build dependencies

pip install ragdata

• Follow these instructions to download the dataset

• Build txtai-arxiv index

python -m ragdata.arxiv.index \
       -d <path to directory with file downloaded in previous step> \
       -o txtai-arxiv

More information

See the following links for more information on the arXiv metadata dataset.

• Dataset on Hugging Face
• Dataset on Kaggle
• Metadata description