topicmodel-multilingual

REDUP multilingual topic model

Multilingual topic model that maps a document to a 125-dimensional topic distribution. The repository contains inference weights and per-language BPE tokenizers for 100 languages.

The dump is built for BigARTM 0.9.2. It is not a Transformers / Diffusers checkpoint: AutoModel and AutoTokenizer are not supported.

Related repositories

Use text_categorization as the historical training reference. For serving embeddings and explanations in production, prefer redup.python.topicmodel.

Repository contents

Total size is about 251 MB.

Languages

Each document must provide modality lang with an ISO-like code from the table below. Tokens are scored under BigARTM class id @{lang}.

How to download

pip install huggingface_hub

from huggingface_hub import snapshot_download

root = snapshot_download(repo_id="redup-ai/topicmodel-multilingual")

Usage

Recommended path: the inference helpers from redup.python.topicmodel.

pip install huggingface_hub bigartm==0.9.2
# install redup-topicmodel from the service repository / package index you use

import asyncio
from types import SimpleNamespace

from redup_topicmodel.topicmodel.worker import TopicModel


def document(document_id: str, tokens: list[str], lang: str):
    return SimpleNamespace(
        document_id=document_id,
        tokens=tokens,
        modalities={"lang": lang},
    )


async def main(root: str):
    model = TopicModel({
        "artifact_root": root,
    })
    pack = SimpleNamespace(documents=[
        document("doc-en", ["hello", "world"], "en"),
        document("doc-ru", ["привет", "мир"], "ru"),
    ])
    result = await model.get_documents_embedding("example", pack)
    for embedding in result["embeddings"]:
        # length-125 topic distribution
        print(len(embedding["values"]), sum(embedding["values"]))


asyncio.run(main(root))

gRPC service

Point the service config at the downloaded directory:

TopicModel:
  artifact_root: /path/from/snapshot_download

Tokenizer only

from redup_topicmodel.topicmodel.bpe import Tokenizers

tokenizers = Tokenizers.load(f"{root}/tokenizers.json.gz")
print(tokenizers["en"].encode("hello world"))

Limitations

• Requires BigARTM and the redup inference stack; not compatible with transformers.AutoModel / AutoTokenizer.
• parameters.bin is a BigARTM pickle — load only trusted artifacts.
• tokenizers.json.gz stores independent BPE packs per language; it is not a Hugging Face tokenizer.json export.
• Short or out-of-vocabulary inputs may produce degenerate topic mass (for example concentrated on topic_0).

License

MIT. See LICENSE.