TiME-da-xs

TiME Danish (da, xs)

Monolingual BERT-style encoder that outputs embeddings for Danish. Distilled from FacebookAI/xlm-roberta-large.

Specs

• language: Danish (da)
• size: xs
• architecture: BERT encoder
• layers: 4
• hidden size: 384
• intermediate size: 1536

Usage (mean pooled embeddings)

from transformers import AutoTokenizer, AutoModel
import torch

repo = "dschulmeist/TiME-da-xs"
tok = AutoTokenizer.from_pretrained(repo)
mdl = AutoModel.from_pretrained(repo)

def mean_pool(last_hidden_state, attention_mask):
    mask = attention_mask.unsqueeze(-1).type_as(last_hidden_state)
    return (last_hidden_state * mask).sum(1) / mask.sum(1).clamp(min=1e-9)

inputs = tok(["example sentence"], padding=True, truncation=True, return_tensors="pt")
outputs = mdl(**inputs)
emb = mean_pool(outputs.last_hidden_state, inputs['attention_mask'])
print(emb.shape)