Downloads · 30 days
0
bert-astronomy-tokenizer
vraj1/bert-astronomy-tokenizer
bert-astronomy-tokenizer is a machine learning model from vraj1. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
WordPiece tokenizer (30k vocab) shared across all astronomy models
Access
Public
Updated Dec 1, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1.3 MB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Dec 1, 2025
- Updated
- Dec 1, 2025
- SHA
- 52d283e1
Model card
From the Hugging Face model README
bert-astronomy-tokenizer
Description
WordPiece tokenizer (30k vocab) shared across all astronomy models
Tokenizer Details
- Type: WordPiece (BERT-style)
- Vocabulary Size: 30,000 tokens
- Special Tokens:
[PAD],[UNK],[CLS],[SEP],[MASK] - Trained On: 95,000 Wikipedia documents (full corpus train split)
- Normalization: Lowercase, NFD, strip accents
- Max Length: 256 tokens
Usage
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast.from_pretrained("vraj1/bert-astronomy-tokenizer")
# Tokenize text
text = "The Hubble telescope orbits Earth."
tokens = tokenizer.tokenize(text)
print(tokens)
# Output: ['the', 'hub', '##ble', 'telescope', 'orbit', '##s', 'earth', '.']
Research Context
This tokenizer is part of a research project studying the effect of corpus composition on language model performance.
Project: Effect of Corpus on Language Model Performance
Institution: [Your University]
Course: NLP - Master's Computer Science
Date: November 2024
- License
- mit
- Languages
- en
- Created
- Dec 1, 2025
- Updated
- Dec 1, 2025