molcrawl-molecule-nat-lang-bert-small

⚠️ Pretraining degeneracy (audit 2026-05-18): empirical inspection shows this checkpoint's encoder is largely collapsed: pair-wise within-sequence hidden-state cosines hover at ≈ 0.999 and the MLM head returns nearly the same top-k tokens regardless of context. The model nominally achieved a low MLM eval_loss but appears to have settled on a degenerate "predict the most frequent token" strategy. Root cause traced to an under-sized BERT pretrain corpus (training_ready_hf_dataset ≈ 4k rows vs ≈ 3.3M available in arrow_splits/). Not recommended for downstream use as-is; consider re-training from arrow_splits/ instead. (Note: the matching -large variant exhibits an even more severe collapse and was therefore not uploaded.)

molcrawl-molecule-nat-lang-bert-small

🚫 Inference is currently unusable on this checkpoint (audit 2026-06-22). Two compounding issues:

1. Tokenizer / model id-space mismatch — the model was trained with a legacy hash-based MinimalTokenizer (vocab_size ≈ 50000, padded to 50008 in config.json), but the tokenizer files saved alongside the checkpoint are the current MoleculeNatLangTokenizer wrapping a standard GPT-2 BPE (vocab_size = 50257). Token ids produced by the saved tokenizer do not map to the same semantics the model was trained on, so any AutoModel.forward(...) returns meaningless outputs.
2. mask_token = None — the saved tokenizer_config.json does not register a mask token, so AutoModelForMaskedLM inference that follows the conventional tokenizer.mask_token workflow raises immediately.

The matching -medium variant does not have the id-space mismatch (its 50264 model vocab is the padded version of GPT-2's 50257). A retrained replacement for -small is in the pipeline and will be pushed in a subsequent release. Do not use this checkpoint for inference or downstream fine-tuning in its current state.

Model Description

BERT-small foundation model (~125M parameters, 12-layer / 768-hidden / 12-head topology, sized to match GPT-2-small) pre-trained on molecule-related natural language text. This -small checkpoint was originally trained with a legacy hash-based MinimalTokenizer (model embedding vocab_size=50008); the GPT-2 BPE tokenizer files currently bundled with the checkpoint do not match the model's id-space — see the warning above for the practical impact.

• Model Type: bert
• Data Type: Molecule-NL
• Training Date: 2026-04-24

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

model = AutoModelForMaskedLM.from_pretrained("kojima-lab/molcrawl-molecule-nat-lang-bert-small")
tokenizer = AutoTokenizer.from_pretrained("kojima-lab/molcrawl-molecule-nat-lang-bert-small")

# Predict masked token
# Use tokenizer.mask_token instead of hardcoded "[MASK]":
# BERT-style tokenizers vary ("[MASK]", "<mask>", etc.)
if tokenizer.mask_token is None:
    raise ValueError("This tokenizer has no mask_token; masked LM inference is not supported.")
prompt = "your input {MASK} sequence".replace("{MASK}", tokenizer.mask_token)
inputs = tokenizer(prompt, return_tensors="pt")
mask_index = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]

with torch.no_grad():
    outputs = model(**inputs)
logits = outputs.logits

predicted_token_id = logits[0, mask_index].argmax(dim=-1)
predicted_token = tokenizer.decode(predicted_token_id)
result = prompt.replace(tokenizer.mask_token, predicted_token)
print(f"Predicted: {result}")

Source Code

Training pipeline, configuration files, and data preparation scripts are available in the MolCrawl GitHub repository: https://github.com/mmai-framework-lab/MolCrawl

License

This model is released under the APACHE-2.0 license.

Citation

If you use this model, please cite:

@misc{molcrawl_molecule_nat_lang_bert_small,
  title={molcrawl-molecule-nat-lang-bert-small},
  author={{RIKEN}},
  year={2026},
  publisher={{Hugging Face}},
  url={{https://huggingface.co/kojima-lab/molcrawl-molecule-nat-lang-bert-small}}
}