Qwen3-14B-chem-dyn-tokenizer

Model Card for summykai/Qwen3-14B-chem-dyn-tokenizer

A Qwen3-14B model augmented with the InternS1 dynamic tokenizer for chemistry/biomolecular text. No full fine-tune was performed; we adapted tokenization and warmed the new token rows so the model is ready for downstream SFT/RL on chemistry tasks.

<p align="center"> <img src="figures/compression_interns1.png" alt="InternS1 tokenizer compression" width="550"/> </p>

Figure: Tokenization efficiency from the InternS1 technical report — SMILES/IUPAC compression ~2.64× vs ~1.44–1.51× for common LLM tokenizers.

Model Details

• Developed by: Summykai
• License: Apache-2.0
• Finetuned from: Qwen/Qwen3-14B
• Model type: QwenForCausalLM
• Languages: Multilingual (inherits from Qwen3)

What changed vs. base Qwen3-14B?

Dynamic tokenizer integration (InternS1)
We ship tokenization_interns1.py and three SentencePiece models:

• tokenizer_SMILES.model (SMILES/SELFIES)
• tokenizer_IUPAC.model (IUPAC names)
• tokenizer_FASTA.model (protein sequences)

The tokenizer auto-detects long chemistry/biomolecular spans and switches to the SP model, or you can wrap with explicit tags:

<SMILES> CC(=O)OC1=CC=CC=C1C(=O)O </SMILES>
<IUPAC> 2-acetoxybenzoic acid </IUPAC>

Logical auto-detect tokens (e.g., <SMILES_AUTO_DETECT>…</SMILES_AUTO_DETECT>) do not consume ids.

Effective vocabulary size: 152,971.

Warm-up of new rows (no full finetune):

• Transformer layers frozen; trained only new embedding rows (+ matching lm_head rows if untied).
• Cross-entropy objective; ~1,500 steps; ~3,986,558 token exposures.
• Stability aids: fp32 CE, grad-clip=1.0, small LR.

Initialization strategy (better than random):
New rows were not left random. We projected domain embeddings into Qwen3’s space (ridge-regularized linear map) to give chemistry tokens meaningful initial locations. After merging, test_merge.py verified max |Δ| emb/head = 0.0.

How to Use

Install: RDKit is strongly recommended for high-quality SMILES auto-detection.

pip install rdkit-pypi sentencepiece transformers
# or via conda:
# conda install -c conda-forge rdkit

from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "summykai/Qwen3-14B-chem-dyn-tokenizer"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, torch_dtype="auto")
prompt = "Name and SMILES for aspirin: <SMILES>CC(=O)OC1=CC=CC=C1C(=O)O</SMILES>"
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))

Files

• Model: model-*.safetensors, model.safetensors.index.json, config.json, generation_config.json
• Tokenizer: vocab.json, merges.txt, tokenizer_config.json, special_tokens_map.json, tokenization_interns1.py, tokenizer_SMILES.model, tokenizer_IUPAC.model, tokenizer_FASTA.model
• Figures: figures/compression_interns1.png

Acknowledgements

• Intern / Shanghai AI Lab — InternS1 tokenizer and report
• Qwen Team — Qwen3-14B base model

Contact

Summykai (HF & GitHub): https://github.com/Summykai