WaterBERT

WaterBERT

WaterBERT is a BERT-base encoder for wastewater- and water-treatment literature. It was obtained by continuing masked-language-model (MLM) pretraining of SciBERT (scivocab, uncased) on a domain corpus of water-treatment research text. The vocabulary is SciBERT's (31,090 tokens); only the weights were adapted.

It is the shared backbone of WaterBERT-NER and WaterBERT-RE, and is intended as a starting point for fine-tuning on water-treatment NLP tasks (entity recognition, relation extraction, classification, retrieval).

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="Mudi12137/WaterBERT")
fill("The Fenton process generates hydroxyl radicals from hydrogen peroxide and [MASK] ions.")
# top prediction: "iron"

For fine-tuning, load it like any BERT checkpoint:

from transformers import AutoModelForTokenClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Mudi12137/WaterBERT")
model = AutoModelForTokenClassification.from_pretrained("Mudi12137/WaterBERT", num_labels=13)

Training

The pretraining corpus and its construction are described in the accompanying paper.

Limitations

• English only, lower-cased input (the tokenizer lower-cases automatically).
• The corpus is research literature (mostly abstracts); performance on other genres such as plant operation logs or regulations has not been evaluated.

License

Apache-2.0, the same as the SciBERT base model.

Citation

See CITATION.cff in the project repository.