electra_small_generator

See our collection for all versions of ELECTRA.

Run ELECTRA with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/electra_small_generator

Paper: ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555) · HF Papers

ELECTRA is Google's BERT-style bidirectional text encoder, pre-trained as a replaced-token discriminator (with a smaller generator producing the corrupted tokens). This repo is the masked-LM (fill-mask) checkpoint. WordPiece tokenizer; mask token [MASK].

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of google/electra-small-generator for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

✨ Quick start (masked-LM (fill-mask))

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from zeromodels.models.electra import ElectraMaskedLM, ElectraTokenizer

mlm = ElectraMaskedLM.from_weights("zeromodels/electra_small_generator")
tokenizer = ElectraTokenizer.from_weights("zeromodels/electra_small_generator")

inputs = tokenizer("The capital of France is [MASK].")
logits = mlm(inputs)  # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.decode([int(logits[0, mask].argmax())]))

Load any ELECTRA variant the same way with from_weights("zeromodels/<variant>"):

Available classes

Load any of these from this repo with from_weights("zeromodels/electra_small_generator") (or on the fly via the hf: prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a hf: fine-tune).

from zeromodels.models.electra import ElectraMaskedLM
model = ElectraMaskedLM.from_weights("zeromodels/electra_small_generator")

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• Prefer ElectraTokenizer.from_weights(...) so WordPiece tokenization matches.
• Downstream tasks (classification / QA / NER) use the discriminator repos; the generator repos are the masked-LM.
• See ELECTRA docs and Loading Weights.
• Community / upstream safetensors still work via the hf: prefix, e.g. ElectraModel.from_weights("hf:google/electra-small-generator").

Special Thanks

A huge thank you to the Google ELECTRA authors for creating and releasing these models.

License: Apache 2.0.