t5-small-nl-to-regex

t5-small-nl-to-regex

google-t5/t5-small fully fine-tuned to translate an English description of a text pattern into a regular expression, on the three DeepRegex corpora (KB13, NL-RX-Synth, NL-RX-Turk).

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("thealper2/t5-small-nl-to-regex")
model = AutoModelForSeq2SeqLM.from_pretrained("thealper2/t5-small-nl-to-regex")

PREFIX = "translate natural language to regex: "

# T5's SentencePiece vocabulary cannot represent \\ ^ ` { } ~, so the
# model emits them as '#b #c #g #l #r #n'. Decode them back:
CODEC = {"#b": "\\", "#c": "^", "#g": "`", "#l": "{", "#r": "}", "#n": "~"}

def decode_regex(text):
    out, i = [], 0
    while i < len(text):
        pair = text[i:i + 2]
        if pair in CODEC:
            out.append(CODEC[pair]); i += 2
        elif pair == "#h":
            out.append("#"); i += 2
        else:
            out.append(text[i]); i += 1
    return "".join(out)

def generate_regex(text):
    ids = tokenizer(PREFIX + text, return_tensors="pt")
    out = model.generate(**ids, max_length=128, num_beams=4, do_sample=False)
    return decode_regex(tokenizer.decode(out[0], skip_special_tokens=True))

generate_regex("lines that contain the word 'dog'")

Output dialect

Outputs follow the DeepRegex regex representation, which is not Python re syntax. It adds two operators:

Matching is whole-line (full-match) semantics. re.compile accepts these strings but reads ~ and & as literal characters, so outputs must be translated before use with Python re.

String literals follow the datasets' own preprocessing: words in quotes are normalised to the placeholders dog, truck, ring, lake in order of appearance. A description mentioning two quoted words yields a regex over dog and truck, which the caller substitutes back.

Training data

20824 examples from the three DeepRegex corpora:

Split 80/10/10 with seed 42 (train 16659 / validation 2082 / test 2083), stratified by corpus. NL-RX-Synth and NL-RX-Turk have byte-identical targ.txt files, so they are split on a shared permutation of the line index; otherwise every NL-RX-Turk test regex would also be an NL-RX-Synth training target.

127 of 2083 test regexes (6.1%) also occur as a training target; these are genuine repeats in the corpora and were not removed.

Training procedure

Full fine-tuning of all parameters — no LoRA, adapters or quantisation.

Evaluation

Held-out test split (2083 examples), beam search with num_beams=4, do_sample=False.

Metrics

• Exact match — string equality after stripping surrounding whitespace only.
• Regex validity — the output parses as a well-formed expression of the dialect.
• DFA equivalence — exact language equivalence decided with automata. Not decidable when either side contains a \b assertion, so it is reported with its coverage.
• Probe agreement — approximate: the prediction and the reference agree on every generated probe string. Finite-sample agreement, not a proof of equivalence.
• Functional accuracy — exact matches plus predictions judged equivalent by DFA equivalence, falling back to probe agreement where DFA equivalence is undecidable. The headline number; it mixes an exact and an approximate criterion and is reported as such.

Overall

Per corpus

Limitations

• Output is in the DeepRegex dialect, not Python re; ~ and & need translation before use.
• Quoted strings are placeholders (dog, truck, ring, lake), inherited from the datasets' preprocessing. The model does not copy real literals.
• The descriptions are short, single-sentence and template-like; behaviour on free-form or multi-sentence requests is untested.
• Probe agreement is finite-sample and can call two regexes equivalent when a longer counter-example exists.
• DFA equivalence cannot decide expressions containing \b, roughly 20% of the corpus.
• 60M parameters and a domain-specific vocabulary — not a general-purpose regex assistant.
• Generated regexes are not validated for catastrophic backtracking; do not run them on untrusted input without review.

Citation

Datasets from Locascio et al., Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledge (EMNLP 2016), and Kushman & Barzilay, Using Semantic Unification to Generate Regular Expressions from Natural Language (NAACL 2013).