legal-bert-tokenizer

dominguesm/legal-bert-tokenizer

legal-bert-tokenizer is a machine learning model from dominguesm. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc-by-sa-4.0.

LegalBERT tokenizer is a word level byte-pair encoding with vocabulary size of 52k tokens (containing the most common words in legal documents), based on the BERTimbau tokenizer. The tokenizer was trained on data prov…

Downloads · 30 days

Access

Public

Updated Mar 13, 2023

Repo size

—

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.json1.2 MB · 75%

At a glance

License: cc-by-sa-4.0
Access: Public
Created: Nov 15, 2022
Updated: Mar 13, 2023
SHA: d1eb0f02

Model card

From the Hugging Face model README

LegalBERT Tokenizer

LegalBERT tokenizer is a word level byte-pair encoding with vocabulary size of 52k tokens (containing the most common words in legal documents), based on the BERTimbau tokenizer. The tokenizer was trained on data provided by the BRAZILIAN SUPREME FEDERAL TRIBUNAL, through the terms of use: LREC 2020. Tokenizer utilize BertTokenizer implementation from transformers.

NOTE: The results of this project do not imply in any way the position of the BRAZILIAN SUPREME FEDERAL TRIBUNAL, all being the sole and exclusive responsibility of the author.

Tokenizer usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dominguesm/legal-bert-tokenizer")

example = ""
tokens = tokenizer.tokenize(example)

Comparison of results

Original Text: De ordem, a Secretaria Judiciária do Supremo Tribunal Federal INTIMA a parte abaixo identificada, ou quem as suas vezes fizer, do inteiro teor do(a) despacho/decisão presente nos autos (art. 270 do Código de Processo Cívil e art 5º da Lei 11.419/2006).

Tokenizer	Tokens	Num. Tokens
BERTimbau	['De', 'ordem', ',', 'a', 'Secretaria', 'Judic', '##iária', 'do', 'Supremo', 'Tribunal', 'Federal', 'IN', '##TI', '##MA', 'a', 'parte', 'abaixo', 'identificada', ',', 'ou', 'quem', 'as', 'suas', 'vezes', 'fiz', '##er', ',', 'do', 'inteiro', 'teor', 'do', '(', 'a', ')', 'despa', '##cho', '/', 'decisão', 'presente', 'nos', 'auto', '##s', '(', 'art', '.', '27', '##0', 'do', 'Código', 'de', 'Processo', 'Cí', '##vil', 'e', 'art', '[UNK]', 'da', 'Lei', '11', '.', '41', '##9', '/', '2006', ')', '.']	66
LegalBERT	`['De', 'ordem', ',', 'a', 'Secretaria', 'Judiciária', 'do', 'Supremo', 'Tribunal', 'Federal', 'INTIMA', 'a', 'parte', 'abaixo', 'identificada', ',', 'ou', 'quem', 'as', 'suas', 'vezes', 'fizer', ',', 'do', 'inteiro', 'teor', 'do', '(', 'a', ')', 'despacho', '/', 'decisão', 'presente', 'nos', 'autos', '(', 'art', '.', '270', 'do', 'Código', 'de', 'Processo', 'Cív', '##il', 'e', 'art', '5º', 'da', 'Lei', '11', '.', '419', '/', '2006', ')', '.']`	58

Citation

If you use this tokenizer, please cite:

@misc {maicon_domingues_2022,
	author       = { {Maicon Domingues} },
	title        = { legal-bert-tokenizer (Revision d8e9d4a) },
	year         = 2022,
	url          = { https://huggingface.co/dominguesm/legal-bert-tokenizer },
	doi          = { 10.57967/hf/0110 },
	publisher    = { Hugging Face }
}

Contacts:

<a href="mailto:[email protected]">[email protected]</a>
NLP.ROCKS

License: cc-by-sa-4.0
Languages: pt
Created: Nov 15, 2022
Updated: Mar 13, 2023