UTC-DeBERTa-large-v2

UTC-DeBERTa-large - universal token classifier

🚀 Meet the second version of our prompt-tuned universal token classification model 🚀

This line of models can perform various information extraction tasks by analysing input prompts and recognizing parts of texts that satisfy prompts. In comparison with the first version, the second one is more general and can recognised as entities, whole sentences, and even paragraphs.

To use a model, just specify a prompt, for example : “Identify all positive aspects of the product mentioned by John: “ and put your target text.

This is a model based on DeBERTaV3-large that was trained on multiple token classification tasks or tasks that can be represented in this way.

Such multi-task fine-tuning enabled better generalization; even small models can be used for zero-shot named entity recognition and demonstrate good performance on reading comprehension tasks.

The model can be used for the following tasks:

• Named entity recognition (NER);
• Open information extraction;
• Question answering;
• Relation extraction;
• Coreference resolution;
• Text cleaning;
• Summarization;

How to use

There are few ways how you can use this model, one of the way is to utilize token-classification pipeline from transformers:

from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

def process(text, prompt, treshold=0.5):
  """
  Processes text by preparing prompt and adjusting indices.
  
  Args:
    text (str): The text to process
    prompt (str): The prompt to prepend to the text
    
  Returns:
    list: A list of dicts with adjusted spans and scores
  """
  # Concatenate text and prompt for full input
  input_ = f"{prompt}\n{text}" 
  results = nlp(input_) # Run NLP on full input
  processed_results = []
  prompt_length = len(prompt) # Get prompt length
  for result in results:
    # check whether score is higher than treshold
    if result['score']<treshold:
        continue
    # Adjust indices by subtracting prompt length
    start = result['start'] - prompt_length 
    # If indexes belongs to the prompt - continue
    if start<0:
        continue
    end = result['end'] - prompt_length
    # Extract span from original text using adjusted indices
    span = text[start:end]
    # Create processed result dict
    processed_result = {
      'span': span,
      'start': start,  
      'end': end,
      'score': result['score']
    }
    processed_results.append(processed_result)
  return processed_results

tokenizer = AutoTokenizer.from_pretrained("knowledgator/UTC-DeBERTa-large-v2")
model = AutoModelForTokenClassification.from_pretrained("knowledgator/UTC-DeBERTa-large-v2")

nlp = pipeline("ner", model=model, tokenizer=tokenizer, aggregation_strategy = 'first')

To use the model for zero-shot named entity recognition, we recommend to utilize the following prompt:

prompt = """Identify the following entity classes in the text:
computer

Text:
"""
text = """Apple was founded as Apple Computer Company on April 1, 1976, by Steve Wozniak, Steve Jobs (1955–2011) and Ronald Wayne to develop and sell Wozniak's Apple I personal computer.
It was incorporated by Jobs and Wozniak as Apple Computer, Inc. in 1977. The company's second computer, the Apple II, became a best seller and one of the first mass-produced microcomputers.
Apple went public in 1980 to instant financial success."""

results = process(text, prompt)

print(results)

To use the model for open information extracttion, put any prompt you want:

prompt = """Extract all positive aspects about the product
"""
text = """I recently purchased the Sony WH-1000XM4 Wireless Noise-Canceling Headphones from Amazon and I must say, I'm thoroughly impressed. The package arrived in New York within 2 days, thanks to Amazon Prime's expedited shipping.

The headphones themselves are remarkable. The noise-canceling feature works like a charm in the bustling city environment, and the 30-hour battery life means I don't have to charge them every day. Connecting them to my Samsung Galaxy S21 was a breeze, and the sound quality is second to none.

I also appreciated the customer service from Amazon when I had a question about the warranty. They responded within an hour and provided all the information I needed.

However, the headphones did not come with a hard case, which was listed in the product description. I contacted Amazon, and they offered a 10% discount on my next purchase as an apology.

Overall, I'd give these headphones a 4.5/5 rating and highly recommend them to anyone looking for top-notch quality in both product and service."""

results = process(text, prompt)

print(results)

To try the model in question answering, just specify question and text passage:

question = """Who are the founders of Microsoft?"""

text = """Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975 to develop and sell BASIC interpreters for the Altair 8800.
During his career at Microsoft, Gates held the positions of chairman, chief executive officer, president and chief software architect, while also being the largest individual shareholder until May 2014."""

input_ = f"{question} {text}"

results = process(text, question)

print(results)

For the text cleaning, please, specify the following prompt, it will recognize the part of the text that should be erased:

prompt = """Clean the following text extracted from the web matching not relevant parts:"""

text = """The mechanism of action was characterized using native mass spectrometry, the thermal shift-binding assay, and enzymatic kinetic studies (Figure ). In the native mass spectrometry binding assay, compound 23R showed dose-dependent binding to SARS-CoV-2 Mpro, similar to the positive control GC376, with a binding stoichiometry of one drug per monomer (Figure A).
Similarly, compound 23R showed dose-dependent stabilization of the SARS-CoV-2 Mpro in the thermal shift binding assay with an apparent Kd value of 9.43 μM, a 9.3-fold decrease compared to ML188 (1) (Figure B). In the enzymatic kinetic studies, 23R was shown to be a noncovalent inhibitor with a Ki value of 0.07 μM (Figure C, D top and middle panels). In comparison, the Ki for the parent compound ML188 (1) is 2.29 μM.
The Lineweaver–Burk or double-reciprocal plot with different compound concentrations yielded an intercept at the Y-axis, suggesting that 23R is a competitive inhibitor similar to ML188 (1) (Figure C, D bottom panel). Buy our T-shirts for the lowerst prices you can find!!!  Overall, the enzymatic kinetic studies confirmed that compound 23R is a noncovalent inhibitor of SARS-CoV-2 Mpro."""

results = process(text, prompt)

print(results)

It's possible to use the model for relation extraction, it allows in N*C operations to extract all relations between entities, where N - number of entities and C - number of classes:

rex_prompt="""
Identify target entity given the following relation: "{}" and the following source entity: "{}"

Text:
"""

text = """Dr. Paul Hammond, a renowned neurologist at Johns Hopkins University, has recently published a paper in the prestigious journal "Nature Neuroscience". """

entity = "Paul Hammond"

relation = "worked at"

prompt = rex_prompt.format(relation, entity)

results = process(text, prompt)

print(results)

To find similar entities in the text, consider the following example:

ent_prompt = "Find all '{}' mentions in the text:"

text = """Several studies have reported its pharmacological activities, including anti-inflammatory, antimicrobial, and antitumoral effects. The effect of E-anethole was studied in the osteosarcoma MG-63 cell line, and the antiproliferative activity was evaluated by an MTT assay. It showed a GI50 value of 60.25 μM with apoptosis induction through the mitochondrial-mediated pathway. Additionally, it induced cell cycle arrest at the G0/G1 phase, up-regulated the expression of p53, caspase-3, and caspase-9, and down-regulated Bcl-xL expression. Moreover, the antitumoral activity of anethole was assessed against oral tumor Ca9-22 cells, and the cytotoxic effects were evaluated by MTT and LDH assays. It demonstrated a LD50 value of 8 μM, and cellular proliferation was 42.7% and 5.2% at anethole concentrations of 3 μM and 30 μM, respectively. It was reported that it could selectively and in a dose-dependent manner decrease cell proliferation and induce apoptosis, as well as induce autophagy, decrease ROS production, and increase glutathione activity. The cytotoxic effect was mediated through NF-kB, MAP kinases, Wnt, caspase-3 and -9, and PARP1 pathways. Additionally, treatment with anethole inhibited cyclin D1 oncogene expression, increased cyclin-dependent kinase inhibitor p21WAF1, up-regulated p53 expression, and inhibited the EMT markers."""

entity = "anethole"

prompt = ent_prompt.format(entity)

results = process(text, prompt)

print(results)

We significantly improved model summarization abilities in comparison to the first version, below is an example:

prompt = "Summarize the following text, highlighting the most important sentences:"

text = """Apple was founded as Apple Computer Company on April 1, 1976, by Steve Wozniak, Steve Jobs (1955–2011) and Ronald Wayne to develop and sell Wozniak's Apple I personal computer. It was incorporated by Jobs and Wozniak as Apple Computer, Inc. in 1977. The company's second computer, the Apple II, became a best seller and one of the first mass-produced microcomputers. Apple went public in 1980 to instant financial success. The company developed computers featuring innovative graphical user interfaces, including the 1984 original Macintosh, announced that year in a critically acclaimed advertisement called "1984". By 1985, the high cost of its products, and power struggles between executives, caused problems. Wozniak stepped back from Apple and pursued other ventures, while Jobs resigned and founded NeXT, taking some Apple employees with him. 
Apple Inc. is an American multinational technology company headquartered in Cupertino, California. Apple is the world's largest technology company by revenue, with US$394.3 billion in 2022 revenue. As of March 2023, Apple is the world's biggest company by market capitalization. As of June 2022, Apple is the fourth-largest personal computer vendor by unit sales and the second-largest mobile phone manufacturer in the world. It is considered one of the Big Five American information technology companies, alongside Alphabet (parent company of Google), Amazon, Meta Platforms, and Microsoft. 
As the market for personal computers expanded and evolved throughout the 1990s, Apple lost considerable market share to the lower-priced duopoly of the Microsoft Windows operating system on Intel-powered PC clones (also known as "Wintel"). In 1997, weeks away from bankruptcy, the company bought NeXT to resolve Apple's unsuccessful operating system strategy and entice Jobs back to the company. Over the next decade, Jobs guided Apple back to profitability through a number of tactics including introducing the iMac, iPod, iPhone and iPad to critical acclaim, launching the "Think different" campaign and other memorable advertising campaigns, opening the Apple Store retail chain, and acquiring numerous companies to broaden the company's product portfolio. When Jobs resigned in 2011 for health reasons, and died two months later, he was succeeded as CEO by Tim Cook"""

results = process(text, prompt)

print(results)

How to run with utca:

First of all, you need to install the package:

pip install utca -U

After that you to create predictor that will run UTC model:

from utca.core import (
    AddData,
    RenameAttribute,
    Flush
)
from utca.implementation.predictors import (
    TokenSearcherPredictor, TokenSearcherPredictorConfig
)
from utca.implementation.tasks import (
    TokenSearcherNER,
    TokenSearcherNERPostprocessor,
)

predictor = TokenSearcherPredictor(
    TokenSearcherPredictorConfig(
        device="cuda:0",
        model="knowledgator/UTC-DeBERTa-large-v2"
    )
)

For NER model you should create the following pipeline:

ner_task = TokenSearcherNER(
    predictor=predictor,
    postprocess=[TokenSearcherNERPostprocessor(
        threshold=0.5
    )]
)

ner_task = TokenSearcherNER()

pipeline = (        
    AddData({"labels": ["scientist", "university", "city"]})         
    | ner_task
    | Flush(keys=["labels"])
    | RenameAttribute("output", "entities")
)

And after that you can put your text for prediction and run the pipeline:

res = pipeline.run({
    "text": """Dr. Paul Hammond, a renowned neurologist at Johns Hopkins University, has recently published a paper in the prestigious journal "Nature Neuroscience". 
His research focuses on a rare genetic mutation, found in less than 0.01% of the population, that appears to prevent the development of Alzheimer's disease. Collaborating with researchers at the University of California, San Francisco, the team is now working to understand the mechanism by which this mutation confers its protective effect. 
Funded by the National Institutes of Health, their research could potentially open new avenues for Alzheimer's treatment."""
})

To use utca for relation extraction construct the following pipeline:

from utca.implementation.tasks import (
    TokenSearcherNER,
    TokenSearcherNERPostprocessor,
    TokenSearcherRelationExtraction,
    TokenSearcherRelationExtractionPostprocessor,
)

pipe = (
    TokenSearcherNER( # TokenSearcherNER task produces classified entities that will be at the "output" key.
        predictor=predictor,
        postprocess=TokenSearcherNERPostprocessor(
            threshold=0.5 # Entity threshold
        )
    ) 
    | RenameAttribute("output", "entities") # Rename output entities from TokenSearcherNER task to use them as inputs in TokenSearcherRelationExtraction
    | TokenSearcherRelationExtraction( # TokenSearcherRelationExtraction is used for relation extraction.
        predictor=predictor,
        postprocess=TokenSearcherRelationExtractionPostprocessor(
            threshold=0.5 # Relation threshold
        )
    )
)

To run pipeline you need to specify parameters for entities and relations:

r = pipe.run({
    "text": text, # Text to process
    "labels": [ # Labels used by TokenSearcherNER for entity extraction
        "scientist",
        "university",
        "city",
        "research",
        "journal",
    ],
    "relations": [{ # Relation parameters
        "relation": "published at", # Relation label. Required parameter.
        "pairs_filter": [("scientist", "journal")], # Optional parameter. It specifies possible members of relations by their entity labels.
        # Here, "scientist" is the entity label of the source, and "journal" is the target's entity label.
        # If provided, only specified pairs will be returned.
    },{
        "relation": "worked at",
        "pairs_filter": [("scientist", "university"), ("scientist", "other")],
        "distance_threshold": 100, # Optional parameter. It specifies the max distance between spans in the text (i.e., the end of the span that is closer to the start of the text and the start of the next one).
    }]
})

print(r["output"])

Benchmarking

Below is a table that highlights the performance of UTC models on the CrossNER dataset. The values represent the Micro F1 scores, with the estimation done at the word level.

Future reading

Check our blogpost - "As GPT4 but for token classification", where we highlighted possible use-cases of the model and why next-token prediction is not the only way to achive amazing zero-shot capabilites. While most of the AI industry is focused on generative AI and decoder-based models, we are committed to developing encoder-based models. We aim to achieve the same level of generalization for such models as their decoder brothers. Encoders have several wonderful properties, such as bidirectional attention, and they are the best choice for many information extraction tasks in terms of efficiency and controllability.

Feedback

We value your input! Share your feedback and suggestions to help us improve our models. Fill out the feedback form

Join Our Discord

Connect with our community on Discord for news, support, and discussion about our models. Join Discord