UTC-T5-large

UTC-T5-large - universal token classificator

🚀 Meet the first prompt-tuned universal token classification model 🚀

This is a model based on flan-T5-large that was trained on multiple token classification tasks or tasks that can be represented in this way.

Such multi-task fine-tuning enabled better generalization; even small models can be used for zero-shot named entity recognition and demonstrate good performance on reading comprehension tasks.

The model can be used for the following tasks:

• Named entity recognition (NER);
• Question answering;
• Relation extraction;
• Coreference resolution;
• Text cleaning;
• Summarization;

How to use

We recommend to use the model with transformers ner pipeline:

from transformers import AutoTokenizer, T5PreTrainedModel, T5Config, T5EncoderModel
from transformers.modeling_outputs import TokenClassifierOutput
from typing import Union, Optional, Tuple
from transformers import pipeline
import torch

class T5EncoderForTokenClassification(T5PreTrainedModel):
    _tied_weights_keys = ["encoder.embed_tokens.weight"]

    def __init__(self, config: T5Config):
        super().__init__(config)
        self.transformer = T5EncoderModel(config)
        self.classification_head = torch.nn.Linear(config.hidden_size, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()

        self.model_parallel = False

    def forward(
        self,
        input_ids: Optional[torch.LongTensor] = None,
        attention_mask: Optional[torch.FloatTensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        inputs_embeds: Optional[torch.FloatTensor] = None,
        labels: Optional[torch.LongTensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, TokenClassifierOutput]:
        r"""
        Returns:

        Example:

        ```python
        >>> from transformers import AutoTokenizer, MTxEncoderForTokenClassification

        >>> tokenizer = AutoTokenizer.from_pretrained("MTx-small")
        >>> model = MTxEncoderForTokenClassification.from_pretrained("MTx-small")
        >>> input_ids = tokenizer(
        ...     "Studies have been shown that owning a dog is good for you", return_tensors="pt"
        ... ).input_ids  # Batch size 1
        >>> outputs = model(input_ids=input_ids)
        >>> logits = outputs.logits
        ```"""
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.transformer(
            input_ids,
            attention_mask=attention_mask,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        sequence_output = outputs[0]

        logits = self.classification_head(sequence_output)

        loss = None
        if labels is not None:
            labels = labels.to(logits.device)
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.config.num_labels), labels.view(-1))

        if not return_dict:
            output = (logits,) + outputs[1:]
            return ((loss,) + output) if loss is not None else output

        return TokenClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

def process(text, prompt, treshold=0.5):
  """
  Processes text by preparing prompt and adjusting indices.
  
  Args:
    text (str): The text to process
    prompt (str): The prompt to prepend to the text
    
  Returns:
    list: A list of dicts with adjusted spans and scores
  """

  # Concatenate text and prompt for full input
  input_ = f"{prompt}\n{text}" 
  
  results = nlp(input_) # Run NLP on full input
  
  processed_results = []

  prompt_length = len(prompt) # Get prompt length
  
  for result in results:
    # check whether score is higher than treshold
    if result['score']<treshold:
        continue
    # Adjust indices by subtracting prompt length
    start = result['start'] - prompt_length 

    # If indexes belongs to the prompt - continue
    if start<0:
        continue

    end = result['end'] - prompt_length
    
    # Extract span from original text using adjusted indices
    span = text[start:end]

    # Create processed result dict
    processed_result = {
      'span': span,
      'start': start,  
      'end': end,
      'score': result['score']
    }

    processed_results.append(processed_result)

  return processed_results

tokenizer = AutoTokenizer.from_pretrained("knowledgator/UTC-T5-large")
model = T5EncoderForTokenClassification.from_pretrained("knowledgator/UTC-T5-large")

nlp = pipeline("ner", model=model, tokenizer=tokenizer, aggregation_strategy = 'first')

To use the model for zero-shot named entity recognition, we recommend to utilize the following prompt:

prompt = """Identify the following entity classes in the text:
computer

Text:
"""
text = """Apple was founded as Apple Computer Company on April 1, 1976, by Steve Wozniak, Steve Jobs (1955–2011) and Ronald Wayne to develop and sell Wozniak's Apple I personal computer.
It was incorporated by Jobs and Wozniak as Apple Computer, Inc. in 1977. The company's second computer, the Apple II, became a best seller and one of the first mass-produced microcomputers.
Apple went public in 1980 to instant financial success."""

results = process(text, prompt)

print(results)

To try the model in question answering, just specify question and text passage:

question = """Who are the founders of Microsoft?"""

text = """Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975 to develop and sell BASIC interpreters for the Altair 8800.
During his career at Microsoft, Gates held the positions of chairman, chief executive officer, president and chief software architect, while also being the largest individual shareholder until May 2014."""

input_ = f"{question} {text}"

results = process(text, question)

print(results)

For the text cleaning, please, specify the following prompt, it will recognize the part of the text that should be erased:

prompt = """Clean the following text extracted from the web matching not relevant parts:"""

text = """The mechanism of action was characterized using native mass spectrometry, the thermal shift-binding assay, and enzymatic kinetic studies (Figure ). In the native mass spectrometry binding assay, compound 23R showed dose-dependent binding to SARS-CoV-2 Mpro, similar to the positive control GC376, with a binding stoichiometry of one drug per monomer (Figure A).
Similarly, compound 23R showed dose-dependent stabilization of the SARS-CoV-2 Mpro in the thermal shift binding assay with an apparent Kd value of 9.43 μM, a 9.3-fold decrease compared to ML188 (1) (Figure B). In the enzymatic kinetic studies, 23R was shown to be a noncovalent inhibitor with a Ki value of 0.07 μM (Figure C, D top and middle panels). In comparison, the Ki for the parent compound ML188 (1) is 2.29 μM.
The Lineweaver–Burk or double-reciprocal plot with different compound concentrations yielded an intercept at the Y-axis, suggesting that 23R is a competitive inhibitor similar to ML188 (1) (Figure C, D bottom panel). Buy our T-shirts for the lowerst prices you can find!!!  Overall, the enzymatic kinetic studies confirmed that compound 23R is a noncovalent inhibitor of SARS-CoV-2 Mpro."""

results = process(text, prompt)

print(results)

It's possible to use the model for relation extraction, it allows in N*C operations to extract all relations between entities, where N - number of entities and C - number of classes:

rex_prompt="""
Identify target entity given the following relation: "{}" and the following source entity: "{}"

Text:
"""

text = """Dr. Paul Hammond, a renowned neurologist at Johns Hopkins University, has recently published a paper in the prestigious journal "Nature Neuroscience". """

entity = "Paul Hammond"

relation = "worked at"

prompt = rex_prompt.format(relation, entity)

results = process(text, prompt)

print(results)

To find similar entities in the text, consider the following example:

ent_prompt = "Find all '{}' mentions in the text:"

text = """Several studies have reported its pharmacological activities, including anti-inflammatory, antimicrobial, and antitumoral effects. The effect of E-anethole was studied in the osteosarcoma MG-63 cell line, and the antiproliferative activity was evaluated by an MTT assay. It showed a GI50 value of 60.25 μM with apoptosis induction through the mitochondrial-mediated pathway. Additionally, it induced cell cycle arrest at the G0/G1 phase, up-regulated the expression of p53, caspase-3, and caspase-9, and down-regulated Bcl-xL expression. Moreover, the antitumoral activity of anethole was assessed against oral tumor Ca9-22 cells, and the cytotoxic effects were evaluated by MTT and LDH assays. It demonstrated a LD50 value of 8 μM, and cellular proliferation was 42.7% and 5.2% at anethole concentrations of 3 μM and 30 μM, respectively. It was reported that it could selectively and in a dose-dependent manner decrease cell proliferation and induce apoptosis, as well as induce autophagy, decrease ROS production, and increase glutathione activity. The cytotoxic effect was mediated through NF-kB, MAP kinases, Wnt, caspase-3 and -9, and PARP1 pathways. Additionally, treatment with anethole inhibited cyclin D1 oncogene expression, increased cyclin-dependent kinase inhibitor p21WAF1, up-regulated p53 expression, and inhibited the EMT markers."""

entity = "anethole"

prompt = ent_prompt.format(entity)

results = process(text, prompt)

print(results)

Currently summarization with UTC model works purely, however, we want to highlight the potential of such approach and use cases when it's beneficial:

prompt = "Summarize the following text, highlighting the most important sentences:"

text = """Apple was founded as Apple Computer Company on April 1, 1976, by Steve Wozniak, Steve Jobs (1955–2011) and Ronald Wayne to develop and sell Wozniak's Apple I personal computer. It was incorporated by Jobs and Wozniak as Apple Computer, Inc. in 1977. The company's second computer, the Apple II, became a best seller and one of the first mass-produced microcomputers. Apple went public in 1980 to instant financial success. The company developed computers featuring innovative graphical user interfaces, including the 1984 original Macintosh, announced that year in a critically acclaimed advertisement called "1984". By 1985, the high cost of its products, and power struggles between executives, caused problems. Wozniak stepped back from Apple and pursued other ventures, while Jobs resigned and founded NeXT, taking some Apple employees with him. 
Apple Inc. is an American multinational technology company headquartered in Cupertino, California. Apple is the world's largest technology company by revenue, with US$394.3 billion in 2022 revenue. As of March 2023, Apple is the world's biggest company by market capitalization. As of June 2022, Apple is the fourth-largest personal computer vendor by unit sales and the second-largest mobile phone manufacturer in the world. It is considered one of the Big Five American information technology companies, alongside Alphabet (parent company of Google), Amazon, Meta Platforms, and Microsoft. 
As the market for personal computers expanded and evolved throughout the 1990s, Apple lost considerable market share to the lower-priced duopoly of the Microsoft Windows operating system on Intel-powered PC clones (also known as "Wintel"). In 1997, weeks away from bankruptcy, the company bought NeXT to resolve Apple's unsuccessful operating system strategy and entice Jobs back to the company. Over the next decade, Jobs guided Apple back to profitability through a number of tactics including introducing the iMac, iPod, iPhone and iPad to critical acclaim, launching the "Think different" campaign and other memorable advertising campaigns, opening the Apple Store retail chain, and acquiring numerous companies to broaden the company's product portfolio. When Jobs resigned in 2011 for health reasons, and died two months later, he was succeeded as CEO by Tim Cook"""

results = process(text, prompt)

print(results)

Future reading

Check our blogpost - "As GPT4 but for token classification", where we highlighted possible use-cases of the model and why next-token prediction is not the only way to achive amazing zero-shot capabilites. While most of the AI industry is focused on generative AI and decoder-based models, we are committed to developing encoder-based models. We aim to achieve the same level of generalization for such models as their decoder brothers. Encoders have several wonderful properties, such as bidirectional attention, and they are the best choice for many information extraction tasks in terms of efficiency and controllability.

Feedback

We value your input! Share your feedback and suggestions to help us improve our models. Fill out the feedback form

Join Our Discord

Connect with our community on Discord for news, support, and discussion about our models. Join Discord