udever-bloom-560m

Model Card for udever-bloom

<!-- Provide a quick summary of what the model is/does. -->

udever-bloom-560m is finetuned from bigscience/bloom-560m via BitFit on MS MARCO Passage Ranking, SNLI and MultiNLI data. It is a universal embedding model across tasks, natural and programming languages. (From the technical view, udever is merely with some minor improvements to sgpt-bloom)

<img width="338" height="259" src="https://user-images.githubusercontent.com/26690193/277643721-cdb7f227-cae5-40e1-b6e1-a201bde00339.png" />

Model Details

Model Description

• Developed by: Alibaba Group
• Model type: Transformer-based Language Model (decoder-only)
• Language(s) (NLP): Multiple; see bloom training data
• Finetuned from model : bigscience/bloom-560m

Model Sources

<!-- Provide the basic links for the model. -->

• Repository: github.com/izhx/uni-rep
• Paper : Language Models are Universal Embedders
• Training Date : 2023-06

Checkpoints

• udever-bloom-560m
• udever-bloom-1b1
• udever-bloom-3b
• udever-bloom-7b1

On ModelScope / 魔搭社区: udever-bloom-560m, udever-bloom-1b1, udever-bloom-3b, udever-bloom-7b1

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import AutoTokenizer, BloomModel

tokenizer = AutoTokenizer.from_pretrained('izhx/udever-bloom-560m')
model = BloomModel.from_pretrained('izhx/udever-bloom-560m')

boq, eoq, bod, eod = '[BOQ]', '[EOQ]', '[BOD]', '[EOD]'
eoq_id, eod_id = tokenizer.convert_tokens_to_ids([eoq, eod])

if tokenizer.padding_side != 'left':
    print('!!!', tokenizer.padding_side)
    tokenizer.padding_side = 'left'


def encode(texts: list, is_query: bool = True, max_length=300):
    bos = boq if is_query else bod
    eos_id = eoq_id if is_query else eod_id
    texts = [bos + t for t in texts]
    encoding = tokenizer(
        texts, truncation=True, max_length=max_length - 1, padding=True
    )
    for ids, mask in zip(encoding['input_ids'], encoding['attention_mask']):
        ids.append(eos_id)
        mask.append(1)
    inputs = tokenizer.pad(encoding, return_tensors='pt')
    with torch.inference_mode():
        outputs = model(**inputs)
        embeds = outputs.last_hidden_state[:, -1]
    return embeds

encode(['I am Bert', 'You are Elmo'])

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

• MS MARCO Passage Ranking, retrieved by (https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/ms_marco/train_bi-encoder_mnrl.py#L86)
• SNLI and MultiNLI (https://sbert.net/datasets/AllNLI.tsv.gz)

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing

MS MARCO hard negatives provided by (https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/ms_marco/train_bi-encoder_mnrl.py#L86). Negatives for SNLI and MultiNLI are randomly sampled.

Training Hyperparameters

• Training regime: tf32, BitFit
• Batch size: 1024
• Epochs: 3
• Optimizer: AdamW
• Learning rate: 1e-4
• Scheduler: constant with warmup.
• Warmup: 0.25 epoch

Evaluation

Table 1: Massive Text Embedding Benchmark MTEB

Table 2: CodeSearchNet

Table 3: Chinese multi-domain retrieval Multi-cpr

More results refer to paper section 3.

Technical Specifications

Model Architecture and Objective

• Model: bigscience/bloom-560m.
• Objective: Constrastive loss with hard negatives (refer to paper section 2.2).

Compute Infrastructure

• Nvidia A100 SXM4 80GB.
• torch 2.0.0, transformers 4.29.2.

Citation

BibTeX:

@article{zhang2023language,
  title={Language Models are Universal Embedders},
  author={Zhang, Xin and Li, Zehan and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan and Zhang, Min},
  journal={arXiv preprint arXiv:2310.08232},
  year={2023}
}