bloom-560m-totto

thealper2/bloom-560m-totto

bigscience/bloom-560m fully fine-tuned on GEM/ToTTo for table-to-text generation: given a Wikipedia table with highlighted cells, it writes a one-sentence description of those cells.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "thealper2/bloom-560m-totto"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).eval()

table = (
    "<page_title> List of national independence days </page_title> "
    "<table> <cell> Senegal <col_header> Country </col_header> </cell> "
    "<cell> 1960 <col_header> Year celebrated </col_header> </cell> </table>"
)
prompt = 'Table:\n{table}\n\nDescription:\n'.format(table=table)

inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(
    **inputs,
    max_new_tokens=64,
    num_beams=4,
    do_sample=False,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:],
                       skip_special_tokens=True).strip())

The prompt format is part of the model: it was trained on Table:\n{table}\n\nDescription:\n followed by the description and EOS, with the loss computed on the description only. Tables are linearized in the standard ToTTo style (<page_title>, <section_title>, <cell>, <col_header>, <row_header>). Feeding a differently formatted prompt will degrade output.

Results

Deterministic beam search (num_beams=4, no sampling) on 1000 examples of the held-out test split. ToTTo's official test split ships without references, so the held-out half of the official validation split is used for reporting.

Training

• Base model: bigscience/bloom-560m (559,214,592 parameters, all of them trained - no adapters)
• Data: 30,000 ToTTo training examples, 1,000 for validation
• Epochs: 1.0 | learning rate: 2e-05 (cosine schedule, warmup ratio 0.03)
• Effective batch size: 16 (2 x 8 accumulation)
• Precision: bf16 mixed precision, fp32 master weights | optimizer: adamw_torch_fused
• Max sequence length: 512 tokens
• Final validation loss: 1.1099 | training loss: 1.2376
• Hardware: single consumer GPU, 15.32 GB peak VRAM, 0.88 h for 1,875 steps

Limitations

• English Wikipedia tables only; other domains and languages are out of distribution.
• The model describes what the highlighted cells say, it does not verify them. It can still hallucinate entities or relations that the table does not support, so do not use its output as a factual source without checking.
• It is a 560M-parameter model trained on a subset of ToTTo, not a general-purpose assistant. Outside the prompt format above it behaves like the untuned base model.
• Inherits the base model's biases and its bigscience-bloom-rail-1.0 use restrictions.

Citation

@inproceedings{parikh2020totto,
  title     = {{ToTTo}: A Controlled Table-To-Text Generation Dataset},
  author    = {Parikh, Ankur P. and Wang, Xuezhi and Gehrmann, Sebastian and
               Faruqui, Manaal and Dhingra, Bhuwan and Yang, Diyi and Das, Dipanjan},
  booktitle = {EMNLP},
  year      = {2020}
}