simctg_rocstories

This model provides a GPT-2 language model trained with SimCTG on the ROCStories benchmark (Mostafazadeh et al., 2016) based on our paper A Contrastive Framework for Neural Text Generation.

We provide a detailed tutorial on how to apply SimCTG and Contrastive Search in our project repo. In the following, we illustrate a brief tutorial on how to use our approach to perform text generation.

1. Installation of SimCTG:

pip install simctg --upgrade

2. Initialize SimCTG Model:

import torch
# load SimCTG language model
from simctg.simctggpt import SimCTGGPT
model_name = r'cambridgeltl/simctg_rocstories'
model = SimCTGGPT(model_name)
model.eval()
tokenizer = model.tokenizer

3. Prepare the Text Prefix:

prompt = r"Accident in the Lab <|endoftext|>"
print ('Prefix is: {}'.format(prompt))
tokens = model.tokenizer.tokenize(prompt)
input_ids = model.tokenizer.convert_tokens_to_ids(tokens)
input_ids = torch.LongTensor(input_ids).view(1,-1)

4. Generate Text with Contrastive Search:

beam_width, alpha, decoding_len = 5, 0.65, 45
output = model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width, 
                                       alpha=alpha, decoding_len=decoding_len) 
print("Output:\n" + 100 * '-')
print(tokenizer.decode(output).split(model.tokenizer.eos_token)[1].strip())
'''
  Prefix is: Accident in the Lab <|endoftext|>
  Output:
  ----------------------------------------------------------------------------------------------------
  Tom went to work one day. He noticed a lab accident in the lab. Tom was worried about his safety at work. 
  Unfortunately the accident didn't go well. Tom wound up leaving early to get back on the job.
'''

For more details of our work, please refer to our main project repo.

5. Citation:

If you find our paper and resources useful, please kindly leave a star and cite our paper. Thanks!

@article{su2022contrastive,
  title={A Contrastive Framework for Neural Text Generation},
  author={Su, Yixuan and Lan, Tian and Wang, Yan and Yogatama, Dani and Kong, Lingpeng and Collier, Nigel},
  journal={arXiv preprint arXiv:2202.06417},
  year={2022}
}