RedPajama-INCITE-Base-3B-v1-dialogue-summary-topic

Model description

The togethercomputer/RedPajama-INCITE-Base-3B-v1 model finetuned for Summary and Topic generation from a dailogue. We use a sample of roughly 1000 data points from the Dialogsum dataset for fine-tuning.

Look at the repo llm-toys for usage and other details.

Try in colab: <a target="_blank" href="https://colab.research.google.com/drive/1MSl8IDLjs3rgEv8cPHbJLR8GHh2ucT3_"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>

Installation

pip install llm-toys

from llm_toys.tasks import SummaryAndTopicGenerator

summary_topic_generator = SummaryAndTopicGenerator()
summary_topic_generator.generate_summary_and_topic(
    """
    #Person1#: I'm so excited for the premiere of the latest Studio Ghibli movie!
    #Person2#: What's got you so hyped?
    #Person1#: Studio Ghibli movies are pure magic! The animation, storytelling, everything is incredible.
    #Person2#: Which movie is it?
    #Person1#: It's called "Whisper of the Wind." It's about a girl on a magical journey to save her village.
    #Person2#: Sounds amazing! I'm in for the premiere.
    #Person1#: Great! We're in for a visual masterpiece and a heartfelt story.
    #Person2#: Can't wait to be transported to their world.
    #Person1#: It'll be an unforgettable experience, for sure!
    """.strip()
)
# {"summary": "#Person1# is excited for the premiere of the latest Studio Ghibli movie.
#              #Person1# thinks the animation, storytelling, and heartfelt story will be unforgettable.
#              #Person2# is also excited for the premiere.",
#  "topic": "Studio ghibli movie"}

Sample training data

{
  "fname": "train_664",
  "dialogue": "#Person1#: Hello, Happy Time Catering Services, Vitoria speaking. How can I help you?\n#Person2#: Hello, Victoria. This is Joe Smith from country holidays. I wondered if you could do some catering for us next week, we are having a small reception. It's to launch our summer holiday advertising campaign. Will you be free?\n#Person1#: When exactly is it? Mr. Smith?\n#Person2#: April 21st, that's Thursday. Oh, sorry, no. It should be Friday.\n#Person1#: Oh, yes I can do that where will you be holding it?\n#Person2#: We thought we'd have that at head office and use the conference room, because there is enough room for everyone there.\n#Person1#: Ok. What sort of things would you like?\n#Person2#: Just a light lunch I think, so that people can eat while they move around and talk to each other. You did some thing similar for us last year. We'd be happy to have the same menu again.\n#Person1#: Right. I'll look at my diary and see what you had last time. Oh, I nearly forgot to ask you how many should I cater for?\n#Person2#: Well, I think most people will be able to come, perhaps around 30. No, let's say 35, to be sure.\n#Person1#: Right, thank you for getting in touch, Mr. Smith. I'll send you confirmation of the arrangements by the end of this week.\n#Person2#: Ok.",
  "summary": "Joe Smith calls Happy Time Catering Service and wants some catering for next week. Victoria asks his requirements and will send him confirmation of the arrangements by the end of this week.",
  "topic": "Catering service"
}

Training params

{
  "batch_size": 1,
  "eval_ratio": 0.05,
  "eval_steps": 100,
  "gradient_accumulation_steps": 4,
  "learning_rate": 0.0001,
  "logging_steps": 100,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "lora_r": 16,
  "max_length": 1024,
  "model_name": "togethercomputer/RedPajama-INCITE-Base-3B-v1",
  "num_train_epochs": 2,
  "seed": 0,
  "task_type": "dialogue_summary_topic",
  "use_aim": True
}

Training curve

Training procedure

The following bitsandbytes quantization config was used during training:

• load_in_8bit: False
• load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: True
• bnb_4bit_compute_dtype: bfloat16

Framework versions

• PEFT 0.4.0.dev0