a_lol_seq_good_prioirty_qlora_hh

HH-RLHF QLoRA adapters trained with Advantage Leftover Lunch RL Sequence (A-LoL seq.)

| Paper | Code |

Model description

We continued QLoRA finetuning on HH-RLHF dataset starting from timdettmers/qlora-hh-rlhf-7b for 1 epoch while only the "chosen" responses and removing the "rejected" responses from the training split. Even within the chosen responses, our method Advantage Leftover Lunch RL (A-LoL), inherently finds 33% of the responses as negative advantage and thus discards them as unfit for training. Despite the low number of training examples, the final adapter trained with A-LoL seq. is able to generate most diverse, safe and helpful responses compared to the baselines.

⚠️ These models are purely intended for research purposes and could produce problematic outputs.

Usage

tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b")
tokenizer.truncation_side = "left"
tokenizer.padding_side = "left"
base_model = AutoModelForCausalLM.from_pretrained(
    "huggyllama/llama-7b",
    load_in_4bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
    max_memory= {i: '40000MB' for i in range(torch.cuda.device_count())},
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.float16,
        bnb_4bit_quant_type='nf4'
    ),
)
model = PeftModel.from_pretrained(base_model, "abaheti95/a_lol_seq_good_prioirty_qlora_hh")

prefix = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n### Human: How would I go about making homemade chewing gum? ### Assistant: Well, it\u2019s pretty easy, but you can either make it a bit better or more interesting if you use xanthan gum, which is a commonly used thickener. Xanthan gum is added to food products as a thickener, gelling agent, emulsifier, and stabilizer, which means that it allows sugar syrups to form gels. If you can get hold of some, you could use it to make really interesting gum! ### Human: Okay what do I do when I have the xanthan gum? What else do I need to add? ### Assistant: Well, the important ingredient is the gums.  You can find many different types at your local pharmacy, which should be able to help you figure out what you need.  You will need:  1) a mixer, 2) sugar, 3) xanthan gum, 4) flavoring, 5) gum base, and 6) a dehydrator, which is a special oven that dries food. ### Human: Why would I need a dehydrator to make gum? ### Assistant:"

current_batch_prefixes_inputs = tokenizer([prefix], max_length = 768 - 128,truncation = True,add_special_tokens=True, padding = True, return_tensors="pt").to(model.device)
# 835 = "###"
predicted_sents = model.generate(**current_batch_prefixes_inputs, max_new_tokens=128,pad_token_id=tokenizer.pad_token_id,num_beams=1,do_sample=False,num_return_sequences = 1, eos_token_id=[835, tokenizer.eos_token_id])
repsonse_tokens = predicted_sents[:, current_batch_prefixes_inputs['input_ids'].shape[-1]:]
responses = tokenizer.batch_decode(repsonse_tokens, skip_special_tokens=True)
# Normalize responses
responses_normalized = [resp.split("\n Human:")[0].split("\nHuman:")[0].split("\n### Human")[0].strip() for resp in responses]
responses_normalized = [resp.replace("###", "").strip() if resp.endswith("###") else resp.strip() for resp in responses_normalized]

We also show the evaluation results of the model on the test set in file: harmless_base_eval_results.jsonl, helpful_base_eval_results.jsonl, helpful_online_eval_results.jsonl and helpful_rejection_eval_results.jsonl.

Framework version and configuration

• PEFT 0.5.0

The following bitsandbytes quantization config was used during training:

• quant_method: bitsandbytes
• load_in_8bit: False
• load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: False
• bnb_4bit_compute_dtype: float16