Inverse-LLaMA3.1-8B-BlackBox

Inverse-LLaMA-3.1-BlackBox-8B-LoRA-Adapter

<!-- Provide a quick summary of what the model is/does. -->

This model is a fine-tuned version of meta-llama/Llama-3.1-8B on the inv_qwen_inf-ins_660k dataset for paper Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts.

Model Description

LoRA Adapter for Inverse-LLaMA-3.1-8B. Please use with the original LLaMA3.1-8B base model.

Uses

Your input should in the form of a model's response, and this inverse model will generate its corresponding input prompt for you.

For instance:

User Input:

Based on the detailed evaluation guideline and format requirement you provided, I'm now evaluating factual consistency of the following summaries to the articles with a score between 0 and 1:

```json
{
  \"article\": \"Gymnast brinn bevan believes he still has a `` small chance '' of competing in the olympic games in rio next summer . Media playback is not supported on this device . The 18-year-old from essex broke the tibia and fibula in his left leg after landing on a concrete floor while vaulting in november . Less than a month before his injury , bevan was part of the first british team to claim a world championships medal , when they won silver in glasgow . `` it hurt my head and heart knowing what i had done , '' he told bbc look east . `` my coach tried to jump in front of me to stop me from seeing what had happened - but i had already seen it . '' Bevan won a series of junior titles before he claimed bronze in the pommel horse at the 2015 european games in baku . He had hoped to continue that progress at the olympics in brazil but , with only 247 days until the games , he knows it will be a challenge to be ready in time . `` the most important thing for me is getting back to full fitness so i can carry on my gymnastics career , '' he said . `` i do n't think the olympic games is fully out of reach , it is going to be tough , but there is still a small chance i can push for a place in that team . '\",\\
  \"summary\": \"Olympic bronze medallist aneurin bevan says he still has a `` small chance '' of competing at rio 2016 despite breaking his leg .\",\\
  \"consistency_score\": 0.0
}
```

Inverse Model Output:

As a highly skilled and meticulous professional fact-checker, your role involves rigorously evaluating the factual consistency of the provided summaries with the respective articles. This evaluation should be performed with a detailed assessment, assigning a score between 0 and 1 based on the accuracy of the summary's alignment with the document's facts.

Your evaluation should adhere to the following format:
{
  \"article\": \"content of the article\",
  \"summary\": \"content of the summary\",
  \"consistency_score\": number between 0 and 1 representing the factual consistency of the summary with the article
}

Carefully read and analyze the entire article before determining the factual consistency score. Ensure that your analysis is thorough and nuanced, considering the intricacies of the information presented.

Article:
Gymnast brinn bevan believes he still has a `` small chance '' of competing in the olympic games in rio next summer . Media playback is not supported on this device . The 18-year-old from essex broke the tibia and fibula in his left leg after landing on a concrete floor while vaulting in november . Less than a month before his injury , bevan was part of the first british team to claim a world championships medal , when they won silver in glasgow . `` it hurt my head and heart knowing what i had done , '' he told bbc look east . `` my coach tried to jump in front of me to stop me from seeing what had happened - but i had already seen it . '' Bevan won a series of junior titles before he claimed bronze in the pommel horse at the 2015 european games in baku . He had hoped to continue that progress at the olympics in brazil but , with only 247 days until the games , he knows it will be a challenge to be ready in time . `` the most important thing for me is getting back to full fitness so i can carry on my gymnastics career , '' he said . `` i do n't think the olympic games is fully out of reach , it is going to be tough , but there is still a small chance i can push for a place in that team . 

Summary:
Olympic bronze medallist aneurin bevan says he still has a `` small chance '' of competing at rio 2016 despite breaking his leg .

Assign the appropriate score for factual consistency between the summary and the article.

Please substitute the samples in the prompt and then you can get the general template.

Direct Use

Due to the format of training samples, please use the corresponding chat template when trying to generate inverse prompts. Here is an example:

base_model_path = "meta-llama/Llama-3.1-8B"
lora_adapter_path = "kou199024/Inverse-LLaMA3.1-8B-BlackBox"
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(model=base_model_path, enable_lora=True, max_lora_rank=256)
prompt = '''
Based on the detailed evaluation guideline and format requirement you provided, I’m now evaluating consistency of the following summaries to the articles with a score between 0 and 1:
```json {{
  "article": "A woman who was allegedly raped and abused by eight men in rotherham changed from a “ lovely girl to an animal ” , her mother told jurors . The witness also said her family had been forced to move to spain to escape her daughter’s alleged abusers. Sheffield crown court also heard how police lost tapes of an interview with defendant sageer hussain in 2003. Eight men, including mr hussain, deny sexually abusing three girls between 1999 and 2003. The mother of one of the alleged victims said in a statement: “her character changed from a lovely girl to an animal. She became horrible.” She said at one stage she discovered a mobile phone in her daughter’s bedroom and rang a number stored under the name ’waleed’. She said a man picked up the phone and said “i ain’t done owt, i ain’t touched her. It isn’t me”. When she asked her daughter about the phone she said she burst into tears and said “they’re raping me, they’re raping me”. She told the court after her daughter went to the police in 2003 her family were repeatedly threatened. “We were so distraught that we sold the business and the home and moved to spain,” she said. Det con andy stephanek, of south yorkshire police, told the court the force had lost the tape of an interview with mr hussain when he was first questioned about the allegations. He said it appeared that “due to the passage of time they’ve been destroyed”. The trial continues.",
  "summary": "The mother of a girl accused of being sexually abused by a gang of men has told a court her daughter’s character changed from “a lovely girl to an animal”.",
  "consistency_score": 0.6666666666666666
}} ```
'''
messages = [
    {"role": "user", "content": prompt}
]
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, max_tokens=8192)

tokenizer = AutoTokenizer.from_pretrained(base_model_path)
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,  # Set to False to strictly disable thinking
)

outputs = llm.generate(
    [text],
    sampling_params,
    lora_request=LoRARequest("inverse_adapter", 1, lora_adapter_path)
)

print(outputs[0].outputs[0].text)

Training Details

Training Procedure

Training Hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 32
• total_train_batch_size: 1024
• total_eval_batch_size: 4
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.95) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3.0

Framework versions

• PEFT 0.12.0
• Transformers 4.46.1
• Pytorch 2.4.0+cu121
• Datasets 3.1.0
• Tokenizers 0.20.3

---

license: apache-2.0