Phi3.1-Simple-Arguments

Phi3.1 Simple Arguments

This model aims to parse simple english arguments, arguments formed of two premises and a conclusion, including two propositions.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: Cristian Desivo
• Model type: LLM
• Language(s) (NLP): English
• License: Apache-2.0
• Finetuned from model: Phi3.1-mini

Model Sources

<!-- Provide the basic links for the model. -->

• Repository: TBD
• Demo: TBD

Quantization

<!-- - **Q4_K_M.gguf** https://huggingface.co/cris177/Qwen2-Simple-Arguments/resolve/main/Qwen2_arguments.Q4_K_M.gguf?download=true -->

Usage

Below we share some code snippets on how to get quickly started with running the model.

llama.cpp server [Recommended]

The recommended way of running the model is with a llama.cpp server running the quantized

Then you can use the following script to use the server's model for inference:

import json
import requests

def llmCall(messages, **args):
    url = "http://localhost:8080/v1/chat/completions"
    headers = {
        "Content-Type": "application/json"
    }
    data = {
        'messages': messages
    }
    for arg in args:
        data[arg] = args[arg]
    response = requests.post(url, headers=headers, json=data)
    return response.json()

def analyze_argument(argument):
    instruction = "Based on the following argument, identify the following elements: premises, conclusion, propositions, type of argument, negation of propositions and validity."
    inputText = "### Input:\n" + argument
    prompt = f"""{instruction}

    {inputText}
    """
    messages=[{"role":"user", "content":prompt}]
    properties = {
        "Premise 1": {"type": "string"},
        "Premise 2": {"type": "string"},
        "Conclusion": {"type": "string"},
        "Type of argument": {"type": "string"},
        "Proposition 1": {"type": "string"},
        "Proposition 2": {"type": "string"},
        "Negation of Proposition 1": {"type": "string"},
        "Negation of Proposition 2": {"type": "string"},
        "Validity": {"type": "string"},
    }
    analysis = llmCall(
        messages=messages,
        max_tokens=1000,
        temperature=0,
        stop=["<|end|>"],
        response_format={
            "type": "json_object",
            "schema": {
                "type": "object",
                "properties": properties,
                "required": list(properties.keys()),
            },
        }
        
        )['choices'][0]['message']['content']
    if analysis.endswith("<|end|>"):
        analysis = analysis[:-5]
    return analysis
argument = "If it's wednesday it's cold, and it's cold, therefore it's wednesday."
output = analyze_argument("If it's wednesday it's cold, and it's cold, therefore it's wednesday.")
print(output)

Output:

{"Premise 1": "If it's wednesday it's cold",
"Premise 2": "It's cold",
"Conclusion": "It is Wednesday",
"Proposition 1": "It is Wednesday",
"Proposition 2": "It is cold",
"Type of argument": "affirming the consequent",
"Negation of Proposition 1": "It is not Wednesday",
"Negation of Proposition 2": "It is not cold",
"Validity": true}

transformers 🤗

First make sure to pip install -U transformers, then use the code below replacing the argument variable for the argument you want to parse:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("cris177/Phi3.1-Simple-Arguments", 
    device_map="auto",)
tokenizer = AutoTokenizer.from_pretrained("cris177/Phi3.1-Simple-Arguments")

argument = "If it's wednesday it's cold, and it's cold, therefore it's wednesday."

instruction = 'Based on the following argument, identify the following elements: premises, conclusion, propositions, type of argument, negation of propositions and validity.'
alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:"""
prompt = alpaca_prompt.format(instruction, argument)
input_ids = tokenizer(prompt, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_length=1000, num_return_sequences=1)
print(tokenizer.decode(outputs[0]))

Output:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Based on the following argument, identify the following elements: premises, conclusion, propositions, type of argument, negation of propositions and validity.

### Input:
If it's wednesday it's cold, and it's cold, therefore it's wednesday.

### Response: 
{"Premise 1": "If it's wednesday it's cold",
"Premise 2": "It's cold",
"Conclusion": "It is Wednesday",
"Proposition 1": "It is Wednesday",
"Proposition 2": "It is cold",
"Type of argument": "affirming the consequent",
"Negation of Proposition 1": "It is not Wednesday",
"Negation of Proposition 2": "It is not cold",
"Validity": "false"}<|endoftext|>

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The model was trained on syntethic data, based on the following types of arguments:

• Modus Ponen
• Modus Tollen
• Affirming Consequent
• Disjunctive Syllogism
• Denying Antecedent
• Invalid Conditional Syllogism

Each argument was constructed by selecting two random propositions (from a list of 400 propositions that was generated beforehand), choosing a type of argument and combining it all with randomly selected connectors (therefore, since, hence, thus, etc).

50k arguments were created to train the model, and 100 to test.

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Training

We used unsloth for memory reduced sped up training.

We trained for one epoch.

Less than 3.5 GB of VRAM were used for training, and it took 3 hours.

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

The model obtains 100% train and test accuracy on our synthetic dataset.