Phi-3-14b-stance-detection

Phi-3-medium-4k-instruct for Stance Detection

Model Description

This model is a fine-tuned version of ´unsloth/Phi-3-medium-4k-instruct´ for stance detection on online discussions. Within an argument map, i.e. a tree structure representing a debate, the model can be used to identify the stance of a claim toward its parent claim. This model is part of SAGESSE, a pipeline for processing data from Reddit threads to create argument maps. In the context of SAGESSE pipeline, the claims that are processed by this stance detection model are extracted using the SAGESSE-EPFL/Mistral-7b-claims-extraction model on Reddit comments.

Training and Fine-Tuning

• Base Model: ´unsloth/Phi-3-medium-4k-instruct´
• Fine-Tuning Data:
  • Claims Extraction Dataset: ~50k claim pairs with relative stance from Kialo
  • Annotation Source: dataset extracted from the debate platform Kialo.
• Fine-Tuning Approach:
  • Technique: Parameter Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA).
  • Training Configuration:
    • Epochs: 2
    • Learning Rate: 2e-4
    • Batch Size: 2
    • Gradient Accumulation Steps: 16
    • Hardware: Single Nvidia A100 GPU with 40GB memory
    • LoRA Rank: 16
    • LoRA Alpha: 16
• Libraries Used: HuggingFace’s Transformers library

Performance

• Evaluation Metric: weighted F1 score over the three classes Positive, Negative and Neutral
• Evaluation Datasets:
• • Kialo test set
  • stances extracted from ChangeMyView v2.0 (used to assess generalization to different domain)
• Performance: Note that the performance have been tested both using directly the fine-tuned model with zero-shots prompting, and the fine-tuned model with 3 relevant shots (chosen by cosine similarity).
• Kialo test set:
  • Fine-Tuned model with zero-shots:
    • F1 Score: 0.874
  • Fine-Tuned model with 3-shots:
    • F1 Score: 0.855
• ChangeMyView:
  • Fine-Tuned model with zero-shots:
    • F1 Score: 0.755
  • Fine-Tuned model with 3-shots:
    • F1 Score: 0.740

Usage

This model is intended for use in systems that require stance detection from text, particularly in contexts like argument mapping, content moderation, or sentiment analysis.

Input Format

The model was used with the following prompt template:

Below is an instruction that describes a task, paired with an input that provides further context.
Write a response that appropriately completes the request.

### Instruction:
You are tasked with stance detection in a debate context. The provided discussions come from a variety of political subreddits on Reddit.
Each discussion includes a parent opinion and various claims that support, oppose, or are neutral to it. Your job is to determine the stance
of the given claim with respect to the parent opinion. Make sure to carefully consider the context and the content of both the parent opinion
and the claim to determine the correct stance. You have three options to choose from for determining the stance: Positive, Negative, and Neutral.

### Input:
Parent Opinion: <parent-claim>
Claim: <claim>

### Response:

Output

The model is used to generate one single token, improving throughput at inference time.

Citation

If you use this model in your research, please cite the following paper:

TBD

Contact Information

For questions or issues, please contact Matteo Santelmo at [email protected].