pnas-witness-llm

Model Card for pnas-witness-llm

This repository contains the model weights and tokenizer for the best model reported in the paper and used in the second experiment (i.e., deberta-v3-large).

For details on how to reproduce this model and the other variants, see our GitHub page here.

Model Sources

• Repository: https://github.com/davedgd/pnas-witness-llm
• Paper: AI assistance improves people’s ability to distinguish correct from incorrect eyewitness lineup identifications

Usage Example

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TextClassificationPipeline
import numpy as np

model_path = 'dobolyilab/pnas-witness-llm'
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSequenceClassification.from_pretrained(model_path)

# provide one or more concatenated input strings consisting of verbal confidence and justification statements
new_data = [
    {'text': "I'm not certain. Maybe familiar."}, 
    {'text': "Pretty certain. His face looks very familiar."}
    ]

pipe = TextClassificationPipeline(model = model, tokenizer = tokenizer, top_k = None)
raw_probs = pipe(new_data, batch_size = 16) # increment batch size depending upon hardware

probs = np.array([[i['score'] for i in item if i['label'] == model.config.id2label[1]][0] for item in raw_probs])
preds = np.where(probs >= 0.5, 1, 0)
labels = np.array([model.config.id2label[i] for i in preds])

print('Probs:', probs)
print('Preds:', preds)
print('Labels:', labels)

Result:

Device set to use mps:0
Probs: [0.26009524 0.67521948]
Preds: [0 1]
Labels: ['Inaccurate' 'Accurate']