literary_evaluation_classifier

Model Card for GVezzani/literary_evaluation_classifier

<!-- Provide a quick summary of what the model is/does. -->

The model is a sequence classifier, finetuned on a corpus of Goodreads book reviews to identify evaluative sentences (distinguishing them from plot summaries, information about the author/genre, and, to put it simply, from any sentence where the reviewer is not explicitly assessing the value of the book).

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

Online book reviews, such as the ones that can be found on Goodreads, are extremely valuable for any scholar interested in literary reception and evaluation, in that they allow to investigate such phenomena on a very large scale and with empirical, data-oriented methodologies. Unfortunatly, they also tend to be higly messy and unstructured, causing the data that can be derived from them to be vexed by an amount of noise that can only in part be countered by increasing the size of one's dataset.

With this model, we offer a tool capable of automatically extract from online book reviews those sentences that contain en explicit evaluation of the book in quesiton. This can help researchers interested in literary evaluation by allowing them to work with cleaner data.

Examples of evaluative sentences:

"And maybe it's because I was so enamored with lines like this"

"This novel that had me swooning so hard love hearts started flickering behind my eyelids"

"Penelope and Vi write flawlessly together"

Examples of non-evaluative sentences:

"I've been trying to figure out what I want to say in this review"

"This is my 'journey' with Stuck-Up Suit"

"I'll admit, I haven't read Cocky Bastard --the first standalone novel Vi Keeland and Penelope Ward co-authored"

• Developed by: Simone Rebora & Gabriele Vezzani
• Model type: AutoModelForSequenceClassification
• Finetuned from: google-bert/bert-base-uncased
• Repository: [More Information Needed]
• Paper: [More Information Needed]

How to Get Started with the Model

Use the code below to get started with the model.

Important The model does not include a tokenizer, we recomend using the one from google-bert/bert-base-uncased.

def predict_text_class(input_text, labels, model, tokenizer): # a simple function to perform the classification

    input_ids = tokenizer(input_text, truncation=True, padding=True, return_tensors="pt")

    with torch.no_grad():
        output = model(**input_ids)

    predicted_labels = output.logits.argmax(dim=1)

    return labels[predicted_labels.item()]

checkpoint = "GVezzani/literary_evaluation_classifier"

# Load the tokenizer from google BERT model
tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

# Load the model
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)

labels = ["val", "no_val"]
sentence = 'This is my journey with Stuck-Up Suit'

#perform classification
label = predict_text_class(sentence, labels=labels, model=model, tokenizer=tokenizer)

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

[More Information Needed]

Training Hyperparameters

train_ratio = 0.9 validation_ratio = 0.1 batch_size: 12 learning_rate=1e-5, num_train_epochs=2 metric_for_best_model='wf1' weight_decay=0.01

Evaluation

          precision    recall  f1-score   support

  no_val      0.895     0.933     0.913       854
     val      0.817     0.731     0.772       349

accuracy                          0.874      1203
macro avg     0.856     0.832     0.843      1203
weighted avg  0.872     0.874     0.872      1203

Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]