LLama3-reviewClassifier

This is a Llama 3:8b based fine-tuned model used for for classifying reviews as useful for improvement of an application or not. A review was considered useful if it contained something useful for the further development or improvement of the application, that is, a user story, bug report, or other specific problems that users have. It does not include general complaints such as “I hate this app”, or encouragements such as “I like your payment feature”, even if these encouragements are directed at a specific feature of the app.

The idea is to use this model as part of a pipeline, where you classify reviews as 'useful' or not, and then you can further tag the useful reviews according to your need and store them. This allows your organisation to send the right reviews to different departments in your organisation to be handled, for example feature requests and bug reports. This model outperformed Chat-bison for this classification task if you weigh the result higher for recall. It had an F1-weighted score of 0.90 for the 550 reviews it was tested against, with an F1-score of 0.92 for the negative class and F1-score of 0.87 for the positive class.

The model is fine-tuned to answers "True" or "False" for inputted reviews. It was trained using unsloth AI, based on Llama-3-8b-bnb-4bit, also using TRL SFTTrainer for 10 epochs with a validation loss of 0.0235. The Lora-adaption weights for this model are also available on my profile.

It was trained on real reviews from the Google Play store using the following prompt:

""" You are an AI trained to classify user reviews for a company called Kivra, which offers a digital mailbox service featuring invoice payment, digital signing, mail scanning, simple bookkeeping, and digital receipts.

Your task is to analyze each user review and determine whether it contains useful information such as feature requests, bug reports, or other valuable feedback that can aid the developers in improving the service. You will classify the review as 'True' if it contains such useful information, and 'False' if it does not.

Classification Guidelines:

• Classify a review as 'True' if it suggests a new feature, reports a bug, complains in a way that highlights a problem, or provides any other information that could lead to an improvement of the service.
• Classify as 'False' if the review only praises the service, comments on its current state without constructive feedback, or is irrelevant to service development.

Example Classifications:

• Review: "It's difficult to increase text size" Classification: True Reason: The review suggests a feature to increase text size, which is useful feedback for enhancing readability.

• Review: "This is the best app to pay bills" Classification: False Reason: The review praises the service but lacks suggestions for improvement or reports of issues.

• Review: "The app crashes every time I try to upload a document." Classification: True Reason: The review reports a specific bug, providing critical information for troubleshooting and improvement."""