24679-Project1-BERT

Model Card: Arxiv AI User Preference Classifier

Application Link

Demo Space: Link to Arxiv AI App Demo Space (e.g., Hugging Face Space)

Model Details

Intended Uses

This classifier is the trigger mechanism for the personalized recommendation engine within the Arxiv AI application.

Core Recommendation Engine Activation

After the user has liked at least one paper, the core recommendation engine is activated when they click the "Save Ratings and Get More Papers" button. By navigating to the Automated Bert Recommendation tab, this classifier's prediction (the "Like" action) is used to analyze the vector embeddings of the liked papers to understand the user's emerging interests. It then generates a new, more refined list of ten papers based on a sophisticated 70/30 split:

• 70% "Exploitation" recommendations: Chosen for their high cosine similarity to the user's selections and representing topics the system is confident the user will like.
• 30% "Exploration" recommendations: Purposefully selected from related fields to introduce novelty and prevent the user's search from becoming too narrow.

This new list is enhanced with transparent justifications; each paper card now includes a "Reason for Recommendation" tag, with messages like "Recommended because you liked 'Attention Is All You Need'" for exploitation picks, or "Exploratory pick from the related field of Computational Linguistics" for exploration ones. This entire process is a continuous loop, allowing the user to progressively refine their recommendations with each interaction.

• Out-of-Scope Use: Should not be used for general-purpose text classification outside the domain of Arxiv papers, for non-English text, or for making critical, high-stakes decisions.

Training Data

The model was trained on a proprietary dataset of user preference data collected by Zachary Zdobinski and Je Choi for the Arxiv AI application.

Data Fields

Evaluation

Evaluation was performed on the test set of 43 examples at epoch 9.0.

Area Under the Curve (AUC)

Environmental Impact

Bias, Risks, and Limitations

• Bias from Data: The model is inherently biased toward the specific preference patterns of the two data creators, Zachary Zdobinski and Je Choi. It will not generalize well to other users without further fine-tuning or a multi-user context.
• Domain Limitation: Performance is expected to degrade significantly on documents outside the topical distribution of the Arxiv papers used in the original data collection.
• Generalization: The model size is relatively small (603 total examples), which may lead to overfitting and poor generalization to truly novel Arxiv papers.

Citation

@misc{arxiv_preference_model_zzd_jch,
  author = {Zdobinski, Zachary and Choi, Je},
  title = {Arxiv AI Application User Preference Data and Classification Model},
  howpublished = {Internal Project Documentation},
  year = {[Information Needed: Year of Release/Creation]} 
}