ofeklubaton-interest-rate-prediction-model

Overiew

This project implements a semantic course recommendation system that helps users discover relevant learning resources based on free-text learning goals. By comparing embeddings between user input and course descriptions, the system retrieves the most relevant courses and uses a language model to generate a structured and personalized learning path. The final output is a clear, goal-oriented learning roadmap presented through a simple and intuitive interface.

Dataset

The dataset is a synthetically generated collection of 10,000 online courses created specifically for content-based recommendation tasks. It was generated using the mistralai/Mistral-7B-Instruct-v0.2 language model, ensuring consistent structure, high semantic quality, and broad topical coverage. The dataset is well suited for experiments involving embeddings, semantic similarity, and text-based recommendation systems.

Exploratory Data Analysis (EDA) During the EDA stage, we aimed to gain a high-level understanding of the synthetic course dataset and its structural properties. We examined the main dataset attributes and explored distributions related to course difficulty, duration, pricing, ratings, and topical coverage in order to characterize the data, even though these features are not directly used in the semantic similarity model. A missing value analysis showed that the dataset is largely complete, with most fields fully populated and only minor sparsity in optional descriptive attributes. These missing values are not critical, as the recommendation system relies primarily on semantic embeddings derived from course descriptions. In addition, we performed several data quality checks to validate the dataset integrity. We verified that all entries follow a consistent schema, contain valid textual descriptions, and align with the expected course structure. Overall, the EDA confirmed that the dataset is clean, well-structured, and suitable for downstream tasks involving embeddings, semantic similarity, and recommendation pipelines.

Outlier Analysis for price: <img src="https://cdn-uploads.huggingface.co/production/uploads/6914689a0152a9457fa7ec88/3_QQFt-12hfKHEb2j_2v1.png" width="700"> Most courses are priced within a typical low-to-mid range, while a small number of courses appear at extremely high prices. These high-priced outliers are rare but likely represent premium offerings rather than data issues, and therefore reflect meaningful variation in the market.

Price distribution by duration: <img src="https://cdn-uploads.huggingface.co/production/uploads/6914689a0152a9457fa7ec88/bS4sbFsLJx9G0MIoABc4U.png" width="700"> Course price shows only a weak relationship with duration, with wide price variation across all course lengths. This suggests that pricing is driven more by course content and perceived value than by duration alone.

Price Variance Across Topics for Similar Duration: <img src="https://cdn-uploads.huggingface.co/production/uploads/6914689a0152a9457fa7ec88/1P5_P_83QwY-IvWNkL695.png" width="700"> Course pricing varies significantly by topic even when duration is fixed. This indicates that pricing is driven primarily by subject value and market positioning rather than by course length.

Course Ratings by Difficulty and Duration: <img src="https://cdn-uploads.huggingface.co/production/uploads/6914689a0152a9457fa7ec88/pdBXKGDfHI9msZBbcLllf.png" width="700"> Advanced courses receive consistently higher ratings across all durations, while beginner ratings decline as course length increases. This suggests that satisfaction depends more on alignment between learner level and course depth than on duration alone.

Topic Difficulty Composition: <img src="https://cdn-uploads.huggingface.co/production/uploads/6914689a0152a9457fa7ec88/UCZBbGQrVzI3CFkjFzNUb.png" width="700"> Topics with a balanced mix of beginner, intermediate, and advanced courses tend to support higher pricing. This suggests that pricing is linked to long-term learning progression and depth rather than accessibility alone.

Embeddings Each course description is converted into a numerical embedding that captures its semantic meaning. We encode course descriptions using a transformer-based text embedding model, mapping all courses into a shared vector space where semantically similar courses are located closer together. All embeddings are L2-normalized, allowing efficient similarity computation using cosine similarity. Embeddings are computed once, stored with course metadata, and reused during inference to support fast and scalable recommendation.

Model Selection Several embedding models were evaluated to balance semantic quality, computational efficiency, and retrieval performance. Each model was compared based on embedding size, similarity distribution, retrieval accuracy (topic-hit@10), and query latency. Although some models achieved higher raw retrieval scores, MiniLM was selected as the final model due to its well-balanced embedding space, which provides clearer separation between courses while maintaining low computational cost and fast inference. This balance makes MiniLM suitable for a practical recommendation system that requires both accuracy and efficiency.