Boston-rideshare-pricing-prediction

🚖 Uber & Lyft Boston Rideshare Data Analysis

<p align="center"> <video width="100%" height="auto" controls autoplay muted loop> <source src="https://cdn-uploads.huggingface.co/production/uploads/69d9149c90277a9a700c29a3/aFECeaUlMClE3UvQWP_lP.mp4" type="video/mp4"> Your browser does not support the video tag. </video> </p>

Boston Ride Fare Predictor 🚕

An advanced AI system for predicting ride-sharing fares and classifying them into service tiers (Budget, Standard, or Premium). The model is based on a Random Forest Regressor trained on Boston's Uber and Lyft datasets.

🎮 Live Interactive Demo

You can test the model in real-time using the interface below:

• Live App: Boston Fare Predictor Space

📊 Dataset Overview

Background

This project explores the dynamic landscape of rideshare services in Boston, MA. The dataset provides a unique opportunity to analyze the interaction between urban mobility and environmental factors by integrating real-world rideshare transactions with high-resolution local weather data.

Data Source

The dataset is sourced from Kaggle ("Uber and Lyft Dataset Boston, MA"). It encompasses a period of rideshare activity in Boston, meticulously merged with weather metrics corresponding to the exact time and location of each trip.

Dataset Scale

• Total Observations: ~693,000 rows.
• Analysis Sample: For computational efficiency and balanced processing, a representative sample of ~50,000 rows is utilized in this project.
• Total Features: 57 initial variables, providing a comprehensive view of each journey.

---

🔍 Features & Variables

The dataset offers a rich mix of data types, making it ideal for advanced feature engineering and predictive modeling:

🔢 Numerical Features

• Trip Metrics: distance (miles), surge_multiplier.
• Weather Metrics: temp (Fahrenheit), humidity, precipIntensity, windSpeed.

🗂️ Categorical Features

• Service Providers: cab_type (Uber vs. Lyft).
• Service Levels: name (e.g., Black, XL, Shared, Lux, UberPool).
• Geography: source and destination (Specific Boston neighborhoods).
• Environment: short_summary (Weather descriptions like "Cloudy", "Rain", or "Clear").

🎯 Target Variable (Label)

• price: A continuous numerical value representing the cost of the ride.

---

❓ The Research Question

"To what extent can we accurately predict rideshare prices in Boston based on trip characteristics, service levels, and real-time weather conditions?"

🛠️ Predictive Features & Target Analysis

The following table summarizes the key variables and the central target variable used in the models:

📊 Part 2: Exploratory Data Analysis (EDA)

Before diving into deep analysis, my first priority was ensuring the dataset's integrity. This phase focused on filtering noise and verifying that my "Bright Orange" visualizations represent a clean, reliable data source.

Key Actions I Took:

• Dimensionality Reduction: I pruned the dataset from 57 initial features down to a streamlined set of high-impact variables, removing redundant IDs and timestamps.
• Missing Value Handling: I successfully addressed null values in the price column. The heatmap below confirms a 100% clean dataset ready for processing.
• Outlier Detection: By plotting Price vs. Distance, I identified and neutralized anomalies—such as ultra-short trips with unexplained high costs—ensuring my models learn from realistic transit patterns.

🌧️ The "Weather Factor" – Impact of Rain on Pricing

One of the most interesting parts of this research was investigating whether adverse weather in Boston directly translates to higher fares for passengers. I analyzed the relationship between precipitation and pricing dynamics.

• Direct Impact vs. Price: I used a Regression Plot (Regplot) to examine if higher rain intensity (precipIntensity) leads to an immediate increase in trip costs.
  • Finding: While there is a slight positive trend, rain alone isn't the primary driver of the base fare.

⏰ Hourly Trends

In this section, I analyzed how the time of day affects rideshare pricing in Boston, focusing on whether certain hours command higher fares on average.

• Hourly Price Stability: I generated a time-series line plot to track the average trip price across a 24-hour cycle.
  • Insight: My analysis shows that the average price remains remarkably stable throughout the day, suggesting that the time of day alone (without considering specific surge events) does not cause massive fluctuations in the base fare.

🌡️ Feature Correlation Analysis

In this critical stage of the EDA, I conducted a deep statistical dive into the mathematical relationships between all features—including the encoded service types—and the final trip price.

• Comprehensive Mapping: I generated an expanded Correlation Heatmap to visualize how each individual variable (from trip distance to specific vehicle tiers) influences the price.
• Key Insights: * The "Price Drivers": I identified that specific luxury service levels (like Black SUV and Lux Black XL) and the trip distance hold the highest positive correlation with the target variable.
  • Feature Pruning: By examining the inter-correlations between variables, I was able to identify and remove redundant features (multi-collinearity), ensuring a more efficient and stable machine learning model.
• Strategic Outcome: This map confirmed that while location and weather play a role, the Service Tier and Distance are the dominant mathematical anchors for pricing in the Boston market.

Brand Comparison

In this phase, I conducted a direct comparative analysis between Uber and Lyft to determine if one brand consistently outprices the other.

• Brand Parity: I utilized a Density Plot to compare the price distributions of Uber and Lyft.
  • Insight: I discovered a near-perfect overlap between the two providers, indicating a highly competitive market where brand choice has a minimal impact on the base fare.

Price vs. Distance by Service Tier

This section analyzes the correlation between travel distance and fare pricing across various service categories from Uber and Lyft.

Key Insights

• Linear Correlation: There is a clear positive linear relationship between distance and price across all tiers.
• Price Premium: The steepness and starting point (intercept) of the regression lines visually demonstrate the price gap between economy services (like UberX) and luxury tiers (like Lux Black).
• Consistency: The relatively narrow confidence intervals (shaded areas) suggest consistent pricing algorithms based on distance within each specific service tier.

---

Part 3: Price Prediction Modeling

Project Objective

The goal of this phase is to predict rideshare trip prices in Boston as a continuous numerical value. Primary Metric: Our objective is to minimize prediction errors, specifically focusing on RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error) to ensure the most reliable price estimations.

Feature Selection & Engineering

Based on the Exploratory Data Analysis (EDA), I identified the following features as the most significant predictors for price:

Justification for Feature Selection

To optimize model performance and maintain interpretability, I performed a strategic reduction of the dataset:

• Multicollinearity Mitigation: Redundant weather variables and technical timestamps were excluded.
• Heatmap Insights: Correlation analysis revealed high multicollinearity among several environmental factors; removing these reduced noise and prevented overfitting.
• Dimensionality Reduction: By keeping only high-impact features, we ensure the model remains efficient without sacrificing predictive power.

Data Preprocessing & Validation

Before training the predictive model, the dataset underwent a rigorous preprocessing pipeline to ensure data integrity and optimal model performance.

Categorical Encoding**

Applied One-Hot Encoding (via pd.get_dummies) to transform categorical variables such as name, source, and destination into numerical formats, enabling the regression algorithms to process categorical data effectively.

Data Splitting & Reproducibility**

• Train-Test Split: Partitioned the dataset into Training (80%) and Testing (20%) sets.
• Random Seed (Reproducibility): I set a fixed random_state=42 to ensure that the data split remains identical every time the code is executed. This ensures that any improvements in model performance are due to tuning and not random variations in the data distribution.

Distribution Validation**

I conducted a comparative analysis of the target variable (price) across both sets:

• Sample Consistency: The split maintained a high volume of data, providing over 500k samples for robust training.
• Distribution Alignment: The KDE (Kernel Density Estimate) plot confirms that the price distribution in the test set mirrors the training set, ensuring a fair and consistent evaluation environment.

🚀Part 4: Model Development: Linear Regression Baseline

Baseline Establishment

I initiated the modeling phase by implementing a Linear Regression model using the scikit-learn framework. The primary objective of this step is to establish a Baseline Performance Benchmark.

By starting with a simpler, interpretable model, I can evaluate the inherent complexity of the prediction task and set a standard for comparison. Linear Regression serves as an ideal baseline as it efficiently models the direct relationships between our features (e.g., distance, service tier) and the target variable (price).

Training & Performance

• Approach: The model was trained using default parameters to capture the fundamental linear trends within the dataset.
• Results: The baseline achieved a Training $R^2$ Score of 0.9345.
• Interpretation: This indicates that approximately 93.45% of the variance in ride prices can be explained by the selected features, confirming a strong linear signal in the data.

This benchmark will be used to measure the "value-add" of more sophisticated algorithms (such as Ensemble methods) in subsequent stages of the project.

📉 Model Evaluation: Baseline Performance

After training, the Linear Regression model was evaluated on the unseen test set (20% of the data) to measure its real-world predictive accuracy.

Performance Metrics

Analysis of Results

The evaluation reveals a high level of precision for a baseline model. Notably, the $R^2$ score on the test set is identical to the training score, demonstrating that the model generalizes exceptionally well and is not suffering from overfitting. These results establish a very strong benchmark for any future, more complex algorithms.

📊 Model Diagnostic Visualizations

To further validate the baseline's performance, I implemented diagnostic plots to visualize the relationship between predicted values and actual market prices.

Actual vs. Predicted Prices**

This scatter plot compares the model's estimations against the ground truth.

• Interpretation: The high density of data points along the 45-degree reference line (dashed red) indicates a very high predictive accuracy. The model consistently captures price trends across the entire range of trip values.

Residuals Distribution (Error Analysis)**

Analysis of the residuals (Errors = Actual - Predicted) is critical for assessing model bias.

• Normal Distribution: The error distribution follows a near-perfect normal curve centered at zero.
• Insight: This symmetry confirms that the model’s predictions are unbiased, meaning it doesn't systematically overprice or underprice rides. The lack of significant skewness suggests that the linear functional form is well-suited for this dataset.

⚖️ Feature Importance: Coefficient Analysis

To understand the underlying logic of the pricing model, I extracted and visualized the Linear Regression Coefficients. This analysis reveals how much each feature contributes to the final fare estimation.

Key Insights from the Model

• Primary Value Drivers: The surge_multiplier and premium service tiers like Lux Black XL have the strongest positive impact on price. This aligns with rideshare pricing strategies where vehicle class and real-time demand significantly inflate costs.
• Discount Factors: Economy and shared services (e.g., UberPool, Shared, UberX) show strong negative coefficients. These represent the price reduction users receive when opting for high-capacity or standard vehicle types compared to the baseline luxury options.
• Distance Impact: The distance feature maintains a consistent positive coefficient, quantifying the marginal cost increase for every additional mile traveled.
• Geographic Influence: Location-based features (e.g., source_North End) show a minimal but measurable impact, suggesting that while the starting point matters, it is secondary to the chosen service tier and distance.

Conclusion

The model effectively prioritizes Service Level and Demand (Surge) as the dominant predictors of price, proving that the user's choice of "how to ride" is often more influential than "where they go."

🛠️ Advanced Feature Engineering & Scaling

To capture complex, non-linear patterns in the data and enhance model accuracy, I implemented several advanced feature engineering techniques:

Interaction Features**

• Distance-Surge Synergy: Created a distance_surge interaction term. This variable accounts for the compounded effect of surge pricing over long distances, providing a more nuanced predictor than evaluating each feature in isolation.

Unsupervised Clustering (K-Means)**

• Deal Categorization: Implemented a K-Means Clustering algorithm ($k=4$) based on the relationship between distance and price.
• Insight: This allows the model to categorize rides into distinct "transaction profiles" (e.g., premium short trips vs. standard long-haul trips), adding a high-level categorical feature that captures latent market segments.

Feature Scaling**

• Standardization: Utilized StandardScaler to transform numerical features (distance, temperature, distance_surge). By ensuring all features have a mean of 0 and a standard deviation of 1, I mitigated potential bias toward features with larger numerical magnitudes, ensuring stable and balanced model training.

📈 Engineering Validation & Visual Analysis

After generating the new features, I performed a visual validation to ensure they provide meaningful signals for the predictive model.

Cluster Discovery (Unsupervised Learning)**

The K-Means Clustering visualization (left) illustrates the distinct "Trip Profiles" identified by the algorithm. By grouping rides based on price-distance coordinates, we successfully segmented the data into four density tiers (represented by the Viridis color scale). This adds a high-level context that helps the model differentiate between economy-short trips and luxury-long hauls.

Interaction Power**

The Interaction Plot (center) validates the logic behind the distance_surge feature. The strong positive slope of the red regression line confirms that the interaction between distance and the surge multiplier is a potent predictor of fare price, capturing price escalations that individual features might miss.

Numerical Uniformity (Scaling)**

The Boxplot of Scaled Features (right) demonstrates the result of our standardization process.

• Outcome: All primary numerical predictors now share a consistent mean and variance.
• Benefit: This prevents "Feature Dominance," ensuring that the regression weights are strictly determined by the feature's predictive importance rather than its original unit scale.

☁️ Environmental Impact Analysis: Rain & Temporal Factors

In this section, I analyzed how external environmental and temporal factors influence pricing dynamics in the Boston rideshare market.

Weather Impact (Rain vs. Clear Skies)**

• Visualization: Boxplot analysis of price distribution across weather conditions.
• Findings: Surprisingly, the presence of rain shows a negligible impact on the median fare. The price distribution remains remarkably stable, suggesting that weather conditions alone are not a primary driver of price volatility in this dataset.

Temporal Factors (Peak vs. Off-Peak)**

• Visualization: Comparative bar chart of average fares during peak and off-peak hours.
• Findings: The data reveals that average prices during peak hours are nearly identical to off-peak periods. While demand may increase, the average fare remains consistent, indicating that surge pricing (captured by our surge_multiplier) might be localized to specific high-intensity events rather than broad time windows.

Strategic Insight

These visualizations justify the model's reliance on Service Tiers and Distance as the dominant predictors. By debunking the assumption that rain or hour-of-day universally inflate prices, we can focus our predictive power on the features that truly drive market variance.

🧬 Advanced Segmentation & Dimensionality Reduction (PCA)

To gain deeper insights into market structures, I performed a multi-stage cluster analysis combining K-Means Clustering with Principal Component Analysis (PCA).

1. Cluster Interpretation & Mapping

By analyzing the centroids of the generated clusters, I identified and labeled four distinct business segments within the Boston rideshare market:

• Standard Short: High-frequency, baseline trips.
• Long Distance: Trips characterized by higher mileage and consistent pricing.
• Premium/Luxury: High-end vehicle tiers regardless of distance.
• Surge/High Demand: Trips significantly impacted by real-time price multipliers.

2. PCA Projection

Since the dataset contains numerous high-dimensional features (encoded locations, service tiers, weather, etc.), I applied PCA to project the data into a 2D space.

• Validation: The resulting visualization (left) demonstrates clear, non-overlapping boundaries between the identified segments.
• Significance: The separation in the PCA plot validates that our clusters are mathematically distinct, providing the predictive model with a powerful engineered feature that encapsulates complex multi-dimensional relationships into a single categorical input.

🏆Part 5: Model Optimization: Enhanced Linear Regression

The Leap from Baseline to Optimized Model

In this final modeling phase, I retrained the Linear Regression algorithm using the enriched dataset. By integrating the advanced features engineered in the previous steps, the model transitioned from processing raw inputs to understanding complex market dynamics.

Integrated Enhancements

The optimized model now accounts for:

• Synergistic Effects: distance_surge interaction.
• Market Segmentation: cluster_id labels and dist_to_centroid.
• External Context: Binary indicators for weather (is_raining) and temporal peaks (is_peak_hour).

Performance Comparison

🌲 Advanced Modeling: Ensemble Methods

To push the boundaries of predictive accuracy, I transitioned from linear modeling to advanced Ensemble Learning algorithms. These models are designed to capture complex, non-linear interactions within the features that a standard regression might overlook.

Implementation Strategy

• Efficient Sampling: To optimize computational performance while maintaining statistical integrity, I utilized a representative sample of 100,000 observations for this phase.
• Algorithmic Diversity: I implemented and compared two industry-standard models:
  1. Random Forest Regressor: An ensemble of 100 decision trees utilizing bagging to reduce variance.
  2. Gradient Boosting Regressor: A boosting technique that builds trees sequentially to minimize residual errors.

Performance Breakthrough

The results represent the highest accuracy achieved in this project:

Key Takeaway

The Random Forest model emerged as the superior predictor, explaining nearly 98% of the price variance. By effectively leveraging the engineered features (clusters and interactions), the model reduced the average prediction error to just $0.83, providing a highly precise solution for rideshare price estimation.

The core objective of this project was to transition from a general estimation to a high-precision predictive model. The final evaluation demonstrates a dramatic performance breakthrough, achieving an approximately 50% reduction in Mean Absolute Error (MAE).

Performance Comparison

🔍 Baseline Comparison Analysis

The significant leap in accuracy is attributed to two primary strategic factors:

1. Model Complexity & Interaction Capture

• The Limitation: The Baseline model assumed a simple linear relationship between features and price.
• The Reality: Rideshare pricing is driven by complex, non-linear interactions. For instance, environmental factors like rain impact premium "Lux" tiers differently than "Shared" economy tiers.
• The Solution: While Linear Regression struggled to capture these subtle nuances, the Random Forest model thrived by effectively mapping these multi-dimensional dependencies.

2. Strategic Feature Engineering

• Contextual Data: The introduction of engineered features (such as Trip Clusters and Distance-Surge interactions) provided the model with "pre-digested" insights.
• Breaking the Ceiling: The original Baseline model lacked this contextual depth, which created an inherent predictive ceiling. By incorporating these engineered features, we removed those limitations and enabled the final model to reach a new level of predictive precision.

---

Conclusion: By combining non-linear ensemble modeling with domain-specific feature engineering, the final model provides highly reliable fare estimations with an average error of less than one dollar ($0.83), making it a robust tool for real-world application.

🏆 Feature Importance: The Impact of Engineering

To finalize the model analysis, I extracted the Gini Importance scores to identify which features were most influential in the Random Forest’s decision-making process.

Key Discovery: Engineering Over Raw Data

The visualization reveals a critical insight: The top 3 most influential features are the Trip Clusters created during the Feature Engineering phase.

• Strategic Success: The fact that trip_cluster_2, trip_cluster_1, and trip_cluster_3 dominate the importance ranking proves that segmenting the market into distinct behavioral profiles provided more predictive power than any single raw input.
• Compound Effects: The engineered interaction distance_surge also ranks higher than standard distance, confirming that the relationship between demand and mileage is a more potent predictor than mileage alone.
• Operational Tiers: Service tiers (e.g., name_Shared, name_UberXL) follow as secondary predictors, showing that once the model understands the trip's "cluster context," it uses the specific service level to fine-tune the final price.

Conclusion: This ranking validates the project's core hypothesis: sophisticated feature engineering, rather than just choosing a complex algorithm, is what truly "breaks the predictive ceiling" and drives model excellence.

🏆Part 6: The Definitive Winner: Random Forest Regressor

After evaluating multiple models, the Random Forest Regressor emerged as the superior solution for predicting rideshare prices in Boston. It achieved an exceptional balance of precision, stability, and computational efficiency.

Model Performance Metrics

• Accuracy: Reached a peak $R^2$ score of 0.9798, effectively explaining nearly 98% of the price variance.
• Error Rate: Achieved the lowest MAE of only $0.83$, cutting the initial baseline error by 50%.
• Stability: Compared to Gradient Boosting, Random Forest provided a more robust balance between training time and predictive consistency, making it the most reliable choice for real-world deployment.

Key Drivers of Success

The victory of the Random Forest model is attributed to three core pillars:

1. Feature Engineering Excellence: Utilizing the engineered distance_surge interaction and trip_cluster provided the model with "pre-digested" insights, allowing it to capture complex demand-price dynamics.
2. Handling Non-Linearity: Unlike Linear Regression, which fits a straight line, Random Forest successfully partitioned data based on complex conditions (e.g., how rain impact varies significantly across different service tiers).
3. Optimized Processing: Systematic scaling and precise One-Hot Encoding allowed the model to process all variables—location, weather, and vehicle type—optimally without distortion from outliers.

Final Verdict: By leveraging ensemble-based complexity and high-quality engineered features, this model delivers a sub-dollar accuracy rate, setting a new standard for fare estimation in the Boston rideshare ecosystem.

🏷️Part 7: Price Categorization: Quantile Binning

To expand the analysis beyond continuous regression, I transformed the target variable into discrete categories. This allows for a Classification approach, focusing on identifying trip price segments rather than exact dollar values.

Binning Strategy: Quantile-Based Discrete Classes

I implemented a Quantile Binning strategy to split the price data into three balanced classes: Low (0), Medium (1), and High (2).

• Methodology: * Used pd.qcut on the training set to ensure an equal distribution of samples across categories.
  • Calculated the price boundaries: $2.5 – $10.5 (Low), $10.5 – $19.0 (Medium), and $19.0 – $42.5 (High).
  • Applied these identical boundaries to the test set using pd.cut to maintain strict evaluation standards and prevent data leakage.
• Validation: The class distribution printout confirms that the dataset remains well-balanced across all three tiers, providing a solid foundation for training a Classification model.

⚖️ Class Balance Verification

Before proceeding to train the classification model, it is essential to ensure that the newly created target categories are well-balanced. An imbalanced dataset can lead to model bias, where the algorithm favors the majority class.

Methodology & Distribution Analysis

I generated a summary table and a corresponding visualization to inspect the distribution of the three price tiers within the training set:

• Classes: 0 (Budget), 1 (Standard), and 2 (Premium).
• Metrics: Calculated both absolute counts and normalized percentages to verify structural integrity.
• Visualization: The Class Balance Check bar plot confirms a nearly uniform distribution across all tiers.

Significance

Having a balanced training set ensures that the classifier will develop equal predictive sensitivity for all price segments. This prevents the model from becoming biased toward any specific tier and ensures that evaluation metrics (like Accuracy and F1-Score) remain reliable indicators of real-world performance.

📊 Training Set Distribution Summary

To ensure a high-quality classification model, I generated a detailed distribution table for the newly defined price tiers within the training dataset.

Data Breakdown

The following table illustrates the balance between the three price segments:

Analysis of Balance

• Consistency: The data is distributed fairly evenly across all three tiers, with the Budget class (0) being the largest and the Standard class (1) the smallest.
• Model Readiness: With each class representing roughly one-third of the total data, the dataset is well-conditioned for multi-class classification. This balanced distribution minimizes the risk of the model developing a majority-class bias, ensuring that the resulting accuracy metrics will be representative of true predictive performance.

Part 8: Multi-Class Classification Modeling

In this phase, I transitioned from price estimation (regression) to price tier prediction (classification). The objective is to identify the most accurate algorithm for categorizing rides into the predefined Budget, Standard, and Premium tiers.

Model Selection & Training Strategy

I implemented a competitive evaluation pipeline using three distinct classification algorithms:

• Logistic Regression: Serving as the statistical baseline for classification.
• Random Forest Classifier: An ensemble method designed to capture complex non-linear patterns.
• K-Nearest Neighbors (KNN): A distance-based algorithm that classifies data points based on their spatial proximity to neighbors ($k=5$).

Execution & Initial Performance

Using an automated training loop, each model was fitted on the categorized training set and evaluated using the Accuracy Score metric on the unseen test set.

Initial Accuracy Results:

• Logistic Regression: 90.67%
• Random Forest: 92.28%
• K-Nearest Neighbors: ~91.2% (estimated)

Key Insight: All models demonstrated high predictive reliability, consistently exceeding 90% accuracy. The Random Forest Classifier emerged as the early leader, further confirming the model's suitability for the intricate pricing structures of the Boston rideshare market.

📉 Model Diagnostics: Confusion Matrix & Classification Reports

To gain a granular understanding of the classifier's performance, I implemented a diagnostic pipeline using Confusion Matrices and detailed Classification Reports for each model.

Diagnostic Tools

• Classification Report: Provided per-class metrics, including Precision, Recall, and F1-Score. This ensures that the model performs consistently across all price tiers (Budget, Standard, and Premium) and doesn't just excel in one at the expense of others.
• Confusion Matrix Visualization: Utilizing ConfusionMatrixDisplay, I mapped the predicted vs. actual labels to visualize where the models were most successful and where they faced "confusion."

Key Evaluation Insights

• Strong Diagonal Alignment: All models exhibit a dominant diagonal line, indicating high predictive accuracy for each specific tier.
• Adjacent Error Logic: Most misclassifications occur between adjacent categories (e.g., Standard being classified as Budget). Notably, there are near-zero errors between extreme tiers (Budget vs. Premium), confirming that the models have successfully captured the fundamental price boundaries of the market.
• Random Forest Superiority: The Confusion Matrix for the Random Forest model displays the highest "True Positive" counts, particularly within the Premium segment, solidifying its position as the most robust classifier for this project.

🔬 Advanced Model Interpretability & Performance Validation

To ensure the highest level of transparency and diagnostic rigor, I implemented two industry-standard evaluation techniques: ROC/AUC Analysis and SHAP Explanations.

ROC Curve Comparison (Class Separation)**

I evaluated the models' ability to distinguish the Premium tier from other categories using the Receiver Operating Characteristic (ROC) curve.

• AUC Score: This metric quantifies the "Separation Power" of each algorithm. A higher Area Under the Curve (AUC) indicates that the model is exceptionally skilled at correctly identifying luxury rides while minimizing false positives.
• Observation: The models demonstrate high sensitivity and specificity, with curves hugging the top-left corner, confirming robust classification thresholds.

SHAP Value Analysis (Model Interpretability)**

To move beyond "Black Box" modeling, I utilized SHAP (SHapley Additive exPlanations) to visualize the global impact of each feature on the model's output.

• Feature Push/Pull: The SHAP summary plot reveals how specific values (e.g., high surge or specific trip clusters) "push" the predicted price higher or lower.
• Validation of Engineering: The analysis clearly shows that our engineered features—specifically the Trip Clusters—carry the most significant predictive weight. This visual evidence confirms that our feature engineering phase was the primary driver of the model's high performance.

---

🏆 The Definitive Winner: Random Forest Classifier

After a rigorous evaluation involving Accuracy metrics, Confusion Matrices, and Interpretability tools, the Random Forest Classifier has been selected as the optimal model for this project.

Why Random Forest Won:

1. Superior Class Separation (AUC/ROC): The model demonstrated the highest AUC score, proving its exceptional ability to distinguish between Budget, Standard, and Premium tiers with minimal false positives. Its decision threshold is the most robust among all tested candidates.

2. Logical Interpretability (SHAP): Through SHAP analysis, we validated that the model's predictions are grounded in meaningful data relationships. It successfully prioritized our engineered features—specifically the Trip Clusters and Distance-Surge interactions—proving that it "understands" the economic drivers of the Boston rideshare market.

3. High-Precision Reliability: With an Accuracy score exceeding 92%, the Random Forest model provides the most stable performance across all segments. Its confusion matrix showed nearly zero errors between extreme classes (Budget vs. Premium), ensuring that high-stakes pricing decisions remain accurate.

Final Verdict

The Random Forest Classifier isn't just a "Black Box"; it is a highly calibrated engine that leverages complex feature interactions to provide actionable, sub-dollar-precise classifications. It stands as the most reliable model for real-world dynamic pricing applications.

🚕 Brand Pricing Dynamics: Uber vs. Lyft

To further validate the economic relevance of the price tiers, I performed a comparative analysis between the two primary market players in the dataset.

Cross-Tabulation Analysis

Using pd.crosstab with index normalization, I mapped the percentage distribution of each price tier—Budget (0), Standard (1), and Premium (2)—across Uber and Lyft. This approach reveals the pricing strategy and market positioning of each brand.

Key Visual Insights

The Stacked Bar Chart (above) illustrates the pricing composition of each fleet:

• Market Presence: Both brands maintain a presence across all three tiers, confirming that the classification system captures broad market segments rather than brand-specific anomalies.
• Pricing Nuances: While the distributions are relatively similar, the data suggests that Uber holds a slightly higher proportion of "Budget" tier rides within this sample, whereas Lyft shows a marginally higher density in the "Premium" segment.
• Model Validation: This consistency ensures that the subsequent classification models will learn to differentiate prices based on trip characteristics (distance, surge, vehicle type) rather than relying solely on brand identity.

🔍 Explainable AI: SHAP Feature Importance

To ensure the classification model's transparency and understand the drivers behind its predictions, I implemented SHAP (SHapley Additive exPlanations) analysis. This method quantifies the contribution of each feature to the final classification across all three price tiers.

Methodology & Computational Efficiency

• Targeted Sampling: I extracted a representative sample of 100 random observations from the test set. This allowed for an efficient calculation of SHAP values without compromising the interpretability of the results.
• Tree-Based Interpretation: Utilizing shap.TreeExplainer optimized for Random Forest, I calculated the impact of each variable on the model's output magnitude.

Visualizing Multi-Class Influence

The SHAP Summary Bar Plot provides a clear breakdown of feature importance, categorized by class (Budget, Standard, and Premium):

• Cluster Dominance: Engineered features, specifically the Trip Clusters, stand out as the most influential variables. Their impact is consistent across all classes, proving they are the primary "anchors" for the model's pricing logic.
• Granular Impact: We can observe how features like name_Shared or distance_surge influence specific classes more than others, providing a nuanced view of what drives a trip to be classified as "Budget" versus "Premium."

Conclusion: This analysis validates that the model relies on logically sound economic indicators, with engineered features providing the most significant predictive signal.

⚙️ Model Optimization: Hyperparameter Tuning

To squeeze every bit of predictive power out of the Random Forest Classifier, I moved beyond default settings and performed a systematic Hyperparameter Optimization phase.

Strategy: Randomized Search Cross-Validation

I utilized RandomizedSearchCV to explore the parameter space efficiently. This approach balances computational cost with high-quality results by sampling a fixed number of parameter combinations.

• Parameters Explored:
  • n_estimators: The number of decision trees in the forest.
  • max_depth: The maximum depth of each tree (controlling for potential over-fitting).
  • min_samples_split: The minimum number of samples required to split an internal node.
• Validation: Implemented a 3-fold Cross-Validation (CV=3) to ensure that the chosen parameters generalize well to unseen data and are not just "memorizing" the training set.

Optimal Configuration Found

The search identified the following "Best Parameters" for the final model:

• n_estimators: 200
• max_depth: 20
• min_samples_split: 5

Outcome: By applying these optimized settings, the model is now fine-tuned to the specific nuances of the Boston rideshare market, ensuring higher stability and more precise classification across all price tiers.

🚀 Model Deployment: Boston Ride Fare Predictor

The final stage of this project involved transitioning from static analysis to a functional AI-powered application. I developed an interactive web interface that allows users to experience the model's predictive power in real-time.

The Interactive Interface

The Boston Ride Fare Predictor enables users to input specific trip parameters:

• Distance (km): Adjustable slider for trip length.
• Surge Multiplier: Real-time demand intensity.
• Vehicle Type: Selection from the available fleet (e.g., Black, Shared, UberXL).

Behind the Scenes: Real-Time Pipeline

To ensure accurate predictions, the application runs a streamlined version of our data pipeline on every user input:

1. Dynamic Feature Engineering: The system automatically injects baseline environmental data (e.g., average Boston temperature) to prevent model errors.
2. One-Hot Encoding & Scaling: Categorical inputs are transformed, and numerical features are normalized using the project's original StandardScaler to maintain consistency with the training environment.
3. Dual-Model Inference: The app simultaneously runs the Random Forest Regressor and Classifier, providing the user with both the estimated dollar fare and the corresponding price tier category.

Outcome: This deployment demonstrates the practical utility of the project, turning complex machine learning logic into a user-friendly tool for strategic ride planning.

Author

Lihi Caremeli

Reichman University · 2026