Identifying Bad Actors on the Uber Carshare (formerly Car Next Door) Platform
The Problem
Uber Carshare (formerly Car Next Door) makes it simple to turn any car into a share car, empowering people to save money, reduce waste and create cleaner, greener, better neighbourhoods. The platform connects a community of car owners to car borrowers, allowing members to borrow cars in hourly blocks before returning them back to the owner.
While the vast majority of Uber Carshares’s members behave well, a small portion (1-2%) of members may misbehave by returning vehicles late, in poor condition, driving erratically or dangerously, or incurring fees or speeding fines during their trips. While a human review system is in place to identify bad actors, it would typically only identify them and intervene once they had already completed 5-6 trips and caused issues to multiple different car lenders on the platform.
Solution
To solve the problem, Max Kelsen worked with Uber Carshare to predict which members were likely to misbehave in future based on their past behaviour. This allows the Uber Carshare team to intervene early before excessive damage has occurred, either by encouraging their members to behave well, or in some cases removing them from the platform.
Max Kelsen combined a range of data sources from member signup, borrowing history, member reviews, high-frequency driving telemetry behaviour to build a picture of a member’s history and identify those who are likely to be bad actors. This allows the Uber Carshare investigations team to proactively triage and engage with members and ensure a great user experience for everyone on the platform.
Challenges and Roadblocks
As part of the project, the Max Kelsen team built a time-series classification model that takes into account the full picture for each member and give a risk score for each member.
One of the challenges within the project was the quality of the labelled data. In the past, only the worst behaved bad actors had been flagged and banned from the platform, but the Uber Carshare team knew a much larger portion of members were exhibiting the same behaviours and hadn’t yet been identified.
Max Kelsen worked with the Uber Carshare team to use machine learning and human in the loop approaches to train models using the existing dataset, and then iteratively review predictions, flag new users as bad actors, and retrain the model with a larger and more accurate dataset.
Max Kelsen deployed the system into AWS using MLOps best practices and all with an infrastructure as code (IaC) approach, with the trained model pipeline generating new predictions each day for all users whose state has changed in the previous 24 hours. A retraining pipeline was implemented together with an automated AWS Code Pipeline CI/CD system, allowing scheduled retraining jobs to automatically run and with approval, promote the latest and most accurate model through to the production Sagemaker Pipelines workflow.
Outcomes
In order to validate the model prior to deployment, Max Kelsen worked with Uber Carshare to validate an initial set of predictions over all users. It was found that of the 100 members that the model identified as likely to be bad actors, 40 of them were flagged for suspension. This is 20-30x better than random guessing.
Once validated and with the model was rolled out to production, Max Kelsen was able to show that by using the machine learning system, Uber Carshare is able to identify bad actors 4 trips earlier than they would without the system, saving hundreds of dollars in fees, fines and damage that would have otherwise occurred. The system continues to run in production and allows the Uber Carshare human review team to be far more efficient, and allows them to focus more of their time delivering an amazing experience to their members.
