Keynote: Robyn Rap

How I Learned to Stop Worrying and Love a Mixed Methods Approach in Recommendation Systems

Delivering value to our users is ultimately the end goal of any machine learning system, but especially so in a high stakes hiring marketplace, where getting a job can change someone’s life. Typically, recommendation and information retrieval systems target a particular metric for online evaluation in an A/B test. But how do we know that our metrics are valid and reliable indicators of success for our users? User studies can serve as an invaluable tool to answer these questions; and yet, many machine learning practitioners have little exposure to these methods or their benefits. In this talk, I share three user study methods that machine learning practitioners can supplement their A/B tests with, including survey methods, eye tracking studies, and qualitative user feedback, drawing on case studies from Indeed.com, the world’s #1 job site. While this talk draws on case studies from the hiring platform space, other practitioners will also benefit from incorporating user studies and feedback into their ML systems. 

Speaker Bio

Robyn Rap is a data science leader at Indeed.com, the world’s #1 job site. She currently serves as Science Lead for Indeed’s Marketplace Platform group, which includes multiple teams of data scientists across the core search engine, trust and safety, aggregation, and search backend. Prior to joining Indeed in 2016, she earned her Ph.D. in Sociology from the University of Texas at Austin. In her spare time, you can find Robyn hiking with her partner, trying not to fall over in yoga class, or playing video games with her cat (currently undefeated).