Human Propelled Machine Learning

NIPS 2014 Workshop

Saturday, December 13, 2014
Montreal, Canada

In typical applications of machine learning (ML), humans typically enter the process at an early stage, in determining an initial representation of the problem and in preparing the data, and at a late stage, in interpreting and making decisions based on the results. Consequently, the bulk of the ML literature deals with such situations. Much less research has been devoted to ML involving “humans-in-the-loop,” where humans play a more intrinsic role in the process, interacting with the ML system to iterate towards a solution to which both humans and machines have contributed. In these situations, the goal is to optimize some quantity that can be obtained only by evaluating human responses and judgments. Examples of this hybrid, “human-in-the-loop” ML approach include:

  • ML-based education, where a scheduling system acquires information about learners with the goal of selecting and recommending optimal lessons;
  • Adaptive testing in psychological surveys, educational assessments, and recommender systems, where the system acquires testees’ responses and selects the next item in an adaptive and automated manner;
  • Interactive topic modeling, where human interpretations of the topics are used to iteratively refine an estimated model;
  • Image classification, where human judgments can be leveraged to improve the quality and information content of image features or classifiers.

The key difference between typical ML problems and problems involving “humans-in-the-loop” and is that in the latter case we aim to fit a model of human behavior as we collect data from subjects and adapt the experiments we conduct based on our model fit. This difference demands flexible and robust algorithms and systems, since the resulting adaptive experimental design depends on potentially unreliable human feedback (e.g., humans might game the system, make mistakes, or act lazily or adversarially). Moreover, the “humans-in-the-loop” paradigm requires a statistical model for human interactions with the environment, which controls how the experimental design adapts to human feedback; such designs are, in general, difficult to construct due to the complex nature of human behavior. Suitable algorithms also need to be very accurate and reliable, since humans prefer a minimal amount of interaction with ML systems; this aspect also prevents the use of computationally intensive parameter selection methods (e.g., a simple grid search over the parameter space). These requirements and real-world constraints render “humans-in-the-loop” ML problems much more challenging than more standard ML problems.

In this workshop, we will focus on the emerging new theories, algorithms, and applications of human-in-the-loop ML algorithms. Creating and estimating statistical models of human behavior and developing computationally efficient and accurate methods will be a focal point of the workshop. This human-behavior aspect of ML has not been well studied in other fields that rely on human inputs such as active learning and experimental design. We will also explore other potential interesting applications involving humans in the loop in different fields, including, for example, education, crowdsourcing, mobile health, pain management, security, defense, psychology, game theory, and economics.

The goal of this workshop is to bring together experts from different fields of ML, cognitive and behavioral sciences, and human-computer interaction (HCI) to explore the interdisciplinary nature of research on this topic. In particular, we aim to elicit new connections among these diverse fields, identify novel tools and models that can be transferred from one to the other, and explore novel ML applications that will benefit from the human-in-the-loop of ML algorithms paradigm. We believe that a successful workshop will lead to new research directions in a variety of areas and will also inspire the development of novel theories and tools.

For more information about the NIPS conference, please visit


  • Richard G. Baraniuk, Rice University
  • Michael C. Mozer, University of Colorado Boulder
  • Divyanshu Vats, Sailthru
  • Christoph Studer, Cornell University
  • Andrew E. Waters, Openstax College
  • Andrew S. Lan, Rice University


8:30--8:40   Opening remarks -- Richard Baraniuk, Rice University and Michael Mozer, CU Boulder

Session 1: Machine Learning in Education
8:40--9:00   Invited talk 1: Jacob Whitehill, HarvardX
Beyond Prediction: First Steps toward Automatic Intervention in MOOC Student Stop-out
9:00--9:20   Invited talk 2: Jonathan Huang, Google
Machine Learning as a Force Multiplier to Empower Instructors and Students in Massive Scale Education
9:20--9:40   Invited talk 3: Emma Brunskill, CMU
Learning to Share and Sharing to Learn
9:40--10:00   Talk 4: Richard Baraniuk, Rice University
Mathematical Language Processing for Automatic Grading and Feedback

10:00--10:30   Coffee Break

Session 2: User-Interactive Modeling
10:30--10:50   Invited talk 5: Yisong Yue, Caltech
Machine Learning for Personalized Clustering
10:50--11:10   Invited talk 6: Nisar Ahmed, CU Boulder
Bayesian Inference for Cooperative Human-Robot Information Fusion
11:10--11:30   Invited talk 7: Jordan Boyd-Graber, CU Boulder
Interactive Topic Modeling
11:30--11:50   Invited talk 8: Beverly Woolf, UMass Amherst
Learning to Teach: Improving Instruction with Machine Learning Techniques

12:00--14:00   Lunch Break

Session P: Poster Session
14:00--14:10   Poster Spotlight
14:10--15:10   Poster Session

Session 3: Modeling Human Perception and Knowledge
15:10--15:30   Invited talk 9: Devi Parikh, Virginia Tech
Beyond Mindless Labeling: Really Leveraging Humans to Build Intelligent Machines
15:30--15:50   Invited talk 10: Pedro Domingues, University of Washington
Knowledge Graph Construction with Tractable Markov Logic
15:50--16:10   Invited talk 11: Todd Coleman, UCSD
Modeling Decision Search Processes via Inverse Optimal Stochastic Planning
16:10--16:30   Invited talk 12: Serge Belongie, Cornell NYC
Visipedia Tool Ecosystem

16:30--17:00   Coffee Break

Session 4: Active Learning
17:00--17:20   Talk 13: Michael Mozer, CU Boulder
Bayesian Optimization: From A/B Testing To A-Z Testing
17:20--17:40   Invited talk 14: Kevin Jamieson, UW Madison
NEXT: A System for Human-Machine Interactive Learning
17:40--18:00   Invited talk 15: Lav Varshney, UIUC
On Mismatched Crowdsourcing

18:00--18:30   General discussion and closing remarks -- Richard Baraniuk, Rice University and Michael Mozer, CU Boulder


Rice University, MS-380 - 6100 Main St - Houston, TX 77005 - USA -