Generalizability: Machine Learning and Humans-in-the-Loop
Files
Description
Automated decision tools, which increasingly rely on machine learning (ML), are used in decision systems that permeate our lives. Examples range from high-stakes decision systems for offering credit, university admissions, and employment, to decision systems serving advertising. Here, we consider data-driven tools that attempt to predict likely behavior of individuals. The debate about ML-based decision-making has spawned an important multi-disciplinary literature, which has focused primarily on fairness, accountability and transparency. For example, the Association for Computing Machinery for the past few years has held a conference for researchers working on these issues. We have been struck, however, by the lack of attention to generalizability in the scholarly and policy discourse about whether and how to incorporate automated decision tools into decision systems. This chapter explores the relationship between generalizability and the division of labor between humans and machines in decision systems. An automated decision tool is generalizable to the extent that it produces outputs that are as correct as the outputs it produced on the data used to create it. The generalizability of an ML model depends on the training process, data availability, and the underlying predictability of the outcome that it models. Ultimately, whether a tool’s generalizability is adequate for a particular decision system depends on how it is deployed, usually in conjunction with human adjudicators. Taking generalizability explicitly into account highlights important aspects of decision system design, as well as important normative trade-offs, that might otherwise be missed. Section 1 provides the conceptual and technical basics underlying our analysis, situating the present discussion in the broader discourse about automated decision-making. It presents a simplified outline of considerations in designing and deploying a decision system, identifying various ways in which automated decision tools could be incorporated, and sketches the steps involved in creating ML models. Section 2 focuses on generalizability and its importance to debates about whether and how to incorporate automated decision tools into decision systems. It relates generalizability to the familiar “rules versus standards” discourse in legal theory and to more traditional data-driven modeling in computer science, social science, and policymaking. It analyzes facets of generalizability that are important for all data-driven models and highlights distinctive ways generalizability interacts with ML models. Section 3 analyzes how human and machine strengths and weaknesses in generalization may affect rulemaking and adjudication. We discuss design stages related to the integration of machine and human decision-making that have received little attention in policy debates and emphasize the importance of these stages to a decision system’s ultimate ability to generalize to real-world cases. In Section 4, we summarize how generalizability concerns should affect the design and implementation of automated decision tools.
Source Publication
Research Handbook on Big Data Law
Source Editors/Authors
Roland Vogl
Publication Date
2021
Recommended Citation
Nay, John and Strandburg, Katherine J., "Generalizability: Machine Learning and Humans-in-the-Loop" (2021). Faculty Chapters. 1661.
https://gretchen.law.nyu.edu/fac-chapt/1661
