Machine learning algorithms are used for decision-making in societally high-stakes settings from child welfare and criminal justice to healthcare and consumer lending. These algorithms are often intended to predict outcomes under a proposed decision. It is challenging to evaluate how well these algorithms perform because we only observe the relevant outcome under a biased sample of the population. In this talk, we explore how to use techniques from causal inference to estimate performance on the full population. We will consider several strategies to account for confounding factors that affect the decision and the outcome. First, we study runtime confounding where all relevant factors are captured in the historical data, but it is either undesirable or impermissible to use some such factors in the prediction model.  Second, we study the setting with unobserved confounders where we can bound the degree to which the outcome varies on average between units receiving different decisions conditional on observed covariates and identified nuisance parameters. We develop debiased machine learning estimators for the learning target and predictive performance estimands under both settings. We present empirical results in the consumer lending and child welfare domains.

Papers: arxiv:2212.09844 and arxiv:2006.16916.