Sorting Algorithms for Qualitative Data to Recover Latent Dimensions with Crowdsourced Judgments

Presenter: James Honaker

Abstract: The Quicksort and Bubble Sort algorithms are commonly implemented procedures in computer science for sorting a set of numbers from low to high in an efficient number of processes using only pairwise comparisons. Because of such algorithms’ reliance on pairwise comparison, they lend themselves to any implementation where a simple judgment requires selecting a winner. We show how such algorithms, adapted for stochastic measurements, are an efficient way to harness human ”crowdsourced” coders who are willing to make brief judgments comparing two pieces of qualitative information (here, sentences of text) to uncover the underlying dimension or structure of the qualitative sources.

As a demonstration of the ability of our approach, we show that correctly structured non-expert judgments of the level of democratization in countries recovers the same information that alternate expert scales of democratization –with large cost and time– estimate.

Our key motivating implementation involves a large collection of written policies describing conditions for eligibility for welfare (TANF) in each state in the US. An open question in the literature on welfare is the existence of a race-to-the-bottom, which necessitates measuring the generosity of complex sets of eligibility rules that differ from state to state and across time.

Existing approaches in the literature have attempted to scale or rank state generosity in welfare policies by either constructing large coding questionnaires (Fellowes and Rowe, 2004) and summing responses, or by attempting factor analysis of all possible raw data on welfare policy (De Jong et al, 2006). The first approach requires understanding all the dimensions that are relevant before constructing the survey implement. The second requires converting all policy documents and rules into quantitative measures.

We show how to obtain a ranking of state welfare generosity without doing harm to the qualitative nature of the sources, and without leveraging expert knowledge to sort the vast collection of textual sources. We present ”crowdsourced” human coders sentences describing one policy measure in each of two states, and ask them which of the two is the more generous (or more flexible, or more lenient) welfare rule. The optimal set of pairwise comparisons is continuously chosen by the sorting algorithm. We compare the rankings of state policies created using judgments from paid human coders through Mechanical Turk, as well as more resource intensive rankings we created to replicate the scaled indices and factor scores used in the previous literature for the most recently available data.

We demonstrate that it is possible to reveal structure, and to organize textual information through ”human processing” by relying on algorithms common to quantitative methods, but without any actual quantification of the qualitative textual sources. Moreover this is a highly resource efficient method to organize large corpora of written information. We demonstrate the powerful performance of non-expert human intelligence, when given sufficiently small structured textual tasks, set out as pairwise comparisons, and show how well understood sorting algorithms can take these human judgments and uncover the latent quantitative ordering of the qualitative sources. The results are very cheap, incredibly fast measures that correlate as strongly with gold standard statistical methods as alternate statistical specifications scale with each other.

Presentation Date: 

Wednesday, February 12, 2014

Presentation Files: 

See also: 2014