Probabilistic machine learning increasingly informs critical decisions in all sectors. To aid the development of trust in these decisions, we develop a taxonomy delineating where trust in an analysis can break down: (1) in the translation of real-world goals to goals on a particular set of available training data, (2) in the translation of abstract goals on the training data to a concrete mathematical problem, (3) in the use of an algorithm to solve the stated mathematical problem, and (4) in the use of a particular code implementation of the chosen algorithm. Our taxonomy highlights steps where existing research work on trust tends to concentrate and also steps where establishing trust is particularly challenging. In this talk, I will detail how trust can fail at each step and illustrate our taxonomy with examples from my recent research.