Response times (RTs) are commonly used to assess cognitive abilities and have recently been employed to assess face and object recognition. However, it is unclear whether face and object processing RTs predict recognition ability beyond accuracy. To test the validity of RT as an assessment of face recognition ability, we examined accuracy and RT on a widely-used face matching assessment modified to collect meaningful RT data, the computerized Benton Face Recognition Test (BFRT-c), and measured whether they predict face recognition ability and DP vs. control group membership. 62 controls and 36 DPs performed the BFRT-c as well as validated measures of face recognition ability: the Cambridge Face Memory Test (CFMT) and a Famous Faces Memory Test (FFMT). We found little-to-no association between BFRT-c accuracy and RT in both controls (r=.07, p=.59) and DPs (r=.03, p=.86). In controls, BFRT-c accuracy robustly predicted CFMT performance (r=.49, p<.001), FFMT performance (r=.43, p<.001), and a CFMT-FFMT composite (r=.54, p<.001), whereas BFRT-c RT was not significantly associated with these measures (all r's<.16, p's>.21). We found that BFRT-c accuracy significantly differed between DPs and controls, but RT failed to differentiate the groups. Results in controls and DPs were replicated with outlier removal. Further, combined scores of BFRT-c accuracy and RT (inverse efficiency score and balanced integration score) did not predict face recognition ability or DP vs. control group membership better than accuracy alone. These results suggest researchers should take caution when using RT to characterize individual differences in face processing or diagnose deficits in prosopagnosia.