Presentation
Experts and Non-Experts Use Diagnostic Aids Inefficiently: A Reanalysis of Cao et al. (2023)
SessionCEDM4: Usability
DescriptionStudies benchmarking automation-assisted decision making against models of optimal use have shown that assisted performance is highly inefficient. However, these exercises have used novice populations performing simplified decision tasks, limiting their generalizability. Cao et al. (2023) described a machine learning algorithm capable of screening CT scans for early signs of pancreatic cancer with extremely high sensitivity (area under the curve ≥ 98.5%) and demonstrated that the algorithm improved human clinicians’ diagnoses. This study reanalyzed the data of Cao et al. (2023) to assess the efficiency of the clinicians’ aid use. Assisted performance was highly inefficient, roughly matching the predictions of a model that assumes participants randomly defer to the aid’s advice with a probability of 50%. Moreover, aid use was equally poor across varying levels of clinician expertise. The results replicate previous findings of poor decision aid use and confirm that they generalize to real-world tasks using expert decision makers.
Contributors
Event Type
Lecture
TimeWednesday, September 11th8:50am - 9:10am MST
LocationFLW Salon C
Cognitive Engineering & Decision Making