1、Issue BriefNovember 2024AI Safety and Automation BiasThe Downside of Human-in-the-LoopAuthorsLauren KahnEmelia S.ProbascoRonnie KinoshitaAI Safety and Automation BiasThe Downside of Human-in-the-LoopAuthorsLauren KahnEmelia S.ProbascoRonnie KinoshitaCenter for Security and Emerging Technology|1 Exec

2、utive Summary Automation bias is the tendency for an individual to over-rely on an automated system.It can lead to increased risk of accidents,errors,and other adverse outcomes when individuals and organizations favor the output or suggestion of the system,even in the face of contradictory informati

3、on.Automation bias can endanger the successful use of artificial intelligence by eroding the users ability to meaningfully control an AI system.As AI systems have proliferated,so too have incidents where these systems have failed or erred in various ways,and human users have failed to correct or rec

4、ognize these behaviors.This study provides a three-tiered framework to understand automation bias by examining the role of users,technical design,and organizations in influencing automation bias.It presents case studies on each of these factors,then offers lessons learned and corresponding recommend

5、ations.User Bias:Tesla Case Study Factors influencing bias:Users personal knowledge,experience,and familiarity with a technology.Users degree of trust and confidence in themselves and the system.Lessons learned from case study:Disparities between user perceptions and system capabilities contribute t

6、obias and may lead to harm.Recommendation:Create and maintain qualification standards for user understanding.User misunderstanding of a systems capabilities or limitations is asignificant contributor to incidents of harm.Since user understanding iscritical to safe operation,system developers and ven