To evaluate the utility of artificial intelligence (AI) models in assisting humans, human-AI collaboration studies usually measure whether AI-assisted users improved their task performance compared to unassisted performance. However, such a performance gain can be explained not only by a synergy of human and AI strengths that enables appropriate reliance on AI, but also by users over-relying on a superior AI regardless of AI prediction quality. We suggest changes to human-AI collaboration studies to account for over-reliance, including replacing performance gain with complementary performance as the study endpoint; controlling, measuring, and reporting over-reliance; and disclosing the negative influence of over-reliance when discussing study results. Our critical analysis aims to improve the scientific rigor of human-AI collaboration studies and to encourage prioritizing collaborative requirements in AI development that enable users’ appropriate reliance on AI.
@inproceedings{miccai_haic2026,author={Jin, Weina and Hamarneh, Ghassan},booktitle={Medical Image Computing and Computer-Assisted Intervention (MICCAI) Workshop on Human-AI Collaboration (MICCAI HAIC)},title={Accounting for Over-Reliance in AI-Assisted Performance Studies},year={2026},keywords={Human-AI collaboration, Human-centered AI, Complementary performance},project={overreliance}}
