Weina Jin

Trying hard to make AI explainable to clinical users

Hi! I am Weina, a PhD student at Dr. Hamarneh’s Medical Image Analysis Lab, Computing Science, Simon Fraser University.

My research is on developing end-user-centered interpretable AI (artificial intelligence), and how to use it to augment doctors’ clinical decision making based on medical image tasks. I’m especially interested in using explanations for better learning, for both AI (enable AI to learn better by forcing explicit representation) and doctors (learn from those explicit representations to accumulate experience from big clinical data).

Previously, I received my Doctor of Medicine (MD) degree from Peking University, and underwent residency training in Neurology. I had worked in the hospital, pharmaceutical and tech company. Now in addtion to my research in training and making sense of artificial intelligence, in my life I am engaged in training and understanding natural intelligence, and writing science fictions.

My research contributions to human-centred AI and its explanation

In my research, I propose the Clinical explainable AI Guidelines based on a systematic evaluation with existing explainable AI algorithms (AAAI22, ICML21 workshop), and a user study with neurosurgeons. The guidelines provide well-specified user-centered requirements and evaluation criteria to develop explainable AI for clinical tasks, and potentially could be extended to other high-stakes domains.

The EUCA framework is another tool to support the development of end-user-centered explainable AI. It provides fine-grained design suggestions for feature-, example-, and rule-based explanations grounded in end-users’ perspectives in my user study. Another work built upon EUCA conducted clinical user requirement analysis for explainable AI via a user study with 25 physicians including radiologists, pathologists, dermatologists, neurologists, etc.

A 3-min video introduction of my research projects:

News

Jun 28, 2022	I am awarded as one out of ten fellows for the Borealis AI Global Fellowship of $10,000. Media report from Borealis AI: The Borealis AI 2021-2022 Fellowships: Inspiring Canada’s AI Research Talent
Jun 20, 2022	Co-organize the Interpretable ML in Healthcare workshop at ICML22.
Dec 9, 2021	Our work on evaluating XAI for medical image task is accepted at AAAI 2022.

Selected publications

AIIM
Evaluating the clinical utility of artificial intelligence assistance and its explanation on the glioma grading task

Jin, Weina, Fatehi, Mostafa, Guo, Ru, and Hamarneh, Ghassan

Artificial Intelligence in Medicine 2024

Weina Jin and Mostafa Fatehi are co-first authors

abstract Bib HTML Paper Blog Code Slides

TLDR: A clinical user study with 35 neurosurgeons and identified that AI and explainable AI are not helpful in the collaborative doctor-AI clinical settings.

Clinical evaluation evidence and model explainability are key gatekeepers to ensure the safe, accountable, and effective use of artificial intelligence (AI) in clinical settings. We conducted a clinical user-centered evaluation with 35 neurosurgeons to assess the utility of AI assistance and its explanation on the glioma grading task. Each participant read 25 brain MRI scans of patients with gliomas, and gave their judgment on the glioma grading without and with the assistance of AI prediction and explanation. The AI model was trained on the BraTS dataset with 88.0% accuracy. The AI explanation was generated using the explainable AI algorithm of SmoothGrad, which was selected from 16 algorithms based on the criterion of being truthful to the AI decision process. Results showed that compared to the average accuracy of 82.5±8.7% when physicians performed the task alone, physicians’ task performance increased to 87.7±7.3% with statistical significance (p-value = 0.002) when assisted by AI prediction, and remained at almost the same level of 88.5±7.0% (p-value = 0.35) with the additional assistance of AI explanation. Based on quantitative and qualitative results, the observed improvement in physicians’ task performance assisted by AI prediction was mainly because physicians’ decision patterns converged to be similar to AI, as physicians only switched their decisions when disagreeing with AI. The insignificant change in physicians’ performance with the additional assistance of AI explanation was because the AI explanations did not provide explicit reasons, contexts, or descriptions of clinical features to help doctors discern potentially incorrect AI predictions. The evaluation showed the clinical utility of AI to assist physicians on the glioma grading task, and identified the limitations and clinical usage gaps of existing explainable AI techniques for future improvement.
@article{JIN2024102751, title = {Evaluating the clinical utility of artificial intelligence assistance and its explanation on the glioma grading task}, journal = {Artificial Intelligence in Medicine}, volume = {148}, pages = {102751}, year = {2024}, issn = {0933-3657}, doi = {https://doi.org/10.1016/j.artmed.2023.102751}, url = {https://www.sciencedirect.com/science/article/pii/S0933365723002658}, author = {Jin, Weina and Fatehi, Mostafa and Guo, Ru and Hamarneh, Ghassan}, keywords = {Artificial intelligence, Neuro-imaging, Neurosurgery, Explainable artificial intelligence, Clinical study, Human-centered artificial intelligence}, video = {https://youtu.be/rhLQQsc8Z8Y}, project = {brain_ai_clinical_study}, }
MedIA
Guidelines and evaluation of clinical explainable AI in medical image analysis

Jin, Weina, Li, Xiaoxiao, Fatehi, Mostafa, and Hamarneh, Ghassan

Medical Image Analysis 2023

abstract arXiv Bib HTML Paper Blog Code

TLDR: The Clinical XAI Guidelines provides criteria that explanation should fulfill in critical decision support.

Explainable artificial intelligence (XAI) is essential for enabling clinical users to get informed decision support from AI and comply with evidence-based medical practice. Applying XAI in clinical settings requires proper evaluation criteria to ensure the explanation technique is both technically sound and clinically useful, but specific support is lacking to achieve this goal. To bridge the research gap, we propose the Clinical XAI Guidelines that consist of five criteria a clinical XAI needs to be optimized for. The guidelines recommend choosing an explanation form based on Guideline 1 (G1) Understandability and G2 Clinical relevance. For the chosen explanation form, its specific XAI technique should be optimized for G3 Truthfulness, G4 Informative plausibility, and G5 Computational efficiency. Following the guidelines, we conducted a systematic evaluation on a novel problem of multi-modal medical image explanation with two clinical tasks, and proposed new evaluation metrics accordingly. Sixteen commonly-used heatmap XAI techniques were evaluated and found to be insufficient for clinical use due to their failure in G3 and G4. Our evaluation demonstrated the use of Clinical XAI Guidelines to support the design and evaluation of clinically viable XAI.
@article{JIN2023102684, title = {Guidelines and evaluation of clinical explainable AI in medical image analysis}, journal = {Medical Image Analysis}, volume = {84}, pages = {102684}, year = {2023}, issn = {1361-8415}, doi = {https://doi.org/10.1016/j.media.2022.102684}, url = {https://www.sciencedirect.com/science/article/pii/S1361841522003127}, author = {Jin, Weina and Li, Xiaoxiao and Fatehi, Mostafa and Hamarneh, Ghassan}, keywords = {Interpretable machine learning, Medical image analysis, Multi-modal medical image, Explainable AI evaluation}, project = {xai_eval}, }
AAAI
Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?

Jin, Weina, Li, Xiaoxiao, and Hamarneh, Ghassan

Proceedings of the AAAI Conference on Artificial Intelligence Jun 2022

Acceptance rate: 15%

abstract arXiv Bib HTML Paper Blog Code Poster Slides

TLDR: Our systematic evaluation showed the examined 16 heatmap algorithms failed to fulfill clinical requirements to correctly indicate AI model decision process or decision quality.

Being able to explain the prediction to clinical end-users is a necessity to leverage the power of artificial intelligence (AI) models for clinical decision support. For medical images, a feature attribution map, or heatmap, is the most common form of explanation that highlights important features for AI models’ prediction. However, it is unknown how well heatmaps perform on explaining decisions on multi-modal medical images, where each image modality or channel visualizes distinct clinical information of the same underlying biomedical phenomenon. Understanding such modality-dependent features is essential for clinical users’ interpretation of AI decisions. To tackle this clinically important but technically ignored problem, we propose the modality-specific feature importance (MSFI) metric. It encodes clinical image and explanation interpretation patterns of modality prioritization and modality-specific feature localization. We conduct a clinical requirement-grounded, systematic evaluation using computational methods and a clinician user study. Results show that the examined 16 heatmap algorithms failed to fulfill clinical requirements to correctly indicate AI model decision process or decision quality. The evaluation and MSFI metric can guide the design and selection of explainable AI algorithms to meet clinical requirements on multi-modal explanation.
@article{Jin_Li_Hamarneh_2022, title = {Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?}, volume = {36}, url = {https://ojs.aaai.org/index.php/AAAI/article/view/21452}, doi = {10.1609/aaai.v36i11.21452}, number = {11}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, author = {Jin, Weina and Li, Xiaoxiao and Hamarneh, Ghassan}, year = {2022}, month = jun, pages = {11945-11953}, project = {xai_eval_aaai}, }
arXiv
EUCA: the End-User-Centered Explainable AI Framework

Jin, Weina, Fan, Jianyu, Gromala, Diane, Pasquier, Philippe, and Hamarneh, Ghassan

Jun 2021

abstract arXiv Bib Paper Blog Code Poster Website

TLDR: EUCA provides design suggestions from end-users’ perspective on explanation forms and goals.

The ability to explain decisions to end-users is a necessity to deploy AI as critical decision support. Yet making AI explainable to non-technical end-users is a relatively ignored and challenging problem. To bridge the gap, we first identify twelve end-user- friendly explanatory forms that do not require technical knowledge to comprehend, including feature-, example-, and rule-based explanations. We then instantiate the explanatory forms as prototyping cards in four AI-assisted critical decision-making tasks, and conduct a user study to co-design low-fidelity prototypes with 32 layperson participants. The results confirm the relevance of using explanatory forms as building blocks of explanations, and identify their proprieties — pros, cons, applicable explanation goals, and design implications. The explanatory forms, their proprieties, and prototyping supports (including a suggested prototyping process, design templates and exemplars, and associated algorithms to actualize explanatory forms) constitute the End-User-Centered explainable AI framework EUCA, and is available at http://weinajin.github.io/end-user-xai. It serves as a practical prototyping toolkit for HCI/AI practitioners and researchers to understand user requirements and build end-user-centered explainable AI.
@article{jin2021euca, title = {{EUCA}: the End-User-Centered Explainable {AI} Framework}, author = {Jin, Weina and Fan, Jianyu and Gromala, Diane and Pasquier, Philippe and Hamarneh, Ghassan}, year = {2021}, archiveprefix = {arXiv}, primaryclass = {cs.HC}, project = {euca}, }

Bio

Weina Jin is a Computer Science Ph.D. student at the Medical Image Analysis Lab, Simon Fraser University, supervised by Professor Ghassan Hamarneh. Weina received her Doctor of Medicine degree from Peking University, and underwent neurological residency training at Peking University First Hospital. She also received research training in human-computer interaction at Simon Fraser University. Weina's research combines her multidisciplinary research experience in medicine, machine learning, and human-computer interaction, and focuses on developing physician-centered explainable artificial intelligence for clinical decision-support. She uses a human-centered approach in the technical development and evaluation to make AI more clinically applicable. Her work has published in AAAI 2022, and received the best poster presentation award in IEEE VIS 2019. She is also the recipient of Borealis AI Global Fellowship 2022.