Measuring Implicit Bias in ICU Notes Using Word-Embedding Neural Network Models

Julien Cobert; Hunter Mills; Albert Lee; Oksana Gologorskaya; Edie Espejo; Sun Young Jeon; W John Boscardin; Timothy A Heintz; Christopher J Kennedy; Deepshikha C Ashana; Allyson Cook Chapman; Karthik Raghunathan; Alex K Smith; Sei J Lee

doi:10.1016/j.chest.2023.12.031

Measuring Implicit Bias in ICU Notes Using Word-Embedding Neural Network Models

Chest. 2024 Jun;165(6):1481-1490. doi: 10.1016/j.chest.2023.12.031. Epub 2024 Jan 8.

Affiliations

¹ Anesthesia Service, San Francisco VA Health Care System, University of California, San Francisco, San Francisco, CA; Department of Anesthesia and Perioperative Care, University of California, San Francisco, San Francisco, CA. Electronic address: Julien.cobert@ucsf.edu.
² Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, CA.
³ Division of Geriatrics, University of California, San Francisco, San Francisco, CA.
⁴ School of Medicine, University of California, San Diego, San Diego, CA.
⁵ Department of Psychiatry, Harvard Medical School, Boston, MA; Center for Precision Psychiatry, Massachusetts General Hospital, Boston, MA.
⁶ Division of Pulmonary, Allergy, and Critical Care Medicine, Duke University, Durham, NC.
⁷ Department of Medicine, the Division of Critical Care and Palliative Medicine, University of California, San Francisco, San Francisco, CA; Department of Surgery, University of California, San Francisco, San Francisco, CA.
⁸ Department of Anesthesia and Perioperative Care, Duke University, Durham, NC.
⁹ Department of Geriatrics, Palliative, and Extended Care, Veterans Affairs Medical Center, University of California, San Francisco, San Francisco, CA; Division of Geriatrics, University of California, San Francisco, San Francisco, CA.

Abstract

Background: Language in nonmedical data sets is known to transmit human-like biases when used in natural language processing (NLP) algorithms that can reinforce disparities. It is unclear if NLP algorithms of medical notes could lead to similar transmissions of biases.

Research question: Can we identify implicit bias in clinical notes, and are biases stable across time and geography?

Study design and methods: To determine whether different racial and ethnic descriptors are similar contextually to stigmatizing language in ICU notes and whether these relationships are stable across time and geography, we identified notes on critically ill adults admitted to the University of California, San Francisco (UCSF), from 2012 through 2022 and to Beth Israel Deaconess Hospital (BIDMC) from 2001 through 2012. Because word meaning is derived largely from context, we trained unsupervised word-embedding algorithms to measure the similarity (cosine similarity) quantitatively of the context between a racial or ethnic descriptor (eg, African-American) and a stigmatizing target word (eg, nonco-operative) or group of words (violence, passivity, noncompliance, nonadherence).

Results: In UCSF notes, Black descriptors were less likely to be similar contextually to violent words compared with White descriptors. Contrastingly, in BIDMC notes, Black descriptors were more likely to be similar contextually to violent words compared with White descriptors. The UCSF data set also showed that Black descriptors were more similar contextually to passivity and noncompliance words compared with Latinx descriptors.

Interpretation: Implicit bias is identifiable in ICU notes. Racial and ethnic group descriptors carry different contextual relationships to stigmatizing words, depending on when and where notes were written. Because NLP models seem able to transmit implicit bias from training data, use of NLP algorithms in clinical prediction could reinforce disparities. Active debiasing strategies may be necessary to achieve algorithmic fairness when using language models in clinical research.

Keywords: critical care; inequity; linguistics; machine learning; natural language processing.

Published by Elsevier Inc.

MeSH terms

Algorithms
Bias
Critical Illness / psychology
Electronic Health Records
Female
Humans
Intensive Care Units*
Male
Natural Language Processing*
Neural Networks, Computer*

Grants and funding

P30 AG044281/AG/NIA NIH HHS/United States