AI In Healthcare Part III: Can AI Break Free from Healthcare's History of Bias?
This is a five-part series examining the unmitigated risks of AI in healthcare
For the better part of the history of Western medicine, race has been treated as biological; it is only recently that institutions have begun peeling back. The seeds for such theory date to the 1700s but have come to shape our society as we know it. It has led to wildly unequal outcomes in everything from housing to healthcare when broken down by race. The Emory study, led by radiologist Judy Gichoya, is troubling because of the possibility the Artificial Intelligence model is tying some biological marker it’s picking up on to the patient’s race.
The use of race as a biological marker has its roots in scientific racism, which is defined as: “An organized system of misusing science to promote false scientific beliefs in which dominant racial and ethnic groups are perceived as being superior,” according to the National Human Genome Research Institute.
In 2003, through the findings of the Human Genome Project, the “scientific” idea of race as biological was irrefutably disproven. The project discovered that modern humans are 99.9% identical in DNA.
This theory took hold in Europe and the United States in the late 18th and 19th centuries, and many of its ideas remained strong well into the 20th. The theory was influenced by the Enlightenment era of the 1700s, in which philosophers began pivoting from Biblical ideas that humans were created equal and started dividing humans into racial species. Influenced by these ideas, scientists began providing “scientific” justification for the catastrophic colonization that Europeans were imposing on the rest of the world. Carl Linnaeus, a Swedish scientist in the 1700s was one of the first to sort humans into a racial hierarchy, with white Europeans on top and people of African descent on the bottom. Samuel Morton, an American scientist and physician, further popularized the theory of a racial hierarchy in the 1800s, when he purported that the size of the skull could determine the intelligence of a race. Morton claimed “Caucasians” had the largest skulls and “Negroes” had the smallest. His findings could never be replicated. It’s also worth noting the skulls he claimed were “Caucasian” were from Egyptian mummies … on the continent of Africa.
In the early 20th century evidence began mounting that these theories were false. Yet, the ideas were not easy to dislodge from a world that had been reformed around this racial hierarchy. Well into the mid and late 20th century, higher crime rates in Black neighborhoods were seen as being pathological. Disparate test scores in Black schools (compared to those in predominantly white schools) were seen as being due to a naturally lower intellect than whites. Black athletes were often denied opportunities to play quarterback in football, to catch or pitch in baseball, and were regularly discriminated against for coaching positions. Even after the logic of a hierarchy was punctured and laws based on equal opportunity were erected, many still looked at race as being biological and it was often used as a marker to define demographic differences in health outcomes. In 2003, through the findings of the Human Genome Project, the “scientific” idea of race as biological was irrefutably disproven. The project discovered that modern humans are 99.9% identical in DNA.
However, as we see with the recent abolition of the spirometer, society, and by extension our healthcare system, are still wrestling with the entrenched idea of race being biological.
In the fall of 2020, while protests in support of Black Lives and over the death of George Floyd continued to grip cities across the United States, the American Medical Association released an official statement recognizing race as a social construct rather than a biological one.
“We believe it is not sufficient for medicine to be non-racist, which is why the AMA is committed to pushing for a shift in thinking from race as a biological risk factor, to a deeper understanding of racism as a determinant of health,” said AMA Board Member Michael Suk, M.D., J.D., M.P.H., M.B.A. in the statement.
It’s important to understand the history of race as a biological vs. a social construct to understand how AI can work to improve disparities vs. recycling them. Defining race as a social construct forces us to look at how racism (both individual and institutional) contributes to social conditions. If an AI tool interprets the impact of racism as being biological (such as lower pulmonary function) it could revert the scientific progress made over the last 30+ years. The same goes for sexism, homophobia, or other forms of discrimination.
"A language model that's trained on lots of clinical notes might learn that when a woman reports that she's in a high level of pain, doctors don't prescribe her anything and they don't send her for follow-up appointments but when a man does that they are sent for follow-up appointments and prescribed medication" – Marzyeh Ghassemi, PhD and researcher at MIT
(Editors’ note: This is why I prefer to think of marginalization as 'systems of power,' given the emphasis on the systems constructed to empower specific identities by disempowering others).
AI is trained on massive amounts of data but without critical minds ensuring the data is accounting for the impact of socially constructed systems of power on the people it deprioritizes, an opening is left to create a hierarchy that is perceived to be neutral. The word “data” often presumes no bias. Still, in a world where disparities in healthcare cannot be untied from socially constructed discrimination, there’s simply no way to train a model to be completely unbiased.
Marzyeh Ghassemi, the MIT researcher, provided a few examples of how bias finds its way around guardrails set up to ensure fairness:
- Numbers: “If you’re a minority, it’s likely there are very few other people in a clinical sample that look like you or have healthcare experiences like you, meaning you are likely to be ignored or misrepresented by the model.”
- Underrepresentation: “Women are half the planet, yet are still often under-sampled, under-recorded, and underrepresented in data sets collected to study a condition.”
- AI models learn from human bias: “In this case, we could still see bad results, because we could be seeing the model learning from the way people in your subgroup are mistreated,” Ghassemi said. “For example, because African American patients are routinely denied access to healthcare, their conditions are often further along when they get to the hospital. The AI model will pick up on that and say, ‘This must be how it's supposed to be because I only ever see examples of these really advanced cases coming in certain patients.’ Or, for example, a language model that's trained on lots of clinical notes might learn that when a woman reports that she's in a high level of pain, doctors don't prescribe her anything and they don't send her for follow-up appointments but when a man does that they are sent for follow-up appointments and prescribed medication. The model will learn the (gender) based on the pronouns in this patient's record and if it’s a woman and they describe that they're in pain, (the model) has learned from millions of patient records, that it’s not supposed to do anything about that.”
It’s not just healthcare. For example, an investigation by The Markup in 2021, revealed that AI models used by financial institutions to determine a borrower’s eligibility for a loan were discriminating based on race. The investigation found that with all factors being equal, lenders were 40% more likely to turn down Latino borrowers than white Americans, 50% more likely to turn down Asian applicants, 70% more likely for Native Americans, and 80% for Black Americans.
“In Europe, you can’t use race because they banned (the use of) social risk scoring (in AI),” Gichoya said of the ban that went into effect in 2023.
But, as Gichoya highlighted in her study, not using race won’t be good enough.
“We are saying that just because you don’t include or collect it if the model is seeing something, it still encodes it."
Stephen Norris is a strategic provider partnerships and management expert with a track record of driving growth and profitability. He has extensive experience building and expanding provider partnerships within the healthcare industry. Norris is skilled in contract negotiation, stakeholder management, and data analysis with a demonstrated ability to lead and motivate teams to deliver exceptional results. He has a deep understanding of the healthcare landscape and a passion for health equity through improving patient outcomes. He is #OpentoWork.
