Artificial-intelligence programs, like the humans who develop and train them, are far from perfect. Whether it’s machine-learning software that analyzes medical images or a generative chatbot, such as ChatGPT, that holds a seemingly organic conversation, algorithm-based technology can make errors and even “hallucinate,” or provide inaccurate information. Perhaps more insidiously, AI can also display biases that get introduced through the massive data troves these programs are trained on—and that are indetectable to many users. Now research suggests human users may unconsciously absorb these automated biases.
Earlier studies demonstrated that biased AI can harm people in already marginalized groups. Some negative impacts are subtle, such as speech-recognition software’s inability to understand non-American accents, which might inconvenience people using smartphones or voice-operated home assistants. Then there are scarier examples—including health-care algorithms that make errors because they’re trained on only a subset of people (such as white people, those of a specific age range or even people with a certain stage of a disease), as well as racially biased police facial-recognition software that could increase wrongful arrests of Black people.
Yet solving the problem may not be as simple as retroactively adjusting algorithms. Once an AI model is out there, influencing people with its bias, the damage is, in a sense, already done. That’s because people who interact with these automated systems could be unconsciously incorporating the skew they encounter into their own future decision-making, as suggested by a psychology study published in October 2023 in Scientific Reports. Crucially, the study demonstrates that bias introduced to a user by an AI model can persist in that person’s behavior even after they stop using the AI program.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
“We already know that artificial intelligence inherits biases from humans,” says the study’s senior researcher, Helena Matute, an experimental psychologist at the University of Deusto in Spain. For example, when the technology publication Rest of World analyzed popular AI image generators in 2023, it found that these programs tended toward ethnic and national stereotypes. But Matute seeks to understand AI-human interactions in the other direction. “The question that we are asking in our laboratory is how artificial intelligence can influence human decisions,” she says.
“The consequences when inaccurate content or guidance originates from artificial intelligence could be severe.” —Celeste Kidd, University of California, Berkeley
Over the course of three experiments, each involving about 200 unique participants, Matute and her co-researcher, Lucía Vicente of the University of Deusto, simulated a simplified medical diagnostic task: they asked the nonexpert participants to categorize images as indicating the presence or absence of a fictional disease. The images were composed of dots of two different colors, and participants were told that these dot arrays represented tissue samples. According to the task parameters, more dots of one color meant a positive result for the illness, whereas more dots of the other color meant that it was negative.
Throughout the different experiments and trials, Matute and Vicente offered subsets of the participants purposefully skewed suggestions that, if followed, would lead them to classify images incorrectly. The scientists described these suggestions as originating from a “diagnostic assistance system based on an artificial-intelligence (AI) algorithm,” they explained in an e-mail. The control group received a series of unlabeled dot images to assess. In contrast, the experimental groups received a series of dot images labeled with “positive” or “negative” assessments from the fake AI. In most instances, the label was correct, but in cases where there were similar numbers of dots of each color, the researchers introduced intentional skew with incorrect answers. In one experimental group, the AI labels tended toward offering false negatives. In a second experimental group, the slant was reversed toward false positives.
The researchers found that the participants who received the fake AI suggestions went on to incorporate the same bias into their future decisions, even after the guidance was no longer offered. For example, if a participant worked with the false-positive suggestions, they tended to continue to make false-positive errors when given new images to assess. This observation held true even though the control groups demonstrated that the task was easy to complete correctly without the AI guidance—and despite 80 percent of participants in one of the experiments noticing that the fictional “AI” made mistakes.
A big caveat is that the study did not involve trained medical professionals or assess any approved diagnostic software, says Joseph Kvedar, a professor of dermatology at Harvard Medical School and editor in chief of npj Digital Medicine. Therefore, Kvedar notes, the study has very limited implications for physicians and the actual AI tools that they use. Keith Dreyer, chief science officer of the American College of Radiology Data Science Institute, agrees and adds that “the premise is not consistent with medical imaging.”
Though not a true medical study, the research offers insight into how people might learn from the biased patterns inadvertently baked into many machine-learning algorithms—and it suggests that AI could influence human behavior for the worse. If we ignore the diagnostic aspect of the fake AI in the study, Kvedar says, the “design of the experiments was almost flawless” from a psychological point of view. Both Dreyer and Kvedar, neither of whom was involved in the study, describe the work as interesting, albeit not surprising.
There’s “real novelty” in the finding that humans might continue to enact an AI’s bias by replicating it beyond the scope of their interactions with a machine-learning model, says Lisa Fazio, an associate professor of psychology and human development at Vanderbilt University, who was not involved in the study. To her, it suggests that even time-limited interactions with problematic AI models or AI-generated outputs can have lasting effects on people.
Consider, for example, the predictive policing software that Santa Cruz, Calif., banned in 2020. Although the city’s police department no longer uses the algorithmic tool to determine where to deploy officers, it’s possible that—after years of use—department officials internalized the software’s likely bias, says Celeste Kidd, an assistant professor of psychology at the University of California, Berkeley, who was also not involved in the study.
It’s widely understood that people learn bias from human sources of information as well. The consequences when inaccurate content or guidance originates from artificial intelligence could be even more severe, however, Kidd says. She has previously studied and written about the unique ways that AI can shift human beliefs. For one, Kidd points out that AI models can easily become even more skewed than humans are. She cites an assessment published by Bloomberg in 2023 that determined that generative AI may display stronger racial and gender biases than people do.
There’s also the risk that humans might ascribe more objectivity to machine-learning tools than to other sources. “The degree to which you are influenced by an information source is related to how intelligent you assess it to be,” Kidd says. People may attribute more authority to AI, she explains, in part because algorithms are often marketed as drawing on the sum of all human knowledge. Matute and Vicente’s study seems to back this idea up in a secondary finding: the researchers noted that participants who self-reported higher levels of trust in automation tended to make more mistakes that mimicked the fake AI’s bias.
Plus, unlike humans, algorithms deliver all outputs—whether correct or not—with seeming “confidence,” Kidd says. In direct human communication, subtle cues of uncertainty are important for how we understand and contextualize information. A long pause, an “um,” a hand gesture or a shift of the eyes might signal that a person isn’t quite positive about what they’re saying. Machines offer no such indicators. “This is a huge problem,” Kidd says. She notes that some AI developers are attempting to retroactively address the issue by adding in uncertainty signals, but it’s difficult to engineer a substitute for the real thing.
Kidd and Matute both claim that a lack of transparency from AI developers about how their tools are trained and built adds to the difficulty of weeding out AI bias. Dreyer agrees, noting that transparency is a problem, even among approved medical AI tools. The U.S. Food and Drug Administration regulates diagnostic machine-learning programs, but there is no uniform federal requirement for data disclosures. The American College of Radiology has been advocating for increased transparency for years and says more work is still necessary. “We need physicians to understand at a high level how these tools work, how they were developed, the characteristics of the training data, how they perform, how they should be used, when they should not be used, and the limitations of the tool,” reads a 2021 article posted on the radiology society’s website.
And it’s not just doctors. In order to minimize the impacts of AI bias, everyone “needs to have a lot more knowledge of how these AI systems work,” Matute says. Otherwise we run the risk of letting algorithmic “black boxes” propel us into a self-defeating cycle in which AI leads to more-biased humans, who in turn create increasingly biased algorithms. “I’m very worried,” Matute adds, “that we are starting a loop, which will be very difficult to get out of.”