The consequences of overrelying on imperfect AI
Professor Marzyeh Ghassemi PhD ’17 PD ‘17 describes how using AI systems in human settings can exacerbate social inequality, and what scientists can do about it
Applications of faulty machine learning models are already worsening existing social problems, says Professor Marzyeh Ghassemi PhD ’17 PD ’17. Ghassemi runs the Healthy ML group, which uses AI to equitably improve human health. In addition to building better models, she uses her expertise to raise awareness about the ways AI can fail.
Ghassemi spoke with The Tech about the negative impacts of applying AI to human systems and about scientists’ responsibility to ensure that the technologies they develop are being used ethically. The following interview has been edited for length and clarity.
The Tech: How and why are machine learning models being used in situations where they can harm minority groups?
Ghassemi: If you train models on one dataset, it’s going to do worse on another dataset, and I think that’s not well communicated.
We’ve had AI scribes for a while now in a healthcare setting. [Assistant Professor] Allison Koenecke of Cornell Tech showed that if you pause for a long time, the OpenAI transcription tool Whisper inserts violent and sexually explicit content into the medical notes in the acoustic pause. Some people with medical conditions or for whom English is not their first language might pause for a longer time. Some of the papers evaluating AI scribes are saying, “Some of the stuff it said wasn’t accurate, and that might cause problems.” There’s a gap between what academic machine learning communities want to show, which is the potential, and the application areas like health that have large amounts of capital.
In the U.S., [some] judges are shown scans of people’s brains, and then for-profit companies will show an AI system and say, “Our model can predict that this person has a violent brain based on their fMRI scan, so they should get longer sentencing.” [MIT Professor] Oliver Rollins has a book coming out on this. That [diagnosis of a violent brain] is not a real thing, but because it comes with the shininess of AI, it seems believable.
TT: What do you think it will take for AI to be deployed equitably?
Ghassemi: We should have better regulation. There’s so many examples in American history where a technology was introduced, and it can do wonderful things, but there were also harms, and so it was regulated. I think we need strong nonpartisan consumer pressure. We need to make it clear to our representatives that this is a forefront issue.
I’ve been told by lawyers that we decide the nitty-gritty details of our regulatory processes based on lawsuits. My hope is that there’s a throughline between some of these lawsuits and better regulation. As more lawsuits are lost [by tech companies], we will see more of an appetite from government officials towards regulation. There’ll be a realization that they need to step up, or they will lose elections. But it’s a difficult process, and we don’t actually know how we’re going to get from one side to another side.
TT: What should MIT do to ensure that these new technologies are used responsibly?
Ghassemi: I think we should have a much stronger science engagement arm. There are some universities that have invested heavily in hiring faculty that have expertise in not just the technical side, but also on the regulation, legislation, and political engagement side. They have experience in how policymakers will want information given to different decision-making bodies.
As a scientist, if you love the thing you work on, it can be very hard to say, “I’m not going to support any use of my technology for anything except for this set of things.” I think that as a technology developer, you have a responsibility to communicate what other people don’t have the training to be aware of. This is not the most common view in science. People vary in their comfort levels on prescribing uses of technology, even if it’s a technology that they themselves develop.
There is a program [MIT’s Science Policy Initiative] where students go to Capitol Hill, meet with staffers or congresspeople, and communicate about important issues like AI and science funding. I’m very proud that MIT has that, and I know so many graduate students who have taken the time to go on that trip. They’re earning expertise here at the institution, and then they’re taking the time to communicate with decision makers. Engaging with policy is very difficult, and I think it’s something that we as a university should invest more in.