Women’s and Gender Studies researchers work to hold AI accountable
From healthcare to government, machine learning models are changing how decisions are made
Over the past few years, artificial intelligence (AI) capabilities have advanced significantly. OpenAI’s discontinued Sora app gave everyone the chance to express their artistic creativity by allowing users to generate entire videos from text prompts that were nearly indistinguishable from reality, and medical diagnosing models have been shown to perform at or above a human level. However, advances in AI technology can just as easily be used in malicious ways. Sora’s videos were realistic enough to lead to concerns over deepfake impersonations, and government surveillance tools use many of the same techniques as models in medical imaging to identify people without due process.
This rapid technological development creates a need for discussions about regulations and the role of AI in social power dynamics. On Feb. 26, the Women’s and Gender Studies (WGS) program hosted a luncheon to address this topic from two different lenses: health and governance. At the luncheon, Institute for Medical Engineering & Science Associate Professor Marzyeh Ghassemi PhD ’17 and Urban Science and Planning Associate Professor Catherine D’Ignazio SM ’14 described their research on the ethical implications of building a future with AI. The room overflowed with people, with several attendees even sitting outside the door just to listen. What emerged from these talks and conversations with experts was a clearer picture of AI’s immediate implications.
More technology gives the potential for more power
When demonstrators took to the streets in Minneapolis to protest the Trump administration’s Operation Metro Surge, ICE agents had AI tools to identify who the protestors were and where they lived. While these tools are far from perfectly accurate, their use still sets a precedent for the government to target and detain specific individuals. This is not the only time the federal government has been publicly involved with AI; more recently, the Trump administration publicly and legally attacked AI company Anthropic for refusing to sell technology that could be used for autonomous weapons or mass surveillance. This government contract was then picked up by Anthropic’s rival company OpenAI.
Such advances in AI give the U.S. government new opportunities to increase their power, leading to a movement that D’Ignazio describes as technofacism. At her WGS luncheon talk, D’Ignazio introduced her conceptualization of technofacism, which builds on existing scholarly research and discussion.
D’Ignazio, who is also the director of the Data + Feminism Lab at MIT, defined technofacism as “the collusion of large technology firms, right-wing billionaires, and tech culture with authoritarian and anti-democratic political agendas.” However, the exact definition is still a work in progress, so she read off of a computer, stopping often to connect with the audience. Her language was precise but casual: for example, when going over the characteristics of facism, she explained how facism relies on the idea that “the nation functions like a family which has a patriarchal daddy figure.”
D’Ignazio’s definition of technofacism is reflected in the experiences of Cindy Cohn, the executive director of the Electronic Frontier Foundation (EFF). Cohn’s professional memoir, Privacy’s Defender: My Thirty-Year Fight Against Digital Surveillance was published last month by the MIT Press. When asked about Professor D’Ignazio’s conceptualization of technofacism, Cohn acknowledged that technology is playing a role in how the federal government exercises its power; however, she cautioned that focusing on the technology could let people “blind [them]selves to the fact that there is just plain, old-fashioned authoritarianism happening right now.”
“I think it is a mistake to think that this current moment was only created by tech, because then you don’t think hard about how to get out of it,” Cohn added.
Cohn explained that the problem is not that these technologies exist, but rather that federal officials are using new technologies for authoritarian purposes. She pointed out that the officials advocating for AI-based surveillance, such as Pete Hegseth, aren’t necessarily driven by technological progress. Instead, they’re using their power to contract out private companies who make surveillance-based machine learning models. For example, the ICE agents identifying protesters are using software like Clearview AI, an app that identifies people by referring to a massive machine learning model trained on photos posted on the internet. Using apps like Flock, they could also point their phones at a car license plate to identify the vehicle’s owner.
D’Ignazio characterizes this software as “AI, tech, and data infrastructure used in the service of fascist state violence,” one of the defining characteristics of technofacism. Other technologies under this category include AI-controlled weapons — the other point of contention between Anthropic and the Trump administration.
D’Ignazio explained that “AI is primed to destroy our civic institutions,” building on a paper by Boston University Professors Woodrow Hartzog and Jessica M. Silbey. Fascist governments could exploit this phenomenon to normalize and justify prejudice: since contamination narratives about women and minorities are a key characteristic of fascist regimes, biased machine learning models create an opportunity to institutionalize discrimination against marginalized groups that might already be misrepresented in the data. Since most of the public won’t know how the machine learning models reach their conclusions, it will be difficult to pinpoint the biases behind them.
Cohn has seen the consequences of these models firsthand. While working with the EFF to track misidentifications made by facial recognition technology for policing, she noticed a disturbing trend: even though the technology has improved considerably, facial recognition software still tends to misidentify people of color most often.
“If you add technology to a racist system, you get a racist system that has technology,” Cohn concluded. “It’s not just the case that making a perfect technology will fix the problems that are actually kind of human-based problems, and that’s true at the front end when you’re training. It’s also true at the back end when they’re using the tool.”
Garbage in, garbage out
To fix issues with human bias in AI, it is first necessary to understand how biased data can influence the outcomes. For example, even if a model performs better than humans at diagnosing patients overall, this statistic often does not take into account rates of underdiagnosis by demographic, nor does it consider how often a model fails to detect a particular disease in an individual of a particular sex, racial, or ethnic category.
One way that models are trained is via the use of existing medical data, which is often biased towards white men. Because marginalized groups are already underdiagnosed, there are fewer existing examples of certain conditions in particular populations, and the model has the potential to overgeneralize those specific cases instead of learning how to properly diagnose cases — a phenomenon called overfitting. Even if demographic data is left out of training data entirely, current AI models can still accurately predict a person’s race from something as simple as a chest X-ray and apply existing stereotypes accordingly; crucially, human doctors cannot make such predictions, and researchers do not know what models use to make those predictions.
Using existing health data to train models also runs the risk of said model using potentially spurious shortcuts. In her talk at the WGS luncheon, Ghassemi joked about a hypothetical model that could predict which patients in a hospital would die with 100% accuracy: while very accurate, this model’s diagnostic criterion was whether or not a patient’s file contained some variation of the words “call the priest.”
Though this example got a laugh out of luncheon attendees, these kinds of shortcuts are a real problem. As another example, Ghassemi showed how a real model designed to identify cancerous moles had developed a more potentially dangerous shortcut: when she gave it a picture of a benign mole on her skin, the model correctly flagged it as benign, but when the model was given a picture of that same mole after Ghassemi’s children had drawn on her skin near the mole, the model marked it as cancerous. Traditional diagnostic criteria for cancerous moles include rapid changes in size or shape, so doctors often make marks around a suspected cancerous mole to track the mole’s appearance; the model then associates those marks with cancer.
Obtaining a more diverse training data set is unfortunately not without its challenges. For Science, Technology, and Society Associate Professor Oliver Rollins, balancing people’s well-being and privacy with AI’s need for data is crucial. He describes the rise of AI in health as “a really interesting kind of dilemma in which [AI] is both set there to improve health outcomes but by doing so, it literally is exposing more people to more dangerous kinds of things.”
According to Rollins, much of this risk can come from the lack of centralized regulation of data. One of his ongoing projects deals with defining “neurodata” and determining whether it is different from health data or genetic data. This distinction is important because data classification directly impacts what can be done with that data.
For example, while Direct-to-Customer genetic testing companies like 23andMe are not prohibited from giving genetic information to third parties, genetic data in medical settings is considered health data and therefore protected information under the Health Insurance Portability and Accountability Act (HIPAA). Adding in the new classification of “AI data” makes protecting privacy even more complicated. If an image generated by a neural scan is used to train an AI model, does that make it AI data or neurodata? If that AI model is designed to diagnose Alzheimer’s, is that data now health data? Because standards for protecting data vary so much, not having a clear pathway to follow can give unfair systems a way to deny their involvement in cases where people get hurt.
AI for the people
As AI becomes a commonplace tool in everything from health research to making basic decisions, it can feel like the world is on a path to becoming completely automated. But to D’Ignazio, this is simply a reflection of a technofascist narrative around AI. “None of this is inevitable,” she told the audience.
While Cohn didn’t comment on the technological capability of AI, she agreed with the idea that the so-called inevitable development of AI is a narrative. Cohn claimed that it is in the best interest of billionaire tech CEOs with expanding AI enterprises to make everyone believe in the inevitability of an AI-dominated future. “That makes me suspicious,” she stated.
To D’Ignazio, the solution to this narrative is “envision[ing] other worlds that are not that one that we’re currently sold right now.” She pointed to an example from her own work at the Data + Feminism Lab, where she organized a project building technology to support activists raising awareness for feminicide, which is violence against women. Instead of assuming a technological solution was necessary, D’Ignazio’s team first worked with the activists to ideate the project, asking if there was anything they could build and what it should be.
Ultimately, D’Ignazio’s group built a model to scan local news for feminicide to help the activists identify what cases to take on. By using an AI model, activists would need to read fewer disturbing cases and consume less violent material, protecting their mental health.
During the design process for the AI model, the researchers offered to further automate the case-finding process by extracting the relevant details for cases that best fit the definition of feminicide and putting them into a spreadsheet. The activists said no. “They considered that part of their labor is witnessing this violence and caring for the women whose lives were affected by it,” D’Ignazio explained. “So they saw this data production work as a kind of collective memory work, and thus it was not appropriate for automation, so we didn’t do it.”
Reflecting on that experience, D’Ignazio proposed an alternative framework for how the world can use AI: one in which technology is developed through open discussions with the people who will be using it, models are small and task-oriented instead of general-purpose, and communities have the chance to build tools even if the tools aren’t profitable. D’Ignazio encouraged the audience to take inspiration from this model in their own technological projects, even when a big data approach to machine learning can feel mainstream. By focusing on a very specific task and including as many perspectives as possible in the ideation process, it becomes easier to notice and fight against the negative impacts of the shortcuts that AI models use to make their decisions. As machine learning plays an increasing role in people’s work, health, privacy, and government, incorporating these interdisciplinary perspectives can play a pivotal role in the quality of people’s lives.