This is part of a weekly series on AI and free speech.
The most important decisions about our lives can turn on factors ranging from unfair to downright absurd. In medieval England, a criminal facing the gallows could save their neck by reciting Psalm 51 from memory — a literacy test meant to identify clergy that became a famous loophole for accused commoners. In France, employers have long sorted job candidates by handwriting analysis, a method researchers later found predicts job performance no better than chance. In Israel, a famous study of parole boards found that favorable rulings peaked right after the judges had lunch and gradually slid toward the bottom as dinner approached. This became known as the “hungry judge effect.”
Society has historically tackled such problems by firing bad decision-makers, passing generally applicable laws governing business practices, or — in the case of England’s “neck verse” — simply repealing the rule. Other times, we might just accept that any qualitative decision carries some inherent arbitrariness — what legal scholar Cass Sunstein and Princeton psychologist Daniel Kahneman called “noise“ in human judgment. Attempting to ensure good decisions by regulating the tools a decision-maker consults — as opposed to the acts and outcomes themselves — is a relatively recent societal initiative. And today, it’s coinciding with algorithms and AI bringing the same unfairness and absurdity to decision-making in digital form.
Instead of memorizing Psalm 51 or perfecting a handwriting sample, people try to game algorithms. As testing of AI video-interview screening tools offered by a Munich startup revealed, a job candidate’s score would rise when she added a bookshelf to her background and fall when the lighting dimmed. A Guardian investigation similarly found resume-screening tools quietly treating a first name like “Thomas” or a keyword like “church” as a predictor of success on the job.
These practices have attracted scrutiny because of the tools’ wide purchase — by one estimate, 90% of U.S. employers use AI tools to screen candidates. And in contexts like health care, we’ve seen automated tools lead to real harm. A 2019 study found that a widely deployed hospital algorithm ranked black patients as healthier than equally sick white patients, because their true health needs were underrepresented in the cost data used to train it. This in turn led to them receiving less care.
What has FIRE been doing in the AI space?
This is the third article in a weekly series on AI and free speech. You can read the first article explaining why the First Amendment is so important in the age of AI here.
Policymakers have responded with a flurry of proposals regulating the use of AI in “consequential decisions” in areas like hiring and health care — that is, decisions about which candidate is hired or which course of treatment is recommended. Illinois has required consent and disclosure for AI-analyzed video interviews since 2020. New York City’s Local Law 144 requires bias audits and candidate notice for automated hiring tools. Colorado enacted the first comprehensive state AI law in 2024 covering consequential decisions across sectors. And in the current session, Connecticut’s recently enacted SB 5 has led a new slate of similar bills through statehouses.
Some readers may have been on board with our application of First Amendment principles to AI earlier in this series, but might feel skeptical or nervous now that we’re dealing with situations where lives and livelihoods are on the line. That’s rational. But be assured: Our application of First Amendment guardrails here follows directly from long-established legal doctrine, and — importantly — leaves intact the most important legal tools in our society’s toolbelt for addressing the targeted harms. Take discrimination, for instance, where Title VII of the Civil Rights Act has long been a potent weapon against business practices with a disparate racial impact — whether that disparate impact is the result of bigotry, unfair screening practices, or AI tools. That legal protection won’t change, nor should it.
The First Amendment problems arrive via what we might call means-specific regulation — laws that target the tools or processes that contribute to a decision rather than the decision itself. As we’ll see, the disclosures required by these laws give the government a window to impose its own perspectives and prejudices about what makes a good or bad decision. And when the tool in question is something people consult for information, like a chatbot, these laws allow the government to place burdens on and set the terms of information access. The result is a classic threat to the First Amendment’s interest in the “free flow of accurate information.”
An age-old institutional challenge
To understand those First Amendment problems, we’ll want to take a deeper dive into why institutions are using AI systems for important decisions in the first place.
The biggest answer is that AI systems can process vast amounts of information more quickly and consistently than any human decision-maker ever could. In many contexts, humans have simply been unable to qualitatively assess all the relevant data needed to make grounded decisions.
This is an age-old challenge for institutions charged with high-stakes decisions, and they have historically been forced to cope with it in imperfect ways. Urgent care clinics, unable to qualitatively assess how in-need every patient that enters the clinic is, mostly order by check-in time. Employers have used blunt screening cutoffs like undergraduate GPA to thin application pools. College admissions officers, overwhelmed by sheer volume, might spend as little as four minutes assessing an application.
AI promises an appealing solution: the judgment of a superhuman reviewer capable of cheaply and carefully reviewing and comparing all the relevant data needed to make good decisions. In practice, these systems identify patterns in historical data and generate predictions or recommendations based on those patterns. Institutions hope the result is lower costs, better outcomes, and less biased decision-making.
Just like a screening cutoff, a handwriting assessment, or any other information shortcut, these systems have real limitations. Training data can be incomplete or skewed in ways that distort outcomes toward the people and qualities represented in it. These concerns are worth taking seriously — and remembering that many of the related harms that may come from the use of AI systems, like discriminatory decision-making, have existing remedies. The use of AI does not exempt anyone from laws governing discrimination, fraud, professional malpractice, privacy, and deceptive trade practices, all of which provide important tools for addressing concrete harms. After all, discrimination is discrimination, fraud is fraud, malpractice is malpractice, regardless of the tool one uses to carry it out.
So the question isn’t whether the government may police discriminatory or negligent decisions. It may. The question is what happens when the government moves beyond regulating decisions and starts regulating what’s behind them — the process and tools that feed into decision-making. And specifically, what happens when the government singles out some tools for regulation — like an AI model — while leaving other tools — like a lottery or a conventional statistical tool — untouched. That’s where two distinctive features of the new laws targeting the use of AI in “consequential decisions” come in: mandatory disclosures and design mandates.
Mandatory disclosures
Mandatory disclosure provisions in the new crop of proposed legislation about AI-based decision-making generally take one of three forms: notice to the people affected by a decision that AI played a role in making it, disclosure of the data used to train or run the system, and audits assessing the system’s potential for harms like bias or discrimination. Often these audits are aimed at encouraging specific anti-bias measures. Colorado’s SB 24-205, for example, mandated the disclosure of the “measures the developer has taken to mitigate known or reasonably foreseeable risks of algorithmic discrimination that may arise from the reasonably foreseeable deployment of the high-risk [AI] system.” The idea there, of course, is that you’d better have some measures in place to disclose.
In requiring developers and deployers to speak when they might otherwise choose not to, these provisions invoke the First Amendment’s compelled speech doctrine.
Now, the First Amendment doesn’t treat all compelled speech the same. “Commercial speech” — which the Supreme Court described in 1985’s Zauderer v. Office of Disciplinary Counsel as speech “about the terms under which … services will be available” — has traditionally been weighed under a more relaxed criterion. Under what’s known as the Zauderer standard, the government can require businesses to disclose information about their products so long as it’s “purely factual and uncontroversial.”
That last part is important, and a real-world example from the Ninth Circuit shows why. If you saw a warning attached to a food or beverage product announcing that it contains acrylamide or glyphosate, you’d be unlikely to go check the science and prudently weigh the risks. More likely, you’d assume there must be a good reason the warning was required, and, subsequently, associate both the ingredient and the product itself with bad news. You’d be surprised to later learn that the harms of acrylamide are fiercely contested, with a major government study finding “no consistent evidence” tying dietary exposure to sickness. The contested basis for the underlying claim is exactly what doomed California’s attempts to mandate disclosures for both chemicals.
Courts have been more permissive when the judgment behind a disclosure is objective. In 2019, the Ninth Circuit upheld a Berkeley ordinance requiring cell phone retailers to post the FCC’s own guidance on radiofrequency exposure at the point of sale: the disclosure was modest, “literally true,” and lacked a subjective basis that might make it misleading. Calorie counts and ingredient lists have long survived on the same footing.
When the basis is subjective or “controversial,” the calculus changes for a few reasons key to the First Amendment analysis.
First, the existence of the disclosure may itself be misleading — consumers will assume it’s tied to an objective, uncontroversial harm akin to “too much sugar can lead to cavities.” By being misleading, the disclosure violates the core principle behind allowing disclosures in the first place: the “free flow of accurate information,” as the Ninth Circuit put it in the Berkeley case. Second, it’s a negative signal against the product, the “reputation and goodwill” of the company behind it, and the underlying component the disclosure attaches to. If we permitted such signals on an arbitrary basis, they could be used by the government to punish products, industries, and companies the government doesn’t like. Finally, that negative signal serves to put the government’s thumb on the scale in the underlying controversy that makes the disclosure “controversial,” like the debate over acrylamide’s harmful effects. And rather than persuading people of its position, the government enlists the targeted companies to voice the government’s contested judgment as though it were their own.
Courts have applied the same logic outside product safety, to judgments that might be political or ethical rather than scientific. In 2024’s X Corp v. Bonta, the Ninth Circuit enjoined California’s social media law AB 587 because it forced platforms to disclose how they define “misinformation,” “hate speech,” and “extremism” — compelling the company to voice a side in “intensely debated and politically fraught topics.” And in 2015’s National Association of Manufacturers v. SEC, the D.C. Circuit struck down the SEC’s requirement that public companies disclose whether their products contain “conflict minerals” — minerals tied to financing conflict in the Democratic Republic of the Congo. The court explained that the requirement saddled disclosing companies with a kind of “moral responsibility” for the Congo war; in other words, the regulation “require[d] an issuer to tell consumers that its products are ethically tainted.”
In short, the government cannot conscript private speakers to broadcast the government’s contested judgment about their own products and practices.
Which brings us back to the AI disclosures.
Yes, we’ve seen real flaws with using AI to help make decisions — recall the AI-based resume screeners rewarding arbitrary keywords. And there are many more examples where that came from. But look closely, and AI doesn’t pose unique harms compared to what it’s replacing. It’s a trade-off: The potential bias of an overwhelmed human decision-maker or a blunt screening cutoff is swapped for the potential bias in a model’s training data. Whether that’s a good trade is itself a subjective judgment, and reasonable people looking at the same facts land in different places.
One Voice: Gagging education board members
Many, many boards for school districts, community colleges, and public universities, and at least one state, have formal policies that limit board members from publicly criticizing actions taken by the boards, speaking to the press, or communicating on social media. These policies, often called “One Voice,” have resulted in punishments of board members …
In fact, whether an AI’s recommendations and outputs are better or fairer than a human’s is, at bottom, one of the most contested debates of the moment — and in many ways a moral one. A model may be more accurate and even less innately biased than a person, but it removes the holistic human touch from a decision-making process in a way many people find morally questionable. It’s part of why the ethics of automation is one of today’s great debates. Rather than letting that debate play out, the government is settling it by drafting the developers and deployers of AI systems to channel the government’s skepticism to the public as if it were their own.
So while a notice that says “this decision was made using an artificial intelligence tool” looks factual and uncontroversial in isolation, context matters. When the state singles out that one input — among the many a decision-maker might consult — for a mandatory warning, it communicates an official judgment: The public should distrust this decision. And it’s doing that on a values-driven basis. No comparable notice attaches when a consequential decision-maker relies on cruder methods for making a decision like a screening cutoff, a conventional statistical tool, a literal roll of the dice, or just a hunch — all of which pose similar threats of bias and unfairness. And importantly, the disclosure attaches simply to the disfavored tool of decision-making — AI — not to any harmful decisions or outcomes. That’s a controversial disclosure.
Our concerns sharpen once you notice what the AI might actually be doing in at least some affected deployments. Laws like Connecticut’s SB 5 subject employers to an array of requirements if they’re consulting an AI system to generate recommendations or other outputs that “materially influence” an employment decision. Read naturally, that could reach a hiring manager who asks a chatbot to pressure-test her impressions of a candidate’s resume or CV if it provides advice that proves sufficiently helpful in the hiring manager’s ultimate decision. On top of the law flagging a potential source of useful information as inherently suspicious, it could also burden any affected people who access that information on account of that subjective suspicion.
And the problem with burdening that information doesn’t stop at compelled disclosures. It’s a basic First Amendment principle that the government can’t burden access to a source of information just because it’s suspicious of what that source might say. We wouldn’t accept special requirements for a manager who consults a friend or a spiritual advisor before a hiring decision because the government harbors skepticism about their advice. The same logic holds when the advisor is a chatbot.
Design mandates
Other restrictions in “consequential decision” bills target the design and governance of AI models themselves. Colorado’s SB 24-205 required developers of “high-risk” AI systems to use “reasonable care” to protect consumers from known or reasonably foreseeable risks of “algorithmic discrimination,” and required documentation about training data, system limitations, foreseeable discrimination risks, testing, data-governance measures, and mitigation steps. But the law’s definition of “algorithmic discrimination” contained a revealing carveout: It excluded certain uses of high-risk AI systems for “expanding an applicant, customer, or participant pool to increase diversity or redress historical discrimination.”
That language shows these laws go a little further than simply promoting fair decision-making. Such regulations require developers and deployers to build the state’s contested fairness judgments into their systems, including which disparate effects must be mitigated, what counts as “diversity,” and what historical discrimination is relevant. xAI’s First Amendment challenge to the law seized on this point, arguing that SB 24-205 did not simply prohibit discriminatory decisions but forced developers like them to conform model design to Colorado’s preferred account of fairness. In part because of these concerns, the law was recently repealed by the legislature and replaced with a version discarding those provisions.
Again, none of the First Amendment arguments above leave the public unprotected from illegal conduct like discrimination or fraud. The tools provided by existing laws — Title VII, the Fair Housing Act, malpractice liability, fraud statutes — remain fully available against a business that discriminates, a doctor who is negligent, or a lender who deceives, regardless of whether the decision was made by a person, a conventional statistical tool, or an AI model. And legislators are free to continue to draft further generally applicable laws governing business practices. The First Amendment analysis doesn’t necessarily implicate any of that. It’s the second, often redundant means-specific layer these bills bolt on top of generally applicable laws that invites the government to impose its suspicions and can burden protected information-gathering.
The law typically resists that layer for good reason. In an uncertain world, consequential decisions will always carry some trade-offs and arbitrariness. Having judges give unfavorable rulings because they’re hungry or a GPA cutoff snubbing otherwise outstanding job candidates is the baked-in price of asking humans to judge other humans at scale, and no disclosure regime is going to cure the problems that brings. The law’s answer to that fact has rarely been policing the specific means with which people might try to make those decisions. It’s certainly never been to limit or burden information sources that might help somebody make a decision. Instead, the answer has traditionally and rightfully been to punish the bad decision or outcome, whoever or whatever produced it.
VERY interesting material. Thanks for putting it together.
Reminds of some emissions regulation that required catalytic converters in all new cars. Then Honda created an engine that ran so clean that it had lower emissions than American cars with a catalytic converter and a catalytic converter didn't help. Except they couldn't sell cars in the US with the new engine without also installing mostly-useless and extremely-expensive catalytic converters anyway, because of the regulation.