California Passes Exclusive Security Rules for AI Firms Despite Tech Firms’ Opposition

SACRAMENTO, Calif. (AP) — California lawmakers voted Tuesday to advance a bill that would require artificial intelligence companies to test their systems and add security measures to prevent them from potentially being manipulated to destroy the state’s power grid or build chemical weapons — scenarios that experts say could happen in the future as the technology advances at the speed of light.

The first-of-its-kind bill aims to reduce the risks posed by AI. It is being fiercely opposed by venture capital and technology firms, including Meta, the parent company of Facebook and Instagram, and Google, who say the rules target developers and should instead focus on those who use and abuse AI systems for harmful purposes.

Democratic state Sen. Scott Wiener, who authored the bill, said the proposal would provide reasonable safety standards to prevent “catastrophic harm” from extremely powerful AI models that may be created in the future.

The requirements would apply only to systems that cost more than $100 million in computing power to train. No current AI model has reached that threshold as of July.

Wiener criticized the campaign by opponents of the bill at a legislative hearing Tuesday, saying it spread misinformation about his measure. His bill does not create new criminal charges for AI developers whose models were used to cause social harm if they tested their systems and took steps to mitigate the risks, Wiener said.

“This bill will not send any AI programmers to jail,” Wiener said. “I would ask people to stop making these claims.”

Under the bill, only the state attorney general will be able to take legal action in the event of violations.

Democratic Gov. Gavin Newsom has touted California as an early adopter and regulator of AI, saying the state could soon deploy generative AI tools to solve highway congestion, make roads safer and provide tax guidance. At the same time, his administration is considering new rules against AI discrimination in hiring practices. He declined to comment on the bill, but warned that overregulation could put the state in a “dangerous situation.”

A growing coalition of tech companies says the requirements will discourage companies from building large AI systems or releasing their technologies as open source.

“The bill would make the AI ecosystem less secure, threaten the open source models that startups and small businesses rely on, rely on nonexistent standards, and introduce regulatory fragmentation,” Rob Sherman, Meta’s vice president and deputy chief privacy officer, wrote in a letter to lawmakers.

Opponents want to wait for more guidance from the federal government. Supporters of the bill have said California can’t wait, citing hard lessons it learned by not acting sooner to restrict social media companies.

The proposal, backed by some of the most respected AI researchers, also calls for the creation of a new state agency to oversee developers and develop best practices.

State lawmakers on Tuesday also considered two ambitious measures to further protect Californians from potential harms from AI. One would combat discrimination related to automation when companies use AI models to review resumes and rental applications. The other would prohibit social media companies from collecting and selling data from people under 18 without their consent or the consent of their guardians.