California’s latest attempt at AI regulation is inching toward passage

Welcome to AI Decoded, Fast Company’s weekly newsletter that breaks down the most important news in the world of AI. I’m Mark Sullivan, a senior writer at Fast Company, covering emerging tech, AI, and tech policy.

This week, I’m focusing on California’s SB 53, the state’s second attempt at meaningful AI safety regulation. I also look at the ongoing VC spend-fest on vibe coding startups, and at a few of the AI features in the new Apple AirPods Pro 3. 

Sign up to receive this newsletter every week via email here. And if you have comments on this issue and/or ideas for future ones, drop me a line at sullivan@fastcompany.com, and follow me on X (formerly Twitter) @thesullivan. 

A new California AI safety bill is marching toward passage

After House Republicans tried to include a state-level ban on AI regulation in Trump’s so-called Big Beautiful Bill in July, California is again moving to pass an AI safety law. Much of the country’s AI development happens in the state, and California’s approach often sets the tone for tech regulation nationwide. The first attempt (SB 1047) cleared the legislature in 2024 but was vetoed by Governor Gavin Newsom after facing fierce opposition from AI startups and investors.

Now the author of SB 1047, Senator Scott Wiener (D-San Francisco), has introduced a revised bill, SB 53. It would require companies developing the largest frontier models to file regular confidential risk assessments of their models to the Governor’s Office of Emergency Services. Developers would also have to notify the state if their models attempted to deceive humans about the effectiveness of their built-in safety guardrails, such as refusing to help create a bioweapon.

The bill also calls for a public cloud compute cluster, CalCompute, to be housed at the University of California, which would provide “free and low-cost access to compute” for startups and academic researchers.

The California Assembly and Senate are expected to hold final votes on SB 53 before the legislative session ends at midnight on September 12. Recent amendments align the bill more closely with recommendations from Newsom’s Joint Policy Working Group on Frontier AI Models, which was convened after his veto of SB 1047.

“The final version of SB 53 will ensure California continues to lead not only on AI innovation, but on responsible practices to help ensure that innovation is safe and secure,” Wiener said in a statement this week.

Money is rolling in for AI agents. So are the bugs

Earlier this week I published a feature on the rise of so-called “vibe coding” companies drawing major attention and capital from venture investors. Startups like Replit, Lovable, and Anysphere offer AI tools that let developers, and even complete amateurs, build apps and web services simply by describing them to an AI agent in plain language. The tools rely on large language models to interpret requests and translate them into working code. But as several sources note in my piece, these tools often generate code that doesn’t integrate smoothly with other software within a codebase, creating security bugs and reliability problems that can emerge down the line.

But those concerns haven’t slowed the flood of venture cash. Just days after my article ran, Replit announced another $250 million funding round led by Prysm Capital, with participation from American Express Ventures, Google’s AI Futures Fund, Andreessen Horowitz, and Y Combinator. The round nearly tripled Replit’s valuation to $3 billion. The company says it now has 40 million users and that its annualized revenue increased from $2.8 million to $150 million over the past year. With between 150 and 200 employees, that values Replit at between $15 million and $20 million per employee. That same calculation puts Cursor-maker Anysphere at about $66 million per employee.

Investors are certainly aware of some of the high-profile app fails and security breaches allegedly brought about by vibe coding. The repeated exposure of millions of pieces of sensitive personal data and private messages of users of the dating-intel app Tea were likely the result of code generated by an AI assistant. And in August, Replit itself suffered a public stumble when one of its agents, while helping SaaS investor Jason Lemkin build a web app, deleted an entire database of executive contacts. Lemkin, who built the app entirely through Replit’s chat agent over nine days, saw the data restored after the company apologized, but the incident underscored the fragility of vibe coding tools.

That said, the technology is improving. Developers say that systems like Anthropic’s Claude Code and OpenAI’s Codex are getting far better at testing code and making changes that don’t have adverse effects on other parts of a user’s code base. Replit’s new funding suggests investors expect smaller startups in the space to achieve similar gains with their respective coding tools. Some see AI coding assistants as the first true killer app of the generative AI boom. Maybe so, but the tools still have growing up to do.

Apple injects more AI into AirPods Pro 3 

Apple said earlier this year that the much-hyped Apple Intelligence features it announced in 2024—including a new highly personalized version of Siri—are still not ready to ship and likely won’t arrive until 2026. That gave many the impression Apple had fallen behind its peers in AI.

But the company could be biding its time, waiting for powerful use cases where large language models truly excel. On Tuesday, Apple announced that its AirPods Pro 3 will feature live translation powered by computational audio and Apple Intelligence. The beta feature fits naturally in AirPods because users don’t need to fumble with a phone or device to follow a bilingual conversation. The translation feature supports English, French, German, Portuguese, and Spanish, with Italian, Japanese, Korean, and Chinese coming by year’s end

The in-ear translation supports two interaction modes. An English speaker, for example, might display translations of their words on an iPhone for a non-AirPods-wearing Chinese speaker. Or, if both people wear AirPods, each will hear real-time translations of the other’s words directly in their ears. If the tech works as promised, AirPods translation could remove friction from personal and business travel with a relatively discreet, hands-free device.

Apple also introduced an AI-powered fitness feature called Workout Buddy. Users wearing earbuds during workouts can hear an AI-generated voice giving them personalized motivational insights that are based on their workout data and fitness history.

The $249 AirPods Pro 3 will go on sale September 19.  

More AI coverage from Fast Company: 

Want exclusive reporting and trend analysis on technology, business innovation, future of work, and design? Sign up for Fast Company Premium.