For years, the world’s most powerful artificial intelligence models have lived inside the walled gardens of massive technology corporations. Companies like Google, Meta, and Microsoft currently dictate the rules, safety standards, and release schedules for this life-changing technology. However, Chris Olah, a senior leader at the AI startup Anthropic, recently issued a warning that could reshape the entire industry. He argues that we must move AI safety and governance outside of the control of Big Tech if we hope to protect humanity’s future.
Olah, a renowned researcher who helped pioneer much of the foundational work on AI interpretability, believes the current system creates a dangerous conflict of interest. When a company earns $1 billion in quarterly profit from a specific AI product, it has a strong incentive to overlook potential risks or “hallucinations” that might hurt its bottom line. According to Olah, we cannot expect these corporations to prioritize safety when their primary goal is satisfying shareholders and maintaining stock prices.
The current model relies on “self-regulation.” This means the companies that build the most dangerous software in the world are also the ones responsible for telling us if it is safe. Olah compares this to letting tobacco companies fund the research that determines whether cigarettes cause cancer. He argues that society needs an independent, well-funded body that operates without the pressure of corporate revenue targets. This group would have the power to audit models before they reach the public, regardless of whether a company’s CEO wants them to wait.
Transparency represents the core of Olah’s proposal. Right now, most AI research happens behind closed doors. Developers treat their training methods and safety data as closely guarded trade secrets, often citing competitive advantage. Olah suggests that we need a “shared library” of safety research. If a researcher at one company discovers a way to prevent an AI from generating harmful content, that knowledge should belong to the public, not just to the board of directors at one specific firm.
The sheer scale of the investment makes this a difficult fight. Big Tech firms pour massive fortunes into GPU clusters and talent acquisition. When a single firm spends $50 billion on new data centers in a short timeframe, it gains a level of influence that dwarfs most government agencies. Olah points out that even the most well-meaning government regulators often feel outmatched by the sheer speed of development in the private sector. The government needs to catch up, or it risks becoming a bystander in the most important technological transition of the century.
Some critics of this view argue that moving safety oversight to a public or independent body would slow down progress. They worry that a slower pace will allow bad actors—such as foreign state-sponsored hacking groups—to seize the lead. However, Olah counters that true safety is actually the fastest path to long-term success. If an AI system causes a global financial shock or accidentally leaks critical infrastructure data, public trust in the entire sector will collapse overnight. A 1.5% chance of catastrophic failure is simply too high when we are talking about systems that control energy grids, financial markets, and healthcare services.
We have seen what happens when the public is left out of the loop. When AI models began generating deepfakes or spreading misinformation, most companies were caught completely flat-footed. They had to scramble to add “guardrails” after the damage was already done. Olah wants to flip this script. He proposes that we shift from a reactive mindset to a proactive one. This means testing for “agentic” risks—where an AI takes physical actions—before the models are even connected to the internet.
Building an independent safety ecosystem will be expensive, but it is necessary for economic stability. Olah suggests that a portion of the massive AI-driven tax revenue should fund this global, nonprofit safety organization. This group would act as a neutral third party, similar to how aviation regulators oversee the safety of airplanes. Airlines compete for customers, but they all follow the same safety rules written by the government. AI companies should operate under the same framework.
The tech industry sits at a crossroads. As companies continue to push for faster and more capable models, the potential for harm increases exponentially. We are moving from simple chatbots that write poetry to complex systems that manage our daily lives. If we don’t demand more accountability now, we risk creating a world where we rely on “black box” systems that no human truly understands.
Olah is not saying we should stop building AI. Instead, he is demanding that we change the culture around it. We need to stop viewing AI safety as a corporate PR task and start viewing it as a global public safety issue. Whether the big corporations will listen remains to be seen, but the argument for an independent safety foundation is gaining steam. It is time for a new chapter in the AI story—one where the public, not just the boardrooms, has a seat at the table.
