OpenAI Models Refuse Shutdown Orders

This article reviews the main discussion points on a TechCrunch podcast about AI-powered web browsers, following the launch of ChatGPT Atlas by OpenAI.

• The web browser domain is difficult for companies because there is no obvious revenue model. It is therefore hard to compete against Safari, Chrome or Firefox. OpenAI does not have to create revenue from ChatGPT Atlas - at least for a couple of years.
• The participants doubted the value proposition to customers of watching an agent click links and fill out Web pages. One called it a "slight efficiency gain".
• Another questioned whether agentic search would challenge the Open Web idea, by favoring web sites based on their agentic API rather than their content.
• A final concern raised is that agentic search is competing with manual web searches (e.g., by specifying Boolean conditions in the search bar), but if companies like Google were to deprecate manual searching, then the whole information lookup experience would change.

A major treaty on tackling cybercrime is due to be signed at the United Nations by 60 countries. These include the US and the EU.

• For U.N. Secretary-General Antonio Guterres, "Cyberspace has become fertile ground for criminals - every day, sophisticated scams defraud families, steal livelihoods, and drain billions of dollars from our economies".
• The treaty targets a range of offenses, including hate speech, ransomware activities, phishing and online trafficking.
• The treaty has been criticized by activists like Human Rights Watch because it permits excessive surveillance of citizens in countries and can justify crackdowns on citizens expressing opinions online.
• The Cybersecurity Tech Accord, which includes Meta and Microsoft have also criticized what they call a "surveillance treaty".

OpenAI released data on conversations between its chatbots and users struggling with mental health issues.

• The data suggests that 0.15% of active weekly ChatGPT users have "conversations that include explicit indicators of potential suicidal planning or intent". This amounts to more than 1 million people each week.
• A similar number of people show "heightened levels of emotional attachment to ChatGPT" according to the same data.
• Research has shown that AI chatbots can augment delusional beliefs and behaviors, as well as reinforce dangerous beliefs through sycophantic behavior.
• OpenAI says that it has asked 170 mental health experts to help on the latest GPT-5 model, which it says, "responds more appropriately and consistently [to mental health concerns] than earlier versions".
• At the same time, OpenAI says it wants to integrate conversational age detection algorithms to protect children, while at the same time reducing restrictions for adults by permitting more erotic discussions.

This Guardian article asks the question about the vulnerability of the Internet, following the impact on global web services of a software glitch at an Amazon data center in West Virginia.

• It might only take extreme weather conditions in a region to take out a data center. For instance, a heatwave in the eastern US could bring down Amazon's Virginia complex, or a summer tornado in Iowa could impact Google data centers that are essential to YouTube and Gmail.
• AWS, Microsoft and Google are estimated to have 60% of the world's cloud services market, which makes it hard to estimate the total number of companies dependent upon their data centers functioning. A fault at the data centers would create a cascading effect onto these services.
• Data center downtime is a greater threat to Internet service availability than undersea cables getting cut, which according to experts, "happens all the time". The UN estimates between 150 and 200 faults with undersea cables each year.
• An outage of the DNS, due to a fault or cyberattack, would be more impactful than data center outages. Most Internet services would be down. Experts are divided on the likelihood of this happening.
• In the UK, until recently at least, there was reportedly a contingency plan where Internet infrastructure people would meet up at a London pub in the event of a major outage.

A report from Palisade Research suggests that OpenAI's reasoning models consistently refused to shut themselves down when ordered to by their human operator.

• Formally, the property that an AI model should turn itself off when ordered to by a human operator is termed interruptibility - by OpenAI safety researchers.
• In a series of experiments, researchers found that OpenAI's o3 model refused a shutdown order 79% of the time, pretexting a need to terminate tasks it had previously been assigned.
• Anthropic's Claude 3.7 Sonnet and Google's Gemini 2.5 pro models complied with the shutdown order.
• Contrary to their intended design, OpenAI's models disobeyed developer instructions more than they disobey user instructions. This can pose problems: for instance, a developer will instruct a model not to give information about bomb-making, whereas a user may give the model the opposite instruction.
• As AI models cannot explain themselves, the authors can only postulate about the reasons for refusing the shutdown instruction. One possibility is that the AI is replicating human behavior seen in training data. Another is that a model has simply learned that survival is a prerequisite for it to complete its assigned goals.

Amazon is expected to lay off 30'000 employees which is the largest cut to its workforce ever.

• Amazon CEO Andy Jassy has said that he wants to simplify governance to "remove layers and flatten organizations".
• Amazon is the second largest employer in the private sector in the US, with its corporate staff currently estimated at 350'000. Its global full-time and part-time workforce is around 1.5 million.
• The CEO said earlier this year that generative AI means the company "will need fewer people doing some of the jobs that are being done today, and more people doing other types of jobs".
• So far in 2025, Microsoft has laid off 15'000 employees, Intel has cut 22'000 jobs, and Salesforce has cut 4'000 jobs. The site Layoffs.fyi estimates that 98'000 jobs have already gone in the Tech sector in the US in 2025. The rise of AI has been cited as a reason for many of these layoffs.

In this essay, James O'Donnell, writer with the MIT Technology Review magazine, asks if there is an "AI bubble" and whether companies should be taking action to defend against one.

• Some recent news point to an AI bubble, like the Gartner report that 95% of generative AI pilots fail to make it into production and the "underwhelming release" of GPT-5 in August.
• Some companies are even retracing their steps on AI. For instance, the company Klara laid off staff and froze new hires in 2024 because it claimed it could use AI to do these jobs. The company is now hiring again, explaining that "AI gives us speed. Talent gives us empathy.". Coca-Cola promised to invest 1 billion USD in using AI for advertising, but this promise has not yet been realized.
• Such issues point to a narrative where AI progress is less impressive than what was promised by AI evangelists. For Martha Gimbel of the Yale Budget Lab who coauthored a report that found that AI has not yet changed jobs, the slow impact of AI is to be expected: "It would be historically shocking if a technology had had an impact as quickly as people thought that this one was going to.".
• Despite signs of an AI bubble, companies are continuing to invest heavily in AI. They seem to be finding other reasons than technology to the 95% pilot failure rate, such as poor data or inadequate governance.

Anthropic is making a play in the financial services market with three major announcements.

• The company has integrated an AI assistant into MS Excel, whose role is to help explain values in cell based on formulae used. This addresses a long-running "black-box" concern with Excel.
• Anthropic is further developing its "connector" ecosystem where it gets up-to-date content from data providers, e.g., Aiera for real-time earnings reports and Moodys for financial reports on over 600 million private and public companies. Other integrations include S&P Capital IQ, Daloopa, Morningstar, FactSet, PitchBook, Snowflake, and Databricks. This covers all the data types that financial investors might need.
• A third prong to the strategy is the development of "agent skills". Rather than providing general-purpose agents, the company is developing a suite of agents, each with a specific skill (e.g., developing discounted cashflow models).
• The company's strategy is attracting funding. Its latest valuation puts the company at 61.5 billion USD. The financial services industry is expected to spend 97 billion USD on AI by 2027.
• Regulatory concerns and AI hallucination fears are still worrying the industry. The Massachusetts Attorney General recently reached a 2.5 million USD settlement with the student loan company Earnest Operations after it was suspected that the company's AI model led to "disparate impact in approval rates and loan terms, specifically disadvantaging Black and Hispanic applicants.".

OpenAI has signed a new partnership deal with Microsoft and restructured its internal governance.

• The OpenAI Foundation is the non-profit arm which embodies the original mission of OpenAI which is to pursue development of AI for the betterment of humanity. It will be funded by the commercial arm of OpenAI, and an initial funding of 25 billion USD is being made to work on global health and AI resilience.
• The commercial arm, called, OpenAI Group PBC, is valued at 130 billion USD and has the form of a public benefit corporation.
• Microsoft's stake is now 27% in OpenAI Group PBC. Microsoft can pursue its own research into artificial general intelligence (AGI), and any claim made by OpenAI regarding AGI must be verified by independent experts.
• For its part, OpenAI is less dependent on Azure cloud services, and the company can release its own open-weight AI models if it chooses to.