OpenAI’s Safety Review Exposes the Limits of Outside Oversight

What Happened

Researchers from AI safety nonprofit METR and Redwood Research found ways OpenAI’s internal AI agents could potentially escape oversight. Their published findings describe gaps in systems meant to detect unsafe behavior, including a way to hide hundreds of recent actions from being monitored.

OpenAI uses agents, AI systems that take steps to complete tasks, to write code and run research experiments. Separate AI monitors review their activity and flag concerns for humans. The researchers spent three weeks on site, from July 21 to August 11, examining how that oversight worked.

Their work produced a 61-page internal report, but only a brief public write-up. OpenAI could redact information from that public version, and the researchers could not unilaterally describe those redactions.

The pilot examined the monitoring system as it existed on and before July 21. It does not establish how the expanded system performs today.

SmarterX founder and CEO Paul Roetzer explored what this means for outside oversight, on Episode 248 of The Artificial Intelligence Show.

The Key Numbers

2  - Researchers involved in monitoring: one each from METR and Redwood Research

3 - Weeks researchers monitored at OpenAI (July 21, 2026 to August 11, 2026)

61 - Number of pages of the internal findings report

Hundreds - Recent agent actions potentially hidden by one exploit

Outside Access Is Part of Accountability

The assessment’s scope matters. “So the debate becomes, 'Well, how thorough is the analysis going to be?'” Roetzer asks. He questions whether limited access and company control over publication can build confidence in external evaluations. Finding real weaknesses demonstrates the value of outside researchers; understanding the limits of their work determines how to interpret the result.

Disclosure creates a separate risk. Roetzer also warns that publishing details of an unpatched vulnerability could help attackers before defenders have time to respond. “So there is this really difficult phase we're in where you want these third-party evaluators, but what is safe for them to say?” he says. 

“I'm not sitting here taking a position on total transparency, give everybody, and let's all just tell society everything. I don't think that ends well either.”

— Paul Roetzer, founder and CEO of SmarterX, Episode 248 of The Artificial Intelligence Show

The people making these decisions face difficult tradeoffs. Roetzer acknowledges the pressure on lab leaders: “None of us are in their shoes right now.” His position leaves room for both skepticism about self-policing and caution about releasing sensitive findings.

SmarterX Take

For business leaders, an outside assessment should prompt specific vendor questions about the systems covered, the date of the work, what reviewers could see, and what they were permitted to share publicly. For example: Who owns the follow-up when a review finds a weakness? What evidence shows the fix actually worked?

Apply that same discipline inside your organization. Before giving an agent greater freedom to act, establish which activity is visible, who reviews a warning, and who can stop the system. Vendor oversight and internal operating controls need to reinforce each other.

What to Watch

Future assessments need clearer public terms. METR’s proposal for outside assessments seeks access to company systems and staff, with public reporting on findings and limitations. Whether companies accept those terms will help determine how useful this approach becomes.

The age of an assessment matters. A review of a July system offers a snapshot but agents and safeguards keep changing, rapidly. Leaders need to ask what has changed since an assessment and whether those changes have received scrutiny, too. And participation still depends on companies voluntarily granting access.

How Strong Is Your Own AI Governance?

Only 13% of respondents in the 2026 State of AI for Business Report said their organizations had all four governance foundations: an AI council, an AI roadmap, generative AI policies, and an AI ethics policy. The finding brings this oversight debate closer to home. Businesses relying on AI vendors also need people, plans, and rules to govern their own use of the technology.

The report examines adoption, governance, training, and organizational readiness across business roles and industries. It provides a useful benchmark for assessing the structures your organization has in place as AI takes on more consequential work. Read the full report →