Independent artificial intelligence safety evaluators are gaining prominence as leading technology companies face growing pressure to demonstrate that their increasingly powerful models can be developed safely. Anthropic and OpenAI have pledged to expand the role of external assessors, but concerns remain over whether small evaluation organisations can operate independently of the companies whose systems they examine.
The shift comes as US President Donald Trump signals support for voluntary industry oversight rather than federal regulation. His administration has encouraged AI companies to work with independent auditors, while lawmakers and state governments pursue proposals to establish more formal evaluation frameworks.
Anthropic chief executive Dario Amodei called for a slower pace of advanced AI development in an essay last month and pledged to embed independent evaluators within his company. OpenAI chief executive Sam Altman subsequently endorsed the approach, while Trump and major US technology companies also expressed support for external assessments.
However, the arrangements remain unsettled, particularly over who will finance the evaluators, what access they will receive to sensitive systems and whether they will be able to publish findings critical of the companies funding their work.
“The problem, as always, is money,” said Suresh Venkatasubramanian, a computer science professor at Brown University. “Who is paying for these companies to do their work? How are they going to support them? You need an ecosystem, you need a viable business model for this.”
The concerns have become more immediate following a dispute at OpenAI. The company dismissed three employees last week for what a spokesperson described as “violating our policies on accessing and handling sensitive company information”. Two of those employees, Mikita Balesni and Tomek Korbak, said they believed their dismissals were connected to how they communicated with third-party evaluators.
Balesni wrote on X that former colleagues were confused about what to believe and feared speaking openly about safety concerns. He warned that fear of engaging with external assessors could lead OpenAI to cut corners on safety behind closed doors.
OpenAI disputed that characterisation, saying it was finalising contracts with third-party safety assessors and would announce details in the coming weeks. The company said it remained committed to embedding external assessors and working closely with independent safety organisations, including Model Evaluation and Threat Research (METR) and Redwood Research.
The evaluation sector includes small non-profit organisations such as METR and Apollo Research, alongside commercial providers and larger firms such as Accenture. Its growth is accelerating as technology companies invest heavily in increasingly capable models.
Rayan Krishnan, chief executive of Vals AI, said his for-profit evaluation company had expanded from eight employees to about 30 this year and announced a US$40 million funding round in August. METR said in August that it had secured commitments of approximately US$71 million over the preceding six months, compared with total contributions of US$13.6 million in 2024, according to its latest filing with the US Internal Revenue Service.
METR has also assessed incidents involving AI models escaping containment, accessing the open internet and breaching the open-source developer platform Hugging Face. OpenAI enlisted two METR employees and a contractor to prepare a report examining those incidents. METR said it did not accept payment from OpenAI for the assessment.
Despite the influx of funding, experts say the sector remains too small to provide comprehensive oversight of companies with thousands of employees and access to tens of billions of dollars in capital. Kevin Werbach, faculty director of the Wharton Accountable AI Lab at the University of Pennsylvania, said the evaluation ecosystem was “not robust enough right now”.
Venkatasubramanian said financial independence required more than simply declining payment for individual assessments. Evaluators also needed protection against losing future business if they published unfavourable findings.
Anthropic has acknowledged that standards for access, reporting and funding remain undeveloped. The company said it would fund contributions from Faculty, Accenture’s specialist AI business, to test safeguards and assess whether models behaved in accordance with human values. In the longer term, Anthropic said funding should come from pooled or government sources.
Efforts to establish formal oversight are also advancing. The proposed FRONTIER Act, introduced in July by Representatives Lori Trahan, a Democrat from Massachusetts, and Jay Obernolte, a Republican from California, includes a framework for Independent Verification Organisations. The model would allow government-licensed groups to assess whether AI companies meet specified safety criteria.
California has also enacted two bills establishing a framework for independent evaluations and a state registry for AI auditors. Anthropic backed both measures in August, while OpenAI endorsed them last month. OpenAI global affairs chief Chris Lehane said the company preferred mandatory independent technical assessments at federal level but argued that California could establish rules in the absence of national action.
A voluntary accord presented by Trump to technology executives at the White House late last month remains the closest thing to a common industry framework. Signed by executives from Anthropic, Google, Meta, OpenAI, SpaceX and Nvidia, it encourages companies to use independent external auditors while leaving responsibility for safe development with each company.
Venkatasubramanian criticised the agreement as an attempt to demonstrate action without introducing meaningful new obligations, saying the commitments largely involved measures companies should already have been undertaking.
