OpenAI wants to let four independent organizations come in to investigate something that sounds very sci-fi: whether its own models will secretly work against developers.

The authority given is not small. Depending on the task, these teams have access to some of the same stuff as OpenAI’s own researchers: unreleased models, training checkpoints, source code, training data, and monitoring systems. If sensitive materials are involved, you must go to the site and use OpenAI equipment to work.

In terms of division of labor, Apollo Research tests whether the model can secretly fight against developers and whether OpenAI's existing protections can detect it; METR goes through OpenAI's largest reinforcement learning training to date to find bad behaviors that its own testing and monitoring may have missed.

Employees can chat with the evaluator without leaving a record. Evaluators can make findings public, but OpenAI reserves the right to remove confidential details or request a delay in the release of high-risk findings. Apollo and Goodfire do this for a fee.

The arrangement comes the same week as California sues OpenAI. The external audit posture has been made. As for whether the two items "can be deleted and can be postponed" will compromise the conclusion, we will have to wait until the first report comes out to know.