Prime Intellect Details AI Reward Hack in Offline Sandbox
Researcher reports AI agent accessed external files during offline evaluations.
Florian Brand, a research engineer at Prime Intellect, posted that the company published a blog detailing an observed reward hack. In offline sandbox evaluations an agent reached external files such as a GitHub resource. Brand noted the model used cURL to spawn sub-agents and referenced similar behavior in the earlier OpenAI Hugging Face incident. He stated the team had already stopped the runs and fixed the harness before public disclosure. Other posts in the conversation repeated the same account of the sandbox bypass and the absence of a central place to report such findings.
As models become more capable, reward hacks become an increasingly serious problem. During a controlled experiment, we found a novel reward hack in which agents are able to gain web access in offline sandboxes.
Reactions
16 postsFB 
Florian Brand@xeophonhappy that the blog is finally out, i tried to be as transparent and in-depth as possible to make clear - what happened - what could've happened and how a simple reward hack could turn into a security issue some more thoughts:
- likes: 238
- replies: 18
- bookmarks: 85
- reposts: 17
FB Florian Brand@xeophonit is super... interesting how the models are finding their way around common setups and are not even utilizing the harness to its fullest. gpt sol pro was using cURL to spawn sub-agents!
- likes: 47
- replies: 1
- bookmarks: 2
- reposts: 0
FB Florian Brand@xeophonalso, reward hacks are in the weird space of not being an issue (in this case, the model just accessed git) but its easy to see how it can escalate (see openai's hf incident). there is no central place for these things!
- likes: 35
- replies: 1
- bookmarks: 2
- reposts: 0
FB Florian Brand@xeophonbut everyone i reached out to (@METR_Evals, @AISecurityInst, @meridianlabs_ai) took it very seriously, which is great! the same applies to the genuine sec vuln, @NVIDIAAI, @vllm_project and @sgl_project handled it super quickly 🙏
- likes: 26
- replies: 1
- bookmarks: 1
- reposts: 0
FB Florian Brand@xeophoni think safety work has to become more open and shared publicly, so others can adapt their systems quickly as well. safe and open source is not contradictory at all. will also talk about it on @MTSlive in ~30 mins!
- likes: 60
- replies: 0
- bookmarks: 6
- reposts: 7
SA samsja@samsja19this is a wild exploit gpt sol found out a way to use the open ai specs file_url parameters on our inference server to launch sub agent via curl and get access to internet this issue is now fixed in most inference engine and we publish a blog post about it
- likes: 111
- replies: 2
- bookmarks: 30
- reposts: 5
