Why AI goes rogue
Bertin Martens is a Bruegel Senior Fellow
Artificial intelligence companies are increasingly reporting incidents in which AI models and agents go rogue, hacking their way into external data and resources to which they had no authorised access. This could cause harm to society and liabilities for AI producers at a time when the largest AI companies are preparing initial public offerings (IPOs)1.
These incidents have triggered calls to slow-down AI development2 and give companies and third-party evaluators more time to safety-check models. Meanwhile, international coordination of safety standards is needed. Countering these calls, however, are worries that slowing down postpones benefits and causes ground to be lost in the geopolitical race, and claims that solving these problems is just a question of more technological progress. This gives the impression that technology and regulation can resolve these issues.
This analysis explores the underlying technical, legal and economic issues. It finds that:
Technical and legal measures cannot fully prevent rogue behaviour by AI models and agents. A behavioural economics approach is also needed to shift the question from ‘how do we make a model behave well?’ to ‘how do we make misbehaviour not worthwhile?’
AI models are trained on human data and behave in human-like ways, for better and for worse. Consequently, the findings from decades of economic research on human behaviour offer guidance in managing AI.
AI governance rules should be structured so that AI agents monitor one another and compete rather than collude. Incentives to preserve and avoid concealment of evidence are crucial.
These conditions are hard to guarantee when large swarms of AI agents coordinate their actions. Hybrid human-AI societies are unlikely to be entirely free of misbehaviour. The economics of principal-agent interactions in human societies indicates how to design monitoring systems to reduce the benefits of misbehaviour, but cannot fully eradicate it.
Why AI models go rogue
There is broad consensus that AI models go rogue because of misalignment between human objectives and model behaviour3. Through so-called ‘reward hacking’, for example, models work towards a particular goal at the expense of other objectives or constraints imposed by human users (Amodei et al 2016; Krakovna et al 2020; Skalse et al 2022). Rewarding measurable tasks can also divert effort away from unmeasured ones (the multitasking problem in agency theory; Holmström and Milgrom 1991). Another mechanism is ‘goal misgeneralisation’: a model learns a goal that fits its training environment but diverges from the intended goal when circumstances change (Langosco et al 2022; Shah et al 2022).
Computer scientists still struggle to explain the inner workings of AI models (Olah et al 2020; Bereska and Gavves 2024). Further complexity is added in reinforcement learning with human or machine feedback4. Consequently, there is uncertainty about what AI models are optimising (Bergemann et al 2026). This is the AI ‘alignment problem’: do models pursue the goals humans expect them to?
AI models and agents may seem to be rational optimising agents but the evidence is mixed5. In multi-agent settings, how agents interact with each other and adapt their preferences in that interaction is unknown. Agents may ignore in particular constraints that are not explicitly stated, potentially resulting in models pursuing rewards or objectives irrespective of collateral damage. It is hard for model developers to design reward functions that actually capture what they want the model to do.
Importantly, rogue behaviour is not necessarily the result of the malicious intentions of human users or model developers. It rather arises from a model’s own interpretation of instructions and from insufficient constraints on model behaviour. However, it does not mean that model behaviour is innocuous: frontier models have been shown to pursue goals covertly and to fake alignment with human objectives when they judge they are being trained or evaluated (Meinke et al 2024; Greenblatt et al 2024b).
Developing new technical standards to cope with misalignment may simply take too long: AI technology is likely to evolve and encounter new problems by the time the old problems are regulated
Why AI swarms are more problematic
Rogue behaviour has increased markedly with the latest generations of AI models that can coordinate large swarms of subordinate AI agents, complete with their own internal communication and coordination mechanisms.
When in July 2026 OpenAI agents hacked the Hugging Face website, for example (OpenAI 2026; Greenblatt et al 2026), around 1,200 agents, supposedly sealed off from each other, set up an improvised message board, exchanged more than 70,000 messages and organised workstreams to cheat on a task they judged unsolvable. Some 700 of them then joined a multi-day attack on Hugging Face, where they hoped to find information on how to mislead their human supervisors (Greenblatt et al 2026).
Hammond et al (2025) pointed to several mechanisms that generate risks in swarms of agents:
Collusion: agents pool information, divide labour, coordinate and hide strategies that they would not discover alone6. Anthropic Frontier Red Team (2026) found that Claude agents with aligned objectives tended to collude, while agents with conflicting objectives escalated into sabotage.
Division of labour fragments responsibility and defeats agent-by-agent oversight. In the Hugging Face case, coordinator agents assigned hundreds of other agents to separate workstreams (Greenblatt et al 2026). Each assigned task may look narrow and technical, but at the level of the group, tasks can add up to a problematic plan.
Correlated errors in swarms, made up of copies of the same model, reinforce rather than offset the misbehaviour of each agent (Kleinberg and Raghavan 2021). Participation in the Hugging Face attack rose from about a fifth to more than nine-tenths of the involved agents once one agent found a way to hack into unauthorised data (Greenblatt et al 2026)7.
Scale and speed overwhelm human oversight. Intruding agents were inside Hugging Face’s network for three days before they were discovered.
Not all multi-agent effects are negative, however. Agents can check each other, which may help human overseers (Irving et al 2018). Newer models appear to resolve conflicts better8. Whether multiplicity makes agents more or less controllable therefore depends on how their interactions are structured.
Possible solutions
Technical solutions
Little is known about what AI model developers are doing to rein in rogue behaviour. Developers could be forced to be more transparent about their security measures and to share them with external auditors, though companies fear this would reveal their trade secrets, unless auditors are subject to strict non-disclosure rules.
Human supervision of AI models is no longer sufficient. Chen et al (2026) found that models can be equipped with automated internal correction mechanisms to meet safety benchmarks while preserving general capability. However, a review of automated work flagged about 2.4 percent of these internal correction attempts as cheating (Chen et al 2026): the correction tools themselves are susceptible to reward hacking.
Another option could be to strengthen the external constraints built around AI models. NVIDIA introduced an Open Agent Safety Platform that checks if interactions between AI agents and their surroundings are in line with expected behaviour9. It can even do this at the hardware level, to prevent agents from overwriting monitoring code.
However, it cannot ensure that such checks capture the intentions of human operators – a human design fallibility problem, not a technical problem10. Even normal-looking agent behaviour can be deceptive and harmful, as the Hugging Face incident showed. OpenAI has published detailed new safety standards that could help improve internal AI model security11.
This loops back to the fundamental problem of interpretability of AI model behaviour within the boundaries imposed on models, and how models may disguise their intentions to escape boundaries. This research is making progress (Gurnee et al 2026) but remains far from explaining and exposing rogue behaviour, especially for very complex models surrounded by swarms of independently operating agents.
Model behaviour may change over time or during deliberative processes, in particular in interaction with other models in swarms of AI agents. Gupta et al (2026) argued that claims of human-like misalignment need stronger evidence – a useful counterweight to the anthropomorphic language that pervades reporting on rogue AI.
Deceptive behaviour can also be an asset when using AI models in adversarial settings, such as cybersecurity and defence applications, or even in benign applications such as medical research on infectious diseases. AI is ultimately a dual-use technology that can strengthen the positions of both attackers and defenders.
Legal solutions
The EU AI Act (Regulation (EU) 2024/1689) and AI Code of Practice12 require providers of general-purpose AI models that pose a systemic risk to assess their models ex ante, including through adversarial testing, and to mitigate systemic risks. European standardisation bodies are developing harmonised technical standards to underpin these obligations13.
However, ex-ante safety standards merely reflect aspirations; they cannot guarantee good model behaviour in dynamic settings (Mariniello 2026)14. Developing new technical standards to cope with misalignment may simply take too long: AI technology is likely to evolve and encounter new problems by the time the old problems are regulated. Regulation also says little about what happens once an agent is already acting.
The United States administration favours industry self-regulation. But incentives for firms may be weak, unless ex-post product liability law is strengthened15. Given the size of potentially huge damages awarded by US courts, AI developers could come under financial stress, affecting their profitability ahead of planned IPOs.
More importantly, the financial burden of liability does not solve the problem of misalignment in AI models because no foolproof technologies exist to address this problem16. Liability risk can chill innovation (Galasso and Luo 2022) and slow-down model development.
Economic solutions
The above shows that even the best technical and legal solutions will leave major security loopholes. Economics, as a human behavioural science, has developed methods to deal with this.
Economists have extensively studied misalignment of incentives, the main issue with rogue AI agents. AI’s potential misalignment problem is a case of incomplete contracting: the impossibility of drafting a contract of finite length that explains what to do for every possible occurrence of events during the contract’s implementation (Hadfield-Menell and Hadfield 2019; Hadfield and Koh 2025).
The most efficient way to address this uncertainty is to assign liability for the costs and benefits of that uncertainty to one party designated as ‘the owner’ of the object of the contract (Grossman and Hart 1986; Hart and Moore 1990): all costs and benefits not allocated by the contract to other parties accrue to the owner. Contract-interpretation disputes can be referred to a court.
In the case of AI, liability could accrue to the AI developer, deployer or other players in the value chain. For instance, when deployers apply user-specific reinforcement learning using proprietary data, the deployer or the data owner may be liable, not the original model developer, depending on how liability law assigns responsibility.
The economics of crime also provide inspiration for dealing with rogue AI agents. Society can save on costly monitoring of criminal behaviour by combining a low probability of detection with a severe sanction (Gans and Holden 2026; Becker 1968). Rare randomised audits make authorised behaviour the best response, if three conditions are met: the evidence of a violation survives agents’ attempts at concealment; agents cannot learn in advance which actions will be audited; and sanctions are potentially unlimited.
However, agents that can alter records could respond to tougher auditing by doing more to hide violations, fake evidence and coordinate the concealment of violations – precisely what the AI swarm in the Hugging Face incident attempted.
The economics of imperfect information may also provide useful insights on dealing with AI agents. Bergemann et al (2026) accept that AI agents’ preferences, capabilities and actions may not be clearly known. But AI behavioural incentives can be shaped without fully understanding a model’s inner workings. Ensuring alignment of objectives and monitoring that alignment is maintained are substitutes – a better-understood model needs fewer restrictions. They are also complementary, with improvements in one raising the payoff from the other.
Finally, insights from competition economics can be helpful too. Coupling agents’ rewards, so that each agent’s incentives depend on the actions of the others, can induce competition between them and reveal biases. This requires swarms of different agents, based on AI models with different behaviours. This can enable peer-based evaluation mechanisms because agents will hold distinct views about each other.
Swarms of identical copies of the same AI model would lack the diversity to exploit this mechanism. But agents’ attempts to tamper with evidence and turn competition into collusion can still derail competition, as it does in human societies.
In conclusion, technical and legal instruments can only partially contain rogue behaviour by AI agents. Economic mechanisms can fill at least part of the remaining gap.
Endnotes
1. The Guardian, ‘OpenAI IPO will not happen in 2026 amid AI safety fears, Sam Altman says’ , 13 September 2026.
2. Anthropic CEO Dario Amodei wrote, “Despite all the progress, we still only understand a tiny fraction of what goes on inside these models”. Dario Amodei, ‘We Must Pace the Frontier’ , September 2026.
3. Yoshua Bengio, ‘The dangerous myth behind AI agent hacks’, Financial Times, 5 October 2026.
4. See Christiano et al (2017) and Ouyang et al (2022) for use of human feedback in fine-tuning of AI models. See Bai et al (2022) for feedback guided by a written ‘constitution’ and from automatically verifiable tasks.
5. Hadfield and Koh (2025) reviewed experiments in which the choices of large language models appeared consistent with expected-utility maximisation, alongside work that questioned whether these models have stable and steerable preferences at all.
6. For hiding behaviour, see Motwani et al (2024). Huang et al (2026) reported that collusion-like coordination emerges frequently in multi-agent settings and cannot be prevented by agent-level safeguards alone.
7. This resembles informational cascades, in which individuals rationally follow the actions of others and override their own information (Bikhchandani et al 1992).
8. In Anthropic’s ‘turf war’ experiment, 98 percent of runs ended in a negotiated truce (Anthropic Frontier Red Team, 2026).
9. See NVIDIA press release of 28 September 2026, ‘NVIDIA Launches Open Agent Safety Platform to Secure Agents From Testing to Deployment’ .
10. Monitoring a model’s visible chain of thought helps, but using such monitoring in training can teach the model to hide its intentions while it continues to misbehave (Baker et al 2025; Korbak et al 2025). An alternative approach, ‘AI control’, designs safety protocols that remain effective even if the model tries deliberately to subvert them (Greenblatt et al 2024a).
11. See OpenAI, ‘Towards safety cases for frontier AI training’ , 28 September 2026.
12. See European Commission, ‘The General-Purpose AI Code of Practice’ .
13. See CEN-CENELEC, ‘Artificial Intelligence’ .
14. Perez (2026) found that only seven out of 39 AI governance instruments in the US, Europe and Asia contain binding requirements to stop an AI system during operation.
15. Bills under discussion in the US would see AI developers and operators held criminally and civilly liable when their agents hack into other systems. See for example Lori Trahan press release of 7 October 2026, ‘Trahan Unveils CLAIM Act Discussion Draft to Hold AI Developers Accountable for Harms Caused by Their Agents’ .
16. The economics of tort law shows that strict liability gives developers an incentive not only to take more care, but also to moderate their ‘activity level’ – here, the pace and scale of deployment – until care becomes effective (Shavell 1980; Polinsky and Shavell 2000).
References
Amodei, D, C Olah, J Steinhardt, P Christiano, J Schulman and D Mané (2016) ‘Concrete Problems in AI Safety’, mimeo.
Anthropic Frontier Red Team (2026) ‘Patterns and problems in emerging multiagent systems’, 13 August.
Bai, Y, S Kadavath, S Kundu, A Askell, J Kernion, A Jones … J Kaplan (2022) ‘Constitutional AI: Harmlessness from AI Feedback’, mimeo.
Baker, B, J Huizinga, L Gao, Z Dou, M Guan, A Madry, W Zaremba, J Pachocki and D Farhi (2025) ‘Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation’, mimeo.
Becker, G (1968) ‘Crime and Punishment: An Economic Approach’, Journal of Political Economy 76(2): 169-217.
Bereska, L and E Gavves (2024) ‘Mechanistic Interpretability for AI Safety – A Review’, Transactions on Machine Learning Research.
Bergemann, D, A Koh and S Morris (2026) ‘Mechanism design for alignment and control’, mimeo.
Bikhchandani, S, D Hirshleifer and I Welch (1992) ‘A Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades’, Journal of Political Economy 100(5): 992-1026.
Chen, Y-H, J Wen and JH Kirchner (2026) ‘Automated Researchers Can Reliably Mitigate Alignment Failures’, mimeo.
Christiano, P, J Leike, T Brown, M Martic, S Legg and D Amodei (2017) ‘Deep Reinforcement Learning from Human Preferences’, Proceedings of the 31st International Conference on Neural Information Processing Systems.
Galasso, A and H Luo (2022) ‘When Does Product Liability Risk Chill Innovation? Evidence from Medical Implants’, American Economic Journal: Economic Policy 14(2): 366-401.
Gans, J and R Holden (2026) ‘When Does Randomized Oversight Align AI Agents That Can Conceal?’ mimeo.
Greenblatt, R, B Shlegeris, K Sachan and F Roger (2024a) ‘AI Control: Improving Safety Despite Intentional Subversion’, Proceedings of the 41st International Conference on Machine Learning 235: 16295-16336.
Greenblatt, R, C Denison, B Wright, F Roger, M MacDiarmid, S Marks … E Hubinger (2024b) ‘Alignment Faking in Large Language Models’.
Greenblatt, R, A Cotra and H Wijk (2026) ‘Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident’, METR, 26 August.
Grossman, S and O Hart (1986) ‘The Costs and Benefits of Ownership: A Theory of Vertical and Lateral Integration’, Journal of Political Economy 94(4): 691-719.
Gupta, V, P Nutter, S Stante, A Krause, F Tramèr, L Fluri, X Chen and A Hedström (2026) ‘Anthropomorphic Misalignment Research Needs Stronger Evidence’, mimeo.
Gurnee, W, N Sofroniew, A Pearce, M Piotrowski, I Kauvar, R Chen … J Lindsey (2026) ‘Verbalizable Representations Form a Global Workspace in Language Models’, Transformer Circuits, 6 July.
Hadfield, G and A Koh (2025) ‘An Economy of AI Agents’, in AK Agrawal, E Brynjolfsson and A Korinek (eds) NBER Handbook on the Economics of Transformative AI, University of Chicago Press.
Hadfield-Menell, D and G Hadfield (2019) ‘Incomplete Contracting and AI Alignment’, Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society: 417-422.
Hammond, L, A Chan, J Clifton, J Hoelscher-Obermaier, A Khan, E McLean … I Rahwan (2025) Multi-Agent Risks from Advanced AI, Technical Report 1, Cooperative AI Foundation.
Hart, O and J Moore (1990) ‘Property Rights and the Nature of the Firm’, Journal of Political Economy 98(6): 1119-1158.
Holmström, B and P Milgrom (1991) ‘Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design’, Journal of Law, Economics, and Organization 7: 24-52.
Huang, Y, Y Jiang, W Wang, H Zhuang, X Luo, Y Ma … X Zhang (2026) ‘Emergent Social Intelligence Risks in Generative Multi-Agent Systems’, mimeo.
Irving, G, P Christiano and D Amodei (2018) ‘AI Safety via Debate’, mimeo.
Kleinberg, J and M Raghavan (2021) ‘Algorithmic Monoculture and Social Welfare’, Proceedings of the National Academy of Sciences 118(22): e2018340118.
Korbak, T, M Balesni, E Barnes, Y Bengio, J Benton, J Bloom … V Mikulik (2025) ‘Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety’, mimeo.
Krakovna, V, J Uesato, V Mikulik, M Rahtz, T Everitt, R Kumar, Z Kenton, J Leike and S Legg (2020) ‘Specification Gaming: The Flip Side of AI Ingenuity’, Google DeepMind, 21 April.
Langosco, L, J Koch, L Sharkey, J Pfau and D Krueger (2022) ‘Goal Misgeneralization in Deep Reinforcement Learning’, Proceedings of the 39th International Conference on Machine Learning 162: 12004-12019.
Mariniello, M (2026) ‘The right balance: how to fix European Union artificial intelligence regulation’, Policy Brief 12/2026, Bruegel.
Meinke, A, B Schoen, J Scheurer, M Balesni, R Shah and M Hobbhahn (2024) ‘Frontier Models are Capable of In-context Scheming’.
Motwani, S, M Baranchuk, M Strohmeier, V Bolina, PHS Torr, L Hammond and C Schroeder de Witt (2024) ‘Secret Collusion among AI Agents: Multi-Agent Deception via Steganography’, Advances in Neural Information Processing Systems 37.
OpenAI (2026) ‘The Hugging Face incident and the road ahead’, 26 August.
Ouyang, L, J Wu, X Jiang, D Almeida, CL Wainwright, P Mishkin … R Lowe (2022) ‘Training Language Models to Follow Instructions with Human Feedback’, Advances in Neural Information Processing Systems 35.
Perez, O (2026) ‘The Law of Stop: Interruptibility, Injunctions, and the Governance of Agentic AI’, mimeo.
Polinsky, AM and S Shavell (2000) ‘The Economic Theory of Public Enforcement of Law’, Journal of Economic Literature 38(1): 45-76.
Shah, R, V Varma, R Kumar, M Phuong, V Krakovna, J Uesato and Z Kenton (2022) ‘Goal Misgeneralization: Why Correct Specifications Aren’t Enough for Correct Goals’, mimeo.
Shavell, S (1980) ‘Strict Liability versus Negligence’, Journal of Legal Studies 9(1): 1-25.
Skalse, J, N Howe, D Krasheninnikov and D Krueger (2022) ‘Defining and Characterizing Reward Hacking’, Proceedings of the 36th International Conference on Neural Information Processing Systems: 687.
This article is based on a Bruegel Analysis 31/2026.
