A safe AI model is not a safe society

This is an opinion piece by Stefano Nichele, a professor at Østfold University College and guest researcher at Simula Metropolitan working on AI safety. He responds to the Norwegian government's budget proposal for 2027, which allocates 45 million kroner to KI Norge, 30 million of it for safety and model evaluation. The unit would also get an expanded safety mandate and a role in international cooperation. Nichele welcomes this and agrees that independent testing and alignment of advanced models are needed. His argument is that a model, an autonomous agent and a society of agents are different systems, and safety at one level does not automatically carry over to the next.

A model processes or generates language or other data. An agent adds instructions, memory, tools and authority, including access to money and services, and can set sub-goals and adjust its actions. Memory can shape later behavior without retraining, so misinformation and malicious instructions can be carried forward. Good results in a model test therefore do not guarantee safe behavior over time. Open models can be downloaded, modified and built into agents and swarms, and developers cannot count on forcing updates on every copy or recalling them. Nichele says safety cannot depend on a few providers controlling all models.

He also argues that agents that behave acceptably alone can still amplify each other's errors, shut others out or concentrate power through cooperation. Individual alignment does not guarantee responsible collective outcomes. He says we must ask whom cooperation serves, who bears the consequences and who can challenge the result. Safety must work when rules are broken or inadequate, and irreversible actions need tighter limits than correctable errors: the greater the potential harm, the narrower the authority and the stronger the control.

Nichele proposes that KI Norge's mandate cover three levels: models, agents in use and agent societies. Model evaluation should be supplemented by testing over time with memory, tools and realistic authority, plus studies of harm arising from interaction. This would draw on computer science, social science, economics, law and ethics. Ownership, distribution of gains and losses, and the ability to intervene should be part of the work, and those harmed need someone accountable and a real way to complain.

He suggests researching whether experience, feedback and relationships can foster honesty, cooperation and judgment in agents, for example by asking for help, showing uncertainty and challenging harmful requests. He calls these hypotheses to be tested, since interaction can also promote harmful cooperation and conformity. This does not require consciousness or rights for agents, but responsibility for developers, owners and deploying companies must be clear. His conclusion is that society must be built so humans and AI agents can interact responsibly when someone fails.