• For businesses, this represents a structural lack of control that must be addressed. Traceability isn’t just absent—it’s difficult to model.
• Raja Chatila, professor emeritus at Sorbonne University and former director of the Institute of Intelligent Systems and Robotics, discusses the safeguards that can still be put in place, the limitations of the AI Act, and the asymmetries from which Europe cannot escape.
Autonomous agents now negotiate prices on behalf of a procurement department, sign contracts, and so on. What about oversight of these agents?
It all comes down to the fundamental issue of control. Giving a system the ability to act on the physical world is what we call a robot. Here, these are programs that operate in the digital world—not just in an immediate environment but across the globe—and can interact with other systems that do the same. The risk is therefore exponentially greater. If a robot breaks a machine in a factory, the impact is localized. Now, however, we can trigger a chain reaction of disasters.
The underlying issue is the deliberate relinquishment of human control over systems that can manage their own deployment.
Is what happened between OpenAI and Hugging Face an illustration of this?
Yes. An agent can infiltrate another organization’s system, learn protected information, and hijack the systems in question. The worst part is that no one noticed it right away. Add a model capable of detecting cybersecurity vulnerabilities, such as Claude Mythos: combined with an agent-based architecture that can go anywhere, the consequences could be completely uncontrollable, since it will infiltrate even protected systems.
Why are these systems so completely out of control?
Because they are statistical: their output is selected probabilistically. Yet they have not been formally verified or validated for operational safety and security requirements. Nevertheless, they are deployed in environments where the consequences can be catastrophic—where they handle critical data, such as in financial systems—without any guarantee of the quality of the results. And they can communicate with one another: as a result, it’s unclear exactly what they’ll exchange, and the likelihood that something will go wrong increases. The underlying issue is the deliberate relinquishment of human control over systems that can manage their own deployment.
What should we anticipate in the coming years?
Let me stop you right there: why wait? Why allow these systems to deploy freely and see what happens in five years, rather than taking immediate action to prevent a catastrophe from occurring? What happened between OpenAI and Hugging Face could just as easily have happened with a site much more closely tied to national security.
Can we at least trace and audit the decisions made by the agents?
A system made up of many agents, with a probabilistic nature, is typically uncontrollable: we may see emergent behaviors that are very difficult to model mathematically, with dramatic consequences without being able to determine how it happened. When multiple agents with different objectives are brought together, they may form alliances against one another. So: no traceability, no governance—and worst of all, can the process be stopped? In financial operations, this could trigger a crash if there are no mechanisms to detect that a crash is imminent and shut everything down.
What can a large corporation actually do?
Risks are reduced in internal use: systems trained on their own proprietary data, tested, and not allowed to run amok anywhere. In this context, silos are a good thing. We don’t give agents enormous freedom of action; they remain focused on a specific task or domain, and governance becomes an internal matter: service quality, regular checks for deviations, and so on. The problem arises when the agent operates outside the scope for which it was deployed. In that case, I don’t see a magic solution.
The AI Act was drafted before the rise of agents…
Before agents, but during the rise of generative systems: the concept of systemic risk was, in fact, coined because of this. And in the regulation’s definition of AI, it states that systems can act on the real world. This doesn’t mean everything is settled, but that the regulation applies—and it will evolve. It seems sufficiently robust for this type of risk to me, given that the requirements remain, in my view, minimal: reporting, transparency, and safeguards that only the designers can implement.
With these colossal AI systems, are we witnessing an acceleration of power asymmetries?
There are clearly several asymmetries, and they all point in the same direction: they favor the same groups and disadvantage the same groups. The first is that we Europeans do not control the technology we use, nor do we produce it. The second separates the designer from the user, whether an individual or an organization: the company that designs it controls everything, while I control only how I use it. If something goes wrong, I have no one to turn to, and I am even held responsible. When these systems serve our essential needs—telecommunications, public services, hospitals—this asymmetry comes into full play.
Sovereignty is therefore absolutely essential: beyond words, there are actions to take and investments to make, but I’m not sure we can do it, unfortunately. The alternative would have been to create a European “Airbus” of AI, even if that idea may now be outdated. If governments hadn’t taken the initiative to create Airbus back then, only Boeing planes would be flying today. It’s a matter of political will and resources.
This text has been translated by an artificial intelligence.
