A subtler form of model error
Most people already know that generative AI can hallucinate. The model may invent a date, source, or event and present it as fact. Such errors are serious, but often detectable: we can compare the response with other sources.
A more difficult case begins when AI makes claim not only about the world but about you. During a conversation, an agent based on an LLM may conclude that you are frustrated, confused, highly engaged, tired, or losing concentration. It might even infer your current psychological or psychophysical state.
How can such an assessment be checked? There is no external database to consult. The evaluation concerns a fleeting state, which can sometimes be difficult even for the person involved to name precisely. Yet, the system may base decisions on it — how to speak to you or what action to take. An AI agent serving a customer may have functions designed to respond to dissatisfaction or uncertainty.
In the case of evaluating human behavior, we are not dealing with classic hallucinations of false information, although the end result is similar. The model generates a description of the person with confidence, but without a sufficiently stable basis. Because the description may sound psychologically convincing and cannot be easily compared with other sources, this error is more difficult to detect.
If the baseline signal is unstable, the same person might be treated differently for reasons unrelated to their actual behavior. Since such inferences are usually invisible, users may have no way to challenge them.
What Orange researchers tested
Researchers from Orange measured how consistently agents based on LLMs make such assessments about people. Instead of asking whether a particular assessment merely seems credible, they repeated the same analyses under identical conditions using the same recordings. They checked whether an agent that appears to know a lot about its conversation partner reliably creates the same picture of the person when the same material is analysed again.
And the consequences? Imagine a customer service agentic system that considers the interlocutor very frustrated, then shortens its responses, limits information, and switches to a de-escalation mode. When the same recording is analyzed again, the system describes the customer as only slightly irritated or even neutral. Nothing about the person has changed. Only the interpretation made by the model has changed. How do we know which response was justified?
Two moving targets: the human and the model
Human behavior naturally changes. The same person may speak differently when tired, in a hurry, or anxious. Traditional psychological measurement has long dealt with this difficulty by checking whether questionnaires or scales are reliable and valid.
Generative AI adds a second source of variability: the model itself. The same input material can lead to different assessments due to probabilistic generation, prompt formulation, context, configuration, or subsequent changes in the model. Human variability is part of the phenomenon being studied. Uncontrolled variability of the model becomes a measurement problem.
When the inferred user state is fed back into an adaptive system, the risk increases. The assessment influences the system’s behavior; this behavior affects the user; the user’s reaction becomes a new source of evaluation. If an error occurs, the system adapts not to the person but to the artifact of its own previous measurement.
A false fact can be checked against another source.A false judgment about your emotions may quietly determine how AI treats you.
Good performance is not the same as reliable measurement
AI systems are often evaluated by asking whether they can correctly classify examples or improve results such as satisfaction, conversion, or task effectiveness. Such tests are important but do not answer the question: if the system re-analyzes the same person under the same conditions, will it reach the same conclusion?
A tool may perform well on average but be very unstable in individual cases. Conversely, the result may be perfectly repeatable but still wrong — like a broken thermometer that always shows the same incorrect temperature. Reliability and validity are related but not the same.
What repeated tests revealed
An experiment described by Krzemińska, Butkiewicz, and Komkowska treated large language models as measurement instruments [3]. Fifteen recordings from a contact center were divided into 552 segments and analyzed repeatedly under identical conditions using three multimodal models. The study involved 213 used or proposed for adaptive systems, covering emotional consistency, engagement, cognitive fit, interaction effectiveness, and safety.
The goal was not to rank models but to assess whether signals are stable enough to support agent decisions about a specific person.
Only 31 of 213 metrics (14.6%), met the rigorous “excellence” threshold of at least 90% stability for a single inference across all three models. After aggregating results at the conversation level, 89 metrics became sufficiently stable for analytical use. However, only 19 of these 89 provided nearly identical assessments across each pair of models.
Practical conclusion
The statement that “AI can recognize user states” is overly optimistic. Some metrics may be reliable within a specific model and context. Others may only be useful after aggregation, for example, for NPS analysis. Still others should not be used as sources of user information at all. Moreover, metrics are not universal. Stability does not automatically transfer from one model to another. The same metric may be stable in one model and context but unstable in another.
Cyberpsychometrics as a quality control layer for AI assessments of humans
is a proposed field of research aimed at testing the quality of user descriptions generated by AI. It draws from psychometrics, the science of psychological measurement, but applies its principles to probabilistic models that can change over time.
It is connected to existing work in the fields of algorithmic profiling, behavioral inference, computational emotion research and the sociology of quantification – all of which study how people become classified, scored or acted upon through data [6][7]. Cyberpsychometrics adds a narrower measurement question: when an AI system generates a user-state metric, is that metric stable, repeatable and meaningful enough for the way it is used?
Its questions are simple: does the model produce the same assessment under the same conditions? Do different models agree? Does the assessment correspond to what the user reports or what trained observers see? Is a given metric reliable enough for real-time decisions or only for statistical summaries?
Unlike a paper questionnaire validated once before use, AI-based measurement may require continuous monitoring because the tool is a model that can change over time. Updating the model, new prompts, different languages, or data environment changes can alter its behavior without changing the application itself.
Explainability is necessary but not sufficient
Explainable AI can tell why a decision was made – for example, “I shortened the response because I detected frustration”. Such an explanation describes the mechanism but does not prove that frustration was measured reliably.
An explanation based on an unstable signal can be logically consistent yet not justify the decision. Cyberpsychometrics emphasizes that before explaining, we must first verify the quality of the signal: how consistently does the system measure the person it claims to understand before explaining any AI decision.
Why this matters for ordinary users
Private interpretation by an AI system can influence how a person is treated. A customer deemed angry might be routed differently. A student considered confused might receive simpler material. An unengaged user could be subjected to stronger persuasion techniques. A person classified as high risk might face additional checks or fewer options.
Users may know, or be informed, that they are interacting with an AI system. In some cases, such transparency is required by regulation [5]. Yet this does not mean that they know when a specific inference about them has been made, what state has been assigned, which threshold has been crossed, or what operational consequence follows. Full real-time disclosure of every intermediate judgment could also disrupt the natural flow of conversation. Transparency as a principle therefore needs an operational complement: quality control of the hidden signals used by the system.
If the baseline signal is reliable, such adjustments can be helpful.
If the baseline signal is unstable, the same person might be treated differently for reasons unrelated to their actual behavior. Since such inferences are usually not visible at the level of each conversational turn, users may have no practical way to challenge them.
From clever demonstrations to responsible adaptive systems
As a first step, a responsible system should distinguish at least three classes of metrics: those sufficiently credible for real-time operation; those useful only after aggregation for analysis; and those not useful at all. This classification should be based on empirical data, not assumptions from model descriptions or an apparent success of a pilot. A pilot may show that adaptive system can work in specific scenario. It does not show whether the signals used for adaptation are reliable enough to be used more broadly.
The next step involves a control layer that can be reactivated, monitor stability over time, detect drift, and warn when a metric should be limited or withdrawn. Organizations deploying AI should set thresholds and decide what actions are permissible at each level of reliability.
The goal is not to prevent AI from adapting to humans but to ensure that adaptation is based on a reliable signal, not fluctuations or fabrications. Before asking how well AI personalizes interactions, we should first ask how consistently it measures the person it claims to understand.
Why Orange Research is working on this
Orange Research is working on responsible AI which begins at the design stage, before a system is deployed and before its effects become visible. The first question for any AI-based service is not only whether the technology works, but who may be affected by it. Who is represented, classified, routed, persuaded, deprioritized or given fewer options? Who may bear the consequences if the system’s interpretation is wrong?
This question is especially important for conversational agents and adaptive systems, because they do not merely answer users. They may also build a working picture of them: their frustration, engagement, attention, urgency, confidence or needs. If this picture is used to change the course of an interaction, then its quality becomes part of the organization’s responsibility. A system designer must therefore ask: what user states are being inferred, what decisions depend on them, how reliable are these signals, and what harm could follow if they are unstable?
Cyberpsychometrics helps make this responsibility operational. It provides a way to distinguish signals that are reliable enough to guide real-time adaptation from those that should be limited to aggregated analysis – or not used at all. The aim is not to discourage adaptive AI, but to ensure that adaptation is based on validated signals rather than untested assumptions about the user.
For Orange Research, this work is also part of a broader Responsible AI approach: investing in research on the effects of technology, identifying potential harms early, and developing tools that help make responsibility operational rather than declarative. This orientation is consistent with Orange’s role as an ambassador of Responsible AI and with the recognition received through the double SRAI Gamechanger award from GPAI.
The next challenge: when AI agents represent people to other AI agents
The problem will become even more complex when AI agents start interacting with other AI agents acting on behalf of users. A personal AI assistant may negotiate, complain, ask for support or make decisions in a person’s name. In such a scenario, the company agent may no longer interpret the human user directly. It may interpret another AI system that represents that user.
This matters because the decision still concerns a human being, even if the human is no longer the immediate participant in the interaction. The chain of inference becomes longer. First, the personal agent builds a working picture of the person it represents: their preferences, goals, urgency, risk tolerance or emotional state. Second, it translates this picture into a particular way of acting, speaking, prioritising or negotiating. Third, the company agent interprets this behaviour and infers the user’s intentions, needs or willingness to accept an offer from the behaviour of the representative agent.
Each step may introduce error. The first agent may build an unstable or incomplete picture of the user. It may then express this picture in a way that distorts the user’s actual preferences or state. The second agent may then misread this expression, especially if its own model, metrics or categories are not compatible with those used by the first agent. In other words, errors may no longer arise from a single inference about a person, but from a sequence of AI-generated representations and reinterpretations.
This issue is particularly important because research on user-state metrics shows that model agreement can be weaker than within-model stability. A metric may be relatively stable inside one model and still not produce the same interpretation in another. In agent-to-agent interaction, this lack of inter-model compatibility becomes a practical risk: one system builds a representation of the user, another system reads it differently, and the consequences are assigned back to the human.
This does not remove the need for cyberpsychometrics. It extends it. Future adaptive systems will need methods for validating not only AI judgments about people, but also the transformations through which people are represented by one AI system and interpreted by another. The key question will no longer be only whether an AI system can infer a user state reliably. It will also be whether this inferred state remains stable, meaningful and accountable when it is passed through another agentic system.
Operational metrics: sufficiently reliable to influence a real-time interaction or automated decision.
Analytical metrics: useful only after aggregation, for example in post hoc analysis or statistical modelling.
Reserve metrics: not sufficiently stable for operational or analytical use and therefore withheld until conditions improve.
Sources :
[1] Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.
[2] Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
[3] Krzeminska, I., Butkiewicz, M., & Komkowska, E. (2026). Can We Trust AI-Inferred User States. A Psychometric Framework for Validating the Reliability of Users States Classification by LLMs in Operational Environments. arXiv:2605.15734. https://arxiv.org/abs/2605.15734
[4] Rai, A. (2020). Explainable AI: From black box to glass box. Journal of the Academy of Marketing Science. https://doi.org/10.1007/s11747-020-00754-8
[5] European Commission. Guidelines on transparency obligations for providers and deployers of AI systems under Article 50 of the AI Act.
[6] Espeland, W. N., & Stevens, M. L. (2008). A sociology of quantification. European Journal of Sociology, 49(3), 401–436.
[7] Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98–125.
