5 minute read
Most benchmarks for AI assistants measure what the system does: whether it completed the task, how accurately, and how fast. These are the right questions to ask of a tool. I would argue they are incomplete questions to ask of a collaborator.
Consider a real-time voice assistant working with a person on a task. The assistant waits until the person has finished speaking, processes the request for a few seconds, and returns a complete and correct answer. By every standard measure, the interaction succeeded. But the person sat idle while waiting, perhaps glanced at their phone, perhaps lost the thread of what they were doing. Repeat this a few dozen times in a session, and the person may be less inclined to use the system again, with none of this showing up in a success rate, an accuracy score, or a latency number.
What is missing is a measure of how well the agent's actions mesh with the human's in time. This is a question the human-robot interaction (HRI) community has spent more than a decade answering, and I would argue that the answers transfer to real-time AI systems more directly than one might expect.
What is fluency?
Guy Hoffman codified a set of objective metrics for collaborative fluency [1]. Human Idle Time (H-IDLE) is the fraction of the task during which the human is waiting. Robot Idle Time (R-IDLE) is the same for the robot. Functional Delay (F-DEL) is the time between the end of one agent's action and the start of the other's, when the second depends on the first. Concurrent Activity (C-ACT) is the fraction of the task during which both are active at once. Alongside these objective measures sits a subjective fluency scale, in which the human rates how smoothly the collaboration felt.
What strikes me about these metrics is that they are defined in terms of temporal activity patterns, not in terms of what the activity is. Nothing in them refers to grasping, handing over, or moving through space. A human and an agent trading turns in a conversation, or working in parallel on a document, produce the same kind of timeline as a human and a robot assembling a part. That is what makes the framework a natural candidate for real-time AI.
Why not just measure latency?
Latency is a property of the agent alone. Fluency is a property of the pair. Two systems with identical response latency can produce very different human idle time, depending on whether the agent acknowledges a request while it works, breaks a long answer into increments, or begins when the human's intent is clear rather than when the human stops speaking. Latency tells you how long the agent took. It does not tell you what the human was doing in the meantime, or whether they knew what to expect.
We saw the difference this makes in our cross-training work [2]. The human and robot switched roles during training, so that each observed how the other preferred the task to be done. The resulting improvement in temporal coordination — 71% more concurrent motion and 41% less human idle time compared to interactive reinforcement learning — was accompanied by higher ratings of trust and of perceived alignment with their preferences. Participants did notice and valued the coordination itself.
Perhaps the most interesting finding in this literature is that anticipation matters beyond efficiency. Hoffman and Breazeal [3] found that a robot acting on a prediction of its partner's next need was perceived as a more fluent teammate, even where the gains in task time were modest. For a voice assistant, responding promptly once intent is clear, or offering the next step as the current one wraps up, is the same phenomenon in a different medium.
What changes when the partner is an AI rather than a robot?
The transfer is not entirely mechanical. A few definitions have to be rethought.
First, activity boundaries. For a robot, activity is motion. For a voice agent, the natural analogue is speech: human activity runs from voice onset to voice offset, and AI activity from the onset of synthesized speech to its end. Text and agentic settings are murkier. Is a developer reading a diff idle? Is the agent idle while it waits for a test suite to finish?
I would argue that in knowledge work the quantity we care about is not idleness but being blocked: the human waiting on the agent with nothing useful to do in the meantime. An agent that keeps the human informed while it works, or hands back something partial to read, is reducing blocked time even when its total response time is unchanged.
Second, concurrency in a single channel. While two people can work simultaneously on separate activities, overlapping speech in a conversation can be disruptive. C-ACT in a voice setting may therefore capture interruptions rather than productive concurrency. In agentic settings, where the AI can work in the background while the human does something else, concurrency regains its original meaning.
Why does this matter?
In the previous post I argued that the objective of a collaborative agent is not only to complete the current task, but to maximize the probability that the human wants to collaborate with it again. Fluency is one of the few process measures that plausibly speaks to that objective. A person who was never left idle, never interrupted at the wrong moment, and never had to wait without knowing why has had a different experience than one who received the same final output on a worse timeline. The output looks identical on a leaderboard. The willingness to come back does not.
The HRI community linked fluency to trust and preference on physical tasks over many years of studies. The deeper question is whether temporal coordination metrics developed for physical collaboration can predict how people experience real-time AI interaction, and ultimately whether they want to collaborate with the system again.
References
[1] G. Hoffman. "Evaluating Fluency in Human-Robot Collaboration." IEEE Transactions on Human-Machine Systems, 2019.
[2] S. Nikolaidis and J. Shah. "Human-Robot Cross-Training: Computational Formulation, Modeling and Evaluation of a Human Team Training Strategy." HRI, 2013, pp. 33–40.
[3] G. Hoffman and C. Breazeal. "Effects of Anticipatory Action on Human-Robot Teamwork: Efficiency, Fluency, and Perception of Team." HRI, 2007.
