AI assistants are rapidly evolving beyond text and image chat boxes into real-time, vision enabled copilots for the physical world. By processing live video streams to guide frontline teams, technicians, and everyday users through multi-step workflows, visual agents have the power to transform productivity. From helping an employee configure a complex device to guiding a manufacturing line worker through a precise operational routine, visual copilots bridge the gap between static instructions and active, hands-on execution. This reflects an ongoing shift from AI systems that are reactive and prompt-induced to systems that are proactive, ambient, and context-aware.
Recent developments from frontier teams including Thinking Machines, Meta, and ModelBest reflect an industry-wide push to build assistants that collaborate with humans in real time. This momentum is also being translated into the enterprise where Agentic Field Service is equipping mobile technicians with real-time guidance to resolve complex jobs faster. Yet for enterprise leaders preparing to deploy vision-language agents into mission critical operations, a fundamental evaluation question remains:
Does an AI copilot’s real time advice actually improve operational task completion?
Traditional benchmarks are unequipped to answer this. Most legacy evaluation frameworks rely on passive, pre-recorded video clips, grading an assistant’s text outputs against a static answer key. In static testing, however, cause and effect are completely decoupled: if an AI issues a critical warning, the person in the pre-recorded video proceeds to make the mistake anyway. Because static benchmarks lack a closed feedback loop, organizations cannot verify whether an assistant’s guidance mitigates real world errors or derails execution.
To solve this blind spot, Salesforce AI Research developed Interactive Visual Gym (IVG)—a simulation framework designed to evaluate and train visual AI assistants in dynamic, closed-loop environments.
Redefining Evaluation Through Dynamic Simulation
Interactive Visual Gym recreates real-world physical workflows, ranging from culinary prep to device system configurations, and pairs them with an autonomous simulated user.
Unlike static video playback, IVG operates on an active feedback loop: the AI assistant analyzes the user’s video feed and provides spoken intervention. The simulated user then alters its actions and decisions in direct response to that advice, altering the state of the digital twin in real time.
Three structural capabilities distinguish IVG from conventional robotic simulators:
- Human-Level Semantic Interaction
Standard robotic simulators mandate micro-level physics controls, such as joint velocities or degree coordinates. IVG operates at the high-level semantic layer where humans naturally communicate—directing an assistant to “move to the prep station with the ingredients” rather than calculating navigation paths. - Programmed Out-of-Plan Disruptions
Real world execution rarely proceeds without friction. The IVG harness can schedule unpredictable operational deviations, such as procedural missteps, mid task plan alterations, clarification questions, or memory check-ins at precise workflow milestones. This stress tests an assistant’s capacity to detect anomalies and course correct a live user under pressure. - Multidimensional User Personas
Frontline workers exhibit varying operational patterns. IVG models diverse workforce archetypes across four behavioral axes:- Domain Expertise: From novices with limited domain knowledge and planning capabilities to experienced users who can anticipate needs and seek minimal intervention.
- AI Trust: Quantifying how quickly a user accepts or questions agent recommendations.
- Execution Velocity: Assessing the preferred pace of task execution, ranging from cautious and deliberate to quick and rapid.
- Interruption Tolerance: The degree of receptiveness to assistant guidance and intervention, ranging from step-by-step support to necessary-only interruptions.
What today’s assistants still find difficult
To establish a rigorous enterprise benchmark, we created IVG-Bench to evaluate 14 leading visual AI assistants across 315 complex scenarios in three operational environments. Even the strongest, Gemini 4 Argon, achieves only 53.4% task success within the time limit, averaged across the three environments, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. Open-weight models—available to download and run locally—generally trail the leading frontier models. We also find that better task success does not necessarily mean better guidance interactions along the way.
Task success and interaction quality measure different aspects of assistance. Points show each model’s mean scores across the three environments.
Guidance can deteriorate as the task continues. For some open-weight models, later responses become less accurate and less relevant to the ongoing activity, revealing a weakness that is difficult to assess through isolated questions about a video.
Interaction quality over task progress for Qwen3.8-27B and Muse Glimmer. Later responses show lower factual grounding, situational relevance, and actionable guidance.
The Human Reality Check
To check whether our evaluation reflects human experience, we invited participants to step into the simulator, complete cooking tasks with the help of AI assistants, and rate their interactions. When real humans tried the assistants, their preferences matched our automated scores nearly 80% of the time, confirming that the benchmark accurately predicts human satisfaction.
How Simulation Accelerates Model Performance
Our simulated environments do more than measure AI performance, they actively improve it. In an early proof of concept, we used the simulator to train an open-source model (Gwen 3.5-4B), incentivizing it to prioritize task completion and error prevention. The results were immediate: the model’s success rate jumped from 33.3% to 42.2% on entirely new tasks it had never seen during training. Beyond the raw numbers, the assistant became noticeably better at delivering the right advice at the right time, stepping in proactively before mistakes happened.
