Imagine a field service technician repairing a dishwasher. Their hands are full, the next appointment is approaching, and a missed step could mean another visit. An AI assistant watching the repair could flag a skipped step while there is still time to act. But an assistant that repeats instructions or interrupts every routine action could make the job harder and risk frontline teams not using it.
BLIP-Live tackles getting the balance of timely help and restraint just right. BLIP-Live is a general model that can support any workflow where a live agent helps people in the moment, across consumer, enterprise, and industry settings.This research explores how proactive video agents can watch an unfolding task, step in at critical moments, and stay quiet during routine work, moving businesses closer to AI assistance that feels like a helpful teammate rather than a distraction.
Why Proactive AI Needs Restraint in the Real World
The potential for proactive assistance extends from workplace training and equipment maintenance to everyday tasks such as setting up an appliance or following a recipe. Guidance becomes more accessible when people can keep their attention on the task instead of stopping to search for instructions. This is a core pillar of Salesforce AI Research’s Ambient Intelligence vision, bringing context-aware help directly into ongoing work—powered by live video agents like BLIP-Live that spot issues proactively, and Interactive Visual Gym, which tests and refines those models in simulation before they reach frontline work.
Video agents today combine live camera and voice interaction with knowledge grounded in product data to help humans through complex procedures. Understanding the product and the current scene provides the foundation for relevant guidance. BLIP-Live addresses a complementary research question: when should a visual assistant speak? Real proactiveness demands restraint: to be genuinely useful in the field, an assistant has to weigh whether its next message is worth interrupting someone’s focus.
Consider a possible field service workflow. A technician is replacing a component and moves toward the next step before completing a required check. A visual assistant could offer a short reminder, then remain quiet as the technician completes the check. Repeating the same reminder throughout the repair would consume attention without adding value.

Teaching AI when silence helps
Recognizing what is happening in a video is only part of the problem. An assistant also needs to distinguish a new opportunity to help from an activity it has already addressed.
BLIP-Live explores building large scale training data that includes both useful response moments and silence periods when the agent should continue observing quietly. This includes what happens after a response: if nothing relevant has changed, the assistant should learn that another message may be unnecessary.
We also investigate how Agent Memory is important to help reduce repetition. The broader lesson is straightforward: an assistant needs to learn how to participate in an ongoing activity, including when to wait. BLIP-Live leverages these learning techniques to build multiple state-of-the-art video agents for exhibiting strong ambient intelligence.

Measuring useful help, including its cost
An evaluation that rewards only catching events can favor an assistant that speaks constantly. That creates a gap between a good score and a good experience.
Our evaluation combines timely coverage with the share of responses that receive credit for addressing an annotated opportunity. Repeated and out-of-window messages count toward the total responses, so unnecessary speech carries a cost. An assistant that never speaks receives no credit either. The goal is to balance timely help with restraint.
In one evaluation across 326 videos filmed from a person’s visual perspective (ego-centric), our BLIP-Live helped reduce unnecessary responses of state-of-the-art models by over 50%. Experiments also indicated clear human preference for our BLIP-Live responses.

What this means for business adoption
For business leaders the need is clear: help employees resolve issues on the first visit, reduce avoidable rework, and make expertise available during critical moments without distraction. BLIP-Live solves for these needs by ensuring the decision of when to speak is part of the intelligence itself—delivering accurate, selective guidance at the exact right moment, especially for people navigating unfamiliar procedures. Next the AI research team behind this technology will be validating the right balance in live workflows, measuring both task outcomes and the experience of the people receiving help.
To learn more, checkout our project page https://bliplive.salesforceresearch.ai.













