When AI Agents Join the Customer Support Team: Lessons From a Successful In-House Test Case

When AI agents join a workforce, they bring a host of new abilities. They’re fluent in multiple languages. They can handle thousands of conversations simultaneously without missing a beat. And they’re available 24 hours a day, seven days a week. 

Yet despite these admirable traits, AI agents, like any early-career employee, still make the occasional misstep — giving the wrong answers or striking the wrong tone. Fortunately, like most employees, they can be taught.

Salesforce learned this firsthand as “customer zero” for Agentforce, the company’s digital labor platform. Starting in fall 2024, select Salesforce customers logging on to help.salesforce.com to ask a question or troubleshoot a problem had the option of working with a digital help agent built on the Agentforce platform.  

Ten months after its debut, Agentforce is handling 45,000 conversations per week — more than 1 million in all by early July — with resolution rates in the 85% range. The numbers continue to rise week by week. 

True, that remains only a small fraction of the 3 to 4 million visits to the Saleforce Help site every month. And yet, by resolving transactional queries, the AI agents are ensuring that humans can focus on the more relationship-driven, nuanced work that people are best at.

That dynamic extends beyond the support team. “My need for headcount has not diminished as the result of AI,” said Zachary Stauber, Senior Director of Agentforce Data & AI within the company’s Digital Success group, who was a key member of the launch team. 

“What has significantly changed, though, is I no longer need to hire people to handle menial tasks” and can instead hire people to focus on work more valuable to the overall business. 

Critically, Salesforce’s agentic customer service implementation is answering key questions that all types of businesses and organizations may have in these early days of the agentic AI era. What does it take to successfully onboard an AI agent? How do you coach a digital co-worker through its mistakes? And what skills do managers need in a truly hybrid workforce, where people and AI work side by side? 

The key to good management, Salesforce learned, is understanding how to bring out AI agents’ best‌ — ‌and use them to better serve customers. As AI agents grow more capable, success will hinge on how well organizations collaborate in a transformation that’s not just technical but also requires a shift in thinking and in culture.

The art of training digital colleagues

After months of planning that began in mid-2024, the Salesforce Customer Support team had mapped out a carefully staged, four-week rollout of its new AI agent‌ — ‌baby steps with limited exposure and extensive monitoring. 

“In our first week, we limited our rollout to only 10% of authenticated users, which gave us fewer than 150 conversations to sift through manually,” said Stauber, who had been part of a small launch team inside the company’s Customer Success group. “We were really nervous. All of us were like, ‘How is this going to work?’” 

These early conversations made clear that coaching ‌AI agents would require human oversight — ‌the digital equivalent of on-the-job learning. 

When customers chose the agent, most of the time Agentforce met their needs. But not always. When customers mentioned needing help with their “live chat” or a product name like “Digital Engagement,” Agentforce grew confused. Rather than providing guidance on those actual services, the AI agent thought users were demanding to speak with a human “support engineer” (Salesforce’s term for its thousands-strong corps of customer service experts).

Another time, Agentforce recommended a competitor’s product to a Salesforce customer. Not exactly the response you’d hope for from your support team. The solution seemed simple: Add instructions that stopped the AI agent from doing that. But then it overcorrected by refusing to answer a customer’s questions about integrating Microsoft Teams because it had been told not to discuss competitors.

Another early glitch: The team had instructed Agentforce to respond too restrictively, blocking even legitimate questions. So the team shifted to a “coaching” rather than “prescribing” methodology. Instead of saying “don’t mention competitors,” for example, they instructed the agents to “act in the best interest of Salesforce.” It was as simple as deciding to “let the LLM be an LLM” rather than treating it like a traditional chatbot with binary rules. 

Those hiccups seemed to confirm the wisdom of the launch team’s decision to take a cautious approach in the early going. “We wanted to have extreme oversight over the experience,” said Emily Winslow, Director of Agentforce Data & AI. 

But by the time CEO Marc Benioff checked in on the effort’s progress a few weeks after the launch, mistakes were rare enough that he encouraged the team to scale more quickly. 

“Marc’s feedback was very clear: You guys are being too conservative,” recalled Bernard Slowey, SVP of Digital Success at Salesforce. “He was like, ‘This technology is here, so let’s get it out there and then let’s learn from it.’” 

With that mandate, the traditional approach of extensive testing and gradual rollouts gave way to rapid iteration and real-time learning. “Our roadmap is in days and weeks, not quarters,” Slowey said.

There were also pleasant surprises in those first months. “We were ready to go in and start debugging, but that wasn’t necessary — which was kind of cool,” Stauber said. It wasn’t bugs that needed fixing. Instead, the work involved continued training of the agent, including fine-tuning agent instructions after observing real customer interactions.

Stauber and his co-workers were also relieved to learn how much Data Cloud‌ — ‌Salesforce’s solution for unifying disparate data across different sources‌ so agents can use it — ‌simplified deployment. The team “hydrated” content from the developer portal — brought it in to Data Cloud — to address some of the gaps in the agent’s responses. “It really changed how we think about building AI,” he said. “We’re so used to stitching together APIs and data sources. But suddenly everything we needed was just there.”

Even so, early on, the AI agent wasn’t able to respond when developers asked more technical questions. But that proved a shortcoming in the instructions rather than a limitation of the machine; no one had yet fed the agent the more technical documentation. 

“It’s no different in our heads than training one of our regular support engineers,” Stauber said. “If you get a human on the phone and they don’t have that piece of information, they can’t answer the question.” 

Yet the solution wasn’t simply giving Agentforce access to every last technical document. The team learned this lesson when a customer asked about a recent software update — and Agentforce referenced outdated release notes from 2018. 

Slowey said these issues were likely due to “content collisions.”

“When there’s too many articles about the same thing, you have poor content hygiene,” he said. “Ultimately, we had to use agents to cull through content and clean it up, and if a piece of content hadn’t been used in a year, we deleted it.”

“There’s a common misconception that with AI, more content equals better answers,” Stauber said. “The reality is that more curated content equals better answers.”

Shaping agentic personalities

Winslow, who spent hours each day reading Agentforce transcripts in those early weeks, was struck by one unexpected pattern: the depth of the conversations she was reading. “We saw much longer conversations than we anticipated — eight turns, 10 turns, with follow-up questions.” The extended dialogues, she said, were “an early indication that this is something completely new.”

The team also noticed that customers evidently felt more comfortable being vulnerable with an AI agent. 

“Customers are more willing to ask Agentforce questions that they might hesitate to ask a human support engineer, likely out of fear of judgment or embarrassment,” Slowey said. “It feels less intimidating to most people.” 

Customers are more willing to ask Agentforce questions that they might hesitate to ask a human support engineer, likely out of fear of judgment or embarrassment. It feels less intimidating to most people.

Bernard Slowey, SVP of Digital Success at Salesforce

Yet, in the early weeks, something was clearly missing from Agentforce’s responses. “There was a very specific way that a human support engineer would display empathy for a customer’s issues and reassure them,’’ Winslow said, “and Agentforce did none of that.” The AI was mimicking the clinical tone of technical documentation, not the conversational human voice Salesforce wanted to project. This was unacceptable for a company that prides itself on its customer relationships.

Still, even more work was needed to make Agentforce sound more natural and empathetic. And so, as Step 2, this past spring, the team hired seasoned conversational designer Hilary Hayes‌ — ‌whose background includes nearly a decade in user experience (UX) and industrial design‌ — ‌to help bring what she calls “more heart” to Agentforce. 

“It was already doing a great job with left-brain tasks — retrieving facts, making connections, surfacing detail,” Hayes said. “What I’m focused on is [what in humans we call] right brain: voice, tone, empathy. How does it greet users? Does it have a sense of humor?” 

Her expertise is becoming even more essential as the Customer Support team works to give Agentforce voice capabilities‌ — ‌the ability to listen and speak‌ — ‌in the coming months.

“Part of my role may be shaping the actual voice — synthetic or otherwise — that people hear when they interact with Agentforce,” Hayes said. Her role is another reminder that deploying AI agents isn’t just about technology. It’s also about creating digital personalities that align with a company’s values and customer service standards.

The human factor

As it turned out, the most delicate part of bringing Agentforce to the support team wasn’t technical — ‌it was human. Agentic AI’s arrival stirred up anxieties among Salesforce support engineers who, until then, had been the main point of contact for customers seeking help. If ‌AI could now answer customer questions, what did that mean for their job security?

In short order, they found that Agentforce worked best as a collaborator, not a replacement. Agentforce proved remarkably fast at certain tasks, such as retrieval and summarization. It could scan multiple articles and synthesize steps back to customers in seconds. But humans remained essential for solving problems that had no simple, off-the-shelf solutions. People draw on experiential knowledge that far exceeds what any AI model can access today. 

“From the start, we made sure that the support engineers felt very connected to this,” said Slowey. “So we’ve included them in planning and improving the experience.” He said the support engineers are integral to training Agentforce, analyzing poor responses and suggesting improvements. 

The company also framed the change as a career opportunity rather than an existential threat. “We’ve positioned Agentforce as another operational efficiency,” said Mahua Choudhury-Hironaga, VP of Assisted Customer Experience, who oversees the support engineers handling online chat. “People can focus on more complex questions and tasks they didn’t have time for before.”

The learnings are also making an impact on how managers think about hiring. 

The focus, Stauber said, has shifted to hiring people who could “actually do something with that information or evaluate it in a way that is aligned specifically to whatever outcomes or goals we’re trying to drive.”

AI and humans working in tandem

These days, customers logging on to Salesforce Help have the ready option to engage with Agentforce. But it’s always a choice, never a requirement. “The moment a customer indicates they want to chat with a human,” Winslow said, “we get them there as fast as possible.” The system recognizes multiple signals that a customer wants human help‌ — ‌from explicit requests to speak to a human to more subtle cues, including profanity or other expressions of frustration. The AI agent is programmed to hand off conversations smoothly and quickly, maintaining context so customers don’t have to repeat their issues to the live support engineer.

Importantly, inside Salesforce, the escalation to a human isn’t considered a failure‌ — ‌often, it’s exactly the right outcome when customers need nuanced problem-solving or simply prefer human interaction. The team views these handoffs as part of a well-functioning system, not something to avoid or minimize. 

To track progress, the team has developed comprehensive scorecards that track resolution rates, escalation levels, response times, and conversation volume. “You need to manage that similar to the way that you’d manage support engineers,” Slowey said.

Proactive agents 

Looking forward, Salesforce envisions a more proactive AI workforce. Current plans for Agentforce include not just voice capabilities but also personalized support based on customer data so they can take actions on behalf of users rather than just providing instructions. Instead of telling a customer the steps to fix an error, for instance, the agent could make the correction itself.

“We want to be a lot more proactive,” Slowey said. He imagines AI agents that check in with customers, recommend learning paths, and even create reminders about uncompleted training. 

We want to be a lot more proactive, Slowey said. He imagines AI agents that check in with customers, recommend learning paths, and even create reminders about uncompleted training. 

In this new world, it’s no longer a question of whether AI will transform customer support — ‌it’s already doing so. The challenge for organizations is how quickly they can learn to manage digital colleagues, coach AI teammates, and strike the right balance between automation and human expertise. For those willing to learn by doing, the rewards are substantial: a workforce that never sleeps and that can scale infinitely.

Perhaps the biggest lesson has been embracing imperfection as a starting point. Traditional enterprise rollouts favor careful testing and gradual deployment. But AI agents learn differently‌. True, they needed to have well-defined guardrails so they stick to the intended tasks, which is a key part of Agentforce development for any use of agentic AI. But within those boundaries, there needs to be room for trial, error, and learning by doing.

The bottom line, Slowey said, is getting Agentforce in the hands of customers to learn and improve. “It’s not always perfect,’’ he said. “But neither are humans.”

Go deeper:

• Read a firsthand account of working with AI and the team’s top five takeaways from deploying Agentforce on Help
• Check out an early account of the deployment
• Find out Salesforce’s three tips to set up AI agents