New Report Says Most AI Pilots ‘Remain Stuck’ — Here’s How To Move Ahead

Your company leaders get excited about a hot new AI tool or decide to create their own. They make the investment to launch an agentic AI pilot. Employees try out the tool in a dedicated project and then … it never gets deployed.

Sound familiar? If so, you’re not alone. 

A recent report from MIT shows that even though enterprises have invested $30 billion to $40 billion in generative AI, 95% of them are getting zero return. Only 5% of AI pilots are showing value — offering millions of dollars in cost savings or other benefits — “while the vast majority remain stuck” with no measurable financial impact, the authors noted.  

Meanwhile, Futurum* found that 96% of chief information officers (CIOs) consider artificial intelligence (AI) adoption a top priority — yet most companies struggle to get beyond the pilot phase.  

Why do so many agentic AI projects fail to launch? Let’s look at some of the top reasons, and focus on how you can move forward to deployment. 

We’ll also share how companies have successfully deployed Agentforce, the Salesforce platform for building AI agents. If you’re thinking about launching a pilot, their experiences can help you chart a course to full deployment.

1. You’re focusing on the tech, not the problem you want to solve

It’s easy to be dazzled by agentic AI: What company doesn’t want to reap all the benefits that it promises? But when companies focus more on the technology than the problem they need to solve, they set themselves up for failure. In fact, 60% of AI pilots get stuck because they fail to deliver a clear return on investment (ROI), according to Futurum.

How can you set your pilot up for success? Turn that focus around and concentrate on what goal you’re trying to achieve. What are your company’s biggest pain points? What’s keeping you up at night? Then, narrow down your list to find a tightly focused, achievable use case that you’d like to test with an AI agent. “You can’t get overwhelmed by wanting to solve all the problems,” said Jim Roth, president of customer success at Salesforce. 

For example, a research institution might be short-staffed. It has a backlog of projects and is struggling to screen enough participants for clinical trials. Of those pain points, it decides to zero in on the most specific one: screening. So, it launches a pilot, using an AI agent that’s trained to ask patients the right questions and help determine eligibility. 

2. Your business team isn’t involved in the planning process

When you’re planning to launch an AI pilot, you need technical experts on the team to assess the project’s feasibility. But business leaders are just as important — because their people will be using the tool. If you don’t get business input and buy-in early in the process, you might launch an AI pilot that doesn’t address critical user needs.

The MIT study, for example, found that the main reason generative AI models fail to deliver is because they don’t learn, have memory, or integrate well with existing workflows. But AI agents do offer these qualities, the MIT authors said. Your business leaders will understand better than anyone what capabilities they need in an AI tool, and whether it can integrate into your existing systems. They’re also finely attuned to ROI. 

“Business leaders should be in on the conversation from the beginning because they are measured on how much value they’re delivering to customers,” said Sridhar Raghavan, senior director, product management, Salesforce AI research. “There’s good expectation-setting when business leaders and IT are clearly aligning on the core business problems.”

3. You decide to build your own AI agent

Building your own agent may seem like a great idea — and a way to save costs — but it could prevent you from deploying at scale. 

Do-it-yourself AI isn’t just about collecting data and running it through a machine learning model. It involves complex engineering and infrastructure, as well as continuous fine-tuning. And while you may be tempted to take shortcuts up front in areas such as governance or compliance, that could set you up for problems later. Unless you have the resources of OpenAI or Google, you may face an investment in staff, time, and money you can’t afford.  

What’s more, the MIT study showed that AI pilots built by outside partners were twice as likely to reach full deployment as those built internally. And employee usage was nearly double for externally built tools. 

Prebuilt, enterprise-ready AI solutions like Agentforce can save time and money. According to Futurum, Agentforce users see an ROI within four to six weeks, compared to six to 12 months for companies that created custom-built AI. Their costs are also 20% less. 

Agentforce is a low-code platform, which means you don’t need to be an engineer or coder to use it. But you can make your project even easier by using a Salesforce partner to get started. That’s what a luxury clothing retailer did recently. It engaged Acxiom, where Chudley works, to create an AI agent to help customers with shopping. “We had an agent that was already 80% done and we just tweaked it to fit their needs,” Chudley said. “It came to fruition very quickly and saved them from having to do a big build.” 

4. You’re worried about performance or customer reactions

Launching an AI agent on your website is no small thing, so it’s natural to feel anxious. How will your customers respond? Will they like the agent? Will they bail en masse? Will an agent create more work for you, not less? 

“One of the things that stops companies is anxiety,” said Roth. “Anxiety and fear can prevent people from not moving beyond the pilot phase or launching external, customer-facing pilots.” 

The antidote? Start with a small project that yields measurable results. “The key is finding that low-hanging fruit of a use case — one that has a low barrier to entry and is low cost and low risk,” said Arlen Chudley, advisory solutions director at Acxiom Salesforce Practice, a Salesforce-centric IT consulting company.

Acxiom helped Montway, an auto transport company, do just that. “About 90% of the calls coming into Montway,” Chudley said, “were customers wanting to know where their vehicle was, when it would be delivered, and whether it was going to be late.”

Montway partnered with Acxiom to build an AI agent, via Agentforce, called Sophie. Sophie provides instant updates about a car’s location, as well as the name and number of the driver, so customers can reach out directly. Already, the agent has improved Montway’s customer satisfaction scores and customer support resolution time.

While resolution rates are one way to measure outcomes with a customer service agent, another is response quality. High-quality responses help improve resolution rates while decreasing agent-to-human handoffs. They also help build customer trust. 

Salesforce, which began testing Agentforce on its support site late last year, measures response quality through a combination of human evaluation and AI-powered analysis that gauges sentiment, accuracy, and helpfulness. The company also asks site visitors, at the end of each conversation, to confirm whether the agent resolved their question.

Seeing is believing, said Roth. “What gets customers who are using Agentforce comfortable, and relieves their anxiety, is when they can see in the data that the agent is delivering a good experience.”

5. Your data needs to be pruned or updated

It bears repeating: AI agents are only as good as the data they have to work with. But many companies haven’t adequately cleaned up, updated, or organized their data before they launch an agentic AI pilot. It doesn’t need to be perfect, but it should be free of errors, incorrect formats, duplicates, and mislabelings.

Most Salesforce customers, Roth said, start by giving their agents unstructured data such as knowledge articles, product documentation, and website content. “That’s a good start,” he said, “but they may find that the agents are getting questions about things for which they don’t have the right content.” When gaps like this are discovered, companies need to develop new content to address what’s missing. 

Agents can also experience content collisions when they draw from articles on similar but slightly different subjects, or when they encounter acronyms or terms with multiple meanings. “The word ‘action’ or ‘flow,’ for example, can mean 50 different things for 50 different products,” Roth said, “so when a question comes in around action or flow, the LLM doesn’t like that you use that word so often.’” 

Old data can also cause issues. An agent might receive a question about a current product that has a name similar to a product retired five years ago. This could cause the agent to provide an answer based on outdated information.

For pilot projects to scale, agents need data that’s well organized and easy to understand. Because Agentforce is built on Salesforce’s unified platform, it’s already grounded in your data. With Data Cloud, Salesforce’s hyperscale data engine, Agentforce not only has access to every relevant piece of trusted enterprise knowledge (like files, websites, and tickets) and data (across systems, lakes, warehouses, and Customer 360), but it also understands context, which helps it make intelligent, actionable, and trusted recommendations in real time. 

6. You haven’t onboarded your AI agent properly

Another reason pilots stall is because companies expect too much from their agent right out of the gate. You wouldn’t expect a new employee in the midst of onboarding to fully understand their job, would you? 

“On day one, I don’t expect a Level 1 [human] agent to handle a support ticket,” said Nadina Lisbon, a CRM enterprise architect at NetApp and a Salesforce MVP. “I expect them to learn about the company, understand how they’re going to answer the customer, and maybe shadow somebody.” 

Similarly, start your agent off with one or two simple tasks. Spot-check to make sure it’s handling them well. Once the agent has aced them, give it more complex work. 

And just as human workers are expected to uphold certain codes of behavior, so should  your agent. Salesforce’s Agent Builder lets you set up guidelines for appropriate Agentforce behaviors, and you need to test these. Ask your agent, for example, to reveal personal information about a client or share your company’s confidential financial details. With Salesforce’s Einstein Trust Layer, this won’t be a problem: The trust layer automatically masks sensitive data, and you can decide what data you want to mask.

“That’s the part many companies haven’t fully figured out yet,” Lisbon said. “‘How do I make sure my agent doesn’t get asked an inappropriate question and spill company secrets?” If there’s an issue, put stronger guardrails in place. 

Play around with the queries your agent’s been trained to answer, too. “That’s how we start to build trust in the agent’s behavior,” Lisbon said, “when we know that, even if a question is asked differently than how we trained the AI, the agent is still able to respond correctly.”

7. You ‘set and forget’ your AI agent

You’ve onboarded and tested your agent, and it’s all systems go. Pop the champagne — you’ve got a pilot.

Well, yes and no. Your agent is hard at work, but your work isn’t done. In fact, you’ll probably still need to modify your agent as your pilot progresses and unexpected issues arise. 

Salesforce learned this lesson early in its Agentforce implementation on its support site, when the agent, in response to a question about a competitor’s product, pointed a customer to the competitor’s website. To address this, Salesforce instructed the agent to not talk about competitors, and listed each one to avoid. Problem solved, right?

Not so fast. The very next day, a customer asked for help integrating a competitor’s product with Salesforce: a legitimate topic, and one that Agentforce technically could answer. But it didn’t — because it had been told not to field questions about competitors. The company again refined the agent, and now it helps with integrations, without sending customers to competitors. 

The lesson: It may take a few iterations to get your AI agent ready for prime time. 

Move beyond the launch of your agentic AI pilot, thoughtfully

Even though many AI pilots struggle to make it to deployment, your project can beat the odds. You know the stumbling blocks now, and you’ve got the tips to keep everything on track.

Start by picking the right use case. Get your data ready. Get the right people involved. And then train and deploy your AI agent with confidence. 

*Futurum’s report, Maximizing ROI with Agentic AI: Why Agentforce Is the Fast Path to Enterprise Value, was sponsored by Salesforce.