Are you looking to understand
how to successfully deploy and maximize
ROI from Agentic AI?
Hi, I'm Nicole Smayling, product
marketing here at Salesforce.
And today I'm talking to Rebecca Wettermann
from a firm that specializes
in research and technology analysis.
Thanks, Nicole. It's great to be here.
Well, we've all heard
about the transformative potential
of AI agents to revolutionize business,
but the journey
to realizing that potential,
it can be costly, complex, and slow.
And today,
we're going to cut through the noise
and focus on how businesses can move
beyond the pilot phase
and accelerate the return on investment
with Agentic AI.
Rebecca and her team have conducted
extensive research Agentic
AI deployments, and we're very fortunate
to have her here today
with us
and she'll share her valuable insights.
Can you tell us a little bit about
your firm, what you do and your role?
So Valoir’s an independent analyst firm
focused on the value of technology.
So a lot of what we do is helping
customers
build the business case
for their technology investments.
We do a lot of independent
research like ROI, case
studies, broader research reports
and surveys to help customers understand
how to measure the value of technology.
And as you can imagine,
in an emerging area like AI,
there's a lot of thought leadership
around helping people understand
how to measure things
that they've never experienced before.
Absolutely. It's new territory we're in.
There was a recent publication from Valoir
all about the key strategies
for how to accelerate your Agentic
And that report was based on in-depth
interviews with Salesforce customers
who've also experimented
with building their own AI solutions.
Can you tell us a little bit more
about that research?
So we reached out to our network
of Salesforce customers, as well as some
that Salesforce introduced us to,
that were early adopters of
AI and Agentforce
to interview them about their experiences.
They represented organizations
of a variety of sizes
across notable geographies and industries.
We're really talking to them about what
they had tried to do with a do it
yourself or DIY approach.
And then what they were able to accomplish
initially with Agentforce.
So can you share with us a little bit
about what were the common challenges
that you heard from organizations
that were starting their journey with Agentic AI.
What we saw was a lot of
initial excitement around generative AI.
I think we were all excited
in the beginning. Right?
And the promise of these agents
organizations started
have a real fear of missing out or FOMO.
However, as they got into actually trying
to build out these agents from scratch
with, you know, the DIY approach,
it presented some real hurdles.
Many projects struggled to move beyond
the pilot stage, even
for companies that had a lot of resource
and AI expertise.
This really led to a shift in mindset
from FOMO to what we termed
fear of messing up, or FOMU, right?
As organizations tried to achieve
acceptable levels of performance
Creating AI that could handle
complex tasks reliably
without human intervention
proved to be a major challenge.
that makes sense. It's all new territory.
And so before we hear about that
FOMU fear of messing up,
could you tell us, you know,
tell our audience a little bit about that?
What were you calling a DIY
do it yourself approach?
What did that mean
when you were doing this research?
So the DIY approach was really about
an organization building
all the necessary components
to build an agent from the ground up.
So, selecting, tuning an LLM,
sometimes even trying to build
and LLM themselves
integrating the various data sources,
building prompts, building those necessary
security features and guardrails,
developing the user interface or
integrating with some existing interface,
and creating those workflows that enabled
the agent to perform its tasks.
Obviously, not all companies were building
from scratch on all of these components,
but they were often leveraging
and cobbling together
kind of different open source tools,
other commercially available
cloud services, in-house and sometimes
external development expertise.
That is significant work.
So why do you think companies
are attempting this DIY approach?
Well, I think it really goes
back to the FOMO?
Everyone on boards
in the CEO's office were asking,
how are we going to differentiate?
How are we going to create
competitive advantage?
It captured everyone's imagination
and folks were getting a lot of pressure
to outpace competitors
and to really deliver on that.
People started doing DIY
because they were getting that pressure
and there just weren't
commercially available platforms yet,
but they were really feeling the heat
to show that they weren't falling behind.
We were all getting told
that everybody else was doing it right?
Some organizations also believe that
they could achieve a competitive advantage
by building their own proprietary
AI agents.
Others wanted to build their own models
and platforms in-house
because they were concerned
about data privacy and leakage.
Finally, to be honest,
there was a lot of hubris.
People thought that because
they could code or understood some basic
algorithms, suddenly
they were prompt engineers and AI experts.
Someone said to me the other day
with generative AI in particular, adoption
pretty quickly outpaced understanding,
So what happened?
How did these DIY pilots,
how did they turn out?
So a few things happened.
First, many organizations
that were attempting DIY deployments
quickly found significant hurdles
that led them
to abandon their in-house efforts.
One of the primary challenges
with achieving an acceptable level
of performance and accuracy,
as our report in our research found,
many of those DIY projects never moved
beyond the pilot phase
because they just couldn't
reach the necessary accuracy.
Even after significant tuning.
They found that on average, DIY efforts
only achieved around a 52% accuracy,
which obviously was unacceptable
for most of them.
This was often due to a lack of sort
of a robust reasoning layer
for complex tasks,
which led to a lot of hallucinations.
Even with a lot of tuning these complex
scenarios to eliminate inaccuracies proved
to be a really big task, the kind of
scenario modeling that could take years for
organizations
with large product catalogs, for example.
The other significant barrier really was
around data integration and management.
That big challenge was building out
and sort of maintaining the AI guardrails
and security.
The development cycles associated
with user interfaces and workflows
was also another big challenge area.
even if they could get something up
and running, which took a lot of work,
they then had to think about the burden
of ongoing maintenance.
Right?
Because you can't just build an agent.
You have to keep managing and tuning all
of those different pieces, particularly
as the LLM vendors kept coming out
with new models and new versions.
Many of the companies we talked with said
they realized through their DIY efforts
that there was just no way to make sure
that their data was really secure.
At the same time as these DIY folks
were realizing that they had maybe
bitten off more than they could chew,
the landscape really started to evolve
with the emergence of platforms
specifically for Agentic AI development, right?
Salesforce announced Agentforce
and other vendors as well, followed suit.
So our research was really looking
at these customers, focusing on
and understanding
the value of this platform based approach,
specifically looking at the experience
of Salesforce customers using Agentforce.
So we found a real acceleration
in time to value for customers
that were building Agentforce
agents, organizations using that platform
that was already built,
tuned and optimized for agent development,
like Agentforce
autonomous agents an average of 16 times
faster than the DIY approach.
We also found a real big increase
in accuracy, with an average
increase of 75% compared to DIY efforts.
So let's dig into that a little bit
more and understand those numbers better.
What key factors contributed
to the significant acceleration
and accuracy
when customers were using Agentforce?
Well, you know, Nicole, we analysts
like to put things into buckets, right?
So we looked at a couple of key areas
where Agentforce
provides a distinct advantage
over a do it yourself strategy.
Okay. We found really 6 areas
that we looked at.
And we talked to customers for each area,
the kind of time and effort
that was involved with model set
up, data and application
integration, prompt engineering,
AI guardrails for security, building out
that user interface and tuning.
So let's start with model setup with DIY
organizations often spent months
evaluating, prototyping
and tuning large language models
or building their own retrieval
augmented generation
RAG databases took a lot of expertise
because Agentforce comes with a pre tuned
LM integrated RAG capabilities.
They were able to largely eliminate
this set up to it.
One customer, for example, told us that
they had developed a beta of Agentforce
within weeks where building it
in their own LLM would have taken months.
another big area that we looked at,
as I mentioned, was data integration,
connecting and preparing data for AI.
Because Agentforce leverages
Salesforce’s Data Cloud, which connects,
unifies and harmonizes that customer
data customers that already had their CRM
data in Salesforce
were able to see quick benefits.
The pre-built RAG capabilities in Data
Cloud, which helped them bring in all that
unstructured data, also really accelerated
that data integration time.
One customer I spoke to, for example,
that had built their RAG in Data Cloud
in just a few hours with the help
of a Salesforce solution engineer,
where they had spent a ton of time
trying to pull all of their product
catalogs into their DIY
approach.
And never really been successful with it.
another key area that we looked at,
as I said, was prompt engineering.
You know,
because Agentforce has prompt builder,
it allows organizations to easily build
and reuse prompt templates,
reduces the need for that
deep, prompt engineering expertise,
and also those pre-built prompts
with out-of-the-box skills and Agentforce
further accelerate that process
of actually building up those workflows.
One customer shared how a manual process
that took them a year to work on in
their DIY project was operationalized
in Agentforce in just a month now,
but probably one of the biggest areas
that we found that was a challenge
for customers in the DIY area was the
AI guardrails and security features.
So the trust layer in Salesforce provides
these essential features with zero data
retention and toxicity detection,
the masking of data going to the LLM,
and all of those capabilities
that enable customers to take advantage of
AI and generative
AI will feel comfortable
and being sure
that they're going to have data security
what's interesting is we found that
a lot of folks had spent a ton of time
trying to figure out how to build their
own trust layer within their DIY projects.
that it would have been like 20
or 30 times the efforts that it took them
to configure trust layer in Salesforce
in an Agentforce. Wow.
No. And some of them had been trying
for more than a year.
What was that big factor that helped them
move beyond the pilot?
Yeah, I mean, it was really
they got to a point where they recognized
that they couldn't get greater accuracy
without putting more data
into their DIY models,
but they couldn't comfortably
put that data in there
because they weren't
sure it could be secure.
Another area obviously
the user application development,
And what we found with Agent Builder
and the sort of drag and drop interface,
the conversational instruction dialogs
automated a lot of that UI
and workflow development process.
So they didn't have to think about
how they were actually going to present it
to the end users.
Customers
estimated that it would take them at least
to develop that customer facing UI
and associated workflows without Agentforce.
And I mean, we're talking like weeks
for folks to get prototypes up and working
where they might have spent
a ton of time and development effort
if they were trying to do it in DIY
fashion.
I mean, each of these categories
has so much significant
work, I can really see how it starts
to stack up.
And it's and it's not just the time
or the work.
It's really the expertise. Right?
There's a big learning curve
for a lot of this stuff.
Like I said before, just because
you know how to code or you know, a
little bit about
AI doesn't make you a prompt engineer.
It doesn't enable you to build out
kind of the trust and security layers
and all these kind of advanced
data features.
Nor, frankly, as one company can,
you probably
cost effectively do it
even if you do have the resources.
the the data security and guardrails,
but also the tuning, right?
We talked about only 50-ish% accuracy
with these DIY projects.
That was a real challenge.
And what we found with Agentforce
with the atlas reasoning engine
and the testing center,
along with the ability to tune
agents with natural language
instructions, allowed business users,
not prompt engineers or AI experts,
to be able to rapidly and iteratively
improve accuracy as they saw
how Agentforce actually worked.
Customers said they had an increase
in accuracy with minor tuning.
They were able to get more accuracy
out of the box initially,
and then really being able
to achieve levels
they thought it would have taken years
for them to do with the DIY approach.
One customers saw their accuracy
go from like 60%
to 85%, pretty much out of the gate.
Thank you for going through all of that.
That was a lot to unpack
in each of those categories.
Well,
thanks for letting me geek out on you
a little bit, but I think it's important
for people to understand,
like all the different layers
that are there.
Absolutely.
I think it's important to do that. Right?
Because not necessarily obvious how much
complexities at each one of those layers.
So you spoke to a lot of customers
while you were doing this research,
and you learned a lot about the use cases.
for a full testimonial today,
but can you talk to us a little bit
about what were the common outcomes
that you saw across the use cases?
So just to frame it out a little bit,
Nicole, people often ask us about
which agent for this use
case is likely to deliver the most value?
And we sum it up like this.
You have to think about the four P's
people, process, price, and potential.
Those Agentforce projects that impact
a lot of people have a lot of process
steps are expensive to do today and
have the potential to really transform
operations
are those that deliver the most value.
So what would fall into each of those
categories?
So self-service that provides
a personalized or individualized
response based on customer history
or particular product
skew is one. You know we talked
about one customer, a manufacturer
that had used RAG
to bring their product catalogs
into Agentforce,
and they had a ton of SKUs. For them
being able to reason through what product
a customer had actually purchased
was the difference between an Agentic
AI approach that
gave them a real answer, versus their DIY
approach that munged to gather data
from multiple product catalogs and
gave them a somewhat nonsensical answer.
That was the approach that really solved
the customer issue and avoided
that call, email, or other interaction
with the customer service rep.
Obviously, it's early days for AI agents
in production for many organizations,
but what we consistently heard from
customers was significant reduction
in the time and resources
required to build and deploy effective
AI agents
and be able to answer these customer
queries
with self-service in a personalized way,
based on customer history and product
and their company data.
They highlighted the ease of use,
the pre-built components,
and the robust security features
as key enablers for them,
and this really drove faster deployment,
higher accuracy, really translating
into quicker realization of benefits
and a faster path to return on investment.
Many felt that they were achieving results
that simply wouldn't have been attainable
with their DIY efforts
because of lack of resources, expertise
or concerns about data security.
That makes sense. That's a lot.
One more question
before we dive into a demo
based on everything you've shared today
about Valoir's research into Agentforce
and the challenges
organizations are facing with DIY,
AI pilots and deployments,
what are the top three recommendations
you would give to organizations
that are looking to get started with Agentic AI?
First just say no to DIY.
make a sweater with a sheep
and a pair of knitting needles.
It doesn't mean you should.
As we talked about, there's
a lot of layers to this DIY products.
You faced numerous hurdles.
You got model set up, you got data
integration, you got prompt engineering.
And it's not just about
the initial set up.
It's also about the cost of maintaining
and governing these projects over time.
And that's where you need the platform
capabilities, the ability
for business users to make changes,
the testing tools, the monitoring tools.
By leveraging a platform
with the pre-built models,
all the integration, organizations
can bypass a lot of the technical work,
a lot of the AI work and see a much
quicker path to return on investment.
I often tell people that this is like
the early days of cloud, when people
still thought they should build
and maintain everything themselves.
Again, sweater, sheep, knitting needles.
Why do it? The second point is data is
really the foundation for successful AI.
This means being able to bring in
both structured and unstructured data,
and having your data architecture
and hygiene and all DIY approaches
really struggle with connecting data
and most importantly,
with ensuring security and privacy.
And what we found was that Salesforce's
investments in Data Cloud
help customers integrate
both Salesforce and non Salesforce data,
which everyone has to ground their agents,
which was really important.
Both for accuracy and for personalization.
A third keypoint, when I would say
is recognize that this is a journey,
not a destination, right?
You have the guardrails, the tools,
tuning an Agentforce
to help customers get basic
AI agents up and running quickly,
but that will help them
see what more they can do.
Early adopters
found their initial efforts, showed them
how they could get even more value
from Agentforce by bringing in more data,
refining flows,
or making instructions more explicit.
The good news is that unlike DIY projects
that require multiple
AI experts and prompt engineers,
Agentforce allows business users
to build and adapt their agents over time,
and as their experience grows,
then they can drive ongoing increasing
ROI from Agentforce.
So no knitting my own sweater.
Get your data in order
and then iterate.
And I would say one final thing, Nicole.
The goal is remember, this is not about
a technology project, right?
This is about delivering value
and delivering outcomes.
So if you think about what
I want to potentially do with Agentforce,
think about those four P's people process,
price and potential
can help you to identify out of the gate
what your best use
cases are going to be for
getting the most value from Agentforce.
I know this has been incredibly helpful
for our audience today.
And now for anyone in our audience
who would like to see this in action.
Let's see a demo of how Agentforce works
to simplify
AI agent development,
reliability and performance.
And get you moving from
pilot to deployment faster.
The promise AI of agents is they'll help us work faster
by enabling us to express a job to be done
and doing that job
autonomously on our behalf.
But companies are struggling to deliver
on this promise, and finding that it takes
much more than a large language model
to make AI agents reliable for business.
A model will never truly know
your business, because your business data
is always changing, and a model's response
will only be as current
as the last time it was trained,
which means you'll need to build
a lot of data retrieval systems
around the model to give your agents
understanding about the jobs
you're asking them to do, and models well,
they have no concept of security.
Once your data goes into the model,
your security permissions are gone.
Everyone can see everything,
which means you'll need to build
security enforcement systems
around the model to protect your data.
It can't actually take action.
It can only recommend action
that you'll need to execute,
which means you'll need to build action
orchestration systems around the model
to turn recommendations
into actions in your business.
And this is just the beginning
of what companies
need to build around a model
to make AI agents reliable.
But what if there was a way to skip
all this and just start building an agent?
Well, that is exactly what Agentforce
enables your company to do.
Because these missing systems are built in
and built
on the Salesforce platform
and integrated with your Customer 360,
which means the only question is
what agent should you build first?
An Agentforce helps with that too,
by providing a library of pre-built
agents, like an AI service
agent that answers customer questions
and deflects incoming cases, or an AI SDR
agent that proactively works leads
and books meetings for your sales reps
or an AI sales coach agent
that attends customer calls and provides
real time tips and objection handling.
Each of these agents
was built on a platform,
and it's a platform you can use to build
your own custom agents,
which you can start doing
by simply describing the agent
you want to build in natural language.
Agentforce will then use Salesforce's
understanding of your business
to auto generate this agent.
Then you'll customize your agents by
describing the topics they can engage with
and the instructions they'll use
when engaging with those topics.
Using natural language descriptions.
And these form the guardrails
that direct your agent's behavior.
And you can test that behavior right here
inside Agent Builder
where you can inspect
exactly how your agents do a job.
It's the reason, step by step, learning
and adapting as they work.
How does Agentforce reason and learn?
Well, as I mentioned earlier,
at the heart of an agent's
capabilities is a large language model.
But the model is an incomplete solution
for turning a job
into an outcome in your business,
which is why Agentforce starts
with data retrieval, which helps
your agents get as much understanding
as possible about the jobs
they're asked to do.
All of that context is
put into a grounded prompt
that is sent to the model, but
we're not asking the model to do the job.
Instead, Agentforce is asking the model
for a plan to do the job,
and then Agentforce orchestrates
the execution of that plan
to produce the desired outcome.
But this is still an incomplete picture
of how this really works
When starting a job, it is rare
that we have perfect understanding.
In fact, it's only by working a job that
we get a full understanding of the job.
This is true for us,
and it's true for your AI agents as well.
Which means to be reliable,
your agents need to be capable of learning
and adapting as they work.
And that's why we created the Atlas
Reasoning Engine, which
coordinates
all these systems into a reasoning loop
that allows your agents to work a job
step by step observing, recommending,
executing, learning, and adapting
as they navigate to the desired outcome.
This is what makes Agentforce
more reliable.
But none of this is possible
without access to the right data
and Agentforce has built in access
to your Salesforce CRM data,
as well as the zero copy external data
you've connected with Data Cloud.
These are integrations
you don't need to build,
but Agentforce also has built in access
to vector databases to power semantic
search and retrieval augmented generation,
which you can use
by making those semantic search results
a part of the prompts
used to ground the model,
which gives your agents a way to retrieve
data that provides
a much deeper understanding of the job
to be done, and data
that is protected by a trust layer
that enforces your company's
security policies and data
that sometimes allows your agent
to recognize that it doesn't know
the answer and instead of hallucinating,
should get a human involved,
which Agentforce also supports
with seamless handoffs to your employees,
because Agentforce is pre integrated
with your customer 360,
which means you don't need to build
a lot of interface integration
to create a collaborative experience
between your agents and your employees.
This is how Agentforce helps
you deliver on the promise of AI agents,
by providing all the missing ingredients
that make AI agents real
and reliable for business.
To get started, visit Agentforce.com.
As you can see, Agentforce provides
a powerful
and intuitive platform for building
and deploying AI agents.
We believe it
truly represents a paradigm shift
in how organizations
can leverage the power of AI.
So we're now going to answer
a few rapid fire questions.
Shall we do it? I'm ready. All right.
One of the biggest challenges we hear from
customers is a fear that they don't have
the right data.
You talked about data. Yeah.
So what do you think is different
about Salesforce's approach
Approach to data? Well,
I think there are two things.
First, for Salesforce customers
that already have their data in CRM,
a lot of the heavy
lifting has been done for them, right?
Second thing is because I'm able to build
Agentforce agents so quickly
and test them so quickly,
I can quickly see as a business user,
if I do have a hole in my data,
or if I have a data effort
that I need to make right.
isn't giving me the right answer,
I can see exactly where I have a gap.
So you're sort of reverse
engineering the data hygiene thing,
which means that business users
can start with simple tasks,
understand
where they may have deficits in their data
or inconsistencies in their data,
and then be able to either
add instructions or find that area of data
to make agents work better.
At identifying that gap, right?
Okay., second, rapid fire question.
you talk about the challenges
that organizations are facing,
you know, in reaching that
acceptable level of accuracy with a
do it yourself approach.
So what specific aspects
of the data management integration,
or other elements become the most critical
when you want to ensure high accuracy?
And how can an organization
prepare for that type of complexity?
Sure, so again, it comes back to you.
not just an accurate answer, but an answer
that is right for that customer?
That means being able to have the customer
data, but also product and company data
all in the same model,
grounding those agents so they can pull
data that's important for that specific
customer, the Atlas reasoning engine,
I'm glad you mentioned that,
because that's a really important part,
too, in terms of being able
to think through
which of these different customer
examples apply.
So I get the right answer
for the right customer.
what tips would you share with our viewers
around the implementation process
and specifically how it relates
to usability and adoption?
folks experimenting with employee
facing agents
first, sometimes before they're ready
to put that out there to the customer.
But what we see is test early and often,
and you want to think about testing
not just for accuracy,
but really thinking about
what is the outcome
and what is the customer experience.
So for example, if you ask for a return
and we have a no returns policy,
a good customer service
agent might recognize that you're
a loyal customer and be able
to offer you some return anyway.
So how do I think about as I get better
with things like Agentforce tuning.
not just for, did
I give you the correct answer, but did I
give you the best answer
for that customer.
Really knowing your customers
and reaching them in a whole new,
more personalized way. Right. Love it.
All right, well,
we've now reached the end of our time.
And Rebecca,
do you have any final thoughts
or anything you'd like to share?
Well Nicole,
the key takeaway really here is that while
the promise of Agentic AI is immense,
the DIY path can be full of challenges
and delays.
The difference with a platform approach
like Agentforce is significantly faster,
more secure way to realize
the benefits of autonomous AI agents
Thank you so much, Rebecca,
for being here today and for sharing
all of your research with us.
It's been very helpful
and very informative.
For those of you who would like to learn
more about Agentforce, please visit
Agentforce.com and you can also find
more information about Valoir
on their website at Valoir.com
Thank you again
to everyone who joined us today.
We hope you found this valuable,
and we look forward to helping you
accelerate your AI journey.