I'm your host, Diane Mizota
at Salesforce headquarters
Over the next 30 minutes,
help.salesforce.com, the customer help
with over 60 million visits
requests handled by Agentforce,
to the tech and teamwork behind it,
to its impact on customers
what we've learned along the way.
Now I want to welcome Jim Roth,
president, Customer Success,
Success here at Salesforce.
Hey, thanks for having me to be here.
So let's start with the milestone.
Salesforce Help just surpassed
1 million support requests.
Congratulations, first of all.
And what does that moment
Yeah it's a it's a big milestone.
where we were able to help
customers real time, 24/7,
without hold music or transfers
getting in the help they needed.
And it's really a celebration
of the fact that we can deliver an easy
Yeah, I love what Jim said there.
portal was always a good experience.
for help portal in the past,
but we always heard from customers,
and we had a lot of content.
And so what we wanted to do
that experience to an agenetic
And that milestone of 1 million
They're finding their answers faster,
getting the resolution that they need.
So it's been an incredible milestone
that we're on the right path
and powered on experience
So tell us about the inspiration
behind Agentforce and the Help Portal.
Like what was your sort of driving
Well, we wanted to deliver that easy
you know, historically when customers
come to self-service portals,
sometimes they have to browse
and read documents, piece it together,
You think about the way search works.
and piece the answers together.
is synthesizing all that information
that answers your specific question.
And it does it real time,
to get to the next human or whatever.
It's doing it right there,
synthesizing the answer you need
Nobody wakes up in the morning saying,
I really hope I need to contact
always a disruption in the day.
And so when you have that moment,
you want to get people in
and out of that experience
And we saw this technology
So Agentforce has handled over
requests helping so many people.
And I want to know, like,
and what has surprised you.
And I think the big thing for me is
this is not a traditional
you kind of release it, people use it,
living, breathing system.
We're constantly looking at it
to see how it's performing,
and we're tweaking the performance
based on what we're seeing.
So it's it's very different
about traditional software.
We have had some funny moments as well.
to put their answer in hip hop lyrics,
which I thought was an interesting one.
When you spit some back at us.
I kind of want to be able to do that
to tell us how to cook spaghetti.
So you get some interesting moments
But yeah, the big learning is
releasing in agenetic AI agent
just like how you would manage
that's also been a learning for us,
we think people feel more comfortable
to an agent than they were a human.
Yeah, because I think what a humans.
that maybe I'm going to come across
a little bit silly here or whatever.
No one knows the conversation
you're having with the agent, right?
It's just you and the agent.
we think people feel more comfortable
asking the questions to learn.
So we're asking those questions
that they would never ask
of our human support reps.
They'll ask questions like,
which products of Salesforce
I own or what's my renewal date?
And historically, customers
would never go through the friction
to kind of open up a ticket
and go ask those questions.
But in a frictionless environment,
maybe they're not worried
they're willing to ask more question.
because it's more engagement.
And those are easy questions
And their understanding of AI too,
knowing what kinds of questions
they can ask and not feeling
that kind of judgment. That's actually.
how are you guys measuring success?
have a couple of metrics we look at.
The first thing we think about
is something called resolution rate,
which is basically a customer
and they don't create a case.
Now that's not a perfect metric, right.
Because you could say, well,
maybe they just went away
So we also ask a question
at the end of an answer of something
which is we basically ask the customer,
did we resolve the issue yes or no?
We think about this is explicit
You need both metrics that come
50,000 conversations a week.
Not everyone responds to that question.
That customer confirmed resolution.
So you have to look at both to see
are we driving in the right direction.
The other thing that we look at is
we also measure what we call CSAT,
which is what companies do
in the service industry today.
And we call that latency.
delivering on that experience.
an easy experience for a customer.
is a phenomenal resource for us
to look at these metrics.
It runs a sentiment analysis
on all of the conversations
that Agent Force is having
and tells us if they're positive,
So we can dig in and understand
where do we need to improve it.
Jim, what are some of the reactions
Yeah I mean customers really view this
as a better or faster experience.
It goes back to this easy an expert
and we can really deliver on that.
An interesting example of
We're really good at English
and we're pretty good at Japanese.
But there's a long tail of languages
and expect us to respond in.
And we aren't as fast at that.
Japanese, we're about to launch
And if you imagine that scenario
instant responses to these questions
in all of those languages.
it might have taken a day or two
to get those answers back to customers.
really like the experience.
the fact that the answers
documentation and content.
that experience across the board
in whatever language you want
with the smartest content
So the experience has been really good.
It's definitely not perfect.
we're using Command Center
to kind of tune the experience
So it's constantly improving.
But we're off to a great start.
And the response has been great.
And one thing to build on
is we're only using our English content
to do the translation for a customer.
industry in the world company Jim.
to translate their content.
They have a lot of English content,
that it's available in today.
So for Agent Force, think about that.
We're able to use that English content.
And it's able to respond in Japanese
and these other languages
that we're going live in.
that's coming soon as well,
sometimes Agentforce needs
with the agent in French,
and not talk to the human in French.
So the other piece of the technology
stack that we're working on
with, is the ability for a human
So the entire experience,
The agent, the digital agent,
into another language like French.
And then if you have to go to a human,
that human could still be
that message is being translated
into the other language as well.
we think there's a huge opportunity
to deliver in ten, 20, 50 languages.
all using the power of AI.
And it's so exciting to sort of
like have that customer experience
for people all over the world. Exactly.
Now, do you have any best practices
trying to follow our lead?
I think the first thing I would say is
you want to start with your data.
It's really important to say
that you have your data ready to go.
the way you would start with
let's get all my content together.
that into, Data Cloud, in our case,
so that the LLM has access to that,
and we alluded to this earlier,
is that sometimes customers are asking
data that is really or questions
that comes from structured data,
like when's my renewal date
And so we've kind of expanded
but also structured data.
the foundation for the AG.
And that's pretty typical.
You know we started with hey,
let's load up the content.
We added more content, added
And that was really the foundation.
that we didn't have content gaps.
We had to create some new content.
We also had to tune the content.
that was in conflict with each other.
where things are kind of in
conflict with each other.
So we had to kind of prune
all that and tune all that.
And that's an ongoing journey.
But I would say the biggest lesson
and we didn't learn this until January,
we started this journey in October,
this other part of service
that you have to deliver,
and how you make customers feel.
So it's not just is your answer
how are you making somebody feel?
That's what defines great service.
And so we think about this
So how do you think about
your digital agent empathy
you would teach a human rep empathy.
we put our human agents through
soft skills training to learn
if a customer gets frustrated or angry?
They want to talk to a supervisor.
And so we had to teach that
to our digital agents as well.
that some people will think about
this is this is a new software
you know, load up the data
And there is a piece of that
But this is as much about
onboarding a new employee
as it is implementing new software.
And just like your new employee,
on your products to start,
and then you would kind of tune them
to kind of coach them on soft skills
and have empathy for customers.
And that is really part of the journey.
a really good example of this,
when a customer has an outage.
Now we have great engineers,
and so it doesn't happen very often,
it is a really make or break moment.
We can either build or erode
a ton of trust just like that.
comes into our agent and says,
hey, I'm having an outage,
it's really important how we show up.
We don't want to reference
or ask them to authenticate.
We want to start just like a human.
so sorry to hear about your situation.
And the LLMs are really good
at picking up on those clues
like outage and downtime and
and they can pick up on that.
But you have to teach the agent
the customer in the right way
and deliver a great service experience.
not only how good is your answer,
but how do you make the customer feel.
And that's really the learning
and we wish we knew it earlier,
with all of our customers
so they can benefit. Yeah.
I think as a customer myself
you don't want to feel like you're
talking to a non-entity. Right.
And I think that's so exciting
and be able to relate to us
ultimately lead to customer success
and that trust in the relationship.
What makes Agent Force work, Bernard?
Think that the massive unlock for us
Like Jim touched on this a little bit
and unstructured content,
like we started with all of our content
that's available in English.
You call that unstructured data,
pieces of content knowledge,
articles, help docs, developer docs.
And so we were able to unlock
all of that with Data Cloud.
And then what we started to do
Things like, who are you?
You know, what products do you use?
Did you have service requests, etc.
So Data Cloud is the unlock that drives
We wouldn't have been able to launch
if we didn't have Data Cloud.
really key element for us.
So that's been a massive part
I think the other thing I would say is,
after you get that launch,
Are we giving the right quality
So the ability to go in and tune
these responses is really important.
When we first started out,
hundred a week, and Bernard
and I were literally reading
every single transcript ourselves
saying, hey, that answer was good,
that one could be better, all of that.
But now we're doing 50,000 a week. Wow.
So you think about, well,
a week and see the patterns
and how to make them better?
actually takes the transcripts,
throws them back into LLMs
and look for patterns in that and
says, hey, you're missing content here.
Or this prompt needs more empathy
because it's not really landing
with customers the right way.
how to resolve this question.
in those patterns at scale.
improvement of the response
quality is the other piece.
like we have content gaps.
We need to tune prompts other things.
And so that's really the key.
just kind of hook up the pipes
flipping the switch and you're done.
Yeah, there is an ongoing process.
you would have ongoing oversight
and management of a human employee.
You have to have ongoing management
of the digital agent as well.
And that's also where an agent builder,
where we spend our time on our topics.
we spend a lot of time as we launched,
We have to go in and rewrite
them, change the language.
And so you spend a lot of time
What's really cool about that is
if I think about what we have today
we have seven topics and eight actions.
We have a lot of products
We have a lot of content.
And you think about that simplicity
Seven topics, eight actions.
That's all it takes to drive,
you know, 1 million support requests.
That's pretty mind blowing.
And it's all written in human easy
human language. No code. Absolutely.
So you spoke about this earlier.
What makes this different
from a typical software development
and that ongoing kind of maintenance.
once you keep training them,
not an old school software release.
You're constantly making it better.
Like we focus on this metric
that we call answer quality,
which is are we getting better?
and the response that the agents given.
tuning to the command center
in Agent Builder to do that.
new products all the time too,
so it's always a challenge
to keep up with those as well.
And I think that's a key lesson
This journey has incredible benefits,
you have to kind of tune this thing
along the way to keep making it better
and stay up with everything
do you have any other learnings?
and one that really resonates.
I think when I talk to customers myself
is we spent a lot of time
at the start of age and force
thinking about our guardrails.
You might ask, what is a guardrail?
There's certain questions
that you might not want your agent
if a customer asks about Marc Benioff,
we don't want to answer that.
to an investor relations page.
It's an example of a guardrail.
We also put a guardrail in place
we don't want you to talk
about competitor products.
If someone comes and says,
is X better than Salesforce?
So we don't want you to answer that.
And then we looked at our conversations
not like the very next day,
Microsoft Teams with Salesforce?
And we have great content
But what happened was our agent
followed the guardrail, right?
Do not talk about our competitors.
we were treating the agent
like an old school chat bot.
So we went back into something
called the Global Instructions,
which is kind of what teaches
and tone of how it show up.
And we said with these words we used,
you are an employee of Salesforce.
Put the best interest of Salesforce
And we took away the guardrail.
Now, if you ask a question,
and it responds in a really good way.
So that was a huge learning for us.
we needed to let the LLM be a LLM and
which is how we were thinking about it
Yeah, that's really cool.
Yeah, big, big learning for us
with many of our customers
because you do really need
what do you not want your agent
And it is so new that like,
I think people would have that tendency
to look at it like a chat bot.
But really you have to kind of expand
your thinking in a whole new way.
Yeah, it's a whole new way
Now let's shift to the people side.
Jim, how is Agentforce changed
and what are you hearing from employees
and how are they adapting?
I mean, one of the things about people
they really are motivated
And our team is no different on that.
to do more proactive things
But typically service teams
where they only have enough capacity
to handle the volume coming at them.
And our team is like that as well.
and what really inspires them
Agentforce is taking more
it's freeing up our teams to go
be more proactive with customers.
How do we go help customers
instead of just responding to them
And that is a huge benefit.
And so customers love that as well.
And our team really loves that.
the important message for us.
We went to them and said, look,
the experts on our products
to help make this agent better.
And when we make the agent better
resolving issues for customers,
that's going to free up your time
more time on the really complex issues.
And so they're very energized by it,
which has been great to see.
that would be more rewarding, right.
To focus on those moments
where you're really helping.
Yeah, I think like to build on
We've always had our support
engineers be part of this project
because our support engineers
are the experts in delivering
support for their products.
They review the conversations
maybe didn't do a good job.
They helped create content
where maybe we're missing content.
So I think having our support
engineers just be part of this
You don't want to set it up
or it's not us versus them.
So bringing them along, making them
And it's been a huge success
congrats on this milestone, but
What's what are we looking forward to
we're only scratching the surface
to be honest of where we are right now.
So we've a lot that we want
Like we talked about the languages,
over the next couple of weeks
to go live in five more languages,
which is pretty exciting.
as the UX for these agents.
we're getting really excited
is that we're actually going
today it's all tech space,
you're having a conversation
and it's not like some robo call.
It's like you're having a conversation
with a human who's guiding you on
how to do something in the product.
And then to help with that,
we're actually getting ready
and Android app of our help portal.
why would I download these apps right?
But imagine you can have a conversation
with Agentforce on your phone
through what you want to do.
So that's a big part for us as well.
The other piece, I'd say is
a lot more personalization.
So using this structured data
from the C360, customer 360
personalize the experience from,
you're using and guide our responses.
That's a big piece for us as well.
So Jim, I'm sure people may be watching
and thinking, how do I get started?
What would you say to them?
I think the key is just to start small,
one scenario and just start
very quickly in literally in minutes
Start playing with the technology,
figure out what data you need
to go turn this on for your customers
as kind of a new channel,
hey, this is new, all new.
Give us feedback and get going.
And especially your customers
that really care about your company
They're going to be all in on it
and give you the feedback.
So I would just say, don't be,
Just get going and start now.
And there's a lot of resources here
at Salesforce to help you.
And we're excited to see you,
I think Jim covered it really well.
to a small percentage of our help
that meant we had like 126
conversations at the start,
and we could look at every single
feel confident in the performance,
the amount of customers who used it
we turned it on to everybody.
and then gradually expand
That's how you go about it.
for this conversation today.
just seen what's possible with
We believe in trust and transparency.
If you'd like to take a deeper look,
page at the link on screen.
I'm Diane Mizota here at Salesforce
HQ and we'll see you next time.
