The Vision of Agents in the Enterprise

Transcript of Salesforce: Agents in the Enterprise | Road to Intelligent Data Apps, 6/21/24

Interview with Vijay Pandiarajan, VP of Product Management, Salesforce Mulesoft

• Salesforce is introducing agents that can accomplish tasks on behalf of users by generating workflows from building blocks
• Agents leverage Salesforce’s operational applications, Customer 360 data model, and integration technology to access data and take action across disparate systems
• Gain insight into how enterprise agents differ from general-purpose consumer agents by having pre-defined business processes as building blocks, access to the state of the business, and the ability to improve by measuring their goals against business outcomes
• In the very near future, agents may be orchestrated to achieve broader business objectives and continuously optimized via feedback loops

In this interview, Vijay Pandiarajan, VP of Product Management at Salesforce, explains the company’s approach to enterprise AI agents. Readers will learn how Salesforce agents leverage the company’s Customer 360 data model and integration capabilities to access data and take coordinated action across multiple systems.

Pandiarajan outlines how these agents can be composed to orchestrate complex workflows to accomplish tasks on behalf of users. He illustrates this with a supply chain example involving multiple specialized agents collaborating under a “chief of staff” agent to replan procurement, logistics, and stocking for a consumer products company.

Enterprise agents like Salesforce’s differ from general purpose AI assistants. Key advantages include having a constrained domain, access to structured business data, and more specific evaluation functions that enable iterative improvement via human feedback.

Looking ahead, Pandiarajan shares a vision for how individual agents may be orchestrated to achieve broader business objectives and continuously optimized through feedback loops.

Transcript

• Relevance of agents and how Salesforce is bringing this vision to life

George Gilbert Our conversation today is about the relevance of agents. But many of the agents you’ll be hearing about in the coming months are attempting to tackle a very hard computer science problem. They’re trying to solve an open-ended planning and reasoning problem to navigate a relatively open world. Salesforce, and some other enterprise vendors, have a unique opportunity because their customers have a map of the state of the customer facing portion of their business in the form of applications and the Data Cloud. They have the equivalent of what researchers call a world model. That world model converts that open-ended planning and reasoning problem into a much simpler search problem. Customers can create an agent, give it a task and tools in the form of a library of workflows, and it can navigate its way to accomplishing the objective.

Today, we’re going to talk about how Salesforce is starting to bring this vision to life. We’re with Vijay Pandiarajan, who’s VP of Product Management for MuleSoft, and the occasion is the introduction of some of the first technology Salesforce is introducing to help developers build agents. Vijay, tell us about your role and what Salesforce is introducing at its event on June 25th.

• Vijay’s role at MuleSoft and Salesforce’s upcoming announcements

Vijay Pandiarajan Yeah, hey George, and hey everyone. Really excited to be here to talk about some of the cool things we’re building out. Cool and impactful, I would say. We have this notion of autonomous agents and how they’re going to help simplify work for various creators and various business users in the enterprise. In particular, we’re going to be talking about AI powered customer experiences at our conference next week, which I think will be super relevant to the many customers that are going to join us. My role at MuleSoft is to think ahead, to think about the problems that customers have faced, the enduring problems and some maybe newer problems. And then being able to take the evolution of technology, especially with the AI technologies that we’ve had here, and bring them to bear in our automation and integration spaces. There’s a lot of new ground to be plowed and new ways to solve enduring problems.

• Initial killer apps for generative AI – software development and customer service

George Gilbert Okay. So we’ve seen a ton of hype around gen AI. We’ve seen Copilots and assistants, basically where it’s almost like a request response pattern of interaction. And it seems within that broad category of where the state of technology is, that there’s two initial killer apps. One is software development, the other is customer service. And it’s clear, within the context of actually each of those, that the next wave of gen AI is agents. So elaborate on the definition of agents that developers will be able to create within the context of these tools that Salesforce is introducing.

• More conceptual thinking and supervision, with agents handling day-to-day coding tasks

Vijay Pandiarajan Yeah, think about the job of any creator. I think developers are the best personification of it, but really there are other creators and builders in your organization as well. People whose job it is to put together interesting and useful applications that help end users get their jobs done. And in this realm, a lot of the work ends up being, you could say at the end, coding work and then there’s a lot of conceptual work. And usually the conceptual work, you might call these people architects or you might call them… This is the best way to build this, so it’s scalable, and it’s secure, and it fits within the system. What we’re going to see is that more of the hands-on day-to- day coding is going to be delegated over to agents themselves, and the intent-based aspects of it, of what the developer’s journey is going to get further emphasized.

So you’re going to be doing more conceptual thinking and supervision, and then the day-to-day coding tasks are going to be handled by these agents. And we’ve sort of seen this happen over the last 18 months with gen AI and the various LLM opportunities that people have had. Every developer I know now has a subscription and is almost sort of doing this on their own in a handcrafted kind of way. We’d love to bring that directly into the tools that people are working in every day, so you’re not sort of handcrafting this. So really a bigger emphasis on intent. And more of the cookbook pieces of here’s a snippet to solve this particular problem, it’s just going to be delegated down to a particular agent. Think there’s a lot of opportunity there.

• Make this concrete with a use case example

George Gilbert Okay, so let’s make this concrete. Let’s talk about a use case that a customer might relate to, where you’ve got individual specialist agents that might relieve a customer working in Salesforce from the tedium of individual tasks, and maybe an example where several of these are working together under the supervision of a user or a user’s own agent.

• A supply chain scenario with a chief of staff agent supported by specialized agents

Vijay Pandiarajan Yeah, think of a, let’s pick a retail scenario, or you’re a logistics manager, supply chain manager at a company. You’ve got many things going on through the year. You’re always planning ahead to some extent. And then as that particular season approaches, you are fine- tuning your stocking and your allocation and the approach that you’re taking. So there’s both a planning aspect and an execution aspect in the supply chain world. And sometimes these two actually collide and you have to make changes. Not everything goes the way you’re expecting it to. So think about a supply chain manager being supported by, let’s call it a chief of staff agent, because this agent really is helping the supply chain manager with a variety of tasks. Obviously, this chief of staff agent is not going to know everything, and that’s where sort of a trusted network of specialized agents come in.

So if you were to think of a scenario where you’re not expecting an unseasonable heat wave, what we’re sort of experiencing right now, and you’ve made your product allocations in a particular way and certain consignment of products to be delivered to certain stores, your chief of staff agent is happily keeping up with some of these anomalies and is alerting you to say, “Look, in about a month or so, we’re expecting unseasonable temperatures in these certain places. And you may want to take some action to make changes to the allocation of goods to these places. And not just that, I’m going to go pull up a plan for you to review and take action on. ” And so this chief of staff agent might look to some other specialized agents to say which stores are going to be in this affected heat wave area?

What subset of products are actually going to change because of this heat wave? Maybe it’s sunscreen and it’s sunshades, and things of that nature. So there’s going to be a subset of product allocations that this agent has to look at. And so that becomes a little bit of the fulfillment agent starts coming into play to say, This is what we were going to send the store, but maybe we should be redistributing that. ” And then from there on, this chief of staff agent is likely looking at a supplier agent as well to say, “Okay, who is the supplier that’s actually supplying us these pieces? Could they supply us more of this or should they be rerouting their shipments? We’re buying the same overall amount, but should they be rerouting some of those shipments to a different place? ” So right here, you saw several specialized agents come into play, right? There’s the inventory agent that came in, there’s the store agent that came in, there’s a fulfillment agent that came in, and even a supplier agent that came in, all sort of working in concert with this chief of staff agent. The ultimate goal of this is to provide a plan or provide a couple of options potentially to the supply chain manager to say, “Which way would you like to go? ” And it could very well be that there are cases where you might have to pay more to acquire these things, or you might have to bring on an additional supplier. So there’s different variations on this, but at the end, the point is the supply chain manager has a lot of help in assessing end goals, and it’s really because of this interconnection of these independent but verified agents that are being pulled together by this chief of staff agent, if you will.

• Agents have specific workflows/tools and generate their own workflow to accomplish tasks

George Gilbert Okay. So to be clear, each agent has a specific set of workflows or tools that it’s programmed to use, and it generates essentially its own workflow as to how to sequence and compose these other workflows to accomplish their specific tasks. So one might be, can this supplier increase my allocation of a certain product? Another agent would be, can my logistics providers move this allocation? And so there must be some mechanism for the agents to coordinate. I don’t know if that means there’s a distributed consensus going on where they’re negotiating on different plans that the individual agents come up with. Is there some sort of coordination mechanism?

• Coordination between agents through policies, constraints, and iterative refinement

Vijay Pandiarajan Yeah, there’s one, just the to find these agents themselves. So the chief of staff agent has a pallet of these specialized agents to go to. Typically, when you’re looking at fulfillment, there’s different things you can optimize for. You could be optimizing for shipping costs at the beginning of the month, and then actually at the end of the month you’re trying to say, “Okay, we’re exceeding our shipping cost budgets. We got to watch out for that. ” But at the beginning of the month, you might just be looking at customer satisfaction. And so yeah, there are some policies in place in which you can sort of dial the knob on which way you should go. In this case, clearly this is about a customer sat and making sure that you don’t run out of things at the particular store. So while the plans are being put together, there are some constraints on the fulfillment side that will help cue the recommendations in the direction that the supply chain manager wants.

George Gilbert Okay, and we’re going to get into unpacking this, but to be clear, it means the actions of and the outcomes of one agent can be constrained by the capabilities, or the constraints and the policy, built into other agents.

Vijay Pandiarajan Yeah, exactly. I think there are both operational policies that the agents can work off of. It’s not unlike an API today, You have API policy management that’s in there. This is a very similar notion. And ultimately, these agents are exposed as APIs to be called and managed, cataloged, and discovered.

• Constraints and goals become policy applied to an API

George Gilbert Okay. So the constraints and the goals become policy that you would’ve applied to an API. They just generate their own workflows with respect to those policies.

Vijay Pandiarajan Yeah. There you go. And what kind of answers you might get back. And this is also iterative, that’s the other thing. It comes back with a plan and then notionally, you should be able to change that depending on what your desired thing is at that moment. You might have other constraints that have to be applied further.

• Inputs and outputs of agents must be strongly typed and matched

George Gilbert And just to be clear, upfront, when someone is building each of the specialist agents, do the equivalent of inputs and outputs have to be strongly typed and matched to the other agents that will be feeding them and consuming their outputs?

Vijay Pandiarajan Yeah. Typically, you want to have some standard ways in which you’re transacting between your agents, and that’s the way you can actually understand what each agent is coming back with and what the next agent is going to interpret and take forward. Certainly, gen AI helps in this form as well, when it’s strongly typed and it’s well-defined, to further extrapolate the meaning of what is being passed. And so that makes for a better recommendation when it comes back. It’s not like, why would you make this recommendation? It doesn’t make any sense.

• Passing inputs/outputs and finding data requires harmonized business objects, not raw data

George Gilbert And then to put a placeholder in this to come back to, this passing the inputs and the outputs, and finding the data that the agents need to consume to do their analysis, that requires harmonized business objects, not raw data. Otherwise, they’ll get lost in the data lake or the operational app.

Vijay Pandiarajan Oh, 100 %. And this is where we feel our Data Cloud really shines, and we work with the data in Data Cloud where the data has been brought together, harmonized, and the interconnections between different things are laid out very clearly.

George Gilbert Okay, let me stop you there just so that… We’ll come back to that when we unpack that layer, but I want to unpack the tools first.

Sure. So let me bring up a screen

George Gilbert that we talked about earlier where we’re going to look at

Vijay Pandiarajan just a schema or a stack of the components. Now, let’s reiterate again, what exactly is coming out on the 25th? And then let’s fit it into this stack.

• Composable agents, AsyncAPI support, and AI-powered composability in the June 25th announcement

Vijay Pandiarajan Yeah, what we’re talking about on the 25th is a composable way to bring these agents together. So this notion of these autonomous agents are going to be brought together in an Async manner as well. So we’re going to be talking about two things. We’re going to be talking about AI powered composability, and we’re also going to be talking about AsyncAPI support. The AsyncAPI support is extremely important as events themselves transpire. How are we proactively bringing that information to bear onto all the work that we’re doing? So it’s an event driven architecture aspect that we have to bring in. But the big news is really around this idea of AI powered composability. And really, this factors in a couple of the things that we’ve already been talking about.

One is how do you get to a future ready foundation. And how can you get to the kinds of efficiencies you have with the intelligent tooling? We talked a little bit right at the start about developers and builders getting an impetus in terms of how they’re building these. And then how do you secure these AI powered experiences? How do you get them in a catalog? How do you find them? We talked about those disparate sets of things.

Okay, so let me pause you

George Gilbert and let’s unpack this piece by piece. Let’s start first with a definition, your definition of what an agent is and what it does.

• Definition of an agent: acts on your behalf, autonomous to different levels, achieves your goals

Vijay Pandiarajan An agent, at the very basic level, acts on your behalf. So there’s a couple of things right there. It takes your persona, it knows what you want, it knows what you have access to and it acts on your behalf. The second thing is it doesn’t need to be guided completely. It’s going to come back for guidance at certain points, but it’s going to be able to go and work autonomously to different levels so that it can achieve the goals that you have. It’s probably the simplest definition right now of what an agent would be. Something that knows who you are, what you want, and can act independently on your behalf.

George Gilbert Okay, so now let’s take it the next step. When we’re talking about acting on your behalf, what we’ve known in LLMs V1 is where they start to get more advanced is tool use. Like they can call a code editor or they might call out to some other tool. In the context of Salesforce, they’re calling out to workflow building blocks. So define for us what some of those building blocks are and how they get built. In other words, what does this composable environment look like and what is it comprised of?

• Anypoint Code Builder with generative flows for integration flows outside Salesforce

Vijay Pandiarajan There’s a few things. One, there’s the infrastructure one to have the catalog of these trusted things. As you create them, they show up in this place and you can validate what they do. The second notion is just how you create them yourself. And we span the spectrum for the different types of creators here. Developers with code, we get into what we’re doing on the Anypoint Code Builder side with generative flows there, integration flows.

George Gilbert Okay, just to be specific, that’s in the quadrant that’s on the connector’s row, but under programmed where you have, Anypoint has always had API integrations, but now Einstein, for Anypoint Code Builder makes it easy to generate those integrations.

Vijay Pandiarajan There you go. And the idea here is that you’re building it with natural language. You’re specifying the intent of what you want. Einstein for Code Builder helps you build that in code.

• Integrations reach beyond Salesforce to access applications and databases in the rest of the application estate

George Gilbert And to be clear, these integrations are for accessing applications and potentially databases outside the Salesforce universe. In other words, this is to reach beyond Salesforce to the rest of your application estate.

Vijay Pandiarajan 100%, right. There’s a wide variety of things that surround your Salesforce implementation that you need access to. Think about what we were talking with the supply chain side. There’s a lot of operational data sitting in these other systems that you would need to access, and this is the way you build those pieces to be able to get to those bits of data.

• How integrations are abstracted into connectors, which then become actions in the flow palette

George Gilbert Okay. Now, explain how integrations get abstracted into connectors and then how connectors become actions that are part of the flow palette of capabilities.

Vijay Pandiarajan Yeah, it’s a great question. Think about integration flows as specific endpoints that you’re accessing data from. But then you end up having a set of related queries that you want answers to. And typically that’s what we’re pulling together as a connector because the connector helps you finish the job to be done. There could be multiple tasks that you’re doing, but the job comprises of all of these tasks. So think of connectors at the job level and the individual integration flows at the task level.

• Connectors span a domain and wrap up related functions to accomplish a job

George Gilbert So just to be clear, like a connector could be like a customer experience or something about procurement that has to involve… Or maybe about supply planning that spans talking to different suppliers and a logistics provider. That might be a supply chain planning connector, even if one doesn’t exist yet.

Vijay Pandiarajan Yeah. I would say usually these connectors are within a particular domain, but there’s probably a set of discrete things that you might want to do. For example, you might want to ask a supplier about what they’re currently sending you today. You might then want to turn around and ask about what about these additional elements themselves? Can we modify what you’re sending us? And it’s a set of tasks that you’re doing with the same entity that likely is going to get wrapped up into a connector so that you have all of the related functions together and it makes sense to be able to exercise all of those conversation points with that third party.

• Connectors get pulled into the Salesforce ecosystem as Invocable Actions

George Gilbert Right. And then that gets pulled together,

Vijay Pandiarajan as you were saying earlier, into our Salesforce ecosystem. We have this little bits of magic called Invocable Action. So that’s one of the ways in which we can get to all of these different systems, interpolate what is going on, and then surface them back up inside Salesforce so you can have access to all of this information. And then it becomes callable from all of the other automation and integration capabilities we have directly within Salesforce as well.

• Different flow types in Salesforce, including Salesforce flows and third-party ones, composed by agents

George Gilbert Okay. So let’s get to that level, which I don’t think we’ve covered yet, which are all the different flow types that could include Salesforce flows and Salesforce action equivalents, as well as these third party ones. And then how an agent might, on its own, figure out a plan for composing these building blocks to accomplish a task.

• Salesforce Flow for admins to assemble flows using low-code/no-code, with generative AI allowing natural language flow creation

Vijay Pandiarajan Yeah. Let’s break those up into two parts, right? Let’s first pick up what’s going on inside the Salesforce ecosystem. We have Salesforce flow, and a different persona really that’s sitting here that’s different from the person sitting in front of Anypoint Code Builder. You have a Salesforce admin who is assembling flows using low code or no code at this point, right? They’re configuring, dragging, and dropping. What we’re really adding here with generative AI is the ability to go from, I would say low-code, to no code, where they can specify a request to create a flow just in natural language. Very similar to what we were just talking about on the ACB side. And you’ll get the same flow built out for you within Flow Builder. And there they can take advantage of several of these actions that we just talked about. You have all these independent agents that are functioning, and now they’ve been made available as Invocable Actions within the Salesforce ecosystem or within Salesforce Salesforce’s builders. And here, these Invocable Actions can then be pulled into a flow directly.

• In Salesforce terminology, actions are discrete tasks while flows are sequences of steps that can include actions

George Gilbert And to be clear, I know I’m interrupting you, but just to be clear, when it’s a Salesforce flow, those are essentially Salesforce actions native to the Salesforce apps, but the actions built on connectors are how you span to other applications?

Vijay Pandiarajan Yeah, I’ll just get our terminology right. Actions tends to be a really overused word, at least in our Salesforce ecosystem. So actions are very specific things that help you reach out and do one thing or two things. They’re actions. They have a definite beginning and an end. And what we were talking about, all these different agents being made available once they come in as Invocable Actions become that set of things. That’s the tentacles that lets you hit whatever the specific things that you want. Flows themselves tend to be a sequence of steps that can include actions within them as well. And so if you’re trying to pull information from different parts of Salesforce, you want to get user input from somebody else, depending on we have various types of flows within Salesforce. And actions can be interspersed within those sequence of steps.

• When building integrations/connectors, developers must handle transaction semantics, error handling, retries, etc.

George Gilbert And also, just to be clear as to where the current state of the technology is today, when you build an integration or a connector, it’s up to the developer to build in the transaction semantics, like how to talk to the external system, how to handle errors or retries, that sort of thing.

Vijay Pandiarajan Yeah, exactly. It tends to be more atomic within that and you know what needs to happen and what’s a good outcome and what’s not. And then there’s always fallback things that in other conditions that you can take care of so that the transaction actually goes through successfully.

• Raw data lakes make it hard for agents and applications to reason over business objects; harmonized data models are needed

George Gilbert Okay. Now, let’s talk about the data estate. We’ve touched on this before, but if we’re dealing with a raw data lake, agents and people would have trouble navigating it. And applications would have trouble essentially making sense of the business objects and communicating between agents or flows. So explain the role of the Customer 360 data model in making it easier to reason about actions that deal with customer data. And then how would customers, now I’m using the word your enterprise customers, how they might bring in additional data and harmonize that so that they can reason not just over the customer data, but maybe the supply chain data or other things.

• The Salesforce data model provides standard and custom objects; the Customer 360 data model in Data Cloud expands this with external data

Vijay Pandiarajan Yeah, for sure. And I think just to set up the background on this, you got to start with the Salesforce data model itself. Even before we think about our Customer 360 pieces in Data Cloud. There’s a very robust data model, this is what all our users love about Salesforce. It helps them express a number of things just directly in the standard objects that we provide, and they can also then write custom objects as well that’s really specific to them. Now, I think the bit that we’re doing with Data Cloud and the Customer 360 has just taken that to a whole entire level. And we’ve seen this in the last couple of years as customers have started using it. And really what makes it so powerful is connecting these ideas beyond what is just inside the Salesforce database itself. And that’s where,

when we see interactions happening in external systems that have a bearing on the user or the customer within Salesforce, we want to know about that and we want all of your intelligent applications to know about that and take advantage of that. It’s that additional information that’s happening off- stage from Salesforce that we want to bring in. And it has to be harmonized. It’s not brought in, and you still have to know what those interactions mean. A very simple example is just users, right? Users inside Salesforce might have different IDs when they’re doing things in other systems as well. So how do you actually harmonize and know that it’s the same person? That’s one really simple form of harmonization, I would say.

And the minute you know that it’s the same person and you look at a set of interactions that have happened in external system, and you overlay that with what you have inside Salesforce, you have a much fuller picture of what has actually happened, or what this poor customer has been through maybe in previous service requests or other purchases they may have made, and you can now really have a better understanding of what their intent is and what would be a good outcome for them. And so at that point, that’s our data graph and that’s our normalization aspects. And now we can take advantage of that with these agents as well to know what is fully transpired. So the recommendations and the choices that the agents are making are more in line with what this user would want.

• The data model is extracted from Salesforce apps into the Data Cloud to add customer experience data and enable point-and-click data engineering

George Gilbert So would it be fair to say that by taking the data model out of the Salesforce operational apps, and I don’t know if I’ve got the terminology quite right, but if there’s an enterprise graph and a data graph, if I understand it, the data graph is the data model for a particular customer, then the scaffolding on which, in the Data Cloud, you add additional elements to the data model related to engagement. So this is where you get the rest of the Customer 360, even if it’s from external systems. The reason I’m bringing this up is by leveraging the data model from Salesforce and then adding some additional customer experience related data model, then all the data engineering that requires so much heavy lifting in the form of code heavy pipelines today becomes a point and click configuration driven process. And so then it’s much easier to bring in just the incremental stuff that adds to that data model. Then you become gravity that attracts the additional stuff.

• Salesforce’s Customer 360 provides the fullest view of the customer by supplementing Salesforce data with external engagement data in a harmonized way

Vijay Pandiarajan I would agree with that. The point of it, I would say, the customer is the customer at the end. There’s nothing that’s really different. It’s just that within Salesforce, we have a view of that and there’s a lot more that’s happening with them. Really all we’re trying to do in the Customer 360 is that you get the fullest extent and the fullest view of that customer as possible. So you’re right, some of this is taking the entities that we have, supplementing the information that we can get from other systems, and importantly harmonizing that over what we already know. And so the data graph gives you the full picture of what has happened. And the minute we get some other engagement data from different system, that overlays it and each bit actually adds another layer. It adds another layer to the fabric that gives you a better and fuller picture of what has happened with that customer.

• The Customer 360 data model can evolve without breaking the no-code/low-code tools that help populate it

George Gilbert So I know we’re now in the foundational data layer. But to be clear, as you’re adding that additional data, the 360 data model can evolve without breaking the no-code, low-code tools that help populate it?

Vijay Pandiarajan Yeah, and that’s one of the features that we have. One of the most popular ways that people interact with Data Cloud today, that use Salesforce flow, is with what we call the Data Cloud-triggered flow. And so if you’ve written triggered flows at all before, a change in the Salesforce database, there’s a change in the lead and something changes, then you want the salesperson to take some action on it, you would write a triggered flow. We wanted exactly the same interaction model to be available to an admin if there was a change in Data Cloud data. Why would we not? We’ve got people really knowing how to act on these changes in the Salesforce data. You can do exactly the same thing on Data Cloud data that’s changing as well. And you can write yourself a Data Cloud-triggered flow and you can make whatever actions that are appropriate when something in Data Cloud has changed. And of course, the purview and footprint of what is in Data Cloud is much larger than what we would have in the traditional Salesforce data model.

George Gilbert Okay. Okay. All right. So we’ve been talking about the foundation that enables agents to have workflows as tools that it figures out how to combine and compose to accomplish a task, and Data Cloud as not only the contextual data foundation, but the harmonized modeled foundation so that it understands the semantics of the data. Now, we need to get to the level… We’ve basically been talking about one or a small number of agents accomplishing a workflow or a sequence of tasks. So now let’s get to the point of how do you align individual agents with larger corporate objectives or constraints, and how you might improve the efficiency of the entire system of agents and processes. And this takes us to the analytics layer,

which I think we’ve put so far, at least on this slide, on process management and observability. And I don’t know if this is the quite right layer to put it in, but we have to look across really not just the behavior of the data but of the agents themselves. So maybe explain how you can align the goals and the constraints of individual agents and their workflows with the overall goals and objectives that you might set for a larger organization or an entire enterprise. And maybe do that with an example.

• Analytics on execution data provides insights into process efficiency and alignment with overall objectives

Vijay Pandiarajan Yeah, no. One of the really interesting things is the efficiency in which work gets done, whether it’s with people or whether it’s with agents. I really think about them as processes with people, systems, and bots, and really we’ve got all of those functioning here. One way to look at the overall effectiveness of a particular process, we gather the analytical information about the execution of these things. The auditability, the trace logs, all of that. We have all of that information. We have the information that’s in the Customer 360 about what the customer has done as well, and the data graphs that are inside Data Cloud. Process mining becomes a really interesting way for us then to look at how are these things transpiring and what is the most efficient way in which these elements can be brought together.

We haven’t said much about that top layer, what we’re calling orchestration, but really a lot of these end-to-end experiences start with defining what an orchestration should look like. And then that’s when these people, systems, and bots are actually working together. We now have the analytical information coming out of that system. And then process mining lets us go back and see how closely were we aligned to what we had initially set out. What is happening with our overall orchestration? Are we actually hitting the goals and the targets that we had? Both in terms of… It could be something like a quality of service. Are we providing the right quality of service, and are we hitting the SLAs that we want? If not, why? And then process mining lets us get into it and understand what is standing in the way of that and what should be changed in the overall orchestration. So I think that’s one of the ways in which you can sort of round trip and look at, are you actually meeting the objectives that you had set out when you designed the system to be how it is.

• Process mining above the orchestration layer provides global system intelligence to improve orchestration and individual agent design

George Gilbert Okay. So in other words, just conceptually, you might think of actually process mining above the orchestration layer, because that’s the global system intelligence. And that would inform how to improve orchestration in the form of how you’re sequencing or even how you’re designing individual agents in the future. Is that a fair way to recap it?

Vijay Pandiarajan Yeah, 100 I almost see something going from analytics as a full loop back into orchestration. And I would sort of put process mining on that and that becomes your full loop.

Okay. And we’re doing this all the time.

Vijay Pandiarajan Customers are doing this all the time

George Gilbert to ensure that they’re staying ahead. Classically, you would call this process improvement. And now we have a very precise way to come at it, and a very data oriented way to come at it to make those improvements.

• Analytics partnership with Apromore for process mining insights

George Gilbert Yeah, you told me so far there’s an integration with an analytics form Apromore.

• Analytics partnership with Apromore for process mining insights

Vijay Pandiarajan Apromore is a business partner we have with the tools to do the process mining for us. We look at the data exhaust that comes out of our execution logs. It gets auto tagged to come into Apromore. And then Apromore is able to come back through, help us find the places where we’ve got the process blockers, understand what those blockers are, what’s causing it. We don’t have enough, I don’t know, approvers on a particular step in the process. Or something else. We can really decipher if it’s a resource constraint or if the process just needs to be redesigned. It could be that a particularly expensive resource is being unnecessarily used, or there could be some additional steps to streamline that. And it comes back with proposals on how the problem could be solved.

• Simulations using historical data to validate proposed process changes before implementation

The cool thing after that is we can run some simulations on it using the data that we have to say, “Hey, had we made this change, what would it have looked like? ” And so then you can have a lot more confidence that the proposed change that you’re going to make is actually going to work. Then of course you’re also constantly tracking it once it goes live to make sure that that’s what’s happening or if something else is actually starting to become the root cause of the problem.

• Future vision: Flows composed by agents include models prescribing actions, with feedback improving models

George Gilbert Okay. So let me try and now step a little bit into the future, and let’s talk about what this might look like. And you tell me how far out this might be. Now, imagine that these flows that are being composed by the agents include models that themselves are prescribing a certain type of action. It’s almost like there’s a planning step within the flows, and I’m sort of putting this together on the fly here, but the idea that because this is Salesforce, these models can be connected to business outcomes in the Salesforce operational apps and potentially even through the integrations and connectors via actions to outcomes in external apps.

And that that feedback can improve the individual models in particular workflows. And then as you have a composite of agent flows, multiple agent flows working together that you are orchestrating and then process mining, you start to be setting up a flywheel where you’re learning at the individual task level and you’re learning at the larger process level. And so then you’ve got this flywheel where your goal as a manager is to architect and orchestrate and optimize this overall system. And it’s not autonomous, but it’s embodying…

Like when we talk about managers building processes, in the past that meant designing organizational processes. Now, you’re embodying more of those processes into software artifacts. Help me flesh out that vision and what that might look like, and how far out it is.

• Much of this vision is technologically feasible today; the key is keeping humans in the loop

Vijay Pandiarajan It’s not as far out as we think it is. I think that’s what’s become really clear in the last 12 months or so. I think a lot of it, in terms of the technology, is probably here. We have all of the elements in place. It’s about sequencing them as we’re exploring and taking the next steps. The biggest thing, in my view here, is how do you keep the humans in the loop as you go ahead and do this? How can you be autonomous and take guidance at the right moments? It’s almost like when you hire a new employee. Some of them are just fabulous in that they’ll go out and figure out what they have to do and they’ll come back exactly at the time when they need to take input. And you want them to be proactive like that, but not also go off of the deep end. I see this as exactly something similar to that.

We can sequence a number of these things, put them into place. And then we need to be able to have the right checkpoints to know that it’s moving in the right direction. And I think that’s one of the things I’m personally really interested in, this human agent interface and this human agent evolution, because it’s not going to be the human by themselves. It hasn’t been for a long time, right? So that’s not particularly revolutionary. But this close interaction that you’re going to have with the agents that you’re working on and then the overall outcomes that you are getting to. I think the combination of those things is going to be really interesting.

• Two levels of improvement: analytic models learning from outcomes, and agents’ plans iteratively refined by human feedback

George Gilbert Okay, so there’s actually two things here. You brought up something that I left out, which is it’s not just the analytic models that are learning from the outcomes, it’s the plans that the agents generate, that the human manager supervises and might tweak or improve so that the next iteration of the plan is either more reliable, or more efficient, or more aligned after the analytics, with the overall objectives. So there’s two levels of improvement. The analytic artifacts are collecting feedback from business outcomes may be related to the operational apps, but then there’s also improvement going on in the planning of each individual agent. Is that a fair way to characterize it?

Vijay Pandiarajan Oh yeah, I think so. I think it’s definitely happening at both levels. If you think about individually, am I getting… The level of feedback could vary, right? Did the logistic agent come back with three viable plans or was one just ridiculous? And I would give it feedback saying, “Look, these two make sense and this one doesn’t,” right? So the feedback doesn’t have to be necessarily super detailed. It could just be… It’s kind of like marking spam in your inbox. You’re saying, “Okay, this looks like spam. That doesn’t.” That’s very black and white. Here, you’re getting into a little bit more nuanced area about was this useful to you or was this not? And I think some of this is how do you take that kind of input and get better at generating the things that are useful to you. I think this idea of going qualitatively, understanding the usefulness or applicability of the recommendation, I think is going to be one of the ways in which you’re defining success and that you’re getting better at being able to quantify these more abstract notions.

• Evaluating agent recommendations is a domain-specific task, providing rich feedback to improve the models

George Gilbert So let’s touch on this. And this may be again, futures, but the recommendation that let’s say an agent comes back with is the result of its execution of a plan. Now, let’s say there’s some user experience that still has to be designed or that’s in the process of being designed for how a user now interacts with an agent to essentially grade or refine its plan or its recommendation. The plan that got it to the recommendation or the recommendation itself, essentially it’s an evaluation function for the agent. And then how that might be incorporated so that that improves the agent’s planning, reliability, and efficiency in the future. Might there be some admin task that collects the successfully evaluated plans, or the positively evaluated plans, then maybe generate synthetic examples, multiple synthetic examples of those, and then does a fine-tuning task that maybe updates an adapter in the model associated with the agent’s planning engine.

Vijay Pandiarajan I’ll tell you, we’re grappling with that today. When we’re talking about creation, we talked about Einstein for Anypoint Code Builder and Einstein for Flow. Think about what’s happening there. It’s an agent, it’s building something for me. I’ve asked it to get the inventory data from SAP and merge it with what’s in Salesforce. I want to flow for that. It’s generated something for me, I’m going to look at it. I’m going to give it feedback and say, “Yeah, that did the job for me.” Or if it’s close enough, I’m going to make a couple of changes to it. And all of that is a learning opportunity right there. One is direct feedback, this worked, it didn’t work. The next bit is more nuanced feedback. 70% was good, I’m going to make the next little bit of the change and then I’m going to use it. I think just the fact that I made changes and then used it is actually another piece of training information as well right there. So to some extent, all of that future stuff, we’re seeing some of those elements right now. It’s just only going to get more refined, and every interaction that you have with that agent in evaluating its output is going to become valuable information to make it generate something more useful for you.

• Salesforce’s approach is use case specific evaluation of agent generated plans to improve the model, not just QA

George Gilbert Okay. So this is critical, because right now, there’s a whole lot of discussion about the difficulty of evaluating models, especially the generative models. And part of that difficulty is it’s a lot of general purpose benchmarks, whereas we’re talking here about use case specific evaluation. And unlike traditional software validation, this is not about QA. This feeds directly back into the fine-tuning to make the next iteration of generated plans, again, more reliable or efficient. And I assume you’ve got some sort of loop where evaluation isn’t about grading the model, it’s about grading the plan to improve the model. Is that a fair assessment?

Vijay Pandiarajan Yeah, 100%. I think you’re still grounded in a domain. And so because you’re in the domain, you have some qualitative assessment of how good the returned information or the return generated artifact is to you. And that can be either a discrete yes, no, or it can be a more analogous, yeah, pretty good, close, but not quite, here’s the difference. And capturing that and then funneling it back, doing the full loop through is what’s going to make this even more effective. And I’ll say the more specific the domain is, the better you’re going to get at getting to the answer that you want. I think the wider it is, it becomes much more complicated. So scoping it down to something like inventory or a very specific task, your chances of success here are going to go up much more quickly.

• Enterprise agents are more tractable than open-ended agents due to harmonized data semantics and rich, domain-specific evaluation signals

George Gilbert Okay. So you’re actually identifying additional things that I didn’t say up in the beginning about why enterprise agents are a more tractable problem than open-ended, open world agents. Which is, it’s not just that you have the state of the business and the semantics, or the state of the business in the form of harmonized data objects that are stateful and evolving with the business process, but that you have an evaluation function for each of these agent flows that is domain specific and therefore has a rich signal for feedback.

Vijay Pandiarajan Yeah, 100%. I think it comes down to expectation, right? There’s an expectation in a particular domain of a certain outcome, and there’s also the level of preciseness that is needed. I would say certain things require a high level of precision and some other things less, and then the human is happy to fill in some of the gaps. So I think you’ve got to understand that domain and you got to understand the variance that you can tolerate. And again, at the end, it’s the usefulness, right? Maybe I haven’t solved it 100% as an agent, but how close did I come and how can I become better at solving the need that’s there? But it comes from that nuance of the expectation, being able to understand the expectation and then being able to close the gap on it.

• In the next 1-2 years, expect better tools to build agents and more agents built into Salesforce for previously manual tasks

George Gilbert Okay. We’ve covered a lot. Any parting thoughts on what our expectations should be when we look out a year or two, about what capabilities might look like then?

Vijay Pandiarajan I think it’s a really exciting time to be in this space right now. I think everything from creator productivity, the way we are creating these agents, and in the ways in which these agents are going to get used in the end outcome, I think it’s going to change a lot of professions really, really fast. I think the important thing is that these tools are provided inbound and in the places where people go to do this work today. So if you have integration and automation tools, we want to make them available right there. And especially in those places where it’s not completely discreet. Like the place where the person, the human can do the last mile part, I think that’s where the productivity is going to come from. I think this is all about human productivity at the end.

And then having these tools just directly available where these people work is going to make a huge difference. And for us at MuleSoft and in Salesforce, that’s the game we’re in. We want to help people be more successful at what they do, and this technology will help us get there. And I think the next 18 months is going to be a pretty exciting time.

George Gilbert So inferring from what I’m hearing you say, perhaps over the next 18 months, we won’t just see better tools for customers to build their own agents, but we’ll see more and more agents built in to Salesforce to accomplish tasks that used to be surfaced through the user interface.

Vijay Pandiarajan 100%. We have a Salesforce Copilot, and we’re all in on making it really, really good and expanding all of the things you can do within the Copilot. We talked about Invocable Actions earlier. A number of the things that we have here are all going to be available. Today, Invocable Actions that are available in Salesforce Copilot. I just think that the quality and the reach and the kinds of things that you can do with these agents is just going to get so much more powerful. It’s all going to come to you within the Salesforce Copilot interface.

George Gilbert Okay. Vijay, that was a wonderful tour of what’s coming. And it was more than just about Salesforce, because we can generalize that to understand agents in the enterprise. And so with that, let’s call it a wrap for today. And of course, we’ll have to have you back to give us an update on the progress and what the future looks like when we look out in a few months. So thanks for joining.

Vijay Pandiarajan Thank you for having me. I’d love to be back. Lots of new things out there, and can’t wait for customers to get their hands on it and make progress in their business.

George Gilbert Okay. Thanks, Vijay. Thank you, George.