Andrew McNamara, Shopify: Why Orders From AI Are Up 13x

Watch on YouTube · Listen on Spotify or Apple Podcasts

Andrew McNamara, VP of Applied ML at Shopify, says AI agents are becoming the new front door to commerce, and that orders reaching Shopify stores from AI are already up 13x. Back for a second conversation with host Liam Lawson, he explains the Universal Commerce Protocol, Shopify's personalized shopping agent in the Shop app, how Sidekick works, and how SimGym uses AI shoppers to test store changes before they reach real customers.

Key takeaways

• Orders from AI are up 13x. Orders coming to Shopify stores from AI are up 13x, and catalog-powered AI search converts about two times better than general AI search. (11:53)
• AI commerce is outpacing social and mobile. Andrew says AI commerce is growing about nine times faster than social commerce and nearly three times faster than mobile did at the same stage. (11:53)
• Agents grow the pie. Shopify sees agents as a new front door to commerce that adds shopping rather than replacing other channels, so it wants merchants to be able to sell everywhere. (01:47)
• The Shop app agent beats traditional search. Shopify's personalized shopping agent has a higher conversion rate than traditional search, and shoppers move from keywords to natural language once they discover it. (06:08)
• Shopping data stays with the shopper. Data from the Shop app agent is used to personalize that individual buyer's experience, much like memory in a personal agent. (09:51)
• Merchants opt in to UCP. Merchants enable a sales channel to expose their catalog to agents that support UCP, then track AI orders, queries and conversions in the Shopify admin. (13:35)
• SimGym tests store changes with AI shoppers. Merchants can run ideas past simulated shoppers before spending live traffic on an A/B test. (37:22)

How are AI agents changing online shopping?

Andrew McNamara says shopping is fundamentally changing, and agents are becoming "the new front door to commerce". People went from retail stores to desktops, and the ways they find things to buy keep multiplying. He dates the latest shift to December, when updated Opus models changed how people used AI for coding, and says the same shift is now reaching shopping.

In Shopify's view, a platform shift like this grows the pie rather than taking shopping away from other channels. More shopping is happening, and a lot of it in agents. Shopify's goal is to be everywhere, so that merchants can sell on any platform in any way.

What is Shopify's shopping agent in the Shop app?

Andrew McNamara describes it as a very personalized shopping assistant inside the Shop app that learns about you as you shop: the products you like, how you buy and your budget. It connects directly to the catalog, shows products on screen and lets you ask about them. It also learns that you shop for different people, such as a dog, a cat or your children.

The agent has a higher conversion rate than traditional search. Once shoppers learn an agent is there, their queries shift from keywords to natural language, and they research more, looking at many products quickly. As Andrew puts it, "people's behaviors are shifting to heavy research when shopping."

How does Shopify use shopping data from its agent?

Andrew McNamara says Shopify uses the data from a shopper's conversations with the agent to personalize that shopper's own experience, similar to how memory works in personal agents like OpenClaw and Hermes. It learns, for example, that one person wants to scroll through many pairs of shoes, while another buying a motherboard has a precise need, and Andrew himself just wants the best waterproof hiking boot.

Asked whether the data is used to train a larger model, Andrew says the team has not had discussions about training a large model on it, and that using it to benefit the individual buyer is what matters.

What is the Universal Commerce Protocol (UCP)?

The Universal Commerce Protocol is a standard for exposing shopping platforms' catalog APIs to AI agents, Andrew McNamara explains. He says it was agreed by a group of e-commerce companies, including Meta, Etsy, Shopify and Mastercard. Shopify's catalog API follows UCP, and merchants who enable the sales channel can have their products appear in agents that have integrated it.

The numbers behind it are large. Andrew says AI commerce is growing about nine times faster than social commerce and nearly three times faster than mobile did at the same stage, and he calls it another generational shift. In Shopify's data, catalog-powered AI search converts about two times better than general AI search, and "orders coming to Shopify stores from AI are actually up 13x."

Merchants get dashboards in the Shopify admin showing orders and conversions from ChatGPT, Google AI Mode, Gemini and other assistants, the top AI queries leading to sales and how they rank for them, plus suggestions when products appear in AI conversations but do not convert. They can use Sidekick to optimize their exposure. Andrew has not heard merchant complaints, which he puts down to the channel being optional. He also sees developers building niche shopping apps, for example for fashion, on top of the catalog API.

How do personal shopping agents use UCP?

Andrew McNamara says many Shopify employees run personal agents on OpenClaw or Hermes, and Shopify CEO Tobi gave his Hermes agent access to the catalog through UCP, a budget and a skill. Tobi says he now periodically receives gifts in the mail from the agent, which he still pays for through its budget. "I know we all wish we could have a personal shopper, but I think agents make that very accessible for us."

Andrew's own habit is to photograph tools and other things he sees during the week. Because the catalog API supports image search, he can hand those photos to his Hermes agent, ask it to research the items and find the best place to buy them, and get an answer the next morning. He still makes the final purchase himself rather than letting the agent ship things automatically.

What are Sidekick app extensions?

Sidekick app extensions let app developers plug their apps into Sidekick, so it can fetch data from an app, update fields and fill forms for the merchant, Andrew McNamara explains. Shopify announced them at its December Editions, opened them to select developers in March and launched with about twenty partners, including Judge.me, Klaviyo, Loop and Matrixify.

He calls the design "MCP shaped" or skill shaped: Sidekick searches for apps relevant to a request, asks the user to choose if there are several, and uses the app if there is one. Merchants can switch each integration on or off. It was one of the most requested features from developers and merchants since Sidekick launched.

What is Shopify Sidekick and how does it work?

Andrew McNamara says "Sidekick is basically an AI co-founder there for you." New merchants use it to brainstorm a business, define a product vision and create themes, while large merchants rely on it for deep analytics in natural language. The idea came from a Shopify survey: merchants who had an entrepreneur in their life to ask questions had a much higher success rate a year later, so Sidekick aims to be that expert, without judgment.

Under the hood, Sidekick is an LLM running in a loop that calls tools and uses skills. It currently runs on Anthropic's Sonnet model, hosted on Google Cloud, which Andrew says is good at agentic tool calling and conversation, though the system can be model agnostic. Its tools include a knowledge base of how to use Shopify drawn from the public help center and Shopify blog posts, natural language to GraphQL for working with a store's data, and analytics.

What is Campaign Autopilot's auto research loop?

Campaign Autopilot runs what Andrew McNamara compares to a Karpathy auto research loop: the merchant sets a budget and guardrails, and the system keeps trying things, measuring what works and iterating, specific to that store and its campaigns. He says it does not draw on other merchants' results.

He explains the general idea with model training: an LLM runs an evaluation against a metric, looks at the errors, finds problems such as mislabeled training data, makes changes, retrains and evaluates again, continuously. At a high level, he says, that is similar to reinforcement learning, except that the LLM itself can suggest the next change.

What is SimGym?

SimGym is a Shopify product that trains AI shoppers to test changes to a store, Andrew McNamara explains. A merchant could send, say, 300 simulated shoppers to complete a task on one version of a store and 300 on another, to compare changes such as moving a button or turning reviews on or off. Shopify validates the simulated shoppers by checking that they reproduce what actually happened when stores made changes in the past.

Because live A/B tests are expensive and bad variants can lose sales, Andrew suggests that "you can actually run all your ideas with SimGym, maybe take the top two and then do a live A B test." He has also run it as a research loop on his own maple syrup store.

How does Shopify decide what to build?

Andrew McNamara says there is no single answer. The team tries to get ahead of the curve and build things people are not yet asking for, while using Twitter and community feedback to learn what it got wrong or did not think of, and making sure Sidekick covers new features across Shopify. Looking ahead, he points to big improvements coming to Sidekick for Shopify's 250 million verified shoppers and millions of merchants.

In his words

"orders coming to Shopify stores from AI are actually up 13x." (11:53)

"people's behaviors are shifting to heavy research when shopping." (06:08)

"I know we all wish we could have a personal shopper, but I think agents make that very accessible for us." (15:55)

"Sidekick is basically an AI co-founder there for you." (30:00)

"you can actually run all your ideas with SimGym, maybe take the top two and then do a live A B test." (37:22)

Chapters

00:00 Introduction and welcome
00:29 What's changed in AI and shopping since their last conversation
01:47 Agents becoming "the new front door to commerce"
04:16 Inside Shop app's personalized shopping agent
07:32 Why data stays personalized to each shopper instead of training a larger model
11:53 What the Universal Commerce Protocol (UCP) is, and orders from AI up 13x
14:58 Merchant tooling for tracking AI-driven traffic and conversions
15:55 The story of Tobi's Hermes agent sending him gifts in the mail
20:48 Andrew's own habit of shopping by taking pictures and searching by image
26:59 Sidekick's app extensions and partner integrations
33:02 Inside Sidekick's architecture: the Sonnet model and knowledge base
35:18 Campaign Autopilot's auto research loop
38:58 SimGym: training AI shoppers to test store changes
42:23 What's next for Shopify's agentic commerce features
44:17 Where to find Andrew

About the guest

Andrew McNamara is VP of Applied ML at Shopify, where he works on Sidekick and Shopify's AI agents. This is his second appearance on The AI Report podcast. Andrew on LinkedIn · Andrew on X

About the host

Liam Lawson is CEO of The AI Report and host of The AI Report Podcast. Meet the team

Full transcript

Edited for readability. Quotes above are word for word from the recording.

Liam Lawson (00:00): All right, Andrew, welcome to the podcast.

Andrew McNamara (00:04): Thanks for having me back.

Liam Lawson (00:06): When did we record, three or four months ago, something like that?

Andrew McNamara (00:09): I think so. Four months ago, probably.

Liam Lawson: It seems like a lot has changed in that time. We were having a bit of a preamble before this. There are a lot of new features, specifically within Shopify, and we were talking about the larger macro ecosystem of what agents are now enabling as well.

Andrew McNamara (00:29): Yes. Even in general, in the last six months, since December, when the significantly updated Opus models came out, we've had a huge shift in how people were using these models for coding and development. Now we're seeing that same shift hit shopping. So it's a totally different world from the last time we chatted.

Liam Lawson (01:02): I want to know more about that from your perspective. For me, the word of the moment for the past six months is agents. I was actually just down in Florida doing a presentation for an HR conference on everything that's possible with agents. People have many different experiences. You could have an OpenClaw or a Hermes agent, and I know you've got some personal anecdotes about this. You can also have agentic experiences within Claude or OpenAI and the different tools you're using. From your perspective, on the Shopify side, what are you noticing in terms of agents and how they intersect with commerce?

Andrew McNamara (01:47): A couple of the core shifts we're seeing are that shopping is just fundamentally changing. We went from retail stores to desktops, and the question is what the surface of tomorrow is that people are shopping on. The ways people are finding things to buy are just multiplying. Agents are a huge part of that. It's like the new front door to commerce, almost. It's been crazy.

One way we think about it is that any time there's a big platform shift or a new way of doing things, it really grows the pie, so to speak. We don't see other shopping services go away. What we really see is that shopping is now available in agents, everyone is using agents, a lot of shopping is happening in that surface, and more shopping is happening overall. So we just keep building for everyone. We want to be everywhere, so that when our merchants want to sell on any kind of platform in any kind of way, we're right there for them, at the head of that curve.

Liam Lawson (03:17): It's interesting. Yesterday I had a conversation with David McIntosh of Instacart. They work with a bunch of retail stores, and they now also have a physical cart enabled with AI that gives you a running total and automatically counts your shopping. They're now integrating a bunch of agentic features as well. You can walk into a store and say, I've got $30 to spend, I want to make this dinner, but my first child is allergic to this and my second child has this problem. Through these agentic experiences, you get a level of convenience and ease that wasn't possible before. I want to know from the Shopify side: how are you enabling these agentic functions for your customers and users? What does that look like?

Andrew McNamara (04:16): There are a couple of different audiences here. There's the buyer, which you were talking about. There's the merchant: how do we do the best thing for them and get them into all these services with as little work as possible on their end, and how do we help them optimize for this and capture the data from it? And then there are buyers who are making their own agents.

One product we shipped recently, inside the Shop app, the purple shopping app, is a very personalized shopping experience and agent directly inside that app. There has always been search functionality in the app, but a very personalized shopping assistant is now available that knows you and learns about you as you shop: what type of products you're interested in, your style of purchasing, your budgeting and so on. It provides a pretty incredible experience, an agent integrated directly with the catalog, showing products right there on the screen. You can select them and ask more about them.

It also learns about different personas. Let's say you shop for different family members. It's going to learn from that: okay, you're shopping for your dog, or you have a cat, or you have children. As you search, it learns and optimizes for who you're shopping for, because not everyone shops for just one person.

Liam Lawson (06:01): What have you noticed in terms of customer feedback?

Andrew McNamara (06:08): We're seeing a higher conversion rate on this than on traditional search, so this is a surface people are using more and more. We're also seeing people's search queries shift from traditional keyword-type queries. Once they learn there's an agent there, their behavior shifts to more natural language queries. We saw this on search engines over a period of time, but it's happening in shopping too, and people's behaviors are shifting to heavy research when shopping: looking at many products quickly and generally doing a lot more research, which leads to higher conversion.

So the feedback is very positive on how accessible it makes shopping and on the improved relevance from the personalization, really getting to know you as a shopper. The more you use it, the more it adapts to you and how you search, and so people come back and use it more and more, because everyone searches differently.

Liam Lawson (07:32): It just becomes more personalized over time. Where my brain goes with this is the data side. You're integrating a feature that personalizes shopping and understands shopping behaviors, customer behaviors and customer insights, and data in general is one of the biggest moats you can have in 2026. You're building an incredibly large dataset of customer buying behavior, potentially one of the largest. How does that work internally? What do the conversations in the executive rooms look like when you're talking about the data you're collecting?

Andrew McNamara (08:18): Our play on personalization is that we use the data from your use of the agent to personalize your shopping experience for you. When some people are browsing, say for shoes, they want tons and tons of shoes. They want to scroll through very quickly, and we can learn that behavior from them using it. Other people might be buying a motherboard or something with a very specific need. You're not going to scroll through a whole bunch of motherboards based on pictures. There are just different kinds of shoppers. When I'm looking for a boot or a shoe, just show me one or two options. I want the best ones for hiking, being up north fishing, being out in the bush. I just want the best waterproof hiking shoe. I don't want to flip through many pictures. So for me, the agent learns that I'm heavy research based and want the exact product, while for someone else it might learn to show them tons and let them scroll. That's how the data comes back and gets hyper-personalized for each individual.

Liam Lawson (09:34): That's how it looks from the customer experience perspective, but what does it look like from a business standpoint? What happens with that data? Are you training internal models? Are there plans to use it for future products? Because it's rich and valuable.

Andrew McNamara (09:51): Right now, for us, it's like how memory would work on OpenClaw or Hermes. We're using it just for users, and it's used for that individual user. How we use this data is important, and using it to benefit the individual buyer is what we're using it for.

Liam Lawson (10:21): So it's not being stored as part of a larger dataset. It's local to that customer.

Andrew McNamara: That's right.

Liam Lawson: Interesting. Is that out of some sort of feedback from customers around data sovereignty? Why that decision? As a business owner, I look at what gives businesses a moat in 2026, when the SaaSpocalypse is a thing, and data is one of the largest things. So I'm curious why the executive team made that decision.

Andrew McNamara (10:53): The decision on using data for personalization?

Liam Lawson (10:56): Yes, or using it only for personalization and not to train a larger model of your own.

Andrew McNamara (11:05): We haven't had discussions on training a large model here.

Liam Lawson (11:31): We can move on to something else. Agentic commerce: talk to me about UCP. I'd heard of it before, and we talked about it in the preamble, so I'm curious if you can break it down for us so we can really decipher what's going on here.

Andrew McNamara (11:53): On the AI commerce side of things, it's growing about nine times faster than social commerce and nearly three times faster than mobile did at this stage. Mobile and social both changed the way a lot of people shop and reshaped commerce, and I think we're in another generational shift in how people shop, and that's agents.

UCP is the Universal Commerce Protocol, a protocol made by a whole bunch of e-commerce companies, like Meta, Etsy, Shopify and Mastercard. There's a whole group of companies that agreed this was a great protocol for exposing catalog APIs for the different shopping platforms. In our data, we see that catalog-powered AI search converts about two times better than general AI search. And orders coming to Shopify stores from AI are actually up 13x.

Liam Lawson (13:10): Sorry, to break that down from a user experience perspective: that's someone typing into ChatGPT or Claude or Perplexity, I want to find boots for fishing in Northern Canada in November, and it links through to a specific Shopify store.

Andrew McNamara (13:25): That's right.

Liam Lawson: And the increase has been enabled by the Universal Commerce Protocol. Can you break down what catalog APIs are?

Andrew McNamara (13:35): A catalog API is an API that follows the Universal Commerce Protocol, and it basically exposes the Shopify catalog. As a merchant, you have the ability to enable this sales channel. If you've enabled it, your products may start to be exposed in agents that have integrated UCP.

Liam Lawson (14:06): And the logic behind it is that for a frontier model, because the catalog is built on UCP, there's more inherent trust. When the AI ranks all these results for what's most trustworthy, because it's built on this and attached to Shopify, it's more likely to recommend it.

Andrew McNamara (14:30): That's right. And we've built dashboards and tooling inside Shopify so that, as a merchant, you can view this data. You can even use Sidekick to help optimize your exposure. So there's a lot of tooling on the merchant side to help understand what's happening.

Liam Lawson (14:58): So, for example, understanding where I rank for specific search terms within a specific AI model.

Andrew McNamara (15:07): Exactly. They can see orders, sales and conversions from ChatGPT, Google AI Mode, Gemini and so on. They can see the top AI queries leading to sales and what they're ranking for on those queries. When products show up in AI conversations but aren't converting, we may have some suggestions on what to fix. So there's lots of tooling inside the Shopify admin for this.

Liam Lawson (15:44): We were talking before, and you mentioned someone at Shopify building their own Hermes agent to get free stuff sent to their home. Explain that for me.

Andrew McNamara (15:55): A lot of Shopify employees are using OpenClaw, or maybe Hermes more recently. Tobi tweeted last week, or a couple of days ago, about his own personal agent. I guess one of them is running on Hermes. He gave it access to UCP, so it can access the catalog from Shopify and other places. He gave it a budget, and I guess created a skill around it, and now he says he periodically receives gifts in the mail from his Hermes agent. It's pretty cool what all of this enables, especially as more and more people have their own shopping agents. It's like your personal AI shopper. I know we all wish we could have a personal shopper, but I think agents make that very accessible for us.

Liam Lawson (16:51): So are the gifts in the mail from free trials the agent signs up for and then cancels? How exactly is he getting free gifts in the mail?

Andrew McNamara (17:00): They're gifts to him, but he's technically still paying for them. His agent is paying for them with a budget the agent has, so it's a gift to himself.

Liam Lawson (17:04): Okay, I've got you.

Andrew McNamara (17:21): But his agents, and all of our agents, learn about us. They know everything we do, not just shopping, but how we run our lives. They help us across the board. So now you can very easily give your agent access to Shopify's catalog, and then it's up to you. You can give it a budget, or have it be proactive and suggest products you might like, and you can build your own experience around how you want to make those purchases. Proactivity is such an important thing as we go into the future, and proactive shopping is a very cool thing, I think. This enables that.

Liam Lawson (17:59): There are obviously a lot of opportunities here. You have Samsung's smart fridges, for example. You can be proactive with these agentic functions, like ordering food in when you're running low on olive oil. There are a bunch of things that could be done. We've been talking from a business-to-consumer standpoint: a merchant on Shopify selling their products wants to rank higher so consumers can find them through AI. Have you thought about the B2B functionality of something like the Universal Commerce Protocol? I wouldn't even know where to start, but there has to be some functionality and use cases. Is there anything you've seen so far?

Andrew McNamara (18:52): Not that I've seen. But I know there are app developers building niche apps that want to create a very specific shopping experience in a certain industry. Instead of a global agent searching everything, somebody may create a niche fashion shopping app. Now they can build very specifically for fashion and use the catalog API just within that industry, so they can build a really niche product. Because this is just a tool available to developers, we see people who aren't even those businesses or merchants creating new apps that use shopping. Merchants benefit because they've chosen the sales channel, and developers benefit because they can create niche apps with instant access to different catalogs. So we're certainly seeing an uptick in these kinds of niche applications.

Liam Lawson (20:17): What are you most excited about personally? We spoke beforehand about the Sidekick app extensions, which essentially function like MCP. We're talking about UCP, agent-to-agent marketplaces and functionality, the agentic functions in Shop. You're super hands-on with this and already seem to be ahead of the curve, so what are you excited about in the future?

Andrew McNamara (20:48): Maybe I'm going to give an underwhelming answer here, and this is unique to me. One of my workflows for buying stuff in general is that I take pictures of things. I see something on TV or in person, or I'm in someone else's workshop and see a cool or useful tool, and I take a picture. Throughout my day or week I take a bunch of pictures, and a lot of my shopping is actually driven by referring back to them. One of the things I find very exciting about the catalog API is that you can also search by image. So not only can you search the UCP catalogs with image search, but as a developer and someone who has a Hermes agent, which I use a lot like a bullet journal, as well as for automated tasks and reminders, I can just dump in all the pictures I've taken and say, can you research these things, find the best place to buy them, and get back to me tomorrow morning. That's one of the huge unlocks I'm personally very excited about. I don't know whether it's just me who shops like that. I imagine maybe people go to stores and take pictures of dresses and so on too. It may not seem that interesting, but I think it's just so powerful. I don't know whether that just fits the way I like to shop and use agents specifically.

Liam Lawson (22:37): It's not something I do personally, but maybe that's because, like you said, you're in workshops quite a lot. I've used tools before, but there are probably some very niche tools where you think, I have no idea what that thing is, let me take a picture and figure it out later.

Andrew McNamara (22:49): Yes. I don't think I'm quite ready for the Tobi scenario of giving it a budget and having it send me gifts all the time, but it's been great to plug into UCP and have it send me recommendations, and then I do the final mile. I know some people just have it ship automatically, which is fantastic, that you can do this kind of thing in the world we live in.

Liam Lawson (23:26): Is there any pushback from your merchants on any of the agentic functions or the Universal Commerce Protocol? From my standpoint it seems like a no-brainer. Google is changing, search is changing, it's all very difficult, and it seems Shopify has merchants' backs: let's help your stuff rank, because the better your business does, the better our business does. But I want to steelman the case for a second. Is there any negative feedback? What are the common complaints, if any?

Andrew McNamara (24:01): I haven't heard any, and I think that's probably because merchants don't have to be on this sales channel. If they see the benefits, they enable it, and if they don't, they just don't. I don't hear many complaints, because the people who don't want to enable it just don't. Giving merchants that flexibility has reduced this.

Liam Lawson (24:32): Another thing we mentioned briefly before, in a similar vein: the Universal Commerce Protocol lets people access these catalog APIs, which reminds me of MCP. We also spoke about Sidekick app extensions, which from what I understand aren't MCP but function in a very similar way. Can you explain what's happening there?

Andrew McNamara (25:01): Back in our December Editions, we talked about Sidekick enabling app extensions, which allow app developers to plug into Sidekick. That covers fetching data that users ask for in Sidekick, as well as updating certain fields, basically using that application via Sidekick: filling forms, kicking into that app and so on. In March we enabled it for a select number of app developers, and we had around twenty partners at launch, which happened at Editions a couple of weeks ago.

The feedback has been very positive. Judge.me, Klaviyo, Loop, Matrixify, all these partners have integrated into Sidekick, and on Twitter I see more and more apps doing this every day. Since Sidekick launched, this was a very heavily requested feature from app developers, and from merchants actually, because they wanted the apps they've installed to be tightly integrated with Sidekick. They were seeing all the value of using Sidekick in general and asking why Sidekick couldn't know about this app and its data. So we've given app developers a way to plug directly into Sidekick.

Like you mentioned, it's MCP shaped, or skill shaped. Sidekick searches for relevant apps for any given user query. If there are multiple relevant apps, it disambiguates. If there's just one, it goes ahead and uses that app and presents the result back to the user.

Andrew McNamara (26:59): Users can also turn these integrations on and off, similar to MCP. It was a great ask from the community, and it took us some time to figure out how to do it exactly right, but we think we really nailed this one.

Liam Lawson (27:18): A lot of this conversation, you've spoken about feedback you see on Twitter or from the community. How do you and your team choose what to build? It sounds like you've got your finger on the pulse and you're ahead of things in enabling these functions. How much do you build from what you believe to be right, and how much from having had enough requests for something?

Andrew McNamara (27:55): There's no straightforward answer. We're certainly trying to be ahead of the curve: what do people want before they want it? What do we see happening, and how do we get ahead of it and build something really cool? How do we build something people aren't even asking for? That's a great way we think about it. At the same time...

Andrew McNamara (28:23): ...Twitter and the community are a great way to ask what we got wrong, or what we didn't think of, what we could be doing that we're not, or what we could be doing better. So that's an important signal as well. And there are things going on elsewhere at Shopify, new features being built, and how do we plug into those and make sure Sidekick has full coverage of the new features across the board. So there's no straightforward answer. It's a little bit of everything.

Liam Lawson (28:58): Who are Shopify's main competitors in this space?

Andrew McNamara (29:05): In which space?

Liam Lawson (29:07): In the marketplace space, Shopify's business in general. Who competes?

Andrew McNamara (29:14): Like the app marketplace? Or, sorry, are we talking about Sidekick now?

Liam Lawson (29:17): No, your entire business model.

Andrew McNamara (29:28): Okay, so we just jumped right out of Sidekick on me. That's a good question. I'm very much in the weeds on Sidekick and agents, so I'm not sure I have the best view or the best answer for this. But you want to talk agents, I'm your guy.

Liam Lawson (29:47): Okay, we can bring it back to agents. We mentioned Sidekick a lot there, and we spoke about it last time, but for people unacquainted with it, tell me a little more about what exactly Sidekick is.

Andrew McNamara (30:00): Sidekick is basically an AI co-founder there for you, right from the beginning when you start your store. We're finding it useful for small or new merchants. They use it to help create their themes and define their product vision and business vision. People come at all different stages. Some people start a Shopify store without any idea, and they can use Sidekick for brainstorming and discovering what kind of store or business they want to start. We also see very large merchants using it heavily for analytics, because you can get very deep insights from your store through natural language, which we're seeing is a huge unlock. Basically, you can do many actions on your store through natural language. It's just a more natural interface for people to do things, and as Sidekick's capabilities increase, we're seeing people run their business a lot through conversation. Sidekick is there to enable that.

One interesting story from the beginning of Sidekick: we did a survey at one point where we looked at merchants who had an entrepreneur in their life they could ask questions, and those who didn't. We surveyed them when they first started their store and a year down the road. The merchants who started their store with an entrepreneur they could ask questions had a much higher success rate than the ones who didn't. So part of the vision for Sidekick was: how can we provide that entrepreneurial expert for merchants, so they can ask questions as if it's a friend? There's no judgment when you ask Sidekick things. In some ways it's easier to have this entrepreneurial expert right there for you if you don't have that in your life.

Liam Lawson (32:24): Break that down a little more for me. We're talking about a co-founder or expert made of artificial intelligence. What does its brain consist of? Which LLM, if any, does it run on? Where's the data coming from? Is it gleaning insights from Shopify? Is it tied to larger datasets? Where does the advice come from? Can you break down the functionality of what makes a great AI co-founder?

Andrew McNamara (33:02): Sidekick's architecture is basically an LLM driving it, and you can think of the LLM as running in a loop, looping through different tool calls or using different skills. Right now we're running on the Sonnet model from Anthropic, hosted on Google's infrastructure, on GCP. That's the main model driving Sidekick right now.

Liam Lawson (33:37): Will that change? Can it be model agnostic?

Andrew McNamara (33:40): Yes, this can be model agnostic. We find Sonnet to be quite good for this use case: agentic tool calling and being conversational.

Liam Lawson (33:51): But that's not a user choice, that's from your side.

Andrew McNamara (33:55): That's right.

Liam Lawson: Okay, cool. Then talk to me about where these insights come from. Where does the knowledge base come from, and how is it updated over time?

Andrew McNamara (34:06): We have knowledge bases internally, basically a knowledge base of how to use Shopify. That's one tool, and I'll come back to it. Then there are tools for things like natural language to GraphQL, which is the underlying language of Shopify. That's what gives Sidekick a lot of its ability to work with and understand your store. There's also one for looking into analytics, and there are many, many different tools. The knowledge base for Shopify is also available publicly, on help.shopify.com. Sidekick is able to look through those documents, as well as Shopify blog posts we've written and so on. Shopify puts a lot of public content out there, and Sidekick can draw on it to provide its answers.

Liam Lawson (35:18): Here's where my brain goes with this. I don't think we've touched on it yet, though we did before the conversation: you have a new feature available, Campaign Autopilot. With all these merchants running on Shopify, you know which merchants are successful and how and why, and which aren't. Is that information being used to feed Sidekick and help it make the right decisions? If you've seen someone in your vertical spend their campaign budget a certain way, use a specific color and hex code in a specific format, and that worked and generated a lot of sales, does that feed in? Are you taking what's right and wrong from other stores and feeding it into Sidekick's brain, or is that siloed to each individual merchant?

Andrew McNamara (36:09): There are a couple of points to touch on here. It's not being used in that way for Campaign Autopilot. With Campaign Autopilot, you give it a budget and specific guardrails, and you can think of it like a Karpathy auto research loop. It's trying certain things, it's hyper-specific to you, it sees what's working, comes back and keeps iterating. It's a research loop for your store, for the exact scenario and the exact campaigns you're trying to run.

Then there's a product called SimGym, where we're training, basically, an AI shopper. Have you heard of SimGym before, or have we talked about it?

Liam Lawson (37:11): I think we spoke about it briefly. From what I remember, you can emulate buyer behavior based on the current conditions of your store. Am I along the right lines?

Andrew McNamara (37:22): Yes. We're training these AI shoppers so that if you show them one version of your store and another version, you can have, say, 300 of them try to complete a task on one version and 300 on the other. You made me think of this with the hex color button idea. If you want to move a button around, or enable or disable reviews, it's basically like running an A/B test without actually having to run a real A/B test with real traffic. If you have many ideas to try but you're not sure, it's expensive to run them on live traffic, and you may lose business on the bad ones. Instead you can actually run all your ideas with SimGym, maybe take the top two and then do a live A/B test.

The way we built this is that people have made changes to their stores in the past. When we create these AI shoppers, we test that, for past changes to stores, our AI shoppers predict the same result as what actually happened on the store. So these AI shoppers, or simulators, are learning to identify real issues that you can then use for A/B testing.

Liam Lawson (38:58): So you go...

Andrew McNamara (39:01): One interesting thing I actually ran on my own store, which is a maple syrup store, is that it's not only for A/B testing. You can also have it run as a research loop on your store, even on your current theme, in the same way as the campaign one. The Karpathy auto research stuff is all the buzz right now in the agentic space.

Liam Lawson (39:25): Can you explain that for the audience in more detail, if possible?

Andrew McNamara (39:29): The idea behind the auto research loop is that there's some metric you have. Let's talk about model training for a second. I want to optimize my model on some metric Y. Instead of me, as a human, analyzing outputs from the model and trying to figure out the best improvements to make to the data, say I'm training a cat or dog classifier and I want to optimize for that metric, you can have an LLM run tests on that metric and then look at the results. It makes suggestions for the next round of model training. Maybe it has identified a bunch of bad labels in your training data, cats that were dogs, or some other objects in there. The LLM can make those recommendations, make the changes on its own and run the scripts to retrain the model. When the model's done training, it runs the evaluations again, looks at them, finds all the errors, and comes up with ideas for how to retrain the model. So it's a loop where an LLM is continuously running, trying to optimize a specific metric. And you can use it for almost anything, not just model training.

Liam Lawson (41:12): Interesting. Are there a lot of parallels between what you've described and how many of these frontier models were developed in the first place, through reinforcement learning?

Andrew McNamara (41:25): There can be, yes.

Liam Lawson (41:27): Where's the difference?

Andrew McNamara (41:32): If you go to a super high level, it could be fairly similar. Reinforcement learning has some state space or environment where you can carry out an action, and there's a reward function: what did this action do, and is it good or bad? Then that signal goes back to the main model. With an LLM driving this loop, the LLM itself can make some of the suggestions, whereas in reinforcement learning you might be using human feedback loops, or preference optimization between two different outcomes. But at a high level it's pretty similar.

Liam Lawson (42:23): Let's flip things back to Shopify to finish off. We've covered a lot: Sidekick app extensions, UCP, agentic commerce, Campaign Autopilot and the agentic assistant in Shop. For a merchant, or for someone who uses Shopify stores, what should we be excited about? Is there anything you can tell us that's recently been made public, or will be by the time this episode releases in about eight weeks?

Andrew McNamara (43:07): I'm not sure I can say too much about upcoming launches, but I can say it's a very exciting time. Since December, agents have been going through a transformational stage in how people use them, both for shopping, which we talked about a lot, and on the Sidekick side. There are some big improvements we can make, and we're excited to share some of those changes. I don't know the timeline, but we're building some really cool stuff, and there's reason for the two hundred and fifty million verified shoppers on Shopify and the millions of merchants to be extremely excited about the future of what we're building.

Liam Lawson (44:05): Andrew, two things. One, thank you for your time. Two, if people want to find out more about who you are and what you do, how do they go about doing that?

Andrew McNamara (44:17): Reach out to me on Twitter, or tag me on Twitter. My Twitter is @DrewCH, D-R-E-W-C-H. Always happy to talk agents, Shopify and commerce.

Liam Lawson (44:32): Awesome. Thank you so much.

Andrew McNamara (44:33): All right, thanks for having me.

Get the five-minute AI briefing

Join 400,000+ business leaders who start their weekday with The AI Report. Subscribe free