The holiday season is around the corner, but letting AI place your orders may still be a ways away, according to a new benchmark from Visa.

Last week, the credit card giant launched its Agentic Commerce Benchmark, claiming to be the first of its kind in testing AI in realistic shopping scenarios and challenges. Visa notes that shopping presents unique challenges, as users don't always know what they are going to buy at the onset. This means that AI models involved in shopping are responsible for distinguishing when they should simply choose to not to buy something, or when it has enough information to justify its decision—a task the benchmark discovered they weren't very good at.

Though it didn't disclose which models were evaluated, Visa claims that the first round of tests involved 22 different AI models from 12 labs that took on 15 shopping challenges, including buying a TV, laptop or personal care appliance. The results overwhelmingly pointed to one outcome: The models are not quite ready for this task yet.

Across testing, the selected AI models achieved a benchmark pass rate of 57.2%. Other notable results included:

• None were perfect: The best performing models passed 82.2% of attempts, the worst scored 28.9%.
• Cost wasn't a differentiator: The five models with the highest average cost per run passed 67.1% of runs, compared with 50.2% for the five least expensive.
• Bad decisions: Around half of the failed runs, 54%, still ended in the AI making a purchase.

The five models that were able to understand a customer's shopping intent, or "hidden buying factors," through conversation flows tended to perform better, scoring an average pass rate of 70.2%. Yet this was inconsistent, as the same models often failed when made to repeat a scenario they had already passed.

Given how far the technology still has to go, the obvious question is when agentic commerce will actually be viable. According to Rubail Birwadker, Visa’s global head of growth, the biggest hurdle may not be technical at all, but whether consumers will trust it. That gap is already visible in Visa’s data: the company’s Trust Index, released in September, found that while 72% of U.S. consumers have used an AI assistant, only 23% trust AI to make payments on their behalf.

“The models are moving fast, and agents are already shopping," said Birwadker. "But the real milestone is when consumers trust them to pay consistently and securely on their behalf. That’s the bar we’re focused on at Visa, and I think we’ll get there sooner than people expect."

Our Deeper View

Out of all the AI application trends that have emerged over the last couple of years, the one that seems the most difficult to bridge is agentic commerce, particularly because it involves giving AI the power to access your finances, which, naturally, people will be reluctant to. It doesn't help that even in controlled environments, AI agents are going rogue, taking actions that are unintended and not permitted, and not giving users the confidence needed to hand over their credit cards. Additionally, a lot of purchases reflect personal taste, something I'm not entirely sure I would ever trust AI to just handle on its own. However, something AI is very good at is working as an idea partner and brainstorming ideas. I think this is a safer, more productive approach to try for embedding AI into commerce, especially as the holiday season quickly approaches.