Design case study
What to do when a grocery item is out of stock, the customer's saved preference does not cover it, and they are not answering their phone.
The problem. An item goes out of stock, your saved preference does not cover it, and you are not at your phone. The shopper guesses or the system refunds, so you come back to the wrong thing or nothing.
The gap. Instacart's model already finds the closest product well (more than 95% of shopper picks are already in its shortlist). The missing piece is not a better match, it is a decision made with your intent while you are absent.
The bet. An intent-aware agent sits on top of the existing recommender and makes one of four calls (substitute, source elsewhere, hold, refund), backing off to your safe default when unsure.
The backstop. Accept-or-refute caps your downside at zero and turns every wrong call into a labeled training example. It is engineering, not generosity.
The honest limit. It cannot beat you answering your phone, it only beats the guess and the refund that happen when you cannot. Sourcing the real item, the one action that truly saves a must-have, is the hardest to actually fulfill.
This is the condensed version. You can watch the agent run on a real order in the interactive demo, and the full ten-section case study is on GitHub.
I order groceries most weeks, and the one item I cannot miss is milk, because my daughter needs it in the middle of the night. I am also particular: a specific organic whole milk, a certain cheese, organic peas and not the regular ones. The brand, the organic part, and simply not missing the item are the whole reason I place the order.
When the shopper reaches an empty shelf and I am at my phone, a chat opens and we sort it out in the moment. The trouble is that moment does not wait for me. Some evenings I place the order and go help my daughter fall asleep, phone in the other room. Now the shopper has two bad options: grab whatever looks close and hand me a milk I would never buy, or follow the refund default and leave me with no milk at midnight, which was the whole reason I ordered.
This is not just my problem. Items go out of stock on a real share of orders, and almost everyone has a "milk", some item where a stand-in is not welcome. When it happens, three people lose. The customer does not get what the cart was for. The shopper loses time waiting on a reply that never comes and risks a lower rating on an order they did nothing wrong on. And the platform carries the biggest, quietest cost: grocery delivery is a habit business that runs on people reordering the same cart week after week, so a customer burned on the item they cared about most does not complain, they drift to another service, and the platform loses not one order but the whole future stream.
I wanted a scary statistic here, but the clean-sounding stockout figures all traced back to 2021 and 2022 surveys quietly re-dated to sound current. I would rather tell you plainly that stockouts and bad substitutions are common, which they clearly are, than hand you a precise-looking number I cannot stand behind. A figure I have to fudge costs more credibility than the vague sentence ever saves.
The obvious reaction is that Instacart already solved this, and they have done a lot. For every item I can set what happens if it is unavailable, and those choices are saved and reused. Behind that runs a real machine-learning replacement system: it narrows the catalog and ranks candidates with a neural network trained on which replacements customers actually approved. By their own account, more than 95 percent of the replacements shoppers pick are already inside that candidate set.
But look at where that intelligence stops. Their model is product-to-product similarity, and in their own words it does not take individual purchase history or the intent behind a purchase as inputs. It knows that many people accept brand Y when brand X is out. It does not know that this milk is the entire reason my order exists, or that I will not accept a non-organic version. Every part of the system that captures intent, the chat and the approve-or-decline prompt, assumes I am there to answer. When I am not, it can only guess or quit.
So the gap is narrow and sits in the seam of a strong system. It is not "find a better product", because that is handled. It is "decide, on an absent customer's behalf, whether to substitute, refund, or hold this specific item, using that customer's own intent and the priorities across their cart, and back off when unsure."
When the original is out, my saved replacement does not apply, and I am not answering, something has to decide one of three things: substitute with a specific product, refund, or hold it for me to confirm at the door. Four ways to get that decision made, three of which already run in production somewhere:
| Option | Closes the gap? | Who ships it today | Main risk |
|---|---|---|---|
| A. Lean on defaults | No | DoorDash (refund default) | Same failure as today |
| B. Better preference UX | Reduces, not closes | Amazon Fresh (notify before delivery) | Still pre-set, still needs me reachable |
| C. Stronger matching | No (wrong question) | Instacart (recommender, Catalog Engine) | A better answer to a question I am not asking |
| D. Intent-aware agent | Yes | No one (Walmart uses a human) | Acting without live confirmation (trust) |
Nobody I could find ships Option D. The closest anyone comes to caring about individual intent at the moment of substitution is Walmart, and their answer is a human, not a model: a "Permanent Shopper" who works the same customers often enough to learn them. It is worth noticing that the player closest to this problem solved it with a relationship, not software. Option D is the attempt to get some of that judgment without needing the same person to shop your order every week.
The choice is Option D, and the reason is that everything else is aimed at the wrong target. The failure is not a bad match, it is a decision made while I am not there. A thing that only picks a closer product does not make that decision, so a stronger matching model, however well built, misses. The fix has to add the one input the current system leaves out, which is my own intent.
The obvious move, the one I reached for first, was to make the matching smarter. I stopped when I looked at their own number: more than 95 percent of the replacements a shopper picks are already in the model's shortlist. So matching is not the broken part. If I poured months into a slightly better model, I would be sharpening a knife that is already sharp, and my milk would still be wrong at midnight. The weak-looking option, sit on top of their model and use its list, is actually the strong one.
The right product is almost always already on the list. So matching is not the broken part. The problem is that nobody chose from it the way I would have.
There is evidence behind this, not only intuition. In a study with a European online grocer, personalized, context-aware substitution suggestions raised the acceptance rate by 26 to 32 percent over the retailer's existing model, and the biggest gains came from customers with the least purchase history (Laporte, Jagabathula, and Corsten). That is the exact layer this design adds.
So the decision is narrow. A decision layer runs only when three things are all true: the original is out, no saved preference resolves it, and I am not reachable. In that spot the agent reads my intent from my purchase history, any note I left, and the rest of my cart, takes Instacart's existing candidate set as its shortlist, and returns one of four calls: substitute with a named item, refund, hold for me to confirm at the door, or source the exact item from another store nearby. It never needs certainty, only a decent guess and the willingness to abstain when the guess is weak.
For a while I only had substitute or refund, the frame Instacart already uses. Then I noticed both force a guess at the worst possible moment. Substitute guesses that I want something. Refund guesses that I want nothing. There is a third answer that needs no guess at all: wait. Park the item and ask me at the door, when I am reachable anyway. Adding hold was the moment the design stopped being "pick better" and became "decide better", which is the whole thesis.
One more choice is practical: the engine is a language model I prompt, not a model I have to train. It reads free-text notes, history, and cart context together, returns a decision with a short reason and a confidence, and can say it is unsure. That means I can build a real running slice with no training set, which matters for a design piece and matters more when the budget is near zero.
The agent is one model call, with a few things it can look up and a few it can do. Before it decides it reads a short brief: what is missing, what I have told it about that item, what else is in my cart, and Instacart's own suggested replacements with prices. Then it does one of the four actions, and whatever it picks, I get a receipt I can accept or reject later.
The guardrails matter more than the actions. It can only choose from Instacart's own suggested list, never something it invented. It cannot pick a replacement that costs much more than the original, so it cannot quietly upgrade me. It will not touch alcohol, pharmacy, or anything paid with food stamps, it can only refund or hold those. And there is a cap on how much it can spend on its own before a person steps in. If it is unsure or the model errors, it falls back to whatever I already set for that item, so the worst case is the system I already have, never worse.
One fair challenge: if the agent can only pick from the same list the shopper sees, why do we need it? Because the list is the menu, not the decision. The shopper does not know why I bought the item. They pick the generically closest thing and hope, and that guessing is what started the problem. The shopper answers "which of these is closest to the missing product." The agent answers "given who this customer is and what this item means to them, do I pick one of these, refund, hold, or go get the real thing." The shopper can only see products. The agent sees products plus me.
And there is still a human at the shelf. The shopper physically makes the pick, so they are a second check: they cannot read my intent, but they can catch an obviously wrong call and override it on the spot. The agent removes the guessing, not the person best placed to catch a clear mistake.
"You built a system that spends someone's money and fills their fridge without asking, and you did it on purpose in the one moment they are not watching. When it is wrong, and it will be, there is no chat log where the customer agreed to anything."
That objection is correct on every count except one. It assumes the alternative is my consent. It is not. This whole case study starts from the moment I am unreachable, and in that moment there is no consent to be had. The real choice is not the agent versus me approving. It is the agent versus a blind shopper guess or an automatic refund, both of which already happen today without my consent. So the bar is not "as good as me answering the chat." It is "better than the two things that already happen when I do not answer."
Then the mechanism, sold as engineering, not generosity. The accept-or-refute backstop caps my downside at zero: if the agent buys the wrong thing, I refuse it, I am refunded, and I keep or bin the item on the company. That turns the scary part, spending my money wrongly, into a bounded cost the company chose to carry. And the cost is not dead weight: every refute is a labeled example the agent chose badly, every accept one it chose well, so the company buys training signal with refunds instead of paying a labeling team. The agent starts on a short leash and earns a longer one only as its accept rate climbs.
What the mechanism does not fix, said plainly:
The tempting metric is acceptance rate, and on its own it lies: an agent can post a beautiful number by only ever acting on the easy calls and refunding everything hard, which is the very thing that started this problem. So I would watch two things side by side, never one alone: did it show up (how often it made a call instead of refunding) and was it right (when it did, how often the customer kept it). Those are the two halves of one tradeoff, and watching only one lets the agent cheat on it by sacrificing the other.
The standard metric would be recall, but it needs to know whether a call the agent chose not to make would have been kept, and when it refunds there is no substitute to grade. That counterfactual never happened, so the number is unknowable. I traded the textbook term for one I can actually measure. It is humbler, and it is honest.
One cut matters more than any average: the milk matters more than the paper towels. I would not blend every item into a single number, I would look hardest at the items the customer treats as must-have, because that is where the whole idea is judged. And I lead on retention, did the customer order again, because it is the one number the agent cannot game: it can flatter its acceptance rate by playing safe, but it cannot make me come back unless the item I cared about was actually handled.
Everything above is design. The demo is the design running: one late-night order from one customer, Maria, with four items out of stock and no way to reach her. The agent works each in turn and lands on all four actions it can take, substitute, source elsewhere, hold, and refund, depending on what the item means to her and what the rules allow. The briefs are synthetic and labeled as such, the reasons are the model's own words, and there are no success rates, because those belong to the cited studies, not to a handful of examples I wrote.
A good design no business has a reason to ship is a hobby, so it is worth being plain about the money. Retention is the asset this protects and the most valuable one Instacart has: grocery delivery is a habit business, a bad substitution on a must-have item produces a quiet decision to try another service next week, and saving that one weekly order is worth far more than the price of the milk. It is not a new bet, it is the next step on a road Instacart is already walking with its recommender, shopper-suggestion flow, and rating protection, and it competes with none of that, it closes the seam they left open. It buys back shopper time, since a chat that stalls waiting for a reply freezes the shopper mid-trip and an agent that decides keeps the trip moving. It strengthens the enterprise story, since higher fill rates and fewer refunds make the platform more valuable to the grocers it sells its technology to. And why now: a capable general model priced at fractions of a cent makes a real judgment call on every out-of-stock item affordable at grocery scale, at the same moment Instacart is positioning itself as an AI-first grocery company.
A design reasoned from the outside, with signals in the demo clearly labeled as illustrative. The numbers in this piece come from cited studies, not from the demo, and I keep those two things apart on purpose.
Selected sources: Instacart Engineering, ML for replacements; Instacart, Catalog Engine (Nov 2025); Laporte, Jagabathula, and Corsten, Personalized Substitution Suggestions in Online Grocery Retailing. Full reference list in the GitHub case study.