An AI phone calling agent, walked second by second through one real restaurant order

The caller opens with: “Can I get one lumberjack slimand one Coke?” That is what the audio says. Run the file through Deepgram and that is what comes back as text. Open the captions array and you can see it on row five.

The agent's first response, ten seconds later, opens with “For your lumberjack slam, how would you like your eggs cooked?” The caller never said “slam.” The agent committed the correction silently inside a 7.025 second think pause, then proceeded straight into modifier intake as if the caller had ordered the dish by its real name from the start.

That moment, and not the LLM, and not the voice quality, and not the latency budget, is the part that determines whether an AI phone calling agent works inside a restaurant. Callers do not speak menu strings. They speak approximations of what they half-read on the takeout site three nights ago. The agent has to translate.

At t=65.985 the caller says “Can you add strawberries, if that's an option?” Note the hedge. The caller does not know the menu well enough to know whether strawberries are a thing on the cheesecake. They are asking, not ordering.

At t=71.345 the agent answers “You got it.” By t=72.145 it has restated “One slice of New York style cheesecake with strawberry topping.” Two quiet things happened in those six seconds. The agent looked up whether the cheesecake item has a strawberry-topping modifier in its modifier tree (it does, that is why the menu has it). Then it normalized the caller's plural “strawberries” to the modifier's canonical singular name. The kitchen printer reads strawberry topping. The cashier reads it. The cook reads it. They all see the menu's word, not the caller's word.

A horizontal AI calling agent platform will gladly transcribe “strawberries” and pass it on. What it cannot do, on its own, is decide whether the modifier exists, attach it to the right line item, and rewrite it to the canonical POS string. That decision lives in the dish ontology, which is restaurant-specific and lives in our onboarding pipeline, not in the speech stack.

The default benchmark for “natural” conversation in the horizontal voice agent guides is sub-600ms response latency. That benchmark is correct as a floor and wrong as a ceiling. On this call the agent's gaps are 1.6, 5.0, and 2.4 seconds, in that order. None of them feel awkward, because the long ones are filled.

At t=10.96, after the caller stops speaking, the agent says “One moment, please.” That is a verbal placeholder that buys the system the time it needs to do the menu lookup and fill the modifier slots before talking again. Without it, the caller would be staring at five seconds of silence. With it, they are not.

At t=37.56, after the caller's modifier choices land, the agent says “Great” in 480ms and rolls straight into the partial readback. Short turn, short wait. Long turn, verbal hold. The threshold is dynamic and the floor is set by what the agent has to look up, not by a fixed 600ms target.

The agent waits until the order is fully assembled (item, all modifiers, name, optional upsell, optional free-form modifier graft), then reads the entire string back in one breath at t=75.42. The caller has 8.64 seconds of agent voice to interrupt with a correction. They do not. They say “Yeah. That's right” at t=85.78.

That is the difference between the order being right at the kitchen printer and the order being right after a refund. A human cashier might or might not read the order back; a reliable phone agent does it every call, regardless of queue depth, and waits for the affirmative before committing. Only after the affirmative does the agent fire “Placing your order now” at t=89.12. The commit step is gated on the readback, not on the original intake.

Most of what is written about AI phone calling agents is written about horizontal platforms (Bland, Retell, Vapi, Synthflow, Lindy, Cognigy, PolyAI). Those platforms are real and they are good. They sell a primitive: a clean speech stack, a fast LLM, a telephony layer, a webhook back to whatever you want. If your team has the engineering bandwidth to build the menu layer on top of one of those primitives, that is a valid path. Bland in particular has a strong developer story for outbound at scale.

The work this page describes, the dish ontology, the alias and phonetic table, the modifier tree, the POS item mapping, the normalization of caller words to canonical modifier names, the gating of commit on a full readback, none of that ships with a horizontal platform. It is built per restaurant, by hand, the first time. After that it is maintained week to week as the menu changes.

That work is what we sell. PieLine ships the call you just heard, for your menu, on your POS, in under 24 hours of onboarding work (mostly on our side; for the owner it is hands-off). It handles up to 20 simultaneous calls during peak. The price is $350 per month for the first 1,000 calls, then $0.50 per call. The first month carries a money-back guarantee.

The agent will not invent a price for a menu item it has never seen. It will not invent an allergen fact. It will not pretend to know whether the kitchen is out of an item; that requires a live POS inventory feed and the absence of one is the right reason to escalate. It will not handle a complaint about last night's order; that is also a named escalation trigger.

When any of those conditions hits, the agent transfers the call with the full transcript attached, so the manager who picks up the line is not starting from zero. The transferred-call rate on live deployments runs under 10%. The other 90%-plus runs end to end on the agent.