web/docs/runtime-gaps.md

What the runtimes showed up in the gate

Found while building and running the buyers in this folder (2026-11-13/14, seed catalogue, demo kitchen mandate). Each item says what happened, where, and the smallest change that would fix it.

Status (14 Nov, branch claude/integrate): 1 fixed (answerUrl, a one-time answer page, channel answer_link; relayed answers stay "relayed, unverified"); 2 fixed in the core, so REST too (POST /api/v1/orders with only requestId; refusal no_current_quote says why when there is none); 3 fixed (x-dock-human-input, tags and x-dock-audience: approver, ?audience=agent); 4 fixed (nextStep and waitingFor on every REST intake, quote and order reply; examples hold placeholders, never a pre-filled answer); 5 fixed (need.deliverOn, an exact day held by the gate); 6 fixed (output schemas published shallow: tools/list 129 KB -> 30 KB, output schemas 109 KB -> 11 KB).

Re-run on 14 Nov afternoon (branch claude/open-interface, 22 tools, partner catalogue, production build)

All seven reached pending_approval with no refusal met, about $0.12 in model cost; transcripts are in transcripts/. Since Friday the tool table gained find_products, find_alternatives, list_orders, withdraw_order and the selling agent's four offer tools; get_order gained waitSeconds. No runtime needed a change for them. What the earlier runs today showed, and what was fixed on our side:

• The runs disturbed each other. Every runtime buys for the same demo kitchen, so the sixth (LangGraph) met five 1,000 L orders still waiting for sign-off and AimoPilot asked "order again?". The scripted buyer could not answer it, and Claude Haiku 5.5 rightly placed nothing (it also offered the one-time answer page). Fix: the runner takes each run's waiting order back through the published withdraw before the next runtime starts.
• The model must use the same key as the server. mint-mandate.ts did not read web/.env.local, so with DOCK_DEMO_KITCHEN_JWK set every mandate would have been refused. Fix: it now loads it, like the demo scripts.
• Google ADK + gemini-3.1-flash-lite put quantity next to need instead of inside it, five times. Our refusal said only "quantity is required", so it fell back to the lower-level tools. Fix: submit_need folds a need's own fields sent beside it into the need. Next run: four calls, the intake path.
• OpenAI Agents SDK + gpt-6-luna called choose_option without by and was refused. Fix: choosing is the agent's own step, so without a name the record says the agent chose. (Buyer answers still require the buyer.)
• Semantic Kernel no longer names any AimoPilot operation. It reads the human-input operations from the document's x-dock-human-input, from the agent view (?audience=agent).

Which needed nothing but the published documents: all seven run on the address, the mandate and what tools/list or /api/v1/openapi.json?audience=agent says. Six read the published human-input marker (_meta["dock/humanInput"] or x-dock-human-input) to route the buyer's answers through their own approval step. Pydantic AI needs not even that: elicitation lets AimoPilot ask the person directly. Each example's runtime-specific settings are its runtime's own (Pydantic AI's tool_error_behavior; Semantic Kernel's four workarounds, listed in RESULTS.md), not AimoPilot's.

1. On clients that cannot elicit, the gate cannot tell a buyer's answer from the model's

Seen with: OpenAI Agents SDK (no elicitation support at all), Google ADK (2025-11-25 handshake), LangGraph (langchain-mcp-adapters pins mcp<2, so 2025-11-25), Semantic Kernel over REST. Only Pydantic AI, the Claude Agent SDK and the Vercel AI SDK reach MCP 2026-07-28 and let AimoPilot ask the buyer itself.

On every other path respond_to_concerns / answer_questions (REST /concerns, /answers) accept whatever by and choice the agent sends. In a Semantic Kernel development run, gpt-6-luna called respondToConcerns with by: "buyer" before asking anyone; only the runtime-side filter in the example stopped it (and the gate would have refused the invented choice id keep_requested_quantity anyway, but not a plain "confirm"). The examples hold the line with each runtime's approval step; an agent built without one would confirm the 1,000 L itself and the ledger would record it as the buyer's answer.

Smallest fix: when the client cannot elicit (human.canAsk false, and always over REST), put a one-time answerUrl on needs_confirmation / needs_input steps (the same token-in-link mechanism as approval links) and tell the agent to show it to the buyer; keep accepting relayed answers, but write them to the ledger as relayed, unverified (the entry already carries relayedBy). Larger alternative: serve 2025-era tools/call statefully (session id + SSE response stream) so elicitation/create can be pushed to the clients that support it on 2025-11-25 (ADK and LangGraph do).

2. The model has to copy a 1 KB signed quote token into place_order

Seen with: OpenAI Agents SDK + gpt-6-luna: in five runs, three met quote_signature_invalid / quote_malformed and two of those never placed the order (five to six mangled copies each, fresh quotes in between, $0.06-0.10 spent instead of $0.01). Google ADK + gemini-3.1-flash-lite and Vercel AI SDK + gpt-6-luna each mangled it once in development and then copied it right. Claude Haiku 5.5 copied it every time.

Small models truncate or alter long opaque strings. The gate does the right thing (refuses, rule quote_signature), but the buyer pays in retries, live lookups and sometimes the whole order.

Smallest fix: in place_order, make quoteToken and lines optional when requestId names an intake in state quoted, and take both from the quote the intake already holds (or accept the short quoteId). The JWS stays in the result for anyone who wants to verify it; the model no longer has to transcribe it.

3. The OpenAPI document does not say which operations need a person, and offers approver operations to agents

Seen with: Semantic Kernel (any OpenAPI-to-tools converter behaves the same).

• MCP marks the two human-input tools with _meta["dock/humanInput"]; /api/v1/openapi.json has no equivalent on respondToConcerns and answerQuestions, so the example names them by operationId (its only AimoPilot-specific line). Fix: add x-dock-human-input: true (same content as the MCP _meta) to those two operations.
• The same document lists decideApproval, approvalChallenge, approvalLink, beginJoin, completeJoin, approverDemoOrder, sandboxDemoOrder, listSandboxes. A converter that loads every operation offers a buyer agent a tool to approve orders. The approval calls are refused without the approver's link token, passkey or session, but they invite the attempt and cost tokens. The example keeps only operations whose security is the mandate. Fix: tag them (for example tags: ["approver"] plus x-dock-audience: "approver") or serve an agent-only view (/api/v1/openapi.json?audience=agent).

4. REST intake steps carry less guidance than MCP's

Seen with: Semantic Kernel. With gpt-5-mini (development run), at quoted it told the buyer four times that it needed a sign-off before it could place the order, and never placed it. With gpt-6-luna (the recorded run, transcripts/semantic-kernel-openapi.txt) it said the same, called the valid quote expired, and opened a second intake before ordering. The MCP result for the same step says "Placing it will ask Jari Mäkelä (owner) to sign off on their own device"; the REST step has only next: { method, path, example }, no nextStep sentence and no waitingFor.

Also, next.example for needs_confirmation pre-fills choice with the first correction (use_100_L): an agent that copies the example, as it is told to do for place_order, silently turns 1,000 L into 100 L.

Fix: add the same nextStep and waitingFor the MCP tools compute (nextStepForIntake, waitingFor in server/mcp/next-step.ts) to the REST intake responses, and make the concern example a placeholder ("choice": "<confirm or a correction id, from the buyer>").

5. "Delivery on Monday" can only be said as "by Monday"

Seen with: every runtime; most visibly OpenAI Agents SDK + gpt-6-luna, which asked the buyer three times whether a Saturday delivery was acceptable before going on. The need has only deliverBy; all three options deliver on the earliest day (Saturday 14 November). A kitchen asking for Monday usually means Monday.

Fix: an optional deliverOn in the need (exact day), used to pick the delivery date of each option, falling back to deliverBy behaviour when absent.

6. The tools' output schemas are twelve times their input schemas, and some runtimes send them to the model

Seen with: Google ADK, which passes each MCP tool's outputSchema to Gemini as the function's response_json_schema. tools/list carries 118 KB of output schemas against 10 KB of input schemas and 6 KB of descriptions, so every ADK request starts at about 50,000 prompt tokens (other runtimes: about 12,000-16,000), and the ADK run costs five to seven times the others on a model with similar prices.

Smallest fix: publish the output schemas with shared definitions ($defs + $ref; zod's toJSONSchema with reused: "ref") and without the field descriptions the input side already explains, or publish a shallow output schema (ok, step.state, nextStep, error) and keep the full shapes in the OpenAPI document.

7. Runtime behaviours the gate should keep in mind (no change needed)

• Server instructions rarely reach the model. Only Pydantic AI has an option to forward them, off by default. AimoPilot's guidance survives because tool descriptions and every result's nextStep repeat it. Keep it that way.
• Refusals (isError with structuredContent) are handled unevenly. Pydantic AI's default raises ModelRetry (the model reads "retry" and the run aborts on the second refusal by one tool); Semantic Kernel drops 4xx bodies entirely. OpenAI Agents, ADK, LangGraph and the AI SDK pass the text through. Because AimoPilot repeats the JSON as text content, the clients that ignore structuredContent still see code, rule and nextStep.
• "Waiting for sign-off, do not retry" was obeyed. No runtime placed an order again after pending_approval; when the scripted buyer kept talking, the AI SDK buyer called get_order instead.
• Custom headers were never a problem for these SDKs. They matter for hosted connectors (ChatGPT, claude.ai, OpenAI's hosted MCP tool, which also needs a public URL); that is what the URL-only connector credential on claude/connector is for.
• The human-input marker is AimoPilot's own. MCP has no standard annotation for "needs a person", so every example that routes those tools through an approval step reads _meta["dock/humanInput"] (the Claude Agent SDK does not expose tool _meta, so that example sends one plain tools/list).

Design note: waiting hours for a sign-off

Today the buyer agent stops when place_order returns pending_approval; the gate holds the order, the approver decides on their phone, and anyone can follow up with get_order. What the runtimes offer for pausing and resuming, from their current documentation:

All of these pause the agent before one of its own tool calls runs, for the agent's human to approve it. AimoPilot's sign-off has a different shape: the tool call has already run, the order exists, and a different person decides on the gate. The state that matters (the order, the approval, the ledger) already lives in the gate and survives restarts; the agent has nothing worth keeping while it waits. A durable runtime would add two things only: resuming the agent automatically when the decision lands, and keeping its conversation across a restart.

Recommendation: keep the current design. If the agent must act on the decision (tell the kitchen, or re-plan after a decline), have the gate push the decision (it already streams /api/v1/events) and resume the agent from its stored messages; a WorkflowAgent with a hook keyed by orderId is the natural fit on the AI SDK if durability is needed in production, but it adds a workflow backend for no visible gain in the demo.