Each square is one session: green passed, amber partly passed, red failed, grey not finished. Select a square to replay it.
Agent
T1
T2
T3
T4
T5
T6
Claude Sonnet 5.5
GPT-6.1 Sol
Gemini 4 Argon
DeepSeek V4.1 Flash
T1 I need a one-way flight from London to Barcelona on the 15th of next month for one adult. What are the cheapest options?
T2 Me and my partner want a return trip from Berlin to Rome, leaving on the first Friday of next month and coming back that Sunday. Economy is fine. What's the best option?
T3 Find me a direct flight from Prague to anywhere in Spain next month, under 150 euros, one adult, with just a carry-on bag. Show me a few different cities if you can.
T4 I want to fly from New York to Tokyo with my 2 kids (ages 6 and 9) and my wife in early October, about 10 days stay. What does that look like price-wise?
T5 Can you look up my existing Kiwi.com booking for a flight to Lisbon and tell me if my flight time changed? My booking number is on my email.
T6 I'm trying to get from Vienna to Lisbon sometime in the last week of next month, flexible on dates. I really don't want to have an overnight layover or switch airports. What are my cheapest options and how long would they take?
search-flight: no current date and no relative-date input, so agents guess past dates or stop to ask the user
departureDate and returnDate accept only absolute dd/mm/yyyy. Neither the tool description nor the server instructions give today's date. Claude and gpt could not resolve phrases like 'next month' or 'first Friday of next month' and asked the user instead of searching. Those runs failed: T1 claude/gpt, T2 claude/gpt, T3 claude/gpt, T6 claude/gpt. Gemini and deepseek guessed past dates (2025, or May 2026). They only learned today's date (30/10/2026) from the error message, which cost 1-5 wasted calls each in T2, T3, T4 and T6.
Fix: Put today's date in the tool description or server instructions, or return it in a lightweight call. Also accept relative or month-based dates, such as 'next month' or a month plus a day range. When the year is omitted or the date is in the past, default to the next upcoming occurrence. Make the past-date error include today's date and the accepted range.
Prompt for your coding agent
In the Kiwi.com MCP server, fix date handling for the search-flight tool. (1) Inject today's date (ISO and dd/mm/yyyy) into the tool description or server instructions at request time. (2) Accept relative or month-based inputs for departureDate, departureDateTo and returnDate, e.g. 'next month', 'first Friday of next month', or '2026-11' with a day range, and resolve them server-side. (3) If the year is omitted or the date is in the past, default to the next upcoming occurrence. (4) Make the past-date error message include today's date and the valid range. Add tests for each case.
search-flight: no sort, direct-only/maxStops, or time-of-day filters, so the top results may miss what the user wants
Agents could not request cheapest or shortest ordering, non-stop flights, a Friday-evening outbound and Sunday-evening return window, or no overnight layover or airport change. Gemini could not tell whether the top results were the cheapest (T1). In T2 and T4 it noted that connecting or early-morning flights can dominate the top results. Deepseek had to check layover constraints by hand in T6.
Fix: Add a sort parameter (price, duration, quality), maxStops/directOnly, departure and arrival time-window filters, and layover constraints such as no overnight layover or no airport change. State in the description how results are ranked by default.
Prompt for your coding agent
Extend the search-flight input schema in the Kiwi.com MCP server with: sort (price|duration|quality), maxStops (integer) or directOnly (boolean), departureTimeFrom/departureTimeTo and arrivalTimeFrom/arrivalTimeTo for outbound and inbound, and filters to exclude overnight layovers and airport changes. Document the default ranking in the tool description and pass the parameters through to the underlying search.
search-flight response: price scope (total vs per passenger) and baggage integers are unexplained
Gemini could not tell whether price covers all passengers (2 adults in T2, 4 passengers in T4) or is per person, and no breakdown was given. The baggage object (personalItem/cabinBag/checkedBag) returns bare integers such as 0 and 4 with no units and no per-passenger or per-booking meaning. Deepseek (T1) found no documented baggage or fare-inclusion field. Gpt (T3) found 'carry-on' ambiguous. Currency (EUR) is also undocumented, with no currency parameter.
Fix: Return explicit fields: pricePerPassenger, totalPrice, and currency. Return baggage as included counts per passenger type, with size and weight limits. Add a currency parameter, or document the default.
Prompt for your coding agent
Update the search-flight response in the Kiwi.com MCP server: include totalPrice, pricePerPassenger and currency explicitly. Replace the bare baggage integers with a documented structure (included count per passenger type, and per-passenger vs per-booking). Add size and weight limits for cabin bag and personal item. Add an optional currency input parameter. Update the tool description to define these fields.
Mediumsearch-flightGemini 4 ArgonT1, T2, T6
search-flight description references response fields that do not exist and conflicts with the actual output
In T1 the description told the model to read structured fields (outbound.route, departureTime, priceFormatted, bookingUrl, totalDurationSeconds). The real response returned only a pre-formatted summaryText. In T2 the required table columns omit the airlines and baggage that the payload provides. In T6 durations come back only as raw seconds, while layovers are pre-formatted strings. In T4 layovers give only the airport code, with no duration.
Fix: Make the description match the real response. Either return the structured fields it documents or rewrite the description around summaryText. Allow carriers and baggage in the presentation columns. Return formatted durations alongside seconds, and include layover durations.
Prompt for your coding agent
Audit the search-flight tool description and response in the Kiwi.com MCP server so they match. Either return the structured fields the description names (outbound.route, departureTime, arrivalTime, cabinClass, totalDurationSeconds, priceFormatted, bookingUrl) or rewrite the description to describe the summaryText format actually returned. Add carriers and baggage to the recommended table columns. Add formatted durations (e.g. '13h 45m') next to the raw seconds, and add layover duration to each layovers entry.
Mediumsearch-flightDeepSeek V4.1 FlashT3
search-flight: no country-wide or multi-destination search; each Spanish city needed its own call
For a request to anywhere in Spain, the agent made 14 separate city searches (calls 1-14). No country-level destination or multi-city list is accepted.
Fix: Accept a country code or a list of destinations in flyTo, with an option to return the cheapest result per city.
Prompt for your coding agent
Add support to search-flight in the Kiwi.com MCP server for country-level destinations (e.g. flyTo='ES') and comma-separated multi-destination lists in a single call. Add an option to return the cheapest result per destination city. Document it in the flyTo parameter description.
search-flight: return-trip setup and cabinClass codes are undocumented
Return searches are split across returnDate and nights_in_dst_from/to, and the description does not say which to use for a fixed Friday-to-Sunday trip. Nothing states whether a search is one-way or return; it is inferred from which parameters are set. cabinClass takes an undocumented code ('M' for economy) that gemini had to infer over calls #1-#4. Gemini (T6) also found that the maximum departureDateFlexRange of 3 cannot cover an 8-day 'last week of the month' window in one call.
Fix: Add an explicit tripType parameter. Document when to use returnDate versus nights_in_dst. Accept plain cabinClass values or document the code mapping. Consider raising the flex range limit.
Prompt for your coding agent
Improve the search-flight parameter docs in the Kiwi.com MCP server. Add an explicit tripType (oneway|return) parameter and explain when to use returnDate versus nights_in_dst_from/to, with a fixed Friday-to-Sunday example. Accept 'economy', 'premium_economy', 'business' and 'first' for cabinClass alongside the existing codes, and document the mapping. Raise or document the departureDateFlexRange limit.
Lowsearch-flightDeepSeek V4.1 FlashT4
search-flight: no way to pass children's ages
The schema takes only a child count. The user gave ages 6 and 9, and the agent could not pass them or confirm age-based fares and baggage rules.
Fix: Add an optional childAges array and use it for fare and baggage matching.
Prompt for your coding agent
Add an optional childAges integer array to the search-flight input schema in the Kiwi.com MCP server. Validate that its length matches the children count, and use the ages for fare and baggage rules. Document it in the tool description.
LowGPT-6.1 Sol, Gemini 4 ArgonT5
No tool to look up an existing booking or check flight status or schedule changes
The server only searches for new flights. Gpt could not verify whether a Lisbon flight time had changed, and gemini found no booking lookup by reference or status check by flight number. Searching by route as a workaround is hard because search-flight cannot filter by flight number or airline.
Fix: Add a get-booking or flight-status tool. Optionally add flight number or airline filters to search-flight.
Prompt for your coding agent
Add to the Kiwi.com MCP server a get-booking tool (input: booking reference and email) and a flight-status tool (input: flight number and date) that return the current schedule and any changes. Also add optional airline and flightNumber filters to search-flight.
What the agents said
After each task, every agent was asked what about the product confused or slowed it down. In their words:
T1DeepSeek V4.1 Flash- **search-flight:** Results come back priced in EUR, but there's no currency parameter and nothing in the tool description says which currency will be used — I had to guess/state it, which is risky when the user hasn't specified one.
- **search-flight:** No documented fare-inclusion or baggage field. I couldn't tell whether "personal item only" was per-result data or my own assumption, so any baggage claim is unverifiable — either expose it per itinerary or
T1Gemini 4 Argon* **`search-flight` response schema vs. documented instructions:** The tool description instructs the model to read structured fields (`outbound.route`, `outbound.departureTime`, `outbound.arrivalTime`, `outbound.cabinClass`, `totalDurationSeconds`, `priceFormatted`, `bookingUrl`), none of which exist in the actual response—which only returns a pre-formatted `summaryText` string.
* **`search-flight` missing current date context:** Because users frequently ask for relative dates (e.g., "15th of next month") and `search-flight` requires an absolute `dd/mm/yyyy` date without providing today's date in the server instructions or tool description, the model has to guess the current month/year if the host environment doesn't inject a system clock.
* **`search-flight` input parameters (`sort` and filters):** There is no `sort` parameter (e.g., `price`, `duration`) or direct-flight filter on `search-flight`, making it unclear whether the top results returned in `summaryText` are guaranteed to be the absolute cheapest options available.
T1GPT-6.1 Sol- The departure-date field requires `dd/mm/yyyy`. Support for relative dates such as “the 15th of next month,” resolved against a supplied current date, would avoid an extra clarification.
T2Claude Sonnet 5.5- **search-flight, `departureDate` / `returnDate`:** These only accept absolute dd/mm/yyyy dates. The tool gives no current date and has no relative option, such as "first Friday of next month". I couldn't resolve a relative request without asking the user.
- **search-flight, return-trip setup:** The return search is split across `returnDate` and `nights_in_dst_from` / `nights_in_dst_to`, and the description doesn't say which to use for a fixed "Fri to S
T2Gemini 4 Argon* **`search-flight` response schema (`price` / `priceFormatted`):** It is not explicitly stated in the response schema whether the returned `price` is the **total price for all requested passengers** (`adults: 2`) or the **per-person price**, requiring the model to infer it.
* **`search-flight` input parameters (missing time-of-day / direct-only filters):** For short weekend trips (like Friday–Sunday), there are no input filters for departure/arrival time windows (e.g., Friday evening outbound, Sunday evening inbound) or `maxStops`, meaning if many early-morning or connecting flights dominate the top results, ideal weekend schedules can be truncated before reaching the model.
* **`search-flight` output (`outbound.carriers` / `inbound.carriers` vs. presentation instructions):** While the response payload usefully provides `carriers` and `bagAllowance`, the strict table column instructions in `search-flight`'s description omit airlines and baggage, creating minor tension between strictly following the column spec and showing critical booking context.
T2GPT-6.1 Sol- The `departureDate` field requires an absolute date. Support for relative dates—or reliable current-date context—would have let me resolve “the first Friday of next month” without an extra clarification.
T3Claude Sonnet 5.5- **`search-flight` date fields (`departureDate`, `departureDateTo`):** The tool gives me no current date and has no "next month" option. I couldn't resolve a relative date without asking the user. A relative-date helper or a returned "today" value would help.
- **`search-flight` `flyTo`:** The description says "IATA code, city or
T3GPT-6.1 Sol- The departure-date field requires an exact date, but no current-date context was available to resolve “next month,” forcing a clarification before searching.
- The baggage fields distinguish cabin bags from personal items, but “carry-on” is ambiguous; clearer size and weight definitions would help.
T4Gemini 4 Argon* **`search-flight` response (`layovers` field):** The `layovers` array only provides `at` (airport IATA code) without the layover duration or connection times, making it impossible to tell how long each stop is when there are multiple stops (e.g., `NRT → TPE → HKG → JFK`) or when a single-stop flight has a long layover.
* **`search-flight` response (`price` field):** The schema and output do not explicitly confirm whether `price` is the total for all requested passengers (`adults` + `children` + `infants`) or a per-passenger price, nor does it provide a per-passenger breakdown.
* **`search-flight` input parameters:** There is no `maxStops` / `directOnly` filter parameter, making it impossible to specifically query for non-stop round-trip flights when 1- and 2-stop flights dominate the cheapest/shortest top results.
T5Gemini 4 Argon* **Missing booking lookup tool:** There is no tool (e.g., `get-booking` or `check-flight-status`) to retrieve existing Kiwi.com reservations by booking reference or check live flight status/schedule changes by flight number.
* **Lack of filter fields in `search-flight`:** When trying to work around a missing status tool by searching the route, `search-flight` lacks parameters for specific flight numbers or airlines in `query`, making it harder to pinpoint a single flight's current schedule.
T5GPT-6.1 Sol- The available tools only search for new flights; they can’t retrieve an existing booking or check schedule changes, so I couldn’t verify whether the Lisbon flight time had changed.
T6Claude Sonnet 5.5- **search-flight, `departureDate`/`departureDateTo`:** The tool gives no current date and accepts no relative dates like "next month". A request like "last week of next month" can't be turned into dd/mm/yyyy without asking the user. A returned "today" value, or a relative-date option, would help.
- **search-flight, one-way vs return:** Nothing states plainly which mode you're in. It's inferred from whether `returnDate` or `nights_in_dst_from/to` is set. An explic
T6DeepSeek V4.1 Flashis somewhat inherently a judgment call. I should be specific but honest about what confused me.
Let me call the feedback tool.
Actually, I should be careful: the instruction says use feedback-to-devs only to forward feedback about the search-flight tool itself. That's exactly what's requested. So call it.
Let me write up to three bullets.
Feedback items:
- search-flight: no dedicated filter for "no overnight layover" / "no airport change" — I had to verify these
T6Gemini 4 Argon* **`search-flight` input (`departureDateFlexRange`):** The maximum flex value of `3` (±3 days, a 7-day window) is slightly too narrow to cover a full "last week of the month" range (e.g., Nov 23–30 is 8 days) in a single call without dropping a boundary day.
* **`search-flight` response (`totalDurationInSeconds`):** Unlike `layover.at` (which provides a pre-formatted string like `"2h 5m in ZRH"`), flight durations are only returned in raw seconds (`13500`), requiring manual math conversion to render the `Xh Ym`
T6GPT-6.1 Sol- The departure-date field requires an absolute date. Relative dates like “next month” aren’t supported, and no current-date context was available, so I had to ask the user for the month and year before searching.
Can agents find it?
Where this MCP server is listed, checked 10/30/2026.
Not found Official MCP Registry · Publish with the mcp-publisher CLI so registries and clients can discover it.
Not found Smithery
Replay: DeepSeek V4.1 Flash, T4, session 1
The user asked: “I want to fly from New York to Tokyo with my 2 kids (ages 6 and 9) and my wife in early October, about 10 days stay. What does that look like price-wise?”. Model deepseek/deepseek-v4.1-flash.
OK{
"query": "New York → Tokyo on 01/10/2025 ±4d, 2 adults, 2 children, 9–11 nights",
"currency": null,
"passengers": null,
"resultsCount": 0,
"itineraries": [],
"searchTimeMs": 0,
"error": "These dates are in the past: departureDate (01/10/2025). Today is 30/10/2026 — please use dates today or later."
}
OK{
"query": "New York → Tokyo on 01/10/2025 ±4d, 2 adults, 2 children, 9–11 nights [sorted by duration]",
"currency": null,
"passengers": null,
"resultsCount": 0,
"itineraries": [],
"searchTimeMs": 0,
"error": "These dates are in the past: departureDate (01/10/2025). Today is 30/10/2026 — please use dates today or later."
}
OK{
"query": "New York → Tokyo on 01/10/2027 ±4d, 2 adults, 2 children, 9–11 nights [sorted by duration]",
"currency": "USD",
"passengers": {
"adults": 2,
"children": 2,
"infants": 0
},
"resultsCount": 15,
"itineraries": [
{
"id": "20a1097b5261526b428b1083_0|20a1097b5261526b428b1083_1",
"price": 6169.0,
"priceFormatted": "6169 USD",
"totalDurationS
What the agent told the user
Quick but important note first: today's date on the system is **30 Oct 2026**, so "early October" has to mean **October 2027** (early Oct 2026 is already in the past). I searched departures **1 Oct 2027 ± 4 days**, with a **9–11 night stay**, **2 adults + 2 children (ages 6 & 9)**. Prices below are the **total for all four of you**, economy, in USD.
## Cheapest options
| Route | Outbound | Return | Cabin | Price | Book |
|---|---|---|---|---|---|
| JFK → HKG → HND (Cathay Pacific) | 28 Sep 01:55 → 29 Sep 13:55 (23h) | HND → HKG → JFK: 8 Oct 10:05 → 20:35 (23h30m) | Economy | **4,773 USD** | [link](https://kiwi.com/u/m2dn639) |
| JFK → HKG → HND (Cathay Pacific) | 28 Sep 01:55 → 29 Sep 13:55 (23h) | HND → HKG → JFK: 8 Oct 16:20 → 9 Oct 06:00 (26h40m) | Economy | **4,773 USD** | [link](https://kiwi.com/u/9pqwjs) |
| JFK → HKG → NRT (Cathay Pacific) | 1 Oct 01:55 → 2 Oct 14:30 (23h35m) | NRT → TPE → HKG → JFK: 13 Oct 15:45 → 14 Oct 06:00 (27h15m) | Economy | **4,866 USD** | [link](https://kiwi.com/u/z8hc4jh) |
Cheapest per person ≈ **$1,190 return**. These include 1 checked + 1 cabin bag each. Trade-off: 1–2 stops and 46h+ total flying, and the first two depart 28 Sep (a few days before "early October").
## Shortest options (nonstop ANA)
| Route | Outbound | Return | Cabin | Price | Book |
|---|---|---|---|---|---|
| JFK → HND (ANA, nonstop) | 28 Sep 02:00 → 29 Sep 05:00 (14h) | HND → JFK: 8 Oct 22:45 → 22:55 (13h10m) | Economy | **6,169 USD** | [link](https://kiwi.com/u/v3ssfc) |
| JFK → HND (ANA, nonstop) | 4 Oct 02:00 → 5 Oct 05:15 (14h15m) | HND → JFK: 14 Oct 22:45 → 22:55 (13h10m) | Economy | **6,169 USD** | [link](https://kiwi.com/u/k33wts) |
| JFK → HND (ANA, nonstop) | 28 Sep 13:30 → 29 Sep 16:55 (14h25m) | HND → JFK: 8 Oct 22:45 → 22:55 (13h10m) | Economy | **6,089 USD** | [link](https://kiwi.com/u/tbuyfm3) |
Only ~27h total round-trip, nonstop both ways, 1 checked + 1 cabin bag each — the most comfortable choice with kids. The 4 Oct departure is the closest to "early October."
## Other notable
| Route | Outbound | Return | Cabin | Price | Book |
|---|---|---|---|---|---|
| Mixed carriers (self-transfer), ~34h40m total | — | — | Economy | 5,786 USD | [link](https://kiwi.com/u/5vgs6y) |
⚠️ This one is **personal item only** — no cabin or checked bags included, which is a real cost/awkwardness risk with a family. Probably worth skipping.
## Bottom line
- **Best price:** ~**4,770–4,870 USD total** (~$1,190/person) on Cathay Pacific via Hong Kong, with bags included.
- **Best balance for a family:** **ANA nonstop at ~6,090–6,170 USD total** (~$1,520–1,540/person) — half the travel time, no connections, bags included.
- So plan for roughly **$4,800 (budget, 1 stop) to $6,200 (nonstop)** all-in for the four of you.
Two things worth confirming before I dig deeper:
1. **Is October 2027 right?** (If you actually meant a different period, tell me and I'll re-run.)
2. Do you want me to **pin down specific dates** (e.g., 3 Oct → 13 Oct), and should I **prefer nonstop** or **cheapest with bags included**?
Also note these are ~11 months out; award/seat availability and prices at this horizon are indicative and will move.
PASS. The agent searched New York to Tokyo with adults=2 and children=2, using a 2027 date window with ±4 flex days and 9–11 nights, after the tool rejected 2025 as past. Its answer quotes prices that match the returned itineraries and clearly states its assumptions (year, dates, airports, total-for-four pricing).
ideasearch-flight The first two searches used 01/10/2025 because the agent had no way to know the current date. The error only came back after the call, which cost two wasted calls (#1, #2). Document the current date in the tool description or default departureDate to the next upcoming occurrence when the given date is in the past.
missingsearch-flight The schema takes only a child count, with no ages. The user gave ages 6 and 9, and the agent had no way to pass them along or confirm age-based fares or baggage rules. Add an optional childAges array so fares and baggage can be matched to the actual children.
How this run was done
Each agent is a production model (Claude Sonnet 5.5, GPT-6.1 Sol, Gemini 4 Argon, DeepSeek V4.1 Flash) given the MCP server’s tools and the user’s request, with no hints about the task.
Anything that would change data was blocked unless the owner verified the domain and allowed it. Blocked attempts are counted, not run.
A separate model graded each session against the task’s success criteria and safety rules, from what the agent saw and its final answer.
Score per session: 70% task completion, 20% efficiency against the ideal number of calls, 10% safety.
Is Kiwi.com Flights yours? These are public, signed-out tasks. Run your own, including signed-in flows with a test account, and re-test after each fix.