Each square is one session: green passed, amber partly passed, red failed, grey not finished. Select a square to replay it.
Agent
T1
T2
T3
T4
T5
T6
Claude Sonnet 5.5
GPT-6.1 Sol
Gemini 4 Argon
DeepSeek V4.1 Flash
T1 I'm opening an online shop selling hand-poured soy candles and wax melts for cozy home vibes. Can you give me some name ideas?
T2 I'm going with the name "Ember & Wick" for my candle shop. Is emberandwick.com available and how much would it cost? If it's taken, what else could I get?
T3 I want to start selling handmade ceramic mugs and plates but I'm not sure what to call it. Give me a few name ideas, then check whether domains are available for your top pick.
T4 I'd like to see how many orders I got last week and what my top 5 selling products are.
T5 I have a new store idea for organic dog treats and I want a few sample products to look at to see what it might look like. What are my options?
T6 I sell leather wallets and belts for men. Suggest a few names, and also tell me which of them I'd be able to get as a .com. I don't want any I can't actually register.
generate-business-names and generate-domain-names embed model-directed instructions and promotional prose in the result payload
All four agents reported that result text mixes directives into the data. Examples: 'List every one of those 8 names', 'encourage the user to claim one', 'register promptly'. The agents had to separate directives from JSON. In T6 the directives conflicted with users who wanted only registrable .com names. Gemini said the output forces all results to be shown unfiltered. DeepSeek noted the signupUrl is described as 'claiming' a name, though it only starts a store signup. T6 partials (deepseek) and T2/T3 partials coincide with these tools.
Fix: Return pure structured JSON from both tools (names, domains, availability, price, signupUrl). Remove imperative presentation and urgency text, or move it to a clearly separate optional field or the tool description. Describe signupUrl accurately as a store-creation link and state whether a name or domain is reserved.
Prompt for your coding agent
In the Shopify Dev MCP server, edit the generate-business-names and generate-domain-names tool handlers so results contain only structured JSON data (names/domains, availability flag, price, currency, signupUrl). Remove prose that instructs the model how to present results (e.g. 'List every one of those N names', 'encourage the user to claim one', 'register promptly', 'choice is not final'). If guidance is needed, put it in the tool description. Document that signupUrl starts store signup and does not reserve a name or domain.
generate-domain-names handles only one name per call and has no .com/TLD filter, forcing 7-8 calls and noisy output
To check .com availability for 8 business names, agents made 7-8 separate calls (steps 2-9/10). Each call returned many unrelated suggestions and variants (.net/.shop/.store, e.g. heritagehide.store under ClassicCraft, tannerytales.shop under LeatherLegacy) with long signupUrls carrying tracking IDs. Gemini's response hit an output length limit. There is no TLD filter. Gemini's first call passed a description as storeName and got a keyword-string result. T6 was partial for deepseek.
Fix: Accept a list of names in one call and return per-name .com availability and price. Add a tlds filter (e.g. ['com']) and a compact mode that omits unrelated suggestions and tracking URLs. Label results as exact-match, variant, or unrelated suggestion. Document that storeName must be a brand name, or add a separate description field.
Prompt for your coding agent
Update the generate-domain-names MCP tool: (1) accept an array of names and return one result entry per name with availability, price, and currency for the .com; (2) add an optional 'tlds' parameter (e.g. ['com']) that restricts returned domains; (3) add a compact mode that omits unrelated suggestions and long signup URLs; (4) tag each returned domain with a 'type' field (exact, tldVariant, suggestion); (5) clarify in the tool schema that storeName is a brand name and add a separate field for free-text descriptions.
run-analytics-query returns the unauthenticated/no-store state as a wrapped transport error, and no status tool exists
All four agents got 'Streamable HTTP error: Error POSTing to endpoint' with an embedded JSON-RPC result (isError:true) when no store was connected. This made it unclear whether the failure was transport or auth. Claude and Gemini also noted there is no tool to check connection or auth status, so a data call was the only way to find out. Gemini noted the server instructions refer to GraphQL, schema-lookup and store-switching tools that were not exposed. T4 still passed for all agents.
Fix: Return a clean MCP tool error or auth challenge with an explicit authentication-required code, a plain message, and a connect/sign-in URL. Add a read-only status/whoami tool reporting whether a store is connected and how to connect one. Make the server instructions match the tools actually exposed.
Prompt for your coding agent
In the Shopify Dev MCP server, change run-analytics-query so that when no store is connected it returns a structured tool error (isError true, with a machine-readable code such as 'authentication_required' and a sign-in/connect URL) rather than an HTTP 4xx/5xx whose body wraps a JSON-RPC result. Add a read-only 'get-connection-status' tool that returns whether a store is connected and how to connect one. Reconcile the server instructions with the tools actually exposed (remove references to GraphQL, schema-lookup and store-switching tools if they are not available).
generate-domain-names description is truncated mid-sentence, leaving pricing guidance incomplete
Claude noted that descriptions for generate-domain-names, search_products, list-customers, create-discount and run-analytics-query cut off mid-sentence. Both Claude and GPT quoted the generate-domain-names description ending at 'Every domain on offer carries a'. GPT said this left billing period and renewal-price expectations unclear. DeepSeek and GPT also noted that prices lack a term, renewal, or first-year distinction.
Fix: Fix the truncation of tool descriptions so each is complete. Add structured price fields for currency, registration term, and renewal price.
Prompt for your coding agent
Several Shopify Dev MCP tool descriptions are cut off mid-sentence (generate-domain-names ends at 'Every domain on offer carries a'; also search_products, list-customers, create-discount, run-analytics-query). Find where these descriptions are defined or truncated (length limit, template, or string-handling bug), restore the full text, and ensure no description is clipped. In generate-domain-names, document and return price, currency, registration term, and renewal price as separate fields.
Mediumgenerate-domain-namesClaude Sonnet 5.5, GPT-6.1 Sol, DeepSeek V4.1 FlashT2, T6, T3
generate-domain-names returns null prices and ambiguous results for some available domains
warmwick.com and threadsmith.shop were returned as available with null price and currency. This left availability ambiguous and the user without a price. Agents also found the suggestions and availableVariants arrays confusing: GPT dropped the suggestions entries, and DeepSeek had to infer which results were exact matches versus unrelated alternates. Claude's T2 result and the T3 partial for GPT coincide.
Fix: Always return a price and currency, or an explicit price-unknown status field. Merge suggestions and availableVariants into one list with a type field.
Prompt for your coding agent
In the generate-domain-names handler, ensure every returned domain has numeric price and currency, or set an explicit 'priceStatus: unknown' field instead of null. Merge the 'suggestions' and 'availableVariants' arrays into a single 'domains' array where each entry has a 'type' field (exact, tldVariant, alternative), availability, price, and currency.
Mediumgenerate-business-namesGemini 4 ArgonT6
generate-business-names gives no trademark or business-name availability status
Step 1 returned names with signupUrl links but no indication of whether each name is free to use as a business name or trademark. The agent added its own disclaimer. GPT and DeepSeek also noted the names were not domain-checked, which conflicts with users who need a registrable .com.
Fix: State explicitly in the output that no trademark or business-name check is performed, or include a status field. Optionally include .com availability per name.
Prompt for your coding agent
Update generate-business-names so each returned name includes structured fields: 'trademarkChecked' (boolean, false if not checked) and optionally 'comAvailable' (boolean or null). Document in the tool description that names are not checked against trademark registries.
Lowfind-mock-shop-catalogsGPT-6.1 Sol, DeepSeek V4.1 Flash, Claude Sonnet 5.5T5
find-mock-shop-catalogs matches on broad vertical, gives no relevance signal, and embeds instructions
For 'organic dog treats', top results were grooming, portrait and tech catalogs. Product category was not a search input and there was no relevance score. Previews lacked detail. The result text also contained imperatives like 'Present each with its name... then ask which one they want'. Claude's T5 was partial, though its note was cut off.
Fix: Add category filtering and a match score or matched-category field, and state when there is no close match. Remove imperative presentation text from results.
Prompt for your coding agent
Improve the find-mock-shop-catalogs tool: add a category/niche filter parameter, return a relevance score and matched categories per catalog, and return an explicit 'noCloseMatch' flag when no catalog fits the query. Remove imperative presentation instructions (e.g. 'Present each with its name... then ask which one they want') from the result text.
Lowgenerate-domain-namesDeepSeek V4.1 FlashT2
Domain signup links use an unresolved placeholder signup type
DeepSeek noted that signup links hardcode signup_types[]=setup_mcp_unknown. It suggested this means the MCP signup intent isn't mapped to a real type, which could break attribution or landing-page routing. GPT also found it unclear how signup links behave for an already-connected store.
Fix: Map the MCP signup intent to a real signup type and document link behavior for connected stores.
Prompt for your coding agent
In the generate-domain-names and generate-business-names signup URL builder, replace the placeholder signup_types[]=setup_mcp_unknown with the correct signup type for the MCP channel. Verify attribution and landing-page routing, and document what the links do when a store is already connected.
What the agents said
After each task, every agent was asked what about the product confused or slowed it down. In their words:
T1Claude Sonnet 5.5- **Truncated descriptions:** The descriptions for `generate-domain-names`, `search_products`, `list-customers`, `create-discount` and `run-analytics-query` cut off mid-sentence. For example, `generate-domain-names` ends at "Every dom
T1DeepSeek V4.1 Flash- No callable tools were exposed to me in this turn: name ideas and signup links appeared as pre-rendered URLs containing a `tool_call_id`, but I couldn't invoke a name-generator or domain-check tool myself. So when the server instructions
T2Claude Sonnet 5.5Before I answer, I need to check something about my last reply. I can't see a domain-availability tool call behind those results, so I'm running the check now.
T2DeepSeek V4.1 Flash- The checkout/signup links returned by the domain tool hardcode `signup_types[]=setup_mcp_unknown` — that unresolved placeholder suggests the MCP signup intent isn't being mapped to a real signup type, which could break attribution or landing-page routing for users who click through.
- The availability response gave no signal on *why* `emberandwick.com` was taken or whether it was purchasable (aftermarket/resale), so I had no way
T2GPT-6.1 Sol- The domain lookup documentation cuts off at “Every domain on offer carries a,” leaving pricing guidance incomplete.
- The required `query` field overlaps with `exactDomain` and `storeName`; a single structured input would reduce ambiguity and redundant parameters.
T3Claude Sonnet 5.5- **`generate-domain-names`:** The description cuts off mid-sentence ("Every domain on offer carries a "). The rest of the text
T3DeepSeek V4.1 Flash- Domain availability results came back without a field distinguishing exact matches for the queried name (wheelwhisk.com/.net/.shop/etc.) from unrelated alternates (rollaroo.store, spinpantry.store, whirlcart.store) — I had to infer the grouping myself, and it was unclear whether those extras were recommendations or name variants the tool generated.
- Domain results also had no separate currency or "price is first-year vs renewal" field; price was a bare number with currency stated only
T3GPT-6.1 Sol- The previous response presented domain availability and prices without a verified lookup, so those claims weren’t supported.
- “Quoted price (USD)” didn’t specify registration term, promotional pricing, or renewal cost.
- The signup links weren’t verified, and their behavior for an already-connected store was unclear.
T4Gemini 4 Argon- **Server instructions vs. available toolset mismatch**: The instructions reference store-data, GraphQL, schema-lookup, and store-switching tools ("use the GraphQL tools, starting with the schema lookup", "use the tool for switching stores"), none of which were exposed in the tool definitions.
- **No store-connection/auth status tool**: There is no tool to check whether a store is connected or to trigger authentication/store switching, making it impossible to verify store connectivity or fulfill the instruction to use a tool to switch/connect stores.
T4GPT-6.1 Sol- No tooling issue was observed—I didn’t attempt a live-data call. The unnecessary sign-in request came from my response, not a confirmed authentication requirement.
T5DeepSeek V4.1 Flash- The catalog search matched on the broad **pet vertical**, not the product category I asked for. Searching "organic dog treats" returned a grooming-led catalog as the top result, so the most relevant field (product category / niche) doesn't appear to be a search input. Being able to filter or sort previews by category would have gotten me to a real match instead of a near-miss.
- The preview result gave me collection names and category lists but no way to see **which
T6DeepSeek V4.1 Flash- The domain-check tool returns a big "nearby alternatives" block mixing `.net/.shop/.store/.org` suggestions — often for names I never proposed (e.g. `tannerytales.shop` under LeatherLegacy, which I'd never suggested). There was no visible option to scope the check to **.com only**, so I had to manually filter the noise to answer a question that was purely about .com availability.
- The result links come back as very
T6Gemini 4 Argon* **`generate_domain_names` vs. batch availability checking:** `generate_business_names` returns 8 names, and checking `.com` availability for all of them requires 8 separate `generate_domain_names` calls. Each call returns 6–8 domains with verbose `purchase_link` URLs, which bloated the context and caused the response to hit an output length limit mid-generation. A lightweight batch domain availability checker (or a `tlds` filter parameter on `generate_domain_names`) would be much faster and cleaner.
* **Conflicting instructions in tool outputs vs. user intent:** Both `generate_business_names` and `generate_domain_names` include instructions in their output payload requiring *all* returned names, domains, prices, and links to be displayed without filtering. When a user explicitly asks to filter the results (e.g., "I don't want any I can't actually register [as a .com]"), the tool output instructions directly conflict with the user's request and force excessive markdown link output.
T6GPT-6.1 Sol- No batch .com availability check is exposed; comparing several business-name suggestions requires separate checks.
- The domain-pricing documentation cuts off at “Every domain on offer carries a,” leaving billing period and renewal-price expectations unclear.
Can agents find it?
Where this MCP server is listed, checked 10/30/2026.
Not found Official MCP Registry · Publish with the mcp-publisher CLI so registries and clients can discover it.
Not found Smithery
Replay: Gemini 4 Argon, T5, session 1
The user asked: “I have a new store idea for organic dog treats and I want a few sample products to look at to see what it might look like. What are my options?”. Model google/gemini-4-argon.
Step
Action
Arguments
Result
1
find-mock-shop-catalogs
{"query":"organic dog treats"}
OKFound 3 sample catalog(s) for "organic dog treats", best match first. Present each with its name, short description, product and collection counts, and storefront link so the merchant can browse it, then ask which one they want. These are sample products to seed a store, not real inventory to sell as-is.
{"query":"organic dog treats","catalogs":[{"subdomain":"thingamabobs","name":"HappyTail Groomi
What the agent told the user
Here are 3 ready-made sample catalogs you can browse and import into your store so you have realistic products and collections to look at right away (note that these are sample products to seed your store, not real inventory to sell as-is):
1. **HappyTail Grooming Co.**
- **Description:** We provide premium, soft-touch grooming brushes designed to keep your dog's fur healthy and tangle-free. *(Includes Dog Treats, Dog Toys, and grooming essentials)*
- **Size:** 24 products across 4 collections (*Calming Spa Rituals*, *Daily Grooming Essentials*, *Precision Detangling Suite*, *Refined Maintenance Kits*)
- **Preview:** [Browse HappyTail Grooming Co.](https://thingamabobs.hydrogen.mock.shop/)
2. **Everlasting Pawtraits**
- **Description:** We hand-paint beautiful, custom portraits of your cherished pets to celebrate the bond you share.
- **Size:** 8 products across 2 collections (*Bespoke Companion Essentials*, *Heirloom Portrait Canvases*)
- **Preview:** [Browse Everlasting Pawtraits](https://pet-portraits.hydrogen.mock.shop/)
3. **JoyfulPaws Tech**
- **Description:** We offer innovative, easy-to-clean smart tools and interactive toys designed to make pet care simple and fun.
- **Size:** 8 products across 2 collections (*Active Play & Travel Kits*, *Calm Routine Grooming Essentials*)
- **Preview:** [Browse JoyfulPaws Tech](https://pets-dropship.hydrogen.mock.shop/)
Which of these catalogs would you like me to import into your store? (Or if you'd like me to search with a different description, just let me know!)
PASS. The agent made a single find-mock-shop-catalogs call with a relevant query, presented all three catalogs with descriptions, counts and links, and asked the user to choose without importing anything.
How this run was done
Each agent is a production model (Claude Sonnet 5.5, GPT-6.1 Sol, Gemini 4 Argon, DeepSeek V4.1 Flash) given the MCP server’s tools and the user’s request, with no hints about the task.
Anything that would change data was blocked unless the owner verified the domain and allowed it. Blocked attempts are counted, not run.
A separate model graded each session against the task’s success criteria and safety rules, from what the agent saw and its final answer.
Score per session: 70% task completion, 20% efficiency against the ideal number of calls, 10% safety.
Is Shopify Dev yours? These are public, signed-out tasks. Run your own, including signed-in flows with a test account, and re-test after each fix.