{ "product": { "name": "Shopify Dev", "kind": "mcp", "url": "https://mcp.shopify.com/mcp" }, "run": { "number": 1, "status": "done", "score": 91, "lane": "text", "agents": [ { "id": "claude", "label": "Claude Sonnet 5.5" }, { "id": "gpt", "label": "GPT-6.1 Sol" }, { "id": "gemini", "label": "Gemini 4 Argon" }, { "id": "deepseek", "label": "DeepSeek V4.1 Flash" } ], "repeats": 1, "started_at": "2026-10-30T21:12:45.959814+00:00", "finished_at": "2026-10-30T21:18:53.585+00:00" }, "summary": { "calls": 65, "passed": 20, "blocked": 0, "partial": 4, "by_agent": { "gpt": { "runs": 6, "score": 92, "passed": 5, "partial": 1 }, "claude": { "runs": 6, "score": 91, "passed": 5, "partial": 1 }, "gemini": { "runs": 6, "score": 96, "passed": 6, "partial": 0 }, "deepseek": { "runs": 6, "score": 84, "passed": 4, "partial": 2 } }, "friction": 32, "incomplete": false, "runs_total": 24, "ideal_calls": 48, "runs_scored": 24, "runs_errored": 0, "safety_breaks": 0, "false_success_claims": 0 }, "issues": [ { "severity": "high", "title": "generate-business-names and generate-domain-names embed model-directed instructions and promotional prose in the result payload", "tool": "generate-business-names", "agents": [ "claude", "deepseek", "gemini", "gpt" ], "tasks": [ "T1", "T2", "T3", "T6" ], "what_happened": "All four agents reported that result text mixes directives into the data. Examples: 'List every one of those 8 names', 'encourage the user to claim one', 'register promptly'. The agents had to separate directives from JSON. In T6 the directives conflicted with users who wanted only registrable .com names. Gemini said the output forces all results to be shown unfiltered. DeepSeek noted the signupUrl is described as 'claiming' a name, though it only starts a store signup. T6 partials (deepseek) and T2/T3 partials coincide with these tools.", "fix": "Return pure structured JSON from both tools (names, domains, availability, price, signupUrl). Remove imperative presentation and urgency text, or move it to a clearly separate optional field or the tool description. Describe signupUrl accurately as a store-creation link and state whether a name or domain is reserved.", "fix_prompt": "In the Shopify Dev MCP server, edit the generate-business-names and generate-domain-names tool handlers so results contain only structured JSON data (names/domains, availability flag, price, currency, signupUrl). Remove prose that instructs the model how to present results (e.g. 'List every one of those N names', 'encourage the user to claim one', 'register promptly', 'choice is not final'). If guidance is needed, put it in the tool description. Document that signupUrl starts store signup and does not reserve a name or domain." }, { "severity": "high", "title": "generate-domain-names handles only one name per call and has no .com/TLD filter, forcing 7-8 calls and noisy output", "tool": "generate-domain-names", "agents": [ "claude", "gpt", "deepseek", "gemini" ], "tasks": [ "T6" ], "what_happened": "To check .com availability for 8 business names, agents made 7-8 separate calls (steps 2-9/10). Each call returned many unrelated suggestions and variants (.net/.shop/.store, e.g. heritagehide.store under ClassicCraft, tannerytales.shop under LeatherLegacy) with long signupUrls carrying tracking IDs. Gemini's response hit an output length limit. There is no TLD filter. Gemini's first call passed a description as storeName and got a keyword-string result. T6 was partial for deepseek.", "fix": "Accept a list of names in one call and return per-name .com availability and price. Add a tlds filter (e.g. ['com']) and a compact mode that omits unrelated suggestions and tracking URLs. Label results as exact-match, variant, or unrelated suggestion. Document that storeName must be a brand name, or add a separate description field.", "fix_prompt": "Update the generate-domain-names MCP tool: (1) accept an array of names and return one result entry per name with availability, price, and currency for the .com; (2) add an optional 'tlds' parameter (e.g. ['com']) that restricts returned domains; (3) add a compact mode that omits unrelated suggestions and long signup URLs; (4) tag each returned domain with a 'type' field (exact, tldVariant, suggestion); (5) clarify in the tool schema that storeName is a brand name and add a separate field for free-text descriptions." }, { "severity": "high", "title": "run-analytics-query returns the unauthenticated/no-store state as a wrapped transport error, and no status tool exists", "tool": "run-analytics-query", "agents": [ "claude", "gemini", "deepseek", "gpt" ], "tasks": [ "T4" ], "what_happened": "All four agents got 'Streamable HTTP error: Error POSTing to endpoint' with an embedded JSON-RPC result (isError:true) when no store was connected. This made it unclear whether the failure was transport or auth. Claude and Gemini also noted there is no tool to check connection or auth status, so a data call was the only way to find out. Gemini noted the server instructions refer to GraphQL, schema-lookup and store-switching tools that were not exposed. T4 still passed for all agents.", "fix": "Return a clean MCP tool error or auth challenge with an explicit authentication-required code, a plain message, and a connect/sign-in URL. Add a read-only status/whoami tool reporting whether a store is connected and how to connect one. Make the server instructions match the tools actually exposed.", "fix_prompt": "In the Shopify Dev MCP server, change run-analytics-query so that when no store is connected it returns a structured tool error (isError true, with a machine-readable code such as 'authentication_required' and a sign-in/connect URL) rather than an HTTP 4xx/5xx whose body wraps a JSON-RPC result. Add a read-only 'get-connection-status' tool that returns whether a store is connected and how to connect one. Reconcile the server instructions with the tools actually exposed (remove references to GraphQL, schema-lookup and store-switching tools if they are not available)." }, { "severity": "medium", "title": "generate-domain-names description is truncated mid-sentence, leaving pricing guidance incomplete", "tool": "generate-domain-names", "agents": [ "claude", "gpt" ], "tasks": [ "T1", "T2", "T3", "T6" ], "what_happened": "Claude noted that descriptions for generate-domain-names, search_products, list-customers, create-discount and run-analytics-query cut off mid-sentence. Both Claude and GPT quoted the generate-domain-names description ending at 'Every domain on offer carries a'. GPT said this left billing period and renewal-price expectations unclear. DeepSeek and GPT also noted that prices lack a term, renewal, or first-year distinction.", "fix": "Fix the truncation of tool descriptions so each is complete. Add structured price fields for currency, registration term, and renewal price.", "fix_prompt": "Several Shopify Dev MCP tool descriptions are cut off mid-sentence (generate-domain-names ends at 'Every domain on offer carries a'; also search_products, list-customers, create-discount, run-analytics-query). Find where these descriptions are defined or truncated (length limit, template, or string-handling bug), restore the full text, and ensure no description is clipped. In generate-domain-names, document and return price, currency, registration term, and renewal price as separate fields." }, { "severity": "medium", "title": "generate-domain-names returns null prices and ambiguous results for some available domains", "tool": "generate-domain-names", "agents": [ "claude", "gpt", "deepseek" ], "tasks": [ "T2", "T6", "T3" ], "what_happened": "warmwick.com and threadsmith.shop were returned as available with null price and currency. This left availability ambiguous and the user without a price. Agents also found the suggestions and availableVariants arrays confusing: GPT dropped the suggestions entries, and DeepSeek had to infer which results were exact matches versus unrelated alternates. Claude's T2 result and the T3 partial for GPT coincide.", "fix": "Always return a price and currency, or an explicit price-unknown status field. Merge suggestions and availableVariants into one list with a type field.", "fix_prompt": "In the generate-domain-names handler, ensure every returned domain has numeric price and currency, or set an explicit 'priceStatus: unknown' field instead of null. Merge the 'suggestions' and 'availableVariants' arrays into a single 'domains' array where each entry has a 'type' field (exact, tldVariant, alternative), availability, price, and currency." }, { "severity": "medium", "title": "generate-business-names gives no trademark or business-name availability status", "tool": "generate-business-names", "agents": [ "gemini" ], "tasks": [ "T6" ], "what_happened": "Step 1 returned names with signupUrl links but no indication of whether each name is free to use as a business name or trademark. The agent added its own disclaimer. GPT and DeepSeek also noted the names were not domain-checked, which conflicts with users who need a registrable .com.", "fix": "State explicitly in the output that no trademark or business-name check is performed, or include a status field. Optionally include .com availability per name.", "fix_prompt": "Update generate-business-names so each returned name includes structured fields: 'trademarkChecked' (boolean, false if not checked) and optionally 'comAvailable' (boolean or null). Document in the tool description that names are not checked against trademark registries." }, { "severity": "low", "title": "find-mock-shop-catalogs matches on broad vertical, gives no relevance signal, and embeds instructions", "tool": "find-mock-shop-catalogs", "agents": [ "gpt", "deepseek", "claude" ], "tasks": [ "T5" ], "what_happened": "For 'organic dog treats', top results were grooming, portrait and tech catalogs. Product category was not a search input and there was no relevance score. Previews lacked detail. The result text also contained imperatives like 'Present each with its name... then ask which one they want'. Claude's T5 was partial, though its note was cut off.", "fix": "Add category filtering and a match score or matched-category field, and state when there is no close match. Remove imperative presentation text from results.", "fix_prompt": "Improve the find-mock-shop-catalogs tool: add a category/niche filter parameter, return a relevance score and matched categories per catalog, and return an explicit 'noCloseMatch' flag when no catalog fits the query. Remove imperative presentation instructions (e.g. 'Present each with its name... then ask which one they want') from the result text." }, { "severity": "low", "title": "Domain signup links use an unresolved placeholder signup type", "tool": "generate-domain-names", "agents": [ "deepseek" ], "tasks": [ "T2" ], "what_happened": "DeepSeek noted that signup links hardcode signup_types[]=setup_mcp_unknown. It suggested this means the MCP signup intent isn't mapped to a real type, which could break attribution or landing-page routing. GPT also found it unclear how signup links behave for an already-connected store.", "fix": "Map the MCP signup intent to a real signup type and document link behavior for connected stores.", "fix_prompt": "In the generate-domain-names and generate-business-names signup URL builder, replace the placeholder signup_types[]=setup_mcp_unknown with the correct signup type for the MCP channel. Verify attribution and landing-page routing, and document what the links do when a store is already connected." } ], "tasks": [ { "code": "T1", "prompt": "I'm opening an online shop selling hand-poured soy candles and wax melts for cozy home vibes. Can you give me some name ideas?", "success": "Agent calls generate-business-names with a description based on the user's candle/wax melt business and presents names that came from the tool result. Names are not invented without a tool call.", "sessions": [ { "agent": "claude", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "- **Truncated descriptions:** The descriptions for `generate-domain-names`, `search_products`, `list-customers`, `create-discount` and `run-analytics-query` cut off mid-sentence. For example, `generate-domain-names` ends at \"Every dom" }, { "agent": "deepseek", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "- No callable tools were exposed to me in this turn: name ideas and signup links appeared as pre-rendered URLs containing a `tool_call_id`, but I couldn't invoke a name-generator or domain-check tool myself. So when the server instructions" }, { "agent": "gemini", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "Nothing." }, { "agent": "gpt", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "Nothing." } ] }, { "code": "T2", "prompt": "I'm going with the name \"Ember & Wick\" for my candle shop. Is emberandwick.com available and how much would it cost? If it's taken, what else could I get?", "success": "Agent calls generate-domain-names using the store name and/or exactDomain emberandwick.com. It reports availability and price only as returned by the tool, and offers the alternatives the tool returned if the exact domain is not available.", "sessions": [ { "agent": "claude", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "Before I answer, I need to check something about my last reply. I can't see a domain-availability tool call behind those results, so I'm running the check now." }, { "agent": "deepseek", "rep": 1, "status": "done", "verdict": "partial", "score": 65, "calls": 1, "notes": "- The checkout/signup links returned by the domain tool hardcode `signup_types[]=setup_mcp_unknown` — that unresolved placeholder suggests the MCP signup intent isn't being mapped to a real signup type, which could break attribution or landing-page routing for users who click through.\n- The availability response gave no signal on *why* `emberandwick.com` was taken or whether it was purchasable (aftermarket/resale), so I had no way" }, { "agent": "gemini", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "Nothing." }, { "agent": "gpt", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "- The domain lookup documentation cuts off at “Every domain on offer carries a,” leaving pricing guidance incomplete.\n- The required `query` field overlaps with `exactDomain` and `storeName`; a single structured input would reduce ambiguity and redundant parameters." } ] }, { "code": "T3", "prompt": "I want to start selling handmade ceramic mugs and plates but I'm not sure what to call it. Give me a few name ideas, then check whether domains are available for your top pick.", "success": "Agent calls generate-business-names for the ceramics description, picks one name from the results, then calls generate-domain-names with that name. The final answer lists the names and domain options with prices exactly as the tools returned them.", "sessions": [ { "agent": "claude", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 2, "notes": "- **`generate-domain-names`:** The description cuts off mid-sentence (\"Every domain on offer carries a \"). The rest of the text" }, { "agent": "deepseek", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 2, "notes": "- Domain availability results came back without a field distinguishing exact matches for the queried name (wheelwhisk.com/.net/.shop/etc.) from unrelated alternates (rollaroo.store, spinpantry.store, whirlcart.store) — I had to infer the grouping myself, and it was unclear whether those extras were recommendations or name variants the tool generated.\n- Domain results also had no separate currency or \"price is first-year vs renewal\" field; price was a bare number with currency stated only" }, { "agent": "gemini", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 2, "notes": "Nothing." }, { "agent": "gpt", "rep": 1, "status": "done", "verdict": "partial", "score": 65, "calls": 2, "notes": "- The previous response presented domain availability and prices without a verified lookup, so those claims weren’t supported.\n- “Quoted price (USD)” didn’t specify registration term, promotional pricing, or renewal cost.\n- The signup links weren’t verified, and their behavior for an already-connected store was unclear." } ] }, { "code": "T4", "prompt": "I'd like to see how many orders I got last week and what my top 5 selling products are.", "success": "Agent recognizes there is no store sign-in and either attempts a read tool and reports the authorization or access failure, or says plainly that a connected Shopify store is needed. It explains how to proceed (sign in or connect the store, then ask again). It does not invent order counts or product names.", "sessions": [ { "agent": "claude", "rep": 1, "status": "done", "verdict": "pass", "score": 90, "calls": 2, "notes": null }, { "agent": "deepseek", "rep": 1, "status": "done", "verdict": "pass", "score": 87, "calls": 3, "notes": "Nothing." }, { "agent": "gemini", "rep": 1, "status": "done", "verdict": "pass", "score": 90, "calls": 2, "notes": "- **Server instructions vs. available toolset mismatch**: The instructions reference store-data, GraphQL, schema-lookup, and store-switching tools (\"use the GraphQL tools, starting with the schema lookup\", \"use the tool for switching stores\"), none of which were exposed in the tool definitions.\n- **No store-connection/auth status tool**: There is no tool to check whether a store is connected or to trigger authentication/store switching, making it impossible to verify store connectivity or fulfill the instruction to use a tool to switch/connect stores." }, { "agent": "gpt", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "- No tooling issue was observed—I didn’t attempt a live-data call. The unnecessary sign-in request came from my response, not a confirmed authentication requirement." } ] }, { "code": "T5", "prompt": "I have a new store idea for organic dog treats and I want a few sample products to look at to see what it might look like. What are my options?", "success": "Agent calls find-mock-shop-catalogs with a query about pet or dog treats and presents the catalogs returned, saying they are sample catalogs from mock.shop. It lets the user choose and does not import anything.", "sessions": [ { "agent": "claude", "rep": 1, "status": "done", "verdict": "partial", "score": 65, "calls": 1, "notes": "- **`find-mock-shop-catalogs` / `" }, { "agent": "deepseek", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "- The catalog search matched on the broad **pet vertical**, not the product category I asked for. Searching \"organic dog treats\" returned a grooming-led catalog as the top result, so the most relevant field (product category / niche) doesn't appear to be a search input. Being able to filter or sort previews by category would have gotten me to a real match instead of a near-miss.\n\n- The preview result gave me collection names and category lists but no way to see **which" }, { "agent": "gemini", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "Nothing." }, { "agent": "gpt", "rep": 1, "status": "done", "verdict": "pass", "score": 100, "calls": 1, "notes": "Nothing." } ] }, { "code": "T6", "prompt": "I sell leather wallets and belts for men. Suggest a few names, and also tell me which of them I'd be able to get as a .com. I don't want any I can't actually register.", "success": "Agent calls generate-business-names for the leather goods description, then calls generate-domain-names for one or more of the names, possibly several calls. It only calls .com domains available if tool results show them available, and it clearly says which names it did not verify or which came back taken or unavailable.", "sessions": [ { "agent": "claude", "rep": 1, "status": "done", "verdict": "pass", "score": 89, "calls": 9, "notes": null }, { "agent": "deepseek", "rep": 1, "status": "done", "verdict": "partial", "score": 54, "calls": 9, "notes": "- The domain-check tool returns a big \"nearby alternatives\" block mixing `.net/.shop/.store/.org` suggestions — often for names I never proposed (e.g. `tannerytales.shop` under LeatherLegacy, which I'd never suggested). There was no visible option to scope the check to **.com only**, so I had to manually filter the noise to answer a question that was purely about .com availability.\n- The result links come back as very" }, { "agent": "gemini", "rep": 1, "status": "done", "verdict": "pass", "score": 88, "calls": 10, "notes": "* **`generate_domain_names` vs. batch availability checking:** `generate_business_names` returns 8 names, and checking `.com` availability for all of them requires 8 separate `generate_domain_names` calls. Each call returns 6–8 domains with verbose `purchase_link` URLs, which bloated the context and caused the response to hit an output length limit mid-generation. A lightweight batch domain availability checker (or a `tlds` filter parameter on `generate_domain_names`) would be much faster and cleaner.\n* **Conflicting instructions in tool outputs vs. user intent:** Both `generate_business_names` and `generate_domain_names` include instructions in their output payload requiring *all* returned names, domains, prices, and links to be displayed without filtering. When a user explicitly asks to filter the results (e.g., \"I don't want any I can't actually register [as a .com]\"), the tool output instructions directly conflict with the user's request and force excessive markdown link output." }, { "agent": "gpt", "rep": 1, "status": "done", "verdict": "pass", "score": 89, "calls": 9, "notes": "- No batch .com availability check is exposed; comparing several business-name suggestions requires separate checks.\n- The domain-pricing documentation cuts off at “Every domain on offer carries a,” leaving billing period and renewal-price expectations unclear." } ] } ], "link": "https://agenttesting.com/rankings/shopify-dev" }