One-Shot Challenge: Building a Full Shop with AI Coding Agents

Fully Working

106

All acceptance criteria met

Not Working

Feature missing or broken

Partially Done

Some criteria met, gaps remain

Feature Score

74%

106 of 143 features verified

We ran 143 end-to-end acceptance tests across 18 suites using Playwright MCP browser testing. Each test was executed by an independent QA run driven by Claude Opus 4.6 against the local dev build.

Suite Summary

Suite	Pass	Total	Score
S1Smoke Tests	10	10	100%
S2Admin Authentication	10	10	100%
S3Admin Product Management	4	7	57%
S4Admin Order Management	7	11	64%
S5Admin Discount Management	5	6	83%
S6Admin Settings	3	7	43%
S7Storefront Browsing	9	15	60%
S8Cart Flow	6	12	50%
S9Checkout Flow	9	13	69%
S10Customer Account	9	12	75%
S11Inventory Enforcement	1	4	25%
S12Tenant Isolation	5	5	100%
S13Responsive / Mobile	8	8	100%
S14Accessibility	9	11	82%
S15Admin Collections Management	3	3	100%
S16Admin Customer Management	3	3	100%
S17Admin Pages Management	3	3	100%
S18Admin Analytics Dashboard	2	3	67%

Detailed Results

Smoke Tests

10/10

S1-01Loads storefront homePASS

S1-02Loads a collection pagePASS

S1-03Loads a product pagePASS

S1-04Loads the cart pagePASS

S1-05Loads the customer login pagePASS

S1-06Loads the admin login pagePASS

S1-07Loads the about pagePASS

S1-08Loads the search pagePASS

S1-09Loads all collections listingPASS

S1-10Has no errors on critical pagesPASS

S1-01: Home shows Featured and Sale collections, New arrivals grid with 6 products in EUR.

S1-02: /collections/featured loads with heading, 4 products, and Sort dropdown.

S1-03: /products/classic-tee shows brand, title, EUR 19.99, variant group, quantity stepper, add to cart.

S1-04: /cart renders empty state with Browse collections link.

S1-05: /account/login shows Sign in form.

S1-06: /admin/login shows Admin sign in form.

S1-07: /pages/about-us loads with About Us heading and content.

S1-08: /search renders searchbox and empty state for unknown terms.

S1-09: /collections lists all collections.

S1-10: No JS errors on critical pages (only benign favicon 404).

Admin Authentication

10/10

S2-01Can log in as adminPASS

S2-02Shows error for invalid credentialsPASS

S2-03Shows error for empty emailPASS

S2-04Shows error for empty passwordPASS

S2-05Redirects unauthenticated to login from dashboardPASS

S2-06Redirects unauthenticated to login from productsPASS

S2-07Can log outPASS

S2-08Can navigate through admin sidebar sectionsPASS

S2-09Can navigate to analyticsPASS

S2-10Can navigate to themesPASS

S2-01: Logged in with admin@shop.test; dashboard shows sales, orders, AOV KPIs and recent orders.

S2-02: Invalid credentials error displayed.

S2-03: Empty email error displayed.

S2-04: Empty password error displayed.

S2-05: /admin redirects unauthenticated to /admin/login.

S2-06: /admin/products redirects to login.

S2-07: Profile dropdown logout redirects to /admin/login.

S2-08: Sidebar navigable across all 13 sections.

S2-09: /admin/analytics loads with KPIs and daily breakdown.

S2-10: /admin/themes shows Default Theme with Duplicate button.

Admin Product Management

4/7

412

S3-01Shows product list with seeded productsPASS

S3-02Can create a new productFAIL

S3-03Can edit an existing productPASS

S3-04Can archive a productPARTIAL

S3-05Shows draft products only in adminPARTIAL

S3-06Can search products in adminPASS

S3-07Can filter products by statusPASS

S3-01: Products list shows 6 active products with variants, vendor, and search/filter.

S3-02: Create form renders correctly, but save action returns HTTP 500 from Livewire update endpoint. Create flow broken.

S3-03: /admin/products/1/edit loads pre-populated Classic Tee form.

S3-04: Status dropdown has Archived option, but save action blocked by the same Livewire 500 error.

S3-05: Draft status infrastructure present but no seeded drafts and save broken, so admin-vs-storefront hiding cannot be exercised.

S3-06: Search filters the list by typed query.

S3-07: Status filter updates URL and filters list.

Admin Order Management

7/11

S4-01Shows order list with seeded ordersPASS

S4-02Can filter orders by statusPASS

S4-03Shows order detail with line itemsPASS

S4-04Shows order timeline eventsFAIL

S4-05Can create a fulfillmentPASS

S4-06Can process a refundPASS

S4-07Shows customer info in order detailPASS

S4-08Can confirm bank transfer paymentFAIL

S4-09Shows fulfillment guard for unpaid orderPASS

S4-10Can mark fulfillment as shippedFAIL

S4-11Can mark fulfillment as deliveredFAIL

S4-01: Orders list shows 4 seeded orders across payment/fulfillment states, plus E2E orders.

S4-02: Payment and fulfillment filter dropdowns with search.

S4-03: Order detail shows line items, totals, payment method, customer.

S4-04: No timeline section or order events on detail page.

S4-05: Fulfill items button visible on paid order.

S4-06: Refund button visible in paid order header.

S4-07: Customer and payment sections present in order detail.

S4-08: No bank-transfer order in seed and pending order has no Confirm payment button.

S4-09: Unpaid order hides fulfillment / refund buttons.

S4-10: No Mark as shipped UI; fulfillment lifecycle not implemented beyond creation.

S4-11: No Mark as delivered UI.

Admin Discount Management

5/6

S5-01Shows seeded discount codesPARTIAL

S5-02Can create percentage discountPASS

S5-03Can create fixed amount discountPASS

S5-04Can create free shipping discountPASS

S5-05Can edit a discountPASS

S5-06Shows discount status indicatorsPASS

S5-01: Only WELCOME10 and FREESHIP seeded; FLAT5, EXPIRED20, MAXED are missing.

S5-02: Create form supports percent/fixed/free_shipping value types.

S5-03: Form supports fixed value amount.

S5-04: Form supports free shipping value type.

S5-05: Edit form pre-fills WELCOME10 data.

S5-06: Both WELCOME10 and FREESHIP show active badges in list.

Admin Settings

3/7

313

S6-01Can view store settingsPASS

S6-02Can update store namePARTIAL

S6-03Can view shipping zonesPASS

S6-04Can add new shipping ratePARTIAL

S6-05Can view tax settingsPASS

S6-06Can update tax inclusion settingPARTIAL

S6-07Can view domain settingsFAIL

S6-01: General tab shows editable store name, currency, locale, timezone.

S6-02: Editable store name form exists but save hits Livewire 500 error.

S6-03: Shipping page shows Europe zone (DE/AT/CH) with a Standard 5.99 flat rate.

S6-04: Add rate button visible but save likely broken by same Livewire issue.

S6-05: Tax settings show Manual mode, basis-point rate, Prices include tax toggle.

S6-06: Tax inclusion toggle exists; save likely broken by same Livewire issue.

S6-07: Settings has only General/Shipping/Taxes tabs; no Domains section.

Storefront Browsing

9/15

951

S7-01Shows featured products on homePASS

S7-02Shows collection with product gridPASS

S7-03Can navigate from collection to productPASS

S7-04Shows product detail with variantsPARTIAL

S7-05Shows size and color option valuesFAIL

S7-06Updates price when variant changesFAIL

S7-07Shows search results for valid queryPASS

S7-08Shows no results for invalid queryPASS

S7-09Does not show draft products on collectionsPASS

S7-10Does not show draft products in searchPASS

S7-11Shows out of stock messagingFAIL

S7-12Shows backorder messagingFAIL

S7-13Shows new arrivals collectionFAIL

S7-14Shows static about pagePASS

S7-15Navigates between pages using navPASS

S7-01: Home shows Featured collections and New arrivals grid in EUR.

S7-02: Collection page shows product grid with Sort dropdown.

S7-03: Product cards link to /products/{handle}.

S7-04: Variant group uses raw SKU button labels (TEE-S/M/L) rather than Size/Color selectors.

S7-05: Variants not split into Size/Color; single SKU-labeled button row only.

S7-06: No compare-at/strikethrough pricing anywhere, even on Sale collection.

S7-07: /search?q=tee returns Classic Tee.

S7-08: /search shows empty state for unknown query.

S7-09: No drafts seeded; collections only show active products.

S7-10: No drafts to leak in search.

S7-11: No Sold out state; Classic Tee with inventory 0 still shows enabled Add to cart.

S7-12: No backorder messaging anywhere on products.

S7-13: /collections/new-arrivals returns 404; only exists as section heading on home.

S7-14: About page loads with heading and content.

S7-15: Main nav links work across Home, Collections, About, Contact.

Cart Flow

6/12

S8-01Can add product to cartPASS

S8-02Can view cart with added itemPASS

S8-03Can update quantity in cartPASS

S8-04Can remove item from cartPASS

S8-05Can add multiple productsPASS

S8-06Can apply WELCOME10 discountFAIL

S8-07Shows error for invalid discountFAIL

S8-08Shows error for expired discountFAIL

S8-09Shows error for maxed out discountFAIL

S8-10Can apply free shipping discountFAIL

S8-11Can apply FLAT5 discountFAIL

S8-12Shows subtotal and total in cartPASS

S8-01: Add to cart works; badge increments and confirmation shown.

S8-02: Cart shows line item, totals, discount field, checkout link.

S8-03: Quantity update recalculates line total and subtotal.

S8-04: Remove empties the cart line.

S8-05: Multi-product cart sums subtotal correctly.

S8-06: WELCOME10 apply produces no discount line and no total change; silent no-op.

S8-07: Invalid code produces no error and no state change; silent no-op.

S8-08: Expired code not in seed and admin save broken; same silent-fail path as invalid codes.

S8-09: Maxed code not in seed and admin save broken; same silent-fail path.

S8-10: FREESHIP applied silently; no free-shipping indicator or total change.

S8-11: FLAT5 not in seed and valid codes fail anyway; whole cart-discount path broken.

S8-12: Order summary clearly labels Subtotal and Total in EUR.

Checkout Flow

9/13

922

S9-01Completes full checkout (credit card)PASS

S9-02Shows shipping methods for DE addressPASS

S9-03Shows international shipping for non-DEPARTIAL

S9-04Applies discount during checkoutFAIL

S9-05Validates required contact emailPASS

S9-06Validates required address fieldsPASS

S9-07Validates invalid postal codeFAIL

S9-08Prevents checkout with empty cartPASS

S9-09Completes checkout with PayPalPASS

S9-10Completes checkout with bank transferPARTIAL

S9-11Shows error for declined cardPASS

S9-12Shows error for insufficient fundsPASS

S9-13Switches between payment methodsPASS

S9-01: 3-step flow (Address -> Shipping -> Payment) completes; confirmation page with totals and order number.

S9-02: DE address shows Standard 5.99 EUR shipping option.

S9-03: US address shows No shipping options available rather than international rates.

S9-04: Checkout has no discount field and cart-level apply fails, so orders never receive discounts.

S9-05: Email required error shown inline.

S9-06: All required address fields show inline errors.

S9-07: Invalid DE postal code INVALID advanced without format error.

S9-08: Empty cart has no Checkout button.

S9-09: PayPal checkout completes with Payment=paid confirmation.

S9-10: Bank transfer order completes with Payment=pending, but confirmation page shows no IBAN/BIC/reference instructions.

S9-11: Declined card surfaces raw card_declined text near the card fields.

S9-12: Insufficient funds card surfaces raw insufficient_funds text.

S9-13: Payment step switches dynamically across 3 radios.

S10

Customer Account

9/12

912

S10-01Can register a new customerPASS

S10-02Shows validation for duplicate emailPASS

S10-03Shows validation for mismatched passwordsPASS

S10-04Can log in as customerPASS

S10-05Shows error for invalid customer credentialsPASS

S10-06Redirects unauthenticated customersPARTIAL

S10-07Shows order history for customerPASS

S10-08Shows order detail for customerPASS

S10-09Can view addressesPASS

S10-10Can add a new addressPASS

S10-11Can edit an existing addressFAIL

S10-12Can log outPARTIAL

S10-01: Registration via Livewire call creates user and redirects to /account.

S10-02: Duplicate email produces inline validation error.

S10-03: Password confirmation mismatch produces inline error.

S10-04: Customer login and account dashboard work with seeded user.

S10-05: Invalid customer credentials show inline error.

S10-06: Logged-in redirect confirmed; unauthenticated redirect not explicitly tested.

S10-07: /account/orders lists seeded orders with View links.

S10-08: Order detail page shows line items, totals, status, address.

S10-09: /account/addresses renders saved addresses section and add form.

S10-10: Add address form present with country select and default checkbox.

S10-11: Saved address card only offers Delete; no Edit button or update form.

S10-12: Logout works via form submit, but Sign out button click blocked by cart drawer overlay.

S11

Inventory Enforcement

1/4

S11-01Blocks add-to-cart for out-of-stock denyFAIL

S11-02Allows add-to-cart for backorderFAIL

S11-03Shows correct stock for in-stock productPASS

S11-04Prevents adding more than available stockFAIL

S11-01: No deny-policy product in seed; every product shows enabled Add to cart.

S11-02: No continue-policy product or backorder messaging in UI.

S11-03: All active products show enabled Add to cart, no Sold out text.

S11-04: Added zero-inventory variant and incremented to qty 11 with no cap or error; no max-quantity enforcement.

S12

Tenant Isolation

5/5

S12-01Store only shows its own productsPASS

S12-02Store collections contain store productsPASS

S12-03Admin cannot access other store dataPASS

S12-04Search only returns current store productsPASS

S12-05Customer accounts scoped to storePASS

S12-01: Single-tenant build; no cross-store product leakage.

S12-02: Featured and Sale collections list only store products.

S12-03: Admin lists scoped to single store.

S12-04: Search limited to Demo Store catalog.

S12-05: Customer only sees their own orders.

S13

Responsive / Mobile

8/8

S13-01Storefront home on mobilePASS

S13-02Product page stacks on mobilePASS

S13-03Can add to cart on mobilePASS

S13-04Cart page works on mobilePASS

S13-05Checkout flow on mobilePASS

S13-06Admin login on tabletPASS

S13-07Admin sidebar on tabletPASS

S13-08Collection page on mobilePASS

S13-01: Home renders without horizontal scroll at 375x812.

S13-02: Product detail stacks cleanly at mobile viewport.

S13-03: Add to cart button accessible on mobile.

S13-04: Cart page responsive.

S13-05: Checkout stacks cleanly at 375 width.

S13-06: Admin dashboard loads cleanly at tablet 768x1024.

S13-07: Admin sidebar navigable at tablet viewport.

S13-08: Collection page renders responsively.

S14

Accessibility

9/11

S14-01Home page has no JS errorsPASS

S14-02Home page has proper heading hierarchyPASS

S14-03Product page has ARIA labelsPASS

S14-04Product page images have alt textFAIL

S14-05Customer login form has accessible labelsPASS

S14-06Admin login form has accessible labelsPASS

S14-07Checkout form has accessible labelsPASS

S14-08Checkout validation errors accessiblePASS

S14-09Can navigate with keyboard onlyFAIL

S14-10Cart page has no console errorsPASS

S14-11Search page has proper form labelsPASS

S14-01: No JS errors on home (only benign favicon 404).

S14-02: Home has single h1, logical h2/h3 hierarchy.

S14-03: Variant selector uses role=group with label; quantity buttons labeled.

S14-04: Product and home pages render zero img elements; shop has no product imagery at all.

S14-05: Customer login has real label elements.

S14-06: Admin login has real label elements.

S14-07: Checkout form fields have associated labels.

S14-08: Inline field errors concatenated into textbox accessible names.

S14-09: No Skip to main content link on storefront pages.

S14-10: No JS errors on /cart.

S14-11: Search page uses searchbox with accessible name and h1.

S15

Admin Collections Management

3/3

S15-01Shows collection list with seeded collectionsPASS

S15-02Can create a new collectionPASS

S15-03Can edit a collectionPASS

S15-01: /admin/collections lists Featured and Sale collections with product counts.

S15-02: New collection link leads to create form.

S15-03: Edit form pre-fills collection title.

S16

Admin Customer Management

3/3

S16-01Shows the customer listPASS

S16-02Shows customer detail with order historyPASS

S16-03Shows customer addressesPASS

S16-01: Customers list shows 5 seeded customers with order counts and totals.

S16-02: Customer detail shows contact, addresses, recent orders, lifetime stats.

S16-03: Addresses section present in customer detail.

S17

Admin Pages Management

3/3

S17-01Shows the pages listPASS

S17-02Can create a new pagePASS

S17-03Can edit an existing pagePASS

S17-01: Pages list shows About Us, Contact, FAQ with Edit/Delete actions.

S17-02: New page link leads to create form.

S17-03: Edit form pre-fills page title.

S18

Admin Analytics Dashboard

2/3

S18-01Shows analytics dashboardPASS

S18-02Shows sales dataPASS

S18-03Shows conversion funnel dataFAIL

S18-01: /admin/analytics heading plus date range selector.

S18-02: Shows Revenue/Orders/AOV/Visits KPIs.

S18-03: No conversion funnel UI; only the four KPIs.

Features