Zapper
We tested Zappy
on ourselves. Three times.
Same rough prompt, sent twice each: once as typed, once through the real Zappy extension. Same account, same model, fresh conversations every time. Where Zappy comes out looking worse, or the result is a wash, that's reported exactly as observed — that's the whole point of doing this.
Benchmark 1
Habit Tracker App
"build me a habit tracker app" — a build test. Zappy's run shipped a tested, deployable project instead of a single sandboxed demo.
Zappy wins clearly
Head to head
What each run actually produced, verified by hand, not read off a chat transcript and trusted.
- ✓ Zero clarifying questions: instantly live and clickable, no install step.
- ✓ Correct streak math, persisted through a real page reload.
- ↔ The trade-off: one file, confined to claude.ai's sandbox: no tests, no build step, nothing deployable as-is.
- ✓ The fuller system: a real, separated Vite/React project (data layer, 3 views, component library), unit-tested and production-built before it shipped.
- ✓ Correctly flagged all 5 real ambiguities: scope, platform, user scale, persistence, feature depth.
- ↔ The trade-off: turn 1 produced a plan first, and it shipped as a ZIP needing a local install: the cost of it being real, deployable software.
The full comparison
| Metric | Without Zappy | With Zappy |
|---|---|---|
| Prompt sent to Claude | Rough, vague, 6 words | Ambiguity resolved, reframed as a plan request |
| Follow-up prompts needed | 0 | 1: "go ahead and build it" |
| Unrequested assumptions | 1: invented a "systems-minded" persona | 0: stated assumptions matched real intent |
| Time to a first glance | ~45–70s, instantly live | ~40–50s for a plan, then a local install |
| System completeness | Single sandboxed file. No tests, no build step: a demo. | Standalone project: separated data layer, 3 views, README. Unit-tested, passing build. Deployable today. |
| Feature completeness | Core ask met; extras offered, not built | Core ask met, plus an unrequested analytics view + JSON export |
What actually mattered
Zappy's run produced the fuller, more real system
Run 1 is one file that only exists inside claude.ai's sandbox. Run 2 is a standalone project with streak-logic edge cases unit-tested before shipping and a passing production build, plus features Run 1 never built unprompted.
That completeness is exactly why it cost an extra turn
The optimized prompt's structure read to Claude as "this is a real project," which is precisely why it built one instead of a quick sandboxed demo.
On "which got me something to glance at fastest," Run 1 wins, no contest. On "which produced the more complete, real piece of software," Zappy's run wins clearly — a standalone, tested, production-built project instead of a sandboxed file, for the cost of one extra message.
Benchmark 2
Small-Business CRM
"build me a crm for my small business" — a harder, more ambiguous build. This time the result is murkier, reported exactly as observed.
Mixed result
Head to head
Real claude.ai, real extension, one clarifying question answered from a fixed ground-truth persona so both runs are judged fairly.
- ✓ Status pipeline with auto-computed repeat-customer badges.
- ✕ Real bug found: clicking "→ Customer" after marking a lead Won froze the entire tab, losing entered data.
- ↔ "Auto-persists" claimed in the response text — false when tested, no actual persistence code.
- ✓ Richer feature set: CSV import, tags/notes, a real dashboard chart, explicit CAD currency (context-grounded, not a guess).
- ✓ Claude caught and fixed its own bug mid-generation, visible in the live transcript, before finishing.
- ↔ Also stayed a single sandboxed artifact this time — Claude was explicit a real build needs a different tool.
The full comparison
| Metric | Without Zappy | With Zappy |
|---|---|---|
| Clarifying questions from Claude | 1 — "Which business is this CRM for?" | 0 — pre-answered by the optimized prompt |
| Currency handling | Generic "$", never committed to a unit | Explicit CAD — context-grounded, not a guess |
| Self-caught bugs during generation | None observed | Yes — fixed its own redundant-state bug live |
| Real bug found during testing | Yes — a conversion button froze the tab | None found in equivalent testing |
| Feature depth vs. rough ask | Leads → Jobs → Customers, status pipeline | Contacts → 5-stage Kanban → Activities → Dashboard, CSV import, tags/notes |
| Output type | Single-file HTML artifact | Single-file JSX artifact — also not a separate project this time |
What actually mattered
"Which is more complete" didn't repeat cleanly this time
Unlike the habit tracker, both runs produced single-artifact demos here — Zappy's run is richer (more entities, a kanban, CSV import) but Claude was explicit that a real deployable build needs a different tool entirely, on both sides.
The real bug this session landed on the unassisted side
Worth resisting the temptation to only report bugs when they favor Zappy — this one didn't, and it's reported anyway.
Zappy's ambiguity-resolution still worked — real gaps named, accurate assumptions stated — and it produced the richer feature set. But it did not produce a more deployable artifact this time; both stayed sandboxed demos. The actionable takeaway: Zappy's value here is the ambiguity-resolution and richer result, not "always ships something more real" — that doesn't automatically generalize to every app idea.
Benchmark 3
Retail AI Adoption Research
"I want to make a research on retail adoption of AI" — a research task, not a build. Zappy's run pulled from stronger sources and caught a misleading stat.
Zappy wins clearly
Head to head
Real claude.ai, real unpacked extension, read directly off the live transcripts — not paraphrased.
- ✕ Every cited source is a low-authority SEO/content-mill site — zero primary research firms.
- ✕ Presents 84–97%, 89%, and 97% adoption in the same response with no caveat they can't all be true.
- ↔ A real clarifying question did appear — but only appended at the very end, after the answer was already written.
| Segment | Adoption |
|---|---|
| E-commerce | 71–77% |
| Grocery | Mod-high |
| Fashion | Uneven |
- ✓ The concrete win: explicitly names and corrects the same "89% adoption" stat Run 1 repeated at face value.
- ✓ Named, primary-adjacent sources (NVIDIA's own survey, Deloitte, TCS, Adobe, McKinsey) alongside trade press.
- ✓ Structured exactly as asked: exec summary, use cases w/ maturity+ROI, segment table, barriers, outlook.
The full comparison
| Metric | Without Zappy | With Zappy |
|---|---|---|
| Web research depth | 1 search, 8 sources | 3 targeted segment-specific searches, 25 sources |
| Source authority | Exclusively SEO/content-mill sites | Named primary-adjacent sources (NVIDIA, Deloitte, TCS, Adobe, McKinsey) |
| Statistical self-awareness | None — conflicting figures, no reconciliation | Explicit upfront: "figures can vary 2–3x" — gives ranges |
| Critique of the "89% adoption" stat | Cited uncritically as fact | Explicitly flagged as "shallow" — names the study behind it |
| Output structure | Informal headers, no exec summary or segment table | Full structured report, separate downloadable artifact |
What actually mattered
The optimization step changed what research got done, not just the prompt
Run 2's three targeted searches — driven directly by the optimized prompt's segment breakdown — pulled from a meaningfully different and stronger source pool. A downstream research-quality effect, not just a prompt-aesthetics one.
A concrete, checkable accuracy correction
Both runs encountered the same "89% adoption" statistic. Run 1 repeated it at face value; Run 2 named it as shallow and cited the specific study showing most of that "adoption" is chatbot-level, not deep integration.
If the question is "which produced research someone could actually trust and use" — Zappy's run wins clearly, on evidence that's independently checkable. It ran deeper, more targeted research, pulled from stronger sources, and caught itself repeating a shallow statistic the unassisted run stated as plain fact.