TestPilot AI

Overview

TestPilot AI explores a practical question for quality engineering: how can a product story become a useful automation workflow without requiring someone to begin from a blank Playwright file?

The POC accepts a plain-English scenario, uses AI to create a structured test plan, generates Playwright TypeScript, validates that output against execution rules, and saves it for review before any browser run begins.

Current workflow

1. describe a user scenario in plain English 2. create a structured QA plan with test cases and risks 3. generate a reviewable Playwright draft for every planned case 4. validate that the draft includes the planned case identifiers, browser actions, assertions, and authentication setup when required 5. save the generated plan and code to Neon Postgres for review 6. let a user queue the approved spec in an isolated GitHub Actions runner, then save the final execution status

The interface is intentionally guided so a first-time automation user can understand what is happening at each step and what they should review next.

Engineering decisions

User-approved execution

TestPilot distinguishes between a generated, validated draft and an executed test. A user explicitly chooses when to run a saved spec; it is never presented as passed before GitHub Actions returns a result.

Guarded code, isolated runner

Generated code is checked before execution and is limited to the QA Demo Bank target. The runner downloads only an approved saved spec, executes it in GitHub Actions, and reports the result back to the TestPilot history.

Public-demo safeguards

The demo is limited to QA Demo Bank scenarios, validates the expected target, caps story length, and limits each IP address to ten plan-generation requests per day. These boundaries protect shared AI capacity and keep the POC focused on a controlled test target.

Reviewable AI output

AI output is treated as a draft to inspect, not as a source of truth. The workflow checks planned-case coverage, browser actions, assertions, authentication setup, and execution policy before it can be queued.

What this project demonstrates

• product thinking applied to a QA workflow
• practical Gemini integration with typed provider errors
• Playwright code generation with execution guardrails
• secure website-triggered GitHub Actions execution
• persistent generated and execution history with Neon Postgres
• clear user experience for people who do not already know test automation

Next phase

The next planned capability is richer failure triage and execution evidence. Test generation, approval, isolated execution, and saved results are already part of the current demo.