How do you tell Claude Code your testing conventions?
Put your testing conventions in a CLAUDE.md file at the root of the repository, because Claude Code reads it at the start of every session. Anthropic's guidance is to include the commands Claude cannot guess, the style rules that differ from defaults, and your testing instructions, and to leave out anything Claude can work out by reading the code.
MARKDOWN
# Testing conventions
- Framework: Playwright Test, TypeScript. Tests live in tests/. Config: playwright.config.ts.
- Run one file: npx playwright test tests/<file>.spec.ts. Run all: npx playwright test.
- Locators: prefer getByRole, getByLabel, and getByText. Use CSS or XPath last, with a comment.
- Assertions: web-first only, such as await expect(locator).toBeVisible(). No waitForTimeout.
- Tests must be independent: no shared state, and each test creates its own data.
- Credentials come from E2E_EMAIL and E2E_PASSWORD. Never print or hard-code them.
- Never weaken an assertion to pass a test. First say whether the test or the app is wrong.
Run /init to generate a starter file from your repository, then trim it, and run /context to confirm Claude loaded it. Keep it short: the memory documentation targets under 200 lines per file because longer files reduce how well Claude follows them. It also says CLAUDE.md is context, not enforced configuration, so anything that must never happen belongs in a hook or a permission rule, covered below.
How do you set up Playwright MCP for Claude Code?
Set up Playwright MCP for Claude Code with one command, which registers a server that lets Claude open pages, read them, and act on them in a real browser.
BASH
claude mcp add --scope project playwright -- npx -y @playwright/mcp@latest
claude mcp list
The --scope project flag writes the server into a .mcp.json file at the repository root, so teammates get the same setup. Without it, claude mcp add stores the server at the default local scope instead of sharing it through the repository. This is the file we got:
JSON
{
"mcpServers": {
"playwright": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"@playwright/mcp@latest"
],
"env": {}
}
}
}
Claude Code asks you to approve a project-scoped server the first time you start a session. Until you do, Claude mcp get playwright to report it as pending approval. Once it is approved, ask Claude to open your login page and describe what it sees.
Playwright MCP works from the page's accessibility tree, not from pixels: each tool returns a structured snapshot of elements, roles, and text, with a reference Claude can act on. That is close to how getByRole finds elements, so what Claude sees maps well onto the locators you want in the test.
There is a cost. Playwright's own comparison says MCP suits specialized agentic loops and exploratory automation, and that the Playwright CLI with skills is the better fit for coding agents such as Claude Code in large codebases, because MCP keeps its tool schemas and page snapshots in context and uses more tokens.
Should you use a Playwright skill for Claude Code instead of MCP?
A Playwright skill for Claude Code is the lighter option: it teaches Claude the Playwright CLI's commands and loads on demand, while MCP keeps its tool definitions in every conversation. Which to choose depends on the job, and you can run more than one.
Option | What it is | Use it when | Trade-off |
|---|
Playwright MCP | A server that drives a browser through the accessibility tree | Claude should explore a live page, confirm locators, and iterate in one session | Higher token cost, per Playwright's comparison |
Playwright CLI skill | playwright-cli plus a skill that teaches Claude its commands | Claude works across a large codebase and you want lower token use | One more tool to install and keep current |
Playwright test agents | Planner, generator, and healer agents from npx playwright init-agents --loop=claude | You want separate plan, write, and repair roles | In our run on 1.56 it replaced an existing .mcp.json |
A skill of your own | A SKILL.md that encodes your team's test-writing procedure | The same request keeps coming back, such as "add a test for this flow" | You maintain it like code |
To install the official skill, add the CLI and then the skill:
BASH
npm install -g @playwright/cli@latest
playwright-cli install --skills
Per Playwright's installation guide, the second command copies a playwright-cli skill into .claude/skills/ in your workspace. Add -g to put it in ~/.claude/skills for every project instead.
A skill of your own is worth writing when you repeat the same request. Anthropic's best-practices guide puts rules that apply everywhere in CLAUDE.md and repeatable workflows in skills. Save this as .claude/skills/new-e2e-test/SKILL.md:
MARKDOWN
---
name: new-e2e-test
description: Write a Playwright test for a user flow, run it, and report the result
disable-model-invocation: true
---
Write a Playwright test for this flow: $ARGUMENTS
1. Read tests/login.spec.ts and follow its structure and locator style.
2. Open the page with the Playwright tools and confirm each locator before using it.
3. Write one file in tests/, one behavior per test, with web-first assertions.
4. Run npx playwright test <file>. If it fails, say if the test or the app is wrong.
5. Report the command you ran and its output. Do not edit existing tests.
Run it with /new-e2e-test checkout with a saved card. The disable-model-invocation: true line means Claude runs the skill only when you call it.
Playwright also ships test agents, introduced in version 1.56. One command scaffolds them for Claude Code:
BASH
npx playwright init-agents --loop=claude
In our run it wrote three agent files, playwright-test-planner.md, playwright-test-generator.md, and playwright-test-healer.md, into .claude/agents/, plus a tests/seed.spec.ts and a .mcp.json with a playwright-test server. It replaced the .mcp.json we already had, so run it first or merge the two servers by hand. The documentation describes the planner as producing a test plan, the generator as turning it into tests, and the healer as repairing failing ones.
How do you get Claude Code to write your first test?
Ask for one flow at a time, name the file, state what a pass looks like, and tell Claude to run the test and show the output before it reports back. Anthropic's guidance is that giving Claude a check it can run is the difference between a session you watch and one you walk away from.
TEXT
Using the Playwright tools, open http://localhost:3000/login and write
tests/login.spec.ts with two tests. First, signing in with the E2E_EMAIL and
E2E_PASSWORD test account lands on /dashboard under the heading "Welcome back".
Second, a wrong password shows the alert "Invalid email or password" and stays
on /login. Follow CLAUDE.md. Then run npx playwright test tests/login.spec.ts
and show me the output.
A reviewable result looks like the file below. It passed against our demo app, and it uses role and label locators, web-first assertions, and credentials from the environment:
TYPESCRIPT
import { test, expect } from '@playwright/test';
const email = process.env.E2E_EMAIL!;
const password = process.env.E2E_PASSWORD!;
test.describe('sign in', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/login');
});
test('signs in with valid credentials', async ({ page }) => {
await page.getByLabel('Email').fill(email);
await page.getByLabel('Password').fill(password);
await page.getByRole('button', { name: 'Sign in' }).click();
await expect(page).toHaveURL(/\/dashboard/);
await expect(page.getByRole('heading', { name: 'Welcome back' })).toBeVisible();
});
test('shows an error for a wrong password', async ({ page }) => {
await page.getByLabel('Email').fill(email);
await page.getByLabel('Password').fill('not-the-password');
await page.getByRole('button', { name: 'Sign in' }).click();
await expect(page.getByRole('alert')).toHaveText('Invalid email or password');
await expect(page).toHaveURL(/\/login/);
});
});
Do not merge on a green run alone. Review the file against Playwright's best practices:
Check | Why it matters | Red flag |
|---|
Locators | Role and label locators survive markup changes better than CSS paths | A selector such as .btn-primary > span:nth-child(2) |
Assertions | Web-first assertions retry until the condition holds | expect(await locator.isVisible()).toBe(true) |
Waits | Fixed sleeps cause flaky tests and slow suites | waitForTimeout(3000) |
Independence | Each test should pass alone and in any order | A test that needs data another test created |
Credentials and data | Secrets and real customer data stay out of the repository | A literal password in the file |
Intent | The test should check the requirement, not just today's behavior | An assertion that copies whatever the page shows now |
Then break the app on purpose. Change the expected text or the button label and confirm the test fails. A test that cannot fail is not a test, and it is an easy mistake to miss in generated code.
How do you run the suite and let Claude Code fix failures?
Run the suite with npx playwright test, point Claude at the failure, and ask it to decide whether the test or the application is wrong before it edits anything.
To see a real failure, we renamed the button in our demo app from "Sign in" to "Log in" and ran the suite. After the default 30-second timeout, Playwright reported:
TEXT
Error: locator.click: Test timeout of 30000ms exceeded.
Call log:
- waiting for getByRole('button', { name: 'Sign in' })
It also wrote an error-context.md file with a page snapshot, which includes the line button "Log in". Claude can read both. If the renaming was intentional, the fix is a one-line change to the locator. If it was not, the right output is a bug report, not an edited test. That judgment is why the prompt asks for a decision first:
TEXT
Run npx playwright test. For each failure, read the error and the page snapshot
in test-results/. Tell me whether the test or the app is wrong. Fix only the
tests that are wrong, keep every assertion at least as strict as before, and
show me the diff.
Three habits keep this loop honest. Read the diff for loosened assertions, because the easiest way to turn a test green is to check less. Confirm a fix is not luck by running npx playwright test --repeat-each=10, which repeats every test ten times. And consider a gate that does not depend on Claude's judgment: Anthropic describes a Stop hook that runs your check as a script and blocks the turn from ending until it passes.
How do you limit what Claude Code can read and run?
Limit Claude Code with permission rules in .claude/settings.json: allow the commands the workflow needs, and deny the files and commands it should never touch.
JSON
{
"permissions": {
"allow": [
"Bash(npx playwright test *)",
"Bash(npm run *)"
],
"deny": [
"Read(./.env)",
"Bash(git push *)"
]
}
}
The permissions documentation shows the same pattern: allow npm run commands, deny git push, and deny reads of .env files. Deny rules are checked first. They are still a guard rail, not a wall. The documentation notes that a push written another way, such as git -C . push, is not matched by a git push rule, so use a hook when an action must never happen.
How do you run Claude Code in CI with GitHub Actions?
Run your Playwright suite in CI exactly as you would without Claude, then add Claude Code as a separate workflow that responds to @claude comments or runs a headless command.
The initializer's GitHub Actions workflow already does the first part. Playwright's CI guide describes a workflow that runs on pushes and pull requests to main or master, installs dependencies and browsers, runs npx playwright test, and uploads the playwright-report folder as an artifact. Keep that job as your gate.
For the Claude side, Anthropic's GitHub Actions documentation gives this workflow, which reacts when someone mentions @claude in a comment. Add your key as the ANTHROPIC_API_KEY repository secret first:
YAML
name: Claude Code
on:
issue_comment:
types: [created]
pull_request_review_comment:
types: [created]
jobs:
claude:
if: contains(github.event.comment.body, '@claude')
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
issues: write
id-token: write
actions: read
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 1
- uses: anthropics/claude-code-action@v1
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
A comment such as "@claude the checkout test fails on this branch, find out why and open a fix" then starts a run. For scripts, the headless mode runs one prompt and exits. Start read-only, so Claude diagnoses and does not edit:
BASH
claude -p "Run npx playwright test and report any failures with a likely cause" \
--allowedTools "Bash(npx playwright test *),Read" \
--permission-mode dontAsk
The --allowedTools flag pre-approves only what the job needs, and --permission-mode dontAsk denies everything else, which matters when no one is there to answer a prompt. Whatever Claude proposes should still arrive as a pull request that a person reviews and that your normal suite checks.
What does Claude Code cover across the six testing steps, and what does it leave with your team?
Claude Code does its best work in test automation, and it helps with the other five testing steps without owning any of them. This table shows where the line falls.
Step | What Claude Code does | What stays with your team |
|---|
01 Test design | Reads a spec, ticket, or pull request and proposes scenarios. The Playwright planner agent writes a test plan for it | Deciding which risks matter, what the acceptance criteria are, and what is out of scope |
02 Test case creation | Turns a described flow into test steps and spec files, through the generator agent or your own skill | Reviewing intent, and keeping a catalog with IDs, owners, and links to requirements. Claude Code holds none of that |
03 Test automation | Writes, runs, and corrects Playwright code in your repository | Conventions, review, test data, environments, and the decision to merge |
04 Test execution | Runs npx playwright test locally, or inside a CI workflow | Schedules, parallel capacity, environments, and secrets. CI keeps the logs, and Claude Code has no run history of its own |
05 QA reporting and analysis | Summarizes a failure from the error and page snapshot. Playwright's HTML report and trace viewer cover a single run | Trends across runs, release decisions, and an evidence trail that people outside the team can read |
06 Test maintenance and optimization | Updates locators and waits when a UI change breaks a test, as the healer agent is built to do | Deciding whether a failure is a product bug, reviewing every diff, and retiring duplicate or low-value tests |
In short, Claude Code writes test code well, but it holds no test estate of its own: no catalog, run history, schedules, governed data, or evidence trail. Whatever of those you have comes from your repository and your CI provider, and your team keeps it current.
What goes wrong when an AI agent writes and fixes your tests?
The most common failure is a test that passes without proving much, and review is where it gets caught. In Sonar's survey of more than 1,100 developers, 96% said they do not fully trust that AI-generated code is functionally correct, yet only 48% always check AI-assisted code before committing it. Tests are code, so the same gap applies to them.
Weakened assertions. A "fix" that loosens a check turns the run green and hides the defect. Guard against it with the CLAUDE.md rule above and a diff review.
Tests that encode a bug. If Claude writes a test from what the page does today, it confirms today's behavior, including mistakes. A person must own the requirement the test checks.
Invented locators and steps. Have Claude confirm each locator in a real browser through MCP or the CLI skill, then run the test.
Flaky passes. One green run proves little. Repeat the test before you trust it.
Exposed secrets and data. Use environment variables, deny rules, and a staging environment with test accounts.
If the product under test is itself built on a language model, assertions need a different approach, which is the subject of how to test generative AI applications.
When is Claude Code enough, and when do you need a test platform?
Claude Code is enough when one team owns a modest set of web tests in one repository and a developer reviews every change. It stops being enough when tests need an owner, a schedule, a history, and an evidence trail that people outside the team can read.
Claude Code is a good fit when:
Developers write and review the tests themselves.
The suite lives in one repository and targets one or two web applications.
A pull request and a CI log are all the evidence anyone asks for.
A platform starts to matter when:
QA, product, and compliance need to see what is covered and what ran, without opening a repository.
Several teams need shared steps, governed test data, and scheduled runs.
Packaged enterprise applications are in scope. If that describes your estate, you are more likely comparing platforms such as Tricentis Tosca than coding assistants.
We build Test Studio, so weigh this section with that in mind. Rebuilding a test estate around a coding assistant is a platform project, not a prompt: you add a catalog, scheduling, governed test data, run history, and reporting, then maintain all of it. An AI test automation platform starts from that estate. In Test Studio the test lands with an ID, reusable steps, variables, a branch, a schedule, and a run history.
See Test Studio run a test against your own build. Book a demo to see it on your application, or download the solution brief for the detail.
If you would rather work with people who have built this before, our forward deployed engineers work inside your team, each carrying a Test Studio license, and are measured on a baseline agreed in writing in week one. Talk to the team.
If the question is wider than testing, an AI SDLC transformation is a bounded engagement: it maps where the hours go across your lifecycle and hands the capability back to your team. It starts with an SDLC assessment.
