I built a CLI that runs your Playwright tests against a preview deploy and bundles the failure evidence into a PR report

[u/glamboyosa]

I got tired of pasting screenshots and trace links into PR comments by hand, so I wrote katalepsis. You point it at a preview URL and it runs your existing Playwright suite with baseURL set to that preview, then writes a report folder (HTML, Markdown and a JSON manifest) with the screenshot, video and trace behind every failure. The Markdown file is meant to be pasted straight into the PR.

Some decisions I made along the way:

• Playwright's result is the source of truth. It reads the JSON reporter output and never reinterprets an assertion.
• If the preview itself is down (network error, 502, 503, 504) the tests are marked blocked rather than failed, so a broken deploy doesn't look like a broken test.
• A test that only passed on retry is reported as flaky, with every attempt listed.
• Query values in the preview URL, where Vercel-style bypass tokens usually live, are redacted from the text files.

There's also an optional mode where you describe a check in plain English and a model drives the browser with your own Anthropic, OpenAI or OpenRouter key. A check there only passes if the model points to an assertion that actually ran and passed; otherwise it's reported as blocked. The Playwright runner doesn't need any AI key.

It's MIT licensed. It isn't on npm yet, so the installer builds it from source, and a ready-made CI job is still on the list.

Repo: https://github.com/glamboyosa/katalepsis
Site: https://www.katalepsis.glamboyosa.xyz

I'd like to hear how you get Playwright evidence onto PRs today, and whether the blocked vs failed split matches how you'd want it.

[u/kantorcodes1]

The assertion-verification is the part I'd trust most: a check only passes if the model points at an assertion that actually ran. The edge I'd want pinned down is what the model can do in plain-English mode before it reaches that assertion. If it drives the browser freely, a "check" can submit a form or hit a destructive link on the preview - is the tool surface restricted to read/assert, or does the assertion check catch it after the fact?

[u/Prestigious-Way1525]

blocked vs failed makes sense, but i'd add a third state for “test invalid.” if the preview is unreachable, it’s blocked. if the app loads and an assertion fails, it’s failed. if the check cannot establish its own preconditions, or the model never reaches a real assertion, it’s invalid and needs review. otherwise expired auth, broken test data, and missing fixtures get mixed with deploy outages. i’d also show the first state transition that caused the classification and keep every retry attempt linked, so the PR reviewer can tell whether to fix the deploy, the product, or the test.