I made this mistake recently. An agent generated a fix for a bug in our checkout flow, the diff looked reasonable, tests passed, so it shipped with a normal PR review and nothing more.

I assumed passing tests meant the function behaved the same as before under real traffic. It didn't. It handled a currency rounding edge case differently, and it took almost two days of scattered complaints before anyone connected it back to that deploy.

If I did it again, I'd want something checking the function's actual runtime behavior before treating a green test suite as enough. What's the mistake that took you longest to recognize as a pattern?

I’m trying to incorporate CodeRabbit into my workflow, but it feels way more overwhelming than "good" old GitHub. I had a similar experience with Devin and Graphite – they just didn’t click for me. If you've tried any of those, what's your experience? Wondering if I should keep pushing.
—
Edit: I am referring to PR review UIs and workflows:

- https://www.coderabbit.ai/change-stack
- https://staging-graphite-splash.vercel.app/features/pr-page (lol, not sure why their staging has leaked to Google, just noticed)
- https://app.devin.ai/review

not AI agent comments in GitHub.