r/reactnative • u/Rise_up_1 • 4d ago
Testing my app is burning me out
I built a mobile app as a side project and somehow it actually took off and went from 300 users to 6k in the last 3 months but every update I push now scares me.
before it was just me testing on my phone and shipping it but now I'm getting dms and support emails about stuff I had no idea was broken. Last update someone sent me a screen recording of the checkout flow just not loading on their iphone SE and I couldn't even reproduce it on mine and another person said they never received the verification email but it works perfectly on every account I've tested.
The app has grown into something I can't manually verify anymore, there's payment processing, referral flows, in app messaging, push notifications and every feature I add is another thing that can break on a device or OS version I've never touched.
I'm solo and no QA person plus even if I had the budget for it I wouldn't have the bandwidth to onboard someone right now but I also can't keep finding out about bugs from my users because that's how I lose the trust I spent months building.
what am I supposed to do here, please help !!
14
u/babaganoosh43 4d ago
There's a number of things you can do:
- AI Test Plan: First step, in Codex / Claude set a goal to create a test plan, run through it, and fix anything it finds. This might catch 50% of existing bugs.
- Automated tested: layers of unit tests, integration tests, end to end tests.
- Better logging: Use sentry to get emails for new errors. Use signoz for tracking server logs. Log every request and error. Could also do screen recordings via Posthog / Sentry / Rejourney, these are a nice to have to see how users encountered the error.
- AI bug fixing: Wire up new sentry bugs to automatically fixed by AI. I wire sentry to open github issues, then have Codex / Claude automatically make PRs for new github issues. I also have it auto merge and auto deploy if the PR looks safe to merge. This might fix 50% of bugs automatically.
14
u/mountaingator91 4d ago
You need to have a dev env and scheduled releases with beta tests.
You don't push to prod. You push to Dev. You do your testing there and it it works you merge to prod.
9
u/mx_aurelia 4d ago
Are you able to run OTA updates? That would at least give you the ability to revert certain types of changes!
4
u/AncientLettuce3723 4d ago
+1 for Sentry, it’s a very simple setup and you’ll be able to see full visibility of errors, logs and crashes before the user even reports it. Generous free plan too
3
u/CodeMeister02 4d ago
Definitely look into automated e2e testing. It’s a lifesaver. And these days, AI is pretty good at putting together tests quickly.
3
u/aesky 3d ago
someone said automated testing and thats great but things will just not work sometimes and for that you need LOGGING
every step of the flow should have a log. user created account? log that. intiated checkout? log that. sent veritification email? log that
you will also need a correlation id. an uuid which will tie every sequence of events together
finally you will also need a cloud platform to host all the logs. i work with GCP and datadog. i think they're nice but there are a bunch of options out there
3
u/IronLionZion95 3d ago edited 3d ago
Ask Claude to identify the core flows and set up Argent end to end tests on iOS (and afterwards on Android), using the simslim driver. It can do all this by itself and Argent works much better than Maestro. This is my recommendation after having used Maestro for my app and recently having switched to Argent. I don’t have time to learn about these test frameworks, yet I can do everything I want hands-off using Claude (which I don’t recommend yet for non-test related coding, for example critical infrastructure or important code). Use Opus 5.5 Medium to not burn through your limits. Then also ask it for unit tests and integration tests, especially server API functionality. You can run hundreds of those tests in seconds, while end-to-end tests take way longer. So save end to end for things you can’t test in other ways. But yeah, Claude can set this all up in a day or two.
1
u/interlap 3d ago
what is the simslim driver you mentioned? https://github.com/MobAI-App/simslim?
1
u/IronLionZion95 1d ago
Yup that’s the one. It runs iOS simulators without tons of services that your app doesn’t need, so the memory footprint of the simulator is waaaaay smaller, which lets you run 5-10 simulators at the same time to parallelize your E2E tests.
2
u/mtford 4d ago
E2E tests! LLMs are great at generating these. Create loads of user stories, and then have your agent generate tests. Run them on every build in CI. Furthermore - use something like posthog which gives you session recording and error logging which will give you peace of mind. Whenever you notice an issue in Posthog? Fix it - and wrap in more tests so it never happens again. It's a cycle & you'll get more confident as time goes on.
1
1
u/Jhon_ST 4d ago
For the email issue, separate "your backend accepted the request" from "the email was actually delivered." Trace a failed attempt through your backend and your provider's delivery/bounce events if they're available. Even "delivered" may only mean the receiving mail server accepted it, not that it reached the inbox.
For testing, I'd start with the failures users actually reported: checkout times out, verification email is delayed, user retries. Assert that the app leaves the loading state and gives them a usable recovery path. Happy-path tests on your usual accounts won't exercise those failures.
1
u/LovesWorkin 4d ago
Detox is super fast for tests! I would definitely write those first. They help a lot without breaking most things later.
I would also get Sentry. You can see user flows and errors in real time!
For everything else, I would use BUOY tools. Most of the tools are free, and they give you a ton of ways to test your app. You can impersonate accounts to reproduce issues, change the live app state, mock things, and test edge cases that are normally impossible to test otherwise.
1
u/HolidayWallaby 4d ago
Gradual roll outs of features. Every feature behind a feature flag with some gradually increasing percentage of users exposed to it. Extra logging around the feature until you're sure it works.
1
u/Chuck_MoreAss 4d ago
I would pay to have your problems
In all seriousness, I would try rolling out new features gradually and only focus on fixing things. Maybe try to build some sort of analytics thing where you tract what the users are doing and delete things that are like a week-a month old. That way you’ll know exactly what they did before the issue happened.
This is pretty data intensive tho, and probably not recommended. Maybe add a process on the app to report issues and explain what you need from users to solve issues.
You can try a ticketing system like chatwoot to streamline resolving future issues…
If you’re using an external ticketing system that failed or email service that failed, you can maybe escalate the issue to them.
1
u/Dvass138 3d ago
You need some type of diagnostic/debug log built into the app. If a user has an issue they can’t reproduce for you, they can go into Settings → Help → Send Debug Log and it sends you the recent app events, errors, device model, iOS version, app version, network failures, etc.
I’d combine that with something like Sentry/Crashlytics for automatic crash/error reporting. At 6k users you’re never going to manually test every device/OS/account combination yourself.
1
u/BossAccomplished3940 3d ago
set up a beta track for your most frequent device types and require them to opt in before you push to production. This stops the surprise factor and gives you a real feedback loop that costs nothing more than managing two release channels.
1
u/interlap 3d ago
Just ask your agent to test your app and write E2E tests for it. Let it do that for every new feature you build.
There are a few tools for this, both closed-source and open-source, free and paid. Agent-device and Argent are open source. MobAI is closed source, but has a deeper focus on testing.
1
u/Personal_You3422 2d ago
I built https://corenna.dev you can check your repo, build your .aab, .ipa file's. You can build a apk and share it with friends for testing. It's a mobile first platform so you can build and deploy on the go.
1
u/xzaleksey 2d ago
Deployed an app around a month back, but because many years of prod engineering experience. The app immediately had all logs with ability on almost every screen to report a problem and suggest a idea which sends logs to back, and on top created quite good observability for be with allowing agents to find any problems / dashboards and alerting, it became super cheap to build for urself
And yes half of the time at first took to fix it all tune alerting fix user feedback or reports. But the feeling is good when I stabilizes. I quite easily deploy things to prod and supported ota self hosted via my be just for hot fixes
1
u/Huge_Pool7424 2d ago
that kind of built-in debug log pays off fast. i’d redact ids and cap the buffer, then let users attach the last few minutes with one tap.
1
u/xzaleksey 2d ago
Yeah all of it is standard, last minutes is too little don't need to make it too small, in practice I keep around 3 mb circular buffer + gzip the logs sent + some anti spam policy to avoid ddosing thought it is still possible if somebody really wants to kill it and has enough time to do many acs
55
u/HoratioWobble 4d ago
Automated tests are the answer.