Product: pay-as-you-go API platform for developers. Stage: live. Engagement: free QA pass, from the outside. Status: sent to the founder; first findings patched.
How it came up
The founder asked on Reddit for people to try the site. I created a free account and used it the way a new customer would: checked the balance, looked at usage, tried to add funds, ran a few requests. I didn't read any code.
A low-balance warning on a brand-new account
The free account came with a starting allowance of requests. After a single request, with almost all of it unused, the dashboard warned that the balance was running low.
Elsewhere on the same billing page, the account status switched back and forth between two contradictory messages while the page was open.
The same request, three different answers
The same request showed a different speed, a different cost, and a different date depending on which page I looked at it on. The cost estimates noticeably overstated the real charge.
For a product that bills by usage, the cost of a request is the number customers care about most. If two pages disagree, they'll trust neither.
If two pages disagree about money, customers trust neither.
Adding funds
- The site rejected the advertised minimum deposit. The real minimum was more than twice as much.
- A fractional deposit showed three amounts that didn't agree with each other.
- A promo code failed with an internal error message for a signed-in user.
A batch form that mislabeled input
A batch form had a limit on how many URLs it would take. Past the limit, it marked the extra URL as a duplicate, which it wasn't, instead of saying the limit had been reached. I didn't submit the batch, so I can't say what the server would have done with it, but a customer would have no idea why their URL was rejected.
What the founder said
I sent the top three findings by message. He thanked me, and a few hours later said everything in that message had already been patched and was waiting to deploy. I sent the full report, with about thirty more items, the next day.
Time spent: a short pass, no code review. Status: sent to the founder; first findings patched.
What this says about AI-built apps
When an AI tool builds a dashboard, it tends to calculate the same number in several places: once for the balance card, once for the usage page, once for the warning. Each version works, and each one is a little different. The fix is boring and reliable: calculate money in one place, on the server, and have every screen show that one answer.