Product: developer platform. Stage: live. Engagement: free QA pass, from the outside. Status: sent to the team; confirmed.
How it came up
I signed up for the platform, connected an AI coding tool to it the way the docs describe, and deployed a small test app. Then I used the dashboard the way a customer would. I didn't have the server code; I looked at the dashboard and the code it sends to the browser.
The core of it worked well. The connection went through on the first try, and the test app was deployed and running in about two minutes. That's the hard part of a product like this, and it was done right.
"Saved," and the old value was gone
The dashboard has a place to store secrets: API keys and passwords your app needs. I added a secret, then added another one with the same name and a different value.
The dashboard said "Saved." It didn't say the name was taken, and it didn't ask whether to replace it. The first value was simply gone, overwritten by the second.
For a store of credentials, that's the most important kind of bug: a success message on top of lost data. A customer who reuses a name by accident replaces a working production key with the wrong one, and the first sign of trouble is their app failing.
A success message on top of lost data is worse than an error.
Two smaller problems in the same area
- Secrets couldn't be deleted from the dashboard. Once added, the only way to change one there was to overwrite it, which is the behavior above.
- Customer-facing logs showed internal infrastructure details, things a customer has no use for and an outsider shouldn't see.
What the team said
I sent those three findings to the team in a message. They replied that I was right about all three. The silent overwrite was already fixed and going out with their next update, and the other two were being built. They asked for the full write-up, which had a few more items in it.
Time spent: a short pass, no code review. Status: confirmed by the team; the overwrite fixed.
What this says about AI-built apps
"Add" and "update" are different operations, and AI-generated code often writes one function that does both. It works in every demo, because a demo never reuses a name. The question to ask of any screen that saves something is: what happens if it already exists?