AI App Review · Case Study

A CRM Whose Records Would Have Quietly Stopped at 5,000

An archive room of neat shelves of file boxes, with newer boxes stacked on the floor past the last shelf

Product: CRM, before its customer beta. Client: anonymous. Engagement: paid code review. Status: written up for the founder as fix before beta.

How it came up

A founder building a CRM, largely with an AI coding tool, hired me to read the code before he opened the beta to customers. I read it as it stood at one specific commit. I didn't run anything against his live site or his customers' data.

A lot of it was good. Every automated check passed, including all of his tests, and none of the outside libraries had a known security problem. The way the app kept one customer's data away from another's was sound. That's worth saying first, because the problem below doesn't show up in any of those checks.

A limit nobody asked for

When the app needed a long list of records, it read them in batches and stopped at about 5,000 records. When it stopped, it wrote one line to the server's log and carried on with whatever it had.

Nobody had asked for that limit. The AI tool that wrote this part of the code added it as a safety measure, and never mentioned it. The founder didn't know it existed. A comment in the code even said the cutoff should be loud. Nothing reached the user.

Nothing errors. Records are just missing.

Where it reached

The same reading routine sat underneath a lot of the product:

  • Every spreadsheet export. A customer with 6,000 contacts would export them and get 5,000, with no warning.
  • Reports. A large report showed a note saying its totals included every record. Past 5,000 records, it would add up the first 5,000 and show a confident, wrong total next to that note.
  • Roll-up totals, like the hours added up on a parent record from its related records.
  • The duplicate check on imports, which would miss anything past the first 5,000.
  • Search, and some filtered list views.

Why the tests didn't catch it

None of the tests checked what happens past 5,000 records, and nothing fails when the limit is hit. The code returns a list, the list looks normal, and every test that checks the list passes. You find this kind of problem by reading what the code does with a large amount of data, by writing a test with more records than the limit, or by having a customer find it for you.

It wasn't far off, either. Five people each logging six entries a day reach 5,000 entries in about 170 working days. A busy customer would hit it in their first year.

The fix was already in the codebase

The app's own backup code did this the right way. It read much further, and it noted when it had stopped early. The fix was to do the same everywhere else: read past the limit where it matters, flag the result when it's cut off, and say so on screen ("Showing the first 5,000 records"). The report note needed correcting at the same time.

Time spent: part of a three-hour code review. Status: written up for the founder before the beta.

What this says about AI-built apps

AI coding tools add guardrails on their own, and that's often a good instinct. The problem is that they don't tell you. A limit you know about is a design decision; a limit you don't know about is a bug waiting for your biggest customer. When you review code an AI wrote, one of the most useful questions you can ask is: what limits did it add, and what happens when one is reached?

Has anyone outside your team read the code?

An AI App Review is two full days in your code, by hand, then a ranked report and a 90-minute walkthrough with your team. It's $1,500, one time.

Request an AI App Review

See what the review covers