There's a moment in every agent I've built where the model's answer stops being something a person reads and becomes something a program acts on. It gets stored, it decides a branch, it's handed to the next step. The instant that's true, prose is a liability. If the next thing that touches the answer is code, then a paragraph of English is a parsing problem you signed yourself up for, and it will break the first time the model phrases the same fact a different way.
The fix isn't a better prompt asking nicely for clean output. It's constraining the output to a defined shape - a JSON schema, a required set of typed fields, a tool-call signature - so the model returns data you can trust structurally, and validation happens at the boundary instead of buried in string parsing three steps downstream.
Parsing Prose Is a Contract You Didn't Write Down
Here's the failure mode, and I've shipped it more than once. You ask a model to classify something or produce a result, it answers in prose, and the answer is right. So you write code to pull the part you need out of the text. Maybe the status is on the last line. Maybe the number comes after the word "total." It works on the ten cases you tested.
Then it fails. Not because the model got the answer wrong, but because it got the answer right and said it differently. "The status is approved" becomes "This should be approved" becomes "Approved (pending your review)." Your regex was keyed on a phrasing, and the phrasing was never a contract. It was a habit the model had on Tuesday.
"It usually says the status on the last line" is not a specification. It's an observation about a sample, and the model never agreed to it. You inferred a format from a few outputs and then treated your inference as a guarantee. The model owes you nothing about where the status appears, because you never told it the status had to appear anywhere in particular. The bug isn't in the parsing. The bug is that there was parsing at all.
What Structured Output Actually Means
Structured output means the model doesn't get to choose the shape. You define the shape, and the generation is constrained to fit it. In practice this shows up three ways, and they're variations on the same idea.
A response schema. You hand the model a JSON schema and the runtime constrains decoding so the output has to be a valid instance of it. Required fields are present. A field typed as a number can't come back as the word "seven." An enum field can only hold one of the values you listed. You don't hope for JSON and then try to parse it; the output is JSON that conforms, or the call errors.
Tool and function-call arguments. When you give a model tools instead of just a chat box, the arguments it passes to a tool are already structured. A tool defined with a typed parameter list is a schema by another name. The model isn't writing you a sentence about what it wants to do; it's filling in status, due_date, and amount as typed values because the tool signature demands them. This is often the cleanest way to get structure, because you were going to define the tool anyway.
Required fields with types. Even without formal schema-constrained decoding, you can specify an exact field set with types and defaults, and validate against it. It's the same discipline: the output is a record with known keys, not a story you mine for keys.
The common thread is that the shape is declared up front and the model's job is to populate it, not to design it.
Validate at the Boundary, Not Three Steps Later
The reason to constrain the shape at the call layer is that it's the one place you can catch a mismatch cheaply. If the model returns something that doesn't fit the schema - a missing required field, a string where a number belongs, an enum value you never defined - you know immediately, at the point of generation, before the bad data has traveled anywhere.
That matters because of how the alternative fails. If you accept loose output and let it flow downstream, a malformed result doesn't announce itself. It gets stored. It gets passed to the next step. The classification that came back as "maybe approved?" instead of one of your three allowed values sails through until, two or three steps later, something tries to branch on it and either throws or, worse, silently does the wrong thing. Now you're debugging a failure far from its cause, reconstructing which step let the bad shape through.
Validation at the boundary collapses that distance to zero. And it gives you a clean recovery: when the output fails the schema, you hand the error back to the model and have it retry. "Your last response was missing the required due_date field and priority was not one of high, medium, low. Return a valid object." The model corrects it, usually on the first retry, because the constraint is now explicit and specific instead of implied. A retry against a schema is a real error-handling path. A regex that quietly matched the wrong substring is not; it's a bug with a delay on it.
This is the same instinct behind having AI review AI: make the machine check the machine at a defined checkpoint, rather than discovering the miss by hand once it's already downstream.
Where Structure Belongs and Where Prose Still Wins
Not every output should be a JSON object, and forcing structure where it doesn't belong is its own mistake. The line is simple: who or what consumes this next?
If the consumer is another process, structure it. Agent results that a later step reads. Extraction where you're pulling fields out to store. Classification where the answer selects a branch. Anything that feeds a database, a conditional, a queue, or another agent. In every one of those, the value of the output is that code can act on it without a human in between, and that's exactly what a defined shape buys you.
If the consumer is a person, prose is right, and structure would just be noise. The summary someone reads over coffee. The explanation of why a change is risky. The draft of an email. Nobody wants a human-facing answer delivered as a field map, and constraining it would strip out the nuance that made it worth reading. A model asked to explain a tradeoff should write sentences.
The mistake in both directions is confusing the two. Don't parse prose you should have gotten as data. Don't schema-constrain something a human was always going to read. The question that sorts it is whether the next reader has hands or a for loop.
Design the Shape First, Like an API Response
The discipline that makes this work is designing the output shape before you write the prompt, the same way you'd design an API response before writing the endpoint. Decide the fields. Decide the types. Decide what's required and what's optional, what the enums are, what a missing value means. The prompt comes after, and its job is to explain how to populate a shape that already exists, not to conjure the shape on the fly.
This is worth doing deliberately because the shape is a contract, and contracts are cheap to change on a whiteboard and expensive to change once three agents depend on them. I run a fleet of agents in production, and their results are consumed by later steps and written into notes I audit later. If those results came back as free-form prose, the whole thing would be unauditable: I couldn't reliably tell what an agent decided, couldn't diff two runs, couldn't gate a downstream step on a field, couldn't trust that "done" meant the same thing today as last week. The structure is what makes the output both usable by the next step and legible to me after the fact. An agent that returns {"action": "created", "record_id": "...", "confidence": "high"} has told me something I can act on and check. An agent that returns "I went ahead and took care of that for you" has told me nothing I can build on.
Worth being precise about the distinction from extracting structure out of messy documents, which is a different job. In a data parsing pipeline, the input is unstructured - a PDF, a scan, an inconsistent form - and you're using the model to impose structure on someone else's mess. This is the other end. Here the model is producing the output, and the point is to constrain its own generation so it's structured at the source. One is about reading chaos. This one is about not creating it.
The shift in mindset is small and it pays off every time. Stop treating the model as something you interview and then transcribe. Treat it as something that returns a record. Define the record, validate the record, and the fragile English-parsing layer that used to sit between the model and your code just disappears.