Local-LLM Document OCR
Self-hosted document OCR that turns scanned PDFs into Markdown using local vision LLMs
What it does
A document processing service that takes a scanned or digital PDF and returns a Markdown transcription. The pipeline renders each PDF page to a PNG image using Apache PDFBox, sends the image to a vision LLM with a transcription prompt, strips any chain-of-thought reasoning tags from the response, and concatenates the per-page Markdown into a final document. The default runtime is Gemma 3 12B running locally on a GPU via LM Studio.
The provider gateway
Under the hood is a provider-agnostic chat gateway with adapters for seven providers: Anthropic, OpenAI, Google Gemini, Meta/Llama (via Groq or local), DeepSeek, xAI/Grok, and any OpenAI-compatible local endpoint. Each adapter translates between the gateway's internal message format and the provider's API. The OCR pipeline calls the gateway, not any specific provider, so switching models is a configuration change.
Why local LLMs for this
OCR of business documents - scanned contracts, invoices, handwritten forms - involves sensitive information. Running the transcription locally means no document content leaves the machine. Gemma 3 12B has strong vision capabilities for a 12B model, and on a modern GPU the throughput is good enough for the batch processing use case. For documents where accuracy matters more than speed, the gateway makes it easy to route to a commercial API instead.
What I learned
Vision LLMs vary a lot in how they handle scanned documents vs digital PDFs. Digital PDFs render cleanly and most models transcribe them well. Scanned documents with skew, low contrast, or handwriting are harder - the model choice matters more. Gemma 3 handles printed scans well. Handwriting is still a weak point for local models at this size.
Stack
Want something like this built?
This is the kind of system I build for clients. Tell me what you're trying to do, and I'll give you a straight answer on whether AI is the right tool.
Book a 30-minute call