Local AI Models: When Data Never Leaves the Office

Andrew Maryasov, AI consultant, founder of Auspex and Grow2.ai. A local AI model is a model that runs on your own hardware, with no network call to an outside provider. For three tasks — speech recognition, document OCR and image classification — it already delivers production-grade quality: Ukrainian in Parakeet-TDT-0.6B-v3 scores 6.79% WER on Fleurs, against an average of 11.97% across languages. The trade-off hasn’t gone away, though. It has simply moved from the model’s quality to the job of running it: the same 258-million-parameter model processes a document in six seconds or in twenty minutes, depending on which engine you run it with.

TL;DR (in 30 seconds)

• Quality is no longer the bottleneck in three narrow tasks: transcription, pulling data out of documents, image classification. This isn’t “almost as good as the cloud” — these are numbers you can work with.
• Ukrainian is supported out of the box. parakeet-tdt-0.6b-v3 (NVIDIA, 600M, CC BY 4.0 license) recognizes 25 European languages with automatic language detection. WER for Ukrainian: 6.79% on Fleurs, 5.10% on CoVoST.
• “Average WER” tells you nothing about your language. In the same model, Slovenian scores 24.03% on Fleurs. The gap with Ukrainian is 3.5x.
• The trade-off has moved to the runtime. Users of granite-docling-258M report 15–20 minutes per document with a naive CPU run, while another user processed their file through llama-server with a bf16 GGUF in ~6 seconds.
• Legally, what disappears isn’t “risk” but a whole class of questions. No cross-border transfer — no GDPR Chapter V. No new sub-processor — no Article 28 agreement.
• Regulatory uncertainty will last longer than planned. The Digital Omnibus — Regulation (EU) 2026/1744 — has been in force since 27 July 2026: obligations for high-risk systems have been pushed back to 2 December 2027 and 2 August 2028.

Six seconds or twenty minutes

granite-docling-258M is an IBM Research model for document conversion, released in September 2025 under the Apache 2.0 license. Two hundred and fifty-eight million parameters. By 2026 standards that’s next to nothing: a hundred times smaller than the models you run through an API.

The Docling project’s tracker has issue #2348, where an engineer describes evaluating the model on a CPU: generating the output for a document takes fifteen to twenty minutes. In discussion #37 on the model’s Hugging Face card, another user complains that even on an RTX 6000 Ada with 48 gigabytes of VRAM a single page takes unacceptably long. Another participant in the thread adds a remark that makes you wince: multi-billion-parameter models in LM Studio produce tokens noticeably faster than this 258-million one.

In the same thread a third participant asks the right question: what exactly are you measuring? And shows their own measurement on a single file. The full pipeline through the Docker UI — about 35 seconds. The same model through llama-server with bf16 quantization in GGUF format — 6 seconds. What’s more, the answer from the “plain,” non-pipeline call turned out to be more accurate.

One model, different engines, a spread of two orders of magnitude.

The question “is a local model good enough” is already closed for a whole class of tasks. The question “can you run it properly” is still open, and that’s the one people get burned on.

Three tasks where a local model is already good enough

What these three tasks have in common: the model doesn’t make anything up; it converts one kind of signal into another — sound into text, pixels into structure, an image into a label. These are narrow, deterministic, high-volume operations, and that’s exactly where small models hold their own.

Speech recognition: Ukrainian included

NVIDIA released parakeet-tdt-0.6b-v3 — an automatic speech recognition model with 600 million parameters. Its predecessor, v2, could only handle English. The third version has been extended to 25 European languages, and Ukrainian is on the list, alongside Polish, Czech, Romanian and Lithuanian.

The numbers from the model card, WER (word error rate, lower is better):

Language	Fleurs	CoVoST
Ukrainian	6.79%	5.10%
Russian	5.51%	3.00%
Portuguese	4.76%	3.96%
Swedish	15.08%	20.16%
Slovenian	24.03%	31.80%
Average across languages	11.97%	11.98%

Ukrainian does noticeably better than average. The model detects the language of the audio on its own — no hint needed. It adds punctuation and capitalization, returns word-level and segment-level timestamps, and handles audio of up to 24 minutes with full attention and up to three hours with local attention. The license is CC BY 4.0 — commercial use is allowed.

Six hundred million parameters. That’s not the kind of weight you buy a server for.

Look at the bottom rows of the table. Slovenian in the same model scores 24% — nearly one word in four. If your audience speaks Slovenian, the previous paragraph doesn’t apply to you: the average on a model card doesn’t describe your language. Look at your own row.

Document OCR: structure, not just text

Classic OCR gives you the text and loses everything else — where the table was, where the formula was, where the figure caption was. For a RAG pipeline that’s a loss you can’t make up for later.

granite-docling-258M is built differently. Under the hood is the Idefics3 architecture, in which IBM swapped the vision encoder for siglip2-base-patch16-512 and the language part for Granite 165M. The model preserves the markup: tables stay tables, formulas stay formulas, code stays code, and the output lands in a DoclingDocument — a structure that’s easy to chunk for search.

Apache 2.0. You can take it, you can modify it, you can ship it inside a product.

And right here is the same runtime trap this article opened with. The model is small, but it’s a full vision-language model: it generates its output token by token for every page. A naive run through transformers on a CPU turns that into a twenty-minute wait. The same weights file under the right engine — seconds.

Classification and vision: what’s already in your pocket

Apple published FastVLM — a vision-language model presented at CVPR 2025, with an open repository. What matters most about it is time to first token, not accuracy: FastVLM-0.5B returns the first token 85 times faster than LLaVA-OneVision and its vision encoder is 3.4 times smaller. The larger FastVLM-7B, built on Qwen2, is 7.9 times faster than Cambrian-1-8B at comparable accuracy. The demo runs right in the browser via WebGPU.

On 8 June 2026 Apple showed the third generation of Apple Foundation Models — five models, from ones that live on the device to server-side ones on Private Cloud Compute. At the framework level it looks mundane: in iOS 26 an app gets a session with a three-billion-parameter on-device model, and there is simply no network in the call path.

This is an important signal, but not because Apple did something unique. It’s because the “on-device model” has stopped being a research genre and become a part of the operating system that every user buys along with the phone.

Where a local model breaks down

There is a boundary, and it runs through three places.

A language that isn’t in your row of the table. We’ve already looked at Slovenian. The general rule: a small model’s quality is unevenly distributed, and that distribution doesn’t correlate at all with how much a language matters to you. Before you roll it out, take two hundred minutes of your own audio — not a benchmark, your own — and measure.

A runtime that degrades quietly. On 27 March 2026, issue #3202 appeared in the Docling tracker; its author claims that TableStructureModel has an unconditional MPS-to-CPU fallback hard-coded into it, which silently switches off acceleration on Apple Silicon. Their measurements: 145.9 seconds on CPU versus 10.4 on MPS, and 715.4 seconds versus 41.1 after switching to MLX — a 14–17x speedup. As of October 2026, the issue is still open.

What interests me in this story is the word “silently.” The model doesn’t crash. The logs are clean. You just get performance fifteen times worse than it could be and write it off as “well, local is slower anyway.” It’s the same class of problem as the blast radius of an AI agent: what costs the most isn’t the thing that broke loudly.

Tasks that call for reasoning. Transcription, extraction, classification — that’s conversion. Text generation, multi-step analysis, long context, dealing with ambiguity — that’s something else. Here a small local model loses, and not by a little. Don’t try to use it in place of the call where a strong model is exactly what you need.

Which class of questions disappears

When you send a scan of a client’s passport to an external API, you’re doing more than one thing, and each of them comes with legal baggage. You’re handing personal data to a processor, which means you need a data processing agreement under Article 28 GDPR. If the processor is outside the EU, Chapter V on cross-border transfers kicks in, with everything that comes with it. If the processing is large-scale or sensitive, you need an impact assessment under Article 35.

A local model doesn’t make you “GDPR compliant.” It does something narrower and more practical: it removes a whole term from the equation. No transfer — no Chapter V. No new sub-processor — no additional Article 28 agreement. The client’s data doesn’t leave the perimeter where it was lawfully sitting anyway.

This isn’t legal advice, and your lawyer still has to look at the specific setup. But “we reduced the risk” and “we removed the grounds for the question” are different sentences.

The second layer is regulatory. The European Commission published the Digital Omnibus on AI on 19 November 2025. The European Parliament approved the agreed text on 17 June 2026 by 423 votes to 57, with 174 abstentions. The Council of the EU gave its final green light on 29 June 2026. The text was published in the Official Journal on 24 July 2026 as Regulation (EU) 2026/1744; it entered into force on 27 July 2026.

What this means in dates: obligations for high-risk systems under Annex III have been pushed back to 2 December 2027, and for AI embedded in regulated products under Annex I, to 2 August 2028. The original deadline was 2 August 2026.

The deadline shift doesn’t mean you can relax: it means the uncertainty will last another year and a half to two years. An architecture in which personal data physically never leaves your perimeter holds up no matter which text ends up in the Official Journal. An architecture built around a specific clause of a specific version of the regulation won’t.

What it actually costs

A local model isn’t “free instead of a subscription.” You’re swapping one cost line for another: instead of per-token billing that grows linearly with volume, you get hardware that’s bought once and engineering time that’s spent constantly.

Count the second part in full: choosing an engine, measuring instead of assuming, noticing that acceleration has quietly switched off, updating the model when the next version comes out. That’s upkeep, not a one-off job.

The arithmetic tips toward local in two cases. The first is volume: when the same type of document goes through the system by the thousands every month, the per-token price stops being a rounding error. The second is when sending data outside either costs you dearly in legal terms or is simply unacceptable to the client.

If you have three hundred documents a month and no personal data, stay on the API. It’ll be cheaper and more straightforward. As for how I decide whether to take on a new tool at all, there’s a separate breakdown.

The routing rule

The practical conclusion I reached while building pipelines that extract data from PDFs and scans for clients goes like this.

Route not by how sensitive the task is, but by what exactly crosses the perimeter.

The raw document, live audio, a face in a photo — these stay inside. A local model turns them into derivatives: text without names, a feature vector, a class label, a structured table. After that, if the next step really does need a strong model, what goes out is the derivative, which contains no personal data.

Recognition, extraction, classification — down, onto your own hardware. Reasoning, generation, long context — up, into the cloud. The line runs between conversion and thinking, not between “important” and “unimportant.”

Three years ago this split was theoretical, because the local half couldn’t deliver on quality. Now it can. The price isn’t a worse result but the obligation to figure out how the thing actually runs on your setup. I laid out which local tools I run myself every day in my AI stack.

Six seconds or twenty minutes. The difference isn’t the model.

FAQ

Do local AI models support Ukrainian?

Yes. NVIDIA’s parakeet-tdt-0.6b-v3 model recognizes Ukrainian among 25 European languages with automatic language detection. WER for Ukrainian is 6.79% on the Fleurs benchmark and 5.10% on CoVoST, which is better than the average across all supported languages (11.97% on Fleurs). The CC BY 4.0 license permits commercial use.

Does a local model mean GDPR compliance?

No. A local model doesn’t automatically make data processing lawful — the legal basis for processing, data minimization and retention periods remain your responsibility. It takes specific legal questions off the table: if data isn’t passed to an external processor, there’s no need for an Article 28 GDPR agreement or for the cross-border transfer mechanisms of Chapter V. A lawyer has to assess the specific setup.

What hardware does a local OCR model need?

Model size matters less than the engine. granite-docling-258M has 258 million parameters, yet with a naive run through transformers on a CPU users report 15–20 minutes per document, while another user processed a file through llama-server with a bf16 GGUF in seconds. Before buying hardware, measure your pipeline on your own documents.

When is a local model worse than a cloud one?

In three cases: when your language is poorly represented in the model (in Parakeet, Slovenian scores 24.03% WER against 6.79% for Ukrainian); when the task calls for reasoning, generation or long context; when the volume is small and the engineering time spent on upkeep costs more than the API bill.

What has changed in the AI Act deadlines?

The European Parliament approved the Digital Omnibus package on 17 June 2026, the Council of the EU gave its final green light on 29 June, and on 24 July 2026 the text appeared in the Official Journal as Regulation (EU) 2026/1744 and entered into force on 27 July. Obligations for high-risk systems under Annex III have been moved to 2 December 2027, and for AI in regulated products under Annex I, to 2 August 2028. The original deadline was 2 August 2026.

Can you run an AI model on a phone?

Yes. Apple’s FastVLM runs on an iPhone and even in the browser via WebGPU, and FastVLM-0.5B returns the first token 85 times faster than LLaVA-OneVision. Starting with iOS 26, apps get access to a three-billion-parameter on-device model through the Foundation Models framework — there’s no network call on that path.

Where do you start the move to local models?

Take one of the three tasks — transcription, OCR, classification — and measure it on your own data, not on a benchmark. First quality on your language and your documents, then speed under a specific engine. If both numbers work for you, move that one task and leave the rest of the pipeline as it is.

What’s next

In 2026 the question “can you trust a local model” splits in two. Whether it’s accurate enough on your data can be measured in half a day, and the answer is usually “yes.” Whether you’ll manage to run it so that it isn’t two hundred times slower than it could be is a question about your team, not about the model.

If people at your company are deciding right now whether to hand client documents to an external API, forward them this article before the decision is made. And if the pipeline is already running and you don’t know what exactly leaves it, come to a discovery session — an hour is usually enough to see which data crosses the perimeter and whether there’s a basis for it.

See also:

• AI Agent Blast Radius: How to Measure Risk Instead of Counting Tokens — on how the credential determines the scale of the damage.
• How I Evaluate a New AI Tool in 20 Minutes — the criterion first, the demo second.
• My AI Stack 2026: What I Actually Use Every Day — which tools actually live in my daily work.
• Which Automations to Implement First: 5 Steps in Order — where to start so there’s something to measure.

Sources:

• parakeet-tdt-0.6b-v3 — NVIDIA model card: 25 languages, WER by language, CC BY 4.0 license.
• Docling issue #3202 — the MPS→CPU fallback in TableStructureModel, 14–17× speedup benchmarks, 27 March 2026.
• IBM Granite-Docling: End-to-end document understanding — IBM announcement, September 2025.
• FastVLM: Efficient Vision Encoding for Vision Language Models — Apple Machine Learning Research, CVPR 2025.
• Introducing the Third Generation of Apple’s Foundation Models — Apple, 8 June 2026.
• Council gives final green light to simplify and streamline AI rules — Council of the EU, 29 June 2026.
• Regulation (EU) 2026/1744 (Digital Omnibus on AI) — EUR-Lex, published in the OJ on 24 July 2026, in force since 27 July 2026.

---

Published by Andrew Maryasov, AI consultant, founder of Auspex and Grow2.ai; implementations are run by Auspex (CRM) and Grow2.ai (AI agents).