CIIFragments Studio is CII-accredited: recover up to 20% of your software development spendLearn more
Back to the blog
Local and sovereign AI: running an LLM on your sensitive dataTech · 6 min

Local and sovereign AI: running an LLM on your sensitive data

Local and sovereign AI: Ollama, vLLM, Mistral, sovereign cloud. Compare costs, quality and architecture to run an LLM on your sensitive data.

KA
Tech Lead

Local AI means running a language model on infrastructure you control, so your documents never leave your perimeter. For healthcare, legal, HR or manufacturing, it is sometimes the only viable option. Here is how to choose between Ollama, vLLM, Mistral and sovereign cloud, with the real trade-offs in cost and quality.

Local AI, sovereign AI, open source: what are we talking about?

These three terms are often confused, yet they describe three different things. Local AI is a model running on a machine you control (workstation, internal server, private cluster). Sovereign AI additionally guarantees control over jurisdiction, operator and reversibility. An open source model (or more precisely an open weight model) is a model whose weights can be downloaded.

Local does not mean open source

A local LLM can perfectly well use a model under a restrictive license. Conversely, an open weight model can run at a US hosting provider. The three criteria are independent, and each must be checked separately.

A point to watch: the license. Some "open" models restrict commercial use or redistribution. Read it before building a product on top.

What sovereignty really covers

Sovereignty is not just about the country where the datacenter sits. A European provider owned by a foreign parent company can remain exposed to the Cloud Act, the US law that allows US authorities to request access to data held by US companies, including outside their territory.

Four questions to ask: who owns the provider, who has administrator access, where the backups are, and whether you can migrate without depending on them.

Why and when should you keep your data in-house?

Keeping your data in-house becomes necessary when the documents processed are regulated, covered by trade secrets or contractually barred from leaving. In other cases, a well-contracted API is often still acceptable.

GDPR: a requirement for control, not a ban

The GDPR does not prohibit AI APIs. It requires you to control the legal basis, the processor role, transfers outside the European Economic Area and the retention period of prompts. The CNIL (the French data protection authority) publishes practical recommendations on the subject.

Local AI reduces the exposure surface. It does not exempt you from documenting the processing or managing access rights.

At Fragments Studio, we see that demand for local AI almost always comes from one of these situations:

  • Health data, which in France requires a host certified HDS (hébergeur de données de santé, the French health data hosting certification).
  • Client files or litigation covered by professional secrecy.
  • HR evaluations, payroll or interviews.
  • Plans, bills of materials or industrial processes covered by trade secrets.

AI Act: local helps, but does not make you compliant

The AI Act classifies uses by risk level. A local model does not make a use compliant: the use case determines the obligations. It does, however, make traceability, log auditing and version control easier.

Our AI Act compliance guide for SMBs details these obligations.

What are the options in 2026: Ollama, vLLM, Mistral or sovereign cloud?

There is no single option: Ollama is for prototyping, vLLM for industrializing, Mistral provides models deployable in Europe, and sovereign cloud (OVHcloud, Scaleway) saves you from buying GPUs. The right choice depends on volume, sensitivity and your operational skills.

What are the options in 2026: Ollama, vLLM, Mistral or sovereign cloud?
OptionMain strengthLimitationRecommended use
OllamaOne-command install, local APILimited concurrencyPrototype, few users
vLLMHigh throughput, multi-GPUMore technical operationsMulti-user production
Mistral modelsOpen weight, European vendorLicense to check depending on the modelSensitive data, European context
Sovereign cloudNo hardware purchaseRecurring cost, dependencyVariable load, EU hosting
Proprietary APIMaximum qualityOutbound data flowsNon-sensitive data

Ollama: the right tool to validate a use case

Ollama is a runtime that downloads and runs a model locally, with an OpenAI-compatible API. It lets you test three models in a day. It is not designed for high concurrency, high availability or fine-grained monitoring.

vLLM: the production engine

vLLM is an inference engine that uses GPU memory efficiently thanks to continuous batching, that is, the dynamic grouping of concurrent requests. It also exposes an OpenAI-compatible API. In return, it requires skills in containers, GPU drivers and monitoring.

Mistral: a European vendor, two ways to use it

Mistral AI offers models via API and open weight models that you can deploy on your own infrastructure, for example with vLLM. For an architecture built on these models, see our Mistral integration page.

Sovereign cloud: OVHcloud and Scaleway

OVHcloud and Scaleway rent GPUs in European datacenters. Check certifications at project time (such as SecNumCloud, the ANSSI qualification for trusted clouds, ANSSI being the French national cybersecurity agency) and the contractual guarantees: they vary between offers.

We regularly deploy on OVHcloud, and the containerized deployment approach matches the one in our article on migrating to ECS Express Mode.

How much does a local LLM really cost compared with proprietary models?

A local LLM is rarely cheaper than an API at low volume: its real benefit is control. According to a Qovery comparison of European GPUs, below roughly 20 billion tokens per month for a 70B-class model, self-hosting is more about sovereignty than savings.

The GPU line item: buy or rent

The same Qovery benchmark reports an H100 PCIe at €2.87 per hour at Scaleway. For a machine running continuously, that comes to about €2,000 per month, before storage and network.

Buying hardware only pays off if the load is stable and heavily used.

The hidden cost: operations

The human cost is most often underestimated. An LLM service must be treated as a critical service:

  • monitoring GPU saturation and response times;
  • managing model updates and regression tests;
  • backups, network security and secrets management;
  • on-call duty if the service becomes essential to teams.

Our position is clear: without someone responsible for operations, a local LLM quickly becomes more expensive than the API it replaces.

Quality: where proprietary models keep the edge

Proprietary models generally remain ahead on complex reasoning, multimodality and following long instructions. Open weight models, on the other hand, are very competitive on extraction, classification and summarization of specialized documents.

Do not rely on generic leaderboards. Build a test set with your own anonymized documents, and measure accuracy, hallucination rate, format compliance, latency and cost per request. A well-chosen small model often beats a large model on a targeted task.

What does the architecture of a local AI connected to your data look like?

A local AI architecture places the model behind an internal gateway that manages permissions, logging and access to documents. The document retrieval principle is detailed in our article on RAG and company data: here, we focus on the infrastructure.

The essential building blocks

The typical flow follows five steps:

  1. The user queries your business application.
  2. An internal gateway authenticates the request and applies read permissions.
  3. The document retrieval service fetches only the authorized documents.
  4. The inference server (Ollama or vLLM) runs the local model.
  5. The response is logged and then returned to the application.

The three rules that prevent unpleasant surprises

  • Segment the network: the inference server must not be able to reach the Internet without an explicit rule.
  • Respect existing permissions: the model must never see a document the user cannot open.
  • Version your models: every model change goes through your regression tests.

The right approach is often hybrid: sensitive data on controlled infrastructure, external API for non-confidential uses.

Frequently asked questions

What is local AI?

Local AI is a language model running on a machine or infrastructure you control, without sending your documents to an external provider. It reduces data exposure but does not exempt you from documenting the processing under the GDPR.

What is the difference between local AI and sovereign AI?

Local AI describes where the model runs. Sovereign AI adds control over jurisdiction, operator and reversibility. A European cloud can be sovereign without being local, and a local server can depend on a foreign model or provider.

How do you run an LLM on your own data?

  • Identify the data that cannot leave your information system.
  • Test several open weight models with Ollama on anonymized documents.
  • Move to vLLM on a dedicated GPU or in a sovereign cloud for production.
  • Add an access gateway, logging and monitoring.

Why does sovereign AI matter for an SMB?

Because sensitive data (healthcare, litigation, HR, manufacturing) engages the company's legal liability. Controlling where it is processed also protects trade secrets and avoids exclusive dependence on a single provider.

Conclusion

Local AI is neither a universal solution nor an automatic saving: it is a choice of control. Ollama validates a use case, vLLM puts it into production, Mistral brings models deployable in Europe and sovereign cloud avoids the hardware investment.

The right trade-off rests on three variables: the sensitivity of your data, your request volume and the expected quality. Start small, measure on your own documents, then size accordingly.


Launch your local and sovereign AI project

Running an LLM on your data is decided on architecture, compliance and cost criteria. We can help you decide and then deploy.

Found our content useful?

Follow Fragments Studio on Google

Add us to your preferred sources and our articles get surfaced first in Top Stories, AI Overviews and AI Mode.

Add to Preferred Sources

Ready to bring your projects to life?

Fragments Studio handles everything: from strategy to production.

Discuss my project
Discuss my project