Downloading a model and running it on a company computer or server appears to offer a simple answer to confidentiality concerns: if the model is “in-house”, data does not leave. This can be an advantage, but it is not an automatic guarantee. An AI system includes runtimes, interfaces, documents, vector databases, logs, backups, libraries, updates and integrations. To determine whether data genuinely remains under control, the entire flow must be examined—not only where the model weights are stored.

Why local AI is back at the centre of the discussion

Open-weight model families have made local inference more accessible. OpenAI describes gpt-oss-20b as capable of running on systems with 16 GB of memory, while Google provides Gemma variants and tools for running them on workstations, servers and devices. The choice is no longer limited to large data centres, although performance and requirements vary significantly according to the model, quantisation and actual workload.

For an organisation, interest comes from practical needs: data residency, confidential documents, offline operation, predictable latency, customisation and reduced dependence on a single provider. These benefits must be compared with hardware, energy, updates, security, monitoring and operational expertise. “Free to download” does not mean “free to operate” or “ready for production”.

Open source and open weight are not synonyms

In commercial language, many models are called open source because their weights can be downloaded. The Open Source Initiative’s Open Source AI Definition, however, requires effective freedoms to use, study, modify and share, together with the preferred form needed to modify the system, including sufficient information about training data and processes.

An open-weight model exposes its final parameters, often under a permissive licence, but may not provide datasets, training pipelines or every component required to reproduce it. This distinction does not determine model quality by itself, but it affects transparency, modifiability, licensing and provider dependency. Before adoption, organisations should therefore inspect the licence, usage policy, provenance, model card, software components and runtime conditions.

“Local” can describe very different architectures

A model may run on an individual laptop, a server inside the corporate network, a managed data centre, a private cloud or a dedicated instance offered by a provider. All of these are often described as private, but they have different boundaries, responsibilities and attack surfaces.

On-premises means that infrastructure is under the organisation’s direct control; private cloud describes a logically or physically isolated cloud environment; self-hosted identifies who operates the application, not necessarily where servers and data reside. Before making privacy claims, organisations need a data-flow diagram, a list of involved parties and the location of prompts, files, embeddings, logs, backups and temporary copies.

Where prompts, documents and logs actually travel

Even when inference runs locally, the application may contact external services for telemetry, updates, authentication, web search, document conversion or connected tools. An internally installed interface may also call remote APIs without making this obvious to users. Firewalls, proxies and connection monitoring should verify actual behaviour rather than relying only on declared configuration.

In RAG systems, documents are extracted, split and converted into numerical representations known as embeddings, which are stored in a vector database. Embeddings, indexes and metadata are derived data, not automatically anonymous data. Conversation history, caches, outputs, system prompts and application logs may also contain confidential information. Retention, encryption, authorisation and deletion must cover the whole pipeline.

The model itself may remain within the scope of personal data

The EDPB explains that whether an AI model is anonymous must be assessed case by case. If people represented in the training data can reasonably be identified, or if personal information can be extracted through queries or access to parameters, the model cannot be treated as anonymous merely because it is described that way.

Running the model internally may reduce disclosures to third parties, but it does not resolve data provenance and lawfulness, minimisation, purpose limitation, data-subject rights or the risk that memorised information may reappear in outputs. These issues should be assessed with the people responsible for data protection, security and the business process. Technical architecture supports those decisions but does not replace legal and organisational assessment.

More control creates more operational responsibility

With a managed service, part of patching, capacity, monitoring and continuity is delegated to the provider. In self-hosting, these responsibilities return to the organisation: authentication, access segregation, encryption, secrets management, updates, dependency vulnerabilities, backups, recovery, usage limits and event logging.

Models and runtimes are also part of the supply chain. Weights, containers and packages should come from verified sources; versions and configurations need to be pinned and tested before release. An unprotected local server can be less secure than a well-governed cloud service. The real benefit is not physical location by itself, but the ability to apply coherent controls and verify them over time.

Cloud, local or hybrid: decide from the use case

Cloud APIs suit projects that need rapid setup, elasticity and advanced models, provided data, contracts, retention, region and subprocessors are compatible with the use case. On-premises deployment is attractive for stable workloads, sensitive data, offline requirements or strong infrastructure control, but it requires internal technical and operational capability. Private cloud sits between these options without removing the need to assess its operator and configuration.

A hybrid architecture may keep classification, anonymisation, document retrieval or sensitive tasks local while using a remote service only for authorised and minimised inputs. It is not automatically the best answer: it adds components, integrations and monitoring points. The decision should compare quality, latency, volume, total cost, permitted data, availability and the consequences of failure.

Start with a pilot that measures privacy too

A useful pilot begins with one process, an authorised dataset and observable success criteria. Before installation, map the data flow, identify every outbound connection, define retention and roles and specify which information must never enter the system. Synthetic or minimised data makes it possible to verify the architecture before introducing real content.

During the pilot, measure response quality, errors, latency, memory consumption, operating cost and human intervention. Authorisation, isolation, backup, recovery and update tests matter as much as model benchmarks. Only after this evidence is available does it make sense to extend the system, select a different model or retain a controlled combination of local and cloud services.

OFFICIAL SOURCES

Further reading.