What Is On-Premise AI — and Is It Worth It for an SME?

What Does "On-Premise AI" Actually Mean?

On-premise AI (also called self-hosted or local AI) refers to running AI models — primarily large language models (LLMs) — on infrastructure the organisation itself controls. Prompts, documents and responses never leave your environment and are not transmitted to an external provider.

In practice "local" is a spectrum: from a single GPU workstation as a pilot, to a server in your own data centre, to a dedicated, single-tenant instance at a Swiss provider (sovereign cloud). What matters is not location alone but who controls the data and the keys.

• On-premise: hardware in your own server room or data centre — full physical control.
• Private/sovereign cloud: a dedicated instance at a Swiss provider, no shared tenancy.
• Cloud API (for comparison): fast and scalable, but data leaves the company.
• Air-gap: fully internet-isolated operation for maximum confidentiality.

Reference Architecture: the Building Blocks of an On-Premise Stack

An on-premise AI system is a layered model. Hardware sits at the bottom, the software that runs the model above it, the applications on top, and operations and security run across all of it. Separating the layers lets you swap models without rebuilding the application.

• Hardware layer: GPU server with sufficient VRAM, fast networking and storage.
• Inference runtime: the software that loads the model and generates responses (e.g. vLLM, Ollama, llama.cpp, Hugging Face TGI).
• Model layer: open-weights models such as Llama, Mistral, Qwen, Gemma or Switzerland's Apertus.
• Knowledge grounding (RAG): vector database, embeddings and retrieval so the model can draw on your own documents.
• Application layer: chat interface, API gateway and integrations into existing systems.
• Operations & security: authentication, access control, logging, monitoring and updates.

Hardware and Software Requirements

The key bottleneck is graphics memory (VRAM). It determines how large a model can be. Quantisation — storing the model with fewer bits per parameter — sharply reduces VRAM needs, usually with only minor quality loss. A rough rule of thumb: VRAM ≈ parameter count × bytes per parameter (4-bit quantisation ≈ 0.5 byte).

• 7–8B-parameter model (quantised): roughly 6–12 GB VRAM — runs on a strong consumer GPU (16–24 GB).
• ~70B-parameter model (quantised): roughly 40–48 GB VRAM — needs a data-centre GPU or multiple cards.
• CPU-only operation is possible but much slower; a GPU is recommended for production load.
• Further sizing factors: number of concurrent users, context length and required throughput (tokens per second).
• The software stack is largely open source and self-hostable; check current hardware prices with the respective vendor.

Security and Compliance Benefits — the Swiss Lens

The strongest benefit of on-premise is data sovereignty: personal data, trade secrets and client files stay inside the company. That eases proof of compliance with the revised Data Protection Act (revFADP/nFADP) and avoids sensitive cross-border transfers. For professions bound by statutory secrecy — lawyers, doctors, fiduciaries (Art. 321 Swiss Criminal Code) — local processing can be decisive.

The EU AI Act matters too: it applies extraterritorially and can affect Swiss companies selling into the EU. On-premise eases traceability and control, but does not replace governance. Important: local does not mean automatically secure — hardening, access control, encryption and patching remain your responsibility.

• No transmission to third parties and no uncontrolled cross-border data transfer.
• Prompts and documents are not fed into third-party model training.
• Air-gapped operation possible for highly sensitive environments.
• Swiss oversight: the FDPIC; clear accountability instead of foreign provider terms.
• But: security is not automatic — you need your own operational security concept.

Realistic Expectations: Cost, Quality, Effort

On-premise is not a set-and-forget move. The quality of good open-weights models is often entirely sufficient for many SME tasks — summarising, drafting, internal search, classification. On complex reasoning, long chains and multilingual nuance, however, the largest cloud models often still lead. That gap is narrowing but will not vanish overnight.

• Cost model: high upfront hardware investment (CapEx) instead of recurring cloud fees (OpEx); payback only above a certain, steady usage volume.
• Operational effort: updates, model swaps, scaling and availability must be handled in-house or via a partner.
• Time to production: weeks to months rather than days — including data integration and testing.
• Don't build everything yourself: appliance or managed-on-prem approaches sharply cut the operational burden.
• Power consumption, cooling and spare hardware belong in the total cost of ownership (TCO).

On-Premise, Cloud or Hybrid? A Decision Guide for SMEs

The question is rarely either/or. Many SMEs do best with a hybrid model: keep sensitive data and recurring standard tasks local, and send peak load and especially demanding tasks to a (ideally Swiss) cloud. Start with one clearly scoped use case and a single GPU server rather than planning a data centre from day one.

• On-premise makes sense when: highly sensitive data, regulated industries, high steady volume, air-gap requirement.
• Cloud makes sense when: fast start, highly variable load, need for frontier models, limited IT resources.
• Hybrid: the best of both — local data sovereignty plus cloud frontier models when needed.
• Pilot first: pick one measurable use case, prove the value, then scale.