Local & Open Source
Self-hosting AI models: what it takes and when it pays off
What "self-hosting AI" actually means
Self-hosting AI means running the model on infrastructure you govern instead of sending your prompts to a third-party API such as OpenAI's or Anthropic's. The models you can self-host are "open-weight" models: their trained parameters are published for download (Llama, Mistral, Qwen, Gemma, DeepSeek and Switzerland's Apertus). You download the weights and serve them yourself.
Self-hosting is a spectrum, not a single choice:
- Local device: a laptop or workstation running Ollama or LM Studio - ideal for prototyping and single-user tasks.
- On-premises server: a GPU machine in your own office or data centre - data never leaves the building.
- Private/dedicated cloud: rented but isolated GPU instances (e.g., a dedicated VPC) - you control the environment without owning hardware.
What it takes: the hardware
The bottleneck is VRAM, the GPU's video memory, which must hold the model plus its context. Bigger models and longer context need more VRAM. Quantization - compressing the weights to 4-bit or 8-bit - cuts requirements dramatically with only modest quality loss. As rough rules of thumb:
- Small (3-8B params), 4-bit: ~4-8 GB VRAM - runs on a consumer GPU or a modern Apple Silicon Mac; good for chat, summarization, RAG.
- Mid (12-34B), 4-bit: ~10-24 GB VRAM - one high-end consumer GPU (e.g., 24 GB) or a workstation card.
- Large (70B), 4-bit: ~40+ GB VRAM - professional GPUs or multi-GPU; noticeably closer to frontier quality.
- Frontier open (100B+/MoE): multiple data-centre GPUs - usually only justified in a private cloud.
The software stack and the skills
Beyond the GPU: concurrency multiplies everything - ten simultaneous users need far more than one. Factor in CPU, RAM, fast storage, electricity and cooling. NVIDIA (CUDA) dominates; Apple Silicon with unified memory is excellent for local single-user work; AMD is catching up but has a smaller ecosystem.
The tooling has matured. To get started, Ollama and LM Studio wrap everything into a near one-click experience. For production, vLLM and Hugging Face TGI add batching, higher throughput and an OpenAI-compatible API so your apps barely change. llama.cpp underpins many of these and runs efficiently even on modest hardware.
The harder part is operations, not installation. Someone must:
- Manage Linux, GPU drivers and CUDA versions
- serve the model reliably, monitor latency and uptime, and handle scaling
- patch security, control access, and keep an audit trail
- evaluate and swap models as better ones ship (the open ecosystem moves fast)
- build the actual value layer: RAG, prompts, guardrails, integration
Cost: self-hosted vs. API
Realistically an SME needs at least one capable engineer or an external partner; underestimating this ongoing effort is the most common reason self-hosting projects disappoint. The economics hinge on volume and utilization. An API charges per token: zero upfront, scales instantly, you pay only for what you use. Self-hosting front-loads cost: you buy or rent GPUs and pay for electricity, cooling and - the biggest hidden item - engineering time, whether the GPU is busy or idle.
The break-even favours self-hosting when utilization is high and steady - a busy, always-on workload amortizes the fixed cost. For low or spiky volume, an idle GPU makes APIs cheaper. Model this honestly, including staff time, before buying hardware. Quick comparison:
- Upfront cost: API none; self-hosted high (hardware) or moderate (rented)
- Marginal cost per request: API per-token; self-hosted near zero once running
- Scaling: API elastic; self-hosted limited by your hardware
- Idle cost: API zero; self-hosted you pay regardless of use
- Ops burden: API minimal; self-hosted significant
Pros and cons at a glance
- Pro - Data sovereignty: prompts and documents never leave your control; decisive for sensitive data.
- Pro - Predictable cost at scale: no per-token bill once the hardware is paid off.
- Pro - Independence: no vendor lock-in, no surprise price or policy changes, works offline/air-gapped.
- Pro - Control: choose the exact model version, customize and fine-tune, tune latency.
- Con - Capital and operations: hardware spend plus continuous engineering.
- Con - Quality gap: the best open models trail the top proprietary ones on hard reasoning - though the gap is narrowing.
- Con - Scaling friction and full responsibility: demand spikes are hard to absorb; security, uptime and compliance are entirely yours, including the treadmill of adopting new models.
When is it worth it for an SME?
Self-hosting makes most sense when at least one of these is true:
It is usually not worth it when volume is low or unpredictable, when you need the strongest possible reasoning, or when you have no operations capacity - then a compliant API or a private-cloud deployment (e.g., a provider's Swiss/EU region or an isolated VPC) often serves better. Many SMEs land on a hybrid: self-host small, high-volume, sensitive tasks; call an API for the hardest ones.
Two Swiss-relevant points. First, self-hosting does not by itself make you compliant - the revDSG, EDOEB guidance and the extraterritorial EU AI Act apply to your AI use regardless of where it runs; self-hosting mainly removes the cross-border data-transfer problem. Second, Switzerland now has Apertus, a fully open large language model from ETH Zurich, EPFL and CSCS - a notable option when transparency and Swiss provenance matter.
- You process sensitive or regulated data (health, legal, financial, HR) where keeping data in-house simplifies revDSG/nFADP compliance and client trust.
- Volume is high and steady enough to amortize hardware.
- You need offline operation, guaranteed latency, or air-gapped environments.
- You already have solid IT/engineering capability or a trusted partner.
A pragmatic roadmap
- Start with a local pilot: run Ollama with a small open model on one workstation and test it on real tasks for a week.
- Measure honestly: quality against your use case, expected monthly volume, and required latency.
- Compare total cost: hardware plus staff time versus an equivalent API bill.
- Decide the tier: local, on-prem, private cloud - or hybrid.
- Invest in the value layer (RAG, integration, guardrails), which matters more than the raw model.
Frequently asked questions
Do I really need an expensive GPU to self-host AI?
No, not to start. A small open model quantized to 4-bit runs on a good consumer GPU (around 8 GB VRAM) or a modern Apple Silicon Mac, which is enough for chat, summarization and RAG for a few users. Expensive professional GPUs only become necessary for large models or many concurrent users.
Does self-hosting automatically make my AI revDSG-compliant?
No. Self-hosting keeps data in-house and removes cross-border transfer risk, which helps - but compliance depends on how you handle personal data overall. The revDSG/nFADP, EDOEB guidance and the EU AI Act still apply. Self-hosting is a strong building block, not a compliance guarantee.
What is the difference between open source and open weights?
"Open weights" means the trained parameters are downloadable and you can run the model yourself; the training data and full recipe may not be public. "Open source" in the strict sense additionally publishes code and often data under an open licence. Most self-hostable models (Llama, Mistral) are open-weight; Apertus is a fully open example.
Which open models are suitable for an SME?
Popular, capable families include Llama, Mistral, Qwen, Gemma and DeepSeek, plus Switzerland's Apertus. Choose by task, language coverage (important for DE/FR/IT), a size that fits your hardware, and licence terms. Test two or three on your own data rather than trusting benchmarks alone.
Is self-hosting the same as Azure OpenAI or a Swiss cloud region?
Not quite. Those are managed/private-cloud options: the provider still runs the model, but in a defined region or isolated tenancy. True self-hosting means you run open-weight models on infrastructure you control. Private cloud is a middle ground - less operational burden, less absolute control.
Is fine-tuning worth it, or is RAG enough?
For most SMEs, retrieval-augmented generation (RAG) - feeding the model your documents at query time - delivers more value for less effort than fine-tuning. Fine-tuning helps for fixed style, format or narrow domain tasks. Start with a good base model plus RAG; consider fine-tuning only if RAG falls short.
Key terms in the glossary
Practical AI for your business
From idea to implementation – we show you what is concretely possible in your case.
Request a demo