Before you start
- A Debian 13 (or recent Ubuntu) machine or VM with at least 16 GB of RAM; a graphics card is strongly recommended
- Docker and Docker Compose installed for the web interface (see Installing Docker on Debian or Ubuntu)
- sudo access, and command-line basics (see Linux commands for beginners)
In plain words
Assistants like Claude run in their publisher's data centres. With a local AI, it's the opposite: an open model runs on your machine, and your questions, documents and answers never leave your network. Ollama is the engine: it downloads the models and runs them. Open WebUI is the interface: a web chat page, similar to the big assistants', reachable from every computer in the house. The result is less powerful than the best online models, but private, free to use and available even without the internet.
For the notions of model, token and quantisation, see Generative AI explained simply.
1. The architecture
| Component | Role | Port |
|---|---|---|
| Ollama | Service that loads models into memory and answers requests (API) | 11434 |
| Open WebUI | Web chat interface, user accounts, history, document upload | 3000 (to 8080 in the container) |
| Your computers | A browser, on the local network or through a VPN | — |
Ollama is installed directly on the system, for simple access to the graphics card; Open WebUI runs in a Docker container.
2. Hardware and choice of model
The limiting factor is the graphics card memory (VRAM). Without a graphics card, Ollama runs on the processor and RAM, but much more slowly: usable for small models, painful beyond that.
| Hardware | Models usable comfortably (quantised to 4 bits) |
|---|---|
| Processor only, 16 GB of memory | 1 to 4 billion parameters, slow |
| 8 GB graphics card | 7 to 8 billion parameters |
| 12 to 16 GB graphics card | Up to 14 billion, or 20 billion depending on the model |
| 24 GB graphics card | Up to 30 to 32 billion |
| Apple silicon Mac | Depends on unified memory: about two thirds usable by the model |
NVIDIA cards are the best supported (CUDA). Recent AMD cards work with ROCm, with a more restricted compatibility list.
Which model?
The Ollama library lists the available models, with their sizes. A few well-known open families:
| Family | Publisher | Strengths |
|---|---|---|
| Mistral, Ministral | Mistral AI (France) | Good at French and other European languages, several sizes |
| Qwen | Alibaba | Very versatile, good at code, many sizes |
| Gemma | Compact and efficient, some versions read images | |
| Llama | Meta | Very widespread, large ecosystem |
| gpt-oss | OpenAI | Reasoning-oriented open-weights models |
| DeepSeek-R1 (distilled versions) | DeepSeek | Detailed reasoning, slower |
Start with a 7 to 8 billion parameter model if you have 8 GB of VRAM, then compare on your own uses: general rankings don't replace a test on your questions.
Proxmox VE: giving the graphics card to a VM
On a hypervisor, there are two approaches: a VM given the graphics card through passthrough (the card is then no longer usable by the host or other VMs), or an LXC container that shares the host's card. Passthrough requires IOMMU (VT-d or AMD-Vi) turned on in the BIOS. See the Proxmox documentation at the bottom of the page and Installing Proxmox VE.
3. Installing Ollama
On an up-to-date Debian 13, with an account that has sudo:
curl -fsSL https://ollama.com/install.sh | sh
The script installs Ollama, creates the ollama system user and the ollama systemd service. For an NVIDIA card, first install the proprietary driver and CUDA (see the Ollama documentation), then check that the card is seen:
nvidia-smi
Check the service:
systemctl status ollama
curl http://127.0.0.1:11434/api/version
{"version":"0.x.y"}
Read a script before running it
curl … | sh runs a downloaded script with the user's rights, then sudo. It's the official method, but you can download it and read it first: curl -fsSL https://ollama.com/install.sh -o install.sh, then less install.sh.
4. First steps with Ollama
Download a model and chat with it (the name and size are chosen in the library):
ollama run mistral
Type your question, then /bye to quit. The useful commands:
| Command | Effect |
|---|---|
ollama pull name |
Downloads a model without running it |
ollama run name |
Starts a chat (downloads the model if needed) |
ollama list |
Lists the downloaded models and their size |
ollama ps |
Shows the models loaded in memory, and whether they're running on the graphics card |
ollama show name |
Details of a model: size, quantisation, context length |
ollama rm name |
Deletes a model |
Check that the model is really using the graphics card:
ollama ps
NAME ID SIZE PROCESSOR CONTEXT UNTIL
mistral:latest 6577803aa9a0 5.7 GB 100% GPU 4096 4 minutes from now
100% GPU: the model fits entirely in VRAM. A split value (48%/52% CPU/GPU) means it's spilling over into RAM, with markedly slower answers: choose a smaller model or a stronger quantisation.
The API can be tested directly:
curl http://127.0.0.1:11434/api/generate -d '{"model": "mistral", "prompt": "Explain RAID 1 in one sentence.", "stream": false}'
5. Tuning Ollama
Settings go through the service's environment variables. Open an override for the service:
sudo systemctl edit ollama
Add in the area provided:
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KEEP_ALIVE=15m"
Then reload:
sudo systemctl daemon-reload
sudo systemctl restart ollama
| Variable | Default value | Role |
|---|---|---|
OLLAMA_HOST |
127.0.0.1:11434 |
Listening address; 0.0.0.0 opens it on all interfaces, needed for Open WebUI in Docker to reach it |
OLLAMA_CONTEXT_LENGTH |
4096 tokens |
Context length: the larger it is, the more the model "remembers" of the conversation and documents, but the more memory it uses |
OLLAMA_KEEP_ALIVE |
5m |
How long a model stays loaded after the last request |
OLLAMA_MODELS |
/usr/share/ollama/.ollama/models |
Models folder; handy for putting them on a larger disk (the ollama user must be able to write to it) |
Ollama's API has no authentication
Anyone who can reach port 11434 can use your models, download others or delete them. Thousands of Ollama servers mistakenly exposed on the internet have been found. Never open this port to the internet, and restrict it to the bare minimum with the firewall (part 7).
6. Installing Open WebUI with Docker Compose
Create a folder for the service:
sudo mkdir -p /opt/open-webui
cd /opt/open-webui
Generate a secret key for sessions:
openssl rand -hex 32
Create the file /opt/open-webui/compose.yaml, replacing the value of WEBUI_SECRET_KEY with the generated key:
services:
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: always
ports:
- "3000:8080"
extra_hosts:
- "host.docker.internal:host-gateway"
environment:
OLLAMA_BASE_URL: "http://host.docker.internal:11434"
WEBUI_SECRET_KEY: "replace-with-the-generated-key"
volumes:
- open-webui:/app/backend/data
volumes:
open-webui:
Start it:
sudo docker compose up -d
sudo docker compose logs -f
When the logs show that the server is listening, open http://server-IP:3000 from a computer on the network.
| Element | Explanation |
|---|---|
ports: "3000:8080" |
The interface is published on the host's port 3000 |
extra_hosts |
Lets the container reach the host, and therefore Ollama, under the name host.docker.internal |
OLLAMA_BASE_URL |
Ollama's address as seen from the container |
open-webui volume |
Accounts, conversations and settings; back it up |
The first account
The first screen offers Create Admin Account: this first account becomes the instance's administrator. Create it immediately (for example spo), with a strong password stored in your password manager: losing this password blocks access to the settings. Sign-ups then close by themselves; the administrator creates the other users' accounts in the admin panel.
Your Ollama models appear in the selector at the top of the conversation. If the list is empty, check OLLAMA_HOST (part 5) and the firewall (part 7).
Updating Open WebUI
cd /opt/open-webui
sudo docker compose pull
sudo docker compose up -d
For Ollama, simply rerun the installation script: it updates the program without touching the models.
7. Securing the whole thing
With ufw, assuming a local network 192.168.1.0/24:
sudo ufw default deny incoming
sudo ufw allow from 192.168.1.0/24 to any port 22 proto tcp
sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp
sudo ufw allow from 172.16.0.0/12 to any port 11434 proto tcp
sudo ufw enable
The third rule only allows Docker's networks (range 172.16.0.0/12) to reach Ollama: Open WebUI can access it, the computers on the network can't. Add a rule for a specific computer if it needs to use the API directly.
Docker and ufw
Ports published by Docker (here 3000) bypass ufw rules, because Docker writes its own firewall rules. To restrict the web interface, publish it only on the local network address ("192.168.1.10:3000:8080") or behind a reverse proxy, and filter at the router.
| Measure | Why |
|---|---|
| No direct exposure to the internet: remote access through a VPN (WireGuard, Tailscale) | Neither Ollama nor an admin interface needs to be visible from outside |
| HTTPS with a reverse proxy (Caddy, Nginx Proxy Manager, Traefik) | Passwords and conversations don't travel in clear text on the network |
| Named accounts, "user" roles for the other members | Each person only sees their own conversations |
Back up the open-webui volume |
Conversations, accounts and settings |
| Regular updates of Ollama and Open WebUI | Both projects move quickly and fix vulnerabilities |
8. Going further
| Use | How |
|---|---|
| Chatting with your documents | Drag a PDF or a text file into the conversation: Open WebUI splits it and gives the relevant passages to the model (RAG) |
| Knowledge bases | In Open WebUI, Workspace > Knowledge: a set of documents that several conversations can query |
| Custom models | A model with standing instructions (for example "answer in numbered procedures") in Workspace > Models |
| Using the API in your scripts | curl http://127.0.0.1:11434/api/chat …, or Ollama's OpenAI-compatible API under /v1/ |
| Audio transcription, image generation | Open WebUI connects to other local services; each one needs yet more graphics memory |
9. If something goes wrong
| Symptom | Likely cause | Solution |
|---|---|---|
| Very slow answers | The model spills out of VRAM or runs on the processor | ollama ps; take a smaller model, reduce OLLAMA_CONTEXT_LENGTH |
nvidia-smi not found or shows no card |
NVIDIA driver missing or badly installed | Install the driver and CUDA, restart |
| No models in Open WebUI | Ollama only listens on 127.0.0.1, or the firewall is blocking |
OLLAMA_HOST=0.0.0.0:11434, ufw rule for 172.16.0.0/12; test from the container: sudo docker exec open-webui curl -s http://host.docker.internal:11434/api/version |
| The model "forgets" the start of a long document | Context too short | Increase OLLAMA_CONTEXT_LENGTH, within the limits of the VRAM |
| Administrator password lost | — | See the reset procedure in the Open WebUI documentation; hence the importance of the password manager |
Key points
- Ollama runs the models, Open WebUI provides the interface: your data stays on your network.
- VRAM decides the model size: 8 GB for a 7 to 8 billion parameter model; check for
100% GPUwithollama ps. OLLAMA_HOST=0.0.0.0:11434so that Open WebUI can reach it, but never expose port 11434: the API has no authentication.- Open WebUI: create the administrator account immediately, back up the volume, update regularly.
- Remote access through a VPN, HTTPS through a reverse proxy, and beware of ports published by Docker that bypass
ufw.