AI / Tutorial

A local AI on your homelab with Ollama and Open WebUI

Running open AI models at home, without data leaving the network: choosing the hardware and the model, installing Ollama on Debian, opening it carefully to the local network, adding the Open WebUI web interface with Docker Compose, and securing the whole thing.

AI Tutorial Intermediate Published October 20, 2026 Translated November 20, 2026 45 min read Version française
An electronic circuit topped with a stylised brain, in blue
Photo: Steve A Johnson on Unsplash

Before you start

In plain words

Assistants like Claude run in their publisher's data centres. With a local AI, it's the opposite: an open model runs on your machine, and your questions, documents and answers never leave your network. Ollama is the engine: it downloads the models and runs them. Open WebUI is the interface: a web chat page, similar to the big assistants', reachable from every computer in the house. The result is less powerful than the best online models, but private, free to use and available even without the internet.

For the notions of model, token and quantisation, see Generative AI explained simply.

1. The architecture

Component Role Port
Ollama Service that loads models into memory and answers requests (API) 11434
Open WebUI Web chat interface, user accounts, history, document upload 3000 (to 8080 in the container)
Your computers A browser, on the local network or through a VPN —

Ollama is installed directly on the system, for simple access to the graphics card; Open WebUI runs in a Docker container.

2. Hardware and choice of model

The limiting factor is the graphics card memory (VRAM). Without a graphics card, Ollama runs on the processor and RAM, but much more slowly: usable for small models, painful beyond that.

Hardware Models usable comfortably (quantised to 4 bits)
Processor only, 16 GB of memory 1 to 4 billion parameters, slow
8 GB graphics card 7 to 8 billion parameters
12 to 16 GB graphics card Up to 14 billion, or 20 billion depending on the model
24 GB graphics card Up to 30 to 32 billion
Apple silicon Mac Depends on unified memory: about two thirds usable by the model

NVIDIA cards are the best supported (CUDA). Recent AMD cards work with ROCm, with a more restricted compatibility list.

Which model?

The Ollama library lists the available models, with their sizes. A few well-known open families:

Family Publisher Strengths
Mistral, Ministral Mistral AI (France) Good at French and other European languages, several sizes
Qwen Alibaba Very versatile, good at code, many sizes
Gemma Google Compact and efficient, some versions read images
Llama Meta Very widespread, large ecosystem
gpt-oss OpenAI Reasoning-oriented open-weights models
DeepSeek-R1 (distilled versions) DeepSeek Detailed reasoning, slower

Start with a 7 to 8 billion parameter model if you have 8 GB of VRAM, then compare on your own uses: general rankings don't replace a test on your questions.

Proxmox VE: giving the graphics card to a VM

On a hypervisor, there are two approaches: a VM given the graphics card through passthrough (the card is then no longer usable by the host or other VMs), or an LXC container that shares the host's card. Passthrough requires IOMMU (VT-d or AMD-Vi) turned on in the BIOS. See the Proxmox documentation at the bottom of the page and Installing Proxmox VE.

3. Installing Ollama

On an up-to-date Debian 13, with an account that has sudo:

curl -fsSL https://ollama.com/install.sh | sh

The script installs Ollama, creates the ollama system user and the ollama systemd service. For an NVIDIA card, first install the proprietary driver and CUDA (see the Ollama documentation), then check that the card is seen:

nvidia-smi

Check the service:

systemctl status ollama
curl http://127.0.0.1:11434/api/version
{"version":"0.x.y"}

Read a script before running it

curl … | sh runs a downloaded script with the user's rights, then sudo. It's the official method, but you can download it and read it first: curl -fsSL https://ollama.com/install.sh -o install.sh, then less install.sh.

4. First steps with Ollama

Download a model and chat with it (the name and size are chosen in the library):

ollama run mistral

Type your question, then /bye to quit. The useful commands:

Command Effect
ollama pull name Downloads a model without running it
ollama run name Starts a chat (downloads the model if needed)
ollama list Lists the downloaded models and their size
ollama ps Shows the models loaded in memory, and whether they're running on the graphics card
ollama show name Details of a model: size, quantisation, context length
ollama rm name Deletes a model

Check that the model is really using the graphics card:

ollama ps
NAME              ID              SIZE      PROCESSOR    CONTEXT    UNTIL
mistral:latest    6577803aa9a0    5.7 GB    100% GPU     4096       4 minutes from now

100% GPU: the model fits entirely in VRAM. A split value (48%/52% CPU/GPU) means it's spilling over into RAM, with markedly slower answers: choose a smaller model or a stronger quantisation.

The API can be tested directly:

curl http://127.0.0.1:11434/api/generate -d '{"model": "mistral", "prompt": "Explain RAID 1 in one sentence.", "stream": false}'

5. Tuning Ollama

Settings go through the service's environment variables. Open an override for the service:

sudo systemctl edit ollama

Add in the area provided:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KEEP_ALIVE=15m"

Then reload:

sudo systemctl daemon-reload
sudo systemctl restart ollama
Variable Default value Role
OLLAMA_HOST 127.0.0.1:11434 Listening address; 0.0.0.0 opens it on all interfaces, needed for Open WebUI in Docker to reach it
OLLAMA_CONTEXT_LENGTH 4096 tokens Context length: the larger it is, the more the model "remembers" of the conversation and documents, but the more memory it uses
OLLAMA_KEEP_ALIVE 5m How long a model stays loaded after the last request
OLLAMA_MODELS /usr/share/ollama/.ollama/models Models folder; handy for putting them on a larger disk (the ollama user must be able to write to it)

Ollama's API has no authentication

Anyone who can reach port 11434 can use your models, download others or delete them. Thousands of Ollama servers mistakenly exposed on the internet have been found. Never open this port to the internet, and restrict it to the bare minimum with the firewall (part 7).

6. Installing Open WebUI with Docker Compose

Create a folder for the service:

sudo mkdir -p /opt/open-webui
cd /opt/open-webui

Generate a secret key for sessions:

openssl rand -hex 32

Create the file /opt/open-webui/compose.yaml, replacing the value of WEBUI_SECRET_KEY with the generated key:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: always
    ports:
      - "3000:8080"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
      WEBUI_SECRET_KEY: "replace-with-the-generated-key"
    volumes:
      - open-webui:/app/backend/data

volumes:
  open-webui:

Start it:

sudo docker compose up -d
sudo docker compose logs -f

When the logs show that the server is listening, open http://server-IP:3000 from a computer on the network.

Element Explanation
ports: "3000:8080" The interface is published on the host's port 3000
extra_hosts Lets the container reach the host, and therefore Ollama, under the name host.docker.internal
OLLAMA_BASE_URL Ollama's address as seen from the container
open-webui volume Accounts, conversations and settings; back it up

The first account

The first screen offers Create Admin Account: this first account becomes the instance's administrator. Create it immediately (for example spo), with a strong password stored in your password manager: losing this password blocks access to the settings. Sign-ups then close by themselves; the administrator creates the other users' accounts in the admin panel.

Your Ollama models appear in the selector at the top of the conversation. If the list is empty, check OLLAMA_HOST (part 5) and the firewall (part 7).

Updating Open WebUI

cd /opt/open-webui
sudo docker compose pull
sudo docker compose up -d

For Ollama, simply rerun the installation script: it updates the program without touching the models.

7. Securing the whole thing

With ufw, assuming a local network 192.168.1.0/24:

sudo ufw default deny incoming
sudo ufw allow from 192.168.1.0/24 to any port 22 proto tcp
sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp
sudo ufw allow from 172.16.0.0/12 to any port 11434 proto tcp
sudo ufw enable

The third rule only allows Docker's networks (range 172.16.0.0/12) to reach Ollama: Open WebUI can access it, the computers on the network can't. Add a rule for a specific computer if it needs to use the API directly.

Docker and ufw

Ports published by Docker (here 3000) bypass ufw rules, because Docker writes its own firewall rules. To restrict the web interface, publish it only on the local network address ("192.168.1.10:3000:8080") or behind a reverse proxy, and filter at the router.

Measure Why
No direct exposure to the internet: remote access through a VPN (WireGuard, Tailscale) Neither Ollama nor an admin interface needs to be visible from outside
HTTPS with a reverse proxy (Caddy, Nginx Proxy Manager, Traefik) Passwords and conversations don't travel in clear text on the network
Named accounts, "user" roles for the other members Each person only sees their own conversations
Back up the open-webui volume Conversations, accounts and settings
Regular updates of Ollama and Open WebUI Both projects move quickly and fix vulnerabilities

8. Going further

Use How
Chatting with your documents Drag a PDF or a text file into the conversation: Open WebUI splits it and gives the relevant passages to the model (RAG)
Knowledge bases In Open WebUI, Workspace > Knowledge: a set of documents that several conversations can query
Custom models A model with standing instructions (for example "answer in numbered procedures") in Workspace > Models
Using the API in your scripts curl http://127.0.0.1:11434/api/chat …, or Ollama's OpenAI-compatible API under /v1/
Audio transcription, image generation Open WebUI connects to other local services; each one needs yet more graphics memory

9. If something goes wrong

Symptom Likely cause Solution
Very slow answers The model spills out of VRAM or runs on the processor ollama ps; take a smaller model, reduce OLLAMA_CONTEXT_LENGTH
nvidia-smi not found or shows no card NVIDIA driver missing or badly installed Install the driver and CUDA, restart
No models in Open WebUI Ollama only listens on 127.0.0.1, or the firewall is blocking OLLAMA_HOST=0.0.0.0:11434, ufw rule for 172.16.0.0/12; test from the container: sudo docker exec open-webui curl -s http://host.docker.internal:11434/api/version
The model "forgets" the start of a long document Context too short Increase OLLAMA_CONTEXT_LENGTH, within the limits of the VRAM
Administrator password lost — See the reset procedure in the Open WebUI documentation; hence the importance of the password manager

Key points

  • Ollama runs the models, Open WebUI provides the interface: your data stays on your network.
  • VRAM decides the model size: 8 GB for a 7 to 8 billion parameter model; check for 100% GPU with ollama ps.
  • OLLAMA_HOST=0.0.0.0:11434 so that Open WebUI can reach it, but never expose port 11434: the API has no authentication.
  • Open WebUI: create the administrator account immediately, back up the volume, update regularly.
  • Remote access through a VPN, HTTPS through a reverse proxy, and beware of ports published by Docker that bypass ufw.

Read next