Local AI (Ollama)

Module id: local_ai · Kind: sidecar · Configured by: OLLAMA_BASE_URL · Provider: ollama

Local AI means chat and embedding models running on your hardware through Ollama — no API key, no per-token cost, no prompt leaving the machine. It is the sovereign option next to the cloud providers, and the reason the standard install is bigger than the minimal one. Opt-in through a Compose profile, because the models are large.

What it adds

  • The Ollama provider: every model you pull becomes selectable in the catalog, with a per-token cost of zero.
  • Import pulled models on the Local AI card of Operate → AI infrastructure → Models & keys — one click lists what Ollama has and adds the rows.
  • The download a local model flow in the admin UI and the automatic first-start download of the default models (bge-m3 for embeddings, gpt-oss:20b for chat when ENABLE_LOCAL_GPT_OSS is on).
  • Local embeddings for RAG and memories, so a fully offline knowledge base is possible.

Without it

Everything works with cloud providers instead; the Ollama rows are hidden and the Local AI card reports the module as not installed. Embeddings then need a cloud embedding model, which the first-run wizard sets up with the provider key you enter.

How to enable

Developer / evaluation stack

Put the profile in .env (so console commands see the URL too) and start:

# .env
COMPOSE_PROFILES=local-ai
OLLAMA_BASE_URL=http://ollama:11434

docker compose up -d
docker compose logs -f ollama          # watch the first model download

The profile exports OLLAMA_BASE_URL=http://ollama:11434 and AUTO_DOWNLOAD_MODELS=true for the app process. Without the line in .env, a docker compose exec backend php bin/console … session does not see the URL — the app works, the console does not.

Production (deploy/compose.yaml)

# deploy/.env
COMPOSE_PROFILES=local-ai          # adds Ollama and Whisper; comma-separate with office if needed
ENABLE_LOCAL_GPT_OSS=false         # true to also pull the 20B local chat model
WHISPER_DEFAULT_MODEL=tiny

Redeploy. The start wrapper sets OLLAMA_BASE_URL, AUTO_DOWNLOAD_MODELS and WHISPER_ENABLED from the profile — do not set them by hand there.

An Ollama you already run

On the host, on a GPU box, or in Kubernetes:

OLLAMA_BASE_URL=http://host.docker.internal:11434    # Ollama on the Docker host
OLLAMA_BASE_URL=http://gpu-node.internal:11434       # a shared GPU server

Restart backend + worker, then Import pulled models on the Local AI card. Pull additional models with ollama pull <name> on that machine (or docker compose exec ollama ollama pull llama3.2:3b for the bundled one) and import again.

Check it works

  • Operate → Feature Status → Local AI (Ollama) shows Available / Ollama is running. The probe is GET /api/tags. Ollama is not reachable means the URL is set but nothing answered — usually the container is still downloading its first model; OLLAMA_BASE_URL is unset means the module is off.
  • Models & keys → Local AI card lists the pulled models; the model selector in chat offers them with a Free badge.
  • Directly:
docker compose exec -T backend curl -sS http://ollama:11434/api/tags

Configuration reference

Variable Default Meaning
OLLAMA_BASE_URL empty (set by the local-ai profile to http://ollama:11434) Base URL of the Ollama API. Empty = module off.

Set by the profile, not by hand: AUTO_DOWNLOAD_MODELS, WHISPER_ENABLED. Related deployment switches: COMPOSE_PROFILES, ENABLE_LOCAL_GPT_OSS, WHISPER_DEFAULT_MODEL.

Sizing: 8 GB RAM is the floor for the minimal stack; plan 16 GB for local AI, more for the 20B chat model, and a GPU if you want interactive speed. Details and the model-download pitfall: Quickstart → Pitfall 1.

Troubleshooting

  • Everything is "downloading" for a long time. First start pulls gigabytes. docker compose logs -f ollama; the UI shows download progress on the Local AI card.
  • Console command says no local AI, the app works. OLLAMA_BASE_URL is only exported for the app process by the profile wrapper. Add it to .env as shown above.
  • Model is pulled but not offered in chat. Import it (Models & keys → Local AI → Import pulled models) and make sure it is active under System Config → AI Models.
  • Slow answers. CPU inference of a 20B model is not interactive. Use a smaller model, a GPU host, or a cloud provider for chat while keeping local embeddings.

Related