SYNAPLAN // DOCS
Launch app

09 / Feature modules

Local AI (Ollama)

On this page 11

Module id: local_ai · Kind: sidecar · Configured by: OLLAMA_BASE_URL · Provider: ollama

Local AI means chat and embedding models running on your hardware through Ollama — no API key, no per-token cost, no prompt leaving the machine. It is the sovereign option next to the cloud providers, and the reason the standard install is bigger than the minimal one. Opt-in through a Compose profile, because the models are large.

What it adds#

  • The Ollama provider: every model you pull becomes selectable in the catalog, with a per-token cost of zero.
  • Import pulled models on the Local AI card of Operate → AI infrastructure → Models & keys — one click lists what Ollama has and adds the rows.
  • The download a local model flow in the admin UI and the automatic first-start download of the default models (bge-m3 for embeddings, gpt-oss:20b for chat when ENABLE_LOCAL_GPT_OSS is on).
  • Local embeddings for RAG and memories, so a fully offline knowledge base is possible.

Without it#

Everything works with cloud providers instead; the Ollama rows are hidden and the Local AI card reports the module as not installed. Embeddings then need a cloud embedding model, which the first-run wizard sets up with the provider key you enter.

How to enable#

Developer / evaluation stack#

Put the profile in .env (so console commands see the URL too) and start:

shell
# .env
COMPOSE_PROFILES=local-ai
OLLAMA_BASE_URL=http://ollama:11434
shell
make up
docker compose logs -f ollama          # watch the first model download

The profile exports OLLAMA_BASE_URL=http://ollama:11434 and AUTO_DOWNLOAD_MODELS=true for the app process. Without the line in .env, a docker compose exec backend php bin/console … session does not see the URL — the app works, the console does not.

Production (deploy/compose.yaml)#

shell
# deploy/.env
COMPOSE_PROFILES=local-ai          # adds Ollama and Whisper; comma-separate with office if needed
ENABLE_LOCAL_GPT_OSS=false         # true to also pull the 20B local chat model
WHISPER_DEFAULT_MODEL=tiny

Redeploy. The start wrapper sets OLLAMA_BASE_URL, AUTO_DOWNLOAD_MODELS and WHISPER_ENABLED from the profile — do not set them by hand there.

An Ollama you already run#

On the host, on a GPU box, or in Kubernetes:

shell
OLLAMA_BASE_URL=http://host.docker.internal:11434    # Ollama on the Docker host
OLLAMA_BASE_URL=http://gpu-node.internal:11434       # a shared GPU server

Restart backend + worker, then Import pulled models on the Local AI card. Pull additional models with ollama pull <name> on that machine (or docker compose exec ollama ollama pull llama3.2:3b for the bundled one) and import again.

When you start COMPOSE_PROFILES=local-ai ENABLE_LOCAL_GPT_OSS=true make up (or pull models onto your own Ollama host), these five models cover embeddings, laptop chat, full local mixes and GPU servers. Together they power the Europe / Local mix in Quick model setup:

Model (ollama pull …) Download size Recommended RAM / VRAM Role & what it unlocks
bge-m3 ~1.2 GB 4 GB RAM Embeddings (VECTORIZE). Pulled automatically with COMPOSE_PROFILES=local-ai. Unlocks local RAG document search and memories (1024 dimensions — identical vector space to the Synaplan Demo and Cloudflare bge-m3).
llama3.2:3b ~2.0 GB 8 GB RAM Compact chat for 8 GB laptops. Fast CPU/Apple-Silicon inference when 16 GB RAM is not available. Pull with docker compose exec ollama ollama pull llama3.2:3b and click Import pulled models. See Air-gapped installation for offline pre-loading.
gpt-oss:20b ~14 GB 16 GB RAM (or 16 GB VRAM) Default local chat & Europe mix. Pulled automatically when ENABLE_LOCAL_GPT_OSS=true. Unlocks private chat, sorting, document analysis and tool use on a single workstation or Apple Silicon Mac.
qwen3.5:35b ~22 GB 24–32 GB RAM / VRAM Multilingual chat, coding & vision. Strong European-language and agentic reasoning option in the local catalog.
gpt-oss:120b ~65 GB GPU server or ≥ 80 GB unified RAM Frontier open-weight reasoning. Full 120B MoE model for shared local GPU boxes (same model family that answers in the Synaplan Demo).

When Ollama finishes downloading a chat model while you are using the Synaplan Demo, the readiness toast offers a one-click Switch to your local models action — it never switches away from the Demo without your click.

Check it works#

  • Operate → Feature Status → Local AI (Ollama) shows Available / Ollama is running. The probe is GET /api/tags. Ollama is not reachable means the URL is set but nothing answered — usually the container is still downloading its first model; OLLAMA_BASE_URL is unset means the module is off.
  • Models & keys → Local AI card lists the pulled models; the model selector in chat offers them with a Free badge.
  • Directly:
shell
docker compose exec -T backend curl -sS http://ollama:11434/api/tags

Configuration reference#

Variable Default Meaning
OLLAMA_BASE_URL empty (set by the local-ai profile to http://ollama:11434) Base URL of the Ollama API. Empty = module off.

Set by the profile, not by hand: AUTO_DOWNLOAD_MODELS, WHISPER_ENABLED. Related deployment switches: COMPOSE_PROFILES, ENABLE_LOCAL_GPT_OSS, WHISPER_DEFAULT_MODEL.

Sizing: 8 GB RAM is the floor for the minimal stack; plan 16 GB for local AI, more for the 20B chat model, and a GPU if you want interactive speed. Details and the model-download pitfall: Quickstart → Pitfall 1.

Troubleshooting#

  • Everything is "downloading" for a long time. First start pulls gigabytes. docker compose logs -f ollama; the UI shows download progress on the Local AI card.
  • Console command says no local AI, the app works. OLLAMA_BASE_URL is only exported for the app process by the profile wrapper. Add it to .env as shown above.
  • Model is pulled but not offered in chat. Import it (Models & keys → Local AI → Import pulled models) and make sure it is active under Manage → Assistants → Models.
  • Slow answers. CPU inference of a 20B model is not interactive. Use a smaller model, a GPU host, or a cloud provider for chat while keeping local embeddings.
↑↓ navigate↵ openEsc close