Migrating to llama.cpp

On macOS with Homebrew:

1

  brew install llama.cpp

That's it. You get llama-server, llama-cli, and the rest of the tools. Metal GPU acceleration works out of the box on Apple Silicon.

You can also grab a pre-built binary from the releases page, or build from source:

1
2
3
4

  git clone https://github.com/ggml-org/llama.cpp
  cd llama.cpp
  cmake -B build
  cmake --build build --config Release

1

  llama-cli -hf unsloth/gemma-4-26B-A4B-it-GGUF:Q4_K_M -cnv

That's the whole thing. What happens:

1. Downloads the model from Hugging Face if you don't have it
2. Loads it into memory with Metal GPU acceleration
3. Starts an interactive chat (-cnv is conversation mode)
4. Frees memory immediately when you quit (Ctrl+C)

The chat template is read from the GGUF metadata – no Modelfile, no configuration. Unlike Ollama there's no background daemon; the process runs, you chat, you quit, it's gone.

Want different parameters? Just add flags:

1
2
3
4
5

  llama-cli \
    -hf unsloth/gemma-4-26B-A4B-it-GGUF:Q8_0 \
    --ctx-size 0 \
    --temp 0.7 \
    -cnv

Compare this to Ollama where changing temperature means creating a Modelfile, running ollama create, and potentially copying 20+ GB of model data.

1

  llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:Q4_K_M --ctx-size 0

This starts an OpenAI-compatible API on http://localhost:8080. Point any tool at it – Continue, aider, Open WebUI, or just curl:

1
2
3
4
5
6
7
8

  curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "gemma-4",
      "messages": [
        {"role": "user", "content": "Explain MoE architectures in two sentences"}
      ]
    }'

Or with Python:

1
2
3
4
5
6
7
8
9

  from openai import OpenAI

  client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")

  response = client.chat.completions.create(
      model="gemma-4",
      messages=[{"role": "user", "content": "Hello!"}],
  )
  print(response.choices[0].message.content)

llama-server includes a built-in web UI. Just start the server and open http://localhost:8080 in your browser. You get a chat interface, no extra app to install.

A bare llama-server serves one model and runs until you stop it. If you want Ollama-style behavior where you can hit one endpoint with different model names and have them auto-load and auto-unload, that's what llama-swap is for.

llama-swap is a lightweight Go proxy that sits in front of llama-server. When a request comes in, it looks at the model field, starts the right llama-server, and proxies the request. When a request comes in for a different model, it stops the old one and starts the new one. After the ttl expires with no requests, the model unloads and memory is freed.

1
2

  brew tap mostlygeek/llama-swap
  brew install llama-swap

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

  models:
    gemma4:
      cmd: llama-server --port ${PORT} -hf unsloth/gemma-4-26B-A4B-it-GGUF:Q4_K_M --ctx-size 0
      ttl: 120
      aliases:
        - gemma-4-26b

    gemma4-31b:
      cmd: llama-server --port ${PORT} -hf unsloth/gemma-4-31B-it-GGUF:Q4_K_M --ctx-size 0
      ttl: 120
      aliases:
        - gemma-4-31b

    gemma4-e2b:
      cmd: llama-server --port ${PORT} -hf unsloth/gemma-4-E2B-it-GGUF:Q4_K_M --ctx-size 0
      ttl: 120

    nemotron:
      cmd: llama-server --port ${PORT} -hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:Q4_K_M --ctx-size 0
      ttl: 120
      aliases:
        - nemotron-3-nano

    nemotron-4b:
      cmd: llama-server --port ${PORT} -hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 --ctx-size 0
      ttl: 120

    qwen3:
      cmd: llama-server --port ${PORT} -hf unsloth/Qwen3-30B-A3B-GGUF:Q4_K_M --ctx-size 0
      ttl: 120
      aliases:
        - qwen3-30b

    qwen3-32b:
      cmd: llama-server --port ${PORT} -hf unsloth/Qwen3-32B-GGUF:Q4_K_M --ctx-size 0
      ttl: 120

    qwen3-coder:
      cmd: llama-server --port ${PORT} -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M --ctx-size 0
      ttl: 120
      aliases:
        - qwen-coder

1

  llama-swap --config config.yaml --listen :8080

1
2
3
4
5
6
7

  # This starts gemma4 automatically
  curl http://localhost:8080/v1/chat/completions \
    -d '{"model": "gemma4", "messages": [{"role": "user", "content": "hi"}]}'

  # This stops gemma4 and starts qwen3-coder
  curl http://localhost:8080/v1/chat/completions \
    -d '{"model": "qwen3-coder", "messages": [{"role": "user", "content": "write fizzbuzz"}]}'

• Performance: community benchmarks show llama.cpp running 1.5-1.8x faster than Ollama on the same hardware.
• New models immediately: GGUFs appear on Hugging Face within hours of a model release. With Ollama you wait for someone to package it for their registry.
• Full quantization range: Ollama only offers a handful of quant levels. On Hugging Face you get IQ2 through BF16.
• No lock-in: models are plain GGUF files shared with any tool.
• Chat templates just work: llama.cpp reads Jinja templates embedded in the GGUF. No Modelfile, no Go template translation.
• No background daemon: nothing running when you're not using it.
• No VC pivot: llama.cpp is MIT-licensed, community-driven, and now part of the Hugging Face ecosystem.

• llama.cpp on GitHub
• Friends Don't Let Friends Use Ollama – detailed history of Ollama's issues
• llama-swap – multi-model hot-swapping proxy
• Gemma 4 26B-A4B GGUF on Hugging Face (Unsloth Dynamic quants)
• Gemma 4 31B GGUF on Hugging Face