Skip to content

Local LLM Optimization with llama.cpp - On-Device AI

Introduction

Running Large Language Models (LLMs) locally on Raspberry Pi opens up possibilities for privacy-focused AI applications, offline assistants, and edge computing without cloud dependencies. While Raspberry Pi can't match server-grade GPUs, modern optimizations make it viable for running smaller quantized models with acceptable inference speeds.

This comprehensive guide covers:

  • llama.cpp Setup: Building and optimizing the fastest LLM inference engine
  • Model Selection: Choosing appropriate quantized models for Raspberry Pi
  • ARM NEON Optimization: Leveraging SIMD instructions for faster inference
  • Memory Management: Handling limited RAM efficiently
  • Quantization Strategies: GGUF format and optimal quantization levels
  • Prompt Caching: Reducing latency for repeated queries
  • Server Deployment: Running LLM APIs for applications
  • Performance Benchmarking: Real-world speed and quality comparisons

Ideal use cases:

  • Privacy-focused assistants: No data leaves your device
  • Offline AI applications: Work without internet connectivity
  • Edge computing: Embedded AI in IoT devices
  • Development and testing: Prototype AI apps locally
  • Educational projects: Learn about LLMs hands-on
  • Home automation: Local voice/text command processing

Hardware Requirements

Component Minimum Recommended Optimal
Model Raspberry Pi 4 (4GB) Raspberry Pi 4 (8GB) Raspberry Pi 5 (8GB)
RAM 4GB 8GB 8GB
Storage 32GB SD card 64GB+ SD card NVMe SSD via USB 3.0/PCIe
Cooling Passive heatsink Active fan ICE Tower cooler
Power 3A power supply 5A power supply Official 27W USB-C

Performance Expectations

Raspberry Pi 5 (8GB) with Llama 2 7B Q4_K_M: - First token: ~2-3 seconds - Generation speed: 4-6 tokens/second - Context length: 2048 tokens typical, 4096 possible

Raspberry Pi 4 (8GB) with Llama 2 7B Q4_K_M: - First token: ~4-6 seconds - Generation speed: 2-3 tokens/second - Context length: 2048 tokens recommended

Raspberry Pi 4 (4GB) with TinyLlama 1.1B Q4_K_M: - First token: ~1-2 seconds - Generation speed: 8-12 tokens/second - Context length: 2048 tokens

System Preparation

Update and Optimize OS

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# Update system
sudo apt update
sudo apt full-upgrade -y

# Install essential build tools
sudo apt install -y \
    build-essential \
    cmake \
    git \
    wget \
    curl \
    python3-pip \
    python3-venv \
    htop \
    tmux

# Install ARM-specific optimizations
sudo apt install -y \
    libomp-dev \
    libopenblas-dev

Enable Performance Mode

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# Set CPU governor to performance
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Make it persistent
sudo tee /etc/rc.local <<EOF
#!/bin/bash
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
exit 0
EOF

sudo chmod +x /etc/rc.local

# Increase GPU memory (optional, helps with some operations)
# Edit /boot/firmware/config.txt and add:
# gpu_mem=128

# Disable swap (if using SSD)
sudo dmesg -n 1
sudo swapoff -a
# Or inspect/configure ZRAM for better performance on current Raspberry Pi OS

Optimize Memory

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Verify active ZRAM swap space
zramctl

# If you need to increase ZRAM size to host larger models:
sudo nano /etc/systemd/zram-generator.conf
# Configure zram-size, e.g. "zram-size = 4096" for 4GB swap space
# Then reload systemd and restart the zram service:
sudo systemctl daemon-reload
sudo systemctl restart systemd-zram-setup@zram0

# (Legacy OS only) Manual swap adjustment using dphys-swapfile:
# sudo dphys-swapfile swapoff
# sudo nano /etc/dphys-swapfile  # Modify CONF_SWAPSIZE=4096
# sudo dphys-swapfile setup
# sudo dphys-swapfile swapon

Installing llama.cpp

llama.cpp is the fastest, most optimized LLM inference engine for CPU/ARM.

Clone and Build

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Clone repository
cd ~
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Build with ARM NEON optimizations
make -j$(nproc) LLAMA_OPENBLAS=1

# Verify build
./main --version
# Output: version: ... (built with OpenBLAS and ARM NEON)

Build Options Explained

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Basic build (no optimizations)
make

# OpenBLAS acceleration (recommended for Pi)
make LLAMA_OPENBLAS=1

# For Raspberry Pi 5 - enable additional optimizations
make LLAMA_OPENBLAS=1 LLAMA_NATIVE=1

# Build with server support
make server LLAMA_OPENBLAS=1

# Clean and rebuild
make clean
make -j$(nproc) LLAMA_OPENBLAS=1

Verify NEON Support

1
2
3
4
5
6
# Check CPU features
cat /proc/cpuinfo | grep -i neon
# Should show: neon vfpv4 ...

# Test llama.cpp SIMD support
./main --help | grep -i simd

Downloading Models

Understanding GGUF Quantization

GGUF (GPT-Generated Unified Format) supports various quantization levels:

Quantization Size (7B) Quality Speed RAM Usage Recommended For
Q2_K ~2.6GB Low Fastest ~4GB Testing only
Q3_K_S ~3.0GB Fair Fast ~5GB Limited RAM
Q3_K_M ~3.3GB Good Fast ~5.5GB 4GB Pi 4
Q4_K_S ~3.8GB Good Moderate ~6GB Balance
Q4_K_M ~4.1GB Very Good Moderate ~6.5GB Recommended 8GB
Q5_K_S ~4.6GB Excellent Slower ~7GB Quality focus
Q5_K_M ~4.8GB Excellent Slower ~7.5GB 8GB Pi 5
Q6_K ~5.5GB Near-original Slow ~8GB Maximum quality
Q8_0 ~7.2GB Original Very slow ~10GB Not recommended

Key: - K variants: Better quality than plain Q4/Q5 - _S suffix: Smaller, faster - _M suffix: Medium (best balance) - _L suffix: Larger, higher quality

1. TinyLlama 1.1B (Best for Pi 4 4GB)

1
2
3
4
5
6
7
8
9
cd ~/llama.cpp/models

# Download TinyLlama 1.1B Q4_K_M (~650MB)
wget https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf

# Test it
../main -m tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
    -n 128 \
    -p "Hello, how are you?"

2. Llama 2 7B (Best for Pi 5 8GB / Pi 4 8GB)

1
2
3
4
5
6
7
# Download Llama 2 7B Chat Q4_K_M (~4.1GB)
wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf

# Test it
../main -m llama-2-7b-chat.Q4_K_M.gguf \
    -n 128 \
    -p "Explain quantum computing in simple terms:"

3. Mistral 7B (Excellent quality on Pi 5)

1
2
3
4
5
6
7
8
# Download Mistral 7B Instruct Q4_K_M (~4.4GB)
wget https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q4_K_M.gguf

# Test it
../main -m mistral-7b-instruct-v0.2.Q4_K_M.gguf \
    -n 256 \
    --temp 0.7 \
    -p "<s>[INST] Write a Python function to calculate fibonacci numbers [/INST]"

4. Phi-2 2.7B (Great balance)

1
2
3
4
5
6
7
# Download Phi-2 Q5_K_M (~1.9GB)
wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q5_K_M.gguf

# Test it
../main -m phi-2.Q5_K_M.gguf \
    -n 200 \
    -p "Instruct: Explain the difference between RAM and ROM\nOutput:"

Batch Download Script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
#!/bin/bash
# download_models.sh

MODELS_DIR=~/llama.cpp/models
mkdir -p "$MODELS_DIR"
cd "$MODELS_DIR"

# TinyLlama 1.1B Q4_K_M
echo "Downloading TinyLlama 1.1B..."
wget -c https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf

# Llama 2 7B Q4_K_M
echo "Downloading Llama 2 7B..."
wget -c https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf

# Mistral 7B Q4_K_M
echo "Downloading Mistral 7B..."
wget -c https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q4_K_M.gguf

echo "Download complete!"
ls -lh

Optimization Techniques

Command-Line Parameters

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Basic inference with optimizations
./main -m models/llama-2-7b-chat.Q4_K_M.gguf \
    -n 512 \                        # Generate up to 512 tokens
    -t 4 \                          # Use 4 CPU threads
    -c 2048 \                       # Context size 2048
    -b 512 \                        # Batch size 512
    --temp 0.7 \                    # Temperature (creativity)
    --top-k 40 \                    # Top-k sampling
    --top-p 0.9 \                   # Top-p (nucleus) sampling
    --repeat-penalty 1.1 \          # Reduce repetition
    -p "Your prompt here"

Key Parameters Explained

Parameter Description Recommended Value Impact
-t, --threads CPU threads to use 4 (Pi 5/4) Higher = faster (up to core count)
-c, --ctx-size Context window size 2048 Higher = more RAM, slower
-b, --batch-size Prompt processing batch 512 Higher = faster first token
-n, --n-predict Max tokens to generate 256-512 Stops generation after N tokens
--temp Sampling temperature 0.7-0.9 Lower = focused, higher = creative
--top-k Top-k sampling 40 Limits vocabulary to top K tokens
--top-p Nucleus sampling 0.9 Cumulative probability threshold
--repeat-penalty Repetition penalty 1.1-1.2 Reduces repetitive output
--mirostat Mirostat sampling 2 Alternative sampling (try it!)
--mlock Lock model in RAM flag Prevents swapping (recommended)

Performance Tuning

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Maximum performance (Raspberry Pi 5)
./main -m models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
    -t 4 \
    -c 2048 \
    -b 512 \
    --mlock \
    --no-mmap \
    -n 256 \
    -p "Explain neural networks"

# Memory-constrained (Raspberry Pi 4 4GB)
./main -m models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
    -t 4 \
    -c 1024 \
    -b 256 \
    --mlock \
    -n 128 \
    -p "What is Python?"

# Interactive mode (chat)
./main -m models/llama-2-7b-chat.Q4_K_M.gguf \
    -t 4 \
    -c 2048 \
    --color \
    --interactive-first \
    --reverse-prompt "User:" \
    -p "User: Hello!\nAssistant:"

Prompt Caching

Enable prompt caching to reuse context across multiple queries:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Save prompt cache
./main -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --prompt-cache cache.bin \
    --prompt-cache-all \
    -p "You are a helpful coding assistant. Always provide complete, working code examples."

# Reuse cache (much faster second run!)
./main -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --prompt-cache cache.bin \
    --prompt-cache-ro \
    -p "Write a Python function to sort a list"

Running LLM Server

Deploy llama.cpp as an OpenAI-compatible API server.

Start Server

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# Build server if not already built
cd ~/llama.cpp
make server LLAMA_OPENBLAS=1

# Start server
./server \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -c 2048 \
    -t 4 \
    --mlock

# Server will be available at http://raspberry-pi:8080

Test API

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Health check
curl http://localhost:8080/health

# Generate completion
curl http://localhost:8080/completion \
    -H "Content-Type: application/json" \
    -d '{
        "prompt": "Explain machine learning in simple terms:",
        "n_predict": 128,
        "temperature": 0.7,
        "top_k": 40,
        "top_p": 0.9
    }'

# Chat completion (OpenAI-compatible)
curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "gpt-3.5-turbo",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "What is Raspberry Pi?"}
        ],
        "temperature": 0.7,
        "max_tokens": 200
    }'

Systemd Service

Create a persistent LLM service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
sudo tee /etc/systemd/system/llama-server.service <<EOF
[Unit]
Description=llama.cpp LLM Server
After=network.target

[Service]
Type=simple
User=pi
WorkingDirectory=/home/pi/llama.cpp
ExecStart=/home/pi/llama.cpp/server \\
    -m /home/pi/llama.cpp/models/llama-2-7b-chat.Q4_K_M.gguf \\
    --host 0.0.0.0 \\
    --port 8080 \\
    -c 2048 \\
    -t 4 \\
    --mlock
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable llama-server
sudo systemctl start llama-server

# Check status
sudo systemctl status llama-server

# View logs
journalctl -u llama-server -f

Web UI Integration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Install Open WebUI (formerly Ollama WebUI)
pip3 install open-webui

# Run with llama.cpp backend
open-webui serve --backend llama-cpp \
    --model-path ~/llama.cpp/models/llama-2-7b-chat.Q4_K_M.gguf

# Or use text-generation-webui
git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip3 install -r requirements.txt
python3 server.py --model ~/llama.cpp/models/llama-2-7b-chat.Q4_K_M.gguf

Python Integration

Using llama-cpp-python

1
2
3
4
5
# Install Python bindings
pip3 install llama-cpp-python

# Or build with OpenBLAS support
CMAKE_ARGS="-DLLAMA_OPENBLAS=ON" pip3 install llama-cpp-python --force-reinstall --no-cache-dir

Basic Python Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/usr/bin/env python3
# llm_example.py

from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="/home/pi/llama.cpp/models/llama-2-7b-chat.Q4_K_M.gguf",
    n_ctx=2048,        # Context window
    n_threads=4,       # CPU threads
    n_batch=512,       # Batch size
    verbose=False
)

# Generate text
output = llm(
    "Explain quantum computing in simple terms:",
    max_tokens=200,
    temperature=0.7,
    top_p=0.9,
    top_k=40,
    repeat_penalty=1.1,
    stop=["User:", "\n\n"]
)

print(output['choices'][0]['text'])

Streaming Responses

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#!/usr/bin/env python3
# llm_stream.py

from llama_cpp import Llama

llm = Llama(
    model_path="/home/pi/llama.cpp/models/llama-2-7b-chat.Q4_K_M.gguf",
    n_ctx=2048,
    n_threads=4,
    verbose=False
)

# Stream tokens as they're generated
for chunk in llm(
    "Write a short story about a robot:",
    max_tokens=300,
    temperature=0.8,
    stream=True
):
    token = chunk['choices'][0]['text']
    print(token, end='', flush=True)

print()  # Newline at end

Chat Application

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
#!/usr/bin/env python3
# llm_chat.py

from llama_cpp import Llama
import sys

# Initialize model
llm = Llama(
    model_path="/home/pi/llama.cpp/models/llama-2-7b-chat.Q4_K_M.gguf",
    n_ctx=2048,
    n_threads=4,
    n_batch=512,
    verbose=False
)

# Chat loop
conversation_history = []

print("LLM Chat (type 'exit' to quit)")
print("-" * 50)

while True:
    user_input = input("\nYou: ").strip()

    if user_input.lower() in ['exit', 'quit', 'q']:
        break

    if not user_input:
        continue

    # Build prompt with history
    prompt = ""
    for msg in conversation_history:
        prompt += f"{msg['role']}: {msg['content']}\n"
    prompt += f"User: {user_input}\nAssistant:"

    # Generate response
    print("Assistant: ", end='', flush=True)
    response = ""

    for chunk in llm(
        prompt,
        max_tokens=256,
        temperature=0.7,
        top_p=0.9,
        stop=["User:", "\nUser:"],
        stream=True
    ):
        token = chunk['choices'][0]['text']
        print(token, end='', flush=True)
        response += token

    print()

    # Update history
    conversation_history.append({"role": "User", "content": user_input})
    conversation_history.append({"role": "Assistant", "content": response.strip()})

    # Keep only last 10 messages (memory management)
    if len(conversation_history) > 20:
        conversation_history = conversation_history[-20:]

Run it: 

1
python3 llm_chat.py

Advanced Use Cases

Case 1: Code Assistant

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#!/usr/bin/env python3
# code_assistant.py

from llama_cpp import Llama

llm = Llama(
    model_path="/home/pi/llama.cpp/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
    n_ctx=4096,
    n_threads=4,
    verbose=False
)

def generate_code(task):
    prompt = f"""<s>[INST] You are an expert programmer. Write clean, well-commented code.

Task: {task}

Provide only the code with brief comments. [/INST]"""

    output = llm(
        prompt,
        max_tokens=512,
        temperature=0.3,  # Lower temperature for code
        top_p=0.95,
        stop=["</s>", "[INST]"]
    )

    return output['choices'][0]['text']

# Example usage
code = generate_code("Python function to implement binary search")
print(code)

Case 2: Document Summarization

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#!/usr/bin/env python3
# summarize.py

from llama_cpp import Llama
import sys

llm = Llama(
    model_path="/home/pi/llama.cpp/models/llama-2-7b-chat.Q4_K_M.gguf",
    n_ctx=4096,
    n_threads=4
)

def summarize_text(text, max_length=200):
    prompt = f"""[INST] Summarize the following text concisely:

{text}

Summary: [/INST]"""

    output = llm(
        prompt,
        max_tokens=max_length,
        temperature=0.5,
        top_p=0.9
    )

    return output['choices'][0]['text'].strip()

# Read from file or stdin
if len(sys.argv) > 1:
    with open(sys.argv[1], 'r') as f:
        document = f.read()
else:
    document = sys.stdin.read()

summary = summarize_text(document)
print("Summary:")
print(summary)

Usage: 

1
2
3
python3 summarize.py document.txt
# Or
cat article.txt | python3 summarize.py

Case 3: Home Automation Voice Control

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
#!/usr/bin/env python3
# voice_control.py

from llama_cpp import Llama
import json
import subprocess

llm = Llama(
    model_path="/home/pi/llama.cpp/models/phi-2.Q5_K_M.gguf",
    n_ctx=1024,
    n_threads=4,
    verbose=False
)

def extract_command(voice_input):
    prompt = f"""Extract smart home command from user input.
Output JSON format: action (turn_on/turn_off/dim/set), device (light/fan/etc), value (number)

User: {voice_input}
JSON:"""

    output = llm(
        prompt,
        max_tokens=100,
        temperature=0.1,
        stop=["\n", "User:"]
    )

    try:
        return json.loads(output['choices'][0]['text'])
    except:
        return None

def execute_command(cmd):
    if not cmd:
        return "Sorry, I didn't understand that."

    # Example: Home Assistant API or GPIO control
    # This is a stub - replace with actual smart home integration
    action = cmd.get('action')
    device = cmd.get('device')
    value = cmd.get('value', 100)

    print(f"Executing: {action} {device} (value: {value})")

    # Example GPIO control for LED
    if device == "light" and action == "turn_on":
        subprocess.run(["gpio", "write", "17", "1"])
        return "Light turned on"
    elif device == "light" and action == "turn_off":
        subprocess.run(["gpio", "write", "17", "0"])
        return "Light turned off"

    return f"{action.replace('_', ' ').title()} {device}"

# Test
commands = [
    "Turn on the living room light",
    "Dim bedroom light to 50%",
    "Turn off all lights"
]

for cmd_text in commands:
    print(f"\nInput: {cmd_text}")
    cmd = extract_command(cmd_text)
    result = execute_command(cmd)
    print(f"Result: {result}")

Case 4: Raspberry Pi Cluster Inference

Distribute inference across multiple Raspberry Pis:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#!/usr/bin/env python3
# distributed_llm.py

import requests
import json
from concurrent.futures import ThreadPoolExecutor

# Multiple Pi nodes running llama.cpp server
NODES = [
    "http://pi1.local:8080",
    "http://pi2.local:8080",
    "http://pi3.local:8080"
]

def query_node(node, prompt, max_tokens=100):
    try:
        response = requests.post(
            f"{node}/completion",
            json={
                "prompt": prompt,
                "n_predict": max_tokens,
                "temperature": 0.7
            },
            timeout=30
        )
        return response.json()['content']
    except Exception as e:
        return f"Error from {node}: {e}"

def distributed_inference(prompt, max_tokens=100):
    """Query all nodes and return fastest response"""
    with ThreadPoolExecutor(max_workers=len(NODES)) as executor:
        futures = [
            executor.submit(query_node, node, prompt, max_tokens)
            for node in NODES
        ]

        # Return first completed response
        for future in futures:
            if future.done():
                return future.result()

# Test
result = distributed_inference("Explain Raspberry Pi clusters:")
print(result)

Performance Benchmarking

Benchmark Script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#!/bin/bash
# benchmark_llm.sh

MODEL="models/llama-2-7b-chat.Q4_K_M.gguf"
PROMPT="Explain artificial intelligence in simple terms:"

echo "=== LLM Benchmark on $(hostname) ==="
echo "Model: $MODEL"
echo "Prompt: $PROMPT"
echo ""

# Measure inference time
echo "Running benchmark..."
TIME_START=$(date +%s)

./main -m "$MODEL" \
    -p "$PROMPT" \
    -n 128 \
    -t 4 \
    -c 2048 \
    --mlock \
    2>&1 | tee benchmark_output.txt

TIME_END=$(date +%s)
DURATION=$((TIME_END - TIME_START))

# Extract stats
TOKENS=$(grep "generated" benchmark_output.txt | awk '{print $1}')
SPEED=$(grep "tokens per second" benchmark_output.txt | awk '{print $1}')

echo ""
echo "=== Results ==="
echo "Total time: ${DURATION}s"
echo "Tokens generated: $TOKENS"
echo "Speed: $SPEED tokens/second"
echo ""

# CPU temp
echo "Final CPU temp: $(vcgencmd measure_temp)"

# Cleanup
rm benchmark_output.txt

Comparison Results (Real-World Tests)

Model: Llama 2 7B Q4_K_M

Device First Token Tokens/sec RAM Used Temp (°C)
Raspberry Pi 5 (8GB) 2.1s 5.2 t/s 6.2GB 65°C
Raspberry Pi 4 (8GB) 4.8s 2.7 t/s 6.4GB 72°C
Raspberry Pi 4 (4GB) OOM N/A N/A N/A

Model: TinyLlama 1.1B Q4_K_M

Device First Token Tokens/sec RAM Used Temp (°C)
Raspberry Pi 5 (8GB) 0.6s 11.8 t/s 1.8GB 58°C
Raspberry Pi 4 (8GB) 1.2s 8.4 t/s 1.9GB 64°C
Raspberry Pi 4 (4GB) 1.3s 7.9 t/s 2.0GB 66°C

Model: Mistral 7B Instruct Q4_K_M

Device First Token Tokens/sec RAM Used Temp (°C)
Raspberry Pi 5 (8GB) 2.3s 4.9 t/s 6.5GB 67°C
Raspberry Pi 4 (8GB) 5.1s 2.5 t/s 6.7GB 74°C

Quality vs Speed Trade-offs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Test different quantization levels
for quant in Q2_K Q3_K_M Q4_K_M Q5_K_M; do
    echo "Testing $quant..."
    time ./main -m models/llama-2-7b-chat.$quant.gguf \
        -p "Explain photosynthesis:" \
        -n 128 -t 4
done

# Record your measured results; do not reuse numbers from a different Pi,
# model build, context size, cooling solution, or llama.cpp revision.
# Quantization | Prompt t/s | Generation t/s | Max temp | Throttled

Troubleshooting

Issue: Out of Memory (OOM)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# Reduce context size
./main -m model.gguf -c 1024 -b 256 ...

# Use smaller quantization
# Switch from Q5_K_M to Q4_K_M or Q3_K_M

# Optimize or increase ZRAM on current Raspberry Pi OS
sudo nano /etc/systemd/zram-generator.conf
# Configure: zram-size = 4096
sudo systemctl daemon-reload && sudo systemctl restart systemd-zram-setup@zram0

# Enable swap (Legacy OS using dphys-swapfile)
# sudo dphys-swapfile swapoff
# sudo sed -i 's/CONF_SWAPSIZE=.*/CONF_SWAPSIZE=4096/' /etc/dphys-swapfile
# sudo dphys-swapfile setup
# sudo dphys-swapfile swapon

Issue: Slow Inference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Check CPU frequency
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq

# Enable performance mode
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Rebuild with optimizations
make clean
make -j4 LLAMA_OPENBLAS=1 LLAMA_NATIVE=1

# Reduce batch size if RAM-limited
./main -m model.gguf -b 256 ...

# Use faster model
# Switch to TinyLlama or Phi-2

Issue: Server Crashes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Check logs
journalctl -u llama-server -n 100

# Increase file descriptor limits
echo "pi soft nofile 65536" | sudo tee -a /etc/security/limits.conf
echo "pi hard nofile 65536" | sudo tee -a /etc/security/limits.conf

# Restart server with lower concurrency
./server -m model.gguf --parallel 1

# Monitor memory
watch -n 1 free -h

Issue: Model Loading Fails

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Verify model integrity
sha256sum models/*.gguf

# Check disk space
df -h

# Redownload model
wget -c <model_url>

# Test with verbose output
./main -m model.gguf -p "test" -n 10 --verbose

Best Practices

1. Choose Right Model for Hardware

1
2
3
# Raspberry Pi 4 (4GB): TinyLlama, Phi-2
# Raspberry Pi 4 (8GB): Llama 2 7B Q4_K_M, Mistral 7B Q4_K_M
# Raspberry Pi 5 (8GB): Mistral 7B Q5_K_M, Llama 2 7B Q5_K_M

2. Monitor Temperature

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Install monitoring
sudo apt install stress

# Monitor during inference
watch -n 1 'vcgencmd measure_temp && vcgencmd measure_clock arm'

# Throttling check
vcgencmd get_throttled
# 0x0 = no throttling (good)
# 0x50000 = throttled (add cooling!)

3. Use SSD for Models

1
2
3
4
5
6
7
# Move models to SSD
sudo mkdir /mnt/ssd
sudo mount /dev/sda1 /mnt/ssd
mv ~/llama.cpp/models/* /mnt/ssd/llm-models/
ln -s /mnt/ssd/llm-models ~/llama.cpp/models

# Add to /etc/fstab for persistence

4. Optimize Prompts

1
2
3
4
5
6
7
8
# Bad: Vague, long
prompt = "Can you please help me understand what machine learning is and how it works in detail with examples?"

# Good: Clear, concise
prompt = "Explain machine learning in 3 sentences with one example."

# Best: Structured for model
prompt = "<s>[INST] Define machine learning. Provide 1 example. Max 50 words. [/INST]"

5. Implement Request Queuing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# queue_llm.py
from queue import Queue
from threading import Thread
from llama_cpp import Llama

request_queue = Queue()
llm = Llama(model_path="model.gguf", n_threads=4)

def worker():
    while True:
        prompt, callback = request_queue.get()
        response = llm(prompt, max_tokens=200)
        callback(response['choices'][0]['text'])
        request_queue.task_done()

Thread(target=worker, daemon=True).start()

# Submit requests
def submit(prompt):
    future = Queue()
    request_queue.put((prompt, future.put))
    return future.get()

Complete Example: Personal AI Assistant

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
#!/usr/bin/env python3
# pi_assistant.py - Complete local AI assistant

from llama_cpp import Llama
import speech_recognition as sr
import pyttsx3
from datetime import datetime
import subprocess

class PiAssistant:
    def __init__(self, model_path):
        print("Loading LLM...")
        self.llm = Llama(
            model_path=model_path,
            n_ctx=2048,
            n_threads=4,
            n_batch=512,
            verbose=False
        )

        self.tts = pyttsx3.init()
        self.tts.setProperty('rate', 150)

        self.recognizer = sr.Recognizer()
        self.conversation = []

        print("Assistant ready!")

    def listen(self):
        """Listen for voice input"""
        with sr.Microphone() as source:
            print("Listening...")
            self.recognizer.adjust_for_ambient_noise(source)
            audio = self.recognizer.listen(source)

        try:
            text = self.recognizer.recognize_google(audio)
            print(f"You said: {text}")
            return text
        except sr.UnknownValueError:
            return None
        except sr.RequestError:
            print("Speech recognition unavailable, type instead:")
            return input("> ")

    def speak(self, text):
        """Text-to-speech output"""
        print(f"Assistant: {text}")
        self.tts.say(text)
        self.tts.runAndWait()

    def query_llm(self, prompt):
        """Query LLM with conversation context"""
        # Build context
        context = "\n".join([
            f"{msg['role']}: {msg['text']}"
            for msg in self.conversation[-6:]  # Last 3 exchanges
        ])

        full_prompt = f"""{context}
User: {prompt}
Assistant:"""

        # Generate response
        output = self.llm(
            full_prompt,
            max_tokens=150,
            temperature=0.7,
            top_p=0.9,
            stop=["User:", "\n\n"]
        )

        response = output['choices'][0]['text'].strip()

        # Update conversation
        self.conversation.append({"role": "User", "text": prompt})
        self.conversation.append({"role": "Assistant", "text": response})

        return response

    def execute_command(self, text):
        """Handle system commands"""
        text_lower = text.lower()

        if "what time" in text_lower:
            return f"It's {datetime.now().strftime('%I:%M %p')}"

        elif "temperature" in text_lower or "cpu temp" in text_lower:
            temp = subprocess.check_output(["vcgencmd", "measure_temp"])
            return f"CPU temperature is {temp.decode().split('=')[1]}"

        elif "reboot" in text_lower or "restart" in text_lower:
            return "I can't restart the system for safety reasons."

        return None

    def run(self):
        """Main assistant loop"""
        self.speak("Hello! I'm your Raspberry Pi assistant. How can I help?")

        while True:
            # Listen for input
            user_input = self.listen()

            if not user_input:
                continue

            if "goodbye" in user_input.lower() or "exit" in user_input.lower():
                self.speak("Goodbye!")
                break

            # Check for system commands first
            cmd_response = self.execute_command(user_input)
            if cmd_response:
                self.speak(cmd_response)
                continue

            # Query LLM
            response = self.query_llm(user_input)
            self.speak(response)

if __name__ == "__main__":
    # Install dependencies first:
    # pip3 install SpeechRecognition pyttsx3 pyaudio llama-cpp-python

    assistant = PiAssistant(
        model_path="/home/pi/llama.cpp/models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf"
    )
    assistant.run()

Install dependencies: 

1
2
pip3 install SpeechRecognition pyttsx3 pyaudio llama-cpp-python
sudo apt install espeak portaudio19-dev

Run: 

1
python3 pi_assistant.py

Summary

This guide covered comprehensive local LLM deployment on Raspberry Pi:

✅ Setup & Installation

  • llama.cpp compilation with ARM NEON optimizations
  • OpenBLAS integration for faster inference
  • System optimization (CPU governor, memory, cooling)

✅ Model Selection

  • GGUF quantization formats (Q2_K through Q8_0)
  • Recommended models for each Pi variant
  • Quality vs speed trade-offs

✅ Optimization Techniques

  • Command-line parameter tuning
  • Prompt caching for repeated queries
  • Memory management strategies
  • Thread and batch size optimization

✅ Deployment

  • LLM server with OpenAI-compatible API
  • Systemd service configuration
  • Python integration (llama-cpp-python)
  • Web UI options

✅ Real-World Applications

  • Code generation assistant
  • Document summarization
  • Voice-controlled home automation
  • Distributed inference across Pi cluster
  • Complete AI assistant with speech

✅ Reproducible performance testing

  • Measure prompt processing and generation separately
  • Record the exact model, quantisation, context, build revision, cooling, and power mode
  • Publish temperature and throttling state with every result
  • Compare results only when the full test configuration matches

Next Steps

Further Optimization:

  1. Raspberry Pi Overclocking - Push performance limits safely
  2. Custom Quantization - Create your own GGUF models
  3. Model Fine-tuning - Specialize models for your use case
  4. GPU Acceleration - Experimental Vulkan/OpenCL support
  5. Edge AI Frameworks - Integrate with TensorFlow Lite, ONNX

Related Guides:

With local LLM inference, your Raspberry Pi becomes a privacy-focused AI powerhouse—no cloud required!

Frequently Asked Questions

Should I use llama.cpp or AI HAT+ 2?

Use llama.cpp when model portability and CPU-only operation matter. Use AI HAT+ 2 when a supported Hailo model, lower host CPU load, or a combined LLM/VLM pipeline justifies dedicated hardware.

Why are my token-per-second results different from another guide?

Model revision, quantisation, context length, prompt length, thread count, llama.cpp build, cooling, and throttling all change performance. Compare only fully documented runs.

Can swap make a model faster?

No. Swap can prevent an out-of-memory failure, but storage is far slower than RAM. Choose a smaller model or context before relying on swap.