diff --git a/README.md b/README.md index a6afc1a..edaa263 100644 --- a/README.md +++ b/README.md @@ -64,11 +64,21 @@ ## Performance -Tested on a Samsung Exynos phone using GPU offload (Vulkan, `BUILD_VULKAN=1`). Tune GPU offload layers (`-ngl`) in the Web UI (**Private Cloud & Services -> Hardware Tweaks**) or the browser extension. +Tested on a Samsung Exynos phone (Xclipse 960 GPU) using GPU offload (Vulkan, packaged `llama-cpp-backend-vulkan`). Tune GPU offload layers (`-ngl`) in the Web UI (**Private Cloud & Services -> Hardware Tweaks**) or the browser extension. -| Device | Model | tok/s | s/article | Battery / 20 articles | Temp change | -|---|---|---|---|---|---| -| Exynos phone, GPU (Vulkan) | Llama-3.2-3B Q4_K_M | measurements coming | measurements coming | measurements coming | measurements coming | +Measured with `llama-bench -p 512 -n 32 -r 1` (Vulkan, `-ngl 99`) inside the Meridian app Console on a real phone: + +| GPU | Model | Threads | Prompt (pp512) | Generation (tg32) | +|---|---|---|---|---| +| Samsung Xclipse 960 (Exynos) | Qwen3 4B Q4_K_M | **2** | **232.29 t/s** | **23.23 t/s** | +| Samsung Xclipse 960 (Exynos) | Qwen3 4B Q4_K_M | 4 | 228.05 t/s | 21.92 t/s | +| Samsung Xclipse 960 (Exynos) | Qwen3 4B Q4_K_M | 8 | 167.78 t/s | 13.79 t/s | + +![Real llama-bench run inside the Meridian app Console on Samsung Xclipse 960 (Vulkan)](screenshots/bench-meridian-xclipse960.jpg) + +*Real llama-bench run inside the Meridian app Console on Samsung Xclipse 960 (Vulkan)* + +With full GPU offload, more CPU threads do not help: **`THREADS=2` is recommended** (fastest prompt reading and generation). `termux/start.sh` uses 2 threads by default when GPU offload is on; override with `THREADS=4 bash ~/nullcloud/termux/start.sh`. Measure your own device: run `bash termux/bench.sh 20` in Airplane Mode with the charger unplugged. It prints real end-to-end tok/s and seconds per article. PRs with your results are welcome. @@ -147,7 +157,15 @@ Need turn-key deployment of **on-device, no cloud AI** for privileged legal disc - **Model download stopped halfway:** run setup again. Partial `*.gguf.part` files are resumed or replaced. - **Share -> Termux does nothing:** install **Termux:API** from F-Droid as well, run `pkg install -y termux-api`, and check that `~/bin/termux-url-opener` exists and is executable (`chmod +x ~/bin/termux-url-opener`). - **Generation stops when the screen turns off:** Android Settings -> Apps -> Termux -> Battery -> **Unrestricted**, and run `termux-wake-lock`. -- **Slow or the GPU is not used:** rebuild with Vulkan: `BUILD_VULKAN=1 bash ~/nullcloud/setup.sh`. If it still fails, lower GPU layers in the Web UI hardware tweaks. +- **GPU is not used (`llama-bench` shows backend `CPU`, or start.sh says "CPU fallback"):** install the Vulkan backend and the Android Vulkan loader: + ```bash + pkg install -y llama-cpp llama-cpp-backend-vulkan + pkg uninstall -y vulkan-loader-generic + pkg install -y vulkan-loader-android + llama-server --list-devices + ``` + You should see a line like `Vulkan0: Samsung Xclipse 960 (...)`. If the package is not available, build from source: `BUILD_VULKAN=1 bash ~/nullcloud/setup.sh`. +- **`llama-server` fails with a `--flash-attn` error:** newer llama.cpp needs a value (`--flash-attn on`). Update Nullcloud with `cd ~/nullcloud && git pull`; `start.sh` also falls back to GPU without Flash-Attention. - **Web UI says "Wrong token":** your token is in `~/.nullcloud-token`. Show it with `cat ~/.nullcloud-token` and paste it once in the Web UI. - **Port 8080 already in use:** `bash ~/nullcloud/termux/stop.sh`, then start again. diff --git a/screenshots/bench-meridian-xclipse960.jpg b/screenshots/bench-meridian-xclipse960.jpg new file mode 100644 index 0000000..e043d0b Binary files /dev/null and b/screenshots/bench-meridian-xclipse960.jpg differ diff --git a/server/ui.html b/server/ui.html index f9c0fef..4480065 100644 --- a/server/ui.html +++ b/server/ui.html @@ -112,7 +112,8 @@

Need help? Setup services

CPU & GPU Hardware Acceleration Tweaks

Tune both GPU layer offload (Vulkan on Android, tested on Exynos; Metal / CUDA / ROCm on desktop) and thread count + quantized KV cache (q8_0/q4_0):

-
Detecting local CPU + GPU engine...
+
Detecting local GPU engine...
+
Measured in Meridian on Samsung Xclipse 960: prompt 232 t/s, generation 23 t/s (Qwen3 4B Q4_K_M). Recommended: 2 CPU threads with full GPU offload.
diff --git a/termux/bench.sh b/termux/bench.sh index fd5fa37..ac53a5f 100755 --- a/termux/bench.sh +++ b/termux/bench.sh @@ -23,7 +23,7 @@ for i in $(seq 1 "$RUNS"); do done read B1 T1 < <(bat) echo -echo "== Nullcloud CPU + GPU benchmark ==" +echo "== Nullcloud GPU benchmark ==" echo "model: ${MODEL_NAME:-local} | gpu: ${GPU_NAME:-Mobile GPU} (-ngl ${GPU_LAYERS:-99})" echo "runs: $RUNS (each ~1,300-token page in, <=200 tokens out)" awk "BEGIN{printf \"end-to-end speed: %.1f tok/s (includes page reading)\n\", $TOK/$SEC}" diff --git a/termux/setup.sh b/termux/setup.sh index 97bc609..915d306 100755 --- a/termux/setup.sh +++ b/termux/setup.sh @@ -12,8 +12,8 @@ URL_1B="https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main MEM_KB="$(awk '/MemTotal:/ {print $2}' /proc/meminfo 2>/dev/null || echo 6000000)" MEM_GB="$(awk "BEGIN{printf \"%.1f\", $MEM_KB / 1048576}")" -# Auto-detect mobile GPU (Snapdragon Adreno, Arm Mali/Immortalis, Samsung Xclipse, or Vulkan ICD) -GPU_NAME="Mobile GPU (Vulkan/OpenCL)" +# Rough GPU guess from sysfs; replaced by llama.cpp's real Vulkan device name after install +GPU_NAME="Mobile GPU (Vulkan)" if [ -r /sys/class/kgsl/kgsl-3d0/gpu_model ]; then GPU_NAME="Adreno $(tr -d '\0\r\n' < /sys/class/kgsl/kgsl-3d0/gpu_model)" elif [ -d /sys/class/misc/mali0 ] || [ -e /dev/mali0 ]; then @@ -48,15 +48,20 @@ fi MODEL_URL="${MODEL_URL:-$DEFAULT_URL}" -echo "== Nullcloud On-Device AI Setup (CPU + GPU Hybrid) ==" +echo "== Nullcloud On-Device AI Setup (GPU-first, Vulkan) ==" echo "Detected RAM: ${MEM_GB} GB | GPU: ${GPU_NAME}" echo "Selecting ${MODEL_NAME} profile (run termux/bench.sh to measure speed)" pkg update -y && pkg upgrade -y -pkg install -y python curl wget jq termux-api unzip poppler vulkan-loader-android 2>/dev/null || \ - pkg install -y python curl wget jq termux-api unzip poppler +pkg install -y python curl wget jq termux-api unzip poppler -if [ "${BUILD_VULKAN:-0}" = "1" ]; then +# Prefer the packaged llama.cpp with the Vulkan backend (no compiling needed). +# vulkan-loader-android uses the phone's real GPU driver; vulkan-loader-generic does not. +pkg uninstall -y vulkan-loader-generic 2>/dev/null || true +pkg install -y vulkan-loader-android 2>/dev/null || true +pkg install -y llama-cpp llama-cpp-backend-vulkan 2>/dev/null || pkg install -y llama-cpp 2>/dev/null || true + +if [ "${BUILD_VULKAN:-0}" = "1" ] || ! command -v llama-server >/dev/null; then echo "Building llama.cpp from source with Vulkan GPU acceleration (-DGGML_VULKAN=ON)..." pkg install -y git cmake clang vulkan-headers vulkan-loader-android shaderc SRC_DIR="$HOME/.cache/llama.cpp-vulkan" @@ -65,9 +70,19 @@ if [ "${BUILD_VULKAN:-0}" = "1" ]; then cmake -S "$SRC_DIR" -B "$SRC_DIR/build" -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release cmake --build "$SRC_DIR/build" --target llama-server -j"$(nproc 2>/dev/null || echo 4)" cp "$SRC_DIR/build/bin/llama-server" "$PREFIX/bin/llama-server" -elif ! command -v llama-server >/dev/null; then - pkg install -y llama-cpp 2>/dev/null || pkg install -y llama.cpp 2>/dev/null || { - echo "!! llama.cpp package not found in your Termux repo. Run with BUILD_VULKAN=1 bash termux/setup.sh to build from source."; exit 1; } +fi + +# Ask llama.cpp for the real GPU name (e.g. "Vulkan0: Samsung Xclipse 960 (...)" -> "Samsung Xclipse 960") +detect_llama_gpu() { + command -v llama-server >/dev/null 2>&1 || return 0 + llama-server --list-devices 2>/dev/null | sed -n 's/^[[:space:]]*Vulkan0:[[:space:]]*\(.*[^[:space:]]\)[[:space:]]*(.*/\1/p' | head -n 1 +} +DETECTED_GPU="$(detect_llama_gpu)" +if [ -n "$DETECTED_GPU" ]; then + GPU_NAME="$DETECTED_GPU" + echo "llama.cpp Vulkan device: $GPU_NAME" +else + echo "!! llama.cpp sees no Vulkan GPU. Check with: llama-server --list-devices (see README Troubleshooting)" fi pip install -r "$ROOT/server/requirements.txt" @@ -101,7 +116,7 @@ echo "$ROOT" > "$HOME/.nullcloud-root" echo echo "== Setup complete (On-Device, No Cloud AI) ==" echo "Profile: $MODEL_NAME (CTX=$CTX_SIZE, MAX_CHARS=$MAX_CHARS)" -echo "Compute: CPU (big.LITTLE + ARM NEON/dotprod) + GPU Offload ($GPU_NAME, -ngl $GPU_LAYERS)" +echo "Compute: GPU offload ($GPU_NAME, -ngl $GPU_LAYERS, Vulkan), THREADS=2 recommended" echo "Your token: $(cat "$HOME/.nullcloud-token")" echo "Start: bash $ROOT/termux/start.sh" echo "Entry points ready:" diff --git a/termux/start.sh b/termux/start.sh index bf64def..e6d79ff 100755 --- a/termux/start.sh +++ b/termux/start.sh @@ -1,7 +1,7 @@ #!/data/data/com.termux/files/usr/bin/bash -# Start Nullcloud on Android (Termux): CPU + GPU hyper-optimized llama-server + local API server. +# Start Nullcloud on Android (Termux): GPU-first (Vulkan) llama-server + local API server. # Set TAILSCALE=1 to bind to your phone's Tailscale IPv4 for Private Cloud laptop access. -# Set GPU_LAYERS=0 for CPU-only, or GPU_LAYERS=99 (default) for Vulkan/OpenCL mobile GPU offload. +# GPU_LAYERS=99 (default) offloads all layers to the Vulkan GPU. THREADS=2 is recommended with full GPU offload. ROOT="$(cd "$(dirname "$0")/.." && pwd)" TOKEN="$(cat "$HOME/.nullcloud-token")" [ -f "$HOME/.nullcloud-profile.env" ] && . "$HOME/.nullcloud-profile.env" @@ -11,11 +11,34 @@ MAX_CHARS="${MAX_CHARS:-6000}" MAX_TOKENS="${MAX_TOKENS:-400}" MODEL_NAME="${MODEL_NAME:-Llama-3.2-3B-Instruct-Q4_K_M}" GPU_LAYERS="${GPU_LAYERS:-99}" -GPU_NAME="${GPU_NAME:-Mobile GPU (Vulkan/OpenCL)}" -# CPU tweak: detect big/performance cores, cap at 4 on 1B profile or 6 on 3B profile for optimal tok/s & thermals +# Ask llama.cpp for the real GPU name (e.g. "Vulkan0: Samsung Xclipse 960 (...)" -> "Samsung Xclipse 960") +detect_llama_gpu() { + command -v llama-server >/dev/null 2>&1 || return 0 + llama-server --list-devices 2>/dev/null | sed -n 's/^[[:space:]]*Vulkan0:[[:space:]]*\(.*[^[:space:]]\)[[:space:]]*(.*/\1/p' | head -n 1 +} +case "${GPU_NAME:-}" in + ""|"Mobile GPU"*|"Vulkan ("*|"Arm Mali / Immortalis GPU") + DETECTED="$(detect_llama_gpu)" + if [ -n "$DETECTED" ]; then + GPU_NAME="$DETECTED" + PROFILE="$HOME/.nullcloud-profile.env" + if [ -f "$PROFILE" ] && grep -q '^GPU_NAME=' "$PROFILE"; then + sed -i "s|^GPU_NAME=.*|GPU_NAME=\"$GPU_NAME\"|" "$PROFILE" + else + echo "GPU_NAME=\"$GPU_NAME\"" >> "$PROFILE" + fi + fi + ;; +esac +GPU_NAME="${GPU_NAME:-Mobile GPU (Vulkan)}" + +# Threads: with full GPU offload, 2 threads gave the best prompt speed in real tests +# (Samsung Xclipse 960, Qwen3 4B Q4_K_M: 2 threads pp512 232.3 / tg32 23.2 t/s, 4 threads 228.1 / 21.9, 8 threads 167.8 / 13.8). CORES="$(nproc 2>/dev/null || echo 4)" -if [ "${MODEL_TIER:-3b}" = "1b" ]; then +if [ "$GPU_LAYERS" -gt 0 ] 2>/dev/null; then + DEFAULT_THREADS=2 +elif [ "${MODEL_TIER:-3b}" = "1b" ]; then DEFAULT_THREADS=$(( CORES > 4 ? 4 : CORES )) else DEFAULT_THREADS=$(( CORES > 6 ? 6 : CORES )) @@ -34,22 +57,31 @@ termux-wake-lock 2>/dev/null pkill -f "llama-server.*--port 8081" 2>/dev/null pkill -f "server/server.py" 2>/dev/null -# Tier 1: GPU offload (-ngl) + quantized KV cache (q8_0/q4_0) + Flash-Attention + CPU big-core threads -ACCEL_ACTIVE="CPU ($THREADS threads) + GPU ($GPU_NAME, -ngl $GPU_LAYERS, KV q8_0/q4_0, Flash-Attn)" +# Tier 1: GPU offload (-ngl) + quantized KV cache (q8_0/q4_0) + Flash-Attention +ACCEL_ACTIVE="GPU ($GPU_NAME, -ngl $GPU_LAYERS, KV q8_0/q4_0, Flash-Attn, $THREADS threads)" if [ "$GPU_LAYERS" -gt 0 ] 2>/dev/null; then nohup llama-server -m "$HOME/nullcloud-models/model.gguf" \ -c "$CTX_SIZE" -t "$THREADS" -ngl "$GPU_LAYERS" -b 512 -ub 128 \ - --cache-type-k q8_0 --cache-type-v q4_0 --flash-attn \ + --cache-type-k q8_0 --cache-type-v q4_0 --flash-attn on \ + --host 127.0.0.1 --port 8081 > "$HOME/nullcloud-llm.log" 2>&1 & + sleep 1 +fi + +# Tier 1b: GPU offload without Flash-Attention / quantized KV (older or stricter llama.cpp builds) +if [ "$GPU_LAYERS" -gt 0 ] 2>/dev/null && ! pgrep -f "llama-server.*--port 8081" >/dev/null 2>&1; then + ACCEL_ACTIVE="GPU ($GPU_NAME, -ngl $GPU_LAYERS, $THREADS threads)" + nohup llama-server -m "$HOME/nullcloud-models/model.gguf" \ + -c "$CTX_SIZE" -t "$THREADS" -ngl "$GPU_LAYERS" \ --host 127.0.0.1 --port 8081 > "$HOME/nullcloud-llm.log" 2>&1 & sleep 1 fi # Tier 2 fallback: CPU-optimized quantized KV cache + Flash-Attention (if GPU offload not compiled or GPU_LAYERS=0) if ! pgrep -f "llama-server.*--port 8081" >/dev/null 2>&1; then - ACCEL_ACTIVE="CPU ($THREADS big-cores, NEON/dotprod, KV q8_0/q4_0, Flash-Attn)" + ACCEL_ACTIVE="CPU fallback ($THREADS threads, KV q8_0/q4_0, Flash-Attn) - GPU did not start, see Troubleshooting" nohup llama-server -m "$HOME/nullcloud-models/model.gguf" \ -c "$CTX_SIZE" -t "$THREADS" -b 512 -ub 128 \ - --cache-type-k q8_0 --cache-type-v q4_0 --flash-attn \ + --cache-type-k q8_0 --cache-type-v q4_0 --flash-attn on \ --host 127.0.0.1 --port 8081 > "$HOME/nullcloud-llm.log" 2>&1 & sleep 1 fi