Real Vulkan benchmarks, --flash-attn on, GPU name detection

README.md

@@ -64,11 +64,21 @@
 
 ## Performance
 
-Tested on a Samsung Exynos phone using GPU offload (Vulkan, `BUILD_VULKAN=1`). Tune GPU offload layers (`-ngl`) in the Web UI (**Private Cloud & Services -> Hardware Tweaks**) or the browser extension.
+Tested on a Samsung Exynos phone (Xclipse 960 GPU) using GPU offload (Vulkan, packaged `llama-cpp-backend-vulkan`). Tune GPU offload layers (`-ngl`) in the Web UI (**Private Cloud & Services -> Hardware Tweaks**) or the browser extension.
 
-| Device | Model | tok/s | s/article | Battery / 20 articles | Temp change |
-|---|---|---|---|---|---|
-| Exynos phone, GPU (Vulkan) | Llama-3.2-3B Q4_K_M | measurements coming | measurements coming | measurements coming | measurements coming |
+Measured with `llama-bench -p 512 -n 32 -r 1` (Vulkan, `-ngl 99`) inside the Meridian app Console on a real phone:
+
+| GPU | Model | Threads | Prompt (pp512) | Generation (tg32) |
+|---|---|---|---|---|
+| Samsung Xclipse 960 (Exynos) | Qwen3 4B Q4_K_M | **2** | **232.29 t/s** | **23.23 t/s** |
+| Samsung Xclipse 960 (Exynos) | Qwen3 4B Q4_K_M | 4 | 228.05 t/s | 21.92 t/s |
+| Samsung Xclipse 960 (Exynos) | Qwen3 4B Q4_K_M | 8 | 167.78 t/s | 13.79 t/s |
+
+![Real llama-bench run inside the Meridian app Console on Samsung Xclipse 960 (Vulkan)](screenshots/bench-meridian-xclipse960.jpg)
+
+*Real llama-bench run inside the Meridian app Console on Samsung Xclipse 960 (Vulkan)*
+
+With full GPU offload, more CPU threads do not help: **`THREADS=2` is recommended** (fastest prompt reading and generation). `termux/start.sh` uses 2 threads by default when GPU offload is on; override with `THREADS=4 bash ~/nullcloud/termux/start.sh`.
 
 Measure your own device: run `bash termux/bench.sh 20` in Airplane Mode with the charger unplugged. It prints real end-to-end tok/s and seconds per article. PRs with your results are welcome.
 
@@ -147,7 +157,15 @@ Need turn-key deployment of **on-device, no cloud AI** for privileged legal disc
 - **Model download stopped halfway:** run setup again. Partial `*.gguf.part` files are resumed or replaced.
 - **Share -> Termux does nothing:** install **Termux:API** from F-Droid as well, run `pkg install -y termux-api`, and check that `~/bin/termux-url-opener` exists and is executable (`chmod +x ~/bin/termux-url-opener`).
 - **Generation stops when the screen turns off:** Android Settings -> Apps -> Termux -> Battery -> **Unrestricted**, and run `termux-wake-lock`.
-- **Slow or the GPU is not used:** rebuild with Vulkan: `BUILD_VULKAN=1 bash ~/nullcloud/setup.sh`. If it still fails, lower GPU layers in the Web UI hardware tweaks.
+- **GPU is not used (`llama-bench` shows backend `CPU`, or start.sh says "CPU fallback"):** install the Vulkan backend and the Android Vulkan loader:
+  ```bash
+  pkg install -y llama-cpp llama-cpp-backend-vulkan
+  pkg uninstall -y vulkan-loader-generic
+  pkg install -y vulkan-loader-android
+  llama-server --list-devices
+  ```
+  You should see a line like `Vulkan0: Samsung Xclipse 960 (...)`. If the package is not available, build from source: `BUILD_VULKAN=1 bash ~/nullcloud/setup.sh`.
+- **`llama-server` fails with a `--flash-attn` error:** newer llama.cpp needs a value (`--flash-attn on`). Update Nullcloud with `cd ~/nullcloud && git pull`; `start.sh` also falls back to GPU without Flash-Attention.
 - **Web UI says "Wrong token":** your token is in `~/.nullcloud-token`. Show it with `cat ~/.nullcloud-token` and paste it once in the Web UI.
 - **Port 8080 already in use:** `bash ~/nullcloud/termux/stop.sh`, then start again.
 

server/ui.html

@@ -112,7 +112,8 @@ <h3>Need help? Setup services</h3>
   <div class="card">
     <h3>CPU &amp; GPU Hardware Acceleration Tweaks</h3>
     <p class="small" style="margin:0 0 8px">Tune both <strong>GPU layer offload</strong> (Vulkan on Android, tested on Exynos; Metal / CUDA / ROCm on desktop) and thread count + quantized KV cache (<code>q8_0/q4_0</code>):</p>
-    <div id="hw-summary" class="small" style="margin-bottom:8px">Detecting local CPU + GPU engine...</div>
+    <div id="hw-summary" class="small" style="margin-bottom:8px">Detecting local GPU engine...</div>
+    <div class="small" style="margin-bottom:8px">Measured in Meridian on Samsung Xclipse 960: prompt 232 t/s, generation 23 t/s (Qwen3 4B Q4_K_M). Recommended: 2 CPU threads with full GPU offload.</div>
     <div class="row">
       <div>
         <label class="small" for="gpu-layers"><strong>GPU Offload Layers (<code>-ngl</code>)</strong></label>

termux/bench.sh

@@ -23,7 +23,7 @@ for i in $(seq 1 "$RUNS"); do
 done
 read B1 T1 < <(bat)
 echo
-echo "== Nullcloud CPU + GPU benchmark =="
+echo "== Nullcloud GPU benchmark =="
 echo "model: ${MODEL_NAME:-local} | gpu: ${GPU_NAME:-Mobile GPU} (-ngl ${GPU_LAYERS:-99})"
 echo "runs: $RUNS  (each ~1,300-token page in, <=200 tokens out)"
 awk "BEGIN{printf \"end-to-end speed: %.1f tok/s (includes page reading)\n\", $TOK/$SEC}"

termux/setup.sh

@@ -12,8 +12,8 @@ URL_1B="https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main
 MEM_KB="$(awk '/MemTotal:/ {print $2}' /proc/meminfo 2>/dev/null || echo 6000000)"
 MEM_GB="$(awk "BEGIN{printf \"%.1f\", $MEM_KB / 1048576}")"
 
-# Auto-detect mobile GPU (Snapdragon Adreno, Arm Mali/Immortalis, Samsung Xclipse, or Vulkan ICD)
-GPU_NAME="Mobile GPU (Vulkan/OpenCL)"
+# Rough GPU guess from sysfs; replaced by llama.cpp's real Vulkan device name after install
+GPU_NAME="Mobile GPU (Vulkan)"
 if [ -r /sys/class/kgsl/kgsl-3d0/gpu_model ]; then
   GPU_NAME="Adreno $(tr -d '\0\r\n' < /sys/class/kgsl/kgsl-3d0/gpu_model)"
 elif [ -d /sys/class/misc/mali0 ] || [ -e /dev/mali0 ]; then
@@ -48,15 +48,20 @@ fi
 
 MODEL_URL="${MODEL_URL:-$DEFAULT_URL}"
 
-echo "== Nullcloud On-Device AI Setup (CPU + GPU Hybrid) =="
+echo "== Nullcloud On-Device AI Setup (GPU-first, Vulkan) =="
 echo "Detected RAM: ${MEM_GB} GB | GPU: ${GPU_NAME}"
 echo "Selecting ${MODEL_NAME} profile (run termux/bench.sh to measure speed)"
 
 pkg update -y && pkg upgrade -y
-pkg install -y python curl wget jq termux-api unzip poppler vulkan-loader-android 2>/dev/null || \
-  pkg install -y python curl wget jq termux-api unzip poppler
+pkg install -y python curl wget jq termux-api unzip poppler
 
-if [ "${BUILD_VULKAN:-0}" = "1" ]; then
+# Prefer the packaged llama.cpp with the Vulkan backend (no compiling needed).
+# vulkan-loader-android uses the phone's real GPU driver; vulkan-loader-generic does not.
+pkg uninstall -y vulkan-loader-generic 2>/dev/null || true
+pkg install -y vulkan-loader-android 2>/dev/null || true
+pkg install -y llama-cpp llama-cpp-backend-vulkan 2>/dev/null || pkg install -y llama-cpp 2>/dev/null || true
+
+if [ "${BUILD_VULKAN:-0}" = "1" ] || ! command -v llama-server >/dev/null; then
   echo "Building llama.cpp from source with Vulkan GPU acceleration (-DGGML_VULKAN=ON)..."
   pkg install -y git cmake clang vulkan-headers vulkan-loader-android shaderc
   SRC_DIR="$HOME/.cache/llama.cpp-vulkan"
@@ -65,9 +70,19 @@ if [ "${BUILD_VULKAN:-0}" = "1" ]; then
   cmake -S "$SRC_DIR" -B "$SRC_DIR/build" -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
   cmake --build "$SRC_DIR/build" --target llama-server -j"$(nproc 2>/dev/null || echo 4)"
   cp "$SRC_DIR/build/bin/llama-server" "$PREFIX/bin/llama-server"
-elif ! command -v llama-server >/dev/null; then
-  pkg install -y llama-cpp 2>/dev/null || pkg install -y llama.cpp 2>/dev/null || {
-    echo "!! llama.cpp package not found in your Termux repo. Run with BUILD_VULKAN=1 bash termux/setup.sh to build from source."; exit 1; }
+fi
+
+# Ask llama.cpp for the real GPU name (e.g. "Vulkan0: Samsung Xclipse 960 (...)" -> "Samsung Xclipse 960")
+detect_llama_gpu() {
+  command -v llama-server >/dev/null 2>&1 || return 0
+  llama-server --list-devices 2>/dev/null | sed -n 's/^[[:space:]]*Vulkan0:[[:space:]]*\(.*[^[:space:]]\)[[:space:]]*(.*/\1/p' | head -n 1
+}
+DETECTED_GPU="$(detect_llama_gpu)"
+if [ -n "$DETECTED_GPU" ]; then
+  GPU_NAME="$DETECTED_GPU"
+  echo "llama.cpp Vulkan device: $GPU_NAME"
+else
+  echo "!! llama.cpp sees no Vulkan GPU. Check with: llama-server --list-devices (see README Troubleshooting)"
 fi
 pip install -r "$ROOT/server/requirements.txt"
 
@@ -101,7 +116,7 @@ echo "$ROOT" > "$HOME/.nullcloud-root"
 echo
 echo "== Setup complete (On-Device, No Cloud AI) =="
 echo "Profile:     $MODEL_NAME (CTX=$CTX_SIZE, MAX_CHARS=$MAX_CHARS)"
-echo "Compute:     CPU (big.LITTLE + ARM NEON/dotprod) + GPU Offload ($GPU_NAME, -ngl $GPU_LAYERS)"
+echo "Compute:     GPU offload ($GPU_NAME, -ngl $GPU_LAYERS, Vulkan), THREADS=2 recommended"
 echo "Your token:  $(cat "$HOME/.nullcloud-token")"
 echo "Start:       bash $ROOT/termux/start.sh"
 echo "Entry points ready:"

termux/start.sh

@@ -1,7 +1,7 @@
 #!/data/data/com.termux/files/usr/bin/bash
-# Start Nullcloud on Android (Termux): CPU + GPU hyper-optimized llama-server + local API server.
+# Start Nullcloud on Android (Termux): GPU-first (Vulkan) llama-server + local API server.
 # Set TAILSCALE=1 to bind to your phone's Tailscale IPv4 for Private Cloud laptop access.
-# Set GPU_LAYERS=0 for CPU-only, or GPU_LAYERS=99 (default) for Vulkan/OpenCL mobile GPU offload.
+# GPU_LAYERS=99 (default) offloads all layers to the Vulkan GPU. THREADS=2 is recommended with full GPU offload.
 ROOT="$(cd "$(dirname "$0")/.." && pwd)"
 TOKEN="$(cat "$HOME/.nullcloud-token")"
 [ -f "$HOME/.nullcloud-profile.env" ] && . "$HOME/.nullcloud-profile.env"
@@ -11,11 +11,34 @@ MAX_CHARS="${MAX_CHARS:-6000}"
 MAX_TOKENS="${MAX_TOKENS:-400}"
 MODEL_NAME="${MODEL_NAME:-Llama-3.2-3B-Instruct-Q4_K_M}"
 GPU_LAYERS="${GPU_LAYERS:-99}"
-GPU_NAME="${GPU_NAME:-Mobile GPU (Vulkan/OpenCL)}"
 
-# CPU tweak: detect big/performance cores, cap at 4 on 1B profile or 6 on 3B profile for optimal tok/s & thermals
+# Ask llama.cpp for the real GPU name (e.g. "Vulkan0: Samsung Xclipse 960 (...)" -> "Samsung Xclipse 960")
+detect_llama_gpu() {
+  command -v llama-server >/dev/null 2>&1 || return 0
+  llama-server --list-devices 2>/dev/null | sed -n 's/^[[:space:]]*Vulkan0:[[:space:]]*\(.*[^[:space:]]\)[[:space:]]*(.*/\1/p' | head -n 1
+}
+case "${GPU_NAME:-}" in
+  ""|"Mobile GPU"*|"Vulkan ("*|"Arm Mali / Immortalis GPU")
+    DETECTED="$(detect_llama_gpu)"
+    if [ -n "$DETECTED" ]; then
+      GPU_NAME="$DETECTED"
+      PROFILE="$HOME/.nullcloud-profile.env"
+      if [ -f "$PROFILE" ] && grep -q '^GPU_NAME=' "$PROFILE"; then
+        sed -i "s|^GPU_NAME=.*|GPU_NAME=\"$GPU_NAME\"|" "$PROFILE"
+      else
+        echo "GPU_NAME=\"$GPU_NAME\"" >> "$PROFILE"
+      fi
+    fi
+    ;;
+esac
+GPU_NAME="${GPU_NAME:-Mobile GPU (Vulkan)}"
+
+# Threads: with full GPU offload, 2 threads gave the best prompt speed in real tests
+# (Samsung Xclipse 960, Qwen3 4B Q4_K_M: 2 threads pp512 232.3 / tg32 23.2 t/s, 4 threads 228.1 / 21.9, 8 threads 167.8 / 13.8).
 CORES="$(nproc 2>/dev/null || echo 4)"
-if [ "${MODEL_TIER:-3b}" = "1b" ]; then
+if [ "$GPU_LAYERS" -gt 0 ] 2>/dev/null; then
+  DEFAULT_THREADS=2
+elif [ "${MODEL_TIER:-3b}" = "1b" ]; then
   DEFAULT_THREADS=$(( CORES > 4 ? 4 : CORES ))
 else
   DEFAULT_THREADS=$(( CORES > 6 ? 6 : CORES ))
@@ -34,22 +57,31 @@ termux-wake-lock 2>/dev/null
 pkill -f "llama-server.*--port 8081" 2>/dev/null
 pkill -f "server/server.py" 2>/dev/null
 
-# Tier 1: GPU offload (-ngl) + quantized KV cache (q8_0/q4_0) + Flash-Attention + CPU big-core threads
-ACCEL_ACTIVE="CPU ($THREADS threads) + GPU ($GPU_NAME, -ngl $GPU_LAYERS, KV q8_0/q4_0, Flash-Attn)"
+# Tier 1: GPU offload (-ngl) + quantized KV cache (q8_0/q4_0) + Flash-Attention
+ACCEL_ACTIVE="GPU ($GPU_NAME, -ngl $GPU_LAYERS, KV q8_0/q4_0, Flash-Attn, $THREADS threads)"
 if [ "$GPU_LAYERS" -gt 0 ] 2>/dev/null; then
   nohup llama-server -m "$HOME/nullcloud-models/model.gguf" \
     -c "$CTX_SIZE" -t "$THREADS" -ngl "$GPU_LAYERS" -b 512 -ub 128 \
-    --cache-type-k q8_0 --cache-type-v q4_0 --flash-attn \
+    --cache-type-k q8_0 --cache-type-v q4_0 --flash-attn on \
+    --host 127.0.0.1 --port 8081 > "$HOME/nullcloud-llm.log" 2>&1 &
+  sleep 1
+fi
+
+# Tier 1b: GPU offload without Flash-Attention / quantized KV (older or stricter llama.cpp builds)
+if [ "$GPU_LAYERS" -gt 0 ] 2>/dev/null && ! pgrep -f "llama-server.*--port 8081" >/dev/null 2>&1; then
+  ACCEL_ACTIVE="GPU ($GPU_NAME, -ngl $GPU_LAYERS, $THREADS threads)"
+  nohup llama-server -m "$HOME/nullcloud-models/model.gguf" \
+    -c "$CTX_SIZE" -t "$THREADS" -ngl "$GPU_LAYERS" \
     --host 127.0.0.1 --port 8081 > "$HOME/nullcloud-llm.log" 2>&1 &
   sleep 1
 fi
 
 # Tier 2 fallback: CPU-optimized quantized KV cache + Flash-Attention (if GPU offload not compiled or GPU_LAYERS=0)
 if ! pgrep -f "llama-server.*--port 8081" >/dev/null 2>&1; then
-  ACCEL_ACTIVE="CPU ($THREADS big-cores, NEON/dotprod, KV q8_0/q4_0, Flash-Attn)"
+  ACCEL_ACTIVE="CPU fallback ($THREADS threads, KV q8_0/q4_0, Flash-Attn) - GPU did not start, see Troubleshooting"
   nohup llama-server -m "$HOME/nullcloud-models/model.gguf" \
     -c "$CTX_SIZE" -t "$THREADS" -b 512 -ub 128 \
-    --cache-type-k q8_0 --cache-type-v q4_0 --flash-attn \
+    --cache-type-k q8_0 --cache-type-v q4_0 --flash-attn on \
     --host 127.0.0.1 --port 8081 > "$HOME/nullcloud-llm.log" 2>&1 &
   sleep 1
 fi