Hitonabi
c48e583790
Engine-Cutover ROCm/HIP -> Vulkan/RADV (gfx1151): +12-22% tg auf MoE (llama-bench
verifiziert, fast 53->65 t/s). ROCm-Build bleibt als Rollback unter /opt/llamacpp.
- Backend: vocab-aware Speculative Decoding. services/gguf_meta.py liest den
Tokenizer-Fingerprint (model/pre/n_vocab) direkt aus dem GGUF-Header (ohne Modell-Load);
register_model + migrate_config haengen nur VOCAB-KOMPATIBLE Drafts an (inkl. --spec-type,
das in dieser llama.cpp-Generation noetig ist). Neue Endpoints /api/models/drafts + /{id}/draft.
- Frontend: idiotensichere Spec-Draft-UI (SpecDraftModal) - nur kompatible Drafts waehlbar,
inkompatible gesperrt mit Begruendung; SPEC/SPEC?-Badge nach echtem Aktiv-Status; Rolle in AddModel.
- maintenance.py: Engine-Update-Quelle -> ggml-org/llama.cpp (Build-Nummer-Vergleich),
ENGINE_PATH=/opt/llamacpp-vulkan.
- Startup-Warmup der brains (deploy/warmup.sh, self-detaching ExecStartPost) + deploy/provision-engine.sh.
- Cleanup: tote LiteLLM gateway/config.yaml + alle Referenzen (config.py/backup.py/backup.sh) entfernt;
README + docs/memory aktualisiert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
69 lines
2.6 KiB
Bash
69 lines
2.6 KiB
Bash
#!/usr/bin/env bash
|
||
# Provisioniert die Inferenz-Engine auf der AI Box (Strix Halo / Ryzen AI MAX+ 395, gfx1151)
|
||
# auf **Vulkan/RADV** — reproduzierbar. Braucht root (sudo).
|
||
#
|
||
# sudo bash ~/mission-control-v2/deploy/provision-engine.sh
|
||
#
|
||
# Hintergrund: Auf gfx1151 ist Vulkan/RADV ggü. ROCm/HIP messbar schneller
|
||
# (Token-Gen +12–22 %, Prefill gleich; auf der Box per llama-bench verifiziert 2026-06-27)
|
||
# UND einfacher. Der ROCm-Build bleibt unter /opt/llamacpp als Rollback liegen.
|
||
set -euo pipefail
|
||
|
||
VULKAN_DIR=/opt/llamacpp-vulkan
|
||
WARMUP_DST=/usr/local/bin/llama-swap-warmup.sh
|
||
DROPIN_DIR=/etc/systemd/system/llama-swap.service.d
|
||
SRC_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||
# Optional gepinnter Build (z.B. b9821). Leer = neuester Release.
|
||
PIN_BUILD="${MC_ENGINE_BUILD:-}"
|
||
|
||
echo "==> 1. RADV-Treiber + Vulkan-Runtime"
|
||
export DEBIAN_FRONTEND=noninteractive
|
||
apt-get update -qq
|
||
apt-get install -y mesa-vulkan-drivers libvulkan1 vulkan-tools curl jq
|
||
|
||
echo "==> 2. llama.cpp Vulkan-Build nach $VULKAN_DIR"
|
||
if [ -n "$PIN_BUILD" ]; then
|
||
TAG="$PIN_BUILD"
|
||
else
|
||
TAG="$(curl -s https://api.github.com/repos/ggml-org/llama.cpp/releases/latest | jq -r .tag_name)"
|
||
fi
|
||
ASSET="llama-${TAG}-bin-ubuntu-vulkan-x64.tar.gz"
|
||
URL="https://github.com/ggml-org/llama.cpp/releases/download/${TAG}/${ASSET}"
|
||
TMP="$(mktemp -d)"
|
||
echo " Lade $URL"
|
||
curl -sL -o "$TMP/v.tgz" "$URL"
|
||
mkdir -p "$VULKAN_DIR"
|
||
tar xzf "$TMP/v.tgz" -C "$TMP"
|
||
# Tarball entpackt nach .../llama-<tag>/ — Inhalt flach nach $VULKAN_DIR
|
||
cp -rf "$TMP"/llama-*/. "$VULKAN_DIR"/
|
||
rm -rf "$TMP"
|
||
test -x "$VULKAN_DIR/llama-server"
|
||
|
||
echo "==> 3. Symlink llama-server -> Vulkan-Build"
|
||
ln -sfn "$VULKAN_DIR/llama-server" /usr/local/bin/llama-server
|
||
|
||
echo "==> 4. systemd Drop-ins für llama-swap (LD_LIBRARY_PATH + Brain-Warmup)"
|
||
mkdir -p "$DROPIN_DIR"
|
||
cat > "$DROPIN_DIR/vulkan.conf" <<EOF
|
||
[Service]
|
||
Environment=LD_LIBRARY_PATH=$VULKAN_DIR
|
||
EOF
|
||
install -m 0755 "$SRC_DIR/warmup.sh" "$WARMUP_DST"
|
||
cat > "$DROPIN_DIR/warmup.conf" <<EOF
|
||
[Service]
|
||
# Nach jedem (Re)Start die brains vorladen. Das Skript detacht sich selbst (blockiert
|
||
# den Start nicht); '-' macht den Aufruf fehlertolerant (kann llama-swap nie failen lassen).
|
||
ExecStartPost=-$WARMUP_DST
|
||
EOF
|
||
|
||
echo "==> 5. Reload + Restart"
|
||
systemctl daemon-reload
|
||
systemctl restart llama-swap
|
||
sleep 3
|
||
systemctl is-active llama-swap
|
||
|
||
echo "==> Fertig. Aktive Engine:"
|
||
readlink -f /usr/local/bin/llama-server
|
||
vulkaninfo --summary 2>/dev/null | grep -m1 deviceName || true
|
||
echo "Rollback auf ROCm: ln -sfn /opt/llamacpp/llama-server /usr/local/bin/llama-server && rm $DROPIN_DIR/vulkan.conf && systemctl daemon-reload && systemctl restart llama-swap"
|