Feat: Vulkan/RADV-Engine + vocab-gepruefte Spec-Drafts + Provisioning/Sync
Engine-Cutover ROCm/HIP -> Vulkan/RADV (gfx1151): +12-22% tg auf MoE (llama-bench
verifiziert, fast 53->65 t/s). ROCm-Build bleibt als Rollback unter /opt/llamacpp.
- Backend: vocab-aware Speculative Decoding. services/gguf_meta.py liest den
Tokenizer-Fingerprint (model/pre/n_vocab) direkt aus dem GGUF-Header (ohne Modell-Load);
register_model + migrate_config haengen nur VOCAB-KOMPATIBLE Drafts an (inkl. --spec-type,
das in dieser llama.cpp-Generation noetig ist). Neue Endpoints /api/models/drafts + /{id}/draft.
- Frontend: idiotensichere Spec-Draft-UI (SpecDraftModal) - nur kompatible Drafts waehlbar,
inkompatible gesperrt mit Begruendung; SPEC/SPEC?-Badge nach echtem Aktiv-Status; Rolle in AddModel.
- maintenance.py: Engine-Update-Quelle -> ggml-org/llama.cpp (Build-Nummer-Vergleich),
ENGINE_PATH=/opt/llamacpp-vulkan.
- Startup-Warmup der brains (deploy/warmup.sh, self-detaching ExecStartPost) + deploy/provision-engine.sh.
- Cleanup: tote LiteLLM gateway/config.yaml + alle Referenzen (config.py/backup.py/backup.sh) entfernt;
README + docs/memory aktualisiert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env bash
|
||||
# Provisioniert die Inferenz-Engine auf der AI Box (Strix Halo / Ryzen AI MAX+ 395, gfx1151)
|
||||
# auf **Vulkan/RADV** — reproduzierbar. Braucht root (sudo).
|
||||
#
|
||||
# sudo bash ~/mission-control-v2/deploy/provision-engine.sh
|
||||
#
|
||||
# Hintergrund: Auf gfx1151 ist Vulkan/RADV ggü. ROCm/HIP messbar schneller
|
||||
# (Token-Gen +12–22 %, Prefill gleich; auf der Box per llama-bench verifiziert 2026-06-27)
|
||||
# UND einfacher. Der ROCm-Build bleibt unter /opt/llamacpp als Rollback liegen.
|
||||
set -euo pipefail
|
||||
|
||||
VULKAN_DIR=/opt/llamacpp-vulkan
|
||||
WARMUP_DST=/usr/local/bin/llama-swap-warmup.sh
|
||||
DROPIN_DIR=/etc/systemd/system/llama-swap.service.d
|
||||
SRC_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
# Optional gepinnter Build (z.B. b9821). Leer = neuester Release.
|
||||
PIN_BUILD="${MC_ENGINE_BUILD:-}"
|
||||
|
||||
echo "==> 1. RADV-Treiber + Vulkan-Runtime"
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -qq
|
||||
apt-get install -y mesa-vulkan-drivers libvulkan1 vulkan-tools curl jq
|
||||
|
||||
echo "==> 2. llama.cpp Vulkan-Build nach $VULKAN_DIR"
|
||||
if [ -n "$PIN_BUILD" ]; then
|
||||
TAG="$PIN_BUILD"
|
||||
else
|
||||
TAG="$(curl -s https://api.github.com/repos/ggml-org/llama.cpp/releases/latest | jq -r .tag_name)"
|
||||
fi
|
||||
ASSET="llama-${TAG}-bin-ubuntu-vulkan-x64.tar.gz"
|
||||
URL="https://github.com/ggml-org/llama.cpp/releases/download/${TAG}/${ASSET}"
|
||||
TMP="$(mktemp -d)"
|
||||
echo " Lade $URL"
|
||||
curl -sL -o "$TMP/v.tgz" "$URL"
|
||||
mkdir -p "$VULKAN_DIR"
|
||||
tar xzf "$TMP/v.tgz" -C "$TMP"
|
||||
# Tarball entpackt nach .../llama-<tag>/ — Inhalt flach nach $VULKAN_DIR
|
||||
cp -rf "$TMP"/llama-*/. "$VULKAN_DIR"/
|
||||
rm -rf "$TMP"
|
||||
test -x "$VULKAN_DIR/llama-server"
|
||||
|
||||
echo "==> 3. Symlink llama-server -> Vulkan-Build"
|
||||
ln -sfn "$VULKAN_DIR/llama-server" /usr/local/bin/llama-server
|
||||
|
||||
echo "==> 4. systemd Drop-ins für llama-swap (LD_LIBRARY_PATH + Brain-Warmup)"
|
||||
mkdir -p "$DROPIN_DIR"
|
||||
cat > "$DROPIN_DIR/vulkan.conf" <<EOF
|
||||
[Service]
|
||||
Environment=LD_LIBRARY_PATH=$VULKAN_DIR
|
||||
EOF
|
||||
install -m 0755 "$SRC_DIR/warmup.sh" "$WARMUP_DST"
|
||||
cat > "$DROPIN_DIR/warmup.conf" <<EOF
|
||||
[Service]
|
||||
# Nach jedem (Re)Start die brains vorladen. Das Skript detacht sich selbst (blockiert
|
||||
# den Start nicht); '-' macht den Aufruf fehlertolerant (kann llama-swap nie failen lassen).
|
||||
ExecStartPost=-$WARMUP_DST
|
||||
EOF
|
||||
|
||||
echo "==> 5. Reload + Restart"
|
||||
systemctl daemon-reload
|
||||
systemctl restart llama-swap
|
||||
sleep 3
|
||||
systemctl is-active llama-swap
|
||||
|
||||
echo "==> Fertig. Aktive Engine:"
|
||||
readlink -f /usr/local/bin/llama-server
|
||||
vulkaninfo --summary 2>/dev/null | grep -m1 deviceName || true
|
||||
echo "Rollback auf ROCm: ln -sfn /opt/llamacpp/llama-server /usr/local/bin/llama-server && rm $DROPIN_DIR/vulkan.conf && systemctl daemon-reload && systemctl restart llama-swap"
|
||||
Reference in New Issue
Block a user