Feat: Vulkan/RADV-Engine + vocab-gepruefte Spec-Drafts + Provisioning/Sync

Engine-Cutover ROCm/HIP -> Vulkan/RADV (gfx1151): +12-22% tg auf MoE (llama-bench
verifiziert, fast 53->65 t/s). ROCm-Build bleibt als Rollback unter /opt/llamacpp.

- Backend: vocab-aware Speculative Decoding. services/gguf_meta.py liest den
  Tokenizer-Fingerprint (model/pre/n_vocab) direkt aus dem GGUF-Header (ohne Modell-Load);
  register_model + migrate_config haengen nur VOCAB-KOMPATIBLE Drafts an (inkl. --spec-type,
  das in dieser llama.cpp-Generation noetig ist). Neue Endpoints /api/models/drafts + /{id}/draft.
- Frontend: idiotensichere Spec-Draft-UI (SpecDraftModal) - nur kompatible Drafts waehlbar,
  inkompatible gesperrt mit Begruendung; SPEC/SPEC?-Badge nach echtem Aktiv-Status; Rolle in AddModel.
- maintenance.py: Engine-Update-Quelle -> ggml-org/llama.cpp (Build-Nummer-Vergleich),
  ENGINE_PATH=/opt/llamacpp-vulkan.
- Startup-Warmup der brains (deploy/warmup.sh, self-detaching ExecStartPost) + deploy/provision-engine.sh.
- Cleanup: tote LiteLLM gateway/config.yaml + alle Referenzen (config.py/backup.py/backup.sh) entfernt;
  README + docs/memory aktualisiert.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Hitonabi
2026-06-27 01:54:29 +02:00
parent 530d77ff1b
commit c48e583790
23 changed files with 1128 additions and 508 deletions
+9 -4
View File
@@ -3,7 +3,8 @@
Komponierbarer Local-AI-Stack für den Bosgame M5. Greenfield-Neuaufbau — Komponierbarer Local-AI-Stack für den Bosgame M5. Greenfield-Neuaufbau —
siehe Architektur-Plan (`docs/` bzw. der genehmigte Plan). siehe Architektur-Plan (`docs/` bzw. der genehmigte Plan).
**Schichten:** Engine (llama-swap) · Routing-Gateway (LiteLLM, `model: auto`) · **Schichten:** Engine (llama-swap, **Vulkan/RADV** auf Strix Halo) · **Builtin-Routing-Gateway**
in MC2 (`model: auto`, kein externer LiteLLM-Dienst — scheitert auf Python 3.14) ·
Mission Control 2.0 (FastAPI + React/shadcn) · Hermes Agent + hermes-webui · Mission Control 2.0 (FastAPI + React/shadcn) · Hermes Agent + hermes-webui ·
Shared Memory (SQLite via MCP). Jede Schicht hinter stabilem Vertrag austauschbar. Shared Memory (SQLite via MCP). Jede Schicht hinter stabilem Vertrag austauschbar.
@@ -13,8 +14,8 @@ Fortschritt & Resume-Guide: siehe [`docs/STATUS.md`](docs/STATUS.md).
- **Phase 0** — FastAPI-Skeleton + React/shadcn-Shell (Cmd+K, Dark, PWA). - **Phase 0** — FastAPI-Skeleton + React/shadcn-Shell (Cmd+K, Dark, PWA).
- **Phase 1** — Compute-Module (fit/caps/sources, portiert), **Discover** (live HF + Fit + Caps), - **Phase 1** — Compute-Module (fit/caps/sources, portiert), **Discover** (live HF + Fit + Caps),
**Engine-Write** (register + `groups`/Ko-Residenz), **LiteLLM-Gateway** (Config + Service, **Engine-Write** (register + `groups`/Ko-Residenz + vocab-geprüfte Spec-Drafts),
`model: auto` + Fallbacks), Frontend **Modelle & Routing** (Caps-Chips, Fit, Discover, Routing-View). **Builtin-Gateway** (`model: auto` + Fallbacks), Frontend **Modelle & Routing** (Caps-Chips, Fit, Discover, Routing-View).
- **Phase 2** — System-Status (CPU/RAM/GPU/Disk), Wartung (restart/self-update, sudo-frei), - **Phase 2** — System-Status (CPU/RAM/GPU/Disk), Wartung (restart/self-update, sudo-frei),
**Connect** (saubere IDE-Snippets → Gateway `model:auto`, LAN-IP-Override). **Connect** (saubere IDE-Snippets → Gateway `model:auto`, LAN-IP-Override).
@@ -59,5 +60,9 @@ cd frontend && npm run build # → frontend/dist
|---|---|---| |---|---|---|
| `MC_LLAMA_SWAP_URL` | `http://127.0.0.1:8080` | Engine | | `MC_LLAMA_SWAP_URL` | `http://127.0.0.1:8080` | Engine |
| `MC_CONFIG_PATH` | `/etc/llama-swap/config.yaml` | llama-swap Config | | `MC_CONFIG_PATH` | `/etc/llama-swap/config.yaml` | llama-swap Config |
| `MC_GATEWAY_URL` | `http://127.0.0.1:4000` | LiteLLM-Gateway | | `MC_GATEWAY_URL` | `http://127.0.0.1:$MC_PORT` | Builtin-Gateway (Teil von MC2, kein externer Dienst) |
| `MC_PORT` | `9000` | MC-Backend-Port | | `MC_PORT` | `9000` | MC-Backend-Port |
| `MC_ENGINE_PATH` | `/opt/llamacpp-vulkan` | Aktive Engine-Binary (Vulkan-Build) |
| `MC_ENGINE_REPO` | `ggml-org/llama.cpp` | Quelle für Engine-Update-Check |
| `MC_DRAFTS_DIR` | `$MODELS/drafts` | Spec-Draft-Modelle (vocab-geprüft) |
| `MC_SPEC_TYPE` | `draft-simple` | Speculative-Decoding-Typ (llama.cpp) |
+14 -7
View File
@@ -33,17 +33,24 @@ CMD_TEMPLATE = os.environ.get("MC_CMD_TEMPLATE", _DEFAULT_CMD_TEMPLATE)
if "{model}" not in CMD_TEMPLATE: if "{model}" not in CMD_TEMPLATE:
CMD_TEMPLATE = _DEFAULT_CMD_TEMPLATE CMD_TEMPLATE = _DEFAULT_CMD_TEMPLATE
DEFAULT_TTL = int(os.environ.get("MC_DEFAULT_TTL", "300")) DEFAULT_TTL = int(os.environ.get("MC_DEFAULT_TTL", "300"))
# Draft-Modell für Speculative Decoding (nur fast/coder, wenn vorhanden). Eine # Verzeichnis mit Draft-Modellen für Speculative Decoding. Beim Hinzufügen eines
# Quelle der Wahrheit für llamaswap.register_model + migrate_config. # fast/coder-Modells wird hieraus automatisch ein **vocab-kompatibler** Draft gewählt
SPEC_DRAFT_MODEL_PATH = os.environ.get( # (Vocab-Check via services.gguf_meta; ein inkompatibler Draft lässt llama.cpp scheitern).
"MC_SPEC_DRAFT_MODEL", f"{MODELS_DIR.as_posix()}/drafts/qwen2.5-1.5b-instruct-q4_k_m.gguf") DRAFTS_DIR = Path(os.environ.get("MC_DRAFTS_DIR", str(MODELS_DIR / "drafts")))
# Optionaler expliziter Default-Draft (leer = Auto-Erkennung aus DRAFTS_DIR). Wird nur
# verwendet, wenn er zum Ziel-Modell vocab-kompatibel ist. (Früher fix qwen2.5 → entfernt,
# weil das mit neueren Vocabs wie Qwen3.6 inkompatibel ist und Spec stillschweigend brach.)
SPEC_DRAFT_MODEL_PATH = os.environ.get("MC_SPEC_DRAFT_MODEL", "")
# Speculative-Decoding-Typ (llama.cpp dieser Generation braucht --spec-type zusätzlich
# zu --spec-draft-model, sonst ist Spec inaktiv).
SPEC_TYPE = os.environ.get("MC_SPEC_TYPE", "draft-simple")
# Env für HuggingFace-Downloads: XET deaktivieren (Hänger bei ~6 MB, siehe v1-Gotcha). # Env für HuggingFace-Downloads: XET deaktivieren (Hänger bei ~6 MB, siehe v1-Gotcha).
HF_DOWNLOAD_ENV = {"HF_HUB_DISABLE_XET": "1"} HF_DOWNLOAD_ENV = {"HF_HUB_DISABLE_XET": "1"}
# --- Routing-Gateway (LiteLLM, model: auto) ---------------------------------- # --- Routing-Gateway (builtin in MC2, model: auto) ---------------------------
# MC2 IST der Gateway (services/gateway.py + routers/gateway_proxy.py). KEIN externer
# LiteLLM-Dienst (scheitert auf Python 3.14). Daher keine Gateway-Config-Datei mehr.
GATEWAY_URL = os.environ.get("MC_GATEWAY_URL", f"http://127.0.0.1:{os.environ.get('MC_PORT', '9000')}").rstrip("/") GATEWAY_URL = os.environ.get("MC_GATEWAY_URL", f"http://127.0.0.1:{os.environ.get('MC_PORT', '9000')}").rstrip("/")
GATEWAY_CONFIG_PATH = Path(os.environ.get(
"MC_GATEWAY_CONFIG", str(Path(__file__).resolve().parent.parent / "gateway" / "config.yaml")))
# --- Hermes Agent (eigener Dienst auf der Box) ------------------------------- # --- Hermes Agent (eigener Dienst auf der Box) -------------------------------
# Gateway (OpenAI-API des Agenten) + standalone Web-UI (nesquena/hermes-webui). # Gateway (OpenAI-API des Agenten) + standalone Web-UI (nesquena/hermes-webui).
+7 -6
View File
@@ -4,8 +4,8 @@ from pathlib import Path
# Add backend directory to sys.path so we can import services # Add backend directory to sys.path so we can import services
sys.path.append(str(Path(__file__).resolve().parent)) sys.path.append(str(Path(__file__).resolve().parent))
from services.llamaswap import read_config, write_config from services.llamaswap import read_config, write_config, spec_draft_flags, _PATH_RE
from config import CONFIG_PATH, SPEC_DRAFT_MODEL_PATH from config import CONFIG_PATH
def migrate(): def migrate():
print(f"Reading config from {CONFIG_PATH}...") print(f"Reading config from {CONFIG_PATH}...")
@@ -16,8 +16,6 @@ def migrate():
cfg = read_config() cfg = read_config()
models = cfg.get("models", {}) models = cfg.get("models", {})
draft_path = SPEC_DRAFT_MODEL_PATH
for name, spec in models.items(): for name, spec in models.items():
if not isinstance(spec, dict): if not isinstance(spec, dict):
continue continue
@@ -36,12 +34,15 @@ def migrate():
aliases = spec.get("aliases", []) aliases = spec.get("aliases", [])
role = aliases[0] if aliases else None role = aliases[0] if aliases else None
# 3. Add parallel and speculative decoding for fast and coder # 3. Add parallel + (nur vocab-kompatibles) Speculative Decoding für fast/coder.
# spec_draft_flags() prüft die Vocab-Kompatibilität und hängt --spec-type an;
# ein inkompatibler Draft (z.B. qwen2.5 ↔ Qwen3.6) wird NICHT gesetzt.
if role in ("fast", "coder"): if role in ("fast", "coder"):
if "--parallel" not in cmd: if "--parallel" not in cmd:
cmd = cmd.strip() + " --parallel 2" cmd = cmd.strip() + " --parallel 2"
if "--spec-draft-model" not in cmd: if "--spec-draft-model" not in cmd:
cmd = cmd.strip() + f" --spec-draft-model {draft_path}" target = mt.group(1) if (mt := _PATH_RE.search(cmd)) else ""
cmd = cmd.strip() + spec_draft_flags(target)
# Update cmd # Update cmd
from ruamel.yaml.scalarstring import LiteralScalarString from ruamel.yaml.scalarstring import LiteralScalarString
+24
View File
@@ -158,6 +158,30 @@ def set_model_ctx(model_id: str, body: CtxReq) -> dict:
return {"ok": True} return {"ok": True}
@router.get("/models/drafts")
def list_drafts(target: str = "") -> dict:
"""Verfügbare Draft-Modelle + ihre Vocab-Kompatibilität zum Ziel-Modell
(target = GGUF-Pfad). Basis für die idiotensichere Spec-Draft-Auswahl im UI."""
return llamaswap.drafts_for(target)
class DraftReq(BaseModel):
draft_path: str | None = None
@router.post("/models/{model_id}/draft")
def set_model_draft(model_id: str, body: DraftReq) -> dict:
"""Setzt/entfernt den Speculative-Decoding-Draft eines Modells. Inkompatible
(oder nicht prüfbare) Drafts werden serverseitig abgelehnt."""
try:
res = llamaswap.set_spec_draft(model_id, body.draft_path)
except PermissionError as exc:
raise HTTPException(500, str(exc))
if not res["ok"]:
raise HTTPException(400 if "kompatib" in res["reason"].lower() else 404, res["reason"])
return res
@router.post("/models/unload") @router.post("/models/unload")
def unload_all_models() -> dict: def unload_all_models() -> dict:
import httpx import httpx
+2 -2
View File
@@ -7,7 +7,7 @@ import shutil
import time import time
from pathlib import Path from pathlib import Path
from config import CONFIG_PATH, GATEWAY_CONFIG_PATH, MEMORY_DB, MODELS_DIR from config import CONFIG_PATH, MEMORY_DB, MODELS_DIR
BACKUP_DIR = Path(MODELS_DIR) / "mc2-backups" BACKUP_DIR = Path(MODELS_DIR) / "mc2-backups"
RETAIN = 7 RETAIN = 7
@@ -31,7 +31,7 @@ def backup_now() -> dict:
dst.mkdir(parents=True, exist_ok=True) dst.mkdir(parents=True, exist_ok=True)
saved = [] saved = []
for src in (MEMORY_DB, Path(str(MEMORY_DB) + "-wal"), Path(str(MEMORY_DB) + "-shm"), for src in (MEMORY_DB, Path(str(MEMORY_DB) + "-wal"), Path(str(MEMORY_DB) + "-shm"),
CONFIG_PATH, GATEWAY_CONFIG_PATH): CONFIG_PATH):
if (name := _safe_copy(src, dst)): if (name := _safe_copy(src, dst)):
saved.append(name) saved.append(name)
# Aufräumen: nur die letzten RETAIN Snapshots behalten. # Aufräumen: nur die letzten RETAIN Snapshots behalten.
+148
View File
@@ -0,0 +1,148 @@
"""
GGUF-Tokenizer-Fingerprint — liest die Tokenizer-Identität direkt aus dem
GGUF-Header (ohne das Modell zu laden), um zu entscheiden, ob ein Draft-Modell
**vocab-kompatibel** mit einem Ziel-Modell ist (Voraussetzung für Speculative
Decoding in llama.cpp — sonst: "draft model vocab type must match target").
Wir lesen nur die Metadaten-KV-Sektion am Dateianfang und brechen ab, sobald
`tokenizer.ggml.tokens` erreicht ist (dessen Länge = n_vocab). model+pre+n_vocab
identifizieren den Tokenizer eindeutig genug, um die in der Praxis relevanten
Fälle zu unterscheiden (Qwen2.5 vs Qwen3 vs Qwen3.6 etc.). Die llama.cpp-Prüfung
beim Laden bleibt der letzte Schiedsrichter.
"""
import struct
from functools import lru_cache
# GGUF value types (https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)
_T_UINT8, _T_INT8, _T_UINT16, _T_INT16, _T_UINT32, _T_INT32, _T_FLOAT32, \
_T_BOOL, _T_STRING, _T_ARRAY, _T_UINT64, _T_INT64, _T_FLOAT64 = range(13)
_SCALAR_FMT = {
_T_UINT8: "<B", _T_INT8: "<b", _T_UINT16: "<H", _T_INT16: "<h",
_T_UINT32: "<I", _T_INT32: "<i", _T_FLOAT32: "<f", _T_BOOL: "<?",
_T_UINT64: "<Q", _T_INT64: "<q", _T_FLOAT64: "<d",
}
_SCALAR_SIZE = {t: struct.calcsize(f) for t, f in _SCALAR_FMT.items()}
_WANT_STRINGS = {"tokenizer.ggml.model", "tokenizer.ggml.pre", "general.architecture"}
class _Reader:
def __init__(self, f):
self.f = f
def read(self, n: int) -> bytes:
b = self.f.read(n)
if len(b) != n:
raise EOFError("unerwartetes Dateiende beim GGUF-Parsen")
return b
def u32(self) -> int:
return struct.unpack("<I", self.read(4))[0]
def u64(self) -> int:
return struct.unpack("<Q", self.read(8))[0]
def gstr(self) -> str:
n = self.u64()
return self.read(n).decode("utf-8", "replace")
def skip_value(self, vtype: int) -> None:
"""Liest einen Wert und verwirft ihn (um den Datei-Pointer korrekt
weiterzuschieben). Arrays werden elementweise konsumiert."""
if vtype == _T_STRING:
self.f.seek(self.u64(), 1)
elif vtype in _SCALAR_SIZE:
self.f.seek(_SCALAR_SIZE[vtype], 1)
elif vtype == _T_ARRAY:
etype = self.u32()
count = self.u64()
if etype == _T_STRING:
for _ in range(count):
self.f.seek(self.u64(), 1)
elif etype in _SCALAR_SIZE:
self.f.seek(_SCALAR_SIZE[etype] * count, 1)
else:
raise ValueError(f"unbekannter Array-Elementtyp {etype}")
else:
raise ValueError(f"unbekannter GGUF-Wertetyp {vtype}")
def _read_fingerprint(path: str) -> dict | None:
"""Liest model/pre/n_vocab aus dem GGUF-Header. None bei Fehler/kein GGUF."""
try:
with open(path, "rb") as fh:
r = _Reader(fh)
if r.read(4) != b"GGUF":
return None
r.u32() # version
r.u64() # tensor_count
kv_count = r.u64()
fp: dict = {"model": None, "pre": None, "arch": None, "n_vocab": None}
for _ in range(kv_count):
key = r.gstr()
vtype = r.u32()
if key == "tokenizer.ggml.tokens" and vtype == _T_ARRAY:
etype = r.u32()
fp["n_vocab"] = r.u64()
# Wir haben alles (model/pre kommen vor tokens) → abbrechen.
if etype != _T_STRING:
return None
break
if key in _WANT_STRINGS and vtype == _T_STRING:
val = r.gstr()
if key == "tokenizer.ggml.model":
fp["model"] = val
elif key == "tokenizer.ggml.pre":
fp["pre"] = val
else:
fp["arch"] = val
else:
r.skip_value(vtype)
if fp["model"] is None and fp["n_vocab"] is None:
return None
return fp
except (OSError, EOFError, ValueError, struct.error):
return None
@lru_cache(maxsize=256)
def _cached(path: str, mtime: float, size: int) -> tuple | None:
fp = _read_fingerprint(path)
if fp is None:
return None
return (fp.get("model"), fp.get("pre"), fp.get("n_vocab"), fp.get("arch"))
def fingerprint(path: str) -> dict | None:
"""Tokenizer-Fingerprint eines GGUF (gecacht nach Pfad+mtime+size).
Returns dict(model, pre, n_vocab, arch) oder None wenn nicht lesbar."""
import os
try:
st = os.stat(path)
except OSError:
return None
t = _cached(path, st.st_mtime, st.st_size)
if t is None:
return None
return {"model": t[0], "pre": t[1], "n_vocab": t[2], "arch": t[3]}
def vocab_key(path: str) -> tuple | None:
"""Vergleichsschlüssel für Vocab-Kompatibilität: (model, pre, n_vocab).
Genau diese Identität verlangt llama.cpp für Speculative Decoding."""
fp = fingerprint(path)
if not fp or fp["n_vocab"] is None:
return None
return (fp["model"], fp["pre"], fp["n_vocab"])
def compatible(target_path: str, draft_path: str) -> bool | None:
"""True/False ob draft vocab-kompatibel zum target ist. None = unbestimmbar
(eine Datei nicht lesbar) → UI behandelt das als 'nicht bestätigt'."""
a = vocab_key(target_path)
b = vocab_key(draft_path)
if a is None or b is None:
return None
return a == b
+110 -3
View File
@@ -13,7 +13,10 @@ import re
import httpx import httpx
from ruamel.yaml.scalarstring import LiteralScalarString from ruamel.yaml.scalarstring import LiteralScalarString
from config import CMD_TEMPLATE, CONFIG_PATH, DEFAULT_TTL, LLAMA_SWAP_URL, SPEC_DRAFT_MODEL_PATH from config import (
CMD_TEMPLATE, CONFIG_PATH, DEFAULT_TTL, DRAFTS_DIR, LLAMA_SWAP_URL,
SPEC_DRAFT_MODEL_PATH, SPEC_TYPE,
)
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
@@ -66,6 +69,11 @@ def _parse_model(name: str, spec: dict) -> dict:
m_draft = re.search(r"--spec-draft-model\s+([^\s]+)", cmd) m_draft = re.search(r"--spec-draft-model\s+([^\s]+)", cmd)
if m_draft: if m_draft:
spec_draft = os.path.basename(m_draft.group(1).replace("'", "").replace('"', "")) spec_draft = os.path.basename(m_draft.group(1).replace("'", "").replace('"', ""))
spec_type = None
if (m_st := re.search(r"--spec-type\s+([^\s]+)", cmd)):
spec_type = m_st.group(1)
# Spec ist nur AKTIV, wenn BEIDES gesetzt ist (--spec-draft-model UND --spec-type).
spec_active = bool(spec_draft and spec_type)
parallel_match = re.search(r"--parallel\s+(\d+)", cmd) parallel_match = re.search(r"--parallel\s+(\d+)", cmd)
parallel_slots = int(parallel_match.group(1)) if parallel_match else 1 parallel_slots = int(parallel_match.group(1)) if parallel_match else 1
@@ -85,6 +93,8 @@ def _parse_model(name: str, spec: dict) -> dict:
"incomplete": not path, "incomplete": not path,
"prompt_cache": prompt_cache, "prompt_cache": prompt_cache,
"spec_draft_model": spec_draft, "spec_draft_model": spec_draft,
"spec_type": spec_type,
"spec_active": spec_active,
"parallel_slots": parallel_slots, "parallel_slots": parallel_slots,
"capabilities": capabilities( "capabilities": capabilities(
name=filename or name, cmd=cmd, name=filename or name, cmd=cmd,
@@ -191,8 +201,10 @@ def register_model(model_path: str, role: str | None = None, ctx: int = 8192,
if role_lower in ("fast", "coder"): if role_lower in ("fast", "coder"):
if "--parallel" not in cmd: if "--parallel" not in cmd:
cmd += " --parallel 2" cmd += " --parallel 2"
if os.path.exists(SPEC_DRAFT_MODEL_PATH) and "--spec-draft-model" not in cmd: # Nur einen VOCAB-KOMPATIBLEN Draft anhängen (sonst scheitert llama.cpp).
cmd += f" --spec-draft-model {SPEC_DRAFT_MODEL_PATH}" # Bei frischem Install existiert die GGUF noch nicht → kein Draft (später im UI setzbar).
if "--spec-draft-model" not in cmd:
cmd += spec_draft_flags(model_path)
cfg.setdefault("models", {})[model_id] = { cfg.setdefault("models", {})[model_id] = {
"cmd": LiteralScalarString(cmd + "\n"), "cmd": LiteralScalarString(cmd + "\n"),
@@ -203,6 +215,101 @@ def register_model(model_path: str, role: str | None = None, ctx: int = 8192,
return model_id return model_id
# --- Speculative-Draft / Vocab-Kompatibilität --------------------------------
def list_drafts() -> list[dict]:
"""Alle Draft-GGUFs in DRAFTS_DIR mit Tokenizer-Fingerprint."""
from services import gguf_meta
out = []
if DRAFTS_DIR.is_dir():
for p in sorted(DRAFTS_DIR.glob("*.gguf")):
out.append({
"path": str(p), "filename": p.name,
"size_bytes": p.stat().st_size if p.exists() else None,
"vocab": gguf_meta.fingerprint(str(p)),
})
return out
def find_compatible_draft(target_path: str) -> str | None:
"""Pfad eines vocab-kompatiblen Drafts für target_path, oder None.
Bevorzugt MC_SPEC_DRAFT_MODEL (falls gesetzt+kompatibel), sonst der erste
kompatible Draft in DRAFTS_DIR. None auch, wenn target (noch) fehlt (dann
nicht verifizierbar → bewusst KEIN Draft anhängen)."""
if not target_path or not os.path.exists(target_path):
return None
from services import gguf_meta
candidates: list[str] = []
if SPEC_DRAFT_MODEL_PATH and os.path.exists(SPEC_DRAFT_MODEL_PATH):
candidates.append(SPEC_DRAFT_MODEL_PATH)
for d in list_drafts():
if d["path"] not in candidates:
candidates.append(d["path"])
for c in candidates:
if gguf_meta.compatible(target_path, c) is True:
return c
return None
def spec_draft_flags(target_path: str) -> str:
"""llama-server-Flags für Speculative Decoding (Draft + --spec-type), oder ''
wenn kein kompatibler Draft existiert. Beides nötig, sonst ist Spec inaktiv."""
d = find_compatible_draft(target_path)
return f" --spec-draft-model {d} --spec-type {SPEC_TYPE}" if d else ""
def drafts_for(target_path: str) -> dict:
"""Für die UI: alle Drafts + ihre Kompatibilität zum Ziel-Modell.
compatible=None heißt 'nicht prüfbar' (Ziel- oder Draft-GGUF fehlt)."""
from services import gguf_meta
exists = bool(target_path and os.path.exists(target_path))
drafts = list_drafts()
for d in drafts:
d["compatible"] = gguf_meta.compatible(target_path, d["path"]) if exists else None
return {
"target_path": target_path,
"target_exists": exists,
"target_vocab": gguf_meta.fingerprint(target_path) if exists else None,
"drafts": drafts,
}
def set_spec_draft(model_id: str, draft_path: str | None) -> dict:
"""Setzt (oder entfernt mit draft_path=None) den Spec-Draft eines Modells.
Validiert die Vocab-Kompatibilität — ein inkompatibler/unprüfbarer Draft wird
abgelehnt (idiotensicher). Returns {ok, reason}."""
cfg = read_config()
spec = (cfg.get("models") or {}).get(model_id)
if not isinstance(spec, dict):
return {"ok": False, "reason": "Modell nicht gefunden"}
cmd = str(spec.get("cmd", ""))
# vorhandene Spec-Flags entfernen (idempotent)
cmd = re.sub(r"\s+--spec-draft-model\s+\S+", "", cmd)
cmd = re.sub(r"\s+--spec-type\s+\S+", "", cmd)
if draft_path:
# relative Angabe (nur Dateiname) gegen DRAFTS_DIR auflösen
if not os.path.isabs(draft_path) and "/" not in draft_path:
draft_path = str(DRAFTS_DIR / draft_path)
if not os.path.exists(draft_path):
return {"ok": False, "reason": "Draft-Datei nicht gefunden"}
from services import gguf_meta
target = ""
if (mt := _PATH_RE.search(cmd)):
target = mt.group(1).replace("'", "").replace('"', "")
comp = gguf_meta.compatible(target, draft_path) if os.path.exists(target) else None
if comp is not True:
reason = ("Draft ist NICHT vocab-kompatibel zum Modell — Speculative Decoding "
"würde beim Laden scheitern."
if comp is False else
"Kompatibilität nicht prüfbar (Modell-GGUF fehlt) — Draft nicht gesetzt.")
return {"ok": False, "reason": reason}
cmd = cmd.rstrip() + f" --spec-draft-model {draft_path} --spec-type {SPEC_TYPE}"
spec["cmd"] = LiteralScalarString(cmd.rstrip() + "\n")
write_config(cfg)
return {"ok": True, "reason": ""}
# --- Groups (Ko-Residenz) ---------------------------------------------------- # --- Groups (Ko-Residenz) ----------------------------------------------------
def set_group(group: str, members: list[str], swap: bool = False, persist: bool = False) -> None: def set_group(group: str, members: list[str], swap: bool = False, persist: bool = False) -> None:
"""llama-swap-`groups`-Eintrag setzen. swap=False → alle Mitglieder dürfen """llama-swap-`groups`-Eintrag setzen. swap=False → alle Mitglieder dürfen
+30 -4
View File
@@ -7,6 +7,7 @@ erweiterte sudoers (siehe docs/BEDIENUNG.md). Lange Ops laufen als jobengine-Job
""" """
import os import os
import re
import subprocess import subprocess
import time import time
from datetime import datetime from datetime import datetime
@@ -21,10 +22,30 @@ SYSTEM_SERVICES = {"llama-swap"}
USER_SERVICES = {"mission-control-2", "hermes-gateway", "hermes-dashboard", "hermes-webui"} USER_SERVICES = {"mission-control-2", "hermes-gateway", "hermes-dashboard", "hermes-webui"}
ENGINE_UPDATE_CMD = os.environ.get("MC_ENGINE_UPDATE_CMD", "") ENGINE_UPDATE_CMD = os.environ.get("MC_ENGINE_UPDATE_CMD", "")
ENGINE_PATH = os.environ.get("MC_ENGINE_PATH", "/opt/llamacpp") # Engine = offizieller Vulkan-Build von ggml-org/llama.cpp (RADV auf Strix Halo).
ENGINE_PATH = os.environ.get("MC_ENGINE_PATH", "/opt/llamacpp-vulkan")
ENGINE_REPO = os.environ.get("MC_ENGINE_REPO", "ggml-org/llama.cpp")
_engine_cache = {"ts": 0.0, "avail": False} _engine_cache = {"ts": 0.0, "avail": False}
def _installed_engine_build() -> int | None:
"""Build-Nummer der installierten llama-server-Binary (z.B. 9821), oder None.
Vulkan-Build braucht LD_LIBRARY_PATH=ENGINE_PATH zum Start von --version."""
bin_path = os.path.join(ENGINE_PATH, "llama-server")
if not os.path.exists(bin_path):
return None
try:
env = dict(os.environ, LD_LIBRARY_PATH=ENGINE_PATH)
out = subprocess.run([bin_path, "--version"], capture_output=True, text=True,
timeout=20, env=env)
txt = (out.stderr or "") + (out.stdout or "")
if (m := re.search(r"build:\s*\S+\s*\((\d+)\)", txt)) or (m := re.search(r"\bb(\d{3,})\b", txt)):
return int(m.group(1))
except Exception:
return None
return None
def _ram_gb() -> float: def _ram_gb() -> float:
return psutil.virtual_memory().total / (1024 ** 3) return psutil.virtual_memory().total / (1024 ** 3)
@@ -45,11 +66,16 @@ def _engine_update_available() -> bool:
return _engine_cache["avail"] return _engine_cache["avail"]
avail = False avail = False
try: try:
rel = httpx.get("https://api.github.com/repos/lemonade-sdk/llamacpp-rocm/releases/latest", rel = httpx.get(f"https://api.github.com/repos/{ENGINE_REPO}/releases/latest",
timeout=6, headers={"User-Agent": "MissionControl2"}).json() timeout=6, headers={"User-Agent": "MissionControl2"}).json()
tag = str(rel.get("tag_name", ""))
latest = int(m.group(1)) if (m := re.search(r"(\d{3,})", tag)) else None
installed = _installed_engine_build()
if latest is not None and installed is not None:
avail = latest > installed # präziser Build-Nummer-Vergleich
else: # Fallback: Release-Datum vs. Engine-mtime
pub = datetime.fromisoformat(rel["published_at"].replace("Z", "+00:00")).timestamp() pub = datetime.fromisoformat(rel["published_at"].replace("Z", "+00:00")).timestamp()
inst = os.path.getmtime(ENGINE_PATH) avail = pub > os.path.getmtime(ENGINE_PATH) + 86400
avail = pub > inst + 86400
except Exception: except Exception:
avail = False avail = False
_engine_cache.update(ts=now, avail=avail) _engine_cache.update(ts=now, avail=avail)
+1 -2
View File
@@ -6,11 +6,10 @@ set -euo pipefail
MODELS_DIR="${MC_MODELS_DIR:-/srv/models}" MODELS_DIR="${MC_MODELS_DIR:-/srv/models}"
MEM_DB="${MC_MEMORY_DB:-$MODELS_DIR/mc2-memory.db}" MEM_DB="${MC_MEMORY_DB:-$MODELS_DIR/mc2-memory.db}"
LSWAP="${MC_CONFIG_PATH:-/etc/llama-swap/config.yaml}" LSWAP="${MC_CONFIG_PATH:-/etc/llama-swap/config.yaml}"
GW="${MC_GATEWAY_CONFIG:-/opt/mission-control-2/gateway/config.yaml}"
DEST="$MODELS_DIR/mc2-backups/$(date +%Y%m%d-%H%M%S)" DEST="$MODELS_DIR/mc2-backups/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$DEST" mkdir -p "$DEST"
for f in "$MEM_DB" "$MEM_DB-wal" "$MEM_DB-shm" "$LSWAP" "$GW"; do for f in "$MEM_DB" "$MEM_DB-wal" "$MEM_DB-shm" "$LSWAP"; do
[ -f "$f" ] && cp -p "$f" "$DEST/" || true [ -f "$f" ] && cp -p "$f" "$DEST/" || true
done done
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env bash
# Provisioniert die Inferenz-Engine auf der AI Box (Strix Halo / Ryzen AI MAX+ 395, gfx1151)
# auf **Vulkan/RADV** — reproduzierbar. Braucht root (sudo).
#
# sudo bash ~/mission-control-v2/deploy/provision-engine.sh
#
# Hintergrund: Auf gfx1151 ist Vulkan/RADV ggü. ROCm/HIP messbar schneller
# (Token-Gen +1222 %, Prefill gleich; auf der Box per llama-bench verifiziert 2026-06-27)
# UND einfacher. Der ROCm-Build bleibt unter /opt/llamacpp als Rollback liegen.
set -euo pipefail
VULKAN_DIR=/opt/llamacpp-vulkan
WARMUP_DST=/usr/local/bin/llama-swap-warmup.sh
DROPIN_DIR=/etc/systemd/system/llama-swap.service.d
SRC_DIR="$(cd "$(dirname "$0")" && pwd)"
# Optional gepinnter Build (z.B. b9821). Leer = neuester Release.
PIN_BUILD="${MC_ENGINE_BUILD:-}"
echo "==> 1. RADV-Treiber + Vulkan-Runtime"
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y mesa-vulkan-drivers libvulkan1 vulkan-tools curl jq
echo "==> 2. llama.cpp Vulkan-Build nach $VULKAN_DIR"
if [ -n "$PIN_BUILD" ]; then
TAG="$PIN_BUILD"
else
TAG="$(curl -s https://api.github.com/repos/ggml-org/llama.cpp/releases/latest | jq -r .tag_name)"
fi
ASSET="llama-${TAG}-bin-ubuntu-vulkan-x64.tar.gz"
URL="https://github.com/ggml-org/llama.cpp/releases/download/${TAG}/${ASSET}"
TMP="$(mktemp -d)"
echo " Lade $URL"
curl -sL -o "$TMP/v.tgz" "$URL"
mkdir -p "$VULKAN_DIR"
tar xzf "$TMP/v.tgz" -C "$TMP"
# Tarball entpackt nach .../llama-<tag>/ — Inhalt flach nach $VULKAN_DIR
cp -rf "$TMP"/llama-*/. "$VULKAN_DIR"/
rm -rf "$TMP"
test -x "$VULKAN_DIR/llama-server"
echo "==> 3. Symlink llama-server -> Vulkan-Build"
ln -sfn "$VULKAN_DIR/llama-server" /usr/local/bin/llama-server
echo "==> 4. systemd Drop-ins für llama-swap (LD_LIBRARY_PATH + Brain-Warmup)"
mkdir -p "$DROPIN_DIR"
cat > "$DROPIN_DIR/vulkan.conf" <<EOF
[Service]
Environment=LD_LIBRARY_PATH=$VULKAN_DIR
EOF
install -m 0755 "$SRC_DIR/warmup.sh" "$WARMUP_DST"
cat > "$DROPIN_DIR/warmup.conf" <<EOF
[Service]
# Nach jedem (Re)Start die brains vorladen. Das Skript detacht sich selbst (blockiert
# den Start nicht); '-' macht den Aufruf fehlertolerant (kann llama-swap nie failen lassen).
ExecStartPost=-$WARMUP_DST
EOF
echo "==> 5. Reload + Restart"
systemctl daemon-reload
systemctl restart llama-swap
sleep 3
systemctl is-active llama-swap
echo "==> Fertig. Aktive Engine:"
readlink -f /usr/local/bin/llama-server
vulkaninfo --summary 2>/dev/null | grep -m1 deviceName || true
echo "Rollback auf ROCm: ln -sfn /opt/llamacpp/llama-server /usr/local/bin/llama-server && rm $DROPIN_DIR/vulkan.conf && systemctl daemon-reload && systemctl restart llama-swap"
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env bash
# Lädt die "brains" (hermes/fast/vision) nach einem llama-swap-(Re)Start vor,
# damit die ERSTE Anfrage nicht kalt ist (llama-swap lädt sonst lazy bei Bedarf).
# Eingehängt als ExecStartPost=-/usr/local/bin/llama-swap-warmup.sh in llama-swap.
#
# Selbst-detachend: der ExecStartPost-Aufruf kehrt sofort zurück (blockiert den
# Service-Start NICHT); die eigentliche Aufwärmung läuft entkoppelt im Hintergrund.
set -u
URL="${MC_LLAMA_SWAP_URL:-http://127.0.0.1:8080}"
BRAINS="${MC_WARMUP_MODELS:-hermes fast vision}"
if [ "${1:-}" != "--inner" ]; then
setsid "$0" --inner >/dev/null 2>&1 &
exit 0
fi
# --- ab hier im entkoppelten Hintergrundprozess ---
for _ in $(seq 1 60); do # auf llama-swap warten (max ~120s)
curl -sf "$URL/v1/models" >/dev/null 2>&1 && break
sleep 2
done
for m in $BRAINS; do
curl -s -m 180 -X POST "$URL/v1/chat/completions" \
-H 'Content-Type: application/json' \
-d "{\"model\":\"$m\",\"max_tokens\":1,\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}" \
>/dev/null 2>&1 || true
done
+19 -1
View File
@@ -23,7 +23,25 @@ metadata:
## Dienste auf der AI Box (Stand 2026-06-27) ## Dienste auf der AI Box (Stand 2026-06-27)
| Dienst | Port | Beschreibung | | Dienst | Port | Beschreibung |
|---|---|---| |---|---|---|
| `llama-swap` (system) | `:8080` | Engine, ROCm/HIP | | `llama-swap` (system) | `:8080` | Engine, **Vulkan/RADV** (seit 2026-06-27, vorher ROCm/HIP) |
### Engine-Backend: Vulkan/RADV (Cutover 2026-06-27)
- **Gemessen:** RADV schlägt ROCm/HIP auf gfx1151 bei tg um **+1222 %** (fast 53→65 t/s roh), Prefill gleich.
„ROCm gewinnt Prefill" gilt hier NICHT; auch bei 32K Tiefe bleibt RADV vorn. hipBLASLt bringt nichts.
- **Setup:** Vulkan-llama.cpp (offizieller Build b9821) in `/opt/llamacpp-vulkan`; Symlink
`/usr/local/bin/llama-server` → dorthin; systemd Drop-in `/etc/systemd/system/llama-swap.service.d/vulkan.conf`
mit `LD_LIBRARY_PATH=/opt/llamacpp-vulkan`. Treiber: `mesa-vulkan-drivers` (RADV STRIX_HALO, Mesa 26.0.3).
- **ROCm-Build bleibt** unter `/opt/llamacpp` (lemonade llamacpp-rocm) als Rollback liegen.
- **Rollback:** `sudo ln -sfn /opt/llamacpp/llama-server /usr/local/bin/llama-server` + Drop-in löschen + `sudo systemctl daemon-reload && sudo systemctl restart llama-swap`.
- **ACHTUNG maintenance.py:** `_engine_update_available()` trackt noch `lemonade-sdk/llamacpp-rocm` — passt nicht
mehr zum aktiven Vulkan-Build (b9821 von ggml-org/llama.cpp). Engine-Update-Quelle anpassen.
- **Spec-Draft (2026-06-27 gefixt):** `qwen2.5-1.5b` war vocab-inkompatibel mit Qwen3.6 UND `--spec-type` fehlte
→ war inaktiv. Auch Qwen3-0.6B ist mit Qwen3.6 inkompatibel → **`fast` läuft jetzt ohne Draft** (kein
kompatibler verfügbar). **`coder` (Qwen3-Coder-Next)**: Qwen3-0.6B IST kompatibel → Draft
`/srv/models/drafts/Qwen3-0.6B-Q8_0.gguf` + `--spec-type draft-simple`, **0,75 Akzeptanz** (echter Coding-Speedup).
Alter `qwen2.5-1.5b-instruct-q4_k_m.gguf` in drafts/ ist jetzt ungenutzt (löschbar, 1,1 GB).
- **Modell-cmds (config.yaml):** llama.cpp dieser Generation braucht `--spec-type` zusätzlich zu
`--spec-draft-model`, sonst ist Spec inaktiv. Drafts müssen **exakt vocab-gleich** zum Target sein.
| `mission-control-2` (user) | `:9001` | MC2 Backend + Frontend | | `mission-control-2` (user) | `:9001` | MC2 Backend + Frontend |
| `hermes-gateway` (user) | `:8642` | Hermes Agent API | | `hermes-gateway` (user) | `:8642` | Hermes Agent API |
| `hermes-webui` (user) | `:8787` | nesquena hermes-webui | | `hermes-webui` (user) | `:8787` | nesquena hermes-webui |
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+2 -2
View File
@@ -7,8 +7,8 @@
<link rel="manifest" href="/manifest.webmanifest" /> <link rel="manifest" href="/manifest.webmanifest" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" /> <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<title>Mission Control 2.0</title> <title>Mission Control 2.0</title>
<script type="module" crossorigin src="/assets/index-D4mBflTE.js"></script> <script type="module" crossorigin src="/assets/index-DPqptZjm.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-C0dfTDFj.css"> <link rel="stylesheet" crossorigin href="/assets/index-CjZVMvhL.css">
</head> </head>
<body> <body>
<div id="root"></div> <div id="root"></div>
@@ -0,0 +1,161 @@
import { useState } from "react"
import { X, Check, Zap, AlertTriangle } from "lucide-react"
import { api, type ModelInfo, type DraftInfo } from "@/lib/api"
import { useDrafts } from "@/lib/queries"
import { fmtSize } from "@/lib/format"
import { cn } from "@/lib/utils"
/**
* Idiotensichere Speculative-Decoding-Konfiguration für EIN Modell.
* Zeigt nur VOCAB-KOMPATIBLE Drafts als wählbar; inkompatible werden gesperrt
* und mit Begründung angezeigt. So kann nie ein kaputter Draft gesetzt werden
* (der das Modell beim Laden scheitern ließe).
*/
export function SpecDraftModal({
model, onClose, onChanged,
}: { model: ModelInfo; onClose: () => void; onChanged: () => void }) {
const { data, isLoading } = useDrafts(model.gguf_path)
const [busy, setBusy] = useState<string | null>(null)
const [err, setErr] = useState("")
const tv = data?.target_vocab
const drafts = data?.drafts ?? []
const compatibles = drafts.filter((d) => d.compatible === true)
const currentFile = model.spec_draft_model
async function apply(draftPath: string | null) {
setBusy(draftPath ?? "__clear__")
setErr("")
try {
await api(`/api/models/${encodeURIComponent(model.name)}/draft`, {
method: "POST",
body: JSON.stringify({ draft_path: draftPath }),
})
onChanged()
onClose()
} catch (e: any) {
setErr(String(e?.message || e))
setBusy(null)
}
}
const vocabLabel = (v?: { pre: string | null; n_vocab: number | null } | null) =>
v ? `${v.pre ?? "?"} · ${v.n_vocab?.toLocaleString() ?? "?"} Tokens` : "—"
return (
<div className="fixed inset-0 z-50 bg-black/60 backdrop-blur-sm flex items-center justify-center p-4">
<div className="w-full max-w-lg rounded-2xl border border-border/80 bg-card p-6 shadow-2xl space-y-4 animate-in fade-in zoom-in-95 duration-200">
<div className="flex items-center justify-between border-b border-border/20 pb-2">
<h3 className="text-xs font-bold uppercase tracking-wider text-primary flex items-center gap-2">
<Zap className="h-4 w-4" /> Speculative Draft
</h3>
<button onClick={onClose} className="p-1 rounded hover:bg-accent text-muted-foreground hover:text-foreground cursor-pointer">
<X className="h-4 w-4" />
</button>
</div>
<div className="text-xs text-muted-foreground leading-relaxed">
Beschleunigt die Token-Generierung mit einem kleinen "Draft"-Modell. Voraussetzung:
der Draft muss den <strong className="text-foreground">exakt gleichen Tokenizer (Vocab)</strong> haben
wie das Modell sonst lehnt llama.cpp es ab.
</div>
{/* Ziel-Vocab */}
<div className="rounded-lg border border-border/40 bg-background/30 px-3 py-2 text-[11px] font-mono flex items-center justify-between">
<span className="text-muted-foreground">{model.name.split("/").pop()?.replace(/\.gguf$/i, "")}</span>
<span className="text-foreground">Vocab: {vocabLabel(tv)}</span>
</div>
{/* Aktueller Zustand */}
{model.spec_active && currentFile && (
<div className="rounded-lg border border-emerald-500/30 bg-emerald-500/5 px-3 py-2 flex items-center justify-between gap-2">
<span className="text-[11px] text-emerald-400 font-mono flex items-center gap-1.5 truncate">
<Check className="h-3.5 w-3.5 shrink-0" /> Aktiv: {currentFile}
</span>
<button
onClick={() => apply(null)}
disabled={busy !== null}
className="h-6 px-2.5 rounded border border-amber-500/30 bg-amber-500/5 hover:bg-amber-500/10 text-amber-400 text-[9px] font-bold uppercase shrink-0 cursor-pointer disabled:opacity-50"
>
Deaktivieren
</button>
</div>
)}
{!data?.target_exists && (
<div className="rounded-lg border border-amber-500/30 bg-amber-500/5 px-3 py-2 text-[11px] text-amber-400 flex items-center gap-2">
<AlertTriangle className="h-3.5 w-3.5 shrink-0" />
Modell-GGUF noch nicht vorhanden Spec-Draft ist nach dem Download konfigurierbar.
</div>
)}
{/* Draft-Liste */}
<div className="space-y-1.5 max-h-64 overflow-y-auto pr-1">
{isLoading ? (
<div className="text-xs text-muted-foreground py-6 text-center">Prüfe Vocab-Kompatibilität</div>
) : drafts.length === 0 ? (
<div className="text-[11px] text-muted-foreground border border-dashed border-border/50 rounded-lg p-4 text-center">
Keine Draft-Modelle in <code className="font-mono">/srv/models/drafts</code>.
Lade ein kleines same-family-Modell (z.B. Qwen3-0.6B) und lege es dort ab.
</div>
) : (
drafts.map((d: DraftInfo) => {
const isCurrent = d.filename === currentFile
const ok = d.compatible === true
return (
<div
key={d.path}
className={cn(
"rounded-lg border px-3 py-2 flex items-center justify-between gap-2 text-left",
ok ? "border-border/40 bg-background/20" : "border-border/20 bg-background/10 opacity-60",
isCurrent && "border-primary/40 bg-primary/10",
)}
>
<div className="min-w-0">
<div className="text-[11px] font-mono font-semibold text-foreground truncate">{d.filename}</div>
<div className="text-[9px] text-muted-foreground font-mono">
{fmtSize(d.size_bytes)} · Vocab: {vocabLabel(d.vocab)}
</div>
</div>
{ok ? (
isCurrent ? (
<span className="text-[9px] font-bold uppercase text-primary flex items-center gap-1 shrink-0"><Check className="h-3.5 w-3.5" /> Aktiv</span>
) : (
<button
onClick={() => apply(d.path)}
disabled={busy !== null}
className="h-7 px-3 rounded-lg border border-emerald-500/30 bg-emerald-500/5 hover:bg-emerald-500/15 text-emerald-400 text-[9px] font-bold uppercase shrink-0 cursor-pointer disabled:opacity-50"
>
Aktivieren
</button>
)
) : (
<span
className="text-[9px] font-bold uppercase text-amber-400/80 flex items-center gap-1 shrink-0"
title={d.compatible === false
? `Inkompatibel: Tokenizer/Vocab weicht ab (Draft ${d.vocab?.pre}/${d.vocab?.n_vocab} ≠ Modell ${tv?.pre}/${tv?.n_vocab}).`
: "Kompatibilität nicht prüfbar (Datei fehlt)."}
>
<AlertTriangle className="h-3.5 w-3.5" /> {d.compatible === false ? "Vocab ≠" : "n/a"}
</span>
)}
</div>
)
})
)}
</div>
{/* Hinweis, wenn Drafts da sind aber keiner kompatibel */}
{!isLoading && data?.target_exists && drafts.length > 0 && compatibles.length === 0 && (
<div className="rounded-lg border border-amber-500/30 bg-amber-500/5 px-3 py-2 text-[11px] text-amber-400 leading-relaxed">
Kein vocab-kompatibler Draft verfügbar Speculative Decoding ist für dieses Modell nicht möglich.
Es braucht einen Draft mit identischem Tokenizer (pre=<span className="font-mono">{tv?.pre}</span>,
n_vocab=<span className="font-mono">{tv?.n_vocab?.toLocaleString()}</span>).
</div>
)}
{err && <div className="text-[10px] text-red-400 font-mono">{err}</div>}
</div>
</div>
)
}
+24
View File
@@ -88,10 +88,34 @@ export interface ModelInfo {
incomplete: boolean incomplete: boolean
prompt_cache: boolean prompt_cache: boolean
spec_draft_model: string | null spec_draft_model: string | null
spec_type: string | null
spec_active: boolean
parallel_slots: number parallel_slots: number
capabilities: Capabilities capabilities: Capabilities
} }
export interface VocabFingerprint {
model: string | null
pre: string | null
n_vocab: number | null
arch: string | null
}
export interface DraftInfo {
path: string
filename: string
size_bytes: number | null
vocab: VocabFingerprint | null
compatible: boolean | null // null = nicht prüfbar (Ziel-GGUF fehlt)
}
export interface DraftsResp {
target_path: string
target_exists: boolean
target_vocab: VocabFingerprint | null
drafts: DraftInfo[]
}
export interface DiscoverModel { export interface DiscoverModel {
name: string name: string
author: string author: string
+10
View File
@@ -8,6 +8,7 @@ import {
type AgentStatus, type AgentStatus,
type ConnectResp, type ConnectResp,
type DiscoverResp, type DiscoverResp,
type DraftsResp,
type Health, type Health,
type Job, type Job,
type Memory, type Memory,
@@ -31,6 +32,7 @@ export const qk = {
agentStatus: ["agent-status"] as const, agentStatus: ["agent-status"] as const,
updates: ["updates"] as const, updates: ["updates"] as const,
discover: ["discover"] as const, discover: ["discover"] as const,
drafts: (target?: string) => ["drafts", target ?? ""] as const,
connect: (params?: string) => ["connect", params ?? ""] as const, connect: (params?: string) => ["connect", params ?? ""] as const,
memory: (q?: string, category?: string) => ["memory", q ?? "", category ?? ""] as const, memory: (q?: string, category?: string) => ["memory", q ?? "", category ?? ""] as const,
} }
@@ -70,6 +72,14 @@ export const useUpdates = (refetchInterval?: number) =>
export const useDiscover = () => export const useDiscover = () =>
useQuery({ queryKey: qk.discover, queryFn: () => api<DiscoverResp>("/api/discover") }) useQuery({ queryKey: qk.discover, queryFn: () => api<DiscoverResp>("/api/discover") })
// Verfügbare Spec-Draft-Modelle + Vocab-Kompatibilität zum Ziel-Modell (GGUF-Pfad).
export const useDrafts = (target?: string) =>
useQuery({
queryKey: qk.drafts(target),
queryFn: () => api<DraftsResp>(`/api/models/drafts?target=${encodeURIComponent(target ?? "")}`),
enabled: !!target,
})
export const useConnect = (params?: string) => export const useConnect = (params?: string) =>
useQuery({ useQuery({
queryKey: qk.connect(params), queryKey: qk.connect(params),
+19 -2
View File
@@ -7,8 +7,10 @@ export function AddModel() {
const [quants, setQuants] = useState<string[]>([]) const [quants, setQuants] = useState<string[]>([])
const [quant, setQuant] = useState("Q4_K_M") const [quant, setQuant] = useState("Q4_K_M")
const [msg, setMsg] = useState("") const [msg, setMsg] = useState("")
const [role, setRole] = useState("")
const [q, setQ] = useState("") const [q, setQ] = useState("")
const [results, setResults] = useState<{ repo: string; downloads: number }[]>([]) const [results, setResults] = useState<{ repo: string; downloads: number }[]>([])
const ROLE_OPTS = ["fast", "heavy", "coder", "vision", "scout"]
async function loadQuants(r?: string) { async function loadQuants(r?: string) {
const rr = r ?? repo const rr = r ?? repo
@@ -43,9 +45,15 @@ export function AddModel() {
try { try {
await api("/api/models/install", { await api("/api/models/install", {
method: "POST", method: "POST",
body: JSON.stringify({ repo, quant, jinja: true }), body: JSON.stringify({ repo, quant, role: role || undefined, jinja: true }),
}) })
setMsg(`Download gestartet: ${repo} (${quant}). Fortschritt wird oben angezeigt.`) setMsg(
`Download gestartet: ${repo} (${quant})${role ? `, Rolle: ${role}` : ""}. ` +
`Fortschritt oben.` +
(role === "fast" || role === "coder"
? " Speculative Draft (Vocab-geprüft) kannst du nach dem Download an der Modellkarte (»Spec«) setzen."
: "")
)
} catch (e) { } catch (e) {
setMsg(`Download-Fehler: ${e}`) setMsg(`Download-Fehler: ${e}`)
} }
@@ -78,6 +86,15 @@ export function AddModel() {
> >
{quants.map((qq) => <option key={qq} value={qq} className="bg-popover text-foreground">{qq}</option>)} {quants.map((qq) => <option key={qq} value={qq} className="bg-popover text-foreground">{qq}</option>)}
</select> </select>
<select
value={role}
onChange={(e) => setRole(e.target.value)}
title="Rolle (optional) — bestimmt Auto-Konfiguration wie parallele Slots"
className="h-9 rounded-lg border border-border/60 bg-background/40 px-3 text-xs outline-none text-foreground font-semibold"
>
<option value="" className="bg-popover text-foreground">Rolle</option>
{ROLE_OPTS.map((r) => <option key={r} value={r} className="bg-popover text-foreground">{r}</option>)}
</select>
<button <button
onClick={install} onClick={install}
className="h-9 px-4 rounded-lg bg-primary text-primary-foreground text-xs font-semibold hover:opacity-90 transition-all cursor-pointer flex items-center gap-1.5" className="h-9 px-4 rounded-lg bg-primary text-primary-foreground text-xs font-semibold hover:opacity-90 transition-all cursor-pointer flex items-center gap-1.5"
+48 -6
View File
@@ -1,12 +1,13 @@
import { useState, useRef, useCallback } from "react" import { useState, useRef, useCallback } from "react"
import { Download, Trash2, Edit3, Activity, HardDrive, X, Check, Copy } from "lucide-react" import { Download, Trash2, Edit3, Activity, HardDrive, X, Check, Copy, Zap } from "lucide-react"
import { api } from "@/lib/api" import { api, type ModelInfo } from "@/lib/api"
import { useModels, useRouting, useConnect, useUpdates, useQueryClient, qk } from "@/lib/queries" import { useModels, useRouting, useConnect, useUpdates, useQueryClient, qk } from "@/lib/queries"
import { useDialog } from "@/lib/useDialog" import { useDialog } from "@/lib/useDialog"
import { CapsChips } from "@/components/CapsChips" import { CapsChips } from "@/components/CapsChips"
import { cn } from "@/lib/utils" import { cn } from "@/lib/utils"
import { fmtSize, fmtCtx } from "@/lib/format" import { fmtSize, fmtCtx } from "@/lib/format"
import { getBrandInfo, ROLES } from "@/components/models/ModelBadges" import { getBrandInfo, ROLES } from "@/components/models/ModelBadges"
import { SpecDraftModal } from "@/components/models/SpecDraftModal"
export function Cockpit() { export function Cockpit() {
const qc = useQueryClient() const qc = useQueryClient()
@@ -26,6 +27,7 @@ export function Cockpit() {
// UI state // UI state
const [activeClient, setActiveClient] = useState<"roocode" | "cursor" | "opencode" | "zed" | "continue" | null>(null) const [activeClient, setActiveClient] = useState<"roocode" | "cursor" | "opencode" | "zed" | "continue" | null>(null)
const [activeRoleForAssign, setActiveRoleForAssign] = useState<string | null>(null) const [activeRoleForAssign, setActiveRoleForAssign] = useState<string | null>(null)
const [specModel, setSpecModel] = useState<ModelInfo | null>(null)
const [copied, setCopied] = useState(false) const [copied, setCopied] = useState(false)
const [hoveredNode, setHoveredNode] = useState<string | null>(null) const [hoveredNode, setHoveredNode] = useState<string | null>(null)
const [viewMode, setViewMode] = useState<"grid" | "list">("grid") const [viewMode, setViewMode] = useState<"grid" | "list">("grid")
@@ -742,11 +744,15 @@ export function Cockpit() {
PC PC
</span> </span>
)} )}
{m.spec_draft_model && ( {m.spec_active ? (
<span className="px-1.5 py-0.5 rounded bg-emerald-500/10 border border-emerald-500/20 text-[9px] font-mono text-emerald-400 font-bold uppercase" title={`Speculative Decoding aktiv (Draft: ${m.spec_draft_model})`}> <span className="px-1.5 py-0.5 rounded bg-emerald-500/10 border border-emerald-500/20 text-[9px] font-mono text-emerald-400 font-bold uppercase" title={`Speculative Decoding aktiv (Draft: ${m.spec_draft_model})`}>
SPEC SPEC
</span> </span>
)} ) : m.spec_draft_model ? (
<span className="px-1.5 py-0.5 rounded bg-amber-500/10 border border-amber-500/20 text-[9px] font-mono text-amber-400 font-bold uppercase" title={`Draft gesetzt (${m.spec_draft_model}), aber INAKTIV — --spec-type fehlt oder Vocab inkompatibel. Über 'Spec' neu konfigurieren.`}>
SPEC?
</span>
) : null}
{m.parallel_slots > 1 && ( {m.parallel_slots > 1 && (
<span className="px-1.5 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-[9px] font-mono text-violet-400 font-bold uppercase" title={`${m.parallel_slots} parallele Slots aktiv`}> <span className="px-1.5 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-[9px] font-mono text-violet-400 font-bold uppercase" title={`${m.parallel_slots} parallele Slots aktiv`}>
SLOTS: {m.parallel_slots} SLOTS: {m.parallel_slots}
@@ -825,6 +831,18 @@ export function Cockpit() {
> >
Ctx Ctx
</button> </button>
<button
onClick={() => setSpecModel(m)}
className={cn(
"h-7 px-2.5 rounded-lg border text-[9px] font-bold uppercase transition-colors cursor-pointer flex items-center gap-1",
m.spec_active ? "border-emerald-500/30 text-emerald-400 hover:bg-emerald-500/10"
: m.spec_draft_model ? "border-amber-500/30 text-amber-400 hover:bg-amber-500/10"
: "border-border/40 text-muted-foreground hover:bg-accent"
)}
title="Speculative Draft konfigurieren (Vocab-geprüft)"
>
<Zap className="h-3 w-3" /> Spec
</button>
<button <button
onClick={() => handleDelete(m.name)} onClick={() => handleDelete(m.name)}
className="h-7 w-7 rounded-lg border border-border/40 text-muted-foreground hover:text-red-400 hover:bg-red-500/5 flex items-center justify-center transition-colors cursor-pointer" className="h-7 w-7 rounded-lg border border-border/40 text-muted-foreground hover:text-red-400 hover:bg-red-500/5 flex items-center justify-center transition-colors cursor-pointer"
@@ -885,11 +903,15 @@ export function Cockpit() {
PC PC
</span> </span>
)} )}
{m.spec_draft_model && ( {m.spec_active ? (
<span className="px-1.5 py-0.5 rounded bg-emerald-500/10 border border-emerald-500/20 text-[8px] font-mono text-emerald-400 font-bold uppercase shrink-0" title={`Speculative Decoding aktiv (Draft: ${m.spec_draft_model})`}> <span className="px-1.5 py-0.5 rounded bg-emerald-500/10 border border-emerald-500/20 text-[8px] font-mono text-emerald-400 font-bold uppercase shrink-0" title={`Speculative Decoding aktiv (Draft: ${m.spec_draft_model})`}>
SPEC SPEC
</span> </span>
)} ) : m.spec_draft_model ? (
<span className="px-1.5 py-0.5 rounded bg-amber-500/10 border border-amber-500/20 text-[8px] font-mono text-amber-400 font-bold uppercase shrink-0" title={`Draft gesetzt (${m.spec_draft_model}), aber INAKTIV — --spec-type fehlt oder Vocab inkompatibel.`}>
SPEC?
</span>
) : null}
{m.parallel_slots > 1 && ( {m.parallel_slots > 1 && (
<span className="px-1.5 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-[8px] font-mono text-violet-400 font-bold uppercase shrink-0" title={`${m.parallel_slots} parallele Slots aktiv`}> <span className="px-1.5 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-[8px] font-mono text-violet-400 font-bold uppercase shrink-0" title={`${m.parallel_slots} parallele Slots aktiv`}>
SLOTS: {m.parallel_slots} SLOTS: {m.parallel_slots}
@@ -944,6 +966,18 @@ export function Cockpit() {
> >
Ctx Ctx
</button> </button>
<button
onClick={() => setSpecModel(m)}
className={cn(
"h-7 px-2.5 rounded-lg border text-[9px] font-bold uppercase transition-all cursor-pointer flex items-center gap-1",
m.spec_active ? "border-emerald-500/30 text-emerald-400 hover:bg-emerald-500/10"
: m.spec_draft_model ? "border-amber-500/30 text-amber-400 hover:bg-amber-500/10"
: "border-border/40 text-muted-foreground hover:bg-accent"
)}
title="Speculative Draft konfigurieren (Vocab-geprüft)"
>
<Zap className="h-3 w-3" /> Spec
</button>
<button <button
onClick={() => handleDelete(m.name)} onClick={() => handleDelete(m.name)}
className="h-7 w-7 rounded-lg flex items-center justify-center border border-border/40 text-muted-foreground hover:text-red-400 hover:bg-red-500/5 transition-all cursor-pointer" className="h-7 w-7 rounded-lg flex items-center justify-center border border-border/40 text-muted-foreground hover:text-red-400 hover:bg-red-500/5 transition-all cursor-pointer"
@@ -1016,6 +1050,14 @@ export function Cockpit() {
</div> </div>
)} )}
{specModel && (
<SpecDraftModal
model={specModel}
onClose={() => setSpecModel(null)}
onChanged={reload}
/>
)}
{dialogElement} {dialogElement}
</div> </div>
) )
-65
View File
@@ -1,65 +0,0 @@
# LiteLLM-Gateway-Config für Mission Control 2.0
# ----------------------------------------------------------------------------
# EIN OpenAI-Endpunkt für ALLE Konsumenten (Hermes + Vibe-Coding-IDEs).
# Hinter dem Gateway sitzt llama-swap (:8080), das die llama-swap-Rollen-Aliase
# (fast/heavy/vision/coder) bei Bedarf lädt — fast+heavy sind via llama-swap
# `groups` (swap:false) ko-resident, daher Delegation ohne Nachlade-Latenz.
#
# Wird von Mission Control verwaltet (services/gateway.py). Start auf der Box:
# litellm --config /opt/mission-control-2/gateway/config.yaml --port 4000
#
# HINWEIS (auf der Box verifizieren): Der KOMPLEXITÄTS-Auto-Router ("model: auto"
# wählt pro Anfrage schnell↔schwer nach Schwierigkeit) ist LiteLLM-versionsabhängig.
# Baseline unten: "auto" == fast, eskaliert per Fallback auf heavy (bei Fehler/
# Kontext-Overflow). Den echten Complexity-Router (docs.litellm.ai/docs/proxy/
# auto_routing) in Phase „Routing-Feinschliff" gegen die installierte Version setzen.
model_list:
# Schnelles Alltags-Hirn (llama-swap-Alias "fast")
- model_name: fast
litellm_params:
model: openai/fast
api_base: http://127.0.0.1:8080/v1
api_key: sk-noauth
# Schweres Reasoning-Hirn (llama-swap-Alias "heavy")
- model_name: heavy
litellm_params:
model: openai/heavy
api_base: http://127.0.0.1:8080/v1
api_key: sk-noauth
# Vision (llama-swap-Alias "vision")
- model_name: vision
litellm_params:
model: openai/vision
api_base: http://127.0.0.1:8080/v1
api_key: sk-noauth
# Coder (llama-swap-Alias "coder")
- model_name: coder
litellm_params:
model: openai/coder
api_base: http://127.0.0.1:8080/v1
api_key: sk-noauth
# "auto" — Baseline: == fast, eskaliert per Fallback auf heavy.
- model_name: auto
litellm_params:
model: openai/fast
api_base: http://127.0.0.1:8080/v1
api_key: sk-noauth
litellm_settings:
# Bei Fehler/Kontext-Overflow von "auto"/"fast" auf "heavy" eskalieren.
fallbacks:
- auto: ["heavy"]
- fast: ["heavy"]
context_window_fallbacks:
- auto: ["heavy"]
- fast: ["heavy"]
num_retries: 1
request_timeout: 600
router_settings:
routing_strategy: simple-shuffle