Feat: Vulkan/RADV-Engine + vocab-gepruefte Spec-Drafts + Provisioning/Sync
Engine-Cutover ROCm/HIP -> Vulkan/RADV (gfx1151): +12-22% tg auf MoE (llama-bench
verifiziert, fast 53->65 t/s). ROCm-Build bleibt als Rollback unter /opt/llamacpp.
- Backend: vocab-aware Speculative Decoding. services/gguf_meta.py liest den
Tokenizer-Fingerprint (model/pre/n_vocab) direkt aus dem GGUF-Header (ohne Modell-Load);
register_model + migrate_config haengen nur VOCAB-KOMPATIBLE Drafts an (inkl. --spec-type,
das in dieser llama.cpp-Generation noetig ist). Neue Endpoints /api/models/drafts + /{id}/draft.
- Frontend: idiotensichere Spec-Draft-UI (SpecDraftModal) - nur kompatible Drafts waehlbar,
inkompatible gesperrt mit Begruendung; SPEC/SPEC?-Badge nach echtem Aktiv-Status; Rolle in AddModel.
- maintenance.py: Engine-Update-Quelle -> ggml-org/llama.cpp (Build-Nummer-Vergleich),
ENGINE_PATH=/opt/llamacpp-vulkan.
- Startup-Warmup der brains (deploy/warmup.sh, self-detaching ExecStartPost) + deploy/provision-engine.sh.
- Cleanup: tote LiteLLM gateway/config.yaml + alle Referenzen (config.py/backup.py/backup.sh) entfernt;
README + docs/memory aktualisiert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -3,7 +3,8 @@
|
|||||||
Komponierbarer Local-AI-Stack für den Bosgame M5. Greenfield-Neuaufbau —
|
Komponierbarer Local-AI-Stack für den Bosgame M5. Greenfield-Neuaufbau —
|
||||||
siehe Architektur-Plan (`docs/` bzw. der genehmigte Plan).
|
siehe Architektur-Plan (`docs/` bzw. der genehmigte Plan).
|
||||||
|
|
||||||
**Schichten:** Engine (llama-swap) · Routing-Gateway (LiteLLM, `model: auto`) ·
|
**Schichten:** Engine (llama-swap, **Vulkan/RADV** auf Strix Halo) · **Builtin-Routing-Gateway**
|
||||||
|
in MC2 (`model: auto`, kein externer LiteLLM-Dienst — scheitert auf Python 3.14) ·
|
||||||
Mission Control 2.0 (FastAPI + React/shadcn) · Hermes Agent + hermes-webui ·
|
Mission Control 2.0 (FastAPI + React/shadcn) · Hermes Agent + hermes-webui ·
|
||||||
Shared Memory (SQLite via MCP). Jede Schicht hinter stabilem Vertrag austauschbar.
|
Shared Memory (SQLite via MCP). Jede Schicht hinter stabilem Vertrag austauschbar.
|
||||||
|
|
||||||
@@ -13,8 +14,8 @@ Fortschritt & Resume-Guide: siehe [`docs/STATUS.md`](docs/STATUS.md).
|
|||||||
|
|
||||||
- **Phase 0** — FastAPI-Skeleton + React/shadcn-Shell (Cmd+K, Dark, PWA).
|
- **Phase 0** — FastAPI-Skeleton + React/shadcn-Shell (Cmd+K, Dark, PWA).
|
||||||
- **Phase 1** — Compute-Module (fit/caps/sources, portiert), **Discover** (live HF + Fit + Caps),
|
- **Phase 1** — Compute-Module (fit/caps/sources, portiert), **Discover** (live HF + Fit + Caps),
|
||||||
**Engine-Write** (register + `groups`/Ko-Residenz), **LiteLLM-Gateway** (Config + Service,
|
**Engine-Write** (register + `groups`/Ko-Residenz + vocab-geprüfte Spec-Drafts),
|
||||||
`model: auto` + Fallbacks), Frontend **Modelle & Routing** (Caps-Chips, Fit, Discover, Routing-View).
|
**Builtin-Gateway** (`model: auto` + Fallbacks), Frontend **Modelle & Routing** (Caps-Chips, Fit, Discover, Routing-View).
|
||||||
|
|
||||||
- **Phase 2** — System-Status (CPU/RAM/GPU/Disk), Wartung (restart/self-update, sudo-frei),
|
- **Phase 2** — System-Status (CPU/RAM/GPU/Disk), Wartung (restart/self-update, sudo-frei),
|
||||||
**Connect** (saubere IDE-Snippets → Gateway `model:auto`, LAN-IP-Override).
|
**Connect** (saubere IDE-Snippets → Gateway `model:auto`, LAN-IP-Override).
|
||||||
@@ -59,5 +60,9 @@ cd frontend && npm run build # → frontend/dist
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `MC_LLAMA_SWAP_URL` | `http://127.0.0.1:8080` | Engine |
|
| `MC_LLAMA_SWAP_URL` | `http://127.0.0.1:8080` | Engine |
|
||||||
| `MC_CONFIG_PATH` | `/etc/llama-swap/config.yaml` | llama-swap Config |
|
| `MC_CONFIG_PATH` | `/etc/llama-swap/config.yaml` | llama-swap Config |
|
||||||
| `MC_GATEWAY_URL` | `http://127.0.0.1:4000` | LiteLLM-Gateway |
|
| `MC_GATEWAY_URL` | `http://127.0.0.1:$MC_PORT` | Builtin-Gateway (Teil von MC2, kein externer Dienst) |
|
||||||
| `MC_PORT` | `9000` | MC-Backend-Port |
|
| `MC_PORT` | `9000` | MC-Backend-Port |
|
||||||
|
| `MC_ENGINE_PATH` | `/opt/llamacpp-vulkan` | Aktive Engine-Binary (Vulkan-Build) |
|
||||||
|
| `MC_ENGINE_REPO` | `ggml-org/llama.cpp` | Quelle für Engine-Update-Check |
|
||||||
|
| `MC_DRAFTS_DIR` | `$MODELS/drafts` | Spec-Draft-Modelle (vocab-geprüft) |
|
||||||
|
| `MC_SPEC_TYPE` | `draft-simple` | Speculative-Decoding-Typ (llama.cpp) |
|
||||||
|
|||||||
+14
-7
@@ -33,17 +33,24 @@ CMD_TEMPLATE = os.environ.get("MC_CMD_TEMPLATE", _DEFAULT_CMD_TEMPLATE)
|
|||||||
if "{model}" not in CMD_TEMPLATE:
|
if "{model}" not in CMD_TEMPLATE:
|
||||||
CMD_TEMPLATE = _DEFAULT_CMD_TEMPLATE
|
CMD_TEMPLATE = _DEFAULT_CMD_TEMPLATE
|
||||||
DEFAULT_TTL = int(os.environ.get("MC_DEFAULT_TTL", "300"))
|
DEFAULT_TTL = int(os.environ.get("MC_DEFAULT_TTL", "300"))
|
||||||
# Draft-Modell für Speculative Decoding (nur fast/coder, wenn vorhanden). Eine
|
# Verzeichnis mit Draft-Modellen für Speculative Decoding. Beim Hinzufügen eines
|
||||||
# Quelle der Wahrheit für llamaswap.register_model + migrate_config.
|
# fast/coder-Modells wird hieraus automatisch ein **vocab-kompatibler** Draft gewählt
|
||||||
SPEC_DRAFT_MODEL_PATH = os.environ.get(
|
# (Vocab-Check via services.gguf_meta; ein inkompatibler Draft lässt llama.cpp scheitern).
|
||||||
"MC_SPEC_DRAFT_MODEL", f"{MODELS_DIR.as_posix()}/drafts/qwen2.5-1.5b-instruct-q4_k_m.gguf")
|
DRAFTS_DIR = Path(os.environ.get("MC_DRAFTS_DIR", str(MODELS_DIR / "drafts")))
|
||||||
|
# Optionaler expliziter Default-Draft (leer = Auto-Erkennung aus DRAFTS_DIR). Wird nur
|
||||||
|
# verwendet, wenn er zum Ziel-Modell vocab-kompatibel ist. (Früher fix qwen2.5 → entfernt,
|
||||||
|
# weil das mit neueren Vocabs wie Qwen3.6 inkompatibel ist und Spec stillschweigend brach.)
|
||||||
|
SPEC_DRAFT_MODEL_PATH = os.environ.get("MC_SPEC_DRAFT_MODEL", "")
|
||||||
|
# Speculative-Decoding-Typ (llama.cpp dieser Generation braucht --spec-type zusätzlich
|
||||||
|
# zu --spec-draft-model, sonst ist Spec inaktiv).
|
||||||
|
SPEC_TYPE = os.environ.get("MC_SPEC_TYPE", "draft-simple")
|
||||||
# Env für HuggingFace-Downloads: XET deaktivieren (Hänger bei ~6 MB, siehe v1-Gotcha).
|
# Env für HuggingFace-Downloads: XET deaktivieren (Hänger bei ~6 MB, siehe v1-Gotcha).
|
||||||
HF_DOWNLOAD_ENV = {"HF_HUB_DISABLE_XET": "1"}
|
HF_DOWNLOAD_ENV = {"HF_HUB_DISABLE_XET": "1"}
|
||||||
|
|
||||||
# --- Routing-Gateway (LiteLLM, model: auto) ----------------------------------
|
# --- Routing-Gateway (builtin in MC2, model: auto) ---------------------------
|
||||||
|
# MC2 IST der Gateway (services/gateway.py + routers/gateway_proxy.py). KEIN externer
|
||||||
|
# LiteLLM-Dienst (scheitert auf Python 3.14). Daher keine Gateway-Config-Datei mehr.
|
||||||
GATEWAY_URL = os.environ.get("MC_GATEWAY_URL", f"http://127.0.0.1:{os.environ.get('MC_PORT', '9000')}").rstrip("/")
|
GATEWAY_URL = os.environ.get("MC_GATEWAY_URL", f"http://127.0.0.1:{os.environ.get('MC_PORT', '9000')}").rstrip("/")
|
||||||
GATEWAY_CONFIG_PATH = Path(os.environ.get(
|
|
||||||
"MC_GATEWAY_CONFIG", str(Path(__file__).resolve().parent.parent / "gateway" / "config.yaml")))
|
|
||||||
|
|
||||||
# --- Hermes Agent (eigener Dienst auf der Box) -------------------------------
|
# --- Hermes Agent (eigener Dienst auf der Box) -------------------------------
|
||||||
# Gateway (OpenAI-API des Agenten) + standalone Web-UI (nesquena/hermes-webui).
|
# Gateway (OpenAI-API des Agenten) + standalone Web-UI (nesquena/hermes-webui).
|
||||||
|
|||||||
@@ -4,8 +4,8 @@ from pathlib import Path
|
|||||||
# Add backend directory to sys.path so we can import services
|
# Add backend directory to sys.path so we can import services
|
||||||
sys.path.append(str(Path(__file__).resolve().parent))
|
sys.path.append(str(Path(__file__).resolve().parent))
|
||||||
|
|
||||||
from services.llamaswap import read_config, write_config
|
from services.llamaswap import read_config, write_config, spec_draft_flags, _PATH_RE
|
||||||
from config import CONFIG_PATH, SPEC_DRAFT_MODEL_PATH
|
from config import CONFIG_PATH
|
||||||
|
|
||||||
def migrate():
|
def migrate():
|
||||||
print(f"Reading config from {CONFIG_PATH}...")
|
print(f"Reading config from {CONFIG_PATH}...")
|
||||||
@@ -16,8 +16,6 @@ def migrate():
|
|||||||
cfg = read_config()
|
cfg = read_config()
|
||||||
models = cfg.get("models", {})
|
models = cfg.get("models", {})
|
||||||
|
|
||||||
draft_path = SPEC_DRAFT_MODEL_PATH
|
|
||||||
|
|
||||||
for name, spec in models.items():
|
for name, spec in models.items():
|
||||||
if not isinstance(spec, dict):
|
if not isinstance(spec, dict):
|
||||||
continue
|
continue
|
||||||
@@ -36,12 +34,15 @@ def migrate():
|
|||||||
aliases = spec.get("aliases", [])
|
aliases = spec.get("aliases", [])
|
||||||
role = aliases[0] if aliases else None
|
role = aliases[0] if aliases else None
|
||||||
|
|
||||||
# 3. Add parallel and speculative decoding for fast and coder
|
# 3. Add parallel + (nur vocab-kompatibles) Speculative Decoding für fast/coder.
|
||||||
|
# spec_draft_flags() prüft die Vocab-Kompatibilität und hängt --spec-type an;
|
||||||
|
# ein inkompatibler Draft (z.B. qwen2.5 ↔ Qwen3.6) wird NICHT gesetzt.
|
||||||
if role in ("fast", "coder"):
|
if role in ("fast", "coder"):
|
||||||
if "--parallel" not in cmd:
|
if "--parallel" not in cmd:
|
||||||
cmd = cmd.strip() + " --parallel 2"
|
cmd = cmd.strip() + " --parallel 2"
|
||||||
if "--spec-draft-model" not in cmd:
|
if "--spec-draft-model" not in cmd:
|
||||||
cmd = cmd.strip() + f" --spec-draft-model {draft_path}"
|
target = mt.group(1) if (mt := _PATH_RE.search(cmd)) else ""
|
||||||
|
cmd = cmd.strip() + spec_draft_flags(target)
|
||||||
|
|
||||||
# Update cmd
|
# Update cmd
|
||||||
from ruamel.yaml.scalarstring import LiteralScalarString
|
from ruamel.yaml.scalarstring import LiteralScalarString
|
||||||
|
|||||||
@@ -158,6 +158,30 @@ def set_model_ctx(model_id: str, body: CtxReq) -> dict:
|
|||||||
return {"ok": True}
|
return {"ok": True}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/models/drafts")
|
||||||
|
def list_drafts(target: str = "") -> dict:
|
||||||
|
"""Verfügbare Draft-Modelle + ihre Vocab-Kompatibilität zum Ziel-Modell
|
||||||
|
(target = GGUF-Pfad). Basis für die idiotensichere Spec-Draft-Auswahl im UI."""
|
||||||
|
return llamaswap.drafts_for(target)
|
||||||
|
|
||||||
|
|
||||||
|
class DraftReq(BaseModel):
|
||||||
|
draft_path: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/models/{model_id}/draft")
|
||||||
|
def set_model_draft(model_id: str, body: DraftReq) -> dict:
|
||||||
|
"""Setzt/entfernt den Speculative-Decoding-Draft eines Modells. Inkompatible
|
||||||
|
(oder nicht prüfbare) Drafts werden serverseitig abgelehnt."""
|
||||||
|
try:
|
||||||
|
res = llamaswap.set_spec_draft(model_id, body.draft_path)
|
||||||
|
except PermissionError as exc:
|
||||||
|
raise HTTPException(500, str(exc))
|
||||||
|
if not res["ok"]:
|
||||||
|
raise HTTPException(400 if "kompatib" in res["reason"].lower() else 404, res["reason"])
|
||||||
|
return res
|
||||||
|
|
||||||
|
|
||||||
@router.post("/models/unload")
|
@router.post("/models/unload")
|
||||||
def unload_all_models() -> dict:
|
def unload_all_models() -> dict:
|
||||||
import httpx
|
import httpx
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import shutil
|
|||||||
import time
|
import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from config import CONFIG_PATH, GATEWAY_CONFIG_PATH, MEMORY_DB, MODELS_DIR
|
from config import CONFIG_PATH, MEMORY_DB, MODELS_DIR
|
||||||
|
|
||||||
BACKUP_DIR = Path(MODELS_DIR) / "mc2-backups"
|
BACKUP_DIR = Path(MODELS_DIR) / "mc2-backups"
|
||||||
RETAIN = 7
|
RETAIN = 7
|
||||||
@@ -31,7 +31,7 @@ def backup_now() -> dict:
|
|||||||
dst.mkdir(parents=True, exist_ok=True)
|
dst.mkdir(parents=True, exist_ok=True)
|
||||||
saved = []
|
saved = []
|
||||||
for src in (MEMORY_DB, Path(str(MEMORY_DB) + "-wal"), Path(str(MEMORY_DB) + "-shm"),
|
for src in (MEMORY_DB, Path(str(MEMORY_DB) + "-wal"), Path(str(MEMORY_DB) + "-shm"),
|
||||||
CONFIG_PATH, GATEWAY_CONFIG_PATH):
|
CONFIG_PATH):
|
||||||
if (name := _safe_copy(src, dst)):
|
if (name := _safe_copy(src, dst)):
|
||||||
saved.append(name)
|
saved.append(name)
|
||||||
# Aufräumen: nur die letzten RETAIN Snapshots behalten.
|
# Aufräumen: nur die letzten RETAIN Snapshots behalten.
|
||||||
|
|||||||
@@ -0,0 +1,148 @@
|
|||||||
|
"""
|
||||||
|
GGUF-Tokenizer-Fingerprint — liest die Tokenizer-Identität direkt aus dem
|
||||||
|
GGUF-Header (ohne das Modell zu laden), um zu entscheiden, ob ein Draft-Modell
|
||||||
|
**vocab-kompatibel** mit einem Ziel-Modell ist (Voraussetzung für Speculative
|
||||||
|
Decoding in llama.cpp — sonst: "draft model vocab type must match target").
|
||||||
|
|
||||||
|
Wir lesen nur die Metadaten-KV-Sektion am Dateianfang und brechen ab, sobald
|
||||||
|
`tokenizer.ggml.tokens` erreicht ist (dessen Länge = n_vocab). model+pre+n_vocab
|
||||||
|
identifizieren den Tokenizer eindeutig genug, um die in der Praxis relevanten
|
||||||
|
Fälle zu unterscheiden (Qwen2.5 vs Qwen3 vs Qwen3.6 etc.). Die llama.cpp-Prüfung
|
||||||
|
beim Laden bleibt der letzte Schiedsrichter.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import struct
|
||||||
|
from functools import lru_cache
|
||||||
|
|
||||||
|
# GGUF value types (https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)
|
||||||
|
_T_UINT8, _T_INT8, _T_UINT16, _T_INT16, _T_UINT32, _T_INT32, _T_FLOAT32, \
|
||||||
|
_T_BOOL, _T_STRING, _T_ARRAY, _T_UINT64, _T_INT64, _T_FLOAT64 = range(13)
|
||||||
|
|
||||||
|
_SCALAR_FMT = {
|
||||||
|
_T_UINT8: "<B", _T_INT8: "<b", _T_UINT16: "<H", _T_INT16: "<h",
|
||||||
|
_T_UINT32: "<I", _T_INT32: "<i", _T_FLOAT32: "<f", _T_BOOL: "<?",
|
||||||
|
_T_UINT64: "<Q", _T_INT64: "<q", _T_FLOAT64: "<d",
|
||||||
|
}
|
||||||
|
_SCALAR_SIZE = {t: struct.calcsize(f) for t, f in _SCALAR_FMT.items()}
|
||||||
|
|
||||||
|
_WANT_STRINGS = {"tokenizer.ggml.model", "tokenizer.ggml.pre", "general.architecture"}
|
||||||
|
|
||||||
|
|
||||||
|
class _Reader:
|
||||||
|
def __init__(self, f):
|
||||||
|
self.f = f
|
||||||
|
|
||||||
|
def read(self, n: int) -> bytes:
|
||||||
|
b = self.f.read(n)
|
||||||
|
if len(b) != n:
|
||||||
|
raise EOFError("unerwartetes Dateiende beim GGUF-Parsen")
|
||||||
|
return b
|
||||||
|
|
||||||
|
def u32(self) -> int:
|
||||||
|
return struct.unpack("<I", self.read(4))[0]
|
||||||
|
|
||||||
|
def u64(self) -> int:
|
||||||
|
return struct.unpack("<Q", self.read(8))[0]
|
||||||
|
|
||||||
|
def gstr(self) -> str:
|
||||||
|
n = self.u64()
|
||||||
|
return self.read(n).decode("utf-8", "replace")
|
||||||
|
|
||||||
|
def skip_value(self, vtype: int) -> None:
|
||||||
|
"""Liest einen Wert und verwirft ihn (um den Datei-Pointer korrekt
|
||||||
|
weiterzuschieben). Arrays werden elementweise konsumiert."""
|
||||||
|
if vtype == _T_STRING:
|
||||||
|
self.f.seek(self.u64(), 1)
|
||||||
|
elif vtype in _SCALAR_SIZE:
|
||||||
|
self.f.seek(_SCALAR_SIZE[vtype], 1)
|
||||||
|
elif vtype == _T_ARRAY:
|
||||||
|
etype = self.u32()
|
||||||
|
count = self.u64()
|
||||||
|
if etype == _T_STRING:
|
||||||
|
for _ in range(count):
|
||||||
|
self.f.seek(self.u64(), 1)
|
||||||
|
elif etype in _SCALAR_SIZE:
|
||||||
|
self.f.seek(_SCALAR_SIZE[etype] * count, 1)
|
||||||
|
else:
|
||||||
|
raise ValueError(f"unbekannter Array-Elementtyp {etype}")
|
||||||
|
else:
|
||||||
|
raise ValueError(f"unbekannter GGUF-Wertetyp {vtype}")
|
||||||
|
|
||||||
|
|
||||||
|
def _read_fingerprint(path: str) -> dict | None:
|
||||||
|
"""Liest model/pre/n_vocab aus dem GGUF-Header. None bei Fehler/kein GGUF."""
|
||||||
|
try:
|
||||||
|
with open(path, "rb") as fh:
|
||||||
|
r = _Reader(fh)
|
||||||
|
if r.read(4) != b"GGUF":
|
||||||
|
return None
|
||||||
|
r.u32() # version
|
||||||
|
r.u64() # tensor_count
|
||||||
|
kv_count = r.u64()
|
||||||
|
fp: dict = {"model": None, "pre": None, "arch": None, "n_vocab": None}
|
||||||
|
for _ in range(kv_count):
|
||||||
|
key = r.gstr()
|
||||||
|
vtype = r.u32()
|
||||||
|
if key == "tokenizer.ggml.tokens" and vtype == _T_ARRAY:
|
||||||
|
etype = r.u32()
|
||||||
|
fp["n_vocab"] = r.u64()
|
||||||
|
# Wir haben alles (model/pre kommen vor tokens) → abbrechen.
|
||||||
|
if etype != _T_STRING:
|
||||||
|
return None
|
||||||
|
break
|
||||||
|
if key in _WANT_STRINGS and vtype == _T_STRING:
|
||||||
|
val = r.gstr()
|
||||||
|
if key == "tokenizer.ggml.model":
|
||||||
|
fp["model"] = val
|
||||||
|
elif key == "tokenizer.ggml.pre":
|
||||||
|
fp["pre"] = val
|
||||||
|
else:
|
||||||
|
fp["arch"] = val
|
||||||
|
else:
|
||||||
|
r.skip_value(vtype)
|
||||||
|
if fp["model"] is None and fp["n_vocab"] is None:
|
||||||
|
return None
|
||||||
|
return fp
|
||||||
|
except (OSError, EOFError, ValueError, struct.error):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=256)
|
||||||
|
def _cached(path: str, mtime: float, size: int) -> tuple | None:
|
||||||
|
fp = _read_fingerprint(path)
|
||||||
|
if fp is None:
|
||||||
|
return None
|
||||||
|
return (fp.get("model"), fp.get("pre"), fp.get("n_vocab"), fp.get("arch"))
|
||||||
|
|
||||||
|
|
||||||
|
def fingerprint(path: str) -> dict | None:
|
||||||
|
"""Tokenizer-Fingerprint eines GGUF (gecacht nach Pfad+mtime+size).
|
||||||
|
Returns dict(model, pre, n_vocab, arch) oder None wenn nicht lesbar."""
|
||||||
|
import os
|
||||||
|
try:
|
||||||
|
st = os.stat(path)
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
t = _cached(path, st.st_mtime, st.st_size)
|
||||||
|
if t is None:
|
||||||
|
return None
|
||||||
|
return {"model": t[0], "pre": t[1], "n_vocab": t[2], "arch": t[3]}
|
||||||
|
|
||||||
|
|
||||||
|
def vocab_key(path: str) -> tuple | None:
|
||||||
|
"""Vergleichsschlüssel für Vocab-Kompatibilität: (model, pre, n_vocab).
|
||||||
|
Genau diese Identität verlangt llama.cpp für Speculative Decoding."""
|
||||||
|
fp = fingerprint(path)
|
||||||
|
if not fp or fp["n_vocab"] is None:
|
||||||
|
return None
|
||||||
|
return (fp["model"], fp["pre"], fp["n_vocab"])
|
||||||
|
|
||||||
|
|
||||||
|
def compatible(target_path: str, draft_path: str) -> bool | None:
|
||||||
|
"""True/False ob draft vocab-kompatibel zum target ist. None = unbestimmbar
|
||||||
|
(eine Datei nicht lesbar) → UI behandelt das als 'nicht bestätigt'."""
|
||||||
|
a = vocab_key(target_path)
|
||||||
|
b = vocab_key(draft_path)
|
||||||
|
if a is None or b is None:
|
||||||
|
return None
|
||||||
|
return a == b
|
||||||
@@ -13,7 +13,10 @@ import re
|
|||||||
import httpx
|
import httpx
|
||||||
from ruamel.yaml.scalarstring import LiteralScalarString
|
from ruamel.yaml.scalarstring import LiteralScalarString
|
||||||
|
|
||||||
from config import CMD_TEMPLATE, CONFIG_PATH, DEFAULT_TTL, LLAMA_SWAP_URL, SPEC_DRAFT_MODEL_PATH
|
from config import (
|
||||||
|
CMD_TEMPLATE, CONFIG_PATH, DEFAULT_TTL, DRAFTS_DIR, LLAMA_SWAP_URL,
|
||||||
|
SPEC_DRAFT_MODEL_PATH, SPEC_TYPE,
|
||||||
|
)
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -66,6 +69,11 @@ def _parse_model(name: str, spec: dict) -> dict:
|
|||||||
m_draft = re.search(r"--spec-draft-model\s+([^\s]+)", cmd)
|
m_draft = re.search(r"--spec-draft-model\s+([^\s]+)", cmd)
|
||||||
if m_draft:
|
if m_draft:
|
||||||
spec_draft = os.path.basename(m_draft.group(1).replace("'", "").replace('"', ""))
|
spec_draft = os.path.basename(m_draft.group(1).replace("'", "").replace('"', ""))
|
||||||
|
spec_type = None
|
||||||
|
if (m_st := re.search(r"--spec-type\s+([^\s]+)", cmd)):
|
||||||
|
spec_type = m_st.group(1)
|
||||||
|
# Spec ist nur AKTIV, wenn BEIDES gesetzt ist (--spec-draft-model UND --spec-type).
|
||||||
|
spec_active = bool(spec_draft and spec_type)
|
||||||
parallel_match = re.search(r"--parallel\s+(\d+)", cmd)
|
parallel_match = re.search(r"--parallel\s+(\d+)", cmd)
|
||||||
parallel_slots = int(parallel_match.group(1)) if parallel_match else 1
|
parallel_slots = int(parallel_match.group(1)) if parallel_match else 1
|
||||||
|
|
||||||
@@ -85,6 +93,8 @@ def _parse_model(name: str, spec: dict) -> dict:
|
|||||||
"incomplete": not path,
|
"incomplete": not path,
|
||||||
"prompt_cache": prompt_cache,
|
"prompt_cache": prompt_cache,
|
||||||
"spec_draft_model": spec_draft,
|
"spec_draft_model": spec_draft,
|
||||||
|
"spec_type": spec_type,
|
||||||
|
"spec_active": spec_active,
|
||||||
"parallel_slots": parallel_slots,
|
"parallel_slots": parallel_slots,
|
||||||
"capabilities": capabilities(
|
"capabilities": capabilities(
|
||||||
name=filename or name, cmd=cmd,
|
name=filename or name, cmd=cmd,
|
||||||
@@ -191,8 +201,10 @@ def register_model(model_path: str, role: str | None = None, ctx: int = 8192,
|
|||||||
if role_lower in ("fast", "coder"):
|
if role_lower in ("fast", "coder"):
|
||||||
if "--parallel" not in cmd:
|
if "--parallel" not in cmd:
|
||||||
cmd += " --parallel 2"
|
cmd += " --parallel 2"
|
||||||
if os.path.exists(SPEC_DRAFT_MODEL_PATH) and "--spec-draft-model" not in cmd:
|
# Nur einen VOCAB-KOMPATIBLEN Draft anhängen (sonst scheitert llama.cpp).
|
||||||
cmd += f" --spec-draft-model {SPEC_DRAFT_MODEL_PATH}"
|
# Bei frischem Install existiert die GGUF noch nicht → kein Draft (später im UI setzbar).
|
||||||
|
if "--spec-draft-model" not in cmd:
|
||||||
|
cmd += spec_draft_flags(model_path)
|
||||||
|
|
||||||
cfg.setdefault("models", {})[model_id] = {
|
cfg.setdefault("models", {})[model_id] = {
|
||||||
"cmd": LiteralScalarString(cmd + "\n"),
|
"cmd": LiteralScalarString(cmd + "\n"),
|
||||||
@@ -203,6 +215,101 @@ def register_model(model_path: str, role: str | None = None, ctx: int = 8192,
|
|||||||
return model_id
|
return model_id
|
||||||
|
|
||||||
|
|
||||||
|
# --- Speculative-Draft / Vocab-Kompatibilität --------------------------------
|
||||||
|
def list_drafts() -> list[dict]:
|
||||||
|
"""Alle Draft-GGUFs in DRAFTS_DIR mit Tokenizer-Fingerprint."""
|
||||||
|
from services import gguf_meta
|
||||||
|
out = []
|
||||||
|
if DRAFTS_DIR.is_dir():
|
||||||
|
for p in sorted(DRAFTS_DIR.glob("*.gguf")):
|
||||||
|
out.append({
|
||||||
|
"path": str(p), "filename": p.name,
|
||||||
|
"size_bytes": p.stat().st_size if p.exists() else None,
|
||||||
|
"vocab": gguf_meta.fingerprint(str(p)),
|
||||||
|
})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def find_compatible_draft(target_path: str) -> str | None:
|
||||||
|
"""Pfad eines vocab-kompatiblen Drafts für target_path, oder None.
|
||||||
|
Bevorzugt MC_SPEC_DRAFT_MODEL (falls gesetzt+kompatibel), sonst der erste
|
||||||
|
kompatible Draft in DRAFTS_DIR. None auch, wenn target (noch) fehlt (dann
|
||||||
|
nicht verifizierbar → bewusst KEIN Draft anhängen)."""
|
||||||
|
if not target_path or not os.path.exists(target_path):
|
||||||
|
return None
|
||||||
|
from services import gguf_meta
|
||||||
|
candidates: list[str] = []
|
||||||
|
if SPEC_DRAFT_MODEL_PATH and os.path.exists(SPEC_DRAFT_MODEL_PATH):
|
||||||
|
candidates.append(SPEC_DRAFT_MODEL_PATH)
|
||||||
|
for d in list_drafts():
|
||||||
|
if d["path"] not in candidates:
|
||||||
|
candidates.append(d["path"])
|
||||||
|
for c in candidates:
|
||||||
|
if gguf_meta.compatible(target_path, c) is True:
|
||||||
|
return c
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def spec_draft_flags(target_path: str) -> str:
|
||||||
|
"""llama-server-Flags für Speculative Decoding (Draft + --spec-type), oder ''
|
||||||
|
wenn kein kompatibler Draft existiert. Beides nötig, sonst ist Spec inaktiv."""
|
||||||
|
d = find_compatible_draft(target_path)
|
||||||
|
return f" --spec-draft-model {d} --spec-type {SPEC_TYPE}" if d else ""
|
||||||
|
|
||||||
|
|
||||||
|
def drafts_for(target_path: str) -> dict:
|
||||||
|
"""Für die UI: alle Drafts + ihre Kompatibilität zum Ziel-Modell.
|
||||||
|
compatible=None heißt 'nicht prüfbar' (Ziel- oder Draft-GGUF fehlt)."""
|
||||||
|
from services import gguf_meta
|
||||||
|
exists = bool(target_path and os.path.exists(target_path))
|
||||||
|
drafts = list_drafts()
|
||||||
|
for d in drafts:
|
||||||
|
d["compatible"] = gguf_meta.compatible(target_path, d["path"]) if exists else None
|
||||||
|
return {
|
||||||
|
"target_path": target_path,
|
||||||
|
"target_exists": exists,
|
||||||
|
"target_vocab": gguf_meta.fingerprint(target_path) if exists else None,
|
||||||
|
"drafts": drafts,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def set_spec_draft(model_id: str, draft_path: str | None) -> dict:
|
||||||
|
"""Setzt (oder entfernt mit draft_path=None) den Spec-Draft eines Modells.
|
||||||
|
Validiert die Vocab-Kompatibilität — ein inkompatibler/unprüfbarer Draft wird
|
||||||
|
abgelehnt (idiotensicher). Returns {ok, reason}."""
|
||||||
|
cfg = read_config()
|
||||||
|
spec = (cfg.get("models") or {}).get(model_id)
|
||||||
|
if not isinstance(spec, dict):
|
||||||
|
return {"ok": False, "reason": "Modell nicht gefunden"}
|
||||||
|
cmd = str(spec.get("cmd", ""))
|
||||||
|
# vorhandene Spec-Flags entfernen (idempotent)
|
||||||
|
cmd = re.sub(r"\s+--spec-draft-model\s+\S+", "", cmd)
|
||||||
|
cmd = re.sub(r"\s+--spec-type\s+\S+", "", cmd)
|
||||||
|
|
||||||
|
if draft_path:
|
||||||
|
# relative Angabe (nur Dateiname) gegen DRAFTS_DIR auflösen
|
||||||
|
if not os.path.isabs(draft_path) and "/" not in draft_path:
|
||||||
|
draft_path = str(DRAFTS_DIR / draft_path)
|
||||||
|
if not os.path.exists(draft_path):
|
||||||
|
return {"ok": False, "reason": "Draft-Datei nicht gefunden"}
|
||||||
|
from services import gguf_meta
|
||||||
|
target = ""
|
||||||
|
if (mt := _PATH_RE.search(cmd)):
|
||||||
|
target = mt.group(1).replace("'", "").replace('"', "")
|
||||||
|
comp = gguf_meta.compatible(target, draft_path) if os.path.exists(target) else None
|
||||||
|
if comp is not True:
|
||||||
|
reason = ("Draft ist NICHT vocab-kompatibel zum Modell — Speculative Decoding "
|
||||||
|
"würde beim Laden scheitern."
|
||||||
|
if comp is False else
|
||||||
|
"Kompatibilität nicht prüfbar (Modell-GGUF fehlt) — Draft nicht gesetzt.")
|
||||||
|
return {"ok": False, "reason": reason}
|
||||||
|
cmd = cmd.rstrip() + f" --spec-draft-model {draft_path} --spec-type {SPEC_TYPE}"
|
||||||
|
|
||||||
|
spec["cmd"] = LiteralScalarString(cmd.rstrip() + "\n")
|
||||||
|
write_config(cfg)
|
||||||
|
return {"ok": True, "reason": ""}
|
||||||
|
|
||||||
|
|
||||||
# --- Groups (Ko-Residenz) ----------------------------------------------------
|
# --- Groups (Ko-Residenz) ----------------------------------------------------
|
||||||
def set_group(group: str, members: list[str], swap: bool = False, persist: bool = False) -> None:
|
def set_group(group: str, members: list[str], swap: bool = False, persist: bool = False) -> None:
|
||||||
"""llama-swap-`groups`-Eintrag setzen. swap=False → alle Mitglieder dürfen
|
"""llama-swap-`groups`-Eintrag setzen. swap=False → alle Mitglieder dürfen
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ erweiterte sudoers (siehe docs/BEDIENUNG.md). Lange Ops laufen als jobengine-Job
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
|
import re
|
||||||
import subprocess
|
import subprocess
|
||||||
import time
|
import time
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
@@ -21,10 +22,30 @@ SYSTEM_SERVICES = {"llama-swap"}
|
|||||||
USER_SERVICES = {"mission-control-2", "hermes-gateway", "hermes-dashboard", "hermes-webui"}
|
USER_SERVICES = {"mission-control-2", "hermes-gateway", "hermes-dashboard", "hermes-webui"}
|
||||||
|
|
||||||
ENGINE_UPDATE_CMD = os.environ.get("MC_ENGINE_UPDATE_CMD", "")
|
ENGINE_UPDATE_CMD = os.environ.get("MC_ENGINE_UPDATE_CMD", "")
|
||||||
ENGINE_PATH = os.environ.get("MC_ENGINE_PATH", "/opt/llamacpp")
|
# Engine = offizieller Vulkan-Build von ggml-org/llama.cpp (RADV auf Strix Halo).
|
||||||
|
ENGINE_PATH = os.environ.get("MC_ENGINE_PATH", "/opt/llamacpp-vulkan")
|
||||||
|
ENGINE_REPO = os.environ.get("MC_ENGINE_REPO", "ggml-org/llama.cpp")
|
||||||
_engine_cache = {"ts": 0.0, "avail": False}
|
_engine_cache = {"ts": 0.0, "avail": False}
|
||||||
|
|
||||||
|
|
||||||
|
def _installed_engine_build() -> int | None:
|
||||||
|
"""Build-Nummer der installierten llama-server-Binary (z.B. 9821), oder None.
|
||||||
|
Vulkan-Build braucht LD_LIBRARY_PATH=ENGINE_PATH zum Start von --version."""
|
||||||
|
bin_path = os.path.join(ENGINE_PATH, "llama-server")
|
||||||
|
if not os.path.exists(bin_path):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
env = dict(os.environ, LD_LIBRARY_PATH=ENGINE_PATH)
|
||||||
|
out = subprocess.run([bin_path, "--version"], capture_output=True, text=True,
|
||||||
|
timeout=20, env=env)
|
||||||
|
txt = (out.stderr or "") + (out.stdout or "")
|
||||||
|
if (m := re.search(r"build:\s*\S+\s*\((\d+)\)", txt)) or (m := re.search(r"\bb(\d{3,})\b", txt)):
|
||||||
|
return int(m.group(1))
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _ram_gb() -> float:
|
def _ram_gb() -> float:
|
||||||
return psutil.virtual_memory().total / (1024 ** 3)
|
return psutil.virtual_memory().total / (1024 ** 3)
|
||||||
|
|
||||||
@@ -45,11 +66,16 @@ def _engine_update_available() -> bool:
|
|||||||
return _engine_cache["avail"]
|
return _engine_cache["avail"]
|
||||||
avail = False
|
avail = False
|
||||||
try:
|
try:
|
||||||
rel = httpx.get("https://api.github.com/repos/lemonade-sdk/llamacpp-rocm/releases/latest",
|
rel = httpx.get(f"https://api.github.com/repos/{ENGINE_REPO}/releases/latest",
|
||||||
timeout=6, headers={"User-Agent": "MissionControl2"}).json()
|
timeout=6, headers={"User-Agent": "MissionControl2"}).json()
|
||||||
pub = datetime.fromisoformat(rel["published_at"].replace("Z", "+00:00")).timestamp()
|
tag = str(rel.get("tag_name", ""))
|
||||||
inst = os.path.getmtime(ENGINE_PATH)
|
latest = int(m.group(1)) if (m := re.search(r"(\d{3,})", tag)) else None
|
||||||
avail = pub > inst + 86400
|
installed = _installed_engine_build()
|
||||||
|
if latest is not None and installed is not None:
|
||||||
|
avail = latest > installed # präziser Build-Nummer-Vergleich
|
||||||
|
else: # Fallback: Release-Datum vs. Engine-mtime
|
||||||
|
pub = datetime.fromisoformat(rel["published_at"].replace("Z", "+00:00")).timestamp()
|
||||||
|
avail = pub > os.path.getmtime(ENGINE_PATH) + 86400
|
||||||
except Exception:
|
except Exception:
|
||||||
avail = False
|
avail = False
|
||||||
_engine_cache.update(ts=now, avail=avail)
|
_engine_cache.update(ts=now, avail=avail)
|
||||||
|
|||||||
+1
-2
@@ -6,11 +6,10 @@ set -euo pipefail
|
|||||||
MODELS_DIR="${MC_MODELS_DIR:-/srv/models}"
|
MODELS_DIR="${MC_MODELS_DIR:-/srv/models}"
|
||||||
MEM_DB="${MC_MEMORY_DB:-$MODELS_DIR/mc2-memory.db}"
|
MEM_DB="${MC_MEMORY_DB:-$MODELS_DIR/mc2-memory.db}"
|
||||||
LSWAP="${MC_CONFIG_PATH:-/etc/llama-swap/config.yaml}"
|
LSWAP="${MC_CONFIG_PATH:-/etc/llama-swap/config.yaml}"
|
||||||
GW="${MC_GATEWAY_CONFIG:-/opt/mission-control-2/gateway/config.yaml}"
|
|
||||||
DEST="$MODELS_DIR/mc2-backups/$(date +%Y%m%d-%H%M%S)"
|
DEST="$MODELS_DIR/mc2-backups/$(date +%Y%m%d-%H%M%S)"
|
||||||
|
|
||||||
mkdir -p "$DEST"
|
mkdir -p "$DEST"
|
||||||
for f in "$MEM_DB" "$MEM_DB-wal" "$MEM_DB-shm" "$LSWAP" "$GW"; do
|
for f in "$MEM_DB" "$MEM_DB-wal" "$MEM_DB-shm" "$LSWAP"; do
|
||||||
[ -f "$f" ] && cp -p "$f" "$DEST/" || true
|
[ -f "$f" ] && cp -p "$f" "$DEST/" || true
|
||||||
done
|
done
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,68 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Provisioniert die Inferenz-Engine auf der AI Box (Strix Halo / Ryzen AI MAX+ 395, gfx1151)
|
||||||
|
# auf **Vulkan/RADV** — reproduzierbar. Braucht root (sudo).
|
||||||
|
#
|
||||||
|
# sudo bash ~/mission-control-v2/deploy/provision-engine.sh
|
||||||
|
#
|
||||||
|
# Hintergrund: Auf gfx1151 ist Vulkan/RADV ggü. ROCm/HIP messbar schneller
|
||||||
|
# (Token-Gen +12–22 %, Prefill gleich; auf der Box per llama-bench verifiziert 2026-06-27)
|
||||||
|
# UND einfacher. Der ROCm-Build bleibt unter /opt/llamacpp als Rollback liegen.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
VULKAN_DIR=/opt/llamacpp-vulkan
|
||||||
|
WARMUP_DST=/usr/local/bin/llama-swap-warmup.sh
|
||||||
|
DROPIN_DIR=/etc/systemd/system/llama-swap.service.d
|
||||||
|
SRC_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
# Optional gepinnter Build (z.B. b9821). Leer = neuester Release.
|
||||||
|
PIN_BUILD="${MC_ENGINE_BUILD:-}"
|
||||||
|
|
||||||
|
echo "==> 1. RADV-Treiber + Vulkan-Runtime"
|
||||||
|
export DEBIAN_FRONTEND=noninteractive
|
||||||
|
apt-get update -qq
|
||||||
|
apt-get install -y mesa-vulkan-drivers libvulkan1 vulkan-tools curl jq
|
||||||
|
|
||||||
|
echo "==> 2. llama.cpp Vulkan-Build nach $VULKAN_DIR"
|
||||||
|
if [ -n "$PIN_BUILD" ]; then
|
||||||
|
TAG="$PIN_BUILD"
|
||||||
|
else
|
||||||
|
TAG="$(curl -s https://api.github.com/repos/ggml-org/llama.cpp/releases/latest | jq -r .tag_name)"
|
||||||
|
fi
|
||||||
|
ASSET="llama-${TAG}-bin-ubuntu-vulkan-x64.tar.gz"
|
||||||
|
URL="https://github.com/ggml-org/llama.cpp/releases/download/${TAG}/${ASSET}"
|
||||||
|
TMP="$(mktemp -d)"
|
||||||
|
echo " Lade $URL"
|
||||||
|
curl -sL -o "$TMP/v.tgz" "$URL"
|
||||||
|
mkdir -p "$VULKAN_DIR"
|
||||||
|
tar xzf "$TMP/v.tgz" -C "$TMP"
|
||||||
|
# Tarball entpackt nach .../llama-<tag>/ — Inhalt flach nach $VULKAN_DIR
|
||||||
|
cp -rf "$TMP"/llama-*/. "$VULKAN_DIR"/
|
||||||
|
rm -rf "$TMP"
|
||||||
|
test -x "$VULKAN_DIR/llama-server"
|
||||||
|
|
||||||
|
echo "==> 3. Symlink llama-server -> Vulkan-Build"
|
||||||
|
ln -sfn "$VULKAN_DIR/llama-server" /usr/local/bin/llama-server
|
||||||
|
|
||||||
|
echo "==> 4. systemd Drop-ins für llama-swap (LD_LIBRARY_PATH + Brain-Warmup)"
|
||||||
|
mkdir -p "$DROPIN_DIR"
|
||||||
|
cat > "$DROPIN_DIR/vulkan.conf" <<EOF
|
||||||
|
[Service]
|
||||||
|
Environment=LD_LIBRARY_PATH=$VULKAN_DIR
|
||||||
|
EOF
|
||||||
|
install -m 0755 "$SRC_DIR/warmup.sh" "$WARMUP_DST"
|
||||||
|
cat > "$DROPIN_DIR/warmup.conf" <<EOF
|
||||||
|
[Service]
|
||||||
|
# Nach jedem (Re)Start die brains vorladen. Das Skript detacht sich selbst (blockiert
|
||||||
|
# den Start nicht); '-' macht den Aufruf fehlertolerant (kann llama-swap nie failen lassen).
|
||||||
|
ExecStartPost=-$WARMUP_DST
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo "==> 5. Reload + Restart"
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl restart llama-swap
|
||||||
|
sleep 3
|
||||||
|
systemctl is-active llama-swap
|
||||||
|
|
||||||
|
echo "==> Fertig. Aktive Engine:"
|
||||||
|
readlink -f /usr/local/bin/llama-server
|
||||||
|
vulkaninfo --summary 2>/dev/null | grep -m1 deviceName || true
|
||||||
|
echo "Rollback auf ROCm: ln -sfn /opt/llamacpp/llama-server /usr/local/bin/llama-server && rm $DROPIN_DIR/vulkan.conf && systemctl daemon-reload && systemctl restart llama-swap"
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Lädt die "brains" (hermes/fast/vision) nach einem llama-swap-(Re)Start vor,
|
||||||
|
# damit die ERSTE Anfrage nicht kalt ist (llama-swap lädt sonst lazy bei Bedarf).
|
||||||
|
# Eingehängt als ExecStartPost=-/usr/local/bin/llama-swap-warmup.sh in llama-swap.
|
||||||
|
#
|
||||||
|
# Selbst-detachend: der ExecStartPost-Aufruf kehrt sofort zurück (blockiert den
|
||||||
|
# Service-Start NICHT); die eigentliche Aufwärmung läuft entkoppelt im Hintergrund.
|
||||||
|
set -u
|
||||||
|
URL="${MC_LLAMA_SWAP_URL:-http://127.0.0.1:8080}"
|
||||||
|
BRAINS="${MC_WARMUP_MODELS:-hermes fast vision}"
|
||||||
|
|
||||||
|
if [ "${1:-}" != "--inner" ]; then
|
||||||
|
setsid "$0" --inner >/dev/null 2>&1 &
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- ab hier im entkoppelten Hintergrundprozess ---
|
||||||
|
for _ in $(seq 1 60); do # auf llama-swap warten (max ~120s)
|
||||||
|
curl -sf "$URL/v1/models" >/dev/null 2>&1 && break
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
|
||||||
|
for m in $BRAINS; do
|
||||||
|
curl -s -m 180 -X POST "$URL/v1/chat/completions" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"model\":\"$m\",\"max_tokens\":1,\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
|
done
|
||||||
@@ -23,7 +23,25 @@ metadata:
|
|||||||
## Dienste auf der AI Box (Stand 2026-06-27)
|
## Dienste auf der AI Box (Stand 2026-06-27)
|
||||||
| Dienst | Port | Beschreibung |
|
| Dienst | Port | Beschreibung |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `llama-swap` (system) | `:8080` | Engine, ROCm/HIP |
|
| `llama-swap` (system) | `:8080` | Engine, **Vulkan/RADV** (seit 2026-06-27, vorher ROCm/HIP) |
|
||||||
|
|
||||||
|
### Engine-Backend: Vulkan/RADV (Cutover 2026-06-27)
|
||||||
|
- **Gemessen:** RADV schlägt ROCm/HIP auf gfx1151 bei tg um **+12–22 %** (fast 53→65 t/s roh), Prefill gleich.
|
||||||
|
„ROCm gewinnt Prefill" gilt hier NICHT; auch bei 32K Tiefe bleibt RADV vorn. hipBLASLt bringt nichts.
|
||||||
|
- **Setup:** Vulkan-llama.cpp (offizieller Build b9821) in `/opt/llamacpp-vulkan`; Symlink
|
||||||
|
`/usr/local/bin/llama-server` → dorthin; systemd Drop-in `/etc/systemd/system/llama-swap.service.d/vulkan.conf`
|
||||||
|
mit `LD_LIBRARY_PATH=/opt/llamacpp-vulkan`. Treiber: `mesa-vulkan-drivers` (RADV STRIX_HALO, Mesa 26.0.3).
|
||||||
|
- **ROCm-Build bleibt** unter `/opt/llamacpp` (lemonade llamacpp-rocm) als Rollback liegen.
|
||||||
|
- **Rollback:** `sudo ln -sfn /opt/llamacpp/llama-server /usr/local/bin/llama-server` + Drop-in löschen + `sudo systemctl daemon-reload && sudo systemctl restart llama-swap`.
|
||||||
|
- **ACHTUNG maintenance.py:** `_engine_update_available()` trackt noch `lemonade-sdk/llamacpp-rocm` — passt nicht
|
||||||
|
mehr zum aktiven Vulkan-Build (b9821 von ggml-org/llama.cpp). Engine-Update-Quelle anpassen.
|
||||||
|
- **Spec-Draft (2026-06-27 gefixt):** `qwen2.5-1.5b` war vocab-inkompatibel mit Qwen3.6 UND `--spec-type` fehlte
|
||||||
|
→ war inaktiv. Auch Qwen3-0.6B ist mit Qwen3.6 inkompatibel → **`fast` läuft jetzt ohne Draft** (kein
|
||||||
|
kompatibler verfügbar). **`coder` (Qwen3-Coder-Next)**: Qwen3-0.6B IST kompatibel → Draft
|
||||||
|
`/srv/models/drafts/Qwen3-0.6B-Q8_0.gguf` + `--spec-type draft-simple`, **0,75 Akzeptanz** (echter Coding-Speedup).
|
||||||
|
Alter `qwen2.5-1.5b-instruct-q4_k_m.gguf` in drafts/ ist jetzt ungenutzt (löschbar, 1,1 GB).
|
||||||
|
- **Modell-cmds (config.yaml):** llama.cpp dieser Generation braucht `--spec-type` zusätzlich zu
|
||||||
|
`--spec-draft-model`, sonst ist Spec inaktiv. Drafts müssen **exakt vocab-gleich** zum Target sein.
|
||||||
| `mission-control-2` (user) | `:9001` | MC2 Backend + Frontend |
|
| `mission-control-2` (user) | `:9001` | MC2 Backend + Frontend |
|
||||||
| `hermes-gateway` (user) | `:8642` | Hermes Agent API |
|
| `hermes-gateway` (user) | `:8642` | Hermes Agent API |
|
||||||
| `hermes-webui` (user) | `:8787` | nesquena hermes-webui |
|
| `hermes-webui` (user) | `:8787` | nesquena hermes-webui |
|
||||||
|
|||||||
-1
File diff suppressed because one or more lines are too long
+1
File diff suppressed because one or more lines are too long
-395
File diff suppressed because one or more lines are too long
+395
File diff suppressed because one or more lines are too long
Vendored
+2
-2
@@ -7,8 +7,8 @@
|
|||||||
<link rel="manifest" href="/manifest.webmanifest" />
|
<link rel="manifest" href="/manifest.webmanifest" />
|
||||||
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
|
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
|
||||||
<title>Mission Control 2.0</title>
|
<title>Mission Control 2.0</title>
|
||||||
<script type="module" crossorigin src="/assets/index-D4mBflTE.js"></script>
|
<script type="module" crossorigin src="/assets/index-DPqptZjm.js"></script>
|
||||||
<link rel="stylesheet" crossorigin href="/assets/index-C0dfTDFj.css">
|
<link rel="stylesheet" crossorigin href="/assets/index-CjZVMvhL.css">
|
||||||
</head>
|
</head>
|
||||||
<body>
|
<body>
|
||||||
<div id="root"></div>
|
<div id="root"></div>
|
||||||
|
|||||||
@@ -0,0 +1,161 @@
|
|||||||
|
import { useState } from "react"
|
||||||
|
import { X, Check, Zap, AlertTriangle } from "lucide-react"
|
||||||
|
import { api, type ModelInfo, type DraftInfo } from "@/lib/api"
|
||||||
|
import { useDrafts } from "@/lib/queries"
|
||||||
|
import { fmtSize } from "@/lib/format"
|
||||||
|
import { cn } from "@/lib/utils"
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Idiotensichere Speculative-Decoding-Konfiguration für EIN Modell.
|
||||||
|
* Zeigt nur VOCAB-KOMPATIBLE Drafts als wählbar; inkompatible werden gesperrt
|
||||||
|
* und mit Begründung angezeigt. So kann nie ein kaputter Draft gesetzt werden
|
||||||
|
* (der das Modell beim Laden scheitern ließe).
|
||||||
|
*/
|
||||||
|
export function SpecDraftModal({
|
||||||
|
model, onClose, onChanged,
|
||||||
|
}: { model: ModelInfo; onClose: () => void; onChanged: () => void }) {
|
||||||
|
const { data, isLoading } = useDrafts(model.gguf_path)
|
||||||
|
const [busy, setBusy] = useState<string | null>(null)
|
||||||
|
const [err, setErr] = useState("")
|
||||||
|
|
||||||
|
const tv = data?.target_vocab
|
||||||
|
const drafts = data?.drafts ?? []
|
||||||
|
const compatibles = drafts.filter((d) => d.compatible === true)
|
||||||
|
const currentFile = model.spec_draft_model
|
||||||
|
|
||||||
|
async function apply(draftPath: string | null) {
|
||||||
|
setBusy(draftPath ?? "__clear__")
|
||||||
|
setErr("")
|
||||||
|
try {
|
||||||
|
await api(`/api/models/${encodeURIComponent(model.name)}/draft`, {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify({ draft_path: draftPath }),
|
||||||
|
})
|
||||||
|
onChanged()
|
||||||
|
onClose()
|
||||||
|
} catch (e: any) {
|
||||||
|
setErr(String(e?.message || e))
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const vocabLabel = (v?: { pre: string | null; n_vocab: number | null } | null) =>
|
||||||
|
v ? `${v.pre ?? "?"} · ${v.n_vocab?.toLocaleString() ?? "?"} Tokens` : "—"
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="fixed inset-0 z-50 bg-black/60 backdrop-blur-sm flex items-center justify-center p-4">
|
||||||
|
<div className="w-full max-w-lg rounded-2xl border border-border/80 bg-card p-6 shadow-2xl space-y-4 animate-in fade-in zoom-in-95 duration-200">
|
||||||
|
<div className="flex items-center justify-between border-b border-border/20 pb-2">
|
||||||
|
<h3 className="text-xs font-bold uppercase tracking-wider text-primary flex items-center gap-2">
|
||||||
|
<Zap className="h-4 w-4" /> Speculative Draft
|
||||||
|
</h3>
|
||||||
|
<button onClick={onClose} className="p-1 rounded hover:bg-accent text-muted-foreground hover:text-foreground cursor-pointer">
|
||||||
|
<X className="h-4 w-4" />
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="text-xs text-muted-foreground leading-relaxed">
|
||||||
|
Beschleunigt die Token-Generierung mit einem kleinen "Draft"-Modell. Voraussetzung:
|
||||||
|
der Draft muss den <strong className="text-foreground">exakt gleichen Tokenizer (Vocab)</strong> haben
|
||||||
|
wie das Modell — sonst lehnt llama.cpp es ab.
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Ziel-Vocab */}
|
||||||
|
<div className="rounded-lg border border-border/40 bg-background/30 px-3 py-2 text-[11px] font-mono flex items-center justify-between">
|
||||||
|
<span className="text-muted-foreground">{model.name.split("/").pop()?.replace(/\.gguf$/i, "")}</span>
|
||||||
|
<span className="text-foreground">Vocab: {vocabLabel(tv)}</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Aktueller Zustand */}
|
||||||
|
{model.spec_active && currentFile && (
|
||||||
|
<div className="rounded-lg border border-emerald-500/30 bg-emerald-500/5 px-3 py-2 flex items-center justify-between gap-2">
|
||||||
|
<span className="text-[11px] text-emerald-400 font-mono flex items-center gap-1.5 truncate">
|
||||||
|
<Check className="h-3.5 w-3.5 shrink-0" /> Aktiv: {currentFile}
|
||||||
|
</span>
|
||||||
|
<button
|
||||||
|
onClick={() => apply(null)}
|
||||||
|
disabled={busy !== null}
|
||||||
|
className="h-6 px-2.5 rounded border border-amber-500/30 bg-amber-500/5 hover:bg-amber-500/10 text-amber-400 text-[9px] font-bold uppercase shrink-0 cursor-pointer disabled:opacity-50"
|
||||||
|
>
|
||||||
|
Deaktivieren
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{!data?.target_exists && (
|
||||||
|
<div className="rounded-lg border border-amber-500/30 bg-amber-500/5 px-3 py-2 text-[11px] text-amber-400 flex items-center gap-2">
|
||||||
|
<AlertTriangle className="h-3.5 w-3.5 shrink-0" />
|
||||||
|
Modell-GGUF noch nicht vorhanden — Spec-Draft ist nach dem Download konfigurierbar.
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Draft-Liste */}
|
||||||
|
<div className="space-y-1.5 max-h-64 overflow-y-auto pr-1">
|
||||||
|
{isLoading ? (
|
||||||
|
<div className="text-xs text-muted-foreground py-6 text-center">Prüfe Vocab-Kompatibilität…</div>
|
||||||
|
) : drafts.length === 0 ? (
|
||||||
|
<div className="text-[11px] text-muted-foreground border border-dashed border-border/50 rounded-lg p-4 text-center">
|
||||||
|
Keine Draft-Modelle in <code className="font-mono">/srv/models/drafts</code>.
|
||||||
|
Lade ein kleines same-family-Modell (z.B. Qwen3-0.6B) und lege es dort ab.
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
drafts.map((d: DraftInfo) => {
|
||||||
|
const isCurrent = d.filename === currentFile
|
||||||
|
const ok = d.compatible === true
|
||||||
|
return (
|
||||||
|
<div
|
||||||
|
key={d.path}
|
||||||
|
className={cn(
|
||||||
|
"rounded-lg border px-3 py-2 flex items-center justify-between gap-2 text-left",
|
||||||
|
ok ? "border-border/40 bg-background/20" : "border-border/20 bg-background/10 opacity-60",
|
||||||
|
isCurrent && "border-primary/40 bg-primary/10",
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
<div className="min-w-0">
|
||||||
|
<div className="text-[11px] font-mono font-semibold text-foreground truncate">{d.filename}</div>
|
||||||
|
<div className="text-[9px] text-muted-foreground font-mono">
|
||||||
|
{fmtSize(d.size_bytes)} · Vocab: {vocabLabel(d.vocab)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{ok ? (
|
||||||
|
isCurrent ? (
|
||||||
|
<span className="text-[9px] font-bold uppercase text-primary flex items-center gap-1 shrink-0"><Check className="h-3.5 w-3.5" /> Aktiv</span>
|
||||||
|
) : (
|
||||||
|
<button
|
||||||
|
onClick={() => apply(d.path)}
|
||||||
|
disabled={busy !== null}
|
||||||
|
className="h-7 px-3 rounded-lg border border-emerald-500/30 bg-emerald-500/5 hover:bg-emerald-500/15 text-emerald-400 text-[9px] font-bold uppercase shrink-0 cursor-pointer disabled:opacity-50"
|
||||||
|
>
|
||||||
|
Aktivieren
|
||||||
|
</button>
|
||||||
|
)
|
||||||
|
) : (
|
||||||
|
<span
|
||||||
|
className="text-[9px] font-bold uppercase text-amber-400/80 flex items-center gap-1 shrink-0"
|
||||||
|
title={d.compatible === false
|
||||||
|
? `Inkompatibel: Tokenizer/Vocab weicht ab (Draft ${d.vocab?.pre}/${d.vocab?.n_vocab} ≠ Modell ${tv?.pre}/${tv?.n_vocab}).`
|
||||||
|
: "Kompatibilität nicht prüfbar (Datei fehlt)."}
|
||||||
|
>
|
||||||
|
<AlertTriangle className="h-3.5 w-3.5" /> {d.compatible === false ? "Vocab ≠" : "n/a"}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
})
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Hinweis, wenn Drafts da sind aber keiner kompatibel */}
|
||||||
|
{!isLoading && data?.target_exists && drafts.length > 0 && compatibles.length === 0 && (
|
||||||
|
<div className="rounded-lg border border-amber-500/30 bg-amber-500/5 px-3 py-2 text-[11px] text-amber-400 leading-relaxed">
|
||||||
|
Kein vocab-kompatibler Draft verfügbar — Speculative Decoding ist für dieses Modell nicht möglich.
|
||||||
|
Es braucht einen Draft mit identischem Tokenizer (pre=<span className="font-mono">{tv?.pre}</span>,
|
||||||
|
n_vocab=<span className="font-mono">{tv?.n_vocab?.toLocaleString()}</span>).
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{err && <div className="text-[10px] text-red-400 font-mono">{err}</div>}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -88,10 +88,34 @@ export interface ModelInfo {
|
|||||||
incomplete: boolean
|
incomplete: boolean
|
||||||
prompt_cache: boolean
|
prompt_cache: boolean
|
||||||
spec_draft_model: string | null
|
spec_draft_model: string | null
|
||||||
|
spec_type: string | null
|
||||||
|
spec_active: boolean
|
||||||
parallel_slots: number
|
parallel_slots: number
|
||||||
capabilities: Capabilities
|
capabilities: Capabilities
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface VocabFingerprint {
|
||||||
|
model: string | null
|
||||||
|
pre: string | null
|
||||||
|
n_vocab: number | null
|
||||||
|
arch: string | null
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DraftInfo {
|
||||||
|
path: string
|
||||||
|
filename: string
|
||||||
|
size_bytes: number | null
|
||||||
|
vocab: VocabFingerprint | null
|
||||||
|
compatible: boolean | null // null = nicht prüfbar (Ziel-GGUF fehlt)
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DraftsResp {
|
||||||
|
target_path: string
|
||||||
|
target_exists: boolean
|
||||||
|
target_vocab: VocabFingerprint | null
|
||||||
|
drafts: DraftInfo[]
|
||||||
|
}
|
||||||
|
|
||||||
export interface DiscoverModel {
|
export interface DiscoverModel {
|
||||||
name: string
|
name: string
|
||||||
author: string
|
author: string
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import {
|
|||||||
type AgentStatus,
|
type AgentStatus,
|
||||||
type ConnectResp,
|
type ConnectResp,
|
||||||
type DiscoverResp,
|
type DiscoverResp,
|
||||||
|
type DraftsResp,
|
||||||
type Health,
|
type Health,
|
||||||
type Job,
|
type Job,
|
||||||
type Memory,
|
type Memory,
|
||||||
@@ -31,6 +32,7 @@ export const qk = {
|
|||||||
agentStatus: ["agent-status"] as const,
|
agentStatus: ["agent-status"] as const,
|
||||||
updates: ["updates"] as const,
|
updates: ["updates"] as const,
|
||||||
discover: ["discover"] as const,
|
discover: ["discover"] as const,
|
||||||
|
drafts: (target?: string) => ["drafts", target ?? ""] as const,
|
||||||
connect: (params?: string) => ["connect", params ?? ""] as const,
|
connect: (params?: string) => ["connect", params ?? ""] as const,
|
||||||
memory: (q?: string, category?: string) => ["memory", q ?? "", category ?? ""] as const,
|
memory: (q?: string, category?: string) => ["memory", q ?? "", category ?? ""] as const,
|
||||||
}
|
}
|
||||||
@@ -70,6 +72,14 @@ export const useUpdates = (refetchInterval?: number) =>
|
|||||||
export const useDiscover = () =>
|
export const useDiscover = () =>
|
||||||
useQuery({ queryKey: qk.discover, queryFn: () => api<DiscoverResp>("/api/discover") })
|
useQuery({ queryKey: qk.discover, queryFn: () => api<DiscoverResp>("/api/discover") })
|
||||||
|
|
||||||
|
// Verfügbare Spec-Draft-Modelle + Vocab-Kompatibilität zum Ziel-Modell (GGUF-Pfad).
|
||||||
|
export const useDrafts = (target?: string) =>
|
||||||
|
useQuery({
|
||||||
|
queryKey: qk.drafts(target),
|
||||||
|
queryFn: () => api<DraftsResp>(`/api/models/drafts?target=${encodeURIComponent(target ?? "")}`),
|
||||||
|
enabled: !!target,
|
||||||
|
})
|
||||||
|
|
||||||
export const useConnect = (params?: string) =>
|
export const useConnect = (params?: string) =>
|
||||||
useQuery({
|
useQuery({
|
||||||
queryKey: qk.connect(params),
|
queryKey: qk.connect(params),
|
||||||
|
|||||||
@@ -7,8 +7,10 @@ export function AddModel() {
|
|||||||
const [quants, setQuants] = useState<string[]>([])
|
const [quants, setQuants] = useState<string[]>([])
|
||||||
const [quant, setQuant] = useState("Q4_K_M")
|
const [quant, setQuant] = useState("Q4_K_M")
|
||||||
const [msg, setMsg] = useState("")
|
const [msg, setMsg] = useState("")
|
||||||
|
const [role, setRole] = useState("")
|
||||||
const [q, setQ] = useState("")
|
const [q, setQ] = useState("")
|
||||||
const [results, setResults] = useState<{ repo: string; downloads: number }[]>([])
|
const [results, setResults] = useState<{ repo: string; downloads: number }[]>([])
|
||||||
|
const ROLE_OPTS = ["fast", "heavy", "coder", "vision", "scout"]
|
||||||
|
|
||||||
async function loadQuants(r?: string) {
|
async function loadQuants(r?: string) {
|
||||||
const rr = r ?? repo
|
const rr = r ?? repo
|
||||||
@@ -43,9 +45,15 @@ export function AddModel() {
|
|||||||
try {
|
try {
|
||||||
await api("/api/models/install", {
|
await api("/api/models/install", {
|
||||||
method: "POST",
|
method: "POST",
|
||||||
body: JSON.stringify({ repo, quant, jinja: true }),
|
body: JSON.stringify({ repo, quant, role: role || undefined, jinja: true }),
|
||||||
})
|
})
|
||||||
setMsg(`Download gestartet: ${repo} (${quant}). Fortschritt wird oben angezeigt.`)
|
setMsg(
|
||||||
|
`Download gestartet: ${repo} (${quant})${role ? `, Rolle: ${role}` : ""}. ` +
|
||||||
|
`Fortschritt oben.` +
|
||||||
|
(role === "fast" || role === "coder"
|
||||||
|
? " Speculative Draft (Vocab-geprüft) kannst du nach dem Download an der Modellkarte (»Spec«) setzen."
|
||||||
|
: "")
|
||||||
|
)
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
setMsg(`Download-Fehler: ${e}`)
|
setMsg(`Download-Fehler: ${e}`)
|
||||||
}
|
}
|
||||||
@@ -78,6 +86,15 @@ export function AddModel() {
|
|||||||
>
|
>
|
||||||
{quants.map((qq) => <option key={qq} value={qq} className="bg-popover text-foreground">{qq}</option>)}
|
{quants.map((qq) => <option key={qq} value={qq} className="bg-popover text-foreground">{qq}</option>)}
|
||||||
</select>
|
</select>
|
||||||
|
<select
|
||||||
|
value={role}
|
||||||
|
onChange={(e) => setRole(e.target.value)}
|
||||||
|
title="Rolle (optional) — bestimmt Auto-Konfiguration wie parallele Slots"
|
||||||
|
className="h-9 rounded-lg border border-border/60 bg-background/40 px-3 text-xs outline-none text-foreground font-semibold"
|
||||||
|
>
|
||||||
|
<option value="" className="bg-popover text-foreground">Rolle…</option>
|
||||||
|
{ROLE_OPTS.map((r) => <option key={r} value={r} className="bg-popover text-foreground">{r}</option>)}
|
||||||
|
</select>
|
||||||
<button
|
<button
|
||||||
onClick={install}
|
onClick={install}
|
||||||
className="h-9 px-4 rounded-lg bg-primary text-primary-foreground text-xs font-semibold hover:opacity-90 transition-all cursor-pointer flex items-center gap-1.5"
|
className="h-9 px-4 rounded-lg bg-primary text-primary-foreground text-xs font-semibold hover:opacity-90 transition-all cursor-pointer flex items-center gap-1.5"
|
||||||
|
|||||||
@@ -1,12 +1,13 @@
|
|||||||
import { useState, useRef, useCallback } from "react"
|
import { useState, useRef, useCallback } from "react"
|
||||||
import { Download, Trash2, Edit3, Activity, HardDrive, X, Check, Copy } from "lucide-react"
|
import { Download, Trash2, Edit3, Activity, HardDrive, X, Check, Copy, Zap } from "lucide-react"
|
||||||
import { api } from "@/lib/api"
|
import { api, type ModelInfo } from "@/lib/api"
|
||||||
import { useModels, useRouting, useConnect, useUpdates, useQueryClient, qk } from "@/lib/queries"
|
import { useModels, useRouting, useConnect, useUpdates, useQueryClient, qk } from "@/lib/queries"
|
||||||
import { useDialog } from "@/lib/useDialog"
|
import { useDialog } from "@/lib/useDialog"
|
||||||
import { CapsChips } from "@/components/CapsChips"
|
import { CapsChips } from "@/components/CapsChips"
|
||||||
import { cn } from "@/lib/utils"
|
import { cn } from "@/lib/utils"
|
||||||
import { fmtSize, fmtCtx } from "@/lib/format"
|
import { fmtSize, fmtCtx } from "@/lib/format"
|
||||||
import { getBrandInfo, ROLES } from "@/components/models/ModelBadges"
|
import { getBrandInfo, ROLES } from "@/components/models/ModelBadges"
|
||||||
|
import { SpecDraftModal } from "@/components/models/SpecDraftModal"
|
||||||
|
|
||||||
export function Cockpit() {
|
export function Cockpit() {
|
||||||
const qc = useQueryClient()
|
const qc = useQueryClient()
|
||||||
@@ -26,6 +27,7 @@ export function Cockpit() {
|
|||||||
// UI state
|
// UI state
|
||||||
const [activeClient, setActiveClient] = useState<"roocode" | "cursor" | "opencode" | "zed" | "continue" | null>(null)
|
const [activeClient, setActiveClient] = useState<"roocode" | "cursor" | "opencode" | "zed" | "continue" | null>(null)
|
||||||
const [activeRoleForAssign, setActiveRoleForAssign] = useState<string | null>(null)
|
const [activeRoleForAssign, setActiveRoleForAssign] = useState<string | null>(null)
|
||||||
|
const [specModel, setSpecModel] = useState<ModelInfo | null>(null)
|
||||||
const [copied, setCopied] = useState(false)
|
const [copied, setCopied] = useState(false)
|
||||||
const [hoveredNode, setHoveredNode] = useState<string | null>(null)
|
const [hoveredNode, setHoveredNode] = useState<string | null>(null)
|
||||||
const [viewMode, setViewMode] = useState<"grid" | "list">("grid")
|
const [viewMode, setViewMode] = useState<"grid" | "list">("grid")
|
||||||
@@ -742,11 +744,15 @@ export function Cockpit() {
|
|||||||
PC
|
PC
|
||||||
</span>
|
</span>
|
||||||
)}
|
)}
|
||||||
{m.spec_draft_model && (
|
{m.spec_active ? (
|
||||||
<span className="px-1.5 py-0.5 rounded bg-emerald-500/10 border border-emerald-500/20 text-[9px] font-mono text-emerald-400 font-bold uppercase" title={`Speculative Decoding aktiv (Draft: ${m.spec_draft_model})`}>
|
<span className="px-1.5 py-0.5 rounded bg-emerald-500/10 border border-emerald-500/20 text-[9px] font-mono text-emerald-400 font-bold uppercase" title={`Speculative Decoding aktiv (Draft: ${m.spec_draft_model})`}>
|
||||||
SPEC
|
SPEC
|
||||||
</span>
|
</span>
|
||||||
)}
|
) : m.spec_draft_model ? (
|
||||||
|
<span className="px-1.5 py-0.5 rounded bg-amber-500/10 border border-amber-500/20 text-[9px] font-mono text-amber-400 font-bold uppercase" title={`Draft gesetzt (${m.spec_draft_model}), aber INAKTIV — --spec-type fehlt oder Vocab inkompatibel. Über 'Spec' neu konfigurieren.`}>
|
||||||
|
SPEC?
|
||||||
|
</span>
|
||||||
|
) : null}
|
||||||
{m.parallel_slots > 1 && (
|
{m.parallel_slots > 1 && (
|
||||||
<span className="px-1.5 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-[9px] font-mono text-violet-400 font-bold uppercase" title={`${m.parallel_slots} parallele Slots aktiv`}>
|
<span className="px-1.5 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-[9px] font-mono text-violet-400 font-bold uppercase" title={`${m.parallel_slots} parallele Slots aktiv`}>
|
||||||
SLOTS: {m.parallel_slots}
|
SLOTS: {m.parallel_slots}
|
||||||
@@ -825,6 +831,18 @@ export function Cockpit() {
|
|||||||
>
|
>
|
||||||
Ctx
|
Ctx
|
||||||
</button>
|
</button>
|
||||||
|
<button
|
||||||
|
onClick={() => setSpecModel(m)}
|
||||||
|
className={cn(
|
||||||
|
"h-7 px-2.5 rounded-lg border text-[9px] font-bold uppercase transition-colors cursor-pointer flex items-center gap-1",
|
||||||
|
m.spec_active ? "border-emerald-500/30 text-emerald-400 hover:bg-emerald-500/10"
|
||||||
|
: m.spec_draft_model ? "border-amber-500/30 text-amber-400 hover:bg-amber-500/10"
|
||||||
|
: "border-border/40 text-muted-foreground hover:bg-accent"
|
||||||
|
)}
|
||||||
|
title="Speculative Draft konfigurieren (Vocab-geprüft)"
|
||||||
|
>
|
||||||
|
<Zap className="h-3 w-3" /> Spec
|
||||||
|
</button>
|
||||||
<button
|
<button
|
||||||
onClick={() => handleDelete(m.name)}
|
onClick={() => handleDelete(m.name)}
|
||||||
className="h-7 w-7 rounded-lg border border-border/40 text-muted-foreground hover:text-red-400 hover:bg-red-500/5 flex items-center justify-center transition-colors cursor-pointer"
|
className="h-7 w-7 rounded-lg border border-border/40 text-muted-foreground hover:text-red-400 hover:bg-red-500/5 flex items-center justify-center transition-colors cursor-pointer"
|
||||||
@@ -885,11 +903,15 @@ export function Cockpit() {
|
|||||||
PC
|
PC
|
||||||
</span>
|
</span>
|
||||||
)}
|
)}
|
||||||
{m.spec_draft_model && (
|
{m.spec_active ? (
|
||||||
<span className="px-1.5 py-0.5 rounded bg-emerald-500/10 border border-emerald-500/20 text-[8px] font-mono text-emerald-400 font-bold uppercase shrink-0" title={`Speculative Decoding aktiv (Draft: ${m.spec_draft_model})`}>
|
<span className="px-1.5 py-0.5 rounded bg-emerald-500/10 border border-emerald-500/20 text-[8px] font-mono text-emerald-400 font-bold uppercase shrink-0" title={`Speculative Decoding aktiv (Draft: ${m.spec_draft_model})`}>
|
||||||
SPEC
|
SPEC
|
||||||
</span>
|
</span>
|
||||||
)}
|
) : m.spec_draft_model ? (
|
||||||
|
<span className="px-1.5 py-0.5 rounded bg-amber-500/10 border border-amber-500/20 text-[8px] font-mono text-amber-400 font-bold uppercase shrink-0" title={`Draft gesetzt (${m.spec_draft_model}), aber INAKTIV — --spec-type fehlt oder Vocab inkompatibel.`}>
|
||||||
|
SPEC?
|
||||||
|
</span>
|
||||||
|
) : null}
|
||||||
{m.parallel_slots > 1 && (
|
{m.parallel_slots > 1 && (
|
||||||
<span className="px-1.5 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-[8px] font-mono text-violet-400 font-bold uppercase shrink-0" title={`${m.parallel_slots} parallele Slots aktiv`}>
|
<span className="px-1.5 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-[8px] font-mono text-violet-400 font-bold uppercase shrink-0" title={`${m.parallel_slots} parallele Slots aktiv`}>
|
||||||
SLOTS: {m.parallel_slots}
|
SLOTS: {m.parallel_slots}
|
||||||
@@ -944,6 +966,18 @@ export function Cockpit() {
|
|||||||
>
|
>
|
||||||
Ctx
|
Ctx
|
||||||
</button>
|
</button>
|
||||||
|
<button
|
||||||
|
onClick={() => setSpecModel(m)}
|
||||||
|
className={cn(
|
||||||
|
"h-7 px-2.5 rounded-lg border text-[9px] font-bold uppercase transition-all cursor-pointer flex items-center gap-1",
|
||||||
|
m.spec_active ? "border-emerald-500/30 text-emerald-400 hover:bg-emerald-500/10"
|
||||||
|
: m.spec_draft_model ? "border-amber-500/30 text-amber-400 hover:bg-amber-500/10"
|
||||||
|
: "border-border/40 text-muted-foreground hover:bg-accent"
|
||||||
|
)}
|
||||||
|
title="Speculative Draft konfigurieren (Vocab-geprüft)"
|
||||||
|
>
|
||||||
|
<Zap className="h-3 w-3" /> Spec
|
||||||
|
</button>
|
||||||
<button
|
<button
|
||||||
onClick={() => handleDelete(m.name)}
|
onClick={() => handleDelete(m.name)}
|
||||||
className="h-7 w-7 rounded-lg flex items-center justify-center border border-border/40 text-muted-foreground hover:text-red-400 hover:bg-red-500/5 transition-all cursor-pointer"
|
className="h-7 w-7 rounded-lg flex items-center justify-center border border-border/40 text-muted-foreground hover:text-red-400 hover:bg-red-500/5 transition-all cursor-pointer"
|
||||||
@@ -1016,6 +1050,14 @@ export function Cockpit() {
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
{specModel && (
|
||||||
|
<SpecDraftModal
|
||||||
|
model={specModel}
|
||||||
|
onClose={() => setSpecModel(null)}
|
||||||
|
onChanged={reload}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
|
||||||
{dialogElement}
|
{dialogElement}
|
||||||
</div>
|
</div>
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -1,65 +0,0 @@
|
|||||||
# LiteLLM-Gateway-Config für Mission Control 2.0
|
|
||||||
# ----------------------------------------------------------------------------
|
|
||||||
# EIN OpenAI-Endpunkt für ALLE Konsumenten (Hermes + Vibe-Coding-IDEs).
|
|
||||||
# Hinter dem Gateway sitzt llama-swap (:8080), das die llama-swap-Rollen-Aliase
|
|
||||||
# (fast/heavy/vision/coder) bei Bedarf lädt — fast+heavy sind via llama-swap
|
|
||||||
# `groups` (swap:false) ko-resident, daher Delegation ohne Nachlade-Latenz.
|
|
||||||
#
|
|
||||||
# Wird von Mission Control verwaltet (services/gateway.py). Start auf der Box:
|
|
||||||
# litellm --config /opt/mission-control-2/gateway/config.yaml --port 4000
|
|
||||||
#
|
|
||||||
# HINWEIS (auf der Box verifizieren): Der KOMPLEXITÄTS-Auto-Router ("model: auto"
|
|
||||||
# wählt pro Anfrage schnell↔schwer nach Schwierigkeit) ist LiteLLM-versionsabhängig.
|
|
||||||
# Baseline unten: "auto" == fast, eskaliert per Fallback auf heavy (bei Fehler/
|
|
||||||
# Kontext-Overflow). Den echten Complexity-Router (docs.litellm.ai/docs/proxy/
|
|
||||||
# auto_routing) in Phase „Routing-Feinschliff" gegen die installierte Version setzen.
|
|
||||||
|
|
||||||
model_list:
|
|
||||||
# Schnelles Alltags-Hirn (llama-swap-Alias "fast")
|
|
||||||
- model_name: fast
|
|
||||||
litellm_params:
|
|
||||||
model: openai/fast
|
|
||||||
api_base: http://127.0.0.1:8080/v1
|
|
||||||
api_key: sk-noauth
|
|
||||||
|
|
||||||
# Schweres Reasoning-Hirn (llama-swap-Alias "heavy")
|
|
||||||
- model_name: heavy
|
|
||||||
litellm_params:
|
|
||||||
model: openai/heavy
|
|
||||||
api_base: http://127.0.0.1:8080/v1
|
|
||||||
api_key: sk-noauth
|
|
||||||
|
|
||||||
# Vision (llama-swap-Alias "vision")
|
|
||||||
- model_name: vision
|
|
||||||
litellm_params:
|
|
||||||
model: openai/vision
|
|
||||||
api_base: http://127.0.0.1:8080/v1
|
|
||||||
api_key: sk-noauth
|
|
||||||
|
|
||||||
# Coder (llama-swap-Alias "coder")
|
|
||||||
- model_name: coder
|
|
||||||
litellm_params:
|
|
||||||
model: openai/coder
|
|
||||||
api_base: http://127.0.0.1:8080/v1
|
|
||||||
api_key: sk-noauth
|
|
||||||
|
|
||||||
# "auto" — Baseline: == fast, eskaliert per Fallback auf heavy.
|
|
||||||
- model_name: auto
|
|
||||||
litellm_params:
|
|
||||||
model: openai/fast
|
|
||||||
api_base: http://127.0.0.1:8080/v1
|
|
||||||
api_key: sk-noauth
|
|
||||||
|
|
||||||
litellm_settings:
|
|
||||||
# Bei Fehler/Kontext-Overflow von "auto"/"fast" auf "heavy" eskalieren.
|
|
||||||
fallbacks:
|
|
||||||
- auto: ["heavy"]
|
|
||||||
- fast: ["heavy"]
|
|
||||||
context_window_fallbacks:
|
|
||||||
- auto: ["heavy"]
|
|
||||||
- fast: ["heavy"]
|
|
||||||
num_retries: 1
|
|
||||||
request_timeout: 600
|
|
||||||
|
|
||||||
router_settings:
|
|
||||||
routing_strategy: simple-shuffle
|
|
||||||
Reference in New Issue
Block a user