Feat: Vulkan/RADV-Engine + vocab-gepruefte Spec-Drafts + Provisioning/Sync
Engine-Cutover ROCm/HIP -> Vulkan/RADV (gfx1151): +12-22% tg auf MoE (llama-bench
verifiziert, fast 53->65 t/s). ROCm-Build bleibt als Rollback unter /opt/llamacpp.
- Backend: vocab-aware Speculative Decoding. services/gguf_meta.py liest den
Tokenizer-Fingerprint (model/pre/n_vocab) direkt aus dem GGUF-Header (ohne Modell-Load);
register_model + migrate_config haengen nur VOCAB-KOMPATIBLE Drafts an (inkl. --spec-type,
das in dieser llama.cpp-Generation noetig ist). Neue Endpoints /api/models/drafts + /{id}/draft.
- Frontend: idiotensichere Spec-Draft-UI (SpecDraftModal) - nur kompatible Drafts waehlbar,
inkompatible gesperrt mit Begruendung; SPEC/SPEC?-Badge nach echtem Aktiv-Status; Rolle in AddModel.
- maintenance.py: Engine-Update-Quelle -> ggml-org/llama.cpp (Build-Nummer-Vergleich),
ENGINE_PATH=/opt/llamacpp-vulkan.
- Startup-Warmup der brains (deploy/warmup.sh, self-detaching ExecStartPost) + deploy/provision-engine.sh.
- Cleanup: tote LiteLLM gateway/config.yaml + alle Referenzen (config.py/backup.py/backup.sh) entfernt;
README + docs/memory aktualisiert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Hitonabi
@@ -3,7 +3,8 @@
|
||||
Komponierbarer Local-AI-Stack für den Bosgame M5. Greenfield-Neuaufbau —
|
||||
siehe Architektur-Plan (`docs/` bzw. der genehmigte Plan).
|
||||
|
||||
**Schichten:** Engine (llama-swap) · Routing-Gateway (LiteLLM, `model: auto`) ·
|
||||
**Schichten:** Engine (llama-swap, **Vulkan/RADV** auf Strix Halo) · **Builtin-Routing-Gateway**
|
||||
in MC2 (`model: auto`, kein externer LiteLLM-Dienst — scheitert auf Python 3.14) ·
|
||||
Mission Control 2.0 (FastAPI + React/shadcn) · Hermes Agent + hermes-webui ·
|
||||
Shared Memory (SQLite via MCP). Jede Schicht hinter stabilem Vertrag austauschbar.
|
||||
|
||||
@@ -13,8 +14,8 @@ Fortschritt & Resume-Guide: siehe [`docs/STATUS.md`](docs/STATUS.md).
|
||||
|
||||
- **Phase 0** — FastAPI-Skeleton + React/shadcn-Shell (Cmd+K, Dark, PWA).
|
||||
- **Phase 1** — Compute-Module (fit/caps/sources, portiert), **Discover** (live HF + Fit + Caps),
|
||||
**Engine-Write** (register + `groups`/Ko-Residenz), **LiteLLM-Gateway** (Config + Service,
|
||||
`model: auto` + Fallbacks), Frontend **Modelle & Routing** (Caps-Chips, Fit, Discover, Routing-View).
|
||||
**Engine-Write** (register + `groups`/Ko-Residenz + vocab-geprüfte Spec-Drafts),
|
||||
**Builtin-Gateway** (`model: auto` + Fallbacks), Frontend **Modelle & Routing** (Caps-Chips, Fit, Discover, Routing-View).
|
||||
|
||||
- **Phase 2** — System-Status (CPU/RAM/GPU/Disk), Wartung (restart/self-update, sudo-frei),
|
||||
**Connect** (saubere IDE-Snippets → Gateway `model:auto`, LAN-IP-Override).
|
||||
@@ -59,5 +60,9 @@ cd frontend && npm run build # → frontend/dist
|
||||
|---|---|---|
|
||||
| `MC_LLAMA_SWAP_URL` | `http://127.0.0.1:8080` | Engine |
|
||||
| `MC_CONFIG_PATH` | `/etc/llama-swap/config.yaml` | llama-swap Config |
|
||||
| `MC_GATEWAY_URL` | `http://127.0.0.1:4000` | LiteLLM-Gateway |
|
||||
| `MC_GATEWAY_URL` | `http://127.0.0.1:$MC_PORT` | Builtin-Gateway (Teil von MC2, kein externer Dienst) |
|
||||
| `MC_PORT` | `9000` | MC-Backend-Port |
|
||||
| `MC_ENGINE_PATH` | `/opt/llamacpp-vulkan` | Aktive Engine-Binary (Vulkan-Build) |
|
||||
| `MC_ENGINE_REPO` | `ggml-org/llama.cpp` | Quelle für Engine-Update-Check |
|
||||
| `MC_DRAFTS_DIR` | `$MODELS/drafts` | Spec-Draft-Modelle (vocab-geprüft) |
|
||||
| `MC_SPEC_TYPE` | `draft-simple` | Speculative-Decoding-Typ (llama.cpp) |
|
||||
|
||||
+14
-7
@@ -33,17 +33,24 @@ CMD_TEMPLATE = os.environ.get("MC_CMD_TEMPLATE", _DEFAULT_CMD_TEMPLATE)
|
||||
if "{model}" not in CMD_TEMPLATE:
|
||||
CMD_TEMPLATE = _DEFAULT_CMD_TEMPLATE
|
||||
DEFAULT_TTL = int(os.environ.get("MC_DEFAULT_TTL", "300"))
|
||||
# Draft-Modell für Speculative Decoding (nur fast/coder, wenn vorhanden). Eine
|
||||
# Quelle der Wahrheit für llamaswap.register_model + migrate_config.
|
||||
SPEC_DRAFT_MODEL_PATH = os.environ.get(
|
||||
"MC_SPEC_DRAFT_MODEL", f"{MODELS_DIR.as_posix()}/drafts/qwen2.5-1.5b-instruct-q4_k_m.gguf")
|
||||
# Verzeichnis mit Draft-Modellen für Speculative Decoding. Beim Hinzufügen eines
|
||||
# fast/coder-Modells wird hieraus automatisch ein **vocab-kompatibler** Draft gewählt
|
||||
# (Vocab-Check via services.gguf_meta; ein inkompatibler Draft lässt llama.cpp scheitern).
|
||||
DRAFTS_DIR = Path(os.environ.get("MC_DRAFTS_DIR", str(MODELS_DIR / "drafts")))
|
||||
# Optionaler expliziter Default-Draft (leer = Auto-Erkennung aus DRAFTS_DIR). Wird nur
|
||||
# verwendet, wenn er zum Ziel-Modell vocab-kompatibel ist. (Früher fix qwen2.5 → entfernt,
|
||||
# weil das mit neueren Vocabs wie Qwen3.6 inkompatibel ist und Spec stillschweigend brach.)
|
||||
SPEC_DRAFT_MODEL_PATH = os.environ.get("MC_SPEC_DRAFT_MODEL", "")
|
||||
# Speculative-Decoding-Typ (llama.cpp dieser Generation braucht --spec-type zusätzlich
|
||||
# zu --spec-draft-model, sonst ist Spec inaktiv).
|
||||
SPEC_TYPE = os.environ.get("MC_SPEC_TYPE", "draft-simple")
|
||||
# Env für HuggingFace-Downloads: XET deaktivieren (Hänger bei ~6 MB, siehe v1-Gotcha).
|
||||
HF_DOWNLOAD_ENV = {"HF_HUB_DISABLE_XET": "1"}
|
||||
|
||||
# --- Routing-Gateway (LiteLLM, model: auto) ----------------------------------
|
||||
# --- Routing-Gateway (builtin in MC2, model: auto) ---------------------------
|
||||
# MC2 IST der Gateway (services/gateway.py + routers/gateway_proxy.py). KEIN externer
|
||||
# LiteLLM-Dienst (scheitert auf Python 3.14). Daher keine Gateway-Config-Datei mehr.
|
||||
GATEWAY_URL = os.environ.get("MC_GATEWAY_URL", f"http://127.0.0.1:{os.environ.get('MC_PORT', '9000')}").rstrip("/")
|
||||
GATEWAY_CONFIG_PATH = Path(os.environ.get(
|
||||
"MC_GATEWAY_CONFIG", str(Path(__file__).resolve().parent.parent / "gateway" / "config.yaml")))
|
||||
|
||||
# --- Hermes Agent (eigener Dienst auf der Box) -------------------------------
|
||||
# Gateway (OpenAI-API des Agenten) + standalone Web-UI (nesquena/hermes-webui).
|
||||
|
||||
@@ -4,8 +4,8 @@ from pathlib import Path
|
||||
# Add backend directory to sys.path so we can import services
|
||||
sys.path.append(str(Path(__file__).resolve().parent))
|
||||
|
||||
from services.llamaswap import read_config, write_config
|
||||
from config import CONFIG_PATH, SPEC_DRAFT_MODEL_PATH
|
||||
from services.llamaswap import read_config, write_config, spec_draft_flags, _PATH_RE
|
||||
from config import CONFIG_PATH
|
||||
|
||||
def migrate():
|
||||
print(f"Reading config from {CONFIG_PATH}...")
|
||||
@@ -16,8 +16,6 @@ def migrate():
|
||||
cfg = read_config()
|
||||
models = cfg.get("models", {})
|
||||
|
||||
draft_path = SPEC_DRAFT_MODEL_PATH
|
||||
|
||||
for name, spec in models.items():
|
||||
if not isinstance(spec, dict):
|
||||
continue
|
||||
@@ -36,12 +34,15 @@ def migrate():
|
||||
aliases = spec.get("aliases", [])
|
||||
role = aliases[0] if aliases else None
|
||||
|
||||
# 3. Add parallel and speculative decoding for fast and coder
|
||||
# 3. Add parallel + (nur vocab-kompatibles) Speculative Decoding für fast/coder.
|
||||
# spec_draft_flags() prüft die Vocab-Kompatibilität und hängt --spec-type an;
|
||||
# ein inkompatibler Draft (z.B. qwen2.5 ↔ Qwen3.6) wird NICHT gesetzt.
|
||||
if role in ("fast", "coder"):
|
||||
if "--parallel" not in cmd:
|
||||
cmd = cmd.strip() + " --parallel 2"
|
||||
if "--spec-draft-model" not in cmd:
|
||||
cmd = cmd.strip() + f" --spec-draft-model {draft_path}"
|
||||
target = mt.group(1) if (mt := _PATH_RE.search(cmd)) else ""
|
||||
cmd = cmd.strip() + spec_draft_flags(target)
|
||||
|
||||
# Update cmd
|
||||
from ruamel.yaml.scalarstring import LiteralScalarString
|
||||
|
||||
@@ -158,6 +158,30 @@ def set_model_ctx(model_id: str, body: CtxReq) -> dict:
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@router.get("/models/drafts")
|
||||
def list_drafts(target: str = "") -> dict:
|
||||
"""Verfügbare Draft-Modelle + ihre Vocab-Kompatibilität zum Ziel-Modell
|
||||
(target = GGUF-Pfad). Basis für die idiotensichere Spec-Draft-Auswahl im UI."""
|
||||
return llamaswap.drafts_for(target)
|
||||
|
||||
|
||||
class DraftReq(BaseModel):
|
||||
draft_path: str | None = None
|
||||
|
||||
|
||||
@router.post("/models/{model_id}/draft")
|
||||
def set_model_draft(model_id: str, body: DraftReq) -> dict:
|
||||
"""Setzt/entfernt den Speculative-Decoding-Draft eines Modells. Inkompatible
|
||||
(oder nicht prüfbare) Drafts werden serverseitig abgelehnt."""
|
||||
try:
|
||||
res = llamaswap.set_spec_draft(model_id, body.draft_path)
|
||||
except PermissionError as exc:
|
||||
raise HTTPException(500, str(exc))
|
||||
if not res["ok"]:
|
||||
raise HTTPException(400 if "kompatib" in res["reason"].lower() else 404, res["reason"])
|
||||
return res
|
||||
|
||||
|
||||
@router.post("/models/unload")
|
||||
def unload_all_models() -> dict:
|
||||
import httpx
|
||||
|
||||
@@ -7,7 +7,7 @@ import shutil
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from config import CONFIG_PATH, GATEWAY_CONFIG_PATH, MEMORY_DB, MODELS_DIR
|
||||
from config import CONFIG_PATH, MEMORY_DB, MODELS_DIR
|
||||
|
||||
BACKUP_DIR = Path(MODELS_DIR) / "mc2-backups"
|
||||
RETAIN = 7
|
||||
@@ -31,7 +31,7 @@ def backup_now() -> dict:
|
||||
dst.mkdir(parents=True, exist_ok=True)
|
||||
saved = []
|
||||
for src in (MEMORY_DB, Path(str(MEMORY_DB) + "-wal"), Path(str(MEMORY_DB) + "-shm"),
|
||||
CONFIG_PATH, GATEWAY_CONFIG_PATH):
|
||||
CONFIG_PATH):
|
||||
if (name := _safe_copy(src, dst)):
|
||||
saved.append(name)
|
||||
# Aufräumen: nur die letzten RETAIN Snapshots behalten.
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
"""
|
||||
GGUF-Tokenizer-Fingerprint — liest die Tokenizer-Identität direkt aus dem
|
||||
GGUF-Header (ohne das Modell zu laden), um zu entscheiden, ob ein Draft-Modell
|
||||
**vocab-kompatibel** mit einem Ziel-Modell ist (Voraussetzung für Speculative
|
||||
Decoding in llama.cpp — sonst: "draft model vocab type must match target").
|
||||
|
||||
Wir lesen nur die Metadaten-KV-Sektion am Dateianfang und brechen ab, sobald
|
||||
`tokenizer.ggml.tokens` erreicht ist (dessen Länge = n_vocab). model+pre+n_vocab
|
||||
identifizieren den Tokenizer eindeutig genug, um die in der Praxis relevanten
|
||||
Fälle zu unterscheiden (Qwen2.5 vs Qwen3 vs Qwen3.6 etc.). Die llama.cpp-Prüfung
|
||||
beim Laden bleibt der letzte Schiedsrichter.
|
||||
"""
|
||||
|
||||
import struct
|
||||
from functools import lru_cache
|
||||
|
||||
# GGUF value types (https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)
|
||||
_T_UINT8, _T_INT8, _T_UINT16, _T_INT16, _T_UINT32, _T_INT32, _T_FLOAT32, \
|
||||
_T_BOOL, _T_STRING, _T_ARRAY, _T_UINT64, _T_INT64, _T_FLOAT64 = range(13)
|
||||
|
||||
_SCALAR_FMT = {
|
||||
_T_UINT8: "<B", _T_INT8: "<b", _T_UINT16: "<H", _T_INT16: "<h",
|
||||
_T_UINT32: "<I", _T_INT32: "<i", _T_FLOAT32: "<f", _T_BOOL: "<?",
|
||||
_T_UINT64: "<Q", _T_INT64: "<q", _T_FLOAT64: "<d",
|
||||
}
|
||||
_SCALAR_SIZE = {t: struct.calcsize(f) for t, f in _SCALAR_FMT.items()}
|
||||
|
||||
_WANT_STRINGS = {"tokenizer.ggml.model", "tokenizer.ggml.pre", "general.architecture"}
|
||||
|
||||
|
||||
class _Reader:
|
||||
def __init__(self, f):
|
||||
self.f = f
|
||||
|
||||
def read(self, n: int) -> bytes:
|
||||
b = self.f.read(n)
|
||||
if len(b) != n:
|
||||
raise EOFError("unerwartetes Dateiende beim GGUF-Parsen")
|
||||
return b
|
||||
|
||||
def u32(self) -> int:
|
||||
return struct.unpack("<I", self.read(4))[0]
|
||||
|
||||
def u64(self) -> int:
|
||||
return struct.unpack("<Q", self.read(8))[0]
|
||||
|
||||
def gstr(self) -> str:
|
||||
n = self.u64()
|
||||
return self.read(n).decode("utf-8", "replace")
|
||||
|
||||
def skip_value(self, vtype: int) -> None:
|
||||
"""Liest einen Wert und verwirft ihn (um den Datei-Pointer korrekt
|
||||
weiterzuschieben). Arrays werden elementweise konsumiert."""
|
||||
if vtype == _T_STRING:
|
||||
self.f.seek(self.u64(), 1)
|
||||
elif vtype in _SCALAR_SIZE:
|
||||
self.f.seek(_SCALAR_SIZE[vtype], 1)
|
||||
elif vtype == _T_ARRAY:
|
||||
etype = self.u32()
|
||||
count = self.u64()
|
||||
if etype == _T_STRING:
|
||||
for _ in range(count):
|
||||
self.f.seek(self.u64(), 1)
|
||||
elif etype in _SCALAR_SIZE:
|
||||
self.f.seek(_SCALAR_SIZE[etype] * count, 1)
|
||||
else:
|
||||
raise ValueError(f"unbekannter Array-Elementtyp {etype}")
|
||||
else:
|
||||
raise ValueError(f"unbekannter GGUF-Wertetyp {vtype}")
|
||||
|
||||
|
||||
def _read_fingerprint(path: str) -> dict | None:
|
||||
"""Liest model/pre/n_vocab aus dem GGUF-Header. None bei Fehler/kein GGUF."""
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
r = _Reader(fh)
|
||||
if r.read(4) != b"GGUF":
|
||||
return None
|
||||
r.u32() # version
|
||||
r.u64() # tensor_count
|
||||
kv_count = r.u64()
|
||||
fp: dict = {"model": None, "pre": None, "arch": None, "n_vocab": None}
|
||||
for _ in range(kv_count):
|
||||
key = r.gstr()
|
||||
vtype = r.u32()
|
||||
if key == "tokenizer.ggml.tokens" and vtype == _T_ARRAY:
|
||||
etype = r.u32()
|
||||
fp["n_vocab"] = r.u64()
|
||||
# Wir haben alles (model/pre kommen vor tokens) → abbrechen.
|
||||
if etype != _T_STRING:
|
||||
return None
|
||||
break
|
||||
if key in _WANT_STRINGS and vtype == _T_STRING:
|
||||
val = r.gstr()
|
||||
if key == "tokenizer.ggml.model":
|
||||
fp["model"] = val
|
||||
elif key == "tokenizer.ggml.pre":
|
||||
fp["pre"] = val
|
||||
else:
|
||||
fp["arch"] = val
|
||||
else:
|
||||
r.skip_value(vtype)
|
||||
if fp["model"] is None and fp["n_vocab"] is None:
|
||||
return None
|
||||
return fp
|
||||
except (OSError, EOFError, ValueError, struct.error):
|
||||
return None
|
||||
|
||||
|
||||
@lru_cache(maxsize=256)
|
||||
def _cached(path: str, mtime: float, size: int) -> tuple | None:
|
||||
fp = _read_fingerprint(path)
|
||||
if fp is None:
|
||||
return None
|
||||
return (fp.get("model"), fp.get("pre"), fp.get("n_vocab"), fp.get("arch"))
|
||||
|
||||
|
||||
def fingerprint(path: str) -> dict | None:
|
||||
"""Tokenizer-Fingerprint eines GGUF (gecacht nach Pfad+mtime+size).
|
||||
Returns dict(model, pre, n_vocab, arch) oder None wenn nicht lesbar."""
|
||||
import os
|
||||
try:
|
||||
st = os.stat(path)
|
||||
except OSError:
|
||||
return None
|
||||
t = _cached(path, st.st_mtime, st.st_size)
|
||||
if t is None:
|
||||
return None
|
||||
return {"model": t[0], "pre": t[1], "n_vocab": t[2], "arch": t[3]}
|
||||
|
||||
|
||||
def vocab_key(path: str) -> tuple | None:
|
||||
"""Vergleichsschlüssel für Vocab-Kompatibilität: (model, pre, n_vocab).
|
||||
Genau diese Identität verlangt llama.cpp für Speculative Decoding."""
|
||||
fp = fingerprint(path)
|
||||
if not fp or fp["n_vocab"] is None:
|
||||
return None
|
||||
return (fp["model"], fp["pre"], fp["n_vocab"])
|
||||
|
||||
|
||||
def compatible(target_path: str, draft_path: str) -> bool | None:
|
||||
"""True/False ob draft vocab-kompatibel zum target ist. None = unbestimmbar
|
||||
(eine Datei nicht lesbar) → UI behandelt das als 'nicht bestätigt'."""
|
||||
a = vocab_key(target_path)
|
||||
b = vocab_key(draft_path)
|
||||
if a is None or b is None:
|
||||
return None
|
||||
return a == b
|
||||
@@ -13,7 +13,10 @@ import re
|
||||
import httpx
|
||||
from ruamel.yaml.scalarstring import LiteralScalarString
|
||||
|
||||
from config import CMD_TEMPLATE, CONFIG_PATH, DEFAULT_TTL, LLAMA_SWAP_URL, SPEC_DRAFT_MODEL_PATH
|
||||
from config import (
|
||||
CMD_TEMPLATE, CONFIG_PATH, DEFAULT_TTL, DRAFTS_DIR, LLAMA_SWAP_URL,
|
||||
SPEC_DRAFT_MODEL_PATH, SPEC_TYPE,
|
||||
)
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -66,6 +69,11 @@ def _parse_model(name: str, spec: dict) -> dict:
|
||||
m_draft = re.search(r"--spec-draft-model\s+([^\s]+)", cmd)
|
||||
if m_draft:
|
||||
spec_draft = os.path.basename(m_draft.group(1).replace("'", "").replace('"', ""))
|
||||
spec_type = None
|
||||
if (m_st := re.search(r"--spec-type\s+([^\s]+)", cmd)):
|
||||
spec_type = m_st.group(1)
|
||||
# Spec ist nur AKTIV, wenn BEIDES gesetzt ist (--spec-draft-model UND --spec-type).
|
||||
spec_active = bool(spec_draft and spec_type)
|
||||
parallel_match = re.search(r"--parallel\s+(\d+)", cmd)
|
||||
parallel_slots = int(parallel_match.group(1)) if parallel_match else 1
|
||||
|
||||
@@ -85,6 +93,8 @@ def _parse_model(name: str, spec: dict) -> dict:
|
||||
"incomplete": not path,
|
||||
"prompt_cache": prompt_cache,
|
||||
"spec_draft_model": spec_draft,
|
||||
"spec_type": spec_type,
|
||||
"spec_active": spec_active,
|
||||
"parallel_slots": parallel_slots,
|
||||
"capabilities": capabilities(
|
||||
name=filename or name, cmd=cmd,
|
||||
@@ -191,8 +201,10 @@ def register_model(model_path: str, role: str | None = None, ctx: int = 8192,
|
||||
if role_lower in ("fast", "coder"):
|
||||
if "--parallel" not in cmd:
|
||||
cmd += " --parallel 2"
|
||||
if os.path.exists(SPEC_DRAFT_MODEL_PATH) and "--spec-draft-model" not in cmd:
|
||||
cmd += f" --spec-draft-model {SPEC_DRAFT_MODEL_PATH}"
|
||||
# Nur einen VOCAB-KOMPATIBLEN Draft anhängen (sonst scheitert llama.cpp).
|
||||
# Bei frischem Install existiert die GGUF noch nicht → kein Draft (später im UI setzbar).
|
||||
if "--spec-draft-model" not in cmd:
|
||||
cmd += spec_draft_flags(model_path)
|
||||
|
||||
cfg.setdefault("models", {})[model_id] = {
|
||||
"cmd": LiteralScalarString(cmd + "\n"),
|
||||
@@ -203,6 +215,101 @@ def register_model(model_path: str, role: str | None = None, ctx: int = 8192,
|
||||
return model_id
|
||||
|
||||
|
||||
# --- Speculative-Draft / Vocab-Kompatibilität --------------------------------
|
||||
def list_drafts() -> list[dict]:
|
||||
"""Alle Draft-GGUFs in DRAFTS_DIR mit Tokenizer-Fingerprint."""
|
||||
from services import gguf_meta
|
||||
out = []
|
||||
if DRAFTS_DIR.is_dir():
|
||||
for p in sorted(DRAFTS_DIR.glob("*.gguf")):
|
||||
out.append({
|
||||
"path": str(p), "filename": p.name,
|
||||
"size_bytes": p.stat().st_size if p.exists() else None,
|
||||
"vocab": gguf_meta.fingerprint(str(p)),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def find_compatible_draft(target_path: str) -> str | None:
|
||||
"""Pfad eines vocab-kompatiblen Drafts für target_path, oder None.
|
||||
Bevorzugt MC_SPEC_DRAFT_MODEL (falls gesetzt+kompatibel), sonst der erste
|
||||
kompatible Draft in DRAFTS_DIR. None auch, wenn target (noch) fehlt (dann
|
||||
nicht verifizierbar → bewusst KEIN Draft anhängen)."""
|
||||
if not target_path or not os.path.exists(target_path):
|
||||
return None
|
||||
from services import gguf_meta
|
||||
candidates: list[str] = []
|
||||
if SPEC_DRAFT_MODEL_PATH and os.path.exists(SPEC_DRAFT_MODEL_PATH):
|
||||
candidates.append(SPEC_DRAFT_MODEL_PATH)
|
||||
for d in list_drafts():
|
||||
if d["path"] not in candidates:
|
||||
candidates.append(d["path"])
|
||||
for c in candidates:
|
||||
if gguf_meta.compatible(target_path, c) is True:
|
||||
return c
|
||||
return None
|
||||
|
||||
|
||||
def spec_draft_flags(target_path: str) -> str:
|
||||
"""llama-server-Flags für Speculative Decoding (Draft + --spec-type), oder ''
|
||||
wenn kein kompatibler Draft existiert. Beides nötig, sonst ist Spec inaktiv."""
|
||||
d = find_compatible_draft(target_path)
|
||||
return f" --spec-draft-model {d} --spec-type {SPEC_TYPE}" if d else ""
|
||||
|
||||
|
||||
def drafts_for(target_path: str) -> dict:
|
||||
"""Für die UI: alle Drafts + ihre Kompatibilität zum Ziel-Modell.
|
||||
compatible=None heißt 'nicht prüfbar' (Ziel- oder Draft-GGUF fehlt)."""
|
||||
from services import gguf_meta
|
||||
exists = bool(target_path and os.path.exists(target_path))
|
||||
drafts = list_drafts()
|
||||
for d in drafts:
|
||||
d["compatible"] = gguf_meta.compatible(target_path, d["path"]) if exists else None
|
||||
return {
|
||||
"target_path": target_path,
|
||||
"target_exists": exists,
|
||||
"target_vocab": gguf_meta.fingerprint(target_path) if exists else None,
|
||||
"drafts": drafts,
|
||||
}
|
||||
|
||||
|
||||
def set_spec_draft(model_id: str, draft_path: str | None) -> dict:
|
||||
"""Setzt (oder entfernt mit draft_path=None) den Spec-Draft eines Modells.
|
||||
Validiert die Vocab-Kompatibilität — ein inkompatibler/unprüfbarer Draft wird
|
||||
abgelehnt (idiotensicher). Returns {ok, reason}."""
|
||||
cfg = read_config()
|
||||
spec = (cfg.get("models") or {}).get(model_id)
|
||||
if not isinstance(spec, dict):
|
||||
return {"ok": False, "reason": "Modell nicht gefunden"}
|
||||
cmd = str(spec.get("cmd", ""))
|
||||
# vorhandene Spec-Flags entfernen (idempotent)
|
||||
cmd = re.sub(r"\s+--spec-draft-model\s+\S+", "", cmd)
|
||||
cmd = re.sub(r"\s+--spec-type\s+\S+", "", cmd)
|
||||
|
||||
if draft_path:
|
||||
# relative Angabe (nur Dateiname) gegen DRAFTS_DIR auflösen
|
||||
if not os.path.isabs(draft_path) and "/" not in draft_path:
|
||||
draft_path = str(DRAFTS_DIR / draft_path)
|
||||
if not os.path.exists(draft_path):
|
||||
return {"ok": False, "reason": "Draft-Datei nicht gefunden"}
|
||||
from services import gguf_meta
|
||||
target = ""
|
||||
if (mt := _PATH_RE.search(cmd)):
|
||||
target = mt.group(1).replace("'", "").replace('"', "")
|
||||
comp = gguf_meta.compatible(target, draft_path) if os.path.exists(target) else None
|
||||
if comp is not True:
|
||||
reason = ("Draft ist NICHT vocab-kompatibel zum Modell — Speculative Decoding "
|
||||
"würde beim Laden scheitern."
|
||||
if comp is False else
|
||||
"Kompatibilität nicht prüfbar (Modell-GGUF fehlt) — Draft nicht gesetzt.")
|
||||
return {"ok": False, "reason": reason}
|
||||
cmd = cmd.rstrip() + f" --spec-draft-model {draft_path} --spec-type {SPEC_TYPE}"
|
||||
|
||||
spec["cmd"] = LiteralScalarString(cmd.rstrip() + "\n")
|
||||
write_config(cfg)
|
||||
return {"ok": True, "reason": ""}
|
||||
|
||||
|
||||
# --- Groups (Ko-Residenz) ----------------------------------------------------
|
||||
def set_group(group: str, members: list[str], swap: bool = False, persist: bool = False) -> None:
|
||||
"""llama-swap-`groups`-Eintrag setzen. swap=False → alle Mitglieder dürfen
|
||||
|
||||
@@ -7,6 +7,7 @@ erweiterte sudoers (siehe docs/BEDIENUNG.md). Lange Ops laufen als jobengine-Job
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
from datetime import datetime
|
||||
@@ -21,10 +22,30 @@ SYSTEM_SERVICES = {"llama-swap"}
|
||||
USER_SERVICES = {"mission-control-2", "hermes-gateway", "hermes-dashboard", "hermes-webui"}
|
||||
|
||||
ENGINE_UPDATE_CMD = os.environ.get("MC_ENGINE_UPDATE_CMD", "")
|
||||
ENGINE_PATH = os.environ.get("MC_ENGINE_PATH", "/opt/llamacpp")
|
||||
# Engine = offizieller Vulkan-Build von ggml-org/llama.cpp (RADV auf Strix Halo).
|
||||
ENGINE_PATH = os.environ.get("MC_ENGINE_PATH", "/opt/llamacpp-vulkan")
|
||||
ENGINE_REPO = os.environ.get("MC_ENGINE_REPO", "ggml-org/llama.cpp")
|
||||
_engine_cache = {"ts": 0.0, "avail": False}
|
||||
|
||||
|
||||
def _installed_engine_build() -> int | None:
|
||||
"""Build-Nummer der installierten llama-server-Binary (z.B. 9821), oder None.
|
||||
Vulkan-Build braucht LD_LIBRARY_PATH=ENGINE_PATH zum Start von --version."""
|
||||
bin_path = os.path.join(ENGINE_PATH, "llama-server")
|
||||
if not os.path.exists(bin_path):
|
||||
return None
|
||||
try:
|
||||
env = dict(os.environ, LD_LIBRARY_PATH=ENGINE_PATH)
|
||||
out = subprocess.run([bin_path, "--version"], capture_output=True, text=True,
|
||||
timeout=20, env=env)
|
||||
txt = (out.stderr or "") + (out.stdout or "")
|
||||
if (m := re.search(r"build:\s*\S+\s*\((\d+)\)", txt)) or (m := re.search(r"\bb(\d{3,})\b", txt)):
|
||||
return int(m.group(1))
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _ram_gb() -> float:
|
||||
return psutil.virtual_memory().total / (1024 ** 3)
|
||||
|
||||
@@ -45,11 +66,16 @@ def _engine_update_available() -> bool:
|
||||
return _engine_cache["avail"]
|
||||
avail = False
|
||||
try:
|
||||
rel = httpx.get("https://api.github.com/repos/lemonade-sdk/llamacpp-rocm/releases/latest",
|
||||
rel = httpx.get(f"https://api.github.com/repos/{ENGINE_REPO}/releases/latest",
|
||||
timeout=6, headers={"User-Agent": "MissionControl2"}).json()
|
||||
pub = datetime.fromisoformat(rel["published_at"].replace("Z", "+00:00")).timestamp()
|
||||
inst = os.path.getmtime(ENGINE_PATH)
|
||||
avail = pub > inst + 86400
|
||||
tag = str(rel.get("tag_name", ""))
|
||||
latest = int(m.group(1)) if (m := re.search(r"(\d{3,})", tag)) else None
|
||||
installed = _installed_engine_build()
|
||||
if latest is not None and installed is not None:
|
||||
avail = latest > installed # präziser Build-Nummer-Vergleich
|
||||
else: # Fallback: Release-Datum vs. Engine-mtime
|
||||
pub = datetime.fromisoformat(rel["published_at"].replace("Z", "+00:00")).timestamp()
|
||||
avail = pub > os.path.getmtime(ENGINE_PATH) + 86400
|
||||
except Exception:
|
||||
avail = False
|
||||
_engine_cache.update(ts=now, avail=avail)
|
||||
|
||||
+1
-2
@@ -6,11 +6,10 @@ set -euo pipefail
|
||||
MODELS_DIR="${MC_MODELS_DIR:-/srv/models}"
|
||||
MEM_DB="${MC_MEMORY_DB:-$MODELS_DIR/mc2-memory.db}"
|
||||
LSWAP="${MC_CONFIG_PATH:-/etc/llama-swap/config.yaml}"
|
||||
GW="${MC_GATEWAY_CONFIG:-/opt/mission-control-2/gateway/config.yaml}"
|
||||
DEST="$MODELS_DIR/mc2-backups/$(date +%Y%m%d-%H%M%S)"
|
||||
|
||||
mkdir -p "$DEST"
|
||||
for f in "$MEM_DB" "$MEM_DB-wal" "$MEM_DB-shm" "$LSWAP" "$GW"; do
|
||||
for f in "$MEM_DB" "$MEM_DB-wal" "$MEM_DB-shm" "$LSWAP"; do
|
||||
[ -f "$f" ] && cp -p "$f" "$DEST/" || true
|
||||
done
|
||||
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env bash
|
||||
# Provisioniert die Inferenz-Engine auf der AI Box (Strix Halo / Ryzen AI MAX+ 395, gfx1151)
|
||||
# auf **Vulkan/RADV** — reproduzierbar. Braucht root (sudo).
|
||||
#
|
||||
# sudo bash ~/mission-control-v2/deploy/provision-engine.sh
|
||||
#
|
||||
# Hintergrund: Auf gfx1151 ist Vulkan/RADV ggü. ROCm/HIP messbar schneller
|
||||
# (Token-Gen +12–22 %, Prefill gleich; auf der Box per llama-bench verifiziert 2026-06-27)
|
||||
# UND einfacher. Der ROCm-Build bleibt unter /opt/llamacpp als Rollback liegen.
|
||||
set -euo pipefail
|
||||
|
||||
VULKAN_DIR=/opt/llamacpp-vulkan
|
||||
WARMUP_DST=/usr/local/bin/llama-swap-warmup.sh
|
||||
DROPIN_DIR=/etc/systemd/system/llama-swap.service.d
|
||||
SRC_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
# Optional gepinnter Build (z.B. b9821). Leer = neuester Release.
|
||||
PIN_BUILD="${MC_ENGINE_BUILD:-}"
|
||||
|
||||
echo "==> 1. RADV-Treiber + Vulkan-Runtime"
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -qq
|
||||
apt-get install -y mesa-vulkan-drivers libvulkan1 vulkan-tools curl jq
|
||||
|
||||
echo "==> 2. llama.cpp Vulkan-Build nach $VULKAN_DIR"
|
||||
if [ -n "$PIN_BUILD" ]; then
|
||||
TAG="$PIN_BUILD"
|
||||
else
|
||||
TAG="$(curl -s https://api.github.com/repos/ggml-org/llama.cpp/releases/latest | jq -r .tag_name)"
|
||||
fi
|
||||
ASSET="llama-${TAG}-bin-ubuntu-vulkan-x64.tar.gz"
|
||||
URL="https://github.com/ggml-org/llama.cpp/releases/download/${TAG}/${ASSET}"
|
||||
TMP="$(mktemp -d)"
|
||||
echo " Lade $URL"
|
||||
curl -sL -o "$TMP/v.tgz" "$URL"
|
||||
mkdir -p "$VULKAN_DIR"
|
||||
tar xzf "$TMP/v.tgz" -C "$TMP"
|
||||
# Tarball entpackt nach .../llama-<tag>/ — Inhalt flach nach $VULKAN_DIR
|
||||
cp -rf "$TMP"/llama-*/. "$VULKAN_DIR"/
|
||||
rm -rf "$TMP"
|
||||
test -x "$VULKAN_DIR/llama-server"
|
||||
|
||||
echo "==> 3. Symlink llama-server -> Vulkan-Build"
|
||||
ln -sfn "$VULKAN_DIR/llama-server" /usr/local/bin/llama-server
|
||||
|
||||
echo "==> 4. systemd Drop-ins für llama-swap (LD_LIBRARY_PATH + Brain-Warmup)"
|
||||
mkdir -p "$DROPIN_DIR"
|
||||
cat > "$DROPIN_DIR/vulkan.conf" <<EOF
|
||||
[Service]
|
||||
Environment=LD_LIBRARY_PATH=$VULKAN_DIR
|
||||
EOF
|
||||
install -m 0755 "$SRC_DIR/warmup.sh" "$WARMUP_DST"
|
||||
cat > "$DROPIN_DIR/warmup.conf" <<EOF
|
||||
[Service]
|
||||
# Nach jedem (Re)Start die brains vorladen. Das Skript detacht sich selbst (blockiert
|
||||
# den Start nicht); '-' macht den Aufruf fehlertolerant (kann llama-swap nie failen lassen).
|
||||
ExecStartPost=-$WARMUP_DST
|
||||
EOF
|
||||
|
||||
echo "==> 5. Reload + Restart"
|
||||
systemctl daemon-reload
|
||||
systemctl restart llama-swap
|
||||
sleep 3
|
||||
systemctl is-active llama-swap
|
||||
|
||||
echo "==> Fertig. Aktive Engine:"
|
||||
readlink -f /usr/local/bin/llama-server
|
||||
vulkaninfo --summary 2>/dev/null | grep -m1 deviceName || true
|
||||
echo "Rollback auf ROCm: ln -sfn /opt/llamacpp/llama-server /usr/local/bin/llama-server && rm $DROPIN_DIR/vulkan.conf && systemctl daemon-reload && systemctl restart llama-swap"
|
||||
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
# Lädt die "brains" (hermes/fast/vision) nach einem llama-swap-(Re)Start vor,
|
||||
# damit die ERSTE Anfrage nicht kalt ist (llama-swap lädt sonst lazy bei Bedarf).
|
||||
# Eingehängt als ExecStartPost=-/usr/local/bin/llama-swap-warmup.sh in llama-swap.
|
||||
#
|
||||
# Selbst-detachend: der ExecStartPost-Aufruf kehrt sofort zurück (blockiert den
|
||||
# Service-Start NICHT); die eigentliche Aufwärmung läuft entkoppelt im Hintergrund.
|
||||
set -u
|
||||
URL="${MC_LLAMA_SWAP_URL:-http://127.0.0.1:8080}"
|
||||
BRAINS="${MC_WARMUP_MODELS:-hermes fast vision}"
|
||||
|
||||
if [ "${1:-}" != "--inner" ]; then
|
||||
setsid "$0" --inner >/dev/null 2>&1 &
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# --- ab hier im entkoppelten Hintergrundprozess ---
|
||||
for _ in $(seq 1 60); do # auf llama-swap warten (max ~120s)
|
||||
curl -sf "$URL/v1/models" >/dev/null 2>&1 && break
|
||||
sleep 2
|
||||
done
|
||||
|
||||
for m in $BRAINS; do
|
||||
curl -s -m 180 -X POST "$URL/v1/chat/completions" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"model\":\"$m\",\"max_tokens\":1,\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}" \
|
||||
>/dev/null 2>&1 || true
|
||||
done
|
||||
@@ -23,7 +23,25 @@ metadata:
|
||||
## Dienste auf der AI Box (Stand 2026-06-27)
|
||||
| Dienst | Port | Beschreibung |
|
||||
|---|---|---|
|
||||
| `llama-swap` (system) | `:8080` | Engine, ROCm/HIP |
|
||||
| `llama-swap` (system) | `:8080` | Engine, **Vulkan/RADV** (seit 2026-06-27, vorher ROCm/HIP) |
|
||||
|
||||
### Engine-Backend: Vulkan/RADV (Cutover 2026-06-27)
|
||||
- **Gemessen:** RADV schlägt ROCm/HIP auf gfx1151 bei tg um **+12–22 %** (fast 53→65 t/s roh), Prefill gleich.
|
||||
„ROCm gewinnt Prefill" gilt hier NICHT; auch bei 32K Tiefe bleibt RADV vorn. hipBLASLt bringt nichts.
|
||||
- **Setup:** Vulkan-llama.cpp (offizieller Build b9821) in `/opt/llamacpp-vulkan`; Symlink
|
||||
`/usr/local/bin/llama-server` → dorthin; systemd Drop-in `/etc/systemd/system/llama-swap.service.d/vulkan.conf`
|
||||
mit `LD_LIBRARY_PATH=/opt/llamacpp-vulkan`. Treiber: `mesa-vulkan-drivers` (RADV STRIX_HALO, Mesa 26.0.3).
|
||||
- **ROCm-Build bleibt** unter `/opt/llamacpp` (lemonade llamacpp-rocm) als Rollback liegen.
|
||||
- **Rollback:** `sudo ln -sfn /opt/llamacpp/llama-server /usr/local/bin/llama-server` + Drop-in löschen + `sudo systemctl daemon-reload && sudo systemctl restart llama-swap`.
|
||||
- **ACHTUNG maintenance.py:** `_engine_update_available()` trackt noch `lemonade-sdk/llamacpp-rocm` — passt nicht
|
||||
mehr zum aktiven Vulkan-Build (b9821 von ggml-org/llama.cpp). Engine-Update-Quelle anpassen.
|
||||
- **Spec-Draft (2026-06-27 gefixt):** `qwen2.5-1.5b` war vocab-inkompatibel mit Qwen3.6 UND `--spec-type` fehlte
|
||||
→ war inaktiv. Auch Qwen3-0.6B ist mit Qwen3.6 inkompatibel → **`fast` läuft jetzt ohne Draft** (kein
|
||||
kompatibler verfügbar). **`coder` (Qwen3-Coder-Next)**: Qwen3-0.6B IST kompatibel → Draft
|
||||
`/srv/models/drafts/Qwen3-0.6B-Q8_0.gguf` + `--spec-type draft-simple`, **0,75 Akzeptanz** (echter Coding-Speedup).
|
||||
Alter `qwen2.5-1.5b-instruct-q4_k_m.gguf` in drafts/ ist jetzt ungenutzt (löschbar, 1,1 GB).
|
||||
- **Modell-cmds (config.yaml):** llama.cpp dieser Generation braucht `--spec-type` zusätzlich zu
|
||||
`--spec-draft-model`, sonst ist Spec inaktiv. Drafts müssen **exakt vocab-gleich** zum Target sein.
|
||||
| `mission-control-2` (user) | `:9001` | MC2 Backend + Frontend |
|
||||
| `hermes-gateway` (user) | `:8642` | Hermes Agent API |
|
||||
| `hermes-webui` (user) | `:8787` | nesquena hermes-webui |
|
||||
|
||||
-1
File diff suppressed because one or more lines are too long
+1
File diff suppressed because one or more lines are too long
-395
File diff suppressed because one or more lines are too long
+395
File diff suppressed because one or more lines are too long
Vendored
+2
-2
@@ -7,8 +7,8 @@
|
||||
<link rel="manifest" href="/manifest.webmanifest" />
|
||||
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
|
||||
<title>Mission Control 2.0</title>
|
||||
<script type="module" crossorigin src="/assets/index-D4mBflTE.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-C0dfTDFj.css">
|
||||
<script type="module" crossorigin src="/assets/index-DPqptZjm.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-CjZVMvhL.css">
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
import { useState } from "react"
|
||||
import { X, Check, Zap, AlertTriangle } from "lucide-react"
|
||||
import { api, type ModelInfo, type DraftInfo } from "@/lib/api"
|
||||
import { useDrafts } from "@/lib/queries"
|
||||
import { fmtSize } from "@/lib/format"
|
||||
import { cn } from "@/lib/utils"
|
||||
|
||||
/**
|
||||
* Idiotensichere Speculative-Decoding-Konfiguration für EIN Modell.
|
||||
* Zeigt nur VOCAB-KOMPATIBLE Drafts als wählbar; inkompatible werden gesperrt
|
||||
* und mit Begründung angezeigt. So kann nie ein kaputter Draft gesetzt werden
|
||||
* (der das Modell beim Laden scheitern ließe).
|
||||
*/
|
||||
export function SpecDraftModal({
|
||||
model, onClose, onChanged,
|
||||
}: { model: ModelInfo; onClose: () => void; onChanged: () => void }) {
|
||||
const { data, isLoading } = useDrafts(model.gguf_path)
|
||||
const [busy, setBusy] = useState<string | null>(null)
|
||||
const [err, setErr] = useState("")
|
||||
|
||||
const tv = data?.target_vocab
|
||||
const drafts = data?.drafts ?? []
|
||||
const compatibles = drafts.filter((d) => d.compatible === true)
|
||||
const currentFile = model.spec_draft_model
|
||||
|
||||
async function apply(draftPath: string | null) {
|
||||
setBusy(draftPath ?? "__clear__")
|
||||
setErr("")
|
||||
try {
|
||||
await api(`/api/models/${encodeURIComponent(model.name)}/draft`, {
|
||||
method: "POST",
|
||||
body: JSON.stringify({ draft_path: draftPath }),
|
||||
})
|
||||
onChanged()
|
||||
onClose()
|
||||
} catch (e: any) {
|
||||
setErr(String(e?.message || e))
|
||||
setBusy(null)
|
||||
}
|
||||
}
|
||||
|
||||
const vocabLabel = (v?: { pre: string | null; n_vocab: number | null } | null) =>
|
||||
v ? `${v.pre ?? "?"} · ${v.n_vocab?.toLocaleString() ?? "?"} Tokens` : "—"
|
||||
|
||||
return (
|
||||
<div className="fixed inset-0 z-50 bg-black/60 backdrop-blur-sm flex items-center justify-center p-4">
|
||||
<div className="w-full max-w-lg rounded-2xl border border-border/80 bg-card p-6 shadow-2xl space-y-4 animate-in fade-in zoom-in-95 duration-200">
|
||||
<div className="flex items-center justify-between border-b border-border/20 pb-2">
|
||||
<h3 className="text-xs font-bold uppercase tracking-wider text-primary flex items-center gap-2">
|
||||
<Zap className="h-4 w-4" /> Speculative Draft
|
||||
</h3>
|
||||
<button onClick={onClose} className="p-1 rounded hover:bg-accent text-muted-foreground hover:text-foreground cursor-pointer">
|
||||
<X className="h-4 w-4" />
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="text-xs text-muted-foreground leading-relaxed">
|
||||
Beschleunigt die Token-Generierung mit einem kleinen "Draft"-Modell. Voraussetzung:
|
||||
der Draft muss den <strong className="text-foreground">exakt gleichen Tokenizer (Vocab)</strong> haben
|
||||
wie das Modell — sonst lehnt llama.cpp es ab.
|
||||
</div>
|
||||
|
||||
{/* Ziel-Vocab */}
|
||||
<div className="rounded-lg border border-border/40 bg-background/30 px-3 py-2 text-[11px] font-mono flex items-center justify-between">
|
||||
<span className="text-muted-foreground">{model.name.split("/").pop()?.replace(/\.gguf$/i, "")}</span>
|
||||
<span className="text-foreground">Vocab: {vocabLabel(tv)}</span>
|
||||
</div>
|
||||
|
||||
{/* Aktueller Zustand */}
|
||||
{model.spec_active && currentFile && (
|
||||
<div className="rounded-lg border border-emerald-500/30 bg-emerald-500/5 px-3 py-2 flex items-center justify-between gap-2">
|
||||
<span className="text-[11px] text-emerald-400 font-mono flex items-center gap-1.5 truncate">
|
||||
<Check className="h-3.5 w-3.5 shrink-0" /> Aktiv: {currentFile}
|
||||
</span>
|
||||
<button
|
||||
onClick={() => apply(null)}
|
||||
disabled={busy !== null}
|
||||
className="h-6 px-2.5 rounded border border-amber-500/30 bg-amber-500/5 hover:bg-amber-500/10 text-amber-400 text-[9px] font-bold uppercase shrink-0 cursor-pointer disabled:opacity-50"
|
||||
>
|
||||
Deaktivieren
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{!data?.target_exists && (
|
||||
<div className="rounded-lg border border-amber-500/30 bg-amber-500/5 px-3 py-2 text-[11px] text-amber-400 flex items-center gap-2">
|
||||
<AlertTriangle className="h-3.5 w-3.5 shrink-0" />
|
||||
Modell-GGUF noch nicht vorhanden — Spec-Draft ist nach dem Download konfigurierbar.
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Draft-Liste */}
|
||||
<div className="space-y-1.5 max-h-64 overflow-y-auto pr-1">
|
||||
{isLoading ? (
|
||||
<div className="text-xs text-muted-foreground py-6 text-center">Prüfe Vocab-Kompatibilität…</div>
|
||||
) : drafts.length === 0 ? (
|
||||
<div className="text-[11px] text-muted-foreground border border-dashed border-border/50 rounded-lg p-4 text-center">
|
||||
Keine Draft-Modelle in <code className="font-mono">/srv/models/drafts</code>.
|
||||
Lade ein kleines same-family-Modell (z.B. Qwen3-0.6B) und lege es dort ab.
|
||||
</div>
|
||||
) : (
|
||||
drafts.map((d: DraftInfo) => {
|
||||
const isCurrent = d.filename === currentFile
|
||||
const ok = d.compatible === true
|
||||
return (
|
||||
<div
|
||||
key={d.path}
|
||||
className={cn(
|
||||
"rounded-lg border px-3 py-2 flex items-center justify-between gap-2 text-left",
|
||||
ok ? "border-border/40 bg-background/20" : "border-border/20 bg-background/10 opacity-60",
|
||||
isCurrent && "border-primary/40 bg-primary/10",
|
||||
)}
|
||||
>
|
||||
<div className="min-w-0">
|
||||
<div className="text-[11px] font-mono font-semibold text-foreground truncate">{d.filename}</div>
|
||||
<div className="text-[9px] text-muted-foreground font-mono">
|
||||
{fmtSize(d.size_bytes)} · Vocab: {vocabLabel(d.vocab)}
|
||||
</div>
|
||||
</div>
|
||||
{ok ? (
|
||||
isCurrent ? (
|
||||
<span className="text-[9px] font-bold uppercase text-primary flex items-center gap-1 shrink-0"><Check className="h-3.5 w-3.5" /> Aktiv</span>
|
||||
) : (
|
||||
<button
|
||||
onClick={() => apply(d.path)}
|
||||
disabled={busy !== null}
|
||||
className="h-7 px-3 rounded-lg border border-emerald-500/30 bg-emerald-500/5 hover:bg-emerald-500/15 text-emerald-400 text-[9px] font-bold uppercase shrink-0 cursor-pointer disabled:opacity-50"
|
||||
>
|
||||
Aktivieren
|
||||
</button>
|
||||
)
|
||||
) : (
|
||||
<span
|
||||
className="text-[9px] font-bold uppercase text-amber-400/80 flex items-center gap-1 shrink-0"
|
||||
title={d.compatible === false
|
||||
? `Inkompatibel: Tokenizer/Vocab weicht ab (Draft ${d.vocab?.pre}/${d.vocab?.n_vocab} ≠ Modell ${tv?.pre}/${tv?.n_vocab}).`
|
||||
: "Kompatibilität nicht prüfbar (Datei fehlt)."}
|
||||
>
|
||||
<AlertTriangle className="h-3.5 w-3.5" /> {d.compatible === false ? "Vocab ≠" : "n/a"}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
})
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Hinweis, wenn Drafts da sind aber keiner kompatibel */}
|
||||
{!isLoading && data?.target_exists && drafts.length > 0 && compatibles.length === 0 && (
|
||||
<div className="rounded-lg border border-amber-500/30 bg-amber-500/5 px-3 py-2 text-[11px] text-amber-400 leading-relaxed">
|
||||
Kein vocab-kompatibler Draft verfügbar — Speculative Decoding ist für dieses Modell nicht möglich.
|
||||
Es braucht einen Draft mit identischem Tokenizer (pre=<span className="font-mono">{tv?.pre}</span>,
|
||||
n_vocab=<span className="font-mono">{tv?.n_vocab?.toLocaleString()}</span>).
|
||||
</div>
|
||||
)}
|
||||
|
||||
{err && <div className="text-[10px] text-red-400 font-mono">{err}</div>}
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -88,10 +88,34 @@ export interface ModelInfo {
|
||||
incomplete: boolean
|
||||
prompt_cache: boolean
|
||||
spec_draft_model: string | null
|
||||
spec_type: string | null
|
||||
spec_active: boolean
|
||||
parallel_slots: number
|
||||
capabilities: Capabilities
|
||||
}
|
||||
|
||||
export interface VocabFingerprint {
|
||||
model: string | null
|
||||
pre: string | null
|
||||
n_vocab: number | null
|
||||
arch: string | null
|
||||
}
|
||||
|
||||
export interface DraftInfo {
|
||||
path: string
|
||||
filename: string
|
||||
size_bytes: number | null
|
||||
vocab: VocabFingerprint | null
|
||||
compatible: boolean | null // null = nicht prüfbar (Ziel-GGUF fehlt)
|
||||
}
|
||||
|
||||
export interface DraftsResp {
|
||||
target_path: string
|
||||
target_exists: boolean
|
||||
target_vocab: VocabFingerprint | null
|
||||
drafts: DraftInfo[]
|
||||
}
|
||||
|
||||
export interface DiscoverModel {
|
||||
name: string
|
||||
author: string
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
type AgentStatus,
|
||||
type ConnectResp,
|
||||
type DiscoverResp,
|
||||
type DraftsResp,
|
||||
type Health,
|
||||
type Job,
|
||||
type Memory,
|
||||
@@ -31,6 +32,7 @@ export const qk = {
|
||||
agentStatus: ["agent-status"] as const,
|
||||
updates: ["updates"] as const,
|
||||
discover: ["discover"] as const,
|
||||
drafts: (target?: string) => ["drafts", target ?? ""] as const,
|
||||
connect: (params?: string) => ["connect", params ?? ""] as const,
|
||||
memory: (q?: string, category?: string) => ["memory", q ?? "", category ?? ""] as const,
|
||||
}
|
||||
@@ -70,6 +72,14 @@ export const useUpdates = (refetchInterval?: number) =>
|
||||
export const useDiscover = () =>
|
||||
useQuery({ queryKey: qk.discover, queryFn: () => api<DiscoverResp>("/api/discover") })
|
||||
|
||||
// Verfügbare Spec-Draft-Modelle + Vocab-Kompatibilität zum Ziel-Modell (GGUF-Pfad).
|
||||
export const useDrafts = (target?: string) =>
|
||||
useQuery({
|
||||
queryKey: qk.drafts(target),
|
||||
queryFn: () => api<DraftsResp>(`/api/models/drafts?target=${encodeURIComponent(target ?? "")}`),
|
||||
enabled: !!target,
|
||||
})
|
||||
|
||||
export const useConnect = (params?: string) =>
|
||||
useQuery({
|
||||
queryKey: qk.connect(params),
|
||||
|
||||
@@ -7,8 +7,10 @@ export function AddModel() {
|
||||
const [quants, setQuants] = useState<string[]>([])
|
||||
const [quant, setQuant] = useState("Q4_K_M")
|
||||
const [msg, setMsg] = useState("")
|
||||
const [role, setRole] = useState("")
|
||||
const [q, setQ] = useState("")
|
||||
const [results, setResults] = useState<{ repo: string; downloads: number }[]>([])
|
||||
const ROLE_OPTS = ["fast", "heavy", "coder", "vision", "scout"]
|
||||
|
||||
async function loadQuants(r?: string) {
|
||||
const rr = r ?? repo
|
||||
@@ -42,10 +44,16 @@ export function AddModel() {
|
||||
setMsg("Download-Job wird initiiert...")
|
||||
try {
|
||||
await api("/api/models/install", {
|
||||
method: "POST",
|
||||
body: JSON.stringify({ repo, quant, jinja: true }),
|
||||
method: "POST",
|
||||
body: JSON.stringify({ repo, quant, role: role || undefined, jinja: true }),
|
||||
})
|
||||
setMsg(`Download gestartet: ${repo} (${quant}). Fortschritt wird oben angezeigt.`)
|
||||
setMsg(
|
||||
`Download gestartet: ${repo} (${quant})${role ? `, Rolle: ${role}` : ""}. ` +
|
||||
`Fortschritt oben.` +
|
||||
(role === "fast" || role === "coder"
|
||||
? " Speculative Draft (Vocab-geprüft) kannst du nach dem Download an der Modellkarte (»Spec«) setzen."
|
||||
: "")
|
||||
)
|
||||
} catch (e) {
|
||||
setMsg(`Download-Fehler: ${e}`)
|
||||
}
|
||||
@@ -71,13 +79,22 @@ export function AddModel() {
|
||||
</button>
|
||||
{quants.length > 0 && (
|
||||
<>
|
||||
<select
|
||||
value={quant}
|
||||
onChange={(e) => setQuant(e.target.value)}
|
||||
<select
|
||||
value={quant}
|
||||
onChange={(e) => setQuant(e.target.value)}
|
||||
className="h-9 rounded-lg border border-border/60 bg-background/40 px-3 text-xs outline-none text-foreground font-semibold"
|
||||
>
|
||||
{quants.map((qq) => <option key={qq} value={qq} className="bg-popover text-foreground">{qq}</option>)}
|
||||
</select>
|
||||
<select
|
||||
value={role}
|
||||
onChange={(e) => setRole(e.target.value)}
|
||||
title="Rolle (optional) — bestimmt Auto-Konfiguration wie parallele Slots"
|
||||
className="h-9 rounded-lg border border-border/60 bg-background/40 px-3 text-xs outline-none text-foreground font-semibold"
|
||||
>
|
||||
<option value="" className="bg-popover text-foreground">Rolle…</option>
|
||||
{ROLE_OPTS.map((r) => <option key={r} value={r} className="bg-popover text-foreground">{r}</option>)}
|
||||
</select>
|
||||
<button
|
||||
onClick={install}
|
||||
className="h-9 px-4 rounded-lg bg-primary text-primary-foreground text-xs font-semibold hover:opacity-90 transition-all cursor-pointer flex items-center gap-1.5"
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
import { useState, useRef, useCallback } from "react"
|
||||
import { Download, Trash2, Edit3, Activity, HardDrive, X, Check, Copy } from "lucide-react"
|
||||
import { api } from "@/lib/api"
|
||||
import { Download, Trash2, Edit3, Activity, HardDrive, X, Check, Copy, Zap } from "lucide-react"
|
||||
import { api, type ModelInfo } from "@/lib/api"
|
||||
import { useModels, useRouting, useConnect, useUpdates, useQueryClient, qk } from "@/lib/queries"
|
||||
import { useDialog } from "@/lib/useDialog"
|
||||
import { CapsChips } from "@/components/CapsChips"
|
||||
import { cn } from "@/lib/utils"
|
||||
import { fmtSize, fmtCtx } from "@/lib/format"
|
||||
import { getBrandInfo, ROLES } from "@/components/models/ModelBadges"
|
||||
import { SpecDraftModal } from "@/components/models/SpecDraftModal"
|
||||
|
||||
export function Cockpit() {
|
||||
const qc = useQueryClient()
|
||||
@@ -26,6 +27,7 @@ export function Cockpit() {
|
||||
// UI state
|
||||
const [activeClient, setActiveClient] = useState<"roocode" | "cursor" | "opencode" | "zed" | "continue" | null>(null)
|
||||
const [activeRoleForAssign, setActiveRoleForAssign] = useState<string | null>(null)
|
||||
const [specModel, setSpecModel] = useState<ModelInfo | null>(null)
|
||||
const [copied, setCopied] = useState(false)
|
||||
const [hoveredNode, setHoveredNode] = useState<string | null>(null)
|
||||
const [viewMode, setViewMode] = useState<"grid" | "list">("grid")
|
||||
@@ -742,11 +744,15 @@ export function Cockpit() {
|
||||
PC
|
||||
</span>
|
||||
)}
|
||||
{m.spec_draft_model && (
|
||||
{m.spec_active ? (
|
||||
<span className="px-1.5 py-0.5 rounded bg-emerald-500/10 border border-emerald-500/20 text-[9px] font-mono text-emerald-400 font-bold uppercase" title={`Speculative Decoding aktiv (Draft: ${m.spec_draft_model})`}>
|
||||
SPEC
|
||||
</span>
|
||||
)}
|
||||
) : m.spec_draft_model ? (
|
||||
<span className="px-1.5 py-0.5 rounded bg-amber-500/10 border border-amber-500/20 text-[9px] font-mono text-amber-400 font-bold uppercase" title={`Draft gesetzt (${m.spec_draft_model}), aber INAKTIV — --spec-type fehlt oder Vocab inkompatibel. Über 'Spec' neu konfigurieren.`}>
|
||||
SPEC?
|
||||
</span>
|
||||
) : null}
|
||||
{m.parallel_slots > 1 && (
|
||||
<span className="px-1.5 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-[9px] font-mono text-violet-400 font-bold uppercase" title={`${m.parallel_slots} parallele Slots aktiv`}>
|
||||
SLOTS: {m.parallel_slots}
|
||||
@@ -825,6 +831,18 @@ export function Cockpit() {
|
||||
>
|
||||
Ctx
|
||||
</button>
|
||||
<button
|
||||
onClick={() => setSpecModel(m)}
|
||||
className={cn(
|
||||
"h-7 px-2.5 rounded-lg border text-[9px] font-bold uppercase transition-colors cursor-pointer flex items-center gap-1",
|
||||
m.spec_active ? "border-emerald-500/30 text-emerald-400 hover:bg-emerald-500/10"
|
||||
: m.spec_draft_model ? "border-amber-500/30 text-amber-400 hover:bg-amber-500/10"
|
||||
: "border-border/40 text-muted-foreground hover:bg-accent"
|
||||
)}
|
||||
title="Speculative Draft konfigurieren (Vocab-geprüft)"
|
||||
>
|
||||
<Zap className="h-3 w-3" /> Spec
|
||||
</button>
|
||||
<button
|
||||
onClick={() => handleDelete(m.name)}
|
||||
className="h-7 w-7 rounded-lg border border-border/40 text-muted-foreground hover:text-red-400 hover:bg-red-500/5 flex items-center justify-center transition-colors cursor-pointer"
|
||||
@@ -885,11 +903,15 @@ export function Cockpit() {
|
||||
PC
|
||||
</span>
|
||||
)}
|
||||
{m.spec_draft_model && (
|
||||
{m.spec_active ? (
|
||||
<span className="px-1.5 py-0.5 rounded bg-emerald-500/10 border border-emerald-500/20 text-[8px] font-mono text-emerald-400 font-bold uppercase shrink-0" title={`Speculative Decoding aktiv (Draft: ${m.spec_draft_model})`}>
|
||||
SPEC
|
||||
</span>
|
||||
)}
|
||||
) : m.spec_draft_model ? (
|
||||
<span className="px-1.5 py-0.5 rounded bg-amber-500/10 border border-amber-500/20 text-[8px] font-mono text-amber-400 font-bold uppercase shrink-0" title={`Draft gesetzt (${m.spec_draft_model}), aber INAKTIV — --spec-type fehlt oder Vocab inkompatibel.`}>
|
||||
SPEC?
|
||||
</span>
|
||||
) : null}
|
||||
{m.parallel_slots > 1 && (
|
||||
<span className="px-1.5 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-[8px] font-mono text-violet-400 font-bold uppercase shrink-0" title={`${m.parallel_slots} parallele Slots aktiv`}>
|
||||
SLOTS: {m.parallel_slots}
|
||||
@@ -937,15 +959,27 @@ export function Cockpit() {
|
||||
>
|
||||
{isRunning ? "Entladen" : "Laden"}
|
||||
</button>
|
||||
<button
|
||||
<button
|
||||
onClick={() => handleSetCtx(m.name, m.ctx)}
|
||||
className="h-7 px-2.5 rounded-lg border border-border/40 text-[9px] font-bold uppercase hover:bg-accent text-muted-foreground transition-all cursor-pointer"
|
||||
title="Kontextlänge anpassen"
|
||||
>
|
||||
Ctx
|
||||
</button>
|
||||
<button
|
||||
onClick={() => handleDelete(m.name)}
|
||||
<button
|
||||
onClick={() => setSpecModel(m)}
|
||||
className={cn(
|
||||
"h-7 px-2.5 rounded-lg border text-[9px] font-bold uppercase transition-all cursor-pointer flex items-center gap-1",
|
||||
m.spec_active ? "border-emerald-500/30 text-emerald-400 hover:bg-emerald-500/10"
|
||||
: m.spec_draft_model ? "border-amber-500/30 text-amber-400 hover:bg-amber-500/10"
|
||||
: "border-border/40 text-muted-foreground hover:bg-accent"
|
||||
)}
|
||||
title="Speculative Draft konfigurieren (Vocab-geprüft)"
|
||||
>
|
||||
<Zap className="h-3 w-3" /> Spec
|
||||
</button>
|
||||
<button
|
||||
onClick={() => handleDelete(m.name)}
|
||||
className="h-7 w-7 rounded-lg flex items-center justify-center border border-border/40 text-muted-foreground hover:text-red-400 hover:bg-red-500/5 transition-all cursor-pointer"
|
||||
title="Modell löschen"
|
||||
>
|
||||
@@ -1016,6 +1050,14 @@ export function Cockpit() {
|
||||
</div>
|
||||
)}
|
||||
|
||||
{specModel && (
|
||||
<SpecDraftModal
|
||||
model={specModel}
|
||||
onClose={() => setSpecModel(null)}
|
||||
onChanged={reload}
|
||||
/>
|
||||
)}
|
||||
|
||||
{dialogElement}
|
||||
</div>
|
||||
)
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
# LiteLLM-Gateway-Config für Mission Control 2.0
|
||||
# ----------------------------------------------------------------------------
|
||||
# EIN OpenAI-Endpunkt für ALLE Konsumenten (Hermes + Vibe-Coding-IDEs).
|
||||
# Hinter dem Gateway sitzt llama-swap (:8080), das die llama-swap-Rollen-Aliase
|
||||
# (fast/heavy/vision/coder) bei Bedarf lädt — fast+heavy sind via llama-swap
|
||||
# `groups` (swap:false) ko-resident, daher Delegation ohne Nachlade-Latenz.
|
||||
#
|
||||
# Wird von Mission Control verwaltet (services/gateway.py). Start auf der Box:
|
||||
# litellm --config /opt/mission-control-2/gateway/config.yaml --port 4000
|
||||
#
|
||||
# HINWEIS (auf der Box verifizieren): Der KOMPLEXITÄTS-Auto-Router ("model: auto"
|
||||
# wählt pro Anfrage schnell↔schwer nach Schwierigkeit) ist LiteLLM-versionsabhängig.
|
||||
# Baseline unten: "auto" == fast, eskaliert per Fallback auf heavy (bei Fehler/
|
||||
# Kontext-Overflow). Den echten Complexity-Router (docs.litellm.ai/docs/proxy/
|
||||
# auto_routing) in Phase „Routing-Feinschliff" gegen die installierte Version setzen.
|
||||
|
||||
model_list:
|
||||
# Schnelles Alltags-Hirn (llama-swap-Alias "fast")
|
||||
- model_name: fast
|
||||
litellm_params:
|
||||
model: openai/fast
|
||||
api_base: http://127.0.0.1:8080/v1
|
||||
api_key: sk-noauth
|
||||
|
||||
# Schweres Reasoning-Hirn (llama-swap-Alias "heavy")
|
||||
- model_name: heavy
|
||||
litellm_params:
|
||||
model: openai/heavy
|
||||
api_base: http://127.0.0.1:8080/v1
|
||||
api_key: sk-noauth
|
||||
|
||||
# Vision (llama-swap-Alias "vision")
|
||||
- model_name: vision
|
||||
litellm_params:
|
||||
model: openai/vision
|
||||
api_base: http://127.0.0.1:8080/v1
|
||||
api_key: sk-noauth
|
||||
|
||||
# Coder (llama-swap-Alias "coder")
|
||||
- model_name: coder
|
||||
litellm_params:
|
||||
model: openai/coder
|
||||
api_base: http://127.0.0.1:8080/v1
|
||||
api_key: sk-noauth
|
||||
|
||||
# "auto" — Baseline: == fast, eskaliert per Fallback auf heavy.
|
||||
- model_name: auto
|
||||
litellm_params:
|
||||
model: openai/fast
|
||||
api_base: http://127.0.0.1:8080/v1
|
||||
api_key: sk-noauth
|
||||
|
||||
litellm_settings:
|
||||
# Bei Fehler/Kontext-Overflow von "auto"/"fast" auf "heavy" eskalieren.
|
||||
fallbacks:
|
||||
- auto: ["heavy"]
|
||||
- fast: ["heavy"]
|
||||
context_window_fallbacks:
|
||||
- auto: ["heavy"]
|
||||
- fast: ["heavy"]
|
||||
num_retries: 1
|
||||
request_timeout: 600
|
||||
|
||||
router_settings:
|
||||
routing_strategy: simple-shuffle
|
||||
Reference in New Issue
Block a user