Feat: Vulkan/RADV-Engine + vocab-gepruefte Spec-Drafts + Provisioning/Sync
Engine-Cutover ROCm/HIP -> Vulkan/RADV (gfx1151): +12-22% tg auf MoE (llama-bench
verifiziert, fast 53->65 t/s). ROCm-Build bleibt als Rollback unter /opt/llamacpp.
- Backend: vocab-aware Speculative Decoding. services/gguf_meta.py liest den
Tokenizer-Fingerprint (model/pre/n_vocab) direkt aus dem GGUF-Header (ohne Modell-Load);
register_model + migrate_config haengen nur VOCAB-KOMPATIBLE Drafts an (inkl. --spec-type,
das in dieser llama.cpp-Generation noetig ist). Neue Endpoints /api/models/drafts + /{id}/draft.
- Frontend: idiotensichere Spec-Draft-UI (SpecDraftModal) - nur kompatible Drafts waehlbar,
inkompatible gesperrt mit Begruendung; SPEC/SPEC?-Badge nach echtem Aktiv-Status; Rolle in AddModel.
- maintenance.py: Engine-Update-Quelle -> ggml-org/llama.cpp (Build-Nummer-Vergleich),
ENGINE_PATH=/opt/llamacpp-vulkan.
- Startup-Warmup der brains (deploy/warmup.sh, self-detaching ExecStartPost) + deploy/provision-engine.sh.
- Cleanup: tote LiteLLM gateway/config.yaml + alle Referenzen (config.py/backup.py/backup.sh) entfernt;
README + docs/memory aktualisiert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -4,8 +4,8 @@ from pathlib import Path
|
||||
# Add backend directory to sys.path so we can import services
|
||||
sys.path.append(str(Path(__file__).resolve().parent))
|
||||
|
||||
from services.llamaswap import read_config, write_config
|
||||
from config import CONFIG_PATH, SPEC_DRAFT_MODEL_PATH
|
||||
from services.llamaswap import read_config, write_config, spec_draft_flags, _PATH_RE
|
||||
from config import CONFIG_PATH
|
||||
|
||||
def migrate():
|
||||
print(f"Reading config from {CONFIG_PATH}...")
|
||||
@@ -16,8 +16,6 @@ def migrate():
|
||||
cfg = read_config()
|
||||
models = cfg.get("models", {})
|
||||
|
||||
draft_path = SPEC_DRAFT_MODEL_PATH
|
||||
|
||||
for name, spec in models.items():
|
||||
if not isinstance(spec, dict):
|
||||
continue
|
||||
@@ -36,12 +34,15 @@ def migrate():
|
||||
aliases = spec.get("aliases", [])
|
||||
role = aliases[0] if aliases else None
|
||||
|
||||
# 3. Add parallel and speculative decoding for fast and coder
|
||||
# 3. Add parallel + (nur vocab-kompatibles) Speculative Decoding für fast/coder.
|
||||
# spec_draft_flags() prüft die Vocab-Kompatibilität und hängt --spec-type an;
|
||||
# ein inkompatibler Draft (z.B. qwen2.5 ↔ Qwen3.6) wird NICHT gesetzt.
|
||||
if role in ("fast", "coder"):
|
||||
if "--parallel" not in cmd:
|
||||
cmd = cmd.strip() + " --parallel 2"
|
||||
if "--spec-draft-model" not in cmd:
|
||||
cmd = cmd.strip() + f" --spec-draft-model {draft_path}"
|
||||
target = mt.group(1) if (mt := _PATH_RE.search(cmd)) else ""
|
||||
cmd = cmd.strip() + spec_draft_flags(target)
|
||||
|
||||
# Update cmd
|
||||
from ruamel.yaml.scalarstring import LiteralScalarString
|
||||
|
||||
Reference in New Issue
Block a user