Feat: Vulkan/RADV-Engine + vocab-gepruefte Spec-Drafts + Provisioning/Sync
Engine-Cutover ROCm/HIP -> Vulkan/RADV (gfx1151): +12-22% tg auf MoE (llama-bench
verifiziert, fast 53->65 t/s). ROCm-Build bleibt als Rollback unter /opt/llamacpp.
- Backend: vocab-aware Speculative Decoding. services/gguf_meta.py liest den
Tokenizer-Fingerprint (model/pre/n_vocab) direkt aus dem GGUF-Header (ohne Modell-Load);
register_model + migrate_config haengen nur VOCAB-KOMPATIBLE Drafts an (inkl. --spec-type,
das in dieser llama.cpp-Generation noetig ist). Neue Endpoints /api/models/drafts + /{id}/draft.
- Frontend: idiotensichere Spec-Draft-UI (SpecDraftModal) - nur kompatible Drafts waehlbar,
inkompatible gesperrt mit Begruendung; SPEC/SPEC?-Badge nach echtem Aktiv-Status; Rolle in AddModel.
- maintenance.py: Engine-Update-Quelle -> ggml-org/llama.cpp (Build-Nummer-Vergleich),
ENGINE_PATH=/opt/llamacpp-vulkan.
- Startup-Warmup der brains (deploy/warmup.sh, self-detaching ExecStartPost) + deploy/provision-engine.sh.
- Cleanup: tote LiteLLM gateway/config.yaml + alle Referenzen (config.py/backup.py/backup.sh) entfernt;
README + docs/memory aktualisiert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -158,6 +158,30 @@ def set_model_ctx(model_id: str, body: CtxReq) -> dict:
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@router.get("/models/drafts")
|
||||
def list_drafts(target: str = "") -> dict:
|
||||
"""Verfügbare Draft-Modelle + ihre Vocab-Kompatibilität zum Ziel-Modell
|
||||
(target = GGUF-Pfad). Basis für die idiotensichere Spec-Draft-Auswahl im UI."""
|
||||
return llamaswap.drafts_for(target)
|
||||
|
||||
|
||||
class DraftReq(BaseModel):
|
||||
draft_path: str | None = None
|
||||
|
||||
|
||||
@router.post("/models/{model_id}/draft")
|
||||
def set_model_draft(model_id: str, body: DraftReq) -> dict:
|
||||
"""Setzt/entfernt den Speculative-Decoding-Draft eines Modells. Inkompatible
|
||||
(oder nicht prüfbare) Drafts werden serverseitig abgelehnt."""
|
||||
try:
|
||||
res = llamaswap.set_spec_draft(model_id, body.draft_path)
|
||||
except PermissionError as exc:
|
||||
raise HTTPException(500, str(exc))
|
||||
if not res["ok"]:
|
||||
raise HTTPException(400 if "kompatib" in res["reason"].lower() else 404, res["reason"])
|
||||
return res
|
||||
|
||||
|
||||
@router.post("/models/unload")
|
||||
def unload_all_models() -> dict:
|
||||
import httpx
|
||||
|
||||
Reference in New Issue
Block a user