Feat: Sprechen-Tab — mit Hermes per Sprache reden (Browser-Voice + 3D-Avatar)

Voll-Duplex Sprach-Interaktion vom lokalen PC mit dem vollen Hermes-Agenten (api_server :8642, OpenAI-kompatibel → gleiche Tools + geteiltes Mem0 wie CLI/Telegram). - Voice-Sidecar (voice_service/, eigenes Py3.12-venv ~/.voice, :8650): STT faster-whisper (medium, de) + gestuftes TTS — Piper (schnell, Default) + Chatterbox (premium, Voice-Cloning, lazy-load, CPU-Start). Analog mem0_service. - Backend: routers/voice.py (Proxy /api/voice/stt|tts|voices + /chat-SSE an Hermes mit Bearer API_SERVER_KEY + X-Hermes-Session-Id für server-seitigen Verlauf). config.py: VOICE_SERVICE_URL + HERMES_API_KEY (Fallback aus ~/.hermes/.env). System-Dienstliste + Wartung (Restart/Logs) um voice-service ergänzt. - Frontend: Sprechen-Tab mit 3D-Avatar (VRM via three-vrm) — Lippensync (Web-Audio-Pegel), Blinzeln, Sentiment-Mimik, Ruhepose. Avatar-Picker (CORS-freie Galerie + .vrm-Upload + URL + VRoid-Hub-Link) + Stimm-Auswahl. Push-to-talk (Knopf/Leertaste). Deps: three, r3f, drei. - Deploy: deploy/voice-service.service + deploy.sh (idempotenter Sidecar-Install, enable, restart). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 23:26:58 +02:00
parent 2360ad173a
commit 8e7ce1b1d3
22 changed files with 1887 additions and 54 deletions
@@ -0,0 +1,85 @@
+// Sequentielle Audio-Wiedergabe für die TTS-Antworten + Pegel-Messung fürs Lippensync.
+//
+// Die einzelnen Satz-WAVs kommen nacheinander rein (satzweise Synthese → niedrige Latenz).
+// Wir spielen sie über EINEN AudioContext geordnet ab und hängen einen AnalyserNode dazwischen,
+// dessen Energie pro Frame in `level.current` (0..1) landet — der 3D-Avatar liest das im
+// useFrame und öffnet den Mund entsprechend. Kein Re-Render pro Frame (Mutable-Ref-Muster).
+
+export class AudioQueue {
+  private ctx: AudioContext
+  private analyser: AnalyserNode
+  private queue: ArrayBuffer[] = []
+  private playing = false
+  private raf = 0
+  private freq: Uint8Array<ArrayBuffer>
+  /** Mutable, vom Avatar pro Frame gelesen. 0 = Mund zu, 1 = weit offen. */
+  readonly level = { current: 0 }
+  onSpeaking?: (speaking: boolean) => void
+
+  constructor() {
+    const Ctor = window.AudioContext || (window as any).webkitAudioContext
+    this.ctx = new Ctor()
+    this.analyser = this.ctx.createAnalyser()
+    this.analyser.fftSize = 256
+    this.analyser.smoothingTimeConstant = 0.6
+    this.analyser.connect(this.ctx.destination)
+    this.freq = new Uint8Array(new ArrayBuffer(this.analyser.frequencyBinCount))
+  }
+
+  async enqueue(buf: ArrayBuffer) {
+    this.queue.push(buf)
+    if (!this.playing) await this.playNext()
+  }
+
+  /** Laufende + wartende Wiedergabe verwerfen (z.B. wenn der Nutzer dazwischenredet). */
+  clear() {
+    this.queue = []
+  }
+
+  private async playNext(): Promise<void> {
+    const buf = this.queue.shift()
+    if (!buf) {
+      this.playing = false
+      this.stopMeter()
+      this.onSpeaking?.(false)
+      return
+    }
+    this.playing = true
+    this.onSpeaking?.(true)
+    if (this.ctx.state === "suspended") {
+      try { await this.ctx.resume() } catch { /* vom User-Gesture freigeschaltet */ }
+    }
+    let audioBuf: AudioBuffer
+    try {
+      audioBuf = await this.ctx.decodeAudioData(buf.slice(0))
+    } catch {
+      return this.playNext()   // kaputtes Segment überspringen
+    }
+    const src = this.ctx.createBufferSource()
+    src.buffer = audioBuf
+    src.connect(this.analyser)
+    src.onended = () => { void this.playNext() }
+    src.start()
+    this.startMeter()
+  }
+
+  private startMeter() {
+    cancelAnimationFrame(this.raf)
+    const tick = () => {
+      this.analyser.getByteFrequencyData(this.freq)
+      // Sprachenergie liegt v.a. in den unteren/mittleren Bändern.
+      const n = Math.min(this.freq.length, 48)
+      let sum = 0
+      for (let i = 2; i < n; i++) sum += this.freq[i]
+      const avg = sum / (n - 2) / 255
+      this.level.current = Math.min(1, avg * 1.9)
+      this.raf = requestAnimationFrame(tick)
+    }
+    tick()
+  }
+
+  private stopMeter() {
+    cancelAnimationFrame(this.raf)
+    this.level.current = 0
+  }
+}