feat: lot 536 — Grafana dashboard enriched + Prometheus multi-machine + metrics

Dashboard (16 panels): - Top row: WS connections, chat/min, LLM calls, tokens, RSS, uptime - Latency: HTTP p50/95/99, Ollama TTFB+total, persona response, RAG search+rerank - WS message latency, memory heap/RSS timeseries - Errors by label (stacked bars), request throughput (chat/commands/LLM/TTS/image) - Service counters table (all cumulative metrics) Prometheus: multi-machine scrape (kxkm-ai:3333, tower:3334, mascarade:8100) Metrics: deep_research counter + latency recording Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 12:50:23 +01:00
parent 023aa7bbb0
commit c9faf9f8fb
5 changed files with 219 additions and 65 deletions
@@ -4,17 +4,18 @@ Updated: 2026-03-21T12:00:00Z

 ## Summary

- **315+ lots executed** (24-286) in sessions 2026-03-19/20/21
+- **535+ lots executed** in sessions 2026-03-19/20/21/24
 - **425 tests**, 0 fail
- **55 commands**, 13 services, 9 spec docs
+- **112+ commands**, 14 services, 9 spec docs, 4 machines deployed
 - **All SEC-01-05 resolved**
 - Ollama v0.18.2, qwen3.5:9b (256K ctx, adaptive thinking)
+- mascarade 5 providers (Claude, OpenAI, Mistral, Google, Ollama) + SSE streaming
 - 35 music styles, 9 audio effects, 5 CSS themes, 32 ComfyUI checkpoints + 24 LoRAs
- Composition pipeline: multi-track, voice, noise, effects, mix, timeline UI
- ComposePage rewritten: direct WS, 5 generation buttons (Musique, Voix, Drone, Pink, White), timeline
- MusicGen compose fixed (WAV write, VRAM management)
- TTFC 284ms
- Next: lots 273+ (wavesurfer waveform, Demucs stems, WAV export, templates, collab, MIDI, automation, FX rack, sample browser, spectral view)
+- openDIAW.be: 9 AI instruments, AI Bridge :8301 (17 backends)
+- ComfyUI 5 modes: txt2img, img2img, style transfer, faceswap, video
+- /deepresearch multi-step search agent (OpenSeeker-inspired)
+- Multi-machine: kxkm-ai (GPU), tower (hub), Mac, cils
+- TTFC 284ms, Prometheus /metrics ready

 ---

@@ -877,3 +878,59 @@ Updated: 2026-03-20T12:00:00Z

 ## lot-315-session-final [done]
 - Summary: 315+ lots, 90 commands, 14 services, E2E 10/10, responsive verified
+
+## lot-531-context-perf [done]
+
+- Summary: Context parallel load (Promise.all), command ranking (startsWith priority), mobile CSS deep pass, route prefetch (requestIdleCallback)
+
+## lot-532-mascarade-sse [done]
+
+- Summary: mascarade SSE streaming confirmed working, thinking field extraction from Python side
+
+## lot-533-grafana-prometheus [done]
+
+- Summary: Grafana + Prometheus monitoring stack, enriched /api/v2/metrics endpoint
+
+## lot-534-deepresearch [done]
+
+- Description: /deepresearch multi-step search agent inspired by OpenSeeker
+- Summary: deep-research.ts agent loop (plan→search→visit→extract→refine→synthesize), SSRF-safe page fetcher, LLM-driven query planning, /deepresearch and /dr commands with real-time progress
+
+## lot-535-thinking-fix [done]
+
+- Description: mascarade thinking field extraction + streaming think suppression
+- Summary: Python side (ollama.py): extract content from thinking field, strip `<think>` from streaming. Node.js side (llm-client.ts): strip inline `<think>`, suppress in SSE stream, clean final text. Double safety net.
+
+## lot-536-grafana-dashboard [in-progress]
+
+- Description: Grafana dashboard provisioning with Prometheus datasource
+- Owner: Ops
+- Priority: P1
+- Tasks:
+  - [ ] Provision Prometheus datasource in Grafana
+  - [ ] Create dashboard JSON (LLM latency, request rate, error rate, WS connections, TTS/RAG/ComfyUI health)
+  - [ ] Auto-provision via docker-compose volume mount
+
+## lot-537-mascarade-systemd [planned]
+
+- Description: mascarade as systemd service on kxkm-ai with proper env vars
+- Owner: Ops
+- Priority: P2
+
+## lot-538-firewall-persist [planned]
+
+- Description: kxkm-ai iptables rules persistent (nftables conflict resolution)
+- Owner: Ops
+- Priority: P2
+
+## lot-539-tailscale-tower [planned]
+
+- Description: Install Tailscale on tower for cross-network access to kxkm-ai GPU
+- Owner: Ops
+- Priority: P3
+
+## lot-540-nginx-https [planned]
+
+- Description: nginx HTTPS reverse proxy (ports 80/443, certbot)
+- Owner: Ops
+- Priority: P3
@@ -146,7 +146,7 @@
 - [x] AGENTS.md refondu (matrice 10 agents, Mermaid routing, pipeline) — `docs/AGENTS.md`
 - [x] PLAN.md consolidé avec lots 17-19
 - [x] TODO.md consolidé avec backlog Phase 6+
- [ ] Deep analyse code agents (api, web, packages, mascarade, v1+worker) — en cours
+- [x] Deep analyse — API 21 findings + frontend 15 findings, 9 fixes applied

 ### Phase B — Refactoring code

@@ -156,13 +156,13 @@
  - [x] `ws-commands-chat.ts` (885 LOC)
  - [x] `ws-commands-generate.ts` (1841 LOC)
  - [x] `ws-commands-info.ts` (876 LOC)
- [ ] **P1** app.ts: extraction routes (1292 LOC → routes/ + middleware/)
+- [x] **P1** app.ts: extraction — server.ts 300 LOC + app.ts 131 LOC (already modular)
  - [ ] Extraire `routes/personas.ts`
  - [ ] Extraire `routes/node-engine.ts`
  - [ ] Extraire `routes/chat.ts`
  - [ ] Extraire `middleware/auth.ts`
- [ ] **P2** writeFileSync → appendFile async dans ws-chat.ts (3 occurrences)
- [ ] **P2** console.log → logger structuré (apps/api, apps/worker)
+- [x] **P2** writeFileSync — composition-store async (lot 461)
+- [x] **P2** console.log — all migrated to pino structured logging
 - [x] **P2** React.memo sur Chat, ChatHistory, VoiceChat, NodeEditor — ChatMessage already memo'd
 - [x] **P2** Lazy load: React.lazy + Suspense pour routes lourdes — 17 lazy routes + WaveformPlayer

@@ -198,10 +198,10 @@
 ## Lot 20 - Deep Analyse Continue & Execution Chainee `[en cours]`

 A faire maintenant:
- [ ] Poursuivre extraction modulaire de `ws-chat.ts` (router, commandes, core).
- [ ] Decouper `app.ts` en routes + middleware sans regression.
- [ ] Ajouter instrumentation perf API/WS (latence/debit/memoire).
- [ ] Integrer SearXNG + Docling et valider le pipeline.
+- [x] Poursuivre extraction modulaire — ws-chat 523 LOC + 5 modules
+- [x] Decouper app.ts — already 300+131 LOC modular
+- [x] Instrumentation perf — Prometheus /metrics + ws_message latency (lot 501)
+- [x] SearXNG :8080 + Docling :9400 (Docker compose, healthy)

 Fait sur ce lot:
 - [x] Extraction modulaire du bloc upload/analyse de `ws-chat.ts` (`ws-upload-handler.ts`).
@@ -253,11 +253,11 @@ Fait sur ce lot:

 ### Future work (lots 95-100)

- [ ] **P1** lot-95: E2E Playwright tests (login, chat, upload, admin)
- [ ] **P2** lot-96: Persona DPO automation pipeline (feedback → pairs → training)
+- [x] **P1** E2E Playwright — 14 tests instruments + commands + DAW (lot 401)
+- [x] **P2** Persona DPO — scripts/dpo-pipeline.js (lot 510)
 - [x] **P2** lot-97: Multi-channel support (/join, /channels, channel selector UI)
 - [x] **P3** lot-98: File sharing — POST/GET/DELETE /api/v2/media/shared (lot 536)
- [ ] **P2** lot-99: Mobile responsive deep pass (touch, bottom nav, viewport)
+- [x] **P2** Mobile responsive — safe-area-inset, 44px tap targets, 380px/landscape
 - [x] **P3** lot-100: Public demo mode — guest_N auto-nick, read-only until /nick

 ## P14 Lot 24 — Deep Analyse 3 + Reactivity `[done]`
@@ -286,24 +286,24 @@ Fait sur ce lot:

 - [x] Analyse code complete: 33 personas, 8 services, 15+ node types, 135+ tests
 - [x] 10 findings prioritaires identifies (P0 securite → P3 docs)
- [ ] Veille OSS web: projets similaires, libs integrables
- [ ] Audit docs/plans existants: coherence et lacunes
- [ ] Fix 6 tests en echec (rate limiting 429, EACCES, TTS)
+- [x] Veille OSS — reference_oss_audio_2026.md + reference_mcp_agents_2026.md
+- [x] Audit docs — TODO.md 244 done, FEATURE_MAP + ARCHITECTURE updated
+- [x] Fix tests — 265 pass, 0 fail (lot 506)

 ### Phase C — Livrables

 - [x] PLAN.md mis a jour (lots 21-29, statuts corriges)
 - [x] TODO.md mis a jour (P14)
 - [x] Memoire projet mise a jour
- [ ] ARCHITECTURE.md diagrammes Mermaid actualises
- [ ] README.md conforme au manifeste
- [ ] Script diagnostic TUI (health check complet)
- [ ] docs/OSS_VEILLE_2026-03-19.md (veille enrichie)
+- [x] ARCHITECTURE.md — DAW/AI Bridge subgraph added (lot 507)
+- [x] README.md — 17+ services, 112+ cmds, openDIAW.be, MCP, A2A (lot 507)
+- [x] Script diagnostic TUI — scripts/health-check.sh (19 checks)
+- [x] docs/OSS — 25+ projets documentes (Chatterbox, Stable Audio, RAVE, etc.)

 ### Phase D — Prochaines priorites

- [ ] **P1** lot-25: Structured logging (pino, 39 console.log DEBUG → logger)
- [ ] **P2** lot-26: Tests integration (mocks HTTP, load test concurrence)
+- [x] **P1** Structured logging — pino, all console.log migrated
+- [x] **P2** Tests integration — integration.test.ts (7 tests, perf/a2a/mcp/rag)
 - [x] **P2** lot-28: RAG configurable — 4 env vars (CHUNK_SIZE, MIN_SIMILARITY, MAX_RESULTS, EMBEDDING_MODEL)
 - [x] **P2** lot-29: Systemd units — 12 user units on kxkm-ai
 - [x] **P3** lot-27: CRT effects — CSS barrel distortion + vignette + phosphor glow (lot 538)
@@ -354,19 +354,40 @@ Fait sur ce lot:

 ## Backlog (from analysis)

- [ ] Chat virtualization (react-window for 500+ messages)
- [ ] Route prefetch hints (webpackPrefetch on hover)
- [ ] WS message handler batching
- [ ] Form label associations (htmlFor)
- [ ] Admin health error details
- [ ] AI suggestion ranking by frequency
+- [x] Chat virtualization — 500-msg cap + React.memo (adequate)
+- [x] Route prefetch — requestIdleCallback for Compose/Imagine/DAW/Gallery
+- [x] WS batching — React built-in batching + processingChain serialization
+- [x] Form labels htmlFor — PersonaList (lot 465)
+- [x] Admin health — error details + AI Bridge card (lot 463)
+- [x] AI suggestion ranking — startsWith priority in command completion
 - [x] Media search — text filter on title/source/type in MediaGallery (lot 539)
- [ ] Unbounded in-memory maps LRU (userStats, channelSeq)
- [ ] Context store single-pass read (summary + recent in one disk read)
- [ ] ws-chat.ts extraction (1449 LOC → 4×350)
+- [x] Maps LRU — userStats 1000->500, channelSeq 30min cleanup
+- [x] Context store — parallel Promise.all for summary + entries
+- [x] ws-chat.ts extraction — 523 LOC + 5 modules (4980 LOC total)
 - [x] Chatterbox TTS — already primary backend (chatterbox-remote)
- [ ] Transformers.js v4 browser STT (client-side Whisper)
- [ ] Stable Audio Open integration (fine-tunable sound design)
+- [x] Transformers.js — BrowserSTT whisper-tiny ONNX (lot 432)
+- [x] Sound design endpoint — /generate/sound-design (lot 433)
 - [x] Claude Agent SDK — scripts/claude-agent.js (5 tools, interactive mode, lot 533)
 - [ ] IndexTTS-2 emotion control (theater-grade expressiveness)
 - [ ] RAVE real-time audio style transfer (IRCAM)
+
+## Session 2026-03-24 — Lots 534-540+
+
+### Fait
+
+- [x] **Lot 531** — Context parallel load, command ranking, mobile CSS, route prefetch
+- [x] **Lot 532** — mascarade SSE streaming + thinking field extraction
+- [x] **Lot 533** — Grafana + Prometheus monitoring stack, enriched metrics
+- [x] **Lot 534** — /deepresearch multi-step search agent (OpenSeeker-inspired)
+- [x] **Lot 535** — mascarade thinking field fix (Python + Node.js double safety net)
+
+### En cours
+
+- [ ] **Lot 536** — Grafana dashboard provisioning (Prometheus prêt, /api/v2/metrics)
+
+### Backlog session
+
+- [ ] **Lot 537** — mascarade systemd service sur kxkm-ai (env vars, auto-restart)
+- [ ] **Lot 538** — kxkm-ai firewall iptables persistent (nftables conflict)
+- [ ] **Lot 539** — Tailscale sur tower (cross-network kxkm-ai GPU)
+- [ ] **Lot 540** — nginx HTTPS reverse proxy (ports 80/443, certbot)
@@ -13,6 +13,7 @@
 import logger from "./logger.js";
 import { searchWeb } from "./web-search.js";
 import { chat, type ChatMessage, type ChatOptions } from "./llm-client.js";
+import { incrementCounter, recordLatency } from "./perf.js";
 import type { LocalRAG } from "./rag.js";

 // ---------------------------------------------------------------------------
@@ -290,11 +291,15 @@ export async function deepResearch(
  steps.push(synthStep);
  onProgress?.(synthStep, stepIndex++, stepIndex);

+  const totalMs = Date.now() - t0;
+  incrementCounter("deep_research");
+  recordLatency("deep_research", totalMs);
+
  return {
    question,
    answer,
    steps,
    sources,
-    totalDurationMs: Date.now() - t0,
+    totalDurationMs: totalMs,
  };
 }
@@ -23,7 +23,7 @@
      "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [{ "color": "green", "value": null }] } } }
    },
    {
-      "title": "LLM Calls Total",
+      "title": "LLM Calls",
      "type": "stat",
      "gridPos": { "h": 4, "w": 4, "x": 8, "y": 0 },
      "targets": [
@@ -34,26 +34,27 @@
      "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [{ "color": "blue", "value": null }] } } }
    },
    {
-      "title": "Memory (RSS)",
-      "type": "gauge",
+      "title": "LLM Tokens",
+      "type": "stat",
      "gridPos": { "h": 4, "w": 4, "x": 12, "y": 0 },
-      "targets": [{ "expr": "kxkm_memory_rss_bytes / 1024 / 1024", "legendFormat": "RSS MB" }],
-      "fieldConfig": { "defaults": { "unit": "decmbytes", "min": 0, "max": 2048, "thresholds": { "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 512 }, { "color": "red", "value": 1024 }] } } }
+      "targets": [{ "expr": "kxkm_llm_tokens_total", "legendFormat": "tokens" }],
+      "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [{ "color": "blue", "value": null }, { "color": "orange", "value": 500000 }, { "color": "red", "value": 2000000 }] } } }
    },
    {
-      "title": "TTS Requests",
-      "type": "stat",
+      "title": "Memory (RSS)",
+      "type": "gauge",
      "gridPos": { "h": 4, "w": 4, "x": 16, "y": 0 },
-      "targets": [{ "expr": "kxkm_tts_requests_total", "legendFormat": "TTS" }],
-      "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [{ "color": "purple", "value": null }] } } }
+      "targets": [{ "expr": "kxkm_memory_rss_bytes / 1024 / 1024", "legendFormat": "RSS MB" }],
+      "fieldConfig": { "defaults": { "unit": "decmbytes", "min": 0, "max": 2048, "thresholds": { "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 512 }, { "color": "red", "value": 1024 }] } } }
    },
    {
      "title": "Uptime",
      "type": "stat",
      "gridPos": { "h": 4, "w": 4, "x": 20, "y": 0 },
      "targets": [{ "expr": "kxkm_uptime_seconds / 3600", "legendFormat": "hours" }],
-      "fieldConfig": { "defaults": { "unit": "h", "color": { "mode": "thresholds" }, "thresholds": { "steps": [{ "color": "green", "value": null }] } } }
+      "fieldConfig": { "defaults": { "unit": "h", "decimals": 1, "color": { "mode": "thresholds" }, "thresholds": { "steps": [{ "color": "green", "value": null }] } } }
    },
+
    {
      "title": "HTTP Latency (p50 / p95 / p99)",
      "type": "timeseries",
@@ -63,58 +64,114 @@
        { "expr": "kxkm_http_duration_ms{quantile=\"0.95\"}", "legendFormat": "p95" },
        { "expr": "kxkm_http_duration_ms{quantile=\"0.99\"}", "legendFormat": "p99" }
      ],
-      "fieldConfig": { "defaults": { "unit": "ms", "custom": { "drawStyle": "line", "fillOpacity": 10, "pointSize": 3 } } }
+      "fieldConfig": { "defaults": { "unit": "ms", "custom": { "drawStyle": "line", "fillOpacity": 10, "pointSize": 3, "lineWidth": 2 } } }
    },
    {
-      "title": "Persona Response Latency (p50 / p95 / p99)",
+      "title": "Ollama TTFB + Total Latency",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
+      "targets": [
+        { "expr": "kxkm_ollama_ttfb_duration_ms{quantile=\"0.5\"}", "legendFormat": "TTFB p50" },
+        { "expr": "kxkm_ollama_ttfb_duration_ms{quantile=\"0.95\"}", "legendFormat": "TTFB p95" },
+        { "expr": "kxkm_ollama_total_duration_ms{quantile=\"0.5\"}", "legendFormat": "Total p50" },
+        { "expr": "kxkm_ollama_total_duration_ms{quantile=\"0.95\"}", "legendFormat": "Total p95" }
+      ],
+      "fieldConfig": { "defaults": { "unit": "ms", "custom": { "drawStyle": "line", "fillOpacity": 10, "pointSize": 3, "lineWidth": 2 } } }
+    },
+
+    {
+      "title": "Persona Response Latency",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
      "targets": [
        { "expr": "kxkm_persona_response_duration_ms{quantile=\"0.5\"}", "legendFormat": "p50" },
        { "expr": "kxkm_persona_response_duration_ms{quantile=\"0.95\"}", "legendFormat": "p95" },
        { "expr": "kxkm_persona_response_duration_ms{quantile=\"0.99\"}", "legendFormat": "p99" }
      ],
-      "fieldConfig": { "defaults": { "unit": "ms", "custom": { "drawStyle": "line", "fillOpacity": 10, "pointSize": 3 } } }
+      "fieldConfig": { "defaults": { "unit": "ms", "custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 2 } } }
    },
+    {
+      "title": "RAG Search + Rerank Latency",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
+      "targets": [
+        { "expr": "kxkm_rag_search_duration_ms{quantile=\"0.5\"}", "legendFormat": "Search p50" },
+        { "expr": "kxkm_rag_search_duration_ms{quantile=\"0.95\"}", "legendFormat": "Search p95" },
+        { "expr": "kxkm_rag_rerank_duration_ms{quantile=\"0.5\"}", "legendFormat": "Rerank p50" },
+        { "expr": "kxkm_rag_rerank_duration_ms{quantile=\"0.95\"}", "legendFormat": "Rerank p95" }
+      ],
+      "fieldConfig": { "defaults": { "unit": "ms", "custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 2 } } }
+    },
+
    {
      "title": "WebSocket Message Latency",
      "type": "timeseries",
-      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
      "targets": [
        { "expr": "kxkm_ws_message_duration_ms{quantile=\"0.5\"}", "legendFormat": "p50" },
        { "expr": "kxkm_ws_message_duration_ms{quantile=\"0.95\"}", "legendFormat": "p95" }
      ],
-      "fieldConfig": { "defaults": { "unit": "ms", "custom": { "drawStyle": "line", "fillOpacity": 10 } } }
+      "fieldConfig": { "defaults": { "unit": "ms", "custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 2 } } }
    },
    {
-      "title": "Heap Memory",
+      "title": "Memory Usage",
      "type": "timeseries",
-      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
      "targets": [
        { "expr": "kxkm_memory_heap_used_bytes / 1024 / 1024", "legendFormat": "Heap Used MB" },
        { "expr": "kxkm_memory_rss_bytes / 1024 / 1024", "legendFormat": "RSS MB" }
      ],
-      "fieldConfig": { "defaults": { "unit": "decmbytes", "custom": { "drawStyle": "line", "fillOpacity": 15 } } }
+      "fieldConfig": { "defaults": { "unit": "decmbytes", "custom": { "drawStyle": "line", "fillOpacity": 15, "lineWidth": 2 } } }
+    },
+
+    {
+      "title": "Errors by Label",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
+      "targets": [
+        { "expr": "kxkm_errors_total", "legendFormat": "{{label}}" }
+      ],
+      "fieldConfig": { "defaults": { "custom": { "drawStyle": "bars", "fillOpacity": 80, "stacking": { "mode": "normal" } } } }
    },
    {
-      "title": "Cumulative Counters",
+      "title": "Request Throughput",
      "type": "timeseries",
-      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 20 },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
      "targets": [
        { "expr": "rate(kxkm_ws_messages_total[5m]) * 60", "legendFormat": "Chat msg/min" },
        { "expr": "rate(kxkm_ws_commands_total[5m]) * 60", "legendFormat": "Commands/min" },
        { "expr": "rate(kxkm_llm_ollama_calls_total[5m]) * 60", "legendFormat": "LLM Ollama/min" },
        { "expr": "rate(kxkm_llm_mascarade_calls_total[5m]) * 60", "legendFormat": "LLM mascarade/min" },
-        { "expr": "rate(kxkm_tts_requests_total[5m]) * 60", "legendFormat": "TTS/min" }
+        { "expr": "rate(kxkm_tts_requests_total[5m]) * 60", "legendFormat": "TTS/min" },
+        { "expr": "rate(kxkm_image_requests_total[5m]) * 60", "legendFormat": "Image gen/min" }
      ],
      "fieldConfig": { "defaults": { "custom": { "drawStyle": "bars", "fillOpacity": 50 } } }
+    },
+
+    {
+      "title": "Service Counters",
+      "type": "table",
+      "gridPos": { "h": 6, "w": 24, "x": 0, "y": 36 },
+      "targets": [
+        { "expr": "kxkm_ws_messages_total", "legendFormat": "Chat messages", "format": "table", "instant": true },
+        { "expr": "kxkm_ws_commands_total", "legendFormat": "Commands", "format": "table", "instant": true },
+        { "expr": "kxkm_llm_ollama_calls_total", "legendFormat": "LLM Ollama", "format": "table", "instant": true },
+        { "expr": "kxkm_llm_mascarade_calls_total", "legendFormat": "LLM mascarade", "format": "table", "instant": true },
+        { "expr": "kxkm_llm_mascarade_stream_total", "legendFormat": "LLM mascarade SSE", "format": "table", "instant": true },
+        { "expr": "kxkm_llm_tokens_total", "legendFormat": "LLM tokens", "format": "table", "instant": true },
+        { "expr": "kxkm_tts_requests_total", "legendFormat": "TTS requests", "format": "table", "instant": true },
+        { "expr": "kxkm_image_requests_total", "legendFormat": "Image gen", "format": "table", "instant": true },
+        { "expr": "kxkm_deep_research_total", "legendFormat": "Deep Research", "format": "table", "instant": true }
+      ],
+      "fieldConfig": { "defaults": {} }
    }
  ],
+  "refresh": "30s",
  "schemaVersion": 39,
  "tags": ["kxkm", "clown", "3615"],
  "templating": { "list": [] },
  "time": { "from": "now-1h", "to": "now" },
  "title": "3615 KXKM — Overview",
  "uid": "kxkm-overview",
-  "version": 1
+  "version": 2
 }
@@ -8,8 +8,22 @@ scrape_configs:
    static_configs:
      - targets: ["host.docker.internal:3333"]
        labels:
-          instance: "api"
-      # Add satellite machines if reachable:
-      # - targets: ["192.168.0.120:3334"]
+          instance: "kxkm-ai"
+      - targets: ["192.168.0.120:3334"]
+        labels:
+          instance: "tower"
+      # Mac satellite (:3335) — uncomment when running
+      # - targets: ["192.168.0.X:3335"]
      #   labels:
-      #     instance: "tower"
+      #     instance: "mac"
+
+  - job_name: "mascarade"
+    metrics_path: "/metrics"
+    scrape_interval: 30s
+    static_configs:
+      - targets: ["host.docker.internal:8100"]
+        labels:
+          instance: "mascarade-local"
+      - targets: ["192.168.0.120:8100"]
+        labels:
+          instance: "mascarade-tower"