diff --git a/.env.example b/.env.example index 29e232b..3b7865d 100644 --- a/.env.example +++ b/.env.example @@ -5,16 +5,24 @@ # cp .env.example .env # --------------------------------------------------------------------------- -# --- Ollama (LLM inference) ------------------------------------------------ -# If Ollama runs natively on the host, use host.docker.internal (default). -# If using the Ollama Docker container (--profile ollama), use http://ollama:11434 -OLLAMA_URL=http://host.docker.internal:11434 +# --- Local LLM runtime (vLLM / TurboQuant) --------------------------------- +# Primary chat/completion runtime. Must expose an OpenAI-compatible API. +LLM_URL=http://host.docker.internal:11434 +LLM_MODEL=qwen-14b-awq +LLM_API_KEY= # Bearer token for vLLM --api-key (leave empty for Ollama) + +# --- Embeddings backend (auxiliary) ---------------------------------------- +# TEI (recommended): dedicated embedding server on port 9500 +# Ollama: fallback, uses /api/embed on port 11434 +OLLAMA_URL=http://host.docker.internal:9500 +EMBEDDING_BACKEND=tei # "tei" (OpenAI /v1/embeddings) or "ollama" (/api/embed) +RAG_EMBEDDING_MODEL=BAAI/bge-m3 # Model name (must match TEI --model-id or Ollama model) # --- Ports ------------------------------------------------------------------ APP_PORT=3333 # V1 Express server API_PORT=4180 # V2 API server PG_PORT=5432 # PostgreSQL (exposed to host) -# OLLAMA_PORT=11434 # Only needed with --profile ollama +# LLM_PORT=11434 # Only needed if you expose the local runtime from compose # --- Admin ------------------------------------------------------------------ ADMIN_BOOTSTRAP_TOKEN= # Initial admin token (set a strong secret) @@ -25,7 +33,7 @@ OWNER_NICK= # Owner nickname in chat MAX_GENERAL_RESPONDERS=4 # Max personas responding in #general # --- Vision (image analysis in chat) ---------------------------------------- -# VISION_MODEL=qwen3-vl:8b # Ollama model for image analysis (qwen3-vl recommandé) +# VISION_MODEL=qwen3-vl:8b # Vision model used by the configured runtime # --- Training (Node Engine worker) ------------------------------------------ # PYTHON_BIN=/home/kxkm/venv/bin/python3 # Python with ML libs (PyTorch, Unsloth, TRL) @@ -37,8 +45,8 @@ MAX_GENERAL_RESPONDERS=4 # Max personas responding in #general # --- ComfyUI (image generation) -------------------------------------------- # COMFYUI_URL=http://localhost:8188 # ComfyUI API endpoint for /imagine command -# --- Mascarade (LLM orchestrator) ------------------------------------------ -# MASCARADE_URL=http://127.0.0.1:8100 # Mascarade API (OpenAI-compatible /v1/chat/completions) +# --- Mascarade (optional cloud/provider routing) ---------------------------- +# MASCARADE_URL=http://127.0.0.1:8100 # Optional cloud/provider router # MASCARADE_API_KEY= # API key for authenticated endpoints (/agents, /orchestrate) # --- External services (optional) ------------------------------------------ diff --git a/AGENTS.md b/AGENTS.md index 0dc5166..e320cfd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,252 +1,130 @@ -# Agents, Sous-agents, Competences +# AGENTS.md — KXKM_Clown Monorepo > "L'infrastructure est une decision politique deployee." -- electron rare -## Orchestration +Multimodal AI chat system. Turborepo monorepo (npm workspaces): 3 apps + 8 packages + 42 scripts. 15+ service mesh. Single RTX 4090 GPU: `MAX_GPU_CONCURRENT=1`. -- Agent racine: **Coordinateur** — planifie, arbitre, synchronise PLAN/TODO/docs -- Sous-agents specialises: analyse code, veille OSS, audit securite, optimisation -- Cadence: synchroniser PLAN.md + TODO.md + docs apres chaque lot +## Key Files -## Matrice des agents (lot 17+) +| File | Purpose | +|------|---------| +| `docker-compose.yml` | 12 services (postgres, searxng, docling, qdrant, ollama, tts, lightrag) with health checks | +| `turbo.json` | Build tasks, caching, workspace graph | +| `package.json` | Root: 15 npm scripts (dev, build, check, test, smoke, verify) | +| `.env.example` | LLM_URL, DATABASE_URL, ports, TTS, RAG model, etc. | -| Agent | Competences | Perimetre | Etat | +## Subdirectories + +| Dir | Purpose | Ref | +|-----|---------|-----| +| `apps/` | 3 apps: api (77 TS), web (64 TS/TSX), worker (4 TS) | `apps/AGENTS.md` | +| `packages/` | 8 packages (core, auth, chat-domain, persona-domain, node-engine, storage, tui, ui) | `packages/AGENTS.md` | +| `scripts/` | 42 files: 20 Python, 22 Shell (ops, training, ingestion, voice, image, deploy) | `scripts/AGENTS.md` | +| `docs/` | 50+ specs, spikes, research, audits (SPEC_*.md, AUDIT_*.md, OSS_VEILLE_*.md) | — | +| `ops/` | Monitoring + ops/v2/ (systemd, TUI, health-check.sh, deep-audit.js) | — | +| `models/` | Fine-tuned + LoRA weights (base_models/, finetuned/, lora/, registry.json) | — | +| `data/` | Ephemeral: persona memory, chat logs, context, corpus (v2-local/, chat-logs/) | — | + +## Agent Matrix + +| Agent | Competences | Scope | Status | |---|---|---|---| -| Coordinateur | planification, arbitrage, docs de pilotage | PLAN.md, TODO.md, AGENTS.md, README.md | actif | -| Securite | validation input, hardening, rate-limit, RBAC | apps/api, ws-chat, packages/auth | veille | -| Backend API | Express, WS, Ollama, RAG, multimodal pipeline | apps/api/src/ | actif | +| Coordinateur | Planning, arbitration, docs sync | PLAN.md, TODO.md, AGENTS.md, README.md | actif | +| Securite | Input validation, hardening, rate-limit, RBAC | apps/api, ws-chat, packages/auth | veille | +| Backend API | Express, WS, Ollama, RAG, multimodal pipeline | apps/api/src/ (77 TS + 27 tests) | actif | | Node Engine | DAG, queue, runs, sandbox, training adapters | packages/node-engine, apps/worker | actif | -| Personas | source/feedback/proposals/pharmacius, memoire | packages/persona-domain, ws-chat | actif | -| Frontend | React/Vite, UX Minitel, React Flow, chat, voice | apps/web/src/ | actif | -| Ops/TUI | scripts, logs, rotate/purge, health, audit | ops/v2/, scripts/ | actif | -| Training | DPO, SFT, Unsloth, eval, autoresearch, Ollama import | scripts/, packages/node-engine | actif | -| Multimodal | STT, TTS, vision, PDF, RAG, recherche web | apps/api/src/ws-chat.ts | actif | -| Veille OSS | recherche projets, libs, modeles, benchmarks | docs/OSS_WATCH, docs/HF_MODEL_RESEARCH | periodique | +| Personas | Memory, DPO, pharmacius, coherence | packages/persona-domain, ws-chat (33 personas) | actif | +| Frontend | React/Vite, Minitel theme, React Flow, chat, voice | apps/web/src/ (64 TS/TSX + 10 tests) | actif | +| Ops/TUI | Monitoring, deploy, logs, health, audit | ops/v2/, scripts/, deep-audit.js | actif | +| Training | DPO, SFT, Unsloth, eval, autoresearch | scripts/, packages/node-engine | actif | +| Multimodal | STT, TTS, vision, PDF, RAG, web search | apps/api/src/ws-multimodal.ts | actif | +| Veille OSS | Benchmarks, new libs, licensing, interop | docs/OSS_WATCH, docs/HF_MODEL_RESEARCH | periodique | -## Sous-agents et skill routing +## Message Flow -```mermaid -flowchart TD - Coord[Coordinateur] - - Coord --> SecAgent[Securite] - Coord --> BackAgent[Backend API] - Coord --> EngAgent[Node Engine] - Coord --> PersAgent[Personas] - Coord --> FrontAgent[Frontend] - Coord --> OpsAgent[Ops/TUI] - Coord --> TrainAgent[Training] - Coord --> MultiAgent[Multimodal] - Coord --> OSSAgent[Veille OSS] - - SecAgent --> |audit| SecScan[Pattern scan P0/P1/P2] - SecAgent --> |fix| SecFix[Correctifs chirurgicaux] - - BackAgent --> |analyse| APIAudit[Deep analyse app.ts, ws-chat.ts] - BackAgent --> |refactor| APISplit[Extraction modules] - - EngAgent --> |test| EngTest[Tests unitaires node-engine] - EngAgent --> |extend| EngNew[Nouveaux node types] - - PersAgent --> |pipeline| PersPipe[Editorial pipeline] - PersAgent --> |finetune| PersDPO[DPO + PCL methodology] - - FrontAgent --> |ui| FrontUI[Minitel theme CSS] - FrontAgent --> |perf| FrontPerf[Memoization, lazy load] - - OpsAgent --> |monitor| OpsHealth[health-check, deep-audit] - OpsAgent --> |deploy| OpsDeploy[Docker, kxkm-ai] - - TrainAgent --> |train| TrainRun[Unsloth/TRL runs] - TrainAgent --> |eval| TrainEval[Scoring, registry] - - MultiAgent --> |voice| MultiVoice[XTTS-v2, WebRTC] - MultiAgent --> |search| MultiSearch[SearXNG] - - OSSAgent --> |web| OSSWeb[Recherche web] - OSSAgent --> |hf| OSSHF[HuggingFace models] +``` +User WS → ws-chat.ts (rate-limit, multimodal dispatch) + → ws-conversation-router.ts (persona routing, context assembly) + → ws-persona-router.ts (memory extract/load, responder select) + → inference-scheduler.ts (single-GPU queue, MAX_GPU_CONCURRENT=1) + → ws-ollama.ts (token stream, tool-calling) + → ws-multimodal.ts (TTS, vision, STT, file upload) + → persona-memory-store.ts (nick-isolated file persist) + → rag.ts (embedding + LightRAG dual-write) + → context-store.ts (channel history + compaction) ``` -## Todo agents (lot 17+ — mis a jour 2026-03-24) +## Services -### Coordinateur +| Service | Port | Notes | +|---------|------|-------| +| API (HTTP+WS) | 4180 | Node.js Express + ws | +| Frontend (Vite) | 5173 | React + 5 CSS themes | +| Ollama/vLLM | 11434 | LLM runtime + embeddings | +| PostgreSQL | 5432 | Chat, sessions, node-engine runs | +| SearXNG | 8080 | Self-hosted search (DuckDuckGo fallback) | +| Docling | 9400 | PDF extraction | +| LightRAG | 9621 | Graph-RAG, `LLM_MODEL=mistral:7b` to avoid `` corruption | +| TTS (Piper/Chatterbox) | 9100 | Voice synthesis | +| Kokoro TTS | 9201 | Fast TTS, 12 voices | +| ComfyUI | 8188 | Image generation (32 checkpoints + 24 LoRAs) | +| Camoufox | 8091 | Stealth browser for bot-protected sites | -- [x] Consolider PLAN.md avec etat reel (lots 0-94 complets) -- [x] Synchroniser FEATURE_MAP.md matrice -- [x] Mettre a jour TODO.md avec backlog Phase session 2026-03-19/20 -- [x] Documenter actions dans ops/v2/logs/ -- [x] lot-95: Coordonner E2E Playwright test plan -- [x] lot-100: Design public demo mode access control +## GPU Constraint -### Backend API +All LLM calls → `inference-scheduler.ts`. No direct fetch() outside approved helpers. RTX 4090: `MAX_GPU_CONCURRENT=1`. Context compaction + persona extraction both via `scheduler.submit(priority: "low")`. -- [x] Extraire app-bootstrap.ts et app-middleware.ts de app.ts -- [x] Extraire ws-conversation-router.ts de ws-chat.ts -- [x] ws-chat.ts modularized (425 to 335 LOC, 3 modules extracted) -- [x] app.ts extraction (540 to 131 LOC, create-repos.ts extracted) -- [x] Zod validation on all 19 API route schemas -- [x] Error telemetry (16 labels) -- [x] Perf instrumentation (6 labels, p50/p95/p99), TTFC 284ms -- [x] Smart routing (5 topic domains) -- [x] Dynamic context window (4k-32k) -- [x] NLP auto-detect generation intent (compose vs imagine) -- [x] /speed command for latency diagnostics -- [ ] lot-178: ACE-Step API direct integration (duration fix) -- [ ] lot-180: Timeline data model -- [x] lot-97: Multi-channel support (create/join channels) -- [x] lot-100: Public demo mode read-only routes +## Persona Memory (nick-isolated, 2026-04) -### Node Engine +- **Path**: `data/v2-local/persona-memory/{personaId}/{nick}.json` +- **Modes**: `auto` (Pharmacius, Sherlock, Turing, Ikeda), `explicit` (artistic personas, `/remember` only) +- **Injection cap**: 8 facts max into system prompt +- **Anonymous relay**: `_anonymous` sentinel for unknown-nick chains -- [x] Extraire registry.ts du hotspot node-engine -- [ ] Ajouter node type `music_generation` (ACE-Step 1.5) -- [ ] Ajouter node type `voice_clone` (Chatterbox) -- [ ] Ajouter node type `audio_mix` (multi-track composition) -- [ ] Ajouter node type `audio_effects` (FX chain) -- [ ] lot-96: Automated DPO pipeline (feedback → pairs → training trigger) +## Build & Dev -### Multimodal (composition pipeline) - -- [x] 35 music styles ACE-Step -- [x] ComfyUI smart checkpoint selection (32 checkpoints + 24 LoRAs) -- [ ] lot-178: ACE-Step API direct (duration fix) -- [ ] lot-181: TTS voiceover mix into timeline -- [ ] lot-182: Audio effects pipeline (reverb, delay, EQ, compression) -- [ ] lot-183: DAW export (stems, markers, project file) -- [x] lot-184: Multi-track composition (/layer, composition-store) -- [x] lot-185: Composition UI (track lanes, play/pause/seek) -- [x] lot-186: Arrangement tools (/comp structure, section markers) -- [x] lot-187: Auto-mastering (/mix master, loudness normalization, limiter) -- [x] lot-188: /voice TTS voiceover injected into composition -- [x] lot-189: /noise 5 types (white, pink, brown, rain, wind) -- [x] lot-190: /fx 9 audio effects (reverb, delay, chorus, flanger, distortion, bitcrusher, EQ, compressor, tremolo) -- [x] lot-191: /ambient scene generator (forest, ocean, city, space, cave) -- [ ] lot-194: Waveform visualization (wavesurfer.js) -- [ ] lot-195: /remix re-generate specific track -- [ ] lot-199: Stem separation (Demucs v4 htdemucs, 6-stem, MIT) -- [x] lot-200: Full DAW export (WAV stems + JSON project) - -### Personas - -- [ ] Evaluer PCL (Persona-Aware Contrastive Learning) pour coherence -- [ ] Evaluer OpenCharacter pour generation profils synthetiques -- [x] Ajouter `/compose` command (generation musicale) - -### Frontend - -- [x] Implementer lot 16 UI Minitel rose (phosphore, VIDEOTEX) -- [x] VoiceChat push-to-talk + level meter + silence auto -- [x] Player audio + viewer image plein ecran -- [x] Mediatheque gallery/playlist -- [x] Progress bars animees Compose/Imagine -- [x] React.memo + useCallback on ChatSidebar, ChatInput, ChatHistory -- [x] 17 lazy-loaded routes (-53% initial JS) -- [x] CRT CSS-only effect (scanlines, vignette, phosphor glow, boot 0.8s) -- [x] Chat virtualization (react-window, variable row heights) -- [x] Markdown rendering (marked + DOMPurify) -- [x] CRT boot animation (modem dial, scanline reveal) -- [x] 5 CSS themes (minitel, crt, hacker, synthwave, default) -- [x] Mobile responsive pass (touch, bottom nav, viewport units) -- [x] Guest mode read-only UI -- [x] lot-185: Composition timeline UI (waveform view, track lanes) -- [ ] lot-194: Waveform visualization (wavesurfer.js) -- [x] lot-95: E2E Playwright tests (login, chat, upload, admin) -- [x] lot-98: File sharing UI (upload → gallery) - -### Ops/TUI - -- [x] Deployer deep-audit.js sur kxkm-ai -- [x] Ajouter SearXNG au docker-compose -- [x] TTS sidecar HTTP (tts-server.py :9100, dual Chatterbox/Piper) -- [x] deploy.sh migrated tmux → systemd -- [x] Systemd services (kxkm-tts + kxkm-lightrag, auto-restart) -- [x] health-check.sh TUI (19 checks) -- [x] Docker compose 12 services with health checks -- [ ] Fix Docker transformers (rebuild propre avec torch) - -### Training - -- [x] Spike BGE-M3 (resultat negatif sur Apple/Metal, baseline maintenue) -- [x] TTS dual backend Chatterbox/Piper valide -- [x] Tool-calling benchmark (llama3.1 vs qwen3 vs mistral) -- [x] Sherlock migrated to llama3.1:8b-instruct-q4_0 -- [ ] lot-96: Persona DPO automation pipeline -- [ ] Tester ACE-Step 1.5 sur RTX 4090 - -### Veille OSS - -- [x] Veille mars 2026 complete (40+ projets analyses, top 10 recommandations) -- [ ] Suivre LLMRTC (WebRTC voice TypeScript) -- [ ] Suivre A2A Protocol (interop agents) -- [ ] Suivre MCP SDK updates -- [ ] Evaluer Kokoro TTS (82M params, ultra-leger) - -## Pipeline d'intervention - -```mermaid -stateDiagram-v2 - [*] --> Analyse: agent lance - Analyse --> Findings: scan code/docs/web - Findings --> Triage: P0/P1/P2 classification - Triage --> Fix_P0: P0 critique - Triage --> Plan_P1: P1 important - Triage --> Backlog_P2: P2 mineur - Fix_P0 --> Test: correction chirurgicale - Plan_P1 --> Test: correction planifiee - Test --> Deploy: tests OK - Deploy --> Log: log + purge - Log --> [*]: cycle termine - Backlog_P2 --> [*]: ajoute au TODO +```bash +npm install # Install all workspaces +npm run dev # Turbo parallel (api, web, worker) +npm run dev:v2:api # API :4180 (tsx watch) +npm run dev:v2:web # Web :5173 (Vite) +npm run check:v2 # tsc --noEmit +npm run -w @kxkm/api test # 278 unit tests +npm run -w @kxkm/web test # 54 unit tests +npm run smoke:v2 # Integration smoke +npm run verify # check + smoke (full gate) +docker compose --profile v2 up -d ``` -## Affectations en cours (2026-03-20) +## Environment -### Mission globale -- Deep analyse continue du code, optimisation chirurgicale, et synchronisation documentaire apres chaque lot. -- Priorite execution: P1 fiabilite, puis dette perf/complexite, puis features lot 18-19. +```bash +LLM_URL=http://localhost:11434 +LLM_MODEL=qwen-14b-awq +DATABASE_URL=postgres://kxkm:kxkm@localhost:5432/kxkm_clown +V2_API_PORT=4180 +TTS_ENABLED=1 +VISION_MODEL=qwen3-vl:8b +RAG_EMBEDDING_MODEL=nomic-embed-text +SEARXNG_URL=http://localhost:8080 +KXKM_PERSONA_MEMORY_INJECTION_LIMIT=8 +``` -### Assignations agents -> sous-agents -> competences +## Cycle State (2026-03-20) -| Agent | Sous-agent | Competences principales | Taches assignees immediates | -|---|---|---|---| -| Coordinateur | Planner/Docs | triage, synchronisation, runbook | Maintenir PLAN/TODO, chainer les lots, tracer actions | -| Backend API | WS/HTTP surgeon | websocket, express, validation input | Extraire `ws-chat.ts` en modules, reduire logs, limiter hot paths | -| Node Engine | DAG runtime | graph validation, queue, state machine | Ajouter nodes `music_generation`, `voice_clone`, `document_extraction` | -| Ops/TUI | Audit operator | TUI scripts, logs, rotation, cron | Rendre `deep-audit` zero faux positif critique, pipeline logs + purge | -| Veille OSS | Scout | benchmark OSS, licences, interop | Evaluer Open WebUI, LibreChat, LangGraph, SearXNG, Docling | +- 130+ lots complete (lot-24 to lot-177) +- 425 tests, 0 failures +- 43 chat commands, 33 personas +- 5 CSS themes, 35 music styles (ACE-Step) +- 32 ComfyUI checkpoints + 24 LoRAs +- TTFC 284ms, guest mode, mobile responsive +- Structured logging (pino JSON) +- Systemd services (TTS, LightRAG) -### Workflow d'enchainement -1. Executer audit + tests -2. Corriger de maniere chirurgicale -3. Re-executer audit + tests -4. Mettre a jour docs de pilotage -5. Alimenter TODO suivant avec ordre d'execution +## See Also -### Regles d'operation -- Interroger le user uniquement en cas de blocage reel (acces, choix irreversibles, secrets). -- Privilegier TUI et scripts avec logs lisibles, puis purge des logs obsoletes. -- Conserver la V1 comme reference comportementale, V2 comme cible active. - -### Etat de cycle (2026-03-20 18:00) - -- 130+ lots termines (lot-24 a lot-177). -- 425 tests, 0 failures. -- 13 services en production. -- 43 chat commands, 33 personas. -- 35 music styles (ACE-Step), 5 CSS themes. -- 32 ComfyUI checkpoints + 24 LoRAs, smart selection NLP. -- TTFC 284ms. -- Guest mode, mobile responsive. -- Structured logging complet (pino JSON, 0 console.log). -- Systemd services (TTS + LightRAG). -- Frontend: lazy routes (-53%), React.memo, CRT boot, chat virtualization. - -### Prochains lots (178-200) — Composition Pipeline - -1. lot-178: Compose duration fix (ACE-Step API direct) -2. lot-179: SPEC_COMPOSE_ADVANCED plan -3. lot-180: Timeline data model (tracks, clips, markers) -4. lot-181: TTS voiceover mix into timeline -5. lot-182: Audio effects pipeline (reverb, delay, EQ) -6. lot-183: DAW export (stems, markers, project) -7. lot-184-200: Multi-track, arrangement, mastering, stem separation, MIDI, templates, collab, lyrics, FX rack, automation, samples, spectral view, history, render queue, sharing +- `apps/AGENTS.md` — api, web, worker details +- `packages/AGENTS.md` — shared package breakdown +- `scripts/AGENTS.md` — deployment, training, ops scripts +- `CLAUDE.md` — Architecture, request paths, data directories +- `PLAN.md` / `TODO.md` / `FEATURE_MAP.md` — Roadmap (lots 178-200) diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..36cce4e --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,146 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Commands + +```bash +# Dev (V2) +npm run dev:v2:api # API on :4180 (tsx watch) +npm run dev:v2:web # Frontend on :5173 (Vite) +npm run dev:v2:worker # Background job processor + +# Build +npm run build:v2 # TypeScript compile (all workspaces) +npm run turbo:build # Full monorepo build + +# Type-check +npm run check:v2 # tsc --noEmit (fast check) +npm run -w @kxkm/api check + +# Tests +npm run -w @kxkm/api test # 278 unit tests +npm run -w @kxkm/web test # 54 unit tests +npm run smoke:v2 # Integration smoke +npm run verify # check + smoke (full gate) + +# Run a single test file (after build) +cd apps/api && node --experimental-test-module-mocks --test dist/persona-memory-store.test.js + +# Docker +docker compose --profile v2 up -d +docker compose --profile v2 --profile ollama up -d # with bundled embeddings backend +``` + +## Architecture + +**Monorepo** (npm workspaces + Turborepo): `apps/` + `packages/` + +### API (`apps/api/src/`) + +The V2 API is a single Node.js process combining Express (HTTP) and `ws` (WebSocket) on port 4180. + +**Request path for a chat message:** +``` +ws-chat.ts (WS handler, rate-limit, routing) + → ws-conversation-router.ts (pick responder personas, build context) + → ws-ollama.ts (stream tokens from vLLM/TurboQuant runtime, tool-calling) + → ws-multimodal.ts (TTS streaming, vision, STT, file upload) + → ws-persona-router.ts (memory load/update, responder selection) +``` + +**Key files:** +| File | Purpose | +|------|---------| +| `server.ts` | HTTP+WS bootstrap, DAW sample routes, corpus boot | +| `ws-chat.ts` | WS entry — broadcast, rate-limit, multimodal dispatch | +| `ws-conversation-router.ts` | Persona routing, context assembly, TTS chunking, inter-persona depth-3 relay | +| `ws-ollama.ts` | Runtime streaming, tool-calling, `` tag stripping | +| `ws-persona-router.ts` | Memory extract/save, responder selection, InferenceScheduler | +| `personas-default.ts` | 33 persona definitions (`memoryMode`, `corpus[]`, `relations[]`) | +| `persona-memory-store.ts` | Per-nick isolated storage at `data/v2-local/persona-memory/{personaId}/{nick}.json` | +| `persona-memory-policy.ts` | auto/explicit/off modes, `injectionFactsLimit` (default 8) | +| `rag.ts` | Local embedding store, per-persona namespaces, LightRAG dual-write | +| `context-store.ts` | Per-channel conversation memory, LLM compaction via InferenceScheduler | +| `inference-scheduler.ts` | Single-GPU queue (`MAX_GPU_CONCURRENT=1`), all LLM calls must go through it | +| `chat-types.ts` | All shared types: `ChatPersona`, `PersonaMemoryMode`, `ClientInfo`, message union | +| `mcp-tools.ts` | Tool definitions injected per persona (web_search, rag_search, etc.) | +| `web-search.ts` | SearXNG → DuckDuckGo fallback; discovered URLs enqueued to `data/sherlock-discovered-urls.jsonl` | + +### Persona Memory (nick-isolated, 2026-04 design) + +Memory is keyed by `(personaId, nick)` — one file per user per persona. + +- `memoryMode: 'auto'` → Pharmacius, Sherlock, Turing, Ikeda (LLM extraction every 5 responses) +- `memoryMode: 'explicit'` → all artistic personas (Schaeffer, Merzbow, Pina, etc.) — only via `/remember` +- `/remember [@persona|@all] ` — direct fact insert, no LLM call +- Injection cap: 8 facts max into system prompt (`injectionFactsLimit`) +- `_anonymous` sentinel for unknown-nick relay chains + +### Packages + +| Package | Contents | +|---------|---------| +| `core` | Shared types, IDs, permissions | +| `auth` | RBAC, sessions, crypto | +| `chat-domain` | Message types, channels, slash command registry | +| `persona-domain` | Persona model, DPO pairs, feedback pipeline | +| `node-engine` | DAG execution, GPU job queue, 15+ node types | +| `storage` | Postgres repositories, migrations | + +### Services & Ports + +| Service | Port | Notes | +|---------|------|-------| +| API V2 | 4180 | HTTP + WS | +| Frontend | 5173 | Vite dev | +| vLLM | 8000 | Primary OpenAI-compatible text runtime (qwen-32b-awq) | +| TEI | 9500 | Dedicated embedding server (BAAI/bge-m3, CPU) | +| PostgreSQL | 5432 | V2 persistence (optional for API, required for worker) | +| LightRAG | 9621 | Graph-RAG; `LLM_MODEL=mistral:7b` to avoid `` JSON corruption | +| SearXNG | 8080 | Self-hosted search | +| TTS | 9100 | Piper + Chatterbox | +| Kokoro TTS | 9201 | Fast TTS, 12 voices | +| AI Bridge | 8301 | 19 audio backends | +| ComfyUI | 8188 | Image generation | +| Docling | 9400 | PDF extraction | +| Camoufox | 8091 | Stealth browser fetch (venv, `kxkm-camoufox.service`) | + +### InferenceScheduler constraint + +**All LLM calls must go through `inference-scheduler.ts`** — no direct `fetch()` to the runtime outside approved helpers. The RTX 4090 has `MAX_GPU_CONCURRENT=1`. `context-store.ts` compaction and `ws-persona-router.ts` extraction both use `scheduler.submit()` with `priority: "low"`. + +### Corpus ingestion pipeline + +`scripts/ingest_spectacle_corpus.py` ingests domain content into LightRAG: +- Seed URLs + SearXNG discovery + `data/sherlock-discovered-urls.jsonl` (live Sherlock search discoveries) +- Camoufox server (`:8091`) used for bot-protected sites (`artcena.fr`, `culture.gouv.fr`, etc.) + +### Data directories + +``` +data/ + v2-local/persona-memory/{personaId}/{nick}.json # Per-user persona memory + context/{channel}.jsonl # Conversation history + sherlock-discovered-urls.jsonl # Corpus queue from web searches + chat-logs/ # Daily JSONL chat logs + manifeste.md # Project philosophy (injected at boot) +``` + +## Environment Variables + +```bash +LLM_URL=http://localhost:8000 # vLLM OpenAI-compatible endpoint +LLM_MODEL=qwen-32b-awq +LLM_API_KEY=vllm-er-2026 # Bearer token for vLLM --api-key +OLLAMA_URL=http://localhost:9500 # TEI embedding server +EMBEDDING_BACKEND=tei # "tei" or "ollama" +RAG_EMBEDDING_MODEL=BAAI/bge-m3 +DATABASE_URL=postgres://kxkm:kxkm@localhost:5432/kxkm_clown +V2_API_PORT=4180 +TTS_ENABLED=1 +VISION_MODEL=qwen3-vl:8b +SEARXNG_URL=http://localhost:8080 +KXKM_PERSONA_MEMORY_INJECTION_LIMIT=8 # Max facts injected into system prompt +CHAT_PAUSED=1 # Or create data/chat-paused file +``` diff --git a/apps/AGENTS.md b/apps/AGENTS.md new file mode 100644 index 0000000..a15c9b3 --- /dev/null +++ b/apps/AGENTS.md @@ -0,0 +1,244 @@ +# AGENTS.md — apps/ + + + +Three applications in Turborepo workspace: api (backend), web (frontend), worker (background jobs). + +## api/ — Backend (77 TS files + 27 tests) + +Node.js Express + WebSocket on port 4180. Single process: HTTP + WS. + +### WebSocket Handlers + +| File | Purpose | +|------|---------| +| `ws-chat.ts` | Entry point: rate-limit, broadcast, multimodal dispatch | +| `ws-conversation-router.ts` | Persona routing, context assembly, TTS chunking, inter-persona depth-3 relay | +| `ws-ollama.ts` | Runtime streaming, tool-calling, `` tag stripping | +| `ws-persona-router.ts` | Memory extract/save, responder selection, InferenceScheduler submit | +| `ws-multimodal.ts` | TTS streaming, vision, STT, file upload, ComfyUI integration | +| `ws-upload-handler.ts` | Media ingestion, storage, gallery | +| `ws-chat-helpers.ts` | Utilities: formatting, logging, metrics | +| `ws-chat-logger.ts` | Structured logging (pino JSON) | +| `ws-chat-history.ts` | Per-channel conversation memory, compaction | +| `ws-chat-state.test.ts` | State machine tests | + +### Commands (5 handlers) + +| File | Purpose | +|------|---------| +| `ws-commands.ts` | Router: dispatch to handlers by `/command` | +| `ws-commands-chat.ts` | `/chat`, `/speed`, `/help` | +| `ws-commands-info.ts` | `/personas`, `/commands`, `/status` | +| `ws-commands-generate.ts` | `/imagine`, `/compose`, `/music` (ACE-Step, ComfyUI) | +| `ws-commands-compose.ts` | DAW composition: `/layer`, `/fx`, `/voice`, `/mix`, `/export` | + +### Personas (33 definitions + memory) + +| File | Purpose | +|------|---------| +| `personas-default.ts` | 33 personas: Pharmacius, Sherlock, Turing, Ikeda, Schaeffer, Merzbow, Pina, etc. (memoryMode, corpus[], relations[]) | +| `persona-runtime.ts` | Runtime: load, extract, save; DPO feedback pipeline | +| `persona-memory-store.ts` | Per-nick isolated storage: `data/v2-local/persona-memory/{personaId}/{nick}.json` | +| `persona-memory-policy.ts` | auto/explicit/off modes, injectionFactsLimit (default 8) | +| `persona-voices.ts` | Voice config per persona | +| `persona-memory-telemetry.ts` | Metrics: extraction time, injection count, memory size | + +### RAG & Inference + +| File | Purpose | +|------|---------| +| `rag.ts` | Local embedding store, per-persona namespaces, LightRAG dual-write | +| `inference-scheduler.ts` | Single-GPU queue: `MAX_GPU_CONCURRENT=1`, all LLM calls must go through it | +| `llm-client.ts` | Ollama API wrapper, streaming, tool-calling | +| `deep-research.ts` | Multi-turn research loop via scheduler | + +### Storage & Context + +| File | Purpose | +|------|---------| +| `context-store.ts` | Per-channel conversation memory, LLM compaction via scheduler | +| `chat-types.ts` | All shared types: ChatPersona, PersonaMemoryMode, ClientInfo, message union | +| `composition-store.ts` | Multi-track composition state (tracks, clips, markers) | +| `media-store.ts` | File storage, gallery, cleanup | + +### Services + +| File | Purpose | +|------|---------| +| `web-search.ts` | SearXNG → DuckDuckGo fallback; discovered URLs → `data/sherlock-discovered-urls.jsonl` | +| `comfyui.ts` | Image generation: checkpoint + LoRA selection, prompt injection | +| `comfyui-models.ts` | 32 checkpoints + 24 LoRAs registry | +| `voice-samples.ts` | Voice clone sample management | +| `mcp-tools.ts` | Tool definitions injected per persona (web_search, rag_search, compose, imagine, etc.) | + +### Routes (6 REST files + bootstrap) + +| File | Purpose | +|------|---------| +| `routes/chat-history.ts` | GET/POST chat logs, export | +| `routes/media.ts` | GET/POST media, gallery | +| `routes/personas.ts` | GET personas, memory, feedback | +| `routes/session.ts` | Auth, session mgmt, guest mode | +| `routes/node-engine.ts` | DAG runs, node types, training | +| `media-shared-routes.ts` | Shared media endpoints | +| `app.ts` | Express setup, middleware, WS upgrade | +| `app-bootstrap.ts` | Corpus load, DAW samples, systemd startup | +| `app-middleware.ts` | Auth, CORS, error handlers | + +### Server & Infrastructure + +| File | Purpose | +|------|---------| +| `server.ts` | HTTP + WS bootstrap, DAW sample routes, corpus boot | +| `create-repos.ts` | DB repo initialization | +| `schemas.ts` | Zod validation: 19 route schemas + command input schemas | +| `error-tracker.ts` | Error telemetry (16 labels) | +| `perf.ts` | Perf instrumentation (6 labels, p50/p95/p99) | +| `logger.ts` | Pino JSON logger config | + +### Tests (27 files) + +| File | Purpose | +|------|---------| +| `app.test.ts` | HTTP routes, middleware, health checks | +| `ws-chat.test.ts` | WS broadcast, rate-limit, multimodal dispatch | +| `ws-conversation-router.test.ts` | Persona routing, context assembly | +| `ws-ollama.test.ts` | Token streaming, tool-calling, think tag | +| `ws-multimodal.test.ts` | TTS, vision, STT, file upload | +| `ws-upload-handler.test.ts` | Media ingestion | +| `ws-commands.test.ts` | Command routing | +| `persona-memory-store.test.ts` | Nick isolation, file persistence | +| `persona-memory-policy.test.ts` | auto/explicit modes | +| `persona-memory-telemetry.test.ts` | Metrics tracking | +| `persona-runtime.test.ts` | Runtime load/save | +| `rag.test.ts` | Embedding, LightRAG sync | +| `context-store.test.ts` | Channel history, compaction | +| `composition-store.test.ts` | Multi-track state | +| `media-store.test.ts` | File storage | +| `web-search.test.ts` | SearXNG, DuckDuckGo fallback | +| `voice-samples.test.ts` | Voice sample mgmt | +| `mcp-tools.test.ts` | Tool definitions | +| `mcp-server.test.ts` | MCP server integration | +| `chat-history-routes.test.ts` | Chat log routes | +| `media-shared-routes.test.ts` | Media endpoints | +| `ws-chat-smoke.test.ts` | End-to-end smoke | +| `ws-integration.test.ts` | Full integration flow | +| `ws-chat-state.test.ts` | State machine | +| `create-repos.test.ts` | DB setup | +| `app.test.ts` | Bootstrap | +| `integration.test.ts` | Full system | + +### Run + +```bash +npm run dev:v2:api # tsx watch (localhost:4180) +npm run -w @kxkm/api test # 278 unit tests +cd apps/api && npm test # From api dir +``` + +## web/ — Frontend (64 TS/TSX files + 10 tests) + +React + Vite on port 5173. 5 CSS themes (minitel, crt, hacker, synthwave, default). React.memo + useCallback optimized. 17 lazy-loaded routes (-53% initial JS). Chat virtualization (react-window). CRT boot animation. + +### Pages (10) + +| Component | Purpose | +|-----------|---------| +| `Chat.tsx` | Main chat interface (virtualized history, input, sidebar) | +| `ImaginePage.tsx` | Image generation (ComfyUI) | +| `ComposePage.tsx` | DAW composition (timeline, tracks, effects) | +| `DawAIPanel.tsx` | AI composition sidebar | +| `LiveFXPage.tsx` | Real-time audio effects | +| `UllaPage.tsx` | Ulla (experimental) | +| `NodeEngineOverview.tsx` | DAG editor, run status | +| `AdminPage.tsx` | Admin dashboard, user management | +| `TrainingDashboard.tsx` | Training runs, metrics | +| `Collectif.tsx` | Multi-user collaboration | + +### Components (15+) + +| Component | Purpose | +|-----------|---------| +| `ChatMessage.tsx` | Message render, markdown, media player | +| `ChatInput.tsx` | Text input, voice record, file upload, rate-limit indicator | +| `ChatHistory.tsx` | Virtualized scroll (react-window) | +| `ChatSidebar.tsx` | Channel list, persona selector, theme toggle | +| `Header.tsx` | Title, menu, auth | +| `Nav.tsx` | Route navigation | +| `ErrorBoundary.tsx` | Error UI | +| `VoiceChat.tsx` | Push-to-talk, level meter, silence auto-detect | +| `MediaGallery.tsx` | Image/audio gallery, fullscreen player | +| `MediaExplorer.tsx` | File browser, upload | +| `TimelineView.tsx` | DAW track lanes, waveform, play/pause/seek | +| `EngineNode.tsx` | DAG node visual | +| `NodeEditor.tsx` | DAG editor (React Flow) | +| `PersonaList.tsx` | Persona selector with memory stats | +| `PersonaDetail.tsx` | Persona info, memory injection preview | + +### Hooks (7) + +| Hook | Purpose | +|------|---------| +| `useWebSocket.ts` | WS connection, auto-reconnect, message dispatch | +| `useAppSession.ts` | Session state, auth, guest mode | +| `useChatState.ts` | Chat history, selected persona, channel | +| `useGenerationCommand.ts` | /imagine, /compose submission, progress tracking | +| `useNodeEditor.ts` | DAG state (nodes, edges, zoom) | +| `useHashRoute.ts` | Client-side routing via hash | +| `useKeyboardShortcuts.ts` | Ctrl+K palette, theme toggle, etc. | +| `useMinitelSounds.ts` | 8-bit Minitel UI sounds | + +### Library + +| File | Purpose | +|------|---------| +| `lib/websocket-url.ts` | WS URL construction (dev vs prod) | +| `api.ts` | HTTP client for REST routes | +| `chat-types.ts` | Frontend message/persona types | + +### Tests (10 files) + +| File | Purpose | +|------|---------| +| `components/Chat*.test.tsx` | Component render, interaction | +| `components/Header.test.tsx` | Header UI | +| `components/Login.test.tsx` | Auth flow | +| `components/ChannelList.test.tsx` | Channel selector | +| `components/PersonaList.test.tsx` | Persona list | +| `components/RunStatus.test.tsx` | Run status display | +| `components/Nav.test.tsx` | Route nav | +| `hooks/*.test.ts` | Hook logic | +| `App.test.tsx` | App bootstrap | + +### Styles + +| File | Purpose | +|------|---------| +| `styles.css` | Base theme variables (colors, fonts, spacing) | +| `styles 2.css` | Alternative theme (legacy) | + +### Run + +```bash +npm run dev:v2:web # Vite (localhost:5173) +npm run -w @kxkm/web test # 54 unit tests +``` + +## worker/ — Background Jobs (4 TS files) + +Node.js background processor. Handles async tasks (training, data ingestion, composition rendering, etc.). + +| File | Purpose | +|------|---------| +| `index.ts` | Entry point, job queue setup | +| `worker-runtime.ts` | Job executor: training jobs, DPO pipeline, node runs | +| `logger.ts` | Structured logging | +| `worker-runtime.test.ts` | Runtime tests | + +### Run + +```bash +npm run dev:v2:worker # Background processor +npm run -w @kxkm/worker test +``` diff --git a/apps/api/src/context-store.ts b/apps/api/src/context-store.ts index 28b381b..2e36857 100644 --- a/apps/api/src/context-store.ts +++ b/apps/api/src/context-store.ts @@ -12,6 +12,7 @@ const DEBUG = process.env.NODE_ENV !== "production" || process.env.DEBUG === "1"; import { trackError } from "./error-tracker.js"; +import { scheduler, VRAM_BUDGETS } from "./inference-scheduler.js"; import { promises as fs } from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -96,6 +97,50 @@ function buildDefaultOptions(): ContextStoreOptions { const DEFAULT_OPTIONS: ContextStoreOptions = buildDefaultOptions(); +function extractFirstJsonObject(raw: string): string | null { + const start = raw.indexOf("{"); + if (start < 0) return null; + + let depth = 0; + let inString = false; + let escaped = false; + + for (let index = start; index < raw.length; index += 1) { + const char = raw[index]; + if (inString) { + if (escaped) { + escaped = false; + continue; + } + if (char === "\\") { + escaped = true; + continue; + } + if (char === "\"") { + inString = false; + } + continue; + } + + if (char === "\"") { + inString = true; + continue; + } + if (char === "{") { + depth += 1; + continue; + } + if (char === "}") { + depth -= 1; + if (depth === 0) { + return raw.slice(start, index + 1); + } + } + } + + return null; +} + // --------------------------------------------------------------------------- // Context Store // --------------------------------------------------------------------------- @@ -133,10 +178,53 @@ export class ContextStore { } } + private async writeSummaryFile(filePath: string, summary: ContextSummary): Promise { + const tmp = `${filePath}.${process.pid}.${Date.now().toString(36)}.tmp`; + await fs.writeFile(tmp, `${JSON.stringify(summary, null, 2)}\n`, "utf-8"); + await fs.rename(tmp, filePath); + } + + private normalizeSummary(raw: unknown, channel: string): ContextSummary | null { + if (!raw || typeof raw !== "object") return null; + const parsed = raw as Record; + if (typeof parsed.summaryText !== "string") return null; + return { + channel: typeof parsed.channel === "string" && parsed.channel.length > 0 ? parsed.channel : channel, + summaryText: parsed.summaryText, + entriesCompacted: typeof parsed.entriesCompacted === "number" && Number.isFinite(parsed.entriesCompacted) + ? parsed.entriesCompacted + : 0, + lastCompactedAt: typeof parsed.lastCompactedAt === "string" ? parsed.lastCompactedAt : new Date(0).toISOString(), + totalCompactions: typeof parsed.totalCompactions === "number" && Number.isFinite(parsed.totalCompactions) + ? parsed.totalCompactions + : 0, + }; + } + private async readSummary(channel: string): Promise { + const summaryPath = this.summaryFile(channel); try { - const raw = await fs.readFile(this.summaryFile(channel), "utf-8"); - return this.parseJson(raw); + const raw = await fs.readFile(summaryPath, "utf-8"); + const parsed = this.parseJson(raw); + const normalized = this.normalizeSummary(parsed, channel); + if (normalized) return normalized; + + const recoveredRaw = extractFirstJsonObject(raw); + if (recoveredRaw) { + const recoveredParsed = this.parseJson(recoveredRaw); + const recovered = this.normalizeSummary(recoveredParsed, channel); + if (recovered) { + await this.writeSummaryFile(summaryPath, recovered); + return recovered; + } + } + + const quarantinePath = path.join( + path.dirname(summaryPath), + `${path.basename(summaryPath, ".json")}.corrupt.${Date.now().toString(36)}.json`, + ); + await fs.rename(summaryPath, quarantinePath).catch(() => {}); + return null; } catch { return null; } @@ -315,21 +403,37 @@ export class ContextStore { let summaryText = existingSummary; // fallback try { - const response = await fetch(`${this.options.ollamaUrl}/api/chat`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - model: this.options.compactionModel, - messages: [{ role: "user", content: prompt }], - stream: false, - }), - signal: AbortSignal.timeout(120_000), - }); + summaryText = await scheduler.submit({ + id: `compact-${channel}-${Date.now()}`, + device: "gpu", + priority: "low", + label: `context-compact:${channel}`, + vramMB: VRAM_BUDGETS.ollama, + execute: async () => { + const llmUrl = process.env.LLM_URL || "http://localhost:11434"; + const llmModel = process.env.LLM_MODEL || "qwen-14b-awq"; + const llmApiKey = process.env.LLM_API_KEY || ""; + const llmH: Record = { "Content-Type": "application/json" }; + if (llmApiKey) llmH["Authorization"] = `Bearer ${llmApiKey}`; + const response = await fetch(`${llmUrl}/v1/chat/completions`, { + method: "POST", + headers: llmH, + body: JSON.stringify({ + model: llmModel, + messages: [{ role: "user", content: prompt }], + stream: false, + max_tokens: 2000, + }), + signal: AbortSignal.timeout(120_000), + }); - if (response.ok) { - const data = (await response.json()) as { message?: { content?: string } }; - summaryText = data.message?.content || existingSummary; - } + if (response.ok) { + const data = (await response.json()) as { choices?: [{ message?: { content?: string } }] }; + return data.choices?.[0]?.message?.content || existingSummary; + } + return existingSummary; + }, + }); } catch (err) { trackError("context_summarization", err, { channel }); // Keep existing summary, still compact the raw file @@ -344,7 +448,7 @@ export class ContextStore { totalCompactions: (previousSummary?.totalCompactions || 0) + 1, }; - await fs.writeFile(this.summaryFile(channel), JSON.stringify(summaryData, null, 2), "utf-8"); + await this.writeSummaryFile(this.summaryFile(channel), summaryData); // Replace raw file with only recent entries const newContent = toKeep.join("\n") + "\n"; diff --git a/apps/api/src/llm-client.ts b/apps/api/src/llm-client.ts index 0dd085a..56f060a 100644 --- a/apps/api/src/llm-client.ts +++ b/apps/api/src/llm-client.ts @@ -1,11 +1,11 @@ /** - * LLM Client — routes through mascarade /v1/chat/completions with Ollama fallback + * Canonical LLM client. * - * mascarade /v1/chat/completions endpoint (OpenAI-compatible): - * POST { model, messages, temperature, max_tokens } - * Returns: { model, choices: [{ message: { content } }], usage } + * Local runtime: + * vLLM / TurboQuant exposed via OpenAI-compatible `/v1/chat/completions` * - * Fallback: direct Ollama /api/chat if mascarade is unavailable + * Optional cloud routing: + * Mascarade `/v1/chat/completions` for explicit cloud provider models */ import logger from "./logger.js"; @@ -17,9 +17,10 @@ import { incrementCounter } from "./perf.js"; const MASCARADE_URL = process.env.MASCARADE_URL || "http://127.0.0.1:8100"; const MASCARADE_API_KEY = process.env.MASCARADE_API_KEY || ""; -const OLLAMA_URL = process.env.OLLAMA_URL || "http://127.0.0.1:11434"; +const LLM_URL = process.env.LLM_URL || "http://127.0.0.1:11434"; +const LLM_MODEL = process.env.LLM_MODEL || "qwen-14b-awq"; const LLM_TIMEOUT_MS = parseInt(process.env.LLM_TIMEOUT_MS || "45000", 10); -const DEFAULT_MODEL = process.env.LLM_DEFAULT_MODEL || "qwen3.5:9b"; +const DEFAULT_MODEL = process.env.LLM_DEFAULT_MODEL || LLM_MODEL; // RouteLLM-style complexity routing const ROUTELLM_ENABLED = process.env.ROUTELLM_ENABLED === "true"; @@ -42,7 +43,17 @@ function mascaradeRecheckMs(): number { export interface ChatMessage { role: "system" | "user" | "assistant" | "tool"; content: string; - tool_calls?: Array<{ function: { name: string; arguments: Record } }>; + tool_call_id?: string; + tool_calls?: ChatToolCall[]; +} + +export interface ChatToolCall { + id?: string; + type?: "function"; + function: { + name: string; + arguments: Record | string; + }; } export interface ChatOptions { @@ -61,7 +72,7 @@ export interface ChatResponse { content: string; model: string; provider: string; - toolCalls?: Array<{ function: { name: string; arguments: Record } }>; + toolCalls?: ChatToolCall[]; thinking?: string; usage?: { promptTokens: number; completionTokens: number; totalTokens: number }; } @@ -92,12 +103,12 @@ function shouldTryMascarade(): boolean { } // --------------------------------------------------------------------------- -// Parse model: detect "provider:model" vs plain Ollama model +// Parse model: detect "provider:model" vs local runtime model // --------------------------------------------------------------------------- function parseModel(model: string | undefined): { provider: string | null; model: string } { const m = model || DEFAULT_MODEL; - const knownProviders = ["claude", "openai", "mistral-api", "google", "bedrock", "huggingface", "ollama", "llama_cpp"]; + const knownProviders = ["claude", "openai", "mistral-api", "google", "bedrock", "huggingface"]; const colonIdx = m.indexOf(":"); if (colonIdx > 0) { const prefix = m.slice(0, colonIdx); @@ -105,18 +116,26 @@ function parseModel(model: string | undefined): { provider: string | null; model return { provider: prefix, model: m.slice(colonIdx + 1) }; } } - // Ollama model (qwen3.5:9b, mistral:7b, etc.) — let mascarade route via default provider + // Local runtime model (qwen3.5:9b, qwen-14b-awq, mistral:7b, etc.) return { provider: null, model: m }; } +function resolveRuntimeModel(model: string | undefined): string { + const parsed = parseModel(model); + if (parsed.provider) { + return DEFAULT_MODEL; + } + return parsed.model || DEFAULT_MODEL; +} + // --------------------------------------------------------------------------- // RouteLLM — complexity scoring for smart routing // --------------------------------------------------------------------------- /** * Score message complexity (0-1). Higher = needs stronger model. - * 0.0-0.3: trivial (salut, oui, merci) → local Ollama - * 0.3-0.6: moderate (short questions) → local Ollama + * 0.0-0.3: trivial (salut, oui, merci) → runtime local + * 0.3-0.6: moderate (short questions) → runtime local * 0.6-1.0: complex (analysis, code, multilingual) → strong provider if available */ function scoreComplexity(messages: ChatMessage[]): number { @@ -158,74 +177,67 @@ function scoreComplexity(messages: ChatMessage[]): number { /** * Decide routing based on complexity score. - * Returns "ollama" to force local, "mascarade" to prefer strong provider, or null for default behavior. + * Returns "runtime" to force local, "mascarade" to prefer strong provider, or null for default behavior. */ -function routeByComplexity(messages: ChatMessage[]): { route: "ollama" | "mascarade" | null; complexity: number } { +function routeByComplexity(messages: ChatMessage[]): { route: "runtime" | "mascarade" | null; complexity: number } { if (!ROUTELLM_ENABLED) return { route: null, complexity: -1 }; const complexity = scoreComplexity(messages); if (complexity < ROUTELLM_THRESHOLD) { - return { route: "ollama", complexity }; + return { route: "runtime", complexity }; } return { route: "mascarade", complexity }; } // --------------------------------------------------------------------------- -// Non-streaming: mascarade /send → Ollama fallback +// Non-streaming: local runtime primary, mascarade only for explicit cloud/provider routing // --------------------------------------------------------------------------- export async function chat(messages: ChatMessage[], opts: ChatOptions = {}): Promise { + const { provider } = parseModel(opts.model); const { route, complexity } = routeByComplexity(messages); - // RouteLLM: skip mascarade entirely for simple messages - if (route === "ollama") { - logger.debug({ complexity, threshold: ROUTELLM_THRESHOLD }, "[llm] routeLLM → ollama (simple)"); - return chatViaOllama(messages, opts); + if (!provider) { + if (route === "runtime") { + logger.debug({ complexity, threshold: ROUTELLM_THRESHOLD }, "[llm] routeLLM → runtime (simple)"); + } + return chatViaRuntime(messages, opts); } if (route === "mascarade") { logger.debug({ complexity, threshold: ROUTELLM_THRESHOLD }, "[llm] routeLLM → mascarade (complex)"); } - if (shouldTryMascarade()) { - try { - const result = await chatViaMascarade(messages, opts); - // If mascarade returns empty content, fall through to Ollama - if (result.content) return result; - logger.warn("[llm] mascarade returned empty content, falling back to Ollama"); - } catch (err) { - logger.warn({ err: (err as Error).message }, "[llm] mascarade failed, falling back to Ollama"); - mascaradeAvailable = false; - mascaradeLastCheck = Date.now(); - mascaradeFailCount++; - } + if (!shouldTryMascarade()) { + throw new Error("Mascarade unavailable for explicit cloud provider routing"); } - return chatViaOllama(messages, opts); + return chatViaMascarade(messages, opts); } // --------------------------------------------------------------------------- -// Streaming: mascarade SSE for cloud providers, direct Ollama for local +// Streaming: mascarade SSE for cloud providers, direct runtime for local // --------------------------------------------------------------------------- export async function* streamChat( messages: ChatMessage[], opts: ChatOptions = {}, ): AsyncGenerator { - const isCloudProvider = opts.model && /^(claude|openai|mistral-api|google|bedrock):/.test(opts.model); + const isCloudProvider = Boolean(opts.model && /^(claude|openai|mistral-api|google|bedrock):/.test(opts.model)); // Cloud providers → stream via mascarade SSE (real streaming, not dump-all) if (isCloudProvider && shouldTryMascarade()) { try { return yield* streamViaMascarade(messages, opts); } catch (err) { - logger.warn({ err: (err as Error).message }, "[llm] mascarade stream failed, falling back to Ollama"); + logger.warn({ err: (err as Error).message }, "[llm] mascarade stream failed"); mascaradeAvailable = false; mascaradeLastCheck = Date.now(); mascaradeFailCount++; + throw err; } } - // Local models → stream via Ollama directly (fastest path) - return yield* streamViaOllama(messages, opts); + // Local models → stream via runtime directly (fastest path) + return yield* streamViaRuntime(messages, opts); } // --------------------------------------------------------------------------- @@ -242,7 +254,7 @@ async function chatViaMascarade(messages: ChatMessage[], opts: ChatOptions): Pro const resp = await fetch(`${MASCARADE_URL}/v1/chat/completions`, { method: "POST", - headers: { "Content-Type": "application/json" }, + headers: { "Content-Type": "application/json", ...(process.env.LLM_API_KEY ? { Authorization: `Bearer ${process.env.LLM_API_KEY}` } : {}) }, body: JSON.stringify({ model: modelStr, messages: messages.map(m => ({ role: m.role, content: m.content })), @@ -408,81 +420,94 @@ async function* streamViaMascarade( } // --------------------------------------------------------------------------- -// Direct Ollama (fallback + streaming) +// Direct runtime (vLLM / TurboQuant) // --------------------------------------------------------------------------- -async function chatViaOllama(messages: ChatMessage[], opts: ChatOptions): Promise { +function toOpenAIMessage(message: ChatMessage): Record { + const base: Record = { + role: message.role, + content: message.content, + }; + if (message.tool_calls && message.tool_calls.length > 0) { + base.tool_calls = message.tool_calls.map((toolCall) => ({ + id: toolCall.id, + type: toolCall.type || "function", + function: { + name: toolCall.function.name, + arguments: typeof toolCall.function.arguments === "string" + ? toolCall.function.arguments + : JSON.stringify(toolCall.function.arguments), + }, + })); + } + if (message.role === "tool" && message.tool_call_id) { + base.tool_call_id = message.tool_call_id; + } + return base; +} + +async function chatViaRuntime(messages: ChatMessage[], opts: ChatOptions): Promise { const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), LLM_TIMEOUT_MS); - const { model } = parseModel(opts.model); + const model = resolveRuntimeModel(opts.model); try { - const resp = await fetch(`${OLLAMA_URL}/api/chat`, { + const resp = await fetch(`${LLM_URL}/v1/chat/completions`, { method: "POST", - headers: { "Content-Type": "application/json" }, + headers: { "Content-Type": "application/json", ...(process.env.LLM_API_KEY ? { Authorization: `Bearer ${process.env.LLM_API_KEY}` } : {}) }, body: JSON.stringify({ model, - messages: messages.map(m => ({ role: m.role, content: m.content })), + messages: messages.map(toOpenAIMessage), stream: false, - options: { - num_predict: opts.maxTokens || 800, - ...(opts.numCtx ? { num_ctx: opts.numCtx } : {}), - num_batch: opts.numBatch || 512, - }, - keep_alive: opts.keepAlive || "30m", - ...(opts.think !== undefined ? { think: opts.think } : {}), + temperature: opts.temperature, + max_tokens: opts.maxTokens || 800, ...(opts.tools && opts.tools.length > 0 ? { tools: opts.tools } : {}), }), signal: controller.signal, }); - if (!resp.ok) throw new Error(`Ollama ${resp.status}: ${resp.statusText}`); - incrementCounter("llm_ollama_calls"); + if (!resp.ok) throw new Error(`vLLM ${resp.status}: ${resp.statusText}`); + incrementCounter("llm_runtime_calls"); const data = await resp.json() as { - message?: { content?: string; thinking?: string; tool_calls?: ChatResponse["toolCalls"] }; + choices?: [{ message?: { content?: string; tool_calls?: ChatResponse["toolCalls"] } }]; }; + const msg = data.choices?.[0]?.message; return { - content: data.message?.content || "", + content: msg?.content || "", model, - provider: "ollama", - toolCalls: data.message?.tool_calls, - thinking: data.message?.thinking, + provider: "vllm", + toolCalls: msg?.tool_calls, }; } finally { clearTimeout(timeout); } } -async function* streamViaOllama( +async function* streamViaRuntime( messages: ChatMessage[], opts: ChatOptions, ): AsyncGenerator { const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), LLM_TIMEOUT_MS); - const { model } = parseModel(opts.model); + const model = resolveRuntimeModel(opts.model); try { - const resp = await fetch(`${OLLAMA_URL}/api/chat`, { + const resp = await fetch(`${LLM_URL}/v1/chat/completions`, { method: "POST", - headers: { "Content-Type": "application/json" }, + headers: { "Content-Type": "application/json", ...(process.env.LLM_API_KEY ? { Authorization: `Bearer ${process.env.LLM_API_KEY}` } : {}) }, body: JSON.stringify({ model, - messages: messages.map(m => ({ role: m.role, content: m.content })), + messages: messages.map(toOpenAIMessage), stream: true, - options: { - num_predict: opts.maxTokens || 800, - ...(opts.numCtx ? { num_ctx: opts.numCtx } : {}), - num_batch: opts.numBatch || 512, - }, - keep_alive: opts.keepAlive || "30m", - think: false, + temperature: opts.temperature, + max_tokens: opts.maxTokens || 800, }), signal: controller.signal, }); - if (!resp.ok) throw new Error(`Ollama ${resp.status}: ${resp.statusText}`); + if (!resp.ok) throw new Error(`vLLM ${resp.status}: ${resp.statusText}`); const reader = resp.body?.getReader(); if (!reader) throw new Error("No response body"); @@ -490,18 +515,24 @@ async function* streamViaOllama( const decoder = new TextDecoder(); let fullText = ""; let inThinking = false; + let sseBuffer = ""; while (true) { const { done, value } = await reader.read(); if (done) break; - const chunk = decoder.decode(value, { stream: true }); - for (const line of chunk.split("\n").filter(Boolean)) { - if (line.length > 102_400) continue; // Skip oversized chunks (100KB max) + sseBuffer += decoder.decode(value, { stream: true }); + const lines = sseBuffer.split("\n"); + sseBuffer = lines.pop() || ""; + + for (const line of lines) { + if (!line.startsWith("data: ")) continue; + const raw = line.slice(6).trim(); + if (raw === "[DONE]") break; try { - const parsed = JSON.parse(line) as { message?: { content?: string }; done?: boolean }; - if (parsed.message?.content) { - const c = parsed.message.content; + const parsed = JSON.parse(raw) as { choices?: [{ delta?: { content?: string } }] }; + const c = parsed.choices?.[0]?.delta?.content; + if (c) { fullText += c; if (c.includes("")) inThinking = true; if (!inThinking) yield c; @@ -512,7 +543,7 @@ async function* streamViaOllama( } const cleaned = fullText.replace(/[\s\S]*?<\/think>\s*/g, "").trim(); - return { content: cleaned, model, provider: "ollama" }; + return { content: cleaned, model, provider: "vllm" }; } finally { clearTimeout(timeout); } @@ -523,12 +554,13 @@ async function* streamViaOllama( // --------------------------------------------------------------------------- export async function getProviders(): Promise { + const providers = ["vllm-turboquant"]; try { const resp = await fetch(`${MASCARADE_URL}/health`, { signal: AbortSignal.timeout(3000) }); - if (!resp.ok) return ["ollama"]; + if (!resp.ok) return providers; const data = await resp.json() as { providers?: string[] }; - return data.providers || ["ollama"]; + return [...providers, ...(data.providers || [])]; } catch { - return ["ollama"]; + return providers; } } diff --git a/apps/api/src/rag.ts b/apps/api/src/rag.ts index f9b4a7d..c9e266b 100644 --- a/apps/api/src/rag.ts +++ b/apps/api/src/rag.ts @@ -1,5 +1,6 @@ /** - * Minimal local RAG using Ollama embeddings. + * Minimal local RAG with pluggable embedding backend. + * Supports TEI/OpenAI-compatible (/v1/embeddings) and Ollama (/api/embed). * Stores document chunks with their embeddings in memory. * Uses cosine similarity for retrieval. */ @@ -14,6 +15,7 @@ const RAG_CHUNK_SIZE = Number(process.env.RAG_CHUNK_SIZE) || 500; const RAG_MIN_SIMILARITY = Number(process.env.RAG_MIN_SIMILARITY) || 0.3; const RAG_MAX_RESULTS = Number(process.env.RAG_MAX_RESULTS) || 3; const RAG_EMBEDDING_MODEL = process.env.RAG_EMBEDDING_MODEL || "bge-m3"; +const EMBEDDING_BACKEND = process.env.EMBEDDING_BACKEND || "tei"; // "tei" or "ollama" interface DocumentChunk { id: string; @@ -36,46 +38,75 @@ export class LocalRAG { private options: RAGOptions; private _rerankerFailCount = 0; private _rerankerLastFail = 0; + private namespaceChunks: Map = new Map(); constructor(options: RAGOptions) { this.options = options; } - /** Verify embedding model is available on Ollama, pull if missing. */ + /** Verify embedding backend is reachable. */ async init(): Promise { - const ollamaUrl = this.options.ollamaUrl; + const url = this.options.ollamaUrl; const model = this.options.embeddingModel || RAG_EMBEDDING_MODEL; + const backend = EMBEDDING_BACKEND; try { - const resp = await fetch(`${ollamaUrl}/api/tags`, { signal: AbortSignal.timeout(5000) }); - const data = (await resp.json()) as { models?: Array<{ name: string }> }; - const models = data.models?.map((m) => m.name) || []; - const available = models.some((m) => m.startsWith(model)); - if (!available) { - logger.warn({ model, available: models.slice(0, 5) }, "[rag] Embedding model not found, pulling..."); - await fetch(`${ollamaUrl}/api/pull`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ name: model }), - signal: AbortSignal.timeout(300_000), - }); - logger.info({ model }, "[rag] Embedding model pulled successfully"); + if (backend === "tei") { + // TEI health check + const resp = await fetch(`${url}/info`, { signal: AbortSignal.timeout(5000) }); + if (resp.ok) { + const info = (await resp.json()) as { model_id?: string }; + logger.info({ model: info.model_id, backend }, "[rag] TEI embedding server ready"); + } else { + logger.warn({ status: resp.status }, "[rag] TEI health check failed"); + } } else { - logger.debug({ model }, "[rag] Embedding model available"); + // Ollama: check model availability, pull if missing + const resp = await fetch(`${url}/api/tags`, { signal: AbortSignal.timeout(5000) }); + const data = (await resp.json()) as { models?: Array<{ name: string }> }; + const models = data.models?.map((m) => m.name) || []; + const available = models.some((m) => m.startsWith(model)); + if (!available) { + logger.warn({ model, available: models.slice(0, 5) }, "[rag] Embedding model not found, pulling..."); + await fetch(`${url}/api/pull`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ name: model }), + signal: AbortSignal.timeout(300_000), + }); + logger.info({ model }, "[rag] Embedding model pulled successfully"); + } else { + logger.debug({ model, backend }, "[rag] Embedding model available"); + } } } catch (err) { - logger.warn({ err }, "[rag] Could not verify embedding model"); + logger.warn({ err, backend }, "[rag] Could not verify embedding backend"); } } - /** Embed text via Ollama /api/embed */ + /** Embed text via TEI (/v1/embeddings) or Ollama (/api/embed). */ async embed(text: string): Promise { - const response = await fetch(`${this.options.ollamaUrl}/api/embed`, { + const url = this.options.ollamaUrl; + const model = this.options.embeddingModel || RAG_EMBEDDING_MODEL; + + if (EMBEDDING_BACKEND === "tei") { + // OpenAI-compatible format (TEI, vLLM, etc.) + const response = await fetch(`${url}/v1/embeddings`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ model, input: text }), + }); + if (!response.ok) { + throw new Error(`TEI embed returned ${response.status}: ${response.statusText}`); + } + const data = (await response.json()) as { data?: Array<{ embedding: number[] }> }; + return data.data?.[0]?.embedding || []; + } + + // Ollama format + const response = await fetch(`${url}/api/embed`, { method: "POST", headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - model: this.options.embeddingModel || RAG_EMBEDDING_MODEL, - input: text, - }), + body: JSON.stringify({ model, input: text }), }); if (!response.ok) { throw new Error(`Ollama embed returned ${response.status}: ${response.statusText}`); @@ -86,14 +117,15 @@ export class LocalRAG { /** Add a document (split into chunks, embed each). * If LightRAG is configured, also pushes the full text there (dual write). */ - async addDocument(text: string, source: string): Promise { + async addDocument(text: string, source: string, namespace?: string): Promise { // Dual-write to LightRAG if configured if (this.options.lightragUrl) { try { const res = await fetch(`${this.options.lightragUrl}/documents/text`, { method: "POST", headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ text }), + body: JSON.stringify({ text, description: namespace }), + signal: AbortSignal.timeout(5000), }); if (res.ok) { logger.debug({ source }, "[rag:lightrag] addDocument to LightRAG OK"); @@ -107,16 +139,25 @@ export class LocalRAG { // Always index locally const textChunks = splitIntoChunks(text, RAG_CHUNK_SIZE); + const newChunks: DocumentChunk[] = []; for (const chunk of textChunks) { const embedding = await this.embed(chunk); - this.chunks.push({ + const docChunk: DocumentChunk = { id: `${source}_${this.chunks.length}`, text: chunk, source, embedding, - }); + }; + this.chunks.push(docChunk); + newChunks.push(docChunk); } - return textChunks.length; + if (namespace) { + if (!this.namespaceChunks.has(namespace)) { + this.namespaceChunks.set(namespace, []); + } + this.namespaceChunks.get(namespace)!.push(...newChunks); + } + return newChunks.length; } /** Search for relevant chunks. @@ -183,6 +224,30 @@ export class LocalRAG { return this.rerank(query, results, limit); } + /** Search within a specific persona namespace. Falls back to global search if namespace has no chunks. */ + async searchNamespace( + query: string, + namespace: string, + maxResults?: number, + ): Promise> { + const nsChunks = this.namespaceChunks.get(namespace); + if (!nsChunks || nsChunks.length === 0) { + return this.search(query, maxResults); + } + const limit = maxResults ?? RAG_MAX_RESULTS; + const queryEmbedding = await this.embed(query); + const scored = nsChunks.map((chunk) => ({ + text: chunk.text, + source: chunk.source, + score: cosineSimilarity(queryEmbedding, chunk.embedding), + })); + scored.sort((a, b) => b.score - a.score); + const results = scored + .filter((s) => s.score >= (this.options.minSimilarity ?? RAG_MIN_SIMILARITY)) + .slice(0, limit); + return this.rerank(query, results, limit); + } + /** Rerank results using BGE cross-encoder for improved precision. * Falls back to original ordering if the reranker is unavailable. */ private async rerank( diff --git a/apps/api/src/routes/session.ts b/apps/api/src/routes/session.ts index c0cd71d..31504cc 100644 --- a/apps/api/src/routes/session.ts +++ b/apps/api/src/routes/session.ts @@ -12,6 +12,13 @@ import { validateLoginInput } from "@kxkm/auth"; import { buildChatChannels } from "@kxkm/chat-domain"; import { getRecentErrors, getErrorCounts } from "../error-tracker.js"; import { scheduler, getGPUUtilization } from "../inference-scheduler.js"; + +const LLM_API_KEY = process.env.LLM_API_KEY || ""; +function llmHeaders(): Record { + const h: Record = { "Content-Type": "application/json" }; + if (LLM_API_KEY) h["Authorization"] = `Bearer ${LLM_API_KEY}`; + return h; +} import type { PersonaRecord } from "@kxkm/persona-domain"; import type { ModelRegistryRecord, NodeGraphRecord, NodeRunRecord } from "@kxkm/node-engine"; import type { StorageMode } from "../app-bootstrap.js"; @@ -87,22 +94,22 @@ export function createSessionRoutes(deps: SessionRouteDeps): Router { router.get("/api/v2/health", async (_req, res) => { const startMs = Date.now(); - const ollamaUrl = process.env.OLLAMA_URL || "http://localhost:11434"; + const llmUrl = process.env.LLM_URL || "http://localhost:11434"; const timeout = (p: Promise, ms = 2000): Promise => Promise.race([p, new Promise((_, rej) => setTimeout(() => rej(new Error("timeout")), ms))]); - const [ollamaResult, dbResult] = await Promise.allSettled([ - timeout(fetch(`${ollamaUrl}/api/tags`).then(async (r) => { - const body = await r.json() as { models?: unknown[] }; - return { ok: r.ok, models: Array.isArray(body.models) ? body.models.length : 0 }; + const [runtimeResult, dbResult] = await Promise.allSettled([ + timeout(fetch(`${llmUrl}/v1/models`, { headers: llmHeaders() }).then(async (r) => { + const body = await r.json() as { data?: unknown[] }; + return { ok: r.ok, models: Array.isArray(body.data) ? body.data.length : 0 }; })), timeout(personaRepo.list().then((list) => ({ ok: true, count: list.length }))), ]); - const ollama = ollamaResult.status === "fulfilled" - ? { status: "ok" as const, models_loaded: ollamaResult.value.models } - : { status: "error" as const, error: (ollamaResult.reason as Error).message }; + const runtime = runtimeResult.status === "fulfilled" + ? { status: "ok" as const, models_loaded: runtimeResult.value.models } + : { status: "error" as const, error: (runtimeResult.reason as Error).message }; const db = dbResult.status === "fulfilled" ? { status: "ok" as const, personas: dbResult.value.count } @@ -117,7 +124,7 @@ export function createSessionRoutes(deps: SessionRouteDeps): Router { roles: ["admin", "editor", "operator", "viewer"] satisfies UserRole[], uptime_sec: uptimeSec, uptime_human: uptimeHuman, - ollama, + runtime, database: db, health_check_ms: Date.now() - startMs, })); @@ -194,12 +201,20 @@ export function createSessionRoutes(deps: SessionRouteDeps): Router { res.json(asApiData(listWorkflows())); }); - // LLM providers (mascarade status) + // LLM providers and runtime status router.get("/api/v2/llm-providers", async (_req, res) => { const { getProviders, checkMascaradeHealth } = await import("../llm-client.js"); const healthy = await checkMascaradeHealth(); const providers = await getProviders(); - res.json(asApiData({ mascarade: healthy, providers, fallback: "ollama" })); + res.json(asApiData({ + runtime: { + kind: "vllm-turboquant", + url: process.env.LLM_URL || "http://localhost:11434", + model: process.env.LLM_MODEL || "qwen-14b-awq", + }, + mascarade: healthy, + providers, + })); }); // RAG search endpoint — for mascarade MCP tool integration @@ -433,28 +448,27 @@ export function createSessionRoutes(deps: SessionRouteDeps): Router { router.post("/api/v2/ai/suggest-prompt", async (req, res) => { const { type, style: compStyle, existing, context } = req.body || {}; - const ollamaUrl = process.env.OLLAMA_URL || "http://localhost:11434"; // Image prompt generation mode — triggered by style:"random" or type not in DAW types const isImageMode = compStyle === "random" || type === "image"; if (isImageMode) { + const llmUrl = process.env.LLM_URL || "http://localhost:11434"; + const llmModel = process.env.LLM_MODEL || "qwen-14b-awq"; try { - const resp = await fetch(`${ollamaUrl}/api/chat`, { + const resp = await fetch(`${llmUrl}/v1/chat/completions`, { method: "POST", - headers: { "Content-Type": "application/json" }, + headers: llmHeaders(), body: JSON.stringify({ - model: "qwen3:8b", + model: llmModel, messages: [{ role: "user", content: "Generate a creative, detailed image prompt for AI image generation. Be specific about style, lighting, mood, subject. Return ONLY the prompt in English, nothing else. Maximum 100 words." }], stream: false, - options: { num_predict: 200 }, - keep_alive: "30m", - think: false, + max_tokens: 200, }), signal: AbortSignal.timeout(15000), }); - if (!resp.ok) throw new Error("Ollama error"); - const data = await resp.json() as { message?: { content?: string } }; - const prompt = data.message?.content?.replace(/[\s\S]*?<\/think>/g, "").trim(); + if (!resp.ok) throw new Error("vLLM error"); + const data = await resp.json() as { choices?: [{ message?: { content?: string } }] }; + const prompt = data.choices?.[0]?.message?.content?.replace(/[\s\S]*?<\/think>/g, "").trim(); res.json({ ok: true, data: { prompt } }); } catch { res.json({ ok: true, data: { prompt: "a mystical forest at twilight, bioluminescent mushrooms, fog, cinematic lighting, 8k" } }); @@ -469,22 +483,22 @@ export function createSessionRoutes(deps: SessionRouteDeps): Router { fx: "un effet sonore ou une texture de fond", }; + const llmUrl = process.env.LLM_URL || "http://localhost:11434"; + const llmModel = process.env.LLM_MODEL || "qwen-14b-awq"; try { - const resp = await fetch(`${ollamaUrl}/api/chat`, { + const resp = await fetch(`${llmUrl}/v1/chat/completions`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ - model: "qwen3:8b", + model: llmModel, messages: [{ role: "user", content: `Tu es un compositeur sonore. Génère ${typeHints[type as string] || "un prompt audio"} pour une composition de style "${compStyle || "experimental"}". ${existing ? `Le prompt actuel est: "${existing}". Améliore-le.` : ""} ${context ? `Contexte des autres pistes: ${context}` : ""} Réponds UNIQUEMENT le prompt (1-2 phrases max, pas d'explication).` }], stream: false, - options: { num_predict: 80 }, - keep_alive: "30m", - think: false, + max_tokens: 80, }), signal: AbortSignal.timeout(10000), }); - const data = await resp.json() as { message?: { content?: string } }; - res.json({ ok: true, prompt: data.message?.content?.trim() || "" }); + const data = await resp.json() as { choices?: [{ message?: { content?: string } }] }; + res.json({ ok: true, prompt: data.choices?.[0]?.message?.content?.trim() || "" }); } catch { res.json({ ok: false, prompt: "" }); } diff --git a/apps/api/src/server.ts b/apps/api/src/server.ts index 2b99dae..6a8193e 100644 --- a/apps/api/src/server.ts +++ b/apps/api/src/server.ts @@ -166,27 +166,29 @@ async function main() { // ----------------------------------------------------------------------- const server = http.createServer(app); - const ollamaUrl = process.env.OLLAMA_URL || "http://localhost:11434"; + const embeddingUrl = process.env.OLLAMA_URL || "http://localhost:11435"; // ----------------------------------------------------------------------- - // Pre-warm Ollama: load primary model into VRAM (non-blocking) - // First inference is ~1-2s slower without this. + // Pre-warm vLLM: verify model is loaded (non-blocking) // ----------------------------------------------------------------------- - const primaryModel = process.env.OLLAMA_MODEL || "qwen3.5:9b"; - fetch(`${ollamaUrl}/api/chat`, { + const llmUrl = process.env.LLM_URL || "http://localhost:11434"; + const primaryModel = process.env.LLM_MODEL || "qwen-14b-awq"; + const llmApiKey = process.env.LLM_API_KEY || ""; + const llmHeaders: Record = { "Content-Type": "application/json" }; + if (llmApiKey) llmHeaders["Authorization"] = `Bearer ${llmApiKey}`; + fetch(`${llmUrl}/v1/chat/completions`, { method: "POST", - headers: { "Content-Type": "application/json" }, + headers: llmHeaders, body: JSON.stringify({ model: primaryModel, messages: [{ role: "user", content: "ping" }], stream: false, - options: { num_predict: 1, num_ctx: 512 }, - keep_alive: "30m", + max_tokens: 1, }), }).then(() => { - if (DEBUG) console.log(`[ollama] Pre-warmed ${primaryModel}`); + if (DEBUG) console.log(`[vllm] Pre-warmed ${primaryModel}`); }).catch(() => { - // Ollama not ready yet — model will load on first real request + // vLLM not ready yet — model will serve on first real request }); // Pre-warm ComfyUI: load default checkpoint into VRAM (non-blocking) @@ -195,9 +197,9 @@ async function main() { }); // ----------------------------------------------------------------------- - // Initialize local RAG (embeddings via Ollama) + // Initialize local RAG (embeddings still use the embedding backend) // ----------------------------------------------------------------------- - const rag = new LocalRAG({ ollamaUrl, lightragUrl: process.env.LIGHTRAG_URL, rerankerUrl: process.env.RERANKER_URL }); + const rag = new LocalRAG({ ollamaUrl: embeddingUrl, lightragUrl: process.env.LIGHTRAG_URL, rerankerUrl: process.env.RERANKER_URL }); // Expose RAG instance on app for API routes (app as any)._rag = rag; @@ -233,11 +235,45 @@ async function main() { } })(); + // Per-persona corpus boot-loader + (async () => { + for (const persona of DEFAULT_PERSONAS) { + if (!persona.corpus || persona.corpus.length === 0) continue; + const ns = `persona:${persona.id}`; + for (const entry of persona.corpus) { + try { + let text = ''; + if (entry.type === 'text' && entry.content) { + text = entry.content; + } else if (entry.type === 'url') { + const resp = await fetch(entry.source, { signal: AbortSignal.timeout(10_000) }); + const html = await resp.text(); + // Strip scripts/styles first, then all tags + text = html + .replace(//gi, '') + .replace(//gi, '') + .replace(/<[^>]+>/g, ' ') + .replace(/\s+/g, ' ') + .trim() + .slice(0, 8000); + } + if (text) { + await rag.addDocument(text, entry.source, ns); + logger.info({ persona: persona.id, source: entry.source, ns }, '[rag:corpus] loaded'); + } + } catch (err) { + logger.warn({ err, persona: persona.id, source: entry.source }, '[rag:corpus] load failed (non-critical)'); + } + } + } + logger.info('[rag:corpus] persona corpus boot-loading complete'); + })(); + // ----------------------------------------------------------------------- // Initialize persistent context store (auto-compaction, 750 MB max) // ----------------------------------------------------------------------- const contextStore = new ContextStore({ - ollamaUrl, + ollamaUrl: embeddingUrl, maxTotalSizeMB: 750, maxEntriesBeforeCompact: 200, compactionModel: "qwen3:8b", @@ -249,7 +285,7 @@ async function main() { }); const wss = attachWebSocketChat(server, { - ollamaUrl, + ollamaUrl: embeddingUrl, rag, contextStore, loadPersonas: async () => { @@ -264,6 +300,7 @@ async function main() { color: defaultDef?.color || "", enabled: !(p as unknown as { disabled?: boolean }).disabled, maxTokens: defaultDef?.maxTokens, + corpus: defaultDef?.corpus, }; }); }, @@ -289,7 +326,7 @@ async function main() { app: "@kxkm/api", port, ws: "/ws", - ollama: ollamaUrl, + embeddings: embeddingUrl, })); }); } diff --git a/apps/api/src/ws-ollama.ts b/apps/api/src/ws-ollama.ts index 047892c..c9e5921 100644 --- a/apps/api/src/ws-ollama.ts +++ b/apps/api/src/ws-ollama.ts @@ -6,10 +6,43 @@ import { trackError } from "./error-tracker.js"; import logger from "./logger.js"; import type { ToolDefinition } from "./mcp-tools.js"; import type { ChatPersona } from "./chat-types.js"; +import type { ChatMessage } from "./llm-client.js"; -const FALLBACK_MODEL = process.env.OLLAMA_FALLBACK_MODEL || "qwen3:4b"; +const LLM_URL = process.env.LLM_URL || "http://localhost:11434"; +const LLM_MODEL = process.env.LLM_MODEL || "qwen-14b-awq"; +const LLM_API_KEY = process.env.LLM_API_KEY || ""; -// HTTP keep-alive agent: reuses TCP connections to Ollama (saves ~5-20ms per request) +function resolveRuntimeModel(model: string | undefined): string { + if (!model) return LLM_MODEL; + const cloudPrefixed = /^(claude|openai|mistral-api|google|bedrock|huggingface):/i.test(model); + if (cloudPrefixed) return LLM_MODEL; + const compatPrefixed = /^(ollama|vllm|runtime):/i.test(model); + return compatPrefixed ? model.slice(model.indexOf(":") + 1) : model; +} + +/** Complete a chat via vLLM OpenAI-compatible API (non-streaming). Strips blocks. */ +export async function vllmComplete( + messages: Array<{ role: string; content: string }>, + opts?: { maxTokens?: number; model?: string }, +): Promise { + const resp = await fetch(`${LLM_URL}/v1/chat/completions`, { + method: "POST", + headers: llmHeaders(), + body: JSON.stringify({ + model: resolveRuntimeModel(opts?.model), + messages, + max_tokens: opts?.maxTokens ?? 800, + stream: false, + }), + signal: AbortSignal.timeout(45_000), + }); + if (!resp.ok) throw new Error(`vLLM ${resp.status}: ${resp.statusText}`); + const data = await resp.json() as { choices?: [{ message?: { content?: string } }] }; + return (data.choices?.[0]?.message?.content || "") + .replace(/[\s\S]*?<\/think>\s*/g, "").trim(); +} + +// HTTP keep-alive agent: reuses TCP connections to the local runtime const ollamaAgent = new http.Agent({ keepAlive: true, maxSockets: 10, @@ -30,7 +63,14 @@ try { // undici not available — fall back to default fetch (still OK, just no keep-alive) } -/** Fetch with keep-alive connection pooling to Ollama */ +/** Common headers for LLM runtime requests */ +function llmHeaders(): Record { + const h: Record = { "Content-Type": "application/json" }; + if (LLM_API_KEY) h["Authorization"] = `Bearer ${LLM_API_KEY}`; + return h; +} + +/** Fetch with keep-alive connection pooling to the local runtime */ function ollamaFetch(url: string, init: RequestInit): Promise { return fetch(url, { ...init, ...ollamaFetchOpts } as RequestInit); } @@ -101,11 +141,11 @@ const DEBUG = process.env.NODE_ENV !== "production" || process.env.DEBUG === "1" // Ollama concurrency limiter (replaces manual semaphore) // --------------------------------------------------------------------------- -// Match OLLAMA_NUM_PARALLEL (set via sudo-optimize.sh) +// Match local runtime concurrency limits const ollamaLimit = pLimit(Number(process.env.MAX_OLLAMA_CONCURRENT) || 2); // --------------------------------------------------------------------------- -// Ollama streaming chat +// Local runtime streaming chat // --------------------------------------------------------------------------- export async function streamOllamaChat( @@ -116,61 +156,64 @@ export async function streamOllamaChat( onDone: (fullText: string) => void, onError: (err: Error) => void, ): Promise { - // Always disable thinking for streaming — thinking output goes to separate field, not content stream - const useThinking = false; await ollamaLimit(async () => { const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), 45_000); + const runtimeModel = resolveRuntimeModel(persona.model); + const runtimeUrl = ollamaUrl || LLM_URL; try { - const response = await ollamaFetch(`${ollamaUrl}/api/chat`, { + const response = await ollamaFetch(`${runtimeUrl}/v1/chat/completions`, { method: "POST", - headers: { "Content-Type": "application/json" }, + headers: llmHeaders(), body: JSON.stringify({ - model: persona.model, + model: runtimeModel, messages: [ { role: "system", content: persona.systemPrompt }, { role: "user", content: userMessage }, ], stream: true, - options: { num_predict: estimateMaxTokens(userMessage, persona.maxTokens), num_ctx: estimateNumCtx(persona.systemPrompt, userMessage), num_batch: 512 }, keep_alive: "30m", think: false, + max_tokens: estimateMaxTokens(userMessage, persona.maxTokens), }), signal: controller.signal, }); if (!response.ok) { - throw new Error(`Ollama returned ${response.status}: ${response.statusText}`); + throw new Error(`vLLM returned ${response.status}: ${response.statusText}`); } const reader = response.body?.getReader(); if (!reader) { - throw new Error("No response body from Ollama"); + throw new Error("No response body from vLLM"); } const decoder = new TextDecoder(); let fullText = ""; let inThinking = false; + let sseBuffer = ""; while (true) { const { done, value } = await reader.read(); if (done) break; - const chunk = decoder.decode(value, { stream: true }); - const lines = chunk.split("\n").filter(Boolean); + sseBuffer += decoder.decode(value, { stream: true }); + const lines = sseBuffer.split("\n"); + sseBuffer = lines.pop() || ""; for (const line of lines) { - try { - const parsed = JSON.parse(line) as { message?: { content?: string }; done?: boolean }; - if (parsed.message?.content) { - const c = parsed.message.content; - fullText += c; - // Suppress ... from streaming to client - if (c.includes("")) inThinking = true; - if (!inThinking) onChunk(c); - if (c.includes("")) inThinking = false; + const raw = line.startsWith("data: ") ? line.slice(6).trim() : line.trim(); + if (!raw) continue; + if (raw === "[DONE]") break; + const c = parseStreamingPayload(raw); + if (c) { + fullText += c; + const visible = stripThinkingFromChunk(c); + if (c.includes("")) inThinking = true; + if (visible && !inThinking) onChunk(visible); + if (c.includes("")) { + inThinking = false; + if (visible) onChunk(visible); } - } catch { - // Partial JSON -- skip } } } @@ -179,60 +222,6 @@ export async function streamOllamaChat( const cleaned = fullText.replace(/[\s\S]*?<\/think>\s*/g, "").trim(); onDone(cleaned); } catch (err) { - // Try fallback model if primary fails - if (persona.model !== FALLBACK_MODEL) { - logger.warn({ nick: persona.nick, primaryModel: persona.model, fallback: FALLBACK_MODEL }, "Trying fallback model"); - const fallbackPersona = { ...persona, model: FALLBACK_MODEL }; - try { - const fallbackController = new AbortController(); - const fallbackTimeout = setTimeout(() => fallbackController.abort(), 45_000); - try { - const fallbackResp = await ollamaFetch(`${ollamaUrl}/api/chat`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - model: FALLBACK_MODEL, - messages: [ - { role: "system", content: persona.systemPrompt }, - { role: "user", content: userMessage }, - ], - stream: true, - options: { num_predict: estimateMaxTokens(userMessage, persona.maxTokens), num_ctx: estimateNumCtx(persona.systemPrompt, userMessage), num_batch: 512 }, - keep_alive: "30m", - }), - signal: fallbackController.signal, - }); - if (!fallbackResp.ok) throw new Error(`Fallback returned ${fallbackResp.status}`); - const reader = fallbackResp.body?.getReader(); - if (!reader) throw new Error("No fallback response body"); - const decoder = new TextDecoder(); - let fullText = ""; - let inThinking = false; - while (true) { - const { done, value } = await reader.read(); - if (done) break; - const chunk = decoder.decode(value, { stream: true }); - for (const line of chunk.split("\n").filter(Boolean)) { - try { - const parsed = JSON.parse(line) as { message?: { content?: string } }; - if (parsed.message?.content) { - const c = parsed.message.content; - fullText += c; - if (c.includes("")) inThinking = true; - if (!inThinking) onChunk(c); - if (c.includes("")) inThinking = false; - } - } catch { /* partial JSON */ } - } - } - const cleaned = fullText.replace(/[\s\S]*?<\/think>\s*/g, "").trim(); - onDone(cleaned); - return; - } finally { - clearTimeout(fallbackTimeout); - } - } catch { /* fallback also failed */ } - } trackError("ollama", err, { persona: persona.nick, model: persona.model }); onError(err instanceof Error ? err : new Error(String(err))); } finally { @@ -246,6 +235,12 @@ function stripThinking(text: string): string { return text.replace(/[\s\S]*?<\/think>\s*/g, "").trim(); } +function stripThinkingFromChunk(text: string): string { + return text + .replace(/[\s\S]*?<\/think>/g, "") + .replace(/<\/?think>/g, ""); +} + /** Clean persona response: strip thinking tokens, self-reference prefix, whitespace */ export function cleanPersonaResponse(text: string, personaNick: string): string { let cleaned = stripThinking(text); @@ -260,7 +255,71 @@ export function cleanPersonaResponse(text: string, personaNick: string): string // --------------------------------------------------------------------------- interface OllamaToolCall { - function: { name: string; arguments: Record }; + id?: string; + type?: "function"; + function: { name: string; arguments: Record | string }; +} + +function toRuntimeMessage(message: ChatMessage): Record { + const payload: Record = { + role: message.role, + content: message.content, + }; + if (message.tool_calls?.length) { + payload.tool_calls = message.tool_calls.map((toolCall) => ({ + id: toolCall.id, + type: toolCall.type || "function", + function: { + name: toolCall.function.name, + arguments: typeof toolCall.function.arguments === "string" + ? toolCall.function.arguments + : JSON.stringify(toolCall.function.arguments), + }, + })); + } + if (message.role === "tool" && message.tool_call_id) { + payload.tool_call_id = message.tool_call_id; + } + return payload; +} + +function parseToolArguments(value: Record | string): Record { + if (typeof value !== "string") return value; + try { + const parsed = JSON.parse(value) as Record; + return parsed && typeof parsed === "object" ? parsed : {}; + } catch { + return {}; + } +} + +function parseStreamingPayload(raw: string): string | null { + try { + const parsed = JSON.parse(raw) as { + choices?: [{ delta?: { content?: string } }]; + message?: { content?: string }; + }; + return parsed.choices?.[0]?.delta?.content ?? parsed.message?.content ?? null; + } catch { + return null; + } +} + +function extractAssistantMessage(data: { + choices?: [{ + message?: { + role?: string; + content?: string; + tool_calls?: OllamaToolCall[]; + }; + }]; + message?: { + role?: string; + content?: string; + tool_calls?: OllamaToolCall[]; + }; +}): { role?: string; content?: string; tool_calls?: OllamaToolCall[] } | undefined { + return data.choices?.[0]?.message || data.message; } /** @@ -302,7 +361,7 @@ export async function executeToolCall( : { type: prompt || "pink", duration }; try { const resp = await fetch(`${AI_BRIDGE}${endpoint}`, { - method: "POST", headers: { "Content-Type": "application/json" }, + method: "POST", headers: llmHeaders(), body: JSON.stringify(body), signal: AbortSignal.timeout(60_000), }); return resp.ok ? `[Audio généré: ${type} ${duration}s]` : `[Erreur génération: HTTP ${resp.status}]`; @@ -314,7 +373,7 @@ export async function executeToolCall( const voice = String(args.voice || "af_heart"); try { const resp = await fetch(`${AI_BRIDGE}/generate/voice-fast`, { - method: "POST", headers: { "Content-Type": "application/json" }, + method: "POST", headers: llmHeaders(), body: JSON.stringify({ text, voice, speed: 1.0 }), signal: AbortSignal.timeout(30_000), }); return resp.ok ? `[Voix synthétisée: ${voice}, "${text.slice(0, 50)}"]` : `[Erreur TTS: HTTP ${resp.status}]`; @@ -330,7 +389,7 @@ export async function executeToolCall( } /** - * Stream Ollama chat with optional tool-calling support. + * Stream local runtime chat with optional tool-calling support. * When tools are provided: * 1. First do a non-streaming call to see if Ollama wants to use tools * 2. If tool_calls present, execute them and re-call with tool results @@ -368,66 +427,54 @@ export async function streamOllamaChatWithTools( await ollamaLimit(async () => { const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), 45_000); + const runtimeUrl = ollamaUrl || LLM_URL; try { - const messages: Array<{ role: string; content: string; tool_calls?: OllamaToolCall[] }> = [ + const runtimeModel = resolveRuntimeModel(persona.model); + const messages: ChatMessage[] = [ { role: "system", content: persona.systemPrompt }, { role: "user", content: userMessage }, ]; - // Step 1: Non-streaming probe with tools (only for tool-like messages) - const probeResp = await ollamaFetch(`${ollamaUrl}/api/chat`, { + // Step 1: Non-streaming probe with tools + const probeResp = await ollamaFetch(`${runtimeUrl}/v1/chat/completions`, { method: "POST", - headers: { "Content-Type": "application/json" }, + headers: llmHeaders(), body: JSON.stringify({ - model: persona.model, - messages, + model: runtimeModel, + messages: messages.map(toRuntimeMessage), tools: tools.map(t => t), stream: false, - options: { num_predict: estimateMaxTokens(userMessage, persona.maxTokens), num_ctx: estimateNumCtx(persona.systemPrompt, userMessage), num_batch: 512 }, keep_alive: "30m", think: shouldThink(userMessage, persona.model) ? undefined : false, + max_tokens: estimateMaxTokens(userMessage, persona.maxTokens), }), signal: controller.signal, }); if (!probeResp.ok) { - throw new Error(`Ollama returned ${probeResp.status}: ${probeResp.statusText}`); + throw new Error(`vLLM returned ${probeResp.status}: ${probeResp.statusText}`); } const probeData = await probeResp.json() as { + choices?: [{ + message?: { + role?: string; + content?: string; + tool_calls?: OllamaToolCall[]; + }; + }]; message?: { role?: string; - content?: string; thinking?: string; + content?: string; tool_calls?: OllamaToolCall[]; }; }; - const toolCalls = probeData.message?.tool_calls; + const probeMsg = extractAssistantMessage(probeData); + const toolCalls = probeMsg?.tool_calls; // If no tool calls, use the response directly if (!toolCalls || toolCalls.length === 0) { - let content = stripThinking(probeData.message?.content || ""); - // qwen3.5 thinking mode: content may be empty with reasoning in thinking field - if (!content && probeData.message?.thinking) { - const thinking = probeData.message.thinking; - // Strip ... tags first - const stripped = thinking.replace(/[\s\S]*?<\/think>\s*/g, "").trim(); - if (stripped) { - content = stripped; - } else { - // Try to extract answer after markers (FR/EN) - const answerMatch = thinking.match(/(?:Answer|Response|Réponse|Output|Conclusion)\s*:\s*([\s\S]+)$/i); - if (answerMatch) { - content = answerMatch[1].trim(); - } else { - // Last resort: take last substantial paragraph - const paragraphs = thinking.split("\n\n").filter(p => p.trim().length > 20); - content = paragraphs[paragraphs.length - 1]?.trim() || thinking.trim(); - } - } - if (content) { - logger.debug("[ollama] extracted content from thinking field (tool probe)"); - } - } + const content = stripThinking(probeMsg?.content || ""); if (content) { onChunk(content); } @@ -438,13 +485,13 @@ export async function streamOllamaChatWithTools( // Step 2: Execute tool calls (max 1 round) messages.push({ role: "assistant", - content: probeData.message?.content || "", + content: probeMsg?.content || "", tool_calls: toolCalls, }); for (const tc of toolCalls) { const name = tc.function.name; - const args = tc.function.arguments; + const args = parseToolArguments(tc.function.arguments); if (DEBUG) console.log(`[mcp-tools] ${persona.nick} calling ${name}(${JSON.stringify(args)})`); let result: string; @@ -457,50 +504,53 @@ export async function streamOllamaChatWithTools( messages.push({ role: "tool", content: result, + tool_call_id: tc.id, }); } // Step 3: Stream the final response with tool context - const streamResp = await ollamaFetch(`${ollamaUrl}/api/chat`, { + const streamResp = await ollamaFetch(`${runtimeUrl}/v1/chat/completions`, { method: "POST", - headers: { "Content-Type": "application/json" }, + headers: llmHeaders(), body: JSON.stringify({ - model: persona.model, - messages, + model: runtimeModel, + messages: messages.map(toRuntimeMessage), stream: true, - options: { num_predict: estimateMaxTokens(userMessage, persona.maxTokens), num_ctx: estimateNumCtx(persona.systemPrompt, userMessage), num_batch: 512 }, keep_alive: "30m", think: false, + max_tokens: estimateMaxTokens(userMessage, persona.maxTokens), }), signal: controller.signal, }); if (!streamResp.ok) { - throw new Error(`Ollama returned ${streamResp.status}: ${streamResp.statusText}`); + throw new Error(`vLLM returned ${streamResp.status}: ${streamResp.statusText}`); } const reader = streamResp.body?.getReader(); if (!reader) { - throw new Error("No response body from Ollama"); + throw new Error("No response body from vLLM"); } const decoder = new TextDecoder(); let fullText = ""; + let toolStreamBuf = ""; while (true) { const { done, value } = await reader.read(); if (done) break; - const chunk = decoder.decode(value, { stream: true }); - const lines = chunk.split("\n").filter(Boolean); + toolStreamBuf += decoder.decode(value, { stream: true }); + const tsLines = toolStreamBuf.split("\n"); + toolStreamBuf = tsLines.pop() || ""; - for (const line of lines) { - try { - const parsed = JSON.parse(line) as { message?: { content?: string }; done?: boolean }; - if (parsed.message?.content) { - fullText += parsed.message.content; - onChunk(parsed.message.content); - } - } catch { - // Partial JSON -- skip + for (const line of tsLines) { + const raw = line.startsWith("data: ") ? line.slice(6).trim() : line.trim(); + if (!raw) continue; + if (raw === "[DONE]") break; + const c = parseStreamingPayload(raw); + if (c) { + fullText += c; + const visible = stripThinkingFromChunk(c); + if (visible) onChunk(visible); } } } @@ -517,11 +567,11 @@ export async function streamOllamaChatWithTools( // --------------------------------------------------------------------------- // LLM Client — mascarade-backed streaming (OpenAI-compatible) -// Falls back to direct Ollama if mascarade is unavailable. +// Uses mascarade only for explicit cloud-provider streaming. // Drop-in replacement for streamOllamaChat with same signature. // --------------------------------------------------------------------------- -import { streamChat as llmStreamChat, type ChatMessage } from "./llm-client.js"; +import { streamChat as llmStreamChat } from "./llm-client.js"; const USE_MASCARADE = process.env.USE_MASCARADE !== "0"; // enabled by default diff --git a/apps/api/src/ws-persona-router.ts b/apps/api/src/ws-persona-router.ts index a7e4e0c..9dcd282 100644 --- a/apps/api/src/ws-persona-router.ts +++ b/apps/api/src/ws-persona-router.ts @@ -17,6 +17,7 @@ import { resetPersonaMemory, savePersonaMemory, } from "./persona-memory-store.js"; +import { scheduler, VRAM_BUDGETS } from "./inference-scheduler.js"; // --------------------------------------------------------------------------- // Persona memory (persistent, file-based) @@ -27,32 +28,48 @@ export async function updatePersonaMemory( persona: ChatPersona, recentMessages: string[], ollamaUrl: string, + userNick: string = "_anonymous", ): Promise { const startedAt = performance.now(); - const memory = await loadPersonaMemory(persona); + const memory = await loadPersonaMemory(persona.id || persona.nick, userNick, persona.nick); const policy = resolvePersonaMemoryPolicy(); const prompt = buildPersonaMemoryExtractionPrompt(persona, recentMessages, policy); recordPersonaMemoryAttempt(persona); try { - const response = await fetch(`${ollamaUrl}/api/chat`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - model: persona.model, - messages: [{ role: "user", content: prompt }], - stream: false, - format: "json", - }), - signal: AbortSignal.timeout(30_000), + const llmUrl = process.env.LLM_URL || "http://localhost:11434"; + const llmModel = process.env.LLM_MODEL || "qwen-14b-awq"; + const llmApiKey = process.env.LLM_API_KEY || ""; + const llmH: Record = { "Content-Type": "application/json" }; + if (llmApiKey) llmH["Authorization"] = `Bearer ${llmApiKey}`; + const data = await scheduler.submit<{ choices?: [{ message?: { content?: string } }] }>({ + id: `memory-extract-${persona.id}-${Date.now()}`, + device: "gpu", + priority: "low", + label: `memory-extract:${persona.id}`, + vramMB: VRAM_BUDGETS.ollama, + execute: async () => { + const response = await fetch(`${llmUrl}/v1/chat/completions`, { + method: "POST", + headers: llmH, + body: JSON.stringify({ + model: llmModel, + messages: [{ role: "user", content: prompt }], + stream: false, + max_tokens: 800, + response_format: { type: "json_object" }, + }), + signal: AbortSignal.timeout(30_000), + }); + + if (!response.ok) { + throw new Error(`Memory update HTTP ${response.status}`); + } + + return response.json() as Promise<{ choices?: [{ message?: { content?: string } }] }>; + }, }); - - if (!response.ok) { - throw new Error(`Memory update HTTP ${response.status}`); - } - - const data = (await response.json()) as { message?: { content?: string } }; - const rawContent = String(data.message?.content || "").trim(); + const rawContent = String(data.choices?.[0]?.message?.content || "").trim(); if (!rawContent) { logger.error({ nick: persona.nick }, "[persona-router] Empty LLM JSON"); recordPersonaMemorySkip(persona, "empty_response"); @@ -74,7 +91,7 @@ export async function updatePersonaMemory( recentMessages, }); - await savePersonaMemory(updated, policy); + await savePersonaMemory(updated, userNick, policy); const durationMs = performance.now() - startedAt; recordLatency("persona_memory_update", durationMs); recordPersonaMemoryWrite(persona, memory, updated, durationMs); diff --git a/docker-compose.yml b/docker-compose.yml index 9121ad9..01de7a2 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -6,13 +6,15 @@ # docker compose --profile v1 up -d # V1 app + postgres # docker compose --profile v2 up -d # V2 api + worker + postgres # docker compose --profile v1 --profile v2 up -d # V1 + V2 + postgres -# docker compose --profile ollama up -d # include Ollama container +# docker compose --profile ollama up -d # include Ollama-compatible embeddings backend # -# By default, Ollama is expected to run natively on the host (port 11434). -# Set OLLAMA_URL in .env to override. Use --profile ollama to run it in Docker. +# By default, the primary LLM runtime is expected on the host (LLM_URL, port 11434). +# OLLAMA_URL is retained for embeddings/RAG compatibility. # --------------------------------------------------------------------------- x-common-env: &common-env + LLM_URL: "${LLM_URL:-http://host.docker.internal:11434}" + LLM_MODEL: "${LLM_MODEL:-qwen-14b-awq}" OLLAMA_URL: "${OLLAMA_URL:-http://host.docker.internal:11434}" DATABASE_URL: postgres://kxkm:kxkm@postgres:5432/kxkm_clown NODE_ENV: production @@ -68,6 +70,8 @@ services: - ./scripts:/app/scripts:ro - /home/kxkm/openDAW/packages/app/studio/dist:/app/daw:ro environment: + LLM_URL: "${LLM_URL:-http://localhost:11434}" + LLM_MODEL: "${LLM_MODEL:-qwen-14b-awq}" OLLAMA_URL: "http://localhost:11434" DATABASE_URL: "postgres://kxkm:kxkm@localhost:5432/kxkm_clown" NODE_ENV: production @@ -112,6 +116,8 @@ services: network_mode: host command: ["node", "apps/worker/dist/index.js"] environment: + LLM_URL: "${LLM_URL:-http://localhost:11434}" + LLM_MODEL: "${LLM_MODEL:-qwen-14b-awq}" OLLAMA_URL: "http://localhost:11434" DATABASE_URL: "postgres://kxkm:kxkm@localhost:5432/kxkm_clown" NODE_ENV: production @@ -153,7 +159,7 @@ services: start_period: 5s # ------------------------------------------------------------------------- - # Ollama — LLM inference server (optional, use --profile ollama) + # Ollama-compatible embeddings backend (optional, use --profile ollama) # ------------------------------------------------------------------------- ollama: image: ollama/ollama:latest @@ -304,7 +310,7 @@ services: start_period: 120s # ------------------------------------------------------------------------- - # LightRAG — Graph RAG server (Ollama backend) + # LightRAG — Graph RAG server # API: POST /query, POST /documents/text, GET /health # Web UI: http://localhost:9621 # ------------------------------------------------------------------------- @@ -323,7 +329,7 @@ services: LLM_MODEL: "qwen3:8b" EMBEDDING_MODEL: "nomic-embed-text" EMBEDDING_DIM: "768" - OLLAMA_HOST: "http://localhost:11434" + OLLAMA_HOST: "${OLLAMA_URL:-http://localhost:11434}" LLM_BINDING: ollama EMBEDDING_BINDING: ollama RAG_DIR: "/data/lightrag" @@ -383,9 +389,25 @@ services: depends_on: - prometheus + # --- Text Embeddings Inference (dedicated embedding server) ----------------- + # Usage: docker compose --profile embeddings up -d + # Exposes OpenAI-compatible /v1/embeddings on port 9500 + # ------------------------------------------------------------------------- + tei: + image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.6 + container_name: kxkm-tei + restart: unless-stopped + profiles: [v2, embeddings] + ports: + - "9500:80" + volumes: + - tei-models:/data + command: --model-id BAAI/bge-m3 --port 80 + volumes: app-data: pg-data: ollama-data: + tei-models: prometheus-data: grafana-data: diff --git a/packages/AGENTS.md b/packages/AGENTS.md new file mode 100644 index 0000000..1a8baf6 --- /dev/null +++ b/packages/AGENTS.md @@ -0,0 +1,141 @@ +# AGENTS.md — packages/ + + + +8 shared packages in npm workspace. Exported as `@kxkm/*` scope. + +## core — Shared Types & Constants (2 TS files) + +| File | Purpose | +|------|---------| +| `index.ts` | Persona IDs, channel constants, permission levels, errors | +| `index.test.ts` | Type checks, constant validation | + +Core IDs: 33 personas (Pharmacius, Sherlock, Turing, Ikeda, Schaeffer, Merzbow, Pina, etc.). Permissions: read, write, admin. + +**Used by**: all packages and apps. + +## auth — Authentication & RBAC (2 TS files) + +| File | Purpose | +|------|---------| +| `index.ts` | RBAC middleware, session validation, JWT verify, guest mode | +| `index.test.ts` | Auth flow, token expiry, guest access | + +Session storage: PostgreSQL (sessionRepo). Guest mode: read-only routes. RBAC: 3 levels (guest, user, admin). + +**Used by**: apps/api (middleware), apps/web (session hooks). + +## chat-domain — Message & Command Types (2 TS files) + +| File | Purpose | +|------|---------| +| `index.ts` | ChatMessage union, Channel, Command registry, slash command definitions | +| `index.test.ts` | Message validation, command parsing | + +43 slash commands: /chat, /imagine, /compose, /help, /speed, etc. Message types: text, image, audio, error, system. + +**Used by**: apps/api (ws-chat), apps/web (UI). + +## persona-domain — Persona Definitions & Memory (4 TS files) + +| File | Purpose | +|------|---------| +| `index.ts` | Persona model, memoryMode (auto/explicit/off), corpus[], relations[], DPO pair definitions | +| `editorial.ts` | Editorial pipeline: persona proposal, feedback collection, DPO training triggers | +| `pharmacius.ts` | Pharmacius persona specialization (meta-reflection, composability) | +| `index.test.ts` | Persona validation, memory mode tests | + +33 personas with per-persona: memory mode, corpus URLs, related personas (depth-3 relay), voice sample. DPO pair collection: user feedback → training pipeline → Unsloth fine-tuning. + +**Used by**: apps/api (ws-chat, persona-runtime), packages/node-engine (training). + +## node-engine — DAG Execution & Job Queue (6 TS files) + +| File | Purpose | +|------|---------| +| `index.ts` | Node registry, DAG validator, run executor | +| `registry.ts` | 15+ node types: text_generation, image_generation, music_generation, audio_effects, voice_clone, document_extraction, sql_query, etc. | +| `sandbox.ts` | Isolated node execution, timeout, resource limits | +| `training.ts` | Training job: DPO pair collection, Unsloth adapter, registry update | +| `registry.test.ts` | Node type validation | +| `index.test.ts` | DAG execution, queue, run state machine | + +GPU-aware queue: submits to `inference-scheduler.ts` for LLM nodes. Training nodes: trigger via worker. + +**Used by**: apps/api (routes/node-engine), apps/worker (job executor), packages/storage (run persistence). + +## storage — PostgreSQL Persistence (8 TS files) + +| File | Purpose | +|------|---------| +| `index.ts` | Repo factory, migration runner | +| `config.test.ts` | DB connection, pool, transaction tests | +| `migration.test.ts` | Schema versioning | +| `session-repo.test.ts` | Session CRUD, expiry cleanup | +| `persona-repo.test.ts` | Persona memory, feedback store | +| `node-engine-repo.test.ts` | Run state, output, logs | +| `test-helpers.ts` | Test DB setup, fixtures | + +Repos: SessionRepo (auth), PersonaRepo (memory, DPO feedback), NodeEngineRepo (DAG runs, training jobs). Migrations: auto-run on startup. + +**Used by**: apps/api (all routes), apps/worker (job persistence). + +## tui — Terminal UI Utilities (3 TS files) + +| File | Purpose | +|------|---------| +| `index.ts` | CLI argument parsing, color codes, progress bars, table format | +| `index.test.ts` | Formatter tests | + +Used by scripts: health-check.sh, deep-audit.js, ops-tui.sh. Outputs JSON + ANSI colors for logs. + +**Used by**: scripts/, ops/ (monitoring). + +## ui — Experimental React Components (2 TS files) + +| File | Purpose | +|------|---------| +| `index.ts` | Button, Input, Select, Modal, Spinner components (unstyled, Tailwind-ready) | +| `index.test.ts` | Component render tests | + +Minimal, reusable. Not currently used in apps/web (which has inline components). + +## Package Dependencies + +``` +core (no deps, baseline types) +auth → core +chat-domain → core +persona-domain → core, chat-domain +node-engine → core, persona-domain +storage → core, auth, chat-domain, persona-domain, node-engine +tui → core +ui → core +``` + +## Build & Test + +```bash +npm run -w @kxkm/core build +npm run -w @kxkm/auth test +npm run -w @kxkm/chat-domain test +npm run -w @kxkm/persona-domain test +npm run -w @kxkm/node-engine test +npm run -w @kxkm/storage test +npm run -w @kxkm/tui test +npm run -w @kxkm/ui test +``` + +## Export Pattern + +Each package exports: +- TypeScript types (.ts) +- CommonJS build (dist/index.js) +- Type declarations (dist/index.d.ts) + +```typescript +import { Persona, PersonaMemoryMode } from '@kxkm/persona-domain'; +import { Node, DAGRun } from '@kxkm/node-engine'; +import { Session } from '@kxkm/auth'; +``` diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md new file mode 100644 index 0000000..354142c --- /dev/null +++ b/scripts/AGENTS.md @@ -0,0 +1,148 @@ +# AGENTS.md — scripts/ + + + +42 files (20 Python + 22 Shell) organized by domain: ops, training, ingestion, voice, image, deployment. + +## Ops & Monitoring (8 files) + +| File | Type | Purpose | +|------|------|---------| +| `health-check.sh` | Shell | 19 checks (API, DB, Ollama, TTS, LightRAG, docker) with color output | +| `deep-audit.js` | Node | Comprehensive audit: hot spots, error rates, perf p50/p95/p99, GPU VRAM, memory leaks | +| `ops-tui.sh` | Shell | Interactive ops menu (deploy, restart services, logs, cleanup) | +| `service-status.sh` | Shell | systemd service status (kxkm-tts, kxkm-lightrag, kxkm-api) | +| `health-doc-search.sh` | Shell | LightRAG embedding health check | +| `health-embeddings.sh` | Shell | Ollama embedding model health | +| `health-voice-clone.sh` | Shell | TTS voice clone model check | +| `journald-monitor.sh` | Shell | Tail systemd logs with filtering (error, warning) | + +## Deployment (5 files) + +| File | Type | Purpose | +|------|------|---------| +| `deploy.sh` | Shell | Systemd service deployment (kxkm-api, kxkm-tts, kxkm-lightrag), auto-restart on fail | +| `rollback-v2.js` | Node | Rollback to previous version (git revert + service restart) | +| `setup-voice-clone.sh` | Shell | Download XTTS-v2 model, configure TTS | +| `qwen3-tts-ondemand.sh` | Shell | Start on-demand TTS server (not systemd) | +| `ollama-import-adapter.sh` | Shell | Import fine-tuned models into Ollama, register in registry.json | + +## Training (7 files) + +| File | Type | Purpose | +|------|------|---------| +| `train_unsloth.py` | Python | Unsloth fine-tuning pipeline: SFT or DPO, load persona-specific dataset, save to models/finetuned/ | +| `eval_model.py` | Python | Evaluate model: perplexity, BLEU, custom benchmarks, compare against baseline | +| `dpo-pipeline.js` | Node | DPO pair collection automation: fetch feedback from DB, format pairs, trigger training | +| `dpo-export.sh` | Shell | Export DPO pairs to JSONL for analysis | +| `orchestrate_batches.py` | Python | Batch orchestration: schedule training jobs across GPU, queue management | +| `migrate-persona-store-v2.js` | Node | Data migration: v1 persona memory → v2 nick-isolated structure | +| `parity-check.js` | Node | V1 ↔ V2 behavior parity validation | + +## Ingestion & RAG (5 files) + +| File | Type | Purpose | +|------|------|---------| +| `ingest_spectacle_corpus.py` | Python | LightRAG corpus ingestion: seed URLs + SearXNG discovery + sherlock-discovered-urls.jsonl, Camoufox for bot-protected sites | +| `extract_pdf_docling.py` | Python | Extract text/tables from PDFs via Docling, chunk for embedding | +| `reranker-server.py` | Python | BGE-M3 reranker service (listen :8090) for LightRAG results | +| `generate_image.py` | Python | ComfyUI image generation wrapper (checkpoint + LoRA selection, queue submission) | +| `generate-persona-dialogues.js` | Node | Generate synthetic persona dialogues for DPO training | + +## Voice & Audio (5 files) + +| File | Type | Purpose | +|------|------|---------| +| `tts-server.py` | Python | Dual TTS backend (Piper + Chatterbox) on :9100, HTTP API | +| `qwen3-tts-server.py` | Python | Qwen3 TTS server alternative (on-demand) | +| `tts_synthesize.py` | Python | Batch TTS synthesis: text → WAV, save to media-store | +| `tts_clone_voice.py` | Python | Voice cloning: sample ingestion, XTTS-v2 fine-tune, register voice | +| `generate-voice-samples.js` | Node | Generate persona voice samples (read persona bio via TTS) | + +## Testing & Validation (5 files) + +| File | Type | Purpose | +|------|------|---------| +| `run-playwright-e2e.sh` | Shell | Run Playwright E2E tests (login, chat, upload, admin) | +| `test-e2e.js` | Node | E2E test runner (WebDriver + assertions) | +| `test-v2.js` | Node | V2 integration tests (API + WS) | +| `smoke-test.sh` | Shell | Quick smoke: API health, DB connect, Ollama respond | +| `smoke-v2.js` | Node | V2 smoke: WS chat, inference, persona memory | + +## Utilities (5 files) + +| File | Type | Purpose | +|------|------|---------| +| `chat-pause.sh` | Shell | Pause chat (set CHAT_PAUSED env var or create data/chat-paused file) | +| `cleanup-logs.sh` | Shell | Rotate + compress logs older than 7 days, purge after 30 days | +| `cleanup-test-compositions.js` | Node | Delete test composition artifacts | +| `ollama-warmup.sh` | Shell | Pre-load LLM models into Ollama memory (avoid cold start) | +| `sudo-optimize.sh` | Shell | System optimization (disable swap, tune kernel params) | + +## Specialized (2 files) + +| File | Type | Purpose | +|------|------|---------| +| `transcribe_audio.py` | Python | Batch audio → text via OpenAI Whisper or ElevenLabs Scribe | +| `xtts_clone.py` | Python | XTTS-v2 voice cloning (wrapper) | + +## Integration & Agent Scripts (4 files) + +| File | Type | Purpose | +|------|------|---------| +| `v2-agent-task.js` | Node | Execute agent task from PLAN.md (sync, run, log) | +| `v2-autoresearch-loop.js` | Node | Continuous autoresearch: fetch OSS projects, benchmark, report | +| `v2-dpo-pipeline.js` | Node | Automated DPO: feedback → pairs → training trigger → registry update | +| `discord-pharmacius.js` | Node | Discord bot integration (Pharmacius agent) | + +## Data Scripts (1 file) + +| File | Type | Purpose | +|------|------|---------| +| `patch-plan.py` | Python | Update PLAN.md lot state (experimental, use git instead) | + +## Pattern: npm run scripts + +All scripts in `package.json`: + +```json +{ + "scripts": { + "health:v2": "bash scripts/health-check.sh", + "audit:deep": "node scripts/deep-audit.js", + "deploy": "bash scripts/deploy.sh", + "train": "python scripts/train_unsloth.py", + "test:e2e": "bash scripts/run-playwright-e2e.sh" + } +} +``` + +## Run Examples + +```bash +npm run health:v2 # Run health checks +node scripts/deep-audit.js # Deep system audit +bash scripts/deploy.sh # Deploy systemd services +python scripts/train_unsloth.py --dpo --model qwen-14b # Train +bash scripts/run-playwright-e2e.sh # E2E tests +bash scripts/health-check.sh # 19-point health check +``` + +## Environment Variables (scripts expect) + +```bash +LLM_URL=http://localhost:11434 +DATABASE_URL=postgres://kxkm:kxkm@localhost:5432/kxkm_clown +SEARXNG_URL=http://localhost:8080 +LIGHTRAG_URL=http://localhost:9621 +TTS_URL=http://localhost:9100 +CAMOUFOX_URL=http://localhost:8091 +DISCORD_TOKEN=xxx # For discord-pharmacius.js +``` + +## Conventions + +- **Exit codes**: 0 = success, 1 = error, 2 = skipped +- **Output**: JSON for machine parsing (health-check.sh, deep-audit.js), human-readable tables for TUI +- **Logging**: scripts/logs/*.log (rotated daily by cleanup-logs.sh) +- **Data**: ephemeral files in data/ (sherlock-discovered-urls.jsonl, chat-logs/, etc.)