feat: wire vLLM + TEI embedding backend
- LLM_API_KEY support across all LLM calls - TEI embedding server (bge-m3, :9500) - Hierarchical AGENTS.md documentation - CLAUDE.md updated for new architecture
This commit is contained in:
Codex Local
+16
-8
@@ -5,16 +5,24 @@
|
||||
# cp .env.example .env
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# --- Ollama (LLM inference) ------------------------------------------------
|
||||
# If Ollama runs natively on the host, use host.docker.internal (default).
|
||||
# If using the Ollama Docker container (--profile ollama), use http://ollama:11434
|
||||
OLLAMA_URL=http://host.docker.internal:11434
|
||||
# --- Local LLM runtime (vLLM / TurboQuant) ---------------------------------
|
||||
# Primary chat/completion runtime. Must expose an OpenAI-compatible API.
|
||||
LLM_URL=http://host.docker.internal:11434
|
||||
LLM_MODEL=qwen-14b-awq
|
||||
LLM_API_KEY= # Bearer token for vLLM --api-key (leave empty for Ollama)
|
||||
|
||||
# --- Embeddings backend (auxiliary) ----------------------------------------
|
||||
# TEI (recommended): dedicated embedding server on port 9500
|
||||
# Ollama: fallback, uses /api/embed on port 11434
|
||||
OLLAMA_URL=http://host.docker.internal:9500
|
||||
EMBEDDING_BACKEND=tei # "tei" (OpenAI /v1/embeddings) or "ollama" (/api/embed)
|
||||
RAG_EMBEDDING_MODEL=BAAI/bge-m3 # Model name (must match TEI --model-id or Ollama model)
|
||||
|
||||
# --- Ports ------------------------------------------------------------------
|
||||
APP_PORT=3333 # V1 Express server
|
||||
API_PORT=4180 # V2 API server
|
||||
PG_PORT=5432 # PostgreSQL (exposed to host)
|
||||
# OLLAMA_PORT=11434 # Only needed with --profile ollama
|
||||
# LLM_PORT=11434 # Only needed if you expose the local runtime from compose
|
||||
|
||||
# --- Admin ------------------------------------------------------------------
|
||||
ADMIN_BOOTSTRAP_TOKEN= # Initial admin token (set a strong secret)
|
||||
@@ -25,7 +33,7 @@ OWNER_NICK= # Owner nickname in chat
|
||||
MAX_GENERAL_RESPONDERS=4 # Max personas responding in #general
|
||||
|
||||
# --- Vision (image analysis in chat) ----------------------------------------
|
||||
# VISION_MODEL=qwen3-vl:8b # Ollama model for image analysis (qwen3-vl recommandé)
|
||||
# VISION_MODEL=qwen3-vl:8b # Vision model used by the configured runtime
|
||||
|
||||
# --- Training (Node Engine worker) ------------------------------------------
|
||||
# PYTHON_BIN=/home/kxkm/venv/bin/python3 # Python with ML libs (PyTorch, Unsloth, TRL)
|
||||
@@ -37,8 +45,8 @@ MAX_GENERAL_RESPONDERS=4 # Max personas responding in #general
|
||||
# --- ComfyUI (image generation) --------------------------------------------
|
||||
# COMFYUI_URL=http://localhost:8188 # ComfyUI API endpoint for /imagine command
|
||||
|
||||
# --- Mascarade (LLM orchestrator) ------------------------------------------
|
||||
# MASCARADE_URL=http://127.0.0.1:8100 # Mascarade API (OpenAI-compatible /v1/chat/completions)
|
||||
# --- Mascarade (optional cloud/provider routing) ----------------------------
|
||||
# MASCARADE_URL=http://127.0.0.1:8100 # Optional cloud/provider router
|
||||
# MASCARADE_API_KEY= # API key for authenticated endpoints (/agents, /orchestrate)
|
||||
|
||||
# --- External services (optional) ------------------------------------------
|
||||
|
||||
@@ -1,252 +1,130 @@
|
||||
# Agents, Sous-agents, Competences
|
||||
# AGENTS.md — KXKM_Clown Monorepo
|
||||
|
||||
> "L'infrastructure est une decision politique deployee." -- electron rare
|
||||
|
||||
## Orchestration
|
||||
Multimodal AI chat system. Turborepo monorepo (npm workspaces): 3 apps + 8 packages + 42 scripts. 15+ service mesh. Single RTX 4090 GPU: `MAX_GPU_CONCURRENT=1`.
|
||||
|
||||
- Agent racine: **Coordinateur** — planifie, arbitre, synchronise PLAN/TODO/docs
|
||||
- Sous-agents specialises: analyse code, veille OSS, audit securite, optimisation
|
||||
- Cadence: synchroniser PLAN.md + TODO.md + docs apres chaque lot
|
||||
## Key Files
|
||||
|
||||
## Matrice des agents (lot 17+)
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `docker-compose.yml` | 12 services (postgres, searxng, docling, qdrant, ollama, tts, lightrag) with health checks |
|
||||
| `turbo.json` | Build tasks, caching, workspace graph |
|
||||
| `package.json` | Root: 15 npm scripts (dev, build, check, test, smoke, verify) |
|
||||
| `.env.example` | LLM_URL, DATABASE_URL, ports, TTS, RAG model, etc. |
|
||||
|
||||
| Agent | Competences | Perimetre | Etat |
|
||||
## Subdirectories
|
||||
|
||||
| Dir | Purpose | Ref |
|
||||
|-----|---------|-----|
|
||||
| `apps/` | 3 apps: api (77 TS), web (64 TS/TSX), worker (4 TS) | `apps/AGENTS.md` |
|
||||
| `packages/` | 8 packages (core, auth, chat-domain, persona-domain, node-engine, storage, tui, ui) | `packages/AGENTS.md` |
|
||||
| `scripts/` | 42 files: 20 Python, 22 Shell (ops, training, ingestion, voice, image, deploy) | `scripts/AGENTS.md` |
|
||||
| `docs/` | 50+ specs, spikes, research, audits (SPEC_*.md, AUDIT_*.md, OSS_VEILLE_*.md) | — |
|
||||
| `ops/` | Monitoring + ops/v2/ (systemd, TUI, health-check.sh, deep-audit.js) | — |
|
||||
| `models/` | Fine-tuned + LoRA weights (base_models/, finetuned/, lora/, registry.json) | — |
|
||||
| `data/` | Ephemeral: persona memory, chat logs, context, corpus (v2-local/, chat-logs/) | — |
|
||||
|
||||
## Agent Matrix
|
||||
|
||||
| Agent | Competences | Scope | Status |
|
||||
|---|---|---|---|
|
||||
| Coordinateur | planification, arbitrage, docs de pilotage | PLAN.md, TODO.md, AGENTS.md, README.md | actif |
|
||||
| Securite | validation input, hardening, rate-limit, RBAC | apps/api, ws-chat, packages/auth | veille |
|
||||
| Backend API | Express, WS, Ollama, RAG, multimodal pipeline | apps/api/src/ | actif |
|
||||
| Coordinateur | Planning, arbitration, docs sync | PLAN.md, TODO.md, AGENTS.md, README.md | actif |
|
||||
| Securite | Input validation, hardening, rate-limit, RBAC | apps/api, ws-chat, packages/auth | veille |
|
||||
| Backend API | Express, WS, Ollama, RAG, multimodal pipeline | apps/api/src/ (77 TS + 27 tests) | actif |
|
||||
| Node Engine | DAG, queue, runs, sandbox, training adapters | packages/node-engine, apps/worker | actif |
|
||||
| Personas | source/feedback/proposals/pharmacius, memoire | packages/persona-domain, ws-chat | actif |
|
||||
| Frontend | React/Vite, UX Minitel, React Flow, chat, voice | apps/web/src/ | actif |
|
||||
| Ops/TUI | scripts, logs, rotate/purge, health, audit | ops/v2/, scripts/ | actif |
|
||||
| Training | DPO, SFT, Unsloth, eval, autoresearch, Ollama import | scripts/, packages/node-engine | actif |
|
||||
| Multimodal | STT, TTS, vision, PDF, RAG, recherche web | apps/api/src/ws-chat.ts | actif |
|
||||
| Veille OSS | recherche projets, libs, modeles, benchmarks | docs/OSS_WATCH, docs/HF_MODEL_RESEARCH | periodique |
|
||||
| Personas | Memory, DPO, pharmacius, coherence | packages/persona-domain, ws-chat (33 personas) | actif |
|
||||
| Frontend | React/Vite, Minitel theme, React Flow, chat, voice | apps/web/src/ (64 TS/TSX + 10 tests) | actif |
|
||||
| Ops/TUI | Monitoring, deploy, logs, health, audit | ops/v2/, scripts/, deep-audit.js | actif |
|
||||
| Training | DPO, SFT, Unsloth, eval, autoresearch | scripts/, packages/node-engine | actif |
|
||||
| Multimodal | STT, TTS, vision, PDF, RAG, web search | apps/api/src/ws-multimodal.ts | actif |
|
||||
| Veille OSS | Benchmarks, new libs, licensing, interop | docs/OSS_WATCH, docs/HF_MODEL_RESEARCH | periodique |
|
||||
|
||||
## Sous-agents et skill routing
|
||||
## Message Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Coord[Coordinateur]
|
||||
|
||||
Coord --> SecAgent[Securite]
|
||||
Coord --> BackAgent[Backend API]
|
||||
Coord --> EngAgent[Node Engine]
|
||||
Coord --> PersAgent[Personas]
|
||||
Coord --> FrontAgent[Frontend]
|
||||
Coord --> OpsAgent[Ops/TUI]
|
||||
Coord --> TrainAgent[Training]
|
||||
Coord --> MultiAgent[Multimodal]
|
||||
Coord --> OSSAgent[Veille OSS]
|
||||
|
||||
SecAgent --> |audit| SecScan[Pattern scan P0/P1/P2]
|
||||
SecAgent --> |fix| SecFix[Correctifs chirurgicaux]
|
||||
|
||||
BackAgent --> |analyse| APIAudit[Deep analyse app.ts, ws-chat.ts]
|
||||
BackAgent --> |refactor| APISplit[Extraction modules]
|
||||
|
||||
EngAgent --> |test| EngTest[Tests unitaires node-engine]
|
||||
EngAgent --> |extend| EngNew[Nouveaux node types]
|
||||
|
||||
PersAgent --> |pipeline| PersPipe[Editorial pipeline]
|
||||
PersAgent --> |finetune| PersDPO[DPO + PCL methodology]
|
||||
|
||||
FrontAgent --> |ui| FrontUI[Minitel theme CSS]
|
||||
FrontAgent --> |perf| FrontPerf[Memoization, lazy load]
|
||||
|
||||
OpsAgent --> |monitor| OpsHealth[health-check, deep-audit]
|
||||
OpsAgent --> |deploy| OpsDeploy[Docker, kxkm-ai]
|
||||
|
||||
TrainAgent --> |train| TrainRun[Unsloth/TRL runs]
|
||||
TrainAgent --> |eval| TrainEval[Scoring, registry]
|
||||
|
||||
MultiAgent --> |voice| MultiVoice[XTTS-v2, WebRTC]
|
||||
MultiAgent --> |search| MultiSearch[SearXNG]
|
||||
|
||||
OSSAgent --> |web| OSSWeb[Recherche web]
|
||||
OSSAgent --> |hf| OSSHF[HuggingFace models]
|
||||
```
|
||||
User WS → ws-chat.ts (rate-limit, multimodal dispatch)
|
||||
→ ws-conversation-router.ts (persona routing, context assembly)
|
||||
→ ws-persona-router.ts (memory extract/load, responder select)
|
||||
→ inference-scheduler.ts (single-GPU queue, MAX_GPU_CONCURRENT=1)
|
||||
→ ws-ollama.ts (token stream, tool-calling)
|
||||
→ ws-multimodal.ts (TTS, vision, STT, file upload)
|
||||
→ persona-memory-store.ts (nick-isolated file persist)
|
||||
→ rag.ts (embedding + LightRAG dual-write)
|
||||
→ context-store.ts (channel history + compaction)
|
||||
```
|
||||
|
||||
## Todo agents (lot 17+ — mis a jour 2026-03-24)
|
||||
## Services
|
||||
|
||||
### Coordinateur
|
||||
| Service | Port | Notes |
|
||||
|---------|------|-------|
|
||||
| API (HTTP+WS) | 4180 | Node.js Express + ws |
|
||||
| Frontend (Vite) | 5173 | React + 5 CSS themes |
|
||||
| Ollama/vLLM | 11434 | LLM runtime + embeddings |
|
||||
| PostgreSQL | 5432 | Chat, sessions, node-engine runs |
|
||||
| SearXNG | 8080 | Self-hosted search (DuckDuckGo fallback) |
|
||||
| Docling | 9400 | PDF extraction |
|
||||
| LightRAG | 9621 | Graph-RAG, `LLM_MODEL=mistral:7b` to avoid `<think>` corruption |
|
||||
| TTS (Piper/Chatterbox) | 9100 | Voice synthesis |
|
||||
| Kokoro TTS | 9201 | Fast TTS, 12 voices |
|
||||
| ComfyUI | 8188 | Image generation (32 checkpoints + 24 LoRAs) |
|
||||
| Camoufox | 8091 | Stealth browser for bot-protected sites |
|
||||
|
||||
- [x] Consolider PLAN.md avec etat reel (lots 0-94 complets)
|
||||
- [x] Synchroniser FEATURE_MAP.md matrice
|
||||
- [x] Mettre a jour TODO.md avec backlog Phase session 2026-03-19/20
|
||||
- [x] Documenter actions dans ops/v2/logs/
|
||||
- [x] lot-95: Coordonner E2E Playwright test plan
|
||||
- [x] lot-100: Design public demo mode access control
|
||||
## GPU Constraint
|
||||
|
||||
### Backend API
|
||||
All LLM calls → `inference-scheduler.ts`. No direct fetch() outside approved helpers. RTX 4090: `MAX_GPU_CONCURRENT=1`. Context compaction + persona extraction both via `scheduler.submit(priority: "low")`.
|
||||
|
||||
- [x] Extraire app-bootstrap.ts et app-middleware.ts de app.ts
|
||||
- [x] Extraire ws-conversation-router.ts de ws-chat.ts
|
||||
- [x] ws-chat.ts modularized (425 to 335 LOC, 3 modules extracted)
|
||||
- [x] app.ts extraction (540 to 131 LOC, create-repos.ts extracted)
|
||||
- [x] Zod validation on all 19 API route schemas
|
||||
- [x] Error telemetry (16 labels)
|
||||
- [x] Perf instrumentation (6 labels, p50/p95/p99), TTFC 284ms
|
||||
- [x] Smart routing (5 topic domains)
|
||||
- [x] Dynamic context window (4k-32k)
|
||||
- [x] NLP auto-detect generation intent (compose vs imagine)
|
||||
- [x] /speed command for latency diagnostics
|
||||
- [ ] lot-178: ACE-Step API direct integration (duration fix)
|
||||
- [ ] lot-180: Timeline data model
|
||||
- [x] lot-97: Multi-channel support (create/join channels)
|
||||
- [x] lot-100: Public demo mode read-only routes
|
||||
## Persona Memory (nick-isolated, 2026-04)
|
||||
|
||||
### Node Engine
|
||||
- **Path**: `data/v2-local/persona-memory/{personaId}/{nick}.json`
|
||||
- **Modes**: `auto` (Pharmacius, Sherlock, Turing, Ikeda), `explicit` (artistic personas, `/remember` only)
|
||||
- **Injection cap**: 8 facts max into system prompt
|
||||
- **Anonymous relay**: `_anonymous` sentinel for unknown-nick chains
|
||||
|
||||
- [x] Extraire registry.ts du hotspot node-engine
|
||||
- [ ] Ajouter node type `music_generation` (ACE-Step 1.5)
|
||||
- [ ] Ajouter node type `voice_clone` (Chatterbox)
|
||||
- [ ] Ajouter node type `audio_mix` (multi-track composition)
|
||||
- [ ] Ajouter node type `audio_effects` (FX chain)
|
||||
- [ ] lot-96: Automated DPO pipeline (feedback → pairs → training trigger)
|
||||
## Build & Dev
|
||||
|
||||
### Multimodal (composition pipeline)
|
||||
|
||||
- [x] 35 music styles ACE-Step
|
||||
- [x] ComfyUI smart checkpoint selection (32 checkpoints + 24 LoRAs)
|
||||
- [ ] lot-178: ACE-Step API direct (duration fix)
|
||||
- [ ] lot-181: TTS voiceover mix into timeline
|
||||
- [ ] lot-182: Audio effects pipeline (reverb, delay, EQ, compression)
|
||||
- [ ] lot-183: DAW export (stems, markers, project file)
|
||||
- [x] lot-184: Multi-track composition (/layer, composition-store)
|
||||
- [x] lot-185: Composition UI (track lanes, play/pause/seek)
|
||||
- [x] lot-186: Arrangement tools (/comp structure, section markers)
|
||||
- [x] lot-187: Auto-mastering (/mix master, loudness normalization, limiter)
|
||||
- [x] lot-188: /voice TTS voiceover injected into composition
|
||||
- [x] lot-189: /noise 5 types (white, pink, brown, rain, wind)
|
||||
- [x] lot-190: /fx 9 audio effects (reverb, delay, chorus, flanger, distortion, bitcrusher, EQ, compressor, tremolo)
|
||||
- [x] lot-191: /ambient scene generator (forest, ocean, city, space, cave)
|
||||
- [ ] lot-194: Waveform visualization (wavesurfer.js)
|
||||
- [ ] lot-195: /remix re-generate specific track
|
||||
- [ ] lot-199: Stem separation (Demucs v4 htdemucs, 6-stem, MIT)
|
||||
- [x] lot-200: Full DAW export (WAV stems + JSON project)
|
||||
|
||||
### Personas
|
||||
|
||||
- [ ] Evaluer PCL (Persona-Aware Contrastive Learning) pour coherence
|
||||
- [ ] Evaluer OpenCharacter pour generation profils synthetiques
|
||||
- [x] Ajouter `/compose` command (generation musicale)
|
||||
|
||||
### Frontend
|
||||
|
||||
- [x] Implementer lot 16 UI Minitel rose (phosphore, VIDEOTEX)
|
||||
- [x] VoiceChat push-to-talk + level meter + silence auto
|
||||
- [x] Player audio + viewer image plein ecran
|
||||
- [x] Mediatheque gallery/playlist
|
||||
- [x] Progress bars animees Compose/Imagine
|
||||
- [x] React.memo + useCallback on ChatSidebar, ChatInput, ChatHistory
|
||||
- [x] 17 lazy-loaded routes (-53% initial JS)
|
||||
- [x] CRT CSS-only effect (scanlines, vignette, phosphor glow, boot 0.8s)
|
||||
- [x] Chat virtualization (react-window, variable row heights)
|
||||
- [x] Markdown rendering (marked + DOMPurify)
|
||||
- [x] CRT boot animation (modem dial, scanline reveal)
|
||||
- [x] 5 CSS themes (minitel, crt, hacker, synthwave, default)
|
||||
- [x] Mobile responsive pass (touch, bottom nav, viewport units)
|
||||
- [x] Guest mode read-only UI
|
||||
- [x] lot-185: Composition timeline UI (waveform view, track lanes)
|
||||
- [ ] lot-194: Waveform visualization (wavesurfer.js)
|
||||
- [x] lot-95: E2E Playwright tests (login, chat, upload, admin)
|
||||
- [x] lot-98: File sharing UI (upload → gallery)
|
||||
|
||||
### Ops/TUI
|
||||
|
||||
- [x] Deployer deep-audit.js sur kxkm-ai
|
||||
- [x] Ajouter SearXNG au docker-compose
|
||||
- [x] TTS sidecar HTTP (tts-server.py :9100, dual Chatterbox/Piper)
|
||||
- [x] deploy.sh migrated tmux → systemd
|
||||
- [x] Systemd services (kxkm-tts + kxkm-lightrag, auto-restart)
|
||||
- [x] health-check.sh TUI (19 checks)
|
||||
- [x] Docker compose 12 services with health checks
|
||||
- [ ] Fix Docker transformers (rebuild propre avec torch)
|
||||
|
||||
### Training
|
||||
|
||||
- [x] Spike BGE-M3 (resultat negatif sur Apple/Metal, baseline maintenue)
|
||||
- [x] TTS dual backend Chatterbox/Piper valide
|
||||
- [x] Tool-calling benchmark (llama3.1 vs qwen3 vs mistral)
|
||||
- [x] Sherlock migrated to llama3.1:8b-instruct-q4_0
|
||||
- [ ] lot-96: Persona DPO automation pipeline
|
||||
- [ ] Tester ACE-Step 1.5 sur RTX 4090
|
||||
|
||||
### Veille OSS
|
||||
|
||||
- [x] Veille mars 2026 complete (40+ projets analyses, top 10 recommandations)
|
||||
- [ ] Suivre LLMRTC (WebRTC voice TypeScript)
|
||||
- [ ] Suivre A2A Protocol (interop agents)
|
||||
- [ ] Suivre MCP SDK updates
|
||||
- [ ] Evaluer Kokoro TTS (82M params, ultra-leger)
|
||||
|
||||
## Pipeline d'intervention
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Analyse: agent lance
|
||||
Analyse --> Findings: scan code/docs/web
|
||||
Findings --> Triage: P0/P1/P2 classification
|
||||
Triage --> Fix_P0: P0 critique
|
||||
Triage --> Plan_P1: P1 important
|
||||
Triage --> Backlog_P2: P2 mineur
|
||||
Fix_P0 --> Test: correction chirurgicale
|
||||
Plan_P1 --> Test: correction planifiee
|
||||
Test --> Deploy: tests OK
|
||||
Deploy --> Log: log + purge
|
||||
Log --> [*]: cycle termine
|
||||
Backlog_P2 --> [*]: ajoute au TODO
|
||||
```bash
|
||||
npm install # Install all workspaces
|
||||
npm run dev # Turbo parallel (api, web, worker)
|
||||
npm run dev:v2:api # API :4180 (tsx watch)
|
||||
npm run dev:v2:web # Web :5173 (Vite)
|
||||
npm run check:v2 # tsc --noEmit
|
||||
npm run -w @kxkm/api test # 278 unit tests
|
||||
npm run -w @kxkm/web test # 54 unit tests
|
||||
npm run smoke:v2 # Integration smoke
|
||||
npm run verify # check + smoke (full gate)
|
||||
docker compose --profile v2 up -d
|
||||
```
|
||||
|
||||
## Affectations en cours (2026-03-20)
|
||||
## Environment
|
||||
|
||||
### Mission globale
|
||||
- Deep analyse continue du code, optimisation chirurgicale, et synchronisation documentaire apres chaque lot.
|
||||
- Priorite execution: P1 fiabilite, puis dette perf/complexite, puis features lot 18-19.
|
||||
```bash
|
||||
LLM_URL=http://localhost:11434
|
||||
LLM_MODEL=qwen-14b-awq
|
||||
DATABASE_URL=postgres://kxkm:kxkm@localhost:5432/kxkm_clown
|
||||
V2_API_PORT=4180
|
||||
TTS_ENABLED=1
|
||||
VISION_MODEL=qwen3-vl:8b
|
||||
RAG_EMBEDDING_MODEL=nomic-embed-text
|
||||
SEARXNG_URL=http://localhost:8080
|
||||
KXKM_PERSONA_MEMORY_INJECTION_LIMIT=8
|
||||
```
|
||||
|
||||
### Assignations agents -> sous-agents -> competences
|
||||
## Cycle State (2026-03-20)
|
||||
|
||||
| Agent | Sous-agent | Competences principales | Taches assignees immediates |
|
||||
|---|---|---|---|
|
||||
| Coordinateur | Planner/Docs | triage, synchronisation, runbook | Maintenir PLAN/TODO, chainer les lots, tracer actions |
|
||||
| Backend API | WS/HTTP surgeon | websocket, express, validation input | Extraire `ws-chat.ts` en modules, reduire logs, limiter hot paths |
|
||||
| Node Engine | DAG runtime | graph validation, queue, state machine | Ajouter nodes `music_generation`, `voice_clone`, `document_extraction` |
|
||||
| Ops/TUI | Audit operator | TUI scripts, logs, rotation, cron | Rendre `deep-audit` zero faux positif critique, pipeline logs + purge |
|
||||
| Veille OSS | Scout | benchmark OSS, licences, interop | Evaluer Open WebUI, LibreChat, LangGraph, SearXNG, Docling |
|
||||
- 130+ lots complete (lot-24 to lot-177)
|
||||
- 425 tests, 0 failures
|
||||
- 43 chat commands, 33 personas
|
||||
- 5 CSS themes, 35 music styles (ACE-Step)
|
||||
- 32 ComfyUI checkpoints + 24 LoRAs
|
||||
- TTFC 284ms, guest mode, mobile responsive
|
||||
- Structured logging (pino JSON)
|
||||
- Systemd services (TTS, LightRAG)
|
||||
|
||||
### Workflow d'enchainement
|
||||
1. Executer audit + tests
|
||||
2. Corriger de maniere chirurgicale
|
||||
3. Re-executer audit + tests
|
||||
4. Mettre a jour docs de pilotage
|
||||
5. Alimenter TODO suivant avec ordre d'execution
|
||||
## See Also
|
||||
|
||||
### Regles d'operation
|
||||
- Interroger le user uniquement en cas de blocage reel (acces, choix irreversibles, secrets).
|
||||
- Privilegier TUI et scripts avec logs lisibles, puis purge des logs obsoletes.
|
||||
- Conserver la V1 comme reference comportementale, V2 comme cible active.
|
||||
|
||||
### Etat de cycle (2026-03-20 18:00)
|
||||
|
||||
- 130+ lots termines (lot-24 a lot-177).
|
||||
- 425 tests, 0 failures.
|
||||
- 13 services en production.
|
||||
- 43 chat commands, 33 personas.
|
||||
- 35 music styles (ACE-Step), 5 CSS themes.
|
||||
- 32 ComfyUI checkpoints + 24 LoRAs, smart selection NLP.
|
||||
- TTFC 284ms.
|
||||
- Guest mode, mobile responsive.
|
||||
- Structured logging complet (pino JSON, 0 console.log).
|
||||
- Systemd services (TTS + LightRAG).
|
||||
- Frontend: lazy routes (-53%), React.memo, CRT boot, chat virtualization.
|
||||
|
||||
### Prochains lots (178-200) — Composition Pipeline
|
||||
|
||||
1. lot-178: Compose duration fix (ACE-Step API direct)
|
||||
2. lot-179: SPEC_COMPOSE_ADVANCED plan
|
||||
3. lot-180: Timeline data model (tracks, clips, markers)
|
||||
4. lot-181: TTS voiceover mix into timeline
|
||||
5. lot-182: Audio effects pipeline (reverb, delay, EQ)
|
||||
6. lot-183: DAW export (stems, markers, project)
|
||||
7. lot-184-200: Multi-track, arrangement, mastering, stem separation, MIDI, templates, collab, lyrics, FX rack, automation, samples, spectral view, history, render queue, sharing
|
||||
- `apps/AGENTS.md` — api, web, worker details
|
||||
- `packages/AGENTS.md` — shared package breakdown
|
||||
- `scripts/AGENTS.md` — deployment, training, ops scripts
|
||||
- `CLAUDE.md` — Architecture, request paths, data directories
|
||||
- `PLAN.md` / `TODO.md` / `FEATURE_MAP.md` — Roadmap (lots 178-200)
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# Dev (V2)
|
||||
npm run dev:v2:api # API on :4180 (tsx watch)
|
||||
npm run dev:v2:web # Frontend on :5173 (Vite)
|
||||
npm run dev:v2:worker # Background job processor
|
||||
|
||||
# Build
|
||||
npm run build:v2 # TypeScript compile (all workspaces)
|
||||
npm run turbo:build # Full monorepo build
|
||||
|
||||
# Type-check
|
||||
npm run check:v2 # tsc --noEmit (fast check)
|
||||
npm run -w @kxkm/api check
|
||||
|
||||
# Tests
|
||||
npm run -w @kxkm/api test # 278 unit tests
|
||||
npm run -w @kxkm/web test # 54 unit tests
|
||||
npm run smoke:v2 # Integration smoke
|
||||
npm run verify # check + smoke (full gate)
|
||||
|
||||
# Run a single test file (after build)
|
||||
cd apps/api && node --experimental-test-module-mocks --test dist/persona-memory-store.test.js
|
||||
|
||||
# Docker
|
||||
docker compose --profile v2 up -d
|
||||
docker compose --profile v2 --profile ollama up -d # with bundled embeddings backend
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
**Monorepo** (npm workspaces + Turborepo): `apps/` + `packages/`
|
||||
|
||||
### API (`apps/api/src/`)
|
||||
|
||||
The V2 API is a single Node.js process combining Express (HTTP) and `ws` (WebSocket) on port 4180.
|
||||
|
||||
**Request path for a chat message:**
|
||||
```
|
||||
ws-chat.ts (WS handler, rate-limit, routing)
|
||||
→ ws-conversation-router.ts (pick responder personas, build context)
|
||||
→ ws-ollama.ts (stream tokens from vLLM/TurboQuant runtime, tool-calling)
|
||||
→ ws-multimodal.ts (TTS streaming, vision, STT, file upload)
|
||||
→ ws-persona-router.ts (memory load/update, responder selection)
|
||||
```
|
||||
|
||||
**Key files:**
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `server.ts` | HTTP+WS bootstrap, DAW sample routes, corpus boot |
|
||||
| `ws-chat.ts` | WS entry — broadcast, rate-limit, multimodal dispatch |
|
||||
| `ws-conversation-router.ts` | Persona routing, context assembly, TTS chunking, inter-persona depth-3 relay |
|
||||
| `ws-ollama.ts` | Runtime streaming, tool-calling, `<think>` tag stripping |
|
||||
| `ws-persona-router.ts` | Memory extract/save, responder selection, InferenceScheduler |
|
||||
| `personas-default.ts` | 33 persona definitions (`memoryMode`, `corpus[]`, `relations[]`) |
|
||||
| `persona-memory-store.ts` | Per-nick isolated storage at `data/v2-local/persona-memory/{personaId}/{nick}.json` |
|
||||
| `persona-memory-policy.ts` | auto/explicit/off modes, `injectionFactsLimit` (default 8) |
|
||||
| `rag.ts` | Local embedding store, per-persona namespaces, LightRAG dual-write |
|
||||
| `context-store.ts` | Per-channel conversation memory, LLM compaction via InferenceScheduler |
|
||||
| `inference-scheduler.ts` | Single-GPU queue (`MAX_GPU_CONCURRENT=1`), all LLM calls must go through it |
|
||||
| `chat-types.ts` | All shared types: `ChatPersona`, `PersonaMemoryMode`, `ClientInfo`, message union |
|
||||
| `mcp-tools.ts` | Tool definitions injected per persona (web_search, rag_search, etc.) |
|
||||
| `web-search.ts` | SearXNG → DuckDuckGo fallback; discovered URLs enqueued to `data/sherlock-discovered-urls.jsonl` |
|
||||
|
||||
### Persona Memory (nick-isolated, 2026-04 design)
|
||||
|
||||
Memory is keyed by `(personaId, nick)` — one file per user per persona.
|
||||
|
||||
- `memoryMode: 'auto'` → Pharmacius, Sherlock, Turing, Ikeda (LLM extraction every 5 responses)
|
||||
- `memoryMode: 'explicit'` → all artistic personas (Schaeffer, Merzbow, Pina, etc.) — only via `/remember`
|
||||
- `/remember [@persona|@all] <text>` — direct fact insert, no LLM call
|
||||
- Injection cap: 8 facts max into system prompt (`injectionFactsLimit`)
|
||||
- `_anonymous` sentinel for unknown-nick relay chains
|
||||
|
||||
### Packages
|
||||
|
||||
| Package | Contents |
|
||||
|---------|---------|
|
||||
| `core` | Shared types, IDs, permissions |
|
||||
| `auth` | RBAC, sessions, crypto |
|
||||
| `chat-domain` | Message types, channels, slash command registry |
|
||||
| `persona-domain` | Persona model, DPO pairs, feedback pipeline |
|
||||
| `node-engine` | DAG execution, GPU job queue, 15+ node types |
|
||||
| `storage` | Postgres repositories, migrations |
|
||||
|
||||
### Services & Ports
|
||||
|
||||
| Service | Port | Notes |
|
||||
|---------|------|-------|
|
||||
| API V2 | 4180 | HTTP + WS |
|
||||
| Frontend | 5173 | Vite dev |
|
||||
| vLLM | 8000 | Primary OpenAI-compatible text runtime (qwen-32b-awq) |
|
||||
| TEI | 9500 | Dedicated embedding server (BAAI/bge-m3, CPU) |
|
||||
| PostgreSQL | 5432 | V2 persistence (optional for API, required for worker) |
|
||||
| LightRAG | 9621 | Graph-RAG; `LLM_MODEL=mistral:7b` to avoid `<think>` JSON corruption |
|
||||
| SearXNG | 8080 | Self-hosted search |
|
||||
| TTS | 9100 | Piper + Chatterbox |
|
||||
| Kokoro TTS | 9201 | Fast TTS, 12 voices |
|
||||
| AI Bridge | 8301 | 19 audio backends |
|
||||
| ComfyUI | 8188 | Image generation |
|
||||
| Docling | 9400 | PDF extraction |
|
||||
| Camoufox | 8091 | Stealth browser fetch (venv, `kxkm-camoufox.service`) |
|
||||
|
||||
### InferenceScheduler constraint
|
||||
|
||||
**All LLM calls must go through `inference-scheduler.ts`** — no direct `fetch()` to the runtime outside approved helpers. The RTX 4090 has `MAX_GPU_CONCURRENT=1`. `context-store.ts` compaction and `ws-persona-router.ts` extraction both use `scheduler.submit()` with `priority: "low"`.
|
||||
|
||||
### Corpus ingestion pipeline
|
||||
|
||||
`scripts/ingest_spectacle_corpus.py` ingests domain content into LightRAG:
|
||||
- Seed URLs + SearXNG discovery + `data/sherlock-discovered-urls.jsonl` (live Sherlock search discoveries)
|
||||
- Camoufox server (`:8091`) used for bot-protected sites (`artcena.fr`, `culture.gouv.fr`, etc.)
|
||||
|
||||
### Data directories
|
||||
|
||||
```
|
||||
data/
|
||||
v2-local/persona-memory/{personaId}/{nick}.json # Per-user persona memory
|
||||
context/{channel}.jsonl # Conversation history
|
||||
sherlock-discovered-urls.jsonl # Corpus queue from web searches
|
||||
chat-logs/ # Daily JSONL chat logs
|
||||
manifeste.md # Project philosophy (injected at boot)
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
LLM_URL=http://localhost:8000 # vLLM OpenAI-compatible endpoint
|
||||
LLM_MODEL=qwen-32b-awq
|
||||
LLM_API_KEY=vllm-er-2026 # Bearer token for vLLM --api-key
|
||||
OLLAMA_URL=http://localhost:9500 # TEI embedding server
|
||||
EMBEDDING_BACKEND=tei # "tei" or "ollama"
|
||||
RAG_EMBEDDING_MODEL=BAAI/bge-m3
|
||||
DATABASE_URL=postgres://kxkm:kxkm@localhost:5432/kxkm_clown
|
||||
V2_API_PORT=4180
|
||||
TTS_ENABLED=1
|
||||
VISION_MODEL=qwen3-vl:8b
|
||||
SEARXNG_URL=http://localhost:8080
|
||||
KXKM_PERSONA_MEMORY_INJECTION_LIMIT=8 # Max facts injected into system prompt
|
||||
CHAT_PAUSED=1 # Or create data/chat-paused file
|
||||
```
|
||||
+244
@@ -0,0 +1,244 @@
|
||||
# AGENTS.md — apps/
|
||||
|
||||
<!-- Parent: ../AGENTS.md -->
|
||||
|
||||
Three applications in Turborepo workspace: api (backend), web (frontend), worker (background jobs).
|
||||
|
||||
## api/ — Backend (77 TS files + 27 tests)
|
||||
|
||||
Node.js Express + WebSocket on port 4180. Single process: HTTP + WS.
|
||||
|
||||
### WebSocket Handlers
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `ws-chat.ts` | Entry point: rate-limit, broadcast, multimodal dispatch |
|
||||
| `ws-conversation-router.ts` | Persona routing, context assembly, TTS chunking, inter-persona depth-3 relay |
|
||||
| `ws-ollama.ts` | Runtime streaming, tool-calling, `<think>` tag stripping |
|
||||
| `ws-persona-router.ts` | Memory extract/save, responder selection, InferenceScheduler submit |
|
||||
| `ws-multimodal.ts` | TTS streaming, vision, STT, file upload, ComfyUI integration |
|
||||
| `ws-upload-handler.ts` | Media ingestion, storage, gallery |
|
||||
| `ws-chat-helpers.ts` | Utilities: formatting, logging, metrics |
|
||||
| `ws-chat-logger.ts` | Structured logging (pino JSON) |
|
||||
| `ws-chat-history.ts` | Per-channel conversation memory, compaction |
|
||||
| `ws-chat-state.test.ts` | State machine tests |
|
||||
|
||||
### Commands (5 handlers)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `ws-commands.ts` | Router: dispatch to handlers by `/command` |
|
||||
| `ws-commands-chat.ts` | `/chat`, `/speed`, `/help` |
|
||||
| `ws-commands-info.ts` | `/personas`, `/commands`, `/status` |
|
||||
| `ws-commands-generate.ts` | `/imagine`, `/compose`, `/music` (ACE-Step, ComfyUI) |
|
||||
| `ws-commands-compose.ts` | DAW composition: `/layer`, `/fx`, `/voice`, `/mix`, `/export` |
|
||||
|
||||
### Personas (33 definitions + memory)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `personas-default.ts` | 33 personas: Pharmacius, Sherlock, Turing, Ikeda, Schaeffer, Merzbow, Pina, etc. (memoryMode, corpus[], relations[]) |
|
||||
| `persona-runtime.ts` | Runtime: load, extract, save; DPO feedback pipeline |
|
||||
| `persona-memory-store.ts` | Per-nick isolated storage: `data/v2-local/persona-memory/{personaId}/{nick}.json` |
|
||||
| `persona-memory-policy.ts` | auto/explicit/off modes, injectionFactsLimit (default 8) |
|
||||
| `persona-voices.ts` | Voice config per persona |
|
||||
| `persona-memory-telemetry.ts` | Metrics: extraction time, injection count, memory size |
|
||||
|
||||
### RAG & Inference
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `rag.ts` | Local embedding store, per-persona namespaces, LightRAG dual-write |
|
||||
| `inference-scheduler.ts` | Single-GPU queue: `MAX_GPU_CONCURRENT=1`, all LLM calls must go through it |
|
||||
| `llm-client.ts` | Ollama API wrapper, streaming, tool-calling |
|
||||
| `deep-research.ts` | Multi-turn research loop via scheduler |
|
||||
|
||||
### Storage & Context
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `context-store.ts` | Per-channel conversation memory, LLM compaction via scheduler |
|
||||
| `chat-types.ts` | All shared types: ChatPersona, PersonaMemoryMode, ClientInfo, message union |
|
||||
| `composition-store.ts` | Multi-track composition state (tracks, clips, markers) |
|
||||
| `media-store.ts` | File storage, gallery, cleanup |
|
||||
|
||||
### Services
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `web-search.ts` | SearXNG → DuckDuckGo fallback; discovered URLs → `data/sherlock-discovered-urls.jsonl` |
|
||||
| `comfyui.ts` | Image generation: checkpoint + LoRA selection, prompt injection |
|
||||
| `comfyui-models.ts` | 32 checkpoints + 24 LoRAs registry |
|
||||
| `voice-samples.ts` | Voice clone sample management |
|
||||
| `mcp-tools.ts` | Tool definitions injected per persona (web_search, rag_search, compose, imagine, etc.) |
|
||||
|
||||
### Routes (6 REST files + bootstrap)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `routes/chat-history.ts` | GET/POST chat logs, export |
|
||||
| `routes/media.ts` | GET/POST media, gallery |
|
||||
| `routes/personas.ts` | GET personas, memory, feedback |
|
||||
| `routes/session.ts` | Auth, session mgmt, guest mode |
|
||||
| `routes/node-engine.ts` | DAG runs, node types, training |
|
||||
| `media-shared-routes.ts` | Shared media endpoints |
|
||||
| `app.ts` | Express setup, middleware, WS upgrade |
|
||||
| `app-bootstrap.ts` | Corpus load, DAW samples, systemd startup |
|
||||
| `app-middleware.ts` | Auth, CORS, error handlers |
|
||||
|
||||
### Server & Infrastructure
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `server.ts` | HTTP + WS bootstrap, DAW sample routes, corpus boot |
|
||||
| `create-repos.ts` | DB repo initialization |
|
||||
| `schemas.ts` | Zod validation: 19 route schemas + command input schemas |
|
||||
| `error-tracker.ts` | Error telemetry (16 labels) |
|
||||
| `perf.ts` | Perf instrumentation (6 labels, p50/p95/p99) |
|
||||
| `logger.ts` | Pino JSON logger config |
|
||||
|
||||
### Tests (27 files)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `app.test.ts` | HTTP routes, middleware, health checks |
|
||||
| `ws-chat.test.ts` | WS broadcast, rate-limit, multimodal dispatch |
|
||||
| `ws-conversation-router.test.ts` | Persona routing, context assembly |
|
||||
| `ws-ollama.test.ts` | Token streaming, tool-calling, think tag |
|
||||
| `ws-multimodal.test.ts` | TTS, vision, STT, file upload |
|
||||
| `ws-upload-handler.test.ts` | Media ingestion |
|
||||
| `ws-commands.test.ts` | Command routing |
|
||||
| `persona-memory-store.test.ts` | Nick isolation, file persistence |
|
||||
| `persona-memory-policy.test.ts` | auto/explicit modes |
|
||||
| `persona-memory-telemetry.test.ts` | Metrics tracking |
|
||||
| `persona-runtime.test.ts` | Runtime load/save |
|
||||
| `rag.test.ts` | Embedding, LightRAG sync |
|
||||
| `context-store.test.ts` | Channel history, compaction |
|
||||
| `composition-store.test.ts` | Multi-track state |
|
||||
| `media-store.test.ts` | File storage |
|
||||
| `web-search.test.ts` | SearXNG, DuckDuckGo fallback |
|
||||
| `voice-samples.test.ts` | Voice sample mgmt |
|
||||
| `mcp-tools.test.ts` | Tool definitions |
|
||||
| `mcp-server.test.ts` | MCP server integration |
|
||||
| `chat-history-routes.test.ts` | Chat log routes |
|
||||
| `media-shared-routes.test.ts` | Media endpoints |
|
||||
| `ws-chat-smoke.test.ts` | End-to-end smoke |
|
||||
| `ws-integration.test.ts` | Full integration flow |
|
||||
| `ws-chat-state.test.ts` | State machine |
|
||||
| `create-repos.test.ts` | DB setup |
|
||||
| `app.test.ts` | Bootstrap |
|
||||
| `integration.test.ts` | Full system |
|
||||
|
||||
### Run
|
||||
|
||||
```bash
|
||||
npm run dev:v2:api # tsx watch (localhost:4180)
|
||||
npm run -w @kxkm/api test # 278 unit tests
|
||||
cd apps/api && npm test # From api dir
|
||||
```
|
||||
|
||||
## web/ — Frontend (64 TS/TSX files + 10 tests)
|
||||
|
||||
React + Vite on port 5173. 5 CSS themes (minitel, crt, hacker, synthwave, default). React.memo + useCallback optimized. 17 lazy-loaded routes (-53% initial JS). Chat virtualization (react-window). CRT boot animation.
|
||||
|
||||
### Pages (10)
|
||||
|
||||
| Component | Purpose |
|
||||
|-----------|---------|
|
||||
| `Chat.tsx` | Main chat interface (virtualized history, input, sidebar) |
|
||||
| `ImaginePage.tsx` | Image generation (ComfyUI) |
|
||||
| `ComposePage.tsx` | DAW composition (timeline, tracks, effects) |
|
||||
| `DawAIPanel.tsx` | AI composition sidebar |
|
||||
| `LiveFXPage.tsx` | Real-time audio effects |
|
||||
| `UllaPage.tsx` | Ulla (experimental) |
|
||||
| `NodeEngineOverview.tsx` | DAG editor, run status |
|
||||
| `AdminPage.tsx` | Admin dashboard, user management |
|
||||
| `TrainingDashboard.tsx` | Training runs, metrics |
|
||||
| `Collectif.tsx` | Multi-user collaboration |
|
||||
|
||||
### Components (15+)
|
||||
|
||||
| Component | Purpose |
|
||||
|-----------|---------|
|
||||
| `ChatMessage.tsx` | Message render, markdown, media player |
|
||||
| `ChatInput.tsx` | Text input, voice record, file upload, rate-limit indicator |
|
||||
| `ChatHistory.tsx` | Virtualized scroll (react-window) |
|
||||
| `ChatSidebar.tsx` | Channel list, persona selector, theme toggle |
|
||||
| `Header.tsx` | Title, menu, auth |
|
||||
| `Nav.tsx` | Route navigation |
|
||||
| `ErrorBoundary.tsx` | Error UI |
|
||||
| `VoiceChat.tsx` | Push-to-talk, level meter, silence auto-detect |
|
||||
| `MediaGallery.tsx` | Image/audio gallery, fullscreen player |
|
||||
| `MediaExplorer.tsx` | File browser, upload |
|
||||
| `TimelineView.tsx` | DAW track lanes, waveform, play/pause/seek |
|
||||
| `EngineNode.tsx` | DAG node visual |
|
||||
| `NodeEditor.tsx` | DAG editor (React Flow) |
|
||||
| `PersonaList.tsx` | Persona selector with memory stats |
|
||||
| `PersonaDetail.tsx` | Persona info, memory injection preview |
|
||||
|
||||
### Hooks (7)
|
||||
|
||||
| Hook | Purpose |
|
||||
|------|---------|
|
||||
| `useWebSocket.ts` | WS connection, auto-reconnect, message dispatch |
|
||||
| `useAppSession.ts` | Session state, auth, guest mode |
|
||||
| `useChatState.ts` | Chat history, selected persona, channel |
|
||||
| `useGenerationCommand.ts` | /imagine, /compose submission, progress tracking |
|
||||
| `useNodeEditor.ts` | DAG state (nodes, edges, zoom) |
|
||||
| `useHashRoute.ts` | Client-side routing via hash |
|
||||
| `useKeyboardShortcuts.ts` | Ctrl+K palette, theme toggle, etc. |
|
||||
| `useMinitelSounds.ts` | 8-bit Minitel UI sounds |
|
||||
|
||||
### Library
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `lib/websocket-url.ts` | WS URL construction (dev vs prod) |
|
||||
| `api.ts` | HTTP client for REST routes |
|
||||
| `chat-types.ts` | Frontend message/persona types |
|
||||
|
||||
### Tests (10 files)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `components/Chat*.test.tsx` | Component render, interaction |
|
||||
| `components/Header.test.tsx` | Header UI |
|
||||
| `components/Login.test.tsx` | Auth flow |
|
||||
| `components/ChannelList.test.tsx` | Channel selector |
|
||||
| `components/PersonaList.test.tsx` | Persona list |
|
||||
| `components/RunStatus.test.tsx` | Run status display |
|
||||
| `components/Nav.test.tsx` | Route nav |
|
||||
| `hooks/*.test.ts` | Hook logic |
|
||||
| `App.test.tsx` | App bootstrap |
|
||||
|
||||
### Styles
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `styles.css` | Base theme variables (colors, fonts, spacing) |
|
||||
| `styles 2.css` | Alternative theme (legacy) |
|
||||
|
||||
### Run
|
||||
|
||||
```bash
|
||||
npm run dev:v2:web # Vite (localhost:5173)
|
||||
npm run -w @kxkm/web test # 54 unit tests
|
||||
```
|
||||
|
||||
## worker/ — Background Jobs (4 TS files)
|
||||
|
||||
Node.js background processor. Handles async tasks (training, data ingestion, composition rendering, etc.).
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `index.ts` | Entry point, job queue setup |
|
||||
| `worker-runtime.ts` | Job executor: training jobs, DPO pipeline, node runs |
|
||||
| `logger.ts` | Structured logging |
|
||||
| `worker-runtime.test.ts` | Runtime tests |
|
||||
|
||||
### Run
|
||||
|
||||
```bash
|
||||
npm run dev:v2:worker # Background processor
|
||||
npm run -w @kxkm/worker test
|
||||
```
|
||||
+121
-17
@@ -12,6 +12,7 @@
|
||||
const DEBUG = process.env.NODE_ENV !== "production" || process.env.DEBUG === "1";
|
||||
|
||||
import { trackError } from "./error-tracker.js";
|
||||
import { scheduler, VRAM_BUDGETS } from "./inference-scheduler.js";
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
@@ -96,6 +97,50 @@ function buildDefaultOptions(): ContextStoreOptions {
|
||||
|
||||
const DEFAULT_OPTIONS: ContextStoreOptions = buildDefaultOptions();
|
||||
|
||||
function extractFirstJsonObject(raw: string): string | null {
|
||||
const start = raw.indexOf("{");
|
||||
if (start < 0) return null;
|
||||
|
||||
let depth = 0;
|
||||
let inString = false;
|
||||
let escaped = false;
|
||||
|
||||
for (let index = start; index < raw.length; index += 1) {
|
||||
const char = raw[index];
|
||||
if (inString) {
|
||||
if (escaped) {
|
||||
escaped = false;
|
||||
continue;
|
||||
}
|
||||
if (char === "\\") {
|
||||
escaped = true;
|
||||
continue;
|
||||
}
|
||||
if (char === "\"") {
|
||||
inString = false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (char === "\"") {
|
||||
inString = true;
|
||||
continue;
|
||||
}
|
||||
if (char === "{") {
|
||||
depth += 1;
|
||||
continue;
|
||||
}
|
||||
if (char === "}") {
|
||||
depth -= 1;
|
||||
if (depth === 0) {
|
||||
return raw.slice(start, index + 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Context Store
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -133,10 +178,53 @@ export class ContextStore {
|
||||
}
|
||||
}
|
||||
|
||||
private async writeSummaryFile(filePath: string, summary: ContextSummary): Promise<void> {
|
||||
const tmp = `${filePath}.${process.pid}.${Date.now().toString(36)}.tmp`;
|
||||
await fs.writeFile(tmp, `${JSON.stringify(summary, null, 2)}\n`, "utf-8");
|
||||
await fs.rename(tmp, filePath);
|
||||
}
|
||||
|
||||
private normalizeSummary(raw: unknown, channel: string): ContextSummary | null {
|
||||
if (!raw || typeof raw !== "object") return null;
|
||||
const parsed = raw as Record<string, unknown>;
|
||||
if (typeof parsed.summaryText !== "string") return null;
|
||||
return {
|
||||
channel: typeof parsed.channel === "string" && parsed.channel.length > 0 ? parsed.channel : channel,
|
||||
summaryText: parsed.summaryText,
|
||||
entriesCompacted: typeof parsed.entriesCompacted === "number" && Number.isFinite(parsed.entriesCompacted)
|
||||
? parsed.entriesCompacted
|
||||
: 0,
|
||||
lastCompactedAt: typeof parsed.lastCompactedAt === "string" ? parsed.lastCompactedAt : new Date(0).toISOString(),
|
||||
totalCompactions: typeof parsed.totalCompactions === "number" && Number.isFinite(parsed.totalCompactions)
|
||||
? parsed.totalCompactions
|
||||
: 0,
|
||||
};
|
||||
}
|
||||
|
||||
private async readSummary(channel: string): Promise<ContextSummary | null> {
|
||||
const summaryPath = this.summaryFile(channel);
|
||||
try {
|
||||
const raw = await fs.readFile(this.summaryFile(channel), "utf-8");
|
||||
return this.parseJson<ContextSummary>(raw);
|
||||
const raw = await fs.readFile(summaryPath, "utf-8");
|
||||
const parsed = this.parseJson<unknown>(raw);
|
||||
const normalized = this.normalizeSummary(parsed, channel);
|
||||
if (normalized) return normalized;
|
||||
|
||||
const recoveredRaw = extractFirstJsonObject(raw);
|
||||
if (recoveredRaw) {
|
||||
const recoveredParsed = this.parseJson<unknown>(recoveredRaw);
|
||||
const recovered = this.normalizeSummary(recoveredParsed, channel);
|
||||
if (recovered) {
|
||||
await this.writeSummaryFile(summaryPath, recovered);
|
||||
return recovered;
|
||||
}
|
||||
}
|
||||
|
||||
const quarantinePath = path.join(
|
||||
path.dirname(summaryPath),
|
||||
`${path.basename(summaryPath, ".json")}.corrupt.${Date.now().toString(36)}.json`,
|
||||
);
|
||||
await fs.rename(summaryPath, quarantinePath).catch(() => {});
|
||||
return null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
@@ -315,21 +403,37 @@ export class ContextStore {
|
||||
|
||||
let summaryText = existingSummary; // fallback
|
||||
try {
|
||||
const response = await fetch(`${this.options.ollamaUrl}/api/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: this.options.compactionModel,
|
||||
messages: [{ role: "user", content: prompt }],
|
||||
stream: false,
|
||||
}),
|
||||
signal: AbortSignal.timeout(120_000),
|
||||
});
|
||||
summaryText = await scheduler.submit<string>({
|
||||
id: `compact-${channel}-${Date.now()}`,
|
||||
device: "gpu",
|
||||
priority: "low",
|
||||
label: `context-compact:${channel}`,
|
||||
vramMB: VRAM_BUDGETS.ollama,
|
||||
execute: async () => {
|
||||
const llmUrl = process.env.LLM_URL || "http://localhost:11434";
|
||||
const llmModel = process.env.LLM_MODEL || "qwen-14b-awq";
|
||||
const llmApiKey = process.env.LLM_API_KEY || "";
|
||||
const llmH: Record<string, string> = { "Content-Type": "application/json" };
|
||||
if (llmApiKey) llmH["Authorization"] = `Bearer ${llmApiKey}`;
|
||||
const response = await fetch(`${llmUrl}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: llmH,
|
||||
body: JSON.stringify({
|
||||
model: llmModel,
|
||||
messages: [{ role: "user", content: prompt }],
|
||||
stream: false,
|
||||
max_tokens: 2000,
|
||||
}),
|
||||
signal: AbortSignal.timeout(120_000),
|
||||
});
|
||||
|
||||
if (response.ok) {
|
||||
const data = (await response.json()) as { message?: { content?: string } };
|
||||
summaryText = data.message?.content || existingSummary;
|
||||
}
|
||||
if (response.ok) {
|
||||
const data = (await response.json()) as { choices?: [{ message?: { content?: string } }] };
|
||||
return data.choices?.[0]?.message?.content || existingSummary;
|
||||
}
|
||||
return existingSummary;
|
||||
},
|
||||
});
|
||||
} catch (err) {
|
||||
trackError("context_summarization", err, { channel });
|
||||
// Keep existing summary, still compact the raw file
|
||||
@@ -344,7 +448,7 @@ export class ContextStore {
|
||||
totalCompactions: (previousSummary?.totalCompactions || 0) + 1,
|
||||
};
|
||||
|
||||
await fs.writeFile(this.summaryFile(channel), JSON.stringify(summaryData, null, 2), "utf-8");
|
||||
await this.writeSummaryFile(this.summaryFile(channel), summaryData);
|
||||
|
||||
// Replace raw file with only recent entries
|
||||
const newContent = toKeep.join("\n") + "\n";
|
||||
|
||||
+116
-84
@@ -1,11 +1,11 @@
|
||||
/**
|
||||
* LLM Client — routes through mascarade /v1/chat/completions with Ollama fallback
|
||||
* Canonical LLM client.
|
||||
*
|
||||
* mascarade /v1/chat/completions endpoint (OpenAI-compatible):
|
||||
* POST { model, messages, temperature, max_tokens }
|
||||
* Returns: { model, choices: [{ message: { content } }], usage }
|
||||
* Local runtime:
|
||||
* vLLM / TurboQuant exposed via OpenAI-compatible `/v1/chat/completions`
|
||||
*
|
||||
* Fallback: direct Ollama /api/chat if mascarade is unavailable
|
||||
* Optional cloud routing:
|
||||
* Mascarade `/v1/chat/completions` for explicit cloud provider models
|
||||
*/
|
||||
|
||||
import logger from "./logger.js";
|
||||
@@ -17,9 +17,10 @@ import { incrementCounter } from "./perf.js";
|
||||
|
||||
const MASCARADE_URL = process.env.MASCARADE_URL || "http://127.0.0.1:8100";
|
||||
const MASCARADE_API_KEY = process.env.MASCARADE_API_KEY || "";
|
||||
const OLLAMA_URL = process.env.OLLAMA_URL || "http://127.0.0.1:11434";
|
||||
const LLM_URL = process.env.LLM_URL || "http://127.0.0.1:11434";
|
||||
const LLM_MODEL = process.env.LLM_MODEL || "qwen-14b-awq";
|
||||
const LLM_TIMEOUT_MS = parseInt(process.env.LLM_TIMEOUT_MS || "45000", 10);
|
||||
const DEFAULT_MODEL = process.env.LLM_DEFAULT_MODEL || "qwen3.5:9b";
|
||||
const DEFAULT_MODEL = process.env.LLM_DEFAULT_MODEL || LLM_MODEL;
|
||||
|
||||
// RouteLLM-style complexity routing
|
||||
const ROUTELLM_ENABLED = process.env.ROUTELLM_ENABLED === "true";
|
||||
@@ -42,7 +43,17 @@ function mascaradeRecheckMs(): number {
|
||||
export interface ChatMessage {
|
||||
role: "system" | "user" | "assistant" | "tool";
|
||||
content: string;
|
||||
tool_calls?: Array<{ function: { name: string; arguments: Record<string, unknown> } }>;
|
||||
tool_call_id?: string;
|
||||
tool_calls?: ChatToolCall[];
|
||||
}
|
||||
|
||||
export interface ChatToolCall {
|
||||
id?: string;
|
||||
type?: "function";
|
||||
function: {
|
||||
name: string;
|
||||
arguments: Record<string, unknown> | string;
|
||||
};
|
||||
}
|
||||
|
||||
export interface ChatOptions {
|
||||
@@ -61,7 +72,7 @@ export interface ChatResponse {
|
||||
content: string;
|
||||
model: string;
|
||||
provider: string;
|
||||
toolCalls?: Array<{ function: { name: string; arguments: Record<string, unknown> } }>;
|
||||
toolCalls?: ChatToolCall[];
|
||||
thinking?: string;
|
||||
usage?: { promptTokens: number; completionTokens: number; totalTokens: number };
|
||||
}
|
||||
@@ -92,12 +103,12 @@ function shouldTryMascarade(): boolean {
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Parse model: detect "provider:model" vs plain Ollama model
|
||||
// Parse model: detect "provider:model" vs local runtime model
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function parseModel(model: string | undefined): { provider: string | null; model: string } {
|
||||
const m = model || DEFAULT_MODEL;
|
||||
const knownProviders = ["claude", "openai", "mistral-api", "google", "bedrock", "huggingface", "ollama", "llama_cpp"];
|
||||
const knownProviders = ["claude", "openai", "mistral-api", "google", "bedrock", "huggingface"];
|
||||
const colonIdx = m.indexOf(":");
|
||||
if (colonIdx > 0) {
|
||||
const prefix = m.slice(0, colonIdx);
|
||||
@@ -105,18 +116,26 @@ function parseModel(model: string | undefined): { provider: string | null; model
|
||||
return { provider: prefix, model: m.slice(colonIdx + 1) };
|
||||
}
|
||||
}
|
||||
// Ollama model (qwen3.5:9b, mistral:7b, etc.) — let mascarade route via default provider
|
||||
// Local runtime model (qwen3.5:9b, qwen-14b-awq, mistral:7b, etc.)
|
||||
return { provider: null, model: m };
|
||||
}
|
||||
|
||||
function resolveRuntimeModel(model: string | undefined): string {
|
||||
const parsed = parseModel(model);
|
||||
if (parsed.provider) {
|
||||
return DEFAULT_MODEL;
|
||||
}
|
||||
return parsed.model || DEFAULT_MODEL;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// RouteLLM — complexity scoring for smart routing
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Score message complexity (0-1). Higher = needs stronger model.
|
||||
* 0.0-0.3: trivial (salut, oui, merci) → local Ollama
|
||||
* 0.3-0.6: moderate (short questions) → local Ollama
|
||||
* 0.0-0.3: trivial (salut, oui, merci) → runtime local
|
||||
* 0.3-0.6: moderate (short questions) → runtime local
|
||||
* 0.6-1.0: complex (analysis, code, multilingual) → strong provider if available
|
||||
*/
|
||||
function scoreComplexity(messages: ChatMessage[]): number {
|
||||
@@ -158,74 +177,67 @@ function scoreComplexity(messages: ChatMessage[]): number {
|
||||
|
||||
/**
|
||||
* Decide routing based on complexity score.
|
||||
* Returns "ollama" to force local, "mascarade" to prefer strong provider, or null for default behavior.
|
||||
* Returns "runtime" to force local, "mascarade" to prefer strong provider, or null for default behavior.
|
||||
*/
|
||||
function routeByComplexity(messages: ChatMessage[]): { route: "ollama" | "mascarade" | null; complexity: number } {
|
||||
function routeByComplexity(messages: ChatMessage[]): { route: "runtime" | "mascarade" | null; complexity: number } {
|
||||
if (!ROUTELLM_ENABLED) return { route: null, complexity: -1 };
|
||||
const complexity = scoreComplexity(messages);
|
||||
if (complexity < ROUTELLM_THRESHOLD) {
|
||||
return { route: "ollama", complexity };
|
||||
return { route: "runtime", complexity };
|
||||
}
|
||||
return { route: "mascarade", complexity };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Non-streaming: mascarade /send → Ollama fallback
|
||||
// Non-streaming: local runtime primary, mascarade only for explicit cloud/provider routing
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export async function chat(messages: ChatMessage[], opts: ChatOptions = {}): Promise<ChatResponse> {
|
||||
const { provider } = parseModel(opts.model);
|
||||
const { route, complexity } = routeByComplexity(messages);
|
||||
|
||||
// RouteLLM: skip mascarade entirely for simple messages
|
||||
if (route === "ollama") {
|
||||
logger.debug({ complexity, threshold: ROUTELLM_THRESHOLD }, "[llm] routeLLM → ollama (simple)");
|
||||
return chatViaOllama(messages, opts);
|
||||
if (!provider) {
|
||||
if (route === "runtime") {
|
||||
logger.debug({ complexity, threshold: ROUTELLM_THRESHOLD }, "[llm] routeLLM → runtime (simple)");
|
||||
}
|
||||
return chatViaRuntime(messages, opts);
|
||||
}
|
||||
|
||||
if (route === "mascarade") {
|
||||
logger.debug({ complexity, threshold: ROUTELLM_THRESHOLD }, "[llm] routeLLM → mascarade (complex)");
|
||||
}
|
||||
|
||||
if (shouldTryMascarade()) {
|
||||
try {
|
||||
const result = await chatViaMascarade(messages, opts);
|
||||
// If mascarade returns empty content, fall through to Ollama
|
||||
if (result.content) return result;
|
||||
logger.warn("[llm] mascarade returned empty content, falling back to Ollama");
|
||||
} catch (err) {
|
||||
logger.warn({ err: (err as Error).message }, "[llm] mascarade failed, falling back to Ollama");
|
||||
mascaradeAvailable = false;
|
||||
mascaradeLastCheck = Date.now();
|
||||
mascaradeFailCount++;
|
||||
}
|
||||
if (!shouldTryMascarade()) {
|
||||
throw new Error("Mascarade unavailable for explicit cloud provider routing");
|
||||
}
|
||||
return chatViaOllama(messages, opts);
|
||||
return chatViaMascarade(messages, opts);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Streaming: mascarade SSE for cloud providers, direct Ollama for local
|
||||
// Streaming: mascarade SSE for cloud providers, direct runtime for local
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export async function* streamChat(
|
||||
messages: ChatMessage[],
|
||||
opts: ChatOptions = {},
|
||||
): AsyncGenerator<string, ChatResponse> {
|
||||
const isCloudProvider = opts.model && /^(claude|openai|mistral-api|google|bedrock):/.test(opts.model);
|
||||
const isCloudProvider = Boolean(opts.model && /^(claude|openai|mistral-api|google|bedrock):/.test(opts.model));
|
||||
|
||||
// Cloud providers → stream via mascarade SSE (real streaming, not dump-all)
|
||||
if (isCloudProvider && shouldTryMascarade()) {
|
||||
try {
|
||||
return yield* streamViaMascarade(messages, opts);
|
||||
} catch (err) {
|
||||
logger.warn({ err: (err as Error).message }, "[llm] mascarade stream failed, falling back to Ollama");
|
||||
logger.warn({ err: (err as Error).message }, "[llm] mascarade stream failed");
|
||||
mascaradeAvailable = false;
|
||||
mascaradeLastCheck = Date.now();
|
||||
mascaradeFailCount++;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
// Local models → stream via Ollama directly (fastest path)
|
||||
return yield* streamViaOllama(messages, opts);
|
||||
// Local models → stream via runtime directly (fastest path)
|
||||
return yield* streamViaRuntime(messages, opts);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -242,7 +254,7 @@ async function chatViaMascarade(messages: ChatMessage[], opts: ChatOptions): Pro
|
||||
|
||||
const resp = await fetch(`${MASCARADE_URL}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
headers: { "Content-Type": "application/json", ...(process.env.LLM_API_KEY ? { Authorization: `Bearer ${process.env.LLM_API_KEY}` } : {}) },
|
||||
body: JSON.stringify({
|
||||
model: modelStr,
|
||||
messages: messages.map(m => ({ role: m.role, content: m.content })),
|
||||
@@ -408,81 +420,94 @@ async function* streamViaMascarade(
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Direct Ollama (fallback + streaming)
|
||||
// Direct runtime (vLLM / TurboQuant)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function chatViaOllama(messages: ChatMessage[], opts: ChatOptions): Promise<ChatResponse> {
|
||||
function toOpenAIMessage(message: ChatMessage): Record<string, unknown> {
|
||||
const base: Record<string, unknown> = {
|
||||
role: message.role,
|
||||
content: message.content,
|
||||
};
|
||||
if (message.tool_calls && message.tool_calls.length > 0) {
|
||||
base.tool_calls = message.tool_calls.map((toolCall) => ({
|
||||
id: toolCall.id,
|
||||
type: toolCall.type || "function",
|
||||
function: {
|
||||
name: toolCall.function.name,
|
||||
arguments: typeof toolCall.function.arguments === "string"
|
||||
? toolCall.function.arguments
|
||||
: JSON.stringify(toolCall.function.arguments),
|
||||
},
|
||||
}));
|
||||
}
|
||||
if (message.role === "tool" && message.tool_call_id) {
|
||||
base.tool_call_id = message.tool_call_id;
|
||||
}
|
||||
return base;
|
||||
}
|
||||
|
||||
async function chatViaRuntime(messages: ChatMessage[], opts: ChatOptions): Promise<ChatResponse> {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), LLM_TIMEOUT_MS);
|
||||
const { model } = parseModel(opts.model);
|
||||
const model = resolveRuntimeModel(opts.model);
|
||||
|
||||
try {
|
||||
const resp = await fetch(`${OLLAMA_URL}/api/chat`, {
|
||||
const resp = await fetch(`${LLM_URL}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
headers: { "Content-Type": "application/json", ...(process.env.LLM_API_KEY ? { Authorization: `Bearer ${process.env.LLM_API_KEY}` } : {}) },
|
||||
body: JSON.stringify({
|
||||
model,
|
||||
messages: messages.map(m => ({ role: m.role, content: m.content })),
|
||||
messages: messages.map(toOpenAIMessage),
|
||||
stream: false,
|
||||
options: {
|
||||
num_predict: opts.maxTokens || 800,
|
||||
...(opts.numCtx ? { num_ctx: opts.numCtx } : {}),
|
||||
num_batch: opts.numBatch || 512,
|
||||
},
|
||||
keep_alive: opts.keepAlive || "30m",
|
||||
...(opts.think !== undefined ? { think: opts.think } : {}),
|
||||
temperature: opts.temperature,
|
||||
max_tokens: opts.maxTokens || 800,
|
||||
...(opts.tools && opts.tools.length > 0 ? { tools: opts.tools } : {}),
|
||||
}),
|
||||
signal: controller.signal,
|
||||
});
|
||||
|
||||
if (!resp.ok) throw new Error(`Ollama ${resp.status}: ${resp.statusText}`);
|
||||
incrementCounter("llm_ollama_calls");
|
||||
if (!resp.ok) throw new Error(`vLLM ${resp.status}: ${resp.statusText}`);
|
||||
incrementCounter("llm_runtime_calls");
|
||||
|
||||
const data = await resp.json() as {
|
||||
message?: { content?: string; thinking?: string; tool_calls?: ChatResponse["toolCalls"] };
|
||||
choices?: [{ message?: { content?: string; tool_calls?: ChatResponse["toolCalls"] } }];
|
||||
};
|
||||
|
||||
const msg = data.choices?.[0]?.message;
|
||||
return {
|
||||
content: data.message?.content || "",
|
||||
content: msg?.content || "",
|
||||
model,
|
||||
provider: "ollama",
|
||||
toolCalls: data.message?.tool_calls,
|
||||
thinking: data.message?.thinking,
|
||||
provider: "vllm",
|
||||
toolCalls: msg?.tool_calls,
|
||||
};
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
|
||||
async function* streamViaOllama(
|
||||
async function* streamViaRuntime(
|
||||
messages: ChatMessage[],
|
||||
opts: ChatOptions,
|
||||
): AsyncGenerator<string, ChatResponse> {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), LLM_TIMEOUT_MS);
|
||||
const { model } = parseModel(opts.model);
|
||||
const model = resolveRuntimeModel(opts.model);
|
||||
|
||||
try {
|
||||
const resp = await fetch(`${OLLAMA_URL}/api/chat`, {
|
||||
const resp = await fetch(`${LLM_URL}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
headers: { "Content-Type": "application/json", ...(process.env.LLM_API_KEY ? { Authorization: `Bearer ${process.env.LLM_API_KEY}` } : {}) },
|
||||
body: JSON.stringify({
|
||||
model,
|
||||
messages: messages.map(m => ({ role: m.role, content: m.content })),
|
||||
messages: messages.map(toOpenAIMessage),
|
||||
stream: true,
|
||||
options: {
|
||||
num_predict: opts.maxTokens || 800,
|
||||
...(opts.numCtx ? { num_ctx: opts.numCtx } : {}),
|
||||
num_batch: opts.numBatch || 512,
|
||||
},
|
||||
keep_alive: opts.keepAlive || "30m",
|
||||
think: false,
|
||||
temperature: opts.temperature,
|
||||
max_tokens: opts.maxTokens || 800,
|
||||
}),
|
||||
signal: controller.signal,
|
||||
});
|
||||
|
||||
if (!resp.ok) throw new Error(`Ollama ${resp.status}: ${resp.statusText}`);
|
||||
if (!resp.ok) throw new Error(`vLLM ${resp.status}: ${resp.statusText}`);
|
||||
|
||||
const reader = resp.body?.getReader();
|
||||
if (!reader) throw new Error("No response body");
|
||||
@@ -490,18 +515,24 @@ async function* streamViaOllama(
|
||||
const decoder = new TextDecoder();
|
||||
let fullText = "";
|
||||
let inThinking = false;
|
||||
let sseBuffer = "";
|
||||
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
|
||||
const chunk = decoder.decode(value, { stream: true });
|
||||
for (const line of chunk.split("\n").filter(Boolean)) {
|
||||
if (line.length > 102_400) continue; // Skip oversized chunks (100KB max)
|
||||
sseBuffer += decoder.decode(value, { stream: true });
|
||||
const lines = sseBuffer.split("\n");
|
||||
sseBuffer = lines.pop() || "";
|
||||
|
||||
for (const line of lines) {
|
||||
if (!line.startsWith("data: ")) continue;
|
||||
const raw = line.slice(6).trim();
|
||||
if (raw === "[DONE]") break;
|
||||
try {
|
||||
const parsed = JSON.parse(line) as { message?: { content?: string }; done?: boolean };
|
||||
if (parsed.message?.content) {
|
||||
const c = parsed.message.content;
|
||||
const parsed = JSON.parse(raw) as { choices?: [{ delta?: { content?: string } }] };
|
||||
const c = parsed.choices?.[0]?.delta?.content;
|
||||
if (c) {
|
||||
fullText += c;
|
||||
if (c.includes("<think>")) inThinking = true;
|
||||
if (!inThinking) yield c;
|
||||
@@ -512,7 +543,7 @@ async function* streamViaOllama(
|
||||
}
|
||||
|
||||
const cleaned = fullText.replace(/<think>[\s\S]*?<\/think>\s*/g, "").trim();
|
||||
return { content: cleaned, model, provider: "ollama" };
|
||||
return { content: cleaned, model, provider: "vllm" };
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
@@ -523,12 +554,13 @@ async function* streamViaOllama(
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export async function getProviders(): Promise<string[]> {
|
||||
const providers = ["vllm-turboquant"];
|
||||
try {
|
||||
const resp = await fetch(`${MASCARADE_URL}/health`, { signal: AbortSignal.timeout(3000) });
|
||||
if (!resp.ok) return ["ollama"];
|
||||
if (!resp.ok) return providers;
|
||||
const data = await resp.json() as { providers?: string[] };
|
||||
return data.providers || ["ollama"];
|
||||
return [...providers, ...(data.providers || [])];
|
||||
} catch {
|
||||
return ["ollama"];
|
||||
return providers;
|
||||
}
|
||||
}
|
||||
|
||||
+94
-29
@@ -1,5 +1,6 @@
|
||||
/**
|
||||
* Minimal local RAG using Ollama embeddings.
|
||||
* Minimal local RAG with pluggable embedding backend.
|
||||
* Supports TEI/OpenAI-compatible (/v1/embeddings) and Ollama (/api/embed).
|
||||
* Stores document chunks with their embeddings in memory.
|
||||
* Uses cosine similarity for retrieval.
|
||||
*/
|
||||
@@ -14,6 +15,7 @@ const RAG_CHUNK_SIZE = Number(process.env.RAG_CHUNK_SIZE) || 500;
|
||||
const RAG_MIN_SIMILARITY = Number(process.env.RAG_MIN_SIMILARITY) || 0.3;
|
||||
const RAG_MAX_RESULTS = Number(process.env.RAG_MAX_RESULTS) || 3;
|
||||
const RAG_EMBEDDING_MODEL = process.env.RAG_EMBEDDING_MODEL || "bge-m3";
|
||||
const EMBEDDING_BACKEND = process.env.EMBEDDING_BACKEND || "tei"; // "tei" or "ollama"
|
||||
|
||||
interface DocumentChunk {
|
||||
id: string;
|
||||
@@ -36,46 +38,75 @@ export class LocalRAG {
|
||||
private options: RAGOptions;
|
||||
private _rerankerFailCount = 0;
|
||||
private _rerankerLastFail = 0;
|
||||
private namespaceChunks: Map<string, DocumentChunk[]> = new Map();
|
||||
|
||||
constructor(options: RAGOptions) {
|
||||
this.options = options;
|
||||
}
|
||||
|
||||
/** Verify embedding model is available on Ollama, pull if missing. */
|
||||
/** Verify embedding backend is reachable. */
|
||||
async init(): Promise<void> {
|
||||
const ollamaUrl = this.options.ollamaUrl;
|
||||
const url = this.options.ollamaUrl;
|
||||
const model = this.options.embeddingModel || RAG_EMBEDDING_MODEL;
|
||||
const backend = EMBEDDING_BACKEND;
|
||||
try {
|
||||
const resp = await fetch(`${ollamaUrl}/api/tags`, { signal: AbortSignal.timeout(5000) });
|
||||
const data = (await resp.json()) as { models?: Array<{ name: string }> };
|
||||
const models = data.models?.map((m) => m.name) || [];
|
||||
const available = models.some((m) => m.startsWith(model));
|
||||
if (!available) {
|
||||
logger.warn({ model, available: models.slice(0, 5) }, "[rag] Embedding model not found, pulling...");
|
||||
await fetch(`${ollamaUrl}/api/pull`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ name: model }),
|
||||
signal: AbortSignal.timeout(300_000),
|
||||
});
|
||||
logger.info({ model }, "[rag] Embedding model pulled successfully");
|
||||
if (backend === "tei") {
|
||||
// TEI health check
|
||||
const resp = await fetch(`${url}/info`, { signal: AbortSignal.timeout(5000) });
|
||||
if (resp.ok) {
|
||||
const info = (await resp.json()) as { model_id?: string };
|
||||
logger.info({ model: info.model_id, backend }, "[rag] TEI embedding server ready");
|
||||
} else {
|
||||
logger.warn({ status: resp.status }, "[rag] TEI health check failed");
|
||||
}
|
||||
} else {
|
||||
logger.debug({ model }, "[rag] Embedding model available");
|
||||
// Ollama: check model availability, pull if missing
|
||||
const resp = await fetch(`${url}/api/tags`, { signal: AbortSignal.timeout(5000) });
|
||||
const data = (await resp.json()) as { models?: Array<{ name: string }> };
|
||||
const models = data.models?.map((m) => m.name) || [];
|
||||
const available = models.some((m) => m.startsWith(model));
|
||||
if (!available) {
|
||||
logger.warn({ model, available: models.slice(0, 5) }, "[rag] Embedding model not found, pulling...");
|
||||
await fetch(`${url}/api/pull`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ name: model }),
|
||||
signal: AbortSignal.timeout(300_000),
|
||||
});
|
||||
logger.info({ model }, "[rag] Embedding model pulled successfully");
|
||||
} else {
|
||||
logger.debug({ model, backend }, "[rag] Embedding model available");
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
logger.warn({ err }, "[rag] Could not verify embedding model");
|
||||
logger.warn({ err, backend }, "[rag] Could not verify embedding backend");
|
||||
}
|
||||
}
|
||||
|
||||
/** Embed text via Ollama /api/embed */
|
||||
/** Embed text via TEI (/v1/embeddings) or Ollama (/api/embed). */
|
||||
async embed(text: string): Promise<number[]> {
|
||||
const response = await fetch(`${this.options.ollamaUrl}/api/embed`, {
|
||||
const url = this.options.ollamaUrl;
|
||||
const model = this.options.embeddingModel || RAG_EMBEDDING_MODEL;
|
||||
|
||||
if (EMBEDDING_BACKEND === "tei") {
|
||||
// OpenAI-compatible format (TEI, vLLM, etc.)
|
||||
const response = await fetch(`${url}/v1/embeddings`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ model, input: text }),
|
||||
});
|
||||
if (!response.ok) {
|
||||
throw new Error(`TEI embed returned ${response.status}: ${response.statusText}`);
|
||||
}
|
||||
const data = (await response.json()) as { data?: Array<{ embedding: number[] }> };
|
||||
return data.data?.[0]?.embedding || [];
|
||||
}
|
||||
|
||||
// Ollama format
|
||||
const response = await fetch(`${url}/api/embed`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: this.options.embeddingModel || RAG_EMBEDDING_MODEL,
|
||||
input: text,
|
||||
}),
|
||||
body: JSON.stringify({ model, input: text }),
|
||||
});
|
||||
if (!response.ok) {
|
||||
throw new Error(`Ollama embed returned ${response.status}: ${response.statusText}`);
|
||||
@@ -86,14 +117,15 @@ export class LocalRAG {
|
||||
|
||||
/** Add a document (split into chunks, embed each).
|
||||
* If LightRAG is configured, also pushes the full text there (dual write). */
|
||||
async addDocument(text: string, source: string): Promise<number> {
|
||||
async addDocument(text: string, source: string, namespace?: string): Promise<number> {
|
||||
// Dual-write to LightRAG if configured
|
||||
if (this.options.lightragUrl) {
|
||||
try {
|
||||
const res = await fetch(`${this.options.lightragUrl}/documents/text`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ text }),
|
||||
body: JSON.stringify({ text, description: namespace }),
|
||||
signal: AbortSignal.timeout(5000),
|
||||
});
|
||||
if (res.ok) {
|
||||
logger.debug({ source }, "[rag:lightrag] addDocument to LightRAG OK");
|
||||
@@ -107,16 +139,25 @@ export class LocalRAG {
|
||||
|
||||
// Always index locally
|
||||
const textChunks = splitIntoChunks(text, RAG_CHUNK_SIZE);
|
||||
const newChunks: DocumentChunk[] = [];
|
||||
for (const chunk of textChunks) {
|
||||
const embedding = await this.embed(chunk);
|
||||
this.chunks.push({
|
||||
const docChunk: DocumentChunk = {
|
||||
id: `${source}_${this.chunks.length}`,
|
||||
text: chunk,
|
||||
source,
|
||||
embedding,
|
||||
});
|
||||
};
|
||||
this.chunks.push(docChunk);
|
||||
newChunks.push(docChunk);
|
||||
}
|
||||
return textChunks.length;
|
||||
if (namespace) {
|
||||
if (!this.namespaceChunks.has(namespace)) {
|
||||
this.namespaceChunks.set(namespace, []);
|
||||
}
|
||||
this.namespaceChunks.get(namespace)!.push(...newChunks);
|
||||
}
|
||||
return newChunks.length;
|
||||
}
|
||||
|
||||
/** Search for relevant chunks.
|
||||
@@ -183,6 +224,30 @@ export class LocalRAG {
|
||||
return this.rerank(query, results, limit);
|
||||
}
|
||||
|
||||
/** Search within a specific persona namespace. Falls back to global search if namespace has no chunks. */
|
||||
async searchNamespace(
|
||||
query: string,
|
||||
namespace: string,
|
||||
maxResults?: number,
|
||||
): Promise<Array<{ text: string; source: string; score: number }>> {
|
||||
const nsChunks = this.namespaceChunks.get(namespace);
|
||||
if (!nsChunks || nsChunks.length === 0) {
|
||||
return this.search(query, maxResults);
|
||||
}
|
||||
const limit = maxResults ?? RAG_MAX_RESULTS;
|
||||
const queryEmbedding = await this.embed(query);
|
||||
const scored = nsChunks.map((chunk) => ({
|
||||
text: chunk.text,
|
||||
source: chunk.source,
|
||||
score: cosineSimilarity(queryEmbedding, chunk.embedding),
|
||||
}));
|
||||
scored.sort((a, b) => b.score - a.score);
|
||||
const results = scored
|
||||
.filter((s) => s.score >= (this.options.minSimilarity ?? RAG_MIN_SIMILARITY))
|
||||
.slice(0, limit);
|
||||
return this.rerank(query, results, limit);
|
||||
}
|
||||
|
||||
/** Rerank results using BGE cross-encoder for improved precision.
|
||||
* Falls back to original ordering if the reranker is unavailable. */
|
||||
private async rerank(
|
||||
|
||||
@@ -12,6 +12,13 @@ import { validateLoginInput } from "@kxkm/auth";
|
||||
import { buildChatChannels } from "@kxkm/chat-domain";
|
||||
import { getRecentErrors, getErrorCounts } from "../error-tracker.js";
|
||||
import { scheduler, getGPUUtilization } from "../inference-scheduler.js";
|
||||
|
||||
const LLM_API_KEY = process.env.LLM_API_KEY || "";
|
||||
function llmHeaders(): Record<string, string> {
|
||||
const h: Record<string, string> = { "Content-Type": "application/json" };
|
||||
if (LLM_API_KEY) h["Authorization"] = `Bearer ${LLM_API_KEY}`;
|
||||
return h;
|
||||
}
|
||||
import type { PersonaRecord } from "@kxkm/persona-domain";
|
||||
import type { ModelRegistryRecord, NodeGraphRecord, NodeRunRecord } from "@kxkm/node-engine";
|
||||
import type { StorageMode } from "../app-bootstrap.js";
|
||||
@@ -87,22 +94,22 @@ export function createSessionRoutes(deps: SessionRouteDeps): Router {
|
||||
|
||||
router.get("/api/v2/health", async (_req, res) => {
|
||||
const startMs = Date.now();
|
||||
const ollamaUrl = process.env.OLLAMA_URL || "http://localhost:11434";
|
||||
const llmUrl = process.env.LLM_URL || "http://localhost:11434";
|
||||
|
||||
const timeout = <T>(p: Promise<T>, ms = 2000): Promise<T> =>
|
||||
Promise.race([p, new Promise<never>((_, rej) => setTimeout(() => rej(new Error("timeout")), ms))]);
|
||||
|
||||
const [ollamaResult, dbResult] = await Promise.allSettled([
|
||||
timeout(fetch(`${ollamaUrl}/api/tags`).then(async (r) => {
|
||||
const body = await r.json() as { models?: unknown[] };
|
||||
return { ok: r.ok, models: Array.isArray(body.models) ? body.models.length : 0 };
|
||||
const [runtimeResult, dbResult] = await Promise.allSettled([
|
||||
timeout(fetch(`${llmUrl}/v1/models`, { headers: llmHeaders() }).then(async (r) => {
|
||||
const body = await r.json() as { data?: unknown[] };
|
||||
return { ok: r.ok, models: Array.isArray(body.data) ? body.data.length : 0 };
|
||||
})),
|
||||
timeout(personaRepo.list().then((list) => ({ ok: true, count: list.length }))),
|
||||
]);
|
||||
|
||||
const ollama = ollamaResult.status === "fulfilled"
|
||||
? { status: "ok" as const, models_loaded: ollamaResult.value.models }
|
||||
: { status: "error" as const, error: (ollamaResult.reason as Error).message };
|
||||
const runtime = runtimeResult.status === "fulfilled"
|
||||
? { status: "ok" as const, models_loaded: runtimeResult.value.models }
|
||||
: { status: "error" as const, error: (runtimeResult.reason as Error).message };
|
||||
|
||||
const db = dbResult.status === "fulfilled"
|
||||
? { status: "ok" as const, personas: dbResult.value.count }
|
||||
@@ -117,7 +124,7 @@ export function createSessionRoutes(deps: SessionRouteDeps): Router {
|
||||
roles: ["admin", "editor", "operator", "viewer"] satisfies UserRole[],
|
||||
uptime_sec: uptimeSec,
|
||||
uptime_human: uptimeHuman,
|
||||
ollama,
|
||||
runtime,
|
||||
database: db,
|
||||
health_check_ms: Date.now() - startMs,
|
||||
}));
|
||||
@@ -194,12 +201,20 @@ export function createSessionRoutes(deps: SessionRouteDeps): Router {
|
||||
res.json(asApiData(listWorkflows()));
|
||||
});
|
||||
|
||||
// LLM providers (mascarade status)
|
||||
// LLM providers and runtime status
|
||||
router.get("/api/v2/llm-providers", async (_req, res) => {
|
||||
const { getProviders, checkMascaradeHealth } = await import("../llm-client.js");
|
||||
const healthy = await checkMascaradeHealth();
|
||||
const providers = await getProviders();
|
||||
res.json(asApiData({ mascarade: healthy, providers, fallback: "ollama" }));
|
||||
res.json(asApiData({
|
||||
runtime: {
|
||||
kind: "vllm-turboquant",
|
||||
url: process.env.LLM_URL || "http://localhost:11434",
|
||||
model: process.env.LLM_MODEL || "qwen-14b-awq",
|
||||
},
|
||||
mascarade: healthy,
|
||||
providers,
|
||||
}));
|
||||
});
|
||||
|
||||
// RAG search endpoint — for mascarade MCP tool integration
|
||||
@@ -433,28 +448,27 @@ export function createSessionRoutes(deps: SessionRouteDeps): Router {
|
||||
|
||||
router.post("/api/v2/ai/suggest-prompt", async (req, res) => {
|
||||
const { type, style: compStyle, existing, context } = req.body || {};
|
||||
const ollamaUrl = process.env.OLLAMA_URL || "http://localhost:11434";
|
||||
|
||||
// Image prompt generation mode — triggered by style:"random" or type not in DAW types
|
||||
const isImageMode = compStyle === "random" || type === "image";
|
||||
if (isImageMode) {
|
||||
const llmUrl = process.env.LLM_URL || "http://localhost:11434";
|
||||
const llmModel = process.env.LLM_MODEL || "qwen-14b-awq";
|
||||
try {
|
||||
const resp = await fetch(`${ollamaUrl}/api/chat`, {
|
||||
const resp = await fetch(`${llmUrl}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
headers: llmHeaders(),
|
||||
body: JSON.stringify({
|
||||
model: "qwen3:8b",
|
||||
model: llmModel,
|
||||
messages: [{ role: "user", content: "Generate a creative, detailed image prompt for AI image generation. Be specific about style, lighting, mood, subject. Return ONLY the prompt in English, nothing else. Maximum 100 words." }],
|
||||
stream: false,
|
||||
options: { num_predict: 200 },
|
||||
keep_alive: "30m",
|
||||
think: false,
|
||||
max_tokens: 200,
|
||||
}),
|
||||
signal: AbortSignal.timeout(15000),
|
||||
});
|
||||
if (!resp.ok) throw new Error("Ollama error");
|
||||
const data = await resp.json() as { message?: { content?: string } };
|
||||
const prompt = data.message?.content?.replace(/<think>[\s\S]*?<\/think>/g, "").trim();
|
||||
if (!resp.ok) throw new Error("vLLM error");
|
||||
const data = await resp.json() as { choices?: [{ message?: { content?: string } }] };
|
||||
const prompt = data.choices?.[0]?.message?.content?.replace(/<think>[\s\S]*?<\/think>/g, "").trim();
|
||||
res.json({ ok: true, data: { prompt } });
|
||||
} catch {
|
||||
res.json({ ok: true, data: { prompt: "a mystical forest at twilight, bioluminescent mushrooms, fog, cinematic lighting, 8k" } });
|
||||
@@ -469,22 +483,22 @@ export function createSessionRoutes(deps: SessionRouteDeps): Router {
|
||||
fx: "un effet sonore ou une texture de fond",
|
||||
};
|
||||
|
||||
const llmUrl = process.env.LLM_URL || "http://localhost:11434";
|
||||
const llmModel = process.env.LLM_MODEL || "qwen-14b-awq";
|
||||
try {
|
||||
const resp = await fetch(`${ollamaUrl}/api/chat`, {
|
||||
const resp = await fetch(`${llmUrl}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: "qwen3:8b",
|
||||
model: llmModel,
|
||||
messages: [{ role: "user", content: `Tu es un compositeur sonore. Génère ${typeHints[type as string] || "un prompt audio"} pour une composition de style "${compStyle || "experimental"}". ${existing ? `Le prompt actuel est: "${existing}". Améliore-le.` : ""} ${context ? `Contexte des autres pistes: ${context}` : ""} Réponds UNIQUEMENT le prompt (1-2 phrases max, pas d'explication).` }],
|
||||
stream: false,
|
||||
options: { num_predict: 80 },
|
||||
keep_alive: "30m",
|
||||
think: false,
|
||||
max_tokens: 80,
|
||||
}),
|
||||
signal: AbortSignal.timeout(10000),
|
||||
});
|
||||
const data = await resp.json() as { message?: { content?: string } };
|
||||
res.json({ ok: true, prompt: data.message?.content?.trim() || "" });
|
||||
const data = await resp.json() as { choices?: [{ message?: { content?: string } }] };
|
||||
res.json({ ok: true, prompt: data.choices?.[0]?.message?.content?.trim() || "" });
|
||||
} catch {
|
||||
res.json({ ok: false, prompt: "" });
|
||||
}
|
||||
|
||||
+52
-15
@@ -166,27 +166,29 @@ async function main() {
|
||||
// -----------------------------------------------------------------------
|
||||
const server = http.createServer(app);
|
||||
|
||||
const ollamaUrl = process.env.OLLAMA_URL || "http://localhost:11434";
|
||||
const embeddingUrl = process.env.OLLAMA_URL || "http://localhost:11435";
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Pre-warm Ollama: load primary model into VRAM (non-blocking)
|
||||
// First inference is ~1-2s slower without this.
|
||||
// Pre-warm vLLM: verify model is loaded (non-blocking)
|
||||
// -----------------------------------------------------------------------
|
||||
const primaryModel = process.env.OLLAMA_MODEL || "qwen3.5:9b";
|
||||
fetch(`${ollamaUrl}/api/chat`, {
|
||||
const llmUrl = process.env.LLM_URL || "http://localhost:11434";
|
||||
const primaryModel = process.env.LLM_MODEL || "qwen-14b-awq";
|
||||
const llmApiKey = process.env.LLM_API_KEY || "";
|
||||
const llmHeaders: Record<string, string> = { "Content-Type": "application/json" };
|
||||
if (llmApiKey) llmHeaders["Authorization"] = `Bearer ${llmApiKey}`;
|
||||
fetch(`${llmUrl}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
headers: llmHeaders,
|
||||
body: JSON.stringify({
|
||||
model: primaryModel,
|
||||
messages: [{ role: "user", content: "ping" }],
|
||||
stream: false,
|
||||
options: { num_predict: 1, num_ctx: 512 },
|
||||
keep_alive: "30m",
|
||||
max_tokens: 1,
|
||||
}),
|
||||
}).then(() => {
|
||||
if (DEBUG) console.log(`[ollama] Pre-warmed ${primaryModel}`);
|
||||
if (DEBUG) console.log(`[vllm] Pre-warmed ${primaryModel}`);
|
||||
}).catch(() => {
|
||||
// Ollama not ready yet — model will load on first real request
|
||||
// vLLM not ready yet — model will serve on first real request
|
||||
});
|
||||
|
||||
// Pre-warm ComfyUI: load default checkpoint into VRAM (non-blocking)
|
||||
@@ -195,9 +197,9 @@ async function main() {
|
||||
});
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Initialize local RAG (embeddings via Ollama)
|
||||
// Initialize local RAG (embeddings still use the embedding backend)
|
||||
// -----------------------------------------------------------------------
|
||||
const rag = new LocalRAG({ ollamaUrl, lightragUrl: process.env.LIGHTRAG_URL, rerankerUrl: process.env.RERANKER_URL });
|
||||
const rag = new LocalRAG({ ollamaUrl: embeddingUrl, lightragUrl: process.env.LIGHTRAG_URL, rerankerUrl: process.env.RERANKER_URL });
|
||||
// Expose RAG instance on app for API routes
|
||||
(app as any)._rag = rag;
|
||||
|
||||
@@ -233,11 +235,45 @@ async function main() {
|
||||
}
|
||||
})();
|
||||
|
||||
// Per-persona corpus boot-loader
|
||||
(async () => {
|
||||
for (const persona of DEFAULT_PERSONAS) {
|
||||
if (!persona.corpus || persona.corpus.length === 0) continue;
|
||||
const ns = `persona:${persona.id}`;
|
||||
for (const entry of persona.corpus) {
|
||||
try {
|
||||
let text = '';
|
||||
if (entry.type === 'text' && entry.content) {
|
||||
text = entry.content;
|
||||
} else if (entry.type === 'url') {
|
||||
const resp = await fetch(entry.source, { signal: AbortSignal.timeout(10_000) });
|
||||
const html = await resp.text();
|
||||
// Strip scripts/styles first, then all tags
|
||||
text = html
|
||||
.replace(/<script[\s\S]*?<\/script>/gi, '')
|
||||
.replace(/<style[\s\S]*?<\/style>/gi, '')
|
||||
.replace(/<[^>]+>/g, ' ')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim()
|
||||
.slice(0, 8000);
|
||||
}
|
||||
if (text) {
|
||||
await rag.addDocument(text, entry.source, ns);
|
||||
logger.info({ persona: persona.id, source: entry.source, ns }, '[rag:corpus] loaded');
|
||||
}
|
||||
} catch (err) {
|
||||
logger.warn({ err, persona: persona.id, source: entry.source }, '[rag:corpus] load failed (non-critical)');
|
||||
}
|
||||
}
|
||||
}
|
||||
logger.info('[rag:corpus] persona corpus boot-loading complete');
|
||||
})();
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Initialize persistent context store (auto-compaction, 750 MB max)
|
||||
// -----------------------------------------------------------------------
|
||||
const contextStore = new ContextStore({
|
||||
ollamaUrl,
|
||||
ollamaUrl: embeddingUrl,
|
||||
maxTotalSizeMB: 750,
|
||||
maxEntriesBeforeCompact: 200,
|
||||
compactionModel: "qwen3:8b",
|
||||
@@ -249,7 +285,7 @@ async function main() {
|
||||
});
|
||||
|
||||
const wss = attachWebSocketChat(server, {
|
||||
ollamaUrl,
|
||||
ollamaUrl: embeddingUrl,
|
||||
rag,
|
||||
contextStore,
|
||||
loadPersonas: async () => {
|
||||
@@ -264,6 +300,7 @@ async function main() {
|
||||
color: defaultDef?.color || "",
|
||||
enabled: !(p as unknown as { disabled?: boolean }).disabled,
|
||||
maxTokens: defaultDef?.maxTokens,
|
||||
corpus: defaultDef?.corpus,
|
||||
};
|
||||
});
|
||||
},
|
||||
@@ -289,7 +326,7 @@ async function main() {
|
||||
app: "@kxkm/api",
|
||||
port,
|
||||
ws: "/ws",
|
||||
ollama: ollamaUrl,
|
||||
embeddings: embeddingUrl,
|
||||
}));
|
||||
});
|
||||
}
|
||||
|
||||
+189
-139
@@ -6,10 +6,43 @@ import { trackError } from "./error-tracker.js";
|
||||
import logger from "./logger.js";
|
||||
import type { ToolDefinition } from "./mcp-tools.js";
|
||||
import type { ChatPersona } from "./chat-types.js";
|
||||
import type { ChatMessage } from "./llm-client.js";
|
||||
|
||||
const FALLBACK_MODEL = process.env.OLLAMA_FALLBACK_MODEL || "qwen3:4b";
|
||||
const LLM_URL = process.env.LLM_URL || "http://localhost:11434";
|
||||
const LLM_MODEL = process.env.LLM_MODEL || "qwen-14b-awq";
|
||||
const LLM_API_KEY = process.env.LLM_API_KEY || "";
|
||||
|
||||
// HTTP keep-alive agent: reuses TCP connections to Ollama (saves ~5-20ms per request)
|
||||
function resolveRuntimeModel(model: string | undefined): string {
|
||||
if (!model) return LLM_MODEL;
|
||||
const cloudPrefixed = /^(claude|openai|mistral-api|google|bedrock|huggingface):/i.test(model);
|
||||
if (cloudPrefixed) return LLM_MODEL;
|
||||
const compatPrefixed = /^(ollama|vllm|runtime):/i.test(model);
|
||||
return compatPrefixed ? model.slice(model.indexOf(":") + 1) : model;
|
||||
}
|
||||
|
||||
/** Complete a chat via vLLM OpenAI-compatible API (non-streaming). Strips <think> blocks. */
|
||||
export async function vllmComplete(
|
||||
messages: Array<{ role: string; content: string }>,
|
||||
opts?: { maxTokens?: number; model?: string },
|
||||
): Promise<string> {
|
||||
const resp = await fetch(`${LLM_URL}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: llmHeaders(),
|
||||
body: JSON.stringify({
|
||||
model: resolveRuntimeModel(opts?.model),
|
||||
messages,
|
||||
max_tokens: opts?.maxTokens ?? 800,
|
||||
stream: false,
|
||||
}),
|
||||
signal: AbortSignal.timeout(45_000),
|
||||
});
|
||||
if (!resp.ok) throw new Error(`vLLM ${resp.status}: ${resp.statusText}`);
|
||||
const data = await resp.json() as { choices?: [{ message?: { content?: string } }] };
|
||||
return (data.choices?.[0]?.message?.content || "")
|
||||
.replace(/<think>[\s\S]*?<\/think>\s*/g, "").trim();
|
||||
}
|
||||
|
||||
// HTTP keep-alive agent: reuses TCP connections to the local runtime
|
||||
const ollamaAgent = new http.Agent({
|
||||
keepAlive: true,
|
||||
maxSockets: 10,
|
||||
@@ -30,7 +63,14 @@ try {
|
||||
// undici not available — fall back to default fetch (still OK, just no keep-alive)
|
||||
}
|
||||
|
||||
/** Fetch with keep-alive connection pooling to Ollama */
|
||||
/** Common headers for LLM runtime requests */
|
||||
function llmHeaders(): Record<string, string> {
|
||||
const h: Record<string, string> = { "Content-Type": "application/json" };
|
||||
if (LLM_API_KEY) h["Authorization"] = `Bearer ${LLM_API_KEY}`;
|
||||
return h;
|
||||
}
|
||||
|
||||
/** Fetch with keep-alive connection pooling to the local runtime */
|
||||
function ollamaFetch(url: string, init: RequestInit): Promise<Response> {
|
||||
return fetch(url, { ...init, ...ollamaFetchOpts } as RequestInit);
|
||||
}
|
||||
@@ -101,11 +141,11 @@ const DEBUG = process.env.NODE_ENV !== "production" || process.env.DEBUG === "1"
|
||||
// Ollama concurrency limiter (replaces manual semaphore)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// Match OLLAMA_NUM_PARALLEL (set via sudo-optimize.sh)
|
||||
// Match local runtime concurrency limits
|
||||
const ollamaLimit = pLimit(Number(process.env.MAX_OLLAMA_CONCURRENT) || 2);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Ollama streaming chat
|
||||
// Local runtime streaming chat
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export async function streamOllamaChat(
|
||||
@@ -116,61 +156,64 @@ export async function streamOllamaChat(
|
||||
onDone: (fullText: string) => void,
|
||||
onError: (err: Error) => void,
|
||||
): Promise<void> {
|
||||
// Always disable thinking for streaming — thinking output goes to separate field, not content stream
|
||||
const useThinking = false;
|
||||
await ollamaLimit(async () => {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), 45_000);
|
||||
const runtimeModel = resolveRuntimeModel(persona.model);
|
||||
const runtimeUrl = ollamaUrl || LLM_URL;
|
||||
|
||||
try {
|
||||
const response = await ollamaFetch(`${ollamaUrl}/api/chat`, {
|
||||
const response = await ollamaFetch(`${runtimeUrl}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
headers: llmHeaders(),
|
||||
body: JSON.stringify({
|
||||
model: persona.model,
|
||||
model: runtimeModel,
|
||||
messages: [
|
||||
{ role: "system", content: persona.systemPrompt },
|
||||
{ role: "user", content: userMessage },
|
||||
],
|
||||
stream: true,
|
||||
options: { num_predict: estimateMaxTokens(userMessage, persona.maxTokens), num_ctx: estimateNumCtx(persona.systemPrompt, userMessage), num_batch: 512 }, keep_alive: "30m", think: false,
|
||||
max_tokens: estimateMaxTokens(userMessage, persona.maxTokens),
|
||||
}),
|
||||
signal: controller.signal,
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Ollama returned ${response.status}: ${response.statusText}`);
|
||||
throw new Error(`vLLM returned ${response.status}: ${response.statusText}`);
|
||||
}
|
||||
|
||||
const reader = response.body?.getReader();
|
||||
if (!reader) {
|
||||
throw new Error("No response body from Ollama");
|
||||
throw new Error("No response body from vLLM");
|
||||
}
|
||||
|
||||
const decoder = new TextDecoder();
|
||||
let fullText = "";
|
||||
let inThinking = false;
|
||||
let sseBuffer = "";
|
||||
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
|
||||
const chunk = decoder.decode(value, { stream: true });
|
||||
const lines = chunk.split("\n").filter(Boolean);
|
||||
sseBuffer += decoder.decode(value, { stream: true });
|
||||
const lines = sseBuffer.split("\n");
|
||||
sseBuffer = lines.pop() || "";
|
||||
|
||||
for (const line of lines) {
|
||||
try {
|
||||
const parsed = JSON.parse(line) as { message?: { content?: string }; done?: boolean };
|
||||
if (parsed.message?.content) {
|
||||
const c = parsed.message.content;
|
||||
fullText += c;
|
||||
// Suppress <think>...</think> from streaming to client
|
||||
if (c.includes("<think>")) inThinking = true;
|
||||
if (!inThinking) onChunk(c);
|
||||
if (c.includes("</think>")) inThinking = false;
|
||||
const raw = line.startsWith("data: ") ? line.slice(6).trim() : line.trim();
|
||||
if (!raw) continue;
|
||||
if (raw === "[DONE]") break;
|
||||
const c = parseStreamingPayload(raw);
|
||||
if (c) {
|
||||
fullText += c;
|
||||
const visible = stripThinkingFromChunk(c);
|
||||
if (c.includes("<think>")) inThinking = true;
|
||||
if (visible && !inThinking) onChunk(visible);
|
||||
if (c.includes("</think>")) {
|
||||
inThinking = false;
|
||||
if (visible) onChunk(visible);
|
||||
}
|
||||
} catch {
|
||||
// Partial JSON -- skip
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -179,60 +222,6 @@ export async function streamOllamaChat(
|
||||
const cleaned = fullText.replace(/<think>[\s\S]*?<\/think>\s*/g, "").trim();
|
||||
onDone(cleaned);
|
||||
} catch (err) {
|
||||
// Try fallback model if primary fails
|
||||
if (persona.model !== FALLBACK_MODEL) {
|
||||
logger.warn({ nick: persona.nick, primaryModel: persona.model, fallback: FALLBACK_MODEL }, "Trying fallback model");
|
||||
const fallbackPersona = { ...persona, model: FALLBACK_MODEL };
|
||||
try {
|
||||
const fallbackController = new AbortController();
|
||||
const fallbackTimeout = setTimeout(() => fallbackController.abort(), 45_000);
|
||||
try {
|
||||
const fallbackResp = await ollamaFetch(`${ollamaUrl}/api/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: FALLBACK_MODEL,
|
||||
messages: [
|
||||
{ role: "system", content: persona.systemPrompt },
|
||||
{ role: "user", content: userMessage },
|
||||
],
|
||||
stream: true,
|
||||
options: { num_predict: estimateMaxTokens(userMessage, persona.maxTokens), num_ctx: estimateNumCtx(persona.systemPrompt, userMessage), num_batch: 512 },
|
||||
keep_alive: "30m",
|
||||
}),
|
||||
signal: fallbackController.signal,
|
||||
});
|
||||
if (!fallbackResp.ok) throw new Error(`Fallback returned ${fallbackResp.status}`);
|
||||
const reader = fallbackResp.body?.getReader();
|
||||
if (!reader) throw new Error("No fallback response body");
|
||||
const decoder = new TextDecoder();
|
||||
let fullText = "";
|
||||
let inThinking = false;
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
const chunk = decoder.decode(value, { stream: true });
|
||||
for (const line of chunk.split("\n").filter(Boolean)) {
|
||||
try {
|
||||
const parsed = JSON.parse(line) as { message?: { content?: string } };
|
||||
if (parsed.message?.content) {
|
||||
const c = parsed.message.content;
|
||||
fullText += c;
|
||||
if (c.includes("<think>")) inThinking = true;
|
||||
if (!inThinking) onChunk(c);
|
||||
if (c.includes("</think>")) inThinking = false;
|
||||
}
|
||||
} catch { /* partial JSON */ }
|
||||
}
|
||||
}
|
||||
const cleaned = fullText.replace(/<think>[\s\S]*?<\/think>\s*/g, "").trim();
|
||||
onDone(cleaned);
|
||||
return;
|
||||
} finally {
|
||||
clearTimeout(fallbackTimeout);
|
||||
}
|
||||
} catch { /* fallback also failed */ }
|
||||
}
|
||||
trackError("ollama", err, { persona: persona.nick, model: persona.model });
|
||||
onError(err instanceof Error ? err : new Error(String(err)));
|
||||
} finally {
|
||||
@@ -246,6 +235,12 @@ function stripThinking(text: string): string {
|
||||
return text.replace(/<think>[\s\S]*?<\/think>\s*/g, "").trim();
|
||||
}
|
||||
|
||||
function stripThinkingFromChunk(text: string): string {
|
||||
return text
|
||||
.replace(/<think>[\s\S]*?<\/think>/g, "")
|
||||
.replace(/<\/?think>/g, "");
|
||||
}
|
||||
|
||||
/** Clean persona response: strip thinking tokens, self-reference prefix, whitespace */
|
||||
export function cleanPersonaResponse(text: string, personaNick: string): string {
|
||||
let cleaned = stripThinking(text);
|
||||
@@ -260,7 +255,71 @@ export function cleanPersonaResponse(text: string, personaNick: string): string
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface OllamaToolCall {
|
||||
function: { name: string; arguments: Record<string, unknown> };
|
||||
id?: string;
|
||||
type?: "function";
|
||||
function: { name: string; arguments: Record<string, unknown> | string };
|
||||
}
|
||||
|
||||
function toRuntimeMessage(message: ChatMessage): Record<string, unknown> {
|
||||
const payload: Record<string, unknown> = {
|
||||
role: message.role,
|
||||
content: message.content,
|
||||
};
|
||||
if (message.tool_calls?.length) {
|
||||
payload.tool_calls = message.tool_calls.map((toolCall) => ({
|
||||
id: toolCall.id,
|
||||
type: toolCall.type || "function",
|
||||
function: {
|
||||
name: toolCall.function.name,
|
||||
arguments: typeof toolCall.function.arguments === "string"
|
||||
? toolCall.function.arguments
|
||||
: JSON.stringify(toolCall.function.arguments),
|
||||
},
|
||||
}));
|
||||
}
|
||||
if (message.role === "tool" && message.tool_call_id) {
|
||||
payload.tool_call_id = message.tool_call_id;
|
||||
}
|
||||
return payload;
|
||||
}
|
||||
|
||||
function parseToolArguments(value: Record<string, unknown> | string): Record<string, unknown> {
|
||||
if (typeof value !== "string") return value;
|
||||
try {
|
||||
const parsed = JSON.parse(value) as Record<string, unknown>;
|
||||
return parsed && typeof parsed === "object" ? parsed : {};
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
function parseStreamingPayload(raw: string): string | null {
|
||||
try {
|
||||
const parsed = JSON.parse(raw) as {
|
||||
choices?: [{ delta?: { content?: string } }];
|
||||
message?: { content?: string };
|
||||
};
|
||||
return parsed.choices?.[0]?.delta?.content ?? parsed.message?.content ?? null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function extractAssistantMessage(data: {
|
||||
choices?: [{
|
||||
message?: {
|
||||
role?: string;
|
||||
content?: string;
|
||||
tool_calls?: OllamaToolCall[];
|
||||
};
|
||||
}];
|
||||
message?: {
|
||||
role?: string;
|
||||
content?: string;
|
||||
tool_calls?: OllamaToolCall[];
|
||||
};
|
||||
}): { role?: string; content?: string; tool_calls?: OllamaToolCall[] } | undefined {
|
||||
return data.choices?.[0]?.message || data.message;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -302,7 +361,7 @@ export async function executeToolCall(
|
||||
: { type: prompt || "pink", duration };
|
||||
try {
|
||||
const resp = await fetch(`${AI_BRIDGE}${endpoint}`, {
|
||||
method: "POST", headers: { "Content-Type": "application/json" },
|
||||
method: "POST", headers: llmHeaders(),
|
||||
body: JSON.stringify(body), signal: AbortSignal.timeout(60_000),
|
||||
});
|
||||
return resp.ok ? `[Audio généré: ${type} ${duration}s]` : `[Erreur génération: HTTP ${resp.status}]`;
|
||||
@@ -314,7 +373,7 @@ export async function executeToolCall(
|
||||
const voice = String(args.voice || "af_heart");
|
||||
try {
|
||||
const resp = await fetch(`${AI_BRIDGE}/generate/voice-fast`, {
|
||||
method: "POST", headers: { "Content-Type": "application/json" },
|
||||
method: "POST", headers: llmHeaders(),
|
||||
body: JSON.stringify({ text, voice, speed: 1.0 }), signal: AbortSignal.timeout(30_000),
|
||||
});
|
||||
return resp.ok ? `[Voix synthétisée: ${voice}, "${text.slice(0, 50)}"]` : `[Erreur TTS: HTTP ${resp.status}]`;
|
||||
@@ -330,7 +389,7 @@ export async function executeToolCall(
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream Ollama chat with optional tool-calling support.
|
||||
* Stream local runtime chat with optional tool-calling support.
|
||||
* When tools are provided:
|
||||
* 1. First do a non-streaming call to see if Ollama wants to use tools
|
||||
* 2. If tool_calls present, execute them and re-call with tool results
|
||||
@@ -368,66 +427,54 @@ export async function streamOllamaChatWithTools(
|
||||
await ollamaLimit(async () => {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), 45_000);
|
||||
const runtimeUrl = ollamaUrl || LLM_URL;
|
||||
|
||||
try {
|
||||
const messages: Array<{ role: string; content: string; tool_calls?: OllamaToolCall[] }> = [
|
||||
const runtimeModel = resolveRuntimeModel(persona.model);
|
||||
const messages: ChatMessage[] = [
|
||||
{ role: "system", content: persona.systemPrompt },
|
||||
{ role: "user", content: userMessage },
|
||||
];
|
||||
|
||||
// Step 1: Non-streaming probe with tools (only for tool-like messages)
|
||||
const probeResp = await ollamaFetch(`${ollamaUrl}/api/chat`, {
|
||||
// Step 1: Non-streaming probe with tools
|
||||
const probeResp = await ollamaFetch(`${runtimeUrl}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
headers: llmHeaders(),
|
||||
body: JSON.stringify({
|
||||
model: persona.model,
|
||||
messages,
|
||||
model: runtimeModel,
|
||||
messages: messages.map(toRuntimeMessage),
|
||||
tools: tools.map(t => t),
|
||||
stream: false,
|
||||
options: { num_predict: estimateMaxTokens(userMessage, persona.maxTokens), num_ctx: estimateNumCtx(persona.systemPrompt, userMessage), num_batch: 512 }, keep_alive: "30m", think: shouldThink(userMessage, persona.model) ? undefined : false,
|
||||
max_tokens: estimateMaxTokens(userMessage, persona.maxTokens),
|
||||
}),
|
||||
signal: controller.signal,
|
||||
});
|
||||
|
||||
if (!probeResp.ok) {
|
||||
throw new Error(`Ollama returned ${probeResp.status}: ${probeResp.statusText}`);
|
||||
throw new Error(`vLLM returned ${probeResp.status}: ${probeResp.statusText}`);
|
||||
}
|
||||
|
||||
const probeData = await probeResp.json() as {
|
||||
choices?: [{
|
||||
message?: {
|
||||
role?: string;
|
||||
content?: string;
|
||||
tool_calls?: OllamaToolCall[];
|
||||
};
|
||||
}];
|
||||
message?: {
|
||||
role?: string;
|
||||
content?: string; thinking?: string;
|
||||
content?: string;
|
||||
tool_calls?: OllamaToolCall[];
|
||||
};
|
||||
};
|
||||
|
||||
const toolCalls = probeData.message?.tool_calls;
|
||||
const probeMsg = extractAssistantMessage(probeData);
|
||||
const toolCalls = probeMsg?.tool_calls;
|
||||
|
||||
// If no tool calls, use the response directly
|
||||
if (!toolCalls || toolCalls.length === 0) {
|
||||
let content = stripThinking(probeData.message?.content || "");
|
||||
// qwen3.5 thinking mode: content may be empty with reasoning in thinking field
|
||||
if (!content && probeData.message?.thinking) {
|
||||
const thinking = probeData.message.thinking;
|
||||
// Strip <think>...</think> tags first
|
||||
const stripped = thinking.replace(/<think>[\s\S]*?<\/think>\s*/g, "").trim();
|
||||
if (stripped) {
|
||||
content = stripped;
|
||||
} else {
|
||||
// Try to extract answer after markers (FR/EN)
|
||||
const answerMatch = thinking.match(/(?:Answer|Response|Réponse|Output|Conclusion)\s*:\s*([\s\S]+)$/i);
|
||||
if (answerMatch) {
|
||||
content = answerMatch[1].trim();
|
||||
} else {
|
||||
// Last resort: take last substantial paragraph
|
||||
const paragraphs = thinking.split("\n\n").filter(p => p.trim().length > 20);
|
||||
content = paragraphs[paragraphs.length - 1]?.trim() || thinking.trim();
|
||||
}
|
||||
}
|
||||
if (content) {
|
||||
logger.debug("[ollama] extracted content from thinking field (tool probe)");
|
||||
}
|
||||
}
|
||||
const content = stripThinking(probeMsg?.content || "");
|
||||
if (content) {
|
||||
onChunk(content);
|
||||
}
|
||||
@@ -438,13 +485,13 @@ export async function streamOllamaChatWithTools(
|
||||
// Step 2: Execute tool calls (max 1 round)
|
||||
messages.push({
|
||||
role: "assistant",
|
||||
content: probeData.message?.content || "",
|
||||
content: probeMsg?.content || "",
|
||||
tool_calls: toolCalls,
|
||||
});
|
||||
|
||||
for (const tc of toolCalls) {
|
||||
const name = tc.function.name;
|
||||
const args = tc.function.arguments;
|
||||
const args = parseToolArguments(tc.function.arguments);
|
||||
if (DEBUG) console.log(`[mcp-tools] ${persona.nick} calling ${name}(${JSON.stringify(args)})`);
|
||||
|
||||
let result: string;
|
||||
@@ -457,50 +504,53 @@ export async function streamOllamaChatWithTools(
|
||||
messages.push({
|
||||
role: "tool",
|
||||
content: result,
|
||||
tool_call_id: tc.id,
|
||||
});
|
||||
}
|
||||
|
||||
// Step 3: Stream the final response with tool context
|
||||
const streamResp = await ollamaFetch(`${ollamaUrl}/api/chat`, {
|
||||
const streamResp = await ollamaFetch(`${runtimeUrl}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
headers: llmHeaders(),
|
||||
body: JSON.stringify({
|
||||
model: persona.model,
|
||||
messages,
|
||||
model: runtimeModel,
|
||||
messages: messages.map(toRuntimeMessage),
|
||||
stream: true,
|
||||
options: { num_predict: estimateMaxTokens(userMessage, persona.maxTokens), num_ctx: estimateNumCtx(persona.systemPrompt, userMessage), num_batch: 512 }, keep_alive: "30m", think: false,
|
||||
max_tokens: estimateMaxTokens(userMessage, persona.maxTokens),
|
||||
}),
|
||||
signal: controller.signal,
|
||||
});
|
||||
|
||||
if (!streamResp.ok) {
|
||||
throw new Error(`Ollama returned ${streamResp.status}: ${streamResp.statusText}`);
|
||||
throw new Error(`vLLM returned ${streamResp.status}: ${streamResp.statusText}`);
|
||||
}
|
||||
|
||||
const reader = streamResp.body?.getReader();
|
||||
if (!reader) {
|
||||
throw new Error("No response body from Ollama");
|
||||
throw new Error("No response body from vLLM");
|
||||
}
|
||||
|
||||
const decoder = new TextDecoder();
|
||||
let fullText = "";
|
||||
let toolStreamBuf = "";
|
||||
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
|
||||
const chunk = decoder.decode(value, { stream: true });
|
||||
const lines = chunk.split("\n").filter(Boolean);
|
||||
toolStreamBuf += decoder.decode(value, { stream: true });
|
||||
const tsLines = toolStreamBuf.split("\n");
|
||||
toolStreamBuf = tsLines.pop() || "";
|
||||
|
||||
for (const line of lines) {
|
||||
try {
|
||||
const parsed = JSON.parse(line) as { message?: { content?: string }; done?: boolean };
|
||||
if (parsed.message?.content) {
|
||||
fullText += parsed.message.content;
|
||||
onChunk(parsed.message.content);
|
||||
}
|
||||
} catch {
|
||||
// Partial JSON -- skip
|
||||
for (const line of tsLines) {
|
||||
const raw = line.startsWith("data: ") ? line.slice(6).trim() : line.trim();
|
||||
if (!raw) continue;
|
||||
if (raw === "[DONE]") break;
|
||||
const c = parseStreamingPayload(raw);
|
||||
if (c) {
|
||||
fullText += c;
|
||||
const visible = stripThinkingFromChunk(c);
|
||||
if (visible) onChunk(visible);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -517,11 +567,11 @@ export async function streamOllamaChatWithTools(
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// LLM Client — mascarade-backed streaming (OpenAI-compatible)
|
||||
// Falls back to direct Ollama if mascarade is unavailable.
|
||||
// Uses mascarade only for explicit cloud-provider streaming.
|
||||
// Drop-in replacement for streamOllamaChat with same signature.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
import { streamChat as llmStreamChat, type ChatMessage } from "./llm-client.js";
|
||||
import { streamChat as llmStreamChat } from "./llm-client.js";
|
||||
|
||||
const USE_MASCARADE = process.env.USE_MASCARADE !== "0"; // enabled by default
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ import {
|
||||
resetPersonaMemory,
|
||||
savePersonaMemory,
|
||||
} from "./persona-memory-store.js";
|
||||
import { scheduler, VRAM_BUDGETS } from "./inference-scheduler.js";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Persona memory (persistent, file-based)
|
||||
@@ -27,32 +28,48 @@ export async function updatePersonaMemory(
|
||||
persona: ChatPersona,
|
||||
recentMessages: string[],
|
||||
ollamaUrl: string,
|
||||
userNick: string = "_anonymous",
|
||||
): Promise<void> {
|
||||
const startedAt = performance.now();
|
||||
const memory = await loadPersonaMemory(persona);
|
||||
const memory = await loadPersonaMemory(persona.id || persona.nick, userNick, persona.nick);
|
||||
const policy = resolvePersonaMemoryPolicy();
|
||||
const prompt = buildPersonaMemoryExtractionPrompt(persona, recentMessages, policy);
|
||||
recordPersonaMemoryAttempt(persona);
|
||||
|
||||
try {
|
||||
const response = await fetch(`${ollamaUrl}/api/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: persona.model,
|
||||
messages: [{ role: "user", content: prompt }],
|
||||
stream: false,
|
||||
format: "json",
|
||||
}),
|
||||
signal: AbortSignal.timeout(30_000),
|
||||
const llmUrl = process.env.LLM_URL || "http://localhost:11434";
|
||||
const llmModel = process.env.LLM_MODEL || "qwen-14b-awq";
|
||||
const llmApiKey = process.env.LLM_API_KEY || "";
|
||||
const llmH: Record<string, string> = { "Content-Type": "application/json" };
|
||||
if (llmApiKey) llmH["Authorization"] = `Bearer ${llmApiKey}`;
|
||||
const data = await scheduler.submit<{ choices?: [{ message?: { content?: string } }] }>({
|
||||
id: `memory-extract-${persona.id}-${Date.now()}`,
|
||||
device: "gpu",
|
||||
priority: "low",
|
||||
label: `memory-extract:${persona.id}`,
|
||||
vramMB: VRAM_BUDGETS.ollama,
|
||||
execute: async () => {
|
||||
const response = await fetch(`${llmUrl}/v1/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: llmH,
|
||||
body: JSON.stringify({
|
||||
model: llmModel,
|
||||
messages: [{ role: "user", content: prompt }],
|
||||
stream: false,
|
||||
max_tokens: 800,
|
||||
response_format: { type: "json_object" },
|
||||
}),
|
||||
signal: AbortSignal.timeout(30_000),
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Memory update HTTP ${response.status}`);
|
||||
}
|
||||
|
||||
return response.json() as Promise<{ choices?: [{ message?: { content?: string } }] }>;
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Memory update HTTP ${response.status}`);
|
||||
}
|
||||
|
||||
const data = (await response.json()) as { message?: { content?: string } };
|
||||
const rawContent = String(data.message?.content || "").trim();
|
||||
const rawContent = String(data.choices?.[0]?.message?.content || "").trim();
|
||||
if (!rawContent) {
|
||||
logger.error({ nick: persona.nick }, "[persona-router] Empty LLM JSON");
|
||||
recordPersonaMemorySkip(persona, "empty_response");
|
||||
@@ -74,7 +91,7 @@ export async function updatePersonaMemory(
|
||||
recentMessages,
|
||||
});
|
||||
|
||||
await savePersonaMemory(updated, policy);
|
||||
await savePersonaMemory(updated, userNick, policy);
|
||||
const durationMs = performance.now() - startedAt;
|
||||
recordLatency("persona_memory_update", durationMs);
|
||||
recordPersonaMemoryWrite(persona, memory, updated, durationMs);
|
||||
|
||||
+28
-6
@@ -6,13 +6,15 @@
|
||||
# docker compose --profile v1 up -d # V1 app + postgres
|
||||
# docker compose --profile v2 up -d # V2 api + worker + postgres
|
||||
# docker compose --profile v1 --profile v2 up -d # V1 + V2 + postgres
|
||||
# docker compose --profile ollama up -d # include Ollama container
|
||||
# docker compose --profile ollama up -d # include Ollama-compatible embeddings backend
|
||||
#
|
||||
# By default, Ollama is expected to run natively on the host (port 11434).
|
||||
# Set OLLAMA_URL in .env to override. Use --profile ollama to run it in Docker.
|
||||
# By default, the primary LLM runtime is expected on the host (LLM_URL, port 11434).
|
||||
# OLLAMA_URL is retained for embeddings/RAG compatibility.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
x-common-env: &common-env
|
||||
LLM_URL: "${LLM_URL:-http://host.docker.internal:11434}"
|
||||
LLM_MODEL: "${LLM_MODEL:-qwen-14b-awq}"
|
||||
OLLAMA_URL: "${OLLAMA_URL:-http://host.docker.internal:11434}"
|
||||
DATABASE_URL: postgres://kxkm:kxkm@postgres:5432/kxkm_clown
|
||||
NODE_ENV: production
|
||||
@@ -68,6 +70,8 @@ services:
|
||||
- ./scripts:/app/scripts:ro
|
||||
- /home/kxkm/openDAW/packages/app/studio/dist:/app/daw:ro
|
||||
environment:
|
||||
LLM_URL: "${LLM_URL:-http://localhost:11434}"
|
||||
LLM_MODEL: "${LLM_MODEL:-qwen-14b-awq}"
|
||||
OLLAMA_URL: "http://localhost:11434"
|
||||
DATABASE_URL: "postgres://kxkm:kxkm@localhost:5432/kxkm_clown"
|
||||
NODE_ENV: production
|
||||
@@ -112,6 +116,8 @@ services:
|
||||
network_mode: host
|
||||
command: ["node", "apps/worker/dist/index.js"]
|
||||
environment:
|
||||
LLM_URL: "${LLM_URL:-http://localhost:11434}"
|
||||
LLM_MODEL: "${LLM_MODEL:-qwen-14b-awq}"
|
||||
OLLAMA_URL: "http://localhost:11434"
|
||||
DATABASE_URL: "postgres://kxkm:kxkm@localhost:5432/kxkm_clown"
|
||||
NODE_ENV: production
|
||||
@@ -153,7 +159,7 @@ services:
|
||||
start_period: 5s
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Ollama — LLM inference server (optional, use --profile ollama)
|
||||
# Ollama-compatible embeddings backend (optional, use --profile ollama)
|
||||
# -------------------------------------------------------------------------
|
||||
ollama:
|
||||
image: ollama/ollama:latest
|
||||
@@ -304,7 +310,7 @@ services:
|
||||
start_period: 120s
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# LightRAG — Graph RAG server (Ollama backend)
|
||||
# LightRAG — Graph RAG server
|
||||
# API: POST /query, POST /documents/text, GET /health
|
||||
# Web UI: http://localhost:9621
|
||||
# -------------------------------------------------------------------------
|
||||
@@ -323,7 +329,7 @@ services:
|
||||
LLM_MODEL: "qwen3:8b"
|
||||
EMBEDDING_MODEL: "nomic-embed-text"
|
||||
EMBEDDING_DIM: "768"
|
||||
OLLAMA_HOST: "http://localhost:11434"
|
||||
OLLAMA_HOST: "${OLLAMA_URL:-http://localhost:11434}"
|
||||
LLM_BINDING: ollama
|
||||
EMBEDDING_BINDING: ollama
|
||||
RAG_DIR: "/data/lightrag"
|
||||
@@ -383,9 +389,25 @@ services:
|
||||
depends_on:
|
||||
- prometheus
|
||||
|
||||
# --- Text Embeddings Inference (dedicated embedding server) -----------------
|
||||
# Usage: docker compose --profile embeddings up -d
|
||||
# Exposes OpenAI-compatible /v1/embeddings on port 9500
|
||||
# -------------------------------------------------------------------------
|
||||
tei:
|
||||
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.6
|
||||
container_name: kxkm-tei
|
||||
restart: unless-stopped
|
||||
profiles: [v2, embeddings]
|
||||
ports:
|
||||
- "9500:80"
|
||||
volumes:
|
||||
- tei-models:/data
|
||||
command: --model-id BAAI/bge-m3 --port 80
|
||||
|
||||
volumes:
|
||||
app-data:
|
||||
pg-data:
|
||||
ollama-data:
|
||||
tei-models:
|
||||
prometheus-data:
|
||||
grafana-data:
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
# AGENTS.md — packages/
|
||||
|
||||
<!-- Parent: ../AGENTS.md -->
|
||||
|
||||
8 shared packages in npm workspace. Exported as `@kxkm/*` scope.
|
||||
|
||||
## core — Shared Types & Constants (2 TS files)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `index.ts` | Persona IDs, channel constants, permission levels, errors |
|
||||
| `index.test.ts` | Type checks, constant validation |
|
||||
|
||||
Core IDs: 33 personas (Pharmacius, Sherlock, Turing, Ikeda, Schaeffer, Merzbow, Pina, etc.). Permissions: read, write, admin.
|
||||
|
||||
**Used by**: all packages and apps.
|
||||
|
||||
## auth — Authentication & RBAC (2 TS files)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `index.ts` | RBAC middleware, session validation, JWT verify, guest mode |
|
||||
| `index.test.ts` | Auth flow, token expiry, guest access |
|
||||
|
||||
Session storage: PostgreSQL (sessionRepo). Guest mode: read-only routes. RBAC: 3 levels (guest, user, admin).
|
||||
|
||||
**Used by**: apps/api (middleware), apps/web (session hooks).
|
||||
|
||||
## chat-domain — Message & Command Types (2 TS files)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `index.ts` | ChatMessage union, Channel, Command registry, slash command definitions |
|
||||
| `index.test.ts` | Message validation, command parsing |
|
||||
|
||||
43 slash commands: /chat, /imagine, /compose, /help, /speed, etc. Message types: text, image, audio, error, system.
|
||||
|
||||
**Used by**: apps/api (ws-chat), apps/web (UI).
|
||||
|
||||
## persona-domain — Persona Definitions & Memory (4 TS files)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `index.ts` | Persona model, memoryMode (auto/explicit/off), corpus[], relations[], DPO pair definitions |
|
||||
| `editorial.ts` | Editorial pipeline: persona proposal, feedback collection, DPO training triggers |
|
||||
| `pharmacius.ts` | Pharmacius persona specialization (meta-reflection, composability) |
|
||||
| `index.test.ts` | Persona validation, memory mode tests |
|
||||
|
||||
33 personas with per-persona: memory mode, corpus URLs, related personas (depth-3 relay), voice sample. DPO pair collection: user feedback → training pipeline → Unsloth fine-tuning.
|
||||
|
||||
**Used by**: apps/api (ws-chat, persona-runtime), packages/node-engine (training).
|
||||
|
||||
## node-engine — DAG Execution & Job Queue (6 TS files)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `index.ts` | Node registry, DAG validator, run executor |
|
||||
| `registry.ts` | 15+ node types: text_generation, image_generation, music_generation, audio_effects, voice_clone, document_extraction, sql_query, etc. |
|
||||
| `sandbox.ts` | Isolated node execution, timeout, resource limits |
|
||||
| `training.ts` | Training job: DPO pair collection, Unsloth adapter, registry update |
|
||||
| `registry.test.ts` | Node type validation |
|
||||
| `index.test.ts` | DAG execution, queue, run state machine |
|
||||
|
||||
GPU-aware queue: submits to `inference-scheduler.ts` for LLM nodes. Training nodes: trigger via worker.
|
||||
|
||||
**Used by**: apps/api (routes/node-engine), apps/worker (job executor), packages/storage (run persistence).
|
||||
|
||||
## storage — PostgreSQL Persistence (8 TS files)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `index.ts` | Repo factory, migration runner |
|
||||
| `config.test.ts` | DB connection, pool, transaction tests |
|
||||
| `migration.test.ts` | Schema versioning |
|
||||
| `session-repo.test.ts` | Session CRUD, expiry cleanup |
|
||||
| `persona-repo.test.ts` | Persona memory, feedback store |
|
||||
| `node-engine-repo.test.ts` | Run state, output, logs |
|
||||
| `test-helpers.ts` | Test DB setup, fixtures |
|
||||
|
||||
Repos: SessionRepo (auth), PersonaRepo (memory, DPO feedback), NodeEngineRepo (DAG runs, training jobs). Migrations: auto-run on startup.
|
||||
|
||||
**Used by**: apps/api (all routes), apps/worker (job persistence).
|
||||
|
||||
## tui — Terminal UI Utilities (3 TS files)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `index.ts` | CLI argument parsing, color codes, progress bars, table format |
|
||||
| `index.test.ts` | Formatter tests |
|
||||
|
||||
Used by scripts: health-check.sh, deep-audit.js, ops-tui.sh. Outputs JSON + ANSI colors for logs.
|
||||
|
||||
**Used by**: scripts/, ops/ (monitoring).
|
||||
|
||||
## ui — Experimental React Components (2 TS files)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `index.ts` | Button, Input, Select, Modal, Spinner components (unstyled, Tailwind-ready) |
|
||||
| `index.test.ts` | Component render tests |
|
||||
|
||||
Minimal, reusable. Not currently used in apps/web (which has inline components).
|
||||
|
||||
## Package Dependencies
|
||||
|
||||
```
|
||||
core (no deps, baseline types)
|
||||
auth → core
|
||||
chat-domain → core
|
||||
persona-domain → core, chat-domain
|
||||
node-engine → core, persona-domain
|
||||
storage → core, auth, chat-domain, persona-domain, node-engine
|
||||
tui → core
|
||||
ui → core
|
||||
```
|
||||
|
||||
## Build & Test
|
||||
|
||||
```bash
|
||||
npm run -w @kxkm/core build
|
||||
npm run -w @kxkm/auth test
|
||||
npm run -w @kxkm/chat-domain test
|
||||
npm run -w @kxkm/persona-domain test
|
||||
npm run -w @kxkm/node-engine test
|
||||
npm run -w @kxkm/storage test
|
||||
npm run -w @kxkm/tui test
|
||||
npm run -w @kxkm/ui test
|
||||
```
|
||||
|
||||
## Export Pattern
|
||||
|
||||
Each package exports:
|
||||
- TypeScript types (.ts)
|
||||
- CommonJS build (dist/index.js)
|
||||
- Type declarations (dist/index.d.ts)
|
||||
|
||||
```typescript
|
||||
import { Persona, PersonaMemoryMode } from '@kxkm/persona-domain';
|
||||
import { Node, DAGRun } from '@kxkm/node-engine';
|
||||
import { Session } from '@kxkm/auth';
|
||||
```
|
||||
@@ -0,0 +1,148 @@
|
||||
# AGENTS.md — scripts/
|
||||
|
||||
<!-- Parent: ../AGENTS.md -->
|
||||
|
||||
42 files (20 Python + 22 Shell) organized by domain: ops, training, ingestion, voice, image, deployment.
|
||||
|
||||
## Ops & Monitoring (8 files)
|
||||
|
||||
| File | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `health-check.sh` | Shell | 19 checks (API, DB, Ollama, TTS, LightRAG, docker) with color output |
|
||||
| `deep-audit.js` | Node | Comprehensive audit: hot spots, error rates, perf p50/p95/p99, GPU VRAM, memory leaks |
|
||||
| `ops-tui.sh` | Shell | Interactive ops menu (deploy, restart services, logs, cleanup) |
|
||||
| `service-status.sh` | Shell | systemd service status (kxkm-tts, kxkm-lightrag, kxkm-api) |
|
||||
| `health-doc-search.sh` | Shell | LightRAG embedding health check |
|
||||
| `health-embeddings.sh` | Shell | Ollama embedding model health |
|
||||
| `health-voice-clone.sh` | Shell | TTS voice clone model check |
|
||||
| `journald-monitor.sh` | Shell | Tail systemd logs with filtering (error, warning) |
|
||||
|
||||
## Deployment (5 files)
|
||||
|
||||
| File | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `deploy.sh` | Shell | Systemd service deployment (kxkm-api, kxkm-tts, kxkm-lightrag), auto-restart on fail |
|
||||
| `rollback-v2.js` | Node | Rollback to previous version (git revert + service restart) |
|
||||
| `setup-voice-clone.sh` | Shell | Download XTTS-v2 model, configure TTS |
|
||||
| `qwen3-tts-ondemand.sh` | Shell | Start on-demand TTS server (not systemd) |
|
||||
| `ollama-import-adapter.sh` | Shell | Import fine-tuned models into Ollama, register in registry.json |
|
||||
|
||||
## Training (7 files)
|
||||
|
||||
| File | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `train_unsloth.py` | Python | Unsloth fine-tuning pipeline: SFT or DPO, load persona-specific dataset, save to models/finetuned/ |
|
||||
| `eval_model.py` | Python | Evaluate model: perplexity, BLEU, custom benchmarks, compare against baseline |
|
||||
| `dpo-pipeline.js` | Node | DPO pair collection automation: fetch feedback from DB, format pairs, trigger training |
|
||||
| `dpo-export.sh` | Shell | Export DPO pairs to JSONL for analysis |
|
||||
| `orchestrate_batches.py` | Python | Batch orchestration: schedule training jobs across GPU, queue management |
|
||||
| `migrate-persona-store-v2.js` | Node | Data migration: v1 persona memory → v2 nick-isolated structure |
|
||||
| `parity-check.js` | Node | V1 ↔ V2 behavior parity validation |
|
||||
|
||||
## Ingestion & RAG (5 files)
|
||||
|
||||
| File | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `ingest_spectacle_corpus.py` | Python | LightRAG corpus ingestion: seed URLs + SearXNG discovery + sherlock-discovered-urls.jsonl, Camoufox for bot-protected sites |
|
||||
| `extract_pdf_docling.py` | Python | Extract text/tables from PDFs via Docling, chunk for embedding |
|
||||
| `reranker-server.py` | Python | BGE-M3 reranker service (listen :8090) for LightRAG results |
|
||||
| `generate_image.py` | Python | ComfyUI image generation wrapper (checkpoint + LoRA selection, queue submission) |
|
||||
| `generate-persona-dialogues.js` | Node | Generate synthetic persona dialogues for DPO training |
|
||||
|
||||
## Voice & Audio (5 files)
|
||||
|
||||
| File | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `tts-server.py` | Python | Dual TTS backend (Piper + Chatterbox) on :9100, HTTP API |
|
||||
| `qwen3-tts-server.py` | Python | Qwen3 TTS server alternative (on-demand) |
|
||||
| `tts_synthesize.py` | Python | Batch TTS synthesis: text → WAV, save to media-store |
|
||||
| `tts_clone_voice.py` | Python | Voice cloning: sample ingestion, XTTS-v2 fine-tune, register voice |
|
||||
| `generate-voice-samples.js` | Node | Generate persona voice samples (read persona bio via TTS) |
|
||||
|
||||
## Testing & Validation (5 files)
|
||||
|
||||
| File | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `run-playwright-e2e.sh` | Shell | Run Playwright E2E tests (login, chat, upload, admin) |
|
||||
| `test-e2e.js` | Node | E2E test runner (WebDriver + assertions) |
|
||||
| `test-v2.js` | Node | V2 integration tests (API + WS) |
|
||||
| `smoke-test.sh` | Shell | Quick smoke: API health, DB connect, Ollama respond |
|
||||
| `smoke-v2.js` | Node | V2 smoke: WS chat, inference, persona memory |
|
||||
|
||||
## Utilities (5 files)
|
||||
|
||||
| File | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `chat-pause.sh` | Shell | Pause chat (set CHAT_PAUSED env var or create data/chat-paused file) |
|
||||
| `cleanup-logs.sh` | Shell | Rotate + compress logs older than 7 days, purge after 30 days |
|
||||
| `cleanup-test-compositions.js` | Node | Delete test composition artifacts |
|
||||
| `ollama-warmup.sh` | Shell | Pre-load LLM models into Ollama memory (avoid cold start) |
|
||||
| `sudo-optimize.sh` | Shell | System optimization (disable swap, tune kernel params) |
|
||||
|
||||
## Specialized (2 files)
|
||||
|
||||
| File | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `transcribe_audio.py` | Python | Batch audio → text via OpenAI Whisper or ElevenLabs Scribe |
|
||||
| `xtts_clone.py` | Python | XTTS-v2 voice cloning (wrapper) |
|
||||
|
||||
## Integration & Agent Scripts (4 files)
|
||||
|
||||
| File | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `v2-agent-task.js` | Node | Execute agent task from PLAN.md (sync, run, log) |
|
||||
| `v2-autoresearch-loop.js` | Node | Continuous autoresearch: fetch OSS projects, benchmark, report |
|
||||
| `v2-dpo-pipeline.js` | Node | Automated DPO: feedback → pairs → training trigger → registry update |
|
||||
| `discord-pharmacius.js` | Node | Discord bot integration (Pharmacius agent) |
|
||||
|
||||
## Data Scripts (1 file)
|
||||
|
||||
| File | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `patch-plan.py` | Python | Update PLAN.md lot state (experimental, use git instead) |
|
||||
|
||||
## Pattern: npm run scripts
|
||||
|
||||
All scripts in `package.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"health:v2": "bash scripts/health-check.sh",
|
||||
"audit:deep": "node scripts/deep-audit.js",
|
||||
"deploy": "bash scripts/deploy.sh",
|
||||
"train": "python scripts/train_unsloth.py",
|
||||
"test:e2e": "bash scripts/run-playwright-e2e.sh"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Run Examples
|
||||
|
||||
```bash
|
||||
npm run health:v2 # Run health checks
|
||||
node scripts/deep-audit.js # Deep system audit
|
||||
bash scripts/deploy.sh # Deploy systemd services
|
||||
python scripts/train_unsloth.py --dpo --model qwen-14b # Train
|
||||
bash scripts/run-playwright-e2e.sh # E2E tests
|
||||
bash scripts/health-check.sh # 19-point health check
|
||||
```
|
||||
|
||||
## Environment Variables (scripts expect)
|
||||
|
||||
```bash
|
||||
LLM_URL=http://localhost:11434
|
||||
DATABASE_URL=postgres://kxkm:kxkm@localhost:5432/kxkm_clown
|
||||
SEARXNG_URL=http://localhost:8080
|
||||
LIGHTRAG_URL=http://localhost:9621
|
||||
TTS_URL=http://localhost:9100
|
||||
CAMOUFOX_URL=http://localhost:8091
|
||||
DISCORD_TOKEN=xxx # For discord-pharmacius.js
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Exit codes**: 0 = success, 1 = error, 2 = skipped
|
||||
- **Output**: JSON for machine parsing (health-check.sh, deep-audit.js), human-readable tables for TUI
|
||||
- **Logging**: scripts/logs/*.log (rotated daily by cleanup-logs.sh)
|
||||
- **Data**: ephemeral files in data/ (sherlock-discovered-urls.jsonl, chat-logs/, etc.)
|
||||
Reference in New Issue
Block a user