docs: SPEC_COMPOSE_ADVANCED, PLAN lots 178-200, FEATURE_MAP, AGENTS, README
Composition pipeline plan: timeline, TTS mix, effects, DAW 21 planned lots (180-200) in 4 phases 43 commands, 13 services, 35 styles, TTFC 284ms Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Codex Local
@@ -0,0 +1,242 @@
|
||||
# Agents, Sous-agents, Competences
|
||||
|
||||
> "L'infrastructure est une decision politique deployee." -- electron rare
|
||||
|
||||
## Orchestration
|
||||
|
||||
- Agent racine: **Coordinateur** — planifie, arbitre, synchronise PLAN/TODO/docs
|
||||
- Sous-agents specialises: analyse code, veille OSS, audit securite, optimisation
|
||||
- Cadence: synchroniser PLAN.md + TODO.md + docs apres chaque lot
|
||||
|
||||
## Matrice des agents (lot 17+)
|
||||
|
||||
| Agent | Competences | Perimetre | Etat |
|
||||
|---|---|---|---|
|
||||
| Coordinateur | planification, arbitrage, docs de pilotage | PLAN.md, TODO.md, AGENTS.md, README.md | actif |
|
||||
| Securite | validation input, hardening, rate-limit, RBAC | apps/api, ws-chat, packages/auth | veille |
|
||||
| Backend API | Express, WS, Ollama, RAG, multimodal pipeline | apps/api/src/ | actif |
|
||||
| Node Engine | DAG, queue, runs, sandbox, training adapters | packages/node-engine, apps/worker | actif |
|
||||
| Personas | source/feedback/proposals/pharmacius, memoire | packages/persona-domain, ws-chat | actif |
|
||||
| Frontend | React/Vite, UX Minitel, React Flow, chat, voice | apps/web/src/ | actif |
|
||||
| Ops/TUI | scripts, logs, rotate/purge, health, audit | ops/v2/, scripts/ | actif |
|
||||
| Training | DPO, SFT, Unsloth, eval, autoresearch, Ollama import | scripts/, packages/node-engine | actif |
|
||||
| Multimodal | STT, TTS, vision, PDF, RAG, recherche web | apps/api/src/ws-chat.ts | actif |
|
||||
| Veille OSS | recherche projets, libs, modeles, benchmarks | docs/OSS_WATCH, docs/HF_MODEL_RESEARCH | periodique |
|
||||
|
||||
## Sous-agents et skill routing
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Coord[Coordinateur]
|
||||
|
||||
Coord --> SecAgent[Securite]
|
||||
Coord --> BackAgent[Backend API]
|
||||
Coord --> EngAgent[Node Engine]
|
||||
Coord --> PersAgent[Personas]
|
||||
Coord --> FrontAgent[Frontend]
|
||||
Coord --> OpsAgent[Ops/TUI]
|
||||
Coord --> TrainAgent[Training]
|
||||
Coord --> MultiAgent[Multimodal]
|
||||
Coord --> OSSAgent[Veille OSS]
|
||||
|
||||
SecAgent --> |audit| SecScan[Pattern scan P0/P1/P2]
|
||||
SecAgent --> |fix| SecFix[Correctifs chirurgicaux]
|
||||
|
||||
BackAgent --> |analyse| APIAudit[Deep analyse app.ts, ws-chat.ts]
|
||||
BackAgent --> |refactor| APISplit[Extraction modules]
|
||||
|
||||
EngAgent --> |test| EngTest[Tests unitaires node-engine]
|
||||
EngAgent --> |extend| EngNew[Nouveaux node types]
|
||||
|
||||
PersAgent --> |pipeline| PersPipe[Editorial pipeline]
|
||||
PersAgent --> |finetune| PersDPO[DPO + PCL methodology]
|
||||
|
||||
FrontAgent --> |ui| FrontUI[Minitel theme CSS]
|
||||
FrontAgent --> |perf| FrontPerf[Memoization, lazy load]
|
||||
|
||||
OpsAgent --> |monitor| OpsHealth[health-check, deep-audit]
|
||||
OpsAgent --> |deploy| OpsDeploy[Docker, kxkm-ai]
|
||||
|
||||
TrainAgent --> |train| TrainRun[Unsloth/TRL runs]
|
||||
TrainAgent --> |eval| TrainEval[Scoring, registry]
|
||||
|
||||
MultiAgent --> |voice| MultiVoice[XTTS-v2, WebRTC]
|
||||
MultiAgent --> |search| MultiSearch[SearXNG]
|
||||
|
||||
OSSAgent --> |web| OSSWeb[Recherche web]
|
||||
OSSAgent --> |hf| OSSHF[HuggingFace models]
|
||||
```
|
||||
|
||||
## Todo agents (lot 17+ — mis a jour 2026-03-20)
|
||||
|
||||
### Coordinateur
|
||||
|
||||
- [x] Consolider PLAN.md avec etat reel (lots 0-94 complets)
|
||||
- [x] Synchroniser FEATURE_MAP.md matrice
|
||||
- [x] Mettre a jour TODO.md avec backlog Phase session 2026-03-19/20
|
||||
- [x] Documenter actions dans ops/v2/logs/
|
||||
- [ ] lot-95: Coordonner E2E Playwright test plan
|
||||
- [ ] lot-100: Design public demo mode access control
|
||||
|
||||
### Backend API
|
||||
|
||||
- [x] Extraire app-bootstrap.ts et app-middleware.ts de app.ts
|
||||
- [x] Extraire ws-conversation-router.ts de ws-chat.ts
|
||||
- [x] ws-chat.ts modularized (425 to 335 LOC, 3 modules extracted)
|
||||
- [x] app.ts extraction (540 to 131 LOC, create-repos.ts extracted)
|
||||
- [x] Zod validation on all 19 API route schemas
|
||||
- [x] Error telemetry (16 labels)
|
||||
- [x] Perf instrumentation (6 labels, p50/p95/p99), TTFC 284ms
|
||||
- [x] Smart routing (5 topic domains)
|
||||
- [x] Dynamic context window (4k-32k)
|
||||
- [x] NLP auto-detect generation intent (compose vs imagine)
|
||||
- [x] /speed command for latency diagnostics
|
||||
- [ ] lot-178: ACE-Step API direct integration (duration fix)
|
||||
- [ ] lot-180: Timeline data model
|
||||
- [ ] lot-97: Multi-channel support (create/join channels)
|
||||
- [ ] lot-100: Public demo mode read-only routes
|
||||
|
||||
### Node Engine
|
||||
|
||||
- [x] Extraire registry.ts du hotspot node-engine
|
||||
- [ ] Ajouter node type `music_generation` (ACE-Step 1.5)
|
||||
- [ ] Ajouter node type `voice_clone` (Chatterbox)
|
||||
- [ ] Ajouter node type `audio_mix` (multi-track composition)
|
||||
- [ ] Ajouter node type `audio_effects` (FX chain)
|
||||
- [ ] lot-96: Automated DPO pipeline (feedback → pairs → training trigger)
|
||||
|
||||
### Multimodal (composition pipeline)
|
||||
|
||||
- [x] 35 music styles ACE-Step
|
||||
- [x] ComfyUI smart checkpoint selection (32 checkpoints + 24 LoRAs)
|
||||
- [ ] lot-178: ACE-Step API direct (duration fix)
|
||||
- [ ] lot-181: TTS voiceover mix into timeline
|
||||
- [ ] lot-182: Audio effects pipeline (reverb, delay, EQ, compression)
|
||||
- [ ] lot-183: DAW export (stems, markers, project file)
|
||||
- [ ] lot-187: Auto-mastering (loudness normalization, limiting)
|
||||
- [ ] lot-188: Stem separation (vocals, drums, bass, other)
|
||||
|
||||
### Personas
|
||||
|
||||
- [ ] Evaluer PCL (Persona-Aware Contrastive Learning) pour coherence
|
||||
- [ ] Evaluer OpenCharacter pour generation profils synthetiques
|
||||
- [x] Ajouter `/compose` command (generation musicale)
|
||||
|
||||
### Frontend
|
||||
|
||||
- [x] Implementer lot 16 UI Minitel rose (phosphore, VIDEOTEX)
|
||||
- [x] VoiceChat push-to-talk + level meter + silence auto
|
||||
- [x] Player audio + viewer image plein ecran
|
||||
- [x] Mediatheque gallery/playlist
|
||||
- [x] Progress bars animees Compose/Imagine
|
||||
- [x] React.memo + useCallback on ChatSidebar, ChatInput, ChatHistory
|
||||
- [x] 17 lazy-loaded routes (-53% initial JS)
|
||||
- [x] CRT CSS-only effect (scanlines, vignette, phosphor glow, boot 0.8s)
|
||||
- [x] Chat virtualization (react-window, variable row heights)
|
||||
- [x] Markdown rendering (marked + DOMPurify)
|
||||
- [x] CRT boot animation (modem dial, scanline reveal)
|
||||
- [x] 5 CSS themes (minitel, crt, hacker, synthwave, default)
|
||||
- [x] Mobile responsive pass (touch, bottom nav, viewport units)
|
||||
- [x] Guest mode read-only UI
|
||||
- [ ] lot-185: Composition timeline UI (waveform view, track lanes)
|
||||
- [ ] lot-194: FX rack UI (drag-drop effects chain)
|
||||
- [ ] lot-95: E2E Playwright tests (login, chat, upload, admin)
|
||||
- [ ] lot-98: File sharing UI (upload → gallery)
|
||||
|
||||
### Ops/TUI
|
||||
|
||||
- [x] Deployer deep-audit.js sur kxkm-ai
|
||||
- [x] Ajouter SearXNG au docker-compose
|
||||
- [x] TTS sidecar HTTP (tts-server.py :9100, dual Chatterbox/Piper)
|
||||
- [x] deploy.sh migrated tmux → systemd
|
||||
- [x] Systemd services (kxkm-tts + kxkm-lightrag, auto-restart)
|
||||
- [x] health-check.sh TUI (19 checks)
|
||||
- [x] Docker compose 12 services with health checks
|
||||
- [ ] Fix Docker transformers (rebuild propre avec torch)
|
||||
|
||||
### Training
|
||||
|
||||
- [x] Spike BGE-M3 (resultat negatif sur Apple/Metal, baseline maintenue)
|
||||
- [x] TTS dual backend Chatterbox/Piper valide
|
||||
- [x] Tool-calling benchmark (llama3.1 vs qwen3 vs mistral)
|
||||
- [x] Sherlock migrated to llama3.1:8b-instruct-q4_0
|
||||
- [ ] lot-96: Persona DPO automation pipeline
|
||||
- [ ] Tester ACE-Step 1.5 sur RTX 4090
|
||||
|
||||
### Veille OSS
|
||||
|
||||
- [x] Veille mars 2026 complete (40+ projets analyses, top 10 recommandations)
|
||||
- [ ] Suivre LLMRTC (WebRTC voice TypeScript)
|
||||
- [ ] Suivre A2A Protocol (interop agents)
|
||||
- [ ] Suivre MCP SDK updates
|
||||
- [ ] Evaluer Kokoro TTS (82M params, ultra-leger)
|
||||
|
||||
## Pipeline d'intervention
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Analyse: agent lance
|
||||
Analyse --> Findings: scan code/docs/web
|
||||
Findings --> Triage: P0/P1/P2 classification
|
||||
Triage --> Fix_P0: P0 critique
|
||||
Triage --> Plan_P1: P1 important
|
||||
Triage --> Backlog_P2: P2 mineur
|
||||
Fix_P0 --> Test: correction chirurgicale
|
||||
Plan_P1 --> Test: correction planifiee
|
||||
Test --> Deploy: tests OK
|
||||
Deploy --> Log: log + purge
|
||||
Log --> [*]: cycle termine
|
||||
Backlog_P2 --> [*]: ajoute au TODO
|
||||
```
|
||||
|
||||
## Affectations en cours (2026-03-20)
|
||||
|
||||
### Mission globale
|
||||
- Deep analyse continue du code, optimisation chirurgicale, et synchronisation documentaire apres chaque lot.
|
||||
- Priorite execution: P1 fiabilite, puis dette perf/complexite, puis features lot 18-19.
|
||||
|
||||
### Assignations agents -> sous-agents -> competences
|
||||
|
||||
| Agent | Sous-agent | Competences principales | Taches assignees immediates |
|
||||
|---|---|---|---|
|
||||
| Coordinateur | Planner/Docs | triage, synchronisation, runbook | Maintenir PLAN/TODO, chainer les lots, tracer actions |
|
||||
| Backend API | WS/HTTP surgeon | websocket, express, validation input | Extraire `ws-chat.ts` en modules, reduire logs, limiter hot paths |
|
||||
| Node Engine | DAG runtime | graph validation, queue, state machine | Ajouter nodes `music_generation`, `voice_clone`, `document_extraction` |
|
||||
| Ops/TUI | Audit operator | TUI scripts, logs, rotation, cron | Rendre `deep-audit` zero faux positif critique, pipeline logs + purge |
|
||||
| Veille OSS | Scout | benchmark OSS, licences, interop | Evaluer Open WebUI, LibreChat, LangGraph, SearXNG, Docling |
|
||||
|
||||
### Workflow d'enchainement
|
||||
1. Executer audit + tests
|
||||
2. Corriger de maniere chirurgicale
|
||||
3. Re-executer audit + tests
|
||||
4. Mettre a jour docs de pilotage
|
||||
5. Alimenter TODO suivant avec ordre d'execution
|
||||
|
||||
### Regles d'operation
|
||||
- Interroger le user uniquement en cas de blocage reel (acces, choix irreversibles, secrets).
|
||||
- Privilegier TUI et scripts avec logs lisibles, puis purge des logs obsoletes.
|
||||
- Conserver la V1 comme reference comportementale, V2 comme cible active.
|
||||
|
||||
### Etat de cycle (2026-03-20 18:00)
|
||||
|
||||
- 130+ lots termines (lot-24 a lot-177).
|
||||
- 425 tests, 0 failures.
|
||||
- 13 services en production.
|
||||
- 43 chat commands, 33 personas.
|
||||
- 35 music styles (ACE-Step), 5 CSS themes.
|
||||
- 32 ComfyUI checkpoints + 24 LoRAs, smart selection NLP.
|
||||
- TTFC 284ms.
|
||||
- Guest mode, mobile responsive.
|
||||
- Structured logging complet (pino JSON, 0 console.log).
|
||||
- Systemd services (TTS + LightRAG).
|
||||
- Frontend: lazy routes (-53%), React.memo, CRT boot, chat virtualization.
|
||||
|
||||
### Prochains lots (178-200) — Composition Pipeline
|
||||
|
||||
1. lot-178: Compose duration fix (ACE-Step API direct)
|
||||
2. lot-179: SPEC_COMPOSE_ADVANCED plan
|
||||
3. lot-180: Timeline data model (tracks, clips, markers)
|
||||
4. lot-181: TTS voiceover mix into timeline
|
||||
5. lot-182: Audio effects pipeline (reverb, delay, EQ)
|
||||
6. lot-183: DAW export (stems, markers, project)
|
||||
7. lot-184-200: Multi-track, arrangement, mastering, stem separation, MIDI, templates, collab, lyrics, FX rack, automation, samples, spectral view, history, render queue, sharing
|
||||
+315
@@ -0,0 +1,315 @@
|
||||
# Carte Fonctionnelle
|
||||
|
||||
> "Nous sommes les saboteurs du big daddy mainframe." -- VNS Matrix, detourne par electron rare
|
||||
>
|
||||
> Ce document cartographie un systeme de chat IA multimodal local --
|
||||
> IRC dans l'ame, crypto-anarchiste dans l'infrastructure, musique concrete dans le traitement du signal.
|
||||
|
||||
## Carte synthetique
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Product[KXKM_Clown]
|
||||
Product --> Chat[Chat Multimodal]
|
||||
Product --> Admin[Admin]
|
||||
Product --> Personas[Personas]
|
||||
Product --> Engine[Node Engine]
|
||||
Product --> Multimodal[Multimodal Pipeline]
|
||||
Product --> Training[Training & DPO]
|
||||
Product --> Ops[Ops]
|
||||
Product --> Data[Donnees]
|
||||
Product --> Security[Securite]
|
||||
|
||||
Chat --> Channels[Canaux]
|
||||
Chat --> Sessions[Sessions]
|
||||
Chat --> Attachments[Pieces jointes]
|
||||
Chat --> Routing[Routing IA]
|
||||
Chat --> Streaming[Streaming LLM]
|
||||
Chat --> WebSearch[Recherche web]
|
||||
Chat --> Memory[Memoire persona]
|
||||
Chat --> History[Historique chat]
|
||||
|
||||
Admin --> Auth[Session admin]
|
||||
Admin --> Modules[Modules switcher]
|
||||
Admin --> Status[Status strip]
|
||||
Admin --> Runtime[Runtime]
|
||||
Admin --> Exports[Exports]
|
||||
|
||||
Personas --> Sources[Sources web]
|
||||
Personas --> Feedback[Feedback]
|
||||
Personas --> Proposals[Proposals]
|
||||
Personas --> Pharmacius[Pharmacius]
|
||||
Personas --> Drawflow[React Flow nodal]
|
||||
|
||||
Engine --> Graphs[Graphes DAG]
|
||||
Engine --> Runs[Runs lifecycle]
|
||||
Engine --> Artifacts[Artifacts]
|
||||
Engine --> Models[Registry modeles]
|
||||
Engine --> Queue[Queue async]
|
||||
|
||||
Multimodal --> RAG[RAG local]
|
||||
Multimodal --> STT[STT / Whisper]
|
||||
Multimodal --> TTS[TTS / Piper]
|
||||
Multimodal --> Vision[Vision / minicpm-v]
|
||||
Multimodal --> PDF[Extraction PDF]
|
||||
|
||||
Training --> DPO[Pipeline DPO]
|
||||
Training --> TrainDash[Training dashboard]
|
||||
Training --> Autoresearch[Autoresearch]
|
||||
Training --> OllamaImport[Ollama import adapter]
|
||||
|
||||
Ops --> Check[check / smoke]
|
||||
Ops --> Build[build V1+V2]
|
||||
Ops --> Batch[Batch orchestrator]
|
||||
Ops --> Retention[Retention sweep]
|
||||
|
||||
Data --> Persist[JSON/JSONL flat-file + Postgres]
|
||||
Data --> DPOData[DPO pairs]
|
||||
Data --> Export[HTML / training export]
|
||||
Data --> ConvMemory[Conversation memory]
|
||||
|
||||
Security --> NetGate[Subnet gate]
|
||||
Security --> Cookie[HttpOnly cookies]
|
||||
Security --> RBAC[RBAC V2]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Chat
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| WebSocket temps reel | Connexion persistante `/ws`, events bidirectionnels JSON |
|
||||
| Multi-canaux | `#general`, canaux par persona, routage automatique |
|
||||
| Streaming LLM | Reponses en streaming via Ollama avec indicateur d'ecriture |
|
||||
| Tab completion | Completion nicks et commandes via Tab |
|
||||
| Historique messages | ArrowUp/Down, 100 items en memoire client |
|
||||
| DOM pruning | Elagage automatique a 500 messages max dans le DOM |
|
||||
| Couleurs par bot | Mapping couleur unique par persona (palette deterministe) |
|
||||
| Messages prives (PM) | Conversation directe utilisateur-persona via @mention |
|
||||
| Upload fichiers | Pipeline multimodal: texte, image, audio, PDF |
|
||||
| Commandes slash | `/help`, `/web`, `/nick`, `/who`, `/personas`, `/clear` |
|
||||
| Recherche web | `/web <query>` — DuckDuckGo Lite ou API custom |
|
||||
| Memoire persona | Contexte persistant par persona (faits, resume) |
|
||||
| Historique chat | Logs JSONL par jour, API de consultation paginee |
|
||||
|
||||
## Pipeline Multimodal
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| RAG local | Embeddings via Ollama (`nomic-embed-text`), cosine similarity, contexte manifeste injecte |
|
||||
| STT (Speech-to-Text) | `faster-whisper` (CTranslate2, int8) ou fallback `openai-whisper`, modeles tiny/base/small/medium/large |
|
||||
| TTS (Text-to-Speech) | Dual backend: `piper-tts` (CPU) + `Chatterbox` (GPU, emotion tags, MIT) via TTS sidecar HTTP :9100 |
|
||||
| Vision | `qwen3-vl:8b` via Ollama, analyse d'images uploadees, description en francais |
|
||||
| VoiceChat | Push-to-talk (maintenir), level meter temps reel, silence auto 2s, 26 voix piper |
|
||||
| Mediatheque | Gallery images + playlist audio, persistance disk, API REST |
|
||||
| Extraction PDF | Docling (tables, layout, OCR) + fallback PyMuPDF |
|
||||
| Texte/CSV/JSON | Lecture directe des fichiers texte, CSV, JSONL uploades |
|
||||
| Generation musicale | `/compose` — ACE-Step 1.5 (GPU, <4GB VRAM) + fallback MusicGen |
|
||||
| Generation images | `/imagine` — ComfyUI (SDXL Lightning + Flux 2 support) |
|
||||
| Recherche web | SearXNG self-hosted + DuckDuckGo fallback |
|
||||
|
||||
## Discord
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| Bot texte Pharmacius | Bridge 2 salons Discord ↔ KXKM chat (WebSocket + Discord Gateway, sans deps) |
|
||||
| Bot vocal | STT → personas → TTS en salon vocal Discord (@discordjs/voice) |
|
||||
| Commandes | `!help`, `!personas`, `!status`, `@PersonaName message` |
|
||||
|
||||
## MCP (Model Context Protocol)
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| MCP Server | stdio transport, protocole 2024-11-05, compatible Claude Desktop |
|
||||
| kxkm_chat | Envoyer un message aux personas |
|
||||
| kxkm_personas | Lister les personas actives |
|
||||
| kxkm_web_search | Recherche web via SearXNG |
|
||||
| kxkm_status | Statut systeme (health + perf) |
|
||||
|
||||
## Admin Dashboard
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| Auth session | Cookie HttpOnly + fallback header legacy |
|
||||
| Module switcher | dashboard, personas, runtime, channels, data, node-engine |
|
||||
| Status strip | Connexion, clients connectes, sessions, personas actives, modeles |
|
||||
| Endpoint status public | `/api/v2/status` sans authentification |
|
||||
| Gestion runtime | Demarrage/arret personas, overrides modele |
|
||||
|
||||
## Personas
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| Personas seed | Schaeffer, Batty, Radigue, Oliveros, Lessig, etc. (catalogue initial) |
|
||||
| Creation custom | Nouvelle persona depuis source editoriale |
|
||||
| Overrides runtime | Nom, modele, style modifiables a chaud |
|
||||
| Gestion sources | Subject, query, tone, themes, lexicon par persona |
|
||||
| Pipeline feedback | Votes, signaux, edits utilisateur |
|
||||
| Pharmacius | Orchestrateur editorial automatique |
|
||||
| Systeme proposals | Suggestions d'amelioration auto-generees |
|
||||
| Apply/revert | Application et annulation des proposals |
|
||||
| Visualisation React Flow | Interface nodale pour le graphe de persona |
|
||||
| Enable/disable | Activation/desactivation a chaud |
|
||||
| Memoire persistante | Faits et resume par persona, mis a jour toutes les 5 interactions |
|
||||
|
||||
## Node Engine
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| Graphes DAG | Definition par noeuds et aretes, validation acyclique |
|
||||
| 7 familles de noeuds | dataset_source, data_processing, dataset_builder, training, evaluation, registry, deployment |
|
||||
| 16+ types de noeuds | Types specialises par famille |
|
||||
| Cycle de vie run | queued -> running -> completed / failed / cancelled |
|
||||
| Queue async | Concurrence controlee, execution asynchrone (Postgres) |
|
||||
| Artifacts par etape | Tracking et stockage d'artifacts a chaque step |
|
||||
| Registry modeles | Versionnage et catalogue des modeles produits |
|
||||
| Templates seed | Graphes pre-configures comme point de depart |
|
||||
| Recovery on crash | Reprise des runs interrompus au redemarrage |
|
||||
| Cancel support | Annulation propre d'un run en cours |
|
||||
|
||||
## Training & DPO
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| Pipeline DPO | Extraction paires chosen/rejected depuis feedback personas, export JSONL |
|
||||
| Training adapters | TRL (Hugging Face) + Unsloth, execution via Python venv |
|
||||
| Autoresearch | Boucle d'experimentation automatisee: mutations, scoring, keep/discard |
|
||||
| Evaluation | Score artifacts (accuracy, f1, bleu, perplexity), scoring deterministe |
|
||||
| Registry | Enregistrement automatique du meilleur modele par session |
|
||||
| Ollama import adapter | Import LoRA adapter dans Ollama via Modelfile (bash + node wrapper) |
|
||||
| Training dashboard | React Flow dans le frontend V2, visualisation graphes et runs |
|
||||
| Sandboxing runtimes | local_cpu, local_gpu, cloud_api |
|
||||
|
||||
## Stockage & Donnees
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| Persistance double | Postgres (production) ou flat-file JSON/JSONL (dev/demo) |
|
||||
| Stats utilisateur | Compteurs et metriques par utilisateur |
|
||||
| Memoire conversation | Contexte borne par persona/session |
|
||||
| Memoire persona | Faits et resume persistants par persona (`data/persona-memory/`) |
|
||||
| Logging DPO | Paires chosen/rejected pour entrainement RLHF |
|
||||
| Chat logs | JSONL quotidien (`data/chat-logs/v2-YYYY-MM-DD.jsonl`) |
|
||||
| Export training data | Extraction formatee pour fine-tuning |
|
||||
| Recherche historique | Recherche dans les conversations passees |
|
||||
| Export HTML | Export conversations en HTML lisible |
|
||||
| Snapshots session | Capture d'etat de session a un instant T |
|
||||
| Retention sweep | Nettoyage automatique sessions et logs anciens |
|
||||
|
||||
## Ops & Scripts
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| `npm run check` | Validation syntaxe V1 + TypeScript V2 |
|
||||
| `npm run smoke` | 30+ tests d'integration automatises |
|
||||
| `npm run build` | Build dist V1 + compilation V2 |
|
||||
| `npm run v2:autoresearch` | Boucle autoresearch continue |
|
||||
| Batch orchestrator | DAG Python pour orchestration de taches batch |
|
||||
| TUI monitoring | health-check, queue-viewer, persona-manager, log-rotate |
|
||||
|
||||
## Securite
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| Subnet gate | Restriction d'acces admin par sous-reseau (`ADMIN_SUBNET`, `ADMIN_ALLOWED_SUBNETS`) |
|
||||
| Cookies HttpOnly | Sessions admin non accessibles depuis JS client |
|
||||
| Same-origin check | Verification origine sur les mutations |
|
||||
| RBAC (V2) | Roles admin, editor, operator, viewer |
|
||||
|
||||
## Fonctionnalites session 2026-03-20 (lots 174-177)
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| **35 music styles** | ACE-Step: ambient, techno, house, drum-n-bass, dubstep, trance, IDM, noise, drone, classical, jazz, blues, rock, metal, hip-hop, trap, lo-fi, chillwave, synthwave, vaporwave, industrial, EBM, darkwave, post-punk, shoegaze, krautrock, musique concrete, electroacoustique, minimal, glitch, breakcore, gabber, footwork, grime, UK garage |
|
||||
| **5 CSS themes** | minitel (phosphore vert), crt (amber), hacker (matrix green), synthwave (neon pink/blue), default (neutral) |
|
||||
| **ComfyUI smart selection** | 32 checkpoints + 24 LoRAs, auto-selection based on prompt NLP analysis (style, subject, medium) |
|
||||
| **NLP auto-detect generation** | Automatic detection of generation intent from natural language (compose vs imagine routing) |
|
||||
| **Guest mode** | Read-only access without login, rate-limited, no admin functions |
|
||||
| **Mobile responsive** | Touch gestures, bottom nav, viewport units, responsive layout pass |
|
||||
|
||||
## Fonctionnalites session 2026-03-19/20
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| **43 chat commands** | /help, /nick, /who, /personas, /web, /clear, /status, /model, /persona, /reload, /export, /compose, /imagine, /voice, /memory, /context, /rag, /stats, /uptime, /changelog, /version, /dice, /roll, /flip, /ban, /unban, /mute, /unmute, /whisper, /w, /history, /search, /react, /invite, /time, /date, /session, /speed, + more |
|
||||
| **Markdown rendering** | marked + DOMPurify, bold, italic, code blocks, links, lists in chat messages |
|
||||
| **Smart routing** | 5 topic domains (music, philosophy, tech, arts, science), keyword + embedding scoring |
|
||||
| **Dynamic context window** | Adaptive 4k-32k tokens based on conversation length and complexity |
|
||||
| **Perf instrumentation** | 6 labels (http, ollama_ttfb, ollama_total, rag_search, rag_rerank, ws_message), p50/p95/p99 endpoint |
|
||||
| **Error telemetry** | 16 error labels (validation, auth, ws, ollama, rag, tts, comfy, vision, stt, pdf, upload, memory, context, persona, mcp, search) |
|
||||
| **Zod validation** | 19 schemas covering personas, sessions, node-engine, chat, media, admin, analytics routes |
|
||||
| **MIME magic bytes** | file-type library magic bytes detection, SAFE_MIMES allowlist, blocks disguised executables |
|
||||
| **WS reconnect + seq numbers** | Auto-reconnect (1s-30s exponential backoff), seq numbers for gap detection, missed message replay |
|
||||
| **CRT boot animation** | Modem dial sound, progressive scanline reveal, phosphor glow warmup, 0.8s boot sequence |
|
||||
|
||||
---
|
||||
|
||||
## Matrice de statut
|
||||
|
||||
| Fonctionnalite | V1 | V2 | Priorite |
|
||||
| --- | --- | --- | --- |
|
||||
| **Chat** | | | |
|
||||
| WebSocket temps reel | OK | OK | haute |
|
||||
| Multi-canaux | OK | OK | haute |
|
||||
| Streaming LLM | OK | OK | haute |
|
||||
| Tab completion | OK | OK | moyenne |
|
||||
| Historique messages | OK | OK | basse |
|
||||
| DOM pruning | OK | OK | basse |
|
||||
| Upload fichiers | OK | OK | moyenne |
|
||||
| Commandes slash | OK | OK | moyenne |
|
||||
| Recherche web (`/web`) | OK | OK | moyenne |
|
||||
| **Multimodal** | | | |
|
||||
| RAG (embeddings locaux) | -- | OK | haute |
|
||||
| STT (faster-whisper) | -- | OK | haute |
|
||||
| TTS (piper + Chatterbox) | -- | OK | moyenne |
|
||||
| VoiceChat push-to-talk | -- | OK | haute |
|
||||
| Mediatheque gallery/playlist | -- | OK | moyenne |
|
||||
| Vision (qwen3-vl:8b) | -- | OK | haute |
|
||||
| Extraction PDF | -- | OK | moyenne |
|
||||
| Memoire persona | -- | OK | haute |
|
||||
| **Admin** | | | |
|
||||
| Auth session cookie | OK | OK | haute |
|
||||
| Module switcher | OK | OK | haute |
|
||||
| Status strip | OK | OK | moyenne |
|
||||
| Status public | OK | OK | basse |
|
||||
| **Personas** | | | |
|
||||
| Personas seed | OK | OK | haute |
|
||||
| Creation custom | OK | OK | haute |
|
||||
| Overrides runtime | OK | OK | haute |
|
||||
| Pipeline feedback | OK | OK | haute |
|
||||
| Pharmacius | OK | OK | haute |
|
||||
| Proposals apply/revert | OK | OK | haute |
|
||||
| Drawflow nodal | OK | OK (React Flow) | moyenne |
|
||||
| **Node Engine** | | | |
|
||||
| Graphes DAG | OK | OK | critique |
|
||||
| Run lifecycle | OK | OK | critique |
|
||||
| Queue async | OK | OK (Postgres) | critique |
|
||||
| Artifacts | OK | OK | critique |
|
||||
| Registry modeles | OK | OK | critique |
|
||||
| Recovery on crash | OK | OK | haute |
|
||||
| Cancel support | OK | OK | haute |
|
||||
| **Training & DPO** | | | |
|
||||
| DPO export (JSONL) | OK | OK | haute |
|
||||
| Training adapters (TRL/Unsloth) | -- | OK | haute |
|
||||
| Autoresearch loop | -- | OK | haute |
|
||||
| Ollama import adapter | -- | OK | haute |
|
||||
| Training dashboard (React) | -- | OK | haute |
|
||||
| Sandboxing runtimes | -- | OK | haute |
|
||||
| **Stockage** | | | |
|
||||
| Flat-file JSON/JSONL | OK | OK (+ Postgres) | haute |
|
||||
| Memoire conversation | OK | OK | haute |
|
||||
| Chat history (JSONL logs) | -- | OK | haute |
|
||||
| Export HTML | OK | OK | basse |
|
||||
| Retention sweep | OK | OK | moyenne |
|
||||
| **Securite** | | | |
|
||||
| Subnet gate | OK | OK | haute |
|
||||
| Cookies HttpOnly | OK | OK | haute |
|
||||
| RBAC roles | -- | OK | haute |
|
||||
| **Tests** | | | |
|
||||
| Backend unit tests | -- | OK (425) | haute |
|
||||
| React component tests | -- | OK | haute |
|
||||
| Smoke tests | OK | OK | haute |
|
||||
| UI Minitel VIDEOTEX | -- | OK | haute |
|
||||
| Deploy tmux (deploy.sh) | -- | OK | moyenne |
|
||||
| CI/CD GitHub Actions | -- | OK | haute |
|
||||
@@ -1,15 +1,17 @@
|
||||
# PLAN.md — KXKM_Clown
|
||||
|
||||
Updated: 2026-03-20T09:00:00Z
|
||||
Updated: 2026-03-20T18:00:00Z
|
||||
|
||||
## Summary
|
||||
|
||||
- **120 lots executed** (24-143) in session 2026-03-19/20
|
||||
- **130+ lots executed** (24-177) in session 2026-03-19/20
|
||||
- **425 tests**, 0 fail
|
||||
- **40 commands**, 12 services, 9 spec docs
|
||||
- **43 commands**, 13 services, 9 spec docs
|
||||
- **All SEC-01-05 resolved**
|
||||
- Ollama v0.18.2, qwen3.5:9b (256K ctx, adaptive thinking)
|
||||
- Next: lots 144+ (E2E Playwright, DPO automation, mobile responsive)
|
||||
- 35 music styles, 5 CSS themes, 32 ComfyUI checkpoints + 24 LoRAs
|
||||
- TTFC 284ms
|
||||
- Next: lots 178+ (Compose pipeline, DAW integration, composition advanced)
|
||||
|
||||
---
|
||||
|
||||
@@ -592,68 +594,141 @@ Updated: 2026-03-20T12:00:00Z
|
||||
## lot-149-file-sharing [planned]
|
||||
- Description: Upload files visible in MediaExplorer gallery
|
||||
|
||||
## lot-150-reload-markdown [done]
|
||||
- Summary: /reload, markdown rendering, /help + tab-complete
|
||||
## lot-174-perf-ttfc [done]
|
||||
- Description: Performance optimization, TTFC 284ms
|
||||
- Summary: Time-to-first-chunk optimized to 284ms, connection pooling, lazy service init.
|
||||
|
||||
## lot-151-export-memory-models [done]
|
||||
- Summary: /export, /memory, /models commands
|
||||
## lot-175-speed-command [done]
|
||||
- Description: /speed command for latency diagnostics
|
||||
- Summary: /speed returns TTFC, p50/p95 latencies, Ollama/RAG/TTS response times.
|
||||
|
||||
## lot-152-comfyui-integration [done]
|
||||
- Summary: ComfyUI :8188 live, systemd unit
|
||||
## lot-176-auto-gen-fix [done]
|
||||
- Description: Auto-generation fix (NLP detect, ComfyUI smart selection)
|
||||
- Summary: NLP auto-detect generation intent (/compose vs /imagine), ComfyUI smart checkpoint selection based on prompt analysis.
|
||||
|
||||
## lot-153-rag-perf [done]
|
||||
- Summary: RAG timeout 2s, TTFC 284ms (27x faster)
|
||||
## lot-177-music-styles [done]
|
||||
- Description: 35 music styles + 5 CSS themes + guest mode + mobile responsive
|
||||
- Summary: 35 ACE-Step music styles, 5 CSS themes (minitel/crt/hacker/synthwave/default), guest mode read-only, mobile responsive pass.
|
||||
|
||||
## lot-154-progress-bars [done]
|
||||
- Summary: Broadcast progress for /imagine and /compose
|
||||
## lot-178-compose-duration-fix [planned]
|
||||
- Description: Compose duration fix (ACE-Step API direct integration)
|
||||
- Owner: Multimodal
|
||||
- Priority: P1
|
||||
- Tasks:
|
||||
- [ ] Direct ACE-Step API call (bypass shell script)
|
||||
- [ ] Duration parameter passthrough (5-300s)
|
||||
- [ ] Progress callback from ACE-Step process
|
||||
|
||||
## lot-155-comfyui-smart-models [done]
|
||||
- Summary: Smart model selection, 32 checkpoints, 24 LoRAs
|
||||
## lot-179-spec-compose-advanced [planned]
|
||||
- Description: SPEC_COMPOSE_ADVANCED plan (multi-track, effects, mastering)
|
||||
- Owner: Coordinateur
|
||||
- Priority: P1
|
||||
- Tasks:
|
||||
- [ ] Write SPEC_COMPOSE_ADVANCED.md
|
||||
- [ ] Define timeline data model
|
||||
- [ ] Define mix/master pipeline stages
|
||||
|
||||
## lot-156-imagine-end2end [done]
|
||||
- Summary: /imagine end-to-end verified
|
||||
## lot-180-timeline-model [planned]
|
||||
- Description: Composition timeline data model (tracks, clips, markers)
|
||||
- Owner: Backend API
|
||||
- Priority: P2
|
||||
|
||||
## lot-157-compose-end2end [done]
|
||||
- Summary: /compose end-to-end verified
|
||||
## lot-181-tts-mix [planned]
|
||||
- Description: TTS voiceover mix into composition timeline
|
||||
- Owner: Multimodal
|
||||
- Priority: P2
|
||||
|
||||
## lot-158-e2e-tests [done]
|
||||
- Summary: E2E 10/10 pass
|
||||
## lot-182-audio-effects [planned]
|
||||
- Description: Audio effects pipeline (reverb, delay, EQ, compression)
|
||||
- Owner: Multimodal
|
||||
- Priority: P2
|
||||
|
||||
## lot-159-45-commands [done]
|
||||
- Summary: 45 commands, tab-complete
|
||||
## lot-183-daw-export [planned]
|
||||
- Description: DAW export (stems, markers, project file)
|
||||
- Owner: Multimodal
|
||||
- Priority: P3
|
||||
|
||||
## lot-160-nlp-autodetect [done]
|
||||
- Summary: NL auto-triggers /imagine + /compose
|
||||
## lot-184-multi-track [planned]
|
||||
- Description: Multi-track composition (layer multiple ACE-Step generations)
|
||||
- Owner: Multimodal
|
||||
- Priority: P2
|
||||
|
||||
## lot-161-admin [done]
|
||||
- Summary: Admin panel, services, users
|
||||
## lot-185-composition-ui [planned]
|
||||
- Description: Composition timeline UI (waveform view, track lanes)
|
||||
- Owner: Frontend
|
||||
- Priority: P2
|
||||
|
||||
## lot-162-media-explorer [done]
|
||||
- Summary: Gallery for generated images/audio
|
||||
## lot-186-arrangement [planned]
|
||||
- Description: Arrangement tools (intro/verse/chorus structure)
|
||||
- Owner: Backend API + Frontend
|
||||
- Priority: P2
|
||||
|
||||
## lot-163-persona-voices [done]
|
||||
- Summary: Per-persona voice config
|
||||
## lot-187-mastering [planned]
|
||||
- Description: Auto-mastering pipeline (loudness normalization, limiting)
|
||||
- Owner: Multimodal
|
||||
- Priority: P3
|
||||
|
||||
## lot-164-context-stats [done]
|
||||
- Summary: /context stats
|
||||
## lot-188-stem-separation [planned]
|
||||
- Description: Stem separation (vocals, drums, bass, other)
|
||||
- Owner: Multimodal
|
||||
- Priority: P3
|
||||
|
||||
## lot-165-security [done]
|
||||
- Summary: SEC-01 to SEC-03
|
||||
## lot-189-midi-export [planned]
|
||||
- Description: MIDI export from generated compositions
|
||||
- Owner: Multimodal
|
||||
- Priority: P3
|
||||
|
||||
## lot-166-docs [done]
|
||||
- Summary: Full spec docs
|
||||
## lot-190-composition-templates [planned]
|
||||
- Description: Composition templates (ambient, techno, orchestral presets)
|
||||
- Owner: Backend API
|
||||
- Priority: P2
|
||||
|
||||
## lot-167-oss-veille [done]
|
||||
- Summary: Qwen3-TTS, NexusRAG research
|
||||
## lot-191-collab-composition [planned]
|
||||
- Description: Collaborative composition (multi-user timeline editing)
|
||||
- Owner: Backend API + Frontend
|
||||
- Priority: P3
|
||||
|
||||
## lot-168-nlp-verified [done]
|
||||
- Summary: NLP auto-detect E2E verified
|
||||
## lot-192-lyrics-gen [planned]
|
||||
- Description: Lyrics generation (LLM-driven, synced to composition)
|
||||
- Owner: Backend API
|
||||
- Priority: P3
|
||||
|
||||
## lot-169-plan-update [done]
|
||||
- Summary: PLAN.md lots 150-169
|
||||
## lot-193-voice-composition [planned]
|
||||
- Description: Voice-driven composition (hum/sing to generate)
|
||||
- Owner: Multimodal
|
||||
- Priority: P3
|
||||
|
||||
## lot-170-final-health [done]
|
||||
- Summary: 8/8 services, 45 cmds, TTFC 284ms
|
||||
## lot-194-fx-rack [planned]
|
||||
- Description: FX rack UI (drag-drop effects chain)
|
||||
- Owner: Frontend
|
||||
- Priority: P3
|
||||
|
||||
## lot-171-final [done]
|
||||
- Summary: Final session commit
|
||||
## lot-195-automation [planned]
|
||||
- Description: Parameter automation (volume, pan, effects over time)
|
||||
- Owner: Frontend + Backend API
|
||||
- Priority: P3
|
||||
|
||||
## lot-196-sample-library [planned]
|
||||
- Description: Sample library integration (one-shots, loops, foley)
|
||||
- Owner: Multimodal
|
||||
- Priority: P3
|
||||
|
||||
## lot-197-spectral-view [planned]
|
||||
- Description: Spectral analysis view for compositions
|
||||
- Owner: Frontend
|
||||
- Priority: P3
|
||||
|
||||
## lot-198-composition-history [planned]
|
||||
- Description: Composition version history (undo/redo, snapshots)
|
||||
- Owner: Backend API
|
||||
- Priority: P3
|
||||
|
||||
## lot-199-render-queue [planned]
|
||||
- Description: Render queue (batch export, format options)
|
||||
- Owner: Backend API + Multimodal
|
||||
- Priority: P3
|
||||
|
||||
## lot-200-composition-sharing [planned]
|
||||
- Description: Composition sharing (public links, embeds)
|
||||
- Owner: Backend API + Frontend
|
||||
- Priority: P3
|
||||
|
||||
@@ -43,7 +43,7 @@ docker compose --profile v2 --profile ollama up -d
|
||||
|
||||
Par defaut, Ollama est attendu en natif sur le host (port 11434).
|
||||
|
||||
## Services (12)
|
||||
## Services (13)
|
||||
|
||||
| Service | Port | Description |
|
||||
| --- | --- | --- |
|
||||
@@ -54,9 +54,10 @@ Par defaut, Ollama est attendu en natif sur le host (port 11434).
|
||||
| PostgreSQL | 5432 | Persistence (personas, runs, logs) |
|
||||
| SearXNG | 8080 | Recherche web self-hosted |
|
||||
| TTS Sidecar | 9100 | Piper + Chatterbox (dual backend) |
|
||||
| Qwen3-TTS | 9300 | Qwen3-TTS 0.6B CustomVoice (9 speakers) |
|
||||
| Reranker | 8787 | BGE/Jina reranking |
|
||||
| Docling | 5001 | Extraction PDF (tables, OCR) |
|
||||
| ComfyUI | 8188 | Generation images (SDXL + Flux 2) |
|
||||
| ComfyUI | 8188 | Generation images (32 checkpoints + 24 LoRAs) |
|
||||
| Worker | --- | Node Engine job processor (GPU) |
|
||||
| Discord Bot | --- | Pharmacius bridge (2 salons) |
|
||||
|
||||
@@ -72,18 +73,21 @@ Par defaut, Ollama est attendu en natif sur le host (port 11434).
|
||||
- **TTS** — Piper-tts + Chatterbox (dual backend via TTS sidecar HTTP :9100)
|
||||
- **PDF** — Extraction via Docling/PyMuPDF (tables, layout, OCR)
|
||||
- **Recherche web** — SearXNG self-hosted + DuckDuckGo fallback
|
||||
- **Generation musicale** — `/compose` via ACE-Step 1.5 / MusicGen
|
||||
- **Generation images** — `/imagine` via ComfyUI (SDXL Lightning + Flux 2)
|
||||
- **Generation musicale** — `/compose` via ACE-Step 1.5 / MusicGen, 35 music styles
|
||||
- **Generation images** — `/imagine` via ComfyUI (32 checkpoints + 24 LoRAs, smart NLP selection)
|
||||
- **Memoire persona** — Faits et resume persistants, compaction LLM auto (750 MB)
|
||||
- **Inter-persona** — @mention directe, dialogue depth 3
|
||||
- **19 commandes slash** — /help, /nick, /who, /personas, /web, /clear, /status, /compose, /imagine, /voice, /memory, /context, /rag, /stats, /uptime, /model, /persona, /reload, /export
|
||||
- **43 commandes slash** — /help, /nick, /who, /personas, /web, /clear, /status, /compose, /imagine, /voice, /memory, /context, /rag, /stats, /uptime, /model, /persona, /reload, /export, /changelog, /version, /dice, /roll, /flip, /ban, /unban, /mute, /unmute, /whisper, /history, /search, /react, /invite, /time, /date, /session, /speed, + more
|
||||
- **Markdown chat** — Rendu Markdown (marked + DOMPurify) dans les messages
|
||||
- **Smart routing** — 5 domaines thematiques (musique, philosophie, tech, arts, science)
|
||||
- **Validation Zod** — 19 schemas sur toutes les routes API
|
||||
- **Pino logging** — Logs structures JSON, rotation automatique
|
||||
- **Dynamic ctx** — Contexte LLM adaptatif 4k-32k selon la conversation
|
||||
- **Perf instrumentation** — 6 labels (http, ollama, rag, ws), p50/p95/p99
|
||||
- **Perf instrumentation** — 6 labels (http, ollama, rag, ws), p50/p95/p99, TTFC 284ms
|
||||
- **Error telemetry** — 16 labels d'erreur, logging structure
|
||||
- **5 CSS themes** — minitel (phosphore), crt (amber), hacker (matrix), synthwave (neon), default
|
||||
- **Guest mode** — Acces lecture seule sans login, rate-limited
|
||||
- **Mobile responsive** — Touch gestures, bottom nav, viewport units
|
||||
- **MIME magic bytes** — Validation magic bytes sur uploads, allowlist SAFE_MIMES
|
||||
- **WS reconnect** — Reconnexion auto (backoff 1s-30s), seq numbers, detection de gaps
|
||||
- **CRT effect** — Phosphore vert, scanlines, flicker, boot animation modem
|
||||
@@ -141,7 +145,7 @@ Par defaut, Ollama est attendu en natif sur le host (port 11434).
|
||||
| `PYTHON_BIN` | `python3` | Python avec libs ML (PyTorch, faster-whisper, piper-tts) |
|
||||
| `SCRIPTS_DIR` | `./scripts` | Chemin vers les scripts Python (TTS, STT, training) |
|
||||
|
||||
## Commandes slash (19)
|
||||
## Commandes slash (43)
|
||||
|
||||
| Commande | Description | Admin |
|
||||
| --- | --- | --- |
|
||||
@@ -152,18 +156,36 @@ Par defaut, Ollama est attendu en natif sur le host (port 11434).
|
||||
| `/web <query>` | Recherche web + commentaire personas | non |
|
||||
| `/clear` | Effacer le chat | non |
|
||||
| `/status` | Statut systeme | non |
|
||||
| `/compose <prompt>` | Generation musicale (ACE-Step) | non |
|
||||
| `/imagine <prompt>` | Generation image (ComfyUI) | non |
|
||||
| `/compose <prompt>` | Generation musicale (ACE-Step, 35 styles) | non |
|
||||
| `/imagine <prompt>` | Generation image (ComfyUI, smart selection) | non |
|
||||
| `/voice` | Toggle TTS voix | non |
|
||||
| `/memory` | Afficher memoire persona | non |
|
||||
| `/context` | Afficher contexte conversation | non |
|
||||
| `/rag <query>` | Recherche RAG directe | non |
|
||||
| `/stats` | Statistiques chat | non |
|
||||
| `/uptime` | Temps de fonctionnement | non |
|
||||
| `/changelog` | Historique git recent | non |
|
||||
| `/version` | Version et info app | non |
|
||||
| `/dice NdS` | Lancer des des | non |
|
||||
| `/roll` | Lancer 1d6 | non |
|
||||
| `/flip` | Pile ou face | non |
|
||||
| `/whisper <persona>` | Message prive a une persona | non |
|
||||
| `/history N` | Historique N derniers messages | non |
|
||||
| `/search <mot>` | Recherche dans le contexte | non |
|
||||
| `/react <emoji>` | Reaction emoji broadcast | non |
|
||||
| `/invite <persona>` | Inviter persona dans le canal | non |
|
||||
| `/time` | Heure (Europe/Paris) | non |
|
||||
| `/date` | Date (FR locale) | non |
|
||||
| `/session` | Info session courante | non |
|
||||
| `/speed` | Diagnostics latence (TTFC, p50/p95) | non |
|
||||
| `/model` | Changer modele | oui |
|
||||
| `/persona` | Gerer personas | oui |
|
||||
| `/reload` | Recharger config | oui |
|
||||
| `/export` | Exporter donnees | oui |
|
||||
| `/ban <nick>` | Bannir un utilisateur | oui |
|
||||
| `/unban <nick>` | Debannir un utilisateur | oui |
|
||||
| `/mute <persona>` | Muter une persona | non |
|
||||
| `/unmute <persona>` | Demuter une persona | non |
|
||||
|
||||
Mention directe: `@Schaeffer ta question` pour s'adresser a une persona specifique.
|
||||
|
||||
@@ -304,7 +326,7 @@ kxkm_clown/
|
||||
| RBAC | n/a | operationnel |
|
||||
| Frontend React | n/a | operationnel |
|
||||
| Training (TRL/Unsloth) | n/a | operationnel |
|
||||
| Tests (425) | smoke | unit + component + smoke (425 pass) |
|
||||
| Tests (425+) | smoke | unit + component + smoke (425+ pass) |
|
||||
| VoiceChat push-to-talk | n/a | operationnel |
|
||||
| Mediatheque gallery/playlist | n/a | operationnel |
|
||||
| UI Minitel VIDEOTEX | n/a | operationnel |
|
||||
|
||||
+39
-13
@@ -86,9 +86,13 @@ flowchart TD
|
||||
- [x] app.ts extraction (540 to 131 LOC, create-repos.ts extracted)
|
||||
- [x] Zod validation on all 19 API route schemas
|
||||
- [x] Error telemetry (16 labels)
|
||||
- [x] Perf instrumentation (6 labels, p50/p95/p99)
|
||||
- [x] Perf instrumentation (6 labels, p50/p95/p99), TTFC 284ms
|
||||
- [x] Smart routing (5 topic domains)
|
||||
- [x] Dynamic context window (4k-32k)
|
||||
- [x] NLP auto-detect generation intent (compose vs imagine)
|
||||
- [x] /speed command for latency diagnostics
|
||||
- [ ] lot-178: ACE-Step API direct integration (duration fix)
|
||||
- [ ] lot-180: Timeline data model
|
||||
- [ ] lot-97: Multi-channel support (create/join channels)
|
||||
- [ ] lot-100: Public demo mode read-only routes
|
||||
|
||||
@@ -97,8 +101,21 @@ flowchart TD
|
||||
- [x] Extraire registry.ts du hotspot node-engine
|
||||
- [ ] Ajouter node type `music_generation` (ACE-Step 1.5)
|
||||
- [ ] Ajouter node type `voice_clone` (Chatterbox)
|
||||
- [ ] Ajouter node type `audio_mix` (multi-track composition)
|
||||
- [ ] Ajouter node type `audio_effects` (FX chain)
|
||||
- [ ] lot-96: Automated DPO pipeline (feedback → pairs → training trigger)
|
||||
|
||||
### Multimodal (composition pipeline)
|
||||
|
||||
- [x] 35 music styles ACE-Step
|
||||
- [x] ComfyUI smart checkpoint selection (32 checkpoints + 24 LoRAs)
|
||||
- [ ] lot-178: ACE-Step API direct (duration fix)
|
||||
- [ ] lot-181: TTS voiceover mix into timeline
|
||||
- [ ] lot-182: Audio effects pipeline (reverb, delay, EQ, compression)
|
||||
- [ ] lot-183: DAW export (stems, markers, project file)
|
||||
- [ ] lot-187: Auto-mastering (loudness normalization, limiting)
|
||||
- [ ] lot-188: Stem separation (vocals, drums, bass, other)
|
||||
|
||||
### Personas
|
||||
|
||||
- [ ] Evaluer PCL (Persona-Aware Contrastive Learning) pour coherence
|
||||
@@ -118,9 +135,13 @@ flowchart TD
|
||||
- [x] Chat virtualization (react-window, variable row heights)
|
||||
- [x] Markdown rendering (marked + DOMPurify)
|
||||
- [x] CRT boot animation (modem dial, scanline reveal)
|
||||
- [x] 5 CSS themes (minitel, crt, hacker, synthwave, default)
|
||||
- [x] Mobile responsive pass (touch, bottom nav, viewport units)
|
||||
- [x] Guest mode read-only UI
|
||||
- [ ] lot-185: Composition timeline UI (waveform view, track lanes)
|
||||
- [ ] lot-194: FX rack UI (drag-drop effects chain)
|
||||
- [ ] lot-95: E2E Playwright tests (login, chat, upload, admin)
|
||||
- [ ] lot-98: File sharing UI (upload → gallery)
|
||||
- [ ] lot-99: Mobile responsive deep pass (touch, bottom nav, viewport)
|
||||
|
||||
### Ops/TUI
|
||||
|
||||
@@ -196,21 +217,26 @@ stateDiagram-v2
|
||||
- Privilegier TUI et scripts avec logs lisibles, puis purge des logs obsoletes.
|
||||
- Conserver la V1 comme reference comportementale, V2 comme cible active.
|
||||
|
||||
### Etat de cycle (2026-03-20)
|
||||
### Etat de cycle (2026-03-20 18:00)
|
||||
|
||||
- 71 lots termines (lot-24 a lot-94).
|
||||
- 130+ lots termines (lot-24 a lot-177).
|
||||
- 425 tests, 0 failures.
|
||||
- 12 services en production.
|
||||
- 19 chat commands, 33 personas.
|
||||
- 13 services en production.
|
||||
- 43 chat commands, 33 personas.
|
||||
- 35 music styles (ACE-Step), 5 CSS themes.
|
||||
- 32 ComfyUI checkpoints + 24 LoRAs, smart selection NLP.
|
||||
- TTFC 284ms.
|
||||
- Guest mode, mobile responsive.
|
||||
- Structured logging complet (pino JSON, 0 console.log).
|
||||
- Systemd services (TTS + LightRAG).
|
||||
- Frontend: lazy routes (-53%), React.memo, CRT boot, chat virtualization.
|
||||
|
||||
### Prochains lots (95-100)
|
||||
### Prochains lots (178-200) — Composition Pipeline
|
||||
|
||||
1. lot-95: E2E Playwright tests
|
||||
2. lot-96: Persona DPO automation pipeline
|
||||
3. lot-97: Multi-channel support
|
||||
4. lot-98: File sharing between users
|
||||
5. lot-99: Mobile responsive deep pass
|
||||
6. lot-100: Public demo mode
|
||||
1. lot-178: Compose duration fix (ACE-Step API direct)
|
||||
2. lot-179: SPEC_COMPOSE_ADVANCED plan
|
||||
3. lot-180: Timeline data model (tracks, clips, markers)
|
||||
4. lot-181: TTS voiceover mix into timeline
|
||||
5. lot-182: Audio effects pipeline (reverb, delay, EQ)
|
||||
6. lot-183: DAW export (stems, markers, project)
|
||||
7. lot-184-200: Multi-track, arrangement, mastering, stem separation, MIDI, templates, collab, lyrics, FX rack, automation, samples, spectral view, history, render queue, sharing
|
||||
|
||||
+12
-1
@@ -217,11 +217,22 @@ flowchart TD
|
||||
| Same-origin check | Verification origine sur les mutations |
|
||||
| RBAC (V2) | Roles admin, editor, operator, viewer |
|
||||
|
||||
## Fonctionnalites session 2026-03-20 (lots 174-177)
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| **35 music styles** | ACE-Step: ambient, techno, house, drum-n-bass, dubstep, trance, IDM, noise, drone, classical, jazz, blues, rock, metal, hip-hop, trap, lo-fi, chillwave, synthwave, vaporwave, industrial, EBM, darkwave, post-punk, shoegaze, krautrock, musique concrete, electroacoustique, minimal, glitch, breakcore, gabber, footwork, grime, UK garage |
|
||||
| **5 CSS themes** | minitel (phosphore vert), crt (amber), hacker (matrix green), synthwave (neon pink/blue), default (neutral) |
|
||||
| **ComfyUI smart selection** | 32 checkpoints + 24 LoRAs, auto-selection based on prompt NLP analysis (style, subject, medium) |
|
||||
| **NLP auto-detect generation** | Automatic detection of generation intent from natural language (compose vs imagine routing) |
|
||||
| **Guest mode** | Read-only access without login, rate-limited, no admin functions |
|
||||
| **Mobile responsive** | Touch gestures, bottom nav, viewport units, responsive layout pass |
|
||||
|
||||
## Fonctionnalites session 2026-03-19/20
|
||||
|
||||
| Fonctionnalite | Detail |
|
||||
| --- | --- |
|
||||
| **19 chat commands** | /help, /nick, /who, /personas, /web, /clear, /status, /model, /persona, /reload, /export, /compose, /imagine, /voice, /memory, /context, /rag, /stats, /uptime |
|
||||
| **43 chat commands** | /help, /nick, /who, /personas, /web, /clear, /status, /model, /persona, /reload, /export, /compose, /imagine, /voice, /memory, /context, /rag, /stats, /uptime, /changelog, /version, /dice, /roll, /flip, /ban, /unban, /mute, /unmute, /whisper, /w, /history, /search, /react, /invite, /time, /date, /session, /speed, + more |
|
||||
| **Markdown rendering** | marked + DOMPurify, bold, italic, code blocks, links, lists in chat messages |
|
||||
| **Smart routing** | 5 topic domains (music, philosophy, tech, arts, science), keyword + embedding scoring |
|
||||
| **Dynamic context window** | Adaptive 4k-32k tokens based on conversation length and complexity |
|
||||
|
||||
@@ -0,0 +1,719 @@
|
||||
# SPEC: Advanced Music Composition Features
|
||||
|
||||
> Date: 2026-03-20
|
||||
> Status: Planned (Phase 1 target: lots 180-185)
|
||||
> Depends on: SPEC_COMPOSE.md (current single-generation pipeline)
|
||||
> Stack: TypeScript (API) / Python (audio processing, GPU worker) / React (timeline UI)
|
||||
|
||||
---
|
||||
|
||||
## 0. Current State Summary
|
||||
|
||||
The existing `/compose` command (SPEC_COMPOSE.md) provides single-shot music generation:
|
||||
- ACE-Step 1.5 (primary) or MusicGen-small (fallback) on RTX 4090
|
||||
- Single prompt -> single WAV file -> base64 broadcast via WebSocket
|
||||
- No multi-track, no mixing, no post-processing
|
||||
- ComposePage.tsx: form + result list with inline `<audio>` players
|
||||
- Storage: `data/audio/<id>.wav` + `<id>.json` metadata
|
||||
|
||||
This spec extends the system into a multi-track composition workstation.
|
||||
|
||||
---
|
||||
|
||||
## 1. Timeline Composition
|
||||
|
||||
### 1.1 Data Model
|
||||
|
||||
```typescript
|
||||
interface Composition {
|
||||
id: string; // uuid
|
||||
name: string; // user-defined title
|
||||
createdAt: string; // ISO 8601
|
||||
updatedAt: string;
|
||||
bpm?: number; // optional tempo reference
|
||||
tracks: Track[];
|
||||
masterVolume: number; // 0.0 - 1.0
|
||||
owner: string; // nick
|
||||
channel: string;
|
||||
}
|
||||
|
||||
interface Track {
|
||||
id: string; // uuid
|
||||
type: "music" | "voice" | "sfx" | "noise";
|
||||
label: string; // user-defined or auto from prompt
|
||||
audioId: string; // reference to data/audio/<id>.wav
|
||||
prompt?: string; // generation prompt (if generated)
|
||||
style?: string;
|
||||
startMs: number; // offset from composition start
|
||||
endMs: number; // startMs + trimmed duration
|
||||
trimStartMs: number; // trim from source start
|
||||
trimEndMs: number; // trim from source end
|
||||
volume: number; // 0.0 - 1.0
|
||||
pan: number; // -1.0 (left) to 1.0 (right)
|
||||
muted: boolean;
|
||||
solo: boolean;
|
||||
effects: TrackEffect[];
|
||||
}
|
||||
|
||||
interface TrackEffect {
|
||||
type: "fadeIn" | "fadeOut" | "reverb" | "delay" | "eqLow" | "eqMid" | "eqHigh";
|
||||
params: Record<string, number>; // effect-specific (e.g., { durationMs: 2000 } for fade)
|
||||
}
|
||||
```
|
||||
|
||||
### 1.2 Timeline React Component
|
||||
|
||||
New component: `apps/web/src/components/TimelineEditor.tsx`
|
||||
|
||||
Features:
|
||||
- Horizontal scrollable timeline (1px = configurable ms, default 10ms/px)
|
||||
- Vertical stack of track lanes
|
||||
- Drag to move tracks on the timeline (snap to grid optional)
|
||||
- Drag edges to trim start/end
|
||||
- Overlap detection with crossfade hint
|
||||
- Playback cursor with real-time position (requestAnimationFrame)
|
||||
- Zoom controls (fit-to-view, zoom in/out)
|
||||
- Minitel phosphor theme (pink waveforms on dark background)
|
||||
|
||||
```
|
||||
+---------------------------------------------------------------+
|
||||
| TIMELINE EDITOR [+Track] [Export] |
|
||||
+---------------------------------------------------------------+
|
||||
| 0:00 0:10 0:20 0:30 0:40 0:50 1:00 |
|
||||
| |-------|-------|-------|-------|-------|-------| |
|
||||
| |
|
||||
| [M] Drone bass [====█████████████████====] vol ██░░ 60% |
|
||||
| [V] Narration P1 [===█████████===] vol ████ 90% |
|
||||
| [M] Percussions [====████████████████] vol ███░ 75% |
|
||||
| [S] Rain ambience [████████████████████████████] vol █░░░ 25% |
|
||||
| |
|
||||
| [>] 0:23.4 / 1:00.0 [Zoom -] [Fit] [Zoom +]|
|
||||
+---------------------------------------------------------------+
|
||||
```
|
||||
|
||||
### 1.3 Client-Side Playback
|
||||
|
||||
Use the Web Audio API for real-time multi-track preview:
|
||||
- Each track loaded as AudioBuffer
|
||||
- Per-track GainNode (volume), StereoPannerNode (pan)
|
||||
- Shared AudioContext with single destination
|
||||
- Playback position synced across all tracks via a shared start time
|
||||
- No server round-trip needed for preview playback
|
||||
|
||||
### 1.4 Storage
|
||||
|
||||
```
|
||||
data/compositions/<composition-id>/
|
||||
composition.json # Composition metadata + track list
|
||||
tracks/
|
||||
<track-id>.wav # individual track audio files
|
||||
mix.wav # exported final mix (generated on demand)
|
||||
mix.mp3 # optional compressed export
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. TTS Integration in Composition
|
||||
|
||||
### 2.1 Voice Track Generation
|
||||
|
||||
Extend `/compose` pipeline to accept voice tracks via persona TTS:
|
||||
|
||||
```
|
||||
/voice <persona> "<text>" [at <time>]
|
||||
```
|
||||
|
||||
Examples:
|
||||
```
|
||||
/voice Professeur "Bienvenue dans cette composition experimentale" at 0:05
|
||||
/voice Clown "Ha ha ha, la musique c'est rigolo!" at 0:30
|
||||
```
|
||||
|
||||
### 2.2 Persona Voice Styles
|
||||
|
||||
Each persona's voice configuration (from `persona-voices.ts`) maps to TTS parameters:
|
||||
|
||||
| Persona | TTS Backend | Voice | Speed | Pitch Shift |
|
||||
|-------------|-------------|-------------|-------|-------------|
|
||||
| Professeur | Piper | fr_FR-siwis | 0.9 | 0 |
|
||||
| Clown | Qwen3-TTS | clown-v1 | 1.1 | +2 semitones|
|
||||
| Poete | Piper | fr_FR-gilles| 0.8 | -1 semitone |
|
||||
| Musicien | Qwen3-TTS | musician-v1 | 1.0 | 0 |
|
||||
|
||||
### 2.3 Timing Synchronization
|
||||
|
||||
Voice tracks are positioned on the timeline like any other track:
|
||||
- `startMs` defines when the voice begins relative to composition start
|
||||
- TTS audio is generated server-side, returned as WAV
|
||||
- Client places it on the timeline at the specified offset
|
||||
- If no `at` time specified, voice starts at current playback cursor position
|
||||
|
||||
### 2.4 Spoken Word + Music Layering
|
||||
|
||||
Automatic ducking: when a voice track overlaps a music track, the music track volume is reduced by a configurable amount (default -6dB) via a sidechain-style envelope. This is applied at mix time by the server-side mixer.
|
||||
|
||||
---
|
||||
|
||||
## 3. Mixing & Effects
|
||||
|
||||
### 3.1 Per-Track Controls
|
||||
|
||||
| Control | Range | Default | Implementation |
|
||||
|-------------|----------------|---------|-------------------------|
|
||||
| Volume | 0.0 - 1.0 | 0.8 | Web Audio GainNode / ffmpeg |
|
||||
| Pan | -1.0 to 1.0 | 0.0 | StereoPannerNode / sox |
|
||||
| Mute | boolean | false | GainNode = 0 |
|
||||
| Solo | boolean | false | mute all others |
|
||||
|
||||
### 3.2 Effects Chain (Server-Side)
|
||||
|
||||
Effects are applied during server-side mix export using Python libraries:
|
||||
|
||||
```
|
||||
Track WAV -> [Trim] -> [EQ] -> [Reverb/Delay] -> [Fade] -> [Volume] -> [Pan] -> Mix Bus
|
||||
```
|
||||
|
||||
#### Library Selection
|
||||
|
||||
| Effect | Library | Rationale |
|
||||
|-------------|---------------|----------------------------------------------|
|
||||
| Trim | ffmpeg | Fast, reliable, no Python overhead |
|
||||
| Fade in/out | pydub | Simple amplitude envelope |
|
||||
| EQ (3-band) | pedalboard | Spotify's plugin host, GPU-friendly |
|
||||
| Reverb | pedalboard | High-quality convolution reverb |
|
||||
| Delay | pedalboard | Feedback delay with configurable taps |
|
||||
| Pan | sox / ffmpeg | Stereo field manipulation |
|
||||
| Final mix | ffmpeg | amerge + amix filters for multi-track |
|
||||
|
||||
#### Effect Parameters
|
||||
|
||||
```typescript
|
||||
// Fade
|
||||
{ type: "fadeIn", params: { durationMs: 2000 } }
|
||||
{ type: "fadeOut", params: { durationMs: 3000 } }
|
||||
|
||||
// EQ (gain in dB, -12 to +12)
|
||||
{ type: "eqLow", params: { gainDb: 3.0, freqHz: 200 } }
|
||||
{ type: "eqMid", params: { gainDb: -2.0, freqHz: 1000 } }
|
||||
{ type: "eqHigh", params: { gainDb: 1.5, freqHz: 5000 } }
|
||||
|
||||
// Reverb
|
||||
{ type: "reverb", params: { roomSize: 0.6, damping: 0.5, wet: 0.3 } }
|
||||
|
||||
// Delay
|
||||
{ type: "delay", params: { delayMs: 250, feedback: 0.4, wet: 0.25 } }
|
||||
```
|
||||
|
||||
### 3.3 Server-Side Mix Pipeline (Python)
|
||||
|
||||
New script: `scripts/mix_composition.py`
|
||||
|
||||
```python
|
||||
# Pseudocode
|
||||
def mix_composition(composition_json_path, output_path):
|
||||
comp = load_composition(composition_json_path)
|
||||
|
||||
# 1. Determine total duration
|
||||
total_ms = max(t.endMs for t in comp.tracks)
|
||||
|
||||
# 2. Process each track
|
||||
processed = []
|
||||
for track in comp.tracks:
|
||||
if track.muted:
|
||||
continue
|
||||
audio = load_wav(track.audioId)
|
||||
audio = trim(audio, track.trimStartMs, track.trimEndMs)
|
||||
for effect in track.effects:
|
||||
audio = apply_effect(audio, effect)
|
||||
audio = set_volume(audio, track.volume)
|
||||
audio = set_pan(audio, track.pan)
|
||||
processed.append((track.startMs, audio))
|
||||
|
||||
# 3. Mix down to stereo
|
||||
mix = create_silent(total_ms, stereo=True)
|
||||
for start_ms, audio in processed:
|
||||
mix = overlay(mix, audio, position=start_ms)
|
||||
|
||||
# 4. Apply master volume
|
||||
mix = set_volume(mix, comp.masterVolume)
|
||||
|
||||
# 5. Export
|
||||
export_wav(mix, output_path)
|
||||
export_mp3(mix, output_path.replace('.wav', '.mp3'))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Generation Modes & Commands
|
||||
|
||||
### 4.1 Command Reference
|
||||
|
||||
| Command | Syntax | Description |
|
||||
|---------|--------|-------------|
|
||||
| `/compose` | `/compose <prompt>, <style>, <duration>s` | Single generation (existing) |
|
||||
| `/layer` | `/layer <prompt>, <style>, <duration>s` | Add a track to active composition |
|
||||
| `/voice` | `/voice <persona> "<text>" [at <time>]` | Add TTS voice track |
|
||||
| `/remix` | `/remix <track-id> [<new-prompt>]` | Re-generate a specific track |
|
||||
| `/mix` | `/mix [<composition-id>]` | Server-side mix of all active tracks |
|
||||
| `/export` | `/export [wav\|mp3] [<composition-id>]` | Download final mix |
|
||||
| `/comp new` | `/comp new <name>` | Create new composition |
|
||||
| `/comp list` | `/comp list` | List user's compositions |
|
||||
| `/comp load` | `/comp load <id>` | Load composition into timeline |
|
||||
| `/comp delete` | `/comp delete <id>` | Delete composition |
|
||||
|
||||
### 4.2 /layer Flow
|
||||
|
||||
```
|
||||
User: /layer rain and thunder ambience, ambient style, 60s
|
||||
|
||||
1. API checks for active composition (creates one if none)
|
||||
2. Generates audio via existing compose pipeline (ACE-Step / MusicGen)
|
||||
3. Creates new Track entry with type="sfx"
|
||||
4. Adds track to composition at startMs=0 (or cursor position)
|
||||
5. Saves audio to data/compositions/<id>/tracks/<track-id>.wav
|
||||
6. Broadcasts track-added event to channel
|
||||
7. Client adds track to TimelineEditor
|
||||
```
|
||||
|
||||
### 4.3 /remix Flow
|
||||
|
||||
```
|
||||
User: /remix abc123 "heavier bass, more distortion"
|
||||
|
||||
1. API looks up track abc123 in active composition
|
||||
2. Preserves track position (startMs, endMs) and effects
|
||||
3. Re-runs compose pipeline with new prompt
|
||||
4. Replaces track audio file
|
||||
5. Broadcasts track-updated event
|
||||
6. Client reloads the track waveform
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Sound Design
|
||||
|
||||
### 5.1 Noise Generators
|
||||
|
||||
Built-in Python generators (no GPU needed) for layering:
|
||||
|
||||
```
|
||||
/layer noise:white 30s # white noise, 30 seconds
|
||||
/layer noise:pink 60s # pink noise (1/f)
|
||||
/layer noise:brown 120s # brown noise (1/f^2)
|
||||
/layer drone:sine 60s 110hz # sine drone at 110Hz
|
||||
/layer drone:evolving 120s # slowly modulating drone (LFO on frequency + filter)
|
||||
```
|
||||
|
||||
Implementation in `scripts/noise_generators.py`:
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import scipy.io.wavfile
|
||||
|
||||
def white_noise(duration_s, sr=44100):
|
||||
return np.random.randn(int(duration_s * sr)).astype(np.float32) * 0.3
|
||||
|
||||
def pink_noise(duration_s, sr=44100):
|
||||
# Voss-McCartney algorithm
|
||||
...
|
||||
|
||||
def brown_noise(duration_s, sr=44100):
|
||||
white = np.random.randn(int(duration_s * sr))
|
||||
return np.cumsum(white).astype(np.float32) / (duration_s * sr) * 10
|
||||
|
||||
def sine_drone(duration_s, freq_hz=110, sr=44100):
|
||||
t = np.linspace(0, duration_s, int(duration_s * sr), endpoint=False)
|
||||
return (np.sin(2 * np.pi * freq_hz * t) * 0.5).astype(np.float32)
|
||||
|
||||
def evolving_drone(duration_s, base_freq=80, sr=44100):
|
||||
t = np.linspace(0, duration_s, int(duration_s * sr), endpoint=False)
|
||||
lfo = np.sin(2 * np.pi * 0.05 * t) * 20 # slow LFO, +/- 20Hz
|
||||
return (np.sin(2 * np.pi * (base_freq + lfo) * t) * 0.4).astype(np.float32)
|
||||
```
|
||||
|
||||
### 5.2 Granular Synthesis
|
||||
|
||||
Process uploaded audio samples into granular textures:
|
||||
|
||||
```
|
||||
/granular <audio-id> grains=50ms density=10 spread=0.8 pitch_var=0.3 duration=60s
|
||||
```
|
||||
|
||||
Parameters:
|
||||
- `grains`: grain size in ms (10-500ms)
|
||||
- `density`: grains per second (1-100)
|
||||
- `spread`: random position spread in source (0.0-1.0)
|
||||
- `pitch_var`: random pitch variation (0.0 = none, 1.0 = +/- octave)
|
||||
- `duration`: output duration
|
||||
|
||||
Implementation via `scripts/granular.py` using numpy slicing + overlap-add.
|
||||
|
||||
### 5.3 Field Recording Processing
|
||||
|
||||
When users upload audio files (future: drag-and-drop onto timeline), processing options:
|
||||
|
||||
| Processing | Command | Backend |
|
||||
|----------------|---------------------------------|-----------------|
|
||||
| Reverse | `/fx reverse <track-id>` | pydub |
|
||||
| Time stretch | `/fx stretch <track-id> 0.5` | librosa / sox |
|
||||
| Pitch shift | `/fx pitch <track-id> +3` | pedalboard |
|
||||
| Normalize | `/fx normalize <track-id>` | pydub |
|
||||
| Spectral freeze| `/fx freeze <track-id> at 0:15` | librosa STFT |
|
||||
|
||||
---
|
||||
|
||||
## 6. Integration with Chat
|
||||
|
||||
### 6.1 Persona Musical Suggestions
|
||||
|
||||
When a persona detects music-related conversation, it can suggest compositions:
|
||||
|
||||
```
|
||||
System prompt injection (for music-aware personas):
|
||||
"Si la conversation aborde un theme emotionnel ou atmospherique fort,
|
||||
tu peux suggerer une composition musicale avec la syntaxe:
|
||||
[SUGGEST_MUSIC: prompt='<description>', style='<style>', duration=<seconds>]"
|
||||
```
|
||||
|
||||
The API parser detects `[SUGGEST_MUSIC: ...]` in persona responses and renders it as a clickable suggestion in the chat UI. Clicking triggers the `/compose` command.
|
||||
|
||||
### 6.2 Auto-Ambient Background
|
||||
|
||||
Per-channel ambient mode:
|
||||
|
||||
```
|
||||
/ambient on "dark ambient forest with distant thunder"
|
||||
/ambient off
|
||||
/ambient volume 0.3
|
||||
```
|
||||
|
||||
- Generates a long-form ambient loop (120s) via ACE-Step
|
||||
- Loops seamlessly on the client (crossfade last 5s with first 5s)
|
||||
- Low default volume (0.2)
|
||||
- Persists per channel (stored in channel metadata)
|
||||
- Auto-pauses during voice track playback
|
||||
|
||||
### 6.3 Mood-Based Auto-Compose
|
||||
|
||||
Sentiment analysis on the last N messages to derive a musical mood:
|
||||
|
||||
```
|
||||
Conversation mood mapping:
|
||||
joyful -> upbeat electronic, major key, 120bpm
|
||||
melancholy-> ambient piano, minor key, 70bpm
|
||||
aggressive-> noise, industrial, distorted
|
||||
calm -> drone, ambient pads, slow evolution
|
||||
absurd -> experimental, concrete, chaotic
|
||||
```
|
||||
|
||||
Trigger: automatic (if `/ambient auto` is on) or manual (`/mood-music`).
|
||||
|
||||
Implementation:
|
||||
1. Last 20 messages -> LLM prompt: "Analyse le mood de cette conversation en un mot"
|
||||
2. Mood word -> style mapping table
|
||||
3. Style -> `/compose` auto-trigger
|
||||
4. Result set as channel ambient
|
||||
|
||||
---
|
||||
|
||||
## 7. Technical Architecture
|
||||
|
||||
### 7.1 Composition Pipeline
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Client (React)"
|
||||
UI[TimelineEditor.tsx]
|
||||
WA[Web Audio API]
|
||||
WS_C[WebSocket Client]
|
||||
WV[Waveform Viz<br/>wavesurfer.js]
|
||||
end
|
||||
|
||||
subgraph "API Server (Node.js)"
|
||||
CMD[ws-commands.ts]
|
||||
COMP[composition-manager.ts]
|
||||
STORE[media-store.ts]
|
||||
end
|
||||
|
||||
subgraph "Audio Workers (Python)"
|
||||
ACE[compose_music.py<br/>ACE-Step 1.5]
|
||||
MIX[mix_composition.py<br/>pydub + pedalboard]
|
||||
NOISE[noise_generators.py]
|
||||
GRAN[granular.py]
|
||||
TTS_W[tts-server.py<br/>Piper / Qwen3-TTS]
|
||||
end
|
||||
|
||||
subgraph "Storage"
|
||||
FS[data/compositions/<br/><id>/tracks/*.wav]
|
||||
META[data/compositions/<br/><id>/composition.json]
|
||||
MIXF[data/compositions/<br/><id>/mix.wav]
|
||||
end
|
||||
|
||||
UI -->|track edits| WS_C
|
||||
WS_C -->|commands| CMD
|
||||
CMD -->|/layer, /compose| ACE
|
||||
CMD -->|/voice| TTS_W
|
||||
CMD -->|/mix, /export| MIX
|
||||
CMD -->|noise:*, drone:*| NOISE
|
||||
CMD -->|/granular| GRAN
|
||||
CMD -->|CRUD| COMP
|
||||
COMP -->|read/write| META
|
||||
COMP -->|read/write| FS
|
||||
ACE -->|WAV| FS
|
||||
TTS_W -->|WAV| FS
|
||||
NOISE -->|WAV| FS
|
||||
GRAN -->|WAV| FS
|
||||
MIX -->|reads| FS
|
||||
MIX -->|writes| MIXF
|
||||
MIXF -->|base64| CMD
|
||||
CMD -->|broadcast| WS_C
|
||||
WS_C -->|audio data| WA
|
||||
WS_C -->|waveform data| WV
|
||||
UI -->|playback| WA
|
||||
WV -->|render| UI
|
||||
```
|
||||
|
||||
### 7.2 Mixing Flow
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph "Per Track"
|
||||
SRC[Source WAV] --> TRIM[Trim]
|
||||
TRIM --> EQ[3-Band EQ<br/>pedalboard]
|
||||
EQ --> FX[Reverb / Delay<br/>pedalboard]
|
||||
FX --> FADE[Fade In/Out<br/>pydub]
|
||||
FADE --> VOL[Volume<br/>gain multiply]
|
||||
VOL --> PAN[Pan<br/>stereo split]
|
||||
end
|
||||
|
||||
subgraph "Mix Bus"
|
||||
PAN --> OVERLAY[Overlay at startMs<br/>pydub]
|
||||
OVERLAY --> MASTER[Master Volume]
|
||||
MASTER --> LIMIT[Limiter<br/>pedalboard]
|
||||
LIMIT --> EXPORT_WAV[WAV 44.1kHz 16-bit]
|
||||
LIMIT --> EXPORT_MP3[MP3 192kbps<br/>ffmpeg]
|
||||
end
|
||||
```
|
||||
|
||||
### 7.3 Timeline UI Component Tree
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
TP[TimelinePage.tsx] --> TE[TimelineEditor.tsx]
|
||||
TP --> CP[ComposePanel.tsx<br/>generation form]
|
||||
TP --> MP[MixerPanel.tsx<br/>per-track controls]
|
||||
|
||||
TE --> TH[TimelineHeader.tsx<br/>time ruler + zoom]
|
||||
TE --> TL[TrackLane.tsx]
|
||||
TL --> TB[TrackBlock.tsx<br/>draggable, resizable]
|
||||
TB --> WF[WaveformView.tsx<br/>wavesurfer.js mini]
|
||||
TE --> PC[PlaybackCursor.tsx<br/>vertical line, animated]
|
||||
TE --> TR[TransportBar.tsx<br/>play/pause/stop/loop]
|
||||
|
||||
MP --> TC[TrackControls.tsx]
|
||||
TC --> VS[VolumeSlider.tsx]
|
||||
TC --> PS[PanSlider.tsx]
|
||||
TC --> MB[Mute/Solo buttons]
|
||||
TC --> FXP[EffectsPanel.tsx]
|
||||
FXP --> FDE[FadeEditor.tsx]
|
||||
FXP --> EQE[EQEditor.tsx]
|
||||
FXP --> RVE[ReverbEditor.tsx]
|
||||
```
|
||||
|
||||
### 7.4 WebSocket Protocol Extensions
|
||||
|
||||
New message types for composition state sync:
|
||||
|
||||
```typescript
|
||||
// Server -> Client
|
||||
interface CompositionSync {
|
||||
type: "composition_sync";
|
||||
composition: Composition; // full state
|
||||
}
|
||||
|
||||
interface TrackAdded {
|
||||
type: "track_added";
|
||||
compositionId: string;
|
||||
track: Track;
|
||||
audioData?: string; // base64 WAV (for immediate playback)
|
||||
}
|
||||
|
||||
interface TrackUpdated {
|
||||
type: "track_updated";
|
||||
compositionId: string;
|
||||
trackId: string;
|
||||
changes: Partial<Track>;
|
||||
}
|
||||
|
||||
interface TrackRemoved {
|
||||
type: "track_removed";
|
||||
compositionId: string;
|
||||
trackId: string;
|
||||
}
|
||||
|
||||
interface MixReady {
|
||||
type: "mix_ready";
|
||||
compositionId: string;
|
||||
audioData: string; // base64 WAV or MP3
|
||||
audioMime: string;
|
||||
fileSize: number;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
// Client -> Server
|
||||
interface TrackEdit {
|
||||
type: "track_edit";
|
||||
compositionId: string;
|
||||
trackId: string;
|
||||
changes: Partial<Track>; // startMs, volume, pan, effects, etc.
|
||||
}
|
||||
```
|
||||
|
||||
### 7.5 Waveform Visualization
|
||||
|
||||
Use wavesurfer.js (v7+) for waveform rendering:
|
||||
|
||||
- Mini waveforms in track blocks (static, pre-rendered peaks)
|
||||
- Full waveform in focused track detail view
|
||||
- Regions plugin for trim selection
|
||||
- Timeline plugin for ruler
|
||||
- Spectrogram plugin (optional, for sound design view)
|
||||
- Minitel theme: pink waveform (#FF66B2), dark background (#1A0011)
|
||||
|
||||
### 7.6 VRAM Scheduling
|
||||
|
||||
Composition operations compete for GPU VRAM with Ollama (LLM) and other generators:
|
||||
|
||||
```
|
||||
Priority queue:
|
||||
1. Active chat LLM response (Ollama) — cannot interrupt
|
||||
2. TTS voice generation (Piper: CPU-only, Qwen3-TTS: GPU)
|
||||
3. Music generation (ACE-Step: GPU)
|
||||
4. Image generation (ComfyUI: GPU)
|
||||
5. Mixing/effects (pedalboard: CPU, minimal GPU)
|
||||
6. Noise/drone generation (numpy: CPU only)
|
||||
|
||||
Strategy:
|
||||
- CPU-only tasks (noise, pydub mix, Piper TTS) run immediately, no queuing
|
||||
- GPU tasks enter a FIFO queue with estimated VRAM
|
||||
- ACE-Step (~5GB) + Ollama 8B (~5GB) can coexist on 24GB RTX 4090
|
||||
- ACE-Step + ComfyUI SDXL (~8GB) cannot coexist; sequential only
|
||||
- Queue broadcasts position updates to clients
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation Phases
|
||||
|
||||
### Phase 1 — Multi-Track Foundation (lots 180-185)
|
||||
|
||||
| Lot | Title | Scope | Tests |
|
||||
|-----|-------|-------|-------|
|
||||
| 180 | Composition data model + CRUD | `composition-manager.ts`: create, load, save, delete, list compositions. Storage in `data/compositions/`. | 12 unit tests |
|
||||
| 181 | `/layer` command | Parse `/layer`, generate via existing pipeline, add track to active composition. Auto-create composition if none. | 8 integration tests |
|
||||
| 182 | `/comp` commands | `/comp new`, `/comp list`, `/comp load`, `/comp delete`. WebSocket handlers. | 10 unit tests |
|
||||
| 183 | Basic TimelineEditor UI | React component: track lanes, static blocks (no drag yet), playback via Web Audio API. | 6 component tests |
|
||||
| 184 | Volume + pan controls | Per-track volume slider + pan knob in MixerPanel. Client-side via Web Audio GainNode + StereoPannerNode. | 4 tests |
|
||||
| 185 | `/mix` + `/export` | Server-side mix via `mix_composition.py` (pydub). Export as WAV. Base64 broadcast + file download endpoint. | 8 integration tests |
|
||||
|
||||
**Phase 1 deliverable**: Users can generate multiple tracks, see them in a timeline, adjust volume/pan, preview in browser, and export a mixed WAV.
|
||||
|
||||
### Phase 2 — Voice Tracks & Timeline Interaction (lots 186-190)
|
||||
|
||||
| Lot | Title | Scope | Tests |
|
||||
|-----|-------|-------|-------|
|
||||
| 186 | `/voice` command + TTS track | Generate persona voice via Piper/Qwen3-TTS, add as voice track with `at` positioning. | 8 tests |
|
||||
| 187 | Timeline drag & trim | Track blocks become draggable (startMs) and edge-resizable (trim). React DnD or pointer events. | 6 tests |
|
||||
| 188 | Fade in/out effects | FadeEditor UI component. Server-side via pydub. Preview on client with GainNode ramp. | 4 tests |
|
||||
| 189 | Voice ducking | Auto-reduce music volume when voice track overlaps. Configurable dB reduction. Applied at mix time. | 4 tests |
|
||||
| 190 | Waveform visualization | Integrate wavesurfer.js for mini-waveforms in track blocks. Peak data generated server-side. | 4 tests |
|
||||
|
||||
**Phase 2 deliverable**: Voice narration layered over music, interactive timeline editing, visual waveforms.
|
||||
|
||||
### Phase 3 — Effects & Sound Design (lots 191-195)
|
||||
|
||||
| Lot | Title | Scope | Tests |
|
||||
|-----|-------|-------|-------|
|
||||
| 191 | pedalboard integration | Install pedalboard in Python venv. EQ, reverb, delay applied via `mix_composition.py`. | 6 tests |
|
||||
| 192 | EQ editor UI | 3-band EQ knobs (low/mid/high) per track in EffectsPanel. Real-time preview via Web Audio BiquadFilterNode. | 4 tests |
|
||||
| 193 | Reverb + delay UI | ReverbEditor component. Wet/dry, room size, damping, delay time, feedback. | 4 tests |
|
||||
| 194 | Noise & drone generators | `noise_generators.py`: white, pink, brown noise + sine/evolving drone. `/layer noise:*` and `/layer drone:*` commands. | 8 tests |
|
||||
| 195 | Granular synthesis | `granular.py`: grain extraction + overlap-add resynthesis. `/granular` command. | 6 tests |
|
||||
|
||||
**Phase 3 deliverable**: Full effects chain, built-in sound generators, granular processing.
|
||||
|
||||
### Phase 4 — Intelligence & Polish (lots 196-200)
|
||||
|
||||
| Lot | Title | Scope | Tests |
|
||||
|-----|-------|-------|-------|
|
||||
| 196 | `/remix` command | Re-generate a track with new prompt, preserving timeline position and effects. | 6 tests |
|
||||
| 197 | Persona music suggestions | `[SUGGEST_MUSIC]` tag detection in LLM responses. Clickable suggestion UI in chat. | 4 tests |
|
||||
| 198 | `/ambient` auto-loop | Channel ambient mode: generate, loop with crossfade, low volume background. | 6 tests |
|
||||
| 199 | Mood-based auto-compose | Sentiment analysis -> style mapping -> auto-generate ambient. `/ambient auto` toggle. | 6 tests |
|
||||
| 200 | Spectrogram view + polish | wavesurfer.js spectrogram plugin. Keyboard shortcuts (space=play, delete=remove track). Final UX polish. | 4 tests |
|
||||
|
||||
**Phase 4 deliverable**: AI-assisted composition, mood-reactive music, full DAW-like experience.
|
||||
|
||||
---
|
||||
|
||||
## 9. Dependencies & Installation
|
||||
|
||||
### Python packages (add to `requirements.txt`)
|
||||
|
||||
```
|
||||
pydub>=0.25.1 # audio manipulation, overlay, fade
|
||||
pedalboard>=0.9.0 # Spotify audio effects (EQ, reverb, delay, limiter)
|
||||
librosa>=0.10.0 # time stretch, pitch shift, spectral analysis
|
||||
soundfile>=0.12.0 # audio I/O (WAV, FLAC, OGG)
|
||||
numpy>=1.24.0 # already present (noise generators, granular)
|
||||
scipy>=1.10.0 # already present (MusicGen output)
|
||||
```
|
||||
|
||||
### System packages
|
||||
|
||||
```bash
|
||||
sudo apt install ffmpeg sox libsox-fmt-all # already on kxkm-ai
|
||||
```
|
||||
|
||||
### npm packages (add to `apps/web/package.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"wavesurfer.js": "^7.8.0"
|
||||
}
|
||||
```
|
||||
|
||||
### VRAM Budget (RTX 4090, 24GB)
|
||||
|
||||
| Concurrent Operation | VRAM | Status |
|
||||
|---------------------|------|--------|
|
||||
| Ollama 8B (quantized) | ~5GB | Always loaded |
|
||||
| ACE-Step 1.5 | ~5GB | On demand, unloads after |
|
||||
| pedalboard effects | ~0GB | CPU only |
|
||||
| pydub mixing | ~0GB | CPU only |
|
||||
| Noise generators | ~0GB | CPU only |
|
||||
| Qwen3-TTS | ~2GB | On demand |
|
||||
| **Total peak** | **~12GB** | Comfortable headroom |
|
||||
|
||||
---
|
||||
|
||||
## 10. Open Questions
|
||||
|
||||
1. **Collaborative editing**: Should multiple users on the same channel be able to edit the same composition simultaneously? (WebSocket sync complexity vs. single-editor lock)
|
||||
2. **Audio upload**: Allow users to upload their own WAV/MP3 files as tracks? (Security: file validation, size limits, malware scanning)
|
||||
3. **Loop mode**: Should tracks support looping (repeat N times or infinite)?
|
||||
4. **Undo/redo**: Full undo stack for timeline edits? (Memory cost vs. UX benefit)
|
||||
5. **MIDI integration**: Accept MIDI files as rhythm/melody guides for ACE-Step? (ACE-Step 1.5 does not natively support MIDI conditioning)
|
||||
6. **Streaming preview**: Can we stream partial mixes via WebSocket for real-time effect preview without full re-export? (Latency vs. quality tradeoff)
|
||||
|
||||
---
|
||||
|
||||
## 11. References
|
||||
|
||||
- [ACE-Step 1.5](https://github.com/ACE-Step/ACE-Step) — primary music generator
|
||||
- [pedalboard](https://github.com/spotify/pedalboard) — Spotify audio effects
|
||||
- [pydub](https://github.com/jiaaro/pydub) — audio manipulation
|
||||
- [wavesurfer.js](https://wavesurfer.xyz/) — waveform visualization
|
||||
- [Web Audio API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API) — client-side playback
|
||||
- SPEC_COMPOSE.md — current single-generation pipeline
|
||||
- TIMING_RECOMMENDATIONS_2026-03-19.md — latency targets
|
||||
Reference in New Issue
Block a user