# Merlin Studio v2 — Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Côté Tower (`lisael-content`) + UI web (`merlin.saillant.cc`), livrer (A) le pipeline de génération des routines illustrées (TTS voix `fr` + Twemoji→`.bin` RGB565A8 + `routines.json` SD + push delta), (B) l'onglet web « 🗓️ Routines » + son API, et (C) le journal des transferts (durée/débit/retry) + l'onglet web « 📤 Transferts » + son API. Le firmware est traité dans un plan séparé ; ce plan respecte le contrat d'interface SD/push partagé. **Architecture :** `tower/lisael_content.py` (module partagé, Python stdlib + Pillow) gère feeds/podcasts + désormais routines + journal transferts. `tower/lisael_server.py` (`:9555`, `ThreadingHTTPServer`, classe `H`) expose `/register` (machine) et l'API web sous Basic Auth. `tower/merlin_ui.html` (vanilla JS, onglets) est l'UI. La box annonce dans `have` (POST `/register`) l'union des basenames de `/sdcard/podcasts` ET `/sdcard/routines` ; Tower calcule le delta routines = `(basenames staging routines) − have` et pousse vers `?name=routines/`. Vérité éditoriale = `routines.json` côté Tower (`/home/clems/lisael-content/routines.json`), régénéré en assets par `build_routine_manifest`. **Tech Stack :** Python 3 stdlib (`http.server`, `urllib`, `json`, `subprocess`, `tempfile`, `threading`, `hashlib`, `time`) + Pillow (`PIL.Image`) + `LVGLImage.py` (script LVGL, `--ofmt BIN`) + `ffmpeg`. UI = HTML/CSS/JS vanilla, pas de framework. Tests hôte = asserts purs `if __name__ == "__main__"` (pattern `tower/test_delta.py` / `tower/test_content.py`), pas de pytest. ## Global Constraints - Python stdlib only (+ Pillow déjà utilisé pour les covers) ; aucune nouvelle dépendance pip. - TTS via la gateway origine tailnet `http://100.78.6.122:9300/v1/audio/speech` (`model=tts-1`, `voice=fr`, `response_format=mp3`) — JAMAIS le domaine public (Cloudflare bloque `Python-urllib`). - Twemoji raw via `https://raw.githubusercontent.com/jdecked/twemoji/main/assets/72x72/.png` avec `User-Agent: curl/8` ; `cp()` strip le VS16 (`U+FE0F`) sinon 404. - Vérif des URLs web publiques (`merlin.saillant.cc`) toujours avec `User-Agent: Mozilla/5.0` (Cloudflare 403 sinon). - Pictos étape = `LVGLImage.py --ofmt BIN --cf RGB565A8` sur PNG 160×160 `.convert("RGBA")` (garde l'alpha de l'emoji) — PAS `RGB565`/`.convert("RGB")` (réservé aux covers podcasts). - Audio = TTS mono mp3 ; si re-transcode nécessaire, `ffmpeg -ac 1 -b:a 48k -c:a libmp3lame` (la sortie gateway est déjà mono mp3, transcode optionnel). - Noms de fichiers SD strictement ASCII, préfixés par id de routine (`matin_`/`soir_`) ; les libellés accentués vivent dans le JSON UTF-8. Pas de collision avec les podcasts (dossier SD distinct `routines/`). - Push delta : le manifest (`routines.json`) part TOUJOURS en dernier (réutiliser la sémantique `compute_delta`). - Toute écriture de fichier d'état (`routines.json`, `transfers.json`) via `_atomic_write_json`. - API web sous Basic Auth `MERLIN_AUTH` (court-circuit `/register` machine sans auth, restreint IP privée via `_is_private`). - Déploiement manuel : `rsync` vers `/home/clems/lisael-content` sur Tower (`clems@192.168.0.120`) puis `sudo systemctl restart lisael-content`. - Commits : sujet ≤ 50 chars, scope sans underscore, PAS d'attribution AI, pas de `--no-verify`. --- ### Task 1: Helper TTS `tts_speak()` dans `lisael_content.py` **Files:** - `tower/lisael_content.py` (ajout) **Interfaces:** - Produces : `def tts_speak(text: str) -> bytes` — POST gateway `/v1/audio/speech`, renvoie les octets mp3 (lève en cas d'échec réseau/HTTP). - Consumes : env `LISAEL_TTS_URL` (défaut `http://100.78.6.122:9300/v1/audio/speech`), env `LISAEL_TTS_VOICE` (défaut `fr`). - [ ] **Step 1:** Ajouter en tête de `tower/lisael_content.py` (près des autres constantes, après `BOX_PORT`) : ```python TTS_URL = os.environ.get("LISAEL_TTS_URL", "http://100.78.6.122:9300/v1/audio/speech") TTS_VOICE = os.environ.get("LISAEL_TTS_VOICE", "fr") ``` - [ ] **Step 2:** Ajouter la fonction (après `transcode`, avant `make_cover`) : ```python def tts_speak(text): """Generate a mono mp3 announcement via the tailnet gateway. Returns bytes. Raises on network/HTTP failure (caller decides degraded behaviour).""" body = json.dumps({"model": "tts-1", "input": text, "voice": TTS_VOICE, "response_format": "mp3"}).encode() req = urllib.request.Request(TTS_URL, data=body, headers={"Content-Type": "application/json"}) with urllib.request.urlopen(req, timeout=30) as r: return r.read() ``` - [ ] **Step 3:** Vérif manuelle (réseau, non unit-testable). Depuis grosmac via Tower : `ssh clems@192.168.0.120 'cd /home/clems/lisael-content && python3 -c "import lisael_content as C; b=C.tts_speak(\"Habille-toi\"); print(len(b)); open(\"/tmp/tts_test.mp3\",\"wb\").write(b)"'` puis vérifier que le fichier fait > 1 Ko et est lisible (`file /tmp/tts_test.mp3` → MPEG ADTS / MP3). Si la gateway n'est pas joignable depuis grosmac, exécuter directement sur Tower. Commit : `feat(tower): add gateway TTS helper`. --- ### Task 2: Helper Twemoji→`.bin` `emoji_cp()` + `emoji_bin()` (TDD pour `emoji_cp`) **Files:** - `tower/lisael_content.py` (ajout) - `tower/test_routines.py` (nouveau fichier de test) **Interfaces:** - Produces : `def emoji_cp(emoji: str) -> str` — codepoints Twemoji joints par `-`, VS16 stripé (pure, testable). - Produces : `def emoji_bin(emoji: str, key: str) -> str` — télécharge le Twemoji, resize 160×160 RGBA, génère `.bin` RGB565A8 dans `STAGING_ROUTINES`, renvoie `".bin"` ou `""` (réseau, non unit-testable). - Consumes : constante `TWEMOJI`, `LVGLIMG`, `STAGING_ROUTINES` (Task 5 ; pour cette task on peut écrire dans un dossier passé/par défaut — voir Step 4). - [ ] **Step 1 (RED):** Créer `tower/test_routines.py` avec en tête le pattern d'isolation de `test_content.py` : ```python import os, tempfile os.environ["LISAEL_BASE"] = tempfile.mkdtemp(prefix="routines_test_") os.environ["LISAEL_STAGING"] = os.path.join(os.environ["LISAEL_BASE"], "podcasts") os.environ["LISAEL_STAGING_ROUTINES"] = os.path.join(os.environ["LISAEL_BASE"], "routines") import lisael_content as C def test_emoji_cp_strips_vs16(): # plain emoji: single codepoint assert C.emoji_cp("\U0001F600") == "1f600" # 😀 # emoji with VS16 (U+FE0F) must be stripped (twemoji 404s otherwise) assert C.emoji_cp("☀️") == "2600" # ☀️ -> 2600 # multi-codepoint (no VS16): joined by '-' assert C.emoji_cp("\U0001F468‍\U0001F4BB").split("-")[0] == "1f468" ``` Et le runner en bas (calqué sur `test_content.py`) : ```python if __name__ == "__main__": for name, fn in sorted(globals().items()): if name.startswith("test_"): fn(); print("ok", name) print("ALL OK") ``` - [ ] **Step 2 (RED run):** `cd tower && python3 test_routines.py` → DOIT échouer (`AttributeError: module ... has no attribute 'emoji_cp'`). - [ ] **Step 3 (GREEN):** Ajouter dans `lisael_content.py` (la constante près des autres, la fonction après `transcode`/`tts_speak`) : ```python TWEMOJI = os.environ.get("LISAEL_TWEMOJI", "https://raw.githubusercontent.com/jdecked/twemoji/main/assets/72x72") def emoji_cp(emoji): """Twemoji filename codepoints, VS16 (U+FE0F) stripped or twemoji 404s.""" return "-".join(f"{ord(c):x}" for c in emoji if c != "️") ``` - [ ] **Step 4 (GREEN run):** `cd tower && python3 test_routines.py` → `ok test_emoji_cp_strips_vs16` puis `ALL OK`. - [ ] **Step 5:** Ajouter `emoji_bin` (après `emoji_cp`). Écrit dans `STAGING_ROUTINES` (défini en Task 5 ; si Task 5 pas encore mergée, ce step suppose la constante présente — l'ordre du plan place Task 5 avant l'usage réel ; ici on référence la constante) : ```python def emoji_bin(emoji, key): """Download the Twemoji for `emoji`, resize 160x160 RGBA, write `.bin` (RGB565A8, keeps alpha) into STAGING_ROUTINES. Returns '.bin' or ''.""" try: url = f"{TWEMOJI}/{emoji_cp(emoji)}.png" raw = urllib.request.urlopen( urllib.request.Request(url, headers={"User-Agent": "curl/8"}), timeout=15, context=ssl.create_default_context()).read() im = Image.open(io.BytesIO(raw)).convert("RGBA").resize((160, 160), Image.LANCZOS) os.makedirs(STAGING_ROUTINES, exist_ok=True) with tempfile.TemporaryDirectory() as td: png = os.path.join(td, f"{key}.png"); im.save(png) subprocess.run([sys.executable, LVGLIMG, "--ofmt", "BIN", "--cf", "RGB565A8", "-o", td, png], check=True, capture_output=True) binp = os.path.join(td, f"{key}.bin") if os.path.exists(binp): os.replace(binp, os.path.join(STAGING_ROUTINES, f"{key}.bin")) return f"{key}.bin" except Exception as e: print("emoji_bin fail", key, emoji, e, flush=True) return "" ``` - [ ] **Step 6:** Vérif manuelle (réseau + LVGLImage). Sur Tower (vérifier d'abord que `LVGLImage.py` existe : `ls -l $LISAEL_LVGLIMG` ou `ls -l /home/clems/lisael-content/LVGLImage.py`) : `cd /home/clems/lisael-content && python3 -c "import lisael_content as C; print(C.emoji_bin('☀️','matin'))"` → doit imprimer `matin.bin` et le fichier doit exister dans le staging routines, taille > 1 Ko. Commit : `feat(tower): twemoji to RGB565A8 bin helper`. --- ### Task 3: Modèle routines — `routines_load()` / `routines_save()` + seed défaut (TDD) **Files:** - `tower/lisael_content.py` (ajout) - `tower/test_routines.py` (ajout de tests) **Interfaces:** - Produces : `def routines_load() -> list` — lit `ROUTINES_JSON` (vérité éditoriale Tower), seed défaut Matin/Soir si absent. - Produces : `def routines_save(routines: list) -> None` — `_atomic_write_json` vers `ROUTINES_JSON`. - Consumes : env `LISAEL_BASE` (chemin `ROUTINES_JSON = BASE/routines.json`). Format de la vérité éditoriale Tower (`routines.json` à la racine `BASE`, distinct du manifest SD `STAGING_ROUTINES/routines.json`) — porte l'emoji source par étape, qui n'apparaît PAS dans le manifest SD : ```json {"routines":[ {"id":"matin","title":"Le matin","emoji":"☀️","steps":[ {"text":"Habille-toi","emoji":"\U0001F455"}, {"text":"Petit-déjeuner","emoji":"\U0001F950"}]}, {"id":"soir","title":"Le soir","emoji":"\U0001F319","steps":[ {"text":"Brosse tes dents","emoji":"\U0001FAA5"}]}]} ``` - [ ] **Step 1 (RED):** Ajouter dans `tower/test_routines.py` : ```python def test_routines_seed_and_roundtrip(): r = C.routines_load() # seeds default if missing ids = [x["id"] for x in r] assert ids == ["matin", "soir"] assert os.path.exists(C.ROUTINES_JSON) assert all("emoji" in x and "title" in x and "steps" in x for x in r) r[0]["steps"].append({"text": "Cartable", "emoji": "\U0001F392"}) C.routines_save(r) r2 = C.routines_load() assert r2[0]["steps"][-1]["text"] == "Cartable" ``` - [ ] **Step 2 (RED run):** `cd tower && python3 test_routines.py` → échoue (`ROUTINES_JSON` / `routines_load` absents). - [ ] **Step 3 (GREEN):** Ajouter dans `lisael_content.py` (constante près de `FEEDS_JSON`, fonctions dans une section `# ---- routines (truth) ----`) : ```python ROUTINES_JSON = os.path.join(BASE, "routines.json") DEFAULT_ROUTINES = [ {"id": "matin", "title": "Le matin", "emoji": "☀️", "steps": [ {"text": "Habille-toi", "emoji": "\U0001F455"}, {"text": "Petit-déjeuner", "emoji": "\U0001F950"}, {"text": "Brosse tes dents", "emoji": "\U0001FAA5"}, {"text": "Prépare ton cartable", "emoji": "\U0001F392"}]}, {"id": "soir", "title": "Le soir", "emoji": "\U0001F319", "steps": [ {"text": "Mets ton pyjama", "emoji": "\U0001F476"}, {"text": "Brosse tes dents", "emoji": "\U0001FAA5"}, {"text": "Lis une histoire", "emoji": "\U0001F4D6"}]}, ] def routines_load(): if os.path.exists(ROUTINES_JSON): try: return json.load(open(ROUTINES_JSON, encoding="utf-8")).get("routines", []) except Exception: pass routines_save([dict(r) for r in DEFAULT_ROUTINES]) return [dict(r) for r in DEFAULT_ROUTINES] def routines_save(routines): _atomic_write_json(ROUTINES_JSON, {"routines": routines}) ``` - [ ] **Step 4 (GREEN run):** `cd tower && python3 test_routines.py` → `ok test_emoji_cp_strips_vs16`, `ok test_routines_seed_and_roundtrip`, `ALL OK`. Commit : `feat(tower): routines truth model load and save`. --- ### Task 4: `build_routine_manifest()` — assets en cache + manifest SD (TDD avec assets factices) **Files:** - `tower/lisael_content.py` (ajout) - `tower/test_routines.py` (ajout de tests) **Interfaces:** - Produces : `def asset_key(rid: str, idx: int) -> str` — base ASCII d'une étape : `f"{rid}_{idx:02d}"` (idx 1-based). Pure, testable. - Produces : `def build_routine_manifest(gen_bin=None, gen_audio=None) -> dict` — pour chaque routine/étape régénère picto `.bin` (si emoji changé) + audio `.mp3` (si texte changé), écrit le manifest SD `STAGING_ROUTINES/routines.json` au format firmware, renvoie `{"generated": n_assets, "skipped": n_cached, "routines": }`. `gen_bin`/`gen_audio` injectables (TDD : factices ; défaut = `emoji_bin` / TTS). Cache via index `STAGING_ROUTINES/.cache.json` (clé asset → `{"emoji":..., "text":...}`). - Consumes : `routines_load()`, `emoji_bin`, `tts_speak`, `STAGING_ROUTINES`, `_atomic_write_json`. Le manifest SD produit (format que le firmware parse — chemins `.bin`/`.mp3` relatifs, PAS d'emoji) : ```json {"routines":[{"id":"matin","title":"Le matin","icon":"matin.bin","steps":[ {"text":"Habille-toi","icon":"matin_01.bin","audio":"matin_01.mp3"}, ...]}, ...]} ``` - [ ] **Step 1 (RED):** Ajouter dans `tower/test_routines.py` : ```python def test_asset_key(): assert C.asset_key("matin", 1) == "matin_01" assert C.asset_key("soir", 12) == "soir_12" def test_build_routine_manifest_caches(): # deterministic content, fake generators that count calls and "write" files C.routines_save([{"id": "matin", "title": "Le matin", "emoji": "☀️", "steps": [{"text": "Habille-toi", "emoji": "\U0001F455"}]}]) os.makedirs(C.STAGING_ROUTINES, exist_ok=True) calls = {"bin": 0, "audio": 0} def fake_bin(emoji, key): calls["bin"] += 1 open(os.path.join(C.STAGING_ROUTINES, key + ".bin"), "wb").close() return key + ".bin" def fake_audio(text, key): calls["audio"] += 1 open(os.path.join(C.STAGING_ROUTINES, key + ".mp3"), "wb").close() return key + ".mp3" r1 = C.build_routine_manifest(gen_bin=fake_bin, gen_audio=fake_audio) # 1 step => 1 step bin + 1 audio + 1 routine icon bin = 2 bin, 1 audio assert calls == {"bin": 2, "audio": 1} man = r1["routines"] assert man[0]["icon"] == "matin.bin" assert man[0]["steps"][0] == {"text": "Habille-toi", "icon": "matin_01.bin", "audio": "matin_01.mp3"} assert os.path.exists(os.path.join(C.STAGING_ROUTINES, "routines.json")) # second build, unchanged content => everything cached, no regeneration calls2 = {"bin": 0, "audio": 0} def fb(e, k): calls2["bin"] += 1; return k + ".bin" def fa(t, k): calls2["audio"] += 1; return k + ".mp3" C.build_routine_manifest(gen_bin=fb, gen_audio=fa) assert calls2 == {"bin": 0, "audio": 0} ``` - [ ] **Step 2 (RED run):** `cd tower && python3 test_routines.py` → échoue (`asset_key` / `build_routine_manifest` absents). - [ ] **Step 3 (GREEN):** Ajouter dans `lisael_content.py` : ```python ROUTINES_CACHE = lambda: os.path.join(STAGING_ROUTINES, ".cache.json") def asset_key(rid, idx): return f"{rid}_{idx:02d}" def _routine_cache_load(): p = ROUTINES_CACHE() if os.path.exists(p): try: return json.load(open(p, encoding="utf-8")) except Exception: pass return {} def build_routine_manifest(gen_bin=None, gen_audio=None): """Regenerate per-step .bin (emoji) + .mp3 (TTS) into STAGING_ROUTINES, skipping assets whose (emoji|text) is unchanged, then write the SD manifest. gen_bin(emoji, key)->bin_name ; gen_audio(text, key)->mp3_name (injectable).""" gen_bin = gen_bin or emoji_bin gen_audio = gen_audio or _gen_step_audio os.makedirs(STAGING_ROUTINES, exist_ok=True) cache = _routine_cache_load() new_cache = {} generated = skipped = 0 manifest = [] for r in routines_load(): rid = r["id"] # routine icon (emoji -> .bin), key = rid icon_key = rid c = cache.get(icon_key) if c and c.get("emoji") == r.get("emoji") and os.path.exists( os.path.join(STAGING_ROUTINES, icon_key + ".bin")): icon = icon_key + ".bin"; skipped += 1 else: icon = gen_bin(r.get("emoji", ""), icon_key) or (icon_key + ".bin"); generated += 1 new_cache[icon_key] = {"emoji": r.get("emoji")} steps = [] for i, st in enumerate(r.get("steps", []), start=1): k = asset_key(rid, i) prev = cache.get(k) or {} bin_name = k + ".bin"; mp3_name = k + ".mp3" # picto if prev.get("emoji") == st.get("emoji") and os.path.exists( os.path.join(STAGING_ROUTINES, bin_name)): skipped += 1 else: gen_bin(st.get("emoji", ""), k); generated += 1 # audio if prev.get("text") == st.get("text") and os.path.exists( os.path.join(STAGING_ROUTINES, mp3_name)): skipped += 1 else: gen_audio(st.get("text", ""), k); generated += 1 new_cache[k] = {"emoji": st.get("emoji"), "text": st.get("text")} steps.append({"text": st.get("text", ""), "icon": bin_name, "audio": mp3_name}) manifest.append({"id": rid, "title": r.get("title", rid), "icon": icon, "steps": steps}) _atomic_write_json(os.path.join(STAGING_ROUTINES, "routines.json"), {"routines": manifest}) _atomic_write_json(ROUTINES_CACHE(), new_cache) return {"generated": generated, "skipped": skipped, "routines": manifest} def _gen_step_audio(text, key): """Default audio generator: TTS -> mono mp3 in STAGING_ROUTINES. '' on failure.""" try: au = tts_speak(text) _atomic_write_bytes(os.path.join(STAGING_ROUTINES, key + ".mp3"), au) return key + ".mp3" except Exception as e: print("step audio fail", key, e, flush=True) return "" ``` Note : la sortie gateway est déjà mono mp3 ; pas de re-transcode `ffmpeg` nécessaire ici (le contrat « mono 48k » est satisfait par la voix `fr` de la gateway). Si un jour la gateway renvoie du stéréo, brancher `transcode` dans `_gen_step_audio`. - [ ] **Step 4 (GREEN run):** `cd tower && python3 test_routines.py` → tous les `ok ...` + `ALL OK`. Commit : `feat(tower): build routine manifest with asset cache`. --- ### Task 5: Staging routines + `push_file(subdir)` généralisé + delta routines (TDD) **Files:** - `tower/lisael_content.py` (modif + ajout) - `tower/test_routines.py` (ajout de tests) **Interfaces:** - Produces : constante `STAGING_ROUTINES` (env `LISAEL_STAGING_ROUTINES`, défaut `BASE/routines`). - Produces : `def routines_staging_files() -> list` — basenames triés de `STAGING_ROUTINES` (hors cachés, hors `.cache.json`). - Modifie : `def push_file(ip, name, subdir="podcasts") -> int` — POST vers `?name=/`, lit depuis le bon staging (`podcasts` → `STAGING`, `routines` → `STAGING_ROUTINES`). Défaut `"podcasts"` = compat totale. - Produces : `def compute_routines_delta(have) -> list` — `compute_delta(routines_staging_files(), have)` (le manifest SD `routines.json` part en dernier). - Consumes : `compute_delta` (réutilisé tel quel : il traite `manifest.json` comme nom de manifest — voir Step 2 pour l'adaptation au nom `routines.json`). - [ ] **Step 1 (RED):** Ajouter dans `tower/test_routines.py` : ```python def test_routines_delta_manifest_last(): os.makedirs(C.STAGING_ROUTINES, exist_ok=True) for f in ("matin.bin", "matin_01.bin", "matin_01.mp3", "routines.json"): open(os.path.join(C.STAGING_ROUTINES, f), "wb").close() d = C.compute_routines_delta([]) assert d[-1] == "routines.json" # manifest always last assert set(d[:-1]) == {"matin.bin", "matin_01.bin", "matin_01.mp3"} # box already has assets -> only manifest moves if assets changed; here all present assert C.compute_routines_delta(["matin.bin", "matin_01.bin", "matin_01.mp3"]) == [] ``` - [ ] **Step 2 (RED run):** `cd tower && python3 test_routines.py` → échoue (`STAGING_ROUTINES`/`compute_routines_delta` absents). - [ ] **Step 3 (GREEN — constante + staging):** Ajouter la constante près de `STAGING` : ```python STAGING_ROUTINES = os.environ.get("LISAEL_STAGING_ROUTINES", os.path.join(BASE, "routines")) ``` Et la fonction (section routines) : ```python def routines_staging_files(): if not os.path.isdir(STAGING_ROUTINES): return [] return sorted(f for f in os.listdir(STAGING_ROUTINES) if not f.startswith(".")) ``` - [ ] **Step 4 (GREEN — delta):** Le `compute_delta` existant traite littéralement `"manifest.json"` comme manifest. Pour réutiliser la logique « manifest en dernier » avec un nom paramétrable, ajouter : ```python def compute_delta_named(staging, have, manifest_name): haveset = set(have) files = [f for f in staging if f != manifest_name and f not in haveset] if files and manifest_name in staging: files.append(manifest_name) return files def compute_routines_delta(have): return compute_delta_named(routines_staging_files(), have, "routines.json") ``` (Ne PAS toucher `compute_delta` — il reste la version podcasts `manifest.json` pour la compat et `test_delta.py`.) - [ ] **Step 5 (GREEN — push_file subdir):** Modifier `push_file` : ```python def push_file(ip, name, subdir="podcasts"): src_dir = STAGING_ROUTINES if subdir == "routines" else STAGING data = open(os.path.join(src_dir, name), "rb").read() url = f"http://{ip}:{BOX_PORT}/put?name={subdir}/{name}" with urllib.request.urlopen(urllib.request.Request(url, data=data, method="POST"), timeout=300) as r: return r.status ``` - [ ] **Step 6 (GREEN run):** `cd tower && python3 test_routines.py` (delta) ET `cd tower && python3 test_delta.py` (régression podcasts → doit toujours afficher `OK`). Commit : `feat(tower): routines staging push and delta`. --- ### Task 6: Journal des transferts — `transfers_begin/mark/end` + persistance (TDD) **Files:** - `tower/lisael_content.py` (ajout) - `tower/test_transfers.py` (nouveau fichier de test) **Interfaces:** - Produces : `transfers_begin(box_ip, files)` — `files = [(name, subdir, size)]` ou `[(name, subdir)]` (taille via `os.path.getsize` si absente) ; initialise le lot actif (tous `pending`). - Produces : `transfers_mark(name, status, error=None)` — `status ∈ {"sending","done","failed"}` ; au passage `done|failed` calcule `duration_s` (monotone, depuis le passage `sending`) et `rate_bps = size/duration_s`. - Produces : `transfers_end()` — vide le lot actif, flush les enregistrements `done|failed` dans `TRANSFERS_JSON` (borné aux ~200 derniers, `_atomic_write_json`). - Produces : `transfers_active() -> dict|None`, `transfers_history(limit=200) -> list`. - Consumes : env `LISAEL_TRANSFERS` (défaut `BASE/transfers.json`), `_atomic_write_json`, `time.monotonic`. Enregistrement par fichier : ```json {"name":"matin_01.mp3","dir":"routines","size":428111,"status":"done", "started":"2026-06-21T09:12:03","ended":"2026-06-21T09:12:07", "duration_s":4.1,"rate_bps":104417,"error":null,"box_ip":"192.168.0.250"} ``` - [ ] **Step 1 (RED):** Créer `tower/test_transfers.py` : ```python import os, tempfile, time, json os.environ["LISAEL_BASE"] = tempfile.mkdtemp(prefix="transfers_test_") os.environ["LISAEL_STAGING"] = os.path.join(os.environ["LISAEL_BASE"], "podcasts") os.environ["LISAEL_STAGING_ROUTINES"] = os.path.join(os.environ["LISAEL_BASE"], "routines") os.environ["LISAEL_TRANSFERS"] = os.path.join(os.environ["LISAEL_BASE"], "transfers.json") import lisael_content as C def test_begin_mark_end_cycle(): C.transfers_begin("192.168.0.250", [("a.mp3", "routines", 1000), ("routines.json", "routines", 50)]) a = C.transfers_active() assert a["box_ip"] == "192.168.0.250" and a["total"] == 2 and a["current"] == 0 assert all(f["status"] == "pending" for f in a["files"]) C.transfers_mark("a.mp3", "sending") assert C.transfers_active()["current"] == 1 time.sleep(0.05) C.transfers_mark("a.mp3", "done") f = next(x for x in C.transfers_active()["files"] if x["name"] == "a.mp3") assert f["status"] == "done" and f["duration_s"] > 0 and f["rate_bps"] > 0 C.transfers_mark("routines.json", "sending") C.transfers_mark("routines.json", "failed", "boom") C.transfers_end() assert C.transfers_active() is None hist = C.transfers_history() names = {h["name"]: h for h in hist} assert names["a.mp3"]["status"] == "done" assert names["routines.json"]["status"] == "failed" and names["routines.json"]["error"] == "boom" def test_history_truncates_200(): for i in range(250): C.transfers_begin("1.2.3.4", [(f"f{i}.mp3", "routines", 10)]) C.transfers_mark(f"f{i}.mp3", "sending"); C.transfers_mark(f"f{i}.mp3", "done") C.transfers_end() assert len(C.transfers_history()) == 200 # the most recent survive assert any(h["name"] == "f249.mp3" for h in C.transfers_history()) if __name__ == "__main__": for name, fn in sorted(globals().items()): if name.startswith("test_"): fn(); print("ok", name) print("ALL OK") ``` - [ ] **Step 2 (RED run):** `cd tower && python3 test_transfers.py` → échoue (`transfers_*` absents). - [ ] **Step 3 (GREEN):** Ajouter `import time` en tête de `lisael_content.py` (sur la ligne d'imports stdlib existante) puis une section `# ---- transfers journal ----` : ```python TRANSFERS_JSON = os.environ.get("LISAEL_TRANSFERS", os.path.join(BASE, "transfers.json")) TRANSFERS_MAX = 200 _active = None # active batch dict or None _active_lock = threading.Lock() def _now_iso(): return time.strftime("%Y-%m-%dT%H:%M:%S") def transfers_begin(box_ip, files): """files: list of (name, subdir[, size]); size resolved via getsize if missing.""" global _active recs = [] for it in files: name, subdir = it[0], it[1] size = it[2] if len(it) > 2 else None if size is None: src = STAGING_ROUTINES if subdir == "routines" else STAGING try: size = os.path.getsize(os.path.join(src, name)) except OSError: size = 0 recs.append({"name": name, "dir": subdir, "size": size, "status": "pending", "started": None, "ended": None, "duration_s": None, "rate_bps": None, "error": None, "box_ip": box_ip, "_t0": None}) with _active_lock: _active = {"box_ip": box_ip, "started": _now_iso(), "total": len(recs), "current": 0, "files": recs} def transfers_mark(name, status, error=None): with _active_lock: if not _active: return for f in _active["files"]: if f["name"] != name: continue if status == "sending": f["status"] = "sending"; f["started"] = _now_iso(); f["_t0"] = time.monotonic() _active["current"] += 1 elif status in ("done", "failed"): f["status"] = status; f["ended"] = _now_iso(); f["error"] = error if f.get("_t0") is not None: dur = max(time.monotonic() - f["_t0"], 0.001) f["duration_s"] = round(dur, 3) f["rate_bps"] = int((f["size"] or 0) / dur) return def transfers_active(): with _active_lock: if not _active: return None # strip internal _t0 for the API view files = [{k: v for k, v in f.items() if k != "_t0"} for f in _active["files"]] return {**_active, "files": files} def _transfers_history_load(): if os.path.exists(TRANSFERS_JSON): try: return json.load(open(TRANSFERS_JSON, encoding="utf-8")) except Exception: pass return [] def transfers_history(limit=TRANSFERS_MAX): return _transfers_history_load()[-limit:] def transfers_end(): global _active with _active_lock: if not _active: return done = [{k: v for k, v in f.items() if k != "_t0"} for f in _active["files"] if f["status"] in ("done", "failed")] _active = None hist = _transfers_history_load() hist.extend(done) hist = hist[-TRANSFERS_MAX:] _atomic_write_json(TRANSFERS_JSON, hist) ``` - [ ] **Step 4 (GREEN run):** `cd tower && python3 test_transfers.py` → `ok test_begin_mark_end_cycle`, `ok test_history_truncates_200`, `ALL OK`. Commit : `feat(tower): transfers journal begin mark end`. --- ### Task 7: Instrumenter `push_batch` + `try_push(subdir)` + `retry_failed` (TDD pour la sélection) **Files:** - `tower/lisael_content.py` (modif + ajout) - `tower/test_transfers.py` (ajout de tests) **Interfaces:** - Modifie : `def push_batch(ip, delta, subdir="podcasts")` — `transfers_begin` au début (chaque fichier `(f, subdir)`), `transfers_mark(f,"sending")` avant `push_file`, `transfers_mark(f,"done"/"failed",err)` après, `transfers_end` à la fin ; passe `subdir` à `push_file`. - Modifie : `def try_push(ip, delta, subdir="podcasts")` — passe `subdir` au thread `push_batch`. - Produces : `def failed_names(box_ip) -> list` — derniers fichiers `failed` connus pour `box_ip` (lot actif sinon historique récent), dédupliqués, en gardant le `dir`. Pure-ish (lit l'état), testable. - Produces : `def retry_failed(box_ip) -> dict` — collecte `failed_names`, relance `try_push` par sous-dossier ; renvoie `{"started": bool, "count": n}`. `409`-able côté API si `count == 0`. - [ ] **Step 1 (RED):** Ajouter dans `tower/test_transfers.py` (test de sélection des échecs sans réseau) : ```python def test_failed_names_from_history(): # seed a finished batch with one failed file for a box C.transfers_begin("9.9.9.9", [("ok.mp3", "routines", 10), ("bad.mp3", "routines", 10)]) C.transfers_mark("ok.mp3", "sending"); C.transfers_mark("ok.mp3", "done") C.transfers_mark("bad.mp3", "sending"); C.transfers_mark("bad.mp3", "failed", "net") C.transfers_end() fn = C.failed_names("9.9.9.9") assert ("bad.mp3", "routines") in fn assert ("ok.mp3", "routines") not in fn # unknown box -> nothing assert C.failed_names("0.0.0.0") == [] ``` - [ ] **Step 2 (RED run):** `cd tower && python3 test_transfers.py` → échoue (`failed_names` absent). - [ ] **Step 3 (GREEN — failed_names + retry_failed):** Ajouter (section transfers) : ```python def failed_names(box_ip): """Failed (name, dir) for box_ip: active batch first, else recent history. Skips names already re-sent successfully later in history.""" seen, failed = {}, [] a = transfers_active() pool = (a["files"] if a and a.get("box_ip") == box_ip else []) \ + [h for h in transfers_history() if h.get("box_ip") == box_ip] # later success overrides earlier failure: iterate chronological, keep last status status_by = {} for h in pool: status_by[h["name"]] = (h["status"], h.get("dir", "podcasts")) out = [] for name, (st, d) in status_by.items(): if st == "failed": out.append((name, d)) return out def retry_failed(box_ip): fails = failed_names(box_ip) if not fails: return {"started": False, "count": 0} by_dir = {} for name, d in fails: by_dir.setdefault(d, []).append(name) started = False for d, names in by_dir.items(): if try_push(box_ip, names, subdir=d): started = True return {"started": started, "count": len(fails)} ``` Note : `transfers_active()["files"]` est ordonné, et l'historique est chronologique (append), donc `status_by` garde bien le dernier statut connu par nom. - [ ] **Step 4 (GREEN — instrument push):** Modifier `push_batch` + `try_push` : ```python def push_batch(ip, delta, subdir="podcasts"): transfers_begin(ip, [(f, subdir) for f in delta]) ok = 0 for f in delta: transfers_mark(f, "sending") try: push_file(ip, f, subdir=subdir); ok += 1 transfers_mark(f, "done") except Exception as e: transfers_mark(f, "failed", str(e)[:200]) print("push fail", f, e, flush=True) transfers_end() print(f"push done {ip} {subdir} {ok}/{len(delta)}", flush=True) with _busy_lock: _busy.discard(ip) def try_push(ip, delta, subdir="podcasts"): """Start a background push unless one already runs for ip. Returns started?""" if not delta: return False with _busy_lock: if ip in _busy: return False _busy.add(ip) threading.Thread(target=push_batch, args=(ip, delta, subdir), daemon=True).start() return True ``` - [ ] **Step 5 (GREEN run):** `cd tower && python3 test_transfers.py` → tous `ok` + `ALL OK`. Régression : `cd tower && python3 test_delta.py` → `OK`. Commit : `feat(tower): instrument push and retry failed`. --- ### Task 8: API `/api/routines` (GET/POST) + `/api/routines/push` dans `lisael_server.py` **Files:** - `tower/lisael_server.py` (modif `do_GET` / `do_POST`) **Interfaces:** - Produces : `GET /api/routines` → `routines_load()` (vérité éditoriale, avec emojis). - Produces : `POST /api/routines` body `{"routines":[...]}` → `routines_save` + `build_routine_manifest` en thread daemon → `202 {"queued": n_steps}`. - Produces : `POST /api/routines/push` → `compute_routines_delta(have)` + `try_push(ip, delta, subdir="routines")` → `{"ip","delta","started"}` ou `409` si pas d'IP. - Consumes : `box_state_load()`, `routines_load/save`, `build_routine_manifest`, `compute_routines_delta`, `try_push`. - [ ] **Step 1:** Dans `do_GET`, après la branche `if path == "/api/box":` (avant `self.send_error(404)`), ajouter : ```python if path == "/api/routines": self._json({"routines": C.routines_load()}); return ``` - [ ] **Step 2:** Dans `do_POST`, après la branche `/api/push` (avant `self.send_error(404)`), ajouter la sauvegarde + régénération en thread (pattern `/api/select`) : ```python if path == "/api/routines": try: b = json.loads(self._body() or b"{}") routines = b["routines"] assert isinstance(routines, list) except Exception as e: self._json({"error": str(e)}, 400); return C.routines_save(routines) n_steps = sum(len(r.get("steps", [])) for r in routines) def work(): try: res = C.build_routine_manifest() print("routines built", res["generated"], "gen", res["skipped"], "cached", flush=True) except Exception as e: print("routines build fail", e, flush=True) threading.Thread(target=work, daemon=True).start() self._json({"queued": n_steps}, 202); return if path == "/api/routines/push": st = C.box_state_load() ip = st.get("ip") if not ip: self._json({"error": "box never registered"}, 409); return delta = C.compute_routines_delta(st.get("have", [])) started = C.try_push(ip, delta, subdir="routines") self._json({"ip": ip, "delta": len(delta), "started": started}); return ``` - [ ] **Step 3:** Mettre à jour le handler `/register` pour que le delta routines soit aussi poussé au boot. Dans `do_POST` branche `/register`, après `started = C.try_push(ip, delta)` ajouter le push routines (la box annonce déjà l'union des basenames dans `have`) : ```python r_delta = C.compute_routines_delta(have) if r_delta: C.try_push(ip, r_delta, subdir="routines") self._json({"queued": len(delta) + len(r_delta), "started": started}); return ``` (Remplace la ligne `self._json({"queued": len(delta), "started": started}); return` existante.) - [ ] **Step 4:** Vérif manuelle locale (pas de box requise). Lancer le serveur en local avec auth désactivée et staging temporaire : ```bash cd tower && LISAEL_BASE=/tmp/merlin_v2 LISAEL_STAGING=/tmp/merlin_v2/podcasts \ LISAEL_STAGING_ROUTINES=/tmp/merlin_v2/routines LISAEL_PORT=9556 \ python3 lisael_server.py & sleep 1 curl -s localhost:9556/api/routines | head -c 400 # -> seed matin/soir curl -s -X POST localhost:9556/api/routines -H 'Content-Type: application/json' \ -d '{"routines":[{"id":"matin","title":"Le matin","emoji":"☀️","steps":[{"text":"Test","emoji":"👕"}]}]}' -i | head -1 # -> 202 curl -s -X POST localhost:9556/api/routines/push -i | head -1 # -> 409 (no box) JSON kill %1 ``` (Le build manifest tentera un vrai TTS/Twemoji ; sur grosmac sans accès gateway il échouera proprement — vérifier que l'API répond bien `202` quoi qu'il en soit, le build est en thread.) Commit : `feat(tower): routines api save build push`. --- ### Task 9: API `/api/transfers` (GET) + `/api/transfers/retry` (POST) **Files:** - `tower/lisael_server.py` (modif `do_GET` / `do_POST`) **Interfaces:** - Produces : `GET /api/transfers` → `{"active": transfers_active(), "history": transfers_history()}`. - Produces : `POST /api/transfers/retry` → `retry_failed(box_ip)` ; `202 {"started","count"}` si `count>0`, sinon `409 {"error": "..."}` (pas d'IP ou rien à relancer). - Consumes : `box_state_load()`, `transfers_active/history`, `retry_failed`. - [ ] **Step 1:** Dans `do_GET`, après la branche `/api/routines` (Task 8), ajouter : ```python if path == "/api/transfers": self._json({"active": C.transfers_active(), "history": C.transfers_history()}); return ``` - [ ] **Step 2:** Dans `do_POST`, après la branche `/api/routines/push` (Task 8), ajouter : ```python if path == "/api/transfers/retry": st = C.box_state_load() ip = st.get("ip") if not ip: self._json({"error": "box never registered"}, 409); return res = C.retry_failed(ip) if not res["count"]: self._json({"error": "rien a relancer"}, 409); return self._json(res, 202); return ``` - [ ] **Step 3:** Vérif manuelle locale (reprend le serveur de la Task 8) : ```bash cd tower && LISAEL_BASE=/tmp/merlin_v2 LISAEL_STAGING=/tmp/merlin_v2/podcasts \ LISAEL_STAGING_ROUTINES=/tmp/merlin_v2/routines LISAEL_PORT=9556 \ python3 lisael_server.py & sleep 1 curl -s localhost:9556/api/transfers # -> {"active":null,"history":[]} curl -s -X POST localhost:9556/api/transfers/retry -i | head -1 # -> 409 JSON kill %1 ``` Commit : `feat(tower): transfers api and retry endpoint`. --- ### Task 10: UI onglet « 🗓️ Routines » dans `merlin_ui.html` **Files:** - `tower/merlin_ui.html` (modif : tabs, panneau, JS) **Interfaces:** - Consumes : `GET /api/routines`, `POST /api/routines`, `POST /api/routines/push` (via le helper `api()` existant). - Produces : édition en mémoire de la liste routines + rendu (2 sections Matin/Soir, étapes texte+emoji, +/−, ↑↓, Enregistrer & pousser). - [ ] **Step 1:** Ajouter le bouton d'onglet dans `.tabs` (après le bouton `tab-box`) : ```html ``` - [ ] **Step 2:** Ajouter le panneau (après `
...
`, avant la fermeture `` de `.wrap`) : ```html
chargement…
``` - [ ] **Step 3:** Étendre `TABS` et `showTab` : ```javascript const TABS = ["podcasts", "fichiers", "box", "routines", "transfers"]; ``` et dans `showTab`, ajouter : ```javascript if (name === "routines") loadRoutines(); ``` (la branche `transfers` sera câblée en Task 11 ; on déclare déjà l'entrée TABS ici pour éviter un second diff sur la même ligne.) - [ ] **Step 4:** Ajouter une palette d'emojis enfant + l'état en mémoire (dans le `