L'électron rare
a96780ba7e
- Created README_FR.md files for various standards and profiles in the `standards` directory, detailing conventions and usage. - Added integration documentation for ZeroClaw, AutoGen, LangGraph, and n8n in the `tools/ai/integrations` directory, outlining their roles and usage. - Introduced a comprehensive README_FR.md for the `tools/cockpit` directory, detailing operator commands and health-check routines. - Added a README_FR.md for the `tools/evals` directory, providing a benchmark framework for electronic prompts. - Created a README_FR.md for the `tools/hw/schops` directory, explaining CLI operations for schematic management. - Added README_FR.md files for the `web` directory and its `project/pcb` subdirectory, outlining the web application structure and PCB viewer inputs.
1.7 KiB
1.7 KiB
T-MA-021 — Benchmark Multi-Provider Mascarade
Benchmark comparatif base models vs fine-tuned sur prompts métier électronique.
Structure
evals/
├── benchmark_providers.py # Runner principal (Mistral, Anthropic, OpenAI)
├── prompts/
│ └── metier_100_template.jsonl # 20 prompts template (à étendre à 100)
└── results/ # Résultats des runs (gitignored)
Usage
# Dry run (pas d'appels API)
python benchmark_providers.py --prompts prompts/metier_100_template.jsonl --dry-run
# Run Mistral uniquement
python benchmark_providers.py --prompts prompts/metier_100_template.jsonl --providers mistral
# Run complet 3 providers
python benchmark_providers.py --prompts prompts/metier_100_template.jsonl --providers mistral,anthropic,openai
# Avec rate limiting adapté au free tier Anthropic (5 RPM)
python benchmark_providers.py --prompts prompts/metier_100_template.jsonl --providers anthropic --rate-limit 13
Format prompts (JSONL)
{"id": "K001", "domain": "kicad", "prompt": "...", "system": "...", "difficulty": "medium"}
Domaines couverts
- KiCad (K001-K040): Schéma, PCB, DRC, symboles, empreintes
- SPICE (S001-S030): Simulation, netlists, analyse AC/DC/transitoire
- Embedded (E001-E020): Firmware STM32/ESP32, drivers, protocoles
- Mixed (M001-M010): Architecture hardware+firmware complète
Env vars
export MISTRAL_API_KEY="708zM4biF4WjZIAROJh2roFiAPK9O7kG"
export ANTHROPIC_API_KEY="sk-ant-api03-..."
export OPENAI_API_KEY="sk-proj-..."