diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ec88416f..20f6f9f9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -39,6 +39,119 @@ Write pure functions where possible. When adding new code, prefer Rust unless th Run `nix fmt` to auto-format your code before submitting. +## Model Cards + +EXO uses TOML-based model cards to define model metadata and capabilities. Model cards are stored in: +- `resources/inference_model_cards/` for text generation models +- `resources/image_model_cards/` for image generation models +- `~/.exo/custom_model_cards/` for user-added custom models + +### Adding a Model Card + +To add a new model, create a TOML file with the following structure: + +```toml +model_id = "mlx-community/Llama-3.2-1B-Instruct-4bit" +n_layers = 16 +hidden_size = 2048 +supports_tensor = true +tasks = ["TextGeneration"] +family = "llama" +quantization = "4bit" +base_model = "Llama 3.2 1B" +capabilities = ["text"] + +[storage_size] +in_bytes = 729808896 +``` + +### Required Fields + +- `model_id`: Hugging Face model identifier +- `n_layers`: Number of transformer layers +- `hidden_size`: Hidden dimension size +- `supports_tensor`: Whether the model supports tensor parallelism +- `tasks`: List of supported tasks (`TextGeneration`, `TextToImage`, `ImageToImage`) +- `family`: Model family (e.g., "llama", "deepseek", "qwen") +- `quantization`: Quantization level (e.g., "4bit", "8bit", "bf16") +- `base_model`: Human-readable base model name +- `capabilities`: List of capabilities (e.g., `["text"]`, `["text", "thinking"]`) + +### Optional Fields + +- `components`: For multi-component models (like image models with separate text encoders and transformers) +- `uses_cfg`: Whether the model uses classifier-free guidance (for image models) +- `trust_remote_code`: Whether to allow remote code execution (defaults to `false` for security) + +### Capabilities + +The `capabilities` field defines what the model can do: +- `text`: Standard text generation +- `thinking`: Model supports chain-of-thought reasoning +- `thinking_toggle`: Thinking can be enabled/disabled via `enable_thinking` parameter +- `image_edit`: Model supports image-to-image editing (FLUX.1-Kontext) + +### Security Note + +By default, `trust_remote_code` is set to `false` for security. Only enable it if the model explicitly requires remote code execution from the Hugging Face hub. + +## API Adapters + +EXO supports multiple API formats through an adapter pattern. Adapters convert API-specific request formats to the internal `TextGenerationTaskParams` format and convert internal token chunks back to API-specific responses. + +### Adapter Architecture + +All adapters live in `src/exo/master/adapters/` and follow the same pattern: + +1. Convert API-specific requests to `TextGenerationTaskParams` +2. Handle both streaming and non-streaming response generation +3. Convert internal `TokenChunk` objects to API-specific formats +4. Manage error handling and edge cases + +### Existing Adapters + +- `chat_completions.py`: OpenAI Chat Completions API +- `claude.py`: Anthropic Claude Messages API +- `responses.py`: OpenAI Responses API +- `ollama.py`: Ollama API (for OpenWebUI compatibility) + +### Adding a New API Adapter + +To add support for a new API format: + +1. Create a new adapter file in `src/exo/master/adapters/` +2. Implement a request conversion function: + ```python + def your_api_request_to_text_generation( + request: YourAPIRequest, + ) -> TextGenerationTaskParams: + # Convert API request to internal format + pass + ``` +3. Implement streaming response generation: + ```python + async def generate_your_api_stream( + command_id: CommandId, + chunk_stream: AsyncGenerator[TokenChunk | ErrorChunk | ToolCallChunk, None], + ) -> AsyncGenerator[str, None]: + # Convert internal chunks to API-specific streaming format + pass + ``` +4. Implement non-streaming response collection: + ```python + async def collect_your_api_response( + command_id: CommandId, + chunk_stream: AsyncGenerator[TokenChunk | ErrorChunk | ToolCallChunk, None], + ) -> AsyncGenerator[str]: + # Collect all chunks and return single response + pass + ``` +5. Register the adapter endpoints in `src/exo/master/api.py` + +The adapter pattern keeps API-specific logic isolated from core inference systems. Internal systems (worker, runner, event sourcing) only see `TextGenerationTaskParams` and `TokenChunk` objects - no API-specific types cross the adapter boundary. + +For detailed API documentation, see [docs/api.md](docs/api.md). + ## Testing EXO relies heavily on manual testing at this point in the project, but this is evolving. Before submitting a change, test both before and after to demonstrate how your change improves behavior. Do the best you can with the hardware you have available - if you need help testing, ask and we'll do our best to assist. Add automated tests where possible - we're actively working to substantially improve our automated testing story. diff --git a/README.md b/README.md index 785f6fba..f1cf9f39 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,8 @@ exo connects all your devices into an AI cluster. Not only does exo enable runni - **Topology-Aware Auto Parallel**: exo figures out the best way to split your model across all available devices based on a realtime view of your device topology. It takes into account device resources and network latency/bandwidth between each link. - **Tensor Parallelism**: exo supports sharding models, for up to 1.8x speedup on 2 devices and 3.2x speedup on 4 devices. - **MLX Support**: exo uses [MLX](https://github.com/ml-explore/mlx) as an inference backend and [MLX distributed](https://ml-explore.github.io/mlx/build/html/usage/distributed.html) for distributed communication. +- **Multiple API Compatibility**: Compatible with OpenAI Chat Completions API, Claude Messages API, OpenAI Responses API, and Ollama API - use your existing tools and clients. +- **Custom Model Support**: Load custom models from HuggingFace hub to expand the range of available models. ## Dashboard @@ -196,6 +198,8 @@ exo follows the [XDG Base Directory Specification](https://specifications.freede - **Configuration files**: `~/.config/exo/` (or `$XDG_CONFIG_HOME/exo/`) - **Data files**: `~/.local/share/exo/` (or `$XDG_DATA_HOME/exo/`) - **Cache files**: `~/.cache/exo/` (or `$XDG_CACHE_HOME/exo/`) +- **Log files**: `~/.cache/exo/exo_log/` (with automatic log rotation) +- **Custom model cards**: `~/.local/share/exo/custom_model_cards/` You can override these locations by setting the corresponding XDG environment variables. @@ -275,8 +279,47 @@ After that, RDMA will be enabled in macOS and exo will take care of the rest. --- +## Environment Variables + +exo supports several environment variables for configuration: + +| Variable | Description | Default | +|----------|-------------|---------| +| `EXO_MODELS_PATH` | Colon-separated paths to search for pre-downloaded models (e.g., on NFS mounts or shared storage) | None | +| `EXO_MODELS_DIR` | Directory where exo downloads and stores models | `~/.local/share/exo/models` (Linux) or `~/.exo/models` (macOS) | +| `EXO_OFFLINE` | Run without internet connection (uses only local models) | `false` | +| `EXO_ENABLE_IMAGE_MODELS` | Enable image model support | `false` | +| `EXO_LIBP2P_NAMESPACE` | Custom namespace for cluster isolation | None | +| `EXO_FAST_SYNCH` | Control MLX_METAL_FAST_SYNCH behavior (for JACCL backend) | Auto | +| `EXO_TRACING_ENABLED` | Enable distributed tracing for performance analysis | `false` | + +**Example usage:** + +```bash +# Use pre-downloaded models from NFS mount +EXO_MODELS_PATH=/mnt/nfs/models:/opt/ai-models uv run exo + +# Run in offline mode +EXO_OFFLINE=true uv run exo + +# Enable image models +EXO_ENABLE_IMAGE_MODELS=true uv run exo + +# Use custom namespace for cluster isolation +EXO_LIBP2P_NAMESPACE=my-dev-cluster uv run exo +``` + +--- + ### Using the API +exo provides multiple API-compatible interfaces for maximum compatibility with existing tools: + +- **OpenAI Chat Completions API** - Compatible with OpenAI clients +- **Claude Messages API** - Compatible with Anthropic's Claude format +- **OpenAI Responses API** - Compatible with OpenAI's Responses format +- **Ollama API** - Compatible with Ollama and tools like OpenWebUI + If you prefer to interact with exo via the API, here is an example creating an instance of a small model (`mlx-community/Llama-3.2-1B-Instruct-4bit`), sending a chat completions request and deleting the instance. --- @@ -366,14 +409,85 @@ When you're done, delete the instance by its ID (find it via `/state` or `/insta curl -X DELETE http://localhost:52415/instance/YOUR_INSTANCE_ID ``` +### Claude Messages API Compatibility + +Use the Claude Messages API format with the `/v1/messages` endpoint: + +```bash +curl -N -X POST http://localhost:52415/v1/messages \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "mlx-community/Llama-3.2-1B-Instruct-4bit", + "messages": [ + {"role": "user", "content": "Hello"} + ], + "max_tokens": 1024, + "stream": true + }' +``` + +### OpenAI Responses API Compatibility + +Use the OpenAI Responses API format with the `/v1/responses` endpoint: + +```bash +curl -N -X POST http://localhost:52415/v1/responses \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "mlx-community/Llama-3.2-1B-Instruct-4bit", + "messages": [ + {"role": "user", "content": "Hello"} + ], + "stream": true + }' +``` + +### Ollama API Compatibility + +exo supports Ollama API endpoints for compatibility with tools like OpenWebUI: + +```bash +# Ollama chat +curl -X POST http://localhost:52415/ollama/api/chat \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "mlx-community/Llama-3.2-1B-Instruct-4bit", + "messages": [ + {"role": "user", "content": "Hello"} + ], + "stream": false + }' + +# List models (Ollama format) +curl http://localhost:52415/ollama/api/tags +``` + +### Custom Model Loading from HuggingFace + +You can add custom models from the HuggingFace hub: + +```bash +curl -X POST http://localhost:52415/models/add \ + -H 'Content-Type: application/json' \ + -d '{ + "model_id": "mlx-community/my-custom-model" + }' +``` + +**Security Note:** + +Custom models requiring `trust_remote_code` in their configuration must be explicitly enabled (default is false) for security. Only enable this if you trust the model's remote code execution. Models are fetched from HuggingFace and stored locally as custom model cards. + **Other useful API endpoints*:** - List all models: `curl http://localhost:52415/models` +- List downloaded models only: `curl http://localhost:52415/models?status=downloaded` +- Search HuggingFace: `curl "http://localhost:52415/models/search?query=llama&limit=10"` - Inspect instance IDs and deployment state: `curl http://localhost:52415/state` For further details, see: -- API basic documentation in [docs/api.md](docs/api.md). +- API documentation in [docs/api.md](docs/api.md). - API types and endpoints in [src/exo/master/api.py](src/exo/master/api.py). --- @@ -432,4 +546,4 @@ On macOS, exo uses the GPU. On Linux, exo currently runs on CPU. We are working ## Contributing -See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on how to contribute to exo. \ No newline at end of file +See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on how to contribute to exo. diff --git a/docs/api.md b/docs/api.md index c25e3bc4..85d0cd3f 100644 --- a/docs/api.md +++ b/docs/api.md @@ -4,7 +4,7 @@ This document describes the REST API exposed by the **EXO** service, as implemen `src/exo/master/api.py` -The API is used to manage model instances in the cluster, inspect cluster state, and perform inference using an OpenAI-compatible interface. +The API is used to manage model instances in the cluster, inspect cluster state, and perform inference using multiple API-compatible interfaces. Base URL example: @@ -144,8 +144,59 @@ Placement result. Returns the list of available models and their metadata. +**Query parameters:** + +* `status`: string (optional) - Filter by `downloaded` to show only downloaded models + **Response:** -Array of model descriptors. +Array of model descriptors including `is_custom` field for custom HuggingFace models. + +### Add Custom Model + +**POST** `/models/add` + +Add a custom model from HuggingFace hub. + +**Request body (example):** + +```json +{ + "model_id": "mlx-community/my-custom-model" +} +``` + +**Response:** +Model descriptor for the added model. + +**Security note:** +Models with `trust_remote_code` enabled in their configuration require explicit opt-in (default is false) for security. + +### Delete Custom Model + +**DELETE** `/models/custom/{model_id}` + +Delete a user-added custom model card. + +**Path parameters:** + +* `model_id`: string, ID of the custom model to delete + +**Response:** +Confirmation JSON with deleted model ID. + +### Search Models + +**GET** `/models/search` + +Search HuggingFace Hub for mlx-community models. + +**Query parameters:** + +* `query`: string (optional) - Search query +* `limit`: integer (default: 20) - Maximum number of results + +**Response:** +Array of HuggingFace model search results. ## 4. Inference / Chat Completions @@ -168,9 +219,123 @@ Executes a chat completion request using an OpenAI-compatible schema. Supports s } ``` +**Request parameters:** + +* `model`: string, required - Model ID to use +* `messages`: array, required - Conversation messages +* `stream`: boolean (default: false) - Enable streaming responses +* `max_tokens`: integer (optional) - Maximum tokens to generate +* `temperature`: float (optional) - Sampling temperature +* `top_p`: float (optional) - Nucleus sampling parameter +* `top_k`: integer (optional) - Top-k sampling parameter +* `stop`: string or array (optional) - Stop sequences +* `seed`: integer (optional) - Random seed for reproducibility +* `enable_thinking`: boolean (optional) - Enable thinking mode for capable models (DeepSeek V3.1, Qwen3, GLM-4.7) +* `tools`: array (optional) - Tool definitions for function calling +* `logprobs`: boolean (optional) - Return log probabilities +* `top_logprobs`: integer (optional) - Number of top log probabilities to return + **Response:** OpenAI-compatible chat completion response. +**Streaming response format:** +When `stream=true`, returns Server-Sent Events (SSE) with format: + +``` +data: {"id":"...","object":"chat.completion","created":...,"model":"...","choices":[...]} + +data: [DONE] +``` + +**Non-streaming response includes usage statistics:** + +```json +{ + "id": "...", + "object": "chat.completion", + "created": 1234567890, + "model": "llama-3.2-1b", + "choices": [{ + "index": 0, + "message": { + "role": "assistant", + "content": "Hello! How can I help you?" + }, + "finish_reason": "stop" + }], + "usage": { + "prompt_tokens": 15, + "completion_tokens": 8, + "total_tokens": 23 + } +} +``` + +**Cancellation:** +You can cancel an active generation by closing the HTTP connection. The server detects the disconnection and stops processing. + +### Claude Messages API + +**POST** `/v1/messages` + +Executes a chat completion request using the Claude Messages API format. Supports streaming and non-streaming modes. + +**Request body (example):** + +```json +{ + "model": "llama-3.2-1b", + "messages": [ + { "role": "user", "content": "Hello" } + ], + "max_tokens": 1024, + "stream": false +} +``` + +**Streaming response format:** +When `stream=true`, returns Server-Sent Events with Claude-specific event types: + +* `message_start` - Message generation started +* `content_block_start` - Content block started +* `content_block_delta` - Incremental content chunk +* `content_block_stop` - Content block completed +* `message_delta` - Message metadata updates +* `message_stop` - Message generation completed + +**Response:** +Claude-compatible messages response. + +### OpenAI Responses API + +**POST** `/v1/responses` + +Executes a chat completion request using the OpenAI Responses API format. Supports streaming and non-streaming modes. + +**Request body (example):** + +```json +{ + "model": "llama-3.2-1b", + "messages": [ + { "role": "user", "content": "Hello" } + ], + "stream": false +} +``` + +**Streaming response format:** +When `stream=true`, returns Server-Sent Events with response-specific event types: + +* `response.created` - Response generation started +* `response.in_progress` - Response is being generated +* `response.output_item.added` - New output item added +* `response.output_item.done` - Output item completed +* `response.done` - Response generation completed + +**Response:** +OpenAI Responses API-compatible response. + ### Benchmarked Chat Completions **POST** `/bench/chat/completions` @@ -181,7 +346,13 @@ Same as `/v1/chat/completions`, but also returns performance and generation stat Same schema as `/v1/chat/completions`. **Response:** -Chat completion plus benchmarking metrics. +Chat completion plus benchmarking metrics including: + +* `prompt_tps` - Tokens per second during prompt processing +* `generation_tps` - Tokens per second during generation +* `prompt_tokens` - Number of prompt tokens +* `generation_tokens` - Number of generated tokens +* `peak_memory_usage` - Peak memory used during generation ### Cancel Command @@ -204,24 +375,148 @@ Cancels an active generation command (text or image). Notifies workers and close Returns 404 if the command is not found or already completed. -## 5. Image Generation & Editing +## 5. Ollama API Compatibility + +EXO provides Ollama API compatibility for tools like OpenWebUI. + +### Ollama Chat + +**POST** `/ollama/api/chat` +**POST** `/ollama/api/api/chat` (alias) +**POST** `/ollama/api/v1/chat` (alias) + +Execute a chat request using Ollama API format. + +**Request body (example):** + +```json +{ + "model": "llama-3.2-1b", + "messages": [ + { "role": "user", "content": "Hello" } + ], + "stream": false +} +``` + +**Response:** +Ollama-compatible chat response. + +### Ollama Generate + +**POST** `/ollama/api/generate` + +Execute a text generation request using Ollama API format. + +**Request body (example):** + +```json +{ + "model": "llama-3.2-1b", + "prompt": "Hello", + "stream": false +} +``` + +**Response:** +Ollama-compatible generation response. + +### Ollama Tags + +**GET** `/ollama/api/tags` +**GET** `/ollama/api/api/tags` (alias) +**GET** `/ollama/api/v1/tags` (alias) + +Returns list of downloaded models in Ollama tags format. + +**Response:** +Array of model tags with metadata. + +### Ollama Show + +**POST** `/ollama/api/show` + +Returns model information in Ollama show format. + +**Request body:** + +```json +{ + "name": "llama-3.2-1b" +} +``` + +**Response:** +Model details including modelfile and family. + +### Ollama PS + +**GET** `/ollama/api/ps` + +Returns list of running models (active instances). + +**Response:** +Array of active model instances. + +### Ollama Version + +**GET** `/ollama/api/version` +**HEAD** `/ollama/` (alias) +**HEAD** `/ollama/api/version` (alias) + +Returns version information for Ollama API compatibility. + +**Response:** + +```json +{ + "version": "exo v1.0" +} +``` + +## 6. Image Generation & Editing ### Image Generation **POST** `/v1/images/generations` -Executes an image generation request using an OpenAI-compatible schema with additional advanced_params. +Executes an image generation request using an OpenAI-compatible schema with additional advanced_params. Supports both streaming and non-streaming modes. **Request body (example):** ```json { "prompt": "a robot playing chess", - "model": "flux-dev", + "model": "exolabs/FLUX.1-dev", + "n": 1, + "size": "1024x1024", "stream": false, + "response_format": "b64_json" } ``` +**Request parameters:** + +* `prompt`: string, required - Text description of the image +* `model`: string, required - Image model ID +* `n`: integer (default: 1) - Number of images to generate +* `size`: string (default: "auto") - Image dimensions. Supported sizes: + - `512x512` + - `768x768` + - `1024x768` + - `768x1024` + - `1024x1024` + - `1024x1536` + - `1536x1024` + - `1024x1365` + - `1365x1024` +* `stream`: boolean (default: false) - Enable streaming for partial images +* `partial_images`: integer (default: 0) - Number of partial images to stream during generation +* `response_format`: string (default: "b64_json") - Either `url` or `b64_json` +* `quality`: string (default: "medium") - Either `high`, `medium`, or `low` +* `output_format`: string (default: "png") - Either `png`, `jpeg`, or `webp` +* `advanced_params`: object (optional) - Advanced generation parameters + **Advanced Parameters (`advanced_params`):** | Parameter | Type | Constraints | Description | @@ -231,8 +526,54 @@ Executes an image generation request using an OpenAI-compatible schema with addi | `guidance` | float | 1.0-20.0 | Classifier-free guidance scale | | `negative_prompt` | string | - | Text describing what to avoid in the image | +**Non-streaming response:** + +```json +{ + "created": 1234567890, + "data": [ + { + "b64_json": "iVBORw0KGgoAAAANSUhEUgAA...", + "url": null + } + ] +} +``` + +**Streaming response format:** +When `stream=true` and `partial_images > 0`, returns Server-Sent Events: + +``` +data: {"type":"partial","image_index":0,"partial_index":1,"total_partials":5,"format":"png","data":{"b64_json":"..."}} + +data: {"type":"final","image_index":0,"format":"png","data":{"b64_json":"..."}} + +data: [DONE] +``` + +### Image Editing + +**POST** `/v1/images/edits` + +Executes an image editing request (img2img) using FLUX.1-Kontext-dev or similar models. + +**Request (multipart/form-data):** + +* `image`: file, required - Input image to edit +* `prompt`: string, required - Text description of desired changes +* `model`: string, required - Image editing model ID (e.g., `exolabs/FLUX.1-Kontext-dev`) +* `n`: integer (default: 1) - Number of edited images to generate +* `size`: string (optional) - Output image dimensions +* `response_format`: string (default: "b64_json") - Either `url` or `b64_json` +* `input_fidelity`: string (default: "low") - Either `low` or `high` - Controls how closely the output follows the input image +* `stream`: string (default: "false") - Enable streaming +* `partial_images`: string (default: "0") - Number of partial images to stream +* `quality`: string (default: "medium") - Either `high`, `medium`, or `low` +* `output_format`: string (default: "png") - Either `png`, `jpeg`, or `webp` +* `advanced_params`: string (optional) - JSON-encoded advanced parameters + **Response:** -OpenAI-compatible image generation response. +Same format as `/v1/images/generations`. ### Benchmarked Image Generation @@ -244,16 +585,15 @@ Same as `/v1/images/generations`, but also returns generation statistics. Same schema as `/v1/images/generations`. **Response:** -Image generation plus benchmarking metrics. +Image generation plus benchmarking metrics including: -### Image Editing - -**POST** `/v1/images/edits` - -Executes an image editing request using an OpenAI-compatible schema with additional advanced_params (same as `/v1/images/generations`). - -**Response:** -Same format as `/v1/images/generations`. +* `seconds_per_step` - Average time per denoising step +* `total_generation_time` - Total generation time +* `num_inference_steps` - Number of inference steps used +* `num_images` - Number of images generated +* `image_width` - Output image width +* `image_height` - Output image height +* `peak_memory_usage` - Peak memory used during generation ### Benchmarked Image Editing @@ -267,37 +607,127 @@ Same schema as `/v1/images/edits`. **Response:** Same format as `/bench/images/generations`, including `generation_stats`. -## 6. Complete Endpoint Summary +### List Images + +**GET** `/images` + +List all stored images. + +**Response:** +Array of image metadata including URLs and expiration times. + +### Get Image + +**GET** `/images/{image_id}` + +Retrieve a stored image by ID. + +**Path parameters:** + +* `image_id`: string, ID of the image + +**Response:** +Image file with appropriate content type. + +## 7. Complete Endpoint Summary ``` +# General GET /node_id GET /state GET /events +# Instance Management POST /instance GET /instance/{instance_id} DELETE /instance/{instance_id} - GET /instance/previews GET /instance/placement POST /place_instance +# Models GET /models GET /v1/models +POST /models/add +DELETE /models/custom/{model_id} +GET /models/search +# Text Generation (OpenAI Chat Completions) POST /v1/chat/completions POST /bench/chat/completions + +# Text Generation (Claude Messages API) +POST /v1/messages + +# Text Generation (OpenAI Responses API) +POST /v1/responses + +# Text Generation (Ollama API) +POST /ollama/api/chat +POST /ollama/api/api/chat +POST /ollama/api/v1/chat +POST /ollama/api/generate +GET /ollama/api/tags +GET /ollama/api/api/tags +GET /ollama/api/v1/tags +POST /ollama/api/show +GET /ollama/api/ps +GET /ollama/api/version +HEAD /ollama/ +HEAD /ollama/api/version + +# Command Control POST /v1/cancel/{command_id} +# Image Generation POST /v1/images/generations POST /bench/images/generations POST /v1/images/edits POST /bench/images/edits +GET /images +GET /images/{image_id} ``` -## 7. Notes +## 8. Notes -* The `/v1/chat/completions` endpoint is compatible with the OpenAI Chat API format, so existing OpenAI clients can be pointed to EXO by changing the base URL. -* The `/v1/images/generations` and `/v1/images/edits` endpoints are compatible with the OpenAI Images API format. -* The instance placement endpoints allow you to plan and preview cluster allocations before actually creating instances. -* The `/events` and `/state` endpoints are primarily intended for operational visibility and debugging. +### API Compatibility + +EXO provides multiple API-compatible interfaces: + +* **OpenAI Chat Completions API** - Compatible with OpenAI clients and tools +* **Claude Messages API** - Compatible with Anthropic's Claude API format +* **OpenAI Responses API** - Compatible with OpenAI's Responses API format +* **Ollama API** - Compatible with Ollama and tools like OpenWebUI + +Existing OpenAI, Claude, or Ollama clients can be pointed to EXO by changing the base URL. + +### Custom Models + +You can add custom models from HuggingFace using the `/models/add` endpoint. Custom models are identified by the `is_custom` field in model list responses. + +**Security:** Models requiring `trust_remote_code` must be explicitly enabled (default is false) for security. Only enable this if you trust the model's remote code. + +### Usage Statistics + +Chat completion responses include usage statistics with: + +* `prompt_tokens` - Number of tokens in the prompt +* `completion_tokens` - Number of tokens generated +* `total_tokens` - Sum of prompt and completion tokens + +### Request Cancellation + +You can cancel active requests by: + +1. Closing the HTTP connection (for streaming requests) +2. Calling `/v1/cancel/{command_id}` (for any request) + +The server detects cancellation and stops processing immediately. + +### Instance Placement + +The instance placement endpoints allow you to plan and preview cluster allocations before creating instances. This helps optimize resource usage across nodes. + +### Observability + +The `/events` and `/state` endpoints are primarily intended for operational visibility and debugging. diff --git a/docs/architecture.md b/docs/architecture.md index 6f60a7b9..daf5d3f2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -28,6 +28,26 @@ There are currently 5 major systems: Implements a distributed algorithm for master election in unstable networking conditions +## API Layer + +The API system uses multiple adapters to support multiple API formats, converting them to a single request / response type. + +### Adapter Pattern + +Adapters convert between external API formats and EXO's internal types: + +``` +Chat Completions → [adapter] → TextGenerationTaskParams → Application +Claude Messages → [adapter] → TextGenerationTaskParams → Application +Responses API → [adapter] → TextGenerationTaskParams → Application +Ollama API → [adapter] → TextGenerationTaskParams → Application +``` + +Each adapter implements two key functions: +1. **Request conversion**: Converts API-specific requests to `TextGenerationTaskParams` +2. **Response generation**: Converts internal `TokenChunk` streams back to API-specific formats (streaming and non-streaming) + + ## Topics There are currently 5 topics: