electron-rare/prima-cpp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/main' into merge/merge_86ca21e_from_upstream

Browse Source
This commit is contained in:
DandinPower
2025-08-07 20:19:18 +08:00
parent 96191d3b6f bdf9d8e74b
commit 7480ffbd30
17 changed files with 1632 additions and 429 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+43 -33
View File
@@ -34,26 +34,26 @@ And, if your devices are more powerful, you could unlock even more possibilities
> Device D4 runs inside a Termux-simulated Linux. Device D1 reads disk data in random mode and D2~D4 read in sequential mode.
**Table 2:** Token latency for Llama models (w/o device selection).
**Table 2:** Token latency for Llama models (with device selection).
| **Model** | **llama.cpp** | **exo** | **dllama** | **prima.cpp** |
|-----------------|---------------|-----------|------------|---------------|
| Llama 3-8B | **15 ms** | 263 ms | 459 ms | 54 ms |
| Llama 3-14B | **20 ms** | - | - | 65 ms |
|----------------|---------------|-----------|------------|---------------|
| Llama 3-8B | 15 ms | 263 ms | 459 ms | **15 ms** |
| Llama 3-14B | 20 ms | - | - | **20 ms** |
| Llama 1-30B | 202 ms | - | - | **72 ms** |
| Llama 3-45B | 328 ms | - | - | **233 ms** |
| Llama 3-60B | 7965 ms | - | - | **468 ms** |
| Llama 1-65B | 8807 ms | - | - | **569 ms** |
| Llama 3-70B | 10120 ms | OOM | OOM | **674 ms** |
**Table 3:** Token latency for Qwen 2.5, QwQ, and DeepSeek R1 models (w/o device selection).
**Table 3:** Token latency for Qwen 2.5, QwQ, and DeepSeek R1 models (with device selection).
| **Model** | **llama.cpp** | **exo** | **dllama** | **prima.cpp** |
|-----------------------------------|---------------|---------------|------------|---------------|
| Qwen-2.5-7B | **14 ms** | 86 ms | - | 44 ms |
| DeepSeek-R1-Distill-Qwen-7B | **14 ms** | 68 ms | - | 52 ms |
| DeepSeek-R1-Distill-Llama-8B | **14 ms** | 77 ms | 435 ms | 59 ms |
| Qwen-2.5-14B | **23 ms** | 31710 ms | - | 65 ms |
| DeepSeek-R1-Distill-Qwen-14B | **24 ms** | 23475 ms | - | 76 ms |
| Qwen-2.5-7B | 14 ms | 86 ms | - | **14 ms** |
| DeepSeek-R1-Distill-Qwen-7B | 14 ms | 68 ms | - | **14 ms** |
| DeepSeek-R1-Distill-Llama-8B | 14 ms | 77 ms | 435 ms | **14 ms** |
| Qwen-2.5-14B | 23 ms | 31710 ms | - | **23 ms** |
| DeepSeek-R1-Distill-Qwen-14B | 24 ms | 23475 ms | - | **24 ms** |
| Qwen-2.5-32B and QwQ-32B | 224 ms | OOM | - | **89 ms** |
| DeepSeek-R1-Distill-Qwen-32B | 232 ms | OOM | - | **93 ms** |
| DeepSeek-R1-Distill-Llama-70B | 10978 ms | OOM | - | **724 ms** |
@@ -61,9 +61,9 @@ And, if your devices are more powerful, you could unlock even more possibilities
> As video recording consumes some RAM, prima.cpp proactively reduces memory usage, resulting in slightly higher latency in the video compared to the table.
> In the old version (w/o device selection), each device is assigned at least one model layer. This would lead to a 1:1:29:1 split for Llama 3-8B, which makes prima.cpp slower than llama.cpp.
> ~~In the old version (w/o device selection), each device is assigned at least one model layer. This would lead to a 1:1:29:1 split for Llama 3-8B, which makes prima.cpp slower than llama.cpp.~~
>
> **New:** In the latest version (with device selection), we will have a 0:0:32:0 split and weak devices removed, then prima.cpp would become llama.cpp when serving small models.
> In the current version (with device selection), we will have a 32:0:0:0 split and weak devices removed, then prima.cpp would become llama.cpp when serving small models.
## 🔑 Key Features
@@ -72,8 +72,10 @@ And, if your devices are more powerful, you could unlock even more possibilities
- - **GPU & CPU Offloading:** If a device has a GPU, you can use both GPU and CPU for inference. For example, when VRAM is full, we can offload some model layers to RAM.
- - **Piped-ring parallelism with prefetching:** Prefetch upcoming layer weights to overlap disk loading latency and use advanced piped-ring parallelism to prevent the "prefetch-release" effect. This new parallelism improves pipeline parallelism by using a ring structure and allows devices to run multiple cycles to predict a new token.
- - **Heterogeneity-aware workload distribution:** A scheduler is designed to optimize workload distribution based on each device's computing power, disk speed, memory, and OS (the OS will affect the disk speed and the memory management strategy). It decides how many model layers a device should handle and how many should run on GPU (if available).
- - **Automatic device selection:** If there are weak devices and removing them would speed up inference, prima.cpp will automatically discover and remove them.
- - **Automatic device selection:** If there are weak devices and removing them would speed up inference, prima.cpp will automatically discover and remove them. This may retain some devices as proxy to prevent the socket connection from being blocked.
- - **Quantization:** We now support Q4K, Q6K, Q80 and IQ1 quantization (GGUF format) and are exploring a Q4K-IQ1 hybrid for a better balance between performance and speed.
- - **Speculative decoding:** We now support speculative decoding, which can [further speed up by up to 80%.](https://github.com/Lizonghang/prima.cpp/discussions/29)
- **Dynamic batching**: We now support concurrent requests from multiple users and batch decoding.
- **Support Models:** We now support hot models like the **Llama, Qwen (and QwQ), and DeepSeek series**. More will be added in future updates.
- **Cross-Platform:** The cluster can consist of devices with different OSs, including macOS, Linux, Android, HarmonyOS, etc. Now, Android and HarmonyOS devices require Termux, and Windows support will be added in future update.
@@ -120,7 +122,8 @@ Before using this project, ensure you have the following dependencies installed:
**Linux (e.g., Ubuntu):**
```shell
sudo apt update -y && sudo apt install -y gcc-9 make cmake fio git wget libzmq3-dev
# Use apt in Linux and pkg in Termux
sudo apt update -y && sudo apt install -y gcc-9 make cmake fio git wget libzmq3-dev curl
```
For HiGHS, download and install from [source](https://github.com/ERGO-Code/HiGHS):
@@ -132,12 +135,13 @@ mkdir build && cd build
cmake ..
make -j$(nproc)
sudo make install
sudo ldconfig
```
**macOS:**
```shell
brew install gcc make cmake fio git wget highs zeromq
brew install gcc make cmake fio git wget highs zeromq curl
```
### Build, Download, and Test
@@ -201,7 +205,7 @@ graph LR;
> **NOTE:** This ring communication is a communication overlay, not the physical topology. These devices are physically fully connected because they all connect to the same Wi-Fi.
> If possible, disable the firewall to prevent the ports needed (e.g., 9000, 10000) been blocked.
> If possible, disable the firewall to prevent the ports needed (e.g., 9000, 10000) been blocked, or you can use `--data-port` (9000, by default) and `--signal-port` (10000, by default) to customize the ports used.
Take QwQ-32B as an example, run the following commands on the devices to launch distributed inference:
@@ -222,6 +226,8 @@ Take QwQ-32B as an example, run the following commands on the devices to launch
Once started, prima.cpp will profile each device and decide how much workload to assign, e.g., how many model layers each device should handle, and how many of them should run on GPU (if available).
> By default, the output layer runs on the CPU. However, if you have enough total VRAM, add `--keep-out-in-cuda` to the master to run it on the GPU.
### Effecient Multi-Split GGUF Support in `prima.cpp`
If your model is split into multiple GGUF files using `gguf-split`, for example by running:
@@ -266,17 +272,15 @@ If a required tensor is missing from the specified splits, the loader will raise
### (Optional) Run with Prebuilt Docker Image
Assume we have a host machine with at least 32 CPU cores, 32 GiB RAM, and 32 GiB VRAM. We simulate 4 homogeneous nodes using Docker containers, with each node allocated 8 CPU cores, 8 GiB RAM, and 8 GiB VRAM. Follow the below steps to get started:
1. Pull our prebuilt Docker image (e.g., [`prima.cpp:1.0.1-cuda`](https://hub.docker.com/repository/docker/lizonghango00o1/prima.cpp/general)) and run 4 containers:
1. Pull our prebuilt Docker image (e.g., [`prima.cpp:1.0.2-cuda`](https://hub.docker.com/repository/docker/lizonghango00o1/prima.cpp/general)) and run 4 containers:
```shell
sudo docker run -dit --name prima-v1 --memory=8gb --memory-swap=8gb --cpus 8 --cpuset-cpus="0-7" --network host --gpus all prima.cpp:1.0.1-cuda
sudo docker run -dit --name prima-v2 --memory=8gb --memory-swap=8gb --cpus 8 --cpuset-cpus="8-15" --network host --gpus all prima.cpp:1.0.1-cuda
sudo docker run -dit --name prima-v3 --memory=8gb --memory-swap=8gb --cpus 8 --cpuset-cpus="16-23" --network host --gpus all prima.cpp:1.0.1-cuda
sudo docker run -dit --name prima-v4 --memory=8gb --memory-swap=8gb --cpus 8 --cpuset-cpus="24-31" --network host --gpus all prima.cpp:1.0.1-cuda
sudo docker run -dit --name prima-v1 --memory=8gb --memory-swap=8gb --cpus 8 --cpuset-cpus="0-7" --network host --gpus all prima.cpp:1.0.2-cuda
sudo docker run -dit --name prima-v2 --memory=8gb --memory-swap=8gb --cpus 8 --cpuset-cpus="8-15" --network host --gpus all prima.cpp:1.0.2-cuda
sudo docker run -dit --name prima-v3 --memory=8gb --memory-swap=8gb --cpus 8 --cpuset-cpus="16-23" --network host --gpus all prima.cpp:1.0.2-cuda
sudo docker run -dit --name prima-v4 --memory=8gb --memory-swap=8gb --cpus 8 --cpuset-cpus="24-31" --network host --gpus all prima.cpp:1.0.2-cuda
```
> If your host machine does not have a GPU, ignore the `--gpus all` option.
2. Download the model file [`qwq-32b-q4_k_m.gguf`](https://huggingface.co/Qwen/QwQ-32B-GGUF) and copy it into each container:
```shell
@@ -287,27 +291,27 @@ sudo docker cp qwq-32b-q4_k_m.gguf prima-v3:/root/prima.cpp/download/
sudo docker cp qwq-32b-q4_k_m.gguf prima-v4:/root/prima.cpp/download/
```
3. (Optional) Enter each container, rebuild prima.cpp if your host machine does not have a GPU:
3. Enter each container and build prima.cpp:
```shell
cd /root/prima.cpp && make clean
make -j$(nproc) # If not rank 0
make USE_HIGHS=1 -j$(nproc) # If rank 0
cd /root/prima.cpp
make GGML_CUDA=1 USE_HIGHS=1 -j$(nproc) # For rank 0
make GGML_CUDA=1 -j$(nproc) # For other ranks
```
4. Enter each container and launch the distributed inference:
```shell
cd /root/prima.cpp
(prima-v1) ./llama-cli -m download/qwq-32b-q4_k_m.gguf -c 1024 -n 256 -p "what is edge AI?" --world 4 --rank 0 --prefetch --gpu-mem 8
(prima-v2) ./llama-cli -m download/qwq-32b-q4_k_m.gguf -c 1024 --world 4 --rank 1 --prefetch --gpu-mem 8
(prima-v3) ./llama-cli -m download/qwq-32b-q4_k_m.gguf -c 1024 --world 4 --rank 2 --prefetch --gpu-mem 8
(prima-v4) ./llama-cli -m download/qwq-32b-q4_k_m.gguf -c 1024 --world 4 --rank 3 --prefetch --gpu-mem 8
(prima-v1) ./llama-cli -m download/qwq-32b-q4_k_m.gguf --world 4 --rank 0 --prefetch --gpu-mem 8 -c 4096 -n 256 -p "what is edge AI?"
(prima-v2) ./llama-cli -m download/qwq-32b-q4_k_m.gguf --world 4 --rank 1 --prefetch --gpu-mem 8
(prima-v3) ./llama-cli -m download/qwq-32b-q4_k_m.gguf --world 4 --rank 2 --prefetch --gpu-mem 8
(prima-v4) ./llama-cli -m download/qwq-32b-q4_k_m.gguf --world 4 --rank 3 --prefetch --gpu-mem 8
```
> If your host machine does not have a GPU, ignore the `--gpu-mem` option.
> You can ignore `--gpu-mem` if you don't want to limit VRAM usage.
> If you update to the latest code, non-rank 0 nodes can omit `-c 1024`.
> Always use `git fetch` to update the local repository.
### Run in Server Mode
You can run prima.cpp in server mode, by launching `llama-server` on the rank 0 device (with `--host` and `--port` specified) and `llama-cli` on the others. Here is an example with 2 devices:
@@ -320,6 +324,8 @@ You can run prima.cpp in server mode, by launching `llama-server` on the rank 0
./llama-cli -m download/qwq-32b-q4_k_m.gguf --world 2 --rank 1 --master 192.168.1.2 --next 192.168.1.2 --prefetch
```
You can specify `-np 4 --cont-batching` when launching `llama-server` to enable concurrent requests.
After that, you can interact with the rank 0 device by calling the Chat Completion API:
```shell
@@ -418,6 +424,10 @@ curl -X POST http://localhost:8080/v1/cancel \
-d '{"task_id": 0}'
```
**9. How to use speculative decoding?**
Please see "[Power prima.cpp with speculative decoding: Further speeds up by up to 80%](https://github.com/Lizonghang/prima.cpp/discussions/29)".
## ❤️ Acknowledgment
This project builds upon the incredible work from the open-source community, especially [ggml, gguf](https://github.com/ggml-org/ggml), and [llama.cpp](https://github.com/ggml-org/llama.cpp). We gratefully acknowledge their contributions.