prima-cpp/CHANGES.md

Introduction

This markdown file is used to demonstrate the change i made compare to original prima.cpp

Configurable Network Port Options

Enable separate configuration of local listening ports and remote node ports for distributed inference, supporting complex networking scenarios including port forwarding.

• commits (1e7ae71)

New CLI Arguments

Local Listening Ports:

• --data_port PORT: Local port where this node listens for data communications
• --signal_port PORT: Local port where this node listens for signal communications

Remote Node Ports (ports that other nodes are listening on):

• --master_data_port PORT: Port that master node is listening on for data
• --next_node_data_port PORT: Port that next node is listening on for data
• --next_node_signal_port PORT: Port that next node is listening on for signals

Port Configuration Logic

Prima.cpp separates local binding from remote addressing:

• Local ports (--data_port, --signal_port): Where this node listens for incoming connections
• Remote ports (--master_*_port, --next_node_*_port): Which ports other nodes are listening on

This design enables port forwarding scenarios where nodes may listen on different ports locally but are accessed through forwarded ports.

Usage Examples

# On rank 0 (master server), run:
./llama-server -m gguf/qwen2.5-7b-instruct-q4_k_m-00001-of-00002.gguf --host 0.0.0.0 --port 8080 --world 2 --rank 0 --data_port 9000 --signal_port 9001 --master 127.0.0.1 --master_data_port 9000 --next 192.168.4.10 --next_node_data_port 9000 --next_node_signal_port 9001

# On rank 1 (worker node), run:
./llama-cli -m gguf/qwen2.5-7b-instruct-q4_k_m-00001-of-00002.gguf --world 2 --rank 1 --data_port 9000 --signal_port 9001 --master 192.168.4.9 --master_data_port 9000 --next 192.168.4.9 --next_node_data_port 9000 --next_node_signal_port 9001

This provides complete control over both local binding and remote connectivity for distributed model inference.

Rank-Specific GGUF Split Loading

Enable selective loading of GGUF split files based on layer assignment, reducing memory usage and download requirements for distributed inference.

• commits (f1f7e37)

New CLI Argument

• --splits SPLIT_LIST: Comma-separated list of split indices to load (e.g., "0,2,3")

Split Loading Logic

When models are split using gguf-split:

./gguf-split --split-max-tensors 128 llama.gguf llama.gguf

You get multiple files:

llama.gguf-00001-of-00004.gguf
llama.gguf-00002-of-00004.gguf  
llama.gguf-00003-of-00004.gguf
llama.gguf-00004-of-00004.gguf

Unlike llama.cpp which loads all splits, prima.cpp enables selective loading based on layer assignment:

• Each rank only loads splits containing its assigned layers
• Reduces memory footprint and network transfer
• Eliminates need to download unused model segments

Layer Assignment Policy

Prima.cpp enforces a specific layer distribution:

• Rank 0 (master): Always owns embedding layers + final transformer blocks
• Rank 1+ (workers): Own contiguous middle transformer blocks
• Rank 1: Always starts from layer 0

Split Selection Strategy

For --n-layer-window "8,8" with --world 2:

• Rank 0: Owns embedding + layers 8-15 → loads splits 0,2,3
• Rank 1: Owns layers 0-7 → loads splits 0,1

Usage Examples

# Rank 0 (master) loads splits containing embedding + final layers
./llama-cli -m llama.gguf-00001-of-00004.gguf --splits 0,2,3 --world 2 --rank 0 --n-layer-window "8,8"

# Rank 1 (worker) loads splits containing initial layers  
./llama-cli -m llama.gguf-00001-of-00004.gguf --splits 0,1 --world 2 --rank 1 --n-layer-window "8,8"

Error Handling

• Missing required tensors trigger immediate error with split suggestions
• Explicitly set --n-layer-window to ensure predictable layer-to-split mapping
• Validate split coverage before distributed execution

This optimization significantly reduces resource requirements for large model inference across multiple nodes.

Communication and Compute Logging

Enable detailed timestamped logging of inter-node communication and computation phases for performance analysis and debugging distributed inference. This feature can be controlled with a CLI flag to reduce overhead when not needed.

• commits (eb0cac1, adad23d, a399e49)

CLI Control

• --enable-comm-compute-log: Enable communication and computation logging (disabled by default for performance)

Logging Categories

Communication Logs:

• [comm][start/end][send_tensors]: Tensor transmission to next node/master
• [comm][start/end][recv_tensors]: Tensor reception from other nodes

Compute Logs:

• [compute][start/end][transformer_blocks]: Layer computation timing
• [compute][start/end][output_linear]: Final output layer processing

Log Format

Each log entry contains:

[rank][timestamp][category][phase][operation][batch_info, description]

• rank: Node rank identifier (0 for master, 1+ for workers)
• timestamp: ISO 8601 timestamp with millisecond precision
• category: comm for communication, compute for computation
• phase: start or end of operation
• operation: Specific function being logged
• batch_info: sbatch_tokens and ubatch_tokens counts
• description: Human-readable operation description

Use Cases

• Performance profiling: Identify communication vs compute bottlenecks
• Gantt chart generation: Visualize parallel execution timeline
• Debugging: Track tensor flow and synchronization issues
• Load balancing: Analyze per-node utilization patterns

Example Output

[0][2025-08-02T12:06:35.050Z][comm][start][send_tensors][sbatch_tokens: 0, ubatch_tokens: 512, send the result to the next node or the master]
[0][2025-08-02T12:06:35.051Z][comm][end][send_tensors][sbatch_tokens: 0, ubatch_tokens: 512, send the result to the next node or the master]
[0][2025-08-02T12:06:35.051Z][comm][start][recv_tensors][sbatch_tokens: 0, ubatch_tokens: 512, receive data from other nodes]
[0][2025-08-02T12:06:35.160Z][comm][end][recv_tensors][sbatch_tokens: 0, ubatch_tokens: 512, receive data from other nodes]
[0][2025-08-02T12:06:35.162Z][compute][start][transformer_blocks][sbatch_tokens: 0, ubatch_tokens: 512]
[0][2025-08-02T12:06:35.165Z][compute][end][transformer_blocks][sbatch_tokens: 0, ubatch_tokens: 512]
[0][2025-08-02T12:06:35.190Z][compute][start][output_linear][sbatch_tokens: 0, ubatch_tokens: 512]
[0][2025-08-02T12:06:35.259Z][compute][end][output_linear][sbatch_tokens: 0, ubatch_tokens: 512]

This logging system provides comprehensive visibility into distributed inference execution patterns.

Communication Tensor Dumping

Enable binary dumping of inter-node tensor communications with comprehensive metadata for detailed analysis of distributed inference data flow.

• commits (e48e955)

New CLI Argument

• --dump-folder FOLDER: Specify directory to dump network communication tensors (disabled if unset)

Binary Dump Format

Each tensor file contains structured binary data with complete shape metadata:

[1 byte: element_type][8 bytes: n_embed][8 bytes: n_tokens][8 bytes: tensor_size][tensor_data]

Header Fields:

• element_type: Data type identifier (FLOAT32 = 0, extensible for other types)
• n_embed: Embedding dimension/width (uint64_t)
• n_tokens: Token count/height (uint64_t)
• tensor_size: Total tensor data size in bytes (uint64_t)
• tensor_data: Raw tensor bytes in native format

Dumping Operations

Send Path (llama_send_tensors):

• Files: send_{counter}.bin
• Shape extracted from tensors->sub_gf_out->ne[]
• Captures outbound tensor data before network transmission

Receive Path (llama_recv_tensors):

• Files: recv_{counter}.bin
• Shape extracted from dims[] parameter
• Captures inbound tensor data after network reception

Usage Examples

# Enable tensor dumping for distributed inference
./llama-cli --model model.gguf --dump-folder ./tensor_dumps --world 2 --rank 0 [other_args]

This feature provides comprehensive visibility into the tensor communication layer for distributed model analysis.

Multi-Node Perplexity Evaluation

Enable distributed perplexity evaluation using llama-perplexity as master coordinator with llama-cli worker nodes for large model assessment.

• commits (c949e53)

Distributed Perplexity Architecture

Master Node (Rank 0):

• Runs llama-perplexity binary
• Coordinates text processing and perplexity calculations
• Manages input text file distribution and result aggregation

Worker Nodes (Rank 1+):

• Run llama-cli binaries in distributed mode
• Process assigned model layers during evaluation
• Participate in tensor communication pipeline

Supported Evaluation Features

• Text file processing: Load evaluation datasets via -f parameter
• Distributed inference: Leverage multi-node processing for large models
• Standard perplexity metrics: Calculate perplexity scores across distributed architecture

Usage Examples

2-Node Setup:

# On rank 0 (master evaluator), run:
./llama-perplexity -m download/qwq-32b-q4_k_m.gguf -f wikitext-2-raw/wiki.test.raw --world 2 --rank 0 --master 192.168.1.2 --next 192.168.1.3

# On rank 1 (worker node), run:
./llama-cli -m download/qwq-32b-q4_k_m.gguf --world 2 --rank 1 --master 192.168.1.2 --next 192.168.1.2

Communication Compression

Enable configurable compression of inter-node tensor communication to reduce bandwidth usage while maintaining flexibility in data type handling. The compression implementation uses BitSqueeze.

• Quantization commits (ff310e7–3a97a82)
• Sparsity commits (45a78f2–9e730dd)
• BitSqueeze commit (73969b4)
• Threshold commit (73969b4)

Purpose

This feature allows each rank to specify the data type used when sending tensors across nodes. Each rank is responsible only for compressing its own outgoing data. On the receiving side, the system automatically inspects the metadata to determine whether the payload is quantized and requires dequantization.

A new CLI argument (--comm_datatype TYPE) sets the communication data type. Supported values:

• f32 (default, no compression)
• integer based (q8_0, q4_0, q2_k, q2_k_fast, iq2_s, iq2_xs, iq2_xxs)
• float based (f32, bf16, fp16, fp8, fp4, mxfp8, mxfp4, nvfp4, nf4, nf4_dq)
• f32_sparsity (no quantization, but allows --comm_sparse_percentage to select the top-k features for each token based on the specified sparsity percentage)

A new CLI argument, comm_compression_threshold N, sets the tokens threshold before compression. Supported values:

• must be larger than negative one
• zero, will send the data based on comm_datatype no matter what
• any value larger than zero, will determine whether to compress to comm_datatype based on the current processed tensors, for example in decoding the number of tokens is small, so if this value is set to ten it might only compress during the prefilling stage, the real situation will depend on your ubatch size and prompt size

Communication Logic

• Send path

  • If f32, tensors are transmitted directly.
  • If other type, tensors are compressed before transmission.

• Receive path

  • Metadata indicates whether the payload is raw or quantized.
  • Compressed payloads are automatically dequantized to float32.

This design ensures that each rank manages compression only for its own outputs, while reception remains transparent.

Example Usage

Rank 0 (send tensors in q8_0 format)

./llama-cli --model model.gguf --world 2 --rank 0 --comm_datatype q8_0

Rank 1 (receive q8_0 data, auto-dequantize to f32, send tensors in q8_0 format)

./llama-cli --model model.gguf --world 2 --rank 1 --comm_datatype q8_0

By default, communication uses f32 with no quantization.

Sparsity Example

./llama-cli --model model.gguf --world 2 --rank 0 --comm_datatype f32_sparsity --comm_sparse_percentage 20

Quantization and Dequantization Timing

To measure the overhead of quantization and dequantization, enable:

--enable-comm-compute-log

This option produces timestamped logs that mark the start and end of quantization and dequantization operations for profiling and debugging.