rltakashigeandGitHub
e23c3a3026
## Motivation Users were reporting GPU timeout errors on Mac Minis, which we never saw on testing with Mac Studios. It also seems to only happen with large models. ## Changes Eval specific distributed operations. ## Why It Works As I wrote in a Slack message: Basically, prefill is too slow for pipeline communications. If there are both communications and GPU operations as part of an mlx graph, the communications become subject to the GPU's 5 second command buffer timeout. For normal generation, I added evals to the communications (only during prefill, as it slows down decode) to do this, fixing GPU timeouts. But we don't do this during warmup, as the prompt is absolutely tiny. This is still too slow on an M4 Pro on some models that it causes a GPU timeout during warmup... ---------------------- This was one of the issues. However, there is another issue: mx.all_gather sometimes reads stale data with FAST_SYNCH enabled. I'm still investigating the root cause, but the code as it is now works on Mac Minis. ## Test Plan ### Manual Testing <img width="2762" height="1808" alt="image" src="https://github.com/user-attachments/assets/27c88542-606c-4551-8f7c-bd2c0471f54e" /> <img width="2820" height="1898" alt="image" src="https://github.com/user-attachments/assets/0ba3478c-ee39-438d-902c-92893db23d05" /> ### Automated Testing Needs a bunch on mac minis
56 lines
1.6 KiB
Bash
Executable File
56 lines
1.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
[ $# -lt 1 ] && {
|
|
echo "Usage: $0 host1 [host2 ...]"
|
|
exit 1
|
|
}
|
|
|
|
[ -z "$(git status --porcelain)" ] || {
|
|
echo "Uncommitted changes"
|
|
exit 1
|
|
}
|
|
|
|
commit=$(git rev-parse HEAD)
|
|
git fetch -q origin
|
|
git branch -r --contains "$commit" | grep -qE '^\s*origin/' || {
|
|
echo "Not pushed to origin"
|
|
exit 1
|
|
}
|
|
hosts=("$@")
|
|
|
|
for host; do
|
|
ssh -T -o BatchMode=yes -o ServerAliveInterval=30 "$host@$host" \
|
|
"EXO_LIBP2P_NAMESPACE=$commit /nix/var/nix/profiles/default/bin/nix build github:exo-explore/exo/$commit" &
|
|
done
|
|
wait
|
|
|
|
cleanup() {
|
|
for host in "${hosts[@]}"; do
|
|
ssh -T -o BatchMode=yes "$host@$host" "pkill -f bin/exo" &
|
|
done
|
|
sleep 1
|
|
jobs -pr | xargs -r kill 2>/dev/null || true
|
|
}
|
|
trap 'cleanup' EXIT INT TERM
|
|
|
|
for host; do
|
|
ssh -T -o BatchMode=yes -o ServerAliveInterval=30 "$host@$host" \
|
|
"EXO_LIBP2P_NAMESPACE=$commit /nix/var/nix/profiles/default/bin/nix run github:exo-explore/exo/$commit" &>/dev/null &
|
|
done
|
|
|
|
for host; do
|
|
echo "Waiting for $host..." 1>&2
|
|
until curl -sf "http://$host:52415/models" &>/dev/null; do sleep 1; done
|
|
done
|
|
|
|
echo "Waiting 30s for cluster setup" 1>&2
|
|
sleep 30
|
|
echo "EXO loaded" 1>&2
|
|
bench_runner="${hosts[0]}"
|
|
mkdir -p "./bench/$commit"
|
|
nix run .#exo-get-all-models-on-cluster -- "$bench_runner" | while IFS= read -r model; do
|
|
echo "running bench for $model" 1>&2
|
|
ssh -Tn -o BatchMode=yes -o ServerAliveInterval=30 "$bench_runner@$bench_runner" "/nix/var/nix/profiles/default/bin/nix run github:exo-explore/exo/$commit#exo-bench -- --model $model --pp 128 4096 --tg 128 --stdout --skip-tensor-ring" >>"./bench/$commit/${model//\//--}.json"
|
|
echo
|
|
done
|