Profiling for Ray Developers¶
This document details, for Ray developers, how to analyze Ray performance.
Getting a stack trace of Ray C++ processes¶
You can use the following GDB command to view the current stack trace of any running Ray process (e.g., raylet). This can be useful for debugging 100% CPU utilization or infinite loops (simply run the command a few times to see what the process is stuck on).
sudo gdb -batch -ex "thread apply all bt" -p <pid>
Note that you can find the pid of the raylet with pgrep raylet.
Installation¶
These instructions are for Ubuntu only. Attempts to get pprof to correctly
symbolize on Mac OS have failed.
sudo apt-get install google-perftools libgoogle-perftools-dev
Launching the to-profile binary¶
If you want to launch Ray in profiling mode, define the following variables:
export RAYLET_PERFTOOLS_PATH=/usr/lib/x86_64-linux-gnu/libprofiler.so
export RAYLET_PERFTOOLS_LOGFILE=/tmp/pprof.out
The file /tmp/pprof.out will be empty until you let the binary run the
target workload for a while and then kill it via ray stop or by
letting the driver exit.
Visualizing the CPU profile¶
The output of pprof can be visualized in many ways. Here we output it as a
zoomable .svg image displaying the call graph annotated with hot paths.
# Use the appropriate path.
RAYLET=ray/python/ray/core/src/ray/raylet/raylet
google-pprof -svg $RAYLET /tmp/pprof.out > /tmp/pprof.svg
# Then open the .svg file with Chrome.
# If you realize the call graph is too large, use -focus=<some function> to zoom
# into subtrees.
google-pprof -focus=epoll_wait -svg $RAYLET /tmp/pprof.out > /tmp/pprof.svg
Here’s a snapshot of an example svg output, taken from the official documentation:
Running Microbenchmarks¶
To run a set of single-node Ray microbenchmarks, use:
ray microbenchmark
You can find the microbenchmark results for Ray releases in the GitHub release logs.
References¶
The pprof documentation.
The gperftools, including libprofiler, tcmalloc, and other goodies.