Profiling for Ray Developers#

This guide helps contributors to the Ray project analyze Ray performance.

Getting a stack trace of Ray C++ processes#

You can use the following GDB command to view the current stack trace of any running Ray process (e.g., raylet). This can be useful for debugging 100% CPU utilization or infinite loops (simply run the command a few times to see what the process is stuck on).

sudo gdb -batch -ex "thread apply all bt" -p <pid>

Note that you can find the pid of the raylet with pgrep raylet.


These instructions are for Ubuntu only. Attempts to get pprof to correctly symbolize on Mac OS have failed.

sudo apt-get install google-perftools libgoogle-perftools-dev

You may need to install graphviz for pprof to generate flame graphs.

sudo apt-get install graphviz

CPU profiling#

To launch Ray in profiling mode and profile Raylet, define the following variables:

export PERFTOOLS_PATH=/usr/lib/x86_64-linux-gnu/
export PERFTOOLS_LOGFILE=/tmp/pprof.out

The file /tmp/pprof.out is empty until you let the binary run the target workload for a while and then kill it via ray stop or by letting the driver exit.

Note: Enabling RAY_RAYLET_PERFTOOLS_PROFILER allows profiling of the Raylet component. To profile other modules, use RAY_{MODULE}_PERFTOOLS_PROFILER, where MODULE represents the uppercase form of the process type, such as GCS_SERVER.

Visualizing the CPU profile#

You can visualize the output of pprof in different ways. Below, the output is a zoomable .svg image displaying the call graph annotated with hot paths.

# Use the appropriate path.

google-pprof -svg $RAYLET /tmp/pprof.out > /tmp/pprof.svg
# Then open the .svg file with Chrome.

# If you realize the call graph is too large, use -focus=<some function> to zoom
# into subtrees.
google-pprof -focus=epoll_wait -svg $RAYLET /tmp/pprof.out > /tmp/pprof.svg

Below is a snapshot of an example SVG output, from the official documentation:

Memory profiling#

To run memory profiling on Ray core components, use Jemalloc (jemalloc/jemalloc). Ray supports environment variables to override LD_PRELOAD on core components.

You can find the component name from For example, if you’d like to profile gcs_server, search PROCESS_TYPE_GCS_SERVER in You can see the value is gcs_server.

Users are supposed to provide 3 env vars for memory profiling.

  • RAY_JEMALLOC_LIB_PATH: The path to the jemalloc shared library .so.

  • RAY_JEMALLOC_CONF: The MALLOC_CONF of jemalloc (comma separated).

  • RAY_JEMALLOC_PROFILE: Comma separated Ray components to run Jemalloc .so. e.g., (“raylet,gcs_server”). Note that the components should match the process type in (It means “RAYLET,GCS_SERVER” won’t work).

# Install jemalloc
tar -xf jemalloc-5.2.1.tar.bz2
cd jemalloc-5.2.1
./configure --enable-prof --enable-prof-libunwind
make install

# Set jemalloc configs through MALLOC_CONF env variable.
# Read
# for all jemalloc configs
# Ray start will profile the GCS server component.
RAY_JEMALLOC_CONF=prof:true,lg_prof_interval:33,lg_prof_sample:17,prof_final:true,prof_leak:true \
RAY_JEMALLOC_LIB_PATH=~/jemalloc-5.2.1/lib/ \
ray start --head

# You should see the following logs.
2021-10-20 19:45:08,175       INFO -- Jemalloc profiling will be used for gcs_server. env vars: {'LD_PRELOAD': '/Users/sangbincho/jemalloc-5.2.1/lib/', 'MALLOC_CONF': 'prof:true,lg_prof_interval:33,lg_prof_sample:17,prof_final:true,prof_leak:true'}

Visualizing the memory profile#

The output files are at the path where you call “ray start”. For the example of profile file “jeprof.15786.0.f.heap”, use following commands to generate the .svg plot.

# Use the appropriate path.

sudo jeprof $RAYLET jeprof.15786.0.f.heap --svg > /tmp/prof.svg
# Then open the .svg file with Chrome.

Running microbenchmarks#

To run a set of single-node Ray microbenchmarks, use:

ray microbenchmark

You can find the microbenchmark results for Ray releases in the GitHub release logs.