Scalability and overhead benchmarks

We conducted a series of microbenchmarks where we evaluated the scalability of Ray Tune and analyzed the performance overhead we observed. The results from these benchmarks are reflected in the documentation, e.g. when we make suggestions on how to remove performance bottlenecks.

This page gives an overview over the experiments we did. For each of these experiments, the goal was to examine the total runtime of the experiment and address issues when the observed overhead compared to the minimal theoretical time was too high (e.g. more than 20% overhead).

In some of the experiments we tweaked the default settings for maximum throughput, e.g. by disabling trial synchronization or result logging. If this is the case, this is stated in the respective benchmark description.

Ray Tune scalability benchmarks overview

Variable

# of trials

Results/second /trial

# of nodes

# CPUs/node

Trial length (s)

Observed runtime

Trial bookkeeping /scheduling overhead

10,000

1

1

16

1

715.27
(625 minimum)

Result throughput (many trials)

1,000

0.1

16

64

100

168.18

Result throughput (many results)

96

10

1

96

100

168.94

Network communication overhead

200

1

200

2

300

2280.82

Long running, 3.75 GB checkpoints

16

Results: 1/60
Checkpoint: 1/900

1

16

86,400

88687.41

XGBoost parameter sweep

16

?

16

64

?

3903

Durable trainable

16

10/60
with 10MB CP

16

2

300

392.42

Below we discuss some insights on results where we observed much overhead.

Result throughput

Result throughput describes the number of results Ray Tune can process in a given timeframe (e.g. “results per second”). The higher the throughput, the more concurrent results can be processed without major delays.

Result throughput is limited by the time it takes to process results. When a trial reports results, it only continues training once the trial executor re-triggered the remote training function. If many trials report results at the same time, each subsequent remote training call is only triggered after handling that trial’s results.

To speed the process up, Ray Tune adaptively buffers results, so that trial training is continued earlier if many trials are running in parallel and report many results at the same time. Still, processing hundreds of results per trial for dozens or hundreds of trials can become a bottleneck.

Main insight: Ray Tune will throw a warning when trial processing becomes a bottleneck. If you notice that this becomes a problem, please follow our guidelines outlined in the FAQ. Generally, it is advised to not report too many results at the same time. Consider increasing the report intervals by a factor of 5-10x.

Below we present more detailed results on the result throughput performance.

Many concurrent trials

In this setup, loggers (CSV, JSON, and TensorboardX) and trial synchronization are disabled, except when explicitly noted.

In this experiment, we’re running many concurrent trials (up to 1,000) on a cluster. We then adjust the reporting frequency (number of results per second) of the trials to measure the throughput limits.

It seems that around 500 total results/second seem to be the threshold for acceptable performance when logging and synchronization are disabled. With logging enabled, around 50-100 results per second can still be managed without too much overhead, but after that measures to decrease incoming results should be considered.

# of trials

Results / second / trial

# Nodes

# CPUs / Node

Length of trial.

Current

1,000

10

16

64

100s

248.39

1,000

1

16

64

100s

175.00

1,000

0.1 with logging

16

64

100s

168.18

384

10

16

64

100s

125.17

256

50

16

64

100s

307.02

256

20

16

64

100s

146.20

256

10

16

64

100s

113.40

256

10 with logging

16

64

100s

436.12

256

0.1 with logging

16

64

100s

106.75

Many results on a single node

In this setup, loggers (CSV, JSON, and TensorboardX) are disabled, except when explicitly noted.

In this experiment, we’re running 96 concurrent trials on a single node. We then adjust the reporting frequency (number of results per second) of the trials to find the throughput limits. Compared to the cluster experiment setup, we report much more often, as we’re running less total trials in parallel.

On a single node, throughput seems to be a bit higher. With logging, handling 1000 results per second seems acceptable in terms of overhead, though you should probably still target for a lower number.

# of trials

Results / second / trial

# Nodes

# CPUs / Node

Length of trial.

Current

96

500

1

96

100s

959.32

96

100

1

96

100s

219.48

96

80

1

96

100s

197.15

96

50

1

96

100s

110.55

96

50 with logging

1

96

100s

702.64

96

10

1

96

100s

103.51

96

10 with logging

1

96

100s

168.94

Network overhead

Running Ray Tune on a distributed setup leads to network communication overhead. This is mostly due to trial synchronization, where results and checkpoints are periodically synchronized and sent via the network. Per default this happens via SSH, where connnection initialization can take between 1 and 2 seconds each time. Since this is a blocking operation that happens on a per-trial basis, running many concurrent trials quickly becomes bottlenecked by this synchronization.

In this experiment, we ran a number of trials on a cluster. Each trial was run on a separate node. We varied the number of concurrent trials (and nodes) to see how much network communication affects total runtime.

Main insight: When running many concurrent trials in a distributed setup, consider using a ray.tune.durable for checkpoint synchronization instead. Another option would be to use a shared storage and disable syncing to driver. The best practices are described here for Kubernetes setups but is applicable for any kind of setup.

In the table below we present more detailed results on the network communication overhead.

# of trials

Results / second / trial

# Nodes

# CPUs / Node

Length of trial

Current

200

1

200

2

300s

2280.82

100

1

100

2

300s

1470

100

0.01

100

2

300s

473.41

50

1

50

2

300s

474.30

50

0.1

50

2

300s

441.54

10

1

10

2

300s

334.37