A Guide To Parallelism and Resources

Parallelism is determined by per trial resources (defaulting to 1 CPU, 0 GPU per trial) and the resources available to Tune (ray.cluster_resources()).

By default, Tune automatically runs N concurrent trials, where N is the number of CPUs (cores) on your machine.

# If you have 4 CPUs on your machine, this will run 4 concurrent trials at a time.
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=10))
results = tuner.fit()

You can override this per trial resources with tune.with_resources. Here you can specify your resource requests using either a dictionary or a PlacementGroupFactory object. In any case, Ray Tune will try to start a placement group for each trial.

# If you have 4 CPUs on your machine, this will run 2 concurrent trials at a time.
tuner = tune.Tuner(tune.with_resources(trainable, {"cpu": 2}, tune_config=tune.TuneConfig(num_samples=10))
results = tuner.fit()

# If you have 4 CPUs on your machine, this will run 1 trial at a time.
tuner = tune.Tuner(tune.with_resources(trainable, {"cpu": 4}, tune_config=tune.TuneConfig(num_samples=10))
results = tuner.fit()

# Fractional values are also supported, (i.e., {"cpu": 0.5}).
tuner = tune.Tuner(tune.with_resources(trainable, {"cpu": 0.5}, tune_config=tune.TuneConfig(num_samples=10))
results = tuner.fit()

Tune will allocate the specified GPU and CPU as specified by tune.with_resources to each individual trial. Even if the trial cannot be scheduled right now, Ray Tune will still try to start the respective placement group. If not enough resources are available, this will trigger autoscaling behavior if you’re using the Ray cluster launcher.

It is also possible to specify memory ("memory", in bytes) and custom resource requirements.

If your trainable function starts more remote workers, you will need to pass so-called placement group factory objects to request these resources. See the PlacementGroupFactory documentation for further information. This also applies if you are using other libraries making use of Ray, such as Modin. Failure to set resources correctly may result in a deadlock, “hanging” the cluster.

Note

The resources specified this way will only be allocated for scheduling Tune trials. These resources will not be enforced on your objective function (Tune trainable) automatically. You will have to make sure your trainable has enough resources to run (e.g. by setting n_jobs for a scikit-learn model accordingly).

How to leverage GPUs?

To leverage GPUs, you must set gpu in tune.with_resources(trainable, resources_per_trial). This will automatically set CUDA_VISIBLE_DEVICES for each trial.

# If you have 8 GPUs, this will run 8 trials at once.
tuner = tune.Tuner(tune.with_resources(trainable, {"gpu": 1}, tune_config=tune.TuneConfig(num_samples=10))
results = tuner.fit()

# If you have 4 CPUs on your machine and 1 GPU, this will run 1 trial at a time.
tuner = tune.Tuner(tune.with_resources(trainable, {"cpu": 2, "gpu": 1}, tune_config=tune.TuneConfig(num_samples=10))
results = tuner.fit()

You can find an example of this in the Keras MNIST example.

Warning

If ‘gpu’ is not set, CUDA_VISIBLE_DEVICES environment variable will be set as empty, disallowing GPU access.

Troubleshooting: Occasionally, you may run into GPU memory issues when running a new trial. This may be due to the previous trial not cleaning up its GPU state fast enough. To avoid this, you can use tune.utils.wait_for_gpu - see docstring.

How to run distributed tuning on a cluster?

To attach to an existing Ray cluster, simply run ray.init before Tuner.fit(). See Starting Ray via the CLI (ray start) for more information about ray.init:

# Connect to an existing distributed Ray cluster
ray.init(address=<ray_address>)
tuner = tune.Tuner(tune.with_resources(trainable, tune.PlacementGroupFactory([{"CPU": 2, "GPU": 1}])), tune_config=tune.TuneConfig(num_samples=100))

Read more in the Tune distributed experiments guide.

How to run distributed training with Tune?

To tune distributed training jobs, you should use Ray AI Runtime (Ray AIR) to use Ray Tune and Ray Train in conjunction with each other. Ray Tune will run multiple trials in parallel, with each trial running distributed training with Ray Train.

How to limit concurrency?

If using a search algorithm, you may want to limit the number of trials that are being evaluated. For example, you may want to serialize the evaluation of trials to do sequential optimization.

In this case, ray.tune.search.ConcurrencyLimiter to limit the amount of concurrency:

algo = BayesOptSearch(utility_kwargs={
    "kind": "ucb",
    "kappa": 2.5,
    "xi": 0.0
})
algo = ConcurrencyLimiter(algo, max_concurrent=4)
scheduler = AsyncHyperBandScheduler()

Note

It is also possible to directly use tune.TuneConfig(max_concurrent_trials=4, ...), which is taken in by Tuner. This automatically wraps the underlying search algorithm in a ConcurrencyLimiter for you.

To understand concurrency limiting in depth, please see ConcurrencyLimiter (tune.search.ConcurrencyLimiter) for more details.