prepare_data_loader#

ray.train.torch.prepare_data_loader(data_loader: torch.utils.data.DataLoader, add_dist_sampler: bool = True, move_to_device: bool = True, auto_transfer: bool = True) → torch.utils.data.DataLoader[source]#

Prepares DataLoader for distributed execution.

This allows you to use the same exact code regardless of number of workers or the device type being used (CPU, GPU, TPU).

Note

This method adds a DistributedSampler to the DataLoader if the number of training workers is greater than 1. If shuffling is enabled on the original DataLoader, then shuffle=True will also be passed into the DistributedSampler constructor. shuffle=False on the original DataLoader also means that shuffling is disabled on the sampler.

With more than 1 worker, calling the DistributedSampler.set_epoch method at the beginning of each epoch before creating the DataLoader iterator is necessary to make shuffling work properly across multiple epochs. Otherwise, the same ordering will be always used. See: https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler # noqa: E501

Examples

>>> import torch
>>> import ray.train.torch
>>> train_dataloader = torch.utils.data.DataLoader(
...     ..., batch_size=..., shuffle=True
... )
>>> train_dataloader = ray.train.torch.prepare_data_loader(train_dataloader)
>>> for epoch in range(10):
...     if ray.train.get_context().get_world_size() > 1:
...         # Required for the distributed sampler to shuffle properly across epochs
...         train_dataloader.sampler.set_epoch(epoch)
...
...     for X, y in train_dataloader:
...         # No need to move data to GPU, this is done by `prepare_data_loader`!
...         # X, y = X.to("cuda"), y.to("cuda")
...         ...
Parameters:
  • data_loader (torch.utils.data.DataLoader) – The DataLoader to prepare.

  • add_dist_sampler (bool) – Whether to add a DistributedSampler to the provided DataLoader.

  • move_to_device (bool) – If set, automatically move the data returned by the data loader to the correct device.

  • auto_transfer (bool) – If True and the device supports streams, such as on GPU or TPU, copies batches to the device on a separate stream while the current stream runs training. Ignored when move_to_device is False or the device doesn’t support streams.

Returns:

The prepared DataLoader.

Return type:

torch.utils.data.DataLoader