prepare_data_loader#
- ray.train.torch.prepare_data_loader(data_loader: torch.utils.data.DataLoader, add_dist_sampler: bool = True, move_to_device: bool = True, auto_transfer: bool = True) torch.utils.data.DataLoader[source]#
Prepares
DataLoaderfor distributed execution.This allows you to use the same exact code regardless of number of workers or the device type being used (CPU, GPU, TPU).
Note
This method adds a
DistributedSamplerto theDataLoaderif the number of training workers is greater than 1. If shuffling is enabled on the originalDataLoader, thenshuffle=Truewill also be passed into theDistributedSamplerconstructor.shuffle=Falseon the originalDataLoaderalso means that shuffling is disabled on the sampler.With more than 1 worker, calling the
DistributedSampler.set_epochmethod at the beginning of each epoch before creating the DataLoader iterator is necessary to make shuffling work properly across multiple epochs. Otherwise, the same ordering will be always used. See: https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler # noqa: E501Examples
>>> import torch >>> import ray.train.torch >>> train_dataloader = torch.utils.data.DataLoader( ... ..., batch_size=..., shuffle=True ... ) >>> train_dataloader = ray.train.torch.prepare_data_loader(train_dataloader) >>> for epoch in range(10): ... if ray.train.get_context().get_world_size() > 1: ... # Required for the distributed sampler to shuffle properly across epochs ... train_dataloader.sampler.set_epoch(epoch) ... ... for X, y in train_dataloader: ... # No need to move data to GPU, this is done by `prepare_data_loader`! ... # X, y = X.to("cuda"), y.to("cuda") ... ...
- Parameters:
data_loader (torch.utils.data.DataLoader) – The DataLoader to prepare.
add_dist_sampler (bool) – Whether to add a DistributedSampler to the provided DataLoader.
move_to_device (bool) – If set, automatically move the data returned by the data loader to the correct device.
auto_transfer (bool) – If
Trueand the device supports streams, such as on GPU or TPU, copies batches to the device on a separate stream while the current stream runs training. Ignored whenmove_to_deviceisFalseor the device doesn’t support streams.
- Returns:
The prepared DataLoader.
- Return type: