get_dataset_shard#

ray.train.get_dataset_shard(dataset_name: str | None = None, state_dict: Dict[str, Any] | None = None) → DataIterator | None#

Returns the ray.data.DataIterator shard for this worker.

Call iter_torch_batches() or to_tf() on this shard to convert it to the appropriate framework-specific data type.

Examples

>>> import ray.train
>>> from ray.train.torch import TorchTrainer
>>> def train_fn_per_worker(config):
...     ...
...     for epoch in range(2):
...         # Trainer will automatically handle sharding.
...         data_shard = ray.train.get_dataset_shard("train")
...         for batch in data_shard.iter_torch_batches():
...             ...
>>> train_dataset = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
>>> trainer = TorchTrainer(
...     train_fn_per_worker,
...     scaling_config=ray.train.ScalingConfig(num_workers=2),
...     datasets={"train": train_dataset}
... )
>>> print("[ignore]"); trainer.fit()
[ignore]...
Parameters:
  • dataset_name (str | None) – If a Dictionary of Datasets was passed to Trainer, then specifies which dataset shard to return.

  • state_dict (Dict[str, Any] | None) – [Experimental] Optional state dict to restore the dataset iterator. This requires data iterator checkpointing to be enabled via ray.train.DataConfig(dataset_checkpoint_configs=...), and the state dict must be one that was saved with DataIterator.state_dict().

Returns:

The DataIterator shard to use for this worker. If no dataset is passed into Trainer, then return None.

Return type:

DataIterator | None