get_dataset_shard#
- ray.train.get_dataset_shard(dataset_name: str | None = None, state_dict: Dict[str, Any] | None = None) DataIterator | None#
Returns the
ray.data.DataIteratorshard for this worker.Call
iter_torch_batches()orto_tf()on this shard to convert it to the appropriate framework-specific data type.Examples
>>> import ray.train >>> from ray.train.torch import TorchTrainer >>> def train_fn_per_worker(config): ... ... ... for epoch in range(2): ... # Trainer will automatically handle sharding. ... data_shard = ray.train.get_dataset_shard("train") ... for batch in data_shard.iter_torch_batches(): ... ... >>> train_dataset = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv") >>> trainer = TorchTrainer( ... train_fn_per_worker, ... scaling_config=ray.train.ScalingConfig(num_workers=2), ... datasets={"train": train_dataset} ... ) >>> print("[ignore]"); trainer.fit() [ignore]...
- Parameters:
dataset_name (str | None) – If a Dictionary of Datasets was passed to
Trainer, then specifies which dataset shard to return.state_dict (Dict[str, Any] | None) – [Experimental] Optional state dict to restore the dataset iterator. This requires data iterator checkpointing to be enabled via
ray.train.DataConfig(dataset_checkpoint_configs=...), and the state dict must be one that was saved withDataIterator.state_dict().
- Returns:
The
DataIteratorshard to use for this worker. If no dataset is passed into Trainer, then return None.- Return type:
DataIterator | None