DatasetCheckpointConfig#

class ray.train.DatasetCheckpointConfig(id_column: str, generate_id_column: bool = False, checkpoint_path: str | None = None, override_filesystem: FileSystem | None = None, delete_checkpoints_after_epoch: bool = True)#

Bases: object

Configuration for training ingest checkpointing.

Parameters:

  • id_column (str) – Name of the ID column in the input dataset. ID values must be unique across all rows in the dataset and must persist during all operators.

  • generate_id_column (bool) – Whether to generate the id_column for each row. Use this when you don’t have a pre-existing id_column in the input dataset. The generated column is removed from the yielded batches. Only Parquet reads on the V2 datasource path are supported.

  • checkpoint_path (str | None) – Path to store the checkpoint data. It can be a path to a cloud object storage (e.g. s3://bucket/path) or a file system path. If the latter, the path must be a network-mounted file system (e.g. /mnt/cluster_storage/) that is accessible to the entire cluster. If not set, defaults to {RunConfig.storage_path}/{RunConfig.name}/ray_data_checkpoints/{dataset_name} configured on the ray.train trainer. Each dataset must use a different checkpoint_path.

  • override_filesystem (FileSystem | None) – Override the pyarrow.fs.FileSystem object used to read/write checkpoint data. Use this when you want to use custom credentials. If unset, this defaults to the filesystem configured in the ray.train.RunConfig when checkpoint_path is also unset. Otherwise, the filesystem is inferred from checkpoint_path.

  • delete_checkpoints_after_epoch (bool) – If True, automatically delete checkpoint data after each epoch completion. This allows for fault tolerance from the latest checkpoint. If you intend to resume from a checkpoint prior to the latest epoch, set this to False. Defaults to True.

PublicAPI (alpha): This API is in alpha and may change before becoming stable.