CheckpointConfig#

class ray.data.checkpoint.interfaces.CheckpointConfig(id_column: str | None = None, checkpoint_path: str | None = None, *, generated_id_column: str | None = None, delete_checkpoint_on_success: bool = True, override_filesystem: pyarrow.fs.FileSystem | None = None, override_backend: CheckpointBackend | None = None, write_num_threads: int = 3, checkpoint_path_partition_filter: PathPartitionFilter | None = None, checkpoint_filter_cls: Type[CheckpointFilter] | None = None, checkpoint_manager_cls: Type[CheckpointManager] | None = None)[source]#

Bases: object

Configuration for checkpointing.

Parameters:

  • id_column (str | None) – Name of the ID column in the input dataset. ID values must be unique across all rows in the dataset and must persist during all operators.

  • checkpoint_path (str | None) – Path to store the checkpoint data. It can be a path to a cloud object storage (e.g. s3://bucket/path) or a file system path. If the latter, the path must be a network-mounted file system (e.g. /mnt/cluster_storage/) that is accessible to the entire cluster. If not set, defaults to RAY_DATA_CHECKPOINT_PATH_BUCKET/ray_data_checkpoint.

  • generated_id_column (str | None) – Name of the ID column to generate a row ID for each row. Use this when you don’t have an id_column in the input dataset. Currently, only Parquet files based data sources are supported for auto-generated row IDs feature.

  • delete_checkpoint_on_success (bool) – If true, automatically delete checkpoint data when the dataset execution succeeds. Only supported for batch-based backend currently.

  • override_filesystem (pyarrow.fs.FileSystem | None) – Override the pyarrow.fs.FileSystem object used to read/write checkpoint data. Use this when you want to use custom credentials.

  • override_backend (CheckpointBackend | None) – Override the CheckpointBackend object used to access the checkpoint backend storage.

  • write_num_threads (int) – Number of threads used to write checkpoint files for completed rows.

  • checkpoint_path_partition_filter (PathPartitionFilter | None) – Filter for checkpoint files to load during restoration when reading from checkpoint_path.

  • checkpoint_filter_cls (Type[CheckpointFilter] | None) – Override the CheckpointFilter subclass used to filter out already-checkpointed rows during restoration. The class is instantiated once per checkpoint filter actor with (checkpoint_config, checkpoint_ref), where checkpoint_ref is the ObjectRef returned by the checkpoint manager’s load_checkpoint (by default, a sorted NumPy array of checkpointed IDs). Defaults to NumpyArrayBasedCheckpointFilter.

  • checkpoint_manager_cls (Type[CheckpointManager] | None) – Override the CheckpointManager subclass used to load checkpoint data during restoration. The class is instantiated on the driver with (checkpoint_config=..., data_context=...) and its load_checkpoint must return an (ObjectRef, int) tuple: the ref is passed opaquely to checkpoint_filter_cls, and the int (size in bytes) feeds the per-actor memory reservation. Typically customized together with checkpoint_filter_cls. Defaults to IdColumnCheckpointManager.

PublicAPI (beta): This API is in beta and may change before becoming stable.

property has_generated_id_column: bool#

Whether this config uses auto-generated row IDs.