Recover lost blocks due to transient failures#
Node failures and preemptions can be common during long training and data processing runs. When a node fails, the blocks whose primary copies were stored in that node’s share of the object store are lost. To Ray Core, Ray Data blocks are objects. By default, Ray Core recovers lost objects by detecting a node loss, finding all objects that were resident on that node, and then re-running the tasks that created them. To learn how that works, see Object fault tolerance. The shortfall of this approach is that block reconstruction happens outside Ray Data’s resource management and scheduling. Without proper resource accounting or backpressure, reconstruction tasks can oversubscribe the cluster, which can lead to excessive spilling or even out-of-memory (OOM) errors.
As of Ray 2.61, Ray Data can reconstruct lost blocks by itself instead of relying on Ray Core. Ray Data accounts for the reconstruction work, and it resubmits only the chain of producers of a lost block, which is the block’s lineage. When a task’s output is lost, Ray Data traces the lost block back to the tasks that read the source data, such as a read task or a from_items task. Then, Ray Data re-runs the chain of tasks that produce the lost block, subject to the existing backpressure and resource accounting of Ray Data.
Ray Data lineage reconstruction is experimental and off by default. Its behavior and configuration may change in future releases.
Note
In early testing, Ray Data lineage reconstruction significantly reduced object spilling after node failures compared to Ray Core lineage reconstruction. This is because reconstruction tasks go through Ray Data’s backpressure and resource accounting. The tradeoff is that end-to-end runtime can increase. Reconstruction tasks wait for resources instead of running right away. Ray Data lineage reconstruction also re-runs the chain of tasks from the first stage in the pipeline. The scheduler batches reconstructions together based on block lineage and prunes out wasted work. Performance benchmarks for lineage reconstruction are in progress. Until they’re published, test your workload with node failures before you turn this on in production.
For the design, see the Ray Enhancement Proposal (REP) for Ray Data lineage reconstruction.
Turn on lineage reconstruction#
Ray Data reconstructs lost blocks when you turn off Ray Core lineage reconstruction for the job. To do that, pass a JobConfig with _disable_job_level_lineage_reconstruction=True to ray.init(). Ray Data reads this setting from the job and turns on its own reconstruction automatically.
import ray
from ray.job_config import JobConfig
ray.init(job_config=JobConfig(_disable_job_level_lineage_reconstruction=True))
ds = ray.data.range(1000).map_batches(lambda batch: {"id": batch["id"] * 2})
for batch in ds.iter_batches():
pass
To check that lineage reconstruction is on, read enable_ray_data_reconstruction on the current DataContext:
print(ray.data.DataContext.get_current().enable_ray_data_reconstruction)
True
Keep the following in mind when you turn on lineage reconstruction:
Turn on lineage reconstruction only through the job config.
DataContext.enable_ray_data_reconstructionreflects the job setting. Don’t set it on theDataContext, because that setting doesn’t propagate to the cluster. If a value on theDataContextconflicts with the job config, Ray Data raises aValueError.Turning on Ray Data lineage reconstruction turns off Ray Core lineage reconstruction automatically. This setting stops the job’s workers from keeping the lineage of their objects, so Ray Core can’t reconstruct any object in the job. Ray Data is the only reconstruction mechanism. If a dataset can’t use Ray Data lineage reconstruction, a lost object fails that dataset. To learn which datasets are supported, see Supported datasets.
Aggregations in your script turn off lineage reconstruction for that dataset. Methods such as
Dataset.count()andDataset.sum()add operators that lineage reconstruction doesn’t support. To keep reconstruction on, consume the dataset withiter_batches(),take_all(), or a write.
Supported datasets#
Ray Data lineage reconstruction supports the following:
Datasets made only of reads and map transformations, such as
map(),map_batches(),filter(), andflat_map(). This includes transformations that run on tasks and on actor pools.Multiple lost blocks in the same dataset.
Consuming the dataset with
iter_batches()ortake_all().
Ray Data turns off lineage reconstruction for the whole dataset if the dataset uses any of the following:
Reconstruction failures#
If Ray Data detects a lost block but can’t start reconstructing it, Ray Data raises a ray.data.exceptions.LineageReconstructionError. For example, reconstruction can’t start when the task that produced the block isn’t tracked in the lineage graph, or when Ray Data no longer holds the input of the read task that the block traces back to. The error message includes both why the block was lost and why reconstruction failed.
LineageReconstructionError is a subclass of ObjectLostError, so code that already catches ObjectLostError also catches it.
Ray Data counts a LineageReconstructionError as a failed block. By default, a failed block fails the dataset. To drop the lost block and continue instead, set max_errored_blocks on the DataContext. To learn more, see Execution configurations.
Troubleshooting#
Ray Data writes lineage reconstruction logs to ray-data.log. To learn where to find this file, see Ray Data logs.
Follow a reconstruction#
Ray Data logs each reconstruction step at the INFO level with the prefix [lineage-reconstruction]. Search for this prefix to see when Ray Data detects a lost block, which read tasks it re-runs, and on which operators.
Inspect the task graph#
Ray Data logs the task graph that it uses for reconstruction at the DEBUG level. For every task, the log records the task ID and the tasks whose output blocks it depends on. When a block is lost, the log also records the reconstruction plan, which is the chain of tasks from the lost block back to its read tasks. Use these lines to check which tasks Ray Data re-runs to rebuild a lost block.
See also#
Job-level checkpointing: Resume a failed job by skipping rows that a previous run processed.
Object fault tolerance: How Ray Core reconstructs lost objects.