PreemptionError#
- class ray.train.PreemptionError(preemption_info: PreemptionInfo, drain_timed_out: bool = False)#
Bases:
TrainingFailedErrorException raised when training is interrupted by node preemption.
Distinct from
WorkerGroupErrorso that a planned preemption consumes a separate retry budget (FailureConfig.max_preemption_failures, default -1 = unlimited) rather thanmax_failures, which is reserved for real failures (OOM, hardware faults, user-code bugs).- Parameters:
preemption_info – Which nodes / world ranks were preempted and the reclaim deadline, for logging and debugging which preemption caused the restart.
drain_timed_out – True when Ray Train stopped waiting because the reclaim deadline passed while workers were still running (and tore them down itself); False when every worker had already exited. Distinguishes a forced teardown from an observed one when debugging.
PublicAPI (alpha): This API is in alpha and may change before becoming stable.