PreemptionError#

class ray.train.PreemptionError(preemption_info: PreemptionInfo, drain_timed_out: bool = False)#

Bases: TrainingFailedError

Exception raised when training is interrupted by node preemption.

Distinct from WorkerGroupError so that a planned preemption consumes a separate retry budget (FailureConfig.max_preemption_failures, default -1 = unlimited) rather than max_failures, which is reserved for real failures (OOM, hardware faults, user-code bugs).

Parameters:
  • preemption_info – Which nodes / world ranks were preempted and the reclaim deadline, for logging and debugging which preemption caused the restart.

  • drain_timed_out – True when Ray Train stopped waiting because the reclaim deadline passed while workers were still running (and tore them down itself); False when every worker had already exited. Distinguishes a forced teardown from an observed one when debugging.

PublicAPI (alpha): This API is in alpha and may change before becoming stable.