FailureConfig

class ray.train.FailureConfig(max_failures: int = 0, fail_fast: bool | str = 'DEPRECATED', controller_failure_limit: int = -1, max_preemption_failures: int = -1)

Bases: FailureConfig

Configuration related to failure handling of each training run.

Parameters:

• max_failures (int) – Tries to recover a run from training worker errors at least this many times. Will recover from the latest checkpoint if present. Setting to -1 will lead to infinite recovery retries. Setting to 0 will disable retries. Defaults to 0.

• controller_failure_limit (int) – [DeveloperAPI] The maximum number of controller failures to tolerate. Setting to -1 will lead to infinite controller retries. Setting to 0 will disable controller retries. Defaults to -1.

• max_preemption_failures (int) – The maximum number of node-preemption interruptions to recover from, counted separately from max_failures (which is reserved for real failures). Will recover from the latest checkpoint if present. Setting to -1 leads to infinite preemption retries; setting to 0 disables them. Defaults to -1.

Methods

Attributes

controller_failure_limit
fail_fast
max_failures
max_preemption_failures