annotate#

ray.train.annotate(message: str, severity: Literal['info', 'warning', 'error'] = 'info', rank_zero_only: bool = True, **fields: Any) → None#

Emit a custom annotation that Grafana can visualize.

Use this to mark a moment of interest on the same timeline as your training metrics, such as an evaluation completing, a learning-rate change, or a phase transition. Annotations are off by default, so this does nothing unless the cluster operator sets RAY_ANNOTATIONS_ENABLED=1 and ships the annotation log lines to Loki. See Overlay event annotations on the dashboards.

You control what the dashboard tooltip shows. Grafana shows the message as the annotation text, and shows any **fields together as a single JSON tag. All custom annotations share one dashboard layer, because Ray emits them under a fixed internal event name, and severity colors them.

Example

>>> import ray.train
>>> def train_func(config):
...    for epoch in range(config["num_epochs"]):
...        # Do training...
...        ray.train.annotate(
...            message=f"Finished epoch {epoch}",
...            epoch=epoch,
...            loss=loss,
...        )
Parameters:

  • message (str) – Human-readable description of the event. Grafana shows it as the annotation text and tooltip. Ray emits it as the message field.

  • severity (Literal['info', 'warning', 'error']) – One of "info", "warning" or "error". Defaults to "info". It selects which of the Train dashboard’s custom annotation layers the event shows under, and so its color.

  • rank_zero_only (bool) – If True, the default, only the rank 0 worker emits the annotation, producing a single point on the dashboard. If False, every worker emits its own annotation and carries ray_train_worker_world_rank for LogQL to distinguish ranks.

  • **fields (Any) – Arbitrary additional key-value pairs to include in the emitted JSON payload, such as epoch=3 or loss=0.1. Ray serializes them together into a single JSON string under the custom_fields key, so the dashboard annotation displays them as one JSON tag. Because custom_fields parses as a single string label, LogQL can’t filter on individual keys within it. Filter on severity, or on the run and rank tags, instead.

DeveloperAPI: This API may change across minor Ray releases.