Tracing

To help debug and monitor Ray applications, Ray integrates with OpenTelemetry to facilitate exporting traces to external tracing stacks such as Jaeger.

Note

Tracing is an experimental feature and under active development. APIs are subject to change.

Installation

First, install OpenTelemetry:

pip install opentelemetry-api==1.1.0
pip install opentelemetry-sdk==1.1.0
pip install opentelemetry-exporter-otlp==1.1.0

Tracing startup hook

To enable tracing, you must provide a tracing startup hook with a function that sets up the Tracer Provider, Remote Span Processors, and Additional Instruments. The tracing startup hook is expected to be a function that is called with no args or kwargs. This hook needs to be available in the Python environment of all the worker processes.

Below is an example tracing startup hook that sets up the default tracing provider, exports spans to files in /tmp/spans, and does not have any additional instruments.

import ray
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    ConsoleSpanExporter,
    SimpleSpanProcessor,
)


def setup_tracing() -> None:
    # Creates /tmp/spans folder
    os.makedirs("/tmp/spans", exist_ok=True)
    # Sets the tracer_provider. This is only allowed once per execution
    # context and will log a warning if attempted multiple times.
    trace.set_tracer_provider(TracerProvider())
    trace.get_tracer_provider().add_span_processor(
        SimpleSpanProcessor(
            ConsoleSpanExporter(
                out=open(f"/tmp/spans/{os.getpid()}.json", "a")
                )
        )
    )

For open-source users who want to experiment with tracing, Ray has a default tracing startup hook that exports spans to the folder /tmp/spans. To run using this default hook, run the following code sample to set up tracing and trace a simple Ray Task.

ray start

$ ray start --head --tracing-startup-hook=ray.util.tracing.setup_local_tmp_tracing:setup_tracing
$ python
    ray.init()
    @ray.remote
    def my_function():
        return 1

    obj_ref = my_function.remote()

ray.init()

ray.init(_tracing_startup_hook="ray.util.tracing.setup_local_tmp_tracing:setup_tracing")

@ray.remote
def my_function():
    return 1

obj_ref = my_function.remote()

If you want to provide your own custom tracing startup hook, provide it in the format of module:attribute where the attribute is the setup_tracing function to be run.

Tracer provider

This configures how to collect traces. View the TracerProvider API here.

Remote span processors

This configures where to export traces to. View the SpanProcessor API here.

Users who want to experiment with tracing can configure their remote span processors to export spans to a local JSON file. Serious users developing locally can push their traces to Jaeger containers via the Jaeger exporter.

Additional instruments

If you are using a library that has built-in tracing support, the setup_tracing function you provide should also patch those libraries. You can find more documentation for the instrumentation of these libraries here.

Custom traces

Add custom tracing in your programs. Within your program, get the tracer object with trace.get_tracer(__name__) and start a new span with tracer.start_as_current_span(...).

See below for a simple example of adding custom tracing.

from opentelemetry import trace

@ray.remote
def my_func():
    tracer = trace.get_tracer(__name__)

    with tracer.start_as_current_span("foo"):
        print("Hello world from OpenTelemetry Python!")