Install RLlib for development#

Develop RLlib locally without compiling Ray by using the setup-dev.py script. The script sets up symlinks between the ray/rllib directory in your local git clone and the matching directory bundled with the pip-installed ray package. Every change you make in your clone’s source files then appears immediately in your installed ray.

If you installed Ray from source using these instructions, don’t use the script. Those steps should already have created the necessary symlinks.

When you use the setup-dev.py script, keep your git branch in sync with the installed Ray binaries. Stay up to date on master and install the latest wheel.

# Clone your fork onto your local machine, e.g.:
git clone https://github.com/[your username]/ray.git
cd ray
# Only enter 'Y' at the first question on linking RLlib.
# This leads to the most stable behavior and you won't have to re-install ray as often.
# If you anticipate making changes to e.g. Tune or Train quite often, consider also symlinking Ray Tune or Train here
# (say 'Y' when asked by the script about creating the Tune or Train symlinks).
python python/ray/setup-dev.py

Contributing to RLlib#

Contributing fixes and enhancements#

File new RLlib-related PRs through Ray’s GitHub repo. The RLlib team welcomes external help from the open-source community. If you’re unsure how to structure a bug-fix or enhancement PR, create a small PR first, then ask questions in its conversation section. For an example of a good first community PR, see this pull request.

Contributing algorithms#

These guidelines cover merging new algorithms into RLlib. RLlib accepts contributions at two levels. The first is an example script, possibly with additional classes in other files. The second is a fully integrated RLlib algorithm in rllib/algorithms.

  • An example algorithm has three requirements:

    • It must subclass Algorithm and implement the training_step() method.

    • It must include the main example script, which demonstrates the algorithm, in a CI test that proves the algorithm learns a task.

    • It should provide capabilities that existing algorithms don’t have.

  • A fully integrated algorithm has four additional requirements:

    • It must provide substantial new capabilities that you can’t add to existing algorithms.

    • It should support custom RLModules.

    • It should use RLlib abstractions and support distributed execution.

    • It should include at least one tuned hyperparameter example. The CI tests this example.

Both integrated and contributed algorithms ship with the ray PyPI package, and Ray’s automated tests cover them.

New features#

The GitHub issues page tracks new feature development, discussions, and priorities. It might not include every development effort.

API stability#

API decorators in the codebase#

Objects and methods annotated with @PublicAPI or @DeveloperAPI on the new API stack, or @OldAPIStack on the old API stack, have the following API compatibility guarantees:

ray.util.annotations.PublicAPI(obj: F) → F[source]
ray.util.annotations.PublicAPI(*, stability: str = 'stable', api_group: str = 'Others') → Callable[[F], F]

Annotation for documenting public APIs.

Public APIs are classes and methods exposed to end users of Ray.

If stability="alpha", the API can be used by advanced users who are tolerant to and expect breaking changes.

If stability="beta", the API is still public and can be used by early users, but are subject to change.

If stability="stable", the APIs will remain backwards compatible across minor Ray releases (e.g., Ray 1.4 -> 1.8).

For a full definition of the stability levels, please refer to the Ray API Stability definitions.

Parameters:

  • *args – When used as a bare @PublicAPI decorator, contains the wrapped function or class as the single positional argument.

  • **kwargs – Supported keyword arguments are stability (one of "stable", "beta", "alpha") and api_group (used only for doc rendering; APIs in the same group are grouped together in the API doc pages).

Returns:

Either the annotated object (when used as @PublicAPI) or a decorator that annotates an object (when used as @PublicAPI(...)).

Examples

>>> from ray.util.annotations import PublicAPI
>>> @PublicAPI
... def func(x):
...     return x

>>> @PublicAPI(stability="beta")
... def func(y):
...     return y
ray.util.annotations.DeveloperAPI(obj: F) → F[source]
ray.util.annotations.DeveloperAPI() → Callable[[F], F]

Annotation for documenting developer APIs.

Developer APIs are lower-level methods explicitly exposed to advanced Ray users and library developers. Their interfaces may change across minor Ray releases.

Parameters:

  • *args – When used as a bare @DeveloperAPI decorator, contains the wrapped function or class as the single positional argument.

  • **kwargs – Reserved for future use; no keyword arguments are currently supported.

Returns:

Either the annotated object (when used as @DeveloperAPI) or a decorator that annotates an object (when used as @DeveloperAPI()).

Examples

>>> from ray.util.annotations import DeveloperAPI
>>> @DeveloperAPI
... def func(x):
...     return x
ray.rllib.utils.annotations.OldAPIStack(obj: F) → F[source]

Decorator for classes/methods/functions belonging to the old API stack.

These should be deprecated at some point after Ray 3.0 (RLlib GA). It is recommended for users to start exploring (and coding against) the new API stack instead.

Benchmarks#

The rl-experiments repo holds many training-run results, and examples/algorithms lists working hyperparameter configurations sorted by algorithm. Benchmark results help the community. If you have results that might interest others, open a pull request to either repo.

Debugging RLlib#

Finding memory leaks in workers#

Keeping the memory usage of long-running workers stable can be challenging. Use the MemoryTrackingCallbacks class to track worker memory usage.

class ray.rllib.callbacks.callbacks.MemoryTrackingCallbacks[source]#

MemoryTrackingCallbacks can be used to trace and track memory usage in rollout workers.

The Memory Tracking Callbacks uses tracemalloc and psutil to track python allocations during rollouts, in training or evaluation.

The tracking data is logged to the custom_metrics of an episode and can therefore be viewed in tensorboard (or in WandB etc..)

Add MemoryTrackingCallbacks callback to the tune config e.g. { …’callbacks’: MemoryTrackingCallbacks …}

Note

This class is meant for debugging and should not be used in production code as tracemalloc incurs a significant slowdown in execution speed.

The callback adds the 20 objects with the highest memory usage in the workers as custom metrics. Monitor these with TensorBoard or other metrics integrations such as Weights & Biases:

../_images/MemoryTrackingCallbacks.png

Troubleshooting#

If you encounter errors such as blas_thread_init: pthread_create: Resource temporarily unavailable when using many workers, set OMP_NUM_THREADS=1. For other resource-limit errors, check the configured system limits with ulimit -a.

To debug unexpected hangs or performance problems, run ray stack to dump the stack traces of all Ray workers on the current node, ray timeline to dump a timeline visualization of tasks to a file, and ray memory to list all object references in the cluster.