Development Tips

Note

Unless otherwise stated, directory and file paths are relative to the project root directory.

Compilation

To speed up compilation, be sure to install Ray with the following commands:

cd python
pip install -e . --verbose

The -e means “editable”, so changes you make to files in the Ray directory will take effect without reinstalling the package. In contrast, if you do python setup.py install, files will be copied from the Ray directory to a directory of Python packages (often something like $HOME/anaconda3/lib/python3.6/site-packages/ray). This means that changes you make to files in the Ray directory will not have any effect.

If you run into Permission Denied errors when running pip install, you can try adding --user. You may also need to run something like sudo chown -R $USER $HOME/anaconda3 (substituting in the appropriate path).

If you make changes to the C++ or Python files, you will need to run the build so C++ code is recompiled and/or Python files are redeployed in the python directory. However, you do not need to rerun pip install -e .. Instead, you can recompile much more quickly by running the following:

bash build.sh

This command is not enough to recompile all C++ unit tests. To do so, see Testing locally.

Developing Ray (Python Only)

Note

Unless otherwise stated, directory and file paths are relative to the project root directory.

RLlib, Tune, Autoscaler, and most Python files do not require you to build and compile Ray. Follow these instructions to develop Ray’s Python files locally.

  1. Pip install the latest Ray wheels. See Latest Snapshots (Nightlies) for instructions.

  2. Fork and clone the project to your machine. Connect your repository to the upstream (main project) ray repository.

git clone https://github.com/[your username]/ray.git
cd ray
git remote add upstream https://github.com/ray-project/ray.git
# Make sure you are up-to-date on master.
  1. Run python python/ray/setup-dev.py. This sets up links between the tune dir (among other directories) in your local repo and the one bundled with the ray package.

Warning

Do not run pip uninstall ray or pip install -U (for Ray or Ray wheels) if setting up your environment this way. To uninstall or upgrade, you must rm -rf the installation site (usually a site-packages/ray location).

Using a local repository for dependencies

If you’d like to build Ray with custom dependencies (for example, with a different version of Cython), you can modify your .bzl file as follows:

http_archive(
  name = "cython",
  ...,
) if False else native.new_local_repository(
  name = "cython",
  build_file = "bazel/BUILD.cython",
  path = "../cython",
)

This replaces the existing http_archive rule with one that references a sibling of your Ray directory (named cython) using the build file provided in the Ray repository (bazel/BUILD.cython). If the dependency already has a Bazel build file in it, you can use native.local_repository instead, and omit build_file.

To test switching back to the original rule, change False to True.

Debugging

Starting processes in a debugger

When processes are crashing, it is often useful to start them in a debugger. Ray currently allows processes to be started in the following:

  • valgrind

  • the valgrind profiler

  • the perftools profiler

  • gdb

  • tmux

To use any of these tools, please make sure that you have them installed on your machine first (gdb and valgrind on MacOS are known to have issues). Then, you can launch a subset of ray processes by adding the environment variable RAY_{PROCESS_NAME}_{DEBUGGER}=1. For instance, if you wanted to start the raylet in valgrind, then you simply need to set the environment variable RAY_RAYLET_VALGRIND=1.

To start a process inside of gdb, the process must also be started inside of tmux. So if you want to start the raylet in gdb, you would start your Python script with the following:

RAY_RAYLET_GDB=1 RAY_RAYLET_TMUX=1 python

You can then list the tmux sessions with tmux ls and attach to the appropriate one.

You can also get a core dump of the raylet process, which is especially useful when filing issues. The process to obtain a core dump is OS-specific, but usually involves running ulimit -c unlimited before starting Ray to allow core dump files to be written.

Inspecting Redis shards

To inspect Redis, you can use the global state API. The easiest way to do this is to start or connect to a Ray cluster with ray.init(), then query the API like so:

ray.init()
ray.nodes()
# Returns current information about the nodes in the cluster, such as:
# [{'ClientID': '2a9d2b34ad24a37ed54e4fcd32bf19f915742f5b',
#   'IsInsertion': True,
#   'NodeManagerAddress': '1.2.3.4',
#   'NodeManagerPort': 43280,
#   'ObjectManagerPort': 38062,
#   'ObjectStoreSocketName': '/tmp/ray/session_2019-01-21_16-28-05_4216/sockets/plasma_store',
#   'RayletSocketName': '/tmp/ray/session_2019-01-21_16-28-05_4216/sockets/raylet',
#   'Resources': {'CPU': 8.0, 'GPU': 1.0}}]

To inspect the primary Redis shard manually, you can also query with commands like the following.

r_primary = ray.worker.global_worker.redis_client
r_primary.keys("*")

To inspect other Redis shards, you will need to create a new Redis client. For example (assuming the relevant IP address is 127.0.0.1 and the relevant port is 1234), you can do this as follows.

import redis
r = redis.StrictRedis(host='127.0.0.1', port=1234)

You can find a list of the relevant IP addresses and ports by running

r_primary.lrange('RedisShards', 0, -1)

Backend logging

The raylet process logs detailed information about events like task execution and object transfers between nodes. To set the logging level at runtime, you can set the RAY_BACKEND_LOG_LEVEL environment variable before starting Ray. For example, you can do:

export RAY_BACKEND_LOG_LEVEL=debug
ray start

This will print any RAY_LOG(DEBUG) lines in the source code to the raylet.err file, which you can find in the Temporary Files.

Testing locally

Testing for Python development

Suppose that one of the tests in a file of tests, e.g., python/ray/tests/test_basic.py, is failing. You can run just that test file locally as follows:

python -m pytest -v python/ray/tests/test_basic.py

However, this will run all of the tests in the file, which can take some time. To run a specific test that is failing, you can do the following instead:

python -m pytest -v python/ray/tests/test_basic.py::test_keyword_args

When running tests, usually only the first test failure matters. A single test failure often triggers the failure of subsequent tests in the same file.

Testing for C++ development

To compile and run all C++ tests, you can run:

bazel test $(bazel query 'kind(cc_test, ...)')

Alternatively, you can also run one specific C++ test. You can use:

bazel test $(bazel query 'kind(cc_test, ...)') --test_filter=ClientConnectionTest --test_output=streamed

Building the Docs

If you make changes that require documentation changes, don’t forget to update the documentation!

When you make documentation changes, build them locally to verify they render correctly. Sphinx is used to generate the documentation.

cd doc
pip install -r requirements-doc.txt
make html

Once done, the docs will be in doc/_build/html. For example, on Mac OSX, you can open the docs (assuming you are still in the doc directory) using open _build/html/index.html.

Creating a pull request

To create a pull request (PR) for your change, first go through the PR template checklist and ensure you’ve completed all the steps.

When you push changes to GitHub, the formatting and verification script ci/travis/format.sh is run first. For pushing to your fork, you can skip this step with git push --no-verify.

Before submitting the PR, you should run this script. If it fails, the push operation will not proceed. This script requires specific versions of the following tools. Installation commands are shown for convenience:

  • yapf version 0.23.0 (pip install yapf==0.23.0)

  • flake8 version 3.7.7 (pip install flake8==3.7.7)

  • flake8-quotes (pip install flake8-quotes)

  • clang-format version 7.0.0 (download this version of Clang from here)

Note: On MacOS X, don’t use HomeBrew to install clang-format, as the only version available is too new.

The Ray project automatically runs continuous integration (CI) tests once a PR is opened using Travis-CI with multiple CI test jobs.

Understand CI test jobs

The Travis CI test folder contains all integration test scripts and they invoke other test scripts via pytest, bazel-based test or other bash scripts. Some of the examples include:

  • Raylet integration tests commands:
    • bazel test //:core_worker_test

    • src/ray/test/run_object_manager_tests.sh

  • Bazel test command:
    • bazel test --build_tests_only //:all

  • Ray serving test commands:
    • python -m pytest python/ray/serve/tests

    • python python/ray/serve/examples/echo_full.py

If a Travis-CI build exception doesn’t appear to be related to your change, please visit this link to check recent tests known to be flaky.

Format and Linting

Installation instructions for the tools mentioned here are discussed above in Creating a pull request.

Running the linter locally: To run the Python linter on a specific file, run flake8 as in this example, flake8 python/ray/worker.py.

Autoformatting code. We use yapf for linting. The config file is .style.yapf. We recommend running scripts/yapf.sh prior to pushing a PR to format any changed files. Note that some projects, such as dataframes and rllib, are currently excluded.

Running CI linter: The Travis CI linter script has multiple components to run. We recommend running ci/travis/format.sh, which runs both linters for Python and C++ codes. In addition, there are other formatting checkers for components like the following:

  • Python REAME format:

cd python
python setup.py check --restructuredtext --strict --metadata
  • Bazel format:

./ci/travis/bazel-format.sh