Editing and managing Python dependencies#

This guide explains how Ray’s Python dependencies are structured, how to add or update one, and how to diagnose the resolution conflicts you’ll most likely hit along the way. Read it before you change anything under python/requirements/ or any requirements_compiled*.txt file.

If you only want to install dependencies to develop or test Ray, see Building Ray from source instead. This page is about changing the dependency set.

How Ray’s dependencies are structured#

Ray resolves dependencies in three layers. Each layer is generated from the one above it and acts as a constraint on the one below it.

Layer	Files	What it is	Generated by
1. Source	`python/requirements/**.txt`	Hand-written direct deps	Most commonly edited
2. Compiled	`python/requirements_compiled*.txt`	One solved version per package across the monorepo	`ci/ci.sh compile_pip_dependencies`
3. Locks	`python/deplocks/*/.lock`	Per-image, fully pinned, SHA256-hashed	`bazelisk run //ci/raydepsets:raydepsets -- build`

The locks are consumed directly by Ray’s Docker and Wanda image builds.

Layer 1 — source requirements#

python/requirements/ holds the hand-written files. In most cases, these are the files you edit to change dependencies across every Python version and image. They list direct dependencies only and express intent (a loose floor, an exact pin, or an environment marker). They’re organized by feature area: ml/tune-requirements.txt, ml/train-requirements.txt, ml/rllib-requirements.txt, ml/data-requirements.txt, serve/, plus top-level files like test-requirements.txt and cloud-requirements.txt.

Layer 2 — compiled requirements#

ci/ci.sh compile_pip_dependencies feeds all the Layer 1 files to pip-compile at once and solves for a single consistent version of every package, direct and transitive. The output is the compiled lockfile, which CI passes as a -c constraint everywhere Ray is installed. Each track is for a different Python version supported by Ray.

File	Covers
`python/requirements_compiled.txt`	The real lockfile for Python 3.10–3.13
`python/requirements_compiled_py3.10.txt`	Symlink → `requirements_compiled.txt`
`python/requirements_compiled_py3.11.txt`	Symlink → `requirements_compiled.txt`
`python/requirements_compiled_py3.12.txt`	Symlink → `requirements_compiled.txt`
`python/requirements_compiled_py3.13.txt`	Symlink → `requirements_compiled.txt`

Don’t hand-edit these files because they’re generated artifacts. Use the ci/ci.sh commands described below to regenerate them.

compile_pip_dependencies runs in a Python 3.11 environment (CI uses the oss-ci-base_test-py3.11 job); that works because pip-compile resolves based on what the requirement files declare, not on the Python it’s currently running.

Layer 3 — lock files#

python/deplocks/**/*.lock are per-image, fully pinned, SHA256-hashed files. Each Docker image, or CI test environment, consumes exactly one lock. The layout under python/deplocks/ mirrors the dependency-set structure: ray_img/ for the main Ray Docker images, docs/ for the docs build, ci/ for CI test environments, llm/ for RayLLM, plus base-image subdirectories. The authoritative mapping of which depset produces which lock lives in ci/raydepsets/configs/*.depsets.yaml.

They’re generated by raydepsets (ci/raydepsets/), which models the relationships between locks as a graph and regenerates every dependent in topological order when one input changes.

raydepsets exists because the locks aren’t independent. The locks have invariants that a bare uv pip compile can’t enforce:

a test environment’s lock must be a strict superset of the base image it runs on, and
all CUDA variants for a given Python version must agree on shared package versions.

Each lock is declared in a ci/raydepsets/configs/*.depsets.yaml file via one of four operations:

Operation	What it does
`compile`	Resolve and lock requirements with `uv pip compile`
`subset`	Extract a subset of an existing lock (validated against the source)
`expand`	Merge several locks (plus optional extra requirements) into one
`relax`	Remove named packages from a lock

relax is a blunt instrument: it deletes lines without re-resolving the graph, so removing a package that something else needs leaves an inconsistent lock.

Every compile call inherits this default set of uv pip compile flags:

--no-header
--generate-hashes
--index-strategy unsafe-best-match
--no-strip-markers
--emit-index-url
--emit-find-links
--unsafe-package setuptools     (unless include_setuptools: true)

--generate-hashes is why locks are SHA256-pinned. --no-strip-markers preserves python_version markers so a single lock can be a constraint at multiple --python-version targets. --index-strategy unsafe-best-match is the same flag you’d pass when reproducing CI’s install resolution locally (see Diagnosing dependency conflicts).

Before each compile runs, raydepsets executes any declared pre-hooks. The common one is ci/raydepsets/pre_hooks/remove-compiled-headers.sh, which copies requirements_compiled*.txt to /tmp/ray-deps/ after stripping --extra-index-url / --find-links lines, so the file is a clean version constraint and GPU index URLs don’t leak into CPU locks.

The locks are consumed verbatim. Docker image builds run uv pip install -r <lock> --no-deps against the matching lock — for example, the docs image installs python/deplocks/docs/docbuild_depset_py3.10.lock. Because --no-deps skips dependency resolution at install time, the pinned versions can’t drift; every image built from the same lock gets the exact same package versions.

The Ray docs build is itself one of these locks: doc/requirements-doc.lock.txt is a symlink to python/deplocks/docs/docbuild_depset_py3.10.lock, so adding a Sphinx extension is a dependency change like any other. For the tool’s full internals, see the raydepsets README.

How to add or update a dependency#

If you have an x86_64 Linux machine with Python 3.11, compile locally (see Compiling locally below). Otherwise, let CI do it and download the regenerated file from the Buildkite Artifacts tab. Either way produces the same lock; pick whichever has the faster feedback loop for you.

1. Edit the Layer 1 file#

Change the relevant python/requirements/**.txt file.

Don’t hand-edit python/requirements_compiled*.txt or any file under python/deplocks/ as they’re build artifacts. Express the change as a source-file pin. The regenerated lock will pick it up.

1a. Keep `requirements.txt` and `setup.py` in sync#

End users install Ray with pip install ray, which reads python/setup.py, not python/requirements.txt. setup.py lists Ray’s runtime dependencies inline in setup_spec.install_requires, and that list has to agree with the matching entries in python/requirements.txt:

The ## setup.py install_requires block in python/requirements.txt mirrors setup_spec.install_requires in python/setup.py. If you add, remove, or re-bound any package in that block, edit both files in the same change.
python/requirements/llm/llm-requirements.txt mirrors setup_spec.extras["llm"] in setup.py. Both files carry an explicit Keep this in sync comment. If you add, remove, or change a version bound for any package listed in both, edit both files in the same change.

Drift between these files ships a broken pip install ray to end users while CI stays green, because CI installs from the compiled lock, not from setup.py. Always check both files when you touch one.

2. Let CI recompile, then commit the result#

The dependencies Buildkite pipeline runs compile_pip_dependencies, then diffs the result against the committed file and fails if they differ, printing:

requirements_compiled.txt is not up to date. Please download it from Artifacts tab and git push the changes.

Follow that message: push your Layer 1 change, let the build: pip-compile dependencies job run, download the regenerated requirements_compiled*.txt from the Buildkite Artifacts tab, commit it, and push again.

By default the compile reuses existing pins as much as possible, so a small change stays small. To force every package to its newest compatible version, a Ray maintainer uncomments the rm ./python/requirements_compiled.txt line in .buildkite/dependencies.rayci.yml for a refresh run. Community contributors typically don’t have the Buildkite permissions to trigger this themselves; ask in your PR if you think a full refresh is needed.

3. Regenerate the Layer 3 locks#

If your change reaches an image’s dependency set, regenerate the locks with raydepsets:

# Regenerate every lock, in dependency order
bazelisk run //ci/raydepsets:raydepsets -- build --all-configs

# Or just one config
bazelisk run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/rayimg.depsets.yaml

# Validate committed locks are current (this is what CI runs)
bazelisk run //ci/raydepsets:raydepsets -- build --all-configs --check

Compiling locally#

You can compile locally instead of waiting on CI, but the compile scripts refuse to run on ARM. ci/ci.sh skips aarch64 and arm64 outright:

Skipping for aarch64

So an Apple Silicon Mac or any ARM box can’t produce a valid lockfile. The GPU and platform-pinned wheels (CUDA torch, jaxlib, the PyTorch Geometric family) don’t resolve there. Compile on an x86_64 / AMD64 Linux machine, in a Python 3.11 environment (the CI compile jobs run under py3.11):

# x86_64 Linux, Python 3.11
ci/ci.sh compile_pip_dependencies            # -> requirements_compiled.txt
bazelisk run //ci/raydepsets:raydepsets -- build --all-configs

Diagnosing dependency conflicts#

When a compile or install fails, use uv to find out why, even though the compile script itself uses pip-compile. uv reports the real incompatibility; pip often hides it behind a generic error.

`resolution-too-deep` almost never means “too deep”#

pip’s backtracking resolver has a hardcoded round limit. When two requirements can never be satisfied together, pip backtracks until it hits that limit and reports resolution-too-deep. This is usually the symptom of some other problem, not the cause.

Re-run the same resolution with uv, which has no such limit and prints the actual conflict. Below is an example of how uv can uncover dependency problems.

# Resolve only, install nothing — surfaces the true error
uv pip install --dry-run "ax-platform==1.2.1" "jaxtyping<0.3.8"

A real example: bumping ax-platform pulled ax-platform → botorch → gpytorch → jaxtyping. jaxtyping>=0.3.8 requires Python>=3.11, but Ray’s CI also builds Python 3.10. pip reported only resolution-too-deep; uv reported the truth:

Because the current Python version (3.10) does not satisfy Python>=3.11 and jaxtyping==0.3.9 depends on Python>=3.11, jaxtyping==0.3.9 cannot be used. And because gpytorch depends on jaxtyping, …

The fix was a one-line floor: jaxtyping<0.3.8 (0.3.7 still supports 3.10).

Reproduce the CI install resolution#

To see what CI’s install step sees, pass the same constraint and requirement files to uv that the install scripts do:

uv pip install --dry-run --index-strategy unsafe-best-match \
  -c python/requirements.txt -c python/requirements_compiled.txt \
  -r python/requirements/ml/tune-requirements.txt

--index-strategy unsafe-best-match matters: the compiled file adds the PyTorch package index, and uv stops at the first index that has a package name (a defense against dependency-confusion attacks). That flag tells uv to check every configured index, matching pip’s default behavior. It’s safe here because all of Ray’s configured indexes are trusted.

Inspect a dependency chain#

pip show ax-platform botorch gpytorch jaxtyping   # who pulls what, what's installed

Edge cases and gotchas#

Platform-specific GPU dependencies#

torch and the PyTorch Geometric packages (torch-scatter, torch-sparse, torch-cluster, torch-spline-conv) ship CPU and GPU builds as distinct PEP 440 versions (1.2.2, 1.2.2+pt27cpu, 1.2.2+pt27cu128) on separate indexes. A single compile pass resolves against one index, so you can’t put both the CPU and GPU requirement files in one dependency set — the resolver is asked to satisfy +cpu and +cu128 of the same package at once and declares the requirements unsatisfiable.

The fix is to keep CPU and GPU as separate dependency sets, each with its own --index. Environment markers don’t work here: a CPU-only and a GPU x86_64 Linux host match the same markers, so a marker-based split installs CUDA wheels on CPU-only machines. This mirrors the existing ci_ml_build_depset (CPU) and ci_ml_gpubuild_depset (GPU) split.

A pin that requires a newer Python is a hidden conflict#

The jaxtyping story above generalizes: a transitive package can tighten its Requires-Python in a new release. The resolver then can’t use it on Ray’s lower supported Python, and you get a confusing failure far from the real cause. When a bump triggers a strange conflict, check the Requires-Python of every newly pulled transitive package, not just your direct change.

A theoretical pin clash isn’t always a real conflict#

uv is Python-version aware. A constraint like etils==1.14.0 that declares Requires-Python: >=3.11 is simply skipped when resolving for Python 3.10, so a lower source pin (etils==1.5.2) wins without conflict. Don’t treat two contradictory-looking pins as a blocker until you’ve reproduced an actual failure with uv pip install --dry-run.

Don’t `pip install -U` past the compiled constraint#

The pins in requirements_compiled.txt are intentional. Slipping a pip install -U <pkg> into a build or CI script after a constrained install silently bumps that package past the locked version and can desync the environment from every other image that shares the lock. If a package needs a newer version, raise it in the Layer 1 source file and recompile.

Changing the compiled file rebuilds every base image#

requirements_compiled.txt is part of the Wanda Docker cache key. Any change to it invalidates the cache and forces a full rebuild of every base image. That can surface pre-existing infrastructure problems that the cache normally hides (an expired apt signing key, a dead package mirror) in your PR, even though your change didn’t cause them. If a base-image build fails on a deps PR with an error unrelated to Python packaging, check whether it reproduces on master with the same cache miss before assuming it’s yours.

Dropping a requirements file drops transitive-only packages#

If you remove a requirements file from a dependency set to resolve a conflict, you also silently drop any package whose only entry point was that file (real example: torchtext, still used by a doctest). Packages also referenced from another file survive. After removing a file, diff the regenerated lock and re-add genuinely needed packages explicitly rather than re-adding the whole conflicting file.

Quick reference#

Task	Command
Recompile Layer 2 (x86_64, py3.11)	`ci/ci.sh compile_pip_dependencies`
Rebuild all Layer 3 locks	`bazelisk run //ci/raydepsets:raydepsets -- build --all-configs`
Rebuild one depset (fast iteration)	`bazelisk run //ci/raydepsets:raydepsets -- build <config>.yaml --name <depset_name>`
Validate locks are current	`bazelisk run //ci/raydepsets:raydepsets -- build --all-configs --check`
Find the true conflict	`uv pip install --dry-run <specs>`
Reproduce CI’s install resolution	`uv pip install --dry-run --index-strategy unsafe-best-match -c python/requirements_compiled.txt -r <file>`
Inspect a dependency chain	`pip show <pkg> ...`