Nested remote functions

A remote function can call other remote functions, which results in nested tasks. Consider the following example:

import ray


@ray.remote
def f():
    return 1


@ray.remote
def g():
    # Call f 4 times and return the resulting object refs.
    return [f.remote() for _ in range(4)]


@ray.remote
def h():
    # Call f 4 times, block until those 4 tasks finish,
    # retrieve the results, and return the values.
    return ray.get([f.remote() for _ in range(4)])

Calling g and h produces the following output:

>>> ray.get(g.remote())
[ObjectRef(b1457ba0911ae84989aae86f89409e953dd9a80e),
 ObjectRef(7c14a1d13a56d8dc01e800761a66f09201104275),
 ObjectRef(99763728ffc1a2c0766a2000ebabded52514e9a6),
 ObjectRef(9c2f372e1933b04b2936bb6f58161285829b9914)]

>>> ray.get(h.remote())
[1, 1, 1, 1]

Note

Define f before you first call g.remote() or h.remote(). Ray pickles g and ships it to the workers the first time you call g.remote(). If you haven’t defined f by then, the definition of g is incomplete.

Yielding resources while blocked

Ray releases a task’s CPU resources while the task blocks. This prevents deadlocks where nested tasks wait for CPU resources that the parent task holds. Consider the following remote function:

@ray.remote(num_cpus=1, num_gpus=1)
def g():
    return ray.get(f.remote())

While a g task runs, it releases its CPU resources when it blocks in the call to ray.get, and it reacquires them when ray.get returns. The task keeps its GPU resources for its whole lifetime because it most likely continues to use GPU memory.