Nested Remote Functions
Nested Remote Functions#
Remote functions can call other remote functions, resulting in nested tasks. For example, consider the following.
import ray @ray.remote def f(): return 1 @ray.remote def g(): # Call f 4 times and return the resulting object refs. return [f.remote() for _ in range(4)] @ray.remote def h(): # Call f 4 times, block until those 4 tasks finish, # retrieve the results, and return the values. return ray.get([f.remote() for _ in range(4)])
h produces the following behavior.
>>> ray.get(g.remote()) [ObjectRef(b1457ba0911ae84989aae86f89409e953dd9a80e), ObjectRef(7c14a1d13a56d8dc01e800761a66f09201104275), ObjectRef(99763728ffc1a2c0766a2000ebabded52514e9a6), ObjectRef(9c2f372e1933b04b2936bb6f58161285829b9914)] >>> ray.get(h.remote()) [1, 1, 1, 1]
One limitation is that the definition of
f must come before the
h because as soon as
g is defined, it
will be pickled and shipped to the workers, and so if
f hasn’t been
defined yet, the definition will be incomplete.
Yielding Resources While Blocked#
Ray will release CPU resources when being blocked. This prevents deadlock cases where the nested tasks are waiting for the CPU resources held by the parent task. Consider the following remote function.
@ray.remote(num_cpus=1, num_gpus=1) def g(): return ray.get(f.remote())
g task is executing, it will release its CPU resources when it gets
blocked in the call to
ray.get. It will reacquire the CPU resources when
ray.get returns. It will retain its GPU resources throughout the lifetime of
the task because the task will most likely continue to use GPU memory.