Object Spilling#
Ray spills objects to a directory in the local filesystem once the object store is full. By default, Ray
spills objects to the temporary directory (for example, /tmp/ray/session_2025-03-28_00-05-20_204810_2814690
).
Spilling to a custom directory#
You can specify a custom directory for spilling objects by setting the
object_spilling_directory
parameter in the ray.init
function or the
--object-spilling-directory
command line option in the ray start
command.
ray.init(object_spilling_directory="/path/to/spill/dir")
ray start --object-spilling-directory=/path/to/spill/dir
For advanced usage and customizations, reach out to the Ray team.
Stats#
When spilling is happening, the following INFO level messages are printed to the Raylet logs-for example, /tmp/ray/session_latest/logs/raylet.out
:
local_object_manager.cc:166: Spilled 50 MiB, 1 objects, write throughput 230 MiB/s
local_object_manager.cc:334: Restored 50 MiB, 1 objects, read throughput 505 MiB/s
You can also view cluster-wide spill stats by using the ray memory
command:
--- Aggregate object store stats across all nodes ---
Plasma memory usage 50 MiB, 1 objects, 50.0% full
Spilled 200 MiB, 4 objects, avg write throughput 570 MiB/s
Restored 150 MiB, 3 objects, avg read throughput 1361 MiB/s
If you only want to display cluster-wide spill stats, use ray memory --stats-only
.