read_lerobot#

Creates a Dataset from a LeRobot v3 dataset.

LeRobot is a platform for sharing datasets and pretrained models for real-world robotics. A LeRobot v3 dataset stores low-dimensional data (state, action, timestamps) in chunked Parquet files and camera observations either as chunked MP4 video files or as encoded images stored inline in the Parquet rows. This reader decodes camera frames (video via torchcodec, images via Pillow) and aligns them with the parquet data using episode metadata.

Output columns include index, episode_index, frame_index, timestamp, state/action vectors, decoded camera frames (as variable-shaped uint8 tensors), task (string), dataset_index (int32, identifies the source root when reading multiple datasets), and stats. stats is a JSON string of the source dataset’s per-feature normalization statistics, keyed by feature name (e.g. {"action": {"mean": [...], "std": [...]}}), for downstream normalization of state/action vectors. It is a per-dataset constant: the same value on every row, dictionary-encoded so it is stored once per block rather than duplicated. It is "{}" when the dataset has no stats.

Examples

Read a LeRobot v3 dataset from a public S3 bucket (anonymous access):

>>> import ray
>>> ds = ray.data.read_lerobot(
...     "s3://anonymous@ray-example-data/lerobot/libero-mini",
... )
>>> ds.schema()

One read task per episode (instead of per file group):

>>> ds = ray.data.read_lerobot(
...     "s3://anonymous@ray-example-data/lerobot/libero-mini",
...     read_granularity="episode",
... )

Read multiple datasets as one (paths may be local or cloud URIs):

>>> ds = ray.data.read_lerobot(
...     ["/path/to/ds1", "/path/to/ds2"],
... )

Parameters:

root – Path or URI to the dataset root (local, gs://, s3://), or a list of such paths to read multiple datasets as one. All roots must share the same video_keys, image_keys, fps, and non-video feature names.
episodes – If given, read only these episode_index values. This is a read-time pushdown – other episodes’ parquet rows and video files are never opened – so it is cheaper than reading everything and filter-ing afterward. Row values are preserved, not renumbered: index, episode_index, frame_index and timestamp keep their original values, so index becomes non-contiguous (gaps where episodes were skipped). Applied per root when reading multiple roots; requesting an episode_index absent from every root raises. None (the default) reads all episodes.
read_granularity – How rows are grouped into the base read tasks. A file group is the set of consecutive episodes whose frames share one physical file (an mp4 per camera for video datasets, or the data parquet for image datasets). "file" (the default) emits one task per file group, so each file is opened once; "episode" emits one task per episode. Use override_num_blocks to tune the final number of output blocks.
filesystem – Filesystem for reading metadata and parquet. A pyarrow FileSystem (wrapped internally with ArrowFSWrapper) or an fsspec AbstractFileSystem. By default it is selected from the URI scheme, including the s3://anonymous@bucket/… convention for public buckets. For credentialed cloud datasets the recommended setup is a pyarrow filesystem together with storage_options (see below): the filesystem covers metadata and parquet, and storage_options supplies the credentials for the by-URI video decode path.
frame_tolerance_s – Max seconds a decoded video frame’s timestamp may differ from a row’s timestamp before it is rejected. None (the default) uses 0.5 / fps — half a frame interval, e.g. ~0.05s at 10fps. Increase to tolerate timestamp jitter; decrease for stricter alignment.
storage_options – fsspec storage options (e.g. credentials or a custom endpoint_url). They supply the credentials for the by-URI video decode path – videos are streamed directly through torchcodec / fsspec, not through filesystem – so pass them alongside a pyarrow filesystem for credentialed cloud video. When filesystem is not given, they also resolve the metadata / parquet filesystem.
num_cpus – The number of CPUs to reserve for each parallel read worker. Video decoding is CPU-intensive, so raising this (and lowering concurrency) can prevent oversubscription.
num_gpus – The number of GPUs to reserve for each parallel read worker. Video frames are decoded on CPU (torchcodec), so this does not accelerate decoding – it only reserves GPUs for the read tasks.
memory – The heap memory in bytes to reserve for each parallel read worker.
ray_remote_args – kwargs passed to ray.remote() in the read tasks.
concurrency – The maximum number of Ray tasks to run concurrently. Set this to control number of tasks to run concurrently. This doesn’t change the total number of tasks run or the total number of output blocks. By default, concurrency is dynamically decided based on the available resources. Use it to cap the number of simultaneous video decoders.
override_num_blocks – Override the number of output blocks from all read tasks. By default this follows read_granularity: one read task per file group (per video file for video datasets, per data-parquet file for image/tabular ones), or per episode, so each file is opened once; raise it (e.g. to your cluster’s CPU count) to parallelize a monolithic dataset across more workers. Splitting a file group re-opens its file(s) once per sub-task, so higher parallelism trades amortized file opens for more concurrency; lowering it merges groups.

Returns:

A Dataset of fully-decoded frames with state, action, camera, task, and metadata columns.

PublicAPI (alpha): This API is in alpha and may change before becoming stable.