ray.data.read_mcap#

ray.data.read_mcap(paths: str | List[str], *, topics: List[str] | Set[str] | None = None, time_range: Tuple[int, int] | TimeRange | None = None, message_types: List[str] | Set[str] | None = None, include_metadata: bool = True, filesystem: pyarrow.fs.FileSystem | None = None, parallelism: int = -1, num_cpus: float | None = None, num_gpus: float | None = None, memory: float | None = None, ray_remote_args: Dict[str, Any] | None = None, meta_provider: BaseFileMetadataProvider | None = None, partition_filter: PathPartitionFilter | None = None, partitioning: Partitioning = None, include_paths: bool = False, ignore_missing_paths: bool = False, shuffle: Literal['files'] | FileShuffleConfig | None = None, file_extensions: List[str] | None = None, concurrency: int | None = None, override_num_blocks: int | None = None) → Dataset[source]#

Create a Dataset from MCAP (Message Capture) files.

MCAP is a format commonly used in robotics and autonomous systems for storing ROS2 messages and other time-series data. This reader provides predicate pushdown optimization for efficient filtering by topics, time ranges, and message types.

Examples

Noindex:

Read all MCAP files in a directory.

>>> import ray
>>> ds = ray.data.read_mcap("s3://bucket/mcap-data/") 
>>> ds.schema()

Read with filtering for specific topics and time range.

>>> from ray.data.datasource import TimeRange  
>>> ds = ray.data.read_mcap( 
...     "s3://bucket/mcap-data/", 
...     topics={"/camera/image_raw", "/lidar/points"}, 
...     time_range=TimeRange(start_time=1000000000, end_time=5000000000), 
...     message_types={"sensor_msgs/Image", "sensor_msgs/PointCloud2"} 
... )

Alternatively, use a tuple for time range (backwards compatible).

>>> ds = ray.data.read_mcap( 
...     "s3://bucket/mcap-data/", 
...     topics={"/camera/image_raw", "/lidar/points"}, 
...     time_range=(1000000000, 5000000000), 
... )

Read multiple local files with include_paths.

>>> ray.data.read_mcap( 
...     ["local:///path/to/file1.mcap", "local:///path/to/file2.mcap"], 
...     include_paths=True 
... )

Read with topic filtering and metadata inclusion.

>>> ds = ray.data.read_mcap( 
...     "data.mcap", 
...     topics={"/camera/image_raw", "/lidar/points"}, 
...     include_metadata=True, 
...     include_paths=True 
... )
Parameters:

  • paths – A single file or directory, or a list of file or directory paths. A list of paths can contain both files and directories.

  • topics – Optional list or set of topic names to include. If specified, only messages from these topics will be read.

  • time_range – Optional time range for filtering messages by timestamp. Can be either a tuple of (start_time, end_time) in nanoseconds (for backwards compatibility) or a TimeRange object. Both values must be non-negative and start_time < end_time.

  • message_types – Optional list or set of message type names (schema names) to include. Only messages with matching schema names will be read.

  • include_metadata – Whether to include MCAP metadata fields in the output. Defaults to True. When True, includes schema, channel, and message metadata.

  • filesystem – The PyArrow filesystem implementation to read from.

  • parallelism – This argument is deprecated. Use override_num_blocks argument.

  • num_cpus – The number of CPUs to reserve for each parallel read worker.

  • num_gpus – The number of GPUs to reserve for each parallel read worker. For example, specify num_gpus=1 to request 1 GPU for each parallel read worker.

  • memory – The heap memory in bytes to reserve for each parallel read worker.

  • ray_remote_args – kwargs passed to ray.remote() in the read tasks.

  • meta_provider – A file metadata provider. Custom metadata providers may be able to resolve file metadata more quickly and/or accurately. In most cases you do not need to set this parameter.

  • partition_filter – A PathPartitionFilter. Use with a custom callback to read only selected partitions of a dataset.

  • partitioning – A Partitioning object that describes how paths are organized. Defaults to None.

  • include_paths – If True, include the path to each file. File paths are stored in the 'path' column.

  • ignore_missing_paths – If True, ignores any file paths in paths that are not found. Defaults to False.

  • shuffle – If setting to “files”, randomly shuffle input files order before read. If setting to FileShuffleConfig, you can pass a seed to shuffle the input files. Defaults to not shuffle with None.

  • file_extensions – A list of file extensions to filter files by. Defaults to ["mcap"].

  • concurrency – The maximum number of Ray tasks to run concurrently. Set this to control number of tasks to run concurrently. This doesn’t change the total number of tasks run or the total number of output blocks. By default, concurrency is dynamically decided based on the available resources.

  • override_num_blocks – Override the number of output blocks from all read tasks. By default, the number of output blocks is dynamically decided based on input data size and available resources. You shouldn’t manually set this value in most cases.

Returns:

Dataset producing records read from the specified MCAP files.

PublicAPI (alpha): This API is in alpha and may change before becoming stable.