ray.data.datasource.ParquetMetadataProvider
ray.data.datasource.ParquetMetadataProvider#
- class ray.data.datasource.ParquetMetadataProvider[source]#
Bases:
ray.data.datasource.file_meta_provider.FileMetadataProvider
Abstract callable that provides block metadata for Arrow Parquet file fragments.
All file fragments should belong to a single dataset block.
Supports optional pre-fetching of ordered metadata for all file fragments in a single batch to help optimize metadata resolution.
- Current subclasses:
DeveloperAPI: This API may change across minor Ray releases.
Methods
Pre-fetches file metadata for all Parquet file fragments in a single batch.