ray.data.datasource.PathPartitionParser.of#

static PathPartitionParser.of(style: PartitionStyle = PartitionStyle.HIVE, base_dir: str | None = None, field_names: List[str] | None = None, field_types: Dict[str, Type[int | float | str | bool]] | None = None, filesystem: pyarrow.fs.FileSystem | None = None) → PathPartitionParser[source]#

Creates a path-based partition parser using a flattened argument list.

Parameters:

  • style – The partition style - may be either HIVE or DIRECTORY.

  • base_dir – “/”-delimited base directory to start searching for partitions (exclusive). File paths outside of this directory will be considered unpartitioned. Specify None or an empty string to search for partitions in all file path directories.

  • field_names – The partition key names. Required for DIRECTORY partitioning. Optional for HIVE partitioning. When non-empty, the order and length of partition key field names must match the order and length of partition directories discovered. Partition key field names are not required to exist in the dataset schema.

  • field_types – A dictionary that maps partition key names to their desired data type. If not provided, the data type default to string.

  • filesystem – Filesystem that will be used for partition path file I/O.

Returns:

The new path-based partition parser.