ray.data.datasource.PathPartitionFilter.of#

static PathPartitionFilter.of(filter_fn: Callable[[Dict[str, str]], bool], style: PartitionStyle = PartitionStyle.HIVE, base_dir: str | None = None, field_names: List[str] | None = None, field_types: Dict[str, Type[int | float | str | bool]] | None = None, filesystem: pyarrow.fs.FileSystem | None = None) → PathPartitionFilter[source]#

Creates a path-based partition filter using a flattened argument list.

Parameters:

  • filter_fn –

    Callback used to filter partitions. Takes a dictionary mapping partition keys to values as input. Unpartitioned files are denoted with an empty input dictionary. Returns True to read a file for that partition or False to skip it. Partition keys and values are always strings read from the filesystem path. For example, this removes all unpartitioned files:

    lambda d: True if d else False

    This raises an assertion error for any unpartitioned file found:

    def do_assert(val, msg):
    assert val, msg

lambda d: do_assert(d, "Expected all files to be partitioned!")

    And this only reads files from January, 2022 partitions:

    lambda d: d["month"] == "January" and d["year"] == "2022"

  • style – The partition style - may be either HIVE or DIRECTORY.

  • base_dir – “/”-delimited base directory to start searching for partitions (exclusive). File paths outside of this directory will be considered unpartitioned. Specify None or an empty string to search for partitions in all file path directories.

  • field_names – The partition key names. Required for DIRECTORY partitioning. Optional for HIVE partitioning. When non-empty, the order and length of partition key field names must match the order and length of partition directories discovered. Partition key field names are not required to exist in the dataset schema.

  • field_types – A dictionary that maps partition key names to their desired data type. If not provided, the data type defaults to string.

  • filesystem – Filesystem that will be used for partition path file I/O.

Returns:

The new path-based partition filter.