ray.data.Dataset.filter#

Dataset.filter(fn: Callable[[Dict[str, Any]], bool] | Callable[[Dict[str, Any]], Iterator[bool]] | _CallableClassProtocol | None = None, expr: str | None = None, *, compute: str | ComputeStrategy = None, fn_args: Iterable[Any] | None = None, fn_kwargs: Dict[str, Any] | None = None, fn_constructor_args: Iterable[Any] | None = None, fn_constructor_kwargs: Dict[str, Any] | None = None, concurrency: int | Tuple[int, int] | None = None, ray_remote_args_fn: Callable[[], Dict[str, Any]] | None = None, **ray_remote_args) → Dataset[source]#

Filter out rows that don’t satisfy the given predicate.

You can use either a function or a callable class or an expression string to perform the transformation. For functions, Ray Data uses stateless Ray tasks. For classes, Ray Data uses stateful Ray actors. For more information, see Stateful Transforms.

Tip

If you use the expr parameter with a Python expression string, Ray Data optimizes your filter with native Arrow interfaces.

Examples

>>> import ray
>>> ds = ray.data.range(100)
>>> ds.filter(expr="id <= 4").take_all()
[{'id': 0}, {'id': 1}, {'id': 2}, {'id': 3}, {'id': 4}]

Time complexity: O(dataset size / parallelism)

Parameters:

  • fn – The predicate to apply to each row, or a class type that can be instantiated to create such a callable.

  • expr – An expression string needs to be a valid Python expression that will be converted to pyarrow.dataset.Expression type.

  • fn_args – Positional arguments to pass to fn after the first argument. These arguments are top-level arguments to the underlying Ray task.

  • fn_kwargs – Keyword arguments to pass to fn. These arguments are top-level arguments to the underlying Ray task.

  • fn_constructor_args – Positional arguments to pass to fn’s constructor. You can only provide this if fn is a callable class. These arguments are top-level arguments in the underlying Ray actor construction task.

  • fn_constructor_kwargs – Keyword arguments to pass to fn’s constructor. This can only be provided if fn is a callable class. These arguments are top-level arguments in the underlying Ray actor construction task.

  • compute – This argument is deprecated. Use concurrency argument.

  • concurrency –

    The semantics of this argument depend on the type of fn:

    • If fn is a function and concurrency isn’t set (default), the actual concurrency is implicitly determined by the available resources and number of input blocks.

    • If fn is a function and concurrency is an int n, Ray Data launches at most n concurrent tasks.

    • If fn is a class and concurrency is an int n, Ray Data uses an actor pool with exactly n workers.

    • If fn is a class and concurrency is a tuple (m, n), Ray Data uses an autoscaling actor pool from m to n workers.

    • If fn is a class and concurrency isn’t set (default), this method raises an error.

  • ray_remote_args_fn – A function that returns a dictionary of remote args passed to each map worker. The purpose of this argument is to generate dynamic arguments for each actor/task, and will be called each time prior to initializing the worker. Args returned from this dict will always override the args in ray_remote_args. Note: this is an advanced, experimental feature.

  • ray_remote_args – Additional resource requirements to request from Ray (e.g., num_gpus=1 to request GPUs for the map tasks). See ray.remote() for details.