Dataset.map(fn: Union[Callable[[Dict[str, Any]], Dict[str, Any]], Callable[[Dict[str, Any]], Iterator[Dict[str, Any]]], _CallableClassProtocol], *, compute: Optional[ray.data._internal.compute.ComputeStrategy] = None, num_cpus: Optional[float] = None, num_gpus: Optional[float] = None, **ray_remote_args) Dataset[source]#

Apply the given function to each record of this dataset.

Note that mapping individual records can be quite slow. Consider using map_batches() for performance.


>>> import ray
>>> # Transform python objects.
>>> ds = ray.data.range(1000)
>>> # The function goes from record (Dict[str, Any]) to record.
>>> ds.map(lambda record: {"id": record["id"] * 2})
+- Dataset(num_blocks=..., num_rows=1000, schema={id: int64})
>>> # Transform Arrow records.
>>> ds = ray.data.from_items(
...     [{"value": i} for i in range(1000)])
>>> ds.map(lambda record: {"v2": record["value"] * 2})
+- Dataset(num_blocks=200, num_rows=1000, schema={value: int64})
>>> # Define a callable class that persists state across
>>> # function invocations for efficiency.
>>> init_model = ... 
>>> class CachedModel:
...    def __init__(self):
...        self.model = init_model()
...    def __call__(self, batch):
...        return self.model(batch)
>>> # Apply the transform in parallel on GPUs. Since
>>> # compute=ActorPoolStrategy(size=8) the transform will be applied on a
>>> # pool of 8 Ray actors, each allocated 1 GPU by Ray.
>>> ds.map(CachedModel, 
...        compute=ray.data.ActorPoolStrategy(size=8),
...        num_gpus=1)

Time complexity: O(dataset size / parallelism)

  • fn – The function to apply to each record, or a class type that can be instantiated to create such a callable. Callable classes are only supported for the actor compute strategy.

  • compute – The compute strategy, either None (default) to use Ray tasks, ray.data.ActorPoolStrategy(size=n) to use a fixed-size actor pool, or ray.data.ActorPoolStrategy(min_size=m, max_size=n) for an autoscaling actor pool.

  • num_cpus – The number of CPUs to reserve for each parallel map worker.

  • num_gpus – The number of GPUs to reserve for each parallel map worker. For example, specify num_gpus=1 to request 1 GPU for each parallel map worker.

  • ray_remote_args – Additional resource requirements to request from ray for each map worker.

See also


Call this method to create new records from existing ones. Unlike map(), a function passed to flat_map() can return multiple records.

flat_map() isn’t recommended because it’s slow; call map_batches() instead.


Call this method to transform batches of data. It’s faster and more flexible than map() and flat_map().