with_columns#

Dataset.with_columns(exprs: Mapping[str, Expr] | UnnestExpr | None = None, *more_exprs: Mapping[str, Expr] | UnnestExpr, compute: ComputeStrategy | None = None, num_cpus: float | None = None, num_gpus: float | None = None, memory: float | None = None, label_selector: Dict[str, str] | None = None, fallback_strategy: List[Dict[str, Any]] | None = None, max_calls: int | None = None, resources: Dict[str, float] | None = None, accelerator_type: str | None = None, runtime_env: Dict[str, Any] | None = None, **ray_remote_args) → Dataset[source]#

Add or overwrite multiple columns via expressions in a single projection.

This is the multi-column counterpart of with_column(). All expressions are evaluated within one projection over the existing columns, which avoids the repeated work of chaining several with_column calls.

In addition to a mapping from column name to expression, positional arguments may be unnest() expressions: each wraps a struct-typed expression and contributes one output column per struct field, named after the fields.

Examples

>>> import ray
>>> from ray.data.expressions import col
>>> ds = ray.data.range(100)
>>> ds.with_columns({
...     "id_2": col("id") * 2,
...     "id_3": col("id") * 3,
... }).show(2)
{'id': 0, 'id_2': 0, 'id_3': 0}
{'id': 1, 'id_2': 2, 'id_3': 3}

See unnest() for expanding a struct-returning UDF into multiple columns, including mixed usage such as ds.with_columns({"a2": col("a") * 2}, unnest(make_features(col("a"), col("b")))).

Parameters:

  • exprs (Mapping[str, Expr] | UnnestExpr | None) – A mapping from new column name to the expression that defines its values, or an unnest() expression. Column order follows the mapping’s insertion order.

  • *more_exprs (Mapping[str, Expr] | UnnestExpr) – Additional mappings or unnest() expressions, appended in argument order.

  • compute (ComputeStrategy | None) – The compute strategy to use for the projection operation.

  • num_cpus (float | None) – The number of CPUs to reserve for each worker.

  • num_gpus (float | None) – The number of GPUs to reserve for each worker.

  • memory (float | None) – The heap memory in bytes to reserve for each worker.

  • label_selector (Dict[str, str] | None) – Labels required on the node where each worker runs.

  • fallback_strategy (List[Dict[str, Any]] | None) – Alternative label requirements that Ray tries in order when label_selector can’t be satisfied.

  • max_calls (int | None) – The maximum number of calls a task worker handles before exiting. This option only applies to task workers.

  • resources (Dict[str, float] | None) – Custom resources to reserve for each worker, expressed as a mapping from resource name to quantity.

  • accelerator_type (str | None) – The accelerator type required on the node where each worker runs.

  • runtime_env (Dict[str, Any] | None) – The runtime environment to use for each worker.

  • **ray_remote_args – Additional resource requirements to request from Ray for the map tasks (e.g., num_gpus=1). This argument is deprecated and will be removed in Ray 2.64.

Returns:

A new dataset with the added or overwritten columns.

Return type:

Dataset

PublicAPI (alpha): This API is in alpha and may change before becoming stable.

On this page