drop_columns#

Dataset.drop_columns(cols: List[str], *, compute: str | None = None, concurrency: int | None = None, num_cpus: float | None = None, num_gpus: float | None = None, memory: float | None = None, label_selector: Dict[str, str] | None = None, fallback_strategy: List[Dict[str, Any]] | None = None, max_calls: int | None = None, resources: Dict[str, float] | None = None, accelerator_type: str | None = None, runtime_env: Dict[str, Any] | None = None, **ray_remote_args) → Dataset[source]#

Drop one or more columns from the dataset.

Examples

>>> import ray
>>> ds = ray.data.read_parquet("s3://anonymous@ray-example-data/iris.parquet")
>>> ds.schema()
Column        Type
------        ----
sepal.length  double
sepal.width   double
petal.length  double
petal.width   double
variety       string
>>> ds.drop_columns(["variety"]).schema()
Column        Type
------        ----
sepal.length  double
sepal.width   double
petal.length  double
petal.width   double

Time complexity: O(dataset size / parallelism)

Parameters:

  • cols (List[str]) – Names of the columns to drop. If any name does not exist, an exception is raised. Column names must be unique. When the input schema is known statically, missing columns are reported at the drop_columns call; otherwise the error surfaces during materialization.

  • compute (str | None) – This argument is deprecated. Use concurrency argument.

  • concurrency (int | None) – The maximum number of Ray workers to use concurrently.

  • num_cpus (float | None) – The number of CPUs to reserve for each worker.

  • num_gpus (float | None) – The number of GPUs to reserve for each worker.

  • memory (float | None) – The heap memory in bytes to reserve for each worker.

  • label_selector (Dict[str, str] | None) – Labels required on the node where each worker runs.

  • fallback_strategy (List[Dict[str, Any]] | None) – Alternative label requirements that Ray tries in order when label_selector can’t be satisfied.

  • max_calls (int | None) – The maximum number of calls a task worker handles before exiting. This option only applies to task workers.

  • resources (Dict[str, float] | None) – Custom resources to reserve for each worker, expressed as a mapping from resource name to quantity.

  • accelerator_type (str | None) – The accelerator type required on the node where each worker runs.

  • runtime_env (Dict[str, Any] | None) – The runtime environment to use for each worker.

  • **ray_remote_args – Additional resource requirements to request from Ray (e.g., num_gpus=1 to request GPUs for the map tasks). See ray.remote() for details. This argument is deprecated and will be removed in Ray 2.64.

Returns:

A new Dataset with the specified columns removed.

Return type:

Dataset

On this page