union

Dataset.union(*other: Dataset) → Dataset

Concatenate Datasets across rows.

The union doesn’t append one dataset after another. By default, it emits blocks in the order the input datasets produce them, so the output order isn’t deterministic. If you set DataContext.execution_options.preserve_order to True, it interleaves blocks round-robin: one block from each dataset in argument order, then the next block from each, until every dataset runs out. Within each dataset, blocks keep their order.

Caution

Unioned datasets aren’t lineage-serializable. As a result, they can’t be used as a tunable hyperparameter in Ray Tune.

Examples

>>> import ray
>>> ctx = ray.data.DataContext.get_current()
>>> ctx.execution_options.preserve_order = True
>>> ds1 = ray.data.range(4, override_num_blocks=2)
>>> ds2 = ray.data.range(4, override_num_blocks=2)
>>> [row["id"] for row in ds1.union(ds2).take_all()]
[0, 1, 0, 1, 2, 3, 2, 3]
>>> ctx.execution_options.preserve_order = False

Parameters:

*other (Dataset) – The datasets to combine with this one. The datasets must have the same schema as this dataset, otherwise the behavior is undefined.

Returns:

A new dataset holding the rows of the input datasets.

Return type:

Dataset