# ray.data.Dataset.std#

Dataset.std(on: Union[None, str, Callable[[ray.data.block.T], Any], List[Union[None, str, Callable[[ray.data.block.T], Any]]]] = None, ddof: int = 1, ignore_nulls: bool = True) ray.data.block.U[source]#

Compute standard deviation over entire dataset.

Note

This operation will trigger execution of the lazy transformations performed on this dataset, and will block until execution completes.

Examples

```>>> import ray
>>> ray.data.range(100).std()
29.011491975882016
>>> ray.data.from_items([
...     (i, i**2)
...     for i in range(100)]).std(lambda x: x[1])
2968.1748039269296
>>> ray.data.range_table(100).std("value", ddof=0)
28.86607004772212
>>> ray.data.from_items([
...     {"A": i, "B": i**2}
...     for i in range(100)]).std(["A", "B"])
{'std(A)': 29.011491975882016, 'std(B)': 2968.1748039269296}
```

Note

This uses Welford’s online method for an accumulator-style computation of the standard deviation. This method was chosen due to it’s numerical stability, and it being computable in a single pass. This may give different (but more accurate) results than NumPy, Pandas, and sklearn, which use a less numerically stable two-pass algorithm. See https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Welford’s_online_algorithm

Parameters
• on

The data subset on which to compute the std.

• For a simple dataset: it can be a callable or a list thereof, and the default is to return a scalar std of all rows.

• For an Arrow dataset: it can be a column name or a list thereof, and the default is to return an `ArrowRow` containing the column-wise std of all columns.

• ddof – Delta Degrees of Freedom. The divisor used in calculations is `N - ddof`, where `N` represents the number of elements.

• ignore_nulls – Whether to ignore null values. If `True`, null values will be ignored when computing the std; if `False`, if a null value is encountered, the output will be None. We consider np.nan, None, and pd.NaT to be null values. Default is `True`.

Returns

The standard deviation result.

For a simple dataset, the output is:

• `on=None`: a scalar representing the std of all rows,

• `on=callable`: a scalar representing the std of the outputs of the callable called on each row,

• `on=[callable_1, ..., calalble_n]`: a tuple of `(std_1, ..., std_n)` representing the std of the outputs of the corresponding callables called on each row.

For an Arrow dataset, the output is:

• `on=None`: an `ArrowRow` containing the column-wise std of all columns,

• `on="col"`: a scalar representing the std of all items in column `"col"`,

• `on=["col_1", ..., "col_n"]`: an n-column `ArrowRow` containing the column-wise std of the provided columns.

If the dataset is empty, all values are null, or any value is null AND `ignore_nulls` is `False`, then the output will be None.