ray.data.from_huggingface#

ray.data.from_huggingface(dataset: datasets.Dataset | datasets.IterableDataset, parallelism: int = -1, concurrency: int | None = None, override_num_blocks: int | None = None) → MaterializedDataset | Dataset[source]#

Create a MaterializedDataset from a Hugging Face Datasets Dataset or a Dataset from a Hugging Face Datasets IterableDataset. For an IterableDataset, we use a streaming implementation to read data.

If the dataset is a public Hugging Face Dataset that is hosted on the Hugging Face Hub and no transformations have been applied, then the hosted parquet files will be passed to read_parquet() to perform a distributed read. All other cases will be done with a single node read.

Example

import ray
import datasets

hf_dataset = datasets.load_dataset("tweet_eval", "emotion")
ray_ds = ray.data.from_huggingface(hf_dataset["train"])
print(ray_ds)

hf_dataset_stream = datasets.load_dataset("tweet_eval", "emotion", streaming=True)
ray_ds_stream = ray.data.from_huggingface(hf_dataset_stream["train"])
print(ray_ds_stream)

MaterializedDataset(
    num_blocks=...,
    num_rows=3257,
    schema={text: string, label: int64}
)
Dataset(
    num_rows=3257,
    schema={text: string, label: int64}
)
Parameters:

  • dataset – A Hugging Face Datasets Dataset or Hugging Face Datasets IterableDataset. DatasetDict and IterableDatasetDict are not supported.

  • parallelism – This argument is deprecated. Use override_num_blocks argument.

  • concurrency – The maximum number of Ray tasks to run concurrently. Set this to control number of tasks to run concurrently. This doesn’t change the total number of tasks run or the total number of output blocks. By default, concurrency is dynamically decided based on the available resources.

  • override_num_blocks – Override the number of output blocks from all read tasks. By default, the number of output blocks is dynamically decided based on input data size and available resources. You shouldn’t manually set this value in most cases.

Returns:

A Dataset holding rows from the Hugging Face Datasets Dataset.