ray.data.read_text#

ray.data.read_text(paths: Union[str, List[str]], *, encoding: str = 'utf-8', errors: str = 'ignore', drop_empty_lines: bool = True, filesystem: Optional[pyarrow.fs.FileSystem] = None, parallelism: int = -1, ray_remote_args: Optional[Dict[str, Any]] = None, arrow_open_stream_args: Optional[Dict[str, Any]] = None, meta_provider: ray.data.datasource.file_meta_provider.BaseFileMetadataProvider = <ray.data.datasource.file_meta_provider.DefaultFileMetadataProvider object>, partition_filter: Optional[ray.data.datasource.partitioning.PathPartitionFilter] = None, partitioning: ray.data.datasource.partitioning.Partitioning = None) → ray.data.dataset.Dataset[str][source]#

Create a dataset from lines stored in text files.

Examples

>>> import ray
>>> # Read a directory of files in remote storage.
>>> ray.data.read_text("s3://bucket/path") 

>>> # Read multiple local files.
>>> ray.data.read_text(["/path/to/file1", "/path/to/file2"])
Parameters

  • paths – A single file path or a list of file paths (or directories).

  • encoding – The encoding of the files (e.g., “utf-8” or “ascii”).

  • errors – What to do with errors on decoding. Specify either “strict”, “ignore”, or “replace”. Defaults to “ignore”.

  • filesystem – The filesystem implementation to read from.

  • parallelism – The requested parallelism of the read. Parallelism may be limited by the number of files of the dataset.

  • ray_remote_args – Kwargs passed to ray.remote in the read tasks and in the subsequent text decoding map task.

  • arrow_open_stream_args – kwargs passed to pyarrow.fs.FileSystem.open_input_stream

  • meta_provider – File metadata provider. Custom metadata providers may be able to resolve file metadata more quickly and/or accurately.

  • partition_filter – Path-based partition filter, if any. Can be used with a custom callback to read only selected partitions of a dataset. By default, this does not filter out any files. If wishing to filter out all file paths except those whose file extension matches e.g. “.txt”, a FileXtensionFilter("txt") can be provided.

  • partitioning – A Partitioning object that describes how paths are organized. Defaults to None.

Returns

Dataset holding lines of text read from the specified paths.

PublicAPI: This API is stable across Ray releases.