ray.data.read_lance#

ray.data.read_lance(uri: str, *, columns: List[str] | None = None, filter: str | None = None, storage_options: Dict[str, str] | None = None, scanner_options: Dict[str, Any] | None = None, ray_remote_args: Dict[str, Any] | None = None, concurrency: int | None = None, override_num_blocks: int | None = None) → Dataset[source]#

Create a Dataset from a Lance Dataset.

Examples

>>> import ray
>>> ds = ray.data.read_lance( 
...     uri="./db_name.lance",
...     columns=["image", "label"],
...     filter="label = 2 AND text IS NOT NULL",
... )
Parameters:

  • uri – The URI of the Lance dataset to read from. Local file paths, S3, and GCS are supported.

  • columns – The columns to read. By default, all columns are read.

  • filter – Read returns only the rows matching the filter. By default, no filter is applied.

  • storage_options – Extra options that make sense for a particular storage connection. This is used to store connection parameters like credentials, endpoint, etc. For more information, see Object Store Configuration.

  • scanner_options – Additional options to configure the LanceDataset.scanner() method, such as batch_size. For more information, see LanceDB API doc

  • ray_remote_args – kwargs passed to remote() in the read tasks.

  • concurrency – The maximum number of Ray tasks to run concurrently. Set this to control number of tasks to run concurrently. This doesn’t change the total number of tasks run or the total number of output blocks. By default, concurrency is dynamically decided based on the available resources.

  • override_num_blocks – Override the number of output blocks from all read tasks. By default, the number of output blocks is dynamically decided based on input data size and available resources. You shouldn’t manually set this value in most cases.

Returns:

A Dataset producing records read from the Lance dataset.