User Guides

If you’re new to Ray Datasets, we recommend starting with the Ray Datasets Quick Start. This user guide will help you navigate the Ray Datasets project and show you how achieve several tasks.

• Creating Datasets
  • Generating Synthetic Data
  • Reading Files From Storage
  • From In-Memory Data
  • From Torch and TensorFlow
  • From 🤗 (Hugging Face) Datasets
  • From MongoDB
  • Reading From SQL Databases
  • Custom Datasources
  • Performance Considerations
• Transforming Datasets
  • Transformations
  • Writing User-defined Functions (UDFs)
  • Configuring Batch Size
  • Compute Strategy
  • Group-bys and aggregations
  • Shuffling data
• Consuming Datasets
  • Retrieving a limited set of rows
  • Iterating over Datasets
  • Splitting Into and Consuming Shards
  • Saving Datasets
• ML Tensor Support
  • Creating Tensor Datasets
  • Transforming / Consuming Tensor Data
  • Saving Tensor Datasets
  • Ragged Tensor Support
  • Disabling Tensor Extension Casting
  • Limitations
• Custom Datasources
  • Read support
  • Write support
  • Putting it all together
• Pipelining Compute
  • Creating a DatasetPipeline
  • Transforming Pipeline Windows
  • Example: Pipelined Batch Inference
  • Pipelines for ML Ingest
• Scheduling, Execution, and Memory Management
  • Scheduling
  • Execution
  • Memory Management
• Performance Tips and Tuning
  • Debugging Statistics
  • Batching Transforms
  • Parquet Column Pruning
  • Parquet Row Pruning
  • Tuning Read Parallelism
  • Enabling Push-Based Shuffle