.rst .pdf

repository open issue suggest edit

User Guides

If you’re new to Ray Datasets, we recommend starting with the Ray Datasets Quick Start. This user guide will help you navigate the Ray Datasets project and show you how achieve several tasks.

• Creating Datasets
  • Generating Synthetic Data
  • Reading Files From Storage
  • From In-Memory Data
  • From Torch and TensorFlow
  • From 🤗 (Hugging Face) Datasets
  • From Image Files (experimental)
  • Custom Datasources
  • Performance Considerations
• Transforming Datasets
  • Transformations
  • Writing UDFs
  • Compute Strategy
• Consuming Datasets
  • Retrieving a limited set of rows
  • Iterating over Datasets
  • Splitting Into and Consuming Shards
  • Saving Datasets
• ML Preprocessing
  • Last-mile Preprocessing
• ML Tensor Support
  • Creating Tensor Datasets
  • Transforming / Consuming Tensor Data
  • Saving Tensor Datasets
  • Disabling Tensor Extension Casting
  • Limitations
• Pipelining Compute
  • Creating a DatasetPipeline
  • Transforming Pipeline Windows
  • Example: Pipelined Batch Inference
  • Pipelines for ML Ingest
• Scheduling, Execution, and Memory Management
  • Scheduling
  • Execution
  • Memory Management
• Performance Tips and Tuning
  • Debugging Statistics
  • Batching Transforms
  • Parquet Column Pruning
  • Parquet Row Pruning
  • Tuning Read Parallelism
  • Enabling Push-Based Shuffle

previous

Key Concepts

next

Creating Datasets