.rst .pdf

repository open issue suggest edit

User Guides

If you’re new to Ray Datasets, we recommend starting with the Ray Datasets Quick Start. This user guide will help you navigate the Ray Datasets project and show you how achieve several tasks, for instance you will learn

• how to load data and preprocess it for machine learning applications,

• how to use Tensors with Ray Datasets,

• how to run Dataset Pipelines in common scenarios,

• and how to tune your Ray Datasets applications for performance.

• Creating Datasets
• Saving Datasets
• Transforming Datasets
• Exchanging Datasets
• Pipelining Compute
  • Creating a DatasetPipeline
  • Per-Window Transformations
• ML Preprocessing
  • Last-mile Preprocessing
• Working with Tensors
  • Tables with tensor columns
  • Single-column tensor datasets
  • Reading existing serialized tensor columns
  • Working with tensor column datasets
  • Writing and reading tensor columns
  • End-to-end workflow with our Pandas extension type
  • Limitations
• Advanced Pipeline Usage
  • Handling Epochs
  • Example: Pipelined Batch Inference
  • Enabling Pipelining
  • Tuning Parallelism
  • Per-Epoch Shuffle Pipeline
  • Pre-repeat vs post-repeat transforms
  • Splitting pipelines for distributed ingest
  • Distributed Ingest with Ray Train
  • Changing Pipeline Structure
• Random Data Access (Experimental)
  • Architecture
  • Performance
  • Fault Tolerance
• Using Custom Datasources
• Performance Tips and Tuning
  • Debugging Statistics
  • Batching Transforms
  • Parquet Column Pruning
  • Tuning Read Parallelism
  • Tuning Max Block Size

previous

Key Concepts

next

Creating Datasets