ProcessorConfig#

class ray.data.llm.ProcessorConfig(*, batch_size: int = 32, accelerator_type: str | None = None, concurrency: int | Tuple[int, int] = 1)[source]#

The processor configuration.

Parameters:

  • batch_size (int) – Configures batch size for the processor. Large batch sizes are likely to saturate the compute resources and could achieve higher throughput. On the other hand, small batch sizes are more fault-tolerant and could reduce bubbles in the data pipeline. You can tune the batch size to balance the throughput and fault-tolerance based on your use case.

  • accelerator_type (str | None) – The accelerator type used by the LLM stage in a processor. Default to None, meaning that only the CPU will be used.

  • concurrency (int | Tuple[int, int]) – The number of workers for data parallelism. Default to 1. If concurrency is a tuple (m, n), Ray creates an autoscaling actor pool that scales between m and n workers (1 <= m <= n). If concurrency is an int n, Ray uses either a fixed pool of n workers or an autoscaling pool from 1 to n workers, depending on the processor and stage.

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'protected_namespaces': (), 'validate_assignment': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].