ray.data.preprocessors.Tokenizer
================================

.. currentmodule:: ray.data.preprocessors

.. autoclass:: Tokenizer
   :show-inheritance:

   
   
   .. rubric:: Methods

   .. autosummary::
      :toctree:

   
      
      ~Tokenizer.fit
      
      ~Tokenizer.fit_transform
      ~Tokenizer.preferred_batch_format
      ~Tokenizer.transform
      ~Tokenizer.transform_batch
      ~Tokenizer.transform_stats