ray.rllib.core.models.catalog.Catalog.get_tokenizer_config#

classmethod Catalog.get_tokenizer_config(observation_space: gymnasium.Space, model_config_dict: dict, view_requirements=-1) → ModelConfig[source]#

Returns a tokenizer config for the given space.

This is useful for recurrent / transformer models that need to tokenize their inputs. By default, RLlib uses the models supported by Catalog out of the box to tokenize.

You should override this method if you want to change the custom tokenizer inside current encoders that Catalog returns without providing the recurrent network as a whole. For example, if you want to define some custom CNN layers as a tokenizer for a recurrent encoder that already includes the recurrent layers and handles the state.

Parameters:

  • observation_space – The observation space to use.

  • model_config_dict – The model config to use.