.ipynb .pdf

repository open issue suggest edit

Binder Colab

Contents

Using RLlib with Tune

Example

Example of using PBT with RLlib.

Note that this requires a cluster with at least 8 GPUs in order for all trials to run concurrently, otherwise PBT will round-robin train the trials which is less efficient (or you can set {“gpu”: 0} to use CPUs for SGD instead).

Note that Tune in general does not need 8 GPUs, and this is just a more computationally demanding example.

import random

from ray import air, tune
from ray.tune.schedulers import PopulationBasedTraining

if __name__ == "__main__":

    # Postprocess the perturbed config to ensure it's still valid
    def explore(config):
        # ensure we collect enough timesteps to do sgd
        if config["train_batch_size"] < config["sgd_minibatch_size"] * 2:
            config["train_batch_size"] = config["sgd_minibatch_size"] * 2
        # ensure we run at least one sgd iter
        if config["num_sgd_iter"] < 1:
            config["num_sgd_iter"] = 1
        return config

    pbt = PopulationBasedTraining(
        time_attr="time_total_s",
        perturbation_interval=120,
        resample_probability=0.25,
        # Specifies the mutations of these hyperparams
        hyperparam_mutations={
            "lambda": lambda: random.uniform(0.9, 1.0),
            "clip_param": lambda: random.uniform(0.01, 0.5),
            "lr": [1e-3, 5e-4, 1e-4, 5e-5, 1e-5],
            "num_sgd_iter": lambda: random.randint(1, 30),
            "sgd_minibatch_size": lambda: random.randint(128, 16384),
            "train_batch_size": lambda: random.randint(2000, 160000),
        },
        custom_explore_fn=explore,
    )
    
    tuner = tune.Tuner(
        "PPO",
        tune_config=tune.TuneConfig(
            metric="episode_reward_mean",
            mode="max",
            scheduler=pbt,
            num_samples=1,
        ),
        param_space={
            "env": "Humanoid-v1",
            "kl_coeff": 1.0,
            "num_workers": 8,
            "num_gpus": 0, # number of GPUs to use
            "model": {"free_log_std": True},
            # These params are tuned from a fixed starting value.
            "lambda": 0.95,
            "clip_param": 0.2,
            "lr": 1e-4,
            # These params start off randomly drawn from a set.
            "num_sgd_iter": tune.choice([10, 20, 30]),
            "sgd_minibatch_size": tune.choice([128, 512, 2048]),
            "train_batch_size": tune.choice([10000, 20000, 40000]),
        },
    )
    results = tuner.fit()

    print("best hyperparameters: ", results.get_best_result().config)

2022-07-22 16:45:04,893	WARNING deprecation.py:48 -- DeprecationWarning: `ray.rllib.execution.buffers` has been deprecated. Use `ray.rllib.utils.replay_buffers` instead. This will raise an error in the future!
2022-07-22 16:45:08,004	INFO services.py:1483 -- View the Ray dashboard at http://127.0.0.1:8274
/Users/kai/coding/ray/python/ray/tune/tune.py:487: UserWarning: Consider boosting PBT performance by enabling `reuse_actors` as well as implementing `reset_config` for Trainable.
  "Consider boosting PBT performance by enabling `reuse_actors` as "

== Status ==
Current time: 2022-07-22 16:45:36 (running for 00:00:25.48)
Memory usage on this node: 11.3/16.0 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.53 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/PPO
Number of trials: 1/1 (1 ERROR)

Trial name	status	loc	num_sgd_iter	sgd_minibatch_size	train_batch_size
PPO_Humanoid-v1_45196_00000	ERROR		30	128	20000

Number of errored trials: 1

Trial name	# failures	error file
PPO_Humanoid-v1_45196_00000	1	/Users/kai/ray_results/PPO/PPO_Humanoid-v1_45196_00000_0_num_sgd_iter=30,sgd_minibatch_size=128,train_batch_size=20000_2022-07-22_16-45-11/error.txt

2022-07-22 16:45:11,206	WARNING trial_runner.py:1517 -- You are trying to access _search_alg interface of TrialRunner in TrialScheduler, which is being restricted. If you believe it is reasonable for your scheduler to access this TrialRunner API, please reach out to Ray team on GitHub. A more strict API access pattern would be enforced starting 1.12s.0
2022-07-22 16:45:11,640	INFO plugin_schema_manager.py:52 -- Loading the default runtime env schemas: ['/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/working_dir_schema.json', '/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/pip_schema.json'].
(PPO pid=53765) 2022-07-22 16:45:21,449	INFO algorithm.py:1855 -- Your framework setting is 'tf', meaning you are using static-graph mode. Set framework='tf2' to enable eager execution with tf2.x. You may also then want to set eager_tracing=True in order to reach similar execution speed as with static-graph mode.
(PPO pid=53765) 2022-07-22 16:45:21,450	WARNING ppo.py:359 -- `train_batch_size` (20000) cannot be achieved with your other settings (num_workers=8 num_envs_per_worker=1 rollout_fragment_length=200)! Auto-adjusting `rollout_fragment_length` to 2500.
(PPO pid=53765) 2022-07-22 16:45:21,450	INFO ppo.py:379 -- In multi-agent mode, policies will be optimized sequentially by the multi-GPU optimizer. Consider setting simple_optimizer=True if this doesn't work for you.
(PPO pid=53765) 2022-07-22 16:45:21,450	INFO algorithm.py:343 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
(PPO pid=53765) 2022-07-22 16:45:36,608	ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
(PPO pid=53765)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 127, in __init__
(PPO pid=53765)     validate=trainer_config.get("validate_workers_after_construction"),
(PPO pid=53765)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 269, in add_workers
(PPO pid=53765)     self.foreach_worker(lambda w: w.assert_healthy())
(PPO pid=53765)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 391, in foreach_worker
(PPO pid=53765)     remote_results = ray.get([w.apply.remote(func) for w in self.remote_workers()])
(PPO pid=53765) ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(PPO pid=53765) KeyError: 'Humanoid-v1'
(PPO pid=53765) 
(PPO pid=53765) During handling of the above exception, another exception occurred:
(PPO pid=53765) 
(PPO pid=53765) ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(PPO pid=53765)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(PPO pid=53765)     return gym.make(env_descriptor, **env_context)
(PPO pid=53765)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(PPO pid=53765)     return registry.make(id, **kwargs)
(PPO pid=53765)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(PPO pid=53765)     spec = self.spec(path)
(PPO pid=53765)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(PPO pid=53765)     id, matching_envs
(PPO pid=53765) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(PPO pid=53765) 
(PPO pid=53765) During handling of the above exception, another exception occurred:
(PPO pid=53765) 
(PPO pid=53765) ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(PPO pid=53765)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(PPO pid=53765)     self.env = env_creator(copy.deepcopy(self.env_context))
(PPO pid=53765)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(PPO pid=53765)     raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(PPO pid=53765) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(PPO pid=53765) a) Not a supported/installed environment.
(PPO pid=53765) b) Not a tune-registered environment creator.
(PPO pid=53765) c) Not a valid env class string.
(PPO pid=53765) 
(PPO pid=53765) Try one of the following:
(PPO pid=53765) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(PPO pid=53765)    For VizDoom support: Install VizDoom
(PPO pid=53765)    (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(PPO pid=53765)    `pip install vizdoomgym`.
(PPO pid=53765)    For PyBullet support: `pip install pybullet`.
(PPO pid=53765) b) To register your custom env, do `from ray import tune;
(PPO pid=53765)    tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(PPO pid=53765)    Then in your config, do `config['env'] = [name]`.
(PPO pid=53765) c) Make sure you provide a fully qualified classpath, e.g.:
(PPO pid=53765)    `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(PPO pid=53765) 
(PPO pid=53765) During handling of the above exception, another exception occurred:
(PPO pid=53765) 
(PPO pid=53765) ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
(PPO pid=53765)   File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 299, in __init__
(PPO pid=53765)     super().__init__(config=config, logger_creator=logger_creator, **kwargs)
(PPO pid=53765)   File "/Users/kai/coding/ray/python/ray/tune/trainable/trainable.py", line 157, in __init__
(PPO pid=53765)     self.setup(copy.deepcopy(self.config))
(PPO pid=53765)   File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 432, in setup
(PPO pid=53765)     raise e.args[0].args[2]
(PPO pid=53765) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(PPO pid=53765) a) Not a supported/installed environment.
(PPO pid=53765) b) Not a tune-registered environment creator.
(PPO pid=53765) c) Not a valid env class string.
(PPO pid=53765) 
(PPO pid=53765) Try one of the following:
(PPO pid=53765) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(PPO pid=53765)    For VizDoom support: Install VizDoom
(PPO pid=53765)    (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(PPO pid=53765)    `pip install vizdoomgym`.
(PPO pid=53765)    For PyBullet support: `pip install pybullet`.
(PPO pid=53765) b) To register your custom env, do `from ray import tune;
(PPO pid=53765)    tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(PPO pid=53765)    Then in your config, do `config['env'] = [name]`.
(PPO pid=53765) c) Make sure you provide a fully qualified classpath, e.g.:
(PPO pid=53765)    `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53785) 2022-07-22 16:45:36,570	ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53785, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b6a39f90>)
(RolloutWorker pid=53785) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53785) 
(RolloutWorker pid=53785) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53785) 
(RolloutWorker pid=53785) ray::RolloutWorker.__init__() (pid=53785, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b6a39f90>)
(RolloutWorker pid=53785)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53785)     return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53785)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53785)     return registry.make(id, **kwargs)
(RolloutWorker pid=53785)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53785)     spec = self.spec(path)
(RolloutWorker pid=53785)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53785)     id, matching_envs
(RolloutWorker pid=53785) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53785) 
(RolloutWorker pid=53785) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53785) 
(RolloutWorker pid=53785) ray::RolloutWorker.__init__() (pid=53785, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b6a39f90>)
(RolloutWorker pid=53785)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53785)     self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53785)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53785)     raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53785) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53785) a) Not a supported/installed environment.
(RolloutWorker pid=53785) b) Not a tune-registered environment creator.
(RolloutWorker pid=53785) c) Not a valid env class string.
(RolloutWorker pid=53785) 
(RolloutWorker pid=53785) Try one of the following:
(RolloutWorker pid=53785) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53785)    For VizDoom support: Install VizDoom
(RolloutWorker pid=53785)    (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53785)    `pip install vizdoomgym`.
(RolloutWorker pid=53785)    For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53785) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53785)    tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53785)    Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53785) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53785)    `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53787) 2022-07-22 16:45:36,570	ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53787, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b29b1990>)
(RolloutWorker pid=53787) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53787) 
(RolloutWorker pid=53787) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53787) 
(RolloutWorker pid=53787) ray::RolloutWorker.__init__() (pid=53787, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b29b1990>)
(RolloutWorker pid=53787)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53787)     return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53787)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53787)     return registry.make(id, **kwargs)
(RolloutWorker pid=53787)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53787)     spec = self.spec(path)
(RolloutWorker pid=53787)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53787)     id, matching_envs
(RolloutWorker pid=53787) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53787) 
(RolloutWorker pid=53787) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53787) 
(RolloutWorker pid=53787) ray::RolloutWorker.__init__() (pid=53787, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b29b1990>)
(RolloutWorker pid=53787)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53787)     self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53787)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53787)     raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53787) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53787) a) Not a supported/installed environment.
(RolloutWorker pid=53787) b) Not a tune-registered environment creator.
(RolloutWorker pid=53787) c) Not a valid env class string.
(RolloutWorker pid=53787) 
(RolloutWorker pid=53787) Try one of the following:
(RolloutWorker pid=53787) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53787)    For VizDoom support: Install VizDoom
(RolloutWorker pid=53787)    (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53787)    `pip install vizdoomgym`.
(RolloutWorker pid=53787)    For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53787) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53787)    tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53787)    Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53787) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53787)    `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53791) 2022-07-22 16:45:36,570	ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53791, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb744490>)
(RolloutWorker pid=53791) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53791) 
(RolloutWorker pid=53791) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53791) 
(RolloutWorker pid=53791) ray::RolloutWorker.__init__() (pid=53791, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb744490>)
(RolloutWorker pid=53791)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53791)     return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53791)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53791)     return registry.make(id, **kwargs)
(RolloutWorker pid=53791)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53791)     spec = self.spec(path)
(RolloutWorker pid=53791)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53791)     id, matching_envs
(RolloutWorker pid=53791) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53791) 
(RolloutWorker pid=53791) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53791) 
(RolloutWorker pid=53791) ray::RolloutWorker.__init__() (pid=53791, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb744490>)
(RolloutWorker pid=53791)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53791)     self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53791)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53791)     raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53791) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53791) a) Not a supported/installed environment.
(RolloutWorker pid=53791) b) Not a tune-registered environment creator.
(RolloutWorker pid=53791) c) Not a valid env class string.
(RolloutWorker pid=53791) 
(RolloutWorker pid=53791) Try one of the following:
(RolloutWorker pid=53791) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53791)    For VizDoom support: Install VizDoom
(RolloutWorker pid=53791)    (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53791)    `pip install vizdoomgym`.
(RolloutWorker pid=53791)    For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53791) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53791)    tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53791)    Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53791) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53791)    `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53788) 2022-07-22 16:45:36,570	ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53788, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb4f53d0>)
(RolloutWorker pid=53788) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53788) 
(RolloutWorker pid=53788) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53788) 
(RolloutWorker pid=53788) ray::RolloutWorker.__init__() (pid=53788, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb4f53d0>)
(RolloutWorker pid=53788)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53788)     return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53788)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53788)     return registry.make(id, **kwargs)
(RolloutWorker pid=53788)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53788)     spec = self.spec(path)
(RolloutWorker pid=53788)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53788)     id, matching_envs
(RolloutWorker pid=53788) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53788) 
(RolloutWorker pid=53788) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53788) 
(RolloutWorker pid=53788) ray::RolloutWorker.__init__() (pid=53788, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb4f53d0>)
(RolloutWorker pid=53788)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53788)     self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53788)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53788)     raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53788) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53788) a) Not a supported/installed environment.
(RolloutWorker pid=53788) b) Not a tune-registered environment creator.
(RolloutWorker pid=53788) c) Not a valid env class string.
(RolloutWorker pid=53788) 
(RolloutWorker pid=53788) Try one of the following:
(RolloutWorker pid=53788) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53788)    For VizDoom support: Install VizDoom
(RolloutWorker pid=53788)    (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53788)    `pip install vizdoomgym`.
(RolloutWorker pid=53788)    For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53788) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53788)    tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53788)    Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53788) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53788)    `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53792) 2022-07-22 16:45:36,570	ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53792, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x117d54a50>)
(RolloutWorker pid=53792) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53792) 
(RolloutWorker pid=53792) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53792) 
(RolloutWorker pid=53792) ray::RolloutWorker.__init__() (pid=53792, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x117d54a50>)
(RolloutWorker pid=53792)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53792)     return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53792)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53792)     return registry.make(id, **kwargs)
(RolloutWorker pid=53792)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53792)     spec = self.spec(path)
(RolloutWorker pid=53792)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53792)     id, matching_envs
(RolloutWorker pid=53792) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53792) 
(RolloutWorker pid=53792) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53792) 
(RolloutWorker pid=53792) ray::RolloutWorker.__init__() (pid=53792, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x117d54a50>)
(RolloutWorker pid=53792)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53792)     self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53792)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53792)     raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53792) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53792) a) Not a supported/installed environment.
(RolloutWorker pid=53792) b) Not a tune-registered environment creator.
(RolloutWorker pid=53792) c) Not a valid env class string.
(RolloutWorker pid=53792) 
(RolloutWorker pid=53792) Try one of the following:
(RolloutWorker pid=53792) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53792)    For VizDoom support: Install VizDoom
(RolloutWorker pid=53792)    (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53792)    `pip install vizdoomgym`.
(RolloutWorker pid=53792)    For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53792) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53792)    tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53792)    Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53792) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53792)    `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53786) 2022-07-22 16:45:36,570	ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(RolloutWorker pid=53786) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53786) 
(RolloutWorker pid=53786) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53786) 
(RolloutWorker pid=53786) ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(RolloutWorker pid=53786)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53786)     return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53786)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53786)     return registry.make(id, **kwargs)
(RolloutWorker pid=53786)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53786)     spec = self.spec(path)
(RolloutWorker pid=53786)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53786)     id, matching_envs
(RolloutWorker pid=53786) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53786) 
(RolloutWorker pid=53786) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53786) 
(RolloutWorker pid=53786) ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(RolloutWorker pid=53786)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53786)     self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53786)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53786)     raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53786) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53786) a) Not a supported/installed environment.
(RolloutWorker pid=53786) b) Not a tune-registered environment creator.
(RolloutWorker pid=53786) c) Not a valid env class string.
(RolloutWorker pid=53786) 
(RolloutWorker pid=53786) Try one of the following:
(RolloutWorker pid=53786) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53786)    For VizDoom support: Install VizDoom
(RolloutWorker pid=53786)    (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53786)    `pip install vizdoomgym`.
(RolloutWorker pid=53786)    For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53786) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53786)    tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53786)    Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53786) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53786)    `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53790) 2022-07-22 16:45:36,570	ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53790, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b35ba250>)
(RolloutWorker pid=53790) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53790) 
(RolloutWorker pid=53790) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53790) 
(RolloutWorker pid=53790) ray::RolloutWorker.__init__() (pid=53790, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b35ba250>)
(RolloutWorker pid=53790)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53790)     return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53790)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53790)     return registry.make(id, **kwargs)
(RolloutWorker pid=53790)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53790)     spec = self.spec(path)
(RolloutWorker pid=53790)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53790)     id, matching_envs
(RolloutWorker pid=53790) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53790) 
(RolloutWorker pid=53790) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53790) 
(RolloutWorker pid=53790) ray::RolloutWorker.__init__() (pid=53790, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b35ba250>)
(RolloutWorker pid=53790)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53790)     self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53790)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53790)     raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53790) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53790) a) Not a supported/installed environment.
(RolloutWorker pid=53790) b) Not a tune-registered environment creator.
(RolloutWorker pid=53790) c) Not a valid env class string.
(RolloutWorker pid=53790) 
(RolloutWorker pid=53790) Try one of the following:
(RolloutWorker pid=53790) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53790)    For VizDoom support: Install VizDoom
(RolloutWorker pid=53790)    (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53790)    `pip install vizdoomgym`.
(RolloutWorker pid=53790)    For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53790) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53790)    tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53790)    Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53790) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53790)    `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53789) 2022-07-22 16:45:36,570	ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53789, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1be518cd0>)
(RolloutWorker pid=53789) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53789) 
(RolloutWorker pid=53789) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53789) 
(RolloutWorker pid=53789) ray::RolloutWorker.__init__() (pid=53789, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1be518cd0>)
(RolloutWorker pid=53789)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53789)     return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53789)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53789)     return registry.make(id, **kwargs)
(RolloutWorker pid=53789)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53789)     spec = self.spec(path)
(RolloutWorker pid=53789)   File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53789)     id, matching_envs
(RolloutWorker pid=53789) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53789) 
(RolloutWorker pid=53789) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53789) 
(RolloutWorker pid=53789) ray::RolloutWorker.__init__() (pid=53789, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1be518cd0>)
(RolloutWorker pid=53789)   File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53789)     self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53789)   File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53789)     raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53789) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53789) a) Not a supported/installed environment.
(RolloutWorker pid=53789) b) Not a tune-registered environment creator.
(RolloutWorker pid=53789) c) Not a valid env class string.
(RolloutWorker pid=53789) 
(RolloutWorker pid=53789) Try one of the following:
(RolloutWorker pid=53789) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53789)    For VizDoom support: Install VizDoom
(RolloutWorker pid=53789)    (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53789)    `pip install vizdoomgym`.
(RolloutWorker pid=53789)    For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53789) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53789)    tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53789)    Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53789) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53789)    `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
2022-07-22 16:45:36,660	ERROR trial_runner.py:921 -- Trial PPO_Humanoid-v1_45196_00000: Error processing event.
ray.tune.error._TuneNoNextExecutorEventError: Traceback (most recent call last):
  File "/Users/kai/coding/ray/python/ray/tune/execution/ray_trial_executor.py", line 989, in get_next_executor_event
    future_result = ray.get(ready_future)
  File "/Users/kai/coding/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/Users/kai/coding/ray/python/ray/_private/worker.py", line 2199, in get
    raise value
  File "python/ray/_raylet.pyx", line 812, in ray._raylet.task_execution_handler
  File "python/ray/_raylet.pyx", line 623, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 772, in ray._raylet.execute_task
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
  File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 127, in __init__
    validate=trainer_config.get("validate_workers_after_construction"),
  File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 269, in add_workers
    self.foreach_worker(lambda w: w.assert_healthy())
  File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 391, in foreach_worker
    remote_results = ray.get([w.apply.remote(func) for w in self.remote_workers()])
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
KeyError: 'Humanoid-v1'

During handling of the above exception, another exception occurred:

ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
  File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
    return gym.make(env_descriptor, **env_context)
  File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
    return registry.make(id, **kwargs)
  File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
    spec = self.spec(path)
  File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
    id, matching_envs
gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])

During handling of the above exception, another exception occurred:

ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
  File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
    self.env = env_creator(copy.deepcopy(self.env_context))
  File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
    raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
a) Not a supported/installed environment.
b) Not a tune-registered environment creator.
c) Not a valid env class string.

Try one of the following:
a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
   For VizDoom support: Install VizDoom
   (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
   `pip install vizdoomgym`.
   For PyBullet support: `pip install pybullet`.
b) To register your custom env, do `from ray import tune;
   tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
   Then in your config, do `config['env'] = [name]`.
c) Make sure you provide a fully qualified classpath, e.g.:
   `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`

During handling of the above exception, another exception occurred:

ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
  File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 299, in __init__
    super().__init__(config=config, logger_creator=logger_creator, **kwargs)
  File "/Users/kai/coding/ray/python/ray/tune/trainable/trainable.py", line 157, in __init__
    self.setup(copy.deepcopy(self.config))
  File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 432, in setup
    raise e.args[0].args[2]
ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
a) Not a supported/installed environment.
b) Not a tune-registered environment creator.
c) Not a valid env class string.

Try one of the following:
a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
   For VizDoom support: Install VizDoom
   (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
   `pip install vizdoomgym`.
   For PyBullet support: `pip install pybullet`.
b) To register your custom env, do `from ray import tune;
   tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
   Then in your config, do `config['env'] = [name]`.
c) Make sure you provide a fully qualified classpath, e.g.:
   `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`

Result for PPO_Humanoid-v1_45196_00000:
  trial_id: '45196_00000'
  

2022-07-22 16:45:36,688	ERROR ray_trial_executor.py:104 -- An exception occurred when trying to stop the Ray actor:Traceback (most recent call last):
  File "/Users/kai/coding/ray/python/ray/tune/execution/ray_trial_executor.py", line 94, in post_stop_cleanup
    ray.get(future, timeout=0)
  File "/Users/kai/coding/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/Users/kai/coding/ray/python/ray/_private/worker.py", line 2199, in get
    raise value
  File "python/ray/_raylet.pyx", line 812, in ray._raylet.task_execution_handler
  File "python/ray/_raylet.pyx", line 623, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 772, in ray._raylet.execute_task
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
  File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 127, in __init__
    validate=trainer_config.get("validate_workers_after_construction"),
  File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 269, in add_workers
    self.foreach_worker(lambda w: w.assert_healthy())
  File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 391, in foreach_worker
    remote_results = ray.get([w.apply.remote(func) for w in self.remote_workers()])
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
KeyError: 'Humanoid-v1'

During handling of the above exception, another exception occurred:

ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
  File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
    return gym.make(env_descriptor, **env_context)
  File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
    return registry.make(id, **kwargs)
  File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
    spec = self.spec(path)
  File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
    id, matching_envs
gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])

During handling of the above exception, another exception occurred:

ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
  File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
    self.env = env_creator(copy.deepcopy(self.env_context))
  File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
    raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
a) Not a supported/installed environment.
b) Not a tune-registered environment creator.
c) Not a valid env class string.

Try one of the following:
a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
   For VizDoom support: Install VizDoom
   (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
   `pip install vizdoomgym`.
   For PyBullet support: `pip install pybullet`.
b) To register your custom env, do `from ray import tune;
   tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
   Then in your config, do `config['env'] = [name]`.
c) Make sure you provide a fully qualified classpath, e.g.:
   `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`

During handling of the above exception, another exception occurred:

ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
  File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 299, in __init__
    super().__init__(config=config, logger_creator=logger_creator, **kwargs)
  File "/Users/kai/coding/ray/python/ray/tune/trainable/trainable.py", line 157, in __init__
    self.setup(copy.deepcopy(self.config))
  File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 432, in setup
    raise e.args[0].args[2]
ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
a) Not a supported/installed environment.
b) Not a tune-registered environment creator.
c) Not a valid env class string.

Try one of the following:
a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
   For VizDoom support: Install VizDoom
   (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
   `pip install vizdoomgym`.
   For PyBullet support: `pip install pybullet`.
b) To register your custom env, do `from ray import tune;
   tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
   Then in your config, do `config['env'] = [name]`.
c) Make sure you provide a fully qualified classpath, e.g.:
   `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`

2022-07-22 16:45:36,795	ERROR tune.py:733 -- Trials did not complete: [PPO_Humanoid-v1_45196_00000]
2022-07-22 16:45:36,795	INFO tune.py:738 -- Total run time: 25.77 seconds (25.48 seconds for the tuning loop).

best hyperparameters:  None

More RLlib Examples

• PB2 PPO Example: Example of optimizing a distributed RLlib algorithm (PPO) with the PB2 scheduler. Uses a small population size of 4, so can train on a laptop.

previous

Model selection and serving with Ray Tune and Ray Serve

next

Tuning XGBoost parameters