Using RLlib with Tune
Contents
Using RLlib with Tune#
Example#
Example of using PBT with RLlib.
Note that this requires a cluster with at least 8 GPUs in order for all trials to run concurrently, otherwise PBT will round-robin train the trials which is less efficient (or you can set {“gpu”: 0} to use CPUs for SGD instead).
Note that Tune in general does not need 8 GPUs, and this is just a more computationally demanding example.
import random
from ray import air, tune
from ray.tune.schedulers import PopulationBasedTraining
if __name__ == "__main__":
# Postprocess the perturbed config to ensure it's still valid
def explore(config):
# ensure we collect enough timesteps to do sgd
if config["train_batch_size"] < config["sgd_minibatch_size"] * 2:
config["train_batch_size"] = config["sgd_minibatch_size"] * 2
# ensure we run at least one sgd iter
if config["num_sgd_iter"] < 1:
config["num_sgd_iter"] = 1
return config
pbt = PopulationBasedTraining(
time_attr="time_total_s",
perturbation_interval=120,
resample_probability=0.25,
# Specifies the mutations of these hyperparams
hyperparam_mutations={
"lambda": lambda: random.uniform(0.9, 1.0),
"clip_param": lambda: random.uniform(0.01, 0.5),
"lr": [1e-3, 5e-4, 1e-4, 5e-5, 1e-5],
"num_sgd_iter": lambda: random.randint(1, 30),
"sgd_minibatch_size": lambda: random.randint(128, 16384),
"train_batch_size": lambda: random.randint(2000, 160000),
},
custom_explore_fn=explore,
)
tuner = tune.Tuner(
"PPO",
tune_config=tune.TuneConfig(
metric="episode_reward_mean",
mode="max",
scheduler=pbt,
num_samples=1,
),
param_space={
"env": "Humanoid-v1",
"kl_coeff": 1.0,
"num_workers": 8,
"num_gpus": 0, # number of GPUs to use
"model": {"free_log_std": True},
# These params are tuned from a fixed starting value.
"lambda": 0.95,
"clip_param": 0.2,
"lr": 1e-4,
# These params start off randomly drawn from a set.
"num_sgd_iter": tune.choice([10, 20, 30]),
"sgd_minibatch_size": tune.choice([128, 512, 2048]),
"train_batch_size": tune.choice([10000, 20000, 40000]),
},
)
results = tuner.fit()
print("best hyperparameters: ", results.get_best_result().config)
2022-07-22 16:45:04,893 WARNING deprecation.py:48 -- DeprecationWarning: `ray.rllib.execution.buffers` has been deprecated. Use `ray.rllib.utils.replay_buffers` instead. This will raise an error in the future!
2022-07-22 16:45:08,004 INFO services.py:1483 -- View the Ray dashboard at http://127.0.0.1:8274
/Users/kai/coding/ray/python/ray/tune/tune.py:487: UserWarning: Consider boosting PBT performance by enabling `reuse_actors` as well as implementing `reset_config` for Trainable.
"Consider boosting PBT performance by enabling `reuse_actors` as "
== Status ==
Current time: 2022-07-22 16:45:36 (running for 00:00:25.48)
Memory usage on this node: 11.3/16.0 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.53 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/PPO
Number of trials: 1/1 (1 ERROR)
Number of errored trials: 1
Current time: 2022-07-22 16:45:36 (running for 00:00:25.48)
Memory usage on this node: 11.3/16.0 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.53 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/PPO
Number of trials: 1/1 (1 ERROR)
| Trial name | status | loc | num_sgd_iter | sgd_minibatch_size | train_batch_size |
|---|---|---|---|---|---|
| PPO_Humanoid-v1_45196_00000 | ERROR | 30 | 128 | 20000 |
Number of errored trials: 1
| Trial name | # failures | error file |
|---|---|---|
| PPO_Humanoid-v1_45196_00000 | 1 | /Users/kai/ray_results/PPO/PPO_Humanoid-v1_45196_00000_0_num_sgd_iter=30,sgd_minibatch_size=128,train_batch_size=20000_2022-07-22_16-45-11/error.txt |
2022-07-22 16:45:11,206 WARNING trial_runner.py:1517 -- You are trying to access _search_alg interface of TrialRunner in TrialScheduler, which is being restricted. If you believe it is reasonable for your scheduler to access this TrialRunner API, please reach out to Ray team on GitHub. A more strict API access pattern would be enforced starting 1.12s.0
2022-07-22 16:45:11,640 INFO plugin_schema_manager.py:52 -- Loading the default runtime env schemas: ['/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/working_dir_schema.json', '/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/pip_schema.json'].
(PPO pid=53765) 2022-07-22 16:45:21,449 INFO algorithm.py:1855 -- Your framework setting is 'tf', meaning you are using static-graph mode. Set framework='tf2' to enable eager execution with tf2.x. You may also then want to set eager_tracing=True in order to reach similar execution speed as with static-graph mode.
(PPO pid=53765) 2022-07-22 16:45:21,450 WARNING ppo.py:359 -- `train_batch_size` (20000) cannot be achieved with your other settings (num_workers=8 num_envs_per_worker=1 rollout_fragment_length=200)! Auto-adjusting `rollout_fragment_length` to 2500.
(PPO pid=53765) 2022-07-22 16:45:21,450 INFO ppo.py:379 -- In multi-agent mode, policies will be optimized sequentially by the multi-GPU optimizer. Consider setting simple_optimizer=True if this doesn't work for you.
(PPO pid=53765) 2022-07-22 16:45:21,450 INFO algorithm.py:343 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
(PPO pid=53765) 2022-07-22 16:45:36,608 ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
(PPO pid=53765) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 127, in __init__
(PPO pid=53765) validate=trainer_config.get("validate_workers_after_construction"),
(PPO pid=53765) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 269, in add_workers
(PPO pid=53765) self.foreach_worker(lambda w: w.assert_healthy())
(PPO pid=53765) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 391, in foreach_worker
(PPO pid=53765) remote_results = ray.get([w.apply.remote(func) for w in self.remote_workers()])
(PPO pid=53765) ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(PPO pid=53765) KeyError: 'Humanoid-v1'
(PPO pid=53765)
(PPO pid=53765) During handling of the above exception, another exception occurred:
(PPO pid=53765)
(PPO pid=53765) ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(PPO pid=53765) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(PPO pid=53765) return gym.make(env_descriptor, **env_context)
(PPO pid=53765) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(PPO pid=53765) return registry.make(id, **kwargs)
(PPO pid=53765) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(PPO pid=53765) spec = self.spec(path)
(PPO pid=53765) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(PPO pid=53765) id, matching_envs
(PPO pid=53765) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(PPO pid=53765)
(PPO pid=53765) During handling of the above exception, another exception occurred:
(PPO pid=53765)
(PPO pid=53765) ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(PPO pid=53765) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(PPO pid=53765) self.env = env_creator(copy.deepcopy(self.env_context))
(PPO pid=53765) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(PPO pid=53765) raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(PPO pid=53765) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(PPO pid=53765) a) Not a supported/installed environment.
(PPO pid=53765) b) Not a tune-registered environment creator.
(PPO pid=53765) c) Not a valid env class string.
(PPO pid=53765)
(PPO pid=53765) Try one of the following:
(PPO pid=53765) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(PPO pid=53765) For VizDoom support: Install VizDoom
(PPO pid=53765) (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(PPO pid=53765) `pip install vizdoomgym`.
(PPO pid=53765) For PyBullet support: `pip install pybullet`.
(PPO pid=53765) b) To register your custom env, do `from ray import tune;
(PPO pid=53765) tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(PPO pid=53765) Then in your config, do `config['env'] = [name]`.
(PPO pid=53765) c) Make sure you provide a fully qualified classpath, e.g.:
(PPO pid=53765) `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(PPO pid=53765)
(PPO pid=53765) During handling of the above exception, another exception occurred:
(PPO pid=53765)
(PPO pid=53765) ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
(PPO pid=53765) File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 299, in __init__
(PPO pid=53765) super().__init__(config=config, logger_creator=logger_creator, **kwargs)
(PPO pid=53765) File "/Users/kai/coding/ray/python/ray/tune/trainable/trainable.py", line 157, in __init__
(PPO pid=53765) self.setup(copy.deepcopy(self.config))
(PPO pid=53765) File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 432, in setup
(PPO pid=53765) raise e.args[0].args[2]
(PPO pid=53765) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(PPO pid=53765) a) Not a supported/installed environment.
(PPO pid=53765) b) Not a tune-registered environment creator.
(PPO pid=53765) c) Not a valid env class string.
(PPO pid=53765)
(PPO pid=53765) Try one of the following:
(PPO pid=53765) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(PPO pid=53765) For VizDoom support: Install VizDoom
(PPO pid=53765) (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(PPO pid=53765) `pip install vizdoomgym`.
(PPO pid=53765) For PyBullet support: `pip install pybullet`.
(PPO pid=53765) b) To register your custom env, do `from ray import tune;
(PPO pid=53765) tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(PPO pid=53765) Then in your config, do `config['env'] = [name]`.
(PPO pid=53765) c) Make sure you provide a fully qualified classpath, e.g.:
(PPO pid=53765) `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53785) 2022-07-22 16:45:36,570 ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53785, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b6a39f90>)
(RolloutWorker pid=53785) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53785)
(RolloutWorker pid=53785) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53785)
(RolloutWorker pid=53785) ray::RolloutWorker.__init__() (pid=53785, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b6a39f90>)
(RolloutWorker pid=53785) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53785) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53785) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53785) return registry.make(id, **kwargs)
(RolloutWorker pid=53785) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53785) spec = self.spec(path)
(RolloutWorker pid=53785) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53785) id, matching_envs
(RolloutWorker pid=53785) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53785)
(RolloutWorker pid=53785) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53785)
(RolloutWorker pid=53785) ray::RolloutWorker.__init__() (pid=53785, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b6a39f90>)
(RolloutWorker pid=53785) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53785) self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53785) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53785) raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53785) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53785) a) Not a supported/installed environment.
(RolloutWorker pid=53785) b) Not a tune-registered environment creator.
(RolloutWorker pid=53785) c) Not a valid env class string.
(RolloutWorker pid=53785)
(RolloutWorker pid=53785) Try one of the following:
(RolloutWorker pid=53785) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53785) For VizDoom support: Install VizDoom
(RolloutWorker pid=53785) (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53785) `pip install vizdoomgym`.
(RolloutWorker pid=53785) For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53785) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53785) tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53785) Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53785) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53785) `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53787) 2022-07-22 16:45:36,570 ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53787, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b29b1990>)
(RolloutWorker pid=53787) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53787)
(RolloutWorker pid=53787) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53787)
(RolloutWorker pid=53787) ray::RolloutWorker.__init__() (pid=53787, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b29b1990>)
(RolloutWorker pid=53787) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53787) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53787) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53787) return registry.make(id, **kwargs)
(RolloutWorker pid=53787) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53787) spec = self.spec(path)
(RolloutWorker pid=53787) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53787) id, matching_envs
(RolloutWorker pid=53787) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53787)
(RolloutWorker pid=53787) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53787)
(RolloutWorker pid=53787) ray::RolloutWorker.__init__() (pid=53787, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b29b1990>)
(RolloutWorker pid=53787) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53787) self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53787) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53787) raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53787) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53787) a) Not a supported/installed environment.
(RolloutWorker pid=53787) b) Not a tune-registered environment creator.
(RolloutWorker pid=53787) c) Not a valid env class string.
(RolloutWorker pid=53787)
(RolloutWorker pid=53787) Try one of the following:
(RolloutWorker pid=53787) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53787) For VizDoom support: Install VizDoom
(RolloutWorker pid=53787) (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53787) `pip install vizdoomgym`.
(RolloutWorker pid=53787) For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53787) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53787) tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53787) Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53787) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53787) `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53791) 2022-07-22 16:45:36,570 ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53791, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb744490>)
(RolloutWorker pid=53791) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53791)
(RolloutWorker pid=53791) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53791)
(RolloutWorker pid=53791) ray::RolloutWorker.__init__() (pid=53791, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb744490>)
(RolloutWorker pid=53791) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53791) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53791) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53791) return registry.make(id, **kwargs)
(RolloutWorker pid=53791) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53791) spec = self.spec(path)
(RolloutWorker pid=53791) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53791) id, matching_envs
(RolloutWorker pid=53791) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53791)
(RolloutWorker pid=53791) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53791)
(RolloutWorker pid=53791) ray::RolloutWorker.__init__() (pid=53791, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb744490>)
(RolloutWorker pid=53791) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53791) self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53791) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53791) raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53791) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53791) a) Not a supported/installed environment.
(RolloutWorker pid=53791) b) Not a tune-registered environment creator.
(RolloutWorker pid=53791) c) Not a valid env class string.
(RolloutWorker pid=53791)
(RolloutWorker pid=53791) Try one of the following:
(RolloutWorker pid=53791) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53791) For VizDoom support: Install VizDoom
(RolloutWorker pid=53791) (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53791) `pip install vizdoomgym`.
(RolloutWorker pid=53791) For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53791) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53791) tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53791) Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53791) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53791) `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53788) 2022-07-22 16:45:36,570 ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53788, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb4f53d0>)
(RolloutWorker pid=53788) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53788)
(RolloutWorker pid=53788) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53788)
(RolloutWorker pid=53788) ray::RolloutWorker.__init__() (pid=53788, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb4f53d0>)
(RolloutWorker pid=53788) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53788) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53788) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53788) return registry.make(id, **kwargs)
(RolloutWorker pid=53788) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53788) spec = self.spec(path)
(RolloutWorker pid=53788) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53788) id, matching_envs
(RolloutWorker pid=53788) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53788)
(RolloutWorker pid=53788) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53788)
(RolloutWorker pid=53788) ray::RolloutWorker.__init__() (pid=53788, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1bb4f53d0>)
(RolloutWorker pid=53788) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53788) self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53788) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53788) raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53788) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53788) a) Not a supported/installed environment.
(RolloutWorker pid=53788) b) Not a tune-registered environment creator.
(RolloutWorker pid=53788) c) Not a valid env class string.
(RolloutWorker pid=53788)
(RolloutWorker pid=53788) Try one of the following:
(RolloutWorker pid=53788) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53788) For VizDoom support: Install VizDoom
(RolloutWorker pid=53788) (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53788) `pip install vizdoomgym`.
(RolloutWorker pid=53788) For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53788) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53788) tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53788) Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53788) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53788) `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53792) 2022-07-22 16:45:36,570 ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53792, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x117d54a50>)
(RolloutWorker pid=53792) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53792)
(RolloutWorker pid=53792) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53792)
(RolloutWorker pid=53792) ray::RolloutWorker.__init__() (pid=53792, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x117d54a50>)
(RolloutWorker pid=53792) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53792) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53792) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53792) return registry.make(id, **kwargs)
(RolloutWorker pid=53792) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53792) spec = self.spec(path)
(RolloutWorker pid=53792) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53792) id, matching_envs
(RolloutWorker pid=53792) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53792)
(RolloutWorker pid=53792) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53792)
(RolloutWorker pid=53792) ray::RolloutWorker.__init__() (pid=53792, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x117d54a50>)
(RolloutWorker pid=53792) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53792) self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53792) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53792) raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53792) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53792) a) Not a supported/installed environment.
(RolloutWorker pid=53792) b) Not a tune-registered environment creator.
(RolloutWorker pid=53792) c) Not a valid env class string.
(RolloutWorker pid=53792)
(RolloutWorker pid=53792) Try one of the following:
(RolloutWorker pid=53792) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53792) For VizDoom support: Install VizDoom
(RolloutWorker pid=53792) (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53792) `pip install vizdoomgym`.
(RolloutWorker pid=53792) For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53792) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53792) tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53792) Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53792) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53792) `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53786) 2022-07-22 16:45:36,570 ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(RolloutWorker pid=53786) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53786)
(RolloutWorker pid=53786) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53786)
(RolloutWorker pid=53786) ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(RolloutWorker pid=53786) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53786) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53786) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53786) return registry.make(id, **kwargs)
(RolloutWorker pid=53786) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53786) spec = self.spec(path)
(RolloutWorker pid=53786) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53786) id, matching_envs
(RolloutWorker pid=53786) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53786)
(RolloutWorker pid=53786) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53786)
(RolloutWorker pid=53786) ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
(RolloutWorker pid=53786) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53786) self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53786) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53786) raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53786) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53786) a) Not a supported/installed environment.
(RolloutWorker pid=53786) b) Not a tune-registered environment creator.
(RolloutWorker pid=53786) c) Not a valid env class string.
(RolloutWorker pid=53786)
(RolloutWorker pid=53786) Try one of the following:
(RolloutWorker pid=53786) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53786) For VizDoom support: Install VizDoom
(RolloutWorker pid=53786) (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53786) `pip install vizdoomgym`.
(RolloutWorker pid=53786) For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53786) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53786) tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53786) Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53786) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53786) `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53790) 2022-07-22 16:45:36,570 ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53790, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b35ba250>)
(RolloutWorker pid=53790) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53790)
(RolloutWorker pid=53790) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53790)
(RolloutWorker pid=53790) ray::RolloutWorker.__init__() (pid=53790, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b35ba250>)
(RolloutWorker pid=53790) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53790) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53790) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53790) return registry.make(id, **kwargs)
(RolloutWorker pid=53790) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53790) spec = self.spec(path)
(RolloutWorker pid=53790) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53790) id, matching_envs
(RolloutWorker pid=53790) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53790)
(RolloutWorker pid=53790) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53790)
(RolloutWorker pid=53790) ray::RolloutWorker.__init__() (pid=53790, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1b35ba250>)
(RolloutWorker pid=53790) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53790) self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53790) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53790) raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53790) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53790) a) Not a supported/installed environment.
(RolloutWorker pid=53790) b) Not a tune-registered environment creator.
(RolloutWorker pid=53790) c) Not a valid env class string.
(RolloutWorker pid=53790)
(RolloutWorker pid=53790) Try one of the following:
(RolloutWorker pid=53790) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53790) For VizDoom support: Install VizDoom
(RolloutWorker pid=53790) (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53790) `pip install vizdoomgym`.
(RolloutWorker pid=53790) For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53790) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53790) tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53790) Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53790) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53790) `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
(RolloutWorker pid=53789) 2022-07-22 16:45:36,570 ERROR worker.py:754 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53789, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1be518cd0>)
(RolloutWorker pid=53789) KeyError: 'Humanoid-v1'
(RolloutWorker pid=53789)
(RolloutWorker pid=53789) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53789)
(RolloutWorker pid=53789) ray::RolloutWorker.__init__() (pid=53789, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1be518cd0>)
(RolloutWorker pid=53789) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=53789) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=53789) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
(RolloutWorker pid=53789) return registry.make(id, **kwargs)
(RolloutWorker pid=53789) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
(RolloutWorker pid=53789) spec = self.spec(path)
(RolloutWorker pid=53789) File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
(RolloutWorker pid=53789) id, matching_envs
(RolloutWorker pid=53789) gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
(RolloutWorker pid=53789)
(RolloutWorker pid=53789) During handling of the above exception, another exception occurred:
(RolloutWorker pid=53789)
(RolloutWorker pid=53789) ray::RolloutWorker.__init__() (pid=53789, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1be518cd0>)
(RolloutWorker pid=53789) File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=53789) self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=53789) File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
(RolloutWorker pid=53789) raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
(RolloutWorker pid=53789) ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
(RolloutWorker pid=53789) a) Not a supported/installed environment.
(RolloutWorker pid=53789) b) Not a tune-registered environment creator.
(RolloutWorker pid=53789) c) Not a valid env class string.
(RolloutWorker pid=53789)
(RolloutWorker pid=53789) Try one of the following:
(RolloutWorker pid=53789) a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
(RolloutWorker pid=53789) For VizDoom support: Install VizDoom
(RolloutWorker pid=53789) (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
(RolloutWorker pid=53789) `pip install vizdoomgym`.
(RolloutWorker pid=53789) For PyBullet support: `pip install pybullet`.
(RolloutWorker pid=53789) b) To register your custom env, do `from ray import tune;
(RolloutWorker pid=53789) tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
(RolloutWorker pid=53789) Then in your config, do `config['env'] = [name]`.
(RolloutWorker pid=53789) c) Make sure you provide a fully qualified classpath, e.g.:
(RolloutWorker pid=53789) `ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
2022-07-22 16:45:36,660 ERROR trial_runner.py:921 -- Trial PPO_Humanoid-v1_45196_00000: Error processing event.
ray.tune.error._TuneNoNextExecutorEventError: Traceback (most recent call last):
File "/Users/kai/coding/ray/python/ray/tune/execution/ray_trial_executor.py", line 989, in get_next_executor_event
future_result = ray.get(ready_future)
File "/Users/kai/coding/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/Users/kai/coding/ray/python/ray/_private/worker.py", line 2199, in get
raise value
File "python/ray/_raylet.pyx", line 812, in ray._raylet.task_execution_handler
File "python/ray/_raylet.pyx", line 623, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 772, in ray._raylet.execute_task
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 127, in __init__
validate=trainer_config.get("validate_workers_after_construction"),
File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 269, in add_workers
self.foreach_worker(lambda w: w.assert_healthy())
File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 391, in foreach_worker
remote_results = ray.get([w.apply.remote(func) for w in self.remote_workers()])
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
KeyError: 'Humanoid-v1'
During handling of the above exception, another exception occurred:
ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
return gym.make(env_descriptor, **env_context)
File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
return registry.make(id, **kwargs)
File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
spec = self.spec(path)
File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
id, matching_envs
gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
During handling of the above exception, another exception occurred:
ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
self.env = env_creator(copy.deepcopy(self.env_context))
File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
a) Not a supported/installed environment.
b) Not a tune-registered environment creator.
c) Not a valid env class string.
Try one of the following:
a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
For VizDoom support: Install VizDoom
(https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
`pip install vizdoomgym`.
For PyBullet support: `pip install pybullet`.
b) To register your custom env, do `from ray import tune;
tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
Then in your config, do `config['env'] = [name]`.
c) Make sure you provide a fully qualified classpath, e.g.:
`ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
During handling of the above exception, another exception occurred:
ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 299, in __init__
super().__init__(config=config, logger_creator=logger_creator, **kwargs)
File "/Users/kai/coding/ray/python/ray/tune/trainable/trainable.py", line 157, in __init__
self.setup(copy.deepcopy(self.config))
File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 432, in setup
raise e.args[0].args[2]
ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
a) Not a supported/installed environment.
b) Not a tune-registered environment creator.
c) Not a valid env class string.
Try one of the following:
a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
For VizDoom support: Install VizDoom
(https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
`pip install vizdoomgym`.
For PyBullet support: `pip install pybullet`.
b) To register your custom env, do `from ray import tune;
tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
Then in your config, do `config['env'] = [name]`.
c) Make sure you provide a fully qualified classpath, e.g.:
`ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
Result for PPO_Humanoid-v1_45196_00000:
trial_id: '45196_00000'
2022-07-22 16:45:36,688 ERROR ray_trial_executor.py:104 -- An exception occurred when trying to stop the Ray actor:Traceback (most recent call last):
File "/Users/kai/coding/ray/python/ray/tune/execution/ray_trial_executor.py", line 94, in post_stop_cleanup
ray.get(future, timeout=0)
File "/Users/kai/coding/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/Users/kai/coding/ray/python/ray/_private/worker.py", line 2199, in get
raise value
File "python/ray/_raylet.pyx", line 812, in ray._raylet.task_execution_handler
File "python/ray/_raylet.pyx", line 623, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 772, in ray._raylet.execute_task
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 127, in __init__
validate=trainer_config.get("validate_workers_after_construction"),
File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 269, in add_workers
self.foreach_worker(lambda w: w.assert_healthy())
File "/Users/kai/coding/ray/python/ray/rllib/evaluation/worker_set.py", line 391, in foreach_worker
remote_results = ray.get([w.apply.remote(func) for w in self.remote_workers()])
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
KeyError: 'Humanoid-v1'
During handling of the above exception, another exception occurred:
ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 50, in _gym_env_creator
return gym.make(env_descriptor, **env_context)
File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 235, in make
return registry.make(id, **kwargs)
File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 128, in make
spec = self.spec(path)
File "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/gym/envs/registration.py", line 187, in spec
id, matching_envs
gym.error.DeprecatedEnv: Env Humanoid-v1 not found (valid versions include ['Humanoid-v2', 'Humanoid-v3'])
During handling of the above exception, another exception occurred:
ray::RolloutWorker.__init__() (pid=53786, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x1193ddd10>)
File "/Users/kai/coding/ray/python/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
self.env = env_creator(copy.deepcopy(self.env_context))
File "/Users/kai/coding/ray/python/ray/rllib/env/utils.py", line 52, in _gym_env_creator
raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
a) Not a supported/installed environment.
b) Not a tune-registered environment creator.
c) Not a valid env class string.
Try one of the following:
a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
For VizDoom support: Install VizDoom
(https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
`pip install vizdoomgym`.
For PyBullet support: `pip install pybullet`.
b) To register your custom env, do `from ray import tune;
tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
Then in your config, do `config['env'] = [name]`.
c) Make sure you provide a fully qualified classpath, e.g.:
`ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
During handling of the above exception, another exception occurred:
ray::PPO.__init__() (pid=53765, ip=127.0.0.1, repr=PPO)
File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 299, in __init__
super().__init__(config=config, logger_creator=logger_creator, **kwargs)
File "/Users/kai/coding/ray/python/ray/tune/trainable/trainable.py", line 157, in __init__
self.setup(copy.deepcopy(self.config))
File "/Users/kai/coding/ray/python/ray/rllib/algorithms/algorithm.py", line 432, in setup
raise e.args[0].args[2]
ray.rllib.utils.error.EnvError: The env string you provided ('Humanoid-v1') is:
a) Not a supported/installed environment.
b) Not a tune-registered environment creator.
c) Not a valid env class string.
Try one of the following:
a) For Atari support: `pip install gym[atari] autorom[accept-rom-license]`.
For VizDoom support: Install VizDoom
(https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
`pip install vizdoomgym`.
For PyBullet support: `pip install pybullet`.
b) To register your custom env, do `from ray import tune;
tune.register('[name]', lambda cfg: [return env obj from here using cfg])`.
Then in your config, do `config['env'] = [name]`.
c) Make sure you provide a fully qualified classpath, e.g.:
`ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv`
2022-07-22 16:45:36,795 ERROR tune.py:733 -- Trials did not complete: [PPO_Humanoid-v1_45196_00000]
2022-07-22 16:45:36,795 INFO tune.py:738 -- Total run time: 25.77 seconds (25.48 seconds for the tuning loop).
best hyperparameters: None
More RLlib Examples#
PB2 PPO Example: Example of optimizing a distributed RLlib algorithm (PPO) with the PB2 scheduler. Uses a small population size of 4, so can train on a laptop.