{
"cells": [
{
"cell_type": "markdown",
"id": "f867a908",
"metadata": {},
"source": [
"# Learning to Play Pong\n",
"\n",
"```{tip}\n",
"For a production-grade implementation of distributed\n",
"reinforcement learning, use [Ray RLlib](https://docs.ray.io/en/master/rllib/index.html).\n",
"```\n",
"\n",
"In this example, we'll train a **very simple** neural network to play Pong using\n",
"Gymnasium.\n",
"\n",
"At a high level, we will use multiple Ray actors to obtain simulation rollouts\n",
"and calculate gradient simultaneously. We will then centralize these\n",
"gradients and update the neural network. The updated neural network will\n",
"then be passed back to each Ray actor for more gradient calculation.\n",
"\n",
"This application is adapted, with minimal modifications, from\n",
"Andrej Karpathy's [source code](https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5)\n",
"(see the accompanying [blog post](http://karpathy.github.io/2016/05/31/rl/)).\n",
"\n",
"```{image} /ray-core/images/pong-arch.svg\n",
":align: center\n",
"```\n",
"\n",
"To run the application, first install some dependencies.\n",
"\n",
"```bash\n",
"pip install gymnasium[atari] gym==0.26.2\n",
"```\n",
"\n",
"At the moment, on a large machine with 64 physical cores, computing an update\n",
"with a batch of size 1 takes about 1 second, a batch of size 10 takes about 2.5\n",
"seconds. A batch of size 60 takes about 3 seconds. On a cluster with 11 nodes,\n",
"each with 18 physical cores, a batch of size 300 takes about 10 seconds. If the\n",
"numbers you see differ from these by much, take a look at the\n",
"**Troubleshooting** section at the bottom of this page and consider\n",
"[submitting an issue](https://github.com/ray-project/ray/issues).\n",
"\n",
"**Note** that these times depend on how long the rollouts take, which in turn\n",
"depends on how well the policy is doing. For example, a really bad policy will\n",
"lose very quickly. As the policy learns, we should expect these numbers to\n",
"increase."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "549e3475",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import os\n",
"import ray\n",
"import time\n",
"\n",
"import gymnasium as gym"
]
},
{
"cell_type": "markdown",
"id": "39e69bfd",
"metadata": {},
"source": [
"## Hyperparameters\n",
"\n",
"Here we'll define a couple of the hyperparameters that are used."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9cb838df",
"metadata": {},
"outputs": [],
"source": [
"H = 200 # The number of hidden layer neurons.\n",
"gamma = 0.99 # The discount factor for reward.\n",
"decay_rate = 0.99 # The decay factor for RMSProp leaky sum of grad^2.\n",
"D = 80 * 80 # The input dimensionality: 80x80 grid.\n",
"learning_rate = 1e-4 # Magnitude of the update."
]
},
{
"cell_type": "markdown",
"id": "a0efd0b2",
"metadata": {},
"source": [
"## Helper Functions\n",
"\n",
"We first define a few helper functions:\n",
"\n",
"1. Preprocessing: The ``preprocess`` function will\n",
"preprocess the original 210x160x3 uint8 frame into a one-dimensional 6400\n",
"float vector.\n",
"\n",
"2. Reward Processing: The ``process_rewards`` function will calculate\n",
"a discounted reward. This formula states that the \"value\" of a\n",
"sampled action is the weighted sum of all rewards afterwards,\n",
"but later rewards are exponentially less important.\n",
"\n",
"3. Rollout: The ``rollout`` function plays an entire game of Pong (until\n",
"either the computer or the RL agent loses)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d20fd47c",
"metadata": {},
"outputs": [],
"source": [
"def preprocess(img):\n",
" # Crop the image.\n",
" img = img[35:195]\n",
" # Downsample by factor of 2.\n",
" img = img[::2, ::2, 0]\n",
" # Erase background (background type 1).\n",
" img[img == 144] = 0\n",
" # Erase background (background type 2).\n",
" img[img == 109] = 0\n",
" # Set everything else (paddles, ball) to 1.\n",
" img[img != 0] = 1\n",
" return img.astype(np.float).ravel()\n",
"\n",
"\n",
"def process_rewards(r):\n",
" \"\"\"Compute discounted reward from a vector of rewards.\"\"\"\n",
" discounted_r = np.zeros_like(r)\n",
" running_add = 0\n",
" for t in reversed(range(0, r.size)):\n",
" # Reset the sum, since this was a game boundary (pong specific!).\n",
" if r[t] != 0:\n",
" running_add = 0\n",
" running_add = running_add * gamma + r[t]\n",
" discounted_r[t] = running_add\n",
" return discounted_r\n",
"\n",
"\n",
"def rollout(model, env):\n",
" \"\"\"Evaluates env and model until the env returns \"Terminated\" or \"Truncated\".\n",
"\n",
" Returns:\n",
" xs: A list of observations\n",
" hs: A list of model hidden states per observation\n",
" dlogps: A list of gradients\n",
" drs: A list of rewards.\n",
"\n",
" \"\"\"\n",
" # Reset the game.\n",
" observation, info = env.reset()\n",
" # Note that prev_x is used in computing the difference frame.\n",
" prev_x = None\n",
" xs, hs, dlogps, drs = [], [], [], []\n",
" terminated = truncated = False\n",
" while not terminated and not truncated:\n",
" cur_x = preprocess(observation)\n",
" x = cur_x - prev_x if prev_x is not None else np.zeros(D)\n",
" prev_x = cur_x\n",
"\n",
" aprob, h = model.policy_forward(x)\n",
" # Sample an action.\n",
" action = 2 if np.random.uniform() < aprob else 3\n",
"\n",
" # The observation.\n",
" xs.append(x)\n",
" # The hidden state.\n",
" hs.append(h)\n",
" y = 1 if action == 2 else 0 # A \"fake label\".\n",
" # The gradient that encourages the action that was taken to be\n",
" # taken (see http://cs231n.github.io/neural-networks-2/#losses if\n",
" # confused).\n",
" dlogps.append(y - aprob)\n",
"\n",
" observation, reward, terminated, truncated, info = env.step(action)\n",
"\n",
" # Record reward (has to be done after we call step() to get reward\n",
" # for previous action).\n",
" drs.append(reward)\n",
" return xs, hs, dlogps, drs"
]
},
{
"cell_type": "markdown",
"id": "7c00c00a",
"metadata": {},
"source": [
"## Neural Network\n",
"\n",
"Here, a neural network is used to define a \"policy\"\n",
"for playing Pong (that is, a function that chooses an action given a state).\n",
"\n",
"To implement a neural network in NumPy, we need to provide helper functions\n",
"for calculating updates and computing the output of the neural network\n",
"given an input, which in our case is an observation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8992067a",
"metadata": {},
"outputs": [],
"source": [
"class Model(object):\n",
" \"\"\"This class holds the neural network weights.\"\"\"\n",
"\n",
" def __init__(self):\n",
" self.weights = {}\n",
" self.weights[\"W1\"] = np.random.randn(H, D) / np.sqrt(D)\n",
" self.weights[\"W2\"] = np.random.randn(H) / np.sqrt(H)\n",
"\n",
" def policy_forward(self, x):\n",
" h = np.dot(self.weights[\"W1\"], x)\n",
" h[h < 0] = 0 # ReLU nonlinearity.\n",
" logp = np.dot(self.weights[\"W2\"], h)\n",
" # Softmax\n",
" p = 1.0 / (1.0 + np.exp(-logp))\n",
" # Return probability of taking action 2, and hidden state.\n",
" return p, h\n",
"\n",
" def policy_backward(self, eph, epx, epdlogp):\n",
" \"\"\"Backward pass to calculate gradients.\n",
"\n",
" Arguments:\n",
" eph: Array of intermediate hidden states.\n",
" epx: Array of experiences (observations).\n",
" epdlogp: Array of logps (output of last layer before softmax).\n",
"\n",
" \"\"\"\n",
" dW2 = np.dot(eph.T, epdlogp).ravel()\n",
" dh = np.outer(epdlogp, self.weights[\"W2\"])\n",
" # Backprop relu.\n",
" dh[eph <= 0] = 0\n",
" dW1 = np.dot(dh.T, epx)\n",
" return {\"W1\": dW1, \"W2\": dW2}\n",
"\n",
" def update(self, grad_buffer, rmsprop_cache, lr, decay):\n",
" \"\"\"Applies the gradients to the model parameters with RMSProp.\"\"\"\n",
" for k, v in self.weights.items():\n",
" g = grad_buffer[k]\n",
" rmsprop_cache[k] = decay * rmsprop_cache[k] + (1 - decay) * g ** 2\n",
" self.weights[k] += lr * g / (np.sqrt(rmsprop_cache[k]) + 1e-5)\n",
"\n",
"\n",
"def zero_grads(grad_buffer):\n",
" \"\"\"Reset the batch gradient buffer.\"\"\"\n",
" for k, v in grad_buffer.items():\n",
" grad_buffer[k] = np.zeros_like(v)"
]
},
{
"cell_type": "markdown",
"id": "c4a847bd",
"metadata": {},
"source": [
"## Parallelizing Gradients\n",
"\n",
"We define an **actor**, which is responsible for taking a model and an env\n",
"and performing a rollout + computing a gradient update."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c95ee2f2",
"metadata": {},
"outputs": [],
"source": [
"# This forces OpenMP to use 1 single thread, which is needed to \n",
"# prevent contention between multiple actors. \n",
"# See https://docs.ray.io/en/latest/ray-core/configure.html for \n",
"# more details. \n",
"os.environ[\"OMP_NUM_THREADS\"] = \"1\"\n",
"# Tell numpy to only use one core. If we don't do this, each actor may\n",
"# try to use all of the cores and the resulting contention may result\n",
"# in no speedup over the serial version. Note that if numpy is using\n",
"# OpenBLAS, then you need to set OPENBLAS_NUM_THREADS=1, and you\n",
"# probably need to do it from the command line (so it happens before\n",
"# numpy is imported).\n",
"os.environ[\"MKL_NUM_THREADS\"] = \"1\"\n",
"\n",
"ray.init()\n",
"\n",
"\n",
"@ray.remote\n",
"class RolloutWorker(object):\n",
" def __init__(self):\n",
" self.env = gym.make(\"GymV26Environment-v0\", env_id=\"ALE/Pong-v5\")\n",
"\n",
" def compute_gradient(self, model):\n",
" # Compute a simulation episode.\n",
" xs, hs, dlogps, drs = rollout(model, self.env)\n",
" reward_sum = sum(drs)\n",
" # Vectorize the arrays.\n",
" epx = np.vstack(xs)\n",
" eph = np.vstack(hs)\n",
" epdlogp = np.vstack(dlogps)\n",
" epr = np.vstack(drs)\n",
"\n",
" # Compute the discounted reward backward through time.\n",
" discounted_epr = process_rewards(epr)\n",
" # Standardize the rewards to be unit normal (helps control the gradient\n",
" # estimator variance).\n",
" discounted_epr -= np.mean(discounted_epr)\n",
" discounted_epr /= np.std(discounted_epr)\n",
" # Modulate the gradient with advantage (the policy gradient magic\n",
" # happens right here).\n",
" epdlogp *= discounted_epr\n",
" return model.policy_backward(eph, epx, epdlogp), reward_sum"
]
},
{
"cell_type": "markdown",
"id": "1ce7f4da",
"metadata": {},
"source": [
"## Running\n",
"\n",
"This example is easy to parallelize because the network can play ten games\n",
"in parallel and no information needs to be shared between the games.\n",
"\n",
"In the loop, the network repeatedly plays games of Pong and\n",
"records a gradient from each game. Every ten games, the gradients are\n",
"combined together and used to update the network."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e353bd1e",
"metadata": {},
"outputs": [],
"source": [
"iterations = 20\n",
"batch_size = 4\n",
"model = Model()\n",
"actors = [RolloutWorker.remote() for _ in range(batch_size)]\n",
"\n",
"running_reward = None\n",
"# \"Xavier\" initialization.\n",
"# Update buffers that add up gradients over a batch.\n",
"grad_buffer = {k: np.zeros_like(v) for k, v in model.weights.items()}\n",
"# Update the rmsprop memory.\n",
"rmsprop_cache = {k: np.zeros_like(v) for k, v in model.weights.items()}\n",
"\n",
"for i in range(1, 1 + iterations):\n",
" model_id = ray.put(model)\n",
" gradient_ids = []\n",
" # Launch tasks to compute gradients from multiple rollouts in parallel.\n",
" start_time = time.time()\n",
" gradient_ids = [actor.compute_gradient.remote(model_id) for actor in actors]\n",
" for batch in range(batch_size):\n",
" [grad_id], gradient_ids = ray.wait(gradient_ids)\n",
" grad, reward_sum = ray.get(grad_id)\n",
" # Accumulate the gradient over batch.\n",
" for k in model.weights:\n",
" grad_buffer[k] += grad[k]\n",
" running_reward = (\n",
" reward_sum\n",
" if running_reward is None\n",
" else running_reward * 0.99 + reward_sum * 0.01\n",
" )\n",
" end_time = time.time()\n",
" print(\n",
" \"Batch {} computed {} rollouts in {} seconds, \"\n",
" \"running mean is {}\".format(\n",
" i, batch_size, end_time - start_time, running_reward\n",
" )\n",
" )\n",
" model.update(grad_buffer, rmsprop_cache, learning_rate, decay_rate)\n",
" zero_grads(grad_buffer)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}