How to write code snippets#
Users learn from example. So, whether you’re writing a docstring or a user guide, include examples that illustrate the relevant APIs. Your examples should run out-of-the-box so that users can copy them and adapt them to their own needs.
This page describes how to write code snippets so that they’re tested in CI, and how to show code that CI can’t run.
Note
The examples in this guide use reStructuredText. If you’re writing Markdown, use MyST syntax. To learn more, read the MyST documentation.
Types of examples#
There are three types of examples: doctest-style, code-output-style, and literalinclude. Use doctest-style in docstrings, and the other two on documentation pages.
doctest-style examples#
doctest-style examples mimic interactive Python sessions. Write them in docstrings, under an Example: section:
Example:
>>> def is_even(x):
... return (x % 2) == 0
>>> is_even(0)
True
>>> is_even(1)
False
They’re rendered like this:
>>> def is_even(x):
... return (x % 2) == 0
>>> is_even(0)
True
>>> is_even(1)
False
Note
Don’t use doctest-style examples on documentation pages. On a page, use code-output-style or literalinclude instead.
code-output-style examples#
code-output-style examples contain ordinary Python code.
.. testcode::
def is_even(x):
return (x % 2) == 0
print(is_even(0))
print(is_even(1))
.. testoutput::
True
False
They’re rendered like this:
def is_even(x):
return (x % 2) == 0
print(is_even(0))
print(is_even(1))
True
False
literalinclude examples#
literalinclude examples display Python modules.
.. literalinclude:: ./doc_code/example_module.py
:language: python
:start-after: __is_even_begin__
:end-before: __is_even_end__
# example_module.py
# fmt: off
# __is_even_begin__
def is_even(x):
return (x % 2) == 0
# __is_even_end__
# fmt: on
They’re rendered like this:
def is_even(x):
return (x % 2) == 0
Which type of example should you write?#
In a docstring, use doctest-style. On a documentation page, choose between code-output-style and literalinclude, whichever best illustrates your API.
Tip
If you’re not sure which style to use on a documentation page, use code-output-style.
When to use doctest-style#
Use doctest-style in docstrings. It suits small examples that emphasize object representations or print intermediate objects:
Example:
>>> import ray
>>> ds = ray.data.range(100)
>>> ds.schema()
Column Type
------ ----
id int64
>>> ds.take(5)
[{'id': 0}, {'id': 1}, {'id': 2}, {'id': 3}, {'id': 4}]
When to use code-output-style#
On a documentation page, use code-output-style to show the code and its expected output, including object representations.
.. testcode::
from typing import Dict
import numpy as np
import ray
ds = ray.data.read_csv("s3://anonymous@air-example-data/iris.csv")
# Compute a "petal area" attribute.
def transform_batch(batch: Dict[str, np.ndarray]) -> Dict[str, np.ndarray]:
vec_a = batch["petal length (cm)"]
vec_b = batch["petal width (cm)"]
batch["petal area (cm^2)"] = np.round(vec_a * vec_b, 2)
return batch
transformed_ds = ds.map_batches(transform_batch)
print(transformed_ds.materialize())
.. testoutput::
shape: (150, 6)
╭───────────────────┬──────────────────┬───────────────────┬──────────────────┬────────┬───────────────────╮
│ sepal length (cm) ┆ sepal width (cm) ┆ petal length (cm) ┆ petal width (cm) ┆ target ┆ petal area (cm^2) │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ double ┆ double ┆ double ┆ double ┆ int64 ┆ double │
╞═══════════════════╪══════════════════╪═══════════════════╪══════════════════╪════════╪═══════════════════╡
│ 5.1 ┆ 3.5 ┆ 1.4 ┆ 0.2 ┆ 0 ┆ 0.28 │
│ 4.9 ┆ 3.0 ┆ 1.4 ┆ 0.2 ┆ 0 ┆ 0.28 │
│ 4.7 ┆ 3.2 ┆ 1.3 ┆ 0.2 ┆ 0 ┆ 0.26 │
│ 4.6 ┆ 3.1 ┆ 1.5 ┆ 0.2 ┆ 0 ┆ 0.3 │
│ 5.0 ┆ 3.6 ┆ 1.4 ┆ 0.2 ┆ 0 ┆ 0.28 │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … │
│ 6.7 ┆ 3.0 ┆ 5.2 ┆ 2.3 ┆ 2 ┆ 11.96 │
│ 6.3 ┆ 2.5 ┆ 5.0 ┆ 1.9 ┆ 2 ┆ 9.5 │
│ 6.5 ┆ 3.0 ┆ 5.2 ┆ 2.0 ┆ 2 ┆ 10.4 │
│ 6.2 ┆ 3.4 ┆ 5.4 ┆ 2.3 ┆ 2 ┆ 12.42 │
│ 5.9 ┆ 3.0 ┆ 5.1 ┆ 1.8 ┆ 2 ┆ 9.18 │
╰───────────────────┴──────────────────┴───────────────────┴──────────────────┴────────┴───────────────────╯
(Showing 10 of 150 rows)
When to use literalinclude#
If you’re writing a longer example that doesn’t show outputs, use literalinclude. For a full guide that CI doesn’t test, see Writing an end-to-end guide.
How to handle hard-to-test examples#
When is it okay to not test an example?#
You don’t need to test examples that depend on external systems such as Weights and Biases.
A testcode block tells readers and maintainers that CI runs the code. Use it only for blocks that CI executes. Show any code that CI doesn’t run as a plain code block.
Skipping doctest-style examples#
To skip a doctest-style example, append # doctest: +SKIP to your Python code.
Example:
>>> import ray
>>> ray.data.read_images("s3://private-bucket") # doctest: +SKIP
Showing code-output-style examples that CI doesn’t run#
To show a code-output-style example that CI doesn’t run, write a plain code-block instead of a testcode block. Don’t add :skipif: True to a testcode block. The block never runs, but it reads as tested code.
.. code-block:: python
from ray.air.integrations.wandb import WandbLoggerCallback
callback = WandbLoggerCallback(
project="Optimization_Project",
api_key_file=...,
log_config=True
)
In MyST Markdown, use a fenced block with a language, such as ```python. To show sample output, add a second plain block with the text language in place of a testoutput block.
How to handle long or non-deterministic outputs#
If your Python code is non-deterministic, or if your output is excessively long, you can skip all or part of the output.
Ignoring doctest-style outputs#
To ignore parts of a doctest-style output, replace problematic sections with ellipses.
>>> import ray
>>> ray.data.read_images("s3://anonymous@ray-example-data/image-datasets/simple")
Dataset(num_rows=..., schema=...)
To ignore an output altogether, write a code-output-style snippet. Don’t use # doctest: +SKIP.
Ignoring code-output-style outputs#
If parts of your output are long or non-deterministic, replace problematic sections with ellipses.
.. testcode::
import ray
ds = ray.data.read_images("s3://anonymous@ray-example-data/image-datasets/simple")
print(ds)
.. testoutput::
Dataset(num_rows=..., schema=...)
If your output is non-deterministic and you want to display a sample output, add :options: +MOCK.
.. testcode::
import random
print(random.random())
.. testoutput::
:options: +MOCK
0.969461416250246
If your output is hard to test and you don’t want to display a sample output, exclude the testoutput.
.. testcode::
print("This output is hidden and untested")
How to add a page’s examples to CI#
CI tests a documentation page’s doctest and testcode examples only if doc/BUILD.bazel names the page. When you add the first runnable example to a page, add the page to its library’s doctest_each list in doc/BUILD.bazel, or to the team:none list if no library owns it. The doc_prose_doctest_targets lint fails a PR that adds a runnable example to a page that no target names, and a PR that leaves a page with no runnable example in a list.
How to test examples with GPUs#
Bazel doctest rules in BUILD.bazel files decide which files CI tests, and a rule with gpu = True runs its files on GPUs. To run an example with GPUs, move its file from the CPU rule to a GPU rule.
For a documentation page under doc/source/, complete the following steps:
Open
doc/BUILD.bazeland find thedoctest_eachrule for the page’s library. It lists each page with a runnable example and gives every page its own test target.Remove the page from that list.
Add the page to the library’s GPU
doctestrule. If the library doesn’t have one, create it.doctest( name = "doctest[train-gpu]", files = [ "source/train/user-guides/using-accelerators.md", ], gpu = True, tags = ["team:ml"], )
For a docstring in a Python module, complete the following steps:
Open the library’s
BUILD.bazelfile, such aspython/ray/train/BUILD.bazel.Add the module to the
excludelist of the library’s CPUdoctestrule glob.Add the module to the library’s GPU
doctestrule. If the library doesn’t have one, create it.doctest( name = "py_doctest[train-gpu]", files = [ "tensorflow/tensorflow_trainer.py", ], gpu = True, tags = ["team:ml"], )
For practical examples, see doc/BUILD.bazel and python/ray/train/BUILD.bazel.
How to locally test examples#
To locally test examples, install the Ray fork of pytest-sphinx.
pip install git+https://github.com/ray-project/pytest-sphinx
Then, run pytest on a module, docstring, or user guide.
pytest --doctest-modules python/ray/data/read_api.py
pytest --doctest-modules python/ray/data/read_api.py::ray.data.read_api.range
pytest --doctest-modules doc/source/data/getting-started.rst
How to debug a failing example#
When a code snippet fails in CI, two questions decide what you need to do about it: what kind of failure is it, and what was the example protecting. To find which step failed, see Per-library docs example tests, which maps each documentation path to the step that runs its examples.
What kind of failure is it?#
The example ran, but its output didn’t match. This is an ordinary test failure. Reproduce it locally with
pytest --doctest-modules <file>and compare the actual output against the expectedtestoutputor>>>block. Either the code changed or the expected output is wrong.The build failed before your example ran. An import error or a
conf.pyerror can abort the build, and the rest of the log is unreliable after an abort. Fix that error first, then re-read the log.A Sphinx warning failed the build. The doc site host (Read the Docs) render gate treats warnings as errors, so a malformed directive or a broken cross-reference fails the build even when your code is correct. This is a markup problem, not a code failure. See the Read the Docs render gate.
What was the example protecting?#
A failing example is a signal, but the right response depends on what the example was there to catch. There are three cases:
A breaking-change detector. The snippet is user-facing code that broke because the library’s behavior changed. Treat the failure as a real signal: either don’t land the change, or land it with breaking-change communication. Once the change is approved, do update the example to match the new behavior, so the page stays correct for the version that ships. What’s wrong is updating it instead of communicating the break, because that hides the change from the users who copied the old example.
An example validator. The example itself is wrong, such as a typo, a bad merge, or a stale import, and just needs to run. Fix the example. Don’t mistake it for a code regression.
A drift indicator. The example still runs, but the prose around it has gone stale relative to what the code now does. Update the narrative, not just the example.
The same example can fail different ways at different times. An IndentationError introduced by a bad merge is an example-validator failure, so fix the snippet. An ImportError from an upstream API change to that same example is a breaking-change signal, so fix the code or communicate the break. The failure text tells you which case you’re in.
Let the intent guide whether to skip. Skipping with # doctest: +SKIP, or moving the example to a plain code block, is right for an example that depends on an external system, but it’s the wrong response to a breaking-change detector. Skipping there hides a real break from users.
Doc edits don’t run example tests by default, so a green PR isn’t evidence that a changed example still runs. When you change an example, add the docs-example-test label to run its tests before you merge.