Writing Unit Tested Code Examples¶

Useful technical documentation contains code examples. Unfortunately, those examples often:

Produce incorrect results
Don’t compile

The ailments above result from many causes, but boil down to one of the following:

The developer fails to test the code example
The code example becomes out-of-sync with the “real” code

For the user, bad examples lead to distrust of not only the documentation, but the overall product.

One of the guiding principles of the Omniverse Documentation System is to make sure code examples are valid. To do this, we present the following method to use your favorite testing framework to ensure your examples both compile and produce the correct result.

Testing Overview¶

The basic idea for unit testing code examples in documentation is fairly simple:

Write the code example as part of a source file that is built and executed in a test.
Add markers in the source file to delineate which parts of the file should be included/extracted as code examples in your documentation.
Use rST’s .. literalinclude:: to “include” the code snippet into your documentation.
Ensure your CI/CD system builds, executes, and checks the result of the test.

The main benefit of this approach is that you’re dynamically referencing “living” code; code that lives as a part of a test. If the test fails or does not compile, your CI/CD system will flag it as such and you’ll be obligated to fix it.

You may encounter a situation where you want to add a code example to your documentation that does not already exists in a test. In such a situation, simply create a new test that does contain the snippet of code you want to include in your documentation. Said differently, treat documentation code examples as first-class citizens in your project. They’re worthy of creating explicit, dedicated tests.

With that said, the code snippets don’t have to come from tests. They can come from SDK code, examples, or even other parts of documentation. The important concept is that the code “included” in the documentation is eventually executed, directly or indirectly, as a part of a test that your CI/CD system can check.

This approach works for not only for source code like C++, Python, MarkDown, reStructuredText, etc.; but also configuration files like TOML and JSON. For real-world examples, search the repo_docs repository for .. literalinclude::.

Testing Example¶

Consider the following code example:

# the swell module is a friendly module. it's an all-around nice module.
# it's swell. to access it, first import it.
import swell

# the magic of the swell module is that it can create a friendly,
# customized message for your favorite person.
customMsg = swell.greet("nathan")

# the resulting message can be printed with print()
print(customMsg)

The test above isn’t a part of this .rst file. Rather, the code snippet above has been “included” as follows via the .. literalinclude:: statement:

.. literalinclude:: ../source/tests/unittest/repo_docs/doc_snippets.py
   :language: python
   :start-after: example-begin unit-tested-example
   :end-before: example-end unit-tested-example
   :dedent:

Above, the .. literalinclude:: file has the following arguments:

file

The file from which to include. This path is relative to the current file.

:language:

A hint to the syntax highlighting engine.

:start-after: / :end-before:

Defines a simple pattern for being/end delimiters. Any text after the :start-after: pattern, up to text matching :end-before:, is included as a part of the snippet.

Here, we’re searching for example-begin unit-tested-example and example-end unit-tested-example.

There’s nothing special about the example-begin / example-end prefix. It’s just a simple pattern. Choose your own prefix, if any, at will.

There are more options to literalinlcude, all of which are covered at its official docs.

Now that we’ve seen the resulting example, let’s look at the actual test file:

import unittest


class ExampleSnippetDocsTestCase(unittest.TestCase):
    def test_swell(self):
        # example-begin unit-tested-example
        # the swell module is a friendly module. it's an all-around nice module.
        # it's swell. to access it, first import it.
        import swell

        # the magic of the swell module is that it can create a friendly,
        # customized message for your favorite person.
        customMsg = swell.greet("nathan")

        # the resulting message can be printed with print()
        print(customMsg)
        # example-end unit-tested-example

        # example-begin unit-tested-example-magic
        assert customMsg == "hi nathan"
        # example-end unit-tested-example-magic

Above, you can see the example-begin unit-tested-example and example-end unit-tested-example markers in the test file as comments. Anything between those markers will be included by the .. literalinclude:: statement above.

The magic is what’s outside of the markers. In particular, the following code which ensures the example produces the desired result:

        assert customMsg == "hi nathan"

If the assert above fails, we know something is busted (either our code or the documentation).

Additionally, if we decide to rename the swell module, or modify the required arguments to the greet method, the test will fail to compile/execute; telling us our documentation is out-of-date.

Running Tests¶

repo_docs does not prescribe which testing framework you should use to run tests. It’s up to you. What’s important is that your documentation code examples are included in one of your tests.

As an example, repo_docs uses the testing philosophy outlined in this document to ensure its examples are valid. In particular, repo_docs uses repo_test to find and run tests in the repo. As a part of repo_docs’ CI/CD (via GitHub), if any test fails, the submitted merge request will not be able to merged, and therefore updated documentation will not be published.

Conclusion¶

Example code snippets are an important part of technical documentation. If those snippets don’t work, users are less likely to adopt your product. In this article, we demonstrate an approach to ensure the snippets of code examples in your documentation not only compile, but produce desired results, thereby increasing the likelihood of user adoption of your product.