Omniverse Telemetry Walkthrough

Adding Structured Logging

This guide covers the process of adding a new structured log schema to an Omniverse application. The end result of this walkthrough should be that your app can emit structured log events for your schema in to the offline log. This guide does not cover the process for having your schema approved by legal or having the events sent to the collection server.

Enable Structured Logging

To use structured logging, you will need to enable the setting: /structuredLog/enable. This setting by default is disabled to prevent applications from unexpectedly sending telemetry data, since some events are sent automatically. For example, a com.nvidia.carbonite.processlifetime.event is sent on framework startup.

Add this to your application’s configuration file to enable telemetry:

[structuredLog]
    "enable" = true

Creating a Schema

The first step to hooking up structured logging is creating a schema for your events. The schema will be used to generate code to send your messages. See Structured Log Message Schemas for a guide on creating your schema. The example schema from example.structuredlog is a good starting point for your new schema. These schemas can either be written in JSON or as a python dictionary. Python dictionary format is very similar to JSON but it has several benefits: comments are supported, strings can be line wrapped and trailing commas are accepted. Note that you can only put python literals in this format; you can’t execute code.

This format is almost identical to JSON except that it allows for comments, multi-line strings and trailing commas.

{
    "name": "example.structuredlog",
    "version": "1.0",
    "namespace": "com.nvidia.carbonite.example.structuredlog",
    "description": "Example schema to demonstrate how to use IStructuredLog."
                   " This sends a dummy startup time event.",
    "flags": [ "fSchemaFlagAnonymizeEvents" ],
    "events": {
        "startup": {
            "privacy": {
                "category": "performance",
                "description": "example event, so categorization is arbitrary"
            },
            "description": "Marks when an app was started and how long startup took."
                           " This data is gathered to demonstrate the telemetry system.",
            "flags": [],
            "properties": {
                "startupTime": {
                    "type": "uint64",
                    "description": "time to startup in nanoseconds"
                },
                "registerTime": {
                    "type": "uint64",
                    "description": "time to register this schema in nanoseconds"
                },
                "exampleString": {
                    "type": "string",
                    "description": "an example string parameter"
                }
            }
        },
        # Multiple events can be added to the events dictionary.
        # Comments and trailing commas are fine in this format.
    }
}

Note that if your data is intended to be used for telemetry, you should avoid creating properties with type object. Hierarchical data is not efficient to manage in a database, so telemetry data should be kept flat.

You can validate that your structured logging schema satisfies requirements of the structured logging system by running the code generator without any output file specified.

./tools/omni.structuredlog/structuredlog.sh my_schema.json

Pure Python Projects

Structured logging can be integrated into python projects without needing to add a build step to the project. This is done with the omni.structuredlog python bindings.

You can generate a python module that performs the event registration for you and adds wrapper functions to more easily send telemetry events. This code generator will also validate the telemetry schema, so you are recommended to use this. This shell code from the root of the Carbonite repo will generate a python module from a JSON schema.

./tools/omni.structuredlog/structuredlog.sh my_schema.json --py=generated_python_module.py

Once you’ve imported this generated python module, all that’s needed to send a structured log messages is to call one of the helpers. Note that if you pass invalid parameters, the structured log message will not be sent and an error will be logged.

    example.structuredlog_generated.startup_send_event(
        registerTime=(end - start) * 1000,
        startupTime=0,
        exampleString="python example string (from a generated helper)",
    )

Inside the Generated Python Module

The internals of the generated python module are straightforward. If you want to use the omni.structuredlog API directly, then you will need to write code similar to this.

The first thing done is to import the omni.structuredlog bindings.

import omni.structuredlog

After the bindings have been imported, your schema is registered with the structured logging system directly from the JSON schema. The returned object is a dictionary of the events that were registered; the keys in the dictionary are the short names of the events. Note that this call may raise an exception if your schema is invalid.

    try:
        events = omni.structuredlog.register_schema(schema)
    except Exception as e:
        omni.log.error("failed to register the schema: " + str(type(e)) + " " + str(e))
        return False

With the schema registered, we can then send structured log events. Note that these calls may throw if the dictionary structure is invalid. The generated code wraps each send call with a helper function to simplify usage.

    try:
        omni.structuredlog.send_event(
            events["startup"],
            {
                "registerTime": (end - start) * 1000,
                "startupTime": 0,
                "exampleString": "python example string (from bindings)",
            },
        )
    except Exception as e:
        omni.log.error("exception thrown: " + str(type(e)) + " " + str(e))
        return False

Generating Code

For C++ apps, you will need to generate a C++ header that gives you an API for your schema. The generated code will allow you to send strongly typed structured log events with minimal performance overhead.

The first step in generating a header is to include the structured log build script in your premake5.lua build script:

dofile('tools/omni.structuredlog/omni.structuredlog.lua')
setup_omni_structuredlog("./")

Also include passthroughs for the various options for the omni.structuredlog tool in your repo.toml script:

[[repo_build.argument]]
name = "--fail-on-write"
help = """
    When enabled, any code generation tool will fail if it needs to write any changes to disk.  This is intended to
    be used on CI systems to ensure that all code changes have been included in any given MR.  If a build step does
    fail due to this option, a local build should be performed first before committing changes to an MR branch."""
kwargs.required = false
kwargs.nargs = 0
extra_premake_args = ["--fail-on-write=1"]
platforms = ["*"] # All platforms.

You can generate code for your schema using omni_structuredlog_schema. The schema parameter is the schema you created. The cpp_output parameter is the output path for the C++ header that will be generated from your schema. The pybind_output parameter is the output path for a python bindings header that can be optionally generated. The namespace argument specifies the C++ namespace for the generated code. The bake_to parameter is optional; if this is specified, it will be the path to write the generated JSON schema to for your simplified schema. If you are using a JSON schema, you should instead add baked = true to the arguments list. See Structured Log Message Schemas for an explanation on the simplifies and full JSON schemas.

    project "example.structuredlog.schemas"
        location (workspaceDir.."/%{prj.name}")
        omni_structuredlog_schema {
            {
                schema = "source/examples/example.structuredlog/example.structuredlog.schema",
                cpp_output = "source/examples/example.structuredlog/example.structuredlog.gen.h",
                pybind_output = "source/examples/example.structuredlog/bindings-python/example.structuredlog.python.h",
                bake_to = "source/examples/example.structuredlog/example.structuredlog.json",
                namespace = "example::structuredlog",
            },
            {
                schema = "source/examples/example.structuredlog.dynamic/example.structuredlog.standalone.schema",
                cpp_output = "source/examples/example.structuredlog.dynamic/example.structuredlog.standalone.gen.h",
                bake_to = "source/examples/example.structuredlog.dynamic/example.structuredlog.standalone.json",
                namespace = "example::structuredlog",
            }
        }

    project "example.structuredlog"
        dependson { "example.structuredlog.bindings.python" }

If more than one schema needs to be built for a given project, they may all be specified in a single call to omni_structuredlog_schema by providing an array of argument objects. Any number of schemas may be added in this manner. Having multiple calls to omni_structuredlog_schema in a single project will work on Linux, but due to some bugs in premake will cause only the first schema to build under Visual Studio.

For example, multiple schemas can be added to the build like this:

        omni_structuredlog_schema {
            {
                schema = "source/plugins/omni.structuredlog/structuredlog.log_consumer.schema",
                bake_to = "source/plugins/omni.structuredlog/structuredlog.log_consumer.json",
                cpp_output = "source/plugins/omni.structuredlog/structuredlog.log_consumer.gen.h",
            },
            {
                schema = "include/omni/structuredlog/StructuredLog.ProcessLifetime.json",
                cpp_output = "include/omni/structuredlog/StructuredLog.ProcessLifetime.gen.h",
                pybind_output = "source/bindings/python/omni.processlifetime/StructuredLog.ProcessLifetime.bindings.python.h",
                baked = true
            }
        }

A generated C++ header can either be generated in its own project with a dependency listed in other projects (to ensure build order), or omni_structuredlog_schema() can be called from within the same project that depends on it. If more than one other project depends on the same generated header, bindings, or script, it is often best to generate them from the schema in a separate project then use dependson in the project to ensure the build order. If only a single project uses the generated code, the omni_structuredlog_schema() call is often better placed within the dependent project itself. In both cases the build order will be correct since the code generation step will run as a pre-build step for the project.

Using the Generated Code in C++ Apps

All that is needed for your structured logging schema to be registered is for you to include the generated header. Once omni.core initializes, your schema will automatically be registered and you’ll be able to send structured log events.

For most use cases, you only need to look at the set of macros at the top of the generated header file. Each of these generated macros will send a structured log event.

    OMNI_EXAMPLE_STRUCTUREDLOG_1_0_STARTUP(startupTime, 0, "example string");

Building the Python Bindings

In addition to the C++ generated header, you can also generate a header that will wrap the C++ API with pybind11, to expose it to python. This functionality exists because it was the original way of accessing structured logging from python. This method of adding python support has a few benefits: the schema won’t be directly included as text in your code, and the performance cost of sending events should be lower.

To add these bindings, you must create a cpp file for these bindings. Unless you have an unusual use case, you can copy the example binding code and change the names.

// Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.
//
// NVIDIA CORPORATION and its licensors retain all intellectual property
// and proprietary rights in and to this software, related documentation
// and any modifications thereto. Any use, reproduction, disclosure or
// distribution of this software and related documentation without an express
// license agreement from NVIDIA CORPORATION is strictly prohibited.
//
#include "example.structuredlog.python.h"

CARB_BINDINGS("example.structuredlog")

namespace
{

PYBIND11_MODULE(structuredlog, m)
{
    example::structuredlog::definePythonModule_example_structuredlog(m);
}

} // namespace

The python bindings just need to have a dependson{} call that points to your schema build target. This is done in example.structuredlog using the define_multi_binding_python() function in Carbonite.

    define_multi_bindings_python {
        project_name = "example.structuredlog.python",
        name = "structuredlog",
        namespace = "example",
        folder = "source/examples/example.structuredlog/bindings-python/",
        dependson = { "example.structuredlog.schemas" }
    }

Projects that use define_bindings_python() can create a build target like this:

project "example.structuredlog.bindings.python"
    dependson { "example.structuredlog.structuredlog_schema" }
    define_bindings_python {
        name = "structuredlog",
        namespace = "example"
    }
    files {"source/examples/example.structuredlog/bindings-python/*.*"}

Using the Generated Code Python Bindings

Using the python bindings is simple. Once you’ve imported your bindings, you just need to create the helper class for your schema.

    inst = example.structuredlog.Schema_example_structuredlog_1_0()

Once you have the class, you can call any of the *_sendEvent() functions to send structured logging events.

    inst.startup_sendEvent(0, int((end - start) * 1000), "python example string")

Testing Your Structured Log Event Integration

Once the generated C++ or python code has been integrated into a project, it should be tested. The easiest way to do this is to run the app or component with the new integration and execute past the point where a new event message would have been sent. Once that messaging code has run, at least one new message from the event should arrive in the log file. By default, the log file can be found in the $HOME/.nvidia-omniverse/logs/ folder. On Linux, $HOME will be the user’s home folder. On Windows, this will be the user’s profile folder (ie: $USERPROFILE).

Locate the log folder and search for a message in the log files with the new event’s identifier. The event identifier can be found in the event’s schema by combining the event prefix for the schema and the name of the given being search for. If the message can be found in the log file and all of the properties of it are correct, the telemetry integration testing can be started. If the message cannot be found or its content cannot is not correct, simply go back to the code and figure out why it’s not being output correctly by stepping through the call in the debugger.

Testing Your Telemetry Integration

Once the new message has been confirmed to be going to the log file correctly, the job of the app’s integration of structured logging is complete. After this point, it is the job of the telemetry transmitter app omni.telemetry.transmitter to get the messages from the log file(s) and send them up to the appropriate data endpoint. To do this, a few steps need to be taken to get the telemetry transmitter app running. The way it is configured and run depends on how it has been integrated into the host app.

The most common way of integrating the telemetry transmitter app would be through the Kit SDK. This has an omni.kit.telemetry extension that handles launching the transmitter app on its startup. If an integration with Kit is not being used, it is the host app’s responsibility to appropriately launch the telemetry transmitter. There is a set of helper functions for doing this in include/omni/structuredlog/Telemetry.h. The main function to do this is omni::telemetry::launchTransmitter().

When the telemetry transmitter app will be sending messages to the default Omniverse data endpoints (test or production), authentication is required. In order to accomplish this, the transmitter must retrieve the current user’s authorization token from the Omniverse Launcher app. If the Launcher app is not running, the transmitter app will go into an idle state until the Launcher becomes available. If the default Omniverse data endpoints are not used, the host app that launches the transmitter is either responsible for getting an appropriate authorization token to the transmitter, or to disable authentication using the /telemetry/authenticate setting.

Before the transmitter can send any new test events to the data endpoint, it must first be given the schema that will validate those events. This schema file is generated by the omni.structuredlog tool using the bake_to option when a ‘.schema’ file is used. Only JSON schemas are accepted by the transmitter tool. By default, the transmitter will download all approved schemas from the schema URL. To test an unapproved schema, the new schema(s) must be injected into the transmitter. This can be done with a debug build of the tool (_build/<platform>/debug/omni.telemetry.transmitter). The /telemetry/schemaFile or /telemetry/schemasDirectory options can be used to specify which set of local JSON schema file(s) get injected into this run of the transmitter app.

Note that running the transmitter for testing purposes often requires that it be launched manually. When doing this, it is easiest to also use the /telemetry/stayAlive setting so that it doesn’t quit on its own once it sees that no Omniverse apps are connected to it.

Once the transmitter has been configured correctly, it should start consuming new events that appear in the log file(s). When injecting test schemas, these will be defult be sent to the Omniverse data staging table. These events can later be analyzed to make sure they will be useful in investigating the necessary trends from the data.