Configuration

repo_man comes with a powerful configuration system to improve flexibility of tools and promote code reusability.

TOML

TOML is used as a configuration format. It has ini like syntax, which makes it human readable and easy to copy and paste between configs. It was choosen as The configuration format for the Omniverse and repo_man followed it.

If you are unfamiliar with it I recommend using any online converter between JSON and TOML (or any other format). In the end they all build a nested dictionary and very similar.

Configuration files

When any tool [tool_name] runs at least 5 configurations merged on top of each other (in order):

repo_man’s repo_tools.toml for tools-wide settings
[tool_name]’s repo_tools.toml for tool-specific settings
repo.toml for repo-specific settings
~/.nvidia-omniverse/config/global_repo.toml for global repo settings.
user.repo.toml for repo-specific per-user settings

Config settings get overridden the further down the sequence you go. So repo.toml settings will override the repo_tools.toml defaults, and user.repo.toml will override anything set earlier in the stack.

Merging (or overriding) happens on leaf values (like integer, string, array), but not on dictionaries (or tables).

repo_tools.toml serves as a source for all default settings for any tool. User can look into _repo/deps/[tool_name]/repo_tools.toml for any settings and copy in repo.toml. We highly recommended to put all settings default values in repo_tools.toml with a comment for a better discoverability.

~/.nvidia-omniverse/config/global_repo.toml is the second to last layer and should be used for setting local values unique to your environment. Examples include: forcing repo tools logging to be verbose, disabling repo_build ccache remote_storage, or setting repo_build max job count. It is not advisable to set core configuration values here such as repo_format clang_format version or repo_build target config. ~ resolves to C:\Users\{username} on Windows at /home/{username} on Linux.

user.repo.toml is the last layer and can be used for setting local values unique to your environment and one particular project. For example you may only wish to build release builds and can override the repo_build default config value for one project.

Example:

repo_build has repo_tools.toml with:

[repo_build]

# All build configs supported
build_configs = ["debug", "release"]

Some repo can only build in release configuration. So it overrides it in repo.toml:

[repo_build]

# Only "release" is supported because of Y:
build_configs = ["release"]

Tokens

repo_man supports tokens to make configuration more flexible. They take a form of ${token} or $token.

They can be escapped using double $$: $${token} will be resolved to ${token}.

Most of them are resolved when configuration is loaded. However some of them are unknown until tool actually runs. Those are left in config as is and it is each tool’s responsibility to resolve them.

Tokens supported by repo_man:

${root} - repo root path.

${config_root} - current config folder path.

${platform} - current host platform. For example windows-x86_64 or linux-x86_64.

${lib_ext} - current platform’s library extension. For example .dll or .so.

${lib_prefix} - current platform’s library prefix. For example lib or empty string.

${bindings_ext} - current platform’s bindings extension. For example .pyd or .so.

${exe_ext} - current platform’s executable extension. For example .exe or empty string.

${shell_ext} - current platform’s shell extension. For example .bat or .sh.

${in_ci} - Set to True if running on gitlab or TeamCity, False otherwise.

${py_ver} - current Python interpreter major and minor version. For example 3.10.4 => 310.

${py_tag} - Python tag per Python packaging guidelines. For example 3.10.4 => py310.

Tokens typically resolved by other tools:

${config} - current build config. For example debug or release.

${platform_target} - target platform (can be different from host platform).

Tokens defined in a config:

New tokens can be defined in any config (repo.toml or repo_tools.toml) in [repo.tokens] section. For example:

[repo.tokens]
foo = 123
bar = "${foo}456"

[some_tool]
x = "${bar}"

Then some_tool.x will be resolved to 123456.

Tokens defined via CLI:

You can also set a token with the --set-token argument. It takes the form of a colon separated key:value pairing. This works nicely with a default token value defined in your repo.toml with said token being overridden perhaps programmatically in CI.

Listing all tokens

For any tool run repo -pt [tool_name] (or --print-tokens) to print all tokens known at configuration time.

Interpolations

Some tokens can take a special form of ${operation:value}. Where operation is a custom interpolation to perform on a token:

Environment variables: `env`

${env:PATH} - value of PATH environment variable.

File content: `file`

${file:path} - content of a file at path. E.g.: "${file:${config_root}/VERSION}".

Other config value: `conf`

${conf:path} - value of other config setting setting at path, e.g. x = "${conf:repo_build.build_configs}" then x would have the same value as repo_build.build_configs setting.

API

When writing a tool use omni.repo.man.resolve_tokens to resolve tokens. Use omni.repo.man.set_token(name, value) to set a new token at runtime. Example:

import omni.repo.man

# Set ${config} to be 'release'
omni.repo.man.set_token("config", "release")
# Now strings with config can be resolved (all other tokens work too):
value = omni.repo.man.resolve_tokens("${platform}-${config}")
print(value) # "windows-x86_64-release"
# dict, list, tuple also supported, but replaced in-place:
value = {"foo": "${platform}-${config}"}
omni.repo.man.resolve_tokens(value)
print(value) # {"foo": "windows-x86_64-release"}

# Token can be removed if None is passed:
omni.repo.man.set_token("config", None)

Filters

Filters are used to apply settings conditionally. They are defined as a key in a dictionary.

To apply settings only on certain platforms use 'platform:[platform]'. For example:

[tool_name]
foo.bar = 3
foo."platform:linux-aarch64".bar = 5

This will set foo.bar to 3 on all platforms except linux-aarch64 where it will be set to 5.

A whole section (dictionary) of settings can be overriden like that.

Array Merging

Arrays can be merged using ++ key. For example,

repo_tools.toml:

[tool_name]
foo = [1, 2]

repo.toml:

[tool_name]
foo."++" = [3, 4]

Will result in tool_name.foo = [1, 2, 3, 4].

If array does not exist it will be created.

Importing Extra Configs

Before applying final repo.toml extra configs can be merged (and thus used as templates) using a import_configs or import_optional_configs key:

[repo]
# Path to additional toml configs merge before applying this one, which must exist
import_configs = [
    "${root}/_repo/deps/repo_kit_tools/kit-template/repo.toml",
]

# Path to additional toml configs merge before applying this one, which can not exist
import_optional_configs = [
    "/home/user/some_config.toml",
]

Paths and Folders

Inside of repo_man’s repo_tools.toml there is a section [repo.folders] which defines default (recommended) paths for various standard folders. This way each tool can use them and each repo can override them.

Here are some of them:

# Recommended/Default folder structure:
[repo.folders]
root = "${root}"
build = "${root}/_build"
compiler = "${root}/_compiler"
packages = "${root}/_build/packages"
host_deps = "${root}/_build/host-deps"
target_deps = "${root}/_build/target-deps"
repo_deps = "${root}/_repo/deps"
pip_packages = "${root}/_build/pip-packages"
...

Typically in tools it looks like this:

def run_tool(options: Dict, config: Dict):
    repo_folders = config.get("repo", {}).get("folders", {})

    # Path to repo root
    root_path = repo_folders["root"]

    # Path to build folder
    build_path = repo_folders["build"]

Command Line Config Overrides

To override (or add) any config setting from command line use a special syntax: --/[key]=[value]. Where [key] is path to a value in config separated by /:

repo test --/repo_test/tracebacks_on_exit=1

Overrides repo_test.tracebacks_on_exit setting to true.

If value is already present in config (which is usually the case) type of that value is used to convert to.

If value is not present (new) a reasonable conversion is attempted:

--/foo/bar=1 - Integer
--/foo/bar=true - Bool
--/foo/bar=0.5 - Float

Quotes can be used to force string:

--/foo/bar="1" - String --/foo/bar='1' - String

Tokens are supported, e.g.:

--/foo/bar=${root} - Path to the root of repo --/foo/bar=${env:HOME} - HOME environment variable etc.

Overriding arrays is not supported.

Use `--` to ignore or passthrough cmd arguments

All command line arguments after -- are ignored. E.g. -- --/foo/bar=2 --help will not override foo/bar config setting and will not show help. Instead it strips ["--/foo/bar=2", "--help"] from sys.argv and passes them to the invoked tool as options.extra_args.

This is useful to pass arguments to some other tool or process.

Typical example is repo_test tool which runs some other test runner under the hood, like doctest or pytest. To pass extra arguments to doctest put them after --:

repo test --suite unittests -- -tc="*[wildcard]*"

Instead of each tool implemting own way to pass extra arguments (e.g. -e="--foo") they can just pass options.extra_args. That also standardizes the way to pass extra arguments to any tool.

Debugging Configuration

For any tool run repo -pr [tool_name] to output resolved configuration.

Using Configuration in tools

In of any tool resolved configuration is passed into the tool run function as python dict. Typically tools would have code like:

def run_tool(options: Dict, config: Dict):
    tool_config = config.get("tool_name", {})

And then:

bar = tool_config.get("foo", {}).get("bar", None)