repo_man - Repo Tools Framework

repo_man (or “Repo Man”) – is a framework for building and running command line tools for code repositories (e.g. build, format, package etc.).

Documentation

Getting Started: Using repo tools

The main goal is to provide one single entry point for any repository: repo.

User clones any repository and runs a single command: repo.bat (windows) / ./repo.sh (linux). repo_man bootstraps itself (users don’t need to install anything) and lists all available tools (subcommands) with a description.

Example of repo output:

λ repo
usage: repo [-h] [-v] [-p] [-pr] [-tb] [TOOL] ...

Repo Tool (repoman):
    One entry point for all repo tools. Pass one of the tools and -h to get help.

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Increase verbosity of logging. Pass -v for
                        info, -vv for verbose.
  -p, --print-config-file
                        Output tool default config file and exit.
  -pr, --print-resolved-config
                        Output tool resolved config and exit.
  -tb, --tracebacks     Enable Python traceback logging to console
                        + TeamCity buildProblem reporting.

Found tools:
    update           Tool to update packman project files with newer versions of packages. By default major part of version is kept the same when looking for a newer version.
    publish          Publish archives (packages) and labels to packman remote.
    packman          Shortcut to packman.
    build            Build system main command.
    format           Format all C++ code (with clang-format) and all python code (with black).
    lint             Run python linters.
    ...

Run as 'repo [TOOL] -h' for more information on a specific tool.

Then a user can run repo [tool_name] to run a particular tool, e.g.: repo build.

Users can learn more about any tool repo [tool_name] -h.

Getting Started: Configuration

Another important goal of repo_man is to provide a common configuration system for all repo tools.

Each repo has repo.toml file in the root where any setting for any repo tool can be overridden.

Each tool has repo_tools.toml file with default settings for that tool (or collection of tools).

Each tool has a namespace in the config. For instance, repo build would be in [repo_build] section. Usually, all used tools are linked into _repo/deps/[tool_name] folder. This way you can find a configuration file for each tool in _repo/deps/[tool_name]/repo_tools.toml. Any setting can then be copied in repo.toml and changed.

Design Goals: Code Sharing (A Story)

Providing a single entry point for all repos greatly improves discoverability and new user experience. However, that was not the only motivation for repo_man. Another important goal of all repo tools is to encourage code reuse across many repositories, their tooling, and scripts. Here is a story to illustrate this, it captures history of “Repo Tools” in a simplified version, explains motivation for each feature:

It all starts with one repo, where repo [tool_name] just calls into tools/tool_name.py. Everyone loves that, Python is easy to change and cross-platform. Any feature or new tool can quickly be added right there. Writing more Python is always preferred over adding configs because it is easy for developers. Code quickly becomes messy, hard to read, and non-reusable.

Then, a few new repos get created. They do the same from a tooling perspective (use the same build system, format code the same way etc.). But each now has a copy of that code. Which quickly starts to diverge. At this point, the only option is to move all tools into packages, shared by all repos. All the custom code is replaced with configuration (like a list of folders to package, instead of nested for-loops). If any bug is found it is fixed once. If a new feature is requested it is implemented once. Code is reused, other repos just update tool versions.

Then, one repo diverges and needs a slightly different behavior. This is where the configuration system comes in. Each new feature can be hidden behind the setting. All repos can safely update, but this one repo can toggle that new feature in repo.toml. The configuration works great here, everyone can opt in and opt out of new features, while most code is reused.

Then, one repo wants something completely different and custom. For example, some code generation during the build. It doesn’t make sense to put it into shared code and pollute shared tools. Creating a new tool in a new repo might be an option, but developers want to quickly iterate on that code. Also, other parts of the build tool are still reused. In this situation, repo_man provides a way to override any tool and write a custom Python code in the repo. Configuration is just not enough here, and we need custom code.

Then, a new repo is created. It is written in a new language and uses a different build system. In that case, repo_man provides a way to just call into an existing tool. If a repository is written in rust and uses cargo build to build, it should be possible to run repo build and it will just run cargo build under the hood.

Finally, we have 100 repositories. And we notice that 50 of those set exactly the same set of settings in repo.toml, while others don’t. Once any setting is changed or added just updating tools is not enough and you want to share those settings, not tools. repo_man provides a way to import shared configuration in repo.toml, which comes from one of the tools those 50 repos use, while others don’t.

Bottom line is that repo_man nudges developers to do more code sharing. Reusing tools is preferred over committing them in each repo. Configuration is preferred over code. However, developers can always escape that when needed and do something completely custom.

repo_tools.toml

This file specifies and documents the configuration keys that repo_man uses.

########################################################################################################################
# Memo:
########################################################################################################################

# Tokens (string substitutions):
#
#   $root         - Repo root path.
#   $config_root  - Current config folder path.
#
# Env var tokens: '$ {env:PATH}', '$ {env:HOMEDRIVE}' etc. (no space after $)
# File content tokens: '$ {file:PATH}' etc. (no space after $)
# Tool config path: '$ {conf:repo.name}' etc. (no space after $)
#
# Platform specific Tokens. Name and examples:
#
# |==============================================================================
# | $platform      | 'windows-x86_64'     | 'linux-x86_64'  | 'macos-universal' |
# |------------------------------------------------------------------------------
# | $lib_ext       | '.dll'               | '.so'           | '.dylib'          |
# | $lib_prefix    | ''                   | 'lib'           | 'lib'             |
# | $bindings_ext  | '.pyd'               | '.so'           | '.so'             |
# | $exe_ext       | '.exe'               | ''              | ''                |
# | $shell_ext     | '.bat'               | '.sh'           | '.sh'             |
# |==============================================================================
#
# |==========================================================
# | $config        | 'debug'              | 'release'       |
# |==========================================================
#
#
# Platform filter:
#
# Use key like 'platform:windows-x86_64' to apply all it's content (value) only on windows platform. It will be merged into
# dictionary tree that contains that as a key.
#


#######################################################################################################################
# [repoman] Core repoman configuration
#######################################################################################################################
[repo]
logging = "warn"

# By default log everything repo tools do into the file:
file_logging = "info"
file_logging_path = "${root}/_repo/repo.log"

# Path to additional toml configs merge before applying this one. Use `import_optional_configs` if non-existence of the path is not an error.
import_configs = []
import_optional_configs = []

# Extra paths to look for repo tools in
extra_tool_paths = []


# This is used only by tools that specify `require_build_config = true` in
# their repo_tools.toml. If this is specified, this allows projects to specify
# `default_build_config` to either "debug" or "release". If the user does not
# specify `--config` on the command line when running the tool, this default
# value will be passed.
# Tools using `require_build_config = true` must store the config value in `args.config`.
default_build_config = "release"

# Recommended/Default folder structure:
[repo.folders]
root = "${root}"
build = "${root}/_build"
compiler = "${root}/_compiler"
installer = "${root}/_installer"
packages = "${root}/_build/packages"
signedpackages = "${root}/_signedpackages"
unsignedpackages = "${root}/_unsignedpackages"
host_deps = "${root}/_build/host-deps"
target_deps = "${root}/_build/target-deps"
repo_deps = "${root}/_repo/deps"
pip_packages = "${root}/_build/pip-packages"
deps_xml_folder = "${root}/deps"
host_deps_xml = "${root}/deps/host-deps.packman.xml"
target_deps_xml = "${root}/deps/target-deps.packman.xml"
assets_xml = "${root}/deps/assets.packman.xml"
prebuild_toml = "${root}/prebuild.toml"
premake_file = "${root}/premake5.lua"
version_file = "${root}/VERSION.md"
changelog_file = "${root}/CHANGELOG.md"
packages_file = "${root}/package.toml"
docs_root = "${root}"
docs_src = "${root}/docs"
docs_dst = "${root}/_build/docs"


# This block is intentionally commented out, as no defaults are provided. This exists as a reference.
# NOTE: There are no defaults within repo_man, you must set all Packman values if you want to
# retrieve your Python executable via Packman.
# There are multiple ways that you can define what Python executable is used by repo_tools.
# By default it will use the Python executable that Packman provides e.g.: tools/packman/python.sh

# You can also point at an absolute/relative path for a python executable via:
# [repo.python_executable]
# python_executable_path = "/usr/bin/python3"

# Another option is runtime pulling a Packman package with a Python executable in it. packman_link_path
# will be prepended to python_executable_path for a complete path.
# [repo.python_executable]
# python_executable_path = "python3${exe_ext}"
# packman_package_name = "python"
# packman_package_version = "3.10.10+nv1-${platform}"
# packman_link_path = "_repo/python/"

# Lastly you might have a Packman dependency file with a Python package that you wish to use. The linkPath
# in the packman dependency will be appended to python_executable_path for a complete path.
# [repo.python_executable]
# python_executable_path = "python3${exe_ext}"
# packman_file_path = "deps/host-deps.packman.xml"
# packman_package_name = "python"

[repo.telemetry]
# By default the NVDF URL is:
# https://gpuwa.nvidia.com/dataflow/{telemetry.nvdataflow-index}_v1/posting
# This is our default NVDF index:
nvdataflow-index = "omni-devplat-metrics-repo-tools"

# [repo.tokens]
# You can specify custom tokens here that you can reference anywhere you would reference a token.
# foo = 123
# bar = "${foo}456"

# These token values will be overridden by any tokens defined via `--set-token` but this can be
# a useful place for default values.

# [repo.rich]
# Replaces python's built-in print with `rich.print`
# replace_builtin_print = true

# Replaces python's `logging.StreamHandler` with `rich.logging.RichHandler`
# replace_log_handler = true

#######################################################################################################################
# [repo_upload] Simple tool to package and upload repo content into packman. Comes with repoman.
#######################################################################################################################

[repo_upload]
command = "upload"
entry_point = "omni.repo.man.upload:setup_repo_tool"

# Tools that come with repoman must be explicitly enabled. To avoid showing tools that might be misconfigured/unwanted.
enabled = false

# Package name to create. It must be specified for tool to become enabled
package_name = ""
# Package version
package_version = ""

# Folder to produce package to
output_folder = "${root}/_build/packages"

# Files to package, with some examples:
files = [
    # ["${root}/omni/repo/man/*.py", "omni/repo/man"],
    # ["*.toml", ""]
]

# Extra script to run before uploading
exec = ""


#######################################################################################################################
# [repo_update] Tool to update packman project files with newer versions of packages. Comes with repoman.
#######################################################################################################################

[repo_update]
command = "update"
entry_point = "omni.repo.man.update:setup_repo_tool"

# Enabled by default as it is pretty safe and has no configuration requried
enabled = true


#######################################################################################################################
# [repo_build_docs] Simple tool to build docs using Sphinx
#######################################################################################################################

[repo_build_docs]
command = "build_docs"
entry_point = "omni.repo.man.build_docs:setup_repo_tool"

# Tools that come with repoman must be explicitly enabled. To avoid showing tools that might be misconfigured/unwanted.
enabled = false

# Sphinx packman package to use
sphinx_version = "3.5.2-py3.7"

# python to use
python_path = "${root}/tools/packman/python${shell_ext}"

# if enabled config (debug/release/etc) can be passed into cmd args. ${config} token can be used in: 'python_path', 'repo.folders.docs_dst', 'extra_sys_paths'
enable_build_config_choice = false

# Additional folders to pass to PYTHONPATH when running Sphinx. Supports glob with wildcards (*)
extra_sys_paths = []


#######################################################################################################################
# [repo_publish] Simple tool to publish archives (packages) and labels to packman remote.
#######################################################################################################################

[repo_publish]
command = "publish"
entry_point = "omni.repo.man.publish:setup_repo_tool"

# Tools that come with repoman must be explicitly enabled. To avoid showing tools that might be misconfigured/unwanted.
enabled = false

# List of package names to choose from
packages = []

# Fail if package already exist on packman remote
fail_if_exist = true

# Fail if no matching packages or labels were located.
# This defaults off for backwards compatibility, but is recommended to be enabled to avoid false success in CI.
fail_if_nothing_found = false

# Search for labels (.txt files) near packages and publish them too
search_for_labels = true

# Default remote to use when packman push/publish
default_remote = "cloudfront"

# Generate labels from package file name using regexp (if matches). Each entry is a pair of regexp and substitution string.
label_regexps = [
	# Example: "omniverse-kit@101.0+master.b44868fc.teamcity.23668.windows-x86_64.debug.7z" -> "omniverse-kit@b44868fc.windows-x86_64.debug.txt"
    #['''(omniverse-kit)@\S+\.(\S+)\.\S+\.\S+\.(\S+)\.(\S+)\.\S+''', '''\1@\2.\3.\4.txt''']
]


#######################################################################################################################
# [repo_build_number] Tool to generate build number according to Omniverse Flow rfc
#######################################################################################################################

[repo_build_number]
command = "build_number"
entry_point = "omni.repo.man.build_number:setup_repo_tool"

# Use new "universal" scheme introduced by Halldor on 2021-12-01
omniverse_flow_version_scheme_2 = false

# 'repo.folder.version_file' setting is used as a path to VERSION file. It must exist for this tool to work.

# Tools that come with repoman must be explicitly enabled. To avoid showing tools that might be misconfigured/unwanted.
enabled = false


#######################################################################################################################
# [repo_packman] A CLI shortcut to packman
#######################################################################################################################

[repo_packman]
command = "packman"
entry_point = "omni.repo.man.packman_shortcut:setup_repo_tool"
packman_path = "${env:PM_INSTALL_PATH}/packman"


#######################################################################################################################
# [repo_poetry] Wrapper over poetry to vendor pip deps
#######################################################################################################################

[repo_poetry]
command = "poetry"
entry_point = "omni.repo.man.poetry:setup_repo_tool"

# Tools that come with repoman must be explicitly enabled. To avoid showing tools that might be misconfigured/unwanted.
enabled = false

# Default CLI args
cli_defaults.venv = "${root}/.venv"
cli_defaults.vendor_req_file = "${root}/vendor-requirements.txt"
cli_defaults.output_req_file = "${root}/requirements.txt"


#######################################################################################################################
# Traceback Reporting
#######################################################################################################################
# In called tools, in this example repo_format, it is possible to enable
# the reporting tracebacks on non-zero sys.exit and exceptions. This will
# push a TeamCity build problem if we're running in teamcity, and dump a
# traceback to the console.

# In the subtool, or your project repo.toml, you can configure both traceback options.
# Repo_* tools will eventually all have these flags enabled.
# Here is an example for repo_format, which would be present in
# repo_format's repo_tools.toml.
#[repo_format]
# Enable to log Python tracebacks to console + upload as teamcity problem
# On Python exception, typically an unexpected tool failure
#tracebacks_on_raise = false

# Note: repo_test is unique in that it disables tracebacks for non-zero system exit.
# This is due to our underlying test suites exitting non-zero on test failure, and
# it is just extra noise if we dump a traceback then. Please don't override this
# unless you want a lot of noise on failure.