Creating a New Carbonite Interface

Overview

Carbonite interfaces are simple yet powerful when used appropriately. There are a few considerations to be made when choosing to make a Carbonite interface versus an ONI one. Carbonite intefaces are best used in situations where global data or state can be used or when an object hierarchy is not necessary. Using a Carbonite interface for these projects can simplify the workflow, but also has some drawbacks and restrictions.

Some benefits of Carbonite interfaces are:

  • Interfaces are simple and can be more efficient to call into. There does not need to be an API wrapper layer in the interface and there are not as many restrictions on what can be passed to functions.

  • Interfaces are quick and easy to implement. A new plugin for a Carbonite interface can be created in a matter of minutes.

  • They match better to some usage cases.

Some of the main drawbacks with Carbonite interfaces are:

  • Interfaces are not reference counted. Since a Carbonite interface is nothing more than a vtable containing function pointers, every caller that attempts to acquire a given interface will receive the same object. There is also no need to release he interface when it is no longer needed. This can lead to problems if one caller decides to unload the plugin. There is no notification to the other callers that acquired the interface that its plugin is being unloaded.

  • Interfaces cannot have data associated with them. Contextual object information returned from a Carbonite interface must go through opaque pointers that then get passed back to other functions in the interface to operate on. This often ends up making the interface flat and monolithic instead of hierarchical.

  • ABI safety and backward compatibility between interface version is difficult to maintain without close attention to detail when making changes. It is easy to break compatibility with previous versions of an interface without necessarily realizing it. This was one of the main motivations for creating ONI and its toolchain.

  • A given Carbonite plugin may only implement a single version of any given interface. If multiple versions of an interface are needed (either using a different backing provider for the interface or implementing a new non-backwards-compatible version of the interface), multiple plugins must be used to contain their implementation.

Here we’ll make an interface called carb::stats::IStats to collect and aggregate simple statistical values.

Project Definitions

The first step in creating a new Carbonite interface (and a plugin for it) is to add a project new definition to the premake5.lua script:

    project "example.carb.stats.plugin"
        define_plugin { ifaces = "source/examples/example.stats/include/carb/stats", impl = "source/examples/example.stats/plugins/carb.stats" }
        dependson { "carb" }
        includedirs { "source/examples/example.stats/include" }
        filter { "system:linux" }
            buildoptions { "-pthread" }
            links { "dl", "pthread" }
        filter {}

This adds a new plugin project called example.carb.stats.plugin. When built, this will produce a plugin such as example.carb.stats.plugin.dll (on Windows). The plugin is set to implement some or all of the interfaces in the source/examples/example.stats/include/carb/stats/ folder with its implementation files in source/examples/example.stats/plugins/carb.stats/.

Define the Interface API

The next step is to create the interface’s main header. According to the coding standards for Carbonite, this should be located in a directory structure that matches the namespace that will be used (ie: include/carb/stats/ in this case). The interface’s header file itself should have the same name as the interface (ie: IStats.h here).

The bare minimum requirements for declaring an interface are that the carb/Interface.h header be included, a #pragma once guard is used, and at least one struct containing a call to the CARB_PLUGIN_INTERFACE() macro be made:

/** A simple global  table of statistics that can be maintained and aggregated.  Statistics
 *  can be added and removed as needed and new values can be aggregated into existing
 *  statistics.  Each statistic's current value can then be retrieved at a later time for
 *  display or analysis.
 */
struct IStats
{
    CARB_PLUGIN_INTERFACE("carb::stats::IStats", 0, 1)

    /** Adds a new statistic value to the table.
     *
     *  @param[in] desc The descriptor of the new statistic value to add.
     *  @returns An identifier for the newly created statistic value if it is successfully added.
     *           If this statistic is no longer needed, it can be removed from the table with
     *           removeStat().
     *  @returns @ref kBadStatId if the new statistic could not be added or an error occurred.
     *
     *  @thread_safety This operation is thread safe.
     */
    StatId(CARB_ABI* addStat)(const StatDesc& desc);

    /** Removes an existing statistic value from the table.
     *
     *  @param[in] stat The identifier of the statistic to be removed.  This identifier is
     *                  acquired from the call to addStat() that originally added it to the
     *                  table.
     *  @returns `true` if the statistic is successfully removed from the table.
     *  @returns `false` if the statistic could not be removed or an error occurred.
     *
     *  @thread_safety This operation is thread safe.
     */
    bool(CARB_ABI* removeStat)(StatId stat);

    /** Adds a new value to be accumulated into a given statistic.
     *
     *  @param[in] stat     The identifier of the statistic to be removed.  This identifier is
     *                      acquired from the call to addStat() that originally added it to the
     *                      table.
     *  @param[in] value    The new value to accumulate into the statistic.  It is the caller's
     *                      responsibility to ensure this new value is appropriate for the given
     *                      statistic.  The new value will be accumulated according to the
     *                      accumulation method chosen when the statistic was first created.
     *  @returns `true` if the new value was successfully accumulated into the given statistic.
     *  @returns `false` if the new value could not be accumulated into the given statistic or the
     *           given statistic could not be found.
     *
     *  @thread_safety This operation is thread safe.
     */
    bool(CARB_ABI* addValue)(StatId stat, const Value& value);

    /** Retrieves the current accumulated value for a given statistic.
     *
     *  @param[in]  stat    The identifier of the statistic to be removed.  This identifier is
     *                      acquired from the call to addStat() that originally added it to the
     *                      table.
     *  @param[out] value   Receives the new value and information for the given statistic.  Note
     *                      that the string values in this returned object will only be valid
     *                      until the statistic object is remvoed.  It is the caller's
     *                      responsibility to ensure the string is either copied as needed
     *                      or to guarantee the statistic will not be removed while its value is
     *                      being used.
     *  @returns `true` if the statistic's current value is successfully retrieved.
     *  @returns `false` if the statistic's value could not be retrieved or the statistic
     *           identifier was not valid.
     *
     *  @thread_safety It is the caller's responsibility to ensure the given statistic is
     *                 not removed while the returned value's strings are being used.  Aside
     *                 from that, this operation is thread safe.
     */
    bool(CARB_ABI* getValue)(StatId stat, StatDesc& value);

    /** Retrieves the total number of statistics in the table.
     *
     *  @returns The total number of statistics currently in the table.  This value can change
     *           at any time if a statistic is removed from the table by another thread.
     *
     *  @thread_safety Retrieving this value is thread safe.  However, the actual size of the
     *                 table may change at any time if another thread modifies it.
     */
    size_t(CARB_ABI* getCount)();
};

Any other requirements for the interface (ie: enums, parameter structs, constants, etc) should be declared either before the interface’s struct or in another header that is included at the top of the file.

The CARB_PLUGIN_INTERFACE() macro call adds some simple boilerplate code to the interface that facilitates discovery in plugins. It needs to know the name of the plugin (with the full namespace) as well as the version number of the interface (major and minor version numbers).

See Interface Design Considerations below for more information on how to best design your interface, stick to our coding standards, and avoid various pitfalls.

Add an Implementation for the Interface

A Carbonite interface may have multiple implementations. Typically these are done so that each implementation handles a specific type of data (ie: carb.dictionary.serializer-json.plugin versus carb.dictionary.serializer-toml.plugin) or each implementation provides functionality from a different backend library (ie: carb.profiler-cpu.plugin versus carb.profiler-tracy.plugin versus carb.profiler-nvtx.plugin). By convention, the backend provider or data handling context is reflected in the plugin’s name itself.

A simple interface like carb::stats::IStats could theoretically be implemented in a single source file. However for example purposes here, we’ll split it up into multiple files as though it was a more complex interface. The following files will be used here:

  • source/examples/example.stats/plugins/carb.stats/Stats.h: A common internal header file for the implementation of the plugin. This should contain any functional definitions needed to implement the interface. This file is relatively straightforward and can simply be examined.

  • source/examples/example.stats/plugins/carb.stats/Stats.cpp: The actual implementation of the carb::stats::IStats interface as a C++ class. This class implementation is relatively straightforward and can simply be examined.

  • source/examples/example.stats/plugins/carb.stats/Interfaces.cpp: The definition of the Carbonite interface itself. This file typically acts as the point where all interfaces in the plugin are exposed to the Carbonite framework. This file includes the simple C wrapper functions used to call through to the C++ implementation class.

Some important parts of the Interfaces.cpp file are required in order to expose your interface’s implementation to the Carbonite framework:

  • The CARB_EXPORTS macro must be defined before including any Carbonite headers. This allows the framework to know that the functions tagged with CARB_EXPORT should be exported from the final built plugin module.

/// This symbol must be defined before any Carbonite headers are included in the source file that
/// makes the call to CARB_PLUGIN_IMPL().  There should be exactly one source file in the plugin
/// that makes that call and all implemented interfaces in that module should be listed in that
/// single call.
///
/// Defining this symbol ensures any functions that are tagged with `CARB_EXPORT` are properly
/// exported from the module.
#define CARB_EXPORTS

#include "Stats.h"

#include <carb/InterfaceUtils.h>
#include <carb/PluginUtils.h>

  • The plugin’s information and interface descriptions must be exported. This is done by calling the CARB_PLUGIN_IMPL() macro, the CARB_PLUGIN_IMPL_DEPS() macro if this plugin depends on other interfaces, and defining one fillInterface() function for each implemented interface.

/// Define the descriptor for this plugin.  This gives the plugin's name, a human readable
/// description of it, the creator/implementor, and whether 'hot' reloading is supported
/// (though that is a largely deprecated feature now and only works properly for a small
/// selection of interfaces).
const struct carb::PluginImplDesc kPluginImpl = { "example.carb.stats.plugin", "Example Carbonite Plugin", "NVIDIA",
                                                  carb::PluginHotReload::eDisabled, "dev" };

/// Define all of the interfaces that are implemented in this plugin.  If more than one
/// interface is implemented, a comma separated list of each [fully qualified] plugin
/// interface name should be given as additional arguments to the macro.  For example,
/// `CARB_PLUGIN_IMPL(kPluginImpl, carb::stats::IStats, carb::stats::IStatsUtils)`.
CARB_PLUGIN_IMPL(kPluginImpl, carb::stats::IStats)

/// Mark that this plugin is not dependent on any other interfaces being available.
CARB_PLUGIN_IMPL_NO_DEPS()

/** Main interface filling function for IStats.
 *
 *  @remarks This function is necessary to fulfill the needs of the CARB_PLUGIN_IMPL() macro
 *           used above.  Since the IStats interface was listed in that call, a matching
 *           fillInterface() for it must be implemented as well.  This fills in the vtable
 *           for the IStats interface for use by callers who try to acquire that interface.
 */
void fillInterface(carb::stats::IStats& iface)
{
    iface.addStat = addStat;
    iface.removeStat = removeStat;
    iface.addValue = addValue;
    iface.getValue = getValue;
    iface.getCount = getCount;
}

  • The plugin’s callback functions. At the very least, every plugin should implement the carbOnPluginStartup[Ex]() and carbOnPluginShutdown() functions. There are other optional callback functions as well. Some of these callbacks are generated by some of the macros that are used. The full list of callback names is in include/carb/PluginCoreUtils.h.

CARB_EXPORT void carbOnPluginStartup()
{
    /// Do any necessary global scope plugin initialization here.  In this case nothing needs
    /// to be done!
}

CARB_EXPORT void carbOnPluginShutdown()
{
    /// Do any necessary global scope plugin shutdown here.  This should include resetting any
    /// global variables or objects to their defaults, to a point where they can be safely
    /// reinitialized in the next carbOnPluginStartup() call.
    ///
    /// Note that because of some dynamic linking behavior on Linux, it is possible that on
    /// unload this module could be shutdown but not actually unloaded from memory.  This would
    /// prevent any global variables from being reset or cleaned up during C++ static
    /// deinitialization.  However, if the plugin were loaded into the process again later,
    /// C++ static initialization would not occur since the module was not previously unloaded.
    /// Thus it is important to perform any and all global and static cleanup explicitly here.
    ///
    /// In our case here, the only thing to do would be to empty the global stats table so that
    /// it's in the expected empty state should the plugin be loaded again.

    g_stats.clear();
}

  • The plugin callbacks are exported from the plugin module itself. These are tagged with CARB_EXPORT in a .cpp file that has CARB_EXPORTS defined at the top. Some of the more interesting callbacks are:

    • carbOnPluginStartup(): If this callback is exported and carbOnPluginStartupEx() is not exported, this callback is performed to notify the plugin that it has been loaded by the Carbonite framework. This callback is intended to handle any initialization tasks that are needed by the library before any interfaces from it are acquired. This is guaranteed to only be called once per module load instance and called before any interfaces in the plugin can be acquired. If the module is unloaded, the startup callback function will be called again if the next time it is loaded.

    • carbOnPluginStartupEx(): This callback is a newer version of the carbOnPluginStartup() callback function. If exported, it will be preferrentially called instead of the older version. It has the same purpose however.

    • carbOnPluginShutdown(): If this callback is exported, it will be called whenever the plugin module is unloaded from the process. This may occur at any time the module is intentionally unloaded, not necessarily just at process exit time. The only exception to this is that it is not called when the process is terminated by the ‘quick shutdown’ method if the carbOnPluginQuickShutdown() callback is also exported. This quick shutdown is triggered using the carb::quickReleaseFrameworkAndTerminate() function. This callback is intended to be the point where the plugin should handle its full set of shutdown tasks. This may include tasks such as releasing or freeing system resources, flushing data to disk, or cleaning up global objects.

    • carbOnPluginQuickShutdown(): If this callback is exported, it will be called when the host app triggers the ‘quick shutdown’ method using carb::quickReleaseFrameworkAndTerminate(). If this callback is not exported, the carbOnPluginShutdown() callback will be performed instead during the quick shutdown of the process. The intention of this specific callback is to only perform the most critical shutdown tasks for the plugin. This may include tasks such as flushing data to disk or saving persistent state. It should accomplish these tasks as quickly as possible and should not worry about tasks such as freeing memory or unloading dependencies.

Once these files are in place, the plugin can be built. It should produce a plugin module that the Carbonite framework can load and discover the carb::stats::IStats interface in.

Using the New Interface

Now that the new plugin has been built, a host app will be able to discover and load it. Let’s create a simple example app. A simple example host app can be found at source/examples/example.stats/example.stats/main.cpp. Much of this example app is specific to the carb::stats::IStats interface itself. The only parts that really need some explanation are:

  • CARB_GLOBALS() must be called at the global scope. This adds all of the global variables needed by the Carbonite framework to the host app. This will include symbols to support functionality such as logging, structured logging, assertions, profiling, and localization. This must be called in exactly one spot in the host app.

/// This macro call must be made at the global namespace level to add all of the Carbonite
/// Framework's global symbols to the executable.  This only needs to be used in host apps
/// and should only be called once in the entire module.  The specific name given here is
/// intended to name the host app itself for use in logging output from the Carbonite
/// framework.
CARB_GLOBALS("example.stats");

  • Before the interface can be used, it must be acquired by the host app. This can be done in one of many ways with the Carbonite framework. The commonly suggested method is to use the carb::getCachedInterface<>() helper functions. These will be the quickest way to find and acquire an interface pointer since the result will be cached internally once found. If the framework unloads the plugin that a cached interface came from, it will automatically be reacquired on the next call to carb::getCachedInterface<>(). The particular usage of each interface differs. Please see each interface’s documentation for usage specifics.

  • Once the interface pointer has been successfully acquired, it may be used by the host app until the framework is shut down. If a particular implementation of an interface is needed and multiple are loaded, the default behavior is to acquire the ‘best’ version of the interface. This often comes down to the one with the highest version number.

  • If a specific version of an interface is required (ie: from a specific plugin), the interface could be acquired either using carb::Framework::acquireInterface<>() directly or using the second parameter to the carb::getCachedInterface<>() template to specify the plugin name. This allows a plugin name or a specific version number to be passed in.

    OMNI_CORE_INIT(argc, argv);

    stats = carb::getCachedInterface<carb::stats::IStats>();

    if (stats == nullptr)
    {
        fputs("ERROR-> failed to acquire the IStats interface.\n", stderr);
        return EXIT_FAILURE;
    }

  • Add a config file for the example app. This lets the Carbonite framework know which plugins it should try to find and load automatically and which default configuration options to use. This is specified in the TOML format. The only portion of it that is strictly important is the pluginsLoaded= list. In order for this config file to be picked up automatically, it must have the same base name as the built application target (ie: “example.stats” in this case) with the “.config.toml” extension and must be located in the same directory as the executable file.

# Specify the list of plugins that this example app depends on and should be found and
# loaded by the Carbonite framework during startup.
pluginsLoaded = [
    "example.carb.stats.plugin",
]

Building and Running the Example App

Building the example app is as easy as running build.bat (Windows) or build.sh (Linux). Alternatively, it can be built directly from within either VS Code (Linux or Windows) or MSVC (Windows).

The example app builds to the location _build/<platform>/<configuration>/example.stats<exe_extension>. This example app doesn’t require any arguments and can be run in one of many ways:

  • Directly on command line or in a window manager such as Explorer (Windows) or Nautilus (Ubuntu).

  • in MSVC on Windows by setting examples/example.stats as the startup app in the Solution Explorer window (right click on the project name and choose “Set As Startup Project”), then running (“Debug” menu -> “Start Debugging” or “Debug” menu -> “Start Without Debugging”).

  • Under VS Code on either Windows or Linux by selecting the example.stats launch profile for your current system.

Running the example app under the debugger and stepping through some of the code may be the most instructional way to figure out how different parts of the app and the example plugin work.

Interface Design Considerations

A keen observer may have noticed some weaknesses in the design of the carb::stats::IStats API. These could be overcome with some redesign of the interface or at least some more care in the implementation of the interface. Some things that could use improvements:

  • Only ABI safe types may be used in the arguments to Carbonite interface functions. These rules should be followed as a list of what types are and are not allowed:

    • The language primitives (ie: int, long, float, char, etc) and pointers or references to the primitive types are always allowed.

    • Locally defined structs and enums, pointers or references to locally defined types, or enums are always allowed.

    • ABI safe built-in types in Carbonite such as omni::string are always allowed.

    • All parameter data types must be plain-old-data using the standard layout.

    • Variadic arguments may be used sparingly if needed.

    • C++ class objects should never be passed into an interface function. They may be returned from one as an opaque pointer as long as the only way to operate on it is for the caller to pass it back into another interface function.

    • No built-in Carbonite helper class types should be passed into or returned from interface functions except for the ones that are explicitly marked as being ABI safe such as omni::string or carb::RString.

    • If an ABI unsafe type (ie: STL container) is to be used, its use must be entirely restricted to inlined code in the header. The object itself (or anything containing it) should never be passed into or returned from an interface function.

    No Carbonite interface function or any struct passed to it should ever use an ABI unsafe type such as an STL object, or anything from a third party library. Any types or enums from third party libraries should be abstracted out with locally defined Carbonite types.

  • Instead of using the carb::stats::Value struct, a carb::variant::Variant could be used. This would allow for a wider variety of data types and give an easier way to specify both the data’s type and its value.

  • The carb::stats::IStats::getValue()`` function has some thread safety (and general usability) concerns. The name and description strings it returns in the output parameter could be invalidated at any time if another thread (or even the same calling thread later on) removes that particular statistic from the table while it’s still being used. This puts an undue requirement for managing all access to the statistics table on the host app. This could be improved in one of the following ways:

    • Change the strings to be omni::string objects instead. This would allow for an ABI safe way of both retrieving and storing the strings for the host app. One consideration would be that the memory may need to be allocated and the strings copied on each access.

    • Change the object stored internally in the table to be reference counted. This would prevent the strings from being deallocated before any caller was done with accessing it. The down side would be that the caller would then be required to release the value object each time it was finished with it.

    • Change the string storage to be in a table that persists for the lifetime of the plugin (or use carb::RString which essentially does the same thing), but at process scope. This would make the retrieved strings safe to access at any point, but could also lead to the table growing in an unmaintainable way with certain usage patterns.

  • If an interface function needs to return a state object to be operated on by other interface functions, it must do so by using an opaque pointer to the object. The caller may not be allowed to directly inspect or operate on the returned object. Only other functions in the same interface or another cooperative interface in the same plugin may know how to access and manipulate the object. A state object from one interface may never be passed to an interface in another plugin to be operated on. An example of an plugin that makes a lot of use of state objects is carb.audio.plugin. That plugin implements multiple interfaces that all

Documentation on an interface is always very important. This is how other developers will discover and figure out how to use your shiny new interface. All public headers should be fully documented:

  • The ideal documentation is such that any possible questions about a given function or object’s usage has at least been touched on.

  • All function documentation should mention any thread safety concerns there may be to calling it, all parameters and return values must be documented.

  • All interfaces should have overview documentation that covers what it does and roughly how it is intended to be used.

  • All API documentation should be in the Doxygen format.

  • No matter how straightforward it is, code _never_ documents itself.

Expanding the Interface Later

Once an interface has been released into the wild and has been used in several places, it becomes more likely that users will request bug fixes, improvements, or changes to the functionality. When doing so, there are a few guidelines to follow to ensure old software doesn’t break if it hasn’t updated to the latest version of the interface’s plugin yet:

  • When adding new functions to the interface, always add to the end of the struct. Software that is still expecting to use an older version of the interface will still work with the newer versions as long as it hasn’t broken the existing ABI at the start of the struct. If a new function is added in the middle of the struct or an existing function is removed, all functions in the interface will be shifted up or down in the vtable of any software expecting to an older version of the interface.

  • Never change parameters or return values on any existing functions in a released interface when making changes to it. This could cause code build for an older version of the interface to behave erratically or incorrectly when calling into those interface functions. Instead, add a new version of the function with the different parameters or return values at the end of the interface struct.

  • When making any change to an interface, its version numbers should be bumped. A minor change that doesn’t break backward compatibility with older versions should simply increment the minor version number. If a change needs to occur that may break older software including deprecating some functions, the major version should be incremented and the minor version reset to zero.