ABI Compatibility
Two binaries (modules, executables, etc.) must have agreement on how data is exchanged between them. While programmers typically think of this in terms of an API (Application Programming Interface) in source code, the hardware requires a binary interpretation of this. This is called the Application Binary Interface.
The ABI therefore is the binary contract between two entities (generally modules) of how functions are called and data is exchanged.
This document is written from the perspective of Carbonite C++ development. Other languages may have similar concepts but vary in implementation.
Importance
Each C/C++ source file (compilation unit) is processed by the compiler to form an object file. These object files are then combined together by a linker to form a binary (executable, static library, dynamic library, etc.). All of these items–object files, executables, static/dynamic libraries–can be generated at different times yet linked together, they must agree on how functions exchange data.
When using plugins as dynamic libraries as Carbonite does, it is monumentally important to ensure that changes can be made to both plugins and the Framework in such a way that ensures backwards compatibility. This allows the API to grow and evolve, but allows binaries built at different times to function correctly.
A similar topic to ABI is that of Interoperability, often shortened to interop. Interop is the ability for two different languages to be able to call functions and exchange data between themselves. Some languages are able to call functions directly with a C calling convention, whereas other languages may require a binding layer–code that is generated to translate function calls and data.
Terminology
“API”
Application Programmer Interface
This is a programmer’s contract between two entities that describes how functions are called and data is exchanged. This contract involves the names of functions that can be called, the parameters and return values that are passed, and other concepts such as atomicity and global state that are considered meta-concepts and not enforced specifically in the code.
“ABI”
Application Binary Interface
Defined above to be the binary contract between two entities (generally modules) of how functions are called and data is exchanged.
This is similar to an “API”, but is a contract at the machine level, describing items such as calling convention, parameter
type, count and order, structure size and binary layout, enum values, etc. Whereas a programmer’s API might think of a function
parameter as an int
, the ABI considers this value as a 32-bit machine word and how it is passed (stack or register)
with no inherent type safety, no concept of signed or unsigned, etc. While an API describes a struct
, the ABI considers
only the binary representation of the data. An API might describe a function by name, but to the ABI this is just an address.
“Breaking ABI”
In terminology below, “Breaking ABI” or “ABI-Breaking” means that a change is made that causes a compiled object with an understanding of the object’s binary layout before the change will no longer work correctly after the change.
Non-exhaustive examples of changes that break ABI (expounded further below):
Function calling convention
Function return type or parameters (including changing pass-by-value to pass-by-reference)
Ordering of members within a
struct
orclass
Type of a member (i.e. changing
size_t
tounsigned
)Offset or alignment (such as by inserting a member)
Size of a member (i.e. changing
char buffer[128]
tochar buffer[256]
)
Making these types of changes is acceptable for an InterfaceType
by increasing the major version parameter in
the CARB_PLUGIN_INTERFACE
macro usage by at least one.
Carbonite interfaces may support multiple versions in order to be backwards compatible.
“ABI Boundary”
An “ABI Boundary” in this document is a potentially fragile point in the code where an ABI-Breaking change may occur. Generally this represents the boundary of a module or executable, since that is the typical granularity of a binary that is built at a given point in time. Calling a function in a different binary module is considered calling across an ABI Boundary. If static libraries or object files are stored in source control and not rebuilt when linking a binary, functions in those object files or static libraries could be considered ABI Boundaries as well.
Any and all parameters, either passed to or returned from an object in a different binary is considered as crossing the ABI Boundary.
Inline functions do not cross a ABI Boundaries and the restrictions below do not apply.
“Semantically Compatible”
Certain changes are allowed to be made as they are semantically compatible with modules built prior
to the change. Carbonite Interfaces use semantic versioning to determine compatibility. The version of an interface is specified
in the CARB_PLUGIN_INTERFACE
macro usage.
Any ABI breaking change must be accompanied by a major version increase of at least one.
Any change that is semantically compatible with older modules built prior to the change must be accompanied by a minor version increased by at least one and retaining the same major version.
Carbonite interfaces may support multiple versions in order to be backwards compatible. In the case of a plugin supporting multiple versions the
version in the CARB_PLUGIN_INTERFACE
macro is the highest version supported.
“Interop”
As stated above, Interop (short for Interoperability) is the ability for two languages to exchange data and call functions between themselves. Many languages (such as Python and Rust) can call functions with a C calling convention, but cannot call functions with a C++ calling convention.
Generally for C++ types we require them to be trivially-copyable
and conform to StandardLayoutType at a bare minimum
to consider them Interop-Safe. Carbonite has the CARB_ASSERT_INTEROP_SAFE
to ensure this.
Interop Safety allows two languages to agree on the data layout of a type, but the code that modifies that type still needs to ensure atomicity, memory ownership and expected data. This is up to the programmer to implement.
Calling Convention
This ABI has many different levels to it. For instance, system calls to an Operating System such as Linux or Windows generally
require calls using the syscall
CPU instruction, but the OS defines a system calling convention–
a contract between the application and the OS that describes the registers or memory that correspond to function arguments and return
values.
Applications are made from the building blocks of object files, static and dynamic libraries, and executables. Since these components can be built at different times, yet need to work together, a calling convention is also used to form a contract in how these different pieces call functions and exchange data.
Different compilers on the same platform should agree on the same convention, which would allow a binary with objects compiled by GCC to call functions in a library of objects compiled with Clang.
C supports different types of platform-specific calling conventions specified per function, such as __cdecl
, __pascal
,
__stdcall
, __fastcall
, etc. The default for 32-bit x86 architecture was __cdecl
. The calling convention indicates
what registers (or where on the stack) arguments are placed in, where the return value is returned, and if the caller or
callee is responsible for cleanup.
Some applicable ABI references that discuss calling conventions:
System V AMD64 ABI (used by Linux and MacOS on x86_64)
Microsoft x64 Calling Convention (x86_64)
ARM AAPCS ABI (used by Linux and MacOS on aarch64)
While calling convention is typically not on the mind of a programmer writing an API, it does affect ABI and therefore should be considered.
Calling convention also comes into play with Interop. Many languages (such as Python and Rust) can call functions with a C calling convention,
but cannot call functions with a C++ calling convention. The distinction here is important: A C calling convention specifies how built-in types
(such as int
and float
) are passed, as well as pointer types (including const char*
-style strings) and even struct
-by-value types
(provided that they are standard layout and trivially copyable). More complicated C++ types (such as omni::string
and omni::function
) can
also be passed by value, but these types require additional specification on top of the C calling convention, such as who is responsible for
destruction and how copying works. This is handled by the C++ calling convention, which makes passing these types by value interop-unsafe.
Carbonite provides the macro CARB_ASSERT_INTEROP_SAFE
which ensures that a type is standard layout and trivially
copyable.
Note
Though references are not part of the C calling convention (as they are a C++ feature), the ABI for references is essentially the same as pointers. They are therefore ABI-safe and interop-safe.
Warning
Changing calling convention, return value or parameters of a function will break its ABI. Changing almost anything about a function will break ABI. Ironically, changing the name of a function will typically not break ABI, but will affect compilation (“API”). This is because source code refers to names whereas the compiled binary code typically does not.
Built-in Types
C/C++ types (e.g. int
, uint32_t
, size_t
, float
, etc.) are always ABI-safe as their characteristics are guaranteed
to never change. However, note that these may be different between platforms/architectures.
These types may be safely used as function parameters, return values, and members of structs and classes.
Pointers and References
Both pointers and references have a well-defined ABI described by the calling convention and are therefore safe to use as members, function parameters and return values provided that the types referenced or pointed to are also ABI safe.
Variadic Arguments
Declaring a function as having variadic arguments (e.g. log(const char* message, ...)
) has an ABI described by the calling
convention and is therefore safe, provided that the types passed through the variadic arguments are also ABI safe.
Endianness
Also typically not thought of by programmers, endianness should be considered
part of the ABI. This is the ordering of bytes in memory of a binary word. A hardware architecture is one specific endianness: big or little.
For big-endian hardware, a 32-bit hexadecimal word 0x01020304
would be represented in memory bytes in the same order: 01 02 03 04
.
For little-endian hardware, the bytes of 0x01020304
are stored backwards: 04 03 02 01
. Carbonite focuses mostly
on little-endian because all supported architectures are little-endian. However, Network Byte Order as specified by the TCP/IP
standard is big-endian. If a function changed to keep the same datatype, but instead required a parameter to be in network
byte order, this would break ABI.
Enum Values
Changing the values of existing enums is an ABI-breaking change for every function that uses them. Adding a new, previously-unassigned enum value is a semantically compatible change.
Struct/Class Layout
Changing the layout of a struct
or class
that is passed across an ABI boundary is likely to affect ABI. See Best Practices
below for some semantically-compatible methods of changing structs and classes.
Consider the following struct:
struct Version {
uint32_t major;
uint32_t minor;
};
This struct has two members: major and minor. It’s size is 8 bytes. Those 8 bytes are layed out in memory as follows:
minor (bytes 4 - 7)
|---------|
00000000 00 00 00 00 00 00 00 00 .. .. .. .. .. .. .. .. |........ |
\_________/
major (bytes 0 - 3)
|---------------------|
total size: 8 bytes
The binary layout of this object represents its ABI. Now consider what would happen if we change the class as follows:
struct Version {
uint32_t major;
char dummy; // Added field
uint32_t minor;
};
The binary representation of this object changes:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 .. .. .. .. |............ |
\__major__/ | (padding) \__minor__/
dummy
|__________________________________|
total size: 12 bytes
Notice how the object changed? The major member is still at the same size and location. However, the size of the struct
changed to 12 bytes instead of 8, and the minor member no longer starts at byte 4 but now at byte 8.
Any module that was compiled with an understanding of Version
in the top code block would not work properly with the
bottom block.
Therefore, we cannot change Version
in this manner without Breaking ABI.
Warning
Changing the type, size, order, or alignment of any member within a struct will break its ABI.
Inheritance
Changing inheritance or inheriting from multiple base classes may break ABI and is not recommended. Inheritance may be changed only if it does not affect the characteristics of any existing members.
Non-virtual Functions
Adding member functions that are non-virtual to an existing struct or class does not break ABI. Typically these functions
will be declared as inline
, otherwise they would be written into an object file (or static library) that must be linked
in order to operate. Functions declared as inline
that cannot actually be compiled in-line where called will be written
into any object files that reference them and then typically be coalesced down to a single function at link time.
Changing parameters, return values, or the body of a non-virtual member function also does not break ABI since this function is essentially copied into whatever module calls it. However, a module will not take advantage of any changes to this function until it is rebuilt with the changes.
Virtual Functions
Changing a class that does not have a v-table to add a v-table by adding a virtual function will break ABI as it causes all members of the class to change characteristics. If the class is a base class for other classes, it will break the ABI of all descendant classes.
Since virtual functions in a class are in order of their declarations, adding a new virtual function to the end of a final
class
(that already has virtual functions) is allowed; this is a semantically compatible change. It is very important that the functions
are added to the end of the declaration, and that the class/struct is final
so that it may not be inherited from.
Members
Members of a struct or class must themselves be ABI-safe in order for a struct or class to be considered ABI-safe.
Caution
It is not ABI-safe to change a struct that is contained as a member within another struct.
Constructor and Initialization
Adding a new type initializer or changing the existing type initializer is a non-ABI-breaking (allowed) change and can be done at any time. However, keep in mind that older modules will not have this change, so previous values of the initializer must be anticipated.
Likewise, adding a new inline constructor to a class is a non-ABI-breaking (allowed) change.
Standard Layout
Data that is exchanged across the ABI Boundary must conform to the C++ named requirements of a standard-layout type. This is often checked with the is_standard_layout type-trait.
Copying and Moving
Data that is passed by value across the ABI Boundary must conform to the C++ is_trivially_copyable type-trait.
C++ Runtime Library Types
C++ Runtime Library Types, such as std::chrono::time_point
, std::vector
, std::string
, std::optional
, std::unique_ptr
, std::function
, std::shared_ptr
, std::weak_ptr
, et al
have no guarantees about ABI safety; their layout could change with an update.
Carbonite considers passing C++ Runtime Library Types to be ABI unsafe.
However, it is acceptable to use these types within inline functions as they do not cross the ABI Boundary.
Warning
Do not pass any C++ Runtime Library Types (or pointers or references to them) across an ABI boundary.
Exceptions
Exceptions are generally C++ Runtime Library Types that inherit from std::exception and as such are not considered by Carbonite to be ABI safe.
It is recommended that all interface functions be declared as noexcept
.
Warning
Do not allow any exceptions to cross an ABI Boundary when unwinding.
ABI-Breaking Changes Summarized
This is a non-exhaustive summation of the ABI-breaking changes mentioned above. Do not do the things listed here without
an ABI-breaking InterfaceType
version change (major version change).
Changing Interface function calling convention, parameters, or return type
Reordering members within an
InterfaceType
Adding members to an
InterfaceType
not at the end (note: adding members to the end is a semantically compatible change)Changing types, size, alignment or other characteristics of members
Best Practices
Use Built-In Types
Built-in data types (i.e. char
, uint32_t
, size_t
, float
, etc.) are always ABI-safe.
Pointers and references are ABI-safe only if the type pointed-to or referenced is also ABI-safe.
Use ABI-Safe Complex Types
Carbonite provides some helper classes that are ABI-safe:
omni::function
- A replacement forstd::function
omni::string
- A replacement forstd::string
carb::RString
(and variations) - A fast string-interning mechanism
Note that not all Carbonite types are intended to be ABI-safe.
Warning
The omni::function
and omni::string
types, while ABI-safe, should only be passed by reference or pointer.
This is because these types are not trivially copyable. This allows maintaining a C calling convention and interop safety.
Add New Interface Functions to the End
Adding interface functions to the end of an InterfaceType
allows the change to be semantically-compatible (minor version change only).
Design Structs to be Changed
It can be advantageous to design structs for future change by including a version of some sort.
Many Windows API structs contain a member that must be initialized to the sizeof
the struct. This is an effort to plan
for future changes in a way that does not break ABI compatibility. New members can then be added to the end of the struct
(avoid removing members, resizing them or adding them anywhere but the end as this drastically complicates processing).
This technique can be used to change a struct in a way that is safe and is employed by Carbonite structures, such as carb::tasking::TaskDesc
and carb::AcquireInterfaceOptions
.
As the struct grows, the size naturally changes, so the size of the struct makes a decent version.
At the first point of the code where the struct is used, promote it to the latest version:
static std::optional<AcquireInterfaceOptions> versionPromote(const AcquireInterfaceOptions& opt)
{
if (CARB_LIKELY(sizeof(opt) == opt.sizeofThis))
{
return opt;
}
// Version promotion to current goes here. Initial size of the struct was 48 bytes.
CARB_LOG_ERROR("Unknown size of AcquireInterfaceOptions struct: %zu", opt.sizeofThis);
return std::nullopt;
}
void* FrameworkImpl::acquireInterface(AcquireInterfaceOptions opt_)
{
auto options = versionPromote(opt_);
if (!options)
return nullptr;
auto& opt = options.value();
// ...
When promoting, if the sizeof
field is not equal to the current sizeof
the struct, keep in mind that every member
with an offset after the sizeof
field is not initialized; reading it will result in undefined behavior.
It is also good practice to do the promote into a copy of the struct (remember from above that structs must be trivially copyable).
Warning
While it may seem like a good idea to pass a struct by value and promote it in-place, this is dangerous and can lead to
stack corruption. This is because of the calling convention: the space for the struct is allocated by the caller, so only
the space for the sizeof
struct is allocated. Writing to members beyond that will corrupt the stack. Instead, pass
the struct by const-reference and copy into a new local struct. Some older Carbonite APIs were written unaware of this
and pass the struct by value.
Warning
When working with arrays of structs, keep in mind that the size of passed-in structs might be different than the current
size, so standard pointer incrementing will not work. Instead, you will have to cast to uint8_t*
(a byte pointer) and
manually calculate the pointer offset.
Example of working with arrays of versioned structs:
void Scheduler::addTasks(TaskDesc* tasks, uint32_t numTasks, Counter* counter)
{
CARB_CHECK(counter != detail::kListOfCounters, "addTasks() does not support list of counters");
// Because the struct may be an older version (different size), we cannot increment the pointer itself. We need to
// handle it at the byte level.
uint8_t* p = reinterpret_cast<uint8_t*>(tasks);
while (numTasks--)
{
TaskDesc* pTask = reinterpret_cast<TaskDesc*>(p);
CARB_FATAL_UNLESS(pTask->size <= sizeof(TaskDesc), "Invalid TaskDesc size");
p += pTask->size;
addTask(*pTask, counter);
}
}
It is good practice to have unit tests for all versions of a struct to ensure that they promote correctly.
Remember to ensure that your struct conforms to C++ type-trait is_standard_layout.
Consider Thread Safety
While not strictly about ABI, it is good Interface design to consider thread safety. For example, this could lead to problems as you cannot atomically get the count and then fill the buffer (also the fill function doesn’t take a count for safety!):
struct Widget;
struct MyInterface {
CARB_PLUGIN_INTERFACE("MyInterface", 1, 0)
size_t(CARB_ABI* getCount)();
void(CARB_ABI* fillWidgets)(Widget** outWidgets);
};
Since no container is currently ABI-safe, one possibility might be to take a function/context:
using WalkWidgetsFn = void(Widget*, void*);
void (CARB_ABI* walkWidgets)(WalkWidgetsFn* fn, void* context);
// Call example:
std::vector<Widget*> widgets;
iWidget->walkWidgets([](Widget* w, void* context) {
static_cast<std::vector<Widget*>*>(context)->push_back(w);
}, &widgets);