OpenVDB  8.1.0
OpenVDB AX

AX C++ Documentation

The following documentation focuses on the C++ design and API of OpenVDB AX, not the front end AX language. For the latter, see the AX Language documentation. For more specific function API information, search for the relevant function doxygen.
Note
These documents are actively being worked on and are subject to change. See the API doxygen comments or contact us for more information.

Contents


Versioning, API and ABI

AX is a self contained library which currently has a one way dependency on OpenVDB; any downstream artifact that uses AX will need to build and link against both OpenVDB and OpenVDB AX. There are a few important factors to be aware of with this integration:
  • OpenVDB AX is tied to the OpenVDB library version. There is no explicit OpenVDB AX version. This includes the OpenVDB namespaced API, the shared library version suffix on relevant platforms and the OpenVDB ABI. It may be possible to build OpenVDB AX against a different version of OpenVDB however this is not explicitly supported.
  • There are no ABI compatibility guarantees for OpenVDB AX components i.e. AX components do not support the transfer of themselves as binary representations across dynamic library boundaries to applications where AX has been compiled at a different location in the OpenVDB repository. Support for this may change in the future.
In general, anything in the public namespace (defined as anything not labelled "internal") comes with standard API guarantees; that is, classes and methods will not change visible behaviour or signatures and deprecations will occur with appropriate replacements (where necessary) prior to removal. However, due to the nature of the project and its infancy, it's encouraged to continue reading below into the relevant AX components. These provide a better description of their intended public consumption and completeness.

OpenVDB AX Repository

The AX repository has been laid out using a modular design. Each subdirectory encompasses a set of similar tools with a fairly consistent (one-way) dependency diagram to other tools in the repository. The following provides a short description of each directory, listed in this ad-hoc dependency order (from highest to lowest):
  • grammar: Provides the flex/bison lexer/grammar rules for generating the OpenVDB AX language, as well as the pre-generated .c files.
  • ast: The definition of an AX Abstract Syntax Tree, its node types and methods to work explicitly with them.
  • compiler: The frontend C++ user interface for building and using AX.
  • codegen: The backend LLVM IR code generation for AX.
Additional tools include (in no particular order):
  • cmd: Command line binary tools for AX.
  • math: Various mathematical methods, typically used by the AX code generators when building native functions.
  • test: Unit testing framework for AX.

The Grammar

The subdirectory grammar contains the rules used to generate an AX abstract syntax tree, as defined by the AX Language specification. These files do not provide an API per-se and are primarily used internally by the ast module. The grammar syntax rules are specified in files with .y suffix and the tokenziers are specified by the .l suffix. These files are designed to work with two UNIX tools, GNU Bison and Flex. Together these tools provide a workflow for developers to create develop a wide range of language parsers. It's encouraged to visit the corresponding manuals for more information on their use.
The AX grammar is defined with a LR parser. In simplest terms, this means that the syntactical rules define a grammar which is parsed bottom-up, left-to-right, deterministically and context free (for the most part). These are fairly typical traits of parsers used to define computer languages. The parser is not designed to be interactive, nor reentrant (is not thread safe, each string is parsed sequentially) however such limitations are not imposed by AX and may be removed in the future. Note that the internal function symbols are correctly prefixed (with ax), so AX should be compatible with other software that also uses Bison/Flex.
The .y and .l files are typically not part of the normal build process. Bison and Flex use them to generate compatible C/C++ code, however this code is always pre-generated and provided with the AX repository within the grammar/generated subdirectory. This C/C++ code is compiled into the AX library. Thus, the grammar is provided for two reasons; to demonstrate how the syntactical rules are implemented and to provide expert users the ability to regenerate the C/C++ bindings (see the build instructions). Importantly, AX does not install any of the files in the grammar folder; the parser is accessed through the ast module.

The AST

The ast module provides:
All methods in this module are designed to work around an AX AST, which represents the syntactical constructs of the AX language as a hierarchy of C++ objects (nodes). The complete AST hierarchy is visualized by the doxygen class graph for an openvdb::ax::ast::Node, which represents the top most node in the AST. The AST API exists in a nested namespace (openvdb::ax::ast) and can be used by clients for more granular control and modifications to an AST. Importantly, the AST API contains the openvdb::ax::ast::parse() method, which invokes the C/C++ functions built by the grammar to iteratively construct an AST from a provided character string.
A number of other tools are provided by the module to work with an AST such as tools to re-interpret the AST back to AX compatible code, printing of the AST's node hierarchy/layout and scanners which are able to query details from AST branches. All these methods use the AX visitor framework, a visitor pattern traversal class strongly influenced by Clang's RecursiveVisitor. Clients can use the AX visitor to implement their own AST analysis or modifications (see vdbaxextend).

Scanners

todo

The Compiler Pipeline

The compiler subdirectory contains the bulk of the C++ API intended to be used directly by clients of OpenVDB AX. It links together the grammar, AST and code generation to produce AX executable objects, forming a pipeline from an initial string of AX code to final execution across some geometry. The following sections detail the core components to the Compiler pipeline.

Logging

Anyone familiar with programming languages will undoubtedly be aware of the huge variety and quantity of feedback that's presented to clients during use. This information, when properly presented, can be invaluable and help to advise users during their usage of the language. AX provides a Logger class which can be used throughout the Compiler pipeline to store and report this information. By default, this class reports all errors to std::cerr and swallows warnings, but can be customised depending on your needs e.g:
// Create a logger which sends errors and warnings to std::cerr
openvdb::ax::Logger
logs([](const std::string& msg) { std::cerr << msg << std::endl; },
[](const std::string& msg) { std::cerr << msg << std::endl; });
logs.setMaxErrors(5); // stop reporting after 5 errors
logs.setWarningsAsErrors(false); // don't count warnings as an error
logs.setPrintLines(true); // print code lines
logs.setNumberedOutput(true); // number each error/warning reported
A Logger should be passed to the relevant pipeline methods as detailed in the following sections.

Executables

The "executable" terminology comes from the binary representation of the compiled function which is invoked. AX executables essentially wrap the generated function calls from the AX codegen with an efficient multi-threaded invoke for the type of geometry being processed. The executable classes are designed to be lightweight to store, modify and copy with a consolidated interface for running and customising execution.
There are two main types of executables; the VolumeExecutable, designed to work with all OpenVDB Volumes and the PointExecutable which works with OpenVDB PointDataGrids. They are not expected to be created directly; instead these objects are returned by the AX Compiler depending on the selected Compiler::compile() function.
Note
There is no inherent limitation to the design of the VolumeExecutable which stops it working on PointDataGrids but it's not particular useful in it's current form and can be confusing so it is explicitly disallowed.
The executables are distinctly detached from the Compiler once they have been built and can be safely used no matter previous or subsequent compilations of AX code. There are a variety of settings on the executables to control runtime behaviour - below demonstrates an example of using this API:
using namespace openvdb;
ax::Compiler compiler; // compiler
a.setName("a");
a.tree().setValueOn({0,0,0}, 1.0f); // set single coordinate to active 1.0f
const ax::VolumeExecutable::Ptr exe1 =
compiler.compile<ax::VolumeExecutable>("f@a += 1.0f;");
exe1->execute(a); // run over active leaf voxels in parallel
ax::VolumeExecutable::Ptr exe2 =
compiler.compile<ax::VolumeExecutable>("f@a += 2.0f;");
// set to process all voxels
exe2->setValueIterator(ax::VolumeExecutable::IterType::ALL);
exe2->execute(a); // run over active and inactive leaf voxels in parallel
std::cout << a.tree().getValue({0,0,0}) << std::endl; // prints 4.0f
std::cout << a.tree().getValue({1,0,0}) << std::endl; // prints 2.0f
Executable Thread Safety
The executables are responsible for launching the compiled AX kernels and may internally use multiple threads. Multiple executables of any type can co-exist and be executed concurrently with unique arguments. For example:
FloatGrid a1, a2; // assume both are named "a"
auto exe1 = compiler.compile<ax::VolumeExecutable>("f@a += 1.0f;");
auto exe2 = compiler.compile<ax::VolumeExecutable>("f@a += 2.0f;");
std::thread t1([&]() { exe1->execute(a1); } );
std::thread t2([&]() {
exe1->execute(a2); // safe even if exe1::execute is being used by another thread
exe2->execute(a2); // safe, can be running alongside another executable
});
This is, however, only safe depending on the access pattern of the AX code, the iteration patterns of the executables and the grid data being fed to the execute methods. When we talk about the thread safety of these classes we are referring to the invocation of their execution method with the same argument data by multiple threads, not how many threads the executables themselves use (which can be configured by the grainSize() setting). Importantly, for a given VDB grid or VDB Points attribute "foo":
  • For The VolumeExecutable it is safe to call VolumeExecutable::execute from multiple threads only if:
    • All relevant AX kernels do not write to grid foo.
  • For The PointExecutable it is safe to call PointExecutable::execute from multiple threads only if:
    • All relevant AX kernels do not write to attribute foo.
    • All relevant AX kernels do not create any attributes or groups.
    • All relevant AX kernels do not modify the position "P" attribute.
For example:
auto exe = compiler.compile<ax::VolumeExecutable>("@a = @b");
// Grid a1 and a2 called "a", b called "b". Grid b is given to both threads.
// This is safe as we guarantee b is only read from
std::thread t1([&]() { GridPtrVec v{a1,b}; exe1->execute(v); } );
std::thread t2([&]() { GridPtrVec v{a2,b}; exe1->execute(v); } );
You can use the tools provided in Scanners.h to query data accesses on AX ASTs to verify access patterns of your data.

Custom Data

The AX language provides two main tokens to access memory which has not been allocated by AX itself. The first token, @ (the "at" symbol), is designed to provide read/write access to geometry attributes. The second token is the dollar character $. This allows read only access to custom data that can be modified in C++ i.e. outside the scope of an AX program. In the same way as the @attribute syntax, the value of this data can change without the need for AX to re-compile the program. It is, however, solely on the C++ clients of AX to make sure that this data is setup correctly when integrating AX into their applications.
Consider this trivial example:
openvdb::v8_1::FloatGrid
Grid< FloatTree > FloatGrid
Definition: openvdb.h:45
openvdb::v8_1::points::index::ALL
@ ALL
Definition: IndexIterator.h:43
Logger.h
Logging system to collect errors and warnings throughout the different stages of parsing and compilat...
openvdb::v8_1::GridPtrVec
std::vector< GridBase::Ptr > GridPtrVec
Definition: Grid.h:514
openvdb
Definition: openvdb/Exceptions.h:13