Merge branch 'REMIX-4240-component-versioning' into 'main'

REMIX-4240 Add handling for renaming components and their properties.

Closes REMIX-4240

See merge request lightspeedrtx/dxvk-remix-nv!1671

Author: MarkEHenderson

documentation/ComponentsAndGraphs.md

@@ -0,0 +1,269 @@
+# Components and Graphs
+
+# ⚠️ Under Construction - not yet ready for use. ⚠️
+
+When creating content to remix games, modders often want to make the replacement content react to the state of the game. The Component system has been created as a way to flexibly detect state and apply changes.
+
+Each individual component represents a small, discrete piece of functionality. These components can be connected together, so that data generated by one component can set the input parameters of another component. This forms a **Graph**, with each component being a single node in that graph. These graphs can create complex and reactive behaviors.
+
+Note that the graphs created must be **Directed Acyclic Graphs (DAGs)** - which means:
+- There must be at least one starting component (with no dependencies)
+- There can be no dependency loops (no component can depend on itself, either directly or indirectly)
+
+## Creating Graphs
+
+Graph creation should be done in the Remix Toolkit.  The graph editor there is still in development.
+
+## Types of Components
+
+There are three major categories of components:
+
+### Trigger Components
+
+These components generate output values by examining the renderable state or generating values based on time. Examples include:
+* Detecting if a given mesh or material hash is used this frame
+* Detecting if the camera is inside some bounding box
+* Animating a number as time passes
+
+### Transform Components
+
+These components assist with mapping outputs to inputs with different types or ranges. Examples include:
+* Adding two values together
+* Outputting a boolean if some value is greater than another value
+* Combining floats into a vector
+* Converting between different data types
+* Applying mathematical operations
+
+### Override Components
+
+These components alter the renderable state. Examples include:
+* Altering configuration
+* Overriding the state of renderable objects such as Lights or Meshes
+* Swapping which replacements are applied to a given mesh hash
+* Altering the time multiplier (pausing or slowing all animations)
+* Modifying material properties
+
+## Component Data Types
+
+The component system supports the following data types as defined in `rtx_graph_types.h`:
+
+| Type | C++ Type | Description | Example Values |
+|------|----------|-------------|----------------|
+| `Bool` | `uint8_t` | Boolean values (stored as uint8_t for variant compatibility) | `true`, `false` |
+| `Float` | `float` | Single precision floating point | `1.0f`, `-3.14f` |
+| `Float2` | `Vector2` | 2D vector of floats | `Vector2(1.0f, 2.0f)` |
+| `Float3` | `Vector3` | 3D vector of floats | `Vector3(1.0f, 2.0f, 3.0f)` |
+| `Color3` | `Vector3` | RGB color (3D vector) | `Vector3(1.0f, 0.5f, 0.0f)` |
+| `Color4` | `Vector4` | RGBA color (4D vector) | `Vector4(1.0f, 0.5f, 0.0f, 1.0f)` |
+| `Int32` | `int32_t` | 32-bit signed integer | `42`, `-100` |
+| `Uint32` | `uint32_t` | 32-bit unsigned integer | `42`, `1000` |
+| `Uint64` | `uint64_t` | 64-bit unsigned integer | `42ULL`, `1000000ULL` |
+| `Prim` | `uint32_t` | USD prim identifier | `101`, `102` |
+
+## Creating Components
+
+Before creating a new component, check the list of existing components to see if one already exists that matches your needs:
+[Component List](components/index.md)
+
+Components are created in C++, using macros. The [TestComponent](https://github.com/NVIDIAGameWorks/dxvk-remix/blob/main/tests/rtx/unit/graph/test_component.h) is a good example component using every data type.
+
+Defining a component has three mandatory pieces: the parameter lists, the component definition, and the `updateRange` function.
+
+### Defining Parameters
+
+Parameters are how components accept Input, store internal State, and expose Output. Each of these are defined in separate lists using macros.
+
+#### Input Parameters
+Inputs can be set to a constant value or connected to another component's Output. They should be read-only during a component's update function.
+
+```cpp
+#define LIST_INPUTS(X) \
+  X(RtComponentPropertyType::Bool, false, inputBool, "Input Bool", "An example of a boolean input parameter") \
+  X(RtComponentPropertyType::Float, 1.f, inputFloat, "Input Float", "An example of a float input parameter") \
+  X(RtComponentPropertyType::Float3, Vector3(1.f, 2.f, 3.f), inputFloat3, "Input Float3", "An example of a Float3 input parameter")
+```
+
+#### State Parameters
+State values are not shared with any other components and can be used however the component needs. They persist between updates.
+
+```cpp
+#define LIST_STATES(X) \
+  X(RtComponentPropertyType::Bool, false, stateBool, "", "An example of a Bool state parameter") \
+  X(RtComponentPropertyType::Float, 2.f, stateFloat, "", "An example of a Float state parameter")
+```
+
+#### Output Parameters
+Output values can be read by other components but should only be set by the owning component.
+
+```cpp
+#define LIST_OUTPUTS(X) \
+  X(RtComponentPropertyType::Bool, false, outputBool, "Output Bool", "An example of a Bool output parameter") \
+  X(RtComponentPropertyType::Float, 3.f, outputFloat, "Output Float", "An example of a Float output parameter")
+```
+
+#### Parameter Options
+
+Each parameter can have optional properties set after the docString:
+
+```cpp
+X(RtComponentPropertyType::Float, 1.f, inputFloat, "Input Float", "test for Float", \
+  property.minValue = 0.0f, property.maxValue = 10.0f, property.optional = true)
+```
+
+Available options include:
+* `property.minValue` / `property.maxValue` - Range constraints (currently UI hints only)
+* `property.optional` - Whether the component functions without this property being set
+* `property.oldUsdNames` - For backwards compatibility when renaming properties
+* `property.enumValues` - For displaying as an enum in the UI
+
+#### Enum Values Example
+
+The `property.enumValues` option allows you to define a set of named values for a property, which will be displayed as a dropdown in the UI. This is particularly useful for properties that should only accept specific predefined values.
+
+First, define your enum class:
+
+```cpp
+enum class LightType : uint32_t {
+  Point = 0,
+  Spot = 1,
+  Directional = 2,
+  Area = 3,
+};
+```
+
+Then use it in your parameter definition:
+
+```cpp
+#define LIST_INPUTS(X) \
+  X(RtComponentPropertyType::Uint32, 0, lightType, "Light Type", "The type of light to create", \
+    property.enumValues = { \
+      {"Point", {LightType::Point, "Omnidirectional point light"}}, \
+      {"Spot", {LightType::Spot, "Directional spot light with falloff"}}, \
+      {"Directional", {LightType::Directional, "Infinite directional light"}}, \
+      {"Area", {LightType::Area, "Area light with physical size"}} \
+    })
+```
+
+The enum values are stored as the underlying type (e.g., `uint32_t` for `LightType`) but displayed with user-friendly names and descriptions in the UI.
+
+### Defining the Component
+
+Invoke the `REMIX_COMPONENT` macro to define your component. Make sure the component name uses UpperCamelCase.
+
+```cpp
+REMIX_COMPONENT( \
+  /* the Component name */ MyComponent, \
+  /* the UI name */        "My Component", \
+  /* the UI categories */  "animation,transform", \
+  /* the doc string */     "A component that does something useful.", \
+  /* the version number */ 1, \
+  LIST_INPUTS, LIST_STATES, LIST_OUTPUTS);
+```
+
+### Defining the `updateRange` function
+
+The `updateRange` function is responsible for updating a batch of components and will usually take the form:
+
+```cpp
+void MyComponent::updateRange(const Rc<DxvkContext>& context, const size_t start, const size_t end) {
+  // Update each component instance in the batch
+  for (size_t i = start; i < end; i++) {
+    // Read input values:
+    if (m_inputBool[i]) {
+      // Use state values
+      m_stateFloat[i] = m_stateFloat[i] + 1.0f;
+    }
+    
+    // Write output values:
+    m_outputFloat[i] = m_stateFloat[i];
+  }
+}
+```
+
+**Important Notes:**
+* Graph updates usually happen between rendering frames, so any state being read will be from frame N, but any changes the graph makes will happen on frame N+1
+* The exception is the first frame the graph exists on - it will be updated once immediately on creation to avoid rendering any default values
+* Always iterate from `start` to `end` to process the correct range of component instances
+* Access properties using the `m_` prefix and array indexing `[i]`
+
+## Component Versioning
+
+Great care needs to be taken when changing components that are already in use to avoid breaking already published data. When updating existing components, follow these guidelines.
+
+### Not a Versioned Change
+
+Making any of these changes does not require incrementing the version number in the Component Definition:
+
+* Change the UI name or tooltip of a property
+* Change the name of a component
+  * Make sure to add the old name to the component definition as `spec.oldNames = {"OriginalName"}`. See `test_component.h` for example.
+* Fix logical bugs in the component
+* Make algorithmic changes in the component
+  * i.e., change the way the component works, but not the inputs or outputs
+
+### Non-Breaking Change
+
+Making these changes requires bumping the version number and will prevent newly defined copies of this component from working in older runtimes.
+
+* Change a property name
+  * Make sure to add the old name to the property definition as `property.oldUsdNames = {"oldName"}`
+  * If a Component's USD prim defines multiple versions of the same property (due to layering or incorrect data migration), the strongest version will be used, as defined by:
+    * Whichever version is defined on the strongest layer
+    * If the strongest layer has multiple versions, then prefer the newest name
+    * If the strongest layer does not have the newest name, use the earliest name in `oldUsdNames`
+  * That means that if there are multiple `oldUsdNames`, then the oldest name should be on the right:
+    ```cpp
+    property.oldUsdNames = {"oldName", "olderName", "originalName"}
+    ```
+* Add a new property
+* Change the type or contents of a state property
+
+### Breaking Change
+
+Rather than making these changes to an existing component, a new component should be made (possibly by copying the old component and changing the name).
+
+* Change the type of an input or output property
+  * We can't automatically convert a type that is connected to another component
+* Change the contents of an input or output property (i.e., float to unorm)
+  * This would need to convert the connected values every frame. Better to just use a different update function.
+* Remove an input or output property
+* Repurpose a component name to a different component type
+  * Once a name maps to a bit of functionality, that pairing should be permanent.
+
+**Note:** The UI name for a component can be changed, so while the new copy could use the original UI name, the original component's UI name could be changed to "My Component (deprecated)".
+
+Also note that ideally, the old component and new component should share as much code as possible via helper functions.
+
+## Component Registration
+
+Components are automatically registered when they are defined using the `REMIX_COMPONENT` macro, and the header file is added to `rtx_component_list.h`. The registration system:
+
+1. Generates a unique hash for each component type
+2. Stores the component specification for runtime lookup
+3. Enables automatic generation of documentation and schemas
+
+## Component Schema
+
+Remix Components are using a subset of the OmniGraph system to enable the Toolkit UI and USD encoding.  It's important to note that while we expose .ogn schema for Remix Components, they are not functional OmniGraph nodes, and the Remix Runtime does not support non-remix OmniGraph components.
+
+## Graph Execution
+
+Components in a graph are executed in topological order based on their connections. The system:
+
+1. Determines the execution order to ensure all inputs are available before components that depend on them
+2. Updates components in batches for performance
+3. Calls `updateRange` on each component batch with the appropriate start/end indices
+4. Optionally calls `applySceneOverrides` for components that need to modify the scene directly
+
+Note that graphs are batched by a topological hash.  This means that large numbers of graphs that have the same component and connections (but different input values) can be updated very performantly.  If a graph has a different component or connection, it will be part of a separate batch.
+
+## Best Practices
+
+1. **Keep components focused**: Each component should do one thing well
+2. **Use meaningful names**: Component and property names should clearly indicate their purpose
+3. **Document thoroughly**: Provide clear docStrings for all components and properties
+4. **Test thoroughly**: Components should be tested with various input combinations
+5. **Consider performance**: Batch operations efficiently and avoid unnecessary computations
+6. **Plan for versioning**: Design components with future changes in mind.  Consider using Enums instead of booleans 
+7. **Use appropriate data types**: Choose the most specific type that fits your needs
+8. **Handle edge cases**: Consider what happens with invalid or missing inputs

src/dxvk/rtx_render/graph/rtx_graph_component_macros.h

@@ -82,6 +82,15 @@
   { \
     RtComponentPropertySpec& property = s_spec.properties[index]; \
     __VA_ARGS__; \
+    if (!property.oldUsdNames.empty()) { \
+      std::string prefix = "inputs:"; \
+      if (property.ioType == RtComponentPropertyIOType::Output) { \
+        prefix = "outputs:"; \
+      } \
+      for (std::string& oldName : property.oldUsdNames) { \
+        oldName = prefix + oldName; \
+      } \
+    } \
     index++; \
   } \
 
@@ -93,7 +102,7 @@
 
 #define REMIX_COMPONENT_WRITE_STATIC_SPEC(componentClass, uiName, categories, docString, versionNumber, X_INPUTS, X_STATES, X_OUTPUTS, ...) \
   static std::once_flag s_once_flag; \
-  static const std::string s_fullName = "lightspeed.trex.components." #componentClass; \
+  static const std::string s_fullName = RtComponentPropertySpec::kUsdNamePrefix + #componentClass; \
   static RtComponentSpec s_spec = { \
     { \
       X_INPUTS(REMIX_COMPONENT_WRITE_PROPERTY_SPEC_INPUT) \

src/dxvk/rtx_render/graph/rtx_graph_types.cpp

@@ -143,14 +143,25 @@ void registerComponentSpec(const RtComponentSpec* spec) {
 
   std::lock_guard<std::mutex> lock(getComponentSpecMapMutex());
 
-  // Check for duplicate registration
-  auto existing = getComponentSpecMap().find(spec->componentType);
-  if (existing != getComponentSpecMap().end()) {
-    Logger::err(str::format("Component spec for type ", spec->name, " already registered. Conflicting component spec: ", existing->second->name));
+  // Check for duplicate registration and insert
+  auto [iterator, wasInserted] = getComponentSpecMap().insert({spec->componentType, spec});
+  if (!wasInserted) {
+    Logger::err(str::format("Component spec for type ", spec->name, " already registered. Conflicting component spec: ", iterator->second->name));
     assert(false && "Multiple component specs mapped to a single ComponentType.");
   }
 
-  getComponentSpecMap()[spec->componentType] = spec;
+  if (!spec->oldNames.empty()) {
+    for (const auto& oldName : spec->oldNames) {
+      std::string fullOldName = RtComponentPropertySpec::kUsdNamePrefix + oldName;
+      RtComponentType oldType = XXH3_64bits(fullOldName.c_str(), fullOldName.size());
+      auto [oldIterator, oldWasInserted] = getComponentSpecMap().insert({oldType, spec});
+      if (!oldWasInserted) {
+        Logger::err(str::format("Component spec for legacy type name ", fullOldName, " already registered. Conflicting component spec: ", oldIterator->second->name));
+        assert(false && "Multiple component specs mapped to a single ComponentType.");
+      }
+    }
+  }
+
 }
 
 const RtComponentSpec* getComponentSpec(const RtComponentType& componentType) {

src/dxvk/rtx_render/graph/rtx_graph_types.h

@@ -152,6 +152,8 @@ using RtComponentType = XXH64_hash_t;
 static const RtComponentType kInvalidComponentType = kEmptyHash;
 
 struct RtComponentPropertySpec {
+  static inline const std::string kUsdNamePrefix = "lightspeed.trex.components.";
+  
   RtComponentPropertyType type;
   RtComponentPropertyValue defaultValue;
   RtComponentPropertyIOType ioType;
@@ -165,11 +167,19 @@ struct RtComponentPropertySpec {
   // To set optional values when using the macros, write them as a comma separated list after the docString. 
   // `property.<name> = <value>`, i.e. `property.minValue = 0.0f, property.maxValue = 1.0f`
 
+  // If this property has been renamed, list the old `usdPropertyName`s here for backwards compatibility.
+  // If multiple definitions for the same property exist, the property on the strongest USD layer will be used.
+  // If multiple definitions for the same property exist on a single layer, `name` will be used first,
+  // followed by the earliest name in `oldUsdNames`.  So the ideal order should be:
+  // property.oldUsdNames = { "thirdName", "secondName", "originalName" }
+  std::vector<std::string> oldUsdNames;
+
   // NOTE: These are currently unenforced on the c++ side, but should be used for OGN generation.
   // TODO: consider enforcing these on the c++ side (between component batch updates?)
   RtComponentPropertyValue minValue = false;
   RtComponentPropertyValue maxValue = false;
 
+
   // Whether the component will function without this property being set.
   // Runtime side all properties have a default value, so this is mostly a UI hint.
   bool optional = false;
@@ -212,6 +222,8 @@ struct RtComponentSpec {
   // Optional functions for component batches.  Set these by adding a lambda to the end of the component definition macro:
   // `spec.applySceneOverrides = [](...) { ... }`
 
+  // If this component has been renamed, list the old `name`s here for backwards compatibility.
+  std::vector<std::string> oldNames;
 
   // Optional function intended for applying values in the graph to renderable objects.  This is called near the top of SceneManager::prepareSceneData.
   std::function<void(const Rc<DxvkContext>& context, RtComponentBatch& batch, const size_t start, const size_t end)> applySceneOverrides;