Herostratus’ legacy

words from a lazy coder

Notes on hacking gfxreconstruct to enable H.265 video encoding

While working on Vulkan Video encode support in Mesa, I needed to capture H.265 encoding traces. gfxreconstruct already handled H.264 video and several other extensions, but VK_KHR_video_encode_h265 was explicitly blocked. Here’s how I unblocked it and what I learned about gfxreconstruct’s code generation machinery along the way.

gfxreconstruct is LunarG’s suite of tools for capturing and replaying graphics API calls. It intercepts Vulkan (and D3D12) calls at the layer level, serializes them into a compressed binary trace file, and can later replay that trace verbatim. This is useful for driver regression testing, GPU bring up, architecture simulation, and bug reporting.

The blocking mechanism #

gfxreconstruct has two independent mechanisms that prevent an extension from being captured.

The first is a runtime blocklist in framework/encode/vulkan_entry_base.cpp. A static array called kVulkanUnsupportedDeviceExtensions lists extension name strings that the layer strips from vkEnumerateDeviceExtensionProperties results. If your extension is on that list, applications cannot even see it when the capture layer is loaded, so they never attempt to use it and nothing gets recorded.

The second is a generation-time exclusion list in the Python code generator. gfxreconstruct does not hand-write capture and replay handlers for each Vulkan function. Instead, it parses the Khronos XML registry (vk.xml, video.xml) and auto-generates thousands of lines of C++ for encoding, decoding, and consuming API calls. The generator has exclusion lists that tell it which extensions and struct families to skip entirely.

VK_KHR_video_encode_h265 was on both lists.

Removing the runtime block #

This part is trivial: open framework/encode/vulkan_entry_base.cpp and delete the line

VK_KHR_VIDEO_ENCODE_H265_EXTENSION_NAME,

from kVulkanUnsupportedDeviceExtensions. One line. After rebuilding, the layer reports the extension to applications. But that is not enough: without generated capture/replay code, intercepted calls would have no handlers.

Removing the generator exclusions #

The code generator lives under framework/generated/. The file khronos_generators/vulkan_generators/vulkan_base_generator.py maintains two exclusion lists:

From _remove_extensions I removed:

VK_KHR_video_decode_h265
VK_KHR_video_encode_h265
VK_KHR_video_maintenance2

From _remove_video_extensions I removed:

vulkan_video_codec_h265std
vulkan_video_codec_h265std_decode
vulkan_video_codec_h265std_encode

I included VK_KHR_video_maintenance2 because its structs interact with video session parameters relevant to H.265 sessions.

The XML patch: when the generator breaks #

After removing the exclusions, I ran the generator and hit an error. The generator could not resolve a len attribute in video.xml for StdVideoH265HrdParameters.

The problematic members are pSubLayerHrdParametersNal and pSubLayerHrdParametersVcl. Both are pointers, and video.xml specifies len="*_max_sub_layers_minus1 + 1" for them. The expression references *_max_sub_layers_minus1, a field that lives in an outer struct (VPS or SPS), not in StdVideoH265HrdParameters itself.

gfxreconstruct’s code generator resolves len expressions by walking the current struct’s members, but it cannot follow cross-struct references. This is a reasonable limitation: the generator would need to understand the full semantics of the Vulkan Video specification to know which outer struct provides the length field.

The fix is a small XML tree patch in gencode.py. Before the generator runs, I strip the len attribute from both members:

hrd_type = video_tree.find('types/type[@name="StdVideoH265HrdParameters"]')
if hrd_type is not None:
    for member_name in ('pSubLayerHrdParametersNal', 'pSubLayerHrdParametersVcl'):
        for member in hrd_type.findall('member'):
            name_elem = member.find('name')
            if name_elem is not None and name_elem.text == member_name:
                member.attrib.pop('len', None)

Without a len attribute, the generator falls back to treating each pointer as pointing to a single element. This is safe for practical capture scenarios where only one sub-layer is in use.

Regenerating the code #

With the generator changes in place, regenerating is a single command:

uv run --with pyparsing python3 framework/generated/generate_vulkan.py

uv run --with pyparsing ensures the pyparsing dependency is available without a manual pip install.

The generator overwrites all files under framework/generated/generated_vulkan_*.cpp and framework/generated/generated_vulkan_*.h. The diff was substantial: hundreds of new functions for encoding and decoding StdVideoH265* structs, plus all the video session parameter handling infrastructure.

The bigger picture #

The interesting part of this exercise was understanding gfxreconstruct’s architecture. The capture layer is not a monolithic block of hand-written interceptors. It is a generator pipeline: Python scripts consume the Khronos XML registry and emit C++ that handles serialization, deserialization, and replay for every struct and function in the Vulkan API surface.

This means enabling a new extension is mostly a matter of telling the generator to stop ignoring it, then fixing any edge cases where the XML description does not match the generator’s assumptions. The actual capture and replay logic comes for free once the generator produces code for the extension’s types and entry points.

If you are considering enabling other video extensions (H.264 encode is still blocked), the same recipe applies: remove from the runtime blocklist, remove from _remove_extensions and _remove_video_extensions, regenerate, and fix any XML len expression issues that surface.