Herostratus’ legacy

words from a lazy coder

DMABuf modifier negotiation in GStreamer

It took almost a year of design and implementation but finally the DMABuf modifier negotiation in GStreamer is merged. Big kudos to all the people involved but mostly to He Junyan, who did the vast majority of the code.## What’s a DMAbuf modifier?

DMABuf are the Linux kernel mechanism to share buffers among different drivers or subsystems. A particular case of DMABuf are the DRM PRIME buffers which are buffers shared by the Display Rendering Manager (DRM) subsystem. They allowed sharing video frames between devices with zero copy.

When we initially added support for DMABuf in GStreamer, we assumed that only color format and size mattered, just as old video frames stored in system memory. But we were wrong. Beside color format and size, also the memory layout has to be considered when sharing DMABufs. By not considering it, the produced output had horrible tiled artifacts in screen. This memory layout is known as modifier, and it’s uniquely described by an uint64 number.

How was designed and implemented? #

First, we wrote a design document for caps negotiation with dmabuf modifiers, where we added a new color format (DMA_DRM) and a new caps field (drm-format). This new caps field holds a string, or a list of strings, composed by the tuple DRM_color_format : DRM_modifier.

Second, we extended the video info object to support DMABuf with helper functions that parse and construct the drm-format field.

Third, we added the dmabuf caps negotiation in glupload. This part was the most difficult one, since the capability of importing DMABufs to OpenGL (which is only available in EGL/GLES) is run-time defined, by querying the hardware. Also, there are two code paths to import frames: direct or RGB-emulated. Direct would be the most efficient, but it depends on the presence of GLES2 API in the driver; while RGB-emulated is imported as a set of RGB images where each component is an image. At the end more than a thousand lines of code were added to the glupload element, beside the code added to EGL context object.

Fourth, and unexpectedly, waylandsink also got DMABuf caps negotiation support.

And lastly, decoders in va plugin merged theirs DMABuf caps negotiation support.

How I can test it? #

You need, of course, to user the current main branch of GStreamer, since it’s just fresh and there’s no release yet. Then you need a box with VA support. And if you inspect, for example, vah264dec, you might see this output if your box is Intel (but also AMD through Mesa is supported though the negotiated memory is linear so far):

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h264
                profile: { (string)main, (string)baseline, (string)high, (string)progressive-high, (string)constrained-high, (string)constrained-baseline }
                  width: [ 1, 4096 ]
                 height: [ 1, 4096 ]
              alignment: au
          stream-format: { (string)avc, (string)avc3, (string)byte-stream }

  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:VAMemory)
                  width: [ 1, 4096 ]
                 height: [ 1, 4096 ]
                 format: NV12
      video/x-raw(memory:DMABuf)
                  width: [ 1, 4096 ]
                 height: [ 1, 4096 ]
                 format: DMA_DRM
             drm-format: NV12:0x0100000000000002
      video/x-raw
                  width: [ 1, 4096 ]
                 height: [ 1, 4096 ]
                 format: NV12

What it’s saying, for memory:DMABuf caps feature, the drm-format to negotiate is NV12:0x0100000000000002.

Now some tests:

NOTE: These commands assume that va decoders are primary ranked (see merge request 2312), and that you’re in a Wayland session.

gst-play-1.0 --flags=0x47 video.file --videosink=waylandsink
GST_GL_API=gles2 gst-play-1.0 --flags=0x47 video.file --videosink=glimagesink
gst-play-1.0 --flags=0x47 video.file --videosink='glupload ! gtkglsink'

Right now it’s required to add --flags=0x47 to playbin because it adds video filters that still don’t negotiate the new DMABuf caps.

GST_GL_API=gles2 instructs GStreamer OpenGL to use GLES2 API, which allows direct importation of YUV images.

Thanks to all the people involved in this effort!

As usual, if you would like to learn more about DMABuf, VA-API, GStreamer or any other open multimedia framework, contact us!