Support GstContext for VA-API elements

Since I started working on gstreamer-vaapi, one of what’s disappointing me is that vaapisink is not so popular even though it should be the best choice on vaapi installed machine. There are some reasonable causes and one of the reasons is probably it doesn’t provide a convinient way to be integrated for application developers.

Until now, we provided a way to set X11 window handle by gst_video_overlay_set_window_handle, which is to tell the overlay to display video output to a specific window. But this is not enough since VA and X11 Display handle is managed internally inside gstreamer-vaapi elements, which means that users can’t handle them by themselves.

In short, there was no way to share display handle created by application. Also we have some additional problems due to this issue as the following.

• If users want to handle multiple display seperatedly, it can’t be possible. bug 754820
• If users run multiple decoding pipelines with vaapisink, performance is down critically since there’s some locks in each vaapisink with same VADisplay. bug 747946

Recently we have merged a series of patches to provide a way to set external VA Display and X11 Display from application via GstContext. GstContext provides a way of sharing not only between elements but also with the application using queries and messages. (For more details, see https://developer.gnome.org/gstreamer/stable/gstreamer-GstContext.html)

By these patches, application can set its own VA Display and X11 Display to VA-API elements as the following:

• Create VADisplay instance by vaGetDisplay, it doesn’t need to be initialized at startup and terminated at endup.
• Call gst_element_set_context with the context to which each display instance is set.

Example: sharing an VADisplay and X11 display with the bus callback, this is almost same as other examples using GstContext.

static GstBusSyncReply
bus_sync_handler (GstBus * bus, GstMessage * msg, gpointer data)
{
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:{
      const gchar *context_type;
      GstContext *context;
      GstStructure *s;
      VADisplay va_display;
      Display *x11_display;

      gst_message_parse_context_type (msg, &context_type);
      gst_println ("Got need context %s from %s", context_type,
          GST_MESSAGE_SRC_NAME (msg));

      if (g_strcmp0 (context_type, "gst.vaapi.app.Display") != 0)
        break;

      x11_display = /* Get X11 Display somehow */
      va_display = vaGetDisplay (x11_display);

      context = gst_context_new ("gst.vaapi.app.Display", TRUE);
      s = gst_context_writable_structure (context);
      gst_structure_set (s, "va-display", G_TYPE_POINTER, va_display, NULL);
      gst_structure_set (s, "x11-display", G_TYPE_POINTER, x11_display, NULL);

      gst_element_set_context (GST_ELEMENT (GST_MESSAGE_SRC (msg)), context);
      gst_context_unref (context);
      break;
    }   
    default:
      break;
  }

  return GST_BUS_PASS;
}

Also you can find the entire example code here.

Furthermore, we know we need to support Wayland for this feature. See bug 705821. There’s already some pending patches but they need to be rebased and modified based on the current way. I’ll be working on this in the near future.

We really want to test this feature more especially in practical cases until next release. I’d appreciate if someone reports any bug or issue and I promise I’d focus on it precisely.

Thanks for reading!