Support GstContext for VA-API elements

|

Since I started working on gstreamer-vaapi, one of what’s disappointing me is that vaapisink is not so popular even though it should be the best choice on vaapi installed machine. There are some reasonable causes and one of the reasons is probably it doesn’t provide a convinient way to be integrated for application developers.

Until now, we provided a way to set X11 window handle by gst_video_overlay_set_window_handle, which is to tell the overlay to display video output to a specific window. But this is not enough since VA and X11 Display handle is managed internally inside gstreamer-vaapi elements, which means that users can’t handle them by themselves.

In short, there was no way to share display handle created by application. Also we have some additional problems due to this issue as the following.

  • If users want to handle multiple display seperatedly, it can’t be possible. bug 754820
  • If users run multiple decoding pipelines with vaapisink, performance is down critically since there’s some locks in each vaapisink with same VADisplay. bug 747946

Recently we have merged a series of patches to provide a way to set external VA Display and X11 Display from application via GstContext. GstContext provides a way of sharing not only between elements but also with the application using queries and messages. (For more details, see https://developer.gnome.org/gstreamer/stable/gstreamer-GstContext.html)

By these patches, application can set its own VA Display and X11 Display to VA-API elements as the following:

  • Create VADisplay instance by vaGetDisplay, it doesn’t need to be initialized at startup and terminated at endup.
  • Call gst_element_set_context with the context to which each display instance is set.

Example: sharing an VADisplay and X11 display with the bus callback, this is almost same as other examples using GstContext.

static GstBusSyncReply
bus_sync_handler (GstBus * bus, GstMessage * msg, gpointer data)
{
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:{
      const gchar *context_type;
      GstContext *context;
      GstStructure *s;
      VADisplay va_display;
      Display *x11_display;

      gst_message_parse_context_type (msg, &context_type);
      gst_println ("Got need context %s from %s", context_type,
          GST_MESSAGE_SRC_NAME (msg));

      if (g_strcmp0 (context_type, "gst.vaapi.app.Display") != 0)
        break;

      x11_display = /* Get X11 Display somehow */
      va_display = vaGetDisplay (x11_display);

      context = gst_context_new ("gst.vaapi.app.Display", TRUE);
      s = gst_context_writable_structure (context);
      gst_structure_set (s, "va-display", G_TYPE_POINTER, va_display, NULL);
      gst_structure_set (s, "x11-display", G_TYPE_POINTER, x11_display, NULL);

      gst_element_set_context (GST_ELEMENT (GST_MESSAGE_SRC (msg)), context);
      gst_context_unref (context);
      break;
    }   
    default:
      break;
  }

  return GST_BUS_PASS;
}

Also you can find the entire example code here.

Furthermore, we know we need to support Wayland for this feature. See bug 705821. There’s already some pending patches but they need to be rebased and modified based on the current way. I’ll be working on this in the near future.

We really want to test this feature more especially in practical cases until next release. I’d appreciate if someone reports any bug or issue and I promise I’d focus on it precisely.

Thanks for reading!

100 commits in GStreamer

|

It’s been 3 years since I’ve started working on GStreamer, meanwhile I contributed over 100 commits fortunately!

Let’s look at my commits in each project in GStreamer.

Now that I write this article, I have made 128 commits.

In Samsung Electronics, which was my previous company that I had been working for, I had a chance to work on gstreamer, which is main multimedia framework on Tizen. Since then, I realized that there are lots of opportunity in open source world and I started enjoying contribution to this project.

This is my first commit:

Yes. It’s just a fix typo. This landed in just five minutes after I proposed and I realized that maintainers are looking at all issues in bugzilla. To be honest, I doubted it a bit. :P

Looking at other commits that I was really happy with.

While I was working on gst-rtsp-server at that time, I found it’s not working properly for RTP retransmission on the server. I reported the issue and discussion went very positive, then my proposed patches landed finally thanks to Sebastian.

This was enhancement of infrastructure for RTSP/RTP in GStreamer, which is providing a way to report stats for sender/receiver.

Then I contributed huge patches of creating new APIs for transformation between SDP and GstCaps including removing duplicated codes. Thanks, Sebastian again.

Until this time I focused on RTSP/RTP streaming on server side since I was working on Miracast on Tizen which uses gst-rtsp-server. At this time I started looking for company so that I could work on open source more closely. Eventually I found Igalia, which is doing great work in open source world including Webkit, Chromium and GStreamer.

Since I joined Igalia I have been focusing on gstreamer-vaapi with my great colleague Victor, who is one of maintainers of GStreamer project. I got to have much more chances to contribute than before. As I said, I worked on RTSP server side before, which means that I should focus on encoder, muxer and networking stuff. But since this move, I got started focusing on playback including decoder and sink to be playable on various platforms.

These are my best patches I think.

By this set of patches, performance of playback on GL/VAAPI has been improved dramatically.

Besides, I have contributed some patches that improve vaapi decoder and encoder, most of them is for h264, which also makes me happy.

During the last three years I worked for GStreamer, I grew up with more capability of SW development, the idea of open source and more deep insight for the world of software. I give a deep appreciation for Igalia that gave me this opportunity, and also I thank you, Victor, for giving me a lot of motivation.

Even at this moment, I’m still working, enjoying and sometimes struggling with GStreamer. I really want to keep continuing this work and find a chance to contribute something new which could be applied on GStreamer.

Thanks for reading!

Libva-rust(libva binding to rust) in development stage

|

Since Rust language appeared in the world, I felt strongly this is the language I should learn.

This is because:

  • Rust guarantees to prevent from common bugs in C/C++ such as memory corruption and race condition, which are very painful to fix whenever you encounter in large project.
  • Rust guarantees it doesn’t lose performance even supporting these features!

I don’t think that Rust aims at replacing C/C++, but it’s worth learning for C/C++ developers like me at least. So I’ve been searching and thinking of what I can do with this new language. In the end of last year, I decided to implement libva bindings to Rust.

Here are advantages of doing this project.

  • I’m working on gstreamer-vaapi project, which means that I’m familiar with VA-API and middleware using this.
  • This kind of binding the existing project to another language makes me understanding the project much more than the moment.
  • Simultaneously, I could also learn new language in the level of practical development.
  • H/W acceleration is a critical feature, especially for laptop or other embedded systems. So this project could be a good option for those trying to use H/W acceleration for playback on linux.

Finally, I did open this internal project on github, named as libva-rust.
There is one example, creating an VASurface and putting raw data to the surface and displaying it only on X11 window.

Let’s see the example code briefly.

let va_disp = VADisplay::initialize(native_display as *mut VANativeDisplay).unwrap();

let va_surface = VASurface::new(&va_disp, WIDTH, HEIGHT, ffi::VA_RT_FORMAT_YUV420, 1).unwrap();

let va_config = VAConfig::new(&va_disp, ffi::VAProfileMPEG2Main, ffi::VAEntrypointVLD, 1).unwrap();

let va_context = VAContext::new(&va_disp,
                                &va_config,
                                &va_surface,
                                WIDTH as i32,
                                HEIGHT as i32,
                                0).unwrap();

Initalization for VA-API.

test_draw::image_generate(&va_disp, &va_image, &va_image_buf);

va_image.put_image(&va_disp,
                   &va_surface,
                   0,
                   0,
                   WIDTH,
                   HEIGHT,
                   0,
                   0,
                   WIDTH,
                   HEIGHT);

Draw raw data to VaapiImage in test_draw.rs and put it to the created surface.

va_surface.put_surface(&va_disp, win, 0, 0, WIDTH, HEIGHT, 0, 0, WIDTH, HEIGHT);

Finally, display it by putting the surface to created X11 window.
It’s simple as you see, but the important first step.

My first goal is providing general and easy a set of “rusty” APIs so that this could be integrated into other rust-multimedia project like rust-media.

Another potential goal is implementation of vaapi plugins in gstreamer, written in Rust. Recently, Sebastian has been working on this(https://github.com/sdroege/rsplugin), I would really like to get involved in this project.

There are tons of things to do for the moment.
Here’s to-do list for now.

  • Implement vp8 decoder first: Simply, it looks easier than h26x decoder. Is there any useful rust h26x parser out there, by the way?
  • Manipulate raw data using Rust apis like Bit/Byte Reader/Writer.
  • Implement general try-catch statement in Rust.
  • Make test cases.
  • Support wayland.

Yes. It has a long way to go still and I don’t have enough time to focus on this project. But I’ll be managing to keep working on this project.

So feel free to use and absolutely welcome contributions including issue report, bug fix, providing a patch, etc.

Thanks!