Tag: gstgoo

OpenMAX: a rant

I used to work with OpenMAX a while ago. I was exploring an approach to wrap OpenMAX components with GStreamer elements. The result was gst-goo. Its purpose was to test the OpenMAX components in complex scenarios and it was only focused for the Texas Instruments’ OpenMAX implementation for the OMAP3 processor.

Some time after we started gst-goo, Felipe Contreras released gst-openmax, which had a more open development, but with a hard set of performance objectives like zero-copy. And also only two implementations were supported at that moment: Bellagio and the TI’s one mentioned before.

Recently, Sebastian Dröge has been working on a redesign of gst-openmax, called gst-omx. He explained the rational behind this new design in his talk in the GStreamer Conference 2011. If you are looking for a good summary of the problems faced when wrapping OpenMAX with GStreamer, because of their semantic impedance mismatch, you should watch his talk.

In my opinion, the key purpose of OpenMAX is to provide a common application interface to a set of different and heterogeneous multimedia components: You could take different implementations, that could offer hardware-accelerated codecs or either any other specialized ones, and build up portable multimedia applications. But this objective has failed utterly: every vendor delivers a incompatible implementation with the others available. One of the causes, as Dröge explained, is because of the specification, it is too ambiguous and open to interpretations.

From my perspective, the problem arises from the need of a library like OpenMAX. It is needed because the implementer wants to hide (or to abstract if you prefer) the control and buffer management of his codification entities. By hiding this, the implementer has the freedom to develop his own stack closely, without any kind of external review.

In order to explain the problem brought by the debauchery in the hind of OpenMAX, let me narrow the scope of the problem: I will not fall on the trap of portability among different operative systems, specially in those of non-Unix. Even more, I will only focus on the ARM architecture of the Linux kernel. Thus, I will not consider the software-based codecs, only the hardware-accelerated ones. The reason upholding these constrains is that, beside the PacketVideo’s OpenCORE, I am not aware of any other successful set of non-Unix / software-based multimedia codecs, interfaced with OpenMAX.

As new and more complex hardware appears, with its own processing units, capable of off-loading the main processor, silicon vendors must deliver also the kernel drivers to operate them. This problem is very recurrent among the ARM vendors, where the seek of added value gives the competitive advantage, and the Linux kernel has the virtues required for a fast time to market.

But these virtues have turned into a ballast: It has been observed excessive churn in the ARM architecture, duplicated code, board-specific data encoded in source files, and conflicts at kernel’s code integration. In other words, every vendor has built up their own software stack without taking care of developing common interfaces among all of them. And this has been particularly true for the hardware-accelerated multimedia components, where OpenMAX promised to the user-space developers what the kernel developers could not achieve.

First we need a clear and unified kernel interface for hardware accelerated multimedia codecs, so the user-space implementations could be straight, lean and clean. Those implementations could be OpenMAX, GStreamer, libav, or whatever we possibly want and need.

But there is hope. Recently there has happened a lot of effort bringing new abstractions and common interfaces for the ARM architecture, so in the future we could expect integrated interfaces for all these new hardware, independently of the vendor.

Though, from my perspective, if we reach this point (and we will), we will have less motivation for a library like OpenMAX, because a high level library, such as GStreamer, it would cover a lot of hardware within a single element. Hence, it is a bit pointless to invest too much in OpenMAX or its wrappers nowadays.

Of course, if you think that I made a mistake along these reasons, I would love to read your comments.

And last but not least, Igalia celebrates its 10th anniversary! Happy igalian is happy 🙂

libgoo & gst-goo

Back in 2007 I started to work integrating OpenMAX IL components into the GStreamer platform.

OpenMAX is a set of programming interfaces, in C language, for portable multimedia processing. Specifically the Integration Layer (IL) defines the interface to communicate with multimedia codecs implemented by hardware or software.

Texas Instrument started to work on an implementation of the OpenMAX IL for their DSP accelerated codecs for OMAP platform.

A quick and rough view of the software architecture implemented to achieve this processing is more or less exposed in the next diagram:

+---------------------+
| OpenMAX IL          |
+---------------------+
| libdspbridge        |
+---------------------+
| Kernel (DSP Bridge) |
+---------------------+

The DSP Bridge driver is a Linux Kernel device driver designed to supply a direct link between the GPP program and the assigned DSP node. Basically the features offered by the driver are:

  • Messaging: Ability to exchange fixed size control messages with DSP
  • Dynamic memory management: Ability to dynamically map files to DSP address space
  • Dynamic loading: Ability to dynamically load new nodes on DSP at run time
  • Power Management: Static and dynamic power management for DSP

The libdspbridge is part of the user-space utilities of the DSP bridge, which purpose is to provide a simple programming interface to the GPP  programs for the driver services.

In the DSP side, using the C/C++ compiler for the C64x+ and the libraries contained in the user-space utilities, it is possible to compile a DSP program  and package it as a DSP node, ready to be controlled by the DSP bridge driver. But right now TI provides a set of out-of-the-box DSP multimedia codecs for non-commercial purposes. These nodes are contained in the tiopenmax package.

So, as I said before, my job was to wrap up the OpenMAX IL components delivered by TI as a GStreamer plug-in. In that way a lot of available multimedia consumers could use the hardware accelerated codecs. But also, our team did the test of the delivered OpenMAX components.

After trying several approaches we came to the conclusion that we need a new layer of software which will provide us

  • Facilitate a great testing coverage of the components without the burden of the upper framework (GStreamer in this case).
  • Improve the code reuse.
  • Use an object oriented programming through GObject.
  • Facilitate the bug’s workaround for each component and maintenance of those workarounds.
  • A playground for experimenting with features such as (OpenMAX specific) tunneling and the (TI specific) DSP Audio Software Framework (DASF).

For those reasons we started to develop an intermediate layer called GOO (GObject OpenMAX).

+---------------------+
| GStreamer / gst-goo |
+---------------------+
| libgoo              |
+---------------------+
| OpenMAX             |
+---------------------+

libgoo is a C language library that wraps OpenMAX using GObject. The follow diagram shows part of its class hierarchy.

                           +--------------+
                           | GooComponent |
                           +--------------+
                                   |
+---------------+ +---------------+ +---------------+ +---------------+
| GooTiAudioEnc | | GooTiAudioDec | | GooTiVideoDec | | GooTiVideoEnc |
+---------------+ +---------------+ +---------------+ +---------------+
        |                 |                 |                 |
 +-------------+   +-------------+  +---------------+ +---------------+
 | GooTiAACEnc |   | GooTiAACDec |  | GooTiMpeg4Dec | | GooTiMpeg4Enc |
 +-------------+   +-------------+  +---------------+ +---------------+

At the top there is GooComponent which represents any OpenMAX component. If the OMX IL implementation is neat and clean, there shouldn’t need to add subclasses underneath it, just parametrize it, and should be ready to use as any other OMX IL component. But reality, as usual, is quite different: Every implementation is different from each other; and to make it worst, each component in a same implementation might behave differently, and that was the case of the TI implementation.

Finally, over libgoo there is gst-goo, the set of GStreamer elements which use the libgoo components. GstGoo also sketched some proof of concepts such as ghost buffers (to be used with the OpenMAX interop profile), and dasfsink and dasfsrc (TI specific).

In those days, before I move to the GStreamer team, an old fellow, Felipe Contreras, worked on gomx, which is the precedent of libgoo, before he got an opportunity in Nokia and started to code on GstOpenMAX. An interesting issue at this point is that Felipec is pushing boldly for a new set of GStreamer elements which ditched OpenMAX and talks directly to the kernel’s DSP bridge: gst-dsp.

What’s the future of libgoo and GstGoo? I couldn’t say. Since I moved to Igalia, I left its development. I’ve heard about a couple companies showed some kind of interest on it, sadly, the current developers are very constrained by the TI workload.