28
Jan 13

Announcing GPhone v0.10

Hi folks!

As many of you may know, lately I have been working on Ekiga and Opal. And, as usually happens to me, I started to wonder how I would re-write that piece of software. My main ideas growth clearly: craft a GObject library, ala WebKitGTK+, wrapping Opal’s classes, paying attention to the gobject-introspection, also, redirecting the Opal’s multimedia rendering to a GStreamer player,  and, finally, write the application with Vala.

The curiosity itched me hard,  so I started to hack, from time to time, these ideas. In mid-November, last year, I had a functional prototype, which only could make phone calls in a disgraceful user interface. But I got my proof of concept. Nonetheless, as I usually do, I didn’t dropped the pet project, continuing the development of more features.

And today, I am pleased to present you,  the release v0.1 of GPhone.

Well, I guess a screencast is mandatory nowadays:

 


24
Jan 13

Igalia WebKit week: welcome Žan and Rego!

This  a new weekly WebKit Igalia’s report. And the last week has been a shaky one.

Let’s start with a warm welcome to Žan Dobersek as collaborator student, who is working hard in a lot of cool matters: gardening the bots, cleaning up the Coverity run output, modifying the WebCoreTestSupport to decrease the dependency WebKit2 API, digging in a stack size problem in JSC, and much more fun stuff.

Meanwhile Joanie, after a lot of restless hacking hours, could fix a crash in tables accessibility, and saving us from many other accessibility crashes. She is working to make the world safer for table users everywhere!

But we have more new faces around: Our dear colleague, Rego, is getting his feet wet, and he started finding and fixing new bugs and he is enabling more tests in the Gtk+ port.

Calvaris is following his efforts for enhancing the user experience with the media controls for HTML5. Here is a recent screenshot of these controls:

https://bug-83869-attachments.webkit.org/attachment.cgi?id=183575

Claudio is pushing the accelerator pedal for the snapshot API in WebKit2. With this API the applications would retrieve a snapshot from a webview as in the Overview mode in Epiphany.

Epiphany's overview mode

Philippe is working on the fullscreen video porting the GStreamerGWorld module to GStreamer 1.0, while he is still hacking on the AudioSourceProvider  for WebAudio.

And the last but not the least, Dape is working real hard on the Qt spell-check support, and he also proposed solutions for Qt WebNotification support.

And that is all for now. See you!


21
Jan 13

Two video streams simultaneously in Ekiga

As we talked earlier, we added support for H.239 in Ekiga, and we were able to show the main role video and the slides role video, one stream at the time.

But now we did a step forward: we wanted to view both videos simultaneously. So, we hacked again Ekiga and we added that feature, when a second video stream arrives, another window is showed with the secondary role stream.

Here is the mandatory screencast:

 


17
Jan 13

Weekly Igalia’s report on WebKit

Hi webkitters,

This weekly report project was supposed to start after the last WebKit hackfest, but my holidays got in between and now I’m recovering of them ;)

Here we go:

In summary, in these last three weeks we have had 15 commits and done 23 reviews.

Martin and Carlos have been working on the authentication mechanisms. Now they can be hooked ,through the web view API, by the applications, which could take control of the dialogues and credentials handling.

Martin has also been dealing with text rendering with complex layouts (such as Arabic). This effort leaded, finally, to the removal of Pango in favor of Harfbuzz.

Now let’s talk about Carlos’ baby monster: the injected bundle patch. As you know, in WebKit2, the engine has been split in two isolated processes the UI and Web processes. The first is in charge of the user interface, and the former deals with HTML, CSS and JavaScript handling. Meanwhile this approach adds more robustness and responsiveness, also imposes more complexity because it is required an IPC mechanism to interact with the Web process. This is particularly hard for accessing to the DOM bindings.

Carlos, since the last year, has been working on his injected bundle patch, which offers a mean to support loading plugins in the web process using injected bundle. Hence, through DBus, an application could load a plugin to communicate, indirectly, with the Web process. This approach is supposed to be the milestone for the DOM bindings in WK2GTK, and also provides a mean to pre-fetch DNS registries. This patch has been happily pushed just recently, in the second week of January.

If this was not enough, Carlos also released the development version of WebKitGTK+ v1.11.4.

Now let us go to the multimedia realm, my favorite land.

Philippe finished the port of his patch for WebAudio support to GStreamer 1.0 as backend. And now he is porting the full-screen support in Gst 0.10 to Gst 1.0 in order to reuse ans share the same base code. Aligned with WebAudio, Philippe is developing a new audio source provider that gathers raw audio data from the MediaPlayer and pipe them into the AudioBus if it is required.

Xabier has been working to deliver a nice and neat HTML5 media controls, using stylable GTK+ controls. And myself, I’m still playing with the audio pitch preservation.

Another great landmark for us is a11y, and here Joanie has been working hard to bring back the accessibility tests on GTK to a sane state. And also keeps her efforts to enable an access to WebKit for Orca.

In other sort of things, Berto has been fighting against a bug on GtkLaunch, which was shown in Epiphany too when displaying only images. Meanwhile, Dape, lurked on spell checking support for Qt WebKit2. And Sergio enabled, by default, the WebP image handling.


25
Sep 12

The thrilling story about the H.239 support in Ekiga

The development of Ekiga 4.0 has begun, and three months ago I started a project aimed to support H.239 in Ekiga.

H.239 is an ITU recommendation and it is titled “Role management and additional media channels for H.300-series terminals”. Its purpose is the definition of procedures for use more than one video channel, and for labelling those channels with a role.

A traditional video-conference has an audio channel, a video channel and an optional data channel. The video channel typically carries the camera image of the participants. The H.239 recommendation defines rules and messages for establishing an additional video channel, often to transmit presentation slides, while still transmitting the video of the presenter.

For presentations in multi-point conferencing, H.239 defines token procedures to guarantee that only one endpoint in the conference sends the additional video channel which is then distributed to all conference participants.

Ekiga depends on Opal, a library that implements, among other things, the protocols used to send voice over IP networks (VoIP). And, according to the documentation, Opal had already support for H.239 since a while, for what we assumed that enable it in Ekiga would be straight forward.

After submitting a few patches to Opal and its base library, PTlib, then I  setup a jhbuild environment to automate the building of Ekiga and its dependencies. Soon, I realized that the task was not going to be as simple as we initially assumed: the support for H.323, in Ekiga, is not as mature as the SIP support.

I have to mention that along the development process, in order to test my scratched code, I used this H.239 application sample, a small MS Windows program which streams two video channels (the main role and the slides role). The good news is that it works fine in Wine.

Well, the true is that the activation of the client support for H.239 in Ekiga was easy. The real problem began with the video output mechanism. Ekiga has a highly complex design for drawing video frames. Too complex in my opinion. One of the first thing I did to understand the video frames display, was to sketch a sequence diagram from when a decoded frame is delivered by Opal to when the frame is displayed in the screen. And here is the result:

As I said, it is too complex for me, but the second stream had to be displayed. Fearfully, I started to refactor some code inside Ekiga and adding more parameters to handle another video stream. But finally, I just stepped forward, and submitted a bunch of patches that finally displays a second stream.

The current outcome is shown in the next screen cast: Screencast Ekiga H.239 (OGV / 7.8M)


20
Jun 12

A GStreamer Video Sink using KMS

The purpose of this blog post is to show the concepts related to the GstKMSSink, a new video sink for GStreamer 1.0, co-developed by Alessandro Decina and myself, done during my hack-fest time in the Igalia’s multimedia team.

One interesting thing to notice is that this element shows it is possible to write DRI clients without the burden of X Window.

Brief introduction to graphics in Linux

If you want to dump images onto your screen, you can simply use the frame buffer device. It provides an abstraction for the graphics hardware and represents the frame buffer of the video hardware. This kernel device allows user application to access the graphics hardware without knowing the low-level details [1].

In GStreamer, we have two options for displaying images using the frame buffer device; or three, if we use OMAP3: fbvideosink, fbdevsink and gst-omapfb.

Nevertheless, since the appearance of the GPUs, the frame buffer device interface has not been sufficient to fulfill all their capabilities. A new kernel interface ought to emerge. And that was the Direct Rendering Manager (DRM).

What in the hell is DRM?

The DRM layer is intended to support the needs of complex graphics devices, usually containing programmable pipelines well suited to 3D graphics acceleration [2]. It deals with [3]:

  1. A DMA queue for graphic buffers transfers [4].
  2. It provides locks for graphics hardware, treating it as shared resource for simultaneous 3D applications [5].
  3. And it provides secure hardware access, preventing clients from escalating privileges [6].

The DRM layer consists of two in-kernel drivers: a generic DRM driver, and another which has specific support for the video hardware [7]. This is possible because the DRM engine is extensible, enabling the device-specific driver to hook out those functionalities that are required by the hardware. For example, in the case of the Intel cards, the Linux kernel driver i915 supports this card and couples its capabilities to the DRM driver.

The device-specific driver, in particular, should cover two main kernel interfaces: the Kernel Mode Settings (KMS) and the Graphics Execution Manager (GEM). Both elements are also exposed to the user-space through the DRM.

With KMS, the user can ask the kernel to enable native resolution in the frame buffer, setting certain display resolution and colour depth mode. One of the benefits of doing it in kernel is that, since the kernel is in complete control of the hardware, it can switch back in the case of failure [8].

In order to allocate command buffers, cursor memory, scanout buffers, etc., the device-specific driver should support a memory manager, and GEM is the manager with more acceptance these days, because of its simplicity [9].

Beside to the graphics memory management, GEM ensures conflict-free sharing of data between applications by managing the memory synchronization. This is important because modern graphics hardware are essentially NUMA environments.

The following diagram shows the components view of the DRM layer:

Direct Rendering Infrastructure

What is the deal with KMS?

KMS is important because on it relies GEM and DRM to allocate frame buffers and to configure the display. And it is important to us because almost all of the ioctls called by the GStreamer element are part of the KMS subset.

Even more, there are some voices saying that KMS is the future replacement for the frame buffer device [10].

To carry out its duties, the KMS identifies five main concepts [11,12]:

Frame buffer:
The frame buffer is just a buffer, in the video memory, that has an image encoded in it as an array of pixels. As KMS configures the ring buffer in this video memory, it holds a the information of this configuration, such as width, height, color depth, bits per pixel, pixel format, and so on.
CRTC:
Stands for Cathode Ray Tube Controller. It reads the data out of the frame buffer and generates the video mode timing. The CRTC also determines what part of the frame buffer is read; e.g., when multi-head is enabled, each CRTC scans out of a different part of the video memory; in clone mode, each CRTC scans out of the same part of the memory.Hence, from the KMS perspective, the CRTC’s abstraction contains the display mode information, including, resolution, depth, polarity, porch, refresh rate, etc. Also, it has the information of the buffer region to display and when to change to the next frame buffer.
Overlay planes:
Overlays are treated a little like CRTCs, but without associated modes our encoder trees hanging off of them: they can be enabled with a specific frame buffer attached at a specific location, but they don’t have to worry about mode setting, though they do need to have an associated CRTC to actually pump their pixels out [13].
Encoder:
The encoder takes the digital bitstream from the CRTC and converts it to the appropriate format across the connector to the monitor.
Connector:
The connector provides the appropriate physical plug for the monitor to connect to, such as HDMI, DVI-D, VGA, S-Video, etc..

And what about this KMSSink?

KMSSink is a first approach towards a video sink as a DRI client. For now it only works in the panda-board with a recent kernel (I guess, 3.3 would make it).

For now it only uses the custom non-tiled buffers and use an overlay plane to display them. So, it is in the to-do, add support for more hardware.

 Bibliography

[1] http://free-electrons.com/kerneldoc/latest/fb/framebuffer.txt
[2] http://free-electrons.com/kerneldoc/latest/DocBook/drm/drmIntroduction.html
[3] https://www.kernel.org/doc/readme/drivers-gpu-drm-README.drm
[4] http://dri.sourceforge.net/doc/drm_low_level.html
[5] http://dri.sourceforge.net/doc/hardware_locking_low_level.html
[6] http://dri.sourceforge.net/doc/security_low_level.html
[7] https://en.wikipedia.org/wiki/Direct_Rendering_Manager
[8] http://www.bitwiz.org.uk/s/how-dri-and-drm-work.html
[9] https://lwn.net/Articles/283798/
[10] http://phoronix.com/forums/showthread.php?23756-KMS-as-anext-gen-Framebuffer
[11] http://elinux.org/images/7/71/Elce11_dae.pdf
[12] http://www.botchco.com/agd5f/?p=51
[13] https://lwn.net/Articles/440192/


10
Apr 12

Aura

In the last weeks Miguel, Calvaris and myself, developed an application for the N9/N950 mobile phone and we called it Aura.

Basically it uses the device’s camera (either the main one or the frontal one) for video recording, as a normal camera application, but also it exposes a set of effects that can be applied, in real time, to the video stream.

For example, here is a video using the historical effect:

Aura is inspired in the Gnome application, Cheese, and it uses many of the effects available in Gnome Video Effects.

The list of effects that were possible to port to the N9/N950 are: dice, edge, flip, historical, hulk, mauve, noir/blanc, optical illusion, quark, radioactive, waveform, ripple, saturation, shagadelic, kung-fu, vertigo and warp.

Besides of these software effects, it is possible to add, simultaneously, another set of effects that the hardware is capable, such as sepia colors. These hardware capabilities do not impose extra processing as the software effects do.

Because of this processing cost, imposed by the non-hardware video effects, Aura has a fixed video resolution. Otherwise the performance would make the application unusable. Also, we had a missing feature: the still image capture. But, hey! there is good news: Aura is fully open source, you can checkout the code at github and we happily accept patches.

Honoring Cheese, the name of Aura is taken from a kind of Finnish blue cheese.

We hope you enjoy this application as we enjoyed developing it.


26
Feb 12

Debian’s mutt with notmuch support

One of my weekend tasks was to reorganize my email environment. For reading email I use mutt, configured to grab the email from an IMAP server. For sending email, I have a minimal exim setup securely relaying to a smart host.

Mutt is a great email browser, but it is very bad at handling IMAP. Besides, I started to need searching through all my email. The solution for the first problem is offlineimap, a program wrote in Python, that “synchronizes emails between two repositories”. It downloads my email from the IMAP server into my laptop, so I work in my email locally, and if I delete an email locally, offlineimap will delete it in the next sync operation.

The solution for the second problem, the search, is notmuch, which is a email indexer, enabling fast searches among a vast mail collection. So, once new mail arrive (or is deleted) with offlineimap, notmuch (de)indexes it.

But another problem appear: how to query to notmch in an integrated way with my mail reader? One solution is provided by mutt-kz, a fork of mutt with notmuch support tightly integrated.

But I use Debian, and I like its package management. So I needed to craft a Debian package for mutt-kz.

I grabbed the Debian’s repository for mutt and re-based, one by one, the patches from mutt-kz.

The result is stored in this repository.

And now, I can query notmuch in mutt and immediately browse the result set.


17
Jan 12

GStreamer video decoder for SysLink

This blog post is another one of our series about SysLink (1, 2): finally I came with a usable GStreamer element for video decoding, which talks directly with the SysLink framework in the OMAP4’s kernel.

As we stated before, SysLink is a set of kernel modules that enables the initialization of remote processors, in a multi-core system (which might be heterogeneous), that run their own operating systems in their own memory space, and also, SysLink, enables the communication between the host processor with the remotes ones. This software and hardware setup could be viewed as an Asymmetric Multi-Processing model.

TI provides a user-space library to access the SysLink services, but I find its implementation a bit clumsy, so I took the challenge of rewrite a part of it, in a simple and straightforward fashion, as gst-dsp does for DSP/Bridge. The result is the interface syslink.h.

Simultaneously, I wrote the utility to load and monitor the operating system into the Cortex-M3 processors for the PandaBoard. This board, such as all the OMAP4-based SoCs, has two ARM Core-M3 as remote processors. Hence, this so called daemon.c, is in charge of loading the firmware images, setting the processor in its running state, allocating the interchange memory areas, and monitoring for any error message.

In order to load the images files into the processors memory areas, it is required to parse the ELF header of the files, and that is the reason of why I decided to depend on libelf, rather than write another ELF parser. Yes, one sad dependency for the daemon. The use of libelf is isolated in elf.h.

When I was developing the daemon, for debugging purposes, I needed to trace the messages generated by the images in the remote processors. For that reason I wrote tracer.c, whose only responsibility is to read and to parse the ring buffer used by the images, in the remote processors, for logging.

Now, in OMAP4, the subsystem comprised by the two Cortex-M3 processors is called Ducati. The first processor is used only for the exchange of notification messages among the host processor and the second M3 processor, where all the multimedia processing is done.

There are at least two images for the second Cortex-M3 processor: DOMX, which is closed source and focused, as far as I know, on the OMX IL interface; and, in the other hand, DCE, which is open source, it’s developed by Rob Clark, and it provides a simple interface of buffers interchange.

My work use DCE, obviously.

But, please, let me go back one step in this component description: in order to send and receive commands between the host processor and one remote processor, SysLink uses a packet based protocol, called Remote Command Messaging, or just RCM for the friends. There are two types of interfaces of RCM, the client and the server. The client interface is used by the applications running in the host processor, and they request services to the server interface, exposed by the systems running in the remote processors, it is accepting the requests and it returns results.

The RCM client interface is in rcm.h.

Above the RCM client, sits my dce.h interface, which is in charge of control the state of the video decoders and it is also in charge of handling the buffers.

But these buffers are tricky. They are not memory areas allocated by a simple malloc, instead they are buffers allocated by a mechanism in the kernel called tiler. The purpose of this mechanism is to provide buffers with capacity of 2D operations by hardware (in other words, cheap and quick in computations). These buffers are shared along all the processing pipeline, so the copies of memory areas are not needed. Of course, in order to achieve this paradise, the video renderer must handle this type of buffers too.

In my code, the interface to the tiler mechanism is in tiler.h.

And finally, the all mighty GStreamer element for video decoding: gstsyslinkvdec.c! Following the spirit of gst-dsp, this element is intended to deal with all the available video decoders in the DCE image, although for now, the H264 decoding is the only one handled.

For now, I have only tested the decoder with fakesink, because the element pushes tiled buffers onto the source pad, and, in order to have an efficient video player, it is required a video renderer that handles this type of tiled buffers. TI is developing one, pvrvideosink, but it depends on EGL, and I would like to avoid X whenever is possible.

I have not measured either the performance of this work compared with the TI’s combo (syslink user-space library / memmgr / libdce / gst-ducati), but I suspect that my approach would be little more efficient, faster, and, at least, simpler ;)

The sad news, as in every hard paced development, all these kernel mechanisms are already deprecated: SysLink and DMM-Tiler will never be mainlined into the kernel, but their successors, rproc/rpmsg and omapdrm, have a good chance. And both have a very different approach since their predecessors. Nevertheless, SysLink is already here and it is being used widely, so this effort has an opportunity for being worthy.

My current task is to decide if I should drop the 2D buffers in the video decoders or if I should develop a video renderer for them.


17
Dec 11

devhelp and jhbuild v2

A couple years ago I wrote a small tip about how to integrate the devhelp installed by your distribution with the documentation generated through jhbuild.

Well, that way is plain wrong. I noticed it when I upgraded to GNOME 3: modifying the environment variable XDG_DATA_DIRS, the gnome-session goes crazy.

So, I found a less intrusive way to make your distro-based devhelp reads the jhbuild-generated documentation:

ln -s /opt/jhbuild/share/gtk-doc/ ~/.local/share/

And that is it!