Author Archives: calvaris

by calvaris

VCR to WebM with GStreamer and hardware encoding

My family had bought many years ago a Panasonic VHS video camera and we had recorded quite a lot of things, holidays, some local shows, etc. I even got paid 5000 pesetas (30€ more than 20 years ago) a couple of times to record weddings in a amateur way. Since my father passed less than a year ago I have wanted to convert those VHS tapes into something that can survive better technologically speaking.

For the job I bought a USB 2.0 dongle and connected it to a VHS VCR through a SCART to RCA cable.

The dongle creates a V4L2 device for video and is detected by Pulseaudio for audio. As I want to see what I am converting live I need to tee both audio and video to the corresponding sinks and the other part would go to to the encoders, muxer and filesink. The command line for that would be:

gst-launch-1.0 matroskamux name=mux ! filesink location=/tmp/test.webm \
v4l2src device=/dev/video2 norm=255 io-mode=mmap ! queue ! vaapipostproc ! tee name=video_t ! \
queue ! vaapivp9enc rate-control=4 bitrate=1536 ! mux.video_0 \
video_t. ! queue ! xvimagesink \
pulsesrc device=alsa_input.usb-MACROSIL_AV_TO_USB2.0-02.analog-stereo ! 'audio/x-raw,rate=48000,channels=2' ! tee name=audio_t ! \
queue ! pulsesink \
audio_t. ! queue ! vorbisenc ! mux.audio_0

As you can see I convert to WebM with VP9 and Vorbis. Something interesting can be passing norm=255 to the v4l2src element so it’s capturing PAL and the rate-control=4 for VBR to the vaapivp9enc element, otherwise it will use cqp as default and file size would end up being huge.

You can see the pipeline, which is beatiful, here:

As you can see, we’re using vaapivp9enc here which is hardware enabled and having this pipeline running in my computer was consuming more or less 20% of CPU with the CPU absolutely relaxed, leaving me the necessary computing power for my daily work. This would not be possible without GStreamer and GStreamer VAAPI plugins, which is what happens with other solutions whose instructions you can find online.

If for some reason you can’t find vaapivp9enc in Debian, you should know there are a couple of packages for the intel drivers and that the one you should install is intel-media-va-driver. Thanks go to my colleague at Igalia Víctor Jáquez, who maintains gstreamer-vaapi and helped me solving this problem.

My workflow for this was converting all tapes into WebM and then cutting them in the different relevant pieces with PiTiVi running GStreamer Editing Services both co-maintained by my colleague at Igalia, Thibault Saunier.

by calvaris

Some rough numbers on WebKit code

My wife asked me for some rough LOC numbers on the WebKit project and I think I could share them with you here as well. They come from r221232. As I’ll take into account some generated code it is relevant to mention that I built WebKitGTK+ with the default CMake options.

First thing I did was running sloccount Source and got the following numbers:

cpp: 2526061 (70.57%)
ansic: 396906 (11.09%)
asm: 207284 (5.79%)
javascript: 175059 (4.89%)
java: 74458 (2.08%)
perl: 73331 (2.05%)
objc: 44422 (1.24%)
python: 38862 (1.09%)
cs: 13011 (0.36%)
ruby: 11605 (0.32%)
xml: 11396 (0.32%)
sh: 3747 (0.10%)
yacc: 2167 (0.06%)
lex: 1007 (0.03%)
lisp: 89 (0.00%)
php: 10 (0.00%)

This number do not include IDL code so I did some grepping to get the number myself that gave me 19632 IDL lines:

$ find Source/ -name ".idl" | xargs cat | grep -ve "^[[:space:]]\/*" -ve "^[[:space:]]*" -ve "^[[:space:]]$" -ve "^[[:space:]][$" -ve "^[[:space:]]};$" | wc -l
19632

The interesting part of the IDL files is that they are used to generate code so those 19632 IDL lines expand to:

ansic: 699140 (65.25%)
cpp: 368720 (34.41%)
python: 1492 (0.14%)
xml: 1040 (0.10%)
javascript: 883 (0.08%)
asm: 169 (0.02%)
perl: 11 (0.00%)

Let’s have a look now at the LayoutTests (they test the functionality of WebCore + the platform). Tests are composed mainly by HTML files so if you run sloccount LayoutTests you get:

javascript: 401159 (76.74%)
python: 87231 (16.69%)
xml: 22978 (4.40%)
php: 4784 (0.92%)
ansic: 3661 (0.70%)
perl: 2726 (0.52%)
sh: 199 (0.04%)

It’s quite interesting to see that sloccount does not consider HTML which is quite relevant when you’re testing a web engine so again, we have to count them manually (thanks to Carlos López who helped me to properly grep here as some binary lines were giving me a headache to get the numbers):

find LayoutTests/ -name ".html" -print0 | xargs -0 cat | strings | grep -Pv "^[[:space:]]$" | wc -l
2205690

You can see 2205690 of “meaningful lines” that combine HTML + other languages that you can see above. I can’t substract here to just get the HTML lines because the number above take into account files with a different extension than HTML, though many of them do include other languages, specially JavaScript.

But the LayoutTests do not include only pure WebKit tests. There are some imported ones so it might be interesting to run the same procedure under LayoutTests/imported to see which ones are imported and not written directly into the WebKit project. I emphasize that because they can be written by WebKit developers in other repositories and actually I can present myself and Youenn Fablet as an example as we wrote tests some tests that were finally moved into the specification and included back later when imported. So again, sloccount LayoutTests/imported:

python: 84803 (59.99%)
javascript: 51794 (36.64%)
ansic: 3661 (2.59%)
php: 575 (0.41%)
xml: 250 (0.18%)
sh: 199 (0.14%)
perl: 86 (0.06%)

The same procedure to count HTML + other stuff lines inside that directory gives a number of 295490:

$ find LayoutTests/imported/ -name ".html" -print0 | xargs -0 cat | strings | grep -Pv "^[[:space:]]$" | wc -l
295490

There are also some other tests that we can talk about, for example the JSTests. I’ll mention already the numbers summed up regarding languages and the manual HTML code (if you made it here, you know the drill already):

javascript: 1713200 (98.64%)
xml: 20665 (1.19%)
perl: 2449 (0.14%)
python: 421 (0.02%)
ruby: 56 (0.00%)
sh: 38 (0.00%)
HTML+stuff: 997

ManualTests:

javascript: 297 (41.02%)
ansic: 187 (25.83%)
java: 118 (16.30%)
xml: 103 (14.23%)
php: 10 (1.38%)
perl: 9 (1.24%)
HTML+stuff: 16026

PerformanceTests:

javascript: 950916 (83.12%)
cpp: 147194 (12.87%)
ansic: 38540 (3.37%)
asm: 5466 (0.48%)
sh: 872 (0.08%)
ruby: 419 (0.04%)
perl: 348 (0.03%)
python: 325 (0.03%)
xml: 5 (0.00%)
HTML+stuff: 238002

TestsWebKitAPI:

cpp: 44753 (99.45%)
ansic: 163 (0.36%)
objc: 76 (0.17%)
xml: 7 (0.02%)
javascript: 1 (0.00%)
HTML+stuff: 3887

And this is all. Remember that these are just some rough statistics, not a “scientific” paper.

Update:

In her expert opinion, in the WebKit project we are devoting around 50% of the total LOC to testing, which makes it a software engineering “textbook” project regarding testing and I think we can be proud of it!

by calvaris

Web Engines Hackfest 2016

From September 26th to 28th we celebrated at the Igalia HQ the 2016 edition of the Web Engines Hackfest. This year we broke all records and got participants from the three main companies behind the three biggest open source web engines, say Mozilla, Google and Apple. Or course, it was not only them, we had some other companies and ourselves. I was active part of the organization and I think we not only did not get any complain but people were comfortable and happy around.

We had several talks (I included the slides and YouTube links):

We had lots and lots of interesting hacking and we also had several breakout sessions:

  • WebKitGTK+ / Epiphany
  • Servo
  • WPE / WebKit for Wayland
  • Layout Models (Grid, Flexbox)
  • WebRTC
  • JavaScript Engines
  • MathML
  • Graphics in WebKit

What I did during the hackfest was working with Enrique and Žan to advance on reviewing our downstream implementation of our GStreamer based of Media Source Extensions (MSE) in order to land it as soon as possible and I can proudly say that we did already (we didn’t finish at the hackfest but managed to do it after it). We broke the bots and pissed off Michael and Carlos but we managed to deactivate it by default and continue working on it upstream.

So summing up, from my point of view and it is not only because I was part of the organization at Igalia, based also in other people’s opinions, I think the hackfest was a success and I think we will continue as we were or maybe growing a bit (no spoilers!).

Finally I would like to thank our gold sponsors Collabora and Igalia and our silver sponsor Mozilla.

by calvaris

WebRTC in WebKit/WPE

For some time I worked at Igalia to enable WebRTC on WebKitForWayland or WPE for the Raspberry Pi 2.

The goal was to have the WebKit WebRTC tests working for a demo. My fellow Igalian Alex was working on the platform itself in WebKit and assisting with some tuning for the Pi on WebKit but the main work needed to be done in OpenWebRTC.

My other fellow Igalian Phil had begun a branch to work on this that was half way with some workarounds. My first task was getting into combat/workaround mode and make OpenWebRTC work with compressed streams from gst-rpicamsrc. OpenWebRTC supported only raw video streams and that Raspberry Pi Cam module GStreamer element provides only H264 encoded ones. I moved some encoders and parsers around, some caps modifications, removed some elements that didn’t work on the Pi and made it work eventually. You can see the result at:

<

video style=”max-width: 100%;” src=”/xrcalvar/files/2016/10/201607-webrtc.mp4″ controls poster=”/xrcalvar/files/2016/10/webrtc-poster.png”>

To make this work by yourselves you needed a custom branch of Buildroot where you could build with the proper plugins enabled also selected the appropriate branches in WPE and OpenWebRTC.

Unfortunately the work was far from being finished so I continued the effort to try to make the arch changes in OpenWebRTC have production quality and that meant do some tasks step by step:

  • Rework the video orientation code: The workaround deactivated it as so far it was being done in GStreamer. In the case of rpicamsrc that can be done by the hardware itself so I cooked a GStreamer interface to enable rotation the same way it was done for the [gl]videoflip elements. The idea would be deprecate the original ones and use the new interface. These landed both in videoflip and glvideoflip. Of course I also implemented it on gst-rpicamsrc here and here and eventually in OpenWebRTC sources.
  • Rework video flip: Once OpenWebRTC sources got orientation support, I could rework the flip both for local and remote feeds.
  • Add gl{down|up}load elements back: There were some issues with the gl elements to upload and download textures, which we had removed. I readded them again.
  • Reworked bins linking: In OpenWebRTC there are some bins that are created to perform some tasks and depending on different circumstances you add or not some elements. I reworked the way those elements are linked so that we don’t have to take into account all the use cases to link them. Now this is easier as the elements are linked as they are the added to the bin.
  • Reworked the renderer_disabled: As in the case for orientation, some elements such as gst-rpicamsrc are able to change color and balance so I added support for that to avoid having that done by GStreamer elements if not necessary. In this case the proper interfaces were already there in GStreamer.
  • Moved the decoding/parsing from the source to the renderer: Before our changes the source was parsing/decoding the remote feeds, local sources were not decoded, just raw was supported. Our workarounds made the local sources decode too but this was not working for all cases. So why decoding at the sources when GStreamer has caps and you can just chain all that to the renderers? So I did eventually, I moved the parsing/decoding to the renderers. This took fixing all the caps negotiation from sources to renderers. Unfortunatelly I think I broke audio on the way, but surely nothing difficult to fix.

This is still a work in progress and now I am changing tasks and handing this over back to my fellow Igalians Phil, who I am sure will do an awesome job together with Alex.

And again, thanks to Igalia for letting me work on this and to Metrological that is sponsoring this work.

by calvaris

Aura

As my colleague Víctor at Igalia has said before in his post, Aura was released to the Nokia Store. Miguel, Víctor and I are quite happy with the result achieved with this app, which intention was to be kind of a port of the Cheese application of the GNOME platform to be used in the N9 or N950 Nokia phones.

The apps allows you to use both cameras (front and principal) to record videos, applying a lot of funny effects (a subset of the GNOME Video Effects) and changing them during the recording. Being Nokia a Finnish company, we decided to name the app after a Finnish Cheese to both honor the GNOME Cheese application and Finland 😉

You can download the app from the Nokia Store where we already got more than 6000 downloads and 100 reviews with a quite good average rating.

You have an example recorded by me with my own phone using the Historical effect and uploaded to Youtube:

And you have even already other videos uploaded to Youtube talking about how Aura works. This one is from a brazilian guy (obrigado!) for FaixaMobi and shows more effects:

Of course, being it free sofware you can also compile it yourself with the code at GitHub and do not be afraid of contributing! The technologies we used were the camerabin element of GStreamer and Qt/QML for the interface where we have the following components:

Aura components UML diagram

  • Main view (aura.qml) with the main interface
  • Controller, which is a mixed QML/C++ object allowing to control the pipeline.
  • Pipeline is a C++ object used by the controller to encapsulate the GStreamer code.
  • PostCapture is also a mixed QML/C++ object that opens the gallery application to show the recorded video and gives you the oportunity of sharing it, deleting it and so on. It uses a C++ controller loaded as a singleton to the context to do some stuff that can only be done in C++. Of course, you can open Gallery yourself and the videos will show up there.
  • EffectManager is a C++ class to load and manage the Effects, which is another C++ class defining how the effect must be applied.
  • Effects (Effects.qml) is a QML component to show the different effects, both software and hardware that Aura can apply. It uses the EffectManager (through the context) to load them and the Controller to apply them.
  • About view (AboutView.qml) is a rework of something done by my colleage Simón Pena and adapted to be used in Aura (Kudos!). It also uses a small AboutViewController to open a Nokia Store URL with the application instead of the browser.
  • ResourceManager is a C++ class used by the Controller to request the proper permissions to record the video.
by calvaris

Folga 29 Marzal

Hoxe estou de folga en contra da reforma laboral. Aínda que teño a sorte de traballar nunha empresa na que non me teño que preocupar por se vou ser despedido xa que son socio dela, considero o meu deber loitar tamén polos dereitos dos demais e tratar de botar para atrás algo que vai acentuar a precarización do traballo e sumirnos aínda máis na maldita crise. Eu achegareime á
manifestación da Praza de Vigo á convocada pola CIG, metade por quizais algo máis de afinidade política, como outra metade por aletoriedade a causa do patetismo que me causa que non se poñan dacordo por algo tan importante coma isto.

by calvaris

Invoking Meego 1.2 Harmattan Gallery

As part of my work at Igalia I am writing an app to record some videos for the Nokia N9. I wanted to get them shown in the gallery so I tried the DBUS approach, that is something similar to what is explained here. The problem is the same they faced, meaning that I got gallery in foreground when it was not previously running, but not otherwise.

The solution to that was using libcontentaction. In order to get this working, first you need to get the vendor name in the tags of your pictures or videos, otherwise Tracker will not index them correctly and this solution can be useless.

With the following solution Gallery will be brought to foreground and show the desired file. Code would be something like:

using ContentAction::Action;

[...]

{
    Action action =
        Action::defaultActionForFile(uri,
            "x-maemo-nepomuk/device-captured-video");
    if (action.isValid()) {
        qDebug() < < Q_FUNC_INFO << "chosen action:" << action.name();
        action.triggerAndWait();
    } else {
        qWarning() << "could not file action for" << m_file;
    }
}

The mime type is one of the defined in the galleryserviceaction.desktop that you can find in the device. For images, you can also check the mime types in that file and maybe you do not need to specify it, but I have not tried this and let it to you. Please, comment me your findings.

by calvaris

Mixed QML/C++ objects reloaded

Some days ago I was writing about how to have mixed QML/C++ objects in a QML application for Igalia, but I faced a problem with the code that I had written. And the problem was that I needed to receive some events and from the QML elements I was loading. Retaking that code:

#define QML_PATH "/path/to/the/qml/files/"

MyObject::MyObject(QDeclarativeItem *parent = 0) :
    QDeclarativeItem(parent)
{
    QDeclarativeEngine engine;
    QDeclarativeComponent component(&engine,
        QUrl::fromLocalFile(QML_PATH "MyObject.qml"));
    QDeclarativeItem *rect =
        dynamic_cast(component.create());
    rect->setParentItem(this);
}

and being MyObject.qml something like:

Button {
    id: "myButton"
    text: "Do something cool"
}

The natural way of connecting that button to some slot you are writing would be something like:

MyObject::MyObject(QDeclarativeItem *parent = 0) :
    QDeclarativeItem(parent)
{
    QDeclarativeEngine engine;
    QDeclarativeComponent component(&engine,
        QUrl::fromLocalFile(QML_PATH "MyObject.qml"));
    QDeclarativeItem *item =
        dynamic_cast(component.create());
    item->setParentItem(this);
    connect(item, SIGNAL(clicked()), this, SLOT(doCoolStuff()));
}

Even if you do this, you are not getting any event in the QML button you declare. It behaves as it were a label or whatever, but the key is the engine. The engine must live while you need the events of the QML component. And the easiest way of getting that done is adding the engine as a private class attribute:

#ifndef MYOBJECT_H
#define MYOBJECT_H

#include 
#include 
#include 

class PostCapture : public QDeclarativeItem
{
    Q_OBJECT

// ...

 private slots:
    void doCoolStuff();

 private:
    QDeclarativeEngine m_engine;
};
#endif

and of course removing the engine stack declaration from the constructor:

MyObject::MyObject(QDeclarativeItem *parent = 0) :
    QDeclarativeItem(parent)
{
    QDeclarativeComponent component(&m_engine,
        QUrl::fromLocalFile(QML_PATH "MyObject.qml"));
    QDeclarativeItem *item =
        dynamic_cast(component.create());
    item->setParentItem(this);
    connect(item, SIGNAL(clicked()), this, SLOT(doCoolStuff()));
}

Voilà.

Of course, if you need to connect to any element that is not the root element, you can always forward the signals and properties from the QML root object or just use the QObject::findChild method to access the right component.

by calvaris

Attended FOSDEM 2012

FOSDEM 2012

This was my first time at FOSDEM (thanks to Igalia for the sponsoring) and I was really impressed by the lots of tracks that you can find there, but as there are so many, you realize that there are many you are not interested at. The feeling is only a bit more intense than at GUADEC or Desktop Summit, because for your interestest, you target more or less the same kind of talks, but in this case, offer is a bit broader.

My thoughts:

  • WebKit and WebKitGtk are in a good shape.
  • We are going to more web based desktops.
  • Tizen is taking off and it is betting for HTML5 and ELF, which looks like a toolkit with very interesting features like Edje being able to write an app with different views for different platforms in an easy way (at least in the paper, because I couldn’t try it).
  • GStreamer is in the right way to 1.0 by improving APIs and features that can boost performance in embedded.
  • I would like to have gone to Wayland talks, but I could not.
  • CERN really rocks working with FLOSS, specially on drivers and even developing their own hardware for their needs. It seems they have the policy of writing and using FLOSS because of being a publicly funded organization. Many other could follow their advice.

About tourist stuff, it was really a pity to be taking some medicins that prevented me to drink the famous Belgian beer, but at least I could visit some famous stuff in Brussels, like the Atomium, Manneken Pis and the Grote Markt. What medicins did allow me was trying the famous waffles and some good chocolate made by Frederic Blondeel.