Share my experience to build Chromium with ICECC

If you’re a Chromium developer, I guess that you’ve suffered from the long build time of Chromium like me. Recently, I’ve set up the icecc build environment for Chromium in the Igalia Korea office. Although there have been some instructions how to build Chromium with ICECC, I think that someone might feel they are a bit difficult. So, I’d like to share my experience how to setup the environment to build Chromium on the icecc with Clang on Linux in order to speed up the build of Chromium.

P.S. Recently Google announced that they will open Goma (Google internal distributed build system) for everyone. Let’s see goma can make us not to use icecc anymore ūüėČ
https://groups.google.com/a/chromium.org/forum/?utm_medium=email&utm_source=footer#!msg/chromium-dev/q7hSGr_JNzg/p44IkGhDDgAJ

Prerequisites in your environment

  1. First, we should install the icecc on your all machines.
    sudo apt-get install icecc [icecc-monitor]
  2. To build Chromium using icecc on the Clang, we have to use some configurations when generating the gn files. To do it as easily as possible, I made a script. So you just download it and register it to $PATH. 
    1. Clone the script project
      $ git clone https://github.com/Gyuyoung/ChromiumBuild.git

      FYI, you can see the detailed configuration here –¬†https://github.com/Gyuyoung/ChromiumBuild/blob/master/buildChromiumICECC.sh

    2. Add your chromium/src path to buildChromiumICECC.sh
      # Please set your path to ICECC_VERSION and CHROMIUM_SRC.
      export CHROMIUM_SRC=$HOME/chromium/src
      export ICECC_VERSION=$HOME/chromium/clang.tar.gz
    3. Register it to PATH environment in .bashrc
      export PATH=/path/to/ChromiumBuild:$PATH
    4. Create a clang toolchain from the patched Chromium version
      I added the process to “sync” argument of buildChromiumICECC.sh. So please just execute the below command before starting compiling. Whenever you run it, clang.tar.gz will be updated every time against the latest Chromium version.

      $ buildChromiumICECC.sh sync

Build

  1. Run an icecc scheduler on a master machine,
    sudo service icecc-scheduler start
  2. Then, run an icecc daemon on each slave machine,
    sudo service iceccd start

    If you run icemon, you can monitor the build status on the icecc.

  3. Start building in chromium/src,
    $ buildChromiumICECC.sh Debug|Release

    When you start building Chromium, you’ll see that the icecc works on the monitor!

Build Time

In my case, I’ve¬† been using 1 laptop and 2 desktops for Chromium build. The HW information is as below,

  • Laptop (Dell XPS 15″ 9560)
    1. CPU: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
    2. RAM: 16G
  • Desktop 1
    1. CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
    2. RAM: 16G
  • Desktop 2
    1. CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
    2. RAM: 8G

I’ve measured how long time I’ve spent to build Chromium with the script in below cases,

  • Laptop
    • Build time on Release with the jumbo build : About 84 min
    • Build time on Release without the jumbo build : About 203 min
  • Laptop + Desktop 1 + Desktop 2
    • Build time on Release with the jumbo build : About 35 min
    • Build time on Release without the jumbo build :¬† About 73 min

But these builds haven’t applied any object caching by ccache yet. If ccache works next time, the build time will be reduced. Besides, the build time is depended on the count of build nodes and performance. So this time can differ from your environment.

Troubleshooting (Ongoing)

  1. Undeclared identifier
    1. Error message
      /home/gyuyoung/chromium/src/third_party/llvm-build/Release+Asserts
       /lib/clang/6.0.0/include/avx512vnniintrin.h:38:20:
       error: use of undeclared identifier '__builtin_ia32_vpdpbusd512_mask'
       return (__m512i) __builtin_ia32_vpdpbusd512_mask ((__v16si) __S, ^
    2. Solution
      Please check if the path of ICECC_VERSION was set correctly.
  2. Loading error libtinfo.so.5
    1. Error message
      usr/bin/clang: error while loading shared libraries: libtinfo.so.5: 
      failed to map segment from shared object
    2. Solution
      Not find a correct fix yet. Just restart the build for now.

Reference

  1. WebKitGTK SpeedUpBuild: https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild
  2. compiling-chromium-with-clang-and-icecc : http://mkollaro.github.io/2015/05/08/compiling-chromium-with-clang-and-icecc/
  3. Rune Lillesveen’s icecc-chromium project:
    https://github.com/lilles/icecc-chromium
Posted in Igalia-Chromium | Tagged , , , | Leave a comment

What is navigator.registerProtocolHandler?

Have you heard that navigator.registerProtocolHandler javascript API?

The API is to give you the power to add your customized scheme(a.k.a. protocol) to your website or web application. If you register your own custom scheme, it can help you to avoid collisions with other DNS, or help other people use it correctly, and be able to claim the moral high ground if they use it incorrectly. In this post, I would like to introduce the feature with some examples, and what I’ve contributed to the feature.

Introduction

The registerProtocolHandler had been started discussing since 2006. Finally, it was introduced in HTML5 for the first time.

This is a simple example of the use of navigator.registerProtocolHandler. Basically, the registerProtocolHandler takes three arguments,

  • scheme – The scheme that you want to handle. For example, mailto, tel, bitcoin, or irc.
  • url – A URL within your web application that can handle the specified scheme.
  • title – The title of the handler.
<script>
     navigator.registerProtocolHandler("web+search",
                                       "https://www.example/search/url=%s",
                                       "web search");
</script>
<a href="web+search:igalia">Igalia</a>

As this example, you can register a custom scheme with a URL that can be combined with the given URL(web+search:igalia)

However, you need to keep in mind some limitations when you use it.

  1. URL should include %s placeholder.
  2. You can register own custom handler only when the URL of the custom handler is same the website origin. If not, the browser will generate a security error.
  3. There are pre-defined schemes in the specification. Except for these schemes, you’re only able to use “web+foo” style scheme. But, the length of “web+foo” is at least five characters including ‘web+’ prefix. If not, the browser will generate a security error.
    • bitcoin, geo, im, irc, ircs, magnet, mailto, mms, news, nntp, openpgp4fpr, sip, sms, smsto, ssh, tel, urn, webcal, wtai, xmpp.
  4. Schemes should not contain any colons. For example, mailto: will generate an error. So, you have to just use mailto.

Status on major browsers

Firefox (3+)

When Firefox meets a webpage which includes navigator.registerProtocolHandler as below,

navigator.registerProtocolHandler("web+search",
                                  "https://www.example/search/url=%s",
                                  "web search");

It will show an alert bar just below the URL bar. Then, it will ask us if we want to add the URL to the Firefox handler list.

After allowing to add it to the supported handler list, you can see it in the Applications section of the settings page (about:preferences#general).

Now, the web+github custom scheme can be used in the site as below,

<a href="web+github:wiki">Wiki</a>
<a href="web+github:Pull requests">PR</a>

Chromium (13+)

Let’s take a look Chrome on the registerProtocolHandler. In Chrome, Chrome shows a new button inside the URL bar instead of the alert bar in Firefox.

If you press the button, a dialog will be shown in order to ask if you allow the website to register the custom scheme.

In the “Handlers” section of the settings (chrome://settings/handlers?search=protocol), you’re able to see that the custom handler was registered.

FYI, other browsers based on Chromium (i.e. Opera, Yandex, Whale, etc.) have handled it similar to Chrome unless each vendor has own specific behavior.

Safari (WebKit)

Though WebKit has supported this feature in WebCore and WebProcess of WebKit2, no browsers based on WebKit have supported this feature with UI. Although I had tried to implement it in UIProcess of WebKit2 through the webkit-dev mailing list so that browsers based on WebKit can support it, Unfortunately, some of Apple engineers had doubts about this feature though there were some agreements to support it in WebKit. So I failed to implement it in UIProcess of WebKit2.

My contributions in WebKit and Chromium

I’ve mainly contributed to apply new changes in the specification, to fix bugs and improve test cases. First,¬†I’d like to introduce the bug that I modified in Chromium recently.

Let’s assume that URL has multiple placeholders(%s).

navigator.registerProtocolHandler("test",
                                  "http://example.com/%s/url=%s",
                                  "title");" 
<a href="test:duplicated_placeholders">this</a>

According to the specification, only first “%s” placeholder should be¬†substituted, not¬†substitute all placeholders. But, Chrome has substituted all placeholders with the given URL as below, even though Firefox has only substituted the first placeholder.

http://example.com/test%3Aduplicated_placeholders/url=test%3Aduplicated_placeholders

So I fixed the problem in¬†[registerProtocolHandler] Only substitute the first “%s” placeholder. The latest Chromium substitutes only first placeholder like below,

http://example.com/test%3Aduplicated_placeholders/url=%s

This is a whole list what I’ve contributed both WebKit and Chromium so far.

  1.  WebKit
  2. Chromium

Summary

So far, we’ve looked at the navigator.registerProtcolHandler for your web application simply. The API can be useful if you want to make users use your web applications like a web-based email client or calendar.

Posted in Igalia-Chromium | Leave a comment

How to make your code on downstream ?

Nowadays most of my projects have been using opensource or worked based on opensource. If we just needs to use it, I think it would be a good situation for you. However we usually have projects that keep the opensource over years for your products. So if you have to hack a lot of modules inside of opensource code, it can be a nightmare when you rebase current source base against the latest opensource after months or years. In this article I would like to share some of my experiences on how to make a downstream patch when we work using opensource.

  1. Try to contribute your patch to the opensource project as much as possible

    • I think this is the best way to reduce our heavy burden that we should maintain in your downstream source code. Even if the opensource project you’re using is being developed fast, you will often face many conflicts during the rebase because it’s likely that original code or architecture have changed frequently in the meantime. To avoid the conflict, it would be best if you contribute your patch to the opensource project as much as possible.
    • Below documents are a good example to explain how to contribute your code to the opensource project.
  2. Make your downstream port

    • We know well that #1 is the best way to reduce our downstream patches though, it is often hard to keep because downstream patches are often too hacky or unstable. Even if you submit a downstream patch to upstream, you might get many review comments or objections from the opensource maintainers. In such case, what else can you do ? In my experience, it was important to separate our downstream implementation from the original code. For example, we can make new TriangleFoo.h/cpp files instead of original Triangle.h/cpp files, then we can use them through the modification of few build scripts.
      1. Figure class
       class Figure {
        public:
            virtual int calculateSize();
       }
       
      2. Triangle class
       class Triangle : public Figure {
        public:
            virtual int calculateSize() override;
       }
      
       Triangle::calculateSize() {
           return width * height / 2;
       }
      
      3. TriangleFoo class
       class TriangleFoo : public Figure {
           public: virtual int calculateSize() override;
       }
       
       TriangleFoo::calculateSize() {
           return new_width * height / 2;
       }
      
      4. Build script. In this example we use cmake,
       list(APPEND Figure_SOURCES
           Figure.cpp
           Triangle.cpp
           TriangleFoo.cpp
       )
       
       list(REMOVE_ITEM Figure_SOURCES
           Triangle.cpp
       )

      We can avoid some conflicts in the next rebase with the latest opensource in the Triangle.h/cpp files. However we still need to modify the TriangleFoo.h/cpp if Figure.h is changed. For example, when a new parameter is added or a return type is changed.

  3. Use #if ~ #endif guard

    • When we only need to modify few lines or just to change logic inside a function, we can use #if ~ #else ~ #endif guard. The guard can help us to know what codes were added by us or modified by us. Besides it might be help us to check easily if the downstream patch generated side effects through turning it off. In my previous projects, most of issues have come from downstream patches because they lacked code review, missed test cases, or were too hacky against original architecture. In such cases, you can check the issue just by turning the guard off. However, if you use #if ~ #endif guard in many places, the usages can mess your code up. So I’d like to recommend you use it only when you really need.
      Triangle::calculateSize()
      {
      #if defined(DOWNSTREAM_ENABLED)
          return new_width * height / 2;
      #else
          return width * height / 2;
      #endif
      }
  4. Try to make a patch per a feature

    • As you may have experienced before, it is very hard to implement a feature with a commit. Even though you succeed in implementing a new feature with a commit perfectly, you may face to touch the implementation again in order to fix a bug or apply new requirements again. In such case your git history will get messier and messier. It will make it difficult to rebase based on the latest opensource. To avoid it, you may have manually merged original implementation with the fixup commits reflected later. But there are two useful git commands for this case – git commit fixup and git rebase –autosquash.
        • git commit –fixup : Automatically marks your commit as a fix of a previous commit.¬†Construct a commit message for use with git rebase –autosquash.
        • git rebase¬†-i¬†–autosquash¬†: Automatically organize merging of these fixup commits and associated normal commits.

    • Example
      There is a good article to explain that explains method [1]. If you need to understand further, it would be good if you visit the URL. Let’s assume that we have 3 commits on your local repository.

      $ git log --oneline
        new commit1 (7ae79f6)
        new commit2 (9e4c1de)
        new commit3 (480ee07)
        previous commit (19c8abf)

      But if we just noticed that we missed to add a comment in commit2, it’s time to use¬†--fixup¬†option.

      $ git add [modified file]
      $ git commit --fixup [new commit2's commit-id]
        (i.e. git commit --fixup 9e4c1de)

      Then you can clean your branch before merging it using - autosquash option.

      $ git rebase -i --autosquash [previous commit id]
        (i.e. git rebase -i autosquash 19c8abf)

 

Reference
[1] http://fle.github.io/git-tip-keep-your-branch-clean-with-fixup-and-autosquash.html

Posted in opensource | Tagged , | Leave a comment