Share my experience to build Chromium with ICECC and Jumbo

If you’re a Chromium developer, I guess that you’ve suffered from the long build time of Chromium like me. Recently, I’ve set up the icecc build environment for Chromium in the Igalia Korea office. Although there have been some instructions how to build Chromium with ICECC, I think that someone might feel they are a bit difficult. So, I’d like to share my experience how to set up the environment to build Chromium on the icecc with Clang on Linux in order to speed up the build of Chromium.

P.S. Recently Google announced that they will open Goma (Google internal distributed build system) for everyone. Let’s see goma can make us not to use icecc anymore 😉
https://groups.google.com/a/chromium.org/forum/?utm_medium=email&utm_source=footer#!msg/chromium-dev/q7hSGr_JNzg/p44IkGhDDgAJ

Prerequisites in your environment

First, we should install the icecc on your all machines.
```
sudo apt-get install icecc ccache [icecc-monitor]
```
To build Chromium using icecc on the Clang, we have to use some configurations when generating the gn files. To do it as easily as possible, I made a script. So you just download it and register it to $PATH.
1. Clone the script project
```
$ git clone https://github.com/Gyuyoung/ChromiumBuild.git
```
  FYI, you can see the detailed configuration here – https://github.com/Gyuyoung/ChromiumBuild/blob/master/buildChromiumICECC.sh
2. Add your chromium/src path to buildChromiumICECC.sh
```
# Please set your path to ICECC_VERSION and CHROMIUM_SRC.
export CHROMIUM_SRC=$HOME/chromium/src
export ICECC_VERSION=$HOME/chromium/clang.tar.gz
```
3. Register it to PATH environment in .bashrc
```
export PATH=/path/to/ChromiumBuild:$PATH
```
4. Link Clang to icecream so that Clang gets redirected trough Icecream
```
$ cd /usr/lib/icecc/bin
$ sudo ln -s /usr/bin/icecc ./clang
$ sudo ln -s /usr/bin/icecc ./clang++
```
5. Add the patched Chromium version of the Clang executable to your PATH right after Icecream
```
$ export PATH="/usr/lib/icecc/bin:$HOME/chromium/src/third_party/llvm-build/Release+Asserts/bin/:$PATH"
```
6. Create a clang toolchain from the patched Chromium version
  I added the process to “sync” argument of buildChromiumICECC.sh. So please just execute the below command before starting compiling. Whenever you run it, clang.tar.gz will be updated every time with the latest Chromium version.
```
$ buildChromiumICECC.sh sync
```

Build

Run an icecc scheduler on a master machine,
```
sudo service icecc-scheduler start
```
Then, run an icecc daemon on each slave machine
```
sudo service iceccd start
```
If you run icemon, you can monitor the build status on the icecc.
Start building in chromium/src
```
$ buildChromiumICECC.sh Debug|Release
```
When you start building Chromium, you’ll see that the icecc works on the monitor!

Build Time

In my case, I’ve been using 1 laptop and 2 desktops for Chromium build. The HW information is as below,

Laptop (Dell XPS 15″ 9560)
1. CPU: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
2. RAM: 16G
3. SSD
Desktop 1
1. CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
2. RAM: 16G
3. SSD
Desktop 2
1. CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
2. RAM: 16G
3. SSD

I’ve measured how long time I’ve spent to build Chromium with the script in below cases,

Laptop
- Build time on Release with the jumbo build: About 84 min
- Build time on Release without the jumbo build: About 203 min
Laptop + Desktop 1 + Desktop 2
- Build time on Release with the jumbo build: About 35 min
- Build time on Release without the jumbo build: About 73 min

But these builds haven’t applied any object caching by ccache yet. If ccache works next time, the build time will be reduced. Besides, the build time is depended on the count of build nodes and performance. So this time can differ from your environment.

Troubleshooting

Undeclared identifier

Error message

/home/gyuyoung/chromium/src/third_party/llvm-build/Release+Asserts
 /lib/clang/6.0.0/include/avx512vnniintrin.h:38:20:
 error: use of undeclared identifier '__builtin_ia32_vpdpbusd512_mask'
 return (__m512i) __builtin_ia32_vpdpbusd512_mask ((__v16si) __S, ^

Solution
Restart to icecc-scheduler and all icecc nodes.

Out of Memory

Error message
```
LLVM ERROR: out of memory
```

Solution

Add more physical RAM to the machine.

Alternatively, we can try to increase the space of swap. (But, this may not solve the OOM issue. In such a case, we should increase the physical RAM)

For example, create a 4G swap file

$ size="4G" && file_swap=/swapfile_$size.img && sudo touch $file_swap && sudo fallocate -l $size /$file_swap && sudo mkswap /$file_swap && sudo swapon -p 20 /$file_swap

Make the swap file permanent

# in your /etc/fstab file
/swapfile    none    swap    sw,pri=10      0       0
/swapfile_4G.img     none    swap    sw,pri=20      0       0

Check swap situation after reboot

$ sudo swapon  -s
Filename       Type     Size        Used    Priority
/swapfile      file     262140      0       10
/swapfile_4G.img       file     4194300     0       20