{"id":11,"date":"2021-12-20T19:45:50","date_gmt":"2021-12-20T22:45:50","guid":{"rendered":"http:\/\/blogs.igalia.com\/gpiccoli\/?p=11"},"modified":"2023-02-12T12:23:19","modified_gmt":"2023-02-12T15:23:19","slug":"booting-upstream-kernel-on-inforce-6640","status":"publish","type":"post","link":"https:\/\/blogs.igalia.com\/gpiccoli\/2021\/12\/booting-upstream-kernel-on-inforce-6640\/","title":{"rendered":"Booting upstream kernel on Inforce 6640"},"content":{"rendered":"\nMy first task in the Core team<\/a> at Igalia<\/a> was to boot a more recent kernel in the Inforce 6640 board<\/a> , which was very interesting given my very limited experience working in the ARM64 world – also, the work aimed to benefit Igalia’s Graphics team<\/a> in their Freedreno<\/a> development. The board itself is pretty powerful and contains a lot of resources; it comes with a quad-core Snapdragon\u2122 820 processor, Adreno\u2122 530 GPU, 4G of RAM plus a fast 64G UFS storage. A picture of my unit below:\n\n\n\n\n
<\/div>\n\n\n\n\n\"i6640_pic\"
\n\n\n\n\n
<\/div>\n\n\n\n\nSo, I got a packed Debian image with 4.14 kernel and after some looking, noticed that the kernel was a modified \/ custom version from Linaro, which they call Qualcomm LT (Landing Tree)<\/a>. Hence, the first idea was to boot this image and see how well it worked. And unsurprisingly, it was working flawlessly – given this was provided by the board vendor, the expectation was that good amount of validation was performed, so we had a starting point here – the working version was the Linaro’s Qualcomm-LT 4.14. In a rush of optimism, I tried to test the latest released mainline kernel at the moment (v5.14) and…it didn’t work! Why would it, right? Where would be the fun?\n\n\n\n\n
<\/div>\n\n\n\n\nSo, next step was to bisect it and I decided to start by using the Linaro’s LT tree – it failed on 5.7+, but succeeded to boot 5.4, although the board seemed pretty slow! First thought was likely a problem in the storage, like a bad device tree config for UFS (the storage device) or a driver change between 4.14 and 5.4. But before looking into that, we needed to understand why kernels 5.4+ weren’t booting…so how to proceed here, if ssh wasn’t working?\n\n\n\n\n
<\/div>\n\n\n\n\nYeah…exactly: serial output to the rescue! So I got some jumper cables and decided to use a Raspberry Pi as the other end for serial connection. By connecting all the proper cables and using “minicom”, I could boot a 5.7 kernel and check the progress..and guess what? Kernel was booting normally!!!
But without PCIe – no network \/ GPU, for example. Also, it was still very slow. And then, an epiphany: it was the device-tree! By dumping the device-tree (from procfs) in both the working 4.14 and the non-working 5.7, a bunch of stuff were missing, like all the PCI structures. But why is that? Somebody just removed stuff from a working device-tree?\n\n\n\n\n
<\/div>\n\n\n\n\nNah, it was just an obvious thing I didn’t realize before: up to kernel 5.6, the Inforce 6640 board was relying on the their “cousin’s” device-tree, the Dragonboard 820c! A pretty similar board, so relying on its DT made the board working like a charm, but in kernel 5.6 a proper device-tree for Inforce 6640 was added<\/a>, although it’s very simple and minimal to boot the board. In order to have a fully working system after that finding, all we needed was to force the usage of the “old” db820c.dts, with the following hack patch:\n\n\n\n\n
<\/div>\n\n\n\n\n\n
--- a\/arch\/arm64\/boot\/dts\/qcom\/Makefile\n+++ b\/arch\/arm64\/boot\/dts\/qcom\/Makefile\n dtb-$(CONFIG_ARCH_QCOM)        += apq8096-db820c.dtb\n-dtb-$(CONFIG_ARCH_QCOM)        += apq8096-ifc6640.dtb\n+#dtb-$(CONFIG_ARCH_QCOM)       += apq8096-ifc6640.dtb\n dtb-$(CONFIG_ARCH_QCOM)        += ipq6018-cp01-c1.dtb<\/code><\/pre>\n\n\n\n\nOK, so now we have a working upstream kernel (with network!), so let’s investigate the remaining: why the board is so slow? And what about graphics, is it working? I started with the graphics and again, had some issues: board had graphics working on Linaro’s LT tree (v5.13), but not in the upstream 5.13\/5.14. Another round of bisects later plus code inspection, and I found the “culprit”: a downstream patch<\/a> that was also required in order to make the GPU work. The patch added a regulator with “always-on” property and a higher voltage than found in upstream. So, changed that and voil\u00e1: we had now a working 5.14 with graphics but…still slow! Moving then to the next journey: why the newer kernels are so much slower?\n\n\n\n\n
<\/div>\n\n\n\n\nIn order to investigate that, I began by comparing the db-820c device-trees among versions, and despite there were differences, nothing rang a bell. So, jumped to sysfs and started investigating there…when it comes to my attention that the CPUs were running slow, very<\/em> slow – CPUs frequency were tiny . Taking a look in the commits and checking cpufreq drivers, the troublemaker was found: after kernel 4.18, a patch<\/a> added a driver for handling OPP<\/a> voltage scaling for Kryo CPUs, which is the case for Snapdragon\u2122 820. After loading this new module, CPU clocks were back to normal\/fast rates, the board was as fast (or even more) than running the original Linaro’s LT 4.14 downstream kernel.\n\n\n\n\n
<\/div>\n\n\n\n\nBut… (yeah, another<\/em> but)…after some simple CPU benchmarks, just some seconds after the beginning of the tests, board rebooted into a firmware error state. It was like as if it couldn’t handle all the CPU speed in the newer kernels. Here, I used the very powerful ftrace tooling to determine the reason of this behavior. And guess what was found?\n\n\n\n\n
<\/div>\n\n\n\n\nCooling! The responsible for that was kernel lack of throttling. The following trace snippet shows how the governor would reduce the CPU clock\/voltage in order to preserve CPUs health:\n\n\n\n\n
<\/div>\n\n\n\n\n\n
 => cpufreq_governor_limits.part.6\n => cpufreq_set_policy\n => cpufreq_update_policy\n => cpufreq_set_cur_state\n => thermal_cdev_update\n => step_wise_throttle\n => handle_thermal_trip\n => thermal_zone_device_update.part.10\n => thermal_zone_device_check\n => process_one_work\n => worker_thread<\/code><\/pre>\n\n\n\n\nThe key here is the function  step_wise_throttle<\/code> which downscale the CPU frequency to prevent from overheating. In order to have it properly working, cooling maps\/trip points are required in the device-tree, it’s present in some downstream patches on Linaro’s kernel – but when that work was upstream’ed, it lacked some portions, hence if we have cpufreq driver and high frequencies, we’re prone to overheating \/ firmware reboots. The patch that fixes that was re-submitted by another person and I tried to get that accepted<\/a>, by “lobbying” in the mailing-list and showing our test results, but up to 5.16-rc6 it isn’t present in Linus tree.\n\n\n\n\n
<\/div>\n\n\n\n\nFinally, worth to mention 2 things: for some reason, building a kernel with ftrace after<\/em> v5.7 prevents it from booting. This led to a very interesting debug, like debug-by-rebooting (using a firmware triggered reboot in assembly), (re-)implementing a very early console to show output much before “earlycon” is available – stay tuned for a blog post about that! Second, we still lack that “regulator-always-on” necessary to have a functional GPU – after submitting a patch<\/a> for that, I’ve discussed that with the maintainer, and it seems we should have the user of such regulator to request it properly – due to other tasks I couldn’t continue this work. So, in summary, if you want to boot an upstream kernel (5.14+) on Inforce 6640, the steps are:\n\n\n\n\n
<\/div>\n\n\n\n\n\n