{"id":91,"date":"2024-07-09T16:21:05","date_gmt":"2024-07-09T19:21:05","guid":{"rendered":"https:\/\/blogs.igalia.com\/gpiccoli\/?p=91"},"modified":"2024-07-09T16:21:05","modified_gmt":"2024-07-09T19:21:05","slug":"presenting-kdumpst-or-how-to-collect-kernel-crash-logs-on-arch-linux","status":"publish","type":"post","link":"https:\/\/blogs.igalia.com\/gpiccoli\/2024\/07\/presenting-kdumpst-or-how-to-collect-kernel-crash-logs-on-arch-linux\/","title":{"rendered":"Presenting kdumpst, or how to collect kernel crash logs on Arch Linux"},"content":{"rendered":"\nIt\u2019s been a long time since I last posted something here \u2013 yet there are interesting news to mention, a bit far from ARM64 (the topic of my previous posts). Let\u2019s talk today about kernel crashes, or even better, how can we collect information if a kernel panic happens on Arch Linux and on SteamOS, the Linux distribution used on the Steam Deck.\n\n\n\n\n
<\/div>\n\n\n\n\nFirst all, it\u2019s worth mentioning some ambiguity in the terms used here. Usually \u201ckernel panic\u201d and \u201ccrash\u201d (or even \u201coops\u201d) are terms used as synonymous, though this is not entirely true. A kernel oops<\/strong> is the error state faced by the kernel in case it finds a bad condition; some usual conditions that lead to oopses are null pointer dereference, specially due to concurrency issues (like bad locking) or user-after-free \/ out-of-bounds accesses. BUG()<\/code> statements in the code and rogue DMA access (due to driver or HW issues) are also a common cause. A kernel panic<\/strong>, on the other hand, is the action the kernel takes once an oops is detected \u2013 and in most cases, the system is typically rebooted as a consequence of these errors.\n\n\n\n\n
<\/div>\n\n\n\n\nAnother possible action, though, is to configure the system to attempt a log collection in the face of these issues, in order to have a post-mortem<\/em> investigation, after a reboot to clean state. And that is where technologies like kdump and pstore come in handy!<\/a>\n\n\n\n\n
<\/div>\n\n\n\n\n\n

Pstore x Kdump – pros and cons<\/strong><\/p>\n\n\n\n\nPstore<\/strong> is a mechanism to have the kernel write some information in a persistent storage backend. The most common scenario is to write the dmesg after an oops condition to the pstore backend, which could be the RAM, a UEFI variable, or even a block device. There are other backends (ACPI ERST tables, for example) and other frontends as well, like the console or ftrace ring buffer, though they\u2019re out of the scope of this introductory post.\n\n\n\n\n

<\/div>\n\n\n\n\nKdump<\/strong> on the other hand is a technology in which the kernel jumps to a new kernel (called \u201ccrash kernel\u201d) when some (user-configurable) condition is observed, like an oops. The \u201cjump\u201d to the new kernel happens through the kexec<\/strong> kernel mechanism, bypassing the FW initialization, so the crash kernel has complete visibility of the state of the old, \u201cbroken\u201d kernel, and is able to collect not only logs, but the full memory image, called vmcore<\/code>. Such memory image could be trimmed to remove userspace data and free pages, and it is usually compressed. All of that is configurable, and the details will be discussed in a future blog post.\n\n\n\n\n
<\/div>\n\n\n\n\nSo, what is the best option?<\/strong> Which one, between pstore and kdump, should be used? As usual in computing (and for everything in life, usually), it depends<\/strong>. The trade-offs here are related to risks involved, which differ between kdump and pstore, the amount of information you want to collect (hence, store!) and the memory \/ time resources one is willing to spend on that.\n\n\n\n\n
<\/div>\n\n\n\n
\n
\"\"<\/figure><\/div>\n\n\n\n
<\/div>\n\n\n\n\nStarting with the resources: kdump requires the kernel to pre-reserve<\/strong> a fixed amount of RAM, that\u2019s never used by the system, regardless of oopses \u2013 and that amount is non-trivial, specially for embedded or any low-resource system, lying in the range of 150-250MB for newer kernels. As per the data collected, of course kdump is capable of collecting more information, in the form of the vmcore<\/code>, but even compressed, the data might be multiple GBs, while a compressed kernel dmesg (that can be enhanced with some extra information, detailed below) is in the order of kBs, maybe a few MBs tops.\n\n\n\n\n
<\/div>\n\n\n\n\nFinally, the risks<\/strong> are present in both pstore and kdump, though they differ a bit: kdump relies on kexec<\/strong>, which is a fast reboot mechanism that bypasses the FW, for the good and the bad. Kexec is for sure faster and more lightweight than a full firmware reboot, but at the same time, we don\u2019t have a standard protocol for PCIe HW resetting, specially on more common architectures like x86 and ARM, so we need to rely on drivers to proper restart the devices; this may lead to issues in the boot time of the crash kernel, or to HW malfunctioning in such kernel, that could incur in a lost opportunity to collect logs, if the storage device doesn\u2019t work fine without a FW reboot, for example, causing the vmcore<\/code> saving step to fail.\n\n\n\n\n
<\/div>\n\n\n\n\nFrom the pstore perspective<\/strong>, the mechanism relies on having the broken kernel to write logs to a persistent storage, so this is inherently unsafe; also, backends may be tricky to deal with, the RAM case being very emblematic. Lots of FW implementations either retrain or just purely corrupt the RAM on boot process, so pstore logs saved to RAM might be useless in some systems. UEFI variables are more stable and should be really persistent, but your mileage may vary: some UEFI implementations may not behave properly, or there may be almost no free space to store even a small dmesg in the UEFI area.\n\n\n\n\n
<\/div>\n\n\n\n\nAll in all, it\u2019s a matter of considering the trade-offs and experimenting<\/strong>. I’ve made a presentation about this topic in the Linux Plumbers 2023, so for the interested readers I suggest you to take a look in the recording<\/a>! Now, for the sake of experimenting, here comes the kdumpst<\/a> tool: an automatic log collection tool for Arch Linux and SteamOS that relies on both pstore and kdump!<\/strong><\/a>\n\n\n\n\n
<\/div>\n\n\n\n\n\n

Presenting kdumpst<\/strong><\/p>\n\n\n\n\nMost common Linux distros come with a kdump tool, like kdump-tools<\/a> for Debian (and Ubuntu), Fedora\u2019s kdump<\/a>, etc. Arch Linux, on the other hand, is missing some high level utility; the Arch wiki<\/a> even recommended a manual process for collecting kernel core dumps, setting everything up \u201cby hand\u201d. The lack of tooling is especially felt on SteamOS, the distribution used by Valve\u2019s Steam Deck, which is a port of Arch Linux. An interesting, and important, feature for SteamOS is the ability to collect logs or any information in case of a kernel crash; but how to ponder and choose between kdump or pstore? The Steam Deck has a powerful HW, but \u201cspending\u201d ~250MB of RAM in every boot just to set kdump seems an overkill; so why not take advantage of both?\n\n\n\n\n

<\/div>\n\n\n\n\nWell, that\u2019s exactly the purpose of kdumpst: to enable users to have a choice and determine in a case-by-case fashion what\u2019s the level of information they wish<\/strong> to have in case of a kernel crash \u2013 the more information is desired (a vmcore<\/code> being the gold standard), the more resources are required for that. So, kdumpst by default relies on pstore\u2019s RAM<\/strong> backend (though the UEFI backend is under implementation), but users can change it to kdump mode, so it will pre-reserve some RAM memory and load the crash kernel. The tool also makes use of some sysctl tunings to panic and reboot under some circumstances (oops, hard lockups, etc) and dump extra information on dmesg, like tasks\u2019 state and memory information (through the panic_print<\/code> tuning) \u2013 all of that part of kdumpst custom settings, which can be easily changed by the user.
\n\n\n\n\n
<\/div>\n\n\n\n\nI plan more detailed blog posts to describe the pstore mechanism and how to use it with kdumpst (and even some tricks out of the tool\u2019s scope), and the same for the kdump facility. But it is worth to give here some hints on how to make use of the tool and explore its capabilities on Arch Linux, a \u201ccrash course\u201d let\u2019s say :-]\n\n\n\n\n
<\/div>\n\n\n\n\n\n

Set-up and experimenting with kdumpst<\/strong><\/p>\n\n\n\n\nFirst of all, you should get the kdumpst package from the AUR and install it. After that, you must enable and start its service, by running\u00a0systemctl enable --now kdumpst-init<\/code>\u00a0. The default mode of operation for kdumpst is based on pstore with RAM backend, so naturally it\u2019ll try to find a region of memory unused by the kernel. It\u2019s not super common to have a region without allocating it previously, but sometimes due to x86 alignment, or a device-tree setting on ARM boards, it might be available.\n\n\n\n\n

<\/div>\n\n\n\n\nIf pstore is properly set, a message like the following would show up in the systemd journal:kdumpst: pstore-RAM was loaded successfully.<\/code>
You can check that using the command: journalctl -b | grep kdumpst<\/code>.\n\n\n\n\n
<\/div>\n\n\n\n\nIf pstore RAM mode fails, the tool will automatically attempts kdump, and the following 2 messages should be present in the logs:\n\n\n\n\n

kdumpst: kexec won't succeed, no reserved memory in this boot\u2026<\/code>
kdumpst: but we automatically set crashkernel for next boot.<\/code><\/p>\n\n\n\n\n\n

<\/div>\n\n\n\n\nThis is expected, since the crash kernel memory allocation requires to be done on boot time. So kdumpst will automatically set the command-line (GRUB only, for now), and on next boot, the following message should be present in the journal:
kdumpst: panic kexec loaded successfully<\/code>.\n\n\n\n\n
<\/div>\n\n\n\n\nIn both cases (pstore or kdump), if the logs report that kdumpst loaded successfully, the system should be ready for collecting the kernel log in case of a crash event. Testing isn\u2019t a bad idea, so how about triggering a \u201cfake\u201d panic to verify? For that, the next 2 commands should do the trick (run them as root user):
echo 1 > \/proc\/sys\/kernel\/sysrq<\/code>
echo c > \/proc\/sysrq-trigger<\/code>\n\n\n\n\n
<\/div>\n\n\n\n\nBe aware that these commands will cause a kernel panic, potentially leading to data corruption, so better to run it on an idle system!
\n\n\n\n\n
<\/div>\n\n\n\n\n
The effect should be a reboot; when the system finishes the initialization process, the following entry should be present in the journal:\n\n\n\n\n

kdumpst: logs saved in \"\/home\/.steamos\/offload\/var\/kdumpst\/logs\"<\/code><\/p>\n\n\n\n\n\n

<\/div>\n\n\n\n\nThat means the tool is properly configured and working, and collected the logs. A zip archive will be present in such directory, containing (among other files) a dmesg of the crash, with some extra information. In case you want to tune the configurables of kdumpst, just take a look in its configuration file, at\u00a0\/var\/share\/kdumpst.d\/00-default<\/code>\u00a0\u2013 one can set various options there, like the default mode (pstore\/kdump), the amount of reserved memory for kdump, the directory used to store the logs, record size of pstore, etc. Another configuration file that may be interesting is the sysctl one, at\u00a0\/usr\/lib\/sysctl.d\/20-panic-sysctls.conf<\/code>, which can be used to fine tune the panic actions for the kernel.\n\n\n\n\n
<\/div>\n\n\n\n\nSo, hopefully, this short introduction has been helpful for Arch Linux users to start experimenting with kdumpst. If you find any issue, have a suggestion for some improvement, or even submit a contribution, feel free to open an issue in the kdumpst gitlab repo<\/a>. Thanks for reading and see you soon in the next posts about pstore and kdump!\n","protected":false},"excerpt":{"rendered":"

It\u2019s been a long time since I last posted something here \u2013 yet there are interesting news to mention, a bit far from ARM64 (the topic of my previous posts). Let\u2019s talk today about kernel crashes, or even better, how can we collect information if a kernel panic happens on Arch Linux and on SteamOS, … Continue reading “Presenting kdumpst, or how to collect kernel crash logs on Arch Linux”<\/span><\/a><\/p>\n","protected":false},"author":67,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-91","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.igalia.com\/gpiccoli\/wp-json\/wp\/v2\/posts\/91","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.igalia.com\/gpiccoli\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.igalia.com\/gpiccoli\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.igalia.com\/gpiccoli\/wp-json\/wp\/v2\/users\/67"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.igalia.com\/gpiccoli\/wp-json\/wp\/v2\/comments?post=91"}],"version-history":[{"count":27,"href":"https:\/\/blogs.igalia.com\/gpiccoli\/wp-json\/wp\/v2\/posts\/91\/revisions"}],"predecessor-version":[{"id":120,"href":"https:\/\/blogs.igalia.com\/gpiccoli\/wp-json\/wp\/v2\/posts\/91\/revisions\/120"}],"wp:attachment":[{"href":"https:\/\/blogs.igalia.com\/gpiccoli\/wp-json\/wp\/v2\/media?parent=91"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.igalia.com\/gpiccoli\/wp-json\/wp\/v2\/categories?post=91"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.igalia.com\/gpiccoli\/wp-json\/wp\/v2\/tags?post=91"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}