When you are developing in the Linux kernel you easily find yourself dealing with a kernel oops, kernel bugs or something alike. Sometimes you can recover from it and continue working, sometimes not. If you are developing drivers and testing them over the same machine... you are doing something wrong.
It is recommended to have a virtual machine or another computer to test your code there, if not you might have productivity problems (you might need to reboot, so it is a waste of time waiting to boot, log in and open the editor to fix the bug) or, even worse, you can lose part of your work because the computer hanged and you forgot to save the file!
In case of having a second machine, you would probably access to it using SSH or telnet. Sometimes, the machine's location is far away from where you are, even in another continent. So you cannot go there, reboot the machine manually and come back in a reasonable time; or you need to bother people to do it for you.
You have a lot of ways to sort out this problem. You can use a remote power switch (for example, this one) but it is quite expensive. Another way is to use some hidden features in the Linux kernel, like the files present in /proc/sys/kernel that I will explain in another post.
However, the topic of this post is to learn how to use the SysRq keys. Now you will understand the presence of the SysRq key on your keyboard!
Definition
As Wikipedia says:
The magic SysRq key is a key combination understood by the Linux kernel, which allows the user to perform various low level commands regardless of the system's state. It is often used to recover from freezes, or to reboot a computer without corrupting the filesystem.
This special key allows you to recover from a crash on your system and send some commans, like shutdown, reboot, sync data with your filesystem, etc.
Enable SysRq support
The support for SysRq commands should be enabled before use it. In the common GNU/Linux distributions, this support is usually enabled by default in the kernels they provided. If you are compiling your own kernel, be sure that you have the following in your .config file:
CONFIG_MAGIC_SYSRQ=y
There is a sysctl to toggle this feature on and off. To enable it:
# echo 1 > /proc/sys/kernel/sysrq
How to use it
On x86, you press the key combo 'Alt' + 'PrintScrn/SysRq' + key. For other architectures you can read the corresponding Linux Documentation file.
Under graphical environments (such as Gnome or KDE) 'Alt'+'PrintScrn/SysRq'+key combination generally only leads to a screenshot being dumped. To avoid this, you should press 'Ctrl' before: 'Ctrl'+'Alt'+'SysRq'+key.
Extracted from the Wikipedia page, here you have the available options:
| Action | QWERTY | Dvorak | AZERTY |
|---|---|---|---|
| Set the console log level, which controls the types of kernel messages that are output to the console | 0 through 9 | 0 through 9 | 0 through 9 (without using shift) |
| Immediately reboot the system, without unmounting or syncing filesystems | b | x | b |
| Reboot kexec and output a crashdump | c | j | c |
| Display all currently held Locks | d | e | d |
| Send the SIGTERM signal to all processes except init (PID 1) | e | . | e |
| Call oom_kill, which kills a process to alleviate an OOM condition | f | u | f |
| When using Kernel Mode Setting, provides emergency support for switching back to the kernel's framebuffer console[4] If the in-kernel debugger 'kdb' is present, enter the debugger. | g | i | g |
| Output a terse help document to the console Any key which is not bound to a command should also perform this action |
h | d | h |
| Send the SIGKILL signal to all processes except init | i | c | i |
| Forcibly "Just thaw it" - filesystems frozen by the FIFREEZE ioctl. | j | h | ? |
| Kill all processes on the current virtual console (Can be used to kill X and svgalib programs, see below) This was originally designed to imitate a Secure Access Key |
k | t | k |
| Shows a stack backtrace for all active CPUs. | l | n | ? |
| Output current memory information to the console | m | m | , |
| Reset the nice level of all high-priority and real-time tasks | n | b | n |
| Shut off the system | o | r | o |
| Output the current registers and flags to the console | p | l | p |
| Display all active high-resolution timers and clock sources. | q | ' | a |
| Switch the keyboard from raw mode, the mode used by programs such as X11 and svgalib, to XLATE mode | r | p | r |
| Sync all mounted filesystems | s | o | s |
| Output a list of current tasks and their information to the console | t | y | t |
| Remount all mounted filesystems in read-only mode | u | g | u |
| Forcefully restores framebuffer console, except for ARM processors, where this key causes ETM buffer dump | v | k | v |
| Display list of blocked (D state) tasks | w | , | z |
| Used by xmon interface on PPC/PowerPC platforms. | x | q | ? |
| Show global CPU Registers (SPARC-64 specific) | y | f | ? |
| Dump the ftrace buffer | z | ; | ? |
How to use SysRq remotely
However, the key combos explained before are just for a local machine with access to a keyboard plugged to it. How do we use these low-level commands in a machine that is a thousand km far away?
Easily, we setup a daemon to connect to it and it will execute this key combos for us. At this case, I will explain sysrqd because is the one I am using to debug one remote machine in Igalia.
Remember that sysrqd opens the 4094 port by default to allow the access via Telnet. You should have to provide a password (later I will explain how) but... the password won't be encrypted, so be aware of the security-risks!
sysrqd setup
Firstly, you need to install it on the remote's GNU/Linux distro. As I am using Debian:
# apt-get install sysrqd
After that, you need to setup the password and change the file permissions:
# echo “mypassword” > /etc/sysrqd.secret # chmod 0600 /etc/sysrqd.secret
And then reboot or start the sysrqd daemon directly:
# /etc/init.d/sysrqd start
Access to sysrqd
Once you have enable sysrqd, you can access to your machine using telnet:
$ telnet <hostname> 4094
It will prompt waiting for your password, then write it (remember that it won't be sent encrypted and it will appear on your screen too!), and finally press your desire commands as they are described in the aforementioned table.
You need to take into account that leaving the Magic SysRq enabled on a production machine can be dangerous, anyone with physical access to the machine can bring it down. It is even worse with sysrqd daemon running, anyone sniffing the network could pick the clear-text password and later access via telnet. So use it for testing and under your own risk!