In Part 2 I explained how ARM32 has two ISAs available (ARM and Thumb), how their feature sets differ in important ways and how software can and often does make use of both even in the same program.
Today we’re going one level higher and look at the ABIs (Application Binary Interface). Whereas an ISA defines what bit sequences make the CPU do what operations, an ABI is a set of conventions software is meant to follow so that it is interoperable with other software targetting the same ABI.
An ABI is adopted by compilers and operating systems. The designers of the instruction set often define some base ABI that compilers only need to extend to define a few missing bits.
One of the biggest underlying reasons getting perf to work in ARM32 is hard is that ARM has deprecated and gone over multiple base ABIs in its lifetime. Furthermore: the details on how frame pointers should work used to be completely non-standard and that has created fragmentation.
As of writing, these are the standard base ABIs defined for ARM:
- ~1987: APCS (Arm Procedure Call Standard (obsolete)). Only applies to ARM.
- ~1997: TPCS (Thumb Procedure Call Standard (obsolete)). Thumb-only counterpart of APCS.
- 1998: ATPCS (ARM-THUMB Procedure Call Standard). Unified ABI for both ARM and Thumb.
- 2003: AAPCS (Procedure Call Standard for the Arm® Architecture). The current base ABI. It applies to both ARM and Thumb.
In this post we’ll learn some typical examples of features an ABI would standardize, ultimately landing in frame pointers as one of such features.
The stack
At its most basic, a CPU does not need to have a concept of a stack. The ARM ISA exemplifies this well: the instructions used for stack manipulation are extremely generic—they can work on any register, support any kind of stack (full descending, full ascending, empty descending, empty ascending) and also work as general “write multiple registers to memory pointed by register” instructions.
But of course, a stack is a very useful abstraction that any non-trivial program will need as it allows to free up registers by saving their data for later use. Different pieces of code (e.g. functions) need to agree on a definition for the stack and rules to use it so they don’t accidentally overwrite each other’s data.
As such, ABIs define the stack; namely: what register points to the current top of the stack, whether it grows towards greater adresses (ascending stack) or toward lower adresses (descending stack), whether the “top of the stack” should be understood as “the last element pushed” (full stack) or “the position a newly pushed element would use” (empty stack), any alignment requirements, and rules for writing to the stack—for example: writes outside of the stack may be considered undefined behavior even in the absence of function calls so that interrupt handlers can make use of the same stack.
On ARM32, virtually all the ABIs1 agree on using a full-descending stack, whose top position is tracked by the r13 register, which for this reason is also give the alias sp (stack pointer). The stack is 4 byte (32 bit) aligned. The AAPCS ABI further requires 8 byte alignment at the public interface (e.g. when calling a library function). Storing data outside of the valid region of the stack (i.e. in addresses lower than sp) is undefined behavior.
Function calls
An ABI defines what a function call is under the hood; namely: how arguments are passed and how are values returned. This part of ABIs is important enough to have its own name: calling convention.
Function arguments
A simple calling convention could require all function arguments be to pushed into the stack. The callee could then access them by addressing the stack pointer. Notably, the cdecl and stdcall calling conventions used in MS Windows 32-bit x86 work in this manner.
Modern calling conventions—including ARM32 AAPCS and all x86-64 calling conventions—use registers in addition to the stack for passing arguments, trading complexity in the ABI for better performance. This is especially a win for functions with few arguments. In many cases, programmers and compilers can strategically do computations directly in the same registers that will be later used as arguments, further reducing the overhead of function calls.
In AAPCS, r0-r3 are filled with the first few arguments. Arguments with types smaller than 32 bit (e.g. char, int16_t) are extended to 32 bits before the call. When an argument has a 64-bit type, it takes two consecutive registers. For more details, see section 6.5, Parameter Passing in the AAPCS specification. Remaining arguments are pushed into the stack.
Callee-saved and caller-saved registers
ABIs often separate user registers in two groups:
- Callee-saved registers
- Caller-saved registers, also known as scratch registers
A function is allowed to write to callee-saved registers at any point, but it must ensure they have their original value on return. This is normally accomplished by pushing (saving) the previous value to the stack before writing and popping it before return, hence the name «callee-saved».
On the other hand, a function is allowed to freely write (scratch) any and all caller-saved registers without preserving their old value anywhere. This also means that, before doing any function call, you must be careful to save any important data in these registers, hence the name «caller-saved».
Having a healthy mix of callee-saved and caller-saved is good for performance, as it reduces the number of stack manipulation operations necessary in typical functions.
In AAPCS, r0-r3 and r12 are caller-saved (i.e. scratch registers). All other general purpose registers are callee-saved.
Returning from a function: return pointers
The calling convention also needs to specify how returning from a function works. The caller must store a return pointer (an address to executable code immediately following the function call) in a well-defined place.
In many calling conventions, including the common ones for x86 and x86-64, the return pointer is passed in the stack. ARM calling conventions are a bit smarter: instead they place the return pointer in a so-called link register (lr). This means that simple leaf functions (functions that do not call other functions) can operate on their arguments and return a value without ever touching the stack.
All ARM calling conventions use r14 as the link register.
Of course, many functions will need to call to other functions before returning, so how does a link register handle this? Quite simply: lr is a callee-saved register! A function that will perform function calls will typically store lr in the stack somewhere in its preamble, do as many function calls as it wants, then restore the old value of lr from the stack.
The l in the bl and blx branch instructions in ARM stands for “link”. It means that the current value of pc will be loaded in lr before the branch. This is the most common way to do function calls in ARM and Thumb.
Special purpose registers
An ABI may reserve certain CPU registers for specific purposes, even if those registers would otherwise be general purpose registers according to the ISA. Compilers do to take this into account when generating code.
AAPCS leaves r9 free to use for any ABI extending AAPCS (e.g. for a specific operating system).
AAPCS defines r12 as the Intra-Procedure-call scratch register (ip). Functions are allowed to use it as a regular scratch register, but its real purpose is to enable writing simple veneers without needing stack manipulation.
Side-tangent: Veneers…? What is a veneer?
(You can safely skip this section If you just want to understand the minimum concepts necessary for frame pointers. Keep reading if you have a more general interest in ARM assembly.)
Instructions in ARM have a fixed size of 32 bits which is as wide as pointer. A consequence of this is that calling functions in arbitrary locations of the program requires a variable number of instructions. Most branches in ARM are done with signed offsets with a range of ±32 MiB (26 bits, of which only the most significant 24 bits are actually encoded in the instruction because ARM instructions are 4-byte-aligned). Thumb is even scrappier: the typical Thumb bl (branch with link) instruction only supports offsets of ±4MiB encoded in 22 bits, which is already more than the 16 bits per instruction of Thumb-1 and it is only possible because the bl ASM instruction gets turned into two machine Thumb-1 instructions designed for this purpose, each containing half of the offset).
Compilers and linkers cope with this limitation by assuming calls will be near enough for the offset to fit. When the offset does not fit, they synthesize some code (the veneer, also sometimes called a trampoline or range extension thunk) in a location close enough for the offset to fit. The veneer will then load the entire address in some register—likely ip— and then use a «branch to address in register» instruction (e.g. bx ip).
From this example the utility of having a scratch register not used for argument passing (ip) becomes apparent: if a scratch register like ip did not exist, the veneer would need to spill some callee-saved register to the stack just to build an address for the bx instruction. Furthermore, the callee couldn’t return directly into the original caller, but it would have to return back into the veneer so that the values from the stack could be restored.
Also of note is that is the fact that the blx instruction (branch with link and exchange ISA, provided an offset or register) did not exist until ARMv5. Before, there was only bx (branch and exchange ISA), which operated on a register. This resulted in a different number of instructions required for the branch depending on the ISA of the target (which may not be known until run-time in the case of shared libraries), and by extension made veneers necessary for interworking between ARM and Thumb.
Frame pointers
A stack frame is the chunk of the stack used by a specific function call, not including nested calls.
A frame pointer points to a fixed location (e.g. the start or end) of the stack frame. When we say that a program uses frame pointers, we refer to that program dedicating a register—the frame pointer register (usually shortened as fp)—to store the frame pointer at all times.
Having a frame pointer allows assembly code to refer to locations in the stack with simple offset from the frame pointer. So, for example, a certain variable saved in the stack will be at fp-4 and will remain at fp-4 even if code later pushes to the stack. This is more useful for humans than compilers, since it’s possible to access the same data from sp as long as you keep track of the required offset changing every time something is pushed or popped.
Something much more useful occurs if code always pushes the old value of the frame pointer register and the return pointer to the stack at call boundaries: it becomes possible to inspect the stack at runtime: the frame pointer register points to the current frame record, from which the previous frame pointer and return pointer are accessible in a fixed offset. This process can be repeated, traversing the stack like a linked list, one stack frame at a time. Before main(), a sentinel value (typically a null pointer) should be pushed instead of a frame pointer so that the end of the linked list can be detected.
Note that the usage of frame pointers is an ABI matter: The above breaks if you call a function in a library that doesn’t use the same exact convention for frame pointers. If that function skips updating the frame pointer register, its stack frames will become invisible. Or worse: if it writes some random value to it, the entire chain of callers will no longer be recoverable.
In x86 ABIs, the Base Pointer register (bp in 16-bit, ebp in 32-bit, rbp in 64-bits) is commonly used to store the frame pointer.
Frame pointers were important for obtaining backtraces in early debuggers, but this is less necessary in modern toolchains. Nowadays compilers can generate debug information (debuginfo) in a format like DWARF that debuggers can use to map any value of the pc register to what function it is part of and how deep in the stack the return pointer is at that particular instruction. This is the same process they use to map local variables to positions in the stack. The process can be repeated until the entire stack trace is obtained.
Since frame pointers became less necessary for debuggers, compiler optimizations that skip updating the frame pointer register or even use it as a general-purpose register have become very common. Omitting frame pointers as an optimization has been the default in gcc for all platforms since 2017, after it had been enabled in most target platforms individually. ARM32 already had this enabled in 2010 and earlier.
While debuginfo is a good alternative to frame pointers to get backtraces in a debugger, other development tools are not so lucky. Recovering call chains from frame pointers is a very lightweight affair: it’s a singly linked list traversal where all the nodes are within consecutive memory. On the other hand, getting a call chain through debuginfo requires having separate debuginfo sections for each object file (e.g. .so library) mapped in memory, search the wanted pc value in their tables and decode the debug tags info to find the address of the return pointer; then repeat this for every other stack frame.
Some tools have tried to adapt to this new normal of not having frame-pointers by adding support for DWARF unwinding. perf is one of them. This is however still unsatisfactory on low-end hardware: perf record -g --call-graph=dwarf cog about:blank will bring a Raspberry Pi 3 to its knees, freezing the entire system until it dies on the hands of the OOM killer. High-end hardware, like desktop PCs, can usually handle using DWARF, but it still results in much higher overhead.
Omitting frame pointers as an optimization has been increasingly reconsidered in recent years, with Fedora 38+, Ubuntu 24.04+ and Arch Linux modifying their build recipes to attempt to build all packages with frame -fno-omit-frame-pointer. Modern CPUs (especially 64-bit CPUs) have enough registers that dedicating one for this purpose is not the hit to performance it once was. Some people would argue that even if there is a slight performance hit, the gains from users and developers being able to use profilers greatly outweight it.
With this modern understanding of what frame pointers are, we will also try to build all packages with frame pointers enabled in an ARM32 environment, so that we can also use a profiler there.
Problem: what are the ARM32 frame pointer ABIs, really?
APCS frames
The (old and deprecated) APCS spec specified an optional frame pointer ABI. When you pass the (now deprecated) -mapcs-frame flag to gcc, you tell it to use the frame pointer structures defined in APCS. Here is a descriptive diagram from page 629 of the Acorn Archimedes‘ Programmer Reference Manual (1987), whose Appendix C is “ARM Procedure Call Standard”:
Here is a simple function that we will use to compare the frame pointer ABIs:
extern void show_sum(int a);
extern void show_diff(int b);
int do_operations(int a, int b) {
int sum = a + b;
show_sum(sum);
int diff = a - b;
show_diff(diff);
int xor = a ^ b;
return xor;
}This is the assembly generated by gcc:
$ arm-linux-gcc -mcpu=cortex-a53 -marm -mapcs-frame -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c add.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
0: e1a0c00d mov ip, sp
4: e92dd830 push {r4, r5, fp, ip, lr, pc}
8: e1a04001 mov r4, r1
c: e1a05000 mov r5, r0
10: e24cb004 sub fp, ip, #4
int sum = a + b;
show_sum(sum);
14: e0800001 add r0, r0, r1
18: ebfffffe bl 0 <show_sum>
int diff = a - b;
show_diff(diff);
1c: e0450004 sub r0, r5, r4
20: ebfffffe bl 0 <show_diff>
int xor = a ^ b;
return xor;
}
24: e0250004 eor r0, r5, r4
28: e89da830 ldm sp, {r4, r5, fp, sp, pc}Here ip is used as a temporary to hold the value of sp before the push. The resulting stack matches the one in the old APCS document as we could expect:
fp points here: | pc at time of push (i.e. savemask pointer) |
| saved lr |
| ip (i.e. the saved sp before the push) |
| saved fp |We can obtain the call chain by looking at the fp register and traversing the linked list of frame records: the saved lr contains the return function for that stack frame, and by extension, who called that function. We continue traversing the stack upwards until we find a NULL pointer indicating the end of the linked list.
AAPCS frames, clang
It may seem weird that I just spend that much time above showing a very old ABI (APCS). Especially because if you look at the modern AAPCS spec today you’ll see frame pointers also defined.
However, if you scroll to the change history, you’ll also notice it was only added in January 2020, whereas AAPCS had been around for 17 years before. This is the real source of problems: as far as I can tell, during these 17 years compilers had no official guidance on any frame pointer ABI for AAPCS.
Making things worse, GCC and clang ended up with similar but mutually incompatible frame pointer ABIs. The official AAPCS frame pointer ABI sided with clang’s. It looks like this:
$ clang -target arm-linux-gnueabihf -g -gdwarf-4 -mcpu=cortex-a53 -c -O2 -fno-omit-frame-pointer -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
0: e92d4830 push {r4, r5, fp, lr}
4: e28db008 add fp, sp, #8
8: e1a05000 mov r5, r0
int sum = a + b;
c: e0810000 add r0, r1, r0
10: e1a04001 mov r4, r1
show_sum(sum);
14: ebfffffe bl 0 <show_sum>
int diff = a - b;
18: e0450004 sub r0, r5, r4
show_diff(diff);
1c: ebfffffe bl 0 <show_diff>
int xor = a ^ b;
20: e0240005 eor r0, r4, r5
return xor;
24: e8bd8830 pop {r4, r5, fp, pc}Only two words are pushed in the stack: fp and lr. The fp register is updated to point to the the most recently saved fp in the stack.
GCC frames
If you pass -marm -fno-omit-frame-pointer to gcc (as of 15.2), this is what the resulting frame pointer ABI looks like:
$ arm-linux-gcc -mcpu=cortex-a53 -marm -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
0: e92d4830 push {r4, r5, fp, lr}
4: e1a04001 mov r4, r1
8: e1a05000 mov r5, r0
c: e28db00c add fp, sp, #12
int sum = a + b;
show_sum(sum);
10: e0800001 add r0, r0, r1
14: ebfffffe bl 0 <show_sum>
int diff = a - b;
show_diff(diff);
18: e0450004 sub r0, r5, r4
1c: ebfffffe bl 0 <show_diff>
int xor = a ^ b;
return xor;
}
20: e0250004 eor r0, r5, r4
24: e8bd8830 pop {r4, r5, fp, pc}This is very similar to what ended up becoming the official AAPCS frame pointer ABI and to what clang does except that fp points to the saved lr, rather than to the saved fp directly. This is what it ends up looking like:
-fno-omit-frame-pointer. The fp register points to the saved lr in the stack.What about Thumb?
While I’ve learned a lot about Thumb while working on this, I would be very cautious trying to make use of frame pointers in Thumb. They’re even messier and currently broken in GCC. For more details, you can keep reading this section.
r7 as frame pointer register
Thumb in stack traces complicates things significantly. Note that r11 is a “high register” in Thumb, which limits in how many different instructions it can be used efficiently. This is especially a problem in the original Thumb-1, which doesn’t have an instruction to push/pop low registers directly and you would typically need 3 additional instructions in the prologue and 3 additional instructions in the epilogue.
The above is much less of a concern with the now ubiquitous Thumb-2, which has 32-bit instructions for push/pop and arithmetic on high registers. However, as a consequence of this historical problem, you will see that in a default Linux setup both gcc and clang use r7 as frame pointer register in Thumb code.
$ clang -target arm-linux-gnueabihf -mcpu=cortex-a53 -mthumb -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
0: b5b0 push {r4, r5, r7, lr}
2: af02 add r7, sp, #8
4: 4605 mov r5, r0
int sum = a + b;
6: 4408 add r0, r1
8: 460c mov r4, r1
show_sum(sum);
a: f7ff fffe bl 0 <show_sum>
int diff = a - b;
e: 1b28 subs r0, r5, r4
show_diff(diff);
10: f7ff fffe bl 0 <show_diff>
int xor = a ^ b;
14: ea84 0005 eor.w r0, r4, r5
return xor;
18: bdb0 pop {r4, r5, r7, pc}The layout clang uses is the same as the official AAPCS ABI, but using r7 instead of the r11. Assuming there is no mixing of ARM and Thumb — the call chain can be recovered with the same traversal algorithm.
On GCC, however, Thumb frame pointers are unviable for call chain recovery, as the GCC sets r7 to the most recent position in the stack, which makes it impossible to know the position of the saved lr and the previous node pointer within the stack without additional information.
Using r11 for both ARM and Thumb
Using different frame pointer registers for different modes is problematic, as code in one mode (typically ARM) will frequently clobber the register used by the other mode (typically Thumb). Both r7 and r11 are caller-saved registers, so the information is there in the stack, but the stack and register values is not enough to locate them.
A simpler approach is to use r11 for both modes, which is much less awkward in Thumb-2. This is what the official AAPCS spec specifies. Recent versions of clang can be told to do this with -mframe-chain=aapcs (or alternatively, -mframe-chain=aapcs+leaf if frame pointers on leaf functions are also desired).
$ clang -target arm-linux-gnueabihf -mcpu=cortex-a53 -mthumb -gdwarf-4 -fno-omit-frame-pointer -mframe-chain=aapcs -O2 -g -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
0: e92d 4830 stmdb sp!, {r4, r5, fp, lr}
4: f10d 0b08 add.w fp, sp, #8
8: 4605 mov r5, r0
int sum = a + b;
a: 4408 add r0, r1
c: 460c mov r4, r1
show_sum(sum);
e: f7ff fffe bl 0 <show_sum>
int diff = a - b;
12: 1b28 subs r0, r5, r4
show_diff(diff);
14: f7ff fffe bl 0 <show_diff>
int xor = a ^ b;
18: ea84 0005 eor.w r0, r4, r5
return xor;
1c: e8bd 8830 ldmia.w sp!, {r4, r5, fp, pc}Frame pointer sadness
There are problems with frame pointers in ARM other than the incompatible ABIs. In fact, I kept finding more and more sadness the more I worked on this post.
GCC frames can’t be unwinded in leaf functions
While -fno-omit-frame-pointer will make GCC push fp to the stack, it doesn’t disable the “link register save elimination” optimization. This messes up with leaf functions. Consider the following trivial function:
void simple_nop(void) {
__asm__("nop");
}$ arm-linux-gcc -mcpu=cortex-a53 -marm -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c simple_nop.c -S && cat simple_nop.s
simple_nop:
@ args = 0, pretend = 0, frame = 0
@ frame_needed = 1, uses_anonymous_args = 0
@ link register save eliminated.
str fp, [sp, #-4]!
add fp, sp, #0
.syntax divided
@ 31 "example.c" 1
nop
@ 0 "" 2
.arm
.syntax unified
add sp, fp, #0
@ sp needed
ldr fp, [sp], #4
bx lr
.size simple_nop, .-simple_nopNotice that, unlike in the previously shown GCC frames, here the fp register is made to point to the saved fp in the stack, rather than the saved lr. This means that an unwinder will (1) wrongly read a next node pointer instead of the function return address that it would expect, (2) will interpret whatever random variable was pushed in the stack immediately before as the next node in the call chain.
I’m surprised I only caught this long after having written an unwinder for GCC frames and having seen it produce sensible flamecharts, but I’m pretty sure this is not just a synthetic problem, since I can see the same problematic assembly by objdump’ing GStreamer libraries.
The deprecated APCS frames (-mapcs-frame) are not affected by this bug.
Getting the top frame is racy during the prologue
At the top of a stack trace you expect to see the currently running function, at the currently running instruction. This is normally obtained by reading the value of the pc register. Then, by looking at the fp register and inspecting the topmost frame record we can see who called this function and traverse the chain.
However, this also means there is a small window of time from the moment a call is done with bl and until the fp register is updated to point to the newly created frame record in which the pc register points to the callee and the caller is recorded in the lr register, but hasn’t been put in a frame record yet.
This results on the caller function seemingly disappearing from the stack trace for a brief moment, as seen here in a instruction-per-instruction simulation:
* 1000(main+0)
* 1004(main+4)
* 1008(main+8)
* 1012(main+12)
* 2000(leaf1+0)
* 2004(leaf1+4)
* 1012(main+12) <- 2008(leaf1+8)
* 1012(main+12) <- 2012(leaf1+12)
* 1012(main+12) <- 2016(leaf1+16)I can’t think of an alternative call chain traversal algorithm that solves this problem using only the stack and registers. While it seems tempting to also make use of the lr register directly to reconstruct the missing frame, that causes other problems later when the callee returns. I’m also not the first person to point out this problem.
Conclusions
Frame pointers in ARM32 are messy — especially compared to ARM64 or x86_64. A standard ABI for them has only been specified few years ago, with support for it is pretty spotty and divided into incompatible alternatives.
The status of frame pointers for ARM32 in GCC is much worse than I thought when I started working on this: my impression now is that the ostensibly deprecated -mapcs-frame is the only frame pointer ABI GCC handles in a reliable way, and I wouldn’t be too surprised if it turned out that any usefulness of the non-APCS frame pointers in ARM32 with GCC is coincidental rather than intentional. This is certainly not helped by the long lasting specification void in this area.
The default frame pointer ABI of GCC is only reliable during non-leaf functions and as long as Thumb is not used. With that big caveat, they’re still potentially useful. Furthermore, the GCC-specific deficiencies I’ve explained here could be fixed, especially now that there is some official ABI guidance.
Clang has positively surprised me during my investigation. I started looking at it only for comparison, but was happy to see support for the new AAPCS frame pointer ABI in Thumb as well, as well as a general good handling of edge cases.
On top of that, seeing the limitations of frame pointers has also made me a bit curious about the alternative approaches for improved accuracy. Most of those require lookup tables and therefore are generally going to be slower, but how much they can be optimized is still an area of ongoing research.
Additional resources
Analyzing code that uses the stack by hand is error prone, so eventually I wrote a simulator for testing frame pointer ABIs and the algorithms to traverse them.
If you want to look at the compiler sources, you’ll be mostly intersted in:
- clang:
llvm/lib/Target/ARM/ARMFrameLowering.cpp. Contains the code for generating prologues and epilogues. SearchFramePtrto find the relevant code. - gcc:
gcc/config/arm/arm.c. Look forHARD_FRAME_POINTER_REGNUM. The prologue contents are emitted inarm_expand_prologue().
- The earliest variant of APCS—referred to as APCS-A—mapped sp to r12 instead of r13. fp was also mapped to r10 instead of r11. This is explained in page 1762 of the RISC OS Programmer Reference Manual (1989), which describes what would later be called APCS-2.
I’m also leaving aside many complexities of APCS variants, most notably chunked stacks, where the program stack wouldn’t be in contiguous memory, but instead in a linked list of chunks, useful for multi-threading in processors without a Memory Management Unit. ↩︎
