Consequences of passing too few register parameters to a C function

In our exploration of calling conventions for various processors on Windows, we learned that in many cases, some of the parameters are passed in registers. Suppose that there is a function that takes two parameters, but you know that the function ignores the second parameter if the first parameter is positive. What happens if you call the function with just one parameter (say, passing zero). The function should ignore the second parameter, so why does it matter that you didn’t pass one? Even though the function doesn’t use the parameter, it still may decide to use the storage for that parameter as a conveniently provided scratch space. For example: int blah(int a, int b) { if (a <= 0) { int c = f1(); f2(a); return c; } else { return f3(a, b); } Is it okay to call blah with zero as its only parameter? You aren’t passing b, but the function doesn’t use b, so why does it matter? Formally, the C and C++ languages say that if you call a function with the wrong number of parameters, the behavior is undefined, so officially, you’ve broken the rules and anything can happen. But let’s look at what types of things could go wrong. If you pass too few parameters on the stack, and it is a callee-clean calling convention, then the callee will clean too many bytes off the stack, resulting in stack imbalance and likely memory corruption. Even if it’s not a callee-clean calling convention, the called function will think that the memory for the parameter is present, and it may use it as scratch space, resulting in memory corruption in the stack frame of the calling function. In our example above, the compiler might realize, “Hey, I don’t need to allocate new memory for the variable c. I can just reuse the memory that holds the now-dead variable b.” In other words, it rewrites the function as int blah(int a, int b) { if (a <= 0) { b = f1(); f2(a); return c; } else { return f3(a, b); } Even if you don’t reserve memory for the variable b, the compiler will assume that you did and overwrite whatever is at the location the reserved memory should have been. But what if the parameters are passed in registers, and you didn’t pass enough of them? On most processors, what happens is that the called function will try to use that register and read whatever uninitialized value happens to be lying in that register. Except on Itanium. One special Itanium quirk is the presence of the “Not a Thing” (NaT) bit, which is a bit attached to each general purpose register that indicates whether the register holds a valid value. The most common ways for a register to enter the NaT state are if it was the result of a failed speculative load, or if it was the result of a mathematical calculation where at least one of the inputs was itself NaT. Therefore, if your uninitialized output register happens to be a NaT left over from an earlier failed speculation, the called function might decide to spill the value onto the stack for safekeeping before using that register for something else. extern bool is_valid(int); int blah2(int a, int b) { if (is_valid(a)) { return f3(a, &b); } else { return 0; } } The compiler realizes that it needs to take the address of b if a is not valid, so it has to spill the value to memory (so that it can have an address). But writing a NaT to memory raises a “NaT consumption” exception, so this function crashes even in the case where it never actually uses the b variable. But wait, there’s more. On Itanium, the function call mechanism is architectural…

This excerpt is published under fair use for community discussion. Read the full article at The Old New Thing.