Sounds like it's having trouble allocating memory for the stack and stack guard. Whatever limit it's hitting though, Linux seems to be able to handle it correctly, which is to kill the process instead of a kernel panic.