1. This is the one where if you "less" a giant log file, your web server will get killed. Awesome.
2. This could be handled by returning NULL.
3. If your server only needs to run a single process, making that one process terminate would be much faster than rebooting the entire machine (commercial-grade servers can take ten minutes to POST).
2. No. Because most allocations by the kernel are not a result of the program calling malloc. (They can be the result of a pagefault when the program tries to write on some memory)
There are a couple of different things going on here:
malloc() generally uses mmap() which is a syscall. It also can use brk()/sbrk(). Some implementations use both.
There is also kmalloc() which is what's broken based on TFA. This is inside the kernel only and is not a syscall.
So you are right, malloc() is not a syscall, but it may or may not use a syscall when you call it. You can, of course, write better code yourself: statically allocate the memory you need, then use malloc() on that big memory chunk. This way you are guaranteed not to cause the system any grief. Either your process starts, or it does not. After it starts, it is in complete control of its own memory.
First of all, malloc implementations that rely on brk are simply broken. The whole idea of a process-wide data segment is broken and leads not only to awkwardness around thread safety safety, also the inability to decommit memory when it's no longer needed. Decent malloc implementations use mmap.
Second, the whole idea that a page isn't really "allocated" until you touch it is just stupid, lazy, and terrible; it doesn't exist on sane operating systems like NT. On decent systems, mmap (or its equivalent) sets aside committed memory and malloc carves out pieces from that block.
It's frustrating to see people who grew up with Linux not only understand how terrible and broken Linux's memory management is, but not realizing that it could be any other way.
NT doesn't have fork. Supporting that requires copy-on-write semantics. And you don't want to stop fork just because you don't have enough memory for a second complete copy of the current processes; you might be about to throw all that away with an exec, and even if not you're likely to keep most of the shared memory shared.
Well, first of NT is... different. Not sure if it's sane or superior or whatever.
Second, you may be right that this behavior causes problems, but it clearly has advantages as well. You advocate copy-on-write semantics elsewhere. Well this behavior is basically that.
Third, dlmalloc() which is pretty widely used, uses sbrk().
Forth, I sense a whole lot of frustration targeted at "these damn Linux kids". Sorry you feel that way. On the plus side, it's Friday, and this particular Linux kid would gladly buy you a beer to help cheer you up.
(Technically, I grew up with NetBSD, not Linux. Wrote a toy allocator for it at one point, and am trying to work on one that works for Linux as well.)
Thats not really true. Not all dynamic allocation schemes rely on mmap, some use the brk and sbrk system calls.
"In the original Unix system, brk and sbrk were the only ways in which applications could acquire additional data space; later versions allowed this to also be done using the mmap call." -https://en.wikipedia.org/wiki/Sbrk
Also mmap itself is also a system call, it usually just was called prior to the failing page allocation, not at the time of allocation failure.
2. This could be handled by returning NULL.
3. If your server only needs to run a single process, making that one process terminate would be much faster than rebooting the entire machine (commercial-grade servers can take ten minutes to POST).