Oh but there actually are ways to signal failure (flushing caches, etc.), and there probably could and should be even more, but still the OOM killer is "needed" as a last resort measure.
I recently saw an interesting Windows VM feature, where a process can indicate that some pages can be sacrificed to the OS if needed, and another function to try to get them back unmodified if by chance they have not been. I don't know if similar Linux syscalls exists, but I found the principle interesting.
You can get a similar effect on recent Linux versions with madvise() with MADV_FREE, although the interface for determining if the page was freed is a bit awkward: the page will be appear as zeroed if the OS decided to free it. Depending on what you are storing there and whether zero data is valid it may be awkward to detect.
You can cancel the free request by performing a write, which is also a bit of an awkward API.
I recently saw an interesting Windows VM feature, where a process can indicate that some pages can be sacrificed to the OS if needed, and another function to try to get them back unmodified if by chance they have not been. I don't know if similar Linux syscalls exists, but I found the principle interesting.