It's important to specify your test scenario here and the behavior you expect, as it's a known issue in general that Linux behaves poorly under OOM conditions. It's not necessarily Firefox specific.
If all of my memory is "in use" but half of it is cached file pages, should Firefox compact its heap and evict cached images? What if all of my memory is spoken for but a bunch of it isn't actually committed, due to overcommit?
Those aren't interesting questions tbh. Obviously most of the time the page cache grows to use all memory, that's the whole point of having it (using all memory that is not committed to applications as a disk cache). Overcommit obviously doesn't create memory pressure because it only exhausts address space, not memory (physical memory + swap). The relevant question is how the system behaves if it needs to allocate pages but can't find any clean (just throw'em out) or unused pages. Linux, and to a lesser degree, Windows start to thrash heavily in these scenarios. I've never seen e.g. Firefox unload a tab when this happens, regardless of OS.
If all of my memory is "in use" but half of it is cached file pages, should Firefox compact its heap and evict cached images? What if all of my memory is spoken for but a bunch of it isn't actually committed, due to overcommit?