Interesting stuff. I get a bit skittish around storage caches because of some scars from some pretty spectacular failures, but when they are working as designed they are quite awesome.
Multiple writers with some using ftrunc(2) that is the nightmare scenario for me. Would be interesting to do a walk through of considering each disk blowing up and the recovery from that.
Excellent! I was just investigating btrfs[0] and
realized that it doesn't offer anything like ZFS's
"2nd level storage" (L2ARC) which is doing
precisely this: keep what is nowadays referred to
as hot data on dedicated (read: solid state)
storage; as much as possible.
In the course of this little recherché I hit upon
bcache and dm-cache but found (Debian) user-land
to be entirely lacking (`modprobe dm-cache`
doesn't raise an eyebrow, though).
Having dm-cache support in LVM -- one of those
projects that figure centrally in my personal
sysadmin lore -- would be just heavenly.
[0] Balanced Tree FS. Similar in outlook and approach
to SUN-now-Oracle's
ZFS and -- oh, irony of ironies -- also under the Oracle umbrela.
Annoying thing about have a filesystem and a separate block layer cache (bcache or dm-cache) is they're duplicating a lot of the same work - filesystem maps from inode:offset to physical offset, cache does the same thing. Both doing their own allocation/reclamation/etc.
I've been turning bcache into a full blown filesystem in my spare time. It's working now - it's fast, and the code is _small_ compared to other filesystems.
Not production ready yet and it'll be awhile before I freeze the on disk format, though.
I have to admit that for a moment I thought: awesome, there is a built-in cache feature now for the llvm compilers and it took me almost a paragraph to recognize that the author was not talking about compilers....
It's a bit surprising that this routine post investigating a new LVM feature turned up on HN and several other high profile sites last night while I was out at a gig.
So I'm afraid I haven't yet had time to do proper testing. The benchmarking in the post is next to useless. I will be looking at how virtual machines behave on top of LVM cache (versus on top of an uncached hard disk), but that will take some time.
Multiple writers with some using ftrunc(2) that is the nightmare scenario for me. Would be interesting to do a walk through of considering each disk blowing up and the recovery from that.