Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You should be able to remove the deduplication data by zfs send/recv within the same pool too, I think


Zfs send/recv sends the blocks as written to the original filesystem (which is why it can be so fast, it doesn’t have to ‘understand’ what it happening or defragment things to read like reading a file does), but that also means undoing or applying dedup won’t work correctly unless it’s screwing with things you probably don’t want it too.

One issue I had is that due to what I eventually tracked down as power issues, I had some corrupted data written to disk under my zfs pool (at the media write later), and I had dedup on.

So dedup, unfortunately, actually made it REALLY suck to fix, because I couldn’t even copy a new version of the file to the same pool! It kept nuking the duplication, and keep the old bad data and I then couldn’t read the copy. :s

It even did this after I deleted everything, because prune couldn’t remove the bad underlying entries because it was having a media failure.

So delete files, scrub, put new files on resulted in them having the exact same failure.

When I nuked the pool and recreated it, it was all fine though.

So yeah, be careful with dedup.


Zfs send/recv actually does send data at a logical level, unless instructed otherwise. There are options to send deduplicated streams, streams maintaining compression, and raw streams but none of those are the default. Also, see my reply to a sibling comment.


Not according to people who know more about ZFS than me.

https://www.truenas.com/community/threads/zfs-dedup-disable-...


So at a pool level you might not be able to turn it off once it's turned on, but you can also turn off deduplication per file system, including in properties you set when receiving a stream. I wasn't confident this would work, but a test proved it can. (chicken_test/dedup_source had deduplication enabled and 16 copies of the same 100MiB file)

  chicken:~# zpool list
  NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    
  HEALTH  ALTROOT
  chicken_test    15G   144M  14.9G        -         -     0%     0%  16.00x    
  ONLINE  -
  chicken:~# zfs send chicken_test/dedup_source@send | zfs recv -o dedup=off 
  chicken_test/nodedup_dest
  chicken:~# zpool list
  NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    
  HEALTH  ALTROOT
  chicken_test    15G  2.29G  12.7G        -         -     0%    15%  16.00x    
  ONLINE  -
  chicken:~# zfs get dedup chicken_test/nodedup_dest
  NAME                       PROPERTY  VALUE          SOURCE
  chicken_test/nodedup_dest  dedup     off            local
  chicken:~# zfs destroy -r chicken_test/dedup_source
  chicken:~# zpool list
  NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    
  HEALTH  ALTROOT
  chicken_test    15G  2.29G  12.7G        -         -     0%    15%  1.00x    
  ONLINE  -




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: