Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I think rsync has a --checksum flag and also supports incremental backups, so probably based on your situation you can understand a logic that represents your access patterns. That is, if you know for sure you personally will not touch the files after 20:00 and no one else will, it's _fairly_ reasonable to assume that if you start rsync after you're done working, then schedule an incremental pass a bit before you start working, anything that was "copied" to your new backup has likely changed in an unintended way, and the list should be pretty small for you to check each morning, if there's anything. Keep in mind stuff like OS deduplication may mess with this (Windows' Dedup would undoubtedly break this schema as it may show that the files are modified after the Optimization job runs)

Alternatively, consider just using a file system that does periodic integrity checks. I know ReFS has integrity streams, and I am pretty sure XFS has something almost exactly the same or better. It won't prevent corruption, but it will give you something you can monitor for when the filesystem reports an issue.

Some combination like this should work.

Similarly, you might be able to come up with a fast trick using stat; with some quick testing on a dummy file in MacOS' ZFS shell, you can do something like:

stat -f %m somedir/*

and compare the resulting value by passing it to sum or something. I am not super familiar with stat in general, so likely I am missing elements that make this unreliable, but I'd consider looking into it further unless someone tells me it's 100% the wrong direction and explains why.



Note that rsync generally runs far, far more slowly with the --checksum flag. And I can't recall it ever saying something like "change(s) in $FileName were only noticed by checksum", so I would have been alerted to quiet disk corruption.

ZFS has checksums, and the 'zpool scrub' command tells it to verify those (on all copies of your data, if you're using RAID).




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: