To learn about the intricacies of how fragmentation can occur in ext3 and some better tools for measuring it, see the fragmentation section of Oliver Diedrich's article 'Tuning the Linux file system Ext3'.
Extents, multiblock allocation, and delayed allocation make ext4 more resistant to fragmentation than ext3, but the developers are still working on a defrag utility.
How does ext* move files around to reduce fragmentation if the disk is full? I'm glad you asked.
Take a look at your filesystem properties:
# tune2fs -l /dev/blah
In there you'll see a property called "Reserved Block Count". That is space reserved for kernel use to avoid fragmentation, continue logging, and for any other tasks the kernel/root user may need to do on a full filesystem. Users cannot write to this and your disk utilities will report your disk being full even if this reserved space is still available.
I believe it is typically 5% of the partition size. If your partition is hundreds of gigabytes then the reserved portion may be tens of gigabytes (multiply the reserved blocks by the block size to get total byte size).
You can change the percentage of this portion with the tune2fs -m flag. I have filled up a 500 GB backup drive with 0% reserved blocks (%5 reserved would have meant that 25GB was unusable) with no issues. Obviously, you probably don't want it set to 0% if you are constantly changing the contents of that partition.
"If you do have problems with fragmentation on Linux, you probably need a larger hard disk. If you actually need to defragment a file system, the simplest way is probably the most reliable: Copy all the files off the partition, erase the partition, then copy the files back onto the partition. The file system will intelligently allocate the files as you copy them back onto the disk."
So... buy a larger hard disk, or get a second disk to copy everything to instead of a defrag utility?
Well, in that case, Windows can't do any better. In order to defrag a filesystem, that requires free space to copy and move files around. If that space isn't there (because the HDD is nearly full), there isn't much you can do.
It's worth noting that while the Windows defrag utility is completely useless without a lot of free space available, many of the third party defrag tools can still work (albeit much more slowly) in that situation.
Of course with SSDs becoming more and more prevalent, this is becoming a non-issue anyway.
I'm a bit puzzled by the second method (with the partition swipe and copy back) .
So, linux scatters the files. My partition is almost full and fragments. Now I copy all the files to a second location and wipe the partition. Now I copy them all back. Won't that just scatter all the files and fragment all over again ?!?
In the first case, the file create & scatter may have occurred in stages, leaving less optimal disk utilization.
By copying files back en mass, fragmentation / allocation will be more optimized.
If you have insufficient disk overhead and continue the file destruction/creation process, you may end up with higher levels of fragmentation in future. This depends on your particular usage scenario.
If you're not underallocating disk, generally you won't have much of a problem.
Well there must be some mistake in what he wrote, you don't have to 'wipe' the partition, merely remove the files (and directories if any) which you copied over before copying them back. When you copy the files 'back' they will be copied sequentially and thus be placed 'scattered' (as in position on harddrive, not fragmented file) with appropriate space left between them for possible growth without needing to fragment them.
This copy method can of course be done on Windows aswell, the only difference is that NTFS's file placement strategy generally results in more fragmentation over time than the strategies employed by Linux filesystems like ext(x). That said all filesystems get fragmented over time, no strategy can avoid this fully.
Also if your partition is very fragmented and you have many large files it can be faster to use this 'copy to another partition' method rather than using a defragmenter which will operate solely on the target partition and thus need to move many parts of files around in the partition to 'fit' in parts of other files. Of course using the copy-method requires you having another partition with enough free space so it might not always be a feasible option.
1. It's not hard to implement such file allocation strategy. If it was superior, other OS'es would have adopted it long ago.
2. Scattering files all over the partition makes the longest seek times. This is actually the most inefficient solution for linux with its apps and sources (usually) consisting of thousands small files.
3. Windows does not need to be defragmented manually as it defragments itself when PC is not in use.
> 2. Scattering files all over the partition makes the longest seek times.
Yes but it isn't guaranteed that there will be less fragmentation on a windows file system. It depends on when was the last defrag, on the way file are created, and your disk occupation.
Also, when you scatter files on different disks it can be more efficient because you can read several cluster simultaneously. But I guess Windows does this as well.
Does this imply for certain contexts, the Windows model is superior?
Say you had a "write/update infrequently, read many times" system profile with 10% disk space avail. It seems like Windows would store these files closer to each other on the drive, making reads more efficient.
Or would that be such a negligible difference that it's not worth mentioning?
It's pretty negligible. I genuinely can't tell the difference between NTFS and ext3 at least performance-wise.
However, the locking semantics on NTFS are shitty unless you know what you are doing (i.e. use KTM/Transactional NTFS) which sometimes skews the statistics a lot.
Windows seems superior to me. It performs better due to seek time reduction, and it defrags and optimizes during idle periods. With Linux you have the worst of both worlds: it's slower and when the disk gets near full you supposedly have no choice but to re-do the disk.
This might be more accurately titled "Why ext2/3/4 doesn't need defragmenting." The advice given in the article:
Copy all the files off the partition, erase the partition, then copy the files back onto the partition. The file system will intelligently allocate the files as you copy them back onto the disk.
does not work for ntfs-3g. Whatever strategy it uses to allocate blocks, it's terrible for streaming reads of large files (last time I checked about 1-2 years ago). So, if you're trying to use the copy-off copy-on method of defragmenting an NTFS partition, don't do it from Linux.
In 30 years working with Unix, I've never encountered a situation where "defragmentation" was the answer. File system too full? Dozens of times. So, what usage pattern am I avoiding to live such a charmed life?
Anecdotally, I run a couple virtual machines on my laptop whose storage is on an external drive that sounds like Geiger counter at Fukushima Reactor 1 when the drive is busy. That is, when I've booted Windows 7. Not when I'm running anything under Windows, Windows just needs to be booted. When I'm running only a Linux VM, it drops to background level.
I can only speak to how BSD does it, but if you append 8k (any small amount) to a file that doesn't have adjacent space available, the filesystem will collect up to the last 64k and rewrite it all out in one block.
The trouble is I did use a filesystem all the way to completely full, and now it's severely fragmented (despite me resizing it larger) and I have no practical way of unfragmenting it.
From the article: "If you actually need to defragment a file system, the simplest way is probably the most reliable: Copy all the files off the partition, erase the partition, then copy the files back onto the partition. The file system will intelligently allocate the files as you copy them back onto the disk."
It's generally a sign that you're running your disks / filesystems overutilized. Retaining 10-20% free space is generally encouraged.
If you do elect to do the "copy out, wipe, copy in" method, you can do this, in general, while not completely losing the use of your system. There are some tricks (union mounts, remounting filesystems) which will allow you to flip between filesystems, in most cases. Specifics will depend on your partitioning scheme, use case, and the filesystems under consideration (usually you'd be dealing with user files under /home, spool or variable files under /var, source trees, etc.).
As the article states, Linux systems generally experience very minimal fragmentation, and dynamically resolve the issue for you, unlike Windows. You can typically run a Linux system for years, given a 10-20% disk reserve, without running into appreciable fragmentation issues. The gains from any sort of manual defragmentation would be minimal compared to the cost of performing the task.
That is true for any filesystem that needs defragging. The entire notion of a disk defragmentor is that it fixes the problem without needing to do this step.
It's like saying "Our car never breaks down so you don't need to be able to repair it. Even if it does break down you can still walk."
There are tools like Shake [1] that can semi-intelligently copy fragmented files around to defragment them, which typically results in less fragments if there is enough free space on the volume.
I had that problem at one point. I vaguely recall that I unmounted my home partition and fscked it at one point to find the fragmented inodes, then there was complicated magic to find the filenames that those inodes referred to, and then there was a script which copied them all to a tmpfs (i.e. into memory), deleted the original, and copied the copy back out.
One of the goals of recent ext4 work was to get support for "online defragmentation" but I don't know to what extent it's automatic, what tools exist, et cetera. It sounds like pyfragtools also has attempted to solve this problem.
When compared to having a (reasonably simple) tool which will do it for free, using only one volume, while still allowing r/w access? Yes, it's impractical.
Other filesystems like ZFS and btrfs utilize copy-on-write to virtually eliminate file fragmentation -- files are copied whenever edits are made to the file.
My understanding is that COW filesystems have really bad fragmentation if you're changing portions of large files -- for example, with virtual machine disk images.
Maybe ZFS/btrfs users don't care, but I would be really hesitant before using a filesystem which does not respect the Unix `shred` tool. Like real paper-shredders, I don't use them very much but when I do use them I really hope that they work.
Its not just zfs/btrfs, ext3/4 has the same cow behavior when data=journal (not default but to note)
And if you're on an ssd its likely doing wear leveling in a similar fashion.
Really the better way to do this than overwriting files in place is to have encrypted files and keys and just reencrypt with a random key you don't retain, or some other method I suppose. But the ability to rewrite to the same spot in the filesystem, for a copy on write filesystem, is rather ludicrous. Hesitate all you want, I prefer cow fs's for most tasks.
They just said copy-on-write. I'm not entirely sure about the exact interplay between shred and btrfs, but shred nominally just writes random data to the file that was already there -- if you copy-on-write that, then you are just copying random data to a different part of the filesystem, not overwriting what's already there.
You could solve this by automatically shredding deleted stuff in the background, but it would probably kill SSDs and other storage devices which you don't want to write to too often.
I don't know how shred works, but based on what you said, it assumes knowledge of how the underlying filesystem works. If there's fault to be had (and there probably isn't), it's not ZFS's.
You really consider being able to shred individual files a critical filesystem feature? Over, say, not having to configure static partition sizes, having snapshots, or being completely immune by design to large classes of data corruption (including all those that require an fsck-like tool to repair)?
ZFS is copy-on-write with snapshot support. If you have files you may need to shred, put them on a different file system; ZFS is oriented towards making it hard to lose data.
What I haven't seen mentioned is that historically, Unix (which inspired Linux) is a server/multi-user OS. All this thinking about "my" disk drives and how "my" large files are allocated is pretty irrelevant when you have dozens of users as well as system daemons accessing different files all of the time. Your drives are seeking all over the place constantly anyway. Fragmentation in such a scenario is much less of an issue.
this doesn't sound right... but maybe thats because I sort of assume any "sufficiently smart file system" has sort of COW (copy on write) like append semantics plus some sort of amortized compaction similar to a reasonable point in the GC design space of copying vs mark-sweep. (realistically most File systems don't do something like that, but its what I imagine "should be" the case)
xfs_fsr improves the organization of mounted filesystems. The reorganization algorithm operates on one file at a time, compacting or otherwise improving the layout of the file extents (contiguous blocks of file data).
xfs also has a tendency to fill up the kernel slab space when stat'ing a bunch of files/directories. This is really annoying because most "memory full" tools don't delve into the slab space to see if the usage is something that will free up automatically.
So Linux DOES need to be defragmented, but the only difference is that it defragments automatically, unlike Windows which requires an external defragmentation tool.
So really the title should be "Why Linux Doesn't Need Defragmentation Tools".
To see how fragmented a file is:
To see how fragmented your free space is http://linux.die.net/man/8/e2freefraghttp://linux.die.net/man/8/filefrag
To learn about the intricacies of how fragmentation can occur in ext3 and some better tools for measuring it, see the fragmentation section of Oliver Diedrich's article 'Tuning the Linux file system Ext3'.
http://www.h-online.com/open/features/Fragmentation-746701.h...
Extents, multiblock allocation, and delayed allocation make ext4 more resistant to fragmentation than ext3, but the developers are still working on a defrag utility.
http://kernelnewbies.org/Ext4