Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

While you are right that it is possible to implement extended file attributes as you describe, even when this is not supported by the operating system, this is extremely inconvenient.

The reason is that you must have complete control over all software run on such a computer and be able to build everything from source, to be sure that all file operations are done using some standard libraries modified by you.

For the file attributes to be reliably associated with the file you must ensure that any file copying, moving, renaming, linking or deleting takes into account the file attributes. If you allow any such operations to be done by a non-modified legacy program, you break the file system. (The fact that even some implementations of extended file attributes in the operating system allow user programs to lose the extended file attributes when they are not aware of them is not a feature, but a serious bug.)

Instead of modifying all file-handling libraries that may be invoked in user programs (e.g. for all programming languages that might be used) it is actually much easier to do the modifications in a single place, i.e. in the file system implementation in the operating system. This can also provide better performance for accessing the file attributes.

Workarounds like those of Windows 95 have appeared after HPFS and its successors, with the purpose of adding extended file attributes to an existing file system, without changing its on-disk data representation.

Like I have already said, the immediate direct successors of HPFS have been both NTFS and Silicon Graphics XFS (which combined features of HPFS with features of the SGI Extent FS), and these two have been developed and launched almost simultaneously.

While there have been many file systems that have been developed later than 1989 and which have ignored HPFS at their conception, all of those that have survived have added sooner or later extended file attributes and B-tree directories. This means that all later file systems have either been influenced by HPFS or they have disappeared.

BSD LFS has started before HPFS (research report in 1988), FAT32 and VFAT were compatible extensions of the older FAT, ext2fs was intended as just a compatible implementation of the old UNIX file systems, with no innovations. ZFS has included since the beginning any features of HPFS, because it was intended to be better than the XFS used by competition (but ZFS has never succeeded to approach XFS in performance, even if it may be better in the provided features and in reliability).



This is a very interesting conversation, thank you!

A couple of minor quibbles: FAT32 is not a compatible extension of the older FAT; ext2fs is not a compatible implementation of any old Unix filesystem; having less features does not always make software worse; the BSD LFS research report is from 01993 rather than 01988 https://www.usenix.org/legacy/publications/library/proceedin... (it credits Ousterhout with introducing the idea of LFSs in 01989); several of the other filesystems I listed that I don't think are significantly influenced by HPFS have not in fact disappeared; and Windows 95 does not, to my knowledge, support extended attributes on existing filesystems.

It's probably worth mentioning that, as you imply, ext2fs did eventually add both extended attributes and B-tree directories, just after being quasi-renamed ext4fs. And I do agree that ext2fs was initially intended to be as boring as possible (without suffering the annoyances of the Minix filesystem) rather than innovative. It just failed to be compatible with BSD FFS.

It's a good point that renaming and hardlinking doesn't automatically take into account such extended attributes implemented as a library; that's a plausible reason to implement them in the filesystem. But implementing them in the filesystem doesn't actually help you with respect to copying, unless copying files is an operation the filesystem supports intrinsically, like "reflinks" on some Linux filesystems such as XFS. You still have to modify your file-copying program to copy the extended attributes when that is desired, whether they're provided by the kernel or by a library. Even if the filesystem supports a copy operation, there are typically cases it doesn't handle, such as copying files onto removable media or network filesystems.

Moreover, whether the attributes are implemented in the kernel or in a library, you also have to modify other utility programs to handle the extended attributes if they are to be preserved; for example, backup programs (including things like rsync, zip, and tar), filesystem integrity checking programs like Tripwire, and network file server programs.

File operations that aren't affected by extended attributes wouldn't need to use the extended-attribute library. Depending on what you wanted to use them for, that could be most of them. And it isn't necessary to reimplement the library in multiple different languages; it's sufficient to implement it in C, Rust, or assembly. (All the other languages would need to have a way to call the C library, but that's also true of extended-attribute system calls.)

If you want to use xattrs for ACLs, you can't implement them in a library. But that seems like a backwards way to look at the situation. You don't need extended attributes in the kernel to implement ACLs; you can extend your filesystem to store ACLs in an ACL-specific place. But, if you do have extended attributes in the kernel, that's a candidate way to store ACLs.

Implementing features like xattrs as libraries (rather than in the kernel) has advantages as well as disadvantages. For example, depending on how you do it, you don't need to modify your backup programs. You can support them on all filesystems instead of just some filesystems. And getting or setting them doesn't necessarily incur system-call overhead, though that probably depends on caching.

I don't think VFAT or exFAT does in fact support either extended file attributes or B-tree directories. According to https://eclecticlight.co/2018/01/12/which-file-systems-and-c..., MacOS fakes xattr support on them using the approach I outlined upthread, but instead of suffixing the filename with .ea, it prefixes it with ._. And I think it does it in the kernel rather than in a library, but I don't really know.

BeOS's ability to find files by extended attributes seems like a reasonable justification for doing them in the kernel instead of in a library. Otherwise, though, it seems like an obvious but debatable design choice like case sensitivity, not an innovation.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: