These are arguably one of the smaller annoyances about apple when not using one myself (so, always) and they have become less relevant over the last years as sharing things via USB has become less common and I don't use dropbox or such with shared folders anymore. People who use git don't tend to check these in, when working with me at least, I make sure of that.
But it's a pretty darn good symbol for how Apple doesn't give a flying fuck about anyone's time/nerves, just dumping these on everyone without asking and letting us deal with their laziness.
Plus, Apple users then typically just rolled their eyes when I point out to them that I've now got all these files on my USB drive, thanks for nothing Apple.
"Why do you even care?"
Yeah, why do I? There's just no reason I should have to sort through these to keep them out of my systems, yet I don't want them there so no I have to, thanks apple... (this was before I got good on the terminal so it was actually quite annoying)
Bottom line: this still gets my blood boiling. Fuck apple, they're the new Microsoft in every way.
It’s just the same as desktop.ini files, it’s where they record the position of icons in storage locations that don’t allow the filesystem to store that information invisibly.
I’m still not sure why you care. You explained that you don’t want them on your ‘systems’ but you haven’t explained how they get there in the first place - you sound like you’re not an Apple user but these files are making it onto your USB drives and this is bad for you for some reason.
Wow, you sound exactly like the apple users I was referring to, except I can't tell whether you actually are one. :)
The point is that yes, "my usb drives" get polluted the second I give them to an apple user even if they just read files off them.
Same happens if an apple user accesses a writable network share on my NAS or my PC.
I care, because I constantly have to be on the look-out then to not accidentally add them to tar files, backups etc.
When I navigate my own file collections I just don't want these things lying around my folders everywhere, distracting me for zero benefit.
That's all work and mind capacity that apple happily, without asking, offloads onto me, because they won't bother _their_ users with it and because wtf am I supposed to do, refuse to interact with all apple systems?
It looks like they store the extended attributes in the __MACOSX tree too. And everything you download from the web has extended attributes describing the original URL and page that linked to it. (With chrome/safari, I haven't checked firefox.)
It looks like the finder "compress" functionality will include __MACOSX, and the command line zip doesn't.
If you run `xattr -l` on something in your Downloads, you should see the kMDItemWhereFroms metadata. mdls shows it too, but also includes other data that is extracted from the file itself.
> Zip files can encode their file names in two ways: CP437, or unicode.
> Each operating system does it wrong, but in a different way. For instance, Mac OS encodes its zip files as unicode, but doesn't set bit 11 correctly, so Python (correctly) reads them as CP437, and garbles the non-ASCII characters in file names.
> I wrote a quick and dirty workaround for Mac OS archives: if the file doesn't exist, encode the name as CP437 and check again. I'll think of something more clever if I ever switch to another OS.
This is my script to “clean up” a typical macOS zip file.
It assumes you’ve started by asking the Finder to “Compress” (which creates a zip file but one riddled with macOS-isms).
#!/bin/bash
zipfile=$1
if [ "x$zipfile" = "x" ] ; then
echo "$0: .zip file expected" >&2
exit 1
fi
zip -d "${zipfile}" "__MACOSX*"
zip -d "${zipfile}" ".DS_Store"
zip -d "${zipfile}" "*/.DS_Store"
unzip -l "${zipfile}" | sort -k 5
Hidden dot files were introduced as a bug and left to linger because they were kind of useful. It all started when someone tried to hide . and .. from the output of ls and messed up the if statement to only check if the first letter in the filename was a period instead of testing for the intended use cases. People then copied that behaviour around because it was a cool new trick, not because it was set up as a standard.
The Windows method, leveraging file attributes, is actually much cleaner in my opinion. You can set the hidden attribute in ZIP files and most tools do for OS specific files and folders, but I don't want my ZIP tool to put files on my file system that I don't get to see first so I always turn them on.
Windows does the same thing with desktop.ini, but I rarely encounter those anymore. It used to be that every ZIP had a bunch of thumbs.db files but Microsoft seems to have cut that out.
...Or does he just remember all past HN threads, and can easily out their entire titles, URLs, dates, and number of posting from memory on the fly, writes poetry about his cat, and is lots of fun at parties, like Data?
There used to be someone on Reddit who knew every scene and episode of The Simpsons and would respond to every comment he was called into to regurgitate his knowledge, so while your comment might be a bit over the top, I'm sure it's not super far from the truth!
@dang, if you see this: Can you fix this for me? It's hard to be a part of the discussion here with this type of thing: "You're posting too fast. Please slow down. Thanks." (I can reply to email sent to the one in my profile for verification, I can't seem to originate emails from there, though, as it's a Fastmail "masked email".)
I use HN Search on the fly but have a bunch of keyboard shortcuts in a browser extension to make it faster. Past explanations here if anyone cares: https://news.ycombinator.com/item?id=28441182.
Since Windows and MacOS don't seem interested in having mutually supported advanced filesystems, it would be nice if we just standardized on a flat file structure to implement portable features.
fat32 is very portable but has a 4GB limit and lacks entered attributes. exfat probably supports everything but until very recent open source drivers it lacked cross platform compatibility.
Plus too many devices are going to be limited to fat32 so we might as well just use that as a common denominator.
It would be nice if a DS_Store system could be used to associate split files (e.g. $NAME.1 $NAME.2, etc.) as a word around for the 4GB limit. User permission attributes and other extended attributes could be stored as well.
What's interesting is that Fat32 had this feature implemented on OS/2 and Win NT via files with the "␠EA.␠SF" suffix. Unfortunately looks like it was dropped in Windows 2000.
exFAT is great other than its lack of journaling, which has bit me in the ass more than a few times. file system durability is really important, especially for one for a drive being frequently moved between environments
I’m not sure this resolves anything. The disk format is no longer secret, true, but the patents are still in place and I don’t see a license anywhere in the link. (Even the one that’s present in the FAT spec released with UEFI is only valid as long as you are using it to implement UEFI.) Linux specifically gets a free pass because Microsoft joined the corresponding patent cartel (the “Open Invention Network”) in ’18, but FOSS at large is still at risk.
(I also don’t see why anyone would implement the allocation-bitmap-based exFAT instead of the extent-based UDF. They haven’t even added a third FAT! ... But that’s an unrelated matter.)
So, why aren't these just stored in a db or some other directory of the OS? Instead of littering the userspace, and no fs-ext required. Also it sounds like these should be per-user-and-dir instead of just per-directory.
So that when you copy/compress/make disk image of/whatever a folder that you have customized (e.g. marked files as red or whatever or moved them in specific locations) they are as you set them when you or someone else opens them in another Mac.
Initially, because the early Mac OS X Finder was a bad enough set of compromises and slow as molasses without adding database lookups to another part of the file system (at a time when HDDs were universally spinning rust and iBooks and PowerBooks often trucked along with 4200rpm drives). The .DS_Store files could be there already in the directory and hidden from users with a flag while Apple figured out whether UFS or HFS+ was their future file system.
Now? Well I/O performance is hardly a consideration for the latest Macs, but they’re there and they work, except the Mac OS X Finder 20 years later while far better than it’s debut performance is still a load of crap so it can still forget your custom view settings for a directory.
Ah, .DS_Store. As a Windows user forced to endure this offensive clutter I curse the fucking bastard that came up with that.
No love either for the twerp who invented desktop.ini or caused thumbs.db to seemingly keep coming back after I repeatedly turn off that feature.
If you're interested in such voodoo arts check out NTFS Alternate File Streams too (which while the bane of many at least have the decency not to spread to filesystems that don't support them).
This wasn't much of an explainer, it didn't even cover all the things that .DS_Store is capable of managing. The wiki does a better job of covering the topic.
It's also not something to get upset about - if .DS_Store or Apple double files (or for that matter Desktop.ini / thumbs.db) are a surprise then it's time to hang up one's coat, they're trivially revealed in the terminal and so predictable and easy to deal with that complaining about them is akin to holding up a dunce sign.
If one has purchased hardware/software that can't cope with them: get a refund, this shit is entry level basic.
Finally I understand the perspective of mac users in this problem. I had assumed that showing hidden files and manually cleaning up the zip archive would be a "simple no-brainer solution".
I just can't comprehend the reasoning of hiding .DS_Store beyond the reach of viewing hidden files. This is user-hostile. Why don't the native zipping tools automatically ignore these files when creating an archive?
No because they include a lot more information than that. For example, where files in the recycle bin came from is stored in DS_Store.
I know this because there's a bug in MacOS they refuse to fix where you can't restore files put into the recycle bin programmatically. This is because the API doesn't write the correct stuff to DS_Store.
Further, the format is proprietary, and I've even read that its structure changes depending on how it's used.
TLDR: .DS_Store files are Desktop Services Stores, containing that folder’s custom attributes, things like icon positions, and in more recent versions of macOS custom settings for the display of file metadata. Among the most important for some users are Finder or Spotlight Comments, which are normally displayed in the Comments section of the Get Info dialog for a file.
On filesystems that support the features: in the file system itself. On filesystems that don't support the features: somewhere on the system itself (like in the registry on Windows) or not at all (for example, when trying to set permissions on FAT32). Other operating systems generally don't create hidden files for this reason.
They do, however, create their own versions of the recycle bin/trash folder. If you use a flash drive on macOS, Windows and some Linux environments, you'll end up with three different recycle bin/trash folders. A bit annoying, but those folders are usually empty anyway.
Windows warns about the inability to copy permissions to FAT32 (in some cases, at least) and some command line tools on Linux do as well, but it depends on the tool.
It would be nice if Linux at least could see the Recycle or Trash folder and use what’s already there. Even better if both macos and Windows started doing the same.
From my knowledge .DS_Store is something everyone hates and no one actually benefits from, I personally hate it with passion and there's several times I wanted to work at Apple to remove it. Is there a reason why macOS still keeps it, do they actually think it's a good idea, or they don't like to listen to users, or they didn't get to do it yet?
> from my knowledge .DS_Store is something everyone hates and no one actually benefits from,
Small correction then; "something non-mac users hate and non-mac users don't benefit from"
I doubt you'll ever hear a Mac user advocate for DS_Store files because by design they do their job best when you never realize they exist. I think for __some__ users once they realize icon positions are maintained there, they'll be a bit more defensive. For many users (myself included), since it can also control view settings for a given folder [0], that I would be pretty peeved to lose.
Computers create junk files all the time and carry a lot of bloat, and furthermore one person's bloat is another person's workflow.
No joke, back when I was still doing general helpdesk support we had some "sort of clever" users who found out that they could recover some project files from %appdata% and relied on this for "auto-saving" their projects. When a new fileserver was spun up and this time the admin forgot (decided not to?) to toggle redirection for Appdata, this user was in for a shock when we had no backup of their Appdata and they had lost months of work.
Personally DS_Store is a very minor annoyance at best, if even noticeable. For version control it's just another file to add to ignore lists, and for network shares it's a very simple MacOS command to disable writing them to network shares. But it's not fair to call it worthless, it has a lot of good uses if you're a Mac user.
.DS_Store files are also a PITA for Mac users who mainly work on the command line. I don't understand why this data isn't stored in filesystem attributes instead of 'user-visible' files (for some reason I didn't have problem with .DS_Store on my previous Mac, but now on the new Mac they're suddenly everywhere again, might be that I somehow disabled them years ago on the old Mac though).
There is an awesome Atom plugin called "ds-store-delete" which allows me to use ctrl-alt-x or click a bright red button in the bottom left corner to delete all .DS_Store files in a workspace.
I just realized that once my girlfriend (against all my advice...) gets her new Mac, these fuckers will show up on our NFS server, huh?
Does anyone know how to prevent that from happening? It's a non-auth NFS, so anyone on the network is supposed to see and use it. So a server-side solution would be preferable, but client-side will do in a pinch.
There’s always been a hidden setting to stop Finder from generating these on a server, but the downside is that it won’t remember things like your girlfriend’s view preferences on the server (window position, sort options, text size, view type e.g. list or icon view).
That setting is also per user account, so if you have a user account for yourself on the same computer you would have to set it for yourself as well.
iptables can forbid her to access the NFS. I have a script to delete them with cron scanning the entire drive, since I have the same weakness for Mac users as you.
It makes sense to hide these files from many macOS users who don't care about them. For developers (who should be able to work with a Terminal) there's no need to resort to any magic or special command to make them visible! They are normal hidden files. A simple `ls -a` will list them too. You also have to put them in .gitignore as you put desktop.ini and other window managers' metadata files.
I wish the author explained in more technical detail or clearer how and where these files cause problems (not doubting that it does). In that case, it would have made a really useful article.
More than the DS_Store files what fascinates me is that when you delete something on a USB stick it doesn't go away, making the life of the stick limited to one write. This, unless you format it, but how many regular users understand it?
The champions in usability...
It just used different words. "Sometimes they’re confused with another annoying but more useful hidden file: shadow files whose names start with ._ which are used to carry extended attribute data as part of the AppleDouble file format used on some FAT file systems."
Yep, .DS_Store files have nothing to do with resource forks.
Resource forks are a filesystem feature (well forks are a filesystem feature and resource forks are an application of that feature). AppleDouble is a way of storing the information that would be in resource forks on filesystems that don't provide forks or similar functionality.
They have everything to do with resource forks. The metadata in DS_Store is stuff that was originally stored in the resource fork on earlier MacOSii; the file's existence is a cludge created to deal with the lack of resource forks when the decision was made to design OS X with POSIX-compatibility.
But it's a pretty darn good symbol for how Apple doesn't give a flying fuck about anyone's time/nerves, just dumping these on everyone without asking and letting us deal with their laziness.
Plus, Apple users then typically just rolled their eyes when I point out to them that I've now got all these files on my USB drive, thanks for nothing Apple. "Why do you even care?" Yeah, why do I? There's just no reason I should have to sort through these to keep them out of my systems, yet I don't want them there so no I have to, thanks apple... (this was before I got good on the terminal so it was actually quite annoying)
Bottom line: this still gets my blood boiling. Fuck apple, they're the new Microsoft in every way.