That was an awesome article! The kind of rabbit hole I’d gladly go down :)
SD cards and USB sticks are just.... weird. I haven’t delved too deeply but maybe someone here knows.... why is this? At this point I almost think of them as ‘not disks’. The particular weirdness I’m thinking of is I’ve seen more than one drive just seem to die, just because you try and repartition them in Windows. Not even filling the drives up.
Is it the case that USB drive and SD card firmware is just crap? Like it makes assumptions about the device other than ‘this is just a set of blocks on flash memory’ and actually needs things on the disk at a file system level to be a certain way for it to work properly? I’m really curious.
Like it makes assumptions about the device other than ‘this is just a set of blocks on flash memory’ and actually needs things on the disk at a file system level to be a certain way for it to work properly?
The wear leveling algorithms are usually written to assume you're using FAT32 as the filesystem and certain parameters of it, but those assumptions are optimisations for wear and speed, not requirements --- AFAIK the last time I looked at this stuff in any deep detail, they weren't so crazy as to try to read the block content to determine the layout/filesystem, but those were still the days when 100k SLC was the norm and 10k MLC met with reliability skepticism.
The race for capacity means flash endurance has taken a steep nosedive with things like TLC/QLC, so if anything it's not really the firmware that's crap, it's the flash itself --- and the firmware is increasingly trying to compensate for it. For the cheapest USB/SD I think the firmware is actually stored in reserved areas of the flash itself, so any corruption has a much higher chance of rendering the device unusable.
Or the awesome progress viewer (pv command). This is cat with some feedback. I use it a lot to image stuff when the media is readable, if you have bad sectors you need some like ddrescue to image it.
Yes USB drive and SD card firmware is just crap as with any embedded software... Firmware is always crap... That said, NAND Flash storage is quirky.
Individual bits on a NAND flash can be addressed or read out in kB "page". Each pages can be "programmed" aka written to, by repeatedly applying desired bit patterns (0b11001001 ... 0b11001001 ... and it sticks). Bits can only be set by writes, not unset. "Depth" of write can be deepened for longer storage or can be used to represent multiple state in the same bit(e.g. 0V means zero, 0.5V means one, 1.0V means two ... that's ternary). State of the art of multi-level write is something like 256 or 1024 levels per cell(bit).
Above is for read and write operations. As said, write ops can set bits, but cannot unset(0b0010 can be overwritten to 0b1111 but not 0b1101). So, in real life, all data in the page is wiped clean before writing. However in erasing context, erase operations take "blocks", of dozens of "pages" together in few MB size where you'd love to specify individual cells or pages. This means a one byte change in a file on a disk necessitate few MB worth of erase. To make it even worse, one cell of the flash lasts only up to 10k program-erase cycles on ancient chips or could be little as 300 cycles on modern designs. Clearly a naive approach destroys NAND way faster that what is commercially acceptable pace. Error margins are even worse, I've read that ECC correction is part of normal read operations or something.
Concerning all these quirks combined, clearly the flash chip interfaces cannot be directly exposed to OS driver, disk LBA addresses cannot be linearly mapped to chip banks in the ways DIP EEPROMs might have been wired in the 80s. Write ops must be cached to minimize reprogramming, and also must be constantly redirected to least used areas. Erase ops must be substituted by marking pages safe to discard. Defragmentation must be handled disk-local to avoid stray active pages scattering. These features called Wear Leveling are present on virtually any flash storage devices, since around high capacity cards started to exceed 16GB in size. From then, the controllers grew to support MLC/TLC/QLC technology(256 level per bit thing) or DRAM-less SLC caching(use part of TLC/QLC chip as SLC as substitute for DRAM write cache - used to be multiple GB of DDR3 in itself at one point) etc etc.
In short, you're completely right that SD cards and USB sticks are "not disks". Large erase units and low endurance of NAND chips, that necessitate intricate nursing, is what makes them "not".
(I'm only a PC nerd, not a NAND expert. Especially numbers above are through my Google-fu so can be off)
> The particular weirdness I’m thinking of is I’ve seen more than one drives just seem to die, just because you try and repartition them in Windows. Not even filling the drives up.
I’ve absolutely had this issue. Lots. I can’t figure out why either. I have about half dozen Micro SD cards that now say they’re not formatted, and error when trying to format them in windows. All of them “broke” during a simple repartition in Disk Manager.
I’ve had minor success with some of them using fdisk to “repair” them but others just fail there too.
then partition for proper cylinder alignment even though experts have disparaged the need for this two decades now,
then format the target FAT32 partition using MS-DOS while actually booted to the W98SE startup floppy, CD, or equivalent (accepting no later DOS version than W98SE of 2001, which all proved to have lesser FAT consistency),
making sure both partition and format are accomplished using 255 heads & 63 sectors/track, correcting and re-doing if necessary, regardless of _native_ device geometry that may come up by default and need to be over-ridden,
you will almost never be as disappointed about compatibility as when you let other tools or devices prepare your FAT32 media.
________________
Even on devices which are intended to never boot the media, best results can also often be obtained when there is a master boot record (MBR) placed at the beginning of sector 0 anyway even though it's theoretically unnecessary. This is in addition to the partition table at the end of sector 0 which is actually essential either way for booting and/or mere storage.
As an an example if you simply create a valid partition table alone on a fully blank USB drive using a Linux tool, you would not yet have a DOS MBR at the beginning of the same sector 0.
Regardless, the partition should be ready for recognition and formatting by MS-DOS as FAT32 using its default geometry for the detected situation. After FAT32 formatting there will then be a volume boot record (VBR) at the starting sector of the partition a certain distance away from sector 0 (best recognition is usually when the VBR is at sector 63).
In MS-DOS, without disturbing the partitioon table, you place a DOS MBR on sector 0 (overwrites MBR zone and current MBR) using an undocumented switch in FDISK; FDISK /MBR when you are booted to floppy and there is no other drive hardware than the target, when the target is recognized as drive 0 by the BIOS and identifed as C:\ by DOS even before formatting, and supposedly when HDDs other than the hardcoded drive 0 target are present as well.
In Windows 10 a BIOS MBR can be written to a chosen sector 0 of a HDD containing a letter volume by BOOTSECT.EXE at the command line using the /MBR option. This occurs while you intentionally overwrite a target VBR simultaneously with an NT6 (BOOTMGR) version which is the main purpose of the BOOTSECT command. If this ends up being done you're probably better off rebooting to DOS and reformatting (/q option for a quick format) to replace with a DOS VBR before using the FAT32 volume.
But the Windows 10 MBR is a good MBR for an otherwise pure DOS HDD.
Unfortunately, occasional devices defy all patterns which Windows versions and sometimes MS-DOS itself are capable of recognizing and/or generating. Designers have poorly selected (sometimes bizarre) partitioning and/or formatting layouts far removed from what was consistent with all that had come before, even though there could be superficial Windows compatibility apparent at one point. This was not progress since FAT32 with LFN is a long-established, stable, now patent-unencumbered standard, and it is more valuable than ever to maintain compatibility back to its foundational 1999 OS version, before important features were compromised in ways which could unfairly make NTFS seem more appropriate by comparison.
For that you may just have to format media in the device itself to achieve its unique desired geometry & layout.
SD cards and USB sticks are just.... weird. I haven’t delved too deeply but maybe someone here knows.... why is this? At this point I almost think of them as ‘not disks’. The particular weirdness I’m thinking of is I’ve seen more than one drive just seem to die, just because you try and repartition them in Windows. Not even filling the drives up.
Is it the case that USB drive and SD card firmware is just crap? Like it makes assumptions about the device other than ‘this is just a set of blocks on flash memory’ and actually needs things on the disk at a file system level to be a certain way for it to work properly? I’m really curious.