superblock checksum mismatch after crash, cannot mount

About SD cards and somesuch...

TL;DR: THINK TWICE before formatting SD cards!!!

What is SD card? One or several NAND flash ICs + controller doing wear leveling and interface translation. It does wear leveling and handles flash blocks translation to show you what you expect, making it look like if it can deal with 512 byte sectors, while flash haves totally different blocking factors.

There'are some bumps on this way. Most of issues come from the fact that flash memory *lacks* "write" operation and uses very specific block operations. There is "page program" and "block erase". Page is typically 1 to 4KiB (+some bytes for ECC, "out of band" data). Erase block is much larger and can usually vary about 256KiB to about 16MiB. Advanced flash storages could prefer to do erase operations in larger erase groups, probably to speed things up by submitting commands to several flash IC at once. This can make preferred erase block size of flash-backed storage even larger.

Once flash page is programmed, it can't be filled with different data unless whole huge erase block is erased by erase operation, clearing all pages in block to their erased state. And then flash can withstand only limited amount of cycles, about several thousands for MLC (2 bit per cell) NAND and only few hundreds rewrites for TLC (3 bit per cell) nand. Then there could be bad blocks, both pre-existing at manufacture time and newly appeared due to wear. And some reserve space to cover new bad blocks. And software expects all these damn 512 byte sectors.

Sounds a bit complicated, isn't it? So there is rather comples translation layer on the way which does translations, bad blocks mapping, wear leveling and so on. In fact even SD card haves rather smart firmware in controller which does many non-trivial operations. But it haves simplified routines, which makes it worse than SSDs.

So what? There're some things you do not expect.

1) Formatting and repartitioning SD card? Generally WORST IDEA EVER. Unless you got idea about NAND flash, how it works and able to guess flash geometry and place filesystem adequately and hint it to use sane blocking factors (hard thing to do), you better do not repartition or format SD cards at all. Factory file sysmte comes pre-formatted in very special way. Generally if you look on new card, you will see fileystem follows special patterns. First, partition table looks a bit empty. There is partition table block and then ... some unused space. While it sounds dumb, this strange thing being done to put partition table in separate erase block, so it never read-modify-written when FAT entries are updated. Should something go wrong, FAR can recover from backup copy. But erased partition table just suxx. Then, FAT tables are aligned in way to fit well around erase block bounds. And of course filesystem blocks should map correctly over NAND pages, because getting filesystem block put into middle of two pages means 2 pages have to be touched when you write single filesystem block just because of bad layout. Incorrect layout can kill write performance of SD card by 2-3 times, especially on small files/small writes and cause extra wear. This is known as write amplification factor.

This said, you can *try* to reformat, BUT no standard OS of firmware formatter will help you with default settings. They can't know geometry of underlying NAND and controller properties. There is no standard, widely accepted way to get such information from card. No matter if you use OS formatter, camera formatter or whatever. YOU WILL RUIN factory format (which is crafted in best possible way) and replace it with another, very likely suboptimal one. So you can easily reduce write speed, get increased wear and just make it unsafe. Up to sudden loss of partition table. So if you're about to try, remember at least the following: put partition table to separate erase block which is never touched. Sudden loss of partition table suxx. So, assuming most common 2-bit MLC NAND, first partition should start at least about 4-32MiB away from card beginning (more alignment does not hurts as long as it is power of 2 and you're ok with lost space). Large alignments of powers of 2 is your friend (in case TLC nand can be not a case due to strange block sizes). Filesystem start should be aligned to large power of 2 factors like units of 32MiB from begin of card. Filesytem blocks should be put on page boundary, avoiding case where filesystem block laid badly over pages boundary (if you lucky, can happen automatically if you align FS start in mentioned way). For these who is really inclined to try this and want to get sane results at cost of some card wearout, there are some tools like "flashbench" which will try to guess actual geometry and best alignment by actually doing some writes and looking on resulting speeds. Obviously, best blocking match leads to best speed as well due to least possible amplification factor.

2) Active writes? Especially small and random? Nice way to quickly kill your card. You see, wear leveling controller could be simplified. Then card often shown as mass storage and I'm not really sure this way card getting any hints about unused regions at all (shouldnt be huge issue for real sd/mmc hosts like Pi but can be problem with USB card readers and somesuch, I guess). If unused region hints are not used (no working DISCARD-like things in effect), wear leveller faces very heavy and suboptimal conditions, where it only haves few extra non-busy blocks for all operations and so it have to deal with read-modiwy-erase-write sequencing far more often than it could be in better conditions. Whole point of DISCARD was to improve operational conditions of wear leveler by giving hints which blocks are not used anymore so it can erase blocks in background and then use them to satisfy request in optimal ways, by just doing what has been requested with minimal overhead. This speeds up write (read-modify-erase-program sequence is slow) and reduces amplification factor, hence less wear. And why so many buzz about amplification? Imagine you'll write 64-bytes file. Due to sector nature it would be at least 512 bytes operation (8x amplification). Maybe more if filesystem only works with blocks. Controller will at least read something like 4KiB page, no matter what, and patch requested 512 bytes inside, and at least program it to new place, should there be free page. So now 64 bytes are 4KiB operation, huh? Buf if there was no free page, controller will have to do some read-patch-erase-write for whole erase block, often 4MiB or more. Or even several blocks + forced garbage collection, if there was no empty erase block to put patched data on (likely scenario if you haven't got DISCARD support on wheels). So, several 4MiB blocks were read, patched and at least one was erased? For that silly 64 byte write? That's what we call amplificaton! Needless to say, if you feed card with small random writes, it will die much, much faster than you can expect from card size multiplied by erase cycles NAND can withstand. Now you probably understand why your card will never reach rated write speed on small files. If you write 100MiB file, it will eventually self-sync on blocking border, making amplification factor close to 1. Not a case for 64-byte files. Btw same applies for SSDs, though their larger capacity, widespread discard support and better controllers and algos make it less daunting problem for sure.

Solution? Use heavy RAM buffering (increase write barrier times?) and try to hint filesystem to aggregate writes to large sequential blocks, preferrably multiple of eraseblock size(in some FS its possible to "ab"use hinting intended for RAID stripes, etc). And DISCARD is your best friend, if you can get it working. Interestingly, CoW based designs are inherently more flash-friendly due to the fact wear leveler is kind of CoW itself and in lucly case it can be almost 1 to 1 mapping. In bad cases write amplification can occur of course. Best case is when file system is explicitly aware of flash blocking factors and uses this knowledge to optimise all sorts of things but its not what you can expect from SD cards and/or "generic" filesystems.

3) Attitude when NAND worn out. SSDs are usually relatively smart in this regard and will just go read only when they're close to expected maximum of erase-reprogram cycles of flash. And SSDs are quite good in ECC abilities. SD cards have cheap and weak controller and so much more dumb and what you see can wary in drastic ways. Sometimes critical structures in NAND could become corrupted to degree controller would fail to initialise card at power up, making data completely unavailable, card is pretty dead at this point. Sometimes controller would miss corrupt data. Sometimes it would be unable to handle it properly. In best case it would go readonly as SSDs do. SSDs have S.M.A.R.T. to help you to get idea how it performs. SD cards.... uhm, there was idea to do similar thing and newest standards added commands with similar meaning, but needless to say its not yet widespread and very optional thing, mostly meant for built-in eMMC "cards". So hell yeah, backup your data. You can't know when it will fail and you can't know how it would fail. Needless to say it makes it easy to suddenly loss all your data.

4) Unexpected power loss on write? Ouch, it hurts! Imagine power got lost in the middle of read-patch-erase-program sequence dealing with large 4+ MiB block, etc. It can happen data were not completely written back to new destination when power loss arrived, possibly resulting in data loss. SSDs are even nasty enough to keep interesting counter - "unsafe shutdown count", it increased each time you've turned off power without properly shutting down SSD first, hence causing risk of mentioned issues. That's why partition table *MUST* live in separate erase block. Else you can once figure out block has been erased but power lost before partition table data were written back. Hence, partition table loss. Sounds cool, isn't it? This case can also violate assumptions of most filesystems about how underlying layer behaves on power loss. Most filesystems just do not expect power loss can cause corruption of something like 4MiB eraseblock with all data it keeps instead of just data involved into current operation. Then, hidden geometry makes it hard to even predict which (meta)data will be damaged. This also affects SSDs to some degree. Toggling power of flash based storages when write is in progress and some time after it (where background logic can operate) is a really bad idea.

5) Sometimes you can face less common stuff like wear leveler hardcoded to deal with FAT32. Fat area can have smaller eraseblocks and wear levler can assume there is FAT and will adjust logic to better handle small frequent writes into this area. You never know what wear leveler will do. So once you trashed factory filesystem, you're really on your own and proceeding on your own risk.

Bottom line: THINK TWICE before formatting SD cards.���{.n�+�������+%�����ݶ��w��{.n�+����{���k~���^n�r���z���h����&���z��z�ޗ�+��+zf���h���~����i��������z_��j:+v���)ߣ�m

Am 24.08.2014 18:59, schrieb Flash ROM:

If you intended to make me hate SD cards then you did a really good job.

I didn't even think once, because actually I was never aware of those things you mentioned. In fact I still don't care about that, because I use my Pis mainly to experiment and to get experience with those failures, but if I ever use them in a real life situation then I'm definitely going to use an external drive and just put the bootloader on SD, without formatting. ;-)

Thank you for your detailed essay!

All cameras have a format function, and this function both partitions and formats (creates a filesystem/volume format). If they can get away with it, so can anyone else.

I think you seriously overestimate the knowledge of camera manufacturer's about the details of flash storage; and any ability to discover it; and any willingness on the part of the flash manufacturer to reveal such underlying details. The whole point of these cards is to completely abstract the reality of the underlying hardware from the application layer - in this case the camera or mobile device using it.

Also, with SDXC exFAT is now specified. And it has only one FAT there isn't a backup FAT. So they're even more difficult to recover data from should things go awry filesystem wise.

It's recommended by the card manufacturers to reformat it in each camera its inserted into. It's the only recommended way to "erase" the sd card for re-use, they don't recommend selectively deleting images. And it's known that one camera's partition table and formatting can irritate another camera make/model if the card isn't reformatted by that camera.

Absurd. It's recommended *by* both device manufacturers (cameras and mobile devices) that consume such cards, as well as the cards' manufacturers. The workflow you're proposing for SD Cards is as a write once device with no practical way of deleting the files on it.

Chris Murphy

If you really know what you are doing, it is possible to determine erase block size by looking at device performance timings, with surprisingly high accuracy (assuming you aren't trying to have software do it for you). I've actually done this before on several occasions, with nearly 100% success.

It's too bad that TFAT didn't catch on, as it would have been great for SD cards if it could be configured to put each FAT on a different erase block.

It's not just cameras that have this issue, a lot of other hardware makes stupid assumptions about the format of media. The first firmware release for the Nintendo Wii for example, chocked if you tried to use an SD card with more than one partition on it, and old desktop versions of Windows won't ever show you anything other than the first partition on an SD card (or most USB storage devices for that matter).