Writing large files onto sync mounted MMC corrupts the FS
Hi people.
I am seeing both VFAT and Ext3 becoming corrupted when using synchronous mount on MMC card (kernel is 2.6.12-rc2-omap1, HW is a test board with Omap 1710). The problem is reproduced by mounting a MMC card synchronously and copying large file onto it.
Typically I insert vfat/ext3 formatted MMC card and do:
mount /dev/mmcblk0p1 /media/mmc1 -t vfat -o sync cp /lib/libc-2.3.2.so /media/mmc1
or
mount /dev/mmcblk0p1 /media/mmc1 -t ext2 -o noatime,sync cp /lib/libc-2.3.2.so /media/mmc1
The results are further below. I have tried to reproduce VFAT problen in desktop PC with loopback device but to no avail. I have used different MMC cards but the problem persists. Disabling DMA reduces the number of error messages or pushes the first occurence a bit further, but does not remove the problem.
Very similar problem appears when writing a large file onto ext2 formatted MMC card. The problem disappears if I mount asynchronoysly. Also similarly the problem on VFAT disappears when mounting asynchronously.
The VFAT problem seems to have appeared around January -- February this year but it is hard to tell the exact point and changeset. For ext2 I haven't done similar search.
Because there are two so similar problems appearing with two separate file-systems, I doubt this is VFAT or Ext3 only problem, but more likely problem in the mmc driver. It would be good to try this out on other on some other hardware using the same mmc core code. Has anybody else seen these?
Regards
Jarkko Lavinen
VFAT problem:
su-18-18:~# mount /dev/mmcblk0p1 /media/mmc1 -t vfat -o sync,rw,noauto,nodev,noexec,nosuid
ke-recv.c:132: iface: org.kernel.kevent, member: mount, path: /org/kernel/block/mmcblk0/mmcblk0p1
su-18-18:~# cp /lib/libc-2.3.2.so /media/mmc1
FAT: Filesystem panic (dev mmcblk0p1)
fat_get_cluster: invalid cluster chain (i_pos 7861)
File system has been set read-only
FAT: Filesystem panic (dev mmcblk0p1)
fat_free_clusters: deleting FAT entry beyond EOF
FAT: Filesystem panic (dev mmcblk0p1)
fat_free: invalid cluster chain (i_pos 7861)
FAT: Filesystem panic (dev mmcblk0p1)
fat_get_cluster: invalid cluster chain (i_pos 7861)
FAT: Filesystem panic (dev mmcblk0p1)
fat_free_clusters: deleting FAT entry beyond EOF
FAT: Filesystem panic (dev mmcblk0p1)
fat_free: invalid cluster chain (i_pos 7861)
cp: Write Error: Input/output errorEXT3 problem:
su-18-18:~# mount /dev/mmcblk0p1 /media/mmc1 -t ext2 -o noatime,sync ke-recv.c:132: iface: org.kernel.kevent, member: mount, path: /org/kernel/block/mmcblk0/mmcblk0p1 su-18-18:~# cp /lib/libc-2.3.2.so /media/mmc1 jffs2_get_inode_nodes(): Data CRC failed on node at 0x05eeaee4: Read 0x0e2f9681, calculated 0xbcd5ccf9 (Unclean reboot without syncing, usually harmless) jffs2_get_inode_nodes(): Data CRC failed on node at 0x05ef66e0: Read 0x1d6fb47e, calculated 0xaf610aef (Unclean reboot without syncing, usually harmless) EXT2-fs error (device mmcblk0p1): ext2_free_blocks: bit already cleared for block 6771 EXT2-fs error (device mmcblk0p1): ext2_free_blocks: bit already cleared for block 6772 EXT2-fs error (device mmcblk0p1): ext2_free_blocks: bit already cleared for block 6773 EXT2-fs error (device mmcblk0p1): ext2_free_blocks: bit already cleared for block 6774 EXT2-fs error (device mmcblk0p1): ext2_free_blocks: bit already cleared for block 6775 su-18-18:~#
Quoted 27 lines · Jarkko Lavinen
I am seeing both VFAT and Ext3 becoming corrupted when using synchronous mount on MMC card (kernel is 2.6.12-rc2-omap1, HW is a test board with Omap 1710). The problem is reproduced by mounting a MMC card synchronously and copying large file onto it.
Typically I insert vfat/ext3 formatted MMC card and do:
mount/dev/mmcblk0p1/media/mmc1 -t vfat -o sync cp/lib/libc-2.3.2.so/media/mmc1
or
mount/dev/mmcblk0p1/media/mmc1 -t ext2 -o noatime,sync cp/lib/libc-2.3.2.so/media/mmc1The results are further below. I have tried to reproduce VFAT problen in desktop PC with loopback device but to no avail. I have used different MMC cards but the problem persists. Disabling DMA reduces the number of error messages or pushes the first occurence a bit further, but does not remove the problem.
Very similar problem appears when writing a large file onto ext2 formatted MMC card. The problem disappears if I mount asynchronoysly. Also similarly the problem on VFAT disappears when mounting asynchronously.
The VFAT problem seems to have appeared around January -- February this year but it is hard to tell the exact point and changeset. For ext2 I haven't done similar search.
Quoted 5 lines
Because there are two so similar problems appearing with two separate file-systems, I doubt this is VFAT or Ext3 only problem, but more likely problem in the mmc driver. It would be good to try this out on other on some other hardware using the same mmc core code. Has anybody else seen these?
Go read the thread on lkml a few weeks ago about how sync + vfat (and others) does nasty destructive things to flash media that doesn't have excelent wear leveling (vfat updates the fat sector for every block of data written when you use sync option, which means you do a lot of writes to one sector of the device, which is not good for devices with a limited number of writes). Typically when flash starts to reach it's limit in number of writes, it starts having trouble holding the value written (some bits start to drift back to one state) which can explain your crc errors and such).
In other words: Don't use sync with flash media. It's a horribly bad idea, and if you look into how flash works (at least the cheaper ones) you will realize why. Anything with a limited number of writes allowed to each sector should avoid rewriting the same sector again and again, and that is what sync does when you use filesystems designed for disks rather than flash. JFFS was designed for flash use and rotates the sectors it uses to store filesystem meta data and has spare space in the filesystem to do wearleveling at the filesystem level. vfat and ext2/3 do not, and it shows. At least write caching/delayed write back elliminates the worst of the rewriting of the meta data sectors.
Len Sorensen
Quoted 11 lines · Lennart Sorensen
In other words: Don't use sync with flash media. It's a horribly bad idea, and if you look into how flash works (at least the cheaper ones) you will realize why. Anything with a limited number of writes allowed to each sector should avoid rewriting the same sector again and again, and that is what sync does when you use filesystems designed for disks rather than flash. JFFS was designed for flash use and rotates the sectors it uses to store filesystem meta data and has spare space in the filesystem to do wearleveling at the filesystem level. vfat and ext2/3 do not, and it shows. At least write caching/delayed write back elliminates the worst of the rewriting of the meta data sectors.
So, we should first ask whether this happens with *new* MMC cards or not.
Signature · 3 lines
Best Regards, Artem B. Bityuckiy, St.-Petersburg, Russia.
Quoted 1 line · ext Artem B. Bityuckiy
So, we should first ask whether this happens with *new* MMC cards or
not.
Flash manufactorers typically claim 100k erase cycles -- some even 1M. If FAT is updated after each written 512 byte block, filling 64MiB card in synchronous mode would cause over 100k FAT updates. One couldn't even fill a new 64MiB card without already breaking it.
With asynchronous mount, one could limit the dirty_ratio from proc to reduce the sync time against sudden power loss. But wouldn't this have the same adverse wearing effect as with synchronous mode once the dirty_ratio is hit and writing still proceeds?
Jarkko Lavinen