Skip to contentSkip to conversation

vfat file system extreme fragmentation on multiprocessor

DISCUSSION Bodo Eggert 2 participants 2 messages Sep 12, 2008 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw
Quoted 13 lines · Lennart Sorensen
Quoted 11 lines · Harun Scheutzow

I like to share the following observation made with different kernels of 2.6.x series, a T7100 Core2Duo CPU (effectively 2 processors). I have not seen such a post while searching.

Two applications compress data at the same time and try to do their best to avoid fragmenting the file system by writing blocks of 50 MByte to a VFAT (FAT32) partition on SATA harddisk, cluster size 8 KByte. Resulting file size is 200 to 250 MByte. It is ok to get 4 to 5 fragments per file. But at random, approximately at every 4th file, there are a few 100 up to more than 4500 (most likely case approx 1500) fragments for each of the two files written in parallel.

[...]

Quoted 4 lines

I don't think fat filesystems have any concept of reserving space for expanding files. It's a pretty simple filesystem after all designed for a single cpu machine with a non-multitasking OS (if you can call DOS an OS).

And it's not designed to allow deleting files if they are open. Ad yet you manage to keep the allocation in memory after removing the directory entry. Keeping a pre-allocation will require a slightly different mechanism, because the pre-allocated space needs to be stolen sometimes.

Quoted 3 lines

Space tends to be allocated from the start of the disk wherever free space is found since otherwise you would have to go searching for the free space, which isn't that efficient.

That's how it used to be done in very ancient DOS versions. Current versions allocate from the most recently allocated cluster (and cache this value in the superblock). It would be perfectly legal to search from the directory's on-disk location onward, too, if you want to emulate ext2 behavior. (You'd pick a random cluster if you'd create a directory.)

[...]

Quoted 3 lines

Now what would happen if you used ftruncate to extend the file you open to a large size, and then started writing it, and then set the size correctly at the end?

The fs would write zeroes instead of the data, doing the same bad allocation, because FAT doesn't support holes.

#3 lore raw

For the records:

Both Windows Vista and XP SP3 produce very little or no fragmentation on same FAT partition and same workload (2 tasks writing in 50 MByte pieces, up to some hundred MByte file size). But XP SP3 has a stupid behavior, too, allocating new space always behind previous allocations, even if old files have already been really deleted (not in trash), at least for one session (not tried what happens after reboot). This can result in multi-GByte holes of free space.

Harun Scheutzow