End of FAT directories

Hello Linux FAT developers and developers of related tools,

there seem to be different interpretations around how to read a FAT directory that contains an entry starting with a NUL byte before further entries *NOT* starting with NUL bytes. I uploaded an example image of this type to [1]. It contains three zero-byte files "WINDOWS", "IS", "GREAT", an empty directory slot and a file with contents named "NOT".

I'm afraid this is not a pure academic post, but I write it because I spent hours on trying to find out why I had problem to make some USB flash drive bootable with syslinux (which follows the MS model on loading files while booting) after copying files with the linux kernel (which follows the Linux model, obviously), additionally dosfsck on that stick showed quite strange results (it dived into a directory that does no longer exist). Another incarnation is mentioned in
https://wiki.physik.fu-berlin.de/linux-minidisc/doku.php?id=himddiskformat
in the section "Filesystem Issues with some Models on Linux". Some of the Sony devices clear only the first half of the cluster containing that subdirectory to zeroes, which works fine if software aborts on the first entry being zero, but obviously makes problem if the second half of the cluster is accessed.

I would really appreciate it if all FOSS working on FAT would agree to one behaviour, and I think we have no choice except being MS compatible (i.e. implement the behaviour as it has always been in DOS and Windows)

It is simply the bug of Sony devices, and can't work on it for both of Windows and Linux (it have to be fixed by fsck).

Assume 0-3 directory entires fine, 4 is zeroed entry, and 5- is crappy data.

1| ENT1        valid
2| ENT2        valid
3| ENT3        valid
4|             zeroed
5| djafdjoi    crappy data

...

Windows will stop at 4, but if windows write new entry as 4, 5- will show those up as crap like linux.

In short, in spec, directory must be initialized by zero (it is same assumption on linux and windows).

Quoted 2 lines

Windows will stop at 4, but if windows write new entry as 4, 5- will show those up as crap like linux.

You are right. Thanks for your input. This means Windows and mtools behave exactly the same way (and thus mtools can not really be considered buggy). I expected Windows to fix up the end, but it does not.

I still would prefer greatly if Windows and Linux at least read media the same way (i.e. stop on the first zeroed entry), I even have a patch for that somewhere around I can dig out.
I furthermore think it is quite important for dosfsck and the linux kernel to behave consistent (think of valid entries after the zeroed entry linking to "lost clusters"). If I prepare patches for the Linux kernel to
- terminate reading at the first zeroed entry

- making sure that if a zeroed entry gets populated, the next entry
gets zeroed unless we hit the end of the directory (deviating from Windows behaviour, intentionally, to keep the filesystem consistent)

And for dosfsck to
- terminate reading at the first zeroed entry
- offer filling the remainder of a directory cluster with zero bytes.
Would this set of patches be acceptable?

Quoted 1 line

- terminate reading at the first zeroed entry

This is acceptable (stop at zero is an simply optimization), and I'll just care about stability and cleanness about it.

Quoted 3 lines

- making sure that if a zeroed entry gets populated, the next entry
gets zeroed unless we hit the end of the directory (deviating from Windows behaviour, intentionally, to keep the filesystem consistent)>

I.e. This is the out of specs anymore. Why can you say those are invalid entries even if Windows honors those entries?

And the above is simulate Windows implementation over specs, why isn't here?

For overwriting the
zeroed entry, why we have to check all entries until see next zeroed (yeah, now, we allowed the crappy data after zeroed)?

My state is simple - the current implement should be fine. I.e. uninitialized directory is the simply bug.

Quoted 4 lines

And for dosfsck to
- terminate reading at the first zeroed entry
- offer filling the remainder of a directory cluster with zero bytes.
Would this set of patches be acceptable?

Sorry. I can't say about dosfsck, I'm not maintainer of it. Ideas, dosfstools maintainer?

Quoted 2 lines

And the above is simulate Windows implementation over specs, why isn't here?

Wow, there actually *is* a formal specification for FAT? I am positively surprised by that. I used to think that the "spec" for FAT is "do as DOS does". On a quick google, I do find the MS EFI FAT32 specification, which is obviuosly non-free. Is there anything else?

Quoted 3 lines

For overwriting the
zeroed entry, why we have to check all entries until see next zeroed (yeah, now, we allowed the crappy data after zeroed)?

My point was not about checking all entries until the next zeroed entry, which is overcomplicating stuff. I meant that if we hit the end (i.e. the first zeroed entry) when searching for free slots, in that case we just clear the one entry directly following the inserted entries. This makes sure that no files magically appear. I *do* understand that they also appear on Windows, though. My point about "modifying a FAT system that passes MS chkdsk with Linux should result in a FAT system that passes MS chkdsk" is in my eyes more important than "Windows damages a file system it considers fine in this situation, so we should do the same."

In the end, there are two valid interpretations on the problem at hand:
- DOS/Windows is buggy in exposing the crap when filling the "hole",
which is proven by chkdsk not complaining about the garbage in the directory, and we should not follow the bug in DOS/Windows

- chkdsk is buggy in not complaining about trailing garbage in
directories, which are proven to be invalid by DOS/Windows messing up on them, and we shouldn't complicate our FAT implementation by trying to work around a bug in chkdsk.

Obviously, I tend to the first interpretation while you prefer the second interpretation. As you are the maintainer, it looks like I have to get away with that.

Quoted 2 lines

My state is simple - the current implement should be fine. I.e. uninitialized directory is the simply bug.

For the uninitialized entries on Hi-MD media, I agree with you that these are caused by a firmware bug of older Sony devices. And to support these devices "good enough", the stop-at-zero would be enough. Regretfully I don't know the history of the USB flash memory I recently got where the root directory was ended by a single entry starting with a NUL byte and containing valid entries past that. The experience with the inconsistent behaviour of different tools on that flash memory made me write the mail about the current state of the FAT implementation.

I think the *most* important fact is to have at least dosfsck and linux consistent about directories with trailing garbage. So I won't even try to submit a patch to the Linux kernel that stops reading at the first unused directory entry if dosfsck will not accept to stop reading at the first unused directory entry.

Quoted 3 lines

For overwriting the
zeroed entry, why we have to check all entries until see next zeroed (yeah, now, we allowed the crappy data after zeroed)?

My point was not about checking all entries until the next zeroed entry, which is overcomplicating stuff. I meant that if we hit the end (i.e. the first zeroed entry) when searching for free slots, in that case we just clear the one entry directly following the inserted entries. This makes sure that no files magically appear. I *do* understand that

In the end, there are two valid interpretations on the problem at hand:
- DOS/Windows is buggy in exposing the crap when filling the "hole",
which is proven by chkdsk not complaining about the garbage in the directory, and we should not follow the bug in DOS/Windows
- chkdsk is buggy in not complaining about trailing garbage in
directories, which are proven to be invalid by DOS/Windows messing up on them, and we shouldn't complicate our FAT implementation by trying to work around a bug in chkdsk.

I think the *most* important fact is to have at least dosfsck and linux consistent about directories with trailing garbage. So I won't even try to submit a patch to the Linux kernel that stops reading at the first unused directory entry if dosfsck will not accept to stop reading at the first unused directory entry.

Quoted 8 lines

Quoted 3 lines

For overwriting the
zeroed entry, why we have to check all entries until see next zeroed (yeah, now, we allowed the crappy data after zeroed)?

My point was not about checking all entries until the next zeroed entry, which is overcomplicating stuff. I meant that if we hit the end (i.e. the first zeroed entry) when searching for free slots, in that case we just clear the one entry directly following the inserted entries. This makes sure that no files magically appear. I *do* understand that

This sounds like a good idea, and costs nothing. I hope we can convince Ogawa...

Quoted 5 lines

I think the *most* important fact is to have at least dosfsck and linux consistent about directories with trailing garbage. So I won't even try to submit a patch to the Linux kernel that stops reading at the first unused directory entry if dosfsck will not accept to stop reading at the first unused directory entry.

They should *not* be consistent.

Kernel should stop at zero invalid entry.

fsck should consider any garbage past zero entry as an error, and zero it out. (Complaining about duplicate blocks is unhelpful but better than nothing. It should really zero the garbage out.)

Am Donnerstag, den 28.04.2011, 15:25 +0200 schrieb Pavel Machek:

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

For overwriting the
zeroed entry, why we have to check all entries until see next zeroed (yeah, now, we allowed the crappy data after zeroed)?

My point was not about checking all entries until the next zeroed entry, which is overcomplicating stuff. I meant that if we hit the end (i.e. the first zeroed entry) when searching for free slots, in that case we just clear the one entry directly following the inserted entries. This makes sure that no files magically appear. I *do* understand that

Quoted 2 lines

This sounds like a good idea, and costs nothing. I hope we can convince Ogawa...

I am going to implement that and publish a patch. If the patch is clean enough, maybe we can convince him. But the freedom of open source can't prevent me from using a patch like that, of course. And maybe it also helps when the patch gets some testing.

Quoted 1 line

Kernel should stop at zero invalid entry.

Quoted 3 lines

fsck should consider any garbage past zero entry as an error, and zero it out. (Complaining about duplicate blocks is unhelpful but better than nothing. It should really zero the garbage out.)

In fact, what you describe is something I would call "consistent". I was not implying that kernel and dosfsck should do exactly the same thing, but I was implying that the kernel and dosfsck should either both or none consider entries past the gap as existing file.
Current state is: both consider past-gap entries valid. Your described state will be: none consider past-gap entries valid. The behaviour you describe is exactly what I would imagine as best way to go. Maybe fsck should ask for "shift post-gap entries/create a dummy deleted entry" vs. "clear post-gap entries".

Quoted 9 lines · Michael Karcher

Am Donnerstag, den 28.04.2011, 15:25 +0200 schrieb Pavel Machek:

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

For overwriting the
zeroed entry, why we have to check all entries until see next zeroed (yeah, now, we allowed the crappy data after zeroed)?

My point was not about checking all entries until the next zeroed entry, which is overcomplicating stuff. I meant that if we hit the end (i.e. the first zeroed entry) when searching for free slots, in that case we just clear the one entry directly following the inserted entries. This makes sure that no files magically appear. I *do* understand that

Quoted 7 lines

Quoted 2 lines

This sounds like a good idea, and costs nothing. I hope we can convince Ogawa...

I am going to implement that and publish a patch. If the patch is clean enough, maybe we can convince him. But the freedom of open source can't prevent me from using a patch like that, of course. And maybe it also helps when the patch gets some testing.

Of course, it's good to publish. However if you can't detect buggy or corrupted directory, I wouldn't ack to help buggy stuff.

So it can be the mount option, but I can't see at all the point it is in-kernel as FS.

Quoted 1 line

Quoted 1 line

Kernel should stop at zero invalid entry.

I'm not complaining to stop at zero. Rather I acked already if it didn't overwrite after zeroed entry.

Quoted 4 lines

Quoted 3 lines

fsck should consider any garbage past zero entry as an error, and zero it out. (Complaining about duplicate blocks is unhelpful but better than nothing. It should really zero the garbage out.)

Quoted 9 lines

In fact, what you describe is something I would call "consistent". I was not implying that kernel and dosfsck should do exactly the same thing, but I was implying that the kernel and dosfsck should either both or none consider entries past the gap as existing file.
Current state is: both consider past-gap entries valid. Your described state will be: none consider past-gap entries valid. The behaviour you describe is exactly what I would imagine as best way to go. Maybe fsck should ask for "shift post-gap entries/create a dummy deleted entry" vs. "clear post-gap entries".

Yes. Also my opinion is the fsck should get ack from user to overwrite data or salvage (has possibility to salvage) if interactive mode, then fixes it.

Thanks.

Quoted 5 lines

Kernel should stop at zero invalid entry.

fsck should consider any garbage past zero entry as an error, and zero it out. (Complaining about duplicate blocks is unhelpful but better than nothing. It should really zero the garbage out.)

In fact, what you describe is something I would call "consistent". I was not implying that kernel and dosfsck should do exactly the same thing, but I was implying that the kernel and dosfsck should either both or