Skip to contentSkip to conversation

PATCH to access old-style FAT fs

DISCUSSION Frodo Looijaard 4 participants 17 messages Jan 26, 2004 – Feb 1, 2004

lore
Share & tools
lore
#0 lore raw

Hi folks,

I have created and attached a new version of my old-style FAT filesystem patch, this time for the 2.6.0 kernel. It can also be found on
http://debian.frodo.looijaard.name/.

Some old implementation of the FAT standard mark the end of the directory file index by inserting a filename beginning with a byte 00. All entries after it should be ignored, even though they are not marked as deleted. At least some EPOC releases (an OS used on Psion PDAs, for example) still use this policy.

The included patch adds the `oldfat' mount option for FAT-based filesystems. Without it, doing an 'ls' on old-style FATs will show a lot of garbage, both old files that have been deleted, and bogus entries that have never been filled.

If you do not use the `oldfat' mount option, FAT filesystems should work exactly as before. I have been very careful not to change the logic of the FAT driver, in order not to break anything. In fact, the patch could probably be optimized a bit, but I wanted to keep it as risk-free as possible.

It should be safe to mount a normal FAT filesystem with the `oldfat' mount option. I have been doing that on and off with some local Win98 filesystems without any trouble. Actually, the only reason I have introduced the mount option is to keep the new logic strictly seperated from the old driver.

All feedback would be very welcome. This patch is released under the GPL. Feel free to include it in stock kernels.

Debian users can install the .deb on http://debian.frodo.looijaard.name
and have the patch applied automatically when compiling 2.6.x kernels.

Thanks,
Frodo

Signature · 4 lines
Frodo Looijaard <[email protected]>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.
#1 lore raw
Followup to:  <[email protected]>
By author:    Frodo Looijaard <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 7 lines

Hi folks,

I have created and attached a new version of my old-style FAT filesystem patch, this time for the 2.6.0 kernel. It can also be found on
http://debian.frodo.looijaard.name/.

Quoted 6 lines

Some old implementation of the FAT standard mark the end of the directory file index by inserting a filename beginning with a byte 00. All entries after it should be ignored, even though they are not marked as deleted. At least some EPOC releases (an OS used on Psion PDAs, for example) still use this policy.

It's not just "old implementations" -- it's the spec.

After reaching a filename beginning with 00, no further data should be assumed to be in that filesystem. MS-DOS itself would only do that when formatting the filesystem, so *all* the subsequent entries would be assumed to start with 00, but that doesn't really seem to be to spec.

-hpa

#2 lore raw
Quoted 17 lines · H. Peter Anvin
Followup to:  <[email protected]>
By author:    Frodo Looijaard <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 7 lines

Hi folks,

I have created and attached a new version of my old-style FAT filesystem patch, this time for the 2.6.0 kernel. It can also be found on
http://debian.frodo.looijaard.name/.

Quoted 6 lines

Some old implementation of the FAT standard mark the end of the directory file index by inserting a filename beginning with a byte 00. All entries after it should be ignored, even though they are not marked as deleted. At least some EPOC releases (an OS used on Psion PDAs, for example) still use this policy.

Quoted 7 lines

It's not just "old implementations" -- it's the spec.

After reaching a filename beginning with 00, no further data should be assumed to be in that filesystem. MS-DOS itself would only do that when formatting the filesystem, so *all* the subsequent entries would be assumed to start with 00, but that doesn't really seem to be to spec.

The new cluster for directory entries must be initialized by 0x00. And, when the directory entry is deleted, the name[0] is updated by 0xe5 not 0x00.

So, if the name[0] is 0x00, it after, all bytes in cluster is 0x00.

The fat driver can stop at name[0] == 0x00, but this is just optimization. The behavior shouldn't change by this.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#3 lore raw
Quoted 9 lines · OGAWA Hirofumi

The new cluster for directory entries must be initialized by 0x00. And, when the directory entry is deleted, the name[0] is updated by 0xe5 not 0x00.

So, if the name[0] is 0x00, it after, all bytes in cluster is 0x00.

The fat driver can stop at name[0] == 0x00, but this is just optimization. The behavior shouldn't change by this.

I looked at the spec, and yes, that is how the spec reads:

If DIR_Name[0] == 0x00, then the directory entry is free (same as for 0xE5), and there are no allocated directory entries after this one (all of the DIR_Name[0] bytes in all of the entries after this one are also set to 0). The special 0 value, rather than the 0xE5 value, indicates to FAT file system driver code that the rest of the entries in this directory do not need to be examined because they are all free.

I guess the original poster has found filesystems which have a 0 followed by garbage. In cases like that, the cardinal rule for FAT is WWDD (What Would DOS Do)... since I'm pretty sure DOS stops examining at that point, we should do the same.

It's the same thing as with using 0xF8 for ending clusters; it's correct according to spec, but WWDD says 0xFF is the right thing.

-hpa

#4 lore raw
Quoted 20 lines · H. Peter Anvin
Quoted 9 lines · OGAWA Hirofumi

The new cluster for directory entries must be initialized by 0x00. And, when the directory entry is deleted, the name[0] is updated by 0xe5 not 0x00.

So, if the name[0] is 0x00, it after, all bytes in cluster is 0x00.

The fat driver can stop at name[0] == 0x00, but this is just optimization. The behavior shouldn't change by this.

I looked at the spec, and yes, that is how the spec reads:

If DIR_Name[0] == 0x00, then the directory entry is free (same as for 0xE5), and there are no allocated directory entries after this one (all of the DIR_Name[0] bytes in all of the entries after this one are also set to 0). The special 0 value, rather than the 0xE5 value, indicates to FAT file system driver code that the rest of the entries in this directory do not need to be examined because they are all free.

Quoted 7 lines

I guess the original poster has found filesystems which have a 0 followed by garbage. In cases like that, the cardinal rule for FAT is WWDD (What Would DOS Do)... since I'm pretty sure DOS stops examining at that point, we should do the same.

It's the same thing as with using 0xF8 for ending clusters; it's correct according to spec, but WWDD says 0xFF is the right thing.

The new cluster for directory entries must be initialized by 0x00. This is required by spec.

If cluster has garbage, the fat driver needs to do such the following part. Stop at DIR_Name[0] == 0 is not enough, and I don't think DOS does this.

@@ -709,7 +731,20 @@
}

- if (IS_FREE((*de)->name)) {
- if (++row == slots)
+ if ((oldfat && ((*de)->name[0] == EOD_FLAG)))
+ last_entry = 1;
+ if (last_entry || IS_FREE((*de)->name)) {
+ if (++row == slots) {
+ if (last_entry) {
+ /* If we encounter a last_entry, we
+ * need to mark the entry after the
+ * one to be inserted as last_entry
+ * now! */
+ if (fat_get_entry(dir,&curr,bh,de,i_pos) > -1) {
+ (*de)->name[0] = 0;
+ mark_inode_dirty(dir);
+ }
+ }
return offset;
+ }

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#5 lore raw
Quoted 4 lines · OGAWA Hirofumi
Quoted 2 lines · H. Peter Anvin

I guess the original poster has found filesystems which have a 0 followed by garbage.

Exactly. At least on those EPOC filesystems, and I seem to have had a old diskette around (perhaps out of the DR-DOS age?) which had the same problem. Regrettably, the diskette got hosed.

Quoted 2 lines

The new cluster for directory entries must be initialized by 0x00. This is required by spec.

Might be. In that case the EPOC filesystems do not conform to spec. Now that would be a first time.

Quoted 3 lines

If cluster has garbage, the fat driver needs to do such the following part. Stop at DIR_Name[0] == 0 is not enough, and I don't think DOS does this.

[Patch fragment to set DIR_Name[0] to 0 when new dir entries are added]

I don't know. I do not have a working MS-DOS partition anymore to test this on.

As I said, I *think* it is safe to have my patch always applied (that is, stop when DIR_Name[0] == 0, and be careful to add a new DIR_Name[0] = 0 entry when new entries are added at the back). It would conform to the standard. But I would not really be surprised if there was yet another FAT implementation somewhere out there that breaks the standard in some other subtle way, which works now but exhibits problems with my patch. That is why I made it a mount option.

Thanks,
Frodo

Signature · 4 lines
Frodo Looijaard <[email protected]>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.
#6 lore raw
Quoted 7 lines · Frodo Looijaard

As I said, I *think* it is safe to have my patch always applied (that is, stop when DIR_Name[0] == 0, and be careful to add a new DIR_Name[0] = 0 entry when new entries are added at the back). It would conform to the standard. But I would not really be surprised if there was yet another FAT implementation somewhere out there that breaks the standard in some other subtle way, which works now but exhibits problems with my patch. That is why I made it a mount option.

"stop when DIR_Name[0] == 0" should be added after cleanup. The option is not needed.

Honestly, I wouldn't like to add the "add a new DIR_Name[0] = 0" part. The option is added easy, but it is not removed easy. And we must maintain it. (BTW, looks like that patch is buggy)

Those stuff makes a change of the normal path difficult really. At the same reason, I removed the fat_cvf stuff.

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#7 lore raw
Quoted 3 lines · OGAWA Hirofumi

"stop when DIR_Name[0] == 0" should be added after cleanup. The option is not needed.

OK. That would be nice. Like I said, I just hope it does not break other FAT implementations, but that is not very likely. At least, that would allow read-only mounted EPOC FAT partitions to be handled correctly.

Quoted 3 lines

Honestly, I wouldn't like to add the "add a new DIR_Name[0] = 0" part. The option is added easy, but it is not removed easy. And we must maintain it.

I understand. I could always maintain that patch separately for those who need it (for read-write mounted EPOC FAT partitions).

Quoted 1 line

(BTW, looks like that patch is buggy)

Could well be. Any suggestions what to look out for?

Thanks,
Frodo

Signature · 4 lines
Frodo Looijaard <[email protected]>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.
#8 lore raw
Quoted 8 lines · Frodo Looijaard
Quoted 3 lines · OGAWA Hirofumi

"stop when DIR_Name[0] == 0" should be added after cleanup. The option is not needed.

OK. That would be nice. Like I said, I just hope it does not break other FAT implementations, but that is not very likely. At least, that would allow read-only mounted EPOC FAT partitions to be handled correctly.

This is spec as hpa saied. So this is right things, and other fat driver also does, AFAIK.

Quoted 6 lines
Quoted 3 lines

Honestly, I wouldn't like to add the "add a new DIR_Name[0] = 0" part. The option is added easy, but it is not removed easy. And we must maintain it.

I understand. I could always maintain that patch separately for those who need it (for read-write mounted EPOC FAT partitions).

Thanks.

Quoted 3 lines
Quoted 1 line

(BTW, looks like that patch is buggy)

Could well be. Any suggestions what to look out for?

@@ -200,4 +202,8 @@
parse_record:
long_slots = 0;
+ if (oldfat && (de->name[0] == EOD_FLAG))
+ last_entry = 1;
+ if (last_entry)

+			continue;
                        ^^^^^^^^

if (de->name[0] == DELETED_FLAG)
continue;

The above should "goto EODir;". Likewise, another "contiure" of fat_search_long().

@@ -709,7 +731,20 @@
}

- if (IS_FREE((*de)->name)) {
- if (++row == slots)
+ if ((oldfat && ((*de)->name[0] == EOD_FLAG)))
+ last_entry = 1;
+ if (last_entry || IS_FREE((*de)->name)) {
+ if (++row == slots) {
+ if (last_entry) {
+ /* If we encounter a last_entry, we
+ * need to mark the entry after the
+ * one to be inserted as last_entry
+ * now! */
+ if (fat_get_entry(dir,&curr,bh,de,i_pos) > -1) {
+ (*de)->name[0] = 0;
+ mark_inode_dirty(dir);

mark_inode_dirty(dir) is not needed, instead of it we should do mark_buffer_dirty(bh).

And this fat_get_entry() updates bh and de, but it should be point to allocated bh and de, not free entry. It's needed by msdos_add_entry().

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#9 lore raw

Hi,

I have run a small test in MS-DOS 5.00:

1) Create a new directory
2) Create five files in it
3) Change the first character of the second filename to 0x00 with an editor
4) Do a DIR listing: only one file is seen
Linux shows four files!
5) Create a new file
6) Do a DIR listing: there are five files

So MS-DOS 5.00 at least does stop when a 0x00 marker is found, but does not write new 0x00 markers when a 0x00 marker is overwritten.

Thanks,
Frodo

Signature · 4 lines
Frodo Looijaard <[email protected]>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.
#10 lore raw
Followup to:  <[email protected]>
By author:    Frodo Looijaard <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 11 lines

I have run a small test in MS-DOS 5.00:

1) Create a new directory
2) Create five files in it
3) Change the first character of the second filename to 0x00 with an editor
4) Do a DIR listing: only one file is seen
Linux shows four files!
5) Create a new file
6) Do a DIR listing: there are five files

Quoted 3 lines

So MS-DOS 5.00 at least does stop when a 0x00 marker is found, but does not write new 0x00 markers when a 0x00 marker is overwritten.

This is of course dangerous, since it could create conflicts.

Thus, I suggest we do write new 0x00 markers; even though it's not required (it only matters if the filesystem is out of spec and therefore by definition corrupt), it would help the Psion, and wouldn't cause any problems for in-spec filesystems.

-hpa

#11 lore raw
Quoted 6 lines · H. Peter Anvin

This is of course dangerous, since it could create conflicts.

Thus, I suggest we do write new 0x00 markers; even though it's not required (it only matters if the filesystem is out of spec and therefore by definition corrupt), it would help the Psion, and wouldn't cause any problems for in-spec filesystems.

Yes, that cluster is dangerous. However, repair should be fsck's job. IMO, for repair we shouldn't add unneeded I/O to normal path.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#12 lore raw
In reply to OGAWA Hirofumi

Hi,

I have attached a newer, better behaving version of my patch:
* Implements new mount option oldfat for FAT-derived filesystems.
* Stops scanning dirs when DIR_Name[0] = 0 when oldfat is set
* Writes a 0 to the next entry DIR_Name[0] when overwriting an entry which has DIR_Name[0] = 0 when oldfat is set

It has been tested with both msdos and vfat filesystems and seems to work well (unlike the patch of a few days ago, which had some issues).

Once you get around to cleaning up the code and making the stop-scanning-dirs-on-zero the default way, the patch can be shrunken to include only the third item.

Quoted 2 lines

The above should "goto EODir;". Likewise, another "contiure" of fat_search_long().

I am not convinced that the goto is always safe, and I am pretty sure that it works now (though not as efficient as possible, perhaps), so I have left that in place for now.

Quoted 5 lines

mark_inode_dirty(dir) is not needed, instead of it we should do mark_buffer_dirty(bh).

And this fat_get_entry() updates bh and de, but it should be point to allocated bh and de, not free entry. It's needed by msdos_add_entry().

Both are now implemented using the new fat_write_zero_entry function.

Thanks for your help,
Frodo

Signature · 4 lines
Frodo Looijaard <[email protected]>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.
#13 lore raw
Quoted 8 lines · Frodo Looijaard

Hi,

I have attached a newer, better behaving version of my patch:
* Implements new mount option oldfat for FAT-derived filesystems.
* Stops scanning dirs when DIR_Name[0] = 0 when oldfat is set
* Writes a 0 to the next entry DIR_Name[0] when overwriting an entry which has DIR_Name[0] = 0 when oldfat is set

Please don't call this "oldfat". There is nothing about this which is "old-style"; it's a workaround for a bug in a specific OS.

-hpa

#14 lore raw
In reply to Frodo Looijaard
Quoted 5 lines · Frodo Looijaard

I have attached a newer, better behaving version of my patch:
* Implements new mount option oldfat for FAT-derived filesystems.
* Stops scanning dirs when DIR_Name[0] = 0 when oldfat is set
* Writes a 0 to the next entry DIR_Name[0] when overwriting an entry which has DIR_Name[0] = 0 when oldfat is set

Just amused myself checking all data I can find about such stuff. And read the replies by Hirofumi and hpa.
I completely agree with them.

The DOS situation is not so much that 0 is an end-of-dir marker. It marks that this entry, and all following entries, has never been used.

For the time being there is no justification for "oldfat". The only place where this behaviour has been reported (not only by you, I saw several references) is on Psion PDAs. So "psion" would be a more obvious name, if a mount option is needed.

And maybe no mount option is needed.
Your 2nd and 3rd points can be done always, I think, unless our FAT code should mimic the DOS FAT behaviour (on non-DOS filesystems).

Andries

#15 lore raw
In reply to Frodo Looijaard
Quoted 6 lines · Frodo Looijaard
Quoted 2 lines

The above should "goto EODir;". Likewise, another "contiure" of fat_search_long().

I am not convinced that the goto is always safe, and I am pretty sure that it works now (though not as efficient as possible, perhaps), so I have left that in place for now.

But, if there is directory entries after 0, you must not allow access to those entries. Because those invalid entries may have reference to the valid data block.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#16 lore raw
In reply to Frodo Looijaard

Hi folks,

another update of the patch, which addresses some of the reactions and concerns which I got in reply:

* The mount option is now called epoc' instead of oldfat'
* fat_search_long now jumps directly to EODir when it finds a zero-marked entry (it took some hard looking, but I belief this is safe).
* fat_readdirx logic is slightly simplified (but can't jump to EODir on a zero-marked entry, regrettably).

Thanks for your feedback!

I'll try to keep this patch up-to-date with new 2.6 kernels. The latest version can always be found on http://debian.frodo.looijaard.name/

Thanks,
Frodo

Signature · 4 lines
Frodo Looijaard <[email protected]>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.