FAT-filesystem EOF marker problem
Hi,
On 2002-Sept-15 a Linux-USB user reported a problem when using a file written by Linux to a CompactFlash (CF) card in an MP3 player. Linux and Windows can still read the CF card correctly, but the MP3 player cannot. I have successfully reproduced this problem.
Initial problem report and thread begin at:
http://marc.theaimsgroup.com/?l=linux-usb-users&m=103210507227709&w=2
I found several Boot Record/BIOS Parameter Block differences, one FAT cluster link difference (EOF marker), and several directory entry differences. I believe that I have proved that the problem lies in the FAT EOF marker difference:
Windows (ME version at least) writes 0xfff for a FAT12 EOF marker. mtools writes 0xfff for a FAT12 EOF marker.
Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker. (Yes, I know that 0xff8 is a valid FAT12 EOF marker.)
This causes some cheap/crappy MP3 player(s) firmware to get lost... yeah, it's a problem with the MP3 player(s).
And Linux users can get around the problem by using mtools instead of a Linux filesystem, but sometimes using a filesystem is just preferred.
Anyone have any suggestions for other workarounds or patches?
I would prefer to see something like (a) Linux uses 0xfff for FAT(12) EOF [and sign-extended for FAT16 and FAT32], or
(b) Linux uses a mount option to indicate the EOF marker value, or (c) Linux dynamically checks what EOF marker value is in the FAT already, and continues to use that value [and continues to use 0xff8 as its default value].
Thanks,
~Randy
Quoted 27 lines · Randy.Dunlap
Hi,
On 2002-Sept-15 a Linux-USB user reported a problem when using a file written by Linux to a CompactFlash (CF) card in an MP3 player. Linux and Windows can still read the CF card correctly, but the MP3 player cannot. I have successfully reproduced this problem.
Initial problem report and thread begin at:
http://marc.theaimsgroup.com/?l=linux-usb-users&m=103210507227709&w=2I found several Boot Record/BIOS Parameter Block differences, one FAT cluster link difference (EOF marker), and several directory entry differences. I believe that I have proved that the problem lies in the FAT EOF marker difference:
Windows (ME version at least) writes
0xffffor a FAT12 EOF marker. mtools writes0xffffor a FAT12 EOF marker.
Linux (v)fat filesystem writes0xff8for a FAT12 EOF marker. (Yes, I know that0xff8is a valid FAT12 EOF marker.)This causes some cheap/crappy MP3 player(s) firmware to get lost... yeah, it's a problem with the MP3 player(s).
And Linux users can get around the problem by using mtools instead of a Linux filesystem, but sometimes using a filesystem is just preferred.
Quoted 8 lines
Anyone have any suggestions for other workarounds or patches?
I would prefer to see something like (a) Linux uses
0xffffor FAT(12) EOF [and sign-extended for FAT16 and FAT32], or
(b) Linux uses a mount option to indicate the EOF marker value, or (c) Linux dynamically checks what EOF marker value is in the FAT already, and continues to use that value [and continues to use0xff8as its default value].
I vote to (a). It should be very easy for 2.5.x. And I think that you explained why we should use standard EOF mark.
So, what do you think of trying (a), until the reason we need values other than standard EOF(fff/ffff/fffffff) is found?
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
On Mon, 23 Sep 2002, OGAWA Hirofumi wrote:
| "Randy.Dunlap" <[email protected]> writes: | | > Hi, | > | > On 2002-Sept-15 a Linux-USB user reported a problem when using | > a file written by Linux to a CompactFlash (CF) card in an MP3 | > player. Linux and Windows can still read the CF card correctly, | > but the MP3 player cannot. I have successfully reproduced this | > problem. | > | > Initial problem report and thread begin at: | > http://marc.theaimsgroup.com/?l=linux-usb-users&m=103210507227709&w=2 | > | > I found several Boot Record/BIOS Parameter Block differences, | > one FAT cluster link difference (EOF marker), and several | > directory entry differences. I believe that I have proved | > that the problem lies in the FAT EOF marker difference: | > | > Windows (ME version at least) writes 0xfff for a FAT12 EOF marker. | > mtools writes 0xfff for a FAT12 EOF marker. | > Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker. | > (Yes, I know that 0xff8 is a valid FAT12 EOF marker.) | > | > This causes some cheap/crappy MP3 player(s) firmware to get lost... | > yeah, it's a problem with the MP3 player(s). | > And Linux users can get around the problem by using mtools | > instead of a Linux filesystem, but sometimes using a filesystem | > is just preferred. | > | > Anyone have any suggestions for other workarounds or patches? | > | > I would prefer to see something like (a) Linux uses 0xfff for | > FAT(12) EOF [and sign-extended for FAT16 and FAT32], or | > (b) Linux uses a mount option to indicate the EOF marker value, | > or (c) Linux dynamically checks what EOF marker value is in | > the FAT already, and continues to use that value [and continues | > to use 0xff8 as its default value]. | | I vote to (a). It should be very easy for 2.5.x. And I think that you | explained why we should use standard EOF mark. | | So, what do you think of trying (a), until the reason we need values | other than standard EOF(fff/ffff/fffffff) is found?
Sure, that sounds fine to me. I expect it to be fairly safe. And if we did (c), we should use 0xfff instead of 0xff8 as the default now that I've had more time to think about it.
Signature · 1 line
~Randy
Quoted 8 lines · Randy.Dunlap
On 2002-Sept-15 a Linux-USB user reported a problem when using a file written by Linux to a CompactFlash (CF) card in an MP3 player. Linux and Windows can still read the CF card correctly, but the MP3 player cannot. I have successfully reproduced this problem.
Initial problem report and thread begin at:
http://marc.theaimsgroup.com/?l=linux-usb-users&m=103210507227709&w=2
There seem to be many more people with this problem. One web page describing it is
http://home.earthlink.net/~ebrombaugh/fattools.html
Quoted 10 lines
Windows (ME version at least) writes
0xffffor a FAT12 EOF marker. mtools writes0xffffor a FAT12 EOF marker.
Linux (v)fat filesystem writes0xff8for a FAT12 EOF marker.I would prefer to see something like (a) Linux uses
0xffffor FAT(12) EOF [and sign-extended for FAT16 and FAT32], or
(b) Linux uses a mount option to indicate the EOF marker value, or (c) Linux dynamically checks what EOF marker value is in the FAT already, and continues to use that value [and continues to use0xff8as its default value].
My docs say: FAT entry 0 contains the media descriptor byte. FAT entry 1 contains the EOF marker, except that the high order two bits can be used for other purposes.
So, I would like to reject (b) - if a device does not work, very few people will get the idea that the end-of-file marker used by the FAT filesystem is to blame. And (a) is reasonable, that is what Microsoft does. And (c) might be replaced by (c'): read the desired EOF marker from FAT entry 1. If that yields an invalid value, use what Microsoft does: 0xfff.
Andries
On Mon, 23 Sep 2002, Andries Brouwer wrote:
| On Sat, Sep 21, 2002 at 10:30:19PM -0700, Randy.Dunlap wrote: | | > On 2002-Sept-15 a Linux-USB user reported a problem when using | > a file written by Linux to a CompactFlash (CF) card in an MP3 | > player. Linux and Windows can still read the CF card correctly, | > but the MP3 player cannot. I have successfully reproduced this | > problem. | > | > Initial problem report and thread begin at: | > http://marc.theaimsgroup.com/?l=linux-usb-users&m=103210507227709&w=2 | | There seem to be many more people with this problem. | One web page describing it is | http://home.earthlink.net/~ebrombaugh/fattools.html
Hm, OK, thanks for the web search that I didn't do...
| > Windows (ME version at least) writes 0xfff for a FAT12 EOF marker. | > mtools writes 0xfff for a FAT12 EOF marker. | > Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker. | | > I would prefer to see something like (a) Linux uses 0xfff for | > FAT(12) EOF [and sign-extended for FAT16 and FAT32], or | > (b) Linux uses a mount option to indicate the EOF marker value, | > or (c) Linux dynamically checks what EOF marker value is in | > the FAT already, and continues to use that value [and continues | > to use 0xff8 as its default value]. | | My docs say: FAT entry 0 contains the media descriptor byte. | FAT entry 1 contains the EOF marker, except that the high order two | bits can be used for other purposes.
And the value that I see there is 0xfff.
| So, I would like to reject (b) - if a device does not work, very few | people will get the idea that the end-of-file marker used by the | FAT filesystem is to blame. And (a) is reasonable, that is what | Microsoft does. And (c) might be replaced by (c'): read the desired | EOF marker from FAT entry 1. If that yields an invalid value, use | what Microsoft does: 0xfff.
Thanks agagin. Most of my FAT specs are long gone and I didn't find this particular nugget on the web. I vote for option (c') in this case.
Signature · 1 line
~Randy
Quoted 24 lines · Randy.Dunlap
| > Windows (ME version at least) writes 0xfff for a FAT12 EOF marker. | > mtools writes 0xfff for a FAT12 EOF marker. | > Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker. | | > I would prefer to see something like (a) Linux uses 0xfff for | > FAT(12) EOF [and sign-extended for FAT16 and FAT32], or | > (b) Linux uses a mount option to indicate the EOF marker value, | > or (c) Linux dynamically checks what EOF marker value is in | > the FAT already, and continues to use that value [and continues | > to use 0xff8 as its default value]. | | My docs say: FAT entry 0 contains the media descriptor byte. | FAT entry 1 contains the EOF marker, except that the high order two | bits can be used for other purposes.And the value that I see there is
0xfff.| So, I would like to reject (b) - if a device does not work, very few | people will get the idea that the end-of-file marker used by the | FAT filesystem is to blame. And (a) is reasonable, that is what | Microsoft does. And (c) might be replaced by (c'): read the desired | EOF marker from FAT entry 1. If that yields an invalid value, use | what Microsoft does: 0xfff.
Quoted 3 lines
Thanks agagin. Most of my FAT specs are long gone and I didn't find this particular nugget on the web. I vote for option (c') in this case.
Umm, Microsoft use 0xfff always, right? And, since entry 1 is set by format command, I worry that this problem may happen again when format command setted value other than 0xfff.
What do you think of above case?
Signature · 1 line
OGAWA Hirofumi <[email protected]>
On Mon, 23 Sep 2002, OGAWA Hirofumi wrote:
| "Randy.Dunlap" <[email protected]> writes: | | > | > Windows (ME version at least) writes 0xfff for a FAT12 EOF marker. | > | > mtools writes 0xfff for a FAT12 EOF marker. | > | > Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker. | > | | > | > I would prefer to see something like (a) Linux uses 0xfff for | > | > FAT(12) EOF [and sign-extended for FAT16 and FAT32], or | > | > (b) Linux uses a mount option to indicate the EOF marker value, | > | > or (c) Linux dynamically checks what EOF marker value is in | > | > the FAT already, and continues to use that value [and continues | > | > to use 0xff8 as its default value]. | > | | > | My docs say: FAT entry 0 contains the media descriptor byte. | > | FAT entry 1 contains the EOF marker, except that the high order two | > | bits can be used for other purposes. | > | > And the value that I see there is 0xfff. | > | > | So, I would like to reject (b) - if a device does not work, very few | > | people will get the idea that the end-of-file marker used by the | > | FAT filesystem is to blame. And (a) is reasonable, that is what | > | Microsoft does. And (c) might be replaced by (c'): read the desired | > | EOF marker from FAT entry 1. If that yields an invalid value, use | > | what Microsoft does: 0xfff. | > | > Thanks agagin. Most of my FAT specs are long gone and I | > didn't find this particular nugget on the web. | > I vote for option (c') in this case. | | Umm, Microsoft use 0xfff always, right? And, since entry 1 is set by | format command, I worry that this problem may happen again when format | command setted value other than 0xfff. | | What do you think of above case?
Um, I guess we can say that Windows ME uses 0xfff always, at least one version of it. Don't know about Windows 98 or 98 SE or 2000 or XP etc. But since FAT entry 1 tells the FAT filesystem what the EOF value is, it shouldn't matter if it changes from one filesystem to another, should it? I don't see a problem with using the value in fat_entry[1]. Just make sure that it's valid (0xff8 - 0xfff for FAT12).
Oh, I see. I'm assuming that this can be done on a per-filesystem basis instead of one global EOF value for all mounted fat filesystems. Does that make more sense? Is it feasible?
I don't see why it couldn't be done this way.
Signature · 1 line
~Randy
Followup to: <[email protected]> By author: "Randy.Dunlap" <[email protected]> In newsgroup: linux.dev.fs.devel
Quoted 9 lines
Um, I guess we can say that Windows ME uses
0xfffalways, at least one version of it. Don't know about Windows 98 or 98 SE or 2000 or XP etc. But since FAT entry 1 tells the FAT filesystem what the EOF value is, it shouldn't matter if it changes from one filesystem to another, should it? I don't see a problem with using the value in fat_entry[1]. Just make sure that it's valid (0xff8-0xffffor FAT12).
ALL versions of DOS since 1.0 uses all ones for the EOC mark.
Under the principle "be liberal in what you receive, conservative in what you send" we should recognize anything >= -8 as EOC, but write -1.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
On 24 Sep 2002, H. Peter Anvin wrote:
| Followup to: <[email protected]> | By author: "Randy.Dunlap" <[email protected]> | In newsgroup: linux.dev.fs.devel | > | > Um, I guess we can say that Windows ME uses 0xfff always, at least | > one version of it. Don't know about Windows 98 or 98 SE or 2000 | > or XP etc. But since FAT entry 1 tells the FAT filesystem what | > the EOF value is, it shouldn't matter if it changes from one | > filesystem to another, should it? I don't see a problem with | > using the value in fat_entry[1]. Just make sure that it's valid | > (0xff8 - 0xfff for FAT12). | > | | ALL versions of DOS since 1.0 uses all ones for the EOC mark. | | Under the principle "be liberal in what you receive, conservative in | what you send" we should recognize anything >= -8 as EOC, but write | -1.
My view exactly.
Signature · 1 line
~Randy
Followup to: <[email protected]> By author: "Randy.Dunlap" <[email protected]> In newsgroup: linux.dev.fs.devel
Quoted 7 lines
| | My docs say: FAT entry 0 contains the media descriptor byte. | FAT entry 1 contains the EOF marker, except that the high order two | bits can be used for other purposes.And the value that I see there is
0xfff.
Fat entry 0 contains the media descriptor byte, and fat entry 1 *used* to contain the EOC mark, but has been retconned as the volume dirty bit (Microsoft OSs don't recognize a volume dirty flag for FAT12, however; this is presumably because of the MS assumption that FAT12 == floppy.):
For FAT16: ClnShutBitMask = 0x8000; HrdErrBitMask = 0x4000; For FAT32: ClnShutBitMask = 0x08000000; HrdErrBitMask = 0x04000000;
Bit ClnShutBitMask If bit is 1, volume is clean . If bit is 0,
volume is dirty . This indicates that the file system driver did not Dismount the volume properly the last time it had the volume mounted. It would be a good idea to run a Chkdsk/Scandisk disk repair utility on it, because it may be damaged.
Bit HrdErrBitMask If this bit is 1, no disk read/write errors were
encountered. If this bit is 0, the file system driver encountered a disk I/O error on the Volume the last time it was mounted, which is an indicator that some sectors may have gone bad on the volume. It would be a good idea to run a Chkdsk/Scandisk disk repair utility that does surface analysis on it to look for new bad sectors.
Either way, I vote we should always use all ones of the EOC mark.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
Quoted 8 lines · H. Peter Anvin
Quoted 5 lines
| My docs say: FAT entry 0 contains the media descriptor byte. | FAT entry 1 contains the EOF marker, except that the high order two | bits can be used for other purposes.And the value that I see there is
0xfff.Fat entry 0 contains the media descriptor byte, and fat entry 1 *used* to contain the EOC mark, but has been retconned as the volume dirty bit
Yes, I know.
(As I see it, it still holds the EOC mark, but only the low order bits are of interest, so the top two bits have a different meaning now.)
Let me mention
http://www.win.tue.nl/~aeb/linux/fs/fat/fat.html
(my notes on the FAT filesystem).
If anyone has corrections or additions I'd like to hear.
Andries
P.S. I checked some old stuff but did not find anything with a EOC other than 0x*fff. I would be interested in hearing about which non-Linux programs used other values.
Followup to: <[email protected]> By author: "Randy.Dunlap" <[email protected]> In newsgroup: linux.dev.fs.devel
Quoted 6 lines
Windows (ME version at least) writes
0xffffor a FAT12 EOF marker. mtools writes0xffffor a FAT12 EOF marker.
Linux (v)fat filesystem writes0xff8for a FAT12 EOF marker. (Yes, I know that0xff8is a valid FAT12 EOF marker.)
Valid, perhaps (the official specification is fuzzy on that subject to say the least), but 0xfff is the standard one and what DOS uses.
The official spec states:
Note that the cluster number whose cluster entry in the FAT contains the EOC mark is allocated to the file and is also the last cluster allocated to the file. Microsoft operating system FAT drivers use the EOC value 0x0FFF for FAT12, 0xFFFF for FAT16, and 0x0FFFFFFF for FAT32 when they set the contents of a cluster to the EOC mark. There are various disk utilities for Microsoft operating systems that use a different value, however.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
By the way, I have posted a copy of the offical FAT spec PDF at:
http://www.zytor.com/~hpa/FatFormat.pdf
... since I can't find the original URL anymore.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
Quoted 5 lines · H. Peter Anvin
By the way, I have posted a copy of the offical FAT spec PDF at:
http://www.zytor.com/~hpa/FatFormat.pdf
... since I can't find the original URL anymore.
ERROR 403: Forbidden.
But maybe it is identical to
http://www.win.tue.nl/~aeb/linux/fs/fat/fatgen102.pdf
?
Quoted 10 lines · Andries Brouwer
Quoted 5 lines · H. Peter Anvin
By the way, I have posted a copy of the offical FAT spec PDF at:
http://www.zytor.com/~hpa/FatFormat.pdf
... since I can't find the original URL anymore.
ERROR 403: Forbidden.
Fixed.
Quoted 4 lines
But maybe it is identical to
http://www.win.tue.nl/~aeb/linux/fs/fat/fatgen102.pdf
Indeed it is.
-hpa