Wrong free clusters count on FAT32
Hi all :)
I have a portable device with a FAT32 formatted hard disk in it, and
everytime I delete a file in the device *using the device itself to do it* the device increases its count of free space and if I plug the device in a Windows system, Windows agrees on the free space. Linux doesn't. Linux believes that the files are still there ocuppying space, and I have to run fsck.vfat to fix the problem.
As far as I've seen, the device is probably not updating correctly
the list of free clusters, but it doesn't seem to worry about it, neither does Windows. So my question is: is there any way of making Linux "bug compatible" with Windows? If Windows itself don't worry about the free cluster count and computes the free space in some other way, then it can be done. I haven't seen anything in mount(8) neither googling.
Apart from not using the device itself to delete files (and probably
not using Windows for that, either) and to run fsck.vfat now and then, is anything I can do to avoid this problem?
Thanks a lot in advance :)
Raúl Núñez de Arenas Coronado
Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net It's my PC and I'll cry if I want to... RAmen!
Quoted 17 lines · DervishD
Hi all :)
I have a portable device with a FAT32 formatted hard disk in it, and
everytime I delete a file in the device *using the device itself to do it* the device increases its count of free space and if I plug the device in a Windows system, Windows agrees on the free space. Linux doesn't. Linux believes that the files are still there ocuppying space, and I have to run fsck.vfat to fix the problem.As far as I've seen, the device is probably not updating correctly
the list of free clusters, but it doesn't seem to worry about it, neither does Windows. So my question is: is there any way of making Linux "bug compatible" with Windows? If Windows itself don't worry about the free cluster count and computes the free space in some other way, then it can be done. I haven't seen anything in mount(8) neither googling.
Quoted 8 lines
Apart from not using the device itself to delete files (and probably
not using Windows for that, either) and to run fsck.vfat now and then, is anything I can do to avoid this problem?Thanks a lot in advance :)
Raúl Núñez de Arenas Coronado
Not that I know how to fix it. But have you tried running chkdsk on a device deleted files in windows. Just that I know, that windows would silently ignore fat errors and will only report them in chkdsk.
Deleting files in windows works fine. There is no problems with it and windows updates everything. I do that all the time.
(I would even theorize that if you delete files on the device and than farther delete more files on windows, than windows will fix the problem on the fly) So it is "bug compatible" with the device, and error recovery compatible with windows.
Boaz
Hi Boaz :)
* Boaz Harrosh <[email protected]> dixit:
Quoted 7 lines
Quoted 4 lines
Apart from not using the device itself to delete files (and probably not using Windows for that, either) and to run fsck.vfat now and then, is anything I can do to avoid this problem?
Not that I know how to fix it. But have you tried running chkdsk on a device deleted files in windows. Just that I know, that windows would silently ignore fat errors and will only report them in chkdsk.
I've done that and the problem is not fixed. I have to use fsck.vfat
instead.
Quoted 2 lines
Deleting files in windows works fine. There is no problems with it and windows updates everything. I do that all the time.
True, I've tested it, too. I'm going to report the problem to the
device manufacturer (it's a Cowon iAUDIO), but probably doing it in the device will be very slow :(
Quoted 3 lines
(I would even theorize that if you delete files on the device and than farther delete more files on windows, than windows will fix the problem on the fly)
Probably, and it may be a good fix, but I seldom plug the device on
Windows.
Thanks for the answer :)
Raúl Núñez de Arenas Coronado
Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net It's my PC and I'll cry if I want to... RAmen!
Quoted 6 lines · DervishD
I have a portable device with a FAT32 formatted hard disk in it, and
everytime I delete a file in the device *using the device itself to do it* the device increases its count of free space and if I plug the device in a Windows system, Windows agrees on the free space. Linux doesn't. Linux believes that the files are still there ocuppying space, and I have to run fsck.vfat to fix the problem.
As I remember: It needs a large amount of time to calculate the free space on a big FAT32 system. So the last free sector count is also stored. When mounting this filesystem you don't need to walk through the whole FAT to calculate the available space, you can use this "cached" value instead. And this cached value seems not to be updated in your portable device.
Maybe it helps, but maybe I'm wrong.
Juergen
Hi Juergen :)
* Juergen Beisert <[email protected]> dixit:
Quoted 10 lines · DervishD
Quoted 6 lines
I have a portable device with a FAT32 formatted hard disk in it, and everytime I delete a file in the device *using the device itself to do it* the device increases its count of free space and if I plug the device in a Windows system, Windows agrees on the free space. Linux doesn't. Linux believes that the files are still there ocuppying space, and I have to run fsck.vfat to fix the problem.
As I remember: It needs a large amount of time to calculate the free space on a big FAT32 system.
Big fat truth, I'm afraid. The thing is that I thought that Linux
did that from time to time to update the count. Obviously, doing it for every statfs call would be very expensive :((
Quoted 4 lines
So the last free sector count is also stored. When mounting this filesystem you don't need to walk through the whole FAT to calculate the available space, you can use this "cached" value instead. And this cached value seems not to be updated in your portable device.
It doesn't, certainly, but Windows doesn't care. Moreover, the
device doesn't seem to recalculate the value on every run (unless it does it lightning fast!), so maybe the number is stored elsewhere (the count can be stored in many places as far as I've read, but I don't know the details).
A mount option to force walking the FAT and getting the real info
could be interesting. That way, it will be only done for certain devices (small disks, for example).
Thanks for your answer :)
Raúl Núñez de Arenas Coronado
Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net It's my PC and I'll cry if I want to... RAmen!
OGAWA Hirofumi
maintainerQuoted 27 lines · DervishD
* Juergen Beisert <[email protected]> dixit:
Quoted 10 lines · DervishD
Quoted 6 lines
I have a portable device with a FAT32 formatted hard disk in it, and everytime I delete a file in the device *using the device itself to do it* the device increases its count of free space and if I plug the device in a Windows system, Windows agrees on the free space. Linux doesn't. Linux believes that the files are still there ocuppying space, and I have to run fsck.vfat to fix the problem.
As I remember: It needs a large amount of time to calculate the free space on a big FAT32 system.
Big fat truth, I'm afraid. The thing is that I thought that Linux
did that from time to time to update the count. Obviously, doing it for every statfs call would be very expensive :((Quoted 4 lines
So the last free sector count is also stored. When mounting this filesystem you don't need to walk through the whole FAT to calculate the available space, you can use this "cached" value instead. And this cached value seems not to be updated in your portable device.
It doesn't, certainly, but Windows doesn't care. Moreover, the
device doesn't seem to recalculate the value on every run (unless it does it lightning fast!), so maybe the number is stored elsewhere (the count can be stored in many places as far as I've read, but I don't know the details).
Quoted 3 lines
A mount option to force walking the FAT and getting the real info
could be interesting. That way, it will be only done for certain devices (small disks, for example).
Yes. It seems that Windows does not update the ->free_clusters correctly. Probably, I think the option is good for now. What do you think about an attached patch?
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Recent Windows doesn't update ->free_clusters correctly. The "nofree" option ignores the ->free_clusters stored on FSINFO.
---
fs/fat/inode.c | 13 ++++++++++--- include/linux/msdos_fs.h | 3 ++- 2 files changed, 12 insertions(+), 4 deletions(-)
fs/fat/inode.c renamed from fs/fat/inode.c~fat_ignore_free_clusters +10 -3
@@ -825,6 +825,8 @@ static int fat_show_options(struct seq_f }if (opts->name_check !='n' )seq_printf (m,",check=%c" , opts->name_check); +if (opts->nofree) +seq_puts (m,",nofree" );if (opts->quiet)seq_puts (m,",quiet" );if (opts->showexec)
@@ -850,7 +852,7 @@ static int fat_show_options(struct seq_fenum { Opt_check_n, Opt_check_r, Opt_check_s, Opt_uid, Opt_gid, - Opt_umask, Opt_dmask, Opt_fmask, Opt_codepage, Opt_nocase, + Opt_umask, Opt_dmask, Opt_fmask, Opt_codepage, Opt_nofree, Opt_nocase, Opt_quiet, Opt_showexec, Opt_debug, Opt_immutable, Opt_dots, Opt_nodots, Opt_charset, Opt_shortname_lower, Opt_shortname_win95,
@@ -872,6 +874,7 @@ static match_table_t fat_tokens = { {Opt_dmask, "dmask=%o"}, {Opt_fmask, "fmask=%o"}, {Opt_codepage, "codepage=%u"}, + {Opt_nofree, "nofree"}, {Opt_nocase, "nocase"}, {Opt_quiet, "quiet"}, {Opt_showexec, "showexec"},
@@ -951,7 +954,7 @@ static int parse_options(char *options, opts->quiet = opts->showexec = opts->sys_immutable = opts->dotsOK =0 ; opts->utf8 = opts->unicode_xlate =0 ; opts->numtail =1 ; - opts->nocase =0 ; + opts->nofree = opts->nocase =0 ; *debug =0 ;if (!options)
@@ -979,6 +982,9 @@ static int parse_options(char *options,case Opt_check_n: opts->name_check ='n' ;break ; +case Opt_nofree: + opts->nofree =1 ; +break ;case Opt_nocase:if (!is_vfat) opts->nocase =1 ;
@@ -1352,7 +1358,8 @@ int fat_fill_super(struct super_block *s sbi->max_cluster = total_clusters +FAT_START_ENT ;/* check the free_clusters, it's not necessarily correct */ -if (sbi->free_clusters !=-1 && sbi->free_clusters > total_clusters) +if ((sbi->free_clusters !=-1 && sbi->free_clusters > total_clusters) || + sbi->options.nofree) sbi->free_clusters =-1 ;/* check the prev_free, it's not necessarily correct */ sbi->prev_free %= sbi->max_cluster;
include/linux/msdos_fs.h renamed from include/linux/msdos_fs.h~fat_ignore_free_clusters +2 -1
@@ -205,7 +205,8 @@ struct fat_mount_options { numtail:1, /* Does first alias have a numeric '~1' type tail? */ atari:1, /* Use Atari GEMDOS variation of MS-DOS fs */ flush:1, /* write things quickly */ - nocase:1; /* Does this need case conversion? 0=need case conversion*/ + nocase:1, /* Does this need case conversion? 0=need case conversion*/ + nofree:1; /* Does use free_clusters */ };#define FAT_HASH_BITS 8
_
Quoted 2 lines · On Sun, 22 Apr 2007 07:42:22 +0900 OGAWA Hirofumi…
Recent Windows doesn't update ->free_clusters correctly. The "nofree" option ignores the ->free_clusters stored on FSINFO.
It'd be better to avoid the new mount option if possible: fatfs is used by a *lot* of people who aren't particualrly expert, and we want to make things easy for them.
Is there some way in which we can work out what's happened and fix it up?
OGAWA Hirofumi
maintainerQuoted 8 lines · Andrew Morton
Quoted 2 lines · On Sun, 22 Apr 2007 07:42:22 +0900 OGAWA Hirofumi…
Recent Windows doesn't update ->free_clusters correctly. The "nofree" option ignores the ->free_clusters stored on FSINFO.
It'd be better to avoid the new mount option if possible: fatfs is used by a *lot* of people who aren't particualrly expert, and we want to make things easy for them.
Ok.
Quoted 1 line
Is there some way in which we can work out what's happened and fix it up?
It seems that the recent Windows changed specification, and it's undocumented. Windows doesn't update ->free_clusters correctly.
Probably, what we can do is to throw away speed of statfs(2) and not using ->free_clusters. (And if possible, recover it by optimization.)
This patch doesn't use ->free_clusters by default. (instead, add "usefree" for forcing to use it)
Signature · 1 line
OGAWA Hirofumi <[email protected]>
It seems that the recent Windows changed specification, and it's undocumented. Windows doesn't update ->free_clusters correctly.
This patch doesn't use ->free_clusters by default. (instead, add "usefree" for forcing to use it)
---
fs/fat/inode.c | 14 +++++++++++--- include/linux/msdos_fs.h | 3 ++- 2 files changed, 13 insertions(+), 4 deletions(-)
fs/fat/inode.c renamed from fs/fat/inode.c~fat_dont-use_free_clusters-for-fat32 +11 -3
@@ -825,6 +825,8 @@ static int fat_show_options(struct seq_f }if (opts->name_check !='n' )seq_printf (m,",check=%c" , opts->name_check); +if (opts->usefree) +seq_puts (m,",usefree" );if (opts->quiet)seq_puts (m,",quiet" );if (opts->showexec)
@@ -850,7 +852,7 @@ static int fat_show_options(struct seq_fenum { Opt_check_n, Opt_check_r, Opt_check_s, Opt_uid, Opt_gid, - Opt_umask, Opt_dmask, Opt_fmask, Opt_codepage, Opt_nocase, + Opt_umask, Opt_dmask, Opt_fmask, Opt_codepage, Opt_usefree, Opt_nocase, Opt_quiet, Opt_showexec, Opt_debug, Opt_immutable, Opt_dots, Opt_nodots, Opt_charset, Opt_shortname_lower, Opt_shortname_win95,
@@ -872,6 +874,7 @@ static match_table_t fat_tokens = { {Opt_dmask, "dmask=%o"}, {Opt_fmask, "fmask=%o"}, {Opt_codepage, "codepage=%u"}, + {Opt_usefree, "usefree"}, {Opt_nocase, "nocase"}, {Opt_quiet, "quiet"}, {Opt_showexec, "showexec"},
@@ -951,7 +954,7 @@ static int parse_options(char *options, opts->quiet = opts->showexec = opts->sys_immutable = opts->dotsOK =0 ; opts->utf8 = opts->unicode_xlate =0 ; opts->numtail =1 ; - opts->nocase =0 ; + opts->usefree = opts->nocase =0 ; *debug =0 ;if (!options)
@@ -979,6 +982,9 @@ static int parse_options(char *options,case Opt_check_n: opts->name_check ='n' ;break ; +case Opt_usefree: + opts->usefree =1 ; +break ;case Opt_nocase:if (!is_vfat) opts->nocase =1 ;
@@ -1306,7 +1312,9 @@ int fat_fill_super(struct super_block *sle32_to_cpu (fsinfo->signature2), sbi->fsinfo_sector); }else { - sbi->free_clusters =le32_to_cpu (fsinfo->free_clusters); +if (sbi->options.usefree) + sbi->free_clusters = +le32_to_cpu (fsinfo->free_clusters); sbi->prev_free =le32_to_cpu (fsinfo->next_cluster); }
include/linux/msdos_fs.h renamed from include/linux/msdos_fs.h~fat_dont-use_free_clusters-for-fat32 +2 -1
@@ -205,7 +205,8 @@ struct fat_mount_options { numtail:1, /* Does first alias have a numeric '~1' type tail? */ atari:1, /* Use Atari GEMDOS variation of MS-DOS fs */ flush:1, /* write things quickly */ - nocase:1; /* Does this need case conversion? 0=need case conversion*/ + nocase:1, /* Does this need case conversion? 0=need case conversion*/ + usefree:1; /* Use free_clusters for FAT32 */ };#define FAT_HASH_BITS 8
_
Hi Ogawa (and Andrew) :)
* OGAWA Hirofumi <[email protected]> dixit:
Quoted 9 lines
Quoted 2 lines · Andrew Morton
Is there some way in which we can work out what's happened and fix it up?
It seems that the recent Windows changed specification, and it's undocumented. Windows doesn't update ->free_clusters correctly.
Probably, what we can do is to throw away speed of statfs(2) and not using ->free_clusters. (And if possible, recover it by optimization.)
Even if Windows updates ->free_clusters correctly, it doesn't even
consider it to compute the number of free clusters, and IMHO Linux should do the same. If speed is a concern, ->free_clusters can be computed at mount time. That way statfs won't slow.
The problem is that if a program writes a file onto the filesystem
without using statfs first to check for free space, the free_clusters entry won't have the real value and the driver may report "disk full" (I haven't read the code for the vfat driver, sorry, so I'm not sure about this) when really there are plenty of clusters to write the new file.
Probably it's stupid to update the free clusters count at mount time
(sorry if so...) but it looks like a good idea to me. And of course, I don't mean to update the value _on disk_, but the kernel's idea of free clusters (so even FAT filesystems mounted R/O will report correct values).
Raúl Núñez de Arenas Coronado
Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net It's my PC and I'll cry if I want to... RAmen!
OGAWA Hirofumi
maintainerQuoted 5 lines · DervishD
The problem is that if a program writes a file onto the filesystem
without using statfs first to check for free space, thefree_clustersentry won't have the real value and the driver may report "disk full" (I haven't read the code for the vfat driver, sorry, so I'm not sure about this) when really there are plenty of clusters to write the new file.
No need to worry about it. If we ignored the ->free_clusters in FSINFO, the fat drivers counts the current free clusters by scaning FAT entries if needed.
Quoted 5 lines
Probably it's stupid to update the free clusters count at mount time
(sorry if so...) but it looks like a good idea to me. And of course, I don't mean to update the value_ondisk_, but the kernel's idea of free clusters (so even FAT filesystems mounted R/O will report correct values).
It would add the limitation to following simple usage,
# mount -t vfat /dev/sda1 /mnt
# cp -a * /mnt
# umountif /dev/sda1 was the large and slow device, "mount" will need several minutes to counts free clusters. I think the user will be hard to accept the several minutes at "mount".
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Hi Ogawa :)
* OGAWA Hirofumi <[email protected]> dixit:
Quoted 10 lines
Quoted 5 lines · DervishD
The problem is that if a program writes a file onto the filesystem without using statfs first to check for free space, the
free_clustersentry won't have the real value and the driver may report "disk full" (I haven't read the code for the vfat driver, sorry, so I'm not sure about this) when really there are plenty of clusters to write the new file.No need to worry about it. If we ignored the ->free_clusters in FSINFO, the fat drivers counts the current free clusters by scaning FAT entries if needed.
Cool! :)
Quoted 8 lines
Quoted 5 lines
Probably it's stupid to update the free clusters count at mount time (sorry if so...) but it looks like a good idea to me. And of course, I don't mean to update the value
_ondisk_, but the kernel's idea of free clusters (so even FAT filesystems mounted R/O will report correct values).It would add the limitation to following simple usage,
Quoted 7 lines
# mount -t vfat /dev/sda1 /mnt # cp -a * /mnt # umountif
/dev/sda1was the large and slow device, "mount" will need several minutes to counts free clusters. I think the user will be hard to accept the several minutes at "mount".
I can carry some tests, but if Windows does that tasks lightning
fast, Linux surely does it faster ;) I don't think, anyway, that having a huge USB disk is a common practice when using "modest" machines.
If you want, I can perform a couple of tests. I have a 80GB disk
that I can connect using an USB adapter and my machine is AMD Athlon XP 1900+ with 1GB of RAM, which looks pretty slow nowadays O:)
Raúl Núñez de Arenas Coronado
Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net It's my PC and I'll cry if I want to... RAmen!
OGAWA Hirofumi
maintainerQuoted 10 lines · DervishD
Quoted 9 lines
It would add the limitation to following simple usage,
# mount -t vfat /dev/sda1 /mnt # cp -a * /mnt # umountif
/dev/sda1was the large and slow device, "mount" will need several minutes to counts free clusters. I think the user will be hard to accept the several minutes at "mount".
Quoted 7 lines
I can carry some tests, but if Windows does that tasks lightning
fast, Linux surely does it faster ;) I don't think, anyway, that having a huge USB disk is a common practice when using "modest" machines.If you want, I can perform a couple of tests. I have a 80GB disk
that I can connect using an USB adapter and my machine is AMD Athlon XP 1900+ with 1GB of RAM, which looks pretty slow nowadays O:)
Yes, I think it's not common practice too. But I don't see why do you want to scanning at the mount.
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Hi Ogawa :)
* OGAWA Hirofumi <[email protected]> dixit:
Quoted 15 lines
Quoted 14 lines · DervishD
Quoted 9 lines
It would add the limitation to following simple usage,
# mount -t vfat /dev/sda1 /mnt # cp -a * /mnt # umountif
/dev/sda1was the large and slow device, "mount" will need several minutes to counts free clusters. I think the user will be hard to accept the several minutes at "mount".I can carry some tests, but if Windows does that tasks lightning
fast, Linux surely does it faster ;) I don't think, anyway, that having a huge USB disk is a common practice when using "modest" machines.
Quoted 6 lines
Quoted 3 lines
If you want, I can perform a couple of tests. I have a 80GB disk
that I can connect using an USB adapter and my machine is AMD Athlon XP 1900+ with 1GB of RAM, which looks pretty slow nowadays O:)Yes, I think it's not common practice too. But I don't see why do you want to scanning at the mount.
Just because I was thinking that, otherwise, the scanning would need
to be done at each statfs call, but I was wrong: once the scanning is done and the count is right, it is written at free_clusters and used afterwards.
I thought the mount was the best point in time for doing this.
Raúl Núñez de Arenas Coronado
Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net It's my PC and I'll cry if I want to... RAmen!
Hi Ogawa :)
* OGAWA Hirofumi <[email protected]> dixit:
Quoted 8 lines
Quoted 3 lines · DervishD
A mount option to force walking the FAT and getting the real info could be interesting. That way, it will be only done for certain devices (small disks, for example).
Yes. It seems that Windows does not update the ->free_clusters correctly. Probably, I think the option is good for now. What do you think about an attached patch?
The option ignores ->free_clusters by default, am I wrong? I'm not
sure about this: given that nobody cared before, chances are that this problem didn't bite anyone, so I think that the default behaviour should be the _old_ behaviour (that is, nofree=0, as if the option never existed).
Moreover, I'm not sure if Windows updates correctly or not thefree_clustes FSINFO entry, this problem has happened (by now) in a portable device only. What I can ensure is that both the portable device and any Windows box I've tested ignores this value and properly calculate the real free space. Linux doesn't because it relies on an invalid value. Again, the fault is in the portable device, but the workaround seems common practice out there...
So I will suggest just "nofree:0" instead of "nofree:1". And thanks
a lot for the patch :))) I'm not sure about this: given that nobody cared before, chances are that this problem didn't bite anyone, so I think that the default behaviour should be the _old_ behaviour (that is, nofree=0, as if the option never existed).
Moreover, I'm not sure if Windows updates correctly or not thefree_clustes FSINFO entry, this problem has happened (by now) in a portable device only. What I can ensure is that both the portable device and any Windows box I've tested ignores this value and properly calculate the real free space. Linux doesn't because it relies on an invalid value. Again, the fault is in the portable device, but the workaround seems common practice out there...
So I will suggest just "nofree:0" instead of "nofree:1". And thanks
a lot for the patch :)))
Raúl Núñez de Arenas Coronado
Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net It's my PC and I'll cry if I want to... RAmen!
Hi,
Quoted 3 lines · OGAWA Hirofumi
Yes. It seems that Windows does not update the ->free_clusters correctly. Probably, I think the option is good for now. What do you think about an attached patch?
Please don't ask me, because I'm not a filesystem expert. Sorry. My suggestion was only a tiny idea.
Juergen