Skip to contentSkip to conversation

Wrong free clusters count on FAT32

DISCUSSION Bodo Eggert 4 participants 10 messages Apr 22, 2007 – Apr 23, 2007 8 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 7 lines · OGAWA Hirofumi
Quoted 7 lines

* Juergen Beisert <[email protected]> dixit:

Quoted 4 lines

So the last free sector count is also stored. When mounting this filesystem you don't need to walk through the whole FAT to calculate the available space, you can use this "cached" value instead. And this cached value seems not to be updated in your portable device.

Quoted 5 lines
Quoted 5 lines

It doesn't, certainly, but Windows doesn't care. Moreover, the
device doesn't seem to recalculate the value on every run (unless it does it lightning fast!), so maybe the number is stored elsewhere (the count can be stored in many places as far as I've read, but I don't know the details).

AFAIR it's stored twice on FAT32, once in a backup sector and once in the superblock or extended superblock (don't remember, I think it was the extended ~). It's not stored on FAT{12,16}.

Quoted 7 lines
Quoted 3 lines

A mount option to force walking the FAT and getting the real info
could be interesting. That way, it will be only done for certain devices (small disks, for example).

Yes. It seems that Windows does not update the ->free_clusters correctly. Probably, I think the option is good for now. What do you think about an attached patch?

Windows _does_ care*, it will pretend the disk to be full. Therefore the stored value *SHOULD* be updated. (I think your patch does this.)

Recalculating the free space is a nice idea, and modern hardware might be fast enough to recalculate the value on mount by default. (I didn't try this for years.) Maybe the default should depend on arch?

About this patch: (news:[email protected])

- usefree is a bad name (I'd suggest recalc_free instead), and your description is too cryptic to be understood by a non-linux FAT expert.
- You forgot to update Documentation/

Signature · 4 lines
Never forget: 2 + 2 = 5 for extremely large values of 2. 

Friß, Spammer: [email protected] [email protected]
 [email protected] [email protected]
#5 lore raw
Quoted 1 line · Bodo Eggert

Windows _does_ care*, it will pretend the disk to be full.

Did you test on 2000 or XP? (e.g. write 0 to free_clusters, then create new file.)

Quoted 1 line

- usefree is a bad name (I'd suggest recalc_free instead),

Is it about nofree option?

Quoted 2 lines

and your description is too cryptic to be understood by a non-linux FAT expert.

Um... why do we need to care about non-linux people in the patch? Anyway, if you correct it, I'll update it gladly.

Quoted 1 line

- You forgot to update Documentation/

Sure. If you can post the patch, it'll be great.

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#6 lore raw
Quoted 3 lines · OGAWA Hirofumi
Quoted 1 line

- You forgot to update Documentation/

Sure. If you can post the patch, it'll be great.

Updated patch is here.

Thanks.

Signature · 108 lines
OGAWA Hirofumi <[email protected]>



It seems that the recent Windows changed specification, and it's
undocumented. Windows doesn't update ->free_clusters correctly.

This patch doesn't use ->free_clusters by default. (instead, add
"usefree" for forcing to use it)

Signed-off-by: OGAWA Hirofumi <[email protected]>
---

 Documentation/filesystems/vfat.txt |    7 +++++++
 fs/fat/inode.c                     |   14 +++++++++++---
 include/linux/msdos_fs.h           |    3 ++-
 3 files changed, 20 insertions(+), 4 deletions(-)

diff -puN fs/fat/inode.c~fat_dont-use_free_clusters-for-fat32 fs/fat/inode.c
--- linux-2.6/fs/fat/inode.c~fat_dont-use_free_clusters-for-fat32	2007-04-22 13:07:05.000000000 +0900
+++ linux-2.6-hirofumi/fs/fat/inode.c	2007-04-22 13:18:07.000000000 +0900
@@ -825,6 +825,8 @@ static int fat_show_options(struct seq_f
 	}
 	if (opts->name_check != 'n')
 		seq_printf(m, ",check=%c", opts->name_check);
+	if (opts->usefree)
+		seq_puts(m, ",usefree");
 	if (opts->quiet)
 		seq_puts(m, ",quiet");
 	if (opts->showexec)
@@ -850,7 +852,7 @@ static int fat_show_options(struct seq_f
 
 enum {
 	Opt_check_n, Opt_check_r, Opt_check_s, Opt_uid, Opt_gid,
-	Opt_umask, Opt_dmask, Opt_fmask, Opt_codepage, Opt_nocase,
+	Opt_umask, Opt_dmask, Opt_fmask, Opt_codepage, Opt_usefree, Opt_nocase,
 	Opt_quiet, Opt_showexec, Opt_debug, Opt_immutable,
 	Opt_dots, Opt_nodots,
 	Opt_charset, Opt_shortname_lower, Opt_shortname_win95,
@@ -872,6 +874,7 @@ static match_table_t fat_tokens = {
 	{Opt_dmask, "dmask=%o"},
 	{Opt_fmask, "fmask=%o"},
 	{Opt_codepage, "codepage=%u"},
+	{Opt_usefree, "usefree"},
 	{Opt_nocase, "nocase"},
 	{Opt_quiet, "quiet"},
 	{Opt_showexec, "showexec"},
@@ -951,7 +954,7 @@ static int parse_options(char *options, 
 	opts->quiet = opts->showexec = opts->sys_immutable = opts->dotsOK =  0;
 	opts->utf8 = opts->unicode_xlate = 0;
 	opts->numtail = 1;
-	opts->nocase = 0;
+	opts->usefree = opts->nocase = 0;
 	*debug = 0;
 
 	if (!options)
@@ -979,6 +982,9 @@ static int parse_options(char *options, 
 		case Opt_check_n:
 			opts->name_check = 'n';
 			break;
+		case Opt_usefree:
+			opts->usefree = 1;
+			break;
 		case Opt_nocase:
 			if (!is_vfat)
 				opts->nocase = 1;
@@ -1306,7 +1312,9 @@ int fat_fill_super(struct super_block *s
 			       le32_to_cpu(fsinfo->signature2),
 			       sbi->fsinfo_sector);
 		} else {
-			sbi->free_clusters = le32_to_cpu(fsinfo->free_clusters);
+			if (sbi->options.usefree)
+				sbi->free_clusters =
+					le32_to_cpu(fsinfo->free_clusters);
 			sbi->prev_free = le32_to_cpu(fsinfo->next_cluster);
 		}
 
diff -puN include/linux/msdos_fs.h~fat_dont-use_free_clusters-for-fat32 include/linux/msdos_fs.h
--- linux-2.6/include/linux/msdos_fs.h~fat_dont-use_free_clusters-for-fat32	2007-04-22 13:07:05.000000000 +0900
+++ linux-2.6-hirofumi/include/linux/msdos_fs.h	2007-04-22 13:09:14.000000000 +0900
@@ -205,7 +205,8 @@ struct fat_mount_options {
 		 numtail:1,       /* Does first alias have a numeric '~1' type tail? */
 		 atari:1,         /* Use Atari GEMDOS variation of MS-DOS fs */
 		 flush:1,	  /* write things quickly */
-		 nocase:1;	  /* Does this need case conversion? 0=need case conversion*/
+		 nocase:1,	  /* Does this need case conversion? 0=need case conversion*/
+		 usefree:1;	  /* Use free_clusters for FAT32 */
 };
 
 #define FAT_HASH_BITS	8
diff -puN Documentation/filesystems/vfat.txt~fat_dont-use_free_clusters-for-fat32 Documentation/filesystems/vfat.txt
--- linux-2.6/Documentation/filesystems/vfat.txt~fat_dont-use_free_clusters-for-fat32	2007-04-22 23:06:21.000000000 +0900
+++ linux-2.6-hirofumi/Documentation/filesystems/vfat.txt	2007-04-22 23:28:19.000000000 +0900
@@ -34,6 +34,13 @@ iocharset=name -- Character set to use f
 		 NOTE: "iocharset=utf8" is not recommended. If unsure,
 		 you should consider the following option instead.
 
+usefree       -- Use the "free clusters" value stored on FSINFO. It'll
+                 be used to determine number of free clusters without
+                 scanning disk. But it's not used by default, because
+                 recent Windows don't update it correctly in some
+                 case. If you are sure the "free clusters" on FSINFO is
+                 correct, by this option you can avoid scanning disk.
+
 utf8=<bool>   -- UTF-8 is the filesystem safe version of Unicode that
 		 is used by the console.  It can be enabled for the
 		 filesystem with this option. If 'uni_xlate' gets set,
_
#7 lore raw

Documentation/filesystems/vfat.txt iocharset=name -- Character set to use f · line 35

Quoted 6 lines · OGAWA Hirofumi

diff -puN Documentation/filesystems/vfat.txt~fat_dont-use_free_clusters-for-fat32 Documentation/filesystems/vfat.txt

Documentation/filesystems/vfat.txt renamed from Documentation/filesystems/vfat.txt~fat_dont-use_free_clusters-for-fat32 +0 -0
@@ -34,6 +34,13 @@ iocharset=name -- Character set to use f
 		 NOTE: "iocharset=utf8" is not recommended. If unsure,
 		 you should consider the following option instead.

"the following option" is no longer correct.

Documentation/filesystems/vfat.txt iocharset=name -- Character set to use f · line 45

Quoted 11 lines
Documentation/filesystems/vfat.txt renamed from Documentation/filesystems/vfat.txt~fat_dont-use_free_clusters-for-fat32 +7 -0
+usefree       -- Use the "free clusters" value stored on FSINFO. It'll
+                 be used to determine number of free clusters without
+                 scanning disk. But it's not used by default, because
+                 recent Windows don't update it correctly in some
+                 case. If you are sure the "free clusters" on FSINFO is
+                 correct, by this option you can avoid scanning disk.
+
 utf8=<bool>   -- UTF-8 is the filesystem safe version of Unicode that
 		 is used by the console.  It can be enabled for the
 		 filesystem with this option. If 'uni_xlate' gets set,

Andreas.

Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
#8 lore raw
Quoted 10 lines · Andreas Schwab

Documentation/filesystems/vfat.txt iocharset=name -- Character set to use f · line 35

Quoted 6 lines · OGAWA Hirofumi

diff -puN Documentation/filesystems/vfat.txt~fat_dont-use_free_clusters-for-fat32 Documentation/filesystems/vfat.txt

Documentation/filesystems/vfat.txt renamed from Documentation/filesystems/vfat.txt~fat_dont-use_free_clusters-for-fat32 +0 -0
@@ -34,6 +34,13 @@ iocharset=name -- Character set to use f
 		 NOTE: "iocharset=utf8" is not recommended. If unsure,
 		 you should consider the following option instead.

"the following option" is no longer correct.

Whoops. Thanks for correcting it.
The patch was fixed.

Signature · 108 lines
OGAWA Hirofumi <[email protected]>



It seems that the recent Windows changed specification, and it's
undocumented. Windows doesn't update ->free_clusters correctly.

This patch doesn't use ->free_clusters by default. (instead, add
"usefree" for forcing to use it)

Signed-off-by: OGAWA Hirofumi <[email protected]>
---

 Documentation/filesystems/vfat.txt |    7 +++++++
 fs/fat/inode.c                     |   14 +++++++++++---
 include/linux/msdos_fs.h           |    3 ++-
 3 files changed, 20 insertions(+), 4 deletions(-)

diff -puN fs/fat/inode.c~fat_dont-use_free_clusters-for-fat32 fs/fat/inode.c
--- linux-2.6/fs/fat/inode.c~fat_dont-use_free_clusters-for-fat32	2007-04-22 13:07:05.000000000 +0900
+++ linux-2.6-hirofumi/fs/fat/inode.c	2007-04-22 13:18:07.000000000 +0900
@@ -825,6 +825,8 @@ static int fat_show_options(struct seq_f
 	}
 	if (opts->name_check != 'n')
 		seq_printf(m, ",check=%c", opts->name_check);
+	if (opts->usefree)
+		seq_puts(m, ",usefree");
 	if (opts->quiet)
 		seq_puts(m, ",quiet");
 	if (opts->showexec)
@@ -850,7 +852,7 @@ static int fat_show_options(struct seq_f
 
 enum {
 	Opt_check_n, Opt_check_r, Opt_check_s, Opt_uid, Opt_gid,
-	Opt_umask, Opt_dmask, Opt_fmask, Opt_codepage, Opt_nocase,
+	Opt_umask, Opt_dmask, Opt_fmask, Opt_codepage, Opt_usefree, Opt_nocase,
 	Opt_quiet, Opt_showexec, Opt_debug, Opt_immutable,
 	Opt_dots, Opt_nodots,
 	Opt_charset, Opt_shortname_lower, Opt_shortname_win95,
@@ -872,6 +874,7 @@ static match_table_t fat_tokens = {
 	{Opt_dmask, "dmask=%o"},
 	{Opt_fmask, "fmask=%o"},
 	{Opt_codepage, "codepage=%u"},
+	{Opt_usefree, "usefree"},
 	{Opt_nocase, "nocase"},
 	{Opt_quiet, "quiet"},
 	{Opt_showexec, "showexec"},
@@ -951,7 +954,7 @@ static int parse_options(char *options, 
 	opts->quiet = opts->showexec = opts->sys_immutable = opts->dotsOK =  0;
 	opts->utf8 = opts->unicode_xlate = 0;
 	opts->numtail = 1;
-	opts->nocase = 0;
+	opts->usefree = opts->nocase = 0;
 	*debug = 0;
 
 	if (!options)
@@ -979,6 +982,9 @@ static int parse_options(char *options, 
 		case Opt_check_n:
 			opts->name_check = 'n';
 			break;
+		case Opt_usefree:
+			opts->usefree = 1;
+			break;
 		case Opt_nocase:
 			if (!is_vfat)
 				opts->nocase = 1;
@@ -1306,7 +1312,9 @@ int fat_fill_super(struct super_block *s
 			       le32_to_cpu(fsinfo->signature2),
 			       sbi->fsinfo_sector);
 		} else {
-			sbi->free_clusters = le32_to_cpu(fsinfo->free_clusters);
+			if (sbi->options.usefree)
+				sbi->free_clusters =
+					le32_to_cpu(fsinfo->free_clusters);
 			sbi->prev_free = le32_to_cpu(fsinfo->next_cluster);
 		}
 
diff -puN include/linux/msdos_fs.h~fat_dont-use_free_clusters-for-fat32 include/linux/msdos_fs.h
--- linux-2.6/include/linux/msdos_fs.h~fat_dont-use_free_clusters-for-fat32	2007-04-22 13:07:05.000000000 +0900
+++ linux-2.6-hirofumi/include/linux/msdos_fs.h	2007-04-22 13:09:14.000000000 +0900
@@ -205,7 +205,8 @@ struct fat_mount_options {
 		 numtail:1,       /* Does first alias have a numeric '~1' type tail? */
 		 atari:1,         /* Use Atari GEMDOS variation of MS-DOS fs */
 		 flush:1,	  /* write things quickly */
-		 nocase:1;	  /* Does this need case conversion? 0=need case conversion*/
+		 nocase:1,	  /* Does this need case conversion? 0=need case conversion*/
+		 usefree:1;	  /* Use free_clusters for FAT32 */
 };
 
 #define FAT_HASH_BITS	8
diff -puN Documentation/filesystems/vfat.txt~fat_dont-use_free_clusters-for-fat32 Documentation/filesystems/vfat.txt
--- linux-2.6/Documentation/filesystems/vfat.txt~fat_dont-use_free_clusters-for-fat32	2007-04-22 23:06:21.000000000 +0900
+++ linux-2.6-hirofumi/Documentation/filesystems/vfat.txt	2007-04-23 00:09:35.000000000 +0900
@@ -57,6 +57,13 @@ nonumtail=<bool> -- When creating 8.3 al
                  currently exist in the directory, 'longfile.txt' will
                  be the short alias instead of 'longfi~1.txt'. 
                   
+usefree       -- Use the "free clusters" value stored on FSINFO. It'll
+                 be used to determine number of free clusters without
+                 scanning disk. But it's not used by default, because
+                 recent Windows don't update it correctly in some
+                 case. If you are sure the "free clusters" on FSINFO is
+                 correct, by this option you can avoid scanning disk.
+
 quiet         -- Stops printing certain warning messages.
 
 check=s|r|n   -- Case sensitivity checking setting.
_
#9 lore raw
In reply to OGAWA Hirofumi
Quoted 6 lines · OGAWA Hirofumi
Quoted 1 line · Bodo Eggert

Windows _does_ care*, it will pretend the disk to be full.

Did you test on 2000 or XP? (e.g. write 0 to free_clusters, then create new file.)

That was back when I still used W98.

Quoted 3 lines
Quoted 1 line

- usefree is a bad name (I'd suggest recalc_free instead),

Is it about nofree option?

Yes. I think recalc_free is way more descriptive.

Quoted 4 lines
Quoted 2 lines

and your description is too cryptic to be understood by a non-linux FAT expert.

Um... why do we need to care about non-linux people in the patch?

Non-experts in Linux FAT implementation details. "recalc sounds like might take long, let's try to disable it" should be all it takes to get a fast mount.

Signature · 2 lines
Top 100 things you don't want the sysadmin to say:
35. Ummm... Didn't you say you turned it off?
#10 lore raw
Quoted 5 lines · Bodo Eggert
Quoted 3 lines
Quoted 1 line

- usefree is a bad name (I'd suggest recalc_free instead),

Is it about nofree option?

Yes. I think recalc_free is way more descriptive.

Recalc is already default on current patch.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#11 lore raw
In reply to Bodo Eggert

Hi Bodo :)

* Bodo Eggert <[email protected]> dixit:

Quoted 14 lines
Quoted 7 lines · OGAWA Hirofumi
Quoted 7 lines

* Juergen Beisert <[email protected]> dixit:

Quoted 4 lines

So the last free sector count is also stored. When mounting this filesystem you don't need to walk through the whole FAT to calculate the available space, you can use this "cached" value instead. And this cached value seems not to be updated in your portable device.

Quoted 5 lines
Quoted 5 lines

It doesn't, certainly, but Windows doesn't care. Moreover, the
device doesn't seem to recalculate the value on every run (unless it does it lightning fast!), so maybe the number is stored elsewhere (the count can be stored in many places as far as I've read, but I don't know the details).

Quoted 3 lines

AFAIR it's stored twice on FAT32, once in a backup sector and once in the superblock or extended superblock (don't remember, I think it was the extended ~). It's not stored on FAT{12,16}.

I hadn't noticed the same problem with FAT16 pendrives, so I
suspected that. Thanks for confirming :)

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net
It's my PC and I'll cry if I want to... RAmen!

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#16 lore raw
Quoted 10 lines · OGAWA Hirofumi
Quoted 5 lines · DervishD

Probably it's stupid to update the free clusters count at mount time
(sorry if so...) but it looks like a good idea to me. And of course, I don't mean to update the value _on disk_, but the kernel's idea of free clusters (so even FAT filesystems mounted R/O will report correct values).

It would add the limitation to following simple usage,

Quoted 7 lines
# mount -t vfat /dev/sda1 /mnt
        # cp -a * /mnt
        # umount

if /dev/sda1 was the large and slow device, "mount" will need several minutes to counts free clusters. I think the user will be hard to accept the several minutes at "mount".

I simulated recalculating the FAT using a suboptimal perl script on my 32 GB FAT32 partition on a 40 GB drive. The results should show the worst case for my system.

The results were below 2 seconds (optimized perl at 0.5 s) on my AMD Athlon XP 2400+. (BTW: I don't think the device speed itself will be relevant in many cases even if you'd asume a slow link, since you'll need to read the FAT anyway.)

I think this is acceptable, and on this system, I'd like to enable recalculating by default. On slower systems, you'll need to chose another default.

I think you'll want that option as a module parameter to make all users happy. And while you're at it, fmask and dmask are good candidates for module parameters, too.-)

# echo 3 > /proc/sys/vm/drop_caches
# /usr/bin/time sh -c "dd if=/dev/hda2 bs=$((77834925/64)) count=4 | perl -e
'"'use POSIX;while (sysread STDIN,$x,4) {$n++ if $x eq "\0\0\0\0"} print
$n*32,"\n"'\'
4+0 records in
4+0 records out
13954720
0.76user 0.60system 0:01.95elapsed 69%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (8major+1583minor)pagefaults 0swaps
# /usr/bin/time sh -c "dd if=/dev/hda2 bs=$((77834925/64)) count=4 | perl -e
'"'use POSIX;while (sysread STDIN,$x,4) {$n++ if $x eq "\0\0\0\0"} print
$n*32,"\n"'\'
4+0 records in
4+0 records out
13955392
0.74user 0.69system 0:01.50elapsed 95%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+1591minor)pagefaults 0swaps
# /usr/bin/time sh -c "dd if=/dev/hda2 bs=$((77834925/64)) count=4 | perl -e
'"'use POSIX;while (sysread STDIN,$x,4096) {for (unpack "L*",$x){$n++ if !$_}}
print $n*32,"\n"'\'
4+0 records in
4+0 records out
13955392
0.43user 0.06system 0:00.50elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+1599minor)pagefaults 0swaps
Signature · 4 lines
The worse the weather, the more you are required to be out in it. 

Friß, Spammer: [email protected] [email protected]
 [email protected] [email protected]
#17 lore raw
Quoted 8 lines · Bodo Eggert
# /usr/bin/time sh -c "dd if=/dev/hda2 bs=$((77834925/64)) count=4 | perl -e
'"'use POSIX;while (sysread STDIN,$x,4) {$n++ if $x eq "\0\0\0\0"} print
$n*32,"\n"'\'
4+0 records in
4+0 records out
13954720
0.76user 0.60system 0:01.95elapsed 69%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (8major+1583minor)pagefaults 0swaps

dd will use readahead, and I'm thinking about usb. Well, anyway, recalc is default in new patch.

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>