Skip to contentSkip to conversation

mkfs.xfs doesn't detect size of storage correctly

DISCUSSION Matthias Schniedermeyer 4 participants 15 messages Jan 29, 2008 – Feb 5, 2008 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hi

Yesterday i bought a 750GB HDD.
I encrypt nearly everything with loop-aes, so i also did it with this HDD.

I create a "fake" partition table and:
losetup -e aes256 -p 0 -o 4096 /dev/loop6 /dev/sdb < key

This creates a loop with everything except the first 4KB, i.e. it leaves out the MBR and another 3,5KB.

/proc/partions shows the correct(tm) size informations for the HDD and the loop:
- snip -

   7     6  732574580 loop6
   8    16  732574584 sdb
   8    17  732572001 sdb1

- snip -

But when i mkfs.xfs the loop
#> mkfs.xfs /dev/loop6

meta-data=/dev/loop6             isize=256    agcount=3, agsize=45785911 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=137357733, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

And mount it:
mount /dev/loop6 /mnt

And least but not least df it:
#> df -m /mnt
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/loop6 536426 5 536422 1% /mnt

There is roughly 1/3 missing.

What can i do to fix this?

#> mkfs.xfs -V
mkfs.xfs version 2.9.5
(Package is from: Debian SID/unstable)

Kernel is 2.6.24 (vanilla, plus of course the external module: loop-aes)

Btw. I have 50 other HDDs in more or less exactly the same setup, (49 HDDs >= 200 GB and <= 500GB and one 1TB, of couse created with my system at an older stage from Feb 2007 onwoard, the 750GB is the first with kernel 2.6.24.) Up to now i can't remember ever "missing" space.

Bis denn

Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.
#1 lore raw
Quoted 36 lines

Hi

Yesterday i bought a 750GB HDD.
I encrypt nearly everything with loop-aes, so i also did it with this HDD.

I create a "fake" partition table and:
losetup -e aes256 -p 0 -o 4096 /dev/loop6 /dev/sdb < key

This creates a loop with everything except the first 4KB, i.e. it leaves out the MBR and another 3,5KB.

/proc/partions shows the correct(tm) size informations for the HDD and the loop:
- snip -

   7     6  732574580 loop6
   8    16  732574584 sdb
   8    17  732572001 sdb1

- snip -

But when i mkfs.xfs the loop
#> mkfs.xfs /dev/loop6
meta-data=/dev/loop6 isize=256 agcount=3, agsize=45785911
blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=137357733, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

And mount it:
mount /dev/loop6 /mnt

Quoted 8 lines

And least but not least df it:
#> df -m /mnt
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/loop6 536426 5 536422 1% /mnt

There is roughly 1/3 missing.

What can i do to fix this?

mkfs.xfs uses the BLKGETSIZE64 ioctl to extract the device size, so the problem is likely in the loop device driver (just a guess). You can use the test program xfs-cmds/xfstests/src/getdevicesize.c to test what that device returns as its size (no XFS-specific code in the test program, so if it returns bad data we've narrowed down the root cause a whole lot).

What does that program produce for your device?

cheers.

--
Nathan

#2 lore raw
Quoted 45 lines · On 29.01.2008 21:15, [email protected] wrote:
Quoted 36 lines

Hi

Yesterday i bought a 750GB HDD.
I encrypt nearly everything with loop-aes, so i also did it with this HDD.

I create a "fake" partition table and:
losetup -e aes256 -p 0 -o 4096 /dev/loop6 /dev/sdb < key

This creates a loop with everything except the first 4KB, i.e. it leaves out the MBR and another 3,5KB.

/proc/partions shows the correct(tm) size informations for the HDD and the loop:
- snip -

   7     6  732574580 loop6
   8    16  732574584 sdb
   8    17  732572001 sdb1

- snip -

But when i mkfs.xfs the loop
#> mkfs.xfs /dev/loop6
meta-data=/dev/loop6 isize=256 agcount=3, agsize=45785911
blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=137357733, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

And mount it:
mount /dev/loop6 /mnt

Quoted 8 lines

And least but not least df it:
#> df -m /mnt
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/loop6 536426 5 536422 1% /mnt

There is roughly 1/3 missing.

What can i do to fix this?

Quoted 8 lines

mkfs.xfs uses the BLKGETSIZE64 ioctl to extract the device size, so the problem is likely in the loop device driver (just a guess). You can use the test program xfs-cmds/xfstests/src/getdevicesize.c to test what that device returns as its size (no XFS-specific code in the test program, so if it returns bad data we've narrowed down the root cause a whole lot).

What does that program produce for your device?

After a little odyssey to actually finding getdevicesize.c :-)

#> wget 'http://oss.sgi.com/cgi-bin/cvsweb.cgi/~checkout~/xfs-cmds/xfstests/src/getdevicesize.c?rev=1.3;content-type=text%2Fplain' -O getdevicesize.c #> gcc -o getdevicesize getdevicesize.c
#> ./getdevicesize /dev/loop6
1465149160 512 byte blocks (BLKGETSIZE64)

Which is 750156369920 bytes or 732574580 KBytes, IOW EXACTLY the number reported in /proc/partions for the loop.

A maybe important detail, i forgot to mention in the original mail is:
The machine has 8GB of RAM, so i compiled the kernel with "64bit=yes" (formaly x86_64), BUT(!) the userspace is 32bit or plain old i386.

I only need the RAM for a large tmpfs. So i didn't reinstall the machine with a 64bit userspace when i switched the hardware from a Dual P3 with 3GB to a Core2Duo with 8GB.

Bis denn

Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.
#3 lore raw
Quoted 3 lines · Matthias Schniedermeyer

A maybe important detail, i forgot to mention in the original mail is:
The machine has 8GB of RAM, so i compiled the kernel with "64bit=yes" (formaly x86_64), BUT(!) the userspace is 32bit or plain old i386.

If you think that's the issue, I'd try compiling the above as a 32-bit binary and re-run.

Although, for a simple sparse-file-backed loopback device w/o loop-aes, that doesn't cause any problem for me:

[root@inode tmp]# ls -l loopfile
-rw-r--r-- 1 root root 750156369920 2008-01-29 08:53 loopfile

[root@inode tmp]# losetup /dev/loop0 loopfile

[root@inode tmp]# ./gds32 /dev/loop0 1465149160 512 byte blocks (BLKGETSIZE64) [root@inode tmp]# ./gds64 /dev/loop0 1465149160 512 byte blocks (BLKGETSIZE64)

-Eric

Quoted 2 lines

Bis denn

#4 lore raw
Quoted 8 lines · Eric Sandeen
Quoted 3 lines · Matthias Schniedermeyer

A maybe important detail, i forgot to mention in the original mail is:
The machine has 8GB of RAM, so i compiled the kernel with "64bit=yes" (formaly x86_64), BUT(!) the userspace is 32bit or plain old i386.

If you think that's the issue, I'd try compiling the above as a 32-bit binary and re-run.

I don't think it's an issue, i just wanted to mention it.

But as soon as i get home i will connect the HDD to an all 32bit system and see if it makes a difference. (I'm still undecided if i should hope that it makes a difference. ;-) )

Bis denn

Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.
#5 lore raw
Quoted 6 lines · Matthias Schniedermeyer
Quoted 5 lines · Eric Sandeen
Quoted 3 lines · Matthias Schniedermeyer

A maybe important detail, i forgot to mention in the original mail is:
The machine has 8GB of RAM, so i compiled the kernel with "64bit=yes" (formaly x86_64), BUT(!) the userspace is 32bit or plain old i386.

Quoted 9 lines
Quoted 2 lines

If you think that's the issue, I'd try compiling the above as a 32-bit binary and re-run.

I don't think it's an issue, i just wanted to mention it.

But as soon as i get home i will connect the HDD to an all 32bit system and see if it makes a difference. (I'm still undecided if i should hope that it makes a difference. ;-) )

well, this can't be hard to find, with a bit of investigation. :)

As Nathan said, mkfs.xfs just uses standard ioctls to get device size, so something is odd... you could sprinkle some printfs into the mkfs code, too, to further debug.

-Eric

Quoted 4 lines

Bis denn

#6 lore raw
Quoted 7 lines · Eric Sandeen
Quoted 6 lines · Matthias Schniedermeyer
Quoted 5 lines · Eric Sandeen
Quoted 3 lines · Matthias Schniedermeyer

A maybe important detail, i forgot to mention in the original mail is:
The machine has 8GB of RAM, so i compiled the kernel with "64bit=yes" (formaly x86_64), BUT(!) the userspace is 32bit or plain old i386.

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

If you think that's the issue, I'd try compiling the above as a 32-bit binary and re-run.

I don't think it's an issue, i just wanted to mention it.

But as soon as i get home i will connect the HDD to an all 32bit system and see if it makes a difference. (I'm still undecided if i should hope that it makes a difference. ;-) )

I'm back home, and i just connected the HDD to my all 32bit system. No difference.

Bis denn

Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.
#7 lore raw
In reply to Matthias Schniedermeyer
Quoted 23 lines · Matthias Schniedermeyer

Hi

Yesterday i bought a 750GB HDD.
I encrypt nearly everything with loop-aes, so i also did it with this HDD.

I create a "fake" partition table and:
losetup -e aes256 -p 0 -o 4096 /dev/loop6 /dev/sdb < key

This creates a loop with everything except the first 4KB, i.e. it leaves out the MBR and another 3,5KB.

/proc/partions shows the correct(tm) size informations for the HDD and the loop:
- snip -

   7     6  732574580 loop6
   8    16  732574584 sdb
   8    17  732572001 sdb1

- snip -

But when i mkfs.xfs the loop
#> mkfs.xfs /dev/loop6

Quoted 8 lines
meta-data=/dev/loop6             isize=256    agcount=3, agsize=45785911 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=137357733, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

I just found a workaround. :-)
As can be seen above the agcount is 3.

For a reason that lays years in the past, when there were issues with the agcount (actually agsize), that to the best of my knowledge are fixed years ago (but still cause a weird feeling whenever i see that word) i just tried '-d agcount=4'

#> mkfs.xfs -l size=1024b -d agcount=4 /dev/loop6 -f

meta-data=/dev/loop6             isize=256    agcount=4, agsize=45785912 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=183143645, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

As can be seen the "blocks"-number is much higher.

#> df -k /mnt

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/loop6           732570484      4256 732566228   1% /mnt

matches(tm) with the loop-size.

So i'm "burned" by agcount/agsize AGAIN, seems my weird feeling about those two words are still with reason. ;-)

Bis denn

Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.
#8 lore raw
Quoted 18 lines · Matthias Schniedermeyer

I just found a workaround. :-)
As can be seen above the agcount is 3.

For a reason that lays years in the past, when there were issues with the agcount (actually agsize), that to the best of my knowledge are fixed years ago (but still cause a weird feeling whenever i see that word) i just tried '-d agcount=4'

#> mkfs.xfs -l size=1024b -d agcount=4 /dev/loop6 -f

meta-data=/dev/loop6             isize=256    agcount=4, agsize=45785912 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=183143645, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0
Quoted 8 lines

As can be seen the "blocks"-number is much higher.

#> df -k /mnt

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/loop6           732570484      4256 732566228   1% /mnt

matches(tm) with the loop-size.

Ohh.. um... Barry? problems :)

[root@magnesium tmp]# mkfs.xfs -dfile,name=xfsfile,size=750156369920
meta-data=xfsfile isize=256 agcount=3, agsize=45785911
blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=137357733, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@magnesium tmp]# bc
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
137357733*4096
562617274368

ouch... that's, um, not the right size :)

-Eric

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw
Quoted 6 lines · Chris Wedgwood
Quoted 2 lines · Eric Sandeen

[root@magnesium tmp]# mkfs.xfs -dfile,name=xfsfile,size=750156369920

Quoted 9 lines

I see this too except I'm pre-creating a file of the right length:

-rw-r--r-- 1 root root 750156369920 Jan 29 12:35 xfsfile

[...]

/root/xfsfile xfs 524G 4.2M 524G 1% /mnt/tmp

I think this fixes it, but it was just a quickie so pls double check :)

if agcount * agsize <= dblocks, then the last ag shouldn't be "small"

-Eric

mkfs/xfs_mkfs.c +4 -2
Index: xfsprogs-2.9.5/mkfs/xfs_mkfs.c
===================================================================
@@ -523,7 +523,8 @@ validate_ag_geometry(
 	 * If the last AG is too small, reduce the filesystem size
 	 * and drop the blocks.
 	 */
-	if ( dblocks % agsize != 0 &&
+	if ( agcount * agsize > dblocks &&
+	     dblocks % agsize != 0 &&
 	     (dblocks % agsize < XFS_AG_MIN_BLOCKS(blocklog))) {
 		fprintf(stderr,
 	_("last AG size %lld blocks too small, minimum size is %lld blocks\n"),
@@ -1808,7 +1809,8 @@ _("size %s specified for log subvolume i
 	 * If the last AG is too small, reduce the filesystem size
 	 * and drop the blocks.
 	 */
-	if ( dblocks % agsize != 0 &&
+	if ( agcount * agsize > dblocks &&
+	     dblocks % agsize != 0 &&
 	     (dblocks % agsize < XFS_AG_MIN_BLOCKS(blocklog))) {
 		dblocks = (xfs_drfsbno_t)((agcount - 1) * agsize);
 		agcount--;
#11 lore raw
In reply to Matthias Schniedermeyer

A little followup.

Today i bought another 750GB HDD, and a 500GB.

Initializing the encryption of the 500GB finished a minute ago and the same bug, of course with a different size (357702MB), was observable.

Workaround also worked in this case (Now its 476937MB). This also proves that the bug is of recent nature, the last 500GB HDD i bought was just over a month ago and this bug didn't show up then.

Bis denn

Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.
#12 lore raw
Quoted 10 lines · Matthias Schniedermeyer

A little followup.

Today i bought another 750GB HDD, and a 500GB.

Initializing the encryption of the 500GB finished a minute ago and the same bug, of course with a different size (357702MB), was observable.

Workaround also worked in this case (Now its 476937MB). This also proves that the bug is of recent nature, the last 500GB HDD i bought was just over a month ago and this bug didn't show up then.

Yes, it's a regression in 2.9.5.

http://oss.sgi.com/cgi-bin/cvsweb.cgi/xfs-cmds/xfsprogs/mkfs/xfs_mkfs.c.diff?r1=1.82;r2=1.83

-Eric

#13 lore raw
In reply to Matthias Schniedermeyer
Quoted 1 line · Matthias Schniedermeyer

There is roughly 1/3 missing.

SGI, have you settled on a fix for this? Latest released xfsprogs, 2.9.5 has this problem, and I've put it in rawhide; it's also in the F9 alpha spin. Should I revert to 2.9.4 if you don't have plans to fix it soon?

Nothing indicates to the user that their fs was misformatted, and there will be a growing number fileystems missing a substantial portion of their space out there....

-Eric

#14 lore raw
Quoted 4 lines · Eric Sandeen
Quoted 1 line · Matthias Schniedermeyer

There is roughly 1/3 missing.

Quoted 8 lines

SGI, have you settled on a fix for this? Latest released xfsprogs, 2.9.5 has this problem, and I've put it in rawhide; it's also in the F9 alpha spin. Should I revert to 2.9.4 if you don't have plans to fix it soon?

Nothing indicates to the user that their fs was misformatted, and there will be a growing number fileystems missing a substantial portion of their space out there....

The big question here is if you can grow the filesystem to fix it.

Before i found the workaround with the agcount i tested to grow the fs, but it didn't work.

So: A correctly(tm) fixed package i think it must include a xfs_grows that can fix a damaged(tm) fs, otherwise people who use such a filesystem (and can't backup/restore) are stuck with missing a large chunk of space.

Bis denn

Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.
#15 lore raw
In reply to Eric Sandeen
Quoted 5 lines · Eric Sandeen
Quoted 1 line · Matthias Schniedermeyer

There is roughly 1/3 missing.

SGI, have you settled on a fix for this?

Yes, Barry is reviewing and testing your proposed fix. The delay was caused by most of the XFS team attending LCA last week. .. sorry.

We'll get a new package out onto OSS asap.

Cheers
-- Mark

Quoted 12 lines

Latest released xfsprogs,
2.9.5 has this problem, and I've put it in rawhide; it's also in the F9 alpha spin. Should I revert to 2.9.4 if you don't have plans to fix it soon?

Nothing indicates to the user that their fs was misformatted, and there will be a growing number fileystems missing a substantial portion of their space out there....

-Eric