Skip to contentSkip to conversation

problems with xfs_growfs after lvextend

DISCUSSION Jimmy Dorff 4 participants 7 messages Sep 23, 2009 – Sep 24, 2009

lore
Share & tools
lore
#0 lore raw

Hello,

I'm having a problem with a corrupt xfs filesystem after attempting to grow the filesystem.

The problem is very similar to this post:
http://www.redhat.com/archives/linux-lvm/2005-November/msg00026.html
Except that xfs_repair never finds a secondary superblock.

Details:
CentOS Linux,
kernel 2.6.18-128.1.16.el5.centos.plus x86_64
lvm2-2.02.40-6.el5
originally: xfsprogs-2.9.4-1.el5.centos.x86_64

This sever has been up and running for a few months w/o problems. Today we added some disks to a 3ware controller. The disks were all tested individually before installation.

Used "tw_cli" to configure a new RAID volume, which appeared in Linux as normal (/dev/sdd).

# pvcreate /dev/sdd
# vgextend array_vg /dev/sdd
# lvextend /dev/array_vg/data --size +12T

This all worked w/o any errors. vgdisplay and lvdisplay all report the correct info and sizes.

The 6TB xfs filesystem on /dev/array_vg/data was mounted as "/srv/data"

# xfs_growfs /srv/data

The size of the filesystem didn't change. I unmounted it and tried again, but no change. However, now I can't mount the filesystem at all.

xfs_check causes xfs_db to use so much memory as to hang up the system. xfs_repair reports:
Phase 1 - find and verify superblock...
superblock read failed, offset 19791209299968, size 2048, ag 96, rval 0

fatal error -- Invalid argument

Also, I've noticed this in syslog:
kernel: attempt to access beyond end of device
kernel: dm-1: rw=0, want=64424509440, limit=38654705664
kernel: I/O error in filesystem ("dm-1") meta-data dev dm-1 block 0xeffffffff ("xfs_read_buf") error 5 buf count 512
kernel: XFS: size check 2 failed

I've trying using xfs_repair from xfsprogs 3.0.3, but it made no difference.

Any suggestions ? Any help understanding why this didn't work ?

Thanks,
Jimmy Dorff

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 19 lines · Jimmy Dorff

Hello,

I'm having a problem with a corrupt xfs filesystem after attempting to grow the filesystem.

The problem is very similar to this post:
http://www.redhat.com/archives/linux-lvm/2005-November/msg00026.html
Except that xfs_repair never finds a secondary superblock.

Details:
CentOS Linux,
kernel 2.6.18-128.1.16.el5.centos.plus x86_64
lvm2-2.02.40-6.el5
originally: xfsprogs-2.9.4-1.el5.centos.x86_64

This sever has been up and running for a few months w/o problems. Today we added some disks to a 3ware controller. The disks were all tested individually before installation.

Quoted 6 lines

Used "tw_cli" to configure a new RAID volume, which appeared in Linux as normal (/dev/sdd).

# pvcreate /dev/sdd
# vgextend array_vg /dev/sdd
# lvextend /dev/array_vg/data --size +12T

that centos kernel had a bug where growing more than 2T (I Think ...) at a time overflowed. So for +12T you probably hit it.

That's probably your bug, and cattelan on this list has helped people recover from it before, I'd suggest not trying to "fix" anything just yet (don't force repair, etc).

It's fixed in the RHEL5.4 xfs code from RH, FWIW.

-Eric

Quoted 31 lines

This all worked w/o any errors. vgdisplay and lvdisplay all report the correct info and sizes.

The 6TB xfs filesystem on /dev/array_vg/data was mounted as "/srv/data"

# xfs_growfs /srv/data

The size of the filesystem didn't change. I unmounted it and tried again, but no change. However, now I can't mount the filesystem at all.

xfs_check causes xfs_db to use so much memory as to hang up the system. xfs_repair reports:
Phase 1 - find and verify superblock...
superblock read failed, offset 19791209299968, size 2048, ag 96, rval 0

fatal error -- Invalid argument

Also, I've noticed this in syslog:
kernel: attempt to access beyond end of device
kernel: dm-1: rw=0, want=64424509440, limit=38654705664
kernel: I/O error in filesystem ("dm-1") meta-data dev dm-1 block
0xeffffffff ("xfs_read_buf") error 5 buf count 512
kernel: XFS: size check 2 failed

I've trying using xfs_repair from xfsprogs 3.0.3, but it made no difference.

Any suggestions ? Any help understanding why this didn't work ?

Quoted 8 lines

Thanks,
Jimmy Dorff

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
In reply to Jimmy Dorff

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 47 lines · Jimmy Dorff

Hello,

I'm having a problem with a corrupt xfs filesystem after attempting
to grow the filesystem.

The problem is very similar to this post:
http://www.redhat.com/archives/linux-lvm/2005-November/msg00026.html
Except that xfs_repair never finds a secondary superblock.

Details: CentOS Linux, kernel 2.6.18-128.1.16.el5.centos.plus x86_64 lvm2-2.02.40-6.el5 originally:
xfsprogs-2.9.4-1.el5.centos.x86_64

This sever has been up and running for a few months w/o problems. Today we added some disks to a 3ware controller. The disks were all
tested individually before installation.

Used "tw_cli" to configure a new RAID volume, which appeared in Linux as normal (/dev/sdd).

# pvcreate /dev/sdd # vgextend array_vg /dev/sdd # lvextend
/dev/array_vg/data --size +12T

This all worked w/o any errors. vgdisplay and lvdisplay all report the correct info and sizes.

The 6TB xfs filesystem on /dev/array_vg/data was mounted as "/srv/data" # xfs_growfs /srv/data

The size of the filesystem didn't change. I unmounted it and tried again, but no change. However, now I can't mount the filesystem at all.

xfs_check causes xfs_db to use so much memory as to hang up the system. xfs_repair reports: Phase 1 - find and verify superblock...
superblock read failed, offset 19791209299968, size 2048, ag 96,
rval 0

fatal error -- Invalid argument

Also, I've noticed this in syslog: kernel: attempt to access beyond end of device kernel: dm-1: rw=0, want=64424509440, limit=38654705664 kernel: I/O error in filesystem ("dm-1")
meta-data dev dm-1 block 0xeffffffff ("xfs_read_buf") error 5
buf count 512 kernel: XFS: size check 2 failed

Quoted 4 lines

I've trying using xfs_repair from xfsprogs 3.0.3, but it made no difference.

Any suggestions ? Any help understanding why this didn't work ?

Sounds like you hit a known bug with xfs_growfs, actually it is a bug with the kernel interface to growfs, the
size used to be a 32 bit number and thus would overflow if trying to grow the filesystem by more than 2TB.

The good new is you can easily restore the the values of the primary super block from one of the other superblocks
using the xfs_db cmd.

The bad news is that if you have run repair you may have messed things up.

basically pull up xfs_db

Quoted 1 line

sb 0 p

... output ..

Quoted 1 line

sb 1 p

... output ...

Send us the result of that and we can figure out which values need to be restored.

- -Russell

Quoted 7 lines

Thanks, Jimmy Dorff

_______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKupUjNRmM+OaGhBgRAhH0AJ9Z98xpNsATGPeJAezQNwSB/sxGZACfRlwy FFgr2iTYOYCHd0BkCsAOW6k=
=SAIq
-----END PGP SIGNATURE-----

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
In reply to Jimmy Dorff
Quoted 1 line · On Mittwoch 23 September 2009 Jimmy Dorff wrote:

Any suggestions ? Any help understanding why this didn't work ?

Hm, hasn't there been a problem with growing more than 2TB at once?

I search in my local box and found this, might help:
May 09: [PATCH] fix overflow in xfs_growfs_data_privat

Maybe kernel update?

mfg zmi

Signature · 10 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
In reply to Russell Cattelan
Quoted 6 lines · Russell Cattelan

The good new is you can easily restore the the values of the primary super block from one of the other superblocks
using the xfs_db cmd.

The bad news is that if you have run repair you may have messed things up.

Quoted 8 lines

basically pull up xfs_db

Quoted 1 line

sb 0 p

... output ..

Quoted 1 line

sb 1 p

... output ...

Send us the result of that and we can figure out which values need to be restored.

Here you go. Thanks for your help!

xfs_db> sb 0
xfs_db> p
magicnum = 0x58465342
blocksize = 4096
dblocks = 8053063680
rblocks = 0
rextents = 0
uuid = bada0c85-0739-4fdb-99fb-6bb7777eb4d0
logstart = 1073741828
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 50331648
agcount = 160
rbmblocks = 0
logblocks = 32768
versionnum = 0x3084
sectsize = 512
inodesize = 256
inopblock = 16
fname = "DATA\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 26
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 103936
ifree = 269
fdblocks = 6499724269
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0
bad_features2 = 0
xfs_db> sb 1
xfs_db> p
magicnum = 0x58465342
blocksize = 4096
dblocks = 1610612736
rblocks = 0
rextents = 0
uuid = bada0c85-0739-4fdb-99fb-6bb7777eb4d0
logstart = 1073741828
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 50331648
agcount = 32
rbmblocks = 0
logblocks = 32768
versionnum = 0x3084
sectsize = 512
inodesize = 256
inopblock = 16
fname = "DATA\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 26
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 103936
ifree = 269
fdblocks = 57273829
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0
bad_features2 = 0

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 10 lines · Jimmy Dorff
Quoted 9 lines · Russell Cattelan

The good new is you can easily restore the the values of the primary super block from one of the other superblocks using the xfs_db cmd.

The bad news is that if you have run repair you may have messed things up.

basically pull up xfs_db

Quoted 1 line

sb 0 p

Quoted 8 lines
Quoted 6 lines

... output ..

Quoted 1 line

sb 1 p

... output ...

Send us the result of that and we can figure out which values need to be restored.

Here you go. Thanks for your help!

Ok good things look ok.
run xfs_db -x to put it into expert mode
use the write cmd to modify your sb 0 values back to what they were.

Quoted 1 line

sb 0 w dblocks = 1610612736 w agcount = 32

That should get the file system back into a state xfs_repair can understand.
I would run xfs_repair -n and see what is says about things. The old last AG might have been resized out to a full AG if it was a partial AG previously
xfs_epair should be able to fix that up, but again run it with -n first to see what is will do.

Once things are back to normal update your xfs kernel module and re-run the grow cmd
and you should be in good shape.

- -Russell

Quoted 18 lines

xfs_db> sb 0 xfs_db> p magicnum = 0x58465342 blocksize = 4096 dblocks = 8053063680 rblocks = 0 rextents = 0 uuid = bada0c85-0739-4fdb-99fb-6bb7777eb4d0 logstart = 1073741828 rootino = 128 rbmino = 129 rsumino = 130 rextsize = 1 agblocks = 50331648 agcount = 160 rbmblocks = 0 logblocks = 32768 versionnum = 0x3084 sectsize = 512 inodesize = 256 inopblock = 16 fname = "DATA\000\000\000\000\000\000\000\000" blocklog = 12 sectlog = 9 inodelog = 8 inopblog = 4 agblklog = 26 rextslog = 0 inprogress = 0
imax_pct = 25 icount = 103936 ifree = 269 fdblocks = 6499724269
frextents = 0 uquotino = 0 gquotino = 0 qflags = 0 flags = 0 shared_vn = 0 inoalignmt = 2 unit = 0 width = 0 dirblklog = 0 logsectlog = 0 logsectsize = 0 logsunit = 0 features2 = 0 bad_features2 = 0 xfs_db> sb 1 xfs_db> p magicnum = 0x58465342 blocksize = 4096 dblocks = 1610612736 rblocks = 0 rextents = 0 uuid = bada0c85-0739-4fdb-99fb-6bb7777eb4d0 logstart = 1073741828 rootino = 128 rbmino = 129 rsumino = 130 rextsize = 1 agblocks = 50331648 agcount = 32 rbmblocks = 0 logblocks = 32768 versionnum =

Quoted 8 lines

0x3084 sectsize = 512 inodesize = 256 inopblock = 16 fname = "DATA\000\000\000\000\000\000\000\000" blocklog = 12 sectlog = 9 inodelog = 8 inopblog = 4 agblklog = 26 rextslog = 0 inprogress = 0
imax_pct = 25 icount = 103936 ifree = 269 fdblocks = 57273829
frextents = 0 uquotino = 0 gquotino = 0 qflags = 0 flags = 0 shared_vn = 0 inoalignmt = 2 unit = 0 width = 0 dirblklog = 0 logsectlog = 0 logsectsize = 0 logsunit = 0 features2 = 0 bad_features2 = 0

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKurasNRmM+OaGhBgRAt4YAJ9aHdZF3iX5hjcrcm6FqS/avk+bBQCfft0U HTAySeS7kYxNvmnwsmalgKw=
=f80e
-----END PGP SIGNATURE-----

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
Quoted 6 lines · Russell Cattelan

Ok good things look ok.
run xfs_db -x to put it into expert mode
use the write cmd to modify your sb 0 values back to what they were.

Quoted 1 line

sb 0 w dblocks = 1610612736 w agcount = 32

That worked perfectly! Much thanks!!

-Jimmy

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs