ext3 filesystem corruption in user mode linux
Hi,
I'm running a stock 2.6.35.4 kernel, on both the host and the guest.
For one test, I've loaded a stage3 gentoo system including portage, created completely on the host, using dd, a 10gig sparse virtual disk, mkfs.ext3, and the gentoo tarballs. (I get fs corruption with Ubuntu guests as well).
If I do some heavy CPU and disk work in the guest, something like this:
# (cd /usr && tar cjf - portage) | tar xjf - # rm -rf portage
Then the filesystem corrupts itself, giving errors like the following in dmesg, during the rm:
EXT3-fs error (device ubda): ext3_lookup: deleted inode referenced: 51715
I've posted my kernel configs online here:
Host: http://foursquare.net/kernel/host-config.txt
UML: http://foursquare.net/kernel/uml-config.txt
I'm not sure what the next debugging step should be. I can test different versions of UML kernels, or test patches, if more testing is needed.
Thanks,
- Chris
Quoted 9 lines · Chris Frey
If I do some heavy CPU and disk work in the guest, something like this:
# (cd /usr && tar cjf - portage) | tar xjf - # rm -rf portageThen the filesystem corrupts itself, giving errors like the following in dmesg, during the rm:
EXT3-fs error (device ubda):
ext3_lookup: deleted inode referenced: 51715
Update:
So far I have been unable to make 2.6.32.21 corrupt its filesystem when running as the UML kernel (Host 2.6.35.4, UML 2.6.32.21)
When running UML 2.6.33.7, I get similar errors:
EXT3-fs error (device ubda): ext3_lookup: deleted inode referenced: 21397
EXT3-fs error (device ubda): ext3_lookup: deleted inode referenced: 13257
EXT3-fs error (device ubda): ext3_lookup: deleted inode referenced: 13257
EXT3-fs error (device ubda): ext3_lookup: deleted inode referenced: 21403
EXT3-fs error (device ubda): ext3_lookup: deleted inode referenced: 21403
EXT3-fs error (device ubda): ext3_lookup: deleted inode referenced: 13262
EXT3-fs error (device ubda): ext3_lookup: deleted inode referenced: 13262
EXT3-fs error (device ubda): htree_dirblock_to_tree: bad entry in directory #13269: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0 EXT3-fs (ubda): warning: empty_dir: bad directory (dir #13269) - no .' or ..'
EXT3-fs (ubda): warning: ext3_rmdir: empty directory has nlink!=2 (3)
If anyone wants me to test a specific UML kernel, please let me know.
- Chris
Quoted 2 lines · Chris Frey
So far I have been unable to make 2.6.32.21 corrupt its filesystem when running as the UML kernel (Host 2.6.35.4, UML 2.6.32.21)
Spoke too soon... 2.6.32.21 has errors too, if pushed hard enough.
- Chris
Quoted 6 lines · Chris Frey
Quoted 2 lines · Chris Frey
So far I have been unable to make 2.6.32.21 corrupt its filesystem when running as the UML kernel (Host 2.6.35.4, UML 2.6.32.21)
Spoke too soon... 2.6.32.21 has errors too, if pushed hard enough.
can you do a git bisect?
Signature · 2 lines
Cheers, //richard
Quoted 9 lines · richard -rw- weinberger
Quoted 6 lines · Chris Frey
Quoted 2 lines · Chris Frey
So far I have been unable to make 2.6.32.21 corrupt its filesystem when running as the UML kernel (Host 2.6.35.4, UML 2.6.32.21)
Spoke too soon... 2.6.32.21 has errors too, if pushed hard enough.
can you do a git bisect?
I've been trying, but so far the range of buggy versions stretches back to 2.6.32.x at least. I'm trying to find a version that doesn't have this issue.
I'm kinda surprised that nobody else has run into this. Or maybe user
mode linux isn't used as much as I thought? Or I'm just doing something
completely bone headed. :-)
- Chris
Quoted 11 lines · Chris Frey
Quoted 9 lines · richard -rw- weinberger
Quoted 6 lines · Chris Frey
Quoted 2 lines · Chris Frey
So far I have been unable to make 2.6.32.21 corrupt its filesystem when running as the UML kernel (Host 2.6.35.4, UML 2.6.32.21)
Spoke too soon... 2.6.32.21 has errors too, if pushed hard enough.
can you do a git bisect?
Quoted 7 lines
I've been trying, but so far the range of buggy versions stretches back to 2.6.32.x at least. I'm trying to find a version that doesn't have this issue.
I'm kinda surprised that nobody else has run into this. Or maybe user mode linux isn't used as much as I thought? Or I'm just doing something completely bone headed. :-)
Hmm, maybe this issue affects only your configuration. When I have some spare time I'll try to trigger this on my setup.
Signature · 2 lines
Cheers, //richard
Quoted 16 lines · richard -rw- weinberger
Quoted 15 lines · Chris Frey
Quoted 9 lines · richard -rw- weinberger
Quoted 6 lines · Chris Frey
Quoted 2 lines · Chris Frey
So far I have been unable to make 2.6.32.21 corrupt its filesystem when running as the UML kernel (Host 2.6.35.4, UML 2.6.32.21)
Spoke too soon... 2.6.32.21 has errors too, if pushed hard enough.
can you do a git bisect?
I've been trying, but so far the range of buggy versions stretches back to 2.6.32.x at least. ?I'm trying to find a version that doesn't have this issue.
Quoted 6 lines
Quoted 3 lines
I'm kinda surprised that nobody else has run into this. ?Or maybe user mode linux isn't used as much as I thought? ? Or I'm just doing something completely bone headed. :-)
Hmm, maybe this issue affects only your configuration. When I have some spare time I'll try to trigger this on my setup.
That would be great. Let me know if you need any specific data from my end.
- Chris
Quoted 4 lines · Chris Frey
Quoted 2 lines
Hmm, maybe this issue affects only your configuration. When I have some spare time I'll try to trigger this on my setup.
That would be great. Let me know if you need any specific data from my end.
One data point that I'm finding so far, is that it is easier to trigger in 2.6.35.x than 2.6.32.x. It also doesn't seem to be dependent on the version of the host kernel. I can reproduce it running 2.6.35.4 or 2.6.32.22.
If you want a small gentoo starting point, I have a 56M compressed ext3 gentoo guest filesystem I can share with those who wish to test. I don't want to make it available to everyone, due to size.
- Chris
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 17 lines · richard -rw- weinberger
Quoted 9 lines · Chris Frey
I'm using a mixture of the following to test. The errors happen during the 'rm'.
Direct copy:
(cd dir && tar cjf - portage) | tar xjf - ; rm -rf portageHostfs copy:
tar xjf/mnt/hostfs/portage-latest.tar.bz2; rm -rf portageQuoted 6 lines
Network copy:
ssh remote "cat portage-latest.tar.bz2" | tar xjf - ; rm -rf portageWith these tests, I'm almost guessing that it might be some missed IRQs or something in the guest, since files that are corrupt often contain all zeros, which would match the sparse filesystem images I'm using.
Quoted 5 lines
Hmm, something really nasty is going one here.
I can reproduce this issue using ext2, ext3 and reiserfs as UML root filesystem. It seems to be a block layer issue.
Tomorrow I'll have a close look at the issue using my openSUSE setup. So far I've used your Gentoo image.
I've also seen the issue with a Ubuntu guest as well. So far, I don't think it matters what OS is in the guest.
Thanks for reproducing the error!
- Chris
Quoted 2 lines · Chris Frey
I'm not sure what the next debugging step should be. I can test different versions of UML kernels, or test patches, if more testing is needed.
Is this the wrong place to post about user mode linux? I searched but I don't see a mailing list specific to ARCH=um.
Thanks,
- Chris
Quoted 8 lines · Chris Frey
Quoted 3 lines · Chris Frey
I'm not sure what the next debugging step should be. I can test different versions of UML kernels, or test patches, if more testing is needed.
Is this the wrong place to post about user mode linux? I searched but I don't see a mailing list specific to ARCH=um.
I would expect here to be OK, but the MAINTAINERS file also lists 2 mailing lists and one web site:
L: [email protected]
L: [email protected]
W: http://user-mode-linux.sourceforge.net
Signature · 1 line
~Randy