fsx for Linux showing up reiserfs problem?
Hi folks,
After reading the article at http://www.kerneltrap.com/article.php?sid=415&mode=thread&order=0
on the FreeBSD guys finding a bunch of NFS bugs with a stress tool,
I took a look at fsx and played with it a little under Linux..
The changes to make it work are trivial, and are at
http://www.codemonkey.org.uk/cruft/fsx-linux.c
(non-existant include & expected mmap() behaviour differences)
I've done a few tests on local filesystems, and so far Ext2 & Ext3 seem to be holding up..
Reiserfs however dies very early into the test..
truncating to largest ever: 0x3f15f
READ BAD DATA: offset = 0x1d3d4, size = 0x962f
OFFSET GOOD BAD RANGE 0x1d3d4 0x177d 0x0000 0x 563
operation# (mod 256) for the bad data unknown, check HOLE and EXTEND ops
Options used were ./fsx -c1234 /mnt/test/testfile
(Although it seems to crash with any -c option)
Looks like an interesting tool, and probably something that should be added to testsuites like Cerberus.
regards,
Dave.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs .
Quoted 2 lines · Dave Jones
I've done a few tests on local filesystems, and so far Ext2 & Ext3 seem to be holding up..
I've tested ext3 on 2.4.17-pre8 and it works fine.
Quoted 7 lines
Reiserfs however dies very early into the test..
truncating to largest ever:
0x3f15f
READ BAD DATA: offset =0x1d3d4, size =0x962fOFFSET GOOD BAD RANGE 0x1d3d4 0x177d 0x0000 0x 563operation# (mod 256) for the bad data unknown, check HOLE and EXTEND ops
I tested with reiserfs on 2.4.9-ac14 and 2.4.15-pre4-xfs and it fails in the same way on both.
I tested on a xfs partition on the 2.4.15-pre4-xfs kernel and didn't get any errors either.
/Martin
Never argue with an idiot. They drag you down to their level, then beat you with experience.
More fun with fsx.
In reply to Dave JonesQuoted 3 lines · Dave Jones
The changes to make it work are trivial, and are at
http://www.codemonkey.org.uk/cruft/fsx-linux.c
(non-existant include & expectedmmap()behaviour differences)
Whilst waiting for the local fs tests to finish/progress I did a quick test with nfs. It dies after a while with an io error.
You can skip over the boring part and go straight to the interesting failure case by using the options -c1234 -b 48870
Should happen quite quickly after that.
Full log of the failcase is at
http://www.codemonkey.org.uk/cruft/nfs-fsx.txt
regards,
Dave.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs
Quoted 20 lines · Dave Jones
Hi folks,
After reading the article at http://www.kerneltrap.com/article.php?sid=415&mode=thread&order=0
on the FreeBSD guys finding a bunch of NFS bugs with a stress tool,
I took a look at fsx and played with it a little under Linux..The changes to make it work are trivial, and are at
http://www.codemonkey.org.uk/cruft/fsx-linux.c
(non-existant include & expectedmmap()behaviour differences)I've done a few tests on local filesystems, and so far Ext2 & Ext3 seem to be holding up..
Reiserfs however dies very early into the test..
truncating to largest ever:
0x3f15f
READ BAD DATA: offset =0x1d3d4, size =0x962fOFFSET GOOD BAD RANGE 0x1d3d4 0x177d 0x0000 0x 563operation# (mod 256) for the bad data unknown, check HOLE and EXTEND ops
Quoted 9 lines
Options used were ./fsx -c1234
/mnt/test/testfile
(Although it seems to crash with any -c option)Looks like an interesting tool, and probably something that should be added to testsuites like Cerberus.
regards,
Dave.
Thanks Dave, Elena and Nikita and Green, take a look at this.
Hans
More fun with fsx.
In reply to Dave JonesQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Dave Jones <[email protected]> writes:
> On Sat, 15 Dec 2001, Dave Jones wrote:
>> The changes to make it work are trivial, and are at >> http://www.codemonkey.org.uk/cruft/fsx-linux.c (non-existant >> include & expected mmap() behaviour differences)
> Whilst waiting for the local fs tests to finish/progress I did > a quick test with nfs. It dies after a while with an io error.
> You can skip over the boring part and go straight to the > interesting failure case by using the options -c1234 -b 48870 > Should happen quite quickly after that.
> Full log of the failcase is at
> http://www.codemonkey.org.uk/cruft/nfs-fsx.txt
Which kernel and mount options?
Cheers,
Trond
Re: More fun with fsx.
Quoted 3 lines · Trond Myklebust
> Full log of the failcase is at
> http://www.codemonkey.org.uk/cruft/nfs-fsx.txt
Which kernel and mount options?
client 2.4.13-ac5, server 2.4.17rc1, default options, nothing special. I'll try to reproduce with a newer kernel on the client.
regards,
Dave.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs
Re: More fun with fsx.
Quoted 5 lines · Dave Jones
Quoted 3 lines
Quoted 2 lines
Full log of the failcase is at
http://www.codemonkey.org.uk/cruft/nfs-fsx.txtWhich kernel and mount options?
client 2.4.13-ac5, server 2.4.17rc1, default options, nothing special. I'll try to reproduce with a newer kernel on the client.
Yup, it's still there with 2.4.17-rc1 as the client too. Exact same failure.
regards,
Dave.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs
Re: More fun with fsx.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Dave Jones <[email protected]> writes:
> Yup, it's still there with 2.4.17-rc1 as the client too. Exact > same failure.
I get a size error when I don't apply the patch that fixes the attribute update race. Can you try with that patch applied? It can be found on
http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif
Cheers,
Trond
Re: More fun with fsx.
Quoted 4 lines · Trond Myklebust
I get a size error when I don't apply the patch that fixes the attribute update race. Can you try with that patch applied? It can be found on
http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif
No change with this applied on client side. Should I bother recompiling server side with this ?
regards,
Dave.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs
Re: More fun with fsx.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Dave Jones <[email protected]> writes:
> On Sun, 16 Dec 2001, Trond Myklebust wrote:
>> I get a size error when I don't apply the patch that fixes the >> attribute update race. Can you try with that patch applied? It >> can be found on
>> http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif
> No change with this applied on client side. Should I bother > recompiling server side with this ?
Nope. It is client-only...
Cheers,
Trond
Re: More fun with fsx.
In reply to Dave JonesQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Dave Jones <[email protected]> writes:
> On Sun, 16 Dec 2001, Dave Jones wrote:
>> > > Full log of the failcase is at
>> > > http://www.codemonkey.org.uk/cruft/nfs-fsx.txt
>> > Which kernel and mount options?
>> client 2.4.13-ac5, server 2.4.17rc1, default options, nothing >> special. I'll try to reproduce with a newer kernel on the >> client.
> Yup, it's still there with 2.4.17-rc1 as the client too. Exact > same failure.
I found the bug. It's a pretty ugly race...
The problem is that although the inode->i_sem semaphore protects you against races with ordinary writes, it does *not* protect you against the memory management routines (such as page_launder()) from calling writepage() on dirty pages.
For most local filesystems, this is not a problem, since they can call vmtruncate() *before* they do their thing on the disk itself. For non-local filesystems, you usually can't do this, since you cannot guarantee that the RPC call will succeed on the server.
The only solution I can see, is to force a synchronous write of those dirty pages to the server prior to calling setattr. I've updated the patch on
http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif
to do this. Fixes the race for me...
Cheers,
Trond
Re: More fun with fsx.
Quoted 1 line · Trond Myklebust
I found the bug. It's a pretty ugly race...
Well, you found _a_ bug perhaps, but not this one.. Still repeatedly fails in exactly the same part with your second patch applied instead.
Dave.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs
Re: More fun with fsx.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Dave Jones <[email protected]> writes:
> On Sun, 16 Dec 2001, Trond Myklebust wrote:
>> I found the bug. It's a pretty ugly race...
> Well, you found _a_ bug perhaps, but not this one.. Still > repeatedly fails in exactly the same part with your second > patch applied instead.
In that case, I'll need a tcpdump of the point at which the error occurs.
I'm unable to produce any problem whatsoever with the new patch applied. Certainly not an EIO: that can normally only occur if you are using soft mounts (which I assume is not the case?) or if the server is screwed up...
Cheers,
Trond
Re: More fun with fsx.
Quoted 2 lines · Trond Myklebust
In that case, I'll need a tcpdump of the point at which the error occurs.
Sure no problem... any particular preferred options ? Want client and server, or just client ?
Quoted 4 lines
I'm unable to produce any problem whatsoever with the new patch applied. Certainly not an EIO: that can normally only occur if you are using soft mounts (which I assume is not the case?) or if the server is screwed up...
This is 'defaults' so, I would guess that is the case looking at the man page. but, I just tested and its repeatable with hard and soft.
regards,
Dave.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs
Re: More fun with fsx.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Dave Jones <[email protected]> writes:
> On Sun, 16 Dec 2001, Trond Myklebust wrote:
>> In that case, I'll need a tcpdump of the point at which the >> error occurs.
> Sure no problem... any particular preferred options ? Want > client and server, or just client ?
Client should suffice. If you could start something like
tcpdump -s 256 -w tcpdump.out
close to the point at which the error occurs, and send me the resulting file, then that should be OK. I'm mainly out to check whether or not the server is the one returning the EIO error, or possibly if it is returning bad post-op attributes. Those are the only remaining possibilities if hard mounts are being used.
BTW: could you also tell me a bit about the server? Is this an ext[23] partition and knfsd? I'm still a bit wary of ReiserFS...
Cheers,
Trond