Skip to contentSkip to conversation

fsx for Linux showing up reiserfs problem?

DISCUSSION Dave Jones 4 participants 15 messages Dec 15, 2001 – Dec 17, 2001

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

Hi folks,
After reading the article at http://www.kerneltrap.com/article.php?sid=415&mode=thread&order=0
on the FreeBSD guys finding a bunch of NFS bugs with a stress tool,
I took a look at fsx and played with it a little under Linux..

The changes to make it work are trivial, and are at
http://www.codemonkey.org.uk/cruft/fsx-linux.c
(non-existant include & expected mmap() behaviour differences)

I've done a few tests on local filesystems, and so far Ext2 & Ext3 seem to be holding up..

Reiserfs however dies very early into the test..

truncating to largest ever: 0x3f15f
READ BAD DATA: offset = 0x1d3d4, size = 0x962f

  OFFSET  GOOD    BAD     RANGE
  0x1d3d4 0x177d  0x0000  0x  563

operation# (mod 256) for the bad data unknown, check HOLE and EXTEND ops

Options used were ./fsx -c1234 /mnt/test/testfile
(Although it seems to crash with any -c option)

Looks like an interesting tool, and probably something that should be added to testsuites like Cerberus.

regards,
Dave.

Signature · 2 lines
| Dave Jones.                    http://www.codemonkey.org.uk
| SuSE Labs .
#1 lore raw
Quoted 2 lines · Dave Jones

I've done a few tests on local filesystems, and so far Ext2 & Ext3 seem to be holding up..

I've tested ext3 on 2.4.17-pre8 and it works fine.

Quoted 7 lines

Reiserfs however dies very early into the test..

truncating to largest ever: 0x3f15f
READ BAD DATA: offset = 0x1d3d4, size = 0x962f

  OFFSET  GOOD    BAD     RANGE
  0x1d3d4 0x177d  0x0000  0x  563

operation# (mod 256) for the bad data unknown, check HOLE and EXTEND ops

I tested with reiserfs on 2.4.9-ac14 and 2.4.15-pre4-xfs and it fails in the same way on both.

I tested on a xfs partition on the 2.4.15-pre4-xfs kernel and didn't get any errors either.

/Martin

Never argue with an idiot. They drag you down to their level, then beat you with experience.

author
#2 lore raw

More fun with fsx.

In reply to Dave Jones
Quoted 3 lines · Dave Jones

The changes to make it work are trivial, and are at
http://www.codemonkey.org.uk/cruft/fsx-linux.c
(non-existant include & expected mmap() behaviour differences)

Whilst waiting for the local fs tests to finish/progress I did a quick test with nfs. It dies after a while with an io error.

You can skip over the boring part and go straight to the interesting failure case by using the options -c1234 -b 48870
Should happen quite quickly after that.

Full log of the failcase is at
http://www.codemonkey.org.uk/cruft/nfs-fsx.txt

regards,
Dave.

Signature · 2 lines
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs
#3 lore raw

More fun with fsx.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Dave Jones <[email protected]> writes:

> On Sat, 15 Dec 2001, Dave Jones wrote:
>> The changes to make it work are trivial, and are at >> http://www.codemonkey.org.uk/cruft/fsx-linux.c (non-existant >> include & expected mmap() behaviour differences)

> Whilst waiting for the local fs tests to finish/progress I did > a quick test with nfs. It dies after a while with an io error.

> You can skip over the boring part and go straight to the > interesting failure case by using the options -c1234 -b 48870 > Should happen quite quickly after that.

> Full log of the failcase is at
> http://www.codemonkey.org.uk/cruft/nfs-fsx.txt

Which kernel and mount options?

Cheers,
Trond

author
#4 lore raw

Re: More fun with fsx.

Quoted 3 lines · Trond Myklebust

> Full log of the failcase is at
> http://www.codemonkey.org.uk/cruft/nfs-fsx.txt
Which kernel and mount options?

client 2.4.13-ac5, server 2.4.17rc1, default options, nothing special. I'll try to reproduce with a newer kernel on the client.

regards,
Dave.

Signature · 2 lines
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs
author
#5 lore raw

Re: More fun with fsx.

Quoted 5 lines · Dave Jones
Quoted 3 lines
Quoted 2 lines

Full log of the failcase is at
http://www.codemonkey.org.uk/cruft/nfs-fsx.txt

Which kernel and mount options?

client 2.4.13-ac5, server 2.4.17rc1, default options, nothing special. I'll try to reproduce with a newer kernel on the client.

Yup, it's still there with 2.4.17-rc1 as the client too. Exact same failure.

regards,
Dave.

Signature · 2 lines
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs
author
#7 lore raw

Re: More fun with fsx.

Quoted 4 lines · Trond Myklebust

I get a size error when I don't apply the patch that fixes the attribute update race. Can you try with that patch applied? It can be found on
http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif

No change with this applied on client side. Should I bother recompiling server side with this ?

regards,
Dave.

Signature · 2 lines
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs
#8 lore raw

Re: More fun with fsx.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Dave Jones <[email protected]> writes:

> On Sun, 16 Dec 2001, Trond Myklebust wrote:
>> I get a size error when I don't apply the patch that fixes the >> attribute update race. Can you try with that patch applied? It >> can be found on
>> http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif

> No change with this applied on client side. Should I bother > recompiling server side with this ?

Nope. It is client-only...

Cheers,
Trond

#9 lore raw

Re: More fun with fsx.

In reply to Dave Jones
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Dave Jones <[email protected]> writes:

> On Sun, 16 Dec 2001, Dave Jones wrote:
>> > > Full log of the failcase is at
>> > > http://www.codemonkey.org.uk/cruft/nfs-fsx.txt
>> > Which kernel and mount options?
>> client 2.4.13-ac5, server 2.4.17rc1, default options, nothing >> special. I'll try to reproduce with a newer kernel on the >> client.

> Yup, it's still there with 2.4.17-rc1 as the client too. Exact > same failure.

I found the bug. It's a pretty ugly race...

The problem is that although the inode->i_sem semaphore protects you against races with ordinary writes, it does *not* protect you against the memory management routines (such as page_launder()) from calling writepage() on dirty pages.
For most local filesystems, this is not a problem, since they can call vmtruncate() *before* they do their thing on the disk itself. For non-local filesystems, you usually can't do this, since you cannot guarantee that the RPC call will succeed on the server.

The only solution I can see, is to force a synchronous write of those dirty pages to the server prior to calling setattr. I've updated the patch on

http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif

to do this. Fixes the race for me...

Cheers,
Trond

author
#10 lore raw

Re: More fun with fsx.

Quoted 1 line · Trond Myklebust

I found the bug. It's a pretty ugly race...

Well, you found _a_ bug perhaps, but not this one.. Still repeatedly fails in exactly the same part with your second patch applied instead.

Dave.

Signature · 2 lines
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs
#11 lore raw

Re: More fun with fsx.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Dave Jones <[email protected]> writes:

> On Sun, 16 Dec 2001, Trond Myklebust wrote:
>> I found the bug. It's a pretty ugly race...

> Well, you found _a_ bug perhaps, but not this one.. Still > repeatedly fails in exactly the same part with your second > patch applied instead.

In that case, I'll need a tcpdump of the point at which the error occurs.

I'm unable to produce any problem whatsoever with the new patch applied. Certainly not an EIO: that can normally only occur if you are using soft mounts (which I assume is not the case?) or if the server is screwed up...

Cheers,
Trond

author
#12 lore raw

Re: More fun with fsx.

Quoted 2 lines · Trond Myklebust

In that case, I'll need a tcpdump of the point at which the error occurs.

Sure no problem... any particular preferred options ? Want client and server, or just client ?

Quoted 4 lines

I'm unable to produce any problem whatsoever with the new patch applied. Certainly not an EIO: that can normally only occur if you are using soft mounts (which I assume is not the case?) or if the server is screwed up...

This is 'defaults' so, I would guess that is the case looking at the man page. but, I just tested and its repeatable with hard and soft.

regards,
Dave.

Signature · 2 lines
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs
#13 lore raw

Re: More fun with fsx.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Dave Jones <[email protected]> writes:

> On Sun, 16 Dec 2001, Trond Myklebust wrote:
>> In that case, I'll need a tcpdump of the point at which the >> error occurs.

> Sure no problem... any particular preferred options ? Want > client and server, or just client ?

Client should suffice. If you could start something like

tcpdump -s 256 -w tcpdump.out

close to the point at which the error occurs, and send me the resulting file, then that should be OK. I'm mainly out to check whether or not the server is the one returning the EIO error, or possibly if it is returning bad post-op attributes. Those are the only remaining possibilities if hard mounts are being used.

BTW: could you also tell me a bit about the server? Is this an ext[23] partition and knfsd? I'm still a bit wary of ReiserFS...

Cheers,
Trond

#14 lore raw
In reply to Dave Jones
Quoted 20 lines · Dave Jones

Hi folks,
After reading the article at http://www.kerneltrap.com/article.php?sid=415&mode=thread&order=0
on the FreeBSD guys finding a bunch of NFS bugs with a stress tool,
I took a look at fsx and played with it a little under Linux..

The changes to make it work are trivial, and are at
http://www.codemonkey.org.uk/cruft/fsx-linux.c
(non-existant include & expected mmap() behaviour differences)

I've done a few tests on local filesystems, and so far Ext2 & Ext3 seem to be holding up..

Reiserfs however dies very early into the test..

truncating to largest ever: 0x3f15f
READ BAD DATA: offset = 0x1d3d4, size = 0x962f

 OFFSET  GOOD    BAD     RANGE
 0x1d3d4 0x177d  0x0000  0x  563

operation# (mod 256) for the bad data unknown, check HOLE and EXTEND ops

Quoted 9 lines

Options used were ./fsx -c1234 /mnt/test/testfile
(Although it seems to crash with any -c option)

Looks like an interesting tool, and probably something that should be added to testsuites like Cerberus.

regards,
Dave.

Thanks Dave, Elena and Nikita and Green, take a look at this.

Hans