Skip to contentSkip to conversation

BUG: unable to handle kernel NULL pointer dereference - nfs v3

DISCUSSION David CHANIAL 4 participants 12 messages Jul 16, 2007 – Sep 4, 2007 NFS server

lore
Share & tools
lore
#0 lore raw

Hi,

I'm not sure is the good place to poste that, and if not - please excuse me.

I was running nfs server v2 since a year on one server, there is few days, i have update my kernel to 2.6.21.3 with support of nfsv3 server.

Somes times per days i have somes crash as below, needing i reboot the server to nfs re-become up.

************

BUG: unable to handle kernel NULL pointer dereference at virtual address 

00000004
printing eip:
c01e7279
*pde = 09ecc001

Oops: 0000 [#1]
SMP
CPU:    0
EIP:    0060:[<c01e7279>]    Not tainted VLI
EFLAGS: 00010246   (2.6.21.3-sdf88-core #9)

EIP is at encode_fsid+0x67/0x89
eax: e5bde8c0 ebx: f7593404 ecx: 00000000 edx: 00000006
esi: dc569048 edi: f75934ec ebp: f7593404 esp: f75f1f18
ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068
Process nfsd (pid: 3386, ti=f75f0000 task=f7b295b0 task.ti=f75f0000)

Stack: 00000000 dc569048 c01e7381 f5bd0df8 c01e5bde 00000000 f6c95000 c01e7fdf
       dc56901c c0519584 c01e7ffc f75934ec f6c95000 c01dd38a f77d68c0 d0ab4200
       c043d99e dc569014 f6c95000 f6c950e0 f7593538 c0519584 c0439f32 00000000
Call Trace:
 [<c01e7381>] encode_fattr3+0xe6/0x131
 [<c01e5bde>] nfsd3_proc_getattr+0xaf/0xb9
 [<c01e7fdf>] nfs3svc_encode_attrstat+0x0/0x3b
 [<c01e7ffc>] nfs3svc_encode_attrstat+0x1d/0x3b
 [<c01dd38a>] nfsd_dispatch+0x13e/0x190
 [<c043d99e>] svcauth_unix_set_client+0x152/0x15e
 [<c0439f32>] svc_process+0x38d/0x636
 [<c01dd14f>] nfsd+0x171/0x26e
 [<c01dcfde>] nfsd+0x0/0x26e
 [<c010312f>] kernel_thread_helper+0x7/0x10
 =======================
Code: e2 08 09 d1 09 c1 eb 10 8b 83 88 00 00 00 8b 40 30 89 c3 89 c1 c1 fb 1f 

89 d8 0f c8 89 06 89 c8 eb 1e
************

Best regards,

Signature · 1 line
David 
#1 lore raw
Quoted 3 lines · On Monday July 16, [email protected] wrote:

Hi,

I'm not sure is the good place to poste that, and if not - please excuse me.

This is the correct place to post this, thanks.

Quoted 10 lines

I was running nfs server v2 since a year on one server, there is few days, i have update my kernel to 2.6.21.3 with support of nfsv3 server.

Somes times per days i have somes crash as below, needing i reboot the server to nfs re-become up.

************

BUG: unable to handle kernel NULL pointer dereference at virtual address 

00000004

^^^^^^^^

This says that it tried to access memory at address '4'. There is no memory there, so it caused the BUG.

Quoted 8 lines

printing eip:
c01e7279
*pde = 09ecc001

Oops: 0000 [#1]
SMP
CPU:    0
EIP:    0060:[<c01e7279>]    Not tainted VLI
EFLAGS: 00010246   (2.6.21.3-sdf88-core #9)
                              ^^^^^^^^^^^

What is "-sdf88-core" ?? Are there any extra patches that we should know about?

Quoted 1 line

EIP is at encode_fsid+0x67/0x89

This is presumably where the illegal access happened.

Quoted 2 lines

eax: e5bde8c0 ebx: f7593404 ecx: 00000000 edx: 00000006
esi: dc569048 edi: f75934ec ebp: f7593404 esp: f75f1f18

Memory accesses are (almost) always relative to the value in some register. Of these registers, the most likely is ecx, with edx a vague possibility.

Quoted 2 lines
Code: e2 08 09 d1 09 c1 eb 10 8b 83 88 00 00 00 8b 40 30 89 c3 89 c1 c1 fb 1f 

89 d8 0f c8 89 06 89 c8 eb 1e

Unfortunately "ksymoops" does seem to decode this into something quite useful enough. Normally one of the numbers has <> around it. Are you should you copied the number across exactly?

This code decodes as:

   0:   e2 08                     loop   a <_EIP+0xa>
   2:   09 d1                     or     %edx,%ecx
   4:   09 c1                     or     %eax,%ecx
   6:   eb 10                     jmp    18 <_EIP+0x18>
   8:   8b 83 88 00 00 00         mov    0x88(%ebx),%eax
   e:   8b 40 30                  mov    0x30(%eax),%eax
  11:   89 c3                     mov    %eax,%ebx

....

From the 'jmp' onwards, that is what I would expect to see in encode_fsid. The code before there doesn't make a lot of sense, so it is hard to pinpoint exactly there the error is.

In any case, there is no place in encode_fsid where an offset of 4 from any register is indexed, nor an offset of -2.
So either there is something wrong with the decoding and displaying of this information, or there is something very wrong with your hardware.

I would suggest:
1/ if possible, run memtest86 on the machine for a while, to make
sure there isn't a problem with the memory.
2/ If the problem happens again, post another report with all the
"oops" information again. Maybe the next time it will be slightly different and will make more sense in some way.

NeilBrown

#2 lore raw

Le mardi 17 juillet 2007 13:13, Neil Brown a écrit :

Quoted 2 lines

What is "-sdf88-core" ?? Are there any extra patches that we should know about?

I have upgraded to 2.6.22.1-sdf90-intel
-sdf90 is a set of a configuration with one little patch :

	http://www.ssi.bg/~ja/#hidden

intel or core juste say for wich CPU it was compiled (the previous kernel "core" was named-error, the kernel really was for a intel (and not core2) cpu)

Quoted 5 lines

   1/ if possible, run memtest86 on the machine for a while, to make sure there isn't a problem with the memory.
   2/ If the problem happens again, post another report with all the "oops" information again.  Maybe the next time it will be slightly different and will make more sense in some way.

Since the kernel update, no crash anymore - for the moment -. If it re-crash i will floow this advices.

Thanks Niel,

Signature · 1 line
David
#3 lore raw
In reply to Neil Brown

Hi Neil,

[ okay, just searching through my lkml folder looking for "unable to handle" :-) ]

Quoted 12 lines · Neil Brown
Quoted 6 lines · On Monday July 16, [email protected] wrote:

************

BUG: unable to handle kernel NULL pointer dereference at virtual address

00000004

EIP is at encode_fsid+0x67/0x89

This is presumably where the illegal access happened.

Quoted 2 lines

eax: e5bde8c0 ebx: f7593404 ecx: 00000000 edx: 00000006
esi: dc569048 edi: f75934ec ebp: f7593404 esp: f75f1f18

Yup, ecx is to blame here ...

Quoted 6 lines
Quoted 2 lines
Code: e2 08 09 d1 09 c1 eb 10 8b 83 88 00 00 00 8b 40 30 89 c3 89 c1 c1 fb 1f

89 d8 0f c8 89 06 89 c8 eb 1e

Unfortunately "ksymoops" does seem to decode this into something quite useful enough. Normally one of the numbers has <> around it. Are you should you copied the number across exactly?

Yes, I think David missed posting the full "Code:" here. Unfortunate.

Quoted 2 lines

In any case, there is no place in encode_fsid where an offset of 4 from any register is indexed, nor an offset of -2.

But I went ahead and disassembled encode_fsid() anyway. I did stumble across a "mov 0x4(%ecx), %edx" -- which turns out to be:

static __be32 *encode_fsid(__be32 *p, struct svc_fh *fhp)

{
	u64 f;
	switch(fsid_source(fhp)) {
	default:
	case FSIDSOURCE_DEV:
		p = xdr_encode_hyper(p, (u64)huge_encode_dev
				(fhp->fh_dentry->d_inode->i_sb->s_dev));
		break;
	case FSIDSOURCE_FSID:
		p = xdr_encode_hyper(p, (u64) fhp->fh_export->ex_fsid);
		break;
	case FSIDSOURCE_UUID:
		f = ((u64*)fhp->fh_export->ex_uuid)[0];
		f ^= ((u64*)fhp->fh_export->ex_uuid)[1]; /* ***** HERE ***** */
		p = xdr_encode_hyper(p, f);
		break;
	}
	return p;
}

Note that fhp->fh_export->ex_uuid is an unsigned char *, which is 4 bytes on an i386 (which is what David's system is). For some reason fhp->fh_export->ex_uuid (%ecx) is NULL here, which leads to the oops. I have _zero_ other knowledge of knfsd code, and not really be of any other use, sorry.

Satyam

#4 lore raw

Ugh, not a good day for me today ... my earlier conclusion was right, but not the reasoning behind it ... hopefully this time I'll do better :-)

Quoted 37 lines · Satyam Sharma

Hi Neil,

[ okay, just searching through my lkml folder looking for "unable to handle" :-) ]

Quoted 12 lines · Neil Brown
Quoted 6 lines · On Monday July 16, [email protected] wrote:

************

BUG: unable to handle kernel NULL pointer dereference at virtual address

00000004

EIP is at encode_fsid+0x67/0x89

This is presumably where the illegal access happened.

Quoted 2 lines

eax: e5bde8c0 ebx: f7593404 ecx: 00000000 edx: 00000006
esi: dc569048 edi: f75934ec ebp: f7593404 esp: f75f1f18

Yup, ecx is to blame here ...

Quoted 6 lines
Quoted 2 lines
Code: e2 08 09 d1 09 c1 eb 10 8b 83 88 00 00 00 8b 40 30 89 c3 89 c1 c1 fb 1f

89 d8 0f c8 89 06 89 c8 eb 1e

Unfortunately "ksymoops" does seem to decode this into something quite useful enough. Normally one of the numbers has <> around it. Are you should you copied the number across exactly?

Yes, I think David missed posting the full "Code:" here. Unfortunate.

Quoted 2 lines

In any case, there is no place in encode_fsid where an offset of 4 from any register is indexed, nor an offset of -2.

But I went ahead and disassembled encode_fsid() anyway. I did stumble across a "mov 0x4(%ecx), %edx" -- which turns out to be:

static __be32 *encode_fsid(__be32 *p, struct svc_fh *fhp)

Quoted 12 lines
{
        u64 f;
        switch(fsid_source(fhp)) {
        default:
        case FSIDSOURCE_DEV:
                p = xdr_encode_hyper(p, (u64)huge_encode_dev
                                (fhp->fh_dentry->d_inode->i_sb->s_dev));
                break;
        case FSIDSOURCE_FSID:
                p = xdr_encode_hyper(p, (u64) fhp->fh_export->ex_fsid);
                break;
        case FSIDSOURCE_UUID:

Whoops ...

Quoted 2 lines
                f = ((u64*)fhp->fh_export->ex_uuid)[0]; /* *** HERE *** */
                f ^= ((u64*)fhp->fh_export->ex_uuid)[1]; /* and not here */

Anyway, %ecx is fhp->fh_export->ex_uuid above. (NULL, therefore 0).

f is u64, so on 32-bit i386, we need to store it in two local registers. gcc does that by fetching 0x4(%ecx) into one register, and (%ecx) into another, both those combined is the first memory load into f.

For the second memory load, again it's a (u64 *) cast, so gcc will fetch 0xc(%ecx) and 0x8(%ecx) separately into two local registers.

And then xor them _separately_. (upper word with upper word, lower word with lower word)

Quoted 1 line
                p = xdr_encode_hyper(p, f);

These are probably the ntohl's -> bswap's.

Quoted 7 lines
                break;
        }
        return p;
}

Note that fhp->fh_export->ex_uuid is an unsigned char *, which is 4 bytes on an i386 (which is what David's system is).

So this wasn't quite right.

Quoted 4 lines

For some
reason fhp->fh_export->ex_uuid (%ecx) is NULL here, which leads to the oops. I have _zero_ other knowledge of knfsd code, and not really be of any other use, sorry.

But this was.

nfs3xdr.o: file format elf32-i386
Disassembly of section .text:
00000232 <encode_fsid>:

    232:	55                   	push   %ebp
    233:	89 e5                	mov    %esp,%ebp
    235:	57                   	push   %edi
    236:	56                   	push   %esi
    237:	89 c6                	mov    %eax,%esi
    239:	53                   	push   %ebx
    23a:	89 d0                	mov    %edx,%eax
    23c:	89 d3                	mov    %edx,%ebx
    23e:	e8 fc ff ff ff       	call   23f <encode_fsid+0xd>
    243:	83 f8 01             	cmp    $0x1,%eax
    246:	74 3e                	je     286 <encode_fsid+0x54>
    248:	83 f8 02             	cmp    $0x2,%eax
    24b:	8d 7e 08             	lea    0x8(%esi),%edi

Note the lea 0x8(%esi),%edi here.

These are the other cases below.

    24e:	74 56                	je     2a6 <encode_fsid+0x74>
    250:	8b 83 84 00 00 00    	mov    0x84(%ebx),%eax
    256:	31 db                	xor    %ebx,%ebx
    258:	8b 40 24             	mov    0x24(%eax),%eax
    25b:	8b 80 08 01 00 00    	mov    0x108(%eax),%eax
    261:	8b 50 08             	mov    0x8(%eax),%edx
    264:	89 d0                	mov    %edx,%eax
    266:	0f b6 ca             	movzbl %dl,%ecx
    269:	c1 e8 14             	shr    $0x14,%eax
    26c:	81 e2 00 ff 0f 00    	and    $0xfff00,%edx
    272:	c1 e0 08             	shl    $0x8,%eax
    275:	09 c1                	or     %eax,%ecx
    277:	89 d8                	mov    %ebx,%eax
    279:	c1 e2 0c             	shl    $0xc,%edx
    27c:	09 d1                	or     %edx,%ecx
    27e:	0f c8                	bswap  %eax
    280:	89 06                	mov    %eax,(%esi)
    282:	89 c8                	mov    %ecx,%eax
    284:	eb 3e                	jmp    2c4 <encode_fsid+0x92>
    286:	8b 83 88 00 00 00    	mov    0x88(%ebx),%eax
    28c:	8b 48 30             	mov    0x30(%eax),%ecx
    28f:	89 cb                	mov    %ecx,%ebx
    291:	c1 fb 1f             	sar    $0x1f,%ebx
    294:	89 d8                	mov    %ebx,%eax
    296:	0f c8                	bswap  %eax
    298:	89 06                	mov    %eax,(%esi)
    29a:	89 c8                	mov    %ecx,%eax
    29c:	0f c8                	bswap  %eax
    29e:	89 46 04             	mov    %eax,0x4(%esi)
    2a1:	8d 46 08             	lea    0x8(%esi),%eax
    2a4:	eb 25                	jmp    2cb <encode_fsid+0x99>

Okay, this is case FSIDSOURCE_UUID:

    2a6:	8b 83 88 00 00 00    	mov    0x88(%ebx),%eax

0x88 bytes is the offset of struct fh_export * in struct svc_fh.

    2ac:	8b 48 34             	mov    0x34(%eax),%ecx

0x34 bytes is the offset of ex_uuid in struct fh_export. %ecx == 0, which means ex_uuid was NULL.

    2af:	8b 51 04             	mov    0x4(%ecx),%edx

Upper word of: ((u64*)fhp->fh_export->ex_uuid)[0]

    2b2:	8b 59 0c             	mov    0xc(%ecx),%ebx

Upper word of: ((u64*)fhp->fh_export->ex_uuid)[1]

    2b5:	8b 01                	mov    (%ecx),%eax

Lower word of: ((u64*)fhp->fh_export->ex_uuid)[0]

    2b7:	8b 49 08             	mov    0x8(%ecx),%ecx

Lower word of: ((u64*)fhp->fh_export->ex_uuid)[1]

2ba: 31 da xor %ebx,%edx

Xor upper with upper.

2bc: 31 c8 xor %ecx,%eax

Xor lower with lower.

So now we finally have the "f ^= ..." thing in %eax and %edx. %eax is lower, %edx is the upper word of "f".

2be: 89 d1 mov %edx,%ecx

gcc moves %edx to %ecx just for kicks.
So now the upper word of "u64 f" is in %ecx.

Looks like xdr_encode_hyper got inlined below.

2c0: 0f c9 bswap %ecx

htonl(upperword)

    2c2:	89 0e                	mov    %ecx,(%esi)

Store that in *p.

2c4: 0f c8 bswap %eax

htonl(lowerword)

    2c6:	89 46 04             	mov    %eax,0x4(%esi)

p++ and store it there in *p.

    2c9:	89 f8                	mov    %edi,%eax

As we noted the "lea 0x8(%esi),%edi" up above, %edi is precisely the operation that does p++ as well as puts return value (__be32 *p) in %eax.

    2cb:	5b                   	pop    %ebx
    2cc:	5e                   	pop    %esi
    2cd:	5f                   	pop    %edi
    2ce:	5d                   	pop    %ebp
    2cf:	c3                   	ret

And we return.

Ahhh ... gcc generated some _beautiful_ code up there. Didn't find a single instruction that shouldn't have been ...

Satyam

#5 lore raw
Quoted 2 lines · On Thursday July 19, [email protected] wrote:

Ugh, not a good day for me today ... my earlier conclusion was right, but not the reasoning behind it ... hopefully this time I'll do better :-)

Looks good. Thanks for your helpful analysis.

Quoted 3 lines

0x34 bytes is the offset of ex_uuid in struct fh_export. %ecx == 0, which means ex_uuid was NULL.

Yup. That shouldn't happen, but I can see that it is possible.

There was another case were ex_uuid could conceivably be referenced while NULL that I fixed a little while ago. Looks like I need to fix this one too. Something like the following.

Thanks,
NeilBrown

### Diffstat output

 ./fs/nfsd/nfsfh.c |   20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)
fs/nfsd/nfsfh.c +15 -5
diff .prev/fs/nfsd/nfsfh.c ./fs/nfsd/nfsfh.c
@@ -566,13 +566,23 @@ enum fsid_source fsid_source(struct svc_
 	case FSID_DEV:
 	case FSID_ENCODE_DEV:
 	case FSID_MAJOR_MINOR:
-		return FSIDSOURCE_DEV;
+		if (fhp->fh_export->ex_dentry->d_inode->i_sb->s_type->fs_flags
+		    & FS_REQUIRES_DEV)
+			return FSIDSOURCE_DEV;
+		break;
 	case FSID_NUM:
-		return FSIDSOURCE_FSID;
-	default:
 		if (fhp->fh_export->ex_flags & NFSEXP_FSID)
 			return FSIDSOURCE_FSID;
-		else
-			return FSIDSOURCE_UUID;
+		break;
+	default:
+		break;
 	}
+	/* either a UUID type filehandle, or the filehandle doesn't
+	 * match the export.
+	 */
+	if (fhp->fh_export->ex_flags & NFSEXP_FSID)
+		return FSIDSOURCE_FSID;
+	if (fhp->fh_export->ex_uuid)
+		return FSIDSOURCE_UUID;
+	return FSIDSOURCE_DEV;
 }
#6 lore raw

Hi,

Quoted 6 lines · Neil Brown
Quoted 2 lines · On Thursday July 19, [email protected] wrote:

Ugh, not a good day for me today ... my earlier conclusion was right, but not the reasoning behind it ... hopefully this time I'll do better :-)

Looks good. Thanks for your helpful analysis.

Thanks, good to know I was of some help :-)

[ For pedantic correctness, of course, s/word/dword/ in the last mail, considering "word" size is still 16-bit on i386 -- but then that's the weird backward-compatibility nomenclature/behaviour of i386. ]

Quoted 41 lines
Quoted 2 lines

0x34 bytes is the offset of ex_uuid in struct fh_export. %ecx == 0, which means ex_uuid was NULL.

Yup. That shouldn't happen, but I can see that it is possible.

There was another case were ex_uuid could conceivably be referenced while NULL that I fixed a little while ago. Looks like I need to fix this one too. Something like the following.

Thanks,
NeilBrown

### Diffstat output

 ./fs/nfsd/nfsfh.c |   20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff .prev/fs/nfsd/nfsfh.c ./fs/nfsd/nfsfh.c

fs/nfsd/nfsfh.c 2007-07-20 12:53:36.000000000 +1000 renamed from fs/nfsd/nfsfh.c 2007-07-13 17:41:48.000000000 +1000 +7 -5
@@ -566,13 +566,23 @@ enum fsid_source fsid_source(struct svc_
        case FSID_DEV:
        case FSID_ENCODE_DEV:
        case FSID_MAJOR_MINOR:
-               return FSIDSOURCE_DEV;
+               if (fhp->fh_export->ex_dentry->d_inode->i_sb->s_type->fs_flags
+                   & FS_REQUIRES_DEV)
+                       return FSIDSOURCE_DEV;
+               break;
        case FSID_NUM:
-               return FSIDSOURCE_FSID;
-       default:
                if (fhp->fh_export->ex_flags & NFSEXP_FSID)
                        return FSIDSOURCE_FSID;
-               else
-                       return FSIDSOURCE_UUID;
+               break;
+       default:
+               break;

fs/nfsd/nfsfh.c 2007-07-20 12:53:36.000000000 +1000 enum fsid_source fsid_source(struct svc_ · line 588

Quoted 10 lines
fs/nfsd/nfsfh.c 2007-07-20 12:53:36.000000000 +1000 renamed from fs/nfsd/nfsfh.c 2007-07-13 17:41:48.000000000 +1000 +8 -0
        }
+       /* either a UUID type filehandle, or the filehandle doesn't
+        * match the export.
+        */
+       if (fhp->fh_export->ex_flags & NFSEXP_FSID)
+               return FSIDSOURCE_FSID;
+       if (fhp->fh_export->ex_uuid)
+               return FSIDSOURCE_UUID;
+       return FSIDSOURCE_DEV;
 }

David, please try this and let us know if it solves your problems.

Thanks,
Satyam

#7 lore raw

Le vendredi 20 juillet 2007 09:13, Satyam Sharma a écrit :

Quoted 1 line

David, please try this and let us know if it solves your problems.

How to ?
Should i have to patch kernel tree myself ? on 2.6.22.1 ?

For the moment, with 2.6.22.1 the probleme has not reappeared.

Best regards,

Signature · 1 line
David
#8 lore raw
Quoted 5 lines · David CHANIAL
Quoted 1 line · Satyam Sharma

David, please try this and let us know if it solves your problems.

How to ?
Should i have to patch kernel tree myself ? on 2.6.22.1 ?

Yes, you can apply the patch Neil just sent to your kernel, re-build, and test that.

Thanks.

#9 lore raw

Le vendredi 20 juillet 2007 15:36, Satyam Sharma a écrit :

Quoted 2 lines

Yes, you can apply the patch Neil just sent to your kernel, re-build, and test that.

Hi, I have no patched the kernel as asked by Neil, but i would notice that with 2.6.22.1 kernel it continue to happend :

http://internetworkpro.org/pastebin/765

I would patch the kernel now and wait to see...

Best regards,

Signature · 1 line
David
#10 lore raw

Hi David,

Quoted 5 lines · David CHANIAL

Le vendredi 20 juillet 2007 15:36, Satyam Sharma a ecrit:

Quoted 2 lines

Yes, you can apply the patch Neil just sent to your kernel, re-build, and test that.

Quoted 7 lines

Hi, I have no patched the kernel as asked by Neil, but i would notice that
with 2.6.22.1 kernel it continue to happend :

http://internetworkpro.org/pastebin/765

I would patch the kernel now and wait to see...

Did you test with that patch applied? Did you manage to hit the BUG again (with or without it)?

Neil, I don't see this one applied upstream as yet -- if this patch indeed solves the oops David saw again (as per the link he has posted) then I'd suggest it be pushed upstream (and -stable too).

Satyam

#11 lore raw

Le Tuesday 04 September 2007 01:55:57 Satyam Sharma, vous avez écrit :

Quoted 2 lines

Did you test with that patch applied? Did you manage to hit the BUG again (with or without it)?

I not have tested this patch.

I will test it as soon as possible (in few days).

Best regards,

Signature · 1 line
David