BUG: unable to handle kernel NULL pointer dereference - nfs v3
Hi,
I'm not sure is the good place to poste that, and if not - please excuse me.
I was running nfs server v2 since a year on one server, there is few days, i have update my kernel to 2.6.21.3 with support of nfsv3 server.
Somes times per days i have somes crash as below, needing i reboot the server to nfs re-become up.
************
BUG: unable to handle kernel NULL pointer dereference at virtual address
00000004
printing eip:c01e7279
*pde = 09ecc001
Oops: 0000 [#1] SMP CPU: 0 EIP: 0060:[<c01e7279>] Not tainted VLI EFLAGS: 00010246 (2.6.21.3-sdf88-core #9)
EIP is at encode_fsid+0x67/0x89
eax: e5bde8c0 ebx: f7593404 ecx: 00000000 edx: 00000006
esi: dc569048 edi: f75934ec ebp: f7593404 esp: f75f1f18
ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068
Process nfsd (pid: 3386, ti=f75f0000 task=f7b295b0 task.ti=f75f0000)
Stack: 00000000 dc569048 c01e7381 f5bd0df8 c01e5bde 00000000 f6c95000 c01e7fdf
dc56901c c0519584 c01e7ffc f75934ec f6c95000 c01dd38a f77d68c0 d0ab4200
c043d99e dc569014 f6c95000 f6c950e0 f7593538 c0519584 c0439f32 00000000
Call Trace:
[<c01e7381>] encode_fattr3+0xe6/0x131
[<c01e5bde>] nfsd3_proc_getattr+0xaf/0xb9
[<c01e7fdf>] nfs3svc_encode_attrstat+0x0/0x3b
[<c01e7ffc>] nfs3svc_encode_attrstat+0x1d/0x3b
[<c01dd38a>] nfsd_dispatch+0x13e/0x190
[<c043d99e>] svcauth_unix_set_client+0x152/0x15e
[<c0439f32>] svc_process+0x38d/0x636
[<c01dd14f>] nfsd+0x171/0x26e
[<c01dcfde>] nfsd+0x0/0x26e
[<c010312f>] kernel_thread_helper+0x7/0x10
=======================
Code: e2 08 09 d1 09 c1 eb 10 8b 83 88 00 00 00 8b 40 30 89 c3 89 c1 c1 fb 1f 89 d8 0f c8 89 06 89 c8 eb 1e
************
Best regards,
Signature · 1 line
David
Quoted 3 lines · On Monday July 16, [email protected] wrote:
Hi,
I'm not sure is the good place to poste that, and if not - please excuse me.
This is the correct place to post this, thanks.
Quoted 10 lines
I was running nfs server v2 since a year on one server, there is few days, i have update my kernel to 2.6.21.3 with support of nfsv3 server.
Somes times per days i have somes crash as below, needing i reboot the server to nfs re-become up.
************
BUG: unable to handle kernel NULL pointer dereference at virtual address00000004
^^^^^^^^
This says that it tried to access memory at address '4'. There is no memory there, so it caused the BUG.
Quoted 8 lines
printing eip:
c01e7279
*pde =09ecc001Oops: 0000 [#1] SMP CPU: 0 EIP: 0060:[<c01e7279>] Not tainted VLI EFLAGS: 00010246 (2.6.21.3-sdf88-core #9)
^^^^^^^^^^^
What is "-sdf88-core" ?? Are there any extra patches that we should know about?
Quoted 1 line
EIP is at
encode_fsid+0x67/0x89
This is presumably where the illegal access happened.
Quoted 2 lines
eax:
e5bde8c0ebx:f7593404ecx: 00000000 edx: 00000006
esi:dc569048edi:f75934ecebp:f7593404esp:f75f1f18
Memory accesses are (almost) always relative to the value in some register. Of these registers, the most likely is ecx, with edx a vague possibility.
Quoted 2 lines
Code: e2 08 09 d1 09 c1 eb 10 8b 83 88 00 00 00 8b 40 30 89 c3 89 c1 c1 fb 1f89 d8 0f c8 89 06 89 c8 eb 1e
Unfortunately "ksymoops" does seem to decode this into something quite useful enough. Normally one of the numbers has <> around it. Are you should you copied the number across exactly?
This code decodes as:
0: e2 08 loop a <_EIP+0xa> 2: 09 d1 or %edx,%ecx 4: 09 c1 or %eax,%ecx 6: eb 10 jmp 18 <_EIP+0x18> 8: 8b 83 88 00 00 00 mov 0x88(%ebx),%eax e: 8b 40 30 mov 0x30(%eax),%eax 11: 89 c3 mov %eax,%ebx
....
From the 'jmp' onwards, that is what I would expect to see in encode_fsid. The code before there doesn't make a lot of sense, so it is hard to pinpoint exactly there the error is.
In any case, there is no place in encode_fsid where an offset of 4 from any register is indexed, nor an offset of -2.
So either there is something wrong with the decoding and displaying of this information, or there is something very wrong with your hardware.
I would suggest:
1/ if possible, run memtest86 on the machine for a while, to make
sure there isn't a problem with the memory.
2/ If the problem happens again, post another report with all the
"oops" information again. Maybe the next time it will be slightly different and will make more sense in some way.
NeilBrown
Le mardi 17 juillet 2007 13:13, Neil Brown a écrit :
Quoted 2 lines
What is "-sdf88-core" ?? Are there any extra patches that we should know about?
I have upgraded to 2.6.22.1-sdf90-intel
-sdf90 is a set of a configuration with one little patch :
http://www.ssi.bg/~ja/#hidden
intel or core juste say for wich CPU it was compiled (the previous kernel "core" was named-error, the kernel really was for a intel (and not core2) cpu)
Quoted 5 lines
1/ if possible, run memtest86 on the machine for a while, to make sure there isn't a problem with the memory.
2/ If the problem happens again, post another report with all the "oops" information again. Maybe the next time it will be slightly different and will make more sense in some way.
Since the kernel update, no crash anymore - for the moment -. If it re-crash i will floow this advices.
Thanks Niel,
Signature · 1 line
David
Hi Neil,
[ okay, just searching through my lkml folder looking for "unable to handle" :-) ]
Quoted 12 lines · Neil Brown
Quoted 6 lines · On Monday July 16, [email protected] wrote:
************
BUG: unable to handle kernel NULL pointer dereference at virtual address00000004
EIP is at
encode_fsid+0x67/0x89This is presumably where the illegal access happened.
Quoted 2 lines
eax:
e5bde8c0ebx:f7593404ecx: 00000000 edx: 00000006
esi:dc569048edi:f75934ecebp:f7593404esp:f75f1f18
Yup, ecx is to blame here ...
Quoted 6 lines
Quoted 2 lines
Code: e2 08 09 d1 09 c1 eb 10 8b 83 88 00 00 00 8b 40 30 89 c3 89 c1 c1 fb 1f89 d8 0f c8 89 06 89 c8 eb 1e
Unfortunately "ksymoops" does seem to decode this into something quite useful enough. Normally one of the numbers has <> around it. Are you should you copied the number across exactly?
Yes, I think David missed posting the full "Code:" here. Unfortunate.
Quoted 2 lines
In any case, there is no place in
encode_fsidwhere an offset of 4 from any register is indexed, nor an offset of -2.
But I went ahead and disassembled encode_fsid() anyway. I did stumble across a "mov 0x4(%ecx), %edx" -- which turns out to be:
static __be32 *encode_fsid(__be32 *p, struct svc_fh *fhp)
{
u64 f;
switch(fsid_source(fhp)) {
default:
case FSIDSOURCE_DEV:
p = xdr_encode_hyper(p, (u64)huge_encode_dev
(fhp->fh_dentry->d_inode->i_sb->s_dev));
break;
case FSIDSOURCE_FSID:
p = xdr_encode_hyper(p, (u64) fhp->fh_export->ex_fsid);
break;
case FSIDSOURCE_UUID:
f = ((u64*)fhp->fh_export->ex_uuid)[0];
f ^= ((u64*)fhp->fh_export->ex_uuid)[1]; /* ***** HERE ***** */
p = xdr_encode_hyper(p, f);
break;
}
return p;
}Note that fhp->fh_export->ex_uuid is an unsigned char *, which is 4 bytes on an i386 (which is what David's system is). For some reason fhp->fh_export->ex_uuid (%ecx) is NULL here, which leads to the oops. I have _zero_ other knowledge of knfsd code, and not really be of any other use, sorry.
Satyam
Ugh, not a good day for me today ... my earlier conclusion was right, but not the reasoning behind it ... hopefully this time I'll do better :-)
Quoted 37 lines · Satyam Sharma
Hi Neil,
[ okay, just searching through my lkml folder looking for "unable to handle" :-) ]
Quoted 12 lines · Neil Brown
Quoted 6 lines · On Monday July 16, [email protected] wrote:
************
BUG: unable to handle kernel NULL pointer dereference at virtual address00000004
EIP is at
encode_fsid+0x67/0x89This is presumably where the illegal access happened.
Quoted 2 lines
eax:
e5bde8c0ebx:f7593404ecx: 00000000 edx: 00000006
esi:dc569048edi:f75934ecebp:f7593404esp:f75f1f18Yup, ecx is to blame here ...
Quoted 6 lines
Quoted 2 lines
Code: e2 08 09 d1 09 c1 eb 10 8b 83 88 00 00 00 8b 40 30 89 c3 89 c1 c1 fb 1f89 d8 0f c8 89 06 89 c8 eb 1e
Unfortunately "ksymoops" does seem to decode this into something quite useful enough. Normally one of the numbers has <> around it. Are you should you copied the number across exactly?
Yes, I think David missed posting the full "Code:" here. Unfortunate.
Quoted 2 lines
In any case, there is no place in
encode_fsidwhere an offset of 4 from any register is indexed, nor an offset of -2.But I went ahead and disassembled
encode_fsid()anyway. I did stumble across a "mov 0x4(%ecx), %edx" -- which turns out to be:static
__be32*encode_fsid(__be32 *p, structsvc_fh*fhp)
Quoted 12 lines
{
u64 f;
switch(fsid_source(fhp)) {
default:
case FSIDSOURCE_DEV:
p = xdr_encode_hyper(p, (u64)huge_encode_dev
(fhp->fh_dentry->d_inode->i_sb->s_dev));
break;
case FSIDSOURCE_FSID:
p = xdr_encode_hyper(p, (u64) fhp->fh_export->ex_fsid);
break;
case FSIDSOURCE_UUID:Whoops ...
Quoted 2 lines
f = ((u64*)fhp->fh_export->ex_uuid)[0]; /* *** HERE *** */
f ^= ((u64*)fhp->fh_export->ex_uuid)[1]; /* and not here */Anyway, %ecx is fhp->fh_export->ex_uuid above. (NULL, therefore 0).
f is u64, so on 32-bit i386, we need to store it in two local registers. gcc does that by fetching 0x4(%ecx) into one register, and (%ecx) into another, both those combined is the first memory load into f.
For the second memory load, again it's a (u64 *) cast, so gcc will fetch 0xc(%ecx) and 0x8(%ecx) separately into two local registers.
And then xor them _separately_. (upper word with upper word, lower word with lower word)
Quoted 1 line
p = xdr_encode_hyper(p, f);
These are probably the ntohl's -> bswap's.
Quoted 7 lines
break; } return p; }Note that
fhp->fh_export->ex_uuidis an unsigned char *, which is 4 bytes on an i386 (which is what David's system is).
So this wasn't quite right.
Quoted 4 lines
For some
reasonfhp->fh_export->ex_uuid(%ecx) is NULL here, which leads to the oops. I have _zero_ other knowledge of knfsd code, and not really be of any other use, sorry.
But this was.
nfs3xdr.o: file format elf32-i386
Disassembly of section .text:
00000232 <encode_fsid>:
232: 55 push %ebp
233: 89 e5 mov %esp,%ebp
235: 57 push %edi
236: 56 push %esi
237: 89 c6 mov %eax,%esi
239: 53 push %ebx
23a: 89 d0 mov %edx,%eax
23c: 89 d3 mov %edx,%ebx
23e: e8 fc ff ff ff call 23f <encode_fsid+0xd>
243: 83 f8 01 cmp $0x1,%eax
246: 74 3e je 286 <encode_fsid+0x54>
248: 83 f8 02 cmp $0x2,%eax
24b: 8d 7e 08 lea 0x8(%esi),%ediNote the lea 0x8(%esi),%edi here.
These are the other cases below.
24e: 74 56 je 2a6 <encode_fsid+0x74>
250: 8b 83 84 00 00 00 mov 0x84(%ebx),%eax
256: 31 db xor %ebx,%ebx
258: 8b 40 24 mov 0x24(%eax),%eax
25b: 8b 80 08 01 00 00 mov 0x108(%eax),%eax
261: 8b 50 08 mov 0x8(%eax),%edx
264: 89 d0 mov %edx,%eax
266: 0f b6 ca movzbl %dl,%ecx
269: c1 e8 14 shr $0x14,%eax
26c: 81 e2 00 ff 0f 00 and $0xfff00,%edx
272: c1 e0 08 shl $0x8,%eax
275: 09 c1 or %eax,%ecx
277: 89 d8 mov %ebx,%eax
279: c1 e2 0c shl $0xc,%edx
27c: 09 d1 or %edx,%ecx
27e: 0f c8 bswap %eax
280: 89 06 mov %eax,(%esi)
282: 89 c8 mov %ecx,%eax
284: eb 3e jmp 2c4 <encode_fsid+0x92>
286: 8b 83 88 00 00 00 mov 0x88(%ebx),%eax
28c: 8b 48 30 mov 0x30(%eax),%ecx
28f: 89 cb mov %ecx,%ebx
291: c1 fb 1f sar $0x1f,%ebx
294: 89 d8 mov %ebx,%eax
296: 0f c8 bswap %eax
298: 89 06 mov %eax,(%esi)
29a: 89 c8 mov %ecx,%eax
29c: 0f c8 bswap %eax
29e: 89 46 04 mov %eax,0x4(%esi)
2a1: 8d 46 08 lea 0x8(%esi),%eax
2a4: eb 25 jmp 2cb <encode_fsid+0x99>Okay, this is case FSIDSOURCE_UUID:
2a6: 8b 83 88 00 00 00 mov 0x88(%ebx),%eax
0x88 bytes is the offset of struct fh_export * in struct svc_fh.
2ac: 8b 48 34 mov 0x34(%eax),%ecx
0x34 bytes is the offset of ex_uuid in struct fh_export. %ecx == 0, which means ex_uuid was NULL.
2af: 8b 51 04 mov 0x4(%ecx),%edx
Upper word of: ((u64*)fhp->fh_export->ex_uuid)[0]
2b2: 8b 59 0c mov 0xc(%ecx),%ebx
Upper word of: ((u64*)fhp->fh_export->ex_uuid)[1]
2b5: 8b 01 mov (%ecx),%eax
Lower word of: ((u64*)fhp->fh_export->ex_uuid)[0]
2b7: 8b 49 08 mov 0x8(%ecx),%ecx
Lower word of: ((u64*)fhp->fh_export->ex_uuid)[1]
2ba: 31 da xor %ebx,%edx
Xor upper with upper.
2bc: 31 c8 xor %ecx,%eax
Xor lower with lower.
So now we finally have the "f ^= ..." thing in %eax and %edx. %eax is lower, %edx is the upper word of "f".
2be: 89 d1 mov %edx,%ecx
gcc moves %edx to %ecx just for kicks.
So now the upper word of "u64 f" is in %ecx.
Looks like xdr_encode_hyper got inlined below.
2c0: 0f c9 bswap %ecx
htonl(upperword)
2c2: 89 0e mov %ecx,(%esi)
Store that in *p.
2c4: 0f c8 bswap %eax
htonl(lowerword)
2c6: 89 46 04 mov %eax,0x4(%esi)
p++ and store it there in *p.
2c9: 89 f8 mov %edi,%eax
As we noted the "lea 0x8(%esi),%edi" up above, %edi is precisely the operation that does p++ as well as puts return value (__be32 *p) in %eax.
2cb: 5b pop %ebx
2cc: 5e pop %esi
2cd: 5f pop %edi
2ce: 5d pop %ebp
2cf: c3 retAnd we return.
Ahhh ... gcc generated some _beautiful_ code up there. Didn't find a single instruction that shouldn't have been ...
Satyam
Quoted 2 lines · On Thursday July 19, [email protected] wrote:
Ugh, not a good day for me today ... my earlier conclusion was right, but not the reasoning behind it ... hopefully this time I'll do better :-)
Looks good. Thanks for your helpful analysis.
Quoted 3 lines
0x34bytes is the offset ofex_uuidin structfh_export. %ecx == 0, which meansex_uuidwas NULL.
Yup. That shouldn't happen, but I can see that it is possible.
There was another case were ex_uuid could conceivably be referenced while NULL that I fixed a little while ago. Looks like I need to fix this one too. Something like the following.
Thanks,
NeilBrown
### Diffstat output
./fs/nfsd/nfsfh.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
fs/nfsd/nfsfh.c +15 -5
@@ -566,13 +566,23 @@ enum fsid_source fsid_source(struct svc_case FSID_DEV :case FSID_ENCODE_DEV :case FSID_MAJOR_MINOR : -return FSIDSOURCE_DEV ; +if (fhp->fh_export->ex_dentry->d_inode->i_sb->s_type->fs_flags + &FS_REQUIRES_DEV ) +return FSIDSOURCE_DEV ; +break ;case FSID_NUM : -return FSIDSOURCE_FSID ; -default :if (fhp->fh_export->ex_flags &NFSEXP_FSID )return FSIDSOURCE_FSID ; -else -return FSIDSOURCE_UUID ; +break ; +default : +break ; } +/* either a UUID type filehandle, or the filehandle doesn't +* match the export. +*/ +if (fhp->fh_export->ex_flags &NFSEXP_FSID ) +return FSIDSOURCE_FSID ; +if (fhp->fh_export->ex_uuid) +return FSIDSOURCE_UUID ; +return FSIDSOURCE_DEV ; }
Hi,
Quoted 6 lines · Neil Brown
Quoted 2 lines · On Thursday July 19, [email protected] wrote:
Ugh, not a good day for me today ... my earlier conclusion was right, but not the reasoning behind it ... hopefully this time I'll do better :-)
Looks good. Thanks for your helpful analysis.
Thanks, good to know I was of some help :-)
[ For pedantic correctness, of course, s/word/dword/ in the last mail, considering "word" size is still 16-bit on i386 -- but then that's the weird backward-compatibility nomenclature/behaviour of i386. ]
Quoted 41 lines
Quoted 2 lines
0x34bytes is the offset ofex_uuidin structfh_export. %ecx == 0, which meansex_uuidwas NULL.Yup. That shouldn't happen, but I can see that it is possible.
There was another case were
ex_uuidcould conceivably be referenced while NULL that I fixed a little while ago. Looks like I need to fix this one too. Something like the following.Thanks,
NeilBrownSigned-off-by Neil Brown <[email protected]>### Diffstat output
./fs/nfsd/nfsfh.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)diff
.prev/fs/nfsd/nfsfh.c./fs/nfsd/nfsfh.cfs/nfsd/nfsfh.c 2007-07-20 12:53:36.000000000 +1000 renamed from fs/nfsd/nfsfh.c 2007-07-13 17:41:48.000000000 +1000 +7
-5@@ -566,13 +566,23 @@ enum fsid_source fsid_source(struct svc_ case FSID_DEV: case FSID_ENCODE_DEV: case FSID_MAJOR_MINOR: - return FSIDSOURCE_DEV; + if (fhp->fh_export->ex_dentry->d_inode->i_sb->s_type->fs_flags + & FS_REQUIRES_DEV) + return FSIDSOURCE_DEV; + break; case FSID_NUM: - return FSIDSOURCE_FSID; - default: if (fhp->fh_export->ex_flags & NFSEXP_FSID) return FSIDSOURCE_FSID; - else - return FSIDSOURCE_UUID; + break; + default: + break;
fs/nfsd/nfsfh.c 2007-07-20 12:53:36.000000000 +1000 enum fsid_source fsid_source(struct svc_ · line 588
Quoted 10 lines
fs/nfsd/nfsfh.c 2007-07-20 12:53:36.000000000 +1000 renamed from fs/nfsd/nfsfh.c 2007-07-13 17:41:48.000000000 +1000 +8
-0} + /* either a UUID type filehandle, or the filehandle doesn't + * match the export. + */ + if (fhp->fh_export->ex_flags & NFSEXP_FSID) + return FSIDSOURCE_FSID; + if (fhp->fh_export->ex_uuid) + return FSIDSOURCE_UUID; + return FSIDSOURCE_DEV; }
David, please try this and let us know if it solves your problems.
Thanks,
Satyam
Le vendredi 20 juillet 2007 09:13, Satyam Sharma a écrit :
Quoted 1 line
David, please try this and let us know if it solves your problems.
How to ?
Should i have to patch kernel tree myself ? on 2.6.22.1 ?
For the moment, with 2.6.22.1 the probleme has not reappeared.
Best regards,
Signature · 1 line
David
Quoted 5 lines · David CHANIAL
Quoted 1 line · Satyam Sharma
David, please try this and let us know if it solves your problems.
How to ?
Should i have to patch kernel tree myself ? on 2.6.22.1 ?
Yes, you can apply the patch Neil just sent to your kernel, re-build, and test that.
Thanks.
Le vendredi 20 juillet 2007 15:36, Satyam Sharma a écrit :
Quoted 2 lines
Yes, you can apply the patch Neil just sent to your kernel, re-build, and test that.
Hi, I have no patched the kernel as asked by Neil, but i would notice that with 2.6.22.1 kernel it continue to happend :
http://internetworkpro.org/pastebin/765
I would patch the kernel now and wait to see...
Best regards,
Signature · 1 line
David
Hi David,
Quoted 5 lines · David CHANIAL
Le vendredi 20 juillet 2007 15:36, Satyam Sharma a ecrit:
Quoted 2 lines
Yes, you can apply the patch Neil just sent to your kernel, re-build, and test that.
Quoted 7 lines
Hi, I have no patched the kernel as asked by Neil, but i would notice that
with 2.6.22.1 kernel it continue to happend :http://internetworkpro.org/pastebin/765
I would patch the kernel now and wait to see...
Did you test with that patch applied? Did you manage to hit the BUG again (with or without it)?
Neil, I don't see this one applied upstream as yet -- if this patch indeed solves the oops David saw again (as per the link he has posted) then I'd suggest it be pushed upstream (and -stable too).
Satyam
Le Tuesday 04 September 2007 01:55:57 Satyam Sharma, vous avez écrit :
Quoted 2 lines
Did you test with that patch applied? Did you manage to hit the BUG again (with or without it)?
I not have tested this patch.
I will test it as soon as possible (in few days).
Best regards,
Signature · 1 line
David