2.6.16: Oops - null ptr in blk_recount_segments?
Hey,
I've seen a few of these since upgrading to 2.6.16 on my x86_64 system:
[14513.188982] Unable to handle kernel NULL pointer dereference at 0000000000000007 RIP:
[14513.188988] <ffffffff8011d900>{page_to_pfn+0}
[14513.188998] PGD 3450d067 PUD 3450e067 PMD 0
[14513.189002] Oops: 0000 [1] SMP
[14513.189005] CPU 0
[14513.189007] Modules linked in: sha256 pktcdvd nfsd lockd sunrpc lp autofs4 thermal fan button processor ac battery ipv6 isofs aes dm_crypt raid5 xor dm_mod fuse usblp usbhid eth1394 snd_seq_dummy snd_seq_oss snd_seq_midi snd_seq_mid
i_event snd_seq snd_via82xx gameport snd_ac97_codec snd_ac97_bus snd_mpu401_uart evdev snd_rawmidi snd_seq_device ohci1394 tuner tvaudio msp3400 psmouse ieee1394 serio_raw floppy pcspkr parport_pc parport bttv video_buf ehci_hcd uhci_h
cd snd_bt87x firmware_class usbcore snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd compat_ioctl32 i2c_algo_bit soundcore v4l2_common btcx_risc ir_common tveeprom i2c_core snd_page_alloc videodev ide_cd cdrom unix
[14513.189040] Pid: 164, comm: pdflush Not tainted 2.6.16 #1
[14513.189044] RIP: 0010:[<ffffffff8011d900>] <ffffffff8011d900>{page_to_pfn+0}
[14513.189050] RSP: 0018:ffff81003f8b5740 EFLAGS: 00010246
[14513.189054] RAX: 0000000000000000 RBX: ffff81002b486c00 RCX: ffff81002b486c00
[14513.189057] RDX: 0000000000000000 RSI: ffff81003e510ec0 RDI: 0000000000000000
[14513.189060] RBP: 0000000000000000 R08: ffff81003e510b40 R09: 0000000000000040
[14513.189064] R10: 00000000ffffffff R11: 0000000000000002 R12: ffff81003e510ec0
[14513.189067] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[14513.189071] FS: 00002adb727dcae0(0000) GS:ffffffff8062c000(0000) knlGS:0000000000000000
[14513.189075] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[14513.189078] CR2: 0000000000000007 CR3: 000000003450c000 CR4: 00000000000006e0
[14513.189082] Process pdflush (pid: 164, threadinfo ffff81003f8b4000, task ffff81003f87b180)
[14513.189084] Stack: ffffffff80288e8d 0000000100362cdd ffffffff8045ab19 0000000000000000
[14513.189091] 0000000100200200 0000000000000000 0000000000000000 ffff81003f13f908
[14513.189096] ffff81003e510ec0 ffff81003f13f908
[14513.189101] Call Trace: <ffffffff80288e8d>{blk_recount_segments+125}
[14513.189113] <ffffffff8045ab19>{schedule_timeout+153} <ffffffff8017df90>{__bio_clone+128}
[14513.189134] <ffffffff8017dfed>{bio_clone+61} <ffffffff80144a70>{autoremove_wake_function+0}
[14513.189143] <ffffffff882576c1>{:dm_crypt:crypt_alloc_buffer+65}
[14513.189157] <ffffffff8825813e>{:dm_crypt:crypt_map+174} <ffffffff8823b463>{:dm_mod:__map_bio+83}
[14513.189190] <ffffffff8823b6a8>{:dm_mod:__clone_and_map+168} <ffffffff8823b8de>{:dm_mod:__split_bio+174}
[14513.189223] <ffffffff8823b9dc>{:dm_mod:dm_request+204} <ffffffff8028b532>{generic_make_request+258}
[14513.189243] <ffffffff8028b628>{submit_bio+216} <ffffffff8017e20f>{__bio_add_page+463}
[14513.189256] <ffffffff8026bd3e>{xfs_submit_ioend_bio+30} <ffffffff8026bec0>{xfs_submit_ioend+144}
[14513.189269] <ffffffff8026cd03>{xfs_page_state_convert+1379} <ffffffff8026d2e0>{linvfs_writepage+176}
[14513.189298] <ffffffff8019edc5>{mpage_writepages+469} <ffffffff8026e3ae>{xfs_buf_rele+62}
[14513.189309] <ffffffff8026d230>{linvfs_writepage+0} <ffffffff80159c5c>{do_writepages+44}
[14513.189336] <ffffffff8019d480>{__sync_single_inode+112} <ffffffff8019d7c1>{__writeback_single_inode+417}
[14513.189349] <ffffffff8823dbf5>{:dm_mod:dm_table_any_congested+21}
[14513.189364] <ffffffff8823bab8>{:dm_mod:dm_any_congested+72} <ffffffff8823dbf5>{:dm_mod:dm_table_any_congested+21}
[14513.189391] <ffffffff8019d9b2>{sync_sb_inodes+482} <ffffffff80144440>{keventd_create_kthread+0}
[14513.189403] <ffffffff8019db15>{writeback_inodes+133} <ffffffff80159a3e>{wb_kupdate+206}
[14513.189423] <ffffffff8015a4d0>{pdflush+0} <ffffffff8015a424>{__pdflush+292}
[14513.189431] <ffffffff8015a50a>{pdflush+58} <ffffffff80159970>{wb_kupdate+0}
[14513.189441] <ffffffff80144402>{kthread+146} <ffffffff8010bba2>{child_rip+8}
[14513.189452] <ffffffff80144440>{keventd_create_kthread+0} <ffffffff80144370>{kthread+0}
[14513.189467] <ffffffff8010bb9a>{child_rip+0}
[14513.189474]
[14513.189475] Code: 48 0f b6 47 07 48 8b 14 c5 80 1c 63 80 48 b8 b7 6d db b6 6d
[14513.189485] RIP <ffffffff8011d900>{page_to_pfn+0} RSP <ffff81003f8b5740>
[14513.189491] CR2: 0000000000000007
[14513.189493] Let me know if you need any additional information.
Cheers,
Peter
Signature · 4 lines
PGP signed and encrypted | .''`. ** Debian GNU/Linux **
messages preferred. | : :' : The universal
| `. `' Operating System
http://www.palfrader.org/ | `- http://www.debian.org/Quoted 3 lines · Peter Palfrader
Hey,
I've seen a few of these since upgrading to 2.6.16 on my
x86_64system:
What did you upgrade from?
....
Quoted 10 lines
[14513.189101] Call Trace: <ffffffff80288e8d>{blk_recount_segments+125}
[14513.189113] <ffffffff8045ab19>{schedule_timeout+153} <ffffffff8017df90>{__bio_clone+128}
[14513.189134] <ffffffff8017dfed>{bio_clone+61} <ffffffff80144a70>{autoremove_wake_function+0}
[14513.189143] <ffffffff882576c1>{:dm_crypt:crypt_alloc_buffer+65}
[14513.189157] <ffffffff8825813e>{:dm_crypt:crypt_map+174} <ffffffff8823b463>{:dm_mod:__map_bio+83}
[14513.189190] <ffffffff8823b6a8>{:dm_mod:__clone_and_map+168} <ffffffff8823b8de>{:dm_mod:__split_bio+174}
[14513.189223] <ffffffff8823b9dc>{:dm_mod:dm_request+204} <ffffffff8028b532>{generic_make_request+258}
[14513.189243] <ffffffff8028b628>{submit_bio+216} <ffffffff8017e20f>{__bio_add_page+463}
[14513.189256] <ffffffff8026bd3e>{xfs_submit_ioend_bio+30} <ffffffff8026bec0>{xfs_submit_ioend+144}
[14513.189269] <ffffffff8026cd03>{xfs_page_state_convert+1379} <ffffffff8026d2e0>{linvfs_writepage+176}This area of XFS changed in 2.6.16; it might be doing something wrong with bios that is blowing up later. Then again, it might be something in dm that is wrong. Does this happen on non-dmcrypt XFS filesystems you have?
I understand that the recognised way to find introduced a reproducable problem like this is to do a git bisect to find the mod that introduced the problem. Is it possible for you to do this?
Cheers,
Dave.
Signature · 3 lines
Dave Chinner R&D Software Enginner SGI Australian Software Group
Quoted 3 lines · David Chinner
Quoted 1 line
I've seen a few of these since upgrading to 2.6.16 on my
x86_64system:What did you upgrade from?
2.6.14.2.
Quoted 11 lines
Quoted 10 lines
[14513.189101] Call Trace: <ffffffff80288e8d>{blk_recount_segments+125} [14513.189113] <ffffffff8045ab19>{schedule_timeout+153} <ffffffff8017df90>{__bio_clone+128} [14513.189134] <ffffffff8017dfed>{bio_clone+61} <ffffffff80144a70>{autoremove_wake_function+0} [14513.189143] <ffffffff882576c1>{:dm_crypt:crypt_alloc_buffer+65} [14513.189157] <ffffffff8825813e>{:dm_crypt:crypt_map+174} <ffffffff8823b463>{:dm_mod:__map_bio+83} [14513.189190] <ffffffff8823b6a8>{:dm_mod:__clone_and_map+168} <ffffffff8823b8de>{:dm_mod:__split_bio+174} [14513.189223] <ffffffff8823b9dc>{:dm_mod:dm_request+204} <ffffffff8028b532>{generic_make_request+258} [14513.189243] <ffffffff8028b628>{submit_bio+216} <ffffffff8017e20f>{__bio_add_page+463} [14513.189256] <ffffffff8026bd3e>{xfs_submit_ioend_bio+30} <ffffffff8026bec0>{xfs_submit_ioend+144} [14513.189269] <ffffffff8026cd03>{xfs_page_state_convert+1379} <ffffffff8026d2e0>{linvfs_writepage+176}
Quoted 3 lines
This area of XFS changed in 2.6.16; it might be doing something wrong with bios that is blowing up later. Then again, it might be something in dm that is wrong. Does this happen on non-dmcrypt XFS filesystems you have?
All the Oopses so far had dm_crypt in their backtrace, but then the non-crypted XFSs don't see that much action. Also, I recently added a new XFS filesytem to the system. Is there a way to tell which of them causes the Oops?
Quoted 3 lines
I understand that the recognised way to find introduced a reproducable problem like this is to do a git bisect to find the mod that introduced the problem. Is it possible for you to do this?
I can certainly try but it usually takes several hours or a at least a few days to manifest here on my desktop.
Signature · 4 lines
PGP signed and encrypted | .''`. ** Debian GNU/Linux **
messages preferred. | : :' : The universal
| `. `' Operating System
http://www.palfrader.org/ | `- http://www.debian.org/Quoted 20 lines · Peter Palfrader
Quoted 3 lines · David Chinner
Quoted 1 line
I've seen a few of these since upgrading to 2.6.16 on my
x86_64system:What did you upgrade from?
2.6.14.2.
Quoted 11 lines
Quoted 10 lines
[14513.189101] Call Trace: <ffffffff80288e8d>{blk_recount_segments+125} [14513.189113] <ffffffff8045ab19>{schedule_timeout+153} <ffffffff8017df90>{__bio_clone+128} [14513.189134] <ffffffff8017dfed>{bio_clone+61} <ffffffff80144a70>{autoremove_wake_function+0} [14513.189143] <ffffffff882576c1>{:dm_crypt:crypt_alloc_buffer+65} [14513.189157] <ffffffff8825813e>{:dm_crypt:crypt_map+174} <ffffffff8823b463>{:dm_mod:__map_bio+83} [14513.189190] <ffffffff8823b6a8>{:dm_mod:__clone_and_map+168} <ffffffff8823b8de>{:dm_mod:__split_bio+174} [14513.189223] <ffffffff8823b9dc>{:dm_mod:dm_request+204} <ffffffff8028b532>{generic_make_request+258} [14513.189243] <ffffffff8028b628>{submit_bio+216} <ffffffff8017e20f>{__bio_add_page+463} [14513.189256] <ffffffff8026bd3e>{xfs_submit_ioend_bio+30} <ffffffff8026bec0>{xfs_submit_ioend+144} [14513.189269] <ffffffff8026cd03>{xfs_page_state_convert+1379} <ffffffff8026d2e0>{linvfs_writepage+176}
Quoted 8 lines
Quoted 3 lines
This area of XFS changed in 2.6.16; it might be doing something wrong with bios that is blowing up later. Then again, it might be something in dm that is wrong. Does this happen on non-dmcrypt XFS filesystems you have?
All the Oopses so far had
dm_cryptin their backtrace, but then the non-crypted XFSs don't see that much action. Also, I recently added a new XFS filesytem to the system. Is there a way to tell which of them causes the Oops?
The only way I know involves prodding the decomposing corpse with a long, pointy stick (i.e. kdb). That probably doesn't help you, though.
Quoted 6 lines
Quoted 3 lines
I understand that the recognised way to find introduced a reproducable problem like this is to do a git bisect to find the mod that introduced the problem. Is it possible for you to do this?
I can certainly try but it usually takes several hours or a at least a few days to manifest here on my desktop.
Well, you can try ruling out the XFS changes - on this page:
These diffs:
2006-01-18
[XFS] Fix a race in xfs_submit_ioend() where we can ... 2006-01-11
[XFS] fix writeback control handling fix a reversed ... [XFS] cluster rewrites We can cluster mapped pages ... [XFS] pass full 64bit offsets to xfs_add_to_ioend [XFS] consolidate some code in xfs_page_state_convert ... [XFS] various fixes for xfs_convert_page fix various ... [XFS] clean up the xfs_offset_to_map interface Currently ... [XFS] use pagevec lookups This reduces the time spend ... [XFS] Initial pass at going directly-to-bio on the ...
Are the ones that change the XFS I/O path from what is in 2.6.14. Try reverting them and see if the problem goes away....
Cheers,
Dave.
Signature · 3 lines
Dave Chinner R&D Software Enginner SGI Australian Software Group
Quoted 8 lines
These diffs:
2006-01-18
[XFS] Fix a race inxfs_submit_ioend()where we can ... 2006-01-11
[XFS] fix writeback control handling fix a reversed ... [XFS] cluster rewrites We can cluster mapped pages ... [...]
I bet on the 3rd...
Jan Engelhardt
Jan Engelhardt schrieb am Dienstag, dem 28. März 2006:
Quoted 10 lines
Quoted 8 lines
These diffs:
2006-01-18
[XFS] Fix a race inxfs_submit_ioend()where we can ... 2006-01-11
[XFS] fix writeback control handling fix a reversed ... [XFS] cluster rewrites We can cluster mapped pages ... [...]I bet on the 3rd...
Some of the patches don't unapply cleanly anymore. I'll see what I can do despite that.
Signature · 1 line
Peter
Quoted 11 lines · Peter Palfrader
Jan Engelhardt schrieb am Dienstag, dem 28. März 2006:
Quoted 9 lines
Quoted 8 lines
These diffs:
2006-01-18
[XFS] Fix a race inxfs_submit_ioend()where we can ... 2006-01-11
[XFS] fix writeback control handling fix a reversed ... [XFS] cluster rewrites We can cluster mapped pages ... [...]
Quoted 4 lines
Quoted 1 line
I bet on the 3rd...
Some of the patches don't unapply cleanly anymore. I'll see what I can do despite that.
You'll be better off trying the bio_clone fix discussed in the other (ext3-bio_clone-panic) thread than go down this route (there is a fix in 2.6.16.1 apparently - start there). Certainly try that before attempting to revert these changes anyway.
cheers.
Signature · 1 line
Nathan
Quoted 4 lines · Nathan Scott
You'll be better off trying the
bio_clonefix discussed in the other (ext3-bio_clone-panic) thread than go down this route (there is a fix in 2.6.16.1 apparently - start there). Certainly try that before attempting to revert these changes anyway.
The problem persists on 2.6.16.1.
lg,
Peter
Quoted 8 lines · Peter Palfrader
Quoted 4 lines · Nathan Scott
You'll be better off trying the
bio_clonefix discussed in the other (ext3-bio_clone-panic) thread than go down this route (there is a fix in 2.6.16.1 apparently - start there). Certainly try that before attempting to revert these changes anyway.The problem persists on 2.6.16.1.
The original oops showed the path into bio_clone() originating from crypt_alloc_buffer() in dm-crypt.c - and blk_recount_segments() is oopsinig in page_to_pfn(), meaning that the most likely cause of this is dm-crypt passing in a bio with invalid ->bi_idx and/or ->bi_vcnt. At least they are not matching what ->bi_io_vec[] holds.
Signature · 1 line
Jens Axboe
Jan Engelhardt schrieb am Dienstag, dem 28. März 2006:
Quoted 10 lines
Quoted 8 lines
These diffs:
2006-01-18
[XFS] Fix a race inxfs_submit_ioend()where we can ... 2006-01-11
[XFS] fix writeback control handling fix a reversed ... [XFS] cluster rewrites We can cluster mapped pages ... [...]I bet on the 3rd...
Just reverting #3[0] on 2.6.16 doesn't fix it either.
Peter, back on 2.6.14.2 for now.
0. one hunk didn't unapply cleanly, and the change was not obvious. I opted for making the beginning of xfs_probe_cluster() look like:
| /* First sum forwards in this page */
| do {
| if (!buffer_uptodate(bh) || (mapped != buffer_mapped(bh)))
| return total;
| total += bh->b_size;
| } while ((bh = bh->b_this_page) != head);