Skip to contentSkip to conversation

2.6.16: Oops - null ptr in blk_recount_segments?

DISCUSSION Peter Palfrader 5 participants 10 messages Mar 27, 2006 – Mar 29, 2006

lore
Share & tools
lore
#0 lore raw

Hey,

I've seen a few of these since upgrading to 2.6.16 on my x86_64 system:

[14513.188982] Unable to handle kernel NULL pointer dereference at 0000000000000007 RIP: 
[14513.188988] <ffffffff8011d900>{page_to_pfn+0}
[14513.188998] PGD 3450d067 PUD 3450e067 PMD 0 
[14513.189002] Oops: 0000 [1] SMP 
[14513.189005] CPU 0 
[14513.189007] Modules linked in: sha256 pktcdvd nfsd lockd sunrpc lp autofs4 thermal fan button processor ac battery ipv6 isofs aes dm_crypt raid5 xor dm_mod fuse usblp usbhid eth1394 snd_seq_dummy snd_seq_oss snd_seq_midi snd_seq_mid
i_event snd_seq snd_via82xx gameport snd_ac97_codec snd_ac97_bus snd_mpu401_uart evdev snd_rawmidi snd_seq_device ohci1394 tuner tvaudio msp3400 psmouse ieee1394 serio_raw floppy pcspkr parport_pc parport bttv video_buf ehci_hcd uhci_h
cd snd_bt87x firmware_class usbcore snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd compat_ioctl32 i2c_algo_bit soundcore v4l2_common btcx_risc ir_common tveeprom i2c_core snd_page_alloc videodev ide_cd cdrom unix
[14513.189040] Pid: 164, comm: pdflush Not tainted 2.6.16 #1
[14513.189044] RIP: 0010:[<ffffffff8011d900>] <ffffffff8011d900>{page_to_pfn+0}
[14513.189050] RSP: 0018:ffff81003f8b5740  EFLAGS: 00010246
[14513.189054] RAX: 0000000000000000 RBX: ffff81002b486c00 RCX: ffff81002b486c00
[14513.189057] RDX: 0000000000000000 RSI: ffff81003e510ec0 RDI: 0000000000000000
[14513.189060] RBP: 0000000000000000 R08: ffff81003e510b40 R09: 0000000000000040
[14513.189064] R10: 00000000ffffffff R11: 0000000000000002 R12: ffff81003e510ec0
[14513.189067] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[14513.189071] FS:  00002adb727dcae0(0000) GS:ffffffff8062c000(0000) knlGS:0000000000000000
[14513.189075] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[14513.189078] CR2: 0000000000000007 CR3: 000000003450c000 CR4: 00000000000006e0
[14513.189082] Process pdflush (pid: 164, threadinfo ffff81003f8b4000, task ffff81003f87b180)
[14513.189084] Stack: ffffffff80288e8d 0000000100362cdd ffffffff8045ab19 0000000000000000 
[14513.189091]        0000000100200200 0000000000000000 0000000000000000 ffff81003f13f908 
[14513.189096]        ffff81003e510ec0 ffff81003f13f908 
[14513.189101] Call Trace: <ffffffff80288e8d>{blk_recount_segments+125}
[14513.189113]        <ffffffff8045ab19>{schedule_timeout+153} <ffffffff8017df90>{__bio_clone+128}
[14513.189134]        <ffffffff8017dfed>{bio_clone+61} <ffffffff80144a70>{autoremove_wake_function+0}
[14513.189143]        <ffffffff882576c1>{:dm_crypt:crypt_alloc_buffer+65}
[14513.189157]        <ffffffff8825813e>{:dm_crypt:crypt_map+174} <ffffffff8823b463>{:dm_mod:__map_bio+83}
[14513.189190]        <ffffffff8823b6a8>{:dm_mod:__clone_and_map+168} <ffffffff8823b8de>{:dm_mod:__split_bio+174}
[14513.189223]        <ffffffff8823b9dc>{:dm_mod:dm_request+204} <ffffffff8028b532>{generic_make_request+258}
[14513.189243]        <ffffffff8028b628>{submit_bio+216} <ffffffff8017e20f>{__bio_add_page+463}
[14513.189256]        <ffffffff8026bd3e>{xfs_submit_ioend_bio+30} <ffffffff8026bec0>{xfs_submit_ioend+144}
[14513.189269]        <ffffffff8026cd03>{xfs_page_state_convert+1379} <ffffffff8026d2e0>{linvfs_writepage+176}
[14513.189298]        <ffffffff8019edc5>{mpage_writepages+469} <ffffffff8026e3ae>{xfs_buf_rele+62}
[14513.189309]        <ffffffff8026d230>{linvfs_writepage+0} <ffffffff80159c5c>{do_writepages+44}
[14513.189336]        <ffffffff8019d480>{__sync_single_inode+112} <ffffffff8019d7c1>{__writeback_single_inode+417}
[14513.189349]        <ffffffff8823dbf5>{:dm_mod:dm_table_any_congested+21}
[14513.189364]        <ffffffff8823bab8>{:dm_mod:dm_any_congested+72} <ffffffff8823dbf5>{:dm_mod:dm_table_any_congested+21}
[14513.189391]        <ffffffff8019d9b2>{sync_sb_inodes+482} <ffffffff80144440>{keventd_create_kthread+0}
[14513.189403]        <ffffffff8019db15>{writeback_inodes+133} <ffffffff80159a3e>{wb_kupdate+206}
[14513.189423]        <ffffffff8015a4d0>{pdflush+0} <ffffffff8015a424>{__pdflush+292}
[14513.189431]        <ffffffff8015a50a>{pdflush+58} <ffffffff80159970>{wb_kupdate+0}
[14513.189441]        <ffffffff80144402>{kthread+146} <ffffffff8010bba2>{child_rip+8}
[14513.189452]        <ffffffff80144440>{keventd_create_kthread+0} <ffffffff80144370>{kthread+0}
[14513.189467]        <ffffffff8010bb9a>{child_rip+0}
[14513.189474] 
[14513.189475] Code: 48 0f b6 47 07 48 8b 14 c5 80 1c 63 80 48 b8 b7 6d db b6 6d 
[14513.189485] RIP <ffffffff8011d900>{page_to_pfn+0} RSP <ffff81003f8b5740>
[14513.189491] CR2: 0000000000000007
[14513.189493]  

Let me know if you need any additional information.

Cheers,
Peter

Signature · 4 lines
 PGP signed and encrypted  |  .''`.  ** Debian GNU/Linux **
    messages preferred.    | : :' :      The  universal
                           | `. `'      Operating System
 http://www.palfrader.org/ |   `-    http://www.debian.org/
#1 lore raw
Quoted 3 lines · Peter Palfrader

Hey,

I've seen a few of these since upgrading to 2.6.16 on my x86_64 system:

What did you upgrade from?

....

Quoted 10 lines
[14513.189101] Call Trace: <ffffffff80288e8d>{blk_recount_segments+125}
[14513.189113]        <ffffffff8045ab19>{schedule_timeout+153} <ffffffff8017df90>{__bio_clone+128}
[14513.189134]        <ffffffff8017dfed>{bio_clone+61} <ffffffff80144a70>{autoremove_wake_function+0}
[14513.189143]        <ffffffff882576c1>{:dm_crypt:crypt_alloc_buffer+65}
[14513.189157]        <ffffffff8825813e>{:dm_crypt:crypt_map+174} <ffffffff8823b463>{:dm_mod:__map_bio+83}
[14513.189190]        <ffffffff8823b6a8>{:dm_mod:__clone_and_map+168} <ffffffff8823b8de>{:dm_mod:__split_bio+174}
[14513.189223]        <ffffffff8823b9dc>{:dm_mod:dm_request+204} <ffffffff8028b532>{generic_make_request+258}
[14513.189243]        <ffffffff8028b628>{submit_bio+216} <ffffffff8017e20f>{__bio_add_page+463}
[14513.189256]        <ffffffff8026bd3e>{xfs_submit_ioend_bio+30} <ffffffff8026bec0>{xfs_submit_ioend+144}
[14513.189269]        <ffffffff8026cd03>{xfs_page_state_convert+1379} <ffffffff8026d2e0>{linvfs_writepage+176}

This area of XFS changed in 2.6.16; it might be doing something wrong with bios that is blowing up later. Then again, it might be something in dm that is wrong. Does this happen on non-dmcrypt XFS filesystems you have?

I understand that the recognised way to find introduced a reproducable problem like this is to do a git bisect to find the mod that introduced the problem. Is it possible for you to do this?

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
R&D Software Enginner
SGI Australian Software Group
#2 lore raw
Quoted 3 lines · David Chinner
Quoted 1 line

I've seen a few of these since upgrading to 2.6.16 on my x86_64 system:

What did you upgrade from?

2.6.14.2.

Quoted 11 lines
Quoted 10 lines
[14513.189101] Call Trace: <ffffffff80288e8d>{blk_recount_segments+125}
[14513.189113]        <ffffffff8045ab19>{schedule_timeout+153} <ffffffff8017df90>{__bio_clone+128}
[14513.189134]        <ffffffff8017dfed>{bio_clone+61} <ffffffff80144a70>{autoremove_wake_function+0}
[14513.189143]        <ffffffff882576c1>{:dm_crypt:crypt_alloc_buffer+65}
[14513.189157]        <ffffffff8825813e>{:dm_crypt:crypt_map+174} <ffffffff8823b463>{:dm_mod:__map_bio+83}
[14513.189190]        <ffffffff8823b6a8>{:dm_mod:__clone_and_map+168} <ffffffff8823b8de>{:dm_mod:__split_bio+174}
[14513.189223]        <ffffffff8823b9dc>{:dm_mod:dm_request+204} <ffffffff8028b532>{generic_make_request+258}
[14513.189243]        <ffffffff8028b628>{submit_bio+216} <ffffffff8017e20f>{__bio_add_page+463}
[14513.189256]        <ffffffff8026bd3e>{xfs_submit_ioend_bio+30} <ffffffff8026bec0>{xfs_submit_ioend+144}
[14513.189269]        <ffffffff8026cd03>{xfs_page_state_convert+1379} <ffffffff8026d2e0>{linvfs_writepage+176}
Quoted 3 lines

This area of XFS changed in 2.6.16; it might be doing something wrong with bios that is blowing up later. Then again, it might be something in dm that is wrong. Does this happen on non-dmcrypt XFS filesystems you have?

All the Oopses so far had dm_crypt in their backtrace, but then the non-crypted XFSs don't see that much action. Also, I recently added a new XFS filesytem to the system. Is there a way to tell which of them causes the Oops?

Quoted 3 lines

I understand that the recognised way to find introduced a reproducable problem like this is to do a git bisect to find the mod that introduced the problem. Is it possible for you to do this?

I can certainly try but it usually takes several hours or a at least a few days to manifest here on my desktop.

Signature · 4 lines
 PGP signed and encrypted  |  .''`.  ** Debian GNU/Linux **
    messages preferred.    | : :' :      The  universal
                           | `. `'      Operating System
 http://www.palfrader.org/ |   `-    http://www.debian.org/
#3 lore raw
Quoted 20 lines · Peter Palfrader
Quoted 3 lines · David Chinner
Quoted 1 line

I've seen a few of these since upgrading to 2.6.16 on my x86_64 system:

What did you upgrade from?

2.6.14.2.

Quoted 11 lines
Quoted 10 lines
[14513.189101] Call Trace: <ffffffff80288e8d>{blk_recount_segments+125}
[14513.189113]        <ffffffff8045ab19>{schedule_timeout+153} <ffffffff8017df90>{__bio_clone+128}
[14513.189134]        <ffffffff8017dfed>{bio_clone+61} <ffffffff80144a70>{autoremove_wake_function+0}
[14513.189143]        <ffffffff882576c1>{:dm_crypt:crypt_alloc_buffer+65}
[14513.189157]        <ffffffff8825813e>{:dm_crypt:crypt_map+174} <ffffffff8823b463>{:dm_mod:__map_bio+83}
[14513.189190]        <ffffffff8823b6a8>{:dm_mod:__clone_and_map+168} <ffffffff8823b8de>{:dm_mod:__split_bio+174}
[14513.189223]        <ffffffff8823b9dc>{:dm_mod:dm_request+204} <ffffffff8028b532>{generic_make_request+258}
[14513.189243]        <ffffffff8028b628>{submit_bio+216} <ffffffff8017e20f>{__bio_add_page+463}
[14513.189256]        <ffffffff8026bd3e>{xfs_submit_ioend_bio+30} <ffffffff8026bec0>{xfs_submit_ioend+144}
[14513.189269]        <ffffffff8026cd03>{xfs_page_state_convert+1379} <ffffffff8026d2e0>{linvfs_writepage+176}
Quoted 8 lines
Quoted 3 lines

This area of XFS changed in 2.6.16; it might be doing something wrong with bios that is blowing up later. Then again, it might be something in dm that is wrong. Does this happen on non-dmcrypt XFS filesystems you have?

All the Oopses so far had dm_crypt in their backtrace, but then the non-crypted XFSs don't see that much action. Also, I recently added a new XFS filesytem to the system. Is there a way to tell which of them causes the Oops?

The only way I know involves prodding the decomposing corpse with a long, pointy stick (i.e. kdb). That probably doesn't help you, though.

Quoted 6 lines
Quoted 3 lines

I understand that the recognised way to find introduced a reproducable problem like this is to do a git bisect to find the mod that introduced the problem. Is it possible for you to do this?

I can certainly try but it usually takes several hours or a at least a few days to manifest here on my desktop.

Well, you can try ruling out the XFS changes - on this page:

http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=history;h=9eeff2395e3cfd05c9b2e6074ff943a34b0c5c21;f=fs/xfs/linux-2.6/xfs_aops.c

These diffs:

2006-01-18
[XFS] Fix a race in xfs_submit_ioend() where we can ... 2006-01-11
[XFS] fix writeback control handling fix a reversed ... [XFS] cluster rewrites We can cluster mapped pages ... [XFS] pass full 64bit offsets to xfs_add_to_ioend [XFS] consolidate some code in xfs_page_state_convert ... [XFS] various fixes for xfs_convert_page fix various ... [XFS] clean up the xfs_offset_to_map interface Currently ... [XFS] use pagevec lookups This reduces the time spend ... [XFS] Initial pass at going directly-to-bio on the ...

Are the ones that change the XFS I/O path from what is in 2.6.14. Try reverting them and see if the problem goes away....

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
R&D Software Enginner
SGI Australian Software Group
#4 lore raw
Quoted 8 lines

These diffs:

2006-01-18
[XFS] Fix a race in xfs_submit_ioend() where we can ... 2006-01-11
[XFS] fix writeback control handling fix a reversed ... [XFS] cluster rewrites We can cluster mapped pages ... [...]

I bet on the 3rd...

Jan Engelhardt

#5 lore raw

Jan Engelhardt schrieb am Dienstag, dem 28. März 2006:

Quoted 10 lines
Quoted 8 lines

These diffs:

2006-01-18
[XFS] Fix a race in xfs_submit_ioend() where we can ... 2006-01-11
[XFS] fix writeback control handling fix a reversed ... [XFS] cluster rewrites We can cluster mapped pages ... [...]

I bet on the 3rd...

Some of the patches don't unapply cleanly anymore. I'll see what I can do despite that.

Signature · 1 line
Peter
#6 lore raw
Quoted 11 lines · Peter Palfrader

Jan Engelhardt schrieb am Dienstag, dem 28. März 2006:

Quoted 9 lines
Quoted 8 lines

These diffs:

2006-01-18
[XFS] Fix a race in xfs_submit_ioend() where we can ... 2006-01-11
[XFS] fix writeback control handling fix a reversed ... [XFS] cluster rewrites We can cluster mapped pages ... [...]

Quoted 4 lines
Quoted 1 line

I bet on the 3rd...

Some of the patches don't unapply cleanly anymore. I'll see what I can do despite that.

You'll be better off trying the bio_clone fix discussed in the other (ext3-bio_clone-panic) thread than go down this route (there is a fix in 2.6.16.1 apparently - start there). Certainly try that before attempting to revert these changes anyway.

cheers.

Signature · 1 line
Nathan
#7 lore raw
Quoted 4 lines · Nathan Scott

You'll be better off trying the bio_clone fix discussed in the other (ext3-bio_clone-panic) thread than go down this route (there is a fix in 2.6.16.1 apparently - start there). Certainly try that before attempting to revert these changes anyway.

The problem persists on 2.6.16.1.

lg,
Peter

#8 lore raw
Quoted 8 lines · Peter Palfrader
Quoted 4 lines · Nathan Scott

You'll be better off trying the bio_clone fix discussed in the other (ext3-bio_clone-panic) thread than go down this route (there is a fix in 2.6.16.1 apparently - start there). Certainly try that before attempting to revert these changes anyway.

The problem persists on 2.6.16.1.

The original oops showed the path into bio_clone() originating from crypt_alloc_buffer() in dm-crypt.c - and blk_recount_segments() is oopsinig in page_to_pfn(), meaning that the most likely cause of this is dm-crypt passing in a bio with invalid ->bi_idx and/or ->bi_vcnt. At least they are not matching what ->bi_io_vec[] holds.

Signature · 1 line
Jens Axboe
#9 lore raw
In reply to Jan Engelhardt

Jan Engelhardt schrieb am Dienstag, dem 28. März 2006:

Quoted 10 lines
Quoted 8 lines

These diffs:

2006-01-18
[XFS] Fix a race in xfs_submit_ioend() where we can ... 2006-01-11
[XFS] fix writeback control handling fix a reversed ... [XFS] cluster rewrites We can cluster mapped pages ... [...]

I bet on the 3rd...

Just reverting #3[0] on 2.6.16 doesn't fix it either.

Peter, back on 2.6.14.2 for now.

0. one hunk didn't unapply cleanly, and the change was not obvious. I opted for making the beginning of xfs_probe_cluster() look like:

|       /* First sum forwards in this page */
|       do {
|               if (!buffer_uptodate(bh) || (mapped != buffer_mapped(bh)))
|                       return total;
|               total += bh->b_size;
|       } while ((bh = bh->b_this_page) != head);