Skip to contentSkip to conversation

Latest kernel NULL pointer deref when running mke2fs

DISCUSSION Richard W.M. Jones 3 participants 8 messages Feb 4, 2014 – Feb 10, 2014 VFS

lore
Share & tools
lore
#0 lore raw

https://bugzilla.redhat.com/show_bug.cgi?id=1061339

It seems to happen when mke2fs issues an ioctl, looks like it might be related to TRIM/discard.

This is under virtualization. The disk is backed by virtio-scsi.

mke2fs -t ext2 -F -b 4096 /dev/VG/LV1
mke2fs 1.42.9 (28-Dec-2013)

[   44.142483] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[   44.142483] IP: [<ffffffff8122040a>] bio_trim+0x1a/0x40
[   44.142483] PGD 1d193067 PUD 1d1c1067 PMD 0 
[   44.142483] Oops: 0000 [#1] SMP 
[   44.142483] Modules linked in: raid1 kvm_amd snd_pcsp snd_pcm kvm snd_timer snd soundcore serio_raw ata_generic pata_acpi virtio_balloon virtio_pci virtio_mmio virtio_net virtio_scsi virtio_blk virtio_console virtio_rng virtio_ring virtio ideapad_laptop sparse_keymap rfkill sym53c8xx scsi_transport_spi crc8 crc_ccitt crc32 crc_itu_t libcrc32c megaraid megaraid_sas megaraid_mbox megaraid_mm
[   44.142483] CPU: 0 PID: 229 Comm: mke2fs Tainted: G        W    3.14.0-0.rc1.git0.1.fc21.x86_64 #1
[   44.142483] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   44.142483] task: ffff88001c100000 ti: ffff88001c0e4000 task.ti: ffff88001c0e4000
[   44.142483] RIP: 0010:[<ffffffff8122040a>]  [<ffffffff8122040a>] bio_trim+0x1a/0x40
[   44.142483] RSP: 0018:ffff88001c0e5b88  EFLAGS: 00000246
[   44.142483] RAX: ffff88001d13f020 RBX: 0000000000000000 RCX: 000000000000b690
[   44.142483] RDX: 0000000000008000 RSI: 0000000000000000 RDI: 0000000000000000
[   44.142483] RBP: ffff88001c0e5b98 R08: 00000000000174a0 R09: ffff88001f0174a0
[   44.142483] R10: 0000000000000000 R11: ffffea0000744fc0 R12: 0000000001000000
[   44.142483] R13: 0000000000000000 R14: ffff88001c0bfe80 R15: ffff88001d16df00
[   44.142483] FS:  00007fe89c7817c0(0000) GS:ffff88001f000000(0000) knlGS:0000000000000000
[   44.142483] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   44.142483] CR2: 0000000000000028 CR3: 000000001c0e7000 CR4: 00000000000006f0
[   44.142483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   44.142483] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[   44.142483] Stack:
[   44.142483]  0000000000000001 0000000000000000 ffff88001c0e5c80 ffffffffa01923f3
[   44.142483]  ffff88001c0e5c50 ffffc90000125040 0000000000008000 ffff88001d16df60
[   44.142483]  0000000000003000 ffff88001c0e5c18 ffffffff00008000 0000000000000001
[   44.142483] Call Trace:
[   44.142483]  [<ffffffffa01923f3>] make_request+0x4c3/0xcd0 [raid1]
[   44.142483]  [<ffffffff810c8ec6>] ? check_preempt_wakeup+0x166/0x250
[   44.142483]  [<ffffffff81555e85>] md_make_request+0xe5/0x230
[   44.142483]  [<ffffffff81326c20>] generic_make_request+0xe0/0x130
[   44.142483]  [<ffffffff81326ce8>] submit_bio+0x78/0x160
[   44.142483]  [<ffffffff81220bfe>] ? bio_alloc_bioset+0x1ce/0x2f0
[   44.142483]  [<ffffffff811fcc73>] ? pollwake+0x73/0x90
[   44.142483]  [<ffffffff8133243b>] blkdev_issue_discard+0x1fb/0x2c0
[   44.142483]  [<ffffffff81336da5>] blkdev_ioctl+0x635/0x7d0
[   44.142483]  [<ffffffff811e83a7>] ? do_sync_write+0x67/0xa0
[   44.142483]  [<ffffffff81222d11>] block_ioctl+0x41/0x50
[   44.142483]  [<ffffffff811fbf90>] do_vfs_ioctl+0x2e0/0x4a0
[   44.142483]  [<ffffffff811fc1f1>] SyS_ioctl+0xa1/0xc0
[   44.142483]  [<ffffffff816fbbe9>] system_call_fastpath+0x16/0x1b
[   44.142483] Code: 01 e9 75 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 54 41 89 d4 41 c1 e4 09 85 f6 53 48 89 fb 75 06 <44> 3b 67 28 74 14 3e 80 63 10 f7 c1 e6 09 48 89 df e8 f0 fe ff 
[   44.142483] RIP  [<ffffffff8122040a>] bio_trim+0x1a/0x40
[   44.142483]  RSP <ffff88001c0e5b88>
[   44.142483] CR2: 0000000000000028
[   44.144483] ---[ end trace f318ded04f590341 ]---

Rich.

Signature · 3 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
#1 lore raw

[ + Kent, Jens, Neil ]

Quoted 28 lines · Richard W.M. Jones

https://bugzilla.redhat.com/show_bug.cgi?id=1061339

It seems to happen when mke2fs issues an ioctl, looks like it might be related to TRIM/discard.

This is under virtualization. The disk is backed by virtio-scsi.

mke2fs -t ext2 -F -b 4096 /dev/VG/LV1
mke2fs 1.42.9 (28-Dec-2013)

[   44.142483] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[   44.142483] IP: [<ffffffff8122040a>] bio_trim+0x1a/0x40
[   44.142483] PGD 1d193067 PUD 1d1c1067 PMD 0
[   44.142483] Oops: 0000 [#1] SMP
[   44.142483] Modules linked in: raid1 kvm_amd snd_pcsp snd_pcm kvm snd_timer snd soundcore serio_raw ata_generic pata_acpi virtio_balloon virtio_pci virtio_mmio virtio_net virtio_scsi virtio_blk virtio_console virtio_rng virtio_ring virtio ideapad_laptop sparse_keymap rfkill sym53c8xx scsi_transport_spi crc8 crc_ccitt crc32 crc_itu_t libcrc32c megaraid megaraid_sas megaraid_mbox megaraid_mm
[   44.142483] CPU: 0 PID: 229 Comm: mke2fs Tainted: G        W    3.14.0-0.rc1.git0.1.fc21.x86_64 #1
[   44.142483] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   44.142483] task: ffff88001c100000 ti: ffff88001c0e4000 task.ti: ffff88001c0e4000
[   44.142483] RIP: 0010:[<ffffffff8122040a>]  [<ffffffff8122040a>] bio_trim+0x1a/0x40
[   44.142483] RSP: 0018:ffff88001c0e5b88  EFLAGS: 00000246
[   44.142483] RAX: ffff88001d13f020 RBX: 0000000000000000 RCX: 000000000000b690
[   44.142483] RDX: 0000000000008000 RSI: 0000000000000000 RDI: 0000000000000000
[   44.142483] RBP: ffff88001c0e5b98 R08: 00000000000174a0 R09: ffff88001f0174a0
[   44.142483] R10: 0000000000000000 R11: ffffea0000744fc0 R12: 0000000001000000
[   44.142483] R13: 0000000000000000 R14: ffff88001c0bfe80 R15: ffff88001d16df00
[   44.142483] FS:  00007fe89c7817c0(0000) GS:ffff88001f000000(0000) knlGS:0000000000000000
[   44.142483] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   44.142483] CR2: 0000000000000028 CR3: 000000001c0e7000 CR4: 00000000000006f0
Quoted 8 lines
[   44.142483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   44.142483] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[   44.142483] Stack:
[   44.142483]  0000000000000001 0000000000000000 ffff88001c0e5c80 ffffffffa01923f3
[   44.142483]  ffff88001c0e5c50 ffffc90000125040 0000000000008000 ffff88001d16df60
[   44.142483]  0000000000003000 ffff88001c0e5c18 ffffffff00008000 0000000000000001
[   44.142483] Call Trace:
[   44.142483]  [<ffffffffa01923f3>] make_request+0x4c3/0xcd0 [raid1]

Based on the oops, we're passing a NULL bio to bio_trim from the MD raid1 make_request.

Not really sure how we get this far, but my guess is it happens here:

                mbio = bio_clone_mddev(bio, GFP_NOIO, mddev);
                bio_trim(mbio, r1_bio->sector - bio->bi_iter.bi_sector, max_sectors);

Guessing mbio is NULL because bio_clone is trying to count the iovecs. bio_for_each_segment expects the bvs to be setup, and since this is a discard bio, they are not.

-chris

#2 lore raw
Quoted 43 lines · Chris Mason

[ + Kent, Jens, Neil ]

Quoted 28 lines · Richard W.M. Jones

https://bugzilla.redhat.com/show_bug.cgi?id=1061339

It seems to happen when mke2fs issues an ioctl, looks like it might be related to TRIM/discard.

This is under virtualization. The disk is backed by virtio-scsi.

mke2fs -t ext2 -F -b 4096 /dev/VG/LV1
mke2fs 1.42.9 (28-Dec-2013)

[   44.142483] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[   44.142483] IP: [<ffffffff8122040a>] bio_trim+0x1a/0x40
[   44.142483] PGD 1d193067 PUD 1d1c1067 PMD 0
[   44.142483] Oops: 0000 [#1] SMP
[   44.142483] Modules linked in: raid1 kvm_amd snd_pcsp snd_pcm kvm snd_timer snd soundcore serio_raw ata_generic pata_acpi virtio_balloon virtio_pci virtio_mmio virtio_net virtio_scsi virtio_blk virtio_console virtio_rng virtio_ring virtio ideapad_laptop sparse_keymap rfkill sym53c8xx scsi_transport_spi crc8 crc_ccitt crc32 crc_itu_t libcrc32c megaraid megaraid_sas megaraid_mbox megaraid_mm
[   44.142483] CPU: 0 PID: 229 Comm: mke2fs Tainted: G        W    3.14.0-0.rc1.git0.1.fc21.x86_64 #1
[   44.142483] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   44.142483] task: ffff88001c100000 ti: ffff88001c0e4000 task.ti: ffff88001c0e4000
[   44.142483] RIP: 0010:[<ffffffff8122040a>]  [<ffffffff8122040a>] bio_trim+0x1a/0x40
[   44.142483] RSP: 0018:ffff88001c0e5b88  EFLAGS: 00000246
[   44.142483] RAX: ffff88001d13f020 RBX: 0000000000000000 RCX: 000000000000b690
[   44.142483] RDX: 0000000000008000 RSI: 0000000000000000 RDI: 0000000000000000
[   44.142483] RBP: ffff88001c0e5b98 R08: 00000000000174a0 R09: ffff88001f0174a0
[   44.142483] R10: 0000000000000000 R11: ffffea0000744fc0 R12: 0000000001000000
[   44.142483] R13: 0000000000000000 R14: ffff88001c0bfe80 R15: ffff88001d16df00
[   44.142483] FS:  00007fe89c7817c0(0000) GS:ffff88001f000000(0000) knlGS:0000000000000000
[   44.142483] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   44.142483] CR2: 0000000000000028 CR3: 000000001c0e7000 CR4: 00000000000006f0
Quoted 8 lines
[   44.142483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   44.142483] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[   44.142483] Stack:
[   44.142483]  0000000000000001 0000000000000000 ffff88001c0e5c80 ffffffffa01923f3
[   44.142483]  ffff88001c0e5c50 ffffc90000125040 0000000000008000 ffff88001d16df60
[   44.142483]  0000000000003000 ffff88001c0e5c18 ffffffff00008000 0000000000000001
[   44.142483] Call Trace:
[   44.142483]  [<ffffffffa01923f3>] make_request+0x4c3/0xcd0 [raid1]

Based on the oops, we're passing a NULL bio to bio_trim from the MD raid1 make_request.

Quoted 8 lines

Not really sure how we get this far, but my guess is it happens here:

                mbio = bio_clone_mddev(bio, GFP_NOIO, mddev);
                bio_trim(mbio, r1_bio->sector - bio->bi_iter.bi_sector, max_sectors);

Guessing mbio is NULL because bio_clone is trying to count the iovecs. bio_for_each_segment expects the bvs to be setup, and since this is a discard bio, they are not.

If you want to see the complete set of commands issued to get to this point, go here:

http://kojipkgs.fedoraproject.org//work/tasks/9085/6489085/build.log

(Sorry it's a rather long log file with much irrelevant info it.)

However if you search for the string

libguestfs: [21323ms] appliance is up

you will see the sequence of commands starting just after that.

Rich.

Signature · 3 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
#3 lore raw
In reply to Chris Mason

Kent Overstreet pointed me to this patch:

https://lkml.org/lkml/2014/2/4/107

Will test it soon.

Rich.

Signature · 4 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
#4 lore raw
Quoted 7 lines · Richard W.M. Jones

Kent Overstreet pointed me to this patch:

https://lkml.org/lkml/2014/2/4/107

Will test it soon.

I don't think this will touch the call path from your oops...at least I don't see how it helps bio_clone do the right thing.

-chris

#5 lore raw
Quoted 1 line · Richard W.M. Jones

https://lkml.org/lkml/2014/2/4/107

Sorry about the delay. I ran into a different kernel bug which needed fixing first.

The patch above *does not* fix this issue.

Rich.

Signature · 4 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
#6 lore raw
Quoted 7 lines · Richard W.M. Jones
Quoted 1 line · Richard W.M. Jones

https://lkml.org/lkml/2014/2/4/107

Sorry about the delay. I ran into a different kernel bug which needed fixing first.

The patch above *does not* fix this issue.

To follow up here, it's actually an MD problem. The following userspace commands are enough to trigger the NULL pointer deref:

  mdadm --create --run test --level raid1 --raid-devices 2 /dev/sda1 /dev/sdb1
  wipefs -a --force /dev/md/test
  mke2fs -t ext4 -F /dev/md/test

There's a full log attached to this comment:

https://bugzilla.redhat.com/show_bug.cgi?id=1061339#c8

Rich.

Signature · 3 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
#7 lore raw
In reply to Chris Mason
Quoted 43 lines · Chris Mason

[ + Kent, Jens, Neil ]

Quoted 28 lines · Richard W.M. Jones

https://bugzilla.redhat.com/show_bug.cgi?id=1061339

It seems to happen when mke2fs issues an ioctl, looks like it might be related to TRIM/discard.

This is under virtualization. The disk is backed by virtio-scsi.

mke2fs -t ext2 -F -b 4096 /dev/VG/LV1
mke2fs 1.42.9 (28-Dec-2013)

[   44.142483] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[   44.142483] IP: [<ffffffff8122040a>] bio_trim+0x1a/0x40
[   44.142483] PGD 1d193067 PUD 1d1c1067 PMD 0
[   44.142483] Oops: 0000 [#1] SMP
[   44.142483] Modules linked in: raid1 kvm_amd snd_pcsp snd_pcm kvm snd_timer snd soundcore serio_raw ata_generic pata_acpi virtio_balloon virtio_pci virtio_mmio virtio_net virtio_scsi virtio_blk virtio_console virtio_rng virtio_ring virtio ideapad_laptop sparse_keymap rfkill sym53c8xx scsi_transport_spi crc8 crc_ccitt crc32 crc_itu_t libcrc32c megaraid megaraid_sas megaraid_mbox megaraid_mm
[   44.142483] CPU: 0 PID: 229 Comm: mke2fs Tainted: G        W    3.14.0-0.rc1.git0.1.fc21.x86_64 #1
[   44.142483] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   44.142483] task: ffff88001c100000 ti: ffff88001c0e4000 task.ti: ffff88001c0e4000
[   44.142483] RIP: 0010:[<ffffffff8122040a>]  [<ffffffff8122040a>] bio_trim+0x1a/0x40
[   44.142483] RSP: 0018:ffff88001c0e5b88  EFLAGS: 00000246
[   44.142483] RAX: ffff88001d13f020 RBX: 0000000000000000 RCX: 000000000000b690
[   44.142483] RDX: 0000000000008000 RSI: 0000000000000000 RDI: 0000000000000000
[   44.142483] RBP: ffff88001c0e5b98 R08: 00000000000174a0 R09: ffff88001f0174a0
[   44.142483] R10: 0000000000000000 R11: ffffea0000744fc0 R12: 0000000001000000
[   44.142483] R13: 0000000000000000 R14: ffff88001c0bfe80 R15: ffff88001d16df00
[   44.142483] FS:  00007fe89c7817c0(0000) GS:ffff88001f000000(0000) knlGS:0000000000000000
[   44.142483] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   44.142483] CR2: 0000000000000028 CR3: 000000001c0e7000 CR4: 00000000000006f0
Quoted 8 lines
[   44.142483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   44.142483] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[   44.142483] Stack:
[   44.142483]  0000000000000001 0000000000000000 ffff88001c0e5c80 ffffffffa01923f3
[   44.142483]  ffff88001c0e5c50 ffffc90000125040 0000000000008000 ffff88001d16df60
[   44.142483]  0000000000003000 ffff88001c0e5c18 ffffffff00008000 0000000000000001
[   44.142483] Call Trace:
[   44.142483]  [<ffffffffa01923f3>] make_request+0x4c3/0xcd0 [raid1]

Based on the oops, we're passing a NULL bio to bio_trim from the MD raid1 make_request.

Quoted 8 lines

Not really sure how we get this far, but my guess is it happens here:

                mbio = bio_clone_mddev(bio, GFP_NOIO, mddev);
                bio_trim(mbio, r1_bio->sector - bio->bi_iter.bi_sector, max_sectors);

Guessing mbio is NULL because bio_clone is trying to count the iovecs. bio_for_each_segment expects the bvs to be setup, and since this is a discard bio, they are not.

Sorry for the delay, just got back. Your analysis looks correct to me - mailing out a patch shortly