Skip to contentSkip to conversation

virtio-blk-pci chokes on some PCI slots

PATCH Markus Armbruster 3 participants 5 messages Jun 16, 2009 – Jun 18, 2009 KVM

lore
Share & tools
lore

Markus Armbruster

author
#0 lore raw

I'm working on make PCI slots configurable in QEMU. While testing the feature for virtio-blk-pci, I ran into a task hang with Fedora kernel 2.6.27.5-117.fc10. Marcelo tested it with a newer kernel, and will follow up with details.

Instead of the full QEMU patch I'm working on, the appended little hack suffices to reproduce the problem.

    $ QEMU_VIRTIO_BLK=9 qemu foo.qcow2 -drive file=bar.img,if=virtio -serial stdio
[...]
    Loading virtio_pci module
    ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
    virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, low) -> IRQ 11
    virtio-pci 0000:00:09.0: can't derive routing for PCI INT A
    virtio-pci 0000:00:09.0: PCI INT A: no GSI - using IRQ 11
    Loading virtio_blk module
     vda:
[tick-tock-tick-tock...]
    <3>INFO: task modprobe:491 blocked for more than 120 seconds.
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    modprobe      D c04303f1     0   491      1
           c6baeae8 00000082 c6baea80 c04303f1 c6bae000 c087667c c0879c00 c0879c00 
           c0879c00 c6b659b0 c6b65c24 c1108c00 00000000 c1108c00 83a43d86 00000002 
           c781aaa8 c6baead0 c04418d0 c6b65c24 00004d88 c781aaa8 00004d88 c6baeb3c 
    Call Trace:
     [<c04303f1>] ? irq_exit+0x5f/0x83
     [<c04418d0>] ? getnstimeofday+0x3c/0xc9
     [<c043fcae>] ? ktime_get_ts+0x4a/0x4e
     [<c06a6110>] io_schedule+0x52/0x8a
     [<c046da7d>] sync_page+0x46/0x4c
     [<c06a64c9>] __wait_on_bit_lock+0x34/0x5e
     [<c046da37>] ? sync_page+0x0/0x4c
     [<c046d9f3>] __lock_page+0x78/0x81
     [<c043cd89>] ? wake_bit_function+0x0/0x43
     [<c046da28>] lock_page+0x2c/0x2f
     [<c046dff4>] read_cache_page_async+0xa4/0xfb
     [<c04b1cec>] ? blkdev_readpage+0x0/0x11
     [<c046e057>] read_cache_page+0xc/0x3c
     [<c04c8ec1>] read_dev_sector+0x34/0x6a
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04ca945>] read_lba+0x69/0xc6
     [<c04cabde>] ? efi_partition+0x84/0x676
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04cabfd>] efi_partition+0xa3/0x676
     [<c042c71a>] ? vprintk+0x2c8/0x2f3
     [<c04a13ab>] ? iget5_locked+0x33/0x114
     [<c051cf05>] ? snprintf+0x15/0x17
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04c94db>] rescan_partitions+0x106/0x242
     [<c041d8ba>] ? need_resched+0x18/0x22
     [<c06a6192>] ? _cond_resched+0x8/0x32
     [<c04b1a4a>] do_open+0x1ff/0x290
     [<c04b1b9d>] __blkdev_get+0x71/0x7c
     [<c04b1bb5>] blkdev_get+0xd/0xf
     [<c04c8fe7>] register_disk+0xf0/0x141
     [<c05111cf>] add_disk+0x34/0x89
     [<c05101fc>] ? exact_match+0x0/0xb
     [<c0510e13>] ? exact_lock+0x0/0x11
     [<c88223b2>] virtblk_probe+0x275/0x2a8 [virtio_blk]
     [<c062236b>] virtio_dev_probe+0x89/0xb3
     [<c05937d0>] driver_probe_device+0xa0/0x136
     [<c05938a0>] __driver_attach+0x3a/0x59
     [<c05931f6>] bus_for_each_dev+0x3b/0x63
     [<c0593675>] driver_attach+0x14/0x16
     [<c0593866>] ? __driver_attach+0x0/0x59
     [<c0592c6a>] bus_add_driver+0x9d/0x1ba
     [<c0593a27>] driver_register+0x81/0xe1
     [<c05112ec>] ? register_blkdev+0xc8/0xdc
     [<c062245a>] register_virtio_driver+0x1f/0x21
     [<c8826022>] init+0x22/0x24 [virtio_blk]
     [<c0401125>] _stext+0x3d/0x115
     [<c8826000>] ? init+0x0/0x24 [virtio_blk]
     [<c044bf7d>] sys_init_module+0x87/0x178
     [<c0403c76>] syscall_call+0x7/0xb
     =======================

Some slots work, e.g. 5, some don't, e.g. 6.

What's going on here?

hw/pc.c +2 -1
index 143b697..15c44ac 100644
@@ -1146,7 +1146,8 @@ static void pc_init1(ram_addr_t ram_size,
         int unit_id = 0;
 
         while ((index = drive_get_index(IF_VIRTIO, 0, unit_id)) != -1) {
-            pci_create_simple(pci_bus, -1, "virtio-blk-pci");
+            char *devfn = getenv("QEMU_VIRTIO_BLK");
+            pci_create_simple(pci_bus, devfn ? atoi(devfn) * 8: -1, "virtio-blk-pci");
             unit_id++;
         }
     }
#1 lore raw
Quoted 86 lines · Markus Armbruster

I'm working on make PCI slots configurable in QEMU. While testing the feature for virtio-blk-pci, I ran into a task hang with Fedora kernel 2.6.27.5-117.fc10. Marcelo tested it with a newer kernel, and will follow up with details.

Instead of the full QEMU patch I'm working on, the appended little hack suffices to reproduce the problem.

    $ QEMU_VIRTIO_BLK=9 qemu foo.qcow2 -drive file=bar.img,if=virtio -serial stdio
[...]
    Loading virtio_pci module
    ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
    virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, low) -> IRQ 11
    virtio-pci 0000:00:09.0: can't derive routing for PCI INT A
    virtio-pci 0000:00:09.0: PCI INT A: no GSI - using IRQ 11
    Loading virtio_blk module
     vda:
[tick-tock-tick-tock...]
    <3>INFO: task modprobe:491 blocked for more than 120 seconds.
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    modprobe      D c04303f1     0   491      1
           c6baeae8 00000082 c6baea80 c04303f1 c6bae000 c087667c c0879c00 c0879c00 
           c0879c00 c6b659b0 c6b65c24 c1108c00 00000000 c1108c00 83a43d86 00000002 
           c781aaa8 c6baead0 c04418d0 c6b65c24 00004d88 c781aaa8 00004d88 c6baeb3c 
    Call Trace:
     [<c04303f1>] ? irq_exit+0x5f/0x83
     [<c04418d0>] ? getnstimeofday+0x3c/0xc9
     [<c043fcae>] ? ktime_get_ts+0x4a/0x4e
     [<c06a6110>] io_schedule+0x52/0x8a
     [<c046da7d>] sync_page+0x46/0x4c
     [<c06a64c9>] __wait_on_bit_lock+0x34/0x5e
     [<c046da37>] ? sync_page+0x0/0x4c
     [<c046d9f3>] __lock_page+0x78/0x81
     [<c043cd89>] ? wake_bit_function+0x0/0x43
     [<c046da28>] lock_page+0x2c/0x2f
     [<c046dff4>] read_cache_page_async+0xa4/0xfb
     [<c04b1cec>] ? blkdev_readpage+0x0/0x11
     [<c046e057>] read_cache_page+0xc/0x3c
     [<c04c8ec1>] read_dev_sector+0x34/0x6a
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04ca945>] read_lba+0x69/0xc6
     [<c04cabde>] ? efi_partition+0x84/0x676
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04cabfd>] efi_partition+0xa3/0x676
     [<c042c71a>] ? vprintk+0x2c8/0x2f3
     [<c04a13ab>] ? iget5_locked+0x33/0x114
     [<c051cf05>] ? snprintf+0x15/0x17
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04c94db>] rescan_partitions+0x106/0x242
     [<c041d8ba>] ? need_resched+0x18/0x22
     [<c06a6192>] ? _cond_resched+0x8/0x32
     [<c04b1a4a>] do_open+0x1ff/0x290
     [<c04b1b9d>] __blkdev_get+0x71/0x7c
     [<c04b1bb5>] blkdev_get+0xd/0xf
     [<c04c8fe7>] register_disk+0xf0/0x141
     [<c05111cf>] add_disk+0x34/0x89
     [<c05101fc>] ? exact_match+0x0/0xb
     [<c0510e13>] ? exact_lock+0x0/0x11
     [<c88223b2>] virtblk_probe+0x275/0x2a8 [virtio_blk]
     [<c062236b>] virtio_dev_probe+0x89/0xb3
     [<c05937d0>] driver_probe_device+0xa0/0x136
     [<c05938a0>] __driver_attach+0x3a/0x59
     [<c05931f6>] bus_for_each_dev+0x3b/0x63
     [<c0593675>] driver_attach+0x14/0x16
     [<c0593866>] ? __driver_attach+0x0/0x59
     [<c0592c6a>] bus_add_driver+0x9d/0x1ba
     [<c0593a27>] driver_register+0x81/0xe1
     [<c05112ec>] ? register_blkdev+0xc8/0xdc
     [<c062245a>] register_virtio_driver+0x1f/0x21
     [<c8826022>] init+0x22/0x24 [virtio_blk]
     [<c0401125>] _stext+0x3d/0x115
     [<c8826000>] ? init+0x0/0x24 [virtio_blk]
     [<c044bf7d>] sys_init_module+0x87/0x178
     [<c0403c76>] syscall_call+0x7/0xb
     =======================

Some slots work, e.g. 5, some don't, e.g. 6.

What's going on here?

hw/pc.c +0 -0
index 143b697..15c44ac 100644

hw/pc.c static void pc_init1(ram_addr_t ram_size, · line 1153

Quoted 9 lines
hw/pc.c +2 -1
@@ -1146,7 +1146,8 @@ static void pc_init1(ram_addr_t ram_size,
         int unit_id = 0;
 
         while ((index = drive_get_index(IF_VIRTIO, 0, unit_id)) != -1) {
-            pci_create_simple(pci_bus, -1, "virtio-blk-pci");
+            char *devfn = getenv("QEMU_VIRTIO_BLK");
+            pci_create_simple(pci_bus, devfn ? atoi(devfn) * 8: -1, "virtio-blk-pci");
             unit_id++;
         }
     }

no softlockup with 2.6.29-06638-g17db0a0, but not very friendly either (note the backing device has no partitions in my case, perhaps that makes a difference).

virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, low) -> IRQ 10
virtio-pci 0000:00:09.0: can't derive routing for PCI INT A virtio-pci 0000:00:09.0: PCI INT A: no GSI - using IRQ 11

Starting udev: Wait timeout. Will continue in the background.[FAILED]

[root@guest ~]# dmesg | grep vda
vda:<6>EXT3 FS on dm-0, internal journal

[root@guest ~]# mount /dev/vda /mnt

Hangs there forever.

author
#2 lore raw
Quoted 106 lines · Marcelo Tosatti
Quoted 86 lines · Markus Armbruster

I'm working on make PCI slots configurable in QEMU. While testing the feature for virtio-blk-pci, I ran into a task hang with Fedora kernel 2.6.27.5-117.fc10. Marcelo tested it with a newer kernel, and will follow up with details.

Instead of the full QEMU patch I'm working on, the appended little hack suffices to reproduce the problem.

    $ QEMU_VIRTIO_BLK=9 qemu foo.qcow2 -drive file=bar.img,if=virtio -serial stdio
[...]
    Loading virtio_pci module
    ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
    virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, low) -> IRQ 11
    virtio-pci 0000:00:09.0: can't derive routing for PCI INT A
    virtio-pci 0000:00:09.0: PCI INT A: no GSI - using IRQ 11
    Loading virtio_blk module
     vda:
[tick-tock-tick-tock...]
    <3>INFO: task modprobe:491 blocked for more than 120 seconds.
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    modprobe      D c04303f1     0   491      1
           c6baeae8 00000082 c6baea80 c04303f1 c6bae000 c087667c c0879c00 c0879c00 
           c0879c00 c6b659b0 c6b65c24 c1108c00 00000000 c1108c00 83a43d86 00000002 
           c781aaa8 c6baead0 c04418d0 c6b65c24 00004d88 c781aaa8 00004d88 c6baeb3c 
    Call Trace:
     [<c04303f1>] ? irq_exit+0x5f/0x83
     [<c04418d0>] ? getnstimeofday+0x3c/0xc9
     [<c043fcae>] ? ktime_get_ts+0x4a/0x4e
     [<c06a6110>] io_schedule+0x52/0x8a
     [<c046da7d>] sync_page+0x46/0x4c
     [<c06a64c9>] __wait_on_bit_lock+0x34/0x5e
     [<c046da37>] ? sync_page+0x0/0x4c
     [<c046d9f3>] __lock_page+0x78/0x81
     [<c043cd89>] ? wake_bit_function+0x0/0x43
     [<c046da28>] lock_page+0x2c/0x2f
     [<c046dff4>] read_cache_page_async+0xa4/0xfb
     [<c04b1cec>] ? blkdev_readpage+0x0/0x11
     [<c046e057>] read_cache_page+0xc/0x3c
     [<c04c8ec1>] read_dev_sector+0x34/0x6a
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04ca945>] read_lba+0x69/0xc6
     [<c04cabde>] ? efi_partition+0x84/0x676
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04cabfd>] efi_partition+0xa3/0x676
     [<c042c71a>] ? vprintk+0x2c8/0x2f3
     [<c04a13ab>] ? iget5_locked+0x33/0x114
     [<c051cf05>] ? snprintf+0x15/0x17
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04c94db>] rescan_partitions+0x106/0x242
     [<c041d8ba>] ? need_resched+0x18/0x22
     [<c06a6192>] ? _cond_resched+0x8/0x32
     [<c04b1a4a>] do_open+0x1ff/0x290
     [<c04b1b9d>] __blkdev_get+0x71/0x7c
     [<c04b1bb5>] blkdev_get+0xd/0xf
     [<c04c8fe7>] register_disk+0xf0/0x141
     [<c05111cf>] add_disk+0x34/0x89
     [<c05101fc>] ? exact_match+0x0/0xb
     [<c0510e13>] ? exact_lock+0x0/0x11
     [<c88223b2>] virtblk_probe+0x275/0x2a8 [virtio_blk]
     [<c062236b>] virtio_dev_probe+0x89/0xb3
     [<c05937d0>] driver_probe_device+0xa0/0x136
     [<c05938a0>] __driver_attach+0x3a/0x59
     [<c05931f6>] bus_for_each_dev+0x3b/0x63
     [<c0593675>] driver_attach+0x14/0x16
     [<c0593866>] ? __driver_attach+0x0/0x59
     [<c0592c6a>] bus_add_driver+0x9d/0x1ba
     [<c0593a27>] driver_register+0x81/0xe1
     [<c05112ec>] ? register_blkdev+0xc8/0xdc
     [<c062245a>] register_virtio_driver+0x1f/0x21
     [<c8826022>] init+0x22/0x24 [virtio_blk]
     [<c0401125>] _stext+0x3d/0x115
     [<c8826000>] ? init+0x0/0x24 [virtio_blk]
     [<c044bf7d>] sys_init_module+0x87/0x178
     [<c0403c76>] syscall_call+0x7/0xb
     =======================

Some slots work, e.g. 5, some don't, e.g. 6.

What's going on here?

hw/pc.c +0 -0
index 143b697..15c44ac 100644

hw/pc.c static void pc_init1(ram_addr_t ram_size, · line 1153

Quoted 9 lines
hw/pc.c +2 -1
@@ -1146,7 +1146,8 @@ static void pc_init1(ram_addr_t ram_size,
         int unit_id = 0;
 
         while ((index = drive_get_index(IF_VIRTIO, 0, unit_id)) != -1) {
-            pci_create_simple(pci_bus, -1, "virtio-blk-pci");
+            char *devfn = getenv("QEMU_VIRTIO_BLK");
+            pci_create_simple(pci_bus, devfn ? atoi(devfn) * 8: -1, "virtio-blk-pci");
             unit_id++;
         }
     }

no softlockup with 2.6.29-06638-g17db0a0, but not very friendly either (note the backing device has no partitions in my case, perhaps that makes a difference).

virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, low) -> IRQ 10
virtio-pci 0000:00:09.0: can't derive routing for PCI INT A virtio-pci 0000:00:09.0: PCI INT A: no GSI - using IRQ 11

Quoted 8 lines

Starting udev: Wait timeout. Will continue in the background.[FAILED]

[root@guest ~]# dmesg | grep vda
vda:<6>EXT3 FS on dm-0, internal journal

[root@guest ~]# mount /dev/vda /mnt

Hangs there forever.

Even better: I can reproduce it with an unpatched upstream QEMU (qemu.git commit cfde4bd9) like this:

    $ qemu foo.qcow2 -net nic -net nic -net nic -drive file=bar.img,if=virtio

virtio-blk-pci ends up in slot 6.

author
#3 lore raw
Quoted 6 lines · Markus Armbruster

Even better: I can reproduce it with an unpatched upstream QEMU (qemu.git commit cfde4bd9) like this:

    $ qemu foo.qcow2 -net nic -net nic -net nic -drive file=bar.img,if=virtio

virtio-blk-pci ends up in slot 6.

And:

    $ qemu foo.qcow2 -net nic,model=virtio -net nic,model=virtio -net nic,model=virtio -net nic,model=virtio

chokes like this:

Setting up hotplug.
Creating block device nodes.
Creating character device nodes.
Loading virtio_pci module
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, low) -> IRQ 10
ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11
virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11 virtio-pci 0000:00:06.0: can't derive routing for PCI INT A virtio-pci 0000:00:06.0: PCI INT A: no GSI - using IRQ 9
virtio-pci 0000:00:07.0: can't derive routing for PCI INT A virtio-pci 0000:00:07.0: PCI INT A: no GSI - using IRQ 11
Loading virtio_blk module

Boot completes, but none of the four NICs work (no /dev/eth*). If I use only two, they end up in "good" slots, and both work fine.

#4 lore raw
In reply to Markus Armbruster
Quoted 11 lines · Markus Armbruster

I'm working on make PCI slots configurable in QEMU. While testing the feature for virtio-blk-pci, I ran into a task hang with Fedora kernel 2.6.27.5-117.fc10. Marcelo tested it with a newer kernel, and will follow up with details.

Some slots work, e.g. 5, some don't, e.g. 6.

What's going on here?

Most likely the interrupt wiring is screwed up. Check your connectors, clear the motherboard of dust (dust + moisture can short out conductors). If that doesn't work, check if the ACPI interrupt routing matches the actual routing by the pci code.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.