Hot-adding disk on VM

Hi all!

According to I see in this document [1], is necessary that is loaded two modules in the guest: acpiphp and pci_hotplug.

The pci_hotplug module is loaded. Nevertheless, in spite of existing the acpiphp module, cannot be loaded:

chandra:~# lsmod|grep pci_hotplug
pci_hotplug            23460  0

chandra:~# modprobe acpiphp
FATAL: Error inserting acpiphp
(/lib/modules/2.6.26-2-686/kernel/drivers/pci/hotplug/acpiphp.ko): No such device

chandra:~# locate acpiphp | grep 686
/lib/modules/2.6.26-2-686/kernel/drivers/pci/hotplug/acpiphp.ko
/lib/modules/2.6.26-2-686/kernel/drivers/pci/hotplug/acpiphp_ibm.ko

Which could be being the problem?

Thanks in advance for your reply.

Regards,
Daniel

[1] http://www.linux-kvm.org/page/Hotadd_pci_devices

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

Hi, all!

I was doing other tests this time with Linux 2.6.31.2 compiled by myself from the sources of kernel.org. This time with loading the acpiphp module, the pci_hotplug module was loaded automatically without I get some type of error message.

central:~# lsmod | grep acpiphp
central:~#
central:~# modprobe acpiphp
central:~#
central:~# lsmod | grep acpiphp
acpiphp                22140  0
pci_hotplug            29024  1 acpiphp

Then, I tried to add a disk of the following way in host:

(qemu) pci_add auto storage file=/dev/vm/central-test,if=virtio OK domain 0, bus 0, slot 5, function 0

In the guest:

Oct 11 18:09:01 central kernel: [  833.403875] pci 0000:00:02.0: BAR 6: bogus alignment [0x0-0x0] flags 0x2
Oct 11 18:09:01 central kernel: [  833.406337] decode_hpp: Could not get hotplug parameters. Use defaults
Oct 11 18:09:01 central kernel: [  833.430492] virtio-pci 0000:00:05.0: enabling device (0000 -> 0001)
Oct 11 18:09:01 central kernel: [  833.452268] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
Oct 11 18:09:01 central kernel: [  833.454668] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
Oct 11 18:09:01 central kernel: [  833.500051] Clocksource tsc unstable (delta = -148257950 ns)
Oct 11 18:09:01 central kernel: [  833.566755]  vda: vda1

Then I attempt to remove the device assuring previously that it is not being used in guest:

(qemu) pci_del 5

In the guest:

Oct 11 18:13:32 central kernel: [ 1104.144179] virtio-pci 0000:00:05.0: PCI INT A disabled

But if I want to add the device again, I get a segmentation fault of the process that handles this virtual machine:

(qemu) pci_add auto storage file=/dev/vm/central-test,if=virtio Connection closed by foreign host.

root@ubuntu:~# tail -f /var/log/messages
[...]
Oct 11 18:21:29 ubuntu kernel: [692002.207026] qemu-system-x86[3481]: segfault at 8d0 ip 000000000041a183 sp 00007fff848ef9f0 error 6 in qemu-system-x86_64[400000+219000]
Oct 11 18:21:29 ubuntu kernel: [692002.291714] br0: port 4(tap2) entering disabled state
Oct 11 18:21:29 ubuntu kernel: [692002.330875] br0: port 4(tap2) entering disabled state

Also I was testing using if=scsi and when I trying to remove the device, can be seen a 'tainted' in guest. In host:

(qemu) pci_add auto storage file=/dev/vm/central-test,if=scsi

In guest:

Oct 11 18:35:51 central kernel: [  199.375026] pci 0000:00:02.0: BAR 6: bogus alignment [0x0-0x0] flags 0x2
Oct 11 18:35:51 central kernel: [  199.377801] decode_hpp: Could not get hotplug parameters. Use defaults
Oct 11 18:35:51 central kernel: [  199.484410] sym53c8xx 0000:00:05.0: enabling device (0000 -> 0003)
Oct 11 18:35:51 central kernel: [  199.495739] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
Oct 11 18:35:51 central kernel: [  199.498165] sym53c8xx 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
Oct 11 18:35:51 central kernel: [  199.503886] sym0: <895a> rev 0x0 at pci 0000:00:05.0 irq 10
Oct 11 18:35:51 central kernel: [  199.508711] sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
Oct 11 18:35:51 central kernel: [  199.512023] sym0: SCSI BUS has been reset.
Oct 11 18:35:51 central kernel: [  199.530575] scsi2 : sym-2.2.3
Oct 11 18:35:54 central kernel: [  202.512229] scsi 2:0:0:0: Direct-Access     QEMU     QEMU HARDDISK    0.10 PQ: 0 ANSI: 3
Oct 11 18:35:54 central kernel: [  202.515687]  target2:0:0: tagged command queuing enabled, command queue depth 16.
Oct 11 18:35:54 central kernel: [  202.519013]  target2:0:0: Beginning Domain Validation
Oct 11 18:35:54 central kernel: [  202.521737]  target2:0:0: Domain Validation skipping write tests
Oct 11 18:35:54 central kernel: [  202.524167]  target2:0:0: Ending Domain Validation
Oct 11 18:35:54 central kernel: [  202.530816] sd 2:0:0:0: Attached scsi generic sg3 type 0
Oct 11 18:35:54 central kernel: [  202.533321] sd 2:0:0:0: [sdc] 10485760 512-byte logical blocks: (5.36 GB/5.00 GiB)
Oct 11 18:35:54 central kernel: [  202.536661] sd 2:0:0:0: [sdc] Write Protect is off
Oct 11 18:35:54 central kernel: [  202.538825] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 11 18:35:54 central kernel: [  202.543108]  sdc: sdc1
Oct 11 18:35:54 central kernel: [  202.557622] sd 2:0:0:0: [sdc] Attached SCSI disk

Then I attempt to remove the device assuring previously that it is not being used in guest:

(qemu) pci_del 5

In the guest:

Oct 11 18:35:54 central kernel: [  202.557622] sd 2:0:0:0: [sdc] Attached SCSI disk                                                
Oct 11 18:38:11 central kernel: [  338.629461] sd 2:0:0:0: [sdc] Synchronizing SCSI cache                                          
Oct 11 18:38:11 central kernel: [  338.631863] sym0: detaching ...                                                                 
Oct 11 18:38:11 central kernel: [  338.633432] sym0: resetting chip                                                                
Oct 11 18:38:11 central kernel: [  338.635014] ------------[] cut here ]------------                                               
Oct 11 18:38:11 central kernel: [  338.636999] WARNING: at /space/src/linux-2.6.31.2/arch/x86/include/asm/dma-mapping.h:132 ___free_dma_mem_cluster+0x78/0xd4 [sym53c8xx]()                                                                                           
Oct 11 18:38:11 central kernel: [  338.637414] Hardware name:                                                                      
Oct 11 18:38:11 central kernel: [  338.637414] Modules linked in: sym53c8xx scsi_transport_spi acpiphp pci_hotplug loop serio_raw psmouse snd_pcsp parport_pc parport snd_pcm snd_timer virtio_net
virtio_balloon snd soundcore snd_page_alloc button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache sg sr_mod cdrom sd_mod piix ide_pci_generic ide_core ata_piix ata_generic libata virtio_pci
virtio_ring virtio scsi_mod thermal fan thermal_sys [last unloaded: scsi_wait_scan]                                                   
Oct 11 18:38:11 central kernel: [  338.637414] Pid: 75, comm: kacpi_notify Not tainted 2.6.31.2-dgb #1                             
Oct 11 18:38:11 central kernel: [  338.637414] Call Trace:                                                                         
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffffa023c92c>] ? ___free_dma_mem_cluster+0x78/0xd4 [sym53c8xx]               
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffffa023c92c>] ? ___free_dma_mem_cluster+0x78/0xd4 [sym53c8xx]               
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff8104244e>] ? warn_slowpath_common+0x77/0xa3                              
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff81199288>] ? acpi_os_execute_deferred+0x0/0x2c                           
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffffa023c92c>] ? ___free_dma_mem_cluster+0x78/0xd4 [sym53c8xx]               
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffffa023c88f>] ? __sym_mfree_dma+0x9d/0xc2 [sym53c8xx]                       
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff81199288>] ? acpi_os_execute_deferred+0x0/0x2c                           
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffffa0236601>] ? sym_hcb_free+0x79/0x1af [sym53c8xx]                         
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffffa023474e>] ? sym_free_resources+0x52/0x71 [sym53c8xx]                    
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffffa023480b>] ? sym_detach+0x9e/0xb5 [sym53c8xx]                            
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffffa0234846>] ? sym2_remove+0x24/0x40 [sym53c8xx]                           
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff8117f2a4>] ? pci_device_remove+0x24/0x47                                 
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff811e4cb4>] ? __device_release_driver+0x77/0xbf                           
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff811e4db2>] ? device_release_driver+0x1e/0x2a                             
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff811e43b6>] ? bus_remove_device+0x9f/0xbe                                 
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff811e2ad8>] ? device_del+0x130/0x16e                                      
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff811e2b1f>] ? device_unregister+0x9/0x12                                  
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff8117b023>] ? pci_stop_bus_device+0x53/0x5f                               
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffffa021ec07>] ? acpiphp_disable_slot+0x95/0x1af [acpiphp]                   
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff81059eec>] ? up+0xe/0x36                                                 
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff81199288>] ? acpi_os_execute_deferred+0x0/0x2c                           
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffffa021f442>] ? handle_hotplug_event_func+0x10c/0x13e [acpiphp]             
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff8119ae91>] ? acpi_bus_get_device+0x1d/0x2e                               
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff8119af29>] ? acpi_bus_notify+0x43/0x79                                   
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff811a8122>] ? acpi_ev_notify_dispatch+0x57/0x61                           
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff811992ab>] ? acpi_os_execute_deferred+0x23/0x2c                          
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff81052992>] ? worker_thread+0x173/0x20f                                   
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff8105699a>] ? autoremove_wake_function+0x0/0x2e                           
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff8105281f>] ? worker_thread+0x0/0x20f                                     
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff8105664c>] ? kthread+0x8b/0x93                                           
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff8100caea>] ? child_rip+0xa/0x20                                          
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff810565c1>] ? kthread+0x0/0x93                                            
Oct 11 18:38:11 central kernel: [  338.637414]  [<ffffffff8100cae0>] ? child_rip+0x0/0x20                                          
Oct 11 18:38:11 central kernel: [  338.637414] ---[] end trace 446c5343fa7322b8 ]---                                               
Oct 11 18:38:11 central kernel: [  338.755839] ------------[] cut here ]------------
[ ... This error appears four times followed in log ... ]
Oct 11 18:38:11 central kernel: [  339.333405] sym53c8xx 0000:00:05.0: PCI INT A disabled

I try to add the disk again:

(qemu) pci_add auto storage file=/dev/vm/central-test,if=scsi OK domain 0, bus 0, slot 5, function 0

But this time in guest is not finished making the detection:

Oct 11 18:43:17 central kernel: [  645.159958] pci 0000:00:02.0: BAR 6: bogus alignment [0x0-0x0] flags 0x2
Oct 11 18:43:17 central kernel: [  645.162594] decode_hpp: Could not get hotplug parameters. Use defaults
Oct 11 18:43:17 central kernel: [  645.172372] sym53c8xx 0000:00:05.0: enabling device (0000 -> 0003)
Oct 11 18:43:17 central kernel: [  645.174952] sym53c8xx 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
Oct 11 18:43:17 central kernel: [  645.180729] sym0: <895a> rev 0x0 at pci 0000:00:05.0 irq 10
Oct 11 18:43:17 central kernel: [  645.202337] sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
Oct 11 18:43:17 central kernel: [  645.208610] sym0: SCSI BUS has been reset.
Oct 11 18:43:17 central kernel: [  645.224164] scsi3 : sym-2.2.3

And from this moment I'm not able to remove the disk.

Which can now be the problem?

Thanks in advance for your reply.

Regards,
Daniel

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

I can actually confirm this behavior. I've been struggling with this for a couple of days now. CC'ing QEMU, as I've discussed this a bit with Anthony Liguori. There's at least two related bugs we've been tracking:
* https://bugs.edge.launchpad.net/ubuntu/+source/qemu-kvm/+bug/432154
* https://bugs.edge.launchpad.net/ubuntu/+source/qemu-kvm/+bug/419590

:-Dustin

Hi, Dustin.

According to I see in Launchpad, there is a fix released for Qemu and qemu-kvm (Ubuntu).

I'm using KVM-88 compiled from the source code obtained of the KVM official site. How I could apply these fix? There is some KVM version that includes these corrections in the official site?

Thanks for your reply.

Regards,
Daniel

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

It's "fix committed" in QEMU, and "Fix Released" in Ubuntu's qemu-kvm.

We in Ubuntu have released version of qemu-kvm-0.11.0 that includes an extra patch from Anthony, which he posted on the mailing list, and committed to QEMU's git tree. He hasn't officially released an upstream QEMU with the fix yet, though.

Not that I know of, yet.

You can pull the patch [1] we're carrying in Ubuntu, and apply it to qemu-kvm-0.11.0 [2] from the kvm website, if you like.

[1] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/karmic/qemu-kvm/karmic/annotate/head%3A/debian/patches/11_fix_virtio-blk_hot_add_after_remove.patch

[2] http://sourceforge.net/projects/kvm/files/

:-Dustin

Hi, Dustin.

I've tested again with Linux 2.6.32 and qemu-kvm-0.12.0-rc2 and this time I could verify that no longer happens the problem with Virtio. That is to say, the virtual disk can be added and be removed every time without problems.

It drew attention to me that whenever I add the virtio device, this appears with a different name (vdc, vdd, vde, etc). This it is the awaited behavior?

Testing the same with if=scsi, I get a tainted again when removing the virtual disk since the first time that I try to remove it. Nevertheless, no longer it seems to have problem to add it again.

Thanks for your reply.

Merry Christmas.

Regards,
Daniel

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598