: Fix xm block-detach

All,
The recent changes to where xenstore stores data about domains (c/s 18562,
in particular) seem to have broken xm block-detach. What happens is that attaching a disk works just fine, but detaching a disk ends with:

Error: 51776 not connected
Usage: xm block-detach <Domain> <DevId> [-f|--force]

Destroy a domain's virtual block device.

The problem basically boils down to where the new code is placing the xenstore entries. Previously, it was storing them in /local/domain/<number>/device/vbd; it is now storing them in /vm/UUID/device/tap. The tap is wrong; tap describes the backend, not the frontend, which should always be vbd. The attached patch fixes this by overriding deviceRoot() in the BlktapController class, similar to how frontendRoot() is done, and then changing devicePath() to use deviceRoot().
There is also a small fix for destroyDevice, to make sure we remove the proper
/vm/UUID/device/vbd entry on removal.
With the patch in place, I was able to successfully block-attach and
block-detach disks again. Note that I only tested this on RHEL-5 xend, which has diverged quite a bit from upstream. However, a quick glance at the code in xen-unstable shows that this is probably a problem there as well; only testing will tell for sure.

diff -urp xen-3.1.0-src.orig/tools/python/xen/xend/server/BlktapController.py xen-3.1.0-src/tools/python/xen/xend/server/BlktapController.py
@@ -16,6 +16,11 @@ class BlktapController(BlkifController):
     def __init__(self, vm):
         BlkifController.__init__(self, vm)
         
+    def deviceRoot(self):
+        """@see DevController#deviceRoot"""
+
+        return "%s/device/vbd" % self.vm.vmpath
+
     def frontendRoot(self):
         """@see DevController#frontendRoot"""
         

diff -urp xen-3.1.0-src.orig/tools/python/xen/xend/server/DevController.py xen-3.1.0-src/tools/python/xen/xend/server/DevController.py
@@ -221,6 +221,8 @@ class DevController:
         # drivers, so this ordering avoids a race).
         self.writeBackend(devid, 'online', "0")
         self.writeBackend(devid, 'state', str(xenbusState['Closing']))
+        devpath = self.devicePath(devid)
+        xstransact.Remove(devpath)
 
         if force:
             frontpath = self.frontendPath(devid)

@@ -487,8 +489,7 @@ class DevController:
     def devicePath(self, devid):
         """Return the /device entry of the given VM. We use it to store
         backend/frontend locations"""
-        return "%s/device/%s/%s" % (self.vm.vmpath,
-                                    self.deviceClass, devid)
+        return "%s/%s" % (self.deviceRoot(), devid)
 
 def hotplugStatusCallback(statusPath, ev, result):
     log.debug("hotplugStatusCallback %s.", statusPath)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Hi Chris,

I could not reproduce the problem by using the latest xen-unstable.

I also found a problem of block tap devices included by c/s 18562, then I have fixed the problem by c/s 18843. But the problem occurred by using xm shutdown or xm destroy or etc, not xm block-detach.

Could you try xm block-detach by using the latest xen-unstable?

Best regards,
Kan

OK, interesting. I'll give it a shot, but it's going to take a little while since I have to build from scratch. I'll report when I'm done.

Thanks,

Chris Lalancette

Ah, now I see. Testing it on xen-unstable does, indeed, show xm block-detach working as expected. There were some changes made in the meantime that actually make it work. That means the first hunk of my changes to DevController.py aren't required. However, I think the other two hunks are actually "correct", even though we don't see the xm block-detach bug in current xen-unstable. That is, they move the device section from /vm/UUID/device/tap to /vm/UUID/device/vbd, which seems more right to me.

Chris Lalancette

Hi Chris,

I have tried the other two hunks of your changes on the latest xen-unstable. I have found two issues. Could you see the attaching file?

1. Information of xm list
When I tried xm list to an active domain, both "vbd" and "tap" were shown. They were same uuid and same uname.

2. Double wait in xend
I checked xend.log after I started a domain. Xend was waiting for both "vdb" and "tap" by using waitForDevices().

If the other two hunks of your changes are correct, I think that there is some kind of lack in your changes yet.

Best regards,
Kan

# xm new vm1
Using config file "/etc/xen/vm1".
# xm list --long vm1
(domain
    (on_crash restart)
    (uuid e3d061d3-d816-c66c-50b9-afcfb44c2203)
    (bootloader_args )
    (vcpus 1)
    (name vm1)
    (on_poweroff destroy)
    (on_reboot restart)
    (cpus (()))
    (bootloader )
    (maxmem 512)
    (memory 256)
    (shadow_memory 0)
    (features )
    (on_xend_start ignore)
    (on_xend_stop ignore)
    (image
        (linux
            (kernel /boot/vmlinux-2.6.18.8-xen)
            (ramdisk /boot/initrd-2.6.18.8-xen-vm1.img)
            (args 'root=/dev/hda1 ro 3 xencons=xvc console=xvc0')
        )
    )
    (status 0)
    (device
        (vif
            (mac 00:16:3e:6e:01:76)
            (uuid 87a00c88-fb69-6500-f8a5-40a50668caec)
        )
    )
    (device
        (tap
            (bootable 1)
            (uname tap:aio:/xen/root-vm1.img)
            (mode w)
            (dev hda1)
            (uuid dedcd624-3567-97d6-496e-5e8feafd8581)
        )
    )
)
# xm start vm1
# xm list --long vm1
(domain
    (domid 2)
    (on_crash restart)
    (uuid e3d061d3-d816-c66c-50b9-afcfb44c2203)
    (bootloader_args )
    (vcpus 1)
    (name vm1)
    (on_poweroff destroy)
    (on_reboot restart)
    (cpus (()))
    (bootloader )
    (maxmem 512)
    (memory 256)
    (shadow_memory 0)
    (features )
    (on_xend_start ignore)
    (on_xend_stop ignore)
    (start_time 1228267781.72)
    (cpu_time 0.864988181)
    (online_vcpus 1)
    (image
        (linux
            (kernel /boot/vmlinux-2.6.18.8-xen)
            (ramdisk /boot/initrd-2.6.18.8-xen-vm1.img)
            (args 'root=/dev/hda1 ro 3 xencons=xvc console=xvc0')
            (notes
                (HV_START_LOW 4118806528)
                (FEATURES
                    'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel'
                )
                (VIRT_BASE 3221225472)
                (GUEST_VERSION 2.6)
                (PADDR_OFFSET 3221225472)
                (GUEST_OS linux)
                (HYPERCALL_PAGE 3222278144)
                (LOADER generic)
                (SUSPEND_CANCEL 1)
                (PAE_MODE yes)
                (ENTRY 3222274048)
                (XEN_VERSION xen-3.0)
            )
        )
    )
    (status 2)
    (state -b----)
    (store_mfn 71728)
    (console_mfn 71727)
    (device
        (vif
            (mac 00:16:3e:6e:01:76)
            (script /etc/xen/scripts/vif-bridge)
            (uuid 87a00c88-fb69-6500-f8a5-40a50668caec)
            (backend 0)
        )
    )
    (device
        (vbd
            (protocol x86_32-abi)
            (uuid dedcd624-3567-97d6-496e-5e8feafd8581)
            (dev hda1:disk)
            (uname tap:aio:/xen/root-vm1.img)
            (mode w)
            (backend 0)
            (bootable 1)
            (VDI )
        )
    )
    (device
        (console
            (protocol vt100)
            (location 2)
            (uuid e878c50a-5dae-4b7e-168b-40dc569ea45b)
        )
    )
    (device
        (tap
            (protocol x86_32-abi)
            (uuid dedcd624-3567-97d6-496e-5e8feafd8581)
            (dev hda1:disk)
            (uname tap:aio:/xen/root-vm1.img)
            (mode w)
            (backend 0)
            (bootable 1)
            (VDI )
        )
    )
)
# xenstore-ls
tool = ""
 xenstored = ""
local = ""
 domain = ""
  0 = ""
   vm = "/vm/00000000-0000-0000-0000-000000000000"
   device = ""
   control = ""
    platform-feature-multiprocessor-suspend = "1"
   error = ""
   memory = ""
    target = "762880"
   cpu = ""
    1 = ""
     availability = "online"
    0 = ""
     availability = "online"
   name = "Domain-0"
   console = ""
    limit = "1048576"
    type = "xenconsoled"
   domid = "0"
   backend = ""
    tap = ""
     2 = ""
      769 = ""
       domain = "vm1"
       frontend = "/local/domain/2/device/vbd/769"
       uuid = "dedcd624-3567-97d6-496e-5e8feafd8581"
       dev = "hda1"
       state = "4"
       params = "aio:/xen/root-vm1.img"
       mode = "w"
       online = "1"
       frontend-id = "2"
       type = "tap"
       sectors = "8385867"
       sector-size = "512"
       info = "0"
       hotplug-status = "connected"
    vif = ""
     2 = ""
      0 = ""
       domain = "vm1"
       handle = "0"
       uuid = "87a00c88-fb69-6500-f8a5-40a50668caec"
       script = "/etc/xen/scripts/vif-bridge"
       state = "4"
       frontend = "/local/domain/2/device/vif/0"
       mac = "00:16:3e:6e:01:76"
       online = "1"
       frontend-id = "2"
       feature-sg = "1"
       feature-gso-tcpv4 = "1"
       feature-rx-copy = "1"
       feature-rx-flip = "0"
       hotplug-status = "connected"
    console = ""
     2 = ""
      0 = ""
       domain = "vm1"
       protocol = "vt100"
       uuid = "e878c50a-5dae-4b7e-168b-40dc569ea45b"
       frontend = "/local/domain/2/device/console/0"
       state = "1"
       location = "2"
       online = "1"
       frontend-id = "2"
  2 = ""
   vm = "/vm/e3d061d3-d816-c66c-50b9-afcfb44c2203"
   device = ""
    vbd = ""
     769 = ""
      virtual-device = "769"
      device-type = "disk"
      protocol = "x86_32-abi"
      backend-id = "0"
      state = "4"
      backend = "/local/domain/0/backend/tap/2/769"
      ring-ref = "8"
      event-channel = "7"
    vif = ""
     0 = ""
      mac = "00:16:3e:6e:01:76"
      handle = "0"
      protocol = "x86_32-abi"
      backend-id = "0"
      state = "4"
      backend = "/local/domain/0/backend/vif/2/0"
      tx-ring-ref = "768"
      rx-ring-ref = "769"
      event-channel = "8"
      request-rx-copy = "1"
      feature-rx-notify = "1"
      feature-no-csum-offload = "0"
      feature-sg = "1"
      feature-gso-tcpv4 = "1"
    console = ""
     0 = ""
      protocol = "x86_32-abi"
      state = "1"
      backend-id = "0"
      backend = "/local/domain/0/backend/console/2/0"
    suspend = ""
     event-channel = "6"
   control = ""
    platform-feature-multiprocessor-suspend = "1"
   error = ""
   memory = ""
    target = "262144"
   device-misc = ""
    vif = ""
     nextDeviceID = "1"
    console = ""
     nextDeviceID = "1"
   console = ""
    ring-ref = "71727"
    port = "2"
    limit = "1048576"
    type = "xenconsoled"
    tty = "/dev/pts/1"
   image = ""
    entry = "3222274048"
    loader = "generic"
    hv-start-low = "4118806528"
    guest-os = "linux"
    features = ""
     writable-descriptor-tables = "1"
     supervisor-mode-kernel = "1"
     pae-pgdir-above-4gb = "1"
     writable-page-tables = "1"
     auto-translated-physmap = "1"
    hypercall-page = "3222278144"
    guest-version = "2.6"
    pae-mode = "yes"
    paddr-offset = "3221225472"
    virt-base = "3221225472"
    suspend-cancel = "1"
    xen-version = "xen-3.0"
   store = ""
    ring-ref = "71728"
    port = "1"
   cpu = ""
    0 = ""
     availability = "online"
   name = "vm1"
   domid = "2"
   serial = ""
    0 = ""
     tty = "/dev/pts/1"
vm = ""
 00000000-0000-0000-0000-000000000000 = ""
  on_xend_stop = "ignore"
  shadow_memory = "0"
  uuid = "00000000-0000-0000-0000-000000000000"
  on_reboot = "restart"
  image = "(linux (kernel ))"
   ostype = "linux"
   kernel = ""
   cmdline = ""
   ramdisk = ""
  on_poweroff = "destroy"
  on_xend_start = "ignore"
  on_crash = "restart"
  xend = ""
   restart_count = "0"
  vcpus = "2"
  vcpu_avail = "3"
  name = "Domain-0"
  memory = "745"
 e3d061d3-d816-c66c-50b9-afcfb44c2203 = ""
  image = "(linux (kernel /boot/vmlinux-2.6.18.8-xen) (ramdisk /boot/initrd-2.6.18.8-xen-vm1.img) (args 'root=/dev/hda1\..."
   ostype = "linux"
   kernel = "/boot/vmlinux-2.6.18.8-xen"
   cmdline = "root=/dev/hda1 ro 3 xencons=xvc console=xvc0"
   ramdisk = "/boot/initrd-2.6.18.8-xen-vm1.img"
  device = ""
   vbd = ""
    769 = ""
     frontend = "/local/domain/2/device/vbd/769"
     frontend-id = "2"
     backend-id = "0"
     backend = "/local/domain/0/backend/tap/2/769"
   vif = ""
    0 = ""
     frontend = "/local/domain/2/device/vif/0"
     frontend-id = "2"
     backend-id = "0"
     backend = "/local/domain/0/backend/vif/2/0"
   console = ""
    0 = ""
     frontend = "/local/domain/2/device/console/0"
     frontend-id = "2"
     backend-id = "0"
     backend = "/local/domain/0/backend/console/2/0"
  on_xend_stop = "ignore"
  shadow_memory = "0"
  uuid = "e3d061d3-d816-c66c-50b9-afcfb44c2203"
  on_reboot = "restart"
  start_time = "1228267781.72"
  on_poweroff = "destroy"
  on_xend_start = "ignore"
  on_crash = "restart"
  xend = ""
   restart_count = "0"
  vcpus = "1"
  vcpu_avail = "1"
  name = "vm1"

[2008-12-03 10:29:41 4121] DEBUG (XendDomainInfo:2077) XendDomainInfo.constructDomain
[2008-12-03 10:29:41 4121] DEBUG (balloon:132) Balloon: 265152 KiB free; need 2048; done.
[2008-12-03 10:29:41 4121] DEBUG (XendDomain:452) Adding Domain: 2
[2008-12-03 10:29:41 4121] DEBUG (XendDomainInfo:2246) XendDomainInfo.initDomain: 2 256
[2008-12-03 10:29:41 4121] DEBUG (XendDomainInfo:2276) _initDomain:shadow_memory=0x0, memory_static_max=0x20000000, memory_static_min=0x0. [2008-12-03 10:29:41 4121] DEBUG (balloon:132) Balloon: 265152 KiB free; need 264192; done.
[2008-12-03 10:29:41 4121] INFO (image:166) buildDomain os=linux dom=2 vcpus=1

[2008-12-03 10:29:41 4121] DEBUG (image:639) domid          = 2
[2008-12-03 10:29:41 4121] DEBUG (image:640) memsize        = 256
[2008-12-03 10:29:41 4121] DEBUG (image:641) image          = /boot/vmlinux-2.6.18.8-xen
[2008-12-03 10:29:41 4121] DEBUG (image:642) store_evtchn   = 1
[2008-12-03 10:29:41 4121] DEBUG (image:643) console_evtchn = 2
[2008-12-03 10:29:41 4121] DEBUG (image:644) cmdline        = root=/dev/hda1 ro 3 xencons=xvc console=xvc0
[2008-12-03 10:29:41 4121] DEBUG (image:645) ramdisk        = /boot/initrd-2.6.18.8-xen-vm1.img
[2008-12-03 10:29:41 4121] DEBUG (image:646) vcpus          = 1
[2008-12-03 10:29:41 4121] DEBUG (image:647) features       = 
[2008-12-03 10:29:41 4121] DEBUG (image:648) flags          = 0

[2008-12-03 10:29:41 4121] INFO (XendDomainInfo:1941) createDevice: tap : {'bootable': 1, 'uname': 'tap:aio:/xen/root-vm1.img', 'mode': 'w', 'dev': 'hda1', 'uuid': 'dedcd624-3567-97d6-496e-5e8feafd8581'}
[2008-12-03 10:29:41 4121] DEBUG (DevController:95) DevController: writing {'virtual-device': '769', 'device-type': 'disk', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/tap/2/769'} to /local/domain/2/device/vbd/769.
[2008-12-03 10:29:41 4121] DEBUG (DevController:97) DevController: writing {'domain': 'vm1', 'frontend': '/local/domain/2/device/vbd/769', 'uuid': 'dedcd624-3567-97d6-496e-5e8feafd8581', 'dev': 'hda1', 'state': '1', 'params': 'aio:/xen/root-vm1.img', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type': 'tap'} to /local/domain/0/backend/tap/2/769.
[2008-12-03 10:29:41 4121] INFO (XendDomainInfo:1941) createDevice: vif : {'mac': '00:16:3e:6e:01:76', 'uuid': '87a00c88-fb69-6500-f8a5-40a50668caec'}
[2008-12-03 10:29:41 4121] DEBUG (DevController:95) DevController: writing {'mac': '00:16:3e:6e:01:76', 'handle': '0', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/2/0'} to /local/domain/2/device/vif/0.
[2008-12-03 10:29:41 4121] DEBUG (DevController:97) DevController: writing {'domain': 'vm1', 'frontend': '/local/domain/2/device/vif/0', 'uuid': '87a00c88-fb69-6500-f8a5-40a50668caec', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:16:3e:6e:01:76', 'frontend-id': '2', 'state': '1', 'online': '1', 'handle': '0'} to /local/domain/0/backend/vif/2/0.
[2008-12-03 10:29:41 4121] DEBUG (XendDomainInfo:2780) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': 'e3d061d3-d816-c66c-50b9-afcfb44c2203', 'on_reboot': 'restart', 'start_time': '1228267781.72', 'on_poweroff': 'destroy', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'image': "(linux (kernel /boot/vmlinux-2.6.18.8-xen) (ramdisk /boot/initrd-2.6.18.8-xen-vm1.img) (args 'root=/dev/hda1 ro 3 xencons=xvc console=xvc0') (notes (HV_START_LOW 4118806528) (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 3221225472) (GUEST_OS linux) (HYPERCALL_PAGE 3222278144) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 3222274048) (XEN_VERSION xen-3.0)))", 'name': 'vm1'} [2008-12-03 10:29:41 4121] DEBUG (XendDomainInfo:1405) Storing domain details: {'console/ring-ref': '71727', 'image/entry': '3222274048', 'console/port': '2', 'store/ring-ref': '71728', 'image/loader': 'generic', 'vm': '/vm/e3d061d3-d816-c66c-50b9-afcfb44c2203', 'control/platform-feature-multiprocessor-suspend': '1', 'image/hv-start-low': '4118806528', 'image/guest-os': 'linux', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '3221225472', 'memory/target': '262144', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/pae-mode': 'yes', 'console/limit': '1048576', 'image/paddr-offset': '3221225472', 'image/hypercall-page': '3222278144', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'console/type': 'xenconsoled', 'image/features/auto-translated-physmap': '1', 'name': 'vm1', 'domid': '2', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2008-12-03 10:29:42 4121] DEBUG (DevController:95) DevController: writing {'protocol': 'x86_32-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/2/0'} to /local/domain/2/device/console/0.
[2008-12-03 10:29:42 4121] DEBUG (DevController:97) DevController: writing {'domain': 'vm1', 'frontend': '/local/domain/2/device/console/0', 'uuid': 'e878c50a-5dae-4b7e-168b-40dc569ea45b', 'frontend-id': '2', 'state': '1', 'location': '2', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/2/0.
[2008-12-03 10:29:42 4121] DEBUG (XendDomainInfo:1489) XendDomainInfo.handleShutdownWatch
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices vif.
[2008-12-03 10:29:42 4121] DEBUG (DevController:144) Waiting for 0.
[2008-12-03 10:29:42 4121] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status.
[2008-12-03 10:29:42 4121] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status.
[2008-12-03 10:29:42 4121] DEBUG (DevController:642) hotplugStatusCallback 1.
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices vscsi.
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices vbd.
[2008-12-03 10:29:42 4121] DEBUG (DevController:144) Waiting for 769.
[2008-12-03 10:29:42 4121] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/tap/2/769/hotplug-status.
[2008-12-03 10:29:42 4121] DEBUG (DevController:642) hotplugStatusCallback 1.
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices irq.
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices vkbd.
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices vfb.
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices console.
[2008-12-03 10:29:42 4121] DEBUG (DevController:144) Waiting for 0.
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices pci.
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices ioports.
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices tap.
[2008-12-03 10:29:42 4121] DEBUG (DevController:144) Waiting for 769.
[2008-12-03 10:29:42 4121] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/tap/2/769/hotplug-status.
[2008-12-03 10:29:42 4121] DEBUG (DevController:642) hotplugStatusCallback 1.
[2008-12-03 10:29:42 4121] DEBUG (DevController:139) Waiting for devices vtpm.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Yes, I see what you mean (both from the data you sent, and from reproducing it on my own). There is probably something missing. Honestly, I'm not sure how sxpr's are generated, so I'll have to look into it. Probably not until tomorrow though. Thanks for testing.

Chris Lalancette