Skip to contentSkip to conversation

PV Passthrough PCI Device Hotplug Support (Tools Part)

RFC PATCH In review Yosuke Iwamatsu 4 participants 8 messages Feb 21, 2008 – Mar 10, 2008

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/3 PV Passthrough PCI Device Hotplug Support (Tools Part)

Tools part of PV PCI hotplug, applied to xen-unstable.hg cs17042.

Most of this part will be changed to adjust latest xen-unstable, since there already exists HVM PCI hotplug code.

Thanks,
-------------------
Yosuke Iwamatsu
NEC Corporation

Tools part of PV PCI hotplug, applied to xen-unstable.hg cs17042.

tools/python/xen/xend/XendDomainInfo.py +93 -0
diff -r e1dde6f8bc87 tools/python/xen/xend/XendDomainInfo.py
@@ -546,6 +546,95 @@ class XendDomainInfo:
         xen.xend.XendDomain.instance().managed_config_save(self)
         return self.getDeviceController(dev_type).sxpr(devid)
 
+    def pci_device_configure(self, dev_sxp, devid = 0):
+        """Configure an existing pci device.
+        
+        @param dev_config: device configuration
+        @type  dev_config: SXP object (parsed config)
+        @param devid:      device id
+        @type  devid:      int
+        @return: Returns True if successfully updated device
+        @rtype: boolean
+        """
+
+        dev_class = sxp.name(dev_sxp)
+        dev_config = {}
+
+        if dev_class != 'pci':
+            return False
+
+        if self.info.is_hvm():
+            return False
+
+        # If pci platform does not exist, create and exit.
+        found = False
+        for existing_type, existing_info in self.info.all_devices_sxpr():
+            if existing_type == 'pci':
+                found = True
+                break
+        if not found:
+            for pci_state in sxp.children(dev_sxp, 'state'):
+                try:
+                    if pci_state[1] != 'Initialising': # not attach
+                        raise XendError("Cannot detach when pci platform "
+                                        "not exist")
+                except IndexError:
+                    raise XendError("Error reading state")
+            self.device_create(dev_sxp)
+            return True
+
+        # Convert device sxp to a dict.
+        # In reconfigure case, the config of PCI device looks like below:
+        #
+        # sxp:
+        # [device, [pci, [dev, [domain, 0], [bus, 0], [slot, 0], [func, 0]],
+        #                [state, 'Initialising']]]
+        #
+        # dict:
+        # {devs: [{slot: 0, bus: 0, domain: 0, func: 0}],
+        #  states: ['Initialising']}
+        #
+        # 'state' parameter takes either 'Initialising' or 'Closing',
+        #  which indicates attach or detach.
+        pci_devs = []
+        for pci_dev in sxp.children(dev_sxp, 'dev'):
+            pci_dev_info = {}
+            for opt_val in pci_dev[1:]:
+                try:
+                    opt, val = opt_val
+                    pci_dev_info[opt] = val
+                except TypeError:
+                    pass
+            pci_devs.append(pci_dev_info)
+        dev_config['devs'] = pci_devs
+        pci_states = []
+        for pci_state in sxp.children(dev_sxp, 'state'):
+            try:
+                pci_states.append(pci_state[1]) 
+            except IndexError:
+                raise XendError("Error reading state")
+        dev_config['states'] = pci_states
+
+        # use DevController.reconfigureDevice to change device config
+        dev_control = self.getDeviceController(dev_class)
+        dev_uuid = dev_control.reconfigureDevice(devid, dev_config)
+
+        dev_control.waitForDevice_reconfigure(devid)
+        num_devs = dev_control.cleanupDevice(devid)
+ 
+        # update XendConfig with new device info
+        if dev_uuid:
+            new_dev_sxp = dev_control.configuration(devid)
+            self.info.device_update(dev_uuid, new_dev_sxp)
+
+        # If there's no device connected, destroy pci here.
+        # To remove config of pci, self.destroyDevice() should be modified.
+        if num_devs == 0:
+            self.destroyDevice('pci', devid)
+        #    self.destroyDevice('pci', devid, False, True)
+
+        return True
+
     def device_configure(self, dev_sxp, devid = None):
         """Configure an existing device.
         
@@ -560,6 +649,10 @@ class XendDomainInfo:
         # convert device sxp to a dict
         dev_class = sxp.name(dev_sxp)
         dev_config = {}
+
+        if dev_class == 'pci':
+            return self.pci_device_configure(dev_sxp)
+
         for opt_val in dev_sxp[1:]:
             try:
                 dev_config[opt_val[0]] = opt_val[1]
tools/python/xen/xend/server/DevController.py +59 -0

98 lines not on this page · raw message

tools/python/xen/xend/server/pciif.py +172 -12

220 lines not on this page · raw message

tools/python/xen/xm/main.py +42 -0

66 lines not on this page · raw message

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#1 lore raw
Quoted 4 lines · Yosuke Iwamatsu

Tools part of PV PCI hotplug, applied to xen-unstable.hg cs17042.

Most of this part will be changed to adjust latest xen-unstable, since there already exists HVM PCI hotplug code.

I applied the other two patches (the Xen and Linux pieces) so please just re-submit this one.

-- Keir

#2 lore raw
Quoted 9 lines · Keir Fraser
Quoted 4 lines · Yosuke Iwamatsu

Tools part of PV PCI hotplug, applied to xen-unstable.hg cs17042.

Most of this part will be changed to adjust latest xen-unstable, since there already exists HVM PCI hotplug code.

I applied the other two patches (the Xen and Linux pieces) so please just re-submit this one.

Thanks!
Now I'm going to look into xend and see how I can get along with the existing HVM pcihp code.

-- Yosuke

#3 lore raw

PATCH 1/1 PV Passthrough PCI Device Hotplug Support (Tools Part)

Hi,

This is the updated patch of PV PCI passthrough hotplug (tools portion).

Changes to xm command:
- Let 'xm pci-attach' command use <domain:bus:slot.func> expression rather than use separated arguments like <domain> <bus> <slot> <func>. This is the same expression of the domain configuration file and I believe it is more user-friendly.
- Let 'xm pci-detach' command use the same <domain:bus:slot.func> expression. Formerly, <virtual slot> was used as the key to indicate which device we were detaching. But for the current implementation of PV PCI hotplug, virtual slots don't have much meaning. I think using the same physical device name in both attachment and detachment is appropriate here.

Changes to xend:
- Both attach and detach commands are received by device_configure() and handled by pci_device_configure().
- HVM specific processings are done in hvm_pci_create_device() and hvm_destroyPCIDevice().

The major problem I encoutered during this work is that, I don't have a vt-d enabled machine and could not confirm that HVM PCI hotplug would still work after modifications :-(
Then I hacked libxc a little to pretend as if there existed a vt-d hardware, and could see HVM PCI attach/detach working correctly. So now I'm pretty sure that we can do both PV and HVM hotplugs with this patch, but I would appreciate it if someone could test this on a real vt-d machine.

Regards,
-------------------
Yosuke Iwamatsu
NEC Corporation

# HG changeset patch
# User [email protected]
# Date 1205112166 -32400
# Node ID 6db8d8a026fc65da4cddd9508bd0da44637031ee # Parent 59b8768d0d0d3bf2125c33adafbd1256e75e2755
tools: add PV Passtrough PCI device hotplug support.

tools/python/xen/xend/XendConfig.py +17 -0
diff -r 59b8768d0d0d -r 6db8d8a026fc tools/python/xen/xend/XendConfig.py
@@ -1407,6 +1407,23 @@ class XendConfig(dict):
                 config = cfg_sxp
 
             dev_type, dev_info = self['devices'][dev_uuid]
+
+            if dev_type == 'pci': # Special case for pci
+                pci_devs = []
+                for pci_dev in sxp.children(config, 'dev'):
+                    pci_dev_info = {}
+                    for opt_val in pci_dev[1:]:
+                        try:
+                            opt, val = opt_val
+                            pci_dev_info[opt] = val
+                        except TypeError:
+                            pass
+                    pci_devs.append(pci_dev_info)
+                self['devices'][dev_uuid] = (dev_type,
+                                             {'devs': pci_devs,
+                                              'uuid': dev_uuid})
+                return True
+                
             for opt_val in config[1:]:
                 try:
                     opt, val = opt_val
tools/python/xen/xend/XendDomainInfo.py +158 -63

279 lines not on this page · raw message

tools/python/xen/xend/server/DevController.py +50 -0

83 lines not on this page · raw message

tools/python/xen/xend/server/pciif.py +197 -32

279 lines not on this page · raw message

tools/python/xen/xm/main.py +31 -24

83 lines not on this page · raw message

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#4 lore raw

Re: [PATCH] PV Passthrough PCI Device Hotplug Support(Tools Part)

Hi Iwamatsu,

def xm_pci_detach(args):

-    arg_check(args, 'pci-detach', 2)
-    dom = args[0]
-    dev = args[1]
-    server.xend.domain.destroyDevice(dom, 'dpci', dev)
+    arg_check(args, 'pci-detach', 2, 2)  <----------------- here!!
+    (dom, pci) = parse_pci_configuration(args, 'Closing')
+    server.xend.domain.device_configure(dom, pci)

+

The last "2" is not necessary.
With your patch, the following strange error message is shown.

# xm pci-detach
Error: 'xm pci-detach' requires between 2 and 2 arguments.
                                ^^^^^^^^^^^^^^^
Usage: xm pci-detach  <Domain> <virtual slot>

Remove a domain's pass-through pci device.

Best regards,
Kan

Quoted 36 lines · Yosuke Iwamatsu

Hi,

This is the updated patch of PV PCI passthrough hotplug (tools portion).

Changes to xm command:
- Let 'xm pci-attach' command use <domain:bus:slot.func> expression rather than use separated arguments like <domain> <bus> <slot> <func>. This is the same expression of the domain configuration file and I believe it is more user-friendly.
- Let 'xm pci-detach' command use the same <domain:bus:slot.func> expression. Formerly, <virtual slot> was used as the key to indicate which device we were detaching. But for the current implementation of PV PCI hotplug, virtual slots don't have much meaning. I think using the same physical device name in both attachment and detachment is appropriate here.

Changes to xend:
- Both attach and detach commands are received by device_configure() and handled by pci_device_configure().
- HVM specific processings are done in hvm_pci_create_device() and hvm_destroyPCIDevice().

The major problem I encoutered during this work is that, I don't have a vt-d enabled machine and could not confirm that HVM PCI hotplug would still work after modifications :-(
Then I hacked libxc a little to pretend as if there existed a vt-d hardware, and could see HVM PCI attach/detach working correctly. So now I'm pretty sure that we can do both PV and HVM hotplugs with this patch, but I would appreciate it if someone could test this on a real vt-d machine.

Regards,
-------------------
Yosuke Iwamatsu
NEC Corporation

#5 lore raw

Re: [PATCH] PV Passthrough PCI Device Hotplug Support(Tools Part)

Quoted 12 lines · Masaki Kanno

Hi Iwamatsu,

def xm_pci_detach(args):

(quoted diff) +4 -4
-    arg_check(args, 'pci-detach', 2)
-    dom = args[0]
-    dev = args[1]
-    server.xend.domain.destroyDevice(dom, 'dpci', dev)
+    arg_check(args, 'pci-detach', 2, 2)  <----------------- here!!
+    (dom, pci) = parse_pci_configuration(args, 'Closing')
+    server.xend.domain.device_configure(dom, pci)
+
Quoted 1 line

The last "2" is not necessary.

All right.
The revised patch is attached.

-- Yosuke

# HG changeset patch
# User [email protected]
# Date 1205116625 -32400
# Node ID 8da2f6ce93bc299274ffa73e06999d693d2513eb # Parent 59b8768d0d0d3bf2125c33adafbd1256e75e2755
tools: Add PV passthrough PCI device hotplug support.

tools/python/xen/xend/XendConfig.py +17 -0
diff -r 59b8768d0d0d -r 8da2f6ce93bc tools/python/xen/xend/XendConfig.py
@@ -1407,6 +1407,23 @@ class XendConfig(dict):
                 config = cfg_sxp
 
             dev_type, dev_info = self['devices'][dev_uuid]
+
+            if dev_type == 'pci': # Special case for pci
+                pci_devs = []
+                for pci_dev in sxp.children(config, 'dev'):
+                    pci_dev_info = {}
+                    for opt_val in pci_dev[1:]:
+                        try:
+                            opt, val = opt_val
+                            pci_dev_info[opt] = val
+                        except TypeError:
+                            pass
+                    pci_devs.append(pci_dev_info)
+                self['devices'][dev_uuid] = (dev_type,
+                                             {'devs': pci_devs,
+                                              'uuid': dev_uuid})
+                return True
+                
             for opt_val in config[1:]:
                 try:
                     opt, val = opt_val
tools/python/xen/xend/XendDomainInfo.py +158 -63

279 lines not on this page · raw message

tools/python/xen/xend/server/DevController.py +50 -0

83 lines not on this page · raw message

tools/python/xen/xend/server/pciif.py +197 -32

279 lines not on this page · raw message

tools/python/xen/xm/main.py +30 -23

82 lines not on this page · raw message

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#6 lore raw
In reply to Yosuke Iwamatsu

Yosuke,

Thanks for you work.
I have tested your patch works on HVM VT-d side.

One suggestion is error handling for user input. e.g. following input error should be caught by xm:

[root@vt-gang32 ~]# xm pci-attach EdwinHVMDomainVtd2 0:2.0.0 (should be
0:2:0.0)

Error: pci: failed to locate device and parse it's resources - [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:00:02.0/driver'
Usage: xm pci-attach <Domain> <domain:bus:slot.func> [virtual slot]

Quoted 36 lines · Yosuke Iwamatsu

Hi,

This is the updated patch of PV PCI passthrough hotplug (tools portion).

Changes to xm command:
- Let 'xm pci-attach' command use <domain:bus:slot.func> expression rather than use separated arguments like <domain> <bus> <slot> <func>. This is the same expression of the domain configuration file and I believe it is more user-friendly.
- Let 'xm pci-detach' command use the same <domain:bus:slot.func> expression. Formerly, <virtual slot> was used as the key to indicate which device we were detaching. But for the current implementation of PV PCI hotplug, virtual slots don't have much meaning. I think using the same physical device name in both attachment and detachment is appropriate here.

Changes to xend:
- Both attach and detach commands are received by device_configure() and handled by pci_device_configure().
- HVM specific processings are done in hvm_pci_create_device() and hvm_destroyPCIDevice().

The major problem I encoutered during this work is that, I don't have a vt-d enabled machine and could not confirm that HVM PCI hotplug would still work after modifications :-(
Then I hacked libxc a little to pretend as if there existed a vt-d hardware, and could see HVM PCI attach/detach working correctly. So now I'm pretty sure that we can do both PV and HVM hotplugs with this patch, but I would appreciate it if someone could test this on a real vt-d machine.

Regards,
-------------------
Yosuke Iwamatsu
NEC Corporation

Signature · 2 lines
best rgds,
edwin
#7 lore raw
Quoted 4 lines · Edwin

Yosuke,

Thanks for you work.
I have tested your patch works on HVM VT-d side.

I'm relieved to hear that. Thank you.

Quoted 7 lines

One suggestion is error handling for user input. e.g. following input error should be caught by xm:

[root@vt-gang32 ~]# xm pci-attach EdwinHVMDomainVtd2 0:2.0.0 (should be 0:2:0.0)

Error: pci: failed to locate device and parse it's resources - [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:00:02.0/driver'
Usage: xm pci-attach <Domain> <domain:bus:slot.func> [virtual slot]

Okay.
The small patch below should fix the issue.
(applied after the previous patch)

Thanks,
Yosuke Iwamatsu

---------

xm: Fix parsing pci device name.

tools/python/xen/xm/main.py +1 -1
diff -r 9233270dfe6a tools/python/xen/xm/main.py
@@ -2240,7 +2240,7 @@ def parse_pci_configuration(args, state)
     pci_match = re.match(r"((?P<domain>[0-9a-fA-F]{1,4})[:,])?" + \
             r"(?P<bus>[0-9a-fA-F]{1,2})[:,]" + \
             r"(?P<slot>[0-9a-fA-F]{1,2})[.,]" + \
-            r"(?P<func>[0-9a-fA-F])", pci_dev_str)
+            r"(?P<func>[0-7])$", pci_dev_str)
     if pci_match == None:
         raise OptionError("Invalid argument: %s %s" % (pci_dev_str,vslt))
     pci_dev_info = pci_match.groupdict('0')