PV Passthrough PCI Device Hotplug Support (Tools Part)
RFC PATCH 1/3 PV Passthrough PCI Device Hotplug Support (Tools Part)
Tools part of PV PCI hotplug, applied to xen-unstable.hg cs17042.
Most of this part will be changed to adjust latest xen-unstable, since there already exists HVM PCI hotplug code.
Thanks,
-------------------
Yosuke Iwamatsu
NEC Corporation
Tools part of PV PCI hotplug, applied to xen-unstable.hg cs17042.
tools/python/xen/xend/XendDomainInfo.py +93 -0
@@ -546,6 +546,95 @@ class XendDomainInfo: xen.xend.XendDomain.instance().managed_config_save(self) return self.getDeviceController(dev_type).sxpr(devid) + def pci_device_configure(self, dev_sxp, devid = 0): + """Configure an existing pci device. + + @param dev_config: device configuration + @type dev_config: SXP object (parsed config) + @param devid: device id + @type devid: int + @return: Returns True if successfully updated device + @rtype: boolean + """ + + dev_class = sxp.name(dev_sxp) + dev_config = {} + + if dev_class != 'pci': + return False + + if self.info.is_hvm(): + return False + + # If pci platform does not exist, create and exit. + found = False + for existing_type, existing_info in self.info.all_devices_sxpr(): + if existing_type == 'pci': + found = True + break + if not found: + for pci_state in sxp.children(dev_sxp, 'state'): + try: + if pci_state[1] != 'Initialising': # not attach + raise XendError("Cannot detach when pci platform " + "not exist") + except IndexError: + raise XendError("Error reading state") + self.device_create(dev_sxp) + return True + + # Convert device sxp to a dict. + # In reconfigure case, the config of PCI device looks like below: + # + # sxp: + # [device, [pci, [dev, [domain, 0], [bus, 0], [slot, 0], [func, 0]], + # [state, 'Initialising']]] + # + # dict: + # {devs: [{slot: 0, bus: 0, domain: 0, func: 0}], + # states: ['Initialising']} + # + # 'state' parameter takes either 'Initialising' or 'Closing', + # which indicates attach or detach. + pci_devs = [] + for pci_dev in sxp.children(dev_sxp, 'dev'): + pci_dev_info = {} + for opt_val in pci_dev[1:]: + try: + opt, val = opt_val + pci_dev_info[opt] = val + except TypeError: + pass + pci_devs.append(pci_dev_info) + dev_config['devs'] = pci_devs + pci_states = [] + for pci_state in sxp.children(dev_sxp, 'state'): + try: + pci_states.append(pci_state[1]) + except IndexError: + raise XendError("Error reading state") + dev_config['states'] = pci_states + + # use DevController.reconfigureDevice to change device config + dev_control = self.getDeviceController(dev_class) + dev_uuid = dev_control.reconfigureDevice(devid, dev_config) + + dev_control.waitForDevice_reconfigure(devid) + num_devs = dev_control.cleanupDevice(devid) + + # update XendConfig with new device info + if dev_uuid: + new_dev_sxp = dev_control.configuration(devid) + self.info.device_update(dev_uuid, new_dev_sxp) + + # If there's no device connected, destroy pci here. + # To remove config of pci, self.destroyDevice() should be modified. + if num_devs == 0: + self.destroyDevice('pci', devid) + # self.destroyDevice('pci', devid, False, True) + + return True + def device_configure(self, dev_sxp, devid = None): """Configure an existing device.
@@ -560,6 +649,10 @@ class XendDomainInfo: # convert device sxp to a dict dev_class = sxp.name(dev_sxp) dev_config = {} + + if dev_class == 'pci': + return self.pci_device_configure(dev_sxp) + for opt_val in dev_sxp[1:]: try: dev_config[opt_val[0]] = opt_val[1]
tools/python/xen/xend/server/DevController.py +59 -0
98 lines not on this page · raw message
tools/python/xen/xend/server/pciif.py +172 -12
220 lines not on this page · raw message
tools/python/xen/xm/main.py +42 -0
66 lines not on this page · raw message
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 4 lines · Yosuke Iwamatsu
Tools part of PV PCI hotplug, applied to xen-unstable.hg cs17042.
Most of this part will be changed to adjust latest xen-unstable, since there already exists HVM PCI hotplug code.
I applied the other two patches (the Xen and Linux pieces) so please just re-submit this one.
-- Keir
Quoted 9 lines · Keir Fraser
Quoted 4 lines · Yosuke Iwamatsu
Tools part of PV PCI hotplug, applied to xen-unstable.hg cs17042.
Most of this part will be changed to adjust latest xen-unstable, since there already exists HVM PCI hotplug code.
I applied the other two patches (the Xen and Linux pieces) so please just re-submit this one.
Thanks!
Now I'm going to look into xend and see how I can get along with the existing HVM pcihp code.
-- Yosuke
PATCH 1/1 PV Passthrough PCI Device Hotplug Support (Tools Part)
Hi,
This is the updated patch of PV PCI passthrough hotplug (tools portion).
Changes to xm command:
- Let 'xm pci-attach' command use <domain:bus:slot.func> expression rather than use separated arguments like <domain> <bus> <slot> <func>. This is the same expression of the domain configuration file and I believe it is more user-friendly.
- Let 'xm pci-detach' command use the same <domain:bus:slot.func> expression. Formerly, <virtual slot> was used as the key to indicate which device we were detaching. But for the current implementation of PV PCI hotplug, virtual slots don't have much meaning. I think using the same physical device name in both attachment and detachment is appropriate here.
Changes to xend:
- Both attach and detach commands are received by device_configure() and handled by pci_device_configure().
- HVM specific processings are done in hvm_pci_create_device() and hvm_destroyPCIDevice().
The major problem I encoutered during this work is that, I don't have a vt-d enabled machine and could not confirm that HVM PCI hotplug would still work after modifications :-(
Then I hacked libxc a little to pretend as if there existed a vt-d hardware, and could see HVM PCI attach/detach working correctly. So now I'm pretty sure that we can do both PV and HVM hotplugs with this patch, but I would appreciate it if someone could test this on a real vt-d machine.
Regards,
-------------------
Yosuke Iwamatsu
NEC Corporation
# HG changeset patch
# User [email protected]
# Date 1205112166 -32400
# Node ID 6db8d8a026fc65da4cddd9508bd0da44637031ee # Parent 59b8768d0d0d3bf2125c33adafbd1256e75e2755
tools: add PV Passtrough PCI device hotplug support.
tools/python/xen/xend/XendConfig.py +17 -0
@@ -1407,6 +1407,23 @@ class XendConfig(dict): config = cfg_sxp dev_type, dev_info = self['devices'][dev_uuid] + + if dev_type == 'pci': # Special case for pci + pci_devs = [] + for pci_dev in sxp.children(config, 'dev'): + pci_dev_info = {} + for opt_val in pci_dev[1:]: + try: + opt, val = opt_val + pci_dev_info[opt] = val + except TypeError: + pass + pci_devs.append(pci_dev_info) + self['devices'][dev_uuid] = (dev_type, + {'devs': pci_devs, + 'uuid': dev_uuid}) + return True + for opt_val in config[1:]: try: opt, val = opt_val
tools/python/xen/xend/XendDomainInfo.py +158 -63
279 lines not on this page · raw message
tools/python/xen/xend/server/DevController.py +50 -0
83 lines not on this page · raw message
tools/python/xen/xend/server/pciif.py +197 -32
279 lines not on this page · raw message
tools/python/xen/xm/main.py +31 -24
83 lines not on this page · raw message
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Re: [PATCH] PV Passthrough PCI Device Hotplug Support(Tools Part)
Hi Iwamatsu,
def xm_pci_detach(args):
- arg_check(args, 'pci-detach', 2) - dom = args[0] - dev = args[1] - server.xend.domain.destroyDevice(dom, 'dpci', dev) + arg_check(args, 'pci-detach', 2, 2) <----------------- here!! + (dom, pci) = parse_pci_configuration(args, 'Closing') + server.xend.domain.device_configure(dom, pci)
+
The last "2" is not necessary.
With your patch, the following strange error message is shown.
# xm pci-detach
Error: 'xm pci-detach' requires between 2 and 2 arguments.
^^^^^^^^^^^^^^^
Usage: xm pci-detach <Domain> <virtual slot>Remove a domain's pass-through pci device.
Best regards,
Kan
Quoted 36 lines · Yosuke Iwamatsu
Hi,
This is the updated patch of PV PCI passthrough hotplug (tools portion).
Changes to xm command:
- Let 'xm pci-attach' command use <domain:bus:slot.func> expression rather than use separated arguments like <domain> <bus> <slot> <func>. This is the same expression of the domain configuration file and I believe it is more user-friendly.
- Let 'xm pci-detach' command use the same <domain:bus:slot.func> expression. Formerly, <virtual slot> was used as the key to indicate which device we were detaching. But for the current implementation of PV PCI hotplug, virtual slots don't have much meaning. I think using the same physical device name in both attachment and detachment is appropriate here.Changes to xend:
- Both attach and detach commands are received bydevice_configure()and handled bypci_device_configure().
- HVM specific processings are done inhvm_pci_create_device()andhvm_destroyPCIDevice().The major problem I encoutered during this work is that, I don't have a vt-d enabled machine and could not confirm that HVM PCI hotplug would still work after modifications :-(
Then I hacked libxc a little to pretend as if there existed a vt-d hardware, and could see HVM PCI attach/detach working correctly. So now I'm pretty sure that we can do both PV and HVM hotplugs with this patch, but I would appreciate it if someone could test this on a real vt-d machine.Regards,
-------------------
Yosuke Iwamatsu
NEC Corporation
Re: [PATCH] PV Passthrough PCI Device Hotplug Support(Tools Part)
Quoted 12 lines · Masaki Kanno
Hi Iwamatsu,
def
xm_pci_detach(args):(quoted diff) +4
-4- arg_check(args, 'pci-detach', 2) - dom = args[0] - dev = args[1] - server.xend.domain.destroyDevice(dom, 'dpci', dev) + arg_check(args, 'pci-detach', 2, 2) <----------------- here!! + (dom, pci) = parse_pci_configuration(args, 'Closing') + server.xend.domain.device_configure(dom, pci) +
Quoted 1 line
The last "2" is not necessary.
All right.
The revised patch is attached.
-- Yosuke
# HG changeset patch
# User [email protected]
# Date 1205116625 -32400
# Node ID 8da2f6ce93bc299274ffa73e06999d693d2513eb # Parent 59b8768d0d0d3bf2125c33adafbd1256e75e2755
tools: Add PV passthrough PCI device hotplug support.
tools/python/xen/xend/XendConfig.py +17 -0
@@ -1407,6 +1407,23 @@ class XendConfig(dict): config = cfg_sxp dev_type, dev_info = self['devices'][dev_uuid] + + if dev_type == 'pci': # Special case for pci + pci_devs = [] + for pci_dev in sxp.children(config, 'dev'): + pci_dev_info = {} + for opt_val in pci_dev[1:]: + try: + opt, val = opt_val + pci_dev_info[opt] = val + except TypeError: + pass + pci_devs.append(pci_dev_info) + self['devices'][dev_uuid] = (dev_type, + {'devs': pci_devs, + 'uuid': dev_uuid}) + return True + for opt_val in config[1:]: try: opt, val = opt_val
tools/python/xen/xend/XendDomainInfo.py +158 -63
279 lines not on this page · raw message
tools/python/xen/xend/server/DevController.py +50 -0
83 lines not on this page · raw message
tools/python/xen/xend/server/pciif.py +197 -32
279 lines not on this page · raw message
tools/python/xen/xm/main.py +30 -23
82 lines not on this page · raw message
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Yosuke,
Thanks for you work.
I have tested your patch works on HVM VT-d side.
One suggestion is error handling for user input. e.g. following input error should be caught by xm:
[root@vt-gang32 ~]# xm pci-attach EdwinHVMDomainVtd2 0:2.0.0 (should be
0:2:0.0)
Error: pci: failed to locate device and parse it's resources - [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:00:02.0/driver'
Usage: xm pci-attach <Domain> <domain:bus:slot.func> [virtual slot]
Quoted 36 lines · Yosuke Iwamatsu
Hi,
This is the updated patch of PV PCI passthrough hotplug (tools portion).
Changes to xm command:
- Let 'xm pci-attach' command use <domain:bus:slot.func> expression rather than use separated arguments like <domain> <bus> <slot> <func>. This is the same expression of the domain configuration file and I believe it is more user-friendly.
- Let 'xm pci-detach' command use the same <domain:bus:slot.func> expression. Formerly, <virtual slot> was used as the key to indicate which device we were detaching. But for the current implementation of PV PCI hotplug, virtual slots don't have much meaning. I think using the same physical device name in both attachment and detachment is appropriate here.Changes to xend:
- Both attach and detach commands are received bydevice_configure()and handled bypci_device_configure().
- HVM specific processings are done inhvm_pci_create_device()andhvm_destroyPCIDevice().The major problem I encoutered during this work is that, I don't have a vt-d enabled machine and could not confirm that HVM PCI hotplug would still work after modifications :-(
Then I hacked libxc a little to pretend as if there existed a vt-d hardware, and could see HVM PCI attach/detach working correctly. So now I'm pretty sure that we can do both PV and HVM hotplugs with this patch, but I would appreciate it if someone could test this on a real vt-d machine.Regards,
-------------------
Yosuke Iwamatsu
NEC Corporation
Signature · 2 lines
best rgds, edwin
Quoted 4 lines · Edwin
Yosuke,
Thanks for you work.
I have tested your patch works on HVM VT-d side.
I'm relieved to hear that. Thank you.
Quoted 7 lines
One suggestion is error handling for user input. e.g. following input error should be caught by xm:
[root@vt-gang32 ~]# xm pci-attach EdwinHVMDomainVtd2 0:2.0.0 (should be 0:2:0.0)
Error: pci: failed to locate device and parse it's resources - [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:00:02.0/driver'
Usage: xm pci-attach <Domain> <domain:bus:slot.func> [virtual slot]
Okay.
The small patch below should fix the issue.
(applied after the previous patch)
Thanks,
Yosuke Iwamatsu
---------
xm: Fix parsing pci device name.
tools/python/xen/xm/main.py +1 -1
@@ -2240,7 +2240,7 @@ def parse_pci_configuration(args, state) pci_match = re.match(r"((?P<domain>[0-9a-fA-F]{1,4})[:,])?" + \ r"(?P<bus>[0-9a-fA-F]{1,2})[:,]" + \ r"(?P<slot>[0-9a-fA-F]{1,2})[.,]" + \ - r"(?P<func>[0-9a-fA-F])", pci_dev_str) + r"(?P<func>[0-7])$", pci_dev_str) if pci_match == None: raise OptionError("Invalid argument: %s %s" % (pci_dev_str,vslt)) pci_dev_info = pci_match.groupdict('0')