Skip to contentSkip to conversation

Inform users about busy device assignment attempt

PATCH In review Alexander Graf 2 participants 7 messages Dec 9, 2009 KVM

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Inform users about busy device assignment attempt

When using -pcidevice on a device that is already in use by a kernel driver all the user gets is the following (very useful) information:

Failed to assign device "04:00.0" : Device or resource busy Failed to deassign device "04:00.0" : Invalid argument Error initializing device pci-assign

Since I usually prefer to have my computer do the thinking for me, I figured it might be a good idea to check and see if a device is actually used by a driver. If so, tell the user.

So with this patch applied you get the following output:

Failed to assign device "04:00.0" : Device or resource busy *** The driver 'igb' is occupying your device 04:00.0. *** Try running "rmmod igb" on the commandline Failed to deassign device "04:00.0" : Invalid argument Error initializing device pci-assign

That should keep people like me from doing the most obvious misuses :-).

---
 hw/device-assignment.c |   21 +++++++++++++++++++--
 1 files changed, 19 insertions(+), 2 deletions(-)
hw/device-assignment.c +19 -2
index beb488c..2612c25 100644
@@ -747,7 +747,8 @@ static uint32_t calc_assigned_dev_id(uint8_t bus, uint8_t devfn)
 static int assign_device(AssignedDevice *dev)
 {
     struct kvm_assigned_pci_dev assigned_dev_data;
-    int r;
+    char dir[128], driver[128], *ns;
+    int r, v;
 
     memset(&assigned_dev_data, 0, sizeof(assigned_dev_data));
     assigned_dev_data.assigned_dev_id  =
@@ -769,9 +770,25 @@ static int assign_device(AssignedDevice *dev)
 #endif
 
     r = kvm_assign_pci_device(kvm_context, &assigned_dev_data);
-    if (r < 0)
+    if (r < 0) {
 	fprintf(stderr, "Failed to assign device \"%s\" : %s\n",
                 dev->dev.qdev.id, strerror(-r));
+
+        snprintf(dir, sizeof(dir),
+                 "/sys/bus/pci/devices/0000:%02x:%02x.%x/driver",
+	         dev->host.bus, dev->host.dev, dev->host.func);
+
+        v = readlink(dir, driver, sizeof(driver));
+        if ((v > 0) && (ns = strrchr(driver, '/'))) {
+            ns++;
+            fprintf(stderr, "*** The driver '%s' is occupying your "
+                                 "device %02x:%02x.%x.\n"
+                            "*** Try running \"rmmod %s\" on the command"
+                                 "line\n",
+                            ns, dev->host.bus, dev->host.dev, dev->host.func,
+                            ns);
+        }
+    }
     return r;
 }
 
Signature · 1 line
1.6.0.2
#1 lore raw
Quoted 13 lines · Alexander Graf

When using -pcidevice on a device that is already in use by a kernel driver all the user gets is the following (very useful) information:

Failed to assign device "04:00.0" : Device or resource busy Failed to deassign device "04:00.0" : Invalid argument Error initializing device pci-assign

Since I usually prefer to have my computer do the thinking for me, I figured it might be a good idea to check and see if a device is actually used by a driver. If so, tell the user.

So with this patch applied you get the following output:

Quoted 7 lines

Failed to assign device "04:00.0" : Device or resource busy *** The driver 'igb' is occupying your device 04:00.0. *** Try running "rmmod igb" on the commandline Failed to deassign device "04:00.0" : Invalid argument Error initializing device pci-assign

That should keep people like me from doing the most obvious misuses :-).

Unconditionally telling people to run rmmod is a pretty dangerous thing todo. If they typod and gave the PCI addr of their disk controller instead of the NIC, they'll be less than happy at the results of our recommended command to "fix" the error. Likewise if they have multiple devices using the same driver & just want to assign one of them. I think it is safer to just have the first bit of your proposed error message

"The device 04:00.0 is in use by the kernel driver 'igb'."

NB 'rmmod' is not the ideal approach for PCI assignment. It is better to explicitly re-bind the device to 'pcistub' because that ensures that no other driver will ever be able to reclaim the device.

Regards,
Daniel

Signature · 4 lines
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
#2 lore raw
Quoted 31 lines · Daniel P. Berrange
Quoted 13 lines · Alexander Graf

When using -pcidevice on a device that is already in use by a kernel driver all the user gets is the following (very useful) information:

Failed to assign device "04:00.0" : Device or resource busy Failed to deassign device "04:00.0" : Invalid argument Error initializing device pci-assign

Since I usually prefer to have my computer do the thinking for me, I figured it might be a good idea to check and see if a device is actually used by a driver. If so, tell the user.

So with this patch applied you get the following output:

Quoted 8 lines

Failed to assign device "04:00.0" : Device or resource busy *** The driver 'igb' is occupying your device 04:00.0. *** Try running "rmmod igb" on the commandline Failed to deassign device "04:00.0" : Invalid argument Error initializing device pci-assign

That should keep people like me from doing the most obvious misuses :-).

Unconditionally telling people to run rmmod is a pretty dangerous thing todo. If they typod and gave the PCI addr of their disk controller instead of the NIC, they'll be less than happy at the results of our recommended command to "fix" the error. Likewise if they have multiple devices using the same driver & just want to assign one of them. I think it is safer to just have the first bit of your proposed error message

Quoted 7 lines

"The device 04:00.0 is in use by the kernel driver 'igb'."

NB 'rmmod' is not the ideal approach for PCI assignment. It is better to explicitly re-bind the device to 'pcistub' because that ensures that no other driver will ever be able to reclaim the device.

Oh - mind to get into detail there? It'd be great if we could tell users an even better way to unbind their device from the driver than rmmod :)

Alex

#3 lore raw
Quoted 12 lines · Alexander Graf
Quoted 11 lines · Daniel P. Berrange

Unconditionally telling people to run rmmod is a pretty dangerous thing todo. If they typod and gave the PCI addr of their disk controller instead of the NIC, they'll be less than happy at the results of our recommended command to "fix" the error. Likewise if they have multiple devices using the same driver & just want to assign one of them. I think it is safer to just have the first bit of your proposed error message

"The device 04:00.0 is in use by the kernel driver 'igb'."

Quoted 7 lines
Quoted 4 lines

NB 'rmmod' is not the ideal approach for PCI assignment. It is better to explicitly re-bind the device to 'pcistub' because that ensures that no other driver will ever be able to reclaim the device.

Oh - mind to get into detail there? It'd be great if we could tell users an even better way to unbind their device from the driver than rmmod :)

The direct low level sysfs way involves the following steps

// Tell pci-stub to accept a particular vendor+product ID binding

  # echo "8086 27cb" > /sys/bus/pci/drivers/pci-stub/new_id

// Remove device from existing PCI driver

  # echo "00:1d.3" > /sys/bus/pci/drivers/e1000/unbind

// Add device to pci-stub PCI driver

  # echo "00:1d.3" > /sys/bus/pci/drivers/pci-stub/bind

// Tell pci-stub to stop accepting a vendor+product ID binding

  # echo "8086 27cb" > /sys/bus/pci/drivers/pci-stub/remove_id

The reason for that last step is that if you have multiple devices of the same vendor+product, you don't want a later hotplug event to bind the new device to pci-stub too !

At a higher level this is also possible with libvirt's nodedev-XXX commands

// Find the name of device you want to detach

   # virsh nodedev-list --tree
    computer
     |
     +- net_computer_loopback
     +- pci_8086_2448
     |   |
     |   +- pci_104c_ac56
     |     
     +- pci_8086_27a0
     +- pci_8086_27a1
     |   |
     ...snip rest of info...

// Check it really is the device you want to detach

   # virsh nodedev-dumpxml pci_8086_27a1
   <device>
     <name>pci_8086_27a1</name>
     <parent>computer</parent>
     <driver>
       <name>pcieport-driver</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>0</bus>
       <slot>1</slot>
       <function>0</function>
       <product id='0x27a1'>Mobile 945GM/PM/GMS, 943/940GML and 945GT Express PCI Express Root Port</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
     </capability>
   </device>

// Detach it from your host OS

   # virsh nodedev-dettach pci_8086_27a1
   Device pci_8086_27a1 dettached

That nodedev-dettach command, does all 4 of the sysfs commands in one go. Those nodedev-XXX commands are all usable, even if you're not using libvirt for actually running guests. If you are using libvirt for running the guests, then you don't even need those steps. Simply set the attribute 'managed=yes' for the <hostdev> in the guest config, and libvirt does the device detach automatically & reattachs when shutting down the guest

Regards,
Daniel

Signature · 4 lines
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
#4 lore raw
Quoted 32 lines · Daniel P. Berrange
Quoted 12 lines · Alexander Graf
Quoted 10 lines · Daniel P. Berrange

Unconditionally telling people to run rmmod is a pretty dangerous thing todo. If they typod and gave the PCI addr of their disk controller instead of the NIC, they'll be less than happy at the results of our recommended command to "fix" the error. Likewise if they have multiple devices using the same driver & just want to assign one of them. I think it is safer to just have the first bit of your proposed error message

"The device 04:00.0 is in use by the kernel driver 'igb'."

Quoted 8 lines
Quoted 5 lines

NB 'rmmod' is not the ideal approach for PCI assignment. It is better to explicitly re-bind the device to 'pcistub' because that ensures that no other driver will ever be able to reclaim the device.

Oh - mind to get into detail there? It'd be great if we could tell users an even better way to unbind their device from the driver than rmmod :)

The direct low level sysfs way involves the following steps

// Tell pci-stub to accept a particular vendor+product ID binding

  # echo "8086 27cb" > /sys/bus/pci/drivers/pci-stub/new_id

// Remove device from existing PCI driver

  # echo "00:1d.3" > /sys/bus/pci/drivers/e1000/unbind

// Add device to pci-stub PCI driver

Quoted 9 lines
  # echo "00:1d.3" > /sys/bus/pci/drivers/pci-stub/bind

// Tell pci-stub to stop accepting a vendor+product ID binding

  # echo "8086 27cb" > /sys/bus/pci/drivers/pci-stub/remove_id

The reason for that last step is that if you have multiple devices of the same vendor+product, you don't want a later hotplug event to bind the new device to pci-stub too !

So what would you think if we'd just print out those 4 commands to the user, so people who don't use libvirt still get guidance when using -pcidevice :). They should be fairly easy to construct from the information we have.

Alex

#5 lore raw
Quoted 43 lines · Alexander Graf
Quoted 32 lines · Daniel P. Berrange
Quoted 12 lines · Alexander Graf
Quoted 10 lines · Daniel P. Berrange

Unconditionally telling people to run rmmod is a pretty dangerous thing todo. If they typod and gave the PCI addr of their disk controller instead of the NIC, they'll be less than happy at the results of our recommended command to "fix" the error. Likewise if they have multiple devices using the same driver & just want to assign one of them. I think it is safer to just have the first bit of your proposed error message

"The device 04:00.0 is in use by the kernel driver 'igb'."

Quoted 8 lines
Quoted 5 lines

NB 'rmmod' is not the ideal approach for PCI assignment. It is better to explicitly re-bind the device to 'pcistub' because that ensures that no other driver will ever be able to reclaim the device.

Oh - mind to get into detail there? It'd be great if we could tell users an even better way to unbind their device from the driver than rmmod :)

The direct low level sysfs way involves the following steps

// Tell pci-stub to accept a particular vendor+product ID binding

  # echo "8086 27cb" > /sys/bus/pci/drivers/pci-stub/new_id

// Remove device from existing PCI driver

  # echo "00:1d.3" > /sys/bus/pci/drivers/e1000/unbind

// Add device to pci-stub PCI driver

Quoted 9 lines
  # echo "00:1d.3" > /sys/bus/pci/drivers/pci-stub/bind

// Tell pci-stub to stop accepting a vendor+product ID binding

  # echo "8086 27cb" > /sys/bus/pci/drivers/pci-stub/remove_id

The reason for that last step is that if you have multiple devices of the same vendor+product, you don't want a later hotplug event to bind the new device to pci-stub too !

Quoted 4 lines

So what would you think if we'd just print out those 4 commands to the user, so people who don't use libvirt still get guidance when using -pcidevice :). They should be fairly easy to construct from the information we have.

I think it is better to put that information in the QEMU docs for the -pcidevice argument where it can be properly explained in detail, outlining the potential consequences of the actions. The error message could direct people to the docs.

Regards,
Daniel

Signature · 4 lines
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
#6 lore raw
Quoted 55 lines · Daniel P. Berrange
Quoted 47 lines · Alexander Graf
Quoted 35 lines · Daniel P. Berrange
Quoted 13 lines · Alexander Graf
Quoted 10 lines · Daniel P. Berrange

Unconditionally telling people to run rmmod is a pretty dangerous thing todo. If they typod and gave the PCI addr of their disk controller instead of the NIC, they'll be less than happy at the results of our recommended command to "fix" the error. Likewise if they have multiple devices using the same driver & just want to assign one of them. I think it is safer to just have the first bit of your proposed error message

"The device 04:00.0 is in use by the kernel driver 'igb'."

Quoted 10 lines
Quoted 6 lines

NB 'rmmod' is not the ideal approach for PCI assignment. It is better to explicitly re-bind the device to 'pcistub' because that ensures that no other driver will ever be able to reclaim the device.

Oh - mind to get into detail there? It'd be great if we could tell users an even better way to unbind their device from the driver than rmmod :)

The direct low level sysfs way involves the following steps

// Tell pci-stub to accept a particular vendor+product ID binding

  # echo "8086 27cb" > /sys/bus/pci/drivers/pci-stub/new_id

// Remove device from existing PCI driver

  # echo "00:1d.3" > /sys/bus/pci/drivers/e1000/unbind

// Add device to pci-stub PCI driver

Quoted 10 lines
  # echo "00:1d.3" > /sys/bus/pci/drivers/pci-stub/bind

// Tell pci-stub to stop accepting a vendor+product ID binding

  # echo "8086 27cb" > /sys/bus/pci/drivers/pci-stub/remove_id

The reason for that last step is that if you have multiple devices of the same vendor+product, you don't want a later hotplug event to bind the new device to pci-stub too !

Quoted 5 lines

So what would you think if we'd just print out those 4 commands to the user, so people who don't use libvirt still get guidance when using -pcidevice :). They should be fairly easy to construct from the information we have.

Quoted 5 lines

I think it is better to put that information in the QEMU docs for the -pcidevice argument where it can be properly explained in detail, outlining the potential consequences of the actions. The error message could direct people to the docs.

I personally like programs that guide me, so I guess it should be in both.

Alex