Skip to contentSkip to conversation

Remove uevent suppression logic from css driver

RFC In review Vineeth Vijayan 5 participants 18 messages Nov 24, 2020 – Jan 19, 2021 2 messages not in the archive

Subsystems (3)
lore
Share & tools
lore

Vineeth Vijayan

author maintainer
#0 lore raw

cover Remove uevent suppression logic from css driver

This RFC is a follow-up for the below mentioned thread.
https://marc.info/?l=linux-s390&m=158591045732735&w=2

As mentioned in the RFD, we tried to remove the uevent suppression logic from the css driver. I think this would help the vfio-cww udev-rules to bind to the subchannel before the io-subchanneli driver.

Here, the uevent is generated after,stsch() and css_sch_is_valid() which makes sure that the subchannel is valid.

Vineeth Vijayan (1):
s390/cio: Remove uevent-suppress from css driver

 drivers/s390/cio/chsc_sch.c     |  5 -----
 drivers/s390/cio/css.c          | 19 -------------------
 drivers/s390/cio/device.c       | 18 ------------------
 drivers/s390/cio/eadm_sch.c     |  5 -----
 drivers/s390/cio/vfio_ccw_drv.c |  5 -----
 5 files changed, 52 deletions(-)
Signature · 1 line
2.17.1
author maintainer
#1 lore raw

RFC 1/1 s390/cio: Remove uevent-suppress from css driver

'commit fa1a8c23eb7d ("s390: cio: Delay uevents for subchannels")' introduced the uevent suppression of subchannels. Even though there are reasons for wanting to delay the uevent, it also introduces problems. i.e On some platforms (qemu), where the udev-rule for the subchannel needs to do driver_override to bind the vfio-ccw driver instead of io_subchannel driver, but the suppressed uevent is generated only when the device is found on the subchannel. By the time it generates the uevent, it makes it difficult for the vfio-ccw udev-rules to work.
This patch removes the uevent-suppress logic from the css driver. The ADD uevent will be generated when there is a valid subchannel and not after binding the valid device. The uevent generates while device_add() during css_sch_device_register() function.

---
 drivers/s390/cio/chsc_sch.c     |  5 -----
 drivers/s390/cio/css.c          | 19 -------------------
 drivers/s390/cio/device.c       | 18 ------------------
 drivers/s390/cio/eadm_sch.c     |  5 -----
 drivers/s390/cio/vfio_ccw_drv.c |  5 -----
 5 files changed, 52 deletions(-)
drivers/s390/cio/chsc_sch.c +0 -5
index 8f080d3fd380..b4057d781fb1 100644
@@ -91,11 +91,6 @@ static int chsc_subchannel_probe(struct subchannel *sch)
 			 sch->schid.ssid, sch->schid.sch_no, ret);
 		dev_set_drvdata(&sch->dev, NULL);
 		kfree(private);
-	} else {
-		if (dev_get_uevent_suppress(&sch->dev)) {
-			dev_set_uevent_suppress(&sch->dev, 0);
-			kobject_uevent(&sch->dev.kobj, KOBJ_ADD);
-		}
 	}
 	return ret;
 }
drivers/s390/cio/css.c +0 -19
index cca1a7c4bb33..6ebc0f7152e9 100644
@@ -449,16 +449,6 @@ int css_register_subchannel(struct subchannel *sch)
 	if (sch->st == SUBCHANNEL_TYPE_IO)
 		sch->dev.type = &io_subchannel_type;
 
-	/*
-	 * We don't want to generate uevents for I/O subchannels that don't
-	 * have a working ccw device behind them since they will be
-	 * unregistered before they can be used anyway, so we delay the add
-	 * uevent until after device recognition was successful.
-	 * Note that we suppress the uevent for all subchannel types;
-	 * the subchannel driver can decide itself when it wants to inform
-	 * userspace of its existence.
-	 */
-	dev_set_uevent_suppress(&sch->dev, 1);
 	css_update_ssd_info(sch);
 	/* make it known to the system */
 	ret = css_sch_device_register(sch);
@@ -467,15 +457,6 @@ int css_register_subchannel(struct subchannel *sch)
 			      sch->schid.ssid, sch->schid.sch_no, ret);
 		return ret;
 	}
-	if (!sch->driver) {
-		/*
-		 * No driver matched. Generate the uevent now so that
-		 * a fitting driver module may be loaded based on the
-		 * modalias.
-		 */
-		dev_set_uevent_suppress(&sch->dev, 0);
-		kobject_uevent(&sch->dev.kobj, KOBJ_ADD);
-	}
 	return ret;
 }
 
drivers/s390/cio/device.c +0 -18
index aab13c78db9f..363c9bbc4a24 100644
@@ -845,14 +845,6 @@ static void io_subchannel_register(struct ccw_device *cdev)
 		adjust_init_count = 0;
 		goto out;
 	}
-	/*
-	 * Now we know this subchannel will stay, we can throw
-	 * our delayed uevent.
-	 */
-	if (dev_get_uevent_suppress(&sch->dev)) {
-		dev_set_uevent_suppress(&sch->dev, 0);
-		kobject_uevent(&sch->dev.kobj, KOBJ_ADD);
-	}
 	/* make it known to the system */
 	ret = ccw_device_add(cdev);
 	if (ret) {
@@ -1055,16 +1047,6 @@ static int io_subchannel_probe(struct subchannel *sch)
 				      "attributes for subchannel "
 				      "0.%x.%04x (rc=%d)\n",
 				      sch->schid.ssid, sch->schid.sch_no, rc);
-		/*
-		 * The console subchannel already has an associated ccw_device.
-		 * Throw the delayed uevent for the subchannel, register
-		 * the ccw_device and exit.
-		 */
-		if (dev_get_uevent_suppress(&sch->dev)) {
-			/* should always be the case for the console */
-			dev_set_uevent_suppress(&sch->dev, 0);
-			kobject_uevent(&sch->dev.kobj, KOBJ_ADD);
-		}
 		cdev = sch_get_cdev(sch);
 		rc = ccw_device_add(cdev);
 		if (rc) {
drivers/s390/cio/eadm_sch.c +0 -5
index 53468ae64b99..f8b43583b487 100644
@@ -243,11 +243,6 @@ static int eadm_subchannel_probe(struct subchannel *sch)
 	spin_lock_irq(&list_lock);
 	list_add(&private->head, &eadm_list);
 	spin_unlock_irq(&list_lock);
-
-	if (dev_get_uevent_suppress(&sch->dev)) {
-		dev_set_uevent_suppress(&sch->dev, 0);
-		kobject_uevent(&sch->dev.kobj, KOBJ_ADD);
-	}
 out:
 	return ret;
 }
drivers/s390/cio/vfio_ccw_drv.c +0 -5
index 8c625b530035..7ccc5035c8fd 100644
@@ -206,11 +206,6 @@ static int vfio_ccw_sch_probe(struct subchannel *sch)
 	if (ret)
 		goto out_disable;
 
-	if (dev_get_uevent_suppress(&sch->dev)) {
-		dev_set_uevent_suppress(&sch->dev, 0);
-		kobject_uevent(&sch->dev.kobj, KOBJ_ADD);
-	}
-
 	VFIO_CCW_MSG_EVENT(4, "bound to subchannel %x.%x.%04x\n",
 			   sch->schid.cssid, sch->schid.ssid,
 			   sch->schid.sch_no);
Signature · 1 line
2.17.1
#2 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 14 lines · On Tue, 24 Nov 2020 10:34:07 +0100 Vineeth Vijayan <[email protected]>…

'commit fa1a8c23eb7d ("s390: cio: Delay uevents for subchannels")' introduced the uevent suppression of subchannels. Even though there are reasons for wanting to delay the uevent, it also introduces problems. i.e On some platforms (qemu), where the udev-rule for the subchannel needs to do driver_override to bind the vfio-ccw driver instead of io_subchannel driver, but the suppressed uevent is generated only when the device is found on the subchannel. By the time it generates the uevent, it makes it difficult for the vfio-ccw udev-rules to work.
This patch removes the uevent-suppress logic from the css driver. The ADD uevent will be generated when there is a valid subchannel and not after binding the valid device. The uevent generates while device_add() during css_sch_device_register() function.

Quoted 8 lines
---
 drivers/s390/cio/chsc_sch.c     |  5 -----
 drivers/s390/cio/css.c          | 19 -------------------
 drivers/s390/cio/device.c       | 18 ------------------
 drivers/s390/cio/eadm_sch.c     |  5 -----
 drivers/s390/cio/vfio_ccw_drv.c |  5 -----
 5 files changed, 52 deletions(-)

While I really like that diffstat, I hope that this is actually safe for userspace programs processing uevents. Previously, we generated the ADD uevent only when all parts were setup and ready to use (including a child ccw_device, for example). Now, the ADD uevent is created earlier, before drivers have done their setup. Do existing udev rules still work as expected?

(...)

drivers/s390/cio/device.c static int io_subchannel_probe(struct subchannel *sch) · line 1061

Quoted 8 lines
drivers/s390/cio/device.c +0 -4
@@ -1055,16 +1047,6 @@ static int io_subchannel_probe(struct subchannel *sch)
 				      "attributes for subchannel "
 				      "0.%x.%04x (rc=%d)\n",
 				      sch->schid.ssid, sch->schid.sch_no, rc);
-		/*
-		 * The console subchannel already has an associated ccw_device.
-		 * Throw the delayed uevent for the subchannel, register
-		 * the ccw_device and exit.

I would keep the comment that we already have a ccw_device here. I.e.

/*
* The console subchannel already has an associated ccw_device.
* Register it and exit.
*/

Quoted 9 lines
drivers/s390/cio/device.c +0 -6
-		 */
-		if (dev_get_uevent_suppress(&sch->dev)) {
-			/* should always be the case for the console */
-			dev_set_uevent_suppress(&sch->dev, 0);
-			kobject_uevent(&sch->dev.kobj, KOBJ_ADD);
-		}
 		cdev = sch_get_cdev(sch);
 		rc = ccw_device_add(cdev);
 		if (rc) {

(...)

#3 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Thank you for looking in to this.

Quoted 25 lines · Cornelia Huck
Quoted 14 lines · On Tue, 24 Nov 2020 10:34:07 +0100 Vineeth Vijayan <[email protected]>…

'commit fa1a8c23eb7d ("s390: cio: Delay uevents for subchannels")' introduced the uevent suppression of subchannels. Even though there are reasons for wanting to delay the uevent, it also introduces problems. i.e On some platforms (qemu), where the udev-rule for the subchannel needs to do driver_override to bind the vfio-ccw driver instead of io_subchannel driver, but the suppressed uevent is generated only when the device is found on the subchannel. By the time it generates the uevent, it makes it difficult for the vfio-ccw udev-rules to work.
This patch removes the uevent-suppress logic from the css driver. The ADD uevent will be generated when there is a valid subchannel and not after binding the valid device. The uevent generates while device_add() during css_sch_device_register() function.

Quoted 8 lines
---
  drivers/s390/cio/chsc_sch.c     |  5 -----
  drivers/s390/cio/css.c          | 19 -------------------
  drivers/s390/cio/device.c       | 18 ------------------
  drivers/s390/cio/eadm_sch.c     |  5 -----
  drivers/s390/cio/vfio_ccw_drv.c |  5 -----
  5 files changed, 52 deletions(-)
Quoted 1 line

While I really like that diffstat, I hope that this is actually safe for userspace programs processing uevents. Previously, we generated the ADD uevent only when all parts were setup and ready to use (including a child ccw_device, for example). Now, the ADD uevent is created earlier, before drivers have done their setup. Do existing udev rules still work as expected? (...)

I am still working on this. I have done minimal tests as of now. Mostly with the available LPARs/zVMs. And it looks ok. But, i know there are many fragile setups out there and the change in the "timing" (The uevents generated earlier) could be a potential issue on that. I will keep you updated on my findings.

About this RFC, i just wanted make sure that we are on same page with regards to the RFD that you shared.

Quoted 26 lines

drivers/s390/cio/device.c static int io_subchannel_probe(struct subchannel *sch) · line 1061

Quoted 8 lines
drivers/s390/cio/device.c +0 -4
@@ -1055,16 +1047,6 @@ static int io_subchannel_probe(struct subchannel *sch)
  				      "attributes for subchannel "
  				      "0.%x.%04x (rc=%d)\n",
  				      sch->schid.ssid, sch->schid.sch_no, rc);
-		/*
-		 * The console subchannel already has an associated ccw_device.
-		 * Throw the delayed uevent for the subchannel, register
-		 * the ccw_device and exit.

I would keep the comment that we already have a ccw_device here. I.e.

/*
* The console subchannel already has an associated ccw_device.
* Register it and exit.
*/

Quoted 9 lines
drivers/s390/cio/device.c +0 -6
-		 */
-		if (dev_get_uevent_suppress(&sch->dev)) {
-			/* should always be the case for the console */
-			dev_set_uevent_suppress(&sch->dev, 0);
-			kobject_uevent(&sch->dev.kobj, KOBJ_ADD);
-		}
  		cdev = sch_get_cdev(sch);
  		rc = ccw_device_add(cdev);
  		if (rc) {

(...)

author maintainer
#4 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Hi Cornelia,

A bit more on this RFC.
I think this is a required change in the CIO layer, especially because there are lot of validations before we call the probe, which makes sure that we are not generating the uevent on an invalid (without a ccw-device connected) subchannel and we dont expect a uevent-storm with the current code-base. So, in anyway, Removing the suppression in the uevent is a cleaner way for the css driver.

But, the more i look at this patch and discuss on this, i think this is not complete. i.e as you know, the main reason for this RFC was the the below thread.
https://marc.info/?l=linux-s390&m=158591045732735&w=2
We are still not solving the problem that was mentioned in that RFD.

There are couple of things which we needs to consider here. With this patch, the uevents are generated before doing the initialization or before finding the ccw-device connected. Which means, the udev-rules have to manage with a non-initialized setup compared to the previous version (Version without this patch). As you mentioned, the current user-space programs which works with this uevent, especially in case of vfio-ccw will have a problem.
Second point is, there is no guarantee on the current code that the uevent generated will be used by udev-rules of vfio-ccw instead of io-subchannel driver. This means, we need to do some more modifications on the udev-rules, which can then decide which driver should bind with the subchannel. I think that is the only way to avoid the problems we are facing with the driver_override.
I would like to get your expert-opinion on the modifications that can be done on the vfio-ccw udev-rules to make it sync with the current patch.

Regards
Vineeth

Quoted 61 lines · Vineeth Vijayan

Thank you for looking in to this.

Quoted 25 lines · Cornelia Huck
Quoted 14 lines · On Tue, 24 Nov 2020 10:34:07 +0100 Vineeth Vijayan <[email protected]>…

'commit fa1a8c23eb7d ("s390: cio: Delay uevents for subchannels")' introduced the uevent suppression of subchannels. Even though there are reasons for wanting to delay the uevent, it also introduces problems. i.e On some platforms (qemu), where the udev-rule for the subchannel needs to do driver_override to bind the vfio-ccw driver instead of io_subchannel driver, but the suppressed uevent is generated only when the device is found on the subchannel. By the time it generates the uevent, it makes it difficult for the vfio-ccw udev-rules to work.
This patch removes the uevent-suppress logic from the css driver. The ADD uevent will be generated when there is a valid subchannel and not after binding the valid device. The uevent generates while device_add() during css_sch_device_register() function.

Quoted 8 lines

---
  drivers/s390/cio/chsc_sch.c     |  5 ----- drivers/s390/cio/css.c          | 19 ------------------- drivers/s390/cio/device.c       | 18 ------------------ drivers/s390/cio/eadm_sch.c     |  5 ----- drivers/s390/cio/vfio_ccw_drv.c |  5 -----

  5 files changed, 52 deletions(-)
Quoted 1 line

While I really like that diffstat, I hope that this is actually safe for userspace programs processing uevents. Previously, we generated the ADD uevent only when all parts were setup and ready to use (including a child ccw_device, for example). Now, the ADD uevent is created earlier, before drivers have done their setup. Do existing udev rules still work as expected? (...)

I am still working on this. I have done minimal tests as of now. Mostly with the available LPARs/zVMs. And it looks ok. But, i know there are many fragile setups out there and the change in the "timing" (The uevents generated earlier) could be a potential issue on that. I will keep you updated on my findings.

About this RFC, i just wanted make sure that we are on same page with regards to the RFD that you shared.

Quoted 28 lines
Quoted 10 lines
(quoted diff) +0 -0
@@ -1055,16 +1047,6 @@ static int io_subchannel_probe(struct 
 subchannel *sch)

                        "attributes for subchannel " "0.%x.%04x (rc=%d)\n", sch->schid.ssid, sch->schid.sch_no, rc); -        /*
-         * The console subchannel already has an associated ccw_device.
-         * Throw the delayed uevent for the subchannel, register -         * the ccw_device and exit.

I would keep the comment that we already have a ccw_device here. I.e.

/*
  * The console subchannel already has an associated ccw_device.
  * Register it and exit.
  */

Quoted 9 lines

-         */
-        if (dev_get_uevent_suppress(&sch->dev)) { -            /* should always be the case for the console */ -            dev_set_uevent_suppress(&sch->dev, 0); -            kobject_uevent(&sch->dev.kobj, KOBJ_ADD); -        }
          cdev = sch_get_cdev(sch);
          rc = ccw_device_add(cdev);
          if (rc) {

(...)

#5 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 10 lines · On Mon, 7 Dec 2020 09:09:48 +0100 Vineeth Vijayan <[email protected]> wrote:

Hi Cornelia,

A bit more on this RFC.
I think this is a required change in the CIO layer, especially because there are lot of validations before we call the probe, which makes sure that we are not generating the uevent on an invalid (without a ccw-device connected) subchannel and we dont expect a uevent-storm with the current code-base. So, in anyway, Removing the suppression in the uevent is a cleaner way for the
css driver.

Agreed. A device being found not operational during actual ccw device setup but not earlier is probably an edge case.

Quoted 7 lines

But, the more i look at this patch and discuss on this, i think this is not complete.
i.e as you know, the main reason for this RFC was the the below thread.
https://marc.info/?l=linux-s390&m=158591045732735&w=2
We are still not solving the problem that was mentioned in that RFD.

Quoted 11 lines

There are couple of things which we needs to consider here. With this patch, the uevents
are generated before doing the initialization or before finding the ccw-device
connected. Which means, the udev-rules have to manage with a non-initialized setup
compared to the previous version (Version without this patch). As you mentioned, the
current user-space programs which works with this uevent, especially in case of vfio-ccw
will have a problem.

IIUC, we'll get the "normal" ADD uevent when the subchannel device is registered (i.e. made visible). For the vfio-ccw case, we want the driverctl rule to match in this case, so that the driver override can be set before the subchannel device ends up being bound to the I/O subchannel driver. So I think that removing the suppression is giving us exactly what we want? Modulo any errors in the initialization sequence we might currently have in the css bus code, of course.

I'm not sure how many rules actually care about events for the subchannel device; the ccw device seems like the more helpful device to watch out for.

Quoted 9 lines

Second point is, there is no guarantee on the current code that the uevent generated
will be used by udev-rules of vfio-ccw instead of io-subchannel driver. This means, we
need to do some more modifications on the udev-rules, which can then decide which
driver should bind with the subchannel. I think that is the only way to avoid the
problems we are facing with the driver_override.

Looking on my LPAR, it seems the driverctl rule is the second in priority; i.e., it will be called before nearly everything else. This should suffice, I hope?

Quoted 3 lines

I would like to get your expert-opinion on the modifications that can be done on the
vfio-ccw udev-rules to make it sync with the current patch.

I think the vfio-ccw specific rules are those we need to worry about the least. Again, from a quick browse on my LPAR, the existing rules seem pretty sane AFAICS. I cannot speak for all the different distros, though :)

One thing that probably should be changed together with the removal of the uevent suppression is the de-registration of the subchannel device by the ccw bus. Likely an edge case, but might cause confusion when a subchannel is immediately gone again.

reviewer
#6 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 19 lines · On Tue, 8 Dec 2020 18:30:54 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 7 lines

But, the more i look at this patch and discuss on this, i think this is not complete.
i.e as you know, the main reason for this RFC was the the below thread.
https://marc.info/?l=linux-s390&m=158591045732735&w=2
We are still not solving the problem that was mentioned in that RFD.

Quoted 11 lines

There are couple of things which we needs to consider here. With this patch, the uevents
are generated before doing the initialization or before finding the ccw-device
connected. Which means, the udev-rules have to manage with a non-initialized setup
compared to the previous version (Version without this patch). As you mentioned, the
current user-space programs which works with this uevent, especially in case of vfio-ccw
will have a problem.

Quoted 8 lines

IIUC, we'll get the "normal" ADD uevent when the subchannel device is registered (i.e. made visible). For the vfio-ccw case, we want the driverctl rule to match in this case, so that the driver override can be set before the subchannel device ends up being bound to the I/O subchannel driver. So I think that removing the suppression is giving us exactly what we want? Modulo any errors in the initialization sequence we might currently have in the css bus code, of course.

I believe, I'm the originator of these concerns, yet I find my concern hard to recognize in the comment of Vineeth, so let me please try to explain this in a different way.

AFAIK the uevent handling is asynchronous with regards to matching and probing, in a sense that there is no synchronization mechanism that ensures, the userspace has had the ADD event handled (e.g. driver_override set_ before the kernel proceeds with matching and probing of the device. Am I wrong about this?

If I'm, with the suppression gone we end up with race, where userspace may or may not set driver_override in time.

The man page of driverctl
(https://manpages.debian.org/testing/driverctl/driverctl.8.en.html)
claims that: "driverctl integrates with udev to support overriding driver selection for both cold- and hotplugged devices from the moment of discovery, ..." and "The driver overrides created by driverctl are persistent across system reboots by default."

Writing to the driver_override sysfs attribute does not auto-rebind. So if we can't ensure being in time to set driver_override for the subchannel before the io_subchannel driver binds, then the userspace should handle this situation (by unbind and bind) to ensure the effectiveness of 'driver override'. I couldn't find that code in driverctl, and I assume if we had that, driver override would work without this patch. Conny, does that sound about right?

My argument is purely speculative. I didn't try this out, but trying stuff out is of limited value with races anyway. Vineeth did you try? If not, I could check this out myself some time later.

Quoted 3 lines

I'm not sure how many rules actually care about events for the subchannel device; the ccw device seems like the more helpful device to watch out for.

I tend to agree, but the problem with vfio-ccw is that (currently) we don't have a ccw device in the host, because we pass-through the subchannel. When we interrogate the subchannel, we do learn if there is a device and if, what is its devno. If I were to run a system with vfio-ccw passthrough, I would want to passthrough the subchannel that talks to the DASD (identified by devno) I need passed through to my guest.

Regards,
Halil

#7 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 59 lines · On Wed, 9 Dec 2020 13:52:03 +0100 Halil Pasic <[email protected]> wrote:
Quoted 19 lines · On Tue, 8 Dec 2020 18:30:54 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 7 lines

But, the more i look at this patch and discuss on this, i think this is not complete.
i.e as you know, the main reason for this RFC was the the below thread.
https://marc.info/?l=linux-s390&m=158591045732735&w=2
We are still not solving the problem that was mentioned in that RFD.

Quoted 11 lines

There are couple of things which we needs to consider here. With this patch, the uevents
are generated before doing the initialization or before finding the ccw-device
connected. Which means, the udev-rules have to manage with a non-initialized setup
compared to the previous version (Version without this patch). As you mentioned, the
current user-space programs which works with this uevent, especially in case of vfio-ccw
will have a problem.

Quoted 8 lines

IIUC, we'll get the "normal" ADD uevent when the subchannel device is registered (i.e. made visible). For the vfio-ccw case, we want the driverctl rule to match in this case, so that the driver override can be set before the subchannel device ends up being bound to the I/O subchannel driver. So I think that removing the suppression is giving us exactly what we want? Modulo any errors in the initialization sequence we might currently have in the css bus code, of course.

I believe, I'm the originator of these concerns, yet I find my concern hard to recognize in the comment of Vineeth, so let me please try to explain this in a different way.

AFAIK the uevent handling is asynchronous with regards to matching and probing, in a sense that there is no synchronization mechanism that ensures, the userspace has had the ADD event handled (e.g. driver_override set_ before the kernel proceeds with matching and probing of the device. Am I wrong about this?

If I'm, with the suppression gone we end up with race, where userspace may or may not set driver_override in time.

The man page of driverctl
(https://manpages.debian.org/testing/driverctl/driverctl.8.en.html)
claims that: "driverctl integrates with udev to support overriding driver selection for both cold- and hotplugged devices from the moment of discovery, ..." and "The driver overrides created by driverctl are persistent across system reboots by default."

Writing to the driver_override sysfs attribute does not auto-rebind. So if we can't ensure being in time to set driver_override for the subchannel before the io_subchannel driver binds, then the userspace should handle this situation (by unbind and bind) to ensure the effectiveness of 'driver override'. I couldn't find that code in driverctl, and I assume if we had that, driver override would work without this patch. Conny, does that sound about right?

Quoted 3 lines

My argument is purely speculative. I didn't try this out, but trying stuff out is of limited value with races anyway. Vineeth did you try? If not, I could check this out myself some time later.

Whenever we delegate policy decisions like that to userspace, we'll end up with uncertainty depending on timing. I don't think that there's any way around it. (FWIW, driver_override for pci behaves just the same, unless I misread the code.)

What removing the uevent suppression does give us is at least a chance to influence the driver that is being bound and not wait until we have a fully setup ccw device as a child as well.

Quoted 5 lines
Quoted 4 lines

I'm not sure how many rules actually care about events for the subchannel device; the ccw device seems like the more helpful device to watch out for.

Quoted 7 lines

I tend to agree, but the problem with vfio-ccw is that (currently) we don't have a ccw device in the host, because we pass-through the subchannel. When we interrogate the subchannel, we do learn if there is a device and if, what is its devno. If I were to run a system with vfio-ccw passthrough, I would want to passthrough the subchannel that talks to the DASD (identified by devno) I need passed through to my guest.

I think that can be solved by simply adding the devno as a variable to the uevent (valid if it's an I/O subchannel; we don't register the subchannel in the first place if dnv is not set.)

reviewer
#8 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 66 lines · On Tue, 15 Dec 2020 19:13:07 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 59 lines · On Wed, 9 Dec 2020 13:52:03 +0100 Halil Pasic <[email protected]> wrote:
Quoted 19 lines · On Tue, 8 Dec 2020 18:30:54 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 7 lines

But, the more i look at this patch and discuss on this, i think this is not complete.
i.e as you know, the main reason for this RFC was the the below thread.
https://marc.info/?l=linux-s390&m=158591045732735&w=2
We are still not solving the problem that was mentioned in that RFD.

Quoted 11 lines

There are couple of things which we needs to consider here. With this patch, the uevents
are generated before doing the initialization or before finding the ccw-device
connected. Which means, the udev-rules have to manage with a non-initialized setup
compared to the previous version (Version without this patch). As you mentioned, the
current user-space programs which works with this uevent, especially in case of vfio-ccw
will have a problem.

Quoted 8 lines

IIUC, we'll get the "normal" ADD uevent when the subchannel device is registered (i.e. made visible). For the vfio-ccw case, we want the driverctl rule to match in this case, so that the driver override can be set before the subchannel device ends up being bound to the I/O subchannel driver. So I think that removing the suppression is giving us exactly what we want? Modulo any errors in the initialization sequence we might currently have in the css bus code, of course.

I believe, I'm the originator of these concerns, yet I find my concern hard to recognize in the comment of Vineeth, so let me please try to explain this in a different way.

AFAIK the uevent handling is asynchronous with regards to matching and probing, in a sense that there is no synchronization mechanism that ensures, the userspace has had the ADD event handled (e.g. driver_override set_ before the kernel proceeds with matching and probing of the device. Am I wrong about this?

If I'm, with the suppression gone we end up with race, where userspace may or may not set driver_override in time.

The man page of driverctl
(https://manpages.debian.org/testing/driverctl/driverctl.8.en.html)
claims that: "driverctl integrates with udev to support overriding driver selection for both cold- and hotplugged devices from the moment of discovery, ..." and "The driver overrides created by driverctl are persistent across system reboots by default."

Writing to the driver_override sysfs attribute does not auto-rebind. So if we can't ensure being in time to set driver_override for the subchannel before the io_subchannel driver binds, then the userspace should handle this situation (by unbind and bind) to ensure the effectiveness of 'driver override'. I couldn't find that code in driverctl, and I assume if we had that, driver override would work without this patch. Conny, does that sound about right?

Quoted 3 lines

My argument is purely speculative. I didn't try this out, but trying stuff out is of limited value with races anyway. Vineeth did you try? If not, I could check this out myself some time later.

Quoted 8 lines

Whenever we delegate policy decisions like that to userspace, we'll end up with uncertainty depending on timing. I don't think that there's any way around it. (FWIW, driver_override for pci behaves just the same, unless I misread the code.)

What removing the uevent suppression does give us is at least a chance to influence the driver that is being bound and not wait until we have a fully setup ccw device as a child as well.

I finally came around to test this. In my experience driverctl works for subchannels and vfio_ccw without this patch, and continues to work with it. I found the code in driverctl that does the unbind and the implicit bind (via drivers_probe after after driver_override was set).

So now I have to ask, how exactly was the original problem diagnosed?

In https://marc.info/?l=linux-s390&m=158591045732735&w=2 there is a paragraph like:

"""
So while there's definitely a good reason for wanting to delay uevents, it is also introducing problems. One is udev rules for subchannels that are supposed to do something before a driver binds (e.g. setting driver_override to bind an I/O subchannel to vfio_ccw instead of io_subchannel) are not effective, as the ADD uevent will only be generated when the io_subchannel driver is already done with doing all setup. Another one is that only the ADD uevent is generated after uevent suppression is lifted; any other uevents that might have been generated are lost.
"""

This is not how driverclt works! I.e. it deals with the situation that the I/O subchannel was already bound to the io_subchannel driver at the time the udev rule installed by driverctl activates (via the mechanism I described above).

Regards,
Halil

#9 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 22 lines · On Sat, 19 Dec 2020 07:33:16 +0100 Halil Pasic <[email protected]> wrote:

I finally came around to test this. In my experience driverctl works for subchannels and vfio_ccw without this patch, and continues to work with it. I found the code in driverctl that does the unbind and the implicit bind (via drivers_probe after after driver_override was set).

So now I have to ask, how exactly was the original problem diagnosed?

In https://marc.info/?l=linux-s390&m=158591045732735&w=2 there is a paragraph like:

"""
So while there's definitely a good reason for wanting to delay uevents, it is also introducing problems. One is udev rules for subchannels that are supposed to do something before a driver binds (e.g. setting driver_override to bind an I/O subchannel to vfio_ccw instead of io_subchannel) are not effective, as the ADD uevent will only be generated when the io_subchannel driver is already done with doing all setup. Another one is that only the ADD uevent is generated after uevent suppression is lifted; any other uevents that might have been generated are lost.
"""

Quoted 4 lines

This is not how driverclt works! I.e. it deals with the situation that the I/O subchannel was already bound to the io_subchannel driver at the time the udev rule installed by driverctl activates (via the mechanism I described above).

That's... weird. It definitely did not work on the LPAR I initially tried it out on!

However, I think removing the suppression still looks like a good idea:
we still have the "any uevent other than ADD will have been lost" problem.

reviewer
#10 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 25 lines · On Mon, 21 Dec 2020 16:46:34 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 22 lines · On Sat, 19 Dec 2020 07:33:16 +0100 Halil Pasic <[email protected]> wrote:

I finally came around to test this. In my experience driverctl works for subchannels and vfio_ccw without this patch, and continues to work with it. I found the code in driverctl that does the unbind and the implicit bind (via drivers_probe after after driver_override was set).

So now I have to ask, how exactly was the original problem diagnosed?

In https://marc.info/?l=linux-s390&m=158591045732735&w=2 there is a paragraph like:

"""
So while there's definitely a good reason for wanting to delay uevents, it is also introducing problems. One is udev rules for subchannels that are supposed to do something before a driver binds (e.g. setting driver_override to bind an I/O subchannel to vfio_ccw instead of io_subchannel) are not effective, as the ADD uevent will only be generated when the io_subchannel driver is already done with doing all setup. Another one is that only the ADD uevent is generated after uevent suppression is lifted; any other uevents that might have been generated are lost.
"""

Quoted 8 lines
Quoted 4 lines

This is not how driverclt works! I.e. it deals with the situation that the I/O subchannel was already bound to the io_subchannel driver at the time the udev rule installed by driverctl activates (via the mechanism I described above).

That's... weird. It definitely did not work on the LPAR I initially tried it out on!

I think Boris told me some weeks ago that it didn't work for him either. I will check with him after the winter sleep.

Quoted 4 lines

However, I think removing the suppression still looks like a good idea:
we still have the "any uevent other than ADD will have been lost" problem.

I agree. I didn't look into the details, in general I think removing quirks specific to 390 (when possible) is a good thing.

Regards,
Halil

#11 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 33 lines · Halil Pasic
Quoted 30 lines · On Mon, 21 Dec 2020 16:46:34 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 26 lines · On Sat, 19 Dec 2020 07:33:16 +0100 Halil Pasic <[email protected]> wrote:

I finally came around to test this. In my experience driverctl works for subchannels and vfio_ccw without this patch, and continues to work with it. I found the code in driverctl that does the unbind and the implicit bind (via drivers_probe after after driver_override was set).

So now I have to ask, how exactly was the original problem diagnosed?

In https://marc.info/?l=linux-s390&m=158591045732735&w=2 there is a paragraph like:

"""
So while there's definitely a good reason for wanting to delay uevents, it is also introducing problems. One is udev rules for subchannels that are supposed to do something before a driver binds (e.g. setting driver_override to bind an I/O subchannel to vfio_ccw instead of io_subchannel) are not effective, as the ADD uevent will only be generated when the io_subchannel driver is already done with doing all setup. Another one is that only the ADD uevent is generated after uevent suppression is lifted; any other uevents that might have been generated are lost.
"""

This is not how driverclt works! I.e. it deals with the situation that the I/O subchannel was already bound to the io_subchannel driver at the time the udev rule installed by driverctl activates (via the mechanism I described above).

Quoted 6 lines
Quoted 3 lines

That's... weird. It definitely did not work on the LPAR I initially tried it out on!

I think Boris told me some weeks ago that it didn't work for him either. I will check with him after the winter sleep.

Yesterday I used driverctl successfully for a subchannel on F33.

Not sure what went wrong a couple of months ago but I cannot reproduce driverctl not working now.

Quoted 5 lines
Quoted 5 lines

However, I think removing the suppression still looks like a good idea:
we still have the "any uevent other than ADD will have been lost" problem.

I totally agree with this.

Quoted 7 lines

I agree. I didn't look into the details, in general I think removing quirks specific to 390 (when possible) is a good thing.

Regards,
Halil

Signature · 8 lines
Mit freundlichen Grüßen/Kind regards
   Boris Fiuczynski

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
reviewer
#12 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 38 lines · On Thu, 14 Jan 2021 14:03:25 +0100 Boris Fiuczynski <[email protected]>…
Quoted 37 lines · Halil Pasic
Quoted 33 lines · On Mon, 21 Dec 2020 16:46:34 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 22 lines · On Sat, 19 Dec 2020 07:33:16 +0100 Halil Pasic <[email protected]> wrote:

I finally came around to test this. In my experience driverctl works for subchannels and vfio_ccw without this patch, and continues to work with it. I found the code in driverctl that does the unbind and the implicit bind (via drivers_probe after after driver_override was set).

So now I have to ask, how exactly was the original problem diagnosed?

In https://marc.info/?l=linux-s390&m=158591045732735&w=2 there is a paragraph like:

"""
So while there's definitely a good reason for wanting to delay uevents, it is also introducing problems. One is udev rules for subchannels that are supposed to do something before a driver binds (e.g. setting driver_override to bind an I/O subchannel to vfio_ccw instead of io_subchannel) are not effective, as the ADD uevent will only be generated when the io_subchannel driver is already done with doing all setup. Another one is that only the ADD uevent is generated after uevent suppression is lifted; any other uevents that might have been generated are lost.
"""

Quoted 4 lines

This is not how driverclt works! I.e. it deals with the situation that the I/O subchannel was already bound to the io_subchannel driver at the time the udev rule installed by driverctl activates (via the mechanism I described above).

That's... weird. It definitely did not work on the LPAR I initially tried it out on!

Quoted 7 lines
Quoted 2 lines

I think Boris told me some weeks ago that it didn't work for him either. I will check with him after the winter sleep.

Yesterday I used driverctl successfully for a subchannel on F33.

Not sure what went wrong a couple of months ago but I cannot reproduce driverctl not working now.

Thanks Boris!

@Conny: IMHO driver_override has to work without this patch. Can you figure out, why did you claim it does not (and provide instructions on how to reproduce the problem)?

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines

However, I think removing the suppression still looks like a good idea:
we still have the "any uevent other than ADD will have been lost" problem.

I totally agree with this.

@Vineeth: I think the best way to move forward is to respin this patch with a commit message, that doesn't argue about driver_override.

Regards,
Halil

[..]

#13 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 46 lines · On Tue, 19 Jan 2021 12:47:24 +0100 Halil Pasic <[email protected]> wrote:
Quoted 43 lines · On Thu, 14 Jan 2021 14:03:25 +0100 Boris Fiuczynski <[email protected]>…
Quoted 33 lines · Halil Pasic
Quoted 30 lines · On Mon, 21 Dec 2020 16:46:34 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 26 lines · On Sat, 19 Dec 2020 07:33:16 +0100 Halil Pasic <[email protected]> wrote:

I finally came around to test this. In my experience driverctl works for subchannels and vfio_ccw without this patch, and continues to work with it. I found the code in driverctl that does the unbind and the implicit bind (via drivers_probe after after driver_override was set).

So now I have to ask, how exactly was the original problem diagnosed?

In https://marc.info/?l=linux-s390&m=158591045732735&w=2 there is a paragraph like:

"""
So while there's definitely a good reason for wanting to delay uevents, it is also introducing problems. One is udev rules for subchannels that are supposed to do something before a driver binds (e.g. setting driver_override to bind an I/O subchannel to vfio_ccw instead of io_subchannel) are not effective, as the ADD uevent will only be generated when the io_subchannel driver is already done with doing all setup. Another one is that only the ADD uevent is generated after uevent suppression is lifted; any other uevents that might have been generated are lost.
"""

This is not how driverclt works! I.e. it deals with the situation that the I/O subchannel was already bound to the io_subchannel driver at the time the udev rule installed by driverctl activates (via the mechanism I described above).

Quoted 6 lines
Quoted 3 lines

That's... weird. It definitely did not work on the LPAR I initially tried it out on!

I think Boris told me some weeks ago that it didn't work for him either. I will check with him after the winter sleep.

Yesterday I used driverctl successfully for a subchannel on F33.

Quoted 8 lines
Quoted 2 lines

Not sure what went wrong a couple of months ago but I cannot reproduce driverctl not working now.

Thanks Boris!

@Conny: IMHO driver_override has to work without this patch. Can you figure out, why did you claim it does not (and provide instructions on how to reproduce the problem)?

This may have been due to other reasons and only looking like a uevent issue at the first glance; however, I do not have that particular setup anymore, so I guess we'll never know.

Quoted 11 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

However, I think removing the suppression still looks like a good idea:
we still have the "any uevent other than ADD will have been lost" problem.

I totally agree with this.

@Vineeth: I think the best way to move forward is to respin this patch with a commit message, that doesn't argue about driver_override.

That sounds good to me.

author maintainer
#14 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 59 lines · Cornelia Huck
Quoted 43 lines · On Tue, 19 Jan 2021 12:47:24 +0100 Halil Pasic <[email protected]> wrote:
Quoted 40 lines · On Thu, 14 Jan 2021 14:03:25 +0100 Boris Fiuczynski <[email protected]>…
Quoted 32 lines · Halil Pasic
Quoted 29 lines · On Mon, 21 Dec 2020 16:46:34 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 26 lines · On Sat, 19 Dec 2020 07:33:16 +0100 Halil Pasic <[email protected]> wrote:

I finally came around to test this. In my experience driverctl works for subchannels and vfio_ccw without this patch, and continues to work with it. I found the code in driverctl that does the unbind and the implicit bind (via drivers_probe after after driver_override was set).

So now I have to ask, how exactly was the original problem diagnosed?

In https://marc.info/?l=linux-s390&m=158591045732735&w=2 there is a paragraph like:

"""
So while there's definitely a good reason for wanting to delay uevents, it is also introducing problems. One is udev rules for subchannels that are supposed to do something before a driver binds (e.g. setting driver_override to bind an I/O subchannel to vfio_ccw instead of io_subchannel) are not effective, as the ADD uevent will only be generated when the io_subchannel driver is already done with doing all setup. Another one is that only the ADD uevent is generated after uevent suppression is lifted; any other uevents that might have been generated are lost.
"""

This is not how driverclt works! I.e. it deals with the situation that the I/O subchannel was already bound to the io_subchannel driver at the time the udev rule installed by driverctl activates (via the mechanism I described above).

Quoted 5 lines
Quoted 3 lines

That's... weird. It definitely did not work on the LPAR I initially tried it out on!

I think Boris told me some weeks ago that it didn't work for him either. I will check with him after the winter sleep.

Yesterday I used driverctl successfully for a subchannel on F33.

Quoted 7 lines
Quoted 2 lines

Not sure what went wrong a couple of months ago but I cannot reproduce driverctl not working now.

Thanks Boris!

@Conny: IMHO driver_override has to work without this patch. Can you figure out, why did you claim it does not (and provide instructions on how to reproduce the problem)?

This may have been due to other reasons and only looking like a uevent issue at the first glance; however, I do not have that particular setup anymore, so I guess we'll never know.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

However, I think removing the suppression still looks like a good idea:
we still have the "any uevent other than ADD will have been lost" problem.

I totally agree with this.

@Vineeth: I think the best way to move forward is to respin this patch with a commit message, that doesn't argue about driver_override.

That sounds good to me.

Thank you @Conny, @Halil and @Boris on the insights.
I see that the driver_override works with/without this patch anyway. But, as mentioned previously, this is more like a cleanup patch. I shall respin the patch as a cleanup patch for CIO layer.

Regards
Vineeth

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

reviewer
#17 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 27 lines · On Wed, 16 Dec 2020 13:07:10 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 16 lines · On Wed, 16 Dec 2020 12:53:41 +0100 Boris Fiuczynski <[email protected]>…
Quoted 11 lines · Cornelia Huck
Quoted 11 lines
Quoted 4 lines

I'm not sure how many rules actually care about events for the subchannel device; the ccw device seems like the more helpful device to watch out for.

I tend to agree, but the problem with vfio-ccw is that (currently) we don't have a ccw device in the host, because we pass-through the subchannel. When we interrogate the subchannel, we do learn if there is a device and if, what is its devno. If I were to run a system with vfio-ccw passthrough, I would want to passthrough the subchannel that talks to the DASD (identified by devno) I need passed through to my guest.

Quoted 4 lines

I think that can be solved by simply adding the devno as a variable to the uevent (valid if it's an I/O subchannel; we don't register the subchannel in the first place if dnv is not set.)

Quoted 7 lines

Providing the devno in the context of the udev event certainly helps if the event consumer would base its actions on it.
As far as I understand the driver_override mechanics driverctl sets the override based on a specified device. In that case the devno would not be looked at and the subchannel would end up with a vfio-ccw driver even so the ccw device might not be the one we want to use as pass-through device.

Quoted 3 lines

Hm, maybe we need to make a change in driverctl that allows per-bus custom rules?

The issue with that is, that this problem ain't bus specific. I.e. it could make perfect sense to driver_override a certain ccw tape device to an alternative tape driver.

The problem is, that the only way driverctl can identify a device is a (name_of_the_bus), device_name_on_the bus) pair. Currently the udev rule installed by driverctl simply ooks fora file /etc/driverctl.d/$env{SUBSYSTEM}-$kernel
which basically encodes the current selection criteria.

Can yo please elaborate on your idea? How would you extend the driverctl cli and how would persistence look like for these custom rules? Would you make driverctl write an udev rule for each such device/custom rule on 'set-override' command instead of file in /etc/driverctl.d?

Regards,
Halil

#18 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 30 lines · On Sat, 19 Dec 2020 08:20:06 +0100 Halil Pasic <[email protected]> wrote:
Quoted 27 lines · On Wed, 16 Dec 2020 13:07:10 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 23 lines · On Wed, 16 Dec 2020 12:53:41 +0100 Boris Fiuczynski <[email protected]>…
Quoted 11 lines · Cornelia Huck
Quoted 11 lines
Quoted 4 lines

I'm not sure how many rules actually care about events for the subchannel device; the ccw device seems like the more helpful device to watch out for.

I tend to agree, but the problem with vfio-ccw is that (currently) we don't have a ccw device in the host, because we pass-through the subchannel. When we interrogate the subchannel, we do learn if there is a device and if, what is its devno. If I were to run a system with vfio-ccw passthrough, I would want to passthrough the subchannel that talks to the DASD (identified by devno) I need passed through to my guest.

Quoted 4 lines

I think that can be solved by simply adding the devno as a variable to the uevent (valid if it's an I/O subchannel; we don't register the subchannel in the first place if dnv is not set.)

Providing the devno in the context of the udev event certainly helps if the event consumer would base its actions on it.
As far as I understand the driver_override mechanics driverctl sets the override based on a specified device. In that case the devno would not be looked at and the subchannel would end up with a vfio-ccw driver even so the ccw device might not be the one we want to use as pass-through device.

Quoted 7 lines
Quoted 3 lines

Hm, maybe we need to make a change in driverctl that allows per-bus custom rules?

The issue with that is, that this problem ain't bus specific. I.e. it could make perfect sense to driver_override a certain ccw tape device to an alternative tape driver.

But ccw does not provide driver_override? Confused.

Quoted 11 lines

The problem is, that the only way driverctl can identify a device is a (name_of_the_bus), device_name_on_the bus) pair. Currently the udev rule installed by driverctl simply ooks fora file /etc/driverctl.d/$env{SUBSYSTEM}-$kernel
which basically encodes the current selection criteria.

Can yo please elaborate on your idea? How would you extend the driverctl cli and how would persistence look like for these custom rules? Would you make driverctl write an udev rule for each such device/custom rule on 'set-override' command instead of file in /etc/driverctl.d?

I have not really looked at how to implement this. But we could have driverctl support an optional "additional_parameters" option, which allows to specify key/value pairs that have to match. I guess that should be dropped into the driverctl config directory, and generate an additional check?

reviewer
#19 lore raw

Re: [RFC 1/1] s390/cio: Remove uevent-suppress from css driver

Quoted 37 lines · On Mon, 21 Dec 2020 16:52:19 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 34 lines · On Sat, 19 Dec 2020 08:20:06 +0100 Halil Pasic <[email protected]> wrote:
Quoted 30 lines · On Wed, 16 Dec 2020 13:07:10 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 16 lines · On Wed, 16 Dec 2020 12:53:41 +0100 Boris Fiuczynski <[email protected]>…
Quoted 11 lines · Cornelia Huck
Quoted 11 lines
Quoted 4 lines

I'm not sure how many rules actually care about events for the subchannel device; the ccw device seems like the more helpful device to watch out for.

I tend to agree, but the problem with vfio-ccw is that (currently) we don't have a ccw device in the host, because we pass-through the subchannel. When we interrogate the subchannel, we do learn if there is a device and if, what is its devno. If I were to run a system with vfio-ccw passthrough, I would want to passthrough the subchannel that talks to the DASD (identified by devno) I need passed through to my guest.

Quoted 4 lines

I think that can be solved by simply adding the devno as a variable to the uevent (valid if it's an I/O subchannel; we don't register the subchannel in the first place if dnv is not set.)

Quoted 7 lines

Providing the devno in the context of the udev event certainly helps if the event consumer would base its actions on it.
As far as I understand the driver_override mechanics driverctl sets the override based on a specified device. In that case the devno would not be looked at and the subchannel would end up with a vfio-ccw driver even so the ccw device might not be the one we want to use as pass-through device.

Hm, maybe we need to make a change in driverctl that allows per-bus custom rules?

Quoted 6 lines
Quoted 3 lines

The issue with that is, that this problem ain't bus specific. I.e. it could make perfect sense to driver_override a certain ccw tape device to an alternative tape driver.

But ccw does not provide driver_override? Confused.

Right, but it could. That is not my point. I was arguing with 'per-bus custom rules'.

Quoted 12 lines
Quoted 11 lines

The problem is, that the only way driverctl can identify a device is a (name_of_the_bus), device_name_on_the bus) pair. Currently the udev rule installed by driverctl simply ooks fora file /etc/driverctl.d/$env{SUBSYSTEM}-$kernel
which basically encodes the current selection criteria.

Can yo please elaborate on your idea? How would you extend the driverctl cli and how would persistence look like for these custom rules? Would you make driverctl write an udev rule for each such device/custom rule on 'set-override' command instead of file in /etc/driverctl.d?

Quoted 6 lines

I have not really looked at how to implement this. But we could have driverctl support an optional "additional_parameters" option, which allows to specify key/value pairs that have to match. I guess that should be dropped into the driverctl config directory, and generate an additional check?

The devil is always in the details. I'm sure something can be done :). Key/value pairs for matching fits IMHO better with new udev rules than with files in the driverctl config dir that somehow generate additiona checks. One thing I didn't understand is, are you proposing an invocation like
driverctl -b css --additional_params="DEVNO=<devno>" \
set-override <subchannel_no> vfio_ccw
or
driverctl -b css --additional_params="DEVNO=<devno>" \
set-override vfio_ccw
?

The first one would safeguard against passing through the wrong devno, but I think, we actually want to be oblivious about the subchannel number.

Regards,
Halil