Skip to contentSkip to conversation

two small mdev fixups

PATCH In review Christoph Hellwig 4 participants 15 messages Jul 26, 2021 – Aug 3, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi all,

two small mdev fixes - one to fix mdev for built-in drivers, and the other one to remove a pointless warning.

#1 lore raw

PATCH 1/2 vfio/mdev: turn mdev_init into a subsys_initcall

Without this setups with buіlt-in mdev and mdev-drivers fail to register like this:

[1.903149] Driver 'intel_vgpu_mdev' was unable to register with bus_type 'mdev' because the bus was not initialized.
---
 drivers/vfio/mdev/mdev_core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/mdev/mdev_core.c +1 -1
index e4581ec093a6..b16606ebafa1 100644
@@ -398,7 +398,7 @@ static void __exit mdev_exit(void)
 	mdev_bus_unregister();
 }
 
-module_init(mdev_init)
+subsys_initcall(mdev_init)
 module_exit(mdev_exit)
 
 MODULE_VERSION(DRIVER_VERSION);
Signature · 1 line
2.30.2
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] vfio/mdev: turn mdev_init into a subsys_initcall

Quoted 9 lines · Christoph Hellwig

Without this setups with buіlt-in mdev and mdev-drivers fail to register like this:

[1.903149] Driver 'intel_vgpu_mdev' was unable to register with bus_type 'mdev' because the bus was not initialized.
---
 drivers/vfio/mdev/mdev_core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH 1/2] vfio/mdev: turn mdev_init into a subsys_initcall

In reply to Christoph Hellwig
Quoted 9 lines · Christoph Hellwig

Without this setups with buіlt-in mdev and mdev-drivers fail to register like this:

[1.903149] Driver 'intel_vgpu_mdev' was unable to register with bus_type 'mdev' because the bus was not initialized.
---
 drivers/vfio/mdev/mdev_core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Jason

#4 lore raw

PATCH 2/2 vfio/mdev: don't warn if ->request is not set

In reply to Christoph Hellwig

Only a single driver actually sets the ->request method, so don't print a scary warning if it isn't.

---
 drivers/vfio/mdev/mdev_core.c | 4 ----
 1 file changed, 4 deletions(-)
drivers/vfio/mdev/mdev_core.c +0 -4
index b16606ebafa1..b314101237fe 100644
@@ -138,10 +138,6 @@ int mdev_register_device(struct device *dev, const struct mdev_parent_ops *ops)
 	if (!dev)
 		return -EINVAL;
 
-	/* Not mandatory, but its absence could be a problem */
-	if (!ops->request)
-		dev_info(dev, "Driver cannot be asked to release device\n");
-
 	mutex_lock(&parent_list_lock);
 
 	/* Check for duplicate */
Signature · 1 line
2.30.2
Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] vfio/mdev: don't warn if ->request is not set

Quoted 13 lines · Christoph Hellwig

Only a single driver actually sets the ->request method, so don't print a scary warning if it isn't.

---
 drivers/vfio/mdev/mdev_core.c | 4 ----
 1 file changed, 4 deletions(-)
drivers/vfio/mdev/mdev_core.c +0 -0
index b16606ebafa1..b314101237fe 100644

drivers/vfio/mdev/mdev_core.c int mdev_register_device(struct device *dev, const struct mdev_parent_ops *ops) · line 143

Quoted 10 lines
drivers/vfio/mdev/mdev_core.c +0 -4
@@ -138,10 +138,6 @@ int mdev_register_device(struct device *dev, const struct mdev_parent_ops *ops)
 	if (!dev)
 		return -EINVAL;
 
-	/* Not mandatory, but its absence could be a problem */
-	if (!ops->request)
-		dev_info(dev, "Driver cannot be asked to release device\n");
-
 	mutex_lock(&parent_list_lock);
 
 	/* Check for duplicate */

We also log a warning if we would like to call ->request() but none was provided, so I think that's fine.

But I wonder why nobody else implements this? Lack of surprise removal?

#6 lore raw

Re: [PATCH 2/2] vfio/mdev: don't warn if ->request is not set

Quoted 1 line · Cornelia Huck

But I wonder why nobody else implements this? Lack of surprise removal?

The only implementation triggers an eventfd that seems to be the same eventfd as the interrupt..

Do you know how this works in userspace? I'm surprised that the interrupt eventfd can trigger an observation that the kernel driver wants to be unplugged?

Jason

#7 lore raw

Re: [PATCH 2/2] vfio/mdev: don't warn if ->request is not set

Quoted 10 lines · On Mon, 26 Jul 2021 20:09:06 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 1 line · Cornelia Huck

But I wonder why nobody else implements this? Lack of surprise removal?

The only implementation triggers an eventfd that seems to be the same eventfd as the interrupt..

Do you know how this works in userspace? I'm surprised that the interrupt eventfd can trigger an observation that the kernel driver wants to be unplugged?

I think we're talking about ccw, but I see QEMU registering separate eventfds for each of the 3 IRQ indexes and the mdev driver specifically triggering the req_trigger...? Thanks,

Alex

#8 lore raw

Re: [PATCH 2/2] vfio/mdev: don't warn if ->request is not set

Quoted 14 lines · Alex Williamson
Quoted 10 lines · On Mon, 26 Jul 2021 20:09:06 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 1 line · Cornelia Huck

But I wonder why nobody else implements this? Lack of surprise removal?

The only implementation triggers an eventfd that seems to be the same eventfd as the interrupt..

Do you know how this works in userspace? I'm surprised that the interrupt eventfd can trigger an observation that the kernel driver wants to be unplugged?

Quoted 5 lines

I think we're talking about ccw, but I see QEMU registering separate eventfds for each of the 3 IRQ indexes and the mdev driver specifically triggering the req_trigger...? Thanks,

Alex

Exactly, ccw has a trigger for normal I/O interrupts, CRW (machine checks), and this one.

#9 lore raw

Re: [PATCH 2/2] vfio/mdev: don't warn if ->request is not set

Quoted 16 lines · Cornelia Huck
Quoted 14 lines · Alex Williamson
Quoted 10 lines · On Mon, 26 Jul 2021 20:09:06 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 1 line · Cornelia Huck

But I wonder why nobody else implements this? Lack of surprise removal?

The only implementation triggers an eventfd that seems to be the same eventfd as the interrupt..

Do you know how this works in userspace? I'm surprised that the interrupt eventfd can trigger an observation that the kernel driver wants to be unplugged?

Quoted 8 lines
Quoted 5 lines

I think we're talking about ccw, but I see QEMU registering separate eventfds for each of the 3 IRQ indexes and the mdev driver specifically triggering the req_trigger...? Thanks,

Alex

Exactly, ccw has a trigger for normal I/O interrupts, CRW (machine checks), and this one.

If it is a dedicated eventfd for 'device being removed' why is it in the CCW implementation and not core code?

Is PCI doing the same?

Jason

#10 lore raw

Re: [PATCH 2/2] vfio/mdev: don't warn if ->request is not set

Quoted 21 lines · On Tue, 27 Jul 2021 14:32:09 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 20 lines · Cornelia Huck
Quoted 18 lines · Alex Williamson
Quoted 10 lines · On Mon, 26 Jul 2021 20:09:06 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 1 line · Cornelia Huck

But I wonder why nobody else implements this? Lack of surprise removal?

The only implementation triggers an eventfd that seems to be the same eventfd as the interrupt..

Do you know how this works in userspace? I'm surprised that the interrupt eventfd can trigger an observation that the kernel driver wants to be unplugged?

I think we're talking about ccw, but I see QEMU registering separate eventfds for each of the 3 IRQ indexes and the mdev driver specifically triggering the req_trigger...? Thanks,

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Alex

Exactly, ccw has a trigger for normal I/O interrupts, CRW (machine checks), and this one.

If it is a dedicated eventfd for 'device being removed' why is it in the CCW implementation and not core code?

The CCW implementation (likewise the vfio-pci implementation) owns the IRQ index address space and the decision to make this a signal to userspace rather than perhaps some handling a device might be able to do internally. For instance an alternate vfio-pci implementation might zap all mmaps, block all r/w access, and turn this into a surprise removal. Another implementation might be more aggressive to sending SIGKILL to the user process. This was the thought behind why vfio-core triggers the driver request callback with a counter, leaving the policy to the driver.

Quoted 1 line

Is PCI doing the same?

Yes, that's where this handling originated. Thanks,

Alex

#11 lore raw

Re: [PATCH 2/2] vfio/mdev: don't warn if ->request is not set

Quoted 29 lines · Alex Williamson
Quoted 26 lines · On Tue, 27 Jul 2021 14:32:09 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 24 lines · Cornelia Huck
Quoted 14 lines · Alex Williamson
Quoted 10 lines · On Mon, 26 Jul 2021 20:09:06 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 1 line · Cornelia Huck

But I wonder why nobody else implements this? Lack of surprise removal?

The only implementation triggers an eventfd that seems to be the same eventfd as the interrupt..

Do you know how this works in userspace? I'm surprised that the interrupt eventfd can trigger an observation that the kernel driver wants to be unplugged?

Quoted 5 lines

I think we're talking about ccw, but I see QEMU registering separate eventfds for each of the 3 IRQ indexes and the mdev driver specifically triggering the req_trigger...? Thanks,

Alex

Exactly, ccw has a trigger for normal I/O interrupts, CRW (machine checks), and this one.

Quoted 7 lines
Quoted 2 lines

If it is a dedicated eventfd for 'device being removed' why is it in the CCW implementation and not core code?

The CCW implementation (likewise the vfio-pci implementation) owns the IRQ index address space and the decision to make this a signal to userspace rather than perhaps some handling a device might be able to do internally.

The core code holds the vfio_device_get() so long as the FD is open. There is no way to pass the wait_for_completion without userspace closing the FD, so there isn't really much choice for the drivers to do beyond signal to userpace to close the FD??

Quoted 2 lines

For instance an alternate vfio-pci implementation might zap all mmaps, block all r/w access, and turn this into a surprise removal.

This is nice, but wouldn't close the FD, so needs core changes anyhow..

Quoted 2 lines

Another implementation might be more aggressive to sending SIGKILL to the user process.

We don't try to revoke FDs from the kernel, it is racy, dangerous and unreliable.

Quoted 2 lines

This was the thought behind why vfio-core triggers the driver request callback with a counter, leaving the policy to the driver.

IMHO subsystem policy does not belong in drivers. Down that road lies a mess for userspace.

Jason

#12 lore raw

Re: [PATCH 2/2] vfio/mdev: don't warn if ->request is not set

Quoted 43 lines · On Tue, 27 Jul 2021 16:03:17 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 29 lines · Alex Williamson
Quoted 26 lines · On Tue, 27 Jul 2021 14:32:09 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 24 lines · Cornelia Huck
Quoted 14 lines · Alex Williamson
Quoted 10 lines · On Mon, 26 Jul 2021 20:09:06 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 1 line · Cornelia Huck

But I wonder why nobody else implements this? Lack of surprise removal?

The only implementation triggers an eventfd that seems to be the same eventfd as the interrupt..

Do you know how this works in userspace? I'm surprised that the interrupt eventfd can trigger an observation that the kernel driver wants to be unplugged?

Quoted 5 lines

I think we're talking about ccw, but I see QEMU registering separate eventfds for each of the 3 IRQ indexes and the mdev driver specifically triggering the req_trigger...? Thanks,

Alex

Exactly, ccw has a trigger for normal I/O interrupts, CRW (machine checks), and this one.

Quoted 7 lines
Quoted 2 lines

If it is a dedicated eventfd for 'device being removed' why is it in the CCW implementation and not core code?

The CCW implementation (likewise the vfio-pci implementation) owns the IRQ index address space and the decision to make this a signal to userspace rather than perhaps some handling a device might be able to do internally.

The core code holds the vfio_device_get() so long as the FD is open. There is no way to pass the wait_for_completion without userspace closing the FD, so there isn't really much choice for the drivers to do beyond signal to userpace to close the FD??

Quoted 5 lines
Quoted 2 lines

For instance an alternate vfio-pci implementation might zap all mmaps, block all r/w access, and turn this into a surprise removal.

This is nice, but wouldn't close the FD, so needs core changes anyhow..

Right, the core would need to be able to handle an FD disconnected from the device, obviously some core changes would be required.

Quoted 5 lines
Quoted 2 lines

Another implementation might be more aggressive to sending SIGKILL to the user process.

We don't try to revoke FDs from the kernel, it is racy, dangerous and unreliable.

I'm not sure how trying to kill the process using an open file becomes a revoke... In fact, the surprise hotplug might just be able to zap mmaps and wait for userspace to generate a SIGBUS.

Quoted 5 lines
Quoted 2 lines

This was the thought behind why vfio-core triggers the driver request callback with a counter, leaving the policy to the driver.

IMHO subsystem policy does not belong in drivers. Down that road lies a mess for userspace.

I think my argument was that to this point it's been driver policy, not subsystem policy. The subsystem policy is to block until the device is released, it's the driver policy whether it has a means to implement something to expedite that. Thanks,

Alex

#14 lore raw
In reply to Christoph Hellwig
Quoted 4 lines · On Mon, 26 Jul 2021 16:35:22 +0200 Christoph Hellwig <[email protected]> wrote:

Hi all,

two small mdev fixes - one to fix mdev for built-in drivers, and the other one to remove a pointless warning.

Applied to vfio next branch for v5.15 with Connie and Jason's R-b. Thanks,

Alex