pci: Add pci device even if the driver failed to attach
PATCH 1/2 pci: Add pci device even if the driver failed to attach
device_attach() returning failure indicates a driver error while trying to probe the device. In such a scenario, the PCI device should still be added in the system and be visible to the user.
This patch partially reverts:
commit ab1a187bba5c ("PCI: Check device_attach() return value always")
--- drivers/pci/bus.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
drivers/pci/bus.c +1 -5
@@ -322,12 +322,8 @@ void pci_bus_add_device(struct pci_dev *dev) dev->match_driver = true; retval =device_attach (&dev->dev); -if (retval <0 && retval != -EPROBE_DEFER ) { +if (retval <0 && retval != -EPROBE_DEFER )pci_warn (dev,"device attach failed (%d)\n" , retval); -pci_proc_detach_device (dev); -pci_remove_sysfs_dev_files (dev); -return ; - }pci_dev_assign_added (dev, true); }
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
PATCH 2/2 pci: Add parameter to disable attaching untrusted devices
Introduce a PCI parameter that disables the automatic attachment of untrusted devices to their drivers.
---
Context:
I set out to implement the approach outlined in
https://lkml.org/lkml/2020/6/9/1331 https://lkml.org/lkml/2020/6/15/1453
But to my surprise, I found that the new hotplugged PCI devices were getting automatically attached to drivers even though /sys/bus/pci/drivers_autoprobe was set to 0.
I realized that the device core's "drivers_autoprobe":
* only disables the *initial* probe of the device (i.e. from device_add()). If a subsystem calls device_attach() explicitly for its devices like PCI subsystem does, the drivers_autoprobe setting does not matter. The core will attach device to the driver. This looks like correct semantic behavior to me because PCI is explicitly calling device_attach(), which is a way to explicitly ask the core to find and attach a driver for a device.
* "drivers_autoprobe" cannot be controlled at boot time (to restrict any drivers before userspace comes up).
The options I considered were:
1) Change device_attach() so that it takes into consideration the drivers_autoprobe property. Not sure if this is semantically correct thing to do though. If I do this, then the only way a driver can be attached to the drivers would be via userspace (/sys/bus/pci/drivers/bind) (Good for our use case though!).
2) Make the drivers_autoprobe property available to PCI to use (currently it is private to device core). The PCI could use this to determine whether or not to call device_attach(). This still leaves the other problem (of not being able to set drivers_autoprobe via command line open).
3) I found the pci_dev->match_driver, which seemed similar to what I am trying to do, but can't be controlled from userspace. I considered populating that field based on drivers_autoprobe (still need (2)). But the problem is that there is the AMD IOMMU driver which is setting this independently, so setting the match_driver based on drivers_autoprobe may not be a good idea. May be we can populate it for untrusted devicesi, based on the parameter that I'm introducing?
4) This patch was my option 4 that helps fix both the problems for me.
drivers/pci/bus.c | 11 ++++++++--- drivers/pci/pci.c | 9 +++++++++ drivers/pci/pci.h | 1 + 3 files changed, 18 insertions(+), 3 deletions(-)
drivers/pci/bus.c +8 -3
@@ -321,9 +321,14 @@ void pci_bus_add_device(struct pci_dev *dev)pci_bridge_d3_update (dev); dev->match_driver = true; - retval =device_attach (&dev->dev); -if (retval <0 && retval != -EPROBE_DEFER ) -pci_warn (dev,"device attach failed (%d)\n" , retval); + +if (dev->untrusted && pci_dont_attach_untrusted_devs) { +pci_info (dev,"not attaching untrusted device\n" ); + }else { + retval =device_attach (&dev->dev); +if (retval <0 && retval != -EPROBE_DEFER ) +pci_warn (dev,"device attach failed (%d)\n" , retval); + }pci_dev_assign_added (dev, true); }
drivers/pci/pci.c +9 -0
@@ -127,6 +127,13 @@ static bool pcie_ats_disabled;/* If set, the PCI config space of each device is printed during boot. */ bool pci_early_dump; +/* +* If set, the devices with "untrusted" flag shall not be attached automatically +* Userspace will need to attach them manually: +* echo <pci device> > /sys/bus/pci/drivers/<driver>/bind +*/ +bool pci_dont_attach_untrusted_devs; +bool pci_ats_disabled (void ) {return pcie_ats_disabled;
@@ -6522,6 +6529,8 @@ static int __init pci_setup(char *str)pci_add_flags (PCI_SCAN_ALL_PCIE_DEVS ); }else if (!strncmp (str,"disable_acs_redir=" ,18 )) { disable_acs_redir_param = str +18 ; + }else if (!strcmp (str,"dont_attach_untrusted_devs" )) { + pci_dont_attach_untrusted_devs = true; }else {pr_err ("PCI: Unknown option `%s'\n" , str); }
drivers/pci/pci.h +1 -0
@@ -13,6 +13,7 @@extern const unsigned char pcie_link_speed[];extern bool pci_early_dump; +extern bool pci_dont_attach_untrusted_devs;bool pcie_cap_has_lnkctl (const struct pci_dev *dev);bool pcie_cap_has_rtctl (const struct pci_dev *dev);
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
Re: [PATCH 2/2] pci: Add parameter to disable attaching untrusted devices
Quoted 2 lines · Rajat Jain
Introduce a PCI parameter that disables the automatic attachment of untrusted devices to their drivers.
You didn't document this new api anywhere :(
Re: [PATCH 2/2] pci: Add parameter to disable attaching untrusted devices
In reply to Rajat JainQuoted 18 lines · Rajat Jain
Introduce a PCI parameter that disables the automatic attachment of untrusted devices to their drivers.
Signed-off-by Rajat Jain <[email protected]>---
Context:I set out to implement the approach outlined in
https://lkml.org/lkml/2020/6/9/1331 https://lkml.org/lkml/2020/6/15/1453But to my surprise, I found that the new hotplugged PCI devices were getting automatically attached to drivers even though
/sys/bus/pci/drivers_autoprobewas set to 0.I realized that the device core's "
drivers_autoprobe":
Quoted 7 lines
* only disables the *initial* probe of the device (i.e. from
device_add()). If a subsystem callsdevice_attach()explicitly for its devices like PCI subsystem does, thedrivers_autoprobesetting does not matter. The core will attach device to the driver. This looks like correct semantic behavior to me because PCI is explicitly callingdevice_attach(), which is a way to explicitly ask the core to find and attach a driver for a device.
Right, but we're doing using device_attach() largely because the driver core doesn't provide any mechanism for deferring the initial probe. I didn't think there was any deeper reason for it, but while looking I noticed that the initial probe can be async and device_attach() forces probing to be synchronous. That has the side effect of serialising all PCI device probing which might be intentional to avoid device renaming due to the change in probe order. Userspace is better at dealing with device names changing now days, but you might still get some people mad at you for changing it.
Quoted 6 lines
2) Make the
drivers_autoprobeproperty available to PCI to use (currently it is private to device core). The PCI could use this to determine whether or not to calldevice_attach(). This still leaves the other problem (of not being able to setdrivers_autoprobevia command line open).
Quoted 6 lines
3) I found the
pci_dev->match_driver, which seemed similar to what I am trying to do, but can't be controlled from userspace. I considered populating that field based ondrivers_autoprobe(still need (2)). But the problem is that there is the AMD IOMMU driver which is setting this independently, so setting thematch_driverbased ondrivers_autoprobemay not be a good idea.
Huh, that's pretty weird. Even with that hack you should be able trigger the bug they're working around by removing the IOMMU device in sysfs and doing a rescan. I wouldn't worry much about making match_device user controllable since you would need to work pretty hard for it to be an issue.
Oliver
Re: [PATCH 2/2] pci: Add parameter to disable attaching untrusted devices
In reply to Rajat JainQuoted 28 lines · Rajat Jain
Introduce a PCI parameter that disables the automatic attachment of untrusted devices to their drivers.
Signed-off-by Rajat Jain <[email protected]>---
Context:I set out to implement the approach outlined in
https://lkml.org/lkml/2020/6/9/1331 https://lkml.org/lkml/2020/6/15/1453But to my surprise, I found that the new hotplugged PCI devices were getting automatically attached to drivers even though
/sys/bus/pci/drivers_autoprobewas set to 0.I realized that the device core's "
drivers_autoprobe":* only disables the *initial* probe of the device (i.e. from
device_add()). If a subsystem callsdevice_attach()explicitly for its devices like PCI subsystem does, thedrivers_autoprobesetting does not matter. The core will attach device to the driver. This looks like correct semantic behavior to me because PCI is explicitly callingdevice_attach(), which is a way to explicitly ask the core to find and attach a driver for a device.* "
drivers_autoprobe" cannot be controlled at boot time (to restrict any drivers before userspace comes up).
Quoted 7 lines
The options I considered were:
1) Change
device_attach()so that it takes into consideration thedrivers_autoprobeproperty. Not sure if this is semantically correct thing to do though. If I do this, then the only way a driver can be attached to the drivers would be via userspace (/sys/bus/pci/drivers/bind) (Good for our use case though!).
This is the correct thing to do here, haven't I been asking you do move this logic into the driver core so that all busses can use it?
Quoted 5 lines
2) Make the
drivers_autoprobeproperty available to PCI to use (currently it is private to device core). The PCI could use this to determine whether or not to calldevice_attach(). This still leaves the other problem (of not being able to setdrivers_autoprobevia command line open).
Ick, command lines are horrible, don't do that if at all possible. On some systems they are not able to be changed which can be good or bad...
Quoted 9 lines
3) I found the
pci_dev->match_driver, which seemed similar to what I am trying to do, but can't be controlled from userspace. I considered populating that field based ondrivers_autoprobe(still need (2)). But the problem is that there is the AMD IOMMU driver which is setting this independently, so setting thematch_driverbased ondrivers_autoprobemay not be a good idea. May be we can populate it for untrusted devicesi, based on the parameter that I'm introducing?4) This patch was my option 4 that helps fix both the problems for me.
I suggest putting some of the above text in the changelog, as it has a lot of good context, while your existing changelog is pretty sparse and does not explain anything...
Quoted 70 lines
drivers/pci/bus.c | 11 ++++++++--- drivers/pci/pci.c | 9 +++++++++ drivers/pci/pci.h | 1 + 3 files changed, 18 insertions(+), 3 deletions(-)drivers/pci/bus.c +8
-3@@ -321,9 +321,14 @@ void pci_bus_add_device(struct pci_dev *dev)pci_bridge_d3_update (dev); dev->match_driver = true; - retval =device_attach (&dev->dev); -if (retval <0 && retval != -EPROBE_DEFER ) -pci_warn (dev,"device attach failed (%d)\n" , retval); + +if (dev->untrusted && pci_dont_attach_untrusted_devs) { +pci_info (dev,"not attaching untrusted device\n" ); + }else { + retval =device_attach (&dev->dev); +if (retval <0 && retval != -EPROBE_DEFER ) +pci_warn (dev,"device attach failed (%d)\n" , retval); + }pci_dev_assign_added (dev, true); }drivers/pci/pci.c +9
-0@@ -127,6 +127,13 @@ static bool pcie_ats_disabled;/* If set, the PCI config space of each device is printed during boot. */ bool pci_early_dump; +/* +* If set, the devices with "untrusted" flag shall not be attached automatically +* Userspace will need to attach them manually: +* echo <pci device> > /sys/bus/pci/drivers/<driver>/bind +*/ +bool pci_dont_attach_untrusted_devs; +bool pci_ats_disabled (void ) {return pcie_ats_disabled;@@ -6522,6 +6529,8 @@ static int __init pci_setup(char *str)pci_add_flags (PCI_SCAN_ALL_PCIE_DEVS ); }else if (!strncmp (str,"disable_acs_redir=" ,18 )) { disable_acs_redir_param = str +18 ; + }else if (!strcmp (str,"dont_attach_untrusted_devs" )) { + pci_dont_attach_untrusted_devs = true; }else {pr_err ("PCI: Unknown option `%s'\n" , str); }drivers/pci/pci.h +1
-0@@ -13,6 +13,7 @@extern const unsigned char pcie_link_speed[];extern bool pci_early_dump; +extern bool pci_dont_attach_untrusted_devs;bool pcie_cap_has_lnkctl (const struct pci_dev *dev);bool pcie_cap_has_rtctl (const struct pci_dev *dev);Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
What happened to the split of "trust" and "internal/external" logic that we discussed before? This seems to ignore all of that and go straight to some form of "we know what we trust, so all is fine!".
It's not obvious what this is really doing here at all, sorry...
greg k-h
Re: [PATCH 2/2] pci: Add parameter to disable attaching untrusted devices
Hello,
Thanks for taking a look.
Quoted 32 lines · Greg Kroah-Hartman
Quoted 30 lines · Rajat Jain
Introduce a PCI parameter that disables the automatic attachment of untrusted devices to their drivers.
Signed-off-by Rajat Jain <[email protected]>---
Context:I set out to implement the approach outlined in
https://lkml.org/lkml/2020/6/9/1331 https://lkml.org/lkml/2020/6/15/1453But to my surprise, I found that the new hotplugged PCI devices were getting automatically attached to drivers even though
/sys/bus/pci/drivers_autoprobewas set to 0.I realized that the device core's "
drivers_autoprobe":* only disables the *initial* probe of the device (i.e. from
device_add()). If a subsystem callsdevice_attach()explicitly for its devices like PCI subsystem does, thedrivers_autoprobesetting does not matter. The core will attach device to the driver. This looks like correct semantic behavior to me because PCI is explicitly callingdevice_attach(), which is a way to explicitly ask the core to find and attach a driver for a device.* "
drivers_autoprobe" cannot be controlled at boot time (to restrict any drivers before userspace comes up).The options I considered were:
Quoted 8 lines
Quoted 5 lines
1) Change
device_attach()so that it takes into consideration thedrivers_autoprobeproperty. Not sure if this is semantically correct thing to do though. If I do this, then the only way a driver can be attached to the drivers would be via userspace (/sys/bus/pci/drivers/bind) (Good for our use case though!).This is the correct thing to do here, haven't I been asking you do move this logic into the driver core so that all busses can use it?
(please see below)
Quoted 9 lines
Quoted 6 lines
2) Make the
drivers_autoprobeproperty available to PCI to use (currently it is private to device core). The PCI could use this to determine whether or not to calldevice_attach(). This still leaves the other problem (of not being able to setdrivers_autoprobevia command line open).Ick, command lines are horrible, don't do that if at all possible. On some systems they are not able to be changed which can be good or bad...
(please see below)
Quoted 9 lines
Quoted 9 lines
3) I found the
pci_dev->match_driver, which seemed similar to what I am trying to do, but can't be controlled from userspace. I considered populating that field based ondrivers_autoprobe(still need (2)). But the problem is that there is the AMD IOMMU driver which is setting this independently, so setting thematch_driverbased ondrivers_autoprobemay not be a good idea. May be we can populate it for untrusted devicesi, based on the parameter that I'm introducing?
Quoted 5 lines
Quoted 1 line
4) This patch was my option 4 that helps fix both the problems for me.
I suggest putting some of the above text in the changelog, as it has a lot of good context, while your existing changelog is pretty sparse and does not explain anything...
Will do.
Quoted 67 lines
Quoted 67 lines
drivers/pci/bus.c | 11 ++++++++--- drivers/pci/pci.c | 9 +++++++++ drivers/pci/pci.h | 1 + 3 files changed, 18 insertions(+), 3 deletions(-)drivers/pci/bus.c +8
-3@@ -321,9 +321,14 @@ void pci_bus_add_device(struct pci_dev *dev)pci_bridge_d3_update (dev); dev->match_driver = true; - retval =device_attach (&dev->dev); -if (retval <0 && retval != -EPROBE_DEFER ) -pci_warn (dev,"device attach failed (%d)\n" , retval); + +if (dev->untrusted && pci_dont_attach_untrusted_devs) { +pci_info (dev,"not attaching untrusted device\n" ); + }else { + retval =device_attach (&dev->dev); +if (retval <0 && retval != -EPROBE_DEFER ) +pci_warn (dev,"device attach failed (%d)\n" , retval); + }pci_dev_assign_added (dev, true); }drivers/pci/pci.c +9
-0@@ -127,6 +127,13 @@ static bool pcie_ats_disabled;/* If set, the PCI config space of each device is printed during boot. */ bool pci_early_dump; +/* +* If set, the devices with "untrusted" flag shall not be attached automatically +* Userspace will need to attach them manually: +* echo <pci device> > /sys/bus/pci/drivers/<driver>/bind +*/ +bool pci_dont_attach_untrusted_devs; +bool pci_ats_disabled (void ) {return pcie_ats_disabled;@@ -6522,6 +6529,8 @@ static int __init pci_setup(char *str)pci_add_flags (PCI_SCAN_ALL_PCIE_DEVS ); }else if (!strncmp (str,"disable_acs_redir=" ,18 )) { disable_acs_redir_param = str +18 ; + }else if (!strcmp (str,"dont_attach_untrusted_devs" )) { + pci_dont_attach_untrusted_devs = true; }else {pr_err ("PCI: Unknown option `%s'\n" , str); }drivers/pci/pci.h +1
-0@@ -13,6 +13,7 @@extern const unsigned char pcie_link_speed[];extern bool pci_early_dump; +extern bool pci_dont_attach_untrusted_devs;
Quoted 8 lines
Quoted 5 lines
drivers/pci/pci.h +0
-1bool pcie_cap_has_lnkctl (const struct pci_dev *dev);bool pcie_cap_has_rtctl (const struct pci_dev *dev); --2.27.0.212.ge8ba1cc988-goog
What happened to the split of "trust" and "internal/external" logic that we discussed before?
a) I think what was decided was introducing a device core "location" property that can be exposed to userspace to help it to decide whether or not to attach a driver to a device. Yes, that is still the plan. (Mild sidenote: userspace may not need to distinguish between internal and external devices if it can assume that no internal PCI devices will show up after "echo 0 > /sys/bus/pci/drivers_autoprobe". But nevertheless...)
b) Note that even with (a) in place, we still need a parameter that can ensure that drivers are not bound to external devices at boot, *before* userspace gets a chance to disable "drivers_autoprobe".
https://lkml.org/lkml/2020/6/15/1453
Is it OK to add such a parameter in device core?
Thanks,
Rajat
Quoted 6 lines
This seems to ignore all of that and go straight to some form of "we know what we trust, so all is fine!".
It's not obvious what this is really doing here at all, sorry...
greg k-h
Re: [PATCH 2/2] pci: Add parameter to disable attaching untrusted devices
Quoted 3 lines · Rajat Jain
a) I think what was decided was introducing a device core "location" property that can be exposed to userspace to help it to decide whether or not to attach a driver to a device. Yes, that is still the plan.
Great, but this patch ignores that and starts to add policy :(
Quoted 4 lines
(Mild sidenote: userspace may not need to distinguish between internal and external devices if it can assume that no internal PCI devices will show up after "echo 0 >
/sys/bus/pci/drivers_autoprobe". But nevertheless...)
It can not assume that.
Quoted 3 lines
b) Note that even with (a) in place, we still need a parameter that can ensure that drivers are not bound to external devices at boot, *before* userspace gets a chance to disable "
drivers_autoprobe".
Why do you think you need that? I kind of doubt you really want this, but ick, if you really do, make it a policy decision that you bake into the kernel as a build option, so that no one else has to use it :)
Quoted 1 line
Ick, please use lore.kernel.org, we don't control lkml.org and it's not all that reliable.
Quoted 1 line
Is it OK to add such a parameter in device core?
You don't have internal/external/wherever in the driver core yet, so don't start adding policy before you get that...
thanks,
greg k-h
Quoted 18 lines · Rajat Jain
device_attach()returning failure indicates a driver error while trying to probe the device. In such a scenario, the PCI device should still be added in the system and be visible to the user.This patch partially reverts:
commitab1a187bba5c("PCI: Checkdevice_attach()return value always")Signed-off-by Rajat Jain <[email protected]>--- drivers/pci/bus.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)drivers/pci/bus.c +0
-0
drivers/pci/bus.c void pci_bus_add_device(struct pci_dev *dev) · line 330
Quoted 10 lines
drivers/pci/bus.c +1
-5@@ -322,12 +322,8 @@ void pci_bus_add_device(struct pci_dev *dev) dev->match_driver = true; retval =device_attach (&dev->dev); -if (retval <0 && retval != -EPROBE_DEFER ) { +if (retval <0 && retval != -EPROBE_DEFER )pci_warn (dev,"device attach failed (%d)\n" , retval); -pci_proc_detach_device (dev); -pci_remove_sysfs_dev_files (dev); -return ; - }
Nice catch, sysfs stuff shouldn't be dependant if a driver is bound to a device or not.
Nit: when you update these patches, can you run "git log --oneline drivers/pci/bus.c" and make your subject lines match the convention? E.g.,
PCI: Add device even if driver attach failed
Quoted 4 lines · Rajat Jain
device_attach()returning failure indicates a driver error while trying to probe the device. In such a scenario, the PCI device should still be added in the system and be visible to the user.
Nit: please wrap logs to fill 75 characters. "git log" adds 4 spaces at the beginning, so 75+4 still fits nicely in 80 columns without wrapping.
Quoted 13 lines
This patch partially reverts:
commitab1a187bba5c("PCI: Checkdevice_attach()return value always")Signed-off-by Rajat Jain <[email protected]>--- drivers/pci/bus.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)drivers/pci/bus.c +0
-0
drivers/pci/bus.c void pci_bus_add_device(struct pci_dev *dev) · line 328
Quoted 8 lines
drivers/pci/bus.c +1
-3@@ -322,12 +322,8 @@ void pci_bus_add_device(struct pci_dev *dev) dev->match_driver = true; retval =device_attach (&dev->dev); -if (retval <0 && retval != -EPROBE_DEFER ) { +if (retval <0 && retval != -EPROBE_DEFER )pci_warn (dev,"device attach failed (%d)\n" , retval); -pci_proc_detach_device (dev); -pci_remove_sysfs_dev_files (dev);
Thanks for catching my bug!
Quoted 8 lines
drivers/pci/bus.c +0
-2-return ; - }pci_dev_assign_added (dev, true); }Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
Quoted 3 lines · Bjorn Helgaas
Nit: when you update these patches, can you run "git log --oneline
drivers/pci/bus.c" and make your subject lines match the convention?
Sorry, will do.
Quoted 5 lines
E.g.,
PCI: Add device even if driver attach failed
On Thu, Jun 25, 2020 at 05:27:09PM -0700, Rajat Jain wrote:
Quoted 8 lines
Quoted 4 lines
device_attach()returning failure indicates a driver error while trying to probe the device. In such a scenario, the PCI device should still be added in the system and be visible to the user.Nit: please wrap logs to fill 75 characters. "git log" adds 4 spaces at the beginning, so 75+4 still fits nicely in 80 columns without wrapping.
Sorry, will do.
Quoted 33 lines
Quoted 14 lines
This patch partially reverts:
commitab1a187bba5c("PCI: Checkdevice_attach()return value always")Signed-off-by Rajat Jain <[email protected]>--- drivers/pci/bus.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)drivers/pci/bus.c +0
-0drivers/pci/bus.c void pci_bus_add_device(struct pci_dev *dev) · line 328
Quoted 8 lines
drivers/pci/bus.c +1
-3@@ -322,12 +322,8 @@ void pci_bus_add_device(struct pci_dev *dev) dev->match_driver = true; retval =device_attach (&dev->dev); -if (retval <0 && retval != -EPROBE_DEFER ) { +if (retval <0 && retval != -EPROBE_DEFER )pci_warn (dev,"device attach failed (%d)\n" , retval); -pci_proc_detach_device (dev); -pci_remove_sysfs_dev_files (dev);Thanks for catching my bug!
Quoted 8 lines
drivers/pci/bus.c +0
-3-return ; - }pci_dev_assign_added (dev, true); } --2.27.0.212.ge8ba1cc988-goog