Skip to contentSkip to conversation

PCI/sysfs: Check IORESOURCE_DISABLED in resource mmap handler

PATCH Superseded Ravi Kumar Bandi 4 participants 10 messages May 12 – May 14 PCI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI/sysfs: Check IORESOURCE_DISABLED in resource mmap handler

pci_mmap_resource() does not check IORESOURCE_DISABLED before mapping a PCI BAR resource into userspace. This allows new mmaps to succeed even after a device has been marked disabled or soft-unplugged by the driver to prevent further access.

Add the check to return -ENODEV when the resource is disabled, blocking new userspace mmaps of BAR resources after device removal.

Tested by marking the PCI BAR resource as disabled and verifying that a subsequent mmap attempt fails with -ENODEV.

$ sudo python3 -c "
import os, mmap, errno
try:
    fd = os.open('/sys/bus/pci/devices/0001:01:00.0/resource0',os.O_RDONLY)
    mmap.mmap(fd, 4096, access=mmap.ACCESS_READ)
    print('mmap succeeded')
except OSError as e:
    print(f'mmap failed - {e}')
"
mmap failed - [Errno 19] No such device
$
---
 drivers/pci/pci-sysfs.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/pci-sysfs.c +3 -0
index d37860841260..61ae259f8ca8 100644
@@ -1089,6 +1089,9 @@ static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a
 	if (ret)
 		return ret;
 
+	if (res->flags & IORESOURCE_DISABLED)
+		return -ENODEV;
+
 	if (res->flags & IORESOURCE_MEM && iomem_is_exclusive(res->start))
 		return -EINVAL;
 
Signature · 1 line
2.47.3
#1 lore raw

Hello,

Quoted 4 lines

pci_mmap_resource() does not check IORESOURCE_DISABLED before mapping a PCI BAR resource into userspace. This allows new mmaps to succeed even after a device has been marked disabled or soft-unplugged by the driver to prevent further access.

Which driver disables resources? Would this be some Amazon-specific thing you are trying to fix? Or are you just manually disabling a given device using sysfs, or something like this?

For "soft-unplugged" device we have pci_dev_set_disconnected(), but this does not check current flags set.

What is your use case here?

Quoted 5 lines

Add the check to return -ENODEV when the resource is disabled, blocking new userspace mmaps of BAR resources after device removal.

Tested by marking the PCI BAR resource as disabled and verifying that a subsequent mmap attempt fails with -ENODEV.

Can you explain how did you do this?

drivers/pci/pci-sysfs.c static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a · line 1094

Quoted 7 lines
drivers/pci/pci-sysfs.c +3 -0
@@ -1089,6 +1089,9 @@ static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a
 	if (ret)
 		return ret;
 
+	if (res->flags & IORESOURCE_DISABLED)
+		return -ENODEV;
+

This probably would be better if it checked IORESOURCE_DISABLED and IORESOURCE_UNSET, but then probably using resource_assigned() would be even better.

Thank you!

Krzysztof

#2 lore raw
Quoted 28 lines · Krzysztof Wilczyński

Hello,

Quoted 4 lines

pci_mmap_resource() does not check IORESOURCE_DISABLED before mapping a PCI BAR resource into userspace. This allows new mmaps to succeed even after a device has been marked disabled or soft-unplugged by the driver to prevent further access.

Which driver disables resources? Would this be some Amazon-specific thing you are trying to fix? Or are you just manually disabling a given device using sysfs, or something like this?

For "soft-unplugged" device we have pci_dev_set_disconnected(), but this does not check current flags set.

What is your use case here?

Quoted 5 lines

Add the check to return -ENODEV when the resource is disabled, blocking new userspace mmaps of BAR resources after device removal.

Tested by marking the PCI BAR resource as disabled and verifying that a subsequent mmap attempt fails with -ENODEV.

Can you explain how did you do this?

Quoted 4 lines
drivers/pci/pci-sysfs.c +0 -0
@@ -1089,6 +1089,9 @@ static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a
 	if (ret)
 		return ret;
 
Quoted 7 lines

drivers/pci/pci-sysfs.c static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a · line 1094

Quoted 3 lines
drivers/pci/pci-sysfs.c +3 -0
+	if (res->flags & IORESOURCE_DISABLED)
+		return -ENODEV;
+

This probably would be better if it checked IORESOURCE_DISABLED and IORESOURCE_UNSET, but then probably using resource_assigned() would be even better.

Yes, resource_assigned() makes more sense.

When considering Krzysztof's sysfs rework series in pci/sysfs, this all should be handled in .is_visible and not in pci_mmap_resource().

Also (FYI), alpha has its own pci_mmap_resource() (IIRC, it still has it even after Krzysztof's series).

Signature · 1 line
 i.
#3 lore raw

Hello,

Quoted 34 lines
Quoted 26 lines
Quoted 4 lines

pci_mmap_resource() does not check IORESOURCE_DISABLED before mapping a PCI BAR resource into userspace. This allows new mmaps to succeed even after a device has been marked disabled or soft-unplugged by the driver to prevent further access.

Which driver disables resources? Would this be some Amazon-specific thing you are trying to fix? Or are you just manually disabling a given device using sysfs, or something like this?

For "soft-unplugged" device we have pci_dev_set_disconnected(), but this does not check current flags set.

What is your use case here?

Quoted 5 lines

Add the check to return -ENODEV when the resource is disabled, blocking new userspace mmaps of BAR resources after device removal.

Tested by marking the PCI BAR resource as disabled and verifying that a subsequent mmap attempt fails with -ENODEV.

Can you explain how did you do this?

Quoted 4 lines
drivers/pci/pci-sysfs.c +0 -0
@@ -1089,6 +1089,9 @@ static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a
 	if (ret)
 		return ret;
 
Quoted 7 lines

drivers/pci/pci-sysfs.c static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a · line 1094

Quoted 3 lines
drivers/pci/pci-sysfs.c +3 -0
+	if (res->flags & IORESOURCE_DISABLED)
+		return -ENODEV;
+

This probably would be better if it checked IORESOURCE_DISABLED and IORESOURCE_UNSET, but then probably using resource_assigned() would be even better.

Quoted 4 lines

Yes, resource_assigned() makes more sense.

When considering Krzysztof's sysfs rework series in pci/sysfs, this all should be handled in .is_visible and not in pci_mmap_resource().

We can't use the resource_assigned() sadly, but the flags is something we definitely can check - I will add this, test, and send new series.

The callback, though, will be called on creation, or if someone calls remove, create or update for the sysfs resource group - no driver does this at the moment (some DRM drivers resize resources, but that's unrelated), so I am not sure how Ravi is doing this at runtime. Perhaps I am looking wrong at the code somewhere...

Quoted 2 lines

Also (FYI), alpha has its own pci_mmap_resource() (IIRC, it still has it even after Krzysztof's series).

For the sparse/dense resources, yes.

Thank you!

Krzysztof

#4 lore raw
Quoted 10 lines · Krzysztof
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

pci_mmap_resource() does not check IORESOURCE_DISABLED before mapping a PCI BAR resource into userspace. This allows new mmaps to succeed even after a device has been marked disabled or soft-unplugged by the driver to prevent further access.

Which driver disables resources? Would this be some Amazon-specific thing you are trying to fix? Or are you just manually disabling a given device using sysfs, or something like this?

What is your use case here?

Thank you Krzysztof, Ilpo for reviewing the patch.

The use case is surprise removal of the PCIe device that does not have a PRSNT# routed to the CPU. Based on other internal states, the proprietary endpoint driver marks the BAR resources as disabled by updating this flag to prevent further userspace access to the mermory regions (mmap) to prevent kernel hang/crash. Currently, BAR memory regions will be allowed to be mmapped using the sysfs resource files even if drivers soft_unplug the device.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

For "soft-unplugged" device we have pci_dev_set_disconnected(), but this does not check current flags set.

Thanks for the suggestion. pci_dev_set_disconnected() couldn't be used directly as it's defined in drivers/pci/pci.h and not part of exported kernel headers, the endpoint driver is currently setting it as below along with the flags:

    xchg(&dev->error_state, pci_channel_io_perm_failure);
    smp_wmb();

    [...]

        pdev->resource[bar].flags |= IORESOURCE_DISABLED;

    [...]

I'll replace checking for IORESOURCE_DISABLED flag
with pci_dev_is_disconnected() instead and make endpoint driver adapt to using pci_dev_set_disconnected().

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Add the check to return -ENODEV when the resource is disabled, blocking new userspace mmaps of BAR resources after device removal.

Tested by marking the PCI BAR resource as disabled and verifying that a subsequent mmap attempt fails with -ENODEV.

Can you explain how did you do this?

Based on the internal logic, endpoint driver sets the resource flags as below to indicate soft_unplug, which pci-sysfs use to to reject mmap of the resources.

    pdev->resource[bar].flags |= IORESOURCE_DISABLED;
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

drivers/pci/pci-sysfs.c static int pci_mmap_resource(struct kobject *kobj · line 1094

Quoted 7 lines
drivers/pci/pci-sysfs.c +3 -0
@@ -1089,6 +1089,9 @@ static int pci_mmap_resource(struct kobject *kobj
  if (ret)
          return ret;
 
+ if (res->flags & IORESOURCE_DISABLED)
+         return -ENODEV;
+

This probably would be better if it checked IORESOURCE_DISABLED and IORESOURCE_UNSET, but then probably using resource_assigned() would be even better.

Thank you, but as you suggested above, I will update the patch to use pci_dev_is_disconnected() instead.

Regards,
Ravi Kumar Bandi

#7 lore raw

PATCH v2 1/1 PCI/sysfs: Check if device disabled in resource mmap handler

In reply to Ravi Kumar Bandi

pci_mmap_resource() does not check if device is enabled before mapping a PCI BAR resource into userspace. This allows new mmaps to succeed even after a device has been marked disabled or soft-unplugged by the driver to prevent further access.

Add the check to return -ENODEV when the resource is disabled, blocking new userspace mmaps of BAR resources after device removal.

Tested by marking the PCI BAR resource as disabled and verifying that a subsequent mmap attempt fails with -ENODEV.

$ sudo python3 -c "
import os, mmap, errno
try:
    fd = os.open('/sys/bus/pci/devices/0001:01:00.0/resource0',os.O_RDONLY)
    mmap.mmap(fd, 4096, access=mmap.ACCESS_READ)
    print('mmap succeeded')
except OSError as e:
    print(f'mmap failed - {e}')
"
mmap failed - [Errno 19] No such device
$
---
 drivers/pci/pci-sysfs.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/pci-sysfs.c +3 -0
index d37860841260..83d580256c4b 100644
@@ -1089,6 +1089,9 @@ static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a
 	if (ret)
 		return ret;
 
+	if (pci_dev_is_disconnected(pdev))
+		return -ENODEV;
+
 	if (res->flags & IORESOURCE_MEM && iomem_is_exclusive(res->start))
 		return -EINVAL;
 
Signature · 1 line
2.47.3
#5 lore raw
In reply to Ravi Kumar Bandi

Hello,

Quoted 2 lines

Thank you, but as you suggested above, I will update the patch to use pci_dev_is_disconnected() instead.

There also exists pci_device_is_present() which would attempt an actual read of a vendor/device IDs, so if this succeeds, then you have a more solid assurance that the device should be working. However, this adds latency, and there might be side-effects, potentially...

I wonder, if for you, you could forfeit setting any flags on a resource, which drivers don't customarily do, such that we would only rely on checking pci_dev_is_disconnected() on our side, so to speak.

We won't be setting any precedent here, as the helpers such as pci_read_config_byte(), perform this check internally already.

Bjorn and Ilpo might have different take on this, though.

Thank you!

Krzysztof

#6 lore raw
Quoted 4 lines · Krzysztof Wilczyński

There also exists pci_device_is_present() which would attempt an actual read of a vendor/device IDs, so if this succeeds, then you have a more solid assurance that the device should be working. However, this adds latency, and there might be side-effects, potentially...

Yeah, exactly - it adds latency and will be more profound if applications mmap the resources periodically.

Quoted 8 lines

I wonder, if for you, you could forfeit setting any flags on a resource, which drivers don't customarily do, such that we would only rely on checking pci_dev_is_disconnected() on our side, so to speak.

We won't be setting any precedent here, as the helpers such as pci_read_config_byte(), perform this check internally already.

Bjorn and Ilpo might have different take on this, though.

Thank you for the suggestion. Yes, we can forfeit setting IORESOURCE_DISABLED and rely solely on pci_dev_is_disconnected() check in pci_mmap_resource(). I have submitted v2 patch, please review.

Thank you.
Ravi Kumar Bandi

#9 lore raw
In reply to Ravi Kumar Bandi
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] IORESOURCE_DISABLED is an enumeration/allocation flag, not a dynamic operational state flag.…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] IORESOURCE_DISABLED is an enumeration/allocation flag, not a dynamic operational state flag.
- [High] The patch fails to check for IORESOURCE_UNSET, leaving a path to map physical address 0.
- [Medium] The check was omitted from the legacy procfs mmap interface and sysfs I/O port read/write handlers. --

commit b337c28a9b32f2246e39d66b4dc57399f253f527
Author: Ravi Kumar Bandi <[email protected]> PCI/sysfs: Check IORESOURCE_DISABLED in resource mmap handler

This commit attempts to block userspace from mapping PCI BAR resources of disabled or soft-unplugged devices. It does so by adding a check for the IORESOURCE_DISABLED flag in the sysfs pci_mmap_resource() handler.

Quoted 4 lines

pci_mmap_resource() does not check IORESOURCE_DISABLED before mapping a PCI BAR resource into userspace. This allows new mmaps to succeed even after a device has been marked disabled or soft-unplugged by the driver to prevent further access.

Is IORESOURCE_DISABLED actually set dynamically when a device is disabled or soft-unplugged by the driver?

Looking at the PCI core, IORESOURCE_DISABLED appears to be an allocation flag used when a resource cannot be assigned structurally (such as a 64-bit BAR on a 32-bit system). It doesn't seem to be dynamically set by pci_disable_device() or soft-unplug paths like pci_stop_and_remove_bus_device().

If the flag is never set during a soft-unplug, will this check successfully prevent the post-removal mmaps as intended?

drivers/pci/pci-sysfs.c static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a · line 1094

Quoted 11 lines
drivers/pci/pci-sysfs.c +3 -0
index d37860841260c..61ae259f8ca80 100644
@@ -1089,6 +1089,9 @@ static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a
 	if (ret)
 		return ret;
 
+	if (res->flags & IORESOURCE_DISABLED)
+		return -ENODEV;
+

Should this also check for IORESOURCE_UNSET?

When the PCI core fails to assign an address to a resource, it sets IORESOURCE_UNSET and sets the resource start address to 0. Since the size remains non-zero, the sysfs files are still created and exposed.

Without checking IORESOURCE_UNSET, could pci_mmap_resource_range() add pci_resource_start() >> PAGE_SHIFT (which is 0) to vma->vm_pgoff, allowing userspace to inadvertently map physical address 0 (System RAM/BIOS) instead of an MMIO region?

Additionally, does this protection need to be extended to other interfaces?

Even with this check in sysfs mmap, userspace might still bypass it using the legacy procfs mmap interface via proc_bus_pci_mmap().

Furthermore, for I/O port BARs (IORESOURCE_IO), the sysfs read and write callbacks (pci_read_resource_io and pci_write_resource_io) directly execute inb/outb instructions without verifying if the resource is disabled. Could an application holding an open file descriptor continue raw hardware accesses and bypass this protection?

Quoted 3 lines
drivers/pci/pci-sysfs.c +0 -0
 	if (res->flags & IORESOURCE_MEM && iomem_is_exclusive(res->start))
 		return -EINVAL;
 
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#8 lore raw

Re: [PATCH v2] PCI/sysfs: Check if device disabled in resource mmap handler

In reply to Ravi Kumar Bandi
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The `pci_dev_is_disconnected()` check fails to block mmaps on soft-unplugged devices and…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The pci_dev_is_disconnected() check fails to block mmaps on soft-unplugged devices and contradicts the commit's stated intent. --

commit fff70a5b941929593fed8b551604aadd9e28748d
Author: Ravi Kumar Bandi <[email protected]>

PCI/sysfs: Check if device disabled in resource mmap handler

This commit adds a check to pci_mmap_resource() to prevent mapping a PCI BAR resource into userspace if the device is disconnected. The stated intent is to block new mmaps after a device has been marked disabled or soft-unplugged.

drivers/pci/pci-sysfs.c static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a · line 1094

Quoted 11 lines
drivers/pci/pci-sysfs.c +3 -0
index d37860841260c..83d580256c4b2 100644
@@ -1089,6 +1089,9 @@ static int pci_mmap_resource(struct kobject *kobj, const struct bin_attribute *a
 	if (ret)
 		return ret;
 
+	if (pci_dev_is_disconnected(pdev))
+		return -ENODEV;
+

The commit message mentions that this change prevents new mmaps after a device has been marked disabled or soft-unplugged. Does this check accurately capture a soft-unplug?

Looking at pci_dev_is_disconnected(), it strictly checks if the device's error_state is pci_channel_io_perm_failure. This state is typically set during surprise hotplug removals or fatal PCIe errors.

During a graceful driver unbind or a soft-removal via sysfs, the error_state remains pci_channel_io_normal. Would a concurrent mmap() on an already-open file descriptor still succeed during the window before the sysfs files are completely removed, bypassing this protection?

Additionally, if the intention was to literally check if the device is disabled (as the commit subject suggests), would that introduce a regression for userspace? Valid userspace tools, such as X servers or VFIO utilities, often map PCI BARs prior to explicitly enabling the device.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1