Skip to contentSkip to conversation

PCI/IOV: Fix out-of-bounds access in sriov_restore_vf_rebar_state()

PATCH In review Marco Nenciarini 4 participants 10 messages Apr 8 – Apr 24

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI/IOV: Fix out-of-bounds access in sriov_restore_vf_rebar_state()

sriov_restore_vf_rebar_state() extracts bar_idx from the VF Resizable BAR control register using a 3-bit field (PCI_VF_REBAR_CTRL_BAR_IDX, bits 0-2), which yields values in the range 0-7. This value is then used to index into dev->sriov->barsz[], which has PCI_SRIOV_NUM_BARS
(6) entries.

If the PCI config space read returns garbage data (e.g. 0xffffffff when the device is no longer accessible on the bus), bar_idx is 7, causing an out-of-bounds array access. UBSAN reports this as:

  UBSAN: array-index-out-of-bounds in drivers/pci/iov.c:948:51
  index 7 is out of range for type 'resource_size_t [6]'

This was observed on an NVIDIA RTX PRO 1000 GPU (GB207GLM) that fell off the PCIe bus during a failed GC6 power state exit. The subsequent pci_restore_state() call triggered the UBSAN splat in sriov_restore_vf_rebar_state() since all config space reads returned 0xffffffff.

Add a bounds check on bar_idx before using it as an array index to prevent the out-of-bounds access.

Fixes 5a8f77e24a30 ("PCI/IOV: Restore VF resizable BAR state after reset")

---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>

 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/iov.c +2 -0
index 00784a60b..521f2cb64 100644
@@ -946,6 +946,8 @@ static void sriov_restore_vf_rebar_state(struct pci_dev *dev)
 
 		pci_read_config_dword(dev, pos + PCI_VF_REBAR_CTRL, &ctrl);
 		bar_idx = FIELD_GET(PCI_VF_REBAR_CTRL_BAR_IDX, ctrl);
+		if (bar_idx >= PCI_SRIOV_NUM_BARS)
+			continue;
 		size = pci_rebar_bytes_to_size(dev->sriov->barsz[bar_idx]);
 		ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE;
 		ctrl |= FIELD_PREP(PCI_VF_REBAR_CTRL_BAR_SIZE, size);
Signature · 1 line
2.47.3
#1 lore raw

Gentle ping on this. It is a one-line bounds check fix for a real UBSAN splat triggered when config space reads return 0xffffffff (device fallen off the bus).

Reviewed-by on v1 1/1
#2 lore raw
In reply to Marco Nenciarini
Quoted 19 lines · Marco Nenciarini

sriov_restore_vf_rebar_state() extracts bar_idx from the VF Resizable BAR control register using a 3-bit field (PCI_VF_REBAR_CTRL_BAR_IDX, bits 0-2), which yields values in the range 0-7. This value is then used to index into dev->sriov->barsz[], which has PCI_SRIOV_NUM_BARS
(6) entries.

If the PCI config space read returns garbage data (e.g. 0xffffffff when the device is no longer accessible on the bus), bar_idx is 7, causing an out-of-bounds array access. UBSAN reports this as:

  UBSAN: array-index-out-of-bounds in drivers/pci/iov.c:948:51
  index 7 is out of range for type 'resource_size_t [6]'

This was observed on an NVIDIA RTX PRO 1000 GPU (GB207GLM) that fell off the PCIe bus during a failed GC6 power state exit. The subsequent pci_restore_state() call triggered the UBSAN splat in sriov_restore_vf_rebar_state() since all config space reads returned 0xffffffff.

Quoted 6 lines

Add a bounds check on bar_idx before using it as an array index to prevent the out-of-bounds access.

Fixes 5a8f77e24a30 ("PCI/IOV: Restore VF resizable BAR state after reset")

Thanks,
-Michał

Quoted 23 lines

---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>

 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/iov.c +2 -0
index 00784a60b..521f2cb64 100644
@@ -946,6 +946,8 @@ static void sriov_restore_vf_rebar_state(struct pci_dev *dev)
 
 		pci_read_config_dword(dev, pos + PCI_VF_REBAR_CTRL, &ctrl);
 		bar_idx = FIELD_GET(PCI_VF_REBAR_CTRL_BAR_IDX, ctrl);
+		if (bar_idx >= PCI_SRIOV_NUM_BARS)
+			continue;
 		size = pci_rebar_bytes_to_size(dev->sriov->barsz[bar_idx]);
 		ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE;
 		ctrl |= FIELD_PREP(PCI_VF_REBAR_CTRL_BAR_SIZE, size);
Signature · 1 line
2.47.3
#3 lore raw
In reply to Marco Nenciarini
Quoted 13 lines · Marco Nenciarini

sriov_restore_vf_rebar_state() extracts bar_idx from the VF Resizable BAR control register using a 3-bit field (PCI_VF_REBAR_CTRL_BAR_IDX, bits 0-2), which yields values in the range 0-7. This value is then used to index into dev->sriov->barsz[], which has PCI_SRIOV_NUM_BARS
(6) entries.

If the PCI config space read returns garbage data (e.g. 0xffffffff when the device is no longer accessible on the bus), bar_idx is 7, causing an out-of-bounds array access. UBSAN reports this as:

  UBSAN: array-index-out-of-bounds in drivers/pci/iov.c:948:51
  index 7 is out of range for type 'resource_size_t [6]'
Quoted 8 lines

This was observed on an NVIDIA RTX PRO 1000 GPU (GB207GLM) that fell off the PCIe bus during a failed GC6 power state exit. The subsequent pci_restore_state() call triggered the UBSAN splat in sriov_restore_vf_rebar_state() since all config space reads returned 0xffffffff.

Add a bounds check on bar_idx before using it as an array index to prevent the out-of-bounds access.

I think pci_restore_rebar_state() has a similar problem: if the device doesn't respond, "nbars" will be 7 (legal values are 1-6), and "bar_idx" will also be 7 (legal values 0-5). We use "bar_idx" for pci_resource_n(), and dev->resource[7] does exist but is not a valid BAR.

Quoted 15 lines
Fixes 5a8f77e24a30 ("PCI/IOV: Restore VF resizable BAR state after reset")

---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>

 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/iov.c +0 -0
index 00784a60b..521f2cb64 100644

drivers/pci/iov.c static void sriov_restore_vf_rebar_state(struct pci_dev *dev) · line 950

Quoted 5 lines
drivers/pci/iov.c +2 -0
@@ -946,6 +946,8 @@ static void sriov_restore_vf_rebar_state(struct pci_dev *dev)
 
 		pci_read_config_dword(dev, pos + PCI_VF_REBAR_CTRL, &ctrl);
 		bar_idx = FIELD_GET(PCI_VF_REBAR_CTRL_BAR_IDX, ctrl);
+		if (bar_idx >= PCI_SRIOV_NUM_BARS)
+			continue;

Both here and in pci_restore_rebar_state(), we blindly use "nbars" derived from a value that might be ~0 because the config read failed. If we fix these, I think we should do something like this so it's obvious why we're checking:

  pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
  if (PCI_POSSIBLE_ERROR(ctrl))
    return;

  nbars = FIELD_GET(PCI_REBAR_CTRL_NBAR_MASK, ctrl);
  for (i = 0; i < nbars; ...) {
    ...
    pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
    if (PCI_POSSIBLE_ERROR(ctrl))
      return;
    bar_idx = ctrl & PCI_REBAR_CTRL_BAR_IDX;
    res = pci_resource_n(pdev, bar_idx);

It's true that "nbars" and "bar_idx" *could* still be invalid even if the config read succeeded, but that would be a device defect.

Quoted 6 lines
drivers/pci/iov.c +0 -0
 		size = pci_rebar_bytes_to_size(dev->sriov->barsz[bar_idx]);
 		ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE;
 		ctrl |= FIELD_PREP(PCI_VF_REBAR_CTRL_BAR_SIZE, size);
Signature · 1 line
2.47.3
#4 lore raw
In reply to Marco Nenciarini

[+cc Rafael, Eric, Alex, Lukas, generic pci_restore_state() question]

Quoted 10 lines · Marco Nenciarini

sriov_restore_vf_rebar_state() extracts bar_idx from the VF Resizable BAR control register using a 3-bit field (PCI_VF_REBAR_CTRL_BAR_IDX, bits 0-2), which yields values in the range 0-7. This value is then used to index into dev->sriov->barsz[], which has PCI_SRIOV_NUM_BARS
(6) entries.

If the PCI config space read returns garbage data (e.g. 0xffffffff when the device is no longer accessible on the bus), bar_idx is 7, causing an out-of-bounds array access. UBSAN reports this as:

Quoted 8 lines
  UBSAN: array-index-out-of-bounds in drivers/pci/iov.c:948:51
  index 7 is out of range for type 'resource_size_t [6]'

This was observed on an NVIDIA RTX PRO 1000 GPU (GB207GLM) that fell off the PCIe bus during a failed GC6 power state exit. The subsequent pci_restore_state() call triggered the UBSAN splat in sriov_restore_vf_rebar_state() since all config space reads returned 0xffffffff.

I think all of pci_restore_state() is problematic for all devices, not just this GPU. If these config reads fail, all the previous config writes probably failed (silently) as well.

And we have this weird retry loop in pci_restore_config_dword():
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/pci.c?id=v7.0#n1766,
which was originally added by
https://git.kernel.org/linus/26f41062f28d ("PCI: check for pci bar restore completion and retry") to fix an actual problem:

On some OEM systems, pci_restore_state() is called while FLR has not yet completed. As a result, PCI BAR register restore is not successful. This fix reads back the restored value and compares it with saved value and re-tries 10 times before giving up.

This just gives me the heebie-jeebies. If we still need this retry loop, it means all the previous state restoration (PCIe, LTR, ASPM, IOV, PRI, ATS, DPC, etc.) probably failed, and we end up with a device where the BARs got restored but none of the previous stuff. That sounds like a mess.

Quoted 29 lines

Add a bounds check on bar_idx before using it as an array index to prevent the out-of-bounds access.

Fixes 5a8f77e24a30 ("PCI/IOV: Restore VF resizable BAR state after reset")

---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>

 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/iov.c +2 -0
index 00784a60b..521f2cb64 100644
@@ -946,6 +946,8 @@ static void sriov_restore_vf_rebar_state(struct pci_dev *dev)
 
 		pci_read_config_dword(dev, pos + PCI_VF_REBAR_CTRL, &ctrl);
 		bar_idx = FIELD_GET(PCI_VF_REBAR_CTRL_BAR_IDX, ctrl);
+		if (bar_idx >= PCI_SRIOV_NUM_BARS)
+			continue;
 		size = pci_rebar_bytes_to_size(dev->sriov->barsz[bar_idx]);
 		ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE;
 		ctrl |= FIELD_PREP(PCI_VF_REBAR_CTRL_BAR_SIZE, size);
Signature · 1 line
2.47.3
#5 lore raw
Quoted 11 lines · Bjorn Helgaas

And we have this weird retry loop in pci_restore_config_dword():
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/pci.c?id=v7.0#n1766,
which was originally added by
https://git.kernel.org/linus/26f41062f28d ("PCI: check for pci bar restore completion and retry") to fix an actual problem:

On some OEM systems, pci_restore_state() is called while FLR has not yet completed. As a result, PCI BAR register restore is not successful. This fix reads back the restored value and compares it with saved value and re-tries 10 times before giving up.

Quoted 5 lines

This just gives me the heebie-jeebies. If we still need this retry loop, it means all the previous state restoration (PCIe, LTR, ASPM, IOV, PRI, ATS, DPC, etc.) probably failed, and we end up with a device where the BARs got restored but none of the previous stuff. That sounds like a mess.

Nowadays we wait for devices to re-appear after reset by polling the Vendor ID register, see the call to pci_dev_wait() in pcie_flr().

It seems we didn't do that back in the day when 26f41062f28d introduced the loop. The commit went into v3.4 and back then, pcie_flr() only waited for 100 msec:

https://elixir.bootlin.com/linux/v3.4/source/drivers/pci/pci.c#L3052

And indeed pci_reset_function() immediately restored config space afterwards:

https://elixir.bootlin.com/linux/v3.4/source/drivers/pci/pci.c#L3285

So I strongly suspect that the loop no longer has a valid raison d'être. Maybe remove it early in the next cycle to get linux-next coverage for 8 weeks and see if anything breaks (which I doubt)?

As to validity of cached config space state in general, see this discussion with Ilpo yesterday, in response to a regression fix I submitted:

https://lore.kernel.org/all/[email protected]/

Thanks,

Lukas

#6 lore raw

v2 cover PCI: Guard Resizable BAR restore against unreachable devices

In reply to Marco Nenciarini

This series addresses Bjorn's review feedback on v1 [1].

v1 bounds-checked bar_idx before indexing dev->sriov->barsz[] in sriov_restore_vf_rebar_state(). Bjorn pointed out that the non-SRIOV sibling pci_restore_rebar_state() has the same issue, and that a PCI_POSSIBLE_ERROR(ctrl) check on the config read makes the intent of the guard more obvious than a post-hoc range check on the extracted field.

v2 therefore adopts PCI_POSSIBLE_ERROR(ctrl) after each Resizable BAR Control read in both functions, bailing out when config reads return the all-ones pattern. Patch 1 covers pci_restore_rebar_state(). Patch 2 covers sriov_restore_vf_rebar_state(), with the NVIDIA GC6 UBSAN splat as motivation.

Note that this changes behavior versus v1: on a bad read we abort the loop instead of skipping just the current BAR. This matches the structure Bjorn suggested in review and is safe because the all-ones pattern means the device is unreachable, so restoring the remaining BARs is moot.

Compile-tested on pci/next (full drivers/pci/ build). The error path cannot be exercised without reproducing the GC6 failure that killed the GPU in the original report.

The broader v1 discussion on the pci_restore_config_dword() retry loop and on save/restore behavior when the device has fallen off the bus is out of scope for this fix. Happy to tackle that separately if there is consensus.

[1] https://lore.kernel.org/all/[email protected]/

Marco Nenciarini (2):
PCI: Skip Resizable BAR restore on read error PCI/IOV: Skip VF Resizable BAR restore on read error

 drivers/pci/iov.c   | 6 ++++++
 drivers/pci/rebar.c | 6 ++++++
 2 files changed, 12 insertions(+)

base-commit: 40286d6379aacfcc053253ef78dc78b09addffda

Signature · 1 line
2.47.3
#7 lore raw

PATCH v2 1/2 PCI: Skip Resizable BAR restore on read error

pci_restore_rebar_state() uses the Resizable BAR Control register to decide how many BARs to restore (nbars) and which BAR each iteration addresses (bar_idx).

When the device does not respond, config reads return the all-ones pattern. Both fields are 3 bits wide, so nbars and bar_idx both evaluate to 7, past the spec's valid ranges for both fields. pci_resource_n() then returns an unrelated resource slot, whose size is used to derive a nonsensical value written back to the Resizable BAR Control register.

Bail out if any Resizable BAR Control read returns the error pattern. No further BARs are touched, which is safe because a config read that returns the error pattern indicates the device is unreachable and restoration is pointless.

Fixes d3252ace0bc6 ("PCI: Restore resized BAR state on resume")

---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>
Cc: Rafael J. Wysocki <[email protected]>
Cc: Eric Chanudet <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Lukas Wunner <[email protected]>

 drivers/pci/rebar.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/pci/rebar.c +6 -0
index 39f8cf3b70d57..11965947c4cb5 100644
@@ -231,6 +231,9 @@ void pci_restore_rebar_state(struct pci_dev *pdev)
 		return;
 
 	pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+	if (PCI_POSSIBLE_ERROR(ctrl))
+		return;
+
 	nbars = FIELD_GET(PCI_REBAR_CTRL_NBAR_MASK, ctrl);
 
 	for (i = 0; i < nbars; i++, pos += 8) {
@@ -238,6 +241,9 @@ void pci_restore_rebar_state(struct pci_dev *pdev)
 		int bar_idx, size;
 
 		pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+		if (PCI_POSSIBLE_ERROR(ctrl))
+			return;
+
 		bar_idx = ctrl & PCI_REBAR_CTRL_BAR_IDX;
 		res = pci_resource_n(pdev, bar_idx);
 		size = pci_rebar_bytes_to_size(resource_size(res));
Signature · 1 line
2.47.3
#8 lore raw

PATCH v2 2/2 PCI/IOV: Skip VF Resizable BAR restore on read error

In reply to Marco Nenciarini

sriov_restore_vf_rebar_state() uses the VF Resizable BAR Control register to decide how many VF BARs to restore (nbars) and which VF BAR each iteration addresses (bar_idx). bar_idx indexes into dev->sriov->barsz[], which has only PCI_SRIOV_NUM_BARS (6) entries.

When the device does not respond, config reads return the all-ones pattern. Both fields are 3 bits wide, so nbars and bar_idx both evaluate to 7. The barsz[] access then goes out of bounds. UBSAN reports this as:

  UBSAN: array-index-out-of-bounds in drivers/pci/iov.c:948:51
  index 7 is out of range for type 'resource_size_t [6]'

Observed on an NVIDIA RTX PRO 1000 GPU (GB207GLM) that fell off the PCIe bus during a failed GC6 power state exit. The subsequent pci_restore_state() invoked sriov_restore_vf_rebar_state() while config reads returned 0xffffffff, triggering the splat.

Bail out if any VF Resizable BAR Control read returns the error pattern. No further VF BARs are touched, which is safe because a config read that returns the error pattern indicates the device is unreachable and restoration is pointless. This mirrors the guard in pci_restore_rebar_state().

Fixes 5a8f77e24a30 ("PCI/IOV: Restore VF resizable BAR state after reset")

---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>
Cc: Rafael J. Wysocki <[email protected]>
Cc: Eric Chanudet <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Lukas Wunner <[email protected]>

 drivers/pci/iov.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/pci/iov.c +6 -0
index 91ac4e37ecb9c..08df9bace13d1 100644
@@ -938,12 +938,18 @@ static void sriov_restore_vf_rebar_state(struct pci_dev *dev)
 		return;
 
 	pci_read_config_dword(dev, pos + PCI_VF_REBAR_CTRL, &ctrl);
+	if (PCI_POSSIBLE_ERROR(ctrl))
+		return;
+
 	nbars = FIELD_GET(PCI_VF_REBAR_CTRL_NBAR_MASK, ctrl);
 
 	for (i = 0; i < nbars; i++, pos += 8) {
 		int bar_idx, size;
 
 		pci_read_config_dword(dev, pos + PCI_VF_REBAR_CTRL, &ctrl);
+		if (PCI_POSSIBLE_ERROR(ctrl))
+			return;
+
 		bar_idx = FIELD_GET(PCI_VF_REBAR_CTRL_BAR_IDX, ctrl);
 		size = pci_rebar_bytes_to_size(dev->sriov->barsz[bar_idx]);
 		ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE;
Signature · 1 line
2.47.3
#9 lore raw

Re: [PATCH v2 0/2] PCI: Guard Resizable BAR restore against unreachable devices

In reply to Marco Nenciarini
Quoted 32 lines · Marco Nenciarini

This series addresses Bjorn's review feedback on v1 [1].

v1 bounds-checked bar_idx before indexing dev->sriov->barsz[] in sriov_restore_vf_rebar_state(). Bjorn pointed out that the non-SRIOV sibling pci_restore_rebar_state() has the same issue, and that a PCI_POSSIBLE_ERROR(ctrl) check on the config read makes the intent of the guard more obvious than a post-hoc range check on the extracted field.

v2 therefore adopts PCI_POSSIBLE_ERROR(ctrl) after each Resizable BAR Control read in both functions, bailing out when config reads return the all-ones pattern. Patch 1 covers pci_restore_rebar_state(). Patch 2 covers sriov_restore_vf_rebar_state(), with the NVIDIA GC6 UBSAN splat as motivation.

Note that this changes behavior versus v1: on a bad read we abort the loop instead of skipping just the current BAR. This matches the structure Bjorn suggested in review and is safe because the all-ones pattern means the device is unreachable, so restoring the remaining BARs is moot.

Compile-tested on pci/next (full drivers/pci/ build). The error path cannot be exercised without reproducing the GC6 failure that killed the GPU in the original report.

The broader v1 discussion on the pci_restore_config_dword() retry loop and on save/restore behavior when the device has fallen off the bus is out of scope for this fix. Happy to tackle that separately if there is consensus.

[1] https://lore.kernel.org/all/[email protected]/

Quoted 7 lines

Marco Nenciarini (2):
PCI: Skip Resizable BAR restore on read error PCI/IOV: Skip VF Resizable BAR restore on read error

 drivers/pci/iov.c   | 6 ++++++
 drivers/pci/rebar.c | 6 ++++++
 2 files changed, 12 insertions(+)

Applied to pci/pm for v7.2, thanks! Will be rebased after v7.1-rc1.

Quoted 4 lines

base-commit: 40286d6379aacfcc053253ef78dc78b09addffda

Signature · 1 line
2.47.3