PCI/IOV: Fix out-of-bounds access in sriov_restore_vf_rebar_state()
Subsystems (2)
PATCH 1/1 PCI/IOV: Fix out-of-bounds access in sriov_restore_vf_rebar_state()
sriov_restore_vf_rebar_state() extracts bar_idx from the VF Resizable BAR control register using a 3-bit field (PCI_VF_REBAR_CTRL_BAR_IDX, bits 0-2), which yields values in the range 0-7. This value is then used to index into dev->sriov->barsz[], which has PCI_SRIOV_NUM_BARS
(6) entries.
If the PCI config space read returns garbage data (e.g. 0xffffffff when the device is no longer accessible on the bus), bar_idx is 7, causing an out-of-bounds array access. UBSAN reports this as:
UBSAN: array-index-out-of-bounds in drivers/pci/iov.c:948:51 index 7 is out of range for type 'resource_size_t [6]'
This was observed on an NVIDIA RTX PRO 1000 GPU (GB207GLM) that fell off the PCIe bus during a failed GC6 power state exit. The subsequent pci_restore_state() call triggered the UBSAN splat in sriov_restore_vf_rebar_state() since all config space reads returned 0xffffffff.
Add a bounds check on bar_idx before using it as an array index to prevent the out-of-bounds access.
---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>
drivers/pci/iov.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/pci/iov.c +2 -0
@@ -946,6 +946,8 @@ static void sriov_restore_vf_rebar_state(struct pci_dev *dev)pci_read_config_dword (dev, pos +PCI_VF_REBAR_CTRL , &ctrl); bar_idx =FIELD_GET (PCI_VF_REBAR_CTRL_BAR_IDX , ctrl); +if (bar_idx >=PCI_SRIOV_NUM_BARS ) +continue ; size =pci_rebar_bytes_to_size (dev->sriov->barsz[bar_idx]); ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE ; ctrl |=FIELD_PREP (PCI_VF_REBAR_CTRL_BAR_SIZE , size);
Signature · 1 line
2.47.3
Gentle ping on this. It is a one-line bounds check fix for a real UBSAN splat triggered when config space reads return 0xffffffff (device fallen off the bus).
Quoted 19 lines · Marco Nenciarini
sriov_restore_vf_rebar_state()extractsbar_idxfrom the VF Resizable BAR control register using a 3-bit field (PCI_VF_REBAR_CTRL_BAR_IDX, bits 0-2), which yields values in the range 0-7. This value is then used to index into dev->sriov->barsz[], which hasPCI_SRIOV_NUM_BARS
(6) entries.If the PCI config space read returns garbage data (e.g.
0xffffffffwhen the device is no longer accessible on the bus),bar_idxis 7, causing an out-of-bounds array access. UBSAN reports this as:UBSAN: array-index-out-of-bounds in drivers/pci/iov.c:948:51 index 7 is out of range for type 'resource_size_t [6]'This was observed on an NVIDIA RTX PRO 1000 GPU (GB207GLM) that fell off the PCIe bus during a failed GC6 power state exit. The subsequent
pci_restore_state()call triggered the UBSAN splat insriov_restore_vf_rebar_state()since all config space reads returned0xffffffff.
Quoted 6 lines
Add a bounds check on
bar_idxbefore using it as an array index to prevent the out-of-bounds access.Fixes 5a8f77e24a30 ("PCI/IOV: Restore VF resizable BAR state after reset")Signed-off-by Marco Nenciarini <[email protected]>
Thanks,
-Michał
Quoted 23 lines
---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>drivers/pci/iov.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/iov.c +2
-0@@ -946,6 +946,8 @@ static void sriov_restore_vf_rebar_state(struct pci_dev *dev)pci_read_config_dword (dev, pos +PCI_VF_REBAR_CTRL , &ctrl); bar_idx =FIELD_GET (PCI_VF_REBAR_CTRL_BAR_IDX , ctrl); +if (bar_idx >=PCI_SRIOV_NUM_BARS ) +continue ; size =pci_rebar_bytes_to_size (dev->sriov->barsz[bar_idx]); ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE ; ctrl |=FIELD_PREP (PCI_VF_REBAR_CTRL_BAR_SIZE , size);Signature · 1 line
2.47.3
Quoted 13 lines · Marco Nenciarini
sriov_restore_vf_rebar_state()extractsbar_idxfrom the VF Resizable BAR control register using a 3-bit field (PCI_VF_REBAR_CTRL_BAR_IDX, bits 0-2), which yields values in the range 0-7. This value is then used to index into dev->sriov->barsz[], which hasPCI_SRIOV_NUM_BARS
(6) entries.If the PCI config space read returns garbage data (e.g.
0xffffffffwhen the device is no longer accessible on the bus),bar_idxis 7, causing an out-of-bounds array access. UBSAN reports this as:UBSAN: array-index-out-of-bounds in drivers/pci/iov.c:948:51 index 7 is out of range for type 'resource_size_t [6]'
Quoted 8 lines
This was observed on an NVIDIA RTX PRO 1000 GPU (GB207GLM) that fell off the PCIe bus during a failed GC6 power state exit. The subsequent
pci_restore_state()call triggered the UBSAN splat insriov_restore_vf_rebar_state()since all config space reads returned0xffffffff.Add a bounds check on
bar_idxbefore using it as an array index to prevent the out-of-bounds access.
I think pci_restore_rebar_state() has a similar problem: if the device doesn't respond, "nbars" will be 7 (legal values are 1-6), and "bar_idx" will also be 7 (legal values 0-5). We use "bar_idx" for pci_resource_n(), and dev->resource[7] does exist but is not a valid BAR.
Quoted 15 lines
Fixes 5a8f77e24a30 ("PCI/IOV: Restore VF resizable BAR state after reset")Signed-off-by Marco Nenciarini <[email protected]>---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>drivers/pci/iov.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/iov.c +0
-0
drivers/pci/iov.c static void sriov_restore_vf_rebar_state(struct pci_dev *dev) · line 950
Quoted 5 lines
drivers/pci/iov.c +2
-0@@ -946,6 +946,8 @@ static void sriov_restore_vf_rebar_state(struct pci_dev *dev)pci_read_config_dword (dev, pos +PCI_VF_REBAR_CTRL , &ctrl); bar_idx =FIELD_GET (PCI_VF_REBAR_CTRL_BAR_IDX , ctrl); +if (bar_idx >=PCI_SRIOV_NUM_BARS ) +continue ;
Both here and in pci_restore_rebar_state(), we blindly use "nbars" derived from a value that might be ~0 because the config read failed. If we fix these, I think we should do something like this so it's obvious why we're checking:
pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
if (PCI_POSSIBLE_ERROR(ctrl))
return;
nbars = FIELD_GET(PCI_REBAR_CTRL_NBAR_MASK, ctrl);
for (i = 0; i < nbars; ...) {
...
pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
if (PCI_POSSIBLE_ERROR(ctrl))
return; bar_idx = ctrl & PCI_REBAR_CTRL_BAR_IDX;
res = pci_resource_n(pdev, bar_idx);It's true that "nbars" and "bar_idx" *could* still be invalid even if the config read succeeded, but that would be a device defect.
Quoted 6 lines
drivers/pci/iov.c +0
-0size =pci_rebar_bytes_to_size (dev->sriov->barsz[bar_idx]); ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE ; ctrl |=FIELD_PREP (PCI_VF_REBAR_CTRL_BAR_SIZE , size);Signature · 1 line
2.47.3
[+cc Rafael, Eric, Alex, Lukas, generic pci_restore_state() question]
Quoted 10 lines · Marco Nenciarini
sriov_restore_vf_rebar_state()extractsbar_idxfrom the VF Resizable BAR control register using a 3-bit field (PCI_VF_REBAR_CTRL_BAR_IDX, bits 0-2), which yields values in the range 0-7. This value is then used to index into dev->sriov->barsz[], which hasPCI_SRIOV_NUM_BARS
(6) entries.If the PCI config space read returns garbage data (e.g.
0xffffffffwhen the device is no longer accessible on the bus),bar_idxis 7, causing an out-of-bounds array access. UBSAN reports this as:
Quoted 8 lines
UBSAN: array-index-out-of-bounds in drivers/pci/iov.c:948:51 index 7 is out of range for type 'resource_size_t [6]'This was observed on an NVIDIA RTX PRO 1000 GPU (GB207GLM) that fell off the PCIe bus during a failed GC6 power state exit. The subsequent
pci_restore_state()call triggered the UBSAN splat insriov_restore_vf_rebar_state()since all config space reads returned0xffffffff.
I think all of pci_restore_state() is problematic for all devices, not just this GPU. If these config reads fail, all the previous config writes probably failed (silently) as well.
And we have this weird retry loop in pci_restore_config_dword():
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/pci.c?id=v7.0#n1766,
which was originally added by
https://git.kernel.org/linus/26f41062f28d ("PCI: check for pci bar restore completion and retry") to fix an actual problem:
On some OEM systems, pci_restore_state() is called while FLR has not yet completed. As a result, PCI BAR register restore is not successful. This fix reads back the restored value and compares it with saved value and re-tries 10 times before giving up.
This just gives me the heebie-jeebies. If we still need this retry loop, it means all the previous state restoration (PCIe, LTR, ASPM, IOV, PRI, ATS, DPC, etc.) probably failed, and we end up with a device where the BARs got restored but none of the previous stuff. That sounds like a mess.
Quoted 29 lines
Add a bounds check on
bar_idxbefore using it as an array index to prevent the out-of-bounds access.Fixes 5a8f77e24a30 ("PCI/IOV: Restore VF resizable BAR state after reset")Signed-off-by Marco Nenciarini <[email protected]>---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>drivers/pci/iov.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/iov.c +2
-0@@ -946,6 +946,8 @@ static void sriov_restore_vf_rebar_state(struct pci_dev *dev)pci_read_config_dword (dev, pos +PCI_VF_REBAR_CTRL , &ctrl); bar_idx =FIELD_GET (PCI_VF_REBAR_CTRL_BAR_IDX , ctrl); +if (bar_idx >=PCI_SRIOV_NUM_BARS ) +continue ; size =pci_rebar_bytes_to_size (dev->sriov->barsz[bar_idx]); ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE ; ctrl |=FIELD_PREP (PCI_VF_REBAR_CTRL_BAR_SIZE , size);Signature · 1 line
2.47.3
Quoted 11 lines · Bjorn Helgaas
And we have this weird retry loop in
pci_restore_config_dword():
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/pci.c?id=v7.0#n1766,
which was originally added by
https://git.kernel.org/linus/26f41062f28d ("PCI: check for pci bar restore completion and retry") to fix an actual problem:On some OEM systems,
pci_restore_state()is called while FLR has not yet completed. As a result, PCI BAR register restore is not successful. This fix reads back the restored value and compares it with saved value and re-tries 10 times before giving up.
Quoted 5 lines
This just gives me the heebie-jeebies. If we still need this retry loop, it means all the previous state restoration (PCIe, LTR, ASPM, IOV, PRI, ATS, DPC, etc.) probably failed, and we end up with a device where the BARs got restored but none of the previous stuff. That sounds like a mess.
Nowadays we wait for devices to re-appear after reset by polling the Vendor ID register, see the call to pci_dev_wait() in pcie_flr().
It seems we didn't do that back in the day when 26f41062f28d introduced the loop. The commit went into v3.4 and back then, pcie_flr() only waited for 100 msec:
https://elixir.bootlin.com/linux/v3.4/source/drivers/pci/pci.c#L3052
And indeed pci_reset_function() immediately restored config space afterwards:
https://elixir.bootlin.com/linux/v3.4/source/drivers/pci/pci.c#L3285
So I strongly suspect that the loop no longer has a valid raison d'être. Maybe remove it early in the next cycle to get linux-next coverage for 8 weeks and see if anything breaks (which I doubt)?
As to validity of cached config space state in general, see this discussion with Ilpo yesterday, in response to a regression fix I submitted:
https://lore.kernel.org/all/[email protected]/
Thanks,
Lukas
v2 cover PCI: Guard Resizable BAR restore against unreachable devices
In reply to Marco NenciariniThis series addresses Bjorn's review feedback on v1 [1].
v1 bounds-checked bar_idx before indexing dev->sriov->barsz[] in sriov_restore_vf_rebar_state(). Bjorn pointed out that the non-SRIOV sibling pci_restore_rebar_state() has the same issue, and that a PCI_POSSIBLE_ERROR(ctrl) check on the config read makes the intent of the guard more obvious than a post-hoc range check on the extracted field.
v2 therefore adopts PCI_POSSIBLE_ERROR(ctrl) after each Resizable BAR Control read in both functions, bailing out when config reads return the all-ones pattern. Patch 1 covers pci_restore_rebar_state(). Patch 2 covers sriov_restore_vf_rebar_state(), with the NVIDIA GC6 UBSAN splat as motivation.
Note that this changes behavior versus v1: on a bad read we abort the loop instead of skipping just the current BAR. This matches the structure Bjorn suggested in review and is safe because the all-ones pattern means the device is unreachable, so restoring the remaining BARs is moot.
Compile-tested on pci/next (full drivers/pci/ build). The error path cannot be exercised without reproducing the GC6 failure that killed the GPU in the original report.
The broader v1 discussion on the pci_restore_config_dword() retry loop and on save/restore behavior when the device has fallen off the bus is out of scope for this fix. Happy to tackle that separately if there is consensus.
[1] https://lore.kernel.org/all/[email protected]/
Marco Nenciarini (2):
PCI: Skip Resizable BAR restore on read error PCI/IOV: Skip VF Resizable BAR restore on read error
drivers/pci/iov.c | 6 ++++++ drivers/pci/rebar.c | 6 ++++++ 2 files changed, 12 insertions(+)
base-commit: 40286d6379aacfcc053253ef78dc78b09addffda
Signature · 1 line
2.47.3
PATCH v2 1/2 PCI: Skip Resizable BAR restore on read error
pci_restore_rebar_state() uses the Resizable BAR Control register to decide how many BARs to restore (nbars) and which BAR each iteration addresses (bar_idx).
When the device does not respond, config reads return the all-ones pattern. Both fields are 3 bits wide, so nbars and bar_idx both evaluate to 7, past the spec's valid ranges for both fields. pci_resource_n() then returns an unrelated resource slot, whose size is used to derive a nonsensical value written back to the Resizable BAR Control register.
Bail out if any Resizable BAR Control read returns the error pattern. No further BARs are touched, which is safe because a config read that returns the error pattern indicates the device is unreachable and restoration is pointless.
---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>
Cc: Rafael J. Wysocki <[email protected]>
Cc: Eric Chanudet <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Lukas Wunner <[email protected]>
drivers/pci/rebar.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/pci/rebar.c +6 -0
@@ -231,6 +231,9 @@ void pci_restore_rebar_state(struct pci_dev *pdev)return ;pci_read_config_dword (pdev, pos +PCI_REBAR_CTRL , &ctrl); +if (PCI_POSSIBLE_ERROR (ctrl)) +return ; + nbars =FIELD_GET (PCI_REBAR_CTRL_NBAR_MASK , ctrl);for (i =0 ; i < nbars; i++, pos +=8 ) {
@@ -238,6 +241,9 @@ void pci_restore_rebar_state(struct pci_dev *pdev)int bar_idx, size;pci_read_config_dword (pdev, pos +PCI_REBAR_CTRL , &ctrl); +if (PCI_POSSIBLE_ERROR (ctrl)) +return ; + bar_idx = ctrl &PCI_REBAR_CTRL_BAR_IDX ; res =pci_resource_n (pdev, bar_idx); size =pci_rebar_bytes_to_size (resource_size (res));
Signature · 1 line
2.47.3
PATCH v2 2/2 PCI/IOV: Skip VF Resizable BAR restore on read error
In reply to Marco Nenciarinisriov_restore_vf_rebar_state() uses the VF Resizable BAR Control register to decide how many VF BARs to restore (nbars) and which VF BAR each iteration addresses (bar_idx). bar_idx indexes into dev->sriov->barsz[], which has only PCI_SRIOV_NUM_BARS (6) entries.
When the device does not respond, config reads return the all-ones pattern. Both fields are 3 bits wide, so nbars and bar_idx both evaluate to 7. The barsz[] access then goes out of bounds. UBSAN reports this as:
UBSAN: array-index-out-of-bounds in drivers/pci/iov.c:948:51 index 7 is out of range for type 'resource_size_t [6]'
Observed on an NVIDIA RTX PRO 1000 GPU (GB207GLM) that fell off the PCIe bus during a failed GC6 power state exit. The subsequent pci_restore_state() invoked sriov_restore_vf_rebar_state() while config reads returned 0xffffffff, triggering the splat.
Bail out if any VF Resizable BAR Control read returns the error pattern. No further VF BARs are touched, which is safe because a config read that returns the error pattern indicates the device is unreachable and restoration is pointless. This mirrors the guard in pci_restore_rebar_state().
---
Cc: Michał Winiarski <[email protected]>
Cc: Ilpo Järvinen <[email protected]>
Cc: Rafael J. Wysocki <[email protected]>
Cc: Eric Chanudet <[email protected]>
Cc: Alex Williamson <[email protected]>
Cc: Lukas Wunner <[email protected]>
drivers/pci/iov.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/pci/iov.c +6 -0
@@ -938,12 +938,18 @@ static void sriov_restore_vf_rebar_state(struct pci_dev *dev)return ;pci_read_config_dword (dev, pos +PCI_VF_REBAR_CTRL , &ctrl); +if (PCI_POSSIBLE_ERROR (ctrl)) +return ; + nbars =FIELD_GET (PCI_VF_REBAR_CTRL_NBAR_MASK , ctrl);for (i =0 ; i < nbars; i++, pos +=8 ) {int bar_idx, size;pci_read_config_dword (dev, pos +PCI_VF_REBAR_CTRL , &ctrl); +if (PCI_POSSIBLE_ERROR (ctrl)) +return ; + bar_idx =FIELD_GET (PCI_VF_REBAR_CTRL_BAR_IDX , ctrl); size =pci_rebar_bytes_to_size (dev->sriov->barsz[bar_idx]); ctrl &= ~PCI_VF_REBAR_CTRL_BAR_SIZE ;
Signature · 1 line
2.47.3
Bjorn Helgaas
AppliedRe: [PATCH v2 0/2] PCI: Guard Resizable BAR restore against unreachable devices
In reply to Marco NenciariniQuoted 32 lines · Marco Nenciarini
This series addresses Bjorn's review feedback on v1 [1].
v1 bounds-checked
bar_idxbefore indexing dev->sriov->barsz[] insriov_restore_vf_rebar_state(). Bjorn pointed out that the non-SRIOV siblingpci_restore_rebar_state()has the same issue, and that aPCI_POSSIBLE_ERROR(ctrl)check on the config read makes the intent of the guard more obvious than a post-hoc range check on the extracted field.v2 therefore adopts
PCI_POSSIBLE_ERROR(ctrl)after each Resizable BAR Control read in both functions, bailing out when config reads return the all-ones pattern. Patch 1 coverspci_restore_rebar_state(). Patch 2 coverssriov_restore_vf_rebar_state(), with the NVIDIA GC6 UBSAN splat as motivation.Note that this changes behavior versus v1: on a bad read we abort the loop instead of skipping just the current BAR. This matches the structure Bjorn suggested in review and is safe because the all-ones pattern means the device is unreachable, so restoring the remaining BARs is moot.
Compile-tested on pci/next (full
drivers/pci/build). The error path cannot be exercised without reproducing the GC6 failure that killed the GPU in the original report.The broader v1 discussion on the
pci_restore_config_dword()retry loop and on save/restore behavior when the device has fallen off the bus is out of scope for this fix. Happy to tackle that separately if there is consensus.
Quoted 7 lines
Marco Nenciarini (2):
PCI: Skip Resizable BAR restore on read error PCI/IOV: Skip VF Resizable BAR restore on read errordrivers/pci/iov.c | 6 ++++++ drivers/pci/rebar.c | 6 ++++++ 2 files changed, 12 insertions(+)
Applied to pci/pm for v7.2, thanks! Will be rebased after v7.1-rc1.
Quoted 4 lines
base-commit:
40286d6379aacfcc053253ef78dc78b09addffdaSignature · 1 line
2.47.3