Skip to contentSkip to conversation

PCI: Do not enable pci atomics on VFs

PATCH Applied Selvin Xavier 3 participants 3 messages Sep 11, 2021 – Sep 14, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: Do not enable pci atomics on VFs

Host crashes when pci_enable_atomic_ops_to_root is called for VFs with virtual buses. The virtual buses added to SR-IOV has bus->self set to NULL and host crashes due to this.

PID: 4481 TASK: ffff89c6941b0000 CPU: 53 COMMAND: "bash"
#0 [ffff9a94817136d8] machine_kexec at ffffffffb90601a4 #1 [ffff9a9481713728] __crash_kexec at ffffffffb9190d5d #2 [ffff9a94817137f0] crash_kexec at ffffffffb9191c4d #3 [ffff9a9481713808] oops_end at ffffffffb9025cd6 #4 [ffff9a9481713828] page_fault_oops at ffffffffb906e417 #5 [ffff9a9481713888] exc_page_fault at ffffffffb9a0ad14 #6 [ffff9a94817138b0] asm_exc_page_fault at ffffffffb9c00ace

    [exception RIP: pcie_capability_read_dword+28]
    RIP: ffffffffb952fd5c  RSP: ffff9a9481713960  RFLAGS: 00010246
    RAX: 0000000000000001  RBX: ffff89c6b1096000  RCX: 0000000000000000
    RDX: ffff9a9481713990  RSI: 0000000000000024  RDI: 0000000000000000
    RBP: 0000000000000080   R8: 0000000000000008   R9: ffff89c64341a2f8
    R10: 0000000000000002  R11: 0000000000000000  R12: ffff89c648bab000
    R13: 0000000000000000  R14: 0000000000000000  R15: ffff89c648bab0c8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff9a9481713988] pci_enable_atomic_ops_to_root at ffffffffb95359a6
 #8 [ffff9a94817139c0] bnxt_qplib_determine_atomics at ffffffffc08c1a33 [bnxt_re]
 #9 [ffff9a94817139d0] bnxt_re_dev_init at ffffffffc08ba2d1 [bnxt_re]
    RIP: 00007f450602f648  RSP: 00007ffe880869e8  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007f450602f648
    RDX: 0000000000000002  RSI: 0000555c566c4a60  RDI: 0000000000000001
    RBP: 0000555c566c4a60   R8: 000000000000000a   R9: 00007f45060c2580
    R10: 000000000000000a  R11: 0000000000000246  R12: 00007f45063026e0
    R13: 0000000000000002  R14: 00007f45062fd880  R15: 0000000000000002
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b

AtomicOp Requester Enable bit in the Device Control 2 register is reserved for VFs and drivers shouldn't enable it for VFs. Adding a check to return EINVAL if pci_enable_atomic_ops_to_root is called with VF pci device.

Fixes 35f5ace5dea4 ("RDMA/bnxt_re: Enable global atomic ops if platform supports")
Fixes 430a23689dea ("PCI: Add pci_enable_atomic_ops_to_root()")
---
 drivers/pci/pci.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/pci/pci.c +8 -0
index aacf575..d968a36 100644
@@ -3702,6 +3702,14 @@ int pci_enable_atomic_ops_to_root(struct pci_dev *dev, u32 cap_mask)
 	struct pci_dev *bridge;
 	u32 cap, ctl2;
 
+	/*
+	 * As per PCIe r5.0, sec 9.3.5.10, the AtomicOp Requester Enable
+	 * bit in the Device Control 2 register is reserved in VFs and the PF
+	 * value applies to all associated VFs. Return -EINVAL if called for VFs.
+	 */
+	if (dev->is_virtfn)
+		return -EINVAL;
+
 	if (!pci_is_pcie(dev))
 		return -EINVAL;
 
Signature · 1 line
2.5.5
Reviewed-by on v1 1/1
#1 lore raw
Quoted 3 lines · Selvin Xavier

Host crashes when pci_enable_atomic_ops_to_root is called for VFs with virtual buses. The virtual buses added to SR-IOV has bus->self set to NULL and host crashes due to this.

^^ I _hate_ to say this, but the extra space isn't ideal. Not
sure if the maintainers will want to hold-up the submission for this (or if they can fix while committing), but something to look out for next time.

Quoted 28 lines

PID: 4481 TASK: ffff89c6941b0000 CPU: 53 COMMAND: "bash"
#0 [ffff9a94817136d8] machine_kexec at ffffffffb90601a4 #1 [ffff9a9481713728] __crash_kexec at ffffffffb9190d5d #2 [ffff9a94817137f0] crash_kexec at ffffffffb9191c4d #3 [ffff9a9481713808] oops_end at ffffffffb9025cd6 #4 [ffff9a9481713828] page_fault_oops at ffffffffb906e417 #5 [ffff9a9481713888] exc_page_fault at ffffffffb9a0ad14 #6 [ffff9a94817138b0] asm_exc_page_fault at ffffffffb9c00ace

    [exception RIP: pcie_capability_read_dword+28]
    RIP: ffffffffb952fd5c  RSP: ffff9a9481713960  RFLAGS: 00010246
    RAX: 0000000000000001  RBX: ffff89c6b1096000  RCX: 0000000000000000
    RDX: ffff9a9481713990  RSI: 0000000000000024  RDI: 0000000000000000
    RBP: 0000000000000080   R8: 0000000000000008   R9: ffff89c64341a2f8
    R10: 0000000000000002  R11: 0000000000000000  R12: ffff89c648bab000
    R13: 0000000000000000  R14: 0000000000000000  R15: ffff89c648bab0c8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff9a9481713988] pci_enable_atomic_ops_to_root at ffffffffb95359a6
 #8 [ffff9a94817139c0] bnxt_qplib_determine_atomics at ffffffffc08c1a33 [bnxt_re]
 #9 [ffff9a94817139d0] bnxt_re_dev_init at ffffffffc08ba2d1 [bnxt_re]
    RIP: 00007f450602f648  RSP: 00007ffe880869e8  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007f450602f648
    RDX: 0000000000000002  RSI: 0000555c566c4a60  RDI: 0000000000000001
    RBP: 0000555c566c4a60   R8: 000000000000000a   R9: 00007f45060c2580
    R10: 000000000000000a  R11: 0000000000000246  R12: 00007f45063026e0
    R13: 0000000000000002  R14: 00007f45062fd880  R15: 0000000000000002
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
Quoted 8 lines

AtomicOp Requester Enable bit in the Device Control 2 register is reserved for VFs and drivers shouldn't enable it for VFs. Adding a check to return EINVAL if pci_enable_atomic_ops_to_root is called with VF pci device.

Fixes 35f5ace5dea4 ("RDMA/bnxt_re: Enable global atomic ops if platform supports")
Fixes 430a23689dea ("PCI: Add pci_enable_atomic_ops_to_root()")

Thanks for this, Selvin. Technically this looks like a good fix.

Quoted 26 lines
---
 drivers/pci/pci.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/pci/pci.c +8 -0
index aacf575..d968a36 100644
@@ -3702,6 +3702,14 @@ int pci_enable_atomic_ops_to_root(struct pci_dev *dev, u32 cap_mask)
 	struct pci_dev *bridge;
 	u32 cap, ctl2;
 
+	/*
+	 * As per PCIe r5.0, sec 9.3.5.10, the AtomicOp Requester Enable
+	 * bit in the Device Control 2 register is reserved in VFs and the PF
+	 * value applies to all associated VFs. Return -EINVAL if called for VFs.
+	 */
+	if (dev->is_virtfn)
+		return -EINVAL;
+
 	if (!pci_is_pcie(dev))
 		return -EINVAL;
 
Signature · 1 line
2.5.5
#2 lore raw
In reply to Selvin Xavier
Quoted 31 lines · Selvin Xavier

Host crashes when pci_enable_atomic_ops_to_root is called for VFs with virtual buses. The virtual buses added to SR-IOV has bus->self set to NULL and host crashes due to this.

PID: 4481 TASK: ffff89c6941b0000 CPU: 53 COMMAND: "bash"
#0 [ffff9a94817136d8] machine_kexec at ffffffffb90601a4 #1 [ffff9a9481713728] __crash_kexec at ffffffffb9190d5d #2 [ffff9a94817137f0] crash_kexec at ffffffffb9191c4d #3 [ffff9a9481713808] oops_end at ffffffffb9025cd6 #4 [ffff9a9481713828] page_fault_oops at ffffffffb906e417 #5 [ffff9a9481713888] exc_page_fault at ffffffffb9a0ad14 #6 [ffff9a94817138b0] asm_exc_page_fault at ffffffffb9c00ace

    [exception RIP: pcie_capability_read_dword+28]
    RIP: ffffffffb952fd5c  RSP: ffff9a9481713960  RFLAGS: 00010246
    RAX: 0000000000000001  RBX: ffff89c6b1096000  RCX: 0000000000000000
    RDX: ffff9a9481713990  RSI: 0000000000000024  RDI: 0000000000000000
    RBP: 0000000000000080   R8: 0000000000000008   R9: ffff89c64341a2f8
    R10: 0000000000000002  R11: 0000000000000000  R12: ffff89c648bab000
    R13: 0000000000000000  R14: 0000000000000000  R15: ffff89c648bab0c8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff9a9481713988] pci_enable_atomic_ops_to_root at ffffffffb95359a6
 #8 [ffff9a94817139c0] bnxt_qplib_determine_atomics at ffffffffc08c1a33 [bnxt_re]
 #9 [ffff9a94817139d0] bnxt_re_dev_init at ffffffffc08ba2d1 [bnxt_re]
    RIP: 00007f450602f648  RSP: 00007ffe880869e8  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007f450602f648
    RDX: 0000000000000002  RSI: 0000555c566c4a60  RDI: 0000000000000001
    RBP: 0000555c566c4a60   R8: 000000000000000a   R9: 00007f45060c2580
    R10: 000000000000000a  R11: 0000000000000246  R12: 00007f45063026e0
    R13: 0000000000000002  R14: 00007f45062fd880  R15: 0000000000000002
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
Quoted 8 lines

AtomicOp Requester Enable bit in the Device Control 2 register is reserved for VFs and drivers shouldn't enable it for VFs. Adding a check to return EINVAL if pci_enable_atomic_ops_to_root is called with VF pci device.

Fixes 35f5ace5dea4 ("RDMA/bnxt_re: Enable global atomic ops if platform supports")
Fixes 430a23689dea ("PCI: Add pci_enable_atomic_ops_to_root()")

Applied to pci/enumeration for v5.16, thanks!

I fixed the extra space and rewrapped the code comment so it fits in 80 columns.

Quoted 26 lines
---
 drivers/pci/pci.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/pci/pci.c +8 -0
index aacf575..d968a36 100644
@@ -3702,6 +3702,14 @@ int pci_enable_atomic_ops_to_root(struct pci_dev *dev, u32 cap_mask)
 	struct pci_dev *bridge;
 	u32 cap, ctl2;
 
+	/*
+	 * As per PCIe r5.0, sec 9.3.5.10, the AtomicOp Requester Enable
+	 * bit in the Device Control 2 register is reserved in VFs and the PF
+	 * value applies to all associated VFs. Return -EINVAL if called for VFs.
+	 */
+	if (dev->is_virtfn)
+		return -EINVAL;
+
 	if (!pci_is_pcie(dev))
 		return -EINVAL;
 
Signature · 1 line
2.5.5