Skip to contentSkip to conversation

VMD firmware first

PATCH New Jon Derrick 3 participants 5 messages Aug 31, 2018 – Sep 4, 2018

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 VMD firmware first

Hi Lorenzo, Keith,

I was hoping to wait for the host_bridge->add_device extensions but had to expedite this. Could we see if we can get this into 4.19?

Jon Derrick (1):
PCI/VMD: Set up firmware first if capable

 arch/x86/pci/common.c        | 17 ++++++++++++++++-
 drivers/pci/controller/vmd.c | 25 ++++++++++++++++++++++++-
 2 files changed, 40 insertions(+), 2 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/1 PCI/VMD: Set up firmware first if capable

Some VMD devices will want to use firmware first error-handling on the entire domain. This is detected by the BIOS setting the VMD endpoint's interface to 0x1.

Detect this condition and propogate it to the entire domain.

---
 arch/x86/pci/common.c        | 17 ++++++++++++++++-
 drivers/pci/controller/vmd.c | 25 ++++++++++++++++++++++++-
 2 files changed, 40 insertions(+), 2 deletions(-)
arch/x86/pci/common.c +16 -1
index d4ec117..f07f2e4 100644
@@ -663,8 +663,23 @@ static void set_dma_domain_ops(struct pci_dev *pdev) {}
 
 static void set_dev_domain_options(struct pci_dev *pdev)
 {
-	if (is_vmd(pdev->bus))
+	if (is_vmd(pdev->bus)) {
+		struct pci_host_bridge *hb;
+		struct pci_dev *vmd;
+
 		pdev->hotplug_user_indicators = 1;
+
+		/*
+		 * The VMD endpoint is not PCIe, so will fail
+		 * pcie_aer_get_firmware_first(). Set and get the raw member
+		 * instead.
+		 */
+		hb = pci_find_host_bridge(pdev->bus);
+		vmd = to_pci_dev(hb->dev.parent);
+
+		pdev->__aer_firmware_first = vmd->__aer_firmware_first;
+		pdev->__aer_firmware_first_valid = vmd->__aer_firmware_first_valid;
+	}
 }
 
 int pcibios_add_device(struct pci_dev *dev)
drivers/pci/controller/vmd.c +24 -1
index fd2dbd7..74a1a04 100644
@@ -44,6 +44,11 @@ enum vmd_features {
 	 * bus numbering
 	 */
 	VMD_FEAT_HAS_BUS_RESTRICTIONS	= (1 << 1),
+
+	/*
+	 * Device may request firmware first error-handling on the domain
+	 */
+	VMD_FEAT_HAS_FIRMWARE_FIRST	= (1 << 2),
 };
 
 /*
@@ -633,6 +638,23 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 			busn_start = 128;
 	}
 
+	/*
+	 * Certain VMD devices may request firmware first error-handling
+	 * support on the domain. These domains are virtual and not described
+	 * by ACPI, so we must set it explicitly. This sets firmware first on
+	 * the endpoint and has a corresponding domain setting in
+	 * arch/x86/pci/common.c
+	 */
+	if (features & VMD_FEAT_HAS_FIRMWARE_FIRST) {
+		u8 interface;
+
+		pci_read_config_byte(vmd->dev, PCI_CLASS_PROG, &interface);
+		if (interface == 0x1) {
+			vmd->dev->__aer_firmware_first = 1;
+			vmd->dev->__aer_firmware_first_valid = 1;
+		}
+	}
+
 	res = &vmd->dev->resource[VMD_CFGBAR];
 	vmd->resources[0] = (struct resource) {
 		.name  = "VMD CFGBAR",
@@ -860,7 +882,8 @@ static int vmd_resume(struct device *dev)
 	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_201D),},
 	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_28C0),
 		.driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW |
-				VMD_FEAT_HAS_BUS_RESTRICTIONS,},
+				VMD_FEAT_HAS_BUS_RESTRICTIONS |
+				VMD_FEAT_HAS_FIRMWARE_FIRST,},
 	{0,}
 };
 MODULE_DEVICE_TABLE(pci, vmd_ids);
Signature · 1 line
1.8.3.1
Reviewed-by on v1 1/1
#2 lore raw

Re: [PATCH] PCI/VMD: Set up firmware first if capable

Quoted 7 lines · Jon Derrick

Some VMD devices will want to use firmware first error-handling on the entire domain. This is detected by the BIOS setting the VMD endpoint's interface to 0x1.

Detect this condition and propogate it to the entire domain.

Just curious, do you know what happens if you encounter a fatal error in a VMD domain with FF enabled? Will this create generic hw error source event for the OS, and if so, does the error record point to the VMD device as the source, or the device within the domain that actually sent the ERR_FATAL?

The implementation looks fine to me.

#3 lore raw

Re: [PATCH] PCI/VMD: Set up firmware first if capable

In reply to Jon Derrick
Quoted 3 lines · Jon Derrick

Some VMD devices will want to use firmware first error-handling on the entire domain. This is detected by the BIOS setting the VMD endpoint's interface to 0x1.

I assume there's some spec that codifies this "programming interface 0x1 means firmware-first". Can you reference that section of the spec? Even if it's not public, it will help people who maintain this in the future.

I also have questions along the lines of Keith's: how are errors reported in the non-firmware-first case, and in the firmware-first case, how are the errors passed on to the OS?

Quoted 59 lines

Detect this condition and propogate it to the entire domain.

---
 arch/x86/pci/common.c        | 17 ++++++++++++++++-
 drivers/pci/controller/vmd.c | 25 ++++++++++++++++++++++++-
 2 files changed, 40 insertions(+), 2 deletions(-)
arch/x86/pci/common.c +16 -1
index d4ec117..f07f2e4 100644
@@ -663,8 +663,23 @@ static void set_dma_domain_ops(struct pci_dev *pdev) {}
 
 static void set_dev_domain_options(struct pci_dev *pdev)
 {
-	if (is_vmd(pdev->bus))
+	if (is_vmd(pdev->bus)) {
+		struct pci_host_bridge *hb;
+		struct pci_dev *vmd;
+
 		pdev->hotplug_user_indicators = 1;
+
+		/*
+		 * The VMD endpoint is not PCIe, so will fail
+		 * pcie_aer_get_firmware_first(). Set and get the raw member
+		 * instead.
+		 */
+		hb = pci_find_host_bridge(pdev->bus);
+		vmd = to_pci_dev(hb->dev.parent);
+
+		pdev->__aer_firmware_first = vmd->__aer_firmware_first;
+		pdev->__aer_firmware_first_valid = vmd->__aer_firmware_first_valid;
+	}
 }
 
 int pcibios_add_device(struct pci_dev *dev)
drivers/pci/controller/vmd.c +7 -0
index fd2dbd7..74a1a04 100644
@@ -44,6 +44,11 @@ enum vmd_features {
 	 * bus numbering
 	 */
 	VMD_FEAT_HAS_BUS_RESTRICTIONS	= (1 << 1),
+
+	/*
+	 * Device may request firmware first error-handling on the domain
+	 */
+	VMD_FEAT_HAS_FIRMWARE_FIRST	= (1 << 2),
 };
 
 /*
@@ -633,6 +638,23 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 			busn_start = 128;
 	}
 
+	/*
+	 * Certain VMD devices may request firmware first error-handling

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 652

Quoted 10 lines
drivers/pci/controller/vmd.c +10 -0
+	 * support on the domain. These domains are virtual and not described
+	 * by ACPI, so we must set it explicitly. This sets firmware first on
+	 * the endpoint and has a corresponding domain setting in
+	 * arch/x86/pci/common.c
+	 */
+	if (features & VMD_FEAT_HAS_FIRMWARE_FIRST) {
+		u8 interface;
+
+		pci_read_config_byte(vmd->dev, PCI_CLASS_PROG, &interface);
+		if (interface == 0x1) {

This test of the programming interface should have a spec reference so it doesn't look magical.

Quoted 9 lines
drivers/pci/controller/vmd.c +5 -0
+			vmd->dev->__aer_firmware_first = 1;
+			vmd->dev->__aer_firmware_first_valid = 1;
+		}
+	}
+
 	res = &vmd->dev->resource[VMD_CFGBAR];
 	vmd->resources[0] = (struct resource) {
 		.name  = "VMD CFGBAR",

drivers/pci/controller/vmd.c static int vmd_resume(struct device *dev) · line 886

Quoted 6 lines
drivers/pci/controller/vmd.c +2 -1
@@ -860,7 +882,8 @@ static int vmd_resume(struct device *dev)
 	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_201D),},
 	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_28C0),
 		.driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW |
-				VMD_FEAT_HAS_BUS_RESTRICTIONS,},
+				VMD_FEAT_HAS_BUS_RESTRICTIONS |
+				VMD_FEAT_HAS_FIRMWARE_FIRST,},

Why is this connected to specific hardware versions? The non-VMD firmware-first functionality is determined by firmware, independent of any particular PCI device.

If there's a spec that says "programming interface 0x1 means something other than firmware-first" on some devices, you could cite it here. If early devices are an exception and programming interface will have a consistent meaning in the future, you might be able to invert this so you only have to mark the early devices instead of every new version.

Quoted 6 lines
drivers/pci/controller/vmd.c +0 -0
 	{0,}
 };
 MODULE_DEVICE_TABLE(pci, vmd_ids);
Signature · 1 line
1.8.3.1
#4 lore raw

Re: [PATCH] PCI/VMD: Set up firmware first if capable

Thanks for the comments.
I'll try to get those answers

Quoted 138 lines · Bjorn Helgaas
Quoted 5 lines · Jon Derrick

Some VMD devices will want to use firmware first error-handling on the
entire domain. This is detected by the BIOS setting the VMD endpoint's
interface to 0x1.

I assume there's some spec that codifies this "programming interface 0x1 means firmware-first". Can you reference that section of the spec? Even if it's not public, it will help people who maintain this in the future.

I also have questions along the lines of Keith's: how are errors reported in the non-firmware-first case, and in the firmware-first case, how are the errors passed on to the OS?

Quoted 79 lines

Detect this condition and propogate it to the entire domain.

---
 arch/x86/pci/common.c        | 17 ++++++++++++++++-
 drivers/pci/controller/vmd.c | 25 ++++++++++++++++++++++++-
 2 files changed, 40 insertions(+), 2 deletions(-)
arch/x86/pci/common.c +8 -1
index d4ec117..f07f2e4 100644
@@ -663,8 +663,23 @@ static void set_dma_domain_ops(struct pci_dev
 *pdev) {}
 
 static void set_dev_domain_options(struct pci_dev *pdev)
 {
-	if (is_vmd(pdev->bus))
+	if (is_vmd(pdev->bus)) {
+		struct pci_host_bridge *hb;
+		struct pci_dev *vmd;
+
 		pdev->hotplug_user_indicators = 1;
+
+		/*
+		 * The VMD endpoint is not PCIe, so will fail
+		 * pcie_aer_get_firmware_first(). Set and get the

raw member

arch/x86/pci/common.c +6 -0
+		 * instead.
+		 */
+		hb = pci_find_host_bridge(pdev->bus);
+		vmd = to_pci_dev(hb->dev.parent);
+
+		pdev->__aer_firmware_first = vmd-
Quoted 1 line

__aer_firmware_first;

arch/x86/pci/common.c +1 -0
+		pdev->__aer_firmware_first_valid = vmd-
Quoted 1 line

__aer_firmware_first_valid;

arch/x86/pci/common.c +1 -0
+	}
 }
 
 int pcibios_add_device(struct pci_dev *dev)
drivers/pci/controller/vmd.c +3 -0
index fd2dbd7..74a1a04 100644
@@ -44,6 +44,11 @@ enum vmd_features {
 	 * bus numbering
 	 */
 	VMD_FEAT_HAS_BUS_RESTRICTIONS	= (1 << 1),
+
+	/*
+	 * Device may request firmware first error-handling on the

domain

drivers/pci/controller/vmd.c +4 -0
+	 */
+	VMD_FEAT_HAS_FIRMWARE_FIRST	= (1 << 2),
 };
 
 /*
@@ -633,6 +638,23 @@ static int vmd_enable_domain(struct vmd_dev
 *vmd, unsigned long features)
 			busn_start = 128;
 	}
 
+	/*
+	 * Certain VMD devices may request firmware first error-

handling

drivers/pci/controller/vmd.c +1 -0
+	 * support on the domain. These domains are virtual and

not described

drivers/pci/controller/vmd.c +1 -0
+	 * by ACPI, so we must set it explicitly. This sets

firmware first on

drivers/pci/controller/vmd.c +7 -0
+	 * the endpoint and has a corresponding domain setting in
+	 * arch/x86/pci/common.c
+	 */
+	if (features & VMD_FEAT_HAS_FIRMWARE_FIRST) {
+		u8 interface;
+
+		pci_read_config_byte(vmd->dev, PCI_CLASS_PROG,

&interface);

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 652

Quoted 1 line
drivers/pci/controller/vmd.c +1 -0
+		if (interface == 0x1) {

This test of the programming interface should have a spec reference so
it doesn't look magical.

Quoted 9 lines
drivers/pci/controller/vmd.c +5 -0
+			vmd->dev->__aer_firmware_first = 1;
+			vmd->dev->__aer_firmware_first_valid = 1;
+		}
+	}
+
 	res = &vmd->dev->resource[VMD_CFGBAR];
 	vmd->resources[0] = (struct resource) {
 		.name  = "VMD CFGBAR",

drivers/pci/controller/vmd.c static int vmd_resume(struct device *dev) · line 888

Quoted 8 lines
drivers/pci/controller/vmd.c +2 -1
@@ -860,7 +882,8 @@ static int vmd_resume(struct device *dev)
 	{PCI_DEVICE(PCI_VENDOR_ID_INTEL,
 PCI_DEVICE_ID_INTEL_VMD_201D),},
 	{PCI_DEVICE(PCI_VENDOR_ID_INTEL,
 PCI_DEVICE_ID_INTEL_VMD_28C0),
 		.driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW |
-				VMD_FEAT_HAS_BUS_RESTRICTIONS,},
+				VMD_FEAT_HAS_BUS_RESTRICTIONS |
+				VMD_FEAT_HAS_FIRMWARE_FIRST,},

Why is this connected to specific hardware versions? The non-VMD firmware-first functionality is determined by firmware, independent of
any particular PCI device.

If there's a spec that says "programming interface 0x1 means something
other than firmware-first" on some devices, you could cite it here. If early devices are an exception and programming interface will have a consistent meaning in the future, you might be able to invert this so you only have to mark the early devices instead of every new version.

Quoted 6 lines
drivers/pci/controller/vmd.c +0 -0
 	{0,}

};
MODULE_DEVICE_TABLE(pci, vmd_ids);

Signature · 1 line
1.8.3.1