PCI: fix system hang issue of Marvell SATA host controller
PATCH 2/2 PCI: fix system hang issue of Marvell SATA host controller
From: Xiangliang Yu <[email protected]>
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
--- drivers/pci/quirks.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-)
drivers/pci/quirks.c +15 -0
@@ -44,6 +44,21 @@ static void quirk_mmio_always_on(struct pci_dev *dev)DECLARE_PCI_FIXUP_CLASS_EARLY (PCI_ANY_ID ,PCI_ANY_ID ,PCI_CLASS_BRIDGE_HOST ,8 , quirk_mmio_always_on); +/* The BAR0 ~ BAR4 of Marvell 9125 device can't be accessed +* by IO resource file, and need to skip the files +*/ +static void quirk_marvell_mask_bar (struct pci_dev *dev) +{ +int i; + +for (i =0 ; i <5 ; i++) +if (dev->resource[i].start) + dev->resource[i].start = + dev->resource[i].end =0 ; +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_MARVELL_EXT ,0x9125 , + quirk_marvell_mask_bar); +/* The Mellanox Tavor device gives false positive parity errors * Mark this device with a broken_parity_status, to allow * PCI scanning code to "skip" this now blacklisted device.
Signature · 1 line
1.7.5.4
Bjorn Helgaas
maintainerQuoted 4 lines · On Thu, Mar 7, 2013 at 7:29 AM, <[email protected]> wrote:
From: Xiangliang Yu <[email protected]>
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
This needs more explanation. We've already read the BARs by the time header quirks are run, so apparently it's not just the mere act of accessing a BAR that causes a hang.
We need to know exactly what's going on here. For example, do BARs 0-4 exist? Does the device decode accesses to the regions described by the BARs? The PCI core has to know what resources the device uses, so if the device decodes accesses, we can't just throw away the start/end information.
Quoted 33 lines
--- drivers/pci/quirks.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-)drivers/pci/quirks.c +15
-1@@ -44,6 +44,21 @@ static void quirk_mmio_always_on(struct pci_dev *dev)DECLARE_PCI_FIXUP_CLASS_EARLY (PCI_ANY_ID ,PCI_ANY_ID ,PCI_CLASS_BRIDGE_HOST ,8 , quirk_mmio_always_on); +/* The BAR0 ~ BAR4 of Marvell 9125 device can't be accessed +* by IO resource file, and need to skip the files +*/ +static void quirk_marvell_mask_bar (struct pci_dev *dev) +{ +int i; + +for (i =0 ; i <5 ; i++) +if (dev->resource[i].start) + dev->resource[i].start = + dev->resource[i].end =0 ; +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_MARVELL_EXT ,0x9125 , + quirk_marvell_mask_bar); +/* The Mellanox Tavor device gives false positive parity errors * Mark this device with a broken_parity_status, to allow * PCI scanning code to "skip" this now blacklisted device. -- 1.7.5.4
Hi, Bjorn
Quoted 7 lines
Quoted 2 lines
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
This needs more explanation. We've already read the BARs by the time header quirks are run, so apparently it's not just the mere act of accessing a BAR that causes a hang.
Quoted 5 lines
We need to know exactly what's going on here. For example, do BARs 0-4 exist? Does the device decode accesses to the regions described by the BARs? The PCI core has to know what resources the device uses, so if the device decodes accesses, we can't just throw away the start/end information.
The BARs 0-4 is exist and the PCI device is enable IO space, but user access the regions file by udevadm command with info parameter, the system will hang. Like this: udevadmin info --attribut-walk --path=/sys/device/pci-device/000:*.
Because the device is just AHCI host controller, don't need the BAR0 ~ 4 region file.
Is my explanation ok for the patch?
Quoted 35 lines
drivers/pci/quirks.c static void quirk_mmio_always_on(struct pci_dev *dev) · line 45
Quoted 12 lines
--- drivers/pci/quirks.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-)drivers/pci/quirks.c +0
-0@@ -44,6 +44,21 @@ static void quirk_mmio_always_on(struct pci_dev *dev) DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_ANY_ID, PCI_ANY_ID, PCI_CLASS_BRIDGE_HOST, 8,
quirk_mmio_always_on);Quoted 22 lines
drivers/pci/quirks.c +15
-1+/* The BAR0 ~ BAR4 of Marvell 9125 device can't be accessed +* by IO resource file, and need to skip the files +*/ +static void quirk_marvell_mask_bar (struct pci_dev *dev) +{ +int i; + +for (i =0 ; i <5 ; i++) +if (dev->resource[i].start) + dev->resource[i].start = + dev->resource[i].end =0 ; +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_MARVELL_EXT ,0x9125 , + quirk_marvell_mask_bar); +/* The Mellanox Tavor device gives false positive parity errors * Mark this device with a broken_parity_status, to allow * PCI scanning code to "skip" this now blacklisted device. -- 1.7.5.4
Bjorn Helgaas
maintainerQuoted 14 lines · Xiangliang Yu
Hi, Bjorn
Quoted 7 lines
Quoted 2 lines
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
This needs more explanation. We've already read the BARs by the time header quirks are run, so apparently it's not just the mere act of accessing a BAR that causes a hang.
Quoted 5 lines
We need to know exactly what's going on here. For example, do BARs 0-4 exist? Does the device decode accesses to the regions described by the BARs? The PCI core has to know what resources the device uses, so if the device decodes accesses, we can't just throw away the start/end information.
Quoted 4 lines
The BARs 0-4 is exist and the PCI device is enable IO space, but user access the regions file by udevadm command with info parameter, the system will hang. Like this: udevadmin info --attribut-walk --path=/sys/device/pci-device/000:*.
Because the device is just AHCI host controller, don't need the BAR0 ~ 4 region file.
Is my explanation ok for the patch?
No, I still don't know what causes the hang; I only know that udevadm can trigger it. I don't want to just paper over the problem until we know what the root cause is.
Does "lspci -H1 -vv" also cause a hang? What about "setpci -s<dev> BASE_ADDRESS_0"? "setpci -H1 -s<dev> BASE_ADDRESS_0"?
Quoted 35 lines
Quoted 35 lines
drivers/pci/quirks.c static void quirk_mmio_always_on(struct pci_dev *dev) · line 45
Quoted 12 lines
--- drivers/pci/quirks.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-)drivers/pci/quirks.c +0
-0@@ -44,6 +44,21 @@ static void quirk_mmio_always_on(struct pci_dev *dev) DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_ANY_ID, PCI_ANY_ID, PCI_CLASS_BRIDGE_HOST, 8,
quirk_mmio_always_on);Quoted 22 lines
drivers/pci/quirks.c +15
-1+/* The BAR0 ~ BAR4 of Marvell 9125 device can't be accessed +* by IO resource file, and need to skip the files +*/ +static void quirk_marvell_mask_bar (struct pci_dev *dev) +{ +int i; + +for (i =0 ; i <5 ; i++) +if (dev->resource[i].start) + dev->resource[i].start = + dev->resource[i].end =0 ; +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_MARVELL_EXT ,0x9125 , + quirk_marvell_mask_bar); +/* The Mellanox Tavor device gives false positive parity errors * Mark this device with a broken_parity_status, to allow * PCI scanning code to "skip" this now blacklisted device. -- 1.7.5.4
Hi, Bjorn
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
This needs more explanation. We've already read the BARs by the time header quirks are run, so apparently it's not just the mere act of accessing a BAR that causes a hang.
Quoted 6 lines
Quoted 5 lines
We need to know exactly what's going on here. For example, do BARs 0-4 exist? Does the device decode accesses to the regions described by the BARs? The PCI core has to know what resources the device uses, so if the device decodes accesses, we can't just throw away the start/end information.
The BARs 0-4 is exist and the PCI device is enable IO space, but user access
the regions file by udevadm command with info parameter, the system will hang.
Quoted 1 line
Like this: udevadmin info --attribut-walk
--path=/sys/device/pci-device/000:*.
Quoted 1 line
Because the device is just AHCI host controller, don't need the BAR0 ~ 4 region
file.
Quoted 8 lines
Quoted 1 line
Is my explanation ok for the patch?
No, I still don't know what causes the hang; I only know that udevadm can trigger it. I don't want to just paper over the problem until we know what the root cause is.
Does "lspci -H1 -vv" also cause a hang? What about "setpci -s<dev>
BASE_ADDRESS_0"? "setpci -H1 -s<dev>BASE_ADDRESS_0"?
The commands are ok because the commands can't find the device after accessing IO port.
The root cause is that accessing of IO port will make the chip go bad. So, the point of the patch is don't export capability of the IO accessing.
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
drivers/pci/quirks.c static void quirk_mmio_always_on(struct pci_dev *dev) · line 45
Quoted 13 lines
--- drivers/pci/quirks.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-)drivers/pci/quirks.c +0
-0@@ -44,6 +44,21 @@ static void quirk_mmio_always_on(struct pci_dev *dev) DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_ANY_ID, PCI_ANY_ID, PCI_CLASS_BRIDGE_HOST, 8,
quirk_mmio_always_on);Quoted 22 lines
drivers/pci/quirks.c +15
-1+/* The BAR0 ~ BAR4 of Marvell 9125 device can't be accessed +* by IO resource file, and need to skip the files +*/ +static void quirk_marvell_mask_bar (struct pci_dev *dev) +{ +int i; + +for (i =0 ; i <5 ; i++) +if (dev->resource[i].start) + dev->resource[i].start = + dev->resource[i].end =0 ; +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_MARVELL_EXT ,0x9125 , + quirk_marvell_mask_bar); +/* The Mellanox Tavor device gives false positive parity errors * Mark this device with a broken_parity_status, to allow * PCI scanning code to "skip" this now blacklisted device. -- 1.7.5.4
Bjorn Helgaas
maintainerQuoted 22 lines · Xiangliang Yu
Hi, Bjorn
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
This needs more explanation. We've already read the BARs by the time header quirks are run, so apparently it's not just the mere act of accessing a BAR that causes a hang.
Quoted 6 lines
Quoted 5 lines
We need to know exactly what's going on here. For example, do BARs 0-4 exist? Does the device decode accesses to the regions described by the BARs? The PCI core has to know what resources the device uses, so if the device decodes accesses, we can't just throw away the start/end information.
The BARs 0-4 is exist and the PCI device is enable IO space, but user access
the regions file by udevadm command with info parameter, the system will hang.
Quoted 1 line
Like this: udevadmin info --attribut-walk
--path=/sys/device/pci-device/000:*.
Quoted 1 line
Because the device is just AHCI host controller, don't need the BAR0 ~ 4 region
file.
Quoted 1 line
Is my explanation ok for the patch?
Quoted 8 lines
Quoted 6 lines
No, I still don't know what causes the hang; I only know that udevadm can trigger it. I don't want to just paper over the problem until we know what the root cause is.
Does "lspci -H1 -vv" also cause a hang? What about "setpci -s<dev>
BASE_ADDRESS_0"? "setpci -H1 -s<dev>BASE_ADDRESS_0"?The commands are ok because the commands can't find the device after accessing IO port.
The root cause is that accessing of IO port will make the chip go bad. So, the point of the patch is don't export capability of the IO accessing.
Ah, so the problem is not with accessing the BAR in config space. The problem is with accessing the I/O port space mapped by the BAR. Is that right?
Does "udevadm info --attribute-walk" really access the device address space mapped by the BARs? That seems surprising to me, and I don't see any indication of it when I try it on an AHCI device on my system:
# udevadm info --attribute-walk --path=/sys/devices/pci0000:00/0000:00:1f.2
Udevadm info starts with the device specified by the devpath and then walks up the chain of parent devices. It prints for every device found, all possible attributes in the udev rules key format. A rule to match, can be composed by the attributes of the device and the attributes from one single parent device.
looking at device '/devices/pci0000:00/0000:00:1f.2':
KERNEL=="0000:00:1f.2"
SUBSYSTEM=="pci"
DRIVER=="ahci"
ATTR{irq}=="40"
ATTR{subsystem_vendor}=="0x17aa"
ATTR{broken_parity_status}=="0"
ATTR{class}=="0x010601"
ATTR{consistent_dma_mask_bits}=="64"
ATTR{dma_mask_bits}=="64"
ATTR{local_cpus}=="00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000000f"
ATTR{device}=="0x3b2f"
ATTR{enable}=="1"
ATTR{msi_bus}==""
ATTR{local_cpulist}=="0-3"
ATTR{vendor}=="0x8086"
ATTR{subsystem_device}=="0x2168"
ATTR{numa_node}=="-1"looking at parent device '/devices/pci0000:00':
KERNELS=="pci0000:00"
SUBSYSTEMS==""
DRIVERS==""Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
drivers/pci/quirks.c static void quirk_mmio_always_on(struct pci_dev *dev) · line 45
Quoted 11 lines
--- drivers/pci/quirks.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-)drivers/pci/quirks.c +0
-0@@ -44,6 +44,21 @@ static void quirk_mmio_always_on(struct pci_dev *dev) DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_ANY_ID, PCI_ANY_ID, PCI_CLASS_BRIDGE_HOST, 8,
quirk_mmio_always_on);Quoted 22 lines
drivers/pci/quirks.c +15
-1+/* The BAR0 ~ BAR4 of Marvell 9125 device can't be accessed +* by IO resource file, and need to skip the files +*/ +static void quirk_marvell_mask_bar (struct pci_dev *dev) +{ +int i; + +for (i =0 ; i <5 ; i++) +if (dev->resource[i].start) + dev->resource[i].start = + dev->resource[i].end =0 ; +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_MARVELL_EXT ,0x9125 , + quirk_marvell_mask_bar); +/* The Mellanox Tavor device gives false positive parity errors * Mark this device with a broken_parity_status, to allow * PCI scanning code to "skip" this now blacklisted device. -- 1.7.5.4
Hi, Bjorn
Quoted 27 lines
Quoted 27 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
This needs more explanation. We've already read the BARs by the time header quirks are run, so apparently it's not just the mere act of accessing a BAR that causes a hang.
Quoted 8 lines
Quoted 6 lines
We need to know exactly what's going on here. For example, do BARs
0-4 exist? Does the device decode accesses to the regions described by the BARs? The PCI core has to know what resources the device uses, so if the device decodes accesses, we can't just throw away the start/end information.The BARs 0-4 is exist and the PCI device is enable IO space, but user access
the regions file by udevadm command with info parameter, the system will hang.
Quoted 1 line
Like this: udevadmin info --attribut-walk
--path=/sys/device/pci-device/000:*.
Quoted 2 lines
Because the device is just AHCI host controller, don't need the BAR0 ~ 4 region
file.
Quoted 1 line
Is my explanation ok for the patch?
No, I still don't know what causes the hang; I only know that udevadm can trigger it. I don't want to just paper over the problem until we know what the root cause is.
Quoted 6 lines
Quoted 2 lines
Does "lspci -H1 -vv" also cause a hang? What about "setpci -s<dev>
BASE_ADDRESS_0"? "setpci -H1 -s<dev>BASE_ADDRESS_0"?The commands are ok because the commands can't find the device after accessing IO port.
The root cause is that accessing of IO port will make the chip go bad. So, the point of the patch is don't export capability of the IO accessing.Ah, so the problem is not with accessing the BAR in config space. The problem is with accessing the I/O port space mapped by the BAR. Is that right?
Yes...
Quoted 1 line
Does "udevadm info --attribute-walk" really access the device address space mapped by the BARs?
The older version maybe will access the space, I just got the info from HP. And I simplify the issue by executing following command:
Cat /sys/devices/pci-device/**/resourceX
I want to set the resources of BAR0 ~ 4 to 0 to avoid the IO accessing by user.
Any question? Thanks!
Quoted 36 lines · Xiangliang Yu
Hi, Bjorn
Quoted 27 lines
Quoted 27 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
This needs more explanation. We've already read the BARs by the time header quirks are run, so apparently it's not just the mere act of accessing a BAR that causes a hang.
Quoted 8 lines
Quoted 6 lines
We need to know exactly what's going on here. For example, do BARs
0-4 exist? Does the device decode accesses to the regions described by the BARs? The PCI core has to know what resources the device uses, so if the device decodes accesses, we can't just throw away the start/end information.The BARs 0-4 is exist and the PCI device is enable IO space, but user access
the regions file by udevadm command with info parameter, the system will hang.
Quoted 1 line
Like this: udevadmin info --attribut-walk
--path=/sys/device/pci-device/000:*.
Quoted 2 lines
Because the device is just AHCI host controller, don't need the BAR0 ~ 4 region
file.
Quoted 1 line
Is my explanation ok for the patch?
No, I still don't know what causes the hang; I only know that udevadm can trigger it. I don't want to just paper over the problem until we know what the root cause is.
Quoted 6 lines
Quoted 2 lines
Does "lspci -H1 -vv" also cause a hang? What about "setpci -s<dev>
BASE_ADDRESS_0"? "setpci -H1 -s<dev>BASE_ADDRESS_0"?The commands are ok because the commands can't find the device after accessing IO port.
The root cause is that accessing of IO port will make the chip go bad. So, the point of the patch is don't export capability of the IO accessing.Ah, so the problem is not with accessing the BAR in config space. The problem is with accessing the I/O port space mapped by the BAR. Is that right?
Quoted 8 lines
Yes...
Quoted 1 line
Does "udevadm info --attribute-walk" really access the device address space mapped by the BARs?
The older version maybe will access the space, I just got the info from HP. And I simplify the issue by executing following command:
Cat/sys/devices/pci-device/**/resourceXI want to set the resources of BAR0 ~ 4 to 0 to avoid the IO accessing by user.
I tried to explain earlier the possible issues with the approach that is currently being put forth. Please review that and if you have any questions ask.
Quoted 2 lines
Any question? Thanks!
Googling and looking at the PCI IDs data base I see that the Marvell 9125 device has been around since sometime around 2010 and that there even seem to be a number of follow-on iterations of the chip (i.e. 9128, 9120, ...). It seems incredibly unlikely that Marvell made a device that has been shipping for 2+ years with five I/O BARs that do not work and we are only now finding out such.
Am I missing something relevant here? Can you verify that this device has is indeed not new and has been successfully used in recent platforms?
You just recently responded with "... I just got the info from HP. ..." so I'm assuming this is an issue that has just been encountered on some type of HP system - is this correct? If so, do you have access to the system to provide the logs I asked for earlier? Also, is there anything special or completely new about this platform that would explain away the arguments for why this is probably not a Marvell device issue?
At this point it seems more likely that there is an issue with the BIOS of the HP system, perhaps a resource duplication/overlap issue much like I talked about earlier.
To understand the root cause and not just band-aid over a symptom we need to get the logs asked for from the system. HP likely needs to get involved and start participating and providing such at this point.
Again, the logs that would be helpful currently are: A 'dmesg' log from the system which was booted using both the "debug" and "ignore_loglevel" boot parameters, a 'lspci -xxx -s<dev>' capture targeting the Marvell 9125 device, and a 'lspci -vv' capture of the system's entire PCI hierarchy.
Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 13 lines · Xiangliang Yu
Hi, Myron
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
Any question? Thanks!
Googling and looking at the PCI IDs data base I see that the Marvell 9125 device has been around since sometime around 2010 and that there even seem to be a number of follow-on iterations of the chip (i.e. 9128, 9120, ...). It seems incredibly unlikely that Marvell made a device that has been shipping for 2+ years with five I/O BARs that do not work and we are only now finding out such.
Quoted 6 lines
Just only 9125 has the issue.
Quoted 3 lines
Am I missing something relevant here? Can you verify that this device has is indeed not new and has been successfully used in recent platforms?
The device can used in recent platforms.
Could you please be a little more explicit (and I'll try to be more specific in my questions) as I was not able to get much, if any, understanding from the responses.
I would like to understand if the 9125 device has had issues corresponding to accessing the I/O Port space mapped by its BARS from the very beginning - i.e. there have been no platforms in the last 2+ years that have been able to successfully drive this device using its I/O BAR accessing methods?
What seems more likely is that only now, due to some new and yet unknown reason, are issues corresponding to accessing the I/O Port space mapped by its BARS occurring - perhaps something to do with a new processor or chipset.
Are you seeing any similar issues when booting Windows on the same platform?
This information could be helpful in tracking down the root cause.
Quoted 11 lines
Quoted 8 lines
You just recently responded with "... I just got the info from HP. ..." so I'm assuming this is an issue that has just been encountered on some type of HP system - is this correct? If so, do you have access to the system to provide the logs I asked for earlier? Also, is there anything special or completely new about this platform that would explain away the arguments for why this is probably not a Marvell device issue?
I can reproduce the issue with following platform:
CPU: Intel i7-3770 3.40GHZ
OS: centos 6.4
6.4 is a fairly old kernel by now - 2.6.32. Have you been able to try an upstream kernel and if so, what were the results?
Quoted 6 lines
Now, the situation is like this:
I captured the PCIE trace with analyzer and found that 1st BE is0x1111when accessing IO port space. But 9125 spec has some limitation, and the BE must be0x0100, to access the 2nd byte only. So, the chip will go to bad.
Great, this is new, interesting, data. Is the 9125 spec publicly accessible and/or could you elaborate on the "some limitation" comment?
I'm fairly sure that PCI Express supports byte-granular accesses to I/O port space (I'll try to read up on this some more as I don't usually work at this low of a level) and it seems unlikely that this area would be broken in a chipset, especially an Intel one.
A byte enable (BE) of 0x1111 suggests the CPU did a 32-bit I/O port read. Does the 9125 device only support one-byte I/O port accesses and when presented with larger request types it doesn't respond properly? I have to admit I don't know what the correct response would be - perhaps a master abort. Do you know what the PCI host controller would return to the CPU so the CPU wouldn't hang in such a case?
Quoted 1 line
Can you tell me what can I do to fix the issue? Thanks!
Once we understand the root cause I'm sure we'll be able to come up with a solution. Let's keep honing in on the problem for now until we get to that understanding.
Quoted 3 lines
A message that is not in the archive (sent off-list or not yet received): [email protected]
Bjorn Helgaas
maintainerQuoted 9 lines · Xiangliang Yu
Hi, Myron
Quoted 8 lines
Quoted 7 lines
Now, the situation is like this:
I captured the PCIE trace with analyzer and found that 1st BE is0x1111when
accessing IO port space. But 9125 spec has some limitation, and the BE must
be0x0100, to access the 2nd byte only. So, the chip will go to bad.
Quoted 4 lines
Quoted 3 lines
Great, this is new, interesting, data. Is the 9125 spec publicly accessible and/or could you elaborate on the "some limitation" comment?
9125 spec is publicly accessible.
Please provide a URL for the spec.
Quoted 5 lines
Quoted 4 lines
A byte enable (BE) of
0x1111suggests the CPU did a 32-bit I/O port read. Does the 9125 device only support one-byte I/O port accesses and when presented with larger request types it doesn't respond properly?Yes, the hardware engineer had confirmed the situation.
Please provide a URL for the erratum describing this 9125 issue.
Bjorn
Hi, Bjorn
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Now, the situation is like this:
I captured the PCIE trace with analyzer and found that 1st BE is0x1111when
accessing IO port space. But 9125 spec has some limitation, and the BE must
be0x0100, to access the 2nd byte only. So, the chip will go to bad.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Great, this is new, interesting, data. Is the 9125 spec publicly accessible and/or could you elaborate on the "some limitation" comment?
9125 spec is publicly accessible.
If you can't see the pic, please open the attachment. Thanks!
Quoted 10 lines · Xiangliang Yu
Hi, Bjorn
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Now, the situation is like this:
I captured the PCIE trace with analyzer and found that 1st BE is0x1111when
accessing IO port space. But 9125 spec has some limitation, and the BE must
be0x0100, to access the 2nd byte only. So, the chip will go to bad.
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Great, this is new, interesting, data. Is the 9125 spec publicly accessible and/or could you elaborate on the "some limitation" comment?
9125 spec is publicly accessible.
If you can't see the pic, please open the attachment. Thanks!
Neither Bjorn nor myself could see the pic (from the previous thread or this thread's attachment).
Quoted 2 lines
Quoted 11 lines · Myron Stowe
Quoted 10 lines · Xiangliang Yu
Hi, Bjorn
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Now, the situation is like this:
I captured the PCIE trace with analyzer and found that 1st BE is0x1111when
accessing IO port space. But 9125 spec has some limitation, and the BE must
be0x0100, to access the 2nd byte only. So, the chip will go to bad.
Quoted 11 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Great, this is new, interesting, data. Is the 9125 spec publicly accessible and/or could you elaborate on the "some limitation" comment?
9125 spec is publicly accessible.
If you can't see the pic, please open the attachment. Thanks!
Neither Bjorn nor myself could see the pic (from the previous thread or this thread's attachment).
Quoted 2 lines
Just an FYI that I proposed a different tact at
https://lkml.org/lkml/2013/3/16/168. For those following this thread you may want to start following that thread also.
Quoted 24 lines · Myron Stowe
Quoted 11 lines · Myron Stowe
Quoted 10 lines · Xiangliang Yu
Hi, Bjorn
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Now, the situation is like this:
I captured the PCIE trace with analyzer and found that 1st BE is0x1111when
accessing IO port space. But 9125 spec has some limitation, and the BE must
be0x0100, to access the 2nd byte only. So, the chip will go to bad.Quoted 11 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Great, this is new, interesting, data. Is the 9125 spec publicly accessible and/or could you elaborate on the "some limitation" comment?
9125 spec is publicly accessible.
If you can't see the pic, please open the attachment. Thanks!
Neither Bjorn nor myself could see the pic (from the previous thread or this thread's attachment).
Quoted 2 lines
Quoted 3 lines
Just an FYI that I proposed a different tact at
https://lkml.org/lkml/2013/3/16/168. For those following this thread you may want to start following that thread also.
I posted a third approach last night. For those of you following this thread there are now three streams:
This stream,
https://lkml.org/lkml/2013/3/16/168 (which is also on the linux-pci list), and https://lkml.org/lkml/2013/3/21/12 (which is also on the linux-pci list).
Perhaps the third time will be a charm
Quoted 22 lines · Xiangliang Yu
Hi, Bjorn
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
This needs more explanation. We've already read the BARs by the time header quirks are run, so apparently it's not just the mere act of accessing a BAR that causes a hang.
Quoted 6 lines
Quoted 5 lines
We need to know exactly what's going on here. For example, do BARs 0-4 exist? Does the device decode accesses to the regions described by the BARs? The PCI core has to know what resources the device uses, so if the device decodes accesses, we can't just throw away the start/end information.
The BARs 0-4 is exist and the PCI device is enable IO space, but user access
the regions file by udevadm command with info parameter, the system will hang.
Quoted 1 line
Like this: udevadmin info --attribut-walk
--path=/sys/device/pci-device/000:*.
Quoted 1 line
Because the device is just AHCI host controller, don't need the BAR0 ~ 4 region
file.
Quoted 1 line
Is my explanation ok for the patch?
Quoted 7 lines
Quoted 6 lines
No, I still don't know what causes the hang; I only know that udevadm can trigger it. I don't want to just paper over the problem until we know what the root cause is.
Does "lspci -H1 -vv" also cause a hang? What about "setpci -s<dev>
BASE_ADDRESS_0"? "setpci -H1 -s<dev>BASE_ADDRESS_0"?The commands are ok because the commands can't find the device after accessing IO port.
Xiangliang:
Sorry but I didn't understand your response above, could you elaborate a little more?
Are the first five BARs of the suspect device all mapping to I/O port space - i.e. similar to something like this (a capture and inclusion of an 'lspci' of the suspect device would be nice to see):
00:1f.2 SATA controller:
Region 0: I/O ports at 1860 [size=8]
Region 1: I/O ports at 1814 [size=4]
Region 2: I/O ports at 1818 [size=8]
Region 3: I/O ports at 1810 [size=4]
Region 4: I/O ports at 1840 [size=32]
Region 5: Memory at f2827000 (32-bit, non-prefetchable) [size=2K]You have done a good job isolating the issue so far. As Bjorn noted; it's looking as if the problem is with accessing the I/O port space mapped by the suspect device's BAR(s), not with accessing the BAR(s) in the device's configuration space.
As you responded positively to earlier, as proposed the suspect device will still actively be decoding accesses to the regions described by the BARs. Because the device is actively decoding the PCI core can't just throw away the BAR's corresponding resource regions, as the patch is currently doing, due to the possibility of another device being added at a later time.
If a subsequent device were added later, the core may need to try and allocate resources for it and, in the worst case scenario, the core could end up allocating resources that conflict with this suspect device as a consequence of the suspect device's original resource allocations having been silently thrown away. The result would be both devices believing they each exclusively own the same set (or subset) of I/O port mappings and thus both actively decoding accesses to such which. A situation that would obviously be disastrous.
There is still something going on here that we still do not understand. Could you please capture the following information to help further isolate the issue:
A 'dmesg' log from the system which was booted using both the
"debug" and "ignore_loglevel" boot parameters, a 'lspci -xxx -s<dev>' capture, and a 'lspci -vv' capture.
Thanks,
Myron
Quoted 42 lines
The root cause is that accessing of IO port will make the chip go bad. So, the point of the patch is don't export capability of the IO accessing.
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
drivers/pci/quirks.c static void quirk_mmio_always_on(struct pci_dev *dev) · line 45
Quoted 13 lines
--- drivers/pci/quirks.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-)drivers/pci/quirks.c +0
-0@@ -44,6 +44,21 @@ static void quirk_mmio_always_on(struct pci_dev *dev) DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_ANY_ID, PCI_ANY_ID, PCI_CLASS_BRIDGE_HOST, 8,
quirk_mmio_always_on);Quoted 14 lines
drivers/pci/quirks.c +13
-0+/* The BAR0 ~ BAR4 of Marvell 9125 device can't be accessed +* by IO resource file, and need to skip the files +*/ +static void quirk_marvell_mask_bar (struct pci_dev *dev) +{ +int i; + +for (i =0 ; i <5 ; i++) +if (dev->resource[i].start) + dev->resource[i].start = + dev->resource[i].end =0 ; +} +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9125,Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/pci/quirks.c +2
-1+ quirk_marvell_mask_bar); +/* The Mellanox Tavor device gives false positive parity errors * Mark this device with a broken_parity_status, to allow * PCI scanning code to "skip" this now blacklisted device. -- 1.7.5.4
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 48 lines · Myron Stowe
Quoted 22 lines · Xiangliang Yu
Hi, Bjorn
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
This needs more explanation. We've already read the BARs by the time header quirks are run, so apparently it's not just the mere act of accessing a BAR that causes a hang.
Quoted 6 lines
Quoted 5 lines
We need to know exactly what's going on here. For example, do BARs 0-4 exist? Does the device decode accesses to the regions described by the BARs? The PCI core has to know what resources the device uses, so if the device decodes accesses, we can't just throw away the start/end information.
The BARs 0-4 is exist and the PCI device is enable IO space, but user access
the regions file by udevadm command with info parameter, the system will hang.
Quoted 1 line
Like this: udevadmin info --attribut-walk
--path=/sys/device/pci-device/000:*.
Quoted 1 line
Because the device is just AHCI host controller, don't need the BAR0 ~ 4 region
file.
Quoted 1 line
Is my explanation ok for the patch?
Quoted 7 lines
Quoted 6 lines
No, I still don't know what causes the hang; I only know that udevadm can trigger it. I don't want to just paper over the problem until we know what the root cause is.
Does "lspci -H1 -vv" also cause a hang? What about "setpci -s<dev>
BASE_ADDRESS_0"? "setpci -H1 -s<dev>BASE_ADDRESS_0"?The commands are ok because the commands can't find the device after accessing IO port.
Xiangliang:
Sorry but I didn't understand your response above, could you elaborate a little more?
Are the first five BARs of the suspect device all mapping to I/O port space - i.e. similar to something like this (a capture and inclusion of an 'lspci' of the suspect device would be nice to see):
00:1f.2 SATA controller: Region 0: I/O ports at 1860 [size=8] Region 1: I/O ports at 1814 [size=4] Region 2: I/O ports at 1818 [size=8] Region 3: I/O ports at 1810 [size=4] Region 4: I/O ports at 1840 [size=32] Region 5: Memory at f2827000 (32-bit, non-prefetchable) [size=2K]
Quoted 4 lines
You have done a good job isolating the issue so far. As Bjorn noted; it's looking as if the problem is with accessing the I/O port space mapped by the suspect device's BAR(s), not with accessing the BAR(s) in the device's configuration space.
It would seem so. My question is what is accessing the IO port space in the first place. BAR5 is the MMIO region used by the AHCI driver. BARs 0-4 are the legacy SFF-compatible ATA ports. Nothing should be messing with those IO ports while AHCI is enabled. It's expected that doing that will break things.
If something in udev is randomly groveling around inside the resource files for those BARs in sysfs, that seems like a really bad thing.
Quoted 69 lines
As you responded positively to earlier, as proposed the suspect device will still actively be decoding accesses to the regions described by the BARs. Because the device is actively decoding the PCI core can't just throw away the BAR's corresponding resource regions, as the patch is currently doing, due to the possibility of another device being added at a later time.
If a subsequent device were added later, the core may need to try and allocate resources for it and, in the worst case scenario, the core could end up allocating resources that conflict with this suspect device as a consequence of the suspect device's original resource allocations having been silently thrown away. The result would be both devices believing they each exclusively own the same set (or subset) of I/O port mappings and thus both actively decoding accesses to such which. A situation that would obviously be disastrous.
There is still something going on here that we still do not understand. Could you please capture the following information to help further isolate the issue:
A 'dmesg' log from the system which was booted using both the
"debug" and "ignore_loglevel" boot parameters, a 'lspci -xxx -s<dev>' capture, and a 'lspci -vv' capture.Thanks,
MyronQuoted 42 lines
The root cause is that accessing of IO port will make the chip go bad. So, the point of the patch is don't export capability of the IO accessing.
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
drivers/pci/quirks.c static void quirk_mmio_always_on(struct pci_dev *dev) · line 45
Quoted 13 lines
--- drivers/pci/quirks.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-)drivers/pci/quirks.c +0
-0@@ -44,6 +44,21 @@ static void quirk_mmio_always_on(struct pci_dev *dev) DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_ANY_ID, PCI_ANY_ID, PCI_CLASS_BRIDGE_HOST, 8,
quirk_mmio_always_on);Quoted 14 lines
drivers/pci/quirks.c +13
-0+/* The BAR0 ~ BAR4 of Marvell 9125 device can't be accessed +* by IO resource file, and need to skip the files +*/ +static void quirk_marvell_mask_bar (struct pci_dev *dev) +{ +int i; + +for (i =0 ; i <5 ; i++) +if (dev->resource[i].start) + dev->resource[i].start = + dev->resource[i].end =0 ; +} +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9125,Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/pci/quirks.c +2
-1+ quirk_marvell_mask_bar); +/* The Mellanox Tavor device gives false positive parity errors * Mark this device with a broken_parity_status, to allow * PCI scanning code to "skip" this now blacklisted device. -- 1.7.5.4
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 73 lines · Robert Hancock
Quoted 66 lines · Myron Stowe
Quoted 39 lines · Xiangliang Yu
Hi, Bjorn
Quoted 36 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Fix system hang issue: if first accessed resource file of BAR0 ~ BAR4, system will hang after executing lspci command
This needs more explanation. We've already read the BARs by the time header quirks are run, so apparently it's not just the mere act of accessing a BAR that causes a hang.
Quoted 8 lines
Quoted 5 lines
We need to know exactly what's going on here. For example, do BARs 0-4 exist? Does the device decode accesses to the regions described by the BARs? The PCI core has to know what resources the device uses, so if the device decodes accesses, we can't just throw away the start/end information.
The BARs 0-4 is exist and the PCI device is enable IO space, but user access
the regions file by udevadm command with info parameter, the system will hang.
Quoted 2 lines
Like this: udevadmin info --attribut-walk
--path=/sys/device/pci-device/000:*.
Quoted 3 lines
Because the device is just AHCI host controller, don't need the BAR0 ~ 4 region
file.
Quoted 2 lines
Is my explanation ok for the patch?
No, I still don't know what causes the hang; I only know that udevadm can trigger it. I don't want to just paper over the problem until we know what the root cause is.
Quoted 5 lines
Quoted 2 lines
Does "lspci -H1 -vv" also cause a hang? What about "setpci -s<dev>
BASE_ADDRESS_0"? "setpci -H1 -s<dev>BASE_ADDRESS_0"?The commands are ok because the commands can't find the device after accessing IO port.
Xiangliang:
Sorry but I didn't understand your response above, could you elaborate a little more?
Are the first five BARs of the suspect device all mapping to I/O port space - i.e. similar to something like this (a capture and inclusion of an 'lspci' of the suspect device would be nice to see):
00:1f.2 SATA controller: Region 0: I/O ports at 1860 [size=8] Region 1: I/O ports at 1814 [size=4] Region 2: I/O ports at 1818 [size=8] Region 3: I/O ports at 1810 [size=4] Region 4: I/O ports at 1840 [size=32] Region 5: Memory at f2827000 (32-bit, non-prefetchable) [size=2K]Quoted 4 lines
You have done a good job isolating the issue so far. As Bjorn noted; it's looking as if the problem is with accessing the I/O port space mapped by the suspect device's BAR(s), not with accessing the BAR(s) in the device's configuration space.
Quoted 8 lines
It would seem so. My question is what is accessing the IO port space in the first place. BAR5 is the MMIO region used by the AHCI driver. BARs 0-4 are the legacy SFF-compatible ATA ports. Nothing should be messing with those IO ports while AHCI is enabled. It's expected that doing that will break things.
If something in udev is randomly groveling around inside the resource files for those BARs in sysfs, that seems like a really bad thing.
Thanks Robert
I'll see if I can get someone knowledgable about udev to look at this since it seems to be the suspect currently.
Hopefully this will lead somewhere as the core isn't, and shouldn't be, concerned with the contents or access limitations of those regions. If the 9125 doesn't respond correctly to 32-bit IO port reads and adjustments for such end up needing to be accounted for then they should to be covered by the driver and not the PCI core.
Quoted 77 lines
Quoted 75 lines
As you responded positively to earlier, as proposed the suspect device will still actively be decoding accesses to the regions described by the BARs. Because the device is actively decoding the PCI core can't just throw away the BAR's corresponding resource regions, as the patch is currently doing, due to the possibility of another device being added at a later time.
If a subsequent device were added later, the core may need to try and allocate resources for it and, in the worst case scenario, the core could end up allocating resources that conflict with this suspect device as a consequence of the suspect device's original resource allocations having been silently thrown away. The result would be both devices believing they each exclusively own the same set (or subset) of I/O port mappings and thus both actively decoding accesses to such which. A situation that would obviously be disastrous.
There is still something going on here that we still do not understand. Could you please capture the following information to help further isolate the issue:
A 'dmesg' log from the system which was booted using both the
"debug" and "ignore_loglevel" boot parameters, a 'lspci -xxx -s<dev>' capture, and a 'lspci -vv' capture.Thanks,
MyronQuoted 46 lines
The root cause is that accessing of IO port will make the chip go bad. So, the point of the patch is don't export capability of the IO accessing.
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 11 lines
--- drivers/pci/quirks.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-)drivers/pci/quirks.c +0
-0drivers/pci/quirks.c static void quirk_mmio_always_on(struct pci_dev · line 46
Quoted 3 lines
drivers/pci/quirks.c +0
-0@@ -44,6 +44,21 @@ static void quirk_mmio_always_on(struct pci_dev *dev) DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_ANY_ID, PCI_ANY_ID, PCI_CLASS_BRIDGE_HOST, 8,
quirk_mmio_always_on);Quoted 15 lines
drivers/pci/quirks.c +13
-0+/* The BAR0 ~ BAR4 of Marvell 9125 device can't be accessed +* by IO resource file, and need to skip the files +*/ +static void quirk_marvell_mask_bar (struct pci_dev *dev) +{ +int i; + +for (i =0 ; i <5 ; i++) +if (dev->resource[i].start) + dev->resource[i].start = + dev->resource[i].end =0 ; +} +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9125,Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/pci/quirks.c +2
-1+ quirk_marvell_mask_bar); +/* The Mellanox Tavor device gives false positive parity errors * Mark this device with a broken_parity_status, to allow * PCI scanning code to "skip" this now blacklisted device. -- 1.7.5.4
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html