PCI: aardvark: Implement workaround for PCIe Completion Timeout
Subsystems (3)
Pali Rohár
maintainerPATCH v2 1/1 PCI: aardvark: Implement workaround for PCIe Completion Timeout
Marvell Armada 3700 Functional Errata, Guidelines, and Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #: 251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.
As a workaround for this erratum, DIS_ORD_CHK flag in Debug Mux Control register must be set. It disables the ordering check in the core between Completions and Posted requests received from the link.
Marvell also suggests to do full memory barrier at the beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize the risk for the race condition documented in the Erratum between the DMA done status reading and the completion of writing to the host memory.
More details about this issue and suggested workarounds are in discussion:
https://lore.kernel.org/linux-pci/BN9PR18MB425154FE5019DCAF2028A1D5DB8D9@BN9PR18MB4251.namprd18.prod.outlook.com/t/#u
It was reported that enabling this workaround fixes instability issues and "Unhandled fault" errors when using 60 GHz WiFi 802.11ad card with Qualcomm QCA6335 chip under significant load which were caused by interrupt status stuck in the outbound CMPLT queue traced back to this erratum.
This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP
Kernel panic - not syncing: Fatal exception in interrupt--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
drivers/pci/controller/pci-aardvark.c +10 -0
@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000
@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;
@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status; +/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ;
Signature · 1 line
2.20.1
Pali Rohár
maintainerHello Elad, could you please review this patch? I have implemented it according your instructions, including that full memory barrier as you described.
Quoted 75 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #: 251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.
As a workaround for this erratum,
DIS_ORD_CHKflag in Debug Mux Control register must be set. It disables the ordering check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize the risk for the race condition documented in the Erratum between the DMA done status reading and the completion of writing to the host memory.
More details about this issue and suggested workarounds are in discussion:
https://lore.kernel.org/linux-pci/BN9PR18MB425154FE5019DCAF2028A1D5DB8D9@BN9PR18MB4251.namprd18.prod.outlook.com/t/#uIt was reported that enabling this workaround fixes instability issues and "Unhandled fault" errors when using 60 GHz WiFi 802.11ad card with Qualcomm QCA6335 chip under significant load which were caused by interrupt status stuck in the outbound CMPLT queue traced back to this erratum.
This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI host controller driver")--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +10
-0@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status; +/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ;Signature · 1 line
2.20.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Elad Nachman
Reviewed-byThanks,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Monday, September 26, 2022 3:35 PM
To: Elad Nachman <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; Rob Herring <[email protected]>; [email protected]; [email protected]; [email protected]; Gregory Clement <[email protected]>; Marek Behún <[email protected]>; Remi Pommarel <[email protected]>; Xogium <[email protected]>; Tomasz Maciej Nowak <[email protected]>
Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion Timeout
External Email
----------------------------------------------------------------------
Hello Elad, could you please review this patch? I have implemented it according your instructions, including that full memory barrier as you described.
Quoted 83 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #:
251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.As a workaround for this erratum,
DIS_ORD_CHKflag in Debug Mux Control register must be set. It disables the ordering check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize the risk for the race condition documented in the Erratum between the DMA done status reading and the completion of writing to the host memory.
More details about this issue and suggested workarounds are in discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_l
inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB4251.namp rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=eTeNT LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONCZMiFDX72 MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME8jNiRgrq 4SDYvqqT0TA&e=It was reported that enabling this workaround fixes instability issues and "Unhandled fault" errors when using 60 GHz WiFi 802.11ad card with Qualcomm
QCA6335 chip under significant load which were caused by interrupt status stuck in the outbound CMPLT queue traced back to this erratum.This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI host controllerdriver")
Cc: [email protected]--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +10
-1@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status; +/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ; --2.20.1
Pali Rohár
maintainerLorenzo, is something more needed for this patch? As it workarounds crashing it is really needed to have it in mainline and backports.
Quoted 93 lines · On Wednesday 28 September 2022 14:05:10 Elad Nachman wrote:
Reviewed-by Elad Nachman <[email protected]>Thanks,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Monday, September 26, 2022 3:35 PM
To: Elad Nachman <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; Rob Herring <[email protected]>; [email protected]; [email protected]; [email protected]; Gregory Clement <[email protected]>; Marek Behún <[email protected]>; Remi Pommarel <[email protected]>; Xogium <[email protected]>; Tomasz Maciej Nowak <[email protected]>
Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion TimeoutExternal Email
----------------------------------------------------------------------
Hello Elad, could you please review this patch? I have implemented it according your instructions, including that full memory barrier as you described.Quoted 74 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #:
251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.As a workaround for this erratum,
DIS_ORD_CHKflag in Debug Mux Control register must be set. It disables the ordering check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize the risk for the race condition documented in the Erratum between the DMA done status reading and the completion of writing to the host memory.
More details about this issue and suggested workarounds are in discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_l
inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB4251.namp rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=eTeNT LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONCZMiFDX72 MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME8jNiRgrq 4SDYvqqT0TA&e=It was reported that enabling this workaround fixes instability issues and "Unhandled fault" errors when using 60 GHz WiFi 802.11ad card with Qualcomm
QCA6335 chip under significant load which were caused by interrupt status stuck in the outbound CMPLT queue traced back to this erratum.This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI host controllerdriver")
Cc: [email protected]--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +7
-0@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status;
Quoted 9 lines
Quoted 9 lines
drivers/pci/controller/pci-aardvark.c +3
-1+/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ; --2.20.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Lorenzo Pieralisi
maintainerQuoted 2 lines · Pali Rohár
Lorenzo, is something more needed for this patch? As it workarounds crashing it is really needed to have it in mainline and backports.
Yes, a clear explanation from Marvell about what this is actually fixing - it took me a while to go through the whole thread but I still don't understand what this patch actually does and why.
An Erratum workaround (if there is any) should define and explain a SW workaround.
(1) Bjorn's concerns in relation to PCI memory model weren't addressed
(2) We don't add undocumented memory barriers to the kernel to "minimize risks". Either we fix a bug or we don't. If we do, write that down and document why the barrier is there and the issue it solves.
I understand that basically you are reverse engineering a HW bug but I am afraid we can't fix the kernel this way - more so with patches going to be backported to stable kernels.
Lorenzo
Quoted 108 lines · On Wednesday 28 September 2022 14:05:10 Elad Nachman wrote:
Quoted 93 lines
Reviewed-by Elad Nachman <[email protected]>Thanks,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Monday, September 26, 2022 3:35 PM
To: Elad Nachman <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; Rob Herring <[email protected]>; [email protected]; [email protected]; [email protected]; Gregory Clement <[email protected]>; Marek Behún <[email protected]>; Remi Pommarel <[email protected]>; Xogium <[email protected]>; Tomasz Maciej Nowak <[email protected]>
Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion TimeoutExternal Email
----------------------------------------------------------------------
Hello Elad, could you please review this patch? I have implemented it according your instructions, including that full memory barrier as you described.Quoted 74 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #:
251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.As a workaround for this erratum,
DIS_ORD_CHKflag in Debug Mux Control register must be set. It disables the ordering check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize the risk for the race condition documented in the Erratum between the DMA done status reading and the completion of writing to the host memory.
More details about this issue and suggested workarounds are in discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_l
inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB4251.namp rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=eTeNT LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONCZMiFDX72 MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME8jNiRgrq 4SDYvqqT0TA&e=It was reported that enabling this workaround fixes instability issues and "Unhandled fault" errors when using 60 GHz WiFi 802.11ad card with Qualcomm
QCA6335 chip under significant load which were caused by interrupt status stuck in the outbound CMPLT queue traced back to this erratum.This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI host controllerdriver")
Cc: [email protected]--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +7
-0@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status;Quoted 9 lines
Quoted 9 lines
drivers/pci/controller/pci-aardvark.c +3
-1+/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ; --2.20.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Pali Rohár
maintainer+ Elad
Could you please look at Lorenzo's comments and help with this fix?
Quoted 130 lines · On Tuesday 04 October 2022 10:02:32 Lorenzo Pieralisi wrote:
Quoted 2 lines · Pali Rohár
Lorenzo, is something more needed for this patch? As it workarounds crashing it is really needed to have it in mainline and backports.
Yes, a clear explanation from Marvell about what this is actually fixing - it took me a while to go through the whole thread but I still don't understand what this patch actually does and why.
An Erratum workaround (if there is any) should define and explain a SW workaround.
(1) Bjorn's concerns in relation to PCI memory model weren't addressed
(2) We don't add undocumented memory barriers to the kernel to "minimize risks". Either we fix a bug or we don't. If we do, write that down and document why the barrier is there and the issue it solves.I understand that basically you are reverse engineering a HW bug but I am afraid we can't fix the kernel this way - more so with patches going to be backported to stable kernels.
Lorenzo
Quoted 108 lines · On Wednesday 28 September 2022 14:05:10 Elad Nachman wrote:
Quoted 93 lines
Reviewed-by Elad Nachman <[email protected]>Thanks,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Monday, September 26, 2022 3:35 PM
To: Elad Nachman <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; Rob Herring <[email protected]>; [email protected]; [email protected]; [email protected]; Gregory Clement <[email protected]>; Marek Behún <[email protected]>; Remi Pommarel <[email protected]>; Xogium <[email protected]>; Tomasz Maciej Nowak <[email protected]>
Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion TimeoutExternal Email
----------------------------------------------------------------------
Hello Elad, could you please review this patch? I have implemented it according your instructions, including that full memory barrier as you described.Quoted 74 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #:
251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.As a workaround for this erratum,
DIS_ORD_CHKflag in Debug Mux Control register must be set. It disables the ordering check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize the risk for the race condition documented in the Erratum between the DMA done status reading and the completion of writing to the host memory.
More details about this issue and suggested workarounds are in discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_l
inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB4251.namp rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=eTeNT LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONCZMiFDX72 MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME8jNiRgrq 4SDYvqqT0TA&e=It was reported that enabling this workaround fixes instability issues and "Unhandled fault" errors when using 60 GHz WiFi 802.11ad card with Qualcomm
QCA6335 chip under significant load which were caused by interrupt status stuck in the outbound CMPLT queue traced back to this erratum.This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI host controllerdriver")
Cc: [email protected]--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +7
-0@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status;Quoted 9 lines
Quoted 9 lines
drivers/pci/controller/pci-aardvark.c +3
-1+/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ; --2.20.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi,
Regarding number #1 :
The H/W erratum means strong ordering mode is not properly implemented in the H/W.
The workaround is to set DIS_ORD_CHK to disable the ordered check, to prevent the issues Pali mentioned.
Leaving it without the workaround means encountering the crashes Pali mentioned.
The H/W was originally designed to have the strong ordering mode, per what is expected of PCI Express, but due to this erratum, it cannot be supported by the taped-out H/W. There is nothing to do about it now in the H/W, as this is a years old SOC.
Software wise the only option is the workaround above, or to accept sporadic crashes of the kernel.
Regarding number #2 :
The impact of the workaround is that the DMA done status could be set before the last data is written to the host memory, causing the host to read incomplete data. This is the reason why I recommended to add a memory barrier. If this is bothering, the memory barrier can be removed from the patch, although I suggest to thoroughly test the data integrity if implementing the patch without the memory barrier.
FYI,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Tuesday, October 4, 2022 11:40 AM
To: Elad Nachman <[email protected]>; Lorenzo Pieralisi <[email protected]>
Cc: Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion Timeout
+ Elad
Could you please look at Lorenzo's comments and help with this fix?
Quoted 150 lines · On Tuesday 04 October 2022 10:02:32 Lorenzo Pieralisi wrote:
Quoted 2 lines · Pali Rohár
Lorenzo, is something more needed for this patch? As it workarounds crashing it is really needed to have it in mainline and backports.
Yes, a clear explanation from Marvell about what this is actually fixing - it took me a while to go through the whole thread but I still don't understand what this patch actually does and why.
An Erratum workaround (if there is any) should define and explain a SW workaround.
(1) Bjorn's concerns in relation to PCI memory model weren't addressed
(2) We don't add undocumented memory barriers to the kernel to "minimize risks". Either we fix a bug or we don't. If we do, write that down and document why the barrier is there and the issue it solves.I understand that basically you are reverse engineering a HW bug but I am afraid we can't fix the kernel this way - more so with patches going to be backported to stable kernels.
Lorenzo
Quoted 128 lines · On Wednesday 28 September 2022 14:05:10 Elad Nachman wrote:
Quoted 109 lines
Reviewed-by Elad Nachman <[email protected]>Thanks,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Monday, September 26, 2022 3:35 PM
To: Elad Nachman <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; Rob Herring <[email protected]>; [email protected]; [email protected]; [email protected]; Gregory Clement <[email protected]>; Marek Behún <[email protected]>; Remi Pommarel <[email protected]>; Xogium <[email protected]>; Tomasz Maciej Nowak <[email protected]>
Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion TimeoutExternal Email
------------------------------------------------------------------
---- Hello Elad, could you please review this patch? I have
implemented it according your instructions, including that full memory barrier as you described.Quoted 80 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and
Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #:
251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.As a workaround for this erratum,
DIS_ORD_CHKflag in Debug Mux Control register must be set. It disables the ordering check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize the risk for the race condition documented in the Erratum between the DMA done status reading and the completion of writing to the host memory.
More details about this issue and suggested workarounds are in discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel
.org_l inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB425 1.namp rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r =eTeNT LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONCZM iFDX72 MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME8j NiRgrq
4SDYvqqT0TA&e=It was reported that enabling this workaround fixes instability issues and "Unhandled fault" errors when using 60 GHz WiFi 802.11ad card with Qualcomm
QCA6335 chip under significant load which were caused by
interrupt status stuck in the outbound CMPLT queue traced back to this erratum.This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI hostcontroller
driver")
Cc: [email protected]--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +7
-0@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status;Quoted 9 lines
Quoted 9 lines
drivers/pci/controller/pci-aardvark.c +3
-1+/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ; --2.20.1
_______________________________________________
linux-arm-kernel mailing list [email protected]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.
org_mailman_listinfo_linux-2Darm-2Dkernel&d=DwIDaQ&c=nKjWec2b6R0mOyP az7xtfQ&r=eTeNTLEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=2NzkT9KLO26k efUOw2nIeSeRnJVZLxEiBXqEoRvDQ0ueww6n4YaXWgAN1uCJX20o&s=nczACS_2jERbA -c4Gfar0-HTA4PtvZdJmsBv8jhW8G0&e=
Pali Rohár
maintainerHello Elad! Thank you very much for detailed explanation.
Lorenzo, it is OK, or do you need some other details?
Quoted 179 lines · On Thursday 06 October 2022 08:14:48 Elad Nachman wrote:
Hi,
Regarding number #1 :
The H/W erratum means strong ordering mode is not properly implemented in the H/W.
The workaround is to setDIS_ORD_CHKto disable the ordered check, to prevent the issues Pali mentioned.
Leaving it without the workaround means encountering the crashes Pali mentioned.
The H/W was originally designed to have the strong ordering mode, per what is expected of PCI Express, but due to this erratum, it cannot be supported by the taped-out H/W. There is nothing to do about it now in the H/W, as this is a years old SOC.
Software wise the only option is the workaround above, or to accept sporadic crashes of the kernel.Regarding number #2 :
The impact of the workaround is that the DMA done status could be set before the last data is written to the host memory, causing the host to read incomplete data. This is the reason why I recommended to add a memory barrier. If this is bothering, the memory barrier can be removed from the patch, although I suggest to thoroughly test the data integrity if implementing the patch without the memory barrier.FYI,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Tuesday, October 4, 2022 11:40 AM
To: Elad Nachman <[email protected]>; Lorenzo Pieralisi <[email protected]>
Cc: Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion Timeout+ Elad
Could you please look at Lorenzo's comments and help with this fix?
Quoted 150 lines · On Tuesday 04 October 2022 10:02:32 Lorenzo Pieralisi wrote:
Quoted 2 lines · Pali Rohár
Lorenzo, is something more needed for this patch? As it workarounds crashing it is really needed to have it in mainline and backports.
Yes, a clear explanation from Marvell about what this is actually fixing - it took me a while to go through the whole thread but I still don't understand what this patch actually does and why.
An Erratum workaround (if there is any) should define and explain a SW workaround.
(1) Bjorn's concerns in relation to PCI memory model weren't addressed
(2) We don't add undocumented memory barriers to the kernel to "minimize risks". Either we fix a bug or we don't. If we do, write that down and document why the barrier is there and the issue it solves.I understand that basically you are reverse engineering a HW bug but I am afraid we can't fix the kernel this way - more so with patches going to be backported to stable kernels.
Lorenzo
Quoted 128 lines · On Wednesday 28 September 2022 14:05:10 Elad Nachman wrote:
Quoted 109 lines
Reviewed-by Elad Nachman <[email protected]>Thanks,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Monday, September 26, 2022 3:35 PM
To: Elad Nachman <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; Rob Herring <[email protected]>; [email protected]; [email protected]; [email protected]; Gregory Clement <[email protected]>; Marek Behún <[email protected]>; Remi Pommarel <[email protected]>; Xogium <[email protected]>; Tomasz Maciej Nowak <[email protected]>
Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion TimeoutExternal Email
------------------------------------------------------------------
---- Hello Elad, could you please review this patch? I have
implemented it according your instructions, including that full memory barrier as you described.Quoted 80 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and
Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #:
251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.As a workaround for this erratum,
DIS_ORD_CHKflag in Debug Mux Control register must be set. It disables the ordering check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize the risk for the race condition documented in the Erratum between the DMA done status reading and the completion of writing to the host memory.
More details about this issue and suggested workarounds are in discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel
.org_l inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB425 1.namp rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r =eTeNT LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONCZM iFDX72 MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME8j NiRgrq
4SDYvqqT0TA&e=It was reported that enabling this workaround fixes instability issues and "Unhandled fault" errors when using 60 GHz WiFi 802.11ad card with Qualcomm
QCA6335 chip under significant load which were caused by
interrupt status stuck in the outbound CMPLT queue traced back to this erratum.This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI hostcontroller
driver")
Cc: [email protected]--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +7
-0@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status;Quoted 9 lines
Quoted 9 lines
drivers/pci/controller/pci-aardvark.c +3
-1+/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ; --2.20.1
_______________________________________________
linux-arm-kernel mailing list [email protected]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.
org_mailman_listinfo_linux-2Darm-2Dkernel&d=DwIDaQ&c=nKjWec2b6R0mOyP az7xtfQ&r=eTeNTLEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=2NzkT9KLO26k efUOw2nIeSeRnJVZLxEiBXqEoRvDQ0ueww6n4YaXWgAN1uCJX20o&s=nczACS_2jERbA -c4Gfar0-HTA4PtvZdJmsBv8jhW8G0&e=
Quoted 15 lines · Elad Nachman
Hi,
Regarding number #1 :
The H/W erratum means strong ordering mode is not properly implemented in the H/W. The workaround is to setDIS_ORD_CHKto disable the ordered check, to prevent the issues Pali mentioned.Leaving it without the workaround means encountering the crashes Pali mentioned. The H/W was originally designed to have the strong ordering mode, per what is expected of PCI Express, but due to this erratum, it cannot be supported by the taped-out H/W. There is nothing to do about it now in the H/W, as this is a years old SOC. Software wise the only option is the workaround above, or to accept sporadic crashes of the kernel.
Quoted 8 lines
Regarding number #2 :
The impact of the workaround is that the DMA done status could be set before the last data is written to the host memory, causing the host to read incomplete data. This is the reason why I recommended to add a memory barrier. If this is bothering, the memory barrier can be removed from the patch, although I suggest to thoroughly test the data integrity if implementing the patch without the memory barrier.
Thank you very much for these details, Elad.
If I understand correctly, this patch avoids many issues (or at least makes them much less likely), but there is still a small possibility of corner cases where a correctly-written driver would work on other platforms but not on this hardware.
Since this definitely improves things dramatically, I think it's worth doing even if it's not absolutely 100%. The commit log and code comments should be enough of a hint.
Quoted 166 lines
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Tuesday, October 4, 2022 11:40 AM
To: Elad Nachman <[email protected]>; Lorenzo Pieralisi <[email protected]>
Cc: Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion Timeout+ Elad
Could you please look at Lorenzo's comments and help with this fix?
Quoted 16 lines · On Tuesday 04 October 2022 10:02:32 Lorenzo Pieralisi wrote:
Quoted 2 lines · Pali Rohár
Lorenzo, is something more needed for this patch? As it workarounds crashing it is really needed to have it in mainline and backports.
Yes, a clear explanation from Marvell about what this is actually fixing - it took me a while to go through the whole thread but I still don't understand what this patch actually does and why.
An Erratum workaround (if there is any) should define and explain a SW workaround.
(1) Bjorn's concerns in relation to PCI memory model weren't addressed
(2) We don't add undocumented memory barriers to the kernel to "minimize risks". Either we fix a bug or we don't. If we do, write that down and document why the barrier is there and the issue it solves.Quoted 134 lines
I understand that basically you are reverse engineering a HW bug but I am afraid we can't fix the kernel this way - more so with patches going to be backported to stable kernels.
Lorenzo
Quoted 128 lines · On Wednesday 28 September 2022 14:05:10 Elad Nachman wrote:
Quoted 109 lines
Reviewed-by Elad Nachman <[email protected]>Thanks,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Monday, September 26, 2022 3:35 PM
To: Elad Nachman <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; Rob Herring <[email protected]>; [email protected]; [email protected]; [email protected]; Gregory Clement <[email protected]>; Marek Behún <[email protected]>; Remi Pommarel <[email protected]>; Xogium <[email protected]>; Tomasz Maciej Nowak <[email protected]>
Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion TimeoutExternal Email
------------------------------------------------------------------
---- Hello Elad, could you please review this patch? I have
implemented it according your instructions, including that full memory barrier as you described.Quoted 80 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and
Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #:
251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.As a workaround for this erratum,
DIS_ORD_CHKflag in Debug Mux Control register must be set. It disables the ordering check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize the risk for the race condition documented in the Erratum between the DMA done status reading and the completion of writing to the host memory.
More details about this issue and suggested workarounds are in discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel
.org_l inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB425 1.namp rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r =eTeNT LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONCZM iFDX72 MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME8j NiRgrq
4SDYvqqT0TA&e=It was reported that enabling this workaround fixes instability issues and "Unhandled fault" errors when using 60 GHz WiFi 802.11ad card with Qualcomm
QCA6335 chip under significant load which were caused by
interrupt status stuck in the outbound CMPLT queue traced back to this erratum.This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI hostcontroller
driver")
Cc: [email protected]--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +7
-0@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status;Quoted 9 lines
Quoted 9 lines
drivers/pci/controller/pci-aardvark.c +3
-1+/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ; --2.20.1
_______________________________________________
linux-arm-kernel mailing list [email protected]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.
org_mailman_listinfo_linux-2Darm-2Dkernel&d=DwIDaQ&c=nKjWec2b6R0mOyP az7xtfQ&r=eTeNTLEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=2NzkT9KLO26k efUOw2nIeSeRnJVZLxEiBXqEoRvDQ0ueww6n4YaXWgAN1uCJX20o&s=nczACS_2jERbA -c4Gfar0-HTA4PtvZdJmsBv8jhW8G0&e=_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Lorenzo Pieralisi
maintainerQuoted 12 lines · Elad Nachman
Hi,
Regarding number #1 :
The H/W erratum means strong ordering mode is not properly implemented in the H/W. The workaround is to setDIS_ORD_CHKto disable the ordered check, to prevent the issues Pali mentioned. Leaving it without the workaround means encountering the crashes Pali mentioned. The H/W was originally designed to have the strong ordering mode, per what is expected of PCI Express, but due to this erratum, it cannot be supported by the taped-out H/W. There is nothing to do about it now in the H/W, as this is a years old SOC. Software wise the only option is the workaround above, or to accept sporadic crashes of the kernel.
It can fix a driver, it can break another one - the PCI ordering model is broken one way or another.
I have no way to determine what setting DIS_ORD_CHK does and what it implies.
What I know is that we are sending this patch to stable kernels and I want to make sure a) it was thoroughly tested on several PCI endpoints and b) it is properly documented.
Quoted 6 lines
The impact of the workaround is that the DMA done status could be set before the last data is written to the host memory, causing the host to read incomplete data. This is the reason why I recommended to add a memory barrier. If this is bothering, the memory barrier can be removed from the patch, although I suggest to thoroughly test the data integrity if implementing the patch without the memory barrier.
It is not bothering, it is that every mb() must be explained in depth.
We are adding an mb() in the *controller* interrupt handler.
I assume this is because that's where the DMA completion IRQ is raised and there we issue an mb() to make sure that endpoint writes into memory completed.
If that's correct, I have no idea how the barrier plays together with DIS_ORD_CHK above. The mb() has no effect on PCI writes queued in the PCI bus, so it may be a plaster but not a fix.
I would like to have a full description of what that mb() is supposed to fix - in details.
When we agree it is fixing something we can add a descriptive comment with the mb() and merge the code.
Thanks,
Lorenzo
Quoted 170 lines
FYI,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Tuesday, October 4, 2022 11:40 AM
To: Elad Nachman <[email protected]>; Lorenzo Pieralisi <[email protected]>
Cc: Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion Timeout+ Elad
Could you please look at Lorenzo's comments and help with this fix?
Quoted 16 lines · On Tuesday 04 October 2022 10:02:32 Lorenzo Pieralisi wrote:
Quoted 2 lines · Pali Rohár
Lorenzo, is something more needed for this patch? As it workarounds crashing it is really needed to have it in mainline and backports.
Yes, a clear explanation from Marvell about what this is actually fixing - it took me a while to go through the whole thread but I still don't understand what this patch actually does and why.
An Erratum workaround (if there is any) should define and explain a SW workaround.
(1) Bjorn's concerns in relation to PCI memory model weren't addressed
(2) We don't add undocumented memory barriers to the kernel to "minimize risks". Either we fix a bug or we don't. If we do, write that down and document why the barrier is there and the issue it solves.Quoted 134 lines
I understand that basically you are reverse engineering a HW bug but I am afraid we can't fix the kernel this way - more so with patches going to be backported to stable kernels.
Lorenzo
Quoted 128 lines · On Wednesday 28 September 2022 14:05:10 Elad Nachman wrote:
Quoted 109 lines
Reviewed-by Elad Nachman <[email protected]>Thanks,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Monday, September 26, 2022 3:35 PM
To: Elad Nachman <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; Rob Herring <[email protected]>; [email protected]; [email protected]; [email protected]; Gregory Clement <[email protected]>; Marek Behún <[email protected]>; Remi Pommarel <[email protected]>; Xogium <[email protected]>; Tomasz Maciej Nowak <[email protected]>
Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion TimeoutExternal Email
------------------------------------------------------------------
---- Hello Elad, could you please review this patch? I have
implemented it according your instructions, including that full memory barrier as you described.Quoted 80 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and
Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #:
251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.As a workaround for this erratum,
DIS_ORD_CHKflag in Debug Mux Control register must be set. It disables the ordering check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize the risk for the race condition documented in the Erratum between the DMA done status reading and the completion of writing to the host memory.
More details about this issue and suggested workarounds are in discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel
.org_l inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB425 1.namp rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r =eTeNT LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONCZM iFDX72 MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME8j NiRgrq
4SDYvqqT0TA&e=It was reported that enabling this workaround fixes instability issues and "Unhandled fault" errors when using 60 GHz WiFi 802.11ad card with Qualcomm
QCA6335 chip under significant load which were caused by
interrupt status stuck in the outbound CMPLT queue traced back to this erratum.This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI hostcontroller
driver")
Cc: [email protected]--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +7
-0@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status;Quoted 9 lines
Quoted 9 lines
drivers/pci/controller/pci-aardvark.c +3
-1+/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ; --2.20.1
_______________________________________________
linux-arm-kernel mailing list [email protected]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.
org_mailman_listinfo_linux-2Darm-2Dkernel&d=DwIDaQ&c=nKjWec2b6R0mOyP az7xtfQ&r=eTeNTLEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=2NzkT9KLO26k efUOw2nIeSeRnJVZLxEiBXqEoRvDQ0ueww6n4YaXWgAN1uCJX20o&s=nczACS_2jERbA -c4Gfar0-HTA4PtvZdJmsBv8jhW8G0&e=_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi,
DIS_ORD_CHK - Disable Ordering Checks.
Setting this bit to 1 disables the ordering check in the core between Completions and Posted requests received from the link
Testing - I think Pali has 3700 boards, I don't have any.
The memory barrier is delaying the ability of the host to access the memory before the update is completed. This is not a bullet proof fix, it is just meant as a way to reduce the probability of this happening. Hence it is optional; I think it is better to have it, but it is not part of the original workaround.
Once again - if the above explanation is not satisfactory, the patch can be reposted without the memory barrier.
FYI,
Elad.
-----Original Message-----
From: Lorenzo Pieralisi <[email protected]>
Sent: Friday, October 21, 2022 11:26 AM
To: Elad Nachman <[email protected]>
Cc: Pali Rohár <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion Timeout
Quoted 12 lines · Elad Nachman
Hi,
Regarding number #1 :
The H/W erratum means strong ordering mode is not properly implemented in the H/W. The workaround is to setDIS_ORD_CHKto disable the ordered check, to prevent the issues Pali mentioned. Leaving it without the workaround means encountering the crashes Pali mentioned. The H/W was originally designed to have the strong ordering mode, per what is expected of PCI Express, but due to this erratum, it cannot be supported by the taped-out H/W. There is nothing to do about it now in the H/W, as this is a years old SOC. Software wise the only option is the workaround above, or to accept sporadic crashes of the kernel.
It can fix a driver, it can break another one - the PCI ordering model is broken one way or another.
I have no way to determine what setting DIS_ORD_CHK does and what it implies.
What I know is that we are sending this patch to stable kernels and I want to make sure a) it was thoroughly tested on several PCI endpoints and b) it is properly documented.
Quoted 6 lines
The impact of the workaround is that the DMA done status could be set before the last data is written to the host memory, causing the host to read incomplete data. This is the reason why I recommended to add a memory barrier. If this is bothering, the memory barrier can be removed from the patch, although I suggest to thoroughly test the data integrity if implementing the patch without the memory barrier.
It is not bothering, it is that every mb() must be explained in depth.
We are adding an mb() in the *controller* interrupt handler.
I assume this is because that's where the DMA completion IRQ is raised and there we issue an mb() to make sure that endpoint writes into memory completed.
If that's correct, I have no idea how the barrier plays together with DIS_ORD_CHK above. The mb() has no effect on PCI writes queued in the PCI bus, so it may be a plaster but not a fix.
I would like to have a full description of what that mb() is supposed to fix - in details.
When we agree it is fixing something we can add a descriptive comment with the mb() and merge the code.
Thanks,
Lorenzo
Quoted 185 lines
FYI,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Tuesday, October 4, 2022 11:40 AM
To: Elad Nachman <[email protected]>; Lorenzo Pieralisi <[email protected]>
Cc: Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion Timeout+ Elad
Could you please look at Lorenzo's comments and help with this fix?
Quoted 17 lines · On Tuesday 04 October 2022 10:02:32 Lorenzo Pieralisi wrote:
Quoted 2 lines · Pali Rohár
Lorenzo, is something more needed for this patch? As it workarounds crashing it is really needed to have it in mainline and backports.
Yes, a clear explanation from Marvell about what this is actually fixing - it took me a while to go through the whole thread but I still don't understand what this patch actually does and why.
An Erratum workaround (if there is any) should define and explain a SW workaround.
(1) Bjorn's concerns in relation to PCI memory model weren't addressed
(2) We don't add undocumented memory barriers to the kernel to "minimize risks". Either we fix a bug or we don't. If we do, write that down and document why the barrier is there and the issue it solves.Quoted 143 lines
I understand that basically you are reverse engineering a HW bug but I am afraid we can't fix the kernel this way - more so with patches going to be backported to stable kernels.
Lorenzo
Quoted 137 lines · On Wednesday 28 September 2022 14:05:10 Elad Nachman wrote:
Quoted 115 lines
Reviewed-by Elad Nachman <[email protected]>Thanks,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Monday, September 26, 2022 3:35 PM
To: Elad Nachman <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; Rob Herring <[email protected]>; [email protected]; [email protected]; [email protected]; Gregory Clement <[email protected]>; Marek Behún <[email protected]>; Remi Pommarel <[email protected]>; Xogium <[email protected]>; Tomasz Maciej Nowak <[email protected]>
Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion TimeoutExternal Email
----------------------------------------------------------------
--
---- Hello Elad, could you please review this patch? I have
implemented it according your instructions, including that full memory barrier as you described.Quoted 85 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and
Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #:
251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.As a workaround for this erratum,
DIS_ORD_CHKflag in Debug
Mux Control register must be set. It disables the ordering
check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the
beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize
the risk for the race condition documented in the Erratum
between the DMA done status reading and the completion of writing to the host memory.More details about this issue and suggested workarounds are in discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kern
el
.org_l inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB4 25
1.namp rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ &r
=eTeNT LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONC ZM
iFDX72 MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME 8j
NiRgrq
4SDYvqqT0TA&e=It was reported that enabling this workaround fixes
instability issues and "Unhandled fault" errors when using 60
GHz WiFi 802.11ad card with Qualcomm
QCA6335 chip under significant load which were caused by
interrupt status stuck in the outbound CMPLT queue traced back to this erratum.This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI hostcontroller
driver")
Cc: [email protected]--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +7
-0@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status;Quoted 9 lines
Quoted 9 lines
drivers/pci/controller/pci-aardvark.c +3
-1+/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ; --2.20.1
_______________________________________________
linux-arm-kernel mailing list [email protected]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.
org_mailman_listinfo_linux-2Darm-2Dkernel&d=DwIDaQ&c=nKjWec2b6R0mO yP az7xtfQ&r=eTeNTLEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=2NzkT9KLO2 6k efUOw2nIeSeRnJVZLxEiBXqEoRvDQ0ueww6n4YaXWgAN1uCJX20o&s=nczACS_2jER bA
-c4Gfar0-HTA4PtvZdJmsBv8jhW8G0&e=_______________________________________________
linux-arm-kernel mailing list
[email protected]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.or
g_mailman_listinfo_linux-2Darm-2Dkernel&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=eTeNTLEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=-FeSXS0gjIWGU0Oxsu2v1CLdlQI6QC3Egv2XQPD76Ng-wAeAtBrOnqEsHp9dmmko&s=OEUpcIK-9uWHm2WsP02xlH1AXc90FoSVzklrfHhQgPY&e=
Lorenzo Pieralisi
maintainerQuoted 7 lines · Elad Nachman
Hi,
DIS_ORD_CHK- Disable Ordering Checks.
Setting this bit to 1 disables the ordering check in the core between Completions and Posted requests received from the link
Quoted 5 lines
Testing - I think Pali has 3700 boards, I don't have any.
The memory barrier is delaying the ability of the host to access the memory before the update is completed. This is not a bullet proof fix, it is just meant as a way to reduce the probability of this happening. Hence it is optional; I think it is better to have it, but it is not part of the original workaround.
Once again - if the above explanation is not satisfactory, the patch can be reposted without the memory barrier.
It is not satisfactory so it should be dropped from the patch.
Thanks,
Lorenzo
Quoted 242 lines
FYI,
Elad.
-----Original Message-----
From: Lorenzo Pieralisi <[email protected]>
Sent: Friday, October 21, 2022 11:26 AM
To: Elad Nachman <[email protected]>
Cc: Pali Rohár <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion TimeoutQuoted 12 lines · Elad Nachman
Hi,
Regarding number #1 :
The H/W erratum means strong ordering mode is not properly implemented in the H/W. The workaround is to setDIS_ORD_CHKto disable the ordered check, to prevent the issues Pali mentioned. Leaving it without the workaround means encountering the crashes Pali mentioned. The H/W was originally designed to have the strong ordering mode, per what is expected of PCI Express, but due to this erratum, it cannot be supported by the taped-out H/W. There is nothing to do about it now in the H/W, as this is a years old SOC. Software wise the only option is the workaround above, or to accept sporadic crashes of the kernel.It can fix a driver, it can break another one - the PCI ordering model is broken one way or another.
I have no way to determine what setting
DIS_ORD_CHKdoes and what it implies.What I know is that we are sending this patch to stable kernels and I want to make sure a) it was thoroughly tested on several PCI endpoints and b) it is properly documented.
Quoted 6 lines
The impact of the workaround is that the DMA done status could be set before the last data is written to the host memory, causing the host to read incomplete data. This is the reason why I recommended to add a memory barrier. If this is bothering, the memory barrier can be removed from the patch, although I suggest to thoroughly test the data integrity if implementing the patch without the memory barrier.
It is not bothering, it is that every
mb()must be explained in depth.We are adding an
mb()in the *controller* interrupt handler.I assume this is because that's where the DMA completion IRQ is raised and there we issue an
mb()to make sure that endpoint writes into memory completed.If that's correct, I have no idea how the barrier plays together with
DIS_ORD_CHKabove. Themb()has no effect on PCI writes queued in the PCI bus, so it may be a plaster but not a fix.I would like to have a full description of what that
mb()is supposed to fix - in details.When we agree it is fixing something we can add a descriptive comment with the
mb()and merge the code.Thanks,
LorenzoQuoted 180 lines
FYI,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Tuesday, October 4, 2022 11:40 AM
To: Elad Nachman <[email protected]>; Lorenzo Pieralisi <[email protected]>
Cc: Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion Timeout+ Elad
Could you please look at Lorenzo's comments and help with this fix?
Quoted 17 lines · On Tuesday 04 October 2022 10:02:32 Lorenzo Pieralisi wrote:
Quoted 2 lines · Pali Rohár
Lorenzo, is something more needed for this patch? As it workarounds crashing it is really needed to have it in mainline and backports.
Yes, a clear explanation from Marvell about what this is actually fixing - it took me a while to go through the whole thread but I still don't understand what this patch actually does and why.
An Erratum workaround (if there is any) should define and explain a SW workaround.
(1) Bjorn's concerns in relation to PCI memory model weren't addressed
(2) We don't add undocumented memory barriers to the kernel to "minimize risks". Either we fix a bug or we don't. If we do, write that down and document why the barrier is there and the issue it solves.Quoted 143 lines
I understand that basically you are reverse engineering a HW bug but I am afraid we can't fix the kernel this way - more so with patches going to be backported to stable kernels.
Lorenzo
Quoted 137 lines · On Wednesday 28 September 2022 14:05:10 Elad Nachman wrote:
Quoted 115 lines
Reviewed-by Elad Nachman <[email protected]>Thanks,
Elad.
-----Original Message-----
From: Pali Rohár <[email protected]>
Sent: Monday, September 26, 2022 3:35 PM
To: Elad Nachman <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; Krzysztof Wilczyński <[email protected]>; Rob Herring <[email protected]>; [email protected]; [email protected]; [email protected]; Gregory Clement <[email protected]>; Marek Behún <[email protected]>; Remi Pommarel <[email protected]>; Xogium <[email protected]>; Tomasz Maciej Nowak <[email protected]>
Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion TimeoutExternal Email
----------------------------------------------------------------
--
---- Hello Elad, could you please review this patch? I have
implemented it according your instructions, including that full memory barrier as you described.Quoted 85 lines · On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote:
Marvell Armada 3700 Functional Errata, Guidelines, and
Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #:
251), that PCIe IP does not support a strong-ordered model for inbound posted vs. outbound completion.As a workaround for this erratum,
DIS_ORD_CHKflag in Debug
Mux Control register must be set. It disables the ordering
check in the core between Completions and Posted requests received from the link.Marvell also suggests to do full memory barrier at the
beginning of aardvark summary interrupt handler before calling interrupt handlers of endpoint drivers in order to minimize
the risk for the race condition documented in the Erratum
between the DMA done status reading and the completion of writing to the host memory.More details about this issue and suggested workarounds are in discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kern
el
.org_l inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB4 25
1.namp rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ &r
=eTeNT LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONC ZM
iFDX72 MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME 8j
NiRgrq
4SDYvqqT0TA&e=It was reported that enabling this workaround fixes
instability issues and "Unhandled fault" errors when using 60
GHz WiFi 802.11ad card with Qualcomm
QCA6335 chip under significant load which were caused by
interrupt status stuck in the outbound CMPLT queue traced back to this erratum.This workaround fixes also kernel panic triggered after some minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip:
Internal error: synchronous external abort: 96000210 [#1] SMP Kernel panic - not syncing: Fatal exception in interruptSigned-off-by Thomas Petazzoni <[email protected]>Signed-off-by Pali Rohár <[email protected]>Fixes 8c39d710363c ("PCI: aardvark: Add Aardvark PCI hostcontroller
driver")
Cc: [email protected]--- drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/pci/controller/pci-aardvark.c +7
-0@@ -210,6 +210,8 @@ enum { };#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) +#define DIS_ORD_CHK BIT(30)/* PCIe core controller registers */ #define CTRL_CORE_BASE_ADDR 0x18000@@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)PCIE_CORE_CTRL2_TD_ENABLE ;advk_writel (pcie, reg,PCIE_CORE_CTRL2_REG ); +/* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ + reg =advk_readl (pcie,DEBUG_MUX_CTRL_REG ); + reg |=DIS_ORD_CHK ; +advk_writel (pcie, reg,DEBUG_MUX_CTRL_REG ); +/* Set lane X1 */ reg =advk_readl (pcie,PCIE_CORE_CTRL0_REG ); reg &= ~LANE_CNT_MSK ;@@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg)struct advk_pcie *pcie = arg;u32 status;Quoted 9 lines
Quoted 9 lines
drivers/pci/controller/pci-aardvark.c +3
-1+/* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ +mb (); + status =advk_readl (pcie,HOST_CTRL_INT_STATUS_REG );if (!(status &PCIE_IRQ_CORE_INT ))return IRQ_NONE ; --2.20.1
_______________________________________________
linux-arm-kernel mailing list [email protected]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.
org_mailman_listinfo_linux-2Darm-2Dkernel&d=DwIDaQ&c=nKjWec2b6R0mO yP az7xtfQ&r=eTeNTLEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=2NzkT9KLO2 6k efUOw2nIeSeRnJVZLxEiBXqEoRvDQ0ueww6n4YaXWgAN1uCJX20o&s=nczACS_2jER bA
-c4Gfar0-HTA4PtvZdJmsBv8jhW8G0&e=Quoted 5 lines
_______________________________________________
linux-arm-kernel mailing list
[email protected]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.or
g_mailman_listinfo_linux-2Darm-2Dkernel&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=eTeNTLEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=-FeSXS0gjIWGU0Oxsu2v1CLdlQI6QC3Egv2XQPD76Ng-wAeAtBrOnqEsHp9dmmko&s=OEUpcIK-9uWHm2WsP02xlH1AXc90FoSVzklrfHhQgPY&e=_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel