PCIe: limit Max Read Request Size on i.MX to 512 bytes
Subsystems (4)
PATCH 1/1 PCIe: limit Max Read Request Size on i.MX to 512 bytes
DWC PCIe controller imposes limits on the Read Request Size that it can handle. For i.MX6 family it's fixed at 512 bytes by default.
If a memory read larger than the limit is requested, the CPU responds with Completer Abort (CA) (on i.MX6 Unsupported Request (UR) is returned instead due to a design error).
The i.MX6 documentation states that the limit can be changed by writing to the PCIE_PL_MRCCR0 register, however there is a fixed (and undocumented) maximum (CX_REMOTE_RD_REQ_SIZE constant). Tests indicate that values larger than 512 bytes don't work, though.
This patch makes the RTL8111 work on i.MX6.
drivers/pci/Kconfig +3 -0
@@ -34,6 +34,9 @@ config PCI_DOMAINS_GENERIC config PCI_SYSCALL bool +config NEED_PCIE_MAX_MRRS + bool + source "drivers/pci/pcie/Kconfig" config PCI_MSI
drivers/pci/controller/dwc/Kconfig +1 -0
@@ -98,6 +98,7 @@ config PCI_IMX6 depends on ARCH_MXC || COMPILE_TEST depends on PCI_MSI_IRQ_DOMAIN select PCIE_DW_HOST + select NEED_PCIE_MAX_MRRS config PCIE_SPEAR13XX bool "STMicroelectronics SPEAr PCIe controller"
drivers/pci/controller/dwc/pci-imx6.c +1 -0
@@ -1148,6 +1148,7 @@ static int imx6_pcie_probe(struct platform_device *pdev) imx6_pcie->vph =NULL ; } + max_pcie_mrrs =512 ;platform_set_drvdata (pdev, imx6_pcie); ret =imx6_pcie_attach_pd (dev);
drivers/pci/pci.c +9 -0
@@ -112,6 +112,10 @@ enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_PEER2PEER;enum pcie_bus_config_types pcie_bus_config =PCIE_BUS_DEFAULT ;#endif +#ifdef CONFIG_NEED_PCIE_MAX_MRRS +u16 max_pcie_mrrs =4096 ;// no limit +#endif +/* * The default CLS is used if arch didn't set CLS explicitly and not * all pci devices agree on the same value. Arch can override either
@@ -5816,6 +5820,11 @@ int pcie_set_readrq(struct pci_dev *dev, int rq) rq = mps; } +#ifdef CONFIG_NEED_PCIE_MAX_MRRS +if (rq > max_pcie_mrrs) + rq = max_pcie_mrrs; +#endif + v = (ffs (rq) -8 ) <<12 ; ret = pcie_capability_clear_and_set_word(dev, PCI_EXP_DEVCTL,
include/linux/pci.h +1 -0
@@ -994,6 +994,7 @@ enum pcie_bus_config_types { };extern enum pcie_bus_config_types pcie_bus_config; +extern u16 max_pcie_mrrs;extern struct bus_type pci_bus_type;
Signature · 5 lines
Krzysztof "Chris" Hałasa Sieć Badawcza Łukasiewicz Przemysłowy Instytut Automatyki i Pomiarów PIAP Al. Jerozolimskie 202, 02-486 Warszawa
Hi Krzysztof,
[...]
Quoted 1 line
This patch makes the RTL8111 work on i.MX6.
Would it be possible to implement this particular MRRS fix as a quirk only for the i.MX6 controller? Unless this is something that we need in the core, a quirk would be preferred over something that changes the PCI core.
An example of such quirk might be the one currently implemented for the Loongson controller, as per:
https://elixir.bootlin.com/linux/v5.14-rc5/source/drivers/pci/controller/pci-loongson.c#L63
Krzysztof
Krzysztof, :-)
Quoted 4 lines
Would it be possible to implement this particular MRRS fix as a quirk only for the i.MX6 controller? Unless this is something that we need in the core, a quirk would be preferred over something that changes the PCI core.
I have briefly considered it, but I think it would be *much* more complicated and error-prone. It also appears that there are more platforms which need it - the old CNS3xxx, which currently subverts the PCIE_BUS_PEER2PEER, the loongson, keystone, maybe all DWC PCIe. Multiplication of the "quirk" code doesn't really look good to me.
TBH I don't think of this as of a "quirk" - all systems have MRRS limits, it just happens that these ones have their limit lower than 4096 bytes. This isn't a limitation of a particular PCIe device, this is a common limit of the whole system.
Also I'm not exactly sure the loongson fixup is complete. It's only done at pci-enable*() time (e.g. for devices which have bigger MRRS after power-up), while e.g. the r8169 driver changes MRRS well after pci-enable*().
This means it needs to stay in/below pcie_get_readrq(), and while it could mean going to ops->write*, it would be a real mess parsing the devices, PCIE capabilities etc.
Now it's basically a few lines in a seldom called routine in pci.c, and the loongson case (and others) can be made simpler (and really fixed) as well.
Signature · 5 lines
Krzysztof "Chris" Hałasa Sieć Badawcza Łukasiewicz Przemysłowy Instytut Automatyki i Pomiarów PIAP Al. Jerozolimskie 202, 02-486 Warszawa
Rob Herring
reviewerQuoted 22 lines · Krzysztof Hałasa
DWC PCIe controller imposes limits on the Read Request Size that it can handle. For i.MX6 family it's fixed at 512 bytes by default.
If a memory read larger than the limit is requested, the CPU responds with Completer Abort (CA) (on i.MX6 Unsupported Request (UR) is returned instead due to a design error).
The i.MX6 documentation states that the limit can be changed by writing to the
PCIE_PL_MRCCR0register, however there is a fixed (and undocumented) maximum (CX_REMOTE_RD_REQ_SIZEconstant). Tests indicate that values larger than 512 bytes don't work, though.This patch makes the RTL8111 work on i.MX6.
Signed-off-by Krzysztof Hałasa <[email protected]>drivers/pci/Kconfig +0
-0
drivers/pci/Kconfig config PCI_DOMAINS_GENERIC · line 37
Quoted 4 lines
drivers/pci/Kconfig +1
-0@@ -34,6 +34,9 @@ config PCI_DOMAINS_GENERIC config PCI_SYSCALL bool +config NEED_PCIE_MAX_MRRS
We don't need a config option for this. It's not much code and it will effectively always be enabled with multi-platform kernels.
Rob
Rob,
Quoted 9 lines · Rob Herring
drivers/pci/Kconfig config PCI_DOMAINS_GENERIC · line 37
Quoted 6 lines
+++ b/drivers/pci/Kconfig
drivers/pci/Kconfig +1
-0@@ -34,6 +34,9 @@ config PCI_DOMAINS_GENERIC config PCI_SYSCALL bool +config NEED_PCIE_MAX_MRRSWe don't need a config option for this. It's not much code and it will effectively always be enabled with multi-platform kernels.
But... non-ARM kernels?
Then perhaps #if CONFIG_ARM?
Signature · 5 lines
Krzysztof "Chris" Hałasa Sieć Badawcza Łukasiewicz Przemysłowy Instytut Automatyki i Pomiarów PIAP Al. Jerozolimskie 202, 02-486 Warszawa
Quoted 13 lines · Krzysztof Hałasa
Krzysztof, :-)
Quoted 4 lines
Would it be possible to implement this particular MRRS fix as a quirk only for the i.MX6 controller? Unless this is something that we need in the core, a quirk would be preferred over something that changes the PCI core.
I have briefly considered it, but I think it would be *much* more complicated and error-prone. It also appears that there are more platforms which need it - the old CNS3xxx, which currently subverts the
PCIE_BUS_PEER2PEER, the loongson, keystone, maybe all DWC PCIe. Multiplication of the "quirk" code doesn't really look good to me.
Quoted 4 lines
TBH I don't think of this as of a "quirk" - all systems have MRRS limits, it just happens that these ones have their limit lower than 4096 bytes. This isn't a limitation of a particular PCIe device, this is a common limit of the whole system.
Do you have a reference for this? I don't see anything in the PCIe spec that suggests platforms must limit MRRS, and it seems that only these ARM-related controllers have this issue. If there *is* a platform connection here, we'll need some way to discover it, e.g., an ACPI _DSM method or similar.
The only guidance in the spec about setting MRRS is that:
- Software must set Max_Read_Request_Size of an isochronous-configured device with a value that does not exceed the Max_Payload_Size set for the device (PCIe r5.0, sec 6.3.4.1)
- The Max_Read_Request_Size mechanism allows improved control of bandwidth allocation in systems where Quality of Service (QoS) is important for the target applications. For example, an arbitration scheme based on counting Requests (and not the sizes of those Requests) provides imprecise bandwidth allocation when some Requesters use much larger sizes than others. The Max_Read_Request_Size mechanism can be used to force more uniform allocation of bandwidth, by restricting the upper size of Read Requests (sec 7.5.3.4 implementation note)
Quoted 8 lines · Bjorn Helgaas
Quoted 4 lines
TBH I don't think of this as of a "quirk" - all systems have MRRS limits, it just happens that these ones have their limit lower than 4096 bytes. This isn't a limitation of a particular PCIe device, this is a common limit of the whole system.
Do you have a reference for this? I don't see anything in the PCIe spec that suggests platforms must limit MRRS, and it seems that only these ARM-related controllers have this issue.
I meant there is always a limit - isn't Max_Read_Request_Size a limit? Device Control Register (Offset 08h) Bit Location 14:12 Max_Read_Request_Size allows for max 4096 bytes, though two values are reserved, so there is room for some easy extension.
- non-ARM (non-DWC?) systems are limited to 4096 bytes
- DWC-based systems are limited to 128, 256, 512 bytes (are there 4096-byte ones?)
That's how I understand it, please correct me if I'm wrong.
Signature · 5 lines
Krzysztof "Chris" Hałasa Sieć Badawcza Łukasiewicz Przemysłowy Instytut Automatyki i Pomiarów PIAP Al. Jerozolimskie 202, 02-486 Warszawa
Richard Zhu
maintainerHi Krzysztof:
Regarding my understanding, that there should be decomposition operations if the
Max_Read_Request_Size is larger than the Max_Payload_size specified by RC port.
The bit0 of AMBA Multiple Outbound Decomposed NP Sub-Requests Control Register(Offset:0x700 + 0x24)
had been set to be 1b1 in default.
Note: The description of this bit.
Enable AMBA Multiple Outbound Decomposed NP Sub- Requests.
This bit when set to '0' disables the possibility of having multiple outstanding non-posted requests that were derived from decomposition of an outbound AMBA request. See Supported AXI Burst Operations for more details. You should not clear this register unless your application master is requesting an amount of read data greater than Max_Read_Request_Size, and the remote device (or switch) is reordering completions that have different tags
Quoted 24 lines
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, August 16, 2021 1:19 PM
To: Bjorn Helgaas <[email protected]>
Cc: Krzysztof Wilczyński <[email protected]>; Bjorn Helgaas <[email protected]>; [email protected]; Artem Lapkin <[email protected]>; Neil Armstrong <[email protected]>; Huacai Chen <[email protected]>; Rob Herring <[email protected]>; Lorenzo Pieralisi <[email protected]>; Richard Zhu <[email protected]>; Lucas Stach <[email protected]>; [email protected]
Subject: Re: [PATCH] PCIe: limit Max Read Request Size on i.MX to 512 bytesQuoted 8 lines · Bjorn Helgaas
Quoted 4 lines
TBH I don't think of this as of a "quirk" - all systems have MRRS limits, it just happens that these ones have their limit lower than 4096 bytes. This isn't a limitation of a particular PCIe device, this is a common limit of the whole system.
Do you have a reference for this? I don't see anything in the PCIe spec that suggests platforms must limit MRRS, and it seems that only these ARM-related controllers have this issue.
Quoted 6 lines
I meant there is always a limit - isn't
Max_Read_Request_Sizea limit? Device Control Register (Offset 08h) Bit Location 14:12Max_Read_Request_Sizeallows for max 4096 bytes, though two values are reserved, so there is room for some easy extension.- non-ARM (non-DWC?) systems are limited to 4096 bytes
[Richard] Regarding to the descriptions listed above, I think that there should no limitations of the Max_payload_size of RC port.
Quoted 2 lines
- DWC-based systems are limited to 128, 256, 512 bytes (are there 4096-byte ones?)
[Richard] The Max_payload_size can be configured from 128bytes to 4KB when integrate the DWC IP into the SOC. On i.MX6 PCIe, this parameter is fixed set to 512 bytes.
BR
Richard
Quoted 8 lines
That's how I understand it, please correct me if I'm wrong. --
Krzysztof "Chris" HałasaSieć Badawcza Łukasiewicz
Przemysłowy Instytut Automatyki i Pomiarów PIAP Al. Jerozolimskie 202, 02-486 Warszawa
Hi Richard,
Please correct me if I got something wrong:
Quoted 3 lines
Regarding my understanding, that there should be decomposition operations if the
Max_Read_Request_Sizeis larger than theMax_Payload_sizespecified by RC port.
I think this means that, for example, a memory read request (a single short physical TLP packet on PCIe, from the peripheral device to the CPU) can request more data than Max_Payload_size (128 bytes on i.MX6). In such case, up to 4 "completion" physical TLP packets are returned by the CPU (up to 512 bytes, with each individual TLP up to 128 bytes as per Max_Payload_size).
The device can't request more than 512 bytes, though - the CPU would not service such request.
Quoted 8 lines
The bit0 of AMBA Multiple Outbound Decomposed NP Sub-Requests Control Register(Offset:0x700 +
0x24)
had been set to be 1b1 in default.Note: The description of this bit.
Enable AMBA Multiple Outbound Decomposed NP Sub- Requests.
This bit when set to '0' disables the possibility of having multiple outstanding non-posted requests that were derived from decomposition of an outbound AMBA request. See Supported AXI Burst Operations for more details.
I think the above means that - when I disable the bit in question - I can't issue memory read requests longer than 128 bytes (max payload size).
This is not exactly clear to me:
Quoted 2 lines
You should not clear this register unless your application master is requesting an amount of read data greater than
Max_Read_Request_Size,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Is "completing" such a request at all possible?
The device shouldn't request more data than its (not CPU's) Max_Read_Request_Size. I. e. if 512 is written to RTL8111's Max_Read_Request_Size, then the Realtek chip will request max 512 bytes at a time.
Quoted 2 lines
and the remote device (or switch) is reordering completions that have different tags
Fortunately, such completions don't seem to be reordered.
However, I'm not sure how setting a bit in the CPU registers could help here. I think the only way *IF* the completions were reordered would be setting MRRS = MPS (= 128 bytes on i.MX6) - so there is nothing that could be reordered.
Signature · 5 lines
Krzysztof "Chris" Hałasa Sieć Badawcza Łukasiewicz Przemysłowy Instytut Automatyki i Pomiarów PIAP Al. Jerozolimskie 202, 02-486 Warszawa