PCI: dwc: Warn only for non-prefetchable memory resource size >4GB
PATCH 1/1 PCI: dwc: Warn only for non-prefetchable memory resource size >4GB
commit 9e73fa02aa009 ("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.
--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/controller/dwc/pcie-designware-host.c +2 -1
@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) +if (upper_32_bits (mem_size) && + !(win->res->flags &IORESOURCE_PREFETCH ))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;
Signature · 1 line
2.17.1
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]
Quoted 27 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +2
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) +if (upper_32_bits (mem_size) && + !(win->res->flags &IORESOURCE_PREFETCH ))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;Signature · 1 line
2.17.1
Quoted 14 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 10 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 14 lines
Quoted 5 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-0drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 9 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) +if (upper_32_bits (mem_size) && + !(win->res->flags &IORESOURCE_PREFETCH ))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;
That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Alan ? I want to understand what's the best course of action before merging these patches.
Lorenzo
Quoted 28 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 24 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 20 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&
Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Yeah. I'm all for a warning if the size is larger than 4GB in case of
non-prefetchable window as one of the ATU outbound translation channels is being used, but, we are not employing any ATU outbound translation channel for prefetchable window and they can be greater than 4GB in size for all right reasons. So, logging a warning for prefetchable region doesn't seem correct to me. Please let me know if my understanding is wrong.
- Vidya Sagar
Quoted 6 lines
Alan ? I want to understand what's the best course of action before merging these patches.
Lorenzo
Quoted 39 lines · Vidya Sagar
Quoted 28 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 24 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 20 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Can DWC maintainers chime in and clarify please ?
Quoted 3 lines
prefetchable window and they can be greater than 4GB in size for all right reasons. So, logging a warning for prefetchable region doesn't seem correct to me. Please let me know if my understanding is wrong.
I think your patch is wrong and it is applied on top of a patch that is wrong too, so I won't apply yours and it is likely I will revert Alan's because it seems to solve nothing (and warn spuriously).
It is time for people who maintain DWC please to speak up because I don't have the HW details required to make a judgment.
Lorenzo
Quoted 7 lines
- Vidya Sagar
Quoted 6 lines
Alan ? I want to understand what's the best course of action before merging these patches.
Lorenzo
Quoted 43 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 39 lines · Vidya Sagar
Quoted 36 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 14 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 10 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Quoted 14 lines
Quoted 5 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-0drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 9 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) +if (upper_32_bits (mem_size) && + !(win->res->flags &IORESOURCE_PREFETCH ))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 7 lines
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?I think so. As I see from the code, ATU's
Region-0 is used for config space translation
Region-1 is used for non-prefetchable memory translation Region-2 is used for I/O translation
So, there is no region reserved for translating prefetchable memory regions.
Quoted 7 lines
Quoted 2 lines
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Not Tegra driver but Tegra HW. Tegra HW doesn't need any ATU outbound translation for prefetchable (for that matter any 1-to-1 mapping to generate memory transactions on the PCIe bus). The Warning is still valid for both Tegra and other DWC based controllers for non-prefetchable memory translation.
Quoted 23 lines
Can DWC maintainers chime in and clarify please ?
Quoted 3 lines
prefetchable window and they can be greater than 4GB in size for all right reasons. So, logging a warning for prefetchable region doesn't seem correct to me. Please let me know if my understanding is wrong.
I think your patch is wrong and it is applied on top of a patch that is wrong too, so I won't apply yours and it is likely I will revert Alan's because it seems to solve nothing (and warn spuriously).
It is time for people who maintain DWC please to speak up because I don't have the HW details required to make a judgment.
Lorenzo
Quoted 7 lines
- Vidya Sagar
Quoted 6 lines
Alan ? I want to understand what's the best course of action before merging these patches.
Lorenzo
On Tue, May 19, 2020 at 10:38:39PM +0530, Vidya Sagar wrote:
[...]
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Quoted 7 lines
Quoted 3 lines
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?I think so. As I see from the code, ATU's
Region-0 is used for config space translation
Region-1 is used for non-prefetchable memory translation Region-2 is used for I/O translation
So, there is no region reserved for translating prefetchable memory regions.
I am confused. Code in dw_pcie_setup_rc() programs a memory region into ATU. What defines that memory as non-prefetchable ?
Code in dw_pcie_host_init() retrieves the bridge windows and there may be multiple memory regions.
How is it determined which one ends up in pp->mem ? DT parsing order ?
I reiterate the point - I need DWC maintainers to chime in to define how this must work.
I am not conviced that your patch is safe to apply and I need to understand how HW works.
Quoted 9 lines
Quoted 6 lines
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Not Tegra driver but Tegra HW. Tegra HW doesn't need any ATU outbound translation for prefetchable (for that matter any 1-to-1 mapping to generate memory transactions on the PCIe bus).
See above.
Lorenzo
Quoted 26 lines
The Warning is still valid for both Tegra and other DWC based controllers for non-prefetchable memory translation.
Quoted 23 lines
Can DWC maintainers chime in and clarify please ?
Quoted 3 lines
prefetchable window and they can be greater than 4GB in size for all right reasons. So, logging a warning for prefetchable region doesn't seem correct to me. Please let me know if my understanding is wrong.
I think your patch is wrong and it is applied on top of a patch that is wrong too, so I won't apply yours and it is likely I will revert Alan's because it seems to solve nothing (and warn spuriously).
It is time for people who maintain DWC please to speak up because I don't have the HW details required to make a judgment.
Lorenzo
Quoted 7 lines
- Vidya Sagar
Quoted 6 lines
Alan ? I want to understand what's the best course of action before merging these patches.
Lorenzo
Quoted 48 lines · Lorenzo Pieralisi
Quoted 39 lines · Vidya Sagar
Quoted 28 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 24 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 20 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 8 lines
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call pci_parse_request_of_pci_ranges(), which performs a simple validation for the IORESOURCE_MEM resource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the -EINVAL value.
In other words, to reach the code that Vidya is changing, it can be only if the resource is a non-prefetchable, any prefetchable resource will be blocked by the previous call, if I'm not mistaken.
Having this in mind, Vidya's change will not make the expected result aimed by him.
I don't see any problem by having resources larger than 4GB, from what I'm seeing in the databook there isn't any restricting related to that as long they don't consume the maximum space that is addressable by the system (depending on if they are 32-bit or 64-bit system address).
To be honest, I'm not seeing a system that could have this resource larger than 4GB, but it might exist some exception that I don't know of, that's why I accepted Alan's patch to warn the user that the resource exceeds the maximum for the 32 bits so that he can be aware that he *might* be consuming the maximum space addressable.
-Gustavo
Quoted 21 lines
Quoted 4 lines
prefetchable window and they can be greater than 4GB in size for all right reasons. So, logging a warning for prefetchable region doesn't seem correct to me. Please let me know if my understanding is wrong.
I think your patch is wrong and it is applied on top of a patch that is wrong too, so I won't apply yours and it is likely I will revert Alan's because it seems to solve nothing (and warn spuriously).
It is time for people who maintain DWC please to speak up because I don't have the HW details required to make a judgment.
Lorenzo
Quoted 7 lines
- Vidya Sagar
Quoted 6 lines
Alan ? I want to understand what's the best course of action before merging these patches.
Lorenzo
Hi Lorenzo,
I came across this issue when implementing a Linux NVMe endpoint function driver under the Linux PCI Endpoint Framework:
https://lwn.net/Articles/804369/
I needed to map up to 128GB of host memory using a single ATU window from the endpoint side because NVMe PRPs can be scattered all over host memory. In the process, I came across this 4GB limitation where the maximum size of memory that can be mapped is limited by what a u32 value can represent.
I submitted a separate patch to fix an undefined behavior that may also happen in dw_pcie_prog_outbound_atu_unroll() under some circumstances when the size of the memory being mapped is greater than what a u32 value can represent.
https://patchwork.kernel.org/patch/11469701/
The above patch has been accepted. However, the variable pp->mem_size in dw_pcie_host_init() is still a u32 whereas the value returned by resource_size() is u64. If the resource size has non-zero upper 32-bits, those upper 32-bits will be lost when assigning:
pp->mem_size = resource_size(pp->mem).
Since current callers seem happy with the existing 4GB implementation and fixing the u32 limit is beyond my available resources and has a long high-impact tail, a warning seemed to be a good choice to highlight this issue in case someone else decides to map a MEM region that is greater than 4GB.
Removing the warning will avoid such discussions. Without this warning, this limitation will go unnoticed and will only impact whoever has to deal with it. It cost me time to figure it out when I had an application that needed a region larger than 4GB. I figured the most I could do about it is to raise the issue by adding a warning.
Regards,
Alan
On Tue, May 19, 2020 at 10:08:54PM +0000, Gustavo Pimentel wrote:
[...]
Quoted 33 lines
Quoted 33 lines
Quoted 18 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.
That code checks if there is *at least* a non-prefetchable resource, that's all it does.
Quoted 3 lines
In other words, to reach the code that Vidya is changing, it can be only if the resource is a non-prefetchable, any prefetchable resource will be blocked by the previous call, if I'm not mistaken.
I think you are mistaken sorry.
Quoted 2 lines
Having this in mind, Vidya's change will not make the expected result aimed by him.
I think Vidya's patch does what he expects, the question is whether it is widely applicable to ALL DWC hosts, that's what I want to know.
Quoted 10 lines
I don't see any problem by having resources larger than 4GB, from what I'm seeing in the databook there isn't any restricting related to that as long they don't consume the maximum space that is addressable by the system (depending on if they are 32-bit or 64-bit system address).
To be honest, I'm not seeing a system that could have this resource larger than 4GB, but it might exist some exception that I don't know of, that's why I accepted Alan's patch to warn the user that the resource exceeds the maximum for the 32 bits so that he can be aware that he *might* be consuming the maximum space addressable.
I think it is most certainly a possibility to have > 4GB prefetchable address spaces so we ought to fix this for good. I still have to understand how the DWC host detects the memory region to be programmed into the ATU given that there is more than one but only 1 ATU memory region AFAICS.
Lorenzo
Quoted 58 lines · Gustavo Pimentel
Quoted 55 lines · Lorenzo Pieralisi
Quoted 39 lines · Vidya Sagar
Quoted 28 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 24 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 20 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.
That's not what the code is doing. pci_parse_request_of_pci_range() will traverse over the whole list of resources that it can find for the given host controller and checks whether one of the resources defines prefetch memory (note the res_valid |= ...). The error will only be returned if no prefetchable memory region was found.
dw_pcie_host_init() will then again traverse the list of resources and it will typically encounter two resource of type IORESOURCE_MEM, one for non-prefetchable memory and another for prefetchable memory.
Vidya's patch is to differentiate between these two resources and allow prefetchable memory regions to exceed sizes of 4 GiB.
That said, I wonder if there isn't a bigger problem at hand here. From looking at the code it doesn't seem like the DWC driver makes any distinction between prefetchable and non-prefetchable memory. Or at least it doesn't allow both to be stored in struct pcie_port.
Am I missing something? Or can anyone explain how we're programming the apertures for prefetchable vs. non-prefetchable memory? Perhaps this is what Vidya was referring to when he said: "we are not using an outbound ATU translation channel for prefetchable memory".
It looks to me like we're also getting partially lucky, or perhaps that is by design, in that Tegra194 defines PCI regions in the following
order: I/O, prefetchable memory, non-prefetchable memory. That means that the DWC core code will overwrite prefetchable memory data with that of non-prefetchable memory and hence the non-prefetchable region ends up stored in struct pcie_port and is then used to program the ATU outbound channel.
Quoted 6 lines
In other words, to reach the code that Vidya is changing, it can be only if the resource is a non-prefetchable, any prefetchable resource will be blocked by the previous call, if I'm not mistaken.
Having this in mind, Vidya's change will not make the expected result aimed by him.
Given the above I think it does. We've also seen this patch eliminate a warning that we were seeing before, so I think it has the intended effect.
Quoted 10 lines
I don't see any problem by having resources larger than 4GB, from what I'm seeing in the databook there isn't any restricting related to that as long they don't consume the maximum space that is addressable by the system (depending on if they are 32-bit or 64-bit system address).
To be honest, I'm not seeing a system that could have this resource larger than 4GB, but it might exist some exception that I don't know of, that's why I accepted Alan's patch to warn the user that the resource exceeds the maximum for the 32 bits so that he can be aware that he *might* be consuming the maximum space addressable.
I think it's pretty common to have this type of prefetchable memory region when you connect discrete GPUs to PCIe. It's not unusual for high-end GPUs to have 8 GiB or even more dedicated video memory and those will typically be mapped to a prefetchable memory region on the PCI device.
Thierry
Quoted 72 lines · Lorenzo Pieralisi
On Tue, May 19, 2020 at 10:08:54PM +0000, Gustavo Pimentel wrote:
[...]
Quoted 33 lines
Quoted 33 lines
Quoted 18 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.That code checks if there is *at least* a non-prefetchable resource, that's all it does.
Quoted 3 lines
In other words, to reach the code that Vidya is changing, it can be only if the resource is a non-prefetchable, any prefetchable resource will be blocked by the previous call, if I'm not mistaken.
I think you are mistaken sorry.
Quoted 2 lines
Having this in mind, Vidya's change will not make the expected result aimed by him.
I think Vidya's patch does what he expects, the question is whether it is widely applicable to ALL DWC hosts, that's what I want to know.
Quoted 10 lines
I don't see any problem by having resources larger than 4GB, from what I'm seeing in the databook there isn't any restricting related to that as long they don't consume the maximum space that is addressable by the system (depending on if they are 32-bit or 64-bit system address).
To be honest, I'm not seeing a system that could have this resource larger than 4GB, but it might exist some exception that I don't know of, that's why I accepted Alan's patch to warn the user that the resource exceeds the maximum for the 32 bits so that he can be aware that he *might* be consuming the maximum space addressable.
Quoted 5 lines
I think it is most certainly a possibility to have > 4GB prefetchable address spaces so we ought to fix this for good. I still have to understand how the DWC host detects the memory region to be programmed into the ATU given that there is more than one but only 1 ATU memory region AFAICS.
Probably best to wait for Vidya to confirm since I'm not altogether familiar with PCI on Tegra194, but looking at the DTS files and the Tegra194 TRM, the prefetchable memory regions are set to a range in 0x1200000000-0x1fffffffff which is a region of the address map that is reserved for "PCIe aperture for > 32-bit OS". Part of that is in use for non-prefetchable memory (and ends up being programmed into the ATU) whereas a much larger part is used for prefetchable memory and is not programmed anywhere, as far as I can tell.
But I think given that this is a designated region of the address map this is probably automatically redirected to the PCIe controller. What I don't know is if that's something Tegra-specific or whether all instantiations have something similar set up.
Thierry
Quoted 91 lines · Thierry Reding
Quoted 58 lines · Gustavo Pimentel
Quoted 55 lines · Lorenzo Pieralisi
Quoted 39 lines · Vidya Sagar
Quoted 28 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 24 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 20 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.That's not what the code is doing.
pci_parse_request_of_pci_range()will traverse over the whole list of resources that it can find for the given host controller and checks whether one of the resources defines prefetch memory (note theres_valid|= ...). The error will only be returned if no prefetchable memory region was found.
dw_pcie_host_init()will then again traverse the list of resources and it will typically encounter two resource of typeIORESOURCE_MEM, one for non-prefetchable memory and another for prefetchable memory.Vidya's patch is to differentiate between these two resources and allow prefetchable memory regions to exceed sizes of 4 GiB.
That said, I wonder if there isn't a bigger problem at hand here. From looking at the code it doesn't seem like the DWC driver makes any distinction between prefetchable and non-prefetchable memory. Or at least it doesn't allow both to be stored in struct
pcie_port.Am I missing something? Or can anyone explain how we're programming the apertures for prefetchable vs. non-prefetchable memory? Perhaps this is what Vidya was referring to when he said: "we are not using an outbound ATU translation channel for prefetchable memory".
Quoted 7 lines
It looks to me like we're also getting partially lucky, or perhaps that is by design, in that Tegra194 defines PCI regions in the following
order: I/O, prefetchable memory, non-prefetchable memory. That means that the DWC core code will overwrite prefetchable memory data with that of non-prefetchable memory and hence the non-prefetchable region ends up stored in structpcie_portand is then used to program the ATU outbound channel.
Well,it is by design. I mean, since the code is not differentiating between prefetchable and non-prefetchable regions, I ordered the entries in 'ranges' property in such a way that 'prefetchable' comes first followed by 'non-prefetchable' entry so that ATU region is used for generating the translation required for 'non-prefetchable' region (which is a non 1-to-1 mapping)
Quoted 31 lines
Quoted 7 lines
In other words, to reach the code that Vidya is changing, it can be only if the resource is a non-prefetchable, any prefetchable resource will be blocked by the previous call, if I'm not mistaken.
Having this in mind, Vidya's change will not make the expected result aimed by him.
Given the above I think it does. We've also seen this patch eliminate a warning that we were seeing before, so I think it has the intended effect.
Quoted 10 lines
I don't see any problem by having resources larger than 4GB, from what I'm seeing in the databook there isn't any restricting related to that as long they don't consume the maximum space that is addressable by the system (depending on if they are 32-bit or 64-bit system address).
To be honest, I'm not seeing a system that could have this resource larger than 4GB, but it might exist some exception that I don't know of, that's why I accepted Alan's patch to warn the user that the resource exceeds the maximum for the 32 bits so that he can be aware that he *might* be consuming the maximum space addressable.
I think it's pretty common to have this type of prefetchable memory region when you connect discrete GPUs to PCIe. It's not unusual for high-end GPUs to have 8 GiB or even more dedicated video memory and those will typically be mapped to a prefetchable memory region on the PCI device.
Thierry
Quoted 79 lines · Thierry Reding
Quoted 72 lines · Lorenzo Pieralisi
On Tue, May 19, 2020 at 10:08:54PM +0000, Gustavo Pimentel wrote:
[...]
Quoted 33 lines
Quoted 33 lines
Quoted 18 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.That code checks if there is *at least* a non-prefetchable resource, that's all it does.
Quoted 3 lines
In other words, to reach the code that Vidya is changing, it can be only if the resource is a non-prefetchable, any prefetchable resource will be blocked by the previous call, if I'm not mistaken.
I think you are mistaken sorry.
Quoted 2 lines
Having this in mind, Vidya's change will not make the expected result aimed by him.
I think Vidya's patch does what he expects, the question is whether it is widely applicable to ALL DWC hosts, that's what I want to know.
Quoted 10 lines
I don't see any problem by having resources larger than 4GB, from what I'm seeing in the databook there isn't any restricting related to that as long they don't consume the maximum space that is addressable by the system (depending on if they are 32-bit or 64-bit system address).
To be honest, I'm not seeing a system that could have this resource larger than 4GB, but it might exist some exception that I don't know of, that's why I accepted Alan's patch to warn the user that the resource exceeds the maximum for the 32 bits so that he can be aware that he *might* be consuming the maximum space addressable.
Quoted 5 lines
I think it is most certainly a possibility to have > 4GB prefetchable address spaces so we ought to fix this for good. I still have to understand how the DWC host detects the memory region to be programmed into the ATU given that there is more than one but only 1 ATU memory region AFAICS.
Quoted 8 lines
Probably best to wait for Vidya to confirm since I'm not altogether
familiar with PCI on Tegra194, but looking at the DTS files and the
Tegra194 TRM, the prefetchable memory regions are set to a range in
0x1200000000-0x1fffffffff which is a region of the address map that
is reserved for "PCIe aperture for > 32-bit OS". Part of that is in
use for non-prefetchable memory (and ends up being programmed into
the ATU) whereas a much larger part is used for prefetchable memory
and is not programmed anywhere, as far as I can tell.Yes. That is true. In case of Tegra194, for 1-to-1 memory translations,
there is no need to use any ATU regions at all as the HW is capable of generating a memory transaction on the bus on its own for any CPU generated reads/writes falling in these apertures and are not captured by ATU windows for generating other (config/IO) types of transactions. Since a part of 64-bit region is used for mapping non-prefetchable BARs of endpoints (which are only 32-bit in size), a translation is required from 64-bit CPU/AXI address to 32-bit PCIe bus address and ATU region is used precisely for this purpose. Since there is no need for ATU translation to do 1-to-1 memory mapping, rest of the 64-bit aperture is used for mapping prefetchable BARs. I'm not sure if this HW behavior is same across DWC implementations. I took a quick look at the 'ranges' property of other DWC implementations and none of them have a region marked as prefetchable...! I'm not sure how kernel handles mapping an endpoint's prefetchable BAR in this case (does it use the 'non-prefetchable' mapping of 'ranges' property for 'prefetchable' BARs as well??)
Quoted 8 lines
But I think given that this is a designated region of the address map this is probably automatically redirected to the PCIe controller. What I don't know is if that's something Tegra-specific or whether all instantiations have something similar set up.
Thierry
Rob Herring
reviewerQuoted 101 lines · Vidya Sagar
Quoted 91 lines · Thierry Reding
Quoted 58 lines · Gustavo Pimentel
Quoted 55 lines · Lorenzo Pieralisi
Quoted 39 lines · Vidya Sagar
Quoted 28 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 24 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 20 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.That's not what the code is doing.
pci_parse_request_of_pci_range()will traverse over the whole list of resources that it can find for the given host controller and checks whether one of the resources defines prefetch memory (note theres_valid|= ...). The error will only be returned if no prefetchable memory region was found.
dw_pcie_host_init()will then again traverse the list of resources and it will typically encounter two resource of typeIORESOURCE_MEM, one for non-prefetchable memory and another for prefetchable memory.Vidya's patch is to differentiate between these two resources and allow prefetchable memory regions to exceed sizes of 4 GiB.
That said, I wonder if there isn't a bigger problem at hand here. From looking at the code it doesn't seem like the DWC driver makes any distinction between prefetchable and non-prefetchable memory. Or at least it doesn't allow both to be stored in struct
pcie_port.Am I missing something? Or can anyone explain how we're programming the apertures for prefetchable vs. non-prefetchable memory? Perhaps this is what Vidya was referring to when he said: "we are not using an outbound ATU translation channel for prefetchable memory".
Quoted 7 lines
It looks to me like we're also getting partially lucky, or perhaps that is by design, in that Tegra194 defines PCI regions in the following
order: I/O, prefetchable memory, non-prefetchable memory. That means that the DWC core code will overwrite prefetchable memory data with that of non-prefetchable memory and hence the non-prefetchable region ends up stored in structpcie_portand is then used to program the ATU outbound channel.
Quoted 6 lines
Well,it is by design. I mean, since the code is not differentiating between prefetchable and non-prefetchable regions, I ordered the entries in 'ranges' property in such a way that 'prefetchable' comes first followed by 'non-prefetchable' entry so that ATU region is used for generating the translation required for 'non-prefetchable' region (which is a non 1-to-1 mapping)
You are getting lucky with your 'design'. Relying on order is fragile (except of course in the places in DT where order is defined, but ranges is not one of them).
Rob
Quoted 109 lines · Rob Herring
Quoted 101 lines · Vidya Sagar
Quoted 91 lines · Thierry Reding
Quoted 58 lines · Gustavo Pimentel
Quoted 55 lines · Lorenzo Pieralisi
Quoted 39 lines · Vidya Sagar
Quoted 28 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 24 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 20 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.That's not what the code is doing.
pci_parse_request_of_pci_range()will traverse over the whole list of resources that it can find for the given host controller and checks whether one of the resources defines prefetch memory (note theres_valid|= ...). The error will only be returned if no prefetchable memory region was found.
dw_pcie_host_init()will then again traverse the list of resources and it will typically encounter two resource of typeIORESOURCE_MEM, one for non-prefetchable memory and another for prefetchable memory.Vidya's patch is to differentiate between these two resources and allow prefetchable memory regions to exceed sizes of 4 GiB.
That said, I wonder if there isn't a bigger problem at hand here. From looking at the code it doesn't seem like the DWC driver makes any distinction between prefetchable and non-prefetchable memory. Or at least it doesn't allow both to be stored in struct
pcie_port.Am I missing something? Or can anyone explain how we're programming the apertures for prefetchable vs. non-prefetchable memory? Perhaps this is what Vidya was referring to when he said: "we are not using an outbound ATU translation channel for prefetchable memory".
Quoted 7 lines
It looks to me like we're also getting partially lucky, or perhaps that is by design, in that Tegra194 defines PCI regions in the following
order: I/O, prefetchable memory, non-prefetchable memory. That means that the DWC core code will overwrite prefetchable memory data with that of non-prefetchable memory and hence the non-prefetchable region ends up stored in structpcie_portand is then used to program the ATU outbound channel.Quoted 6 lines
Well,it is by design. I mean, since the code is not differentiating between prefetchable and non-prefetchable regions, I ordered the entries in 'ranges' property in such a way that 'prefetchable' comes first followed by 'non-prefetchable' entry so that ATU region is used for generating the translation required for 'non-prefetchable' region (which is a non 1-to-1 mapping)
Quoted 3 lines
You are getting lucky with your 'design'. Relying on order is fragile (except of course in the places in DT where order is defined, but ranges is not one of them).
Yeah, I think the DWC core should be improved to differentiate between the two types of memory resources. There shouldn't be a need to encode any ordering because the type is already part of the value in the ranges property.
Thierry
Quoted 110 lines · Thierry Reding
Quoted 109 lines · Rob Herring
Quoted 107 lines · Vidya Sagar
Quoted 91 lines · Thierry Reding
Quoted 58 lines · Gustavo Pimentel
Quoted 55 lines · Lorenzo Pieralisi
Quoted 39 lines · Vidya Sagar
Quoted 28 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 24 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 20 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.That's not what the code is doing.
pci_parse_request_of_pci_range()will traverse over the whole list of resources that it can find for the given host controller and checks whether one of the resources defines prefetch memory (note theres_valid|= ...). The error will only be returned if no prefetchable memory region was found.
dw_pcie_host_init()will then again traverse the list of resources and it will typically encounter two resource of typeIORESOURCE_MEM, one for non-prefetchable memory and another for prefetchable memory.Vidya's patch is to differentiate between these two resources and allow prefetchable memory regions to exceed sizes of 4 GiB.
That said, I wonder if there isn't a bigger problem at hand here. From looking at the code it doesn't seem like the DWC driver makes any distinction between prefetchable and non-prefetchable memory. Or at least it doesn't allow both to be stored in struct
pcie_port.Am I missing something? Or can anyone explain how we're programming the apertures for prefetchable vs. non-prefetchable memory? Perhaps this is what Vidya was referring to when he said: "we are not using an outbound ATU translation channel for prefetchable memory".
Quoted 7 lines
It looks to me like we're also getting partially lucky, or perhaps that is by design, in that Tegra194 defines PCI regions in the following
order: I/O, prefetchable memory, non-prefetchable memory. That means that the DWC core code will overwrite prefetchable memory data with that of non-prefetchable memory and hence the non-prefetchable region ends up stored in structpcie_portand is then used to program the ATU outbound channel.Well,it is by design. I mean, since the code is not differentiating between prefetchable and non-prefetchable regions, I ordered the entries in 'ranges' property in such a way that 'prefetchable' comes first followed by 'non-prefetchable' entry so that ATU region is used for generating the translation required for 'non-prefetchable' region (which is a non 1-to-1 mapping)
Quoted 8 lines
Quoted 3 lines
You are getting lucky with your 'design'. Relying on order is fragile (except of course in the places in DT where order is defined, but ranges is not one of them).
Yeah, I think the DWC core should be improved to differentiate between the two types of memory resources. There shouldn't be a need to encode any ordering because the type is already part of the value in the ranges property.
DWC resources handling is broken beyond belief. In practical terms, I think the best thing I can do is dropping:
9e73fa02aa00 ("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits")
from my pci/dwc branch. However, the ATU programming API must be fixed and this reliance on DT entries ordering avoided - it is really bad practice (and it prevents us from reworking kernel code in ways that are legitimate but would break owing to DT assumptions).
So yes, the DWC host bridge code must be updated asap - this is not acceptable.
Thanks,
Lorenzo
Quoted 30 lines · Alan Mikhak
Hi Lorenzo,
I came across this issue when implementing a Linux NVMe endpoint function driver under the Linux PCI Endpoint Framework:
https://lwn.net/Articles/804369/I needed to map up to 128GB of host memory using a single ATU window from the endpoint side because NVMe PRPs can be scattered all over host memory. In the process, I came across this 4GB limitation where the maximum size of memory that can be mapped is limited by what a u32 value can represent.
I submitted a separate patch to fix an undefined behavior that may also happen in
dw_pcie_prog_outbound_atu_unroll()under some circumstances when the size of the memory being mapped is greater than what a u32 value can represent.
https://patchwork.kernel.org/patch/11469701/The above patch has been accepted. However, the variable
pp->mem_sizeindw_pcie_host_init()is still a u32 whereas the value returned byresource_size()is u64. If the resource size has non-zero upper 32-bits, those upper 32-bits will be lost when assigning:
pp->mem_size=resource_size(pp->mem).Since current callers seem happy with the existing 4GB implementation and fixing the u32 limit is beyond my available resources and has a long high-impact tail, a warning seemed to be a good choice to highlight this issue in case someone else decides to map a MEM region that is greater than 4GB.
Quoted 5 lines
Removing the warning will avoid such discussions. Without this warning, this limitation will go unnoticed and will only impact whoever has to deal with it. It cost me time to figure it out when I had an application that needed a region larger than 4GB. I figured the most I could do about it is to raise the issue by adding a warning.
You did the right thing (and you helped me unearth some major deficiencies in current DWC code). Unfortunately I have to drop:
9e73fa02aa00 ("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits")
because it triggers regressions (and it is still not in the mainline, IMO there would be more if we send it upstream).
I will keep:
e1fc129219a8 ("PCI: dwc: Program outbound ATU upper limit register")
because it is a step in the right direction and makes sense on its own.
Thanks for all the effort you put into this.
Lorenzo
Quoted 8 lines
Regards,
Alan
Quoted 125 lines · Lorenzo Pieralisi
Quoted 110 lines · Thierry Reding
Quoted 109 lines · Rob Herring
Quoted 107 lines · Vidya Sagar
Quoted 91 lines · Thierry Reding
Quoted 58 lines · Gustavo Pimentel
Quoted 55 lines · Lorenzo Pieralisi
Quoted 39 lines · Vidya Sagar
Quoted 28 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 24 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 20 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.That's not what the code is doing.
pci_parse_request_of_pci_range()will traverse over the whole list of resources that it can find for the given host controller and checks whether one of the resources defines prefetch memory (note theres_valid|= ...). The error will only be returned if no prefetchable memory region was found.
dw_pcie_host_init()will then again traverse the list of resources and it will typically encounter two resource of typeIORESOURCE_MEM, one for non-prefetchable memory and another for prefetchable memory.Vidya's patch is to differentiate between these two resources and allow prefetchable memory regions to exceed sizes of 4 GiB.
That said, I wonder if there isn't a bigger problem at hand here. From looking at the code it doesn't seem like the DWC driver makes any distinction between prefetchable and non-prefetchable memory. Or at least it doesn't allow both to be stored in struct
pcie_port.Am I missing something? Or can anyone explain how we're programming the apertures for prefetchable vs. non-prefetchable memory? Perhaps this is what Vidya was referring to when he said: "we are not using an outbound ATU translation channel for prefetchable memory".
Quoted 7 lines
It looks to me like we're also getting partially lucky, or perhaps that is by design, in that Tegra194 defines PCI regions in the following
order: I/O, prefetchable memory, non-prefetchable memory. That means that the DWC core code will overwrite prefetchable memory data with that of non-prefetchable memory and hence the non-prefetchable region ends up stored in structpcie_portand is then used to program the ATU outbound channel.Well,it is by design. I mean, since the code is not differentiating between prefetchable and non-prefetchable regions, I ordered the entries in 'ranges' property in such a way that 'prefetchable' comes first followed by 'non-prefetchable' entry so that ATU region is used for generating the translation required for 'non-prefetchable' region (which is a non 1-to-1 mapping)
Quoted 8 lines
Quoted 3 lines
You are getting lucky with your 'design'. Relying on order is fragile (except of course in the places in DT where order is defined, but ranges is not one of them).
Yeah, I think the DWC core should be improved to differentiate between the two types of memory resources. There shouldn't be a need to encode any ordering because the type is already part of the value in the ranges property.
DWC resources handling is broken beyond belief. In practical terms, I think the best thing I can do is dropping:
9e73fa02aa00("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits")
Quoted 7 lines
from my pci/dwc branch. However, the ATU programming API must be fixed and this reliance on DT entries ordering avoided - it is really bad practice (and it prevents us from reworking kernel code in ways that are legitimate but would break owing to DT assumptions).
So yes, the DWC host bridge code must be updated asap - this is not acceptable.
Vidya, would you have any spare cycles to look into this a bit since you're already familiar? I think for starters it would be good to add a special case to the IORESOURCE_MEM case in dw_pcie_host_init() that deals with IORESOURCE_PREFETCH set and then store the result in a separate struct resource in struct pcie_port, something roughly along the lines of:
struct pcie_port {
...
struct resource *mem;
struct resource *prefetch;
...
};
...
int dw_pcie_host_init(struct pcie_port *pp)
{
...
resource_list_for_each_entry(win, &bridge->windows) {
switch (resource_type(win->res)) {
...
case IORESOURCE_MEM:
if (win->res.flags & IORESOURCE_PREFETCH) {
pp->prefetch = win->res;
...
} else {
pp->mem = win->res;
...
}
break;
...
}
...
}I suppose for the non-prefetchable memory we could leave the warning in because they can never be larger than 32 bits anyway. Then again, I'm not sure the check is actually fully correct. My recollection is that non-prefetchable memory needs to be completely within the 4 GiB range, rather than just the base and the size. So I think something like the base starting at 3 GiB and then spanning 2 GiB would be valid according to the current check, but I don't think it's valid according to the specification.
The other interesting datapoint to have would be whether the DWC core always has 1:1 mappings for prefetchable memory. If so, I think it might be useful to still parse them, even if nothing in the driver is using them. But I don't know what would be a good way to find out if that's really the case.
I also saw, like you did, that none of the other, non-Tegra device trees specify any prefetchable memory for the DWC, so I don't understand how they would work. Perhaps they just don't support prefetchable memory?
If you don't have the time to do this I could possibly take a stab at it but there are a few other things I need to look into, so I probably won't get around to this within the next two or so weeks.
Thierry
Quoted 185 lines · Thierry Reding
Quoted 125 lines · Lorenzo Pieralisi
Quoted 110 lines · Thierry Reding
Quoted 109 lines · Rob Herring
Quoted 107 lines · Vidya Sagar
Quoted 91 lines · Thierry Reding
Quoted 58 lines · Gustavo Pimentel
Quoted 55 lines · Lorenzo Pieralisi
Quoted 39 lines · Vidya Sagar
Quoted 28 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 24 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 20 lines · Vidya Sagar
commit
9e73fa02aa009("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by Vidya Sagar <[email protected]>--- drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/pci/controller/dwc/pcie-designware-host.c +1
-1@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->mem = win->res; pp->mem->name ="MEM" ; mem_size =resource_size (pp->mem); -if (upper_32_bits (mem_size)) + if (upper_32_bits(mem_size) &&Quoted 8 lines
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct pcie_port *pp) · line 373
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ !(win->res->flags & IORESOURCE_PREFETCH))dev_warn (dev,"MEM resource size exceeds max for 32 bits\n" ); pp->mem_size = mem_size; pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.That's not what the code is doing.
pci_parse_request_of_pci_range()will traverse over the whole list of resources that it can find for the given host controller and checks whether one of the resources defines prefetch memory (note theres_valid|= ...). The error will only be returned if no prefetchable memory region was found.
dw_pcie_host_init()will then again traverse the list of resources and it will typically encounter two resource of typeIORESOURCE_MEM, one for non-prefetchable memory and another for prefetchable memory.Vidya's patch is to differentiate between these two resources and allow prefetchable memory regions to exceed sizes of 4 GiB.
That said, I wonder if there isn't a bigger problem at hand here. From looking at the code it doesn't seem like the DWC driver makes any distinction between prefetchable and non-prefetchable memory. Or at least it doesn't allow both to be stored in struct
pcie_port.Am I missing something? Or can anyone explain how we're programming the apertures for prefetchable vs. non-prefetchable memory? Perhaps this is what Vidya was referring to when he said: "we are not using an outbound ATU translation channel for prefetchable memory".
Quoted 7 lines
It looks to me like we're also getting partially lucky, or perhaps that is by design, in that Tegra194 defines PCI regions in the following
order: I/O, prefetchable memory, non-prefetchable memory. That means that the DWC core code will overwrite prefetchable memory data with that of non-prefetchable memory and hence the non-prefetchable region ends up stored in structpcie_portand is then used to program the ATU outbound channel.Well,it is by design. I mean, since the code is not differentiating between prefetchable and non-prefetchable regions, I ordered the entries in 'ranges' property in such a way that 'prefetchable' comes first followed by 'non-prefetchable' entry so that ATU region is used for generating the translation required for 'non-prefetchable' region (which is a non 1-to-1 mapping)
Quoted 8 lines
Quoted 3 lines
You are getting lucky with your 'design'. Relying on order is fragile (except of course in the places in DT where order is defined, but ranges is not one of them).
Yeah, I think the DWC core should be improved to differentiate between the two types of memory resources. There shouldn't be a need to encode any ordering because the type is already part of the value in the ranges property.
DWC resources handling is broken beyond belief. In practical terms, I think the best thing I can do is dropping:
9e73fa02aa00("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits")Quoted 7 lines
from my pci/dwc branch. However, the ATU programming API must be fixed and this reliance on DT entries ordering avoided - it is really bad practice (and it prevents us from reworking kernel code in ways that are legitimate but would break owing to DT assumptions).
So yes, the DWC host bridge code must be updated asap - this is not acceptable.
Vidya, would you have any spare cycles to look into this a bit since you're already familiar? I think for starters it would be good to add a special case to the
IORESOURCE_MEMcase indw_pcie_host_init()that deals withIORESOURCE_PREFETCHset and then store the result in a separate struct resource in structpcie_port, something roughly along the lines of:struct pcie_port { ... struct resource *mem; struct resource *prefetch; ... }; ... int dw_pcie_host_init(struct pcie_port *pp) { ... resource_list_for_each_entry(win, &bridge->windows) { switch (resource_type(win->res)) { ... case IORESOURCE_MEM: if (win->res.flags & IORESOURCE_PREFETCH) { pp->prefetch = win->res; ... } else { pp->mem = win->res; ... } break; ... } ... }I suppose for the non-prefetchable memory we could leave the warning in because they can never be larger than 32 bits anyway. Then again, I'm not sure the check is actually fully correct. My recollection is that non-prefetchable memory needs to be completely within the 4 GiB range, rather than just the base and the size. So I think something like the base starting at 3 GiB and then spanning 2 GiB would be valid according to the current check, but I don't think it's valid according to the specification.
The other interesting datapoint to have would be whether the DWC core always has 1:1 mappings for prefetchable memory. If so, I think it might be useful to still parse them, even if nothing in the driver is using them. But I don't know what would be a good way to find out if that's really the case.
Quoted 7 lines
I also saw, like you did, that none of the other, non-Tegra device trees specify any prefetchable memory for the DWC, so I don't understand how they would work. Perhaps they just don't support prefetchable memory?
If you don't have the time to do this I could possibly take a stab at it but there are a few other things I need to look into, so I probably won't get around to this within the next two or so weeks.
Sure. I'll try to come up with a patch set to address this at DWC core level.
Thanks,
Vidya Sagar
Quoted 3 lines
Thierry
Quoted 190 lines · Vidya Sagar
Quoted 187 lines · Thierry Reding
Quoted 127 lines · Lorenzo Pieralisi
Quoted 112 lines · Thierry Reding
Quoted 111 lines · Rob Herring
Quoted 109 lines · Vidya Sagar
Quoted 93 lines · Thierry Reding
Quoted 60 lines · Gustavo Pimentel
Quoted 57 lines · Lorenzo Pieralisi
Quoted 41 lines · Vidya Sagar
Quoted 30 lines · Lorenzo Pieralisi
External email: Use caution opening links or attachments
Quoted 26 lines · Bjorn Helgaas
[+cc Alan; please cc authors of relevant commits,
updated Andrew's email address]Quoted 22 lines · Vidya Sagar
commit 9e73fa02aa009 ("PCI: dwc: Warn if MEM resource size exceeds max for
32-bits") enables warning for MEM resources of size >4GB but prefetchable
memory resources also come under this category where sizes can go beyond
4GB. Avoid logging a warning for prefetchable memory resources.Signed-off-by: Vidya Sagar <[email protected]>
---
drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 42fbfe2a1b8f..a29396529ea4 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -366,7 +366,8 @@ int dw_pcie_host_init(struct pcie_port
*pp)
pp->mem = win->res;
pp->mem->name = "MEM";
mem_size = resource_size(pp->mem);
- if (upper_32_bits(mem_size))
+ if (upper_32_bits(mem_size) &&
+ !(win->res->flags &Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
IORESOURCE_PREFETCH))
dev_warn(dev, "MEM resource size exceeds max for 32 bits\n");
pp->mem_size = mem_size;
pp->mem_bus_addr = pp->mem->start - win->offset;That warning was added for a reason - why should not we log legitimate warnings ? AFAIU having resources larger than 4GB can lead to undefined behaviour given the current ATU programming API.
Quoted 3 lines
Yeah. I'm all for a warning if the size is larger than 4GB in case of non-prefetchable window as one of the ATU outbound translation channels is being used,
Is it true for all DWC host controllers ? Or there may be another exception whereby we would be forced to disable this warning altogether ?
Quoted 1 line
but, we are not employing any ATU outbound translation channel for
What does this mean ? "we are not employing any ATU outbound...", is this the tegra driver ? And what guarantees that this warning is not legitimate on DWC host controllers that do use the ATU outbound translation for prefetchable windows ?
Quoted 8 lines
Quoted 1 line
Can DWC maintainers chime in and clarify please ?
Before this code section, there is the following function call
pci_parse_request_of_pci_ranges(), which performs a simple validation for theIORESOURCE_MEMresource type.
This validation checks if the resource is marked as prefetchable, if so, an error message "non-prefetchable memory resource required" is given and a return code with the-EINVALvalue.That's not what the code is doing.
pci_parse_request_of_pci_range()will traverse over the whole list of resources that it can find for the given host controller and checks whether one of the resources defines prefetch memory (note theres_valid|= ...). The error will only be returned if no prefetchable memory region was found.
dw_pcie_host_init()will then again traverse the list of resources and it will typically encounter two resource of typeIORESOURCE_MEM, one for non-prefetchable memory and another for prefetchable memory.Vidya's patch is to differentiate between these two resources and allow prefetchable memory regions to exceed sizes of 4 GiB.
That said, I wonder if there isn't a bigger problem at hand here. From looking at the code it doesn't seem like the DWC driver makes any distinction between prefetchable and non-prefetchable memory. Or at least it doesn't allow both to be stored in struct
pcie_port.Am I missing something? Or can anyone explain how we're programming the apertures for prefetchable vs. non-prefetchable memory? Perhaps this is what Vidya was referring to when he said: "we are not using an outbound ATU translation channel for prefetchable memory".
Quoted 7 lines
It looks to me like we're also getting partially lucky, or perhaps that is by design, in that Tegra194 defines PCI regions in the following
order: I/O, prefetchable memory, non-prefetchable memory. That means that the DWC core code will overwrite prefetchable memory data with that of non-prefetchable memory and hence the non-prefetchable region ends up stored in structpcie_portand is then used to program the ATU outbound channel.Well,it is by design. I mean, since the code is not differentiating between prefetchable and non-prefetchable regions, I ordered the entries in 'ranges' property in such a way that 'prefetchable' comes first followed by 'non-prefetchable' entry so that ATU region is used for generating the translation required for 'non-prefetchable' region (which is a non 1-to-1 mapping)
Quoted 8 lines
Quoted 3 lines
You are getting lucky with your 'design'. Relying on order is fragile (except of course in the places in DT where order is defined, but ranges is not one of them).
Yeah, I think the DWC core should be improved to differentiate between the two types of memory resources. There shouldn't be a need to encode any ordering because the type is already part of the value in the ranges property.
DWC resources handling is broken beyond belief. In practical terms, I think the best thing I can do is dropping:
9e73fa02aa00("PCI: dwc: Warn if MEM resource size exceeds max for 32-bits")Quoted 7 lines
from my pci/dwc branch. However, the ATU programming API must be fixed and this reliance on DT entries ordering avoided - it is really bad practice (and it prevents us from reworking kernel code in ways that are legitimate but would break owing to DT assumptions).
So yes, the DWC host bridge code must be updated asap - this is not acceptable.
Vidya, would you have any spare cycles to look into this a bit since you're already familiar? I think for starters it would be good to add a special case to the
IORESOURCE_MEMcase indw_pcie_host_init()that deals withIORESOURCE_PREFETCHset and then store the result in a separate struct resource in structpcie_port, something roughly along the lines of:struct
pcie_port{
...
struct resource *mem;
struct resource *prefetch;
...
};...
int
dw_pcie_host_init(struct pcie_port *pp)
{
...
resource_list_for_each_entry(win, &bridge->windows){ switch (resource_type(win->res)) {
...
caseIORESOURCE_MEM:
if (win->res.flags&IORESOURCE_PREFETCH) {pp->prefetch=win->res;
...
} else {
pp->mem=win->res;
...
}
break;
...
}
...
}I suppose for the non-prefetchable memory we could leave the warning in because they can never be larger than 32 bits anyway. Then again, I'm not sure the check is actually fully correct. My recollection is that non-prefetchable memory needs to be completely within the 4 GiB range, rather than just the base and the size. So I think something like the base starting at 3 GiB and then spanning 2 GiB would be valid according to the current check, but I don't think it's valid according to the specification.
The other interesting datapoint to have would be whether the DWC core always has 1:1 mappings for prefetchable memory. If so, I think it might be useful to still parse them, even if nothing in the driver is using them. But I don't know what would be a good way to find out if that's really the case.
Quoted 8 lines
Quoted 7 lines
I also saw, like you did, that none of the other, non-Tegra device trees specify any prefetchable memory for the DWC, so I don't understand how they would work. Perhaps they just don't support prefetchable memory?
If you don't have the time to do this I could possibly take a stab at it but there are a few other things I need to look into, so I probably won't get around to this within the next two or so weeks.
Sure. I'll try to come up with a patch set to address this at DWC core level.
New patch set for review is available @ http://patchwork.ozlabs.org/project/linux-pci/list/?series=180799
Thanks,
Vidya Sagar
Quoted 6 lines
Thanks,
Vidya SagarQuoted 3 lines
Thierry