SPARC/PCI: Correct 64-bit non-pref -> pref BAR resources
Subsystems (3)
PATCH RESEND 1/1 SPARC/PCI: Correct 64-bit non-pref -> pref BAR resources
SPARC T5-2 dts describes some PCI BARs as 64-bit resources without pref(etchable) bit (0x83... vs 0xc3... in assigned-addresses) for address ranges above the 4G threshold. Such resources cannot be placed into a non-prefetchable PCI bridge window that is capable only to 32-bit addressing. As such, it looks the platform is improperly describe by dts.
Kernel detect this problem (see the IORESOURCE_PREFETCH check in pci_find_parent_resource()) and fails to assign these BAR resources to the resource tree due to lack of a compatible bridge window.
Prior to the commit 754babaaf333 ("sparc/PCI: Remove pcibios_enable_device() as they do nothing extra") SPARC arch code did not test whether device resources were successfully in the resource tree when enabling a device, effectively hiding the problem. After removing the arch specific enable code, PCI core's pci_enable_resources() refuses to enable the device when it finds not all mem resources are assigned, and therefore mpt3sas can't be enabled:
pci 0001:04:00.0: reg 0x14: [mem 0x801110000000-0x80111000ffff 64bit]
pci 0001:04:00.0: reg 0x1c: [mem 0x801110040000-0x80111007ffff 64bit]
pci 0001:04:00.0: BAR 1 [mem 0x801110000000-0x80111000ffff 64bit]: can't claim; no compatible bridge window pci 0001:04:00.0: BAR 3 [mem 0x801110040000-0x80111007ffff 64bit]: can't claim; no compatible bridge window mpt3sas 0001:04:00.0: BAR 1 [mem size 0x00010000 64bit]: not assigned; can't enable device
For clarity, this filtered log only shows failures for one mpt3sas device but other devices fail similarly. In the reported case, the end result with all the failures is an unbootable system.
Things appeared to "work" before the commit 754babaaf333 ("sparc/PCI:
Remove pcibios_enable_device() as they do nothing extra") because the resource tree is agnostic to whether PCI BAR resources are properly in the tree or not. So as long as there was a parent resource (e.g. a root bus resource) that contains the address range, the resource tree code just places resource request underneath it without any consideration to the intermediate BAR resource. While it worked, it's incorrect setup still.
Add of fixup to set IORESOURCE_PREFETCH flag for a 64-bit PCI resource that has the end address above 4G requiring placement into the prefetchable window. Also log the issue.
---
Resending with linux-pci@ ML.
Any comments on the approach are welcome. E.g., is the fixup done at a correct level? Should it be targeted specifically to the known failures (how?) to avoid hiding more platform description problems?
It seems VF BARs still have 64-bit non-pref despite this change, AFAICT, those are read directly from the device's config space so would require ordinary quirks. None of them result in device enable failing though so the issue is orthogonal to the one being fixed here.
If suggesting a different approach, please do realize my knowledge about OF code is generally very limited (and I'm not sure how directly the fixup code in other archs, mainly ppc, can be used as an example how to do fixups with sparc).
--- arch/sparc/kernel/pci.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
arch/sparc/kernel/pci.c +23 -0
@@ -181,6 +181,28 @@ static int __init ofpci_debug(char *str)__setup ("ofpci_debug=" , ofpci_debug); +static void of_fixup_pci_pref (struct pci_dev *dev,int index, +struct resource *res) +{ +struct pci_bus_region region; + +if (!(res->flags &IORESOURCE_MEM_64 )) +return ; + +if (!resource_size (res)) +return ; + +pcibios_resource_to_bus (dev->bus, ®ion, res); +if (region.end <= ~((u32 )0 )) +return ; + +if (!(res->flags &IORESOURCE_PREFETCH )) { + res->flags |=IORESOURCE_PREFETCH ; +pci_info (dev,"reg 0x%x: fixup: pref added to 64-bit resource\n" , + index); + } +} +static unsigned long pci_parse_of_flags (u32 addr0) {unsigned long flags =0 ;
@@ -244,6 +266,7 @@ static void pci_parse_of_addrs(struct platform_device *op, res->end = op_res->end; res->flags = flags; res->name =pci_name (dev); +of_fixup_pci_pref (dev, i, res);pci_info (dev,"reg 0x%x: %pR\n" , i, res); }
base-commit: 3a8660878839faadb4f1a6dd72c3179c1df56787
Signature · 1 line
2.39.5
[top-posting to facilitate processing]
The issue fixed by below patch is still on my list of tracked regression, which got me wondering what's up here. Did it fall through the cracks? Is there some good reason why this approach was dropped? Or Was the regression maybe fixed already and I just missed it?
Ciao, Thorsten
Quoted 111 lines · Ilpo Järvinen
SPARC T5-2 dts describes some PCI BARs as 64-bit resources without pref(etchable) bit (
0x83... vs0xc3... in assigned-addresses) for address ranges above the 4G threshold. Such resources cannot be placed into a non-prefetchable PCI bridge window that is capable only to 32-bit addressing. As such, it looks the platform is improperly describe by dts.Kernel detect this problem (see the
IORESOURCE_PREFETCHcheck inpci_find_parent_resource()) and fails to assign these BAR resources to the resource tree due to lack of a compatible bridge window.Prior to the commit
754babaaf333("sparc/PCI: Removepcibios_enable_device()as they do nothing extra") SPARC arch code did not test whether device resources were successfully in the resource tree when enabling a device, effectively hiding the problem. After removing the arch specific enable code, PCI core'spci_enable_resources()refuses to enable the device when it finds not all mem resources are assigned, and therefore mpt3sas can't be enabled:pci 0001:04:00.0: reg
0x14: [mem 0x801110000000-0x80111000ffff 64bit]
pci 0001:04:00.0: reg0x1c: [mem 0x801110040000-0x80111007ffff 64bit]
pci 0001:04:00.0: BAR 1 [mem 0x801110000000-0x80111000ffff 64bit]: can't claim; no compatible bridge window pci 0001:04:00.0: BAR 3 [mem 0x801110040000-0x80111007ffff 64bit]: can't claim; no compatible bridge window mpt3sas 0001:04:00.0: BAR 1 [mem size0x0001000064bit]: not assigned; can't enable deviceFor clarity, this filtered log only shows failures for one mpt3sas device but other devices fail similarly. In the reported case, the end result with all the failures is an unbootable system.
Things appeared to "work" before the commit
754babaaf333("sparc/PCI:
Removepcibios_enable_device()as they do nothing extra") because the resource tree is agnostic to whether PCI BAR resources are properly in the tree or not. So as long as there was a parent resource (e.g. a root bus resource) that contains the address range, the resource tree code just places resource request underneath it without any consideration to the intermediate BAR resource. While it worked, it's incorrect setup still.Add of fixup to set
IORESOURCE_PREFETCHflag for a 64-bit PCI resource that has the end address above 4G requiring placement into the prefetchable window. Also log the issue.Fixes 754babaaf333 ("sparc/PCI: Remove pcibios_enable_device() as they do nothing extra")Reported-by Nathaniel Roach <[email protected]>Tested-by Nathaniel Roach <[email protected]>Signed-off-by Ilpo Järvinen <[email protected]>---
Resending with linux-pci@ ML.
Any comments on the approach are welcome. E.g., is the fixup done at a correct level? Should it be targeted specifically to the known failures (how?) to avoid hiding more platform description problems?
It seems VF BARs still have 64-bit non-pref despite this change, AFAICT, those are read directly from the device's config space so would require ordinary quirks. None of them result in device enable failing though so the issue is orthogonal to the one being fixed here.
If suggesting a different approach, please do realize my knowledge about OF code is generally very limited (and I'm not sure how directly the fixup code in other archs, mainly ppc, can be used as an example how to do fixups with sparc).
--- arch/sparc/kernel/pci.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)arch/sparc/kernel/pci.c +23
-0@@ -181,6 +181,28 @@ static int __init ofpci_debug(char *str)__setup ("ofpci_debug=" , ofpci_debug); +static void of_fixup_pci_pref (struct pci_dev *dev,int index, +struct resource *res) +{ +struct pci_bus_region region; + +if (!(res->flags &IORESOURCE_MEM_64 )) +return ; + +if (!resource_size (res)) +return ; + +pcibios_resource_to_bus (dev->bus, ®ion, res); +if (region.end <= ~((u32 )0 )) +return ; + +if (!(res->flags &IORESOURCE_PREFETCH )) { + res->flags |=IORESOURCE_PREFETCH ; +pci_info (dev,"reg 0x%x: fixup: pref added to 64-bit resource\n" , + index); + } +} +static unsigned long pci_parse_of_flags (u32 addr0) {unsigned long flags =0 ;@@ -244,6 +266,7 @@ static void pci_parse_of_addrs(struct platform_device *op, res->end = op_res->end; res->flags = flags; res->name =pci_name (dev); +of_fixup_pci_pref (dev, i, res);pci_info (dev,"reg 0x%x: %pR\n" , i, res); }base-commit:
3a8660878839faadb4f1a6dd72c3179c1df56787
Quoted 6 lines · Linux regression tracking (Thorsten Leemhuis)
[top-posting to facilitate processing]
The issue fixed by below patch is still on my list of tracked regression, which got me wondering what's up here. Did it fall through the cracks? Is there some good reason why this approach was dropped? Or Was the regression maybe fixed already and I just missed it?
Hi Thorsten,
There isn't other solution to this AFAIK so you've not missed anything.
I'm used to PCI related changes often taking long time to handle so I'm just patiently waiting. :-)
--
i.
Quoted 115 lines
Ciao, Thorsten
Quoted 111 lines · Ilpo Järvinen
SPARC T5-2 dts describes some PCI BARs as 64-bit resources without pref(etchable) bit (
0x83... vs0xc3... in assigned-addresses) for address ranges above the 4G threshold. Such resources cannot be placed into a non-prefetchable PCI bridge window that is capable only to 32-bit addressing. As such, it looks the platform is improperly describe by dts.Kernel detect this problem (see the
IORESOURCE_PREFETCHcheck inpci_find_parent_resource()) and fails to assign these BAR resources to the resource tree due to lack of a compatible bridge window.Prior to the commit
754babaaf333("sparc/PCI: Removepcibios_enable_device()as they do nothing extra") SPARC arch code did not test whether device resources were successfully in the resource tree when enabling a device, effectively hiding the problem. After removing the arch specific enable code, PCI core'spci_enable_resources()refuses to enable the device when it finds not all mem resources are assigned, and therefore mpt3sas can't be enabled:pci 0001:04:00.0: reg
0x14: [mem 0x801110000000-0x80111000ffff 64bit]
pci 0001:04:00.0: reg0x1c: [mem 0x801110040000-0x80111007ffff 64bit]
pci 0001:04:00.0: BAR 1 [mem 0x801110000000-0x80111000ffff 64bit]: can't claim; no compatible bridge window pci 0001:04:00.0: BAR 3 [mem 0x801110040000-0x80111007ffff 64bit]: can't claim; no compatible bridge window mpt3sas 0001:04:00.0: BAR 1 [mem size0x0001000064bit]: not assigned; can't enable deviceFor clarity, this filtered log only shows failures for one mpt3sas device but other devices fail similarly. In the reported case, the end result with all the failures is an unbootable system.
Things appeared to "work" before the commit
754babaaf333("sparc/PCI:
Removepcibios_enable_device()as they do nothing extra") because the resource tree is agnostic to whether PCI BAR resources are properly in the tree or not. So as long as there was a parent resource (e.g. a root bus resource) that contains the address range, the resource tree code just places resource request underneath it without any consideration to the intermediate BAR resource. While it worked, it's incorrect setup still.Add of fixup to set
IORESOURCE_PREFETCHflag for a 64-bit PCI resource that has the end address above 4G requiring placement into the prefetchable window. Also log the issue.Fixes 754babaaf333 ("sparc/PCI: Remove pcibios_enable_device() as they do nothing extra")Reported-by Nathaniel Roach <[email protected]>Tested-by Nathaniel Roach <[email protected]>Signed-off-by Ilpo Järvinen <[email protected]>---
Resending with linux-pci@ ML.
Any comments on the approach are welcome. E.g., is the fixup done at a correct level? Should it be targeted specifically to the known failures (how?) to avoid hiding more platform description problems?
It seems VF BARs still have 64-bit non-pref despite this change, AFAICT, those are read directly from the device's config space so would require ordinary quirks. None of them result in device enable failing though so the issue is orthogonal to the one being fixed here.
If suggesting a different approach, please do realize my knowledge about OF code is generally very limited (and I'm not sure how directly the fixup code in other archs, mainly ppc, can be used as an example how to do fixups with sparc).
--- arch/sparc/kernel/pci.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)arch/sparc/kernel/pci.c +23
-0@@ -181,6 +181,28 @@ static int __init ofpci_debug(char *str)__setup ("ofpci_debug=" , ofpci_debug); +static void of_fixup_pci_pref (struct pci_dev *dev,int index, +struct resource *res) +{ +struct pci_bus_region region; + +if (!(res->flags &IORESOURCE_MEM_64 )) +return ; + +if (!resource_size (res)) +return ; + +pcibios_resource_to_bus (dev->bus, ®ion, res); +if (region.end <= ~((u32 )0 )) +return ; + +if (!(res->flags &IORESOURCE_PREFETCH )) { + res->flags |=IORESOURCE_PREFETCH ; +pci_info (dev,"reg 0x%x: fixup: pref added to 64-bit resource\n" , + index); + } +} +static unsigned long pci_parse_of_flags (u32 addr0) {unsigned long flags =0 ;@@ -244,6 +266,7 @@ static void pci_parse_of_addrs(struct platform_device *op, res->end = op_res->end; res->flags = flags; res->name =pci_name (dev); +of_fixup_pci_pref (dev, i, res);pci_info (dev,"reg 0x%x: %pR\n" , i, res); }base-commit:
3a8660878839faadb4f1a6dd72c3179c1df56787
[+cc Adrian]
Quoted 42 lines · Ilpo Järvinen
SPARC T5-2 dts describes some PCI BARs as 64-bit resources without pref(etchable) bit (
0x83... vs0xc3... in assigned-addresses) for address ranges above the 4G threshold. Such resources cannot be placed into a non-prefetchable PCI bridge window that is capable only to 32-bit addressing. As such, it looks the platform is improperly describe by dts.Kernel detect this problem (see the
IORESOURCE_PREFETCHcheck inpci_find_parent_resource()) and fails to assign these BAR resources to the resource tree due to lack of a compatible bridge window.Prior to the commit
754babaaf333("sparc/PCI: Removepcibios_enable_device()as they do nothing extra") SPARC arch code did not test whether device resources were successfully in the resource tree when enabling a device, effectively hiding the problem. After removing the arch specific enable code, PCI core'spci_enable_resources()refuses to enable the device when it finds not all mem resources are assigned, and therefore mpt3sas can't be enabled:pci 0001:04:00.0: reg
0x14: [mem 0x801110000000-0x80111000ffff 64bit]
pci 0001:04:00.0: reg0x1c: [mem 0x801110040000-0x80111007ffff 64bit]
pci 0001:04:00.0: BAR 1 [mem 0x801110000000-0x80111000ffff 64bit]: can't claim; no compatible bridge window pci 0001:04:00.0: BAR 3 [mem 0x801110040000-0x80111007ffff 64bit]: can't claim; no compatible bridge window mpt3sas 0001:04:00.0: BAR 1 [mem size0x0001000064bit]: not assigned; can't enable deviceFor clarity, this filtered log only shows failures for one mpt3sas device but other devices fail similarly. In the reported case, the end result with all the failures is an unbootable system.
Things appeared to "work" before the commit
754babaaf333("sparc/PCI:
Removepcibios_enable_device()as they do nothing extra") because the resource tree is agnostic to whether PCI BAR resources are properly in the tree or not. So as long as there was a parent resource (e.g. a root bus resource) that contains the address range, the resource tree code just places resource request underneath it without any consideration to the intermediate BAR resource. While it worked, it's incorrect setup still.Add of fixup to set
IORESOURCE_PREFETCHflag for a 64-bit PCI resource that has the end address above 4G requiring placement into the prefetchable window. Also log the issue.
Quoted 5 lines
Fixes 754babaaf333 ("sparc/PCI: Remove pcibios_enable_device() as they do nothing extra")Reported-by Nathaniel Roach <[email protected]>Tested-by Nathaniel Roach <[email protected]>Signed-off-by Ilpo Järvinen <[email protected]>
Originally posted as attachment in response to Adrian's pointer to Nathaniel's regression report:
https://lore.kernel.org/r/[email protected]
This seems reasonable to me and addresses a v6.18 regression, so I put this on pci/for-linus for v6.19.
Quoted 4 lines
---
Resending with linux-pci@ ML.
Quoted 8 lines
Any comments on the approach are welcome. E.g., is the fixup done at a correct level? Should it be targeted specifically to the known failures (how?) to avoid hiding more platform description problems?
It seems VF BARs still have 64-bit non-pref despite this change, AFAICT, those are read directly from the device's config space so would require ordinary quirks. None of them result in device enable failing though so the issue is orthogonal to the one being fixed here.
The VF BARs that don't have PCI_BASE_ADDRESS_MEM_PREFETCH set is a generic question, not anything sparc-specific, right?
As of the "Removing Prefetchable Terminology" ECN
(https://members.pcisig.com/wg/PCI-SIG/document/20839) that was incorporated in PCIe r6.3, PCI_BASE_ADDRESS_MEM_PREFETCH (bit 3) is deprecated. There's a recommendation that it be set for 64-bit BARs and cleared for 32-bit BARs, but I suppose we'll see 64-bit VF BARs with it cleared as in this case.
We might need to rework the PCI_BASE_ADDRESS_MEM_PREFETCH handling to accommodate this. I suspect we'll still need to pay attention to it at least for conventional PCI, but there might be some wiggle room for PCIe.
Quoted 54 lines
If suggesting a different approach, please do realize my knowledge about OF code is generally very limited (and I'm not sure how directly the fixup code in other archs, mainly ppc, can be used as an example how to do fixups with sparc).
--- arch/sparc/kernel/pci.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)arch/sparc/kernel/pci.c +23
-0@@ -181,6 +181,28 @@ static int __init ofpci_debug(char *str)__setup ("ofpci_debug=" , ofpci_debug); +static void of_fixup_pci_pref (struct pci_dev *dev,int index, +struct resource *res) +{ +struct pci_bus_region region; + +if (!(res->flags &IORESOURCE_MEM_64 )) +return ; + +if (!resource_size (res)) +return ; + +pcibios_resource_to_bus (dev->bus, ®ion, res); +if (region.end <= ~((u32 )0 )) +return ; + +if (!(res->flags &IORESOURCE_PREFETCH )) { + res->flags |=IORESOURCE_PREFETCH ; +pci_info (dev,"reg 0x%x: fixup: pref added to 64-bit resource\n" , + index); + } +} +static unsigned long pci_parse_of_flags (u32 addr0) {unsigned long flags =0 ;@@ -244,6 +266,7 @@ static void pci_parse_of_addrs(struct platform_device *op, res->end = op_res->end; res->flags = flags; res->name =pci_name (dev); +of_fixup_pci_pref (dev, i, res);pci_info (dev,"reg 0x%x: %pR\n" , i, res); }base-commit:
3a8660878839faadb4f1a6dd72c3179c1df56787Signature · 1 line
2.39.5
Quoted 67 lines · Bjorn Helgaas
[+cc Adrian]
Quoted 42 lines · Ilpo Järvinen
SPARC T5-2 dts describes some PCI BARs as 64-bit resources without pref(etchable) bit (
0x83... vs0xc3... in assigned-addresses) for address ranges above the 4G threshold. Such resources cannot be placed into a non-prefetchable PCI bridge window that is capable only to 32-bit addressing. As such, it looks the platform is improperly describe by dts.Kernel detect this problem (see the
IORESOURCE_PREFETCHcheck inpci_find_parent_resource()) and fails to assign these BAR resources to the resource tree due to lack of a compatible bridge window.Prior to the commit
754babaaf333("sparc/PCI: Removepcibios_enable_device()as they do nothing extra") SPARC arch code did not test whether device resources were successfully in the resource tree when enabling a device, effectively hiding the problem. After removing the arch specific enable code, PCI core'spci_enable_resources()refuses to enable the device when it finds not all mem resources are assigned, and therefore mpt3sas can't be enabled:pci 0001:04:00.0: reg
0x14: [mem 0x801110000000-0x80111000ffff 64bit]
pci 0001:04:00.0: reg0x1c: [mem 0x801110040000-0x80111007ffff 64bit]
pci 0001:04:00.0: BAR 1 [mem 0x801110000000-0x80111000ffff 64bit]: can't claim; no compatible bridge window pci 0001:04:00.0: BAR 3 [mem 0x801110040000-0x80111007ffff 64bit]: can't claim; no compatible bridge window mpt3sas 0001:04:00.0: BAR 1 [mem size0x0001000064bit]: not assigned; can't enable deviceFor clarity, this filtered log only shows failures for one mpt3sas device but other devices fail similarly. In the reported case, the end result with all the failures is an unbootable system.
Things appeared to "work" before the commit
754babaaf333("sparc/PCI:
Removepcibios_enable_device()as they do nothing extra") because the resource tree is agnostic to whether PCI BAR resources are properly in the tree or not. So as long as there was a parent resource (e.g. a root bus resource) that contains the address range, the resource tree code just places resource request underneath it without any consideration to the intermediate BAR resource. While it worked, it's incorrect setup still.Add of fixup to set
IORESOURCE_PREFETCHflag for a 64-bit PCI resource that has the end address above 4G requiring placement into the prefetchable window. Also log the issue.Quoted 5 lines
Fixes 754babaaf333 ("sparc/PCI: Remove pcibios_enable_device() as they do nothing extra")Reported-by Nathaniel Roach <[email protected]>Tested-by Nathaniel Roach <[email protected]>Signed-off-by Ilpo Järvinen <[email protected]>Originally posted as attachment in response to Adrian's pointer to Nathaniel's regression report:
https://lore.kernel.org/r/[email protected]
This seems reasonable to me and addresses a v6.18 regression, so I put this on pci/for-linus for v6.19.
Quoted 8 lines
---
Resending with linux-pci@ ML.
Any comments on the approach are welcome. E.g., is the fixup done at a correct level? Should it be targeted specifically to the known failures (how?) to avoid hiding more platform description problems?
Quoted 7 lines
Quoted 4 lines
It seems VF BARs still have 64-bit non-pref despite this change, AFAICT, those are read directly from the device's config space so would require ordinary quirks. None of them result in device enable failing though so the issue is orthogonal to the one being fixed here.
The VF BARs that don't have
PCI_BASE_ADDRESS_MEM_PREFETCHset is a generic question, not anything sparc-specific, right?
What I tried to say is that on this platform, we have such cases, but that being said they do not cause regression (and they cannot be "fixed" by the same means either).
None of these BARs (non-VF and VF ones) would not get assigned because of the IORESOURCE_PREFETCH check in pci_find_parent_resource() but post-754babaaf333 ("sparc/PCI: Remove pcibios_enable_device() as they do nothing extra"), enable device just doesn't like that. But because VF BARs are treated "optional", only non-VF BARs cause enable fail (unrelated to this issue, I got asked this week why VF resources are optional and realized I don't know the answer myself).
...
Quoted 6 lines
As of the "Removing Prefetchable Terminology" ECN
(https://members.pcisig.com/wg/PCI-SIG/document/20839) that was incorporated in PCIe r6.3,PCI_BASE_ADDRESS_MEM_PREFETCH(bit 3) is deprecated. There's a recommendation that it be set for 64-bit BARs and cleared for 32-bit BARs, but I suppose we'll see 64-bit VF BARs with it cleared as in this case.
I'd missed this change (IIRC, I only read CBs only up to r6.2 and then r6.4 to r7.0).
...If we look VF BAR flags on general level, could we now actually set IORESOURCE_PREFETCH in decode_bar() if it's a 64-bit BAR on a PCIe device? That should make those VF BARs and other non-pref 64-bit "right" and it would be rather straightforward way to differentiate PCI vs PCIe use cases.
Quoted 4 lines
We might need to rework the
PCI_BASE_ADDRESS_MEM_PREFETCHhandling to accommodate this. I suspect we'll still need to pay attention to it at least for conventional PCI, but there might be some wiggle room for PCIe.
Abolish 64-bit non-pref goes to non-prefetchable window rule, or more like replace it with some rule that depends on conventional PCI being present or not, doesn't look that complicated change thanks to the addition of pbus_select_window_for_type() which consolidates these decision into a central place (but it would require finding out if there's conventional PCI involved which requires more work such as walking the device hierarchy or maintaining flag which tells that information). Only the assignment still run on its own fallback rules independent of pbus_select_window_for_type() and of course that pci_find_parent_resource() (this should probably be renamed to _for_addr or something along those lines to better align with pbus_select_window_for_type()).
Out of those two approaches, to keep conventional PCI compability, it would be probably more straightforward to just make any PCIe 64-bit BAR IORESOURCE_PREFETCH but that isn't as clearly connected to the intended logic decision which is "conventional PCI or not?" but I guess a comment in decode_bar() and perhaps in ioport.h would be good enough the explain it.
Related to this, the root bus resource flags are a mess, IORESOURCE_MEM_64 is not always there even if addresses are clearly 64-bit and parent resource selection there should probably be agnostic to IORESOURCE_PREFETCH (which is currently not done as cleanly as it could be but just as artifacts the fallbacks have).
Signature · 1 line
i.
Quoted 55 lines
Quoted 54 lines
If suggesting a different approach, please do realize my knowledge about OF code is generally very limited (and I'm not sure how directly the fixup code in other archs, mainly ppc, can be used as an example how to do fixups with sparc).
--- arch/sparc/kernel/pci.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)arch/sparc/kernel/pci.c +23
-0@@ -181,6 +181,28 @@ static int __init ofpci_debug(char *str)__setup ("ofpci_debug=" , ofpci_debug); +static void of_fixup_pci_pref (struct pci_dev *dev,int index, +struct resource *res) +{ +struct pci_bus_region region; + +if (!(res->flags &IORESOURCE_MEM_64 )) +return ; + +if (!resource_size (res)) +return ; + +pcibios_resource_to_bus (dev->bus, ®ion, res); +if (region.end <= ~((u32 )0 )) +return ; + +if (!(res->flags &IORESOURCE_PREFETCH )) { + res->flags |=IORESOURCE_PREFETCH ; +pci_info (dev,"reg 0x%x: fixup: pref added to 64-bit resource\n" , + index); + } +} +static unsigned long pci_parse_of_flags (u32 addr0) {unsigned long flags =0 ;@@ -244,6 +266,7 @@ static void pci_parse_of_addrs(struct platform_device *op, res->end = op_res->end; res->flags = flags; res->name =pci_name (dev); +of_fixup_pci_pref (dev, i, res);pci_info (dev,"reg 0x%x: %pR\n" , i, res); }base-commit:
3a8660878839faadb4f1a6dd72c3179c1df56787Signature · 1 line
2.39.5
Quoted 78 lines · Ilpo Järvinen
Quoted 65 lines · Bjorn Helgaas
Quoted 42 lines · Ilpo Järvinen
SPARC T5-2 dts describes some PCI BARs as 64-bit resources without pref(etchable) bit (
0x83... vs0xc3... in assigned-addresses) for address ranges above the 4G threshold. Such resources cannot be placed into a non-prefetchable PCI bridge window that is capable only to 32-bit addressing. As such, it looks the platform is improperly describe by dts.Kernel detect this problem (see the
IORESOURCE_PREFETCHcheck inpci_find_parent_resource()) and fails to assign these BAR resources to the resource tree due to lack of a compatible bridge window.Prior to the commit
754babaaf333("sparc/PCI: Removepcibios_enable_device()as they do nothing extra") SPARC arch code did not test whether device resources were successfully in the resource tree when enabling a device, effectively hiding the problem. After removing the arch specific enable code, PCI core'spci_enable_resources()refuses to enable the device when it finds not all mem resources are assigned, and therefore mpt3sas can't be enabled:pci 0001:04:00.0: reg
0x14: [mem 0x801110000000-0x80111000ffff 64bit]
pci 0001:04:00.0: reg0x1c: [mem 0x801110040000-0x80111007ffff 64bit]
pci 0001:04:00.0: BAR 1 [mem 0x801110000000-0x80111000ffff 64bit]: can't claim; no compatible bridge window pci 0001:04:00.0: BAR 3 [mem 0x801110040000-0x80111007ffff 64bit]: can't claim; no compatible bridge window mpt3sas 0001:04:00.0: BAR 1 [mem size0x0001000064bit]: not assigned; can't enable deviceFor clarity, this filtered log only shows failures for one mpt3sas device but other devices fail similarly. In the reported case, the end result with all the failures is an unbootable system.
Things appeared to "work" before the commit
754babaaf333("sparc/PCI:
Removepcibios_enable_device()as they do nothing extra") because the resource tree is agnostic to whether PCI BAR resources are properly in the tree or not. So as long as there was a parent resource (e.g. a root bus resource) that contains the address range, the resource tree code just places resource request underneath it without any consideration to the intermediate BAR resource. While it worked, it's incorrect setup still.Add of fixup to set
IORESOURCE_PREFETCHflag for a 64-bit PCI resource that has the end address above 4G requiring placement into the prefetchable window. Also log the issue.Quoted 5 lines
Fixes 754babaaf333 ("sparc/PCI: Remove pcibios_enable_device() as they do nothing extra")Reported-by Nathaniel Roach <[email protected]>Tested-by Nathaniel Roach <[email protected]>Signed-off-by Ilpo Järvinen <[email protected]>Originally posted as attachment in response to Adrian's pointer to Nathaniel's regression report:
https://lore.kernel.org/r/[email protected]
This seems reasonable to me and addresses a v6.18 regression, so I put this on pci/for-linus for v6.19.
Quoted 8 lines
---
Resending with linux-pci@ ML.
Any comments on the approach are welcome. E.g., is the fixup done at a correct level? Should it be targeted specifically to the known failures (how?) to avoid hiding more platform description problems?
Quoted 7 lines
Quoted 4 lines
It seems VF BARs still have 64-bit non-pref despite this change, AFAICT, those are read directly from the device's config space so would require ordinary quirks. None of them result in device enable failing though so the issue is orthogonal to the one being fixed here.
The VF BARs that don't have
PCI_BASE_ADDRESS_MEM_PREFETCHset is a generic question, not anything sparc-specific, right?What I tried to say is that on this platform, we have such cases, but that being said they do not cause regression (and they cannot be "fixed" by the same means either).
Quoted 7 lines
None of these BARs (non-VF and VF ones) would not get assigned because of the
IORESOURCE_PREFETCHcheck inpci_find_parent_resource()but post-754babaaf333 ("sparc/PCI: Removepcibios_enable_device()as they do nothing extra"), enable device just doesn't like that. But because VF BARs are treated "optional", only non-VF BARs cause enable fail (unrelated to this issue, I got asked this week why VF resources are optional and realized I don't know the answer myself).
I don't know either. I think we currently try to assign the max possible VF resources based on TotalVFs. If that allocation fails, none of the VFs are usable, even if we could have allocated smaller areas and at least used *some* of the VFs.
That seems like it might be a little too aggressive because it makes a device that supports a huge number of VFs completely unusable on a platform with limited MMIO space, when we could in principle make it usable with fewer VFs.
Bjorn