Regression caused by "PCI: Shrink decoding-disabled window while sizing BARs"
Hello,
Since next-20141117, PCI support is broken on ARM Marvell EBU platforms. next-20141114 was working fine. The problem we see is:
pci 0000:03:00.0: reg 0x10: can't handle BAR larger than 4GB (size 0xfffffffffff00000)
Therefore, the Marvell-specific MBus window is not setup, and the first access to the PCI device registers causes a kernel panic.
You can compare a successful boot:
With a failed boot:
It turns out that if I take next-20141117 and revert the two following commits:
7ea945f0bb49423451dbf222f62c6fabc857aaac PCI: Shrink decoding-disabled window while sizing BARs 3a02517d5e2aef7cd4524468525a6292c81cfad1 PCI: Support 64-bit bridge windows if we have 64-bit dma_addr_t
Then the PCI support works again.
Note that the Marvell pci-mvebu driver emulates its own PCI bridge, in order to support the dynamic creation of MBus windows (this is a Marvell specific concept). Therefore, there is a possibility that the issue might be that our emulation is not 100% correct, which I'll be checking today.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
Hello,
Quoted 4 lines · Thomas Petazzoni
Since next-20141117, PCI support is broken on ARM Marvell EBU platforms. next-20141114 was working fine. The problem we see is:
pci 0000:03:00.0: reg
0x10: can't handle BAR larger than 4GB (size0xfffffffffff00000)
Another issue (possibly related, or not), is the following compile time warning:
/home/thomas/projets/linux-2.6/drivers/pci/probe.c: In function ‘pci_read_bridge_mmio_pref’:/home/thomas/projets/linux-2.6/drivers/pci/probe.c:430:5: warning: left shift count >= width of type
base |= ((dma_addr_t) mem_base_hi) << 32;
^
/home/thomas/projets/linux-2.6/drivers/pci/probe.c:431:5: warning: left shift count >= width of type
limit |= ((dma_addr_t) mem_limit_hi) << 32;
^This happens because dma_addr_t is 32 bits on ARM platforms, which isn't properly taken into account by the commit that introduces support for 64 bits DMA addresses.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
Hello,
Quoted 11 lines · Thomas Petazzoni
Another issue (possibly related, or not), is the following compile time warning:
/home/thomas/projets/linux-2.6/drivers/pci/probe.c: In function ‘pci_read_bridge_mmio_pref’:/home/thomas/projets/linux-2.6/drivers/pci/probe.c:430:5: warning: left shift count >= width of typebase |= ((dma_addr_t) mem_base_hi) << 32; ^ /home/thomas/projets/linux-2.6/drivers/pci/probe.c:431:5: warning: left shift count >= width of type limit |= ((dma_addr_t) mem_limit_hi) << 32; ^
Quoted 3 lines
This happens because
dma_addr_tis 32 bits on ARM platforms, which isn't properly taken into account by the commit that introduces support for 64 bits DMA addresses.
Not sure about this one: the commit introducing the issue is in next-20141117, but not in the current pci/next of Bjorn. Was it reverted? Am I looking at the wrong place?
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
Bjorn Helgaas
maintainer[+cc Yinghai]
Quoted 15 lines · Thomas Petazzoni
Hello,
Quoted 11 lines · Thomas Petazzoni
Another issue (possibly related, or not), is the following compile time warning:
/home/thomas/projets/linux-2.6/drivers/pci/probe.c: In function ‘pci_read_bridge_mmio_pref’:/home/thomas/projets/linux-2.6/drivers/pci/probe.c:430:5: warning: left shift count >= width of typebase |= ((dma_addr_t) mem_base_hi) << 32; ^ /home/thomas/projets/linux-2.6/drivers/pci/probe.c:431:5: warning: left shift count >= width of type limit |= ((dma_addr_t) mem_limit_hi) << 32; ^
Quoted 7 lines
Quoted 3 lines
This happens because
dma_addr_tis 32 bits on ARM platforms, which isn't properly taken into account by the commit that introduces support for 64 bits DMA addresses.Not sure about this one: the commit introducing the issue is in next-20141117, but not in the current pci/next of Bjorn. Was it reverted? Am I looking at the wrong place?
Yes, that change (3a02517d5e2a ("PCI: Support 64-bit bridge windows if we have 64-bit dma_addr_t")) is in my for-linus branch because it fixes a regression and I intend to merge it for v3.18 (after we figure out these issues, of course).
I saw this warning yesterday, but haven't heard from Yinghai yet.
Bjorn
PATCH 1/1 PCI: fix probe.c warning on !CONFIG_ARCH_DMA_ADDR_T_64BIT platforms
Commit 3a02517d5e2a ("PCI: Support 64-bit bridge windows if we have 64-bit dma_addr_t") modified the pci_read_bridge_mmio_pref() function to support 64 bits bridge windows if the dma_addr_t type if 64 bits.
However, even though it accounts for platforms where dma_addr_t is 32 bits, it introduced a compile-time warning on such platforms:
drivers/pci/probe.c: In function ‘pci_read_bridge_mmio_pref’:drivers/pci/probe.c:430:5: warning: left shift count >= width of type
base |= ((dma_addr_t) mem_base_hi) << 32;
^
drivers/pci/probe.c:431:5: warning: left shift count >= width of type
limit |= ((dma_addr_t) mem_limit_hi) << 32;This is due to the fact that the code that gets used on platforms where dma_addr_t is 64 bits is also compiled on platforms where dma_addr_t is 32 bits.
To solve this, this patch switches from using the runtime 'sizeof(dma_addr_t) < 8' test to a compile time test on CONFIG_ARCH_DMA_ADDR_T_64BIT, which is the configuration option used by <linux/types.h> to decide whether dma_addr_t is 32 bits or 64 bits.
Note that in the case mentionned in the commit log of 3a02517d5e2a, i.e x86 32 bits with PAE enabled, CONFIG_ARCH_DMA_ADDR_T_64BIT is enabled, because this option is enabled when either x86-64 is used, or x86 with HIGHMEM64G.
---
Applies on top of pci/for-linus
drivers/pci/probe.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/pci/probe.c +8 -8
@@ -429,15 +429,15 @@ static void pci_read_bridge_mmio_pref(struct pci_bus *child)* this, just assume they are not being used. */ if (mem_base_hi <= mem_limit_hi) { -if (sizeof (dma_addr_t) <8 ) { -if (mem_base_hi || mem_limit_hi) { -dev_err (&dev->dev,"can't handle 64-bit address space for bridge\n" ); -return ; - } - }else { - base |= ((dma_addr_t ) mem_base_hi) <<32 ; - limit |= ((dma_addr_t ) mem_limit_hi) <<32 ; +#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT + base |= ((dma_addr_t ) mem_base_hi) <<32 ; + limit |= ((dma_addr_t ) mem_limit_hi) <<32 ; +#else +if (mem_base_hi || mem_limit_hi) { +dev_err (&dev->dev,"can't handle 64-bit address space for bridge\n" ); +return ; } +#endif } }if (base <= limit) {
Signature · 1 line
2.1.0
Quoted 8 lines · Bjorn Helgaas
[+cc Yinghai]
Yes, that change (
3a02517d5e2a("PCI: Support 64-bit bridge windows if we have 64-bitdma_addr_t")) is in my for-linus branch because it fixes a regression and I intend to merge it for v3.18 (after we figure out these issues, of course).I saw this warning yesterday, but haven't heard from Yinghai yet.
Will send updated version with u64 cast.
Thanks
Yinghai
Dear Yinghai Lu,
Quoted 11 lines · Yinghai Lu
Quoted 8 lines · Bjorn Helgaas
[+cc Yinghai]
Yes, that change (
3a02517d5e2a("PCI: Support 64-bit bridge windows if we have 64-bitdma_addr_t")) is in my for-linus branch because it fixes a regression and I intend to merge it for v3.18 (after we figure out these issues, of course).I saw this warning yesterday, but haven't heard from Yinghai yet.
Will send updated version with u64 cast.
Is a u64 cast really the appropriate solution here? Have you seen my proposed fix "[PATCH] PCI: fix probe.c warning
on !CONFIG_ARCH_DMA_ADDR_T_64BIT platforms" ?
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
Quoted 3 lines · Thomas Petazzoni
Is a u64 cast really the appropriate solution here? Have you seen my proposed fix "[PATCH] PCI: fix
probe.cwarning
on !CONFIG_ARCH_DMA_ADDR_T_64BIT platforms" ?
Yes, that I saw your patch.
We want to make that to be consistent with __pci_read_bases() and avoid MACRO.
I think the warning is wrong. aka compile should omit out that branch. as sizeof(dma_addr_t) on !CONFIG_ARCH_DMA_ADDR_T_64BIT
is 4 and it is smaller than 8. and those are const, so the compiler should not ...
add the cast just shut off the compiler wrong warning.
code segment:
if (sizeof(dma_addr_t) < 8) {
if (mem_base_hi || mem_limit_hi) {
dev_err(&dev->dev, "can't
handle 64-bit address space for bridge\n");
return;
}
} else {
base |= (dma_addr_t)(((u64)mem_base_hi)<<32);
limit |= (dma_addr_t)(((u64)mem_limit_hi)<<32);
}Thanks
Yinghai
Bjorn Helgaas
maintainerQuoted 20 lines · Yinghai Lu
Quoted 3 lines · Thomas Petazzoni
Is a u64 cast really the appropriate solution here? Have you seen my proposed fix "[PATCH] PCI: fix
probe.cwarning
on !CONFIG_ARCH_DMA_ADDR_T_64BIT platforms" ?Yes, that I saw your patch.
We want to make that to be consistent with
__pci_read_bases()and avoid MACRO.I think the warning is wrong. aka compile should omit out that branch. as
sizeof(dma_addr_t)on !CONFIG_ARCH_DMA_ADDR_T_64BIT
is 4 and it is smaller than 8. and those are const, so the compiler should not ...add the cast just shut off the compiler wrong warning.
code segment:
Quoted 12 lines
if (sizeof(dma_addr_t) < 8) {
if (mem_base_hi || mem_limit_hi) {
dev_err(&dev->dev, "can't
handle 64-bit address space for bridge\n");
return;
}
} else {
base |= (dma_addr_t)(((u64)mem_base_hi)<<32);
limit |= (dma_addr_t)(((u64)mem_limit_hi)<<32);
}I propose the following third alternative, which always computes full 64-bit base/limit, then casts them to the dma_addr_t size and compares to see if anything got chopped off.
This is on top of pci/for-linus. Whatever we settle on, I'll fold it into the original patch before asking Linus to pull it.
Bjorn
drivers/pci/probe.c +15 -11
@@ -407,6 +407,7 @@ static void pci_read_bridge_mmio_pref(struct pci_bus *child) {struct pci_dev *dev = child->self;u16 mem_base_lo, mem_limit_lo; +u64 base64, limit64;dma_addr_t base, limit;struct pci_bus_region region;struct resource *res;
@@ -414,8 +415,8 @@ static void pci_read_bridge_mmio_pref(struct pci_bus *child) res = child->resource[2 ];pci_read_config_word (dev,PCI_PREF_MEMORY_BASE , &mem_base_lo);pci_read_config_word (dev,PCI_PREF_MEMORY_LIMIT , &mem_limit_lo); - base = ((dma_addr_t ) mem_base_lo &PCI_PREF_RANGE_MASK ) <<16 ; - limit = ((dma_addr_t ) mem_limit_lo &PCI_PREF_RANGE_MASK ) <<16 ; + base64 = (mem_base_lo &PCI_PREF_RANGE_MASK ) <<16 ; + limit64 = (mem_limit_lo &PCI_PREF_RANGE_MASK ) <<16 ;if ((mem_base_lo &PCI_PREF_RANGE_TYPE_MASK ) ==PCI_PREF_RANGE_TYPE_64 ) {u32 mem_base_hi, mem_limit_hi;
@@ -429,17 +430,20 @@ static void pci_read_bridge_mmio_pref(struct pci_bus *child)* this, just assume they are not being used. */ if (mem_base_hi <= mem_limit_hi) { -if (sizeof (dma_addr_t) <8 ) { -if (mem_base_hi || mem_limit_hi) { -dev_err (&dev->dev,"can't handle 64-bit address space for bridge\n" ); -return ; - } - }else { - base |= ((dma_addr_t ) mem_base_hi) <<32 ; - limit |= ((dma_addr_t ) mem_limit_hi) <<32 ; - } + base64 |= (u64 ) mem_base_hi <<32 ; + limit64 |= (u64 ) mem_limit_hi <<32 ; } } + + base = (dma_addr_t ) base64; + limit = (dma_addr_t ) limit64; + +if (base != base64) { +dev_err (&dev->dev,"can't handle bridge window above 4GB (bus address %#010llx)\n" , + (unsigned long long ) base64); +return ; + } +if (base <= limit) { res->flags = (mem_base_lo &PCI_PREF_RANGE_TYPE_MASK ) |IORESOURCE_MEM |IORESOURCE_PREFETCH ;
Quoted 47 lines · Bjorn Helgaas
I propose the following third alternative, which always computes full 64-bit base/limit, then casts them to the
dma_addr_tsize and compares to see if anything got chopped off.This is on top of pci/for-linus. Whatever we settle on, I'll fold it into the original patch before asking Linus to pull it.
drivers/pci/probe.c +5
-11@@ -407,6 +407,7 @@ static void pci_read_bridge_mmio_pref(struct pci_bus *child) {struct pci_dev *dev = child->self;u16 mem_base_lo, mem_limit_lo; +u64 base64, limit64;dma_addr_t base, limit;struct pci_bus_region region;struct resource *res;@@ -414,8 +415,8 @@ static void pci_read_bridge_mmio_pref(struct pci_bus *child) res = child->resource[2 ];pci_read_config_word (dev,PCI_PREF_MEMORY_BASE , &mem_base_lo);pci_read_config_word (dev,PCI_PREF_MEMORY_LIMIT , &mem_limit_lo); - base = ((dma_addr_t ) mem_base_lo &PCI_PREF_RANGE_MASK ) <<16 ; - limit = ((dma_addr_t ) mem_limit_lo &PCI_PREF_RANGE_MASK ) <<16 ; + base64 = (mem_base_lo &PCI_PREF_RANGE_MASK ) <<16 ; + limit64 = (mem_limit_lo &PCI_PREF_RANGE_MASK ) <<16 ;if ((mem_base_lo &PCI_PREF_RANGE_TYPE_MASK ) ==PCI_PREF_RANGE_TYPE_64 ) {u32 mem_base_hi, mem_limit_hi;@@ -429,17 +430,20 @@ static void pci_read_bridge_mmio_pref(struct pci_bus *child)* this, just assume they are not being used. */ if (mem_base_hi <= mem_limit_hi) { -if (sizeof (dma_addr_t) <8 ) { -if (mem_base_hi || mem_limit_hi) { -dev_err (&dev->dev,"can't handle 64-bit address space for bridge\n" ); -return ; - } - }else { - base |= ((dma_addr_t ) mem_base_hi) <<32 ; - limit |= ((dma_addr_t ) mem_limit_hi) <<32 ; - } + base64 |= (u64 ) mem_base_hi <<32 ; + limit64 |= (u64 ) mem_limit_hi <<32 ; } }
drivers/pci/probe.c static void pci_read_bridge_mmio_pref(struct pci_bus *child) · line 446
Quoted 10 lines
drivers/pci/probe.c +10
-0+ + base = (dma_addr_t ) base64; + limit = (dma_addr_t ) limit64; + +if (base != base64) { +dev_err (&dev->dev,"can't handle bridge window above 4GB (bus address %#010llx)\n" , + (unsigned long long ) base64); +return ; + } +
Yes, that is better. on 64bit, compiler will omit last 7 lines for comparing...
Thanks
Yinghai
PATCH 1/1 PCI: fixup __pci_read_base() after refactoring
In reply to Thomas PetazzoniIn commit 7ea945f0bb49 ("PCI: Shrink decoding-disabled window while sizing BARs"), Myron Stowe refactored the code of __pci_read_base() in order to reduce the amount of time spent with decoding disabled.
However, contrary to what was said in the commit log, the commit does introduce some functional change: the pci_size() function that used to be called *before* the BAR size check is done is now called *after* the BAR size check is done.
This causes some failures on certain platforms (namely ARM Marvell EBU platforms, equipped for example with a PCIe SATA card, or a PCIe USB3 XHCI controller):
pci 0000:03:00.0: reg 0x10: can't handle BAR larger than 4GB (size 0xfffffffffff00000)
This problem didn't exist before this commit, due to pci_size() being called before doing the PCI BAR size check. Therefore, this commit fixes the problem by restoring the initial order of the operation, by calling pci_size() before doing the PCI BAR size check.
---
Note: this fix has been tested to work correctly for me, and the PCI messages I get are now identical to the ones I was getting with 3.18-rc5. However, please review my patch carefully, as I must admit I do not fully understand all the implications of the change as well as the code in __pci_read_base().
Applies on top of pci/next
--- drivers/pci/probe.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/pci/probe.c +8 -8
@@ -246,6 +246,14 @@ int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type,if (!sz64) goto fail; + sz64 =pci_size (l64, sz64, mask64); + +if (!sz64) { +dev_info (&dev->dev,"%sreg 0x%x: invalid BAR (can't size)\n" , +FW_BUG , pos); + goto fail; + } +if (res->flags &IORESOURCE_MEM_64 ) {if ((sizeof (dma_addr_t) <8 ||sizeof (resource_size_t) <8 ) && sz64 >0x100000000ULL ) {
@@ -268,14 +276,6 @@ int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type, } } - sz64 =pci_size (l64, sz64, mask64); - -if (!sz64) { -dev_info (&dev->dev,"%sreg 0x%x: invalid BAR (can't size)\n" , -FW_BUG , pos); - goto fail; - } - region.start = l64; region.end = l64 + sz64;
Signature · 1 line
2.1.0
Re: [PATCH] PCI: fixup __pci_read_base() after refactoring
Quoted 29 lines · Thomas Petazzoni
In commit
7ea945f0bb49("PCI: Shrink decoding-disabled window while sizing BARs"), Myron Stowe refactored the code of__pci_read_base()in order to reduce the amount of time spent with decoding disabled.However, contrary to what was said in the commit log, the commit does introduce some functional change: the
pci_size()function that used to be called *before* the BAR size check is done is now called *after* the BAR size check is done.This causes some failures on certain platforms (namely ARM Marvell EBU platforms, equipped for example with a PCIe SATA card, or a PCIe USB3 XHCI controller):
pci 0000:03:00.0: reg
0x10: can't handle BAR larger than 4GB (size0xfffffffffff00000)This problem didn't exist before this commit, due to
pci_size()being called before doing the PCI BAR size check. Therefore, this commit fixes the problem by restoring the initial order of the operation, by callingpci_size()before doing the PCI BAR size check.Signed-off-by Thomas Petazzoni <[email protected]>Fixes 7ea945f0bb49 ("PCI: Shrink decoding-disabled window while sizing BARs")---
Note: this fix has been tested to work correctly for me, and the PCI messages I get are now identical to the ones I was getting with 3.18-rc5. However, please review my patch carefully, as I must admit I do not fully understand all the implications of the change as well as the code in__pci_read_base().
Quoted 6 lines
Applies on top of pci/next
Signed-off-by Thomas Petazzoni <[email protected]>--- drivers/pci/probe.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
I started seeing this same issue on Tegra today and I can confirm that this patch fixes this regression:
Re: [PATCH] PCI: fixup __pci_read_base() after refactoring
In reply to Thomas PetazzoniQuoted 15 lines · Thomas Petazzoni
In commit
7ea945f0bb49("PCI: Shrink decoding-disabled window while sizing BARs"), Myron Stowe refactored the code of__pci_read_base()in order to reduce the amount of time spent with decoding disabled.However, contrary to what was said in the commit log, the commit does introduce some functional change: the
pci_size()function that used to be called *before* the BAR size check is done is now called *after* the BAR size check is done.This causes some failures on certain platforms (namely ARM Marvell EBU platforms, equipped for example with a PCIe SATA card, or a PCIe USB3 XHCI controller):
pci 0000:03:00.0: reg
0x10: can't handle BAR larger than 4GB (size0xfffffffffff00000)
Quoted 8 lines
This problem didn't exist before this commit, due to
pci_size()being called before doing the PCI BAR size check. Therefore, this commit fixes the problem by restoring the initial order of the operation, by callingpci_size()before doing the PCI BAR size check.Signed-off-by Thomas Petazzoni <[email protected]>Fixes 7ea945f0bb49 ("PCI: Shrink decoding-disabled window while sizing BARs")
I also boot tested this on armada-370-mirabox (multi_v7_defconfig and mvebu_v7_defconfig) where I originally noticed the problem, and it's booting fine again.
and, you forgot something: ;)
Kevin
Bjorn Helgaas
maintainerRe: [PATCH] PCI: fixup __pci_read_base() after refactoring
In reply to Thomas PetazzoniQuoted 29 lines · Thomas Petazzoni
In commit
7ea945f0bb49("PCI: Shrink decoding-disabled window while sizing BARs"), Myron Stowe refactored the code of__pci_read_base()in order to reduce the amount of time spent with decoding disabled.However, contrary to what was said in the commit log, the commit does introduce some functional change: the
pci_size()function that used to be called *before* the BAR size check is done is now called *after* the BAR size check is done.This causes some failures on certain platforms (namely ARM Marvell EBU platforms, equipped for example with a PCIe SATA card, or a PCIe USB3 XHCI controller):
pci 0000:03:00.0: reg
0x10: can't handle BAR larger than 4GB (size0xfffffffffff00000)This problem didn't exist before this commit, due to
pci_size()being called before doing the PCI BAR size check. Therefore, this commit fixes the problem by restoring the initial order of the operation, by callingpci_size()before doing the PCI BAR size check.Signed-off-by Thomas Petazzoni <[email protected]>Fixes 7ea945f0bb49 ("PCI: Shrink decoding-disabled window while sizing BARs")---
Note: this fix has been tested to work correctly for me, and the PCI messages I get are now identical to the ones I was getting with 3.18-rc5. However, please review my patch carefully, as I must admit I do not fully understand all the implications of the change as well as the code in__pci_read_base().
Quoted 3 lines
Applies on top of pci/next
Signed-off-by Thomas Petazzoni <[email protected]>
I folded essentially this patch into "PCI: Shrink decoding-disabled window while sizing BARs" and re-merged that into my "next" branch. Thanks!
Bjorn
Quoted 41 lines
--- drivers/pci/probe.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)drivers/pci/probe.c +8
-8@@ -246,6 +246,14 @@ int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type,if (!sz64) goto fail; + sz64 =pci_size (l64, sz64, mask64); + +if (!sz64) { +dev_info (&dev->dev,"%sreg 0x%x: invalid BAR (can't size)\n" , +FW_BUG , pos); + goto fail; + } +if (res->flags &IORESOURCE_MEM_64 ) {if ((sizeof (dma_addr_t) <8 ||sizeof (resource_size_t) <8 ) && sz64 >0x100000000ULL ) {@@ -268,14 +276,6 @@ int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type, } } - sz64 =pci_size (l64, sz64, mask64); - -if (!sz64) { -dev_info (&dev->dev,"%sreg 0x%x: invalid BAR (can't size)\n" , -FW_BUG , pos); - goto fail; - } - region.start = l64; region.end = l64 + sz64;Signature · 1 line
2.1.0