PCI: rcar: Always allocate MSI addresses in 32bit space
Subsystems (4)
PATCH 1/1 PCI: rcar: Always allocate MSI addresses in 32bit space
From: Marek Vasut <[email protected]>
This fixes MSI operation on legacy PCI cards, which cannot issue 64bit MSIs. The R-Car controller only has one MSI trigger address instead of two, one for 64bit and one for 32bit MSI, set the address to 32bit PCIe space so that legacy PCI cards can also trigger MSIs.
--- drivers/pci/controller/pcie-rcar-host.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/controller/pcie-rcar-host.c +1 -1
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) }/* setup MSI data target */ - msi->pages =__get_free_pages (GFP_KERNEL ,0 ); + msi->pages =__get_free_pages (GFP_KERNEL |GFP_DMA32 ,0 );rcar_pcie_hw_enable_msi (host);return 0 ;
Signature · 1 line
2.28.0
Hi Marek,
Quoted 9 lines · On Fri, Oct 16, 2020 at 2:04 PM <[email protected]> wrote:
From: Marek Vasut <[email protected]>
This fixes MSI operation on legacy PCI cards, which cannot issue 64bit MSIs. The R-Car controller only has one MSI trigger address instead of two, one for 64bit and one for 32bit MSI, set the address to 32bit PCIe space so that legacy PCI cards can also trigger MSIs.
Fixes 290c1fb35860 ("PCI: rcar: Add MSI support for PCIe")Signed-off-by Marek Vasut <[email protected]>
Thanks for your patch!
Seems to work, on both R-Car M2-W and M3-W, as virt_to_phys((void *)msi->pages) points to RAM below the 4 GiB limit, so
drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 756
Quoted 8 lines
drivers/pci/controller/pcie-rcar-host.c +1
-1@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) }/* setup MSI data target */ - msi->pages =__get_free_pages (GFP_KERNEL ,0 ); + msi->pages =__get_free_pages (GFP_KERNEL |GFP_DMA32 ,0 );
BTW, can this fail, especially now this is allocated from a more limited pool?
drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 759
Quoted 3 lines
drivers/pci/controller/pcie-rcar-host.c +0
-0rcar_pcie_hw_enable_msi (host);return 0 ;
Regardless:
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Quoted 1 line · Geert Uytterhoeven
Hi Marek,
Hi,
[...]
Quoted 11 lines
drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 756
Quoted 8 lines
drivers/pci/controller/pcie-rcar-host.c +1
-1@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) }/* setup MSI data target */ - msi->pages =__get_free_pages (GFP_KERNEL ,0 ); + msi->pages =__get_free_pages (GFP_KERNEL |GFP_DMA32 ,0 );BTW, can this fail, especially now this is allocated from a more limited pool?
I am pretty sure this can fail on systems that don't have DRAM below 4 GiB , but that is never the case on any hardware with this controller.
[...]
Hello Marek-san,
Quoted 9 lines
From: [email protected], Sent: Friday, October 16, 2020 9:05 PM
From: Marek Vasut <[email protected]>
This fixes MSI operation on legacy PCI cards, which cannot issue 64bit MSIs. The R-Car controller only has one MSI trigger address instead of two, one for 64bit and one for 32bit MSI, set the address to 32bit PCIe space so that legacy PCI cards can also trigger MSIs.
Quoted 8 lines
Fixes 290c1fb35860 ("PCI: rcar: Add MSI support for PCIe")Signed-off-by Marek Vasut <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Geert Uytterhoeven <[email protected]>Cc Lorenzo Pieralisi <[email protected]>Cc Wolfram Sang <[email protected]>Cc Yoshihiro Shimoda <[email protected]>
Thank you for the patch!
And I tested on R-Car H3. So,
Best regards,
Yoshihiro Shimoda
Quoted 24 lines · On Fri, Oct 16, 2020 at 02:04:31PM +0200, [email protected] wrote:
From: Marek Vasut <[email protected]>
This fixes MSI operation on legacy PCI cards, which cannot issue 64bit MSIs. The R-Car controller only has one MSI trigger address instead of two, one for 64bit and one for 32bit MSI, set the address to 32bit PCIe space so that legacy PCI cards can also trigger MSIs.
Fixes 290c1fb35860 ("PCI: rcar: Add MSI support for PCIe")Signed-off-by Marek Vasut <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Geert Uytterhoeven <[email protected]>Cc Lorenzo Pieralisi <[email protected]>Cc Wolfram Sang <[email protected]>Cc Yoshihiro Shimoda <[email protected]>--- drivers/pci/controller/pcie-rcar-host.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/controller/pcie-rcar-host.c +0
-0
drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 756
Quoted 5 lines
drivers/pci/controller/pcie-rcar-host.c +1
-1@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) }/* setup MSI data target */ - msi->pages =__get_free_pages (GFP_KERNEL ,0 ); + msi->pages =__get_free_pages (GFP_KERNEL |GFP_DMA32 ,0 );
This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).
Can't you just pick up a non-DMA-able address < 4GB (ie outside the host controller inbound window range) and use it as doorbell address instead ?
Thanks,
Lorenzo
Quoted 6 lines
drivers/pci/controller/pcie-rcar-host.c +0
-0rcar_pcie_hw_enable_msi (host);return 0 ;Signature · 1 line
2.28.0
On 12/10/20 7:11 PM, Lorenzo Pieralisi wrote:
[...]
Quoted 7 lines
Quoted 7 lines
drivers/pci/controller/pcie-rcar-host.c +0
-0@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) }
Quoted 5 lines
drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 756
Quoted 3 lines
drivers/pci/controller/pcie-rcar-host.c +1
-1/* setup MSI data target */ - msi->pages =__get_free_pages (GFP_KERNEL ,0 ); + msi->pages =__get_free_pages (GFP_KERNEL |GFP_DMA32 ,0 );This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).
How come? I would expect GFP_DMA32 allocates a buffer below 4 GiB in any case.
[...]
Quoted 16 lines · Marek Vasut
On 12/10/20 7:11 PM, Lorenzo Pieralisi wrote:
[...]
Quoted 11 lines
drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 755
Quoted 9 lines
drivers/pci/controller/pcie-rcar-host.c +1
-1@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) }/* setup MSI data target */ - msi->pages =__get_free_pages (GFP_KERNEL ,0 ); + msi->pages =__get_free_pages (GFP_KERNEL |GFP_DMA32 ,0 );This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).
Quoted 2 lines
How come? I would expect
GFP_DMA32allocates a buffer below 4 GiB in any case.
For ARM LPAE allocation falls back to ZONE_NORMAL that happens to work because if there is memory > 4GB it ends up in ZONE_HIGHMEM, so this patch should still work on ARM LPAE too.
Regardless, thoughts above the alternative approach (that saves you a page allocation) ?
Lorenzo
Quoted 20 lines · Lorenzo Pieralisi
Quoted 16 lines · Marek Vasut
On 12/10/20 7:11 PM, Lorenzo Pieralisi wrote:
[...]
Quoted 11 lines
drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 755
Quoted 9 lines
drivers/pci/controller/pcie-rcar-host.c +1
-1@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) }/* setup MSI data target */ - msi->pages =__get_free_pages (GFP_KERNEL ,0 ); + msi->pages =__get_free_pages (GFP_KERNEL |GFP_DMA32 ,0 );This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).
Quoted 2 lines
How come? I would expect
GFP_DMA32allocates a buffer below 4 GiB in any case.
Quoted 6 lines
For ARM LPAE allocation falls back to
ZONE_NORMALthat happens to work because if there is memory > 4GB it ends up inZONE_HIGHMEM, so this patch should still work on ARM LPAE too.Regardless, thoughts above the alternative approach (that saves you a page allocation) ?
Since this is a bugfix, I would prefer it to be minimal.
Also, in case there was some yet undiscovered hardware bug which would let the MSI write through, having unused memory as the MSI destination address would only lead to write into that memory -- instead of a write into some other address.
Changing this to some hard-coded address (any suggestions?) can be a subsequent patch.
Quoted 21 lines · Marek Vasut
Quoted 20 lines · Lorenzo Pieralisi
Quoted 18 lines · Marek Vasut
On 12/10/20 7:11 PM, Lorenzo Pieralisi wrote:
[...]
Quoted 11 lines
drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 755
Quoted 9 lines
drivers/pci/controller/pcie-rcar-host.c +1
-1@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) }/* setup MSI data target */ - msi->pages =__get_free_pages (GFP_KERNEL ,0 ); + msi->pages =__get_free_pages (GFP_KERNEL |GFP_DMA32 ,0 );This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).
How come? I would expect
GFP_DMA32allocates a buffer below 4 GiB in any case.
Quoted 8 lines
Quoted 6 lines
For ARM LPAE allocation falls back to
ZONE_NORMALthat happens to work because if there is memory > 4GB it ends up inZONE_HIGHMEM, so this patch should still work on ARM LPAE too.Regardless, thoughts above the alternative approach (that saves you a page allocation) ?
Since this is a bugfix, I would prefer it to be minimal.
Yes, I agree with you on that.
Quoted 7 lines
Also, in case there was some yet undiscovered hardware bug which would let the MSI write through, having unused memory as the MSI destination address would only lead to write into that memory -- instead of a write into some other address.
Changing this to some hard-coded address (any suggestions?) can be a subsequent patch.
The idea was taking the address from the host controller inbound window (ie an address outside the dma-ranges ~(dma-ranges) and < 4GB), it should not matter which one. I agree though that this can be a subsequent patch even though usually we send for -rc* only fixes for patches that hit the previous merge window - this seems a quite longstanding (I traced it back to v3.16) one so it would wait till v5.12, there is time to refactor it.
Thanks,
Lorenzo
On 12/21/20 11:01 AM, Lorenzo Pieralisi wrote:
[...]
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines
drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 755
Quoted 7 lines
drivers/pci/controller/pcie-rcar-host.c +1
-1@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) }/* setup MSI data target */ - msi->pages =__get_free_pages (GFP_KERNEL ,0 ); + msi->pages =__get_free_pages (GFP_KERNEL |GFP_DMA32 ,0 );This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).
Quoted 2 lines
How come? I would expect
GFP_DMA32allocates a buffer below 4 GiB in any case.For ARM LPAE allocation falls back to
ZONE_NORMALthat happens to work because if there is memory > 4GB it ends up inZONE_HIGHMEM, so this patch should still work on ARM LPAE too.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Regardless, thoughts above the alternative approach (that saves you a page allocation) ?
Since this is a bugfix, I would prefer it to be minimal.
Yes, I agree with you on that.
Then maybe it makes sense to apply this bugfix so others can benefit from it too ?
Quoted 11 lines
Quoted 7 lines
Also, in case there was some yet undiscovered hardware bug which would let the MSI write through, having unused memory as the MSI destination address would only lead to write into that memory -- instead of a write into some other address.
Changing this to some hard-coded address (any suggestions?) can be a subsequent patch.
The idea was taking the address from the host controller inbound window (ie an address outside the dma-ranges ~(dma-ranges) and < 4GB), it should not matter which one.
Wouldn't that make the code quite unnecessarily complex for no gain ?
The above fix does just that in one line, unless there is some code in the PCI subsystem to select such an address already ?
Quoted 5 lines
I agree though that this can be a
subsequent patch even though usually we send for -rc* only fixes for patches that hit the previous merge window - this seems a quite longstanding (I traced it back to v3.16) one so it would wait till v5.12, there is time to refactor it.
I see, I was not aware of this policy toward bugfixes.
Quoted 24 lines · Marek Vasut
On 12/21/20 11:01 AM, Lorenzo Pieralisi wrote:
[...]
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 10 lines
Quoted 9 lines
drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 755
Quoted 7 lines
drivers/pci/controller/pcie-rcar-host.c +1
-1@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) }/* setup MSI data target */ - msi->pages =__get_free_pages (GFP_KERNEL ,0 ); + msi->pages =__get_free_pages (GFP_KERNEL |GFP_DMA32 ,0 );This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).
Quoted 2 lines
How come? I would expect
GFP_DMA32allocates a buffer below 4 GiB in any case.For ARM LPAE allocation falls back to
ZONE_NORMALthat happens to work because if there is memory > 4GB it ends up inZONE_HIGHMEM, so this patch should still work on ARM LPAE too.Regardless, thoughts above the alternative approach (that saves you a page allocation) ?
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Since this is a bugfix, I would prefer it to be minimal.
Yes, I agree with you on that.
Then maybe it makes sense to apply this bugfix so others can benefit from it too ?
I will apply it shortly, thanks.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Also, in case there was some yet undiscovered hardware bug which would let the MSI write through, having unused memory as the MSI destination address would only lead to write into that memory -- instead of a write into some other address.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Changing this to some hard-coded address (any suggestions?) can be a subsequent patch.
The idea was taking the address from the host controller inbound window (ie an address outside the dma-ranges ~(dma-ranges) and < 4GB), it should not matter which one.
Wouldn't that make the code quite unnecessarily complex for no gain ?
Well, there is a gain, current code is allocating a page of memory - there is no need to do that and I don't think that what I am asking is complex.
Again, I will merge this patch but please have a look to check if what I ask above is a possibility.
Thanks,
Lorenzo
Quoted 10 lines
The above fix does just that in one line, unless there is some code in the PCI subsystem to select such an address already ?
Quoted 5 lines
I agree though that this can be a
subsequent patch even though usually we send for -rc* only fixes for patches that hit the previous merge window - this seems a quite longstanding (I traced it back to v3.16) one so it would wait till v5.12, there is time to refactor it.I see, I was not aware of this policy toward bugfixes.
Lorenzo Pieralisi
AppliedQuoted 4 lines · On Fri, 16 Oct 2020 14:04:31 +0200, [email protected] wrote:
This fixes MSI operation on legacy PCI cards, which cannot issue 64bit MSIs. The R-Car controller only has one MSI trigger address instead of two, one for 64bit and one for 32bit MSI, set the address to 32bit PCIe space so that legacy PCI cards can also trigger MSIs.
Applied to pci/rcar, thanks!
[1/1] PCI: rcar: Always allocate MSI addresses in 32bit space
https://git.kernel.org/lpieralisi/pci/c/c4e0fec2f7
Thanks,
Lorenzo