Skip to contentSkip to conversation

PCI: rcar: Always allocate MSI addresses in 32bit space

PATCH Applied [email protected] 4 participants 12 messages Oct 16, 2020 – Jan 15, 2021

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: rcar: Always allocate MSI addresses in 32bit space

From: Marek Vasut <[email protected]>

This fixes MSI operation on legacy PCI cards, which cannot issue 64bit MSIs. The R-Car controller only has one MSI trigger address instead of two, one for 64bit and one for 32bit MSI, set the address to 32bit PCIe space so that legacy PCI cards can also trigger MSIs.

Fixes 290c1fb35860 ("PCI: rcar: Add MSI support for PCIe")
Cc Bjorn Helgaas <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc Lorenzo Pieralisi <[email protected]>
Cc Wolfram Sang <[email protected]>
Cc Yoshihiro Shimoda <[email protected]>
---
 drivers/pci/controller/pcie-rcar-host.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/controller/pcie-rcar-host.c +1 -1
index 1194d5f3341b..ac5c7d7573a6 100644
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host)
 	}
 
 	/* setup MSI data target */
-	msi->pages = __get_free_pages(GFP_KERNEL, 0);
+	msi->pages = __get_free_pages(GFP_KERNEL | GFP_DMA32, 0);
 	rcar_pcie_hw_enable_msi(host);
 
 	return 0;
Signature · 1 line
2.28.0
Tested-by Reviewed-by on v1 1/1
#1 lore raw

Hi Marek,

Quoted 9 lines · On Fri, Oct 16, 2020 at 2:04 PM <[email protected]> wrote:

From: Marek Vasut <[email protected]>

This fixes MSI operation on legacy PCI cards, which cannot issue 64bit MSIs. The R-Car controller only has one MSI trigger address instead of two, one for 64bit and one for 32bit MSI, set the address to 32bit PCIe space so that legacy PCI cards can also trigger MSIs.

Fixes 290c1fb35860 ("PCI: rcar: Add MSI support for PCIe")

Thanks for your patch!

Seems to work, on both R-Car M2-W and M3-W, as virt_to_phys((void *)msi->pages) points to RAM below the 4 GiB limit, so

drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 756

Quoted 8 lines
drivers/pci/controller/pcie-rcar-host.c +1 -1
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host)
        }
 
        /* setup MSI data target */
-       msi->pages = __get_free_pages(GFP_KERNEL, 0);
+       msi->pages = __get_free_pages(GFP_KERNEL | GFP_DMA32, 0);

BTW, can this fail, especially now this is allocated from a more limited pool?

drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 759

Quoted 3 lines
drivers/pci/controller/pcie-rcar-host.c +0 -0
        rcar_pcie_hw_enable_msi(host);
 
        return 0;

Regardless:

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#2 lore raw
Quoted 1 line · Geert Uytterhoeven

Hi Marek,

Hi,

[...]

Quoted 11 lines

drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 756

Quoted 8 lines
drivers/pci/controller/pcie-rcar-host.c +1 -1
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host)
        }
 
        /* setup MSI data target */
-       msi->pages = __get_free_pages(GFP_KERNEL, 0);
+       msi->pages = __get_free_pages(GFP_KERNEL | GFP_DMA32, 0);

BTW, can this fail, especially now this is allocated from a more limited pool?

I am pretty sure this can fail on systems that don't have DRAM below 4 GiB , but that is never the case on any hardware with this controller.

[...]

maintainer Reviewed-by Tested-by on v1 1/1
#3 lore raw
In reply to [email protected]

Hello Marek-san,

Quoted 9 lines

From: [email protected], Sent: Friday, October 16, 2020 9:05 PM

From: Marek Vasut <[email protected]>

This fixes MSI operation on legacy PCI cards, which cannot issue 64bit MSIs. The R-Car controller only has one MSI trigger address instead of two, one for 64bit and one for 32bit MSI, set the address to 32bit PCIe space so that legacy PCI cards can also trigger MSIs.

Quoted 8 lines
Fixes 290c1fb35860 ("PCI: rcar: Add MSI support for PCIe")
Cc Bjorn Helgaas <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc Lorenzo Pieralisi <[email protected]>
Cc Wolfram Sang <[email protected]>
Cc Yoshihiro Shimoda <[email protected]>

Thank you for the patch!

And I tested on R-Car H3. So,

Best regards,
Yoshihiro Shimoda

#4 lore raw
In reply to [email protected]
Quoted 24 lines · On Fri, Oct 16, 2020 at 02:04:31PM +0200, [email protected] wrote:

From: Marek Vasut <[email protected]>

This fixes MSI operation on legacy PCI cards, which cannot issue 64bit MSIs. The R-Car controller only has one MSI trigger address instead of two, one for 64bit and one for 32bit MSI, set the address to 32bit PCIe space so that legacy PCI cards can also trigger MSIs.

Fixes 290c1fb35860 ("PCI: rcar: Add MSI support for PCIe")
Cc Bjorn Helgaas <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc Lorenzo Pieralisi <[email protected]>
Cc Wolfram Sang <[email protected]>
Cc Yoshihiro Shimoda <[email protected]>
---
 drivers/pci/controller/pcie-rcar-host.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/controller/pcie-rcar-host.c +0 -0
index 1194d5f3341b..ac5c7d7573a6 100644

drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 756

Quoted 5 lines
drivers/pci/controller/pcie-rcar-host.c +1 -1
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host)
 	}
 
 	/* setup MSI data target */
-	msi->pages = __get_free_pages(GFP_KERNEL, 0);
+	msi->pages = __get_free_pages(GFP_KERNEL | GFP_DMA32, 0);

This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).

Can't you just pick up a non-DMA-able address < 4GB (ie outside the host controller inbound window range) and use it as doorbell address instead ?

Thanks,
Lorenzo

Quoted 6 lines
drivers/pci/controller/pcie-rcar-host.c +0 -0
 	rcar_pcie_hw_enable_msi(host);
 
 	return 0;
Signature · 1 line
2.28.0
#5 lore raw

On 12/10/20 7:11 PM, Lorenzo Pieralisi wrote:

[...]

Quoted 7 lines
Quoted 7 lines
drivers/pci/controller/pcie-rcar-host.c +0 -0
index 1194d5f3341b..ac5c7d7573a6 100644
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host)
  	}
 
Quoted 5 lines

drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 756

Quoted 3 lines
drivers/pci/controller/pcie-rcar-host.c +1 -1
  	/* setup MSI data target */
-	msi->pages = __get_free_pages(GFP_KERNEL, 0);
+	msi->pages = __get_free_pages(GFP_KERNEL | GFP_DMA32, 0);

This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).

How come? I would expect GFP_DMA32 allocates a buffer below 4 GiB in any case.

[...]

#6 lore raw
Quoted 16 lines · Marek Vasut

On 12/10/20 7:11 PM, Lorenzo Pieralisi wrote:

[...]

Quoted 11 lines

drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 755

Quoted 9 lines
drivers/pci/controller/pcie-rcar-host.c +1 -1
index 1194d5f3341b..ac5c7d7573a6 100644
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host)
  	}
  	/* setup MSI data target */
-	msi->pages = __get_free_pages(GFP_KERNEL, 0);
+	msi->pages = __get_free_pages(GFP_KERNEL | GFP_DMA32, 0);

This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).

Quoted 2 lines

How come? I would expect GFP_DMA32 allocates a buffer below 4 GiB in any case.

For ARM LPAE allocation falls back to ZONE_NORMAL that happens to work because if there is memory > 4GB it ends up in ZONE_HIGHMEM, so this patch should still work on ARM LPAE too.

Regardless, thoughts above the alternative approach (that saves you a page allocation) ?

Lorenzo

#7 lore raw
Quoted 20 lines · Lorenzo Pieralisi
Quoted 16 lines · Marek Vasut

On 12/10/20 7:11 PM, Lorenzo Pieralisi wrote:

[...]

Quoted 11 lines

drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 755

Quoted 9 lines
drivers/pci/controller/pcie-rcar-host.c +1 -1
index 1194d5f3341b..ac5c7d7573a6 100644
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host)
   	}
   	/* setup MSI data target */
-	msi->pages = __get_free_pages(GFP_KERNEL, 0);
+	msi->pages = __get_free_pages(GFP_KERNEL | GFP_DMA32, 0);

This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).

Quoted 2 lines

How come? I would expect GFP_DMA32 allocates a buffer below 4 GiB in any case.

Quoted 6 lines

For ARM LPAE allocation falls back to ZONE_NORMAL that happens to work because if there is memory > 4GB it ends up in ZONE_HIGHMEM, so this patch should still work on ARM LPAE too.

Regardless, thoughts above the alternative approach (that saves you a page allocation) ?

Since this is a bugfix, I would prefer it to be minimal.

Also, in case there was some yet undiscovered hardware bug which would let the MSI write through, having unused memory as the MSI destination address would only lead to write into that memory -- instead of a write into some other address.

Changing this to some hard-coded address (any suggestions?) can be a subsequent patch.

#8 lore raw
Quoted 21 lines · Marek Vasut
Quoted 20 lines · Lorenzo Pieralisi
Quoted 18 lines · Marek Vasut

On 12/10/20 7:11 PM, Lorenzo Pieralisi wrote:

[...]

Quoted 11 lines

drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 755

Quoted 9 lines
drivers/pci/controller/pcie-rcar-host.c +1 -1
index 1194d5f3341b..ac5c7d7573a6 100644
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host)
   	}
   	/* setup MSI data target */
-	msi->pages = __get_free_pages(GFP_KERNEL, 0);
+	msi->pages = __get_free_pages(GFP_KERNEL | GFP_DMA32, 0);

This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).

How come? I would expect GFP_DMA32 allocates a buffer below 4 GiB in any case.

Quoted 8 lines
Quoted 6 lines

For ARM LPAE allocation falls back to ZONE_NORMAL that happens to work because if there is memory > 4GB it ends up in ZONE_HIGHMEM, so this patch should still work on ARM LPAE too.

Regardless, thoughts above the alternative approach (that saves you a page allocation) ?

Since this is a bugfix, I would prefer it to be minimal.

Yes, I agree with you on that.

Quoted 7 lines

Also, in case there was some yet undiscovered hardware bug which would let the MSI write through, having unused memory as the MSI destination address would only lead to write into that memory -- instead of a write into some other address.

Changing this to some hard-coded address (any suggestions?) can be a subsequent patch.

The idea was taking the address from the host controller inbound window (ie an address outside the dma-ranges ~(dma-ranges) and < 4GB), it should not matter which one. I agree though that this can be a subsequent patch even though usually we send for -rc* only fixes for patches that hit the previous merge window - this seems a quite longstanding (I traced it back to v3.16) one so it would wait till v5.12, there is time to refactor it.

Thanks,
Lorenzo

#9 lore raw

On 12/21/20 11:01 AM, Lorenzo Pieralisi wrote:

[...]

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines

drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 755

Quoted 7 lines
drivers/pci/controller/pcie-rcar-host.c +1 -1
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host)
    	}
    	/* setup MSI data target */
-	msi->pages = __get_free_pages(GFP_KERNEL, 0);
+	msi->pages = __get_free_pages(GFP_KERNEL | GFP_DMA32, 0);

This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).

Quoted 2 lines

How come? I would expect GFP_DMA32 allocates a buffer below 4 GiB in any case.

For ARM LPAE allocation falls back to ZONE_NORMAL that happens to work because if there is memory > 4GB it ends up in ZONE_HIGHMEM, so this patch should still work on ARM LPAE too.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Regardless, thoughts above the alternative approach (that saves you a page allocation) ?

Since this is a bugfix, I would prefer it to be minimal.

Yes, I agree with you on that.

Then maybe it makes sense to apply this bugfix so others can benefit from it too ?

Quoted 11 lines
Quoted 7 lines

Also, in case there was some yet undiscovered hardware bug which would let the MSI write through, having unused memory as the MSI destination address would only lead to write into that memory -- instead of a write into some other address.

Changing this to some hard-coded address (any suggestions?) can be a subsequent patch.

The idea was taking the address from the host controller inbound window (ie an address outside the dma-ranges ~(dma-ranges) and < 4GB), it should not matter which one.

Wouldn't that make the code quite unnecessarily complex for no gain ?
The above fix does just that in one line, unless there is some code in the PCI subsystem to select such an address already ?

Quoted 5 lines

I agree though that this can be a
subsequent patch even though usually we send for -rc* only fixes for patches that hit the previous merge window - this seems a quite longstanding (I traced it back to v3.16) one so it would wait till v5.12, there is time to refactor it.

I see, I was not aware of this policy toward bugfixes.

#10 lore raw
Quoted 24 lines · Marek Vasut

On 12/21/20 11:01 AM, Lorenzo Pieralisi wrote:

[...]

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 10 lines
Quoted 9 lines

drivers/pci/controller/pcie-rcar-host.c static int rcar_pcie_enable_msi(struct rcar_pcie_host *host) · line 755

Quoted 7 lines
drivers/pci/controller/pcie-rcar-host.c +1 -1
@@ -753,7 +753,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie_host *host)
    	}
    	/* setup MSI data target */
-	msi->pages = __get_free_pages(GFP_KERNEL, 0);
+	msi->pages = __get_free_pages(GFP_KERNEL | GFP_DMA32, 0);

This does not do what you want on !CONFIG_ZONE_DMA32 (ie arm LPAE).

Quoted 2 lines

How come? I would expect GFP_DMA32 allocates a buffer below 4 GiB in any case.

For ARM LPAE allocation falls back to ZONE_NORMAL that happens to work because if there is memory > 4GB it ends up in ZONE_HIGHMEM, so this patch should still work on ARM LPAE too.

Regardless, thoughts above the alternative approach (that saves you a page allocation) ?

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Since this is a bugfix, I would prefer it to be minimal.

Yes, I agree with you on that.

Then maybe it makes sense to apply this bugfix so others can benefit from it too ?

I will apply it shortly, thanks.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Also, in case there was some yet undiscovered hardware bug which would let the MSI write through, having unused memory as the MSI destination address would only lead to write into that memory -- instead of a write into some other address.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Changing this to some hard-coded address (any suggestions?) can be a subsequent patch.

The idea was taking the address from the host controller inbound window (ie an address outside the dma-ranges ~(dma-ranges) and < 4GB), it should not matter which one.

Wouldn't that make the code quite unnecessarily complex for no gain ?

Well, there is a gain, current code is allocating a page of memory - there is no need to do that and I don't think that what I am asking is complex.

Again, I will merge this patch but please have a look to check if what I ask above is a possibility.

Thanks,
Lorenzo

Quoted 10 lines

The above fix does just that in one line, unless there is some code in the PCI subsystem to select such an address already ?

Quoted 5 lines

I agree though that this can be a
subsequent patch even though usually we send for -rc* only fixes for patches that hit the previous merge window - this seems a quite longstanding (I traced it back to v3.16) one so it would wait till v5.12, there is time to refactor it.

I see, I was not aware of this policy toward bugfixes.

#11 lore raw
In reply to [email protected]
Quoted 4 lines · On Fri, 16 Oct 2020 14:04:31 +0200, [email protected] wrote:

This fixes MSI operation on legacy PCI cards, which cannot issue 64bit MSIs. The R-Car controller only has one MSI trigger address instead of two, one for 64bit and one for 32bit MSI, set the address to 32bit PCIe space so that legacy PCI cards can also trigger MSIs.

Applied to pci/rcar, thanks!

[1/1] PCI: rcar: Always allocate MSI addresses in 32bit space

      https://git.kernel.org/lpieralisi/pci/c/c4e0fec2f7

Thanks,
Lorenzo