PCI: dwc: Strengthen the MSI address allocation logic
PATCH v3 1/1 PCI: dwc: Strengthen the MSI address allocation logic
There can be platforms that do not use/have 32-bit DMA addresses but want to enumerate endpoints which support only 32-bit MSI address. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.
If there vendor driver has already setup the MSI address using some mechanism, use the same. This method can be used by the platforms described above to support EPs they wish to.
Else, if the memory region is not reserved, try to allocate a 32-bit IOVA. Additionally, if this allocation also fails, attempt a 64-bit allocation for probe to be successful. If the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address will not work.
---
Changelog since v2:
- If the vendor driver has setup the msi_data, use the same
Changelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails
.../pci/controller/dwc/pcie-designware-host.c | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +20 -6
@@ -374,10 +374,18 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)* order not to miss MSI TLPs from those devices the MSI target * address has to be within the lowest 4GB. * -* Note until there is a better alternative found the reservation is -* done by allocating from the artificially limited DMA-coherent -* memory. +* Check if the vendor driver has setup the MSI address already. If yes, +* pick up the same. This will be helpful for platforms that do not +* use/have 32-bit DMA addresses but want to use endpoints which support +* only 32-bit MSI address. +* Else, if the memory region is not reserved, try to allocate a 32-bit +* IOVA. Additionally, if this allocation also fails, attempt a 64-bit +* allocation. If the 64-bit MSI address is allocated, then the EPs +* supporting 32-bit MSI address will not work. */ +if (pp->msi_data) +return 0 ; + ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 ));if (ret)dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" );
@@ -385,9 +393,15 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); -dw_pcie_free_msi (pp); -return -ENOMEM ; +dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" ); +dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL ); +if (!msi_vaddr) { +dev_err (dev,"Failed to alloc and map MSI data\n" ); +dw_pcie_free_msi (pp); +return -ENOMEM ; + } }return 0 ;
Signature · 1 line
2.43.0.594.gd9cf4e227d-goog
Quoted 32 lines · Ajay Agarwal
There can be platforms that do not use/have 32-bit DMA addresses but want to enumerate endpoints which support only 32-bit MSI address. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.
If there vendor driver has already setup the MSI address using some mechanism, use the same. This method can be used by the platforms described above to support EPs they wish to.
Else, if the memory region is not reserved, try to allocate a 32-bit IOVA. Additionally, if this allocation also fails, attempt a 64-bit allocation for probe to be successful. If the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address will not work.
Signed-off-by Ajay Agarwal <[email protected]>---
Changelog since v2:
- If the vendor driver has setup themsi_data, use the sameChangelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails.../pci/controller/dwc/pcie-designware-host.c | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/pci/controller/dwc/pcie-designware-host.c +0
-0
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 380
Quoted 7 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-3@@ -374,10 +374,18 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)* order not to miss MSI TLPs from those devices the MSI target * address has to be within the lowest 4GB. * -* Note until there is a better alternative found the reservation is -* done by allocating from the artificially limited DMA-coherent -* memory.
Why do you keep deleting this statement? The driver still uses the DMA-coherent memory as a workaround. Your solution doesn't solve the problem completely. This is another workaround. One more time: the correct solution would be to allocate a 32-bit address or some range within the 4GB PCIe bus memory with no _RAM_ or some other IO behind. Your solution relies on the platform firmware/glue-driver doing that, which isn't universally applicable. So please don't drop the comment.
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 379
Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* Check if the vendor driver has setup the MSI address already. If yes, +* pick up the same.
This is inferred from the code below. So drop it.
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 380
Quoted 3 lines
This will be helpful for platforms that do not
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* use/have 32-bit DMA addresses but want to use endpoints which support +* only 32-bit MSI address.
Please merge it into the first part of the comment as like: "Permit the platforms to override the MSI target address if they have a free PCIe-bus memory specifically reserved for that."
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 384
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4
-0+* Else, if the memory region is not reserved, try to allocate a 32-bit +* IOVA. Additionally, if this allocation also fails, attempt a 64-bit +* allocation. If the 64-bit MSI address is allocated, then the EPs +* supporting 32-bit MSI address will not work.
This is easily inferred from the code below. So drop it.
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 386
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0*/ +if (pp->msi_data)
Note this is a physical address for which even zero value might be valid. In this case it's the address of the PCIe bus space for which AFAICS zero isn't reserved for something special.
Quoted 21 lines
drivers/pci/controller/dwc/pcie-designware-host.c +11
-3+return 0 ; + ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 ));if (ret)dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" );@@ -385,9 +393,15 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); -dw_pcie_free_msi (pp); -return -ENOMEM ; +dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" ); +dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL ); +if (!msi_vaddr) { +dev_err (dev,"Failed to alloc and map MSI data\n" ); +dw_pcie_free_msi (pp); +return -ENOMEM ; + }
Quoted 3 lines · Robin Murphy
Yeah, something like that. Personally I'd still be tempted to try some mildly more involved logic to just have a single
dev_warn(), but I think that's less important than just having something which clearly works.
I guess this can be done but in a bit clumsy way. Like this:
ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)) ||
!dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL);
if (ret) {
dev_warn(dev, "Failed to allocate 32-bit MSI target address\n");
dma_set_coherent_mask(dev, DMA_BIT_MASK(64));
ret = !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL);
if (ret) {
dev_err(dev, "Failed to allocate MSI target address\n");
return -ENOMEM;
}
}Not sure whether it's much better than what Ajay suggested but at least it has a single warning string describing the error, and we can drop the unused msi_vaddr variable.
-Serge(y)
Quoted 6 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-0}return 0 ;Signature · 1 line
2.43.0.594.gd9cf4e227d-goog
Quoted 37 lines · Serge Semin
Quoted 35 lines · Ajay Agarwal
There can be platforms that do not use/have 32-bit DMA addresses but want to enumerate endpoints which support only 32-bit MSI address. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.
If there vendor driver has already setup the MSI address using some mechanism, use the same. This method can be used by the platforms described above to support EPs they wish to.
Else, if the memory region is not reserved, try to allocate a 32-bit IOVA. Additionally, if this allocation also fails, attempt a 64-bit allocation for probe to be successful. If the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address will not work.
Signed-off-by Ajay Agarwal <[email protected]>---
Changelog since v2:
- If the vendor driver has setup themsi_data, use the sameChangelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails.../pci/controller/dwc/pcie-designware-host.c | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/pci/controller/dwc/pcie-designware-host.c +0
-0@@ -374,10 +374,18 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)* order not to miss MSI TLPs from those devices the MSI target * address has to be within the lowest 4GB. *
Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 379
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-3-* Note until there is a better alternative found the reservation is -* done by allocating from the artificially limited DMA-coherent -* memory. Why do you keep deleting this statement? The driver still uses the DMA-coherent memory as a workaround. Your solution doesn't solve the problem completely. This is another workaround. One more time: the correct solution would be to allocate a 32-bit address or some range within the 4GB PCIe bus memory with no _RAM_ or some other IO behind. Your solution relies on the platform
firmware/glue-driverdoing that, which isn't universally applicable. So please don't drop the comment.
ACK.
Quoted 5 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 378
Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* Check if the vendor driver has setup the MSI address already. If yes, +* pick up the same. This is inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 380
Quoted 3 lines
This will be helpful for platforms that do not
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* use/have 32-bit DMA addresses but want to use endpoints which support +* only 32-bit MSI address. Please merge it into the first part of the comment as like: "Permit the platforms to override the MSI target address if they have a free PCIe-bus memory specifically reserved for that."
ACK.
Quoted 7 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 384
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4
-0+* Else, if the memory region is not reserved, try to allocate a 32-bit +* IOVA. Additionally, if this allocation also fails, attempt a 64-bit +* allocation. If the 64-bit MSI address is allocated, then the EPs +* supporting 32-bit MSI address will not work. This is easily inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 386
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0*/ +if (pp->msi_data)Note this is a physical address for which even zero value might be valid. In this case it's the address of the PCIe bus space for which AFAICS zero isn't reserved for something special.
That is a fair point. What do you suggest we do? Shall we define another op set_msi_data (like init/msi_init/start_link) and if it is defined by the vendor, then call it? Then vendor has to set the pp->msi_data there? Let me know.
Quoted 29 lines
Quoted 21 lines
drivers/pci/controller/dwc/pcie-designware-host.c +11
-3+return 0 ; + ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 ));if (ret)dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" );@@ -385,9 +393,15 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); -dw_pcie_free_msi (pp); -return -ENOMEM ; +dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" ); +dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL ); +if (!msi_vaddr) { +dev_err (dev,"Failed to alloc and map MSI data\n" ); +dw_pcie_free_msi (pp); +return -ENOMEM ; + }Quoted 3 lines · Robin Murphy
Yeah, something like that. Personally I'd still be tempted to try some mildly more involved logic to just have a single
dev_warn(), but I think that's less important than just having something which clearly works.I guess this can be done but in a bit clumsy way. Like this:
Quoted 10 lines
ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)) ||
!dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL);
if (ret) {
dev_warn(dev, "Failed to allocate 32-bit MSI target address\n");
dma_set_coherent_mask(dev, DMA_BIT_MASK(64));
ret = !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL);
if (ret) {
dev_err(dev, "Failed to allocate MSI target address\n");
return -ENOMEM;As you pointed out already, this looks pretty clumsy. I think we should stick to the more descriptive and readable code that I suggested.
Quoted 15 lines
} }Not sure whether it's much better than what Ajay suggested but at least it has a single warning string describing the error, and we can drop the unused
msi_vaddrvariable.-Serge(y)
Quoted 6 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-0}return 0 ;Signature · 1 line
2.43.0.594.gd9cf4e227d-goog
Quoted 87 lines · Ajay Agarwal
Quoted 37 lines · Serge Semin
Quoted 35 lines · Ajay Agarwal
There can be platforms that do not use/have 32-bit DMA addresses but want to enumerate endpoints which support only 32-bit MSI address. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.
If there vendor driver has already setup the MSI address using some mechanism, use the same. This method can be used by the platforms described above to support EPs they wish to.
Else, if the memory region is not reserved, try to allocate a 32-bit IOVA. Additionally, if this allocation also fails, attempt a 64-bit allocation for probe to be successful. If the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address will not work.
Signed-off-by Ajay Agarwal <[email protected]>---
Changelog since v2:
- If the vendor driver has setup themsi_data, use the sameChangelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails.../pci/controller/dwc/pcie-designware-host.c | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/pci/controller/dwc/pcie-designware-host.c +0
-0@@ -374,10 +374,18 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)* order not to miss MSI TLPs from those devices the MSI target * address has to be within the lowest 4GB. * Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 379
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-3-* Note until there is a better alternative found the reservation is -* done by allocating from the artificially limited DMA-coherent -* memory. Why do you keep deleting this statement? The driver still uses the DMA-coherent memory as a workaround. Your solution doesn't solve the problem completely. This is another workaround. One more time: the correct solution would be to allocate a 32-bit address or some range within the 4GB PCIe bus memory with no _RAM_ or some other IO behind. Your solution relies on the platform
firmware/glue-driverdoing that, which isn't universally applicable. So please don't drop the comment.ACK.
Quoted 5 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 378
Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* Check if the vendor driver has setup the MSI address already. If yes, +* pick up the same. This is inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 380
Quoted 3 lines
This will be helpful for platforms that do not
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* use/have 32-bit DMA addresses but want to use endpoints which support +* only 32-bit MSI address. Please merge it into the first part of the comment as like: "Permit the platforms to override the MSI target address if they have a free PCIe-bus memory specifically reserved for that."
ACK.
Quoted 7 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 384
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4
-0+* Else, if the memory region is not reserved, try to allocate a 32-bit +* IOVA. Additionally, if this allocation also fails, attempt a 64-bit +* allocation. If the 64-bit MSI address is allocated, then the EPs +* supporting 32-bit MSI address will not work. This is easily inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 386
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0*/ +if (pp->msi_data)Note this is a physical address for which even zero value might be valid. In this case it's the address of the PCIe bus space for which AFAICS zero isn't reserved for something special.
Quoted 4 lines
That is a fair point. What do you suggest we do? Shall we define another op
set_msi_data(likeinit/msi_init/start_link) and if it is defined by the vendor, then call it? Then vendor has to set thepp->msi_datathere? Let me know.
You can define a new capability flag here drivers/pci/controller/dwc/pcie-designware.h (see DW_PCIE_CAP_* macros) , set it in the glue driver by means of the dw_pcie_cap_set() macro function and instead of checking msi_data value test the flag for being set by dw_pcie_cap_is().
Quoted 35 lines
Quoted 35 lines
Quoted 22 lines
drivers/pci/controller/dwc/pcie-designware-host.c +11
-3+return 0 ; + ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 ));if (ret)dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" );@@ -385,9 +393,15 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); -dw_pcie_free_msi (pp); -return -ENOMEM ; +dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" ); +dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL ); +if (!msi_vaddr) { +dev_err (dev,"Failed to alloc and map MSI data\n" ); +dw_pcie_free_msi (pp); +return -ENOMEM ; + }Quoted 3 lines · Robin Murphy
Yeah, something like that. Personally I'd still be tempted to try some mildly more involved logic to just have a single
dev_warn(), but I think that's less important than just having something which clearly works.I guess this can be done but in a bit clumsy way. Like this:
ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)) || !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_warn(dev, "Failed to allocate 32-bit MSI target address\n");
Quoted 8 lines
Quoted 5 lines
dma_set_coherent_mask(dev, DMA_BIT_MASK(64)); ret = !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_err(dev, "Failed to allocate MSI target address\n"); return -ENOMEM;As you pointed out already, this looks pretty clumsy. I think we should stick to the more descriptive and readable code that I suggested.
I do not know which solution is better really. Both have pros and cons. Let's wait for Bjorn, Mani or Robin opinion about this.
-Serge(y)
Quoted 16 lines
Quoted 16 lines
} }Not sure whether it's much better than what Ajay suggested but at least it has a single warning string describing the error, and we can drop the unused
msi_vaddrvariable.-Serge(y)
Quoted 6 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-0}return 0 ;Signature · 1 line
2.43.0.594.gd9cf4e227d-goog
Quoted 93 lines · Serge Semin
Quoted 87 lines · Ajay Agarwal
Quoted 37 lines · Serge Semin
Quoted 35 lines · Ajay Agarwal
There can be platforms that do not use/have 32-bit DMA addresses but want to enumerate endpoints which support only 32-bit MSI address. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.
If there vendor driver has already setup the MSI address using some mechanism, use the same. This method can be used by the platforms described above to support EPs they wish to.
Else, if the memory region is not reserved, try to allocate a 32-bit IOVA. Additionally, if this allocation also fails, attempt a 64-bit allocation for probe to be successful. If the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address will not work.
Signed-off-by Ajay Agarwal <[email protected]>---
Changelog since v2:
- If the vendor driver has setup themsi_data, use the sameChangelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails.../pci/controller/dwc/pcie-designware-host.c | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/pci/controller/dwc/pcie-designware-host.c +0
-0@@ -374,10 +374,18 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)* order not to miss MSI TLPs from those devices the MSI target * address has to be within the lowest 4GB. * Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 379
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-3-* Note until there is a better alternative found the reservation is -* done by allocating from the artificially limited DMA-coherent -* memory. Why do you keep deleting this statement? The driver still uses the DMA-coherent memory as a workaround. Your solution doesn't solve the problem completely. This is another workaround. One more time: the correct solution would be to allocate a 32-bit address or some range within the 4GB PCIe bus memory with no _RAM_ or some other IO behind. Your solution relies on the platform
firmware/glue-driverdoing that, which isn't universally applicable. So please don't drop the comment.ACK.
Quoted 5 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 378
Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* Check if the vendor driver has setup the MSI address already. If yes, +* pick up the same. This is inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 380
Quoted 3 lines
This will be helpful for platforms that do not
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* use/have 32-bit DMA addresses but want to use endpoints which support +* only 32-bit MSI address. Please merge it into the first part of the comment as like: "Permit the platforms to override the MSI target address if they have a free PCIe-bus memory specifically reserved for that."
ACK.
Quoted 7 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 384
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4
-0+* Else, if the memory region is not reserved, try to allocate a 32-bit +* IOVA. Additionally, if this allocation also fails, attempt a 64-bit +* allocation. If the 64-bit MSI address is allocated, then the EPs +* supporting 32-bit MSI address will not work. This is easily inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 386
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0*/ +if (pp->msi_data)Note this is a physical address for which even zero value might be valid. In this case it's the address of the PCIe bus space for which AFAICS zero isn't reserved for something special.
Quoted 4 lines
That is a fair point. What do you suggest we do? Shall we define another op
set_msi_data(likeinit/msi_init/start_link) and if it is defined by the vendor, then call it? Then vendor has to set thepp->msi_datathere? Let me know.
Quoted 6 lines
You can define a new capability flag here
drivers/pci/controller/dwc/pcie-designware.h(see DW_PCIE_CAP_* macros) , set it in the glue driver by means of thedw_pcie_cap_set()macro function and instead of checkingmsi_datavalue test the flag for being set bydw_pcie_cap_is().
Sure, good suggestion. ACK.
Quoted 41 lines
Quoted 41 lines
Quoted 40 lines
Quoted 22 lines
drivers/pci/controller/dwc/pcie-designware-host.c +11
-3+return 0 ; + ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 ));if (ret)dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" );@@ -385,9 +393,15 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); -dw_pcie_free_msi (pp); -return -ENOMEM ; +dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" ); +dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL ); +if (!msi_vaddr) { +dev_err (dev,"Failed to alloc and map MSI data\n" ); +dw_pcie_free_msi (pp); +return -ENOMEM ; + }Quoted 3 lines · Robin Murphy
Yeah, something like that. Personally I'd still be tempted to try some mildly more involved logic to just have a single
dev_warn(), but I think that's less important than just having something which clearly works.I guess this can be done but in a bit clumsy way. Like this:
ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)) || !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_warn(dev, "Failed to allocate 32-bit MSI target address\n"); dma_set_coherent_mask(dev, DMA_BIT_MASK(64)); ret = !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_err(dev, "Failed to allocate MSI target address\n"); return -ENOMEM;
Quoted 8 lines
Quoted 2 lines
As you pointed out already, this looks pretty clumsy. I think we should stick to the more descriptive and readable code that I suggested.
I do not know which solution is better really. Both have pros and cons. Let's wait for Bjorn, Mani or Robin opinion about this.
-Serge(y)
Bjorn/Mani/Robin,
Can you please provide your comment?
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
} }Not sure whether it's much better than what Ajay suggested but at least it has a single warning string describing the error, and we can drop the unused
msi_vaddrvariable.-Serge(y)
Quoted 6 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-0}return 0 ;Signature · 1 line
2.43.0.594.gd9cf4e227d-goog
Quoted 146 lines · Ajay Agarwal
Quoted 93 lines · Serge Semin
Quoted 87 lines · Ajay Agarwal
Quoted 37 lines · Serge Semin
Quoted 35 lines · Ajay Agarwal
There can be platforms that do not use/have 32-bit DMA addresses but want to enumerate endpoints which support only 32-bit MSI address. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.
If there vendor driver has already setup the MSI address using some mechanism, use the same. This method can be used by the platforms described above to support EPs they wish to.
Else, if the memory region is not reserved, try to allocate a 32-bit IOVA. Additionally, if this allocation also fails, attempt a 64-bit allocation for probe to be successful. If the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address will not work.
Signed-off-by Ajay Agarwal <[email protected]>---
Changelog since v2:
- If the vendor driver has setup themsi_data, use the sameChangelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails.../pci/controller/dwc/pcie-designware-host.c | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/pci/controller/dwc/pcie-designware-host.c +0
-0@@ -374,10 +374,18 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)* order not to miss MSI TLPs from those devices the MSI target * address has to be within the lowest 4GB. * Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 379
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-3-* Note until there is a better alternative found the reservation is -* done by allocating from the artificially limited DMA-coherent -* memory. Why do you keep deleting this statement? The driver still uses the DMA-coherent memory as a workaround. Your solution doesn't solve the problem completely. This is another workaround. One more time: the correct solution would be to allocate a 32-bit address or some range within the 4GB PCIe bus memory with no _RAM_ or some other IO behind. Your solution relies on the platform
firmware/glue-driverdoing that, which isn't universally applicable. So please don't drop the comment.ACK.
Quoted 5 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 378
Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* Check if the vendor driver has setup the MSI address already. If yes, +* pick up the same. This is inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c
Quoted 3 lines
This will be helpful for platforms that do not
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* use/have 32-bit DMA addresses but want to use endpoints which support +* only 32-bit MSI address. Please merge it into the first part of the comment as like: "Permit the platforms to override the MSI target address if they have a free PCIe-bus memory specifically reserved for that."
ACK.
Quoted 7 lines
drivers/pci/controller/dwc/pcie-designware-host.c
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4
-0+* Else, if the memory region is not reserved, try to allocate a 32-bit +* IOVA. Additionally, if this allocation also fails, attempt a 64-bit +* allocation. If the 64-bit MSI address is allocated, then the EPs +* supporting 32-bit MSI address will not work. This is easily inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c
Quoted 3 lines
*/
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+if (pp->msi_data)Note this is a physical address for which even zero value might be valid. In this case it's the address of the PCIe bus space for which AFAICS zero isn't reserved for something special.
Quoted 4 lines
That is a fair point. What do you suggest we do? Shall we define another op
set_msi_data(likeinit/msi_init/start_link) and if it is defined by the vendor, then call it? Then vendor has to set thepp->msi_datathere? Let me know.Quoted 6 lines
You can define a new capability flag here
drivers/pci/controller/dwc/pcie-designware.h(see DW_PCIE_CAP_* macros) , set it in the glue driver by means of thedw_pcie_cap_set()macro function and instead of checkingmsi_datavalue test the flag for being set bydw_pcie_cap_is().Sure, good suggestion. ACK.
Quoted 44 lines
Quoted 43 lines
Quoted 30 lines
Quoted 22 lines
drivers/pci/controller/dwc/pcie-designware-host.c +11
-3+return 0 ; + ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 ));if (ret)dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" );@@ -385,9 +393,15 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); -dw_pcie_free_msi (pp); -return -ENOMEM ; +dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" ); +dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL ); +if (!msi_vaddr) { +dev_err (dev,"Failed to alloc and map MSI data\n" ); +dw_pcie_free_msi (pp); +return -ENOMEM ; + }Quoted 3 lines · Robin Murphy
Yeah, something like that. Personally I'd still be tempted to try some mildly more involved logic to just have a single
dev_warn(), but I think that's less important than just having something which clearly works.I guess this can be done but in a bit clumsy way. Like this:
Quoted 10 lines
ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)) || !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_warn(dev, "Failed to allocate 32-bit MSI target address\n"); dma_set_coherent_mask(dev, DMA_BIT_MASK(64)); ret = !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_err(dev, "Failed to allocate MSI target address\n"); return -ENOMEM;As you pointed out already, this looks pretty clumsy. I think we should stick to the more descriptive and readable code that I suggested.
Quoted 7 lines
Quoted 5 lines
I do not know which solution is better really. Both have pros and cons. Let's wait for Bjorn, Mani or Robin opinion about this.
-Serge(y)
Bjorn/Mani/Robin,
Can you please provide your comment?
FWIW I had a go at sketching out what I think I'd do, on top of the v3 patch. Apparently I'm not in a too-clever-for-my-own-good mood today, since what came out seems to have ended up pretty much just simplifying the pre-existing code. I'll leave the choice up to you.
Thanks,
Robin.
----->8-----
drivers/pci/controller/dwc/pcie-designware-host.c +6 -8
@@ -328,7 +328,7 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)struct dw_pcie *pci =to_dw_pcie_from_pp (pp);struct device *dev = pci->dev;struct platform_device *pdev =to_platform_device (dev); -u64 *msi_vaddr; +u64 *msi_vaddr =NULL ;int ret;u32 ctrl, num_ctrls;
@@ -387,18 +387,16 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)return 0 ; ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 )); -if (ret) -dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" ); - - msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, -GFP_KERNEL ); +if (!ret) + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL );if (!msi_vaddr) { -dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" ); +dev_warn (dev,"Failed to configure 32-bit MSI address. Devices with only 32-bit MSI support may not work properly\n" );dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); +dev_err (dev,"Failed to configure MSI address\n" );dw_pcie_free_msi (pp);return -ENOMEM ; }
Quoted 189 lines · Robin Murphy
Quoted 146 lines · Ajay Agarwal
Quoted 93 lines · Serge Semin
Quoted 87 lines · Ajay Agarwal
Quoted 37 lines · Serge Semin
Quoted 35 lines · Ajay Agarwal
There can be platforms that do not use/have 32-bit DMA addresses but want to enumerate endpoints which support only 32-bit MSI address. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.
If there vendor driver has already setup the MSI address using some mechanism, use the same. This method can be used by the platforms described above to support EPs they wish to.
Else, if the memory region is not reserved, try to allocate a 32-bit IOVA. Additionally, if this allocation also fails, attempt a 64-bit allocation for probe to be successful. If the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address will not work.
Signed-off-by Ajay Agarwal <[email protected]>---
Changelog since v2:
- If the vendor driver has setup themsi_data, use the sameChangelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails.../pci/controller/dwc/pcie-designware-host.c | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/pci/controller/dwc/pcie-designware-host.c +0
-0@@ -374,10 +374,18 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)* order not to miss MSI TLPs from those devices the MSI target * address has to be within the lowest 4GB. * Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 379
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-3-* Note until there is a better alternative found the reservation is -* done by allocating from the artificially limited DMA-coherent -* memory. Why do you keep deleting this statement? The driver still uses the DMA-coherent memory as a workaround. Your solution doesn't solve the problem completely. This is another workaround. One more time: the correct solution would be to allocate a 32-bit address or some range within the 4GB PCIe bus memory with no _RAM_ or some other IO behind. Your solution relies on the platform
firmware/glue-driverdoing that, which isn't universally applicable. So please don't drop the comment.ACK.
Quoted 5 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 378
Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* Check if the vendor driver has setup the MSI address already. If yes, +* pick up the same. This is inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c
Quoted 3 lines
This will be helpful for platforms that do not
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* use/have 32-bit DMA addresses but want to use endpoints which support +* only 32-bit MSI address. Please merge it into the first part of the comment as like: "Permit the platforms to override the MSI target address if they have a free PCIe-bus memory specifically reserved for that."
ACK.
Quoted 7 lines
drivers/pci/controller/dwc/pcie-designware-host.c
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4
-0+* Else, if the memory region is not reserved, try to allocate a 32-bit +* IOVA. Additionally, if this allocation also fails, attempt a 64-bit +* allocation. If the 64-bit MSI address is allocated, then the EPs +* supporting 32-bit MSI address will not work. This is easily inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c
Quoted 3 lines
*/
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+if (pp->msi_data)Note this is a physical address for which even zero value might be valid. In this case it's the address of the PCIe bus space for which AFAICS zero isn't reserved for something special.
Quoted 4 lines
That is a fair point. What do you suggest we do? Shall we define another op
set_msi_data(likeinit/msi_init/start_link) and if it is defined by the vendor, then call it? Then vendor has to set thepp->msi_datathere? Let me know.Quoted 6 lines
You can define a new capability flag here
drivers/pci/controller/dwc/pcie-designware.h(see DW_PCIE_CAP_* macros) , set it in the glue driver by means of thedw_pcie_cap_set()macro function and instead of checkingmsi_datavalue test the flag for being set bydw_pcie_cap_is().Sure, good suggestion. ACK.
Quoted 44 lines
Quoted 43 lines
Quoted 30 lines
Quoted 22 lines
drivers/pci/controller/dwc/pcie-designware-host.c +11
-3+return 0 ; + ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 ));if (ret)dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" );@@ -385,9 +393,15 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); -dw_pcie_free_msi (pp); -return -ENOMEM ; +dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" ); +dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL ); +if (!msi_vaddr) { +dev_err (dev,"Failed to alloc and map MSI data\n" ); +dw_pcie_free_msi (pp); +return -ENOMEM ; + }Quoted 3 lines · Robin Murphy
Yeah, something like that. Personally I'd still be tempted to try some mildly more involved logic to just have a single
dev_warn(), but I think that's less important than just having something which clearly works.I guess this can be done but in a bit clumsy way. Like this:
Quoted 10 lines
ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)) || !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_warn(dev, "Failed to allocate 32-bit MSI target address\n"); dma_set_coherent_mask(dev, DMA_BIT_MASK(64)); ret = !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_err(dev, "Failed to allocate MSI target address\n"); return -ENOMEM;As you pointed out already, this looks pretty clumsy. I think we should stick to the more descriptive and readable code that I suggested.
Quoted 7 lines
Quoted 5 lines
I do not know which solution is better really. Both have pros and cons. Let's wait for Bjorn, Mani or Robin opinion about this.
-Serge(y)
Bjorn/Mani/Robin,
Can you please provide your comment?FWIW I had a go at sketching out what I think I'd do, on top of the v3 patch. Apparently I'm not in a too-clever-for-my-own-good mood today, since what came out seems to have ended up pretty much just simplifying the pre-existing code. I'll leave the choice up to you.
Thanks,
Robin.----->8-----
drivers/pci/controller/dwc/pcie-designware-host.c +4
-7@@ -328,7 +328,7 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)struct dw_pcie *pci =to_dw_pcie_from_pp (pp);struct device *dev = pci->dev;struct platform_device *pdev =to_platform_device (dev); -u64 *msi_vaddr; +u64 *msi_vaddr =NULL ;int ret;u32 ctrl, num_ctrls;@@ -387,18 +387,16 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)return 0 ; ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 )); -if (ret) -dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" ); - - msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, -GFP_KERNEL ); +if (!ret) + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL );if (!msi_vaddr) { -dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" );
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 401
Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-1+dev_warn (dev,"Failed to configure 32-bit MSI address. Devices with only 32-bit MSI support may not work properly\n" );dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); +dev_err (dev,"Failed to configure MSI address\n" );dw_pcie_free_msi (pp);return -ENOMEM ; }
Thanks Robin for your suggestion. I will keep my patch but pick up the dev_err prints from your suggestion.
Manivannan Sadhasivam
maintainerQuoted 146 lines · Ajay Agarwal
Quoted 93 lines · Serge Semin
Quoted 87 lines · Ajay Agarwal
Quoted 37 lines · Serge Semin
Quoted 35 lines · Ajay Agarwal
There can be platforms that do not use/have 32-bit DMA addresses but want to enumerate endpoints which support only 32-bit MSI address. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.
If there vendor driver has already setup the MSI address using some mechanism, use the same. This method can be used by the platforms described above to support EPs they wish to.
Else, if the memory region is not reserved, try to allocate a 32-bit IOVA. Additionally, if this allocation also fails, attempt a 64-bit allocation for probe to be successful. If the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address will not work.
Signed-off-by Ajay Agarwal <[email protected]>---
Changelog since v2:
- If the vendor driver has setup themsi_data, use the sameChangelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails.../pci/controller/dwc/pcie-designware-host.c | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/pci/controller/dwc/pcie-designware-host.c +0
-0@@ -374,10 +374,18 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)* order not to miss MSI TLPs from those devices the MSI target * address has to be within the lowest 4GB. * Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 379
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-3-* Note until there is a better alternative found the reservation is -* done by allocating from the artificially limited DMA-coherent -* memory. Why do you keep deleting this statement? The driver still uses the DMA-coherent memory as a workaround. Your solution doesn't solve the problem completely. This is another workaround. One more time: the correct solution would be to allocate a 32-bit address or some range within the 4GB PCIe bus memory with no _RAM_ or some other IO behind. Your solution relies on the platform
firmware/glue-driverdoing that, which isn't universally applicable. So please don't drop the comment.ACK.
Quoted 5 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 378
Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* Check if the vendor driver has setup the MSI address already. If yes, +* pick up the same. This is inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 380
Quoted 3 lines
This will be helpful for platforms that do not
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* use/have 32-bit DMA addresses but want to use endpoints which support +* only 32-bit MSI address. Please merge it into the first part of the comment as like: "Permit the platforms to override the MSI target address if they have a free PCIe-bus memory specifically reserved for that."
ACK.
Quoted 7 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 384
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4
-0+* Else, if the memory region is not reserved, try to allocate a 32-bit +* IOVA. Additionally, if this allocation also fails, attempt a 64-bit +* allocation. If the 64-bit MSI address is allocated, then the EPs +* supporting 32-bit MSI address will not work. This is easily inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 386
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0*/ +if (pp->msi_data)Note this is a physical address for which even zero value might be valid. In this case it's the address of the PCIe bus space for which AFAICS zero isn't reserved for something special.
Quoted 4 lines
That is a fair point. What do you suggest we do? Shall we define another op
set_msi_data(likeinit/msi_init/start_link) and if it is defined by the vendor, then call it? Then vendor has to set thepp->msi_datathere? Let me know.Quoted 6 lines
You can define a new capability flag here
drivers/pci/controller/dwc/pcie-designware.h(see DW_PCIE_CAP_* macros) , set it in the glue driver by means of thedw_pcie_cap_set()macro function and instead of checkingmsi_datavalue test the flag for being set bydw_pcie_cap_is().Sure, good suggestion. ACK.
Quoted 44 lines
Quoted 43 lines
Quoted 30 lines
Quoted 22 lines
drivers/pci/controller/dwc/pcie-designware-host.c +11
-3+return 0 ; + ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 ));if (ret)dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" );@@ -385,9 +393,15 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); -dw_pcie_free_msi (pp); -return -ENOMEM ; +dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" ); +dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL ); +if (!msi_vaddr) { +dev_err (dev,"Failed to alloc and map MSI data\n" ); +dw_pcie_free_msi (pp); +return -ENOMEM ; + }Quoted 3 lines · Robin Murphy
Yeah, something like that. Personally I'd still be tempted to try some mildly more involved logic to just have a single
dev_warn(), but I think that's less important than just having something which clearly works.I guess this can be done but in a bit clumsy way. Like this:
Quoted 10 lines
ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)) || !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_warn(dev, "Failed to allocate 32-bit MSI target address\n"); dma_set_coherent_mask(dev, DMA_BIT_MASK(64)); ret = !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_err(dev, "Failed to allocate MSI target address\n"); return -ENOMEM;As you pointed out already, this looks pretty clumsy. I think we should stick to the more descriptive and readable code that I suggested.
Quoted 8 lines
Quoted 5 lines
I do not know which solution is better really. Both have pros and cons. Let's wait for Bjorn, Mani or Robin opinion about this.
-Serge(y)
Bjorn/Mani/Robin,
Can you please provide your comment?
Sergey's suggestion masks out the error coming from "dma_set_coherent_mask()" by using the same error log. This is equivalent to printing the same error log for both API failures:
ret = dma_set_coherent_mask() if (ret) dev_warn(dev, "Failed to allocate 32-bit MSI target address\n"); ret = dmam_alloc_coherent() if (ret) dev_warn(dev, "Failed to allocate 32-bit MSI target address\n");
Which doesn't look correct to me. So let's stick to what Ajay had initially.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 168 lines · Manivannan Sadhasivam
Quoted 146 lines · Ajay Agarwal
Quoted 93 lines · Serge Semin
Quoted 87 lines · Ajay Agarwal
Quoted 37 lines · Serge Semin
Quoted 35 lines · Ajay Agarwal
There can be platforms that do not use/have 32-bit DMA addresses but want to enumerate endpoints which support only 32-bit MSI address. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.
If there vendor driver has already setup the MSI address using some mechanism, use the same. This method can be used by the platforms described above to support EPs they wish to.
Else, if the memory region is not reserved, try to allocate a 32-bit IOVA. Additionally, if this allocation also fails, attempt a 64-bit allocation for probe to be successful. If the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address will not work.
Signed-off-by Ajay Agarwal <[email protected]>---
Changelog since v2:
- If the vendor driver has setup themsi_data, use the sameChangelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails.../pci/controller/dwc/pcie-designware-host.c | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-)drivers/pci/controller/dwc/pcie-designware-host.c +0
-0@@ -374,10 +374,18 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)* order not to miss MSI TLPs from those devices the MSI target * address has to be within the lowest 4GB. * Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 379
Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0
-3-* Note until there is a better alternative found the reservation is -* done by allocating from the artificially limited DMA-coherent -* memory. Why do you keep deleting this statement? The driver still uses the DMA-coherent memory as a workaround. Your solution doesn't solve the problem completely. This is another workaround. One more time: the correct solution would be to allocate a 32-bit address or some range within the 4GB PCIe bus memory with no _RAM_ or some other IO behind. Your solution relies on the platform
firmware/glue-driverdoing that, which isn't universally applicable. So please don't drop the comment.ACK.
Quoted 5 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 378
Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* Check if the vendor driver has setup the MSI address already. If yes, +* pick up the same. This is inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 380
Quoted 3 lines
This will be helpful for platforms that do not
drivers/pci/controller/dwc/pcie-designware-host.c +2
-0+* use/have 32-bit DMA addresses but want to use endpoints which support +* only 32-bit MSI address. Please merge it into the first part of the comment as like: "Permit the platforms to override the MSI target address if they have a free PCIe-bus memory specifically reserved for that."
ACK.
Quoted 7 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 384
Quoted 4 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4
-0+* Else, if the memory region is not reserved, try to allocate a 32-bit +* IOVA. Additionally, if this allocation also fails, attempt a 64-bit +* allocation. If the 64-bit MSI address is allocated, then the EPs +* supporting 32-bit MSI address will not work. This is easily inferred from the code below. So drop it.
ACK.
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 386
Quoted 3 lines
*/
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+if (pp->msi_data)Note this is a physical address for which even zero value might be valid. In this case it's the address of the PCIe bus space for which AFAICS zero isn't reserved for something special.
Quoted 4 lines
That is a fair point. What do you suggest we do? Shall we define another op
set_msi_data(likeinit/msi_init/start_link) and if it is defined by the vendor, then call it? Then vendor has to set thepp->msi_datathere? Let me know.Quoted 6 lines
You can define a new capability flag here
drivers/pci/controller/dwc/pcie-designware.h(see DW_PCIE_CAP_* macros) , set it in the glue driver by means of thedw_pcie_cap_set()macro function and instead of checkingmsi_datavalue test the flag for being set bydw_pcie_cap_is().Sure, good suggestion. ACK.
Quoted 44 lines
Quoted 43 lines
Quoted 30 lines
Quoted 22 lines
drivers/pci/controller/dwc/pcie-designware-host.c +11
-3+return 0 ; + ret =dma_set_coherent_mask (dev,DMA_BIT_MASK (32 ));if (ret)dev_warn (dev,"Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n" );@@ -385,9 +393,15 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data,GFP_KERNEL );if (!msi_vaddr) { -dev_err (dev,"Failed to alloc and map MSI data\n" ); -dw_pcie_free_msi (pp); -return -ENOMEM ; +dev_warn (dev,"Failed to alloc 32-bit MSI data. Attempting 64-bit now\n" ); +dma_set_coherent_mask (dev,DMA_BIT_MASK (64 )); + msi_vaddr =dmam_alloc_coherent (dev,sizeof (u64), &pp->msi_data, +GFP_KERNEL ); +if (!msi_vaddr) { +dev_err (dev,"Failed to alloc and map MSI data\n" ); +dw_pcie_free_msi (pp); +return -ENOMEM ; + }Quoted 3 lines · Robin Murphy
Yeah, something like that. Personally I'd still be tempted to try some mildly more involved logic to just have a single
dev_warn(), but I think that's less important than just having something which clearly works.I guess this can be done but in a bit clumsy way. Like this:
Quoted 10 lines
ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)) || !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_warn(dev, "Failed to allocate 32-bit MSI target address\n"); dma_set_coherent_mask(dev, DMA_BIT_MASK(64)); ret = !dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (ret) { dev_err(dev, "Failed to allocate MSI target address\n"); return -ENOMEM;As you pointed out already, this looks pretty clumsy. I think we should stick to the more descriptive and readable code that I suggested.
Quoted 8 lines
Quoted 5 lines
I do not know which solution is better really. Both have pros and cons. Let's wait for Bjorn, Mani or Robin opinion about this.
-Serge(y)
Bjorn/Mani/Robin,
Can you please provide your comment?Sergey's suggestion masks out the error coming from "
dma_set_coherent_mask()" by using the same error log. This is equivalent to printing the same error log for both API failures:ret = dma_set_coherent_mask() if (ret) dev_warn(dev, "Failed to allocate 32-bit MSI target address\n"); ret = dmam_alloc_coherent() if (ret) dev_warn(dev, "Failed to allocate 32-bit MSI target address\n");
Quoted 1 line
Which doesn't look correct to me. So let's stick to what Ajay had initially.
My approach did the same, and it was intentional - the significant information for the user is that some devices may go wrong because we couldn't support 32-bit MSIs; the intimate details of exactly *why* we couldn't support 32-bit MSIs don't really matter, and are unlikely to be actionable. Furthermore, setting a 32-bit DMA mask should never fail on any reasonable platform where this driver would actually run, so there's no practical value in anticipating it - handling that particular return value is really just a matter of logical correctness.
Thanks,
Robin.