dma: mark unmapped DMA scatter/gather invalid
Subsystems (2)
PATCH 1/1 dma: mark unmapped DMA scatter/gather invalid
This patch updates dma_direct_unmap_sg() to mark each scatter/gather entry invalid, after it's unmapped. This fixes two issues:
1. It makes the unmapping code able to tolerate a double unmap.
2. It prevents the NVMe driver from erroneously treating an unmapped DMA address as mapped.
The bug that motivated this patch was the following sequence, which occurred within the NVMe driver, with the kernel flag swiotlb=force.
* NVMe driver calls dma_direct_map_sg()
* dma_direct_map_sg() fails part way through the scatter gather/list
* dma_direct_map_sg() calls dma_direct_unmap_sg() to unmap any entries succeeded.
* NVMe driver calls dma_direct_unmap_sg(), redundantly, leading to a double unmap, which is a bug.
With this patch, a hadoop workload running on a cluster of three AMD SEV VMs, is able to succeed. Without the patch, the hadoop workload suffers application-level and even VM-level failures.
--- kernel/dma/direct.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/dma/direct.c +3 -1
@@ -374,9 +374,11 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl,struct scatterlist *sg;int i; -for_each_sg (sgl, sg, nents, i) +for_each_sg (sgl, sg, nents, i) {dma_direct_unmap_page (dev, sg->dma_address,sg_dma_len (sg), dir, attrs); + sg->dma_address =DMA_MAPPING_ERROR ; + } }EXPORT_SYMBOL (dma_direct_unmap_sg);#endif
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Robin Murphy
reviewerQuoted 2 lines · Marc Orr
This patch updates
dma_direct_unmap_sg()to mark each scatter/gather entry invalid, after it's unmapped. This fixes two issues:
s/fixes/bodges around (badly)/
Quoted 7 lines
1. It makes the unmapping code able to tolerate a double unmap.
2. It prevents the NVMe driver from erroneously treating an unmapped DMA address as mapped.The bug that motivated this patch was the following sequence, which occurred within the NVMe driver, with the kernel flag
swiotlb=force.
Quoted 6 lines
* NVMe driver calls
dma_direct_map_sg()
*dma_direct_map_sg()fails part way through the scatter gather/list
*dma_direct_map_sg()callsdma_direct_unmap_sg()to unmap any entries succeeded.
* NVMe driver callsdma_direct_unmap_sg(), redundantly, leading to a double unmap, which is a bug.
So why not just fix that actual bug?
Quoted 17 lines
With this patch, a hadoop workload running on a cluster of three AMD SEV VMs, is able to succeed. Without the patch, the hadoop workload suffers application-level and even VM-level failures.
Tested-by Jianxiong Gao <[email protected]>Tested-by Marc Orr <[email protected]>Reviewed-by Jianxiong Gao <[email protected]>Signed-off-by Marc Orr <[email protected]>--- kernel/dma/direct.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)kernel/dma/direct.c +0
-0
kernel/dma/direct.c void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, · line 380
Quoted 8 lines
kernel/dma/direct.c +2
-1@@ -374,9 +374,11 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl,struct scatterlist *sg;int i; -for_each_sg (sgl, sg, nents, i) +for_each_sg (sgl, sg, nents, i) {dma_direct_unmap_page (dev, sg->dma_address,sg_dma_len (sg), dir, attrs); + sg->dma_address =DMA_MAPPING_ERROR ;
There are more DMA API backends than just dma-direct, so while this might help paper over bugs when SWIOTLB is in use, it's not going to have any effect when those same bugs are hit under other circumstances. Once again, the moral of the story is that effort is better spent just fixing the bugs ;)
Robin.
kernel/dma/direct.c void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, · line 384
Quoted 5 lines
kernel/dma/direct.c +1
-0+ } }EXPORT_SYMBOL (dma_direct_unmap_sg);#endif
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Greg KH
maintainerQuoted 21 lines · Marc Orr
This patch updates
dma_direct_unmap_sg()to mark each scatter/gather entry invalid, after it's unmapped. This fixes two issues:1. It makes the unmapping code able to tolerate a double unmap.
2. It prevents the NVMe driver from erroneously treating an unmapped DMA address as mapped.The bug that motivated this patch was the following sequence, which occurred within the NVMe driver, with the kernel flag
swiotlb=force.* NVMe driver calls
dma_direct_map_sg()
*dma_direct_map_sg()fails part way through the scatter gather/list
*dma_direct_map_sg()callsdma_direct_unmap_sg()to unmap any entries succeeded.
* NVMe driver callsdma_direct_unmap_sg(), redundantly, leading to a double unmap, which is a bug.With this patch, a hadoop workload running on a cluster of three AMD SEV VMs, is able to succeed. Without the patch, the hadoop workload suffers application-level and even VM-level failures.
Quoted 7 lines
Tested-by Jianxiong Gao <[email protected]>Tested-by Marc Orr <[email protected]>Reviewed-by Jianxiong Gao <[email protected]>Signed-off-by Marc Orr <[email protected]>--- kernel/dma/direct.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
<formletter>
This is not the correct way to submit patches for inclusion in the stable kernel tree. Please read:
https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
for how to do this properly.
</formletter>
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 1 line · Marc Orr
minus [email protected], per gregkh@'s email.
Quoted 8 lines
Quoted 8 lines
kernel/dma/direct.c +0
-0@@ -374,9 +374,11 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl,struct scatterlist *sg;int i;
Quoted 11 lines
kernel/dma/direct.c void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, · line 380
Quoted 5 lines
kernel/dma/direct.c +2
-1-for_each_sg (sgl, sg, nents, i) +for_each_sg (sgl, sg, nents, i) {dma_direct_unmap_page (dev, sg->dma_address,sg_dma_len (sg), dir, attrs); + sg->dma_address =DMA_MAPPING_ERROR ;There are more DMA API backends than just dma-direct, so while this might help paper over bugs when SWIOTLB is in use, it's not going to have any effect when those same bugs are hit under other circumstances. Once again, the moral of the story is that effort is better spent just fixing the bugs ;)
Thanks for the quick feedback. What is the correct fix? I understand the first half. The NVMe driver should be updated to not call unmap on an address that has already been unmapped within the DMA direct code. Where I'm less certain is how to communicate to the NVMe driver that the mapping failed. In particular, the NVMe code explicitly checks if the first DMA address in the scatter/gather list is set to DMA_MAPPING_ERROR. Thus, don't we need to update the DMA direct code to propagate DMA_MAPPING_ERROR back up to the driver, via the scatter/gather struct?
I skimmed arch/arm/mm/dma-mapping.c, just now. I can see that this code sets the address within the scatter/gather struct to DMA_MAPPING_ERROR before trying to map an IO address and write it into the struct. Is this a good example to follow?
Thanks,
Marc
Robin Murphy
reviewerQuoted 24 lines · Marc Orr
Quoted 1 line · Marc Orr
minus [email protected], per gregkh@'s email.
Quoted 8 lines
Quoted 8 lines
kernel/dma/direct.c +0
-0@@ -374,9 +374,11 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl,struct scatterlist *sg;int i;Quoted 11 lines
kernel/dma/direct.c void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, · line 380
Quoted 5 lines
kernel/dma/direct.c +2
-1-for_each_sg (sgl, sg, nents, i) +for_each_sg (sgl, sg, nents, i) {dma_direct_unmap_page (dev, sg->dma_address,sg_dma_len (sg), dir, attrs); + sg->dma_address =DMA_MAPPING_ERROR ;There are more DMA API backends than just dma-direct, so while this might help paper over bugs when SWIOTLB is in use, it's not going to have any effect when those same bugs are hit under other circumstances. Once again, the moral of the story is that effort is better spent just fixing the bugs ;)
Quoted 9 lines
Thanks for the quick feedback. What is the correct fix? I understand the first half. The NVMe driver should be updated to not call unmap on an address that has already been unmapped within the DMA direct code. Where I'm less certain is how to communicate to the NVMe driver that the mapping failed. In particular, the NVMe code explicitly checks if the first DMA address in the scatter/gather list is set to
DMA_MAPPING_ERROR. Thus, don't we need to update the DMA direct code to propagateDMA_MAPPING_ERRORback up to the driver, via the scatter/gather struct?
Erm, you check the return value of dma_map_sg(). If it's zero, the request failed; if it's nonzero, that's how many DMA segments you now have to process. See Documentation/core-api/dma-api.rst.
The only guarantee offered about the state of the scatterlist itself is that if it is successfully mapped, then the dma_address and dma_length fields are valid for that many segments, and if that is fewer than the total number of physical segments then the next one after the final DMA segment will have a dma_length of 0. In particular there are no guarantees at all about the state if the mapping was unsuccessful.
If a driver is failing to keep track of the success/failure status and later down the line trying to guess what to do with a list that may or may not have been mapped, then frankly that driver should be redesigned because that is a terrible anti-pattern. At the very very least it should explicitly encode its own "known bad" state upon failure that it can then reliably recognise later.
Robin.
Quoted 8 lines
I skimmed
arch/arm/mm/dma-mapping.c, just now. I can see that this code sets the address within the scatter/gather struct toDMA_MAPPING_ERRORbefore trying to map an IO address and write it into the struct. Is this a good example to follow?Thanks,
Marc
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 41 lines
Quoted 20 lines
Quoted 8 lines
Quoted 8 lines
kernel/dma/direct.c +0
-0@@ -374,9 +374,11 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl,struct scatterlist *sg;int i;Quoted 11 lines
kernel/dma/direct.c void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, · line 380
Quoted 5 lines
kernel/dma/direct.c +2
-1-for_each_sg (sgl, sg, nents, i) +for_each_sg (sgl, sg, nents, i) {dma_direct_unmap_page (dev, sg->dma_address,sg_dma_len (sg), dir, attrs); + sg->dma_address =DMA_MAPPING_ERROR ;There are more DMA API backends than just dma-direct, so while this might help paper over bugs when SWIOTLB is in use, it's not going to have any effect when those same bugs are hit under other circumstances. Once again, the moral of the story is that effort is better spent just fixing the bugs ;)
Quoted 9 lines
Thanks for the quick feedback. What is the correct fix? I understand the first half. The NVMe driver should be updated to not call unmap on an address that has already been unmapped within the DMA direct code. Where I'm less certain is how to communicate to the NVMe driver that the mapping failed. In particular, the NVMe code explicitly checks if the first DMA address in the scatter/gather list is set to
DMA_MAPPING_ERROR. Thus, don't we need to update the DMA direct code to propagateDMA_MAPPING_ERRORback up to the driver, via the scatter/gather struct?Erm, you check the return value of
dma_map_sg(). If it's zero, the request failed; if it's nonzero, that's how many DMA segments you now have to process. SeeDocumentation/core-api/dma-api.rst.The only guarantee offered about the state of the scatterlist itself is that if it is successfully mapped, then the
dma_addressanddma_lengthfields are valid for that many segments, and if that is fewer than the total number of physical segments then the next one after the final DMA segment will have adma_lengthof 0. In particular there are no guarantees at all about the state if the mapping was unsuccessful.
Quoted 6 lines
If a driver is failing to keep track of the success/failure status and later down the line trying to guess what to do with a list that may or may not have been mapped, then frankly that driver should be redesigned because that is a terrible anti-pattern. At the very very least it should explicitly encode its own "known bad" state upon failure that it can then reliably recognise later.
Got it now. I'll get to work on a patch for the NVMe driver to fix the bug. Thanks for all of these pointers. They are immensely helpful.
Thanks,
Marc
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu