Skip to contentSkip to conversation

media: videobuf2-dma-sg: limit the sg segment size

PATCH In review Anle Pan 7 participants 32 messages Aug 27, 2023 – Sep 18, 2023 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore

Anle Pan

author
#0 lore raw

PATCH 1/1 media: videobuf2-dma-sg: limit the sg segment size

When allocating from pages, the size of the sg segment is unlimited and the default is UINT_MAX. This will cause the DMA stream mapping failed later with a "swiotlb buffer full" error. The default maximum mapping size is 128 slots x 2K = 256K, determined by "IO_TLB_SEGSIZE". To fix the issue, limit the sg segment size according to "dma_max_mapping_size" to match the mapping limit.

---
 drivers/media/common/videobuf2/videobuf2-dma-sg.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/media/common/videobuf2/videobuf2-dma-sg.c +7 -2
index fa69158a65b1..b608a7c5f240 100644
@@ -105,6 +105,7 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
 	struct sg_table *sgt;
 	int ret;
 	int num_pages;
+	size_t max_segment = 0;
 
 	if (WARN_ON(!dev) || WARN_ON(!size))
 		return ERR_PTR(-EINVAL);
@@ -134,8 +135,12 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
 	if (ret)
 		goto fail_pages_alloc;
 
-	ret = sg_alloc_table_from_pages(buf->dma_sgt, buf->pages,
-			buf->num_pages, 0, size, GFP_KERNEL);
+	if (dev)
+		max_segment = dma_max_mapping_size(dev);
+	if (max_segment == 0)
+		max_segment = UINT_MAX;
+	ret = sg_alloc_table_from_pages_segment(buf->dma_sgt, buf->pages,
+		buf->num_pages, 0, size, max_segment, GFP_KERNEL);
 	if (ret)
 		goto fail_table_alloc;
 
Signature · 1 line
2.25.1
maintainer
#1 lore raw

Hi Anle,

Quoted 4 lines · Anle Pan

When allocating from pages, the size of the sg segment is unlimited and the default is UINT_MAX. This will cause the DMA stream mapping failed later with a "swiotlb buffer full" error.

Thanks for the patch. Good catch.

Quoted 24 lines

The default maximum mapping
size is 128 slots x 2K = 256K, determined by "IO_TLB_SEGSIZE". To fix the issue, limit the sg segment size according to "dma_max_mapping_size" to match the mapping limit.

---
 drivers/media/common/videobuf2/videobuf2-dma-sg.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -0
index fa69158a65b1..b608a7c5f240 100644
@@ -105,6 +105,7 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
        struct sg_table *sgt;
        int ret;
        int num_pages;
+       size_t max_segment = 0;
 
        if (WARN_ON(!dev) || WARN_ON(!size))
                return ERR_PTR(-EINVAL);
@@ -134,8 +135,12 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
        if (ret)

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 143

Quoted 10 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +6 -2
                goto fail_pages_alloc;
 
-       ret = sg_alloc_table_from_pages(buf->dma_sgt, buf->pages,
-                       buf->num_pages, 0, size, GFP_KERNEL);
+       if (dev)
+               max_segment = dma_max_mapping_size(dev);
+       if (max_segment == 0)
+               max_segment = UINT_MAX;
+       ret = sg_alloc_table_from_pages_segment(buf->dma_sgt, buf->pages,
+               buf->num_pages, 0, size, max_segment, GFP_KERNEL);

One thing that I'm not sure about here is that we use sg_alloc_table_from_pages_segment(), but we actually don't pass the max segment size (as returned by dma_get_max_seg_size()) to it. I'm also not exactly sure what's the difference between "max mapping size" and "max seg size".
+Robin Murphy +Christoph Hellwig I think we could benefit from your expertise here.

Generally looking at videobuf2-dma-sg, I feel like we would benefit from some kind of dma_alloc_table_from_pages() that simply takes the struct dev pointer and does everything necessary.

Best regards,
Tomasz

reviewer
#2 lore raw
Quoted 36 lines · Tomasz Figa

Hi Anle,

Quoted 4 lines · Anle Pan

When allocating from pages, the size of the sg segment is unlimited and the default is UINT_MAX. This will cause the DMA stream mapping failed later with a "swiotlb buffer full" error.

Thanks for the patch. Good catch.

Quoted 26 lines

The default maximum mapping
size is 128 slots x 2K = 256K, determined by "IO_TLB_SEGSIZE". To fix the issue, limit the sg segment size according to "dma_max_mapping_size" to match the mapping limit.

---
  drivers/media/common/videobuf2/videobuf2-dma-sg.c | 9 +++++++--
  1 file changed, 7 insertions(+), 2 deletions(-)
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -0
index fa69158a65b1..b608a7c5f240 100644
@@ -105,6 +105,7 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
         struct sg_table *sgt;
         int ret;
         int num_pages;
+       size_t max_segment = 0;
 
         if (WARN_ON(!dev) || WARN_ON(!size))
                 return ERR_PTR(-EINVAL);
@@ -134,8 +135,12 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
         if (ret)
                 goto fail_pages_alloc;
 
Quoted 3 lines

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 138

Quoted 3 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -2
-       ret = sg_alloc_table_from_pages(buf->dma_sgt, buf->pages,
-                       buf->num_pages, 0, size, GFP_KERNEL);
+       if (dev)

dev can't be NULL, see the context above.

Quoted 6 lines

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 143

Quoted 5 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +5 -0
+               max_segment = dma_max_mapping_size(dev);
+       if (max_segment == 0)
+               max_segment = UINT_MAX;
+       ret = sg_alloc_table_from_pages_segment(buf->dma_sgt, buf->pages,
+               buf->num_pages, 0, size, max_segment, GFP_KERNEL);
Quoted 7 lines

One thing that I'm not sure about here is that we use sg_alloc_table_from_pages_segment(), but we actually don't pass the max segment size (as returned by dma_get_max_seg_size()) to it. I'm also not exactly sure what's the difference between "max mapping size" and "max seg size".
+Robin Murphy +Christoph Hellwig I think we could benefit from your expertise here.

dma_get_max_seg_size() represents a capability of the device itself, namely the largest contiguous range it can be programmed to access in a single DMA descriptor/register/whatever. Conversely, dma_max_mapping_size() is a capablity of the DMA API implementation, and represents the largest contiguous mapping it is guaranteed to be able to handle (each segment in the case of dma_map_sg(), or the whole thing for dma_map_page()). Most likely the thing you want here is min_not_zero(max_seg_size, max_mapping_size).

Quoted 3 lines

Generally looking at videobuf2-dma-sg, I feel like we would benefit from some kind of dma_alloc_table_from_pages() that simply takes the struct dev pointer and does everything necessary.

Possibly; this code already looks lifted from drm_prime_pages_to_sg(), and if it's needed here then presumably vb2_dma_sg_get_userptr() also needs it, at the very least.

Thanks,
Robin.

#3 lore raw
Quoted 3 lines · Robin Murphy

dma_get_max_seg_size() represents a capability of the device itself, namely the largest contiguous range it can be programmed to access in a single DMA descriptor/register/whatever.

Yes. In a way it's a bit odd that it ended up in a field in struct device, as the feature might actually be different for different DMA engines or features in a device. If I was to redesign it from scratch I'd just pass it to dma_map_sg.

Quoted 7 lines
Quoted 3 lines

Generally looking at videobuf2-dma-sg, I feel like we would benefit from some kind of dma_alloc_table_from_pages() that simply takes the struct dev pointer and does everything necessary.

Possibly; this code already looks lifted from drm_prime_pages_to_sg(), and if it's needed here then presumably vb2_dma_sg_get_userptr() also needs it, at the very least.

Yes, there's tons of them. But I'd feel really bad adding even more struct scatterlist based APIs given how bad of a data structure that is.

maintainer
#4 lore raw
Quoted 15 lines · Christoph Hellwig
Quoted 3 lines · Robin Murphy

dma_get_max_seg_size() represents a capability of the device itself, namely the largest contiguous range it can be programmed to access in a single DMA descriptor/register/whatever.

Yes. In a way it's a bit odd that it ended up in a field in struct device, as the feature might actually be different for different DMA engines or features in a device. If I was to redesign it from scratch I'd just pass it to dma_map_sg.

Quoted 4 lines
Quoted 3 lines

Generally looking at videobuf2-dma-sg, I feel like we would benefit from some kind of dma_alloc_table_from_pages() that simply takes the struct dev pointer and does everything necessary.

Quoted 6 lines
Quoted 3 lines

Possibly; this code already looks lifted from drm_prime_pages_to_sg(), and if it's needed here then presumably vb2_dma_sg_get_userptr() also needs it, at the very least.

Yes, there's tons of them. But I'd feel really bad adding even more struct scatterlist based APIs given how bad of a data structure that is.

Do we see anything replacing it widely anywhere on the short-middle term horizon? I think we could possibly migrate vb2 to use that new thing internally and just provide some compatibility X to scatterlist conversion function for the drivers.

#5 lore raw
Quoted 4 lines · Tomasz Figa

Do we see anything replacing it widely anywhere on the short-middle term horizon? I think we could possibly migrate vb2 to use that new thing internally and just provide some compatibility X to scatterlist conversion function for the drivers.

Jason said at LSF/MM that he had a prototype for a mapping API that takes a phys/len array as input and dma_addr/len a output, which really is the right thing to do, especially for dmabuf.

Jason, what's the status of your work?

#6 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines · Tomasz Figa

Do we see anything replacing it widely anywhere on the short-middle term horizon? I think we could possibly migrate vb2 to use that new thing internally and just provide some compatibility X to scatterlist conversion function for the drivers.

Jason said at LSF/MM that he had a prototype for a mapping API that takes a phys/len array as input and dma_addr/len a output, which really is the right thing to do, especially for dmabuf.

Yes, still a prototype. Given the change in direction some of the assumptions of the list design will need some adjusting.

I felt there wasn't much justification to add a list without also supporting the P2P and it was not looking very good to give the DMA API proper p2p support without also connecting it to lists somehow.

Anyhow, I had drafted a basic list datastructure and starting implementation that is sort of structured in away that is similar to xarray (eg with fixed chunks, generic purpose, etc)

https://github.com/jgunthorpe/linux/commit/58d7e0578a09d9cd2360be515208bcd74ade5958

I would still call it an experiment as the auto-sizing entry approach needs benchmarking to see what it costs.

Quoted 1 line

Jason, what's the status of your work?

After we talked I changed the order of the work, instead of starting with the SG list side, I wanted to progress on the DMA API side and then build any list infrastructure on that new API.

Your idea to use a new API that could allocate IOVA and then map phys to IOVA as a DMA API entry point looked good to me, and it is a perfect map for what RDMA's ODP stuff needs. So I changed direction to rework ODP and introduce the DMA API parts as the first step.

I had to put it aside due to the volume of iommu/vfio stuff going on right now. However, I think I will have someone who can work on the ODP driver part this month

Regardless, RDMA really needs some kind of generic lists to hold CPU and physical address ranges, so I still see that as being part of the ultimate solution.

Jason

#7 lore raw
Quoted 13 lines · Jason Gunthorpe
Quoted 5 lines
Quoted 1 line

conversion function for the drivers.

Jason said at LSF/MM that he had a prototype for a mapping API that takes a phys/len array as input and dma_addr/len a output, which really is the right thing to do, especially for dmabuf.

Yes, still a prototype. Given the change in direction some of the assumptions of the list design will need some adjusting.

I felt there wasn't much justification to add a list without also supporting the P2P and it was not looking very good to give the DMA API proper p2p support without also connecting it to lists somehow.

Quoted 5 lines

Anyhow, I had drafted a basic list datastructure and starting implementation that is sort of structured in away that is similar to xarray (eg with fixed chunks, generic purpose, etc)

https://github.com/jgunthorpe/linux/commit/58d7e0578a09d9cd2360be515208bcd74ade5958

This seems fairly complicated complicated, and the entry seems pretty large for a bio_vec replacement or a dma_addr_t+len tuple, which both should be (sizeof(phys_addr_t) + sizeof(u32) + the size of flags if needed, which for 64-bit would fit into the padding from 96 bytes to 128 bytes anyway.

#8 lore raw
Quoted 18 lines · Christoph Hellwig
Quoted 17 lines · Jason Gunthorpe
Quoted 5 lines
Quoted 1 line

conversion function for the drivers.

Jason said at LSF/MM that he had a prototype for a mapping API that takes a phys/len array as input and dma_addr/len a output, which really is the right thing to do, especially for dmabuf.

Yes, still a prototype. Given the change in direction some of the assumptions of the list design will need some adjusting.

I felt there wasn't much justification to add a list without also supporting the P2P and it was not looking very good to give the DMA API proper p2p support without also connecting it to lists somehow.

Anyhow, I had drafted a basic list datastructure and starting implementation that is sort of structured in away that is similar to xarray (eg with fixed chunks, generic purpose, etc)

Quoted 6 lines
Quoted 1 line

https://github.com/jgunthorpe/linux/commit/58d7e0578a09d9cd2360be515208bcd74ade5958

This seems fairly complicated complicated, and the entry seems pretty large for a bio_vec replacement or a dma_addr_t+len tuple, which both should be (sizeof(phys_addr_t) + sizeof(u32) + the size of flags if needed, which for 64-bit would fit into the padding from 96 bytes to 128 bytes anyway.

The entry is variable sized, so it depends on what is stuffed in it. For alot of common use cases, especially RDMA page lists, it will be able to use an 8 byte entry. This is pretty much the most space efficient it could be.

There are RDMA use cases where we end up holding huge numbers of pages for a long time just so we can eventually unpin them. It is a nice outcome if that could use 8 bytes/folio.

The primary alternative I see is a fixed 16 bytes/entry with a 64 bit address and ~60 bit length + ~4 bits of flags. This is closer to bio, simpler and faster, but makes the RDMA cases 2x bigger.

Which are the right trade offs, or not, I don't know yet. I wanted to experiment with what this would look like for a bit.

With your direction I felt we could safely keep bio as it is and cheaply make a fast DMA mapper for it. Provide something like this as the 'kitchen sink' version for dmabuf/rdma/etc that are a little different.

Jason

#9 lore raw
Quoted 4 lines · Jason Gunthorpe

The entry is variable sized, so it depends on what is stuffed in it. For alot of common use cases, especially RDMA page lists, it will be able to use an 8 byte entry. This is pretty much the most space efficient it could be.

How do you get away with a 8 byte entry for addr+len?

Quoted 3 lines

The primary alternative I see is a fixed 16 bytes/entry with a 64 bit address and ~60 bit length + ~4 bits of flags. This is closer to bio, simpler and faster, but makes the RDMA cases 2x bigger.

That's what I'd expect.

Quoted 4 lines

With your direction I felt we could safely keep bio as it is and cheaply make a fast DMA mapper for it. Provide something like this as the 'kitchen sink' version for dmabuf/rdma/etc that are a little different.

So for the first version I see no need to change the bio_vec representation as part of this project, but at the same time the bio_vec representation causes problems for other reasons. So I want to change it anyway.

#10 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 4 lines · Jason Gunthorpe

The entry is variable sized, so it depends on what is stuffed in it. For alot of common use cases, especially RDMA page lists, it will be able to use an 8 byte entry. This is pretty much the most space efficient it could be.

How do you get away with a 8 byte entry for addr+len?

It's a compression. The basic base/length/flags has alot of zero bits in common cases.

I was drafting:

2 bits for 'encoding == 8 bytes'
2 bits for flags
28 bits for length
32 bits for address >> 12

So if the range has zero bits in the right places then it fits in 8 bytes.

Otherwise the compressor will choose a 16 byte entry:

2 bits for 'encoding == 16 bytes'
2 bits for flags
36 bits for length
64 bits for address
24 bits for offset

And a 24 byte entry with 36 bits of flags and no limitations.

So we can store anything, but common cases of page lists will use only 8 bytes/entry.

This is a classical compression trade off, better space efficiency for long term storage, worse iteration efficiency.

Quoted 7 lines
Quoted 4 lines

With your direction I felt we could safely keep bio as it is and cheaply make a fast DMA mapper for it. Provide something like this as the 'kitchen sink' version for dmabuf/rdma/etc that are a little different.

So for the first version I see no need to change the bio_vec representation as part of this project,

Right

Quoted 2 lines

but at the same time the bio_vec representation causes problems for other reasons. So I want to change it anyway.

I don't feel competent in this area, so I'm not sure what this will be.

I was hoping to come with some data and benchmarks and we consider options. The appeal of smaller long term memory footprint for the RDMA use case is interesting enough to look at it.

Jason

A message that is not in the archive (sent off-list or not yet received): DB9PR04MB92841D8BC1122D5A4210F78987E6A@DB9PR04MB9284.eurprd04.prod.outlook.com

reviewer
#12 lore raw
Quoted 7 lines · Hui Fang

Hi, guys
Thanks for your time.
I wonder if only NXP met the "swiotlb buffer full" issue.
In theory, when format is YUYV, those resolutions no greater than 320x240 (153600 bytes, which less than the max mapping size 256K ) can't meet the issue. But resolutions no less than 640x480 (307200 bytes), may have chances to trigger the issue.
On our side, we tested on 1080p, easy to reproduce.
Or maybe "dma_max_mapping_size(dev)" is much bigger than 256K on your side?

I would imagine that in most cases, people either have an IOMMU, or they don't have more RAM than the device can access directly.

And in fact I think we've missed looking deep enough and that's the real problem here - the overall context is a buffer allocator, so if SWIOTLB is bouncing anything in that dma_map_sg() call, then it means we're allocating a buffer for a device out of pages that that device can't actually access, which seems fundamentally bad. Unfortunately there isn't currently an easy way to do the right thing - dma-debug would probably get very unhappy about scraping a bunch of dma_alloc_pages() allocations into a scatterlist and subsequently calling dma_sync_sg() on it, while dma_alloc_noncontiguous() does at least return a scatterlist but would be overly restrictive since we don't need it to be dma-contiguous either. I guess we could do with some sort of dma_alloc_sgt() API that joins the relevant internal bits together to replace vb2_dma_sg_alloc_compacted() with proper DMA-mask-aware allocation, and probably also allocates the sg_table as well?

Thanks,
Robin.

Quoted 34 lines

BRs,
Fang Hui
________________________________
From: Christoph Hellwig <[email protected]>
Sent: Tuesday, August 29, 2023 11:04 PM
To: Robin Murphy <[email protected]>
Cc: Tomasz Figa <[email protected]>; Anle Pan <[email protected]>; Christoph Hellwig <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Hui Fang <[email protected]>
Subject: [EXT] Re: [PATCH] media: videobuf2-dma-sg: limit the sg segment size

Caution: This is an external email. Please take care when clicking links or opening attachments. When in doubt, report the message using the 'Report this email' button

Quoted 3 lines · Robin Murphy

dma_get_max_seg_size() represents a capability of the device itself, namely the largest contiguous range it can be programmed to access in a single DMA descriptor/register/whatever.

Yes. In a way it's a bit odd that it ended up in a field in struct device, as the feature might actually be different for different DMA engines or features in a device. If I was to redesign it from scratch I'd just pass it to dma_map_sg.

Quoted 7 lines
Quoted 3 lines

Generally looking at videobuf2-dma-sg, I feel like we would benefit from some kind of dma_alloc_table_from_pages() that simply takes the struct dev pointer and does everything necessary.

Possibly; this code already looks lifted from drm_prime_pages_to_sg(), and if it's needed here then presumably vb2_dma_sg_get_userptr() also needs it, at the very least.

Yes, there's tons of them. But I'd feel really bad adding even more struct scatterlist based APIs given how bad of a data structure that is.

maintainer
#13 lore raw
In reply to Robin Murphy
Quoted 58 lines · Robin Murphy
Quoted 36 lines · Tomasz Figa

Hi Anle,

Quoted 4 lines · Anle Pan

When allocating from pages, the size of the sg segment is unlimited and the default is UINT_MAX. This will cause the DMA stream mapping failed later with a "swiotlb buffer full" error.

Thanks for the patch. Good catch.

Quoted 26 lines

The default maximum mapping
size is 128 slots x 2K = 256K, determined by "IO_TLB_SEGSIZE". To fix the issue, limit the sg segment size according to "dma_max_mapping_size" to match the mapping limit.

---
  drivers/media/common/videobuf2/videobuf2-dma-sg.c | 9 +++++++--
  1 file changed, 7 insertions(+), 2 deletions(-)
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -0
index fa69158a65b1..b608a7c5f240 100644
@@ -105,6 +105,7 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
         struct sg_table *sgt;
         int ret;
         int num_pages;
+       size_t max_segment = 0;
 
         if (WARN_ON(!dev) || WARN_ON(!size))
                 return ERR_PTR(-EINVAL);
@@ -134,8 +135,12 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
         if (ret)
                 goto fail_pages_alloc;
 
Quoted 3 lines

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 138

Quoted 3 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -2
-       ret = sg_alloc_table_from_pages(buf->dma_sgt, buf->pages,
-                       buf->num_pages, 0, size, GFP_KERNEL);
+       if (dev)

dev can't be NULL, see the context above.

Quoted 6 lines

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 143

Quoted 5 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +5 -0
+               max_segment = dma_max_mapping_size(dev);
+       if (max_segment == 0)
+               max_segment = UINT_MAX;
+       ret = sg_alloc_table_from_pages_segment(buf->dma_sgt, buf->pages,
+               buf->num_pages, 0, size, max_segment, GFP_KERNEL);
Quoted 7 lines

One thing that I'm not sure about here is that we use sg_alloc_table_from_pages_segment(), but we actually don't pass the max segment size (as returned by dma_get_max_seg_size()) to it. I'm also not exactly sure what's the difference between "max mapping size" and "max seg size".
+Robin Murphy +Christoph Hellwig I think we could benefit from your expertise here.

Quoted 9 lines

dma_get_max_seg_size() represents a capability of the device itself, namely the largest contiguous range it can be programmed to access in a single DMA descriptor/register/whatever. Conversely, dma_max_mapping_size() is a capablity of the DMA API implementation, and represents the largest contiguous mapping it is guaranteed to be able to handle (each segment in the case of dma_map_sg(), or the whole thing for dma_map_page()). Most likely the thing you want here is min_not_zero(max_seg_size, max_mapping_size).

The extra complexity needed here convinces me even more that we need a helper...

Quoted 7 lines
Quoted 3 lines

Generally looking at videobuf2-dma-sg, I feel like we would benefit from some kind of dma_alloc_table_from_pages() that simply takes the struct dev pointer and does everything necessary.

Possibly; this code already looks lifted from drm_prime_pages_to_sg(), and if it's needed here then presumably vb2_dma_sg_get_userptr() also needs it, at the very least.

That code probably predates drm_prime_pages_to_sg(), but that's probably also the reason that it has its own issues...
I'm tempted to send a patch adding such a helper, but Christoph didn't sound very keen on that idea. Hmm.

Best regards,
Tomasz

#14 lore raw
In reply to Robin Murphy

Hi all,

Quoted 61 lines · Robin Murphy
Quoted 36 lines · Tomasz Figa

Hi Anle,

Quoted 4 lines · Anle Pan

When allocating from pages, the size of the sg segment is unlimited and the default is UINT_MAX. This will cause the DMA stream mapping failed later with a "swiotlb buffer full" error.

Thanks for the patch. Good catch.

Quoted 26 lines

The default maximum mapping
size is 128 slots x 2K = 256K, determined by "IO_TLB_SEGSIZE". To fix the issue, limit the sg segment size according to "dma_max_mapping_size" to match the mapping limit.

---
  drivers/media/common/videobuf2/videobuf2-dma-sg.c | 9 +++++++--

  1 file changed, 7 insertions(+), 2 deletions(-)
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -0
index fa69158a65b1..b608a7c5f240 100644
@@ -105,6 +105,7 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
          struct sg_table *sgt;
          int ret;
          int num_pages;
+       size_t max_segment = 0;
 
          if (WARN_ON(!dev) || WARN_ON(!size))
                  return ERR_PTR(-EINVAL);
@@ -134,8 +135,12 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
          if (ret)
                  goto fail_pages_alloc;
 
Quoted 3 lines

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 138

Quoted 3 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -2
-       ret = sg_alloc_table_from_pages(buf->dma_sgt, buf->pages,
-                       buf->num_pages, 0, size, GFP_KERNEL);
+       if (dev)

dev can't be NULL, see the context above.

Quoted 6 lines

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 143

Quoted 5 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +5 -0
+               max_segment = dma_max_mapping_size(dev);
+       if (max_segment == 0)
+               max_segment = UINT_MAX;
+       ret = sg_alloc_table_from_pages_segment(buf->dma_sgt, buf->pages,
+               buf->num_pages, 0, size, max_segment, GFP_KERNEL);
Quoted 7 lines

One thing that I'm not sure about here is that we use sg_alloc_table_from_pages_segment(), but we actually don't pass the max segment size (as returned by dma_get_max_seg_size()) to it. I'm also not exactly sure what's the difference between "max mapping size" and "max seg size".
+Robin Murphy +Christoph Hellwig I think we could benefit from your expertise here.

dma_get_max_seg_size() represents a capability of the device itself, namely the largest contiguous range it can be programmed to access in a single DMA descriptor/register/whatever. Conversely, dma_max_mapping_size() is a capablity of the DMA API implementation, and represents the largest contiguous mapping it is guaranteed to be able to handle (each segment in the case of dma_map_sg(), or the whole thing for dma_map_page()). Most likely the thing you want here is min_not_zero(max_seg_size, max_mapping_size).

Quoted 5 lines
Quoted 3 lines

Generally looking at videobuf2-dma-sg, I feel like we would benefit from some kind of dma_alloc_table_from_pages() that simply takes the struct dev pointer and does everything necessary.

Possibly; this code already looks lifted from drm_prime_pages_to_sg(), and if it's needed here then presumably vb2_dma_sg_get_userptr() also needs it, at the very least.

I also see sg_alloc_table_from_pages being called in vb2_dc_get_userptr in videobuf2-dma-contig.c, presumably that needs the same fix?

From what I gather from this thread there are no improved helpers on the immediate horizon, so this issue has to be fixed in videobuf2 for now.

So this requires a v2 that fixes this also in vb2_dma_sg_get_userptr and vb2_dc_get_userptr, correct? If so, then it would be nice if Anle can post a v2 with those changes.

Note that when I grepped for sg_alloc_table_from_pages_segment users, I noticed that in most cases dma_max_mapping_size is used, but in one case it uses dma_get_max_seg_size (drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c). I have no idea what the difference is between the two.

One small change that would make sg_alloc_table_from_pages_segment() a bit easier to work with is if it would replace a max_segment value of 0 with UINT_MAX. Then you can just stick in dma_max_mapping_size(dev) as the argument.

Alternatively, if we can be certain that dma_max_mapping_size(dev) never returns 0, then that 'if (max_segment == 0)' part can just be dropped.

I also wonder if any of the other (non-media) users of sg_alloc_table_from_pages would need to use sg_alloc_table_from_pages_segment as well. Is this really media specific? Or is media just the most likely subsystem to hit this issue due to the large amounts of memory it uses?

Regards,

Hans

maintainer
#15 lore raw
Quoted 69 lines · Hans Verkuil

Hi all,

Quoted 65 lines · Robin Murphy
Quoted 36 lines · Tomasz Figa

Hi Anle,

Quoted 4 lines · Anle Pan

When allocating from pages, the size of the sg segment is unlimited and the default is UINT_MAX. This will cause the DMA stream mapping failed later with a "swiotlb buffer full" error.

Thanks for the patch. Good catch.

Quoted 26 lines

The default maximum mapping
size is 128 slots x 2K = 256K, determined by "IO_TLB_SEGSIZE". To fix the issue, limit the sg segment size according to "dma_max_mapping_size" to match the mapping limit.

---
  drivers/media/common/videobuf2/videobuf2-dma-sg.c | 9 +++++++--
  1 file changed, 7 insertions(+), 2 deletions(-)
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -0
index fa69158a65b1..b608a7c5f240 100644
@@ -105,6 +105,7 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
         struct sg_table *sgt;
         int ret;
         int num_pages;
+       size_t max_segment = 0;
 
         if (WARN_ON(!dev) || WARN_ON(!size))
                 return ERR_PTR(-EINVAL);
@@ -134,8 +135,12 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
         if (ret)
                 goto fail_pages_alloc;
 
Quoted 3 lines

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 138

Quoted 3 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -2
-       ret = sg_alloc_table_from_pages(buf->dma_sgt, buf->pages,
-                       buf->num_pages, 0, size, GFP_KERNEL);
+       if (dev)

dev can't be NULL, see the context above.

Quoted 6 lines

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 143

Quoted 5 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +5 -0
+               max_segment = dma_max_mapping_size(dev);
+       if (max_segment == 0)
+               max_segment = UINT_MAX;
+       ret = sg_alloc_table_from_pages_segment(buf->dma_sgt, buf->pages,
+               buf->num_pages, 0, size, max_segment, GFP_KERNEL);
Quoted 7 lines

One thing that I'm not sure about here is that we use sg_alloc_table_from_pages_segment(), but we actually don't pass the max segment size (as returned by dma_get_max_seg_size()) to it. I'm also not exactly sure what's the difference between "max mapping size" and "max seg size".
+Robin Murphy +Christoph Hellwig I think we could benefit from your expertise here.

dma_get_max_seg_size() represents a capability of the device itself, namely the largest contiguous range it can be programmed to access in a single DMA descriptor/register/whatever. Conversely, dma_max_mapping_size() is a capablity of the DMA API implementation, and represents the largest contiguous mapping it is guaranteed to be able to handle (each segment in the case of dma_map_sg(), or the whole thing for dma_map_page()). Most likely the thing you want here is min_not_zero(max_seg_size, max_mapping_size).

Quoted 3 lines

Generally looking at videobuf2-dma-sg, I feel like we would benefit from some kind of dma_alloc_table_from_pages() that simply takes the struct dev pointer and does everything necessary.

Quoted 7 lines
Quoted 1 line

Possibly; this code already looks lifted from drm_prime_pages_to_sg(), and if it's needed here then presumably vb2_dma_sg_get_userptr() also needs it, at the very least.

I also see sg_alloc_table_from_pages being called in vb2_dc_get_userptr in videobuf2-dma-contig.c, presumably that needs the same fix?

From what I gather from this thread there are no improved helpers on the immediate horizon, so this issue has to be fixed in videobuf2 for now.

Agreed. (Although I suspect the real issue that NXP is having isn't really this and this is just a side effect.)

Quoted 3 lines

So this requires a v2 that fixes this also in vb2_dma_sg_get_userptr and vb2_dc_get_userptr, correct? If so, then it would be nice if Anle can post a v2 with those changes.

If we need to fix 3 different callers, could we at least add an internal vb2 helper for this (e.g. vb2_asg_alloc_table_from_pages())?

Quoted 5 lines

Note that when I grepped for sg_alloc_table_from_pages_segment users, I noticed that in most cases dma_max_mapping_size is used, but in one case it uses dma_get_max_seg_size (drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c). I have no idea what the difference is between the two.

FWIW, Robin explained that earlier in the thread. To be fully correct, we need to consider both...

Quoted 7 lines

One small change that would make sg_alloc_table_from_pages_segment() a bit easier to work with is if it would replace a max_segment value of 0 with UINT_MAX. Then you can just stick in dma_max_mapping_size(dev) as the argument.

Alternatively, if we can be certain that dma_max_mapping_size(dev) never returns 0, then that 'if (max_segment == 0)' part can just be dropped.

That's also the reason I wanted a helper. Not sure if it's so easy to change the calling convention now.

Quoted 5 lines

I also wonder if any of the other (non-media) users of sg_alloc_table_from_pages would need to use sg_alloc_table_from_pages_segment as well. Is this really media specific? Or is media just the most likely subsystem to hit this issue due to the large amounts of memory it uses?

I don't think it's media-specific, but also as I said earlier, it should be really rare to hit it - swiotlb isn't normally expected for reasonably capable systems and buffer allocation being done correctly.

Best regards,
Tomasz

#16 lore raw
Quoted 82 lines · Tomasz Figa
Quoted 69 lines · Hans Verkuil

Hi all,

Quoted 65 lines · Robin Murphy
Quoted 36 lines · Tomasz Figa

Hi Anle,

Quoted 4 lines · Anle Pan

When allocating from pages, the size of the sg segment is unlimited and the default is UINT_MAX. This will cause the DMA stream mapping failed later with a "swiotlb buffer full" error.

Thanks for the patch. Good catch.

Quoted 26 lines

The default maximum mapping
size is 128 slots x 2K = 256K, determined by "IO_TLB_SEGSIZE". To fix the issue, limit the sg segment size according to "dma_max_mapping_size" to match the mapping limit.

---
  drivers/media/common/videobuf2/videobuf2-dma-sg.c | 9 +++++++--
  1 file changed, 7 insertions(+), 2 deletions(-)
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -0
index fa69158a65b1..b608a7c5f240 100644
@@ -105,6 +105,7 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
         struct sg_table *sgt;
         int ret;
         int num_pages;
+       size_t max_segment = 0;
 
         if (WARN_ON(!dev) || WARN_ON(!size))
                 return ERR_PTR(-EINVAL);
@@ -134,8 +135,12 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
         if (ret)
                 goto fail_pages_alloc;
 
Quoted 3 lines

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 138

Quoted 3 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -2
-       ret = sg_alloc_table_from_pages(buf->dma_sgt, buf->pages,
-                       buf->num_pages, 0, size, GFP_KERNEL);
+       if (dev)

dev can't be NULL, see the context above.

Quoted 6 lines

drivers/media/common/videobuf2/videobuf2-dma-sg.c static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, · line 143

Quoted 5 lines
drivers/media/common/videobuf2/videobuf2-dma-sg.c +5 -0
+               max_segment = dma_max_mapping_size(dev);
+       if (max_segment == 0)
+               max_segment = UINT_MAX;
+       ret = sg_alloc_table_from_pages_segment(buf->dma_sgt, buf->pages,
+               buf->num_pages, 0, size, max_segment, GFP_KERNEL);
Quoted 7 lines

One thing that I'm not sure about here is that we use sg_alloc_table_from_pages_segment(), but we actually don't pass the max segment size (as returned by dma_get_max_seg_size()) to it. I'm also not exactly sure what's the difference between "max mapping size" and "max seg size".
+Robin Murphy +Christoph Hellwig I think we could benefit from your expertise here.

dma_get_max_seg_size() represents a capability of the device itself, namely the largest contiguous range it can be programmed to access in a single DMA descriptor/register/whatever. Conversely, dma_max_mapping_size() is a capablity of the DMA API implementation, and represents the largest contiguous mapping it is guaranteed to be able to handle (each segment in the case of dma_map_sg(), or the whole thing for dma_map_page()). Most likely the thing you want here is min_not_zero(max_seg_size, max_mapping_size).

Quoted 3 lines

Generally looking at videobuf2-dma-sg, I feel like we would benefit from some kind of dma_alloc_table_from_pages() that simply takes the struct dev pointer and does everything necessary.

Quoted 7 lines
Quoted 1 line

Possibly; this code already looks lifted from drm_prime_pages_to_sg(), and if it's needed here then presumably vb2_dma_sg_get_userptr() also needs it, at the very least.

I also see sg_alloc_table_from_pages being called in vb2_dc_get_userptr in videobuf2-dma-contig.c, presumably that needs the same fix?

From what I gather from this thread there are no improved helpers on the immediate horizon, so this issue has to be fixed in videobuf2 for now.

Agreed. (Although I suspect the real issue that NXP is having isn't really this and this is just a side effect.)

Quoted 1 line
Quoted 5 lines
Quoted 2 lines

So this requires a v2 that fixes this also in vb2_dma_sg_get_userptr and vb2_dc_get_userptr, correct? If so, then it would be nice if Anle can post a v2 with those changes.

If we need to fix 3 different callers, could we at least add an internal vb2 helper for this (e.g. vb2_asg_alloc_table_from_pages())?

Definitely.

Quoted 9 lines
Quoted 6 lines

Note that when I grepped for sg_alloc_table_from_pages_segment users, I noticed that in most cases dma_max_mapping_size is used, but in one case it uses dma_get_max_seg_size (drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c). I have no idea what the difference is between the two.

FWIW, Robin explained that earlier in the thread. To be fully correct, we need to consider both...

Ah, I missed that explanation. Yes, it looks like both need to be considered.

Quoted 11 lines
Quoted 8 lines

One small change that would make sg_alloc_table_from_pages_segment() a bit easier to work with is if it would replace a max_segment value of 0 with UINT_MAX. Then you can just stick in dma_max_mapping_size(dev) as the argument.

Alternatively, if we can be certain that dma_max_mapping_size(dev) never returns 0, then that 'if (max_segment == 0)' part can just be dropped.

That's also the reason I wanted a helper. Not sure if it's so easy to change the calling convention now.

There are not all that many users of sg_alloc_table_from_pages(_segment), dma_get_max_seg_size and dma_max_mapping_size, so it doesn't look too difficult for a sufficiently motivated person (not me!) to make changes here.

Regards,

Hans

Quoted 13 lines
Quoted 6 lines

I also wonder if any of the other (non-media) users of sg_alloc_table_from_pages would need to use sg_alloc_table_from_pages_segment as well. Is this really media specific? Or is media just the most likely subsystem to hit this issue due to the large amounts of memory it uses?

I don't think it's media-specific, but also as I said earlier, it should be really rare to hit it - swiotlb isn't normally expected for reasonably capable systems and buffer allocation being done correctly.

Best regards,
Tomasz

#17 lore raw
In reply to Anle Pan
Quoted 7 lines · Anle Pan

When allocating from pages, the size of the sg segment is unlimited and the default is UINT_MAX. This will cause the DMA stream mapping failed later with a "swiotlb buffer full" error. The default maximum mapping size is 128 slots x 2K = 256K, determined by "IO_TLB_SEGSIZE".
To fix the issue, limit the sg segment size according to "dma_max_mapping_size" to match the mapping limit.

I wonder if only NXP met the "swiotlb buffer full" issue. In theory, when format is YUYV, those resolutions no greater than 320x240 (153600 bytes, which less than the max mapping size 256K ) can't meet the issue. But resolutions no less than 640x480 (307200 bytes), may have chances to trigger the issue.

BRs,
FangHui

maintainer
#18 lore raw
Quoted 10 lines · Hui Fang
Quoted 7 lines · Anle Pan

When allocating from pages, the size of the sg segment is unlimited and the default is UINT_MAX. This will cause the DMA stream mapping failed later with a "swiotlb buffer full" error. The default maximum mapping size is 128 slots x 2K = 256K, determined by "IO_TLB_SEGSIZE".
To fix the issue, limit the sg segment size according to "dma_max_mapping_size" to match the mapping limit.

Quoted 5 lines

I wonder if only NXP met the "swiotlb buffer full" issue. In theory, when format is YUYV, those resolutions no greater than 320x240 (153600 bytes, which less than the max mapping size 256K ) can't meet the issue. But resolutions no less than 640x480 (307200 bytes), may have chances to trigger the issue.

I think a combination of a device that can support scatter-gather, but requires swiotlb is kind of rare. Most drivers would require a single contiguous DMA address (and thus use videobuf2-dma-contig) and physical discontinuity would be handled by an IOMMU (transparently to vb2).

I guess one thing to ask you about your specific setup is whether it's expected that the swiotlb ends up being used at all?

Best regards,
Tomasz

#19 lore raw
Quoted 1 line · Tomasz Figa

On Wed, Aug 30, 2023 at 5:50 PM Hui Fang <[email protected]> wrote:

......

Quoted 6 lines
Quoted 5 lines

I wonder if only NXP met the "swiotlb buffer full" issue. In theory, when format is YUYV, those resolutions no greater than 320x240 (153600 bytes, which less than the max mapping size 256K ) can't meet the issue. But resolutions no less than 640x480 (307200 bytes), may have chances to trigger the issue.

Quoted 7 lines

I think a combination of a device that can support scatter-gather, but requires swiotlb is kind of rare. Most drivers would require a single contiguous DMA address (and thus use videobuf2-dma-contig) and physical discontinuity would be handled by an IOMMU (transparently to vb2).

I guess one thing to ask you about your specific setup is whether it's expected that the swiotlb ends up being used at all?

Yes, the swiotlb ends up being used. We met the issue when bring up DeviceAsWebCam (android-14 new feature, android device works as a usb camera).

BRs,
Fang Hui

maintainer
#20 lore raw
Quoted 15 lines · Hui Fang
Quoted 1 line · Tomasz Figa

On Wed, Aug 30, 2023 at 5:50 PM Hui Fang <[email protected]> wrote:

......

Quoted 11 lines
Quoted 5 lines

I wonder if only NXP met the "swiotlb buffer full" issue. In theory, when format is YUYV, those resolutions no greater than 320x240 (153600 bytes, which less than the max mapping size 256K ) can't meet the issue. But resolutions no less than 640x480 (307200 bytes), may have chances to trigger the issue.

I think a combination of a device that can support scatter-gather, but requires swiotlb is kind of rare. Most drivers would require a single contiguous DMA address (and thus use videobuf2-dma-contig) and physical discontinuity would be handled by an IOMMU (transparently to vb2).

Quoted 5 lines
Quoted 2 lines

I guess one thing to ask you about your specific setup is whether it's expected that the swiotlb ends up being used at all?

Yes, the swiotlb ends up being used. We met the issue when bring up DeviceAsWebCam (android-14 new feature, android device works as a usb camera).

I see. I guess the mapping is done by the USB gadget controller driver? Could you point us to the exact driver that's used?

Just to clarify, swiotlb should only be needed in the very extreme fallback case, because of the performance impact of the memory copy back and forth. The right approach would depend on the DMA capabilities of your device, though.

Best regards,
Tomasz

#21 lore raw
Quoted 7 lines · Tomasz Figa

I see. I guess the mapping is done by the USB gadget controller driver? Could you point us to the exact driver that's used?

Just to clarify, swiotlb should only be needed in the very extreme fallback case, because of the performance impact of the memory copy back and forth. The right approach would depend on the DMA capabilities of your device, though.

[  138.493943][ T2104] Call trace:
[  138.497090][ T2104]  vb2_dma_sg_alloc+0x2ec/0x2fc
[  138.501808][ T2104]  __vb2_queue_alloc+0x224/0x724
[  138.506608][ T2104]  vb2_core_reqbufs+0x374/0x528
[  138.511320][ T2104]  vb2_reqbufs+0xe0/0xf4
[  138.515428][ T2104]  uvcg_alloc_buffers+0x18/0x34
[  138.520159][ T2104]  uvc_v4l2_reqbufs+0x38/0x54
[  138.524703][ T2104]  v4l_reqbufs+0x68/0x80
[  138.528820][ T2104]  __video_do_ioctl+0x370/0x4dc
[  138.533535][ T2104]  video_usercopy+0x43c/0xb38
[  138.538076][ T2104]  video_ioctl2+0x18/0x28
[  138.542272][ T2104]  v4l2_ioctl+0x6c/0x84
[  138.546291][ T2104]  __arm64_sys_ioctl+0xa8/0xe4
[  138.550928][ T2104]  invoke_syscall+0x58/0x114
[  138.555389][ T2104]  el0_svc_common+0x88/0xfc
[  138.559755][ T2104]  do_el0_svc+0x2c/0xb8
[  138.563776][ T2104]  el0_svc+0x2c/0xa4
[  138.567544][ T2104]  el0t_64_sync_handler+0x68/0xb4
[  138.572434][ T2104]  el0t_64_sync+0x1a4/0x1a8
[  138.576803][ T2104] Code: 17ffffcb 928002b3 d4210000 17ffffc8 (d4210000) 
[  138.583598][ T2104] ---[ end trace 0000000000000000 ]---

Also, below should explain why vb2_dma_sg_alloc is used. We tested on 8mp with use dwc3 controller.

In drivers/usb/dwc3/gadget.c:
dwc->gadget->sg_supported = true;

In drivers/usb/gadget/function/uvc_queue.c

if (cdev->gadget->sg_supported) {
	queue->queue.mem_ops = &vb2_dma_sg_memops;
	queue->use_sg = 1;
} else {
	queue->queue.mem_ops = &vb2_vmalloc_memops;
}

BRs,
Fang Hui

maintainer
#22 lore raw
Quoted 39 lines · Hui Fang
Quoted 7 lines · Tomasz Figa

I see. I guess the mapping is done by the USB gadget controller driver? Could you point us to the exact driver that's used?

Just to clarify, swiotlb should only be needed in the very extreme fallback case, because of the performance impact of the memory copy back and forth. The right approach would depend on the DMA capabilities of your device, though.

[  138.493943][ T2104] Call trace:
[  138.497090][ T2104]  vb2_dma_sg_alloc+0x2ec/0x2fc
[  138.501808][ T2104]  __vb2_queue_alloc+0x224/0x724
[  138.506608][ T2104]  vb2_core_reqbufs+0x374/0x528
[  138.511320][ T2104]  vb2_reqbufs+0xe0/0xf4
[  138.515428][ T2104]  uvcg_alloc_buffers+0x18/0x34
[  138.520159][ T2104]  uvc_v4l2_reqbufs+0x38/0x54
[  138.524703][ T2104]  v4l_reqbufs+0x68/0x80
[  138.528820][ T2104]  __video_do_ioctl+0x370/0x4dc
[  138.533535][ T2104]  video_usercopy+0x43c/0xb38
[  138.538076][ T2104]  video_ioctl2+0x18/0x28
[  138.542272][ T2104]  v4l2_ioctl+0x6c/0x84
[  138.546291][ T2104]  __arm64_sys_ioctl+0xa8/0xe4
[  138.550928][ T2104]  invoke_syscall+0x58/0x114
[  138.555389][ T2104]  el0_svc_common+0x88/0xfc
[  138.559755][ T2104]  do_el0_svc+0x2c/0xb8
[  138.563776][ T2104]  el0_svc+0x2c/0xa4
[  138.567544][ T2104]  el0t_64_sync_handler+0x68/0xb4
[  138.572434][ T2104]  el0t_64_sync+0x1a4/0x1a8
[  138.576803][ T2104] Code: 17ffffcb 928002b3 d4210000 17ffffc8 (d4210000)
[  138.583598][ T2104] ---[ end trace 0000000000000000 ]---

Also, below should explain why vb2_dma_sg_alloc is used. We tested on 8mp with use dwc3 controller.

In drivers/usb/dwc3/gadget.c:
dwc->gadget->sg_supported = true;

Quoted 8 lines

In drivers/usb/gadget/function/uvc_queue.c

if (cdev->gadget->sg_supported) {
        queue->queue.mem_ops = &vb2_dma_sg_memops;
        queue->use_sg = 1;
} else {
        queue->queue.mem_ops = &vb2_vmalloc_memops;
}

That all makes sense, but it still doesn't answer the real question on why swiotlb ends up being used. I think you may want to trace what happens in the DMA mapping ops implementation on your system causing it to use swiotlb.

Best regards,
Tomasz

#23 lore raw
Quoted 4 lines · Tomasz Figa

That all makes sense, but it still doesn't answer the real question on why swiotlb ends up being used. I think you may want to trace what happens in the DMA mapping ops implementation on your system causing it to use swiotlb.

Add log and feed invalid data to low buffer on purpose, it's confirmed that swiotlb is actually used.

Got log as
"[ 846.570271][ T138] software IO TLB: ==== swiotlb_bounce: DMA_TO_DEVICE,
dst 000000004589fa38, src 00000000c6d7e8d8, srcPhy 5504139264, size 4096".

" srcPhy 5504139264" is larger than 4G (8mp has DRAM over 5G). And "CONFIG_ZONE_DMA32=y" in kernel config, so swiotlb static is used. Also, the host (win10) side can't get valid image.

Code as below.

drivers/media/common/videobuf2/videobuf2-dma-sg.c +2 -0
index 7f83a86e6810..de03704ce695 100644
@@ -98,6 +98,7 @@ static int vb2_dma_sg_alloc_compacted(struct vb2_dma_sg_buf *buf,
        return 0;
 }
 
+bool g_v4l2 = false;
 static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
                              unsigned long size)
 {
@@ -144,6 +145,7 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
        if (ret)
                goto fail_table_alloc;
 
+       g_v4l2 = true;
        pr_info("==== vb2_dma_sg_alloc, call sg_alloc_table_from_pages_segment,
			size %d, max_segment %d\n", (int)size, (int)max_segment);
kernel/dma/swiotlb.c +9 -0
index dac01ace03a0..a2cda646a02f 100644
@@ -523,6 +523,7 @@ static unsigned int swiotlb_align_offset(struct device *dev, u64 addr)
        return addr & dma_get_min_align_mask(dev) & (IO_TLB_SIZE - 1);
 }
 
+extern bool g_v4l2;
 /*
  * Bounce: copy the swiotlb buffer from or back to the original dma location
  */
@@ -591,8 +592,19 @@ static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size
                }
        } else if (dir == DMA_TO_DEVICE) {
                memcpy(vaddr, phys_to_virt(orig_addr), size);
+               if (g_v4l2) {
+                       static unsigned char val;
+                       val++;
+                       memset(vaddr, val, size);
+
+                       pr_info("====xx %s: DMA_TO_DEVICE, dst %p, src %p, srcPhy %llu, size %zu\n",
+                               __func__, vaddr, phys_to_virt(orig_addr), orig_addr, size);
+               }
        } else {
                memcpy(phys_to_virt(orig_addr), vaddr, size);
        }
 }
 

BRs,
Fang Hui

maintainer
#24 lore raw
Quoted 10 lines · Hui Fang
Quoted 4 lines · Tomasz Figa

That all makes sense, but it still doesn't answer the real question on why swiotlb ends up being used. I think you may want to trace what happens in the DMA mapping ops implementation on your system causing it to use swiotlb.

Add log and feed invalid data to low buffer on purpose, it's confirmed that swiotlb is actually used.

Yes, that we already know. But why?

Quoted 61 lines

Got log as
"[ 846.570271][ T138] software IO TLB: ==== swiotlb_bounce: DMA_TO_DEVICE,
dst 000000004589fa38, src 00000000c6d7e8d8, srcPhy 5504139264, size 4096".

" srcPhy 5504139264" is larger than 4G (8mp has DRAM over 5G). And "CONFIG_ZONE_DMA32=y" in kernel config, so swiotlb static is used. Also, the host (win10) side can't get valid image.

Code as below.

drivers/media/common/videobuf2/videobuf2-dma-sg.c +2 -0
index 7f83a86e6810..de03704ce695 100644
@@ -98,6 +98,7 @@ static int vb2_dma_sg_alloc_compacted(struct vb2_dma_sg_buf *buf,
        return 0;
 }
 
+bool g_v4l2 = false;
 static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
                              unsigned long size)
 {
@@ -144,6 +145,7 @@ static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev,
        if (ret)
                goto fail_table_alloc;
 
+       g_v4l2 = true;
        pr_info("==== vb2_dma_sg_alloc, call sg_alloc_table_from_pages_segment,
                        size %d, max_segment %d\n", (int)size, (int)max_segment);
kernel/dma/swiotlb.c +9 -0
index dac01ace03a0..a2cda646a02f 100644
@@ -523,6 +523,7 @@ static unsigned int swiotlb_align_offset(struct device *dev, u64 addr)
        return addr & dma_get_min_align_mask(dev) & (IO_TLB_SIZE - 1);
 }
 
+extern bool g_v4l2;
 /*
  * Bounce: copy the swiotlb buffer from or back to the original dma location
  */
@@ -591,8 +592,19 @@ static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size
                }
        } else if (dir == DMA_TO_DEVICE) {
                memcpy(vaddr, phys_to_virt(orig_addr), size);
+               if (g_v4l2) {
+                       static unsigned char val;
+                       val++;
+                       memset(vaddr, val, size);
+
+                       pr_info("====xx %s: DMA_TO_DEVICE, dst %p, src %p, srcPhy %llu, size %zu\n",
+                               __func__, vaddr, phys_to_virt(orig_addr), orig_addr, size);
+               }
        } else {
                memcpy(phys_to_virt(orig_addr), vaddr, size);
        }
 }
 

BRs,
Fang Hui

#25 lore raw

On Tue, Sep 12 2023 at 11:22 AM Tomasz Figa <[email protected]>

Quoted 9 lines · Hui Fang
Quoted 2 lines

On Wed, Sep 6, 2023 at 18:28 PM Tomasz Figa <[email protected]>

Quoted 5 lines · wrote:
Quoted 4 lines

That all makes sense, but it still doesn't answer the real question on why swiotlb ends up being used. I think you may want to trace what happens in the DMA mapping ops implementation on your system causing it to use swiotlb.

Quoted 5 lines
Quoted 3 lines

Add log and feed invalid data to low buffer on purpose, it's confirmed that swiotlb is actually used.

Yes, that we already know. But why?

The physical address of v4l2 buffer is large than 4G (5504139264), so the swiotlb is used. "[ 846.570271][ T138] software IO TLB: ==== swiotlb_bounce: DMA_TO_DEVICE,
dst 000000004589fa38, src 00000000c6d7e8d8, srcPhy 5504139264, size 4096".

BRs,
Fang Hui

maintainer
#26 lore raw
Quoted 15 lines · Hui Fang

On Tue, Sep 12 2023 at 11:22 AM Tomasz Figa <[email protected]>

Quoted 13 lines · Hui Fang
Quoted 2 lines

On Wed, Sep 6, 2023 at 18:28 PM Tomasz Figa <[email protected]>

Quoted 8 lines · wrote:
Quoted 4 lines

That all makes sense, but it still doesn't answer the real question on why swiotlb ends up being used. I think you may want to trace what happens in the DMA mapping ops implementation on your system causing it to use swiotlb.

Add log and feed invalid data to low buffer on purpose, it's confirmed that swiotlb is actually used.

Quoted 6 lines
Quoted 1 line

Yes, that we already know. But why?

The physical address of v4l2 buffer is large than 4G (5504139264), so the swiotlb is used. "[ 846.570271][ T138] software IO TLB: ==== swiotlb_bounce: DMA_TO_DEVICE,
dst 000000004589fa38, src 00000000c6d7e8d8, srcPhy 5504139264, size 4096".

Is your DMA device restricted only to the bottom-most 4 GB (32-bit DMA address)? If yes, would it make sense to also allocate from that area rather than bouncing the memory?

Best regards,
Tomasz

#27 lore raw
Quoted 3 lines · Tomasz Figa

Is your DMA device restricted only to the bottom-most 4 GB (32-bit DMA address)? If yes, would it make sense to also allocate from that area rather than bouncing the memory?

The DMA device use 32-bit DMA address.
From user space, can't control the v4l2 buffer address, may still change the code of vb2_dma_sg_alloc().

BRs,
Fang Hui

maintainer
#28 lore raw
Quoted 9 lines · Hui Fang
Quoted 3 lines · Tomasz Figa

Is your DMA device restricted only to the bottom-most 4 GB (32-bit DMA address)? If yes, would it make sense to also allocate from that area rather than bouncing the memory?

The DMA device use 32-bit DMA address.
From user space, can't control the v4l2 buffer address, may still change the code of vb2_dma_sg_alloc().

Right. You may want to try modifying vb2_dma_sg_alloc_compacted() to use dma_alloc_pages() instead of alloc_pages().

Best regards,
Tomasz

#29 lore raw
Quoted 2 lines · Tomasz Figa

Right. You may want to try modifying vb2_dma_sg_alloc_compacted() to use dma_alloc_pages() instead of alloc_pages().

Thanks for your suggestion, it works. And it's a better resolution since no need an extra copy from high buffer to low buffer.

BRs,
Fang Hui

maintainer
#30 lore raw
Quoted 7 lines · Hui Fang
Quoted 2 lines · Tomasz Figa

Right. You may want to try modifying vb2_dma_sg_alloc_compacted() to use dma_alloc_pages() instead of alloc_pages().

Thanks for your suggestion, it works. And it's a better resolution since no need an extra copy from high buffer to low buffer.

Great to hear! Could you submit a patch? Would appreciate adding

above the Signed-off-by line if you don't mind. Thanks.

Best regards,
Tomasz

#31 lore raw
Quoted 5 lines · Tomasz Figa

Great to hear! Could you submit a patch? Would appreciate adding

above the Signed-off-by line if you don't mind. Thanks.

Sure. Will verified on other different i.mx boards, then push.

BRs,
Fang Hui