Skip to contentSkip to conversation

drm/xen-front: Make shmem backed display buffer coherent

PATCH Superseded Oleksandr Andrushchenko 5 participants 31 messages Jan 15, 2019 – Jan 30, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 drm/xen-front: Make shmem backed display buffer coherent

From: Oleksandr Andrushchenko <[email protected]>

When GEM backing storage is allocated with drm_gem_get_pages the backing pages may be cached, thus making it possible that the backend sees only partial content of the buffer which may lead to screen artifacts. Make sure that the frontend's memory is coherent and the backend always sees correct display buffer content.

Fixes c575b7eeb89f ("drm/xen-front: Add support for Xen PV display frontend")

---
Changes since v1:
- Remove GFP_USER|__GFP_DMA32 mapping flags (Gerd)
- Use drm_prime_pages_to_sg directly (Noralf)

 drivers/gpu/drm/xen/xen_drm_front_gem.c | 38 ++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 4 deletions(-)
drivers/gpu/drm/xen/xen_drm_front_gem.c +34 -4
index 28bc501af450..0b0d9b4f97dc 100644
@@ -32,8 +32,11 @@ struct xen_gem_object {
 	/* set for buffers allocated by the backend */
 	bool be_alloc;
 
-	/* this is for imported PRIME buffer */
-	struct sg_table *sgt_imported;
+	/*
+	 * this is for imported PRIME buffer or the one allocated via
+	 * drm_gem_get_pages.
+	 */
+	struct sg_table *sgt;
 };
 
 static inline struct xen_gem_object *
@@ -124,8 +127,28 @@ static struct xen_gem_object *gem_create(struct drm_device *dev, size_t size)
 		goto fail;
 	}
 
+	xen_obj->sgt = drm_prime_pages_to_sg(xen_obj->pages,
+					     xen_obj->num_pages);
+	if (IS_ERR_OR_NULL(xen_obj->sgt)) {
+		ret = PTR_ERR(xen_obj->sgt);
+		xen_obj->sgt = NULL;
+		goto fail_put_pages;
+	}
+
+	if (!dma_map_sg(dev->dev, xen_obj->sgt->sgl, xen_obj->sgt->nents,
+			DMA_BIDIRECTIONAL)) {
+		ret = -EFAULT;
+		goto fail_free_sgt;
+	}
+
 	return xen_obj;
 
+fail_free_sgt:
+	sg_free_table(xen_obj->sgt);
+	xen_obj->sgt = NULL;
+fail_put_pages:
+	drm_gem_put_pages(&xen_obj->base, xen_obj->pages, true, false);
+	xen_obj->pages = NULL;
 fail:
 	DRM_ERROR("Failed to allocate buffer with size %zu\n", size);
 	return ERR_PTR(ret);
@@ -148,7 +171,7 @@ void xen_drm_front_gem_free_object_unlocked(struct drm_gem_object *gem_obj)
 	struct xen_gem_object *xen_obj = to_xen_gem_obj(gem_obj);
 
 	if (xen_obj->base.import_attach) {
-		drm_prime_gem_destroy(&xen_obj->base, xen_obj->sgt_imported);
+		drm_prime_gem_destroy(&xen_obj->base, xen_obj->sgt);
 		gem_free_pages_array(xen_obj);
 	} else {
 		if (xen_obj->pages) {
@@ -157,6 +180,13 @@ void xen_drm_front_gem_free_object_unlocked(struct drm_gem_object *gem_obj)
 							xen_obj->pages);
 				gem_free_pages_array(xen_obj);
 			} else {
+				if (xen_obj->sgt) {
+					dma_unmap_sg(xen_obj->base.dev->dev,
+						     xen_obj->sgt->sgl,
+						     xen_obj->sgt->nents,
+						     DMA_BIDIRECTIONAL);
+					sg_free_table(xen_obj->sgt);
+				}
 				drm_gem_put_pages(&xen_obj->base,
 						  xen_obj->pages, true, false);
 			}
@@ -202,7 +232,7 @@ xen_drm_front_gem_import_sg_table(struct drm_device *dev,
 	if (ret < 0)
 		return ERR_PTR(ret);
 
-	xen_obj->sgt_imported = sgt;
+	xen_obj->sgt = sgt;
 
 	ret = drm_prime_sg_to_page_addr_arrays(sgt, xen_obj->pages,
 					       NULL, xen_obj->num_pages);
Signature · 1 line
2.20.1
#1 lore raw

Hi,

drivers/gpu/drm/xen/xen_drm_front_gem.c static struct xen_gem_object *gem_create(struct drm_device *dev, size_t size) · line 142

Quoted 5 lines
drivers/gpu/drm/xen/xen_drm_front_gem.c +5 -0
+	if (!dma_map_sg(dev->dev, xen_obj->sgt->sgl, xen_obj->sgt->nents,
+			DMA_BIDIRECTIONAL)) {
+		ret = -EFAULT;
+		goto fail_free_sgt;
+	}

Hmm, so it seems the arm guys could not come up with a suggestion how to solve that one in a better way. Ok, lets go with this then.

But didn't we agree that this deserves a comment exmplaining the purpose of the dma_map_sg() call is to flush caches and that there is no actual DMA happening here?

cheers,
Gerd

#2 lore raw
Quoted 11 lines · Gerd Hoffmann

Hi,

drivers/gpu/drm/xen/xen_drm_front_gem.c static struct xen_gem_object *gem_create(struct drm_device *dev, size_t size) · line 142

Quoted 5 lines
drivers/gpu/drm/xen/xen_drm_front_gem.c +5 -0
+	if (!dma_map_sg(dev->dev, xen_obj->sgt->sgl, xen_obj->sgt->nents,
+			DMA_BIDIRECTIONAL)) {
+		ret = -EFAULT;
+		goto fail_free_sgt;
+	}

Hmm, so it seems the arm guys could not come up with a suggestion how to solve that one in a better way. Ok, lets go with this then.

Quoted 3 lines

But didn't we agree that this deserves a comment exmplaining the purpose of the dma_map_sg() call is to flush caches and that there is no actual DMA happening here?

Using a dma mapping call to flush caches is complete no-go. But the real question is why you'd even want to flush cashes if you do not want a dma mapping?

This whole issue keeps getting more and more confusing.

#3 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Gerd Hoffmann
   Hi,

drivers/gpu/drm/xen/xen_drm_front_gem.c static struct xen_gem_object *gem_create(struct drm_device *dev, size_t size) · line 142

Quoted 5 lines
drivers/gpu/drm/xen/xen_drm_front_gem.c +5 -0
+	if (!dma_map_sg(dev->dev, xen_obj->sgt->sgl, xen_obj->sgt->nents,
+			DMA_BIDIRECTIONAL)) {
+		ret = -EFAULT;
+		goto fail_free_sgt;
+	}

Hmm, so it seems the arm guys could not come up with a suggestion how to solve that one in a better way. Ok, lets go with this then.

Quoted 8 lines
Quoted 3 lines

But didn't we agree that this deserves a comment exmplaining the purpose of the dma_map_sg() call is to flush caches and that there is no actual DMA happening here?

Using a dma mapping call to flush caches is complete no-go. But the real question is why you'd even want to flush cashes if you do not want a dma mapping?

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

#4 lore raw
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code. These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

#5 lore raw
Quoted 13 lines · Christoph Hellwig
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

Quoted 8 lines

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code. These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

I'll have to get even deeper into it. Initially I looked at the code, but didn't find anything useful. Or maybe I have just overlooked obvious things there

Thank you,
Oleksandr

#6 lore raw

(+ Stefano)

Hi,

Sorry for jumping late in the conversation.

Quoted 20 lines · Oleksandr Andrushchenko
Quoted 19 lines · Christoph Hellwig
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code. These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

Quoted 5 lines
Quoted 2 lines

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

I'll have to get even deeper into it. Initially I looked at the code, but didn't find anything useful. Or maybe I have just overlooked obvious things there

From Xen on Arm ABI:

"All memory which is shared with other entities in the system (including the hypervisor and other guests) must reside in memory which is mapped as Normal Inner Write-Back Outer Write-Back Inner-Shareable. This applies to:
- hypercall arguments passed via a pointer to guest memory.
- memory shared via the grant table mechanism (including PV I/O rings etc).
- memory shared with the hypervisor (struct shared_info, struct vcpu_info, the grant table, etc).
"

So you should not need any cache maintenance here. Can you provide more details on the memory attribute you use for memory shared in both the backend and frontend?

Cheers,

Quoted 8 lines

Thank you,
Oleksandr
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Signature · 1 line
Julien Grall
#7 lore raw
Quoted 47 lines · Julien Grall

(+ Stefano)

Hi,

Sorry for jumping late in the conversation.

Quoted 21 lines · Oleksandr Andrushchenko
Quoted 20 lines · Christoph Hellwig
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code.  These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

Quoted 5 lines
Quoted 2 lines

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

I'll have to get even deeper into it. Initially I looked at the code, but didn't find anything useful. Or maybe I have just overlooked obvious things there

From Xen on Arm ABI:

"All memory which is shared with other entities in the system (including the hypervisor and other guests) must reside in memory which is mapped as Normal Inner Write-Back Outer Write-Back Inner-Shareable.
This applies to:
  - hypercall arguments passed via a pointer to guest memory.
  - memory shared via the grant table mechanism (including PV I/O rings etc).
  - memory shared with the hypervisor (struct shared_info, struct vcpu_info, the grant table, etc).
"

Quoted 4 lines

So you should not need any cache maintenance here. Can you provide more details on the memory attribute you use for memory shared in both the backend and frontend?

It takes quite some time to collect this (because many components are involved in the
use-case), but for now the pages in the guest domain are:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

Quoted 11 lines

Cheers,

Quoted 8 lines

Thank you,
Oleksandr
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

Hello,

Quoted 46 lines · Oleksandr Andrushchenko
Quoted 41 lines · Julien Grall
Quoted 21 lines · Oleksandr Andrushchenko
Quoted 20 lines · Christoph Hellwig
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code.  These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

Quoted 5 lines
Quoted 2 lines

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

I'll have to get even deeper into it. Initially I looked at the code, but didn't find anything useful. Or maybe I have just overlooked obvious things there

From Xen on Arm ABI:

"All memory which is shared with other entities in the system (including the hypervisor and other guests) must reside in memory which is mapped as Normal Inner Write-Back Outer Write-Back Inner-Shareable.
This applies to:
  - hypercall arguments passed via a pointer to guest memory.
  - memory shared via the grant table mechanism (including PV I/O rings etc).
  - memory shared with the hypervisor (struct shared_info, struct vcpu_info, the grant table, etc).
"

Quoted 4 lines

So you should not need any cache maintenance here. Can you provide more details on the memory attribute you use for memory shared in both the backend and frontend?

Quoted 5 lines

It takes quite some time to collect this (because many components are involved in the
use-case), but for now the pages in the guest domain are:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

So that's the attribute for the page mapped in the frontend, right? How about the backend side?

Also, could that page be handed to the graphic card correctly? If so, is your graphic card coherent?

If one of your components is mapping with non-cacheable attributes then you are already not following the Xen Arm ABI. In that case, we would need to discuss how to extend the ABI.

Cheers,

Signature · 1 line
Julien Grall
#9 lore raw

Hello, Julien!

Quoted 49 lines · Julien Grall

Hello,

Quoted 46 lines · Oleksandr Andrushchenko
Quoted 45 lines · Julien Grall
Quoted 21 lines · Oleksandr Andrushchenko
Quoted 20 lines · Christoph Hellwig
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code.  These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

Quoted 5 lines
Quoted 2 lines

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

I'll have to get even deeper into it. Initially I looked at the code, but didn't find anything useful. Or maybe I have just overlooked obvious things there

 From Xen on Arm ABI:

"All memory which is shared with other entities in the system (including the hypervisor and other guests) must reside in memory which is mapped as Normal Inner Write-Back Outer Write-Back Inner-Shareable.
This applies to:
   - hypercall arguments passed via a pointer to guest memory.
   - memory shared via the grant table mechanism (including PV I/O rings etc).
   - memory shared with the hypervisor (struct shared_info, struct vcpu_info, the grant table, etc).
"

So you should not need any cache maintenance here. Can you provide more details on the memory attribute you use for memory shared in both the backend and frontend?

Quoted 8 lines
Quoted 5 lines

It takes quite some time to collect this (because many components are involved in the
use-case), but for now the pages in the guest domain are:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

So that's the attribute for the page mapped in the frontend, right? How about the backend side?

Please see below

Quoted 2 lines

Also, could that page be handed to the graphic card correctly?

Yes, if we use zero-copying. But please see below

Quoted 1 line

If so, is your graphic card coherent?

Yes, it is

Quoted 7 lines

If one of your components is mapping with non-cacheable attributes then you are already not following the Xen Arm ABI. In that case, we would need to discuss how to extend the ABI.

Cheers,

Well, I didn't get the attributes of pages at the backend side, but IMO those
do not matter in my use-case (for simplicity I am not using zero-copying at backend side):

1. Frontend device allocates display buffer pages which come from shmem and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

2. Frontend grants references to these pages and shares those with the backend

3. Backend is a user-space application (Weston client), so it uses gntdev kernel
driver to mmap the pages. The pages, which are used by gntdev, are those coming
from the Xen balloon driver and I believe they are all normal memory and shouldn't be non-cached.

4. Once the frontend starts displaying it flips the buffers and backend does *memcpy*
from the frontend-backend shared buffer into Weston's buffer. This means no HW at the backend side touches the shared buffer.

5. I can see distorted picture.

Previously I used setup with zero-copying, so then the picture becomes more complicated
in terms of buffers and how those used by the backed, but anyways it seems that the
very basic scenario with memory copying doesn't work for me.

Using DMA API on frontend's side does help - no artifacts are seen. This is why I'm thinking that this is related to frontend/kernel side rather then to
the backend side. This is why I'm thinking this is related to caches and trying to figure
out what can be done here instead of using DMA API.

Thank you,
Olkesandr

#10 lore raw
Quoted 1 line · Oleksandr Andrushchenko

Hello, Julien!

Hi,

Quoted 5 lines · Julien Grall

Well, I didn't get the attributes of pages at the backend side, but IMO those
do not matter in my use-case (for simplicity I am not using zero-copying at backend side):

They are actually important no matter what is your use case. If you access the same physical page with different attributes, then you are asking for trouble.

This is why Xen imposes all the pages shared to have their memory attributes following some rules. Actually, speaking with Mark R., we may want to tight a bit more the attributes.

Quoted 5 lines

1. Frontend device allocates display buffer pages which come from shmem and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

My knowledge of Xen DRM is inexistent. However, looking at the code in 5.0-rc2, I don't seem to find the same attributes. For instance xen_drm_front_gem_prime_vmap and gem_mmap_obj are using pgprot_writecombine. So it looks like, the mapping will be non-cacheable on Arm64.

Can you explain how you came up to these attributes?

Quoted 24 lines

2. Frontend grants references to these pages and shares those with the backend

3. Backend is a user-space application (Weston client), so it uses gntdev kernel
driver to mmap the pages. The pages, which are used by gntdev, are those coming
from the Xen balloon driver and I believe they are all normal memory and shouldn't be non-cached.

4. Once the frontend starts displaying it flips the buffers and backend does *memcpy*
from the frontend-backend shared buffer into Weston's buffer. This means no HW at the backend side touches the shared buffer.

5. I can see distorted picture.

Previously I used setup with zero-copying, so then the picture becomes more complicated
in terms of buffers and how those used by the backed, but anyways it seems that the
very basic scenario with memory copying doesn't work for me.

Quoted 6 lines

Using DMA API on frontend's side does help - no artifacts are seen. This is why I'm thinking that this is related to frontend/kernel side rather then to
the backend side. This is why I'm thinking this is related to caches and trying to figure
out what can be done here instead of using DMA API.

We actually never required to use cache flush in other PV protocol, so I still don't understand why the PV DRM should be different here.

To me, it looks like that you are either missing some barriers or the memory attributes are not correct.

Cheers,

Signature · 1 line
Julien Grall
#11 lore raw

Hello, Julien!
Sorry for the late reply - it took quite some time to collect the data requested.

Quoted 14 lines · Julien Grall
Quoted 1 line · Oleksandr Andrushchenko

Hello, Julien!

Hi,

Quoted 6 lines · Julien Grall

Well, I didn't get the attributes of pages at the backend side, but IMO those
do not matter in my use-case (for simplicity I am not using zero-copying at
backend side):

Quoted 3 lines

They are actually important no matter what is your use case. If you access the same physical page with different attributes, then you are asking for trouble.

So, we have:

DomU: frontend side
====================
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

DomD: backend side
====================
PTE_USER + !PTE_RDONLY + PTE_PXN + PTE_NG + PTE_CONT + PTE_TABLE_BIT + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

From the above it seems that I don't violate cached/non-cached
agreement here

Quoted 11 lines

This is why Xen imposes all the pages shared to have their memory attributes following some rules. Actually, speaking with Mark R., we may want to tight a bit more the attributes.

Quoted 5 lines

1. Frontend device allocates display buffer pages which come from shmem and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

Quoted 7 lines

My knowledge of Xen DRM is inexistent. However, looking at the code in 5.0-rc2, I don't seem to find the same attributes. For instance xen_drm_front_gem_prime_vmap and gem_mmap_obj are using pgprot_writecombine. So it looks like, the mapping will be non-cacheable on Arm64.

Can you explain how you came up to these attributes?

pgprot_writecombine is PTE_ATTRINDX(MT_NORMAL_NC), so it seems to be applicable here? [1]

Quoted 32 lines
Quoted 25 lines

2. Frontend grants references to these pages and shares those with the backend

3. Backend is a user-space application (Weston client), so it uses gntdev kernel
driver to mmap the pages. The pages, which are used by gntdev, are those coming
from the Xen balloon driver and I believe they are all normal memory and shouldn't be non-cached.

4. Once the frontend starts displaying it flips the buffers and backend does *memcpy*
from the frontend-backend shared buffer into Weston's buffer. This means no HW at the backend side touches the shared buffer.

5. I can see distorted picture.

Previously I used setup with zero-copying, so then the picture becomes more complicated
in terms of buffers and how those used by the backed, but anyways it seems that the
very basic scenario with memory copying doesn't work for me.

Quoted 6 lines

Using DMA API on frontend's side does help - no artifacts are seen. This is why I'm thinking that this is related to frontend/kernel side rather then to
the backend side. This is why I'm thinking this is related to caches and trying to figure
out what can be done here instead of using DMA API.

Quoted 2 lines

We actually never required to use cache flush in other PV protocol, so I still don't understand why the PV DRM should be different here.

Well, you are right. But at the same time not flushing the buffer makes troubles,
so this is why I am trying to figure out what is wrong here.

Quoted 2 lines

To me, it looks like that you are either missing some barriers

Barriers for the buffer? Not sure what you mean here. Even more, we have a use case
when the buffer is not touched by CPU in DomD and is solely owned by the HW.

Quoted 1 line

or the memory attributes are not correct.

Please see above - I do need your advice here

Quoted 3 lines

Cheers,

Thank you very much for your time,
Oleksandr

[1]
https://elixir.bootlin.com/linux/v5.0-rc2/source/arch/arm64/include/asm/pgtable.h#L414

#12 lore raw
Quoted 1 line · Oleksandr Andrushchenko

Hello, Julien!

Hi,

Quoted 18 lines · Julien Grall
Quoted 14 lines · Oleksandr Andrushchenko
Quoted 1 line

Hello, Julien!

Hi,

Quoted 6 lines · Julien Grall

Well, I didn't get the attributes of pages at the backend side, but IMO those
do not matter in my use-case (for simplicity I am not using zero-copying at
backend side):

Quoted 3 lines

They are actually important no matter what is your use case. If you access the same physical page with different attributes, then you are asking for trouble.

Quoted 6 lines

So, we have:

DomU: frontend side
====================
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

I still don't understand how you came up with MT_NORMAL when you seem to confirm...

Quoted 25 lines

DomD: backend side
====================
PTE_USER + !PTE_RDONLY + PTE_PXN + PTE_NG + PTE_CONT + PTE_TABLE_BIT + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

From the above it seems that I don't violate cached/non-cached
agreement here

Quoted 17 lines

This is why Xen imposes all the pages shared to have their memory attributes following some rules. Actually, speaking with Mark R., we may want to tight a bit more the attributes.

Quoted 5 lines

1. Frontend device allocates display buffer pages which come from shmem and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

My knowledge of Xen DRM is inexistent. However, looking at the code in 5.0-rc2, I don't seem to find the same attributes. For instance xen_drm_front_gem_prime_vmap and gem_mmap_obj are using pgprot_writecombine. So it looks like, the mapping will be non-cacheable on Arm64.

Quoted 3 lines
Quoted 1 line

Can you explain how you came up to these attributes?

pgprot_writecombine is PTE_ATTRINDX(MT_NORMAL_NC), so it seems to be applicable here? [1]

... that MT_NORMAL_NC is used for the frontend pages.

MT_NORMAL_NC is different from MT_NORMAL. The use of the former will result to non-cacheable memory while the latter will result to cacheable memory.

To me, this looks like the exact reason why you see artifact on the display buffer. As the author of this code, can you explain why you decided to use pgprot_writecombine here instead of relying on the default VMA prot?

[...]

Quoted 5 lines
Quoted 2 lines

We actually never required to use cache flush in other PV protocol, so I still don't understand why the PV DRM should be different here.

Well, you are right. But at the same time not flushing the buffer makes troubles,
so this is why I am trying to figure out what is wrong here.

The cache flush is likely hiding the real problem rather than solving it.

Quoted 3 lines
Quoted 2 lines

To me, it looks like that you are either missing some barriers

Barriers for the buffer? Not sure what you mean here.

If you share information between two entities, you may need some ordering so the information are seen consistently by the consumer side. This can be achieved by using barriers.

Quoted 3 lines

Even more, we have
a use case
when the buffer is not touched by CPU in DomD and is solely owned by the HW.

Memory ordering issues are subtle. The fact that one of your use-case works does not imply that barriers are not necessary. I am not saying there are a missing barriers, I am only pointed out potential reasons.

Anyway, I don't think your problem is a missing barriers here. It is more likely because of mismatch memory attributes (see above).

Cheers,

Signature · 1 line
Julien Grall
#13 lore raw
Quoted 72 lines · Julien Grall
Quoted 1 line · Oleksandr Andrushchenko

Hello, Julien!

Hi,

Quoted 19 lines · Julien Grall
Quoted 15 lines · Oleksandr Andrushchenko
Quoted 1 line

Hello, Julien!

Hi,

Quoted 7 lines · Julien Grall

Well, I didn't get the attributes of pages at the backend side, but IMO
those
do not matter in my use-case (for simplicity I am not using zero-copying at
backend side):

Quoted 3 lines

They are actually important no matter what is your use case. If you access the same physical page with different attributes, then you are asking for trouble.

Quoted 6 lines

So, we have:

DomU: frontend side
====================
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

I still don't understand how you came up with MT_NORMAL when you seem to confirm...

Quoted 26 lines

DomD: backend side
====================
PTE_USER + !PTE_RDONLY + PTE_PXN + PTE_NG + PTE_CONT + PTE_TABLE_BIT + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

  From the above it seems that I don't violate cached/non-cached agreement here

Quoted 18 lines

This is why Xen imposes all the pages shared to have their memory attributes following some rules. Actually, speaking with Mark R., we may want to tight a bit more the attributes.

Quoted 6 lines

1. Frontend device allocates display buffer pages which come from shmem
and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

My knowledge of Xen DRM is inexistent. However, looking at the code in 5.0-rc2, I don't seem to find the same attributes. For instance xen_drm_front_gem_prime_vmap and gem_mmap_obj are using pgprot_writecombine. So it looks like, the mapping will be non-cacheable on Arm64.

Quoted 3 lines
Quoted 1 line

Can you explain how you came up to these attributes?

pgprot_writecombine is PTE_ATTRINDX(MT_NORMAL_NC), so it seems to be applicable here? [1]

... that MT_NORMAL_NC is used for the frontend pages.

MT_NORMAL_NC is different from MT_NORMAL. The use of the former will result to non-cacheable memory while the latter will result to cacheable memory.

Quoted 6 lines

To me, this looks like the exact reason why you see artifact on the display buffer. As the author of this code, can you explain why you decided to use pgprot_writecombine here instead of relying on the default VMA prot?

[...]

Sorry for late reply, it took me quite some time to re-check all the use-cases
that we have with PV DRM.
First of all I found a bug in my debug code which reported page attributes and
now I can confirm that all the use-cases sue MT_NORMAL, both backend and frontend.
You are right: there is no reason to have pgprot_writecombine and indeed I have
to rely on almost default VMA prot. I came up with the following (used by omap drm,
for example):

(quoted diff) +0 -0

b/drivers/gpu/drm/xen/xen_drm_front_gem.c
index ae28ad4b4254..867800a2ed42 100644

drivers/gpu/drm/xen/xen_drm_front_gem.c +1 -2
@@ -238,8 +238,7 @@ static int gem_mmap_obj(struct xen_gem_object *xen_obj,
         vma->vm_flags &= ~VM_PFNMAP;
         vma->vm_flags |= VM_MIXEDMAP;
         vma->vm_pgoff = 0;
-       vma->vm_page_prot =
- pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
+       vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
 
         {
                 int cnt = xen_obj->num_pages > 5 ? 5 : xen_obj->num_pages;
@@ -295,7 +294,7 @@ void *xen_drm_front_gem_prime_vmap(struct 

drm_gem_object *gem_obj)
                return NULL;

        return vmap(xen_obj->pages, xen_obj->num_pages,
-                   VM_MAP, pgprot_writecombine(PAGE_KERNEL)); +                   VM_MAP, PAGE_KERNEL);
 }

With the above all the artifacts are gone now as page attributes are the same across
all domains. So, I consider this as a fix and will send it as v3 or better drop this patch and have a new one.

THANK YOU for helping figuring this out!

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

We actually never required to use cache flush in other PV protocol, so I still don't understand why the PV DRM should be different here.

Well, you are right. But at the same time not flushing the buffer makes troubles,
so this is why I am trying to figure out what is wrong here.

The cache flush is likely hiding the real problem rather than solving it.

It does hide the real issue. And I have confirmed this

Quoted 13 lines
Quoted 3 lines
Quoted 2 lines

To me, it looks like that you are either missing some barriers

Barriers for the buffer? Not sure what you mean here.

If you share information between two entities, you may need some ordering so the information are seen consistently by the consumer side. This can be achieved by using barriers.

Quoted 4 lines

Even more, we have
a use case
when the buffer is not touched by CPU in DomD and is solely owned by the HW.

Quoted 7 lines

Memory ordering issues are subtle. The fact that one of your use-case works does not imply that barriers are not necessary. I am not saying there are a missing barriers, I am only pointed out potential reasons.

Anyway, I don't think your problem is a missing barriers here. It is more likely because of mismatch memory attributes (see above).

Yes, please see above

Quoted 2 lines

Cheers,

Thank you all for helping with the correct fix,
Oleksandr

#14 lore raw
In reply to Julien Grall
Quoted 72 lines · Julien Grall
Quoted 1 line · Oleksandr Andrushchenko

Hello, Julien!

Hi,

Quoted 19 lines · Julien Grall
Quoted 15 lines · Oleksandr Andrushchenko
Quoted 1 line

Hello, Julien!

Hi,

Quoted 7 lines · Julien Grall

Well, I didn't get the attributes of pages at the backend side, but IMO
those
do not matter in my use-case (for simplicity I am not using zero-copying at
backend side):

Quoted 3 lines

They are actually important no matter what is your use case. If you access the same physical page with different attributes, then you are asking for trouble.

Quoted 6 lines

So, we have:

DomU: frontend side
====================
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

I still don't understand how you came up with MT_NORMAL when you seem to confirm...

Quoted 26 lines

DomD: backend side
====================
PTE_USER + !PTE_RDONLY + PTE_PXN + PTE_NG + PTE_CONT + PTE_TABLE_BIT + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

  From the above it seems that I don't violate cached/non-cached agreement here

Quoted 18 lines

This is why Xen imposes all the pages shared to have their memory attributes following some rules. Actually, speaking with Mark R., we may want to tight a bit more the attributes.

Quoted 6 lines

1. Frontend device allocates display buffer pages which come from shmem
and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

My knowledge of Xen DRM is inexistent. However, looking at the code in 5.0-rc2, I don't seem to find the same attributes. For instance xen_drm_front_gem_prime_vmap and gem_mmap_obj are using pgprot_writecombine. So it looks like, the mapping will be non-cacheable on Arm64.

Quoted 3 lines
Quoted 1 line

Can you explain how you came up to these attributes?

pgprot_writecombine is PTE_ATTRINDX(MT_NORMAL_NC), so it seems to be applicable here? [1]

... that MT_NORMAL_NC is used for the frontend pages.

MT_NORMAL_NC is different from MT_NORMAL. The use of the former will result to non-cacheable memory while the latter will result to cacheable memory.

Quoted 6 lines

To me, this looks like the exact reason why you see artifact on the display buffer. As the author of this code, can you explain why you decided to use pgprot_writecombine here instead of relying on the default VMA prot?

[...]

Sorry for late reply, it took me quite some time to re-check all the use-cases
that we have with PV DRM.
First of all I found a bug in my debug code which reported page attributes and
now I can confirm that all the use-cases sue MT_NORMAL, both backend and frontend.
You are right: there is no reason to have pgprot_writecombine and indeed I have
to rely on almost default VMA prot. I came up with the following (used by omap drm,
for example):

(quoted diff) +0 -0

b/drivers/gpu/drm/xen/xen_drm_front_gem.c
index ae28ad4b4254..867800a2ed42 100644

drivers/gpu/drm/xen/xen_drm_front_gem.c +1 -2
@@ -238,8 +238,7 @@ static int gem_mmap_obj(struct xen_gem_object *xen_obj,
         vma->vm_flags &= ~VM_PFNMAP;
         vma->vm_flags |= VM_MIXEDMAP;
         vma->vm_pgoff = 0;
-       vma->vm_page_prot =
- pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
+       vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
 
         {
                 int cnt = xen_obj->num_pages > 5 ? 5 : xen_obj->num_pages;
@@ -295,7 +294,7 @@ void *xen_drm_front_gem_prime_vmap(struct 

drm_gem_object *gem_obj)
                return NULL;

        return vmap(xen_obj->pages, xen_obj->num_pages,
-                   VM_MAP, pgprot_writecombine(PAGE_KERNEL)); +                   VM_MAP, PAGE_KERNEL);
 }

With the above all the artifacts are gone now as page attributes are the same across
all domains. So, I consider this as a fix and will send it as v3 or better drop this patch and have a new one.

THANK YOU for helping figuring this out!

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

We actually never required to use cache flush in other PV protocol, so I still don't understand why the PV DRM should be different here.

Well, you are right. But at the same time not flushing the buffer makes troubles,
so this is why I am trying to figure out what is wrong here.

The cache flush is likely hiding the real problem rather than solving it.

It does hide the real issue. And I have confirmed this

Quoted 13 lines
Quoted 3 lines
Quoted 2 lines

To me, it looks like that you are either missing some barriers

Barriers for the buffer? Not sure what you mean here.

If you share information between two entities, you may need some ordering so the information are seen consistently by the consumer side. This can be achieved by using barriers.

Quoted 4 lines

Even more, we have
a use case
when the buffer is not touched by CPU in DomD and is solely owned by the HW.

Quoted 7 lines

Memory ordering issues are subtle. The fact that one of your use-case works does not imply that barriers are not necessary. I am not saying there are a missing barriers, I am only pointed out potential reasons.

Anyway, I don't think your problem is a missing barriers here. It is more likely because of mismatch memory attributes (see above).

Yes, please see above

Quoted 2 lines

Cheers,

Thank you all for helping with the correct fix,
Oleksandr
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw
In reply to Julien Grall

Dropped in favor of https://lkml.org/lkml/2019/1/29/910

Quoted 109 lines · Julien Grall
Quoted 1 line · Oleksandr Andrushchenko

Hello, Julien!

Hi,

Quoted 19 lines · Julien Grall
Quoted 15 lines · Oleksandr Andrushchenko
Quoted 1 line

Hello, Julien!

Hi,

Quoted 7 lines · Julien Grall

Well, I didn't get the attributes of pages at the backend side, but IMO
those
do not matter in my use-case (for simplicity I am not using zero-copying at
backend side):

Quoted 3 lines

They are actually important no matter what is your use case. If you access the same physical page with different attributes, then you are asking for trouble.

Quoted 6 lines

So, we have:

DomU: frontend side
====================
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

I still don't understand how you came up with MT_NORMAL when you seem to confirm...

Quoted 26 lines

DomD: backend side
====================
PTE_USER + !PTE_RDONLY + PTE_PXN + PTE_NG + PTE_CONT + PTE_TABLE_BIT + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

  From the above it seems that I don't violate cached/non-cached agreement here

Quoted 18 lines

This is why Xen imposes all the pages shared to have their memory attributes following some rules. Actually, speaking with Mark R., we may want to tight a bit more the attributes.

Quoted 6 lines

1. Frontend device allocates display buffer pages which come from shmem
and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

My knowledge of Xen DRM is inexistent. However, looking at the code in 5.0-rc2, I don't seem to find the same attributes. For instance xen_drm_front_gem_prime_vmap and gem_mmap_obj are using pgprot_writecombine. So it looks like, the mapping will be non-cacheable on Arm64.

Quoted 3 lines
Quoted 1 line

Can you explain how you came up to these attributes?

pgprot_writecombine is PTE_ATTRINDX(MT_NORMAL_NC), so it seems to be applicable here? [1]

... that MT_NORMAL_NC is used for the frontend pages.

MT_NORMAL_NC is different from MT_NORMAL. The use of the former will result to non-cacheable memory while the latter will result to cacheable memory.

To me, this looks like the exact reason why you see artifact on the display buffer. As the author of this code, can you explain why you decided to use pgprot_writecombine here instead of relying on the default VMA prot?

[...]

Quoted 5 lines
Quoted 2 lines

We actually never required to use cache flush in other PV protocol, so I still don't understand why the PV DRM should be different here.

Well, you are right. But at the same time not flushing the buffer makes troubles,
so this is why I am trying to figure out what is wrong here.

The cache flush is likely hiding the real problem rather than solving it.

Quoted 3 lines
Quoted 2 lines

To me, it looks like that you are either missing some barriers

Barriers for the buffer? Not sure what you mean here.

If you share information between two entities, you may need some ordering so the information are seen consistently by the consumer side. This can be achieved by using barriers.

Quoted 4 lines

Even more, we have
a use case
when the buffer is not touched by CPU in DomD and is solely owned by the HW.

Memory ordering issues are subtle. The fact that one of your use-case works does not imply that barriers are not necessary. I am not saying there are a missing barriers, I am only pointed out potential reasons.

Anyway, I don't think your problem is a missing barriers here. It is more likely because of mismatch memory attributes (see above).

Cheers,

#16 lore raw
In reply to Julien Grall

Dropped in favor of https://lkml.org/lkml/2019/1/29/910

Quoted 100 lines · Julien Grall
Quoted 1 line · Oleksandr Andrushchenko

Hello, Julien!

Hi,

Quoted 19 lines · Julien Grall
Quoted 15 lines · Oleksandr Andrushchenko
Quoted 1 line

Hello, Julien!

Hi,

Quoted 7 lines · Julien Grall

Well, I didn't get the attributes of pages at the backend side, but IMO
those
do not matter in my use-case (for simplicity I am not using zero-copying at
backend side):

Quoted 3 lines

They are actually important no matter what is your use case. If you access the same physical page with different attributes, then you are asking for trouble.

Quoted 6 lines

So, we have:

DomU: frontend side
====================
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

I still don't understand how you came up with MT_NORMAL when you seem to confirm...

Quoted 26 lines

DomD: backend side
====================
PTE_USER + !PTE_RDONLY + PTE_PXN + PTE_NG + PTE_CONT + PTE_TABLE_BIT + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

  From the above it seems that I don't violate cached/non-cached agreement here

Quoted 18 lines

This is why Xen imposes all the pages shared to have their memory attributes following some rules. Actually, speaking with Mark R., we may want to tight a bit more the attributes.

Quoted 6 lines

1. Frontend device allocates display buffer pages which come from shmem
and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

My knowledge of Xen DRM is inexistent. However, looking at the code in 5.0-rc2, I don't seem to find the same attributes. For instance xen_drm_front_gem_prime_vmap and gem_mmap_obj are using pgprot_writecombine. So it looks like, the mapping will be non-cacheable on Arm64.

Quoted 3 lines
Quoted 1 line

Can you explain how you came up to these attributes?

pgprot_writecombine is PTE_ATTRINDX(MT_NORMAL_NC), so it seems to be applicable here? [1]

... that MT_NORMAL_NC is used for the frontend pages.

MT_NORMAL_NC is different from MT_NORMAL. The use of the former will result to non-cacheable memory while the latter will result to cacheable memory.

To me, this looks like the exact reason why you see artifact on the display buffer. As the author of this code, can you explain why you decided to use pgprot_writecombine here instead of relying on the default VMA prot?

[...]

Quoted 5 lines
Quoted 2 lines

We actually never required to use cache flush in other PV protocol, so I still don't understand why the PV DRM should be different here.

Well, you are right. But at the same time not flushing the buffer makes troubles,
so this is why I am trying to figure out what is wrong here.

The cache flush is likely hiding the real problem rather than solving it.

Quoted 3 lines
Quoted 2 lines

To me, it looks like that you are either missing some barriers

Barriers for the buffer? Not sure what you mean here.

If you share information between two entities, you may need some ordering so the information are seen consistently by the consumer side. This can be achieved by using barriers.

Quoted 4 lines

Even more, we have
a use case
when the buffer is not touched by CPU in DomD and is solely owned by the HW.

Quoted 9 lines

Memory ordering issues are subtle. The fact that one of your use-case works does not imply that barriers are not necessary. I am not saying there are a missing barriers, I am only pointed out potential reasons.

Anyway, I don't think your problem is a missing barriers here. It is more likely because of mismatch memory attributes (see above).

Cheers,

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#17 lore raw
In reply to Oleksandr Andrushchenko
Quoted 1 line · Oleksandr Andrushchenko

Hello, Julien!

Hi,

Quoted 18 lines · Julien Grall
Quoted 14 lines · Oleksandr Andrushchenko
Quoted 1 line

Hello, Julien!

Hi,

Quoted 6 lines · Julien Grall

Well, I didn't get the attributes of pages at the backend side, but IMO those
do not matter in my use-case (for simplicity I am not using zero-copying at
backend side):

Quoted 3 lines

They are actually important no matter what is your use case. If you access the same physical page with different attributes, then you are asking for trouble.

Quoted 6 lines

So, we have:

DomU: frontend side
====================
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

I still don't understand how you came up with MT_NORMAL when you seem to confirm...

Quoted 25 lines

DomD: backend side
====================
PTE_USER + !PTE_RDONLY + PTE_PXN + PTE_NG + PTE_CONT + PTE_TABLE_BIT + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

From the above it seems that I don't violate cached/non-cached
agreement here

Quoted 17 lines

This is why Xen imposes all the pages shared to have their memory attributes following some rules. Actually, speaking with Mark R., we may want to tight a bit more the attributes.

Quoted 5 lines

1. Frontend device allocates display buffer pages which come from shmem and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

My knowledge of Xen DRM is inexistent. However, looking at the code in 5.0-rc2, I don't seem to find the same attributes. For instance xen_drm_front_gem_prime_vmap and gem_mmap_obj are using pgprot_writecombine. So it looks like, the mapping will be non-cacheable on Arm64.

Quoted 3 lines
Quoted 1 line

Can you explain how you came up to these attributes?

pgprot_writecombine is PTE_ATTRINDX(MT_NORMAL_NC), so it seems to be applicable here? [1]

... that MT_NORMAL_NC is used for the frontend pages.

MT_NORMAL_NC is different from MT_NORMAL. The use of the former will result to non-cacheable memory while the latter will result to cacheable memory.

To me, this looks like the exact reason why you see artifact on the display buffer. As the author of this code, can you explain why you decided to use pgprot_writecombine here instead of relying on the default VMA prot?

[...]

Quoted 5 lines
Quoted 2 lines

We actually never required to use cache flush in other PV protocol, so I still don't understand why the PV DRM should be different here.

Well, you are right. But at the same time not flushing the buffer makes troubles,
so this is why I am trying to figure out what is wrong here.

The cache flush is likely hiding the real problem rather than solving it.

Quoted 3 lines
Quoted 2 lines

To me, it looks like that you are either missing some barriers

Barriers for the buffer? Not sure what you mean here.

If you share information between two entities, you may need some ordering so the information are seen consistently by the consumer side. This can be achieved by using barriers.

Quoted 3 lines

Even more, we have
a use case
when the buffer is not touched by CPU in DomD and is solely owned by the HW.

Memory ordering issues are subtle. The fact that one of your use-case works does not imply that barriers are not necessary. I am not saying there are a missing barriers, I am only pointed out potential reasons.

Anyway, I don't think your problem is a missing barriers here. It is more likely because of mismatch memory attributes (see above).

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#18 lore raw
In reply to Julien Grall

Hello, Julien!
Sorry for the late reply - it took quite some time to collect the data requested.

Quoted 14 lines · Julien Grall
Quoted 1 line · Oleksandr Andrushchenko

Hello, Julien!

Hi,

Quoted 6 lines · Julien Grall

Well, I didn't get the attributes of pages at the backend side, but IMO those
do not matter in my use-case (for simplicity I am not using zero-copying at
backend side):

Quoted 3 lines

They are actually important no matter what is your use case. If you access the same physical page with different attributes, then you are asking for trouble.

So, we have:

DomU: frontend side
====================
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

DomD: backend side
====================
PTE_USER + !PTE_RDONLY + PTE_PXN + PTE_NG + PTE_CONT + PTE_TABLE_BIT + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

From the above it seems that I don't violate cached/non-cached
agreement here

Quoted 11 lines

This is why Xen imposes all the pages shared to have their memory attributes following some rules. Actually, speaking with Mark R., we may want to tight a bit more the attributes.

Quoted 5 lines

1. Frontend device allocates display buffer pages which come from shmem and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

Quoted 7 lines

My knowledge of Xen DRM is inexistent. However, looking at the code in 5.0-rc2, I don't seem to find the same attributes. For instance xen_drm_front_gem_prime_vmap and gem_mmap_obj are using pgprot_writecombine. So it looks like, the mapping will be non-cacheable on Arm64.

Can you explain how you came up to these attributes?

pgprot_writecombine is PTE_ATTRINDX(MT_NORMAL_NC), so it seems to be applicable here? [1]

Quoted 32 lines
Quoted 25 lines

2. Frontend grants references to these pages and shares those with the backend

3. Backend is a user-space application (Weston client), so it uses gntdev kernel
driver to mmap the pages. The pages, which are used by gntdev, are those coming
from the Xen balloon driver and I believe they are all normal memory and shouldn't be non-cached.

4. Once the frontend starts displaying it flips the buffers and backend does *memcpy*
from the frontend-backend shared buffer into Weston's buffer. This means no HW at the backend side touches the shared buffer.

5. I can see distorted picture.

Previously I used setup with zero-copying, so then the picture becomes more complicated
in terms of buffers and how those used by the backed, but anyways it seems that the
very basic scenario with memory copying doesn't work for me.

Quoted 6 lines

Using DMA API on frontend's side does help - no artifacts are seen. This is why I'm thinking that this is related to frontend/kernel side rather then to
the backend side. This is why I'm thinking this is related to caches and trying to figure
out what can be done here instead of using DMA API.

Quoted 2 lines

We actually never required to use cache flush in other PV protocol, so I still don't understand why the PV DRM should be different here.

Well, you are right. But at the same time not flushing the buffer makes troubles,
so this is why I am trying to figure out what is wrong here.

Quoted 2 lines

To me, it looks like that you are either missing some barriers

Barriers for the buffer? Not sure what you mean here. Even more, we have a use case
when the buffer is not touched by CPU in DomD and is solely owned by the HW.

Quoted 1 line

or the memory attributes are not correct.

Please see above - I do need your advice here

Quoted 3 lines

Cheers,

Thank you very much for your time,
Oleksandr

[1]
https://elixir.bootlin.com/linux/v5.0-rc2/source/arch/arm64/include/asm/pgtable.h#L414
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#19 lore raw
In reply to Oleksandr Andrushchenko
Quoted 1 line · Oleksandr Andrushchenko

Hello, Julien!

Hi,

Quoted 5 lines · Julien Grall

Well, I didn't get the attributes of pages at the backend side, but IMO those
do not matter in my use-case (for simplicity I am not using zero-copying at backend side):

They are actually important no matter what is your use case. If you access the same physical page with different attributes, then you are asking for trouble.

This is why Xen imposes all the pages shared to have their memory attributes following some rules. Actually, speaking with Mark R., we may want to tight a bit more the attributes.

Quoted 5 lines

1. Frontend device allocates display buffer pages which come from shmem and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

My knowledge of Xen DRM is inexistent. However, looking at the code in 5.0-rc2, I don't seem to find the same attributes. For instance xen_drm_front_gem_prime_vmap and gem_mmap_obj are using pgprot_writecombine. So it looks like, the mapping will be non-cacheable on Arm64.

Can you explain how you came up to these attributes?

Quoted 24 lines

2. Frontend grants references to these pages and shares those with the backend

3. Backend is a user-space application (Weston client), so it uses gntdev kernel
driver to mmap the pages. The pages, which are used by gntdev, are those coming
from the Xen balloon driver and I believe they are all normal memory and shouldn't be non-cached.

4. Once the frontend starts displaying it flips the buffers and backend does *memcpy*
from the frontend-backend shared buffer into Weston's buffer. This means no HW at the backend side touches the shared buffer.

5. I can see distorted picture.

Previously I used setup with zero-copying, so then the picture becomes more complicated
in terms of buffers and how those used by the backed, but anyways it seems that the
very basic scenario with memory copying doesn't work for me.

Quoted 6 lines

Using DMA API on frontend's side does help - no artifacts are seen. This is why I'm thinking that this is related to frontend/kernel side rather then to
the backend side. This is why I'm thinking this is related to caches and trying to figure
out what can be done here instead of using DMA API.

We actually never required to use cache flush in other PV protocol, so I still don't understand why the PV DRM should be different here.

To me, it looks like that you are either missing some barriers or the memory attributes are not correct.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#20 lore raw
In reply to Julien Grall

Hello, Julien!

Quoted 49 lines · Julien Grall

Hello,

Quoted 46 lines · Oleksandr Andrushchenko
Quoted 45 lines · Julien Grall
Quoted 21 lines · Oleksandr Andrushchenko
Quoted 20 lines · Christoph Hellwig
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code.  These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

Quoted 5 lines
Quoted 2 lines

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

I'll have to get even deeper into it. Initially I looked at the code, but didn't find anything useful. Or maybe I have just overlooked obvious things there

 From Xen on Arm ABI:

"All memory which is shared with other entities in the system (including the hypervisor and other guests) must reside in memory which is mapped as Normal Inner Write-Back Outer Write-Back Inner-Shareable.
This applies to:
   - hypercall arguments passed via a pointer to guest memory.
   - memory shared via the grant table mechanism (including PV I/O rings etc).
   - memory shared with the hypervisor (struct shared_info, struct vcpu_info, the grant table, etc).
"

So you should not need any cache maintenance here. Can you provide more details on the memory attribute you use for memory shared in both the backend and frontend?

Quoted 8 lines
Quoted 5 lines

It takes quite some time to collect this (because many components are involved in the
use-case), but for now the pages in the guest domain are:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

So that's the attribute for the page mapped in the frontend, right? How about the backend side?

Please see below

Quoted 2 lines

Also, could that page be handed to the graphic card correctly?

Yes, if we use zero-copying. But please see below

Quoted 1 line

If so, is your graphic card coherent?

Yes, it is

Quoted 7 lines

If one of your components is mapping with non-cacheable attributes then you are already not following the Xen Arm ABI. In that case, we would need to discuss how to extend the ABI.

Cheers,

Well, I didn't get the attributes of pages at the backend side, but IMO those
do not matter in my use-case (for simplicity I am not using zero-copying at backend side):

1. Frontend device allocates display buffer pages which come from shmem and have these attributes:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

2. Frontend grants references to these pages and shares those with the backend

3. Backend is a user-space application (Weston client), so it uses gntdev kernel
driver to mmap the pages. The pages, which are used by gntdev, are those coming
from the Xen balloon driver and I believe they are all normal memory and shouldn't be non-cached.

4. Once the frontend starts displaying it flips the buffers and backend does *memcpy*
from the frontend-backend shared buffer into Weston's buffer. This means no HW at the backend side touches the shared buffer.

5. I can see distorted picture.

Previously I used setup with zero-copying, so then the picture becomes more complicated
in terms of buffers and how those used by the backed, but anyways it seems that the
very basic scenario with memory copying doesn't work for me.

Using DMA API on frontend's side does help - no artifacts are seen. This is why I'm thinking that this is related to frontend/kernel side rather then to
the backend side. This is why I'm thinking this is related to caches and trying to figure
out what can be done here instead of using DMA API.

Thank you,
Olkesandr
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#21 lore raw
In reply to Oleksandr Andrushchenko

Hello,

Quoted 46 lines · Oleksandr Andrushchenko
Quoted 41 lines · Julien Grall
Quoted 21 lines · Oleksandr Andrushchenko
Quoted 20 lines · Christoph Hellwig
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code.  These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

Quoted 5 lines
Quoted 2 lines

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

I'll have to get even deeper into it. Initially I looked at the code, but didn't find anything useful. Or maybe I have just overlooked obvious things there

From Xen on Arm ABI:

"All memory which is shared with other entities in the system (including the hypervisor and other guests) must reside in memory which is mapped as Normal Inner Write-Back Outer Write-Back Inner-Shareable.
This applies to:
  - hypercall arguments passed via a pointer to guest memory.
  - memory shared via the grant table mechanism (including PV I/O rings etc).
  - memory shared with the hypervisor (struct shared_info, struct vcpu_info, the grant table, etc).
"

Quoted 4 lines

So you should not need any cache maintenance here. Can you provide more details on the memory attribute you use for memory shared in both the backend and frontend?

Quoted 5 lines

It takes quite some time to collect this (because many components are involved in the
use-case), but for now the pages in the guest domain are:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

So that's the attribute for the page mapped in the frontend, right? How about the backend side?

Also, could that page be handed to the graphic card correctly? If so, is your graphic card coherent?

If one of your components is mapping with non-cacheable attributes then you are already not following the Xen Arm ABI. In that case, we would need to discuss how to extend the ABI.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#22 lore raw
In reply to Julien Grall
Quoted 47 lines · Julien Grall

(+ Stefano)

Hi,

Sorry for jumping late in the conversation.

Quoted 21 lines · Oleksandr Andrushchenko
Quoted 20 lines · Christoph Hellwig
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code.  These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

Quoted 5 lines
Quoted 2 lines

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

I'll have to get even deeper into it. Initially I looked at the code, but didn't find anything useful. Or maybe I have just overlooked obvious things there

From Xen on Arm ABI:

"All memory which is shared with other entities in the system (including the hypervisor and other guests) must reside in memory which is mapped as Normal Inner Write-Back Outer Write-Back Inner-Shareable.
This applies to:
  - hypercall arguments passed via a pointer to guest memory.
  - memory shared via the grant table mechanism (including PV I/O rings etc).
  - memory shared with the hypervisor (struct shared_info, struct vcpu_info, the grant table, etc).
"

Quoted 4 lines

So you should not need any cache maintenance here. Can you provide more details on the memory attribute you use for memory shared in both the backend and frontend?

It takes quite some time to collect this (because many components are involved in the
use-case), but for now the pages in the guest domain are:
!PTE_RDONLY + PTE_PXN + PTE_SHARED + PTE_AF + PTE_UXN + PTE_ATTRINDX(MT_NORMAL)

Quoted 11 lines

Cheers,

Quoted 8 lines

Thank you,
Oleksandr
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#23 lore raw
In reply to Oleksandr Andrushchenko

(+ Stefano)

Hi,

Sorry for jumping late in the conversation.

Quoted 20 lines · Oleksandr Andrushchenko
Quoted 19 lines · Christoph Hellwig
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code. These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

Quoted 5 lines
Quoted 2 lines

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

I'll have to get even deeper into it. Initially I looked at the code, but didn't find anything useful. Or maybe I have just overlooked obvious things there

From Xen on Arm ABI:

"All memory which is shared with other entities in the system (including the hypervisor and other guests) must reside in memory which is mapped as Normal Inner Write-Back Outer Write-Back Inner-Shareable. This applies to:
- hypercall arguments passed via a pointer to guest memory.
- memory shared via the grant table mechanism (including PV I/O rings etc).
- memory shared with the hypervisor (struct shared_info, struct vcpu_info, the grant table, etc).
"

So you should not need any cache maintenance here. Can you provide more details on the memory attribute you use for memory shared in both the backend and frontend?

Cheers,

Quoted 8 lines

Thank you,
Oleksandr
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#24 lore raw
In reply to Christoph Hellwig
Quoted 13 lines · Christoph Hellwig
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

Quoted 8 lines

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code. These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

I'll have to get even deeper into it. Initially I looked at the code, but didn't find anything useful. Or maybe I have just overlooked obvious things there

Thank you,
Oleksandr
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#25 lore raw
In reply to Oleksandr Andrushchenko
Quoted 12 lines · Oleksandr Andrushchenko
Quoted 1 line

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them

The right answer would be to implement cache maintainance hooks for this case in the Xen arch code. These would basically look the same as the low-level cache maintainance used by the DMA ops, but without going through the DMA mapping layer, in fact they should probably reuse the same low-level assembly routines.

I don't think this is the first usage of such Xen buffer sharing, so what do the other users do?

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#26 lore raw
In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Gerd Hoffmann
   Hi,

drivers/gpu/drm/xen/xen_drm_front_gem.c static struct xen_gem_object *gem_create(struct drm_device *dev, size_t size) · line 142

Quoted 5 lines
drivers/gpu/drm/xen/xen_drm_front_gem.c +5 -0
+	if (!dma_map_sg(dev->dev, xen_obj->sgt->sgl, xen_obj->sgt->nents,
+			DMA_BIDIRECTIONAL)) {
+		ret = -EFAULT;
+		goto fail_free_sgt;
+	}

Hmm, so it seems the arm guys could not come up with a suggestion how to solve that one in a better way. Ok, lets go with this then.

Quoted 8 lines
Quoted 3 lines

But didn't we agree that this deserves a comment exmplaining the purpose of the dma_map_sg() call is to flush caches and that there is no actual DMA happening here?

Using a dma mapping call to flush caches is complete no-go. But the real question is why you'd even want to flush cashes if you do not want a dma mapping?

This whole issue keeps getting more and more confusing.

Well, I don't really do DMA here, but instead the buffers in question are shared with other Xen domain, so effectively it could be thought of some sort of DMA here, where the "device" is that remote domain. If the buffers are not flushed then the remote part sees some inconsistency which in my case results in artifacts on screen while displaying the buffers. When buffers are allocated via DMA API then there are no artifacts; if buffers are allocated with shmem + DMA mapping then there are no artifacts as well.
The only offending use-case is when I use shmem backed buffers, but do not flush them
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#27 lore raw
In reply to Gerd Hoffmann
Quoted 11 lines · Gerd Hoffmann

Hi,

drivers/gpu/drm/xen/xen_drm_front_gem.c static struct xen_gem_object *gem_create(struct drm_device *dev, size_t size) · line 142

Quoted 5 lines
drivers/gpu/drm/xen/xen_drm_front_gem.c +5 -0
+	if (!dma_map_sg(dev->dev, xen_obj->sgt->sgl, xen_obj->sgt->nents,
+			DMA_BIDIRECTIONAL)) {
+		ret = -EFAULT;
+		goto fail_free_sgt;
+	}

Hmm, so it seems the arm guys could not come up with a suggestion how to solve that one in a better way. Ok, lets go with this then.

Quoted 3 lines

But didn't we agree that this deserves a comment exmplaining the purpose of the dma_map_sg() call is to flush caches and that there is no actual DMA happening here?

Using a dma mapping call to flush caches is complete no-go. But the real question is why you'd even want to flush cashes if you do not want a dma mapping?

This whole issue keeps getting more and more confusing.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#28 lore raw
In reply to Gerd Hoffmann
Quoted 10 lines · Gerd Hoffmann
   Hi,

drivers/gpu/drm/xen/xen_drm_front_gem.c static struct xen_gem_object *gem_create(struct drm_device *dev, size_t size) · line 142

Quoted 5 lines
drivers/gpu/drm/xen/xen_drm_front_gem.c +5 -0
+	if (!dma_map_sg(dev->dev, xen_obj->sgt->sgl, xen_obj->sgt->nents,
+			DMA_BIDIRECTIONAL)) {
+		ret = -EFAULT;
+		goto fail_free_sgt;
+	}

Hmm, so it seems the arm guys could not come up with a suggestion how to solve that one in a better way. Ok, lets go with this then.

Quoted 3 lines

But didn't we agree that this deserves a comment exmplaining the purpose of the dma_map_sg() call is to flush caches and that there is no actual DMA happening here?

My bad, will add the comment

Quoted 3 lines

cheers,
Gerd

Thank you,
Oleksandr

#29 lore raw
In reply to Gerd Hoffmann
Quoted 10 lines · Gerd Hoffmann
   Hi,

drivers/gpu/drm/xen/xen_drm_front_gem.c static struct xen_gem_object *gem_create(struct drm_device *dev, size_t size) · line 142

Quoted 5 lines
drivers/gpu/drm/xen/xen_drm_front_gem.c +5 -0
+	if (!dma_map_sg(dev->dev, xen_obj->sgt->sgl, xen_obj->sgt->nents,
+			DMA_BIDIRECTIONAL)) {
+		ret = -EFAULT;
+		goto fail_free_sgt;
+	}

Hmm, so it seems the arm guys could not come up with a suggestion how to solve that one in a better way. Ok, lets go with this then.

Quoted 3 lines

But didn't we agree that this deserves a comment exmplaining the purpose of the dma_map_sg() call is to flush caches and that there is no actual DMA happening here?

My bad, will add the comment

Quoted 3 lines

cheers,
Gerd

Thank you,
Oleksandr
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#30 lore raw
In reply to Oleksandr Andrushchenko

Hi,

drivers/gpu/drm/xen/xen_drm_front_gem.c static struct xen_gem_object *gem_create(struct drm_device *dev, size_t size) · line 142

Quoted 5 lines
drivers/gpu/drm/xen/xen_drm_front_gem.c +5 -0
+	if (!dma_map_sg(dev->dev, xen_obj->sgt->sgl, xen_obj->sgt->nents,
+			DMA_BIDIRECTIONAL)) {
+		ret = -EFAULT;
+		goto fail_free_sgt;
+	}

Hmm, so it seems the arm guys could not come up with a suggestion how to solve that one in a better way. Ok, lets go with this then.

But didn't we agree that this deserves a comment exmplaining the purpose of the dma_map_sg() call is to flush caches and that there is no actual DMA happening here?

cheers,
Gerd

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel