drm: msm: run into issues
Subsystems (4)
Hi Rob,
I run into issues with msm drm driver while implementing a test application which use v4l2 vidc (venus) decoder driver to decode videos and msm drm driver to display the decoded frames. The v4l2 test application use msm drm as dmabuf exporter by DRM_IOCTL_MODE_CREATE_DUMB and DRM_IOCTL_PRIME_HANDLE_TO_FD from libdrm.
So far the test app is able to decode and display using dmabuf type of buffers (so, to honest it is slower than using mmap & memcpy but this is another story). The problems start when destroying drm buffers by call to DRM_IOCTL_MODE_DESTROY_DUMB ioctl and also when closing dmabuf fd. The issues I'm seeing are:
- the first and the major one happens in msm_gem_free_object() where we are trying to free sg table which table is already freed by drm_prime core in dmabuf .detach callback. In fact the msm_gem_free_object is called by dmabuf .release callback which should be happened after .detach. I find weird to call sg_table_free in .detach callback and did not understand why it is there.
- the second one is again in msm_gem.c::put_pages dma_unmap_sg call. Some background, vidc (venus) driver use dma-iommu infrastructure copped from qcom downstream kernel to abstract iommu map/unmap [1]. On the other side when drm driver is exporter it use swiotbl [2] dma map operations, dma_map_sg will fill sg->dma_address with phy addresses. So when vidc call dma_map_sg the sg dma_address is filled with iova address then drm call dma_unmap_sg expecting phy addresses.
If I didn't express myself well, see the hack bellow which I made to prevent kernel crash.
Any thoughts?
drivers/gpu/drm/drm_prime.c +0 -0
@@ -155,10 +155,10 @@ static void drm_gem_map_detach(struct dma_buf
*dma_buf,
if (prime_attach->dir != DMA_NONE)
dma_unmap_sg(attach->dev, sgt->sgl, sgt->nents,
prime_attach->dir);
- sg_free_table(sgt);
+/* sg_free_table(sgt);*/
}- kfree(sgt);
+/* kfree(sgt);*/
kfree(prime_attach);
attach->priv = NULL;}
drivers/gpu/drm/msm/msm_gem.c +1 -1
@@ -116,9 +116,10 @@ static void put_pages(struct drm_gem_object *obj)/* For non-cached buffers, ensure the new pages are clean * because display controller, GPU, etc. are not coherent: */ -if (msm_obj->flags & (MSM_BO_WC |MSM_BO_UNCACHED )) +/* if (msm_obj->flags & (MSM_BO_WC|MSM_BO_UNCACHED)) dma_unmap_sg(obj->dev->dev, msm_obj->sgt->sgl, msm_obj->sgt->nents,
DMA_BIDIRECTIONAL);
+*/
sg_free_table(msm_obj->sgt);
kfree(msm_obj->sgt);Signature · 8 lines
regards, Stan [1] https://git.linaro.org/landing-teams/working/qualcomm/kernel.git/blob/refs/heads/release/qcomlt-4.0:/arch/arm64/mm/dma-mapping.c#l1315 [2] https://git.linaro.org/landing-teams/working/qualcomm/kernel.git/blob/refs/heads/release/qcomlt-4.0:/arch/arm64/mm/dma-mapping.c#l350
Quoted 14 lines · Stanimir Varbanov
Hi Rob,
I run into issues with msm drm driver while implementing a test application which use v4l2 vidc (venus) decoder driver to decode videos and msm drm driver to display the decoded frames. The v4l2 test application use msm drm as dmabuf exporter by
DRM_IOCTL_MODE_CREATE_DUMBandDRM_IOCTL_PRIME_HANDLE_TO_FDfrom libdrm.So far the test app is able to decode and display using dmabuf type of buffers (so, to honest it is slower than using mmap & memcpy but this is another story). The problems start when destroying drm buffers by call to
DRM_IOCTL_MODE_DESTROY_DUMBioctl and also when closing dmabuf fd. The issues I'm seeing are:
Quoted 6 lines
- the first and the major one happens in
msm_gem_free_object()where we are trying to free sg table which table is already freed bydrm_primecore in dmabuf .detach callback. In fact themsm_gem_free_objectis called by dmabuf .release callback which should be happened after .detach. I find weird to callsg_table_freein .detach callback and did not understand why it is there.
so, I think in msm_gem_prime_get_sg_table() we need to create a duplicate sgt.. since dma_buf_map_attachment() (and therefore ->gem_prime_get_sg_table()) returns ownership of the sgt. So it is not correct to be returning our own internal copy.
Quoted 7 lines
- the second one is again in
msm_gem.c::put_pagesdma_unmap_sgcall. Some background, vidc (venus) driver use dma-iommu infrastructure copped from qcom downstream kernel to abstract iommu map/unmap [1]. On the other side when drm driver is exporter it use swiotbl [2] dma map operations,dma_map_sgwill fillsg->dma_addresswith phy addresses. So when vidc calldma_map_sgthe sgdma_addressis filled with iova address then drm calldma_unmap_sgexpecting phy addresses.
hmm.. so drm_gem_map_dma_buf() should dma_map_sg() using attach->dev (which should be vidc, not drm). Maybe something unexepected was happening there since we were incorrectly returning our own sgt (which had already been dma_map_sg()'d w/ the drm device for cache maintenance?
Could you try:
-------------
(quoted diff) +0 -0
b/drivers/gpu/drm/msm/msm_gem_prime.c
index dd7a7ab..831461b 100644
drivers/gpu/drm/msm/msm_gem_prime.c +6 -2
@@ -23,8 +23,12 @@struct sg_table *msm_gem_prime_get_sg_table (struct drm_gem_object *obj) {struct msm_gem_object *msm_obj =to_msm_bo (obj); -BUG_ON (!msm_obj->sgt);/* should have already pinned! */ -return msm_obj->sgt; +int npages = obj->size >>PAGE_SHIFT ; + +if (WARN_ON (!msm_obj->pages))/* should have already pinned! */ +return NULL ; + +return drm_prime_pages_to_sg (msm_obj->pages, npages); }void *msm_gem_prime_vmap (struct drm_gem_object *obj)
-------------
Quoted 50 lines
If I didn't express myself well, see the hack bellow which I made to prevent kernel crash.
Any thoughts?
drivers/gpu/drm/drm_prime.c +2
-2@@ -155,10 +155,10 @@ static void drm_gem_map_detach(struct dma_buf *dma_buf, if (prime_attach->dir != DMA_NONE)dma_unmap_sg (attach->dev, sgt->sgl, sgt->nents, prime_attach->dir); -sg_free_table (sgt); +/* sg_free_table(sgt);*/ } -kfree (sgt); +/* kfree(sgt);*/ kfree (prime_attach); attach->priv =NULL ;}
drivers/gpu/drm/msm/msm_gem.c +2
-2@@ -116,9 +116,10 @@ static void put_pages(struct drm_gem_object *obj)/* For non-cached buffers, ensure the new pages are clean * because display controller, GPU, etc. are not coherent: */ -if (msm_obj->flags & (MSM_BO_WC |MSM_BO_UNCACHED )) +/* if (msm_obj->flags & (MSM_BO_WC|MSM_BO_UNCACHED)) dma_unmap_sg(obj->dev->dev, msm_obj->sgt->sgl, msm_obj->sgt->nents, DMA_BIDIRECTIONAL); +*/ sg_free_table (msm_obj->sgt);kfree (msm_obj->sgt); --regards,
Stan
Quoted 36 lines · Rob Clark
Quoted 14 lines · Stanimir Varbanov
Hi Rob,
I run into issues with msm drm driver while implementing a test application which use v4l2 vidc (venus) decoder driver to decode videos and msm drm driver to display the decoded frames. The v4l2 test application use msm drm as dmabuf exporter by
DRM_IOCTL_MODE_CREATE_DUMBandDRM_IOCTL_PRIME_HANDLE_TO_FDfrom libdrm.So far the test app is able to decode and display using dmabuf type of buffers (so, to honest it is slower than using mmap & memcpy but this is another story). The problems start when destroying drm buffers by call to
DRM_IOCTL_MODE_DESTROY_DUMBioctl and also when closing dmabuf fd. The issues I'm seeing are:Quoted 6 lines
- the first and the major one happens in
msm_gem_free_object()where we are trying to free sg table which table is already freed bydrm_primecore in dmabuf .detach callback. In fact themsm_gem_free_objectis called by dmabuf .release callback which should be happened after .detach. I find weird to callsg_table_freein .detach callback and did not understand why it is there.so, I think in
msm_gem_prime_get_sg_table()we need to create a duplicate sgt.. sincedma_buf_map_attachment()(and therefore ->gem_prime_get_sg_table()) returns ownership of the sgt. So it is not correct to be returning our own internal copy.Quoted 7 lines
- the second one is again in
msm_gem.c::put_pagesdma_unmap_sgcall. Some background, vidc (venus) driver use dma-iommu infrastructure copped from qcom downstream kernel to abstract iommu map/unmap [1]. On the other side when drm driver is exporter it use swiotbl [2] dma map operations,dma_map_sgwill fillsg->dma_addresswith phy addresses. So when vidc calldma_map_sgthe sgdma_addressis filled with iova address then drm calldma_unmap_sgexpecting phy addresses.
Quoted 5 lines
hmm.. so
drm_gem_map_dma_buf()shoulddma_map_sg()usingattach->dev(which should be vidc, not drm). Maybe something unexepected was happening there since we were incorrectly returning our own sgt (which had already been dma_map_sg()'d w/ the drm device for cache maintenance?
In fact this is consequence of the first issue.
Quoted 13 lines
Could you try:
-------------
drivers/gpu/drm/msm/msm_gem_prime.c +0
-0@@ -23,8 +23,12 @@struct sg_table *msm_gem_prime_get_sg_table (struct drm_gem_object *obj) {struct msm_gem_object *msm_obj =to_msm_bo (obj);
drivers/gpu/drm/msm/msm_gem_prime.c line 33
Quoted 10 lines
drivers/gpu/drm/msm/msm_gem_prime.c +6
-2-BUG_ON (!msm_obj->sgt);/* should have already pinned! */ -return msm_obj->sgt; +int npages = obj->size >>PAGE_SHIFT ; + +if (WARN_ON (!msm_obj->pages))/* should have already pinned! */ +return NULL ; + +return drm_prime_pages_to_sg (msm_obj->pages, npages); }
Brilliant diff, duplicating sgt fixed both issues!
Care to fix this in mainline?
Signature · 2 lines
regards, Stan
Quoted 71 lines · Stanimir Varbanov
Quoted 36 lines · Rob Clark
Quoted 14 lines · Stanimir Varbanov
Hi Rob,
I run into issues with msm drm driver while implementing a test application which use v4l2 vidc (venus) decoder driver to decode videos and msm drm driver to display the decoded frames. The v4l2 test application use msm drm as dmabuf exporter by
DRM_IOCTL_MODE_CREATE_DUMBandDRM_IOCTL_PRIME_HANDLE_TO_FDfrom libdrm.So far the test app is able to decode and display using dmabuf type of buffers (so, to honest it is slower than using mmap & memcpy but this is another story). The problems start when destroying drm buffers by call to
DRM_IOCTL_MODE_DESTROY_DUMBioctl and also when closing dmabuf fd. The issues I'm seeing are:Quoted 6 lines
- the first and the major one happens in
msm_gem_free_object()where we are trying to free sg table which table is already freed bydrm_primecore in dmabuf .detach callback. In fact themsm_gem_free_objectis called by dmabuf .release callback which should be happened after .detach. I find weird to callsg_table_freein .detach callback and did not understand why it is there.so, I think in
msm_gem_prime_get_sg_table()we need to create a duplicate sgt.. sincedma_buf_map_attachment()(and therefore ->gem_prime_get_sg_table()) returns ownership of the sgt. So it is not correct to be returning our own internal copy.Quoted 7 lines
- the second one is again in
msm_gem.c::put_pagesdma_unmap_sgcall. Some background, vidc (venus) driver use dma-iommu infrastructure copped from qcom downstream kernel to abstract iommu map/unmap [1]. On the other side when drm driver is exporter it use swiotbl [2] dma map operations,dma_map_sgwill fillsg->dma_addresswith phy addresses. So when vidc calldma_map_sgthe sgdma_addressis filled with iova address then drm calldma_unmap_sgexpecting phy addresses.Quoted 5 lines
hmm.. so
drm_gem_map_dma_buf()shoulddma_map_sg()usingattach->dev(which should be vidc, not drm). Maybe something unexepected was happening there since we were incorrectly returning our own sgt (which had already been dma_map_sg()'d w/ the drm device for cache maintenance?In fact this is consequence of the first issue.
Quoted 13 lines
Could you try:
-------------
drivers/gpu/drm/msm/msm_gem_prime.c +0
-0@@ -23,8 +23,12 @@struct sg_table *msm_gem_prime_get_sg_table (struct drm_gem_object *obj) {struct msm_gem_object *msm_obj =to_msm_bo (obj);drivers/gpu/drm/msm/msm_gem_prime.c line 33
Quoted 10 lines
drivers/gpu/drm/msm/msm_gem_prime.c +6
-2-BUG_ON (!msm_obj->sgt);/* should have already pinned! */ -return msm_obj->sgt; +int npages = obj->size >>PAGE_SHIFT ; + +if (WARN_ON (!msm_obj->pages))/* should have already pinned! */ +return NULL ; + +return drm_prime_pages_to_sg (msm_obj->pages, npages); }Brilliant diff, duplicating sgt fixed both issues!
Quoted 1 line
Care to fix this in mainline?
will do, just sent patch.. I'll include it for 4.2-fixes shortly
BR,
-R
Quoted 3 lines
--
regards,
Stan