From: =?UTF-8?Q?Christian_K=c3=b6nig?= Subject: Re: [PATCH v4 05/10] drm/ttm: Add vmap/vunmap to TTM and TTM GEM helpers Date: Fri, 16 Oct 2020 11:41:18 +0200 Message-ID: <64130e2a-0e45-60da-2929-6378f59bfe97@amd.com> References: <20201015123806.32416-1-tzimmermann@suse.de> <20201015123806.32416-6-tzimmermann@suse.de> <935d5771-5645-62a6-849c-31e286db1e30@amd.com> <20201015164909.GC401619@phenom.ffwll.local> <20201015195204.1745fe7f@linux-uq9g> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Return-path: In-Reply-To: <20201015195204.1745fe7f@linux-uq9g> Content-Language: en-US To: Thomas Zimmermann , Daniel Vetter Cc: luben.tuikov@amd.com, airlied@linux.ie, nouveau@lists.freedesktop.org, dri-devel@lists.freedesktop.org, chris@chris-wilson.co.uk, melissa.srw@gmail.com, ray.huang@amd.com, kraxel@redhat.com, sam@ravnborg.org, emil.velikov@collabora.com, linux-samsung-soc@vger.kernel.org, jy0922.shim@samsung.com, lima@lists.freedesktop.org, oleksandr_andrushchenko@epam.com, krzk@kernel.org, steven.price@arm.com, linux-rockchip@lists.infradead.org, kgene@kernel.org, bskeggs@redhat.com, linux+etnaviv@armlinux.org.uk, spice-devel@lists.freedesktop.org, alyssa.rosenzweig@collabora.com, etnaviv@lists.freedesktop.org, hdegoede@redhat.com, xen-devel@lists.xenproject.org, virtualization@lists.linux-foundation.org, sean@poorly.run, apaneers@amd.com, linux-arm-kernel@lists.infradead.org, linaro-mm-sig List-Id: nouveau.vger.kernel.org Am 15.10.20 um 19:52 schrieb Thomas Zimmermann: > Hi > > On Thu, 15 Oct 2020 18:49:09 +0200 Daniel Vetter wrote: > >> On Thu, Oct 15, 2020 at 04:08:13PM +0200, Christian König wrote: >>> Am 15.10.20 um 14:38 schrieb Thomas Zimmermann: >>>> The new functions ttm_bo_{vmap,vunmap}() map and unmap a TTM BO in >>>> kernel address space. The mapping's address is returned as struct >>>> dma_buf_map. Each function is a simplified version of TTM's existing >>>> kmap code. Both functions respect the memory's location ani/or >>>> writecombine flags. >>>> >>>> On top TTM's functions, GEM TTM helpers got drm_gem_ttm_{vmap,vunmap}(), >>>> two helpers that convert a GEM object into the TTM BO and forward the >>>> call to TTM's vmap/vunmap. These helpers can be dropped into the rsp >>>> GEM object callbacks. >>>> >>>> v4: >>>> * drop ttm_kmap_obj_to_dma_buf() in favor of vmap helpers >>>> (Daniel, Christian) >>> Bunch of minor comments below, but over all look very solid to me. >> Yeah I think just duplicating the ttm bo map stuff for vmap is indeed the >> cleanest. And then we can maybe push the combinatorial monster into >> vmwgfx, which I think is the only user after this series. Or perhaps a >> dedicated set of helpers to map an invidual page (again using the >> dma_buf_map stuff). > From a quick look, I'd say it should be possible to have the same interface > for kmap/kunmap as for vmap/vunmap (i.e., parameters are bo and dma-buf-map). > All mapping state can be deduced from this. And struct ttm_bo_kmap_obj can be > killed off entirely. Yes, that would be rather nice to have. Thanks, Christian. > > Best regards > Thomas > >> I'll let Christian with the details, but at a high level this is >> definitely >> >> Acked-by: Daniel Vetter >> >> Thanks a lot for doing all this. >> -Daniel >> >>>> Signed-off-by: Thomas Zimmermann >>>> --- >>>> drivers/gpu/drm/drm_gem_ttm_helper.c | 38 +++++++++++++++ >>>> drivers/gpu/drm/ttm/ttm_bo_util.c | 72 ++++++++++++++++++++++++++++ >>>> include/drm/drm_gem_ttm_helper.h | 6 +++ >>>> include/drm/ttm/ttm_bo_api.h | 28 +++++++++++ >>>> include/linux/dma-buf-map.h | 20 ++++++++ >>>> 5 files changed, 164 insertions(+) >>>> >>>> diff --git a/drivers/gpu/drm/drm_gem_ttm_helper.c >>>> b/drivers/gpu/drm/drm_gem_ttm_helper.c index 0e4fb9ba43ad..db4c14d78a30 >>>> 100644 --- a/drivers/gpu/drm/drm_gem_ttm_helper.c >>>> +++ b/drivers/gpu/drm/drm_gem_ttm_helper.c >>>> @@ -49,6 +49,44 @@ void drm_gem_ttm_print_info(struct drm_printer *p, >>>> unsigned int indent, } >>>> EXPORT_SYMBOL(drm_gem_ttm_print_info); >>>> +/** >>>> + * drm_gem_ttm_vmap() - vmap &ttm_buffer_object >>>> + * @gem: GEM object. >>>> + * @map: [out] returns the dma-buf mapping. >>>> + * >>>> + * Maps a GEM object with ttm_bo_vmap(). This function can be used as >>>> + * &drm_gem_object_funcs.vmap callback. >>>> + * >>>> + * Returns: >>>> + * 0 on success, or a negative errno code otherwise. >>>> + */ >>>> +int drm_gem_ttm_vmap(struct drm_gem_object *gem, >>>> + struct dma_buf_map *map) >>>> +{ >>>> + struct ttm_buffer_object *bo = drm_gem_ttm_of_gem(gem); >>>> + >>>> + return ttm_bo_vmap(bo, map); >>>> + >>>> +} >>>> +EXPORT_SYMBOL(drm_gem_ttm_vmap); >>>> + >>>> +/** >>>> + * drm_gem_ttm_vunmap() - vunmap &ttm_buffer_object >>>> + * @gem: GEM object. >>>> + * @map: dma-buf mapping. >>>> + * >>>> + * Unmaps a GEM object with ttm_bo_vunmap(). This function can be used >>>> as >>>> + * &drm_gem_object_funcs.vmap callback. >>>> + */ >>>> +void drm_gem_ttm_vunmap(struct drm_gem_object *gem, >>>> + struct dma_buf_map *map) >>>> +{ >>>> + struct ttm_buffer_object *bo = drm_gem_ttm_of_gem(gem); >>>> + >>>> + ttm_bo_vunmap(bo, map); >>>> +} >>>> +EXPORT_SYMBOL(drm_gem_ttm_vunmap); >>>> + >>>> /** >>>> * drm_gem_ttm_mmap() - mmap &ttm_buffer_object >>>> * @gem: GEM object. >>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c >>>> b/drivers/gpu/drm/ttm/ttm_bo_util.c index bdee4df1f3f2..80c42c774c7d >>>> 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_util.c >>>> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c >>>> @@ -32,6 +32,7 @@ >>>> #include >>>> #include >>>> #include >>>> +#include >>>> #include >>>> #include >>>> #include >>>> @@ -526,6 +527,77 @@ void ttm_bo_kunmap(struct ttm_bo_kmap_obj *map) >>>> } >>>> EXPORT_SYMBOL(ttm_bo_kunmap); >>>> +int ttm_bo_vmap(struct ttm_buffer_object *bo, struct dma_buf_map *map) >>>> +{ >>>> + struct ttm_resource *mem = &bo->mem; >>>> + int ret; >>>> + >>>> + ret = ttm_mem_io_reserve(bo->bdev, mem); >>>> + if (ret) >>>> + return ret; >>>> + >>>> + if (mem->bus.is_iomem) { >>>> + void __iomem *vaddr_iomem; >>>> + unsigned long size = bo->num_pages << PAGE_SHIFT; >>> Please use uint64_t here and make sure to cast bo->num_pages before >>> shifting. >>> >>> We have an unit tests of allocating a 8GB BO and that should work on a >>> 32bit machine as well :) >>> >>>> + >>>> + if (mem->bus.addr) >>>> + vaddr_iomem = (void *)(((u8 *)mem->bus.addr)); >>>> + else if (mem->placement & TTM_PL_FLAG_WC) >>> I've just nuked the TTM_PL_FLAG_WC flag in drm-misc-next. There is a new >>> mem->bus.caching enum as replacement. >>> >>>> + vaddr_iomem = ioremap_wc(mem->bus.offset, >>>> size); >>>> + else >>>> + vaddr_iomem = ioremap(mem->bus.offset, size); >>>> + >>>> + if (!vaddr_iomem) >>>> + return -ENOMEM; >>>> + >>>> + dma_buf_map_set_vaddr_iomem(map, vaddr_iomem); >>>> + >>>> + } else { >>>> + struct ttm_operation_ctx ctx = { >>>> + .interruptible = false, >>>> + .no_wait_gpu = false >>>> + }; >>>> + struct ttm_tt *ttm = bo->ttm; >>>> + pgprot_t prot; >>>> + void *vaddr; >>>> + >>>> + BUG_ON(!ttm); >>> I think we can drop this, populate will just crash badly anyway. >>> >>>> + >>>> + ret = ttm_tt_populate(bo->bdev, ttm, &ctx); >>>> + if (ret) >>>> + return ret; >>>> + >>>> + /* >>>> + * We need to use vmap to get the desired page >>>> protection >>>> + * or to make the buffer object look contiguous. >>>> + */ >>>> + prot = ttm_io_prot(mem->placement, PAGE_KERNEL); >>> The calling convention has changed on drm-misc-next as well, but should be >>> trivial to adapt. >>> >>> Regards, >>> Christian. >>> >>>> + vaddr = vmap(ttm->pages, bo->num_pages, 0, prot); >>>> + if (!vaddr) >>>> + return -ENOMEM; >>>> + >>>> + dma_buf_map_set_vaddr(map, vaddr); >>>> + } >>>> + >>>> + return 0; >>>> +} >>>> +EXPORT_SYMBOL(ttm_bo_vmap); >>>> + >>>> +void ttm_bo_vunmap(struct ttm_buffer_object *bo, struct dma_buf_map >>>> *map) +{ >>>> + if (dma_buf_map_is_null(map)) >>>> + return; >>>> + >>>> + if (map->is_iomem) >>>> + iounmap(map->vaddr_iomem); >>>> + else >>>> + vunmap(map->vaddr); >>>> + dma_buf_map_clear(map); >>>> + >>>> + ttm_mem_io_free(bo->bdev, &bo->mem); >>>> +} >>>> +EXPORT_SYMBOL(ttm_bo_vunmap); >>>> + >>>> static int ttm_bo_wait_free_node(struct ttm_buffer_object *bo, >>>> bool dst_use_tt) >>>> { >>>> diff --git a/include/drm/drm_gem_ttm_helper.h >>>> b/include/drm/drm_gem_ttm_helper.h index 118cef76f84f..7c6d874910b8 >>>> 100644 --- a/include/drm/drm_gem_ttm_helper.h >>>> +++ b/include/drm/drm_gem_ttm_helper.h >>>> @@ -10,11 +10,17 @@ >>>> #include >>>> #include >>>> +struct dma_buf_map; >>>> + >>>> #define drm_gem_ttm_of_gem(gem_obj) \ >>>> container_of(gem_obj, struct ttm_buffer_object, base) >>>> void drm_gem_ttm_print_info(struct drm_printer *p, unsigned int >>>> indent, const struct drm_gem_object *gem); >>>> +int drm_gem_ttm_vmap(struct drm_gem_object *gem, >>>> + struct dma_buf_map *map); >>>> +void drm_gem_ttm_vunmap(struct drm_gem_object *gem, >>>> + struct dma_buf_map *map); >>>> int drm_gem_ttm_mmap(struct drm_gem_object *gem, >>>> struct vm_area_struct *vma); >>>> diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h >>>> index 37102e45e496..2c59a785374c 100644 >>>> --- a/include/drm/ttm/ttm_bo_api.h >>>> +++ b/include/drm/ttm/ttm_bo_api.h >>>> @@ -48,6 +48,8 @@ struct ttm_bo_global; >>>> struct ttm_bo_device; >>>> +struct dma_buf_map; >>>> + >>>> struct drm_mm_node; >>>> struct ttm_placement; >>>> @@ -494,6 +496,32 @@ int ttm_bo_kmap(struct ttm_buffer_object *bo, >>>> unsigned long start_page, */ >>>> void ttm_bo_kunmap(struct ttm_bo_kmap_obj *map); >>>> +/** >>>> + * ttm_bo_vmap >>>> + * >>>> + * @bo: The buffer object. >>>> + * @map: pointer to a struct dma_buf_map representing the map. >>>> + * >>>> + * Sets up a kernel virtual mapping, using ioremap or vmap to the >>>> + * data in the buffer object. The parameter @map returns the virtual >>>> + * address as struct dma_buf_map. Unmap the buffer with >>>> ttm_bo_vunmap(). >>>> + * >>>> + * Returns >>>> + * -ENOMEM: Out of memory. >>>> + * -EINVAL: Invalid range. >>>> + */ >>>> +int ttm_bo_vmap(struct ttm_buffer_object *bo, struct dma_buf_map *map); >>>> + >>>> +/** >>>> + * ttm_bo_vunmap >>>> + * >>>> + * @bo: The buffer object. >>>> + * @map: Object describing the map to unmap. >>>> + * >>>> + * Unmaps a kernel map set up by ttm_bo_vmap(). >>>> + */ >>>> +void ttm_bo_vunmap(struct ttm_buffer_object *bo, struct dma_buf_map >>>> *map); + >>>> /** >>>> * ttm_bo_mmap_obj - mmap memory backed by a ttm buffer object. >>>> * >>>> diff --git a/include/linux/dma-buf-map.h b/include/linux/dma-buf-map.h >>>> index fd1aba545fdf..2e8bbecb5091 100644 >>>> --- a/include/linux/dma-buf-map.h >>>> +++ b/include/linux/dma-buf-map.h >>>> @@ -45,6 +45,12 @@ >>>> * >>>> * dma_buf_map_set_vaddr(&map. 0xdeadbeaf); >>>> * >>>> + * To set an address in I/O memory, use dma_buf_map_set_vaddr_iomem(). >>>> + * >>>> + * .. code-block:: c >>>> + * >>>> + * dma_buf_map_set_vaddr_iomem(&map. 0xdeadbeaf); >>>> + * >>>> * Test if a mapping is valid with either dma_buf_map_is_set() or >>>> * dma_buf_map_is_null(). >>>> * >>>> @@ -118,6 +124,20 @@ static inline void dma_buf_map_set_vaddr(struct >>>> dma_buf_map *map, void *vaddr) map->is_iomem = false; >>>> } >>>> +/** >>>> + * dma_buf_map_set_vaddr_iomem - Sets a dma-buf mapping structure to >>>> an address in I/O memory >>>> + * @map: The dma-buf mapping structure >>>> + * @vaddr_iomem: An I/O-memory address >>>> + * >>>> + * Sets the address and the I/O-memory flag. >>>> + */ >>>> +static inline void dma_buf_map_set_vaddr_iomem(struct dma_buf_map *map, >>>> + void __iomem >>>> *vaddr_iomem) +{ >>>> + map->vaddr_iomem = vaddr_iomem; >>>> + map->is_iomem = true; >>>> +} >>>> + >>>> /** >>>> * dma_buf_map_is_equal - Compares two dma-buf mapping structures for >>>> equality >>>> * @lhs: The dma-buf mapping structure > >