DRM GPUVA Manager & Nouveau VM_BIND UAPI
Subsystems (4)
v8 cover DRM GPUVA Manager & Nouveau VM_BIND UAPI
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This patch series provides a new UAPI for the Nouveau driver in order to support Vulkan features, such as sparse bindings and sparse residency.
Furthermore, with the DRM GPUVA manager it provides a new DRM core feature to keep track of GPU virtual address (VA) mappings in a more generic way.
The DRM GPUVA manager is indented to help drivers implement userspace-manageable GPU VA spaces in reference to the Vulkan API. In order to achieve this goal it serves the following purposes in this context.
1) Provide infrastructure to track GPU VA allocations and mappings, using an interval tree (RB-tree).
2) Generically connect GPU VA mappings to their backing buffers, in particular DRM GEM objects.
3) Provide a common implementation to perform more complex mapping operations on the GPU VA space. In particular splitting and merging of GPU VA mappings, e.g. for intersecting mapping requests or partial unmap requests.
The new VM_BIND Nouveau UAPI build on top of the DRM GPUVA manager, itself providing the following new interfaces.
1) Initialize a GPU VA space via the new DRM_IOCTL_NOUVEAU_VM_INIT ioctl for UMDs to specify the portion of VA space managed by the kernel and userspace, respectively.
2) Allocate and free a VA space region as well as bind and unbind memory to the GPUs VA space via the new DRM_IOCTL_NOUVEAU_VM_BIND ioctl.
3) Execute push buffers with the new DRM_IOCTL_NOUVEAU_EXEC ioctl.
Both, DRM_IOCTL_NOUVEAU_VM_BIND and DRM_IOCTL_NOUVEAU_EXEC, make use of the DRM scheduler to queue jobs and support asynchronous processing with DRM syncobjs as synchronization mechanism.
By default DRM_IOCTL_NOUVEAU_VM_BIND does synchronous processing, DRM_IOCTL_NOUVEAU_EXEC supports asynchronous processing only.
The new VM_BIND UAPI for Nouveau makes also use of drm_exec (execution context for GEM buffers) by Christian König. Since the patch implementing drm_exec was not yet merged into drm-next it is part of this series, as well as a small fix for this patch, which was found while testing this series.
This patch series is also available at [1].
There is a Mesa NVK merge request by Dave Airlie [2] implementing the corresponding userspace parts for this series.
The Vulkan CTS test suite passes the sparse binding and sparse residency test cases for the new UAPI together with Dave's Mesa work.
There are also some test cases in the igt-gpu-tools project [3] for the new UAPI and hence the DRM GPU VA manager. However, most of them are testing the DRM GPU VA manager's logic through Nouveau's new UAPI and should be considered just as helper for implementation.
However, I absolutely intend to change those test cases to proper kunit test cases for the DRM GPUVA manager, once and if we agree on it's usefulness and design.
[1] https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next /
https://gitlab.freedesktop.org/nouvelles/kernel/-/merge_requests/1
[2] https://gitlab.freedesktop.org/nouveau/mesa/-/merge_requests/150/
[3] https://gitlab.freedesktop.org/dakr/igt-gpu-tools/-/tree/wip_nouveau_vm_bind
Changes in V2:
==============
Nouveau:
- Reworked the Nouveau VM_BIND UAPI to avoid memory allocations in fence signalling critical sections. Updates to the VA space are split up in three separate stages, where only the 2. stage executes in a fence signalling critical section:
1. update the VA space, allocate new structures and page tables
2. (un-)map the requested memory bindings
3. free structures and page tables
- Separated generic job scheduler code from specific job implementations.
- Separated the EXEC and VM_BIND implementation of the UAPI.
- Reworked the locking parts of the nvkm/vmm RAW interface, such that (un-)map operations can be executed in fence signalling critical sections.
GPUVA Manager:
- made drm_gpuva_regions optional for users of the GPUVA manager
- allow NULL GEMs for drm_gpuva entries
- swichted from drm_mm to maple_tree for track drm_gpuva / drm_gpuva_region entries
- provide callbacks for users to allocate custom drm_gpuva_op structures to allow inheritance
- added user bits to drm_gpuva_flags
- added a prefetch operation type in order to support generating prefetch operations in the same way other operations generated
- hand the responsibility for mutual exclusion for a GEM's drm_gpuva list to the user; simplified corresponding (un-)link functions
Maple Tree:
- I added two maple tree patches to the series, one to support custom tree walk macros and one to hand the locking responsibility to the user of the GPUVA manager without pre-defined lockdep checks.
Changes in V3:
==============
Nouveau:
- Reworked the Nouveau VM_BIND UAPI to do the job cleanup (including page table cleanup) within a workqueue rather than the job_free() callback of the scheduler itself. A job_free() callback can stall the execution (run() callback) of the next job in the queue. Since the page table cleanup requires to take the same locks as need to be taken for page table allocation, doing it directly in the job_free() callback would still violate the fence signalling critical path.
- Separated Nouveau fence allocation and emit, such that we do not violate the fence signalling critical path in EXEC jobs.
- Implement "regions" (for handling sparse mappings through PDEs and dual page tables) within Nouveau.
- Drop the requirement for every mapping to be contained within a region.
- Add necassary synchronization of VM_BIND job operation sequences in order to work around limitations in page table handling. This will be addressed in a future re-work of Nouveau's page table handling.
- Fixed a couple of race conditions found through more testing. Thanks to Dave for consitently trying to break it. :-)
GPUVA Manager:
- Implement pre-allocation capabilities for tree modifications within fence signalling critical sections.
- Implement accessors to to apply tree modification while walking the GPUVA tree in order to actually support processing of drm_gpuva_ops through callbacks in fence signalling critical sections rather than through pre-allocated operation lists.
- Remove merging of GPUVAs; the kernel has limited to none knowlege about the semantics of mapping sequences. Hence, merging is purely speculative. It seems that gaining a significant (or at least a measurable) performance increase through merging is way more likely to happen when userspace is responsible for merging mappings up to the next larger page size if possible.
- Since merging was removed, regions pretty much loose their right to exist. They might still be useful for handling dual page tables or similar mechanisms, but since Nouveau seems to be the only driver having a need for this for now, regions were removed from the GPUVA manager.
- Fixed a couple of maple_tree related issues; thanks to Liam for helping me out.
Changes in V4:
==============
Nouveau:
- Refactored how specific VM_BIND and EXEC jobs are created and how their arguments are passed to the generic job implementation.
- Fixed a UAF race condition where bind job ops could have been freed already while still waiting for a job cleanup to finish. This is due to in certain cases we need to wait for mappings actually being unmapped before creating sparse regions in the same area.
- Re-based the code onto drm_exec v4 patch.
GPUVA Manager:
- Fixed a maple tree related bug when pre-allocating MA states. (Boris Brezillion)
- Made struct drm_gpuva_fn_ops a const object in all occurrences. (Boris Brezillion)
Changes in V5:
==============
Nouveau:
- Link and unlink GPUVAs outside the fence signalling critical path in nouveau_uvmm_bind_job_submit() holding the dma-resv lock. Mutual exclusion of BO evicts causing mapping invalidation and regular mapping operations is ensured with dma-fences.
GPUVA Manager:
- Removed the separate GEMs GPUVA list lock. Link and unlink as well as iterating the GEM's GPUVA list should be protected with the GEM's dma-resv lock instead.
- Renamed DRM_GPUVA_EVICTED flag to DRM_GPUVA_INVALIDATED. Mappings do not get eviced, they might get invalidated due to eviction.
- Maple tree uses the 'unsinged long' type for node entries. While this works for GPU VA spaces larger than 32-bit on 64-bit kernel, the GPU VA space is limited to 32-bit on 32-bit kernels as well.
As long as we do not have a 64-bit capable maple tree for 32-bit kernels, the GPU VA manager contains checks to throw warnings when GPU VA entries exceed the maple tree's storage capabilities.
- Extended the Documentation and added example code as requested by Donald Robson.
Changes in V6
=============
Nouveau:
- Re-based the code onto drm_exec v5 patch.
GPUVA Manager:
- Switch from maple tree to RB-tree.
It turned out that mas_preallocate() requires the maple tree not to change in between pre-allocating nodes with mas_preallocate() and inserting an entry with the help of the pre-allocated memory (mas_insert_prealloc()).
However, considering that drivers typically implement interfaces where jobs to create GPU mappings can be submitted by userspace, are queued up by the kernel and are processed asynchronously in dma-fence signalling critical paths, this is a major issue. In the ioctl() used to submit a job we'd need to pre-allocated memory with mas_preallocate(), however, previously queued up jobs could concurrently alter the maple tree resulting in potentially insufficient pre-allocated memory for the currently submitted job on execution time.
There is a detailed and still ongoing discussion about this topic one the -mm list [1]. So far the only solution seems to be to use GFP_ATOMIC and allocate memory directly in the fence signalling critical path, where we need it. However, I think that is not what we want to rely on.
I think we should definitely continue in trying to find a solution on how to fit in the maple tree (or how to make the maple tree fit in). However, for now it seems to be more expedient to move on using a RB-tree.
[1] https://lore.kernel.org/lkml/[email protected]/
- Provide a flag to let driver optionally provide their own lock to lock linking and unlinking of GPUVAs to GEM objects. The DRM GPUVA manager still does not take the locks itself, but rather contains lockdep checks on either the GEMs dma-resv lock (default) or, if DRM_GPUVA_MANAGER_LOCK_EXTERN is set, the driver provided lock. (Boris Brezillon)
Changes in V7
=============
Nouveau:
- Rebase to drm_exec v7.
- Move drm_gem_gpuva_init() before ttm_bo_init_validate(), but after initialization of the corresponding dma-resv.
GPUVA Manager:
- Fix drm_gpuva_find_first() range parameter in drm_gpuva_for_each_va* macros. (Boris)
- Simplify drm_gpuva_for_each_va* macros using a __drm_gpuva_next() helper. (Boris)
- Move lockdep checks for an optional external GEM gpuva list lock out of the GPUVA Manager to drm_gem.h. (Boris)
- Fix code style issues pointed out by Thomas.
- Switch to EXPORT_SYMBOL_GPL(). (Christoph)
Changes in V8
=============
Nouveau:
- n/a
GPUVA Manager:
- Fix documentation about locking the GEMs GPUVA list. (Donald)
- Fix a few minor checkpatch warnings.
Danilo Krummrich (12):
drm: manager to keep track of GPUs VA mappings
drm: debugfs: provide infrastructure to dump a DRM GPU VA space drm/nouveau: new VM_BIND uapi interfaces
drm/nouveau: get vmm via nouveau_cli_vmm() drm/nouveau: bo: initialize GEM GPU VA interface drm/nouveau: move usercopy helpers to nouveau_drv.h drm/nouveau: fence: separate fence alloc and emit drm/nouveau: fence: fail to emit when fence context is killed drm/nouveau: chan: provide nouveau_channel_kill() drm/nouveau: nvkm/vmm: implement raw ops to manage uvmm drm/nouveau: implement new VM_BIND uAPI
drm/nouveau: debugfs: implement DRM GPU VA debugfs
Documentation/gpu/driver-uapi.rst | 11 + Documentation/gpu/drm-mm.rst | 36 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/drm_debugfs.c | 40 + drivers/gpu/drm/drm_gem.c | 3 + drivers/gpu/drm/drm_gpuva_mgr.c | 1728 +++++++++++++++ drivers/gpu/drm/nouveau/Kbuild | 3 + drivers/gpu/drm/nouveau/Kconfig | 2 + drivers/gpu/drm/nouveau/dispnv04/crtc.c | 9 +- drivers/gpu/drm/nouveau/include/nvif/if000c.h | 26 +- drivers/gpu/drm/nouveau/include/nvif/vmm.h | 19 +- .../gpu/drm/nouveau/include/nvkm/subdev/mmu.h | 20 +- drivers/gpu/drm/nouveau/nouveau_abi16.c | 24 + drivers/gpu/drm/nouveau/nouveau_abi16.h | 1 + drivers/gpu/drm/nouveau/nouveau_bo.c | 209 +- drivers/gpu/drm/nouveau/nouveau_bo.h | 2 +- drivers/gpu/drm/nouveau/nouveau_chan.c | 22 +- drivers/gpu/drm/nouveau/nouveau_chan.h | 1 + drivers/gpu/drm/nouveau/nouveau_debugfs.c | 39 + drivers/gpu/drm/nouveau/nouveau_dmem.c | 9 +- drivers/gpu/drm/nouveau/nouveau_drm.c | 27 +- drivers/gpu/drm/nouveau/nouveau_drv.h | 94 +- drivers/gpu/drm/nouveau/nouveau_exec.c | 414 ++++ drivers/gpu/drm/nouveau/nouveau_exec.h | 54 + drivers/gpu/drm/nouveau/nouveau_fence.c | 23 +- drivers/gpu/drm/nouveau/nouveau_fence.h | 5 +- drivers/gpu/drm/nouveau/nouveau_gem.c | 62 +- drivers/gpu/drm/nouveau/nouveau_mem.h | 5 + drivers/gpu/drm/nouveau/nouveau_prime.c | 2 +- drivers/gpu/drm/nouveau/nouveau_sched.c | 462 ++++ drivers/gpu/drm/nouveau/nouveau_sched.h | 123 + drivers/gpu/drm/nouveau/nouveau_svm.c | 2 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 1970 +++++++++++++++++ drivers/gpu/drm/nouveau/nouveau_uvmm.h | 107 + drivers/gpu/drm/nouveau/nouveau_vmm.c | 4 +- drivers/gpu/drm/nouveau/nvif/vmm.c | 100 +- .../gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.c | 213 +- drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.c | 197 +- drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.h | 25 + .../drm/nouveau/nvkm/subdev/mmu/vmmgf100.c | 16 +- .../drm/nouveau/nvkm/subdev/mmu/vmmgp100.c | 16 +- .../gpu/drm/nouveau/nvkm/subdev/mmu/vmmnv50.c | 27 +- include/drm/drm_debugfs.h | 25 + include/drm/drm_drv.h | 6 + include/drm/drm_gem.h | 79 + include/drm/drm_gpuva_mgr.h | 706 ++++++ include/uapi/drm/nouveau_drm.h | 209 ++ 47 files changed, 6938 insertions(+), 240 deletions(-) create mode 100644 drivers/gpu/drm/drm_gpuva_mgr.c create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.c create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.h create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.c create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.h create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.c create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.h create mode 100644 include/drm/drm_gpuva_mgr.h
base-commit: 41639b3a8b0f1f194dfe0577d99db70613f78626
Signature · 1 line
2.41.0
PATCH v8 1/12 drm: manager to keep track of GPUs VA mappings
Add infrastructure to keep track of GPU virtual address (VA) mappings with a decicated VA space manager implementation.
New UAPIs, motivated by Vulkan sparse memory bindings graphics drivers start implementing, allow userspace applications to request multiple and arbitrary GPU VA mappings of buffer objects. The DRM GPU VA manager is intended to serve the following purposes in this context.
1) Provide infrastructure to track GPU VA allocations and mappings, making using an interval tree (RB-tree).
2) Generically connect GPU VA mappings to their backing buffers, in particular DRM GEM objects.
3) Provide a common implementation to perform more complex mapping operations on the GPU VA space. In particular splitting and merging of GPU VA mappings, e.g. for intersecting mapping requests or partial unmap requests.
--- Documentation/gpu/drm-mm.rst | 36 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/drm_gem.c | 3 + drivers/gpu/drm/drm_gpuva_mgr.c | 1728 +++++++++++++++++++++++++++++++ include/drm/drm_drv.h | 6 + include/drm/drm_gem.h | 79 ++ include/drm/drm_gpuva_mgr.h | 706 +++++++++++++ 7 files changed, 2559 insertions(+) create mode 100644 drivers/gpu/drm/drm_gpuva_mgr.c create mode 100644 include/drm/drm_gpuva_mgr.h
Documentation/gpu/drm-mm.rst +36 -0
@@ -466,6 +466,42 @@ DRM MM Range Allocator Function References .. kernel-doc:: drivers/gpu/drm/drm_mm.c :export: +DRM GPU VA Manager +================== + +Overview +-------- + +.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c + :doc: Overview + +Split and Merge +--------------- + +.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c + :doc: Split and Merge + +Locking +------- + +.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c + :doc: Locking + +Examples +-------- + +.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c + :doc: Examples + +DRM GPU VA Manager Function References +-------------------------------------- + +.. kernel-doc:: include/drm/drm_gpuva_mgr.h + :internal: + +.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c + :export: + DRM Buddy Allocator ===================
drivers/gpu/drm/Makefile +1 -0
@@ -45,6 +45,7 @@ drm-y := \ drm_vblank.o \ drm_vblank_work.o \ drm_vma_manager.o \ + drm_gpuva_mgr.o \ drm_writeback.o drm-$(CONFIG_DRM_LEGACY) += \ drm_agpsupport.o \
drivers/gpu/drm/drm_gem.c +3 -0
@@ -164,6 +164,9 @@ void drm_gem_private_object_init(struct drm_device *dev,if (!obj->resv) obj->resv = &obj->_resv; +if (drm_core_check_feature (dev,DRIVER_GEM_GPUVA )) +drm_gem_gpuva_init (obj); +drm_vma_node_reset (&obj->vma_node);INIT_LIST_HEAD (&obj->lru_node); }
drivers/gpu/drm/drm_gpuva_mgr.c new file +1728 -0
1,728 lines not on this page · raw message
include/drm/drm_drv.h +6 -0
@@ -104,6 +104,12 @@ enum drm_driver_feature {* acceleration should be handled by two drivers that are connected using auxiliary bus. */ DRIVER_COMPUTE_ACCEL =BIT (7 ), +/** +* @DRIVER_GEM_GPUVA: +* +* Driver supports user defined GPU VA bindings for GEM objects. +*/ + DRIVER_GEM_GPUVA = BIT(8),/* IMPORTANT: Below are all the legacy flags, add new ones above. */
include/drm/drm_gem.h +79 -0
95 lines not on this page · raw message
include/drm/drm_gpuva_mgr.h new file +706 -0
706 lines not on this page · raw message
Signature · 1 line
2.41.0
Re: [PATCH drm-misc-next v8 01/12] drm: manager to keep track of GPUs VA mappings
Quoted 26 lines · Danilo Krummrich
Add infrastructure to keep track of GPU virtual address (VA) mappings with a decicated VA space manager implementation.
New UAPIs, motivated by Vulkan sparse memory bindings graphics drivers start implementing, allow userspace applications to request multiple and arbitrary GPU VA mappings of buffer objects. The DRM GPU VA manager is intended to serve the following purposes in this context.
1) Provide infrastructure to track GPU VA allocations and mappings, making using an interval tree (RB-tree).
2) Generically connect GPU VA mappings to their backing buffers, in particular DRM GEM objects.
3) Provide a common implementation to perform more complex mapping operations on the GPU VA space. In particular splitting and merging of GPU VA mappings, e.g. for intersecting mapping requests or partial unmap requests.
Acked-by Thomas Hellström <[email protected]>Acked-by Matthew Brost <[email protected]>Reviewed-by Boris Brezillon <[email protected]>Tested-by Matthew Brost <[email protected]>Tested-by Donald Robson <[email protected]>Suggested-by Dave Airlie <[email protected]>Signed-off-by Danilo Krummrich <[email protected]>
[...]
Quoted 5 lines
drivers/gpu/drm/drm_gpuva_mgr.c new file +0
-0
[...]
drivers/gpu/drm/drm_gpuva_mgr.c
Quoted 8 lines
drivers/gpu/drm/drm_gpuva_mgr.c new file +8
-0+static bool +drm_gpuva_check_overflow (u64 addr,u64 range) +{ +u64 end; + +return WARN (check_add_overflow (addr, range, &end), +"GPUVA address limited to %lu bytes.\n" ,sizeof (end)); +}
This produces a warning on 32 bit systems as sizeof() isn't necessarily an unsigned long. The fix below silences the warning.
Thanks,
Steve
---8<-----
From 9c7356580362b6ac4673724f18ea6e8453b52913 Mon Sep 17 00:00:00 2001
From: Steven Price <[email protected]>
Date: Thu, 20 Jul 2023 10:58:09 +0100
Subject: [PATCH] drm: manager: Fix printk format for size_t
sizeof() returns a size_t which may be different to an unsigned long. Use the correct format specifier of '%zu' to prevent compiler warnings.
--- drivers/gpu/drm/drm_gpuva_mgr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/drm_gpuva_mgr.c +1 -1
@@ -619,7 +619,7 @@ drm_gpuva_check_overflow(u64 addr, u64 range)u64 end;return WARN (check_add_overflow (addr, range, &end), -"GPUVA address limited to %lu bytes.\n" ,sizeof (end)); +"GPUVA address limited to %zu bytes.\n" ,sizeof (end)); } static bool
Signature · 1 line
2.39.2
Re: [PATCH drm-misc-next v8 01/12] drm: manager to keep track of GPUs VA mappings
Quoted 38 lines · Steven Price
Quoted 26 lines · Danilo Krummrich
Add infrastructure to keep track of GPU virtual address (VA) mappings with a decicated VA space manager implementation.
New UAPIs, motivated by Vulkan sparse memory bindings graphics drivers start implementing, allow userspace applications to request multiple and arbitrary GPU VA mappings of buffer objects. The DRM GPU VA manager is intended to serve the following purposes in this context.
1) Provide infrastructure to track GPU VA allocations and mappings, making using an interval tree (RB-tree).
2) Generically connect GPU VA mappings to their backing buffers, in particular DRM GEM objects.
3) Provide a common implementation to perform more complex mapping operations on the GPU VA space. In particular splitting and merging of GPU VA mappings, e.g. for intersecting mapping requests or partial unmap requests.
Acked-by Thomas Hellström <[email protected]>Acked-by Matthew Brost <[email protected]>Reviewed-by Boris Brezillon <[email protected]>Tested-by Matthew Brost <[email protected]>Tested-by Donald Robson <[email protected]>Suggested-by Dave Airlie <[email protected]>Signed-off-by Danilo Krummrich <[email protected]>[...]
Quoted 5 lines
drivers/gpu/drm/drm_gpuva_mgr.c new file +0
-0[...]
Quoted 11 lines
drivers/gpu/drm/drm_gpuva_mgr.c
Quoted 8 lines
drivers/gpu/drm/drm_gpuva_mgr.c new file +8
-0+static bool +drm_gpuva_check_overflow (u64 addr,u64 range) +{ +u64 end; + +return WARN (check_add_overflow (addr, range, &end), +"GPUVA address limited to %lu bytes.\n" ,sizeof (end)); +}This produces a warning on 32 bit systems as
sizeof()isn't necessarily an unsigned long. The fix below silences the warning.
Thank you for fixing this up! Applied both of your patches to drm-misc-next.
- Danilo
Quoted 33 lines
Thanks,
Steve
---8<-----
From9c7356580362b6ac4673724f18ea6e8453b52913Mon Sep 17 00:00:00 2001
From: Steven Price <[email protected]>
Date: Thu, 20 Jul 2023 10:58:09 +0100
Subject: [PATCH] drm: manager: Fix printk format forsize_t
sizeof()returns asize_twhich may be different to an unsigned long. Use the correct format specifier of '%zu' to prevent compiler warnings.Fixes e6303f323b1a ("drm: manager to keep track of GPUs VA mappings")Signed-off-by Steven Price <[email protected]>--- drivers/gpu/drm/drm_gpuva_mgr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/drm_gpuva_mgr.c +1
-1@@ -619,7 +619,7 @@ drm_gpuva_check_overflow(u64 addr, u64 range)u64 end;return WARN (check_add_overflow (addr, range, &end), -"GPUVA address limited to %lu bytes.\n" ,sizeof (end)); +"GPUVA address limited to %zu bytes.\n" ,sizeof (end)); } static bool
Maxime Ripard
maintainerRe: [PATCH drm-misc-next v8 01/12] drm: manager to keep track of GPUs VA mappings
In reply to Danilo KrummrichHi Danilo,
Quoted 19 lines · Danilo Krummrich
Add infrastructure to keep track of GPU virtual address (VA) mappings with a decicated VA space manager implementation.
New UAPIs, motivated by Vulkan sparse memory bindings graphics drivers start implementing, allow userspace applications to request multiple and arbitrary GPU VA mappings of buffer objects. The DRM GPU VA manager is intended to serve the following purposes in this context.
1) Provide infrastructure to track GPU VA allocations and mappings, making using an interval tree (RB-tree).
2) Generically connect GPU VA mappings to their backing buffers, in particular DRM GEM objects.
3) Provide a common implementation to perform more complex mapping operations on the GPU VA space. In particular splitting and merging of GPU VA mappings, e.g. for intersecting mapping requests or partial unmap requests.
Quoted 7 lines
Acked-by Thomas Hellström <[email protected]>Acked-by Matthew Brost <[email protected]>Reviewed-by Boris Brezillon <[email protected]>Tested-by Matthew Brost <[email protected]>Tested-by Donald Robson <[email protected]>Suggested-by Dave Airlie <[email protected]>Signed-off-by Danilo Krummrich <[email protected]>
For some reason this breaks the drm_exec kunit patches:
$ ./tools/testing/kunit/kunit.py run --kunitconfig=drivers/gpu/drm/tests --cross_compile aarch64-linux-gnu- --arch arm64 --raw_output drm_exec.test_lock
[13:31:14] Configuring KUnit Kernel ...
[13:31:14] Building KUnit Kernel ...
Populating config with:
$ make ARCH=arm64 O=.kunit olddefconfig CROSS_COMPILE=aarch64-linux-gnu-
Building with:
$ make ARCH=arm64 O=.kunit --jobs=32 CROSS_COMPILE=aarch64-linux-gnu-
[13:31:16] Starting KUnit Kernel (1/1)...
Running tests with:
$ qemu-system-aarch64 -nodefaults -m 1024 -kernel .kunit/arch/arm64/boot/Image.gz -append 'kunit.filter_glob=drm_exec.test_lock kunit.enable=1 console=ttyAMA0 kunit_shutdown=reboot' -no-reboot -nographic -serial stdio -machine virt -cpu cortex-a57
KTAP version 1
1..1
KTAP version 1
# Subtest: drm_exec
1..1
Unable to handle kernel NULL pointer dereference at virtual address 00000000000000b0
Mem abort info:
ESR = 0x0000000096000005
EC = 0x25: DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
FSC = 0x05: level 1 translation fault
Data abort info:
ISV = 0, ISS = 0x00000005
CM = 0, WnR = 0
[00000000000000b0] user address but active_mm is swapper
Internal error: Oops: 0000000096000005 [#1] SMP
CPU: 0 PID: 21 Comm: kunit_try_catch Tainted: G N 6.4.0-rc7-02032-ge6303f323b1a #17
Hardware name: linux,dummy-virt (DT)
pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : drm_gem_private_object_init+0x54/0xa8
lr : drm_gem_private_object_init+0x44/0xa8
sp : ffffffc008763c10
x29: ffffffc008763c10 x28: 0000000000000000 x27: 0000000000000000
x26: 0000000000000000 x25: 0000000000000000 x24: ffffffc00800bb90
x23: ffffffc00820d3e4 x22: ffffff8000fb8880 x21: ffffffc008763db8
x20: ffffffc0086171d8 x19: ffffffc008763cb8 x18: ffffffffffffffff
x17: ffffffc0085b0da8 x16: 000000005a832b7d x15: 0000000000000000
x14: 0000000000000001 x13: 0000000000000000 x12: ffffff8000cbe380
x11: ffffff8000ca0000 x10: 00000000000007b0 x9 : 0000000000000000
x8 : ffffffc008763cb8 x7 : 0000000000000000 x6 : 000000000000003f
x5 : 0000000000000040 x4 : 0000000000000008 x3 : 0000000000000030
x2 : ffffffc0085b9118 x1 : 0000000000000000 x0 : 0000000000000000
Call trace:
drm_gem_private_object_init+0x54/0xa8
test_lock+0x58/0xf4
kunit_try_run_case+0x48/0xa8
kunit_generic_run_threadfn_adapter+0x20/0x2c
kthread+0xd4/0xd8
ret_from_fork+0x10/0x20
Code: f9407e60 b40002a0 f9401a80 b9406a81 (b940b000)
---[ end trace 0000000000000000 ]---
^CERROR:root:Build interruption occurred. Cleaning console.
qemu-system-aarch64: terminating on signal 2
[13:31:17] Elapsed time: 3.396s total, 0.001s configuring, 1.978s building, 1.417s runningMaxime
Re: [PATCH drm-misc-next v8 01/12] drm: manager to keep track of GPUs VA mappings
Quoted 30 lines · On Fri, 28 Jul 2023 13:31:36 +0200 Maxime Ripard <[email protected]> wrote:
Hi Danilo,
Quoted 19 lines · Danilo Krummrich
Add infrastructure to keep track of GPU virtual address (VA) mappings with a decicated VA space manager implementation.
New UAPIs, motivated by Vulkan sparse memory bindings graphics drivers start implementing, allow userspace applications to request multiple and arbitrary GPU VA mappings of buffer objects. The DRM GPU VA manager is intended to serve the following purposes in this context.
1) Provide infrastructure to track GPU VA allocations and mappings, making using an interval tree (RB-tree).
2) Generically connect GPU VA mappings to their backing buffers, in particular DRM GEM objects.
3) Provide a common implementation to perform more complex mapping operations on the GPU VA space. In particular splitting and merging of GPU VA mappings, e.g. for intersecting mapping requests or partial unmap requests.
Quoted 7 lines
Acked-by Thomas Hellström <[email protected]>Acked-by Matthew Brost <[email protected]>Reviewed-by Boris Brezillon <[email protected]>Tested-by Matthew Brost <[email protected]>Tested-by Donald Robson <[email protected]>Suggested-by Dave Airlie <[email protected]>Signed-off-by Danilo Krummrich <[email protected]>
Quoted 1 line
For some reason this breaks the
drm_execkunit patches:
Fix available here [1].
Maxime Ripard
maintainerRe: [PATCH drm-misc-next v8 01/12] drm: manager to keep track of GPUs VA mappings
Hi Boris,
Quoted 33 lines · Boris Brezillon
Quoted 30 lines · On Fri, 28 Jul 2023 13:31:36 +0200 Maxime Ripard <[email protected]> wrote:
Hi Danilo,
Quoted 26 lines · Danilo Krummrich
Add infrastructure to keep track of GPU virtual address (VA) mappings with a decicated VA space manager implementation.
New UAPIs, motivated by Vulkan sparse memory bindings graphics drivers start implementing, allow userspace applications to request multiple and arbitrary GPU VA mappings of buffer objects. The DRM GPU VA manager is intended to serve the following purposes in this context.
1) Provide infrastructure to track GPU VA allocations and mappings, making using an interval tree (RB-tree).
2) Generically connect GPU VA mappings to their backing buffers, in particular DRM GEM objects.
3) Provide a common implementation to perform more complex mapping operations on the GPU VA space. In particular splitting and merging of GPU VA mappings, e.g. for intersecting mapping requests or partial unmap requests.
Acked-by Thomas Hellström <[email protected]>Acked-by Matthew Brost <[email protected]>Reviewed-by Boris Brezillon <[email protected]>Tested-by Matthew Brost <[email protected]>Tested-by Donald Robson <[email protected]>Suggested-by Dave Airlie <[email protected]>Signed-off-by Danilo Krummrich <[email protected]>
Quoted 5 lines
Quoted 1 line
For some reason this breaks the
drm_execkunit patches:Fix available here [1].
Thanks for pointing it out :)
Maxime
PATCH v8 2/12 drm: debugfs: provide infrastructure to dump a DRM GPU VA space
In reply to Danilo KrummrichThis commit adds a function to dump a DRM GPU VA space and a macro for drivers to register the struct drm_info_list 'gpuvas' entry.
Most likely, most drivers might maintain one DRM GPU VA space per struct drm_file, but there might also be drivers not having a fixed relation between DRM GPU VA spaces and a DRM core infrastructure, hence we need the indirection via the driver iterating it's maintained DRM GPU VA spaces.
--- drivers/gpu/drm/drm_debugfs.c | 40 +++++++++++++++++++++++++++++++++++ include/drm/drm_debugfs.h | 25 ++++++++++++++++++++++ 2 files changed, 65 insertions(+)
drivers/gpu/drm/drm_debugfs.c +40 -0
@@ -39,6 +39,7 @@#include <drm/drm_file.h> #include <drm/drm_gem.h> #include <drm/drm_managed.h> +#include <drm/drm_gpuva_mgr.h> #include "drm_crtc_internal.h" #include "drm_internal.h"
@@ -175,6 +176,45 @@ static const struct file_operations drm_debugfs_fops = { .release = single_release, }; +/** +* drm_debugfs_gpuva_info - dump the given DRM GPU VA space +* @m: pointer to the &seq_file to write +* @mgr: the &drm_gpuva_manager representing the GPU VA space +* +* Dumps the GPU VA mappings of a given DRM GPU VA manager. +* +* For each DRM GPU VA space drivers should call this function from their +* &drm_info_list's show callback. +* +* Returns: 0 on success, -ENODEV if the &mgr is not initialized +*/ +int drm_debugfs_gpuva_info (struct seq_file *m, +struct drm_gpuva_manager *mgr) +{ +struct drm_gpuva *va, *kva = &mgr->kernel_alloc_node; + +if (!mgr->name) +return -ENODEV ; + +seq_printf (m,"DRM GPU VA space (%s) [0x%016llx;0x%016llx]\n" , + mgr->name, mgr->mm_start, mgr->mm_start + mgr->mm_range); +seq_printf (m,"Kernel reserved node [0x%016llx;0x%016llx]\n" , + kva->va.addr, kva->va.addr + kva->va.range); +seq_puts (m,"\n" ); +seq_puts (m," VAs | start | range | end | object | object offset\n" ); +seq_puts (m,"-------------------------------------------------------------------------------------------------------------\n" ); +drm_gpuva_for_each_va (va, mgr) { +if (unlikely (va == kva)) +continue ; + +seq_printf (m," | 0x%016llx | 0x%016llx | 0x%016llx | 0x%016llx | 0x%016llx\n" , + va->va.addr, va->va.range, va->va.addr + va->va.range, + (u64 )va->gem.obj, va->gem.offset); + } + +return 0 ; +} +EXPORT_SYMBOL (drm_debugfs_gpuva_info);/** * drm_debugfs_create_files - Initialize a given set of debugfs files for DRM
include/drm/drm_debugfs.h +25 -0
@@ -34,6 +34,22 @@#include <linux/types.h> #include <linux/seq_file.h> + +#include <drm/drm_gpuva_mgr.h> + +/** +* DRM_DEBUGFS_GPUVA_INFO - &drm_info_list entry to dump a GPU VA space +* @show: the &drm_info_list's show callback +* @data: driver private data +* +* Drivers should use this macro to define a &drm_info_list entry to provide a +* debugfs file for dumping the GPU VA space regions and mappings. +* +* For each DRM GPU VA space drivers should call drm_debugfs_gpuva_info() from +* their @show callback. +*/ +#define DRM_DEBUGFS_GPUVA_INFO (show, data) {"gpuvas", show, DRIVER_GEM_GPUVA, data} +/** * struct drm_info_list - debugfs info list entry *
@@ -134,6 +150,9 @@ void drm_debugfs_add_file(struct drm_device *dev, const char *name,void drm_debugfs_add_files (struct drm_device *dev,const struct drm_debugfs_info *files,int count); + +int drm_debugfs_gpuva_info (struct seq_file *m, +struct drm_gpuva_manager *mgr);#else static inline void drm_debugfs_create_files(const struct drm_info_list *files, int count, struct dentry *root,
@@ -155,6 +174,12 @@ static inline void drm_debugfs_add_files(struct drm_device *dev, const struct drm_debugfs_info *files, int count) {} + +static inline int drm_debugfs_gpuva_info (struct seq_file *m, +struct drm_gpuva_manager *mgr) +{ +return 0 ; +}#endif #endif /* _DRM_DEBUGFS_H_ */
Signature · 1 line
2.41.0
Re: [PATCH drm-misc-next v8 02/12] drm: debugfs: provide infrastructure to dump a DRM GPU VA space
Quoted 55 lines · Danilo Krummrich
This commit adds a function to dump a DRM GPU VA space and a macro for drivers to register the struct
drm_info_list'gpuvas' entry.Most likely, most drivers might maintain one DRM GPU VA space per struct
drm_file, but there might also be drivers not having a fixed relation between DRM GPU VA spaces and a DRM core infrastructure, hence we need the indirection via the driver iterating it's maintained DRM GPU VA spaces.Reviewed-by Boris Brezillon <[email protected]>Signed-off-by Danilo Krummrich <[email protected]>--- drivers/gpu/drm/drm_debugfs.c | 40 +++++++++++++++++++++++++++++++++++ include/drm/drm_debugfs.h | 25 ++++++++++++++++++++++ 2 files changed, 65 insertions(+)drivers/gpu/drm/drm_debugfs.c +25
-0@@ -39,6 +39,7 @@#include <drm/drm_file.h> #include <drm/drm_gem.h> #include <drm/drm_managed.h> +#include <drm/drm_gpuva_mgr.h> #include "drm_crtc_internal.h" #include "drm_internal.h" @@ -175,6 +176,45 @@ static const struct file_operations drm_debugfs_fops = { .release = single_release, }; +/** +* drm_debugfs_gpuva_info - dump the given DRM GPU VA space +* @m: pointer to the &seq_file to write +* @mgr: the &drm_gpuva_manager representing the GPU VA space +* +* Dumps the GPU VA mappings of a given DRM GPU VA manager. +* +* For each DRM GPU VA space drivers should call this function from their +* &drm_info_list's show callback. +* +* Returns: 0 on success, -ENODEV if the &mgr is not initialized +*/ +int drm_debugfs_gpuva_info (struct seq_file *m, +struct drm_gpuva_manager *mgr) +{ +struct drm_gpuva *va, *kva = &mgr->kernel_alloc_node; + +if (!mgr->name) +return -ENODEV ; + +seq_printf (m,"DRM GPU VA space (%s) [0x%016llx;0x%016llx]\n" , + mgr->name, mgr->mm_start, mgr->mm_start + mgr->mm_range); +seq_printf (m,"Kernel reserved node [0x%016llx;0x%016llx]\n" , + kva->va.addr, kva->va.addr + kva->va.range);
drivers/gpu/drm/drm_debugfs.c static const struct file_operations drm_debugfs_fops = { · line 212
Quoted 10 lines
drivers/gpu/drm/drm_debugfs.c +10
-0+seq_puts (m,"\n" ); +seq_puts (m," VAs | start | range | end | object | object offset\n" ); +seq_puts (m,"-------------------------------------------------------------------------------------------------------------\n" ); +drm_gpuva_for_each_va (va, mgr) { +if (unlikely (va == kva)) +continue ; + +seq_printf (m," | 0x%016llx | 0x%016llx | 0x%016llx | 0x%016llx | 0x%016llx\n" , + va->va.addr, va->va.range, va->va.addr + va->va.range, + (u64 )va->gem.obj, va->gem.offset);
Casting a pointer to u64 generates a warning on 32 bit systems:
drivers/gpu/drm/drm_debugfs.c:212:7: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
212 | (u64)va->gem.obj, va->gem.offset);
| ^Adding an extra cast to uintptr_t shuts the compiler up (alternatively a proper pointer type like %px could be used in the format string, but that does make the table layout harder).
The patch below fixes the warning for me.
Steve
----8<----
From bea59724d68fec0b3a56f82a42d0e4e59c514565 Mon Sep 17 00:00:00 2001
From: Steven Price <[email protected]>
Date: Thu, 20 Jul 2023 11:45:11 +0100
Subject: [PATCH] drm: debugfs: Silence warning from cast
Casting a pointer to an integer of a different size generates a warning from the compiler. First cast the pointer to a pointer-sized type to keep the compiler happy.
--- drivers/gpu/drm/drm_debugfs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/drm_debugfs.c +1 -1
@@ -209,7 +209,7 @@ int drm_debugfs_gpuva_info(struct seq_file *m,seq_printf (m," | 0x%016llx | 0x%016llx | 0x%016llx | 0x%016llx | 0x%016llx\n" , va->va.addr, va->va.range, va->va.addr + va->va.range, - (u64 )va->gem.obj, va->gem.offset); + (u64 )(uintptr_t )va->gem.obj, va->gem.offset); }return 0 ;
Signature · 1 line
2.39.2
PATCH v8 3/12 drm/nouveau: new VM_BIND uapi interfaces
In reply to Danilo KrummrichThis commit provides the interfaces for the new UAPI motivated by the Vulkan API. It allows user mode drivers (UMDs) to:
1) Initialize a GPU virtual address (VA) space via the new DRM_IOCTL_NOUVEAU_VM_INIT ioctl. UMDs can provide a kernel reserved VA area.
2) Bind and unbind GPU VA space mappings via the new DRM_IOCTL_NOUVEAU_VM_BIND ioctl.
3) Execute push buffers with the new DRM_IOCTL_NOUVEAU_EXEC ioctl.
Both, DRM_IOCTL_NOUVEAU_VM_BIND and DRM_IOCTL_NOUVEAU_EXEC support asynchronous processing with DRM syncobjs as synchronization mechanism.
The default DRM_IOCTL_NOUVEAU_VM_BIND is synchronous processing, DRM_IOCTL_NOUVEAU_EXEC supports asynchronous processing only.
--- Documentation/gpu/driver-uapi.rst | 8 ++ include/uapi/drm/nouveau_drm.h | 209 ++++++++++++++++++++++++++++++ 2 files changed, 217 insertions(+)
Documentation/gpu/driver-uapi.rst +8 -0
@@ -6,3 +6,11 @@ drm/i915 uAPI ============= .. kernel-doc:: include/uapi/drm/i915_drm.h + +drm/nouveau uAPI +================ + +VM_BIND / EXEC uAPI +------------------- + +.. kernel-doc:: include/uapi/drm/nouveau_drm.h
include/uapi/drm/nouveau_drm.h +209 -0
227 lines not on this page · raw message
Signature · 1 line
2.41.0
Re: [PATCH drm-misc-next v8 03/12] drm/nouveau: new VM_BIND uapi interfaces
Quoted 85 lines · Danilo Krummrich
This commit provides the interfaces for the new UAPI motivated by the Vulkan API. It allows user mode drivers (UMDs) to:
1) Initialize a GPU virtual address (VA) space via the new
DRM_IOCTL_NOUVEAU_VM_INITioctl. UMDs can provide a kernel reserved VA area.2) Bind and unbind GPU VA space mappings via the new
DRM_IOCTL_NOUVEAU_VM_BINDioctl.3) Execute push buffers with the new
DRM_IOCTL_NOUVEAU_EXECioctl.Both,
DRM_IOCTL_NOUVEAU_VM_BINDandDRM_IOCTL_NOUVEAU_EXECsupport asynchronous processing with DRM syncobjs as synchronization mechanism.The default
DRM_IOCTL_NOUVEAU_VM_BINDis synchronous processing,DRM_IOCTL_NOUVEAU_EXECsupports asynchronous processing only.Co-authored-by Dave Airlie <[email protected]>Signed-off-by Danilo Krummrich <[email protected]>--- Documentation/gpu/driver-uapi.rst | 8 ++ include/uapi/drm/nouveau_drm.h | 209 ++++++++++++++++++++++++++++++ 2 files changed, 217 insertions(+)Documentation/gpu/driver-uapi.rst +8
-0@@ -6,3 +6,11 @@ drm/i915 uAPI ============= .. kernel-doc:: include/uapi/drm/i915_drm.h + +drm/nouveau uAPI +================ + +VM_BIND / EXEC uAPI +------------------- + +.. kernel-doc:: include/uapi/drm/nouveau_drm.hinclude/uapi/drm/nouveau_drm.h +11
-0@@ -126,6 +126,209 @@ struct drm_nouveau_gem_cpu_fini {__u32 handle; }; +/** +* struct drm_nouveau_sync - sync object +* +* This structure serves as synchronization mechanism for (potentially) +* asynchronous operations such as EXEC or VM_BIND. +*/ +struct drm_nouveau_sync { +/** +* @flags: the flags for a sync object +* +* The first 8 bits are used to determine the type of the sync object.
include/uapi/drm/nouveau_drm.h +22
-0+*/ +__u32 flags; +#define DRM_NOUVEAU_SYNC_SYNCOBJ 0x0 +#define DRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ 0x1 +#define DRM_NOUVEAU_SYNC_TYPE_MASK 0xf +/** +* @handle: the handle of the sync object +*/ +__u32 handle; +/** +* @timeline_value: +* +* The timeline point of the sync object in case the syncobj is of +* type DRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ. +*/ +__u64 timeline_value; +}; + +/** +* struct drm_nouveau_vm_init - GPU VA space init structure +* +* Used to initialize the GPU's VA space for a user client, telling the
Quoted 4 lines
kernel
+ * which portion of the VA space is managed by the UMD and kernel respectively.
I assume this has to be called quite early. Like maybe before any BOs or channels are created? In any case, it'd be nice to have that documented.
include/uapi/drm/nouveau_drm.h
Quoted 13 lines
+ */
include/uapi/drm/nouveau_drm.h +3
-0+struct drm_nouveau_vm_init { +/** +* @unmanaged_addr: start address of the kernel managed VA space region
include/uapi/drm/nouveau_drm.h +4
-0+*/ +__u64 unmanaged_addr; +/** +* @unmanaged_size: size of the kernel managed VA space region in bytes
include/uapi/drm/nouveau_drm.h +2
-0+*/ +__u64 unmanaged_size;
Over-all, I think this is the right API. My only concern is with the word "unmanaged". None of the VA space is unmanaged. Some is userspace-managed and some is kernel-managed. I guess "unmanaged" kinda makes sense because this is coming from userspace and it's saying which bits it manages and which bits it doesn't. Still seems clunky to me. Maybe kernel_managed? IDK, that feels weird too. Since we're already using UMD in this file, we could call it kmd_managed. IDK. 🤷🏻♀️
Yeah, I know this is a total bikeshed color thing and I'm not going to block anything based on it. 😅 Just wanted to see if we can come up with anything better. It's documented and that's the important thing.
Quoted 69 lines
include/uapi/drm/nouveau_drm.h +7
-0+}; + +/** +* struct drm_nouveau_vm_bind_op - VM_BIND operation +* +* This structure represents a single VM_BIND operation. UMDs should pass +* an array of this structure via struct drm_nouveau_vm_bind's &op_ptr field.
+ */include/uapi/drm/nouveau_drm.h +10
-0+struct drm_nouveau_vm_bind_op { +/** +* @op: the operation type +*/ +__u32 op; +/** +* @DRM_NOUVEAU_VM_BIND_OP_MAP: +* +* Map a GEM object to the GPU's VA space. Optionally, the +* &DRM_NOUVEAU_VM_BIND_SPARSE flag can be passed to instruct the kernel to
+ * create sparse mappings for the given range.
+ */include/uapi/drm/nouveau_drm.h +5
-0+#define DRM_NOUVEAU_VM_BIND_OP_MAP 0x0 +/** +* @DRM_NOUVEAU_VM_BIND_OP_UNMAP: +* +* Unmap an existing mapping in the GPU's VA space. If the region the mapping
+ * is located in is a sparse region, new sparse mappings are created where the
+ * unmapped (memory backed) mapping was mapped previously. To remove a sparse
+ * region the&DRM_NOUVEAU_VM_BIND_SPARSEmust be set.
+ */include/uapi/drm/nouveau_drm.h +23
-0+#define DRM_NOUVEAU_VM_BIND_OP_UNMAP 0x1 +/** +* @flags: the flags for a &drm_nouveau_vm_bind_op +*/ +__u32 flags; +/** +* @DRM_NOUVEAU_VM_BIND_SPARSE: +* +* Indicates that an allocated VA space region should be sparse. +*/ +#define DRM_NOUVEAU_VM_BIND_SPARSE (1 << 8) +/** +* @handle: the handle of the DRM GEM object to map +*/ +__u32 handle; +/** +* @pad: 32 bit padding, should be 0 +*/ +__u32 pad; +/** +* @addr: +* +* the address the VA space region or (memory backed) mapping should be mapped to
include/uapi/drm/nouveau_drm.h +10
-0+*/ +__u64 addr; +/** +* @bo_offset: the offset within the BO backing the mapping +*/ +__u64 bo_offset; +/** +* @range: the size of the requested mapping in bytes +*/ +__u64 range;
include/uapi/drm/nouveau_drm.h struct drm_nouveau_gem_cpu_fini { · line 244
Quoted 10 lines
include/uapi/drm/nouveau_drm.h +10
-0+}; + +/** +* struct drm_nouveau_vm_bind - structure for DRM_IOCTL_NOUVEAU_VM_BIND +*/ +struct drm_nouveau_vm_bind { +/** +* @op_count: the number of &drm_nouveau_vm_bind_op +*/ +__u32 op_count;
I've chatted a bit with Dave on IRC about this but both VM_BIND and EXEC should support op_count == 0 and do exactly the same thing that they would do if there were real ops. In the case of vm_bind, that just means wait on the waits and then signal the signals. In particular, it should NOT just return success and do nothing. Dave has a patch for this for EXEC but IDK if VM_BIND needs any attention. Of course, if it's not ASYNC, then quickly doing nothing after validating inputs is acceptable.
Quoted 60 lines
include/uapi/drm/nouveau_drm.h +8
-0+/** +* @flags: the flags for a &drm_nouveau_vm_bind ioctl +*/ +__u32 flags; +/** +* @DRM_NOUVEAU_VM_BIND_RUN_ASYNC: +* +* Indicates that the given VM_BIND operation should be executed asynchronously
+ * by the kernel.
+ *
+ * If this flag is not supplied the kernel executes the associated operations
+ * synchronously and doesn't accept any&drm_nouveau_syncobjects.
+ */include/uapi/drm/nouveau_drm.h +7
-0+#define DRM_NOUVEAU_VM_BIND_RUN_ASYNC 0x1 +/** +* @wait_count: the number of wait &drm_nouveau_syncs +*/ +__u32 wait_count; +/** +* @sig_count: the number of &drm_nouveau_syncs to signal when finished
include/uapi/drm/nouveau_drm.h +20
-0+*/ +__u32 sig_count; +/** +* @wait_ptr: pointer to &drm_nouveau_syncs to wait for +*/ +__u64 wait_ptr; +/** +* @sig_ptr: pointer to &drm_nouveau_syncs to signal when finished +*/ +__u64 sig_ptr; +/** +* @op_ptr: pointer to the &drm_nouveau_vm_bind_ops to execute +*/ +__u64 op_ptr; +}; + +/** +* struct drm_nouveau_exec_push - EXEC push operation +* +* This structure represents a single EXEC push operation. UMDs should pass an
+ * array of this structure via struct drm_nouveau_exec's&push_ptrfield.
+ */include/uapi/drm/nouveau_drm.h +13
-0+struct drm_nouveau_exec_push { +/** +* @va: the virtual address of the push buffer mapping +*/ +__u64 va; +/** +* @va_len: the length of the push buffer mapping +*/ +__u64 va_len; +}; + +/** +* struct drm_nouveau_exec - structure for DRM_IOCTL_NOUVEAU_EXEC
include/uapi/drm/nouveau_drm.h struct drm_nouveau_gem_cpu_fini { · line 309
Quoted 10 lines
include/uapi/drm/nouveau_drm.h +10
-0+*/ +struct drm_nouveau_exec { +/** +* @channel: the channel to execute the push buffer in +*/ +__u32 channel; +/** +* @push_count: the number of &drm_nouveau_exec_push ops +*/ +__u32 push_count;
Same comment as above. We want push_count == 0 to behave the same as any other EXEC just without anything new. In particular, it needs to wait on all the waits as well as the previous EXECs on that channel and then signal the sigs. I know Dave has a patch for this and it's working quite well in my testing.
Other than that, everything looks good. I'm still re-reading all the NVK patches but they've been working quite well in my testing this week apart from a perf issue I need to dig into. I'll give a real RB once we're sure we all agree on the semantics of _count.
~Faith
Quoted 55 lines
include/uapi/drm/nouveau_drm.h +6
-0+/** +* @wait_count: the number of wait &drm_nouveau_syncs +*/ +__u32 wait_count; +/** +* @sig_count: the number of &drm_nouveau_syncs to signal when finished
include/uapi/drm/nouveau_drm.h +21
-0+*/ +__u32 sig_count; +/** +* @wait_ptr: pointer to &drm_nouveau_syncs to wait for +*/ +__u64 wait_ptr; +/** +* @sig_ptr: pointer to &drm_nouveau_syncs to signal when finished +*/ +__u64 sig_ptr; +/** +* @push_ptr: pointer to &drm_nouveau_exec_push ops +*/ +__u64 push_ptr; +}; +#define DRM_NOUVEAU_GETPARAM 0x00/* deprecated */ #define DRM_NOUVEAU_SETPARAM 0x01/* deprecated */ #define DRM_NOUVEAU_CHANNEL_ALLOC 0x02/* deprecated */ @@ -136,6 +339,9 @@ struct drm_nouveau_gem_cpu_fini {#define DRM_NOUVEAU_NVIF 0x07#define DRM_NOUVEAU_SVM_INIT 0x08#define DRM_NOUVEAU_SVM_BIND 0x09 +#define DRM_NOUVEAU_VM_INIT 0x10 +#define DRM_NOUVEAU_VM_BIND 0x11 +#define DRM_NOUVEAU_EXEC 0x12#define DRM_NOUVEAU_GEM_NEW 0x40#define DRM_NOUVEAU_GEM_PUSHBUF 0x41#define DRM_NOUVEAU_GEM_CPU_PREP 0x42@@ -197,6 +403,9 @@ struct drm_nouveau_svm_bind {#define DRM_IOCTL_NOUVEAU_GEM_CPU_FINI DRM_IOW (DRM_COMMAND_BASE + DRM_NOUVEAU_GEM_CPU_FINI, struct drm_nouveau_gem_cpu_fini)#define DRM_IOCTL_NOUVEAU_GEM_INFO DRM_IOWR(DRM_COMMAND_BASE + DRM_NOUVEAU_GEM_INFO, struct drm_nouveau_gem_info) +#define DRM_IOCTL_NOUVEAU_VM_INIT DRM_IOWR(DRM_COMMAND_BASE + DRM_NOUVEAU_VM_INIT, struct drm_nouveau_vm_init) +#define DRM_IOCTL_NOUVEAU_VM_BIND DRM_IOWR(DRM_COMMAND_BASE +
DRM_NOUVEAU_VM_BIND, structdrm_nouveau_vm_bind)
+#defineDRM_IOCTL_NOUVEAU_EXECDRM_IOWR(DRM_COMMAND_BASE +DRM_NOUVEAU_EXEC, structdrm_nouveau_exec)
#ifdefined(__cplusplus)
}
#endif
--
2.41.0
Re: [PATCH drm-misc-next v8 03/12] drm/nouveau: new VM_BIND uapi interfaces
Quoted 96 lines · Faith Ekstrand
On Wed, Jul 19, 2023 at 7:15 PM Danilo Krummrich <[email protected] <mailto:[email protected]>> wrote:
This commit provides the interfaces for the new UAPI motivated by the Vulkan API. It allows user mode drivers (UMDs) to:
1) Initialize a GPU virtual address (VA) space via the new
DRM_IOCTL_NOUVEAU_VM_INITioctl. UMDs can provide a kernel reserved VA area.2) Bind and unbind GPU VA space mappings via the new
DRM_IOCTL_NOUVEAU_VM_BINDioctl.3) Execute push buffers with the new
DRM_IOCTL_NOUVEAU_EXECioctl.Both,
DRM_IOCTL_NOUVEAU_VM_BINDandDRM_IOCTL_NOUVEAU_EXECsupport asynchronous processing with DRM syncobjs as synchronization mechanism.The default
DRM_IOCTL_NOUVEAU_VM_BINDis synchronous processing,DRM_IOCTL_NOUVEAU_EXECsupports asynchronous processing only.Co-authored-by Dave Airlie <[email protected]<mailto:[email protected]>>
Signed-off-by Danilo Krummrich <[email protected]<mailto:[email protected]>>
---
Documentation/gpu/driver-uapi.rst | 8 ++ include/uapi/drm/nouveau_drm.h | 209 ++++++++++++++++++++++++++++++ 2 files changed, 217 insertions(+)diff --git
a/Documentation/gpu/driver-uapi.rstb/Documentation/gpu/driver-uapi.rstindex4411e6919a3d..9c7ca6e33a68100644 ---a/Documentation/gpu/driver-uapi.rst+++b/Documentation/gpu/driver-uapi.rst@@ -6,3 +6,11 @@ drm/i915 uAPI
=============.. kernel-doc::
include/uapi/drm/i915_drm.h
+
+drm/nouveau uAPI
+================
+
+VM_BIND / EXEC uAPI
+-------------------
+
+.. kernel-doc::include/uapi/drm/nouveau_drm.hdiff --gita/include/uapi/drm/nouveau_drm.hb/include/uapi/drm/nouveau_drm.h
index853a327433d3..4d3a70529637100644
---a/include/uapi/drm/nouveau_drm.h
+++b/include/uapi/drm/nouveau_drm.h
@@ -126,6 +126,209 @@ structdrm_nouveau_gem_cpu_fini{
__u32handle;
};+/**
+ * structdrm_nouveau_sync- sync object
+ *
+ * This structure serves as synchronization mechanism for (potentially)
+ * asynchronous operations such as EXEC orVM_BIND.
+ */
+structdrm_nouveau_sync{
+ /**
+ * @flags: the flags for a sync object
+ *
+ * The first 8 bits are used to determine the type of the sync object.
+ */
+ __u32 flags;
+#defineDRM_NOUVEAU_SYNC_SYNCOBJ0x0
+#defineDRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ0x1
+#defineDRM_NOUVEAU_SYNC_TYPE_MASK0xf
+ /**
+ * @handle: the handle of the sync object
+ */
+ __u32 handle;
+ /**
+ * @timeline_value:
+ *
+ * The timeline point of the sync object in case the syncobj is of
+ * typeDRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ.
+ */
+ __u64timeline_value;
+};
+
+/**
+ * structdrm_nouveau_vm_init- GPU VA space init structure
+ *
+ * Used to initialize the GPU's VA space for a user client, telling the kernel
+ * which portion of the VA space is managed by the UMD and kernel respectively.
Quoted 1 line
I assume this has to be called quite early. Like maybe before any BOs or channels are created? In any case, it'd be nice to have that documented.
Exactly, doing any of those will disable the new uAPI entirely if it wasn't yet initialized. I will add some documentation for this.
Quoted 15 lines
+ */
+structdrm_nouveau_vm_init{
+ /**
+ * @unmanaged_addr: start address of the kernel managed VA space region
+ */
+ __u64unmanaged_addr;
+ /**
+ * @unmanaged_size: size of the kernel managed VA space region in bytes
+ */
+ __u64unmanaged_size;
Quoted 1 line
Over-all, I think this is the right API. My only concern is with the word "unmanaged". None of the VA space is unmanaged. Some is userspace-managed and some is kernel-managed. I guess "unmanaged" kinda makes sense because this is coming from userspace and it's saying which bits it manages and which bits it doesn't. Still seems clunky to me. Maybe
kernel_managed? IDK, that feels weird too. Since we're already using UMD in this file, we could call itkmd_managed. IDK. 🤷🏻♀️
kernel_managed / kmd_managed both sounds fine to me. I'm good with either one.
Quoted 84 lines
Yeah, I know this is a total bikeshed color thing and I'm not going to block anything based on it. 😅 Just wanted to see if we can come up with anything better. It's documented and that's the important thing.
+};
+
+/**
+ * structdrm_nouveau_vm_bind_op-VM_BINDoperation
+ *
+ * This structure represents a singleVM_BINDoperation. UMDs should pass
+ * an array of this structure via struct drm_nouveau_vm_bind's&op_ptrfield.
+ */
+structdrm_nouveau_vm_bind_op{
+ /**
+ * @op: the operation type
+ */
+ __u32 op;
+/**
+ * @DRM_NOUVEAU_VM_BIND_OP_MAP:
+ *
+ * Map a GEM object to the GPU's VA space. Optionally, the
+ *&DRM_NOUVEAU_VM_BIND_SPARSEflag can be passed to instruct the kernel to
+ * create sparse mappings for the given range.
+ */
+#defineDRM_NOUVEAU_VM_BIND_OP_MAP0x0
+/**
+ * @DRM_NOUVEAU_VM_BIND_OP_UNMAP:
+ *
+ * Unmap an existing mapping in the GPU's VA space. If the region the mapping
+ * is located in is a sparse region, new sparse mappings are created where the
+ * unmapped (memory backed) mapping was mapped previously. To remove a sparse
+ * region the&DRM_NOUVEAU_VM_BIND_SPARSEmust be set.
+ */
+#defineDRM_NOUVEAU_VM_BIND_OP_UNMAP0x1
+ /**
+ * @flags: the flags for a&drm_nouveau_vm_bind_op
+ */
+ __u32 flags;
+/**
+ * @DRM_NOUVEAU_VM_BIND_SPARSE:
+ *
+ * Indicates that an allocated VA space region should be sparse.
+ */
+#defineDRM_NOUVEAU_VM_BIND_SPARSE(1 << 8)
+ /**
+ * @handle: the handle of the DRM GEM object to map + */
+ __u32 handle;
+ /**
+ * @pad: 32 bit padding, should be 0
+ */
+ __u32 pad;
+ /**
+ * @addr:
+ *
+ * the address the VA space region or (memory backed) mapping should be mapped to
+ */
+ __u64 addr;
+ /**
+ * @bo_offset: the offset within the BO backing the mapping + */
+ __u64bo_offset;
+ /**
+ * @range: the size of the requested mapping in bytes + */
+ __u64 range;
+};
+
+/**
+ * structdrm_nouveau_vm_bind- structure forDRM_IOCTL_NOUVEAU_VM_BIND
+ */
+structdrm_nouveau_vm_bind{
+ /**
+ * @op_count: the number of&drm_nouveau_vm_bind_op+ */
+ __u32op_count;
Quoted 1 line
I've chatted a bit with Dave on IRC about this but both
VM_BINDand EXEC should supportop_count == 0and do exactly the same thing that they would do if there were real ops. In the case ofvm_bind, that just means wait on the waits and then signal the signals. In particular, it should NOT just return success and do nothing. Dave has a patch for this for EXEC but IDK ifVM_BINDneeds any attention. Of course, if it's not ASYNC, then quickly doing nothing after validating inputs is acceptable.
What will this be used for? I guess it would not be important to be executed in order with "regular" (non-noop) jobs? Because the only thing this would tell you is that e.g. for VM_BIND all previous binds completed, which is what we have syncobjs for.
- Danilo
Quoted 139 lines
+ /**
+ * @flags: the flags for a&drm_nouveau_vm_bindioctl + */
+ __u32 flags;
+/**
+ * @DRM_NOUVEAU_VM_BIND_RUN_ASYNC:
+ *
+ * Indicates that the givenVM_BINDoperation should be executed asynchronously
+ * by the kernel.
+ *
+ * If this flag is not supplied the kernel executes the associated operations
+ * synchronously and doesn't accept any&drm_nouveau_syncobjects.
+ */
+#defineDRM_NOUVEAU_VM_BIND_RUN_ASYNC0x1
+ /**
+ * @wait_count: the number of wait&drm_nouveau_syncs+ */
+ __u32wait_count;
+ /**
+ * @sig_count: the number of&drm_nouveau_syncsto signal when finished
+ */
+ __u32sig_count;
+ /**
+ * @wait_ptr: pointer to&drm_nouveau_syncsto wait for + */
+ __u64wait_ptr;
+ /**
+ * @sig_ptr: pointer to&drm_nouveau_syncsto signal when finished
+ */
+ __u64sig_ptr;
+ /**
+ * @op_ptr: pointer to the&drm_nouveau_vm_bind_opsto execute + */
+ __u64op_ptr;
+};
+
+/**
+ * structdrm_nouveau_exec_push- EXEC push operation
+ *
+ * This structure represents a single EXEC push operation. UMDs should pass an
+ * array of this structure via struct drm_nouveau_exec's&push_ptrfield.
+ */
+structdrm_nouveau_exec_push{
+ /**
+ * @va: the virtual address of the push buffer mapping + */
+ __u64 va;
+ /**
+ * @va_len: the length of the push buffer mapping + */
+ __u64va_len;
+};
+
+/**
+ * structdrm_nouveau_exec- structure forDRM_IOCTL_NOUVEAU_EXEC
+ */
+structdrm_nouveau_exec{
+ /**
+ * @channel: the channel to execute the push buffer in + */
+ __u32 channel;
+ /**
+ * @push_count: the number of&drm_nouveau_exec_pushops + */
+ __u32push_count;Same comment as above. We want
push_count == 0to behave the same as any other EXEC just without anything new. In particular, it needs to wait on all the waits as well as the previous EXECs on that channel and then signal the sigs. I know Dave has a patch for this and it's working quite well in my testing.Other than that, everything looks good. I'm still re-reading all the NVK patches but they've been working quite well in my testing this week apart from a perf issue I need to dig into. I'll give a real RB once we're sure we all agree on the semantics of
_count.~Faith
+ /** + * @wait_count: the number of wait &drm_nouveau_syncs + */ + __u32 wait_count; + /** + * @sig_count: the number of &drm_nouveau_syncs to signal when finished + */ + __u32 sig_count; + /** + * @wait_ptr: pointer to &drm_nouveau_syncs to wait for + */ + __u64 wait_ptr; + /** + * @sig_ptr: pointer to &drm_nouveau_syncs to signal when finished + */ + __u64 sig_ptr; + /** + * @push_ptr: pointer to &drm_nouveau_exec_push ops + */ + __u64 push_ptr; +}; + #define DRM_NOUVEAU_GETPARAM 0x00 /* deprecated */ #define DRM_NOUVEAU_SETPARAM 0x01 /* deprecated */ #define DRM_NOUVEAU_CHANNEL_ALLOC 0x02 /* deprecated */ @@ -136,6 +339,9 @@ struct drm_nouveau_gem_cpu_fini { #define DRM_NOUVEAU_NVIF 0x07 #define DRM_NOUVEAU_SVM_INIT 0x08 #define DRM_NOUVEAU_SVM_BIND 0x09 +#define DRM_NOUVEAU_VM_INIT 0x10 +#define DRM_NOUVEAU_VM_BIND 0x11 +#define DRM_NOUVEAU_EXEC 0x12 #define DRM_NOUVEAU_GEM_NEW 0x40 #define DRM_NOUVEAU_GEM_PUSHBUF 0x41 #define DRM_NOUVEAU_GEM_CPU_PREP 0x42 @@ -197,6 +403,9 @@ struct drm_nouveau_svm_bind { #define DRM_IOCTL_NOUVEAU_GEM_CPU_FINI DRM_IOW (DRM_COMMAND_BASE + DRM_NOUVEAU_GEM_CPU_FINI, struct drm_nouveau_gem_cpu_fini) #define DRM_IOCTL_NOUVEAU_GEM_INFO DRM_IOWR(DRM_COMMAND_BASE + DRM_NOUVEAU_GEM_INFO, struct drm_nouveau_gem_info) +#define DRM_IOCTL_NOUVEAU_VM_INIT DRM_IOWR(DRM_COMMAND_BASE + DRM_NOUVEAU_VM_INIT, struct drm_nouveau_vm_init) +#define DRM_IOCTL_NOUVEAU_VM_BIND DRM_IOWR(DRM_COMMAND_BASE + DRM_NOUVEAU_VM_BIND, struct drm_nouveau_vm_bind) +#define DRM_IOCTL_NOUVEAU_EXEC DRM_IOWR(DRM_COMMAND_BASE + DRM_NOUVEAU_EXEC, struct drm_nouveau_exec) #if defined(__cplusplus) } #endif -- 2.41.0
Re: [PATCH drm-misc-next v8 03/12] drm/nouveau: new VM_BIND uapi interfaces
Quoted 104 lines · Danilo Krummrich
Quoted 9 lines · Faith Ekstrand
On Wed, Jul 19, 2023 at 7:15 PM Danilo Krummrich <[email protected] <mailto:[email protected]>> wrote:
This commit provides the interfaces for the new UAPI motivated by the Vulkan API. It allows user mode drivers (UMDs) to:
1) Initialize a GPU virtual address (VA) space via the new
DRM_IOCTL_NOUVEAU_VM_INITioctl. UMDs can provide a kernelreserved
Quoted 9 lines
VA area.
2) Bind and unbind GPU VA space mappings via the new
DRM_IOCTL_NOUVEAU_VM_BINDioctl.3) Execute push buffers with the new
DRM_IOCTL_NOUVEAU_EXECioctl.Both,
DRM_IOCTL_NOUVEAU_VM_BINDandDRM_IOCTL_NOUVEAU_EXECsupport asynchronous processing with DRM syncobjs as synchronizationmechanism.
Quoted 11 lines
The default
DRM_IOCTL_NOUVEAU_VM_BINDis synchronous processing,DRM_IOCTL_NOUVEAU_EXECsupports asynchronous processing only.Co-authored-by Dave Airlie <[email protected]<mailto:[email protected]>>
Signed-off-by Danilo Krummrich <[email protected]<mailto:[email protected]>>
---
Documentation/gpu/driver-uapi.rst| 8 ++
include/uapi/drm/nouveau_drm.h| 209++++++++++++++++++++++++++++++
Quoted 24 lines
2 files changed, 217 insertions(+)diff --git a/Documentation/gpu/driver-uapi.rst b/Documentation/gpu/driver-uapi.rst index 4411e6919a3d..9c7ca6e33a68 100644 --- a/Documentation/gpu/driver-uapi.rst +++ b/Documentation/gpu/driver-uapi.rst @@ -6,3 +6,11 @@ drm/i915 uAPI ============= .. kernel-doc:: include/uapi/drm/i915_drm.h + +drm/nouveau uAPI +================ + +VM_BIND / EXEC uAPI +------------------- + +.. kernel-doc:: include/uapi/drm/nouveau_drm.h diff --git a/include/uapi/drm/nouveau_drm.h b/include/uapi/drm/nouveau_drm.h index 853a327433d3..4d3a70529637 100644 --- a/include/uapi/drm/nouveau_drm.h +++ b/include/uapi/drm/nouveau_drm.hQuoted 8 lines
@@ -126,6 +126,209 @@ struct drm_nouveau_gem_cpu_fini { __u32 handle; }; +/** + * struct drm_nouveau_sync - sync object + * + * This structure serves as synchronization mechanism for(potentially)
Quoted 36 lines
+ * asynchronous operations such as EXEC or
VM_BIND.
+ */
+structdrm_nouveau_sync{
+ /**
+ * @flags: the flags for a sync object
+ *
+ * The first 8 bits are used to determine the type of the
sync object.
+ */
+__u32flags;
+#defineDRM_NOUVEAU_SYNC_SYNCOBJ0x0
+#defineDRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ0x1
+#defineDRM_NOUVEAU_SYNC_TYPE_MASK0xf
+ /**
+ * @handle: the handle of the sync object
+ */
+__u32handle;
+ /**
+ * @timeline_value:
+ *
+ * The timeline point of the sync object in case the syncobj
is of
+ * typeDRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ.
+ */
+__u64timeline_value;
+};
+
+/**
+ * structdrm_nouveau_vm_init- GPU VA space init structure
+ *
+ * Used to initialize the GPU's VA space for a user client, telling the kernel
+ * which portion of the VA space is managed by the UMD and kernel respectively.
Quoted 6 lines
Quoted 2 lines
I assume this has to be called quite early. Like maybe before any BOs or channels are created? In any case, it'd be nice to have that documented.
Exactly, doing any of those will disable the new uAPI entirely if it wasn't yet initialized. I will add some documentation for this.
Thanks!
Quoted 23 lines
Quoted 15 lines
+ */
+structdrm_nouveau_vm_init{
+ /**
+ * @unmanaged_addr: start address of the kernel managed VA
space region
+ */
+__u64unmanaged_addr;
+ /**
+ * @unmanaged_size: size of the kernel managed VA space
region in bytes
+ */
+__u64unmanaged_size;Quoted 7 lines
Over-all, I think this is the right API. My only concern is with the word "unmanaged". None of the VA space is unmanaged. Some is userspace-managed and some is kernel-managed. I guess "unmanaged" kinda makes sense because this is coming from userspace and it's saying which bits it manages and which bits it doesn't. Still seems clunky to me. Maybe
kernel_managed? IDK, that feels weird too. Since we're already using UMD in this file, we could call itkmd_managed. IDK. 🤷🏻♀️
Quoted 3 lines
kernel_managed/kmd_managedboth sounds fine to me. I'm good with either one.
Let's go with kernel_managed then, unless anyone objects.
Quoted 95 lines
Quoted 70 lines
Yeah, I know this is a total bikeshed color thing and I'm not going to block anything based on it. 😅 Just wanted to see if we can come up with anything better. It's documented and that's the important thing.
+};
+
+/**
+ * structdrm_nouveau_vm_bind_op-VM_BINDoperation
+ *
+ * This structure represents a singleVM_BINDoperation. UMDs should pass
+ * an array of this structure via struct drm_nouveau_vm_bind's&op_ptrfield.
+ */
+structdrm_nouveau_vm_bind_op{
+ /**
+ * @op: the operation type
+ */
+__u32op;
+/**
+ * @DRM_NOUVEAU_VM_BIND_OP_MAP:
+ *
+ * Map a GEM object to the GPU's VA space. Optionally, the
+ *&DRM_NOUVEAU_VM_BIND_SPARSEflag can be passed to instruct the kernel to
+ * create sparse mappings for the given range.
+ */
+#defineDRM_NOUVEAU_VM_BIND_OP_MAP0x0
+/**
+ * @DRM_NOUVEAU_VM_BIND_OP_UNMAP:
+ *
+ * Unmap an existing mapping in the GPU's VA space. If the region the mapping
+ * is located in is a sparse region, new sparse mappings are created where the
+ * unmapped (memory backed) mapping was mapped previously. To remove a sparse
+ * region the&DRM_NOUVEAU_VM_BIND_SPARSEmust be set.
+ */
+#defineDRM_NOUVEAU_VM_BIND_OP_UNMAP0x1
+ /**
+ * @flags: the flags for a&drm_nouveau_vm_bind_op
+ */
+__u32flags;
+/**
+ * @DRM_NOUVEAU_VM_BIND_SPARSE:
+ *
+ * Indicates that an allocated VA space region should be sparse.
+ */
+#defineDRM_NOUVEAU_VM_BIND_SPARSE(1 << 8)
+ /**
+ * @handle: the handle of the DRM GEM object to map
+ */
+__u32handle;
+ /**
+ * @pad: 32 bit padding, should be 0
+ */
+__u32pad;
+ /**
+ * @addr:
+ *
+ * the address the VA space region or (memory backed)
mapping should be mapped to
+ */
+__u64addr;
+ /**
+ * @bo_offset: the offset within the BO backing the mapping
+ */
+__u64bo_offset;Quoted 8 lines
+ /**
+ * @range: the size of the requested mapping in bytes
+ */
+__u64range;
+};
+
+/**
+ * structdrm_nouveau_vm_bind- structure for
DRM_IOCTL_NOUVEAU_VM_BINDQuoted 8 lines
+ */
+structdrm_nouveau_vm_bind{
+ /**
+ * @op_count: the number of&drm_nouveau_vm_bind_op
+ */
+__u32op_count;Quoted 7 lines
I've chatted a bit with Dave on IRC about this but both
VM_BINDand EXEC should supportop_count == 0and do exactly the same thing that they would do if there were real ops. In the case ofvm_bind, that just means wait on the waits and then signal the signals. In particular, it should NOT just return success and do nothing. Dave has a patch for this for EXEC but IDK ifVM_BINDneeds any attention. Of course, if it's not ASYNC, then quickly doing nothing after validating inputs is acceptable.
Quoted 5 lines
What will this be used for? I guess it would not be important to be executed in order with "regular" (non-noop) jobs? Because the only thing this would tell you is that e.g. for
VM_BINDall previous binds completed, which is what we have syncobjs for.
Yes, exactly that. Effectively, it allows you to add more signal objects to the last submitted job after the fact. Vulkan allows submits with zero command buffers and they have to behave the same as submits that actually do work. We also use this internally in Mesa to implement things like vkQueueWaitForIdle. (It's actually a little more subtle than that because the new signals will also wait on any waits in the zero-size exec.)
The standard driver work-around for this which Mesa Vulkan drivers carry is to have a no-op pushbuf that you stash somewhere. Whenever command_buffer_count == 0, you submit that one instead to trick the kernel into thinking it's doing work. Since we're building a new UAPI, though, we may as well just support this corner case directly in the kernel driver.
~Faith
Quoted 152 lines
- Danilo
Quoted 29 lines
+ /**
+ * @flags: the flags for a&drm_nouveau_vm_bindioctl
+ */
+__u32flags;
+/**
+ * @DRM_NOUVEAU_VM_BIND_RUN_ASYNC:
+ *
+ * Indicates that the givenVM_BINDoperation should be executed asynchronously
+ * by the kernel.
+ *
+ * If this flag is not supplied the kernel executes the associated operations
+ * synchronously and doesn't accept any&drm_nouveau_syncobjects.
+ */
+#defineDRM_NOUVEAU_VM_BIND_RUN_ASYNC0x1
+ /**
+ * @wait_count: the number of wait&drm_nouveau_syncs
+ */
+__u32wait_count;
+ /**
+ * @sig_count: the number of&drm_nouveau_syncsto signal
when finished
+ */
+__u32sig_count;
+ /**
+ * @wait_ptr: pointer to&drm_nouveau_syncsto wait for
+ */Quoted 8 lines
+
__u64wait_ptr;
+ /**
+ * @sig_ptr: pointer to&drm_nouveau_syncsto signal when
finished
+ */
+__u64sig_ptr;
+ /**
+ * @op_ptr: pointer to the&drm_nouveau_vm_bind_opstoexecute
Quoted 95 lines
+ */
+__u64op_ptr;
+};
+
+/**
+ * structdrm_nouveau_exec_push- EXEC push operation
+ *
+ * This structure represents a single EXEC push operation. UMDs should pass an
+ * array of this structure via struct drm_nouveau_exec's&push_ptrfield.
+ */
+structdrm_nouveau_exec_push{
+ /**
+ * @va: the virtual address of the push buffer mapping
+ */
+__u64va;
+ /**
+ * @va_len: the length of the push buffer mapping
+ */
+__u64va_len;
+};
+
+/**
+ * structdrm_nouveau_exec- structure forDRM_IOCTL_NOUVEAU_EXEC
+ */
+structdrm_nouveau_exec{
+ /**
+ * @channel: the channel to execute the push buffer in
+ */
+__u32channel;
+ /**
+ * @push_count: the number of&drm_nouveau_exec_pushops
+ */
+__u32push_count;Same comment as above. We want
push_count == 0to behave the same as any other EXEC just without anything new. In particular, it needs to wait on all the waits as well as the previous EXECs on that channel and then signal the sigs. I know Dave has a patch for this and it's working quite well in my testing.Other than that, everything looks good. I'm still re-reading all the NVK patches but they've been working quite well in my testing this week apart from a perf issue I need to dig into. I'll give a real RB once we're sure we all agree on the semantics of
_count.~Faith
+ /**
+ * @wait_count: the number of wait&drm_nouveau_syncs
+ */
+__u32wait_count;
+ /**
+ * @sig_count: the number of&drm_nouveau_syncsto signal
when finished
+ */
+__u32sig_count;
+ /**
+ * @wait_ptr: pointer to&drm_nouveau_syncsto wait for
+ */
+__u64wait_ptr;
+ /**
+ * @sig_ptr: pointer to&drm_nouveau_syncsto signal when
finished
+ */
+__u64sig_ptr;
+ /**
+ * @push_ptr: pointer to&drm_nouveau_exec_pushops
+ */
+__u64push_ptr;
+};
+
#defineDRM_NOUVEAU_GETPARAM0x00/* deprecated */
#defineDRM_NOUVEAU_SETPARAM0x01/* deprecated */
#defineDRM_NOUVEAU_CHANNEL_ALLOC0x02/* deprecated */
@@ -136,6 +339,9 @@ structdrm_nouveau_gem_cpu_fini{
#defineDRM_NOUVEAU_NVIF0x07
#defineDRM_NOUVEAU_SVM_INIT0x08
#defineDRM_NOUVEAU_SVM_BIND0x09
+#defineDRM_NOUVEAU_VM_INIT0x10
+#defineDRM_NOUVEAU_VM_BIND0x11
+#defineDRM_NOUVEAU_EXEC0x12
#defineDRM_NOUVEAU_GEM_NEW0x40
#defineDRM_NOUVEAU_GEM_PUSHBUF0x41
#defineDRM_NOUVEAU_GEM_CPU_PREP0x42
@@ -197,6 +403,9 @@ structdrm_nouveau_svm_bind{
#defineDRM_IOCTL_NOUVEAU_GEM_CPU_FINIDRM_IOW
(DRM_COMMAND_BASE+DRM_NOUVEAU_GEM_CPU_FINI, structdrm_nouveau_gem_cpu_fini)
#defineDRM_IOCTL_NOUVEAU_GEM_INFODRM_IOWR(DRM_COMMAND_BASE +DRM_NOUVEAU_GEM_INFO, struct
drm_nouveau_gem_info)Quoted 8 lines
+#define
DRM_IOCTL_NOUVEAU_VM_INITDRM_IOWR(DRM_COMMAND_BASE + DRM_NOUVEAU_VM_INIT, struct drm_nouveau_vm_init)
+#defineDRM_IOCTL_NOUVEAU_VM_BINDDRM_IOWR(DRM_COMMAND_BASE + DRM_NOUVEAU_VM_BIND, struct drm_nouveau_vm_bind)
+#defineDRM_IOCTL_NOUVEAU_EXEC
DRM_IOWR(DRM_COMMAND_BASE +DRM_NOUVEAU_EXEC, struct
drm_nouveau_exec)Quoted 6 lines
#if defined(__cplusplus) } #endif -- 2.41.0
PATCH v8 4/12 drm/nouveau: get vmm via nouveau_cli_vmm()
In reply to Danilo KrummrichProvide a getter function for the client's current vmm context. Since we'll add a new (u)vmm context for UMD bindings in subsequent commits, this will keep the code clean.
--- drivers/gpu/drm/nouveau/nouveau_bo.c | 2 +- drivers/gpu/drm/nouveau/nouveau_chan.c | 2 +- drivers/gpu/drm/nouveau/nouveau_drv.h | 9 +++++++++ drivers/gpu/drm/nouveau/nouveau_gem.c | 6 +++--- 4 files changed, 14 insertions(+), 5 deletions(-)
drivers/gpu/drm/nouveau/nouveau_bo.c +1 -1
@@ -204,7 +204,7 @@ nouveau_bo_alloc(struct nouveau_cli *cli, u64 *size, int *align, u32 domain,struct nouveau_drm *drm = cli->drm;struct nouveau_bo *nvbo;struct nvif_mmu *mmu = &cli->mmu; -struct nvif_vmm *vmm = cli->svm.cli ? &cli->svm.vmm : &cli->vmm.vmm; +struct nvif_vmm *vmm = &nouveau_cli_vmm (cli)->vmm;int i, pi =-1 ;if (!*size) {
drivers/gpu/drm/nouveau/nouveau_chan.c +1 -1
@@ -148,7 +148,7 @@ nouveau_channel_prep(struct nouveau_drm *drm, struct nvif_device *device, chan->device = device; chan->drm = drm; - chan->vmm = cli->svm.cli ? &cli->svm : &cli->vmm; + chan->vmm =nouveau_cli_vmm (cli);atomic_set (&chan->killed,0 );/* allocate memory for dma push buffer */
drivers/gpu/drm/nouveau/nouveau_drv.h +9 -0
@@ -112,6 +112,15 @@ struct nouveau_cli_work {struct dma_fence_cb cb; }; +static inline struct nouveau_vmm * +nouveau_cli_vmm (struct nouveau_cli *cli) +{ +if (cli->svm.cli) +return &cli->svm; + +return &cli->vmm; +} +void nouveau_cli_work_queue (struct nouveau_cli *,struct dma_fence *,struct nouveau_cli_work *);
drivers/gpu/drm/nouveau/nouveau_gem.c +3 -3
@@ -103,7 +103,7 @@ nouveau_gem_object_open(struct drm_gem_object *gem, struct drm_file *file_priv)struct nouveau_bo *nvbo =nouveau_gem_object (gem);struct nouveau_drm *drm =nouveau_bdev (nvbo->bo.bdev);struct device *dev = drm->dev->dev; -struct nouveau_vmm *vmm = cli->svm.cli ? &cli->svm : &cli->vmm; +struct nouveau_vmm *vmm =nouveau_cli_vmm (cli);struct nouveau_vma *vma;int ret;
@@ -180,7 +180,7 @@ nouveau_gem_object_close(struct drm_gem_object *gem, struct drm_file *file_priv)struct nouveau_bo *nvbo =nouveau_gem_object (gem);struct nouveau_drm *drm =nouveau_bdev (nvbo->bo.bdev);struct device *dev = drm->dev->dev; -struct nouveau_vmm *vmm = cli->svm.cli ? &cli->svm : & cli->vmm; +struct nouveau_vmm *vmm =nouveau_cli_vmm (cli);struct nouveau_vma *vma;int ret;
@@ -269,7 +269,7 @@ nouveau_gem_info(struct drm_file *file_priv, struct drm_gem_object *gem, {struct nouveau_cli *cli =nouveau_cli (file_priv);struct nouveau_bo *nvbo =nouveau_gem_object (gem); -struct nouveau_vmm *vmm = cli->svm.cli ? &cli->svm : &cli->vmm; +struct nouveau_vmm *vmm =nouveau_cli_vmm (cli);struct nouveau_vma *vma;if (is_power_of_2 (nvbo->valid_domains))
Signature · 1 line
2.41.0
PATCH v8 5/12 drm/nouveau: bo: initialize GEM GPU VA interface
In reply to Danilo KrummrichInitialize the GEM's DRM GPU VA manager interface in preparation for the (u)vmm implementation, provided by subsequent commits, to make use of it.
--- drivers/gpu/drm/nouveau/nouveau_bo.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/gpu/drm/nouveau/nouveau_bo.c +3 -0
@@ -215,11 +215,14 @@ nouveau_bo_alloc(struct nouveau_cli *cli, u64 *size, int *align, u32 domain, nvbo =kzalloc (sizeof (struct nouveau_bo ),GFP_KERNEL );if (!nvbo)return ERR_PTR (-ENOMEM ); +INIT_LIST_HEAD (&nvbo->head);INIT_LIST_HEAD (&nvbo->entry);INIT_LIST_HEAD (&nvbo->vma_list); nvbo->bo.bdev = &drm->ttm.bdev; +drm_gem_gpuva_init (&nvbo->bo.base); +/* This is confusing, and doesn't actually mean we want an uncached * mapping, but is what NOUVEAU_GEM_DOMAIN_COHERENT gets translated * into in nouveau_gem_new().
Signature · 1 line
2.41.0
PATCH v8 6/12 drm/nouveau: move usercopy helpers to nouveau_drv.h
In reply to Danilo KrummrichMove the usercopy helpers to a common driver header file to make it usable for the new API added in subsequent commits.
--- drivers/gpu/drm/nouveau/nouveau_drv.h | 26 ++++++++++++++++++++++++++ drivers/gpu/drm/nouveau/nouveau_gem.c | 26 -------------------------- 2 files changed, 26 insertions(+), 26 deletions(-)
drivers/gpu/drm/nouveau/nouveau_drv.h +26 -0
@@ -130,6 +130,32 @@ nouveau_cli(struct drm_file *fpriv)return fpriv ? fpriv->driver_priv :NULL ; } +static inline void +u_free (void *addr) +{ +kvfree (addr); +} + +static inline void * +u_memcpya (uint64_t user,unsigned nmemb,unsigned size) +{ +void *mem; +void __user *userptr = (void __force __user *)(uintptr_t )user; + + size *= nmemb; + + mem =kvmalloc (size,GFP_KERNEL ); +if (!mem) +return ERR_PTR (-ENOMEM ); + +if (copy_from_user (mem, userptr, size)) { +u_free (mem); +return ERR_PTR (-EFAULT ); + } + +return mem; +} +#include <nvif/object.h> #include <nvif/parent.h>
drivers/gpu/drm/nouveau/nouveau_gem.c +0 -26
@@ -613,32 +613,6 @@ nouveau_gem_pushbuf_validate(struct nouveau_channel *chan,return 0 ; } -static inline void -u_free (void *addr) -{ -kvfree (addr); -} - -static inline void * -u_memcpya (uint64_t user,unsigned nmemb,unsigned size) -{ -void *mem; -void __user *userptr = (void __force __user *)(uintptr_t )user; - - size *= nmemb; - - mem =kvmalloc (size,GFP_KERNEL ); -if (!mem) -return ERR_PTR (-ENOMEM ); - -if (copy_from_user (mem, userptr, size)) { -u_free (mem); -return ERR_PTR (-EFAULT ); - } - -return mem; -} - static int nouveau_gem_pushbuf_reloc_apply(struct nouveau_cli *cli, struct drm_nouveau_gem_pushbuf *req,
Signature · 1 line
2.41.0
PATCH v8 7/12 drm/nouveau: fence: separate fence alloc and emit
In reply to Danilo KrummrichThe new (VM_BIND) UAPI exports DMA fences through DRM syncobjs. Hence, in order to emit fences within DMA fence signalling critical sections (e.g. as typically done in the DRM GPU schedulers run_job() callback) we need to separate fence allocation and fence emitting.
--- drivers/gpu/drm/nouveau/dispnv04/crtc.c | 9 ++++- drivers/gpu/drm/nouveau/nouveau_bo.c | 52 +++++++++++++++---------- drivers/gpu/drm/nouveau/nouveau_chan.c | 6 ++- drivers/gpu/drm/nouveau/nouveau_dmem.c | 9 +++-- drivers/gpu/drm/nouveau/nouveau_fence.c | 16 +++----- drivers/gpu/drm/nouveau/nouveau_fence.h | 3 +- drivers/gpu/drm/nouveau/nouveau_gem.c | 5 ++- 7 files changed, 59 insertions(+), 41 deletions(-)
drivers/gpu/drm/nouveau/dispnv04/crtc.c +8 -1
@@ -1122,11 +1122,18 @@ nv04_page_flip_emit(struct nouveau_channel *chan,PUSH_NVSQ (push,NV_SW ,NV_SW_PAGE_FLIP ,0x00000000 );PUSH_KICK (push); - ret =nouveau_fence_new (chan, false, pfence); + ret =nouveau_fence_new (pfence);if (ret) goto fail; + ret =nouveau_fence_emit (*pfence, chan); +if (ret) + goto fail_fence_unref; +return 0 ; + +fail_fence_unref: +nouveau_fence_unref (pfence); fail:spin_lock_irqsave (&dev->event_lock, flags);list_del (&s->head);
drivers/gpu/drm/nouveau/nouveau_bo.c +31 -21
60 lines not on this page · raw message
drivers/gpu/drm/nouveau/nouveau_chan.c +4 -2
@@ -62,9 +62,11 @@ nouveau_channel_idle(struct nouveau_channel *chan)struct nouveau_fence *fence =NULL ;int ret; - ret =nouveau_fence_new (chan, false, &fence); + ret =nouveau_fence_new (&fence);if (!ret) { - ret =nouveau_fence_wait (fence, false, false); + ret =nouveau_fence_emit (fence, chan); +if (!ret) + ret =nouveau_fence_wait (fence, false, false);nouveau_fence_unref (&fence); }
drivers/gpu/drm/nouveau/nouveau_dmem.c +6 -3
@@ -209,7 +209,8 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) goto done; } -nouveau_fence_new (dmem->migrate.chan, false, &fence); +if (!nouveau_fence_new (&fence)) +nouveau_fence_emit (fence, dmem->migrate.chan);migrate_vma_pages (&args);nouveau_dmem_fence_done (&fence);dma_unmap_page (drm->dev->dev, dma_addr,PAGE_SIZE ,DMA_BIDIRECTIONAL );
@@ -402,7 +403,8 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk) } } -nouveau_fence_new (chunk->drm->dmem->migrate.chan, false, &fence); +if (!nouveau_fence_new (&fence)) +nouveau_fence_emit (fence, chunk->drm->dmem->migrate.chan);migrate_device_pages (src_pfns, dst_pfns, npages);nouveau_dmem_fence_done (&fence);migrate_device_finalize (src_pfns, dst_pfns, npages);
@@ -675,7 +677,8 @@ static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm, addr +=PAGE_SIZE ; } -nouveau_fence_new (drm->dmem->migrate.chan, false, &fence); +if (!nouveau_fence_new (&fence)) +nouveau_fence_emit (fence, chunk->drm->dmem->migrate.chan);migrate_vma_pages (args);nouveau_dmem_fence_done (&fence);nouveau_pfns_map (svmm, args->vma->vm_mm, args->start, pfns, i);
drivers/gpu/drm/nouveau/nouveau_fence.c +5 -11
@@ -210,6 +210,9 @@ nouveau_fence_emit(struct nouveau_fence *fence, struct nouveau_channel *chan)struct nouveau_fence_priv *priv = (void *)chan->drm->fence;int ret; +if (unlikely (!chan->fence)) +return -ENODEV ; + fence->channel = chan; fence->timeout = jiffies + (15 *HZ );
@@ -396,25 +399,16 @@ nouveau_fence_unref(struct nouveau_fence **pfence) }int -nouveau_fence_new (struct nouveau_channel *chan,bool sysmem, -struct nouveau_fence **pfence) +nouveau_fence_new (struct nouveau_fence **pfence) {struct nouveau_fence *fence; -int ret =0 ; - -if (unlikely (!chan->fence)) -return -ENODEV ; fence =kzalloc (sizeof (*fence),GFP_KERNEL );if (!fence)return -ENOMEM ; - ret =nouveau_fence_emit (fence, chan); -if (ret) -nouveau_fence_unref (&fence); - *pfence = fence; -return ret; +return 0 ; }static const char *nouveau_fence_get_get_driver_name (struct dma_fence *fence)
drivers/gpu/drm/nouveau/nouveau_fence.h +1 -2
@@ -17,8 +17,7 @@ struct nouveau_fence {unsigned long timeout; }; -int nouveau_fence_new (struct nouveau_channel *,bool sysmem, -struct nouveau_fence **); +int nouveau_fence_new (struct nouveau_fence **);void nouveau_fence_unref (struct nouveau_fence **);int nouveau_fence_emit (struct nouveau_fence *,struct nouveau_channel *);
drivers/gpu/drm/nouveau/nouveau_gem.c +4 -1
@@ -873,8 +873,11 @@ nouveau_gem_ioctl_pushbuf(struct drm_device *dev, void *data, } } - ret =nouveau_fence_new (chan, false, &fence); + ret =nouveau_fence_new (&fence); +if (!ret) + ret =nouveau_fence_emit (fence, chan);if (ret) { +nouveau_fence_unref (&fence);NV_PRINTK (err, cli,"error fencing pushbuf: %d\n" , ret);WIND_RING (chan); goto out;
Signature · 1 line
2.41.0
PATCH v8 8/12 drm/nouveau: fence: fail to emit when fence context is killed
In reply to Danilo KrummrichThe new VM_BIND UAPI implementation introduced in subsequent commits will allow asynchronous jobs processing push buffers and emitting fences.
If a fence context is killed, e.g. due to a channel fault, jobs which are already queued for execution might still emit new fences. In such a case a job would hang forever.
To fix that, fail to emit a new fence on a killed fence context with -ENODEV to unblock the job.
--- drivers/gpu/drm/nouveau/nouveau_fence.c | 7 +++++++ drivers/gpu/drm/nouveau/nouveau_fence.h | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-)
drivers/gpu/drm/nouveau/nouveau_fence.c +7 -0
@@ -96,6 +96,7 @@ nouveau_fence_context_kill(struct nouveau_fence_chan *fctx, int error)if (nouveau_fence_signal (fence))nvif_event_block (&fctx->event); } + fctx->killed =1 ;spin_unlock_irqrestore (&fctx->lock, flags); }
@@ -229,6 +230,12 @@ nouveau_fence_emit(struct nouveau_fence *fence, struct nouveau_channel *chan)dma_fence_get (&fence->base);spin_lock_irq (&fctx->lock); +if (unlikely (fctx->killed)) { +spin_unlock_irq (&fctx->lock); +dma_fence_put (&fence->base); +return -ENODEV ; + } +if (nouveau_fence_update (chan, fctx))nvif_event_block (&fctx->event);
drivers/gpu/drm/nouveau/nouveau_fence.h +1 -1
@@ -44,7 +44,7 @@ struct nouveau_fence_chan {char name[32 ];struct nvif_event event; -int notify_ref, dead; +int notify_ref, dead, killed; };struct nouveau_fence_priv {
Signature · 1 line
2.41.0
PATCH v8 9/12 drm/nouveau: chan: provide nouveau_channel_kill()
In reply to Danilo KrummrichThe new VM_BIND UAPI implementation introduced in subsequent commits will allow asynchronous jobs processing push buffers and emitting fences.
If a job times out, we need a way to recover from this situation. For now, simply kill the channel to unblock all hung up jobs and signal userspace that the device is dead on the next EXEC or VM_BIND ioctl.
--- drivers/gpu/drm/nouveau/nouveau_chan.c | 14 +++++++++++--- drivers/gpu/drm/nouveau/nouveau_chan.h | 1 + 2 files changed, 12 insertions(+), 3 deletions(-)
drivers/gpu/drm/nouveau/nouveau_chan.c +11 -3
@@ -40,6 +40,14 @@ MODULE_PARM_DESC(vram_pushbuf, "Create DMA push buffers in VRAM");int nouveau_vram_pushbuf;module_param_named (vram_pushbuf, nouveau_vram_pushbuf, int,0400 ); +void +nouveau_channel_kill (struct nouveau_channel *chan) +{ +atomic_set (&chan->killed,1 ); +if (chan->fence) +nouveau_fence_context_kill (chan->fence, -ENODEV ); +} +static int nouveau_channel_killed (struct nvif_event *event,void *repv,u32 repc) {
@@ -47,9 +55,9 @@ nouveau_channel_killed(struct nvif_event *event, void *repv, u32 repc)struct nouveau_cli *cli = (void *)chan->user.client;NV_PRINTK (warn, cli,"channel %d killed!\n" , chan->chid); -atomic_set (&chan->killed,1 ); -if (chan->fence) -nouveau_fence_context_kill (chan->fence, -ENODEV ); + +if (unlikely (!atomic_read (&chan->killed))) +nouveau_channel_kill (chan);return NVIF_EVENT_DROP ; }
drivers/gpu/drm/nouveau/nouveau_chan.h +1 -0
@@ -65,6 +65,7 @@ int nouveau_channel_new(struct nouveau_drm *, struct nvif_device *, bool priv, u32 vram, u32 gart, struct nouveau_channel **); void nouveau_channel_del(struct nouveau_channel **);int nouveau_channel_idle (struct nouveau_channel *); +void nouveau_channel_kill (struct nouveau_channel *);extern int nouveau_vram_pushbuf;
Signature · 1 line
2.41.0
PATCH v8 10/12 drm/nouveau: nvkm/vmm: implement raw ops to manage uvmm
In reply to Danilo KrummrichThe new VM_BIND UAPI uses the DRM GPU VA manager to manage the VA space. Hence, we a need a way to manipulate the MMUs page tables without going through the internal range allocator implemented by nvkm/vmm.
This patch adds a raw interface for nvkm/vmm to pass the resposibility for managing the address space and the corresponding map/unmap/sparse operations to the upper layers.
--- drivers/gpu/drm/nouveau/include/nvif/if000c.h | 26 ++- drivers/gpu/drm/nouveau/include/nvif/vmm.h | 19 +- .../gpu/drm/nouveau/include/nvkm/subdev/mmu.h | 20 +- drivers/gpu/drm/nouveau/nouveau_svm.c | 2 +- drivers/gpu/drm/nouveau/nouveau_vmm.c | 4 +- drivers/gpu/drm/nouveau/nvif/vmm.c | 100 +++++++- .../gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.c | 213 ++++++++++++++++-- drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.c | 197 ++++++++++++---- drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.h | 25 ++ .../drm/nouveau/nvkm/subdev/mmu/vmmgf100.c | 16 +- .../drm/nouveau/nvkm/subdev/mmu/vmmgp100.c | 16 +- .../gpu/drm/nouveau/nvkm/subdev/mmu/vmmnv50.c | 27 ++- 12 files changed, 566 insertions(+), 99 deletions(-)
drivers/gpu/drm/nouveau/include/nvif/if000c.h +25 -1
@@ -3,7 +3,10 @@struct nvif_vmm_v0 {__u8 version;__u8 page_nr; -__u8 managed; +#define NVIF_VMM_V0_TYPE_UNMANAGED 0x00 +#define NVIF_VMM_V0_TYPE_MANAGED 0x01 +#define NVIF_VMM_V0_TYPE_RAW 0x02 +__u8 type;__u8 pad03[5 ];__u64 addr;__u64 size;
@@ -17,6 +20,7 @@ struct nvif_vmm_v0 {#define NVIF_VMM_V0_UNMAP 0x04#define NVIF_VMM_V0_PFNMAP 0x05#define NVIF_VMM_V0_PFNCLR 0x06 +#define NVIF_VMM_V0_RAW 0x07#define NVIF_VMM_V0_MTHD (i) ((i) + 0x80)struct nvif_vmm_page_v0 {
@@ -66,6 +70,26 @@ struct nvif_vmm_unmap_v0 {__u64 addr; }; +struct nvif_vmm_raw_v0 { +__u8 version; +#define NVIF_VMM_RAW_V0_GET 0x0 +#define NVIF_VMM_RAW_V0_PUT 0x1 +#define NVIF_VMM_RAW_V0_MAP 0x2 +#define NVIF_VMM_RAW_V0_UNMAP 0x3 +#define NVIF_VMM_RAW_V0_SPARSE 0x4 +__u8 op; +__u8 sparse; +__u8 ref; +__u8 shift; +__u32 argc; +__u8 pad01[7 ]; +__u64 addr; +__u64 size; +__u64 offset; +__u64 memory; +__u64 argv; +}; +struct nvif_vmm_pfnmap_v0 {__u8 version;__u8 page;
drivers/gpu/drm/nouveau/include/nvif/vmm.h +17 -2
@@ -4,6 +4,12 @@struct nvif_mem ;struct nvif_mmu ; +enum nvif_vmm_type { +UNMANAGED , +MANAGED , +RAW , +}; +enum nvif_vmm_get {ADDR ,PTES ,
@@ -30,8 +36,9 @@ struct nvif_vmm {int page_nr; }; -int nvif_vmm_ctor (struct nvif_mmu *,const char *name,s32 oclass,bool managed, -u64 addr,u64 size,void *argv,u32 argc,struct nvif_vmm *); +int nvif_vmm_ctor (struct nvif_mmu *,const char *name,s32 oclass, +enum nvif_vmm_type ,u64 addr,u64 size,void *argv,u32 argc, +struct nvif_vmm *);void nvif_vmm_dtor (struct nvif_vmm *);int nvif_vmm_get (struct nvif_vmm *,enum nvif_vmm_get ,bool sparse,u8 page,u8 align,u64 size,struct nvif_vma *);
@@ -39,4 +46,12 @@ void nvif_vmm_put(struct nvif_vmm *, struct nvif_vma *);int nvif_vmm_map (struct nvif_vmm *,u64 addr,u64 size,void *argv,u32 argc,struct nvif_mem *,u64 offset);int nvif_vmm_unmap (struct nvif_vmm *,u64 ); + +int nvif_vmm_raw_get (struct nvif_vmm *vmm,u64 addr,u64 size,u8 shift); +int nvif_vmm_raw_put (struct nvif_vmm *vmm,u64 addr,u64 size,u8 shift); +int nvif_vmm_raw_map (struct nvif_vmm *vmm,u64 addr,u64 size,u8 shift, +void *argv,u32 argc,struct nvif_mem *mem,u64 offset); +int nvif_vmm_raw_unmap (struct nvif_vmm *vmm,u64 addr,u64 size, +u8 shift,bool sparse); +int nvif_vmm_raw_sparse (struct nvif_vmm *vmm,u64 addr,u64 size,bool ref);#endif
drivers/gpu/drm/nouveau/include/nvkm/subdev/mmu.h +19 -1
@@ -17,6 +17,7 @@ struct nvkm_vma { bool part:1; /* Region was split from an allocated region by map(). */ bool busy:1; /* Region busy (for temporarily preventing user access). */ bool mapped:1; /* Region contains valid pages. */ + bool no_comp:1; /* Force no memory compression. */struct nvkm_memory *memory;/* Memory currently mapped into VMA. */ struct nvkm_tags *tags;/* Compression tag reference. */ };
@@ -27,10 +28,26 @@ struct nvkm_vmm {const char *name;u32 debug;struct kref kref; -struct mutex mutex; + +struct { +struct mutex vmm; +struct mutex ref; +struct mutex map; + } mutex;u64 start;u64 limit; +struct { +struct { +u64 addr; +u64 size; + } p; +struct { +u64 addr; +u64 size; + } n; +bool raw; + } managed;struct nvkm_vmm_pt *pd;struct list_head join;
@@ -70,6 +87,7 @@ struct nvkm_vmm_map {const struct nvkm_vmm_page *page; +bool no_comp;struct nvkm_tags *tags;u64 next;u64 type;
drivers/gpu/drm/nouveau/nouveau_svm.c +1 -1
@@ -350,7 +350,7 @@ nouveau_svmm_init(struct drm_device *dev, void *data,* VMM instead of the standard one. */ ret = nvif_vmm_ctor(&cli->mmu, "svmVmm", - cli->vmm.vmm.object.oclass, true, + cli->vmm.vmm.object.oclass, MANAGED, args->unmanaged_addr, args->unmanaged_size, &(struct gp100_vmm_v0) { .fault_replay = true,
drivers/gpu/drm/nouveau/nouveau_vmm.c +2 -2
@@ -128,8 +128,8 @@ nouveau_vmm_fini(struct nouveau_vmm *vmm)int nouveau_vmm_init (struct nouveau_cli *cli,s32 oclass,struct nouveau_vmm *vmm) { -int ret =nvif_vmm_ctor (&cli->mmu,"drmVmm" , oclass, false,PAGE_SIZE , -0 ,NULL ,0 , &vmm->vmm); +int ret =nvif_vmm_ctor (&cli->mmu,"drmVmm" , oclass,UNMANAGED , +PAGE_SIZE ,0 ,NULL ,0 , &vmm->vmm);if (ret)return ret;
drivers/gpu/drm/nouveau/nvif/vmm.c +97 -3
120 lines not on this page · raw message
drivers/gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.c +195 -18
324 lines not on this page · raw message
drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.c +152 -45
306 lines not on this page · raw message
drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.h +25 -0
@@ -163,6 +163,7 @@ int nvkm_vmm_new_(const struct nvkm_vmm_func *, struct nvkm_mmu *, u32 pd_header, bool managed, u64 addr, u64 size, struct lock_class_key *, const char *name, struct nvkm_vmm **); +struct nvkm_vma *nvkm_vma_new (u64 addr,u64 size);struct nvkm_vma *nvkm_vmm_node_search (struct nvkm_vmm *,u64 addr);struct nvkm_vma *nvkm_vmm_node_split (struct nvkm_vmm *,struct nvkm_vma *,u64 addr,u64 size);
@@ -173,6 +174,30 @@ void nvkm_vmm_put_locked(struct nvkm_vmm *, struct nvkm_vma *);void nvkm_vmm_unmap_locked (struct nvkm_vmm *,struct nvkm_vma *,bool pfn);void nvkm_vmm_unmap_region (struct nvkm_vmm *,struct nvkm_vma *); +int nvkm_vmm_raw_get (struct nvkm_vmm *vmm,u64 addr,u64 size,u8 refd); +void nvkm_vmm_raw_put (struct nvkm_vmm *vmm,u64 addr,u64 size,u8 refd); +void nvkm_vmm_raw_unmap (struct nvkm_vmm *vmm,u64 addr,u64 size, +bool sparse,u8 refd); +int nvkm_vmm_raw_sparse (struct nvkm_vmm *,u64 addr,u64 size,bool ref); + +static inline bool +nvkm_vmm_in_managed_range (struct nvkm_vmm *vmm,u64 start,u64 size) +{ +u64 p_start = vmm->managed.p.addr; +u64 p_end = p_start + vmm->managed.p.size; +u64 n_start = vmm->managed.n.addr; +u64 n_end = n_start + vmm->managed.n.size; +u64 end = start + size; + +if (start >= p_start && end <= p_end) +return true; + +if (start >= n_start && end <= n_end) +return true; + +return false; +} +#define NVKM_VMM_PFN_ADDR 0xfffffffffffff000ULL#define NVKM_VMM_PFN_ADDR_SHIFT 12#define NVKM_VMM_PFN_APER 0x00000000000000f0ULL
drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgf100.c +9 -7
@@ -287,15 +287,17 @@ gf100_vmm_valid(struct nvkm_vmm *vmm, void *argv, u32 argc,return -EINVAL ; } - ret =nvkm_memory_tags_get (memory, device, tags, - nvkm_ltc_tags_clear, - &map->tags); -if (ret) { -VMM_DEBUG (vmm,"comp %d" , ret); -return ret; +if (!map->no_comp) { + ret =nvkm_memory_tags_get (memory, device, tags, + nvkm_ltc_tags_clear, + &map->tags); +if (ret) { +VMM_DEBUG (vmm,"comp %d" , ret); +return ret; + } } -if (map->tags->mn) { +if (!map->no_comp && map->tags->mn) {u64 tags = map->tags->mn->offset + (map->offset >>17 );if (page->shift ==17 || !gm20x) { map->type |= tags <<44 ;
drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgp100.c +9 -7
@@ -453,15 +453,17 @@ gp100_vmm_valid(struct nvkm_vmm *vmm, void *argv, u32 argc,return -EINVAL ; } - ret =nvkm_memory_tags_get (memory, device, tags, - nvkm_ltc_tags_clear, - &map->tags); -if (ret) { -VMM_DEBUG (vmm,"comp %d" , ret); -return ret; +if (!map->no_comp) { + ret =nvkm_memory_tags_get (memory, device, tags, + nvkm_ltc_tags_clear, + &map->tags); +if (ret) { +VMM_DEBUG (vmm,"comp %d" , ret); +return ret; + } } -if (map->tags->mn) { +if (!map->no_comp && map->tags->mn) { tags = map->tags->mn->offset + (map->offset >>16 ); map->ctag |= ((1ULL << page->shift) >>16 ) <<36 ; map->type |= tags <<36 ;
drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmnv50.c +15 -12
@@ -296,19 +296,22 @@ nv50_vmm_valid(struct nvkm_vmm *vmm, void *argv, u32 argc,return -EINVAL ; } - ret =nvkm_memory_tags_get (memory, device, tags,NULL , - &map->tags); -if (ret) { -VMM_DEBUG (vmm,"comp %d" , ret); -return ret; - } +if (!map->no_comp) { + ret =nvkm_memory_tags_get (memory, device, tags,NULL , + &map->tags); +if (ret) { +VMM_DEBUG (vmm,"comp %d" , ret); +return ret; + } -if (map->tags->mn) { -u32 tags = map->tags->mn->offset + (map->offset >>16 ); - map->ctag |= (u64 )comp <<49 ; - map->type |= (u64 )comp <<47 ; - map->type |= (u64 )tags <<49 ; - map->next |= map->ctag; +if (map->tags->mn) { +u32 tags = map->tags->mn->offset + + (map->offset >>16 ); + map->ctag |= (u64 )comp <<49 ; + map->type |= (u64 )comp <<47 ; + map->type |= (u64 )tags <<49 ; + map->next |= map->ctag; + } } }
Signature · 1 line
2.41.0
PATCH v8 11/12 drm/nouveau: implement new VM_BIND uAPI
In reply to Danilo KrummrichThis commit provides the implementation for the new uapi motivated by the Vulkan API. It allows user mode drivers (UMDs) to:
1) Initialize a GPU virtual address (VA) space via the new DRM_IOCTL_NOUVEAU_VM_INIT ioctl for UMDs to specify the portion of VA space managed by the kernel and userspace, respectively.
2) Allocate and free a VA space region as well as bind and unbind memory to the GPUs VA space via the new DRM_IOCTL_NOUVEAU_VM_BIND ioctl. UMDs can request the named operations to be processed either synchronously or asynchronously. It supports DRM syncobjs (incl. timelines) as synchronization mechanism. The management of the GPU VA mappings is implemented with the DRM GPU VA manager.
3) Execute push buffers with the new DRM_IOCTL_NOUVEAU_EXEC ioctl. The execution happens asynchronously. It supports DRM syncobj (incl. timelines) as synchronization mechanism. DRM GEM object locking is handled with drm_exec.
Both, DRM_IOCTL_NOUVEAU_VM_BIND and DRM_IOCTL_NOUVEAU_EXEC, use the DRM GPU scheduler for the asynchronous paths.
--- Documentation/gpu/driver-uapi.rst | 3 + drivers/gpu/drm/nouveau/Kbuild | 3 + drivers/gpu/drm/nouveau/Kconfig | 2 + drivers/gpu/drm/nouveau/nouveau_abi16.c | 24 + drivers/gpu/drm/nouveau/nouveau_abi16.h | 1 + drivers/gpu/drm/nouveau/nouveau_bo.c | 156 +- drivers/gpu/drm/nouveau/nouveau_bo.h | 2 +- drivers/gpu/drm/nouveau/nouveau_drm.c | 27 +- drivers/gpu/drm/nouveau/nouveau_drv.h | 59 +- drivers/gpu/drm/nouveau/nouveau_exec.c | 414 +++++ drivers/gpu/drm/nouveau/nouveau_exec.h | 54 + drivers/gpu/drm/nouveau/nouveau_gem.c | 25 +- drivers/gpu/drm/nouveau/nouveau_mem.h | 5 + drivers/gpu/drm/nouveau/nouveau_prime.c | 2 +- drivers/gpu/drm/nouveau/nouveau_sched.c | 462 ++++++ drivers/gpu/drm/nouveau/nouveau_sched.h | 123 ++ drivers/gpu/drm/nouveau/nouveau_uvmm.c | 1970 +++++++++++++++++++++++ drivers/gpu/drm/nouveau/nouveau_uvmm.h | 107 ++ 18 files changed, 3372 insertions(+), 67 deletions(-) create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.c create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.h create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.c create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.h create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.c create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.h
Documentation/gpu/driver-uapi.rst +3 -0
@@ -13,4 +13,7 @@ drm/nouveau uAPI VM_BIND / EXEC uAPI ------------------- +.. kernel-doc:: drivers/gpu/drm/nouveau/nouveau_exec.c + :doc: Overview + .. kernel-doc:: include/uapi/drm/nouveau_drm.h
drivers/gpu/drm/nouveau/Kbuild +3 -0
@@ -47,6 +47,9 @@ nouveau-y += nouveau_prime.o nouveau-y += nouveau_sgdma.o nouveau-y += nouveau_ttm.o nouveau-y += nouveau_vmm.o +nouveau-y += nouveau_exec.o +nouveau-y += nouveau_sched.o +nouveau-y += nouveau_uvmm.o # DRM - modesetting nouveau-$(CONFIG_DRM_NOUVEAU_BACKLIGHT) += nouveau_backlight.o
drivers/gpu/drm/nouveau/Kconfig +2 -0
@@ -10,6 +10,8 @@ config DRM_NOUVEAU select DRM_KMS_HELPER select DRM_TTM select DRM_TTM_HELPER + select DRM_EXEC + select DRM_SCHED select I2C select I2C_ALGOBIT select BACKLIGHT_CLASS_DEVICE if DRM_NOUVEAU_BACKLIGHT
drivers/gpu/drm/nouveau/nouveau_abi16.c +24 -0
@@ -35,6 +35,7 @@#include "nouveau_chan.h" #include "nouveau_abi16.h" #include "nouveau_vmm.h" +#include "nouveau_sched.h" static struct nouveau_abi16 *nouveau_abi16 (struct drm_file *file_priv)
@@ -125,6 +126,17 @@ nouveau_abi16_chan_fini(struct nouveau_abi16 *abi16, {struct nouveau_abi16_ntfy *ntfy, *temp; +/* When a client exits without waiting for it's queued up jobs to +* finish it might happen that we fault the channel. This is due to +* drm_file_free() calling drm_gem_release() before the postclose() +* callback. Hence, we can't tear down this scheduler entity before +* uvmm mappings are unmapped. Currently, we can't detect this case. +* +* However, this should be rare and harmless, since the channel isn't +* needed anymore. +*/ +nouveau_sched_entity_fini (&chan->sched_entity); +/* wait for all activity to stop before cleaning up */ if (chan->chan)nouveau_channel_idle (chan->chan);
@@ -261,6 +273,13 @@ nouveau_abi16_ioctl_channel_alloc(ABI16_IOCTL_ARGS)if (!drm->channel)return nouveau_abi16_put (abi16, -ENODEV ); +/* If uvmm wasn't initialized until now disable it completely to prevent +* userspace from mixing up UAPIs. +* +* The client lock is already acquired by nouveau_abi16_get(). +*/ +__nouveau_cli_uvmm_disable (cli); + device = &abi16->device; engine =NV_DEVICE_HOST_RUNLIST_ENGINES_GR ;
@@ -304,6 +323,11 @@ nouveau_abi16_ioctl_channel_alloc(ABI16_IOCTL_ARGS)if (ret) goto done; + ret =nouveau_sched_entity_init (&chan->sched_entity, &drm->sched, + drm->sched_wq); +if (ret) + goto done; + init->channel = chan->chan->chid;if (device->info.family >=NV_DEVICE_INFO_V0_TESLA )
drivers/gpu/drm/nouveau/nouveau_abi16.h +1 -0
@@ -26,6 +26,7 @@ struct nouveau_abi16_chan {struct nouveau_bo *ntfy;struct nouveau_vma *ntfy_vma;struct nvkm_mm heap; +struct nouveau_sched_entity sched_entity; };struct nouveau_abi16 {
drivers/gpu/drm/nouveau/nouveau_bo.c +99 -57
207 lines not on this page · raw message
drivers/gpu/drm/nouveau/nouveau_bo.h +1 -1
@@ -73,7 +73,7 @@ extern struct ttm_device_funcs nouveau_bo_driver;void nouveau_bo_move_init (struct nouveau_drm *);struct nouveau_bo *nouveau_bo_alloc (struct nouveau_cli *,u64 *size,int *align, -u32 domain,u32 tile_mode,u32 tile_flags); +u32 domain,u32 tile_mode,u32 tile_flags,bool internal);int nouveau_bo_init (struct nouveau_bo *,u64 size,int align,u32 domain,struct sg_table *sg,struct dma_resv *robj); int nouveau_bo_new(struct nouveau_cli *, u64 size, int align, u32 domain,
drivers/gpu/drm/nouveau/nouveau_drm.c +25 -2
@@ -68,6 +68,9 @@#include "nouveau_platform.h" #include "nouveau_svm.h" #include "nouveau_dmem.h" +#include "nouveau_exec.h" +#include "nouveau_uvmm.h" +#include "nouveau_sched.h" DECLARE_DYNDBG_CLASSMAP(drm_debug_classes, DD_CLASS_TYPE_DISJOINT_BITS, 0, "DRM_UT_CORE",
@@ -196,6 +199,8 @@ nouveau_cli_fini(struct nouveau_cli *cli)WARN_ON (!list_empty (&cli->worker));usif_client_fini (cli); +nouveau_uvmm_fini (&cli->uvmm); +nouveau_sched_entity_fini (&cli->sched_entity);nouveau_vmm_fini (&cli->svm);nouveau_vmm_fini (&cli->vmm);nvif_mmu_dtor (&cli->mmu);
@@ -301,6 +306,12 @@ nouveau_cli_init(struct nouveau_drm *drm, const char *sname, } cli->mem = &mems[ret]; + + ret =nouveau_sched_entity_init (&cli->sched_entity, &drm->sched, + drm->sched_wq); +if (ret) + goto done; +return 0 ; done:if (ret)
@@ -554,10 +565,14 @@ nouveau_drm_device_init(struct drm_device *dev)nvif_parent_ctor (&nouveau_parent, &drm->parent); drm->master.base.object.parent = &drm->parent; - ret =nouveau_cli_init (drm,"DRM-master" , &drm->master); + ret =nouveau_sched_init (drm);if (ret) goto fail_alloc; + ret =nouveau_cli_init (drm,"DRM-master" , &drm->master); +if (ret) + goto fail_sched; + ret =nouveau_cli_init (drm,"DRM" , &drm->client);if (ret) goto fail_master;
@@ -614,7 +629,6 @@ nouveau_drm_device_init(struct drm_device *dev) }return 0 ; - fail_dispinit:nouveau_display_destroy (dev); fail_dispctor:
@@ -627,6 +641,8 @@ nouveau_drm_device_init(struct drm_device *dev)nouveau_cli_fini (&drm->client); fail_master:nouveau_cli_fini (&drm->master); +fail_sched: +nouveau_sched_fini (drm); fail_alloc:nvif_parent_dtor (&drm->parent);kfree (drm);
@@ -678,6 +694,8 @@ nouveau_drm_device_fini(struct drm_device *dev) }mutex_unlock (&drm->clients_lock); +nouveau_sched_fini (drm); +nouveau_cli_fini (&drm->client);nouveau_cli_fini (&drm->master);nvif_parent_dtor (&drm->parent);
@@ -1179,6 +1197,9 @@ nouveau_ioctls[] = {DRM_IOCTL_DEF_DRV (NOUVEAU_GEM_CPU_PREP , nouveau_gem_ioctl_cpu_prep,DRM_RENDER_ALLOW ),DRM_IOCTL_DEF_DRV (NOUVEAU_GEM_CPU_FINI , nouveau_gem_ioctl_cpu_fini,DRM_RENDER_ALLOW ),DRM_IOCTL_DEF_DRV (NOUVEAU_GEM_INFO , nouveau_gem_ioctl_info,DRM_RENDER_ALLOW ), +DRM_IOCTL_DEF_DRV (NOUVEAU_VM_INIT , nouveau_uvmm_ioctl_vm_init,DRM_RENDER_ALLOW ), +DRM_IOCTL_DEF_DRV (NOUVEAU_VM_BIND , nouveau_uvmm_ioctl_vm_bind,DRM_RENDER_ALLOW ), + DRM_IOCTL_DEF_DRV(NOUVEAU_EXEC, nouveau_exec_ioctl_exec, DRM_RENDER_ALLOW), };long
@@ -1226,6 +1247,8 @@ nouveau_driver_fops = {static struct drm_driver driver_stub = { .driver_features =DRIVER_GEM | +DRIVER_SYNCOBJ |DRIVER_SYNCOBJ_TIMELINE | +DRIVER_GEM_GPUVA |DRIVER_MODESET |DRIVER_RENDER , .open = nouveau_drm_open,
drivers/gpu/drm/nouveau/nouveau_drv.h +57 -2
99 lines not on this page · raw message
drivers/gpu/drm/nouveau/nouveau_exec.c new file +414 -0
414 lines not on this page · raw message
drivers/gpu/drm/nouveau/nouveau_exec.h new file +54 -0
54 lines not on this page · raw message
drivers/gpu/drm/nouveau/nouveau_gem.c +21 -4
@@ -120,7 +120,11 @@ nouveau_gem_object_open(struct drm_gem_object *gem, struct drm_file *file_priv) goto out; } - ret =nouveau_vma_new (nvbo, vmm, &vma); +/* only create a VMA on binding */ +if (!nouveau_cli_uvmm (cli)) + ret =nouveau_vma_new (nvbo, vmm, &vma); +else + ret =0 ;pm_runtime_mark_last_busy (dev);pm_runtime_put_autosuspend (dev); out:
@@ -187,6 +191,9 @@ nouveau_gem_object_close(struct drm_gem_object *gem, struct drm_file *file_priv)if (vmm->vmm.object.oclass <NVIF_CLASS_VMM_NV50 )return ; +if (nouveau_cli_uvmm (cli)) +return ; + ret =ttm_bo_reserve (&nvbo->bo, false, false,NULL );if (ret)return ;
@@ -231,7 +238,7 @@ nouveau_gem_new(struct nouveau_cli *cli, u64 size, int align, uint32_t domain, domain |=NOUVEAU_GEM_DOMAIN_CPU ; nvbo =nouveau_bo_alloc (cli, &size, &align, domain, tile_mode, - tile_flags); + tile_flags, false);if (IS_ERR (nvbo))return PTR_ERR (nvbo);
@@ -279,13 +286,15 @@ nouveau_gem_info(struct drm_file *file_priv, struct drm_gem_object *gem,else rep->domain =NOUVEAU_GEM_DOMAIN_VRAM ; rep->offset = nvbo->offset; -if (vmm->vmm.object.oclass >=NVIF_CLASS_VMM_NV50 ) { +if (vmm->vmm.object.oclass >=NVIF_CLASS_VMM_NV50 && + !nouveau_cli_uvmm (cli)) { vma =nouveau_vma_find (nvbo, vmm);if (!vma)return -EINVAL ; rep->offset = vma->addr; - } + }else + rep->offset =0 ; rep->size = nvbo->bo.base.size; rep->map_handle =drm_vma_node_offset_addr (&nvbo->bo.base.vma_node);
@@ -310,6 +319,11 @@ nouveau_gem_ioctl_new(struct drm_device *dev, void *data,struct nouveau_bo *nvbo =NULL ;int ret =0 ; +/* If uvmm wasn't initialized until now disable it completely to prevent +* userspace from mixing up UAPIs. +*/ +nouveau_cli_uvmm_disable (cli); + ret =nouveau_gem_new (cli, req->info.size, req->align, req->info.domain, req->info.tile_mode, req->info.tile_flags, &nvbo);
@@ -721,6 +735,9 @@ nouveau_gem_ioctl_pushbuf(struct drm_device *dev, void *data,if (unlikely (!abi16))return -ENOMEM ; +if (unlikely (nouveau_cli_uvmm (cli))) +return -ENOSYS ; +list_for_each_entry (temp, &abi16->channels, head) {if (temp->chan->chid == req->channel) { chan = temp->chan;
drivers/gpu/drm/nouveau/nouveau_mem.h +5 -0
@@ -35,4 +35,9 @@ int nouveau_mem_vram(struct ttm_resource *, bool contig, u8 page);int nouveau_mem_host (struct ttm_resource *,struct ttm_tt *);void nouveau_mem_fini (struct nouveau_mem *);int nouveau_mem_map (struct nouveau_mem *,struct nvif_vmm *,struct nvif_vma *); +int +nouveau_mem_map_fixed (struct nouveau_mem *mem, +struct nvif_vmm *vmm, +u8 kind,u64 addr, +u64 offset,u64 range);#endif
drivers/gpu/drm/nouveau/nouveau_prime.c +1 -1
@@ -50,7 +50,7 @@ struct drm_gem_object *nouveau_gem_prime_import_sg_table(struct drm_device *dev,dma_resv_lock (robj,NULL ); nvbo =nouveau_bo_alloc (&drm->client, &size, &align, -NOUVEAU_GEM_DOMAIN_GART ,0 ,0 ); +NOUVEAU_GEM_DOMAIN_GART ,0 ,0 , true);if (IS_ERR (nvbo)) { obj =ERR_CAST (nvbo); goto unlock;
drivers/gpu/drm/nouveau/nouveau_sched.c new file +462 -0
462 lines not on this page · raw message
drivers/gpu/drm/nouveau/nouveau_sched.h new file +123 -0
123 lines not on this page · raw message
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +1970 -0
1,387 lines not on this page · raw message
drivers/gpu/drm/nouveau/nouveau_uvmm.h new file +107 -0
107 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.41.0
Part of the diff is not shown here. View the raw message.
Re: [PATCH drm-misc-next v8 11/12] drm/nouveau: implement new VM_BIND uAPI
Quoted 14 lines · Danilo Krummrich
This commit provides the implementation for the new uapi motivated by the Vulkan API. It allows user mode drivers (UMDs) to:
1) Initialize a GPU virtual address (VA) space via the new
DRM_IOCTL_NOUVEAU_VM_INITioctl for UMDs to specify the portion of VA space managed by the kernel and userspace, respectively.2) Allocate and free a VA space region as well as bind and unbind memory to the GPUs VA space via the new
DRM_IOCTL_NOUVEAU_VM_BINDioctl. UMDs can request the named operations to be processed either synchronously or asynchronously. It supports DRM syncobjs (incl. timelines) as synchronization mechanism. The management of the GPU VA mappings is implemented with the DRM GPU VA manager.
Quoted 8 lines
3) Execute push buffers with the new
DRM_IOCTL_NOUVEAU_EXECioctl. The execution happens asynchronously. It supports DRM syncobj (incl. timelines) as synchronization mechanism. DRM GEM object locking is handled withdrm_exec.Both,
DRM_IOCTL_NOUVEAU_VM_BINDandDRM_IOCTL_NOUVEAU_EXEC, use the DRM GPU scheduler for the asynchronous paths.
IDK where the best place to talk about this is but this seems as good as any.
I've been looking into why the Vulkan CTS runs about 2x slower for me on the new UAPI and I created a little benchmark to facilitate testing:
https://gitlab.freedesktop.org/mesa/crucible/-/merge_requests/141
The test, roughly, does the following:
1. Allocates and binds 1000 BOs
2. Constructs a pushbuf that executes a no-op compute shader.
3. Does a single EXEC/wait combo to warm up the kernel
4. Loops 10,000 times, doing SYNCOBJ_RESET (fast), EXEC, and thenSYNCOBJ_WAIT and times the loop
Of course, there's a bit of userspace driver overhead but that's negledgable.
If you drop the top patch which allocates 1k buffers, the submit time on the old uAPI is 54 us/exec vs. 66 us/exec on the new UAPI. This includes the time to do a SYNCOBJ_RESET (fast), EXEC, and SYNCOBJ_WAIT. The Intel driver, by comparison, is 33us/exec so it's not syncobj overhead. This is a bit concerning (you'd think the new thing would be faster) but what really has me concerned is the 1k buffer case.
If you include the top patch in the crucible MR, it allocates 1000 BOs and VM_BINDs them. All the binding is done before the warmup EXEC. Suddenly, the submit time jumps to 257 us/exec with the new UAPI. The old UAPI is much worse (1134 us/exec) but that's not the point. Once we've done the first EXEC and created our VM bindings, the cost per EXEC shouldn't change at all based on the number of BOs bound. Part of the point of VM_BIND is to get all that binding logic and BO walking off the EXEC path.
Normally, I wouldn't be too worried about a little performance problem like this. This is the first implementation and we can improve it later. I get that. However, I suspect the solution to this problem involves more UAPI and I want to make sure we have it all before we call this all done and dusted and land it.
The way AMD solves this problem as well as the new Xe driver for Intel is to have a concept of internal vs. external BOs. Basically, there's an INTERNAL bit specified somewhere in BO creation that has a few userspace implications:
1. In the Xe world where VMs are objects, INTERNAL BOs are assigned a VM
on creation and can never be bound to any other VM.
2. Any attempt to export an INTERNAL BO via prime or a similar mechanism
will fail with -EINVAL (I think?).
Inside the kernel driver, all the internal BOs on a VM (or DRM file in the case of nouveau/AMD since they don't have VM objects) share a single dma_resv which allows you to avoid having to walk lists of BOs and take locks on every exec. Instead, you can just look at the fences on the dma_resv for the VM. There's still a BO list associated with the VM for external BOs but, in most Vulkan applications, there are less than a half dozen external BOs total. Meanwhile, the hundreds or thousands of BOs used entirely internally to the application basically count as one BO when it comes to locking overhead.
I'm not actually trying to dictate kernel driver design here. If one dma_resv doesn't work, fine. I don't care as long as EXEC scales. However, given that the solution in all the other drivers involves a BO create flag nouveau doesn't have, we need to either add that or prove that we can get EXEC to scale without it.
~Faith
Quoted 4,053 lines
--- Documentation/gpu/driver-uapi.rst | 3 + drivers/gpu/drm/nouveau/Kbuild | 3 + drivers/gpu/drm/nouveau/Kconfig | 2 + drivers/gpu/drm/nouveau/nouveau_abi16.c | 24 + drivers/gpu/drm/nouveau/nouveau_abi16.h | 1 + drivers/gpu/drm/nouveau/nouveau_bo.c | 156 +- drivers/gpu/drm/nouveau/nouveau_bo.h | 2 +- drivers/gpu/drm/nouveau/nouveau_drm.c | 27 +- drivers/gpu/drm/nouveau/nouveau_drv.h | 59 +- drivers/gpu/drm/nouveau/nouveau_exec.c | 414 +++++ drivers/gpu/drm/nouveau/nouveau_exec.h | 54 + drivers/gpu/drm/nouveau/nouveau_gem.c | 25 +- drivers/gpu/drm/nouveau/nouveau_mem.h | 5 + drivers/gpu/drm/nouveau/nouveau_prime.c | 2 +- drivers/gpu/drm/nouveau/nouveau_sched.c | 462 ++++++ drivers/gpu/drm/nouveau/nouveau_sched.h | 123 ++ drivers/gpu/drm/nouveau/nouveau_uvmm.c | 1970 +++++++++++++++++++++++ drivers/gpu/drm/nouveau/nouveau_uvmm.h | 107 ++ 18 files changed, 3372 insertions(+), 67 deletions(-) create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.c create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.h create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.c create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.h create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.c create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.hDocumentation/gpu/driver-uapi.rst +3
-0@@ -13,4 +13,7 @@ drm/nouveau uAPI VM_BIND / EXEC uAPI ------------------- +.. kernel-doc:: drivers/gpu/drm/nouveau/nouveau_exec.c + :doc: Overview + .. kernel-doc:: include/uapi/drm/nouveau_drm.hdrivers/gpu/drm/nouveau/Kbuild +3
-0@@ -47,6 +47,9 @@ nouveau-y += nouveau_prime.o nouveau-y += nouveau_sgdma.o nouveau-y += nouveau_ttm.o nouveau-y += nouveau_vmm.o +nouveau-y += nouveau_exec.o +nouveau-y += nouveau_sched.o +nouveau-y += nouveau_uvmm.o # DRM - modesetting nouveau-$(CONFIG_DRM_NOUVEAU_BACKLIGHT) += nouveau_backlight.odrivers/gpu/drm/nouveau/Kconfig +2
-0@@ -10,6 +10,8 @@ config DRM_NOUVEAU select DRM_KMS_HELPER select DRM_TTM select DRM_TTM_HELPER + select DRM_EXEC + select DRM_SCHED select I2C select I2C_ALGOBIT select BACKLIGHT_CLASS_DEVICE if DRM_NOUVEAU_BACKLIGHTdrivers/gpu/drm/nouveau/nouveau_abi16.c +6
-0@@ -35,6 +35,7 @@#include "nouveau_chan.h" #include "nouveau_abi16.h" #include "nouveau_vmm.h" +#include "nouveau_sched.h" static struct nouveau_abi16 *nouveau_abi16 (struct drm_file *file_priv)@@ -125,6 +126,17 @@ nouveau_abi16_chan_fini(struct nouveau_abi16 *abi16, {struct nouveau_abi16_ntfy *ntfy, *temp; +/* When a client exits without waiting for it's queued up jobs to +* finish it might happen that we fault the channel. This is due to +* drm_file_free() calling drm_gem_release() before the postclose() +* callback. Hence, we can't tear down this scheduler entity before +* uvmm mappings are unmapped. Currently, we can't detect this case.
drivers/gpu/drm/nouveau/nouveau_abi16.c +2
-0+* +* However, this should be rare and harmless, since the channel isn't
drivers/gpu/drm/nouveau/nouveau_abi16.c +5
-0+* needed anymore. +*/ +nouveau_sched_entity_fini (&chan->sched_entity); +/* wait for all activity to stop before cleaning up */ if (chan->chan)nouveau_channel_idle (chan->chan);@@ -261,6 +273,13 @@ nouveau_abi16_ioctl_channel_alloc(ABI16_IOCTL_ARGS)if (!drm->channel)return nouveau_abi16_put (abi16, -ENODEV ); +/* If uvmm wasn't initialized until now disable it completely to prevent
drivers/gpu/drm/nouveau/nouveau_abi16.c +11
-0+* userspace from mixing up UAPIs. +* +* The client lock is already acquired by nouveau_abi16_get(). +*/ +__nouveau_cli_uvmm_disable (cli); + device = &abi16->device; engine =NV_DEVICE_HOST_RUNLIST_ENGINES_GR ;@@ -304,6 +323,11 @@ nouveau_abi16_ioctl_channel_alloc(ABI16_IOCTL_ARGS)if (ret) goto done; + ret =nouveau_sched_entity_init (&chan->sched_entity, &drm->sched, + drm->sched_wq); +if (ret) + goto done; + init->channel = chan->chan->chid;if (device->info.family >=NV_DEVICE_INFO_V0_TESLA )drivers/gpu/drm/nouveau/nouveau_abi16.h +1
-0@@ -26,6 +26,7 @@ struct nouveau_abi16_chan {struct nouveau_bo *ntfy;struct nouveau_vma *ntfy_vma;struct nvkm_mm heap; +struct nouveau_sched_entity sched_entity; };struct nouveau_abi16 {drivers/gpu/drm/nouveau/nouveau_bo.c +1
-1@@ -199,7 +199,7 @@ nouveau_bo_fixup_align(struct nouveau_bo *nvbo, int *align, u64 *size)struct nouveau_bo *nouveau_bo_alloc (struct nouveau_cli *cli,u64 *size,int *align,u32 domain, -u32 tile_mode,u32 tile_flags) +u32 tile_mode,u32 tile_flags,bool internal) {struct nouveau_drm *drm = cli->drm; struct nouveau_bo *nvbo;drivers/gpu/drm/nouveau/nouveau_bo.c +0
-2@@ -221,8 +221,6 @@ nouveau_bo_alloc(struct nouveau_cli *cli, u64 *size, int *align, u32 domain, INIT_LIST_HEAD(&nvbo->vma_list); nvbo->bo.bdev = &drm->ttm.bdev; -drm_gem_gpuva_init (&nvbo->bo.base); -/* This is confusing, and doesn't actually mean we want an uncached * mapping, but is what NOUVEAU_GEM_DOMAIN_COHERENT gets translated * into in
nouveau_gem_new().drivers/gpu/drm/nouveau/nouveau_bo.c +24
-5@@ -235,68 +233,103 @@ nouveau_bo_alloc(struct nouveau_cli *cli, u64 *size, int *align, u32 domain, nvbo->force_coherent = true; } -if (cli->device.info.family >=NV_DEVICE_INFO_V0_FERMI ) { - nvbo->kind = (tile_flags &0x0000ff00 ) >>8 ; -if (!nvif_mmu_kind_valid (mmu, nvbo->kind)) { -kfree (nvbo); -return ERR_PTR (-EINVAL ); + nvbo->contig = !(tile_flags &NOUVEAU_GEM_TILE_NONCONTIG ); +if (!nouveau_cli_uvmm (cli) || internal) { +/* for BO noVM allocs, don't assign kinds */ +if (cli->device.info.family >=NV_DEVICE_INFO_V0_FERMI ) { + nvbo->kind = (tile_flags &0x0000ff00 ) >>8 ; +if (!nvif_mmu_kind_valid (mmu, nvbo->kind)) { +kfree (nvbo); +return ERR_PTR (-EINVAL ); + } + + nvbo->comp = mmu->kind[nvbo->kind] != nvbo->kind; + }else if (cli->device.info.family >=NV_DEVICE_INFO_V0_TESLA ) { + nvbo->kind = (tile_flags &0x00007f00 ) >>8 ; + nvbo->comp = (tile_flags &0x00030000 ) >>16 ; +if (!nvif_mmu_kind_valid (mmu, nvbo->kind)) { +kfree (nvbo); +return ERR_PTR (-EINVAL ); + } + }else { + nvbo->zeta = (tile_flags &0x00000007 ); + } + nvbo->mode = tile_mode; + +/* Determine the desirable target GPU page size for the buffer. */
drivers/gpu/drm/nouveau/nouveau_bo.c +2
-0+for (i =0 ; i < vmm->page_nr; i++) { +/* Because we cannot currently allow VMM maps to fail
+ * during buffer migration, we need to determine
page
+ * size for the buffer up-front, and pre-allocate
itsdrivers/gpu/drm/nouveau/nouveau_bo.c +3
-0+* page tables. +* +* Skip page sizes that can't support needed domains.
drivers/gpu/drm/nouveau/nouveau_bo.c +2
-0+*/ + if (cli->device.info.family >
NV_DEVICE_INFO_V0_CURIE&&
+ (domain &NOUVEAU_GEM_DOMAIN_VRAM) &&
!vmm->page[i].vram)drivers/gpu/drm/nouveau/nouveau_bo.c +3
-0+continue ; + if ((domain & NOUVEAU_GEM_DOMAIN_GART) && + (!vmm->page[i].host || vmm->page[i].shift >
PAGE_SHIFT))drivers/gpu/drm/nouveau/nouveau_bo.c +3
-0+continue ; + +/* Select this page size if it's the first that supports
+ * the potential memory domains, or when it's
compatibledrivers/gpu/drm/nouveau/nouveau_bo.c +6
-0+* with the requested compression settings. +*/ +if (pi <0 || !nvbo->comp || vmm->page[i].comp) + pi = i; + +/* Stop once the buffer is larger than the current page size. */
drivers/gpu/drm/nouveau/nouveau_bo.c +3
-22+if (*size >=1ULL << vmm->page[i].shift) +break ; } - nvbo->comp = mmu->kind[nvbo->kind] != nvbo->kind; - }else -if (cli->device.info.family >=NV_DEVICE_INFO_V0_TESLA ) { - nvbo->kind = (tile_flags &0x00007f00 ) >>8 ; - nvbo->comp = (tile_flags &0x00030000 ) >>16 ; -if (!nvif_mmu_kind_valid (mmu, nvbo->kind)) { +if (WARN_ON (pi <0 )) {kfree (nvbo);return ERR_PTR (-EINVAL ); } - }else { - nvbo->zeta = (tile_flags &0x00000007 ); - } - nvbo->mode = tile_mode; - nvbo->contig = !(tile_flags &NOUVEAU_GEM_TILE_NONCONTIG ); -/* Determine the desirable target GPU page size for the buffer. */ -for (i =0 ; i < vmm->page_nr; i++) { -/* Because we cannot currently allow VMM maps to fail -* during buffer migration, we need to determine page -* size for the buffer up-front, and pre-allocate its -* page tables. -* -* Skip page sizes that can't support needed domains. -*/ - if (cli->device.info.family > NV_DEVICE_INFO_V0_CURIE && - (domain & NOUVEAU_GEM_DOMAIN_VRAM) &&!vmm->page[i].vram)
drivers/gpu/drm/nouveau/nouveau_bo.c +0
-3-continue ; - if ((domain & NOUVEAU_GEM_DOMAIN_GART) && - (!vmm->page[i].host || vmm->page[i].shift >
PAGE_SHIFT))drivers/gpu/drm/nouveau/nouveau_bo.c +0
-10-continue ; - -/* Select this page size if it's the first that supports -* the potential memory domains, or when it's compatible -* with the requested compression settings. -*/ -if (pi <0 || !nvbo->comp || vmm->page[i].comp) - pi = i; - -/* Stop once the buffer is larger than the current page size. */
drivers/gpu/drm/nouveau/nouveau_bo.c +1
-3-if (*size >=1ULL << vmm->page[i].shift) -break ; - } +/* Disable compression if suitable settings couldn't be found. */
drivers/gpu/drm/nouveau/nouveau_bo.c +13
-4+if (nvbo->comp && !vmm->page[pi].comp) { +if (mmu->object.oclass >=NVIF_CLASS_MMU_GF100 ) + nvbo->kind = mmu->kind[nvbo->kind]; + nvbo->comp =0 ; + } + nvbo->page = vmm->page[pi].shift; + }else { +/* reject other tile flags when in VM mode. */ +if (tile_mode) +return ERR_PTR (-EINVAL ); +if (tile_flags & ~NOUVEAU_GEM_TILE_NONCONTIG ) +return ERR_PTR (-EINVAL ); -if (WARN_ON (pi <0 )) { -kfree (nvbo); -return ERR_PTR (-EINVAL ); - } +/* Determine the desirable target GPU page size for the buffer. */
drivers/gpu/drm/nouveau/nouveau_bo.c +2
-0+for (i =0 ; i < vmm->page_nr; i++) { +/* Because we cannot currently allow VMM maps to fail
+ * during buffer migration, we need to determine
page
+ * size for the buffer up-front, and pre-allocate
itsdrivers/gpu/drm/nouveau/nouveau_bo.c +3
-0+* page tables. +* +* Skip page sizes that can't support needed domains.
drivers/gpu/drm/nouveau/nouveau_bo.c +2
-0+*/ + if ((domain & NOUVEAU_GEM_DOMAIN_VRAM) &&!vmm->page[i].vram)
drivers/gpu/drm/nouveau/nouveau_bo.c +3
-0+continue ; + if ((domain & NOUVEAU_GEM_DOMAIN_GART) && + (!vmm->page[i].host || vmm->page[i].shift >
PAGE_SHIFT))drivers/gpu/drm/nouveau/nouveau_bo.c +4
-5+continue ; -/* Disable compression if suitable settings couldn't be found. */ -if (nvbo->comp && !vmm->page[pi].comp) { -if (mmu->object.oclass >=NVIF_CLASS_MMU_GF100 ) - nvbo->kind = mmu->kind[nvbo->kind]; - nvbo->comp =0 ; +if (pi <0 ) + pi = i; +/* Stop once the buffer is larger than the current page size. */
drivers/gpu/drm/nouveau/nouveau_bo.c +9
-1+if (*size >=1ULL << vmm->page[i].shift) +break ; + } +if (WARN_ON (pi <0 )) { +kfree (nvbo); +return ERR_PTR (-EINVAL ); + } + nvbo->page = vmm->page[pi].shift; } - nvbo->page = vmm->page[pi].shift;nouveau_bo_fixup_align (nvbo, align, size);@@ -313,6 +346,13 @@ nouveau_bo_init(struct nouveau_bo *nvbo, u64 size, int align, u32 domain, nouveau_bo_placement_set(nvbo, domain, 0);INIT_LIST_HEAD (&nvbo->io_reserve_lru); +/* At this point we're guaranteed the dma-resv of the
drm_gem_objectisdrivers/gpu/drm/nouveau/nouveau_bo.c +2
-0+* initialized. We also can't do it after ttm_bo_init_validate(), +* because during validation we might receive a bo_move() callback where
drivers/gpu/drm/nouveau/nouveau_bo.c +4
-0+* we already need to iterate the GEMs GPUVA list. +*/ +drm_gem_gpuva_init (&nvbo->bo.base); + ret = ttm_bo_init_validate(nvbo->bo.bdev, &nvbo->bo, type, &nvbo->placement, align >> PAGE_SHIFT,false,
sg, robj, nouveau_bo_del_ttm);drivers/gpu/drm/nouveau/nouveau_bo.c +2
-1@@ -334,7 +374,7 @@ nouveau_bo_new(struct nouveau_cli *cli, u64 size, int align, int ret; nvbo =nouveau_bo_alloc (cli, &size, &align, domain, tile_mode, - tile_flags); + tile_flags, true);if (IS_ERR (nvbo))return PTR_ERR (nvbo);@@ -948,6 +988,7 @@ static void nouveau_bo_move_ntfy(struct ttm_buffer_object *bo, list_for_each_entry(vma, &nvbo->vma_list, head) {nouveau_vma_map (vma, mem); } +nouveau_uvmm_bo_map_all (nvbo, mem); }else {list_for_each_entry (vma, &nvbo->vma_list, head) {ret = dma_resv_wait_timeout(bo->base.resv,drivers/gpu/drm/nouveau/nouveau_bo.c +1
-0@@ -956,6 +997,7 @@ static void nouveau_bo_move_ntfy(struct ttm_buffer_object *bo,WARN_ON (ret <=0 );nouveau_vma_unmap (vma); } +nouveau_uvmm_bo_unmap_all (nvbo); }if (
new_reg)drivers/gpu/drm/nouveau/nouveau_bo.h +1
-1@@ -73,7 +73,7 @@ extern struct ttm_device_funcs nouveau_bo_driver;void nouveau_bo_move_init (struct nouveau_drm *);struct nouveau_bo *nouveau_bo_alloc (struct nouveau_cli *,u64 *size,int *align, -u32 domain,u32 tile_mode,u32 tile_flags ); + u32 domain, u32 tile_mode, u32 tile_flags, bool internal);int nouveau_bo_init(struct
nouveau_bo*, u64 size, int align, u32 domain,struct sg_table *sg, struct dma_resv *robj);int nouveau_bo_new(struct
nouveau_cli*, u64 size, int align, u32 domain,drivers/gpu/drm/nouveau/nouveau_drm.c +11
-0@@ -68,6 +68,9 @@#include "nouveau_platform.h" #include "nouveau_svm.h" #include "nouveau_dmem.h" +#include "nouveau_exec.h" +#include "nouveau_uvmm.h" +#include "nouveau_sched.h" DECLARE_DYNDBG_CLASSMAP(drm_debug_classes, DD_CLASS_TYPE_DISJOINT_BITS, 0, "DRM_UT_CORE",@@ -196,6 +199,8 @@ nouveau_cli_fini(struct nouveau_cli *cli)WARN_ON (!list_empty (&cli->worker));usif_client_fini (cli); +nouveau_uvmm_fini (&cli->uvmm); +nouveau_sched_entity_fini (&cli->sched_entity);nouveau_vmm_fini (&cli->svm);nouveau_vmm_fini (&cli->vmm);nvif_mmu_dtor (&cli->mmu);@@ -301,6 +306,12 @@ nouveau_cli_init(struct nouveau_drm *drm, const char *sname, } cli->mem = &mems[ret]; + + ret =nouveau_sched_entity_init (&cli->sched_entity, &drm->sched, + drm->sched_wq); +if (ret) + goto done; +return 0 ; done:if (ret)
drivers/gpu/drm/nouveau/nouveau_drm.c +10
-2@@ -554,10 +565,14 @@ nouveau_drm_device_init(struct drm_device *dev)nvif_parent_ctor (&nouveau_parent, &drm->parent); drm->master.base.object.parent = &drm->parent; - ret =nouveau_cli_init (drm,"DRM-master" , &drm->master); + ret =nouveau_sched_init (drm);if (ret) goto fail_alloc; + ret =nouveau_cli_init (drm,"DRM-master" , &drm->master); +if (ret) + goto fail_sched; + ret =nouveau_cli_init (drm,"DRM" , &drm->client);if (ret) goto fail_master;@@ -614,7 +629,6 @@ nouveau_drm_device_init(struct drm_device *dev) }return 0 ; - fail_dispinit:nouveau_display_destroy (dev); fail_dispctor:@@ -627,6 +641,8 @@ nouveau_drm_device_init(struct drm_device *dev)nouveau_cli_fini (&drm->client); fail_master:nouveau_cli_fini (&drm->master); +fail_sched: +nouveau_sched_fini (drm); fail_alloc:nvif_parent_dtor (&drm->parent);kfree (drm);@@ -678,6 +694,8 @@ nouveau_drm_device_fini(struct drm_device *dev) }mutex_unlock (&drm->clients_lock); +nouveau_sched_fini (drm); +nouveau_cli_fini (&drm->client);nouveau_cli_fini (&drm->master);nvif_parent_dtor (&drm->parent);@@ -1179,6 +1197,9 @@ nouveau_ioctls[] = {DRM_IOCTL_DEF_DRV (NOUVEAU_GEM_CPU_PREP , nouveau_gem_ioctl_cpu_prep,DRM_RENDER_ALLOW ),DRM_IOCTL_DEF_DRV (NOUVEAU_GEM_CPU_FINI , nouveau_gem_ioctl_cpu_fini,DRM_RENDER_ALLOW ),DRM_IOCTL_DEF_DRV (NOUVEAU_GEM_INFO , nouveau_gem_ioctl_info,DRM_RENDER_ALLOW ), + DRM_IOCTL_DEF_DRV(NOUVEAU_VM_INIT, nouveau_uvmm_ioctl_vm_init,
DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(NOUVEAU_VM_BIND,nouveau_uvmm_ioctl_vm_bind,DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(NOUVEAU_EXEC,nouveau_exec_ioctl_exec,DRM_RENDER_ALLOW),
};long
drivers/gpu/drm/nouveau/nouveau_drm.c +2
-0@@ -1226,6 +1247,8 @@ nouveau_driver_fops = {static struct drm_driver driver_stub = { .driver_features =DRIVER_GEM | +DRIVER_SYNCOBJ |DRIVER_SYNCOBJ_TIMELINE | +DRIVER_GEM_GPUVA |DRIVER_MODESET |DRIVER_RENDER , .open = nouveau_drm_open,drivers/gpu/drm/nouveau/nouveau_drv.h +57
-2@@ -10,8 +10,8 @@#define DRIVER_DATE "20120801"#define DRIVER_MAJOR 1 -#define DRIVER_MINOR 3 -#define DRIVER_PATCHLEVEL 1 +#define DRIVER_MINOR 4 +#define DRIVER_PATCHLEVEL 0/* * 1.1.1: @@ -63,7 +63,9 @@ struct platform_device;#include "nouveau_fence.h" #include "nouveau_bios.h" +#include "nouveau_sched.h" #include "nouveau_vmm.h" +#include "nouveau_uvmm.h" struct nouveau_drm_tile {struct nouveau_fence *fence;@@ -91,6 +93,10 @@ struct nouveau_cli {struct nvif_mmu mmu;struct nouveau_vmm vmm;struct nouveau_vmm svm; +struct nouveau_uvmm uvmm; + +struct nouveau_sched_entity sched_entity; +const struct nvif_mclass *mem;struct list_head head;@@ -112,15 +118,60 @@ struct nouveau_cli_work {struct dma_fence_cb cb; }; +static inline struct nouveau_uvmm * +nouveau_cli_uvmm (struct nouveau_cli *cli) +{ +if (!cli || !cli->uvmm.vmm.cli) +return NULL ; + +return &cli->uvmm; +} + +static inline struct nouveau_uvmm * +nouveau_cli_uvmm_locked (struct nouveau_cli *cli) +{ +struct nouveau_uvmm *uvmm; + +mutex_lock (&cli->mutex); + uvmm =nouveau_cli_uvmm (cli); +mutex_unlock (&cli->mutex); + +return uvmm; +} +static inline struct nouveau_vmm *nouveau_cli_vmm (struct nouveau_cli *cli) { +struct nouveau_uvmm *uvmm; + + uvmm =nouveau_cli_uvmm (cli); +if (uvmm) +return &uvmm->vmm; +if (cli->svm.cli)return &cli->svm;return &cli->vmm; } +static inline void +__nouveau_cli_uvmm_disable (struct nouveau_cli *cli) +{ +struct nouveau_uvmm *uvmm; + + uvmm =nouveau_cli_uvmm (cli); +if (!uvmm) + cli->uvmm.disabled = true; +} + +static inline void +nouveau_cli_uvmm_disable (struct nouveau_cli *cli) +{ +mutex_lock (&cli->mutex); +__nouveau_cli_uvmm_disable (cli); +mutex_unlock (&cli->mutex); +} +void nouveau_cli_work_queue (struct nouveau_cli *,struct dma_fence *,struct nouveau_cli_work *);@@ -257,6 +308,10 @@ struct nouveau_drm {struct mutex lock;bool component_registered; } audio; + +struct drm_gpu_scheduler sched; +struct workqueue_struct *sched_wq; + }; static inline struct nouveau_drm *drivers/gpu/drm/nouveau/nouveau_exec.c new file +6
-0@@ -0,0 +1,414 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright (c) 2022 Red Hat. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors:
+ * Danilo Krummrich <[email protected]>
+ *
+ */drivers/gpu/drm/nouveau/nouveau_exec.c new file +17
-0+ +#include <drm/drm_exec.h> + +#include "nouveau_drv.h" +#include "nouveau_gem.h" +#include "nouveau_mem.h" +#include "nouveau_dma.h" +#include "nouveau_exec.h" +#include "nouveau_abi16.h" +#include "nouveau_chan.h" +#include "nouveau_sched.h" +#include "nouveau_uvmm.h" + +/** +* DOC: Overview +* +* Nouveau's VM_BIND / EXEC UAPI consists of three ioctls:
DRM_NOUVEAU_VM_INIT,
+ *DRM_NOUVEAU_VM_BINDandDRM_NOUVEAU_EXEC.
+ *
+ * In order to use the UAPI firstly a user client must initialize the VA space
+ * using theDRM_NOUVEAU_VM_INITioctl specifying which region of the VA space
+ * should be managed by the kernel and which by the UMD.
+ *
+ * TheDRM_NOUVEAU_VM_BINDioctl provides clients an interface to manage the
+ * userspace-managable portion of the VA space. It provides operations to map
+ * and unmap memory. Mappings may be flagged as sparse. Sparse mappings are not
+ * backed by a GEM object and the kernel will ignore GEM handles provided
+ * alongside a sparse mapping.
+ *
+ * Userspace may request memory backed mappings either within or outside of the
+ * bounds (but not crossing those bounds) of a previously mapped sparse
+ * mapping. Subsequently requested memory backed mappings within a sparse
+ * mapping will take precedence over the corresponding range of the sparse
+ * mapping. If such memory backed mappings are unmapped the kernel will make
+ * sure that the corresponding sparse mapping will take their place again.
+ * Requests to unmap a sparse mapping that still contains memory backed mappings
+ * will result in those memory backed mappings being unmapped first.
+ *
+ * Unmap requests are not bound to the range of existing mappings and can even
+ * overlap the bounds of sparse mappings. For such a request the kernel will
+ * make sure to unmap all memory backed mappings within the given range,
+ * splitting up memory backed mappings which are only partially contained
+ * within the given range. Unmap requests with the sparse flag set must match
+ * the range of a previously mapped sparse mapping exactly though.
+ *
+ * While the kernel generally permits arbitrary sequences and ranges of memory
+ * backed mappings being mapped and unmapped, either within a single or multiple
+ *VM_BINDioctl calls, there are some restrictions for sparse mappings.
+ *
+ * The kernel does not permit to:
+ * - unmap non-existent sparse mappings
+ * - unmap a sparse mapping and map a new sparse mapping overlapping
the range
+ * of the previously unmapped sparse mapping within the sameVM_BIND
ioctl
+ * - unmap a sparse mapping and map new memory backed mappings
overlapping the
+ * range of the previously unmapped sparse mapping within the sameVM_BIND
+ * ioctl
+ *
+ * When using theVM_BINDioctl to request the kernel to map memory to a given
+ * virtual address in the GPU's VA space there is no guarantee that the actual
+ * mappings are created in the GPU's MMU. If the given memory is swapped out
+ * at the time the bind operation is executed the kernel will stash the mapping
+ * details into it's internal alloctor and create the actual MMU mappings once
+ * the memory is swapped back in. While this is transparent for userspace, it is
+ * guaranteed that all the backing memory is swapped back in and all the memory
+ * mappings, as requested by userspace previously, are actually mapped once the
+ *DRM_NOUVEAU_EXECioctl is called to submit an exec job.
+ *
+ * AVM_BINDjob can be executed either synchronously or asynchronously. If
+ * exectued asynchronously, userspace may provide a list of syncobjs this job
+ * will wait for and/or a list of syncobj the kernel will signal once the
+ *VM_BINDjob finished execution. If executed synchronously the ioctl will
+ * block until the bind job is finished. For synchronous jobs the kernel will
+ * not permit any syncobjs submitted to the kernel.
+ *
+ * To execute a push buffer the UAPI provides theDRM_NOUVEAU_EXECioctl. EXEC
+ * jobs are always executed asynchronously, and, equal toVM_BINDjobs, provide
+ * the option to synchronize them with syncobjs.
+ *
+ * Besides that, EXEC jobs can be scheduled for a specified channel to execute on.
+ *
+ * SinceVM_BINDjobs update the GPU's VA space on job submit, EXEC jobs do have
+ * an up to date view of the VA space. However, the actual mappings might still
+ * be pending. Hence, EXEC jobs require to have the particular fences - of
+ * the correspondingVM_BINDjobs they depent on - attached to them.
+ */drivers/gpu/drm/nouveau/nouveau_exec.c new file +67
-0+ +static int +nouveau_exec_job_submit (struct nouveau_job *job) +{ +struct nouveau_exec_job *exec_job =to_nouveau_exec_job (job); +struct nouveau_cli *cli = exec_job->base.cli; +struct nouveau_uvmm *uvmm =nouveau_cli_uvmm (cli); +struct drm_exec *exec = &job->exec; +struct drm_gem_object *obj; +unsigned long index; +int ret; + + ret =nouveau_fence_new (&exec_job->fence); +if (ret) +return ret; + +nouveau_uvmm_lock (uvmm); +drm_exec_until_all_locked (exec) { +struct drm_gpuva *va; + +drm_gpuva_for_each_va (va, &uvmm->umgr) { +if (unlikely (va == &uvmm->umgr.kernel_alloc_node)) +continue ; + + ret =drm_exec_prepare_obj (exec, va->gem.obj,1 ); +drm_exec_retry_on_contention (exec); +if (ret) { +nouveau_uvmm_unlock (uvmm); +return ret; + } + } + } +nouveau_uvmm_unlock (uvmm); + +drm_exec_for_each_locked_object (exec, index, obj) { +struct nouveau_bo *nvbo =nouveau_gem_object (obj); + + ret =nouveau_bo_validate (nvbo, true, false); +if (ret) +return ret; + } + +return 0 ; +} + +static struct dma_fence * +nouveau_exec_job_run (struct nouveau_job *job) +{ +struct nouveau_exec_job *exec_job =to_nouveau_exec_job (job); +struct nouveau_channel *chan = exec_job->chan; +struct nouveau_fence *fence = exec_job->fence; +int i, ret; + + ret =nouveau_dma_wait (chan, exec_job->push.count +1 ,16 ); +if (ret) { +NV_PRINTK (err, job->cli,"nv50cal_space: %d\n" , ret); +return ERR_PTR (ret); + } + +for (i =0 ; i < exec_job->push.count; i++) { +nv50_dma_push (chan, exec_job->push.s[i].va, + exec_job->push.s[i].va_len); + } + + ret =nouveau_fence_emit (fence, chan); +if (ret) { + NV_PRINTK(err, job->cli, "error fencing pushbuf: %d\n",ret);
drivers/gpu/drm/nouveau/nouveau_exec.c new file +215
-0+WIND_RING (chan); +return ERR_PTR (ret); + } + + exec_job->fence =NULL ; + +return &fence->base; +} + +static void +nouveau_exec_job_free (struct nouveau_job *job) +{ +struct nouveau_exec_job *exec_job =to_nouveau_exec_job (job); + +nouveau_job_free (job); + +nouveau_fence_unref (&exec_job->fence); +kfree (exec_job->push.s); +kfree (exec_job); +} + +static enum drm_gpu_sched_stat +nouveau_exec_job_timeout (struct nouveau_job *job) +{ +struct nouveau_exec_job *exec_job =to_nouveau_exec_job (job); +struct nouveau_channel *chan = exec_job->chan; + +if (unlikely (!atomic_read (&chan->killed))) +nouveau_channel_kill (chan); + +NV_PRINTK (warn, job->cli,"job timeout, channel %d killed!\n" , + chan->chid); + +nouveau_sched_entity_fini (job->entity); + +return DRM_GPU_SCHED_STAT_ENODEV ; +} + +static struct nouveau_job_ops nouveau_exec_job_ops = { + .submit = nouveau_exec_job_submit, + .run = nouveau_exec_job_run, + .free = nouveau_exec_job_free, + .timeout = nouveau_exec_job_timeout, +}; + +int +nouveau_exec_job_init (struct nouveau_exec_job **pjob, +struct nouveau_exec_job_args *__args) +{ +struct nouveau_exec_job *job; +struct nouveau_job_args args = {}; +int ret; + + job = *pjob =kzalloc (sizeof (*job),GFP_KERNEL ); +if (!job) +return -ENOMEM ; + + job->push.count = __args->push.count; + job->push.s =kmemdup (__args->push.s, +sizeof (*__args->push.s) * + __args->push.count, +GFP_KERNEL ); +if (!job->push.s) { + ret = -ENOMEM ; + goto err_free_job; + } + + job->chan = __args->chan; + + args.sched_entity = __args->sched_entity; + args.file_priv = __args->file_priv; + + args.in_sync.count = __args->in_sync.count; + args.in_sync.s = __args->in_sync.s; + + args.out_sync.count = __args->out_sync.count; + args.out_sync.s = __args->out_sync.s; + + args.ops = &nouveau_exec_job_ops; + args.resv_usage =DMA_RESV_USAGE_WRITE ; + + ret =nouveau_job_init (&job->base, &args); +if (ret) + goto err_free_pushs; + +return 0 ; + +err_free_pushs: +kfree (job->push.s); +err_free_job: +kfree (job); + *pjob =NULL ; + +return ret; +} + +static int +nouveau_exec (struct nouveau_exec_job_args *args) +{ +struct nouveau_exec_job *job; +int ret; + + ret =nouveau_exec_job_init (&job, args); +if (ret) +return ret; + + ret =nouveau_job_submit (&job->base); +if (ret) + goto err_job_fini; + +return 0 ; + +err_job_fini: +nouveau_job_fini (&job->base); +return ret; +} + +static int +nouveau_exec_ucopy (struct nouveau_exec_job_args *args, +struct drm_nouveau_exec __user *req) +{ +struct drm_nouveau_sync **s; +u32 inc = req->wait_count; +u64 ins = req->wait_ptr; +u32 outc = req->sig_count; +u64 outs = req->sig_ptr; +u32 pushc = req->push_count; +u64 pushs = req->push_ptr; +int ret; + + args->push.count = pushc; + args->push.s =u_memcpya (pushs, pushc,sizeof (*args->push.s)); +if (IS_ERR (args->push.s)) +return PTR_ERR (args->push.s); + +if (inc) { + s = &args->in_sync.s; + + args->in_sync.count = inc; + *s =u_memcpya (ins, inc,sizeof (**s)); +if (IS_ERR (*s)) { + ret =PTR_ERR (*s); + goto err_free_pushs; + } + } + +if (outc) { + s = &args->out_sync.s; + + args->out_sync.count = outc; + *s =u_memcpya (outs, outc,sizeof (**s)); +if (IS_ERR (*s)) { + ret =PTR_ERR (*s); + goto err_free_ins; + } + } + +return 0 ; + +err_free_pushs: +u_free (args->push.s); +err_free_ins: +u_free (args->in_sync.s); +return ret; +} + +static void +nouveau_exec_ufree (struct nouveau_exec_job_args *args) +{ +u_free (args->push.s); +u_free (args->in_sync.s); +u_free (args->out_sync.s); +} + +int +nouveau_exec_ioctl_exec (struct drm_device *dev, +void __user *data, +struct drm_file *file_priv) +{ +struct nouveau_abi16 *abi16 =nouveau_abi16_get (file_priv); +struct nouveau_cli *cli =nouveau_cli (file_priv); +struct nouveau_abi16_chan *chan16; +struct nouveau_channel *chan =NULL ; +struct nouveau_exec_job_args args = {}; +struct drm_nouveau_exec __user *req = data; +int ret =0 ; + +if (unlikely (!abi16)) +return -ENOMEM ; + +/* abi16 locks already */ +if (unlikely (!nouveau_cli_uvmm (cli))) +return nouveau_abi16_put (abi16, -ENOSYS ); + +list_for_each_entry (chan16, &abi16->channels, head) { +if (chan16->chan->chid == req->channel) { + chan = chan16->chan; +break ; + } + } + +if (!chan) +return nouveau_abi16_put (abi16, -ENOENT ); + +if (unlikely (atomic_read (&chan->killed))) +return nouveau_abi16_put (abi16, -ENODEV ); + +if (!chan->dma.ib_max) +return nouveau_abi16_put (abi16, -ENOSYS ); + +if (unlikely (req->push_count ==0 )) + goto out; + +if (unlikely (req->push_count >NOUVEAU_GEM_MAX_PUSH )) { + NV_PRINTK(err, cli, "pushbuf push count exceeds limit: %dmax %d\n",
drivers/gpu/drm/nouveau/nouveau_exec.c new file +21
-0+ req->push_count, NOUVEAU_GEM_MAX_PUSH); +return nouveau_abi16_put (abi16, -EINVAL ); + } + + ret =nouveau_exec_ucopy (&args, req); +if (ret) + goto out; + + args.sched_entity = &chan16->sched_entity; + args.file_priv = file_priv; + args.chan = chan; + + ret =nouveau_exec (&args); +if (ret) + goto out_free_args; + +out_free_args: +nouveau_exec_ufree (&args); +out: +return nouveau_abi16_put (abi16, ret); +}drivers/gpu/drm/nouveau/nouveau_exec.h new file +54
-0@@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: MIT */ + +#ifndef __NOUVEAU_EXEC_H__ +#define __NOUVEAU_EXEC_H__ + +#include <drm/drm_exec.h> + +#include "nouveau_drv.h" +#include "nouveau_sched.h" + +struct nouveau_exec_job_args { +struct drm_file *file_priv; +struct nouveau_sched_entity *sched_entity; + +struct drm_exec exec; +struct nouveau_channel *chan; + +struct { +struct drm_nouveau_sync *s; +u32 count; + } in_sync; + +struct { +struct drm_nouveau_sync *s; +u32 count; + } out_sync; + +struct { +struct drm_nouveau_exec_push *s; +u32 count; + } push; +}; + +struct nouveau_exec_job { +struct nouveau_job base; +struct nouveau_fence *fence; +struct nouveau_channel *chan; + +struct { +struct drm_nouveau_exec_push *s; +u32 count; + } push; +}; + +#define to_nouveau_exec_job (job) \ + container_of((job), struct nouveau_exec_job, base) + +int nouveau_exec_job_init (struct nouveau_exec_job **job, +struct nouveau_exec_job_args *args); + +int nouveau_exec_ioctl_exec (struct drm_device *dev,void __user *data, +struct drm_file *file_priv); + +#endif drivers/gpu/drm/nouveau/nouveau_gem.c +5
-1@@ -120,7 +120,11 @@ nouveau_gem_object_open(struct drm_gem_object *gem, struct drm_file *file_priv) goto out; } - ret =nouveau_vma_new (nvbo, vmm, &vma); +/* only create a VMA on binding */ +if (!nouveau_cli_uvmm (cli)) + ret =nouveau_vma_new (nvbo, vmm, &vma); +else + ret =0 ;pm_runtime_mark_last_busy (dev);pm_runtime_put_autosuspend (dev);out:
drivers/gpu/drm/nouveau/nouveau_gem.c +3
-0@@ -187,6 +191,9 @@ nouveau_gem_object_close(struct drm_gem_object *gem, struct drm_file *file_priv)if (vmm->vmm.object.oclass <NVIF_CLASS_VMM_NV50 )return ; +if (nouveau_cli_uvmm (cli)) +return ; + ret =ttm_bo_reserve (&nvbo->bo, false, false,NULL );if (ret)return;drivers/gpu/drm/nouveau/nouveau_gem.c +5
-3@@ -231,7 +238,7 @@ nouveau_gem_new(struct nouveau_cli *cli, u64 size, int align, uint32_t domain, domain |=NOUVEAU_GEM_DOMAIN_CPU ; nvbo =nouveau_bo_alloc (cli, &size, &align, domain, tile_mode, - tile_flags); + tile_flags, false);if (IS_ERR (nvbo))return PTR_ERR (nvbo);@@ -279,13 +286,15 @@ nouveau_gem_info(struct drm_file *file_priv, structdrm_gem_object *gem, else rep->domain = NOUVEAU_GEM_DOMAIN_VRAM; rep->offset = nvbo->offset; -if (vmm->vmm.object.oclass >=NVIF_CLASS_VMM_NV50 ) { +if (vmm->vmm.object.oclass >=NVIF_CLASS_VMM_NV50 && + !nouveau_cli_uvmm (cli)) { vma =nouveau_vma_find (nvbo, vmm);if (!vma)return -EINVAL ; rep->offset = vma->addr; - } + }else + rep->offset =0 ; rep->size = nvbo->bo.base.size;rep->map_handle =
drm_vma_node_offset_addr(&nvbo->bo.base.vma_node);drivers/gpu/drm/nouveau/nouveau_gem.c +1
-0@@ -310,6 +319,11 @@ nouveau_gem_ioctl_new(struct drm_device *dev, void *data, struct nouveau_bo *nvbo = NULL;int ret =0 ; +/* If uvmm wasn't initialized until now disable it completely to prevent
drivers/gpu/drm/nouveau/nouveau_gem.c +7
-0+* userspace from mixing up UAPIs. +*/ +nouveau_cli_uvmm_disable (cli); + ret =nouveau_gem_new (cli, req->info.size, req->align, req->info.domain, req->info.tile_mode, req->info.tile_flags, &nvbo);@@ -721,6 +735,9 @@ nouveau_gem_ioctl_pushbuf(struct drm_device *dev, void *data, if (unlikely(!abi16)) return -ENOMEM; +if (unlikely (nouveau_cli_uvmm (cli))) +return -ENOSYS ; +list_for_each_entry (temp, &abi16->channels, head) {if (temp->chan->chid == req->channel) {chan = temp->chan;drivers/gpu/drm/nouveau/nouveau_mem.h +0
-0@@ -35,4 +35,9 @@ int nouveau_mem_vram(struct ttm_resource *, bool contig, u8 page);int nouveau_mem_host (struct ttm_resource *,struct ttm_tt *);void nouveau_mem_fini (struct nouveau_mem *); int nouveau_mem_map(struct nouveau_mem *, struct nvif_vmm *, struct
nvif_vma*);drivers/gpu/drm/nouveau/nouveau_mem.h +5
-0+int +nouveau_mem_map_fixed (struct nouveau_mem *mem, +struct nvif_vmm *vmm, +u8 kind,u64 addr, +u64 offset,u64 range);#endif drivers/gpu/drm/nouveau/nouveau_prime.c +1
-1@@ -50,7 +50,7 @@ struct drm_gem_object *nouveau_gem_prime_import_sg_table(struct drm_device *dev, dma_resv_lock(robj, NULL); nvbo =nouveau_bo_alloc (&drm->client, &size, &align, -NOUVEAU_GEM_DOMAIN_GART ,0 ,0 ); +NOUVEAU_GEM_DOMAIN_GART ,0 ,0 , true);if (IS_ERR (nvbo)) { obj =ERR_CAST (nvbo);goto unlock;drivers/gpu/drm/nouveau/nouveau_sched.c new file +6
-0@@ -0,0 +1,462 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright (c) 2022 Red Hat. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors:
+ * Danilo Krummrich <[email protected]>
+ *
+ */drivers/gpu/drm/nouveau/nouveau_sched.c new file +17
-0+ +#include <linux/slab.h> +#include <drm/gpu_scheduler.h> +#include <drm/drm_syncobj.h> + +#include "nouveau_drv.h" +#include "nouveau_gem.h" +#include "nouveau_mem.h" +#include "nouveau_dma.h" +#include "nouveau_exec.h" +#include "nouveau_abi16.h" +#include "nouveau_sched.h" + +/* FIXME +* +* We want to make sure that jobs currently executing can't be deferred by +* other jobs competing for the hardware. Otherwise we might end up with job
+ * timeouts just because of too many clients submitting too many jobs. We don't
+ * want jobs to time out because of system load, but because of the job being
+ * too bulky.
+ *
+ * For now allow for up to 16 concurrent jobs in flight until we know how many
+ * rings the hardware can process in parallel.
+ */drivers/gpu/drm/nouveau/nouveau_sched.c new file +58
-0+#define NOUVEAU_SCHED_HW_SUBMISSIONS 16 +#define NOUVEAU_SCHED_JOB_TIMEOUT_MS 10000 + +int +nouveau_job_init (struct nouveau_job *job, +struct nouveau_job_args *args) +{ +struct nouveau_sched_entity *entity = args->sched_entity; +int ret; + + job->file_priv = args->file_priv; + job->cli =nouveau_cli (args->file_priv); + job->entity = entity; + + job->sync = args->sync; + job->resv_usage = args->resv_usage; + + job->ops = args->ops; + + job->in_sync.count = args->in_sync.count; +if (job->in_sync.count) { +if (job->sync) +return -EINVAL ; + + job->in_sync.data =kmemdup (args->in_sync.s, +sizeof (*args->in_sync.s) * + args->in_sync.count, +GFP_KERNEL ); +if (!job->in_sync.data) +return -ENOMEM ; + } + + job->out_sync.count = args->out_sync.count; +if (job->out_sync.count) { +if (job->sync) { + ret = -EINVAL ; + goto err_free_in_sync; + } + + job->out_sync.data =kmemdup (args->out_sync.s, +sizeof (*args->out_sync.s) * + args->out_sync.count, +GFP_KERNEL ); +if (!job->out_sync.data) { + ret = -ENOMEM ; + goto err_free_in_sync; + } + + job->out_sync.objs =kcalloc (job->out_sync.count, +sizeof (*job->out_sync.objs), +GFP_KERNEL ); +if (!job->out_sync.objs) { + ret = -ENOMEM ; + goto err_free_out_sync; + } + + job->out_sync.chains = kcalloc(job->out_sync.count, +
sizeof(*job->out_sync.chains),drivers/gpu/drm/nouveau/nouveau_sched.c new file +81
-0+ GFP_KERNEL); +if (!job->out_sync.chains) { + ret = -ENOMEM ; + goto err_free_objs; + } + + } + + ret =drm_sched_job_init (&job->base, &entity->base,NULL ); +if (ret) + goto err_free_chains; + + job->state =NOUVEAU_JOB_INITIALIZED ; + +return 0 ; + +err_free_chains: +kfree (job->out_sync.chains); +err_free_objs: +kfree (job->out_sync.objs); +err_free_out_sync: +kfree (job->out_sync.data); +err_free_in_sync: +kfree (job->in_sync.data); +return ret; +} + +void +nouveau_job_free (struct nouveau_job *job) +{ +kfree (job->in_sync.data); +kfree (job->out_sync.data); +kfree (job->out_sync.objs); +kfree (job->out_sync.chains); +} + +void nouveau_job_fini (struct nouveau_job *job) +{ +dma_fence_put (job->done_fence); +drm_sched_job_cleanup (&job->base); + job->ops->free (job); +} + +static int +sync_find_fence (struct nouveau_job *job, +struct drm_nouveau_sync *sync, +struct dma_fence **fence) +{ +u32 stype = sync->flags &DRM_NOUVEAU_SYNC_TYPE_MASK ; +u64 point =0 ; +int ret; + +if (stype !=DRM_NOUVEAU_SYNC_SYNCOBJ && + stype !=DRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ ) +return -EOPNOTSUPP ; + +if (stype ==DRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ ) + point = sync->timeline_value; + + ret =drm_syncobj_find_fence (job->file_priv, + sync->handle, point, + sync->flags, fence); +if (ret) +return ret; + +return 0 ; +} + +static int +nouveau_job_add_deps (struct nouveau_job *job) +{ +struct dma_fence *in_fence =NULL ; +int ret, i; + +for (i =0 ; i < job->in_sync.count; i++) { +struct drm_nouveau_sync *sync = &job->in_sync.data[i]; + + ret =sync_find_fence (job, sync, &in_fence); +if (ret) { + NV_PRINTK(warn, job->cli, + "Failed to find syncobj (-> in):handle=%d\n",
drivers/gpu/drm/nouveau/nouveau_sched.c new file +50
-0+ sync->handle); +return ret; + } + + ret =drm_sched_job_add_dependency (&job->base, in_fence); +if (ret) +return ret; + } + +return 0 ; +} + +static void +nouveau_job_fence_attach_cleanup (struct nouveau_job *job) +{ +int i; + +for (i =0 ; i < job->out_sync.count; i++) { +struct drm_syncobj *obj = job->out_sync.objs[i]; +struct dma_fence_chain *chain = job->out_sync.chains[i]; + +if (obj) +drm_syncobj_put (obj); + +if (chain) +dma_fence_chain_free (chain); + } +} + +static int +nouveau_job_fence_attach_prepare (struct nouveau_job *job) +{ +int i, ret; + +for (i =0 ; i < job->out_sync.count; i++) { +struct drm_nouveau_sync *sync = &job->out_sync.data[i]; +struct drm_syncobj **pobj = &job->out_sync.objs[i]; +struct dma_fence_chain **pchain = &job->out_sync.chains[i]; +u32 stype = sync->flags &DRM_NOUVEAU_SYNC_TYPE_MASK ; + +if (stype !=DRM_NOUVEAU_SYNC_SYNCOBJ && + stype !=DRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ ) { + ret = -EINVAL ; + goto err_sync_cleanup; + } + + *pobj =drm_syncobj_find (job->file_priv, sync->handle); +if (!*pobj) { + NV_PRINTK(warn, job->cli, + "Failed to find syncobj (-> out):handle=%d\n",
drivers/gpu/drm/nouveau/nouveau_sched.c new file +64
-0+ sync->handle); + ret = -ENOENT ; + goto err_sync_cleanup; + } + +if (stype ==DRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ ) { + *pchain =dma_fence_chain_alloc (); +if (!*pchain) { + ret = -ENOMEM ; + goto err_sync_cleanup; + } + } + } + +return 0 ; + +err_sync_cleanup: +nouveau_job_fence_attach_cleanup (job); +return ret; +} + +static void +nouveau_job_fence_attach (struct nouveau_job *job) +{ +struct dma_fence *fence = job->done_fence; +int i; + +for (i =0 ; i < job->out_sync.count; i++) { +struct drm_nouveau_sync *sync = &job->out_sync.data[i]; +struct drm_syncobj **pobj = &job->out_sync.objs[i]; +struct dma_fence_chain **pchain = &job->out_sync.chains[i]; +u32 stype = sync->flags &DRM_NOUVEAU_SYNC_TYPE_MASK ; + +if (stype ==DRM_NOUVEAU_SYNC_TIMELINE_SYNCOBJ ) { +drm_syncobj_add_point (*pobj, *pchain, fence, + sync->timeline_value); + }else { +drm_syncobj_replace_fence (*pobj, fence); + } + +drm_syncobj_put (*pobj); + *pobj =NULL ; + *pchain =NULL ; + } +} + +static void +nouveau_job_resv_add_fence (struct nouveau_job *job) +{ +struct drm_exec *exec = &job->exec; +struct drm_gem_object *obj; +unsigned long index; + +drm_exec_for_each_locked_object (exec, index, obj) { +struct dma_resv *resv = obj->resv; + +dma_resv_add_fence (resv, job->done_fence, job->resv_usage); + } +} + +int +nouveau_job_submit (struct nouveau_job *job) +{ + struct nouveau_sched_entity *entity =
to_nouveau_sched_entity(job->base.entity);drivers/gpu/drm/nouveau/nouveau_sched.c new file +12
-0+struct dma_fence *done_fence =NULL ; +int ret; + + ret =nouveau_job_add_deps (job); +if (ret) + goto err; + + ret =nouveau_job_fence_attach_prepare (job); +if (ret) + goto err; + +/* Make sure the job appears on the sched_entity's queue in the same
drivers/gpu/drm/nouveau/nouveau_sched.c new file +132
-0+* order as it was submitted. +*/ +mutex_lock (&entity->mutex); + +drm_exec_init (&job->exec,DRM_EXEC_INTERRUPTIBLE_WAIT | +DRM_EXEC_IGNORE_DUPLICATES ); + +/* Guarantee jobs we won't fail after the submit() callback +* returned successfully. +*/ +if (job->ops->submit) { + ret = job->ops->submit (job); +if (ret) + goto err_cleanup; + } + +drm_sched_job_arm (&job->base); + job->done_fence =dma_fence_get (&job->base.s_fence->finished); +if (job->sync) + done_fence =dma_fence_get (job->done_fence); + +nouveau_job_fence_attach (job); +nouveau_job_resv_add_fence (job); + +drm_exec_fini (&job->exec); + +/* Set job state before pushing the job to the scheduler, +* such that we do not overwrite the job state set in run(). +*/ + job->state =NOUVEAU_JOB_SUBMIT_SUCCESS ; + +drm_sched_entity_push_job (&job->base); + +mutex_unlock (&entity->mutex); + +if (done_fence) { +dma_fence_wait (done_fence, true); +dma_fence_put (done_fence); + } + +return 0 ; + +err_cleanup: +drm_exec_fini (&job->exec); +mutex_unlock (&entity->mutex); +nouveau_job_fence_attach_cleanup (job); +err: + job->state =NOUVEAU_JOB_SUBMIT_FAILED ; +return ret; +} + +bool +nouveau_sched_entity_qwork (struct nouveau_sched_entity *entity, +struct work_struct *work) +{ +return queue_work (entity->sched_wq, work); +} + +static struct dma_fence * +nouveau_job_run (struct nouveau_job *job) +{ +struct dma_fence *fence; + + fence = job->ops->run (job); +if (unlikely (IS_ERR (fence))) + job->state =NOUVEAU_JOB_RUN_FAILED ; +else + job->state =NOUVEAU_JOB_RUN_SUCCESS ; + +return fence; +} + +static struct dma_fence * +nouveau_sched_run_job (struct drm_sched_job *sched_job) +{ +struct nouveau_job *job =to_nouveau_job (sched_job); + +return nouveau_job_run (job); +} + +static enum drm_gpu_sched_stat +nouveau_sched_timedout_job (struct drm_sched_job *sched_job) +{ +struct nouveau_job *job =to_nouveau_job (sched_job); + +NV_PRINTK (warn, job->cli,"Job timed out.\n" ); + +if (job->ops->timeout) +return job->ops->timeout (job); + +return DRM_GPU_SCHED_STAT_ENODEV ; +} + +static void +nouveau_sched_free_job (struct drm_sched_job *sched_job) +{ +struct nouveau_job *job =to_nouveau_job (sched_job); + +nouveau_job_fini (job); +} + +int nouveau_sched_entity_init (struct nouveau_sched_entity *entity, +struct drm_gpu_scheduler *sched, +struct workqueue_struct *sched_wq) +{ +mutex_init (&entity->mutex); +spin_lock_init (&entity->job.list.lock); +INIT_LIST_HEAD (&entity->job.list.head); +init_waitqueue_head (&entity->job.wq); + + entity->sched_wq = sched_wq; +return drm_sched_entity_init (&entity->base, +DRM_SCHED_PRIORITY_NORMAL , + &sched,1 ,NULL ); +} + +void +nouveau_sched_entity_fini (struct nouveau_sched_entity *entity) +{ +drm_sched_entity_destroy (&entity->base); +} + +static const struct drm_sched_backend_ops nouveau_sched_ops = { + .run_job = nouveau_sched_run_job, + .timedout_job = nouveau_sched_timedout_job, + .free_job = nouveau_sched_free_job, +}; + +int nouveau_sched_init (struct nouveau_drm *drm) +{ +struct drm_gpu_scheduler *sched = &drm->sched; + long job_hang_limit =
msecs_to_jiffies(NOUVEAU_SCHED_JOB_TIMEOUT_MS);drivers/gpu/drm/nouveau/nouveau_sched.c new file +7
-0+ + drm->sched_wq =create_singlethread_workqueue ("nouveau_sched_wq" ); +if (!drm->sched_wq) +return ENOMEM ; + + return drm_sched_init(sched, &nouveau_sched_ops, + NOUVEAU_SCHED_HW_SUBMISSIONS, 0,
job_hang_limit,drivers/gpu/drm/nouveau/nouveau_sched.c new file +8
-0+ NULL, NULL, "nouveau_sched", drm->dev->dev); +} + +void nouveau_sched_fini (struct nouveau_drm *drm) +{ +destroy_workqueue (drm->sched_wq); +drm_sched_fini (&drm->sched); +}drivers/gpu/drm/nouveau/nouveau_sched.h new file +123
-0@@ -0,0 +1,123 @@ +/* SPDX-License-Identifier: MIT */ + +#ifndef NOUVEAU_SCHED_H +#define NOUVEAU_SCHED_H + +#include <linux/types.h> + +#include <drm/drm_exec.h> +#include <drm/gpu_scheduler.h> + +#include "nouveau_drv.h" + +#define to_nouveau_job (sched_job) \ + container_of((sched_job), struct nouveau_job, base) + +struct nouveau_job_ops ; + +enum nouveau_job_state { +NOUVEAU_JOB_UNINITIALIZED =0 , +NOUVEAU_JOB_INITIALIZED , +NOUVEAU_JOB_SUBMIT_SUCCESS , +NOUVEAU_JOB_SUBMIT_FAILED , +NOUVEAU_JOB_RUN_SUCCESS , +NOUVEAU_JOB_RUN_FAILED , +}; + +struct nouveau_job_args { +struct drm_file *file_priv; +struct nouveau_sched_entity *sched_entity; + +enum dma_resv_usage resv_usage; +bool sync; + +struct { +struct drm_nouveau_sync *s; +u32 count; + } in_sync; + +struct { +struct drm_nouveau_sync *s; +u32 count; + } out_sync; + +struct nouveau_job_ops *ops; +}; + +struct nouveau_job { +struct drm_sched_job base; + +enum nouveau_job_state state; + +struct nouveau_sched_entity *entity; + +struct drm_file *file_priv; +struct nouveau_cli *cli; + +struct drm_exec exec; +enum dma_resv_usage resv_usage; +struct dma_fence *done_fence; + +bool sync; + +struct { +struct drm_nouveau_sync *data; +u32 count; + } in_sync; + +struct { +struct drm_nouveau_sync *data; +struct drm_syncobj **objs; +struct dma_fence_chain **chains; +u32 count; + } out_sync; + +struct nouveau_job_ops { +int (*submit)(struct nouveau_job *); +struct dma_fence *(*run)(struct nouveau_job *); +void (*free)(struct nouveau_job *); +enum drm_gpu_sched_stat (*timeout)(struct nouveau_job *); + } *ops; +}; + +int nouveau_job_ucopy_syncs (struct nouveau_job_args *args, +u32 inc,u64 ins, +u32 outc,u64 outs); + +int nouveau_job_init (struct nouveau_job *job, +struct nouveau_job_args *args); +void nouveau_job_free (struct nouveau_job *job); + +int nouveau_job_submit (struct nouveau_job *job); +void nouveau_job_fini (struct nouveau_job *job); + +#define to_nouveau_sched_entity (entity) \ + container_of((entity), struct nouveau_sched_entity, base) + +struct nouveau_sched_entity { +struct drm_sched_entity base; +struct mutex mutex; + +struct workqueue_struct *sched_wq; + +struct { +struct { +struct list_head head; +spinlock_t lock; + } list; +struct wait_queue_head wq; + } job; +}; + +int nouveau_sched_entity_init (struct nouveau_sched_entity *entity, +struct drm_gpu_scheduler *sched, +struct workqueue_struct *sched_wq); +void nouveau_sched_entity_fini (struct nouveau_sched_entity *entity); + +bool nouveau_sched_entity_qwork (struct nouveau_sched_entity *entity, +struct work_struct *work); + +int nouveau_sched_init (struct nouveau_drm *drm); +void nouveau_sched_fini (struct nouveau_drm *drm); + +#endif drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +6
-0@@ -0,0 +1,1970 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright (c) 2022 Red Hat. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors:
+ * Danilo Krummrich <[email protected]>
+ *
+ */drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +5
-0+ +/* +* Locking: +* +* The uvmm mutex protects any operations on the GPU VA space provided by the
+ * DRM GPU VA manager.
+ *
+ * The GEMsdma_resvlock protects the GEMs GPUVA list, hence link/unlink of a
+ * mapping to it's backing GEM must be performed under this lock.
+ *
+ * Actual map/unmap operations within the fence signalling critical path are
+ * protected by installing DMA fences to the corresponding GEMs DMA
+ * reservations, such that concurrent BO moves, which itself walk the GEMs GPUVA
+ * list in order to map/unmap it's entries, can't occur concurrently.
+ *
+ * Accessing theDRM_GPUVA_INVALIDATEDflag doesn't need any separate
+ * protection, since there are no accesses other than from BO move callbacks
+ * and from the fence signalling critical path, which are already protected by
+ * the corresponding GEMs DMA reservation fence.
+ */drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +18
-0+ +#include "nouveau_drv.h" +#include "nouveau_gem.h" +#include "nouveau_mem.h" +#include "nouveau_uvmm.h" + +#include <nvif/vmm.h> +#include <nvif/mem.h> + +#include <nvif/class.h> +#include <nvif/if000c.h> +#include <nvif/if900d.h> + +#define NOUVEAU_VA_SPACE_BITS 47/* FIXME */ +#define NOUVEAU_VA_SPACE_START 0x0 +#define NOUVEAU_VA_SPACE_END (1ULL << NOUVEAU_VA_SPACE_BITS) + +#define list_last_op (_ops) list_last_entry(_ops, struct bind_job_op,entry)
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +5
-0+#define list_prev_op (_op) list_prev_entry(_op, entry) +#define list_for_each_op (_op, _ops) list_for_each_entry(_op, _ops, entry) +#define list_for_each_op_from_reverse (_op, _ops) \ + list_for_each_entry_from_reverse(_op, _ops, entry) +#define list_for_each_op_safe (_op, _n, _ops)
list_for_each_entry_safe(_op, _n, _ops, entry)drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +485
-0+ +enum vm_bind_op { +OP_MAP =DRM_NOUVEAU_VM_BIND_OP_MAP , +OP_UNMAP =DRM_NOUVEAU_VM_BIND_OP_UNMAP , +OP_MAP_SPARSE , +OP_UNMAP_SPARSE , +}; + +struct nouveau_uvma_prealloc { +struct nouveau_uvma *map; +struct nouveau_uvma *prev; +struct nouveau_uvma *next; +}; + +struct bind_job_op { +struct list_head entry; + +enum vm_bind_op op; +u32 flags; + +struct { +u64 addr; +u64 range; + } va; + +struct { +u32 handle; +u64 offset; +struct drm_gem_object *obj; + } gem; + +struct nouveau_uvma_region *reg; +struct nouveau_uvma_prealloc new; +struct drm_gpuva_ops *ops; +}; + +struct uvmm_map_args { +struct nouveau_uvma_region *region; +u64 addr; +u64 range; +u8 kind; +}; + +static int +nouveau_uvmm_vmm_sparse_ref (struct nouveau_uvmm *uvmm, +u64 addr,u64 range) +{ +struct nvif_vmm *vmm = &uvmm->vmm.vmm; + +return nvif_vmm_raw_sparse (vmm, addr, range, true); +} + +static int +nouveau_uvmm_vmm_sparse_unref (struct nouveau_uvmm *uvmm, +u64 addr,u64 range) +{ +struct nvif_vmm *vmm = &uvmm->vmm.vmm; + +return nvif_vmm_raw_sparse (vmm, addr, range, false); +} + +static int +nouveau_uvmm_vmm_get (struct nouveau_uvmm *uvmm, +u64 addr,u64 range) +{ +struct nvif_vmm *vmm = &uvmm->vmm.vmm; + +return nvif_vmm_raw_get (vmm, addr, range,PAGE_SHIFT ); +} + +static int +nouveau_uvmm_vmm_put (struct nouveau_uvmm *uvmm, +u64 addr,u64 range) +{ +struct nvif_vmm *vmm = &uvmm->vmm.vmm; + +return nvif_vmm_raw_put (vmm, addr, range,PAGE_SHIFT ); +} + +static int +nouveau_uvmm_vmm_unmap (struct nouveau_uvmm *uvmm, +u64 addr,u64 range,bool sparse) +{ +struct nvif_vmm *vmm = &uvmm->vmm.vmm; + +return nvif_vmm_raw_unmap (vmm, addr, range,PAGE_SHIFT , sparse); +} + +static int +nouveau_uvmm_vmm_map (struct nouveau_uvmm *uvmm, +u64 addr,u64 range, +u64 bo_offset,u8 kind, +struct nouveau_mem *mem) +{ +struct nvif_vmm *vmm = &uvmm->vmm.vmm; +union { +struct gf100_vmm_map_v0 gf100; + } args; +u32 argc =0 ; + +switch (vmm->object.oclass) { +case NVIF_CLASS_VMM_GF100 : +case NVIF_CLASS_VMM_GM200 : +case NVIF_CLASS_VMM_GP100 : + args.gf100.version =0 ; +if (mem->mem.type &NVIF_MEM_VRAM ) + args.gf100.vol =0 ; +else + args.gf100.vol =1 ; + args.gf100.ro =0 ; + args.gf100.priv =0 ; + args.gf100.kind = kind; + argc =sizeof (args.gf100); +break ; +default : +WARN_ON (1 ); +return -ENOSYS ; + } + +return nvif_vmm_raw_map (vmm, addr, range,PAGE_SHIFT , + &args, argc, + &mem->mem, bo_offset); +} + +static int +nouveau_uvma_region_sparse_unref (struct nouveau_uvma_region *reg) +{ +u64 addr = reg->va.addr; +u64 range = reg->va.range; + +return nouveau_uvmm_vmm_sparse_unref (reg->uvmm, addr, range); +} + +static int +nouveau_uvma_vmm_put (struct nouveau_uvma *uvma) +{ +u64 addr = uvma->va.va.addr; +u64 range = uvma->va.va.range; + +return nouveau_uvmm_vmm_put (uvma->uvmm, addr, range); +} + +static int +nouveau_uvma_map (struct nouveau_uvma *uvma, +struct nouveau_mem *mem) +{ +u64 addr = uvma->va.va.addr; +u64 offset = uvma->va.gem.offset; +u64 range = uvma->va.va.range; + +return nouveau_uvmm_vmm_map (uvma->uvmm, addr, range, + offset, uvma->kind, mem); +} + +static int +nouveau_uvma_unmap (struct nouveau_uvma *uvma) +{ +u64 addr = uvma->va.va.addr; +u64 range = uvma->va.va.range; +bool sparse = !!uvma->region; + +if (drm_gpuva_invalidated (&uvma->va)) +return 0 ; + +return nouveau_uvmm_vmm_unmap (uvma->uvmm, addr, range, sparse); +} + +static int +nouveau_uvma_alloc (struct nouveau_uvma **puvma) +{ + *puvma =kzalloc (sizeof (**puvma),GFP_KERNEL ); +if (!*puvma) +return -ENOMEM ; + +return 0 ; +} + +static void +nouveau_uvma_free (struct nouveau_uvma *uvma) +{ +kfree (uvma); +} + +static int +__nouveau_uvma_insert (struct nouveau_uvmm *uvmm, +struct nouveau_uvma *uvma) +{ +return drm_gpuva_insert (&uvmm->umgr, &uvma->va); +} + +static int +nouveau_uvma_insert (struct nouveau_uvmm *uvmm, +struct nouveau_uvma *uvma, +struct nouveau_uvma_region *region, +struct drm_gem_object *obj, +u64 bo_offset,u64 addr, +u64 range,u8 kind) +{ +int ret; + + uvma->uvmm = uvmm; + uvma->region = region; + uvma->kind = kind; + uvma->va.va.addr = addr; + uvma->va.va.range = range; + uvma->va.gem.offset = bo_offset; + uvma->va.gem.obj = obj; + + ret =__nouveau_uvma_insert (uvmm, uvma); +if (ret) +return ret; + +return 0 ; +} + +static void +nouveau_uvma_remove (struct nouveau_uvma *uvma) +{ +drm_gpuva_remove (&uvma->va); +} + +static void +nouveau_uvma_gem_get (struct nouveau_uvma *uvma) +{ +drm_gem_object_get (uvma->va.gem.obj); +} + +static void +nouveau_uvma_gem_put (struct nouveau_uvma *uvma) +{ +drm_gem_object_put (uvma->va.gem.obj); +} + +static int +nouveau_uvma_region_alloc (struct nouveau_uvma_region **preg) +{ + *preg =kzalloc (sizeof (**preg),GFP_KERNEL ); +if (!*preg) +return -ENOMEM ; + +kref_init (&(*preg)->kref); + +return 0 ; +} + +static void +nouveau_uvma_region_free (struct kref *kref) +{ +struct nouveau_uvma_region *reg = + container_of(kref, struct nouveau_uvma_region, kref); + +kfree (reg); +} + +static void +nouveau_uvma_region_get (struct nouveau_uvma_region *reg) +{ +kref_get (®->kref); +} + +static void +nouveau_uvma_region_put (struct nouveau_uvma_region *reg) +{ +kref_put (®->kref, nouveau_uvma_region_free); +} + +static int +__nouveau_uvma_region_insert (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_region *reg) +{ +u64 addr = reg->va.addr; +u64 range = reg->va.range; +u64 last = addr + range -1 ; +MA_STATE (mas, &uvmm->region_mt, addr, addr); + +if (unlikely (mas_walk (&mas))) { +mas_unlock (&mas); +return -EEXIST ; + } + +if (unlikely (mas.last < last)) { +mas_unlock (&mas); +return -EEXIST ; + } + + mas.index = addr; + mas.last = last; + +mas_store_gfp (&mas, reg,GFP_KERNEL ); + + reg->uvmm = uvmm; + +return 0 ; +} + +static int +nouveau_uvma_region_insert (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_region *reg, +u64 addr,u64 range) +{ +int ret; + + reg->uvmm = uvmm; + reg->va.addr = addr; + reg->va.range = range; + + ret =__nouveau_uvma_region_insert (uvmm, reg); +if (ret) +return ret; + +return 0 ; +} + +static void +nouveau_uvma_region_remove (struct nouveau_uvma_region *reg) +{ +struct nouveau_uvmm *uvmm = reg->uvmm; +MA_STATE (mas, &uvmm->region_mt, reg->va.addr,0 ); + +mas_erase (&mas); +} + +static int +nouveau_uvma_region_create (struct nouveau_uvmm *uvmm, +u64 addr,u64 range) +{ +struct nouveau_uvma_region *reg; +int ret; + +if (!drm_gpuva_interval_empty (&uvmm->umgr, addr, range)) +return -ENOSPC ; + + ret =nouveau_uvma_region_alloc (®); +if (ret) +return ret; + + ret =nouveau_uvma_region_insert (uvmm, reg, addr, range); +if (ret) + goto err_free_region; + + ret =nouveau_uvmm_vmm_sparse_ref (uvmm, addr, range); +if (ret) + goto err_region_remove; + +return 0 ; + +err_region_remove: +nouveau_uvma_region_remove (reg); +err_free_region: +nouveau_uvma_region_put (reg); +return ret; +} + +static struct nouveau_uvma_region * +nouveau_uvma_region_find_first (struct nouveau_uvmm *uvmm, +u64 addr,u64 range) +{ +MA_STATE (mas, &uvmm->region_mt, addr,0 ); + +return mas_find (&mas, addr + range -1 ); +} + +static struct nouveau_uvma_region * +nouveau_uvma_region_find (struct nouveau_uvmm *uvmm, +u64 addr,u64 range) +{ +struct nouveau_uvma_region *reg; + + reg =nouveau_uvma_region_find_first (uvmm, addr, range); +if (!reg) +return NULL ; + +if (reg->va.addr != addr || + reg->va.range != range) +return NULL ; + +return reg; +} + +static bool +nouveau_uvma_region_empty (struct nouveau_uvma_region *reg) +{ +struct nouveau_uvmm *uvmm = reg->uvmm; + +return drm_gpuva_interval_empty (&uvmm->umgr, + reg->va.addr, + reg->va.range); +} + +static int +__nouveau_uvma_region_destroy (struct nouveau_uvma_region *reg) +{ +struct nouveau_uvmm *uvmm = reg->uvmm; +u64 addr = reg->va.addr; +u64 range = reg->va.range; + +if (!nouveau_uvma_region_empty (reg)) +return -EBUSY ; + +nouveau_uvma_region_remove (reg); +nouveau_uvmm_vmm_sparse_unref (uvmm, addr, range); +nouveau_uvma_region_put (reg); + +return 0 ; +} + +static int +nouveau_uvma_region_destroy (struct nouveau_uvmm *uvmm, +u64 addr,u64 range) +{ +struct nouveau_uvma_region *reg; + + reg =nouveau_uvma_region_find (uvmm, addr, range); +if (!reg) +return -ENOENT ; + +return __nouveau_uvma_region_destroy (reg); +} + +static void +nouveau_uvma_region_dirty (struct nouveau_uvma_region *reg) +{ + +init_completion (®->complete); + reg->dirty = true; +} + +static void +nouveau_uvma_region_complete (struct nouveau_uvma_region *reg) +{ +complete_all (®->complete); +} + +static void +op_map_prepare_unwind (struct nouveau_uvma *uvma) +{ +nouveau_uvma_gem_put (uvma); +nouveau_uvma_remove (uvma); +nouveau_uvma_free (uvma); +} + +static void +op_unmap_prepare_unwind (struct drm_gpuva *va) +{ +drm_gpuva_insert (va->mgr, va); +} + +static void +nouveau_uvmm_sm_prepare_unwind (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops, +struct drm_gpuva_op *last, +struct uvmm_map_args *args) +{ +struct drm_gpuva_op *op = last; +u64 vmm_get_start = args ? args->addr :0 ; +u64 vmm_get_end = args ? args->addr + args->range :0 ; + +/* Unwind GPUVA space. */ +drm_gpuva_for_each_op_from_reverse (op, ops) { +switch (op->op) { +case DRM_GPUVA_OP_MAP : +op_map_prepare_unwind (new->map); +break ; +case DRM_GPUVA_OP_REMAP : { +struct drm_gpuva_op_remap *r = &op->remap; + +if (r->next) +op_map_prepare_unwind (new->next); + +if (r->prev) +op_map_prepare_unwind (new->prev); + +op_unmap_prepare_unwind (r->unmap->va); +break ; + } +case DRM_GPUVA_OP_UNMAP : +op_unmap_prepare_unwind (op->unmap.va); +break ; +default : +break ; + } + } + +/* Unmap operation don't allocate page tables, hence skip the following
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +146
-0+* page table unwind. +*/ +if (!args) +return ; + +drm_gpuva_for_each_op (op, ops) { +switch (op->op) { +case DRM_GPUVA_OP_MAP : { +u64 vmm_get_range = vmm_get_end - vmm_get_start; + +if (vmm_get_range) +nouveau_uvmm_vmm_put (uvmm, vmm_get_start, + vmm_get_range); +break ; + } +case DRM_GPUVA_OP_REMAP : { +struct drm_gpuva_op_remap *r = &op->remap; +struct drm_gpuva *va = r->unmap->va; +u64 ustart = va->va.addr; +u64 urange = va->va.range; +u64 uend = ustart + urange; + +if (r->prev) + vmm_get_start = uend; + +if (r->next) + vmm_get_end = ustart; + +if (r->prev && r->next) + vmm_get_start = vmm_get_end =0 ; + +break ; + } +case DRM_GPUVA_OP_UNMAP : { +struct drm_gpuva_op_unmap *u = &op->unmap; +struct drm_gpuva *va = u->va; +u64 ustart = va->va.addr; +u64 urange = va->va.range; +u64 uend = ustart + urange; + +/* Nothing to do for mappings we merge with. */ +if (uend == vmm_get_start || + ustart == vmm_get_end) +break ; + +if (ustart > vmm_get_start) { +u64 vmm_get_range = ustart - vmm_get_start; + +nouveau_uvmm_vmm_put (uvmm, vmm_get_start, + vmm_get_range); + } + vmm_get_start = uend; +break ; + } +default : +break ; + } + +if (op == last) +break ; + } +} + +static void +nouveau_uvmm_sm_map_prepare_unwind (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops, +u64 addr,u64 range) +{ +struct drm_gpuva_op *last =drm_gpuva_last_op (ops); +struct uvmm_map_args args = { + .addr = addr, + .range = range, + }; + +nouveau_uvmm_sm_prepare_unwind (uvmm, new, ops, last, &args); +} + +static void +nouveau_uvmm_sm_unmap_prepare_unwind (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops) +{ +struct drm_gpuva_op *last =drm_gpuva_last_op (ops); + +nouveau_uvmm_sm_prepare_unwind (uvmm, new, ops, last,NULL ); +} + +static int +op_map_prepare (struct nouveau_uvmm *uvmm, +struct nouveau_uvma **puvma, +struct drm_gpuva_op_map *m, +struct uvmm_map_args *args) +{ +struct nouveau_uvma *uvma; +int ret; + + ret =nouveau_uvma_alloc (&uvma); +if (ret) + goto err; + + ret =nouveau_uvma_insert (uvmm, uvma, args->region, + m->gem.obj, m->gem.offset, + m->va.addr, m->va.range, + args->kind); +if (ret) + goto err_free_uvma; + +/* Keep a reference until this uvma is destroyed. */ +nouveau_uvma_gem_get (uvma); + + *puvma = uvma; +return 0 ; + +err_free_uvma: +nouveau_uvma_free (uvma); +err: + *puvma =NULL ; +return ret; +} + +static void +op_unmap_prepare (struct drm_gpuva_op_unmap *u) +{ +struct nouveau_uvma *uvma =uvma_from_va (u->va); + +nouveau_uvma_remove (uvma); +} + +static int +nouveau_uvmm_sm_prepare (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops, +struct uvmm_map_args *args) +{ +struct drm_gpuva_op *op; +u64 vmm_get_start = args ? args->addr :0 ; +u64 vmm_get_end = args ? args->addr + args->range :0 ; +int ret; + +drm_gpuva_for_each_op (op, ops) { +switch (op->op) { +case DRM_GPUVA_OP_MAP : { +u64 vmm_get_range = vmm_get_end - vmm_get_start; + + ret = op_map_prepare(uvmm, &new->map, &op->map,args);
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +5
-0+if (ret) + goto unwind; + +if (args && vmm_get_range) { + ret = nouveau_uvmm_vmm_get(uvmm,
vmm_get_start,drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +22
-0+ vmm_get_range); +if (ret) { +op_map_prepare_unwind (new->map); + goto unwind; + } + } +break ; + } +case DRM_GPUVA_OP_REMAP : { +struct drm_gpuva_op_remap *r = &op->remap; +struct drm_gpuva *va = r->unmap->va; +struct uvmm_map_args remap_args = { + .kind =uvma_from_va (va)->kind, + }; +u64 ustart = va->va.addr; +u64 urange = va->va.range; +u64 uend = ustart + urange; + +op_unmap_prepare (r->unmap); + +if (r->prev) { + ret = op_map_prepare(uvmm, &new->prev,
r->prev,drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +10
-0+ &remap_args); +if (ret) + goto unwind; + +if (args) + vmm_get_start = uend; + } + +if (r->next) { + ret = op_map_prepare(uvmm, &new->next,
r->next,drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +37
-0+ &remap_args); +if (ret) { +if (r->prev) +op_map_prepare_unwind (new->prev); + goto unwind; + } + +if (args) + vmm_get_end = ustart; + } + +if (args && (r->prev && r->next)) + vmm_get_start = vmm_get_end =0 ; + +break ; + } +case DRM_GPUVA_OP_UNMAP : { +struct drm_gpuva_op_unmap *u = &op->unmap; +struct drm_gpuva *va = u->va; +u64 ustart = va->va.addr; +u64 urange = va->va.range; +u64 uend = ustart + urange; + +op_unmap_prepare (u); + +if (!args) +break ; + +/* Nothing to do for mappings we merge with. */ +if (uend == vmm_get_start || + ustart == vmm_get_end) +break ; + +if (ustart > vmm_get_start) { +u64 vmm_get_range = ustart - vmm_get_start; + + ret = nouveau_uvmm_vmm_get(uvmm,
vmm_get_start,drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +59
-0+ vmm_get_range); +if (ret) { +op_unmap_prepare_unwind (va); + goto unwind; + } + } + vmm_get_start = uend; + +break ; + } +default : + ret = -EINVAL ; + goto unwind; + } + } + +return 0 ; + +unwind: +if (op !=drm_gpuva_first_op (ops)) +nouveau_uvmm_sm_prepare_unwind (uvmm, new, ops, +drm_gpuva_prev_op (op), + args); +return ret; +} + +static int +nouveau_uvmm_sm_map_prepare (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct nouveau_uvma_region *region, +struct drm_gpuva_ops *ops, +u64 addr,u64 range,u8 kind) +{ +struct uvmm_map_args args = { + .region = region, + .addr = addr, + .range = range, + .kind = kind, + }; + +return nouveau_uvmm_sm_prepare (uvmm, new, ops, &args); +} + +static int +nouveau_uvmm_sm_unmap_prepare (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops) +{ +return nouveau_uvmm_sm_prepare (uvmm, new, ops,NULL ); +} + +static struct drm_gem_object * +op_gem_obj (struct drm_gpuva_op *op) +{ +switch (op->op) { +case DRM_GPUVA_OP_MAP : +return op->map.gem.obj; +case DRM_GPUVA_OP_REMAP : +/* Actually, we're looking for the GEMs backing remap.prev and
+ * remap.next, but since this is a remap they're identical
todrivers/gpu/drm/nouveau/nouveau_uvmm.c new file +130
-0+* the GEM backing the unmapped GPUVA. +*/ +return op->remap.unmap->va->gem.obj; +case DRM_GPUVA_OP_UNMAP : +return op->unmap.va->gem.obj; +default : +WARN (1 ,"Unknown operation.\n" ); +return NULL ; + } +} + +static void +op_map (struct nouveau_uvma *uvma) +{ +struct nouveau_bo *nvbo =nouveau_gem_object (uvma->va.gem.obj); + +nouveau_uvma_map (uvma,nouveau_mem (nvbo->bo.resource)); +} + +static void +op_unmap (struct drm_gpuva_op_unmap *u) +{ +struct drm_gpuva *va = u->va; +struct nouveau_uvma *uvma =uvma_from_va (va); + +/* nouveau_uvma_unmap() does not unmap if backing BO is evicted. */ +if (!u->keep) +nouveau_uvma_unmap (uvma); +} + +static void +op_unmap_range (struct drm_gpuva_op_unmap *u, +u64 addr,u64 range) +{ +struct nouveau_uvma *uvma =uvma_from_va (u->va); +bool sparse = !!uvma->region; + +if (!drm_gpuva_invalidated (u->va)) +nouveau_uvmm_vmm_unmap (uvma->uvmm, addr, range, sparse); +} + +static void +op_remap (struct drm_gpuva_op_remap *r, +struct nouveau_uvma_prealloc *new) +{ +struct drm_gpuva_op_unmap *u = r->unmap; +struct nouveau_uvma *uvma =uvma_from_va (u->va); +u64 addr = uvma->va.va.addr; +u64 range = uvma->va.va.range; + +if (r->prev) + addr = r->prev->va.addr + r->prev->va.range; + +if (r->next) + range = r->next->va.addr - addr; + +op_unmap_range (u, addr, range); +} + +static int +nouveau_uvmm_sm (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops) +{ +struct drm_gpuva_op *op; + +drm_gpuva_for_each_op (op, ops) { +switch (op->op) { +case DRM_GPUVA_OP_MAP : +op_map (new->map); +break ; +case DRM_GPUVA_OP_REMAP : +op_remap (&op->remap, new); +break ; +case DRM_GPUVA_OP_UNMAP : +op_unmap (&op->unmap); +break ; +default : +break ; + } + } + +return 0 ; +} + +static int +nouveau_uvmm_sm_map (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops) +{ +return nouveau_uvmm_sm (uvmm, new, ops); +} + +static int +nouveau_uvmm_sm_unmap (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops) +{ +return nouveau_uvmm_sm (uvmm, new, ops); +} + +static void +nouveau_uvmm_sm_cleanup (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops,bool unmap) +{ +struct drm_gpuva_op *op; + +drm_gpuva_for_each_op (op, ops) { +switch (op->op) { +case DRM_GPUVA_OP_MAP : +break ; +case DRM_GPUVA_OP_REMAP : { +struct drm_gpuva_op_remap *r = &op->remap; +struct drm_gpuva_op_map *p = r->prev; +struct drm_gpuva_op_map *n = r->next; +struct drm_gpuva *va = r->unmap->va; +struct nouveau_uvma *uvma =uvma_from_va (va); + +if (unmap) { +u64 addr = va->va.addr; +u64 end = addr + va->va.range; + +if (p) + addr = p->va.addr + p->va.range; + +if (n) + end = n->va.addr; + + nouveau_uvmm_vmm_put(uvmm, addr, end -addr);
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +42
-0+ } + +nouveau_uvma_gem_put (uvma); +nouveau_uvma_free (uvma); +break ; + } +case DRM_GPUVA_OP_UNMAP : { +struct drm_gpuva_op_unmap *u = &op->unmap; +struct drm_gpuva *va = u->va; +struct nouveau_uvma *uvma =uvma_from_va (va); + +if (unmap) +nouveau_uvma_vmm_put (uvma); + +nouveau_uvma_gem_put (uvma); +nouveau_uvma_free (uvma); +break ; + } +default : +break ; + } + } +} + +static void +nouveau_uvmm_sm_map_cleanup (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops) +{ +nouveau_uvmm_sm_cleanup (uvmm, new, ops, false); +} + +static void +nouveau_uvmm_sm_unmap_cleanup (struct nouveau_uvmm *uvmm, +struct nouveau_uvma_prealloc *new, +struct drm_gpuva_ops *ops) +{ +nouveau_uvmm_sm_cleanup (uvmm, new, ops, true); +} + +static int +nouveau_uvmm_validate_range(struct nouveau_uvmm *uvmm, u64 addr, u64range)
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +78
-0+{ +u64 end = addr + range; +u64 unmanaged_end = uvmm->unmanaged_addr + + uvmm->unmanaged_size; + +if (addr & ~PAGE_MASK ) +return -EINVAL ; + +if (range & ~PAGE_MASK ) +return -EINVAL ; + +if (end <= addr) +return -EINVAL ; + +if (addr <NOUVEAU_VA_SPACE_START || + end >NOUVEAU_VA_SPACE_END ) +return -EINVAL ; + +if (addr < unmanaged_end && + end > uvmm->unmanaged_addr) +return -EINVAL ; + +return 0 ; +} + +static int +nouveau_uvmm_bind_job_alloc (struct nouveau_uvmm_bind_job **pjob) +{ + *pjob =kzalloc (sizeof (**pjob),GFP_KERNEL ); +if (!*pjob) +return -ENOMEM ; + +kref_init (&(*pjob)->kref); + +return 0 ; +} + +static void +nouveau_uvmm_bind_job_free (struct kref *kref) +{ +struct nouveau_uvmm_bind_job *job = + container_of(kref, struct nouveau_uvmm_bind_job, kref); + +nouveau_job_free (&job->base); +kfree (job); +} + +static void +nouveau_uvmm_bind_job_get (struct nouveau_uvmm_bind_job *job) +{ +kref_get (&job->kref); +} + +static void +nouveau_uvmm_bind_job_put (struct nouveau_uvmm_bind_job *job) +{ +kref_put (&job->kref, nouveau_uvmm_bind_job_free); +} + +static int +bind_validate_op (struct nouveau_job *job, +struct bind_job_op *op) +{ +struct nouveau_uvmm *uvmm =nouveau_cli_uvmm (job->cli); +struct drm_gem_object *obj = op->gem.obj; + +if (op->op ==OP_MAP ) { +if (op->gem.offset & ~PAGE_MASK ) +return -EINVAL ; + +if (obj->size <= op->gem.offset) +return -EINVAL ; + +if (op->va.range > (obj->size - op->gem.offset)) +return -EINVAL ; + } + + return nouveau_uvmm_validate_range(uvmm, op->va.addr,
op->va.range);drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +93
-0+} + +static void +bind_validate_map_sparse (struct nouveau_job *job,u64 addr,u64 range) +{ +struct nouveau_uvmm_bind_job *bind_job; +struct nouveau_sched_entity *entity = job->entity; +struct bind_job_op *op; +u64 end = addr + range; + +again: +spin_lock (&entity->job.list.lock); +list_for_each_entry (bind_job, &entity->job.list.head, entry) { +list_for_each_op (op, &bind_job->ops) { +if (op->op ==OP_UNMAP ) { +u64 op_addr = op->va.addr; +u64 op_end = op_addr + op->va.range; + +if (!(end <= op_addr || addr >= op_end)) { +nouveau_uvmm_bind_job_get (bind_job); +spin_unlock (&entity->job.list.lock); +wait_for_completion (&bind_job->complete); +nouveau_uvmm_bind_job_put (bind_job); + goto again; + } + } + } + } +spin_unlock (&entity->job.list.lock); +} + +static int +bind_validate_map_common (struct nouveau_job *job,u64 addr,u64 range, +bool sparse) +{ +struct nouveau_uvmm *uvmm =nouveau_cli_uvmm (job->cli); +struct nouveau_uvma_region *reg; +u64 reg_addr, reg_end; +u64 end = addr + range; + +again: +nouveau_uvmm_lock (uvmm); + reg =nouveau_uvma_region_find_first (uvmm, addr, range); +if (!reg) { +nouveau_uvmm_unlock (uvmm); +return 0 ; + } + +/* Generally, job submits are serialized, hence only +* dirty regions can be modified concurrently. */ +if (reg->dirty) { +nouveau_uvma_region_get (reg); +nouveau_uvmm_unlock (uvmm); +wait_for_completion (®->complete); +nouveau_uvma_region_put (reg); + goto again; + } +nouveau_uvmm_unlock (uvmm); + +if (sparse) +return -ENOSPC ; + + reg_addr = reg->va.addr; + reg_end = reg_addr + reg->va.range; + +/* Make sure the mapping is either outside of a +* region or fully enclosed by a region. +*/ +if (reg_addr > addr || reg_end < end) +return -ENOSPC ; + +return 0 ; +} + +static int +bind_validate_region (struct nouveau_job *job) +{ +struct nouveau_uvmm_bind_job *bind_job =to_uvmm_bind_job (job); +struct bind_job_op *op; +int ret; + +list_for_each_op (op, &bind_job->ops) { +u64 op_addr = op->va.addr; +u64 op_range = op->va.range; +bool sparse = false; + +switch (op->op) { +case OP_MAP_SPARSE : + sparse = true; +bind_validate_map_sparse (job, op_addr, op_range); + fallthrough; +case OP_MAP : + ret = bind_validate_map_common(job, op_addr,
op_range,drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +14
-0+ sparse); +if (ret) +return ret; +break ; +default : +break ; + } + } + +return 0 ; +} + +static void +bind_link_gpuvas(struct drm_gpuva_ops *ops, struct nouveau_uvma_prealloc*new)
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +48
-0+{ +struct drm_gpuva_op *op; + +drm_gpuva_for_each_op (op, ops) { +switch (op->op) { +case DRM_GPUVA_OP_MAP : +drm_gpuva_link (&new->map->va); +break ; +case DRM_GPUVA_OP_REMAP : +if (op->remap.prev) +drm_gpuva_link (&new->prev->va); +if (op->remap.next) +drm_gpuva_link (&new->next->va); +drm_gpuva_unlink (op->remap.unmap->va); +break ; +case DRM_GPUVA_OP_UNMAP : +drm_gpuva_unlink (op->unmap.va); +break ; +default : +break ; + } + } +} + +static int +nouveau_uvmm_bind_job_submit (struct nouveau_job *job) +{ +struct nouveau_uvmm *uvmm =nouveau_cli_uvmm (job->cli); +struct nouveau_uvmm_bind_job *bind_job =to_uvmm_bind_job (job); +struct nouveau_sched_entity *entity = job->entity; +struct drm_exec *exec = &job->exec; +struct bind_job_op *op; +int ret; + +list_for_each_op (op, &bind_job->ops) { +if (op->op ==OP_MAP ) { + op->gem.obj =drm_gem_object_lookup (job->file_priv, + op->gem.handle); +if (!op->gem.obj) +return -ENOENT ; + } + + ret =bind_validate_op (job, op); +if (ret) +return ret; + } + +/* If a sparse region or mapping overlaps a dirty region, we need to
+ * wait for the region to complete the unbind process. This is due
todrivers/gpu/drm/nouveau/nouveau_uvmm.c new file +8
-0+* how page table management is currently implemented. A future +* implementation might change this. +*/ + ret =bind_validate_region (job); +if (ret) +return ret; + +/* Once we start modifying the GPU VA space we need to keep holding the
+ * uvmm lock until we can't fail anymore. This is due to the set
of GPU
+ * VA space changes must appear atomically and we need to be able
todrivers/gpu/drm/nouveau/nouveau_uvmm.c new file +15
-0+* unwind all GPU VA space changes on failure. +*/ +nouveau_uvmm_lock (uvmm); +list_for_each_op (op, &bind_job->ops) { +switch (op->op) { +case OP_MAP_SPARSE : + ret =nouveau_uvma_region_create (uvmm, + op->va.addr, + op->va.range); +if (ret) + goto unwind_continue; + +break ; +case OP_UNMAP_SPARSE : + op->reg = nouveau_uvma_region_find(uvmm,
op->va.addr,drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +7
-0+ op->va.range); +if (!op->reg || op->reg->dirty) { + ret = -ENOENT ; + goto unwind_continue; + } + + op->ops =drm_gpuva_sm_unmap_ops_create(&uvmm->umgr,
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +37
-0+ op->va.addr, + op->va.range); +if (IS_ERR (op->ops)) { + ret =PTR_ERR (op->ops); + goto unwind_continue; + } + + ret =nouveau_uvmm_sm_unmap_prepare (uvmm, &op->new, + op->ops); +if (ret) { +drm_gpuva_ops_free (&uvmm->umgr, op->ops); + op->ops =NULL ; + op->reg =NULL ; + goto unwind_continue; + } + +nouveau_uvma_region_dirty (op->reg); + +break ; +case OP_MAP : { +struct nouveau_uvma_region *reg; + + reg =nouveau_uvma_region_find_first (uvmm, + op->va.addr, + op->va.range); +if (reg) { +u64 reg_addr = reg->va.addr; +u64 reg_end = reg_addr + reg->va.range; +u64 op_addr = op->va.addr; +u64 op_end = op_addr + op->va.range; + +if (unlikely (reg->dirty)) { + ret = -EINVAL ; + goto unwind_continue; + } + +/* Make sure the mapping is either outside of a
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +3
-0+* region or fully enclosed by a region. +*/ + if (reg_addr > op_addr || reg_end <
op_end) {drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +20
-0+ ret = -ENOSPC ; + goto unwind_continue; + } + } + + op->ops =drm_gpuva_sm_map_ops_create (&uvmm->umgr, + op->va.addr, + op->va.range, + op->gem.obj, + op->gem.offset); +if (IS_ERR (op->ops)) { + ret =PTR_ERR (op->ops); + goto unwind_continue; + } + + ret = nouveau_uvmm_sm_map_prepare(uvmm, &op->new, + reg, op->ops, + op->va.addr, + op->va.range, + op->flags &
0xff);drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +10
-0+if (ret) { +drm_gpuva_ops_free (&uvmm->umgr, op->ops); + op->ops =NULL ; + goto unwind_continue; + } + +break ; + } +case OP_UNMAP : + op->ops =drm_gpuva_sm_unmap_ops_create(&uvmm->umgr,
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +31
-0+ op->va.addr, + op->va.range); +if (IS_ERR (op->ops)) { + ret =PTR_ERR (op->ops); + goto unwind_continue; + } + + ret =nouveau_uvmm_sm_unmap_prepare (uvmm, &op->new, + op->ops); +if (ret) { +drm_gpuva_ops_free (&uvmm->umgr, op->ops); + op->ops =NULL ; + goto unwind_continue; + } + +break ; +default : + ret = -EINVAL ; + goto unwind_continue; + } + } + +drm_exec_until_all_locked (exec) { +list_for_each_op (op, &bind_job->ops) { +struct drm_gpuva_op *va_op; + +if (IS_ERR_OR_NULL (op->ops)) +continue ; + +drm_gpuva_for_each_op (va_op, op->ops) { + struct drm_gem_object *obj =
op_gem_obj(va_op);drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +24
-0+ +if (unlikely (!obj)) +continue ; + + ret =drm_exec_prepare_obj (exec, obj,1 ); +drm_exec_retry_on_contention (exec); +if (ret) { + op =list_last_op (&bind_job->ops); + goto unwind; + } + } + } + } + +list_for_each_op (op, &bind_job->ops) { +struct drm_gpuva_op *va_op; + +if (IS_ERR_OR_NULL (op->ops)) +continue ; + +drm_gpuva_for_each_op (va_op, op->ops) { +struct drm_gem_object *obj =op_gem_obj (va_op); + +/* Don't validate GEMs backing mappings we're about to
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +21
-0+* unmap, it's not worth the effort. +*/ +if (unlikely (va_op->op ==DRM_GPUVA_OP_UNMAP )) +continue ; + +if (unlikely (!obj)) +continue ; + + ret =nouveau_bo_validate (nouveau_gem_object (obj), + true, false); +if (ret) { + op =list_last_op (&bind_job->ops); + goto unwind; + } + } + } + +/* Link and unlink GPUVAs while holding the dma_resv lock. +* +* As long as we validate() all GEMs and add fences to all GEMs DMA +* reservations backing map and remap operations we can be sure there
+ * won't be any concurrent (in)validations during job execution,
hencedrivers/gpu/drm/nouveau/nouveau_uvmm.c new file +4
-0+* we're safe to check drm_gpuva_invalidated() within the fence +* signalling critical path without holding a separate lock. +* +* GPUVAs about to be unmapped are safe as well, since they're unlinked
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +3
-0+* already. +* +* GEMs from map and remap operations must be validated before linking
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +5
-0+* their corresponding mappings to prevent the actual PT update to +* happen right away in validate() rather than asynchronously as +* intended. +* +* Note that after linking and unlinking the GPUVAs in this loop this
+ * function cannot fail anymore, hence there is no need for an
unwinddrivers/gpu/drm/nouveau/nouveau_uvmm.c new file +33
-0+* path. +*/ +list_for_each_op (op, &bind_job->ops) { +switch (op->op) { +case OP_UNMAP_SPARSE : +case OP_MAP : +case OP_UNMAP : +bind_link_gpuvas (op->ops, &op->new); +break ; +default : +break ; + } + } +nouveau_uvmm_unlock (uvmm); + +spin_lock (&entity->job.list.lock); +list_add (&bind_job->entry, &entity->job.list.head); +spin_unlock (&entity->job.list.lock); + +return 0 ; + +unwind_continue: + op =list_prev_op (op); +unwind: +list_for_each_op_from_reverse (op, &bind_job->ops) { +switch (op->op) { +case OP_MAP_SPARSE : +nouveau_uvma_region_destroy (uvmm, op->va.addr, + op->va.range); +break ; +case OP_UNMAP_SPARSE : +__nouveau_uvma_region_insert (uvmm, op->reg); + nouveau_uvmm_sm_unmap_prepare_unwind(uvmm,
&op->new,drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +10
-0+ op->ops); +break ; +case OP_MAP : +nouveau_uvmm_sm_map_prepare_unwind (uvmm, &op->new, + op->ops, + op->va.addr, + op->va.range); +break ; +case OP_UNMAP : + nouveau_uvmm_sm_unmap_prepare_unwind(uvmm,
&op->new,drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +35
-0+ op->ops); +break ; + } + +drm_gpuva_ops_free (&uvmm->umgr, op->ops); + op->ops =NULL ; + op->reg =NULL ; + } + +nouveau_uvmm_unlock (uvmm); +return ret; +} + +static struct dma_fence * +nouveau_uvmm_bind_job_run (struct nouveau_job *job) +{ +struct nouveau_uvmm_bind_job *bind_job =to_uvmm_bind_job (job); +struct nouveau_uvmm *uvmm =nouveau_cli_uvmm (job->cli); +struct bind_job_op *op; +int ret =0 ; + +list_for_each_op (op, &bind_job->ops) { +switch (op->op) { +case OP_MAP_SPARSE : +/* noop */ +break ; +case OP_MAP : + ret =nouveau_uvmm_sm_map (uvmm, &op->new, op->ops); +if (ret) + goto out; +break ; +case OP_UNMAP_SPARSE : + fallthrough; +case OP_UNMAP : + ret = nouveau_uvmm_sm_unmap(uvmm, &op->new,
op->ops);drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +26
-0+if (ret) + goto out; +break ; + } + } + +out: +if (ret) +NV_PRINTK (err, job->cli,"bind job failed: %d\n" , ret); +return ERR_PTR (ret); +} + +static void +nouveau_uvmm_bind_job_free_work_fn (struct work_struct *work) +{ +struct nouveau_uvmm_bind_job *bind_job = + container_of(work, struct nouveau_uvmm_bind_job, work); +struct nouveau_job *job = &bind_job->base; +struct nouveau_uvmm *uvmm =nouveau_cli_uvmm (job->cli); +struct nouveau_sched_entity *entity = job->entity; +struct bind_job_op *op, *next; + +list_for_each_op (op, &bind_job->ops) { +struct drm_gem_object *obj = op->gem.obj; + +/* When nouveau_uvmm_bind_job_submit() fails op->ops and
op->regdrivers/gpu/drm/nouveau/nouveau_uvmm.c new file +9
-0+* will be NULL, hence skip the cleanup. +*/ Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
&op->new,drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +20
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
&op->new,drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +19
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
list to
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +378
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
memory.");
drivers/gpu/drm/nouveau/nouveau_uvmm.c new file +8
-0Lines of this file are not shown. View the raw message.
drivers/gpu/drm/nouveau/nouveau_uvmm.h new file +79
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
nouveau_uvmm_bind_job, base)drivers/gpu/drm/nouveau/nouveau_uvmm.h new file +9
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
*mem);
drivers/gpu/drm/nouveau/nouveau_uvmm.h new file +19
-1Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
2.41.0
Part of the diff is not shown here. View the raw message.
Dave Airlie
maintainerRe: [PATCH drm-misc-next v8 11/12] drm/nouveau: implement new VM_BIND uAPI
Quoted 42 lines · Faith Ekstrand
Quoted 15 lines · Danilo Krummrich
This commit provides the implementation for the new uapi motivated by the Vulkan API. It allows user mode drivers (UMDs) to:
1) Initialize a GPU virtual address (VA) space via the new
DRM_IOCTL_NOUVEAU_VM_INITioctl for UMDs to specify the portion of VA space managed by the kernel and userspace, respectively.2) Allocate and free a VA space region as well as bind and unbind memory to the GPUs VA space via the new
DRM_IOCTL_NOUVEAU_VM_BINDioctl. UMDs can request the named operations to be processed either synchronously or asynchronously. It supports DRM syncobjs (incl. timelines) as synchronization mechanism. The management of the GPU VA mappings is implemented with the DRM GPU VA manager.Quoted 7 lines
3) Execute push buffers with the new
DRM_IOCTL_NOUVEAU_EXECioctl. The execution happens asynchronously. It supports DRM syncobj (incl. timelines) as synchronization mechanism. DRM GEM object locking is handled withdrm_exec.Both,
DRM_IOCTL_NOUVEAU_VM_BINDandDRM_IOCTL_NOUVEAU_EXEC, use the DRM GPU scheduler for the asynchronous paths.IDK where the best place to talk about this is but this seems as good as any.
I've been looking into why the Vulkan CTS runs about 2x slower for me on the new UAPI and I created a little benchmark to facilitate testing:
https://gitlab.freedesktop.org/mesa/crucible/-/merge_requests/141
The test, roughly, does the following:
1. Allocates and binds 1000 BOs
2. Constructs a pushbuf that executes a no-op compute shader.
3. Does a single EXEC/wait combo to warm up the kernel
4. Loops 10,000 times, doingSYNCOBJ_RESET(fast), EXEC, and thenSYNCOBJ_WAITand times the loopOf course, there's a bit of userspace driver overhead but that's negledgable.
If you drop the top patch which allocates 1k buffers, the submit time on the old uAPI is 54 us/exec vs. 66 us/exec on the new UAPI. This includes the time to do a
SYNCOBJ_RESET(fast), EXEC, andSYNCOBJ_WAIT. The Intel driver, by comparison, is 33us/exec so it's not syncobj overhead. This is a bit concerning (you'd think the new thing would be faster) but what really has me concerned is the 1k buffer case.
Quoted 7 lines
If you include the top patch in the crucible MR, it allocates 1000 BOs and
VM_BINDsthem. All the binding is done before the warmup EXEC. Suddenly, the submit time jumps to 257 us/exec with the new UAPI. The old UAPI is much worse (1134 us/exec) but that's not the point. Once we've done the first EXEC and created our VM bindings, the cost per EXEC shouldn't change at all based on the number of BOs bound. Part of the point ofVM_BINDis to get all that binding logic and BO walking off the EXEC path.Normally, I wouldn't be too worried about a little performance problem like this. This is the first implementation and we can improve it later. I get that. However, I suspect the solution to this problem involves more UAPI and I want to make sure we have it all before we call this all done and dusted and land it.
The way AMD solves this problem as well as the new Xe driver for Intel is to have a concept of internal vs. external BOs. Basically, there's an INTERNAL bit specified somewhere in BO creation that has a few userspace implications:
1. In the Xe world where VMs are objects, INTERNAL BOs are assigned a VM on creation and can never be bound to any other VM.
2. Any attempt to export an INTERNAL BO via prime or a similar mechanism will fail with-EINVAL(I think?).
Okay I've done a first pass at implementing this,
https://gitlab.freedesktop.org/nouvelles/kernel/-/commit/005a0005ff80ec85f3e9a314cae0576b7441076c
Danilo we should probably pick that up for the next iteration. with corresponding mesa work it gets the latency for me from 180us down to 70us.
Dave.
Re: [PATCH drm-misc-next v8 11/12] drm/nouveau: implement new VM_BIND uAPI
In reply to Faith EkstrandQuoted 49 lines · Faith Ekstrand
On Wed, Jul 19, 2023 at 7:15 PM Danilo Krummrich <[email protected] <mailto:[email protected]>> wrote:
This commit provides the implementation for the new uapi motivated by the
Vulkan API. It allows user mode drivers (UMDs) to:1) Initialize a GPU virtual address (VA) space via the new
DRM_IOCTL_NOUVEAU_VM_INITioctl for UMDs to specify the portion
of VA
space managed by the kernel and userspace, respectively.2) Allocate and free a VA space region as well as bind and unbind memory to the GPUs VA space via the new
DRM_IOCTL_NOUVEAU_VM_BINDioctl. UMDs can request the named operations to be processed either synchronously or asynchronously. It supports DRM syncobjs (incl. timelines) as synchronization mechanism. The management
of the
GPU VA mappings is implemented with the DRM GPU VA manager.3) Execute push buffers with the new
DRM_IOCTL_NOUVEAU_EXECioctl. The execution happens asynchronously. It supports DRM syncobj (incl. timelines) as synchronization mechanism. DRM GEM object locking is handled withdrm_exec.Both,
DRM_IOCTL_NOUVEAU_VM_BINDandDRM_IOCTL_NOUVEAU_EXEC, use the DRM GPU scheduler for the asynchronous paths.IDK where the best place to talk about this is but this seems as good as any.
I've been looking into why the Vulkan CTS runs about 2x slower for me on the new UAPI and I created a little benchmark to facilitate testing:
https://gitlab.freedesktop.org/mesa/crucible/-/merge_requests/141 <https://gitlab.freedesktop.org/mesa/crucible/-/merge_requests/141>
The test, roughly, does the following:
1. Allocates and binds 1000 BOs
2. Constructs a pushbuf that executes a no-op compute shader.
3. Does a single EXEC/wait combo to warm up the kernel
4. Loops 10,000 times, doingSYNCOBJ_RESET(fast), EXEC, and thenSYNCOBJ_WAITand times the loopOf course, there's a bit of userspace driver overhead but that's negledgable.
If you drop the top patch which allocates 1k buffers, the submit time on the old uAPI is 54 us/exec vs. 66 us/exec on the new UAPI. This includes the time to do a
SYNCOBJ_RESET(fast), EXEC, and SYNCOBJ_WAIT.The Intel driver, by comparison, is 33us/exec so it's not syncobj overhead. This is a bit concerning (you'd think the new thing would be faster) but what really has me concerned is the 1k buffer case.If you include the top patch in the crucible MR, it allocates 1000 BOs and
VM_BINDsthem. All the binding is done before the warmup EXEC. Suddenly, the submit time jumps to 257 us/exec with the new UAPI. The old UAPI is much worse (1134 us/exec) but that's not the point. Once we've done the first EXEC and created our VM bindings, the cost per EXEC shouldn't change at all based on the number of BOs bound. Part of the point ofVM_BINDis to get all that binding logic and BO walking off the EXEC path.Normally, I wouldn't be too worried about a little performance problem like this. This is the first implementation and we can improve it later. I get that. However, I suspect the solution to this problem involves more UAPI and I want to make sure we have it all before we call this all done and dusted and land it.
Quoted 7 lines
The way AMD solves this problem as well as the new Xe driver for Intel is to have a concept of internal vs. external BOs. Basically, there's an INTERNAL bit specified somewhere in BO creation that has a few userspace implications:
1. In the Xe world where VMs are objects, INTERNAL BOs are assigned a
VM on creation and can never be bound to any other VM.
2. Any attempt to export an INTERNAL BO via prime or a similar
mechanism will fail with-EINVAL(I think?).Inside the kernel driver, all the internal BOs on a VM (or DRM file in the case of nouveau/AMD since they don't have VM objects) share a single
dma_resvwhich allows you to avoid having to walk lists of BOs and take locks on every exec. Instead, you can just look at the fences on thedma_resvfor the VM. There's still a BO list associated with the VM for external BOs but, in most Vulkan applications, there are less than a half dozen external BOs total. Meanwhile, the hundreds or thousands of BOs used entirely internally to the application basically count as one BO when it comes to locking overhead.
I am aware of that and I have some WIP patches [1] to generalize a common dma-resv within the GPUVA manager which basically represents a GPU-VM. It also keeps track of external GEM objects and evicted objects, such that on EXEC we only need to validate objects needing validation, rather than all of them. Hence, it should be faster than with Daves patch having a common dma-resv only.
In [1] I also picked up Daves code to allow for noop jobs to be submitted as well as the NOUVEAU_GEM_DOMAIN_NO_SHARE flag.
This seems to work fine with yours and Daves latest mesa work (670c301a9845a3fc795fd48a1e6714e75b388245).
Your crucible bench.submit-latency test goes down to 51us on my machine with those patches.
I am unsure though, if we should aim for a common solution within the GPUVA manager directly or if we should do it driver specific in a first shot. I discussed this patch with Matt and I know that XE looks for having a generalized solution as well. However, it surely needs some more care and polish and feedback from other drivers perspective.
[1] https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next-vm-resv
Quoted 2 lines
I'm not actually trying to dictate kernel driver design here. If one
dma_resvdoesn't work, fine. I don't care as long as EXEC scales. However, given that the solution in all the other drivers involves a BO create flag nouveau doesn't have, we need to either add that or prove that we can get EXEC to scale without it.
From #nouveau:
<gfxstrand> CTSing now
<gfxstrand> It looks like it's still going to take 1.5 hours.
I may have an idea what could be the issue, let me explain.
Currently, there is a single drm_gpu_scheduler having a drm_sched_entity per client (for VM_BIND jobs) and a drm_sched_entity per channel (for EXEC jobs).
For VM_BIND jobs the corresponding PT[E]s are allocated before the job is pushed to the corresponding drm_sched_entity. The PT[E]s are freed by the schedulers free() callback pushing work to a single threaded workqueue doing the actual free. (We can't do it in the free() callback directly, since to free PT[E]s we need to hold a mutex we also need to hold while allocating them.)
Because of how the page table handling in Nouveau is implemented currently there are some ordering restrictions when it comes to allocating and freeing PT[E]s. For instance, we can't allocate PT[E]s for sparse regions before the PT[E]s of previously removed memory backed mappings *within the same address range* aren't freed. The same applies vice versa and for sparse mapping replacing sparse mapping. For memory backed mappings (also for those within sparse regions) we do *not* have such ordering requirements.
So, let's assume userspace removes a sparse region A[0x0, 0x8000000] and asks for a couple of new memory backed mappings within or crossing this range; the kernel needs to wait for A not only to be unmapped, but also the backing PT[E]s to be freed before it can even allocate the PT[E]s for the new memory backed mappings.
Now, let's have a look what the gpu schedulers main loop does. Before picking the next entity to schedule a job for, it tries to fetch the first job from the pending_list and checks whether its dma-fence is signaled already and whether the job can be cleaned up. Subsequent jobs on the pending_list are not taken into consideration. Hence, it might well be that the first job on the pending_list isn't signaled yet, but subsequent jobs are and hence *could* be cleaned up.
Normally, this shouldn't be a problem, since we wouldn't really care *when* resources are cleaned up as long as they are eventually. However, with the ordering restrictions the page table handling gives us we might actually care about the "when".
For instance, it could happen that the first job on the pending list is a rather long running EXEC job (1) scheduled from client A on some channel. The next job on the pending list could be a VM_BIND job (2) from client B removing a sparse region, which is finished already but is blocked to be cleaned up until the EXEC job (1) from client A is finished and cleaned up. Now, a subsequent VM_BIND job (3) from client B creating a new memory backed mapping in the same address range as the sparse region removed by job (2) would need to wait for (2) to be cleaned up. Ultimately, we can expect client B to submit an EXEC job that needs to wait for the corresponding mappings to be created, namely the VM_BIND job (3).
Clearly in order to address this we need to rework the page table handling in Nouveau to get rid of those ordering restrictions.
Temporarily, we could also try to run a secondary drm_gpu_scheduler instance, one for VM_BINDs and one for EXECs maybe...
However, I would not expect this to be an issue in real applications, especially if mesa takes a little care not to re-use certain address space areas right away to avoid running into such wait conditions.
For parallel VK CTS runs I could imagine that we run into such cases from time to time though.
- Danilo
Quoted 3 lines
~Faith
Re: [PATCH drm-misc-next v8 11/12] drm/nouveau: implement new VM_BIND uAPI
Quoted 191 lines · Danilo Krummrich
Quoted 7 lines · Faith Ekstrand
On Wed, Jul 19, 2023 at 7:15 PM Danilo Krummrich <[email protected] <mailto:[email protected]>> wrote:
This commit provides the implementation for the new uapi motivated by the
Vulkan API. It allows user mode drivers (UMDs) to:Quoted 6 lines
1) Initialize a GPU virtual address (VA) space via the new
DRM_IOCTL_NOUVEAU_VM_INITioctl for UMDs to specify the portion
of VA
space managed by the kernel and userspace, respectively.2) Allocate and free a VA space region as well as bind and unbind
memory
Quoted 8 lines
to the GPUs VA space via the new
DRM_IOCTL_NOUVEAU_VM_BINDioctl. UMDs can request the named operations to be processed either synchronously or asynchronously. It supports DRM syncobjs (incl. timelines) as synchronization mechanism. The management
of the
GPU VA mappings is implemented with the DRM GPU VA manager.3) Execute push buffers with the new
DRM_IOCTL_NOUVEAU_EXECioctl.The
Quoted 2 lines
execution happens asynchronously. It supports DRM syncobj (incl. timelines) as synchronization mechanism. DRM GEM object locking
is
Quoted 3 lines
handled with
drm_exec.Both,
DRM_IOCTL_NOUVEAU_VM_BINDandDRM_IOCTL_NOUVEAU_EXEC, use theDRM
Quoted 53 lines
GPU scheduler for the asynchronous paths.
IDK where the best place to talk about this is but this seems as good as any.
I've been looking into why the Vulkan CTS runs about 2x slower for me on the new UAPI and I created a little benchmark to facilitate testing:
https://gitlab.freedesktop.org/mesa/crucible/-/merge_requests/141
<https://gitlab.freedesktop.org/mesa/crucible/-/merge_requests/141>The test, roughly, does the following:
1. Allocates and binds 1000 BOs
2. Constructs a pushbuf that executes a no-op compute shader.
3. Does a single EXEC/wait combo to warm up the kernel
4. Loops 10,000 times, doingSYNCOBJ_RESET(fast), EXEC, and thenSYNCOBJ_WAITand times the loopOf course, there's a bit of userspace driver overhead but that's negledgable.
If you drop the top patch which allocates 1k buffers, the submit time on the old uAPI is 54 us/exec vs. 66 us/exec on the new UAPI. This includes the time to do a
SYNCOBJ_RESET(fast), EXEC, and SYNCOBJ_WAIT.The Intel driver, by comparison, is 33us/exec so it's not syncobj overhead. This is a bit concerning (you'd think the new thing would be faster) but what really has me concerned is the 1k buffer case.If you include the top patch in the crucible MR, it allocates 1000 BOs and
VM_BINDsthem. All the binding is done before the warmup EXEC. Suddenly, the submit time jumps to 257 us/exec with the new UAPI. The old UAPI is much worse (1134 us/exec) but that's not the point. Once we've done the first EXEC and created our VM bindings, the cost per EXEC shouldn't change at all based on the number of BOs bound. Part of the point ofVM_BINDis to get all that binding logic and BO walking off the EXEC path.Normally, I wouldn't be too worried about a little performance problem like this. This is the first implementation and we can improve it later. I get that. However, I suspect the solution to this problem involves more UAPI and I want to make sure we have it all before we call this all done and dusted and land it.
The way AMD solves this problem as well as the new Xe driver for Intel is to have a concept of internal vs. external BOs. Basically, there's an INTERNAL bit specified somewhere in BO creation that has a few userspace implications:
1. In the Xe world where VMs are objects, INTERNAL BOs are assigned a
VM on creation and can never be bound to any other VM.
2. Any attempt to export an INTERNAL BO via prime or a similar
mechanism will fail with-EINVAL(I think?).Quoted 9 lines
Inside the kernel driver, all the internal BOs on a VM (or DRM file in the case of nouveau/AMD since they don't have VM objects) share a single
dma_resvwhich allows you to avoid having to walk lists of BOs and take locks on every exec. Instead, you can just look at the fences on thedma_resvfor the VM. There's still a BO list associated with the VM for external BOs but, in most Vulkan applications, there are less than a half dozen external BOs total. Meanwhile, the hundreds or thousands of BOs used entirely internally to the application basically count as one BO when it comes to locking overhead.I am aware of that and I have some WIP patches [1] to generalize a common dma-resv within the GPUVA manager which basically represents a GPU-VM. It also keeps track of external GEM objects and evicted objects, such that on EXEC we only need to validate objects needing validation, rather than all of them. Hence, it should be faster than with Daves patch having a common dma-resv only.
In [1] I also picked up Daves code to allow for noop jobs to be submitted as well as the
NOUVEAU_GEM_DOMAIN_NO_SHAREflag.This seems to work fine with yours and Daves latest mesa work (
670c301a9845a3fc795fd48a1e6714e75b388245).Your crucible bench.submit-latency test goes down to 51us on my machine with those patches.
I am unsure though, if we should aim for a common solution within the GPUVA manager directly or if we should do it driver specific in a first shot. I discussed this patch with Matt and I know that XE looks for having a generalized solution as well. However, it surely needs some more care and polish and feedback from other drivers perspective.
[1]
https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next-vm-resv
Quoted 6 lines
I'm not actually trying to dictate kernel driver design here. If one
dma_resvdoesn't work, fine. I don't care as long as EXEC scales. However, given that the solution in all the other drivers involves a BO create flag nouveau doesn't have, we need to either add that or prove that we can get EXEC to scale without it.From #nouveau:
<gfxstrand> CTSing now
<gfxstrand> It looks like it's still going to take 1.5 hours.I may have an idea what could be the issue, let me explain.
Currently, there is a single
drm_gpu_schedulerhaving adrm_sched_entityper client (forVM_BINDjobs) and adrm_sched_entityper channel (for EXEC jobs).For
VM_BINDjobs the corresponding PT[E]s are allocated before the job is pushed to the correspondingdrm_sched_entity. The PT[E]s are freed by the schedulersfree()callback pushing work to a single threaded workqueue doing the actual free. (We can't do it in thefree()callback directly, since to free PT[E]s we need to hold a mutex we also need to hold while allocating them.)Because of how the page table handling in Nouveau is implemented currently there are some ordering restrictions when it comes to allocating and freeing PT[E]s. For instance, we can't allocate PT[E]s for sparse regions before the PT[E]s of previously removed memory backed mappings *within the same address range* aren't freed. The same applies vice versa and for sparse mapping replacing sparse mapping. For memory backed mappings (also for those within sparse regions) we do *not* have such ordering requirements.
So, let's assume userspace removes a sparse region A[0x0,
0x8000000] and asks for a couple of new memory backed mappings within or crossing this range; the kernel needs to wait for A not only to be unmapped, but also the backing PT[E]s to be freed before it can even allocate the PT[E]s for the new memory backed mappings.Now, let's have a look what the gpu schedulers main loop does. Before picking the next entity to schedule a job for, it tries to fetch the first job from the
pending_listand checks whether its dma-fence is signaled already and whether the job can be cleaned up. Subsequent jobs on thepending_listare not taken into consideration. Hence, it might well be that the first job on thepending_listisn't signaled yet, but subsequent jobs are and hence *could* be cleaned up.Normally, this shouldn't be a problem, since we wouldn't really care *when* resources are cleaned up as long as they are eventually. However, with the ordering restrictions the page table handling gives us we might actually care about the "when".
For instance, it could happen that the first job on the pending list is a rather long running EXEC job (1) scheduled from client A on some channel. The next job on the pending list could be a
VM_BINDjob (2) from client B removing a sparse region, which is finished already but is blocked to be cleaned up until the EXEC job (1) from client A is finished and cleaned up. Now, a subsequentVM_BINDjob (3) from client B creating a new memory backed mapping in the same address range as the sparse region removed by job (2) would need to wait for (2) to be cleaned up. Ultimately, we can expect client B to submit an EXEC job that needs to wait for the corresponding mappings to be created, namely theVM_BINDjob (3).Clearly in order to address this we need to rework the page table handling in Nouveau to get rid of those ordering restrictions.
Temporarily, we could also try to run a secondary
drm_gpu_schedulerinstance, one forVM_BINDsand one for EXECs maybe...
Quoted 7 lines
However, I would not expect this to be an issue in real applications, especially if mesa takes a little care not to re-use certain address space areas right away to avoid running into such wait conditions.
For parallel VK CTS runs I could imagine that we run into such cases from time to time though.
Thanks for the detailed write-up! That would definitely explain it. If I remember, I'll try to do a single-threaded run or two. If your theory is correct, there should be no real perf difference when running single-threaded. Those runs will take a long time, though, so I'll have to run them over night. I'll let you know in a few days once I have the results.
If this theory holds, then I'm not concerned about the performance of the API itself. It would still be good to see if we can find a way to reduce the cross-process drag in the implementation but that's a perf optimization we can do later.
Does it actually matter? Yes, it kinda does. No, it probably doesn't matter for games because you're typically only running one game at a time. From a development PoV, however, if it makes CI take longer then that slows down development and that's not good for the users, either.
~Faith
Quoted 7 lines
- Danilo
Quoted 3 lines
~Faith
Re: [PATCH drm-misc-next v8 11/12] drm/nouveau: implement new VM_BIND uAPI
Quoted 241 lines · Faith Ekstrand
On Mon, Jul 24, 2023 at 9:04 PM Danilo Krummrich <[email protected] <mailto:[email protected]>> wrote:
On 7/22/23 17:12, Faith Ekstrand wrote:
> On Wed, Jul 19, 2023 at 7:15 PM Danilo Krummrich <[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>
> This commit provides the implementation for the new uapi
motivated
> by the
> Vulkan API. It allows user mode drivers (UMDs) to:
>
> 1) Initialize a GPU virtual address (VA) space via the new > DRM_IOCTL_NOUVEAU_VM_INIT ioctl for UMDs to specify the
portion
> of VA
> space managed by the kernel and userspace, respectively. >
> 2) Allocate and free a VA space region as well as bind and
unbind memory
> to the GPUs VA space via the new
DRM_IOCTL_NOUVEAU_VM_BINDioctl.
> UMDs can request the named operations to be processed either > synchronously or asynchronously. It supports DRM syncobjs > (incl. timelines) as synchronization mechanism. The
management
> of the
> GPU VA mappings is implemented with the DRM GPU VA manager. >
> 3) Execute push buffers with the newDRM_IOCTL_NOUVEAU_EXEC
ioctl. The
> execution happens asynchronously. It supports DRM syncobj
(incl.
> timelines) as synchronization mechanism. DRM GEM object
locking is
> handled withdrm_exec.
>
> Both,DRM_IOCTL_NOUVEAU_VM_BINDandDRM_IOCTL_NOUVEAU_EXEC,
use the DRM
> GPU scheduler for the asynchronous paths.
>
>
> IDK where the best place to talk about this is but this seems as
good as
> any.
>
> I've been looking into why the Vulkan CTS runs about 2x slower
for me on
> the new UAPI and I created a little benchmark to facilitate testing:
>
> https://gitlab.freedesktop.org/mesa/crucible/-/merge_requests/141
<https://gitlab.freedesktop.org/mesa/crucible/-/merge_requests/141>
>
<https://gitlab.freedesktop.org/mesa/crucible/-/merge_requests/141
<https://gitlab.freedesktop.org/mesa/crucible/-/merge_requests/141>>
>
> The test, roughly, does the following:
> 1. Allocates and binds 1000 BOs
> 2. Constructs a pushbuf that executes a no-op compute shader. > 3. Does a single EXEC/wait combo to warm up the kernel > 4. Loops 10,000 times, doingSYNCOBJ_RESET(fast), EXEC, and then >SYNCOBJ_WAITand times the loop
>
> Of course, there's a bit of userspace driver overhead but that's > negledgable.
>
> If you drop the top patch which allocates 1k buffers, the submit
time on
> the old uAPI is 54 us/exec vs. 66 us/exec on the new UAPI. This
includes
> the time to do aSYNCOBJ_RESET(fast), EXEC, and SYNCOBJ_WAIT.The
Intel
> driver, by comparison, is 33us/exec so it's not syncobj overhead.
This
> is a bit concerning (you'd think the new thing would be faster)
but what
> really has me concerned is the 1k buffer case.
>
> If you include the top patch in the crucible MR, it allocates
1000 BOs
> andVM_BINDsthem. All the binding is done before the warmup EXEC. > Suddenly, the submit time jumps to 257 us/exec with the new UAPI.
The
> old UAPI is much worse (1134 us/exec) but that's not the point. Once > we've done the first EXEC and created our VM bindings, the cost
per EXEC
> shouldn't change at all based on the number of BOs bound. Part
of the
> point ofVM_BINDis to get all that binding logic and BO walking
off the
> EXEC path.
>
> Normally, I wouldn't be too worried about a little performance
problem
> like this. This is the first implementation and we can improve it
later.
> I get that. However, I suspect the solution to this problem involves > more UAPI and I want to make sure we have it all before we call
this all
> done and dusted and land it.
>
> The way AMD solves this problem as well as the new Xe driver for
Intel
> is to have a concept of internal vs. external BOs. Basically,
there's an
> INTERNAL bit specified somewhere in BO creation that has a few
userspace
> implications:
> 1. In the Xe world where VMs are objects, INTERNAL BOs are
assigned a
> VM on creation and can never be bound to any other VM.
> 2. Any attempt to export an INTERNAL BO via prime or a similar > mechanism will fail with-EINVAL(I think?).
>
> Inside the kernel driver, all the internal BOs on a VM (or DRM
file in
> the case of nouveau/AMD since they don't have VM objects) share a
single
>dma_resvwhich allows you to avoid having to walk lists of BOs
and take
> locks on every exec. Instead, you can just look at the fences on the >dma_resvfor the VM. There's still a BO list associated with the
VM for
> external BOs but, in most Vulkan applications, there are less than a > half dozen external BOs total. Meanwhile, the hundreds or
thousands of
> BOs used entirely internally to the application basically count
as one
> BO when it comes to locking overhead.I am aware of that and I have some WIP patches [1] to generalize a common dma-resv within the GPUVA manager which basically represents a GPU-VM. It also keeps track of external GEM objects and evicted objects,
such that on EXEC we only need to validate objects needing validation, rather than all of them. Hence, it should be faster than with Daves patch having a common dma-resv only.In [1] I also picked up Daves code to allow for noop jobs to be submitted as well as the
NOUVEAU_GEM_DOMAIN_NO_SHAREflag.This seems to work fine with yours and Daves latest mesa work (
670c301a9845a3fc795fd48a1e6714e75b388245).Your crucible bench.submit-latency test goes down to 51us on my machine with those patches.
I am unsure though, if we should aim for a common solution within the GPUVA manager directly or if we should do it driver specific in a first shot. I discussed this patch with Matt and I know that XE looks for having a generalized solution as well. However, it surely needs some more care and polish and feedback from other drivers perspective.
[1]
https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next-vm-resv <https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next-vm-resv>>
> I'm not actually trying to dictate kernel driver design here. If one >dma_resvdoesn't work, fine. I don't care as long as EXEC scales. > However, given that the solution in all the other drivers
involves a BO
> create flag nouveau doesn't have, we need to either add that or
prove
> that we can get EXEC to scale without it.From #nouveau:
<gfxstrand> CTSing now
<gfxstrand> It looks like it's still going to take 1.5 hours.I may have an idea what could be the issue, let me explain.
Currently, there is a single
drm_gpu_schedulerhaving adrm_sched_entity
per client (forVM_BINDjobs) and adrm_sched_entityper channel (for EXEC jobs).For
VM_BINDjobs the corresponding PT[E]s are allocated before the job is pushed to the correspondingdrm_sched_entity. The PT[E]s are freed by
the schedulersfree()callback pushing work to a single threaded workqueue doing the actual free. (We can't do it in thefree()callback directly, since to free PT[E]s we need to hold a mutex we also need to hold while allocating them.)Because of how the page table handling in Nouveau is implemented currently there are some ordering restrictions when it comes to allocating and freeing PT[E]s. For instance, we can't allocate PT[E]s for sparse regions before the PT[E]s of previously removed memory backed
mappings *within the same address range* aren't freed. The same applies vice versa and for sparse mapping replacing sparse mapping. For memory backed mappings (also for those within sparse regions) we do *not* have such ordering requirements.So, let's assume userspace removes a sparse region A[0x0,
0x8000000] and
asks for a couple of new memory backed mappings within or crossing this range; the kernel needs to wait for A not only to be unmapped, but also the backing PT[E]s to be freed before it can even allocate the PT[E]s for the new memory backed mappings.Now, let's have a look what the gpu schedulers main loop does. Before picking the next entity to schedule a job for, it tries to fetch the first job from the
pending_listand checks whether its dma-fence is signaled already and whether the job can be cleaned up. Subsequent jobs on thepending_listare not taken into consideration. Hence, it might well be that the first job on thepending_listisn't signaled yet, but subsequent jobs are and hence *could* be cleaned up.Normally, this shouldn't be a problem, since we wouldn't really care *when* resources are cleaned up as long as they are eventually. However,
with the ordering restrictions the page table handling gives us we might
actually care about the "when".For instance, it could happen that the first job on the pending list is a rather long running EXEC job (1) scheduled from client A on some channel. The next job on the pending list could be a
VM_BINDjob (2) from client B removing a sparse region, which is finished already but is
blocked to be cleaned up until the EXEC job (1) from client A is finished and cleaned up. Now, a subsequentVM_BINDjob (3) from client B
creating a new memory backed mapping in the same address range as the sparse region removed by job (2) would need to wait for (2) to be cleaned up. Ultimately, we can expect client B to submit an EXEC job that needs to wait for the corresponding mappings to be created, namely theVM_BINDjob (3).Clearly in order to address this we need to rework the page table handling in Nouveau to get rid of those ordering restrictions.
Temporarily, we could also try to run a secondary
drm_gpu_schedulerinstance, one forVM_BINDsand one for EXECs maybe...However, I would not expect this to be an issue in real applications, especially if mesa takes a little care not to re-use certain address space areas right away to avoid running into such wait conditions.
Quoted 5 lines
For parallel VK CTS runs I could imagine that we run into such cases from time to time though.
Thanks for the detailed write-up! That would definitely explain it. If I remember, I'll try to do a single-threaded run or two. If your theory is correct, there should be no real perf difference when running single-threaded. Those runs will take a long time, though, so I'll have to run them over night. I'll let you know in a few days once I have the results.
I can also push a separate branch where I just print out a warning whenever we run into such a condition including the time we were waiting for things to complete. I can probably push something later today.
Quoted 2 lines
If this theory holds, then I'm not concerned about the performance of the API itself. It would still be good to see if we can find a way to reduce the cross-process drag in the implementation but that's a perf optimization we can do later.
From the kernel side I think the only thing we could really do is to temporarily run a secondary drm_gpu_scheduler instance, one for VM_BINDs and one for EXECs until we got the new page table handling in place.
However, the UMD could avoid such conditions more effectively, since it controls the address space. Namely, avoid re-using the same region of the address space right away in certain cases. For instance, instead of replacing a sparse region A[0x0, 0x4000000] with a larger sparse region B[0x0, 0x8000000], replace it with B'[0x4000000, 0xC000000] if possible.
However, just mentioning this for completeness. The UMD surely shouldn't probably even temporarily work around such a kernel limitation.
Anyway, before doing any of those, let's see if the theory holds and we're actually running into such cases.
Quoted 2 lines
Does it actually matter? Yes, it kinda does. No, it probably doesn't matter for games because you're typically only running one game at a time. From a development PoV, however, if it makes CI take longer then that slows down development and that's not good for the users, either.
Fully agree.
- Danilo
Quoted 9 lines
~Faith
- Danilo > > ~Faith >
Re: [PATCH drm-misc-next v8 11/12] drm/nouveau: implement new VM_BIND uAPI
Quoted 4 lines · Danilo Krummrich
Quoted 1 line · Faith Ekstrand
Thanks for the detailed write-up! That would definitely explain it. If I remember, I'll try to do a single-threaded run or two. If your theory is correct, there should be no real perf difference when running single-threaded. Those runs will take a long time, though, so I'll have to run them over night. I'll let you know in a few days once I have the results.
I can also push a separate branch where I just print out a warning whenever we run into such a condition including the time we were waiting for things to complete. I can probably push something later today.
https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next-track-stalls
It prints out the duration of every wait as well as the total wait time since boot.
- Danilo
Quoted 29 lines
Quoted 3 lines
If this theory holds, then I'm not concerned about the performance of the API itself. It would still be good to see if we can find a way to reduce the cross-process drag in the implementation but that's a perf optimization we can do later.
From the kernel side I think the only thing we could really do is to
temporarily run a secondarydrm_gpu_schedulerinstance, one forVM_BINDsand one for EXECs until we got the new page table handling in place.However, the UMD could avoid such conditions more effectively, since it controls the address space. Namely, avoid re-using the same region of the address space right away in certain cases. For instance, instead of replacing a sparse region A[0x0,
0x4000000] with a larger sparse region B[0x0,0x8000000], replace it with B'[0x4000000,0xC000000] if possible.However, just mentioning this for completeness. The UMD surely shouldn't probably even temporarily work around such a kernel limitation.
Anyway, before doing any of those, let's see if the theory holds and we're actually running into such cases.
Quoted 2 lines
Does it actually matter? Yes, it kinda does. No, it probably doesn't matter for games because you're typically only running one game at a time. From a development PoV, however, if it makes CI take longer then that slows down development and that's not good for the users, either.
Fully agree.
- Danilo
Quoted 9 lines
~Faith
- Danilo
>
> ~Faith
>
Re: [PATCH drm-misc-next v8 11/12] drm/nouveau: implement new VM_BIND uAPI
Quoted 14 lines · Danilo Krummrich
Quoted 11 lines · Danilo Krummrich
Quoted 6 lines · Faith Ekstrand
Thanks for the detailed write-up! That would definitely explain it. If I remember, I'll try to do a single-threaded run or two. If your theory is correct, there should be no real perf difference when running single-threaded. Those runs will take a long time, though, so I'll have to run them over night. I'll let you know in a few days once I have the results.
I can also push a separate branch where I just print out a warning whenever we run into such a condition including the time we were waiting for things to complete. I can probably push something later today.
Quoted 36 lines
https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next-track-stalls
It prints out the duration of every wait as well as the total wait time since boot.
- Danilo
Quoted 28 lines
Quoted 6 lines
If this theory holds, then I'm not concerned about the performance of the API itself. It would still be good to see if we can find a way to reduce the cross-process drag in the implementation but that's a perf optimization we can do later.
From the kernel side I think the only thing we could really do is to
temporarily run a secondarydrm_gpu_schedulerinstance, one forVM_BINDsand one for EXECs until we got the new page table handling in place.However, the UMD could avoid such conditions more effectively, since it controls the address space. Namely, avoid re-using the same region of the address space right away in certain cases. For instance, instead of replacing a sparse region A[0x0,
0x4000000] with a larger sparse region B[0x0,0x8000000], replace it with B'[0x4000000,0xC000000] if possible.However, just mentioning this for completeness. The UMD surely shouldn't probably even temporarily work around such a kernel limitation.
Anyway, before doing any of those, let's see if the theory holds and we're actually running into such cases.
Quoted 5 lines
Does it actually matter? Yes, it kinda does. No, it probably doesn't matter for games because you're typically only running one game at a time. From a development PoV, however, if it makes CI take longer then that slows down development and that's not good for the users, either.
I've run a bunch of tests over the weekend and It's starting to look like the added CTS time might be from the newly enabled synchronization tests themselves. We enabled timeline semaphores as well as semaphore and fence sharing along with the new uAPI and I did not expect those to be nearly that heavy-weight so I didn't think of that as a likely factor. I'm doing a couple more runs to confirm but what I'm seeing right now seems to indicate that the new API with the old feature set has about the same run time now that the submit overhead issue has been fixed.
I think this means that we can proceed under the assumption that there are no more serious perf issues with the new API. I'm happy to RB/ACK the next version of the API and implementation patches, as long as it has the new NO_SHARE BO create flag and properly rejects exports of NO_SHARE BOs, even if not all of the dma_resv plumbing is fully baked.
~Faith
Re: [PATCH drm-misc-next v8 11/12] drm/nouveau: implement new VM_BIND uAPI
Quoted 57 lines · Faith Ekstrand
Quoted 34 lines · Danilo Krummrich
Quoted 10 lines · Danilo Krummrich
Quoted 6 lines · Faith Ekstrand
Thanks for the detailed write-up! That would definitely explain it. If I remember, I'll try to do a single-threaded run or two. If your theory is correct, there should be no real perf difference when running single-threaded. Those runs will take a long time, though, so I'll have to run them over night. I'll let you know in a few days once I have the results.
I can also push a separate branch where I just print out a warning whenever we run into such a condition including the time we were
waiting
Quoted 1 line
for things to complete. I can probably push something later today.
https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next-track-stalls
It prints out the duration of every wait as well as the total wait time since boot.
- Danilo
Quoted 9 lines
Quoted 6 lines
If this theory holds, then I'm not concerned about the performance of the API itself. It would still be good to see if we can find a way to reduce the cross-process drag in the implementation but that's a perf optimization we can do later.
From the kernel side I think the only thing we could really do is to
temporarily run a secondarydrm_gpu_schedulerinstance, one for
VM_BINDsQuoted 2 lines
and one for EXECs until we got the new page table handling in place.
Quoted 19 lines
Quoted 7 lines
However, the UMD could avoid such conditions more effectively, since it controls the address space. Namely, avoid re-using the same region of the address space right away in certain cases. For instance, instead of replacing a sparse region A[0x0,
0x4000000] with a larger sparse region B[0x0,0x8000000], replace it with B'[0x4000000,0xC000000] if possible.However, just mentioning this for completeness. The UMD surely
shouldn't
Quoted 10 lines
probably even temporarily work around such a kernel limitation.
Anyway, before doing any of those, let's see if the theory holds and we're actually running into such cases.
Quoted 5 lines
Does it actually matter? Yes, it kinda does. No, it probably doesn't matter for games because you're typically only running one game at a time. From a development PoV, however, if it makes CI take longer then that slows down development and that's not good for the users, either.
Quoted 9 lines
I've run a bunch of tests over the weekend and It's starting to look like the added CTS time might be from the newly enabled synchronization tests themselves. We enabled timeline semaphores as well as semaphore and fence sharing along with the new uAPI and I did not expect those to be nearly that heavy-weight so I didn't think of that as a likely factor. I'm doing a couple more runs to confirm but what I'm seeing right now seems to indicate that the new API with the old feature set has about the same run time now that the submit overhead issue has been fixed.
My last two runs have come back and confirmed this theory. With the BO fixes, all remaining slow-downs are coming from newly added tests which turn out to be very slow tests.
~Faith
Quoted 8 lines
I think this means that we can proceed under the assumption that there are no more serious perf issues with the new API. I'm happy to RB/ACK the next version of the API and implementation patches, as long as it has the new
NO_SHAREBO create flag and properly rejects exports ofNO_SHAREBOs, even if not all of thedma_resvplumbing is fully baked.~Faith
PATCH v8 12/12 drm/nouveau: debugfs: implement DRM GPU VA debugfs
In reply to Danilo KrummrichProvide the driver indirection iterating over all DRM GPU VA spaces to enable the common 'gpuvas' debugfs file for dumping DRM GPU VA spaces.
--- drivers/gpu/drm/nouveau/nouveau_debugfs.c | 39 +++++++++++++++++++++++ 1 file changed, 39 insertions(+)
drivers/gpu/drm/nouveau/nouveau_debugfs.c +39 -0
@@ -203,6 +203,44 @@ nouveau_debugfs_pstate_open(struct inode *inode, struct file *file)return single_open (file, nouveau_debugfs_pstate_get, inode->i_private); } +static void +nouveau_debugfs_gpuva_regions (struct seq_file *m,struct nouveau_uvmm *uvmm) +{ +MA_STATE (mas, &uvmm->region_mt,0 ,0 ); +struct nouveau_uvma_region *reg; + +seq_puts (m," VA regions | start | range | end \n" ); +seq_puts (m,"----------------------------------------------------------------------------\n" ); +mas_for_each (&mas, reg,ULONG_MAX ) +seq_printf (m," | 0x%016llx | 0x%016llx | 0x%016llx\n" , + reg->va.addr, reg->va.range, reg->va.addr + reg->va.range); +} + +static int +nouveau_debugfs_gpuva (struct seq_file *m,void *data) +{ +struct drm_info_node *node = (struct drm_info_node *) m->private; +struct nouveau_drm *drm =nouveau_drm (node->minor->dev); +struct nouveau_cli *cli; + +mutex_lock (&drm->clients_lock); +list_for_each_entry (cli, &drm->clients, head) { +struct nouveau_uvmm *uvmm =nouveau_cli_uvmm (cli); + +if (!uvmm) +continue ; + +nouveau_uvmm_lock (uvmm); +drm_debugfs_gpuva_info (m, &uvmm->umgr); +seq_puts (m,"\n" ); +nouveau_debugfs_gpuva_regions (m, uvmm); +nouveau_uvmm_unlock (uvmm); + } +mutex_unlock (&drm->clients_lock); + +return 0 ; +} +static const struct file_operations nouveau_pstate_fops = { .owner =THIS_MODULE , .open = nouveau_debugfs_pstate_open,
@@ -214,6 +252,7 @@ static const struct file_operations nouveau_pstate_fops = {static struct drm_info_list nouveau_debugfs_list[] = { {"vbios.rom" , nouveau_debugfs_vbios_image,0 ,NULL }, {"strap_peek" , nouveau_debugfs_strap_peek,0 ,NULL }, +DRM_DEBUGFS_GPUVA_INFO (nouveau_debugfs_gpuva,NULL ), };#define NOUVEAU_DEBUGFS_ENTRIES ARRAY_SIZE(nouveau_debugfs_list)
Signature · 1 line
2.41.0