drm: Add a driver for CSF-based Mali GPUs
Subsystems (6)
Boris Brezillon
maintainerv3 cover drm: Add a driver for CSF-based Mali GPUs
Hello,
This is the 3rd version of the kernel driver for Mali CSF-based GPUs.
With all the DRM dependencies being merged (drm-sched single entity and drm_gpuvm), I thought now was a good time to post a new version. Note that the iommu series we depend on [1] has been merged recently. The only remaining dependency that hasn't been merged yet is this rather trival drm_gpuvm [2] patch.
As for v2, I pushed a branch based on drm-misc-next and containing all the dependencies that are not yet available in drm-misc-next here[3], and another [4] containing extra patches to have things working on rk3588. The CSF firmware binary can be found here[5], and should be placed under /lib/firmware/arm/mali/arch10.8/mali_csffw.bin.
The mesa MR adding v10 support on top of panthor is available here [6].
Regarding the GPL2+MIT relicensing, Liviu did an audit and found two more people that I didn't spot initially: Clément Péron for the devfreq code, and Alexey Sheplyakov for some bits in panthor_gpu.c. Both are Cc-ed on the relevant patches. The rest of the code is either new, or covered by the Linaro, Arm and Collabora acks.
And here is a non-exhaustive changelog, check each commit for a detailed changelog.
v3;
- Quite a few changes at the MMU/sched level to make the fix some race conditions and deadlocks
- Addition of the a sync-only VM_BIND operation (to support vkQueueSparseBind with zero commands).
- Addition of a VM_GET_STATE ioctl
- Various cosmetic changes (see the commit changelogs for more details)
- Various fixes (see the commit changelogs for more details)
v2:
- Rename the driver (pancsf -> panthor)
- Split the commit adding the driver to ease review
- Use drm_sched for dependency tracking/job submission
- Add a VM_BIND ioctl
- Add the concept of exclusive VM for BOs that are only ever mapped to a single VM
- Document the code and uAPI
- Add a DT binding doc
Regards,
Boris
[1]https://lore.kernel.org/linux-arm-kernel/[email protected]/T/ [2]https://patchwork.freedesktop.org/patch/570380/ [3]https://gitlab.freedesktop.org/panfrost/linux/-/tree/panthor-v3 [4]https://gitlab.freedesktop.org/panfrost/linux/-/tree/panthor-v3+rk3588 [5]https://gitlab.com/firefly-linux/external/libmali/-/raw/firefly/firmware/g610/mali_csffw.bin [6]https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26358
Boris Brezillon (13):
drm/panthor: Add uAPI
drm/panthor: Add GPU register definitions
drm/panthor: Add the device logical block
drm/panthor: Add the GPU logical block
drm/panthor: Add GEM logical block
drm/panthor: Add the devfreq logical block
drm/panthor: Add the MMU/VM logical block
drm/panthor: Add the FW logical block
drm/panthor: Add the heap logical block
drm/panthor: Add the scheduler logical block
drm/panthor: Add the driver frontend block
drm/panthor: Allow driver compilation
drm/panthor: Add an entry to MAINTAINERS
Liviu Dudau (1):
dt-bindings: gpu: mali-valhall-csf: Add support for Arm Mali CSF GPUs
.../bindings/gpu/arm,mali-valhall-csf.yaml | 147 + Documentation/gpu/driver-uapi.rst | 5 + MAINTAINERS | 11 + drivers/gpu/drm/Kconfig | 2 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/panthor/Kconfig | 23 + drivers/gpu/drm/panthor/Makefile | 15 + drivers/gpu/drm/panthor/panthor_devfreq.c | 283 ++ drivers/gpu/drm/panthor/panthor_devfreq.h | 25 + drivers/gpu/drm/panthor/panthor_device.c | 497 +++ drivers/gpu/drm/panthor/panthor_device.h | 381 ++ drivers/gpu/drm/panthor/panthor_drv.c | 1454 +++++++ drivers/gpu/drm/panthor/panthor_fw.c | 1332 +++++++ drivers/gpu/drm/panthor/panthor_fw.h | 504 +++ drivers/gpu/drm/panthor/panthor_gem.c | 227 ++ drivers/gpu/drm/panthor/panthor_gem.h | 144 + drivers/gpu/drm/panthor/panthor_gpu.c | 481 +++ drivers/gpu/drm/panthor/panthor_gpu.h | 52 + drivers/gpu/drm/panthor/panthor_heap.c | 517 +++ drivers/gpu/drm/panthor/panthor_heap.h | 36 + drivers/gpu/drm/panthor/panthor_mmu.c | 2653 +++++++++++++ drivers/gpu/drm/panthor/panthor_mmu.h | 101 + drivers/gpu/drm/panthor/panthor_regs.h | 237 ++ drivers/gpu/drm/panthor/panthor_sched.c | 3410 +++++++++++++++++ drivers/gpu/drm/panthor/panthor_sched.h | 48 + include/uapi/drm/panthor_drm.h | 892 +++++ 26 files changed, 13478 insertions(+) create mode 100644 Documentation/devicetree/bindings/gpu/arm,mali-valhall-csf.yaml create mode 100644 drivers/gpu/drm/panthor/Kconfig create mode 100644 drivers/gpu/drm/panthor/Makefile create mode 100644 drivers/gpu/drm/panthor/panthor_devfreq.c create mode 100644 drivers/gpu/drm/panthor/panthor_devfreq.h create mode 100644 drivers/gpu/drm/panthor/panthor_device.c create mode 100644 drivers/gpu/drm/panthor/panthor_device.h create mode 100644 drivers/gpu/drm/panthor/panthor_drv.c create mode 100644 drivers/gpu/drm/panthor/panthor_fw.c create mode 100644 drivers/gpu/drm/panthor/panthor_fw.h create mode 100644 drivers/gpu/drm/panthor/panthor_gem.c create mode 100644 drivers/gpu/drm/panthor/panthor_gem.h create mode 100644 drivers/gpu/drm/panthor/panthor_gpu.c create mode 100644 drivers/gpu/drm/panthor/panthor_gpu.h create mode 100644 drivers/gpu/drm/panthor/panthor_heap.c create mode 100644 drivers/gpu/drm/panthor/panthor_heap.h create mode 100644 drivers/gpu/drm/panthor/panthor_mmu.c create mode 100644 drivers/gpu/drm/panthor/panthor_mmu.h create mode 100644 drivers/gpu/drm/panthor/panthor_regs.h create mode 100644 drivers/gpu/drm/panthor/panthor_sched.c create mode 100644 drivers/gpu/drm/panthor/panthor_sched.h create mode 100644 include/uapi/drm/panthor_drm.h
Signature · 1 line
2.43.0
Boris Brezillon
maintainerPATCH v3 1/14 drm/panthor: Add uAPI
Panthor follows the lead of other recently submitted drivers with ioctls allowing us to support modern Vulkan features, like sparse memory binding:
- Pretty standard GEM management ioctls (BO_CREATE and BO_MMAP_OFFSET), with the 'exclusive-VM' bit to speed-up BO reservation on job submission
- VM management ioctls (VM_CREATE, VM_DESTROY and VM_BIND). The VM_BIND ioctl is loosely based on the Xe model, and can handle both asynchronous and synchronous requests
- GPU execution context creation/destruction, tiler heap context creation and job submission. Those ioctls reflect how the hardware/scheduler works and are thus driver specific.
We also have a way to expose IO regions, such that the usermode driver can directly access specific/well-isolate registers, like the LATEST_FLUSH register used to implement cache-flush reduction.
This uAPI intentionally keeps usermode queues out of the scope, which explains why doorbell registers and command stream ring-buffers are not directly exposed to userspace.
v3:
- Add the concept of sync-only VM operation
- Fix support for 32-bit userspace
- Rework drm_panthor_vm_create to pass the user VA size instead of the kernel VA size (suggested by Robin Murphy)
- Typo fixes
- Explicitly cast enums with top bit set to avoid compiler warnings in -pedantic mode.
- Drop property core_group_count as it can be easily calculated by the number of bits set in l2_present.
--- Documentation/gpu/driver-uapi.rst | 5 + include/uapi/drm/panthor_drm.h | 892 ++++++++++++++++++++++++++++++ 2 files changed, 897 insertions(+) create mode 100644 include/uapi/drm/panthor_drm.h
Documentation/gpu/driver-uapi.rst +5 -0
@@ -17,3 +17,8 @@ VM_BIND / EXEC uAPI :doc: Overview .. kernel-doc:: include/uapi/drm/nouveau_drm.h + +drm/panthor uAPI +================ + +.. kernel-doc:: include/uapi/drm/panthor_drm.h
include/uapi/drm/panthor_drm.h new file +892 -0
892 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v3 01/14] drm/panthor: Add uAPI
Quoted 32 lines · Boris Brezillon
Panthor follows the lead of other recently submitted drivers with ioctls allowing us to support modern Vulkan features, like sparse memory binding:
- Pretty standard GEM management ioctls (
BO_CREATEandBO_MMAP_OFFSET), with the 'exclusive-VM' bit to speed-up BO reservation on job submission
- VM management ioctls (VM_CREATE,VM_DESTROYandVM_BIND). TheVM_BINDioctl is loosely based on the Xe model, and can handle both asynchronous and synchronous requests
- GPU execution context creation/destruction, tiler heap context creation and job submission. Those ioctls reflect how the hardware/scheduler works and are thus driver specific.We also have a way to expose IO regions, such that the usermode driver can directly access specific/well-isolate registers, like the
LATEST_FLUSHregister used to implement cache-flush reduction.This uAPI intentionally keeps usermode queues out of the scope, which explains why doorbell registers and command stream ring-buffers are not directly exposed to userspace.
v3:
- Add the concept of sync-only VM operation
- Fix support for 32-bit userspace
- Reworkdrm_panthor_vm_createto pass the user VA size instead of the kernel VA size (suggested by Robin Murphy)
- Typo fixes
- Explicitly cast enums with top bit set to avoid compiler warnings in -pedantic mode.
- Drop propertycore_group_countas it can be easily calculated by the number of bits set inl2_present.
Quoted 2 lines
Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>
NIT: I believe the signed-off lines should be like:
So yours at the end not mine (because you're the one posting it) and a "Co-developed-by" to explain why there's a sign-off from me. I'm guessing this is actually my fault since I sent the branch to you with my changes with the sign offs like this.
Otherwise everything looks good:
Thanks,
Steve
Quoted 917 lines
--- Documentation/gpu/driver-uapi.rst | 5 + include/uapi/drm/panthor_drm.h | 892 ++++++++++++++++++++++++++++++ 2 files changed, 897 insertions(+) create mode 100644 include/uapi/drm/panthor_drm.hDocumentation/gpu/driver-uapi.rst +5
-0@@ -17,3 +17,8 @@ VM_BIND / EXEC uAPI :doc: Overview .. kernel-doc:: include/uapi/drm/nouveau_drm.h + +drm/panthor uAPI +================ + +.. kernel-doc:: include/uapi/drm/panthor_drm.hinclude/uapi/drm/panthor_drm.h new file +892
-0@@ -0,0 +1,892 @@ +/* SPDX-License-Identifier: MIT */ +/* Copyright (C) 2023 Collabora ltd. */ +#ifndef _PANTHOR_DRM_H_ +#define _PANTHOR_DRM_H_ + +#include "drm.h" + +#if defined(__cplusplus) +extern "C" { +#endif + +/** +* DOC: Introduction +* +* This documentation describes the Panthor IOCTLs. +* +* Just a few generic rules about the data passed to the Panthor IOCTLs: +* +* - Structures must be aligned on 64-bit/8-byte. If the object is not +* naturally aligned, a padding field must be added. +* - Fields must be explicitly aligned to their natural type alignment with +* pad[0..N] fields. +* - All padding fields will be checked by the driver to make sure they are +* zeroed. +* - Flags can be added, but not removed/replaced. +* - New fields can be added to the main structures (the structures +* directly passed to the ioctl). Those fields can be added at the end of +* the structure, or replace existing padding fields. Any new field being +* added must preserve the behavior that existed before those fields were +* added when a value of zero is passed. +* - New fields can be added to indirect objects (objects pointed by the +* main structure), iff those objects are passed a size to reflect the +* size known by the userspace driver (see drm_panthor_obj_array::stride +* or drm_panthor_dev_query::size). +* - If the kernel driver is too old to know some fields, those will be +* ignored if zero, and otherwise rejected (and so will be zero on output). +* - If userspace is too old to know some fields, those will be zeroed +* (input) before the structure is parsed by the kernel driver. +* - Each new flag/field addition must come with a driver version update so +* the userspace driver doesn't have to trial and error to know which +* flags are supported. +* - Structures should not contain unions, as this would defeat the +* extensibility of such structures. +* - IOCTLs can't be removed or replaced. New IOCTL IDs should be placed +* at the end of the drm_panthor_ioctl_id enum. +*/ + +/** +* DOC: MMIO regions exposed to userspace. +* +* .. c:macro:: DRM_PANTHOR_USER_MMIO_OFFSET +* +* File offset for all MMIO regions being exposed to userspace. Don't use +* this value directly, use DRM_PANTHOR_USER_<name>_OFFSET values instead. +* pgoffset passed to mmap2() is an unsigned long, which forces us to use a +* different offset on 32-bit and 64-bit systems. +* +* .. c:macro:: DRM_PANTHOR_USER_FLUSH_ID_MMIO_OFFSET +* +* File offset for the LATEST_FLUSH_ID register. The Userspace driver controls +* GPU cache flushing through CS instructions, but the flush reduction +* mechanism requires a flush_id. This flush_id could be queried with an +* ioctl, but Arm provides a well-isolated register page containing only this +* read-only register, so let's expose this page through a static mmap offset +* and allow direct mapping of this MMIO region so we can avoid the +* user <-> kernel round-trip. +*/ +#define DRM_PANTHOR_USER_MMIO_OFFSET_32BIT (1ull << 43) +#define DRM_PANTHOR_USER_MMIO_OFFSET_64BIT (1ull << 56) +#define DRM_PANTHOR_USER_MMIO_OFFSET (sizeof(unsigned long) < 8 ? \ + DRM_PANTHOR_USER_MMIO_OFFSET_32BIT : \ + DRM_PANTHOR_USER_MMIO_OFFSET_64BIT) +#define DRM_PANTHOR_USER_FLUSH_ID_MMIO_OFFSET (DRM_PANTHOR_USER_MMIO_OFFSET | 0) + +/** +* DOC: IOCTL IDs +* +* enum drm_panthor_ioctl_id - IOCTL IDs +* +* Place new ioctls at the end, don't re-order, don't replace or remove entries. +* +* These IDs are not meant to be used directly. Use the DRM_IOCTL_PANTHOR_xxx +* definitions instead. +*/ +enum drm_panthor_ioctl_id { +/** @DRM_PANTHOR_DEV_QUERY: Query device information. */ +DRM_PANTHOR_DEV_QUERY =0 , + +/** @DRM_PANTHOR_VM_CREATE: Create a VM. */ +DRM_PANTHOR_VM_CREATE , + +/** @DRM_PANTHOR_VM_DESTROY: Destroy a VM. */ +DRM_PANTHOR_VM_DESTROY , + +/** @DRM_PANTHOR_VM_BIND: Bind/unbind memory to a VM. */ +DRM_PANTHOR_VM_BIND , + +/** @DRM_PANTHOR_BO_CREATE: Create a buffer object. */ +DRM_PANTHOR_BO_CREATE , + +/** +* @DRM_PANTHOR_BO_MMAP_OFFSET: Get the file offset to pass to +* mmap to map a GEM object. +*/ +DRM_PANTHOR_BO_MMAP_OFFSET , + +/** @DRM_PANTHOR_GROUP_CREATE: Create a scheduling group. */ +DRM_PANTHOR_GROUP_CREATE , + +/** @DRM_PANTHOR_GROUP_DESTROY: Destroy a scheduling group. */ +DRM_PANTHOR_GROUP_DESTROY , + +/** +* @DRM_PANTHOR_GROUP_SUBMIT: Submit jobs to queues belonging +* to a specific scheduling group. +*/ +DRM_PANTHOR_GROUP_SUBMIT , + +/** @DRM_PANTHOR_GROUP_GET_STATE: Get the state of a scheduling group. */ +DRM_PANTHOR_GROUP_GET_STATE , + +/** @DRM_PANTHOR_TILER_HEAP_CREATE: Create a tiler heap. */ +DRM_PANTHOR_TILER_HEAP_CREATE , + +/** @DRM_PANTHOR_TILER_HEAP_DESTROY: Destroy a tiler heap. */ +DRM_PANTHOR_TILER_HEAP_DESTROY , +}; + +/** +* DRM_IOCTL_PANTHOR() - Build a Panthor IOCTL number +* @__access: Access type. Must be R, W or RW. +* @__id: One of the DRM_PANTHOR_xxx id. +* @__type: Suffix of the type being passed to the IOCTL. +* +* Don't use this macro directly, use the DRM_IOCTL_PANTHOR_xxx +* values instead. +* +* Return: An IOCTL number to be passed to ioctl() from userspace. +*/ +#define DRM_IOCTL_PANTHOR (__access, __id, __type) \ + DRM_IO ## __access(DRM_COMMAND_BASE + DRM_PANTHOR_ ## __id, \ + struct drm_panthor_ ## __type) + +#define DRM_IOCTL_PANTHOR_DEV_QUERY \ + DRM_IOCTL_PANTHOR(WR, DEV_QUERY, dev_query) +#define DRM_IOCTL_PANTHOR_VM_CREATE \ + DRM_IOCTL_PANTHOR(WR, VM_CREATE, vm_create) +#define DRM_IOCTL_PANTHOR_VM_DESTROY \ + DRM_IOCTL_PANTHOR(WR, VM_DESTROY, vm_destroy) +#define DRM_IOCTL_PANTHOR_VM_BIND \ + DRM_IOCTL_PANTHOR(WR, VM_BIND, vm_bind) +#define DRM_IOCTL_PANTHOR_BO_CREATE \ + DRM_IOCTL_PANTHOR(WR, BO_CREATE, bo_create) +#define DRM_IOCTL_PANTHOR_BO_MMAP_OFFSET \ + DRM_IOCTL_PANTHOR(WR, BO_MMAP_OFFSET, bo_mmap_offset) +#define DRM_IOCTL_PANTHOR_GROUP_CREATE \ + DRM_IOCTL_PANTHOR(WR, GROUP_CREATE, group_create) +#define DRM_IOCTL_PANTHOR_GROUP_DESTROY \ + DRM_IOCTL_PANTHOR(WR, GROUP_DESTROY, group_destroy) +#define DRM_IOCTL_PANTHOR_GROUP_SUBMIT \ + DRM_IOCTL_PANTHOR(WR, GROUP_SUBMIT, group_submit) +#define DRM_IOCTL_PANTHOR_GROUP_GET_STATE \ + DRM_IOCTL_PANTHOR(WR, GROUP_GET_STATE, group_get_state) +#define DRM_IOCTL_PANTHOR_TILER_HEAP_CREATE \ + DRM_IOCTL_PANTHOR(WR, TILER_HEAP_CREATE, tiler_heap_create) +#define DRM_IOCTL_PANTHOR_TILER_HEAP_DESTROY \ + DRM_IOCTL_PANTHOR(WR, TILER_HEAP_DESTROY, tiler_heap_destroy) + +/** +* DOC: IOCTL arguments +*/ + +/** +* struct drm_panthor_obj_array - Object array. +* +* This object is used to pass an array of objects whose size is subject to changes in +* future versions of the driver. In order to support this mutability, we pass a stride +* describing the size of the object as known by userspace. +* +* You shouldn't fill drm_panthor_obj_array fields directly. You should instead use +* the DRM_PANTHOR_OBJ_ARRAY() macro that takes care of initializing the stride to +* the object size. +*/ +struct drm_panthor_obj_array { +/** @stride: Stride of object struct. Used for versioning. */ +__u32 stride; + +/** @count: Number of objects in the array. */ +__u32 count; + +/** @array: User pointer to an array of objects. */ +__u64 array; +}; + +/** +* DRM_PANTHOR_OBJ_ARRAY() - Initialize a drm_panthor_obj_array field. +* @cnt: Number of elements in the array. +* @ptr: Pointer to the array to pass to the kernel. +* +* Macro initializing a drm_panthor_obj_array based on the object size as known +* by userspace. +*/ +#define DRM_PANTHOR_OBJ_ARRAY (cnt, ptr) \ + { .stride = sizeof((ptr)[0]), .count = (cnt), .array = (__u64)(uintptr_t)(ptr) } + +/** +* enum drm_panthor_sync_op_flags - Synchronization operation flags. +*/ +enum drm_panthor_sync_op_flags { +/** @DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_MASK: Synchronization handle type mask. */ +DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_MASK =0xff , + +/** @DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_SYNCOBJ: Synchronization object type. */ +DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_SYNCOBJ =0 , + +/** +* @DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_TIMELINE_SYNCOBJ: Timeline synchronization +* object type. +*/ +DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_TIMELINE_SYNCOBJ =1 , + +/** @DRM_PANTHOR_SYNC_OP_WAIT: Wait operation. */ +DRM_PANTHOR_SYNC_OP_WAIT =0 <<31 , + +/** @DRM_PANTHOR_SYNC_OP_SIGNAL: Signal operation. */ +DRM_PANTHOR_SYNC_OP_SIGNAL = (int )(1u <<31 ), +}; + +/** +* struct drm_panthor_sync_op - Synchronization operation. +*/ +struct drm_panthor_sync_op { +/** @flags: Synchronization operation flags. Combination of DRM_PANTHOR_SYNC_OP values. */ +__u32 flags; + +/** @handle: Sync handle. */ +__u32 handle; + +/** +* @timeline_value: MBZ if +* (flags & DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_MASK) != +* DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_TIMELINE_SYNCOBJ. +*/ +__u64 timeline_value; +}; + +/** +* enum drm_panthor_dev_query_type - Query type +* +* Place new types at the end, don't re-order, don't remove or replace. +*/ +enum drm_panthor_dev_query_type { +/** @DRM_PANTHOR_DEV_QUERY_GPU_INFO: Query GPU information. */ +DRM_PANTHOR_DEV_QUERY_GPU_INFO =0 , + +/** @DRM_PANTHOR_DEV_QUERY_CSIF_INFO: Query command-stream interface information. */ +DRM_PANTHOR_DEV_QUERY_CSIF_INFO , +}; + +/** +* struct drm_panthor_gpu_info - GPU information +* +* Structure grouping all queryable information relating to the GPU. +*/ +struct drm_panthor_gpu_info { +/** @gpu_id : GPU ID. */ +__u32 gpu_id; +#define DRM_PANTHOR_ARCH_MAJOR (x) ((x) >> 28) +#define DRM_PANTHOR_ARCH_MINOR (x) (((x) >> 24) & 0xf) +#define DRM_PANTHOR_ARCH_REV (x) (((x) >> 20) & 0xf) +#define DRM_PANTHOR_PRODUCT_MAJOR (x) (((x) >> 16) & 0xf) +#define DRM_PANTHOR_VERSION_MAJOR (x) (((x) >> 12) & 0xf) +#define DRM_PANTHOR_VERSION_MINOR (x) (((x) >> 4) & 0xff) +#define DRM_PANTHOR_VERSION_STATUS (x) ((x) & 0xf) + +/** @gpu_rev: GPU revision. */ +__u32 gpu_rev; + +/** @csf_id: Command stream frontend ID. */ +__u32 csf_id; +#define DRM_PANTHOR_CSHW_MAJOR (x) (((x) >> 26) & 0x3f) +#define DRM_PANTHOR_CSHW_MINOR (x) (((x) >> 20) & 0x3f) +#define DRM_PANTHOR_CSHW_REV (x) (((x) >> 16) & 0xf) +#define DRM_PANTHOR_MCU_MAJOR (x) (((x) >> 10) & 0x3f) +#define DRM_PANTHOR_MCU_MINOR (x) (((x) >> 4) & 0x3f) +#define DRM_PANTHOR_MCU_REV (x) ((x) & 0xf) + +/** @l2_features: L2-cache features. */ +__u32 l2_features; + +/** @tiler_features: Tiler features. */ +__u32 tiler_features; + +/** @mem_features: Memory features. */ +__u32 mem_features; + +/** @mmu_features: MMU features. */ +__u32 mmu_features; +#define DRM_PANTHOR_MMU_VA_BITS (x) ((x) & 0xff) + +/** @thread_features: Thread features. */ +__u32 thread_features; + +/** @max_threads: Maximum number of threads. */ +__u32 max_threads; + +/** @thread_max_workgroup_size: Maximum workgroup size. */ +__u32 thread_max_workgroup_size; + +/** +* @thread_max_barrier_size: Maximum number of threads that can wait +* simultaneously on a barrier. +*/ +__u32 thread_max_barrier_size; + +/** @coherency_features: Coherency features. */ +__u32 coherency_features; + +/** @texture_features: Texture features. */ +__u32 texture_features[4 ]; + +/** @as_present: Bitmask encoding the number of address-space exposed by the MMU. */ +__u32 as_present; + +/** @shader_present: Bitmask encoding the shader cores exposed by the GPU. */ +__u64 shader_present; + +/** @l2_present: Bitmask encoding the L2 caches exposed by the GPU. */ +__u64 l2_present; + +/** @tiler_present: Bitmask encoding the tiler units exposed by the GPU. */ +__u64 tiler_present; +}; + +/** +* struct drm_panthor_csif_info - Command stream interface information +* +* Structure grouping all queryable information relating to the command stream interface. +*/ +struct drm_panthor_csif_info { +/** @csg_slot_count: Number of command stream group slots exposed by the firmware. */ +__u32 csg_slot_count; + +/** @cs_slot_count: Number of command stream slots per group. */ +__u32 cs_slot_count; + +/** @cs_reg_count: Number of command stream registers. */ +__u32 cs_reg_count; + +/** @scoreboard_slot_count: Number of scoreboard slots. */ +__u32 scoreboard_slot_count; + +/** +* @unpreserved_cs_reg_count: Number of command stream registers reserved by +* the kernel driver to call a userspace command stream. +* +* All registers can be used by a userspace command stream, but the +* [cs_slot_count - unpreserved_cs_reg_count .. cs_slot_count] registers are +* used by the kernel when DRM_PANTHOR_IOCTL_GROUP_SUBMIT is called. +*/ +__u32 unpreserved_cs_reg_count; + +/** +* @pad: Padding field, set to zero. +*/ +__u32 pad; +}; + +/** +* struct drm_panthor_dev_query - Arguments passed to DRM_PANTHOR_IOCTL_DEV_QUERY +*/ +struct drm_panthor_dev_query { +/** @type: the query type (see drm_panthor_dev_query_type). */ +__u32 type; + +/** +* @size: size of the type being queried. +* +* If pointer is NULL, size is updated by the driver to provide the +* output structure size. If pointer is not NULL, the driver will +* only copy min(size, actual_structure_size) bytes to the pointer, +* and update the size accordingly. This allows us to extend query +* types without breaking userspace. +*/ +__u32 size; + +/** +* @pointer: user pointer to a query type struct. +* +* Pointer can be NULL, in which case, nothing is copied, but the +* actual structure size is returned. If not NULL, it must point to +* a location that's large enough to hold size bytes. +*/ +__u64 pointer; +}; + +/** +* struct drm_panthor_vm_create - Arguments passed to DRM_PANTHOR_IOCTL_VM_CREATE +*/ +struct drm_panthor_vm_create { +/** @flags: VM flags, MBZ. */ +__u32 flags; + +/** @id: Returned VM ID. */ +__u32 id; + +/** +* @user_va_range: Size of the VA space reserved for user objects. +* +* The kernel will pick the remaining space to map kernel-only objects to the +* VM (heap chunks, heap context, ring buffers, kernel synchronization objects, +* ...). If the space left for kernel objects is too small, kernel object +* allocation will fail further down the road. One can use +* drm_panthor_gpu_info::mmu_features to extract the total virtual address +* range, and chose a user_va_range that leaves some space to the kernel. +* +* If user_va_range is zero, the kernel will pick a sensible value based on +* TASK_SIZE and the virtual range supported by the GPU MMU (the kernel/user +* split should leave enough VA space for userspace processes to support SVM, +* while still allowing the kernel to map some amount of kernel objects in +* the kernel VA range). The value chosen by the driver will be returned in +* @user_va_range. +* +* User VA space always starts at 0x0, kernel VA space is always placed after +* the user VA range. +*/ +__u64 user_va_range; +}; + +/** +* struct drm_panthor_vm_destroy - Arguments passed to DRM_PANTHOR_IOCTL_VM_DESTROY +*/ +struct drm_panthor_vm_destroy { +/** @id: ID of the VM to destroy. */ +__u32 id; + +/** @pad: MBZ. */ +__u32 pad; +}; + +/** +* enum drm_panthor_vm_bind_op_flags - VM bind operation flags +*/ +enum drm_panthor_vm_bind_op_flags { +/** +* @DRM_PANTHOR_VM_BIND_OP_MAP_READONLY: Map the memory read-only. +* +* Only valid with DRM_PANTHOR_VM_BIND_OP_TYPE_MAP. +*/ +DRM_PANTHOR_VM_BIND_OP_MAP_READONLY =1 <<0 , + +/** +* @DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC: Map the memory not-executable. +* +* Only valid with DRM_PANTHOR_VM_BIND_OP_TYPE_MAP. +*/ +DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC =1 <<1 , + +/** +* @DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED: Map the memory uncached. +* +* Only valid with DRM_PANTHOR_VM_BIND_OP_TYPE_MAP. +*/ +DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED =1 <<2 , + +/** +* @DRM_PANTHOR_VM_BIND_OP_TYPE_MASK: Mask used to determine the type of operation. +*/ +DRM_PANTHOR_VM_BIND_OP_TYPE_MASK = (int )(0xfu <<28 ), + +/** @DRM_PANTHOR_VM_BIND_OP_TYPE_MAP: Map operation. */ +DRM_PANTHOR_VM_BIND_OP_TYPE_MAP =0 <<28 , + +/** @DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP: Unmap operation. */ +DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP =1 <<28 , + +/** +* @DRM_PANTHOR_VM_BIND_OP_TYPE_SYNC_ONLY: No VM operation. +* +* Just serves as a synchronization point on a VM queue. +* +* Only valid if %DRM_PANTHOR_VM_BIND_ASYNC is set in drm_panthor_vm_bind::flags, +* and drm_panthor_vm_bind_op::syncs contains at least one element. +*/ +DRM_PANTHOR_VM_BIND_OP_TYPE_SYNC_ONLY =2 <<28 , +}; + +/** +* struct drm_panthor_vm_bind_op - VM bind operation +*/ +struct drm_panthor_vm_bind_op { +/** @flags: Combination of drm_panthor_vm_bind_op_flags flags. */ +__u32 flags; + +/** +* @bo_handle: Handle of the buffer object to map. +* MBZ for unmap or sync-only operations. +*/ +__u32 bo_handle; + +/** +* @bo_offset: Buffer object offset. +* MBZ for unmap or sync-only operations. +*/ +__u64 bo_offset; + +/** +* @va: Virtual address to map/unmap. +* MBZ for sync-only operations. +*/ +__u64 va; + +/** +* @size: Size to map/unmap. +* MBZ for sync-only operations. +*/ +__u64 size; + +/** +* @syncs: Array of struct drm_panthor_sync_op synchronization +* operations. +* +* This array must be empty if %DRM_PANTHOR_VM_BIND_ASYNC is not set on +* the drm_panthor_vm_bind object containing this VM bind operation. +* +* This array shall not be empty for sync-only operations. +*/ +struct drm_panthor_obj_array syncs; + +}; + +/** +* enum drm_panthor_vm_bind_flags - VM bind flags +*/ +enum drm_panthor_vm_bind_flags { +/** +* @DRM_PANTHOR_VM_BIND_ASYNC: VM bind operations are queued to the VM +* queue instead of being executed synchronously. +*/ +DRM_PANTHOR_VM_BIND_ASYNC =1 <<0 , +}; + +/** +* struct drm_panthor_vm_bind - Arguments passed to DRM_IOCTL_PANTHOR_VM_BIND +*/ +struct drm_panthor_vm_bind { +/** @vm_id: VM targeted by the bind request. */ +__u32 vm_id; + +/** @flags: Combination of drm_panthor_vm_bind_flags flags. */ +__u32 flags; + +/** @ops: Array of struct drm_panthor_vm_bind_op bind operations. */ +struct drm_panthor_obj_array ops; +}; + +/** +* enum drm_panthor_bo_flags - Buffer object flags, passed at creation time. +*/ +enum drm_panthor_bo_flags { +/** @DRM_PANTHOR_BO_NO_MMAP: The buffer object will never be CPU-mapped in userspace. */ +DRM_PANTHOR_BO_NO_MMAP = (1 <<0 ), +}; + +/** +* struct drm_panthor_bo_create - Arguments passed to DRM_IOCTL_PANTHOR_BO_CREATE. +*/ +struct drm_panthor_bo_create { +/** +* @size: Requested size for the object +* +* The (page-aligned) allocated size for the object will be returned. +*/ +__u64 size; + +/** +* @flags: Flags. Must be a combination of drm_panthor_bo_flags flags. +*/ +__u32 flags; + +/** +* @exclusive_vm_id: Exclusive VM this buffer object will be mapped to. +* +* If not zero, the field must refer to a valid VM ID, and implies that: +* - the buffer object will only ever be bound to that VM +* - cannot be exported as a PRIME fd +*/ +__u32 exclusive_vm_id; + +/** +* @handle: Returned handle for the object. +* +* Object handles are nonzero. +*/ +__u32 handle; + +/** @pad: MBZ. */ +__u32 pad; +}; + +/** +* struct drm_panthor_bo_mmap_offset - Arguments passed to DRM_IOCTL_PANTHOR_BO_MMAP_OFFSET. +*/ +struct drm_panthor_bo_mmap_offset { +/** @handle: Handle of the object we want an mmap offset for. */ +__u32 handle; + +/** @pad: MBZ. */ +__u32 pad; + +/** @offset: The fake offset to use for subsequent mmap calls. */ +__u64 offset; +}; + +/** +* struct drm_panthor_queue_create - Queue creation arguments. +*/ +struct drm_panthor_queue_create { +/** +* @priority: Defines the priority of queues inside a group. Goes from 0 to 15, +* 15 being the highest priority. +*/ +__u8 priority; + +/** @pad: Padding fields, MBZ. */ +__u8 pad[3 ]; + +/** @ringbuf_size: Size of the ring buffer to allocate to this queue. */ +__u32 ringbuf_size; +}; + +/** +* enum drm_panthor_group_priority - Scheduling group priority +*/ +enum drm_panthor_group_priority { +/** @PANTHOR_GROUP_PRIORITY_LOW: Low priority group. */ +PANTHOR_GROUP_PRIORITY_LOW =0 , + +/** @PANTHOR_GROUP_PRIORITY_MEDIUM: Medium priority group. */ +PANTHOR_GROUP_PRIORITY_MEDIUM , + +/** @PANTHOR_GROUP_PRIORITY_HIGH: High priority group. */ +PANTHOR_GROUP_PRIORITY_HIGH , +}; + +/** +* struct drm_panthor_group_create - Arguments passed to DRM_IOCTL_PANTHOR_GROUP_CREATE +*/ +struct drm_panthor_group_create { +/** @queues: Array of drm_panthor_queue_create elements. */ +struct drm_panthor_obj_array queues; + +/** +* @max_compute_cores: Maximum number of cores that can be used by compute +* jobs across CS queues bound to this group. +* +* Must be less or equal to the number of bits set in @compute_core_mask. +*/ +__u8 max_compute_cores; + +/** +* @max_fragment_cores: Maximum number of cores that can be used by fragment +* jobs across CS queues bound to this group. +* +* Must be less or equal to the number of bits set in @fragment_core_mask. +*/ +__u8 max_fragment_cores; + +/** +* @max_tiler_cores: Maximum number of tilers that can be used by tiler jobs +* across CS queues bound to this group. +* +* Must be less or equal to the number of bits set in @tiler_core_mask. +*/ +__u8 max_tiler_cores; + +/** @priority: Group priority (see enum drm_panthor_group_priority). */ +__u8 priority; + +/** @pad: Padding field, MBZ. */ +__u32 pad; + +/** +* @compute_core_mask: Mask encoding cores that can be used for compute jobs. +* +* This field must have at least @max_compute_cores bits set. +* +* The bits set here should also be set in drm_panthor_gpu_info::shader_present. +*/ +__u64 compute_core_mask; + +/** +* @fragment_core_mask: Mask encoding cores that can be used for fragment jobs. +* +* This field must have at least @max_fragment_cores bits set. +* +* The bits set here should also be set in drm_panthor_gpu_info::shader_present. +*/ +__u64 fragment_core_mask; + +/** +* @tiler_core_mask: Mask encoding cores that can be used for tiler jobs. +* +* This field must have at least @max_tiler_cores bits set. +* +* The bits set here should also be set in drm_panthor_gpu_info::tiler_present. +*/ +__u64 tiler_core_mask; + +/** +* @vm_id: VM ID to bind this group to. +* +* All submission to queues bound to this group will use this VM. +*/ +__u32 vm_id; + +/** +* @group_handle: Returned group handle. Passed back when submitting jobs or +* destroying a group. +*/ +__u32 group_handle; +}; + +/** +* struct drm_panthor_group_destroy - Arguments passed to DRM_IOCTL_PANTHOR_GROUP_DESTROY +*/ +struct drm_panthor_group_destroy { +/** @group_handle: Group to destroy */ +__u32 group_handle; + +/** @pad: Padding field, MBZ. */ +__u32 pad; +}; + +/** +* struct drm_panthor_queue_submit - Job submission arguments. +* +* This is describing the userspace command stream to call from the kernel +* command stream ring-buffer. Queue submission is always part of a group +* submission, taking one or more jobs to submit to the underlying queues. +*/ +struct drm_panthor_queue_submit { +/** @queue_index: Index of the queue inside a group. */ +__u32 queue_index; + +/** +* @stream_size: Size of the command stream to execute. +* +* Must be 64-bit/8-byte aligned (the size of a CS instruction) +* +* Can be zero if stream_addr is zero too. +*/ +__u32 stream_size; + +/** +* @stream_addr: GPU address of the command stream to execute. +* +* Must be aligned on 64-byte. +* +* Can be zero is stream_size is zero too. +*/ +__u64 stream_addr; + +/** +* @latest_flush: FLUSH_ID read at the time the stream was built. +* +* This allows cache flush elimination for the automatic +* flush+invalidate(all) done at submission time, which is needed to +* ensure the GPU doesn't get garbage when reading the indirect command +* stream buffers. If you want the cache flush to happen +* unconditionally, pass a zero here. +*/ +__u32 latest_flush; + +/** @pad: MBZ. */ +__u32 pad; + +/** @syncs: Array of struct drm_panthor_sync_op sync operations. */ +struct drm_panthor_obj_array syncs; +}; + +/** +* struct drm_panthor_group_submit - Arguments passed to DRM_IOCTL_PANTHOR_VM_BIND +*/ +struct drm_panthor_group_submit { +/** @group_handle: Handle of the group to queue jobs to. */ +__u32 group_handle; + +/** @pad: MBZ. */ +__u32 pad; + +/** @queue_submits: Array of drm_panthor_queue_submit objects. */ +struct drm_panthor_obj_array queue_submits; +}; + +/** +* enum drm_panthor_group_state_flags - Group state flags +*/ +enum drm_panthor_group_state_flags { +/** +* @DRM_PANTHOR_GROUP_STATE_TIMEDOUT: Group had unfinished jobs. +* +* When a group ends up with this flag set, no jobs can be submitted to its queues. +*/ +DRM_PANTHOR_GROUP_STATE_TIMEDOUT =1 <<0 , + +/** +* @DRM_PANTHOR_GROUP_STATE_FATAL_FAULT: Group had fatal faults. +* +* When a group ends up with this flag set, no jobs can be submitted to its queues. +*/ +DRM_PANTHOR_GROUP_STATE_FATAL_FAULT =1 <<1 , +}; + +/** +* struct drm_panthor_group_get_state - Arguments passed to DRM_IOCTL_PANTHOR_GROUP_GET_STATE +* +* Used to query the state of a group and decide whether a new group should be created to +* replace it. +*/ +struct drm_panthor_group_get_state { +/** @group_handle: Handle of the group to query state on */ +__u32 group_handle; + +/** +* @state: Combination of DRM_PANTHOR_GROUP_STATE_* flags encoding the +* group state. +*/ +__u32 state; + +/** @fatal_queues: Bitmask of queues that faced fatal faults. */ +__u32 fatal_queues; + +/** @pad: MBZ */ +__u32 pad; +}; + +/** +* struct drm_panthor_tiler_heap_create - Arguments passed to DRM_IOCTL_PANTHOR_TILER_HEAP_CREATE +*/ +struct drm_panthor_tiler_heap_create { +/** @vm_id: VM ID the tiler heap should be mapped to */ +__u32 vm_id; + +/** @initial_chunk_count: Initial number of chunks to allocate. */ +__u32 initial_chunk_count; + +/** @chunk_size: Chunk size. Must be a power of two at least 256KB large. */ +__u32 chunk_size; + +/** @max_chunks: Maximum number of chunks that can be allocated. */ +__u32 max_chunks; + +/** +* @target_in_flight: Maximum number of in-flight render passes. +* +* If the heap has more than tiler jobs in-flight, the FW will wait for render +* passes to finish before queuing new tiler jobs. +*/ +__u32 target_in_flight; + +/** @handle: Returned heap handle. Passed back to DESTROY_TILER_HEAP. */ +__u32 handle; + +/** @tiler_heap_ctx_gpu_va: Returned heap GPU virtual address returned */ +__u64 tiler_heap_ctx_gpu_va; + +/** +* @first_heap_chunk_gpu_va: First heap chunk. +* +* The tiler heap is formed of heap chunks forming a single-link list. This +* is the first element in the list. +*/ +__u64 first_heap_chunk_gpu_va; +}; + +/** +* struct drm_panthor_tiler_heap_destroy - Arguments passed to DRM_IOCTL_PANTHOR_TILER_HEAP_DESTROY +*/ +struct drm_panthor_tiler_heap_destroy { +/** @handle: Handle of the tiler heap to destroy */ +__u32 handle; + +/** @pad: Padding field, MBZ. */ +__u32 pad; +}; + +#if defined(__cplusplus) +} +#endif + +#endif /* _PANTHOR_DRM_H_ */
Re: [PATCH v3 01/14] drm/panthor: Add uAPI
In reply to Boris BrezillonQuoted 19 lines
include/uapi/drm/panthor_drm.h new file +13
-0@@ -0,0 +1,892 @@ +/* SPDX-License-Identifier: MIT */ +/* Copyright (C) 2023 Collabora ltd. */ +#ifndef _PANTHOR_DRM_H_ +#define _PANTHOR_DRM_H_ + +#include "drm.h" + +#if defined(__cplusplus) +extern "C" { +#endif + +/** + * DOC: Introduction
...
include/uapi/drm/panthor_drm.h line 794
Quoted 13 lines
include/uapi/drm/panthor_drm.h new file +13
-0+/** +* struct drm_panthor_group_submit - Arguments passed to DRM_IOCTL_PANTHOR_VM_BIND +*/ +struct drm_panthor_group_submit { +/** @group_handle: Handle of the group to queue jobs to. */ +__u32 group_handle; + +/** @pad: MBZ. */ +__u32 pad; + +/** @queue_submits: Array of drm_panthor_queue_submit objects. */ +struct drm_panthor_obj_array queue_submits; +};
Hi! Very minor nit - but shouldn't the comment above say DRM_IOCTL_PANTHOR_GROUP_SUBMIT, not VM_BIND?
Quoted 88 lines
include/uapi/drm/panthor_drm.h new file +88
-0+ +/** +* enum drm_panthor_group_state_flags - Group state flags +*/ +enum drm_panthor_group_state_flags { +/** +* @DRM_PANTHOR_GROUP_STATE_TIMEDOUT: Group had unfinished jobs. +* +* When a group ends up with this flag set, no jobs can be submitted to its queues. +*/ +DRM_PANTHOR_GROUP_STATE_TIMEDOUT =1 <<0 , + +/** +* @DRM_PANTHOR_GROUP_STATE_FATAL_FAULT: Group had fatal faults. +* +* When a group ends up with this flag set, no jobs can be submitted to its queues. +*/ +DRM_PANTHOR_GROUP_STATE_FATAL_FAULT =1 <<1 , +}; + +/** +* struct drm_panthor_group_get_state - Arguments passed to DRM_IOCTL_PANTHOR_GROUP_GET_STATE +* +* Used to query the state of a group and decide whether a new group should be created to +* replace it. +*/ +struct drm_panthor_group_get_state { +/** @group_handle: Handle of the group to query state on */ +__u32 group_handle; + +/** +* @state: Combination of DRM_PANTHOR_GROUP_STATE_* flags encoding the +* group state. +*/ +__u32 state; + +/** @fatal_queues: Bitmask of queues that faced fatal faults. */ +__u32 fatal_queues; + +/** @pad: MBZ */ +__u32 pad; +}; + +/** +* struct drm_panthor_tiler_heap_create - Arguments passed to DRM_IOCTL_PANTHOR_TILER_HEAP_CREATE +*/ +struct drm_panthor_tiler_heap_create { +/** @vm_id: VM ID the tiler heap should be mapped to */ +__u32 vm_id; + +/** @initial_chunk_count: Initial number of chunks to allocate. */ +__u32 initial_chunk_count; + +/** @chunk_size: Chunk size. Must be a power of two at least 256KB large. */ +__u32 chunk_size; + +/** @max_chunks: Maximum number of chunks that can be allocated. */ +__u32 max_chunks; + +/** +* @target_in_flight: Maximum number of in-flight render passes. +* +* If the heap has more than tiler jobs in-flight, the FW will wait for render +* passes to finish before queuing new tiler jobs. +*/ +__u32 target_in_flight; + +/** @handle: Returned heap handle. Passed back to DESTROY_TILER_HEAP. */ +__u32 handle; + +/** @tiler_heap_ctx_gpu_va: Returned heap GPU virtual address returned */ +__u64 tiler_heap_ctx_gpu_va; + +/** +* @first_heap_chunk_gpu_va: First heap chunk. +* +* The tiler heap is formed of heap chunks forming a single-link list. This +* is the first element in the list. +*/ +__u64 first_heap_chunk_gpu_va; +}; + +/** +* struct drm_panthor_tiler_heap_destroy - Arguments passed to DRM_IOCTL_PANTHOR_TILER_HEAP_DESTROY +*/ +struct drm_panthor_tiler_heap_destroy { +/** @handle: Handle of the tiler heap to destroy */ +__u32 handle;
include/uapi/drm/panthor_drm.h line 892
Quoted 10 lines
include/uapi/drm/panthor_drm.h new file +10
-0+ +/** @pad: Padding field, MBZ. */ +__u32 pad; +}; + +#if defined(__cplusplus) +} +#endif + +#endif /* _PANTHOR_DRM_H_ */
Cheers,
Chris
Boris Brezillon
maintainerRe: [PATCH v3 01/14] drm/panthor: Add uAPI
Quoted 8 lines · On Mon, 18 Dec 2023 13:20:07 +0000 Chris Diamand <[email protected]>…
Quoted 8 lines
include/uapi/drm/panthor_drm.h new file +8
-0+/** +* struct drm_panthor_group_submit - Arguments passed to DRM_IOCTL_PANTHOR_VM_BIND +*/ +struct drm_panthor_group_submit { +/** @group_handle: Handle of the group to queue jobs to. */ +__u32 group_handle; + +/** @pad: MBZ. */
Quoted 8 lines
include/uapi/drm/panthor_drm.h line 794
Quoted 5 lines
include/uapi/drm/panthor_drm.h new file +5
-0+__u32 pad; + +/** @queue_submits: Array of drm_panthor_queue_submit objects. */ +struct drm_panthor_obj_array queue_submits; +};Hi! Very minor nit - but shouldn't the comment above say
DRM_IOCTL_PANTHOR_GROUP_SUBMIT, notVM_BIND?
Certainly, I'll fix it in v4.
Boris Brezillon
maintainerPATCH v3 2/14 drm/panthor: Add GPU register definitions
In reply to Boris BrezillonThose are the registers directly accessible through the MMIO range.
FW registers are exposed in panthor_fw.h.
v3:
- Add macros to extract GPU ID info
- Formatting changes
- Remove AS_TRANSCFG_ADRMODE_LEGACY - it doesn't exist post-CSF
- Remove CSF_GPU_LATEST_FLUSH_ID_DEFAULT
- Add GPU_L2_FEATURES_LINE_SIZE for extracting the GPU cache line size
--- drivers/gpu/drm/panthor/panthor_regs.h | 237 +++++++++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_regs.h
drivers/gpu/drm/panthor/panthor_regs.h new file +237 -0
@@ -0,0 +1,237 @@ +/* SPDX-License-Identifier: GPL-2.0 or MIT */ +/* Copyright 2018 Marty E. Plummer <[email protected]> */ +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ +/* +* Register definitions based on mali_kbase_gpu_regmap.h and +* mali_kbase_gpu_regmap_csf.h +* (C) COPYRIGHT 2010-2022 ARM Limited. All rights reserved. +*/ +#ifndef __PANTHOR_REGS_H__ +#define __PANTHOR_REGS_H__ + +#define GPU_ID 0x0 +#define GPU_ARCH_MAJOR (x) ((x) >> 28) +#define GPU_ARCH_MINOR (x) (((x) & GENMASK(27, 24)) >> 24) +#define GPU_ARCH_REV (x) (((x) & GENMASK(23, 20)) >> 20) +#define GPU_PROD_MAJOR (x) (((x) & GENMASK(19, 16)) >> 16) +#define GPU_VER_MAJOR (x) (((x) & GENMASK(15, 12)) >> 12) +#define GPU_VER_MINOR (x) (((x) & GENMASK(11, 4)) >> 4) +#define GPU_VER_STATUS (x) ((x) & GENMASK(3, 0)) + +#define GPU_L2_FEATURES 0x4 +#define GPU_L2_FEATURES_LINE_SIZE (x) (1 << ((x) & GENMASK(7, 0))) + +#define GPU_TILER_FEATURES 0xC +#define GPU_MEM_FEATURES 0x10 +#define GROUPS_L2_COHERENT BIT(0) + +#define GPU_MMU_FEATURES 0x14 +#define GPU_MMU_FEATURES_VA_BITS (x) ((x) & GENMASK(7, 0)) +#define GPU_MMU_FEATURES_PA_BITS (x) (((x) >> 8) & GENMASK(7, 0)) +#define GPU_AS_PRESENT 0x18 +#define GPU_CSF_ID 0x1C + +#define GPU_INT_RAWSTAT 0x20 +#define GPU_INT_CLEAR 0x24 +#define GPU_INT_MASK 0x28 +#define GPU_INT_STAT 0x2c +#define GPU_IRQ_FAULT BIT(0) +#define GPU_IRQ_PROTM_FAULT BIT(1) +#define GPU_IRQ_RESET_COMPLETED BIT(8) +#define GPU_IRQ_POWER_CHANGED BIT(9) +#define GPU_IRQ_POWER_CHANGED_ALL BIT(10) +#define GPU_IRQ_CLEAN_CACHES_COMPLETED BIT(17) +#define GPU_IRQ_DOORBELL_MIRROR BIT(18) +#define GPU_IRQ_MCU_STATUS_CHANGED BIT(19) +#define GPU_CMD 0x30 +#define GPU_CMD_DEF (type, payload) ((type) | ((payload) << 8)) +#define GPU_SOFT_RESET GPU_CMD_DEF(1, 1) +#define GPU_HARD_RESET GPU_CMD_DEF(1, 2) +#define CACHE_CLEAN BIT(0) +#define CACHE_INV BIT(1) +#define GPU_FLUSH_CACHES (l2, lsc, oth) \ + GPU_CMD_DEF(4, ((l2) << 0) | ((lsc) << 4) | ((oth) << 8)) + +#define GPU_STATUS 0x34 +#define GPU_STATUS_ACTIVE BIT(0) +#define GPU_STATUS_PWR_ACTIVE BIT(1) +#define GPU_STATUS_PAGE_FAULT BIT(4) +#define GPU_STATUS_PROTM_ACTIVE BIT(7) +#define GPU_STATUS_DBG_ENABLED BIT(8) + +#define GPU_FAULT_STATUS 0x3C +#define GPU_FAULT_ADDR_LO 0x40 +#define GPU_FAULT_ADDR_HI 0x44 + +#define GPU_PWR_KEY 0x50 +#define GPU_PWR_KEY_UNLOCK 0x2968A819 +#define GPU_PWR_OVERRIDE0 0x54 +#define GPU_PWR_OVERRIDE1 0x58 + +#define GPU_TIMESTAMP_OFFSET_LO 0x88 +#define GPU_TIMESTAMP_OFFSET_HI 0x8C +#define GPU_CYCLE_COUNT_LO 0x90 +#define GPU_CYCLE_COUNT_HI 0x94 +#define GPU_TIMESTAMP_LO 0x98 +#define GPU_TIMESTAMP_HI 0x9C + +#define GPU_THREAD_MAX_THREADS 0xA0 +#define GPU_THREAD_MAX_WORKGROUP_SIZE 0xA4 +#define GPU_THREAD_MAX_BARRIER_SIZE 0xA8 +#define GPU_THREAD_FEATURES 0xAC + +#define GPU_TEXTURE_FEATURES (n) (0xB0 + ((n) * 4)) + +#define GPU_SHADER_PRESENT_LO 0x100 +#define GPU_SHADER_PRESENT_HI 0x104 +#define GPU_TILER_PRESENT_LO 0x110 +#define GPU_TILER_PRESENT_HI 0x114 +#define GPU_L2_PRESENT_LO 0x120 +#define GPU_L2_PRESENT_HI 0x124 + +#define SHADER_READY_LO 0x140 +#define SHADER_READY_HI 0x144 +#define TILER_READY_LO 0x150 +#define TILER_READY_HI 0x154 +#define L2_READY_LO 0x160 +#define L2_READY_HI 0x164 + +#define SHADER_PWRON_LO 0x180 +#define SHADER_PWRON_HI 0x184 +#define TILER_PWRON_LO 0x190 +#define TILER_PWRON_HI 0x194 +#define L2_PWRON_LO 0x1A0 +#define L2_PWRON_HI 0x1A4 + +#define SHADER_PWROFF_LO 0x1C0 +#define SHADER_PWROFF_HI 0x1C4 +#define TILER_PWROFF_LO 0x1D0 +#define TILER_PWROFF_HI 0x1D4 +#define L2_PWROFF_LO 0x1E0 +#define L2_PWROFF_HI 0x1E4 + +#define SHADER_PWRTRANS_LO 0x200 +#define SHADER_PWRTRANS_HI 0x204 +#define TILER_PWRTRANS_LO 0x210 +#define TILER_PWRTRANS_HI 0x214 +#define L2_PWRTRANS_LO 0x220 +#define L2_PWRTRANS_HI 0x224 + +#define SHADER_PWRACTIVE_LO 0x240 +#define SHADER_PWRACTIVE_HI 0x244 +#define TILER_PWRACTIVE_LO 0x250 +#define TILER_PWRACTIVE_HI 0x254 +#define L2_PWRACTIVE_LO 0x260 +#define L2_PWRACTIVE_HI 0x264 + +#define GPU_REVID 0x280 + +#define GPU_COHERENCY_FEATURES 0x300 +#define GPU_COHERENCY_PROT_BIT (name) BIT(GPU_COHERENCY_ ## name) + +#define GPU_COHERENCY_PROTOCOL 0x304 +#define GPU_COHERENCY_ACE 0 +#define GPU_COHERENCY_ACE_LITE 1 +#define GPU_COHERENCY_NONE 31 + +#define MCU_CONTROL 0x700 +#define MCU_CONTROL_ENABLE 1 +#define MCU_CONTROL_AUTO 2 +#define MCU_CONTROL_DISABLE 0 + +#define MCU_STATUS 0x704 +#define MCU_STATUS_DISABLED 0 +#define MCU_STATUS_ENABLED 1 +#define MCU_STATUS_HALT 2 +#define MCU_STATUS_FATAL 3 + +/* Job Control regs */ +#define JOB_INT_RAWSTAT 0x1000 +#define JOB_INT_CLEAR 0x1004 +#define JOB_INT_MASK 0x1008 +#define JOB_INT_STAT 0x100c +#define JOB_INT_GLOBAL_IF BIT(31) +#define JOB_INT_CSG_IF (x) BIT(x) + +/* MMU regs */ +#define MMU_INT_RAWSTAT 0x2000 +#define MMU_INT_CLEAR 0x2004 +#define MMU_INT_MASK 0x2008 +#define MMU_INT_STAT 0x200c + +/* AS_COMMAND register commands */ + +#define MMU_BASE 0x2400 +#define MMU_AS_SHIFT 6 +#define MMU_AS (as) (MMU_BASE + ((as) << MMU_AS_SHIFT)) + +#define AS_TRANSTAB_LO (as) (MMU_AS(as) + 0x0) +#define AS_TRANSTAB_HI (as) (MMU_AS(as) + 0x4) +#define AS_MEMATTR_LO (as) (MMU_AS(as) + 0x8) +#define AS_MEMATTR_HI (as) (MMU_AS(as) + 0xC) +#define AS_MEMATTR_AARCH64_INNER_ALLOC_IMPL (2 << 2) +#define AS_MEMATTR_AARCH64_INNER_ALLOC_EXPL (w, r) ((3 << 2) | \ + ((w) ? BIT(0) : 0) | \ + ((r) ? BIT(1) : 0)) +#define AS_MEMATTR_AARCH64_SH_MIDGARD_INNER (0 << 4) +#define AS_MEMATTR_AARCH64_SH_CPU_INNER (1 << 4) +#define AS_MEMATTR_AARCH64_SH_CPU_INNER_SHADER_COH (2 << 4) +#define AS_MEMATTR_AARCH64_SHARED (0 << 6) +#define AS_MEMATTR_AARCH64_INNER_OUTER_NC (1 << 6) +#define AS_MEMATTR_AARCH64_INNER_OUTER_WB (2 << 6) +#define AS_MEMATTR_AARCH64_FAULT (3 << 6) +#define AS_LOCKADDR_LO (as) (MMU_AS(as) + 0x10) +#define AS_LOCKADDR_HI (as) (MMU_AS(as) + 0x14) +#define AS_COMMAND (as) (MMU_AS(as) + 0x18) +#define AS_COMMAND_NOP 0 +#define AS_COMMAND_UPDATE 1 +#define AS_COMMAND_LOCK 2 +#define AS_COMMAND_UNLOCK 3 +#define AS_COMMAND_FLUSH_PT 4 +#define AS_COMMAND_FLUSH_MEM 5 +#define AS_LOCK_REGION_MIN_SIZE (1ULL << 15) +#define AS_FAULTSTATUS (as) (MMU_AS(as) + 0x1C) +#define AS_FAULTSTATUS_ACCESS_TYPE_MASK (0x3 << 8) +#define AS_FAULTSTATUS_ACCESS_TYPE_ATOMIC (0x0 << 8) +#define AS_FAULTSTATUS_ACCESS_TYPE_EX (0x1 << 8) +#define AS_FAULTSTATUS_ACCESS_TYPE_READ (0x2 << 8) +#define AS_FAULTSTATUS_ACCESS_TYPE_WRITE (0x3 << 8) +#define AS_FAULTADDRESS_LO (as) (MMU_AS(as) + 0x20) +#define AS_FAULTADDRESS_HI (as) (MMU_AS(as) + 0x24) +#define AS_STATUS (as) (MMU_AS(as) + 0x28) +#define AS_STATUS_AS_ACTIVE BIT(0) +#define AS_TRANSCFG_LO (as) (MMU_AS(as) + 0x30) +#define AS_TRANSCFG_HI (as) (MMU_AS(as) + 0x34) +#define AS_TRANSCFG_ADRMODE_UNMAPPED (1 << 0) +#define AS_TRANSCFG_ADRMODE_IDENTITY (2 << 0) +#define AS_TRANSCFG_ADRMODE_AARCH64_4K (6 << 0) +#define AS_TRANSCFG_ADRMODE_AARCH64_64K (8 << 0) +#define AS_TRANSCFG_INA_BITS (x) ((x) << 6) +#define AS_TRANSCFG_OUTA_BITS (x) ((x) << 14) +#define AS_TRANSCFG_SL_CONCAT BIT(22) +#define AS_TRANSCFG_PTW_MEMATTR_NC (1 << 24) +#define AS_TRANSCFG_PTW_MEMATTR_WB (2 << 24) +#define AS_TRANSCFG_PTW_SH_NS (0 << 28) +#define AS_TRANSCFG_PTW_SH_OS (2 << 28) +#define AS_TRANSCFG_PTW_SH_IS (3 << 28) +#define AS_TRANSCFG_PTW_RA BIT(30) +#define AS_TRANSCFG_DISABLE_HIER_AP BIT(33) +#define AS_TRANSCFG_DISABLE_AF_FAULT BIT(34) +#define AS_TRANSCFG_WXN BIT(35) +#define AS_TRANSCFG_XREADABLE BIT(36) +#define AS_FAULTEXTRA_LO (as) (MMU_AS(as) + 0x38) +#define AS_FAULTEXTRA_HI (as) (MMU_AS(as) + 0x3C) + +#define CSF_GPU_LATEST_FLUSH_ID 0x10000 + +#define CSF_DOORBELL (i) (0x80000 + ((i) * 0x10000)) +#define CSF_GLB_DOORBELL_ID 0 + +#define gpu_write (dev, reg, data) \ + writel(data, (dev)->iomem + (reg)) + +#define gpu_read (dev, reg) \ + readl((dev)->iomem + (reg)) + +#endif
Signature · 1 line
2.43.0
Re: [PATCH v3 02/14] drm/panthor: Add GPU register definitions
Quoted 11 lines · Boris Brezillon
Those are the registers directly accessible through the MMIO range.
FW registers are exposed in
panthor_fw.h.v3:
- Add macros to extract GPU ID info
- Formatting changes
- RemoveAS_TRANSCFG_ADRMODE_LEGACY- it doesn't exist post-CSF
- RemoveCSF_GPU_LATEST_FLUSH_ID_DEFAULT
- AddGPU_L2_FEATURES_LINE_SIZEfor extracting the GPU cache line size
Quoted 5 lines
Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>
Thanks,
Steve
Quoted 248 lines
--- drivers/gpu/drm/panthor/panthor_regs.h | 237 +++++++++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_regs.hdrivers/gpu/drm/panthor/panthor_regs.h new file +237
-0@@ -0,0 +1,237 @@ +/* SPDX-License-Identifier: GPL-2.0 or MIT */ +/* Copyright 2018 Marty E. Plummer <[email protected]> */ +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ +/* +* Register definitions based on mali_kbase_gpu_regmap.h and +* mali_kbase_gpu_regmap_csf.h +* (C) COPYRIGHT 2010-2022 ARM Limited. All rights reserved. +*/ +#ifndef __PANTHOR_REGS_H__ +#define __PANTHOR_REGS_H__ + +#define GPU_ID 0x0 +#define GPU_ARCH_MAJOR (x) ((x) >> 28) +#define GPU_ARCH_MINOR (x) (((x) & GENMASK(27, 24)) >> 24) +#define GPU_ARCH_REV (x) (((x) & GENMASK(23, 20)) >> 20) +#define GPU_PROD_MAJOR (x) (((x) & GENMASK(19, 16)) >> 16) +#define GPU_VER_MAJOR (x) (((x) & GENMASK(15, 12)) >> 12) +#define GPU_VER_MINOR (x) (((x) & GENMASK(11, 4)) >> 4) +#define GPU_VER_STATUS (x) ((x) & GENMASK(3, 0)) + +#define GPU_L2_FEATURES 0x4 +#define GPU_L2_FEATURES_LINE_SIZE (x) (1 << ((x) & GENMASK(7, 0))) + +#define GPU_TILER_FEATURES 0xC +#define GPU_MEM_FEATURES 0x10 +#define GROUPS_L2_COHERENT BIT(0) + +#define GPU_MMU_FEATURES 0x14 +#define GPU_MMU_FEATURES_VA_BITS (x) ((x) & GENMASK(7, 0)) +#define GPU_MMU_FEATURES_PA_BITS (x) (((x) >> 8) & GENMASK(7, 0)) +#define GPU_AS_PRESENT 0x18 +#define GPU_CSF_ID 0x1C + +#define GPU_INT_RAWSTAT 0x20 +#define GPU_INT_CLEAR 0x24 +#define GPU_INT_MASK 0x28 +#define GPU_INT_STAT 0x2c +#define GPU_IRQ_FAULT BIT(0) +#define GPU_IRQ_PROTM_FAULT BIT(1) +#define GPU_IRQ_RESET_COMPLETED BIT(8) +#define GPU_IRQ_POWER_CHANGED BIT(9) +#define GPU_IRQ_POWER_CHANGED_ALL BIT(10) +#define GPU_IRQ_CLEAN_CACHES_COMPLETED BIT(17) +#define GPU_IRQ_DOORBELL_MIRROR BIT(18) +#define GPU_IRQ_MCU_STATUS_CHANGED BIT(19) +#define GPU_CMD 0x30 +#define GPU_CMD_DEF (type, payload) ((type) | ((payload) << 8)) +#define GPU_SOFT_RESET GPU_CMD_DEF(1, 1) +#define GPU_HARD_RESET GPU_CMD_DEF(1, 2) +#define CACHE_CLEAN BIT(0) +#define CACHE_INV BIT(1) +#define GPU_FLUSH_CACHES (l2, lsc, oth) \ + GPU_CMD_DEF(4, ((l2) << 0) | ((lsc) << 4) | ((oth) << 8)) + +#define GPU_STATUS 0x34 +#define GPU_STATUS_ACTIVE BIT(0) +#define GPU_STATUS_PWR_ACTIVE BIT(1) +#define GPU_STATUS_PAGE_FAULT BIT(4) +#define GPU_STATUS_PROTM_ACTIVE BIT(7) +#define GPU_STATUS_DBG_ENABLED BIT(8) + +#define GPU_FAULT_STATUS 0x3C +#define GPU_FAULT_ADDR_LO 0x40 +#define GPU_FAULT_ADDR_HI 0x44 + +#define GPU_PWR_KEY 0x50 +#define GPU_PWR_KEY_UNLOCK 0x2968A819 +#define GPU_PWR_OVERRIDE0 0x54 +#define GPU_PWR_OVERRIDE1 0x58 + +#define GPU_TIMESTAMP_OFFSET_LO 0x88 +#define GPU_TIMESTAMP_OFFSET_HI 0x8C +#define GPU_CYCLE_COUNT_LO 0x90 +#define GPU_CYCLE_COUNT_HI 0x94 +#define GPU_TIMESTAMP_LO 0x98 +#define GPU_TIMESTAMP_HI 0x9C + +#define GPU_THREAD_MAX_THREADS 0xA0 +#define GPU_THREAD_MAX_WORKGROUP_SIZE 0xA4 +#define GPU_THREAD_MAX_BARRIER_SIZE 0xA8 +#define GPU_THREAD_FEATURES 0xAC + +#define GPU_TEXTURE_FEATURES (n) (0xB0 + ((n) * 4)) + +#define GPU_SHADER_PRESENT_LO 0x100 +#define GPU_SHADER_PRESENT_HI 0x104 +#define GPU_TILER_PRESENT_LO 0x110 +#define GPU_TILER_PRESENT_HI 0x114 +#define GPU_L2_PRESENT_LO 0x120 +#define GPU_L2_PRESENT_HI 0x124 + +#define SHADER_READY_LO 0x140 +#define SHADER_READY_HI 0x144 +#define TILER_READY_LO 0x150 +#define TILER_READY_HI 0x154 +#define L2_READY_LO 0x160 +#define L2_READY_HI 0x164 + +#define SHADER_PWRON_LO 0x180 +#define SHADER_PWRON_HI 0x184 +#define TILER_PWRON_LO 0x190 +#define TILER_PWRON_HI 0x194 +#define L2_PWRON_LO 0x1A0 +#define L2_PWRON_HI 0x1A4 + +#define SHADER_PWROFF_LO 0x1C0 +#define SHADER_PWROFF_HI 0x1C4 +#define TILER_PWROFF_LO 0x1D0 +#define TILER_PWROFF_HI 0x1D4 +#define L2_PWROFF_LO 0x1E0 +#define L2_PWROFF_HI 0x1E4 + +#define SHADER_PWRTRANS_LO 0x200 +#define SHADER_PWRTRANS_HI 0x204 +#define TILER_PWRTRANS_LO 0x210 +#define TILER_PWRTRANS_HI 0x214 +#define L2_PWRTRANS_LO 0x220 +#define L2_PWRTRANS_HI 0x224 + +#define SHADER_PWRACTIVE_LO 0x240 +#define SHADER_PWRACTIVE_HI 0x244 +#define TILER_PWRACTIVE_LO 0x250 +#define TILER_PWRACTIVE_HI 0x254 +#define L2_PWRACTIVE_LO 0x260 +#define L2_PWRACTIVE_HI 0x264 + +#define GPU_REVID 0x280 + +#define GPU_COHERENCY_FEATURES 0x300 +#define GPU_COHERENCY_PROT_BIT (name) BIT(GPU_COHERENCY_ ## name) + +#define GPU_COHERENCY_PROTOCOL 0x304 +#define GPU_COHERENCY_ACE 0 +#define GPU_COHERENCY_ACE_LITE 1 +#define GPU_COHERENCY_NONE 31 + +#define MCU_CONTROL 0x700 +#define MCU_CONTROL_ENABLE 1 +#define MCU_CONTROL_AUTO 2 +#define MCU_CONTROL_DISABLE 0 + +#define MCU_STATUS 0x704 +#define MCU_STATUS_DISABLED 0 +#define MCU_STATUS_ENABLED 1 +#define MCU_STATUS_HALT 2 +#define MCU_STATUS_FATAL 3 + +/* Job Control regs */ +#define JOB_INT_RAWSTAT 0x1000 +#define JOB_INT_CLEAR 0x1004 +#define JOB_INT_MASK 0x1008 +#define JOB_INT_STAT 0x100c +#define JOB_INT_GLOBAL_IF BIT(31) +#define JOB_INT_CSG_IF (x) BIT(x) + +/* MMU regs */ +#define MMU_INT_RAWSTAT 0x2000 +#define MMU_INT_CLEAR 0x2004 +#define MMU_INT_MASK 0x2008 +#define MMU_INT_STAT 0x200c + +/* AS_COMMAND register commands */ + +#define MMU_BASE 0x2400 +#define MMU_AS_SHIFT 6 +#define MMU_AS (as) (MMU_BASE + ((as) << MMU_AS_SHIFT)) + +#define AS_TRANSTAB_LO (as) (MMU_AS(as) + 0x0) +#define AS_TRANSTAB_HI (as) (MMU_AS(as) + 0x4) +#define AS_MEMATTR_LO (as) (MMU_AS(as) + 0x8) +#define AS_MEMATTR_HI (as) (MMU_AS(as) + 0xC) +#define AS_MEMATTR_AARCH64_INNER_ALLOC_IMPL (2 << 2) +#define AS_MEMATTR_AARCH64_INNER_ALLOC_EXPL (w, r) ((3 << 2) | \ + ((w) ? BIT(0) : 0) | \ + ((r) ? BIT(1) : 0)) +#define AS_MEMATTR_AARCH64_SH_MIDGARD_INNER (0 << 4) +#define AS_MEMATTR_AARCH64_SH_CPU_INNER (1 << 4) +#define AS_MEMATTR_AARCH64_SH_CPU_INNER_SHADER_COH (2 << 4) +#define AS_MEMATTR_AARCH64_SHARED (0 << 6) +#define AS_MEMATTR_AARCH64_INNER_OUTER_NC (1 << 6) +#define AS_MEMATTR_AARCH64_INNER_OUTER_WB (2 << 6) +#define AS_MEMATTR_AARCH64_FAULT (3 << 6) +#define AS_LOCKADDR_LO (as) (MMU_AS(as) + 0x10) +#define AS_LOCKADDR_HI (as) (MMU_AS(as) + 0x14) +#define AS_COMMAND (as) (MMU_AS(as) + 0x18) +#define AS_COMMAND_NOP 0 +#define AS_COMMAND_UPDATE 1 +#define AS_COMMAND_LOCK 2 +#define AS_COMMAND_UNLOCK 3 +#define AS_COMMAND_FLUSH_PT 4 +#define AS_COMMAND_FLUSH_MEM 5 +#define AS_LOCK_REGION_MIN_SIZE (1ULL << 15) +#define AS_FAULTSTATUS (as) (MMU_AS(as) + 0x1C) +#define AS_FAULTSTATUS_ACCESS_TYPE_MASK (0x3 << 8) +#define AS_FAULTSTATUS_ACCESS_TYPE_ATOMIC (0x0 << 8) +#define AS_FAULTSTATUS_ACCESS_TYPE_EX (0x1 << 8) +#define AS_FAULTSTATUS_ACCESS_TYPE_READ (0x2 << 8) +#define AS_FAULTSTATUS_ACCESS_TYPE_WRITE (0x3 << 8) +#define AS_FAULTADDRESS_LO (as) (MMU_AS(as) + 0x20) +#define AS_FAULTADDRESS_HI (as) (MMU_AS(as) + 0x24) +#define AS_STATUS (as) (MMU_AS(as) + 0x28) +#define AS_STATUS_AS_ACTIVE BIT(0) +#define AS_TRANSCFG_LO (as) (MMU_AS(as) + 0x30) +#define AS_TRANSCFG_HI (as) (MMU_AS(as) + 0x34) +#define AS_TRANSCFG_ADRMODE_UNMAPPED (1 << 0) +#define AS_TRANSCFG_ADRMODE_IDENTITY (2 << 0) +#define AS_TRANSCFG_ADRMODE_AARCH64_4K (6 << 0) +#define AS_TRANSCFG_ADRMODE_AARCH64_64K (8 << 0) +#define AS_TRANSCFG_INA_BITS (x) ((x) << 6) +#define AS_TRANSCFG_OUTA_BITS (x) ((x) << 14) +#define AS_TRANSCFG_SL_CONCAT BIT(22) +#define AS_TRANSCFG_PTW_MEMATTR_NC (1 << 24) +#define AS_TRANSCFG_PTW_MEMATTR_WB (2 << 24) +#define AS_TRANSCFG_PTW_SH_NS (0 << 28) +#define AS_TRANSCFG_PTW_SH_OS (2 << 28) +#define AS_TRANSCFG_PTW_SH_IS (3 << 28) +#define AS_TRANSCFG_PTW_RA BIT(30) +#define AS_TRANSCFG_DISABLE_HIER_AP BIT(33) +#define AS_TRANSCFG_DISABLE_AF_FAULT BIT(34) +#define AS_TRANSCFG_WXN BIT(35) +#define AS_TRANSCFG_XREADABLE BIT(36) +#define AS_FAULTEXTRA_LO (as) (MMU_AS(as) + 0x38) +#define AS_FAULTEXTRA_HI (as) (MMU_AS(as) + 0x3C) + +#define CSF_GPU_LATEST_FLUSH_ID 0x10000 + +#define CSF_DOORBELL (i) (0x80000 + ((i) * 0x10000)) +#define CSF_GLB_DOORBELL_ID 0 + +#define gpu_write (dev, reg, data) \ + writel(data, (dev)->iomem + (reg)) + +#define gpu_read (dev, reg) \ + readl((dev)->iomem + (reg)) + +#endif
Boris Brezillon
maintainerPATCH v3 3/14 drm/panthor: Add the device logical block
In reply to Boris BrezillonThe panthor driver is designed in a modular way, where each logical block is dealing with a specific HW-block or software feature. In order for those blocks to communicate with each other, we need a central panthor_device collecting all the blocks, and exposing some common features, like interrupt handling, power management, reset, ...
This what this panthor_device logical block is about.
v3:
- Add acks for the MIT+GPL2 relicensing
- Fix 32-bit support
- Shorten the sections protected by panthor_device::pm::mmio_lock to fix lock ordering issues.
- Rename panthor_device::pm::lock into panthor_device::pm::mmio_lock to better reflect what this lock is protecting
- Use dev_err_probe()
- Make sure we call drm_dev_exit() when something fails half-way in panthor_device_reset_work()
- Replace CSF_GPU_LATEST_FLUSH_ID_DEFAULT with a constant '1' and a comment to explain. Also remove setting the dummy flush ID on suspend.
- Remove drm_WARN_ON() in panthor_exception_name()
- Check pirq->suspended in panthor_xxx_irq_raw_handler()
--- drivers/gpu/drm/panthor/panthor_device.c | 497 +++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_device.h | 381 +++++++++++++++++ 2 files changed, 878 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_device.c create mode 100644 drivers/gpu/drm/panthor/panthor_device.h
drivers/gpu/drm/panthor/panthor_device.c new file +497 -0
@@ -0,0 +1,497 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2018 Marty E. Plummer <[email protected]> */ +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ + +#include <linux/clk.h> +#include <linux/reset.h> +#include <linux/platform_device.h> +#include <linux/pm_domain.h> +#include <linux/pm_runtime.h> +#include <linux/regulator/consumer.h> + +#include <drm/drm_drv.h> +#include <drm/drm_managed.h> + +#include "panthor_sched.h" +#include "panthor_device.h" +#include "panthor_devfreq.h" +#include "panthor_gpu.h" +#include "panthor_fw.h" +#include "panthor_mmu.h" +#include "panthor_regs.h" + +static int panthor_clk_init (struct panthor_device *ptdev) +{ + ptdev->clks.core =devm_clk_get (ptdev->base.dev,NULL ); +if (IS_ERR (ptdev->clks.core)) +return dev_err_probe (ptdev->base.dev, +PTR_ERR (ptdev->clks.core), +"get 'core' clock failed" ); + + ptdev->clks.stacks =devm_clk_get_optional (ptdev->base.dev,"stacks" ); +if (IS_ERR (ptdev->clks.stacks)) +return dev_err_probe (ptdev->base.dev, +PTR_ERR (ptdev->clks.stacks), +"get 'stacks' clock failed" ); + + ptdev->clks.coregroup =devm_clk_get_optional (ptdev->base.dev,"coregroup" ); +if (IS_ERR (ptdev->clks.coregroup)) +return dev_err_probe (ptdev->base.dev, +PTR_ERR (ptdev->clks.coregroup), +"get 'coregroup' clock failed" ); + +drm_info (&ptdev->base,"clock rate = %lu\n" ,clk_get_rate (ptdev->clks.core)); +return 0 ; +} + +void panthor_device_unplug (struct panthor_device *ptdev) +{ +/* FIXME: This is racy. */ +if (drm_dev_is_unplugged (&ptdev->base)) +return ; + +drm_WARN_ON (&ptdev->base,pm_runtime_get_sync (ptdev->base.dev) <0 ); + +/* Call drm_dev_unplug() so any access to HW block happening after +* that point get rejected. +*/ +drm_dev_unplug (&ptdev->base); + +/* Now, try to cleanly shutdown the GPU before the device resources +* get reclaimed. +*/ +panthor_sched_unplug (ptdev); +panthor_fw_unplug (ptdev); +panthor_mmu_unplug (ptdev); +panthor_gpu_unplug (ptdev); + +pm_runtime_dont_use_autosuspend (ptdev->base.dev); +pm_runtime_put_sync_suspend (ptdev->base.dev); +} + +static void panthor_device_reset_cleanup (struct drm_device *ddev,void *data) +{ + struct panthor_device *ptdev = container_of(ddev, struct panthor_device, base); + +cancel_work_sync (&ptdev->reset.work); +destroy_workqueue (ptdev->reset.wq); +} + +static void panthor_device_reset_work (struct work_struct *work) +{ + struct panthor_device *ptdev = container_of(work, struct panthor_device, reset.work); +int ret =0 , cookie; + +if (atomic_read (&ptdev->pm.state) !=PANTHOR_DEVICE_PM_STATE_ACTIVE ) { +/* +* No need for a reset as the device has been (or will be) +* powered down +*/ +atomic_set (&ptdev->reset.pending,0 ); +return ; + } + +if (!drm_dev_enter (&ptdev->base, &cookie)) +return ; + +panthor_sched_pre_reset (ptdev); +panthor_fw_pre_reset (ptdev, true); +panthor_mmu_pre_reset (ptdev); +panthor_gpu_soft_reset (ptdev); +panthor_gpu_l2_power_on (ptdev); +panthor_mmu_post_reset (ptdev); + ret =panthor_fw_post_reset (ptdev); +if (ret) + goto out_dev_exit; + +atomic_set (&ptdev->reset.pending,0 ); +panthor_sched_post_reset (ptdev); + +if (ret) { +panthor_device_unplug (ptdev); +drm_err (&ptdev->base,"Failed to boot MCU after reset, making device unusable." ); + } + +out_dev_exit: +drm_dev_exit (cookie); +} + +static bool panthor_device_is_initialized (struct panthor_device *ptdev) +{ +return !!ptdev->scheduler; +} + +static void panthor_device_free_page (struct drm_device *ddev,void *data) +{ +free_page ((unsigned long )data); +} + +int panthor_device_init (struct panthor_device *ptdev) +{ +struct resource *res; +struct page *p; +int ret; + + ptdev->coherent =device_get_dma_attr (ptdev->base.dev) ==DEV_DMA_COHERENT ; + +drmm_mutex_init (&ptdev->base, &ptdev->pm.mmio_lock); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_SUSPENDED ); + p =alloc_page (GFP_KERNEL | __GFP_ZERO); +if (!p) +return -ENOMEM ; + + ptdev->pm.dummy_latest_flush =page_address (p); + ret =drmm_add_action_or_reset (&ptdev->base, panthor_device_free_page, + ptdev->pm.dummy_latest_flush); +if (ret) +return ret; + +/* +* Set the dummy page holding the latest flush to 1. This will cause the +* flush to avoided as we know it isn't necessary if the submission +* happens while the dummy page is mapped. Zero cannot be used because +* that means 'always flush'. +*/ + *ptdev->pm.dummy_latest_flush =1 ; + +INIT_WORK (&ptdev->reset.work, panthor_device_reset_work); + ptdev->reset.wq =alloc_ordered_workqueue ("panthor-reset-wq" ,0 ); +if (!ptdev->reset.wq) +return -ENOMEM ; + + ret =drmm_add_action_or_reset (&ptdev->base, panthor_device_reset_cleanup,NULL ); +if (ret) +return ret; + + ret =panthor_clk_init (ptdev); +if (ret) +return ret; + + ret =panthor_devfreq_init (ptdev); +if (ret) +return ret; + + ptdev->iomem =devm_platform_get_and_ioremap_resource (to_platform_device (ptdev->base.dev), +0 , &res); +if (IS_ERR (ptdev->iomem)) +return PTR_ERR (ptdev->iomem); + + ptdev->phys_addr = res->start; + + ret =devm_pm_runtime_enable (ptdev->base.dev); +if (ret) +return ret; + + ret =pm_runtime_resume_and_get (ptdev->base.dev); +if (ret) +return ret; + + ret =panthor_gpu_init (ptdev); +if (ret) + goto err_rpm_put; + + ret =panthor_mmu_init (ptdev); +if (ret) + goto err_rpm_put; + + ret =panthor_fw_init (ptdev); +if (ret) + goto err_rpm_put; + + ret =panthor_sched_init (ptdev); +if (ret) + goto err_rpm_put; + +/* ~3 frames */ +pm_runtime_set_autosuspend_delay (ptdev->base.dev,50 ); +pm_runtime_use_autosuspend (ptdev->base.dev); +pm_runtime_put_autosuspend (ptdev->base.dev); +return 0 ; + +err_rpm_put: +pm_runtime_put_sync_suspend (ptdev->base.dev); +return ret; +} + +#define PANTHOR_EXCEPTION (id) \ + [DRM_PANTHOR_EXCEPTION_ ## id] = { \ + .name = #id, \ + } + +struct panthor_exception_info { +const char *name; +}; + +static const struct panthor_exception_info panthor_exception_infos[] = { +PANTHOR_EXCEPTION (OK ), +PANTHOR_EXCEPTION (TERMINATED ), +PANTHOR_EXCEPTION (KABOOM ), +PANTHOR_EXCEPTION (EUREKA ), +PANTHOR_EXCEPTION (ACTIVE ), +PANTHOR_EXCEPTION (CS_RES_TERM ), +PANTHOR_EXCEPTION (CS_CONFIG_FAULT ), +PANTHOR_EXCEPTION (CS_ENDPOINT_FAULT ), +PANTHOR_EXCEPTION (CS_BUS_FAULT ), +PANTHOR_EXCEPTION (CS_INSTR_INVALID ), +PANTHOR_EXCEPTION (CS_CALL_STACK_OVERFLOW ), +PANTHOR_EXCEPTION (CS_INHERIT_FAULT ), +PANTHOR_EXCEPTION (INSTR_INVALID_PC ), +PANTHOR_EXCEPTION (INSTR_INVALID_ENC ), +PANTHOR_EXCEPTION (INSTR_BARRIER_FAULT ), +PANTHOR_EXCEPTION (DATA_INVALID_FAULT ), +PANTHOR_EXCEPTION (TILE_RANGE_FAULT ), +PANTHOR_EXCEPTION (ADDR_RANGE_FAULT ), +PANTHOR_EXCEPTION (IMPRECISE_FAULT ), +PANTHOR_EXCEPTION (OOM ), +PANTHOR_EXCEPTION (CSF_FW_INTERNAL_ERROR ), +PANTHOR_EXCEPTION (CSF_RES_EVICTION_TIMEOUT ), +PANTHOR_EXCEPTION (GPU_BUS_FAULT ), +PANTHOR_EXCEPTION (GPU_SHAREABILITY_FAULT ), +PANTHOR_EXCEPTION (SYS_SHAREABILITY_FAULT ), +PANTHOR_EXCEPTION (GPU_CACHEABILITY_FAULT ), +PANTHOR_EXCEPTION (TRANSLATION_FAULT_0 ), +PANTHOR_EXCEPTION (TRANSLATION_FAULT_1 ), +PANTHOR_EXCEPTION (TRANSLATION_FAULT_2 ), +PANTHOR_EXCEPTION (TRANSLATION_FAULT_3 ), +PANTHOR_EXCEPTION (TRANSLATION_FAULT_4 ), +PANTHOR_EXCEPTION (PERM_FAULT_0 ), +PANTHOR_EXCEPTION (PERM_FAULT_1 ), +PANTHOR_EXCEPTION (PERM_FAULT_2 ), +PANTHOR_EXCEPTION (PERM_FAULT_3 ), +PANTHOR_EXCEPTION (ACCESS_FLAG_1 ), +PANTHOR_EXCEPTION (ACCESS_FLAG_2 ), +PANTHOR_EXCEPTION (ACCESS_FLAG_3 ), +PANTHOR_EXCEPTION (ADDR_SIZE_FAULT_IN ), +PANTHOR_EXCEPTION (ADDR_SIZE_FAULT_OUT0 ), +PANTHOR_EXCEPTION (ADDR_SIZE_FAULT_OUT1 ), +PANTHOR_EXCEPTION (ADDR_SIZE_FAULT_OUT2 ), +PANTHOR_EXCEPTION (ADDR_SIZE_FAULT_OUT3 ), +PANTHOR_EXCEPTION (MEM_ATTR_FAULT_0 ), +PANTHOR_EXCEPTION (MEM_ATTR_FAULT_1 ), +PANTHOR_EXCEPTION (MEM_ATTR_FAULT_2 ), +PANTHOR_EXCEPTION (MEM_ATTR_FAULT_3 ), +}; + +const char *panthor_exception_name (struct panthor_device *ptdev,u32 exception_code) +{ +if (exception_code >=ARRAY_SIZE (panthor_exception_infos) || + !panthor_exception_infos[exception_code].name) +return "Unknown exception type" ; + +return panthor_exception_infos[exception_code].name; +} + +static vm_fault_t panthor_mmio_vm_fault (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct panthor_device *ptdev = vma->vm_private_data; +u64 id = (u64 )vma->vm_pgoff <<PAGE_SHIFT ; +unsigned long pfn; +pgprot_t pgprot; +vm_fault_t ret; +bool active; +int cookie; + +if (!drm_dev_enter (&ptdev->base, &cookie)) +return VM_FAULT_SIGBUS ; + +mutex_lock (&ptdev->pm.mmio_lock); + active =atomic_read (&ptdev->pm.state) ==PANTHOR_DEVICE_PM_STATE_ACTIVE ; + +switch (panthor_device_mmio_offset (id)) { +case DRM_PANTHOR_USER_FLUSH_ID_MMIO_OFFSET : +if (active) + pfn =__phys_to_pfn (ptdev->phys_addr +CSF_GPU_LATEST_FLUSH_ID ); +else + pfn =virt_to_pfn (ptdev->pm.dummy_latest_flush); +break ; + +default : + ret =VM_FAULT_SIGBUS ; + goto out_unlock; + } + + pgprot = vma->vm_page_prot; +if (active) + pgprot =pgprot_noncached (pgprot); + + ret =vmf_insert_pfn_prot (vma, vmf->address, pfn, pgprot); + +out_unlock: +mutex_unlock (&ptdev->pm.mmio_lock); +drm_dev_exit (cookie); +return ret; +} + +static const struct vm_operations_struct panthor_mmio_vm_ops = { + .fault = panthor_mmio_vm_fault, +}; + +int panthor_device_mmap_io (struct panthor_device *ptdev,struct vm_area_struct *vma) +{ +u64 id = (u64 )vma->vm_pgoff <<PAGE_SHIFT ; + +switch (panthor_device_mmio_offset (id)) { +case DRM_PANTHOR_USER_FLUSH_ID_MMIO_OFFSET : +if (vma->vm_end - vma->vm_start !=PAGE_SIZE || + (vma->vm_flags & (VM_WRITE |VM_EXEC ))) +return -EINVAL ; + +break ; + +default : +return -EINVAL ; + } + +/* Defer actual mapping to the fault handler. */ + vma->vm_private_data = ptdev; + vma->vm_ops = &panthor_mmio_vm_ops; +vm_flags_set (vma, +VM_IO |VM_DONTCOPY |VM_DONTEXPAND | +VM_NORESERVE |VM_DONTDUMP |VM_PFNMAP ); +return 0 ; +} + +#ifdef CONFIG_PM +int panthor_device_resume (struct device *dev) +{ +struct panthor_device *ptdev =dev_get_drvdata (dev); +int ret, cookie; + +if (atomic_read (&ptdev->pm.state) !=PANTHOR_DEVICE_PM_STATE_SUSPENDED ) +return -EINVAL ; + +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_RESUMING ); + + ret =clk_prepare_enable (ptdev->clks.core); +if (ret) + goto err_set_suspended; + + ret =clk_prepare_enable (ptdev->clks.stacks); +if (ret) + goto err_disable_core_clk; + + ret =clk_prepare_enable (ptdev->clks.coregroup); +if (ret) + goto err_disable_stacks_clk; + + ret =panthor_devfreq_resume (ptdev); +if (ret) + goto err_disable_coregroup_clk; + +if (panthor_device_is_initialized (ptdev) && +drm_dev_enter (&ptdev->base, &cookie)) { +panthor_gpu_resume (ptdev); +panthor_mmu_resume (ptdev); + ret =drm_WARN_ON (&ptdev->base,panthor_fw_resume (ptdev)); +if (!ret) +panthor_sched_resume (ptdev); + +drm_dev_exit (cookie); + +if (ret) + goto err_devfreq_suspend; + } + +if (atomic_read (&ptdev->reset.pending)) +queue_work (ptdev->reset.wq, &ptdev->reset.work); + +/* Clear all IOMEM mappings pointing to this device after we've +* resumed. This way the fake mappings pointing to the dummy pages +* are removed and the real iomem mapping will be restored on next +* access. +*/ +mutex_lock (&ptdev->pm.mmio_lock); +unmap_mapping_range (ptdev->base.anon_inode->i_mapping, +DRM_PANTHOR_USER_MMIO_OFFSET ,0 ,1 ); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_ACTIVE ); +mutex_unlock (&ptdev->pm.mmio_lock); +return 0 ; + +err_devfreq_suspend: +panthor_devfreq_suspend (ptdev); + +err_disable_coregroup_clk: +clk_disable_unprepare (ptdev->clks.coregroup); + +err_disable_stacks_clk: +clk_disable_unprepare (ptdev->clks.stacks); + +err_disable_core_clk: +clk_disable_unprepare (ptdev->clks.core); + +err_set_suspended: +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_SUSPENDED ); +return ret; +} + +int panthor_device_suspend (struct device *dev) +{ +struct panthor_device *ptdev =dev_get_drvdata (dev); +int ret, cookie; + +if (atomic_read (&ptdev->pm.state) !=PANTHOR_DEVICE_PM_STATE_ACTIVE ) +return -EINVAL ; + +/* Clear all IOMEM mappings pointing to this device before we +* shutdown the power-domain and clocks. Failing to do that results +* in external aborts when the process accesses the iomem region. +* We change the state and call unmap_mapping_range() with the +* mmio_lock held to make sure the vm_fault handler won't set up +* invalid mappings. +*/ +mutex_lock (&ptdev->pm.mmio_lock); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_SUSPENDING ); +unmap_mapping_range (ptdev->base.anon_inode->i_mapping, +DRM_PANTHOR_USER_MMIO_OFFSET ,0 ,1 ); +mutex_unlock (&ptdev->pm.mmio_lock); + +if (panthor_device_is_initialized (ptdev) && +drm_dev_enter (&ptdev->base, &cookie)) { +cancel_work_sync (&ptdev->reset.work); + +/* We prepare everything as if we were resetting the GPU. +* The end of the reset will happen in the resume path though. +*/ +panthor_sched_suspend (ptdev); +panthor_fw_suspend (ptdev); +panthor_mmu_suspend (ptdev); +panthor_gpu_suspend (ptdev); +drm_dev_exit (cookie); + } + + ret =panthor_devfreq_suspend (ptdev); +if (ret) { +if (panthor_device_is_initialized (ptdev) && +drm_dev_enter (&ptdev->base, &cookie)) { +panthor_gpu_resume (ptdev); +panthor_mmu_resume (ptdev); +drm_WARN_ON (&ptdev->base,panthor_fw_resume (ptdev)); +panthor_sched_resume (ptdev); +drm_dev_exit (cookie); + } + + goto err_set_active; + } + +clk_disable_unprepare (ptdev->clks.coregroup); +clk_disable_unprepare (ptdev->clks.stacks); +clk_disable_unprepare (ptdev->clks.core); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_SUSPENDED ); +return 0 ; + +err_set_active: +/* If something failed and we have to revert back to an +* active state, we also need to clear the MMIO userspace +* mappings, so any dumb pages that were mapped while we +* were trying to suspend gets invalidated. +*/ +mutex_lock (&ptdev->pm.mmio_lock); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_ACTIVE ); +unmap_mapping_range (ptdev->base.anon_inode->i_mapping, +DRM_PANTHOR_USER_MMIO_OFFSET ,0 ,1 ); +mutex_unlock (&ptdev->pm.mmio_lock); +return ret; +} +#endif
drivers/gpu/drm/panthor/panthor_device.h new file +381 -0
381 lines not on this page · raw message
Signature · 1 line
2.43.0
Steven Price
maintainerRe: [PATCH v3 03/14] drm/panthor: Add the device logical block
Quoted 23 lines · Boris Brezillon
The panthor driver is designed in a modular way, where each logical block is dealing with a specific HW-block or software feature. In order for those blocks to communicate with each other, we need a central
panthor_devicecollecting all the blocks, and exposing some common features, like interrupt handling, power management, reset, ...This what this
panthor_devicelogical block is about.v3:
- Add acks for the MIT+GPL2 relicensing
- Fix 32-bit support
- Shorten the sections protected bypanthor_device::pm::mmio_lockto fix lock ordering issues.
- Renamepanthor_device::pm::lockintopanthor_device::pm::mmio_lockto better reflect what this lock is protecting
- Usedev_err_probe()
- Make sure we calldrm_dev_exit()when something fails half-way inpanthor_device_reset_work()
- ReplaceCSF_GPU_LATEST_FLUSH_ID_DEFAULTwith a constant '1' and a comment to explain. Also remove setting the dummy flush ID on suspend.
- Removedrm_WARN_ON()inpanthor_exception_name()
- Checkpirq->suspendedinpanthor_xxx_irq_raw_handler()
Quoted 5 lines
Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>
We still have the "FIXME: this is racy" and there's something wrong in panthor_device_reset_work() (see below). But otherwise looks good.
Quoted 12 lines
--- drivers/gpu/drm/panthor/panthor_device.c | 497 +++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_device.h | 381 +++++++++++++++++ 2 files changed, 878 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_device.c create mode 100644 drivers/gpu/drm/panthor/panthor_device.hdrivers/gpu/drm/panthor/panthor_device.c new file +0
-0
<snip>
Quoted 16 lines
drivers/gpu/drm/panthor/panthor_device.c new file +16
-0+ +static void panthor_device_reset_work (struct work_struct *work) +{ + struct panthor_device *ptdev = container_of(work, struct panthor_device, reset.work); +int ret =0 , cookie; + +if (atomic_read (&ptdev->pm.state) !=PANTHOR_DEVICE_PM_STATE_ACTIVE ) { +/* +* No need for a reset as the device has been (or will be) +* powered down +*/ +atomic_set (&ptdev->reset.pending,0 ); +return ; + } + +if (!drm_dev_enter (&ptdev->base, &cookie))
drivers/gpu/drm/panthor/panthor_device.c line 105
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_device.c new file +10
-0+return ; + +panthor_sched_pre_reset (ptdev); +panthor_fw_pre_reset (ptdev, true); +panthor_mmu_pre_reset (ptdev); +panthor_gpu_soft_reset (ptdev); +panthor_gpu_l2_power_on (ptdev); +panthor_mmu_post_reset (ptdev); + ret =panthor_fw_post_reset (ptdev); +if (ret)
If ret is true then we branch, so...
drivers/gpu/drm/panthor/panthor_device.c line 111
Quoted 6 lines
drivers/gpu/drm/panthor/panthor_device.c new file +6
-0+ goto out_dev_exit; + +atomic_set (&ptdev->reset.pending,0 ); +panthor_sched_post_reset (ptdev); + +if (ret) {
...this cannot ever be reached. I think the out_dev_exit label should be earlier (and renamed?)
drivers/gpu/drm/panthor/panthor_device.c line 118
Quoted 7 lines
drivers/gpu/drm/panthor/panthor_device.c new file +7
-0+panthor_device_unplug (ptdev); +drm_err (&ptdev->base,"Failed to boot MCU after reset, making device unusable." ); + } + +out_dev_exit: +drm_dev_exit (cookie); +}
Thanks,
Steve
Boris Brezillon
maintainerRe: [PATCH v3 03/14] drm/panthor: Add the device logical block
Quoted 24 lines · On Wed, 6 Dec 2023 16:55:42 +0000 Steven Price <[email protected]> wrote:
Quoted 23 lines · Boris Brezillon
The panthor driver is designed in a modular way, where each logical block is dealing with a specific HW-block or software feature. In order for those blocks to communicate with each other, we need a central
panthor_devicecollecting all the blocks, and exposing some common features, like interrupt handling, power management, reset, ...This what this
panthor_devicelogical block is about.v3:
- Add acks for the MIT+GPL2 relicensing
- Fix 32-bit support
- Shorten the sections protected bypanthor_device::pm::mmio_lockto fix lock ordering issues.
- Renamepanthor_device::pm::lockintopanthor_device::pm::mmio_lockto better reflect what this lock is protecting
- Usedev_err_probe()
- Make sure we calldrm_dev_exit()when something fails half-way inpanthor_device_reset_work()
- ReplaceCSF_GPU_LATEST_FLUSH_ID_DEFAULTwith a constant '1' and a comment to explain. Also remove setting the dummy flush ID on suspend.
- Removedrm_WARN_ON()inpanthor_exception_name()
- Checkpirq->suspendedinpanthor_xxx_irq_raw_handler()
Quoted 7 lines
Quoted 5 lines
Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>We still have the "FIXME: this is racy"
Yeah, I still didn't find a proper solution for that.
Quoted 47 lines
and there's something wrong in
panthor_device_reset_work()(see below). But otherwise looks good.Quoted 12 lines
--- drivers/gpu/drm/panthor/panthor_device.c | 497 +++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_device.h | 381 +++++++++++++++++ 2 files changed, 878 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_device.c create mode 100644 drivers/gpu/drm/panthor/panthor_device.hdrivers/gpu/drm/panthor/panthor_device.c new file +0
-0<snip>
Quoted 16 lines
drivers/gpu/drm/panthor/panthor_device.c new file +16
-0+ +static void panthor_device_reset_work (struct work_struct *work) +{ + struct panthor_device *ptdev = container_of(work, struct panthor_device, reset.work); +int ret =0 , cookie; + +if (atomic_read (&ptdev->pm.state) !=PANTHOR_DEVICE_PM_STATE_ACTIVE ) { +/* +* No need for a reset as the device has been (or will be) +* powered down +*/ +atomic_set (&ptdev->reset.pending,0 ); +return ; + } + +if (!drm_dev_enter (&ptdev->base, &cookie))drivers/gpu/drm/panthor/panthor_device.c line 105
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_device.c new file +10
-0+return ; + +panthor_sched_pre_reset (ptdev); +panthor_fw_pre_reset (ptdev, true); +panthor_mmu_pre_reset (ptdev); +panthor_gpu_soft_reset (ptdev); +panthor_gpu_l2_power_on (ptdev); +panthor_mmu_post_reset (ptdev); + ret =panthor_fw_post_reset (ptdev); +if (ret)If ret is true then we branch, so...
Quoted 3 lines
drivers/gpu/drm/panthor/panthor_device.c line 108
Quoted 3 lines
drivers/gpu/drm/panthor/panthor_device.c new file +3
-0+ goto out_dev_exit; + +atomic_set (&ptdev->reset.pending,0 );
I think we want to clear the reset pending bit even if the post reset failed.
Quoted 6 lines
drivers/gpu/drm/panthor/panthor_device.c line 111
Quoted 3 lines
drivers/gpu/drm/panthor/panthor_device.c new file +3
-0+panthor_sched_post_reset (ptdev); + +if (ret) {...this cannot ever be reached. I think the
out_dev_exitlabel should be earlier (and renamed?)
Uh, actually we can't call panthor_device_unplug() when we're in the drm_dev_enter/exit() section, this would cause a deadlock. This should be moved at the end of the function, but I can't remember if I had another good reason to move it here...
Quoted 12 lines
drivers/gpu/drm/panthor/panthor_device.c line 118
Quoted 8 lines
drivers/gpu/drm/panthor/panthor_device.c new file +7
-0+panthor_device_unplug (ptdev); +drm_err (&ptdev->base,"Failed to boot MCU after reset, making device unusable." ); + } + +out_dev_exit: +drm_dev_exit (cookie); +}Thanks,
Steve
Boris Brezillon
maintainerRe: [PATCH v3 03/14] drm/panthor: Add the device logical block
Quoted 33 lines · On Thu, 7 Dec 2023 09:12:43 +0100 Boris Brezillon…
Quoted 30 lines · On Wed, 6 Dec 2023 16:55:42 +0000 Steven Price <[email protected]> wrote:
Quoted 28 lines · Boris Brezillon
The panthor driver is designed in a modular way, where each logical block is dealing with a specific HW-block or software feature. In order for those blocks to communicate with each other, we need a central
panthor_devicecollecting all the blocks, and exposing some common features, like interrupt handling, power management, reset, ...This what this
panthor_devicelogical block is about.v3:
- Add acks for the MIT+GPL2 relicensing
- Fix 32-bit support
- Shorten the sections protected bypanthor_device::pm::mmio_lockto fix lock ordering issues.
- Renamepanthor_device::pm::lockintopanthor_device::pm::mmio_lockto better reflect what this lock is protecting
- Usedev_err_probe()
- Make sure we calldrm_dev_exit()when something fails half-way inpanthor_device_reset_work()
- ReplaceCSF_GPU_LATEST_FLUSH_ID_DEFAULTwith a constant '1' and a comment to explain. Also remove setting the dummy flush ID on suspend.
- Removedrm_WARN_ON()inpanthor_exception_name()
- Checkpirq->suspendedinpanthor_xxx_irq_raw_handler()Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>
Quoted 3 lines
Quoted 1 line
We still have the "FIXME: this is racy"
Yeah, I still didn't find a proper solution for that.
This [1] should fix the race condition in the unplug path.
[1]https://gitlab.freedesktop.org/panfrost/linux/-/commit/b79b28069e524ae7fea22a9a158b870eab2d5f9a
Steven Price
maintainerRe: [PATCH v3 03/14] drm/panthor: Add the device logical block
Quoted 36 lines · Boris Brezillon
Quoted 33 lines · On Thu, 7 Dec 2023 09:12:43 +0100 Boris Brezillon…
Quoted 30 lines · On Wed, 6 Dec 2023 16:55:42 +0000 Steven Price <[email protected]> wrote:
Quoted 28 lines · Boris Brezillon
The panthor driver is designed in a modular way, where each logical block is dealing with a specific HW-block or software feature. In order for those blocks to communicate with each other, we need a central
panthor_devicecollecting all the blocks, and exposing some common features, like interrupt handling, power management, reset, ...This what this
panthor_devicelogical block is about.v3:
- Add acks for the MIT+GPL2 relicensing
- Fix 32-bit support
- Shorten the sections protected bypanthor_device::pm::mmio_lockto fix lock ordering issues.
- Renamepanthor_device::pm::lockintopanthor_device::pm::mmio_lockto better reflect what this lock is protecting
- Usedev_err_probe()
- Make sure we calldrm_dev_exit()when something fails half-way inpanthor_device_reset_work()
- ReplaceCSF_GPU_LATEST_FLUSH_ID_DEFAULTwith a constant '1' and a comment to explain. Also remove setting the dummy flush ID on suspend.
- Removedrm_WARN_ON()inpanthor_exception_name()
- Checkpirq->suspendedinpanthor_xxx_irq_raw_handler()Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
We still have the "FIXME: this is racy"
Yeah, I still didn't find a proper solution for that.
This [1] should fix the race condition in the unplug path.
[1]https://gitlab.freedesktop.org/panfrost/linux/-/commit/b79b28069e524ae7fea22a9a158b870eab2d5f9a
Looks like it should do the job. I'm surprised that we're the only ones to face this though.
Looking at the new imagination driver it appears there's a similar problem:
pvr_device_lost() uses a boolean 'lost' to track multiple calls but that boolean isn't protected by any specific lock (AFAICT). Indeed pvr_device_lost() calls drm_dev_unplug() while in a drm_dev_enter() critical section (see pvr_mmu_flush_exec()). If I'm not mistaken that's the same problem we discussed and isn't allowed? drm_dev_unplug() will synchronise on the SRCU that drm_dev_enter() is holding.
+CC: Frank, Donald, Matt from Imagination.
Steve
Boris Brezillon
maintainerRe: [PATCH v3 03/14] drm/panthor: Add the device logical block
Quoted 39 lines · On Thu, 7 Dec 2023 10:23:55 +0000 Steven Price <[email protected]> wrote:
Quoted 38 lines · Boris Brezillon
Quoted 35 lines · On Thu, 7 Dec 2023 09:12:43 +0100 Boris Brezillon…
Quoted 31 lines · On Wed, 6 Dec 2023 16:55:42 +0000 Steven Price <[email protected]> wrote:
Quoted 23 lines · Boris Brezillon
The panthor driver is designed in a modular way, where each logical block is dealing with a specific HW-block or software feature. In order for those blocks to communicate with each other, we need a central
panthor_devicecollecting all the blocks, and exposing some common features, like interrupt handling, power management, reset, ...This what this
panthor_devicelogical block is about.v3:
- Add acks for the MIT+GPL2 relicensing
- Fix 32-bit support
- Shorten the sections protected bypanthor_device::pm::mmio_lockto fix lock ordering issues.
- Renamepanthor_device::pm::lockintopanthor_device::pm::mmio_lockto better reflect what this lock is protecting
- Usedev_err_probe()
- Make sure we calldrm_dev_exit()when something fails half-way inpanthor_device_reset_work()
- ReplaceCSF_GPU_LATEST_FLUSH_ID_DEFAULTwith a constant '1' and a comment to explain. Also remove setting the dummy flush ID on suspend.
- Removedrm_WARN_ON()inpanthor_exception_name()
- Checkpirq->suspendedinpanthor_xxx_irq_raw_handler()Quoted 5 lines
Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>We still have the "FIXME: this is racy"
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Yeah, I still didn't find a proper solution for that.
This [1] should fix the race condition in the unplug path.
[1]https://gitlab.freedesktop.org/panfrost/linux/-/commit/b79b28069e524ae7fea22a9a158b870eab2d5f9a
Looks like it should do the job. I'm surprised that we're the only ones to face this though.
Most drivers just have one path where they call drm_dev_unplug():
the device removal callback, which is only called once per device. The only exception where I see more than one occurrence are the amdgpu and powervr drivers. I guess amdgpu has some tricks to serialize _unplug operations, and powervr is probably buggy as you pointed out.
Quoted 8 lines
Looking at the new imagination driver it appears there's a similar problem:
pvr_device_lost()uses a boolean 'lost' to track multiple calls but that boolean isn't protected by any specific lock (AFAICT). Indeedpvr_device_lost()callsdrm_dev_unplug()while in adrm_dev_enter()critical section (seepvr_mmu_flush_exec()). If I'm not mistaken that's the same problem we discussed and isn't allowed?
It is, indeed. That means there's a deadlock when pvr_device_lost() is called from the MMU code. And I guess the race condition with concurrent pvr_device_lost() callers exists too (unless I missed something, and calls to pvr_device_lost() are serialized with another lock).
Quoted 7 lines
drm_dev_unplug()will
synchronise on the SRCU thatdrm_dev_enter()is holding.+CC: Frank, Donald, Matt from Imagination.
Steve
Re: [EXTERNAL] Re: [PATCH v3 03/14] drm/panthor: Add the device logical block
In reply to Steven PriceThanks for the bug report!
Donald
Quoted 63 lines · Steven Price
*** CAUTION: This email originates from a source not known to Imagination Technologies. Think before you click a link or open an attachment ***
Quoted 36 lines · Boris Brezillon
Quoted 33 lines · On Thu, 7 Dec 2023 09:12:43 +0100 Boris Brezillon…
Quoted 30 lines · On Wed, 6 Dec 2023 16:55:42 +0000 Steven Price <[email protected]> wrote:
Quoted 28 lines · Boris Brezillon
The panthor driver is designed in a modular way, where each logical block is dealing with a specific HW-block or software feature. In order for those blocks to communicate with each other, we need a central
panthor_devicecollecting all the blocks, and exposing some common features, like interrupt handling, power management, reset, ...This what this
panthor_devicelogical block is about.v3:
- Add acks for the MIT+GPL2 relicensing
- Fix 32-bit support
- Shorten the sections protected bypanthor_device::pm::mmio_lockto fix lock ordering issues.
- Renamepanthor_device::pm::lockintopanthor_device::pm::mmio_lockto better reflect what this lock is protecting
- Usedev_err_probe()
- Make sure we calldrm_dev_exit()when something fails half-way inpanthor_device_reset_work()
- ReplaceCSF_GPU_LATEST_FLUSH_ID_DEFAULTwith a constant '1' and a comment to explain. Also remove setting the dummy flush ID on suspend.
- Removedrm_WARN_ON()inpanthor_exception_name()
- Checkpirq->suspendedinpanthor_xxx_irq_raw_handler()Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>Quoted 7 lines
Quoted 3 lines
Quoted 1 line
We still have the "FIXME: this is racy"
Yeah, I still didn't find a proper solution for that.
This [1] should fix the race condition in the unplug path.
Looks like it should do the job. I'm surprised that we're the only ones to face this though.
Looking at the new imagination driver it appears there's a similar problem:
pvr_device_lost()uses a boolean 'lost' to track multiple calls but that boolean isn't protected by any specific lock (AFAICT). Indeedpvr_device_lost()callsdrm_dev_unplug()while in adrm_dev_enter()critical section (seepvr_mmu_flush_exec()). If I'm not mistaken that's the same problem we discussed and isn't allowed?drm_dev_unplug()will synchronise on the SRCU thatdrm_dev_enter()is holding.+CC: Frank, Donald, Matt from Imagination.
Steve
Liviu Dudau
maintainerRe: [PATCH v3 03/14] drm/panthor: Add the device logical block
In reply to Boris BrezillonHi Boris,
Quoted 244 lines · Boris Brezillon
The panthor driver is designed in a modular way, where each logical block is dealing with a specific HW-block or software feature. In order for those blocks to communicate with each other, we need a central
panthor_devicecollecting all the blocks, and exposing some common features, like interrupt handling, power management, reset, ...This what this
panthor_devicelogical block is about.v3:
- Add acks for the MIT+GPL2 relicensing
- Fix 32-bit support
- Shorten the sections protected bypanthor_device::pm::mmio_lockto fix lock ordering issues.
- Renamepanthor_device::pm::lockintopanthor_device::pm::mmio_lockto better reflect what this lock is protecting
- Usedev_err_probe()
- Make sure we calldrm_dev_exit()when something fails half-way inpanthor_device_reset_work()
- ReplaceCSF_GPU_LATEST_FLUSH_ID_DEFAULTwith a constant '1' and a comment to explain. Also remove setting the dummy flush ID on suspend.
- Removedrm_WARN_ON()inpanthor_exception_name()
- Checkpirq->suspendedinpanthor_xxx_irq_raw_handler()Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>--- drivers/gpu/drm/panthor/panthor_device.c | 497 +++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_device.h | 381 +++++++++++++++++ 2 files changed, 878 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_device.c create mode 100644 drivers/gpu/drm/panthor/panthor_device.hdrivers/gpu/drm/panthor/panthor_device.c new file +203
-0@@ -0,0 +1,497 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2018 Marty E. Plummer <[email protected]> */ +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ + +#include <linux/clk.h> +#include <linux/reset.h> +#include <linux/platform_device.h> +#include <linux/pm_domain.h> +#include <linux/pm_runtime.h> +#include <linux/regulator/consumer.h> + +#include <drm/drm_drv.h> +#include <drm/drm_managed.h> + +#include "panthor_sched.h" +#include "panthor_device.h" +#include "panthor_devfreq.h" +#include "panthor_gpu.h" +#include "panthor_fw.h" +#include "panthor_mmu.h" +#include "panthor_regs.h" + +static int panthor_clk_init (struct panthor_device *ptdev) +{ + ptdev->clks.core =devm_clk_get (ptdev->base.dev,NULL ); +if (IS_ERR (ptdev->clks.core)) +return dev_err_probe (ptdev->base.dev, +PTR_ERR (ptdev->clks.core), +"get 'core' clock failed" ); + + ptdev->clks.stacks =devm_clk_get_optional (ptdev->base.dev,"stacks" ); +if (IS_ERR (ptdev->clks.stacks)) +return dev_err_probe (ptdev->base.dev, +PTR_ERR (ptdev->clks.stacks), +"get 'stacks' clock failed" ); + + ptdev->clks.coregroup =devm_clk_get_optional (ptdev->base.dev,"coregroup" ); +if (IS_ERR (ptdev->clks.coregroup)) +return dev_err_probe (ptdev->base.dev, +PTR_ERR (ptdev->clks.coregroup), +"get 'coregroup' clock failed" ); + +drm_info (&ptdev->base,"clock rate = %lu\n" ,clk_get_rate (ptdev->clks.core)); +return 0 ; +} + +void panthor_device_unplug (struct panthor_device *ptdev) +{ +/* FIXME: This is racy. */ +if (drm_dev_is_unplugged (&ptdev->base)) +return ; + +drm_WARN_ON (&ptdev->base,pm_runtime_get_sync (ptdev->base.dev) <0 ); + +/* Call drm_dev_unplug() so any access to HW block happening after +* that point get rejected. +*/ +drm_dev_unplug (&ptdev->base); + +/* Now, try to cleanly shutdown the GPU before the device resources +* get reclaimed. +*/ +panthor_sched_unplug (ptdev); +panthor_fw_unplug (ptdev); +panthor_mmu_unplug (ptdev); +panthor_gpu_unplug (ptdev); + +pm_runtime_dont_use_autosuspend (ptdev->base.dev); +pm_runtime_put_sync_suspend (ptdev->base.dev); +} + +static void panthor_device_reset_cleanup (struct drm_device *ddev,void *data) +{ + struct panthor_device *ptdev = container_of(ddev, struct panthor_device, base); + +cancel_work_sync (&ptdev->reset.work); +destroy_workqueue (ptdev->reset.wq); +} + +static void panthor_device_reset_work (struct work_struct *work) +{ + struct panthor_device *ptdev = container_of(work, struct panthor_device, reset.work); +int ret =0 , cookie; + +if (atomic_read (&ptdev->pm.state) !=PANTHOR_DEVICE_PM_STATE_ACTIVE ) { +/* +* No need for a reset as the device has been (or will be) +* powered down +*/ +atomic_set (&ptdev->reset.pending,0 ); +return ; + } + +if (!drm_dev_enter (&ptdev->base, &cookie)) +return ; + +panthor_sched_pre_reset (ptdev); +panthor_fw_pre_reset (ptdev, true); +panthor_mmu_pre_reset (ptdev); +panthor_gpu_soft_reset (ptdev); +panthor_gpu_l2_power_on (ptdev); +panthor_mmu_post_reset (ptdev); + ret =panthor_fw_post_reset (ptdev); +if (ret) + goto out_dev_exit; + +atomic_set (&ptdev->reset.pending,0 ); +panthor_sched_post_reset (ptdev); + +if (ret) { +panthor_device_unplug (ptdev); +drm_err (&ptdev->base,"Failed to boot MCU after reset, making device unusable." ); + } + +out_dev_exit: +drm_dev_exit (cookie); +} + +static bool panthor_device_is_initialized (struct panthor_device *ptdev) +{ +return !!ptdev->scheduler; +} + +static void panthor_device_free_page (struct drm_device *ddev,void *data) +{ +free_page ((unsigned long )data); +} + +int panthor_device_init (struct panthor_device *ptdev) +{ +struct resource *res; +struct page *p; +int ret; + + ptdev->coherent =device_get_dma_attr (ptdev->base.dev) ==DEV_DMA_COHERENT ; + +drmm_mutex_init (&ptdev->base, &ptdev->pm.mmio_lock); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_SUSPENDED ); + p =alloc_page (GFP_KERNEL | __GFP_ZERO); +if (!p) +return -ENOMEM ; + + ptdev->pm.dummy_latest_flush =page_address (p); + ret =drmm_add_action_or_reset (&ptdev->base, panthor_device_free_page, + ptdev->pm.dummy_latest_flush); +if (ret) +return ret; + +/* +* Set the dummy page holding the latest flush to 1. This will cause the +* flush to avoided as we know it isn't necessary if the submission +* happens while the dummy page is mapped. Zero cannot be used because +* that means 'always flush'. +*/ + *ptdev->pm.dummy_latest_flush =1 ; + +INIT_WORK (&ptdev->reset.work, panthor_device_reset_work); + ptdev->reset.wq =alloc_ordered_workqueue ("panthor-reset-wq" ,0 ); +if (!ptdev->reset.wq) +return -ENOMEM ; + + ret =drmm_add_action_or_reset (&ptdev->base, panthor_device_reset_cleanup,NULL ); +if (ret) +return ret; + + ret =panthor_clk_init (ptdev); +if (ret) +return ret; + + ret =panthor_devfreq_init (ptdev); +if (ret) +return ret; + + ptdev->iomem =devm_platform_get_and_ioremap_resource (to_platform_device (ptdev->base.dev), +0 , &res); +if (IS_ERR (ptdev->iomem)) +return PTR_ERR (ptdev->iomem); + + ptdev->phys_addr = res->start; + + ret =devm_pm_runtime_enable (ptdev->base.dev); +if (ret) +return ret; + + ret =pm_runtime_resume_and_get (ptdev->base.dev); +if (ret) +return ret; + + ret =panthor_gpu_init (ptdev); +if (ret) + goto err_rpm_put; + + ret =panthor_mmu_init (ptdev); +if (ret) + goto err_rpm_put; + + ret =panthor_fw_init (ptdev); +if (ret) + goto err_rpm_put; + + ret =panthor_sched_init (ptdev); +if (ret)
drivers/gpu/drm/panthor/panthor_device.c line 213
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_device.c new file +10
-0+ goto err_rpm_put; + +/* ~3 frames */ +pm_runtime_set_autosuspend_delay (ptdev->base.dev,50 ); +pm_runtime_use_autosuspend (ptdev->base.dev); +pm_runtime_put_autosuspend (ptdev->base.dev); +return 0 ; + +err_rpm_put: +pm_runtime_put_sync_suspend (ptdev->base.dev);
I'm afraid this is not enough to cleanup the driver if firmware fails to load or if panthor_sched_init() fails. I was playing with another device that's based on RK3588 and forgot to move the firmware to the new location. I've got a kernel crash with an SError due to failure to load the firmware. It was in panthor_mmu_irq_raw_handler() which fired because the MMU was not unplugged.
My local fix was:
-- >8 --------------------------------------------------------------
drivers/gpu/drm/panthor/panthor_device.c +9 -3
@@ -224,15 +224,15 @@ int panthor_device_init(struct panthor_device *ptdev) ret =panthor_mmu_init (ptdev);if (ret) - goto err_rpm_put; + goto err_mmu_init; ret =panthor_fw_init (ptdev);if (ret) - goto err_rpm_put; + goto err_fw_init; ret =panthor_sched_init (ptdev);if (ret) - goto err_rpm_put; + goto err_sched_init;/* ~3 frames */ pm_runtime_set_autosuspend_delay (ptdev->base.dev,50 );
@@ -240,6 +240,12 @@ int panthor_device_init(struct panthor_device *ptdev)pm_runtime_put_autosuspend (ptdev->base.dev);return 0 ; +err_sched_init: +panthor_fw_unplug (ptdev); +err_fw_init: +panthor_mmu_unplug (ptdev); +err_mmu_init: +panthor_gpu_unplug (ptdev); err_rpm_put:pm_runtime_put_sync_suspend (ptdev->base.dev);return ret;
-- >8 -------------------------------------------------------------
I can send a fully formed patch if you agree with the fix.
Best regards,
Liviu
Quoted 674 lines
drivers/gpu/drm/panthor/panthor_device.c new file +284
-0+return ret; +} + +#define PANTHOR_EXCEPTION (id) \ + [DRM_PANTHOR_EXCEPTION_ ## id] = { \ + .name = #id, \ + } + +struct panthor_exception_info { +const char *name; +}; + +static const struct panthor_exception_info panthor_exception_infos[] = { +PANTHOR_EXCEPTION (OK ), +PANTHOR_EXCEPTION (TERMINATED ), +PANTHOR_EXCEPTION (KABOOM ), +PANTHOR_EXCEPTION (EUREKA ), +PANTHOR_EXCEPTION (ACTIVE ), +PANTHOR_EXCEPTION (CS_RES_TERM ), +PANTHOR_EXCEPTION (CS_CONFIG_FAULT ), +PANTHOR_EXCEPTION (CS_ENDPOINT_FAULT ), +PANTHOR_EXCEPTION (CS_BUS_FAULT ), +PANTHOR_EXCEPTION (CS_INSTR_INVALID ), +PANTHOR_EXCEPTION (CS_CALL_STACK_OVERFLOW ), +PANTHOR_EXCEPTION (CS_INHERIT_FAULT ), +PANTHOR_EXCEPTION (INSTR_INVALID_PC ), +PANTHOR_EXCEPTION (INSTR_INVALID_ENC ), +PANTHOR_EXCEPTION (INSTR_BARRIER_FAULT ), +PANTHOR_EXCEPTION (DATA_INVALID_FAULT ), +PANTHOR_EXCEPTION (TILE_RANGE_FAULT ), +PANTHOR_EXCEPTION (ADDR_RANGE_FAULT ), +PANTHOR_EXCEPTION (IMPRECISE_FAULT ), +PANTHOR_EXCEPTION (OOM ), +PANTHOR_EXCEPTION (CSF_FW_INTERNAL_ERROR ), +PANTHOR_EXCEPTION (CSF_RES_EVICTION_TIMEOUT ), +PANTHOR_EXCEPTION (GPU_BUS_FAULT ), +PANTHOR_EXCEPTION (GPU_SHAREABILITY_FAULT ), +PANTHOR_EXCEPTION (SYS_SHAREABILITY_FAULT ), +PANTHOR_EXCEPTION (GPU_CACHEABILITY_FAULT ), +PANTHOR_EXCEPTION (TRANSLATION_FAULT_0 ), +PANTHOR_EXCEPTION (TRANSLATION_FAULT_1 ), +PANTHOR_EXCEPTION (TRANSLATION_FAULT_2 ), +PANTHOR_EXCEPTION (TRANSLATION_FAULT_3 ), +PANTHOR_EXCEPTION (TRANSLATION_FAULT_4 ), +PANTHOR_EXCEPTION (PERM_FAULT_0 ), +PANTHOR_EXCEPTION (PERM_FAULT_1 ), +PANTHOR_EXCEPTION (PERM_FAULT_2 ), +PANTHOR_EXCEPTION (PERM_FAULT_3 ), +PANTHOR_EXCEPTION (ACCESS_FLAG_1 ), +PANTHOR_EXCEPTION (ACCESS_FLAG_2 ), +PANTHOR_EXCEPTION (ACCESS_FLAG_3 ), +PANTHOR_EXCEPTION (ADDR_SIZE_FAULT_IN ), +PANTHOR_EXCEPTION (ADDR_SIZE_FAULT_OUT0 ), +PANTHOR_EXCEPTION (ADDR_SIZE_FAULT_OUT1 ), +PANTHOR_EXCEPTION (ADDR_SIZE_FAULT_OUT2 ), +PANTHOR_EXCEPTION (ADDR_SIZE_FAULT_OUT3 ), +PANTHOR_EXCEPTION (MEM_ATTR_FAULT_0 ), +PANTHOR_EXCEPTION (MEM_ATTR_FAULT_1 ), +PANTHOR_EXCEPTION (MEM_ATTR_FAULT_2 ), +PANTHOR_EXCEPTION (MEM_ATTR_FAULT_3 ), +}; + +const char *panthor_exception_name (struct panthor_device *ptdev,u32 exception_code) +{ +if (exception_code >=ARRAY_SIZE (panthor_exception_infos) || + !panthor_exception_infos[exception_code].name) +return "Unknown exception type" ; + +return panthor_exception_infos[exception_code].name; +} + +static vm_fault_t panthor_mmio_vm_fault (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct panthor_device *ptdev = vma->vm_private_data; +u64 id = (u64 )vma->vm_pgoff <<PAGE_SHIFT ; +unsigned long pfn; +pgprot_t pgprot; +vm_fault_t ret; +bool active; +int cookie; + +if (!drm_dev_enter (&ptdev->base, &cookie)) +return VM_FAULT_SIGBUS ; + +mutex_lock (&ptdev->pm.mmio_lock); + active =atomic_read (&ptdev->pm.state) ==PANTHOR_DEVICE_PM_STATE_ACTIVE ; + +switch (panthor_device_mmio_offset (id)) { +case DRM_PANTHOR_USER_FLUSH_ID_MMIO_OFFSET : +if (active) + pfn =__phys_to_pfn (ptdev->phys_addr +CSF_GPU_LATEST_FLUSH_ID ); +else + pfn =virt_to_pfn (ptdev->pm.dummy_latest_flush); +break ; + +default : + ret =VM_FAULT_SIGBUS ; + goto out_unlock; + } + + pgprot = vma->vm_page_prot; +if (active) + pgprot =pgprot_noncached (pgprot); + + ret =vmf_insert_pfn_prot (vma, vmf->address, pfn, pgprot); + +out_unlock: +mutex_unlock (&ptdev->pm.mmio_lock); +drm_dev_exit (cookie); +return ret; +} + +static const struct vm_operations_struct panthor_mmio_vm_ops = { + .fault = panthor_mmio_vm_fault, +}; + +int panthor_device_mmap_io (struct panthor_device *ptdev,struct vm_area_struct *vma) +{ +u64 id = (u64 )vma->vm_pgoff <<PAGE_SHIFT ; + +switch (panthor_device_mmio_offset (id)) { +case DRM_PANTHOR_USER_FLUSH_ID_MMIO_OFFSET : +if (vma->vm_end - vma->vm_start !=PAGE_SIZE || + (vma->vm_flags & (VM_WRITE |VM_EXEC ))) +return -EINVAL ; + +break ; + +default : +return -EINVAL ; + } + +/* Defer actual mapping to the fault handler. */ + vma->vm_private_data = ptdev; + vma->vm_ops = &panthor_mmio_vm_ops; +vm_flags_set (vma, +VM_IO |VM_DONTCOPY |VM_DONTEXPAND | +VM_NORESERVE |VM_DONTDUMP |VM_PFNMAP ); +return 0 ; +} + +#ifdef CONFIG_PM +int panthor_device_resume (struct device *dev) +{ +struct panthor_device *ptdev =dev_get_drvdata (dev); +int ret, cookie; + +if (atomic_read (&ptdev->pm.state) !=PANTHOR_DEVICE_PM_STATE_SUSPENDED ) +return -EINVAL ; + +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_RESUMING ); + + ret =clk_prepare_enable (ptdev->clks.core); +if (ret) + goto err_set_suspended; + + ret =clk_prepare_enable (ptdev->clks.stacks); +if (ret) + goto err_disable_core_clk; + + ret =clk_prepare_enable (ptdev->clks.coregroup); +if (ret) + goto err_disable_stacks_clk; + + ret =panthor_devfreq_resume (ptdev); +if (ret) + goto err_disable_coregroup_clk; + +if (panthor_device_is_initialized (ptdev) && +drm_dev_enter (&ptdev->base, &cookie)) { +panthor_gpu_resume (ptdev); +panthor_mmu_resume (ptdev); + ret =drm_WARN_ON (&ptdev->base,panthor_fw_resume (ptdev)); +if (!ret) +panthor_sched_resume (ptdev); + +drm_dev_exit (cookie); + +if (ret) + goto err_devfreq_suspend; + } + +if (atomic_read (&ptdev->reset.pending)) +queue_work (ptdev->reset.wq, &ptdev->reset.work); + +/* Clear all IOMEM mappings pointing to this device after we've +* resumed. This way the fake mappings pointing to the dummy pages +* are removed and the real iomem mapping will be restored on next +* access. +*/ +mutex_lock (&ptdev->pm.mmio_lock); +unmap_mapping_range (ptdev->base.anon_inode->i_mapping, +DRM_PANTHOR_USER_MMIO_OFFSET ,0 ,1 ); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_ACTIVE ); +mutex_unlock (&ptdev->pm.mmio_lock); +return 0 ; + +err_devfreq_suspend: +panthor_devfreq_suspend (ptdev); + +err_disable_coregroup_clk: +clk_disable_unprepare (ptdev->clks.coregroup); + +err_disable_stacks_clk: +clk_disable_unprepare (ptdev->clks.stacks); + +err_disable_core_clk: +clk_disable_unprepare (ptdev->clks.core); + +err_set_suspended: +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_SUSPENDED ); +return ret; +} + +int panthor_device_suspend (struct device *dev) +{ +struct panthor_device *ptdev =dev_get_drvdata (dev); +int ret, cookie; + +if (atomic_read (&ptdev->pm.state) !=PANTHOR_DEVICE_PM_STATE_ACTIVE ) +return -EINVAL ; + +/* Clear all IOMEM mappings pointing to this device before we +* shutdown the power-domain and clocks. Failing to do that results +* in external aborts when the process accesses the iomem region. +* We change the state and call unmap_mapping_range() with the +* mmio_lock held to make sure the vm_fault handler won't set up +* invalid mappings. +*/ +mutex_lock (&ptdev->pm.mmio_lock); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_SUSPENDING ); +unmap_mapping_range (ptdev->base.anon_inode->i_mapping, +DRM_PANTHOR_USER_MMIO_OFFSET ,0 ,1 ); +mutex_unlock (&ptdev->pm.mmio_lock); + +if (panthor_device_is_initialized (ptdev) && +drm_dev_enter (&ptdev->base, &cookie)) { +cancel_work_sync (&ptdev->reset.work); + +/* We prepare everything as if we were resetting the GPU. +* The end of the reset will happen in the resume path though. +*/ +panthor_sched_suspend (ptdev); +panthor_fw_suspend (ptdev); +panthor_mmu_suspend (ptdev); +panthor_gpu_suspend (ptdev); +drm_dev_exit (cookie); + } + + ret =panthor_devfreq_suspend (ptdev); +if (ret) { +if (panthor_device_is_initialized (ptdev) && +drm_dev_enter (&ptdev->base, &cookie)) { +panthor_gpu_resume (ptdev); +panthor_mmu_resume (ptdev); +drm_WARN_ON (&ptdev->base,panthor_fw_resume (ptdev)); +panthor_sched_resume (ptdev); +drm_dev_exit (cookie); + } + + goto err_set_active; + } + +clk_disable_unprepare (ptdev->clks.coregroup); +clk_disable_unprepare (ptdev->clks.stacks); +clk_disable_unprepare (ptdev->clks.core); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_SUSPENDED ); +return 0 ; + +err_set_active: +/* If something failed and we have to revert back to an +* active state, we also need to clear the MMIO userspace +* mappings, so any dumb pages that were mapped while we +* were trying to suspend gets invalidated. +*/ +mutex_lock (&ptdev->pm.mmio_lock); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_ACTIVE ); +unmap_mapping_range (ptdev->base.anon_inode->i_mapping, +DRM_PANTHOR_USER_MMIO_OFFSET ,0 ,1 ); +mutex_unlock (&ptdev->pm.mmio_lock); +return ret; +} +#endif drivers/gpu/drm/panthor/panthor_device.h new file +381
-0@@ -0,0 +1,381 @@ +/* SPDX-License-Identifier: GPL-2.0 or MIT */ +/* Copyright 2018 Marty E. Plummer <[email protected]> */ +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ + +#ifndef __PANTHOR_DEVICE_H__ +#define __PANTHOR_DEVICE_H__ + +#include <linux/atomic.h> +#include <linux/io-pgtable.h> +#include <linux/regulator/consumer.h> +#include <linux/sched.h> +#include <linux/spinlock.h> +#include <drm/drm_device.h> +#include <drm/drm_mm.h> +#include <drm/gpu_scheduler.h> +#include <drm/panthor_drm.h> + +struct panthor_csf ; +struct panthor_csf_ctx ; +struct panthor_device ; +struct panthor_gpu ; +struct panthor_group_pool ; +struct panthor_heap_pool ; +struct panthor_job ; +struct panthor_mmu ; +struct panthor_fw ; +struct panthor_perfcnt ; +struct panthor_vm ; +struct panthor_vm_pool ; + +/** +* enum panthor_device_pm_state - PM state +*/ +enum panthor_device_pm_state { +/** @PANTHOR_DEVICE_PM_STATE_SUSPENDED: Device is suspended. */ +PANTHOR_DEVICE_PM_STATE_SUSPENDED =0 , + +/** @PANTHOR_DEVICE_PM_STATE_RESUMING: Device is being resumed. */ +PANTHOR_DEVICE_PM_STATE_RESUMING , + +/** @PANTHOR_DEVICE_PM_STATE_ACTIVE: Device is active. */ +PANTHOR_DEVICE_PM_STATE_ACTIVE , + +/** @PANTHOR_DEVICE_PM_STATE_SUSPENDING: Device is being suspended. */ +PANTHOR_DEVICE_PM_STATE_SUSPENDING , +}; + +/** +* struct panthor_irq - IRQ data +* +* Used to automate IRQ handling for the 3 different IRQs we have in this driver. +*/ +struct panthor_irq { +/** @ptdev: Panthor device */ +struct panthor_device *ptdev; + +/** @irq: IRQ number. */ +int irq; + +/** @mask: Current mask being applied to xxx_INT_MASK. */ +u32 mask; + +/** @suspended: Set to true when the IRQ is suspended. */ +atomic_t suspended; +}; + +/** +* struct panthor_device - Panthor device +*/ +struct panthor_device { +/** @base: Base drm_device. */ +struct drm_device base; + +/** @phys_addr: Physical address of the iomem region. */ +phys_addr_t phys_addr; + +/** @iomem: CPU mapping of the IOMEM region. */ +void __iomem *iomem; + +/** @clks: GPU clocks. */ +struct { +/** @core: Core clock. */ +struct clk *core; + +/** @stacks: Stacks clock. This clock is optional. */ +struct clk *stacks; + +/** @coregroup: Core group clock. This clock is optional. */ +struct clk *coregroup; + } clks; + +/** @coherent: True if the CPU/GPU are memory coherent. */ +bool coherent; + +/** @gpu_info: GPU information. */ +struct drm_panthor_gpu_info gpu_info; + +/** @csif_info: Command stream interface information. */ +struct drm_panthor_csif_info csif_info; + +/** @gpu: GPU management data. */ +struct panthor_gpu *gpu; + +/** @fw: FW management data. */ +struct panthor_fw *fw; + +/** @mmu: MMU management data. */ +struct panthor_mmu *mmu; + +/** @scheduler: Scheduler management data. */ +struct panthor_scheduler *scheduler; + +/** @devfreq: Device frequency scaling management data. */ +struct panthor_devfreq *devfreq; + +/** @reset: Reset related fields. */ +struct { +/** @wq: Ordered worqueud used to schedule reset operations. */ +struct workqueue_struct *wq; + +/** @work: Reset work. */ +struct work_struct work; + +/** @pending: Set to true if a reset is pending. */ +atomic_t pending; + } reset; + +/** @pm: Power management related data. */ +struct { +/** @state: Power state. */ +atomic_t state; + +/** +* @mmio_lock: Lock protecting MMIO userspace CPU mappings. +* +* This is needed to ensure we map the dummy IO pages when +* the device is being suspended, and the real IO pages when +* the device is being resumed. We can't just do with the +* state atomicity to deal with this race. +*/ +struct mutex mmio_lock; + +/** +* @dummy_latest_flush: Dummy LATEST_FLUSH page. +* +* Used to replace the real LATEST_FLUSH page when the GPU +* is suspended. +*/ +u32 *dummy_latest_flush; + } pm; +}; + +/** +* struct panthor_file - Panthor file +*/ +struct panthor_file { +/** @ptdev: Device attached to this file. */ +struct panthor_device *ptdev; + +/** @vms: VM pool attached to this file. */ +struct panthor_vm_pool *vms; + +/** @groups: Scheduling group pool attached to this file. */ +struct panthor_group_pool *groups; +}; + +int panthor_device_init (struct panthor_device *ptdev); +void panthor_device_unplug (struct panthor_device *ptdev); + +/** +* panthor_device_schedule_reset() - Schedules a reset operation +*/ +static inline void panthor_device_schedule_reset (struct panthor_device *ptdev) +{ +if (!atomic_cmpxchg (&ptdev->reset.pending,0 ,1 ) && +atomic_read (&ptdev->pm.state) ==PANTHOR_DEVICE_PM_STATE_ACTIVE ) +queue_work (ptdev->reset.wq, &ptdev->reset.work); +} + +/** +* panthor_device_reset_is_pending() - Checks if a reset is pending. +* +* Return: true if a reset is pending, false otherwise. +*/ +static inline bool panthor_device_reset_is_pending (struct panthor_device *ptdev) +{ +return atomic_read (&ptdev->reset.pending) !=0 ; +} + +int panthor_device_mmap_io (struct panthor_device *ptdev, +struct vm_area_struct *vma); + +int panthor_device_resume (struct device *dev); +int panthor_device_suspend (struct device *dev); + +enum drm_panthor_exception_type { +DRM_PANTHOR_EXCEPTION_OK =0x00 , +DRM_PANTHOR_EXCEPTION_TERMINATED =0x04 , +DRM_PANTHOR_EXCEPTION_KABOOM =0x05 , +DRM_PANTHOR_EXCEPTION_EUREKA =0x06 , +DRM_PANTHOR_EXCEPTION_ACTIVE =0x08 , +DRM_PANTHOR_EXCEPTION_CS_RES_TERM =0x0f , +DRM_PANTHOR_EXCEPTION_MAX_NON_FAULT =0x3f , +DRM_PANTHOR_EXCEPTION_CS_CONFIG_FAULT =0x40 , +DRM_PANTHOR_EXCEPTION_CS_ENDPOINT_FAULT =0x44 , +DRM_PANTHOR_EXCEPTION_CS_BUS_FAULT =0x48 , +DRM_PANTHOR_EXCEPTION_CS_INSTR_INVALID =0x49 , +DRM_PANTHOR_EXCEPTION_CS_CALL_STACK_OVERFLOW =0x4a , +DRM_PANTHOR_EXCEPTION_CS_INHERIT_FAULT =0x4b , +DRM_PANTHOR_EXCEPTION_INSTR_INVALID_PC =0x50 , +DRM_PANTHOR_EXCEPTION_INSTR_INVALID_ENC =0x51 , +DRM_PANTHOR_EXCEPTION_INSTR_BARRIER_FAULT =0x55 , +DRM_PANTHOR_EXCEPTION_DATA_INVALID_FAULT =0x58 , +DRM_PANTHOR_EXCEPTION_TILE_RANGE_FAULT =0x59 , +DRM_PANTHOR_EXCEPTION_ADDR_RANGE_FAULT =0x5a , +DRM_PANTHOR_EXCEPTION_IMPRECISE_FAULT =0x5b , +DRM_PANTHOR_EXCEPTION_OOM =0x60 , +DRM_PANTHOR_EXCEPTION_CSF_FW_INTERNAL_ERROR =0x68 , +DRM_PANTHOR_EXCEPTION_CSF_RES_EVICTION_TIMEOUT =0x69 , +DRM_PANTHOR_EXCEPTION_GPU_BUS_FAULT =0x80 , +DRM_PANTHOR_EXCEPTION_GPU_SHAREABILITY_FAULT =0x88 , +DRM_PANTHOR_EXCEPTION_SYS_SHAREABILITY_FAULT =0x89 , +DRM_PANTHOR_EXCEPTION_GPU_CACHEABILITY_FAULT =0x8a , +DRM_PANTHOR_EXCEPTION_TRANSLATION_FAULT_0 =0xc0 , +DRM_PANTHOR_EXCEPTION_TRANSLATION_FAULT_1 =0xc1 , +DRM_PANTHOR_EXCEPTION_TRANSLATION_FAULT_2 =0xc2 , +DRM_PANTHOR_EXCEPTION_TRANSLATION_FAULT_3 =0xc3 , +DRM_PANTHOR_EXCEPTION_TRANSLATION_FAULT_4 =0xc4 , +DRM_PANTHOR_EXCEPTION_PERM_FAULT_0 =0xc8 , +DRM_PANTHOR_EXCEPTION_PERM_FAULT_1 =0xc9 , +DRM_PANTHOR_EXCEPTION_PERM_FAULT_2 =0xca , +DRM_PANTHOR_EXCEPTION_PERM_FAULT_3 =0xcb , +DRM_PANTHOR_EXCEPTION_ACCESS_FLAG_1 =0xd9 , +DRM_PANTHOR_EXCEPTION_ACCESS_FLAG_2 =0xda , +DRM_PANTHOR_EXCEPTION_ACCESS_FLAG_3 =0xdb , +DRM_PANTHOR_EXCEPTION_ADDR_SIZE_FAULT_IN =0xe0 , +DRM_PANTHOR_EXCEPTION_ADDR_SIZE_FAULT_OUT0 =0xe4 , +DRM_PANTHOR_EXCEPTION_ADDR_SIZE_FAULT_OUT1 =0xe5 , +DRM_PANTHOR_EXCEPTION_ADDR_SIZE_FAULT_OUT2 =0xe6 , +DRM_PANTHOR_EXCEPTION_ADDR_SIZE_FAULT_OUT3 =0xe7 , +DRM_PANTHOR_EXCEPTION_MEM_ATTR_FAULT_0 =0xe8 , +DRM_PANTHOR_EXCEPTION_MEM_ATTR_FAULT_1 =0xe9 , +DRM_PANTHOR_EXCEPTION_MEM_ATTR_FAULT_2 =0xea , +DRM_PANTHOR_EXCEPTION_MEM_ATTR_FAULT_3 =0xeb , +}; + +/** +* panthor_exception_is_fault() - Checks if an exception is a fault. +* +* Return: true if the exception is a fault, false otherwise. +*/ +static inline bool +panthor_exception_is_fault (u32 exception_code) +{ +return exception_code >DRM_PANTHOR_EXCEPTION_MAX_NON_FAULT ; +} + +const char *panthor_exception_name (struct panthor_device *ptdev, +u32 exception_code); + +/** +* PANTHOR_IRQ_HANDLER() - Define interrupt handlers and the interrupt +* registration function. +* +* The boiler-plate to gracefully deal with shared interrupts is +* auto-generated. All you have to do is call PANTHOR_IRQ_HANDLER() +* just after the actual handler. The handler prototype is: +* +* void (*handler)(struct panthor_device *, u32 status); +*/ +#define PANTHOR_IRQ_HANDLER (__name, __reg_prefix, __handler) \ +static irqreturn_t panthor_ ## __name ## _irq_raw_handler(int irq, void *data) \ +{ \ + struct panthor_irq *pirq = data; \ + struct panthor_device *ptdev = pirq->ptdev; \ + \ + if (atomic_read(&pirq->suspended)) \ + return IRQ_NONE; \ + if (!gpu_read(ptdev, __reg_prefix ## _INT_STAT)) \ + return IRQ_NONE; \ + \ + gpu_write(ptdev, __reg_prefix ## _INT_MASK, 0); \ + return IRQ_WAKE_THREAD; \ +} \ + \ +static irqreturn_t panthor_ ## __name ## _irq_threaded_handler(int irq, void *data) \ +{ \ + struct panthor_irq *pirq = data; \ + struct panthor_device *ptdev = pirq->ptdev; \ + irqreturn_t ret = IRQ_NONE; \ + \ + while (true) { \ + u32 status = gpu_read(ptdev, __reg_prefix ## _INT_RAWSTAT) & pirq->mask; \ + \ + if (!status) \ + break; \ + \ + gpu_write(ptdev, __reg_prefix ## _INT_CLEAR, status); \ + \ + __handler(ptdev, status); \ + ret = IRQ_HANDLED; \ + } \ + \ + if (!atomic_read(&pirq->suspended)) \ + gpu_write(ptdev, __reg_prefix ## _INT_MASK, pirq->mask); \ + \ + return ret; \ +} \ + \ +static inline void panthor_ ## __name ## _irq_suspend(struct panthor_irq *pirq) \ +{ \ + int cookie; \ + \ + atomic_set(&pirq->suspended, true); \ + \ + if (drm_dev_enter(&pirq->ptdev->base, &cookie)) { \ + gpu_write(pirq->ptdev, __reg_prefix ## _INT_MASK, 0); \ + synchronize_irq(pirq->irq); \ + drm_dev_exit(cookie); \ + } \ + \ + pirq->mask = 0; \ +} \ + \ +static inline void panthor_ ## __name ## _irq_resume(struct panthor_irq *pirq, u32 mask) \ +{ \ + int cookie; \ + \ + atomic_set(&pirq->suspended, false); \ + pirq->mask = mask; \ + \ + if (drm_dev_enter(&pirq->ptdev->base, &cookie)) { \ + gpu_write(pirq->ptdev, __reg_prefix ## _INT_CLEAR, mask); \ + gpu_write(pirq->ptdev, __reg_prefix ## _INT_MASK, mask); \ + drm_dev_exit(cookie); \ + } \ +} \ + \ +static int panthor_request_ ## __name ## _irq(struct panthor_device *ptdev, \ + struct panthor_irq *pirq, \ + int irq, u32 mask) \ +{ \ + pirq->ptdev = ptdev; \ + pirq->irq = irq; \ + panthor_ ## __name ## _irq_resume(pirq, mask); \ + \ + return devm_request_threaded_irq(ptdev->base.dev, irq, \ + panthor_ ## __name ## _irq_raw_handler, \ + panthor_ ## __name ## _irq_threaded_handler, \ + IRQF_SHARED, KBUILD_MODNAME "-" # __name, \ + pirq); \ +} + +/** +* panthor_device_mmio_offset() - Turn a user MMIO offset into a kernel one +* @offset: Offset to convert. +* +* With 32-bit systems being limited by the 32-bit representation of mmap2's +* pgoffset field, we need to make the MMIO offset arch specific. This function +* converts a user MMIO offset into something the kernel driver understands. +* +* If the kernel and userspace architecture match, the offset is unchanged. If +* the kernel is 64-bit and userspace is 32-bit, the offset is adjusted to match +* 64-bit offsets. 32-bit kernel with 64-bit userspace is impossible. +* +* Return: Adjusted offset. +*/ +static inline u64 panthor_device_mmio_offset (u64 offset) +{ +#ifdef CONFIG_ARM64 +if (test_tsk_thread_flag (current,TIF_32BIT )) + offset +=DRM_PANTHOR_USER_MMIO_OFFSET_64BIT -DRM_PANTHOR_USER_MMIO_OFFSET_32BIT ; +#endif + +return offset; +} + +extern struct workqueue_struct *panthor_cleanup_wq; + +#endif Signature · 1 line
2.43.0
Signature · 8 lines
====================
| I would like to |
| fix the world, |
| but they're not |
| giving me the |
\ source code! /
---------------
¯\_(ツ)_/¯Boris Brezillon
maintainerRe: [PATCH v3 03/14] drm/panthor: Add the device logical block
Quoted 255 lines · On Fri, 22 Dec 2023 13:26:22 +0000 Liviu Dudau <[email protected]> wrote:
Hi Boris,
Quoted 252 lines · Boris Brezillon
The panthor driver is designed in a modular way, where each logical block is dealing with a specific HW-block or software feature. In order for those blocks to communicate with each other, we need a central
panthor_devicecollecting all the blocks, and exposing some common features, like interrupt handling, power management, reset, ...This what this
panthor_devicelogical block is about.v3:
- Add acks for the MIT+GPL2 relicensing
- Fix 32-bit support
- Shorten the sections protected bypanthor_device::pm::mmio_lockto fix lock ordering issues.
- Renamepanthor_device::pm::lockintopanthor_device::pm::mmio_lockto better reflect what this lock is protecting
- Usedev_err_probe()
- Make sure we calldrm_dev_exit()when something fails half-way inpanthor_device_reset_work()
- ReplaceCSF_GPU_LATEST_FLUSH_ID_DEFAULTwith a constant '1' and a comment to explain. Also remove setting the dummy flush ID on suspend.
- Removedrm_WARN_ON()inpanthor_exception_name()
- Checkpirq->suspendedinpanthor_xxx_irq_raw_handler()Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>--- drivers/gpu/drm/panthor/panthor_device.c | 497 +++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_device.h | 381 +++++++++++++++++ 2 files changed, 878 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_device.c create mode 100644 drivers/gpu/drm/panthor/panthor_device.hdrivers/gpu/drm/panthor/panthor_device.c new file +211
-0@@ -0,0 +1,497 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2018 Marty E. Plummer <[email protected]> */ +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ + +#include <linux/clk.h> +#include <linux/reset.h> +#include <linux/platform_device.h> +#include <linux/pm_domain.h> +#include <linux/pm_runtime.h> +#include <linux/regulator/consumer.h> + +#include <drm/drm_drv.h> +#include <drm/drm_managed.h> + +#include "panthor_sched.h" +#include "panthor_device.h" +#include "panthor_devfreq.h" +#include "panthor_gpu.h" +#include "panthor_fw.h" +#include "panthor_mmu.h" +#include "panthor_regs.h" + +static int panthor_clk_init (struct panthor_device *ptdev) +{ + ptdev->clks.core =devm_clk_get (ptdev->base.dev,NULL ); +if (IS_ERR (ptdev->clks.core)) +return dev_err_probe (ptdev->base.dev, +PTR_ERR (ptdev->clks.core), +"get 'core' clock failed" ); + + ptdev->clks.stacks =devm_clk_get_optional (ptdev->base.dev,"stacks" ); +if (IS_ERR (ptdev->clks.stacks)) +return dev_err_probe (ptdev->base.dev, +PTR_ERR (ptdev->clks.stacks), +"get 'stacks' clock failed" ); + + ptdev->clks.coregroup =devm_clk_get_optional (ptdev->base.dev,"coregroup" ); +if (IS_ERR (ptdev->clks.coregroup)) +return dev_err_probe (ptdev->base.dev, +PTR_ERR (ptdev->clks.coregroup), +"get 'coregroup' clock failed" ); + +drm_info (&ptdev->base,"clock rate = %lu\n" ,clk_get_rate (ptdev->clks.core)); +return 0 ; +} + +void panthor_device_unplug (struct panthor_device *ptdev) +{ +/* FIXME: This is racy. */ +if (drm_dev_is_unplugged (&ptdev->base)) +return ; + +drm_WARN_ON (&ptdev->base,pm_runtime_get_sync (ptdev->base.dev) <0 ); + +/* Call drm_dev_unplug() so any access to HW block happening after +* that point get rejected. +*/ +drm_dev_unplug (&ptdev->base); + +/* Now, try to cleanly shutdown the GPU before the device resources +* get reclaimed. +*/ +panthor_sched_unplug (ptdev); +panthor_fw_unplug (ptdev); +panthor_mmu_unplug (ptdev); +panthor_gpu_unplug (ptdev); + +pm_runtime_dont_use_autosuspend (ptdev->base.dev); +pm_runtime_put_sync_suspend (ptdev->base.dev); +} + +static void panthor_device_reset_cleanup (struct drm_device *ddev,void *data) +{ + struct panthor_device *ptdev = container_of(ddev, struct panthor_device, base); + +cancel_work_sync (&ptdev->reset.work); +destroy_workqueue (ptdev->reset.wq); +} + +static void panthor_device_reset_work (struct work_struct *work) +{ + struct panthor_device *ptdev = container_of(work, struct panthor_device, reset.work); +int ret =0 , cookie; + +if (atomic_read (&ptdev->pm.state) !=PANTHOR_DEVICE_PM_STATE_ACTIVE ) { +/* +* No need for a reset as the device has been (or will be) +* powered down +*/ +atomic_set (&ptdev->reset.pending,0 ); +return ; + } + +if (!drm_dev_enter (&ptdev->base, &cookie)) +return ; + +panthor_sched_pre_reset (ptdev); +panthor_fw_pre_reset (ptdev, true); +panthor_mmu_pre_reset (ptdev); +panthor_gpu_soft_reset (ptdev); +panthor_gpu_l2_power_on (ptdev); +panthor_mmu_post_reset (ptdev); + ret =panthor_fw_post_reset (ptdev); +if (ret) + goto out_dev_exit; + +atomic_set (&ptdev->reset.pending,0 ); +panthor_sched_post_reset (ptdev); + +if (ret) { +panthor_device_unplug (ptdev); +drm_err (&ptdev->base,"Failed to boot MCU after reset, making device unusable." ); + } + +out_dev_exit: +drm_dev_exit (cookie); +} + +static bool panthor_device_is_initialized (struct panthor_device *ptdev) +{ +return !!ptdev->scheduler; +} + +static void panthor_device_free_page (struct drm_device *ddev,void *data) +{ +free_page ((unsigned long )data); +} + +int panthor_device_init (struct panthor_device *ptdev) +{ +struct resource *res; +struct page *p; +int ret; + + ptdev->coherent =device_get_dma_attr (ptdev->base.dev) ==DEV_DMA_COHERENT ; + +drmm_mutex_init (&ptdev->base, &ptdev->pm.mmio_lock); +atomic_set (&ptdev->pm.state,PANTHOR_DEVICE_PM_STATE_SUSPENDED ); + p =alloc_page (GFP_KERNEL | __GFP_ZERO); +if (!p) +return -ENOMEM ; + + ptdev->pm.dummy_latest_flush =page_address (p); + ret =drmm_add_action_or_reset (&ptdev->base, panthor_device_free_page, + ptdev->pm.dummy_latest_flush); +if (ret) +return ret; + +/* +* Set the dummy page holding the latest flush to 1. This will cause the +* flush to avoided as we know it isn't necessary if the submission +* happens while the dummy page is mapped. Zero cannot be used because +* that means 'always flush'. +*/ + *ptdev->pm.dummy_latest_flush =1 ; + +INIT_WORK (&ptdev->reset.work, panthor_device_reset_work); + ptdev->reset.wq =alloc_ordered_workqueue ("panthor-reset-wq" ,0 ); +if (!ptdev->reset.wq) +return -ENOMEM ; + + ret =drmm_add_action_or_reset (&ptdev->base, panthor_device_reset_cleanup,NULL ); +if (ret) +return ret; + + ret =panthor_clk_init (ptdev); +if (ret) +return ret; + + ret =panthor_devfreq_init (ptdev); +if (ret) +return ret; + + ptdev->iomem =devm_platform_get_and_ioremap_resource (to_platform_device (ptdev->base.dev), +0 , &res); +if (IS_ERR (ptdev->iomem)) +return PTR_ERR (ptdev->iomem); + + ptdev->phys_addr = res->start; + + ret =devm_pm_runtime_enable (ptdev->base.dev); +if (ret) +return ret; + + ret =pm_runtime_resume_and_get (ptdev->base.dev); +if (ret) +return ret; + + ret =panthor_gpu_init (ptdev); +if (ret) + goto err_rpm_put; + + ret =panthor_mmu_init (ptdev); +if (ret) + goto err_rpm_put; + + ret =panthor_fw_init (ptdev); +if (ret) + goto err_rpm_put; + + ret =panthor_sched_init (ptdev); +if (ret) + goto err_rpm_put; + +/* ~3 frames */ +pm_runtime_set_autosuspend_delay (ptdev->base.dev,50 ); +pm_runtime_use_autosuspend (ptdev->base.dev); +pm_runtime_put_autosuspend (ptdev->base.dev); +return 0 ; +
Quoted 8 lines
drivers/gpu/drm/panthor/panthor_device.c line 213
Quoted 2 lines
drivers/gpu/drm/panthor/panthor_device.c new file +2
-0+err_rpm_put: +pm_runtime_put_sync_suspend (ptdev->base.dev);I'm afraid this is not enough to cleanup the driver if firmware fails to load or if
panthor_sched_init()fails. I was playing with another device that's based on RK3588 and forgot to move the firmware to the new location. I've got a kernel crash with an SError due to failure to load the firmware. It was inpanthor_mmu_irq_raw_handler()which fired because the MMU was not unplugged.
Uh, right. I hate the fact we have to explicitly call the unplug() functions in that case, but I also don't have a better solution...
Quoted 9 lines
My local fix was:
-- >8 --------------------------------------------------------------
drivers/gpu/drm/panthor/panthor_device.c +0
-0
drivers/gpu/drm/panthor/panthor_device.c int panthor_device_init(struct panthor_device *ptdev) · line 227
Quoted 5 lines
drivers/gpu/drm/panthor/panthor_device.c +1
-1@@ -224,15 +224,15 @@ int panthor_device_init(struct panthor_device *ptdev) ret =panthor_mmu_init (ptdev);if (ret) - goto err_rpm_put; + goto err_mmu_init;
All the label names encode the unwind operation they're taking care of, so s/err_mmu_init/err_unplug_gpu/.
Quoted 29 lines
drivers/gpu/drm/panthor/panthor_device.c +8
-3ret =panthor_fw_init (ptdev);if (ret) - goto err_rpm_put; + goto err_fw_init; ret =panthor_sched_init (ptdev);if (ret) - goto err_rpm_put; + goto err_sched_init;/* ~3 frames */ pm_runtime_set_autosuspend_delay (ptdev->base.dev,50 );@@ -240,6 +240,12 @@ int panthor_device_init(struct panthor_device *ptdev)pm_runtime_put_autosuspend (ptdev->base.dev);return 0 ; +err_sched_init: +panthor_fw_unplug (ptdev); +err_fw_init: +panthor_mmu_unplug (ptdev); +err_mmu_init: +panthor_gpu_unplug (ptdev); err_rpm_put:pm_runtime_put_sync_suspend (ptdev->base.dev);return ret; -- >8 -------------------------------------------------------------
Quoted 1 line
I can send a fully formed patch if you agree with the fix.
I'll take care of send when preparing v4. Thanks for the offer.
Boris Brezillon
maintainerPATCH v3 4/14 drm/panthor: Add the GPU logical block
In reply to Boris BrezillonHandles everything that's not related to the FW, the MMU or the scheduler. This is the block dealing with the GPU property retrieval, the GPU block power on/off logic, and some global operations, like global cache flushing.
v3:
- Add acks for the MIT/GPL2 relicensing
- Use macros to extract GPU ID info
- Make sure we reset clear pending_reqs bits when wait_event_timeout() times out but the corresponding bit is cleared in GPU_INT_RAWSTAT (can happen if the IRQ is masked or HW takes to long to call the IRQ handler)
- GPU_MODEL now takes separate arch and product majors to be more readable.
- Drop GPU_IRQ_MCU_STATUS_CHANGED from interrupt mask.
- Handle GPU_IRQ_PROTM_FAULT correctly (don't output registers that are not updated for protected interrupts).
- Minor code tidy ups
--- drivers/gpu/drm/panthor/panthor_gpu.c | 481 ++++++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_gpu.h | 52 +++ 2 files changed, 533 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_gpu.c create mode 100644 drivers/gpu/drm/panthor/panthor_gpu.h
drivers/gpu/drm/panthor/panthor_gpu.c new file +481 -0
@@ -0,0 +1,481 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2018 Marty E. Plummer <[email protected]> */ +/* Copyright 2019 Linaro, Ltd., Rob Herring <[email protected]> */ +/* Copyright 2019 Collabora ltd. */ + +#include <linux/bitfield.h> +#include <linux/bitmap.h> +#include <linux/delay.h> +#include <linux/dma-mapping.h> +#include <linux/interrupt.h> +#include <linux/io.h> +#include <linux/iopoll.h> +#include <linux/platform_device.h> +#include <linux/pm_runtime.h> + +#include <drm/drm_drv.h> +#include <drm/drm_managed.h> + +#include "panthor_device.h" +#include "panthor_gpu.h" +#include "panthor_regs.h" + +/** +* struct panthor_gpu - GPU block management data. +*/ +struct panthor_gpu { +/** @irq: GPU irq. */ +struct panthor_irq irq; + +/** @reqs_lock: Lock protecting access to pending_reqs. */ +spinlock_t reqs_lock; + +/** @pending_reqs: Pending GPU requests. */ +u32 pending_reqs; + +/** @reqs_acked: GPU request wait queue. */ +wait_queue_head_t reqs_acked; +}; + +/** +* struct panthor_model - GPU model description +*/ +struct panthor_model { +/** @name: Model name. */ +const char *name; + +/** @arch_major: Major version number of architecture. */ +u8 arch_major; + +/** @product_major: Major version number of product. */ +u8 product_major; +}; + +/** +* GPU_MODEL() - Define a GPU model. A GPU product can be uniquely identified +* by a combination of the major architecture version and the major product +* version. +* @name: Name for the GPU model. +* @_arch_major: Architecture major. +* @_product_major: Product major. +*/ +#define GPU_MODEL (_name, _arch_major, _product_major) \ +{\ + .name = __stringify(_name), \ + .arch_major = _arch_major, \ + .product_major = _product_major, \ +} + +static const struct panthor_model gpu_models[] = { +GPU_MODEL (g610,10 ,7 ), + {}, +}; + +#define GPU_INTERRUPTS_MASK \ + (GPU_IRQ_FAULT | \ + GPU_IRQ_PROTM_FAULT | \ + GPU_IRQ_RESET_COMPLETED | \ + GPU_IRQ_CLEAN_CACHES_COMPLETED) + +static void panthor_gpu_init_info (struct panthor_device *ptdev) +{ +const struct panthor_model *model; +u32 arch_major, product_major; +u32 major, minor, status; +unsigned int i; + + ptdev->gpu_info.gpu_id =gpu_read (ptdev,GPU_ID ); + ptdev->gpu_info.csf_id =gpu_read (ptdev,GPU_CSF_ID ); + ptdev->gpu_info.gpu_rev =gpu_read (ptdev,GPU_REVID ); + ptdev->gpu_info.l2_features =gpu_read (ptdev,GPU_L2_FEATURES ); + ptdev->gpu_info.tiler_features =gpu_read (ptdev,GPU_TILER_FEATURES ); + ptdev->gpu_info.mem_features =gpu_read (ptdev,GPU_MEM_FEATURES ); + ptdev->gpu_info.mmu_features =gpu_read (ptdev,GPU_MMU_FEATURES ); + ptdev->gpu_info.thread_features =gpu_read (ptdev,GPU_THREAD_FEATURES ); + ptdev->gpu_info.max_threads =gpu_read (ptdev,GPU_THREAD_MAX_THREADS ); + ptdev->gpu_info.thread_max_workgroup_size =gpu_read (ptdev,GPU_THREAD_MAX_WORKGROUP_SIZE ); + ptdev->gpu_info.thread_max_barrier_size =gpu_read (ptdev,GPU_THREAD_MAX_BARRIER_SIZE ); + ptdev->gpu_info.coherency_features =gpu_read (ptdev,GPU_COHERENCY_FEATURES ); +for (i =0 ; i <4 ; i++) + ptdev->gpu_info.texture_features[i] =gpu_read (ptdev,GPU_TEXTURE_FEATURES (i)); + + ptdev->gpu_info.as_present =gpu_read (ptdev,GPU_AS_PRESENT ); + + ptdev->gpu_info.shader_present =gpu_read (ptdev,GPU_SHADER_PRESENT_LO ); + ptdev->gpu_info.shader_present |= (u64 )gpu_read (ptdev,GPU_SHADER_PRESENT_HI ) <<32 ; + + ptdev->gpu_info.tiler_present =gpu_read (ptdev,GPU_TILER_PRESENT_LO ); + ptdev->gpu_info.tiler_present |= (u64 )gpu_read (ptdev,GPU_TILER_PRESENT_HI ) <<32 ; + + ptdev->gpu_info.l2_present =gpu_read (ptdev,GPU_L2_PRESENT_LO ); + ptdev->gpu_info.l2_present |= (u64 )gpu_read (ptdev,GPU_L2_PRESENT_HI ) <<32 ; + + arch_major =GPU_ARCH_MAJOR (ptdev->gpu_info.gpu_id); + product_major =GPU_PROD_MAJOR (ptdev->gpu_info.gpu_id); + major =GPU_VER_MAJOR (ptdev->gpu_info.gpu_id); + minor =GPU_VER_MINOR (ptdev->gpu_info.gpu_id); + status =GPU_VER_STATUS (ptdev->gpu_info.gpu_id); + +for (model = gpu_models; model->name; model++) { +if (model->arch_major == arch_major && + model->product_major == product_major) +break ; + } + +drm_info (&ptdev->base, +"mali-%s id 0x%x major 0x%x minor 0x%x status 0x%x" , + model->name ?:"unknown" , ptdev->gpu_info.gpu_id >>16 , + major, minor, status); + +drm_info (&ptdev->base, +"Features: L2:%#x Tiler:%#x Mem:%#x MMU:%#x AS:%#x" , + ptdev->gpu_info.l2_features, + ptdev->gpu_info.tiler_features, + ptdev->gpu_info.mem_features, + ptdev->gpu_info.mmu_features, + ptdev->gpu_info.as_present); + +drm_info (&ptdev->base, +"shader_present=0x%0llx l2_present=0x%0llx tiler_present=0x%0llx" , + ptdev->gpu_info.shader_present, ptdev->gpu_info.l2_present, + ptdev->gpu_info.tiler_present); +} + +static void panthor_gpu_irq_handler (struct panthor_device *ptdev,u32 status) +{ +if (status &GPU_IRQ_FAULT ) { +u32 fault_status =gpu_read (ptdev,GPU_FAULT_STATUS ); +u64 address = ((u64 )gpu_read (ptdev,GPU_FAULT_ADDR_HI ) <<32 ) | +gpu_read (ptdev,GPU_FAULT_ADDR_LO ); + +drm_warn (&ptdev->base,"GPU Fault 0x%08x (%s) at 0x%016llx\n" , + fault_status,panthor_exception_name (ptdev, fault_status &0xFF ), + address); + } +if (status &GPU_IRQ_PROTM_FAULT ) +drm_warn (&ptdev->base,"GPU Fault in protected mode\n" ); + +spin_lock (&ptdev->gpu->reqs_lock); +if (status & ptdev->gpu->pending_reqs) { + ptdev->gpu->pending_reqs &= ~status; +wake_up_all (&ptdev->gpu->reqs_acked); + } +spin_unlock (&ptdev->gpu->reqs_lock); +} +PANTHOR_IRQ_HANDLER (gpu,GPU , panthor_gpu_irq_handler); + +/** +* panthor_gpu_unplug() - Called when the GPU is unplugged. +* @ptdev: Device to unplug. +*/ +void panthor_gpu_unplug (struct panthor_device *ptdev) +{ +unsigned long flags; + +/* Make sure the IRQ handler is not running after that point. */ +panthor_gpu_irq_suspend (&ptdev->gpu->irq); + +/* Wake-up all waiters. */ +spin_lock_irqsave (&ptdev->gpu->reqs_lock, flags); + ptdev->gpu->pending_reqs =0 ; +wake_up_all (&ptdev->gpu->reqs_acked); +spin_unlock_irqrestore (&ptdev->gpu->reqs_lock, flags); +} + +/** +* panthor_gpu_init() - Initialize the GPU block +* @ptdev: Device. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_init (struct panthor_device *ptdev) +{ +struct panthor_gpu *gpu; +u32 pa_bits; +int ret, irq; + + gpu =drmm_kzalloc (&ptdev->base,sizeof (*gpu),GFP_KERNEL ); +if (!gpu) +return -ENOMEM ; + +spin_lock_init (&gpu->reqs_lock); +init_waitqueue_head (&gpu->reqs_acked); + ptdev->gpu = gpu; +panthor_gpu_init_info (ptdev); + +dma_set_max_seg_size (ptdev->base.dev,UINT_MAX ); + pa_bits =GPU_MMU_FEATURES_PA_BITS (ptdev->gpu_info.mmu_features); + ret =dma_set_mask_and_coherent (ptdev->base.dev,DMA_BIT_MASK (pa_bits)); +if (ret) +return ret; + + irq =platform_get_irq_byname (to_platform_device (ptdev->base.dev),"gpu" ); +if (irq <=0 ) +return ret; + + ret =panthor_request_gpu_irq (ptdev, &ptdev->gpu->irq, irq,GPU_INTERRUPTS_MASK ); +if (ret) +return ret; + +return 0 ; +} + +/** +* panthor_gpu_block_power_off() - Power-off a specific block of the GPU +* @ptdev: Device. +* @blk_name: Block name. +* @pwroff_reg: Power-off register for this block. +* @pwrtrans_reg: Power transition register for this block. +* @mask: Sub-elements to power-off. +* @timeout_us: Timeout in microseconds. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_block_power_off (struct panthor_device *ptdev, +const char *blk_name, +u32 pwroff_reg,u32 pwrtrans_reg, +u64 mask,u32 timeout_us) +{ +u32 val, i; +int ret; + +for (i =0 ; i <2 ; i++) { +u32 mask32 = mask >> (i *32 ); + +if (!mask32) +continue ; + + ret =readl_relaxed_poll_timeout (ptdev->iomem + pwrtrans_reg + (i *4 ), + val, !(mask32 & val), +100 , timeout_us); +if (ret) { +drm_err (&ptdev->base,"timeout waiting on %s:%llx power transition" , + blk_name, mask); +return ret; + } + } + +if (mask &GENMASK (31 ,0 )) +gpu_write (ptdev, pwroff_reg, mask); + +if (mask >>32 ) +gpu_write (ptdev, pwroff_reg +4 , mask >>32 ); + +for (i =0 ; i <2 ; i++) { +u32 mask32 = mask >> (i *32 ); + +if (!mask32) +continue ; + + ret =readl_relaxed_poll_timeout (ptdev->iomem + pwrtrans_reg + (i *4 ), + val, !(mask & val), +100 , timeout_us); +if (ret) { +drm_err (&ptdev->base,"timeout waiting on %s:%llx power transition" , + blk_name, mask); +return ret; + } + } + +return 0 ; +} + +/** +* panthor_gpu_block_power_on() - Power-on a specific block of the GPU +* @ptdev: Device. +* @blk_name: Block name. +* @pwron_reg: Power-on register for this block. +* @pwrtrans_reg: Power transition register for this block. +* @rdy_reg: Power transition ready register. +* @mask: Sub-elements to power-on. +* @timeout_us: Timeout in microseconds. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_block_power_on (struct panthor_device *ptdev, +const char *blk_name, +u32 pwron_reg,u32 pwrtrans_reg, +u32 rdy_reg,u64 mask,u32 timeout_us) +{ +u32 val, i; +int ret; + +for (i =0 ; i <2 ; i++) { +u32 mask32 = mask >> (i *32 ); + +if (!mask32) +continue ; + + ret =readl_relaxed_poll_timeout (ptdev->iomem + pwrtrans_reg + (i *4 ), + val, !(mask32 & val), +100 , timeout_us); +if (ret) { +drm_err (&ptdev->base,"timeout waiting on %s:%llx power transition" , + blk_name, mask); +return ret; + } + } + +if (mask &GENMASK (31 ,0 )) +gpu_write (ptdev, pwron_reg, mask); + +if (mask >>32 ) +gpu_write (ptdev, pwron_reg +4 , mask >>32 ); + +for (i =0 ; i <2 ; i++) { +u32 mask32 = mask >> (i *32 ); + +if (!mask32) +continue ; + + ret =readl_relaxed_poll_timeout (ptdev->iomem + rdy_reg + (i *4 ), + val, (mask32 & val) == mask32, +100 , timeout_us); +if (ret) { +drm_err (&ptdev->base,"timeout waiting on %s:%llx readyness" , + blk_name, mask); +return ret; + } + } + +return 0 ; +} + +/** +* panthor_gpu_l2_power_on() - Power-on the L2-cache +* @ptdev: Device. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_l2_power_on (struct panthor_device *ptdev) +{ +if (ptdev->gpu_info.l2_present !=1 ) { +/* +* Only support one core group now. +* ~(l2_present - 1) unsets all bits in l2_present except +* the bottom bit. (l2_present - 2) has all the bits in +* the first core group set. AND them together to generate +* a mask of cores in the first core group. +*/ +u64 core_mask = ~(ptdev->gpu_info.l2_present -1 ) & + (ptdev->gpu_info.l2_present -2 ); +drm_info_once (&ptdev->base,"using only 1st core group (%lu cores from %lu)\n" , +hweight64 (core_mask), +hweight64 (ptdev->gpu_info.shader_present)); + } + +return panthor_gpu_power_on (ptdev,L2 ,1 ,20000 ); +} + +/** +* panthor_gpu_flush_caches() - Flush caches +* @ptdev: Device. +* @l2: L2 flush type. +* @lsc: LSC flush type. +* @other: Other flush type. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_flush_caches (struct panthor_device *ptdev, +u32 l2,u32 lsc,u32 other) +{ +bool timedout = false; +unsigned long flags; + +spin_lock_irqsave (&ptdev->gpu->reqs_lock, flags); +if (!drm_WARN_ON (&ptdev->base, + ptdev->gpu->pending_reqs &GPU_IRQ_CLEAN_CACHES_COMPLETED )) { + ptdev->gpu->pending_reqs |=GPU_IRQ_CLEAN_CACHES_COMPLETED ; +gpu_write (ptdev,GPU_CMD ,GPU_FLUSH_CACHES (l2, lsc, other)); + } +spin_unlock_irqrestore (&ptdev->gpu->reqs_lock, flags); + +if (!wait_event_timeout (ptdev->gpu->reqs_acked, + !(ptdev->gpu->pending_reqs &GPU_IRQ_CLEAN_CACHES_COMPLETED ), +msecs_to_jiffies (100 ))) { +spin_lock_irqsave (&ptdev->gpu->reqs_lock, flags); +if ((ptdev->gpu->pending_reqs &GPU_IRQ_CLEAN_CACHES_COMPLETED ) !=0 && + !(gpu_read (ptdev,GPU_INT_RAWSTAT ) &GPU_IRQ_CLEAN_CACHES_COMPLETED )) + timedout = true; +else + ptdev->gpu->pending_reqs &= ~GPU_IRQ_CLEAN_CACHES_COMPLETED ; +spin_unlock_irqrestore (&ptdev->gpu->reqs_lock, flags); + } + +if (timedout) { +drm_err (&ptdev->base,"Flush caches timeout" ); +return -ETIMEDOUT ; + } + +return 0 ; +} + +/** +* panthor_gpu_soft_reset() - Issue a soft-reset +* @ptdev: Device. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_soft_reset (struct panthor_device *ptdev) +{ +bool timedout = false; +unsigned long flags; + +spin_lock_irqsave (&ptdev->gpu->reqs_lock, flags); +if (!drm_WARN_ON (&ptdev->base, + ptdev->gpu->pending_reqs &GPU_IRQ_RESET_COMPLETED )) { + ptdev->gpu->pending_reqs |=GPU_IRQ_RESET_COMPLETED ; +gpu_write (ptdev,GPU_INT_CLEAR ,GPU_IRQ_RESET_COMPLETED ); +gpu_write (ptdev,GPU_CMD ,GPU_SOFT_RESET ); + } +spin_unlock_irqrestore (&ptdev->gpu->reqs_lock, flags); + +if (!wait_event_timeout (ptdev->gpu->reqs_acked, + !(ptdev->gpu->pending_reqs &GPU_IRQ_RESET_COMPLETED ), +msecs_to_jiffies (100 ))) { +spin_lock_irqsave (&ptdev->gpu->reqs_lock, flags); +if ((ptdev->gpu->pending_reqs &GPU_IRQ_RESET_COMPLETED ) !=0 && + !(gpu_read (ptdev,GPU_INT_RAWSTAT ) &GPU_IRQ_RESET_COMPLETED )) + timedout = true; +else + ptdev->gpu->pending_reqs &= ~GPU_IRQ_RESET_COMPLETED ; +spin_unlock_irqrestore (&ptdev->gpu->reqs_lock, flags); + } + +if (timedout) { +drm_err (&ptdev->base,"Soft reset timeout" ); +return -ETIMEDOUT ; + } + +return 0 ; +} + +/** +* panthor_gpu_suspend() - Suspend the GPU block. +* @ptdev: Device. +* +* Suspend the GPU irq. This should be called last in the suspend procedure, +* after all other blocks have been suspented. +*/ +void panthor_gpu_suspend (struct panthor_device *ptdev) +{ +/* +* It may be preferable to simply power down the L2, but for now just +* soft-reset which will leave the L2 powered down. +*/ +panthor_gpu_soft_reset (ptdev); +panthor_gpu_irq_suspend (&ptdev->gpu->irq); +} + +/** +* panthor_gpu_resume() - Resume the GPU block. +* @ptdev: Device. +* +* Resume the IRQ handler and power-on the L2-cache. +* The FW takes care of powering the other blocks. +*/ +void panthor_gpu_resume (struct panthor_device *ptdev) +{ +panthor_gpu_irq_resume (&ptdev->gpu->irq,GPU_INTERRUPTS_MASK ); +panthor_gpu_l2_power_on (ptdev); +}
drivers/gpu/drm/panthor/panthor_gpu.h new file +52 -0
52 lines not on this page · raw message
Signature · 1 line
2.43.0
Steven Price
maintainerRe: [PATCH v3 04/14] drm/panthor: Add the GPU logical block
Quoted 19 lines · Boris Brezillon
Handles everything that's not related to the FW, the MMU or the scheduler. This is the block dealing with the GPU property retrieval, the GPU block power on/off logic, and some global operations, like global cache flushing.
v3:
- Add acks for the MIT/GPL2 relicensing
- Use macros to extract GPU ID info
- Make sure we reset clearpending_reqsbits whenwait_event_timeout()times out but the corresponding bit is cleared inGPU_INT_RAWSTAT(can happen if the IRQ is masked or HW takes to long to call the IRQ handler)
-GPU_MODELnow takes separate arch and product majors to be more readable.
- DropGPU_IRQ_MCU_STATUS_CHANGEDfrom interrupt mask.
- HandleGPU_IRQ_PROTM_FAULTcorrectly (don't output registers that are not updated for protected interrupts).
- Minor code tidy ups
Quoted 6 lines
Cc Alexey Sheplyakov <[email protected]> # MIT+GPL2 relicensingSigned-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>
LGTM!
Reviewed: Steven Price <[email protected]>
Quoted 552 lines
--- drivers/gpu/drm/panthor/panthor_gpu.c | 481 ++++++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_gpu.h | 52 +++ 2 files changed, 533 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_gpu.c create mode 100644 drivers/gpu/drm/panthor/panthor_gpu.hdrivers/gpu/drm/panthor/panthor_gpu.c new file +481
-0@@ -0,0 +1,481 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2018 Marty E. Plummer <[email protected]> */ +/* Copyright 2019 Linaro, Ltd., Rob Herring <[email protected]> */ +/* Copyright 2019 Collabora ltd. */ + +#include <linux/bitfield.h> +#include <linux/bitmap.h> +#include <linux/delay.h> +#include <linux/dma-mapping.h> +#include <linux/interrupt.h> +#include <linux/io.h> +#include <linux/iopoll.h> +#include <linux/platform_device.h> +#include <linux/pm_runtime.h> + +#include <drm/drm_drv.h> +#include <drm/drm_managed.h> + +#include "panthor_device.h" +#include "panthor_gpu.h" +#include "panthor_regs.h" + +/** +* struct panthor_gpu - GPU block management data. +*/ +struct panthor_gpu { +/** @irq: GPU irq. */ +struct panthor_irq irq; + +/** @reqs_lock: Lock protecting access to pending_reqs. */ +spinlock_t reqs_lock; + +/** @pending_reqs: Pending GPU requests. */ +u32 pending_reqs; + +/** @reqs_acked: GPU request wait queue. */ +wait_queue_head_t reqs_acked; +}; + +/** +* struct panthor_model - GPU model description +*/ +struct panthor_model { +/** @name: Model name. */ +const char *name; + +/** @arch_major: Major version number of architecture. */ +u8 arch_major; + +/** @product_major: Major version number of product. */ +u8 product_major; +}; + +/** +* GPU_MODEL() - Define a GPU model. A GPU product can be uniquely identified +* by a combination of the major architecture version and the major product +* version. +* @name: Name for the GPU model. +* @_arch_major: Architecture major. +* @_product_major: Product major. +*/ +#define GPU_MODEL (_name, _arch_major, _product_major) \ +{\ + .name = __stringify(_name), \ + .arch_major = _arch_major, \ + .product_major = _product_major, \ +} + +static const struct panthor_model gpu_models[] = { +GPU_MODEL (g610,10 ,7 ), + {}, +}; + +#define GPU_INTERRUPTS_MASK \ + (GPU_IRQ_FAULT | \ + GPU_IRQ_PROTM_FAULT | \ + GPU_IRQ_RESET_COMPLETED | \ + GPU_IRQ_CLEAN_CACHES_COMPLETED) + +static void panthor_gpu_init_info (struct panthor_device *ptdev) +{ +const struct panthor_model *model; +u32 arch_major, product_major; +u32 major, minor, status; +unsigned int i; + + ptdev->gpu_info.gpu_id =gpu_read (ptdev,GPU_ID ); + ptdev->gpu_info.csf_id =gpu_read (ptdev,GPU_CSF_ID ); + ptdev->gpu_info.gpu_rev =gpu_read (ptdev,GPU_REVID ); + ptdev->gpu_info.l2_features =gpu_read (ptdev,GPU_L2_FEATURES ); + ptdev->gpu_info.tiler_features =gpu_read (ptdev,GPU_TILER_FEATURES ); + ptdev->gpu_info.mem_features =gpu_read (ptdev,GPU_MEM_FEATURES ); + ptdev->gpu_info.mmu_features =gpu_read (ptdev,GPU_MMU_FEATURES ); + ptdev->gpu_info.thread_features =gpu_read (ptdev,GPU_THREAD_FEATURES ); + ptdev->gpu_info.max_threads =gpu_read (ptdev,GPU_THREAD_MAX_THREADS ); + ptdev->gpu_info.thread_max_workgroup_size =gpu_read (ptdev,GPU_THREAD_MAX_WORKGROUP_SIZE ); + ptdev->gpu_info.thread_max_barrier_size =gpu_read (ptdev,GPU_THREAD_MAX_BARRIER_SIZE ); + ptdev->gpu_info.coherency_features =gpu_read (ptdev,GPU_COHERENCY_FEATURES ); +for (i =0 ; i <4 ; i++) + ptdev->gpu_info.texture_features[i] =gpu_read (ptdev,GPU_TEXTURE_FEATURES (i)); + + ptdev->gpu_info.as_present =gpu_read (ptdev,GPU_AS_PRESENT ); + + ptdev->gpu_info.shader_present =gpu_read (ptdev,GPU_SHADER_PRESENT_LO ); + ptdev->gpu_info.shader_present |= (u64 )gpu_read (ptdev,GPU_SHADER_PRESENT_HI ) <<32 ; + + ptdev->gpu_info.tiler_present =gpu_read (ptdev,GPU_TILER_PRESENT_LO ); + ptdev->gpu_info.tiler_present |= (u64 )gpu_read (ptdev,GPU_TILER_PRESENT_HI ) <<32 ; + + ptdev->gpu_info.l2_present =gpu_read (ptdev,GPU_L2_PRESENT_LO ); + ptdev->gpu_info.l2_present |= (u64 )gpu_read (ptdev,GPU_L2_PRESENT_HI ) <<32 ; + + arch_major =GPU_ARCH_MAJOR (ptdev->gpu_info.gpu_id); + product_major =GPU_PROD_MAJOR (ptdev->gpu_info.gpu_id); + major =GPU_VER_MAJOR (ptdev->gpu_info.gpu_id); + minor =GPU_VER_MINOR (ptdev->gpu_info.gpu_id); + status =GPU_VER_STATUS (ptdev->gpu_info.gpu_id); + +for (model = gpu_models; model->name; model++) { +if (model->arch_major == arch_major && + model->product_major == product_major) +break ; + } + +drm_info (&ptdev->base, +"mali-%s id 0x%x major 0x%x minor 0x%x status 0x%x" , + model->name ?:"unknown" , ptdev->gpu_info.gpu_id >>16 , + major, minor, status); + +drm_info (&ptdev->base, +"Features: L2:%#x Tiler:%#x Mem:%#x MMU:%#x AS:%#x" , + ptdev->gpu_info.l2_features, + ptdev->gpu_info.tiler_features, + ptdev->gpu_info.mem_features, + ptdev->gpu_info.mmu_features, + ptdev->gpu_info.as_present); + +drm_info (&ptdev->base, +"shader_present=0x%0llx l2_present=0x%0llx tiler_present=0x%0llx" , + ptdev->gpu_info.shader_present, ptdev->gpu_info.l2_present, + ptdev->gpu_info.tiler_present); +} + +static void panthor_gpu_irq_handler (struct panthor_device *ptdev,u32 status) +{ +if (status &GPU_IRQ_FAULT ) { +u32 fault_status =gpu_read (ptdev,GPU_FAULT_STATUS ); +u64 address = ((u64 )gpu_read (ptdev,GPU_FAULT_ADDR_HI ) <<32 ) | +gpu_read (ptdev,GPU_FAULT_ADDR_LO ); + +drm_warn (&ptdev->base,"GPU Fault 0x%08x (%s) at 0x%016llx\n" , + fault_status,panthor_exception_name (ptdev, fault_status &0xFF ), + address); + } +if (status &GPU_IRQ_PROTM_FAULT ) +drm_warn (&ptdev->base,"GPU Fault in protected mode\n" ); + +spin_lock (&ptdev->gpu->reqs_lock); +if (status & ptdev->gpu->pending_reqs) { + ptdev->gpu->pending_reqs &= ~status; +wake_up_all (&ptdev->gpu->reqs_acked); + } +spin_unlock (&ptdev->gpu->reqs_lock); +} +PANTHOR_IRQ_HANDLER (gpu,GPU , panthor_gpu_irq_handler); + +/** +* panthor_gpu_unplug() - Called when the GPU is unplugged. +* @ptdev: Device to unplug. +*/ +void panthor_gpu_unplug (struct panthor_device *ptdev) +{ +unsigned long flags; + +/* Make sure the IRQ handler is not running after that point. */ +panthor_gpu_irq_suspend (&ptdev->gpu->irq); + +/* Wake-up all waiters. */ +spin_lock_irqsave (&ptdev->gpu->reqs_lock, flags); + ptdev->gpu->pending_reqs =0 ; +wake_up_all (&ptdev->gpu->reqs_acked); +spin_unlock_irqrestore (&ptdev->gpu->reqs_lock, flags); +} + +/** +* panthor_gpu_init() - Initialize the GPU block +* @ptdev: Device. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_init (struct panthor_device *ptdev) +{ +struct panthor_gpu *gpu; +u32 pa_bits; +int ret, irq; + + gpu =drmm_kzalloc (&ptdev->base,sizeof (*gpu),GFP_KERNEL ); +if (!gpu) +return -ENOMEM ; + +spin_lock_init (&gpu->reqs_lock); +init_waitqueue_head (&gpu->reqs_acked); + ptdev->gpu = gpu; +panthor_gpu_init_info (ptdev); + +dma_set_max_seg_size (ptdev->base.dev,UINT_MAX ); + pa_bits =GPU_MMU_FEATURES_PA_BITS (ptdev->gpu_info.mmu_features); + ret =dma_set_mask_and_coherent (ptdev->base.dev,DMA_BIT_MASK (pa_bits)); +if (ret) +return ret; + + irq =platform_get_irq_byname (to_platform_device (ptdev->base.dev),"gpu" ); +if (irq <=0 ) +return ret; + + ret =panthor_request_gpu_irq (ptdev, &ptdev->gpu->irq, irq,GPU_INTERRUPTS_MASK ); +if (ret) +return ret; + +return 0 ; +} + +/** +* panthor_gpu_block_power_off() - Power-off a specific block of the GPU +* @ptdev: Device. +* @blk_name: Block name. +* @pwroff_reg: Power-off register for this block. +* @pwrtrans_reg: Power transition register for this block. +* @mask: Sub-elements to power-off. +* @timeout_us: Timeout in microseconds. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_block_power_off (struct panthor_device *ptdev, +const char *blk_name, +u32 pwroff_reg,u32 pwrtrans_reg, +u64 mask,u32 timeout_us) +{ +u32 val, i; +int ret; + +for (i =0 ; i <2 ; i++) { +u32 mask32 = mask >> (i *32 ); + +if (!mask32) +continue ; + + ret =readl_relaxed_poll_timeout (ptdev->iomem + pwrtrans_reg + (i *4 ), + val, !(mask32 & val), +100 , timeout_us); +if (ret) { +drm_err (&ptdev->base,"timeout waiting on %s:%llx power transition" , + blk_name, mask); +return ret; + } + } + +if (mask &GENMASK (31 ,0 )) +gpu_write (ptdev, pwroff_reg, mask); + +if (mask >>32 ) +gpu_write (ptdev, pwroff_reg +4 , mask >>32 ); + +for (i =0 ; i <2 ; i++) { +u32 mask32 = mask >> (i *32 ); + +if (!mask32) +continue ; + + ret =readl_relaxed_poll_timeout (ptdev->iomem + pwrtrans_reg + (i *4 ), + val, !(mask & val), +100 , timeout_us); +if (ret) { +drm_err (&ptdev->base,"timeout waiting on %s:%llx power transition" , + blk_name, mask); +return ret; + } + } + +return 0 ; +} + +/** +* panthor_gpu_block_power_on() - Power-on a specific block of the GPU +* @ptdev: Device. +* @blk_name: Block name. +* @pwron_reg: Power-on register for this block. +* @pwrtrans_reg: Power transition register for this block. +* @rdy_reg: Power transition ready register. +* @mask: Sub-elements to power-on. +* @timeout_us: Timeout in microseconds. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_block_power_on (struct panthor_device *ptdev, +const char *blk_name, +u32 pwron_reg,u32 pwrtrans_reg, +u32 rdy_reg,u64 mask,u32 timeout_us) +{ +u32 val, i; +int ret; + +for (i =0 ; i <2 ; i++) { +u32 mask32 = mask >> (i *32 ); + +if (!mask32) +continue ; + + ret =readl_relaxed_poll_timeout (ptdev->iomem + pwrtrans_reg + (i *4 ), + val, !(mask32 & val), +100 , timeout_us); +if (ret) { +drm_err (&ptdev->base,"timeout waiting on %s:%llx power transition" , + blk_name, mask); +return ret; + } + } + +if (mask &GENMASK (31 ,0 )) +gpu_write (ptdev, pwron_reg, mask); + +if (mask >>32 ) +gpu_write (ptdev, pwron_reg +4 , mask >>32 ); + +for (i =0 ; i <2 ; i++) { +u32 mask32 = mask >> (i *32 ); + +if (!mask32) +continue ; + + ret =readl_relaxed_poll_timeout (ptdev->iomem + rdy_reg + (i *4 ), + val, (mask32 & val) == mask32, +100 , timeout_us); +if (ret) { +drm_err (&ptdev->base,"timeout waiting on %s:%llx readyness" , + blk_name, mask); +return ret; + } + } + +return 0 ; +} + +/** +* panthor_gpu_l2_power_on() - Power-on the L2-cache +* @ptdev: Device. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_l2_power_on (struct panthor_device *ptdev) +{ +if (ptdev->gpu_info.l2_present !=1 ) { +/* +* Only support one core group now. +* ~(l2_present - 1) unsets all bits in l2_present except +* the bottom bit. (l2_present - 2) has all the bits in +* the first core group set. AND them together to generate +* a mask of cores in the first core group. +*/ +u64 core_mask = ~(ptdev->gpu_info.l2_present -1 ) & + (ptdev->gpu_info.l2_present -2 ); +drm_info_once (&ptdev->base,"using only 1st core group (%lu cores from %lu)\n" , +hweight64 (core_mask), +hweight64 (ptdev->gpu_info.shader_present)); + } + +return panthor_gpu_power_on (ptdev,L2 ,1 ,20000 ); +} + +/** +* panthor_gpu_flush_caches() - Flush caches +* @ptdev: Device. +* @l2: L2 flush type. +* @lsc: LSC flush type. +* @other: Other flush type. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_flush_caches (struct panthor_device *ptdev, +u32 l2,u32 lsc,u32 other) +{ +bool timedout = false; +unsigned long flags; + +spin_lock_irqsave (&ptdev->gpu->reqs_lock, flags); +if (!drm_WARN_ON (&ptdev->base, + ptdev->gpu->pending_reqs &GPU_IRQ_CLEAN_CACHES_COMPLETED )) { + ptdev->gpu->pending_reqs |=GPU_IRQ_CLEAN_CACHES_COMPLETED ; +gpu_write (ptdev,GPU_CMD ,GPU_FLUSH_CACHES (l2, lsc, other)); + } +spin_unlock_irqrestore (&ptdev->gpu->reqs_lock, flags); + +if (!wait_event_timeout (ptdev->gpu->reqs_acked, + !(ptdev->gpu->pending_reqs &GPU_IRQ_CLEAN_CACHES_COMPLETED ), +msecs_to_jiffies (100 ))) { +spin_lock_irqsave (&ptdev->gpu->reqs_lock, flags); +if ((ptdev->gpu->pending_reqs &GPU_IRQ_CLEAN_CACHES_COMPLETED ) !=0 && + !(gpu_read (ptdev,GPU_INT_RAWSTAT ) &GPU_IRQ_CLEAN_CACHES_COMPLETED )) + timedout = true; +else + ptdev->gpu->pending_reqs &= ~GPU_IRQ_CLEAN_CACHES_COMPLETED ; +spin_unlock_irqrestore (&ptdev->gpu->reqs_lock, flags); + } + +if (timedout) { +drm_err (&ptdev->base,"Flush caches timeout" ); +return -ETIMEDOUT ; + } + +return 0 ; +} + +/** +* panthor_gpu_soft_reset() - Issue a soft-reset +* @ptdev: Device. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_gpu_soft_reset (struct panthor_device *ptdev) +{ +bool timedout = false; +unsigned long flags; + +spin_lock_irqsave (&ptdev->gpu->reqs_lock, flags); +if (!drm_WARN_ON (&ptdev->base, + ptdev->gpu->pending_reqs &GPU_IRQ_RESET_COMPLETED )) { + ptdev->gpu->pending_reqs |=GPU_IRQ_RESET_COMPLETED ; +gpu_write (ptdev,GPU_INT_CLEAR ,GPU_IRQ_RESET_COMPLETED ); +gpu_write (ptdev,GPU_CMD ,GPU_SOFT_RESET ); + } +spin_unlock_irqrestore (&ptdev->gpu->reqs_lock, flags); + +if (!wait_event_timeout (ptdev->gpu->reqs_acked, + !(ptdev->gpu->pending_reqs &GPU_IRQ_RESET_COMPLETED ), +msecs_to_jiffies (100 ))) { +spin_lock_irqsave (&ptdev->gpu->reqs_lock, flags); +if ((ptdev->gpu->pending_reqs &GPU_IRQ_RESET_COMPLETED ) !=0 && + !(gpu_read (ptdev,GPU_INT_RAWSTAT ) &GPU_IRQ_RESET_COMPLETED )) + timedout = true; +else + ptdev->gpu->pending_reqs &= ~GPU_IRQ_RESET_COMPLETED ; +spin_unlock_irqrestore (&ptdev->gpu->reqs_lock, flags); + } + +if (timedout) { +drm_err (&ptdev->base,"Soft reset timeout" ); +return -ETIMEDOUT ; + } + +return 0 ; +} + +/** +* panthor_gpu_suspend() - Suspend the GPU block. +* @ptdev: Device. +* +* Suspend the GPU irq. This should be called last in the suspend procedure, +* after all other blocks have been suspented. +*/ +void panthor_gpu_suspend (struct panthor_device *ptdev) +{ +/* +* It may be preferable to simply power down the L2, but for now just +* soft-reset which will leave the L2 powered down. +*/ +panthor_gpu_soft_reset (ptdev); +panthor_gpu_irq_suspend (&ptdev->gpu->irq); +} + +/** +* panthor_gpu_resume() - Resume the GPU block. +* @ptdev: Device. +* +* Resume the IRQ handler and power-on the L2-cache. +* The FW takes care of powering the other blocks. +*/ +void panthor_gpu_resume (struct panthor_device *ptdev) +{ +panthor_gpu_irq_resume (&ptdev->gpu->irq,GPU_INTERRUPTS_MASK ); +panthor_gpu_l2_power_on (ptdev); +}drivers/gpu/drm/panthor/panthor_gpu.h new file +52
-0@@ -0,0 +1,52 @@ +/* SPDX-License-Identifier: GPL-2.0 or MIT */ +/* Copyright 2018 Marty E. Plummer <[email protected]> */ +/* Copyright 2019 Collabora ltd. */ + +#ifndef __PANTHOR_GPU_H__ +#define __PANTHOR_GPU_H__ + +struct panthor_device ; + +int panthor_gpu_init (struct panthor_device *ptdev); +void panthor_gpu_unplug (struct panthor_device *ptdev); +void panthor_gpu_suspend (struct panthor_device *ptdev); +void panthor_gpu_resume (struct panthor_device *ptdev); + +int panthor_gpu_block_power_on (struct panthor_device *ptdev, +const char *blk_name, +u32 pwron_reg,u32 pwrtrans_reg, +u32 rdy_reg,u64 mask,u32 timeout_us); +int panthor_gpu_block_power_off (struct panthor_device *ptdev, +const char *blk_name, +u32 pwroff_reg,u32 pwrtrans_reg, +u64 mask,u32 timeout_us); + +/** +* panthor_gpu_power_on() - Power on the GPU block. +* +* Return: 0 on success, a negative error code otherwise. +*/ +#define panthor_gpu_power_on (ptdev, type, mask, timeout_us) \ + panthor_gpu_block_power_on(ptdev, #type, \ + type ## _PWRON_LO, \ + type ## _PWRTRANS_LO, \ + type ## _READY_LO, \ + mask, timeout_us) + +/** +* panthor_gpu_power_off() - Power off the GPU block. +* +* Return: 0 on success, a negative error code otherwise. +*/ +#define panthor_gpu_power_off (ptdev, type, mask, timeout_us) \ + panthor_gpu_block_power_off(ptdev, #type, \ + type ## _PWROFF_LO, \ + type ## _PWRTRANS_LO, \ + mask, timeout_us) + +int panthor_gpu_l2_power_on (struct panthor_device *ptdev); +int panthor_gpu_flush_caches (struct panthor_device *ptdev, +u32 l2,u32 lsc,u32 other); +int panthor_gpu_soft_reset (struct panthor_device *ptdev); + +#endif
Boris Brezillon
maintainerPATCH v3 5/14 drm/panthor: Add GEM logical block
In reply to Boris BrezillonAnything relating to GEM object management is placed here. Nothing particularly interesting here, given the implementation is based on drm_gem_shmem_object, which is doing most of the work.
v3:
- Add acks for the MIT/GPL2 relicensing
- Provide a panthor_kernel_bo abstraction for buffer objects managed by the kernel (will replace panthor_fw_mem and be used everywhere we were using panthor_gem_create_and_map() before)
- Adjust things to match drm_gpuvm changes
- Change return of panthor_gem_create_with_handle() to int
--- drivers/gpu/drm/panthor/panthor_gem.c | 227 ++++++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_gem.h | 144 ++++++++++++++++ 2 files changed, 371 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_gem.c create mode 100644 drivers/gpu/drm/panthor/panthor_gem.h
drivers/gpu/drm/panthor/panthor_gem.c new file +227 -0
@@ -0,0 +1,227 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ + +#include <linux/err.h> +#include <linux/slab.h> +#include <linux/dma-buf.h> +#include <linux/dma-mapping.h> + +#include <drm/panthor_drm.h> + +#include "panthor_device.h" +#include "panthor_gem.h" +#include "panthor_mmu.h" + +static void panthor_gem_free_object (struct drm_gem_object *obj) +{ +struct panthor_gem_object *bo =to_panthor_bo (obj); +struct drm_gem_object *vm_root_gem = bo->exclusive_vm_root_gem; + +drm_gem_free_mmap_offset (&bo->base.base); +mutex_destroy (&bo->gpuva_list_lock); +drm_gem_shmem_free (&bo->base); +drm_gem_object_put (vm_root_gem); +} + +/** +* panthor_kernel_bo_destroy() - Destroy a kernel buffer object +* @vm: The VM this BO was mapped to. +* @bo: Kernel buffer object to destroy. +*/ +void panthor_kernel_bo_destroy (struct panthor_vm *vm, +struct panthor_kernel_bo *bo) +{ +int ret; + +panthor_kernel_bo_vunmap (bo); + +if (drm_WARN_ON (bo->obj->dev, +to_panthor_bo (bo->obj)->exclusive_vm_root_gem !=panthor_vm_root_gem (vm))) + goto out_free_bo; + +if (!vm) + goto out_free_bo; + + ret =panthor_vm_unmap_range (vm, bo->va_node.start, +panthor_kernel_bo_size (bo)); +if (ret) + goto out_free_bo; + +panthor_vm_free_va (vm, &bo->va_node); +drm_gem_object_put (bo->obj); + +out_free_bo: +kfree (bo); +} + +/** +* panthor_kernel_bo_create() - Create and map a GEM object to a VM +* @ptdev: Device. +* @vm: VM to map the GEM to. If NULL, the kernel object is not GPU mapped. +* @size: Size of the buffer object. +* @bo_flags: Combination of drm_panthor_bo_flags flags. +* @vm_map_flags: Combination of drm_panthor_vm_bind_op_flags (only those +* that are related to map operations). +* @gpu_va: GPU address assigned when mapping to the VM. +* If gpu_va == PANTHOR_VM_KERNEL_AUTO_VA, the virtual address will be +* automatically allocated. +* +* Return: A valid pointer in case of success, an ERR_PTR() otherwise. +*/ +struct panthor_kernel_bo * +panthor_kernel_bo_create (struct panthor_device *ptdev,struct panthor_vm *vm, +size_t size,u32 bo_flags,u32 vm_map_flags, +u64 gpu_va) +{ +struct drm_gem_shmem_object *obj; +struct panthor_kernel_bo *kbo; +struct panthor_gem_object *bo; +int ret; + + kbo =kzalloc (sizeof (*kbo),GFP_KERNEL ); +if (!kbo) +return ERR_PTR (-ENOMEM ); + + obj =drm_gem_shmem_create (&ptdev->base, size); +if (IS_ERR (obj)) { + ret =PTR_ERR (obj); + goto err_free_bo; + } + + bo =to_panthor_bo (&obj->base); + size = obj->base.size; + kbo->obj = &obj->base; + bo->flags = bo_flags; + +if (!vm) +return 0 ; + + ret =panthor_vm_alloc_va (vm, gpu_va, size, &kbo->va_node); +if (ret) + goto err_put_obj; + + ret =panthor_vm_map_bo_range (vm, bo,0 , size, kbo->va_node.start, vm_map_flags); +if (ret) + goto err_free_va; + + bo->exclusive_vm_root_gem =panthor_vm_root_gem (vm); +drm_gem_object_get (bo->exclusive_vm_root_gem); + bo->base.base.resv = bo->exclusive_vm_root_gem->resv; +return kbo; + +err_free_va: +panthor_vm_free_va (vm, &kbo->va_node); + +err_put_obj: +drm_gem_object_put (&obj->base); + +err_free_bo: +kfree (kbo); +return ERR_PTR (ret); +} + +static int panthor_gem_mmap (struct drm_gem_object *obj,struct vm_area_struct *vma) +{ +struct panthor_gem_object *bo =to_panthor_bo (obj); + +/* Don't allow mmap on objects that have the NO_MMAP flag set. */ +if (bo->flags &DRM_PANTHOR_BO_NO_MMAP ) +return -EINVAL ; + +return drm_gem_shmem_object_mmap (obj, vma); +} + +static struct dma_buf * +panthor_gem_prime_export (struct drm_gem_object *obj,int flags) +{ +/* We can't export GEMs that have an exclusive VM. */ +if (to_panthor_bo (obj)->exclusive_vm_root_gem) +return ERR_PTR (-EINVAL ); + +return drm_gem_prime_export (obj, flags); +} + +static const struct drm_gem_object_funcs panthor_gem_funcs = { + .free = panthor_gem_free_object, + .print_info = drm_gem_shmem_object_print_info, + .pin = drm_gem_shmem_object_pin, + .unpin = drm_gem_shmem_object_unpin, + .get_sg_table = drm_gem_shmem_object_get_sg_table, + .vmap = drm_gem_shmem_object_vmap, + .vunmap = drm_gem_shmem_object_vunmap, + .mmap = panthor_gem_mmap, + .export = panthor_gem_prime_export, + .vm_ops = &drm_gem_shmem_vm_ops, +}; + +/** +* panthor_gem_create_object - Implementation of driver->gem_create_object. +* @ddev: DRM device +* @size: Size in bytes of the memory the object will reference +* +* This lets the GEM helpers allocate object structs for us, and keep +* our BO stats correct. +*/ +struct drm_gem_object *panthor_gem_create_object (struct drm_device *ddev,size_t size) +{ + struct panthor_device *ptdev = container_of(ddev, struct panthor_device, base); +struct panthor_gem_object *obj; + + obj =kzalloc (sizeof (*obj),GFP_KERNEL ); +if (!obj) +return ERR_PTR (-ENOMEM ); + + obj->base.base.funcs = &panthor_gem_funcs; + obj->base.map_wc = !ptdev->coherent; +mutex_init (&obj->gpuva_list_lock); +drm_gem_gpuva_set_lock (&obj->base.base, &obj->gpuva_list_lock); + +return &obj->base.base; +} + +/** +* panthor_gem_create_with_handle() - Create a GEM object and attach it to a handle. +* @file: DRM file. +* @ddev: DRM device. +* @exclusive_vm: Exclusive VM. Not NULL if the GEM object can't be shared. +* @size: Size of the GEM object to allocate. +* @flags: Combination of drm_panthor_bo_flags flags. +* @handle: Pointer holding the handle pointing to the new GEM object. +* +* Return: Zero on success +*/ +int +panthor_gem_create_with_handle (struct drm_file *file, +struct drm_device *ddev, +struct panthor_vm *exclusive_vm, +size_t size, +u32 flags,u32 *handle) +{ +int ret; +struct drm_gem_shmem_object *shmem; +struct panthor_gem_object *bo; + + shmem =drm_gem_shmem_create (ddev, size); +if (IS_ERR (shmem)) +return PTR_ERR (shmem); + + bo =to_panthor_bo (&shmem->base); + bo->flags = flags; + +if (exclusive_vm) { + bo->exclusive_vm_root_gem =panthor_vm_root_gem (exclusive_vm); +drm_gem_object_get (bo->exclusive_vm_root_gem); + bo->base.base.resv = bo->exclusive_vm_root_gem->resv; + } + +/* +* Allocate an id of idr table where the obj is registered +* and handle has the id what user can see. +*/ + ret =drm_gem_handle_create (file, &shmem->base, handle); +/* drop reference from allocate - handle holds it now. */ +drm_gem_object_put (&shmem->base); + +return ret; +}
drivers/gpu/drm/panthor/panthor_gem.h new file +144 -0
144 lines not on this page · raw message
Signature · 1 line
2.43.0
Steven Price
maintainerRe: [PATCH v3 05/14] drm/panthor: Add GEM logical block
Quoted 12 lines · Boris Brezillon
Anything relating to GEM object management is placed here. Nothing particularly interesting here, given the implementation is based on
drm_gem_shmem_object, which is doing most of the work.v3:
- Add acks for the MIT/GPL2 relicensing
- Provide apanthor_kernel_boabstraction for buffer objects managed by the kernel (will replacepanthor_fw_memand be used everywhere we were usingpanthor_gem_create_and_map()before)
- Adjust things to matchdrm_gpuvmchanges
- Change return ofpanthor_gem_create_with_handle()to int
Quoted 5 lines
Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>
Looks generally good, but see below about the 'vm' argument to the kernel_bo functions.
Quoted 47 lines
--- drivers/gpu/drm/panthor/panthor_gem.c | 227 ++++++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_gem.h | 144 ++++++++++++++++ 2 files changed, 371 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_gem.c create mode 100644 drivers/gpu/drm/panthor/panthor_gem.hdrivers/gpu/drm/panthor/panthor_gem.c new file +34
-0@@ -0,0 +1,227 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ + +#include <linux/err.h> +#include <linux/slab.h> +#include <linux/dma-buf.h> +#include <linux/dma-mapping.h> + +#include <drm/panthor_drm.h> + +#include "panthor_device.h" +#include "panthor_gem.h" +#include "panthor_mmu.h" + +static void panthor_gem_free_object (struct drm_gem_object *obj) +{ +struct panthor_gem_object *bo =to_panthor_bo (obj); +struct drm_gem_object *vm_root_gem = bo->exclusive_vm_root_gem; + +drm_gem_free_mmap_offset (&bo->base.base); +mutex_destroy (&bo->gpuva_list_lock); +drm_gem_shmem_free (&bo->base); +drm_gem_object_put (vm_root_gem); +} + +/** +* panthor_kernel_bo_destroy() - Destroy a kernel buffer object +* @vm: The VM this BO was mapped to. +* @bo: Kernel buffer object to destroy. +*/ +void panthor_kernel_bo_destroy (struct panthor_vm *vm, +struct panthor_kernel_bo *bo) +{
drivers/gpu/drm/panthor/panthor_gem.c line 44
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_gem.c new file +10
-0+int ret; + +panthor_kernel_bo_vunmap (bo); + +if (drm_WARN_ON (bo->obj->dev, +to_panthor_bo (bo->obj)->exclusive_vm_root_gem !=panthor_vm_root_gem (vm))) + goto out_free_bo; + +if (!vm) + goto out_free_bo;
Don't we want to call drm_gem_object_put() in this case? Although I'm not sure I've entirely got my head around the !vm case yet. I'm a little suspicious it's left over?
Quoted 44 lines
drivers/gpu/drm/panthor/panthor_gem.c new file +44
-0+ + ret =panthor_vm_unmap_range (vm, bo->va_node.start, +panthor_kernel_bo_size (bo)); +if (ret) + goto out_free_bo; + +panthor_vm_free_va (vm, &bo->va_node); +drm_gem_object_put (bo->obj); + +out_free_bo: +kfree (bo); +} + +/** +* panthor_kernel_bo_create() - Create and map a GEM object to a VM +* @ptdev: Device. +* @vm: VM to map the GEM to. If NULL, the kernel object is not GPU mapped. +* @size: Size of the buffer object. +* @bo_flags: Combination of drm_panthor_bo_flags flags. +* @vm_map_flags: Combination of drm_panthor_vm_bind_op_flags (only those +* that are related to map operations). +* @gpu_va: GPU address assigned when mapping to the VM. +* If gpu_va == PANTHOR_VM_KERNEL_AUTO_VA, the virtual address will be +* automatically allocated. +* +* Return: A valid pointer in case of success, an ERR_PTR() otherwise. +*/ +struct panthor_kernel_bo * +panthor_kernel_bo_create (struct panthor_device *ptdev,struct panthor_vm *vm, +size_t size,u32 bo_flags,u32 vm_map_flags, +u64 gpu_va) +{ +struct drm_gem_shmem_object *obj; +struct panthor_kernel_bo *kbo; +struct panthor_gem_object *bo; +int ret; + + kbo =kzalloc (sizeof (*kbo),GFP_KERNEL ); +if (!kbo) +return ERR_PTR (-ENOMEM ); + + obj =drm_gem_shmem_create (&ptdev->base, size); +if (IS_ERR (obj)) { + ret =PTR_ERR (obj);
drivers/gpu/drm/panthor/panthor_gem.c line 98
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_gem.c new file +10
-0+ goto err_free_bo; + } + + bo =to_panthor_bo (&obj->base); + size = obj->base.size; + kbo->obj = &obj->base; + bo->flags = bo_flags; + +if (!vm) +return 0 ;
This doesn't look right - I'd expect kbo to be returned? (Or an error if !vm isn't actually supported).
I've had a look at the rest of the driver and I can't find a user of the !vm case. So either I'm missing something (quite plausible) or we should just make the vm argument compulsory and simplify this a bit.
Steve
Quoted 279 lines
drivers/gpu/drm/panthor/panthor_gem.c new file +129
-0+ + ret =panthor_vm_alloc_va (vm, gpu_va, size, &kbo->va_node); +if (ret) + goto err_put_obj; + + ret =panthor_vm_map_bo_range (vm, bo,0 , size, kbo->va_node.start, vm_map_flags); +if (ret) + goto err_free_va; + + bo->exclusive_vm_root_gem =panthor_vm_root_gem (vm); +drm_gem_object_get (bo->exclusive_vm_root_gem); + bo->base.base.resv = bo->exclusive_vm_root_gem->resv; +return kbo; + +err_free_va: +panthor_vm_free_va (vm, &kbo->va_node); + +err_put_obj: +drm_gem_object_put (&obj->base); + +err_free_bo: +kfree (kbo); +return ERR_PTR (ret); +} + +static int panthor_gem_mmap (struct drm_gem_object *obj,struct vm_area_struct *vma) +{ +struct panthor_gem_object *bo =to_panthor_bo (obj); + +/* Don't allow mmap on objects that have the NO_MMAP flag set. */ +if (bo->flags &DRM_PANTHOR_BO_NO_MMAP ) +return -EINVAL ; + +return drm_gem_shmem_object_mmap (obj, vma); +} + +static struct dma_buf * +panthor_gem_prime_export (struct drm_gem_object *obj,int flags) +{ +/* We can't export GEMs that have an exclusive VM. */ +if (to_panthor_bo (obj)->exclusive_vm_root_gem) +return ERR_PTR (-EINVAL ); + +return drm_gem_prime_export (obj, flags); +} + +static const struct drm_gem_object_funcs panthor_gem_funcs = { + .free = panthor_gem_free_object, + .print_info = drm_gem_shmem_object_print_info, + .pin = drm_gem_shmem_object_pin, + .unpin = drm_gem_shmem_object_unpin, + .get_sg_table = drm_gem_shmem_object_get_sg_table, + .vmap = drm_gem_shmem_object_vmap, + .vunmap = drm_gem_shmem_object_vunmap, + .mmap = panthor_gem_mmap, + .export = panthor_gem_prime_export, + .vm_ops = &drm_gem_shmem_vm_ops, +}; + +/** +* panthor_gem_create_object - Implementation of driver->gem_create_object. +* @ddev: DRM device +* @size: Size in bytes of the memory the object will reference +* +* This lets the GEM helpers allocate object structs for us, and keep +* our BO stats correct. +*/ +struct drm_gem_object *panthor_gem_create_object (struct drm_device *ddev,size_t size) +{ + struct panthor_device *ptdev = container_of(ddev, struct panthor_device, base); +struct panthor_gem_object *obj; + + obj =kzalloc (sizeof (*obj),GFP_KERNEL ); +if (!obj) +return ERR_PTR (-ENOMEM ); + + obj->base.base.funcs = &panthor_gem_funcs; + obj->base.map_wc = !ptdev->coherent; +mutex_init (&obj->gpuva_list_lock); +drm_gem_gpuva_set_lock (&obj->base.base, &obj->gpuva_list_lock); + +return &obj->base.base; +} + +/** +* panthor_gem_create_with_handle() - Create a GEM object and attach it to a handle. +* @file: DRM file. +* @ddev: DRM device. +* @exclusive_vm: Exclusive VM. Not NULL if the GEM object can't be shared. +* @size: Size of the GEM object to allocate. +* @flags: Combination of drm_panthor_bo_flags flags. +* @handle: Pointer holding the handle pointing to the new GEM object. +* +* Return: Zero on success +*/ +int +panthor_gem_create_with_handle (struct drm_file *file, +struct drm_device *ddev, +struct panthor_vm *exclusive_vm, +size_t size, +u32 flags,u32 *handle) +{ +int ret; +struct drm_gem_shmem_object *shmem; +struct panthor_gem_object *bo; + + shmem =drm_gem_shmem_create (ddev, size); +if (IS_ERR (shmem)) +return PTR_ERR (shmem); + + bo =to_panthor_bo (&shmem->base); + bo->flags = flags; + +if (exclusive_vm) { + bo->exclusive_vm_root_gem =panthor_vm_root_gem (exclusive_vm); +drm_gem_object_get (bo->exclusive_vm_root_gem); + bo->base.base.resv = bo->exclusive_vm_root_gem->resv; + } + +/* +* Allocate an id of idr table where the obj is registered +* and handle has the id what user can see. +*/ + ret =drm_gem_handle_create (file, &shmem->base, handle); +/* drop reference from allocate - handle holds it now. */ +drm_gem_object_put (&shmem->base); + +return ret; +}drivers/gpu/drm/panthor/panthor_gem.h new file +144
-0@@ -0,0 +1,144 @@ +/* SPDX-License-Identifier: GPL-2.0 or MIT */ +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ + +#ifndef __PANTHOR_GEM_H__ +#define __PANTHOR_GEM_H__ + +#include <drm/drm_gem_shmem_helper.h> +#include <drm/drm_mm.h> + +#include <linux/iosys-map.h> +#include <linux/rwsem.h> + +struct panthor_vm ; + +/** +* struct panthor_gem_object - Driver specific GEM object. +*/ +struct panthor_gem_object { +/** @base: Inherit from drm_gem_shmem_object. */ +struct drm_gem_shmem_object base; + +/** +* @exclusive_vm_root_gem: Root GEM of the exclusive VM this GEM object +* is attached to. +* +* If @exclusive_vm_root_gem != NULL, any attempt to bind the GEM to a +* different VM will fail. +* +* All FW memory objects have this field set to the root GEM of the MCU +* VM. +*/ +struct drm_gem_object *exclusive_vm_root_gem; + +/** +* @gpuva_list_lock: Custom GPUVA lock. +* +* Used to protect insertion of drm_gpuva elements to the +* drm_gem_object.gpuva.list list. +* +* We can't use the GEM resv for that, because drm_gpuva_link() is +* called in a dma-signaling path, where we're not allowed to take +* resv locks. +*/ +struct mutex gpuva_list_lock; + +/** @flags: Combination of drm_panthor_bo_flags flags. */ +u32 flags; +}; + +/** +* struct panthor_kernel_bo - Kernel buffer object. +* +* These objects are only manipulated by the kernel driver and not +* directly exposed to the userspace. The GPU address of a kernel +* BO might be passed to userspace though. +*/ +struct panthor_kernel_bo { +/** +* @obj: The GEM object backing this kernel buffer object. +*/ +struct drm_gem_object *obj; + +/** +* @va_node: VA space allocated to this GEM. +*/ +struct drm_mm_node va_node; + +/** +* @kmap: Kernel CPU mapping of @gem. +*/ +void *kmap; +}; + +static inline +struct panthor_gem_object *to_panthor_bo (struct drm_gem_object *obj) +{ + return container_of(to_drm_gem_shmem_obj(obj), struct panthor_gem_object, base); +} + +struct drm_gem_object *panthor_gem_create_object (struct drm_device *ddev,size_t size); + +struct drm_gem_object * +panthor_gem_prime_import_sg_table (struct drm_device *ddev, +struct dma_buf_attachment *attach, +struct sg_table *sgt); + +int +panthor_gem_create_with_handle (struct drm_file *file, +struct drm_device *ddev, +struct panthor_vm *exclusive_vm, +size_t size, +u32 flags, +uint32_t *handle); + +static inline u64 +panthor_kernel_bo_gpuva (struct panthor_kernel_bo *bo) +{ +return bo->va_node.start; +} + +static inline size_t +panthor_kernel_bo_size (struct panthor_kernel_bo *bo) +{ +return bo->obj->size; +} + +static inline int +panthor_kernel_bo_vmap (struct panthor_kernel_bo *bo) +{ +struct iosys_map map; +int ret; + +if (bo->kmap) +return 0 ; + + ret =drm_gem_vmap_unlocked (bo->obj, &map); +if (ret) +return ret; + + bo->kmap = map.vaddr; +return 0 ; +} + +static inline void +panthor_kernel_bo_vunmap (struct panthor_kernel_bo *bo) +{ +if (bo->kmap) { +struct iosys_map map =IOSYS_MAP_INIT_VADDR (bo->kmap); + +drm_gem_vunmap_unlocked (bo->obj, &map); + bo->kmap =NULL ; + } +} + +struct panthor_kernel_bo * +panthor_kernel_bo_create (struct panthor_device *ptdev,struct panthor_vm *vm, +size_t size,u32 bo_flags,u32 vm_map_flags, +u64 gpu_va); + +void panthor_kernel_bo_destroy (struct panthor_vm *vm, +struct panthor_kernel_bo *bo); + +#endif /* __PANTHOR_GEM_H__ */
Boris Brezillon
maintainerRe: [PATCH v3 05/14] drm/panthor: Add GEM logical block
Quoted 58 lines · On Thu, 7 Dec 2023 16:38:33 +0000 Steven Price <[email protected]> wrote:
Quoted 44 lines
drivers/gpu/drm/panthor/panthor_gem.c new file +44
-0+ + ret =panthor_vm_unmap_range (vm, bo->va_node.start, +panthor_kernel_bo_size (bo)); +if (ret) + goto out_free_bo; + +panthor_vm_free_va (vm, &bo->va_node); +drm_gem_object_put (bo->obj); + +out_free_bo: +kfree (bo); +} + +/** +* panthor_kernel_bo_create() - Create and map a GEM object to a VM +* @ptdev: Device. +* @vm: VM to map the GEM to. If NULL, the kernel object is not GPU mapped. +* @size: Size of the buffer object. +* @bo_flags: Combination of drm_panthor_bo_flags flags. +* @vm_map_flags: Combination of drm_panthor_vm_bind_op_flags (only those +* that are related to map operations). +* @gpu_va: GPU address assigned when mapping to the VM. +* If gpu_va == PANTHOR_VM_KERNEL_AUTO_VA, the virtual address will be +* automatically allocated. +* +* Return: A valid pointer in case of success, an ERR_PTR() otherwise. +*/ +struct panthor_kernel_bo * +panthor_kernel_bo_create (struct panthor_device *ptdev,struct panthor_vm *vm, +size_t size,u32 bo_flags,u32 vm_map_flags, +u64 gpu_va) +{ +struct drm_gem_shmem_object *obj; +struct panthor_kernel_bo *kbo; +struct panthor_gem_object *bo; +int ret; + + kbo =kzalloc (sizeof (*kbo),GFP_KERNEL ); +if (!kbo) +return ERR_PTR (-ENOMEM ); + + obj =drm_gem_shmem_create (&ptdev->base, size); +if (IS_ERR (obj)) { + ret =PTR_ERR (obj);drivers/gpu/drm/panthor/panthor_gem.c line 98
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_gem.c new file +10
-0+ goto err_free_bo; + } + + bo =to_panthor_bo (&obj->base); + size = obj->base.size; + kbo->obj = &obj->base; + bo->flags = bo_flags; + +if (!vm) +return 0 ;This doesn't look right - I'd expect kbo to be returned? (Or an error if !vm isn't actually supported).
Quoted 3 lines
I've had a look at the rest of the driver and I can't find a user of the !vm case. So either I'm missing something (quite plausible) or we should just make the vm argument compulsory and simplify this a bit.
Hm, I can't remember why I made the GPU VM mapping optional for private BOs, so I'll just make it mandatory as you suggest, and we can revisit it later if we have an actual use case for that.
Boris Brezillon
maintainerPATCH v3 6/14 drm/panthor: Add the devfreq logical block
In reply to Boris BrezillonEvery thing related to devfreq in placed in panthor_devfreq.c, and helpers that can be called by other logical blocks are exposed through panthor_devfreq.h.
This implementation is loosely based on the panfrost implementation, the only difference being that we don't count device users, because the idle/active state will be managed by the scheduler logic.
v3:
- Add acks for the MIT/GPL2 relicensing
v2:
- Added in v2
--- drivers/gpu/drm/panthor/panthor_devfreq.c | 283 ++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_devfreq.h | 25 ++ 2 files changed, 308 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_devfreq.c create mode 100644 drivers/gpu/drm/panthor/panthor_devfreq.h
drivers/gpu/drm/panthor/panthor_devfreq.c new file +283 -0
@@ -0,0 +1,283 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2019 Collabora ltd. */ + +#include <linux/clk.h> +#include <linux/devfreq.h> +#include <linux/devfreq_cooling.h> +#include <linux/platform_device.h> +#include <linux/pm_opp.h> + +#include <drm/drm_managed.h> + +#include "panthor_device.h" +#include "panthor_devfreq.h" + +/** +* struct panthor_devfreq - Device frequency management +*/ +struct panthor_devfreq { +/** @devfreq: devfreq device. */ +struct devfreq *devfreq; + +/** @gov_data: Governor data. */ +struct devfreq_simple_ondemand_data gov_data; + +/** @busy_time: Busy time. */ +ktime_t busy_time; + +/** @idle_time: Idle time. */ +ktime_t idle_time; + +/** @time_last_update: Last update time. */ +ktime_t time_last_update; + +/** @last_busy_state: True if the GPU was busy last time we updated the state. */ +bool last_busy_state; + +/* +* @lock: Lock used to protect busy_time, idle_time, time_last_update and +* last_busy_state. +* +* These fields can be accessed concurrently by panthor_devfreq_get_dev_status() +* and panthor_devfreq_record_{busy,idle}(). +*/ +spinlock_t lock; +}; + +static void panthor_devfreq_update_utilization (struct panthor_devfreq *pdevfreq) +{ +ktime_t now, last; + + now =ktime_get (); + last = pdevfreq->time_last_update; + +if (pdevfreq->last_busy_state) + pdevfreq->busy_time +=ktime_sub (now, last); +else + pdevfreq->idle_time +=ktime_sub (now, last); + + pdevfreq->time_last_update = now; +} + +static int panthor_devfreq_target (struct device *dev,unsigned long *freq, +u32 flags) +{ +struct dev_pm_opp *opp; + + opp =devfreq_recommended_opp (dev, freq, flags); +if (IS_ERR (opp)) +return PTR_ERR (opp); +dev_pm_opp_put (opp); + +return dev_pm_opp_set_rate (dev, *freq); +} + +static void panthor_devfreq_reset (struct panthor_devfreq *pdevfreq) +{ + pdevfreq->busy_time =0 ; + pdevfreq->idle_time =0 ; + pdevfreq->time_last_update =ktime_get (); +} + +static int panthor_devfreq_get_dev_status (struct device *dev, +struct devfreq_dev_status *status) +{ +struct panthor_device *ptdev =dev_get_drvdata (dev); +struct panthor_devfreq *pdevfreq = ptdev->devfreq; +unsigned long irqflags; + + status->current_frequency =clk_get_rate (ptdev->clks.core); + +spin_lock_irqsave (&pdevfreq->lock, irqflags); + +panthor_devfreq_update_utilization (pdevfreq); + + status->total_time =ktime_to_ns (ktime_add (pdevfreq->busy_time, + pdevfreq->idle_time)); + + status->busy_time =ktime_to_ns (pdevfreq->busy_time); + +panthor_devfreq_reset (pdevfreq); + +spin_unlock_irqrestore (&pdevfreq->lock, irqflags); + +drm_dbg (&ptdev->base,"busy %lu total %lu %lu %% freq %lu MHz\n" , + status->busy_time, status->total_time, + status->busy_time / (status->total_time /100 ), + status->current_frequency /1000 /1000 ); + +return 0 ; +} + +static struct devfreq_dev_profile panthor_devfreq_profile = { + .timer =DEVFREQ_TIMER_DELAYED , + .polling_ms =50 ,/* ~3 frames */ + .target = panthor_devfreq_target, + .get_dev_status = panthor_devfreq_get_dev_status, +}; + +int panthor_devfreq_init (struct panthor_device *ptdev) +{ +/* There's actually 2 regulators (mali and sram), but the OPP core only +* supports one. +* +* We assume the sram regulator is coupled with the mali one and let +* the coupling logic deal with voltage updates. +*/ +static const char *const reg_names[] = {"mali" ,NULL }; +struct thermal_cooling_device *cooling; +struct device *dev = ptdev->base.dev; +struct panthor_devfreq *pdevfreq; +struct dev_pm_opp *opp; +unsigned long cur_freq; +int ret; + + pdevfreq =drmm_kzalloc (&ptdev->base,sizeof (*ptdev->devfreq),GFP_KERNEL ); +if (!pdevfreq) +return -ENOMEM ; + + ptdev->devfreq = pdevfreq; + + ret =devm_pm_opp_set_regulators (dev, reg_names); +if (ret) { +if (ret != -EPROBE_DEFER ) +DRM_DEV_ERROR (dev,"Couldn't set OPP regulators\n" ); + +return ret; + } + + ret =devm_pm_opp_of_add_table (dev); +if (ret) +return ret; + +spin_lock_init (&pdevfreq->lock); + +panthor_devfreq_reset (pdevfreq); + + cur_freq =clk_get_rate (ptdev->clks.core); + + opp =devfreq_recommended_opp (dev, &cur_freq,0 ); +if (IS_ERR (opp)) +return PTR_ERR (opp); + + panthor_devfreq_profile.initial_freq = cur_freq; + +/* Regulator coupling only takes care of synchronizing/balancing voltage +* updates, but the coupled regulator needs to be enabled manually. +* +* We use devm_regulator_get_enable_optional() and keep the sram supply +* enabled until the device is removed, just like we do for the mali +* supply, which is enabled when dev_pm_opp_set_opp(dev, opp) is called, +* and disabled when the opp_table is torn down, using the devm action. +* +* If we really care about disabling regulators on suspend, we should: +* - use devm_regulator_get_optional() here +* - call dev_pm_opp_set_opp(dev, NULL) before leaving this function +* (this disables the regulator passed to the OPP layer) +* - call dev_pm_opp_set_opp(dev, NULL) and +* regulator_disable(ptdev->regulators.sram) in +* panthor_devfreq_suspend() +* - call dev_pm_opp_set_opp(dev, default_opp) and +* regulator_enable(ptdev->regulators.sram) in +* panthor_devfreq_resume() +* +* But without knowing if it's beneficial or not (in term of power +* consumption), or how much it slows down the suspend/resume steps, +* let's just keep regulators enabled for the device lifetime. +*/ + ret =devm_regulator_get_enable_optional (dev,"sram" ); +if (ret && ret != -ENODEV ) { +if (ret != -EPROBE_DEFER ) +DRM_DEV_ERROR (dev,"Couldn't retrieve/enable sram supply\n" ); +return ret; + } + +/* +* Set the recommend OPP this will enable and configure the regulator +* if any and will avoid a switch off by regulator_late_cleanup() +*/ + ret =dev_pm_opp_set_opp (dev, opp); +if (ret) { +DRM_DEV_ERROR (dev,"Couldn't set recommended OPP\n" ); +return ret; + } + +dev_pm_opp_put (opp); + +/* +* Setup default thresholds for the simple_ondemand governor. +* The values are chosen based on experiments. +*/ + pdevfreq->gov_data.upthreshold =45 ; + pdevfreq->gov_data.downdifferential =5 ; + + pdevfreq->devfreq =devm_devfreq_add_device (dev, &panthor_devfreq_profile, +DEVFREQ_GOV_SIMPLE_ONDEMAND , + &pdevfreq->gov_data); +if (IS_ERR (pdevfreq->devfreq)) { +DRM_DEV_ERROR (dev,"Couldn't initialize GPU devfreq\n" ); + ret =PTR_ERR (pdevfreq->devfreq); + pdevfreq->devfreq =NULL ; +return ret; + } + + cooling =devfreq_cooling_em_register (pdevfreq->devfreq,NULL ); +if (IS_ERR (cooling)) +DRM_DEV_INFO (dev,"Failed to register cooling device\n" ); + +return 0 ; +} + +int panthor_devfreq_resume (struct panthor_device *ptdev) +{ +struct panthor_devfreq *pdevfreq = ptdev->devfreq; + +if (!pdevfreq->devfreq) +return 0 ; + +panthor_devfreq_reset (pdevfreq); + +return devfreq_resume_device (pdevfreq->devfreq); +} + +int panthor_devfreq_suspend (struct panthor_device *ptdev) +{ +struct panthor_devfreq *pdevfreq = ptdev->devfreq; + +if (!pdevfreq->devfreq) +return 0 ; + +return devfreq_suspend_device (pdevfreq->devfreq); +} + +void panthor_devfreq_record_busy (struct panthor_device *ptdev) +{ +struct panthor_devfreq *pdevfreq = ptdev->devfreq; +unsigned long irqflags; + +if (!pdevfreq->devfreq) +return ; + +spin_lock_irqsave (&pdevfreq->lock, irqflags); + +panthor_devfreq_update_utilization (pdevfreq); + pdevfreq->last_busy_state = true; + +spin_unlock_irqrestore (&pdevfreq->lock, irqflags); +} + +void panthor_devfreq_record_idle (struct panthor_device *ptdev) +{ +struct panthor_devfreq *pdevfreq = ptdev->devfreq; +unsigned long irqflags; + +if (!pdevfreq->devfreq) +return ; + +spin_lock_irqsave (&pdevfreq->lock, irqflags); + +panthor_devfreq_update_utilization (pdevfreq); + pdevfreq->last_busy_state = false; + +spin_unlock_irqrestore (&pdevfreq->lock, irqflags); +}
drivers/gpu/drm/panthor/panthor_devfreq.h new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 or MIT */ +/* Copyright 2019 Collabora ltd. */ + +#ifndef __PANTHOR_DEVFREQ_H__ +#define __PANTHOR_DEVFREQ_H__ + +#include <linux/devfreq.h> +#include <linux/spinlock.h> +#include <linux/ktime.h> + +struct devfreq ; +struct thermal_cooling_device ; + +struct panthor_device ; +struct panthor_devfreq ; + +int panthor_devfreq_init (struct panthor_device *ptdev); + +int panthor_devfreq_resume (struct panthor_device *ptdev); +int panthor_devfreq_suspend (struct panthor_device *ptdev); + +void panthor_devfreq_record_busy (struct panthor_device *ptdev); +void panthor_devfreq_record_idle (struct panthor_device *ptdev); + +#endif /* __PANTHOR_DEVFREQ_H__ */
Signature · 1 line
2.43.0
Re: [PATCH v3 06/14] drm/panthor: Add the devfreq logical block
Hi Boris,
Quoted 9 lines · Boris Brezillon
Every thing related to devfreq in placed in
panthor_devfreq.c, and helpers that can be called by other logical blocks are exposed throughpanthor_devfreq.h.This implementation is loosely based on the panfrost implementation, the only difference being that we don't count device users, because the idle/active state will be managed by the scheduler logic.
Quoted 7 lines
v3:
- Add acks for the MIT/GPL2 relicensingv2:
- Added in v2Cc Clément Péron <[email protected]> # MIT+GPL2 relicensing
For the MIT+GPL2 relicensing.
Quoted 335 lines
Reviewed-by Steven Price <[email protected]>Signed-off-by Boris Brezillon <[email protected]>--- drivers/gpu/drm/panthor/panthor_devfreq.c | 283 ++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_devfreq.h | 25 ++ 2 files changed, 308 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_devfreq.c create mode 100644 drivers/gpu/drm/panthor/panthor_devfreq.hdrivers/gpu/drm/panthor/panthor_devfreq.c new file +283
-0@@ -0,0 +1,283 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2019 Collabora ltd. */ + +#include <linux/clk.h> +#include <linux/devfreq.h> +#include <linux/devfreq_cooling.h> +#include <linux/platform_device.h> +#include <linux/pm_opp.h> + +#include <drm/drm_managed.h> + +#include "panthor_device.h" +#include "panthor_devfreq.h" + +/** +* struct panthor_devfreq - Device frequency management +*/ +struct panthor_devfreq { +/** @devfreq: devfreq device. */ +struct devfreq *devfreq; + +/** @gov_data: Governor data. */ +struct devfreq_simple_ondemand_data gov_data; + +/** @busy_time: Busy time. */ +ktime_t busy_time; + +/** @idle_time: Idle time. */ +ktime_t idle_time; + +/** @time_last_update: Last update time. */ +ktime_t time_last_update; + +/** @last_busy_state: True if the GPU was busy last time we updated the state. */ +bool last_busy_state; + +/* +* @lock: Lock used to protect busy_time, idle_time, time_last_update and +* last_busy_state. +* +* These fields can be accessed concurrently by panthor_devfreq_get_dev_status() +* and panthor_devfreq_record_{busy,idle}(). +*/ +spinlock_t lock; +}; + +static void panthor_devfreq_update_utilization (struct panthor_devfreq *pdevfreq) +{ +ktime_t now, last; + + now =ktime_get (); + last = pdevfreq->time_last_update; + +if (pdevfreq->last_busy_state) + pdevfreq->busy_time +=ktime_sub (now, last); +else + pdevfreq->idle_time +=ktime_sub (now, last); + + pdevfreq->time_last_update = now; +} + +static int panthor_devfreq_target (struct device *dev,unsigned long *freq, +u32 flags) +{ +struct dev_pm_opp *opp; + + opp =devfreq_recommended_opp (dev, freq, flags); +if (IS_ERR (opp)) +return PTR_ERR (opp); +dev_pm_opp_put (opp); + +return dev_pm_opp_set_rate (dev, *freq); +} + +static void panthor_devfreq_reset (struct panthor_devfreq *pdevfreq) +{ + pdevfreq->busy_time =0 ; + pdevfreq->idle_time =0 ; + pdevfreq->time_last_update =ktime_get (); +} + +static int panthor_devfreq_get_dev_status (struct device *dev, +struct devfreq_dev_status *status) +{ +struct panthor_device *ptdev =dev_get_drvdata (dev); +struct panthor_devfreq *pdevfreq = ptdev->devfreq; +unsigned long irqflags; + + status->current_frequency =clk_get_rate (ptdev->clks.core); + +spin_lock_irqsave (&pdevfreq->lock, irqflags); + +panthor_devfreq_update_utilization (pdevfreq); + + status->total_time =ktime_to_ns (ktime_add (pdevfreq->busy_time, + pdevfreq->idle_time)); + + status->busy_time =ktime_to_ns (pdevfreq->busy_time); + +panthor_devfreq_reset (pdevfreq); + +spin_unlock_irqrestore (&pdevfreq->lock, irqflags); + +drm_dbg (&ptdev->base,"busy %lu total %lu %lu %% freq %lu MHz\n" , + status->busy_time, status->total_time, + status->busy_time / (status->total_time /100 ), + status->current_frequency /1000 /1000 ); + +return 0 ; +} + +static struct devfreq_dev_profile panthor_devfreq_profile = { + .timer =DEVFREQ_TIMER_DELAYED , + .polling_ms =50 ,/* ~3 frames */ + .target = panthor_devfreq_target, + .get_dev_status = panthor_devfreq_get_dev_status, +}; + +int panthor_devfreq_init (struct panthor_device *ptdev) +{ +/* There's actually 2 regulators (mali and sram), but the OPP core only +* supports one. +* +* We assume the sram regulator is coupled with the mali one and let +* the coupling logic deal with voltage updates. +*/ +static const char *const reg_names[] = {"mali" ,NULL }; +struct thermal_cooling_device *cooling; +struct device *dev = ptdev->base.dev; +struct panthor_devfreq *pdevfreq; +struct dev_pm_opp *opp; +unsigned long cur_freq; +int ret; + + pdevfreq =drmm_kzalloc (&ptdev->base,sizeof (*ptdev->devfreq),GFP_KERNEL ); +if (!pdevfreq) +return -ENOMEM ; + + ptdev->devfreq = pdevfreq; + + ret =devm_pm_opp_set_regulators (dev, reg_names); +if (ret) { +if (ret != -EPROBE_DEFER ) +DRM_DEV_ERROR (dev,"Couldn't set OPP regulators\n" ); + +return ret; + } + + ret =devm_pm_opp_of_add_table (dev); +if (ret) +return ret; + +spin_lock_init (&pdevfreq->lock); + +panthor_devfreq_reset (pdevfreq); + + cur_freq =clk_get_rate (ptdev->clks.core); + + opp =devfreq_recommended_opp (dev, &cur_freq,0 ); +if (IS_ERR (opp)) +return PTR_ERR (opp); + + panthor_devfreq_profile.initial_freq = cur_freq; + +/* Regulator coupling only takes care of synchronizing/balancing voltage +* updates, but the coupled regulator needs to be enabled manually. +* +* We use devm_regulator_get_enable_optional() and keep the sram supply +* enabled until the device is removed, just like we do for the mali +* supply, which is enabled when dev_pm_opp_set_opp(dev, opp) is called, +* and disabled when the opp_table is torn down, using the devm action. +* +* If we really care about disabling regulators on suspend, we should: +* - use devm_regulator_get_optional() here +* - call dev_pm_opp_set_opp(dev, NULL) before leaving this function +* (this disables the regulator passed to the OPP layer) +* - call dev_pm_opp_set_opp(dev, NULL) and +* regulator_disable(ptdev->regulators.sram) in +* panthor_devfreq_suspend() +* - call dev_pm_opp_set_opp(dev, default_opp) and +* regulator_enable(ptdev->regulators.sram) in +* panthor_devfreq_resume() +* +* But without knowing if it's beneficial or not (in term of power +* consumption), or how much it slows down the suspend/resume steps, +* let's just keep regulators enabled for the device lifetime. +*/ + ret =devm_regulator_get_enable_optional (dev,"sram" ); +if (ret && ret != -ENODEV ) { +if (ret != -EPROBE_DEFER ) +DRM_DEV_ERROR (dev,"Couldn't retrieve/enable sram supply\n" ); +return ret; + } + +/* +* Set the recommend OPP this will enable and configure the regulator +* if any and will avoid a switch off by regulator_late_cleanup() +*/ + ret =dev_pm_opp_set_opp (dev, opp); +if (ret) { +DRM_DEV_ERROR (dev,"Couldn't set recommended OPP\n" ); +return ret; + } + +dev_pm_opp_put (opp); + +/* +* Setup default thresholds for the simple_ondemand governor. +* The values are chosen based on experiments. +*/ + pdevfreq->gov_data.upthreshold =45 ; + pdevfreq->gov_data.downdifferential =5 ; + + pdevfreq->devfreq =devm_devfreq_add_device (dev, &panthor_devfreq_profile, +DEVFREQ_GOV_SIMPLE_ONDEMAND , + &pdevfreq->gov_data); +if (IS_ERR (pdevfreq->devfreq)) { +DRM_DEV_ERROR (dev,"Couldn't initialize GPU devfreq\n" ); + ret =PTR_ERR (pdevfreq->devfreq); + pdevfreq->devfreq =NULL ; +return ret; + } + + cooling =devfreq_cooling_em_register (pdevfreq->devfreq,NULL ); +if (IS_ERR (cooling)) +DRM_DEV_INFO (dev,"Failed to register cooling device\n" ); + +return 0 ; +} + +int panthor_devfreq_resume (struct panthor_device *ptdev) +{ +struct panthor_devfreq *pdevfreq = ptdev->devfreq; + +if (!pdevfreq->devfreq) +return 0 ; + +panthor_devfreq_reset (pdevfreq); + +return devfreq_resume_device (pdevfreq->devfreq); +} + +int panthor_devfreq_suspend (struct panthor_device *ptdev) +{ +struct panthor_devfreq *pdevfreq = ptdev->devfreq; + +if (!pdevfreq->devfreq) +return 0 ; + +return devfreq_suspend_device (pdevfreq->devfreq); +} + +void panthor_devfreq_record_busy (struct panthor_device *ptdev) +{ +struct panthor_devfreq *pdevfreq = ptdev->devfreq; +unsigned long irqflags; + +if (!pdevfreq->devfreq) +return ; + +spin_lock_irqsave (&pdevfreq->lock, irqflags); + +panthor_devfreq_update_utilization (pdevfreq); + pdevfreq->last_busy_state = true; + +spin_unlock_irqrestore (&pdevfreq->lock, irqflags); +} + +void panthor_devfreq_record_idle (struct panthor_device *ptdev) +{ +struct panthor_devfreq *pdevfreq = ptdev->devfreq; +unsigned long irqflags; + +if (!pdevfreq->devfreq) +return ; + +spin_lock_irqsave (&pdevfreq->lock, irqflags); + +panthor_devfreq_update_utilization (pdevfreq); + pdevfreq->last_busy_state = false; + +spin_unlock_irqrestore (&pdevfreq->lock, irqflags); +}drivers/gpu/drm/panthor/panthor_devfreq.h new file +25
-1@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 or MIT */ +/* Copyright 2019 Collabora ltd. */ + +#ifndef __PANTHOR_DEVFREQ_H__ +#define __PANTHOR_DEVFREQ_H__ + +#include <linux/devfreq.h> +#include <linux/spinlock.h> +#include <linux/ktime.h> + +struct devfreq ; +struct thermal_cooling_device ; + +struct panthor_device ; +struct panthor_devfreq ; + +int panthor_devfreq_init (struct panthor_device *ptdev); + +int panthor_devfreq_resume (struct panthor_device *ptdev); +int panthor_devfreq_suspend (struct panthor_device *ptdev); + +void panthor_devfreq_record_busy (struct panthor_device *ptdev); +void panthor_devfreq_record_idle (struct panthor_device *ptdev); + +#endif /* __PANTHOR_DEVFREQ_H__ */ --2.43.0
Boris Brezillon
maintainerPATCH v3 7/14 drm/panthor: Add the MMU/VM logical block
In reply to Boris BrezillonMMU and VM management is related and placed in the same source file.
Page table updates are delegated to the io-pgtable-arm driver that's in the iommu subsystem.
The VM management logic is based on drm_gpuva_mgr, and is assuming the VA space is mostly managed by the usermode driver, except for a reserved portion of this VA-space that's used for kernel objects (like the heap contexts/chunks).
Both asynchronous and synchronous VM operations are supported, and internal helpers are exposed to allow other logical blocks to map their buffers in the GPU VA space.
There's one VM_BIND queue per-VM (meaning the Vulkan driver can only expose one sparse-binding queue), and this bind queue is managed with a 1:1 drm_sched_entity:drm_gpu_scheduler, such that each VM gets its own independent execution queue, avoiding VM operation serialization at the device level (things are still serialized at the VM level).
The rest is just implementation details that are hopefully well explained in the documentation.
v3:
- Add acks for the MIT/GPL2 relicensing
- Propagate MMU faults to the scheduler
- Move pages pinning/unpinning out of the dma_signalling path
- Fix 32-bit support
- Rework the user/kernel VA range calculation
- Make the auto-VA range explicit (auto-VA range doesn't cover the full kernel-VA range on the MCU VM)
- Let callers of panthor_vm_alloc_va() allocate the drm_mm_node (embedded in panthor_kernel_bo now)
- Adjust things to match the latest drm_gpuvm changes (extobj tracking, resv prep and more)
- Drop the per-AS lock and use slots_lock (fixes a race on vm->as.id)
- Set as.id to -1 when reusing an address space from the LRU list
- Drop misleading comment about page faults
- Remove check for irq being assigned in panthor_mmu_unplug()
--- drivers/gpu/drm/panthor/panthor_mmu.c | 2653 +++++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_mmu.h | 101 + 2 files changed, 2754 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_mmu.c create mode 100644 drivers/gpu/drm/panthor/panthor_mmu.h
drivers/gpu/drm/panthor/panthor_mmu.c new file +2653 -0
@@ -0,0 +1,2653 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ + +#include <drm/drm_debugfs.h> +#include <drm/drm_drv.h> +#include <drm/drm_exec.h> +#include <drm/drm_gpuvm.h> +#include <drm/drm_managed.h> +#include <drm/gpu_scheduler.h> +#include <drm/panthor_drm.h> + +#include <linux/atomic.h> +#include <linux/bitfield.h> +#include <linux/delay.h> +#include <linux/dma-mapping.h> +#include <linux/interrupt.h> +#include <linux/io.h> +#include <linux/iopoll.h> +#include <linux/io-pgtable.h> +#include <linux/iommu.h> +#include <linux/kmemleak.h> +#include <linux/platform_device.h> +#include <linux/pm_runtime.h> +#include <linux/rwsem.h> +#include <linux/sched.h> +#include <linux/shmem_fs.h> +#include <linux/sizes.h> + +#include "panthor_device.h" +#include "panthor_heap.h" +#include "panthor_mmu.h" +#include "panthor_sched.h" +#include "panthor_gem.h" +#include "panthor_regs.h" + +#define MAX_AS_SLOTS 32 + +struct panthor_vm ; + +/** +* struct panthor_as_slot - Address space slot +*/ +struct panthor_as_slot { +/** @vm: VM bound to this slot. NULL is no VM is bound. */ +struct panthor_vm *vm; +}; + +/** +* struct panthor_mmu - MMU related data +*/ +struct panthor_mmu { +/** @irq: The MMU irq. */ +struct panthor_irq irq; + +/** @as: Address space related fields. +* +* The GPU has a limited number of address spaces (AS) slots, forcing +* us to re-assign them to re-assign slots on-demand. +*/ +struct { +/** @slots_lock: Lock protecting access to all other AS fields. */ +struct mutex slots_lock; + +/** @alloc_mask: Bitmask encoding the allocated slots. */ +unsigned long alloc_mask; + +/** @faulty_mask: Bitmask encoding the faulty slots. */ +unsigned long faulty_mask; + +/** @slots: VMs currently bound to the AS slots. */ +struct panthor_as_slot slots[MAX_AS_SLOTS ]; + +/** +* @lru_list: List of least recently used VMs. +* +* We use this list to pick a VM to evict when all slots are +* used. +* +* There should be no more active VMs than there are AS slots, +* so this LRU is just here to keep VMs bound until there's +* a need to release a slot, thus avoid unnecessary TLB/cache +* flushes. +*/ +struct list_head lru_list; + } as; + +/** @vm: VMs management fields */ +struct { +/** @lock: Lock protecting access to list. */ +struct mutex lock; + +/** @list: List containing all VMs. */ +struct list_head list; + +/** @reset_in_progress: True if a reset is in progress. */ +bool reset_in_progress; + +/** @wq: Workqueue used for the VM_BIND queues. */ +struct workqueue_struct *wq; + } vm; +}; + +/** +* struct panthor_vm_pool - VM pool object +*/ +struct panthor_vm_pool { +/** @xa: Array used for VM handle tracking. */ +struct xarray xa; +}; + +/** +* struct panthor_vma - GPU mapping object +* +* This is used to track GEM mappings in GPU space. +*/ +struct panthor_vma { +/** @base: Inherits from drm_gpuva. */ +struct drm_gpuva base; + +/** @node: Used to implement deferred release of VMAs. */ +struct list_head node; + +/** +* @flags: Combination of drm_panthor_vm_bind_op_flags. +* +* Only map related flags are accepted. +*/ +u32 flags; +}; + +/** +* struct panthor_vm_op_ctx - VM operation context +* +* With VM operations potentially taking place in a dma-signaling path, we +* need to make sure everything that might require resource allocation is +* pre-allocated upfront. This is what this operation context is far. +* +* We also collect resources that have been freed, so we can release them +* asynchronously, and let the VM_BIND scheduler process the next VM_BIND +* request. +*/ +struct panthor_vm_op_ctx { +/** @rsvd_page_tables: Pages reserved for the MMU page table update. */ +struct { +/** @count: Number of pages reserved. */ +u32 count; + +/** @ptr: Point to the first unused page in the @pages table. */ +u32 ptr; + +/** +* @page: Array of pages that can be used for an MMU page table update. +* +* After an VM operation, there might be free pages left in this array. +* They should be returned to the pt_cache as part of the op_ctx cleanup. +*/ +void **pages; + } rsvd_page_tables; + +/** @flags: Combination of drm_panthor_vm_bind_op_flags. */ +u32 flags; + +/** @va: Virtual range targeted by the VM operation. */ +struct { +/** @addr: Start address. */ +u64 addr; + +/** @range: Range size. */ +u64 range; + } va; + +/** +* @returned_vmas: List of panthor_vma objects returned after a VM operation. +* +* For unmap operations, this will contain all VMAs that were covered by the +* specified VA range. +* +* For map operations, this will contain all VMAs that previously mapped to +* the specified VA range. +* +* Those VMAs, and the resources they point to will be released as part of +* the op_ctx cleanup operation. +*/ +struct list_head returned_vmas; + +/** @map: Fields specific to a map operation. */ +struct { +/** @vm_bo: Buffer object to map. */ +struct drm_gpuvm_bo *vm_bo; + +/** @bo_offset: Offset in the buffer object. */ +u64 bo_offset; + +/** +* @sgt: sg-table pointing to pages backing the GEM object. +* +* This is gathered at job creation time, such that we don't have +* to allocate in ::run_job(). +*/ +struct sg_table *sgt; + +/** +* @prev_vma: Pre-allocated VMA object to deal with a remap situation. +* +* If the map request covers a region that's inside another VMA, the +* previous VMA will be split, requiring instantiation of a maximum of +* two new VMA objects. +*/ +struct panthor_vma *prev_vma; + +/** +* @new_vma: The new VMA object that will be inserted to the VA tree. +*/ +struct panthor_vma *new_vma; + +/** +* @next_vma: Pre-allocated VMA object to deal with a remap situation. +* +* See @prev_vma. +*/ +struct panthor_vma *next_vma; + } map; +}; + +/** +* struct panthor_vm - VM object +* +* A VM is an object representing a GPU (or MCU) virtual address space. +* It embeds the MMU page table for this address space, a tree containing +* all the virtual mappings of GEM objects, and other things needed to manage +* the VM. +* +* Except for the MCU VM, which is managed by the kernel, all other VMs are +* created by userspace and mostly managed by userspace, using the +* %DRM_IOCTL_PANTHOR_VM_BIND ioctl. +* +* A portion of the virtual address space is reserved for kernel objects, +* like heap chunks, and userspace gets to decide how much of the virtual +* address space is left to the kernel (half of the virtual address space +* by default). +*/ +struct panthor_vm { +/** +* @base: Inherit from drm_gpuvm. +* +* We delegate all the VA management to the common drm_gpuvm framework +* and only implement hooks to update the MMU page table. +*/ +struct drm_gpuvm base; + +/** +* @sched: Scheduler used for asynchronous VM_BIND request. +* +* We use a 1:1 scheduler here. +*/ +struct drm_gpu_scheduler sched; + +/** +* @entity: Scheduling entity representing the VM_BIND queue. +* +* There's currently one bind queue per VM. It doesn't make sense to +* allow more given the VM operations are serialized anyway. +*/ +struct drm_sched_entity entity; + +/** @ptdev: Device. */ +struct panthor_device *ptdev; + +/** @memattr: Value to program to the AS_MEMATTR register. */ +u64 memattr; + +/** @pgtbl_ops: Page table operations. */ +struct io_pgtable_ops *pgtbl_ops; + +/** @root_page_table: Stores the root page table pointer. */ +void *root_page_table; + +/** +* @op_lock: Lock used to serialize operations on a VM. +* +* The serialization of jobs queued to the VM_BIND queue is already +* taken care of by drm_sched, but we need to serialize synchronous +* and asynchronous VM_BIND request. This is what this lock is for. +*/ +struct mutex op_lock; + +/** +* @op_ctx: The context attached to the currently executing VM operation. +* +* NULL when no operation is in progress. +*/ +struct panthor_vm_op_ctx *op_ctx; + +/** +* @mm: Memory management object representing the auto-VA/kernel-VA. +* +* Used to auto-allocate VA space for kernel-managed objects (tiler +* heaps, ...). +* +* For the MCU VM, this is managing the VA range that's used to map +* all shared interfaces. +* +* For user VMs, the range is specified by userspace, and must not +* exceed half of the VA space addressable. +*/ +struct drm_mm mm; + +/** @mm_lock: Lock protecting the @mm field. */ +struct mutex mm_lock; + +/** @kernel_auto_va: Automatic VA-range for kernel BOs. */ +struct { +/** @start: Start of the automatic VA-range for kernel BOs. */ +u64 start; + +/** @size: Size of the automatic VA-range for kernel BOs. */ +u64 end; + } kernel_auto_va; + +/** @as: Address space related fields. */ +struct { +/** +* @id: ID of the address space this VM is bound to. +* +* A value of -1 means the VM is inactive/not bound. +*/ +int id; + +/** +* @lru_node: Used to instead the VM in the panthor_mmu::as::lru_list. +* +* Active VMs should not be inserted in the LRU list. +*/ +struct list_head lru_node; + } as; + +/** +* @heaps: Tiler heap related fields. +*/ +struct { +/** +* @pool: The heap pool attached to this VM. +* +* Will stay NULL until someone creates a heap context on this VM. +*/ +struct panthor_heap_pool *pool; + +/** @lock: Lock used to protect access to @pool. */ +struct mutex lock; + } heaps; + +/** @node: Used to insert the VM in the panthor_mmu::vm::list. */ +struct list_head node; + +/** @for_mcu: True if this is the MCU VM. */ +bool for_mcu; + +/** +* @destroyed: True if the VM was destroyed. +* +* No further bind requests should be queued to a destroyed VM. +*/ +bool destroyed; + +/** +* @unusable: True if the VM has turned unusable because something +* bad happened during an asynchronous request. +* +* We don't try to recover from such failures, because this implies +* informing userspace about the specific operation that failed, and +* hoping the userspace driver can replay things from there. This all +* sounds very complicated for little gain. +* +* Instead, we should just flag the VM as unusable, and fail any +* further request targeting this VM. +* +* We also provide a way to query a VM state, so userspace can destroy +* it and create a new one. +* +* As an analogy, this would be mapped to a VK_ERROR_DEVICE_LOST +* situation, where the logical device needs to be re-created. +*/ +bool unusable; + +/** +* @unhandled_fault: Unhandled fault happened. +* +* This should be reported to the scheduler, and the queue/group be +* flagged as faulty as a result. +*/ +bool unhandled_fault; +}; + +/** +* struct panthor_vm_bind_job - VM bind job +*/ +struct panthor_vm_bind_job { +/** @base: Inherit from drm_sched_job. */ +struct drm_sched_job base; + +/** @refcount: Reference count. */ +struct kref refcount; + +/** @cleanup_op_ctx_work: Work used to cleanup the VM operation context. */ +struct work_struct cleanup_op_ctx_work; + +/** @vm: VM targeted by the VM operation. */ +struct panthor_vm *vm; + +/** @ctx: Operation context. */ +struct panthor_vm_op_ctx ctx; +}; + +/** +* @pt_cache: Cache used to allocate MMU page tables. +* +* The pre-allocation pattern forces us to over-allocate to plan for +* the worst case scenario, and return the pages we didn't use. +* +* Having a kmem_cache allows us to speed allocations. +*/ +static struct kmem_cache *pt_cache; + +/** +* alloc_pt() - Custom page table allocator +* @cookie: Cookie passed at page table allocation time. +* @size: Size of the page table. This size should be fixed, +* and determined at creation time based on the granule size. +* @gfp: GFP flags. +* +* We want a custom allocator so we can use a cache for page table +* allocations and amortize the cost of the over-reservation that's +* done to allow asynchronous VM operations. +* +* Return: non-NULL on success, NULL if the allocation failed for any +* reason. +*/ +static void *alloc_pt (void *cookie,size_t size,gfp_t gfp) +{ +struct panthor_vm *vm = cookie; +void *page; + +/* Allocation of the root page table happening during init. */ +if (unlikely (!vm->pgtbl_ops)) { +struct page *p; + +drm_WARN_ON (&vm->ptdev->base, vm->op_ctx); + p =alloc_pages_node (dev_to_node (vm->ptdev->base.dev), + gfp | __GFP_ZERO,get_order (size)); + page = p ?page_address (p) :NULL ; + vm->root_page_table = page; +return page; + } + +/* We're not supposed to have anything bigger than 4k here, because we picked a +* 4k granule size at init time. +*/ +if (drm_WARN_ON (&vm->ptdev->base, size !=SZ_4K )) +return NULL ; + +/* We must have some op_ctx attached to the VM and it must have at least one +* free page. +*/ +if (drm_WARN_ON (&vm->ptdev->base, !vm->op_ctx) || +drm_WARN_ON (&vm->ptdev->base, + vm->op_ctx->rsvd_page_tables.ptr >= vm->op_ctx->rsvd_page_tables.count)) +return NULL ; + + page = vm->op_ctx->rsvd_page_tables.pages[vm->op_ctx->rsvd_page_tables.ptr++]; +memset (page,0 ,SZ_4K ); + +/* Page table entries don't use virtual addresses, which trips out +* kmemleak. kmemleak_alloc_phys() might work, but physical addresses +* are mixed with other fields, and I fear kmemleak won't detect that +* either. +* +* Let's just ignore memory passed to the page-table driver for now. +*/ +kmemleak_ignore (page); +return page; +} + +/** +* @free_pt() - Custom page table free function +* @cookie: Cookie passed at page table allocation time. +* @data: Page table to free. +* @size: Size of the page table. This size should be fixed, +* and determined at creation time based on the granule size. +*/ +static void free_pt (void *cookie,void *data,size_t size) +{ +struct panthor_vm *vm = cookie; + +if (unlikely (vm->root_page_table == data)) { +free_pages ((unsigned long )data,get_order (size)); +return ; + } + +if (drm_WARN_ON (&vm->ptdev->base, size !=SZ_4K )) +return ; + +/* Return the page to the pt_cache. */ +kmem_cache_free (pt_cache, data); +} + +static int wait_ready (struct panthor_device *ptdev,u32 as_nr) +{ +int ret; +u32 val; + +/* Wait for the MMU status to indicate there is no active command, in +* case one is pending. +*/ + ret =readl_relaxed_poll_timeout_atomic (ptdev->iomem +AS_STATUS (as_nr), + val, !(val &AS_STATUS_AS_ACTIVE ), +10 ,100000 ); + +if (ret) { +panthor_device_schedule_reset (ptdev); +drm_err (&ptdev->base,"AS_ACTIVE bit stuck\n" ); + } + +return ret; +} + +static int write_cmd (struct panthor_device *ptdev,u32 as_nr,u32 cmd) +{ +int status; + +/* write AS_COMMAND when MMU is ready to accept another command */ + status =wait_ready (ptdev, as_nr); +if (!status) +gpu_write (ptdev,AS_COMMAND (as_nr), cmd); + +return status; +} + +static void lock_region (struct panthor_device *ptdev,u32 as_nr, +u64 region_start,u64 size) +{ +u8 region_width; +u64 region; +u64 region_end = region_start + size; + +if (!size) +return ; + +/* +* The locked region is a naturally aligned power of 2 block encoded as +* log2 minus(1). +* Calculate the desired start/end and look for the highest bit which +* differs. The smallest naturally aligned block must include this bit +* change, the desired region starts with this bit (and subsequent bits) +* zeroed and ends with the bit (and subsequent bits) set to one. +*/ + region_width =max (fls64 (region_start ^ (region_end -1 )), +const_ilog2 (AS_LOCK_REGION_MIN_SIZE )) -1 ; + +/* +* Mask off the low bits of region_start (which would be ignored by +* the hardware anyway) +*/ + region_start &=GENMASK_ULL (63 , region_width); + + region = region_width | region_start; + +/* Lock the region that needs to be updated */ +gpu_write (ptdev,AS_LOCKADDR_LO (as_nr),lower_32_bits (region)); +gpu_write (ptdev,AS_LOCKADDR_HI (as_nr),upper_32_bits (region)); +write_cmd (ptdev, as_nr,AS_COMMAND_LOCK ); +} + +static int mmu_hw_do_operation_locked (struct panthor_device *ptdev,int as_nr, +u64 iova,u64 size,u32 op) +{ +lockdep_assert_held (&ptdev->mmu->as.slots_lock); + +if (as_nr <0 ) +return 0 ; + +if (op !=AS_COMMAND_UNLOCK ) +lock_region (ptdev, as_nr, iova, size); + +/* Run the MMU operation */ +write_cmd (ptdev, as_nr, op); + +/* Wait for the flush to complete */ +return wait_ready (ptdev, as_nr); +} + +static int mmu_hw_do_operation (struct panthor_vm *vm, +u64 iova,u64 size,u32 op) +{ +struct panthor_device *ptdev = vm->ptdev; +int ret; + +mutex_lock (&ptdev->mmu->as.slots_lock); + ret =mmu_hw_do_operation_locked (ptdev, vm->as.id, iova, size, op); +mutex_unlock (&ptdev->mmu->as.slots_lock); + +return ret; +} + +static int panthor_mmu_as_enable (struct panthor_device *ptdev,u32 as_nr, +u64 transtab,u64 transcfg,u64 memattr) +{ +int ret; + + ret =mmu_hw_do_operation_locked (ptdev, as_nr,0 , ~0ULL ,AS_COMMAND_FLUSH_MEM ); +if (ret) +return ret; + +gpu_write (ptdev,AS_TRANSTAB_LO (as_nr),lower_32_bits (transtab)); +gpu_write (ptdev,AS_TRANSTAB_HI (as_nr),upper_32_bits (transtab)); + +gpu_write (ptdev,AS_MEMATTR_LO (as_nr),lower_32_bits (memattr)); +gpu_write (ptdev,AS_MEMATTR_HI (as_nr),upper_32_bits (memattr)); + +gpu_write (ptdev,AS_TRANSCFG_LO (as_nr),lower_32_bits (transcfg)); +gpu_write (ptdev,AS_TRANSCFG_HI (as_nr),upper_32_bits (transcfg)); + +return write_cmd (ptdev, as_nr,AS_COMMAND_UPDATE ); +} + +static int panthor_mmu_as_disable (struct panthor_device *ptdev,u32 as_nr) +{ +int ret; + + ret =mmu_hw_do_operation_locked (ptdev, as_nr,0 , ~0ULL ,AS_COMMAND_FLUSH_MEM ); +if (ret) +return ret; + +gpu_write (ptdev,AS_TRANSTAB_LO (as_nr),0 ); +gpu_write (ptdev,AS_TRANSTAB_HI (as_nr),0 ); + +gpu_write (ptdev,AS_MEMATTR_LO (as_nr),0 ); +gpu_write (ptdev,AS_MEMATTR_HI (as_nr),0 ); + +gpu_write (ptdev,AS_TRANSCFG_LO (as_nr),AS_TRANSCFG_ADRMODE_UNMAPPED ); +gpu_write (ptdev,AS_TRANSCFG_HI (as_nr),0 ); + +return write_cmd (ptdev, as_nr,AS_COMMAND_UPDATE ); +} + +static u32 panthor_mmu_fault_mask (struct panthor_device *ptdev,u32 value) +{ +/* Bits 16 to 31 mean REQ_COMPLETE. */ +return value &GENMASK (15 ,0 ); +} + +static u32 panthor_mmu_as_fault_mask (struct panthor_device *ptdev,u32 as) +{ +return BIT (as); +} + +/** +* panthor_vm_has_unhandled_faults() - Check if a VM has unhandled faults +* @vm: VM to check. +* +* Return: true if the VM has unhandled faults, false otherwise. +*/ +bool panthor_vm_has_unhandled_faults (struct panthor_vm *vm) +{ +return vm->unhandled_fault; +} + +/** +* panthor_vm_active() - Flag a VM as active +* @VM: VM to flag as active. +* +* Assigns an address space to a VM so it can be used by the GPU/MCU. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_active (struct panthor_vm *vm) +{ +struct panthor_device *ptdev = vm->ptdev; +struct io_pgtable_cfg *cfg = &io_pgtable_ops_to_pgtable (vm->pgtbl_ops)->cfg; +int ret =0 , as, cookie; +u64 transtab, transcfg; + +if (!drm_dev_enter (&ptdev->base, &cookie)) +return -ENODEV ; + +mutex_lock (&ptdev->mmu->as.slots_lock); + + as = vm->as.id; +if (as >=0 ) { +/* Unhandled pagefault on this AS, the MMU was disabled. We need to +* re-enable the MMU after clearing+unmasking the AS interrupts. +*/ +if (ptdev->mmu->as.faulty_mask &panthor_mmu_as_fault_mask (ptdev, as)) + goto out_enable_as; + + goto out_unlock; + } + +/* Check for a free AS */ +if (vm->for_mcu) { +drm_WARN_ON (&ptdev->base, ptdev->mmu->as.alloc_mask &BIT (0 )); + as =0 ; + }else { + as =ffz (ptdev->mmu->as.alloc_mask |BIT (0 )); + } + +if (!(BIT (as) & ptdev->gpu_info.as_present)) { +struct panthor_vm *lru_vm; + + lru_vm =list_first_entry_or_null (&ptdev->mmu->as.lru_list, + struct panthor_vm, + as.lru_node); +if (drm_WARN_ON (&ptdev->base, !lru_vm)) { + ret = -EBUSY ; + goto out_unlock; + } + +list_del_init (&lru_vm->as.lru_node); + as = lru_vm->as.id; + + lru_vm->as.id =-1 ; + }else { +set_bit (as, &ptdev->mmu->as.alloc_mask); + } + +/* Assign the free or reclaimed AS to the FD */ + vm->as.id = as; + ptdev->mmu->as.slots[as].vm = vm; + +out_enable_as: + transtab = cfg->arm_lpae_s1_cfg.ttbr; + transcfg =AS_TRANSCFG_PTW_MEMATTR_WB | +AS_TRANSCFG_PTW_RA | +AS_TRANSCFG_ADRMODE_AARCH64_4K ; +if (ptdev->coherent) + transcfg |=AS_TRANSCFG_PTW_SH_OS ; + +/* If the VM is re-activated, we clear the fault. */ + vm->unhandled_fault = false; + +/* Unhandled pagefault on this AS, clear the fault and re-enable interrupts +* before enabling the AS. +*/ +if (ptdev->mmu->as.faulty_mask &panthor_mmu_as_fault_mask (ptdev, as)) { +gpu_write (ptdev,MMU_INT_CLEAR ,panthor_mmu_as_fault_mask (ptdev, as)); + ptdev->mmu->as.faulty_mask &= ~panthor_mmu_as_fault_mask (ptdev, as); +gpu_write (ptdev,MMU_INT_MASK , ~ptdev->mmu->as.faulty_mask); + } + + ret =panthor_mmu_as_enable (vm->ptdev, vm->as.id, transtab, transcfg, vm->memattr); + +out_unlock: +mutex_unlock (&ptdev->mmu->as.slots_lock); +drm_dev_exit (cookie); +return ret; +} + +/** +* panthor_vm_idle() - Flag a VM idle +* @VM: VM to flag as idle. +* +* When we know the GPU is done with the VM (no more jobs to process), +* we can relinquish the AS slot attached to this VM, if any. +* +* We don't release the slot immediately, but instead place the VM in +* the LRU list, so it can be evicted if another VM needs an AS slot. +* This way, VMs keep attached to the AS they were given until we run +* out of free slot, limiting the number of MMU operations (TLB flush +* and other AS updates). +*/ +void panthor_vm_idle (struct panthor_vm *vm) +{ +struct panthor_device *ptdev = vm->ptdev; + +mutex_lock (&ptdev->mmu->as.slots_lock); +if (vm->as.id >=0 &&list_empty (&vm->as.lru_node)) +list_add_tail (&vm->as.lru_node, &ptdev->mmu->as.lru_list); +mutex_unlock (&ptdev->mmu->as.slots_lock); +} + +static void panthor_vm_stop (struct panthor_vm *vm) +{ +drm_sched_stop (&vm->sched,NULL ); +} + +static void panthor_vm_start (struct panthor_vm *vm) +{ +drm_sched_start (&vm->sched, true); +} + +/** +* panthor_vm_as() - Get the AS slot attached to a VM +* @vm: VM to get the AS slot of. +* +* Return: -1 if the VM is not assigned an AS slot yet, >= 0 otherwise. +*/ +int panthor_vm_as (struct panthor_vm *vm) +{ +return vm->as.id; +} + +static size_t get_pgsize (u64 addr,size_t size,size_t *count) +{ +/* +* io-pgtable only operates on multiple pages within a single table +* entry, so we need to split at boundaries of the table size, i.e. +* the next block size up. The distance from address A to the next +* boundary of block size B is logically B - A % B, but in unsigned +* two's complement where B is a power of two we get the equivalence +* B - A % B == (B - A) % B == (n * B - A) % B, and choose n = 0 :) +*/ +size_t blk_offset = -addr %SZ_2M ; + +if (blk_offset || size <SZ_2M ) { + *count =min_not_zero (blk_offset, size) /SZ_4K ; +return SZ_4K ; + } + blk_offset = -addr %SZ_1G ?:SZ_1G ; + *count =min (blk_offset, size) /SZ_2M ; +return SZ_2M ; +} + +static int panthor_vm_flush_range (struct panthor_vm *vm,u64 iova,u64 size) +{ +struct panthor_device *ptdev = vm->ptdev; +int ret =0 , cookie; + +if (vm->as.id <0 ) +return 0 ; + +/* If the device is unplugged, we just silently skip the flush. */ +if (!drm_dev_enter (&ptdev->base, &cookie)) +return 0 ; + +/* Flush the PTs only if we're already awake */ +if (pm_runtime_active (ptdev->base.dev)) + ret =mmu_hw_do_operation (vm, iova, size,AS_COMMAND_FLUSH_PT ); + +drm_dev_exit (cookie); +return ret; +} + +static int panthor_vm_unmap_pages (struct panthor_vm *vm,u64 iova,u64 size) +{ +struct panthor_device *ptdev = vm->ptdev; +struct io_pgtable_ops *ops = vm->pgtbl_ops; +u64 offset =0 ; + +drm_dbg (&ptdev->base,"unmap: as=%d, iova=%llx, len=%llx" , vm->as.id, iova, size); + +while (offset < size) { +size_t unmapped_sz =0 , pgcount; +size_t pgsize =get_pgsize (iova + offset, size - offset, &pgcount); + + unmapped_sz = ops->unmap_pages (ops, iova + offset, pgsize, pgcount,NULL ); + +if (drm_WARN_ON (&ptdev->base, unmapped_sz != pgsize * pgcount)) { +drm_err (&ptdev->base,"failed to unmap range %llx-%llx (requested range %llx-%llx)\n" , + iova + offset + unmapped_sz, + iova + offset + pgsize * pgcount, + iova, iova + size); +panthor_vm_flush_range (vm, iova, offset + unmapped_sz); +return -EINVAL ; + } + offset += unmapped_sz; + } + +return panthor_vm_flush_range (vm, iova, size); +} + +static int +panthor_vm_map_pages (struct panthor_vm *vm,u64 iova,int prot, +struct sg_table *sgt,u64 offset,u64 size) +{ +struct panthor_device *ptdev = vm->ptdev; +unsigned int count; +struct scatterlist *sgl; +struct io_pgtable_ops *ops = vm->pgtbl_ops; +u64 start_iova = iova; +int ret; + +if (!size) +return 0 ; + +for_each_sgtable_dma_sg (sgt, sgl, count) { +dma_addr_t paddr =sg_dma_address (sgl); +size_t len =sg_dma_len (sgl); + +if (len <= offset) { + offset -= len; +continue ; + } + + paddr -= offset; + len -= offset; + +if (size >=0 ) { + len =min_t (size_t, len, size); + size -= len; + } + +drm_dbg (&ptdev->base,"map: as=%d, iova=%llx, paddr=%pad, len=%zx" , + vm->as.id, iova, &paddr, len); + +while (len) { +size_t pgcount, mapped =0 ; +size_t pgsize =get_pgsize (iova | paddr, len, &pgcount); + + ret = ops->map_pages (ops, iova, paddr, pgsize, pgcount, prot, +GFP_KERNEL , &mapped); + iova += mapped; + paddr += mapped; + len -= mapped; + +if (drm_WARN_ON (&ptdev->base, !ret && !mapped)) + ret = -ENOMEM ; + +if (ret) { +/* If something failed, unmap what we've already mapped before +* returning. The unmap call is not supposed to fail. +*/ +drm_WARN_ON (&ptdev->base, +panthor_vm_unmap_pages (vm, start_iova, + iova - start_iova)); +return ret; + } + } + +if (!size) +break ; + } + +return panthor_vm_flush_range (vm, start_iova, iova - start_iova); +} + +static int flags_to_prot (u32 flags) +{ +int prot =0 ; + +if (flags &DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC ) + prot |=IOMMU_NOEXEC ; + +if (!(flags &DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED )) + prot |=IOMMU_CACHE ; + +if (flags &DRM_PANTHOR_VM_BIND_OP_MAP_READONLY ) + prot |=IOMMU_READ ; +else + prot |=IOMMU_READ |IOMMU_WRITE ; + +return prot; +} + +/** +* panthor_vm_alloc_va() - Allocate a region in the auto-va space +* @VM: VM to allocate a region on. +* @size: Size of the region. +* +* Some GPU objects, like heap chunks, are fully managed by the kernel and +* need to be mapped to the userspace VM, in the region reserved for kernel +* objects. +* +* This function takes care of allocating a region in this reserved space. +* +* Return: A valid pointer on success, and ERR_PTR() otherwise. +*/ +int +panthor_vm_alloc_va (struct panthor_vm *vm,u64 va,u64 size, +struct drm_mm_node *va_node) +{ +int ret; + +if (!size || (size & ~PAGE_MASK )) +return -EINVAL ; + +if (va !=PANTHOR_VM_KERNEL_AUTO_VA && (va & ~PAGE_MASK )) +return -EINVAL ; + +mutex_lock (&vm->mm_lock); +if (va !=PANTHOR_VM_KERNEL_AUTO_VA ) { +memset (va_node,0 ,sizeof (*va_node)); + va_node->start = va; + va_node->size = size; + ret =drm_mm_reserve_node (&vm->mm, va_node); + }else { + ret =drm_mm_insert_node_in_range (&vm->mm, va_node, size, + size >=SZ_2M ?SZ_2M :SZ_4K , +0 , vm->kernel_auto_va.start, + vm->kernel_auto_va.end, +DRM_MM_INSERT_BEST ); + } +mutex_unlock (&vm->mm_lock); + +return ret; +} + +/** +* panthor_vm_free_va() - Free a region allocated with panthor_vm_alloc_va() +* @VM: VM to free the region on. +* @mm_node: Memory node representing the region to free. +*/ +void panthor_vm_free_va (struct panthor_vm *vm,struct drm_mm_node *va_node) +{ +mutex_lock (&vm->mm_lock); +drm_mm_remove_node (va_node); +mutex_unlock (&vm->mm_lock); +} + +static void panthor_vm_bo_put (struct drm_gpuvm_bo *vm_bo) +{ +struct panthor_gem_object *bo =to_panthor_bo (vm_bo->obj); +bool unpin; + +/* We must retain the GEM before calling drm_gpuvm_bo_put(), +* otherwise the mutex might be destroyed while we hold it. +*/ +drm_gem_object_get (&bo->base.base); +mutex_lock (&bo->gpuva_list_lock); + unpin =drm_gpuvm_bo_put (vm_bo); +mutex_unlock (&bo->gpuva_list_lock); + +/* If the vm_bo object was destroyed, release the pin reference that +* was hold by this object. +*/ +if (unpin && !bo->base.base.import_attach) +drm_gem_shmem_unpin (&bo->base); + +drm_gem_object_put (&bo->base.base); +} + +static void panthor_vm_cleanup_op_ctx (struct panthor_vm_op_ctx *op_ctx, +struct panthor_vm *vm) +{ +struct panthor_vma *vma, *tmp_vma; + +u32 remaining_pt_count = op_ctx->rsvd_page_tables.count - + op_ctx->rsvd_page_tables.ptr; + +if (remaining_pt_count) { +kmem_cache_free_bulk (pt_cache, remaining_pt_count, + op_ctx->rsvd_page_tables.pages + + op_ctx->rsvd_page_tables.ptr); + } + +kfree (op_ctx->rsvd_page_tables.pages); +memset (&op_ctx->rsvd_page_tables,0 ,sizeof (op_ctx->rsvd_page_tables)); + +if (op_ctx->map.vm_bo) +panthor_vm_bo_put (op_ctx->map.vm_bo); + +kfree (op_ctx->map.new_vma); +kfree (op_ctx->map.next_vma); +kfree (op_ctx->map.prev_vma); +memset (&op_ctx->map,0 ,sizeof (op_ctx->map)); + +list_for_each_entry_safe (vma, tmp_vma, &op_ctx->returned_vmas, node) { +list_del (&vma->node); +panthor_vm_bo_put (vma->base.vm_bo); +kfree (vma); + } +} + +#define PANTHOR_VM_BIND_OP_MAP_FLAGS \ + (DRM_PANTHOR_VM_BIND_OP_MAP_READONLY | \ + DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC | \ + DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED | \ + DRM_PANTHOR_VM_BIND_OP_TYPE_MASK) + +static int panthor_vm_prepare_map_op_ctx (struct panthor_vm_op_ctx *op_ctx, +struct panthor_vm *vm, +struct panthor_gem_object *bo, +u64 offset, +u64 size,u64 va, +u32 flags) +{ +struct drm_gpuvm_bo *preallocated_vm_bo; +struct sg_table *sgt =NULL ; +u64 pt_count; +int ret; + +if (!bo) +return -EINVAL ; + +if ((flags & ~PANTHOR_VM_BIND_OP_MAP_FLAGS ) || + (flags &DRM_PANTHOR_VM_BIND_OP_TYPE_MASK ) !=DRM_PANTHOR_VM_BIND_OP_TYPE_MAP ) +return -EINVAL ; + +/* Make sure the VA and size are aligned and in-bounds. */ +if (size > bo->base.base.size || offset > bo->base.base.size - size) +return -EINVAL ; + +/* If the BO has an exclusive VM attached, it can't be mapped to other VMs. */ +if (bo->exclusive_vm_root_gem && + bo->exclusive_vm_root_gem !=panthor_vm_root_gem (vm)) +return -EINVAL ; + +memset (op_ctx,0 ,sizeof (*op_ctx)); +INIT_LIST_HEAD (&op_ctx->returned_vmas); + op_ctx->flags = flags; + op_ctx->va.range = size; + op_ctx->va.addr = va; + + op_ctx->map.new_vma =kzalloc (sizeof (*op_ctx->map.new_vma),GFP_KERNEL ); + op_ctx->map.next_vma =kzalloc (sizeof (*op_ctx->map.next_vma),GFP_KERNEL ); + op_ctx->map.prev_vma =kzalloc (sizeof (*op_ctx->map.prev_vma),GFP_KERNEL ); +if (!op_ctx->map.new_vma || !op_ctx->map.next_vma || !op_ctx->map.prev_vma) { + ret = -ENOMEM ; + goto err_cleanup; + } + +if (!bo->base.base.import_attach) { +/* Pre-reserve the BO pages, so the map operation doesn't have to +* allocate. +*/ + ret =drm_gem_shmem_pin (&bo->base); +if (ret) + goto err_cleanup; + } + + sgt =drm_gem_shmem_get_pages_sgt (&bo->base); +if (IS_ERR (sgt)) { +if (!bo->base.base.import_attach) +drm_gem_shmem_unpin (&bo->base); + + ret =PTR_ERR (sgt); + goto err_cleanup; + } + + op_ctx->map.sgt = sgt; + + preallocated_vm_bo =drm_gpuvm_bo_create (&vm->base, &bo->base.base); +if (!preallocated_vm_bo) { +if (!bo->base.base.import_attach) +drm_gem_shmem_unpin (&bo->base); + + ret = -ENOMEM ; + goto err_cleanup; + } + +mutex_lock (&bo->gpuva_list_lock); + op_ctx->map.vm_bo =drm_gpuvm_bo_obtain_prealloc (preallocated_vm_bo); +mutex_unlock (&bo->gpuva_list_lock); + +/* If the a vm_bo for this <VM,BO> combination exists, it already +* retains a pin ref, and we can release the one we took earlier. +* +* If our pre-allocated vm_bo is picked, it now retains the pin ref, +* which will be released in panthor_vm_bo_put(). +*/ +if (preallocated_vm_bo != op_ctx->map.vm_bo && + !bo->base.base.import_attach) +drm_gem_shmem_unpin (&bo->base); + + op_ctx->map.bo_offset = offset; + +/* L1, L2 and L3 page tables. +* We could optimize L3 allocation by iterating over the sgt and merging +* 2M contiguous blocks, but it's simpler to over-provision and return +* the pages if they're not used. +*/ + pt_count = ((ALIGN (va + size,1ull <<39 ) -ALIGN_DOWN (va,1ull <<39 )) >>39 ) + + ((ALIGN (va + size,1ull <<30 ) -ALIGN_DOWN (va,1ull <<30 )) >>30 ) + + ((ALIGN (va + size,1ull <<21 ) -ALIGN_DOWN (va,1ull <<21 )) >>21 ); + + op_ctx->rsvd_page_tables.pages =kcalloc (pt_count, +sizeof (*op_ctx->rsvd_page_tables.pages), +GFP_KERNEL ); +if (!op_ctx->rsvd_page_tables.pages) + goto err_cleanup; + + ret =kmem_cache_alloc_bulk (pt_cache,GFP_KERNEL , pt_count, + op_ctx->rsvd_page_tables.pages); + op_ctx->rsvd_page_tables.count = ret; +if (ret != pt_count) { + ret = -ENOMEM ; + goto err_cleanup; + } + +/* Insert BO into the extobj list last, when we know nothing can fail. */ +drm_gpuvm_bo_extobj_add (op_ctx->map.vm_bo); + +return 0 ; + +err_cleanup: +panthor_vm_cleanup_op_ctx (op_ctx, vm); +return ret; +} + +static int panthor_vm_prepare_unmap_op_ctx (struct panthor_vm_op_ctx *op_ctx, +struct panthor_vm *vm, +u64 va,u64 size) +{ +u32 pt_count =0 ; +int ret; + +memset (op_ctx,0 ,sizeof (*op_ctx)); +INIT_LIST_HEAD (&op_ctx->returned_vmas); + op_ctx->va.range = size; + op_ctx->va.addr = va; + op_ctx->flags =DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP ; + +/* Pre-allocate L3 page tables to account for the split-2M-block +* situation on unmap. +*/ +if (va !=ALIGN (va,SZ_2M )) + pt_count++; + +if (va + size !=ALIGN (va + size,SZ_2M ) && +ALIGN (va + size,SZ_2M ) !=ALIGN (va,SZ_2M )) + pt_count++; + +if (pt_count) { + op_ctx->rsvd_page_tables.pages =kcalloc (pt_count, +sizeof (*op_ctx->rsvd_page_tables.pages), +GFP_KERNEL ); +if (!op_ctx->rsvd_page_tables.pages) + goto err_cleanup; + + ret =kmem_cache_alloc_bulk (pt_cache,GFP_KERNEL , pt_count, + op_ctx->rsvd_page_tables.pages); +if (ret != pt_count) { + ret = -ENOMEM ; + goto err_cleanup; + } + op_ctx->rsvd_page_tables.count = pt_count; + } + +return 0 ; + +err_cleanup: +panthor_vm_cleanup_op_ctx (op_ctx, vm); +return ret; +} + +static void panthor_vm_prepare_sync_only_op_ctx (struct panthor_vm_op_ctx *op_ctx, +struct panthor_vm *vm) +{ +memset (op_ctx,0 ,sizeof (*op_ctx)); +INIT_LIST_HEAD (&op_ctx->returned_vmas); + op_ctx->flags =DRM_PANTHOR_VM_BIND_OP_TYPE_SYNC_ONLY ; +} + +/** +* panthor_vm_get_bo_for_va() - Get the GEM object mapped at a virtual address +* @vm: VM to look into. +* @va: Virtual address to search for. +* @bo_offset: Offset of the GEM object mapped at this virtual address. +* Only valid on success. +* +* The object returned by this function might no longer be mapped when the +* function returns. It's the caller responsibility to ensure there's no +* concurrent map/unmap operations making the returned value invalid, or +* make sure it doesn't matter if the object is no longer mapped. +* +* Return: A valid pointer on success, an ERR_PTR() otherwise. +*/ +struct panthor_gem_object * +panthor_vm_get_bo_for_va (struct panthor_vm *vm,u64 va,u64 *bo_offset) +{ +struct panthor_gem_object *bo =ERR_PTR (-ENOENT ); +struct drm_gpuva *gpuva; +struct panthor_vma *vma; + +/* Take the VM lock to prevent concurrent map/unmap operations. */ +mutex_lock (&vm->op_lock); + gpuva =drm_gpuva_find_first (&vm->base, va,1 ); + vma = gpuva ? container_of(gpuva, struct panthor_vma, base) : NULL; +if (vma && vma->base.gem.obj) { +drm_gem_object_get (vma->base.gem.obj); + bo =to_panthor_bo (vma->base.gem.obj); + *bo_offset = vma->base.gem.offset; + } +mutex_unlock (&vm->op_lock); + +return bo; +} + +#define PANTHOR_VM_MIN_KERNEL_VA_SIZE SZ_256M + +static u64 +panthor_vm_create_get_user_va_range (const struct drm_panthor_vm_create *args, +u64 full_va_range) +{ +u64 user_va_range; + +/* Make sure we have a minimum amount of VA space for kernel objects. */ +if (full_va_range <PANTHOR_VM_MIN_KERNEL_VA_SIZE ) +return 0 ; + +if (args->user_va_range) { +/* Use the user provided value if != 0. */ + user_va_range = args->user_va_range; + }else if (TASK_SIZE_OF (current) < full_va_range) { +/* If the task VM size is smaller than the GPU VA range, pick this +* as our default user VA range, so userspace can CPU/GPU map buffers +* at the same address. +*/ + user_va_range =TASK_SIZE_OF (current); + }else { +/* If the GPU VA range is smaller than the task VM size, we +* just have to live with the fact we won't be able to map +* all buffers at the same GPU/CPU address. +* +* If the GPU VA range is bigger than 4G (more than 32-bit of +* VA), we split the range in two, and assign half of it to +* the user and the other half to the kernel, if it's not, we +* keep the kernel VA space as small as possible. +*/ + user_va_range = full_va_range >SZ_4G ? + full_va_range /2 : + full_va_range -PANTHOR_VM_MIN_KERNEL_VA_SIZE ; + } + +if (full_va_range -PANTHOR_VM_MIN_KERNEL_VA_SIZE < user_va_range) + user_va_range = full_va_range -PANTHOR_VM_MIN_KERNEL_VA_SIZE ; + +return user_va_range; +} + +#define PANTHOR_VM_CREATE_FLAGS 0 + +static int +panthor_vm_create_check_args (const struct panthor_device *ptdev, +const struct drm_panthor_vm_create *args, +u64 *kernel_va_start,u64 *kernel_va_range) +{ +u32 va_bits =GPU_MMU_FEATURES_VA_BITS (ptdev->gpu_info.mmu_features); +u64 full_va_range =1ull <<min_t (u32, va_bits,sizeof (unsigned long ) *8 ); +u64 user_va_range; + +if (args->flags & ~PANTHOR_VM_CREATE_FLAGS ) +return -EINVAL ; + + user_va_range =panthor_vm_create_get_user_va_range (args, full_va_range); +if (!user_va_range || (args->user_va_range && args->user_va_range > user_va_range)) +return -EINVAL ; + +/* Pick a kernel VA range that's a power of two, to have a clear split. */ + *kernel_va_range =rounddown_pow_of_two (full_va_range - user_va_range); + *kernel_va_start = full_va_range - *kernel_va_range; +return 0 ; +} + +/* +* Only 32 VMs per open file. If that becomes a limiting factor, we can +* increase this number. +*/ +#define PANTHOR_MAX_VMS_PER_FILE 32 + +/** +* panthor_vm_pool_create_vm() - Create a VM +* @pool: The VM to create this VM on. +* @kernel_va_start: Start of the region reserved for kernel objects. +* @kernel_va_range: Size of the region reserved for kernel objects. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_pool_create_vm (struct panthor_device *ptdev, +struct panthor_vm_pool *pool, +struct drm_panthor_vm_create *args) +{ +u64 kernel_va_start, kernel_va_range; +struct panthor_vm *vm; +int ret; +u32 id; + + ret =panthor_vm_create_check_args (ptdev, args, &kernel_va_start, &kernel_va_range); +if (ret) +return ret; + + vm =panthor_vm_create (ptdev, false, kernel_va_start, kernel_va_range, + kernel_va_start, kernel_va_range); +if (IS_ERR (vm)) +return PTR_ERR (vm); + + ret =xa_alloc (&pool->xa, &id, vm, +XA_LIMIT (1 ,PANTHOR_MAX_VMS_PER_FILE ),GFP_KERNEL ); + +if (ret) { +panthor_vm_put (vm); +return ret; + } + + args->user_va_range = kernel_va_start; +return id; +} + +static void panthor_vm_destroy (struct panthor_vm *vm) +{ +if (!vm) +return ; + + vm->destroyed = true; + +mutex_lock (&vm->heaps.lock); +panthor_heap_pool_destroy (vm->heaps.pool); + vm->heaps.pool =NULL ; +mutex_unlock (&vm->heaps.lock); + +drm_WARN_ON (&vm->ptdev->base, +panthor_vm_unmap_range (vm, vm->base.mm_start, vm->base.mm_range)); +panthor_vm_put (vm); +} + +/** +* panthor_vm_destroy() - Destroy a VM. +* @pool: VM pool. +* @handle: VM handle. +* +* This function doesn't free the VM object or its resources, it just kills +* all mappings, and makes sure nothing can be mapped after that point. +* +* If there was any active jobs at the time this function is called, these +* jobs should experience page faults and be killed as a result. +* +* The VM resources are freed when the last reference on the VM object is +* dropped. +*/ +int panthor_vm_pool_destroy_vm (struct panthor_vm_pool *pool,u32 handle) +{ +struct panthor_vm *vm; + + vm =xa_erase (&pool->xa, handle); + +panthor_vm_destroy (vm); + +return vm ?0 : -EINVAL ; +} + +/** +* panthor_vm_pool_get_vm() - Retrieve VM object bound to a VM handle +* @pool: VM pool to check. +* @handle: Handle of the VM to retrieve. +* +* Return: A valid pointer if the VM exists, NULL otherwise. +*/ +struct panthor_vm * +panthor_vm_pool_get_vm (struct panthor_vm_pool *pool,u32 handle) +{ +struct panthor_vm *vm; + + vm =panthor_vm_get (xa_load (&pool->xa, handle)); + +return vm; +} + +/** +* panthor_vm_pool_destroy() - Destroy a VM pool. +* @pfile: File. +* +* Destroy all VMs in the pool, and release the pool resources. +* +* Note that VMs can outlive the pool they were created from if other +* objects hold a reference to there VMs. +*/ +void panthor_vm_pool_destroy (struct panthor_file *pfile) +{ +struct panthor_vm *vm; +unsigned long i; + +if (!pfile->vms) +return ; + +xa_for_each (&pfile->vms->xa, i, vm) +panthor_vm_destroy (vm); + +xa_destroy (&pfile->vms->xa); +kfree (pfile->vms); +} + +/** +* panthor_vm_pool_create() - Create a VM pool +* @pfile: File. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_pool_create (struct panthor_file *pfile) +{ + pfile->vms =kzalloc (sizeof (*pfile->vms),GFP_KERNEL ); +if (!pfile->vms) +return -ENOMEM ; + +xa_init_flags (&pfile->vms->xa,XA_FLAGS_ALLOC1 ); +return 0 ; +} + +/* dummy TLB ops, the real TLB flush happens in panthor_vm_flush_range() */ +static void mmu_tlb_flush_all (void *cookie) +{ +} + +static void mmu_tlb_flush_walk (unsigned long iova,size_t size,size_t granule,void *cookie) +{ +} + +static const struct iommu_flush_ops mmu_tlb_ops = { + .tlb_flush_all = mmu_tlb_flush_all, + .tlb_flush_walk = mmu_tlb_flush_walk, +}; + +static const char *access_type_name (struct panthor_device *ptdev, +u32 fault_status) +{ +switch (fault_status &AS_FAULTSTATUS_ACCESS_TYPE_MASK ) { +case AS_FAULTSTATUS_ACCESS_TYPE_ATOMIC : +return "ATOMIC" ; +case AS_FAULTSTATUS_ACCESS_TYPE_READ : +return "READ" ; +case AS_FAULTSTATUS_ACCESS_TYPE_WRITE : +return "WRITE" ; +case AS_FAULTSTATUS_ACCESS_TYPE_EX : +return "EXECUTE" ; +default : +drm_WARN_ON (&ptdev->base,1 ); +return NULL ; + } +} + +static void panthor_mmu_irq_handler (struct panthor_device *ptdev,u32 status) +{ +bool has_unhandled_faults = false; + + status =panthor_mmu_fault_mask (ptdev, status); +while (status) { +u32 as =ffs (status | (status >>16 )) -1 ; +u32 mask =panthor_mmu_as_fault_mask (ptdev, as); +u32 new_int_mask; +u64 addr; +u32 fault_status; +u32 exception_type; +u32 access_type; +u32 source_id; + + fault_status =gpu_read (ptdev,AS_FAULTSTATUS (as)); + addr =gpu_read (ptdev,AS_FAULTADDRESS_LO (as)); + addr |= (u64 )gpu_read (ptdev,AS_FAULTADDRESS_HI (as)) <<32 ; + +/* decode the fault status */ + exception_type = fault_status &0xFF ; + access_type = (fault_status >>8 ) &0x3 ; + source_id = (fault_status >>16 ); + +mutex_lock (&ptdev->mmu->as.slots_lock); + + ptdev->mmu->as.faulty_mask |= mask; + new_int_mask = +panthor_mmu_fault_mask (ptdev, ~ptdev->mmu->as.faulty_mask); + +/* terminal fault, print info about the fault */ +drm_err (&ptdev->base, +"Unhandled Page fault in AS%d at VA 0x%016llX\n" +"raw fault status: 0x%X\n" +"decoded fault status: %s\n" +"exception type 0x%X: %s\n" +"access type 0x%X: %s\n" +"source id 0x%X\n" , + as, addr, + fault_status, + (fault_status & (1 <<10 ) ?"DECODER FAULT" :"SLAVE FAULT" ), + exception_type,panthor_exception_name (ptdev, exception_type), + access_type,access_type_name (ptdev, fault_status), + source_id); + +/* Ignore MMU interrupts on this AS until it's been +* re-enabled. +*/ + ptdev->mmu->irq.mask = new_int_mask; +gpu_write (ptdev,MMU_INT_MASK , new_int_mask); + +if (ptdev->mmu->as.slots[as].vm) + ptdev->mmu->as.slots[as].vm->unhandled_fault = true; + +/* Disable the MMU to kill jobs on this AS. */ +panthor_mmu_as_disable (ptdev, as); +mutex_unlock (&ptdev->mmu->as.slots_lock); + + status &= ~mask; + has_unhandled_faults = true; + } + +if (has_unhandled_faults) +panthor_sched_report_mmu_fault (ptdev); +} +PANTHOR_IRQ_HANDLER (mmu,MMU , panthor_mmu_irq_handler); + +/** +* panthor_mmu_suspend() - Suspend the MMU logic +* @ptdev: Device. +* +* All we do here is de-assign the AS slots on all active VMs, so things +* get flushed to the main memory, and no further access to these VMs are +* possible. +* +* We also suspend the MMU IRQ. +*/ +void panthor_mmu_suspend (struct panthor_device *ptdev) +{ +mutex_lock (&ptdev->mmu->as.slots_lock); +for (u32 i =0 ; i <ARRAY_SIZE (ptdev->mmu->as.slots); i++) { +struct panthor_vm *vm = ptdev->mmu->as.slots[i].vm; + +if (vm) { +drm_WARN_ON (&ptdev->base,panthor_mmu_as_disable (ptdev, i)); + vm->as.id =-1 ; +list_del_init (&vm->as.lru_node); + ptdev->mmu->as.slots[i].vm =NULL ; + } + } +mutex_unlock (&ptdev->mmu->as.slots_lock); + +panthor_mmu_irq_suspend (&ptdev->mmu->irq); +} + +/** +* panthor_mmu_resume() - Resume the MMU logic +* @ptdev: Device. +* +* Resume the IRQ. +* +* We don't re-enable previously active VMs. We assume other parts of the +* driver will call panthor_vm_active() on the VMs they intend to use. +*/ +void panthor_mmu_resume (struct panthor_device *ptdev) +{ +mutex_lock (&ptdev->mmu->as.slots_lock); + ptdev->mmu->as.alloc_mask =0 ; + ptdev->mmu->as.faulty_mask =0 ; +mutex_unlock (&ptdev->mmu->as.slots_lock); + +panthor_mmu_irq_resume (&ptdev->mmu->irq,panthor_mmu_fault_mask (ptdev, ~0 )); +} + +/** +* panthor_mmu_pre_reset() - Prepare for a reset +* @ptdev: Device. +* +* Suspend the IRQ, and make sure all VM_BIND queues are stopped, so we +* don't get asked to do a VM operation while the GPU is down. +* +* We don't cleanly shutdown the AS slots here, because the reset might +* come from an AS_ACTIVE_BIT stuck situation. +*/ +void panthor_mmu_pre_reset (struct panthor_device *ptdev) +{ +struct panthor_vm *vm; + +panthor_mmu_irq_suspend (&ptdev->mmu->irq); + +mutex_lock (&ptdev->mmu->vm.lock); + ptdev->mmu->vm.reset_in_progress = true; +list_for_each_entry (vm, &ptdev->mmu->vm.list, node) +panthor_vm_stop (vm); +mutex_unlock (&ptdev->mmu->vm.lock); +} + +/** +* panthor_mmu_post_reset() - Restore things after a reset +* @ptdev: Device. +* +* Put the MMU logic back in action after a reset. That implies resuming the +* IRQ and re-enabling the VM_BIND queues. +*/ +void panthor_mmu_post_reset (struct panthor_device *ptdev) +{ +struct panthor_vm *vm; + +mutex_lock (&ptdev->mmu->as.slots_lock); + +/* Now that the reset is effective, we can assume that none of the +* AS slots are setup, and clear the faulty flags too. +*/ + ptdev->mmu->as.alloc_mask =0 ; + ptdev->mmu->as.faulty_mask =0 ; + +for (u32 i =0 ; i <ARRAY_SIZE (ptdev->mmu->as.slots); i++) { +struct panthor_vm *vm = ptdev->mmu->as.slots[i].vm; + +if (vm) { + vm->as.id =-1 ; +list_del_init (&vm->as.lru_node); + ptdev->mmu->as.slots[i].vm =NULL ; + } + } + +mutex_unlock (&ptdev->mmu->as.slots_lock); + +panthor_mmu_irq_resume (&ptdev->mmu->irq,panthor_mmu_fault_mask (ptdev, ~0 )); + +/* Restart the VM_BIND queues. */ +mutex_lock (&ptdev->mmu->vm.lock); +list_for_each_entry (vm, &ptdev->mmu->vm.list, node) { +panthor_vm_start (vm); + } + ptdev->mmu->vm.reset_in_progress = false; +mutex_unlock (&ptdev->mmu->vm.lock); +} + +static void panthor_vm_free (struct drm_gpuvm *gpuvm) +{ + struct panthor_vm *vm = container_of(gpuvm, struct panthor_vm, base); +struct panthor_device *ptdev = vm->ptdev; + +mutex_lock (&vm->heaps.lock); +if (drm_WARN_ON (&ptdev->base, vm->heaps.pool)) +panthor_heap_pool_destroy (vm->heaps.pool); +mutex_unlock (&vm->heaps.lock); +mutex_destroy (&vm->heaps.lock); + +mutex_lock (&ptdev->mmu->vm.lock); +list_del (&vm->node); +/* Restore the scheduler state so we can call drm_sched_entity_destroy() +* and drm_sched_fini(). If get there, that means we have no job left +* and no new jobs can be queued, so we can start the scheduler without +* risking interfering with the reset. +*/ +if (ptdev->mmu->vm.reset_in_progress) +panthor_vm_start (vm); +mutex_unlock (&ptdev->mmu->vm.lock); + +drm_sched_entity_destroy (&vm->entity); +drm_sched_fini (&vm->sched); + +mutex_lock (&ptdev->mmu->as.slots_lock); +if (vm->as.id >=0 ) { +int cookie; + +if (drm_dev_enter (&ptdev->base, &cookie)) { +panthor_mmu_as_disable (ptdev, vm->as.id); +drm_dev_exit (cookie); + } + + ptdev->mmu->as.slots[vm->as.id].vm =NULL ; +clear_bit (vm->as.id, &ptdev->mmu->as.alloc_mask); +list_del (&vm->as.lru_node); + } +mutex_unlock (&ptdev->mmu->as.slots_lock); + +free_io_pgtable_ops (vm->pgtbl_ops); + +drm_mm_takedown (&vm->mm); +kfree (vm); +} + +/** +* panthor_vm_put() - Release a reference on a VM +* @vm: VM to release the reference on. Can be NULL. +*/ +void panthor_vm_put (struct panthor_vm *vm) +{ +static_assert (offsetof(struct panthor_vm , base) ==0 ); +drm_gpuvm_put (&vm->base); +} + +/** +* panthor_vm_get() - Get a VM reference +* @vm: VM to get the reference on. Can be NULL. +* +* Return: @vm value. +*/ +struct panthor_vm *panthor_vm_get (struct panthor_vm *vm) +{ +if (vm) +drm_gpuvm_get (&vm->base); + +return vm; +} + +/** +* panthor_vm_get_heap_pool() - Get the heap pool attached to a VM +* @vm: VM to query the heap pool on. +* @create: True if the heap pool should be created when it doesn't exist. +* +* Heap pools are per-VM. This function allows one to retrieve the heap pool +* attached to a VM. +* +* If no heap pool exists yet, and @create is true, we create one. +* +* The returned panthor_heap_pool should be released with panthor_heap_pool_put(). +* +* Return: A valid pointer on success, an ERR_PTR() otherwise. +*/ +struct panthor_heap_pool *panthor_vm_get_heap_pool (struct panthor_vm *vm,bool create) +{ +struct panthor_heap_pool *pool; + +mutex_lock (&vm->heaps.lock); +if (!vm->heaps.pool && create) { +if (vm->destroyed) + pool =ERR_PTR (-EINVAL ); +else + pool =panthor_heap_pool_create (vm->ptdev, vm); + +if (!IS_ERR (pool)) + vm->heaps.pool =panthor_heap_pool_get (pool); + }else { + pool =panthor_heap_pool_get (vm->heaps.pool); + } +mutex_unlock (&vm->heaps.lock); + +return pool; +} + +static u64 mair_to_memattr (u64 mair) +{ +u64 memattr =0 ; +u32 i; + +for (i =0 ; i <8 ; i++) { +u8 in_attr = mair >> (8 * i), out_attr; +u8 outer = in_attr >>4 , inner = in_attr &0xf ; + +/* For caching to be enabled, inner and outer caching policy +* have to be both write-back, if one of them is write-through +* or non-cacheable, we just choose non-cacheable. Device +* memory is also translated to non-cacheable. +*/ +if (!(outer &3 ) || !(outer &4 ) || !(inner &4 )) { + out_attr =AS_MEMATTR_AARCH64_INNER_OUTER_NC | +AS_MEMATTR_AARCH64_SH_MIDGARD_INNER | +AS_MEMATTR_AARCH64_INNER_ALLOC_EXPL (false, false); + }else { +/* Use SH_CPU_INNER mode so SH_IS, which is used when +* IOMMU_CACHE is set, actually maps to the standard +* definition of inner-shareable and not Mali's +* internal-shareable mode. +*/ + out_attr =AS_MEMATTR_AARCH64_INNER_OUTER_WB | +AS_MEMATTR_AARCH64_SH_CPU_INNER | +AS_MEMATTR_AARCH64_INNER_ALLOC_EXPL (inner &1 , inner &2 ); + } + + memattr |= (u64 )out_attr << (8 * i); + } + +return memattr; +} + +static void panthor_vma_link (struct panthor_vm *vm, +struct panthor_vma *vma, +struct drm_gpuvm_bo *vm_bo) +{ +struct panthor_gem_object *bo =to_panthor_bo (vma->base.gem.obj); + +mutex_lock (&bo->gpuva_list_lock); +drm_gpuva_link (&vma->base, vm_bo); +drm_gpuvm_bo_put (vm_bo); +mutex_unlock (&bo->gpuva_list_lock); +} + +static void panthor_vma_unlink (struct panthor_vm *vm, +struct panthor_vma *vma) +{ +struct panthor_gem_object *bo =to_panthor_bo (vma->base.gem.obj); +struct drm_gpuvm_bo *vm_bo =drm_gpuvm_bo_get (vma->base.vm_bo); + +mutex_lock (&bo->gpuva_list_lock); +drm_gpuva_unlink (&vma->base); +mutex_unlock (&bo->gpuva_list_lock); + +/* drm_gpuva_unlink() release the vm_bo, but we manually retained it +* when entering this function, so we can implement deferred VMA +* destruction. Re-assign it here. +*/ + vma->base.vm_bo = vm_bo; +list_add_tail (&vma->node, &vm->op_ctx->returned_vmas); +} + +static void panthor_vma_init (struct panthor_vma *vma,u32 flags) +{ +INIT_LIST_HEAD (&vma->node); + vma->flags = flags; +} + +#define PANTHOR_VM_MAP_FLAGS \ + (DRM_PANTHOR_VM_BIND_OP_MAP_READONLY | \ + DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC | \ + DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED) + +static int panthor_gpuva_sm_step_map (struct drm_gpuva_op *op,void *priv) +{ +struct panthor_vm *vm = priv; +struct panthor_vm_op_ctx *op_ctx = vm->op_ctx; +struct panthor_vma *vma = op_ctx->map.new_vma; +int ret; + +panthor_vma_init (vma, op_ctx->flags &PANTHOR_VM_MAP_FLAGS ); + + ret =panthor_vm_map_pages (vm, op->map.va.addr,flags_to_prot (vma->flags), + op_ctx->map.sgt, op->map.gem.offset, + op->map.va.range); +if (ret) +return ret; + +/* Ref owned by the mapping now, clear the obj field so we don't release the +* pinning/obj ref behind GPUVA's back. +*/ +drm_gpuva_map (&vm->base, &vma->base, &op->map); +panthor_vma_link (vm, op_ctx->map.new_vma, op_ctx->map.vm_bo); + op_ctx->map.vm_bo =NULL ; + op_ctx->map.new_vma =NULL ; +return 0 ; +} + +static int panthor_gpuva_sm_step_remap (struct drm_gpuva_op *op, +void *priv) +{ + struct panthor_vma *unmap_vma = container_of(op->remap.unmap->va, struct panthor_vma, base); +struct panthor_vm *vm = priv; +struct panthor_vm_op_ctx *op_ctx = vm->op_ctx; +struct drm_gpuva *prev_va =NULL , *next_va =NULL ; +u64 unmap_start, unmap_range; +int ret; + +drm_gpuva_op_remap_to_unmap_range (&op->remap, &unmap_start, &unmap_range); + ret =panthor_vm_unmap_pages (vm, unmap_start, unmap_range); +if (ret) +return ret; + +if (op->remap.prev) { +panthor_vma_init (op_ctx->map.prev_vma, unmap_vma->flags); + prev_va = &op_ctx->map.prev_vma->base; + } + +if (op->remap.next) { +panthor_vma_init (op_ctx->map.next_vma, unmap_vma->flags); + next_va = &op_ctx->map.next_vma->base; + } + +drm_gpuva_remap (prev_va, next_va, &op->remap); + +if (prev_va) { +panthor_vma_link (vm, op_ctx->map.prev_vma, op->remap.unmap->va->vm_bo); + op_ctx->map.prev_vma =NULL ; + } + +if (next_va) { +panthor_vma_link (vm, op_ctx->map.next_vma, op->remap.unmap->va->vm_bo); + op_ctx->map.next_vma =NULL ; + } + +panthor_vma_unlink (vm, unmap_vma); +return 0 ; +} + +static int panthor_gpuva_sm_step_unmap (struct drm_gpuva_op *op, +void *priv) +{ + struct panthor_vma *unmap_vma = container_of(op->unmap.va, struct panthor_vma, base); +struct panthor_vm *vm = priv; +int ret; + + ret =panthor_vm_unmap_pages (vm, unmap_vma->base.va.addr, + unmap_vma->base.va.range); +if (drm_WARN_ON (&vm->ptdev->base, ret)) +return ret; + +drm_gpuva_unmap (&op->unmap); +panthor_vma_unlink (vm, unmap_vma); +return 0 ; +} + +static const struct drm_gpuvm_ops panthor_gpuvm_ops = { + .vm_free = panthor_vm_free, + .sm_step_map = panthor_gpuva_sm_step_map, + .sm_step_remap = panthor_gpuva_sm_step_remap, + .sm_step_unmap = panthor_gpuva_sm_step_unmap, +}; + +/** +* panthor_vm_resv() - Get the dma_resv object attached to a VM. +* @vm: VM to get the dma_resv of. +* +* Return: A dma_resv object. +*/ +struct dma_resv *panthor_vm_resv (struct panthor_vm *vm) +{ +return drm_gpuvm_resv (&vm->base); +} + +struct drm_gem_object *panthor_vm_root_gem (struct panthor_vm *vm) +{ +if (!vm) +return NULL ; + +return vm->base.r_obj; +} + +static int +panthor_vm_exec_op (struct panthor_vm *vm,struct panthor_vm_op_ctx *op, +bool flag_vm_unusable_on_failure) +{ +u32 op_type = op->flags &DRM_PANTHOR_VM_BIND_OP_TYPE_MASK ; +int ret; + +if (op_type ==DRM_PANTHOR_VM_BIND_OP_TYPE_SYNC_ONLY ) +return 0 ; + +mutex_lock (&vm->op_lock); + vm->op_ctx = op; +switch (op_type) { +case DRM_PANTHOR_VM_BIND_OP_TYPE_MAP : +if (vm->unusable) { + ret = -EINVAL ; +break ; + } + + ret =drm_gpuvm_sm_map (&vm->base, vm, op->va.addr, op->va.range, + op->map.vm_bo->obj, op->map.bo_offset); +break ; + +case DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP : + ret =drm_gpuvm_sm_unmap (&vm->base, vm, op->va.addr, op->va.range); +break ; + +default : + ret = -EINVAL ; +break ; + } + +if (ret && flag_vm_unusable_on_failure) + vm->unusable = true; + + vm->op_ctx =NULL ; +mutex_unlock (&vm->op_lock); + +return ret; +} + +static struct dma_fence * +panthor_vm_bind_run_job (struct drm_sched_job *sched_job) +{ + struct panthor_vm_bind_job *job = container_of(sched_job, struct panthor_vm_bind_job, base); +bool cookie; +int ret; + +/* Not only we report an error whose result is propagated to the +* drm_sched finished fence, but we also flag the VM as unusable, because +* a failure in the async VM_BIND results in an inconsistent state. VM needs +* to be destroyed and recreated. +*/ + cookie =dma_fence_begin_signalling (); + ret =panthor_vm_exec_op (job->vm, &job->ctx, true); +dma_fence_end_signalling (cookie); + +return ret ?ERR_PTR (ret) :NULL ; +} + +static void panthor_vm_bind_job_release (struct kref *kref) +{ + struct panthor_vm_bind_job *job = container_of(kref, struct panthor_vm_bind_job, refcount); + +if (job->base.s_fence) +drm_sched_job_cleanup (&job->base); + +panthor_vm_cleanup_op_ctx (&job->ctx, job->vm); +panthor_vm_put (job->vm); +kfree (job); +} + +/** +* panthor_vm_bind_job_put() - Release a VM_BIND job reference +* @sched_job: Job to release the reference on. +*/ +void panthor_vm_bind_job_put (struct drm_sched_job *sched_job) +{ +struct panthor_vm_bind_job *job = + container_of(sched_job, struct panthor_vm_bind_job, base); + +if (sched_job) +kref_put (&job->refcount, panthor_vm_bind_job_release); +} + +static void +panthor_vm_bind_free_job (struct drm_sched_job *sched_job) +{ +struct panthor_vm_bind_job *job = + container_of(sched_job, struct panthor_vm_bind_job, base); + +drm_sched_job_cleanup (sched_job); + +/* Do the heavy cleanups asynchronously, so we're out of the +* dma-signaling path and can acquire dma-resv locks safely. +*/ +queue_work (panthor_cleanup_wq, &job->cleanup_op_ctx_work); +} + +static enum drm_gpu_sched_stat +panthor_vm_bind_timedout_job (struct drm_sched_job *sched_job) +{ +WARN (1 ,"VM_BIND ops are synchronous for now, there should be no timeout!" ); +return DRM_GPU_SCHED_STAT_NOMINAL ; +} + +static const struct drm_sched_backend_ops panthor_vm_bind_ops = { + .run_job = panthor_vm_bind_run_job, + .free_job = panthor_vm_bind_free_job, + .timedout_job = panthor_vm_bind_timedout_job, +}; + +/** +* panthor_vm_create() - Create a VM +* @ptdev: Device. +* @for_mcu: True if this is the FW MCU VM. +* @kernel_va_start: Start of the range reserved for kernel BO mapping. +* @kernel_va_size: Size of the range reserved for kernel BO mapping. +* @auto_kernel_va_start: Start of the auto-VA kernel range. +* @auto_kernel_va_size: Size of the auto-VA kernel range. +* +* Return: A valid pointer on success, an ERR_PTR() otherwise. +*/ +struct panthor_vm * +panthor_vm_create (struct panthor_device *ptdev,bool for_mcu, +u64 kernel_va_start,u64 kernel_va_size, +u64 auto_kernel_va_start,u64 auto_kernel_va_size) +{ +u32 va_bits =GPU_MMU_FEATURES_VA_BITS (ptdev->gpu_info.mmu_features); +u32 pa_bits =GPU_MMU_FEATURES_PA_BITS (ptdev->gpu_info.mmu_features); +u64 full_va_range =1ull <<min_t (u32, va_bits,sizeof (unsigned long ) *8 ); +struct drm_gem_object *dummy_gem; +struct drm_gpu_scheduler *sched; +struct io_pgtable_cfg pgtbl_cfg; +u64 mair, min_va, va_range; +struct panthor_vm *vm; +int ret; + + vm =kzalloc (sizeof (*vm),GFP_KERNEL ); +if (!vm) +return ERR_PTR (-ENOMEM ); + +/* We allocate a dummy GEM for the VM. */ + dummy_gem =drm_gpuvm_resv_object_alloc (&ptdev->base); +if (!dummy_gem) { + ret = -ENOMEM ; + goto err_free_vm; + } + +mutex_init (&vm->heaps.lock); + vm->for_mcu = for_mcu; + vm->ptdev = ptdev; +mutex_init (&vm->op_lock); + +if (for_mcu) { +/* CSF MCU is a cortex M7, and can only address 4G */ + min_va =0 ; + va_range =SZ_4G ; + }else { + min_va =0 ; + va_range = full_va_range; + } + +mutex_init (&vm->mm_lock); +drm_mm_init (&vm->mm, kernel_va_start, kernel_va_size); + vm->kernel_auto_va.start = auto_kernel_va_start; + vm->kernel_auto_va.end = vm->kernel_auto_va.start + auto_kernel_va_size -1 ; + +INIT_LIST_HEAD (&vm->node); +INIT_LIST_HEAD (&vm->as.lru_node); + vm->as.id =-1 ; + + pgtbl_cfg = (struct io_pgtable_cfg ) { + .pgsize_bitmap =SZ_4K |SZ_2M , + .ias = va_bits, + .oas = pa_bits, + .coherent_walk = ptdev->coherent, + .tlb = &mmu_tlb_ops, + .iommu_dev = ptdev->base.dev, + .alloc = alloc_pt, + .free = free_pt, + }; + + vm->pgtbl_ops =alloc_io_pgtable_ops (ARM_64_LPAE_S1 , &pgtbl_cfg, vm); +if (!vm->pgtbl_ops) { + ret = -EINVAL ; + goto err_mm_takedown; + } + +/* Bind operations are synchronous for now, no timeout needed. */ + ret =drm_sched_init (&vm->sched, &panthor_vm_bind_ops, ptdev->mmu->vm.wq, +1 ,1 ,0 , +MAX_SCHEDULE_TIMEOUT ,NULL ,NULL , +"panthor-vm-bind" , ptdev->base.dev); +if (ret) + goto err_free_io_pgtable; + + sched = &vm->sched; + ret =drm_sched_entity_init (&vm->entity,0 , &sched,1 ,NULL ); +if (ret) + goto err_sched_fini; + + mair =io_pgtable_ops_to_pgtable (vm->pgtbl_ops)->cfg.arm_lpae_s1_cfg.mair; + vm->memattr =mair_to_memattr (mair); + +mutex_lock (&ptdev->mmu->vm.lock); +list_add_tail (&vm->node, &ptdev->mmu->vm.list); + +/* If a reset is in progress, stop the scheduler. */ +if (ptdev->mmu->vm.reset_in_progress) +panthor_vm_stop (vm); +mutex_unlock (&ptdev->mmu->vm.lock); + +/* We intentionally leave the reserved range to zero, because we want kernel VMAs +* to be handled the same way user VMAs are. +*/ +drm_gpuvm_init (&vm->base, + for_mcu ?"panthor-MCU-VM" :"panthor-GPU-VM" , +0 , &ptdev->base, dummy_gem, min_va, va_range,0 ,0 , + &panthor_gpuvm_ops); +drm_gem_object_put (dummy_gem); +return vm; + +err_sched_fini: +drm_sched_fini (&vm->sched); + +err_free_io_pgtable: +free_io_pgtable_ops (vm->pgtbl_ops); + +err_mm_takedown: +drm_mm_takedown (&vm->mm); +drm_gem_object_put (dummy_gem); + +err_free_vm: +kfree (vm); +return ERR_PTR (ret); +} + +static int +panthor_vm_bind_prepare_op_ctx (struct drm_file *file, +struct panthor_vm *vm, +const struct drm_panthor_vm_bind_op *op, +struct panthor_vm_op_ctx *op_ctx) +{ +struct drm_gem_object *gem; +int ret; + +/* Aligned on page size. */ +if ((op->va | op->size) & ~PAGE_MASK ) +return -EINVAL ; + +switch (op->flags &DRM_PANTHOR_VM_BIND_OP_TYPE_MASK ) { +case DRM_PANTHOR_VM_BIND_OP_TYPE_MAP : + gem =drm_gem_object_lookup (file, op->bo_handle); + ret =panthor_vm_prepare_map_op_ctx (op_ctx, vm, + gem ?to_panthor_bo (gem) :NULL , + op->bo_offset, + op->size, + op->va, + op->flags); +drm_gem_object_put (gem); +return ret; + +case DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP : +if (op->flags & ~DRM_PANTHOR_VM_BIND_OP_TYPE_MASK ) +return -EINVAL ; + +if (op->bo_handle || op->bo_offset) +return -EINVAL ; + +return panthor_vm_prepare_unmap_op_ctx (op_ctx, vm, op->va, op->size); + +case DRM_PANTHOR_VM_BIND_OP_TYPE_SYNC_ONLY : +if (op->flags & ~DRM_PANTHOR_VM_BIND_OP_TYPE_MASK ) +return -EINVAL ; + +if (op->bo_handle || op->bo_offset) +return -EINVAL ; + +if (op->va || op->size) +return -EINVAL ; + +if (!op->syncs.count) +return -EINVAL ; + +panthor_vm_prepare_sync_only_op_ctx (op_ctx, vm); +return 0 ; + +default : +return -EINVAL ; + } +} + +static void panthor_vm_bind_job_cleanup_op_ctx_work (struct work_struct *work) +{ +struct panthor_vm_bind_job *job = + container_of(work, struct panthor_vm_bind_job, cleanup_op_ctx_work); + +panthor_vm_cleanup_op_ctx (&job->ctx, job->vm); +panthor_vm_bind_job_put (&job->base); +} + +/** +* panthor_vm_bind_job_create() - Create a VM_BIND job +* @file: File. +* @vm: VM targeted by the VM_BIND job. +* @op: VM operation data. +* +* Return: A valid pointer on success, an ERR_PTR() otherwise. +*/ +struct drm_sched_job * +panthor_vm_bind_job_create (struct drm_file *file, +struct panthor_vm *vm, +const struct drm_panthor_vm_bind_op *op) +{ +struct panthor_vm_bind_job *job; +int ret; + +if (!vm) +return ERR_PTR (-EINVAL ); + +if (vm->destroyed || vm->unusable) +return ERR_PTR (-EINVAL ); + + job =kzalloc (sizeof (*job),GFP_KERNEL ); +if (!job) +return ERR_PTR (-ENOMEM ); + + ret =panthor_vm_bind_prepare_op_ctx (file, vm, op, &job->ctx); +if (ret) { +kfree (job); +return ERR_PTR (ret); + } + +INIT_WORK (&job->cleanup_op_ctx_work, panthor_vm_bind_job_cleanup_op_ctx_work); +kref_init (&job->refcount); + job->vm =panthor_vm_get (vm); + + ret =drm_sched_job_init (&job->base, &vm->entity,1 , vm); +if (ret) + goto err_put_job; + +return &job->base; + +err_put_job: +panthor_vm_bind_job_put (&job->base); +return ERR_PTR (ret); +} + +/** +* panthor_vm_bind_job_prepare_resvs() - Prepare VM_BIND job dma_resvs +* @exec: The locking/preparation context. +* @sched_job: The job to prepare resvs on. +* +* Locks and prepare the VM resv. +* +* If this is a map operation, locks and prepares the GEM resv. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_bind_job_prepare_resvs (struct drm_exec *exec, +struct drm_sched_job *sched_job) +{ + struct panthor_vm_bind_job *job = container_of(sched_job, struct panthor_vm_bind_job, base); +int ret; + +/* Acquire the VM lock an reserve a slot for this VM bind job. */ + ret =drm_gpuvm_prepare_vm (&job->vm->base, exec,1 ); +if (ret) +return ret; + +if (job->ctx.map.vm_bo) { +/* Lock/prepare the GEM being mapped. */ + ret =drm_exec_prepare_obj (exec, job->ctx.map.vm_bo->obj,1 ); +if (ret) +return ret; + } + +return 0 ; +} + +/** +* panthor_vm_bind_job_update_resvs() - Update the resv objects touched by a job +* @exec: drm_exec context. +* @sched_job: Job to update the resvs on. +*/ +void panthor_vm_bind_job_update_resvs (struct drm_exec *exec, +struct drm_sched_job *sched_job) +{ + struct panthor_vm_bind_job *job = container_of(sched_job, struct panthor_vm_bind_job, base); + +/* Explicit sync => we just register our job finished fence as bookkeep. */ +drm_gpuvm_resv_add_fence (&job->vm->base, exec, + &sched_job->s_fence->finished, +DMA_RESV_USAGE_BOOKKEEP , +DMA_RESV_USAGE_BOOKKEEP ); +} + +void panthor_vm_update_resvs (struct panthor_vm *vm,struct drm_exec *exec, +struct dma_fence *fence, +enum dma_resv_usage private_usage, +enum dma_resv_usage extobj_usage) +{ +drm_gpuvm_resv_add_fence (&vm->base, exec, fence, private_usage, extobj_usage); +} + +/** +* panthor_vm_bind_exec_sync_op() - Execute a VM_BIND operation synchronously. +* @file: File. +* @vm: VM targeted by the VM operation. +* @op: Data describing the VM operation. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_bind_exec_sync_op (struct drm_file *file, +struct panthor_vm *vm, +struct drm_panthor_vm_bind_op *op) +{ +struct panthor_vm_op_ctx op_ctx; +int ret; + +/* No sync objects allowed on synchronous operations. */ +if (op->syncs.count) +return -EINVAL ; + +if (!op->size) +return 0 ; + + ret =panthor_vm_bind_prepare_op_ctx (file, vm, op, &op_ctx); +if (ret) +return ret; + + ret =panthor_vm_exec_op (vm, &op_ctx, false); +panthor_vm_cleanup_op_ctx (&op_ctx, vm); + +return ret; +} + +/** +* panthor_vm_map_bo_range() - Map a GEM object range to a VM +* @vm: VM to map the GEM to. +* @bo: GEM object to map. +* @offset: Offset in the GEM object. +* @size: Size to map. +* @va: Virtual address to map the object to. +* @flags: Combination of drm_panthor_vm_bind_op_flags flags. +* Only map-related flags are valid. +* +* Internal use only. For userspace requests, use +* panthor_vm_bind_exec_sync_op() instead. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_map_bo_range (struct panthor_vm *vm,struct panthor_gem_object *bo, +u64 offset,u64 size,u64 va,u32 flags) +{ +struct panthor_vm_op_ctx op_ctx; +int ret; + + ret =panthor_vm_prepare_map_op_ctx (&op_ctx, vm, bo, offset, size, va, flags); +if (ret) +return ret; + + ret =panthor_vm_exec_op (vm, &op_ctx, false); +panthor_vm_cleanup_op_ctx (&op_ctx, vm); + +return ret; +} + +/** +* panthor_vm_unmap_range() - Unmap a portion of the VA space +* @vm: VM to unmap the region from. +* @va: Virtual address to unmap. Must be 4k aligned. +* @size: Size of the region to unmap. Must be 4k aligned. +* +* Internal use only. For userspace requests, use +* panthor_vm_bind_exec_sync_op() instead. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_unmap_range (struct panthor_vm *vm,u64 va,u64 size) +{ +struct panthor_vm_op_ctx op_ctx; +int ret; + + ret =panthor_vm_prepare_unmap_op_ctx (&op_ctx, vm, va, size); +if (ret) +return ret; + + ret =panthor_vm_exec_op (vm, &op_ctx, false); +panthor_vm_cleanup_op_ctx (&op_ctx, vm); + +return ret; +} + +/** +* panthor_vm_prepare_mapped_bos_resvs() - Prepare resvs on VM BOs. +* @exec: Locking/preparation context. +* @vm: VM targeted by the GPU job. +* @slot_count: Number of slots to reserve. +* +* GPU jobs assume all BOs bound to the VM at the time the job is submitted +* are available when the job is executed. In order to guarantee that, we +* need to reserve a slot on all BOs mapped to a VM and update this slot with +* the job fence after its submission. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_prepare_mapped_bos_resvs (struct drm_exec *exec,struct panthor_vm *vm, +u32 slot_count) +{ +int ret; + +/* Acquire the VM lock an reserve a slot for this GPU job. */ + ret =drm_gpuvm_prepare_vm (&vm->base, exec, slot_count); +if (ret) +return ret; + +/* VM operations are not protected by the VM resv-lock. We need to +* take the op_lock to make sure the shared_bos list is not updated +* while we're walking it. +*/ +return drm_gpuvm_prepare_objects (&vm->base, exec, slot_count); +} + +/** +* panthor_mmu_unplug() - Unplug the MMU logic +* @ptdev: Device. +* +* No access to the MMU regs should be done after this function is called. +* We suspend the IRQ and disable all VMs to guarantee that. +*/ +void panthor_mmu_unplug (struct panthor_device *ptdev) +{ +panthor_mmu_irq_suspend (&ptdev->mmu->irq); + +mutex_lock (&ptdev->mmu->as.slots_lock); +for (u32 i =0 ; i <ARRAY_SIZE (ptdev->mmu->as.slots); i++) { +struct panthor_vm *vm = ptdev->mmu->as.slots[i].vm; + +if (vm) { +drm_WARN_ON (&ptdev->base,panthor_mmu_as_disable (ptdev, i)); + vm->as.id =-1 ; +list_del_init (&vm->as.lru_node); +clear_bit (i, &ptdev->mmu->as.alloc_mask); + ptdev->mmu->as.slots[i].vm =NULL ; + } + } +mutex_unlock (&ptdev->mmu->as.slots_lock); +} + +static void panthor_mmu_release_wq (struct drm_device *ddev,void *res) +{ +destroy_workqueue (res); +} + +/** +* panthor_mmu_init() - Initialize the MMU logic. +* @ptdev: Device. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_mmu_init (struct panthor_device *ptdev) +{ +struct panthor_mmu *mmu; +int ret, irq; + + mmu =drmm_kzalloc (&ptdev->base,sizeof (*mmu),GFP_KERNEL ); +if (!mmu) +return -ENOMEM ; + +INIT_LIST_HEAD (&mmu->as.lru_list); + +drmm_mutex_init (&ptdev->base, &mmu->as.slots_lock); +INIT_LIST_HEAD (&mmu->vm.list); +drmm_mutex_init (&ptdev->base, &mmu->vm.lock); + + ptdev->mmu = mmu; + + irq =platform_get_irq_byname (to_platform_device (ptdev->base.dev),"mmu" ); +if (irq <=0 ) +return -ENODEV ; + + ret =panthor_request_mmu_irq (ptdev, &mmu->irq, irq, +panthor_mmu_fault_mask (ptdev, ~0 )); +if (ret) +return ret; + + mmu->vm.wq =alloc_workqueue ("panthor-vm-bind" ,WQ_UNBOUND ,0 ); +if (!mmu->vm.wq) +return -ENOMEM ; + +return drmm_add_action_or_reset (&ptdev->base, panthor_mmu_release_wq, mmu->vm.wq); +} + +#ifdef CONFIG_DEBUG_FS +static int show_vm_gpuvas (struct panthor_vm *vm,struct seq_file *m) +{ +int ret; + +mutex_lock (&vm->op_lock); + ret =drm_debugfs_gpuva_info (m, &vm->base); +mutex_unlock (&vm->op_lock); + +return ret; +} + +static int show_each_vm (struct seq_file *m,void *arg) +{ +struct drm_info_node *node = (struct drm_info_node *)m->private; +struct drm_device *ddev = node->minor->dev; + struct panthor_device *ptdev = container_of(ddev, struct panthor_device, base); +int (*show)(struct panthor_vm *,struct seq_file *) = node->info_ent->data; +struct panthor_vm *vm; +int ret =0 ; + +mutex_lock (&ptdev->mmu->vm.lock); +list_for_each_entry (vm, &ptdev->mmu->vm.list, node) { + ret =show (vm, m); +if (ret <0 ) +break ; + +seq_puts (m,"\n" ); + } +mutex_unlock (&ptdev->mmu->vm.lock); + +return ret; +} + +static struct drm_info_list panthor_mmu_debugfs_list[] = { +DRM_DEBUGFS_GPUVA_INFO (show_each_vm, show_vm_gpuvas), +}; + +/** +* panthor_mmu_debugfs_init() - Initialize MMU debugfs entries +* @minor: Minor. +*/ +void panthor_mmu_debugfs_init (struct drm_minor *minor) +{ +drm_debugfs_create_files (panthor_mmu_debugfs_list, +ARRAY_SIZE (panthor_mmu_debugfs_list), + minor->debugfs_root, minor); +} +#endif /* CONFIG_DEBUG_FS */ + +/** +* panthor_mmu_pt_cache_init() - Initialize the page table cache. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_mmu_pt_cache_init (void ) +{ + pt_cache =kmem_cache_create ("panthor-mmu-pt" ,SZ_4K ,SZ_4K ,0 ,NULL ); +if (!pt_cache) +return -ENOMEM ; + +return 0 ; +} + +/** +* panthor_mmu_pt_cache_fini() - Destroy the page table cache. +*/ +void panthor_mmu_pt_cache_fini (void ) +{ +kmem_cache_destroy (pt_cache); +}
drivers/gpu/drm/panthor/panthor_mmu.h new file +101 -0
101 lines not on this page · raw message
Signature · 1 line
2.43.0
Steven Price
maintainerRe: [PATCH v3 07/14] drm/panthor: Add the MMU/VM logical block
Quoted 40 lines · Boris Brezillon
MMU and VM management is related and placed in the same source file.
Page table updates are delegated to the io-pgtable-arm driver that's in the iommu subsystem.
The VM management logic is based on
drm_gpuva_mgr, and is assuming the VA space is mostly managed by the usermode driver, except for a reserved portion of this VA-space that's used for kernel objects (like the heap contexts/chunks).Both asynchronous and synchronous VM operations are supported, and internal helpers are exposed to allow other logical blocks to map their buffers in the GPU VA space.
There's one
VM_BINDqueue per-VM (meaning the Vulkan driver can only expose one sparse-binding queue), and this bind queue is managed with a 1:1 drm_sched_entity:drm_gpu_scheduler, such that each VM gets its own independent execution queue, avoiding VM operation serialization at the device level (things are still serialized at the VM level).The rest is just implementation details that are hopefully well explained in the documentation.
v3:
- Add acks for the MIT/GPL2 relicensing
- Propagate MMU faults to the scheduler
- Move pages pinning/unpinning out of thedma_signallingpath
- Fix 32-bit support
- Rework the user/kernel VA range calculation
- Make the auto-VA range explicit (auto-VA range doesn't cover the full kernel-VA range on the MCU VM)
- Let callers ofpanthor_vm_alloc_va()allocate thedrm_mm_node(embedded inpanthor_kernel_bonow)
- Adjust things to match the latestdrm_gpuvmchanges (extobj tracking, resv prep and more)
- Drop the per-AS lock and useslots_lock(fixes a race onvm->as.id)
- Set as.id to -1 when reusing an address space from the LRU list
- Drop misleading comment about page faults
- Remove check for irq being assigned inpanthor_mmu_unplug()
Quoted 5 lines
Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>
A few comments below, but nothing major.
Quoted 448 lines
--- drivers/gpu/drm/panthor/panthor_mmu.c | 2653 +++++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_mmu.h | 101 + 2 files changed, 2754 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_mmu.c create mode 100644 drivers/gpu/drm/panthor/panthor_mmu.hdrivers/gpu/drm/panthor/panthor_mmu.c new file +435
-0@@ -0,0 +1,2653 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ + +#include <drm/drm_debugfs.h> +#include <drm/drm_drv.h> +#include <drm/drm_exec.h> +#include <drm/drm_gpuvm.h> +#include <drm/drm_managed.h> +#include <drm/gpu_scheduler.h> +#include <drm/panthor_drm.h> + +#include <linux/atomic.h> +#include <linux/bitfield.h> +#include <linux/delay.h> +#include <linux/dma-mapping.h> +#include <linux/interrupt.h> +#include <linux/io.h> +#include <linux/iopoll.h> +#include <linux/io-pgtable.h> +#include <linux/iommu.h> +#include <linux/kmemleak.h> +#include <linux/platform_device.h> +#include <linux/pm_runtime.h> +#include <linux/rwsem.h> +#include <linux/sched.h> +#include <linux/shmem_fs.h> +#include <linux/sizes.h> + +#include "panthor_device.h" +#include "panthor_heap.h" +#include "panthor_mmu.h" +#include "panthor_sched.h" +#include "panthor_gem.h" +#include "panthor_regs.h" + +#define MAX_AS_SLOTS 32 + +struct panthor_vm ; + +/** +* struct panthor_as_slot - Address space slot +*/ +struct panthor_as_slot { +/** @vm: VM bound to this slot. NULL is no VM is bound. */ +struct panthor_vm *vm; +}; + +/** +* struct panthor_mmu - MMU related data +*/ +struct panthor_mmu { +/** @irq: The MMU irq. */ +struct panthor_irq irq; + +/** @as: Address space related fields. +* +* The GPU has a limited number of address spaces (AS) slots, forcing +* us to re-assign them to re-assign slots on-demand. +*/ +struct { +/** @slots_lock: Lock protecting access to all other AS fields. */ +struct mutex slots_lock; + +/** @alloc_mask: Bitmask encoding the allocated slots. */ +unsigned long alloc_mask; + +/** @faulty_mask: Bitmask encoding the faulty slots. */ +unsigned long faulty_mask; + +/** @slots: VMs currently bound to the AS slots. */ +struct panthor_as_slot slots[MAX_AS_SLOTS ]; + +/** +* @lru_list: List of least recently used VMs. +* +* We use this list to pick a VM to evict when all slots are +* used. +* +* There should be no more active VMs than there are AS slots, +* so this LRU is just here to keep VMs bound until there's +* a need to release a slot, thus avoid unnecessary TLB/cache +* flushes. +*/ +struct list_head lru_list; + } as; + +/** @vm: VMs management fields */ +struct { +/** @lock: Lock protecting access to list. */ +struct mutex lock; + +/** @list: List containing all VMs. */ +struct list_head list; + +/** @reset_in_progress: True if a reset is in progress. */ +bool reset_in_progress; + +/** @wq: Workqueue used for the VM_BIND queues. */ +struct workqueue_struct *wq; + } vm; +}; + +/** +* struct panthor_vm_pool - VM pool object +*/ +struct panthor_vm_pool { +/** @xa: Array used for VM handle tracking. */ +struct xarray xa; +}; + +/** +* struct panthor_vma - GPU mapping object +* +* This is used to track GEM mappings in GPU space. +*/ +struct panthor_vma { +/** @base: Inherits from drm_gpuva. */ +struct drm_gpuva base; + +/** @node: Used to implement deferred release of VMAs. */ +struct list_head node; + +/** +* @flags: Combination of drm_panthor_vm_bind_op_flags. +* +* Only map related flags are accepted. +*/ +u32 flags; +}; + +/** +* struct panthor_vm_op_ctx - VM operation context +* +* With VM operations potentially taking place in a dma-signaling path, we +* need to make sure everything that might require resource allocation is +* pre-allocated upfront. This is what this operation context is far. +* +* We also collect resources that have been freed, so we can release them +* asynchronously, and let the VM_BIND scheduler process the next VM_BIND +* request. +*/ +struct panthor_vm_op_ctx { +/** @rsvd_page_tables: Pages reserved for the MMU page table update. */ +struct { +/** @count: Number of pages reserved. */ +u32 count; + +/** @ptr: Point to the first unused page in the @pages table. */ +u32 ptr; + +/** +* @page: Array of pages that can be used for an MMU page table update. +* +* After an VM operation, there might be free pages left in this array. +* They should be returned to the pt_cache as part of the op_ctx cleanup. +*/ +void **pages; + } rsvd_page_tables; + +/** @flags: Combination of drm_panthor_vm_bind_op_flags. */ +u32 flags; + +/** @va: Virtual range targeted by the VM operation. */ +struct { +/** @addr: Start address. */ +u64 addr; + +/** @range: Range size. */ +u64 range; + } va; + +/** +* @returned_vmas: List of panthor_vma objects returned after a VM operation. +* +* For unmap operations, this will contain all VMAs that were covered by the +* specified VA range. +* +* For map operations, this will contain all VMAs that previously mapped to +* the specified VA range. +* +* Those VMAs, and the resources they point to will be released as part of +* the op_ctx cleanup operation. +*/ +struct list_head returned_vmas; + +/** @map: Fields specific to a map operation. */ +struct { +/** @vm_bo: Buffer object to map. */ +struct drm_gpuvm_bo *vm_bo; + +/** @bo_offset: Offset in the buffer object. */ +u64 bo_offset; + +/** +* @sgt: sg-table pointing to pages backing the GEM object. +* +* This is gathered at job creation time, such that we don't have +* to allocate in ::run_job(). +*/ +struct sg_table *sgt; + +/** +* @prev_vma: Pre-allocated VMA object to deal with a remap situation. +* +* If the map request covers a region that's inside another VMA, the +* previous VMA will be split, requiring instantiation of a maximum of +* two new VMA objects. +*/ +struct panthor_vma *prev_vma; + +/** +* @new_vma: The new VMA object that will be inserted to the VA tree. +*/ +struct panthor_vma *new_vma; + +/** +* @next_vma: Pre-allocated VMA object to deal with a remap situation. +* +* See @prev_vma. +*/ +struct panthor_vma *next_vma; + } map; +}; + +/** +* struct panthor_vm - VM object +* +* A VM is an object representing a GPU (or MCU) virtual address space. +* It embeds the MMU page table for this address space, a tree containing +* all the virtual mappings of GEM objects, and other things needed to manage +* the VM. +* +* Except for the MCU VM, which is managed by the kernel, all other VMs are +* created by userspace and mostly managed by userspace, using the +* %DRM_IOCTL_PANTHOR_VM_BIND ioctl. +* +* A portion of the virtual address space is reserved for kernel objects, +* like heap chunks, and userspace gets to decide how much of the virtual +* address space is left to the kernel (half of the virtual address space +* by default). +*/ +struct panthor_vm { +/** +* @base: Inherit from drm_gpuvm. +* +* We delegate all the VA management to the common drm_gpuvm framework +* and only implement hooks to update the MMU page table. +*/ +struct drm_gpuvm base; + +/** +* @sched: Scheduler used for asynchronous VM_BIND request. +* +* We use a 1:1 scheduler here. +*/ +struct drm_gpu_scheduler sched; + +/** +* @entity: Scheduling entity representing the VM_BIND queue. +* +* There's currently one bind queue per VM. It doesn't make sense to +* allow more given the VM operations are serialized anyway. +*/ +struct drm_sched_entity entity; + +/** @ptdev: Device. */ +struct panthor_device *ptdev; + +/** @memattr: Value to program to the AS_MEMATTR register. */ +u64 memattr; + +/** @pgtbl_ops: Page table operations. */ +struct io_pgtable_ops *pgtbl_ops; + +/** @root_page_table: Stores the root page table pointer. */ +void *root_page_table; + +/** +* @op_lock: Lock used to serialize operations on a VM. +* +* The serialization of jobs queued to the VM_BIND queue is already +* taken care of by drm_sched, but we need to serialize synchronous +* and asynchronous VM_BIND request. This is what this lock is for. +*/ +struct mutex op_lock; + +/** +* @op_ctx: The context attached to the currently executing VM operation. +* +* NULL when no operation is in progress. +*/ +struct panthor_vm_op_ctx *op_ctx; + +/** +* @mm: Memory management object representing the auto-VA/kernel-VA. +* +* Used to auto-allocate VA space for kernel-managed objects (tiler +* heaps, ...). +* +* For the MCU VM, this is managing the VA range that's used to map +* all shared interfaces. +* +* For user VMs, the range is specified by userspace, and must not +* exceed half of the VA space addressable. +*/ +struct drm_mm mm; + +/** @mm_lock: Lock protecting the @mm field. */ +struct mutex mm_lock; + +/** @kernel_auto_va: Automatic VA-range for kernel BOs. */ +struct { +/** @start: Start of the automatic VA-range for kernel BOs. */ +u64 start; + +/** @size: Size of the automatic VA-range for kernel BOs. */ +u64 end; + } kernel_auto_va; + +/** @as: Address space related fields. */ +struct { +/** +* @id: ID of the address space this VM is bound to. +* +* A value of -1 means the VM is inactive/not bound. +*/ +int id; + +/** +* @lru_node: Used to instead the VM in the panthor_mmu::as::lru_list. +* +* Active VMs should not be inserted in the LRU list. +*/ +struct list_head lru_node; + } as; + +/** +* @heaps: Tiler heap related fields. +*/ +struct { +/** +* @pool: The heap pool attached to this VM. +* +* Will stay NULL until someone creates a heap context on this VM. +*/ +struct panthor_heap_pool *pool; + +/** @lock: Lock used to protect access to @pool. */ +struct mutex lock; + } heaps; + +/** @node: Used to insert the VM in the panthor_mmu::vm::list. */ +struct list_head node; + +/** @for_mcu: True if this is the MCU VM. */ +bool for_mcu; + +/** +* @destroyed: True if the VM was destroyed. +* +* No further bind requests should be queued to a destroyed VM. +*/ +bool destroyed; + +/** +* @unusable: True if the VM has turned unusable because something +* bad happened during an asynchronous request. +* +* We don't try to recover from such failures, because this implies +* informing userspace about the specific operation that failed, and +* hoping the userspace driver can replay things from there. This all +* sounds very complicated for little gain. +* +* Instead, we should just flag the VM as unusable, and fail any +* further request targeting this VM. +* +* We also provide a way to query a VM state, so userspace can destroy +* it and create a new one. +* +* As an analogy, this would be mapped to a VK_ERROR_DEVICE_LOST +* situation, where the logical device needs to be re-created. +*/ +bool unusable; + +/** +* @unhandled_fault: Unhandled fault happened. +* +* This should be reported to the scheduler, and the queue/group be +* flagged as faulty as a result. +*/ +bool unhandled_fault; +}; + +/** +* struct panthor_vm_bind_job - VM bind job +*/ +struct panthor_vm_bind_job { +/** @base: Inherit from drm_sched_job. */ +struct drm_sched_job base; + +/** @refcount: Reference count. */ +struct kref refcount; + +/** @cleanup_op_ctx_work: Work used to cleanup the VM operation context. */ +struct work_struct cleanup_op_ctx_work; + +/** @vm: VM targeted by the VM operation. */ +struct panthor_vm *vm; + +/** @ctx: Operation context. */ +struct panthor_vm_op_ctx ctx; +}; + +/** +* @pt_cache: Cache used to allocate MMU page tables. +* +* The pre-allocation pattern forces us to over-allocate to plan for +* the worst case scenario, and return the pages we didn't use. +* +* Having a kmem_cache allows us to speed allocations. +*/ +static struct kmem_cache *pt_cache; + +/** +* alloc_pt() - Custom page table allocator +* @cookie: Cookie passed at page table allocation time. +* @size: Size of the page table. This size should be fixed, +* and determined at creation time based on the granule size. +* @gfp: GFP flags. +* +* We want a custom allocator so we can use a cache for page table +* allocations and amortize the cost of the over-reservation that's +* done to allow asynchronous VM operations. +*
drivers/gpu/drm/panthor/panthor_mmu.c line 445
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+* Return: non-NULL on success, NULL if the allocation failed for any +* reason. +*/ +static void *alloc_pt (void *cookie,size_t size,gfp_t gfp) +{ +struct panthor_vm *vm = cookie; +void *page; + +/* Allocation of the root page table happening during init. */ +if (unlikely (!vm->pgtbl_ops)) {
I'm not that keen on using pgtbl_ops as the proxy for this. Can we use root_page_table instead?
At the moment if the IOMMU code ever did multiple allocations during alloc_io_pgtable_ops() then we'd overwrite root_page_table and screw up on the free path.
If we use root_page_table == NULL as the check then things will 'cleanly' fail by falling through to the non-root case in that case.
Of course this really looks like we should have had a different allocator for the root table but I'm not (re)opening that can of worms! ;)
And of course it doesn't make any sense for the IOMMU code to do multiple allocations so this is all rather academic - but maybe one day there will be a different page table structure (16K pages maybe?).
Quoted 41 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +41
-0+struct page *p; + +drm_WARN_ON (&vm->ptdev->base, vm->op_ctx); + p =alloc_pages_node (dev_to_node (vm->ptdev->base.dev), + gfp | __GFP_ZERO,get_order (size)); + page = p ?page_address (p) :NULL ; + vm->root_page_table = page; +return page; + } + +/* We're not supposed to have anything bigger than 4k here, because we picked a +* 4k granule size at init time. +*/ +if (drm_WARN_ON (&vm->ptdev->base, size !=SZ_4K )) +return NULL ; + +/* We must have some op_ctx attached to the VM and it must have at least one +* free page. +*/ +if (drm_WARN_ON (&vm->ptdev->base, !vm->op_ctx) || +drm_WARN_ON (&vm->ptdev->base, + vm->op_ctx->rsvd_page_tables.ptr >= vm->op_ctx->rsvd_page_tables.count)) +return NULL ; + + page = vm->op_ctx->rsvd_page_tables.pages[vm->op_ctx->rsvd_page_tables.ptr++]; +memset (page,0 ,SZ_4K ); + +/* Page table entries don't use virtual addresses, which trips out +* kmemleak. kmemleak_alloc_phys() might work, but physical addresses +* are mixed with other fields, and I fear kmemleak won't detect that +* either. +* +* Let's just ignore memory passed to the page-table driver for now. +*/ +kmemleak_ignore (page); +return page; +} + +/** +* @free_pt() - Custom page table free function +* @cookie: Cookie passed at page table allocation time.
drivers/gpu/drm/panthor/panthor_mmu.c line 496
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+* @data: Page table to free. +* @size: Size of the page table. This size should be fixed, +* and determined at creation time based on the granule size. +*/ +static void free_pt (void *cookie,void *data,size_t size) +{ +struct panthor_vm *vm = cookie; + +if (unlikely (vm->root_page_table == data)) { +free_pages ((unsigned long )data,get_order (size));
Maybe add "vm->root_page_table = NULL;"?
Quoted 451 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +451
-0+return ; + } + +if (drm_WARN_ON (&vm->ptdev->base, size !=SZ_4K )) +return ; + +/* Return the page to the pt_cache. */ +kmem_cache_free (pt_cache, data); +} + +static int wait_ready (struct panthor_device *ptdev,u32 as_nr) +{ +int ret; +u32 val; + +/* Wait for the MMU status to indicate there is no active command, in +* case one is pending. +*/ + ret =readl_relaxed_poll_timeout_atomic (ptdev->iomem +AS_STATUS (as_nr), + val, !(val &AS_STATUS_AS_ACTIVE ), +10 ,100000 ); + +if (ret) { +panthor_device_schedule_reset (ptdev); +drm_err (&ptdev->base,"AS_ACTIVE bit stuck\n" ); + } + +return ret; +} + +static int write_cmd (struct panthor_device *ptdev,u32 as_nr,u32 cmd) +{ +int status; + +/* write AS_COMMAND when MMU is ready to accept another command */ + status =wait_ready (ptdev, as_nr); +if (!status) +gpu_write (ptdev,AS_COMMAND (as_nr), cmd); + +return status; +} + +static void lock_region (struct panthor_device *ptdev,u32 as_nr, +u64 region_start,u64 size) +{ +u8 region_width; +u64 region; +u64 region_end = region_start + size; + +if (!size) +return ; + +/* +* The locked region is a naturally aligned power of 2 block encoded as +* log2 minus(1). +* Calculate the desired start/end and look for the highest bit which +* differs. The smallest naturally aligned block must include this bit +* change, the desired region starts with this bit (and subsequent bits) +* zeroed and ends with the bit (and subsequent bits) set to one. +*/ + region_width =max (fls64 (region_start ^ (region_end -1 )), +const_ilog2 (AS_LOCK_REGION_MIN_SIZE )) -1 ; + +/* +* Mask off the low bits of region_start (which would be ignored by +* the hardware anyway) +*/ + region_start &=GENMASK_ULL (63 , region_width); + + region = region_width | region_start; + +/* Lock the region that needs to be updated */ +gpu_write (ptdev,AS_LOCKADDR_LO (as_nr),lower_32_bits (region)); +gpu_write (ptdev,AS_LOCKADDR_HI (as_nr),upper_32_bits (region)); +write_cmd (ptdev, as_nr,AS_COMMAND_LOCK ); +} + +static int mmu_hw_do_operation_locked (struct panthor_device *ptdev,int as_nr, +u64 iova,u64 size,u32 op) +{ +lockdep_assert_held (&ptdev->mmu->as.slots_lock); + +if (as_nr <0 ) +return 0 ; + +if (op !=AS_COMMAND_UNLOCK ) +lock_region (ptdev, as_nr, iova, size); + +/* Run the MMU operation */ +write_cmd (ptdev, as_nr, op); + +/* Wait for the flush to complete */ +return wait_ready (ptdev, as_nr); +} + +static int mmu_hw_do_operation (struct panthor_vm *vm, +u64 iova,u64 size,u32 op) +{ +struct panthor_device *ptdev = vm->ptdev; +int ret; + +mutex_lock (&ptdev->mmu->as.slots_lock); + ret =mmu_hw_do_operation_locked (ptdev, vm->as.id, iova, size, op); +mutex_unlock (&ptdev->mmu->as.slots_lock); + +return ret; +} + +static int panthor_mmu_as_enable (struct panthor_device *ptdev,u32 as_nr, +u64 transtab,u64 transcfg,u64 memattr) +{ +int ret; + + ret =mmu_hw_do_operation_locked (ptdev, as_nr,0 , ~0ULL ,AS_COMMAND_FLUSH_MEM ); +if (ret) +return ret; + +gpu_write (ptdev,AS_TRANSTAB_LO (as_nr),lower_32_bits (transtab)); +gpu_write (ptdev,AS_TRANSTAB_HI (as_nr),upper_32_bits (transtab)); + +gpu_write (ptdev,AS_MEMATTR_LO (as_nr),lower_32_bits (memattr)); +gpu_write (ptdev,AS_MEMATTR_HI (as_nr),upper_32_bits (memattr)); + +gpu_write (ptdev,AS_TRANSCFG_LO (as_nr),lower_32_bits (transcfg)); +gpu_write (ptdev,AS_TRANSCFG_HI (as_nr),upper_32_bits (transcfg)); + +return write_cmd (ptdev, as_nr,AS_COMMAND_UPDATE ); +} + +static int panthor_mmu_as_disable (struct panthor_device *ptdev,u32 as_nr) +{ +int ret; + + ret =mmu_hw_do_operation_locked (ptdev, as_nr,0 , ~0ULL ,AS_COMMAND_FLUSH_MEM ); +if (ret) +return ret; + +gpu_write (ptdev,AS_TRANSTAB_LO (as_nr),0 ); +gpu_write (ptdev,AS_TRANSTAB_HI (as_nr),0 ); + +gpu_write (ptdev,AS_MEMATTR_LO (as_nr),0 ); +gpu_write (ptdev,AS_MEMATTR_HI (as_nr),0 ); + +gpu_write (ptdev,AS_TRANSCFG_LO (as_nr),AS_TRANSCFG_ADRMODE_UNMAPPED ); +gpu_write (ptdev,AS_TRANSCFG_HI (as_nr),0 ); + +return write_cmd (ptdev, as_nr,AS_COMMAND_UPDATE ); +} + +static u32 panthor_mmu_fault_mask (struct panthor_device *ptdev,u32 value) +{ +/* Bits 16 to 31 mean REQ_COMPLETE. */ +return value &GENMASK (15 ,0 ); +} + +static u32 panthor_mmu_as_fault_mask (struct panthor_device *ptdev,u32 as) +{ +return BIT (as); +} + +/** +* panthor_vm_has_unhandled_faults() - Check if a VM has unhandled faults +* @vm: VM to check. +* +* Return: true if the VM has unhandled faults, false otherwise. +*/ +bool panthor_vm_has_unhandled_faults (struct panthor_vm *vm) +{ +return vm->unhandled_fault; +} + +/** +* panthor_vm_active() - Flag a VM as active +* @VM: VM to flag as active. +* +* Assigns an address space to a VM so it can be used by the GPU/MCU. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_active (struct panthor_vm *vm) +{ +struct panthor_device *ptdev = vm->ptdev; +struct io_pgtable_cfg *cfg = &io_pgtable_ops_to_pgtable (vm->pgtbl_ops)->cfg; +int ret =0 , as, cookie; +u64 transtab, transcfg; + +if (!drm_dev_enter (&ptdev->base, &cookie)) +return -ENODEV ; + +mutex_lock (&ptdev->mmu->as.slots_lock); + + as = vm->as.id; +if (as >=0 ) { +/* Unhandled pagefault on this AS, the MMU was disabled. We need to +* re-enable the MMU after clearing+unmasking the AS interrupts. +*/ +if (ptdev->mmu->as.faulty_mask &panthor_mmu_as_fault_mask (ptdev, as)) + goto out_enable_as; + + goto out_unlock; + } + +/* Check for a free AS */ +if (vm->for_mcu) { +drm_WARN_ON (&ptdev->base, ptdev->mmu->as.alloc_mask &BIT (0 )); + as =0 ; + }else { + as =ffz (ptdev->mmu->as.alloc_mask |BIT (0 )); + } + +if (!(BIT (as) & ptdev->gpu_info.as_present)) { +struct panthor_vm *lru_vm; + + lru_vm =list_first_entry_or_null (&ptdev->mmu->as.lru_list, + struct panthor_vm, + as.lru_node); +if (drm_WARN_ON (&ptdev->base, !lru_vm)) { + ret = -EBUSY ; + goto out_unlock; + } + +list_del_init (&lru_vm->as.lru_node); + as = lru_vm->as.id; + + lru_vm->as.id =-1 ; + }else { +set_bit (as, &ptdev->mmu->as.alloc_mask); + } + +/* Assign the free or reclaimed AS to the FD */ + vm->as.id = as; + ptdev->mmu->as.slots[as].vm = vm; + +out_enable_as: + transtab = cfg->arm_lpae_s1_cfg.ttbr; + transcfg =AS_TRANSCFG_PTW_MEMATTR_WB | +AS_TRANSCFG_PTW_RA | +AS_TRANSCFG_ADRMODE_AARCH64_4K ; +if (ptdev->coherent) + transcfg |=AS_TRANSCFG_PTW_SH_OS ; + +/* If the VM is re-activated, we clear the fault. */ + vm->unhandled_fault = false; + +/* Unhandled pagefault on this AS, clear the fault and re-enable interrupts +* before enabling the AS. +*/ +if (ptdev->mmu->as.faulty_mask &panthor_mmu_as_fault_mask (ptdev, as)) { +gpu_write (ptdev,MMU_INT_CLEAR ,panthor_mmu_as_fault_mask (ptdev, as)); + ptdev->mmu->as.faulty_mask &= ~panthor_mmu_as_fault_mask (ptdev, as); +gpu_write (ptdev,MMU_INT_MASK , ~ptdev->mmu->as.faulty_mask); + } + + ret =panthor_mmu_as_enable (vm->ptdev, vm->as.id, transtab, transcfg, vm->memattr); + +out_unlock: +mutex_unlock (&ptdev->mmu->as.slots_lock); +drm_dev_exit (cookie); +return ret; +} + +/** +* panthor_vm_idle() - Flag a VM idle +* @VM: VM to flag as idle. +* +* When we know the GPU is done with the VM (no more jobs to process), +* we can relinquish the AS slot attached to this VM, if any. +* +* We don't release the slot immediately, but instead place the VM in +* the LRU list, so it can be evicted if another VM needs an AS slot. +* This way, VMs keep attached to the AS they were given until we run +* out of free slot, limiting the number of MMU operations (TLB flush +* and other AS updates). +*/ +void panthor_vm_idle (struct panthor_vm *vm) +{ +struct panthor_device *ptdev = vm->ptdev; + +mutex_lock (&ptdev->mmu->as.slots_lock); +if (vm->as.id >=0 &&list_empty (&vm->as.lru_node)) +list_add_tail (&vm->as.lru_node, &ptdev->mmu->as.lru_list); +mutex_unlock (&ptdev->mmu->as.slots_lock); +} + +static void panthor_vm_stop (struct panthor_vm *vm) +{ +drm_sched_stop (&vm->sched,NULL ); +} + +static void panthor_vm_start (struct panthor_vm *vm) +{ +drm_sched_start (&vm->sched, true); +} + +/** +* panthor_vm_as() - Get the AS slot attached to a VM +* @vm: VM to get the AS slot of. +* +* Return: -1 if the VM is not assigned an AS slot yet, >= 0 otherwise. +*/ +int panthor_vm_as (struct panthor_vm *vm) +{ +return vm->as.id; +} + +static size_t get_pgsize (u64 addr,size_t size,size_t *count) +{ +/* +* io-pgtable only operates on multiple pages within a single table +* entry, so we need to split at boundaries of the table size, i.e. +* the next block size up. The distance from address A to the next +* boundary of block size B is logically B - A % B, but in unsigned +* two's complement where B is a power of two we get the equivalence +* B - A % B == (B - A) % B == (n * B - A) % B, and choose n = 0 :) +*/ +size_t blk_offset = -addr %SZ_2M ; + +if (blk_offset || size <SZ_2M ) { + *count =min_not_zero (blk_offset, size) /SZ_4K ; +return SZ_4K ; + } + blk_offset = -addr %SZ_1G ?:SZ_1G ; + *count =min (blk_offset, size) /SZ_2M ; +return SZ_2M ; +} + +static int panthor_vm_flush_range (struct panthor_vm *vm,u64 iova,u64 size) +{ +struct panthor_device *ptdev = vm->ptdev; +int ret =0 , cookie; + +if (vm->as.id <0 ) +return 0 ; + +/* If the device is unplugged, we just silently skip the flush. */ +if (!drm_dev_enter (&ptdev->base, &cookie)) +return 0 ; + +/* Flush the PTs only if we're already awake */ +if (pm_runtime_active (ptdev->base.dev)) + ret =mmu_hw_do_operation (vm, iova, size,AS_COMMAND_FLUSH_PT ); + +drm_dev_exit (cookie); +return ret; +} + +static int panthor_vm_unmap_pages (struct panthor_vm *vm,u64 iova,u64 size) +{ +struct panthor_device *ptdev = vm->ptdev; +struct io_pgtable_ops *ops = vm->pgtbl_ops; +u64 offset =0 ; + +drm_dbg (&ptdev->base,"unmap: as=%d, iova=%llx, len=%llx" , vm->as.id, iova, size); + +while (offset < size) { +size_t unmapped_sz =0 , pgcount; +size_t pgsize =get_pgsize (iova + offset, size - offset, &pgcount); + + unmapped_sz = ops->unmap_pages (ops, iova + offset, pgsize, pgcount,NULL ); + +if (drm_WARN_ON (&ptdev->base, unmapped_sz != pgsize * pgcount)) { +drm_err (&ptdev->base,"failed to unmap range %llx-%llx (requested range %llx-%llx)\n" , + iova + offset + unmapped_sz, + iova + offset + pgsize * pgcount, + iova, iova + size); +panthor_vm_flush_range (vm, iova, offset + unmapped_sz); +return -EINVAL ; + } + offset += unmapped_sz; + } + +return panthor_vm_flush_range (vm, iova, size); +} + +static int +panthor_vm_map_pages (struct panthor_vm *vm,u64 iova,int prot, +struct sg_table *sgt,u64 offset,u64 size) +{ +struct panthor_device *ptdev = vm->ptdev; +unsigned int count; +struct scatterlist *sgl; +struct io_pgtable_ops *ops = vm->pgtbl_ops; +u64 start_iova = iova; +int ret; + +if (!size) +return 0 ; + +for_each_sgtable_dma_sg (sgt, sgl, count) { +dma_addr_t paddr =sg_dma_address (sgl); +size_t len =sg_dma_len (sgl); + +if (len <= offset) { + offset -= len; +continue ; + } + + paddr -= offset; + len -= offset; + +if (size >=0 ) { + len =min_t (size_t, len, size); + size -= len; + } + +drm_dbg (&ptdev->base,"map: as=%d, iova=%llx, paddr=%pad, len=%zx" , + vm->as.id, iova, &paddr, len); + +while (len) { +size_t pgcount, mapped =0 ; +size_t pgsize =get_pgsize (iova | paddr, len, &pgcount); + + ret = ops->map_pages (ops, iova, paddr, pgsize, pgcount, prot, +GFP_KERNEL , &mapped); + iova += mapped; + paddr += mapped; + len -= mapped; + +if (drm_WARN_ON (&ptdev->base, !ret && !mapped)) + ret = -ENOMEM ; + +if (ret) { +/* If something failed, unmap what we've already mapped before +* returning. The unmap call is not supposed to fail. +*/ +drm_WARN_ON (&ptdev->base, +panthor_vm_unmap_pages (vm, start_iova, + iova - start_iova)); +return ret; + } + } + +if (!size) +break ; + } + +return panthor_vm_flush_range (vm, start_iova, iova - start_iova); +} + +static int flags_to_prot (u32 flags) +{ +int prot =0 ; + +if (flags &DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC ) + prot |=IOMMU_NOEXEC ; + +if (!(flags &DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED )) + prot |=IOMMU_CACHE ; + +if (flags &DRM_PANTHOR_VM_BIND_OP_MAP_READONLY ) + prot |=IOMMU_READ ;
drivers/gpu/drm/panthor/panthor_mmu.c line 957
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+else + prot |=IOMMU_READ |IOMMU_WRITE ; + +return prot; +} + +/** +* panthor_vm_alloc_va() - Allocate a region in the auto-va space +* @VM: VM to allocate a region on. +* @size: Size of the region.
kerneldoc needs updating for the new arguments.
drivers/gpu/drm/panthor/panthor_mmu.c line 965
Quoted 8 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +8
-0+* +* Some GPU objects, like heap chunks, are fully managed by the kernel and +* need to be mapped to the userspace VM, in the region reserved for kernel +* objects. +* +* This function takes care of allocating a region in this reserved space. +* +* Return: A valid pointer on success, and ERR_PTR() otherwise.
Returns an error code not a pointer.
Quoted 6 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +6
-0+*/ +int +panthor_vm_alloc_va (struct panthor_vm *vm,u64 va,u64 size, +struct drm_mm_node *va_node) +{ +int ret;
drivers/gpu/drm/panthor/panthor_mmu.c line 981
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+ +if (!size || (size & ~PAGE_MASK )) +return -EINVAL ; + +if (va !=PANTHOR_VM_KERNEL_AUTO_VA && (va & ~PAGE_MASK )) +return -EINVAL ; + +mutex_lock (&vm->mm_lock); +if (va !=PANTHOR_VM_KERNEL_AUTO_VA ) { +memset (va_node,0 ,sizeof (*va_node));
This memset() seems redundant. I certainly can't see why it's only required on this path.
Quoted 365 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +365
-0+ va_node->start = va; + va_node->size = size; + ret =drm_mm_reserve_node (&vm->mm, va_node); + }else { + ret =drm_mm_insert_node_in_range (&vm->mm, va_node, size, + size >=SZ_2M ?SZ_2M :SZ_4K , +0 , vm->kernel_auto_va.start, + vm->kernel_auto_va.end, +DRM_MM_INSERT_BEST ); + } +mutex_unlock (&vm->mm_lock); + +return ret; +} + +/** +* panthor_vm_free_va() - Free a region allocated with panthor_vm_alloc_va() +* @VM: VM to free the region on. +* @mm_node: Memory node representing the region to free. +*/ +void panthor_vm_free_va (struct panthor_vm *vm,struct drm_mm_node *va_node) +{ +mutex_lock (&vm->mm_lock); +drm_mm_remove_node (va_node); +mutex_unlock (&vm->mm_lock); +} + +static void panthor_vm_bo_put (struct drm_gpuvm_bo *vm_bo) +{ +struct panthor_gem_object *bo =to_panthor_bo (vm_bo->obj); +bool unpin; + +/* We must retain the GEM before calling drm_gpuvm_bo_put(), +* otherwise the mutex might be destroyed while we hold it. +*/ +drm_gem_object_get (&bo->base.base); +mutex_lock (&bo->gpuva_list_lock); + unpin =drm_gpuvm_bo_put (vm_bo); +mutex_unlock (&bo->gpuva_list_lock); + +/* If the vm_bo object was destroyed, release the pin reference that +* was hold by this object. +*/ +if (unpin && !bo->base.base.import_attach) +drm_gem_shmem_unpin (&bo->base); + +drm_gem_object_put (&bo->base.base); +} + +static void panthor_vm_cleanup_op_ctx (struct panthor_vm_op_ctx *op_ctx, +struct panthor_vm *vm) +{ +struct panthor_vma *vma, *tmp_vma; + +u32 remaining_pt_count = op_ctx->rsvd_page_tables.count - + op_ctx->rsvd_page_tables.ptr; + +if (remaining_pt_count) { +kmem_cache_free_bulk (pt_cache, remaining_pt_count, + op_ctx->rsvd_page_tables.pages + + op_ctx->rsvd_page_tables.ptr); + } + +kfree (op_ctx->rsvd_page_tables.pages); +memset (&op_ctx->rsvd_page_tables,0 ,sizeof (op_ctx->rsvd_page_tables)); + +if (op_ctx->map.vm_bo) +panthor_vm_bo_put (op_ctx->map.vm_bo); + +kfree (op_ctx->map.new_vma); +kfree (op_ctx->map.next_vma); +kfree (op_ctx->map.prev_vma); +memset (&op_ctx->map,0 ,sizeof (op_ctx->map)); + +list_for_each_entry_safe (vma, tmp_vma, &op_ctx->returned_vmas, node) { +list_del (&vma->node); +panthor_vm_bo_put (vma->base.vm_bo); +kfree (vma); + } +} + +#define PANTHOR_VM_BIND_OP_MAP_FLAGS \ + (DRM_PANTHOR_VM_BIND_OP_MAP_READONLY | \ + DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC | \ + DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED | \ + DRM_PANTHOR_VM_BIND_OP_TYPE_MASK) + +static int panthor_vm_prepare_map_op_ctx (struct panthor_vm_op_ctx *op_ctx, +struct panthor_vm *vm, +struct panthor_gem_object *bo, +u64 offset, +u64 size,u64 va, +u32 flags) +{ +struct drm_gpuvm_bo *preallocated_vm_bo; +struct sg_table *sgt =NULL ; +u64 pt_count; +int ret; + +if (!bo) +return -EINVAL ; + +if ((flags & ~PANTHOR_VM_BIND_OP_MAP_FLAGS ) || + (flags &DRM_PANTHOR_VM_BIND_OP_TYPE_MASK ) !=DRM_PANTHOR_VM_BIND_OP_TYPE_MAP ) +return -EINVAL ; + +/* Make sure the VA and size are aligned and in-bounds. */ +if (size > bo->base.base.size || offset > bo->base.base.size - size) +return -EINVAL ; + +/* If the BO has an exclusive VM attached, it can't be mapped to other VMs. */ +if (bo->exclusive_vm_root_gem && + bo->exclusive_vm_root_gem !=panthor_vm_root_gem (vm)) +return -EINVAL ; + +memset (op_ctx,0 ,sizeof (*op_ctx)); +INIT_LIST_HEAD (&op_ctx->returned_vmas); + op_ctx->flags = flags; + op_ctx->va.range = size; + op_ctx->va.addr = va; + + op_ctx->map.new_vma =kzalloc (sizeof (*op_ctx->map.new_vma),GFP_KERNEL ); + op_ctx->map.next_vma =kzalloc (sizeof (*op_ctx->map.next_vma),GFP_KERNEL ); + op_ctx->map.prev_vma =kzalloc (sizeof (*op_ctx->map.prev_vma),GFP_KERNEL ); +if (!op_ctx->map.new_vma || !op_ctx->map.next_vma || !op_ctx->map.prev_vma) { + ret = -ENOMEM ; + goto err_cleanup; + } + +if (!bo->base.base.import_attach) { +/* Pre-reserve the BO pages, so the map operation doesn't have to +* allocate. +*/ + ret =drm_gem_shmem_pin (&bo->base); +if (ret) + goto err_cleanup; + } + + sgt =drm_gem_shmem_get_pages_sgt (&bo->base); +if (IS_ERR (sgt)) { +if (!bo->base.base.import_attach) +drm_gem_shmem_unpin (&bo->base); + + ret =PTR_ERR (sgt); + goto err_cleanup; + } + + op_ctx->map.sgt = sgt; + + preallocated_vm_bo =drm_gpuvm_bo_create (&vm->base, &bo->base.base); +if (!preallocated_vm_bo) { +if (!bo->base.base.import_attach) +drm_gem_shmem_unpin (&bo->base); + + ret = -ENOMEM ; + goto err_cleanup; + } + +mutex_lock (&bo->gpuva_list_lock); + op_ctx->map.vm_bo =drm_gpuvm_bo_obtain_prealloc (preallocated_vm_bo); +mutex_unlock (&bo->gpuva_list_lock); + +/* If the a vm_bo for this <VM,BO> combination exists, it already +* retains a pin ref, and we can release the one we took earlier. +* +* If our pre-allocated vm_bo is picked, it now retains the pin ref, +* which will be released in panthor_vm_bo_put(). +*/ +if (preallocated_vm_bo != op_ctx->map.vm_bo && + !bo->base.base.import_attach) +drm_gem_shmem_unpin (&bo->base); + + op_ctx->map.bo_offset = offset; + +/* L1, L2 and L3 page tables. +* We could optimize L3 allocation by iterating over the sgt and merging +* 2M contiguous blocks, but it's simpler to over-provision and return +* the pages if they're not used. +*/ + pt_count = ((ALIGN (va + size,1ull <<39 ) -ALIGN_DOWN (va,1ull <<39 )) >>39 ) + + ((ALIGN (va + size,1ull <<30 ) -ALIGN_DOWN (va,1ull <<30 )) >>30 ) + + ((ALIGN (va + size,1ull <<21 ) -ALIGN_DOWN (va,1ull <<21 )) >>21 ); + + op_ctx->rsvd_page_tables.pages =kcalloc (pt_count, +sizeof (*op_ctx->rsvd_page_tables.pages), +GFP_KERNEL ); +if (!op_ctx->rsvd_page_tables.pages) + goto err_cleanup; + + ret =kmem_cache_alloc_bulk (pt_cache,GFP_KERNEL , pt_count, + op_ctx->rsvd_page_tables.pages); + op_ctx->rsvd_page_tables.count = ret; +if (ret != pt_count) { + ret = -ENOMEM ; + goto err_cleanup; + } + +/* Insert BO into the extobj list last, when we know nothing can fail. */ +drm_gpuvm_bo_extobj_add (op_ctx->map.vm_bo); + +return 0 ; + +err_cleanup: +panthor_vm_cleanup_op_ctx (op_ctx, vm); +return ret; +} + +static int panthor_vm_prepare_unmap_op_ctx (struct panthor_vm_op_ctx *op_ctx, +struct panthor_vm *vm, +u64 va,u64 size) +{ +u32 pt_count =0 ; +int ret; + +memset (op_ctx,0 ,sizeof (*op_ctx)); +INIT_LIST_HEAD (&op_ctx->returned_vmas); + op_ctx->va.range = size; + op_ctx->va.addr = va; + op_ctx->flags =DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP ; + +/* Pre-allocate L3 page tables to account for the split-2M-block +* situation on unmap. +*/ +if (va !=ALIGN (va,SZ_2M )) + pt_count++; + +if (va + size !=ALIGN (va + size,SZ_2M ) && +ALIGN (va + size,SZ_2M ) !=ALIGN (va,SZ_2M )) + pt_count++; + +if (pt_count) { + op_ctx->rsvd_page_tables.pages =kcalloc (pt_count, +sizeof (*op_ctx->rsvd_page_tables.pages), +GFP_KERNEL ); +if (!op_ctx->rsvd_page_tables.pages) + goto err_cleanup; + + ret =kmem_cache_alloc_bulk (pt_cache,GFP_KERNEL , pt_count, + op_ctx->rsvd_page_tables.pages); +if (ret != pt_count) { + ret = -ENOMEM ; + goto err_cleanup; + } + op_ctx->rsvd_page_tables.count = pt_count; + } + +return 0 ; + +err_cleanup: +panthor_vm_cleanup_op_ctx (op_ctx, vm); +return ret; +} + +static void panthor_vm_prepare_sync_only_op_ctx (struct panthor_vm_op_ctx *op_ctx, +struct panthor_vm *vm) +{ +memset (op_ctx,0 ,sizeof (*op_ctx)); +INIT_LIST_HEAD (&op_ctx->returned_vmas); + op_ctx->flags =DRM_PANTHOR_VM_BIND_OP_TYPE_SYNC_ONLY ; +} + +/** +* panthor_vm_get_bo_for_va() - Get the GEM object mapped at a virtual address +* @vm: VM to look into. +* @va: Virtual address to search for. +* @bo_offset: Offset of the GEM object mapped at this virtual address. +* Only valid on success. +* +* The object returned by this function might no longer be mapped when the +* function returns. It's the caller responsibility to ensure there's no +* concurrent map/unmap operations making the returned value invalid, or +* make sure it doesn't matter if the object is no longer mapped. +* +* Return: A valid pointer on success, an ERR_PTR() otherwise. +*/ +struct panthor_gem_object * +panthor_vm_get_bo_for_va (struct panthor_vm *vm,u64 va,u64 *bo_offset) +{ +struct panthor_gem_object *bo =ERR_PTR (-ENOENT ); +struct drm_gpuva *gpuva; +struct panthor_vma *vma; + +/* Take the VM lock to prevent concurrent map/unmap operations. */ +mutex_lock (&vm->op_lock); + gpuva =drm_gpuva_find_first (&vm->base, va,1 ); + vma = gpuva ? container_of(gpuva, struct panthor_vma, base) : NULL; +if (vma && vma->base.gem.obj) { +drm_gem_object_get (vma->base.gem.obj); + bo =to_panthor_bo (vma->base.gem.obj); + *bo_offset = vma->base.gem.offset; + } +mutex_unlock (&vm->op_lock); + +return bo; +} + +#define PANTHOR_VM_MIN_KERNEL_VA_SIZE SZ_256M + +static u64 +panthor_vm_create_get_user_va_range (const struct drm_panthor_vm_create *args, +u64 full_va_range) +{ +u64 user_va_range; + +/* Make sure we have a minimum amount of VA space for kernel objects. */ +if (full_va_range <PANTHOR_VM_MIN_KERNEL_VA_SIZE ) +return 0 ; + +if (args->user_va_range) { +/* Use the user provided value if != 0. */ + user_va_range = args->user_va_range; + }else if (TASK_SIZE_OF (current) < full_va_range) { +/* If the task VM size is smaller than the GPU VA range, pick this +* as our default user VA range, so userspace can CPU/GPU map buffers +* at the same address. +*/ + user_va_range =TASK_SIZE_OF (current); + }else { +/* If the GPU VA range is smaller than the task VM size, we +* just have to live with the fact we won't be able to map +* all buffers at the same GPU/CPU address. +* +* If the GPU VA range is bigger than 4G (more than 32-bit of +* VA), we split the range in two, and assign half of it to +* the user and the other half to the kernel, if it's not, we +* keep the kernel VA space as small as possible. +*/ + user_va_range = full_va_range >SZ_4G ? + full_va_range /2 : + full_va_range -PANTHOR_VM_MIN_KERNEL_VA_SIZE ; + } + +if (full_va_range -PANTHOR_VM_MIN_KERNEL_VA_SIZE < user_va_range) + user_va_range = full_va_range -PANTHOR_VM_MIN_KERNEL_VA_SIZE ; + +return user_va_range; +} + +#define PANTHOR_VM_CREATE_FLAGS 0 + +static int +panthor_vm_create_check_args (const struct panthor_device *ptdev, +const struct drm_panthor_vm_create *args, +u64 *kernel_va_start,u64 *kernel_va_range) +{ +u32 va_bits =GPU_MMU_FEATURES_VA_BITS (ptdev->gpu_info.mmu_features); +u64 full_va_range =1ull <<min_t (u32, va_bits,sizeof (unsigned long ) *8 ); +u64 user_va_range; + +if (args->flags & ~PANTHOR_VM_CREATE_FLAGS ) +return -EINVAL ; + + user_va_range =panthor_vm_create_get_user_va_range (args, full_va_range); +if (!user_va_range || (args->user_va_range && args->user_va_range > user_va_range)) +return -EINVAL ; + +/* Pick a kernel VA range that's a power of two, to have a clear split. */ + *kernel_va_range =rounddown_pow_of_two (full_va_range - user_va_range); + *kernel_va_start = full_va_range - *kernel_va_range; +return 0 ; +} + +/* +* Only 32 VMs per open file. If that becomes a limiting factor, we can +* increase this number.
drivers/gpu/drm/panthor/panthor_mmu.c line 1356
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+*/ +#define PANTHOR_MAX_VMS_PER_FILE 32 + +/** +* panthor_vm_pool_create_vm() - Create a VM +* @pool: The VM to create this VM on. +* @kernel_va_start: Start of the region reserved for kernel objects. +* @kernel_va_range: Size of the region reserved for kernel objects. +* +* Return: 0 on success, a negative error code otherwise.
Actually returns the (positive) id on success.
Quoted 384 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +384
-0+*/ +int panthor_vm_pool_create_vm (struct panthor_device *ptdev, +struct panthor_vm_pool *pool, +struct drm_panthor_vm_create *args) +{ +u64 kernel_va_start, kernel_va_range; +struct panthor_vm *vm; +int ret; +u32 id; + + ret =panthor_vm_create_check_args (ptdev, args, &kernel_va_start, &kernel_va_range); +if (ret) +return ret; + + vm =panthor_vm_create (ptdev, false, kernel_va_start, kernel_va_range, + kernel_va_start, kernel_va_range); +if (IS_ERR (vm)) +return PTR_ERR (vm); + + ret =xa_alloc (&pool->xa, &id, vm, +XA_LIMIT (1 ,PANTHOR_MAX_VMS_PER_FILE ),GFP_KERNEL ); + +if (ret) { +panthor_vm_put (vm); +return ret; + } + + args->user_va_range = kernel_va_start; +return id; +} + +static void panthor_vm_destroy (struct panthor_vm *vm) +{ +if (!vm) +return ; + + vm->destroyed = true; + +mutex_lock (&vm->heaps.lock); +panthor_heap_pool_destroy (vm->heaps.pool); + vm->heaps.pool =NULL ; +mutex_unlock (&vm->heaps.lock); + +drm_WARN_ON (&vm->ptdev->base, +panthor_vm_unmap_range (vm, vm->base.mm_start, vm->base.mm_range)); +panthor_vm_put (vm); +} + +/** +* panthor_vm_destroy() - Destroy a VM. +* @pool: VM pool. +* @handle: VM handle. +* +* This function doesn't free the VM object or its resources, it just kills +* all mappings, and makes sure nothing can be mapped after that point. +* +* If there was any active jobs at the time this function is called, these +* jobs should experience page faults and be killed as a result. +* +* The VM resources are freed when the last reference on the VM object is +* dropped. +*/ +int panthor_vm_pool_destroy_vm (struct panthor_vm_pool *pool,u32 handle) +{ +struct panthor_vm *vm; + + vm =xa_erase (&pool->xa, handle); + +panthor_vm_destroy (vm); + +return vm ?0 : -EINVAL ; +} + +/** +* panthor_vm_pool_get_vm() - Retrieve VM object bound to a VM handle +* @pool: VM pool to check. +* @handle: Handle of the VM to retrieve. +* +* Return: A valid pointer if the VM exists, NULL otherwise. +*/ +struct panthor_vm * +panthor_vm_pool_get_vm (struct panthor_vm_pool *pool,u32 handle) +{ +struct panthor_vm *vm; + + vm =panthor_vm_get (xa_load (&pool->xa, handle)); + +return vm; +} + +/** +* panthor_vm_pool_destroy() - Destroy a VM pool. +* @pfile: File. +* +* Destroy all VMs in the pool, and release the pool resources. +* +* Note that VMs can outlive the pool they were created from if other +* objects hold a reference to there VMs. +*/ +void panthor_vm_pool_destroy (struct panthor_file *pfile) +{ +struct panthor_vm *vm; +unsigned long i; + +if (!pfile->vms) +return ; + +xa_for_each (&pfile->vms->xa, i, vm) +panthor_vm_destroy (vm); + +xa_destroy (&pfile->vms->xa); +kfree (pfile->vms); +} + +/** +* panthor_vm_pool_create() - Create a VM pool +* @pfile: File. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_pool_create (struct panthor_file *pfile) +{ + pfile->vms =kzalloc (sizeof (*pfile->vms),GFP_KERNEL ); +if (!pfile->vms) +return -ENOMEM ; + +xa_init_flags (&pfile->vms->xa,XA_FLAGS_ALLOC1 ); +return 0 ; +} + +/* dummy TLB ops, the real TLB flush happens in panthor_vm_flush_range() */ +static void mmu_tlb_flush_all (void *cookie) +{ +} + +static void mmu_tlb_flush_walk (unsigned long iova,size_t size,size_t granule,void *cookie) +{ +} + +static const struct iommu_flush_ops mmu_tlb_ops = { + .tlb_flush_all = mmu_tlb_flush_all, + .tlb_flush_walk = mmu_tlb_flush_walk, +}; + +static const char *access_type_name (struct panthor_device *ptdev, +u32 fault_status) +{ +switch (fault_status &AS_FAULTSTATUS_ACCESS_TYPE_MASK ) { +case AS_FAULTSTATUS_ACCESS_TYPE_ATOMIC : +return "ATOMIC" ; +case AS_FAULTSTATUS_ACCESS_TYPE_READ : +return "READ" ; +case AS_FAULTSTATUS_ACCESS_TYPE_WRITE : +return "WRITE" ; +case AS_FAULTSTATUS_ACCESS_TYPE_EX : +return "EXECUTE" ; +default : +drm_WARN_ON (&ptdev->base,1 ); +return NULL ; + } +} + +static void panthor_mmu_irq_handler (struct panthor_device *ptdev,u32 status) +{ +bool has_unhandled_faults = false; + + status =panthor_mmu_fault_mask (ptdev, status); +while (status) { +u32 as =ffs (status | (status >>16 )) -1 ; +u32 mask =panthor_mmu_as_fault_mask (ptdev, as); +u32 new_int_mask; +u64 addr; +u32 fault_status; +u32 exception_type; +u32 access_type; +u32 source_id; + + fault_status =gpu_read (ptdev,AS_FAULTSTATUS (as)); + addr =gpu_read (ptdev,AS_FAULTADDRESS_LO (as)); + addr |= (u64 )gpu_read (ptdev,AS_FAULTADDRESS_HI (as)) <<32 ; + +/* decode the fault status */ + exception_type = fault_status &0xFF ; + access_type = (fault_status >>8 ) &0x3 ; + source_id = (fault_status >>16 ); + +mutex_lock (&ptdev->mmu->as.slots_lock); + + ptdev->mmu->as.faulty_mask |= mask; + new_int_mask = +panthor_mmu_fault_mask (ptdev, ~ptdev->mmu->as.faulty_mask); + +/* terminal fault, print info about the fault */ +drm_err (&ptdev->base, +"Unhandled Page fault in AS%d at VA 0x%016llX\n" +"raw fault status: 0x%X\n" +"decoded fault status: %s\n" +"exception type 0x%X: %s\n" +"access type 0x%X: %s\n" +"source id 0x%X\n" , + as, addr, + fault_status, + (fault_status & (1 <<10 ) ?"DECODER FAULT" :"SLAVE FAULT" ), + exception_type,panthor_exception_name (ptdev, exception_type), + access_type,access_type_name (ptdev, fault_status), + source_id); + +/* Ignore MMU interrupts on this AS until it's been +* re-enabled. +*/ + ptdev->mmu->irq.mask = new_int_mask; +gpu_write (ptdev,MMU_INT_MASK , new_int_mask); + +if (ptdev->mmu->as.slots[as].vm) + ptdev->mmu->as.slots[as].vm->unhandled_fault = true; + +/* Disable the MMU to kill jobs on this AS. */ +panthor_mmu_as_disable (ptdev, as); +mutex_unlock (&ptdev->mmu->as.slots_lock); + + status &= ~mask; + has_unhandled_faults = true; + } + +if (has_unhandled_faults) +panthor_sched_report_mmu_fault (ptdev); +} +PANTHOR_IRQ_HANDLER (mmu,MMU , panthor_mmu_irq_handler); + +/** +* panthor_mmu_suspend() - Suspend the MMU logic +* @ptdev: Device. +* +* All we do here is de-assign the AS slots on all active VMs, so things +* get flushed to the main memory, and no further access to these VMs are +* possible. +* +* We also suspend the MMU IRQ. +*/ +void panthor_mmu_suspend (struct panthor_device *ptdev) +{ +mutex_lock (&ptdev->mmu->as.slots_lock); +for (u32 i =0 ; i <ARRAY_SIZE (ptdev->mmu->as.slots); i++) { +struct panthor_vm *vm = ptdev->mmu->as.slots[i].vm; + +if (vm) { +drm_WARN_ON (&ptdev->base,panthor_mmu_as_disable (ptdev, i)); + vm->as.id =-1 ; +list_del_init (&vm->as.lru_node); + ptdev->mmu->as.slots[i].vm =NULL ; + } + } +mutex_unlock (&ptdev->mmu->as.slots_lock); + +panthor_mmu_irq_suspend (&ptdev->mmu->irq); +} + +/** +* panthor_mmu_resume() - Resume the MMU logic +* @ptdev: Device. +* +* Resume the IRQ. +* +* We don't re-enable previously active VMs. We assume other parts of the +* driver will call panthor_vm_active() on the VMs they intend to use. +*/ +void panthor_mmu_resume (struct panthor_device *ptdev) +{ +mutex_lock (&ptdev->mmu->as.slots_lock); + ptdev->mmu->as.alloc_mask =0 ; + ptdev->mmu->as.faulty_mask =0 ; +mutex_unlock (&ptdev->mmu->as.slots_lock); + +panthor_mmu_irq_resume (&ptdev->mmu->irq,panthor_mmu_fault_mask (ptdev, ~0 )); +} + +/** +* panthor_mmu_pre_reset() - Prepare for a reset +* @ptdev: Device. +* +* Suspend the IRQ, and make sure all VM_BIND queues are stopped, so we +* don't get asked to do a VM operation while the GPU is down. +* +* We don't cleanly shutdown the AS slots here, because the reset might +* come from an AS_ACTIVE_BIT stuck situation. +*/ +void panthor_mmu_pre_reset (struct panthor_device *ptdev) +{ +struct panthor_vm *vm; + +panthor_mmu_irq_suspend (&ptdev->mmu->irq); + +mutex_lock (&ptdev->mmu->vm.lock); + ptdev->mmu->vm.reset_in_progress = true; +list_for_each_entry (vm, &ptdev->mmu->vm.list, node) +panthor_vm_stop (vm); +mutex_unlock (&ptdev->mmu->vm.lock); +} + +/** +* panthor_mmu_post_reset() - Restore things after a reset +* @ptdev: Device. +* +* Put the MMU logic back in action after a reset. That implies resuming the +* IRQ and re-enabling the VM_BIND queues. +*/ +void panthor_mmu_post_reset (struct panthor_device *ptdev) +{ +struct panthor_vm *vm; + +mutex_lock (&ptdev->mmu->as.slots_lock); + +/* Now that the reset is effective, we can assume that none of the +* AS slots are setup, and clear the faulty flags too. +*/ + ptdev->mmu->as.alloc_mask =0 ; + ptdev->mmu->as.faulty_mask =0 ; + +for (u32 i =0 ; i <ARRAY_SIZE (ptdev->mmu->as.slots); i++) { +struct panthor_vm *vm = ptdev->mmu->as.slots[i].vm; + +if (vm) { + vm->as.id =-1 ; +list_del_init (&vm->as.lru_node); + ptdev->mmu->as.slots[i].vm =NULL ; + } + } + +mutex_unlock (&ptdev->mmu->as.slots_lock); + +panthor_mmu_irq_resume (&ptdev->mmu->irq,panthor_mmu_fault_mask (ptdev, ~0 )); + +/* Restart the VM_BIND queues. */ +mutex_lock (&ptdev->mmu->vm.lock); +list_for_each_entry (vm, &ptdev->mmu->vm.list, node) { +panthor_vm_start (vm); + } + ptdev->mmu->vm.reset_in_progress = false; +mutex_unlock (&ptdev->mmu->vm.lock); +} + +static void panthor_vm_free (struct drm_gpuvm *gpuvm) +{ + struct panthor_vm *vm = container_of(gpuvm, struct panthor_vm, base); +struct panthor_device *ptdev = vm->ptdev; + +mutex_lock (&vm->heaps.lock); +if (drm_WARN_ON (&ptdev->base, vm->heaps.pool)) +panthor_heap_pool_destroy (vm->heaps.pool); +mutex_unlock (&vm->heaps.lock); +mutex_destroy (&vm->heaps.lock); + +mutex_lock (&ptdev->mmu->vm.lock); +list_del (&vm->node); +/* Restore the scheduler state so we can call drm_sched_entity_destroy() +* and drm_sched_fini(). If get there, that means we have no job left +* and no new jobs can be queued, so we can start the scheduler without +* risking interfering with the reset. +*/ +if (ptdev->mmu->vm.reset_in_progress) +panthor_vm_start (vm); +mutex_unlock (&ptdev->mmu->vm.lock); + +drm_sched_entity_destroy (&vm->entity); +drm_sched_fini (&vm->sched); + +mutex_lock (&ptdev->mmu->as.slots_lock); +if (vm->as.id >=0 ) { +int cookie; + +if (drm_dev_enter (&ptdev->base, &cookie)) { +panthor_mmu_as_disable (ptdev, vm->as.id); +drm_dev_exit (cookie); + } + + ptdev->mmu->as.slots[vm->as.id].vm =NULL ; +clear_bit (vm->as.id, &ptdev->mmu->as.alloc_mask); +list_del (&vm->as.lru_node); + } +mutex_unlock (&ptdev->mmu->as.slots_lock); + +free_io_pgtable_ops (vm->pgtbl_ops); + +drm_mm_takedown (&vm->mm);
drivers/gpu/drm/panthor/panthor_mmu.c line 1750
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+kfree (vm); +} + +/** +* panthor_vm_put() - Release a reference on a VM +* @vm: VM to release the reference on. Can be NULL. +*/ +void panthor_vm_put (struct panthor_vm *vm) +{ +static_assert (offsetof(struct panthor_vm , base) ==0 );
Yuk! ;)
I'd prefer:
drm_gpuvm_put(vm ? &vm->base : NULL);
which my compiler turns into the same thing rather than relying on the type punning. You can keep the static_assert if you like, but I don't like relying on it for correct code generation. Although I'll admit I couldn't actually get the compiler to produce incorrect code when I tried.
Quoted 749 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +749
-0+drm_gpuvm_put (&vm->base); +} + +/** +* panthor_vm_get() - Get a VM reference +* @vm: VM to get the reference on. Can be NULL. +* +* Return: @vm value. +*/ +struct panthor_vm *panthor_vm_get (struct panthor_vm *vm) +{ +if (vm) +drm_gpuvm_get (&vm->base); + +return vm; +} + +/** +* panthor_vm_get_heap_pool() - Get the heap pool attached to a VM +* @vm: VM to query the heap pool on. +* @create: True if the heap pool should be created when it doesn't exist. +* +* Heap pools are per-VM. This function allows one to retrieve the heap pool +* attached to a VM. +* +* If no heap pool exists yet, and @create is true, we create one. +* +* The returned panthor_heap_pool should be released with panthor_heap_pool_put(). +* +* Return: A valid pointer on success, an ERR_PTR() otherwise. +*/ +struct panthor_heap_pool *panthor_vm_get_heap_pool (struct panthor_vm *vm,bool create) +{ +struct panthor_heap_pool *pool; + +mutex_lock (&vm->heaps.lock); +if (!vm->heaps.pool && create) { +if (vm->destroyed) + pool =ERR_PTR (-EINVAL ); +else + pool =panthor_heap_pool_create (vm->ptdev, vm); + +if (!IS_ERR (pool)) + vm->heaps.pool =panthor_heap_pool_get (pool); + }else { + pool =panthor_heap_pool_get (vm->heaps.pool); + } +mutex_unlock (&vm->heaps.lock); + +return pool; +} + +static u64 mair_to_memattr (u64 mair) +{ +u64 memattr =0 ; +u32 i; + +for (i =0 ; i <8 ; i++) { +u8 in_attr = mair >> (8 * i), out_attr; +u8 outer = in_attr >>4 , inner = in_attr &0xf ; + +/* For caching to be enabled, inner and outer caching policy +* have to be both write-back, if one of them is write-through +* or non-cacheable, we just choose non-cacheable. Device +* memory is also translated to non-cacheable. +*/ +if (!(outer &3 ) || !(outer &4 ) || !(inner &4 )) { + out_attr =AS_MEMATTR_AARCH64_INNER_OUTER_NC | +AS_MEMATTR_AARCH64_SH_MIDGARD_INNER | +AS_MEMATTR_AARCH64_INNER_ALLOC_EXPL (false, false); + }else { +/* Use SH_CPU_INNER mode so SH_IS, which is used when +* IOMMU_CACHE is set, actually maps to the standard +* definition of inner-shareable and not Mali's +* internal-shareable mode. +*/ + out_attr =AS_MEMATTR_AARCH64_INNER_OUTER_WB | +AS_MEMATTR_AARCH64_SH_CPU_INNER | +AS_MEMATTR_AARCH64_INNER_ALLOC_EXPL (inner &1 , inner &2 ); + } + + memattr |= (u64 )out_attr << (8 * i); + } + +return memattr; +} + +static void panthor_vma_link (struct panthor_vm *vm, +struct panthor_vma *vma, +struct drm_gpuvm_bo *vm_bo) +{ +struct panthor_gem_object *bo =to_panthor_bo (vma->base.gem.obj); + +mutex_lock (&bo->gpuva_list_lock); +drm_gpuva_link (&vma->base, vm_bo); +drm_gpuvm_bo_put (vm_bo); +mutex_unlock (&bo->gpuva_list_lock); +} + +static void panthor_vma_unlink (struct panthor_vm *vm, +struct panthor_vma *vma) +{ +struct panthor_gem_object *bo =to_panthor_bo (vma->base.gem.obj); +struct drm_gpuvm_bo *vm_bo =drm_gpuvm_bo_get (vma->base.vm_bo); + +mutex_lock (&bo->gpuva_list_lock); +drm_gpuva_unlink (&vma->base); +mutex_unlock (&bo->gpuva_list_lock); + +/* drm_gpuva_unlink() release the vm_bo, but we manually retained it +* when entering this function, so we can implement deferred VMA +* destruction. Re-assign it here. +*/ + vma->base.vm_bo = vm_bo; +list_add_tail (&vma->node, &vm->op_ctx->returned_vmas); +} + +static void panthor_vma_init (struct panthor_vma *vma,u32 flags) +{ +INIT_LIST_HEAD (&vma->node); + vma->flags = flags; +} + +#define PANTHOR_VM_MAP_FLAGS \ + (DRM_PANTHOR_VM_BIND_OP_MAP_READONLY | \ + DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC | \ + DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED) + +static int panthor_gpuva_sm_step_map (struct drm_gpuva_op *op,void *priv) +{ +struct panthor_vm *vm = priv; +struct panthor_vm_op_ctx *op_ctx = vm->op_ctx; +struct panthor_vma *vma = op_ctx->map.new_vma; +int ret; + +panthor_vma_init (vma, op_ctx->flags &PANTHOR_VM_MAP_FLAGS ); + + ret =panthor_vm_map_pages (vm, op->map.va.addr,flags_to_prot (vma->flags), + op_ctx->map.sgt, op->map.gem.offset, + op->map.va.range); +if (ret) +return ret; + +/* Ref owned by the mapping now, clear the obj field so we don't release the +* pinning/obj ref behind GPUVA's back. +*/ +drm_gpuva_map (&vm->base, &vma->base, &op->map); +panthor_vma_link (vm, op_ctx->map.new_vma, op_ctx->map.vm_bo); + op_ctx->map.vm_bo =NULL ; + op_ctx->map.new_vma =NULL ; +return 0 ; +} + +static int panthor_gpuva_sm_step_remap (struct drm_gpuva_op *op, +void *priv) +{ + struct panthor_vma *unmap_vma = container_of(op->remap.unmap->va, struct panthor_vma, base); +struct panthor_vm *vm = priv; +struct panthor_vm_op_ctx *op_ctx = vm->op_ctx; +struct drm_gpuva *prev_va =NULL , *next_va =NULL ; +u64 unmap_start, unmap_range; +int ret; + +drm_gpuva_op_remap_to_unmap_range (&op->remap, &unmap_start, &unmap_range); + ret =panthor_vm_unmap_pages (vm, unmap_start, unmap_range); +if (ret) +return ret; + +if (op->remap.prev) { +panthor_vma_init (op_ctx->map.prev_vma, unmap_vma->flags); + prev_va = &op_ctx->map.prev_vma->base; + } + +if (op->remap.next) { +panthor_vma_init (op_ctx->map.next_vma, unmap_vma->flags); + next_va = &op_ctx->map.next_vma->base; + } + +drm_gpuva_remap (prev_va, next_va, &op->remap); + +if (prev_va) { +panthor_vma_link (vm, op_ctx->map.prev_vma, op->remap.unmap->va->vm_bo); + op_ctx->map.prev_vma =NULL ; + } + +if (next_va) { +panthor_vma_link (vm, op_ctx->map.next_vma, op->remap.unmap->va->vm_bo); + op_ctx->map.next_vma =NULL ; + } + +panthor_vma_unlink (vm, unmap_vma); +return 0 ; +} + +static int panthor_gpuva_sm_step_unmap (struct drm_gpuva_op *op, +void *priv) +{ + struct panthor_vma *unmap_vma = container_of(op->unmap.va, struct panthor_vma, base); +struct panthor_vm *vm = priv; +int ret; + + ret =panthor_vm_unmap_pages (vm, unmap_vma->base.va.addr, + unmap_vma->base.va.range); +if (drm_WARN_ON (&vm->ptdev->base, ret)) +return ret; + +drm_gpuva_unmap (&op->unmap); +panthor_vma_unlink (vm, unmap_vma); +return 0 ; +} + +static const struct drm_gpuvm_ops panthor_gpuvm_ops = { + .vm_free = panthor_vm_free, + .sm_step_map = panthor_gpuva_sm_step_map, + .sm_step_remap = panthor_gpuva_sm_step_remap, + .sm_step_unmap = panthor_gpuva_sm_step_unmap, +}; + +/** +* panthor_vm_resv() - Get the dma_resv object attached to a VM. +* @vm: VM to get the dma_resv of. +* +* Return: A dma_resv object. +*/ +struct dma_resv *panthor_vm_resv (struct panthor_vm *vm) +{ +return drm_gpuvm_resv (&vm->base); +} + +struct drm_gem_object *panthor_vm_root_gem (struct panthor_vm *vm) +{ +if (!vm) +return NULL ; + +return vm->base.r_obj; +} + +static int +panthor_vm_exec_op (struct panthor_vm *vm,struct panthor_vm_op_ctx *op, +bool flag_vm_unusable_on_failure) +{ +u32 op_type = op->flags &DRM_PANTHOR_VM_BIND_OP_TYPE_MASK ; +int ret; + +if (op_type ==DRM_PANTHOR_VM_BIND_OP_TYPE_SYNC_ONLY ) +return 0 ; + +mutex_lock (&vm->op_lock); + vm->op_ctx = op; +switch (op_type) { +case DRM_PANTHOR_VM_BIND_OP_TYPE_MAP : +if (vm->unusable) { + ret = -EINVAL ; +break ; + } + + ret =drm_gpuvm_sm_map (&vm->base, vm, op->va.addr, op->va.range, + op->map.vm_bo->obj, op->map.bo_offset); +break ; + +case DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP : + ret =drm_gpuvm_sm_unmap (&vm->base, vm, op->va.addr, op->va.range); +break ; + +default : + ret = -EINVAL ; +break ; + } + +if (ret && flag_vm_unusable_on_failure) + vm->unusable = true; + + vm->op_ctx =NULL ; +mutex_unlock (&vm->op_lock); + +return ret; +} + +static struct dma_fence * +panthor_vm_bind_run_job (struct drm_sched_job *sched_job) +{ + struct panthor_vm_bind_job *job = container_of(sched_job, struct panthor_vm_bind_job, base); +bool cookie; +int ret; + +/* Not only we report an error whose result is propagated to the +* drm_sched finished fence, but we also flag the VM as unusable, because +* a failure in the async VM_BIND results in an inconsistent state. VM needs +* to be destroyed and recreated. +*/ + cookie =dma_fence_begin_signalling (); + ret =panthor_vm_exec_op (job->vm, &job->ctx, true); +dma_fence_end_signalling (cookie); + +return ret ?ERR_PTR (ret) :NULL ; +} + +static void panthor_vm_bind_job_release (struct kref *kref) +{ + struct panthor_vm_bind_job *job = container_of(kref, struct panthor_vm_bind_job, refcount); + +if (job->base.s_fence) +drm_sched_job_cleanup (&job->base); + +panthor_vm_cleanup_op_ctx (&job->ctx, job->vm); +panthor_vm_put (job->vm); +kfree (job); +} + +/** +* panthor_vm_bind_job_put() - Release a VM_BIND job reference +* @sched_job: Job to release the reference on. +*/ +void panthor_vm_bind_job_put (struct drm_sched_job *sched_job) +{ +struct panthor_vm_bind_job *job = + container_of(sched_job, struct panthor_vm_bind_job, base); + +if (sched_job) +kref_put (&job->refcount, panthor_vm_bind_job_release); +} + +static void +panthor_vm_bind_free_job (struct drm_sched_job *sched_job) +{ +struct panthor_vm_bind_job *job = + container_of(sched_job, struct panthor_vm_bind_job, base); + +drm_sched_job_cleanup (sched_job); + +/* Do the heavy cleanups asynchronously, so we're out of the +* dma-signaling path and can acquire dma-resv locks safely. +*/ +queue_work (panthor_cleanup_wq, &job->cleanup_op_ctx_work); +} + +static enum drm_gpu_sched_stat +panthor_vm_bind_timedout_job (struct drm_sched_job *sched_job) +{ +WARN (1 ,"VM_BIND ops are synchronous for now, there should be no timeout!" ); +return DRM_GPU_SCHED_STAT_NOMINAL ; +} + +static const struct drm_sched_backend_ops panthor_vm_bind_ops = { + .run_job = panthor_vm_bind_run_job, + .free_job = panthor_vm_bind_free_job, + .timedout_job = panthor_vm_bind_timedout_job, +}; + +/** +* panthor_vm_create() - Create a VM +* @ptdev: Device. +* @for_mcu: True if this is the FW MCU VM. +* @kernel_va_start: Start of the range reserved for kernel BO mapping. +* @kernel_va_size: Size of the range reserved for kernel BO mapping. +* @auto_kernel_va_start: Start of the auto-VA kernel range. +* @auto_kernel_va_size: Size of the auto-VA kernel range. +* +* Return: A valid pointer on success, an ERR_PTR() otherwise. +*/ +struct panthor_vm * +panthor_vm_create (struct panthor_device *ptdev,bool for_mcu, +u64 kernel_va_start,u64 kernel_va_size, +u64 auto_kernel_va_start,u64 auto_kernel_va_size) +{ +u32 va_bits =GPU_MMU_FEATURES_VA_BITS (ptdev->gpu_info.mmu_features); +u32 pa_bits =GPU_MMU_FEATURES_PA_BITS (ptdev->gpu_info.mmu_features); +u64 full_va_range =1ull <<min_t (u32, va_bits,sizeof (unsigned long ) *8 ); +struct drm_gem_object *dummy_gem; +struct drm_gpu_scheduler *sched; +struct io_pgtable_cfg pgtbl_cfg; +u64 mair, min_va, va_range; +struct panthor_vm *vm; +int ret; + + vm =kzalloc (sizeof (*vm),GFP_KERNEL ); +if (!vm) +return ERR_PTR (-ENOMEM ); + +/* We allocate a dummy GEM for the VM. */ + dummy_gem =drm_gpuvm_resv_object_alloc (&ptdev->base); +if (!dummy_gem) { + ret = -ENOMEM ; + goto err_free_vm; + } + +mutex_init (&vm->heaps.lock); + vm->for_mcu = for_mcu; + vm->ptdev = ptdev; +mutex_init (&vm->op_lock); + +if (for_mcu) { +/* CSF MCU is a cortex M7, and can only address 4G */ + min_va =0 ; + va_range =SZ_4G ; + }else { + min_va =0 ; + va_range = full_va_range; + } + +mutex_init (&vm->mm_lock); +drm_mm_init (&vm->mm, kernel_va_start, kernel_va_size); + vm->kernel_auto_va.start = auto_kernel_va_start; + vm->kernel_auto_va.end = vm->kernel_auto_va.start + auto_kernel_va_size -1 ; + +INIT_LIST_HEAD (&vm->node); +INIT_LIST_HEAD (&vm->as.lru_node); + vm->as.id =-1 ; + + pgtbl_cfg = (struct io_pgtable_cfg ) { + .pgsize_bitmap =SZ_4K |SZ_2M , + .ias = va_bits, + .oas = pa_bits, + .coherent_walk = ptdev->coherent, + .tlb = &mmu_tlb_ops, + .iommu_dev = ptdev->base.dev, + .alloc = alloc_pt, + .free = free_pt, + }; + + vm->pgtbl_ops =alloc_io_pgtable_ops (ARM_64_LPAE_S1 , &pgtbl_cfg, vm); +if (!vm->pgtbl_ops) { + ret = -EINVAL ; + goto err_mm_takedown; + } + +/* Bind operations are synchronous for now, no timeout needed. */ + ret =drm_sched_init (&vm->sched, &panthor_vm_bind_ops, ptdev->mmu->vm.wq, +1 ,1 ,0 , +MAX_SCHEDULE_TIMEOUT ,NULL ,NULL , +"panthor-vm-bind" , ptdev->base.dev); +if (ret) + goto err_free_io_pgtable; + + sched = &vm->sched; + ret =drm_sched_entity_init (&vm->entity,0 , &sched,1 ,NULL ); +if (ret) + goto err_sched_fini; + + mair =io_pgtable_ops_to_pgtable (vm->pgtbl_ops)->cfg.arm_lpae_s1_cfg.mair; + vm->memattr =mair_to_memattr (mair); + +mutex_lock (&ptdev->mmu->vm.lock); +list_add_tail (&vm->node, &ptdev->mmu->vm.list); + +/* If a reset is in progress, stop the scheduler. */ +if (ptdev->mmu->vm.reset_in_progress) +panthor_vm_stop (vm); +mutex_unlock (&ptdev->mmu->vm.lock); + +/* We intentionally leave the reserved range to zero, because we want kernel VMAs +* to be handled the same way user VMAs are. +*/ +drm_gpuvm_init (&vm->base, + for_mcu ?"panthor-MCU-VM" :"panthor-GPU-VM" , +0 , &ptdev->base, dummy_gem, min_va, va_range,0 ,0 , + &panthor_gpuvm_ops); +drm_gem_object_put (dummy_gem); +return vm; + +err_sched_fini: +drm_sched_fini (&vm->sched); + +err_free_io_pgtable: +free_io_pgtable_ops (vm->pgtbl_ops); + +err_mm_takedown: +drm_mm_takedown (&vm->mm); +drm_gem_object_put (dummy_gem); + +err_free_vm: +kfree (vm); +return ERR_PTR (ret); +} + +static int +panthor_vm_bind_prepare_op_ctx (struct drm_file *file, +struct panthor_vm *vm, +const struct drm_panthor_vm_bind_op *op, +struct panthor_vm_op_ctx *op_ctx) +{ +struct drm_gem_object *gem; +int ret; + +/* Aligned on page size. */ +if ((op->va | op->size) & ~PAGE_MASK ) +return -EINVAL ; + +switch (op->flags &DRM_PANTHOR_VM_BIND_OP_TYPE_MASK ) { +case DRM_PANTHOR_VM_BIND_OP_TYPE_MAP : + gem =drm_gem_object_lookup (file, op->bo_handle); + ret =panthor_vm_prepare_map_op_ctx (op_ctx, vm, + gem ?to_panthor_bo (gem) :NULL , + op->bo_offset, + op->size, + op->va, + op->flags); +drm_gem_object_put (gem); +return ret; + +case DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP : +if (op->flags & ~DRM_PANTHOR_VM_BIND_OP_TYPE_MASK ) +return -EINVAL ; + +if (op->bo_handle || op->bo_offset) +return -EINVAL ; + +return panthor_vm_prepare_unmap_op_ctx (op_ctx, vm, op->va, op->size); + +case DRM_PANTHOR_VM_BIND_OP_TYPE_SYNC_ONLY : +if (op->flags & ~DRM_PANTHOR_VM_BIND_OP_TYPE_MASK ) +return -EINVAL ; + +if (op->bo_handle || op->bo_offset) +return -EINVAL ; + +if (op->va || op->size) +return -EINVAL ; + +if (!op->syncs.count) +return -EINVAL ; + +panthor_vm_prepare_sync_only_op_ctx (op_ctx, vm); +return 0 ; + +default : +return -EINVAL ; + } +} + +static void panthor_vm_bind_job_cleanup_op_ctx_work (struct work_struct *work) +{ +struct panthor_vm_bind_job *job = + container_of(work, struct panthor_vm_bind_job, cleanup_op_ctx_work); + +panthor_vm_cleanup_op_ctx (&job->ctx, job->vm); +panthor_vm_bind_job_put (&job->base); +} + +/** +* panthor_vm_bind_job_create() - Create a VM_BIND job +* @file: File. +* @vm: VM targeted by the VM_BIND job. +* @op: VM operation data. +* +* Return: A valid pointer on success, an ERR_PTR() otherwise. +*/ +struct drm_sched_job * +panthor_vm_bind_job_create (struct drm_file *file, +struct panthor_vm *vm, +const struct drm_panthor_vm_bind_op *op) +{ +struct panthor_vm_bind_job *job; +int ret; + +if (!vm) +return ERR_PTR (-EINVAL ); + +if (vm->destroyed || vm->unusable) +return ERR_PTR (-EINVAL ); + + job =kzalloc (sizeof (*job),GFP_KERNEL ); +if (!job) +return ERR_PTR (-ENOMEM ); + + ret =panthor_vm_bind_prepare_op_ctx (file, vm, op, &job->ctx); +if (ret) { +kfree (job); +return ERR_PTR (ret); + } + +INIT_WORK (&job->cleanup_op_ctx_work, panthor_vm_bind_job_cleanup_op_ctx_work); +kref_init (&job->refcount); + job->vm =panthor_vm_get (vm); + + ret =drm_sched_job_init (&job->base, &vm->entity,1 , vm); +if (ret) + goto err_put_job; + +return &job->base; + +err_put_job: +panthor_vm_bind_job_put (&job->base); +return ERR_PTR (ret); +} + +/** +* panthor_vm_bind_job_prepare_resvs() - Prepare VM_BIND job dma_resvs +* @exec: The locking/preparation context. +* @sched_job: The job to prepare resvs on. +* +* Locks and prepare the VM resv. +* +* If this is a map operation, locks and prepares the GEM resv. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_bind_job_prepare_resvs (struct drm_exec *exec, +struct drm_sched_job *sched_job) +{ + struct panthor_vm_bind_job *job = container_of(sched_job, struct panthor_vm_bind_job, base); +int ret; + +/* Acquire the VM lock an reserve a slot for this VM bind job. */ + ret =drm_gpuvm_prepare_vm (&job->vm->base, exec,1 ); +if (ret) +return ret; + +if (job->ctx.map.vm_bo) { +/* Lock/prepare the GEM being mapped. */ + ret =drm_exec_prepare_obj (exec, job->ctx.map.vm_bo->obj,1 ); +if (ret) +return ret; + } + +return 0 ; +} + +/** +* panthor_vm_bind_job_update_resvs() - Update the resv objects touched by a job +* @exec: drm_exec context. +* @sched_job: Job to update the resvs on. +*/ +void panthor_vm_bind_job_update_resvs (struct drm_exec *exec, +struct drm_sched_job *sched_job) +{ + struct panthor_vm_bind_job *job = container_of(sched_job, struct panthor_vm_bind_job, base); + +/* Explicit sync => we just register our job finished fence as bookkeep. */ +drm_gpuvm_resv_add_fence (&job->vm->base, exec, + &sched_job->s_fence->finished, +DMA_RESV_USAGE_BOOKKEEP , +DMA_RESV_USAGE_BOOKKEEP ); +} + +void panthor_vm_update_resvs (struct panthor_vm *vm,struct drm_exec *exec, +struct dma_fence *fence, +enum dma_resv_usage private_usage, +enum dma_resv_usage extobj_usage) +{ +drm_gpuvm_resv_add_fence (&vm->base, exec, fence, private_usage, extobj_usage); +} + +/** +* panthor_vm_bind_exec_sync_op() - Execute a VM_BIND operation synchronously. +* @file: File. +* @vm: VM targeted by the VM operation. +* @op: Data describing the VM operation. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_bind_exec_sync_op (struct drm_file *file, +struct panthor_vm *vm, +struct drm_panthor_vm_bind_op *op) +{ +struct panthor_vm_op_ctx op_ctx; +int ret; + +/* No sync objects allowed on synchronous operations. */ +if (op->syncs.count) +return -EINVAL ; + +if (!op->size) +return 0 ; + + ret =panthor_vm_bind_prepare_op_ctx (file, vm, op, &op_ctx); +if (ret) +return ret; + + ret =panthor_vm_exec_op (vm, &op_ctx, false); +panthor_vm_cleanup_op_ctx (&op_ctx, vm); + +return ret; +} + +/** +* panthor_vm_map_bo_range() - Map a GEM object range to a VM +* @vm: VM to map the GEM to. +* @bo: GEM object to map. +* @offset: Offset in the GEM object. +* @size: Size to map. +* @va: Virtual address to map the object to. +* @flags: Combination of drm_panthor_vm_bind_op_flags flags. +* Only map-related flags are valid. +* +* Internal use only. For userspace requests, use +* panthor_vm_bind_exec_sync_op() instead. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_map_bo_range (struct panthor_vm *vm,struct panthor_gem_object *bo, +u64 offset,u64 size,u64 va,u32 flags) +{ +struct panthor_vm_op_ctx op_ctx; +int ret; + + ret =panthor_vm_prepare_map_op_ctx (&op_ctx, vm, bo, offset, size, va, flags); +if (ret) +return ret; + + ret =panthor_vm_exec_op (vm, &op_ctx, false); +panthor_vm_cleanup_op_ctx (&op_ctx, vm); + +return ret; +} + +/** +* panthor_vm_unmap_range() - Unmap a portion of the VA space +* @vm: VM to unmap the region from. +* @va: Virtual address to unmap. Must be 4k aligned. +* @size: Size of the region to unmap. Must be 4k aligned. +* +* Internal use only. For userspace requests, use +* panthor_vm_bind_exec_sync_op() instead. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_unmap_range (struct panthor_vm *vm,u64 va,u64 size) +{ +struct panthor_vm_op_ctx op_ctx; +int ret; + + ret =panthor_vm_prepare_unmap_op_ctx (&op_ctx, vm, va, size); +if (ret) +return ret; + + ret =panthor_vm_exec_op (vm, &op_ctx, false); +panthor_vm_cleanup_op_ctx (&op_ctx, vm); + +return ret; +} + +/** +* panthor_vm_prepare_mapped_bos_resvs() - Prepare resvs on VM BOs. +* @exec: Locking/preparation context. +* @vm: VM targeted by the GPU job. +* @slot_count: Number of slots to reserve. +* +* GPU jobs assume all BOs bound to the VM at the time the job is submitted +* are available when the job is executed. In order to guarantee that, we +* need to reserve a slot on all BOs mapped to a VM and update this slot with +* the job fence after its submission. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_prepare_mapped_bos_resvs (struct drm_exec *exec,struct panthor_vm *vm, +u32 slot_count) +{ +int ret;
drivers/gpu/drm/panthor/panthor_mmu.c line 2509
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+ +/* Acquire the VM lock an reserve a slot for this GPU job. */ + ret =drm_gpuvm_prepare_vm (&vm->base, exec, slot_count); +if (ret) +return ret; + +/* VM operations are not protected by the VM resv-lock. We need to +* take the op_lock to make sure the shared_bos list is not updated +* while we're walking it. +*/
Is the above comment stale? AFAIK the shared_bos list doesn't exist anymore and this doesn't appear to relate to anything here.
Steve
Quoted 251 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +144
-0+return drm_gpuvm_prepare_objects (&vm->base, exec, slot_count); +} + +/** +* panthor_mmu_unplug() - Unplug the MMU logic +* @ptdev: Device. +* +* No access to the MMU regs should be done after this function is called. +* We suspend the IRQ and disable all VMs to guarantee that. +*/ +void panthor_mmu_unplug (struct panthor_device *ptdev) +{ +panthor_mmu_irq_suspend (&ptdev->mmu->irq); + +mutex_lock (&ptdev->mmu->as.slots_lock); +for (u32 i =0 ; i <ARRAY_SIZE (ptdev->mmu->as.slots); i++) { +struct panthor_vm *vm = ptdev->mmu->as.slots[i].vm; + +if (vm) { +drm_WARN_ON (&ptdev->base,panthor_mmu_as_disable (ptdev, i)); + vm->as.id =-1 ; +list_del_init (&vm->as.lru_node); +clear_bit (i, &ptdev->mmu->as.alloc_mask); + ptdev->mmu->as.slots[i].vm =NULL ; + } + } +mutex_unlock (&ptdev->mmu->as.slots_lock); +} + +static void panthor_mmu_release_wq (struct drm_device *ddev,void *res) +{ +destroy_workqueue (res); +} + +/** +* panthor_mmu_init() - Initialize the MMU logic. +* @ptdev: Device. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_mmu_init (struct panthor_device *ptdev) +{ +struct panthor_mmu *mmu; +int ret, irq; + + mmu =drmm_kzalloc (&ptdev->base,sizeof (*mmu),GFP_KERNEL ); +if (!mmu) +return -ENOMEM ; + +INIT_LIST_HEAD (&mmu->as.lru_list); + +drmm_mutex_init (&ptdev->base, &mmu->as.slots_lock); +INIT_LIST_HEAD (&mmu->vm.list); +drmm_mutex_init (&ptdev->base, &mmu->vm.lock); + + ptdev->mmu = mmu; + + irq =platform_get_irq_byname (to_platform_device (ptdev->base.dev),"mmu" ); +if (irq <=0 ) +return -ENODEV ; + + ret =panthor_request_mmu_irq (ptdev, &mmu->irq, irq, +panthor_mmu_fault_mask (ptdev, ~0 )); +if (ret) +return ret; + + mmu->vm.wq =alloc_workqueue ("panthor-vm-bind" ,WQ_UNBOUND ,0 ); +if (!mmu->vm.wq) +return -ENOMEM ; + +return drmm_add_action_or_reset (&ptdev->base, panthor_mmu_release_wq, mmu->vm.wq); +} + +#ifdef CONFIG_DEBUG_FS +static int show_vm_gpuvas (struct panthor_vm *vm,struct seq_file *m) +{ +int ret; + +mutex_lock (&vm->op_lock); + ret =drm_debugfs_gpuva_info (m, &vm->base); +mutex_unlock (&vm->op_lock); + +return ret; +} + +static int show_each_vm (struct seq_file *m,void *arg) +{ +struct drm_info_node *node = (struct drm_info_node *)m->private; +struct drm_device *ddev = node->minor->dev; + struct panthor_device *ptdev = container_of(ddev, struct panthor_device, base); +int (*show)(struct panthor_vm *,struct seq_file *) = node->info_ent->data; +struct panthor_vm *vm; +int ret =0 ; + +mutex_lock (&ptdev->mmu->vm.lock); +list_for_each_entry (vm, &ptdev->mmu->vm.list, node) { + ret =show (vm, m); +if (ret <0 ) +break ; + +seq_puts (m,"\n" ); + } +mutex_unlock (&ptdev->mmu->vm.lock); + +return ret; +} + +static struct drm_info_list panthor_mmu_debugfs_list[] = { +DRM_DEBUGFS_GPUVA_INFO (show_each_vm, show_vm_gpuvas), +}; + +/** +* panthor_mmu_debugfs_init() - Initialize MMU debugfs entries +* @minor: Minor. +*/ +void panthor_mmu_debugfs_init (struct drm_minor *minor) +{ +drm_debugfs_create_files (panthor_mmu_debugfs_list, +ARRAY_SIZE (panthor_mmu_debugfs_list), + minor->debugfs_root, minor); +} +#endif /* CONFIG_DEBUG_FS */ + +/** +* panthor_mmu_pt_cache_init() - Initialize the page table cache. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_mmu_pt_cache_init (void ) +{ + pt_cache =kmem_cache_create ("panthor-mmu-pt" ,SZ_4K ,SZ_4K ,0 ,NULL ); +if (!pt_cache) +return -ENOMEM ; + +return 0 ; +} + +/** +* panthor_mmu_pt_cache_fini() - Destroy the page table cache. +*/ +void panthor_mmu_pt_cache_fini (void ) +{ +kmem_cache_destroy (pt_cache); +}drivers/gpu/drm/panthor/panthor_mmu.h new file +101
-0@@ -0,0 +1,101 @@ +/* SPDX-License-Identifier: GPL-2.0 or MIT */ +/* Copyright 2019 Linaro, Ltd, Rob Herring <[email protected]> */ +/* Copyright 2023 Collabora ltd. */ + +#ifndef __PANTHOR_MMU_H__ +#define __PANTHOR_MMU_H__ + +#include <linux/dma-resv.h> + +struct drm_exec ; +struct drm_sched_job ; +struct panthor_gem_object ; +struct panthor_heap_pool ; +struct panthor_vm ; +struct panthor_vma ; +struct panthor_mmu ; + +int panthor_mmu_init (struct panthor_device *ptdev); +void panthor_mmu_unplug (struct panthor_device *ptdev); +void panthor_mmu_pre_reset (struct panthor_device *ptdev); +void panthor_mmu_post_reset (struct panthor_device *ptdev); +void panthor_mmu_suspend (struct panthor_device *ptdev); +void panthor_mmu_resume (struct panthor_device *ptdev); + +int panthor_vm_map_bo_range (struct panthor_vm *vm,struct panthor_gem_object *bo, +u64 offset,u64 size,u64 va,u32 flags); +int panthor_vm_unmap_range (struct panthor_vm *vm,u64 va,u64 size); +struct panthor_gem_object * +panthor_vm_get_bo_for_va (struct panthor_vm *vm,u64 va,u64 *bo_offset); + +int panthor_vm_active (struct panthor_vm *vm); +void panthor_vm_idle (struct panthor_vm *vm); +int panthor_vm_as (struct panthor_vm *vm); + +struct panthor_heap_pool * +panthor_vm_get_heap_pool (struct panthor_vm *vm,bool create); + +struct panthor_vm *panthor_vm_get (struct panthor_vm *vm); +void panthor_vm_put (struct panthor_vm *vm); +struct panthor_vm *panthor_vm_create (struct panthor_device *ptdev,bool for_mcu, +u64 kernel_va_start,u64 kernel_va_size, +u64 kernel_auto_va_start, +u64 kernel_auto_va_size); + +int panthor_vm_prepare_mapped_bos_resvs (struct drm_exec *exec, +struct panthor_vm *vm, +u32 slot_count); +int panthor_vm_add_bos_resvs_deps_to_job (struct panthor_vm *vm, +struct drm_sched_job *job); +void panthor_vm_add_job_fence_to_bos_resvs (struct panthor_vm *vm, +struct drm_sched_job *job); + +struct dma_resv *panthor_vm_resv (struct panthor_vm *vm); +struct drm_gem_object *panthor_vm_root_gem (struct panthor_vm *vm); + +void panthor_vm_pool_destroy (struct panthor_file *pfile); +int panthor_vm_pool_create (struct panthor_file *pfile); +int panthor_vm_pool_create_vm (struct panthor_device *ptdev, +struct panthor_vm_pool *pool, +struct drm_panthor_vm_create *args); +int panthor_vm_pool_destroy_vm (struct panthor_vm_pool *pool,u32 handle); +struct panthor_vm *panthor_vm_pool_get_vm (struct panthor_vm_pool *pool,u32 handle); + +bool panthor_vm_has_unhandled_faults (struct panthor_vm *vm); + +/* +* PANTHOR_VM_KERNEL_AUTO_VA: Use this magic address when you want the GEM +* logic to auto-allocate the virtual address in the reserved kernel VA range. +*/ +#define PANTHOR_VM_KERNEL_AUTO_VA ~0ull + +int panthor_vm_alloc_va (struct panthor_vm *vm,u64 va,u64 size, +struct drm_mm_node *va_node); +void panthor_vm_free_va (struct panthor_vm *vm,struct drm_mm_node *va_node); + +int panthor_vm_bind_exec_sync_op (struct drm_file *file, +struct panthor_vm *vm, +struct drm_panthor_vm_bind_op *op); + +struct drm_sched_job * +panthor_vm_bind_job_create (struct drm_file *file, +struct panthor_vm *vm, +const struct drm_panthor_vm_bind_op *op); +void panthor_vm_bind_job_put (struct drm_sched_job *job); +int panthor_vm_bind_job_prepare_resvs (struct drm_exec *exec, +struct drm_sched_job *job); +void panthor_vm_bind_job_update_resvs (struct drm_exec *exec,struct drm_sched_job *job); + +void panthor_vm_update_resvs (struct panthor_vm *vm,struct drm_exec *exec, +struct dma_fence *fence, +enum dma_resv_usage private_usage, +enum dma_resv_usage extobj_usage); + +int panthor_mmu_pt_cache_init (void ); +void panthor_mmu_pt_cache_fini (void ); + +#ifdef CONFIG_DEBUG_FS +void panthor_mmu_debugfs_init (struct drm_minor *minor); +#endif + +#endif
Boris Brezillon
maintainerRe: [PATCH v3 07/14] drm/panthor: Add the MMU/VM logical block
Hi Steve,
Quoted 16 lines · On Fri, 8 Dec 2023 14:28:05 +0000 Steven Price <[email protected]> wrote:
Quoted 16 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +16
-0+/** +* alloc_pt() - Custom page table allocator +* @cookie: Cookie passed at page table allocation time. +* @size: Size of the page table. This size should be fixed, +* and determined at creation time based on the granule size. +* @gfp: GFP flags. +* +* We want a custom allocator so we can use a cache for page table +* allocations and amortize the cost of the over-reservation that's +* done to allow asynchronous VM operations. +* +* Return: non-NULL on success, NULL if the allocation failed for any +* reason. +*/ +static void *alloc_pt (void *cookie,size_t size,gfp_t gfp) +{
Quoted 8 lines
drivers/gpu/drm/panthor/panthor_mmu.c line 445
Quoted 5 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +5
-0+struct panthor_vm *vm = cookie; +void *page; + +/* Allocation of the root page table happening during init. */ +if (unlikely (!vm->pgtbl_ops)) {I'm not that keen on using
pgtbl_opsas the proxy for this. Can we useroot_page_tableinstead?
Definitely, I actually intended to test ->root_page_table when I introduced this field, but somehow forgot to update this part of the code.
Quoted 60 lines
At the moment if the IOMMU code ever did multiple allocations during
alloc_io_pgtable_ops()then we'd overwriteroot_page_tableand screw up on the free path.If we use
root_page_table== NULL as the check then things will 'cleanly' fail by falling through to the non-root case in that case.Of course this really looks like we should have had a different allocator for the root table but I'm not (re)opening that can of worms! ;)
And of course it doesn't make any sense for the IOMMU code to do multiple allocations so this is all rather academic - but maybe one day there will be a different page table structure (16K pages maybe?).
Quoted 45 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +45
-0+struct page *p; + +drm_WARN_ON (&vm->ptdev->base, vm->op_ctx); + p =alloc_pages_node (dev_to_node (vm->ptdev->base.dev), + gfp | __GFP_ZERO,get_order (size)); + page = p ?page_address (p) :NULL ; + vm->root_page_table = page; +return page; + } + +/* We're not supposed to have anything bigger than 4k here, because we picked a +* 4k granule size at init time. +*/ +if (drm_WARN_ON (&vm->ptdev->base, size !=SZ_4K )) +return NULL ; + +/* We must have some op_ctx attached to the VM and it must have at least one +* free page. +*/ +if (drm_WARN_ON (&vm->ptdev->base, !vm->op_ctx) || +drm_WARN_ON (&vm->ptdev->base, + vm->op_ctx->rsvd_page_tables.ptr >= vm->op_ctx->rsvd_page_tables.count)) +return NULL ; + + page = vm->op_ctx->rsvd_page_tables.pages[vm->op_ctx->rsvd_page_tables.ptr++]; +memset (page,0 ,SZ_4K ); + +/* Page table entries don't use virtual addresses, which trips out +* kmemleak. kmemleak_alloc_phys() might work, but physical addresses +* are mixed with other fields, and I fear kmemleak won't detect that +* either. +* +* Let's just ignore memory passed to the page-table driver for now. +*/ +kmemleak_ignore (page); +return page; +} + +/** +* @free_pt() - Custom page table free function +* @cookie: Cookie passed at page table allocation time. +* @data: Page table to free. +* @size: Size of the page table. This size should be fixed, +* and determined at creation time based on the granule size. +*/
Quoted 8 lines
drivers/gpu/drm/panthor/panthor_mmu.c line 496
Quoted 6 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +6
-0+static void free_pt (void *cookie,void *data,size_t size) +{ +struct panthor_vm *vm = cookie; + +if (unlikely (vm->root_page_table == data)) { +free_pages ((unsigned long )data,get_order (size));Maybe add "
vm->root_page_table= NULL;"?
Sure.
Quoted 3 lines
Quoted 3 lines
(quoted diff) +2
-0+ return; + }
[...]
Quoted 6 lines
Quoted 4 lines
+/**
+ *panthor_vm_alloc_va()- Allocate a region in the auto-va space
+ * @VM: VM to allocate a region on.
+ * @size: Size of the region.kerneldoc needs updating for the new arguments.
Will fix.
Quoted 11 lines
Quoted 9 lines
+ *
+ * Some GPU objects, like heap chunks, are fully managed by the kernel and
+ * need to be mapped to the userspace VM, in the region reserved for kernel
+ * objects.
+ *
+ * This function takes care of allocating a region in this reserved space.
+ *
+ * Return: A valid pointer on success, andERR_PTR()otherwise.Returns an error code not a pointer.
And that too.
Quoted 11 lines
Quoted 11 lines
+ */
drivers/gpu/drm/panthor/panthor_mmu.c new file +9
-0+int +panthor_vm_alloc_va (struct panthor_vm *vm,u64 va,u64 size, +struct drm_mm_node *va_node) +{ +int ret; + +if (!size || (size & ~PAGE_MASK )) +return -EINVAL ; +
Quoted 8 lines
drivers/gpu/drm/panthor/panthor_mmu.c line 981
Quoted 6 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +6
-0+if (va !=PANTHOR_VM_KERNEL_AUTO_VA && (va & ~PAGE_MASK )) +return -EINVAL ; + +mutex_lock (&vm->mm_lock); +if (va !=PANTHOR_VM_KERNEL_AUTO_VA ) { +memset (va_node,0 ,sizeof (*va_node));This
memset()seems redundant.
If we assume the va_node is initialized to zero, it's indeed redundant. I'll update the doc to make this a caller requirement.
Quoted 2 lines
I certainly can't see why it's only
required on this path.
drm_mm_insert_node_in_range() seems to assign all fields explicitly, while, according to the doc [1], drm_mm_reserve_node() wants the caller to make sure the struct is zero-initialized, except for the start and size fields.
Quoted 15 lines
Quoted 15 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +14
-0+ va_node->start = va; + va_node->size = size; + ret =drm_mm_reserve_node (&vm->mm, va_node); + }else { + ret =drm_mm_insert_node_in_range (&vm->mm, va_node, size, + size >=SZ_2M ?SZ_2M :SZ_4K , +0 , vm->kernel_auto_va.start, + vm->kernel_auto_va.end, +DRM_MM_INSERT_BEST ); + } +mutex_unlock (&vm->mm_lock); + +return ret; +}
[...]
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +7
-0+/* +* Only 32 VMs per open file. If that becomes a limiting factor, we can +* increase this number. +*/ +#define PANTHOR_MAX_VMS_PER_FILE 32 + +/**
Quoted 8 lines
Quoted 6 lines
+ *
panthor_vm_pool_create_vm()- Create a VM
+ * @pool: The VM to create this VM on.
+ * @kernel_va_start: Start of the region reserved for kernel objects.
+ * @kernel_va_range: Size of the region reserved for kernel objects.
+ *
+ * Return: 0 on success, a negative error code otherwise.Actually returns the (positive) id on success.
Will fix.
Quoted 31 lines
Quoted 31 lines
+ */
drivers/gpu/drm/panthor/panthor_mmu.c new file +29
-0+int panthor_vm_pool_create_vm (struct panthor_device *ptdev, +struct panthor_vm_pool *pool, +struct drm_panthor_vm_create *args) +{ +u64 kernel_va_start, kernel_va_range; +struct panthor_vm *vm; +int ret; +u32 id; + + ret =panthor_vm_create_check_args (ptdev, args, &kernel_va_start, &kernel_va_range); +if (ret) +return ret; + + vm =panthor_vm_create (ptdev, false, kernel_va_start, kernel_va_range, + kernel_va_start, kernel_va_range); +if (IS_ERR (vm)) +return PTR_ERR (vm); + + ret =xa_alloc (&pool->xa, &id, vm, +XA_LIMIT (1 ,PANTHOR_MAX_VMS_PER_FILE ),GFP_KERNEL ); + +if (ret) { +panthor_vm_put (vm); +return ret; + } + + args->user_va_range = kernel_va_start; +return id; +}
[...]
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c line 1750
Quoted 7 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +7
-0+/** +* panthor_vm_put() - Release a reference on a VM +* @vm: VM to release the reference on. Can be NULL. +*/ +void panthor_vm_put (struct panthor_vm *vm) +{ +static_assert (offsetof(struct panthor_vm , base) ==0 );Yuk! ;)
Quoted 8 lines
I'd prefer:
drm_gpuvm_put(vm ? &vm->base : NULL);which my compiler turns into the same thing rather than relying on the type punning. You can keep the
static_assertif you like, but I don't like relying on it for correct code generation. Although I'll admit I couldn't actually get the compiler to produce incorrect code when I tried.
Sure, I'll pick your suggestion here.
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +2
-0+drm_gpuvm_put (&vm->base); +}
[...]
Quoted 23 lines
Quoted 23 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +23
-0+ +/** +* panthor_vm_prepare_mapped_bos_resvs() - Prepare resvs on VM BOs. +* @exec: Locking/preparation context. +* @vm: VM targeted by the GPU job. +* @slot_count: Number of slots to reserve. +* +* GPU jobs assume all BOs bound to the VM at the time the job is submitted +* are available when the job is executed. In order to guarantee that, we +* need to reserve a slot on all BOs mapped to a VM and update this slot with +* the job fence after its submission. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_prepare_mapped_bos_resvs (struct drm_exec *exec,struct panthor_vm *vm, +u32 slot_count) +{ +int ret; + +/* Acquire the VM lock an reserve a slot for this GPU job. */ + ret =drm_gpuvm_prepare_vm (&vm->base, exec, slot_count); +if (ret) +return ret;
Quoted 8 lines
drivers/gpu/drm/panthor/panthor_mmu.c line 2509
Quoted 5 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +5
-0+ +/* VM operations are not protected by the VM resv-lock. We need to +* take the op_lock to make sure the shared_bos list is not updated +* while we're walking it. +*/ Is the above comment stale? AFAIK the
shared_boslist doesn't exist anymore and this doesn't appear to relate to anything here.
Oops, indeed. That predates to transition to drm_gpuvm for the VM <-> BO association.
Thanks for the review!
[1]https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_mm.c#L441
Boris Brezillon
maintainerRe: [PATCH v3 07/14] drm/panthor: Add the MMU/VM logical block
In reply to Boris BrezillonQuoted 28 lines · On Mon, 4 Dec 2023 18:33:00 +0100 Boris Brezillon…
drivers/gpu/drm/panthor/panthor_mmu.c new file +28
-0+static int panthor_vm_prepare_unmap_op_ctx (struct panthor_vm_op_ctx *op_ctx, +struct panthor_vm *vm, +u64 va,u64 size) +{ +u32 pt_count =0 ; +int ret; + +memset (op_ctx,0 ,sizeof (*op_ctx)); +INIT_LIST_HEAD (&op_ctx->returned_vmas); + op_ctx->va.range = size; + op_ctx->va.addr = va; + op_ctx->flags =DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP ; + +/* Pre-allocate L3 page tables to account for the split-2M-block +* situation on unmap. +*/ +if (va !=ALIGN (va,SZ_2M )) + pt_count++; + +if (va + size !=ALIGN (va + size,SZ_2M ) && +ALIGN (va + size,SZ_2M ) !=ALIGN (va,SZ_2M )) + pt_count++; + +if (pt_count) { + op_ctx->rsvd_page_tables.pages =kcalloc (pt_count, +sizeof (*op_ctx->rsvd_page_tables.pages), +GFP_KERNEL ); +if (!op_ctx->rsvd_page_tables.pages)
drivers/gpu/drm/panthor/panthor_mmu.c line 1226
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+ goto err_cleanup; + + ret =kmem_cache_alloc_bulk (pt_cache,GFP_KERNEL , pt_count, + op_ctx->rsvd_page_tables.pages); +if (ret != pt_count) { + ret = -ENOMEM ; + goto err_cleanup; + } + op_ctx->rsvd_page_tables.count = pt_count; + }
We also need to allocate {prev,next}_vma objects, because partial unmaps will call sm_step_remap(), which leads to a NULL deref if we don't pre-allocate these objects. I'm reworking the code to have a 3-entry array of preallocated VMAs instead of having ->map.{prev,new,next}_vma fields. We can then populate this array when preparing an op_ctx, and pick from this preallocated pool in sm_step_[re]map().
Quoted 7 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +7
-0+ +return 0 ; + +err_cleanup: +panthor_vm_cleanup_op_ctx (op_ctx, vm); +return ret; +}
Boris Brezillon
maintainerRe: [PATCH v3 07/14] drm/panthor: Add the MMU/VM logical block
In reply to Boris BrezillonQuoted 10 lines · On Mon, 4 Dec 2023 18:33:00 +0100 Boris Brezillon…
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+/** +* panthor_vm_idle() - Flag a VM idle +* @VM: VM to flag as idle. +* +* When we know the GPU is done with the VM (no more jobs to process), +* we can relinquish the AS slot attached to this VM, if any. +* +* We don't release the slot immediately, but instead place the VM in +* the LRU list, so it can be evicted if another VM needs an AS slot. +* This way, VMs keep attached to the AS they were given until we run
drivers/gpu/drm/panthor/panthor_mmu.c line 777
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+* out of free slot, limiting the number of MMU operations (TLB flush +* and other AS updates). +*/ +void panthor_vm_idle (struct panthor_vm *vm) +{ +struct panthor_device *ptdev = vm->ptdev; + +mutex_lock (&ptdev->mmu->as.slots_lock); +if (vm->as.id >=0 &&list_empty (&vm->as.lru_node)) +list_add_tail (&vm->as.lru_node, &ptdev->mmu->as.lru_list);
This doesn't work as soon as we have more than one scheduling group per VM. If one of them is being scheduled out while the other stays on a FW slot, the VM becomes idle and its AS might be re-assigned to a different group. We need to refcnt the active users here, and only return the AS when this active_cnt reaches 0.
Quoted 2 lines
(quoted diff) +2
-0+ mutex_unlock(&ptdev->mmu->as.slots_lock); +}
Boris Brezillon
maintainerRe: [PATCH v3 07/14] drm/panthor: Add the MMU/VM logical block
In reply to Boris BrezillonQuoted 18 lines · On Mon, 4 Dec 2023 18:33:00 +0100 Boris Brezillon…
drivers/gpu/drm/panthor/panthor_mmu.c new file +18
-0+/** +* panthor_vm_active() - Flag a VM as active +* @VM: VM to flag as active. +* +* Assigns an address space to a VM so it can be used by the GPU/MCU. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_vm_active (struct panthor_vm *vm) +{ +struct panthor_device *ptdev = vm->ptdev; +struct io_pgtable_cfg *cfg = &io_pgtable_ops_to_pgtable (vm->pgtbl_ops)->cfg; +int ret =0 , as, cookie; +u64 transtab, transcfg; + +if (!drm_dev_enter (&ptdev->base, &cookie)) +return -ENODEV ; +
drivers/gpu/drm/panthor/panthor_mmu.c line 695
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+mutex_lock (&ptdev->mmu->as.slots_lock); + + as = vm->as.id; +if (as >=0 ) { +/* Unhandled pagefault on this AS, the MMU was disabled. We need to +* re-enable the MMU after clearing+unmasking the AS interrupts. +*/ +if (ptdev->mmu->as.faulty_mask &panthor_mmu_as_fault_mask (ptdev, as)) + goto out_enable_as; +
When the context was idle and we make it active, we never remove the VM from the LRU list, which might cause this AS to be re-assigned to someone else while the GPU is still using the AS. I'll fix that along the active_cnt issue I mentioned in my previous reply.
Quoted 61 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +61
-0+ goto out_unlock; + } + +/* Check for a free AS */ +if (vm->for_mcu) { +drm_WARN_ON (&ptdev->base, ptdev->mmu->as.alloc_mask &BIT (0 )); + as =0 ; + }else { + as =ffz (ptdev->mmu->as.alloc_mask |BIT (0 )); + } + +if (!(BIT (as) & ptdev->gpu_info.as_present)) { +struct panthor_vm *lru_vm; + + lru_vm =list_first_entry_or_null (&ptdev->mmu->as.lru_list, + struct panthor_vm, + as.lru_node); +if (drm_WARN_ON (&ptdev->base, !lru_vm)) { + ret = -EBUSY ; + goto out_unlock; + } + +list_del_init (&lru_vm->as.lru_node); + as = lru_vm->as.id; + + lru_vm->as.id =-1 ; + }else { +set_bit (as, &ptdev->mmu->as.alloc_mask); + } + +/* Assign the free or reclaimed AS to the FD */ + vm->as.id = as; + ptdev->mmu->as.slots[as].vm = vm; + +out_enable_as: + transtab = cfg->arm_lpae_s1_cfg.ttbr; + transcfg =AS_TRANSCFG_PTW_MEMATTR_WB | +AS_TRANSCFG_PTW_RA | +AS_TRANSCFG_ADRMODE_AARCH64_4K ; +if (ptdev->coherent) + transcfg |=AS_TRANSCFG_PTW_SH_OS ; + +/* If the VM is re-activated, we clear the fault. */ + vm->unhandled_fault = false; + +/* Unhandled pagefault on this AS, clear the fault and re-enable interrupts +* before enabling the AS. +*/ +if (ptdev->mmu->as.faulty_mask &panthor_mmu_as_fault_mask (ptdev, as)) { +gpu_write (ptdev,MMU_INT_CLEAR ,panthor_mmu_as_fault_mask (ptdev, as)); + ptdev->mmu->as.faulty_mask &= ~panthor_mmu_as_fault_mask (ptdev, as); +gpu_write (ptdev,MMU_INT_MASK , ~ptdev->mmu->as.faulty_mask); + } + + ret =panthor_mmu_as_enable (vm->ptdev, vm->as.id, transtab, transcfg, vm->memattr); + +out_unlock: +mutex_unlock (&ptdev->mmu->as.slots_lock); +drm_dev_exit (cookie); +return ret; +}
Boris Brezillon
maintainerRe: [PATCH v3 07/14] drm/panthor: Add the MMU/VM logical block
In reply to Boris BrezillonQuoted 19 lines · On Mon, 4 Dec 2023 18:33:00 +0100 Boris Brezillon…
drivers/gpu/drm/panthor/panthor_mmu.c new file +19
-0+static int panthor_gpuva_sm_step_remap (struct drm_gpuva_op *op, +void *priv) +{ + struct panthor_vma *unmap_vma = container_of(op->remap.unmap->va, struct panthor_vma, base); +struct panthor_vm *vm = priv; +struct panthor_vm_op_ctx *op_ctx = vm->op_ctx; +struct drm_gpuva *prev_va =NULL , *next_va =NULL ; +u64 unmap_start, unmap_range; +int ret; + +drm_gpuva_op_remap_to_unmap_range (&op->remap, &unmap_start, &unmap_range); + ret =panthor_vm_unmap_pages (vm, unmap_start, unmap_range); +if (ret) +return ret; + +if (op->remap.prev) { +panthor_vma_init (op_ctx->map.prev_vma, unmap_vma->flags); + prev_va = &op_ctx->map.prev_vma->base; + }
drivers/gpu/drm/panthor/panthor_mmu.c line 1932
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +10
-0+ +if (op->remap.next) { +panthor_vma_init (op_ctx->map.next_vma, unmap_vma->flags); + next_va = &op_ctx->map.next_vma->base; + } + +drm_gpuva_remap (prev_va, next_va, &op->remap); + +if (prev_va) { +panthor_vma_link (vm, op_ctx->map.prev_vma, op->remap.unmap->va->vm_bo);
panthor_vma_link() transfers the vm_bo ownership to the vma object, we need to take a reference if we want to keep vm_bo's refcount balanced.
Quoted 11 lines
drivers/gpu/drm/panthor/panthor_mmu.c new file +11
-0+ op_ctx->map.prev_vma =NULL ; + } + +if (next_va) { +panthor_vma_link (vm, op_ctx->map.next_vma, op->remap.unmap->va->vm_bo); + op_ctx->map.next_vma =NULL ; + } + +panthor_vma_unlink (vm, unmap_vma); +return 0 ; +}
Boris Brezillon
maintainerPATCH v3 8/14 drm/panthor: Add the FW logical block
In reply to Boris BrezillonContains everything that's FW related, that includes the code dealing with the microcontroller unit (MCU) that's running the FW, and anything related to allocating memory shared between the FW and the CPU.
A few global FW events are processed in the IRQ handler, the rest is forwarded to the scheduler, since scheduling is the primary reason for the FW existence, and also the main source of FW <-> kernel interactions.
v3:
- Make the FW path more future-proof (Liviu)
- Use one waitqueue for all FW events
- Simplify propagation of FW events to the scheduler logic
- Drop the panthor_fw_mem abstraction and use panthor_kernel_bo instead
- Account for the panthor_vm changes
- Replace magic number with 0x7fffffff with ~0 to better signify that it's the maximum permitted value.
- More accurate rounding when computing the firmware timeout.
- Add a 'sub iterator' helper function. This also adds a check that a firmware entry doesn't overflow the firmware image.
- Drop __packed from FW structures, natural alignment is good enough.
- Other minor code improvements.
--- drivers/gpu/drm/panthor/panthor_fw.c | 1332 ++++++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_fw.h | 504 ++++++++++ 2 files changed, 1836 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_fw.c create mode 100644 drivers/gpu/drm/panthor/panthor_fw.h
drivers/gpu/drm/panthor/panthor_fw.c new file +1332 -0
@@ -0,0 +1,1332 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT +/* Copyright 2023 Collabora ltd. */ + +#ifdef CONFIG_ARM_ARCH_TIMER +#include <asm/arch_timer.h> +#endif + +#include <linux/clk.h> +#include <linux/dma-mapping.h> +#include <linux/firmware.h> +#include <linux/iopoll.h> +#include <linux/iosys-map.h> +#include <linux/mutex.h> +#include <linux/platform_device.h> + +#include <drm/drm_drv.h> +#include <drm/drm_managed.h> + +#include "panthor_device.h" +#include "panthor_gem.h" +#include "panthor_gpu.h" +#include "panthor_regs.h" +#include "panthor_fw.h" +#include "panthor_mmu.h" +#include "panthor_sched.h" + +#define CSF_FW_NAME "mali_csffw.bin" + +#define PING_INTERVAL_MS 12000 +#define PROGRESS_TIMEOUT_CYCLES (5ull * 500 * 1024 * 1024) +#define PROGRESS_TIMEOUT_SCALE_SHIFT 10 +#define IDLE_HYSTERESIS_US 800 +#define PWROFF_HYSTERESIS_US 10000 + +/** +* struct panthor_fw_binary_hdr - Firmware binary header. +*/ +struct panthor_fw_binary_hdr { +/** @magic: Magic value to check binary validity. */ +u32 magic; +#define CSF_FW_BINARY_HEADER_MAGIC 0xc3f13a6e + +/** @minor: Minor FW version. */ +u8 minor; + +/** @major: Major FW version. */ +u8 major; +#define CSF_FW_BINARY_HEADER_MAJOR_MAX 0 + +/** @padding1: MBZ. */ +u16 padding1; + +/** @version_hash: FW version hash. */ +u32 version_hash; + +/** @padding2: MBZ. */ +u32 padding2; + +/** @size: FW binary size. */ +u32 size; +}; + +/** +* enum panthor_fw_binary_entry_type - Firmware binary entry type +*/ +enum panthor_fw_binary_entry_type { +/** @CSF_FW_BINARY_ENTRY_TYPE_IFACE: Host <-> FW interface. */ +CSF_FW_BINARY_ENTRY_TYPE_IFACE =0 , + +/** @CSF_FW_BINARY_ENTRY_TYPE_CONFIG: FW config. */ +CSF_FW_BINARY_ENTRY_TYPE_CONFIG =1 , + +/** @CSF_FW_BINARY_ENTRY_TYPE_FUTF_TEST: Unit-tests. */ +CSF_FW_BINARY_ENTRY_TYPE_FUTF_TEST =2 , + +/** @CSF_FW_BINARY_ENTRY_TYPE_TRACE_BUFFER: Trace buffer interface. */ +CSF_FW_BINARY_ENTRY_TYPE_TRACE_BUFFER =3 , + +/** @CSF_FW_BINARY_ENTRY_TYPE_TIMELINE_METADATA: Timeline metadata interface. */ +CSF_FW_BINARY_ENTRY_TYPE_TIMELINE_METADATA =4 , +}; + +#define CSF_FW_BINARY_ENTRY_TYPE (ehdr) ((ehdr) & 0xff) +#define CSF_FW_BINARY_ENTRY_SIZE (ehdr) (((ehdr) >> 8) & 0xff) +#define CSF_FW_BINARY_ENTRY_UPDATE BIT(30) +#define CSF_FW_BINARY_ENTRY_OPTIONAL BIT(31) + +#define CSF_FW_BINARY_IFACE_ENTRY_RD_RD BIT(0) +#define CSF_FW_BINARY_IFACE_ENTRY_RD_WR BIT(1) +#define CSF_FW_BINARY_IFACE_ENTRY_RD_EX BIT(2) +#define CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_NONE (0 << 3) +#define CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_CACHED (1 << 3) +#define CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_UNCACHED_COHERENT (2 << 3) +#define CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_CACHED_COHERENT (3 << 3) +#define CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_MASK GENMASK(4, 3) +#define CSF_FW_BINARY_IFACE_ENTRY_RD_PROT BIT(5) +#define CSF_FW_BINARY_IFACE_ENTRY_RD_SHARED BIT(30) +#define CSF_FW_BINARY_IFACE_ENTRY_RD_ZERO BIT(31) + +#define CSF_FW_BINARY_IFACE_ENTRY_RD_SUPPORTED_FLAGS \ + (CSF_FW_BINARY_IFACE_ENTRY_RD_RD | \ + CSF_FW_BINARY_IFACE_ENTRY_RD_WR | \ + CSF_FW_BINARY_IFACE_ENTRY_RD_EX | \ + CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_MASK | \ + CSF_FW_BINARY_IFACE_ENTRY_RD_PROT | \ + CSF_FW_BINARY_IFACE_ENTRY_RD_SHARED | \ + CSF_FW_BINARY_IFACE_ENTRY_RD_ZERO) + +/** +* struct panthor_fw_binary_section_entry_hdr - Describes a section of FW binary +*/ +struct panthor_fw_binary_section_entry_hdr { +/** @flags: Section flags. */ +u32 flags; + +/** @va: MCU virtual range to map this binary section to. */ +struct { +/** @start: Start address. */ +u32 start; + +/** @end: End address. */ +u32 end; + } va; + +/** @data: Data to initialize the FW section with. */ +struct { +/** @start: Start offset in the FW binary. */ +u32 start; + +/** @end: End offset in the FW binary. */ +u32 end; + } data; +}; + +/** +* struct panthor_fw_binary_iter - Firmware binary iterator +* +* Used to parse a firmware binary. +*/ +struct panthor_fw_binary_iter { +/** @data: FW binary data. */ +const void *data; + +/** @size: FW binary size. */ +size_t size; + +/** @offset: Iterator offset. */ +size_t offset; +}; + +/** +* struct panthor_fw_section - FW section +*/ +struct panthor_fw_section { +/** @node: Used to keep track of FW sections. */ +struct list_head node; + +/** @flags: Section flags, as encoded in the FW binary. */ +u32 flags; + +/** @mem: Section memory. */ +struct panthor_kernel_bo *mem; + +/** +* @name: Name of the section, as specified in the binary. +* +* Can be NULL. +*/ +const char *name; + +/** +* @data: Initial data copied to the FW memory. +* +* We keep data around so we can reload sections after a reset. +*/ +struct { +/** @buf: Buffed used to store init data. */ +const void *buf; + +/** @size: Size of @buf in bytes. */ +size_t size; + } data; +}; + +#define CSF_MCU_SHARED_REGION_START 0x04000000ULL +#define CSF_MCU_SHARED_REGION_SIZE 0x04000000ULL + +#define MIN_CS_PER_CSG 8 +#define MIN_CSGS 3 +#define MAX_CSG_PRIO 0xf + +#define CSF_IFACE_VERSION (major, minor, patch) \ + (((major) << 24) | ((minor) << 16) | (patch)) +#define CSF_IFACE_VERSION_MAJOR (v) ((v) >> 24) +#define CSF_IFACE_VERSION_MINOR (v) (((v) >> 16) & 0xff) +#define CSF_IFACE_VERSION_PATCH (v) ((v) & 0xffff) + +#define CSF_GROUP_CONTROL_OFFSET 0x1000 +#define CSF_STREAM_CONTROL_OFFSET 0x40 +#define CSF_UNPRESERVED_REG_COUNT 4 + +/** +* struct panthor_fw_iface - FW interfaces +*/ +struct panthor_fw_iface { +/** @global: Global interface. */ +struct panthor_fw_global_iface global; + +/** @groups: Group slot interfaces. */ +struct panthor_fw_csg_iface groups[MAX_CSGS ]; + +/** @streams: Command stream slot interfaces. */ +struct panthor_fw_cs_iface streams[MAX_CSGS ][MAX_CS_PER_CSG ]; +}; + +/** +* struct panthor_fw - Firmware management +*/ +struct panthor_fw { +/** @vm: MCU VM. */ +struct panthor_vm *vm; + +/** @sections: List of FW sections. */ +struct list_head sections; + +/** @shared_section: The section containing the FW interfaces. */ +struct panthor_fw_section *shared_section; + +/** @iface: FW interfaces. */ +struct panthor_fw_iface iface; + +/** @watchdog: Collection of fields relating to the FW watchdog. */ +struct { +/** @ping_work: Delayed work used to ping the FW. */ +struct delayed_work ping_work; + } watchdog; + +/** +* @req_waitqueue: FW request waitqueue. +* +* Everytime a request is sent to a command stream group or the global +* interface, the caller will first busy wait for the request to be +* acknowledged, and then fallback to a sleeping wait. +* +* This wait queue is here to support the sleeping wait flavor. +*/ +wait_queue_head_t req_waitqueue; + +/** @booted: True is the FW is booted */ +bool booted; + +/** +* @fast_reset: True if the post_reset logic can proceed with a fast reset. +* +* A fast reset is just a reset where the driver doesn't reload the FW sections. +* +* Any time the firmware is properly suspended, a fast reset can take place. +* On the other hand, if the halt operation failed, the driver will reload +* all sections to make sure we start from a fresh state. +*/ +bool fast_reset; + +/** @irq: Job irq data. */ +struct panthor_irq irq; +}; + +struct panthor_vm *panthor_fw_vm (struct panthor_device *ptdev) +{ +return ptdev->fw->vm; +} + +/** +* panthor_fw_get_glb_iface() - Get the global interface +* @ptdev: Device. +* +* Return: The global interface. +*/ +struct panthor_fw_global_iface * +panthor_fw_get_glb_iface (struct panthor_device *ptdev) +{ +return &ptdev->fw->iface.global; +} + +/** +* panthor_fw_get_csg_iface() - Get a command stream group slot interface +* @ptdev: Device. +* @csg_slot: Index of the command stream group slot. +* +* Return: The command stream group slot interface. +*/ +struct panthor_fw_csg_iface * +panthor_fw_get_csg_iface (struct panthor_device *ptdev,u32 csg_slot) +{ +if (drm_WARN_ON (&ptdev->base, csg_slot >=MAX_CSGS )) +return NULL ; + +return &ptdev->fw->iface.groups[csg_slot]; +} + +/** +* panthor_fw_get_cs_iface() - Get a command stream slot interface +* @ptdev: Device. +* @csg_slot: Index of the command stream group slot. +* @cs_slot: Index of the command stream slot. +* +* Return: The command stream slot interface. +*/ +struct panthor_fw_cs_iface * +panthor_fw_get_cs_iface (struct panthor_device *ptdev,u32 csg_slot,u32 cs_slot) +{ +if (drm_WARN_ON (&ptdev->base, csg_slot >=MAX_CSGS || cs_slot >MAX_CS_PER_CSG )) +return NULL ; + +return &ptdev->fw->iface.streams[csg_slot][cs_slot]; +} + +/** +* panthor_fw_conv_timeout() - Convert a timeout into a cycle-count +* @ptdev: Device. +* @timeout_us: Timeout expressed in micro-seconds. +* +* The FW has two timer sources: the GPU counter or arch-timer. We need +* to express timeouts in term of number of cycles and specify which +* timer source should be used. +* +* Return: A value suitable for timeout fields in the global interface. +*/ +static u32 panthor_fw_conv_timeout (struct panthor_device *ptdev,u32 timeout_us) +{ +bool use_cycle_counter = false; +u32 timer_rate =0 ; +u64 mod_cycles; + +#ifdef CONFIG_ARM_ARCH_TIMER + timer_rate =arch_timer_get_cntfrq (); +#endif + +if (!timer_rate) { + use_cycle_counter = true; + timer_rate =clk_get_rate (ptdev->clks.core); + } + +if (drm_WARN_ON (&ptdev->base, !timer_rate)) { +/* We couldn't get a valid clock rate, let's just pick the +* maximum value so the FW still handles the core +* power on/off requests. +*/ +return GLB_TIMER_VAL (~0 ) | +GLB_TIMER_SOURCE_GPU_COUNTER ; + } + + mod_cycles =DIV_ROUND_UP_ULL ((u64 )timeout_us * timer_rate, +1000000ull <<10 ); +if (drm_WARN_ON (&ptdev->base, mod_cycles >GLB_TIMER_VAL (~0 ))) + mod_cycles =GLB_TIMER_VAL (~0 ); + +return GLB_TIMER_VAL (mod_cycles) | + (use_cycle_counter ?GLB_TIMER_SOURCE_GPU_COUNTER :0 ); +} + +static int panthor_fw_binary_iter_read (struct panthor_device *ptdev, +struct panthor_fw_binary_iter *iter, +void *out,size_t size) +{ +size_t new_offset = iter->offset + size; + +if (new_offset > iter->size || new_offset < iter->offset) { +drm_err (&ptdev->base,"Firmware too small\n" ); +return -EINVAL ; + } + +memcpy (out, iter->data + iter->offset, size); + iter->offset = new_offset; +return 0 ; +} + +static int panthor_fw_binary_sub_iter_init (struct panthor_device *ptdev, +struct panthor_fw_binary_iter *iter, +struct panthor_fw_binary_iter *sub_iter, +size_t size) +{ +size_t new_offset = iter->offset + size; + +if (new_offset > iter->size || new_offset < iter->offset) { +drm_err (&ptdev->base,"Firmware entry too long\n" ); +return -EINVAL ; + } + + sub_iter->offset =0 ; + sub_iter->data = iter->data + iter->offset; + sub_iter->size = size; + iter->offset = new_offset; +return 0 ; +} + +static void panthor_fw_init_section_mem (struct panthor_device *ptdev, +struct panthor_fw_section *section) +{ +bool was_mapped = !!section->mem->kmap; +int ret; + +if (!section->data.size && + !(section->flags &CSF_FW_BINARY_IFACE_ENTRY_RD_ZERO )) +return ; + + ret =panthor_kernel_bo_vmap (section->mem); +if (drm_WARN_ON (&ptdev->base, ret)) +return ; + +memcpy (section->mem->kmap, section->data.buf, section->data.size); +if (section->flags &CSF_FW_BINARY_IFACE_ENTRY_RD_ZERO ) { +memset (section->mem->kmap + section->data.size,0 , +panthor_kernel_bo_size (section->mem) - section->data.size); + } + +if (!was_mapped) +panthor_kernel_bo_vunmap (section->mem); +} + +/** +* panthor_fw_alloc_queue_iface_mem() - Allocate a ring-buffer interfaces. +* @ptdev: Device. +* @input: Pointer holding the input interface on success. +* Should be ignored on failure. +* @output: Pointer holding the output interface on success. +* Should be ignored on failure. +* @input_fw_va: Pointer holding the input interface FW VA on success. +* Should be ignored on failure. +* @output_fw_va: Pointer holding the output interface FW VA on success. +* Should be ignored on failure. +* +* Allocates panthor_fw_ringbuf_{input,out}_iface interfaces. The input +* interface is at offset 0, and the output interface at offset 4096. +* +* Return: A valid pointer in case of success, an ERR_PTR() otherwise. +*/ +struct panthor_kernel_bo * +panthor_fw_alloc_queue_iface_mem (struct panthor_device *ptdev, +struct panthor_fw_ringbuf_input_iface **input, +const struct panthor_fw_ringbuf_output_iface **output, +u32 *input_fw_va,u32 *output_fw_va) +{ +struct panthor_kernel_bo *mem; +int ret; + + mem =panthor_kernel_bo_create (ptdev, ptdev->fw->vm,SZ_8K , +DRM_PANTHOR_BO_NO_MMAP , +DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC | +DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED , +PANTHOR_VM_KERNEL_AUTO_VA ); +if (IS_ERR (mem)) +return mem; + + ret =panthor_kernel_bo_vmap (mem); +if (ret) { +panthor_kernel_bo_destroy (panthor_fw_vm (ptdev), mem); +return ERR_PTR (ret); + } + +memset (mem->kmap,0 ,panthor_kernel_bo_size (mem)); + *input = mem->kmap; + *output = mem->kmap +SZ_4K ; + *input_fw_va =panthor_kernel_bo_gpuva (mem); + *output_fw_va = *input_fw_va +SZ_4K ; + +return mem; +} + +/** +* panthor_fw_alloc_suspend_buf_mem() - Allocate a suspend buffer for a command stream group. +* @ptdev: Device. +* @size: Size of the suspend buffer. +* +* Return: A valid pointer in case of success, an ERR_PTR() otherwise. +*/ +struct panthor_kernel_bo * +panthor_fw_alloc_suspend_buf_mem (struct panthor_device *ptdev,size_t size) +{ +return panthor_kernel_bo_create (ptdev,panthor_fw_vm (ptdev), size, +DRM_PANTHOR_BO_NO_MMAP , +DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC , +PANTHOR_VM_KERNEL_AUTO_VA ); +} + +static int panthor_fw_load_section_entry (struct panthor_device *ptdev, +const struct firmware *fw, +struct panthor_fw_binary_iter *iter, +u32 ehdr) +{ +struct panthor_fw_binary_section_entry_hdr hdr; +struct panthor_fw_section *section; +u32 section_size; +u32 name_len; +int ret; + + ret =panthor_fw_binary_iter_read (ptdev, iter, &hdr,sizeof (hdr)); +if (ret) +return ret; + +if (hdr.data.end < hdr.data.start) { +drm_err (&ptdev->base,"Firmware corrupted, data.end < data.start (0x%x < 0x%x)\n" , + hdr.data.end, hdr.data.start); +return -EINVAL ; + } + +if (hdr.va.end < hdr.va.start) { +drm_err (&ptdev->base,"Firmware corrupted, hdr.va.end < hdr.va.start (0x%x < 0x%x)\n" , + hdr.va.end, hdr.va.start); +return -EINVAL ; + } + +if (hdr.data.end > fw->size) { +drm_err (&ptdev->base,"Firmware corrupted, file truncated? data_end=0x%x > fw size=0x%zx\n" , + hdr.data.end, fw->size); +return -EINVAL ; + } + +if ((hdr.va.start & ~PAGE_MASK ) !=0 || + (hdr.va.end & ~PAGE_MASK ) !=0 ) { +drm_err (&ptdev->base,"Firmware corrupted, virtual addresses not page aligned: 0x%x-0x%x\n" , + hdr.va.start, hdr.va.end); +return -EINVAL ; + } + +if (hdr.flags & ~CSF_FW_BINARY_IFACE_ENTRY_RD_SUPPORTED_FLAGS ) { +drm_err (&ptdev->base,"Firmware contains interface with unsupported flags (0x%x)\n" , + hdr.flags); +return -EINVAL ; + } + +if (hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_PROT ) { +drm_warn (&ptdev->base, +"Firmware protected mode entry not be supported, ignoring" ); +return 0 ; + } + +if (hdr.va.start ==CSF_MCU_SHARED_REGION_START && + !(hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_SHARED )) { +drm_err (&ptdev->base, +"Interface at 0x%llx must be shared" ,CSF_MCU_SHARED_REGION_START ); +return -EINVAL ; + } + + name_len = iter->size - iter->offset; + + section =drmm_kzalloc (&ptdev->base,sizeof (*section),GFP_KERNEL ); +if (!section) +return -ENOMEM ; + +list_add_tail (§ion->node, &ptdev->fw->sections); + section->flags = hdr.flags; + section->data.size = hdr.data.end - hdr.data.start; + +if (section->data.size >0 ) { +void *data =drmm_kmalloc (&ptdev->base, section->data.size,GFP_KERNEL ); + +if (!data) +return -ENOMEM ; + +memcpy (data, fw->data + hdr.data.start, section->data.size); + section->data.buf = data; + } + +if (name_len >0 ) { +char *name =drmm_kmalloc (&ptdev->base, name_len +1 ,GFP_KERNEL ); + +if (!name) +return -ENOMEM ; + +memcpy (name, iter->data + iter->offset, name_len); + name[name_len] ='\0' ; + section->name = name; + } + + section_size = hdr.va.end - hdr.va.start; +if (section_size) { +u32 cache_mode = hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_MASK ; +struct panthor_gem_object *bo; +u32 vm_map_flags =0 ; +struct sg_table *sgt; +u64 va = hdr.va.start; + +if (!(hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_WR )) + vm_map_flags |=DRM_PANTHOR_VM_BIND_OP_MAP_READONLY ; + +if (!(hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_EX )) + vm_map_flags |=DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC ; + +/* TODO: CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_*_COHERENT are mapped to +* non-cacheable for now. We might want to introduce a new +* IOMMU_xxx flag (or abuse IOMMU_MMIO, which maps to device +* memory and is currently not used by our driver) for +* AS_MEMATTR_AARCH64_SHARED memory, so we can take benefit +* of IO-coherent systems. +*/ +if (cache_mode !=CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_CACHED ) + vm_map_flags |=DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED ; + + section->mem =panthor_kernel_bo_create (ptdev,panthor_fw_vm (ptdev), + section_size, +DRM_PANTHOR_BO_NO_MMAP , + vm_map_flags, va); +if (IS_ERR (section->mem)) +return PTR_ERR (section->mem); + +if (drm_WARN_ON (&ptdev->base, section->mem->va_node.start != hdr.va.start)) +return -EINVAL ; + +if (section->flags &CSF_FW_BINARY_IFACE_ENTRY_RD_SHARED ) { + ret =panthor_kernel_bo_vmap (section->mem); +if (ret) +return ret; + } + +panthor_fw_init_section_mem (ptdev, section); + + bo =to_panthor_bo (section->mem->obj); + sgt =drm_gem_shmem_get_pages_sgt (&bo->base); +if (IS_ERR (sgt)) +return PTR_ERR (section->mem); + +dma_sync_sgtable_for_device (ptdev->base.dev, sgt,DMA_TO_DEVICE ); + } + +if (hdr.va.start ==CSF_MCU_SHARED_REGION_START ) + ptdev->fw->shared_section = section; + +return 0 ; +} + +static void +panthor_reload_fw_sections (struct panthor_device *ptdev,bool full_reload) +{ +struct panthor_fw_section *section; + +list_for_each_entry (section, &ptdev->fw->sections, node) { +struct sg_table *sgt; + +if (!full_reload && !(section->flags &CSF_FW_BINARY_IFACE_ENTRY_RD_WR )) +continue ; + +panthor_fw_init_section_mem (ptdev, section); + sgt =drm_gem_shmem_get_pages_sgt (&to_panthor_bo (section->mem->obj)->base); +if (!drm_WARN_ON (&ptdev->base,IS_ERR_OR_NULL (sgt))) +dma_sync_sgtable_for_device (ptdev->base.dev, sgt,DMA_TO_DEVICE ); + } +} + +static int panthor_fw_load_entry (struct panthor_device *ptdev, +const struct firmware *fw, +struct panthor_fw_binary_iter *iter) +{ +struct panthor_fw_binary_iter eiter; +u32 ehdr; +int ret; + + ret =panthor_fw_binary_iter_read (ptdev, iter, &ehdr,sizeof (ehdr)); +if (ret) +return ret; + +if ((iter->offset %sizeof (u32)) || + (CSF_FW_BINARY_ENTRY_SIZE (ehdr) %sizeof (u32))) { +drm_err (&ptdev->base,"Firmware entry isn't 32 bit aligned, offset=0x%x size=0x%x\n" , + (u32)(iter->offset -sizeof (u32)),CSF_FW_BINARY_ENTRY_SIZE (ehdr)); +return -EINVAL ; + } + +if (panthor_fw_binary_sub_iter_init (ptdev, iter, &eiter, +CSF_FW_BINARY_ENTRY_SIZE (ehdr) -sizeof (ehdr))) +return -EINVAL ; + +switch (CSF_FW_BINARY_ENTRY_TYPE (ehdr)) { +case CSF_FW_BINARY_ENTRY_TYPE_IFACE : +return panthor_fw_load_section_entry (ptdev, fw, &eiter, ehdr); + +/* FIXME: handle those entry types? */ +case CSF_FW_BINARY_ENTRY_TYPE_CONFIG : +case CSF_FW_BINARY_ENTRY_TYPE_FUTF_TEST : +case CSF_FW_BINARY_ENTRY_TYPE_TRACE_BUFFER : +case CSF_FW_BINARY_ENTRY_TYPE_TIMELINE_METADATA : +return 0 ; +default : +break ; + } + +if (ehdr &CSF_FW_BINARY_ENTRY_OPTIONAL ) +return 0 ; + +drm_err (&ptdev->base, +"Unsupported non-optional entry type %u in firmware\n" , +CSF_FW_BINARY_ENTRY_TYPE (ehdr)); +return -EINVAL ; +} + +static int panthor_fw_load (struct panthor_device *ptdev) +{ +const struct firmware *fw =NULL ; +struct panthor_fw_binary_iter iter = {}; +struct panthor_fw_binary_hdr hdr; +char fw_path[128 ]; +int ret; + +snprintf (fw_path,sizeof (fw_path),"arm/mali/arch%d.%d/%s" , + (u32 )GPU_ARCH_MAJOR (ptdev->gpu_info.gpu_id), + (u32 )GPU_ARCH_MINOR (ptdev->gpu_info.gpu_id), +CSF_FW_NAME ); + + ret =request_firmware (&fw, fw_path, ptdev->base.dev); +if (ret) { +drm_err (&ptdev->base,"Failed to load firmware image '%s'\n" , +CSF_FW_NAME ); +return ret; + } + + iter.data = fw->data; + iter.size = fw->size; + ret =panthor_fw_binary_iter_read (ptdev, &iter, &hdr,sizeof (hdr)); +if (ret) + goto out; + +if (hdr.magic !=CSF_FW_BINARY_HEADER_MAGIC ) { + ret = -EINVAL ; +drm_err (&ptdev->base,"Invalid firmware magic\n" ); + goto out; + } + +if (hdr.major !=CSF_FW_BINARY_HEADER_MAJOR_MAX ) { + ret = -EINVAL ; +drm_err (&ptdev->base,"Unsupported firmware binary header version %d.%d (expected %d.x)\n" , + hdr.major, hdr.minor,CSF_FW_BINARY_HEADER_MAJOR_MAX ); + goto out; + } + +if (hdr.size > iter.size) { +drm_err (&ptdev->base,"Firmware image is truncated\n" ); + goto out; + } + + iter.size = hdr.size; + +while (iter.offset < hdr.size) { + ret =panthor_fw_load_entry (ptdev, fw, &iter); +if (ret) + goto out; + } + +if (!ptdev->fw->shared_section) { +drm_err (&ptdev->base,"Shared interface region not found\n" ); + ret = -EINVAL ; + goto out; + } + +out: +release_firmware (fw); +return ret; +} + +/** +* iface_fw_to_cpu_addr() - Turn an MCU address into a CPU address +* @ptdev: Device. +* @mcu_va: MCU address. +* +* Return: NULL if the address is not part of the shared section, non-NULL otherwise. +*/ +static void *iface_fw_to_cpu_addr (struct panthor_device *ptdev,u32 mcu_va) +{ +u64 shared_mem_start =panthor_kernel_bo_gpuva (ptdev->fw->shared_section->mem); +u64 shared_mem_end = shared_mem_start + +panthor_kernel_bo_size (ptdev->fw->shared_section->mem); +if (mcu_va < shared_mem_start || mcu_va >= shared_mem_end) +return NULL ; + +return ptdev->fw->shared_section->mem->kmap + (mcu_va - shared_mem_start); +} + +static int panthor_init_cs_iface (struct panthor_device *ptdev, +unsigned int csg_idx,unsigned int cs_idx) +{ +struct panthor_fw_global_iface *glb_iface =panthor_fw_get_glb_iface (ptdev); +struct panthor_fw_csg_iface *csg_iface =panthor_fw_get_csg_iface (ptdev, csg_idx); +struct panthor_fw_cs_iface *cs_iface = &ptdev->fw->iface.streams[csg_idx][cs_idx]; +u64 shared_section_sz =panthor_kernel_bo_size (ptdev->fw->shared_section->mem); +u32 iface_offset =CSF_GROUP_CONTROL_OFFSET + + (csg_idx * glb_iface->control->group_stride) + +CSF_STREAM_CONTROL_OFFSET + + (cs_idx * csg_iface->control->stream_stride); +struct panthor_fw_cs_iface *first_cs_iface = +panthor_fw_get_cs_iface (ptdev,0 ,0 ); + +if (iface_offset +sizeof (*cs_iface) >= shared_section_sz) +return -EINVAL ; + +spin_lock_init (&cs_iface->lock); + cs_iface->control = ptdev->fw->shared_section->mem->kmap + iface_offset; + cs_iface->input =iface_fw_to_cpu_addr (ptdev, cs_iface->control->input_va); + cs_iface->output =iface_fw_to_cpu_addr (ptdev, cs_iface->control->output_va); + +if (!cs_iface->input || !cs_iface->output) { +drm_err (&ptdev->base,"Invalid stream control interface input/output VA" ); +return -EINVAL ; + } + +if (cs_iface != first_cs_iface) { +if (cs_iface->control->features != first_cs_iface->control->features) { +drm_err (&ptdev->base,"Expecting identical CS slots" ); +return -EINVAL ; + } + }else { +u32 reg_count =CS_FEATURES_WORK_REGS (cs_iface->control->features); + + ptdev->csif_info.cs_reg_count = reg_count; + ptdev->csif_info.unpreserved_cs_reg_count =CSF_UNPRESERVED_REG_COUNT ; + } + +return 0 ; +} + +static bool compare_csg (const struct panthor_fw_csg_control_iface *a, +const struct panthor_fw_csg_control_iface *b) +{ +if (a->features != b->features) +return false; +if (a->suspend_size != b->suspend_size) +return false; +if (a->protm_suspend_size != b->protm_suspend_size) +return false; +if (a->stream_num != b->stream_num) +return false; +return true; +} + +static int panthor_init_csg_iface (struct panthor_device *ptdev, +unsigned int csg_idx) +{ +struct panthor_fw_global_iface *glb_iface =panthor_fw_get_glb_iface (ptdev); +struct panthor_fw_csg_iface *csg_iface = &ptdev->fw->iface.groups[csg_idx]; +u64 shared_section_sz =panthor_kernel_bo_size (ptdev->fw->shared_section->mem); +u32 iface_offset =CSF_GROUP_CONTROL_OFFSET + (csg_idx * glb_iface->control->group_stride); +unsigned int i; + +if (iface_offset +sizeof (*csg_iface) >= shared_section_sz) +return -EINVAL ; + +spin_lock_init (&csg_iface->lock); + csg_iface->control = ptdev->fw->shared_section->mem->kmap + iface_offset; + csg_iface->input =iface_fw_to_cpu_addr (ptdev, csg_iface->control->input_va); + csg_iface->output =iface_fw_to_cpu_addr (ptdev, csg_iface->control->output_va); + +if (csg_iface->control->stream_num <MIN_CS_PER_CSG || + csg_iface->control->stream_num >MAX_CS_PER_CSG ) +return -EINVAL ; + +if (!csg_iface->input || !csg_iface->output) { +drm_err (&ptdev->base,"Invalid group control interface input/output VA" ); +return -EINVAL ; + } + +if (csg_idx >0 ) { +struct panthor_fw_csg_iface *first_csg_iface = +panthor_fw_get_csg_iface (ptdev,0 ); + +if (!compare_csg (first_csg_iface->control, csg_iface->control)) { +drm_err (&ptdev->base,"Expecting identical CSG slots" ); +return -EINVAL ; + } + } + +for (i =0 ; i < csg_iface->control->stream_num; i++) { +int ret =panthor_init_cs_iface (ptdev, csg_idx, i); + +if (ret) +return ret; + } + +return 0 ; +} + +static u32 panthor_get_instr_features (struct panthor_device *ptdev) +{ +struct panthor_fw_global_iface *glb_iface =panthor_fw_get_glb_iface (ptdev); + +if (glb_iface->control->version <CSF_IFACE_VERSION (1 ,1 ,0 )) +return 0 ; + +return glb_iface->control->instr_features; +} + +static int panthor_fw_init_ifaces (struct panthor_device *ptdev) +{ +struct panthor_fw_global_iface *glb_iface = &ptdev->fw->iface.global; +unsigned int i; + +if (!ptdev->fw->shared_section->mem->kmap) +return -EINVAL ; + +spin_lock_init (&glb_iface->lock); + glb_iface->control = ptdev->fw->shared_section->mem->kmap; + +if (!glb_iface->control->version) { +drm_err (&ptdev->base,"Firmware version is 0. Firmware may have failed to boot" ); +return -EINVAL ; + } + + glb_iface->input =iface_fw_to_cpu_addr (ptdev, glb_iface->control->input_va); + glb_iface->output =iface_fw_to_cpu_addr (ptdev, glb_iface->control->output_va); +if (!glb_iface->input || !glb_iface->output) { +drm_err (&ptdev->base,"Invalid global control interface input/output VA" ); +return -EINVAL ; + } + +if (glb_iface->control->group_num >MAX_CSGS || + glb_iface->control->group_num <MIN_CSGS ) { +drm_err (&ptdev->base,"Invalid number of control groups" ); +return -EINVAL ; + } + +for (i =0 ; i < glb_iface->control->group_num; i++) { +int ret =panthor_init_csg_iface (ptdev, i); + +if (ret) +return ret; + } + +drm_info (&ptdev->base,"CSF FW v%d.%d.%d, Features %#x Instrumentation features %#x" , +CSF_IFACE_VERSION_MAJOR (glb_iface->control->version), +CSF_IFACE_VERSION_MINOR (glb_iface->control->version), +CSF_IFACE_VERSION_PATCH (glb_iface->control->version), + glb_iface->control->features, +panthor_get_instr_features (ptdev)); +return 0 ; +} + +static void panthor_fw_init_global_iface (struct panthor_device *ptdev) +{ +struct panthor_fw_global_iface *glb_iface =panthor_fw_get_glb_iface (ptdev); + +/* Enable all cores. */ + glb_iface->input->core_en_mask = ptdev->gpu_info.shader_present; + +/* Setup timers. */ + glb_iface->input->poweroff_timer =panthor_fw_conv_timeout (ptdev,PWROFF_HYSTERESIS_US ); + glb_iface->input->progress_timer =PROGRESS_TIMEOUT_CYCLES >>PROGRESS_TIMEOUT_SCALE_SHIFT ; + glb_iface->input->idle_timer =panthor_fw_conv_timeout (ptdev,IDLE_HYSTERESIS_US ); + +/* Enable interrupts we care about. */ + glb_iface->input->ack_irq_mask =GLB_CFG_ALLOC_EN | +GLB_PING | +GLB_CFG_PROGRESS_TIMER | +GLB_CFG_POWEROFF_TIMER | +GLB_IDLE_EN | +GLB_IDLE ; + +panthor_fw_update_reqs (glb_iface, req,GLB_IDLE_EN ,GLB_IDLE_EN ); +panthor_fw_toggle_reqs (glb_iface, req, ack, +GLB_CFG_ALLOC_EN | +GLB_CFG_POWEROFF_TIMER | +GLB_CFG_PROGRESS_TIMER ); + +gpu_write (ptdev,CSF_DOORBELL (CSF_GLB_DOORBELL_ID ),1 ); + +/* Kick the watchdog. */ +mod_delayed_work (ptdev->reset.wq, &ptdev->fw->watchdog.ping_work, +msecs_to_jiffies (PING_INTERVAL_MS )); +} + +static void panthor_job_irq_handler (struct panthor_device *ptdev,u32 status) +{ +if (!ptdev->fw->booted && (status &JOB_INT_GLOBAL_IF )) + ptdev->fw->booted = true; + +wake_up_all (&ptdev->fw->req_waitqueue); + +/* If the FW is not booted, don't process IRQs, just flag the FW as booted. */ +if (!ptdev->fw->booted) +return ; + +panthor_sched_report_fw_events (ptdev, status); +} +PANTHOR_IRQ_HANDLER (job,JOB , panthor_job_irq_handler); + +static int panthor_fw_start (struct panthor_device *ptdev) +{ +bool timedout = false; + + ptdev->fw->booted = false; +panthor_job_irq_resume (&ptdev->fw->irq, ~0 ); +gpu_write (ptdev,MCU_CONTROL ,MCU_CONTROL_AUTO ); + +if (!wait_event_timeout (ptdev->fw->req_waitqueue, + ptdev->fw->booted, +msecs_to_jiffies (1000 ))) { +if (!ptdev->fw->booted && + !(gpu_read (ptdev,JOB_INT_STAT ) &JOB_INT_GLOBAL_IF )) + timedout = true; + } + +if (timedout) { +drm_err (&ptdev->base,"Failed to boot MCU" ); +return -ETIMEDOUT ; + } + +return 0 ; +} + +static void panthor_fw_stop (struct panthor_device *ptdev) +{ +u32 status; + +gpu_write (ptdev,MCU_CONTROL ,MCU_CONTROL_DISABLE ); +if (readl_poll_timeout (ptdev->iomem +MCU_STATUS , status, + status ==MCU_STATUS_DISABLED ,10 ,100000 )) +drm_err (&ptdev->base,"Failed to stop MCU" ); +} + +/** +* panthor_fw_pre_reset() - Call before a reset. +* @ptdev: Device. +* @on_hang: true if the reset was triggered on a GPU hang. +* +* If the reset is not triggered on a hang, we try to gracefully halt the +* MCU, so we can do a fast-reset when panthor_fw_post_reset() is called. +*/ +void panthor_fw_pre_reset (struct panthor_device *ptdev,bool on_hang) +{ +/* Make sure we won't be woken up by a ping. */ +cancel_delayed_work_sync (&ptdev->fw->watchdog.ping_work); + + ptdev->fw->fast_reset = false; + +if (!on_hang) { +struct panthor_fw_global_iface *glb_iface =panthor_fw_get_glb_iface (ptdev); +u32 status; + +panthor_fw_update_reqs (glb_iface, req,GLB_HALT ,GLB_HALT ); +gpu_write (ptdev,CSF_DOORBELL (CSF_GLB_DOORBELL_ID ),1 ); +if (!readl_poll_timeout (ptdev->iomem +MCU_STATUS , status, + status ==MCU_STATUS_HALT ,10 ,100000 ) && + glb_iface->output->halt_status ==PANTHOR_FW_HALT_OK ) { + ptdev->fw->fast_reset = true; + }else { +drm_warn (&ptdev->base,"Failed to cleanly suspend MCU" ); + } + +/* The FW detects 0 -> 1 transitions. Make sure we reset +* the HALT bit before the FW is rebooted. +*/ +panthor_fw_update_reqs (glb_iface, req,0 ,GLB_HALT ); + } + +panthor_job_irq_suspend (&ptdev->fw->irq); +} + +/** +* panthor_fw_post_reset() - Call after a reset. +* @ptdev: Device. +* +* Start the FW. If this is not a fast reset, all FW sections are reloaded to +* make sure we can recover from a memory corruption. +*/ +int panthor_fw_post_reset (struct panthor_device *ptdev) +{ +int ret; + +/* Make the MCU VM active. */ + ret =panthor_vm_active (ptdev->fw->vm); +if (ret) +return ret; + +/* Reload all sections, including RO ones. We're not supposed +* to end up here anyway, let's just assume the overhead of +* reloading everything is acceptable. +*/ +if (!ptdev->fw->fast_reset) +panthor_reload_fw_sections (ptdev, true); + + ret =panthor_fw_start (ptdev); +if (ret) +return ret; + +/* We must re-initialize the global interface even on fast-reset. */ +panthor_fw_init_global_iface (ptdev); +return 0 ; +} + +/** +* panthor_fw_unplug() - Called when the device is unplugged. +* @ptdev: Device. +* +* This function must make sure all pending operations are flushed before +* will release device resources, thus preventing any interaction with +* the HW. +* +* If there is still FW-related work running after this function returns, +* they must use drm_dev_{enter,exit}() and skip any HW access when +* drm_dev_enter() returns false. +*/ +void panthor_fw_unplug (struct panthor_device *ptdev) +{ +struct panthor_fw_section *section; + +cancel_delayed_work_sync (&ptdev->fw->watchdog.ping_work); + +/* Make sure the IRQ handler can be called after that point. */ +if (ptdev->fw->irq.irq) +panthor_job_irq_suspend (&ptdev->fw->irq); + +panthor_fw_stop (ptdev); + +if (ptdev->fw->vm) +panthor_vm_idle (ptdev->fw->vm); + +list_for_each_entry (section, &ptdev->fw->sections, node) +panthor_kernel_bo_destroy (panthor_fw_vm (ptdev), section->mem); + +panthor_vm_put (ptdev->fw->vm); + +panthor_gpu_power_off (ptdev,L2 , ptdev->gpu_info.l2_present,20000 ); +} + +/** +* panthor_fw_wait_acks() - Wait for requests to be acknowledged by the FW. +* @req_ptr: Pointer to the req register. +* @ack_ptr: Pointer to the ack register. +* @wq: Wait queue to use for the sleeping wait. +* @req_mask: Mask of requests to wait for. +* @acked: Pointer to field that's updated with the acked requests. +* If the function returns 0, *acked == req_mask. +* @timeout_ms: Timeout expressed in milliseconds. +* +* Return: 0 on success, -ETIMEDOUT otherwise. +*/ +static int panthor_fw_wait_acks (const u32 *req_ptr,const u32 *ack_ptr, +wait_queue_head_t *wq, +u32 req_mask,u32 *acked, +u32 timeout_ms) +{ +u32 ack, req =READ_ONCE (*req_ptr) & req_mask; +int ret; + +/* Busy wait for a few µsecs before falling back to a sleeping wait. */ + *acked = req_mask; + ret =read_poll_timeout_atomic (READ_ONCE , ack, + (ack & req_mask) == req, +0 ,10 ,0 , + *ack_ptr); +if (!ret) +return 0 ; + +if (wait_event_timeout (*wq, (READ_ONCE (*ack_ptr) & req_mask) == req, +msecs_to_jiffies (timeout_ms))) +return 0 ; + +/* Check one last time, in case we were not woken up for some reason. */ + ack =READ_ONCE (*ack_ptr); +if ((ack & req_mask) == req) +return 0 ; + + *acked = ~(req ^ ack) & req_mask; +return -ETIMEDOUT ; +} + +/** +* panthor_fw_glb_wait_acks() - Wait for global requests to be acknowledged. +* @ptdev: Device. +* @req_mask: Mask of requests to wait for. +* @acked: Pointer to field that's updated with the acked requests. +* If the function returns 0, *acked == req_mask. +* @timeout_ms: Timeout expressed in milliseconds. +* +* Return: 0 on success, -ETIMEDOUT otherwise. +*/ +int panthor_fw_glb_wait_acks (struct panthor_device *ptdev, +u32 req_mask,u32 *acked, +u32 timeout_ms) +{ +struct panthor_fw_global_iface *glb_iface =panthor_fw_get_glb_iface (ptdev); + +/* GLB_HALT doesn't get acked through the FW interface. */ +if (drm_WARN_ON (&ptdev->base, req_mask & (~GLB_REQ_MASK |GLB_HALT ))) +return -EINVAL ; + +return panthor_fw_wait_acks (&glb_iface->input->req, + &glb_iface->output->ack, + &ptdev->fw->req_waitqueue, + req_mask, acked, timeout_ms); +} + +/** +* panthor_fw_csg_wait_acks() - Wait for command stream group requests to be acknowledged. +* @ptdev: Device. +* @csg_slot: CSG slot ID. +* @req_mask: Mask of requests to wait for. +* @acked: Pointer to field that's updated with the acked requests. +* If the function returns 0, *acked == req_mask. +* @timeout_ms: Timeout expressed in milliseconds. +* +* Return: 0 on success, -ETIMEDOUT otherwise. +*/ +int panthor_fw_csg_wait_acks (struct panthor_device *ptdev,u32 csg_slot, +u32 req_mask,u32 *acked,u32 timeout_ms) +{ +struct panthor_fw_csg_iface *csg_iface =panthor_fw_get_csg_iface (ptdev, csg_slot); +int ret; + +if (drm_WARN_ON (&ptdev->base, req_mask & ~CSG_REQ_MASK )) +return -EINVAL ; + + ret =panthor_fw_wait_acks (&csg_iface->input->req, + &csg_iface->output->ack, + &ptdev->fw->req_waitqueue, + req_mask, acked, timeout_ms); + +/* +* Check that all bits in the state field were updated, is any mismatch +* then clear all bits in the state field. This allows code to do +* (acked & CSG_STATE_MASK) and get the right value. +*/ + +if ((*acked &CSG_STATE_MASK ) !=CSG_STATE_MASK ) + *acked &= ~CSG_STATE_MASK ; + +return ret; +} + +/** +* panthor_fw_ring_csg_doorbells() - Ring command stream group doorbells. +* @ptdev: Device. +* @csg_mask: Bitmask encoding the command stream group doorbells to ring. +* +* This function is toggling bits in the doorbell_req and ringing the +* global doorbell. It doesn't require a user doorbell to be attached to +* the group. +*/ +void panthor_fw_ring_csg_doorbells (struct panthor_device *ptdev,u32 csg_mask) +{ +struct panthor_fw_global_iface *glb_iface =panthor_fw_get_glb_iface (ptdev); + +panthor_fw_toggle_reqs (glb_iface, doorbell_req, doorbell_ack, csg_mask); +gpu_write (ptdev,CSF_DOORBELL (CSF_GLB_DOORBELL_ID ),1 ); +} + +static void panthor_fw_ping_work (struct work_struct *work) +{ + struct panthor_fw *fw = container_of(work, struct panthor_fw, watchdog.ping_work.work); +struct panthor_device *ptdev = fw->irq.ptdev; +struct panthor_fw_global_iface *glb_iface =panthor_fw_get_glb_iface (ptdev); +u32 acked; +int ret; + +if (panthor_device_reset_is_pending (ptdev)) +return ; + +panthor_fw_toggle_reqs (glb_iface, req, ack,GLB_PING ); +gpu_write (ptdev,CSF_DOORBELL (CSF_GLB_DOORBELL_ID ),1 ); + + ret =panthor_fw_glb_wait_acks (ptdev,GLB_PING , &acked,100 ); +if (ret) { +panthor_device_schedule_reset (ptdev); +drm_err (&ptdev->base,"FW ping timeout, scheduling a reset" ); + }else { +mod_delayed_work (ptdev->reset.wq, &fw->watchdog.ping_work, +msecs_to_jiffies (PING_INTERVAL_MS )); + } +} + +/** +* panthor_fw_init() - Initialize FW related data. +* @ptdev: Device. +* +* Return: 0 on success, a negative error code otherwise. +*/ +int panthor_fw_init (struct panthor_device *ptdev) +{ +struct panthor_fw *fw; +int ret, irq; + + fw =drmm_kzalloc (&ptdev->base,sizeof (*fw),GFP_KERNEL ); +if (!fw) +return -ENOMEM ; + + ptdev->fw = fw; +init_waitqueue_head (&fw->req_waitqueue); +INIT_LIST_HEAD (&fw->sections); +INIT_DELAYED_WORK (&fw->watchdog.ping_work, panthor_fw_ping_work); + + irq =platform_get_irq_byname (to_platform_device (ptdev->base.dev),"job" ); +if (irq <=0 ) +return -ENODEV ; + + ret =panthor_request_job_irq (ptdev, &fw->irq, irq,0 ); +if (ret) { +drm_err (&ptdev->base,"failed to request job irq" ); +return ret; + } + + ret =panthor_gpu_l2_power_on (ptdev); +if (ret) +return ret; + + fw->vm =panthor_vm_create (ptdev, true, +0 ,SZ_4G , +CSF_MCU_SHARED_REGION_START , +CSF_MCU_SHARED_REGION_SIZE ); +if (IS_ERR (fw->vm)) { + ret =PTR_ERR (fw->vm); + fw->vm =NULL ; + goto err_unplug_fw; + } + + ret =panthor_fw_load (ptdev); +if (ret) + goto err_unplug_fw; + + ret =panthor_vm_active (fw->vm); +if (ret) + goto err_unplug_fw; + + ret =panthor_fw_start (ptdev); +if (ret) + goto err_unplug_fw; + + ret =panthor_fw_init_ifaces (ptdev); +if (ret) + goto err_unplug_fw; + +panthor_fw_init_global_iface (ptdev); +return 0 ; + +err_unplug_fw: +panthor_fw_unplug (ptdev); +return ret; +}
drivers/gpu/drm/panthor/panthor_fw.h new file +504 -0
504 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v3 08/14] drm/panthor: Add the FW logical block
Quoted 23 lines · Boris Brezillon
Contains everything that's FW related, that includes the code dealing with the microcontroller unit (MCU) that's running the FW, and anything related to allocating memory shared between the FW and the CPU.
A few global FW events are processed in the IRQ handler, the rest is forwarded to the scheduler, since scheduling is the primary reason for the FW existence, and also the main source of FW <-> kernel interactions.
v3:
- Make the FW path more future-proof (Liviu)
- Use one waitqueue for all FW events
- Simplify propagation of FW events to the scheduler logic
- Drop thepanthor_fw_memabstraction and usepanthor_kernel_boinstead
- Account for thepanthor_vmchanges
- Replace magic number with0x7fffffffwith ~0 to better signify that it's the maximum permitted value.
- More accurate rounding when computing the firmware timeout.
- Add a 'sub iterator' helper function. This also adds a check that a firmware entry doesn't overflow the firmware image.
- Drop__packedfrom FW structures, natural alignment is good enough.
- Other minor code improvements.
Quoted 2 lines
Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>
One typo below (I think it might even have been mine...), but otherwise
Quoted 12 lines
--- drivers/gpu/drm/panthor/panthor_fw.c | 1332 ++++++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_fw.h | 504 ++++++++++ 2 files changed, 1836 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_fw.c create mode 100644 drivers/gpu/drm/panthor/panthor_fw.hdrivers/gpu/drm/panthor/panthor_fw.c new file +0
-0
<snip>
Quoted 17 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +17
-0+/** +* panthor_fw_csg_wait_acks() - Wait for command stream group requests to be acknowledged. +* @ptdev: Device. +* @csg_slot: CSG slot ID. +* @req_mask: Mask of requests to wait for. +* @acked: Pointer to field that's updated with the acked requests. +* If the function returns 0, *acked == req_mask. +* @timeout_ms: Timeout expressed in milliseconds. +* +* Return: 0 on success, -ETIMEDOUT otherwise. +*/ +int panthor_fw_csg_wait_acks (struct panthor_device *ptdev,u32 csg_slot, +u32 req_mask,u32 *acked,u32 timeout_ms) +{ +struct panthor_fw_csg_iface *csg_iface =panthor_fw_get_csg_iface (ptdev, csg_slot); +int ret; +
drivers/gpu/drm/panthor/panthor_fw.c line 1214
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +10
-0+if (drm_WARN_ON (&ptdev->base, req_mask & ~CSG_REQ_MASK )) +return -EINVAL ; + + ret =panthor_fw_wait_acks (&csg_iface->input->req, + &csg_iface->output->ack, + &ptdev->fw->req_waitqueue, + req_mask, acked, timeout_ms); + +/* +* Check that all bits in the state field were updated, is any mismatch
NIT: ..., *if* any mismatch
Steve
Re: [PATCH v3 08/14] drm/panthor: Add the FW logical block
In reply to Boris BrezillonQuoted 37 lines · Boris Brezillon
Contains everything that's FW related, that includes the code dealing with the microcontroller unit (MCU) that's running the FW, and anything related to allocating memory shared between the FW and the CPU.
A few global FW events are processed in the IRQ handler, the rest is forwarded to the scheduler, since scheduling is the primary reason for the FW existence, and also the main source of FW <-> kernel interactions.
v3:
- Make the FW path more future-proof (Liviu)
- Use one waitqueue for all FW events
- Simplify propagation of FW events to the scheduler logic
- Drop thepanthor_fw_memabstraction and usepanthor_kernel_boinstead
- Account for thepanthor_vmchanges
- Replace magic number with0x7fffffffwith ~0 to better signify that it's the maximum permitted value.
- More accurate rounding when computing the firmware timeout.
- Add a 'sub iterator' helper function. This also adds a check that a firmware entry doesn't overflow the firmware image.
- Drop__packedfrom FW structures, natural alignment is good enough.
- Other minor code improvements.Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>--- drivers/gpu/drm/panthor/panthor_fw.c | 1332 ++++++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_fw.h | 504 ++++++++++ 2 files changed, 1836 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_fw.c create mode 100644 drivers/gpu/drm/panthor/panthor_fw.hdrivers/gpu/drm/panthor/panthor_fw.c new file +0
-0
...
Quoted 110 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +110
-0+static int panthor_fw_load_section_entry (struct panthor_device *ptdev, +const struct firmware *fw, +struct panthor_fw_binary_iter *iter, +u32 ehdr) +{ +struct panthor_fw_binary_section_entry_hdr hdr; +struct panthor_fw_section *section; +u32 section_size; +u32 name_len; +int ret; + + ret =panthor_fw_binary_iter_read (ptdev, iter, &hdr,sizeof (hdr)); +if (ret) +return ret; + +if (hdr.data.end < hdr.data.start) { +drm_err (&ptdev->base,"Firmware corrupted, data.end < data.start (0x%x < 0x%x)\n" , + hdr.data.end, hdr.data.start); +return -EINVAL ; + } + +if (hdr.va.end < hdr.va.start) { +drm_err (&ptdev->base,"Firmware corrupted, hdr.va.end < hdr.va.start (0x%x < 0x%x)\n" , + hdr.va.end, hdr.va.start); +return -EINVAL ; + } + +if (hdr.data.end > fw->size) { +drm_err (&ptdev->base,"Firmware corrupted, file truncated? data_end=0x%x > fw size=0x%zx\n" , + hdr.data.end, fw->size); +return -EINVAL ; + } + +if ((hdr.va.start & ~PAGE_MASK ) !=0 || + (hdr.va.end & ~PAGE_MASK ) !=0 ) { +drm_err (&ptdev->base,"Firmware corrupted, virtual addresses not page aligned: 0x%x-0x%x\n" , + hdr.va.start, hdr.va.end); +return -EINVAL ; + } + +if (hdr.flags & ~CSF_FW_BINARY_IFACE_ENTRY_RD_SUPPORTED_FLAGS ) { +drm_err (&ptdev->base,"Firmware contains interface with unsupported flags (0x%x)\n" , + hdr.flags); +return -EINVAL ; + } + +if (hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_PROT ) { +drm_warn (&ptdev->base, +"Firmware protected mode entry not be supported, ignoring" ); +return 0 ; + } + +if (hdr.va.start ==CSF_MCU_SHARED_REGION_START && + !(hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_SHARED )) { +drm_err (&ptdev->base, +"Interface at 0x%llx must be shared" ,CSF_MCU_SHARED_REGION_START ); +return -EINVAL ; + } + + name_len = iter->size - iter->offset; + + section =drmm_kzalloc (&ptdev->base,sizeof (*section),GFP_KERNEL ); +if (!section) +return -ENOMEM ; + +list_add_tail (§ion->node, &ptdev->fw->sections); + section->flags = hdr.flags; + section->data.size = hdr.data.end - hdr.data.start; + +if (section->data.size >0 ) { +void *data =drmm_kmalloc (&ptdev->base, section->data.size,GFP_KERNEL ); + +if (!data) +return -ENOMEM ; + +memcpy (data, fw->data + hdr.data.start, section->data.size); + section->data.buf = data; + } + +if (name_len >0 ) { +char *name =drmm_kmalloc (&ptdev->base, name_len +1 ,GFP_KERNEL ); + +if (!name) +return -ENOMEM ; + +memcpy (name, iter->data + iter->offset, name_len); + name[name_len] ='\0' ; + section->name = name; + } + + section_size = hdr.va.end - hdr.va.start; +if (section_size) { +u32 cache_mode = hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_MASK ; +struct panthor_gem_object *bo; +u32 vm_map_flags =0 ; +struct sg_table *sgt; +u64 va = hdr.va.start; + +if (!(hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_WR )) + vm_map_flags |=DRM_PANTHOR_VM_BIND_OP_MAP_READONLY ; + +if (!(hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_EX )) + vm_map_flags |=DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC ; + +/* TODO: CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_*_COHERENT are mapped to +* non-cacheable for now. We might want to introduce a new +* IOMMU_xxx flag (or abuse IOMMU_MMIO, which maps to device +* memory and is currently not used by our driver) for +* AS_MEMATTR_AARCH64_SHARED memory, so we can take benefit +* of IO-coherent systems.
drivers/gpu/drm/panthor/panthor_fw.c line 604
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +10
-0+*/ +if (cache_mode !=CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_CACHED ) + vm_map_flags |=DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED ; + + section->mem =panthor_kernel_bo_create (ptdev,panthor_fw_vm (ptdev), + section_size, +DRM_PANTHOR_BO_NO_MMAP , + vm_map_flags, va); +if (IS_ERR (section->mem)) +return PTR_ERR (section->mem);
There's a small bug here - if panthor_kernel_bo_create() fails, section->mem will be left containing an error value, not NULL or a valid pointer.
This means that on rmmod, panthor_fw_unplug() will try and free the BO even though it was never allocated, causing a kernel panic.
The solution is obviously to only assign section->mem if the allocation actually succeeded.
However I also wonder if the list_add_tail() call should be moved to later in the function, after we know everything else has succeeded, or if this function needs some more error handling in general.
Quoted 26 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +26
-0+ +if (drm_WARN_ON (&ptdev->base, section->mem->va_node.start != hdr.va.start)) +return -EINVAL ; + +if (section->flags &CSF_FW_BINARY_IFACE_ENTRY_RD_SHARED ) { + ret =panthor_kernel_bo_vmap (section->mem); +if (ret) +return ret; + } + +panthor_fw_init_section_mem (ptdev, section); + + bo =to_panthor_bo (section->mem->obj); + sgt =drm_gem_shmem_get_pages_sgt (&bo->base); +if (IS_ERR (sgt)) +return PTR_ERR (section->mem); + +dma_sync_sgtable_for_device (ptdev->base.dev, sgt,DMA_TO_DEVICE ); + } + +if (hdr.va.start ==CSF_MCU_SHARED_REGION_START ) + ptdev->fw->shared_section = section; + +return 0 ; +} +
...
Quoted 19 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +19
-0+/** +* panthor_fw_unplug() - Called when the device is unplugged. +* @ptdev: Device. +* +* This function must make sure all pending operations are flushed before +* will release device resources, thus preventing any interaction with +* the HW. +* +* If there is still FW-related work running after this function returns, +* they must use drm_dev_{enter,exit}() and skip any HW access when +* drm_dev_enter() returns false. +*/ +void panthor_fw_unplug (struct panthor_device *ptdev) +{ +struct panthor_fw_section *section; + +cancel_delayed_work_sync (&ptdev->fw->watchdog.ping_work); + +/* Make sure the IRQ handler can be called after that point. */
drivers/gpu/drm/panthor/panthor_fw.c line 1113
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +10
-0+if (ptdev->fw->irq.irq) +panthor_job_irq_suspend (&ptdev->fw->irq); + +panthor_fw_stop (ptdev); + +if (ptdev->fw->vm) +panthor_vm_idle (ptdev->fw->vm); + +list_for_each_entry (section, &ptdev->fw->sections, node) +panthor_kernel_bo_destroy (panthor_fw_vm (ptdev), section->mem);
Here's where we destroy the potentially invalid section->mem.
Note that the issues are twofold:
Firstly, section->mem could be an error pointer as mentioned earlier. Secondly, panthor_kernel_bo_destroy() doesn't actually check for error values or even NULL.
It would probably be worth adding NULL checks to panthor_kernel_bo_destroy() and panthor_kernel_bo_vunmap() to protect against this.
drivers/gpu/drm/panthor/panthor_fw.c line 1119
Quoted 6 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +6
-0+ +panthor_vm_put (ptdev->fw->vm); + +panthor_gpu_power_off (ptdev,L2 , ptdev->gpu_info.l2_present,20000 ); +} +
Cheers,
Chris
Boris Brezillon
maintainerRe: [PATCH v3 08/14] drm/panthor: Add the FW logical block
Quoted 164 lines · On Mon, 18 Dec 2023 21:25:16 +0000 Chris Diamand <[email protected]>…
Quoted 37 lines · Boris Brezillon
Contains everything that's FW related, that includes the code dealing with the microcontroller unit (MCU) that's running the FW, and anything related to allocating memory shared between the FW and the CPU.
A few global FW events are processed in the IRQ handler, the rest is forwarded to the scheduler, since scheduling is the primary reason for the FW existence, and also the main source of FW <-> kernel interactions.
v3:
- Make the FW path more future-proof (Liviu)
- Use one waitqueue for all FW events
- Simplify propagation of FW events to the scheduler logic
- Drop thepanthor_fw_memabstraction and usepanthor_kernel_boinstead
- Account for thepanthor_vmchanges
- Replace magic number with0x7fffffffwith ~0 to better signify that it's the maximum permitted value.
- More accurate rounding when computing the firmware timeout.
- Add a 'sub iterator' helper function. This also adds a check that a firmware entry doesn't overflow the firmware image.
- Drop__packedfrom FW structures, natural alignment is good enough.
- Other minor code improvements.Signed-off-by Boris Brezillon <[email protected]>Signed-off-by Steven Price <[email protected]>--- drivers/gpu/drm/panthor/panthor_fw.c | 1332 ++++++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_fw.h | 504 ++++++++++ 2 files changed, 1836 insertions(+) create mode 100644 drivers/gpu/drm/panthor/panthor_fw.c create mode 100644 drivers/gpu/drm/panthor/panthor_fw.hdrivers/gpu/drm/panthor/panthor_fw.c new file +0
-0...
Quoted 110 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +110
-0+static int panthor_fw_load_section_entry (struct panthor_device *ptdev, +const struct firmware *fw, +struct panthor_fw_binary_iter *iter, +u32 ehdr) +{ +struct panthor_fw_binary_section_entry_hdr hdr; +struct panthor_fw_section *section; +u32 section_size; +u32 name_len; +int ret; + + ret =panthor_fw_binary_iter_read (ptdev, iter, &hdr,sizeof (hdr)); +if (ret) +return ret; + +if (hdr.data.end < hdr.data.start) { +drm_err (&ptdev->base,"Firmware corrupted, data.end < data.start (0x%x < 0x%x)\n" , + hdr.data.end, hdr.data.start); +return -EINVAL ; + } + +if (hdr.va.end < hdr.va.start) { +drm_err (&ptdev->base,"Firmware corrupted, hdr.va.end < hdr.va.start (0x%x < 0x%x)\n" , + hdr.va.end, hdr.va.start); +return -EINVAL ; + } + +if (hdr.data.end > fw->size) { +drm_err (&ptdev->base,"Firmware corrupted, file truncated? data_end=0x%x > fw size=0x%zx\n" , + hdr.data.end, fw->size); +return -EINVAL ; + } + +if ((hdr.va.start & ~PAGE_MASK ) !=0 || + (hdr.va.end & ~PAGE_MASK ) !=0 ) { +drm_err (&ptdev->base,"Firmware corrupted, virtual addresses not page aligned: 0x%x-0x%x\n" , + hdr.va.start, hdr.va.end); +return -EINVAL ; + } + +if (hdr.flags & ~CSF_FW_BINARY_IFACE_ENTRY_RD_SUPPORTED_FLAGS ) { +drm_err (&ptdev->base,"Firmware contains interface with unsupported flags (0x%x)\n" , + hdr.flags); +return -EINVAL ; + } + +if (hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_PROT ) { +drm_warn (&ptdev->base, +"Firmware protected mode entry not be supported, ignoring" ); +return 0 ; + } + +if (hdr.va.start ==CSF_MCU_SHARED_REGION_START && + !(hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_SHARED )) { +drm_err (&ptdev->base, +"Interface at 0x%llx must be shared" ,CSF_MCU_SHARED_REGION_START ); +return -EINVAL ; + } + + name_len = iter->size - iter->offset; + + section =drmm_kzalloc (&ptdev->base,sizeof (*section),GFP_KERNEL ); +if (!section) +return -ENOMEM ; + +list_add_tail (§ion->node, &ptdev->fw->sections); + section->flags = hdr.flags; + section->data.size = hdr.data.end - hdr.data.start; + +if (section->data.size >0 ) { +void *data =drmm_kmalloc (&ptdev->base, section->data.size,GFP_KERNEL ); + +if (!data) +return -ENOMEM ; + +memcpy (data, fw->data + hdr.data.start, section->data.size); + section->data.buf = data; + } + +if (name_len >0 ) { +char *name =drmm_kmalloc (&ptdev->base, name_len +1 ,GFP_KERNEL ); + +if (!name) +return -ENOMEM ; + +memcpy (name, iter->data + iter->offset, name_len); + name[name_len] ='\0' ; + section->name = name; + } + + section_size = hdr.va.end - hdr.va.start; +if (section_size) { +u32 cache_mode = hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_MASK ; +struct panthor_gem_object *bo; +u32 vm_map_flags =0 ; +struct sg_table *sgt; +u64 va = hdr.va.start; + +if (!(hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_WR )) + vm_map_flags |=DRM_PANTHOR_VM_BIND_OP_MAP_READONLY ; + +if (!(hdr.flags &CSF_FW_BINARY_IFACE_ENTRY_RD_EX )) + vm_map_flags |=DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC ; + +/* TODO: CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_*_COHERENT are mapped to +* non-cacheable for now. We might want to introduce a new +* IOMMU_xxx flag (or abuse IOMMU_MMIO, which maps to device +* memory and is currently not used by our driver) for +* AS_MEMATTR_AARCH64_SHARED memory, so we can take benefit +* of IO-coherent systems. drivers/gpu/drm/panthor/panthor_fw.c line 604
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +10
-0+*/ +if (cache_mode !=CSF_FW_BINARY_IFACE_ENTRY_RD_CACHE_MODE_CACHED ) + vm_map_flags |=DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED ; + + section->mem =panthor_kernel_bo_create (ptdev,panthor_fw_vm (ptdev), + section_size, +DRM_PANTHOR_BO_NO_MMAP , + vm_map_flags, va); +if (IS_ERR (section->mem)) +return PTR_ERR (section->mem);There's a small bug here - if
panthor_kernel_bo_create()fails,section->memwill be left containing an error value, not NULL or a valid pointer.
Quoted 4 lines
This means that on rmmod,
panthor_fw_unplug()will try and free the BO even though it was never allocated, causing a kernel panic.The solution is obviously to only assign
section->memif the allocation actually succeeded.
Indeed.
Quoted 4 lines
However I also wonder if the
list_add_tail()call should be moved to later in the function, after we know everything else has succeeded, or if this function needs some more error handling in general.
Actually, I did the insertion early such that any initialization step that needs to be undone if something fails in the middle doesn't have to take place in some additional error path in this function, but only in panthor_fw_unplug() instead. What I lack though, is an
if (!IS_ERR_OR_NULL(section->mem)) panthor_kernel_bo_destroy()
in panthor_fw_unplug().
Quoted 62 lines
Quoted 27 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +26
-0+ +if (drm_WARN_ON (&ptdev->base, section->mem->va_node.start != hdr.va.start)) +return -EINVAL ; + +if (section->flags &CSF_FW_BINARY_IFACE_ENTRY_RD_SHARED ) { + ret =panthor_kernel_bo_vmap (section->mem); +if (ret) +return ret; + } + +panthor_fw_init_section_mem (ptdev, section); + + bo =to_panthor_bo (section->mem->obj); + sgt =drm_gem_shmem_get_pages_sgt (&bo->base); +if (IS_ERR (sgt)) +return PTR_ERR (section->mem); + +dma_sync_sgtable_for_device (ptdev->base.dev, sgt,DMA_TO_DEVICE ); + } + +if (hdr.va.start ==CSF_MCU_SHARED_REGION_START ) + ptdev->fw->shared_section = section; + +return 0 ; +} +...
Quoted 19 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +19
-0+/** +* panthor_fw_unplug() - Called when the device is unplugged. +* @ptdev: Device. +* +* This function must make sure all pending operations are flushed before +* will release device resources, thus preventing any interaction with +* the HW. +* +* If there is still FW-related work running after this function returns, +* they must use drm_dev_{enter,exit}() and skip any HW access when +* drm_dev_enter() returns false. +*/ +void panthor_fw_unplug (struct panthor_device *ptdev) +{ +struct panthor_fw_section *section; + +cancel_delayed_work_sync (&ptdev->fw->watchdog.ping_work); + +/* Make sure the IRQ handler can be called after that point. */ drivers/gpu/drm/panthor/panthor_fw.c line 1113
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +10
-0+if (ptdev->fw->irq.irq) +panthor_job_irq_suspend (&ptdev->fw->irq); + +panthor_fw_stop (ptdev); + +if (ptdev->fw->vm) +panthor_vm_idle (ptdev->fw->vm); + +list_for_each_entry (section, &ptdev->fw->sections, node) +panthor_kernel_bo_destroy (panthor_fw_vm (ptdev), section->mem);Here's where we destroy the potentially invalid
section->mem.
Quoted 5 lines
Note that the issues are twofold:
Firstly,section->memcould be an error pointer as mentioned earlier. Secondly,panthor_kernel_bo_destroy()doesn't actually check for error values or even NULL.It would probably be worth adding NULL checks to
panthor_kernel_bo_destroy()andpanthor_kernel_bo_vunmap()to protect against this.
Or we just add an
if (IS_ERR_OR_NULL(section->mem))
check here instead, unless we have other call sites where kernel_bo can be ERR_OR_NULL.
Quoted 10 lines
drivers/gpu/drm/panthor/panthor_fw.c line 1119
Quoted 7 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +6
-0+ +panthor_vm_put (ptdev->fw->vm); + +panthor_gpu_power_off (ptdev,L2 , ptdev->gpu_info.l2_present,20000 ); +} +Cheers,
Chris
Boris Brezillon
maintainerRe: [PATCH v3 08/14] drm/panthor: Add the FW logical block
In reply to Chris DiamandQuoted 22 lines · On Mon, 18 Dec 2023 21:25:16 +0000 Chris Diamand <[email protected]>…
Quoted 18 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +7
-0+void panthor_fw_unplug (struct panthor_device *ptdev) +{ +struct panthor_fw_section *section; + +cancel_delayed_work_sync (&ptdev->fw->watchdog.ping_work); + +/* Make sure the IRQ handler can be called after that point. */
drivers/gpu/drm/panthor/panthor_fw.c new file +10
-0+if (ptdev->fw->irq.irq) +panthor_job_irq_suspend (&ptdev->fw->irq); + +panthor_fw_stop (ptdev); + +if (ptdev->fw->vm) +panthor_vm_idle (ptdev->fw->vm); + +list_for_each_entry (section, &ptdev->fw->sections, node) + panthor_kernel_bo_destroy(panthor_fw_vm(ptdev),Quoted 1 line
section->mem);Here's where we destroy the potentially invalid
section->mem.
Quoted 8 lines
Note that the issues are twofold:
Firstly,section->memcould be an error pointer as mentioned earlier. Secondly,panthor_kernel_bo_destroy()doesn't actually check for error values or even NULL.It would probably be worth adding NULL checks to
panthor_kernel_bo_destroy()andpanthor_kernel_bo_vunmap()to protect against this.
I ended up implementing your suggestion, because it simplifies all call-sites in panthor_sched.c too. The new version defer the IS_ERR_OR_NULL() check to panthor_kernel_bo_destroy().
Re: [PATCH v3 08/14] drm/panthor: Add the FW logical block
Quoted 30 lines · Boris Brezillon
Quoted 27 lines · On Mon, 18 Dec 2023 21:25:16 +0000 Chris Diamand <[email protected]>…
Quoted 18 lines
drivers/gpu/drm/panthor/panthor_fw.c new file +7
-0+void panthor_fw_unplug (struct panthor_device *ptdev) +{ +struct panthor_fw_section *section; + +cancel_delayed_work_sync (&ptdev->fw->watchdog.ping_work); + +/* Make sure the IRQ handler can be called after that point. */
drivers/gpu/drm/panthor/panthor_fw.c new file +10
-0+if (ptdev->fw->irq.irq) +panthor_job_irq_suspend (&ptdev->fw->irq); + +panthor_fw_stop (ptdev); + +if (ptdev->fw->vm) +panthor_vm_idle (ptdev->fw->vm); + +list_for_each_entry (section, &ptdev->fw->sections, node) + panthor_kernel_bo_destroy(panthor_fw_vm(ptdev),Quoted 1 line
section->mem);Here's where we destroy the potentially invalid
section->mem.Note that the issues are twofold:
Firstly,section->memcould be an error pointer as mentioned earlier. Secondly,panthor_kernel_bo_destroy()doesn't actually check for error values or even NULL.
Quoted 7 lines
Quoted 3 lines
It would probably be worth adding NULL checks to
panthor_kernel_bo_destroy()andpanthor_kernel_bo_vunmap()to protect against this.I ended up implementing your suggestion, because it simplifies all call-sites in
panthor_sched.ctoo. The new version defer theIS_ERR_OR_NULL()check topanthor_kernel_bo_destroy().
Sounds good to me - thanks!