Skip to contentSkip to conversation

drm/pancsf: Add a new driver for Mali CSF-based GPUs

RFC PATCH Superseded Boris Brezillon 4 participants 10 messages Feb 1, 2023 – Feb 6, 2023

Subsystems (2)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 drm/pancsf: Add a new driver for Mali CSF-based GPUs

Mali v10 (second Valhal iteration) and later GPUs replaced the Job Manager block by a command stream based interface called CSF (for Command Stream Frontend). This interface is not only turning the job chain based submission model into a command stream based one, but also introducing FW-assisted scheduling of command stream queues. This is a fundamental shift in both how userspace is supposed to submit jobs, but also how the driver is architectured. We initially tried to retrofit the CSF model into panfrost, but this ended up introducing unneeded complexity to the existing driver, which we all know is a potential source of regression.

So here comes a brand new driver for CSF-based hardware. This is a preliminary version and some important features are missing (like devfreq , PM support and a memory shrinker implem, to name a few). The goal of this RFC is to gather some preliminary feedback on both the uAPI and some basic building blocks, like the MMU/VM code, the tiler heap allocation logic...

It's also here to give concrete code to refer to for the discussion around scheduling and VM_BIND support that started on the Xe/nouveau threads[1][2]. Right now, I'm still using a custom timesharing-based scheduler, but I plan to give Daniel's suggestion a try (having one drm_gpu_scheduler per drm_sched_entity, and replacing the tick-based scheduler by some group slot manager with an LRU-based group eviction mechanism). I also have a bunch of things I need to figure out regarding the VM-based memory management code. The current design assumes explicit syncs everywhere, but we don't use resv objects yet. I see other modern drivers are adding BOOKKEEP fences to the VM resv object and using this VM resv to synchronize with kernel operations on the VM, but we currently don't do any of that. As Daniel pointed out it's likely to become an issue when we throw the memory shrinker into the mix. And of course, the plan is to transition to the drm_gpuva_manager infrastructure being discussed here [2] before merging the driver. Kind of related to this shrinker topic, I'm wondering if it wouldn't make sense to use the TTM infra for our buffer management (AFAIU, we'd get LRU-based BO eviction for free, without needing to expose an MADVISE(DONT_NEED) kind of API), but I'm a bit worried about the extra complexity this would pull in.

Note that DT bindings are currently undocumented. For those who really care, they're based on the panfrost bindings, so I don't expect any pain points on that front. I'll provide a proper doc once all other aspects have been sorted out.

Regards,

Boris

[1]https://lore.kernel.org/dri-devel/[email protected]/ [2]https://lore.kernel.org/lkml/Y8jOCE%[email protected]/

Cc Alyssa Rosenzweig <[email protected]>
Cc Steven Price <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Daniel Vetter <[email protected]>
Cc Daniel Stone <[email protected]>
Cc Jason Ekstrand <[email protected]>
---
 drivers/gpu/drm/Kconfig                |    2 +
 drivers/gpu/drm/Makefile               |    1 +
 drivers/gpu/drm/pancsf/Kconfig         |   15 +
 drivers/gpu/drm/pancsf/Makefile        |   14 +
 drivers/gpu/drm/pancsf/pancsf_device.c |  391 ++++
 drivers/gpu/drm/pancsf/pancsf_device.h |  168 ++
 drivers/gpu/drm/pancsf/pancsf_drv.c    |  812 +++++++
 drivers/gpu/drm/pancsf/pancsf_gem.c    |  161 ++
 drivers/gpu/drm/pancsf/pancsf_gem.h    |   45 +
 drivers/gpu/drm/pancsf/pancsf_gpu.c    |  381 ++++
 drivers/gpu/drm/pancsf/pancsf_gpu.h    |   40 +
 drivers/gpu/drm/pancsf/pancsf_heap.c   |  337 +++
 drivers/gpu/drm/pancsf/pancsf_heap.h   |   30 +
 drivers/gpu/drm/pancsf/pancsf_mcu.c    |  891 ++++++++
 drivers/gpu/drm/pancsf/pancsf_mcu.h    |  313 +++
 drivers/gpu/drm/pancsf/pancsf_mmu.c    | 1345 +++++++++++
 drivers/gpu/drm/pancsf/pancsf_mmu.h    |   51 +
 drivers/gpu/drm/pancsf/pancsf_regs.h   |  225 ++
 drivers/gpu/drm/pancsf/pancsf_sched.c  | 2837 ++++++++++++++++++++++++
 drivers/gpu/drm/pancsf/pancsf_sched.h  |   68 +
 include/uapi/drm/pancsf_drm.h          |  414 ++++
 21 files changed, 8541 insertions(+)
 create mode 100644 drivers/gpu/drm/pancsf/Kconfig
 create mode 100644 drivers/gpu/drm/pancsf/Makefile
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_device.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_device.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_drv.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_gem.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_gem.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_gpu.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_gpu.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_heap.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_heap.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_mcu.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_mcu.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_mmu.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_mmu.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_regs.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_sched.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_sched.h
 create mode 100644 include/uapi/drm/pancsf_drm.h
drivers/gpu/drm/Kconfig +2 -0
index 315cbdf61979..29c18d9f2980 100644
@@ -344,6 +344,8 @@ source "drivers/gpu/drm/vboxvideo/Kconfig"
 
 source "drivers/gpu/drm/lima/Kconfig"
 
+source "drivers/gpu/drm/pancsf/Kconfig"
+
 source "drivers/gpu/drm/panfrost/Kconfig"
 
 source "drivers/gpu/drm/aspeed/Kconfig"
drivers/gpu/drm/Makefile +1 -0
index cc637343d87b..582d2e1fe274 100644
@@ -188,6 +188,7 @@ obj-$(CONFIG_DRM_TVE200) += tve200/
 obj-$(CONFIG_DRM_XEN) += xen/
 obj-$(CONFIG_DRM_VBOXVIDEO) += vboxvideo/
 obj-$(CONFIG_DRM_LIMA)  += lima/
+obj-$(CONFIG_DRM_PANCSF) += pancsf/
 obj-$(CONFIG_DRM_PANFROST) += panfrost/
 obj-$(CONFIG_DRM_ASPEED_GFX) += aspeed/
 obj-$(CONFIG_DRM_MCDE) += mcde/
drivers/gpu/drm/pancsf/Kconfig new file +15 -0
new file mode 100644
index 000000000000..b3937eb240fc
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0
+
+config DRM_PANCSF
+	tristate "PanCSF (DRM support for ARM Mali CSF-based GPUs)"
+	depends on DRM
+	depends on ARM || ARM64 || (COMPILE_TEST && !GENERIC_ATOMIC64)
+	depends on MMU
+	select DRM_SCHED
+	select IOMMU_SUPPORT
+	select IOMMU_IO_PGTABLE_LPAE
+	select DRM_GEM_SHMEM_HELPER
+	select PM_DEVFREQ
+	select DEVFREQ_GOV_SIMPLE_ONDEMAND
+	help
+	  DRM driver for ARM Mali CSF-based GPUs.
drivers/gpu/drm/pancsf/Makefile new file +14 -0
new file mode 100644
index 000000000000..94a4f2d34331
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0
+
+pancsf-y := \
+	pancsf_device.o \
+	pancsf_drv.o \
+	pancsf_gem.o \
+	pancsf_gpu.o \
+	pancsf_heap.o \
+	pancsf_heap.o \
+	pancsf_mcu.o \
+	pancsf_mmu.o \
+	pancsf_sched.o
+
+obj-$(CONFIG_DRM_PANCSF) += pancsf.o
drivers/gpu/drm/pancsf/pancsf_device.c new file +391 -0

391 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_device.h new file +168 -0

168 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_drv.c new file +812 -0

812 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_gem.c new file +161 -0

161 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_gem.h new file +45 -0

45 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_gpu.c new file +381 -0

381 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_gpu.h new file +40 -0
new file mode 100644
index 000000000000..1ee39b8b10b0
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2018 Marty E. Plummer <[email protected]> */
+/* Copyright 2019 Collabora ltd. */
+
+#ifndef __PANCSF_GPU_H__
+#define __PANCSF_GPU_H__
+
+struct pancsf_device;
+
+int pancsf_gpu_init(struct pancsf_device *pfdev);
+void pancsf_gpu_fini(struct pancsf_device *pfdev);
+
+int pancsf_gpu_block_power_on(struct pancsf_device *pfdev,
+			      const char *blk_name,
+			      u32 pwron_reg, u32 pwrtrans_reg,
+			      u32 rdy_reg, u64 mask, u32 timeout_us);
+int pancsf_gpu_block_power_off(struct pancsf_device *pfdev,
+			       const char *blk_name,
+			       u32 pwroff_reg, u32 pwrtrans_reg,
+			       u64 mask, u32 timeout_us);
+
+#define pancsf_gpu_power_on(pfdev, type, mask, timeout_us) \
+	pancsf_gpu_block_power_on(pfdev, #type, \
+				    type ## _PWRON_LO, \
+				    type ## _PWRTRANS_LO, \
+				    type ## _READY_LO, \
+				    mask, timeout_us)
+
+#define pancsf_gpu_power_off(pfdev, type, mask, timeout_us) \
+	pancsf_gpu_block_power_off(pfdev, #type, \
+				     type ## _PWROFF_LO, \
+				     type ## _PWRTRANS_LO, \
+				     mask, timeout_us)
+
+int pancsf_gpu_l2_power_on(struct pancsf_device *pfdev);
+int pancsf_gpu_flush_caches(struct pancsf_device *pfdev,
+			    u32 l2, u32 lsc, u32 other);
+int pancsf_gpu_soft_reset(struct pancsf_device *pfdev);
+
+#endif
drivers/gpu/drm/pancsf/pancsf_heap.c new file +337 -0

337 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_heap.h new file +30 -0
new file mode 100644
index 000000000000..ed395ab33b5e
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2023 Collabora ltd. */
+
+#ifndef __PANCSF_HEAP_H__
+#define __PANCSF_HEAP_H__
+
+#include <linux/types.h>
+
+struct pancsf_device;
+struct pancsf_heap_pool;
+struct pancsf_vm;
+
+int pancsf_heap_create(struct pancsf_heap_pool *pool,
+		       u32 initial_chunk_count,
+		       u32 chunk_size,
+		       u32 max_chunks,
+		       u32 target_in_flight,
+		       u64 *heap_ctx_gpu_va,
+		       u64 *first_chunk_gpu_va);
+int pancsf_heap_destroy(struct pancsf_heap_pool *pool, u32 handle);
+struct pancsf_heap_pool *
+pancsf_heap_pool_create(struct pancsf_device *pfdev, struct pancsf_vm *vm);
+void pancsf_heap_pool_destroy(struct pancsf_heap_pool *pool);
+int pancsf_heap_grow(struct pancsf_heap_pool *pool,
+		     u64 heap_gpu_va,
+		     u32 renderpasses_in_flight,
+		     u32 pending_frag_count,
+		     u64 *new_chunk_gpu_va);
+
+#endif
drivers/gpu/drm/pancsf/pancsf_mcu.c new file +891 -0

591 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_mcu.h new file +313 -0

313 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_mmu.c new file +1345 -0

1,345 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_mmu.h new file +51 -0

51 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_regs.h new file +225 -0

225 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_sched.c new file +2837 -0

2,837 lines not on this page · raw message

drivers/gpu/drm/pancsf/pancsf_sched.h new file +68 -0

68 lines not on this page · raw message

include/uapi/drm/pancsf_drm.h new file +414 -0

414 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.39.0

Part of the diff is not shown here. View the raw message.

#1 lore raw
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in…

Hi Boris,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on drm-misc/drm-misc-next]
[also build test WARNING on drm/drm-next drm-exynos/exynos-drm-next drm-intel/for-linux-next drm-intel/for-linux-next-fixes drm-tip/drm-tip linus/master v6.2-rc6 next-20230201] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Boris-Brezillon/drm-pancsf-Add-a-new-driver-for-Mali-CSF-based-GPUs/20230201-172815
base:   git://anongit.freedesktop.org/drm/drm-misc drm-misc-next
patch link:    https://lore.kernel.org/r/20230201084832.1708866-1-boris.brezillon%40collabora.com
patch subject: [RFC PATCH] drm/pancsf: Add a new driver for Mali CSF-based GPUs
config: sparc-allyesconfig (https://download.01.org/0day-ci/archive/20230201/[email protected]/config)
compiler: sparc64-linux-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/794ce1c0209242bf1496fc4195ca707d45b37e73
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Boris-Brezillon/drm-pancsf-Add-a-new-driver-for-Mali-CSF-based-GPUs/20230201-172815 git checkout 794ce1c0209242bf1496fc4195ca707d45b37e73
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=sparc olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=sparc SHELL=/bin/bash drivers/gpu/

If you fix the issue, kindly add following tag where applicable | Reported-by: kernel test robot <[email protected]>

All warnings (new ones prefixed by >>):

>> drivers/gpu/drm/pancsf/pancsf_drv.c:65:5: warning: no previous prototype for 'pancsf_ioctl_vm_create' [-Wmissing-prototypes]
      65 | int pancsf_ioctl_vm_create(struct drm_device *ddev, void *data,
         |     ^~~~~~~~~~~~~~~~~~~~~~
>> drivers/gpu/drm/pancsf/pancsf_drv.c:84:5: warning: no previous prototype for 'pancsf_ioctl_vm_destroy' [-Wmissing-prototypes]
      84 | int pancsf_ioctl_vm_destroy(struct drm_device *ddev, void *data,
         |     ^~~~~~~~~~~~~~~~~~~~~~~
>> drivers/gpu/drm/pancsf/pancsf_drv.c:258:1: warning: no previous prototype for 'pancsf_free_sync_signal_array' [-Wmissing-prototypes]
     258 | pancsf_free_sync_signal_array(struct pancsf_sync_signal_array *array)
         | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> drivers/gpu/drm/pancsf/pancsf_drv.c:273:1: warning: no previous prototype for 'pancsf_collect_sync_signal_array' [-Wmissing-prototypes]
     273 | pancsf_collect_sync_signal_array(struct drm_file *file,
         | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   drivers/gpu/drm/pancsf/pancsf_drv.c: In function 'pancsf_collect_sync_signal_array':
>> drivers/gpu/drm/pancsf/pancsf_drv.c:290:21: warning: variable 'ret' set but not used [-Wunused-but-set-variable]
     290 |                 int ret;
         |                     ^~~
--
   drivers/gpu/drm/pancsf/pancsf_gem.c: In function 'pancsf_gem_prime_import_sg_table':
>> drivers/gpu/drm/pancsf/pancsf_gem.c:153:35: warning: variable 'bo' set but not used [-Wunused-but-set-variable]
     153 |         struct pancsf_gem_object *bo;
         |                                   ^~
--
>> drivers/gpu/drm/pancsf/pancsf_mcu.c:248:1: warning: no previous prototype for 'pancsf_fw_mem_alloc' [-Wmissing-prototypes]
     248 | pancsf_fw_mem_alloc(struct pancsf_device *pfdev,
         | ^~~~~~~~~~~~~~~~~~~
--
>> drivers/gpu/drm/pancsf/pancsf_mmu.c:394:5: warning: no previous prototype for 'pancsf_vm_unmap_pages' [-Wmissing-prototypes]
     394 | int pancsf_vm_unmap_pages(struct pancsf_vm *vm, u64 iova, size_t size)
         |     ^~~~~~~~~~~~~~~~~~~~~
   drivers/gpu/drm/pancsf/pancsf_mmu.c: In function 'pancsf_vm_unmap_pages':
>> drivers/gpu/drm/pancsf/pancsf_mmu.c:399:13: warning: variable 'ret' set but not used [-Wunused-but-set-variable]
     399 |         int ret;
         |             ^~~
--
   drivers/gpu/drm/pancsf/pancsf_sched.c: In function 'pancsf_prog_csg_slot_locked':
>> drivers/gpu/drm/pancsf/pancsf_sched.c:608:46: warning: variable 'glb_iface' set but not used [-Wunused-but-set-variable]
     608 |         const struct pancsf_fw_global_iface *glb_iface;
         |                                              ^~~~~~~~~
   drivers/gpu/drm/pancsf/pancsf_sched.c: In function 'pancsf_handle_cs_fatal':
>> drivers/gpu/drm/pancsf/pancsf_sched.c:667:42: warning: variable 'csg_iface' set but not used [-Wunused-but-set-variable]
     667 |         const struct pancsf_fw_cs_iface *csg_iface;
         |                                          ^~~~~~~~~
   drivers/gpu/drm/pancsf/pancsf_sched.c: In function 'pancsf_handle_tiler_oom':
>> drivers/gpu/drm/pancsf/pancsf_sched.c:725:13: warning: variable 'info' set but not used [-Wunused-but-set-variable]
     725 |         u64 info, heap_address, new_chunk_va;
         |             ^~~~
>> drivers/gpu/drm/pancsf/pancsf_sched.c:723:13: warning: variable 'fault' set but not used [-Wunused-but-set-variable]
     723 |         u32 fault, vt_start, vt_end, frag_end;
         |             ^~~~~
>> drivers/gpu/drm/pancsf/pancsf_sched.c:722:30: warning: variable 'stream' set but not used [-Wunused-but-set-variable]
     722 |         struct pancsf_queue *stream;
         |                              ^~~~~~
   drivers/gpu/drm/pancsf/pancsf_sched.c: In function 'tick_ctx_init':
   drivers/gpu/drm/pancsf/pancsf_sched.c:1478:40: warning: variable 'glb_iface' set but not used [-Wunused-but-set-variable]
    1478 |         struct pancsf_fw_global_iface *glb_iface;
         |                                        ^~~~~~~~~
   drivers/gpu/drm/pancsf/pancsf_sched.c: In function 'tick_ctx_apply':
   drivers/gpu/drm/pancsf/pancsf_sched.c:1651:59: warning: variable 'csg_iface' set but not used [-Wunused-but-set-variable]
    1651 |                         const struct pancsf_fw_csg_iface *csg_iface;
         |                                                           ^~~~~~~~~
   drivers/gpu/drm/pancsf/pancsf_sched.c:1638:40: warning: variable 'glb_iface' set but not used [-Wunused-but-set-variable]
    1638 |         struct pancsf_fw_global_iface *glb_iface;
         |                                        ^~~~~~~~~
   drivers/gpu/drm/pancsf/pancsf_sched.c: In function 'pancsf_sched_pre_reset':
>> drivers/gpu/drm/pancsf/pancsf_sched.c:2153:13: warning: variable 'ret' set but not used [-Wunused-but-set-variable]
    2153 |         int ret;
         |             ^~~
   drivers/gpu/drm/pancsf/pancsf_sched.c: In function 'pancsf_group_sync_upd_work':
>> drivers/gpu/drm/pancsf/pancsf_sched.c:2316:52: warning: variable 'fence' set but not used [-Wunused-but-set-variable]
    2316 |                         struct pancsf_queue_fence *fence;
         |                                                    ^~~~~

vim +/pancsf_ioctl_vm_create +65 drivers/gpu/drm/pancsf/pancsf_drv.c

    64	
  > 65	int pancsf_ioctl_vm_create(struct drm_device *ddev, void *data,
    66				   struct drm_file *file)
    67	{
    68		struct pancsf_device *pfdev = ddev->dev_private;
    69		struct pancsf_file *pfile = file->driver_priv;
    70		struct drm_pancsf_vm_create *args = data;
    71		int ret;
    72	
    73		if (args->flags & ~PANCSF_VM_CREATE_FLAGS)
    74			return -EINVAL;
    75	
    76		ret = pancsf_vm_pool_create_vm(pfdev, pfile->vms);
    77		if (ret < 0)
    78			return ret;
    79	
    80		args->id = ret;
    81		return 0;
    82	}
    83	
  > 84	int pancsf_ioctl_vm_destroy(struct drm_device *ddev, void *data,
    85				    struct drm_file *file)
    86	{
    87		struct pancsf_file *pfile = file->driver_priv;
    88		struct drm_pancsf_vm_destroy *args = data;
    89	
    90		pancsf_vm_pool_destroy_vm(pfile->vms, args->id);
    91		return 0;
    92	}
    93	
    94	#define PANCSF_BO_FLAGS		0
    95	
    96	static int pancsf_ioctl_bo_create(struct drm_device *ddev, void *data,
    97					  struct drm_file *file)
    98	{
    99		struct pancsf_gem_object *bo;
   100		struct drm_pancsf_bo_create *args = data;
   101	
   102		if (!args->size || args->pad ||
   103		    (args->flags & ~PANCSF_BO_FLAGS))
   104			return -EINVAL;
   105	
   106		bo = pancsf_gem_create_with_handle(file, ddev, args->size, args->flags,
   107						   &args->handle);
   108		if (IS_ERR(bo))
   109			return PTR_ERR(bo);
   110	
   111		return 0;
   112	}
   113	
   114	#define PANCSF_VMA_MAP_FLAGS (PANCSF_VMA_MAP_READONLY | \
   115				      PANCSF_VMA_MAP_NOEXEC | \
   116				      PANCSF_VMA_MAP_UNCACHED | \
   117				      PANCSF_VMA_MAP_FRAG_SHADER | \
   118				      PANCSF_VMA_MAP_ON_FAULT | \
   119				      PANCSF_VMA_MAP_AUTO_VA)
   120	
   121	static int pancsf_ioctl_vm_map(struct drm_device *ddev, void *data,
   122				       struct drm_file *file)
   123	{
   124		struct pancsf_file *pfile = file->driver_priv;
   125		struct drm_pancsf_vm_map *args = data;
   126		struct drm_gem_object *gem;
   127		struct pancsf_vm *vm;
   128		int ret;
   129	
   130		if (args->flags & ~PANCSF_VMA_MAP_FLAGS)
   131			return -EINVAL;
   132	
   133		gem = drm_gem_object_lookup(file, args->bo_handle);
   134		if (!gem)
   135			return -EINVAL;
   136	
   137		vm = pancsf_vm_pool_get_vm(pfile->vms, args->vm_id);
   138		if (vm) {
   139			ret = pancsf_vm_map_bo_range(vm, to_pancsf_bo(gem), args->bo_offset,
   140						     args->size, &args->va, args->flags);
   141		} else {
   142			ret = -EINVAL;
   143		}
   144	
   145		pancsf_vm_put(vm);
   146		drm_gem_object_put(gem);
   147		return ret;
   148	}
   149	
   150	#define PANCSF_VMA_UNMAP_FLAGS 0
   151	
   152	static int pancsf_ioctl_vm_unmap(struct drm_device *ddev, void *data,
   153					 struct drm_file *file)
   154	{
   155		struct pancsf_file *pfile = file->driver_priv;
   156		struct drm_pancsf_vm_unmap *args = data;
   157		struct pancsf_vm *vm;
   158		int ret;
   159	
   160		if (args->flags & ~PANCSF_VMA_UNMAP_FLAGS)
   161			return -EINVAL;
   162	
   163		vm = pancsf_vm_pool_get_vm(pfile->vms, args->vm_id);
   164		if (vm)
   165			ret = pancsf_vm_unmap_range(vm, args->va, args->size);
   166		else
   167			ret = -EINVAL;
   168	
   169		pancsf_vm_put(vm);
   170		return ret;
   171	}
   172	
   173	static void *pancsf_get_obj_array(struct drm_pancsf_obj_array *in, u32 min_stride)
   174	{
   175		u32 elem_size = min_t(u32, in->stride, min_stride);
   176		int ret = 0;
   177		void *out;
   178	
   179		if (in->stride < min_stride)
   180			return ERR_PTR(-EINVAL);
   181	
   182		out = kvmalloc_array(in->count, elem_size, GFP_KERNEL);
   183		if (!out)
   184			return ERR_PTR(-ENOMEM);
   185	
   186		if (elem_size == in->stride) {
   187			if (copy_from_user(out, u64_to_user_ptr(in->array), elem_size * in->count))
   188				ret = -EFAULT;
   189		} else {
   190			void __user *in_ptr = u64_to_user_ptr(in->array);
   191			void *out_ptr = out;
   192			u32 i;
   193	
   194			for (i = 0; i < in->count; i++) {
   195				if (copy_from_user(out_ptr, in_ptr, elem_size)) {
   196					ret = -EFAULT;
   197					break;
   198				}
   199			}
   200		}
   201	
   202		if (ret) {
   203			kvfree(out);
   204			return ERR_PTR(ret);
   205		}
   206	
   207		return out;
   208	}
   209	
   210	static int pancsf_add_job_deps(struct drm_file *file, struct pancsf_job *job,
   211				       struct drm_pancsf_sync_op *sync_ops, u32 sync_op_count)
   212	{
   213		u32 i;
   214	
   215		for (i = 0; i < sync_op_count; i++) {
   216			struct dma_fence *fence;
   217			int ret;
   218	
   219			if (sync_ops[i].op_type != DRM_PANCSF_SYNC_OP_WAIT)
   220				continue;
   221	
   222			switch (sync_ops[i].handle_type) {
   223			case DRM_PANCSF_SYNC_HANDLE_TYPE_SYNCOBJ:
   224			case DRM_PANCSF_SYNC_HANDLE_TYPE_TIMELINE_SYNCOBJ:
   225				ret = drm_syncobj_find_fence(file, sync_ops[i].handle,
   226							     sync_ops[i].timeline_value,
   227							     0, &fence);
   228				if (ret)
   229					return ret;
   230	
   231				ret = pancsf_add_job_dep(job, fence);
   232				if (ret) {
   233					dma_fence_put(fence);
   234					return ret;
   235				}
   236				break;
   237	
   238			default:
   239				return -EINVAL;
   240			}
   241		}
   242	
   243		return 0;
   244	}
   245	
   246	struct pancsf_sync_signal {
   247		struct drm_syncobj *syncobj;
   248		struct dma_fence_chain *chain;
   249		u64 point;
   250	};
   251	
   252	struct pancsf_sync_signal_array {
   253		struct pancsf_sync_signal *signals;
   254		u32 count;
   255	};
   256	
   257	void
 > 258	pancsf_free_sync_signal_array(struct pancsf_sync_signal_array *array)

259 {
260 u32 i;
261
262 for (i = 0; i < array->count; i++) {
263 drm_syncobj_put(array->signals[i].syncobj);
264 dma_fence_chain_free(array->signals[i].chain);
265 }
266
267 kvfree(array->signals);
268 array->signals = NULL;
269 array->count = 0;
270 }
271

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#2 lore raw
In reply to Boris Brezillon
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in…

Hi Boris,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on drm-misc/drm-misc-next]
[also build test WARNING on drm/drm-next drm-exynos/exynos-drm-next drm-intel/for-linux-next drm-intel/for-linux-next-fixes drm-tip/drm-tip linus/master v6.2-rc6 next-20230201] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Boris-Brezillon/drm-pancsf-Add-a-new-driver-for-Mali-CSF-based-GPUs/20230201-172815
base:   git://anongit.freedesktop.org/drm/drm-misc drm-misc-next
patch link:    https://lore.kernel.org/r/20230201084832.1708866-1-boris.brezillon%40collabora.com
patch subject: [RFC PATCH] drm/pancsf: Add a new driver for Mali CSF-based GPUs
config: arc-allyesconfig (https://download.01.org/0day-ci/archive/20230202/[email protected]/config)
compiler: arceb-elf-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/794ce1c0209242bf1496fc4195ca707d45b37e73
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Boris-Brezillon/drm-pancsf-Add-a-new-driver-for-Mali-CSF-based-GPUs/20230201-172815 git checkout 794ce1c0209242bf1496fc4195ca707d45b37e73
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=arc olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=arc SHELL=/bin/bash drivers/gpu/drm/pancsf/

If you fix the issue, kindly add following tag where applicable | Reported-by: kernel test robot <[email protected]>

All warnings (new ones prefixed by >>):

drivers/gpu/drm/pancsf/pancsf_drv.c:65:5: warning: no previous prototype for 'pancsf_ioctl_vm_create' [-Wmissing-prototypes]
65 | int pancsf_ioctl_vm_create(struct drm_device *ddev, void *data,
| ^~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/pancsf/pancsf_drv.c:84:5: warning: no previous prototype for 'pancsf_ioctl_vm_destroy' [-Wmissing-prototypes]
84 | int pancsf_ioctl_vm_destroy(struct drm_device *ddev, void *data,
| ^~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/pancsf/pancsf_drv.c:258:1: warning: no previous prototype for 'pancsf_free_sync_signal_array' [-Wmissing-prototypes]
258 | pancsf_free_sync_signal_array(struct pancsf_sync_signal_array *array)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/pancsf/pancsf_drv.c:273:1: warning: no previous prototype for 'pancsf_collect_sync_signal_array' [-Wmissing-prototypes]
273 | pancsf_collect_sync_signal_array(struct drm_file *file,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/pancsf/pancsf_drv.c: In function 'pancsf_collect_sync_signal_array':
drivers/gpu/drm/pancsf/pancsf_drv.c:290:21: warning: variable 'ret' set but not used [-Wunused-but-set-variable]

     290 |                 int ret;
         |                     ^~~
   drivers/gpu/drm/pancsf/pancsf_drv.c: In function 'pancsf_mmap_io':
>> drivers/gpu/drm/pancsf/pancsf_drv.c:628:9: warning: conversion from 'long long unsigned int' to 'long unsigned int' changes value from '18446462598732840960' to '0' [-Woverflow]
     628 |         case DRM_PANCSF_USER_FLUSH_ID_MMIO_OFFSET:
         |         ^~~~

vim +628 drivers/gpu/drm/pancsf/pancsf_drv.c

618
619 static int pancsf_mmap_io(struct file *filp, struct vm_area_struct *vma)
620 {
621 struct drm_file *priv = filp->private_data;
622 struct pancsf_file *pfile = priv->driver_priv;
623 struct pancsf_device *pfdev = pfile->pfdev;
624 phys_addr_t phys_offset;
625 size_t size;
626
627 switch (vma->vm_pgoff << PAGE_SHIFT) {

 > 628		case DRM_PANCSF_USER_FLUSH_ID_MMIO_OFFSET:
   629			size = PAGE_SIZE;
   630			phys_offset = 0;
   631			break;
   632		default:
   633			return -EINVAL;
   634		}
   635	
   636		if (vma->vm_end - vma->vm_start != size)
   637			return -EINVAL;
   638	
   639		vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE |
   640				 VM_DONTDUMP | VM_PFNMAP;
   641	
   642		vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
   643	
   644		return io_remap_pfn_range(vma,
   645					  vma->vm_start,
   646					  (pfdev->phys_addr + phys_offset) >> PAGE_SHIFT,
   647					  size,
   648					  vma->vm_page_prot);

649 }
650

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#3 lore raw
In reply to Boris Brezillon

Hi Boris,

Thanks for the post - it's great to see the progress!

Quoted 10 lines · Boris Brezillon

Mali v10 (second Valhal iteration) and later GPUs replaced the Job Manager block by a command stream based interface called CSF (for Command Stream Frontend). This interface is not only turning the job chain based submission model into a command stream based one, but also introducing FW-assisted scheduling of command stream queues. This is a fundamental shift in both how userspace is supposed to submit jobs, but also how the driver is architectured. We initially tried to retrofit the CSF model into panfrost, but this ended up introducing unneeded complexity to the existing driver, which we all know is a potential source of regression.

While I agree there's some big differences which effectively mandate splitting the driver I do think there are some parts which make a lot of sense to share.

For example pancsf_regs.h and panfrost_regs.h are really quite similar and I think could easily be combined. The clock/regulator code is pretty much a direct copy/paste (just adding support for more clocks), etc.

What would be ideal is factoring out 'generic' parts from panfrost and then being able to use them from pancsf.

I had a go at starting that:

https://gitlab.arm.com/linux-arm/linux-sp/-/tree/pancsf-refactor

(lightly tested for Panfrost, only build tested for pancsf).

That saves around 200 lines overall and avoids needing to maintain two lots of clock/regulator code. There's definite scope for sharing (most) register definitions between the drivers and quite possibly some of the MMU/memory code (although there's diminishing returns there).

Quoted 6 lines

So here comes a brand new driver for CSF-based hardware. This is a preliminary version and some important features are missing (like devfreq , PM support and a memory shrinker implem, to name a few). The goal of this RFC is to gather some preliminary feedback on both the uAPI and some basic building blocks, like the MMU/VM code, the tiler heap allocation logic...

At the moment I don't have any CSF hardware available, so this review is a pure code review. I'll try to organise some hardware and do some testing, but it's probably going to take a while to arrive and get setup.

Quoted 12 lines

It's also here to give concrete code to refer to for the discussion around scheduling and VM_BIND support that started on the Xe/nouveau threads[1][2]. Right now, I'm still using a custom timesharing-based scheduler, but I plan to give Daniel's suggestion a try (having one drm_gpu_scheduler per drm_sched_entity, and replacing the tick-based scheduler by some group slot manager with an LRU-based group eviction mechanism). I also have a bunch of things I need to figure out regarding the VM-based memory management code. The current design assumes explicit syncs everywhere, but we don't use resv objects yet. I see other modern drivers are adding BOOKKEEP fences to the VM resv object and using this VM resv to synchronize with kernel operations on the VM, but we currently don't do any of that. As Daniel pointed out it's likely to

Quoted 8 lines

become an issue when we throw the memory shrinker into the mix. And of course, the plan is to transition to the drm_gpuva_manager infrastructure being discussed here [2] before merging the driver. Kind of related to this shrinker topic, I'm wondering if it wouldn't make sense to use the TTM infra for our buffer management (AFAIU, we'd get LRU-based BO eviction for free, without needing to expose an MADVISE(DONT_NEED) kind of API), but I'm a bit worried about the extra complexity this would pull in.

I'm afraid I haven't done much review of this yet - my knowledge of how this is done in kbase is lacking as I was already leaving GPU around the time this was being implemented... but since it's all about to change perhaps that's for the best ;)

Quoted 62 lines

Note that DT bindings are currently undocumented. For those who really care, they're based on the panfrost bindings, so I don't expect any pain points on that front. I'll provide a proper doc once all other aspects have been sorted out.

Regards,

Boris

[1]https://lore.kernel.org/dri-devel/[email protected]/ [2]https://lore.kernel.org/lkml/Y8jOCE%[email protected]/

Cc Alyssa Rosenzweig <[email protected]>
Cc Steven Price <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Daniel Vetter <[email protected]>
Cc Daniel Stone <[email protected]>
Cc Jason Ekstrand <[email protected]>
---
 drivers/gpu/drm/Kconfig                |    2 +
 drivers/gpu/drm/Makefile               |    1 +
 drivers/gpu/drm/pancsf/Kconfig         |   15 +
 drivers/gpu/drm/pancsf/Makefile        |   14 +
 drivers/gpu/drm/pancsf/pancsf_device.c |  391 ++++
 drivers/gpu/drm/pancsf/pancsf_device.h |  168 ++
 drivers/gpu/drm/pancsf/pancsf_drv.c    |  812 +++++++
 drivers/gpu/drm/pancsf/pancsf_gem.c    |  161 ++
 drivers/gpu/drm/pancsf/pancsf_gem.h    |   45 +
 drivers/gpu/drm/pancsf/pancsf_gpu.c    |  381 ++++
 drivers/gpu/drm/pancsf/pancsf_gpu.h    |   40 +
 drivers/gpu/drm/pancsf/pancsf_heap.c   |  337 +++
 drivers/gpu/drm/pancsf/pancsf_heap.h   |   30 +
 drivers/gpu/drm/pancsf/pancsf_mcu.c    |  891 ++++++++
 drivers/gpu/drm/pancsf/pancsf_mcu.h    |  313 +++
 drivers/gpu/drm/pancsf/pancsf_mmu.c    | 1345 +++++++++++
 drivers/gpu/drm/pancsf/pancsf_mmu.h    |   51 +
 drivers/gpu/drm/pancsf/pancsf_regs.h   |  225 ++
 drivers/gpu/drm/pancsf/pancsf_sched.c  | 2837 ++++++++++++++++++++++++
 drivers/gpu/drm/pancsf/pancsf_sched.h  |   68 +
 include/uapi/drm/pancsf_drm.h          |  414 ++++
 21 files changed, 8541 insertions(+)
 create mode 100644 drivers/gpu/drm/pancsf/Kconfig
 create mode 100644 drivers/gpu/drm/pancsf/Makefile
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_device.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_device.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_drv.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_gem.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_gem.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_gpu.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_gpu.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_heap.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_heap.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_mcu.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_mcu.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_mmu.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_mmu.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_regs.h
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_sched.c
 create mode 100644 drivers/gpu/drm/pancsf/pancsf_sched.h
 create mode 100644 include/uapi/drm/pancsf_drm.h

<snip>

Quoted 24 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +18 -0
new file mode 100644
index 000000000000..05c4df13ba90
@@ -0,0 +1,812 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2018 Marty E. Plummer <[email protected]> */
+/* Copyright 2019 Linaro, Ltd., Rob Herring <[email protected]> */
+/* Copyright 2019 Collabora ltd. */
+
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/pagemap.h>
+#include <linux/pm_runtime.h>
+#include <drm/pancsf_drm.h>
+#include <drm/drm_drv.h>
+#include <drm/drm_ioctl.h>
+#include <drm/drm_syncobj.h>
+#include <drm/drm_utils.h>
+#include <drm/drm_debugfs.h>
+
+#include "pancsf_sched.h"
+#include "pancsf_device.h"

drivers/gpu/drm/pancsf/pancsf_drv.c line 28

Quoted 10 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +10 -0
+#include "pancsf_gem.h"
+#include "pancsf_heap.h"
+#include "pancsf_mcu.h"
+#include "pancsf_mmu.h"
+#include "pancsf_gpu.h"
+
+#define DRM_PANCSF_SYNC_OP_MIN_SIZE		24
+#define DRM_PANCSF_QUEUE_SUBMIT_MIN_SIZE	32
+#define DRM_PANCSF_QUEUE_CREATE_MIN_SIZE	8
+#define DRM_PANCSF_VM_BIND_OP_MIN_SIZE		48

I'm not sure why these are #defines rather than using sizeof()?

Quoted 17 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +17 -0
+static int pancsf_ioctl_dev_query(struct drm_device *ddev, void *data, struct drm_file *file)
+{
+	struct drm_pancsf_dev_query *args = data;
+	struct pancsf_device *pfdev = ddev->dev_private;
+	const void *src;
+	size_t src_size;
+
+	switch (args->type) {
+	case DRM_PANCSF_DEV_QUERY_GPU_INFO:
+		src_size = sizeof(pfdev->gpu_info);
+		src = &pfdev->gpu_info;
+		break;
+	case DRM_PANCSF_DEV_QUERY_CSIF_INFO:
+		src_size = sizeof(pfdev->csif_info);
+		src = &pfdev->csif_info;
+		break;
+	default:

drivers/gpu/drm/pancsf/pancsf_drv.c line 56

Quoted 10 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +10 -0
+		return -EINVAL;
+	}
+
+	if (!args->pointer) {
+		args->size = src_size;
+		return 0;
+	}
+
+	args->size = min_t(unsigned long, src_size, args->size);
+	if (copy_to_user((void __user *)(uintptr_t)args->pointer, src, args->size))

NIT: use u64_to_user_ptr()

Quoted 132 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +132 -0
+		return -EFAULT;
+
+	return 0;
+}
+
+#define PANCSF_MAX_VMS_PER_FILE		32
+#define PANCSF_VM_CREATE_FLAGS		0
+
+int pancsf_ioctl_vm_create(struct drm_device *ddev, void *data,
+			   struct drm_file *file)
+{
+	struct pancsf_device *pfdev = ddev->dev_private;
+	struct pancsf_file *pfile = file->driver_priv;
+	struct drm_pancsf_vm_create *args = data;
+	int ret;
+
+	if (args->flags & ~PANCSF_VM_CREATE_FLAGS)
+		return -EINVAL;
+
+	ret = pancsf_vm_pool_create_vm(pfdev, pfile->vms);
+	if (ret < 0)
+		return ret;
+
+	args->id = ret;
+	return 0;
+}
+
+int pancsf_ioctl_vm_destroy(struct drm_device *ddev, void *data,
+			    struct drm_file *file)
+{
+	struct pancsf_file *pfile = file->driver_priv;
+	struct drm_pancsf_vm_destroy *args = data;
+
+	pancsf_vm_pool_destroy_vm(pfile->vms, args->id);
+	return 0;
+}
+
+#define PANCSF_BO_FLAGS		0
+
+static int pancsf_ioctl_bo_create(struct drm_device *ddev, void *data,
+				  struct drm_file *file)
+{
+	struct pancsf_gem_object *bo;
+	struct drm_pancsf_bo_create *args = data;
+
+	if (!args->size || args->pad ||
+	    (args->flags & ~PANCSF_BO_FLAGS))
+		return -EINVAL;
+
+	bo = pancsf_gem_create_with_handle(file, ddev, args->size, args->flags,
+					   &args->handle);
+	if (IS_ERR(bo))
+		return PTR_ERR(bo);
+
+	return 0;
+}
+
+#define PANCSF_VMA_MAP_FLAGS (PANCSF_VMA_MAP_READONLY | \
+			      PANCSF_VMA_MAP_NOEXEC | \
+			      PANCSF_VMA_MAP_UNCACHED | \
+			      PANCSF_VMA_MAP_FRAG_SHADER | \
+			      PANCSF_VMA_MAP_ON_FAULT | \
+			      PANCSF_VMA_MAP_AUTO_VA)
+
+static int pancsf_ioctl_vm_map(struct drm_device *ddev, void *data,
+			       struct drm_file *file)
+{
+	struct pancsf_file *pfile = file->driver_priv;
+	struct drm_pancsf_vm_map *args = data;
+	struct drm_gem_object *gem;
+	struct pancsf_vm *vm;
+	int ret;
+
+	if (args->flags & ~PANCSF_VMA_MAP_FLAGS)
+		return -EINVAL;
+
+	gem = drm_gem_object_lookup(file, args->bo_handle);
+	if (!gem)
+		return -EINVAL;
+
+	vm = pancsf_vm_pool_get_vm(pfile->vms, args->vm_id);
+	if (vm) {
+		ret = pancsf_vm_map_bo_range(vm, to_pancsf_bo(gem), args->bo_offset,
+					     args->size, &args->va, args->flags);
+	} else {
+		ret = -EINVAL;
+	}
+
+	pancsf_vm_put(vm);
+	drm_gem_object_put(gem);
+	return ret;
+}
+
+#define PANCSF_VMA_UNMAP_FLAGS 0
+
+static int pancsf_ioctl_vm_unmap(struct drm_device *ddev, void *data,
+				 struct drm_file *file)
+{
+	struct pancsf_file *pfile = file->driver_priv;
+	struct drm_pancsf_vm_unmap *args = data;
+	struct pancsf_vm *vm;
+	int ret;
+
+	if (args->flags & ~PANCSF_VMA_UNMAP_FLAGS)
+		return -EINVAL;
+
+	vm = pancsf_vm_pool_get_vm(pfile->vms, args->vm_id);
+	if (vm)
+		ret = pancsf_vm_unmap_range(vm, args->va, args->size);
+	else
+		ret = -EINVAL;
+
+	pancsf_vm_put(vm);
+	return ret;
+}
+
+static void *pancsf_get_obj_array(struct drm_pancsf_obj_array *in, u32 min_stride)
+{
+	u32 elem_size = min_t(u32, in->stride, min_stride);
+	int ret = 0;
+	void *out;
+
+	if (in->stride < min_stride)
+		return ERR_PTR(-EINVAL);
+
+	out = kvmalloc_array(in->count, elem_size, GFP_KERNEL);
+	if (!out)
+		return ERR_PTR(-ENOMEM);
+
+	if (elem_size == in->stride) {
+		if (copy_from_user(out, u64_to_user_ptr(in->array), elem_size * in->count))
+			ret = -EFAULT;

drivers/gpu/drm/pancsf/pancsf_drv.c line 198

Quoted 10 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +10 -0
+	} else {
+		void __user *in_ptr = u64_to_user_ptr(in->array);
+		void *out_ptr = out;
+		u32 i;
+
+		for (i = 0; i < in->count; i++) {
+			if (copy_from_user(out_ptr, in_ptr, elem_size)) {
+				ret = -EFAULT;
+				break;
+			}

Missing the increment of out_ptr and in_ptr.

Quoted 105 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +105 -0
+		}
+	}
+
+	if (ret) {
+		kvfree(out);
+		return ERR_PTR(ret);
+	}
+
+	return out;
+}
+
+static int pancsf_add_job_deps(struct drm_file *file, struct pancsf_job *job,
+			       struct drm_pancsf_sync_op *sync_ops, u32 sync_op_count)
+{
+	u32 i;
+
+	for (i = 0; i < sync_op_count; i++) {
+		struct dma_fence *fence;
+		int ret;
+
+		if (sync_ops[i].op_type != DRM_PANCSF_SYNC_OP_WAIT)
+			continue;
+
+		switch (sync_ops[i].handle_type) {
+		case DRM_PANCSF_SYNC_HANDLE_TYPE_SYNCOBJ:
+		case DRM_PANCSF_SYNC_HANDLE_TYPE_TIMELINE_SYNCOBJ:
+			ret = drm_syncobj_find_fence(file, sync_ops[i].handle,
+						     sync_ops[i].timeline_value,
+						     0, &fence);
+			if (ret)
+				return ret;
+
+			ret = pancsf_add_job_dep(job, fence);
+			if (ret) {
+				dma_fence_put(fence);
+				return ret;
+			}
+			break;
+
+		default:
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+struct pancsf_sync_signal {
+	struct drm_syncobj *syncobj;
+	struct dma_fence_chain *chain;
+	u64 point;
+};
+
+struct pancsf_sync_signal_array {
+	struct pancsf_sync_signal *signals;
+	u32 count;
+};
+
+void
+pancsf_free_sync_signal_array(struct pancsf_sync_signal_array *array)
+{
+	u32 i;
+
+	for (i = 0; i < array->count; i++) {
+		drm_syncobj_put(array->signals[i].syncobj);
+		dma_fence_chain_free(array->signals[i].chain);
+	}
+
+	kvfree(array->signals);
+	array->signals = NULL;
+	array->count = 0;
+}
+
+int
+pancsf_collect_sync_signal_array(struct drm_file *file,
+				 struct drm_pancsf_sync_op *sync_ops, u32 sync_op_count,
+				 struct pancsf_sync_signal_array *array)
+{
+	u32 count = 0, i;
+	int ret;
+
+	for (i = 0; i < sync_op_count; i++) {
+		if (sync_ops[i].op_type == DRM_PANCSF_SYNC_OP_SIGNAL)
+			count++;
+	}
+
+	array->signals = kvmalloc_array(count, sizeof(*array->signals), GFP_KERNEL | __GFP_ZERO);
+	if (!array->signals)
+		return -ENOMEM;
+
+	for (i = 0; i < sync_op_count; i++) {
+		int ret;
+
+		if (sync_ops[i].op_type != DRM_PANCSF_SYNC_OP_SIGNAL)
+			continue;
+
+		switch (sync_ops[i].handle_type) {
+		case DRM_PANCSF_SYNC_HANDLE_TYPE_TIMELINE_SYNCOBJ:
+			array->signals[array->count].chain = dma_fence_chain_alloc();
+			if (!array->signals[array->count].chain) {
+				ret = -ENOMEM;
+				goto err;
+			}
+
+			array->signals[array->count].point = sync_ops[i].timeline_value;

drivers/gpu/drm/pancsf/pancsf_drv.c line 313

Quoted 10 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +10 -0
+			fallthrough;
+
+		case DRM_PANCSF_SYNC_HANDLE_TYPE_SYNCOBJ:
+			array->signals[array->count].syncobj = drm_syncobj_find(file, sync_ops[i].handle);
+			if (!array->signals[array->count].syncobj) {
+				ret = -EINVAL;
+				goto err;
+			}
+
+			array->count++;

NIT: it's not obvious in this function that array->count is set to 0 initially. It looks like it's correct because pancsf_ioctl_group_submit() does an alloc with __GFP_ZERO, but it would be much clearer with a local variable and assigning array->count at the end.

Quoted 4 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +4 -0
+			break;
+
+		default:
+			ret = -EINVAL;

drivers/gpu/drm/pancsf/pancsf_drv.c line 327

Quoted 10 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +10 -0
+			goto err;
+		}
+	}
+
+	return 0;
+
+err:
+	pancsf_free_sync_signal_array(array);
+	return ret;
+}

<gigantic snip - sorry, I phased out somewhat reading through all the code - hopefully when I get a platform and start playing with it properly I can give some meaningful feedback. Nothing jumped out as looking seriously wrong>

Quoted 119 lines
include/uapi/drm/pancsf_drm.h new file +113 -0
new file mode 100644
index 000000000000..d770d1d1037c
@@ -0,0 +1,414 @@
+/* SPDX-License-Identifier: MIT */
+/* Copyright (C) 2023 Collabora ltd. */
+#ifndef _PANCSF_DRM_H_
+#define _PANCSF_DRM_H_
+
+#include "drm.h"
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+/*
+ * Userspace driver controls GPU cache flushling through CS instructions, but
+ * the flush reduction mechanism requires a flush_id. This flush_id could be
+ * queried with an ioctl, but Arm provides a well-isolated register page
+ * containing only this read-only register, so let's expose this page through
+ * a static mmap offset and allow direct mapping of this MMIO region so we
+ * can avoid the user <-> kernel round-trip.
+ */
+#define DRM_PANCSF_USER_MMIO_OFFSET		(0xffffull << 48)
+#define DRM_PANCSF_USER_FLUSH_ID_MMIO_OFFSET	(DRM_PANCSF_USER_MMIO_OFFSET | 0)
+
+/* Place new ioctls at the end, don't re-oder. */
+enum drm_pancsf_ioctl_id {
+	DRM_PANCSF_DEV_QUERY = 0,
+	DRM_PANCSF_VM_CREATE,
+	DRM_PANCSF_VM_DESTROY,
+	DRM_PANCSF_BO_CREATE,
+	DRM_PANCSF_BO_MMAP_OFFSET,
+	DRM_PANCSF_VM_MAP,
+	DRM_PANCSF_VM_UNMAP,
+	DRM_PANCSF_GROUP_CREATE,
+	DRM_PANCSF_GROUP_DESTROY,
+	DRM_PANCSF_TILER_HEAP_CREATE,
+	DRM_PANCSF_TILER_HEAP_DESTROY,
+	DRM_PANCSF_GROUP_SUBMIT,
+};
+
+#define DRM_IOCTL_PANCSF_DEV_QUERY		DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_DEV_QUERY, struct drm_pancsf_dev_query)
+#define DRM_IOCTL_PANCSF_VM_CREATE		DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_VM_CREATE, struct drm_pancsf_vm_create)
+#define DRM_IOCTL_PANCSF_VM_DESTROY		DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_VM_DESTROY, struct drm_pancsf_vm_destroy)
+#define DRM_IOCTL_PANCSF_BO_CREATE		DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_BO_CREATE, struct drm_pancsf_bo_create)
+#define DRM_IOCTL_PANCSF_BO_MMAP_OFFSET		DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_BO_MMAP_OFFSET, struct drm_pancsf_bo_mmap_offset)
+#define DRM_IOCTL_PANCSF_VM_MAP			DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_VM_MAP, struct drm_pancsf_vm_map)
+#define DRM_IOCTL_PANCSF_VM_UNMAP		DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_VM_UNMAP, struct drm_pancsf_vm_unmap)
+#define DRM_IOCTL_PANCSF_GROUP_CREATE		DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_GROUP_CREATE, struct drm_pancsf_group_create)
+#define DRM_IOCTL_PANCSF_GROUP_DESTROY		DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_GROUP_DESTROY, struct drm_pancsf_group_destroy)
+#define DRM_IOCTL_PANCSF_TILER_HEAP_CREATE	DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_TILER_HEAP_CREATE, struct drm_pancsf_tiler_heap_create)
+#define DRM_IOCTL_PANCSF_TILER_HEAP_DESTROY	DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_TILER_HEAP_DESTROY, struct drm_pancsf_tiler_heap_destroy)
+#define DRM_IOCTL_PANCSF_GROUP_SUBMIT		DRM_IOWR(DRM_COMMAND_BASE + DRM_PANCSF_GROUP_SUBMIT, struct drm_pancsf_group_submit)
+
+/* Place new types at the end, don't re-oder. */
+enum drm_pancsf_dev_query_type {
+	DRM_PANCSF_DEV_QUERY_GPU_INFO = 0,
+	DRM_PANCSF_DEV_QUERY_CSIF_INFO,
+};
+
+struct drm_pancsf_gpu_info {
+#define DRM_PANCSF_ARCH_MAJOR(x)		((x) >> 28)
+#define DRM_PANCSF_ARCH_MINOR(x)		(((x) >> 24) & 0xf)
+#define DRM_PANCSF_ARCH_REV(x)			(((x) >> 20) & 0xf)
+#define DRM_PANCSF_PRODUCT_MAJOR(x)		(((x) >> 16) & 0xf)
+#define DRM_PANCSF_VERSION_MAJOR(x)		(((x) >> 12) & 0xf)
+#define DRM_PANCSF_VERSION_MINOR(x)		(((x) >> 4) & 0xff)
+#define DRM_PANCSF_VERSION_STATUS(x)		((x) & 0xf)
+	__u32 gpu_id;
+	__u32 gpu_rev;
+#define DRM_PANCSF_CSHW_MAJOR(x)		(((x) >> 26) & 0x3f)
+#define DRM_PANCSF_CSHW_MINOR(x)		(((x) >> 20) & 0x3f)
+#define DRM_PANCSF_CSHW_REV(x)			(((x) >> 16) & 0xf)
+#define DRM_PANCSF_MCU_MAJOR(x)			(((x) >> 10) & 0x3f)
+#define DRM_PANCSF_MCU_MINOR(x)			(((x) >> 4) & 0x3f)
+#define DRM_PANCSF_MCU_REV(x)			((x) & 0xf)
+	__u32 csf_id;
+	__u32 l2_features;
+	__u32 tiler_features;
+	__u32 mem_features;
+	__u32 mmu_features;
+	__u32 thread_features;
+	__u32 max_threads;
+	__u32 thread_max_workgroup_size;
+	__u32 thread_max_barrier_size;
+	__u32 coherency_features;
+	__u32 texture_features[4];
+	__u32 as_present;
+	__u32 core_group_count;
+	__u64 shader_present;
+	__u64 l2_present;
+	__u64 tiler_present;
+};
+
+struct drm_pancsf_csif_info {
+	__u32 csg_slot_count;
+	__u32 cs_slot_count;
+	__u32 cs_reg_count;
+	__u32 scoreboard_slot_count;
+	__u32 unpreserved_cs_reg_count;
+};
+
+struct drm_pancsf_dev_query {
+	/** @type: the query type (see enum drm_pancsf_dev_query_type). */
+	__u32 type;
+
+	/**
+	 * @size: size of the type being queried.
+	 *
+	 * If pointer is NULL, size is updated by the driver to provide the
+	 * output structure size. If pointer is not NULL, the the driver will
+	 * only copy min(size, actual_structure_size) bytes to the pointer,
+	 * and update the size accordingly. This allows us to extend query
+	 * types without breaking userspace.
+	 */
+	__u32 size;

include/uapi/drm/pancsf_drm.h line 123

Quoted 10 lines
include/uapi/drm/pancsf_drm.h new file +10 -0
+
+	/**
+	 * @pointer: user pointer to a query type struct.
+	 *
+	 * Pointer can be NULL, in which case, nothing is copied, but the
+	 * actual structure size is returned. If not NULL, it must point to
+	 * a location that's large enough to hold size bytes.
+	 */
+	__u64 pointer;
+};

Genuine question: is there something wrong with the panfrost 'get_param' ioctl where individual features are queried one-by-one, rather than passing a big structure back to user space.

I ask because we've had issues in the past with trying to 'deprecate' registers - if a new version of the hardware stops providing a (meaningful) value for a register then it's hard to fix up the structures. The get_param method means it's possible to return a failure for just the register(s) that have disappeared.

There is obviously overhead iterating over all the register that user space cares about. Another option (used by kbase) is to return some form of structured data so a missing property can be encoded.

Quoted 30 lines
include/uapi/drm/pancsf_drm.h new file +30 -0
+
+struct drm_pancsf_vm_create {
+	/** @flags: VM flags, MBZ. */
+	__u32 flags;
+
+	/** @id: Returned VM ID */
+	__u32 id;
+};
+
+struct drm_pancsf_vm_destroy {
+	/** @id: ID of the VM to destroy */
+	__u32 id;
+
+	/** @pad: MBZ. */
+	__u32 pad;
+};
+
+struct drm_pancsf_bo_create {
+	/**
+	 * @size: Requested size for the object
+	 *
+	 * The (page-aligned) allocated size for the object will be returned.
+	 */
+	__u64 size;
+
+	/**
+	 * @flags: Flags, currently unused, MBZ.
+	 */
+	__u32 flags;
+

include/uapi/drm/pancsf_drm.h line 163

Quoted 10 lines
include/uapi/drm/pancsf_drm.h new file +10 -0
+	/**
+	 * @vm_id: Attached VM, if any
+	 *
+	 * If a VM is specified, this BO must:
+	 *
+	 *  1. Only ever be bound to that VM.
+	 *
+	 *  2. Cannot be exported as a PRIME fd.
+	 */
+	__u32 vm_id;

Unless I'm mistaken this (vm_id) isn't used (or even checked) by the current code.

Quoted 8 lines
include/uapi/drm/pancsf_drm.h new file +8 -0
+
+	/**
+	 * @handle: Returned handle for the object.
+	 *
+	 * Object handles are nonzero.
+	 */
+	__u32 handle;
+

include/uapi/drm/pancsf_drm.h line 181

Quoted 10 lines
include/uapi/drm/pancsf_drm.h new file +10 -0
+	/* @pad: MBZ. */
+	__u32 pad;
+};
+
+struct drm_pancsf_bo_mmap_offset {
+	/** @handle: Handle for the object being mapped. */
+	__u32 handle;
+
+	/** @pad: MBZ. */
+	__u32 pad;

pancsf_ioctl_bo_mmap_offset is missing validation for this pad field.

Quoted 11 lines
include/uapi/drm/pancsf_drm.h new file +11 -0
+
+	/** @offset: The fake offset to use for subsequent mmap call */
+	__u64 offset;
+};
+
+#define PANCSF_VMA_MAP_READONLY		0x1
+#define PANCSF_VMA_MAP_NOEXEC		0x2
+#define PANCSF_VMA_MAP_UNCACHED		0x4
+#define PANCSF_VMA_MAP_FRAG_SHADER	0x8
+#define PANCSF_VMA_MAP_ON_FAULT		0x10
+#define PANCSF_VMA_MAP_AUTO_VA		0x20

include/uapi/drm/pancsf_drm.h line 202

Quoted 10 lines
include/uapi/drm/pancsf_drm.h new file +10 -0
+
+struct drm_pancsf_vm_map {
+	/** @vm_id: VM to map BO range to */
+	__u32 vm_id;
+
+	/** @flags: Combination of PANCSF_VMA_MAP_ flags */
+	__u32 flags;
+
+	/** @pad: padding field, MBZ. */
+	__u32 pad;

Missing validation in pancsf_ioctl_vm_map.

Quoted 13 lines
include/uapi/drm/pancsf_drm.h new file +13 -0
+
+	/** @bo_handle: Buffer object to map. */
+	__u32 bo_handle;
+
+	/** @bo_offset: Buffer object offset. */
+	__u64 bo_offset;
+
+	/**
+	 * @va: Virtual address to map the BO to. Mapping address returned here if
+	 *	PANCSF_VMA_MAP_ON_FAULT is set.
+	 */
+	__u64 va;
+

include/uapi/drm/pancsf_drm.h line 225

Quoted 10 lines
include/uapi/drm/pancsf_drm.h new file +10 -0
+	/** @size: Size to map. */
+	__u64 size;
+};
+
+struct drm_pancsf_vm_unmap {
+	/** @vm_id: VM to map BO range to */
+	__u32 vm_id;
+
+	/** @flags: MBZ. */
+	__u32 flags;

Validation missing

Quoted 20 lines
include/uapi/drm/pancsf_drm.h new file +20 -0
+
+	/** @va: Virtual address to unmap. */
+	__u64 va;
+
+	/** @size: Size to unmap. */
+	__u64 size;
+};
+
+enum drm_pancsf_sync_op_type {
+	DRM_PANCSF_SYNC_OP_WAIT = 0,
+	DRM_PANCSF_SYNC_OP_SIGNAL,
+};
+
+enum drm_pancsf_sync_handle_type {
+	DRM_PANCSF_SYNC_HANDLE_TYPE_SYNCOBJ = 0,
+	DRM_PANCSF_SYNC_HANDLE_TYPE_TIMELINE_SYNCOBJ,
+};
+
+struct drm_pancsf_sync_op {
+	/** @op_type: Sync operation type. */

include/uapi/drm/pancsf_drm.h line 255

Quoted 10 lines
include/uapi/drm/pancsf_drm.h new file +10 -0
+	__u32 op_type;
+
+	/** @handle_type: Sync handle type. */
+	__u32 handle_type;
+
+	/** @handle: Sync handle. */
+	__u32 handle;
+
+	/** @flags: MBZ. */
+	__u32 flags;

I don't see any proper validation that op_type/handle_type/flags are not invalid.

Quoted 42 lines
include/uapi/drm/pancsf_drm.h new file +42 -0
+
+	/** @timeline_value: MBZ if handle_type != DRM_PANCSF_SYNC_HANDLE_TYPE_TIMELINE_SYNCOBJ. */
+	__u64 timeline_value;
+};
+
+struct drm_pancsf_obj_array {
+	/** @stride: Stride of object struct. Used for versioning. */
+	__u32 stride;
+
+	/** @count: Number of objects in the array. */
+	__u32 count;
+
+	/** @array: User pointer to an array of objects. */
+	__u64 array;
+};
+
+#define DRM_PANCSF_OBJ_ARRAY(cnt, ptr) \
+	{ .stride = sizeof(ptr[0]), .count = cnt, .array = (__u64)(uintptr_t)ptr }
+
+struct drm_pancsf_queue_submit {
+	/** @queue_index: Index of the queue inside a group. */
+	__u32 queue_index;
+
+	/** @stream_size: Size of the command stream to execute. */
+	__u32 stream_size;
+
+	/** @stream_addr: GPU address of the command stream to execute. */
+	__u64 stream_addr;
+
+	/** @syncs: Array of sync operations. */
+	struct drm_pancsf_obj_array syncs;
+};
+
+struct drm_pancsf_group_submit {
+	/** @group_handle: Handle of the group to queue jobs to. */
+	__u32 group_handle;
+
+	/** @syncs: Array of queue submit operations. */
+	struct drm_pancsf_obj_array queue_submits;
+};
+
+struct drm_pancsf_queue_create {

include/uapi/drm/pancsf_drm.h line 307

Quoted 10 lines
include/uapi/drm/pancsf_drm.h new file +10 -0
+	/**
+	 * @priority: Defines the priority of queues inside a group. Goes from 0 to 15,
+	 *	      15 being the highest priority.
+	 */
+	__u8 priority;
+
+	/** @pad: Padding fields, MBZ. */
+	__u8 pad[3];
+
+	/** @ringbuf_size: Size of the ring buffer to allocate to this queue. */

(Must be PAGE_SIZE aligned and at most 64k)

include/uapi/drm/pancsf_drm.h line 318

Quoted 11 lines
include/uapi/drm/pancsf_drm.h new file +11 -0
+	__u32 ringbuf_size;
+};
+
+enum drm_pancsf_group_priority {
+	PANCSF_GROUP_PRIORITY_LOW = 0,
+	PANCSF_GROUP_PRIORITY_MEDIUM,
+	PANCSF_GROUP_PRIORITY_HIGH,
+};
+
+struct drm_pancsf_group_create {
+	/** @queues: Array of drm_pancsf_create_cs_queue elements. */

NIT: s/drm_pancsf_create_cs_queue/drm_pancsf_queue_create/

Quoted 44 lines
include/uapi/drm/pancsf_drm.h new file +44 -0
+	struct drm_pancsf_obj_array queues;
+
+	/**
+	 * @max_compute_cores: Maximum number of cores that can be
+	 *		       used by compute jobs across CS queues
+	 *		       bound to this group.
+	 */
+	__u8 max_compute_cores;
+
+	/**
+	 * @max_fragment_cores: Maximum number of cores that can be
+	 *			used by fragment jobs across CS queues
+	 *			bound to this group.
+	 */
+	__u8 max_fragment_cores;
+
+	/**
+	 * @max_tiler_cores: Maximum number of tilers that can be
+	 *		     used by tiler jobs across CS queues
+	 *		     bound to this group.
+	 */
+	__u8 max_tiler_cores;
+
+	/** @priority: Group priority (see drm_drm_pancsf_cs_group_priority). */
+	__u8 priority;
+
+	/** @compute_core_mask: Mask encoding cores that can be used for compute jobs. */
+	__u64 compute_core_mask;
+
+	/** @fragment_core_mask: Mask encoding cores that can be used for fragment jobs. */
+	__u64 fragment_core_mask;
+
+	/** @tiler_core_mask: Mask encoding cores that can be used for tiler jobs. */
+	__u64 tiler_core_mask;
+
+	/**
+	 * @vm_id: VM ID to bind this group to. All submission to queues bound to this
+	 *	   group will use this VM.
+	 */
+	__u32 vm_id;
+
+	/*
+	 * @group_handle: Returned group handle. Passed back when submitting jobs or
+	 *		  destroying a group.

include/uapi/drm/pancsf_drm.h line 372

Quoted 10 lines
include/uapi/drm/pancsf_drm.h new file +10 -0
+	 */
+	__u32 group_handle;
+};
+
+struct drm_pancsf_group_destroy {
+	/** @group_handle: Group to destroy */
+	__u32 group_handle;
+
+	/** @pad: Padding field, MBZ. */
+	__u32 pad;

No validation for padding

Quoted 25 lines
include/uapi/drm/pancsf_drm.h new file +25 -0
+};
+
+struct drm_pancsf_tiler_heap_create {
+	/** @vm_id: VM ID the tiler heap should be mapped to */
+	__u32 vm_id;
+
+	/** @initial_chunk_count: Initial number of chunks to allocate. */
+	__u32 initial_chunk_count;
+
+	/** @chunk_size: Chunk size. Must be a power of two at least 256KB large. */
+	__u32 chunk_size;
+
+	/* @max_chunks: Maximum number of chunks that can be allocated. */
+	__u32 max_chunks;
+
+	/** @target_in_flight: Maximum number of in-flight render passes.
+	 * If exceeded the FW will wait for render passes to finish before
+	 * queuing new tiler jobs.
+	 */
+	__u32 target_in_flight;
+
+	/** @handle: Returned heap handle. Passed back to DESTROY_TILER_HEAP. */
+	__u32 handle;
+
+	/** @tiler_heap_ctx_gpu_va: Returned heap GPU virtual address returned */

include/uapi/drm/pancsf_drm.h line 407

Quoted 10 lines
include/uapi/drm/pancsf_drm.h new file +10 -0
+	__u64 tiler_heap_ctx_gpu_va;
+	__u64 first_heap_chunk_gpu_va;
+};
+
+struct drm_pancsf_tiler_heap_destroy {
+	/** @handle: Handle of the tiler heap to destroy */
+	__u32 handle;
+
+	/** @pad: Padding field, MBZ. */
+	__u32 pad;

Validation

include/uapi/drm/pancsf_drm.h line 414

Quoted 7 lines
include/uapi/drm/pancsf_drm.h new file +7 -0
+};
+
+#if defined(__cplusplus)
+}
+#endif
+
+#endif /* _PANCSF_DRM_H_ */

Thanks once again for this. I hope to be able to give it a spin soon!

Steve

#4 lore raw
Quoted 24 lines
Quoted 10 lines

Mali v10 (second Valhal iteration) and later GPUs replaced the Job Manager block by a command stream based interface called CSF (for Command Stream Frontend). This interface is not only turning the job chain based submission model into a command stream based one, but also introducing FW-assisted scheduling of command stream queues. This is a fundamental shift in both how userspace is supposed to submit jobs, but also how the driver is architectured. We initially tried to retrofit the CSF model into panfrost, but this ended up introducing unneeded complexity to the existing driver, which we all know is a potential source of regression.

While I agree there's some big differences which effectively mandate splitting the driver I do think there are some parts which make a lot of sense to share.

For example pancsf_regs.h and panfrost_regs.h are really quite similar and I think could easily be combined. The clock/regulator code is pretty much a direct copy/paste (just adding support for more clocks), etc.

What would be ideal is factoring out 'generic' parts from panfrost and then being able to use them from pancsf.

I had a go at starting that:

Quoted 8 lines

https://gitlab.arm.com/linux-arm/linux-sp/-/tree/pancsf-refactor

(lightly tested for Panfrost, only build tested for pancsf).

That saves around 200 lines overall and avoids needing to maintain two lots of clock/regulator code. There's definite scope for sharing (most) register definitions between the drivers and quite possibly some of the MMU/memory code (although there's diminishing returns there).

200 lines saved in a 5kloc+ driver doesn't seem worth much, especially against the added testing combinatorics, TBH. The main reason I can see to unify is if we want VM_BIND (and related goodies) on JM hardware too. That's only really for Vulkan and I really don't see the case for Vulkan on anything older than Valhall at this point. So it comes down to whether we want to start Vulkan at v9 or skip to v10. The separate panfrost/pancsf drivers approach strongly favours the latter.

#5 lore raw
Quoted 33 lines · Alyssa Rosenzweig
Quoted 24 lines
Quoted 10 lines

Mali v10 (second Valhal iteration) and later GPUs replaced the Job Manager block by a command stream based interface called CSF (for Command Stream Frontend). This interface is not only turning the job chain based submission model into a command stream based one, but also introducing FW-assisted scheduling of command stream queues. This is a fundamental shift in both how userspace is supposed to submit jobs, but also how the driver is architectured. We initially tried to retrofit the CSF model into panfrost, but this ended up introducing unneeded complexity to the existing driver, which we all know is a potential source of regression.

While I agree there's some big differences which effectively mandate splitting the driver I do think there are some parts which make a lot of sense to share.

For example pancsf_regs.h and panfrost_regs.h are really quite similar and I think could easily be combined. The clock/regulator code is pretty much a direct copy/paste (just adding support for more clocks), etc.

What would be ideal is factoring out 'generic' parts from panfrost and then being able to use them from pancsf.

I had a go at starting that:

Quoted 8 lines

https://gitlab.arm.com/linux-arm/linux-sp/-/tree/pancsf-refactor

(lightly tested for Panfrost, only build tested for pancsf).

That saves around 200 lines overall and avoids needing to maintain two lots of clock/regulator code. There's definite scope for sharing (most) register definitions between the drivers and quite possibly some of the MMU/memory code (although there's diminishing returns there).

Quoted 7 lines

200 lines saved in a 5kloc+ driver doesn't seem worth much, especially against the added testing combinatorics, TBH. The main reason I can see to unify is if we want VM_BIND (and related goodies) on JM hardware too. That's only really for Vulkan and I really don't see the case for Vulkan on anything older than Valhall at this point. So it comes down to whether we want to start Vulkan at v9 or skip to v10. The separate panfrost/pancsf drivers approach strongly favours the latter.

While I agree 200 lines isn't much in the grand scheme what I really don't want is to have to maintain two (almost) identical versions of the same code. I agree with the concept entirely of having a separate .ko and not trying to keep it all "one driver". Just, for the bits where it's clearly copy/pasted from the existing Panfrost, lets move that code into a common file and build it into both drivers.

Ultimately the 200 line saving was just a couple of hours this morning - indeed I was 'reviewing' by comparing against Panfrost and thinking "if it works in Panfrost it must be correct" - the review would be even easier if it wasn't new code ;)

And as far as I'm aware the changes I'm proposing don't make any difference to testing - I'm not sure I understand that statement.

The MMU/memory code I'm undecided on. There's clearly copied code there but quite a few differences. If we can unify and get extra goodies for Panfrost then it's worth doing, if the unification is going to be hard or risk regressions then perhaps not - especially if Mesa isn't going to get the features (which depends whether anyone working on Mesa wants to work on Vulkan for Bifrost).

Anyway, just to be clear I don't want to stand in the way of getting this merged. If necessary the refactor can be done on top afterwards (indeed that's what I've got in my repo).

Thanks,

Steve

#6 lore raw
In reply to Steven Price

Hi Steven,

Quoted 3 lines · On Fri, 3 Feb 2023 15:41:38 +0000 Steven Price <[email protected]> wrote:

Hi Boris,

Thanks for the post - it's great to see the progress!

Thanks for chiming in!

Quoted 13 lines · Boris Brezillon
Quoted 10 lines

Mali v10 (second Valhal iteration) and later GPUs replaced the Job Manager block by a command stream based interface called CSF (for Command Stream Frontend). This interface is not only turning the job chain based submission model into a command stream based one, but also introducing FW-assisted scheduling of command stream queues. This is a fundamental shift in both how userspace is supposed to submit jobs, but also how the driver is architectured. We initially tried to retrofit the CSF model into panfrost, but this ended up introducing unneeded complexity to the existing driver, which we all know is a potential source of regression.

Quoted 6 lines

While I agree there's some big differences which effectively mandate splitting the driver I do think there are some parts which make a lot of sense to share.

For example pancsf_regs.h and panfrost_regs.h are really quite similar and I think could easily be combined.

For registers, I'm not so sure. I mean, yes, most of them are identical, but some disappeared, while others were kept but with a different layout (see GPU_CMD), or bits re-purposed for different meaning (MMU_INT_STAT where BUS_FAULT just became OPERATION_COMPLETE). This makes the whole thing very confusing, so I'd rather keep those definitions separate for my own sanity (spent a bit of time trying to understand why my GPU command was doing nothing or why I was receiving BUS_FAULT interrupts, before realizing I was referring to the old layout) :-).

Quoted 2 lines

The clock/regulator code is pretty
much a direct copy/paste (just adding support for more clocks), etc.

Clock and regulators, maybe, but there's not much to be shared here. I mean, Linux already provides quite a few helpers making the clk/regulator retrieval/enabling/disabling pretty straightforward. Not to mention that, by keeping them separate, we don't really need to deal with old Mali HW quirks, and we can focus on new HW bugs instead :-).

Quoted 3 lines

What would be ideal is factoring out 'generic' parts from panfrost and then being able to use them from pancsf.

I've been refactoring this pancsf driver a few times already, and I must admit I'd prefer to keep things separate, at least for the initial submission. If we see things that should be shared, then we can do that in a follow-up series, but I think it's a bit premature to do it now.

Quoted 6 lines

I had a go at starting that:

https://gitlab.arm.com/linux-arm/linux-sp/-/tree/pancsf-refactor

(lightly tested for Panfrost, only build tested for pancsf).

Thanks, I'll have a look.

Quoted 5 lines

That saves around 200 lines overall and avoids needing to maintain two lots of clock/regulator code. There's definite scope for sharing (most) register definitions between the drivers and quite possibly some of the MMU/memory code (although there's diminishing returns there).

Yeah, actually the MMU code is likely to diverge even more if we want to support VM_BIND (require pre-allocating pages for the page-table updates, so the map/unmap operations can't fail in the run_job path), so I'm not sure it's a good idea to share that bit, at least not until we have a clearer idea of how we want things done.

Quoted 10 lines
Quoted 7 lines

So here comes a brand new driver for CSF-based hardware. This is a preliminary version and some important features are missing (like devfreq , PM support and a memory shrinker implem, to name a few). The goal of this RFC is to gather some preliminary feedback on both the uAPI and some basic building blocks, like the MMU/VM code, the tiler heap allocation logic...

At the moment I don't have any CSF hardware available, so this review is a pure code review.

That's still very useful.

Quoted 2 lines

I'll try to organise some hardware and do some
testing, but it's probably going to take a while to arrive and get setup.

I'm actively working on the driver, and fixing things as I go, so let me know when you're ready to test and I'll point you to the latest version.

Quoted 6 lines

drivers/gpu/drm/pancsf/pancsf_drv.c line 28

Quoted 4 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +4 -0
+#define DRM_PANCSF_SYNC_OP_MIN_SIZE		24
+#define DRM_PANCSF_QUEUE_SUBMIT_MIN_SIZE	32
+#define DRM_PANCSF_QUEUE_CREATE_MIN_SIZE	8
+#define DRM_PANCSF_VM_BIND_OP_MIN_SIZE		48  

I'm not sure why these are #defines rather than using sizeof()?

Yeah, I don't really like that. Those min sizes are here to deal with potential new versions of the various objects passed as arrays referenced from the main ioctl struct, so we can't really use sizeof(struct drm_pancsf_xxx), because the size will change when those structs are extended.

I'm considering using some macro trickery based on offsetof(struct drm_pancsf_xxx, last_mandatory_field) + sizeof(drm_pancsf_xxx.last_mandatory_field) to get
the minimum struct size instead of those static defines. On the other hand, this should never change, so it's only a pain point while the initial struct are under discussion.

Quoted 22 lines
Quoted 22 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +21 -0
+static int pancsf_ioctl_dev_query(struct drm_device *ddev, void *data, struct drm_file *file)
+{
+	struct drm_pancsf_dev_query *args = data;
+	struct pancsf_device *pfdev = ddev->dev_private;
+	const void *src;
+	size_t src_size;
+
+	switch (args->type) {
+	case DRM_PANCSF_DEV_QUERY_GPU_INFO:
+		src_size = sizeof(pfdev->gpu_info);
+		src = &pfdev->gpu_info;
+		break;
+	case DRM_PANCSF_DEV_QUERY_CSIF_INFO:
+		src_size = sizeof(pfdev->csif_info);
+		src = &pfdev->csif_info;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	if (!args->pointer) {
Quoted 8 lines

drivers/gpu/drm/pancsf/pancsf_drv.c line 56

Quoted 6 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +6 -0
+		args->size = src_size;
+		return 0;
+	}
+
+	args->size = min_t(unsigned long, src_size, args->size);
+	if (copy_to_user((void __user *)(uintptr_t)args->pointer, src, args->size))  

NIT: use u64_to_user_ptr()

Sure, will fix that.

Quoted 27 lines
Quoted 27 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +25 -0
+		return -EFAULT;
+
+	return 0;
+}
+
 
+static void *pancsf_get_obj_array(struct drm_pancsf_obj_array *in, u32 min_stride)
+{
+	u32 elem_size = min_t(u32, in->stride, min_stride);
+	int ret = 0;
+	void *out;
+
+	if (in->stride < min_stride)
+		return ERR_PTR(-EINVAL);
+
+	out = kvmalloc_array(in->count, elem_size, GFP_KERNEL);
+	if (!out)
+		return ERR_PTR(-ENOMEM);
+
+	if (elem_size == in->stride) {
+		if (copy_from_user(out, u64_to_user_ptr(in->array), elem_size * in->count))
+			ret = -EFAULT;
+	} else {
+		void __user *in_ptr = u64_to_user_ptr(in->array);
+		void *out_ptr = out;
+		u32 i;
Quoted 8 lines

drivers/gpu/drm/pancsf/pancsf_drv.c line 198

Quoted 6 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +6 -0
+
+		for (i = 0; i < in->count; i++) {
+			if (copy_from_user(out_ptr, in_ptr, elem_size)) {
+				ret = -EFAULT;
+				break;
+			}  

Missing the increment of out_ptr and in_ptr.

Actually, it's just one of the problem here. There's a confusion between min_stride and actual object size, which I didn't notice because they match right now, but as soon as we extend the base structs it will break. The following version should fix that:

static void *pancsf_get_obj_array(struct drm_pancsf_obj_array *in, u32 min_stride, u32 obj_size)

{
        u32 cpy_elem_size = min_t(u32, in->stride, obj_size);
        int ret = 0;
        void *out;

        if (in->stride < min_stride)
                return ERR_PTR(-EINVAL);

        out = kvmalloc_array(in->count, obj_size, GFP_KERNEL | __GFP_ZERO);
        if (!out)
                return ERR_PTR(-ENOMEM);

        if (obj_size == in->stride) {
                if (copy_from_user(out, u64_to_user_ptr(in->array), obj_size * in->count))
                        ret = -EFAULT;
        } else {
                void __user *in_ptr = u64_to_user_ptr(in->array);
                void *out_ptr = out;
                u32 i;

                for (i = 0; i < in->count; i++) {
                        if (copy_from_user(out_ptr, in_ptr, cpy_elem_size)) {
                                ret = -EFAULT;
                                break;
                        }

                        out_ptr += obj_size;
                        in_ptr += in->stride;
                }
        }

        if (ret) {
                kvfree(out);
                return ERR_PTR(ret);
        }

        return out;
}
Quoted 53 lines
Quoted 53 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +49 -0
+		}
+	}
+
+	if (ret) {
+		kvfree(out);
+		return ERR_PTR(ret);
+	}
+
+	return out;
+}
 
 
 
+int
+pancsf_collect_sync_signal_array(struct drm_file *file,
+				 struct drm_pancsf_sync_op *sync_ops, u32 sync_op_count,
+				 struct pancsf_sync_signal_array *array)
+{
+	u32 count = 0, i;
+	int ret;
+
+	for (i = 0; i < sync_op_count; i++) {
+		if (sync_ops[i].op_type == DRM_PANCSF_SYNC_OP_SIGNAL)
+			count++;
+	}
+
+	array->signals = kvmalloc_array(count, sizeof(*array->signals), GFP_KERNEL | __GFP_ZERO);
+	if (!array->signals)
+		return -ENOMEM;
+
+	for (i = 0; i < sync_op_count; i++) {
+		int ret;
+
+		if (sync_ops[i].op_type != DRM_PANCSF_SYNC_OP_SIGNAL)
+			continue;
+
+		switch (sync_ops[i].handle_type) {
+		case DRM_PANCSF_SYNC_HANDLE_TYPE_TIMELINE_SYNCOBJ:
+			array->signals[array->count].chain = dma_fence_chain_alloc();
+			if (!array->signals[array->count].chain) {
+				ret = -ENOMEM;
+				goto err;
+			}
+
+			array->signals[array->count].point = sync_ops[i].timeline_value;
+			fallthrough;
+
+		case DRM_PANCSF_SYNC_HANDLE_TYPE_SYNCOBJ:
+			array->signals[array->count].syncobj = drm_syncobj_find(file, sync_ops[i].handle);
+			if (!array->signals[array->count].syncobj) {
+				ret = -EINVAL;
+				goto err;
Quoted 8 lines

drivers/gpu/drm/pancsf/pancsf_drv.c line 313

Quoted 3 lines
drivers/gpu/drm/pancsf/pancsf_drv.c new file +3 -0
+			}
+
+			array->count++;  

NIT: it's not obvious in this function that array->count is set to 0 initially. It looks like it's correct because pancsf_ioctl_group_submit() does an alloc with __GFP_ZERO, but it would be much clearer with a local variable and assigning array->count at the end.

Definitely.

Quoted 79 lines
Quoted 79 lines
include/uapi/drm/pancsf_drm.h new file +77 -0
+			break;
+
+		default:
+			ret = -EINVAL;
+			goto err;
+		}
+	}
+
+	return 0;
+
+err:
+	pancsf_free_sync_signal_array(array);
+	return ret;
+}  
 
+
+struct drm_pancsf_gpu_info {
+#define DRM_PANCSF_ARCH_MAJOR(x)		((x) >> 28)
+#define DRM_PANCSF_ARCH_MINOR(x)		(((x) >> 24) & 0xf)
+#define DRM_PANCSF_ARCH_REV(x)			(((x) >> 20) & 0xf)
+#define DRM_PANCSF_PRODUCT_MAJOR(x)		(((x) >> 16) & 0xf)
+#define DRM_PANCSF_VERSION_MAJOR(x)		(((x) >> 12) & 0xf)
+#define DRM_PANCSF_VERSION_MINOR(x)		(((x) >> 4) & 0xff)
+#define DRM_PANCSF_VERSION_STATUS(x)		((x) & 0xf)
+	__u32 gpu_id;
+	__u32 gpu_rev;
+#define DRM_PANCSF_CSHW_MAJOR(x)		(((x) >> 26) & 0x3f)
+#define DRM_PANCSF_CSHW_MINOR(x)		(((x) >> 20) & 0x3f)
+#define DRM_PANCSF_CSHW_REV(x)			(((x) >> 16) & 0xf)
+#define DRM_PANCSF_MCU_MAJOR(x)			(((x) >> 10) & 0x3f)
+#define DRM_PANCSF_MCU_MINOR(x)			(((x) >> 4) & 0x3f)
+#define DRM_PANCSF_MCU_REV(x)			((x) & 0xf)
+	__u32 csf_id;
+	__u32 l2_features;
+	__u32 tiler_features;
+	__u32 mem_features;
+	__u32 mmu_features;
+	__u32 thread_features;
+	__u32 max_threads;
+	__u32 thread_max_workgroup_size;
+	__u32 thread_max_barrier_size;
+	__u32 coherency_features;
+	__u32 texture_features[4];
+	__u32 as_present;
+	__u32 core_group_count;
+	__u64 shader_present;
+	__u64 l2_present;
+	__u64 tiler_present;
+};
+
+struct drm_pancsf_csif_info {
+	__u32 csg_slot_count;
+	__u32 cs_slot_count;
+	__u32 cs_reg_count;
+	__u32 scoreboard_slot_count;
+	__u32 unpreserved_cs_reg_count;
+};
+
+struct drm_pancsf_dev_query {
+	/** @type: the query type (see enum drm_pancsf_dev_query_type). */
+	__u32 type;
+
+	/**
+	 * @size: size of the type being queried.
+	 *
+	 * If pointer is NULL, size is updated by the driver to provide the
+	 * output structure size. If pointer is not NULL, the the driver will
+	 * only copy min(size, actual_structure_size) bytes to the pointer,
+	 * and update the size accordingly. This allows us to extend query
+	 * types without breaking userspace.
+	 */
+	__u32 size;
+
+	/**
+	 * @pointer: user pointer to a query type struct.
+	 *
+	 * Pointer can be NULL, in which case, nothing is copied, but the
+	 * actual structure size is returned. If not NULL, it must point to
Quoted 8 lines

include/uapi/drm/pancsf_drm.h line 123

Quoted 4 lines
include/uapi/drm/pancsf_drm.h new file +4 -0
+	 * a location that's large enough to hold size bytes.
+	 */
+	__u64 pointer;
+};  

Genuine question: is there something wrong with the panfrost 'get_param' ioctl where individual features are queried one-by-one, rather than passing a big structure back to user space.

Well, I've just seen the Xe driver exposing things this way, and I thought it was a good idea, but I don't have a strong opinion here, and if others think it's preferable to stick to GET_PARAM, I'm fine with that too.

Quoted 6 lines

I ask because we've had issues in the past with trying to 'deprecate' registers - if a new version of the hardware stops providing a (meaningful) value for a register then it's hard to fix up the structures. The get_param method means it's possible to return a failure for just the register(s) that have disappeared.

There's still the option to fork the struct and add a new type if things differ too much, or just extend the existing info struct, with new flags to invalidate previous fields and new fields for new stuff.

Quoted 4 lines

There is obviously overhead iterating over all the register that user space cares about. Another option (used by kbase) is to return some form of structured data so a missing property can be encoded.

I'll have a look at how kbase does that. Thanks for the pointer.

Quoted 18 lines
Quoted 18 lines
include/uapi/drm/pancsf_drm.h new file +18 -0
+struct drm_pancsf_bo_create {
+	/**
+	 * @size: Requested size for the object
+	 *
+	 * The (page-aligned) allocated size for the object will be returned.
+	 */
+	__u64 size;
+
+	/**
+	 * @flags: Flags, currently unused, MBZ.
+	 */
+	__u32 flags;
+
+	/**
+	 * @vm_id: Attached VM, if any
+	 *
+	 * If a VM is specified, this BO must:
+	 *
Quoted 8 lines

include/uapi/drm/pancsf_drm.h line 163

Quoted 5 lines
include/uapi/drm/pancsf_drm.h new file +5 -0
+	 *  1. Only ever be bound to that VM.
+	 *
+	 *  2. Cannot be exported as a PRIME fd.
+	 */
+	__u32 vm_id;  

Unless I'm mistaken this (vm_id) isn't used (or even checked) by the current code.

Yeah, it will be used once we hook up the VM resv object and make all private BOs point to this resv (instead of their own). Didn't really see a need for that yet, since we don't have a shrinker reclaiming BOs behind our back, and the new uAPI is all about letting the userspace driver deal with object lifetime and passing explicit syncs. I might have missed a use case where this is required though...

Quoted 13 lines
Quoted 13 lines
include/uapi/drm/pancsf_drm.h new file +12 -0
+
+	/**
+	 * @handle: Returned handle for the object.
+	 *
+	 * Object handles are nonzero.
+	 */
+	__u32 handle;
+
+	/* @pad: MBZ. */
+	__u32 pad;
+};
+
Quoted 8 lines

include/uapi/drm/pancsf_drm.h line 181

Quoted 6 lines
include/uapi/drm/pancsf_drm.h new file +6 -0
+struct drm_pancsf_bo_mmap_offset {
+	/** @handle: Handle for the object being mapped. */
+	__u32 handle;
+
+	/** @pad: MBZ. */
+	__u32 pad;  

pancsf_ioctl_bo_mmap_offset is missing validation for this pad field.

I suspect most of them do, because I've added them progressively along with other fields to keep things 64-bit aligned. I'll do a pass and add missing checks before submitting v2.

Quoted 46 lines
Quoted 46 lines
include/uapi/drm/pancsf_drm.h new file +46 -0
+
+	/** @timeline_value: MBZ if handle_type != DRM_PANCSF_SYNC_HANDLE_TYPE_TIMELINE_SYNCOBJ. */
+	__u64 timeline_value;
+};
+
+struct drm_pancsf_obj_array {
+	/** @stride: Stride of object struct. Used for versioning. */
+	__u32 stride;
+
+	/** @count: Number of objects in the array. */
+	__u32 count;
+
+	/** @array: User pointer to an array of objects. */
+	__u64 array;
+};
+
+#define DRM_PANCSF_OBJ_ARRAY(cnt, ptr) \
+	{ .stride = sizeof(ptr[0]), .count = cnt, .array = (__u64)(uintptr_t)ptr }
+
+struct drm_pancsf_queue_submit {
+	/** @queue_index: Index of the queue inside a group. */
+	__u32 queue_index;
+
+	/** @stream_size: Size of the command stream to execute. */
+	__u32 stream_size;
+
+	/** @stream_addr: GPU address of the command stream to execute. */
+	__u64 stream_addr;
+
+	/** @syncs: Array of sync operations. */
+	struct drm_pancsf_obj_array syncs;
+};
+
+struct drm_pancsf_group_submit {
+	/** @group_handle: Handle of the group to queue jobs to. */
+	__u32 group_handle;
+
+	/** @syncs: Array of queue submit operations. */
+	struct drm_pancsf_obj_array queue_submits;
+};
+
+struct drm_pancsf_queue_create {
+	/**
+	 * @priority: Defines the priority of queues inside a group. Goes from 0 to 15,
+	 *	      15 being the highest priority.
+	 */
Quoted 8 lines

include/uapi/drm/pancsf_drm.h line 307

Quoted 6 lines
include/uapi/drm/pancsf_drm.h new file +6 -0
+	__u8 priority;
+
+	/** @pad: Padding fields, MBZ. */
+	__u8 pad[3];
+
+	/** @ringbuf_size: Size of the ring buffer to allocate to this queue. */  

(Must be PAGE_SIZE aligned and at most 64k)

I'll add it to the doc. Thanks

Quoted 9 lines

include/uapi/drm/pancsf_drm.h line 414

Quoted 6 lines
include/uapi/drm/pancsf_drm.h new file +6 -0
+
+#if defined(__cplusplus)
+}
+#endif
+
+#endif /* _PANCSF_DRM_H_ */  

Thanks once again for this. I hope to be able to give it a spin soon!

And thanks for your preliminary review. That's really appreciated.

Boris

#7 lore raw
Quoted 71 lines
Quoted 62 lines
Quoted 62 lines
include/uapi/drm/pancsf_drm.h new file +62 -0
+struct drm_pancsf_gpu_info {
+#define DRM_PANCSF_ARCH_MAJOR(x)		((x) >> 28)
+#define DRM_PANCSF_ARCH_MINOR(x)		(((x) >> 24) & 0xf)
+#define DRM_PANCSF_ARCH_REV(x)			(((x) >> 20) & 0xf)
+#define DRM_PANCSF_PRODUCT_MAJOR(x)		(((x) >> 16) & 0xf)
+#define DRM_PANCSF_VERSION_MAJOR(x)		(((x) >> 12) & 0xf)
+#define DRM_PANCSF_VERSION_MINOR(x)		(((x) >> 4) & 0xff)
+#define DRM_PANCSF_VERSION_STATUS(x)		((x) & 0xf)
+	__u32 gpu_id;
+	__u32 gpu_rev;
+#define DRM_PANCSF_CSHW_MAJOR(x)		(((x) >> 26) & 0x3f)
+#define DRM_PANCSF_CSHW_MINOR(x)		(((x) >> 20) & 0x3f)
+#define DRM_PANCSF_CSHW_REV(x)			(((x) >> 16) & 0xf)
+#define DRM_PANCSF_MCU_MAJOR(x)			(((x) >> 10) & 0x3f)
+#define DRM_PANCSF_MCU_MINOR(x)			(((x) >> 4) & 0x3f)
+#define DRM_PANCSF_MCU_REV(x)			((x) & 0xf)
+	__u32 csf_id;
+	__u32 l2_features;
+	__u32 tiler_features;
+	__u32 mem_features;
+	__u32 mmu_features;
+	__u32 thread_features;
+	__u32 max_threads;
+	__u32 thread_max_workgroup_size;
+	__u32 thread_max_barrier_size;
+	__u32 coherency_features;
+	__u32 texture_features[4];
+	__u32 as_present;
+	__u32 core_group_count;
+	__u64 shader_present;
+	__u64 l2_present;
+	__u64 tiler_present;
+};
+
+struct drm_pancsf_csif_info {
+	__u32 csg_slot_count;
+	__u32 cs_slot_count;
+	__u32 cs_reg_count;
+	__u32 scoreboard_slot_count;
+	__u32 unpreserved_cs_reg_count;
+};
+
+struct drm_pancsf_dev_query {
+	/** @type: the query type (see enum drm_pancsf_dev_query_type). */
+	__u32 type;
+
+	/**
+	 * @size: size of the type being queried.
+	 *
+	 * If pointer is NULL, size is updated by the driver to provide the
+	 * output structure size. If pointer is not NULL, the the driver will
+	 * only copy min(size, actual_structure_size) bytes to the pointer,
+	 * and update the size accordingly. This allows us to extend query
+	 * types without breaking userspace.
+	 */
+	__u32 size;
+
+	/**
+	 * @pointer: user pointer to a query type struct.
+	 *
+	 * Pointer can be NULL, in which case, nothing is copied, but the
+	 * actual structure size is returned. If not NULL, it must point to
Quoted 8 lines

include/uapi/drm/pancsf_drm.h line 123

Quoted 4 lines
include/uapi/drm/pancsf_drm.h new file +4 -0
+	 * a location that's large enough to hold size bytes.
+	 */
+	__u64 pointer;
+};  

Genuine question: is there something wrong with the panfrost 'get_param' ioctl where individual features are queried one-by-one, rather than passing a big structure back to user space.

Quoted 3 lines

Well, I've just seen the Xe driver exposing things this way, and I thought it was a good idea, but I don't have a strong opinion here, and if others think it's preferable to stick to GET_PARAM, I'm fine with that too.

I vastly prefer the info struct, GET_PARAM isn't a great interface when there are large numbers of properties to query... Actually I just suggested to Lina that she adopt this approach for Asahi instead of the current GET_PARAM ioctl we have (downstream for now).

It isn't a *big* deal but GET_PARAM doesn't really seem better on any axes.

Quoted 4 lines
Quoted 4 lines

I ask because we've had issues in the past with trying to 'deprecate' registers - if a new version of the hardware stops providing a (meaningful) value for a register then it's hard to fix up the structures.

I'm not sure this is a big deal. If the register no longer exists (meaningfully), zero it out in the info structure and trust userspace to interpret meaningfully based on the GPU. If registers are getting dropped between revisions, that's obviously not great. But this should only change at major architecture boundaries; I don't see the added value of doing the interpretation in kernel instead of userspace. I say this with my userspace hat on, of course ;-)

Quoted 5 lines
Quoted 3 lines

There is obviously overhead iterating over all the register that user space cares about. Another option (used by kbase) is to return some form of structured data so a missing property can be encoded.

I'll have a look at how kbase does that. Thanks for the pointer.

I'd be fine with the kbase approach but I don't really see the added value over what Boris proposed in the RFC, tbh.

#8 lore raw
Quoted 71 lines · Alyssa Rosenzweig
Quoted 71 lines
Quoted 70 lines
Quoted 56 lines
include/uapi/drm/pancsf_drm.h new file +56 -0
+struct drm_pancsf_gpu_info {
+#define DRM_PANCSF_ARCH_MAJOR(x)		((x) >> 28)
+#define DRM_PANCSF_ARCH_MINOR(x)		(((x) >> 24) & 0xf)
+#define DRM_PANCSF_ARCH_REV(x)			(((x) >> 20) & 0xf)
+#define DRM_PANCSF_PRODUCT_MAJOR(x)		(((x) >> 16) & 0xf)
+#define DRM_PANCSF_VERSION_MAJOR(x)		(((x) >> 12) & 0xf)
+#define DRM_PANCSF_VERSION_MINOR(x)		(((x) >> 4) & 0xff)
+#define DRM_PANCSF_VERSION_STATUS(x)		((x) & 0xf)
+	__u32 gpu_id;
+	__u32 gpu_rev;
+#define DRM_PANCSF_CSHW_MAJOR(x)		(((x) >> 26) & 0x3f)
+#define DRM_PANCSF_CSHW_MINOR(x)		(((x) >> 20) & 0x3f)
+#define DRM_PANCSF_CSHW_REV(x)			(((x) >> 16) & 0xf)
+#define DRM_PANCSF_MCU_MAJOR(x)			(((x) >> 10) & 0x3f)
+#define DRM_PANCSF_MCU_MINOR(x)			(((x) >> 4) & 0x3f)
+#define DRM_PANCSF_MCU_REV(x)			((x) & 0xf)
+	__u32 csf_id;
+	__u32 l2_features;
+	__u32 tiler_features;
+	__u32 mem_features;
+	__u32 mmu_features;
+	__u32 thread_features;
+	__u32 max_threads;
+	__u32 thread_max_workgroup_size;
+	__u32 thread_max_barrier_size;
+	__u32 coherency_features;
+	__u32 texture_features[4];
+	__u32 as_present;
+	__u32 core_group_count;
+	__u64 shader_present;
+	__u64 l2_present;
+	__u64 tiler_present;
+};
+
+struct drm_pancsf_csif_info {
+	__u32 csg_slot_count;
+	__u32 cs_slot_count;
+	__u32 cs_reg_count;
+	__u32 scoreboard_slot_count;
+	__u32 unpreserved_cs_reg_count;
+};
+
+struct drm_pancsf_dev_query {
+	/** @type: the query type (see enum drm_pancsf_dev_query_type). */
+	__u32 type;
+
+	/**
+	 * @size: size of the type being queried.
+	 *
+	 * If pointer is NULL, size is updated by the driver to provide the
+	 * output structure size. If pointer is not NULL, the the driver will
+	 * only copy min(size, actual_structure_size) bytes to the pointer,
+	 * and update the size accordingly. This allows us to extend query
+	 * types without breaking userspace.
+	 */
+	__u32 size;

include/uapi/drm/pancsf_drm.h line 123

Quoted 10 lines
include/uapi/drm/pancsf_drm.h new file +10 -0
+
+	/**
+	 * @pointer: user pointer to a query type struct.
+	 *
+	 * Pointer can be NULL, in which case, nothing is copied, but the
+	 * actual structure size is returned. If not NULL, it must point to
+	 * a location that's large enough to hold size bytes.
+	 */
+	__u64 pointer;
+};  

Genuine question: is there something wrong with the panfrost 'get_param' ioctl where individual features are queried one-by-one, rather than passing a big structure back to user space.

Quoted 8 lines
Quoted 3 lines

Well, I've just seen the Xe driver exposing things this way, and I thought it was a good idea, but I don't have a strong opinion here, and if others think it's preferable to stick to GET_PARAM, I'm fine with that too.

I vastly prefer the info struct, GET_PARAM isn't a great interface when there are large numbers of properties to query... Actually I just suggested to Lina that she adopt this approach for Asahi instead of the current GET_PARAM ioctl we have (downstream for now).

Ok, good to know there is some preference here - like I said this was a genuine question: I'm not trying to say this is wrong.

Quoted 8 lines

It isn't a *big* deal but GET_PARAM doesn't really seem better on any axes.

Quoted 4 lines
Quoted 4 lines

I ask because we've had issues in the past with trying to 'deprecate' registers - if a new version of the hardware stops providing a (meaningful) value for a register then it's hard to fix up the structures.

Quoted 7 lines

I'm not sure this is a big deal. If the register no longer exists (meaningfully), zero it out in the info structure and trust userspace to interpret meaningfully based on the GPU. If registers are getting dropped between revisions, that's obviously not great. But this should only change at major architecture boundaries; I don't see the added value of doing the interpretation in kernel instead of userspace. I say this with my userspace hat on, of course ;-)

Just some background:

In the early days of the Midgard DDK driver there was a structure much like the one proposed here in which the kernel dumped the various feature registers and passed it as a large struct to user space. User space then ended up using that struct directly in various bits of code all over the place.

We then ended up with the problem that it was easy to add new properties to that struct (including derived and hideously badly defined ones like "gpu_available_memory_size") but basically impossible to remove anything since the struct definition was shared between user and kernel. The kernel couldn't drop anything because old user space might need it, and user space picked up the definition from the kernel so these problematic members were always there to tempt user space coders.

By packing the data into a structured blob it provided the ability to:

* Provide a logical separation between user and kernel - each could
have their own structure and it was definitively not uABI.

* The kernel could provide backwards compat values for properties we
wanted to kill and user space could simply ignore them. They wouldn't be around to tempt future user space coders.

* New properties could be added without breaking old user space, and
without forever bloating the structure if we wanted to back out the change (at worst you just burn an ID value for the structured blob).

* (In theory) it's possible for user space to identify that a property
isn't present (e.g. running on an old kernel) rather than trying to populate every field with a dummy value. As far as I remember this never actually happened... user space just made up values for what was missing ;)

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

There is obviously overhead iterating over all the register that user space cares about. Another option (used by kbase) is to return some form of structured data so a missing property can be encoded.

I'll have a look at how kbase does that. Thanks for the pointer.

I'd be fine with the kbase approach but I don't really see the added value over what Boris proposed in the RFC, tbh.

My main concern is making that structure uABI. Which is because 6 years ago I had to clean up the mess that we had in the DDK ;)

Although I'll admit that most of the problems we had with the DDK was user space developers wanting information that the GPU driver shouldn't have been providing (maximum amount of memory available, clock speeds etc) or derived properties that user space could have calculated itself (e.g. decoded GPU ID). I guess this is also one of the problems with developing a driver in parallel with the hardware - things get added to try the idea out, and not always reverted if it turns out badly (or even cleaned up if it's a good idea).

Panfrost/PanCSF thankfully have much cleaner sets of properties exposed.

Steve

#9 lore raw
In reply to Boris Brezillon
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in…

Hi Boris,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on drm-misc/drm-misc-next]
[also build test ERROR on drm/drm-next drm-exynos/exynos-drm-next drm-intel/for-linux-next drm-intel/for-linux-next-fixes drm-tip/drm-tip linus/master v6.2-rc6 next-20230203] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Boris-Brezillon/drm-pancsf-Add-a-new-driver-for-Mali-CSF-based-GPUs/20230201-172815
base:   git://anongit.freedesktop.org/drm/drm-misc drm-misc-next
patch link:    https://lore.kernel.org/r/20230201084832.1708866-1-boris.brezillon%40collabora.com
patch subject: [RFC PATCH] drm/pancsf: Add a new driver for Mali CSF-based GPUs
config: arc-allyesconfig (https://download.01.org/0day-ci/archive/20230206/[email protected]/config)
compiler: arceb-elf-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/794ce1c0209242bf1496fc4195ca707d45b37e73
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Boris-Brezillon/drm-pancsf-Add-a-new-driver-for-Mali-CSF-based-GPUs/20230201-172815 git checkout 794ce1c0209242bf1496fc4195ca707d45b37e73
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=arc olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=arc SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable | Reported-by: kernel test robot <[email protected]>

All errors (new ones prefixed by >>):

drivers/gpu/drm/pancsf/pancsf_mcu.c:248:1: warning: no previous prototype for 'pancsf_fw_mem_alloc' [-Wmissing-prototypes]
248 | pancsf_fw_mem_alloc(struct pancsf_device *pfdev,
| ^~~~~~~~~~~~~~~~~~~
In file included from arch/arc/include/asm/thread_info.h:16,
from include/linux/thread_info.h:60,
from include/asm-generic/preempt.h:5,
from ./arch/arc/include/generated/asm/preempt.h:1,
from include/linux/preempt.h:78,
from include/linux/rcupdate.h:27,
from include/linux/rculist.h:11,
from include/linux/pid.h:5,
from include/linux/sched.h:14,
from include/linux/ratelimit.h:6,
from include/linux/dev_printk.h:16,
from include/linux/device.h:15,
from include/linux/dma-mapping.h:7,
from drivers/gpu/drm/pancsf/pancsf_mcu.c:4:
drivers/gpu/drm/pancsf/pancsf_mcu.c: In function 'pancsf_fw_load_section_entry':

>> arch/arc/include/asm/page.h:82:37: error: expected ')' before '{' token
      82 | #define __pgprot(x)     ((pgprot_t) { (x) })
         |                                     ^
   arch/arc/include/asm/pgtable-bits-arcv2.h:49:25: note: in expansion of macro '__pgprot'
      49 | #define PAGE_KERNEL     __pgprot(___DEF | _PAGE_GLOBAL \
         |                         ^~~~~~~~
   include/linux/highmem-internal.h:32:19: note: in expansion of macro 'PAGE_KERNEL'
      32 | #define kmap_prot PAGE_KERNEL
         |                   ^~~~~~~~~~~
   drivers/gpu/drm/pancsf/pancsf_mcu.c:443:34: note: in expansion of macro 'kmap_prot'
     443 |                         pgprot_t kmap_prot = PAGE_KERNEL;
         |                                  ^~~~~~~~~
>> arch/arc/include/asm/page.h:82:44: error: expected statement before ')' token
      82 | #define __pgprot(x)     ((pgprot_t) { (x) })
         |                                            ^
   arch/arc/include/asm/pgtable-bits-arcv2.h:49:25: note: in expansion of macro '__pgprot'
      49 | #define PAGE_KERNEL     __pgprot(___DEF | _PAGE_GLOBAL \
         |                         ^~~~~~~~
   drivers/gpu/drm/pancsf/pancsf_mcu.c:443:46: note: in expansion of macro 'PAGE_KERNEL'
     443 |                         pgprot_t kmap_prot = PAGE_KERNEL;
         |                                              ^~~~~~~~~~~

vim +82 arch/arc/include/asm/page.h

5dda4dc570ac41 Vineet Gupta     2013-01-18  80  
5dda4dc570ac41 Vineet Gupta     2013-01-18  81  #define pgprot_val(x)	((x).pgprot)
5dda4dc570ac41 Vineet Gupta     2013-01-18 @82  #define __pgprot(x)	((pgprot_t) { (x) })

4368902bb90f0e Gilad Ben-Yossef 2013-01-22 83 #define pte_pgprot(x) __pgprot(pte_val(x))
4368902bb90f0e Gilad Ben-Yossef 2013-01-22 84

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests