Skip to contentSkip to conversation

Support fdinfo runtime and memory stats on Panthor

PATCH Applied Adrián Larumbe 5 participants 14 messages Sep 23, 2024 – Oct 2, 2024

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v8 cover Support fdinfo runtime and memory stats on Panthor

This patch series enables userspace utilities like gputop and nvtop to query a render context's fdinfo file and figure out rates of engine and memory utilisation.

Previous discussion can be found at
https://lore.kernel.org/dri-devel/[email protected]/

Changelog:
v8:
- Fixed uninitialised stack variable bug that was triggering an invalid memory reference.
- Added a few R-b tags to commits
v7:
- Fixed some kernel test bot-reported documentation and sign mismatch errors.
- Defined convenience macros for specifying CS instructions according to their profiled status.
- Explicitly initialised instruction count for structure containing a job's instructions when calculating its amount of credits for the scheduler.
- Some minor cosmetic nits.
v6:
- Addressed some nits and style issues.
- Enforced static assert equality of instruction buffer when calculating job credits or copying them into the ringbuffer.
- Added explanation to the way in which job credits and profiled job size is done.
- Broke down fdinfo enablement patch into two, one of them dealing with adding support for calculating the current and top operating device frequencies
- Fixed race at the time drm file-wide profiling stats are gathered from groups.
v5:
- Moved profiling information slots into a per-queue BO and away from syncobjs.
- Decide on size of profiling slots BO from size of CS for minimal profiled job
- Turn job and device profiling flag into a bit mask so that individual metrics can be enabled separately.
- Shrunk ringbuffer slot size to that of a cache line.
- Track profiling slot indeces separately from the job's queue ringbuffer's
- Emit CS instructions one by one and tag them depending on profiling mask
- New helper for calculating job credits depending on profiling flags
- Add Documentation file for sysfs profiling knob
- fdinfo will only show engines or cycles tags if these are respectively enabled.
v4:
- Fixed wrong assignment location for frequency values in Panthor's devfreq
- Removed the last two commits about registering size of internal BO's
- Rearranged patch series so that sysfs knob is done last and all the previous time sampling and fdinfo show dependencies are already in place
v3:
- Fixed some nits and removed useless bounds check in panthor_sched.c
- Added support for sysfs profiling knob and optional job accounting
- Added new patches for calculating size of internal BO's
v2:
- Split original first patch in two, one for FW CS cycle and timestamp calculations and job accounting memory management, and a second one
that enables fdinfo.
- Moved NUM_INSTRS_PER_SLOT to the file prelude
- Removed nelem variable from the group's struct definition.
- Precompute size of group's syncobj BO to avoid code duplication.
- Some minor nits.

Adrián Larumbe (5):
drm/panthor: introduce job cycle and timestamp accounting drm/panthor: record current and maximum device clock frequencies drm/panthor: add DRM fdinfo support
drm/panthor: enable fdinfo for memory stats drm/panthor: add sysfs knob for enabling job profiling

 .../testing/sysfs-driver-panthor-profiling    |  10 +
 Documentation/gpu/panthor.rst                 |  46 +++
 drivers/gpu/drm/panthor/panthor_devfreq.c     |  18 +-
 drivers/gpu/drm/panthor/panthor_device.h      |  36 ++
 drivers/gpu/drm/panthor/panthor_drv.c         |  73 ++++
 drivers/gpu/drm/panthor/panthor_gem.c         |  12 +
 drivers/gpu/drm/panthor/panthor_sched.c       | 384 +++++++++++++++---
 drivers/gpu/drm/panthor/panthor_sched.h       |   2 +
 8 files changed, 531 insertions(+), 50 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-driver-panthor-profiling
 create mode 100644 Documentation/gpu/panthor.rst
Signature · 1 line
2.46.0
#1 lore raw

PATCH v8 1/5 drm/panthor: introduce job cycle and timestamp accounting

Enable calculations of job submission times in clock cycles and wall time. This is done by expanding the boilerplate command stream when running a job to include instructions that compute said times right before and after a user CS.

A separate kernel BO is created per queue to store those values. Jobs can access their sampled data through an index different from that of the queue's ringbuffer. The reason for this is saving memory on the profiling information kernel BO, since the amount of simultaneous profiled jobs we can write into the queue's ringbuffer might be much smaller than for regular jobs, as the former take more CSF instructions.

This commit is done in preparation for enabling DRM fdinfo support in the Panthor driver, which depends on the numbers calculated herein.

A profile mode mask has been added that will in a future commit allow UM to toggle performance metric sampling behaviour, which is disabled by default to save power. When a ringbuffer CS is constructed, timestamp and cycling sampling instructions are added depending on the enabled flags in the profiling mask.

A helper was provided that calculates the number of instructions for a given set of enablement mask, and these are passed as the number of credits when initialising a DRM scheduler job.

---
 drivers/gpu/drm/panthor/panthor_device.h |  22 ++
 drivers/gpu/drm/panthor/panthor_sched.c  | 328 +++++++++++++++++++----
 2 files changed, 301 insertions(+), 49 deletions(-)
drivers/gpu/drm/panthor/panthor_device.h +22 -0
index e388c0472ba7..a48e30d0af30 100644
@@ -66,6 +66,25 @@ struct panthor_irq {
 	atomic_t suspended;
 };
 
+/**
+ * enum panthor_device_profiling_mode - Profiling state
+ */
+enum panthor_device_profiling_flags {
+	/** @PANTHOR_DEVICE_PROFILING_DISABLED: Profiling is disabled. */
+	PANTHOR_DEVICE_PROFILING_DISABLED = 0,
+
+	/** @PANTHOR_DEVICE_PROFILING_CYCLES: Sampling job cycles. */
+	PANTHOR_DEVICE_PROFILING_CYCLES = BIT(0),
+
+	/** @PANTHOR_DEVICE_PROFILING_TIMESTAMP: Sampling job timestamp. */
+	PANTHOR_DEVICE_PROFILING_TIMESTAMP = BIT(1),
+
+	/** @PANTHOR_DEVICE_PROFILING_ALL: Sampling everything. */
+	PANTHOR_DEVICE_PROFILING_ALL =
+	PANTHOR_DEVICE_PROFILING_CYCLES |
+	PANTHOR_DEVICE_PROFILING_TIMESTAMP,
+};
+
 /**
  * struct panthor_device - Panthor device
  */
@@ -162,6 +181,9 @@ struct panthor_device {
 		 */
 		struct page *dummy_latest_flush;
 	} pm;
+
+	/** @profile_mask: User-set profiling flags for job accounting. */
+	u32 profile_mask;
 };
 
 /**
drivers/gpu/drm/panthor/panthor_sched.c +279 -49

429 lines not on this page · raw message

Signature · 1 line
2.46.0
maintainer
#2 lore raw

Re: [PATCH v8 1/5] drm/panthor: introduce job cycle and timestamp accounting

Quoted 11 lines · On Tue, 24 Sep 2024 00:06:21 +0100 Adrián Larumbe…
drivers/gpu/drm/panthor/panthor_sched.c +11 -0
+static u32 calc_profiling_ringbuf_num_slots(struct panthor_device *ptdev,
+				       u32 cs_ringbuf_size)
+{
+	u32 min_profiled_job_instrs = U32_MAX;
+	u32 last_flag = fls(PANTHOR_DEVICE_PROFILING_ALL);
+
+	/*
+	 * We want to calculate the minimum size of a profiled job's CS,
+	 * because since they need additional instructions for the sampling
+	 * of performance metrics, they might take up further slots in
+	 * the queue's ringbuffer. This means we might not need as many job

drivers/gpu/drm/panthor/panthor_sched.c static const struct drm_sched_backend_ops panthor_queue_sched_ops = { · line 3194

Quoted 10 lines
drivers/gpu/drm/panthor/panthor_sched.c +10 -0
+	 * slots for keeping track of their profiling information. What we
+	 * need is the maximum number of slots we should allocate to this end,
+	 * which matches the maximum number of profiled jobs we can place
+	 * simultaneously in the queue's ring buffer.
+	 * That has to be calculated separately for every single job profiling
+	 * flag, but not in the case job profiling is disabled, since unprofiled
+	 * jobs don't need to keep track of this at all.
+	 */
+	for (u32 i = 0; i < last_flag; i++) {
+		if (BIT(i) & PANTHOR_DEVICE_PROFILING_ALL)

I'll get rid of this check when applying, as suggested by Steve. Steve, with this modification do you want me to add your R-b?

BTW, I've also fixed a bunch of checkpatch errors/warnings, so you might want to run checkpatch --strict next time.

Quoted 6 lines
drivers/gpu/drm/panthor/panthor_sched.c +6 -0
+			min_profiled_job_instrs =
+				min(min_profiled_job_instrs, calc_job_credits(BIT(i)));
+	}
+
+	return DIV_ROUND_UP(cs_ringbuf_size, min_profiled_job_instrs * sizeof(u64));
+}
maintainer
#3 lore raw

Re: [PATCH v8 1/5] drm/panthor: introduce job cycle and timestamp accounting

Quoted 19 lines · Boris Brezillon
Quoted 16 lines · On Tue, 24 Sep 2024 00:06:21 +0100 Adrián Larumbe…
drivers/gpu/drm/panthor/panthor_sched.c +16 -0
+static u32 calc_profiling_ringbuf_num_slots(struct panthor_device *ptdev,
+				       u32 cs_ringbuf_size)
+{
+	u32 min_profiled_job_instrs = U32_MAX;
+	u32 last_flag = fls(PANTHOR_DEVICE_PROFILING_ALL);
+
+	/*
+	 * We want to calculate the minimum size of a profiled job's CS,
+	 * because since they need additional instructions for the sampling
+	 * of performance metrics, they might take up further slots in
+	 * the queue's ringbuffer. This means we might not need as many job
+	 * slots for keeping track of their profiling information. What we
+	 * need is the maximum number of slots we should allocate to this end,
+	 * which matches the maximum number of profiled jobs we can place
+	 * simultaneously in the queue's ring buffer.
+	 * That has to be calculated separately for every single job profiling
Quoted 8 lines

drivers/gpu/drm/panthor/panthor_sched.c static const struct drm_sched_backend_ops panthor_queue_sched_ops = { · line 3194

Quoted 5 lines
drivers/gpu/drm/panthor/panthor_sched.c +5 -0
+	 * flag, but not in the case job profiling is disabled, since unprofiled
+	 * jobs don't need to keep track of this at all.
+	 */
+	for (u32 i = 0; i < last_flag; i++) {
+		if (BIT(i) & PANTHOR_DEVICE_PROFILING_ALL)

I'll get rid of this check when applying, as suggested by Steve. Steve, with this modification do you want me to add your R-b?

Yes, please do.

Thanks,
Steve

Quoted 9 lines

BTW, I've also fixed a bunch of checkpatch errors/warnings, so you might want to run checkpatch --strict next time.

Quoted 6 lines
drivers/gpu/drm/panthor/panthor_sched.c +6 -0
+			min_profiled_job_instrs =
+				min(min_profiled_job_instrs, calc_job_credits(BIT(i)));
+	}
+
+	return DIV_ROUND_UP(cs_ringbuf_size, min_profiled_job_instrs * sizeof(u64));
+}
#4 lore raw

PATCH v8 2/5 drm/panthor: record current and maximum device clock frequencies

In reply to Adrián Larumbe

In order to support UM in calculating rates of GPU utilisation, the current operating and maximum GPU clock frequencies must be recorded during device initialisation, and also during OPP state transitions.

---
 drivers/gpu/drm/panthor/panthor_devfreq.c | 18 +++++++++++++++++-
 drivers/gpu/drm/panthor/panthor_device.h  |  6 ++++++
 2 files changed, 23 insertions(+), 1 deletion(-)
drivers/gpu/drm/panthor/panthor_devfreq.c +17 -1
index c6d3c327cc24..9d0f891b9b53 100644
@@ -62,14 +62,20 @@ static void panthor_devfreq_update_utilization(struct panthor_devfreq *pdevfreq)
 static int panthor_devfreq_target(struct device *dev, unsigned long *freq,
 				  u32 flags)
 {
+	struct panthor_device *ptdev = dev_get_drvdata(dev);
 	struct dev_pm_opp *opp;
+	int err;
 
 	opp = devfreq_recommended_opp(dev, freq, flags);
 	if (IS_ERR(opp))
 		return PTR_ERR(opp);
 	dev_pm_opp_put(opp);
 
-	return dev_pm_opp_set_rate(dev, *freq);
+	err = dev_pm_opp_set_rate(dev, *freq);
+	if (!err)
+		ptdev->current_frequency = *freq;
+
+	return err;
 }
 
 static void panthor_devfreq_reset(struct panthor_devfreq *pdevfreq)
@@ -130,6 +136,7 @@ int panthor_devfreq_init(struct panthor_device *ptdev)
 	struct panthor_devfreq *pdevfreq;
 	struct dev_pm_opp *opp;
 	unsigned long cur_freq;
+	unsigned long freq = ULONG_MAX;
 	int ret;
 
 	pdevfreq = drmm_kzalloc(&ptdev->base, sizeof(*ptdev->devfreq), GFP_KERNEL);
@@ -161,6 +168,7 @@ int panthor_devfreq_init(struct panthor_device *ptdev)
 		return PTR_ERR(opp);
 
 	panthor_devfreq_profile.initial_freq = cur_freq;
+	ptdev->current_frequency = cur_freq;
 
 	/* Regulator coupling only takes care of synchronizing/balancing voltage
 	 * updates, but the coupled regulator needs to be enabled manually.
@@ -204,6 +212,14 @@ int panthor_devfreq_init(struct panthor_device *ptdev)
 
 	dev_pm_opp_put(opp);
 
+	/* Find the fastest defined rate  */
+	opp = dev_pm_opp_find_freq_floor(dev, &freq);
+	if (IS_ERR(opp))
+		return PTR_ERR(opp);
+	ptdev->fast_rate = freq;
+
+	dev_pm_opp_put(opp);
+
 	/*
 	 * Setup default thresholds for the simple_ondemand governor.
 	 * The values are chosen based on experiments.
drivers/gpu/drm/panthor/panthor_device.h +6 -0
index a48e30d0af30..2109905813e8 100644
@@ -184,6 +184,12 @@ struct panthor_device {
 
 	/** @profile_mask: User-set profiling flags for job accounting. */
 	u32 profile_mask;
+
+	/** @current_frequency: Device clock frequency at present. Set by DVFS*/
+	unsigned long current_frequency;
+
+	/** @fast_rate: Maximum device clock frequency. Set by DVFS */
+	unsigned long fast_rate;
 };
 
 /**
Signature · 1 line
2.46.0
#5 lore raw

PATCH v8 3/5 drm/panthor: add DRM fdinfo support

In reply to Adrián Larumbe

Drawing from the FW-calculated values in a previous commit, we can increase the numbers for an open file by collecting them from finished jobs when updating their group synchronisation objects.

Display of fdinfo key-value pairs is governed by a bitmask that is by default unset in the present commit, and supporting manual toggle of it will be the matter of a later commit.

---
 drivers/gpu/drm/panthor/panthor_device.h |  8 ++++
 drivers/gpu/drm/panthor/panthor_drv.c    | 34 ++++++++++++++
 drivers/gpu/drm/panthor/panthor_sched.c  | 56 ++++++++++++++++++++++++
 drivers/gpu/drm/panthor/panthor_sched.h  |  2 +
 4 files changed, 100 insertions(+)
drivers/gpu/drm/panthor/panthor_device.h +8 -0
index 2109905813e8..0e68f5a70d20 100644
@@ -192,6 +192,11 @@ struct panthor_device {
 	unsigned long fast_rate;
 };
 
+struct panthor_gpu_usage {
+	u64 time;
+	u64 cycles;
+};
+
 /**
  * struct panthor_file - Panthor file
  */
@@ -204,6 +209,9 @@ struct panthor_file {
 
 	/** @groups: Scheduling group pool attached to this file. */
 	struct panthor_group_pool *groups;
+
+	/** @stats: cycle and timestamp measures for job execution. */
+	struct panthor_gpu_usage stats;
 };
 
 int panthor_device_init(struct panthor_device *ptdev);
drivers/gpu/drm/panthor/panthor_drv.c +34 -0
index 0caf9e9a8c45..233b265c0819 100644
@@ -13,6 +13,7 @@
 #include <linux/pagemap.h>
 #include <linux/platform_device.h>
 #include <linux/pm_runtime.h>
+#include <linux/time64.h>
 
 #include <drm/drm_auth.h>
 #include <drm/drm_debugfs.h>
@@ -1414,6 +1415,37 @@ static int panthor_mmap(struct file *filp, struct vm_area_struct *vma)
 	return ret;
 }
 
+static void panthor_gpu_show_fdinfo(struct panthor_device *ptdev,
+				    struct panthor_file *pfile,
+				    struct drm_printer *p)
+{
+	if (ptdev->profile_mask & PANTHOR_DEVICE_PROFILING_ALL)
+		panthor_fdinfo_gather_group_samples(pfile);
+
+	if (ptdev->profile_mask & PANTHOR_DEVICE_PROFILING_TIMESTAMP) {
+#ifdef CONFIG_ARM_ARCH_TIMER
+		drm_printf(p, "drm-engine-panthor:\t%llu ns\n",
+			   DIV_ROUND_UP_ULL((pfile->stats.time * NSEC_PER_SEC),
+					    arch_timer_get_cntfrq()));
+#endif
+	}
+	if (ptdev->profile_mask & PANTHOR_DEVICE_PROFILING_CYCLES)
+		drm_printf(p, "drm-cycles-panthor:\t%llu\n", pfile->stats.cycles);
+
+	drm_printf(p, "drm-maxfreq-panthor:\t%lu Hz\n", ptdev->fast_rate);
+	drm_printf(p, "drm-curfreq-panthor:\t%lu Hz\n", ptdev->current_frequency);
+}
+
+static void panthor_show_fdinfo(struct drm_printer *p, struct drm_file *file)
+{
+	struct drm_device *dev = file->minor->dev;
+	struct panthor_device *ptdev = container_of(dev, struct panthor_device, base);
+
+	panthor_gpu_show_fdinfo(ptdev, file->driver_priv, p);
+
+	drm_show_memory_stats(p, file);
+}
+
 static const struct file_operations panthor_drm_driver_fops = {
 	.open = drm_open,
 	.release = drm_release,
@@ -1423,6 +1455,7 @@ static const struct file_operations panthor_drm_driver_fops = {
 	.read = drm_read,
 	.llseek = noop_llseek,
 	.mmap = panthor_mmap,
+	.show_fdinfo = drm_show_fdinfo,
 };
 
 #ifdef CONFIG_DEBUG_FS
@@ -1442,6 +1475,7 @@ static const struct drm_driver panthor_drm_driver = {
 			   DRIVER_SYNCOBJ_TIMELINE | DRIVER_GEM_GPUVA,
 	.open = panthor_open,
 	.postclose = panthor_postclose,
+	.show_fdinfo = panthor_show_fdinfo,
 	.ioctls = panthor_drm_driver_ioctls,
 	.num_ioctls = ARRAY_SIZE(panthor_drm_driver_ioctls),
 	.fops = &panthor_drm_driver_fops,
drivers/gpu/drm/panthor/panthor_sched.c +56 -0

86 lines not on this page · raw message

drivers/gpu/drm/panthor/panthor_sched.h +2 -0
index 3a30d2328b30..5ae6b4bde7c5 100644
@@ -47,4 +47,6 @@ void panthor_sched_resume(struct panthor_device *ptdev);
 void panthor_sched_report_mmu_fault(struct panthor_device *ptdev);
 void panthor_sched_report_fw_events(struct panthor_device *ptdev, u32 events);
 
+void panthor_fdinfo_gather_group_samples(struct panthor_file *pfile);
+
 #endif
Signature · 1 line
2.46.0
#6 lore raw

Re: [PATCH v8 3/5] drm/panthor: add DRM fdinfo support

kernel test robot noticed the following build errors:

Hi Adrián,

kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on v6.11 next-20240927]
[cannot apply to drm-misc/drm-misc-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Adri-n-Larumbe/drm-panthor-introduce-job-cycle-and-timestamp-accounting/20240924-071018
base:   linus/master
patch link:    https://lore.kernel.org/r/20240923230912.2207320-4-adrian.larumbe%40collabora.com
patch subject: [PATCH v8 3/5] drm/panthor: add DRM fdinfo support
config: arm-randconfig-002-20240929 (https://download.01.org/0day-ci/archive/20240929/[email protected]/config)
compiler: arm-linux-gnueabi-gcc (GCC) 14.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240929/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from include/linux/math64.h:6,
from include/linux/time.h:6,
from include/linux/stat.h:19,
from include/linux/module.h:13,
from drivers/gpu/drm/panthor/panthor_drv.c:7:
drivers/gpu/drm/panthor/panthor_drv.c: In function 'panthor_gpu_show_fdinfo':

>> drivers/gpu/drm/panthor/panthor_drv.c:1389:45: error: implicit declaration of function 'arch_timer_get_cntfrq' [-Wimplicit-function-declaration]
    1389 |                                             arch_timer_get_cntfrq()));
         |                                             ^~~~~~~~~~~~~~~~~~~~~
   include/linux/math.h:40:39: note: in definition of macro 'DIV_ROUND_DOWN_ULL'
      40 |         ({ unsigned long long _tmp = (ll); do_div(_tmp, d); _tmp; })
         |                                       ^~
   drivers/gpu/drm/panthor/panthor_drv.c:1388:28: note: in expansion of macro 'DIV_ROUND_UP_ULL'
    1388 |                            DIV_ROUND_UP_ULL((pfile->stats.time * NSEC_PER_SEC),
         |                            ^~~~~~~~~~~~~~~~

vim +/arch_timer_get_cntfrq +1389 drivers/gpu/drm/panthor/panthor_drv.c

1377
1378 static void panthor_gpu_show_fdinfo(struct panthor_device *ptdev,
1379 struct panthor_file *pfile,
1380 struct drm_printer *p)
1381 {
1382 if (ptdev->profile_mask & PANTHOR_DEVICE_PROFILING_ALL)
1383 panthor_fdinfo_gather_group_samples(pfile);
1384
1385 if (ptdev->profile_mask & PANTHOR_DEVICE_PROFILING_TIMESTAMP) {
1386 #ifdef CONFIG_ARM_ARCH_TIMER
1387 drm_printf(p, "drm-engine-panthor:\t%llu ns\n",
1388 DIV_ROUND_UP_ULL((pfile->stats.time * NSEC_PER_SEC),

> 1389						    arch_timer_get_cntfrq()));

1390 #endif
1391 }
1392 if (ptdev->profile_mask & PANTHOR_DEVICE_PROFILING_CYCLES)
1393 drm_printf(p, "drm-cycles-panthor:\t%llu\n", pfile->stats.cycles);
1394
1395 drm_printf(p, "drm-maxfreq-panthor:\t%lu Hz\n", ptdev->fast_rate);
1396 drm_printf(p, "drm-curfreq-panthor:\t%lu Hz\n", ptdev->current_frequency);
1397 }
1398

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
maintainer
#7 lore raw

Re: [PATCH v8 3/5] drm/panthor: add DRM fdinfo support

Quoted 11 lines · On Sun, 29 Sep 2024 08:25:38 +0800 kernel test robot <[email protected]> wrote:

Hi Adrián,

kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on v6.11 next-20240927]
[cannot apply to drm-misc/drm-misc-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

Quoted 7 lines
url:    https://github.com/intel-lab-lkp/linux/commits/Adri-n-Larumbe/drm-panthor-introduce-job-cycle-and-timestamp-accounting/20240924-071018
base:   linus/master
patch link:    https://lore.kernel.org/r/20240923230912.2207320-4-adrian.larumbe%40collabora.com
patch subject: [PATCH v8 3/5] drm/panthor: add DRM fdinfo support
config: arm-randconfig-002-20240929 (https://download.01.org/0day-ci/archive/20240929/[email protected]/config)
compiler: arm-linux-gnueabi-gcc (GCC) 14.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240929/[email protected]/reproduce)

I gave this a try with Adrian's series applied on top of drm-misc-next, and I couldn't reproduce the issue.

Quoted 50 lines

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from include/linux/math64.h:6,
from include/linux/time.h:6,
from include/linux/stat.h:19,
from include/linux/module.h:13,
from drivers/gpu/drm/panthor/panthor_drv.c:7:
drivers/gpu/drm/panthor/panthor_drv.c: In function 'panthor_gpu_show_fdinfo':

Quoted 1 line
Quoted 1 line

drivers/gpu/drm/panthor/panthor_drv.c:1389:45: error: implicit declaration of function 'arch_timer_get_cntfrq' [-Wimplicit-function-declaration]

    1389 |                                             arch_timer_get_cntfrq()));
         |                                             ^~~~~~~~~~~~~~~~~~~~~
   include/linux/math.h:40:39: note: in definition of macro 'DIV_ROUND_DOWN_ULL'
      40 |         ({ unsigned long long _tmp = (ll); do_div(_tmp, d); _tmp; })
         |                                       ^~
   drivers/gpu/drm/panthor/panthor_drv.c:1388:28: note: in expansion of macro 'DIV_ROUND_UP_ULL'
    1388 |                            DIV_ROUND_UP_ULL((pfile->stats.time * NSEC_PER_SEC),
         |                            ^~~~~~~~~~~~~~~~

vim +/arch_timer_get_cntfrq +1389 drivers/gpu/drm/panthor/panthor_drv.c

1377
1378 static void panthor_gpu_show_fdinfo(struct panthor_device *ptdev,
1379 struct panthor_file *pfile,
1380 struct drm_printer *p)
1381 {
1382 if (ptdev->profile_mask & PANTHOR_DEVICE_PROFILING_ALL)
1383 panthor_fdinfo_gather_group_samples(pfile);
1384
1385 if (ptdev->profile_mask & PANTHOR_DEVICE_PROFILING_TIMESTAMP) {
1386 #ifdef CONFIG_ARM_ARCH_TIMER
1387 drm_printf(p, "drm-engine-panthor:\t%llu ns\n",
1388 DIV_ROUND_UP_ULL((pfile->stats.time * NSEC_PER_SEC),

Quoted 1 line

1389 arch_timer_get_cntfrq()));

1390 #endif
1391 }
1392 if (ptdev->profile_mask & PANTHOR_DEVICE_PROFILING_CYCLES)
1393 drm_printf(p, "drm-cycles-panthor:\t%llu\n", pfile->stats.cycles);
1394
1395 drm_printf(p, "drm-maxfreq-panthor:\t%lu Hz\n", ptdev->fast_rate);
1396 drm_printf(p, "drm-curfreq-panthor:\t%lu Hz\n", ptdev->current_frequency);
1397 }
1398

maintainer
#8 lore raw

Re: [PATCH v8 3/5] drm/panthor: add DRM fdinfo support

In reply to Adrián Larumbe

drivers/gpu/drm/panthor/panthor_sched.c void panthor_sched_post_reset(struct panthor_device *ptdev, bool reset_failed) · line 2837

Quoted 10 lines · On Tue, 24 Sep 2024 00:06:23 +0100 Adrián Larumbe…
drivers/gpu/drm/panthor/panthor_sched.c +10 -0
+static void update_fdinfo_stats(struct panthor_job *job)
+{
+	struct panthor_group *group = job->group;
+	struct panthor_queue *queue = group->queues[job->queue_idx];
+	struct panthor_gpu_usage *fdinfo = &group->fdinfo.data;
+	struct panthor_job_profiling_data *times;
+
+	times = (struct panthor_job_profiling_data *)
+		((unsigned long) queue->profiling.slots->kmap +
+		 (job->profiling.slot * sizeof(struct panthor_job_profiling_data)));

The casting done here is a bit of a mess, so I replaced it by:

	struct panthor_job_profiling_data *slots = queue->profiling.slots->kmap;
	struct panthor_job_profiling_data *data = &slots[job->profiling.slot];
Quoted 8 lines
drivers/gpu/drm/panthor/panthor_sched.c +8 -0
+
+	mutex_lock(&group->fdinfo.lock);
+	if (job->profiling.mask & PANTHOR_DEVICE_PROFILING_CYCLES)
+		fdinfo->cycles += times->cycles.after - times->cycles.before;
+	if (job->profiling.mask & PANTHOR_DEVICE_PROFILING_TIMESTAMP)
+		fdinfo->time += times->time.after - times->time.before;
+	mutex_unlock(&group->fdinfo.lock);
+}
#9 lore raw

PATCH v8 4/5 drm/panthor: enable fdinfo for memory stats

In reply to Adrián Larumbe

Implement drm object's status callback.

Also, we consider a PRIME imported BO to be resident if its matching dma_buf has an open attachment, which means its backing storage had already been allocated.

---
 drivers/gpu/drm/panthor/panthor_gem.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/gpu/drm/panthor/panthor_gem.c +12 -0
index 38f560864879..c60b599665d8 100644
@@ -145,6 +145,17 @@ panthor_gem_prime_export(struct drm_gem_object *obj, int flags)
 	return drm_gem_prime_export(obj, flags);
 }
 
+static enum drm_gem_object_status panthor_gem_status(struct drm_gem_object *obj)
+{
+	struct panthor_gem_object *bo = to_panthor_bo(obj);
+	enum drm_gem_object_status res = 0;
+
+	if (bo->base.base.import_attach || bo->base.pages)
+		res |= DRM_GEM_OBJECT_RESIDENT;
+
+	return res;
+}
+
 static const struct drm_gem_object_funcs panthor_gem_funcs = {
 	.free = panthor_gem_free_object,
 	.print_info = drm_gem_shmem_object_print_info,
@@ -154,6 +165,7 @@ static const struct drm_gem_object_funcs panthor_gem_funcs = {
 	.vmap = drm_gem_shmem_object_vmap,
 	.vunmap = drm_gem_shmem_object_vunmap,
 	.mmap = panthor_gem_mmap,
+	.status = panthor_gem_status,
 	.export = panthor_gem_prime_export,
 	.vm_ops = &drm_gem_shmem_vm_ops,
 };
Signature · 1 line
2.46.0
#10 lore raw

PATCH v8 5/5 drm/panthor: add sysfs knob for enabling job profiling

In reply to Adrián Larumbe

This commit introduces a DRM device sysfs attribute that lets UM control the job accounting status in the device. The knob variable had been brought in as part of a previous commit, but now we're able to fix it manually.

As sysfs files are part of a driver's uAPI, describe its legitimate input values and output format in a documentation file.

---
 .../testing/sysfs-driver-panthor-profiling    | 10 ++++
 Documentation/gpu/panthor.rst                 | 46 +++++++++++++++++++
 drivers/gpu/drm/panthor/panthor_drv.c         | 39 ++++++++++++++++
 3 files changed, 95 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-driver-panthor-profiling
 create mode 100644 Documentation/gpu/panthor.rst
Documentation/ABI/testing/sysfs-driver-panthor-profiling new file +10 -0
new file mode 100644
index 000000000000..af05fccedc15
@@ -0,0 +1,10 @@
+What:		/sys/bus/platform/drivers/panthor/.../profiling
+Date:		September 2024
+KernelVersion:	6.11.0
+Contact:	Adrian Larumbe <[email protected]>
+Description:
+		Bitmask to enable drm fdinfo's job profiling measurements.
+		Valid values are:
+		0: Don't enable fdinfo job profiling sources.
+		1: Enable GPU cycle measurements for running jobs.
+		2: Enable GPU timestamp sampling for running jobs.
Documentation/gpu/panthor.rst new file +46 -0

46 lines not on this page · raw message

drivers/gpu/drm/panthor/panthor_drv.c +39 -0
index 233b265c0819..6f47d9d1d86a 100644
@@ -1513,6 +1513,44 @@ static void panthor_remove(struct platform_device *pdev)
 	panthor_device_unplug(ptdev);
 }
 
+static ssize_t profiling_show(struct device *dev,
+			      struct device_attribute *attr,
+			      char *buf)
+{
+	struct panthor_device *ptdev = dev_get_drvdata(dev);
+
+	return sysfs_emit(buf, "%d\n", ptdev->profile_mask);
+}
+
+static ssize_t profiling_store(struct device *dev,
+			       struct device_attribute *attr,
+			       const char *buf, size_t len)
+{
+	struct panthor_device *ptdev = dev_get_drvdata(dev);
+	u32 value;
+	int err;
+
+	err = kstrtou32(buf, 0, &value);
+	if (err)
+		return err;
+
+	if ((value & ~PANTHOR_DEVICE_PROFILING_ALL) != 0)
+		return -EINVAL;
+
+	ptdev->profile_mask = value;
+
+	return len;
+}
+
+static DEVICE_ATTR_RW(profiling);
+
+static struct attribute *panthor_attrs[] = {
+	&dev_attr_profiling.attr,
+	NULL,
+};
+
+ATTRIBUTE_GROUPS(panthor);
+
 static const struct of_device_id dt_match[] = {
 	{ .compatible = "rockchip,rk3588-mali" },
 	{ .compatible = "arm,mali-valhall-csf" },
@@ -1532,6 +1570,7 @@ static struct platform_driver panthor_driver = {
 		.name = "panthor",
 		.pm = pm_ptr(&panthor_pm_ops),
 		.of_match_table = dt_match,
+		.dev_groups = panthor_groups,
 	},
 };
 
Signature · 1 line
2.46.0
maintainer Applied
#11 lore raw
In reply to Adrián Larumbe
Quoted 53 lines · On Tue, 24 Sep 2024 00:06:20 +0100 Adrián Larumbe…

This patch series enables userspace utilities like gputop and nvtop to query a render context's fdinfo file and figure out rates of engine and memory utilisation.

Previous discussion can be found at
https://lore.kernel.org/dri-devel/[email protected]/

Changelog:
v8:
- Fixed uninitialised stack variable bug that was triggering an invalid memory reference.
- Added a few R-b tags to commits
v7:
- Fixed some kernel test bot-reported documentation and sign mismatch errors.
- Defined convenience macros for specifying CS instructions according to their profiled status.
- Explicitly initialised instruction count for structure containing a job's instructions when calculating its amount of credits for the scheduler.
- Some minor cosmetic nits.
v6:
- Addressed some nits and style issues.
- Enforced static assert equality of instruction buffer when calculating job credits or copying them into the ringbuffer.
- Added explanation to the way in which job credits and profiled job size is done.
- Broke down fdinfo enablement patch into two, one of them dealing with adding support for calculating the current and top operating device frequencies
- Fixed race at the time drm file-wide profiling stats are gathered from groups.
v5:
- Moved profiling information slots into a per-queue BO and away from syncobjs.
- Decide on size of profiling slots BO from size of CS for minimal profiled job
- Turn job and device profiling flag into a bit mask so that individual metrics can be enabled separately.
- Shrunk ringbuffer slot size to that of a cache line.
- Track profiling slot indeces separately from the job's queue ringbuffer's
- Emit CS instructions one by one and tag them depending on profiling mask
- New helper for calculating job credits depending on profiling flags
- Add Documentation file for sysfs profiling knob
- fdinfo will only show engines or cycles tags if these are respectively enabled.
v4:
- Fixed wrong assignment location for frequency values in Panthor's devfreq
- Removed the last two commits about registering size of internal BO's
- Rearranged patch series so that sysfs knob is done last and all the previous time sampling and fdinfo show dependencies are already in place
v3:
- Fixed some nits and removed useless bounds check in panthor_sched.c
- Added support for sysfs profiling knob and optional job accounting
- Added new patches for calculating size of internal BO's
v2:
- Split original first patch in two, one for FW CS cycle and timestamp calculations and job accounting memory management, and a second one
that enables fdinfo.
- Moved NUM_INSTRS_PER_SLOT to the file prelude
- Removed nelem variable from the group's struct definition.
- Precompute size of group's syncobj BO to avoid code duplication.
- Some minor nits.

Quoted 6 lines

Adrián Larumbe (5):
drm/panthor: introduce job cycle and timestamp accounting drm/panthor: record current and maximum device clock frequencies drm/panthor: add DRM fdinfo support
drm/panthor: enable fdinfo for memory stats drm/panthor: add sysfs knob for enabling job profiling

Queued to drm-misc-next after applying the few modifications I mentioned. Also added Steve's ack (given on IRC) on the first patch.

Quoted 13 lines
 .../testing/sysfs-driver-panthor-profiling    |  10 +
 Documentation/gpu/panthor.rst                 |  46 +++
 drivers/gpu/drm/panthor/panthor_devfreq.c     |  18 +-
 drivers/gpu/drm/panthor/panthor_device.h      |  36 ++
 drivers/gpu/drm/panthor/panthor_drv.c         |  73 ++++
 drivers/gpu/drm/panthor/panthor_gem.c         |  12 +
 drivers/gpu/drm/panthor/panthor_sched.c       | 384 +++++++++++++++---
 drivers/gpu/drm/panthor/panthor_sched.h       |   2 +
 8 files changed, 531 insertions(+), 50 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-driver-panthor-profiling
 create mode 100644 Documentation/gpu/panthor.rst
#12 lore raw
Quoted 2 lines · Boris Brezillon

Queued to drm-misc-next after applying the few modifications I mentioned. Also added Steve's ack (given on IRC) on the first patch.

Can we have the drm-tip rebuild conflict resolution too, please?

drivers/gpu/drm/panthor/panthor_drv.c +0 -0
diff --cc drivers/gpu/drm/panthor/panthor_drv.c
index c520f156e2d7,f9b93f84d611..000000000000
@@@ -1383,7 -1476,7 +1476,11 @@@ static const struct file_operations pan
        .read = drm_read,
        .llseek = noop_llseek,
        .mmap = panthor_mmap,
++<<<<<<< HEAD
 +      .fop_flags = FOP_UNSIGNED_OFFSET,
++=======
+       .show_fdinfo = drm_show_fdinfo,
++>>>>>>> drm-misc/drm-misc-next
  };
  
  #ifdef CONFIG_DEBUG_FS
Quoted 15 lines
Quoted 14 lines

 .../testing/sysfs-driver-panthor-profiling    |  10 +
 Documentation/gpu/panthor.rst                 |  46 +++
 drivers/gpu/drm/panthor/panthor_devfreq.c     |  18 +-
 drivers/gpu/drm/panthor/panthor_device.h      |  36 ++
 drivers/gpu/drm/panthor/panthor_drv.c         |  73 ++++
 drivers/gpu/drm/panthor/panthor_gem.c         |  12 +
 drivers/gpu/drm/panthor/panthor_sched.c       | 384 +++++++++++++++---
 drivers/gpu/drm/panthor/panthor_sched.h       |   2 +
 8 files changed, 531 insertions(+), 50 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-driver-panthor-profiling
 create mode 100644 Documentation/gpu/panthor.rst
Signature · 1 line
Jani Nikula, Intel
maintainer
#13 lore raw
Quoted 5 lines · On Wed, 02 Oct 2024 12:26:49 +0300 Jani Nikula <[email protected]>…
Quoted 2 lines · Boris Brezillon

Queued to drm-misc-next after applying the few modifications I mentioned. Also added Steve's ack (given on IRC) on the first patch.

Can we have the drm-tip rebuild conflict resolution too, please?

Oops, sorry. Should be good now.

Quoted 17 lines
drivers/gpu/drm/panthor/panthor_drv.c +4 -0
diff --cc drivers/gpu/drm/panthor/panthor_drv.c
index c520f156e2d7,f9b93f84d611..000000000000
@@@ -1383,7 -1476,7 +1476,11 @@@ static const struct file_operations pan
        .read = drm_read,
        .llseek = noop_llseek,
        .mmap = panthor_mmap,
++<<<<<<< HEAD
 +      .fop_flags = FOP_UNSIGNED_OFFSET,
++=======
+       .show_fdinfo = drm_show_fdinfo,
++>>>>>>> drm-misc/drm-misc-next  
  };
 
  #ifdef CONFIG_DEBUG_FS