Skip to contentSkip to conversation

Initial Xe driver submission

RFC PATCH In review Matthew Brost 14 participants 87 messages Dec 22, 2022 – Mar 9, 2023 Page 1 of 2

Subsystems (13)
lore
Share & tools
lore

Matthew Brost

author maintainer
#0 lore raw

cover Initial Xe driver submission

Hello,

This is a submission for Xe, a new driver for Intel GPUs that supports both integrated and discrete platforms starting with Tiger Lake (first platform with Intel Xe Architecture). The intention of this new driver is to have a fresh base to work from that is unencumbered by older platforms, whilst also taking the opportunity to rearchitect our driver to increase sharing across the drm subsystem, both leveraging and allowing us to contribute more towards other shared components like TTM and drm/scheduler. The memory model is based on VM bind which is similar to the i915 implementation. Likewise the execbuf implementation for Xe is very similar to execbuf3 in the i915 [1].

The code is at a stage where it is already functional and has experimental support for multiple platforms starting from Tiger Lake, with initial support implemented in Mesa (for Iris and Anv, our OpenGL and Vulkan drivers), as well as in NEO (for OpenCL and Level0). A Mesa MR has been posted [2] and NEO implementation will be released publicly early next year. We also have a suite of IGTs for XE that will appear on the IGT list shortly.

It has been built with the assumption of supporting multiple architectures from the get-go, right now with tests running both on X86 and ARM hosts. And we intend to continue working on it and improving on it as part of the kernel community upstream.

The new Xe driver leverages a lot from i915 and work on i915 continues as we ready Xe for production throughout 2023.

As for display, the intent is to share the display code with the i915 driver so that there is maximum reuse there. Currently this is being done by compiling the display code twice, but alternatives to that are under consideration and we want to have more discussion on what the best final solution will look like over the next few months. Right now, work is ongoing in refactoring the display codebase to remove as much as possible any unnecessary dependencies on i915 specific data structures there..

We currently have 2 submission backends, execlists and GuC. The execlist is meant mostly for testing and is not fully functional while GuC backend is fully functional. As with the i915 and GuC submission, in Xe the GuC firmware is required and should be placed in /lib/firmware/xe.

The GuC firmware can be found in the below location:
https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/i915

The easiest way to setup firmware is:
cp -r /lib/firmware/i915 /lib/firmware/xe

The code has been organized such that we have all patches that touch areas outside of drm/xe first for review, and then the actual new driver in a separate commit. The code which is outside of drm/xe is included in this RFC while drm/xe is not due to the size of the commit. The drm/xe is code is available in a public repo listed below.

Xe driver commit:
https://cgit.freedesktop.org/drm/drm-xe/commit/?h=drm-xe-next&id=9cb016ebbb6a275f57b1cb512b95d5a842391ad7

Xe kernel repo:
https://cgit.freedesktop.org/drm/drm-xe/

There's a lot of work still to happen on Xe but we're very excited about it and wanted to share it early and welcome feedback and discussion.

Cheers,
Matthew Brost

[1] https://patchwork.freedesktop.org/series/105879/
[2] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20418

Maarten Lankhorst (12):
drm/amd: Convert amdgpu to use suballocation helper. drm/radeon: Use the drm suballocation manager implementation. drm/i915: Remove gem and overlay frontbuffer tracking drm/i915/display: Neuter frontbuffer tracking harder drm/i915/display: Add more macros to remove all direct calls to uncore drm/i915/display: Remove all uncore mmio accesses in favor of intel_de drm/i915: Rename find_section to find_bdb_section drm/i915/regs: Set DISPLAY_MMIO_BASE to 0 for xe drm/i915/display: Fix a use-after-free when intel_edp_init_connector
fails
drm/i915/display: Remaining changes to make xe compile sound/hda: Allow XE as i915 replacement for sound mei/hdcp: Also enable for XE

Matthew Brost (5):
drm/sched: Convert drm scheduler to use a work queue rather than
kthread
drm/sched: Add generic scheduler message interface drm/sched: Start run wq before TDR in drm_sched_start drm/sched: Submit job before starting TDR
drm/sched: Add helper to set TDR timeout

Thomas Hellström (3):
drm/suballoc: Introduce a generic suballocation manager
drm: Add a gpu page-table walker helper
drm/ttm: Don't print error message if eviction was interrupted

 drivers/gpu/drm/Kconfig                       |   5 +
 drivers/gpu/drm/Makefile                      |   4 +
 drivers/gpu/drm/amd/amdgpu/Kconfig            |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu.h           |  26 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c   |  14 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |  12 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c        |   5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  23 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h      |   3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_sa.c        | 320 +-----------------
 drivers/gpu/drm/drm_pt_walk.c                 | 159 +++++++++
 drivers/gpu/drm/drm_suballoc.c                | 301 ++++++++++++++++
 drivers/gpu/drm/i915/Makefile                 |   2 +-
 drivers/gpu/drm/i915/display/hsw_ips.c        |   7 +-
 drivers/gpu/drm/i915/display/i9xx_plane.c     |   1 +
 drivers/gpu/drm/i915/display/intel_atomic.c   |   2 +
 .../gpu/drm/i915/display/intel_atomic_plane.c |  25 +-
 .../gpu/drm/i915/display/intel_backlight.c    |   2 +-
 drivers/gpu/drm/i915/display/intel_bios.c     |  71 ++--
 drivers/gpu/drm/i915/display/intel_bw.c       |  36 +-
 drivers/gpu/drm/i915/display/intel_cdclk.c    |  68 ++--
 drivers/gpu/drm/i915/display/intel_color.c    |   1 +
 drivers/gpu/drm/i915/display/intel_crtc.c     |  14 +-
 drivers/gpu/drm/i915/display/intel_cursor.c   |  14 +-
 drivers/gpu/drm/i915/display/intel_de.h       |  38 +++
 drivers/gpu/drm/i915/display/intel_display.c  | 155 +++++++--
 drivers/gpu/drm/i915/display/intel_display.h  |   9 +-
 .../gpu/drm/i915/display/intel_display_core.h |   5 +-
 .../drm/i915/display/intel_display_debugfs.c  |   8 +
 .../drm/i915/display/intel_display_power.c    |  40 ++-
 .../drm/i915/display/intel_display_power.h    |   6 +
 .../i915/display/intel_display_power_map.c    |   7 +
 .../i915/display/intel_display_power_well.c   |  24 +-
 .../drm/i915/display/intel_display_reg_defs.h |   4 +
 .../drm/i915/display/intel_display_trace.h    |   6 +
 .../drm/i915/display/intel_display_types.h    |  32 +-
 drivers/gpu/drm/i915/display/intel_dmc.c      |  17 +-
 drivers/gpu/drm/i915/display/intel_dp.c       |  11 +-
 drivers/gpu/drm/i915/display/intel_dp_aux.c   |   6 +
 drivers/gpu/drm/i915/display/intel_dpio_phy.c |   9 +-
 drivers/gpu/drm/i915/display/intel_dpio_phy.h |  15 +
 drivers/gpu/drm/i915/display/intel_dpll.c     |   8 +-
 drivers/gpu/drm/i915/display/intel_dpll_mgr.c |   4 +
 drivers/gpu/drm/i915/display/intel_drrs.c     |   1 +
 drivers/gpu/drm/i915/display/intel_dsb.c      | 124 +++++--
 drivers/gpu/drm/i915/display/intel_dsi_vbt.c  |  26 +-
 drivers/gpu/drm/i915/display/intel_fb.c       | 108 ++++--
 drivers/gpu/drm/i915/display/intel_fb_pin.c   |   6 -
 drivers/gpu/drm/i915/display/intel_fbc.c      |  49 ++-
 drivers/gpu/drm/i915/display/intel_fbdev.c    | 108 +++++-
 .../gpu/drm/i915/display/intel_frontbuffer.c  | 103 +-----
 .../gpu/drm/i915/display/intel_frontbuffer.h  |  67 +---
 drivers/gpu/drm/i915/display/intel_gmbus.c    |   2 +-
 drivers/gpu/drm/i915/display/intel_hdcp.c     |   9 +-
 drivers/gpu/drm/i915/display/intel_hdmi.c     |   1 -
 .../gpu/drm/i915/display/intel_lpe_audio.h    |   8 +
 .../drm/i915/display/intel_modeset_setup.c    |  11 +-
 drivers/gpu/drm/i915/display/intel_opregion.c |   2 +-
 drivers/gpu/drm/i915/display/intel_overlay.c  |  14 -
 .../gpu/drm/i915/display/intel_pch_display.h  |  16 +
 .../gpu/drm/i915/display/intel_pch_refclk.h   |   8 +
 drivers/gpu/drm/i915/display/intel_pipe_crc.c |   1 +
 .../drm/i915/display/intel_plane_initial.c    |   3 +-
 drivers/gpu/drm/i915/display/intel_psr.c      |   1 +
 drivers/gpu/drm/i915/display/intel_sprite.c   |  21 ++
 drivers/gpu/drm/i915/display/intel_vbt_defs.h |   2 +-
 drivers/gpu/drm/i915/display/intel_vga.c      |   5 +
 drivers/gpu/drm/i915/display/skl_scaler.c     |   2 +
 .../drm/i915/display/skl_universal_plane.c    |  52 ++-
 drivers/gpu/drm/i915/display/skl_watermark.c  |  25 +-
 drivers/gpu/drm/i915/gem/i915_gem_clflush.c   |   4 -
 drivers/gpu/drm/i915/gem/i915_gem_domain.c    |   7 -
 .../gpu/drm/i915/gem/i915_gem_execbuffer.c    |   2 -
 drivers/gpu/drm/i915/gem/i915_gem_object.c    |  25 --
 drivers/gpu/drm/i915/gem/i915_gem_object.h    |  22 --
 drivers/gpu/drm/i915/gem/i915_gem_phys.c      |   4 -
 drivers/gpu/drm/i915/gt/intel_gt_regs.h       |   3 +-
 drivers/gpu/drm/i915/i915_driver.c            |   1 +
 drivers/gpu/drm/i915/i915_gem.c               |   8 -
 drivers/gpu/drm/i915/i915_gem_gtt.c           |   1 -
 drivers/gpu/drm/i915/i915_reg_defs.h          |   8 +
 drivers/gpu/drm/i915/i915_vma.c               |  12 -
 drivers/gpu/drm/radeon/radeon.h               |  55 +--
 drivers/gpu/drm/radeon/radeon_ib.c            |  12 +-
 drivers/gpu/drm/radeon/radeon_object.h        |  25 +-
 drivers/gpu/drm/radeon/radeon_sa.c            | 314 ++---------------
 drivers/gpu/drm/radeon/radeon_semaphore.c     |   6 +-
 drivers/gpu/drm/scheduler/sched_main.c        | 182 +++++++---
 drivers/gpu/drm/ttm/ttm_bo.c                  |   3 +-
 drivers/misc/mei/hdcp/Kconfig                 |   2 +-
 drivers/misc/mei/hdcp/mei_hdcp.c              |   3 +-
 include/drm/drm_pt_walk.h                     | 161 +++++++++
 include/drm/drm_suballoc.h                    | 112 ++++++
 include/drm/gpu_scheduler.h                   |  41 ++-
 sound/hda/hdac_i915.c                         |  17 +-
 sound/pci/hda/hda_intel.c                     |  56 +--
 sound/soc/intel/avs/core.c                    |  13 +-
 sound/soc/sof/intel/hda.c                     |   7 +-
 98 files changed, 2076 insertions(+), 1325 deletions(-)
 create mode 100644 drivers/gpu/drm/drm_pt_walk.c
 create mode 100644 drivers/gpu/drm/drm_suballoc.c
 create mode 100644 include/drm/drm_pt_walk.h
 create mode 100644 include/drm/drm_suballoc.h
Signature · 1 line
2.37.3
author maintainer
#1 lore raw

RFC PATCH 1/20 drm/suballoc: Introduce a generic suballocation manager

From: Thomas Hellström <[email protected]>

Initially we tried to leverage the amdgpu suballocation manager. It turnes out, however, that it tries extremely hard not to enable signalling on the fences that hold the memory up for freeing, which makes it hard to understand and to fix potential issues with it.

So in a simplification effort, introduce a drm suballocation manager as a wrapper around an existing allocator (drm_mm) and to avoid using queues for freeing, thus avoiding throttling on free which is an undesired feature as typically the throttling needs to be done uninterruptible.

This variant is probably more cpu-hungry but can be improved at the cost of additional complexity. Ideas for that are documented in the drm_suballoc.c file.

---
 drivers/gpu/drm/Kconfig        |   4 +
 drivers/gpu/drm/Makefile       |   3 +
 drivers/gpu/drm/drm_suballoc.c | 301 +++++++++++++++++++++++++++++++++
 include/drm/drm_suballoc.h     | 112 ++++++++++++
 4 files changed, 420 insertions(+)
 create mode 100644 drivers/gpu/drm/drm_suballoc.c
 create mode 100644 include/drm/drm_suballoc.h
drivers/gpu/drm/Kconfig +4 -0
index 663ea8f9966d..ad231a68c2a5 100644
@@ -233,6 +233,10 @@ config DRM_GEM_SHMEM_HELPER
 	help
 	  Choose this if you need the GEM shmem helper functions
 
+config DRM_SUBALLOC_HELPER
+	tristate
+	depends on DRM
+
 config DRM_SCHED
 	tristate
 	depends on DRM
drivers/gpu/drm/Makefile +3 -0
index 496fa5a6147a..23ad760884b2 100644
@@ -88,6 +88,9 @@ obj-$(CONFIG_DRM_GEM_DMA_HELPER) += drm_dma_helper.o
 drm_shmem_helper-y := drm_gem_shmem_helper.o
 obj-$(CONFIG_DRM_GEM_SHMEM_HELPER) += drm_shmem_helper.o
 
+drm_suballoc_helper-y := drm_suballoc.o
+obj-$(CONFIG_DRM_SUBALLOC_HELPER) += drm_suballoc_helper.o
+
 drm_vram_helper-y := drm_gem_vram_helper.o
 obj-$(CONFIG_DRM_VRAM_HELPER) += drm_vram_helper.o
 
drivers/gpu/drm/drm_suballoc.c new file +301 -0

301 lines not on this page · raw message

include/drm/drm_suballoc.h new file +112 -0

112 lines not on this page · raw message

Signature · 1 line
2.37.3
author maintainer
#2 lore raw

RFC PATCH 2/20 drm/amd: Convert amdgpu to use suballocation helper.

In reply to Matthew Brost

From: Maarten Lankhorst <[email protected]>

Now that we have a generic suballocation helper, Use it in amdgpu. The debug output is slightly different and suballocation may be slightly more cpu-hungry.

---
 drivers/gpu/drm/Kconfig                    |   1 +
 drivers/gpu/drm/amd/amdgpu/Kconfig         |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  26 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c     |   5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |  23 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h   |   3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_sa.c     | 320 ++-------------------
 7 files changed, 43 insertions(+), 336 deletions(-)
drivers/gpu/drm/Kconfig +1 -0
index ad231a68c2a5..de45c3c059f0 100644
@@ -78,6 +78,7 @@ config DRM_KUNIT_TEST
 	select DRM_DISPLAY_HELPER
 	select DRM_LIB_RANDOM
 	select DRM_KMS_HELPER
+	select DRM_SUBALLOC_HELPER
 	select DRM_BUDDY
 	select DRM_EXPORT_FOR_TESTS if m
 	select DRM_KUNIT_TEST_HELPERS
drivers/gpu/drm/amd/amdgpu/Kconfig +1 -0
index 5fcd510f1abb..eef179b81d0f 100644
@@ -16,6 +16,7 @@ config DRM_AMDGPU
 	select BACKLIGHT_CLASS_DEVICE
 	select INTERVAL_TREE
 	select DRM_BUDDY
+	select DRM_SUBALLOC_HELPER
 	# amdgpu depends on ACPI_VIDEO when ACPI is enabled, for select to work
 	# ACPI_VIDEO's dependencies must also be selected.
 	select INPUT if ACPI
drivers/gpu/drm/amd/amdgpu/amdgpu.h +4 -22
index 2644cd991210..009903e21d83 100644
@@ -422,29 +422,11 @@ struct amdgpu_clock {
  * alignment).
  */
 
-#define AMDGPU_SA_NUM_FENCE_LISTS	32
-
 struct amdgpu_sa_manager {
-	wait_queue_head_t	wq;
-	struct amdgpu_bo	*bo;
-	struct list_head	*hole;
-	struct list_head	flist[AMDGPU_SA_NUM_FENCE_LISTS];
-	struct list_head	olist;
-	unsigned		size;
-	uint64_t		gpu_addr;
-	void			*cpu_ptr;
-	uint32_t		domain;
-	uint32_t		align;
-};
-
-/* sub-allocation buffer */
-struct amdgpu_sa_bo {
-	struct list_head		olist;
-	struct list_head		flist;
-	struct amdgpu_sa_manager	*manager;
-	unsigned			soffset;
-	unsigned			eoffset;
-	struct dma_fence	        *fence;
+	struct drm_suballoc_manager	base;
+	struct amdgpu_bo		*bo;
+	uint64_t			gpu_addr;
+	void				*cpu_ptr;
 };
 
 int amdgpu_fence_slab_init(void);
drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c +2 -3
index bcccc348dbe2..5621b63c7f42 100644
@@ -69,7 +69,7 @@ int amdgpu_ib_get(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 
 	if (size) {
 		r = amdgpu_sa_bo_new(&adev->ib_pools[pool_type],
-				      &ib->sa_bo, size, 256);
+				      &ib->sa_bo, size);
 		if (r) {
 			dev_err(adev->dev, "failed to get a new IB (%d)\n", r);
 			return r;
@@ -309,8 +309,7 @@ int amdgpu_ib_pool_init(struct amdgpu_device *adev)
 
 	for (i = 0; i < AMDGPU_IB_POOL_MAX; i++) {
 		r = amdgpu_sa_bo_manager_init(adev, &adev->ib_pools[i],
-					      AMDGPU_IB_POOL_SIZE,
-					      AMDGPU_GPU_PAGE_SIZE,
+					      AMDGPU_IB_POOL_SIZE, 256,
 					      AMDGPU_GEM_DOMAIN_GTT);
 		if (r)
 			goto error;
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +15 -8
index 93207badf83f..568baf15d5b1 100644
@@ -336,15 +336,22 @@ uint32_t amdgpu_bo_get_preferred_domain(struct amdgpu_device *adev,
 /*
  * sub allocation
  */
+static inline struct amdgpu_sa_manager *
+to_amdgpu_sa_manager(struct drm_suballoc_manager *manager)
+{
+	return container_of(manager, struct amdgpu_sa_manager, base);
+}
 
-static inline uint64_t amdgpu_sa_bo_gpu_addr(struct amdgpu_sa_bo *sa_bo)
+static inline uint64_t amdgpu_sa_bo_gpu_addr(struct drm_suballoc *sa_bo)
 {
-	return sa_bo->manager->gpu_addr + sa_bo->soffset;
+	return to_amdgpu_sa_manager(sa_bo->manager)->gpu_addr +
+		drm_suballoc_soffset(sa_bo);
 }
 
-static inline void * amdgpu_sa_bo_cpu_addr(struct amdgpu_sa_bo *sa_bo)
+static inline void * amdgpu_sa_bo_cpu_addr(struct drm_suballoc *sa_bo)
 {
-	return sa_bo->manager->cpu_ptr + sa_bo->soffset;
+	return to_amdgpu_sa_manager(sa_bo->manager)->cpu_ptr +
+		drm_suballoc_soffset(sa_bo);
 }
 
 int amdgpu_sa_bo_manager_init(struct amdgpu_device *adev,
@@ -355,11 +362,11 @@ void amdgpu_sa_bo_manager_fini(struct amdgpu_device *adev,
 int amdgpu_sa_bo_manager_start(struct amdgpu_device *adev,
 				      struct amdgpu_sa_manager *sa_manager);
 int amdgpu_sa_bo_new(struct amdgpu_sa_manager *sa_manager,
-		     struct amdgpu_sa_bo **sa_bo,
-		     unsigned size, unsigned align);
+		     struct drm_suballoc **sa_bo,
+		     unsigned size);
 void amdgpu_sa_bo_free(struct amdgpu_device *adev,
-			      struct amdgpu_sa_bo **sa_bo,
-			      struct dma_fence *fence);
+		       struct drm_suballoc **sa_bo,
+		       struct dma_fence *fence);
 #if defined(CONFIG_DEBUG_FS)
 void amdgpu_sa_bo_dump_debug_info(struct amdgpu_sa_manager *sa_manager,
 					 struct seq_file *m);
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h +2 -1
index f752c7ae7f60..6dd2a3b7e434 100644
@@ -27,6 +27,7 @@
 #include <drm/amdgpu_drm.h>
 #include <drm/gpu_scheduler.h>
 #include <drm/drm_print.h>
+#include <drm/drm_suballoc.h>
 
 struct amdgpu_device;
 struct amdgpu_ring;
@@ -92,7 +93,7 @@ enum amdgpu_ib_pool_type {
 };
 
 struct amdgpu_ib {
-	struct amdgpu_sa_bo		*sa_bo;
+	struct drm_suballoc		*sa_bo;
 	uint32_t			length_dw;
 	uint64_t			gpu_addr;
 	uint32_t			*ptr;
drivers/gpu/drm/amd/amdgpu/amdgpu_sa.c +18 -302

371 lines not on this page · raw message

Signature · 1 line
2.37.3
author maintainer
#3 lore raw

RFC PATCH 3/20 drm/radeon: Use the drm suballocation manager implementation.

In reply to Matthew Brost

From: Maarten Lankhorst <[email protected]>

Use the generic suballocation helper.
Note that the generic suballocator only allows a single alignment, so we may waste a few more bytes for radeon_semaphore, shouldn't be a big deal, could be re-added if needed. Also, similar to amdgpu, debug output changes slightly and suballocator cpu usage may be slightly higher.

---
 drivers/gpu/drm/radeon/radeon.h           |  55 +---
 drivers/gpu/drm/radeon/radeon_ib.c        |  12 +-
 drivers/gpu/drm/radeon/radeon_object.h    |  25 +-
 drivers/gpu/drm/radeon/radeon_sa.c        | 314 ++--------------------
 drivers/gpu/drm/radeon/radeon_semaphore.c |   6 +-
 5 files changed, 55 insertions(+), 357 deletions(-)
drivers/gpu/drm/radeon/radeon.h +8 -47
index 57e20780a458..d19a4b1c1a8f 100644
@@ -79,6 +79,7 @@
 
 #include <drm/drm_gem.h>
 #include <drm/drm_audio_component.h>
+#include <drm/drm_suballoc.h>
 
 #include "radeon_family.h"
 #include "radeon_mode.h"
@@ -511,52 +512,12 @@ struct radeon_bo {
 };
 #define gem_to_radeon_bo(gobj) container_of((gobj), struct radeon_bo, tbo.base)
 
-/* sub-allocation manager, it has to be protected by another lock.
- * By conception this is an helper for other part of the driver
- * like the indirect buffer or semaphore, which both have their
- * locking.
- *
- * Principe is simple, we keep a list of sub allocation in offset
- * order (first entry has offset == 0, last entry has the highest
- * offset).
- *
- * When allocating new object we first check if there is room at
- * the end total_size - (last_object_offset + last_object_size) >=
- * alloc_size. If so we allocate new object there.
- *
- * When there is not enough room at the end, we start waiting for
- * each sub object until we reach object_offset+object_size >=
- * alloc_size, this object then become the sub object we return.
- *
- * Alignment can't be bigger than page size.
- *
- * Hole are not considered for allocation to keep things simple.
- * Assumption is that there won't be hole (all object on same
- * alignment).
- */
 struct radeon_sa_manager {
-	wait_queue_head_t	wq;
-	struct radeon_bo	*bo;
-	struct list_head	*hole;
-	struct list_head	flist[RADEON_NUM_RINGS];
-	struct list_head	olist;
-	unsigned		size;
-	uint64_t		gpu_addr;
-	void			*cpu_ptr;
-	uint32_t		domain;
-	uint32_t		align;
-};
-
-struct radeon_sa_bo;
-
-/* sub-allocation buffer */
-struct radeon_sa_bo {
-	struct list_head		olist;
-	struct list_head		flist;
-	struct radeon_sa_manager	*manager;
-	unsigned			soffset;
-	unsigned			eoffset;
-	struct radeon_fence		*fence;
+	struct drm_suballoc_manager	base;
+	struct radeon_bo		*bo;
+	uint64_t			gpu_addr;
+	void				*cpu_ptr;
+	u32 domain;
 };
 
 /*
@@ -587,7 +548,7 @@ int radeon_mode_dumb_mmap(struct drm_file *filp,
  * Semaphores.
  */
 struct radeon_semaphore {
-	struct radeon_sa_bo	*sa_bo;
+	struct drm_suballoc	*sa_bo;
 	signed			waiters;
 	uint64_t		gpu_addr;
 };
@@ -816,7 +777,7 @@ void radeon_irq_kms_disable_hpd(struct radeon_device *rdev, unsigned hpd_mask);
  */
 
 struct radeon_ib {
-	struct radeon_sa_bo		*sa_bo;
+	struct drm_suballoc		*sa_bo;
 	uint32_t			length_dw;
 	uint64_t			gpu_addr;
 	uint32_t			*ptr;
drivers/gpu/drm/radeon/radeon_ib.c +5 -7
index 62b116727b4f..63fcfe65d814 100644
@@ -61,7 +61,7 @@ int radeon_ib_get(struct radeon_device *rdev, int ring,
 {
 	int r;
 
-	r = radeon_sa_bo_new(rdev, &rdev->ring_tmp_bo, &ib->sa_bo, size, 256);
+	r = radeon_sa_bo_new(&rdev->ring_tmp_bo, &ib->sa_bo, size);
 	if (r) {
 		dev_err(rdev->dev, "failed to get a new IB (%d)\n", r);
 		return r;
@@ -77,7 +77,7 @@ int radeon_ib_get(struct radeon_device *rdev, int ring,
 		/* ib pool is bound at RADEON_VA_IB_OFFSET in virtual address
 		 * space and soffset is the offset inside the pool bo
 		 */
-		ib->gpu_addr = ib->sa_bo->soffset + RADEON_VA_IB_OFFSET;
+		ib->gpu_addr = drm_suballoc_soffset(ib->sa_bo) + RADEON_VA_IB_OFFSET;
 	} else {
 		ib->gpu_addr = radeon_sa_bo_gpu_addr(ib->sa_bo);
 	}
@@ -97,7 +97,7 @@ int radeon_ib_get(struct radeon_device *rdev, int ring,
 void radeon_ib_free(struct radeon_device *rdev, struct radeon_ib *ib)
 {
 	radeon_sync_free(rdev, &ib->sync, ib->fence);
-	radeon_sa_bo_free(rdev, &ib->sa_bo, ib->fence);
+	radeon_sa_bo_free(&ib->sa_bo, ib->fence);
 	radeon_fence_unref(&ib->fence);
 }
 
@@ -201,8 +201,7 @@ int radeon_ib_pool_init(struct radeon_device *rdev)
 
 	if (rdev->family >= CHIP_BONAIRE) {
 		r = radeon_sa_bo_manager_init(rdev, &rdev->ring_tmp_bo,
-					      RADEON_IB_POOL_SIZE*64*1024,
-					      RADEON_GPU_PAGE_SIZE,
+					      RADEON_IB_POOL_SIZE*64*1024, 256,
 					      RADEON_GEM_DOMAIN_GTT,
 					      RADEON_GEM_GTT_WC);
 	} else {
@@ -210,8 +209,7 @@ int radeon_ib_pool_init(struct radeon_device *rdev)
 		 * to the command stream checking
 		 */
 		r = radeon_sa_bo_manager_init(rdev, &rdev->ring_tmp_bo,
-					      RADEON_IB_POOL_SIZE*64*1024,
-					      RADEON_GPU_PAGE_SIZE,
+					      RADEON_IB_POOL_SIZE*64*1024, 256,
 					      RADEON_GEM_DOMAIN_GTT, 0);
 	}
 	if (r) {
drivers/gpu/drm/radeon/radeon_object.h +15 -10
index 0a6ef49e990a..b7c5087a7dbc 100644
@@ -169,15 +169,22 @@ extern void radeon_bo_fence(struct radeon_bo *bo, struct radeon_fence *fence,
 /*
  * sub allocation
  */
+static inline struct radeon_sa_manager *
+to_radeon_sa_manager(struct drm_suballoc_manager *manager)
+{
+	return container_of(manager, struct radeon_sa_manager, base);
+}
 
-static inline uint64_t radeon_sa_bo_gpu_addr(struct radeon_sa_bo *sa_bo)
+static inline uint64_t radeon_sa_bo_gpu_addr(struct drm_suballoc *sa_bo)
 {
-	return sa_bo->manager->gpu_addr + sa_bo->soffset;
+	return to_radeon_sa_manager(sa_bo->manager)->gpu_addr +
+		drm_suballoc_soffset(sa_bo);
 }
 
-static inline void * radeon_sa_bo_cpu_addr(struct radeon_sa_bo *sa_bo)
+static inline void * radeon_sa_bo_cpu_addr(struct drm_suballoc *sa_bo)
 {
-	return sa_bo->manager->cpu_ptr + sa_bo->soffset;
+	return to_radeon_sa_manager(sa_bo->manager)->cpu_ptr +
+		drm_suballoc_soffset(sa_bo);
 }
 
 extern int radeon_sa_bo_manager_init(struct radeon_device *rdev,
@@ -190,12 +197,10 @@ extern int radeon_sa_bo_manager_start(struct radeon_device *rdev,
 				      struct radeon_sa_manager *sa_manager);
 extern int radeon_sa_bo_manager_suspend(struct radeon_device *rdev,
 					struct radeon_sa_manager *sa_manager);
-extern int radeon_sa_bo_new(struct radeon_device *rdev,
-			    struct radeon_sa_manager *sa_manager,
-			    struct radeon_sa_bo **sa_bo,
-			    unsigned size, unsigned align);
-extern void radeon_sa_bo_free(struct radeon_device *rdev,
-			      struct radeon_sa_bo **sa_bo,
+extern int radeon_sa_bo_new(struct radeon_sa_manager *sa_manager,
+			    struct drm_suballoc **sa_bo,
+			    unsigned size);
+extern void radeon_sa_bo_free(struct drm_suballoc **sa_bo,
 			      struct radeon_fence *fence);
 #if defined(CONFIG_DEBUG_FS)
 extern void radeon_sa_bo_dump_debug_info(struct radeon_sa_manager *sa_manager,
drivers/gpu/drm/radeon/radeon_sa.c +24 -290

362 lines not on this page · raw message

drivers/gpu/drm/radeon/radeon_semaphore.c +3 -3
index 221e59476f64..3e2b0bf0d55d 100644
@@ -40,8 +40,8 @@ int radeon_semaphore_create(struct radeon_device *rdev,
 	if (*semaphore == NULL) {
 		return -ENOMEM;
 	}
-	r = radeon_sa_bo_new(rdev, &rdev->ring_tmp_bo,
-			     &(*semaphore)->sa_bo, 8, 8);
+	r = radeon_sa_bo_new(&rdev->ring_tmp_bo,
+			     &(*semaphore)->sa_bo, 8);
 	if (r) {
 		kfree(*semaphore);
 		*semaphore = NULL;
@@ -100,7 +100,7 @@ void radeon_semaphore_free(struct radeon_device *rdev,
 		dev_err(rdev->dev, "semaphore %p has more waiters than signalers,"
 			" hardware lockup imminent!\n", *semaphore);
 	}
-	radeon_sa_bo_free(rdev, &(*semaphore)->sa_bo, fence);
+	radeon_sa_bo_free(&(*semaphore)->sa_bo, fence);
 	kfree(*semaphore);
 	*semaphore = NULL;
 }
Signature · 1 line
2.37.3
author maintainer
#4 lore raw

RFC PATCH 4/20 drm/sched: Convert drm scheduler to use a work queue rather than kthread

In reply to Matthew Brost

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

2. In XE submissions are done via programming a ring buffer (circular buffer), a drm_gpu_scheduler provides a limit on number of jobs, if the limit of number jobs is set to RING_SIZE / MAX_SIZE_PER_JOB we get flow control on the ring for free.

A problem with this design is currently a drm_gpu_scheduler uses a kthread for submission / job cleanup. This doesn't scale if a large number of drm_gpu_scheduler are used. To work around the scaling issue, use a worker rather than kthread for submission / job cleanup.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  14 +--
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  12 +-
 drivers/gpu/drm/scheduler/sched_main.c      | 124 ++++++++++++--------
 include/drm/gpu_scheduler.h                 |  13 +-
 4 files changed, 93 insertions(+), 70 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +7 -7
index f60753f97ac5..9c2a10aeb0b3 100644
@@ -1489,9 +1489,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused)
 	for (i = 0; i < AMDGPU_MAX_RINGS; i++) {
 		struct amdgpu_ring *ring = adev->rings[i];
 
-		if (!ring || !ring->sched.thread)
+		if (!ring || !ring->sched.ready)
 			continue;
-		kthread_park(ring->sched.thread);
+		drm_sched_run_wq_stop(&ring->sched);
 	}
 
 	seq_printf(m, "run ib test:\n");
@@ -1505,9 +1505,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused)
 	for (i = 0; i < AMDGPU_MAX_RINGS; i++) {
 		struct amdgpu_ring *ring = adev->rings[i];
 
-		if (!ring || !ring->sched.thread)
+		if (!ring || !ring->sched.ready)
 			continue;
-		kthread_unpark(ring->sched.thread);
+		drm_sched_run_wq_start(&ring->sched);
 	}
 
 	up_write(&adev->reset_domain->sem);
@@ -1727,7 +1727,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
 
 	ring = adev->rings[val];
 
-	if (!ring || !ring->funcs->preempt_ib || !ring->sched.thread)
+	if (!ring || !ring->funcs->preempt_ib || !ring->sched.ready)
 		return -EINVAL;
 
 	/* the last preemption failed */
@@ -1745,7 +1745,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
 		goto pro_end;
 
 	/* stop the scheduler */
-	kthread_park(ring->sched.thread);
+	drm_sched_run_wq_stop(&ring->sched);
 
 	/* preempt the IB */
 	r = amdgpu_ring_preempt_ib(ring);
@@ -1779,7 +1779,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
 
 failure:
 	/* restart the scheduler */
-	kthread_unpark(ring->sched.thread);
+	drm_sched_run_wq_start(&ring->sched);
 
 	up_read(&adev->reset_domain->sem);
 
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +6 -6
index 076ae400d099..9552929ccf87 100644
@@ -4577,7 +4577,7 @@ bool amdgpu_device_has_job_running(struct amdgpu_device *adev)
 	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 		struct amdgpu_ring *ring = adev->rings[i];
 
-		if (!ring || !ring->sched.thread)
+		if (!ring || !ring->sched.ready)
 			continue;
 
 		spin_lock(&ring->sched.job_list_lock);
@@ -4708,7 +4708,7 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
 	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 		struct amdgpu_ring *ring = adev->rings[i];
 
-		if (!ring || !ring->sched.thread)
+		if (!ring || !ring->sched.ready)
 			continue;
 
 		/*clear job fence from fence drv to avoid force_completion
@@ -5247,7 +5247,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
 		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 			struct amdgpu_ring *ring = tmp_adev->rings[i];
 
-			if (!ring || !ring->sched.thread)
+			if (!ring || !ring->sched.ready)
 				continue;
 
 			drm_sched_stop(&ring->sched, job ? &job->base : NULL);
@@ -5321,7 +5321,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
 		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 			struct amdgpu_ring *ring = tmp_adev->rings[i];
 
-			if (!ring || !ring->sched.thread)
+			if (!ring || !ring->sched.ready)
 				continue;
 
 			drm_sched_start(&ring->sched, true);
@@ -5648,7 +5648,7 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta
 		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 			struct amdgpu_ring *ring = adev->rings[i];
 
-			if (!ring || !ring->sched.thread)
+			if (!ring || !ring->sched.ready)
 				continue;
 
 			drm_sched_stop(&ring->sched, NULL);
@@ -5776,7 +5776,7 @@ void amdgpu_pci_resume(struct pci_dev *pdev)
 	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 		struct amdgpu_ring *ring = adev->rings[i];
 
-		if (!ring || !ring->sched.thread)
+		if (!ring || !ring->sched.ready)
 			continue;
 
 		drm_sched_start(&ring->sched, true);
drivers/gpu/drm/scheduler/sched_main.c +72 -52

230 lines not on this page · raw message

include/drm/gpu_scheduler.h +8 -5
index ca857ec9e7eb..ff50f3c289cd 100644
@@ -456,17 +456,16 @@ struct drm_sched_backend_ops {
  * @timeout: the time after which a job is removed from the scheduler.
  * @name: name of the ring for which this scheduler is being used.
  * @sched_rq: priority wise array of run queues.
- * @wake_up_worker: the wait queue on which the scheduler sleeps until a job
- *                  is ready to be scheduled.
  * @job_scheduled: once @drm_sched_entity_do_release is called the scheduler
  *                 waits on this wait queue until all the scheduled jobs are
  *                 finished.
  * @hw_rq_count: the number of jobs currently in the hardware queue.
  * @job_id_count: used to assign unique id to the each job.
+ * @run_wq: workqueue used to queue @work_run
  * @timeout_wq: workqueue used to queue @work_tdr
+ * @work_run: schedules jobs and cleans up entities
  * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the
  *            timeout interval is over.
- * @thread: the kthread on which the scheduler which run.
  * @pending_list: the list of jobs which are currently in the job queue.
  * @job_list_lock: lock to protect the pending_list.
  * @hang_limit: once the hangs by a job crosses this limit then it is marked
@@ -475,6 +474,7 @@ struct drm_sched_backend_ops {
  * @_score: score used when the driver doesn't provide one
  * @ready: marks if the underlying HW is ready to work
  * @free_guilty: A hit to time out handler to free the guilty job.
+ * @pause_run_wq: pause queuing of @work_run on @run_wq
  * @dev: system &struct device
  *
  * One scheduler is implemented for each hardware ring.
@@ -485,13 +485,13 @@ struct drm_gpu_scheduler {
 	long				timeout;
 	const char			*name;
 	struct drm_sched_rq		sched_rq[DRM_SCHED_PRIORITY_COUNT];
-	wait_queue_head_t		wake_up_worker;
 	wait_queue_head_t		job_scheduled;
 	atomic_t			hw_rq_count;
 	atomic64_t			job_id_count;
+	struct workqueue_struct		*run_wq;
 	struct workqueue_struct		*timeout_wq;
+	struct work_struct		work_run;
 	struct delayed_work		work_tdr;
-	struct task_struct		*thread;
 	struct list_head		pending_list;
 	spinlock_t			job_list_lock;
 	int				hang_limit;
@@ -499,6 +499,7 @@ struct drm_gpu_scheduler {
 	atomic_t                        _score;
 	bool				ready;
 	bool				free_guilty;
+	bool				pause_run_wq;
 	struct device			*dev;
 };
 
@@ -529,6 +530,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity,
 
 void drm_sched_job_cleanup(struct drm_sched_job *job);
 void drm_sched_wakeup(struct drm_gpu_scheduler *sched);
+void drm_sched_run_wq_stop(struct drm_gpu_scheduler *sched);
+void drm_sched_run_wq_start(struct drm_gpu_scheduler *sched);
 void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad);
 void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery);
 void drm_sched_resubmit_jobs(struct drm_gpu_scheduler *sched);
Signature · 1 line
2.37.3
#5 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 18 lines · Matthew Brost

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

2. In XE submissions are done via programming a ring buffer (circular buffer), a drm_gpu_scheduler provides a limit on number of jobs, if the limit of number jobs is set to RING_SIZE / MAX_SIZE_PER_JOB we get flow control on the ring for free.

Quoted 4 lines

A problem with this design is currently a drm_gpu_scheduler uses a kthread for submission / job cleanup. This doesn't scale if a large number of drm_gpu_scheduler are used. To work around the scaling issue, use a worker rather than kthread for submission / job cleanup.

You might want to enable CONFIG_DRM_MSM in your kconfig, I think you missed a part

BR,
-R

Quoted 433 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  14 +--
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  12 +-
 drivers/gpu/drm/scheduler/sched_main.c      | 124 ++++++++++++--------
 include/drm/gpu_scheduler.h                 |  13 +-
 4 files changed, 93 insertions(+), 70 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +7 -7
index f60753f97ac5..9c2a10aeb0b3 100644
@@ -1489,9 +1489,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused)
        for (i = 0; i < AMDGPU_MAX_RINGS; i++) {
                struct amdgpu_ring *ring = adev->rings[i];
 
-               if (!ring || !ring->sched.thread)
+               if (!ring || !ring->sched.ready)
                        continue;
-               kthread_park(ring->sched.thread);
+               drm_sched_run_wq_stop(&ring->sched);
        }
 
        seq_printf(m, "run ib test:\n");
@@ -1505,9 +1505,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused)
        for (i = 0; i < AMDGPU_MAX_RINGS; i++) {
                struct amdgpu_ring *ring = adev->rings[i];
 
-               if (!ring || !ring->sched.thread)
+               if (!ring || !ring->sched.ready)
                        continue;
-               kthread_unpark(ring->sched.thread);
+               drm_sched_run_wq_start(&ring->sched);
        }
 
        up_write(&adev->reset_domain->sem);
@@ -1727,7 +1727,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
 
        ring = adev->rings[val];
 
-       if (!ring || !ring->funcs->preempt_ib || !ring->sched.thread)
+       if (!ring || !ring->funcs->preempt_ib || !ring->sched.ready)
                return -EINVAL;
 
        /* the last preemption failed */
@@ -1745,7 +1745,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
                goto pro_end;
 
        /* stop the scheduler */
-       kthread_park(ring->sched.thread);
+       drm_sched_run_wq_stop(&ring->sched);
 
        /* preempt the IB */
        r = amdgpu_ring_preempt_ib(ring);
@@ -1779,7 +1779,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
 
 failure:
        /* restart the scheduler */
-       kthread_unpark(ring->sched.thread);
+       drm_sched_run_wq_start(&ring->sched);
 
        up_read(&adev->reset_domain->sem);
 
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +6 -6
index 076ae400d099..9552929ccf87 100644
@@ -4577,7 +4577,7 @@ bool amdgpu_device_has_job_running(struct amdgpu_device *adev)
        for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                struct amdgpu_ring *ring = adev->rings[i];
 
-               if (!ring || !ring->sched.thread)
+               if (!ring || !ring->sched.ready)
                        continue;
 
                spin_lock(&ring->sched.job_list_lock);
@@ -4708,7 +4708,7 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
        for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                struct amdgpu_ring *ring = adev->rings[i];
 
-               if (!ring || !ring->sched.thread)
+               if (!ring || !ring->sched.ready)
                        continue;
 
                /*clear job fence from fence drv to avoid force_completion
@@ -5247,7 +5247,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
                for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                        struct amdgpu_ring *ring = tmp_adev->rings[i];
 
-                       if (!ring || !ring->sched.thread)
+                       if (!ring || !ring->sched.ready)
                                continue;
 
                        drm_sched_stop(&ring->sched, job ? &job->base : NULL);
@@ -5321,7 +5321,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
                for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                        struct amdgpu_ring *ring = tmp_adev->rings[i];
 
-                       if (!ring || !ring->sched.thread)
+                       if (!ring || !ring->sched.ready)
                                continue;
 
                        drm_sched_start(&ring->sched, true);
@@ -5648,7 +5648,7 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta
                for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                        struct amdgpu_ring *ring = adev->rings[i];
 
-                       if (!ring || !ring->sched.thread)
+                       if (!ring || !ring->sched.ready)
                                continue;
 
                        drm_sched_stop(&ring->sched, NULL);
@@ -5776,7 +5776,7 @@ void amdgpu_pci_resume(struct pci_dev *pdev)
        for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                struct amdgpu_ring *ring = adev->rings[i];
 
-               if (!ring || !ring->sched.thread)
+               if (!ring || !ring->sched.ready)
                        continue;
 
                drm_sched_start(&ring->sched, true);
drivers/gpu/drm/scheduler/sched_main.c +72 -52
index 27d52ffbb808..8c64045d0692 100644
@@ -44,7 +44,6 @@
  * The jobs in a entity are always scheduled in the order that they were pushed.
  */
 
-#include <linux/kthread.h>
 #include <linux/wait.h>
 #include <linux/sched.h>
 #include <linux/completion.h>
@@ -251,6 +250,53 @@ drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq)
        return rb ? rb_entry(rb, struct drm_sched_entity, rb_tree_node) : NULL;
 }
 
+/**
+ * drm_sched_run_wq_stop - stop scheduler run worker
+ *
+ * @sched: scheduler instance to stop run worker
+ */
+void drm_sched_run_wq_stop(struct drm_gpu_scheduler *sched)
+{
+       sched->pause_run_wq = true;
+       smp_wmb();
+
+       cancel_work_sync(&sched->work_run);
+}
+EXPORT_SYMBOL(drm_sched_run_wq_stop);
+
+/**
+ * drm_sched_run_wq_start - start scheduler run worker
+ *
+ * @sched: scheduler instance to start run worker
+ */
+void drm_sched_run_wq_start(struct drm_gpu_scheduler *sched)
+{
+       sched->pause_run_wq = false;
+       smp_wmb();
+
+       queue_work(sched->run_wq, &sched->work_run);
+}
+EXPORT_SYMBOL(drm_sched_run_wq_start);
+
+/**
+ * drm_sched_run_wq_queue - queue scheduler run worker
+ *
+ * @sched: scheduler instance to queue run worker
+ */
+static void drm_sched_run_wq_queue(struct drm_gpu_scheduler *sched)
+{
+       smp_rmb();
+
+       /*
+        * Try not to schedule work if pause_run_wq set but not the end of world
+        * if we do as either it will be cancelled by the above
+        * cancel_work_sync, or drm_sched_main turns into a NOP while
+        * pause_run_wq is set.
+        */
+       if (!sched->pause_run_wq)
+               queue_work(sched->run_wq, &sched->work_run);
+}
+
 /**
  * drm_sched_job_done - complete a job
  * @s_job: pointer to the job which is done
@@ -270,7 +316,7 @@ static void drm_sched_job_done(struct drm_sched_job *s_job)
        dma_fence_get(&s_fence->finished);
        drm_sched_fence_finished(s_fence);
        dma_fence_put(&s_fence->finished);
-       wake_up_interruptible(&sched->wake_up_worker);
+       drm_sched_run_wq_queue(sched);
 }
 
 /**
@@ -433,7 +479,7 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad)
 {
        struct drm_sched_job *s_job, *tmp;
 
-       kthread_park(sched->thread);
+       drm_sched_run_wq_stop(sched);
 
        /*
         * Reinsert back the bad job here - now it's safe as
@@ -546,7 +592,7 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery)
                spin_unlock(&sched->job_list_lock);
        }
 
-       kthread_unpark(sched->thread);
+       drm_sched_run_wq_start(sched);
 }
 EXPORT_SYMBOL(drm_sched_start);
 
@@ -831,7 +877,7 @@ static bool drm_sched_ready(struct drm_gpu_scheduler *sched)
 void drm_sched_wakeup(struct drm_gpu_scheduler *sched)
 {
        if (drm_sched_ready(sched))
-               wake_up_interruptible(&sched->wake_up_worker);
+               drm_sched_run_wq_queue(sched);
 }
 
 /**
@@ -941,60 +987,42 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list,
 }
 EXPORT_SYMBOL(drm_sched_pick_best);
 
-/**
- * drm_sched_blocked - check if the scheduler is blocked
- *
- * @sched: scheduler instance
- *
- * Returns true if blocked, otherwise false.
- */
-static bool drm_sched_blocked(struct drm_gpu_scheduler *sched)
-{
-       if (kthread_should_park()) {
-               kthread_parkme();
-               return true;
-       }
-
-       return false;
-}
-
 /**
  * drm_sched_main - main scheduler thread
  *
  * @param: scheduler instance
- *
- * Returns 0.
  */
-static int drm_sched_main(void *param)
+static void drm_sched_main(struct work_struct *w)
 {
-       struct drm_gpu_scheduler *sched = (struct drm_gpu_scheduler *)param;
+       struct drm_gpu_scheduler *sched =
+               container_of(w, struct drm_gpu_scheduler, work_run);
        int r;
 
-       sched_set_fifo_low(current);
-
-       while (!kthread_should_stop()) {
-               struct drm_sched_entity *entity = NULL;
+       while (!READ_ONCE(sched->pause_run_wq)) {
+               struct drm_sched_entity *entity;
                struct drm_sched_fence *s_fence;
                struct drm_sched_job *sched_job;
                struct dma_fence *fence;
-               struct drm_sched_job *cleanup_job = NULL;
+               struct drm_sched_job *cleanup_job;
 
-               wait_event_interruptible(sched->wake_up_worker,
-                                        (cleanup_job = drm_sched_get_cleanup_job(sched)) ||
-                                        (!drm_sched_blocked(sched) &&
-                                         (entity = drm_sched_select_entity(sched))) ||
-                                        kthread_should_stop());
+               cleanup_job = drm_sched_get_cleanup_job(sched);
+               entity = drm_sched_select_entity(sched);
 
                if (cleanup_job)
                        sched->ops->free_job(cleanup_job);
 
-               if (!entity)
+               if (!entity) {
+                       if (!cleanup_job)
+                               break;
                        continue;
+               }
 
                sched_job = drm_sched_entity_pop_job(entity);
 
                if (!sched_job) {
                        complete_all(&entity->entity_idle);
+                       if (!cleanup_job)
+                               break;
                        continue;
                }
 
@@ -1022,14 +1050,14 @@ static int drm_sched_main(void *param)
                                          r);
                } else {
                        if (IS_ERR(fence))
-                               dma_fence_set_error(&s_fence->finished, PTR_ERR(fence));
+                               dma_fence_set_error(&s_fence->finished,
+                                                   PTR_ERR(fence));
 
                        drm_sched_job_done(sched_job);
                }
 
                wake_up(&sched->job_scheduled);
        }
-       return 0;
 }
 
 /**
@@ -1054,35 +1082,28 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,
                   long timeout, struct workqueue_struct *timeout_wq,
                   atomic_t *score, const char *name, struct device *dev)
 {
-       int i, ret;
+       int i;
        sched->ops = ops;
        sched->hw_submission_limit = hw_submission;
        sched->name = name;
        sched->timeout = timeout;
        sched->timeout_wq = timeout_wq ? : system_wq;
+       sched->run_wq = system_wq;      /* FIXME: Let user pass this in */
        sched->hang_limit = hang_limit;
        sched->score = score ? score : &sched->_score;
        sched->dev = dev;
        for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++)
                drm_sched_rq_init(sched, &sched->sched_rq[i]);
 
-       init_waitqueue_head(&sched->wake_up_worker);
        init_waitqueue_head(&sched->job_scheduled);
        INIT_LIST_HEAD(&sched->pending_list);
        spin_lock_init(&sched->job_list_lock);
        atomic_set(&sched->hw_rq_count, 0);
        INIT_DELAYED_WORK(&sched->work_tdr, drm_sched_job_timedout);
+       INIT_WORK(&sched->work_run, drm_sched_main);
        atomic_set(&sched->_score, 0);
        atomic64_set(&sched->job_id_count, 0);
-
-       /* Each scheduler will run on a seperate kernel thread */
-       sched->thread = kthread_run(drm_sched_main, sched, sched->name);
-       if (IS_ERR(sched->thread)) {
-               ret = PTR_ERR(sched->thread);
-               sched->thread = NULL;
-               DRM_DEV_ERROR(sched->dev, "Failed to create scheduler for %s.\n", name);
-               return ret;
-       }
+       sched->pause_run_wq = false;
 
        sched->ready = true;
        return 0;
@@ -1101,8 +1122,7 @@ void drm_sched_fini(struct drm_gpu_scheduler *sched)
        struct drm_sched_entity *s_entity;
        int i;
 
-       if (sched->thread)
-               kthread_stop(sched->thread);
+       drm_sched_run_wq_stop(sched);
 
        for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) {
                struct drm_sched_rq *rq = &sched->sched_rq[i];
include/drm/gpu_scheduler.h +8 -6
index ca857ec9e7eb..ff50f3c289cd 100644
@@ -456,17 +456,16 @@ struct drm_sched_backend_ops {
  * @timeout: the time after which a job is removed from the scheduler.
  * @name: name of the ring for which this scheduler is being used.
  * @sched_rq: priority wise array of run queues.
- * @wake_up_worker: the wait queue on which the scheduler sleeps until a job
- *                  is ready to be scheduled.
  * @job_scheduled: once @drm_sched_entity_do_release is called the scheduler
  *                 waits on this wait queue until all the scheduled jobs are
  *                 finished.
  * @hw_rq_count: the number of jobs currently in the hardware queue.
  * @job_id_count: used to assign unique id to the each job.
+ * @run_wq: workqueue used to queue @work_run
  * @timeout_wq: workqueue used to queue @work_tdr
+ * @work_run: schedules jobs and cleans up entities
  * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the
  *            timeout interval is over.
- * @thread: the kthread on which the scheduler which run.
  * @pending_list: the list of jobs which are currently in the job queue.
  * @job_list_lock: lock to protect the pending_list.
  * @hang_limit: once the hangs by a job crosses this limit then it is marked
@@ -475,6 +474,7 @@ struct drm_sched_backend_ops {
  * @_score: score used when the driver doesn't provide one
  * @ready: marks if the underlying HW is ready to work
  * @free_guilty: A hit to time out handler to free the guilty job.
+ * @pause_run_wq: pause queuing of @work_run on @run_wq
  * @dev: system &struct device
  *
  * One scheduler is implemented for each hardware ring.
@@ -485,13 +485,13 @@ struct drm_gpu_scheduler {
        long                            timeout;
        const char                      *name;
        struct drm_sched_rq             sched_rq[DRM_SCHED_PRIORITY_COUNT];
-       wait_queue_head_t               wake_up_worker;
        wait_queue_head_t               job_scheduled;
        atomic_t                        hw_rq_count;
        atomic64_t                      job_id_count;
+       struct workqueue_struct         *run_wq;
        struct workqueue_struct         *timeout_wq;
+       struct work_struct              work_run;
        struct delayed_work             work_tdr;
-       struct task_struct              *thread;
        struct list_head                pending_list;
        spinlock_t                      job_list_lock;
        int                             hang_limit;
@@ -499,6 +499,7 @@ struct drm_gpu_scheduler {
        atomic_t                        _score;
        bool                            ready;
        bool                            free_guilty;
+       bool                            pause_run_wq;
        struct device                   *dev;
 };
 
@@ -529,6 +530,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity,
 
 void drm_sched_job_cleanup(struct drm_sched_job *job);
 void drm_sched_wakeup(struct drm_gpu_scheduler *sched);
+void drm_sched_run_wq_stop(struct drm_gpu_scheduler *sched);
+void drm_sched_run_wq_start(struct drm_gpu_scheduler *sched);
 void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad);
 void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery);
 void drm_sched_resubmit_jobs(struct drm_gpu_scheduler *sched);
--

2.37.3

author maintainer
#6 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 24 lines · Rob Clark
Quoted 18 lines · Matthew Brost

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

2. In XE submissions are done via programming a ring buffer (circular buffer), a drm_gpu_scheduler provides a limit on number of jobs, if the limit of number jobs is set to RING_SIZE / MAX_SIZE_PER_JOB we get flow control on the ring for free.

Quoted 4 lines

A problem with this design is currently a drm_gpu_scheduler uses a kthread for submission / job cleanup. This doesn't scale if a large number of drm_gpu_scheduler are used. To work around the scaling issue, use a worker rather than kthread for submission / job cleanup.

Quoted 6 lines

You might want to enable CONFIG_DRM_MSM in your kconfig, I think you missed a part

BR,
-R

Thanks for feedback Rob, yes indeed we missed updating the MSM driver. Fixed up in our Xe repo and will be fixed in the next rev on the list.

Matt

Quoted 433 lines
Quoted 433 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  14 +--
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  12 +-
 drivers/gpu/drm/scheduler/sched_main.c      | 124 ++++++++++++--------
 include/drm/gpu_scheduler.h                 |  13 +-
 4 files changed, 93 insertions(+), 70 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +7 -7
index f60753f97ac5..9c2a10aeb0b3 100644
@@ -1489,9 +1489,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused)
        for (i = 0; i < AMDGPU_MAX_RINGS; i++) {
                struct amdgpu_ring *ring = adev->rings[i];
 
-               if (!ring || !ring->sched.thread)
+               if (!ring || !ring->sched.ready)
                        continue;
-               kthread_park(ring->sched.thread);
+               drm_sched_run_wq_stop(&ring->sched);
        }
 
        seq_printf(m, "run ib test:\n");
@@ -1505,9 +1505,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused)
        for (i = 0; i < AMDGPU_MAX_RINGS; i++) {
                struct amdgpu_ring *ring = adev->rings[i];
 
-               if (!ring || !ring->sched.thread)
+               if (!ring || !ring->sched.ready)
                        continue;
-               kthread_unpark(ring->sched.thread);
+               drm_sched_run_wq_start(&ring->sched);
        }
 
        up_write(&adev->reset_domain->sem);
@@ -1727,7 +1727,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
 
        ring = adev->rings[val];
 
-       if (!ring || !ring->funcs->preempt_ib || !ring->sched.thread)
+       if (!ring || !ring->funcs->preempt_ib || !ring->sched.ready)
                return -EINVAL;
 
        /* the last preemption failed */
@@ -1745,7 +1745,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
                goto pro_end;
 
        /* stop the scheduler */
-       kthread_park(ring->sched.thread);
+       drm_sched_run_wq_stop(&ring->sched);
 
        /* preempt the IB */
        r = amdgpu_ring_preempt_ib(ring);
@@ -1779,7 +1779,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
 
 failure:
        /* restart the scheduler */
-       kthread_unpark(ring->sched.thread);
+       drm_sched_run_wq_start(&ring->sched);
 
        up_read(&adev->reset_domain->sem);
 
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +6 -6
index 076ae400d099..9552929ccf87 100644
@@ -4577,7 +4577,7 @@ bool amdgpu_device_has_job_running(struct amdgpu_device *adev)
        for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                struct amdgpu_ring *ring = adev->rings[i];
 
-               if (!ring || !ring->sched.thread)
+               if (!ring || !ring->sched.ready)
                        continue;
 
                spin_lock(&ring->sched.job_list_lock);
@@ -4708,7 +4708,7 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
        for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                struct amdgpu_ring *ring = adev->rings[i];
 
-               if (!ring || !ring->sched.thread)
+               if (!ring || !ring->sched.ready)
                        continue;
 
                /*clear job fence from fence drv to avoid force_completion
@@ -5247,7 +5247,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
                for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                        struct amdgpu_ring *ring = tmp_adev->rings[i];
 
-                       if (!ring || !ring->sched.thread)
+                       if (!ring || !ring->sched.ready)
                                continue;
 
                        drm_sched_stop(&ring->sched, job ? &job->base : NULL);
@@ -5321,7 +5321,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
                for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                        struct amdgpu_ring *ring = tmp_adev->rings[i];
 
-                       if (!ring || !ring->sched.thread)
+                       if (!ring || !ring->sched.ready)
                                continue;
 
                        drm_sched_start(&ring->sched, true);
@@ -5648,7 +5648,7 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta
                for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                        struct amdgpu_ring *ring = adev->rings[i];
 
-                       if (!ring || !ring->sched.thread)
+                       if (!ring || !ring->sched.ready)
                                continue;
 
                        drm_sched_stop(&ring->sched, NULL);
@@ -5776,7 +5776,7 @@ void amdgpu_pci_resume(struct pci_dev *pdev)
        for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                struct amdgpu_ring *ring = adev->rings[i];
 
-               if (!ring || !ring->sched.thread)
+               if (!ring || !ring->sched.ready)
                        continue;
 
                drm_sched_start(&ring->sched, true);
drivers/gpu/drm/scheduler/sched_main.c +72 -52
index 27d52ffbb808..8c64045d0692 100644
@@ -44,7 +44,6 @@
  * The jobs in a entity are always scheduled in the order that they were pushed.
  */
 
-#include <linux/kthread.h>
 #include <linux/wait.h>
 #include <linux/sched.h>
 #include <linux/completion.h>
@@ -251,6 +250,53 @@ drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq)
        return rb ? rb_entry(rb, struct drm_sched_entity, rb_tree_node) : NULL;
 }
 
+/**
+ * drm_sched_run_wq_stop - stop scheduler run worker
+ *
+ * @sched: scheduler instance to stop run worker
+ */
+void drm_sched_run_wq_stop(struct drm_gpu_scheduler *sched)
+{
+       sched->pause_run_wq = true;
+       smp_wmb();
+
+       cancel_work_sync(&sched->work_run);
+}
+EXPORT_SYMBOL(drm_sched_run_wq_stop);
+
+/**
+ * drm_sched_run_wq_start - start scheduler run worker
+ *
+ * @sched: scheduler instance to start run worker
+ */
+void drm_sched_run_wq_start(struct drm_gpu_scheduler *sched)
+{
+       sched->pause_run_wq = false;
+       smp_wmb();
+
+       queue_work(sched->run_wq, &sched->work_run);
+}
+EXPORT_SYMBOL(drm_sched_run_wq_start);
+
+/**
+ * drm_sched_run_wq_queue - queue scheduler run worker
+ *
+ * @sched: scheduler instance to queue run worker
+ */
+static void drm_sched_run_wq_queue(struct drm_gpu_scheduler *sched)
+{
+       smp_rmb();
+
+       /*
+        * Try not to schedule work if pause_run_wq set but not the end of world
+        * if we do as either it will be cancelled by the above
+        * cancel_work_sync, or drm_sched_main turns into a NOP while
+        * pause_run_wq is set.
+        */
+       if (!sched->pause_run_wq)
+               queue_work(sched->run_wq, &sched->work_run);
+}
+
 /**
  * drm_sched_job_done - complete a job
  * @s_job: pointer to the job which is done
@@ -270,7 +316,7 @@ static void drm_sched_job_done(struct drm_sched_job *s_job)
        dma_fence_get(&s_fence->finished);
        drm_sched_fence_finished(s_fence);
        dma_fence_put(&s_fence->finished);
-       wake_up_interruptible(&sched->wake_up_worker);
+       drm_sched_run_wq_queue(sched);
 }
 
 /**
@@ -433,7 +479,7 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad)
 {
        struct drm_sched_job *s_job, *tmp;
 
-       kthread_park(sched->thread);
+       drm_sched_run_wq_stop(sched);
 
        /*
         * Reinsert back the bad job here - now it's safe as
@@ -546,7 +592,7 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery)
                spin_unlock(&sched->job_list_lock);
        }
 
-       kthread_unpark(sched->thread);
+       drm_sched_run_wq_start(sched);
 }
 EXPORT_SYMBOL(drm_sched_start);
 
@@ -831,7 +877,7 @@ static bool drm_sched_ready(struct drm_gpu_scheduler *sched)
 void drm_sched_wakeup(struct drm_gpu_scheduler *sched)
 {
        if (drm_sched_ready(sched))
-               wake_up_interruptible(&sched->wake_up_worker);
+               drm_sched_run_wq_queue(sched);
 }
 
 /**
@@ -941,60 +987,42 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list,
 }
 EXPORT_SYMBOL(drm_sched_pick_best);
 
-/**
- * drm_sched_blocked - check if the scheduler is blocked
- *
- * @sched: scheduler instance
- *
- * Returns true if blocked, otherwise false.
- */
-static bool drm_sched_blocked(struct drm_gpu_scheduler *sched)
-{
-       if (kthread_should_park()) {
-               kthread_parkme();
-               return true;
-       }
-
-       return false;
-}
-
 /**
  * drm_sched_main - main scheduler thread
  *
  * @param: scheduler instance
- *
- * Returns 0.
  */
-static int drm_sched_main(void *param)
+static void drm_sched_main(struct work_struct *w)
 {
-       struct drm_gpu_scheduler *sched = (struct drm_gpu_scheduler *)param;
+       struct drm_gpu_scheduler *sched =
+               container_of(w, struct drm_gpu_scheduler, work_run);
        int r;
 
-       sched_set_fifo_low(current);
-
-       while (!kthread_should_stop()) {
-               struct drm_sched_entity *entity = NULL;
+       while (!READ_ONCE(sched->pause_run_wq)) {
+               struct drm_sched_entity *entity;
                struct drm_sched_fence *s_fence;
                struct drm_sched_job *sched_job;
                struct dma_fence *fence;
-               struct drm_sched_job *cleanup_job = NULL;
+               struct drm_sched_job *cleanup_job;
 
-               wait_event_interruptible(sched->wake_up_worker,
-                                        (cleanup_job = drm_sched_get_cleanup_job(sched)) ||
-                                        (!drm_sched_blocked(sched) &&
-                                         (entity = drm_sched_select_entity(sched))) ||
-                                        kthread_should_stop());
+               cleanup_job = drm_sched_get_cleanup_job(sched);
+               entity = drm_sched_select_entity(sched);
 
                if (cleanup_job)
                        sched->ops->free_job(cleanup_job);
 
-               if (!entity)
+               if (!entity) {
+                       if (!cleanup_job)
+                               break;
                        continue;
+               }
 
                sched_job = drm_sched_entity_pop_job(entity);
 
                if (!sched_job) {
                        complete_all(&entity->entity_idle);
+                       if (!cleanup_job)
+                               break;
                        continue;
                }
 
@@ -1022,14 +1050,14 @@ static int drm_sched_main(void *param)
                                          r);
                } else {
                        if (IS_ERR(fence))
-                               dma_fence_set_error(&s_fence->finished, PTR_ERR(fence));
+                               dma_fence_set_error(&s_fence->finished,
+                                                   PTR_ERR(fence));
 
                        drm_sched_job_done(sched_job);
                }
 
                wake_up(&sched->job_scheduled);
        }
-       return 0;
 }
 
 /**
@@ -1054,35 +1082,28 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,
                   long timeout, struct workqueue_struct *timeout_wq,
                   atomic_t *score, const char *name, struct device *dev)
 {
-       int i, ret;
+       int i;
        sched->ops = ops;
        sched->hw_submission_limit = hw_submission;
        sched->name = name;
        sched->timeout = timeout;
        sched->timeout_wq = timeout_wq ? : system_wq;
+       sched->run_wq = system_wq;      /* FIXME: Let user pass this in */
        sched->hang_limit = hang_limit;
        sched->score = score ? score : &sched->_score;
        sched->dev = dev;
        for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++)
                drm_sched_rq_init(sched, &sched->sched_rq[i]);
 
-       init_waitqueue_head(&sched->wake_up_worker);
        init_waitqueue_head(&sched->job_scheduled);
        INIT_LIST_HEAD(&sched->pending_list);
        spin_lock_init(&sched->job_list_lock);
        atomic_set(&sched->hw_rq_count, 0);
        INIT_DELAYED_WORK(&sched->work_tdr, drm_sched_job_timedout);
+       INIT_WORK(&sched->work_run, drm_sched_main);
        atomic_set(&sched->_score, 0);
        atomic64_set(&sched->job_id_count, 0);
-
-       /* Each scheduler will run on a seperate kernel thread */
-       sched->thread = kthread_run(drm_sched_main, sched, sched->name);
-       if (IS_ERR(sched->thread)) {
-               ret = PTR_ERR(sched->thread);
-               sched->thread = NULL;
-               DRM_DEV_ERROR(sched->dev, "Failed to create scheduler for %s.\n", name);
-               return ret;
-       }
+       sched->pause_run_wq = false;
 
        sched->ready = true;
        return 0;
@@ -1101,8 +1122,7 @@ void drm_sched_fini(struct drm_gpu_scheduler *sched)
        struct drm_sched_entity *s_entity;
        int i;
 
-       if (sched->thread)
-               kthread_stop(sched->thread);
+       drm_sched_run_wq_stop(sched);
 
        for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) {
                struct drm_sched_rq *rq = &sched->sched_rq[i];
include/drm/gpu_scheduler.h +8 -6
index ca857ec9e7eb..ff50f3c289cd 100644
@@ -456,17 +456,16 @@ struct drm_sched_backend_ops {
  * @timeout: the time after which a job is removed from the scheduler.
  * @name: name of the ring for which this scheduler is being used.
  * @sched_rq: priority wise array of run queues.
- * @wake_up_worker: the wait queue on which the scheduler sleeps until a job
- *                  is ready to be scheduled.
  * @job_scheduled: once @drm_sched_entity_do_release is called the scheduler
  *                 waits on this wait queue until all the scheduled jobs are
  *                 finished.
  * @hw_rq_count: the number of jobs currently in the hardware queue.
  * @job_id_count: used to assign unique id to the each job.
+ * @run_wq: workqueue used to queue @work_run
  * @timeout_wq: workqueue used to queue @work_tdr
+ * @work_run: schedules jobs and cleans up entities
  * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the
  *            timeout interval is over.
- * @thread: the kthread on which the scheduler which run.
  * @pending_list: the list of jobs which are currently in the job queue.
  * @job_list_lock: lock to protect the pending_list.
  * @hang_limit: once the hangs by a job crosses this limit then it is marked
@@ -475,6 +474,7 @@ struct drm_sched_backend_ops {
  * @_score: score used when the driver doesn't provide one
  * @ready: marks if the underlying HW is ready to work
  * @free_guilty: A hit to time out handler to free the guilty job.
+ * @pause_run_wq: pause queuing of @work_run on @run_wq
  * @dev: system &struct device
  *
  * One scheduler is implemented for each hardware ring.
@@ -485,13 +485,13 @@ struct drm_gpu_scheduler {
        long                            timeout;
        const char                      *name;
        struct drm_sched_rq             sched_rq[DRM_SCHED_PRIORITY_COUNT];
-       wait_queue_head_t               wake_up_worker;
        wait_queue_head_t               job_scheduled;
        atomic_t                        hw_rq_count;
        atomic64_t                      job_id_count;
+       struct workqueue_struct         *run_wq;
        struct workqueue_struct         *timeout_wq;
+       struct work_struct              work_run;
        struct delayed_work             work_tdr;
-       struct task_struct              *thread;
        struct list_head                pending_list;
        spinlock_t                      job_list_lock;
        int                             hang_limit;
@@ -499,6 +499,7 @@ struct drm_gpu_scheduler {
        atomic_t                        _score;
        bool                            ready;
        bool                            free_guilty;
+       bool                            pause_run_wq;
        struct device                   *dev;
 };
 
@@ -529,6 +530,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity,
 
 void drm_sched_job_cleanup(struct drm_sched_job *job);
 void drm_sched_wakeup(struct drm_gpu_scheduler *sched);
+void drm_sched_run_wq_stop(struct drm_gpu_scheduler *sched);
+void drm_sched_run_wq_start(struct drm_gpu_scheduler *sched);
 void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad);
 void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery);
 void drm_sched_resubmit_jobs(struct drm_gpu_scheduler *sched);
--

2.37.3

#7 lore raw

Re: [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

In reply to Matthew Brost

Hello Matthew,

Quoted 11 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a FW-assisted scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because I felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof. Giving drm_sched_entity exlusive access to a drm_gpu_scheduler probably might help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs. Here are a few things I noted while working on the drm_sched-based PoC:

- The complexity to suspend/resume streams and recover from failures remains quite important (because everything is still very asynchronous under the hood). Sure, you don't have to do this fancy timeslice-based scheduling, but that's still a lot of code, and AFAICT, it didn't integrate well with drm_sched TDR (my previous attempt at reconciling them has been unsuccessful, but maybe your patches would help there)
- You lose one of the nice thing that's brought by timeslice-based
scheduling: a tiny bit of fairness. That is, if one stream is queuing a compute job that's monopolizing the GPU core, you know the kernel part of the scheduler will eventually evict it and let other streams with same or higher priority run, even before the job timeout kicks in.
- Stream slots exposed by the Arm FW are not exactly HW queues that run things concurrently. The FW can decide to let only the stream with the highest priority get access to the various HW resources (GPU cores, tiler, ...), and let other streams starve. That means you might get spurious timeouts on some jobs/sched-entities while they didn't even get a chance to run.

So overall, and given I'm no longer the only one having to deal with a FW scheduler that's designed with timeslice scheduling in mind, I'm wondering if it's not time to design a common timeslice-based scheduler instead of trying to bend drivers to use the model enforced by drm_sched. But that's just my 2 cents, of course.

Regards,

Boris

#8 lore raw

Re: [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 17 lines · On Fri, 30 Dec 2022 11:20:42 +0100 Boris Brezillon…

Hello Matthew,

Quoted 11 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

Quoted 12 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a FW-assisted scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because I felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof. Giving drm_sched_entity exlusive access to a drm_gpu_scheduler probably might help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue selection is dumb because there's only one entity ever bound to the scheduler (the one that's part of the xe_guc_engine object which also contains the drm_gpu_scheduler instance). I guess the main issue we'd have on Arm is the fact that the stream doesn't necessarily get scheduled when ->run_job() is called, it can be placed in the runnable queue and be picked later by the kernel-side scheduler when a FW slot gets released. That can probably be sorted out by manually disabling the job timer and re-enabling it when the stream gets picked by the scheduler. But my main concern remains, we're basically abusing drm_sched here.

For the Arm driver, that means turning the following sequence

1. wait for job deps
2. queue job to ringbuf and push the stream to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
3. stream gets picked by the timeslice scheduler and sent to the FW for execution

into

1. queue job to entity which takes care of waiting for job deps for us
2. schedule a drm_sched_main iteration
3. the only available entity is picked, and the first job from this entity is dequeued. ->run_job() is called: the job is queued to the ringbuf and the stream is pushed to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
4. stream gets picked by the timeslice scheduler and sent to the FW for execution

That's one extra step we don't really need. To sum-up, yes, all the job/entity tracking might be interesting to share/re-use, but I wonder if we couldn't have that without pulling out the scheduling part of drm_sched, or maybe I'm missing something, and there's something in drm_gpu_scheduler you really need.

#9 lore raw

Re: [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 72 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…
Quoted 17 lines · On Fri, 30 Dec 2022 11:20:42 +0100 Boris Brezillon…

Hello Matthew,

Quoted 11 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

Quoted 12 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a FW-assisted scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because I felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof. Giving drm_sched_entity exlusive access to a drm_gpu_scheduler probably might help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue selection is dumb because there's only one entity ever bound to the scheduler (the one that's part of the xe_guc_engine object which also contains the drm_gpu_scheduler instance). I guess the main issue we'd have on Arm is the fact that the stream doesn't necessarily get scheduled when ->run_job() is called, it can be placed in the runnable queue and be picked later by the kernel-side scheduler when a FW slot gets released. That can probably be sorted out by manually disabling the job timer and re-enabling it when the stream gets picked by the scheduler. But my main concern remains, we're basically abusing drm_sched here.

For the Arm driver, that means turning the following sequence

1. wait for job deps
2. queue job to ringbuf and push the stream to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
3. stream gets picked by the timeslice scheduler and sent to the FW for execution

into

1. queue job to entity which takes care of waiting for job deps for us
2. schedule a drm_sched_main iteration
3. the only available entity is picked, and the first job from this entity is dequeued. ->run_job() is called: the job is queued to the ringbuf and the stream is pushed to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
4. stream gets picked by the timeslice scheduler and sent to the FW for execution

Quoted 5 lines

That's one extra step we don't really need. To sum-up, yes, all the job/entity tracking might be interesting to share/re-use, but I wonder if we couldn't have that without pulling out the scheduling part of drm_sched, or maybe I'm missing something, and there's something in drm_gpu_scheduler you really need.

On second thought, that's probably an acceptable overhead (not even sure the extra step I was mentioning exists in practice, because dep fence signaled state is checked as part of the drm_sched_main iteration, so that's basically replacing the worker I schedule to check job deps), and I like the idea of being able to re-use drm_sched dep-tracking without resorting to invasive changes to the existing logic, so I'll probably give it a try.

#10 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 81 lines · Boris Brezillon
Quoted 72 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…
Quoted 17 lines · On Fri, 30 Dec 2022 11:20:42 +0100 Boris Brezillon…

Hello Matthew,

Quoted 11 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

Quoted 12 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a FW-assisted scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because I felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof. Giving drm_sched_entity exlusive access to a drm_gpu_scheduler probably might help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue selection is dumb because there's only one entity ever bound to the scheduler (the one that's part of the xe_guc_engine object which also contains the drm_gpu_scheduler instance). I guess the main issue we'd have on Arm is the fact that the stream doesn't necessarily get scheduled when ->run_job() is called, it can be placed in the runnable queue and be picked later by the kernel-side scheduler when a FW slot gets released. That can probably be sorted out by manually disabling the job timer and re-enabling it when the stream gets picked by the scheduler. But my main concern remains, we're basically abusing drm_sched here.

For the Arm driver, that means turning the following sequence

1. wait for job deps
2. queue job to ringbuf and push the stream to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
3. stream gets picked by the timeslice scheduler and sent to the FW for execution

into

1. queue job to entity which takes care of waiting for job deps for us
2. schedule a drm_sched_main iteration
3. the only available entity is picked, and the first job from this entity is dequeued. ->run_job() is called: the job is queued to the ringbuf and the stream is pushed to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
4. stream gets picked by the timeslice scheduler and sent to the FW for execution

Quoted 5 lines

That's one extra step we don't really need. To sum-up, yes, all the job/entity tracking might be interesting to share/re-use, but I wonder if we couldn't have that without pulling out the scheduling part of drm_sched, or maybe I'm missing something, and there's something in drm_gpu_scheduler you really need.

Quoted 7 lines

On second thought, that's probably an acceptable overhead (not even sure the extra step I was mentioning exists in practice, because dep fence signaled state is checked as part of the drm_sched_main iteration, so that's basically replacing the worker I schedule to check job deps), and I like the idea of being able to re-use drm_sched dep-tracking without resorting to invasive changes to the existing logic, so I'll probably give it a try.

I agree with the concerns and think that how Xe proposes to integrate with drm_sched is a problem, or at least significantly inelegant.

AFAICT it proposes to have 1:1 between *userspace* created contexts (per context _and_ engine) and drm_sched. I am not sure avoiding invasive changes to the shared code is in the spirit of the overall idea and instead opportunity should be used to look at way to refactor/improve drm_sched.

Even on the low level, the idea to replace drm_sched threads with workers has a few problems.

To start with, the pattern of:

while (not_stopped) {
keep picking jobs
}

Feels fundamentally in disagreement with workers (while obviously fits perfectly with the current kthread design).

Secondly, it probably demands separate workers (not optional), otherwise behaviour of shared workqueues has either the potential to explode number kernel threads anyway, or add latency.

What would be interesting to learn is whether the option of refactoring drm_sched to deal with out of order completion was considered and what were the conclusions.

Second option perhaps to split out the drm_sched code into parts which would lend themselves more to "pick and choose" of its functionalities. Specifically, Xe wants frontend dependency tracking, but not any scheduling really (neither least busy drm_sched, neither FIFO/RQ entity picking), so even having all these data structures in memory is a waste.

With the first option then the end result could be drm_sched per engine class (hardware view), which I think fits with the GuC model. Give all schedulable contexts (entities) to the GuC and then mostly forget about them. Timeslicing and re-ordering and all happens transparently to the kernel from that point until completion.

Or with the second option you would build on some smaller refactored sub-components of drm_sched, by maybe splitting the dependency tracking from scheduling (RR/FIFO entity picking code).

Second option is especially a bit vague and I haven't thought about the required mechanics, but it just appeared too obvious the proposed design has a bit too much impedance mismatch.

Oh and as a side note, when I went into the drm_sched code base to remind myself how things worked, it is quite easy to find some FIXME comments which suggest people working on it are unsure of locking desing there and such. So perhaps that all needs cleanup too, I mean would benefit from refactoring/improving work as brainstormed above anyway.

Regards,

Tvrtko

#11 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Hi,

Quoted 90 lines · On Tue, 3 Jan 2023 13:02:15 +0000 Tvrtko Ursulin…
Quoted 81 lines · Boris Brezillon
Quoted 72 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…
Quoted 17 lines · On Fri, 30 Dec 2022 11:20:42 +0100 Boris Brezillon…

Hello Matthew,

Quoted 11 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

Quoted 12 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a FW-assisted scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because I felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof. Giving drm_sched_entity exlusive access to a drm_gpu_scheduler probably might help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue selection is dumb because there's only one entity ever bound to the scheduler (the one that's part of the xe_guc_engine object which also contains the drm_gpu_scheduler instance). I guess the main issue we'd have on Arm is the fact that the stream doesn't necessarily get scheduled when ->run_job() is called, it can be placed in the runnable queue and be picked later by the kernel-side scheduler when a FW slot gets released. That can probably be sorted out by manually disabling the job timer and re-enabling it when the stream gets picked by the scheduler. But my main concern remains, we're basically abusing drm_sched here.

For the Arm driver, that means turning the following sequence

1. wait for job deps
2. queue job to ringbuf and push the stream to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
3. stream gets picked by the timeslice scheduler and sent to the FW for execution

into

1. queue job to entity which takes care of waiting for job deps for us
2. schedule a drm_sched_main iteration
3. the only available entity is picked, and the first job from this entity is dequeued. ->run_job() is called: the job is queued to the ringbuf and the stream is pushed to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
4. stream gets picked by the timeslice scheduler and sent to the FW for execution

Quoted 5 lines

That's one extra step we don't really need. To sum-up, yes, all the job/entity tracking might be interesting to share/re-use, but I wonder if we couldn't have that without pulling out the scheduling part of drm_sched, or maybe I'm missing something, and there's something in drm_gpu_scheduler you really need.

Quoted 7 lines

On second thought, that's probably an acceptable overhead (not even sure the extra step I was mentioning exists in practice, because dep fence signaled state is checked as part of the drm_sched_main iteration, so that's basically replacing the worker I schedule to check job deps), and I like the idea of being able to re-use drm_sched dep-tracking without resorting to invasive changes to the existing logic, so I'll probably give it a try.

Quoted 2 lines

I agree with the concerns and think that how Xe proposes to integrate with drm_sched is a problem, or at least significantly inelegant.

Okay, so it looks like I'm not the only one to be bothered by the way Xe tries to bypass the drm_sched limitations :-).

Quoted 19 lines

AFAICT it proposes to have 1:1 between *userspace* created contexts (per context _and_ engine) and drm_sched. I am not sure avoiding invasive changes to the shared code is in the spirit of the overall idea and instead opportunity should be used to look at way to refactor/improve drm_sched.

Even on the low level, the idea to replace drm_sched threads with workers has a few problems.

To start with, the pattern of:

   while (not_stopped) {
	keep picking jobs
   }

Feels fundamentally in disagreement with workers (while obviously fits perfectly with the current kthread design).

Quoted 7 lines

Secondly, it probably demands separate workers (not optional), otherwise behaviour of shared workqueues has either the potential to explode number kernel threads anyway, or add latency.

What would be interesting to learn is whether the option of refactoring drm_sched to deal with out of order completion was considered and what were the conclusions.

I might be wrong, but I don't think the fundamental issue here is the out-of-order completion thing that's mentioned in the commit message. It just feels like this is a symptom of the impedance mismatch we have between priority+FIFO-based job scheduling and priority+timeslice-based queue scheduling (a queue being represented by a drm_sched_entity in drm_sched).

Quoted 6 lines

Second option perhaps to split out the drm_sched code into parts which would lend themselves more to "pick and choose" of its functionalities. Specifically, Xe wants frontend dependency tracking, but not any scheduling really (neither least busy drm_sched, neither FIFO/RQ entity picking), so even having all these data structures in memory is a waste.

Same thing for the panfrost+CSF driver I was mentioning in my previous emails.

Quoted 6 lines

With the first option then the end result could be drm_sched per engine class (hardware view), which I think fits with the GuC model. Give all schedulable contexts (entities) to the GuC and then mostly forget about them. Timeslicing and re-ordering and all happens transparently to the kernel from that point until completion.

Yep, that would work. I guess it would mean creating an intermediate abstract/interface to schedule entities and then implement this interface for the simple HW-engine+job-scheduling case, so that existing drm_sched users don't see a difference, and new drivers that need to interface with FW-assisted schedulers can implement the higher-lever entity scheduling interface. Don't know what this interface would look like though.

Quoted 4 lines

Or with the second option you would build on some smaller refactored sub-components of drm_sched, by maybe splitting the dependency tracking from scheduling (RR/FIFO entity picking code).

What I've done so far is duplicate the dep-tracking logic in the driver. It's not that much code, but it would be nice to not have to duplicate it in the first place...

Regards,

Boris

author maintainer
#12 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

In reply to Tvrtko Ursulin
Quoted 90 lines · Tvrtko Ursulin
Quoted 80 lines · Boris Brezillon
Quoted 71 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…
Quoted 16 lines · On Fri, 30 Dec 2022 11:20:42 +0100 Boris Brezillon…

Hello Matthew,

Quoted 11 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

Quoted 12 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a FW-assisted scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because I felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof. Giving drm_sched_entity exlusive access to a drm_gpu_scheduler probably might help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue selection is dumb because there's only one entity ever bound to the scheduler (the one that's part of the xe_guc_engine object which also contains the drm_gpu_scheduler instance). I guess the main issue we'd have on Arm is the fact that the stream doesn't necessarily get scheduled when ->run_job() is called, it can be placed in the runnable queue and be picked later by the kernel-side scheduler when a FW slot gets released. That can probably be sorted out by manually disabling the job timer and re-enabling it when the stream gets picked by the scheduler. But my main concern remains, we're basically abusing drm_sched here.

For the Arm driver, that means turning the following sequence

1. wait for job deps
2. queue job to ringbuf and push the stream to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
3. stream gets picked by the timeslice scheduler and sent to the FW for execution

into

1. queue job to entity which takes care of waiting for job deps for us
2. schedule a drm_sched_main iteration
3. the only available entity is picked, and the first job from this entity is dequeued. ->run_job() is called: the job is queued to the ringbuf and the stream is pushed to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
4. stream gets picked by the timeslice scheduler and sent to the FW for execution

Quoted 5 lines

That's one extra step we don't really need. To sum-up, yes, all the job/entity tracking might be interesting to share/re-use, but I wonder if we couldn't have that without pulling out the scheduling part of drm_sched, or maybe I'm missing something, and there's something in drm_gpu_scheduler you really need.

Quoted 7 lines

On second thought, that's probably an acceptable overhead (not even sure the extra step I was mentioning exists in practice, because dep fence signaled state is checked as part of the drm_sched_main iteration, so that's basically replacing the worker I schedule to check job deps), and I like the idea of being able to re-use drm_sched dep-tracking without resorting to invasive changes to the existing logic, so I'll probably give it a try.

Quoted 3 lines

I agree with the concerns and think that how Xe proposes to integrate with drm_sched is a problem, or at least significantly inelegant.

Inelegant is a matter of opinion, I actually rather like this solution.

BTW this isn't my design rather this was Jason's idea.

Quoted 5 lines

AFAICT it proposes to have 1:1 between *userspace* created contexts (per context _and_ engine) and drm_sched. I am not sure avoiding invasive changes to the shared code is in the spirit of the overall idea and instead opportunity should be used to look at way to refactor/improve drm_sched.

Yes, it is 1:1 *userspace* engines and drm_sched.

I'm not really prepared to make large changes to DRM scheduler at the moment for Xe as they are not really required nor does Boris seem they will be required for his work either. I am interested to see what Boris comes up with.

Quoted 12 lines

Even on the low level, the idea to replace drm_sched threads with workers has a few problems.

To start with, the pattern of:

while (not_stopped) {
keep picking jobs
}

Feels fundamentally in disagreement with workers (while obviously fits perfectly with the current kthread design).

The while loop breaks and worker exists if no jobs are ready.

Quoted 4 lines

Secondly, it probably demands separate workers (not optional), otherwise behaviour of shared workqueues has either the potential to explode number kernel threads anyway, or add latency.

Right now the system_unbound_wq is used which does have a limit on the number of threads, right? I do have a FIXME to allow a worker to be passed in similar to TDR.

WRT to latency, the 1:1 ratio could actually have lower latency as 2 GPU schedulers can be pushing jobs into the backend / cleaning up jobs in parallel.

Quoted 4 lines

What would be interesting to learn is whether the option of refactoring drm_sched to deal with out of order completion was considered and what were the conclusions.

I coded this up a while back when trying to convert the i915 to the DRM scheduler it isn't all that hard either. The free flow control on the ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE) is really what sold me on the this design.

Quoted 6 lines

Second option perhaps to split out the drm_sched code into parts which would lend themselves more to "pick and choose" of its functionalities. Specifically, Xe wants frontend dependency tracking, but not any scheduling really (neither least busy drm_sched, neither FIFO/RQ entity picking), so even having all these data structures in memory is a waste.

I don't think that we are wasting memory is a very good argument for making intrusive changes to the DRM scheduler.

Quoted 6 lines

With the first option then the end result could be drm_sched per engine class (hardware view), which I think fits with the GuC model. Give all schedulable contexts (entities) to the GuC and then mostly forget about them. Timeslicing and re-ordering and all happens transparently to the kernel from that point until completion.

Out-of-order problem still exists here.

Quoted 8 lines

Or with the second option you would build on some smaller refactored sub-components of drm_sched, by maybe splitting the dependency tracking from scheduling (RR/FIFO entity picking code).

Second option is especially a bit vague and I haven't thought about the required mechanics, but it just appeared too obvious the proposed design has a bit too much impedance mismatch.

IMO ROI on this is low and again lets see what Boris comes up with.

Matt

Quoted 9 lines

Oh and as a side note, when I went into the drm_sched code base to remind myself how things worked, it is quite easy to find some FIXME comments which suggest people working on it are unsure of locking desing there and such. So perhaps that all needs cleanup too, I mean would benefit from refactoring/improving work as brainstormed above anyway.

Regards,

Tvrtko

author maintainer
#13 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 132 lines · Matthew Brost
Quoted 90 lines · Tvrtko Ursulin
Quoted 80 lines · Boris Brezillon
Quoted 71 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…
Quoted 16 lines · On Fri, 30 Dec 2022 11:20:42 +0100 Boris Brezillon…

Hello Matthew,

Quoted 11 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

Quoted 12 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a FW-assisted scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because I felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof. Giving drm_sched_entity exlusive access to a drm_gpu_scheduler probably might help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue selection is dumb because there's only one entity ever bound to the scheduler (the one that's part of the xe_guc_engine object which also contains the drm_gpu_scheduler instance). I guess the main issue we'd have on Arm is the fact that the stream doesn't necessarily get scheduled when ->run_job() is called, it can be placed in the runnable queue and be picked later by the kernel-side scheduler when a FW slot gets released. That can probably be sorted out by manually disabling the job timer and re-enabling it when the stream gets picked by the scheduler. But my main concern remains, we're basically abusing drm_sched here.

For the Arm driver, that means turning the following sequence

1. wait for job deps
2. queue job to ringbuf and push the stream to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
3. stream gets picked by the timeslice scheduler and sent to the FW for execution

into

1. queue job to entity which takes care of waiting for job deps for us
2. schedule a drm_sched_main iteration
3. the only available entity is picked, and the first job from this entity is dequeued. ->run_job() is called: the job is queued to the ringbuf and the stream is pushed to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
4. stream gets picked by the timeslice scheduler and sent to the FW for execution

Quoted 5 lines

That's one extra step we don't really need. To sum-up, yes, all the job/entity tracking might be interesting to share/re-use, but I wonder if we couldn't have that without pulling out the scheduling part of drm_sched, or maybe I'm missing something, and there's something in drm_gpu_scheduler you really need.

Quoted 7 lines

On second thought, that's probably an acceptable overhead (not even sure the extra step I was mentioning exists in practice, because dep fence signaled state is checked as part of the drm_sched_main iteration, so that's basically replacing the worker I schedule to check job deps), and I like the idea of being able to re-use drm_sched dep-tracking without resorting to invasive changes to the existing logic, so I'll probably give it a try.

Quoted 3 lines

I agree with the concerns and think that how Xe proposes to integrate with drm_sched is a problem, or at least significantly inelegant.

Inelegant is a matter of opinion, I actually rather like this solution.

BTW this isn't my design rather this was Jason's idea.

Quoted 5 lines

AFAICT it proposes to have 1:1 between *userspace* created contexts (per context _and_ engine) and drm_sched. I am not sure avoiding invasive changes to the shared code is in the spirit of the overall idea and instead opportunity should be used to look at way to refactor/improve drm_sched.

Yes, it is 1:1 *userspace* engines and drm_sched.

I'm not really prepared to make large changes to DRM scheduler at the moment for Xe as they are not really required nor does Boris seem they will be required for his work either. I am interested to see what Boris comes up with.

Quoted 12 lines

Even on the low level, the idea to replace drm_sched threads with workers has a few problems.

To start with, the pattern of:

while (not_stopped) {
keep picking jobs
}

Feels fundamentally in disagreement with workers (while obviously fits perfectly with the current kthread design).

The while loop breaks and worker exists if no jobs are ready.

Quoted 4 lines

Secondly, it probably demands separate workers (not optional), otherwise behaviour of shared workqueues has either the potential to explode number kernel threads anyway, or add latency.

Quoted 8 lines

Right now the system_unbound_wq is used which does have a limit on the number of threads, right? I do have a FIXME to allow a worker to be passed in similar to TDR.

WRT to latency, the 1:1 ratio could actually have lower latency as 2 GPU schedulers can be pushing jobs into the backend / cleaning up jobs in parallel.

Thought of one more point here where why in Xe we absolutely want a 1 to 1 ratio between entity and scheduler - the way we implement timeslicing for preempt fences.

Let me try to explain.

Preempt fences are implemented via the generic messaging interface [1] with suspend / resume messages. If a suspend messages is received to soon after calling resume (this is per entity) we simply sleep in the suspend call thus giving the entity a timeslice. This completely falls apart with a many to 1 relationship as now a entity waiting for a timeslice blocks the other entities. Could we work aroudn this, sure but just another bunch of code we'd have to add in Xe. Being to freely sleep in backend without affecting other entities is really, really nice IMO and I bet Xe isn't the only driver that is going to feel this way.

Last thing I'll say regardless of how anyone feels about Xe using a 1 to 1 relationship this patch IMO makes sense as I hope we can all agree a workqueue scales better than kthreads.

Matt

[1] https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1

Quoted 51 lines
Quoted 4 lines

What would be interesting to learn is whether the option of refactoring drm_sched to deal with out of order completion was considered and what were the conclusions.

I coded this up a while back when trying to convert the i915 to the DRM scheduler it isn't all that hard either. The free flow control on the ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE) is really what sold me on the this design.

Quoted 6 lines

Second option perhaps to split out the drm_sched code into parts which would lend themselves more to "pick and choose" of its functionalities. Specifically, Xe wants frontend dependency tracking, but not any scheduling really (neither least busy drm_sched, neither FIFO/RQ entity picking), so even having all these data structures in memory is a waste.

I don't think that we are wasting memory is a very good argument for making intrusive changes to the DRM scheduler.

Quoted 6 lines

With the first option then the end result could be drm_sched per engine class (hardware view), which I think fits with the GuC model. Give all schedulable contexts (entities) to the GuC and then mostly forget about them. Timeslicing and re-ordering and all happens transparently to the kernel from that point until completion.

Out-of-order problem still exists here.

Quoted 8 lines

Or with the second option you would build on some smaller refactored sub-components of drm_sched, by maybe splitting the dependency tracking from scheduling (RR/FIFO entity picking code).

Second option is especially a bit vague and I haven't thought about the required mechanics, but it just appeared too obvious the proposed design has a bit too much impedance mismatch.

IMO ROI on this is low and again lets see what Boris comes up with.

Matt

Quoted 9 lines

Oh and as a side note, when I went into the drm_sched code base to remind myself how things worked, it is quite easy to find some FIXME comments which suggest people working on it are unsure of locking desing there and such. So perhaps that all needs cleanup too, I mean would benefit from refactoring/improving work as brainstormed above anyway.

Regards,

Tvrtko

#14 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 158 lines · Matthew Brost
Quoted 132 lines · Matthew Brost
Quoted 90 lines · Tvrtko Ursulin
Quoted 80 lines · Boris Brezillon
Quoted 71 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…
Quoted 16 lines · On Fri, 30 Dec 2022 11:20:42 +0100 Boris Brezillon…

Hello Matthew,

Quoted 11 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

Quoted 12 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a FW-assisted scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because I felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof. Giving drm_sched_entity exlusive access to a drm_gpu_scheduler probably might help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue selection is dumb because there's only one entity ever bound to the scheduler (the one that's part of the xe_guc_engine object which also contains the drm_gpu_scheduler instance). I guess the main issue we'd have on Arm is the fact that the stream doesn't necessarily get scheduled when ->run_job() is called, it can be placed in the runnable queue and be picked later by the kernel-side scheduler when a FW slot gets released. That can probably be sorted out by manually disabling the job timer and re-enabling it when the stream gets picked by the scheduler. But my main concern remains, we're basically abusing drm_sched here.

For the Arm driver, that means turning the following sequence

1. wait for job deps
2. queue job to ringbuf and push the stream to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
3. stream gets picked by the timeslice scheduler and sent to the FW for execution

into

1. queue job to entity which takes care of waiting for job deps for us
2. schedule a drm_sched_main iteration
3. the only available entity is picked, and the first job from this entity is dequeued. ->run_job() is called: the job is queued to the ringbuf and the stream is pushed to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
4. stream gets picked by the timeslice scheduler and sent to the FW for execution

Quoted 5 lines

That's one extra step we don't really need. To sum-up, yes, all the job/entity tracking might be interesting to share/re-use, but I wonder if we couldn't have that without pulling out the scheduling part of drm_sched, or maybe I'm missing something, and there's something in drm_gpu_scheduler you really need.

Quoted 7 lines

On second thought, that's probably an acceptable overhead (not even sure the extra step I was mentioning exists in practice, because dep fence signaled state is checked as part of the drm_sched_main iteration, so that's basically replacing the worker I schedule to check job deps), and I like the idea of being able to re-use drm_sched dep-tracking without resorting to invasive changes to the existing logic, so I'll probably give it a try.

Quoted 3 lines

I agree with the concerns and think that how Xe proposes to integrate with drm_sched is a problem, or at least significantly inelegant.

Inelegant is a matter of opinion, I actually rather like this solution.

BTW this isn't my design rather this was Jason's idea.

Quoted 5 lines

AFAICT it proposes to have 1:1 between *userspace* created contexts (per context _and_ engine) and drm_sched. I am not sure avoiding invasive changes to the shared code is in the spirit of the overall idea and instead opportunity should be used to look at way to refactor/improve drm_sched.

Yes, it is 1:1 *userspace* engines and drm_sched.

I'm not really prepared to make large changes to DRM scheduler at the moment for Xe as they are not really required nor does Boris seem they will be required for his work either. I am interested to see what Boris comes up with.

Quoted 12 lines

Even on the low level, the idea to replace drm_sched threads with workers has a few problems.

To start with, the pattern of:

   while (not_stopped) {
	keep picking jobs
   }

Feels fundamentally in disagreement with workers (while obviously fits perfectly with the current kthread design).

The while loop breaks and worker exists if no jobs are ready.

Quoted 4 lines

Secondly, it probably demands separate workers (not optional), otherwise behaviour of shared workqueues has either the potential to explode number kernel threads anyway, or add latency.

Quoted 8 lines

Right now the system_unbound_wq is used which does have a limit on the number of threads, right? I do have a FIXME to allow a worker to be passed in similar to TDR.

WRT to latency, the 1:1 ratio could actually have lower latency as 2 GPU schedulers can be pushing jobs into the backend / cleaning up jobs in parallel.

Thought of one more point here where why in Xe we absolutely want a 1 to 1 ratio between entity and scheduler - the way we implement timeslicing for preempt fences.

Let me try to explain.

Preempt fences are implemented via the generic messaging interface [1] with suspend / resume messages. If a suspend messages is received to soon after calling resume (this is per entity) we simply sleep in the suspend call thus giving the entity a timeslice. This completely falls apart with a many to 1 relationship as now a entity waiting for a timeslice blocks the other entities. Could we work aroudn this, sure but just another bunch of code we'd have to add in Xe. Being to freely sleep in backend without affecting other entities is really, really nice IMO and I bet Xe isn't the only driver that is going to feel this way.

Quoted 3 lines

Last thing I'll say regardless of how anyone feels about Xe using a 1 to 1 relationship this patch IMO makes sense as I hope we can all agree a workqueue scales better than kthreads.

I don't know for sure what will scale better and for what use case, combination of CPU cores vs number of GPU engines to keep busy vs other system activity. But I wager someone is bound to ask for some numbers to make sure proposal is not negatively affecting any other drivers.

In any case that's a low level question caused by the high level design decision. So I'd think first focus on the high level - which is the 1:1 mapping of entity to scheduler instance proposal.

Fundamentally it will be up to the DRM maintainers and the community to bless your approach. And it is important to stress 1:1 is about userspace contexts, so I believe unlike any other current scheduler user. And also important to stress this effectively does not make Xe _really_ use the scheduler that much.

I can only offer my opinion, which is that the two options mentioned in this thread (either improve drm scheduler to cope with what is required, or split up the code so you can use just the parts of drm_sched which you want - which is frontend dependency tracking) shouldn't be so readily dismissed, given how I think the idea was for the new driver to work less in a silo and more in the community (not do kludges to workaround stuff because it is thought to be too hard to improve common code), but fundamentally, "goto previous paragraph" for what I am concerned.

Regards,

Tvrtko

P.S. And as a related side note, there are more areas where drm_sched could be improved, like for instance priority handling.
Take a look at msm_submitqueue_create / msm_gpu_convert_priority / get_sched_entity to see how msm works around the drm_sched hardcoded limit of available priority levels, in order to avoid having to leave a hw capability unused. I suspect msm would be happier if they could have all priority levels equal in terms of whether they apply only at the frontend level or completely throughout the pipeline.

Quoted 53 lines

[1] https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1

Quoted 51 lines
Quoted 4 lines

What would be interesting to learn is whether the option of refactoring drm_sched to deal with out of order completion was considered and what were the conclusions.

I coded this up a while back when trying to convert the i915 to the DRM scheduler it isn't all that hard either. The free flow control on the ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE) is really what sold me on the this design.

Quoted 6 lines

Second option perhaps to split out the drm_sched code into parts which would lend themselves more to "pick and choose" of its functionalities. Specifically, Xe wants frontend dependency tracking, but not any scheduling really (neither least busy drm_sched, neither FIFO/RQ entity picking), so even having all these data structures in memory is a waste.

I don't think that we are wasting memory is a very good argument for making intrusive changes to the DRM scheduler.

Quoted 6 lines

With the first option then the end result could be drm_sched per engine class (hardware view), which I think fits with the GuC model. Give all schedulable contexts (entities) to the GuC and then mostly forget about them. Timeslicing and re-ordering and all happens transparently to the kernel from that point until completion.

Out-of-order problem still exists here.

Quoted 8 lines

Or with the second option you would build on some smaller refactored sub-components of drm_sched, by maybe splitting the dependency tracking from scheduling (RR/FIFO entity picking code).

Second option is especially a bit vague and I haven't thought about the required mechanics, but it just appeared too obvious the proposed design has a bit too much impedance mismatch.

IMO ROI on this is low and again lets see what Boris comes up with.

Matt

Quoted 9 lines

Oh and as a side note, when I went into the drm_sched code base to remind myself how things worked, it is quite easy to find some FIXME comments which suggest people working on it are unsure of locking desing there and such. So perhaps that all needs cleanup too, I mean would benefit from refactoring/improving work as brainstormed above anyway.

Regards,

Tvrtko

#15 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 116 lines · Tvrtko Ursulin
Quoted 10 lines · Matthew Brost
Quoted 9 lines · Matthew Brost
Quoted 8 lines · Tvrtko Ursulin
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also

because I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think

I

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue selection is dumb because there's only one entity ever bound to the scheduler (the one that's part of the xe_guc_engine object which also contains the drm_gpu_scheduler instance). I guess the main issue we'd have on Arm is the fact that the stream doesn't necessarily get scheduled when ->run_job() is called, it can be placed in the

runnable

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

queue and be picked later by the kernel-side scheduler when a FW slot gets released. That can probably be sorted out by manually disabling

the

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

job timer and re-enabling it when the stream gets picked by the scheduler. But my main concern remains, we're basically abusing drm_sched here.

For the Arm driver, that means turning the following sequence

1. wait for job deps
2. queue job to ringbuf and push the stream to the runnable queue (if it wasn't queued already). Wakeup the timeslice

scheduler

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

to re-evaluate (if the stream is not on a FW slot already)
3. stream gets picked by the timeslice scheduler and sent to the FW

for

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

execution

into

1. queue job to entity which takes care of waiting for job deps for us
2. schedule a drm_sched_main iteration
3. the only available entity is picked, and the first job from this

     entity is dequeued. ->run_job() is called: the job is queued to

the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

ringbuf and the stream is pushed to the runnable queue (if it

wasn't

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

queued already). Wakeup the timeslice scheduler to re-evaluate

(if

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the stream is not on a FW slot already)
4. stream gets picked by the timeslice scheduler and sent to the FW

for

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

execution

That's one extra step we don't really need. To sum-up, yes, all the job/entity tracking might be interesting to share/re-use, but I

wonder

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines

if we couldn't have that without pulling out the scheduling part of drm_sched, or maybe I'm missing something, and there's something in drm_gpu_scheduler you really need.

On second thought, that's probably an acceptable overhead (not even sure the extra step I was mentioning exists in practice, because dep fence signaled state is checked as part of the drm_sched_main iteration, so that's basically replacing the worker I schedule to check job deps), and I like the idea of being able to re-use drm_sched dep-tracking without resorting to invasive changes to the existing logic, so I'll probably give it a try.

Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

I agree with the concerns and think that how Xe proposes to integrate

with

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

drm_sched is a problem, or at least significantly inelegant.

Inelegant is a matter of opinion, I actually rather like this solution.

BTW this isn't my design rather this was Jason's idea.

Sure, throw me under the bus, why don't you? :-P Nah, it's all fine. It's my design and I'm happy to defend it or be blamed for it in the history books as the case may be.

Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

AFAICT it proposes to have 1:1 between *userspace* created contexts

(per

Quoted 1 line
Quoted 1 line
Quoted 1 line

context _and_ engine) and drm_sched. I am not sure avoiding invasive

changes

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

to the shared code is in the spirit of the overall idea and instead opportunity should be used to look at way to refactor/improve

drm_sched.

Maybe? I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor. (More on that later.) There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier. Not ideal, but not the end of the world either. Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

Quoted 22 lines
Quoted 8 lines
Quoted 8 lines

Yes, it is 1:1 *userspace* engines and drm_sched.

I'm not really prepared to make large changes to DRM scheduler at the moment for Xe as they are not really required nor does Boris seem they will be required for his work either. I am interested to see what Boris comes up with.

Quoted 1 line

Even on the low level, the idea to replace drm_sched threads with

workers

Quoted 12 lines
Quoted 12 lines
Quoted 10 lines

has a few problems.

To start with, the pattern of:

   while (not_stopped) {
    keep picking jobs
   }

Feels fundamentally in disagreement with workers (while obviously fits perfectly with the current kthread design).

The while loop breaks and worker exists if no jobs are ready.

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Quoted 32 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Secondly, it probably demands separate workers (not optional),

otherwise

Quoted 1 line
Quoted 1 line
Quoted 1 line

behaviour of shared workqueues has either the potential to explode

number

Quoted 28 lines
Quoted 11 lines
Quoted 2 lines

kernel threads anyway, or add latency.

Right now the system_unbound_wq is used which does have a limit on the number of threads, right? I do have a FIXME to allow a worker to be passed in similar to TDR.

WRT to latency, the 1:1 ratio could actually have lower latency as 2 GPU schedulers can be pushing jobs into the backend / cleaning up jobs in parallel.

Thought of one more point here where why in Xe we absolutely want a 1 to 1 ratio between entity and scheduler - the way we implement timeslicing for preempt fences.

Let me try to explain.

Preempt fences are implemented via the generic messaging interface [1] with suspend / resume messages. If a suspend messages is received to soon after calling resume (this is per entity) we simply sleep in the suspend call thus giving the entity a timeslice. This completely falls apart with a many to 1 relationship as now a entity waiting for a timeslice blocks the other entities. Could we work aroudn this, sure but just another bunch of code we'd have to add in Xe. Being to freely sleep in backend without affecting other entities is really, really nice IMO and I bet Xe isn't the only driver that is going to feel this way.

Quoted 9 lines
Quoted 3 lines

Last thing I'll say regardless of how anyone feels about Xe using a 1 to 1 relationship this patch IMO makes sense as I hope we can all agree a workqueue scales better than kthreads.

I don't know for sure what will scale better and for what use case, combination of CPU cores vs number of GPU engines to keep busy vs other system activity. But I wager someone is bound to ask for some numbers to make sure proposal is not negatively affecting any other drivers.

Then let them ask. Waving your hands vaguely in the direction of the rest of DRM and saying "Uh, someone (not me) might object" is profoundly unhelpful. Sure, someone might. That's why it's on dri-devel. If you think there's someone in particular who might have a useful opinion on this, throw them in the CC so they don't miss the e-mail thread.

Or are you asking for numbers? If so, what numbers are you asking for?

Also, If we're talking about a design that might paint us into an Intel-HW-specific hole, that would be one thing. But we're not. We're talking about switching which kernel threading/task mechanism to use for what's really a very generic problem. The core Xe design works without this patch (just with more kthreads). If we land this patch or something like it and get it wrong and it causes a performance problem for someone down the line, we can revisit it.

Quoted 10 lines

In any case that's a low level question caused by the high level design decision. So I'd think first focus on the high level - which is the 1:1 mapping of entity to scheduler instance proposal.

Fundamentally it will be up to the DRM maintainers and the community to bless your approach. And it is important to stress 1:1 is about userspace contexts, so I believe unlike any other current scheduler user. And also important to stress this effectively does not make Xe _really_ use the scheduler that much.

I don't think this makes Xe nearly as much of a one-off as you think it does. I've already told the Asahi team working on Apple M1/2 hardware to do it this way and it seems to be a pretty good mapping for them. I believe this is roughly the plan for nouveau as well. It's not the way it currently works for anyone because most other groups aren't doing FW scheduling yet. In the world of FW scheduling and hardware designed to support userspace direct-to-FW submit, I think the design makes perfect sense (see below) and I expect we'll see more drivers move in this direction as those drivers evolve. (AMD is doing some customish thing for how with gpu_scheduler on the front-end somehow. I've not dug into those details.)

Quoted 10 lines

I can only offer my opinion, which is that the two options mentioned in this thread (either improve drm scheduler to cope with what is required, or split up the code so you can use just the parts of drm_sched which you want - which is frontend dependency tracking) shouldn't be so readily dismissed, given how I think the idea was for the new driver to work less in a silo and more in the community (not do kludges to workaround stuff because it is thought to be too hard to improve common code), but fundamentally, "goto previous paragraph" for what I am concerned.

Meta comment: It appears as if you're falling into the standard i915 team trap of having an internal discussion about what the community discussion might look like instead of actually having the community discussion. If you are seriously concerned about interactions with other drivers or whether or setting common direction, the right way to do that is to break a patch or two out into a separate RFC series and tag a handful of driver maintainers. Trying to predict the questions other people might ask is pointless. Cc them and asking for their input instead.

Quoted 13 lines

Regards,

Tvrtko

P.S. And as a related side note, there are more areas where drm_sched could be improved, like for instance priority handling.
Take a look at msm_submitqueue_create / msm_gpu_convert_priority / get_sched_entity to see how msm works around the drm_sched hardcoded limit of available priority levels, in order to avoid having to leave a hw capability unused. I suspect msm would be happier if they could have all priority levels equal in terms of whether they apply only at the frontend level or completely throughout the pipeline.

Quoted 13 lines
Quoted 4 lines

[1] https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1

Quoted 2 lines
Quoted 2 lines

What would be interesting to learn is whether the option of refactoring drm_sched to deal with out of order completion was considered and what

were

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

the conclusions.

I coded this up a while back when trying to convert the i915 to the DRM scheduler it isn't all that hard either. The free flow control on the ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE) is really what sold me on the this design.

You're not the only one to suggest supporting out-of-order completion. However, it's tricky and breaks a lot of internal assumptions of the scheduler. It also reduces functionality a bit because it can no longer automatically rate-limit HW/FW queues which are often fixed-size. (Ok, yes, it probably could but it becomes a substantially harder problem.)

It also seems like a worse mapping to me. The goal here is to turn submissions on a userspace-facing engine/queue into submissions to a FW queue submissions, sorting out any dma_fence dependencies. Matt's description of saying this is a 1:1 mapping between sched/entity doesn't tell the whole story. It's a 1:1:1 mapping between xe_engine, gpu_scheduler, and GuC FW engine. Why make it a 1:something:1 mapping? Why is that better?

There are two places where this 1:1:1 mapping is causing problems:

1. It creates lots of kthreads. This is what this patch is trying to
solve. IDK if it's solving it the best way but that's the goal.

2. There are a far more limited number of communication queues between the
kernel and GuC for more meta things like pausing and resuming queues, getting events back from GuC, etc. Unless we're in a weird pressure scenario, the amount of traffic on this queue should be low so we can probably just have one per physical device. The vast majority of kernel -> GuC communication should be on the individual FW queue rings and maybe smashing in-memory doorbells.

Doing out-of-order completion sort-of solves the 1 but does nothing for 2 and actually makes managing FW queues harder because we no longer have built-in rate limiting. Seems like a net loss to me.

Quoted 13 lines
Quoted 1 line
Quoted 1 line

Second option perhaps to split out the drm_sched code into parts which

would

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

lend themselves more to "pick and choose" of its functionalities. Specifically, Xe wants frontend dependency tracking, but not any

scheduling

Quoted 1 line
Quoted 1 line
Quoted 1 line

really (neither least busy drm_sched, neither FIFO/RQ entity picking),

so

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

even having all these data structures in memory is a waste.

I don't think that we are wasting memory is a very good argument for making intrusive changes to the DRM scheduler.

Worse than that, I think the "we could split it up" kind-of misses the point of the way Xe is using drm/scheduler. It's not just about re-using a tiny bit of dependency tracking code. Using the scheduler in this way provides a clean separation between front-end and back-end. The job of the userspace-facing ioctl code is to shove things on the scheduler. The job of the run_job callback is to encode the job into the FW queue format, stick it in the FW queue ring, and maybe smash a doorbell. Everything else happens in terms of managing those queues side-band. The gpu_scheduler code manages the front-end queues and Xe manages the FW queues via the Kernel <-> GuC communication rings. From a high level, this is a really clean design. There are potentially some sticky bits around the dual-use of dma_fence for scheduling and memory management but none of those are solved by breaking the DRM scheduler into chunks or getting rid of the 1:1:1 mapping.

If we split it out, we're basically asking the driver to implement a bunch of kthread or workqueue stuff, all the ring rate-limiting, etc. It may not be all that much code but also, why? To save a few bytes of memory per engine? Each engine already has 32K(ish) worth of context state and a similar size ring to communicate with the FW. No one is going to notice an extra CPU data structure.

I'm not seeing a solid argument against the 1:1:1 design here other than that it doesn't seem like the way DRM scheduler was intended to be used. I won't argue that. It's not. But it is a fairly natural way to take advantage of the benefits the DRM scheduler does provide while also mapping it to hardware that was designed for userspace direct-to-FW submit.

--Jason

Quoted 37 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines

With the first option then the end result could be drm_sched per engine class (hardware view), which I think fits with the GuC model. Give all schedulable contexts (entities) to the GuC and then mostly forget about them. Timeslicing and re-ordering and all happens transparently to the kernel from that point until completion.

Out-of-order problem still exists here.

Quoted 2 lines

Or with the second option you would build on some smaller refactored sub-components of drm_sched, by maybe splitting the dependency

tracking from

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

scheduling (RR/FIFO entity picking code).

Second option is especially a bit vague and I haven't thought about the required mechanics, but it just appeared too obvious the proposed

design has

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

a bit too much impedance mismatch.

IMO ROI on this is low and again lets see what Boris comes up with.

Matt

Quoted 1 line

Oh and as a side note, when I went into the drm_sched code base to

remind

Quoted 1 line
Quoted 1 line
Quoted 1 line

myself how things worked, it is quite easy to find some FIXME comments

which

Quoted 1 line
Quoted 1 line
Quoted 1 line

suggest people working on it are unsure of locking desing there and

such. So

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

perhaps that all needs cleanup too, I mean would benefit from refactoring/improving work as brainstormed above anyway.

Regards,

Tvrtko

#16 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 38 lines

>>> AFAICT it proposes to have 1:1 between *userspace* created
contexts (per
>>> context _and_ engine) and drm_sched. I am not sure avoiding
invasive changes
>>> to the shared code is in the spirit of the overall idea and instead >>> opportunity should be used to look at way to refactor/improve
drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.)  There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

>> Yes, it is 1:1 *userspace* engines and drm_sched. >>
>> I'm not really prepared to make large changes to DRM scheduler
at the
>> moment for Xe as they are not really required nor does Boris
seem they
>> will be required for his work either. I am interested to see
what Boris
>> comes up with.
>>
>>> Even on the low level, the idea to replace drm_sched threads
with workers
>>> has a few problems.
>>>
>>> To start with, the pattern of:
>>>
>>>    while (not_stopped) {
>>>     keep picking jobs
>>>    }
>>>
>>> Feels fundamentally in disagreement with workers (while
obviously fits
>>> perfectly with the current kthread design).
>>
>> The while loop breaks and worker exists if no jobs are ready.

Quoted 1 line

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal. """

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

Unbound workers have no CPU / cache locality either and no connection with the CPU scheduler to optimize scheduling patterns. This may matter either on large systems or on small ones. Whereas the current design allows for scheduler to notice userspace CPU thread keeps waking up the same drm scheduler kernel thread, and so it can keep them on the same CPU, the unbound workers lose that ability and so 2nd CPU might be getting woken up from low sleep for every submission.

Hence, apart from being a bit of a impedance mismatch, the proposal has the potential to change performance and power patterns and both large and small machines.

Quoted 55 lines

>>> Secondly, it probably demands separate workers (not optional),
otherwise
>>> behaviour of shared workqueues has either the potential to
explode number
>>> kernel threads anyway, or add latency.
>>>
>>
>> Right now the system_unbound_wq is used which does have a limit
on the
>> number of threads, right? I do have a FIXME to allow a worker to be >> passed in similar to TDR.
>>
>> WRT to latency, the 1:1 ratio could actually have lower latency
as 2 GPU
>> schedulers can be pushing jobs into the backend / cleaning up
jobs in
>> parallel.
>>
>
> Thought of one more point here where why in Xe we absolutely want
a 1 to
> 1 ratio between entity and scheduler - the way we implement
timeslicing
> for preempt fences.
>
> Let me try to explain.
>
> Preempt fences are implemented via the generic messaging
interface [1]
> with suspend / resume messages. If a suspend messages is received to > soon after calling resume (this is per entity) we simply sleep in the > suspend call thus giving the entity a timeslice. This completely
falls
> apart with a many to 1 relationship as now a entity waiting for a > timeslice blocks the other entities. Could we work aroudn this,
sure but
> just another bunch of code we'd have to add in Xe. Being to
freely sleep
> in backend without affecting other entities is really, really
nice IMO
> and I bet Xe isn't the only driver that is going to feel this way. >
> Last thing I'll say regardless of how anyone feels about Xe using
a 1 to
> 1 relationship this patch IMO makes sense as I hope we can all
agree a
> workqueue scales better than kthreads.

I don't know for sure what will scale better and for what use case, combination of CPU cores vs number of GPU engines to keep busy vs other system activity. But I wager someone is bound to ask for some numbers to
make sure proposal is not negatively affecting any other drivers.

Quoted 3 lines

Then let them ask.  Waving your hands vaguely in the direction of the rest of DRM and saying "Uh, someone (not me) might object" is profoundly unhelpful.  Sure, someone might.  That's why it's on dri-devel.  If you think there's someone in particular who might have a useful opinion on this, throw them in the CC so they don't miss the e-mail thread.

Or are you asking for numbers?  If so, what numbers are you asking for?

It was a heads up to the Xe team in case people weren't appreciating how the proposed change has the potential influence power and performance across the board. And nothing in the follow up discussion made me think it was considered so I don't think it was redundant to raise it.

In my experience it is typical that such core changes come with some numbers. Which is in case of drm scheduler is tricky and probably requires explicitly asking everyone to test (rather than count on "don't miss the email thread"). Real products can fail to ship due ten mW here or there. Like suddenly an extra core prevented from getting into deep sleep.

If that was "profoundly unhelpful" so be it.

Quoted 1 line

Also, If we're talking about a design that might paint us into an Intel-HW-specific hole, that would be one thing.  But we're not.  We're talking about switching which kernel threading/task mechanism to use for what's really a very generic problem.  The core Xe design works without this patch (just with more kthreads).  If we land this patch or something like it and get it wrong and it causes a performance problem for someone down the line, we can revisit it.

For some definition of "it works" - I really wouldn't suggest shipping a kthread per user context at any point.

Quoted 25 lines

In any case that's a low level question caused by the high level design decision. So I'd think first focus on the high level - which is the 1:1 mapping of entity to scheduler instance proposal.

Fundamentally it will be up to the DRM maintainers and the community to bless your approach. And it is important to stress 1:1 is about userspace contexts, so I believe unlike any other current scheduler user. And also important to stress this effectively does not make Xe _really_ use the scheduler that much.

I don't think this makes Xe nearly as much of a one-off as you think it does.  I've already told the Asahi team working on Apple M1/2 hardware to do it this way and it seems to be a pretty good mapping for them. I believe this is roughly the plan for nouveau as well.  It's not the way it currently works for anyone because most other groups aren't doing FW scheduling yet.  In the world of FW scheduling and hardware designed to support userspace direct-to-FW submit, I think the design makes perfect sense (see below) and I expect we'll see more drivers move in this direction as those drivers evolve.  (AMD is doing some customish thing for how with gpu_scheduler on the front-end somehow. I've not dug into those details.)

I can only offer my opinion, which is that the two options mentioned in this thread (either improve drm scheduler to cope with what is required,
or split up the code so you can use just the parts of drm_sched which you want - which is frontend dependency tracking) shouldn't be so readily dismissed, given how I think the idea was for the new driver to work less in a silo and more in the community (not do kludges to workaround stuff because it is thought to be too hard to improve common code), but fundamentally, "goto previous paragraph" for what I am concerned.

Quoted 1 line

Meta comment:  It appears as if you're falling into the standard i915 team trap of having an internal discussion about what the community discussion might look like instead of actually having the community discussion.  If you are seriously concerned about interactions with other drivers or whether or setting common direction, the right way to do that is to break a patch or two out into a separate RFC series and tag a handful of driver maintainers.  Trying to predict the questions other people might ask is pointless. Cc them and asking for their input instead.

I don't follow you here. It's not an internal discussion - I am raising my concerns on the design publicly. I am supposed to write a patch to show something, but am allowed to comment on a RFC series?

It is "drm/sched: Convert drm scheduler to use a work queue rather than kthread" which should have Cc-ed _everyone_ who use drm scheduler.

Quoted 34 lines

Regards,

Tvrtko

P.S. And as a related side note, there are more areas where drm_sched could be improved, like for instance priority handling. Take a look at msm_submitqueue_create / msm_gpu_convert_priority / get_sched_entity to see how msm works around the drm_sched hardcoded limit of available priority levels, in order to avoid having to leave a hw capability unused. I suspect msm would be happier if they could have all priority levels equal in terms of whether they apply only at the frontend level or completely throughout the pipeline.

> [1]
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
>
>>> What would be interesting to learn is whether the option of
refactoring
>>> drm_sched to deal with out of order completion was considered
and what were
>>> the conclusions.
>>>
>>
>> I coded this up a while back when trying to convert the i915 to
the DRM
>> scheduler it isn't all that hard either. The free flow control
on the
>> ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE) is
really what
>> sold me on the this design.

Quoted 3 lines

You're not the only one to suggest supporting out-of-order completion. However, it's tricky and breaks a lot of internal assumptions of the scheduler. It also reduces functionality a bit because it can no longer automatically rate-limit HW/FW queues which are often fixed-size.  (Ok, yes, it probably could but it becomes a substantially harder problem.)

It also seems like a worse mapping to me.  The goal here is to turn submissions on a userspace-facing engine/queue into submissions to a FW queue submissions, sorting out any dma_fence dependencies.  Matt's description of saying this is a 1:1 mapping between sched/entity doesn't tell the whole story. It's a 1:1:1 mapping between xe_engine, gpu_scheduler, and GuC FW engine.  Why make it a 1:something:1 mapping? Why is that better?

As I have stated before, what I think what would fit well for Xe is one drm_scheduler per engine class. In specific terms on our current hardware, one drm scheduler instance for render, compute, blitter, video and video enhance. Userspace contexts remain scheduler entities.

That way you avoid the whole kthread/kworker story and you have it actually use the entity picking code in the scheduler, which may be useful when the backend is congested.

Yes you have to solve the out of order problem so in my mind that is something to discuss. What the problem actually is (just TDR?), how tricky and why etc.

And yes you lose the handy LRCA ring buffer size management so you'd have to make those entities not runnable in some other way.

Regarding the argument you raise below - would any of that make the frontend / backend separation worse and why? Do you think it is less natural? If neither is true then all remains is that it appears extra work to support out of order completion of entities has been discounted in favour of an easy but IMO inelegant option.

Quoted 7 lines

There are two places where this 1:1:1 mapping is causing problems:

 1. It creates lots of kthreads. This is what this patch is trying to
solve. IDK if it's solving it the best way but that's the goal.

 2. There are a far more limited number of communication queues between
the kernel and GuC for more meta things like pausing and resuming queues, getting events back from GuC, etc. Unless we're in a weird pressure scenario, the amount of traffic on this queue should be low so we can probably just have one per physical device.  The vast majority of kernel -> GuC communication should be on the individual FW queue rings and maybe smashing in-memory doorbells.

I don't follow your terminology here. I suppose you are talking about global GuC CT and context ringbuffers. If so then isn't "far more limited" actually one?

Regards,

Tvrtko

Quoted 67 lines

Doing out-of-order completion sort-of solves the 1 but does nothing for 2 and actually makes managing FW queues harder because we no longer have built-in rate limiting.  Seems like a net loss to me.

>>> Second option perhaps to split out the drm_sched code into
parts which would
>>> lend themselves more to "pick and choose" of its functionalities. >>> Specifically, Xe wants frontend dependency tracking, but not
any scheduling
>>> really (neither least busy drm_sched, neither FIFO/RQ entity
picking), so
>>> even having all these data structures in memory is a waste. >>>
>>
>> I don't think that we are wasting memory is a very good argument for >> making intrusive changes to the DRM scheduler.

Worse than that, I think the "we could split it up" kind-of misses the point of the way Xe is using drm/scheduler.  It's not just about re-using a tiny bit of dependency tracking code.  Using the scheduler in this way provides a clean separation between front-end and back-end. The job of the userspace-facing ioctl code is to shove things on the scheduler.  The job of the run_job callback is to encode the job into the FW queue format, stick it in the FW queue ring, and maybe smash a doorbell.  Everything else happens in terms of managing those queues side-band.  The gpu_scheduler code manages the front-end queues and Xe manages the FW queues via the Kernel <-> GuC communication rings.  From a high level, this is a really clean design.  There are potentially some sticky bits around the dual-use of dma_fence for scheduling and memory management but none of those are solved by breaking the DRM scheduler into chunks or getting rid of the 1:1:1 mapping.

If we split it out, we're basically asking the driver to implement a bunch of kthread or workqueue stuff, all the ring rate-limiting, etc. It may not be all that much code but also, why?  To save a few bytes of memory per engine?  Each engine already has 32K(ish) worth of context state and a similar size ring to communicate with the FW.  No one is going to notice an extra CPU data structure.

I'm not seeing a solid argument against the 1:1:1 design here other than that it doesn't seem like the way DRM scheduler was intended to be used.  I won't argue that.  It's not.  But it is a fairly natural way to take advantage of the benefits the DRM scheduler does provide while also mapping it to hardware that was designed for userspace direct-to-FW submit.

--Jason

>>> With the first option then the end result could be drm_sched
per engine
>>> class (hardware view), which I think fits with the GuC model.
Give all
>>> schedulable contexts (entities) to the GuC and then mostly
forget about
>>> them. Timeslicing and re-ordering and all happens transparently
to the
>>> kernel from that point until completion.
>>>
>>
>> Out-of-order problem still exists here.
>>
>>> Or with the second option you would build on some smaller
refactored
>>> sub-components of drm_sched, by maybe splitting the dependency
tracking from
>>> scheduling (RR/FIFO entity picking code).
>>>
>>> Second option is especially a bit vague and I haven't thought
about the
>>> required mechanics, but it just appeared too obvious the
proposed design has
>>> a bit too much impedance mismatch.
>>>
>>
>> IMO ROI on this is low and again lets see what Boris comes up with. >>
>> Matt
>>
>>> Oh and as a side note, when I went into the drm_sched code base
to remind
>>> myself how things worked, it is quite easy to find some FIXME
comments which
>>> suggest people working on it are unsure of locking desing there
and such. So
>>> perhaps that all needs cleanup too, I mean would benefit from >>> refactoring/improving work as brainstormed above anyway. >>>
>>> Regards,
>>>
>>> Tvrtko

#17 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 61 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 38 lines

     >>> AFAICT it proposes to have 1:1 between *userspace* created contexts (per
     >>> context _and_ engine) and drm_sched. I am not sure avoiding invasive changes
     >>> to the shared code is in the spirit of the overall idea and instead >>> opportunity should be used to look at way to refactor/improve drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.)  There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

     >> Yes, it is 1:1 *userspace* engines and drm_sched. >>
     >> I'm not really prepared to make large changes to DRM scheduler at the
     >> moment for Xe as they are not really required nor does Boris seem they
     >> will be required for his work either. I am interested to see what Boris
     >> comes up with.
     >>
     >>> Even on the low level, the idea to replace drm_sched threads with workers
     >>> has a few problems.
     >>>
     >>> To start with, the pattern of:
     >>>
     >>>    while (not_stopped) {
     >>>     keep picking jobs
     >>>    }
     >>>
     >>> Feels fundamentally in disagreement with workers (while obviously fits
     >>> perfectly with the current kthread design).
     >>
     >> The while loop breaks and worker exists if no jobs are ready.

Quoted 1 line

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal. """

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Quoted 5 lines

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

To make matters worse, if I follow the code correctly, all these per user context worker thread / work items end up contending on the same lock or circular buffer, both are one instance per GPU:

guc_engine_run_job
-> submit_engine

    a) wq_item_append
        -> wq_wait_for_space
          -> msleep
    b) xe_guc_ct_send
        -> guc_ct_send
          -> mutex_lock(&ct->lock);
          -> later a potential msleep in h2g_has_room

Regards,

Tvrtko

author maintainer
#18 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 99 lines · Tvrtko Ursulin
Quoted 88 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 47 lines

     >>> AFAICT it proposes to have 1:1 between *userspace* created contexts (per
     >>> context _and_ engine) and drm_sched. I am not sure avoiding invasive changes
     >>> to the shared code is in the spirit of the overall idea and instead
     >>> opportunity should be used to look at way to refactor/improve drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.) There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

     >> Yes, it is 1:1 *userspace* engines and drm_sched. >>
     >> I'm not really prepared to make large changes to DRM scheduler at the
     >> moment for Xe as they are not really required nor does Boris seem they
     >> will be required for his work either. I am interested to see what Boris
     >> comes up with.
     >>
     >>> Even on the low level, the idea to replace drm_sched threads with workers
     >>> has a few problems.
     >>>
     >>> To start with, the pattern of:
     >>>
     >>>    while (not_stopped) {
     >>>     keep picking jobs
     >>>    }
     >>>
     >>> Feels fundamentally in disagreement with workers (while obviously fits
     >>> perfectly with the current kthread design).
     >>
     >> The while loop breaks and worker exists if no jobs are ready.

Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 4 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

To make matters worse, if I follow the code correctly, all these per user context worker thread / work items end up contending on the same lock or circular buffer, both are one instance per GPU:

Quoted 5 lines

guc_engine_run_job
-> submit_engine

    a) wq_item_append
        -> wq_wait_for_space
          -> msleep

a) is dedicated per xe_engine

Also you missed the step of programming the ring which is dedicated per xe_engine

Quoted 4 lines
    b) xe_guc_ct_send
        -> guc_ct_send
          -> mutex_lock(&ct->lock);
          -> later a potential msleep in h2g_has_room

Techincally there is 1 instance per GT not GPU, yes this is shared but in practice there will always be space in the CT channel so contention on the lock should be rare.

I haven't read your rather long reply yet, but also FWIW using a workqueue has suggested by AMD (original authors of the DRM scheduler) when we ran this design by them.

Matt

Quoted 4 lines

Regards,

Tvrtko

#19 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 100 lines · Matthew Brost
Quoted 99 lines · Tvrtko Ursulin
Quoted 88 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 47 lines

     >>> AFAICT it proposes to have 1:1 between *userspace* created contexts (per
     >>> context _and_ engine) and drm_sched. I am not sure avoiding invasive changes
     >>> to the shared code is in the spirit of the overall idea and
instead
     >>> opportunity should be used to look at way to refactor/improve drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.) There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

     >> Yes, it is 1:1 *userspace* engines and drm_sched. >>
     >> I'm not really prepared to make large changes to DRM scheduler at the
     >> moment for Xe as they are not really required nor does Boris seem they
     >> will be required for his work either. I am interested to see what Boris
     >> comes up with.
     >>
     >>> Even on the low level, the idea to replace drm_sched threads with workers
     >>> has a few problems.
     >>>
     >>> To start with, the pattern of:
     >>>
     >>>    while (not_stopped) {
     >>>     keep picking jobs
     >>>    }
     >>>
     >>> Feels fundamentally in disagreement with workers (while obviously fits
     >>> perfectly with the current kthread design).
     >>
     >> The while loop breaks and worker exists if no jobs are ready.

Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 4 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

To make matters worse, if I follow the code correctly, all these per user context worker thread / work items end up contending on the same lock or circular buffer, both are one instance per GPU:

Quoted 7 lines
Quoted 5 lines

guc_engine_run_job
-> submit_engine

     a) wq_item_append
         -> wq_wait_for_space
           -> msleep

a) is dedicated per xe_engine

Hah true, what its for then? I thought throttling the LRCA ring is done via:

drm_sched_init(&ge->sched, &drm_sched_ops,

		 e->lrc[0].ring.size / MAX_JOB_SIZE_BYTES,

Is there something more to throttle other than the ring? It is throttling something using msleeps..

Quoted 1 line

Also you missed the step of programming the ring which is dedicated per xe_engine

I was trying to quickly find places which serialize on something in the backend, ringbuffer emission did not seem to do that but maybe I missed something.

Quoted 9 lines
Quoted 5 lines
     b) xe_guc_ct_send
         -> guc_ct_send
           -> mutex_lock(&ct->lock);
           -> later a potential msleep in h2g_has_room

Techincally there is 1 instance per GT not GPU, yes this is shared but in practice there will always be space in the CT channel so contention on the lock should be rare.

Yeah I used the term GPU to be more understandable to outside audience.

I am somewhat disappointed that the Xe opportunity hasn't been used to improve upon the CT communication bottlenecks. I mean those backoff sleeps and lock contention. I wish there would be a single thread in charge of the CT channel and internal users (other parts of the driver) would be able to send their requests to it in a more efficient manner, with less lock contention and centralized backoff.

Quoted 3 lines

I haven't read your rather long reply yet, but also FWIW using a workqueue has suggested by AMD (original authors of the DRM scheduler) when we ran this design by them.

Commit message says nothing about that. ;)

Regards,

Tvrtko

author maintainer
#20 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 108 lines · Tvrtko Ursulin
Quoted 106 lines · Matthew Brost
Quoted 104 lines · Tvrtko Ursulin
Quoted 88 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 47 lines

     >>> AFAICT it proposes to have 1:1 between *userspace* created contexts (per
     >>> context _and_ engine) and drm_sched. I am not sure avoiding invasive changes
     >>> to the shared code is in the spirit of the overall idea and
instead
     >>> opportunity should be used to look at way to refactor/improve drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.) There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

     >> Yes, it is 1:1 *userspace* engines and drm_sched. >>
     >> I'm not really prepared to make large changes to DRM scheduler at the
     >> moment for Xe as they are not really required nor does Boris seem they
     >> will be required for his work either. I am interested to see what Boris
     >> comes up with.
     >>
     >>> Even on the low level, the idea to replace drm_sched threads with workers
     >>> has a few problems.
     >>>
     >>> To start with, the pattern of:
     >>>
     >>>    while (not_stopped) {
     >>>     keep picking jobs
     >>>    }
     >>>
     >>> Feels fundamentally in disagreement with workers (while obviously fits
     >>> perfectly with the current kthread design).
     >>
     >> The while loop breaks and worker exists if no jobs are ready.

Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 4 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

To make matters worse, if I follow the code correctly, all these per user context worker thread / work items end up contending on the same lock or circular buffer, both are one instance per GPU:

guc_engine_run_job
-> submit_engine

     a) wq_item_append
         -> wq_wait_for_space
           -> msleep
Quoted 4 lines
Quoted 1 line

a) is dedicated per xe_engine

Hah true, what its for then? I thought throttling the LRCA ring is done via:

This is a per guc_id 'work queue' which is used for parallel submission (e.g. multiple LRC tail values need to written atomically by the GuC). Again in practice there should always be space.

Quoted 12 lines

drm_sched_init(&ge->sched, &drm_sched_ops,

		 e->lrc[0].ring.size / MAX_JOB_SIZE_BYTES,

Is there something more to throttle other than the ring? It is throttling something using msleeps..

Quoted 1 line

Also you missed the step of programming the ring which is dedicated per xe_engine

I was trying to quickly find places which serialize on something in the backend, ringbuffer emission did not seem to do that but maybe I missed something.

xe_ring_ops vfunc emit_job is called to write the ring.

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
     b) xe_guc_ct_send
         -> guc_ct_send
           -> mutex_lock(&ct->lock);
           -> later a potential msleep in h2g_has_room

Techincally there is 1 instance per GT not GPU, yes this is shared but in practice there will always be space in the CT channel so contention on the lock should be rare.

Quoted 9 lines

Yeah I used the term GPU to be more understandable to outside audience.

I am somewhat disappointed that the Xe opportunity hasn't been used to improve upon the CT communication bottlenecks. I mean those backoff sleeps and lock contention. I wish there would be a single thread in charge of the CT channel and internal users (other parts of the driver) would be able to send their requests to it in a more efficient manner, with less lock contention and centralized backoff.

Well the CT backend was more or less a complete rewrite. Mutexes actually work rather well to ensure fairness compared to the spin locks used in the i915. This code was pretty heavily reviewed by Daniel and both of us landed a big mutex for all of the CT code compared to the 3 or 4 spin locks used in the i915.

Quoted 6 lines
Quoted 3 lines

I haven't read your rather long reply yet, but also FWIW using a workqueue has suggested by AMD (original authors of the DRM scheduler) when we ran this design by them.

Commit message says nothing about that. ;)

Yea I missed that, will fix in the next rev. Just dug through my emails and Christian suggested a work queue and Andrey also gave some input on the DRM scheduler design.

Also in the next will likely update the run_wq to be passed in by the user.

Matt

Quoted 3 lines

Regards,

Tvrtko

#21 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 109 lines · Matthew Brost
Quoted 108 lines · Tvrtko Ursulin
Quoted 106 lines · Matthew Brost
Quoted 104 lines · Tvrtko Ursulin
Quoted 88 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 47 lines

     >>> AFAICT it proposes to have 1:1 between *userspace* created contexts (per
     >>> context _and_ engine) and drm_sched. I am not sure avoiding invasive changes
     >>> to the shared code is in the spirit of the overall idea and
instead
     >>> opportunity should be used to look at way to refactor/improve drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.) There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

     >> Yes, it is 1:1 *userspace* engines and drm_sched. >>
     >> I'm not really prepared to make large changes to DRM scheduler at the
     >> moment for Xe as they are not really required nor does Boris seem they
     >> will be required for his work either. I am interested to see what Boris
     >> comes up with.
     >>
     >>> Even on the low level, the idea to replace drm_sched threads with workers
     >>> has a few problems.
     >>>
     >>> To start with, the pattern of:
     >>>
     >>>    while (not_stopped) {
     >>>     keep picking jobs
     >>>    }
     >>>
     >>> Feels fundamentally in disagreement with workers (while obviously fits
     >>> perfectly with the current kthread design).
     >>
     >> The while loop breaks and worker exists if no jobs are ready.

Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 4 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

To make matters worse, if I follow the code correctly, all these per user context worker thread / work items end up contending on the same lock or circular buffer, both are one instance per GPU:

guc_engine_run_job
-> submit_engine

      a) wq_item_append
          -> wq_wait_for_space
            -> msleep
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

a) is dedicated per xe_engine

Hah true, what its for then? I thought throttling the LRCA ring is done via:

This is a per guc_id 'work queue' which is used for parallel submission (e.g. multiple LRC tail values need to written atomically by the GuC). Again in practice there should always be space.

Speaking of guc id, where does blocking when none are available happen in the non parallel case?

Quoted 6 lines
Quoted 6 lines
   drm_sched_init(&ge->sched, &drm_sched_ops,
		 e->lrc[0].ring.size / MAX_JOB_SIZE_BYTES,

Is there something more to throttle other than the ring? It is throttling something using msleeps..

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Also you missed the step of programming the ring which is dedicated per xe_engine

I was trying to quickly find places which serialize on something in the backend, ringbuffer emission did not seem to do that but maybe I missed something.

xe_ring_ops vfunc emit_job is called to write the ring.

Right but does it serialize between different contexts, I didn't spot that it does in which case it wasn't relevant to the sub story.

Quoted 20 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
      b) xe_guc_ct_send
          -> guc_ct_send
            -> mutex_lock(&ct->lock);
            -> later a potential msleep in h2g_has_room

Techincally there is 1 instance per GT not GPU, yes this is shared but in practice there will always be space in the CT channel so contention on the lock should be rare.

Quoted 9 lines

Yeah I used the term GPU to be more understandable to outside audience.

I am somewhat disappointed that the Xe opportunity hasn't been used to improve upon the CT communication bottlenecks. I mean those backoff sleeps and lock contention. I wish there would be a single thread in charge of the CT channel and internal users (other parts of the driver) would be able to send their requests to it in a more efficient manner, with less lock contention and centralized backoff.

Quoted 5 lines

Well the CT backend was more or less a complete rewrite. Mutexes actually work rather well to ensure fairness compared to the spin locks used in the i915. This code was pretty heavily reviewed by Daniel and both of us landed a big mutex for all of the CT code compared to the 3 or 4 spin locks used in the i915.

Are the "nb" sends gone? But that aside, I wasn't meaning just the locking but the high level approach. Never mind.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I haven't read your rather long reply yet, but also FWIW using a workqueue has suggested by AMD (original authors of the DRM scheduler) when we ran this design by them.

Commit message says nothing about that. ;)

Quoted 6 lines

Yea I missed that, will fix in the next rev. Just dug through my emails and Christian suggested a work queue and Andrey also gave some input on the DRM scheduler design.

Also in the next will likely update the run_wq to be passed in by the user.

Yes, and IMO that may need to be non-optional.

Regards,

Tvrtko

author maintainer
#22 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 116 lines · Tvrtko Ursulin
Quoted 114 lines · Matthew Brost
Quoted 112 lines · Tvrtko Ursulin
Quoted 100 lines · Matthew Brost
Quoted 99 lines · Tvrtko Ursulin
Quoted 88 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 47 lines

     >>> AFAICT it proposes to have 1:1 between *userspace* created contexts (per
     >>> context _and_ engine) and drm_sched. I am not sure avoiding invasive changes
     >>> to the shared code is in the spirit of the overall idea and
instead
     >>> opportunity should be used to look at way to refactor/improve drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.) There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

     >> Yes, it is 1:1 *userspace* engines and drm_sched. >>
     >> I'm not really prepared to make large changes to DRM scheduler at the
     >> moment for Xe as they are not really required nor does Boris seem they
     >> will be required for his work either. I am interested to see what Boris
     >> comes up with.
     >>
     >>> Even on the low level, the idea to replace drm_sched threads with workers
     >>> has a few problems.
     >>>
     >>> To start with, the pattern of:
     >>>
     >>>    while (not_stopped) {
     >>>     keep picking jobs
     >>>    }
     >>>
     >>> Feels fundamentally in disagreement with workers (while obviously fits
     >>> perfectly with the current kthread design).
     >>
     >> The while loop breaks and worker exists if no jobs are ready.

Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 4 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

To make matters worse, if I follow the code correctly, all these per user context worker thread / work items end up contending on the same lock or circular buffer, both are one instance per GPU:

Quoted 7 lines
Quoted 5 lines

guc_engine_run_job
-> submit_engine

      a) wq_item_append
          -> wq_wait_for_space
            -> msleep

a) is dedicated per xe_engine

Hah true, what its for then? I thought throttling the LRCA ring is done via:

Quoted 7 lines
Quoted 3 lines

This is a per guc_id 'work queue' which is used for parallel submission (e.g. multiple LRC tail values need to written atomically by the GuC). Again in practice there should always be space.

Speaking of guc id, where does blocking when none are available happen in the non parallel case?

We have 64k guc_ids on native, 1k guc_ids with 64k VFs. Either way we think that is more than enough and can just reject xe_engine creation if we run out of guc_ids. If this proves to false, we can fix this but the guc_id stealing the i915 is rather complicated and hopefully not needed.

We will limit the number of guc_ids allowed per user pid to reasonible number to prevent a DoS. Elevated pids (e.g. IGTs) will be able do to whatever they want.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
   drm_sched_init(&ge->sched, &drm_sched_ops,
		 e->lrc[0].ring.size / MAX_JOB_SIZE_BYTES,

Is there something more to throttle other than the ring? It is throttling something using msleeps..

Quoted 1 line

Also you missed the step of programming the ring which is dedicated per xe_engine

I was trying to quickly find places which serialize on something in the backend, ringbuffer emission did not seem to do that but maybe I missed something.

Quoted 5 lines
Quoted 1 line

xe_ring_ops vfunc emit_job is called to write the ring.

Right but does it serialize between different contexts, I didn't spot that it does in which case it wasn't relevant to the sub story.

Right just saying this is an additional step that is done in parallel between xe_engines.

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 9 lines
Quoted 5 lines
      b) xe_guc_ct_send
          -> guc_ct_send
            -> mutex_lock(&ct->lock);
            -> later a potential msleep in h2g_has_room

Techincally there is 1 instance per GT not GPU, yes this is shared but in practice there will always be space in the CT channel so contention on the lock should be rare.

Yeah I used the term GPU to be more understandable to outside audience.

I am somewhat disappointed that the Xe opportunity hasn't been used to improve upon the CT communication bottlenecks. I mean those backoff sleeps and lock contention. I wish there would be a single thread in charge of the CT channel and internal users (other parts of the driver) would be able to send their requests to it in a more efficient manner, with less lock contention and centralized backoff.

Quoted 9 lines
Quoted 5 lines

Well the CT backend was more or less a complete rewrite. Mutexes actually work rather well to ensure fairness compared to the spin locks used in the i915. This code was pretty heavily reviewed by Daniel and both of us landed a big mutex for all of the CT code compared to the 3 or 4 spin locks used in the i915.

Are the "nb" sends gone? But that aside, I wasn't meaning just the locking but the high level approach. Never mind.

xe_guc_ct_send is non-blocking, xe_guc_ct_send_block is blocking. I don't think the later is used yet.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I haven't read your rather long reply yet, but also FWIW using a workqueue has suggested by AMD (original authors of the DRM scheduler) when we ran this design by them.

Commit message says nothing about that. ;)

Quoted 9 lines
Quoted 6 lines

Yea I missed that, will fix in the next rev. Just dug through my emails and Christian suggested a work queue and Andrey also gave some input on the DRM scheduler design.

Also in the next will likely update the run_wq to be passed in by the user.

Yes, and IMO that may need to be non-optional.

Yea, will fix.

Matt

Quoted 3 lines

Regards,

Tvrtko

#23 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 113 lines · Matthew Brost
Quoted 103 lines · Tvrtko Ursulin
Quoted 102 lines · Matthew Brost
Quoted 101 lines · Tvrtko Ursulin
Quoted 100 lines · Matthew Brost
Quoted 99 lines · Tvrtko Ursulin
Quoted 85 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 47 lines

     >>> AFAICT it proposes to have 1:1 between *userspace* created contexts (per
     >>> context _and_ engine) and drm_sched. I am not sure avoiding invasive changes
     >>> to the shared code is in the spirit of the overall idea and
instead
     >>> opportunity should be used to look at way to refactor/improve drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.) There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

     >> Yes, it is 1:1 *userspace* engines and drm_sched. >>
     >> I'm not really prepared to make large changes to DRM scheduler at the
     >> moment for Xe as they are not really required nor does Boris seem they
     >> will be required for his work either. I am interested to see what Boris
     >> comes up with.
     >>
     >>> Even on the low level, the idea to replace drm_sched threads with workers
     >>> has a few problems.
     >>>
     >>> To start with, the pattern of:

        >>>
        >>>    while (not_stopped) {
        >>>     keep picking jobs
        >>>    }
        >>>
        >>> Feels fundamentally in disagreement with workers (while
       obviously fits
        >>> perfectly with the current kthread design).
        >>
        >> The while loop breaks and worker exists if no jobs are ready.
Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 4 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

To make matters worse, if I follow the code correctly, all these per user context worker thread / work items end up contending on the same lock or circular buffer, both are one instance per GPU:

guc_engine_run_job

    -> submit_engine
       a) wq_item_append
           -> wq_wait_for_space
             -> msleep
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

a) is dedicated per xe_engine

Hah true, what its for then? I thought throttling the LRCA ring is done via:

This is a per guc_id 'work queue' which is used for parallel submission (e.g. multiple LRC tail values need to written atomically by the GuC). Again in practice there should always be space.

Speaking of guc id, where does blocking when none are available happen in the non parallel case?

Quoted 8 lines

We have 64k guc_ids on native, 1k guc_ids with 64k VFs. Either way we think that is more than enough and can just reject xe_engine creation if we run out of guc_ids. If this proves to false, we can fix this but the guc_id stealing the i915 is rather complicated and hopefully not needed.

We will limit the number of guc_ids allowed per user pid to reasonible number to prevent a DoS. Elevated pids (e.g. IGTs) will be able do to whatever they want.

What about doorbells? As some point, we will have to start using those and they are a much more limited resource - 256 total and way less with VFs.

John.

author maintainer
#24 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 119 lines · John Harrison
Quoted 118 lines · Matthew Brost
Quoted 103 lines · Tvrtko Ursulin
Quoted 102 lines · Matthew Brost
Quoted 101 lines · Tvrtko Ursulin
Quoted 100 lines · Matthew Brost
Quoted 99 lines · Tvrtko Ursulin
Quoted 85 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 47 lines

     >>> AFAICT it proposes to have 1:1 between *userspace* created contexts (per
     >>> context _and_ engine) and drm_sched. I am not sure avoiding invasive changes
     >>> to the shared code is in the spirit of the overall idea and
instead
     >>> opportunity should be used to look at way to refactor/improve drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.) There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

     >> Yes, it is 1:1 *userspace* engines and drm_sched. >>
     >> I'm not really prepared to make large changes to DRM scheduler at the
     >> moment for Xe as they are not really required nor does Boris seem they
     >> will be required for his work either. I am interested to see what Boris
     >> comes up with.
     >>
     >>> Even on the low level, the idea to replace drm_sched threads with workers
     >>> has a few problems.
     >>>
     >>> To start with, the pattern of:

        >>>
        >>>    while (not_stopped) {
        >>>     keep picking jobs
        >>>    }
        >>>
        >>> Feels fundamentally in disagreement with workers (while
       obviously fits
        >>> perfectly with the current kthread design).
        >>
        >> The while loop breaks and worker exists if no jobs are ready.
Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 4 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

To make matters worse, if I follow the code correctly, all these per user context worker thread / work items end up contending on the same lock or circular buffer, both are one instance per GPU:

guc_engine_run_job

    -> submit_engine
       a) wq_item_append
           -> wq_wait_for_space
             -> msleep
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

a) is dedicated per xe_engine

Hah true, what its for then? I thought throttling the LRCA ring is done via:

This is a per guc_id 'work queue' which is used for parallel submission (e.g. multiple LRC tail values need to written atomically by the GuC). Again in practice there should always be space.

Speaking of guc id, where does blocking when none are available happen in the non parallel case?

We have 64k guc_ids on native, 1k guc_ids with 64k VFs. Either way we think that is more than enough and can just reject xe_engine creation if we run out of guc_ids. If this proves to false, we can fix this but the guc_id stealing the i915 is rather complicated and hopefully not needed.

Quoted 6 lines
Quoted 3 lines

We will limit the number of guc_ids allowed per user pid to reasonible number to prevent a DoS. Elevated pids (e.g. IGTs) will be able do to whatever they want.

What about doorbells? As some point, we will have to start using those and they are a much more limited resource - 256 total and way less with VFs.

We haven't thought about that one yet, will figure this one out when we implement this.

Matt

Quoted 2 lines

John.

#25 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

In reply to Tvrtko Ursulin
Quoted 78 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 5 lines

>>> AFAICT it proposes to have 1:1 between *userspace* created
contexts (per
>>> context _and_ engine) and drm_sched. I am not sure avoiding
invasive changes
>>> to the shared code is in the spirit of the overall idea and

instead

Quoted 40 lines

>>> opportunity should be used to look at way to refactor/improve
drm_sched.

Maybe? I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor. (More on that later.) There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier. Not ideal, but not the end of the world either. Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

>> Yes, it is 1:1 *userspace* engines and drm_sched. >>
>> I'm not really prepared to make large changes to DRM scheduler
at the
>> moment for Xe as they are not really required nor does Boris
seem they
>> will be required for his work either. I am interested to see
what Boris
>> comes up with.
>>
>>> Even on the low level, the idea to replace drm_sched threads
with workers
>>> has a few problems.
>>>
>>> To start with, the pattern of:
>>>
>>> while (not_stopped) {
>>> keep picking jobs
>>> }
>>>
>>> Feels fundamentally in disagreement with workers (while
obviously fits
>>> perfectly with the current kthread design).
>>
>> The while loop breaks and worker exists if no jobs are ready.

Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Quoted 5 lines

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts.

Those are pretty much the same in practice. Rather, user threads is typically an upper bound on the number of contexts. Yes, a single user thread could have a bunch of contexts but basically nothing does that except IGT. In real-world usage, it's at most one context per user thread.

Quoted 18 lines

That
is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

Unbound workers have no CPU / cache locality either and no connection with the CPU scheduler to optimize scheduling patterns. This may matter either on large systems or on small ones. Whereas the current design allows for scheduler to notice userspace CPU thread keeps waking up the same drm scheduler kernel thread, and so it can keep them on the same CPU, the unbound workers lose that ability and so 2nd CPU might be getting woken up from low sleep for every submission.

Quoted 4 lines

Hence, apart from being a bit of a impedance mismatch, the proposal has the potential to change performance and power patterns and both large and small machines.

Ok, thanks for explaining the issue you're seeing in more detail. Yes, deferred kwork does appear to mismatch somewhat with what the scheduler needs or at least how it's worked in the past. How much impact will that mismatch have? Unclear.

Quoted 72 lines
Quoted 10 lines

>>> Secondly, it probably demands separate workers (not optional),
otherwise
>>> behaviour of shared workqueues has either the potential to
explode number
>>> kernel threads anyway, or add latency.
>>>
>>
>> Right now the system_unbound_wq is used which does have a limit
on the
>> number of threads, right? I do have a FIXME to allow a worker to

be

Quoted 12 lines

>> passed in similar to TDR.
>>
>> WRT to latency, the 1:1 ratio could actually have lower latency
as 2 GPU
>> schedulers can be pushing jobs into the backend / cleaning up
jobs in
>> parallel.
>>
>
> Thought of one more point here where why in Xe we absolutely want
a 1 to
> 1 ratio between entity and scheduler - the way we implement

Quoted 8 lines

timeslicing
> for preempt fences.
>
> Let me try to explain.
>
> Preempt fences are implemented via the generic messaging
interface [1]
> with suspend / resume messages. If a suspend messages is received

to

Quoted 1 line

> soon after calling resume (this is per entity) we simply sleep in

the

Quoted 17 lines

> suspend call thus giving the entity a timeslice. This completely
falls
> apart with a many to 1 relationship as now a entity waiting for a > timeslice blocks the other entities. Could we work aroudn this,
sure but
> just another bunch of code we'd have to add in Xe. Being to
freely sleep
> in backend without affecting other entities is really, really
nice IMO
> and I bet Xe isn't the only driver that is going to feel this way. >
> Last thing I'll say regardless of how anyone feels about Xe using
a 1 to
> 1 relationship this patch IMO makes sense as I hope we can all
agree a
> workqueue scales better than kthreads.

Quoted 2 lines

I don't know for sure what will scale better and for what use case, combination of CPU cores vs number of GPU engines to keep busy vs

other

Quoted 5 lines

system activity. But I wager someone is bound to ask for some numbers to
make sure proposal is not negatively affecting any other drivers.

Quoted 7 lines

Then let them ask. Waving your hands vaguely in the direction of the rest of DRM and saying "Uh, someone (not me) might object" is profoundly unhelpful. Sure, someone might. That's why it's on dri-devel. If you think there's someone in particular who might have a useful opinion on this, throw them in the CC so they don't miss the e-mail thread.

Or are you asking for numbers? If so, what numbers are you asking for?

It was a heads up to the Xe team in case people weren't appreciating how the proposed change has the potential influence power and performance across the board. And nothing in the follow up discussion made me think it was considered so I don't think it was redundant to raise it.

Quoted 9 lines

In my experience it is typical that such core changes come with some numbers. Which is in case of drm scheduler is tricky and probably requires explicitly asking everyone to test (rather than count on "don't miss the email thread"). Real products can fail to ship due ten mW here or there. Like suddenly an extra core prevented from getting into deep sleep.

If that was "profoundly unhelpful" so be it.

With your above explanation, it makes more sense what you're asking. It's still not something Matt is likely to be able to provide on his own. We need to tag some other folks and ask them to test it out. We could play around a bit with it on Xe but it's not exactly production grade yet and is going to hit this differently from most. Likely candidates are probably AMD and Freedreno.

Quoted 11 lines
Quoted 7 lines

Also, If we're talking about a design that might paint us into an Intel-HW-specific hole, that would be one thing. But we're not. We're talking about switching which kernel threading/task mechanism to use for what's really a very generic problem. The core Xe design works without this patch (just with more kthreads). If we land this patch or something like it and get it wrong and it causes a performance problem for someone down the line, we can revisit it.

For some definition of "it works" - I really wouldn't suggest shipping a kthread per user context at any point.

You have yet to elaborate on why. What resources is it consuming that's going to be a problem? Are you anticipating CPU affinity problems? Or does it just seem wasteful?

I think I largely agree that it's probably unnecessary/wasteful but reducing the number of kthreads seems like a tractable problem to solve regardless of where we put the gpu_scheduler object. Is this the right solution? Maybe not. It was also proposed at one point that we could split the scheduler into two pieces: A scheduler which owns the kthread, and a back-end which targets some HW ring thing where you can have multiple back-ends per scheduler. That's certainly more invasive from a DRM scheduler internal API PoV but would solve the kthread problem in a way that's more similar to what we have now.

Quoted 51 lines
Quoted 1 line

In any case that's a low level question caused by the high level

design

Quoted 1 line

decision. So I'd think first focus on the high level - which is the

1:1

Quoted 3 lines

mapping of entity to scheduler instance proposal.

Fundamentally it will be up to the DRM maintainers and the community

to

Quoted 18 lines

bless your approach. And it is important to stress 1:1 is about userspace contexts, so I believe unlike any other current scheduler user. And also important to stress this effectively does not make Xe _really_ use the scheduler that much.

I don't think this makes Xe nearly as much of a one-off as you think it does. I've already told the Asahi team working on Apple M1/2 hardware to do it this way and it seems to be a pretty good mapping for them. I believe this is roughly the plan for nouveau as well. It's not the way it currently works for anyone because most other groups aren't doing FW scheduling yet. In the world of FW scheduling and hardware designed to support userspace direct-to-FW submit, I think the design makes perfect sense (see below) and I expect we'll see more drivers move in this direction as those drivers evolve. (AMD is doing some customish thing for how with gpu_scheduler on the front-end somehow. I've not dug into those details.)

Quoted 1 line

I can only offer my opinion, which is that the two options mentioned

in

Quoted 5 lines

this thread (either improve drm scheduler to cope with what is required,
or split up the code so you can use just the parts of drm_sched which you want - which is frontend dependency tracking) shouldn't be so readily dismissed, given how I think the idea was for the new driver

to

Quoted 2 lines

work less in a silo and more in the community (not do kludges to workaround stuff because it is thought to be too hard to improve

common

Quoted 4 lines

code), but fundamentally, "goto previous paragraph" for what I am concerned.

Quoted 9 lines

Meta comment: It appears as if you're falling into the standard i915 team trap of having an internal discussion about what the community discussion might look like instead of actually having the community discussion. If you are seriously concerned about interactions with other drivers or whether or setting common direction, the right way to do that is to break a patch or two out into a separate RFC series and tag a handful of driver maintainers. Trying to predict the questions other people might ask is pointless. Cc them and asking for their input instead.

Quoted 4 lines

I don't follow you here. It's not an internal discussion - I am raising my concerns on the design publicly. I am supposed to write a patch to show something, but am allowed to comment on a RFC series?

I may have misread your tone a bit. It felt a bit like too many discussions I've had in the past where people are trying to predict what others will say instead of just asking them. Reading it again, I was probably jumping to conclusions a bit. Sorry about that.

Quoted 3 lines

It is "drm/sched: Convert drm scheduler to use a work queue rather than kthread" which should have Cc-ed _everyone_ who use drm scheduler.

Yeah, it probably should have. I think that's mostly what I've been trying to say.

Quoted 51 lines
Quoted 10 lines

Regards,

Tvrtko

P.S. And as a related side note, there are more areas where drm_sched could be improved, like for instance priority handling. Take a look at msm_submitqueue_create / msm_gpu_convert_priority / get_sched_entity to see how msm works around the drm_sched hardcoded limit of available priority levels, in order to avoid having to

leave a

Quoted 1 line

hw capability unused. I suspect msm would be happier if they could

have

Quoted 30 lines

all priority levels equal in terms of whether they apply only at the frontend level or completely throughout the pipeline.

> [1]
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1

>
>>> What would be interesting to learn is whether the option of
refactoring
>>> drm_sched to deal with out of order completion was considered
and what were
>>> the conclusions.
>>>
>>
>> I coded this up a while back when trying to convert the i915 to
the DRM
>> scheduler it isn't all that hard either. The free flow control
on the
>> ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE) is
really what
>> sold me on the this design.

You're not the only one to suggest supporting out-of-order completion. However, it's tricky and breaks a lot of internal assumptions of the scheduler. It also reduces functionality a bit because it can no longer automatically rate-limit HW/FW queues which are often fixed-size. (Ok, yes, it probably could but it becomes a substantially harder problem.)

Quoted 7 lines

It also seems like a worse mapping to me. The goal here is to turn submissions on a userspace-facing engine/queue into submissions to a FW queue submissions, sorting out any dma_fence dependencies. Matt's description of saying this is a 1:1 mapping between sched/entity doesn't tell the whole story. It's a 1:1:1 mapping between xe_engine, gpu_scheduler, and GuC FW engine. Why make it a 1:something:1 mapping? Why is that better?

Quoted 5 lines

As I have stated before, what I think what would fit well for Xe is one drm_scheduler per engine class. In specific terms on our current hardware, one drm scheduler instance for render, compute, blitter, video and video enhance. Userspace contexts remain scheduler entities.

And this is where we fairly strongly disagree. More in a bit.

Quoted 4 lines

That way you avoid the whole kthread/kworker story and you have it actually use the entity picking code in the scheduler, which may be useful when the backend is congested.

What back-end congestion are you referring to here? Running out of FW queue IDs? Something else?

Quoted 4 lines

Yes you have to solve the out of order problem so in my mind that is something to discuss. What the problem actually is (just TDR?), how tricky and why etc.

Quoted 9 lines

And yes you lose the handy LRCA ring buffer size management so you'd have to make those entities not runnable in some other way.

Regarding the argument you raise below - would any of that make the frontend / backend separation worse and why? Do you think it is less natural? If neither is true then all remains is that it appears extra work to support out of order completion of entities has been discounted in favour of an easy but IMO inelegant option.

Broadly speaking, the kernel needs to stop thinking about GPU scheduling in terms of scheduling jobs and start thinking in terms of scheduling contexts/engines. There is still some need for scheduling individual jobs but that is only for the purpose of delaying them as needed to resolve dma_fence dependencies. Once dependencies are resolved, they get shoved onto the context/engine queue and from there the kernel only really manages whole contexts/engines. This is a major architectural shift, entirely different from the way i915 scheduling works. It's also different from the historical usage of DRM scheduler which I think is why this all looks a bit funny.

To justify this architectural shift, let's look at where we're headed. In the glorious future...

1. Userspace submits directly to firmware queues. The kernel has no
visibility whatsoever into individual jobs. At most it can pause/resume FW contexts as needed to handle eviction and memory management.

2. Because of 1, apart from handing out the FW queue IDs at the beginning,
the kernel can't really juggle them that much. Depending on FW design, it may be able to pause a client, give its IDs to another, and then resume it later when IDs free up. What it's not doing is juggling IDs on a job-by-job basis like i915 currently is.

3. Long-running compute jobs may not complete for days. This means that
memory management needs to happen in terms of pause/resume of entire contexts/engines using the memory rather than based on waiting for individual jobs to complete or pausing individual jobs until the memory is available.

4. Synchronization happens via userspace memory fences (UMF) and the
kernel is mostly unaware of most dependencies and when a context/engine is or is not runnable. Instead, it keeps as many of them minimally active (memory is available, even if it's in system RAM) as possible and lets the FW sort out dependencies. (There may need to be some facility for sleeping a context until a memory change similar to futex() or poll() for userspace threads. There are some details TBD.)

Are there potential problems that will need to be solved here? Yes. Is it a good design? Well, Microsoft has been living in this future for half a decade or better and it's working quite well for them. It's also the way all modern game consoles work. It really is just Linux that's stuck with the same old job model we've had since the monumental shift to DRI2.

To that end, one of the core goals of the Xe project was to make the driver internally behave as close to the above model as possible while keeping the old-school job model as a very thin layer on top. As the broader ecosystem problems (window-system support for UMF, for instance) are solved, that layer can be peeled back. The core driver will already be ready for it.

To that end, the point of the DRM scheduler in Xe isn't to schedule jobs. It's to resolve syncobj and dma-buf implicit sync dependencies and stuff jobs into their respective context/engine queue once they're ready. All the actual scheduling happens in firmware and any scheduling the kernel does to deal with contention, oversubscriptions, too many contexts, etc. is between contexts/engines, not individual jobs. Sure, the individual job visibility is nice, but if we design around it, we'll never get to the glorious future.

I really need to turn the above (with a bit more detail) into a blog post.... Maybe I'll do that this week.

In any case, I hope that provides more insight into why Xe is designed the way it is and why I'm pushing back so hard on trying to make it more of a "classic" driver as far as scheduling is concerned. Are there potential problems here? Yes, that's why Xe has been labeled a prototype. Are such radical changes necessary to get to said glorious future? Yes, I think they are. Will it be worth it? I believe so.

Quoted 13 lines

There are two places where this 1:1:1 mapping is causing problems:

Quoted 11 lines

1. It creates lots of kthreads. This is what this patch is trying to
solve. IDK if it's solving it the best way but that's the goal.

2. There are a far more limited number of communication queues between
the kernel and GuC for more meta things like pausing and resuming queues, getting events back from GuC, etc. Unless we're in a weird pressure scenario, the amount of traffic on this queue should be low so we can probably just have one per physical device. The vast majority of kernel -> GuC communication should be on the individual FW queue rings and maybe smashing in-memory doorbells.

Quoted 4 lines

I don't follow your terminology here. I suppose you are talking about global GuC CT and context ringbuffers. If so then isn't "far more limited" actually one?

I thought there could be more than one but I think in practice it's just the one.

--Jason

Quoted 100 lines

Regards,

Tvrtko

Quoted 7 lines

Doing out-of-order completion sort-of solves the 1 but does nothing for 2 and actually makes managing FW queues harder because we no longer have built-in rate limiting. Seems like a net loss to me.

>>> Second option perhaps to split out the drm_sched code into
parts which would
>>> lend themselves more to "pick and choose" of its

functionalities.

Quoted 8 lines

>>> Specifically, Xe wants frontend dependency tracking, but not
any scheduling
>>> really (neither least busy drm_sched, neither FIFO/RQ entity
picking), so
>>> even having all these data structures in memory is a waste. >>>
>>
>> I don't think that we are wasting memory is a very good argument

for

Quoted 25 lines

>> making intrusive changes to the DRM scheduler.

Worse than that, I think the "we could split it up" kind-of misses the point of the way Xe is using drm/scheduler. It's not just about re-using a tiny bit of dependency tracking code. Using the scheduler in this way provides a clean separation between front-end and back-end. The job of the userspace-facing ioctl code is to shove things on the scheduler. The job of the run_job callback is to encode the job into the FW queue format, stick it in the FW queue ring, and maybe smash a doorbell. Everything else happens in terms of managing those queues side-band. The gpu_scheduler code manages the front-end queues and Xe manages the FW queues via the Kernel <-> GuC communication rings. From a high level, this is a really clean design. There are potentially some sticky bits around the dual-use of dma_fence for scheduling and memory management but none of those are solved by breaking the DRM scheduler into chunks or getting rid of the 1:1:1 mapping.

If we split it out, we're basically asking the driver to implement a bunch of kthread or workqueue stuff, all the ring rate-limiting, etc. It may not be all that much code but also, why? To save a few bytes of memory per engine? Each engine already has 32K(ish) worth of context state and a similar size ring to communicate with the FW. No one is going to notice an extra CPU data structure.

Quoted 5 lines

I'm not seeing a solid argument against the 1:1:1 design here other than that it doesn't seem like the way DRM scheduler was intended to be used. I won't argue that. It's not. But it is a fairly natural way to take advantage of the benefits the DRM scheduler does provide while also mapping it to hardware that was designed for userspace direct-to-FW

submit.

Quoted 22 lines

--Jason

>>> With the first option then the end result could be drm_sched
per engine
>>> class (hardware view), which I think fits with the GuC model.
Give all
>>> schedulable contexts (entities) to the GuC and then mostly
forget about
>>> them. Timeslicing and re-ordering and all happens transparently
to the
>>> kernel from that point until completion.
>>>
>>
>> Out-of-order problem still exists here.
>>
>>> Or with the second option you would build on some smaller
refactored
>>> sub-components of drm_sched, by maybe splitting the dependency
tracking from
>>> scheduling (RR/FIFO entity picking code).
>>>

Quoted 8 lines

>>> Second option is especially a bit vague and I haven't thought
about the
>>> required mechanics, but it just appeared too obvious the
proposed design has
>>> a bit too much impedance mismatch.
>>>
>>
>> IMO ROI on this is low and again lets see what Boris comes up

with.

Quoted 16 lines
     >>
     >> Matt
     >>
     >>> Oh and as a side note, when I went into the drm_sched code base
    to remind
     >>> myself how things worked, it is quite easy to find some FIXME
    comments which
     >>> suggest people working on it are unsure of locking desing there
    and such. So
     >>> perhaps that all needs cleanup too, I mean would benefit from
     >>> refactoring/improving work as brainstormed above anyway.
     >>>
     >>> Regards,
     >>>
     >>> Tvrtko
#26 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 100 lines · Jason Ekstrand

On Tue, Jan 10, 2023 at 5:28 AM Tvrtko Ursulin <[email protected] <mailto:[email protected]>> wrote:

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

>      >>> AFAICT it proposes to have 1:1 between *userspace* created >     contexts (per
>      >>> context _and_ engine) and drm_sched. I am not sure avoiding >     invasive changes
>      >>> to the shared code is in the spirit of the overall idea
and instead
>      >>> opportunity should be used to look at way to
refactor/improve
>     drm_sched.
>
>
> Maybe?  I'm not convinced that what Xe is doing is an abuse at
all or
> really needs to drive a re-factor.  (More on that later.)
There's only
> one real issue which is that it fires off potentially a lot of
kthreads.
> Even that's not that bad given that kthreads are pretty light and
you're
> not likely to have more kthreads than userspace threads which are
much
> heavier.  Not ideal, but not the end of the world either.
Definitely
> something we can/should optimize but if we went through with Xe
without
> this patch, it would probably be mostly ok.
>
>      >> Yes, it is 1:1 *userspace* engines and drm_sched. >      >>
>      >> I'm not really prepared to make large changes to DRM
scheduler
>     at the
>      >> moment for Xe as they are not really required nor does Boris >     seem they
>      >> will be required for his work either. I am interested to see >     what Boris
>      >> comes up with.
>      >>
>      >>> Even on the low level, the idea to replace drm_sched threads >     with workers
>      >>> has a few problems.
>      >>>
>      >>> To start with, the pattern of:
>      >>>
>      >>>    while (not_stopped) {
>      >>>     keep picking jobs
>      >>>    }
>      >>>
>      >>> Feels fundamentally in disagreement with workers (while >     obviously fits
>      >>> perfectly with the current kthread design).
>      >>
>      >> The while loop breaks and worker exists if no jobs are ready. >
>
> I'm not very familiar with workqueues. What are you saying would fit > better? One scheduling job per work item rather than one big work
item
> which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch
because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued
work items (more interesting when mixing up with the system_unbound_wq),
in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts.

Quoted 1 line

Those are pretty much the same in practice.  Rather, user threads is typically an upper bound on the number of contexts.  Yes, a single user thread could have a bunch of contexts but basically nothing does that except IGT.  In real-world usage, it's at most one context per user thread.

Typically is the key here. But I am not sure it is good enough. Consider this example - Intel Flex 170:

* Delivers up to 36 streams 1080p60 transcode throughput per card.
* When scaled to 10 cards in a 4U server configuration, it can support
up to 360 streams of HEVC/HEVC 1080p60 transcode throughput.

One transcode stream from my experience typically is 3-4 GPU contexts (buffer travels from vcs -> rcs -> vcs, maybe vecs) used from a single CPU thread. 4 contexts * 36 streams = 144 active contexts. Multiply by 60fps = 8640 jobs submitted and completed per second.

144 active contexts in the proposed scheme means possibly means 144 kernel worker threads spawned (driven by 36 transcode CPU threads). (I don't think the pools would scale down given all are constantly pinged at 60fps.)

And then each of 144 threads goes to grab the single GuC CT mutex. First threads are being made schedulable, then put to sleep as mutex contention is hit, then woken again as mutexes are getting released, rinse, repeat.

(And yes this backend contention is there regardless of 1:1:1, it would require a different re-design to solve that. But it is just a question whether there are 144 contending threads, or just 6 with the thread per engine class scheme.)

Then multiply all by 10 for a 4U server use case and you get 1440 worker kthreads, yes 10 more CT locks, but contending on how many CPU cores? Just so they can grab a timeslice and maybe content on a mutex as the next step.

This example is where it would hurt on large systems. Imagine only an even wider media transcode card...

Second example is only a single engine class used (3d desktop?) but with a bunch of not-runnable jobs queued and waiting on a fence to signal. Implicit or explicit dependencies doesn't matter. Then the fence signals and call backs run. N work items get scheduled, but they all submit to the same HW engine. So we end up with:

        /-- wi1 --\
       / ..     .. \
 cb --+---  wi.. ---+-- rq1 -- .. -- rqN
       \ ..    ..  /
        \-- wiN --/

All that we have achieved is waking up N CPUs to contend on the same lock and effectively insert the job into the same single HW queue. I don't see any positives there.

This example I think can particularly hurt small / low power devices because of needless waking up of many cores for no benefit. Granted, I don't have a good feel on how common this pattern is in practice.

Quoted 123 lines

That
is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Several problems there.

It is fundamentally pointless to have potentially that many more threads
than the number of CPU cores - it simply creates a scheduling storm.

Unbound workers have no CPU / cache locality either and no connection with the CPU scheduler to optimize scheduling patterns. This may matter either on large systems or on small ones. Whereas the current design allows for scheduler to notice userspace CPU thread keeps waking up the same drm scheduler kernel thread, and so it can keep them on the same CPU, the unbound workers lose that ability and so 2nd CPU might be getting woken up from low sleep for every submission.

Hence, apart from being a bit of a impedance mismatch, the proposal has the potential to change performance and power patterns and both large and small machines.

Ok, thanks for explaining the issue you're seeing in more detail.  Yes, deferred kwork does appear to mismatch somewhat with what the scheduler needs or at least how it's worked in the past.  How much impact will that mismatch have?  Unclear.

>      >>> Secondly, it probably demands separate workers (not
optional),
>     otherwise
>      >>> behaviour of shared workqueues has either the potential to >     explode number
>      >>> kernel threads anyway, or add latency.
>      >>>
>      >>
>      >> Right now the system_unbound_wq is used which does have a
limit
>     on the
>      >> number of threads, right? I do have a FIXME to allow a
worker to be
>      >> passed in similar to TDR.
>      >>
>      >> WRT to latency, the 1:1 ratio could actually have lower
latency
>     as 2 GPU
>      >> schedulers can be pushing jobs into the backend / cleaning up >     jobs in
>      >> parallel.
>      >>
>      >
>      > Thought of one more point here where why in Xe we
absolutely want
>     a 1 to
>      > 1 ratio between entity and scheduler - the way we implement >     timeslicing
>      > for preempt fences.
>      >
>      > Let me try to explain.
>      >
>      > Preempt fences are implemented via the generic messaging >     interface [1]
>      > with suspend / resume messages. If a suspend messages is
received to
>      > soon after calling resume (this is per entity) we simply
sleep in the
>      > suspend call thus giving the entity a timeslice. This
completely
>     falls
>      > apart with a many to 1 relationship as now a entity
waiting for a
>      > timeslice blocks the other entities. Could we work aroudn
this,
>     sure but
>      > just another bunch of code we'd have to add in Xe. Being to >     freely sleep
>      > in backend without affecting other entities is really, really >     nice IMO
>      > and I bet Xe isn't the only driver that is going to feel
this way.
>      >
>      > Last thing I'll say regardless of how anyone feels about
Xe using
>     a 1 to
>      > 1 relationship this patch IMO makes sense as I hope we can all >     agree a
>      > workqueue scales better than kthreads.
>
>     I don't know for sure what will scale better and for what use
case,
>     combination of CPU cores vs number of GPU engines to keep
busy vs other
>     system activity. But I wager someone is bound to ask for some >     numbers to
>     make sure proposal is not negatively affecting any other drivers. >
>
> Then let them ask.  Waving your hands vaguely in the direction of
the
> rest of DRM and saying "Uh, someone (not me) might object" is
profoundly
> unhelpful.  Sure, someone might.  That's why it's on dri-devel.
If you
> think there's someone in particular who might have a useful
opinion on
> this, throw them in the CC so they don't miss the e-mail thread. >
> Or are you asking for numbers?  If so, what numbers are you
asking for?

It was a heads up to the Xe team in case people weren't appreciating how
the proposed change has the potential influence power and performance across the board. And nothing in the follow up discussion made me think it was considered so I don't think it was redundant to raise it.

In my experience it is typical that such core changes come with some numbers. Which is in case of drm scheduler is tricky and probably requires explicitly asking everyone to test (rather than count on "don't
miss the email thread"). Real products can fail to ship due ten mW here or there. Like suddenly an extra core prevented from getting into deep sleep.

Quoted 4 lines

If that was "profoundly unhelpful" so be it.

With your above explanation, it makes more sense what you're asking. It's still not something Matt is likely to be able to provide on his own.  We need to tag some other folks and ask them to test it out.  We could play around a bit with it on Xe but it's not exactly production grade yet and is going to hit this differently from most.  Likely candidates are probably AMD and Freedreno.

Whoever is setup to check out power and performance would be good to give it a spin, yes.

PS. I don't think I was asking Matt to test with other devices. To start with I think Xe is a team effort. I was asking for more background on the design decision since patch 4/20 does not say anything on that angle, nor later in the thread it was IMO sufficiently addressed.

Quoted 12 lines

> Also, If we're talking about a design that might paint us into an > Intel-HW-specific hole, that would be one thing.  But we're not.
We're
> talking about switching which kernel threading/task mechanism to
use for
> what's really a very generic problem.  The core Xe design works
without
> this patch (just with more kthreads).  If we land this patch or > something like it and get it wrong and it causes a performance
problem
> for someone down the line, we can revisit it.

Quoted 6 lines

For some definition of "it works" - I really wouldn't suggest shipping a
kthread per user context at any point.

You have yet to elaborate on why. What resources is it consuming that's going to be a problem? Are you anticipating CPU affinity problems? Or does it just seem wasteful?

Well I don't know, commit message says the approach does not scale. :)

Quoted 70 lines

I think I largely agree that it's probably unnecessary/wasteful but reducing the number of kthreads seems like a tractable problem to solve regardless of where we put the gpu_scheduler object.  Is this the right solution?  Maybe not.  It was also proposed at one point that we could split the scheduler into two pieces: A scheduler which owns the kthread, and a back-end which targets some HW ring thing where you can have multiple back-ends per scheduler.  That's certainly more invasive from a DRM scheduler internal API PoV but would solve the kthread problem in a way that's more similar to what we have now.

>     In any case that's a low level question caused by the high
level design
>     decision. So I'd think first focus on the high level - which
is the 1:1
>     mapping of entity to scheduler instance proposal.
>
>     Fundamentally it will be up to the DRM maintainers and the
community to
>     bless your approach. And it is important to stress 1:1 is about >     userspace contexts, so I believe unlike any other current
scheduler
>     user. And also important to stress this effectively does not
make Xe
>     _really_ use the scheduler that much.
>
>
> I don't think this makes Xe nearly as much of a one-off as you
think it
> does.  I've already told the Asahi team working on Apple M1/2
hardware
> to do it this way and it seems to be a pretty good mapping for
them. I
> believe this is roughly the plan for nouveau as well.  It's not
the way
> it currently works for anyone because most other groups aren't
doing FW
> scheduling yet.  In the world of FW scheduling and hardware
designed to
> support userspace direct-to-FW submit, I think the design makes
perfect
> sense (see below) and I expect we'll see more drivers move in this > direction as those drivers evolve.  (AMD is doing some customish
thing
> for how with gpu_scheduler on the front-end somehow. I've not dug
into
> those details.)
>
>     I can only offer my opinion, which is that the two options
mentioned in
>     this thread (either improve drm scheduler to cope with what is >     required,
>     or split up the code so you can use just the parts of
drm_sched which
>     you want - which is frontend dependency tracking) shouldn't be so >     readily dismissed, given how I think the idea was for the new
driver to
>     work less in a silo and more in the community (not do kludges to >     workaround stuff because it is thought to be too hard to
improve common
>     code), but fundamentally, "goto previous paragraph" for what I am >     concerned.
>
>
> Meta comment:  It appears as if you're falling into the standard
i915
> team trap of having an internal discussion about what the community > discussion might look like instead of actually having the community > discussion.  If you are seriously concerned about interactions with > other drivers or whether or setting common direction, the right
way to
> do that is to break a patch or two out into a separate RFC series
and
> tag a handful of driver maintainers.  Trying to predict the
questions
> other people might ask is pointless. Cc them and asking for their
input
> instead.

Quoted 6 lines

I don't follow you here. It's not an internal discussion - I am raising my concerns on the design publicly. I am supposed to write a patch to show something, but am allowed to comment on a RFC series?

I may have misread your tone a bit.  It felt a bit like too many discussions I've had in the past where people are trying to predict what others will say instead of just asking them.  Reading it again, I was probably jumping to conclusions a bit.  Sorry about that.

Okay no problem, thanks. In any case we don't have to keep discussing it, since I wrote one or two emails ago it is fundamentally on the maintainers and community to ack the approach. I only felt like RFC did not explain the potential downsides sufficiently so I wanted to probe that area a bit.

Quoted 80 lines

It is "drm/sched: Convert drm scheduler to use a work queue rather than kthread" which should have Cc-ed _everyone_ who use drm scheduler.

Yeah, it probably should have.  I think that's mostly what I've been trying to say.

>
>     Regards,
>
>     Tvrtko
>
>     P.S. And as a related side note, there are more areas where
drm_sched
>     could be improved, like for instance priority handling.
>     Take a look at msm_submitqueue_create /
msm_gpu_convert_priority /
>     get_sched_entity to see how msm works around the drm_sched
hardcoded
>     limit of available priority levels, in order to avoid having
to leave a
>     hw capability unused. I suspect msm would be happier if they
could have
>     all priority levels equal in terms of whether they apply only
at the
>     frontend level or completely throughout the pipeline.
>
>      > [1]
>
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
>
 <https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1 <https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>>
>      >
>      >>> What would be interesting to learn is whether the option of
>     refactoring
>      >>> drm_sched to deal with out of order completion was
considered
>     and what were
>      >>> the conclusions.
>      >>>
>      >>
>      >> I coded this up a while back when trying to convert the
i915 to
>     the DRM
>      >> scheduler it isn't all that hard either. The free flow
control
>     on the
>      >> ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE) is
>     really what
>      >> sold me on the this design.
>
>
> You're not the only one to suggest supporting out-of-order
completion.
> However, it's tricky and breaks a lot of internal assumptions of the
> scheduler. It also reduces functionality a bit because it can no
longer
> automatically rate-limit HW/FW queues which are often
fixed-size.  (Ok,
> yes, it probably could but it becomes a substantially harder
problem.)
>
> It also seems like a worse mapping to me.  The goal here is to turn
> submissions on a userspace-facing engine/queue into submissions
to a FW
> queue submissions, sorting out any dma_fence dependencies.  Matt's
> description of saying this is a 1:1 mapping between sched/entity
doesn't
> tell the whole story. It's a 1:1:1 mapping between xe_engine,
> gpu_scheduler, and GuC FW engine.  Why make it a 1:something:1
mapping?
> Why is that better?

As I have stated before, what I think what would fit well for Xe is one drm_scheduler per engine class. In specific terms on our current hardware, one drm scheduler instance for render, compute, blitter, video
and video enhance. Userspace contexts remain scheduler entities.

Quoted 8 lines

And this is where we fairly strongly disagree.  More in a bit.

That way you avoid the whole kthread/kworker story and you have it actually use the entity picking code in the scheduler, which may be useful when the backend is congested.

What back-end congestion are you referring to here?  Running out of FW queue IDs?  Something else?

CT channel, number of context ids.

Quoted 33 lines

Yes you have to solve the out of order problem so in my mind that is something to discuss. What the problem actually is (just TDR?), how tricky and why etc.

And yes you lose the handy LRCA ring buffer size management so you'd have to make those entities not runnable in some other way.

Regarding the argument you raise below - would any of that make the frontend / backend separation worse and why? Do you think it is less natural? If neither is true then all remains is that it appears extra work to support out of order completion of entities has been discounted in favour of an easy but IMO inelegant option.

Broadly speaking, the kernel needs to stop thinking about GPU scheduling in terms of scheduling jobs and start thinking in terms of scheduling contexts/engines.  There is still some need for scheduling individual jobs but that is only for the purpose of delaying them as needed to resolve dma_fence dependencies.  Once dependencies are resolved, they get shoved onto the context/engine queue and from there the kernel only really manages whole contexts/engines.  This is a major architectural shift, entirely different from the way i915 scheduling works.  It's also different from the historical usage of DRM scheduler which I think is why this all looks a bit funny.

To justify this architectural shift, let's look at where we're headed. In the glorious future...

 1. Userspace submits directly to firmware queues.  The kernel has no
visibility whatsoever into individual jobs.  At most it can pause/resume FW contexts as needed to handle eviction and memory management.

 2. Because of 1, apart from handing out the FW queue IDs at the
beginning, the kernel can't really juggle them that much.  Depending on FW design, it may be able to pause a client, give its IDs to another, and then resume it later when IDs free up.  What it's not doing is juggling IDs on a job-by-job basis like i915 currently is.

 3. Long-running compute jobs may not complete for days.  This means
that memory management needs to happen in terms of pause/resume of entire contexts/engines using the memory rather than based on waiting for individual jobs to complete or pausing individual jobs until the memory is available.

 4. Synchronization happens via userspace memory fences (UMF) and the
kernel is mostly unaware of most dependencies and when a context/engine is or is not runnable.  Instead, it keeps as many of them minimally active (memory is available, even if it's in system RAM) as possible and lets the FW sort out dependencies.  (There may need to be some facility for sleeping a context until a memory change similar to futex() or poll() for userspace threads.  There are some details TBD.)

Are there potential problems that will need to be solved here?  Yes.  Is it a good design?  Well, Microsoft has been living in this future for half a decade or better and it's working quite well for them.  It's also the way all modern game consoles work.  It really is just Linux that's stuck with the same old job model we've had since the monumental shift to DRI2.

Quoted 7 lines

To that end, one of the core goals of the Xe project was to make the driver internally behave as close to the above model as possible while keeping the old-school job model as a very thin layer on top.  As the broader ecosystem problems (window-system support for UMF, for instance) are solved, that layer can be peeled back.  The core driver will already be ready for it.

To that end, the point of the DRM scheduler in Xe isn't to schedule jobs.  It's to resolve syncobj and dma-buf implicit sync dependencies and stuff jobs into their respective context/engine queue once they're ready.  All the actual scheduling happens in firmware and any scheduling the kernel does to deal with contention, oversubscriptions, too many contexts, etc. is between contexts/engines, not individual jobs.  Sure, the individual job visibility is nice, but if we design around it, we'll never get to the glorious future.

I really need to turn the above (with a bit more detail) into a blog post.... Maybe I'll do that this week.

In any case, I hope that provides more insight into why Xe is designed the way it is and why I'm pushing back so hard on trying to make it more of a "classic" driver as far as scheduling is concerned.  Are there potential problems here?  Yes, that's why Xe has been labeled a prototype.  Are such radical changes necessary to get to said glorious future?  Yes, I think they are.  Will it be worth it?  I believe so.

Right, that's all solid I think. My takeaway is that frontend priority sorting and that stuff isn't needed and that is okay. And that there are multiple options to maybe improve drm scheduler, like the fore mentioned making it deal with out of order, or split into functional components, or split frontend/backend what you suggested. For most of them cost vs benefit is more or less not completely clear, neither how much effort was invested to look into them.

One thing I missed from this explanation is how drm_scheduler per engine class interferes with the high level concepts. And I did not manage to pick up on what exactly is the TDR problem in that case. Maybe the two are one and the same.

Bottom line is I still have the concern that conversion to kworkers has an opportunity to regress. Possibly more opportunity for some Xe use cases than to affect other vendors, since they would still be using per physical engine / queue scheduler instances.

And to put my money where my mouth is I will try to put testing Xe inside the full blown ChromeOS environment in my team plans. It would probably also be beneficial if Xe team could take a look at real world behaviour of the extreme transcode use cases too. If the stack is ready for that and all. It would be better to know earlier rather than later if there is a fundamental issue.

For the patch at hand, and the cover letter, it certainly feels it would benefit to record the past design discussion had with AMD folks, to explicitly copy other drivers, and to record the theoretical pros and cons of threads vs unbound workers as I have tried to highlight them.

Regards,

Tvrtko

author maintainer
#27 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 113 lines · Tvrtko Ursulin
Quoted 102 lines · Jason Ekstrand

On Tue, Jan 10, 2023 at 5:28 AM Tvrtko Ursulin <[email protected] <mailto:[email protected]>> wrote:

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

>      >>> AFAICT it proposes to have 1:1 between *userspace* created >     contexts (per
>      >>> context _and_ engine) and drm_sched. I am not sure avoiding >     invasive changes
>      >>> to the shared code is in the spirit of the overall idea
and instead
>      >>> opportunity should be used to look at way to
refactor/improve
>     drm_sched.
>
>
> Maybe?  I'm not convinced that what Xe is doing is an abuse at
all or
> really needs to drive a re-factor.  (More on that later.)
There's only
> one real issue which is that it fires off potentially a lot of
kthreads.
> Even that's not that bad given that kthreads are pretty light and
you're
> not likely to have more kthreads than userspace threads which are
much
> heavier.  Not ideal, but not the end of the world either.
Definitely
> something we can/should optimize but if we went through with Xe
without
> this patch, it would probably be mostly ok.
>
>      >> Yes, it is 1:1 *userspace* engines and drm_sched. >      >>
>      >> I'm not really prepared to make large changes to DRM
scheduler
>     at the
>      >> moment for Xe as they are not really required nor does Boris >     seem they
>      >> will be required for his work either. I am interested to see >     what Boris
>      >> comes up with.
>      >>
>      >>> Even on the low level, the idea to replace drm_sched threads >     with workers
>      >>> has a few problems.
>      >>>
>      >>> To start with, the pattern of:
>      >>>
>      >>>    while (not_stopped) {
>      >>>     keep picking jobs
>      >>>    }
>      >>>
>      >>> Feels fundamentally in disagreement with workers (while >     obviously fits
>      >>> perfectly with the current kthread design).
>      >>
>      >> The while loop breaks and worker exists if no jobs are ready. >
>
> I'm not very familiar with workqueues. What are you saying would fit > better? One scheduling job per work item rather than one big work
item
> which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch
because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued
work items (more interesting when mixing up with the system_unbound_wq),
in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts.

Quoted 5 lines

Those are pretty much the same in practice.  Rather, user threads is typically an upper bound on the number of contexts.  Yes, a single user thread could have a bunch of contexts but basically nothing does that except IGT.  In real-world usage, it's at most one context per user thread.

Typically is the key here. But I am not sure it is good enough. Consider this example - Intel Flex 170:

Quoted 9 lines

* Delivers up to 36 streams 1080p60 transcode throughput per card.
* When scaled to 10 cards in a 4U server configuration, it can support up
to 360 streams of HEVC/HEVC 1080p60 transcode throughput.

One transcode stream from my experience typically is 3-4 GPU contexts (buffer travels from vcs -> rcs -> vcs, maybe vecs) used from a single CPU thread. 4 contexts * 36 streams = 144 active contexts. Multiply by 60fps = 8640 jobs submitted and completed per second.

See my reply with my numbers based running xe_exec_threads, on a TGL we are getting 33711 jobs per sec /w 640 xe_engines. This seems to scale just fine.

Quoted 8 lines

144 active contexts in the proposed scheme means possibly means 144 kernel worker threads spawned (driven by 36 transcode CPU threads). (I don't think the pools would scale down given all are constantly pinged at 60fps.)

And then each of 144 threads goes to grab the single GuC CT mutex. First threads are being made schedulable, then put to sleep as mutex contention is hit, then woken again as mutexes are getting released, rinse, repeat.

Quoted 9 lines

(And yes this backend contention is there regardless of 1:1:1, it would require a different re-design to solve that. But it is just a question whether there are 144 contending threads, or just 6 with the thread per engine class scheme.)

Then multiply all by 10 for a 4U server use case and you get 1440 worker kthreads, yes 10 more CT locks, but contending on how many CPU cores? Just so they can grab a timeslice and maybe content on a mutex as the next step.

Same as above, this seems to scale just fine as I bet the above example of 33711 job per sec is limited by a single GuC context switching rather than Xe being about to feed the GuC. Also certainly a server in this configuration is going to a CPU much faster than the TGL I was using.

Also did another quick change to use 1280 xe_engines in xe_exec_threads:

root@DUT025-TGLU:igt-gpu-tools# xe_exec_threads --r threads-basic
IGT-Version: 1.26-ge26de4b2 (x86_64) (Linux: 6.1.0-rc1-xe+ x86_64)
Starting subtest: threads-basic
Subtest threads-basic: SUCCESS (1.198s)

More or less same results as 640 xe_engines.

Quoted 16 lines

This example is where it would hurt on large systems. Imagine only an even wider media transcode card...

Second example is only a single engine class used (3d desktop?) but with a bunch of not-runnable jobs queued and waiting on a fence to signal. Implicit or explicit dependencies doesn't matter. Then the fence signals and call backs run. N work items get scheduled, but they all submit to the same HW engine. So we end up with:

        /-- wi1 --\
       / ..     .. \
 cb --+---  wi.. ---+-- rq1 -- .. -- rqN
       \ ..    ..  /
        \-- wiN --/
Quoted 4 lines

All that we have achieved is waking up N CPUs to contend on the same lock and effectively insert the job into the same single HW queue. I don't see any positives there.

I've said this before, the CT channel in practice isn't going to be full so the section of code protected by the mutex is really, really small. The mutex really shouldn't ever have contention. Also does a mutex spin for small period of time before going to sleep? I seem to recall some type of core lock did this, if we can use a lock that spins for short period of time this argument falls apart.

Quoted 165 lines

This example I think can particularly hurt small / low power devices because of needless waking up of many cores for no benefit. Granted, I don't have a good feel on how common this pattern is in practice.

Quoted 129 lines

That
is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Several problems there.

It is fundamentally pointless to have potentially that many more threads
than the number of CPU cores - it simply creates a scheduling storm.

Unbound workers have no CPU / cache locality either and no connection with the CPU scheduler to optimize scheduling patterns. This may matter either on large systems or on small ones. Whereas the current design allows for scheduler to notice userspace CPU thread keeps waking up the same drm scheduler kernel thread, and so it can keep them on the same CPU, the unbound workers lose that ability and so 2nd CPU might be getting woken up from low sleep for every submission.

Hence, apart from being a bit of a impedance mismatch, the proposal has the potential to change performance and power patterns and both large and small machines.

Ok, thanks for explaining the issue you're seeing in more detail.  Yes, deferred kwork does appear to mismatch somewhat with what the scheduler needs or at least how it's worked in the past.  How much impact will that mismatch have?  Unclear.

>      >>> Secondly, it probably demands separate workers (not
optional),
>     otherwise
>      >>> behaviour of shared workqueues has either the potential to >     explode number
>      >>> kernel threads anyway, or add latency.
>      >>>
>      >>
>      >> Right now the system_unbound_wq is used which does have a
limit
>     on the
>      >> number of threads, right? I do have a FIXME to allow a
worker to be
>      >> passed in similar to TDR.
>      >>
>      >> WRT to latency, the 1:1 ratio could actually have lower
latency
>     as 2 GPU
>      >> schedulers can be pushing jobs into the backend / cleaning up >     jobs in
>      >> parallel.
>      >>
>      >
>      > Thought of one more point here where why in Xe we
absolutely want
>     a 1 to
>      > 1 ratio between entity and scheduler - the way we implement >     timeslicing
>      > for preempt fences.
>      >
>      > Let me try to explain.
>      >
>      > Preempt fences are implemented via the generic messaging >     interface [1]
>      > with suspend / resume messages. If a suspend messages is
received to
>      > soon after calling resume (this is per entity) we simply
sleep in the
>      > suspend call thus giving the entity a timeslice. This
completely
>     falls
>      > apart with a many to 1 relationship as now a entity
waiting for a
>      > timeslice blocks the other entities. Could we work aroudn
this,
>     sure but
>      > just another bunch of code we'd have to add in Xe. Being to >     freely sleep
>      > in backend without affecting other entities is really, really >     nice IMO
>      > and I bet Xe isn't the only driver that is going to feel
this way.
>      >
>      > Last thing I'll say regardless of how anyone feels about
Xe using
>     a 1 to
>      > 1 relationship this patch IMO makes sense as I hope we can all >     agree a
>      > workqueue scales better than kthreads.
>
>     I don't know for sure what will scale better and for what use
case,
>     combination of CPU cores vs number of GPU engines to keep
busy vs other
>     system activity. But I wager someone is bound to ask for some >     numbers to
>     make sure proposal is not negatively affecting any other drivers. >
>
> Then let them ask.  Waving your hands vaguely in the direction of
the
> rest of DRM and saying "Uh, someone (not me) might object" is
profoundly
> unhelpful.  Sure, someone might.  That's why it's on dri-devel.
If you
> think there's someone in particular who might have a useful
opinion on
> this, throw them in the CC so they don't miss the e-mail thread. >
> Or are you asking for numbers?  If so, what numbers are you
asking for?

It was a heads up to the Xe team in case people weren't appreciating how
the proposed change has the potential influence power and performance across the board. And nothing in the follow up discussion made me think it was considered so I don't think it was redundant to raise it.

In my experience it is typical that such core changes come with some numbers. Which is in case of drm scheduler is tricky and probably requires explicitly asking everyone to test (rather than count on "don't
miss the email thread"). Real products can fail to ship due ten mW here or there. Like suddenly an extra core prevented from getting into deep sleep.

If that was "profoundly unhelpful" so be it.

Quoted 6 lines

With your above explanation, it makes more sense what you're asking. It's still not something Matt is likely to be able to provide on his own.  We need to tag some other folks and ask them to test it out.  We could play around a bit with it on Xe but it's not exactly production grade yet and is going to hit this differently from most.  Likely candidates are probably AMD and Freedreno.

Whoever is setup to check out power and performance would be good to give it a spin, yes.

PS. I don't think I was asking Matt to test with other devices. To start with I think Xe is a team effort. I was asking for more background on the design decision since patch 4/20 does not say anything on that angle, nor later in the thread it was IMO sufficiently addressed.

Quoted 17 lines

> Also, If we're talking about a design that might paint us into an > Intel-HW-specific hole, that would be one thing.  But we're not.
We're
> talking about switching which kernel threading/task mechanism to
use for
> what's really a very generic problem.  The core Xe design works
without
> this patch (just with more kthreads).  If we land this patch or > something like it and get it wrong and it causes a performance
problem
> for someone down the line, we can revisit it.

For some definition of "it works" - I really wouldn't suggest shipping a
kthread per user context at any point.

Quoted 6 lines
Quoted 3 lines

You have yet to elaborate on why. What resources is it consuming that's going to be a problem? Are you anticipating CPU affinity problems? Or does it just seem wasteful?

Well I don't know, commit message says the approach does not scale. :)

I don't think we want a user interface to directly be able to create a kthread, that seems like a bad idea which Christian pointed out to us off the list last March.

Quoted 286 lines
Quoted 83 lines

I think I largely agree that it's probably unnecessary/wasteful but reducing the number of kthreads seems like a tractable problem to solve regardless of where we put the gpu_scheduler object.  Is this the right solution?  Maybe not.  It was also proposed at one point that we could split the scheduler into two pieces: A scheduler which owns the kthread, and a back-end which targets some HW ring thing where you can have multiple back-ends per scheduler.  That's certainly more invasive from a DRM scheduler internal API PoV but would solve the kthread problem in a way that's more similar to what we have now.

>     In any case that's a low level question caused by the high
level design
>     decision. So I'd think first focus on the high level - which
is the 1:1
>     mapping of entity to scheduler instance proposal.
>
>     Fundamentally it will be up to the DRM maintainers and the
community to
>     bless your approach. And it is important to stress 1:1 is about >     userspace contexts, so I believe unlike any other current
scheduler
>     user. And also important to stress this effectively does not
make Xe
>     _really_ use the scheduler that much.
>
>
> I don't think this makes Xe nearly as much of a one-off as you
think it
> does.  I've already told the Asahi team working on Apple M1/2
hardware
> to do it this way and it seems to be a pretty good mapping for
them. I
> believe this is roughly the plan for nouveau as well.  It's not
the way
> it currently works for anyone because most other groups aren't
doing FW
> scheduling yet.  In the world of FW scheduling and hardware
designed to
> support userspace direct-to-FW submit, I think the design makes
perfect
> sense (see below) and I expect we'll see more drivers move in this > direction as those drivers evolve.  (AMD is doing some customish
thing
> for how with gpu_scheduler on the front-end somehow. I've not dug
into
> those details.)
>
>     I can only offer my opinion, which is that the two options
mentioned in
>     this thread (either improve drm scheduler to cope with what is >     required,
>     or split up the code so you can use just the parts of
drm_sched which
>     you want - which is frontend dependency tracking) shouldn't be so >     readily dismissed, given how I think the idea was for the new
driver to
>     work less in a silo and more in the community (not do kludges to >     workaround stuff because it is thought to be too hard to
improve common
>     code), but fundamentally, "goto previous paragraph" for what I am >     concerned.
>
>
> Meta comment:  It appears as if you're falling into the standard
i915
> team trap of having an internal discussion about what the community > discussion might look like instead of actually having the community > discussion.  If you are seriously concerned about interactions with > other drivers or whether or setting common direction, the right
way to
> do that is to break a patch or two out into a separate RFC series
and
> tag a handful of driver maintainers.  Trying to predict the
questions
> other people might ask is pointless. Cc them and asking for their
input
> instead.

I don't follow you here. It's not an internal discussion - I am raising my concerns on the design publicly. I am supposed to write a patch to show something, but am allowed to comment on a RFC series?

Quoted 4 lines

I may have misread your tone a bit.  It felt a bit like too many discussions I've had in the past where people are trying to predict what others will say instead of just asking them.  Reading it again, I was probably jumping to conclusions a bit.  Sorry about that.

Okay no problem, thanks. In any case we don't have to keep discussing it, since I wrote one or two emails ago it is fundamentally on the maintainers and community to ack the approach. I only felt like RFC did not explain the potential downsides sufficiently so I wanted to probe that area a bit.

Quoted 84 lines

It is "drm/sched: Convert drm scheduler to use a work queue rather than kthread" which should have Cc-ed _everyone_ who use drm scheduler.

Yeah, it probably should have.  I think that's mostly what I've been trying to say.

>
>     Regards,
>
>     Tvrtko
>
>     P.S. And as a related side note, there are more areas where
drm_sched
>     could be improved, like for instance priority handling. >     Take a look at msm_submitqueue_create /
msm_gpu_convert_priority /
>     get_sched_entity to see how msm works around the drm_sched
hardcoded
>     limit of available priority levels, in order to avoid having
to leave a
>     hw capability unused. I suspect msm would be happier if they
could have
>     all priority levels equal in terms of whether they apply only
at the
>     frontend level or completely throughout the pipeline. >
>      > [1]
>
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
>
 <https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>>
>      >
>      >>> What would be interesting to learn is whether the option of >     refactoring
>      >>> drm_sched to deal with out of order completion was
considered
>     and what were
>      >>> the conclusions.
>      >>>
>      >>
>      >> I coded this up a while back when trying to convert the
i915 to
>     the DRM
>      >> scheduler it isn't all that hard either. The free flow
control
>     on the
>      >> ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE) is >     really what
>      >> sold me on the this design.
>
>
> You're not the only one to suggest supporting out-of-order
completion.
> However, it's tricky and breaks a lot of internal assumptions of the > scheduler. It also reduces functionality a bit because it can no
longer
> automatically rate-limit HW/FW queues which are often
fixed-size.  (Ok,
> yes, it probably could but it becomes a substantially harder
problem.)
>
> It also seems like a worse mapping to me.  The goal here is to turn > submissions on a userspace-facing engine/queue into submissions
to a FW
> queue submissions, sorting out any dma_fence dependencies.  Matt's > description of saying this is a 1:1 mapping between sched/entity
doesn't
> tell the whole story. It's a 1:1:1 mapping between xe_engine, > gpu_scheduler, and GuC FW engine.  Why make it a 1:something:1
mapping?
> Why is that better?

As I have stated before, what I think what would fit well for Xe is one drm_scheduler per engine class. In specific terms on our current hardware, one drm scheduler instance for render, compute, blitter, video
and video enhance. Userspace contexts remain scheduler entities.

And this is where we fairly strongly disagree.  More in a bit.

Quoted 7 lines

That way you avoid the whole kthread/kworker story and you have it actually use the entity picking code in the scheduler, which may be useful when the backend is congested.

What back-end congestion are you referring to here?  Running out of FW queue IDs?  Something else?

CT channel, number of context ids.

Quoted 79 lines

Yes you have to solve the out of order problem so in my mind that is something to discuss. What the problem actually is (just TDR?), how tricky and why etc.

And yes you lose the handy LRCA ring buffer size management so you'd have to make those entities not runnable in some other way.

Regarding the argument you raise below - would any of that make the frontend / backend separation worse and why? Do you think it is less natural? If neither is true then all remains is that it appears extra work to support out of order completion of entities has been discounted in favour of an easy but IMO inelegant option.

Broadly speaking, the kernel needs to stop thinking about GPU scheduling in terms of scheduling jobs and start thinking in terms of scheduling contexts/engines.  There is still some need for scheduling individual jobs but that is only for the purpose of delaying them as needed to resolve dma_fence dependencies.  Once dependencies are resolved, they get shoved onto the context/engine queue and from there the kernel only really manages whole contexts/engines.  This is a major architectural shift, entirely different from the way i915 scheduling works.  It's also different from the historical usage of DRM scheduler which I think is why this all looks a bit funny.

To justify this architectural shift, let's look at where we're headed. In the glorious future...

 1. Userspace submits directly to firmware queues.  The kernel has no
visibility whatsoever into individual jobs.  At most it can pause/resume FW contexts as needed to handle eviction and memory management.

 2. Because of 1, apart from handing out the FW queue IDs at the
beginning, the kernel can't really juggle them that much.  Depending on FW design, it may be able to pause a client, give its IDs to another, and then resume it later when IDs free up.  What it's not doing is juggling IDs on a job-by-job basis like i915 currently is.

 3. Long-running compute jobs may not complete for days.  This means
that memory management needs to happen in terms of pause/resume of entire contexts/engines using the memory rather than based on waiting for individual jobs to complete or pausing individual jobs until the memory is available.

 4. Synchronization happens via userspace memory fences (UMF) and the
kernel is mostly unaware of most dependencies and when a context/engine is or is not runnable.  Instead, it keeps as many of them minimally active (memory is available, even if it's in system RAM) as possible and lets the FW sort out dependencies.  (There may need to be some facility for sleeping a context until a memory change similar to futex() or poll() for userspace threads.  There are some details TBD.)

Are there potential problems that will need to be solved here?  Yes.  Is it a good design?  Well, Microsoft has been living in this future for half a decade or better and it's working quite well for them.  It's also the way all modern game consoles work.  It really is just Linux that's stuck with the same old job model we've had since the monumental shift to DRI2.

To that end, one of the core goals of the Xe project was to make the driver internally behave as close to the above model as possible while keeping the old-school job model as a very thin layer on top.  As the broader ecosystem problems (window-system support for UMF, for instance) are solved, that layer can be peeled back.  The core driver will already be ready for it.

To that end, the point of the DRM scheduler in Xe isn't to schedule jobs.  It's to resolve syncobj and dma-buf implicit sync dependencies and stuff jobs into their respective context/engine queue once they're ready.  All the actual scheduling happens in firmware and any scheduling the kernel does to deal with contention, oversubscriptions, too many contexts, etc. is between contexts/engines, not individual jobs.  Sure, the individual job visibility is nice, but if we design around it, we'll never get to the glorious future.

I really need to turn the above (with a bit more detail) into a blog post.... Maybe I'll do that this week.

Quoted 6 lines

In any case, I hope that provides more insight into why Xe is designed the way it is and why I'm pushing back so hard on trying to make it more of a "classic" driver as far as scheduling is concerned.  Are there potential problems here?  Yes, that's why Xe has been labeled a prototype.  Are such radical changes necessary to get to said glorious future?  Yes, I think they are.  Will it be worth it?  I believe so.

Right, that's all solid I think. My takeaway is that frontend priority sorting and that stuff isn't needed and that is okay. And that there are multiple options to maybe improve drm scheduler, like the fore mentioned making it deal with out of order, or split into functional components, or split frontend/backend what you suggested. For most of them cost vs benefit is more or less not completely clear, neither how much effort was invested to look into them.

One thing I missed from this explanation is how drm_scheduler per engine class interferes with the high level concepts. And I did not manage to pick up on what exactly is the TDR problem in that case. Maybe the two are one and the same.

Quoted 5 lines

Bottom line is I still have the concern that conversion to kworkers has an opportunity to regress. Possibly more opportunity for some Xe use cases than to affect other vendors, since they would still be using per physical engine / queue scheduler instances.

We certainly don't want to affect other vendors but I haven't yet heard any push back from other vendors. I don't think speculating about potential problems is helpful.

Quoted 7 lines

And to put my money where my mouth is I will try to put testing Xe inside the full blown ChromeOS environment in my team plans. It would probably also be beneficial if Xe team could take a look at real world behaviour of the extreme transcode use cases too. If the stack is ready for that and all. It would be better to know earlier rather than later if there is a fundamental issue.

We don't have a media UMD yet it will be tough to test at this point in time. Also not sure when Xe is going to be POR for a Chrome product either so porting Xe into ChromeOS likely isn't a top priority for your team. I know from experience that porting things into ChromeOS isn't trivial as I've support several of these efforts. Not saying don't do this just mentioning the realities of what you are suggesting.

Matt

Quoted 8 lines

For the patch at hand, and the cover letter, it certainly feels it would benefit to record the past design discussion had with AMD folks, to explicitly copy other drivers, and to record the theoretical pros and cons of threads vs unbound workers as I have tried to highlight them.

Regards,

Tvrtko

#28 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 1 line · Matthew Brost

On Wed, Jan 11, 2023 at 08:50:37AM +0000, Tvrtko Ursulin wrote:

[snip]

Quoted 21 lines
Quoted 16 lines

This example is where it would hurt on large systems. Imagine only an even wider media transcode card...

Second example is only a single engine class used (3d desktop?) but with a bunch of not-runnable jobs queued and waiting on a fence to signal. Implicit or explicit dependencies doesn't matter. Then the fence signals and call backs run. N work items get scheduled, but they all submit to the same HW engine. So we end up with:

         /-- wi1 --\
        / ..     .. \
  cb --+---  wi.. ---+-- rq1 -- .. -- rqN
        \ ..    ..  /
         \-- wiN --/
Quoted 4 lines

All that we have achieved is waking up N CPUs to contend on the same lock and effectively insert the job into the same single HW queue. I don't see any positives there.

Quoted 6 lines

I've said this before, the CT channel in practice isn't going to be full so the section of code protected by the mutex is really, really small. The mutex really shouldn't ever have contention. Also does a mutex spin for small period of time before going to sleep? I seem to recall some type of core lock did this, if we can use a lock that spins for short period of time this argument falls apart.

This argument already fell apart when we established it's the system_wq and not the unbound one. So a digression only - it did not fall apart because of the CT channel not being ever congested, there would still be the question of what's the point to wake up N cpus when there is a single work channel in the backend.

You would have been able to bypass all that by inserting work items directly, not via the scheduler workers. I thought that was what Jason was implying when he mentioned that a better frontend/backend drm scheduler split was considered at some point.

Because for 1:1:1, where GuC is truly 1, it does seem it would work better if that sort of a split would enable you to queue directly into the backend bypassing the kthread/worker / wait_on wake_up dance.

Would that work? From drm_sched_entity_push_job directly to the backend - not waking up but *calling* the equivalent of drm_sched_main.

Quoted 19 lines
Quoted 13 lines

Right, that's all solid I think. My takeaway is that frontend priority sorting and that stuff isn't needed and that is okay. And that there are multiple options to maybe improve drm scheduler, like the fore mentioned making it deal with out of order, or split into functional components, or split frontend/backend what you suggested. For most of them cost vs benefit is more or less not completely clear, neither how much effort was invested to look into them.

One thing I missed from this explanation is how drm_scheduler per engine class interferes with the high level concepts. And I did not manage to pick up on what exactly is the TDR problem in that case. Maybe the two are one and the same.

Quoted 5 lines

Bottom line is I still have the concern that conversion to kworkers has an opportunity to regress. Possibly more opportunity for some Xe use cases than to affect other vendors, since they would still be using per physical engine / queue scheduler instances.

Quoted 3 lines

We certainly don't want to affect other vendors but I haven't yet heard any push back from other vendors. I don't think speculating about potential problems is helpful.

I haven't had any push back on the drm cgroup controller either. :D

Quoted 8 lines
Quoted 7 lines

And to put my money where my mouth is I will try to put testing Xe inside the full blown ChromeOS environment in my team plans. It would probably also be beneficial if Xe team could take a look at real world behaviour of the extreme transcode use cases too. If the stack is ready for that and all. It would be better to know earlier rather than later if there is a fundamental issue.

Quoted 6 lines

We don't have a media UMD yet it will be tough to test at this point in time. Also not sure when Xe is going to be POR for a Chrome product either so porting Xe into ChromeOS likely isn't a top priority for your team. I know from experience that porting things into ChromeOS isn't trivial as I've support several of these efforts. Not saying don't do this just mentioning the realities of what you are suggesting.

I know, I only said I'd put it in the plans, not that it will happen tomorrow.

Regards,

Tvrtko

#29 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

In reply to Tvrtko Ursulin
Quoted 118 lines · Tvrtko Ursulin
Quoted 7 lines · Jason Ekstrand

On Tue, Jan 10, 2023 at 5:28 AM Tvrtko Ursulin <[email protected] <mailto:[email protected]>>

wrote:

Quoted 5 lines
    On 09/01/2023 17:27, Jason Ekstrand wrote:

    [snip]

     >      >>> AFAICT it proposes to have 1:1 between *userspace*

created

Quoted 2 lines

> contexts (per
> >>> context _and_ engine) and drm_sched. I am not sure

avoiding

Quoted 22 lines

> invasive changes
> >>> to the shared code is in the spirit of the overall idea
and instead
> >>> opportunity should be used to look at way to
refactor/improve
> drm_sched.
>
>
> Maybe? I'm not convinced that what Xe is doing is an abuse at
all or
> really needs to drive a re-factor. (More on that later.)
There's only
> one real issue which is that it fires off potentially a lot of
kthreads.
> Even that's not that bad given that kthreads are pretty light and
you're
> not likely to have more kthreads than userspace threads which are
much
> heavier. Not ideal, but not the end of the world either.
Definitely
> something we can/should optimize but if we went through with Xe
without

Quoted 8 lines

> this patch, it would probably be mostly ok. >
> >> Yes, it is 1:1 *userspace* engines and drm_sched.
> >>
> >> I'm not really prepared to make large changes to DRM
scheduler
> at the
> >> moment for Xe as they are not really required nor does

Boris

Quoted 2 lines

> seem they
> >> will be required for his work either. I am interested to

see

Quoted 4 lines

> what Boris
> >> comes up with.
> >>
> >>> Even on the low level, the idea to replace drm_sched

threads

Quoted 6 lines

> with workers
> >>> has a few problems.
> >>>
> >>> To start with, the pattern of:
> >>>
> >>> while (not_stopped) {

Quoted 8 lines

> >>> keep picking jobs
> >>> }
> >>>
> >>> Feels fundamentally in disagreement with workers (while
> obviously fits
> >>> perfectly with the current kthread design).
> >>
> >> The while loop breaks and worker exists if no jobs are

ready.

Quoted 3 lines
     >
     >
     > I'm not very familiar with workqueues. What are you saying would

fit

Quoted 11 lines

> better? One scheduling job per work item rather than one big work
item
> which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch
because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked

this

Quoted 8 lines

random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent

work

Quoted 17 lines

processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued
work items (more interesting when mixing up with the system_unbound_wq),
in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts.

Quoted 4 lines

Those are pretty much the same in practice. Rather, user threads is typically an upper bound on the number of contexts. Yes, a single user thread could have a bunch of contexts but basically nothing does that except IGT. In real-world usage, it's at most one context per user

Quoted 9 lines

thread.

Typically is the key here. But I am not sure it is good enough. Consider this example - Intel Flex 170:

* Delivers up to 36 streams 1080p60 transcode throughput per card.
* When scaled to 10 cards in a 4U server configuration, it can support
up to 360 streams of HEVC/HEVC 1080p60 transcode throughput.

I had a feeling it was going to be media.... 😅

Quoted 10 lines

One transcode stream from my experience typically is 3-4 GPU contexts (buffer travels from vcs -> rcs -> vcs, maybe vecs) used from a single CPU thread. 4 contexts * 36 streams = 144 active contexts. Multiply by 60fps = 8640 jobs submitted and completed per second.

144 active contexts in the proposed scheme means possibly means 144 kernel worker threads spawned (driven by 36 transcode CPU threads). (I don't think the pools would scale down given all are constantly pinged at 60fps.)

Quoted 5 lines

And then each of 144 threads goes to grab the single GuC CT mutex. First threads are being made schedulable, then put to sleep as mutex contention is hit, then woken again as mutexes are getting released, rinse, repeat.

Why is every submission grabbing the GuC CT mutex? I've not read the GuC back-end yet but I was under the impression that most run_job() would be just shoving another packet into a ring buffer. If we have to send the GuC a message on the control ring every single time we submit a job, that's pretty horrible.

--Jason

(And yes this backend contention is there regardless of 1:1:1, it would

Quoted 538 lines

require a different re-design to solve that. But it is just a question whether there are 144 contending threads, or just 6 with the thread per engine class scheme.)

Then multiply all by 10 for a 4U server use case and you get 1440 worker kthreads, yes 10 more CT locks, but contending on how many CPU cores? Just so they can grab a timeslice and maybe content on a mutex as the next step.

This example is where it would hurt on large systems. Imagine only an even wider media transcode card...

Second example is only a single engine class used (3d desktop?) but with a bunch of not-runnable jobs queued and waiting on a fence to signal. Implicit or explicit dependencies doesn't matter. Then the fence signals and call backs run. N work items get scheduled, but they all submit to the same HW engine. So we end up with:

         /-- wi1 --\
        / ..     .. \
  cb --+---  wi.. ---+-- rq1 -- .. -- rqN
        \ ..    ..  /
         \-- wiN --/

All that we have achieved is waking up N CPUs to contend on the same lock and effectively insert the job into the same single HW queue. I don't see any positives there.

This example I think can particularly hurt small / low power devices because of needless waking up of many cores for no benefit. Granted, I don't have a good feel on how common this pattern is in practice.

Quoted 3 lines

That
is the number which drives the maximum number of not-runnable jobs

that

Quoted 1 line

can become runnable at once, and hence spawn that many work items,

and

Quoted 10 lines

in turn unbound worker threads.

Several problems there.

It is fundamentally pointless to have potentially that many more threads
than the number of CPU cores - it simply creates a scheduling storm.

Unbound workers have no CPU / cache locality either and no connection with the CPU scheduler to optimize scheduling patterns. This may

matter

Quoted 2 lines

either on large systems or on small ones. Whereas the current design allows for scheduler to notice userspace CPU thread keeps waking up

the

Quoted 5 lines

same drm scheduler kernel thread, and so it can keep them on the same CPU, the unbound workers lose that ability and so 2nd CPU might be getting woken up from low sleep for every submission.

Hence, apart from being a bit of a impedance mismatch, the proposal

has

Quoted 9 lines

the potential to change performance and power patterns and both large and small machines.

Ok, thanks for explaining the issue you're seeing in more detail. Yes, deferred kwork does appear to mismatch somewhat with what the scheduler needs or at least how it's worked in the past. How much impact will that mismatch have? Unclear.

Quoted 4 lines

> >>> Secondly, it probably demands separate workers (not
optional),
> otherwise
> >>> behaviour of shared workqueues has either the potential

to

Quoted 7 lines

> explode number
> >>> kernel threads anyway, or add latency.
> >>>
> >>
> >> Right now the system_unbound_wq is used which does have a
limit
> on the

Quoted 8 lines

> >> number of threads, right? I do have a FIXME to allow a
worker to be
> >> passed in similar to TDR.
> >>
> >> WRT to latency, the 1:1 ratio could actually have lower
latency
> as 2 GPU
> >> schedulers can be pushing jobs into the backend /

cleaning up

Quoted 22 lines

> jobs in
> >> parallel.
> >>
> >
> > Thought of one more point here where why in Xe we
absolutely want
> a 1 to
> > 1 ratio between entity and scheduler - the way we implement
> timeslicing
> > for preempt fences.
> >
> > Let me try to explain.
> >
> > Preempt fences are implemented via the generic messaging
> interface [1]
> > with suspend / resume messages. If a suspend messages is
received to
> > soon after calling resume (this is per entity) we simply
sleep in the
> > suspend call thus giving the entity a timeslice. This
completely
> falls

Quoted 8 lines

> > apart with a many to 1 relationship as now a entity
waiting for a
> > timeslice blocks the other entities. Could we work aroudn
this,
> sure but
> > just another bunch of code we'd have to add in Xe. Being to
> freely sleep
> > in backend without affecting other entities is really,

really

Quoted 8 lines

> nice IMO
> > and I bet Xe isn't the only driver that is going to feel
this way.
> >
> > Last thing I'll say regardless of how anyone feels about
Xe using
> a 1 to
> > 1 relationship this patch IMO makes sense as I hope we can

all

Quoted 10 lines

> agree a
> > workqueue scales better than kthreads.
>
> I don't know for sure what will scale better and for what use
case,
> combination of CPU cores vs number of GPU engines to keep
busy vs other
> system activity. But I wager someone is bound to ask for some
> numbers to
> make sure proposal is not negatively affecting any other

drivers.

Quoted 15 lines

>
>
> Then let them ask. Waving your hands vaguely in the direction of
the
> rest of DRM and saying "Uh, someone (not me) might object" is
profoundly
> unhelpful. Sure, someone might. That's why it's on dri-devel.
If you
> think there's someone in particular who might have a useful
opinion on
> this, throw them in the CC so they don't miss the e-mail thread. >
> Or are you asking for numbers? If so, what numbers are you
asking for?

Quoted 4 lines

It was a heads up to the Xe team in case people weren't appreciating how
the proposed change has the potential influence power and performance across the board. And nothing in the follow up discussion made me

think

Quoted 7 lines

it was considered so I don't think it was redundant to raise it.

In my experience it is typical that such core changes come with some numbers. Which is in case of drm scheduler is tricky and probably requires explicitly asking everyone to test (rather than count on "don't
miss the email thread"). Real products can fail to ship due ten mW

here

Quoted 1 line

or there. Like suddenly an extra core prevented from getting into

deep

Quoted 11 lines

sleep.

If that was "profoundly unhelpful" so be it.

With your above explanation, it makes more sense what you're asking. It's still not something Matt is likely to be able to provide on his own. We need to tag some other folks and ask them to test it out. We could play around a bit with it on Xe but it's not exactly production grade yet and is going to hit this differently from most. Likely candidates are probably AMD and Freedreno.

Whoever is setup to check out power and performance would be good to give it a spin, yes.

PS. I don't think I was asking Matt to test with other devices. To start with I think Xe is a team effort. I was asking for more background on the design decision since patch 4/20 does not say anything on that angle, nor later in the thread it was IMO sufficiently addressed.

Quoted 12 lines

> Also, If we're talking about a design that might paint us into an > Intel-HW-specific hole, that would be one thing. But we're not.
We're
> talking about switching which kernel threading/task mechanism to
use for
> what's really a very generic problem. The core Xe design works
without
> this patch (just with more kthreads). If we land this patch or > something like it and get it wrong and it causes a performance
problem
> for someone down the line, we can revisit it.

Quoted 8 lines

For some definition of "it works" - I really wouldn't suggest shipping a
kthread per user context at any point.

You have yet to elaborate on why. What resources is it consuming that's going to be a problem? Are you anticipating CPU affinity problems? Or does it just seem wasteful?

Well I don't know, commit message says the approach does not scale. :)

Quoted 10 lines

I think I largely agree that it's probably unnecessary/wasteful but reducing the number of kthreads seems like a tractable problem to solve regardless of where we put the gpu_scheduler object. Is this the right solution? Maybe not. It was also proposed at one point that we could split the scheduler into two pieces: A scheduler which owns the kthread, and a back-end which targets some HW ring thing where you can have multiple back-ends per scheduler. That's certainly more invasive from a DRM scheduler internal API PoV but would solve the kthread problem in a way that's more similar to what we have now.

Quoted 9 lines

> In any case that's a low level question caused by the high
level design
> decision. So I'd think first focus on the high level - which
is the 1:1
> mapping of entity to scheduler instance proposal.
>
> Fundamentally it will be up to the DRM maintainers and the
community to
> bless your approach. And it is important to stress 1:1 is

about

Quoted 27 lines

> userspace contexts, so I believe unlike any other current
scheduler
> user. And also important to stress this effectively does not
make Xe
> _really_ use the scheduler that much.
>
>
> I don't think this makes Xe nearly as much of a one-off as you
think it
> does. I've already told the Asahi team working on Apple M1/2
hardware
> to do it this way and it seems to be a pretty good mapping for
them. I
> believe this is roughly the plan for nouveau as well. It's not
the way
> it currently works for anyone because most other groups aren't
doing FW
> scheduling yet. In the world of FW scheduling and hardware
designed to
> support userspace direct-to-FW submit, I think the design makes
perfect
> sense (see below) and I expect we'll see more drivers move in this > direction as those drivers evolve. (AMD is doing some customish
thing
> for how with gpu_scheduler on the front-end somehow. I've not dug
into
> those details.)

Quoted 8 lines

>
> I can only offer my opinion, which is that the two options
mentioned in
> this thread (either improve drm scheduler to cope with what is
> required,
> or split up the code so you can use just the parts of
drm_sched which
> you want - which is frontend dependency tracking) shouldn't

be so

Quoted 3 lines

> readily dismissed, given how I think the idea was for the new
driver to
> work less in a silo and more in the community (not do kludges

to

Quoted 3 lines

> workaround stuff because it is thought to be too hard to
improve common
> code), but fundamentally, "goto previous paragraph" for what

I am

Quoted 6 lines
     >     concerned.
     >
     >
     > Meta comment:  It appears as if you're falling into the standard
    i915
     > team trap of having an internal discussion about what the

community

Quoted 1 line

> discussion might look like instead of actually having the

community

Quoted 1 line

> discussion. If you are seriously concerned about interactions

with

Quoted 11 lines

> other drivers or whether or setting common direction, the right
way to
> do that is to break a patch or two out into a separate RFC series
and
> tag a handful of driver maintainers. Trying to predict the
questions
> other people might ask is pointless. Cc them and asking for their
input
> instead.

I don't follow you here. It's not an internal discussion - I am

raising

Quoted 8 lines

my concerns on the design publicly. I am supposed to write a patch to show something, but am allowed to comment on a RFC series?

I may have misread your tone a bit. It felt a bit like too many discussions I've had in the past where people are trying to predict what others will say instead of just asking them. Reading it again, I was probably jumping to conclusions a bit. Sorry about that.

Okay no problem, thanks. In any case we don't have to keep discussing it, since I wrote one or two emails ago it is fundamentally on the maintainers and community to ack the approach. I only felt like RFC did not explain the potential downsides sufficiently so I wanted to probe that area a bit.

Quoted 1 line

It is "drm/sched: Convert drm scheduler to use a work queue rather

than

Quoted 25 lines

kthread" which should have Cc-ed _everyone_ who use drm scheduler.

Yeah, it probably should have. I think that's mostly what I've been trying to say.

     >
     >     Regards,
     >
     >     Tvrtko
     >
     >     P.S. And as a related side note, there are more areas where
    drm_sched
     >     could be improved, like for instance priority handling.
     >     Take a look at msm_submitqueue_create /
    msm_gpu_convert_priority /
     >     get_sched_entity to see how msm works around the drm_sched
    hardcoded
     >     limit of available priority levels, in order to avoid having
    to leave a
     >     hw capability unused. I suspect msm would be happier if they
    could have
     >     all priority levels equal in terms of whether they apply only
    at the
     >     frontend level or completely throughout the pipeline.
Quoted 8 lines
     >
     >      > [1]
     >
    https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
    <https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1

     >
      <

https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1 <
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>>

Quoted 2 lines

> >
> >>> What would be interesting to learn is whether the option

of

Quoted 6 lines

> refactoring
> >>> drm_sched to deal with out of order completion was
considered
> and what were
> >>> the conclusions.
> >>>

Quoted 8 lines

> >>
> >> I coded this up a while back when trying to convert the
i915 to
> the DRM
> >> scheduler it isn't all that hard either. The free flow
control
> on the
> >> ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE)

is

Quoted 7 lines

> really what
> >> sold me on the this design.
>
>
> You're not the only one to suggest supporting out-of-order
completion.
> However, it's tricky and breaks a lot of internal assumptions of

the

Quoted 8 lines

> scheduler. It also reduces functionality a bit because it can no
longer
> automatically rate-limit HW/FW queues which are often
fixed-size. (Ok,
> yes, it probably could but it becomes a substantially harder
problem.)
>
> It also seems like a worse mapping to me. The goal here is to

turn

Quoted 11 lines

> submissions on a userspace-facing engine/queue into submissions
to a FW
> queue submissions, sorting out any dma_fence dependencies. Matt's > description of saying this is a 1:1 mapping between sched/entity
doesn't
> tell the whole story. It's a 1:1:1 mapping between xe_engine, > gpu_scheduler, and GuC FW engine. Why make it a 1:something:1
mapping?
> Why is that better?

As I have stated before, what I think what would fit well for Xe is

one

Quoted 8 lines

drm_scheduler per engine class. In specific terms on our current hardware, one drm scheduler instance for render, compute, blitter, video
and video enhance. Userspace contexts remain scheduler entities.

And this is where we fairly strongly disagree. More in a bit.

Quoted 7 lines

That way you avoid the whole kthread/kworker story and you have it actually use the entity picking code in the scheduler, which may be useful when the backend is congested.

What back-end congestion are you referring to here? Running out of FW queue IDs? Something else?

CT channel, number of context ids.

Quoted 5 lines

Yes you have to solve the out of order problem so in my mind that is something to discuss. What the problem actually is (just TDR?), how tricky and why etc.

Quoted 7 lines

And yes you lose the handy LRCA ring buffer size management so you'd have to make those entities not runnable in some other way.

Regarding the argument you raise below - would any of that make the frontend / backend separation worse and why? Do you think it is less natural? If neither is true then all remains is that it appears extra work to support out of order completion of entities has been

discounted

Quoted 67 lines

in favour of an easy but IMO inelegant option.

Broadly speaking, the kernel needs to stop thinking about GPU scheduling in terms of scheduling jobs and start thinking in terms of scheduling contexts/engines. There is still some need for scheduling individual jobs but that is only for the purpose of delaying them as needed to resolve dma_fence dependencies. Once dependencies are resolved, they get shoved onto the context/engine queue and from there the kernel only really manages whole contexts/engines. This is a major architectural shift, entirely different from the way i915 scheduling works. It's also different from the historical usage of DRM scheduler which I think is why this all looks a bit funny.

To justify this architectural shift, let's look at where we're headed. In the glorious future...

1. Userspace submits directly to firmware queues. The kernel has no
visibility whatsoever into individual jobs. At most it can pause/resume FW contexts as needed to handle eviction and memory management.

2. Because of 1, apart from handing out the FW queue IDs at the
beginning, the kernel can't really juggle them that much. Depending on FW design, it may be able to pause a client, give its IDs to another, and then resume it later when IDs free up. What it's not doing is juggling IDs on a job-by-job basis like i915 currently is.

3. Long-running compute jobs may not complete for days. This means
that memory management needs to happen in terms of pause/resume of entire contexts/engines using the memory rather than based on waiting for individual jobs to complete or pausing individual jobs until the memory is available.

4. Synchronization happens via userspace memory fences (UMF) and the
kernel is mostly unaware of most dependencies and when a context/engine is or is not runnable. Instead, it keeps as many of them minimally active (memory is available, even if it's in system RAM) as possible and lets the FW sort out dependencies. (There may need to be some facility for sleeping a context until a memory change similar to futex() or poll() for userspace threads. There are some details TBD.)

Are there potential problems that will need to be solved here? Yes. Is it a good design? Well, Microsoft has been living in this future for half a decade or better and it's working quite well for them. It's also the way all modern game consoles work. It really is just Linux that's stuck with the same old job model we've had since the monumental shift to DRI2.

To that end, one of the core goals of the Xe project was to make the driver internally behave as close to the above model as possible while keeping the old-school job model as a very thin layer on top. As the broader ecosystem problems (window-system support for UMF, for instance) are solved, that layer can be peeled back. The core driver will already be ready for it.

To that end, the point of the DRM scheduler in Xe isn't to schedule jobs. It's to resolve syncobj and dma-buf implicit sync dependencies and stuff jobs into their respective context/engine queue once they're ready. All the actual scheduling happens in firmware and any scheduling the kernel does to deal with contention, oversubscriptions, too many contexts, etc. is between contexts/engines, not individual jobs. Sure, the individual job visibility is nice, but if we design around it, we'll never get to the glorious future.

I really need to turn the above (with a bit more detail) into a blog post.... Maybe I'll do that this week.

Quoted 6 lines

In any case, I hope that provides more insight into why Xe is designed the way it is and why I'm pushing back so hard on trying to make it more of a "classic" driver as far as scheduling is concerned. Are there potential problems here? Yes, that's why Xe has been labeled a prototype. Are such radical changes necessary to get to said glorious future? Yes, I think they are. Will it be worth it? I believe so.

Right, that's all solid I think. My takeaway is that frontend priority sorting and that stuff isn't needed and that is okay. And that there are multiple options to maybe improve drm scheduler, like the fore mentioned making it deal with out of order, or split into functional components, or split frontend/backend what you suggested. For most of them cost vs benefit is more or less not completely clear, neither how much effort was invested to look into them.

One thing I missed from this explanation is how drm_scheduler per engine class interferes with the high level concepts. And I did not manage to pick up on what exactly is the TDR problem in that case. Maybe the two are one and the same.

Bottom line is I still have the concern that conversion to kworkers has an opportunity to regress. Possibly more opportunity for some Xe use cases than to affect other vendors, since they would still be using per physical engine / queue scheduler instances.

And to put my money where my mouth is I will try to put testing Xe inside the full blown ChromeOS environment in my team plans. It would probably also be beneficial if Xe team could take a look at real world behaviour of the extreme transcode use cases too. If the stack is ready for that and all. It would be better to know earlier rather than later if there is a fundamental issue.

For the patch at hand, and the cover letter, it certainly feels it would benefit to record the past design discussion had with AMD folks, to explicitly copy other drivers, and to record the theoretical pros and cons of threads vs unbound workers as I have tried to highlight them.

Regards,

Tvrtko

author maintainer
#30 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 126 lines · Jason Ekstrand
Quoted 123 lines · Tvrtko Ursulin
Quoted 7 lines · Jason Ekstrand

On Tue, Jan 10, 2023 at 5:28 AM Tvrtko Ursulin <[email protected] <mailto:[email protected]>>

wrote:

Quoted 5 lines
    On 09/01/2023 17:27, Jason Ekstrand wrote:

    [snip]

     >      >>> AFAICT it proposes to have 1:1 between *userspace*

created

Quoted 2 lines

> contexts (per
> >>> context _and_ engine) and drm_sched. I am not sure

avoiding

Quoted 22 lines

> invasive changes
> >>> to the shared code is in the spirit of the overall idea
and instead
> >>> opportunity should be used to look at way to
refactor/improve
> drm_sched.
>
>
> Maybe? I'm not convinced that what Xe is doing is an abuse at
all or
> really needs to drive a re-factor. (More on that later.)
There's only
> one real issue which is that it fires off potentially a lot of
kthreads.
> Even that's not that bad given that kthreads are pretty light and
you're
> not likely to have more kthreads than userspace threads which are
much
> heavier. Not ideal, but not the end of the world either.
Definitely
> something we can/should optimize but if we went through with Xe
without

Quoted 8 lines

> this patch, it would probably be mostly ok. >
> >> Yes, it is 1:1 *userspace* engines and drm_sched.
> >>
> >> I'm not really prepared to make large changes to DRM
scheduler
> at the
> >> moment for Xe as they are not really required nor does

Boris

Quoted 2 lines

> seem they
> >> will be required for his work either. I am interested to

see

Quoted 4 lines

> what Boris
> >> comes up with.
> >>
> >>> Even on the low level, the idea to replace drm_sched

threads

Quoted 6 lines

> with workers
> >>> has a few problems.
> >>>
> >>> To start with, the pattern of:
> >>>
> >>> while (not_stopped) {

Quoted 8 lines

> >>> keep picking jobs
> >>> }
> >>>
> >>> Feels fundamentally in disagreement with workers (while
> obviously fits
> >>> perfectly with the current kthread design).
> >>
> >> The while loop breaks and worker exists if no jobs are

ready.

Quoted 3 lines
     >
     >
     > I'm not very familiar with workqueues. What are you saying would

fit

Quoted 11 lines

> better? One scheduling job per work item rather than one big work
item
> which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch
because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked

this

Quoted 8 lines

random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent

work

Quoted 17 lines

processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued
work items (more interesting when mixing up with the system_unbound_wq),
in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts.

Quoted 4 lines

Those are pretty much the same in practice. Rather, user threads is typically an upper bound on the number of contexts. Yes, a single user thread could have a bunch of contexts but basically nothing does that except IGT. In real-world usage, it's at most one context per user

thread.

Typically is the key here. But I am not sure it is good enough. Consider this example - Intel Flex 170:

Quoted 7 lines
Quoted 4 lines

* Delivers up to 36 streams 1080p60 transcode throughput per card.
* When scaled to 10 cards in a 4U server configuration, it can support
up to 360 streams of HEVC/HEVC 1080p60 transcode throughput.

I had a feeling it was going to be media.... 😅

Yea wondering the media UMD can be rewritten to use less xe_engines, it is massive rewrite for VM bind + no implicit dependencies so let's just pile on some more work?

Quoted 17 lines
Quoted 11 lines

One transcode stream from my experience typically is 3-4 GPU contexts (buffer travels from vcs -> rcs -> vcs, maybe vecs) used from a single CPU thread. 4 contexts * 36 streams = 144 active contexts. Multiply by 60fps = 8640 jobs submitted and completed per second.

144 active contexts in the proposed scheme means possibly means 144 kernel worker threads spawned (driven by 36 transcode CPU threads). (I don't think the pools would scale down given all are constantly pinged at 60fps.)

Quoted 5 lines

And then each of 144 threads goes to grab the single GuC CT mutex. First threads are being made schedulable, then put to sleep as mutex contention is hit, then woken again as mutexes are getting released, rinse, repeat.

Quoted 6 lines

Why is every submission grabbing the GuC CT mutex? I've not read the GuC back-end yet but I was under the impression that most run_job() would be just shoving another packet into a ring buffer. If we have to send the GuC a message on the control ring every single time we submit a job, that's pretty horrible.

Run job writes the ring buffer and moves the tail as the first step (no lock required). Next it needs to tell the GuC the xe_engine LRC tail has moved, this is done from a single Host to GuC channel which is circular buffer, the writing of the channel protected by the mutex. There are little more nuances too but in practice there is always space in the channel so the time mutex needs to held is really, really small (check cached credits, write 3 dwords in payload, write 1 dword to move tail). I also believe mutexes in Linux are hybrid where they spin for a little bit before sleeping and certainly if there is space in the channel we shouldn't sleep mutex contention.

As far as this being horrible, well didn't design the GuC and this how it is implemented for KMD based submission. We also have 256 doorbells so we wouldn't need a lock but I think are other issues with that design too which need to be worked out in the Xe2 / Xe3 timeframe.

Also if you see my follow up response Xe is ~33k execs per second with the current implementation on a 8 core (or maybe 8 thread) TGL which seems to fine to me.

Matt

Quoted 542 lines

--Jason

(And yes this backend contention is there regardless of 1:1:1, it would

Quoted 538 lines

require a different re-design to solve that. But it is just a question whether there are 144 contending threads, or just 6 with the thread per engine class scheme.)

Then multiply all by 10 for a 4U server use case and you get 1440 worker kthreads, yes 10 more CT locks, but contending on how many CPU cores? Just so they can grab a timeslice and maybe content on a mutex as the next step.

This example is where it would hurt on large systems. Imagine only an even wider media transcode card...

Second example is only a single engine class used (3d desktop?) but with a bunch of not-runnable jobs queued and waiting on a fence to signal. Implicit or explicit dependencies doesn't matter. Then the fence signals and call backs run. N work items get scheduled, but they all submit to the same HW engine. So we end up with:

         /-- wi1 --\
        / ..     .. \
  cb --+---  wi.. ---+-- rq1 -- .. -- rqN
        \ ..    ..  /
         \-- wiN --/

All that we have achieved is waking up N CPUs to contend on the same lock and effectively insert the job into the same single HW queue. I don't see any positives there.

This example I think can particularly hurt small / low power devices because of needless waking up of many cores for no benefit. Granted, I don't have a good feel on how common this pattern is in practice.

Quoted 3 lines

That
is the number which drives the maximum number of not-runnable jobs

that

Quoted 1 line

can become runnable at once, and hence spawn that many work items,

and

Quoted 10 lines

in turn unbound worker threads.

Several problems there.

It is fundamentally pointless to have potentially that many more threads
than the number of CPU cores - it simply creates a scheduling storm.

Unbound workers have no CPU / cache locality either and no connection with the CPU scheduler to optimize scheduling patterns. This may

matter

Quoted 2 lines

either on large systems or on small ones. Whereas the current design allows for scheduler to notice userspace CPU thread keeps waking up

the

Quoted 5 lines

same drm scheduler kernel thread, and so it can keep them on the same CPU, the unbound workers lose that ability and so 2nd CPU might be getting woken up from low sleep for every submission.

Hence, apart from being a bit of a impedance mismatch, the proposal

has

Quoted 9 lines

the potential to change performance and power patterns and both large and small machines.

Ok, thanks for explaining the issue you're seeing in more detail. Yes, deferred kwork does appear to mismatch somewhat with what the scheduler needs or at least how it's worked in the past. How much impact will that mismatch have? Unclear.

Quoted 4 lines

> >>> Secondly, it probably demands separate workers (not
optional),
> otherwise
> >>> behaviour of shared workqueues has either the potential

to

Quoted 7 lines

> explode number
> >>> kernel threads anyway, or add latency.
> >>>
> >>
> >> Right now the system_unbound_wq is used which does have a
limit
> on the

Quoted 8 lines

> >> number of threads, right? I do have a FIXME to allow a
worker to be
> >> passed in similar to TDR.
> >>
> >> WRT to latency, the 1:1 ratio could actually have lower
latency
> as 2 GPU
> >> schedulers can be pushing jobs into the backend /

cleaning up

Quoted 22 lines

> jobs in
> >> parallel.
> >>
> >
> > Thought of one more point here where why in Xe we
absolutely want
> a 1 to
> > 1 ratio between entity and scheduler - the way we implement
> timeslicing
> > for preempt fences.
> >
> > Let me try to explain.
> >
> > Preempt fences are implemented via the generic messaging
> interface [1]
> > with suspend / resume messages. If a suspend messages is
received to
> > soon after calling resume (this is per entity) we simply
sleep in the
> > suspend call thus giving the entity a timeslice. This
completely
> falls

Quoted 8 lines

> > apart with a many to 1 relationship as now a entity
waiting for a
> > timeslice blocks the other entities. Could we work aroudn
this,
> sure but
> > just another bunch of code we'd have to add in Xe. Being to
> freely sleep
> > in backend without affecting other entities is really,

really

Quoted 8 lines

> nice IMO
> > and I bet Xe isn't the only driver that is going to feel
this way.
> >
> > Last thing I'll say regardless of how anyone feels about
Xe using
> a 1 to
> > 1 relationship this patch IMO makes sense as I hope we can

all

Quoted 10 lines

> agree a
> > workqueue scales better than kthreads.
>
> I don't know for sure what will scale better and for what use
case,
> combination of CPU cores vs number of GPU engines to keep
busy vs other
> system activity. But I wager someone is bound to ask for some
> numbers to
> make sure proposal is not negatively affecting any other

drivers.

Quoted 15 lines

>
>
> Then let them ask. Waving your hands vaguely in the direction of
the
> rest of DRM and saying "Uh, someone (not me) might object" is
profoundly
> unhelpful. Sure, someone might. That's why it's on dri-devel.
If you
> think there's someone in particular who might have a useful
opinion on
> this, throw them in the CC so they don't miss the e-mail thread. >
> Or are you asking for numbers? If so, what numbers are you
asking for?

Quoted 4 lines

It was a heads up to the Xe team in case people weren't appreciating how
the proposed change has the potential influence power and performance across the board. And nothing in the follow up discussion made me

think

Quoted 7 lines

it was considered so I don't think it was redundant to raise it.

In my experience it is typical that such core changes come with some numbers. Which is in case of drm scheduler is tricky and probably requires explicitly asking everyone to test (rather than count on "don't
miss the email thread"). Real products can fail to ship due ten mW

here

Quoted 1 line

or there. Like suddenly an extra core prevented from getting into

deep

Quoted 11 lines

sleep.

If that was "profoundly unhelpful" so be it.

With your above explanation, it makes more sense what you're asking. It's still not something Matt is likely to be able to provide on his own. We need to tag some other folks and ask them to test it out. We could play around a bit with it on Xe but it's not exactly production grade yet and is going to hit this differently from most. Likely candidates are probably AMD and Freedreno.

Whoever is setup to check out power and performance would be good to give it a spin, yes.

PS. I don't think I was asking Matt to test with other devices. To start with I think Xe is a team effort. I was asking for more background on the design decision since patch 4/20 does not say anything on that angle, nor later in the thread it was IMO sufficiently addressed.

Quoted 12 lines

> Also, If we're talking about a design that might paint us into an > Intel-HW-specific hole, that would be one thing. But we're not.
We're
> talking about switching which kernel threading/task mechanism to
use for
> what's really a very generic problem. The core Xe design works
without
> this patch (just with more kthreads). If we land this patch or > something like it and get it wrong and it causes a performance
problem
> for someone down the line, we can revisit it.

Quoted 8 lines

For some definition of "it works" - I really wouldn't suggest shipping a
kthread per user context at any point.

You have yet to elaborate on why. What resources is it consuming that's going to be a problem? Are you anticipating CPU affinity problems? Or does it just seem wasteful?

Well I don't know, commit message says the approach does not scale. :)

Quoted 10 lines

I think I largely agree that it's probably unnecessary/wasteful but reducing the number of kthreads seems like a tractable problem to solve regardless of where we put the gpu_scheduler object. Is this the right solution? Maybe not. It was also proposed at one point that we could split the scheduler into two pieces: A scheduler which owns the kthread, and a back-end which targets some HW ring thing where you can have multiple back-ends per scheduler. That's certainly more invasive from a DRM scheduler internal API PoV but would solve the kthread problem in a way that's more similar to what we have now.

Quoted 9 lines

> In any case that's a low level question caused by the high
level design
> decision. So I'd think first focus on the high level - which
is the 1:1
> mapping of entity to scheduler instance proposal.
>
> Fundamentally it will be up to the DRM maintainers and the
community to
> bless your approach. And it is important to stress 1:1 is

about

Quoted 27 lines

> userspace contexts, so I believe unlike any other current
scheduler
> user. And also important to stress this effectively does not
make Xe
> _really_ use the scheduler that much.
>
>
> I don't think this makes Xe nearly as much of a one-off as you
think it
> does. I've already told the Asahi team working on Apple M1/2
hardware
> to do it this way and it seems to be a pretty good mapping for
them. I
> believe this is roughly the plan for nouveau as well. It's not
the way
> it currently works for anyone because most other groups aren't
doing FW
> scheduling yet. In the world of FW scheduling and hardware
designed to
> support userspace direct-to-FW submit, I think the design makes
perfect
> sense (see below) and I expect we'll see more drivers move in this > direction as those drivers evolve. (AMD is doing some customish
thing
> for how with gpu_scheduler on the front-end somehow. I've not dug
into
> those details.)

Quoted 8 lines

>
> I can only offer my opinion, which is that the two options
mentioned in
> this thread (either improve drm scheduler to cope with what is
> required,
> or split up the code so you can use just the parts of
drm_sched which
> you want - which is frontend dependency tracking) shouldn't

be so

Quoted 3 lines

> readily dismissed, given how I think the idea was for the new
driver to
> work less in a silo and more in the community (not do kludges

to

Quoted 3 lines

> workaround stuff because it is thought to be too hard to
improve common
> code), but fundamentally, "goto previous paragraph" for what

I am

Quoted 6 lines
     >     concerned.
     >
     >
     > Meta comment:  It appears as if you're falling into the standard
    i915
     > team trap of having an internal discussion about what the

community

Quoted 1 line

> discussion might look like instead of actually having the

community

Quoted 1 line

> discussion. If you are seriously concerned about interactions

with

Quoted 11 lines

> other drivers or whether or setting common direction, the right
way to
> do that is to break a patch or two out into a separate RFC series
and
> tag a handful of driver maintainers. Trying to predict the
questions
> other people might ask is pointless. Cc them and asking for their
input
> instead.

I don't follow you here. It's not an internal discussion - I am

raising

Quoted 8 lines

my concerns on the design publicly. I am supposed to write a patch to show something, but am allowed to comment on a RFC series?

I may have misread your tone a bit. It felt a bit like too many discussions I've had in the past where people are trying to predict what others will say instead of just asking them. Reading it again, I was probably jumping to conclusions a bit. Sorry about that.

Okay no problem, thanks. In any case we don't have to keep discussing it, since I wrote one or two emails ago it is fundamentally on the maintainers and community to ack the approach. I only felt like RFC did not explain the potential downsides sufficiently so I wanted to probe that area a bit.

Quoted 1 line

It is "drm/sched: Convert drm scheduler to use a work queue rather

than

Quoted 25 lines

kthread" which should have Cc-ed _everyone_ who use drm scheduler.

Yeah, it probably should have. I think that's mostly what I've been trying to say.

     >
     >     Regards,
     >
     >     Tvrtko
     >
     >     P.S. And as a related side note, there are more areas where
    drm_sched
     >     could be improved, like for instance priority handling.
     >     Take a look at msm_submitqueue_create /
    msm_gpu_convert_priority /
     >     get_sched_entity to see how msm works around the drm_sched
    hardcoded
     >     limit of available priority levels, in order to avoid having
    to leave a
     >     hw capability unused. I suspect msm would be happier if they
    could have
     >     all priority levels equal in terms of whether they apply only
    at the
     >     frontend level or completely throughout the pipeline.
Quoted 8 lines
     >
     >      > [1]
     >
    https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
    <https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1

     >
      <

https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1 <
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>>

Quoted 2 lines

> >
> >>> What would be interesting to learn is whether the option

of

Quoted 6 lines

> refactoring
> >>> drm_sched to deal with out of order completion was
considered
> and what were
> >>> the conclusions.
> >>>

Quoted 8 lines

> >>
> >> I coded this up a while back when trying to convert the
i915 to
> the DRM
> >> scheduler it isn't all that hard either. The free flow
control
> on the
> >> ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE)

is

Quoted 7 lines

> really what
> >> sold me on the this design.
>
>
> You're not the only one to suggest supporting out-of-order
completion.
> However, it's tricky and breaks a lot of internal assumptions of

the

Quoted 8 lines

> scheduler. It also reduces functionality a bit because it can no
longer
> automatically rate-limit HW/FW queues which are often
fixed-size. (Ok,
> yes, it probably could but it becomes a substantially harder
problem.)
>
> It also seems like a worse mapping to me. The goal here is to

turn

Quoted 11 lines

> submissions on a userspace-facing engine/queue into submissions
to a FW
> queue submissions, sorting out any dma_fence dependencies. Matt's > description of saying this is a 1:1 mapping between sched/entity
doesn't
> tell the whole story. It's a 1:1:1 mapping between xe_engine, > gpu_scheduler, and GuC FW engine. Why make it a 1:something:1
mapping?
> Why is that better?

As I have stated before, what I think what would fit well for Xe is

one

Quoted 8 lines

drm_scheduler per engine class. In specific terms on our current hardware, one drm scheduler instance for render, compute, blitter, video
and video enhance. Userspace contexts remain scheduler entities.

And this is where we fairly strongly disagree. More in a bit.

Quoted 7 lines

That way you avoid the whole kthread/kworker story and you have it actually use the entity picking code in the scheduler, which may be useful when the backend is congested.

What back-end congestion are you referring to here? Running out of FW queue IDs? Something else?

CT channel, number of context ids.

Quoted 5 lines

Yes you have to solve the out of order problem so in my mind that is something to discuss. What the problem actually is (just TDR?), how tricky and why etc.

Quoted 7 lines

And yes you lose the handy LRCA ring buffer size management so you'd have to make those entities not runnable in some other way.

Regarding the argument you raise below - would any of that make the frontend / backend separation worse and why? Do you think it is less natural? If neither is true then all remains is that it appears extra work to support out of order completion of entities has been

discounted

Quoted 67 lines

in favour of an easy but IMO inelegant option.

Broadly speaking, the kernel needs to stop thinking about GPU scheduling in terms of scheduling jobs and start thinking in terms of scheduling contexts/engines. There is still some need for scheduling individual jobs but that is only for the purpose of delaying them as needed to resolve dma_fence dependencies. Once dependencies are resolved, they get shoved onto the context/engine queue and from there the kernel only really manages whole contexts/engines. This is a major architectural shift, entirely different from the way i915 scheduling works. It's also different from the historical usage of DRM scheduler which I think is why this all looks a bit funny.

To justify this architectural shift, let's look at where we're headed. In the glorious future...

1. Userspace submits directly to firmware queues. The kernel has no
visibility whatsoever into individual jobs. At most it can pause/resume FW contexts as needed to handle eviction and memory management.

2. Because of 1, apart from handing out the FW queue IDs at the
beginning, the kernel can't really juggle them that much. Depending on FW design, it may be able to pause a client, give its IDs to another, and then resume it later when IDs free up. What it's not doing is juggling IDs on a job-by-job basis like i915 currently is.

3. Long-running compute jobs may not complete for days. This means
that memory management needs to happen in terms of pause/resume of entire contexts/engines using the memory rather than based on waiting for individual jobs to complete or pausing individual jobs until the memory is available.

4. Synchronization happens via userspace memory fences (UMF) and the
kernel is mostly unaware of most dependencies and when a context/engine is or is not runnable. Instead, it keeps as many of them minimally active (memory is available, even if it's in system RAM) as possible and lets the FW sort out dependencies. (There may need to be some facility for sleeping a context until a memory change similar to futex() or poll() for userspace threads. There are some details TBD.)

Are there potential problems that will need to be solved here? Yes. Is it a good design? Well, Microsoft has been living in this future for half a decade or better and it's working quite well for them. It's also the way all modern game consoles work. It really is just Linux that's stuck with the same old job model we've had since the monumental shift to DRI2.

To that end, one of the core goals of the Xe project was to make the driver internally behave as close to the above model as possible while keeping the old-school job model as a very thin layer on top. As the broader ecosystem problems (window-system support for UMF, for instance) are solved, that layer can be peeled back. The core driver will already be ready for it.

To that end, the point of the DRM scheduler in Xe isn't to schedule jobs. It's to resolve syncobj and dma-buf implicit sync dependencies and stuff jobs into their respective context/engine queue once they're ready. All the actual scheduling happens in firmware and any scheduling the kernel does to deal with contention, oversubscriptions, too many contexts, etc. is between contexts/engines, not individual jobs. Sure, the individual job visibility is nice, but if we design around it, we'll never get to the glorious future.

I really need to turn the above (with a bit more detail) into a blog post.... Maybe I'll do that this week.

Quoted 6 lines

In any case, I hope that provides more insight into why Xe is designed the way it is and why I'm pushing back so hard on trying to make it more of a "classic" driver as far as scheduling is concerned. Are there potential problems here? Yes, that's why Xe has been labeled a prototype. Are such radical changes necessary to get to said glorious future? Yes, I think they are. Will it be worth it? I believe so.

Right, that's all solid I think. My takeaway is that frontend priority sorting and that stuff isn't needed and that is okay. And that there are multiple options to maybe improve drm scheduler, like the fore mentioned making it deal with out of order, or split into functional components, or split frontend/backend what you suggested. For most of them cost vs benefit is more or less not completely clear, neither how much effort was invested to look into them.

One thing I missed from this explanation is how drm_scheduler per engine class interferes with the high level concepts. And I did not manage to pick up on what exactly is the TDR problem in that case. Maybe the two are one and the same.

Bottom line is I still have the concern that conversion to kworkers has an opportunity to regress. Possibly more opportunity for some Xe use cases than to affect other vendors, since they would still be using per physical engine / queue scheduler instances.

And to put my money where my mouth is I will try to put testing Xe inside the full blown ChromeOS environment in my team plans. It would probably also be beneficial if Xe team could take a look at real world behaviour of the extreme transcode use cases too. If the stack is ready for that and all. It would be better to know earlier rather than later if there is a fundamental issue.

For the patch at hand, and the cover letter, it certainly feels it would benefit to record the past design discussion had with AMD folks, to explicitly copy other drivers, and to record the theoretical pros and cons of threads vs unbound workers as I have tried to highlight them.

Regards,

Tvrtko

#31 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 152 lines · Matthew Brost
Quoted 7 lines · Jason Ekstrand
Quoted 4 lines · Tvrtko Ursulin
Quoted 2 lines · Jason Ekstrand

On Tue, Jan 10, 2023 at 5:28 AM Tvrtko Ursulin
<[email protected] <mailto:

[email protected]>>

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

wrote:

    On 09/01/2023 17:27, Jason Ekstrand wrote:

    [snip]
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

> >>> AFAICT it proposes to have 1:1 between *userspace*

created

Quoted 2 lines

> contexts (per
> >>> context _and_ engine) and drm_sched. I am not sure

avoiding

Quoted 2 lines

> invasive changes
> >>> to the shared code is in the spirit of the overall

idea

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

and instead
> >>> opportunity should be used to look at way to
refactor/improve
> drm_sched.
>

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

>
> Maybe? I'm not convinced that what Xe is doing is an abuse at
all or
> really needs to drive a re-factor. (More on that later.)
There's only
> one real issue which is that it fires off potentially a lot of
kthreads.
> Even that's not that bad given that kthreads are pretty light

and

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

you're
> not likely to have more kthreads than userspace threads which

are

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

much
> heavier. Not ideal, but not the end of the world either.
Definitely
> something we can/should optimize but if we went through with

Xe

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

without
> this patch, it would probably be mostly ok. >
> >> Yes, it is 1:1 *userspace* engines and drm_sched.
> >>
> >> I'm not really prepared to make large changes to DRM
scheduler
> at the
> >> moment for Xe as they are not really required nor does

Quoted 3 lines
Quoted 3 lines

Boris

Quoted 2 lines

> seem they
> >> will be required for his work either. I am interested

to

Quoted 6 lines
Quoted 6 lines

see

Quoted 4 lines

> what Boris
> >> comes up with.
> >>
> >>> Even on the low level, the idea to replace drm_sched

threads

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

> with workers
> >>> has a few problems.
> >>>
> >>> To start with, the pattern of:
> >>>
> >>> while (not_stopped) {
> >>> keep picking jobs
> >>> }
> >>>
> >>> Feels fundamentally in disagreement with workers

(while

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

> obviously fits
> >>> perfectly with the current kthread design).
> >>
> >> The while loop breaks and worker exists if no jobs are

ready.

Quoted 3 lines
     >
     >
     > I'm not very familiar with workqueues. What are you saying

would

Quoted 2 lines
Quoted 2 lines

fit

Quoted 1 line

> better? One scheduling job per work item rather than one big

work

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

item
> which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which

is

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

potentially unbound in runtime. But it is a bit moot conceptual mismatch
because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've

picked

Quoted 2 lines
Quoted 2 lines

this

Quoted 1 line

random spot to consolidate what I have already mentioned and

perhaps

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume

many

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

cycles. That means maintaining just enough concurrency to prevent

work

Quoted 6 lines

processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the

users.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

"""

Given the unbound queues will be spawned on demand to service all queued
work items (more interesting when mixing up with the system_unbound_wq),
in the proposed design the number of instantiated worker threads

does

Quoted 1 line
Quoted 1 line
Quoted 1 line

not correspond to the number of user threads (as you have

elsewhere

Quoted 1 line
Quoted 1 line
Quoted 1 line

stated), but pessimistically to the number of active user

contexts.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Those are pretty much the same in practice. Rather, user threads is typically an upper bound on the number of contexts. Yes, a single

user

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

thread could have a bunch of contexts but basically nothing does that except IGT. In real-world usage, it's at most one context per user

thread.

Typically is the key here. But I am not sure it is good enough.

Consider

Quoted 4 lines
Quoted 4 lines

this example - Intel Flex 170:

* Delivers up to 36 streams 1080p60 transcode throughput per card.
* When scaled to 10 cards in a 4U server configuration, it can

support

Quoted 3 lines
Quoted 2 lines

up to 360 streams of HEVC/HEVC 1080p60 transcode throughput.

Quoted 7 lines
Quoted 2 lines

I had a feeling it was going to be media.... 😅

Yea wondering the media UMD can be rewritten to use less xe_engines, it is massive rewrite for VM bind + no implicit dependencies so let's just pile on some more work?

It could probably use fewer than it does today. It currently creates and throws away contexts like crazy, or did last I looked at it. However, the nature of media encode is that it often spreads across two or three different types of engines. There's not much you can do to change that.

Quoted 26 lines
Quoted 6 lines
Quoted 6 lines

One transcode stream from my experience typically is 3-4 GPU contexts (buffer travels from vcs -> rcs -> vcs, maybe vecs) used from a single CPU thread. 4 contexts * 36 streams = 144 active contexts. Multiply by 60fps = 8640 jobs submitted and completed per second.

Quoted 6 lines
Quoted 6 lines

144 active contexts in the proposed scheme means possibly means 144 kernel worker threads spawned (driven by 36 transcode CPU threads). (I don't think the pools would scale down given all are constantly pinged at 60fps.)

And then each of 144 threads goes to grab the single GuC CT mutex.

First

Quoted 8 lines
Quoted 4 lines

threads are being made schedulable, then put to sleep as mutex contention is hit, then woken again as mutexes are getting released, rinse, repeat.

Why is every submission grabbing the GuC CT mutex? I've not read the GuC back-end yet but I was under the impression that most run_job() would be just shoving another packet into a ring buffer. If we have to send the

GuC

Quoted 3 lines

a message on the control ring every single time we submit a job, that's pretty horrible.

Quoted 11 lines

Run job writes the ring buffer and moves the tail as the first step (no lock required). Next it needs to tell the GuC the xe_engine LRC tail has moved, this is done from a single Host to GuC channel which is circular buffer, the writing of the channel protected by the mutex. There are little more nuances too but in practice there is always space in the channel so the time mutex needs to held is really, really small (check cached credits, write 3 dwords in payload, write 1 dword to move tail). I also believe mutexes in Linux are hybrid where they spin for a little bit before sleeping and certainly if there is space in the channel we shouldn't sleep mutex contention.

Ok, that makes sense. It's maybe a bit clunky and it'd be nice if we had some way to batch things up a bit so we only have to poke the GuC channel once for every batch of things rather than once per job. That's maybe something we can look into as a future improvement; not fundamental.

Generally, though, it sounds like contention could be a real problem if we end up ping-ponging that lock between cores. It's going to depend on how much work it takes to get the next ready thing vs. the cost of that atomic. But, also, anything we do is going to potentially run into contention problems. *shrug* If we were going to go for one-per-HW-engine, we may as well go one-per-device and then we wouldn't need the lock. Off the top of my head, that doesn't sound great either but IDK.

Quoted 5 lines

As far as this being horrible, well didn't design the GuC and this how it is implemented for KMD based submission. We also have 256 doorbells so we wouldn't need a lock but I think are other issues with that design too which need to be worked out in the Xe2 / Xe3 timeframe.

Yeah, not blaming you. Just surprised, that's all. How does it work for userspace submission? What would it look like if the kernel emulated userspace submission? Is that even possible?

What are these doorbell things? How do they play into it?

Quoted 4 lines

Also if you see my follow up response Xe is ~33k execs per second with the current implementation on a 8 core (or maybe 8 thread) TGL which seems to fine to me.

33k exec/sec is about 500/frame which should be fine. 500 is a lot for a single frame. I typically tell game devs to shoot for dozens per frame. The important thing is that it stays low even with hundreds of memory objects bound. (Xe should be just fine there.)

--Jason

Quoted 669 lines

Matt

Quoted 9 lines

--Jason

(And yes this backend contention is there regardless of 1:1:1, it would

Quoted 5 lines

require a different re-design to solve that. But it is just a question whether there are 144 contending threads, or just 6 with the thread per engine class scheme.)

Then multiply all by 10 for a 4U server use case and you get 1440

worker

Quoted 8 lines
Quoted 8 lines

kthreads, yes 10 more CT locks, but contending on how many CPU cores? Just so they can grab a timeslice and maybe content on a mutex as the next step.

This example is where it would hurt on large systems. Imagine only an even wider media transcode card...

Second example is only a single engine class used (3d desktop?) but

with

Quoted 2 lines
Quoted 2 lines

a bunch of not-runnable jobs queued and waiting on a fence to signal. Implicit or explicit dependencies doesn't matter. Then the fence

signals

Quoted 14 lines
Quoted 14 lines

and call backs run. N work items get scheduled, but they all submit to the same HW engine. So we end up with:

         /-- wi1 --\
        / ..     .. \
  cb --+---  wi.. ---+-- rq1 -- .. -- rqN
        \ ..    ..  /
         \-- wiN --/

All that we have achieved is waking up N CPUs to contend on the same lock and effectively insert the job into the same single HW queue. I don't see any positives there.

Quoted 7 lines
Quoted 7 lines

This example I think can particularly hurt small / low power devices because of needless waking up of many cores for no benefit. Granted, I don't have a good feel on how common this pattern is in practice.

Quoted 3 lines

That
is the number which drives the maximum number of not-runnable

jobs

Quoted 2 lines
Quoted 2 lines

that

Quoted 1 line

can become runnable at once, and hence spawn that many work

items,

Quoted 8 lines
Quoted 8 lines

and

Quoted 7 lines

in turn unbound worker threads.

Several problems there.

It is fundamentally pointless to have potentially that many more threads
than the number of CPU cores - it simply creates a scheduling

storm.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Unbound workers have no CPU / cache locality either and no

connection

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

with the CPU scheduler to optimize scheduling patterns. This may

matter

Quoted 1 line

either on large systems or on small ones. Whereas the current

design

Quoted 1 line
Quoted 1 line
Quoted 1 line

allows for scheduler to notice userspace CPU thread keeps waking

up

Quoted 2 lines
Quoted 2 lines

the

Quoted 1 line

same drm scheduler kernel thread, and so it can keep them on the

same

Quoted 1 line
Quoted 1 line
Quoted 1 line

CPU, the unbound workers lose that ability and so 2nd CPU might

be

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

getting woken up from low sleep for every submission.

Hence, apart from being a bit of a impedance mismatch, the

proposal

Quoted 2 lines
Quoted 2 lines

has

Quoted 1 line

the potential to change performance and power patterns and both

large

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

and small machines.

Ok, thanks for explaining the issue you're seeing in more detail.

Yes,

Quoted 1 line
Quoted 1 line
Quoted 1 line

deferred kwork does appear to mismatch somewhat with what the

scheduler

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

needs or at least how it's worked in the past. How much impact will that mismatch have? Unclear.

> >>> Secondly, it probably demands separate workers (not
optional),
> otherwise
> >>> behaviour of shared workqueues has either the

potential

Quoted 6 lines
Quoted 6 lines

to

Quoted 5 lines

> explode number
> >>> kernel threads anyway, or add latency.
> >>>
> >>
> >> Right now the system_unbound_wq is used which does

have a

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

limit
> on the
> >> number of threads, right? I do have a FIXME to allow a
worker to be
> >> passed in similar to TDR.
> >>
> >> WRT to latency, the 1:1 ratio could actually have

lower

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

latency
> as 2 GPU
> >> schedulers can be pushing jobs into the backend /

cleaning up

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

> jobs in
> >> parallel.
> >>
> >
> > Thought of one more point here where why in Xe we
absolutely want
> a 1 to
> > 1 ratio between entity and scheduler - the way we

implement

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

> timeslicing
> > for preempt fences.
> >
> > Let me try to explain.
> >
> > Preempt fences are implemented via the generic

messaging

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

> interface [1]
> > with suspend / resume messages. If a suspend messages

is

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

received to
> > soon after calling resume (this is per entity) we

simply

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

sleep in the
> > suspend call thus giving the entity a timeslice. This
completely
> falls
> > apart with a many to 1 relationship as now a entity
waiting for a
> > timeslice blocks the other entities. Could we work

aroudn

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

this,
> sure but
> > just another bunch of code we'd have to add in Xe.

Being to

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

> freely sleep
> > in backend without affecting other entities is really,

really

Quoted 2 lines

> nice IMO
> > and I bet Xe isn't the only driver that is going to

feel

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

this way.
> >
> > Last thing I'll say regardless of how anyone feels

about

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Xe using
> a 1 to
> > 1 relationship this patch IMO makes sense as I hope we

can

Quoted 5 lines
Quoted 5 lines

all

Quoted 4 lines

> agree a
> > workqueue scales better than kthreads.
>
> I don't know for sure what will scale better and for what

use

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

case,
> combination of CPU cores vs number of GPU engines to keep
busy vs other
> system activity. But I wager someone is bound to ask for

some

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

> numbers to
> make sure proposal is not negatively affecting any other

drivers.

Quoted 3 lines
     >
     >
     > Then let them ask.  Waving your hands vaguely in the

direction of

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

the
> rest of DRM and saying "Uh, someone (not me) might object" is
profoundly
> unhelpful. Sure, someone might. That's why it's on

dri-devel.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

If you
> think there's someone in particular who might have a useful
opinion on
> this, throw them in the CC so they don't miss the e-mail

thread.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

>
> Or are you asking for numbers? If so, what numbers are you
asking for?

It was a heads up to the Xe team in case people weren't

appreciating

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

how
the proposed change has the potential influence power and

performance

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

across the board. And nothing in the follow up discussion made me

think

Quoted 3 lines

it was considered so I don't think it was redundant to raise it.

In my experience it is typical that such core changes come with

some

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

numbers. Which is in case of drm scheduler is tricky and probably requires explicitly asking everyone to test (rather than count on "don't
miss the email thread"). Real products can fail to ship due ten

mW

Quoted 11 lines
Quoted 11 lines

here

Quoted 1 line

or there. Like suddenly an extra core prevented from getting into

deep

Quoted 8 lines

sleep.

If that was "profoundly unhelpful" so be it.

With your above explanation, it makes more sense what you're asking. It's still not something Matt is likely to be able to provide on his own. We need to tag some other folks and ask them to test it out.

We

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

could play around a bit with it on Xe but it's not exactly production grade yet and is going to hit this differently from most. Likely candidates are probably AMD and Freedreno.

Whoever is setup to check out power and performance would be good to give it a spin, yes.

PS. I don't think I was asking Matt to test with other devices. To

start

Quoted 5 lines
Quoted 5 lines

with I think Xe is a team effort. I was asking for more background on the design decision since patch 4/20 does not say anything on that angle, nor later in the thread it was IMO sufficiently addressed.

Quoted 1 line

> Also, If we're talking about a design that might paint us

into an

Quoted 1 line
Quoted 1 line
Quoted 1 line

> Intel-HW-specific hole, that would be one thing. But we're

not.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

We're
> talking about switching which kernel threading/task mechanism

to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

use for
> what's really a very generic problem. The core Xe design

works

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

without
> this patch (just with more kthreads). If we land this patch

or

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

> something like it and get it wrong and it causes a performance
problem
> for someone down the line, we can revisit it.

For some definition of "it works" - I really wouldn't suggest shipping a
kthread per user context at any point.

You have yet to elaborate on why. What resources is it consuming

that's

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

going to be a problem? Are you anticipating CPU affinity problems? Or does it just seem wasteful?

Well I don't know, commit message says the approach does not scale. :)

Quoted 2 lines

I think I largely agree that it's probably unnecessary/wasteful but reducing the number of kthreads seems like a tractable problem to

solve

Quoted 1 line
Quoted 1 line
Quoted 1 line

regardless of where we put the gpu_scheduler object. Is this the

right

Quoted 1 line
Quoted 1 line
Quoted 1 line

solution? Maybe not. It was also proposed at one point that we

could

Quoted 1 line
Quoted 1 line
Quoted 1 line

split the scheduler into two pieces: A scheduler which owns the

kthread,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

and a back-end which targets some HW ring thing where you can have multiple back-ends per scheduler. That's certainly more invasive

from a

Quoted 1 line
Quoted 1 line
Quoted 1 line

DRM scheduler internal API PoV but would solve the kthread problem

in a

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

way that's more similar to what we have now.

> In any case that's a low level question caused by the high
level design
> decision. So I'd think first focus on the high level -

which

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

is the 1:1
> mapping of entity to scheduler instance proposal.
>
> Fundamentally it will be up to the DRM maintainers and the
community to
> bless your approach. And it is important to stress 1:1 is

about

Quoted 3 lines

> userspace contexts, so I believe unlike any other current
scheduler
> user. And also important to stress this effectively does

not

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

make Xe
> _really_ use the scheduler that much.
>
>
> I don't think this makes Xe nearly as much of a one-off as you
think it
> does. I've already told the Asahi team working on Apple M1/2
hardware
> to do it this way and it seems to be a pretty good mapping for
them. I
> believe this is roughly the plan for nouveau as well. It's

not

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

the way
> it currently works for anyone because most other groups aren't
doing FW
> scheduling yet. In the world of FW scheduling and hardware
designed to
> support userspace direct-to-FW submit, I think the design

makes

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

perfect
> sense (see below) and I expect we'll see more drivers move in

this

Quoted 1 line
Quoted 1 line
Quoted 1 line

> direction as those drivers evolve. (AMD is doing some

customish

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

thing
> for how with gpu_scheduler on the front-end somehow. I've not

dug

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

into
> those details.)
>
> I can only offer my opinion, which is that the two options
mentioned in
> this thread (either improve drm scheduler to cope with

what is

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

> required,
> or split up the code so you can use just the parts of
drm_sched which
> you want - which is frontend dependency tracking)

shouldn't

Quoted 2 lines
Quoted 2 lines

be so

Quoted 1 line

> readily dismissed, given how I think the idea was for the

new

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

driver to
> work less in a silo and more in the community (not do

kludges

Quoted 4 lines
Quoted 4 lines

to

Quoted 3 lines

> workaround stuff because it is thought to be too hard to
improve common
> code), but fundamentally, "goto previous paragraph" for

what

Quoted 5 lines
Quoted 5 lines

I am

Quoted 4 lines
     >     concerned.
     >
     >
     > Meta comment:  It appears as if you're falling into the

standard

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

i915
> team trap of having an internal discussion about what the

community

Quoted 1 line

> discussion might look like instead of actually having the

community

Quoted 1 line

> discussion. If you are seriously concerned about interactions

with

Quoted 1 line

> other drivers or whether or setting common direction, the

right

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

way to
> do that is to break a patch or two out into a separate RFC

series

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

and
> tag a handful of driver maintainers. Trying to predict the
questions
> other people might ask is pointless. Cc them and asking for

their

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

input
> instead.

I don't follow you here. It's not an internal discussion - I am

raising

Quoted 1 line

my concerns on the design publicly. I am supposed to write a

patch to

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

show something, but am allowed to comment on a RFC series?

I may have misread your tone a bit. It felt a bit like too many discussions I've had in the past where people are trying to predict

what

Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

others will say instead of just asking them. Reading it again, I was probably jumping to conclusions a bit. Sorry about that.

Okay no problem, thanks. In any case we don't have to keep discussing it, since I wrote one or two emails ago it is fundamentally on the maintainers and community to ack the approach. I only felt like RFC did not explain the potential downsides sufficiently so I wanted to probe that area a bit.

Quoted 1 line

It is "drm/sched: Convert drm scheduler to use a work queue

rather

Quoted 2 lines
Quoted 2 lines

than

Quoted 1 line

kthread" which should have Cc-ed _everyone_ who use drm

scheduler.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Yeah, it probably should have. I think that's mostly what I've been trying to say.

     >
     >     Regards,
     >
     >     Tvrtko
     >
     >     P.S. And as a related side note, there are more areas

where

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drm_sched
> could be improved, like for instance priority handling.
> Take a look at msm_submitqueue_create /
msm_gpu_convert_priority /
> get_sched_entity to see how msm works around the drm_sched
hardcoded
> limit of available priority levels, in order to avoid

having

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

to leave a
> hw capability unused. I suspect msm would be happier if

they

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

could have
> all priority levels equal in terms of whether they apply

only

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
    at the
     >     frontend level or completely throughout the pipeline.
     >
     >      > [1]
     >

https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1

Quoted 1 line
Quoted 1 line
Quoted 1 line
    <

https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
     >
      <

https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1 <
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>>

Quoted 2 lines

> >
> >>> What would be interesting to learn is whether the

option

Quoted 9 lines
Quoted 9 lines

of

Quoted 8 lines

> refactoring
> >>> drm_sched to deal with out of order completion was
considered
> and what were
> >>> the conclusions.
> >>>
> >>
> >> I coded this up a while back when trying to convert

the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

i915 to
> the DRM
> >> scheduler it isn't all that hard either. The free flow
control
> on the
> >> ring (e.g. set job limit == SIZE OF RING / MAX JOB

SIZE)

Quoted 8 lines
Quoted 8 lines

is

Quoted 7 lines

> really what
> >> sold me on the this design.
>
>
> You're not the only one to suggest supporting out-of-order
completion.
> However, it's tricky and breaks a lot of internal assumptions

of

Quoted 2 lines
Quoted 2 lines

the

Quoted 1 line

> scheduler. It also reduces functionality a bit because it can

no

Quoted 9 lines
Quoted 9 lines
Quoted 7 lines

longer
> automatically rate-limit HW/FW queues which are often
fixed-size. (Ok,
> yes, it probably could but it becomes a substantially harder
problem.)
>
> It also seems like a worse mapping to me. The goal here is to

turn

Quoted 1 line

> submissions on a userspace-facing engine/queue into

submissions

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

to a FW
> queue submissions, sorting out any dma_fence dependencies.

Matt's

Quoted 1 line
Quoted 1 line
Quoted 1 line

> description of saying this is a 1:1 mapping between

sched/entity

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

doesn't
> tell the whole story. It's a 1:1:1 mapping between xe_engine, > gpu_scheduler, and GuC FW engine. Why make it a 1:something:1
mapping?
> Why is that better?

As I have stated before, what I think what would fit well for Xe

is

Quoted 3 lines
Quoted 3 lines

one

Quoted 2 lines

drm_scheduler per engine class. In specific terms on our current hardware, one drm scheduler instance for render, compute,

blitter,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

video
and video enhance. Userspace contexts remain scheduler entities.

And this is where we fairly strongly disagree. More in a bit.

That way you avoid the whole kthread/kworker story and you have

it

Quoted 1 line
Quoted 1 line
Quoted 1 line

actually use the entity picking code in the scheduler, which may

be

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

useful when the backend is congested.

What back-end congestion are you referring to here? Running out of

FW

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

queue IDs? Something else?

CT channel, number of context ids.

Quoted 2 lines

Yes you have to solve the out of order problem so in my mind

that is

Quoted 1 line
Quoted 1 line
Quoted 1 line

something to discuss. What the problem actually is (just TDR?),

how

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

tricky and why etc.

And yes you lose the handy LRCA ring buffer size management so

you'd

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

have to make those entities not runnable in some other way.

Regarding the argument you raise below - would any of that make

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

frontend / backend separation worse and why? Do you think it is

less

Quoted 1 line
Quoted 1 line
Quoted 1 line

natural? If neither is true then all remains is that it appears

extra

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

work to support out of order completion of entities has been

discounted

Quoted 4 lines

in favour of an easy but IMO inelegant option.

Broadly speaking, the kernel needs to stop thinking about GPU

scheduling

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

in terms of scheduling jobs and start thinking in terms of scheduling contexts/engines. There is still some need for scheduling individual jobs but that is only for the purpose of delaying them as needed to resolve dma_fence dependencies. Once dependencies are resolved, they get shoved onto the context/engine queue and from there the kernel

only

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

really manages whole contexts/engines. This is a major architectural shift, entirely different from the way i915 scheduling works. It's

also

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

different from the historical usage of DRM scheduler which I think is why this all looks a bit funny.

To justify this architectural shift, let's look at where we're

headed.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

In the glorious future...

1. Userspace submits directly to firmware queues. The kernel has

no

Quoted 1 line
Quoted 1 line
Quoted 1 line

visibility whatsoever into individual jobs. At most it can

pause/resume

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

FW contexts as needed to handle eviction and memory management.

2. Because of 1, apart from handing out the FW queue IDs at the
beginning, the kernel can't really juggle them that much. Depending

on

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

FW design, it may be able to pause a client, give its IDs to another, and then resume it later when IDs free up. What it's not doing is juggling IDs on a job-by-job basis like i915 currently is.

3. Long-running compute jobs may not complete for days. This means
that memory management needs to happen in terms of pause/resume of entire contexts/engines using the memory rather than based on waiting for individual jobs to complete or pausing individual jobs until the memory is available.

4. Synchronization happens via userspace memory fences (UMF) and

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

kernel is mostly unaware of most dependencies and when a

context/engine

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

is or is not runnable. Instead, it keeps as many of them minimally active (memory is available, even if it's in system RAM) as possible

and

Quoted 1 line
Quoted 1 line
Quoted 1 line

lets the FW sort out dependencies. (There may need to be some

facility

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

for sleeping a context until a memory change similar to futex() or poll() for userspace threads. There are some details TBD.)

Are there potential problems that will need to be solved here?

Yes. Is

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

it a good design? Well, Microsoft has been living in this future for half a decade or better and it's working quite well for them. It's

also

Quoted 1 line
Quoted 1 line
Quoted 1 line

the way all modern game consoles work. It really is just Linux

that's

Quoted 1 line
Quoted 1 line
Quoted 1 line

stuck with the same old job model we've had since the monumental

shift

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

to DRI2.

To that end, one of the core goals of the Xe project was to make the driver internally behave as close to the above model as possible

while

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

keeping the old-school job model as a very thin layer on top. As the broader ecosystem problems (window-system support for UMF, for

instance)

Quoted 1 line
Quoted 1 line
Quoted 1 line

are solved, that layer can be peeled back. The core driver will

already

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

be ready for it.

To that end, the point of the DRM scheduler in Xe isn't to schedule jobs. It's to resolve syncobj and dma-buf implicit sync dependencies and stuff jobs into their respective context/engine queue once

they're

Quoted 1 line
Quoted 1 line
Quoted 1 line

ready. All the actual scheduling happens in firmware and any

scheduling

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the kernel does to deal with contention, oversubscriptions, too many contexts, etc. is between contexts/engines, not individual jobs.

Sure,

Quoted 1 line
Quoted 1 line
Quoted 1 line

the individual job visibility is nice, but if we design around it,

we'll

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

never get to the glorious future.

I really need to turn the above (with a bit more detail) into a blog post.... Maybe I'll do that this week.

In any case, I hope that provides more insight into why Xe is

designed

Quoted 1 line
Quoted 1 line
Quoted 1 line

the way it is and why I'm pushing back so hard on trying to make it

more

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

of a "classic" driver as far as scheduling is concerned. Are there potential problems here? Yes, that's why Xe has been labeled a prototype. Are such radical changes necessary to get to said

glorious

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

future? Yes, I think they are. Will it be worth it? I believe so.

Right, that's all solid I think. My takeaway is that frontend priority sorting and that stuff isn't needed and that is okay. And that there

are

Quoted 1 line
Quoted 1 line

multiple options to maybe improve drm scheduler, like the fore

mentioned

Quoted 6 lines
Quoted 6 lines

making it deal with out of order, or split into functional components, or split frontend/backend what you suggested. For most of them cost vs benefit is more or less not completely clear, neither how much effort was invested to look into them.

One thing I missed from this explanation is how drm_scheduler per

engine

Quoted 9 lines
Quoted 9 lines

class interferes with the high level concepts. And I did not manage to pick up on what exactly is the TDR problem in that case. Maybe the two are one and the same.

Bottom line is I still have the concern that conversion to kworkers has an opportunity to regress. Possibly more opportunity for some Xe use cases than to affect other vendors, since they would still be using per physical engine / queue scheduler instances.

Quoted 8 lines
Quoted 8 lines

And to put my money where my mouth is I will try to put testing Xe inside the full blown ChromeOS environment in my team plans. It would probably also be beneficial if Xe team could take a look at real world behaviour of the extreme transcode use cases too. If the stack is ready for that and all. It would be better to know earlier rather than later if there is a fundamental issue.

For the patch at hand, and the cover letter, it certainly feels it

would

Quoted 8 lines
Quoted 8 lines

benefit to record the past design discussion had with AMD folks, to explicitly copy other drivers, and to record the theoretical pros and cons of threads vs unbound workers as I have tried to highlight them.

Regards,

Tvrtko

#32 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 23 lines · Jason Ekstrand

On Wed, Jan 11, 2023 at 4:32 PM Matthew Brost <[email protected]> wrote:

On Wed, Jan 11, 2023 at 04:18:01PM -0600, Jason Ekstrand wrote:
> On Wed, Jan 11, 2023 at 2:50 AM Tvrtko Ursulin <
> [email protected]> wrote:
>
> >
[snip]
> >
> > Typically is the key here. But I am not sure it is good enough. Consider
> > this example - Intel Flex 170:
> >
> >   * Delivers up to 36 streams 1080p60 transcode throughput per card.
> >   * When scaled to 10 cards in a 4U server configuration, it can support
> > up to 360 streams of HEVC/HEVC 1080p60 transcode throughput. > >
>
> I had a feeling it was going to be media.... 😅
>

Quoted 8 lines

Yea wondering the media UMD can be rewritten to use less xe_engines, it
is massive rewrite for VM bind + no implicit dependencies so let's just
pile on some more work?

It could probably use fewer than it does today.  It currently creates and throws away contexts like crazy, or did last I looked at it.  However, the nature of media encode is that it often spreads across two or three different types of engines.  There's not much you can do to change that.

And as per Tvrtko's example, you get media servers that transcode huge numbers of tiny streams in parallel. Almost no work per frame but 100s of independent streams being run concurrently. That means many 100s of contexts all trying to run at 30fps. I recall a specific bug about thundering herds - hundreds (thousands?) of waiting threads all being woken up at once because some request had completed.

Quoted 64 lines
Quoted 2 lines
Quoted 2 lines

One transcode stream from my experience typically is 3-4 GPU

contexts
> > (buffer travels from vcs -> rcs -> vcs, maybe vecs) used from a single
> > CPU thread. 4 contexts * 36 streams = 144 active contexts. Multiply by
> > 60fps = 8640 jobs submitted and completed per second. > >
> > 144 active contexts in the proposed scheme means possibly means 144
> > kernel worker threads spawned (driven by 36 transcode CPU threads). (I
> > don't think the pools would scale down given all are constantly pinged
> > at 60fps.)
> >
> > And then each of 144 threads goes to grab the single GuC CT mutex. First
> > threads are being made schedulable, then put to sleep as mutex > > contention is hit, then woken again as mutexes are getting released,
> > rinse, repeat.
> >
>
> Why is every submission grabbing the GuC CT mutex? I've not read the GuC
> back-end yet but I was under the impression that most run_job() would be
> just shoving another packet into a ring buffer.  If we have to send the GuC
> a message on the control ring every single time we submit a job, that's
> pretty horrible.
>

Run job writes the ring buffer and moves the tail as the first step (no
lock required). Next it needs to tell the GuC the xe_engine LRC tail has
moved, this is done from a single Host to GuC channel which is circular
buffer, the writing of the channel protected by the mutex. There are little more nuances too but in practice there is always space in the channel so the time mutex needs to held is really, really small (check cached credits, write 3 dwords in payload, write 1 dword to move
tail). I also believe mutexes in Linux are hybrid where they spin for a
little bit before sleeping and certainly if there is space in the channel we shouldn't sleep mutex contention.

Ok, that makes sense.  It's maybe a bit clunky and it'd be nice if we had some way to batch things up a bit so we only have to poke the GuC channel once for every batch of things rather than once per job.  That's maybe something we can look into as a future improvement; not fundamental.

Generally, though, it sounds like contention could be a real problem if we end up ping-ponging that lock between cores.  It's going to depend on how much work it takes to get the next ready thing vs. the cost of that atomic.  But, also, anything we do is going to potentially run into contention problems.  *shrug*  If we were going to go for one-per-HW-engine, we may as well go one-per-device and then we wouldn't need the lock.  Off the top of my head, that doesn't sound great either but IDK.

As far as this being horrible, well didn't design the GuC and this how it is implemented for KMD based submission. We also have 256 doorbells so we wouldn't need a lock but I think are other issues with that design
too which need to be worked out in the Xe2 / Xe3 timeframe.

Quoted 3 lines

Yeah, not blaming you.  Just surprised, that's all.  How does it work for userspace submission?  What would it look like if the kernel emulated userspace submission?  Is that even possible?

What are these doorbell things?  How do they play into it?

Basically a bank of MMIO space reserved per 'entity' where a write to that MMIO space becomes an named interrupt to GuC. You can assign each doorbell to a specific GuC context. So writing to that doorbell address is effectively the same as sending a SCHEDULE_CONTEXT H2G message from the KMD for that context. But the advantage is you ring the doorbell from user land with no call into the kernel at all. Or from within the kernel, you can do it without needing any locks at all. Problem is, we have 64K contexts in GuC but only 256 doorbells in the hardware. Less if using SRIOV. So the "per 'entity'" part because somewhat questionable as to exactly what the 'entity' is. And hence we just haven't bothered supporting them in Linux because a) no direct submission from user land yet, and b) as Matthew says entire chain of IOCTL from UMD to kernel to acquiring a lock and sending the H2G has generally been fast enough. The latency only becomes an issue for ULLS people but for them, even the doorbells from user space are too high a latency because that still potentially involves the GuC having to do some scheduling and context switch type action.

John.

Quoted 797 lines

Also if you see my follow up response Xe is ~33k execs per second with the current implementation on a 8 core (or maybe 8 thread) TGL which seems to fine to me.

33k exec/sec is about 500/frame which should be fine. 500 is a lot for a single frame.  I typically tell game devs to shoot for dozens per frame.  The important thing is that it stays low even with hundreds of memory objects bound. (Xe should be just fine there.)

--Jason

Matt

> --Jason
>
>
> (And yes this backend contention is there regardless of 1:1:1, it would
> > require a different re-design to solve that. But it is just a question
> > whether there are 144 contending threads, or just 6 with the thread per
> > engine class scheme.)
> >
> > Then multiply all by 10 for a 4U server use case and you get 1440 worker
> > kthreads, yes 10 more CT locks, but contending on how many CPU cores?
> > Just so they can grab a timeslice and maybe content on a mutex as the
> > next step.
> >
> > This example is where it would hurt on large systems. Imagine only an
> > even wider media transcode card...
> >
> > Second example is only a single engine class used (3d desktop?) but with
> > a bunch of not-runnable jobs queued and waiting on a fence to signal.
> > Implicit or explicit dependencies doesn't matter. Then the fence signals
> > and call backs run. N work items get scheduled, but they all submit to
> > the same HW engine. So we end up with:
> >
> >          /-- wi1 --\
> >         / ..     .. \
> >   cb --+---  wi.. ---+-- rq1 -- .. -- rqN
> >         \ ..    ..  /
> >          \-- wiN --/
> >
> >
> > All that we have achieved is waking up N CPUs to contend on the same
> > lock and effectively insert the job into the same single HW queue. I
> > don't see any positives there.
> >
> > This example I think can particularly hurt small / low power devices
> > because of needless waking up of many cores for no benefit. Granted, I
> > don't have a good feel on how common this pattern is in practice. > >
> > >
> > >     That
> > >     is the number which drives the maximum number of not-runnable jobs
> > that
> > >     can become runnable at once, and hence spawn that many work items,
> > and
> > >     in turn unbound worker threads.
> > >
> > >     Several problems there.
> > >
> > >     It is fundamentally pointless to have potentially that many more
> > >     threads
> > >     than the number of CPU cores - it simply creates a scheduling storm.
> > >
> > >     Unbound workers have no CPU / cache locality either and no connection
> > >     with the CPU scheduler to optimize scheduling patterns. This may
> > matter
> > >     either on large systems or on small ones. Whereas the current design
> > >     allows for scheduler to notice userspace CPU thread keeps waking up
> > the
> > >     same drm scheduler kernel thread, and so it can keep them on the same
> > >     CPU, the unbound workers lose that ability and so 2nd CPU might be
> > >     getting woken up from low sleep for every submission. > > >
> > >     Hence, apart from being a bit of a impedance mismatch, the proposal
> > has
> > >     the potential to change performance and power patterns and both large
> > >     and small machines.
> > >
> > >
> > > Ok, thanks for explaining the issue you're seeing in more detail.  Yes,
> > > deferred kwork does appear to mismatch somewhat with what the scheduler
> > > needs or at least how it's worked in the past.  How much impact will
> > > that mismatch have?  Unclear.
> > >
> > >      >      >>> Secondly, it probably demands separate workers (not
> > >     optional),
> > >      >     otherwise
> > >      >      >>> behaviour of shared workqueues has either the potential
> > to
> > >      >     explode number
> > >      >      >>> kernel threads anyway, or add latency. > > >      >      >>>
> > >      >      >>
> > >      >      >> Right now the system_unbound_wq is used which does have a
> > >     limit
> > >      >     on the
> > >      >      >> number of threads, right? I do have a FIXME to allow a
> > >     worker to be
> > >      >      >> passed in similar to TDR.
> > >      >      >>
> > >      >      >> WRT to latency, the 1:1 ratio could actually have lower
> > >     latency
> > >      >     as 2 GPU
> > >      >      >> schedulers can be pushing jobs into the backend / > > cleaning up
> > >      >     jobs in
> > >      >      >> parallel.
> > >      >      >>
> > >      >      >
> > >      >      > Thought of one more point here where why in Xe we > > >     absolutely want
> > >      >     a 1 to
> > >      >      > 1 ratio between entity and scheduler - the way we implement
> > >      >     timeslicing
> > >      >      > for preempt fences.
> > >      >      >
> > >      >      > Let me try to explain.
> > >      >      >
> > >      >      > Preempt fences are implemented via the generic messaging
> > >      >     interface [1]
> > >      >      > with suspend / resume messages. If a suspend messages is
> > >     received to
> > >      >      > soon after calling resume (this is per entity) we simply
> > >     sleep in the
> > >      >      > suspend call thus giving the entity a timeslice. This
> > >     completely
> > >      >     falls
> > >      >      > apart with a many to 1 relationship as now a entity
> > >     waiting for a
> > >      >      > timeslice blocks the other entities. Could we work aroudn
> > >     this,
> > >      >     sure but
> > >      >      > just another bunch of code we'd have to add in Xe. Being to
> > >      >     freely sleep
> > >      >      > in backend without affecting other entities is really,
> > really
> > >      >     nice IMO
> > >      >      > and I bet Xe isn't the only driver that is going to feel
> > >     this way.
> > >      >      >
> > >      >      > Last thing I'll say regardless of how anyone feels about
> > >     Xe using
> > >      >     a 1 to
> > >      >      > 1 relationship this patch IMO makes sense as I hope we can
> > all
> > >      >     agree a
> > >      >      > workqueue scales better than kthreads. > > >      >
> > >      >     I don't know for sure what will scale better and for what use
> > >     case,
> > >      >     combination of CPU cores vs number of GPU engines to keep
> > >     busy vs other
> > >      >     system activity. But I wager someone is bound to ask for some
> > >      >     numbers to
> > >      >     make sure proposal is not negatively affecting any other
> > drivers.
> > >      >
> > >      >
> > >      > Then let them ask.  Waving your hands vaguely in the direction of
> > >     the
> > >      > rest of DRM and saying "Uh, someone (not me) might object" is
> > >     profoundly
> > >      > unhelpful.  Sure, someone might. That's why it's on dri-devel.
> > >     If you
> > >      > think there's someone in particular who might have a useful
> > >     opinion on
> > >      > this, throw them in the CC so they don't miss the e-mail thread.
> > >      >
> > >      > Or are you asking for numbers?  If so, what numbers are you
> > >     asking for?
> > >
> > >     It was a heads up to the Xe team in case people weren't appreciating
> > >     how
> > >     the proposed change has the potential influence power and performance
> > >     across the board. And nothing in the follow up discussion made me
> > think
> > >     it was considered so I don't think it was redundant to raise it.
> > >
> > >     In my experience it is typical that such core changes come with some
> > >     numbers. Which is in case of drm scheduler is tricky and probably
> > >     requires explicitly asking everyone to test (rather than count on
> > >     "don't
> > >     miss the email thread"). Real products can fail to ship due ten mW
> > here
> > >     or there. Like suddenly an extra core prevented from getting into
> > deep
> > >     sleep.
> > >
> > >     If that was "profoundly unhelpful" so be it.
> > >
> > >
> > > With your above explanation, it makes more sense what you're asking.
> > > It's still not something Matt is likely to be able to provide on his
> > > own.  We need to tag some other folks and ask them to test it out.  We
> > > could play around a bit with it on Xe but it's not exactly production
> > > grade yet and is going to hit this differently from most. Likely
> > > candidates are probably AMD and Freedreno.
> >
> > Whoever is setup to check out power and performance would be good to
> > give it a spin, yes.
> >
> > PS. I don't think I was asking Matt to test with other devices. To start
> > with I think Xe is a team effort. I was asking for more background on
> > the design decision since patch 4/20 does not say anything on that > > angle, nor later in the thread it was IMO sufficiently addressed. > >
> > >      > Also, If we're talking about a design that might paint us into an
> > >      > Intel-HW-specific hole, that would be one thing.  But we're not.
> > >     We're
> > >      > talking about switching which kernel threading/task mechanism to
> > >     use for
> > >      > what's really a very generic problem.  The core Xe design works
> > >     without
> > >      > this patch (just with more kthreads).  If we land this patch or
> > >      > something like it and get it wrong and it causes a performance
> > >     problem
> > >      > for someone down the line, we can revisit it. > > >
> > >     For some definition of "it works" - I really wouldn't suggest
> > >     shipping a
> > >     kthread per user context at any point.
> > >
> > >
> > > You have yet to elaborate on why. What resources is it consuming that's
> > > going to be a problem? Are you anticipating CPU affinity problems? Or
> > > does it just seem wasteful?
> >
> > Well I don't know, commit message says the approach does not scale. :)
> >
> > > I think I largely agree that it's probably unnecessary/wasteful but
> > > reducing the number of kthreads seems like a tractable problem to solve
> > > regardless of where we put the gpu_scheduler object.  Is this the right
> > > solution?  Maybe not.  It was also proposed at one point that we could
> > > split the scheduler into two pieces: A scheduler which owns the kthread,
> > > and a back-end which targets some HW ring thing where you can have
> > > multiple back-ends per scheduler.  That's certainly more invasive from a
> > > DRM scheduler internal API PoV but would solve the kthread problem in a
> > > way that's more similar to what we have now.
> > >
> > >      >     In any case that's a low level question caused by the high
> > >     level design
> > >      >     decision. So I'd think first focus on the high level - which
> > >     is the 1:1
> > >      >     mapping of entity to scheduler instance proposal. > > >      >
> > >      >     Fundamentally it will be up to the DRM maintainers and the
> > >     community to
> > >      >     bless your approach. And it is important to stress 1:1 is
> > about
> > >      >     userspace contexts, so I believe unlike any other current
> > >     scheduler
> > >      >     user. And also important to stress this effectively does not
> > >     make Xe
> > >      >     _really_ use the scheduler that much.
> > >      >
> > >      >
> > >      > I don't think this makes Xe nearly as much of a one-off as you
> > >     think it
> > >      > does.  I've already told the Asahi team working on Apple M1/2
> > >     hardware
> > >      > to do it this way and it seems to be a pretty good mapping for
> > >     them. I
> > >      > believe this is roughly the plan for nouveau as well.  It's not
> > >     the way
> > >      > it currently works for anyone because most other groups aren't
> > >     doing FW
> > >      > scheduling yet.  In the world of FW scheduling and hardware
> > >     designed to
> > >      > support userspace direct-to-FW submit, I think the design makes
> > >     perfect
> > >      > sense (see below) and I expect we'll see more drivers move in this
> > >      > direction as those drivers evolve. (AMD is doing some customish
> > >     thing
> > >      > for how with gpu_scheduler on the front-end somehow. I've not dug
> > >     into
> > >      > those details.)
> > >      >
> > >      >     I can only offer my opinion, which is that the two options
> > >     mentioned in
> > >      >     this thread (either improve drm scheduler to cope with what is
> > >      >     required,
> > >      >     or split up the code so you can use just the parts of > > >     drm_sched which
> > >      >     you want - which is frontend dependency tracking) shouldn't
> > be so
> > >      >     readily dismissed, given how I think the idea was for the new
> > >     driver to
> > >      >     work less in a silo and more in the community (not do kludges
> > to
> > >      >     workaround stuff because it is thought to be too hard to
> > >     improve common
> > >      >     code), but fundamentally, "goto previous paragraph" for what
> > I am
> > >      >     concerned.
> > >      >
> > >      >
> > >      > Meta comment:  It appears as if you're falling into the standard
> > >     i915
> > >      > team trap of having an internal discussion about what the > > community
> > >      > discussion might look like instead of actually having the > > community
> > >      > discussion.  If you are seriously concerned about interactions
> > with
> > >      > other drivers or whether or setting common direction, the right
> > >     way to
> > >      > do that is to break a patch or two out into a separate RFC series
> > >     and
> > >      > tag a handful of driver maintainers.  Trying to predict the
> > >     questions
> > >      > other people might ask is pointless. Cc them and asking for their
> > >     input
> > >      > instead.
> > >
> > >     I don't follow you here. It's not an internal discussion
- I am
> > raising
> > >     my concerns on the design publicly. I am supposed to write a patch to
> > >     show something, but am allowed to comment on a RFC series? > > >
> > >
> > > I may have misread your tone a bit.  It felt a bit like too many > > > discussions I've had in the past where people are trying to predict what
> > > others will say instead of just asking them. Reading it again, I was
> > > probably jumping to conclusions a bit.  Sorry about that. > >
> > Okay no problem, thanks. In any case we don't have to keep discussing
> > it, since I wrote one or two emails ago it is fundamentally on the > > maintainers and community to ack the approach. I only felt like RFC did
> > not explain the potential downsides sufficiently so I wanted to probe
> > that area a bit.
> >
> > >     It is "drm/sched: Convert drm scheduler to use a work queue rather
> > than
> > >     kthread" which should have Cc-ed _everyone_ who use drm scheduler.
> > >
> > >
> > > Yeah, it probably should have.  I think that's mostly what I've been
> > > trying to say.
> > >
> > >      >
> > >      >     Regards,
> > >      >
> > >      >     Tvrtko
> > >      >
> > >      >     P.S. And as a related side note, there are more areas where
> > >     drm_sched
> > >      >     could be improved, like for instance priority handling.
> > >      >     Take a look at msm_submitqueue_create /
> > >     msm_gpu_convert_priority /
> > >      >     get_sched_entity to see how msm works around the drm_sched
> > >     hardcoded
> > >      >     limit of available priority levels, in order to avoid having
> > >     to leave a
> > >      >     hw capability unused. I suspect msm would be happier if they
> > >     could have
> > >      >     all priority levels equal in terms of whether they apply only
> > >     at the
> > >      >     frontend level or completely throughout the pipeline. > > >      >
> > >      >      > [1]
> > >      >
> > >
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
> > >
 <https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
> > >
> > >      >
> > >       <
> >
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
<
> >
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>>>
> > >      >      >
> > >      >      >>> What would be interesting to learn is whether the option
> > of
> > >      >     refactoring
> > >      >      >>> drm_sched to deal with out of order completion was
> > >     considered
> > >      >     and what were
> > >      >      >>> the conclusions.
> > >      >      >>>
> > >      >      >>
> > >      >      >> I coded this up a while back when trying to convert the
> > >     i915 to
> > >      >     the DRM
> > >      >      >> scheduler it isn't all that hard either. The free flow
> > >     control
> > >      >     on the
> > >      >      >> ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE)
> > is
> > >      >     really what
> > >      >      >> sold me on the this design.
> > >      >
> > >      >
> > >      > You're not the only one to suggest supporting out-of-order
> > >     completion.
> > >      > However, it's tricky and breaks a lot of internal assumptions of
> > the
> > >      > scheduler. It also reduces functionality a bit because it can no
> > >     longer
> > >      > automatically rate-limit HW/FW queues which are often > > >     fixed-size.  (Ok,
> > >      > yes, it probably could but it becomes a substantially harder
> > >     problem.)
> > >      >
> > >      > It also seems like a worse mapping to me.  The goal here is to
> > turn
> > >      > submissions on a userspace-facing engine/queue into submissions
> > >     to a FW
> > >      > queue submissions, sorting out any dma_fence dependencies.  Matt's
> > >      > description of saying this is a 1:1 mapping between sched/entity
> > >     doesn't
> > >      > tell the whole story. It's a 1:1:1 mapping between xe_engine,
> > >      > gpu_scheduler, and GuC FW engine. Why make it a 1:something:1
> > >     mapping?
> > >      > Why is that better?
> > >
> > >     As I have stated before, what I think what would fit well for Xe is
> > one
> > >     drm_scheduler per engine class. In specific terms on our current
> > >     hardware, one drm scheduler instance for render, compute, blitter,
> > >     video
> > >     and video enhance. Userspace contexts remain scheduler entities.
> > >
> > >
> > > And this is where we fairly strongly disagree.  More in a bit. > > >
> > >     That way you avoid the whole kthread/kworker story and you have it
> > >     actually use the entity picking code in the scheduler, which may be
> > >     useful when the backend is congested.
> > >
> > >
> > > What back-end congestion are you referring to here?  Running out of FW
> > > queue IDs?  Something else?
> >
> > CT channel, number of context ids.
> >
> > >
> > >     Yes you have to solve the out of order problem so in my mind that is
> > >     something to discuss. What the problem actually is (just TDR?), how
> > >     tricky and why etc.
> > >
> > >     And yes you lose the handy LRCA ring buffer size management so you'd
> > >     have to make those entities not runnable in some other way. > > >
> > >     Regarding the argument you raise below - would any of that make the
> > >     frontend / backend separation worse and why? Do you think it is less
> > >     natural? If neither is true then all remains is that it appears extra
> > >     work to support out of order completion of entities has been > > discounted
> > >     in favour of an easy but IMO inelegant option.
> > >
> > >
> > > Broadly speaking, the kernel needs to stop thinking about GPU scheduling
> > > in terms of scheduling jobs and start thinking in terms of scheduling
> > > contexts/engines.  There is still some need for scheduling individual
> > > jobs but that is only for the purpose of delaying them as needed to
> > > resolve dma_fence dependencies.  Once dependencies are resolved, they
> > > get shoved onto the context/engine queue and from there the kernel only
> > > really manages whole contexts/engines.  This is a major architectural
> > > shift, entirely different from the way i915 scheduling works.  It's also
> > > different from the historical usage of DRM scheduler which I think is
> > > why this all looks a bit funny.
> > >
> > > To justify this architectural shift, let's look at where we're headed.
> > > In the glorious future...
> > >
> > >   1. Userspace submits directly to firmware queues.  The kernel has no
> > > visibility whatsoever into individual jobs. At most it can pause/resume
> > > FW contexts as needed to handle eviction and memory management. > > >
> > >   2. Because of 1, apart from handing out the FW queue IDs at the
> > > beginning, the kernel can't really juggle them that much. Depending on
> > > FW design, it may be able to pause a client, give its IDs to another,
> > > and then resume it later when IDs free up. What it's not doing is
> > > juggling IDs on a job-by-job basis like i915 currently is. > > >
> > >   3. Long-running compute jobs may not complete for days. This means
> > > that memory management needs to happen in terms of pause/resume of
> > > entire contexts/engines using the memory rather than based on waiting
> > > for individual jobs to complete or pausing individual jobs until the
> > > memory is available.
> > >
> > >   4. Synchronization happens via userspace memory fences (UMF) and the
> > > kernel is mostly unaware of most dependencies and when a context/engine
> > > is or is not runnable.  Instead, it keeps as many of them minimally
> > > active (memory is available, even if it's in system RAM) as possible and
> > > lets the FW sort out dependencies.  (There may need to be some facility
> > > for sleeping a context until a memory change similar to futex() or
> > > poll() for userspace threads.  There are some details TBD.) > > >
> > > Are there potential problems that will need to be solved here?  Yes.  Is
> > > it a good design?  Well, Microsoft has been living in this future for
> > > half a decade or better and it's working quite well for them.  It's also
> > > the way all modern game consoles work.  It really is just Linux that's
> > > stuck with the same old job model we've had since the monumental shift
> > > to DRI2.
> > >
> > > To that end, one of the core goals of the Xe project was to make the
> > > driver internally behave as close to the above model as possible while
> > > keeping the old-school job model as a very thin layer on top.  As the
> > > broader ecosystem problems (window-system support for UMF, for instance)
> > > are solved, that layer can be peeled back. The core driver will already
> > > be ready for it.
> > >
> > > To that end, the point of the DRM scheduler in Xe isn't to schedule
> > > jobs.  It's to resolve syncobj and dma-buf implicit sync dependencies
> > > and stuff jobs into their respective context/engine queue once they're
> > > ready.  All the actual scheduling happens in firmware and any scheduling
> > > the kernel does to deal with contention, oversubscriptions, too many
> > > contexts, etc. is between contexts/engines, not individual jobs.  Sure,
> > > the individual job visibility is nice, but if we design around it, we'll
> > > never get to the glorious future.
> > >
> > > I really need to turn the above (with a bit more detail) into a blog
> > > post.... Maybe I'll do that this week.
> > >
> > > In any case, I hope that provides more insight into why Xe is designed
> > > the way it is and why I'm pushing back so hard on trying to make it more
> > > of a "classic" driver as far as scheduling is concerned. Are there
> > > potential problems here?  Yes, that's why Xe has been labeled a > > > prototype.  Are such radical changes necessary to get to said glorious
> > > future?  Yes, I think they are.  Will it be worth it?  I believe so.
> >
> > Right, that's all solid I think. My takeaway is that frontend priority
> > sorting and that stuff isn't needed and that is okay. And that there are
> > multiple options to maybe improve drm scheduler, like the fore mentioned
> > making it deal with out of order, or split into functional components,
> > or split frontend/backend what you suggested. For most of them cost vs
> > benefit is more or less not completely clear, neither how much effort
> > was invested to look into them.
> >
> > One thing I missed from this explanation is how drm_scheduler per engine
> > class interferes with the high level concepts. And I did not manage to
> > pick up on what exactly is the TDR problem in that case. Maybe the two
> > are one and the same.
> >
> > Bottom line is I still have the concern that conversion to kworkers has
> > an opportunity to regress. Possibly more opportunity for some Xe use
> > cases than to affect other vendors, since they would still be using per
> > physical engine / queue scheduler instances.
> >
> > And to put my money where my mouth is I will try to put testing Xe > > inside the full blown ChromeOS environment in my team plans. It would
> > probably also be beneficial if Xe team could take a look at real world
> > behaviour of the extreme transcode use cases too. If the stack is ready
> > for that and all. It would be better to know earlier rather than later
> > if there is a fundamental issue.
> >
> > For the patch at hand, and the cover letter, it certainly feels it would
> > benefit to record the past design discussion had with AMD folks, to
> > explicitly copy other drivers, and to record the theoretical pros and
> > cons of threads vs unbound workers as I have tried to highlight them.
> >
> > Regards,
> >
> > Tvrtko
> >

author maintainer
#33 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 141 lines · John Harrison
Quoted 31 lines · Jason Ekstrand

On Wed, Jan 11, 2023 at 4:32 PM Matthew Brost <[email protected]> wrote:

On Wed, Jan 11, 2023 at 04:18:01PM -0600, Jason Ekstrand wrote:
> On Wed, Jan 11, 2023 at 2:50 AM Tvrtko Ursulin <
> [email protected]> wrote:
>
> >
[snip]
> >
> > Typically is the key here. But I am not sure it is good enough. Consider
> > this example - Intel Flex 170:
> >
> >   * Delivers up to 36 streams 1080p60 transcode throughput per card.
> >   * When scaled to 10 cards in a 4U server configuration, it can support
> > up to 360 streams of HEVC/HEVC 1080p60 transcode throughput. > >
>
> I had a feeling it was going to be media.... 😅
>

Yea wondering the media UMD can be rewritten to use less xe_engines, it
is massive rewrite for VM bind + no implicit dependencies so let's just
pile on some more work?

Quoted 5 lines

It could probably use fewer than it does today.  It currently creates and throws away contexts like crazy, or did last I looked at it. However, the nature of media encode is that it often spreads across two or three different types of engines.  There's not much you can do to change that.

And as per Tvrtko's example, you get media servers that transcode huge numbers of tiny streams in parallel. Almost no work per frame but 100s of independent streams being run concurrently. That means many 100s of contexts all trying to run at 30fps. I recall a specific bug about thundering herds - hundreds (thousands?) of waiting threads all being woken up at once because some request had completed.

Quoted 75 lines
Quoted 2 lines
Quoted 2 lines

One transcode stream from my experience typically is 3-4 GPU

contexts
> > (buffer travels from vcs -> rcs -> vcs, maybe vecs) used from a single
> > CPU thread. 4 contexts * 36 streams = 144 active contexts. Multiply by
> > 60fps = 8640 jobs submitted and completed per second. > >
> > 144 active contexts in the proposed scheme means possibly means 144
> > kernel worker threads spawned (driven by 36 transcode CPU threads). (I
> > don't think the pools would scale down given all are constantly pinged
> > at 60fps.)
> >
> > And then each of 144 threads goes to grab the single GuC CT mutex. First
> > threads are being made schedulable, then put to sleep as mutex > > contention is hit, then woken again as mutexes are getting released,
> > rinse, repeat.
> >
>
> Why is every submission grabbing the GuC CT mutex? I've not read the GuC
> back-end yet but I was under the impression that most run_job() would be
> just shoving another packet into a ring buffer.  If we have to send the GuC
> a message on the control ring every single time we submit a job, that's
> pretty horrible.
>

Run job writes the ring buffer and moves the tail as the first step (no
lock required). Next it needs to tell the GuC the xe_engine LRC tail has
moved, this is done from a single Host to GuC channel which is circular
buffer, the writing of the channel protected by the mutex. There are little more nuances too but in practice there is always space in the channel so the time mutex needs to held is really, really small (check cached credits, write 3 dwords in payload, write 1 dword to move
tail). I also believe mutexes in Linux are hybrid where they spin for a
little bit before sleeping and certainly if there is space in the channel we shouldn't sleep mutex contention.

Ok, that makes sense.  It's maybe a bit clunky and it'd be nice if we had some way to batch things up a bit so we only have to poke the GuC channel once for every batch of things rather than once per job.  That's maybe something we can look into as a future improvement; not fundamental.

Generally, though, it sounds like contention could be a real problem if we end up ping-ponging that lock between cores.  It's going to depend on how much work it takes to get the next ready thing vs. the cost of that atomic.  But, also, anything we do is going to potentially run into contention problems.  *shrug*  If we were going to go for one-per-HW-engine, we may as well go one-per-device and then we wouldn't need the lock.  Off the top of my head, that doesn't sound great either but IDK.

As far as this being horrible, well didn't design the GuC and this how it is implemented for KMD based submission. We also have 256 doorbells so we wouldn't need a lock but I think are other issues with that design
too which need to be worked out in the Xe2 / Xe3 timeframe.

Quoted 5 lines

Yeah, not blaming you.  Just surprised, that's all.  How does it work for userspace submission?  What would it look like if the kernel emulated userspace submission?  Is that even possible?

What are these doorbell things?  How do they play into it?

Basically a bank of MMIO space reserved per 'entity' where a write to that MMIO space becomes an named interrupt to GuC. You can assign each doorbell to a specific GuC context. So writing to that doorbell address is effectively the same as sending a SCHEDULE_CONTEXT H2G message from the KMD for that context. But the advantage is you ring the doorbell from user land with no call into the kernel at all. Or from within the kernel, you can do it without needing any locks at all. Problem is, we have 64K contexts in GuC but only 256 doorbells in the hardware. Less if using SRIOV. So the "per 'entity'" part because somewhat questionable as to exactly what the 'entity' is. And hence we just haven't bothered supporting them in Linux because a) no direct submission from user land yet, and b) as Matthew says entire chain of IOCTL from UMD to kernel to acquiring a lock and sending the H2G has generally been fast enough. The latency only becomes an issue for ULLS people but for them, even the doorbells from user space are too high a latency because that still potentially involves the GuC having to do some scheduling and context switch type action.

Quoted 2 lines

John.

I talked with Jason on IRC last week about doorbells and we came up with the idea after chatting to allocate the doorbells with a greedy algorithm which results in the first 256 xe_engine each getting their own doorbell thus avoid contention on the CT channel / lock (this is still KMD submission).

Coded up a prototype for this and initial test results of xe_exec_threads /w 245 user xe_engines, 5 threads, and 40k total execs are an average of .824s vs. 923s for /w and w/o doorbells. Or in other words 49714 execs per seconds /w doorbells vs. 44353 without. This seems to indicate using doorbells can provide a performance improvement. Also Jason and I reasoned we should be able to use doorbells 99% of the time aside from maybe some wacky media use cases. I also plan on following up with the media UMD to see if we can get them to use less xe_engines.

Matt

Quoted 801 lines
Quoted 801 lines

Also if you see my follow up response Xe is ~33k execs per second with the current implementation on a 8 core (or maybe 8 thread) TGL which seems to fine to me.

33k exec/sec is about 500/frame which should be fine. 500 is a lot for a single frame.  I typically tell game devs to shoot for dozens per frame.  The important thing is that it stays low even with hundreds of memory objects bound. (Xe should be just fine there.)

--Jason

Matt

> --Jason
>
>
> (And yes this backend contention is there regardless of 1:1:1, it would
> > require a different re-design to solve that. But it is just a question
> > whether there are 144 contending threads, or just 6 with the thread per
> > engine class scheme.)
> >
> > Then multiply all by 10 for a 4U server use case and you get 1440 worker
> > kthreads, yes 10 more CT locks, but contending on how many CPU cores?
> > Just so they can grab a timeslice and maybe content on a mutex as the
> > next step.
> >
> > This example is where it would hurt on large systems. Imagine only an
> > even wider media transcode card...
> >
> > Second example is only a single engine class used (3d desktop?) but with
> > a bunch of not-runnable jobs queued and waiting on a fence to signal.
> > Implicit or explicit dependencies doesn't matter. Then the fence signals
> > and call backs run. N work items get scheduled, but they all submit to
> > the same HW engine. So we end up with:
> >
> >          /-- wi1 --\
> >         / ..     .. \
> >   cb --+---  wi.. ---+-- rq1 -- .. -- rqN
> >         \ ..    ..  /
> >          \-- wiN --/
> >
> >
> > All that we have achieved is waking up N CPUs to contend on the same
> > lock and effectively insert the job into the same single HW queue. I
> > don't see any positives there.
> >
> > This example I think can particularly hurt small / low power devices
> > because of needless waking up of many cores for no benefit. Granted, I
> > don't have a good feel on how common this pattern is in practice. > >
> > >
> > >     That
> > >     is the number which drives the maximum number of not-runnable jobs
> > that
> > >     can become runnable at once, and hence spawn that many work items,
> > and
> > >     in turn unbound worker threads.
> > >
> > >     Several problems there.
> > >
> > >     It is fundamentally pointless to have potentially that many more
> > >     threads
> > >     than the number of CPU cores - it simply creates a scheduling storm.
> > >
> > >     Unbound workers have no CPU / cache locality either and no connection
> > >     with the CPU scheduler to optimize scheduling patterns. This may
> > matter
> > >     either on large systems or on small ones. Whereas the current design
> > >     allows for scheduler to notice userspace CPU thread keeps waking up
> > the
> > >     same drm scheduler kernel thread, and so it can keep them on the same
> > >     CPU, the unbound workers lose that ability and so 2nd CPU might be
> > >     getting woken up from low sleep for every submission. > > >
> > >     Hence, apart from being a bit of a impedance mismatch, the proposal
> > has
> > >     the potential to change performance and power patterns and both large
> > >     and small machines.
> > >
> > >
> > > Ok, thanks for explaining the issue you're seeing in more detail.  Yes,
> > > deferred kwork does appear to mismatch somewhat with what the scheduler
> > > needs or at least how it's worked in the past.  How much impact will
> > > that mismatch have?  Unclear.
> > >
> > >      >      >>> Secondly, it probably demands separate workers (not
> > >     optional),
> > >      >     otherwise
> > >      >      >>> behaviour of shared workqueues has either the potential
> > to
> > >      >     explode number
> > >      >      >>> kernel threads anyway, or add latency. > > >      >      >>>
> > >      >      >>
> > >      >      >> Right now the system_unbound_wq is used which does have a
> > >     limit
> > >      >     on the
> > >      >      >> number of threads, right? I do have a FIXME to allow a
> > >     worker to be
> > >      >      >> passed in similar to TDR.
> > >      >      >>
> > >      >      >> WRT to latency, the 1:1 ratio could actually have lower
> > >     latency
> > >      >     as 2 GPU
> > >      >      >> schedulers can be pushing jobs into the backend / > > cleaning up
> > >      >     jobs in
> > >      >      >> parallel.
> > >      >      >>
> > >      >      >
> > >      >      > Thought of one more point here where why in Xe we > > >     absolutely want
> > >      >     a 1 to
> > >      >      > 1 ratio between entity and scheduler - the way we implement
> > >      >     timeslicing
> > >      >      > for preempt fences.
> > >      >      >
> > >      >      > Let me try to explain.
> > >      >      >
> > >      >      > Preempt fences are implemented via the generic messaging
> > >      >     interface [1]
> > >      >      > with suspend / resume messages. If a suspend messages is
> > >     received to
> > >      >      > soon after calling resume (this is per entity) we simply
> > >     sleep in the
> > >      >      > suspend call thus giving the entity a timeslice. This
> > >     completely
> > >      >     falls
> > >      >      > apart with a many to 1 relationship as now a entity
> > >     waiting for a
> > >      >      > timeslice blocks the other entities. Could we work aroudn
> > >     this,
> > >      >     sure but
> > >      >      > just another bunch of code we'd have to add in Xe. Being to
> > >      >     freely sleep
> > >      >      > in backend without affecting other entities is really,
> > really
> > >      >     nice IMO
> > >      >      > and I bet Xe isn't the only driver that is going to feel
> > >     this way.
> > >      >      >
> > >      >      > Last thing I'll say regardless of how anyone feels about
> > >     Xe using
> > >      >     a 1 to
> > >      >      > 1 relationship this patch IMO makes sense as I hope we can
> > all
> > >      >     agree a
> > >      >      > workqueue scales better than kthreads. > > >      >
> > >      >     I don't know for sure what will scale better and for what use
> > >     case,
> > >      >     combination of CPU cores vs number of GPU engines to keep
> > >     busy vs other
> > >      >     system activity. But I wager someone is bound to ask for some
> > >      >     numbers to
> > >      >     make sure proposal is not negatively affecting any other
> > drivers.
> > >      >
> > >      >
> > >      > Then let them ask.  Waving your hands vaguely in the direction of
> > >     the
> > >      > rest of DRM and saying "Uh, someone (not me) might object" is
> > >     profoundly
> > >      > unhelpful.  Sure, someone might. That's why it's on dri-devel.
> > >     If you
> > >      > think there's someone in particular who might have a useful
> > >     opinion on
> > >      > this, throw them in the CC so they don't miss the e-mail thread.
> > >      >
> > >      > Or are you asking for numbers?  If so, what numbers are you
> > >     asking for?
> > >
> > >     It was a heads up to the Xe team in case people weren't appreciating
> > >     how
> > >     the proposed change has the potential influence power and performance
> > >     across the board. And nothing in the follow up discussion made me
> > think
> > >     it was considered so I don't think it was redundant to raise it.
> > >
> > >     In my experience it is typical that such core changes come with some
> > >     numbers. Which is in case of drm scheduler is tricky and probably
> > >     requires explicitly asking everyone to test (rather than count on
> > >     "don't
> > >     miss the email thread"). Real products can fail to ship due ten mW
> > here
> > >     or there. Like suddenly an extra core prevented from getting into
> > deep
> > >     sleep.
> > >
> > >     If that was "profoundly unhelpful" so be it.
> > >
> > >
> > > With your above explanation, it makes more sense what you're asking.
> > > It's still not something Matt is likely to be able to provide on his
> > > own.  We need to tag some other folks and ask them to test it out.  We
> > > could play around a bit with it on Xe but it's not exactly production
> > > grade yet and is going to hit this differently from most. Likely
> > > candidates are probably AMD and Freedreno.
> >
> > Whoever is setup to check out power and performance would be good to
> > give it a spin, yes.
> >
> > PS. I don't think I was asking Matt to test with other devices. To start
> > with I think Xe is a team effort. I was asking for more background on
> > the design decision since patch 4/20 does not say anything on that > > angle, nor later in the thread it was IMO sufficiently addressed. > >
> > >      > Also, If we're talking about a design that might paint us into an
> > >      > Intel-HW-specific hole, that would be one thing.  But we're not.
> > >     We're
> > >      > talking about switching which kernel threading/task mechanism to
> > >     use for
> > >      > what's really a very generic problem.  The core Xe design works
> > >     without
> > >      > this patch (just with more kthreads).  If we land this patch or
> > >      > something like it and get it wrong and it causes a performance
> > >     problem
> > >      > for someone down the line, we can revisit it. > > >
> > >     For some definition of "it works" - I really wouldn't suggest
> > >     shipping a
> > >     kthread per user context at any point.
> > >
> > >
> > > You have yet to elaborate on why. What resources is it consuming that's
> > > going to be a problem? Are you anticipating CPU affinity problems? Or
> > > does it just seem wasteful?
> >
> > Well I don't know, commit message says the approach does not scale. :)
> >
> > > I think I largely agree that it's probably unnecessary/wasteful but
> > > reducing the number of kthreads seems like a tractable problem to solve
> > > regardless of where we put the gpu_scheduler object.  Is this the right
> > > solution?  Maybe not.  It was also proposed at one point that we could
> > > split the scheduler into two pieces: A scheduler which owns the kthread,
> > > and a back-end which targets some HW ring thing where you can have
> > > multiple back-ends per scheduler.  That's certainly more invasive from a
> > > DRM scheduler internal API PoV but would solve the kthread problem in a
> > > way that's more similar to what we have now.
> > >
> > >      >     In any case that's a low level question caused by the high
> > >     level design
> > >      >     decision. So I'd think first focus on the high level - which
> > >     is the 1:1
> > >      >     mapping of entity to scheduler instance proposal. > > >      >
> > >      >     Fundamentally it will be up to the DRM maintainers and the
> > >     community to
> > >      >     bless your approach. And it is important to stress 1:1 is
> > about
> > >      >     userspace contexts, so I believe unlike any other current
> > >     scheduler
> > >      >     user. And also important to stress this effectively does not
> > >     make Xe
> > >      >     _really_ use the scheduler that much.
> > >      >
> > >      >
> > >      > I don't think this makes Xe nearly as much of a one-off as you
> > >     think it
> > >      > does.  I've already told the Asahi team working on Apple M1/2
> > >     hardware
> > >      > to do it this way and it seems to be a pretty good mapping for
> > >     them. I
> > >      > believe this is roughly the plan for nouveau as well.  It's not
> > >     the way
> > >      > it currently works for anyone because most other groups aren't
> > >     doing FW
> > >      > scheduling yet.  In the world of FW scheduling and hardware
> > >     designed to
> > >      > support userspace direct-to-FW submit, I think the design makes
> > >     perfect
> > >      > sense (see below) and I expect we'll see more drivers move in this
> > >      > direction as those drivers evolve. (AMD is doing some customish
> > >     thing
> > >      > for how with gpu_scheduler on the front-end somehow. I've not dug
> > >     into
> > >      > those details.)
> > >      >
> > >      >     I can only offer my opinion, which is that the two options
> > >     mentioned in
> > >      >     this thread (either improve drm scheduler to cope with what is
> > >      >     required,
> > >      >     or split up the code so you can use just the parts of > > >     drm_sched which
> > >      >     you want - which is frontend dependency tracking) shouldn't
> > be so
> > >      >     readily dismissed, given how I think the idea was for the new
> > >     driver to
> > >      >     work less in a silo and more in the community (not do kludges
> > to
> > >      >     workaround stuff because it is thought to be too hard to
> > >     improve common
> > >      >     code), but fundamentally, "goto previous paragraph" for what
> > I am
> > >      >     concerned.
> > >      >
> > >      >
> > >      > Meta comment:  It appears as if you're falling into the standard
> > >     i915
> > >      > team trap of having an internal discussion about what the > > community
> > >      > discussion might look like instead of actually having the > > community
> > >      > discussion.  If you are seriously concerned about interactions
> > with
> > >      > other drivers or whether or setting common direction, the right
> > >     way to
> > >      > do that is to break a patch or two out into a separate RFC series
> > >     and
> > >      > tag a handful of driver maintainers.  Trying to predict the
> > >     questions
> > >      > other people might ask is pointless. Cc them and asking for their
> > >     input
> > >      > instead.
> > >
> > >     I don't follow you here. It's not an internal discussion
- I am
> > raising
> > >     my concerns on the design publicly. I am supposed to write a patch to
> > >     show something, but am allowed to comment on a RFC series? > > >
> > >
> > > I may have misread your tone a bit.  It felt a bit like too many > > > discussions I've had in the past where people are trying to predict what
> > > others will say instead of just asking them. Reading it again, I was
> > > probably jumping to conclusions a bit.  Sorry about that. > >
> > Okay no problem, thanks. In any case we don't have to keep discussing
> > it, since I wrote one or two emails ago it is fundamentally on the > > maintainers and community to ack the approach. I only felt like RFC did
> > not explain the potential downsides sufficiently so I wanted to probe
> > that area a bit.
> >
> > >     It is "drm/sched: Convert drm scheduler to use a work queue rather
> > than
> > >     kthread" which should have Cc-ed _everyone_ who use drm scheduler.
> > >
> > >
> > > Yeah, it probably should have.  I think that's mostly what I've been
> > > trying to say.
> > >
> > >      >
> > >      >     Regards,
> > >      >
> > >      >     Tvrtko
> > >      >
> > >      >     P.S. And as a related side note, there are more areas where
> > >     drm_sched
> > >      >     could be improved, like for instance priority handling.
> > >      >     Take a look at msm_submitqueue_create /
> > >     msm_gpu_convert_priority /
> > >      >     get_sched_entity to see how msm works around the drm_sched
> > >     hardcoded
> > >      >     limit of available priority levels, in order to avoid having
> > >     to leave a
> > >      >     hw capability unused. I suspect msm would be happier if they
> > >     could have
> > >      >     all priority levels equal in terms of whether they apply only
> > >     at the
> > >      >     frontend level or completely throughout the pipeline. > > >      >
> > >      >      > [1]
> > >      >
> > >
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
> > >
 <https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
> > >
> > >      >
> > >       <
> >
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
<
> >
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>>>
> > >      >      >
> > >      >      >>> What would be interesting to learn is whether the option
> > of
> > >      >     refactoring
> > >      >      >>> drm_sched to deal with out of order completion was
> > >     considered
> > >      >     and what were
> > >      >      >>> the conclusions.
> > >      >      >>>
> > >      >      >>
> > >      >      >> I coded this up a while back when trying to convert the
> > >     i915 to
> > >      >     the DRM
> > >      >      >> scheduler it isn't all that hard either. The free flow
> > >     control
> > >      >     on the
> > >      >      >> ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE)
> > is
> > >      >     really what
> > >      >      >> sold me on the this design.
> > >      >
> > >      >
> > >      > You're not the only one to suggest supporting out-of-order
> > >     completion.
> > >      > However, it's tricky and breaks a lot of internal assumptions of
> > the
> > >      > scheduler. It also reduces functionality a bit because it can no
> > >     longer
> > >      > automatically rate-limit HW/FW queues which are often > > >     fixed-size.  (Ok,
> > >      > yes, it probably could but it becomes a substantially harder
> > >     problem.)
> > >      >
> > >      > It also seems like a worse mapping to me.  The goal here is to
> > turn
> > >      > submissions on a userspace-facing engine/queue into submissions
> > >     to a FW
> > >      > queue submissions, sorting out any dma_fence dependencies.  Matt's
> > >      > description of saying this is a 1:1 mapping between sched/entity
> > >     doesn't
> > >      > tell the whole story. It's a 1:1:1 mapping between xe_engine,
> > >      > gpu_scheduler, and GuC FW engine. Why make it a 1:something:1
> > >     mapping?
> > >      > Why is that better?
> > >
> > >     As I have stated before, what I think what would fit well for Xe is
> > one
> > >     drm_scheduler per engine class. In specific terms on our current
> > >     hardware, one drm scheduler instance for render, compute, blitter,
> > >     video
> > >     and video enhance. Userspace contexts remain scheduler entities.
> > >
> > >
> > > And this is where we fairly strongly disagree.  More in a bit. > > >
> > >     That way you avoid the whole kthread/kworker story and you have it
> > >     actually use the entity picking code in the scheduler, which may be
> > >     useful when the backend is congested.
> > >
> > >
> > > What back-end congestion are you referring to here?  Running out of FW
> > > queue IDs?  Something else?
> >
> > CT channel, number of context ids.
> >
> > >
> > >     Yes you have to solve the out of order problem so in my mind that is
> > >     something to discuss. What the problem actually is (just TDR?), how
> > >     tricky and why etc.
> > >
> > >     And yes you lose the handy LRCA ring buffer size management so you'd
> > >     have to make those entities not runnable in some other way. > > >
> > >     Regarding the argument you raise below - would any of that make the
> > >     frontend / backend separation worse and why? Do you think it is less
> > >     natural? If neither is true then all remains is that it appears extra
> > >     work to support out of order completion of entities has been > > discounted
> > >     in favour of an easy but IMO inelegant option.
> > >
> > >
> > > Broadly speaking, the kernel needs to stop thinking about GPU scheduling
> > > in terms of scheduling jobs and start thinking in terms of scheduling
> > > contexts/engines.  There is still some need for scheduling individual
> > > jobs but that is only for the purpose of delaying them as needed to
> > > resolve dma_fence dependencies.  Once dependencies are resolved, they
> > > get shoved onto the context/engine queue and from there the kernel only
> > > really manages whole contexts/engines.  This is a major architectural
> > > shift, entirely different from the way i915 scheduling works.  It's also
> > > different from the historical usage of DRM scheduler which I think is
> > > why this all looks a bit funny.
> > >
> > > To justify this architectural shift, let's look at where we're headed.
> > > In the glorious future...
> > >
> > >   1. Userspace submits directly to firmware queues.  The kernel has no
> > > visibility whatsoever into individual jobs. At most it can pause/resume
> > > FW contexts as needed to handle eviction and memory management. > > >
> > >   2. Because of 1, apart from handing out the FW queue IDs at the
> > > beginning, the kernel can't really juggle them that much. Depending on
> > > FW design, it may be able to pause a client, give its IDs to another,
> > > and then resume it later when IDs free up. What it's not doing is
> > > juggling IDs on a job-by-job basis like i915 currently is. > > >
> > >   3. Long-running compute jobs may not complete for days. This means
> > > that memory management needs to happen in terms of pause/resume of
> > > entire contexts/engines using the memory rather than based on waiting
> > > for individual jobs to complete or pausing individual jobs until the
> > > memory is available.
> > >
> > >   4. Synchronization happens via userspace memory fences (UMF) and the
> > > kernel is mostly unaware of most dependencies and when a context/engine
> > > is or is not runnable.  Instead, it keeps as many of them minimally
> > > active (memory is available, even if it's in system RAM) as possible and
> > > lets the FW sort out dependencies.  (There may need to be some facility
> > > for sleeping a context until a memory change similar to futex() or
> > > poll() for userspace threads.  There are some details TBD.) > > >
> > > Are there potential problems that will need to be solved here?  Yes.  Is
> > > it a good design?  Well, Microsoft has been living in this future for
> > > half a decade or better and it's working quite well for them.  It's also
> > > the way all modern game consoles work.  It really is just Linux that's
> > > stuck with the same old job model we've had since the monumental shift
> > > to DRI2.
> > >
> > > To that end, one of the core goals of the Xe project was to make the
> > > driver internally behave as close to the above model as possible while
> > > keeping the old-school job model as a very thin layer on top.  As the
> > > broader ecosystem problems (window-system support for UMF, for instance)
> > > are solved, that layer can be peeled back. The core driver will already
> > > be ready for it.
> > >
> > > To that end, the point of the DRM scheduler in Xe isn't to schedule
> > > jobs.  It's to resolve syncobj and dma-buf implicit sync dependencies
> > > and stuff jobs into their respective context/engine queue once they're
> > > ready.  All the actual scheduling happens in firmware and any scheduling
> > > the kernel does to deal with contention, oversubscriptions, too many
> > > contexts, etc. is between contexts/engines, not individual jobs.  Sure,
> > > the individual job visibility is nice, but if we design around it, we'll
> > > never get to the glorious future.
> > >
> > > I really need to turn the above (with a bit more detail) into a blog
> > > post.... Maybe I'll do that this week.
> > >
> > > In any case, I hope that provides more insight into why Xe is designed
> > > the way it is and why I'm pushing back so hard on trying to make it more
> > > of a "classic" driver as far as scheduling is concerned. Are there
> > > potential problems here?  Yes, that's why Xe has been labeled a > > > prototype.  Are such radical changes necessary to get to said glorious
> > > future?  Yes, I think they are.  Will it be worth it?  I believe so.
> >
> > Right, that's all solid I think. My takeaway is that frontend priority
> > sorting and that stuff isn't needed and that is okay. And that there are
> > multiple options to maybe improve drm scheduler, like the fore mentioned
> > making it deal with out of order, or split into functional components,
> > or split frontend/backend what you suggested. For most of them cost vs
> > benefit is more or less not completely clear, neither how much effort
> > was invested to look into them.
> >
> > One thing I missed from this explanation is how drm_scheduler per engine
> > class interferes with the high level concepts. And I did not manage to
> > pick up on what exactly is the TDR problem in that case. Maybe the two
> > are one and the same.
> >
> > Bottom line is I still have the concern that conversion to kworkers has
> > an opportunity to regress. Possibly more opportunity for some Xe use
> > cases than to affect other vendors, since they would still be using per
> > physical engine / queue scheduler instances.
> >
> > And to put my money where my mouth is I will try to put testing Xe > > inside the full blown ChromeOS environment in my team plans. It would
> > probably also be beneficial if Xe team could take a look at real world
> > behaviour of the extreme transcode use cases too. If the stack is ready
> > for that and all. It would be better to know earlier rather than later
> > if there is a fundamental issue.
> >
> > For the patch at hand, and the cover letter, it certainly feels it would
> > benefit to record the past design discussion had with AMD folks, to
> > explicitly copy other drivers, and to record the theoretical pros and
> > cons of threads vs unbound workers as I have tried to highlight them.
> >
> > Regards,
> >
> > Tvrtko
> >

author maintainer
#34 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

In reply to Tvrtko Ursulin
Quoted 85 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 45 lines

>>> AFAICT it proposes to have 1:1 between *userspace* created
contexts (per
>>> context _and_ engine) and drm_sched. I am not sure avoiding
invasive changes
>>> to the shared code is in the spirit of the overall idea and instead >>> opportunity should be used to look at way to refactor/improve
drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.)  There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

>> Yes, it is 1:1 *userspace* engines and drm_sched. >>
>> I'm not really prepared to make large changes to DRM scheduler
at the
>> moment for Xe as they are not really required nor does Boris
seem they
>> will be required for his work either. I am interested to see
what Boris
>> comes up with.
>>
>>> Even on the low level, the idea to replace drm_sched threads
with workers
>>> has a few problems.
>>>
>>> To start with, the pattern of:
>>>
>>>    while (not_stopped) {
>>>     keep picking jobs
>>>    }
>>>
>>> Feels fundamentally in disagreement with workers (while
obviously fits
>>> perfectly with the current kthread design).
>>
>> The while loop breaks and worker exists if no jobs are ready.

Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 5 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

We can use a different work queue if this is an issue, have a FIXME which indicates we should allow the user to pass in the work queue.

Quoted 8 lines

Unbound workers have no CPU / cache locality either and no connection with the CPU scheduler to optimize scheduling patterns. This may matter either on large systems or on small ones. Whereas the current design allows for scheduler to notice userspace CPU thread keeps waking up the same drm scheduler kernel thread, and so it can keep them on the same CPU, the unbound workers lose that ability and so 2nd CPU might be getting woken up from low sleep for every submission.

I guess I don't understand kthread vs. workqueue scheduling internals.

Quoted 4 lines

Hence, apart from being a bit of a impedance mismatch, the proposal has the potential to change performance and power patterns and both large and small machines.

We are going to have to test this out I suppose and play around to see if this design has any real world impacts. As Jason said, yea probably will need a bit of help here from others. Will CC relavent parties on next rev.

Quoted 84 lines
Quoted 55 lines

>>> Secondly, it probably demands separate workers (not optional),
otherwise
>>> behaviour of shared workqueues has either the potential to
explode number
>>> kernel threads anyway, or add latency.
>>>
>>
>> Right now the system_unbound_wq is used which does have a limit
on the
>> number of threads, right? I do have a FIXME to allow a worker to be >> passed in similar to TDR.
>>
>> WRT to latency, the 1:1 ratio could actually have lower latency
as 2 GPU
>> schedulers can be pushing jobs into the backend / cleaning up
jobs in
>> parallel.
>>
>
> Thought of one more point here where why in Xe we absolutely want
a 1 to
> 1 ratio between entity and scheduler - the way we implement
timeslicing
> for preempt fences.
>
> Let me try to explain.
>
> Preempt fences are implemented via the generic messaging
interface [1]
> with suspend / resume messages. If a suspend messages is received to > soon after calling resume (this is per entity) we simply sleep in the > suspend call thus giving the entity a timeslice. This completely
falls
> apart with a many to 1 relationship as now a entity waiting for a > timeslice blocks the other entities. Could we work aroudn this,
sure but
> just another bunch of code we'd have to add in Xe. Being to
freely sleep
> in backend without affecting other entities is really, really
nice IMO
> and I bet Xe isn't the only driver that is going to feel this way. >
> Last thing I'll say regardless of how anyone feels about Xe using
a 1 to
> 1 relationship this patch IMO makes sense as I hope we can all
agree a
> workqueue scales better than kthreads.

I don't know for sure what will scale better and for what use case, combination of CPU cores vs number of GPU engines to keep busy vs other system activity. But I wager someone is bound to ask for some numbers to
make sure proposal is not negatively affecting any other drivers.

Quoted 7 lines

Then let them ask.  Waving your hands vaguely in the direction of the rest of DRM and saying "Uh, someone (not me) might object" is profoundly unhelpful.  Sure, someone might.  That's why it's on dri-devel.  If you think there's someone in particular who might have a useful opinion on this, throw them in the CC so they don't miss the e-mail thread.

Or are you asking for numbers?  If so, what numbers are you asking for?

It was a heads up to the Xe team in case people weren't appreciating how the proposed change has the potential influence power and performance across the board. And nothing in the follow up discussion made me think it was considered so I don't think it was redundant to raise it.

In my experience it is typical that such core changes come with some numbers. Which is in case of drm scheduler is tricky and probably requires explicitly asking everyone to test (rather than count on "don't miss the email thread"). Real products can fail to ship due ten mW here or there. Like suddenly an extra core prevented from getting into deep sleep.

If that was "profoundly unhelpful" so be it.

Quoted 7 lines

Also, If we're talking about a design that might paint us into an Intel-HW-specific hole, that would be one thing.  But we're not.  We're talking about switching which kernel threading/task mechanism to use for what's really a very generic problem.  The core Xe design works without this patch (just with more kthreads).  If we land this patch or something like it and get it wrong and it causes a performance problem for someone down the line, we can revisit it.

Quoted 3 lines

For some definition of "it works" - I really wouldn't suggest shipping a kthread per user context at any point.

Yea, this is why using a workqueue rathre than a kthread was suggested to me by AMD. I should've put a suggested by on the commit message, need to dig through my emails and figure out who exactly suggested this.

Quoted 45 lines
Quoted 35 lines

In any case that's a low level question caused by the high level design decision. So I'd think first focus on the high level - which is the 1:1 mapping of entity to scheduler instance proposal.

Fundamentally it will be up to the DRM maintainers and the community to bless your approach. And it is important to stress 1:1 is about userspace contexts, so I believe unlike any other current scheduler user. And also important to stress this effectively does not make Xe _really_ use the scheduler that much.

I don't think this makes Xe nearly as much of a one-off as you think it does.  I've already told the Asahi team working on Apple M1/2 hardware to do it this way and it seems to be a pretty good mapping for them. I believe this is roughly the plan for nouveau as well.  It's not the way it currently works for anyone because most other groups aren't doing FW scheduling yet.  In the world of FW scheduling and hardware designed to support userspace direct-to-FW submit, I think the design makes perfect sense (see below) and I expect we'll see more drivers move in this direction as those drivers evolve.  (AMD is doing some customish thing for how with gpu_scheduler on the front-end somehow. I've not dug into those details.)

I can only offer my opinion, which is that the two options mentioned in this thread (either improve drm scheduler to cope with what is required,
or split up the code so you can use just the parts of drm_sched which you want - which is frontend dependency tracking) shouldn't be so readily dismissed, given how I think the idea was for the new driver to work less in a silo and more in the community (not do kludges to workaround stuff because it is thought to be too hard to improve common code), but fundamentally, "goto previous paragraph" for what I am concerned.

Quoted 9 lines

Meta comment:  It appears as if you're falling into the standard i915 team trap of having an internal discussion about what the community discussion might look like instead of actually having the community discussion.  If you are seriously concerned about interactions with other drivers or whether or setting common direction, the right way to do that is to break a patch or two out into a separate RFC series and tag a handful of driver maintainers.  Trying to predict the questions other people might ask is pointless. Cc them and asking for their input instead.

Quoted 7 lines

I don't follow you here. It's not an internal discussion - I am raising my concerns on the design publicly. I am supposed to write a patch to show something, but am allowed to comment on a RFC series?

It is "drm/sched: Convert drm scheduler to use a work queue rather than kthread" which should have Cc-ed _everyone_ who use drm scheduler.

Yea, will do on next rev.

Quoted 48 lines
Quoted 40 lines

Regards,

Tvrtko

P.S. And as a related side note, there are more areas where drm_sched could be improved, like for instance priority handling. Take a look at msm_submitqueue_create / msm_gpu_convert_priority / get_sched_entity to see how msm works around the drm_sched hardcoded limit of available priority levels, in order to avoid having to leave a hw capability unused. I suspect msm would be happier if they could have all priority levels equal in terms of whether they apply only at the frontend level or completely throughout the pipeline.

> [1]
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
>
>>> What would be interesting to learn is whether the option of
refactoring
>>> drm_sched to deal with out of order completion was considered
and what were
>>> the conclusions.
>>>
>>
>> I coded this up a while back when trying to convert the i915 to
the DRM
>> scheduler it isn't all that hard either. The free flow control
on the
>> ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE) is
really what
>> sold me on the this design.

You're not the only one to suggest supporting out-of-order completion. However, it's tricky and breaks a lot of internal assumptions of the scheduler. It also reduces functionality a bit because it can no longer automatically rate-limit HW/FW queues which are often fixed-size.  (Ok, yes, it probably could but it becomes a substantially harder problem.)

Quoted 7 lines

It also seems like a worse mapping to me.  The goal here is to turn submissions on a userspace-facing engine/queue into submissions to a FW queue submissions, sorting out any dma_fence dependencies.  Matt's description of saying this is a 1:1 mapping between sched/entity doesn't tell the whole story. It's a 1:1:1 mapping between xe_engine, gpu_scheduler, and GuC FW engine.  Why make it a 1:something:1 mapping? Why is that better?

Quoted 5 lines

As I have stated before, what I think what would fit well for Xe is one drm_scheduler per engine class. In specific terms on our current hardware, one drm scheduler instance for render, compute, blitter, video and video enhance. Userspace contexts remain scheduler entities.

I disagree.

Quoted 4 lines

That way you avoid the whole kthread/kworker story and you have it actually use the entity picking code in the scheduler, which may be useful when the backend is congested.

In practice the backend shouldn't be congested but if it is a mutex provides fairness probably better than using a shared scheduler. Also what you are suggesting doesn't make sense at all as the congestion is per-GT, so if anything we should use 1 scheduler per-GT not per engine class.

Quoted 4 lines

Yes you have to solve the out of order problem so in my mind that is something to discuss. What the problem actually is (just TDR?), how tricky and why etc.

Cleanup of jobs, TDR, replaying jobs, etc... It has decent amount of impact.

Quoted 3 lines

And yes you lose the handy LRCA ring buffer size management so you'd have to make those entities not runnable in some other way.

Also we lose our preempt fence implemenation too. Again I don't see how the design you are suggesting is a win.

Quoted 19 lines

Regarding the argument you raise below - would any of that make the frontend / backend separation worse and why? Do you think it is less natural? If neither is true then all remains is that it appears extra work to support out of order completion of entities has been discounted in favour of an easy but IMO inelegant option.

Quoted 5 lines

There are two places where this 1:1:1 mapping is causing problems:

 1. It creates lots of kthreads. This is what this patch is trying to
solve. IDK if it's solving it the best way but that's the goal.

Quoted 7 lines

 2. There are a far more limited number of communication queues between
the kernel and GuC for more meta things like pausing and resuming queues, getting events back from GuC, etc. Unless we're in a weird pressure scenario, the amount of traffic on this queue should be low so we can probably just have one per physical device.  The vast majority of kernel -> GuC communication should be on the individual FW queue rings and maybe smashing in-memory doorbells.

Quoted 4 lines

I don't follow your terminology here. I suppose you are talking about global GuC CT and context ringbuffers. If so then isn't "far more limited" actually one?

We have 1 GuC GT per-GT.

Matt

Quoted 96 lines

Regards,

Tvrtko

Quoted 92 lines

Doing out-of-order completion sort-of solves the 1 but does nothing for 2 and actually makes managing FW queues harder because we no longer have built-in rate limiting.  Seems like a net loss to me.

>>> Second option perhaps to split out the drm_sched code into
parts which would
>>> lend themselves more to "pick and choose" of its functionalities. >>> Specifically, Xe wants frontend dependency tracking, but not
any scheduling
>>> really (neither least busy drm_sched, neither FIFO/RQ entity
picking), so
>>> even having all these data structures in memory is a waste. >>>
>>
>> I don't think that we are wasting memory is a very good argument for >> making intrusive changes to the DRM scheduler.

Worse than that, I think the "we could split it up" kind-of misses the point of the way Xe is using drm/scheduler.  It's not just about re-using a tiny bit of dependency tracking code.  Using the scheduler in this way provides a clean separation between front-end and back-end. The job of the userspace-facing ioctl code is to shove things on the scheduler.  The job of the run_job callback is to encode the job into the FW queue format, stick it in the FW queue ring, and maybe smash a doorbell.  Everything else happens in terms of managing those queues side-band.  The gpu_scheduler code manages the front-end queues and Xe manages the FW queues via the Kernel <-> GuC communication rings.  From a high level, this is a really clean design.  There are potentially some sticky bits around the dual-use of dma_fence for scheduling and memory management but none of those are solved by breaking the DRM scheduler into chunks or getting rid of the 1:1:1 mapping.

If we split it out, we're basically asking the driver to implement a bunch of kthread or workqueue stuff, all the ring rate-limiting, etc. It may not be all that much code but also, why?  To save a few bytes of memory per engine?  Each engine already has 32K(ish) worth of context state and a similar size ring to communicate with the FW.  No one is going to notice an extra CPU data structure.

I'm not seeing a solid argument against the 1:1:1 design here other than that it doesn't seem like the way DRM scheduler was intended to be used.  I won't argue that.  It's not.  But it is a fairly natural way to take advantage of the benefits the DRM scheduler does provide while also mapping it to hardware that was designed for userspace direct-to-FW submit.

--Jason

>>> With the first option then the end result could be drm_sched
per engine
>>> class (hardware view), which I think fits with the GuC model.
Give all
>>> schedulable contexts (entities) to the GuC and then mostly
forget about
>>> them. Timeslicing and re-ordering and all happens transparently
to the
>>> kernel from that point until completion.
>>>
>>
>> Out-of-order problem still exists here.
>>
>>> Or with the second option you would build on some smaller
refactored
>>> sub-components of drm_sched, by maybe splitting the dependency
tracking from
>>> scheduling (RR/FIFO entity picking code).
>>>
>>> Second option is especially a bit vague and I haven't thought
about the
>>> required mechanics, but it just appeared too obvious the
proposed design has
>>> a bit too much impedance mismatch.
>>>
>>
>> IMO ROI on this is low and again lets see what Boris comes up with. >>
>> Matt
>>
>>> Oh and as a side note, when I went into the drm_sched code base
to remind
>>> myself how things worked, it is quite easy to find some FIXME
comments which
>>> suggest people working on it are unsure of locking desing there
and such. So
>>> perhaps that all needs cleanup too, I mean would benefit from >>> refactoring/improving work as brainstormed above anyway. >>>
>>> Regards,
>>>
>>> Tvrtko

author maintainer
#35 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 104 lines · Matthew Brost
Quoted 85 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 45 lines

>>> AFAICT it proposes to have 1:1 between *userspace* created
contexts (per
>>> context _and_ engine) and drm_sched. I am not sure avoiding
invasive changes
>>> to the shared code is in the spirit of the overall idea and instead >>> opportunity should be used to look at way to refactor/improve
drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.)  There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

>> Yes, it is 1:1 *userspace* engines and drm_sched. >>
>> I'm not really prepared to make large changes to DRM scheduler
at the
>> moment for Xe as they are not really required nor does Boris
seem they
>> will be required for his work either. I am interested to see
what Boris
>> comes up with.
>>
>>> Even on the low level, the idea to replace drm_sched threads
with workers
>>> has a few problems.
>>>
>>> To start with, the pattern of:
>>>
>>>    while (not_stopped) {
>>>     keep picking jobs
>>>    }
>>>
>>> Feels fundamentally in disagreement with workers (while
obviously fits
>>> perfectly with the current kthread design).
>>
>> The while loop breaks and worker exists if no jobs are ready.

Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 5 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

We can use a different work queue if this is an issue, have a FIXME which indicates we should allow the user to pass in the work queue.

Quoted 8 lines

Unbound workers have no CPU / cache locality either and no connection with the CPU scheduler to optimize scheduling patterns. This may matter either on large systems or on small ones. Whereas the current design allows for scheduler to notice userspace CPU thread keeps waking up the same drm scheduler kernel thread, and so it can keep them on the same CPU, the unbound workers lose that ability and so 2nd CPU might be getting woken up from low sleep for every submission.

Quoted 2 lines

I guess I don't understand kthread vs. workqueue scheduling internals.

Looked into this and we are not using unbound workers rather we are just using the system_wq which is indeed bound. Again we can change this so a user can just pass in worker too. After doing a of research bound workers allows the scheduler to use locality too avoid that exact problem your reading.

TL;DR I'm not buying any of these arguments although it is possible I am missing something.

Matt

Quoted 363 lines
Quoted 4 lines

Hence, apart from being a bit of a impedance mismatch, the proposal has the potential to change performance and power patterns and both large and small machines.

We are going to have to test this out I suppose and play around to see if this design has any real world impacts. As Jason said, yea probably will need a bit of help here from others. Will CC relavent parties on next rev.

Quoted 84 lines
Quoted 55 lines

>>> Secondly, it probably demands separate workers (not optional),
otherwise
>>> behaviour of shared workqueues has either the potential to
explode number
>>> kernel threads anyway, or add latency.
>>>
>>
>> Right now the system_unbound_wq is used which does have a limit
on the
>> number of threads, right? I do have a FIXME to allow a worker to be >> passed in similar to TDR.
>>
>> WRT to latency, the 1:1 ratio could actually have lower latency
as 2 GPU
>> schedulers can be pushing jobs into the backend / cleaning up
jobs in
>> parallel.
>>
>
> Thought of one more point here where why in Xe we absolutely want
a 1 to
> 1 ratio between entity and scheduler - the way we implement
timeslicing
> for preempt fences.
>
> Let me try to explain.
>
> Preempt fences are implemented via the generic messaging
interface [1]
> with suspend / resume messages. If a suspend messages is received to > soon after calling resume (this is per entity) we simply sleep in the > suspend call thus giving the entity a timeslice. This completely
falls
> apart with a many to 1 relationship as now a entity waiting for a > timeslice blocks the other entities. Could we work aroudn this,
sure but
> just another bunch of code we'd have to add in Xe. Being to
freely sleep
> in backend without affecting other entities is really, really
nice IMO
> and I bet Xe isn't the only driver that is going to feel this way. >
> Last thing I'll say regardless of how anyone feels about Xe using
a 1 to
> 1 relationship this patch IMO makes sense as I hope we can all
agree a
> workqueue scales better than kthreads.

I don't know for sure what will scale better and for what use case, combination of CPU cores vs number of GPU engines to keep busy vs other system activity. But I wager someone is bound to ask for some numbers to
make sure proposal is not negatively affecting any other drivers.

Quoted 7 lines

Then let them ask.  Waving your hands vaguely in the direction of the rest of DRM and saying "Uh, someone (not me) might object" is profoundly unhelpful.  Sure, someone might.  That's why it's on dri-devel.  If you think there's someone in particular who might have a useful opinion on this, throw them in the CC so they don't miss the e-mail thread.

Or are you asking for numbers?  If so, what numbers are you asking for?

It was a heads up to the Xe team in case people weren't appreciating how the proposed change has the potential influence power and performance across the board. And nothing in the follow up discussion made me think it was considered so I don't think it was redundant to raise it.

In my experience it is typical that such core changes come with some numbers. Which is in case of drm scheduler is tricky and probably requires explicitly asking everyone to test (rather than count on "don't miss the email thread"). Real products can fail to ship due ten mW here or there. Like suddenly an extra core prevented from getting into deep sleep.

If that was "profoundly unhelpful" so be it.

Quoted 7 lines

Also, If we're talking about a design that might paint us into an Intel-HW-specific hole, that would be one thing.  But we're not.  We're talking about switching which kernel threading/task mechanism to use for what's really a very generic problem.  The core Xe design works without this patch (just with more kthreads).  If we land this patch or something like it and get it wrong and it causes a performance problem for someone down the line, we can revisit it.

Quoted 3 lines

For some definition of "it works" - I really wouldn't suggest shipping a kthread per user context at any point.

Yea, this is why using a workqueue rathre than a kthread was suggested to me by AMD. I should've put a suggested by on the commit message, need to dig through my emails and figure out who exactly suggested this.

Quoted 45 lines
Quoted 35 lines

In any case that's a low level question caused by the high level design decision. So I'd think first focus on the high level - which is the 1:1 mapping of entity to scheduler instance proposal.

Fundamentally it will be up to the DRM maintainers and the community to bless your approach. And it is important to stress 1:1 is about userspace contexts, so I believe unlike any other current scheduler user. And also important to stress this effectively does not make Xe _really_ use the scheduler that much.

I don't think this makes Xe nearly as much of a one-off as you think it does.  I've already told the Asahi team working on Apple M1/2 hardware to do it this way and it seems to be a pretty good mapping for them. I believe this is roughly the plan for nouveau as well.  It's not the way it currently works for anyone because most other groups aren't doing FW scheduling yet.  In the world of FW scheduling and hardware designed to support userspace direct-to-FW submit, I think the design makes perfect sense (see below) and I expect we'll see more drivers move in this direction as those drivers evolve.  (AMD is doing some customish thing for how with gpu_scheduler on the front-end somehow. I've not dug into those details.)

I can only offer my opinion, which is that the two options mentioned in this thread (either improve drm scheduler to cope with what is required,
or split up the code so you can use just the parts of drm_sched which you want - which is frontend dependency tracking) shouldn't be so readily dismissed, given how I think the idea was for the new driver to work less in a silo and more in the community (not do kludges to workaround stuff because it is thought to be too hard to improve common code), but fundamentally, "goto previous paragraph" for what I am concerned.

Quoted 9 lines

Meta comment:  It appears as if you're falling into the standard i915 team trap of having an internal discussion about what the community discussion might look like instead of actually having the community discussion.  If you are seriously concerned about interactions with other drivers or whether or setting common direction, the right way to do that is to break a patch or two out into a separate RFC series and tag a handful of driver maintainers.  Trying to predict the questions other people might ask is pointless. Cc them and asking for their input instead.

Quoted 7 lines

I don't follow you here. It's not an internal discussion - I am raising my concerns on the design publicly. I am supposed to write a patch to show something, but am allowed to comment on a RFC series?

It is "drm/sched: Convert drm scheduler to use a work queue rather than kthread" which should have Cc-ed _everyone_ who use drm scheduler.

Yea, will do on next rev.

Quoted 48 lines
Quoted 40 lines

Regards,

Tvrtko

P.S. And as a related side note, there are more areas where drm_sched could be improved, like for instance priority handling. Take a look at msm_submitqueue_create / msm_gpu_convert_priority / get_sched_entity to see how msm works around the drm_sched hardcoded limit of available priority levels, in order to avoid having to leave a hw capability unused. I suspect msm would be happier if they could have all priority levels equal in terms of whether they apply only at the frontend level or completely throughout the pipeline.

> [1]
https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1
<https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1>
>
>>> What would be interesting to learn is whether the option of
refactoring
>>> drm_sched to deal with out of order completion was considered
and what were
>>> the conclusions.
>>>
>>
>> I coded this up a while back when trying to convert the i915 to
the DRM
>> scheduler it isn't all that hard either. The free flow control
on the
>> ring (e.g. set job limit == SIZE OF RING / MAX JOB SIZE) is
really what
>> sold me on the this design.

You're not the only one to suggest supporting out-of-order completion. However, it's tricky and breaks a lot of internal assumptions of the scheduler. It also reduces functionality a bit because it can no longer automatically rate-limit HW/FW queues which are often fixed-size.  (Ok, yes, it probably could but it becomes a substantially harder problem.)

Quoted 7 lines

It also seems like a worse mapping to me.  The goal here is to turn submissions on a userspace-facing engine/queue into submissions to a FW queue submissions, sorting out any dma_fence dependencies.  Matt's description of saying this is a 1:1 mapping between sched/entity doesn't tell the whole story. It's a 1:1:1 mapping between xe_engine, gpu_scheduler, and GuC FW engine.  Why make it a 1:something:1 mapping? Why is that better?

Quoted 5 lines

As I have stated before, what I think what would fit well for Xe is one drm_scheduler per engine class. In specific terms on our current hardware, one drm scheduler instance for render, compute, blitter, video and video enhance. Userspace contexts remain scheduler entities.

I disagree.

Quoted 4 lines

That way you avoid the whole kthread/kworker story and you have it actually use the entity picking code in the scheduler, which may be useful when the backend is congested.

In practice the backend shouldn't be congested but if it is a mutex provides fairness probably better than using a shared scheduler. Also what you are suggesting doesn't make sense at all as the congestion is per-GT, so if anything we should use 1 scheduler per-GT not per engine class.

Quoted 4 lines

Yes you have to solve the out of order problem so in my mind that is something to discuss. What the problem actually is (just TDR?), how tricky and why etc.

Cleanup of jobs, TDR, replaying jobs, etc... It has decent amount of impact.

Quoted 3 lines

And yes you lose the handy LRCA ring buffer size management so you'd have to make those entities not runnable in some other way.

Also we lose our preempt fence implemenation too. Again I don't see how the design you are suggesting is a win.

Quoted 19 lines

Regarding the argument you raise below - would any of that make the frontend / backend separation worse and why? Do you think it is less natural? If neither is true then all remains is that it appears extra work to support out of order completion of entities has been discounted in favour of an easy but IMO inelegant option.

Quoted 5 lines

There are two places where this 1:1:1 mapping is causing problems:

 1. It creates lots of kthreads. This is what this patch is trying to
solve. IDK if it's solving it the best way but that's the goal.

Quoted 7 lines

 2. There are a far more limited number of communication queues between
the kernel and GuC for more meta things like pausing and resuming queues, getting events back from GuC, etc. Unless we're in a weird pressure scenario, the amount of traffic on this queue should be low so we can probably just have one per physical device.  The vast majority of kernel -> GuC communication should be on the individual FW queue rings and maybe smashing in-memory doorbells.

Quoted 4 lines

I don't follow your terminology here. I suppose you are talking about global GuC CT and context ringbuffers. If so then isn't "far more limited" actually one?

We have 1 GuC GT per-GT.

Matt

Quoted 96 lines

Regards,

Tvrtko

Quoted 92 lines

Doing out-of-order completion sort-of solves the 1 but does nothing for 2 and actually makes managing FW queues harder because we no longer have built-in rate limiting.  Seems like a net loss to me.

>>> Second option perhaps to split out the drm_sched code into
parts which would
>>> lend themselves more to "pick and choose" of its functionalities. >>> Specifically, Xe wants frontend dependency tracking, but not
any scheduling
>>> really (neither least busy drm_sched, neither FIFO/RQ entity
picking), so
>>> even having all these data structures in memory is a waste. >>>
>>
>> I don't think that we are wasting memory is a very good argument for >> making intrusive changes to the DRM scheduler.

Worse than that, I think the "we could split it up" kind-of misses the point of the way Xe is using drm/scheduler.  It's not just about re-using a tiny bit of dependency tracking code.  Using the scheduler in this way provides a clean separation between front-end and back-end. The job of the userspace-facing ioctl code is to shove things on the scheduler.  The job of the run_job callback is to encode the job into the FW queue format, stick it in the FW queue ring, and maybe smash a doorbell.  Everything else happens in terms of managing those queues side-band.  The gpu_scheduler code manages the front-end queues and Xe manages the FW queues via the Kernel <-> GuC communication rings.  From a high level, this is a really clean design.  There are potentially some sticky bits around the dual-use of dma_fence for scheduling and memory management but none of those are solved by breaking the DRM scheduler into chunks or getting rid of the 1:1:1 mapping.

If we split it out, we're basically asking the driver to implement a bunch of kthread or workqueue stuff, all the ring rate-limiting, etc. It may not be all that much code but also, why?  To save a few bytes of memory per engine?  Each engine already has 32K(ish) worth of context state and a similar size ring to communicate with the FW.  No one is going to notice an extra CPU data structure.

I'm not seeing a solid argument against the 1:1:1 design here other than that it doesn't seem like the way DRM scheduler was intended to be used.  I won't argue that.  It's not.  But it is a fairly natural way to take advantage of the benefits the DRM scheduler does provide while also mapping it to hardware that was designed for userspace direct-to-FW submit.

--Jason

>>> With the first option then the end result could be drm_sched
per engine
>>> class (hardware view), which I think fits with the GuC model.
Give all
>>> schedulable contexts (entities) to the GuC and then mostly
forget about
>>> them. Timeslicing and re-ordering and all happens transparently
to the
>>> kernel from that point until completion.
>>>
>>
>> Out-of-order problem still exists here.
>>
>>> Or with the second option you would build on some smaller
refactored
>>> sub-components of drm_sched, by maybe splitting the dependency
tracking from
>>> scheduling (RR/FIFO entity picking code).
>>>
>>> Second option is especially a bit vague and I haven't thought
about the
>>> required mechanics, but it just appeared too obvious the
proposed design has
>>> a bit too much impedance mismatch.
>>>
>>
>> IMO ROI on this is low and again lets see what Boris comes up with. >>
>> Matt
>>
>>> Oh and as a side note, when I went into the drm_sched code base
to remind
>>> myself how things worked, it is quite easy to find some FIXME
comments which
>>> suggest people working on it are unsure of locking desing there
and such. So
>>> perhaps that all needs cleanup too, I mean would benefit from >>> refactoring/improving work as brainstormed above anyway. >>>
>>> Regards,
>>>
>>> Tvrtko

#36 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 108 lines · Matthew Brost
Quoted 104 lines · Matthew Brost
Quoted 85 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 45 lines

>>> AFAICT it proposes to have 1:1 between *userspace* created
contexts (per
>>> context _and_ engine) and drm_sched. I am not sure avoiding
invasive changes
>>> to the shared code is in the spirit of the overall idea and instead >>> opportunity should be used to look at way to refactor/improve
drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.)  There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

>> Yes, it is 1:1 *userspace* engines and drm_sched. >>
>> I'm not really prepared to make large changes to DRM scheduler
at the
>> moment for Xe as they are not really required nor does Boris
seem they
>> will be required for his work either. I am interested to see
what Boris
>> comes up with.
>>
>>> Even on the low level, the idea to replace drm_sched threads
with workers
>>> has a few problems.
>>>
>>> To start with, the pattern of:
>>>
>>>    while (not_stopped) {
>>>     keep picking jobs
>>>    }
>>>
>>> Feels fundamentally in disagreement with workers (while
obviously fits
>>> perfectly with the current kthread design).
>>
>> The while loop breaks and worker exists if no jobs are ready.

Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 5 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

We can use a different work queue if this is an issue, have a FIXME which indicates we should allow the user to pass in the work queue.

Quoted 8 lines

Unbound workers have no CPU / cache locality either and no connection with the CPU scheduler to optimize scheduling patterns. This may matter either on large systems or on small ones. Whereas the current design allows for scheduler to notice userspace CPU thread keeps waking up the same drm scheduler kernel thread, and so it can keep them on the same CPU, the unbound workers lose that ability and so 2nd CPU might be getting woken up from low sleep for every submission.

Quoted 2 lines

I guess I don't understand kthread vs. workqueue scheduling internals.

Quoted 8 lines

Looked into this and we are not using unbound workers rather we are just using the system_wq which is indeed bound. Again we can change this so a user can just pass in worker too. After doing a of research bound workers allows the scheduler to use locality too avoid that exact problem your reading.

TL;DR I'm not buying any of these arguments although it is possible I am missing something.

Well you told me it's using unbound.. message id [email protected]:

"""
Right now the system_unbound_wq is used which does have a limit on the number of threads, right? I do have a FIXME to allow a worker to be passed in similar to TDR.
"""

With bound workers you will indeed get CPU locality. I am not sure what it will do in terms of concurrency. If it will serialize work items to fewer spawned workers that will be good for the CT contention issue, but may negatively affect latency. And possibly preemption / time slicing decisions since the order of submitting to the backend will not be in the order of context priority, hence high prio may be submitted right after low and immediately trigger preemption.

Anyway, since you are not buying any arguments on paper perhaps you are more open towards testing. If you would adapt gem_wsim for Xe you would be able to spawn N simulated transcode sessions on any Gen11+ machine and try it out.

For example:

gem_wsim -w benchmarks/wsim/media_load_balance_fhd26u7.wsim -c 36 -r 600

That will run you 36 parallel transcoding sessions streams for 600 frames each. No client setup needed whatsoever apart from compiling IGT.

In the past that was quite a handy tool to identify scheduling issues, or validate changes against. All workloads with the media prefix have actually been hand crafted by looking at what real media pipelines do with real data. Few years back at least.

It could show you real world behaviour of the kworkers approach and it could also enable you to cross reference any power and performance changes relative to i915. Background story there is that media servers like to fit N streams to a server and if a change comes along which suddenly makes only N-1 stream fit before dropping out of realtime, that's a big problem.

If you will believe me there is value in that kind of testing I am happy to help you add Xe support to the tool, time permitting so possibly guidance only at the moment.

Regards,

Tvrtko

author maintainer
#37 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 118 lines · Tvrtko Ursulin
Quoted 107 lines · Matthew Brost
Quoted 104 lines · Matthew Brost
Quoted 85 lines · Tvrtko Ursulin

On 09/01/2023 17:27, Jason Ekstrand wrote:

[snip]

Quoted 45 lines

>>> AFAICT it proposes to have 1:1 between *userspace* created
contexts (per
>>> context _and_ engine) and drm_sched. I am not sure avoiding
invasive changes
>>> to the shared code is in the spirit of the overall idea and instead >>> opportunity should be used to look at way to refactor/improve
drm_sched.

Maybe?  I'm not convinced that what Xe is doing is an abuse at all or really needs to drive a re-factor.  (More on that later.)  There's only one real issue which is that it fires off potentially a lot of kthreads. Even that's not that bad given that kthreads are pretty light and you're not likely to have more kthreads than userspace threads which are much heavier.  Not ideal, but not the end of the world either.  Definitely something we can/should optimize but if we went through with Xe without this patch, it would probably be mostly ok.

>> Yes, it is 1:1 *userspace* engines and drm_sched. >>
>> I'm not really prepared to make large changes to DRM scheduler
at the
>> moment for Xe as they are not really required nor does Boris
seem they
>> will be required for his work either. I am interested to see
what Boris
>> comes up with.
>>
>>> Even on the low level, the idea to replace drm_sched threads
with workers
>>> has a few problems.
>>>
>>> To start with, the pattern of:
>>>
>>>    while (not_stopped) {
>>>     keep picking jobs
>>>    }
>>>
>>> Feels fundamentally in disagreement with workers (while
obviously fits
>>> perfectly with the current kthread design).
>>
>> The while loop breaks and worker exists if no jobs are ready.

Quoted 3 lines

I'm not very familiar with workqueues. What are you saying would fit better? One scheduling job per work item rather than one big work item which handles all available jobs?

Yes and no, it indeed IMO does not fit to have a work item which is potentially unbound in runtime. But it is a bit moot conceptual mismatch because it is a worst case / theoretical, and I think due more fundamental concerns.

If we have to go back to the low level side of things, I've picked this random spot to consolidate what I have already mentioned and perhaps expand.

To start with, let me pull out some thoughts from workqueue.rst:

"""
Generally, work items are not expected to hog a CPU and consume many cycles. That means maintaining just enough concurrency to prevent work processing from stalling should be optimal.
"""

For unbound queues:
"""
The responsibility of regulating concurrency level is on the users. """

Given the unbound queues will be spawned on demand to service all queued work items (more interesting when mixing up with the system_unbound_wq), in the proposed design the number of instantiated worker threads does not correspond to the number of user threads (as you have elsewhere stated), but pessimistically to the number of active user contexts. That is the number which drives the maximum number of not-runnable jobs that can become runnable at once, and hence spawn that many work items, and in turn unbound worker threads.

Quoted 5 lines

Several problems there.

It is fundamentally pointless to have potentially that many more threads than the number of CPU cores - it simply creates a scheduling storm.

We can use a different work queue if this is an issue, have a FIXME which indicates we should allow the user to pass in the work queue.

Quoted 8 lines

Unbound workers have no CPU / cache locality either and no connection with the CPU scheduler to optimize scheduling patterns. This may matter either on large systems or on small ones. Whereas the current design allows for scheduler to notice userspace CPU thread keeps waking up the same drm scheduler kernel thread, and so it can keep them on the same CPU, the unbound workers lose that ability and so 2nd CPU might be getting woken up from low sleep for every submission.

Quoted 1 line

I guess I don't understand kthread vs. workqueue scheduling internals.

Quoted 8 lines

Looked into this and we are not using unbound workers rather we are just using the system_wq which is indeed bound. Again we can change this so a user can just pass in worker too. After doing a of research bound workers allows the scheduler to use locality too avoid that exact problem your reading.

TL;DR I'm not buying any of these arguments although it is possible I am missing something.

Quoted 9 lines

Well you told me it's using unbound.. message id
[email protected]:

"""
Right now the system_unbound_wq is used which does have a limit on the number of threads, right? I do have a FIXME to allow a worker to be passed in similar to TDR.
"""

Yea, my mistake. A quick look at the shows we are using system_wq (same as TDR).

Quoted 8 lines

With bound workers you will indeed get CPU locality. I am not sure what it will do in terms of concurrency. If it will serialize work items to fewer spawned workers that will be good for the CT contention issue, but may negatively affect latency. And possibly preemption / time slicing decisions since the order of submitting to the backend will not be in the order of context priority, hence high prio may be submitted right after low and immediately trigger preemption.

We should probably use system_highpri_wq for high priority contexts (xe_engine).

Quoted 9 lines

Anyway, since you are not buying any arguments on paper perhaps you are more open towards testing. If you would adapt gem_wsim for Xe you would be able to spawn N simulated transcode sessions on any Gen11+ machine and try it out.

For example:

gem_wsim -w benchmarks/wsim/media_load_balance_fhd26u7.wsim -c 36 -r 600

Quoted 8 lines

That will run you 36 parallel transcoding sessions streams for 600 frames each. No client setup needed whatsoever apart from compiling IGT.

In the past that was quite a handy tool to identify scheduling issues, or validate changes against. All workloads with the media prefix have actually been hand crafted by looking at what real media pipelines do with real data. Few years back at least.

Porting this is non-trivial as this is 2.5k. Also in Xe we are trending to use UMD benchmarks to determine if there are performance problems as in the i915 we had tons microbenchmarks / IGT benchmarks that we found meant absolutely nothing. Can't say if this benchmark falls into that category.

We VK and compute benchmarks running and haven't found any major issues yet. The media UMD hasn't been ported because of the VM bind dependency so I can't say if there are any issues with the media UMD + Xe.

What I can do hack up xe_exec_threads to really hammer Xe - change it to 128x xe_engines + 8k execs per thread. Each exec is super simple, it just stores a dword. It creates a thread per hardware engine, so on TGL this is 5x threads.

Results below:

root@DUT025-TGLU:mbrost# xe_exec_threads --r threads-basic
IGT-Version: 1.26-ge26de4b2 (x86_64) (Linux: 6.1.0-rc1-xe+ x86_64)
Starting subtest: threads-basic
Subtest threads-basic: SUCCESS (1.215s)
root@DUT025-TGLU:mbrost# dumptrace | grep job | wc
  40960  491520 7401728
root@DUT025-TGLU:mbrost# dumptrace | grep engine | wc
    645    7095   82457

So with 640 xe_engines (5x are VM engines) it takes 1.215 seconds test time to run 40960 execs. That seems to indicate we do not have a scheduling problem.

This is 8 core (or at least 8 threads) TGL:

root@DUT025-TGLU:mbrost# cat /proc/cpuinfo
...
processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 140
model name      : 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
stepping        : 1
microcode       : 0x3a
cpu MHz         : 2344.098
cache size      : 12288 KB
physical id     : 0
siblings        : 8
core id         : 3
cpu cores       : 4
...

Enough data to be convinced there is not issue with this design? I can also hack up Xe to use less GPU schedulers /w a kthreads but again that isn't trivial and doesn't seem necessary based on these results.

Quoted 9 lines

It could show you real world behaviour of the kworkers approach and it could also enable you to cross reference any power and performance changes relative to i915. Background story there is that media servers like to fit N streams to a server and if a change comes along which suddenly makes only N-1 stream fit before dropping out of realtime, that's a big problem.

If you will believe me there is value in that kind of testing I am happy to help you add Xe support to the tool, time permitting so possibly guidance only at the moment.

If we want to port the tool I wont stop you and provide support if you struggle with the uAPI but based on my results above I don't think this is necessary.

Matt

Quoted 4 lines

Regards,

Tvrtko

#38 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 1 line · Matthew Brost

On Wed, Jan 11, 2023 at 09:09:45AM +0000, Tvrtko Ursulin wrote:

[snip]

Quoted 18 lines
Quoted 9 lines

Anyway, since you are not buying any arguments on paper perhaps you are more open towards testing. If you would adapt gem_wsim for Xe you would be able to spawn N simulated transcode sessions on any Gen11+ machine and try it out.

For example:

gem_wsim -w benchmarks/wsim/media_load_balance_fhd26u7.wsim -c 36 -r 600

Quoted 8 lines

That will run you 36 parallel transcoding sessions streams for 600 frames each. No client setup needed whatsoever apart from compiling IGT.

In the past that was quite a handy tool to identify scheduling issues, or validate changes against. All workloads with the media prefix have actually been hand crafted by looking at what real media pipelines do with real data. Few years back at least.

Quoted 5 lines

Porting this is non-trivial as this is 2.5k. Also in Xe we are trending to use UMD benchmarks to determine if there are performance problems as in the i915 we had tons microbenchmarks / IGT benchmarks that we found meant absolutely nothing. Can't say if this benchmark falls into that category.

I explained what it does so it was supposed to be obvious it is not a micro benchmark.

2.5k what, lines of code? Difficulty of adding Xe support does not scale with LOC but with how much it uses the kernel API. You'd essentially need to handle context/engine creation and different execbuf.

It's not trivial no, but it would save you downloading gigabytes of test streams, building a bunch of tools and libraries etc, and so overall in my experience it *significantly* improves the driver development turn-around time.

Quoted 42 lines

We VK and compute benchmarks running and haven't found any major issues yet. The media UMD hasn't been ported because of the VM bind dependency so I can't say if there are any issues with the media UMD + Xe.

What I can do hack up xe_exec_threads to really hammer Xe - change it to 128x xe_engines + 8k execs per thread. Each exec is super simple, it just stores a dword. It creates a thread per hardware engine, so on TGL this is 5x threads.

Results below:

root@DUT025-TGLU:mbrost# xe_exec_threads --r threads-basic
IGT-Version: 1.26-ge26de4b2 (x86_64) (Linux: 6.1.0-rc1-xe+ x86_64)
Starting subtest: threads-basic
Subtest threads-basic: SUCCESS (1.215s)
root@DUT025-TGLU:mbrost# dumptrace | grep job | wc
   40960  491520 7401728
root@DUT025-TGLU:mbrost# dumptrace | grep engine | wc
     645    7095   82457

So with 640 xe_engines (5x are VM engines) it takes 1.215 seconds test time to run 40960 execs. That seems to indicate we do not have a scheduling problem.

This is 8 core (or at least 8 threads) TGL:

root@DUT025-TGLU:mbrost# cat /proc/cpuinfo
...
processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 140
model name      : 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
stepping        : 1
microcode       : 0x3a
cpu MHz         : 2344.098
cache size      : 12288 KB
physical id     : 0
siblings        : 8
core id         : 3
cpu cores       : 4
...
Quoted 3 lines

Enough data to be convinced there is not issue with this design? I can also hack up Xe to use less GPU schedulers /w a kthreads but again that isn't trivial and doesn't seem necessary based on these results.

Not yet. It's not only about how many somethings per second you can do. It is also about what effect to the rest of the system it creates.

Anyway I think you said in different sub-thread you will move away from system_wq, so we can close this one. With that plan at least I don't have to worry my mouse will stutter and audio glitch while Xe is churning away.

Regards,

Tvrtko

author maintainer
#39 lore raw

Re: [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

In reply to Boris Brezillon
Quoted 36 lines · Boris Brezillon
Quoted 33 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…
Quoted 29 lines · On Fri, 30 Dec 2022 11:20:42 +0100 Boris Brezillon…

Hello Matthew,

Quoted 11 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem.

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a FW-assisted scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because I felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof. Giving drm_sched_entity exlusive access to a drm_gpu_scheduler probably might help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Quoted 6 lines
Quoted 6 lines

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

FWIW this design was also ran by AMD quite a while ago (off the list) and we didn't get any serious push back. Things can change however...

Quoted 11 lines
Quoted 11 lines

selection is dumb because there's only one entity ever bound to the scheduler (the one that's part of the xe_guc_engine object which also contains the drm_gpu_scheduler instance). I guess the main issue we'd have on Arm is the fact that the stream doesn't necessarily get scheduled when ->run_job() is called, it can be placed in the runnable queue and be picked later by the kernel-side scheduler when a FW slot gets released. That can probably be sorted out by manually disabling the job timer and re-enabling it when the stream gets picked by the scheduler. But my main concern remains, we're basically abusing drm_sched here.

That's a matter of opinion, yes we are using it slightly differently than anyone else but IMO the fact the DRM scheduler works for the Xe use case with barely any changes is a testament to its design.

Quoted 11 lines
Quoted 11 lines

For the Arm driver, that means turning the following sequence

1. wait for job deps
2. queue job to ringbuf and push the stream to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
3. stream gets picked by the timeslice scheduler and sent to the FW for execution

into

Quoted 11 lines
Quoted 11 lines

1. queue job to entity which takes care of waiting for job deps for us
2. schedule a drm_sched_main iteration
3. the only available entity is picked, and the first job from this entity is dequeued. ->run_job() is called: the job is queued to the ringbuf and the stream is pushed to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
4. stream gets picked by the timeslice scheduler and sent to the FW for execution

Yes, an extra step but you get to use all the nice DRM scheduler functions for dependency tracking. Also in our case we really want a single entry point in the backend (the work queue). Also see [1] which helped us seal a bunch of races we had in the i915 by using a single entry point. All these benefits are why we landed on the DRM scheduler and it has worked of rather nicely compared to the i915.

[1] https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1

Quoted 6 lines
Quoted 5 lines

That's one extra step we don't really need. To sum-up, yes, all the job/entity tracking might be interesting to share/re-use, but I wonder if we couldn't have that without pulling out the scheduling part of drm_sched, or maybe I'm missing something, and there's something in drm_gpu_scheduler you really need.

Quoted 7 lines

On second thought, that's probably an acceptable overhead (not even sure the extra step I was mentioning exists in practice, because dep fence signaled state is checked as part of the drm_sched_main iteration, so that's basically replacing the worker I schedule to check job deps), and I like the idea of being able to re-use drm_sched dep-tracking without resorting to invasive changes to the existing logic, so I'll probably give it a try.

Let me know how this goes.

Matt

#40 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

FWIW this design was also ran by AMD quite a while ago (off the list)

Quoted 2 lines

and we didn't get any serious push back. Things can change however...

Yup, AMD and NVIDIA both want this, more-or-less.

Quoted 67 lines
Quoted 7 lines
Quoted 7 lines

selection is dumb because there's only one entity ever bound to the scheduler (the one that's part of the xe_guc_engine object which also contains the drm_gpu_scheduler instance). I guess the main issue we'd have on Arm is the fact that the stream doesn't necessarily get scheduled when ->run_job() is called, it can be placed in the runnable queue and be picked later by the kernel-side scheduler when a FW slot gets released. That can probably be sorted out by manually disabling

the

Quoted 4 lines
Quoted 4 lines

job timer and re-enabling it when the stream gets picked by the scheduler. But my main concern remains, we're basically abusing drm_sched here.

That's a matter of opinion, yes we are using it slightly differently than anyone else but IMO the fact the DRM scheduler works for the Xe use case with barely any changes is a testament to its design.

Quoted 11 lines
Quoted 11 lines

For the Arm driver, that means turning the following sequence

1. wait for job deps
2. queue job to ringbuf and push the stream to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
3. stream gets picked by the timeslice scheduler and sent to the FW for execution

into

Quoted 11 lines
Quoted 11 lines

1. queue job to entity which takes care of waiting for job deps for us
2. schedule a drm_sched_main iteration
3. the only available entity is picked, and the first job from this entity is dequeued. ->run_job() is called: the job is queued to the ringbuf and the stream is pushed to the runnable queue (if it wasn't queued already). Wakeup the timeslice scheduler to re-evaluate (if the stream is not on a FW slot already)
4. stream gets picked by the timeslice scheduler and sent to the FW for execution

Yes, an extra step but you get to use all the nice DRM scheduler functions for dependency tracking. Also in our case we really want a single entry point in the backend (the work queue). Also see [1] which helped us seal a bunch of races we had in the i915 by using a single entry point. All these benefits are why we landed on the DRM scheduler and it has worked of rather nicely compared to the i915.

[1] https://patchwork.freedesktop.org/patch/515857/?series=112189&rev=1

Quoted 6 lines
Quoted 5 lines

That's one extra step we don't really need. To sum-up, yes, all the job/entity tracking might be interesting to share/re-use, but I wonder if we couldn't have that without pulling out the scheduling part of drm_sched, or maybe I'm missing something, and there's something in drm_gpu_scheduler you really need.

Quoted 7 lines

On second thought, that's probably an acceptable overhead (not even sure the extra step I was mentioning exists in practice, because dep fence signaled state is checked as part of the drm_sched_main iteration, so that's basically replacing the worker I schedule to check job deps), and I like the idea of being able to re-use drm_sched dep-tracking without resorting to invasive changes to the existing logic, so I'll probably give it a try.

Let me know how this goes.

Matt

#41 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

Regards,

Boris

#42 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#43 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Hi Daniel,

Quoted 138 lines · On Mon, 9 Jan 2023 21:40:21 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Quoted 8 lines

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Would you mind pointing me to the amdgpu code you're mentioning here? Still have a hard time seeing what TTM has to do with scheduling, but I also don't know much about TTM, so I'll keep digging.

Quoted 5 lines

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.

Honestly, I don't think there's much difference between those two cases already. There's just a bunch of additional code to schedule queues on FW slots for the limited-number-of-FW-slots case, which, right now, is driver specific. The job queuing front-end pretty much achieves what drm_sched does already: queuing job to entities, checking deps, submitting job to HW (in our case, writing to the command stream ring buffer). Things start to differ after that point: once a scheduling entity has pending jobs, we add it to one of the runnable queues (one queue per prio) and kick the kernel-side timeslice-based scheduler to re-evaluate, if needed.

I'm all for using generic code when it makes sense, even if that means adding this common code when it doesn't exists, but I don't want to be dragged into some major refactoring that might take years to land. Especially if pancsf is the first FW-assisted-scheduler-with-few-FW-slot driver.

Here's a link to my WIP branch [1], and here is the scheduler logic
[2] if you want to have a look. Don't pay too much attention to the driver uAPI (it's being redesigned).

Regards,

Boris

[1]https://gitlab.freedesktop.org/bbrezillon/linux/-/tree/pancsf [2]https://gitlab.freedesktop.org/bbrezillon/linux/-/blob/pancsf/drivers/gpu/drm/pancsf/pancsf_sched.c

#44 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 148 lines · Boris Brezillon

Hi Daniel,

Quoted 142 lines · On Mon, 9 Jan 2023 21:40:21 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

Quoted 8 lines
Quoted 4 lines

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Would you mind pointing me to the amdgpu code you're mentioning here? Still have a hard time seeing what TTM has to do with scheduling, but I also don't know much about TTM, so I'll keep digging.

ttm is about moving stuff in&out of a limited space and gives you some nice tooling for pipelining it all. It doesn't care whether that space is vram or some limited id space. vmwgfx used ttm as an id manager iirc.

Quoted 16 lines
Quoted 4 lines

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.

Honestly, I don't think there's much difference between those two cases already. There's just a bunch of additional code to schedule queues on FW slots for the limited-number-of-FW-slots case, which, right now, is driver specific. The job queuing front-end pretty much achieves what drm_sched does already: queuing job to entities, checking deps, submitting job to HW (in our case, writing to the command stream ring buffer). Things start to differ after that point: once a scheduling entity has pending jobs, we add it to one of the runnable queues (one queue per prio) and kick the kernel-side timeslice-based scheduler to re-evaluate, if needed.

Quoted 5 lines

I'm all for using generic code when it makes sense, even if that means adding this common code when it doesn't exists, but I don't want to be dragged into some major refactoring that might take years to land. Especially if pancsf is the first FW-assisted-scheduler-with-few-FW-slot driver.

I don't see where there's a major refactoring that you're getting dragged into?

Yes there's a huge sprawling discussion right now, but I think that's just largely people getting confused.

Wrt the actual id assignment stuff, in amdgpu at least it's few lines of code. See the amdgpu_vmid_grab stuff for the simplest starting point.

And also yes a scheduler frontend for dependency sorting shouldn't really be a that big thing, so there's not going to be huge amounts of code sharing in the end. It's the conceptual sharing, and sharing stuff like drm_sched_entity to eventual build some cross driver gpu context stuff on top that really is going to matter.

Also like I mentioned, at least in some cases i915-guc might also have a need for fw scheduler slot allocation for a bunch of running things.

Finally I'm a bit confused why you're building a time sharing scheduler in the kernel if you have one in fw already. Or do I get that part wrong?
-Daniel

Quoted 10 lines

Here's a link to my WIP branch [1], and here is the scheduler logic
[2] if you want to have a look. Don't pay too much attention to the driver uAPI (it's being redesigned).

Regards,

Boris

[1]https://gitlab.freedesktop.org/bbrezillon/linux/-/tree/pancsf [2]https://gitlab.freedesktop.org/bbrezillon/linux/-/blob/pancsf/drivers/gpu/drm/pancsf/pancsf_sched.c

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#45 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Hi Daniel,

Quoted 155 lines · On Wed, 11 Jan 2023 22:47:02 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 153 lines · Boris Brezillon

Hi Daniel,

Quoted 146 lines · On Mon, 9 Jan 2023 21:40:21 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Quoted 8 lines
Quoted 3 lines

Would you mind pointing me to the amdgpu code you're mentioning here? Still have a hard time seeing what TTM has to do with scheduling, but I also don't know much about TTM, so I'll keep digging.

ttm is about moving stuff in&out of a limited space and gives you some nice tooling for pipelining it all. It doesn't care whether that space is vram or some limited id space. vmwgfx used ttm as an id manager iirc.

Ok.

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.

Honestly, I don't think there's much difference between those two cases already. There's just a bunch of additional code to schedule queues on FW slots for the limited-number-of-FW-slots case, which, right now, is driver specific. The job queuing front-end pretty much achieves what drm_sched does already: queuing job to entities, checking deps, submitting job to HW (in our case, writing to the command stream ring buffer). Things start to differ after that point: once a scheduling entity has pending jobs, we add it to one of the runnable queues (one queue per prio) and kick the kernel-side timeslice-based scheduler to re-evaluate, if needed.

Quoted 7 lines
Quoted 5 lines

I'm all for using generic code when it makes sense, even if that means adding this common code when it doesn't exists, but I don't want to be dragged into some major refactoring that might take years to land. Especially if pancsf is the first FW-assisted-scheduler-with-few-FW-slot driver.

I don't see where there's a major refactoring that you're getting dragged into?

Oh, no, I'm not saying this is the case just yet, just wanted to make sure we're on the same page :-).

Quoted 3 lines

Yes there's a huge sprawling discussion right now, but I think that's just largely people getting confused.

I definitely am :-).

Quoted 4 lines

Wrt the actual id assignment stuff, in amdgpu at least it's few lines of code. See the amdgpu_vmid_grab stuff for the simplest starting point.

Ok, thanks for the pointers. I'll have a look and see how I could use that. I guess that's about getting access to the FW slots with some sort of priority+FIFO ordering guarantees given by TTM. If that's the case, I'll have to think about it, because that's a major shift from what we're doing now, and I'm afraid this could lead to starving non-resident entities if all resident entities keep receiving new jobs to execute. Unless we put some sort of barrier when giving access to a slot, so we evict the entity when it's done executing the stuff it had when it was given access to this slot. But then, again, there are other constraints to take into account for the Arm Mali CSF case:

- it's more efficient to update all FW slots at once, because each update of a slot might require updating priorities of the other slots (FW mandates unique slot priorities, and those priorities depend on the entity priority/queue-ordering)
- context/FW slot switches have a non-negligible cost (FW needs to suspend the context and save the state every time there such a switch), so, limiting the number of FW slot updates might prove important

Quoted 4 lines

And also yes a scheduler frontend for dependency sorting shouldn't really be a that big thing, so there's not going to be huge amounts of code sharing in the end.

Agreed.

Quoted 3 lines

It's the conceptual sharing, and sharing
stuff like drm_sched_entity to eventual build some cross driver gpu context stuff on top that really is going to matter.

And I agree with that too.

Quoted 3 lines

Also like I mentioned, at least in some cases i915-guc might also have a need for fw scheduler slot allocation for a bunch of running things.

Ok.

Quoted 4 lines

Finally I'm a bit confused why you're building a time sharing scheduler in the kernel if you have one in fw already. Or do I get that part wrong?

It's here to overcome the low number of FW-slot (which is as low as 8 on the HW I'm testing on). If you don't do time sharing scheduling kernel-side, you have no guarantee of fairness, since one could keep queuing jobs to an entity/queue, making it permanently resident, without giving a chance to non-resident entities/queues to ever run. To sum-up, the scheduler is not entirely handled by the FW, it's a mixed design, where part of it is in the FW (scheduling between currently active entities passed to the FW), and the other part in the kernel driver (rotating runnable entities on the limited amount of FW slots we have). But overall, it shouldn't make a difference compared to Xe. The fact some of the scheduling happens kernel-side is completely opaque to the drm_sched_entity frontend if we go the Xe way (one drm_gpu_scheduler per drm_sched_entity, real scheduling is handled by some black box, either entirely in the FW, or with shared responsibility between FW and kernel).

Regards,

Boris

#46 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 221 lines · Boris Brezillon

Hi Daniel,

Quoted 155 lines · On Wed, 11 Jan 2023 22:47:02 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 153 lines · Boris Brezillon

Hi Daniel,

Quoted 146 lines · On Mon, 9 Jan 2023 21:40:21 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Quoted 8 lines
Quoted 3 lines

Would you mind pointing me to the amdgpu code you're mentioning here? Still have a hard time seeing what TTM has to do with scheduling, but I also don't know much about TTM, so I'll keep digging.

ttm is about moving stuff in&out of a limited space and gives you some nice tooling for pipelining it all. It doesn't care whether that space is vram or some limited id space. vmwgfx used ttm as an id manager iirc.

Ok.

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.

Honestly, I don't think there's much difference between those two cases already. There's just a bunch of additional code to schedule queues on FW slots for the limited-number-of-FW-slots case, which, right now, is driver specific. The job queuing front-end pretty much achieves what drm_sched does already: queuing job to entities, checking deps, submitting job to HW (in our case, writing to the command stream ring buffer). Things start to differ after that point: once a scheduling entity has pending jobs, we add it to one of the runnable queues (one queue per prio) and kick the kernel-side timeslice-based scheduler to re-evaluate, if needed.

Quoted 7 lines
Quoted 5 lines

I'm all for using generic code when it makes sense, even if that means adding this common code when it doesn't exists, but I don't want to be dragged into some major refactoring that might take years to land. Especially if pancsf is the first FW-assisted-scheduler-with-few-FW-slot driver.

I don't see where there's a major refactoring that you're getting dragged into?

Oh, no, I'm not saying this is the case just yet, just wanted to make sure we're on the same page :-).

Quoted 3 lines

Yes there's a huge sprawling discussion right now, but I think that's just largely people getting confused.

I definitely am :-).

Quoted 4 lines

Wrt the actual id assignment stuff, in amdgpu at least it's few lines of code. See the amdgpu_vmid_grab stuff for the simplest starting point.

Ok, thanks for the pointers. I'll have a look and see how I could use that. I guess that's about getting access to the FW slots with some sort of priority+FIFO ordering guarantees given by TTM. If that's the case, I'll have to think about it, because that's a major shift from what we're doing now, and I'm afraid this could lead to starving non-resident entities if all resident entities keep receiving new jobs to execute. Unless we put some sort of barrier when giving access to a slot, so we evict the entity when it's done executing the stuff it had when it was given access to this slot. But then, again, there are other constraints to take into account for the Arm Mali CSF case:

Quoted 8 lines

- it's more efficient to update all FW slots at once, because each update of a slot might require updating priorities of the other slots (FW mandates unique slot priorities, and those priorities depend on the entity priority/queue-ordering)
- context/FW slot switches have a non-negligible cost (FW needs to suspend the context and save the state every time there such a switch), so, limiting the number of FW slot updates might prove important

I frankly think you're overworrying. When you have 31+ contexts running at the same time, you have bigger problems. At that point there's two use-cases:
1. system is overloaded, the user will reach for reset button anyway
2. temporary situation, all you have to do is be roughly fair enough to get through it before case 1 happens.

Trying to write a perfect scheduler for this before we have actual benchmarks that justify the effort seems like pretty serious overkill. That's why I think the simplest solution is the one we should have:

- drm/sched frontend. If you get into slot exhaustion that alone will ensure enough fairness

- LRU list of slots, with dma_fence so you can pipeline/batch up changes as needed (but I honestly wouldn't worry about the batching before you've shown an actual need for this in some benchmark/workload, even piglit shouldn't have this many things running concurrently I think, you don't have that many cpu cores). Between drm/sched and the lru you will have an emergent scheduler that cycles through all runnable gpu jobs.

- If you want to go fancy, have eviction tricks like skipping currently still active gpu context with higher priority than the one that you need to find a slot for.

- You don't need time slicing in this, not even for compute. compute is done with preempt context fences, if you give them a minimum scheduling quanta you'll have a very basic round robin scheduler as an emergent thing.

Any workload were it matters will be scheduled by the fw directly, with drm/sched only being the dma_fence dependcy sorter. My take is that if you spend more than a hundred or so lines with slot allocation logic (excluding the hw code to load/unload a slot) you're probably doing some serious overengineering.

Quoted 23 lines
Quoted 3 lines

And also yes a scheduler frontend for dependency sorting shouldn't really be a that big thing, so there's not going to be huge amounts of code sharing in the end.

Agreed.

Quoted 3 lines

It's the conceptual sharing, and sharing
stuff like drm_sched_entity to eventual build some cross driver gpu context stuff on top that really is going to matter.

And I agree with that too.

Quoted 3 lines

Also like I mentioned, at least in some cases i915-guc might also have a need for fw scheduler slot allocation for a bunch of running things.

Ok.

Quoted 4 lines

Finally I'm a bit confused why you're building a time sharing scheduler in the kernel if you have one in fw already. Or do I get that part wrong?

Quoted 15 lines

It's here to overcome the low number of FW-slot (which is as low as 8 on the HW I'm testing on). If you don't do time sharing scheduling kernel-side, you have no guarantee of fairness, since one could keep queuing jobs to an entity/queue, making it permanently resident, without giving a chance to non-resident entities/queues to ever run. To sum-up, the scheduler is not entirely handled by the FW, it's a mixed design, where part of it is in the FW (scheduling between currently active entities passed to the FW), and the other part in the kernel driver (rotating runnable entities on the limited amount of FW slots we have). But overall, it shouldn't make a difference compared to Xe. The fact some of the scheduling happens kernel-side is completely opaque to the drm_sched_entity frontend if we go the Xe way (one drm_gpu_scheduler per drm_sched_entity, real scheduling is handled by some black box, either entirely in the FW, or with shared responsibility between FW and kernel).

See above. I don't think you need three scheduler (dma_fence sorting frontend, kernel round robin, fw round robin here). I'm pretty sure you do not _want_ 3 schedulers. And if you just take the 3 pieces above, you will have a scheduler that's Fair Enough (tm) even when you have more than 31 context.

I would frankly not even be surprised if you can get away with full stalls, so not even the dma_fence pipelining needed. even if you stall out a handful of context, there should still be 20+ available for the fw to schedule and keep the gpu busy. After all, this is still a gpu, there's only 2 things you need:
- fair enough to avoid completely stalling out some app and the user reaching the reset button
- throughput. as long as you can keep enough runnable slots for the fw to schedule, it really shouldn't matter how shoddily you push in new stuff.

Cheers, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#47 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 238 lines · On Thu, 12 Jan 2023 10:32:18 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 221 lines · Boris Brezillon

Hi Daniel,

Quoted 155 lines · On Wed, 11 Jan 2023 22:47:02 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 153 lines · Boris Brezillon

Hi Daniel,

Quoted 146 lines · On Mon, 9 Jan 2023 21:40:21 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Quoted 8 lines
Quoted 3 lines

Would you mind pointing me to the amdgpu code you're mentioning here? Still have a hard time seeing what TTM has to do with scheduling, but I also don't know much about TTM, so I'll keep digging.

ttm is about moving stuff in&out of a limited space and gives you some nice tooling for pipelining it all. It doesn't care whether that space is vram or some limited id space. vmwgfx used ttm as an id manager iirc.

Ok.

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.

Honestly, I don't think there's much difference between those two cases already. There's just a bunch of additional code to schedule queues on FW slots for the limited-number-of-FW-slots case, which, right now, is driver specific. The job queuing front-end pretty much achieves what drm_sched does already: queuing job to entities, checking deps, submitting job to HW (in our case, writing to the command stream ring buffer). Things start to differ after that point: once a scheduling entity has pending jobs, we add it to one of the runnable queues (one queue per prio) and kick the kernel-side timeslice-based scheduler to re-evaluate, if needed.

Quoted 7 lines
Quoted 5 lines

I'm all for using generic code when it makes sense, even if that means adding this common code when it doesn't exists, but I don't want to be dragged into some major refactoring that might take years to land. Especially if pancsf is the first FW-assisted-scheduler-with-few-FW-slot driver.

I don't see where there's a major refactoring that you're getting dragged into?

Oh, no, I'm not saying this is the case just yet, just wanted to make sure we're on the same page :-).

Quoted 3 lines

Yes there's a huge sprawling discussion right now, but I think that's just largely people getting confused.

I definitely am :-).

Quoted 4 lines

Wrt the actual id assignment stuff, in amdgpu at least it's few lines of code. See the amdgpu_vmid_grab stuff for the simplest starting point.

Ok, thanks for the pointers. I'll have a look and see how I could use that. I guess that's about getting access to the FW slots with some sort of priority+FIFO ordering guarantees given by TTM. If that's the case, I'll have to think about it, because that's a major shift from what we're doing now, and I'm afraid this could lead to starving non-resident entities if all resident entities keep receiving new jobs to execute. Unless we put some sort of barrier when giving access to a slot, so we evict the entity when it's done executing the stuff it had when it was given access to this slot. But then, again, there are other constraints to take into account for the Arm Mali CSF case:

Quoted 8 lines

- it's more efficient to update all FW slots at once, because each update of a slot might require updating priorities of the other slots (FW mandates unique slot priorities, and those priorities depend on the entity priority/queue-ordering)
- context/FW slot switches have a non-negligible cost (FW needs to suspend the context and save the state every time there such a switch), so, limiting the number of FW slot updates might prove important

I frankly think you're overworrying. When you have 31+ contexts running at the same time, you have bigger problems. At that point there's two use-cases:
1. system is overloaded, the user will reach for reset button anyway
2. temporary situation, all you have to do is be roughly fair enough to get through it before case 1 happens.

Quoted 6 lines

Trying to write a perfect scheduler for this before we have actual benchmarks that justify the effort seems like pretty serious overkill. That's why I think the simplest solution is the one we should have:

- drm/sched frontend. If you get into slot exhaustion that alone will ensure enough fairness

We're talking about the CS ring buffer slots here, right?

Quoted 17 lines

- LRU list of slots, with dma_fence so you can pipeline/batch up changes as needed (but I honestly wouldn't worry about the batching before you've shown an actual need for this in some benchmark/workload, even piglit shouldn't have this many things running concurrently I think, you don't have that many cpu cores). Between drm/sched and the lru you will have an emergent scheduler that cycles through all runnable gpu jobs.

- If you want to go fancy, have eviction tricks like skipping currently still active gpu context with higher priority than the one that you need to find a slot for.

- You don't need time slicing in this, not even for compute. compute is done with preempt context fences, if you give them a minimum scheduling quanta you'll have a very basic round robin scheduler as an emergent thing.

Quoted 5 lines

Any workload were it matters will be scheduled by the fw directly, with drm/sched only being the dma_fence dependcy sorter. My take is that if you spend more than a hundred or so lines with slot allocation logic (excluding the hw code to load/unload a slot) you're probably doing some serious overengineering.

Let me see if I got this right:

- we still keep a 1:1 drm_gpu_scheduler:drm_sched_entity approach, where hw_submission_limit == available_slots_in_ring_buf
- when ->run_job() is called, we write the RUN_JOB() instruction sequence to the next available ringbuf slot and queue the entity to the FW-slot queue
* if a slot is directly available, we program the slot directly
* if no slots are available, but some slots are done with the jobs they were given (last job fence signaled), we evict the LRU entity (possibly taking priority into account) and use this slot for the new entity
* if no slots are available and all currently assigned slots contain busy entities, we queue the entity to a pending list (possibly one list per prio)

I'll need to make sure this still works with the concept of group (it's not a single queue we schedule, it's a group of queues, meaning that we have N fences to watch to determine if the slot is busy or not, but that should be okay).

#48 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 289 lines · On Thu, 12 Jan 2023 11:11:03 +0100 Boris Brezillon…
Quoted 238 lines · On Thu, 12 Jan 2023 10:32:18 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 221 lines · Boris Brezillon

Hi Daniel,

Quoted 155 lines · On Wed, 11 Jan 2023 22:47:02 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 153 lines · Boris Brezillon

Hi Daniel,

Quoted 146 lines · On Mon, 9 Jan 2023 21:40:21 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1
mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not
guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Quoted 8 lines
Quoted 3 lines

Would you mind pointing me to the amdgpu code you're mentioning here? Still have a hard time seeing what TTM has to do with scheduling, but I also don't know much about TTM, so I'll keep digging.

ttm is about moving stuff in&out of a limited space and gives you some nice tooling for pipelining it all. It doesn't care whether that space is vram or some limited id space. vmwgfx used ttm as an id manager iirc.

Ok.

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.

Honestly, I don't think there's much difference between those two cases already. There's just a bunch of additional code to schedule queues on FW slots for the limited-number-of-FW-slots case, which, right now, is driver specific. The job queuing front-end pretty much achieves what drm_sched does already: queuing job to entities, checking deps, submitting job to HW (in our case, writing to the command stream ring buffer). Things start to differ after that point: once a scheduling entity has pending jobs, we add it to one of the runnable queues (one queue per prio) and kick the kernel-side timeslice-based scheduler to re-evaluate, if needed.

Quoted 7 lines
Quoted 5 lines

I'm all for using generic code when it makes sense, even if that means adding this common code when it doesn't exists, but I don't want to be dragged into some major refactoring that might take years to land. Especially if pancsf is the first FW-assisted-scheduler-with-few-FW-slot driver.

I don't see where there's a major refactoring that you're getting dragged into?

Oh, no, I'm not saying this is the case just yet, just wanted to make sure we're on the same page :-).

Quoted 3 lines

Yes there's a huge sprawling discussion right now, but I think that's just largely people getting confused.

I definitely am :-).

Quoted 4 lines

Wrt the actual id assignment stuff, in amdgpu at least it's few lines of code. See the amdgpu_vmid_grab stuff for the simplest starting point.

Ok, thanks for the pointers. I'll have a look and see how I could use that. I guess that's about getting access to the FW slots with some sort of priority+FIFO ordering guarantees given by TTM. If that's the case, I'll have to think about it, because that's a major shift from what we're doing now, and I'm afraid this could lead to starving non-resident entities if all resident entities keep receiving new jobs to execute. Unless we put some sort of barrier when giving access to a slot, so we evict the entity when it's done executing the stuff it had when it was given access to this slot. But then, again, there are other constraints to take into account for the Arm Mali CSF case:

Quoted 8 lines

- it's more efficient to update all FW slots at once, because each update of a slot might require updating priorities of the other slots (FW mandates unique slot priorities, and those priorities depend on the entity priority/queue-ordering)
- context/FW slot switches have a non-negligible cost (FW needs to suspend the context and save the state every time there such a switch), so, limiting the number of FW slot updates might prove important

I frankly think you're overworrying. When you have 31+ contexts running at the same time, you have bigger problems. At that point there's two use-cases:
1. system is overloaded, the user will reach for reset button anyway
2. temporary situation, all you have to do is be roughly fair enough to get through it before case 1 happens.

Quoted 6 lines

Trying to write a perfect scheduler for this before we have actual benchmarks that justify the effort seems like pretty serious overkill. That's why I think the simplest solution is the one we should have:

- drm/sched frontend. If you get into slot exhaustion that alone will ensure enough fairness

We're talking about the CS ring buffer slots here, right?

Quoted 17 lines

- LRU list of slots, with dma_fence so you can pipeline/batch up changes as needed (but I honestly wouldn't worry about the batching before you've shown an actual need for this in some benchmark/workload, even piglit shouldn't have this many things running concurrently I think, you don't have that many cpu cores). Between drm/sched and the lru you will have an emergent scheduler that cycles through all runnable gpu jobs.

- If you want to go fancy, have eviction tricks like skipping currently still active gpu context with higher priority than the one that you need to find a slot for.

- You don't need time slicing in this, not even for compute. compute is done with preempt context fences, if you give them a minimum scheduling quanta you'll have a very basic round robin scheduler as an emergent thing.

Quoted 5 lines

Any workload were it matters will be scheduled by the fw directly, with drm/sched only being the dma_fence dependcy sorter. My take is that if you spend more than a hundred or so lines with slot allocation logic (excluding the hw code to load/unload a slot) you're probably doing some serious overengineering.

Let me see if I got this right:

- we still keep a 1:1 drm_gpu_scheduler:drm_sched_entity approach, where hw_submission_limit == available_slots_in_ring_buf
- when ->run_job() is called, we write the RUN_JOB() instruction sequence to the next available ringbuf slot and queue the entity to the FW-slot queue
* if a slot is directly available, we program the slot directly
* if no slots are available, but some slots are done with the jobs they were given (last job fence signaled), we evict the LRU entity (possibly taking priority into account) and use this slot for the new entity
* if no slots are available and all currently assigned slots contain busy entities, we queue the entity to a pending list (possibly one list per prio)

Quoted 4 lines

I'll need to make sure this still works with the concept of group (it's not a single queue we schedule, it's a group of queues, meaning that we have N fences to watch to determine if the slot is busy or not, but that should be okay).

Oh, there's one other thing I forgot to mention: the FW scheduler is not entirely fair, it does take the slot priority (which has to be unique across all currently assigned slots) into account when scheduling groups. So, ideally, we'd want to rotate group priorities when they share the same drm_sched_priority (probably based on the position in the LRU).

#49 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 278 lines · Boris Brezillon
Quoted 275 lines · On Thu, 12 Jan 2023 11:11:03 +0100 Boris Brezillon…
Quoted 238 lines · On Thu, 12 Jan 2023 10:32:18 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 221 lines · Boris Brezillon

Hi Daniel,

Quoted 155 lines · On Wed, 11 Jan 2023 22:47:02 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 153 lines · Boris Brezillon

Hi Daniel,

Quoted 146 lines · On Mon, 9 Jan 2023 21:40:21 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

On Thu, 22 Dec 2022 14:21:11 -0800
Matthew Brost <[email protected]> wrote:

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1
mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not
guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of
issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does
the scheduling between those N command streams, the kernel driver
does timeslice scheduling to update the command streams passed to the
FW). I must admit I gave up on using drm_sched at some point, mostly
because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a
timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's
coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Quoted 8 lines
Quoted 3 lines

Would you mind pointing me to the amdgpu code you're mentioning here? Still have a hard time seeing what TTM has to do with scheduling, but I also don't know much about TTM, so I'll keep digging.

ttm is about moving stuff in&out of a limited space and gives you some nice tooling for pipelining it all. It doesn't care whether that space is vram or some limited id space. vmwgfx used ttm as an id manager iirc.

Ok.

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.

Honestly, I don't think there's much difference between those two cases already. There's just a bunch of additional code to schedule queues on FW slots for the limited-number-of-FW-slots case, which, right now, is driver specific. The job queuing front-end pretty much achieves what drm_sched does already: queuing job to entities, checking deps, submitting job to HW (in our case, writing to the command stream ring buffer). Things start to differ after that point: once a scheduling entity has pending jobs, we add it to one of the runnable queues (one queue per prio) and kick the kernel-side timeslice-based scheduler to re-evaluate, if needed.

Quoted 7 lines
Quoted 5 lines

I'm all for using generic code when it makes sense, even if that means adding this common code when it doesn't exists, but I don't want to be dragged into some major refactoring that might take years to land. Especially if pancsf is the first FW-assisted-scheduler-with-few-FW-slot driver.

I don't see where there's a major refactoring that you're getting dragged into?

Oh, no, I'm not saying this is the case just yet, just wanted to make sure we're on the same page :-).

Quoted 3 lines

Yes there's a huge sprawling discussion right now, but I think that's just largely people getting confused.

I definitely am :-).

Quoted 4 lines

Wrt the actual id assignment stuff, in amdgpu at least it's few lines of code. See the amdgpu_vmid_grab stuff for the simplest starting point.

Ok, thanks for the pointers. I'll have a look and see how I could use that. I guess that's about getting access to the FW slots with some sort of priority+FIFO ordering guarantees given by TTM. If that's the case, I'll have to think about it, because that's a major shift from what we're doing now, and I'm afraid this could lead to starving non-resident entities if all resident entities keep receiving new jobs to execute. Unless we put some sort of barrier when giving access to a slot, so we evict the entity when it's done executing the stuff it had when it was given access to this slot. But then, again, there are other constraints to take into account for the Arm Mali CSF case:

Quoted 8 lines

- it's more efficient to update all FW slots at once, because each update of a slot might require updating priorities of the other slots (FW mandates unique slot priorities, and those priorities depend on the entity priority/queue-ordering)
- context/FW slot switches have a non-negligible cost (FW needs to suspend the context and save the state every time there such a switch), so, limiting the number of FW slot updates might prove important

I frankly think you're overworrying. When you have 31+ contexts running at the same time, you have bigger problems. At that point there's two use-cases:
1. system is overloaded, the user will reach for reset button anyway
2. temporary situation, all you have to do is be roughly fair enough to get through it before case 1 happens.

Quoted 6 lines

Trying to write a perfect scheduler for this before we have actual benchmarks that justify the effort seems like pretty serious overkill. That's why I think the simplest solution is the one we should have:

- drm/sched frontend. If you get into slot exhaustion that alone will ensure enough fairness

We're talking about the CS ring buffer slots here, right?

Quoted 17 lines

- LRU list of slots, with dma_fence so you can pipeline/batch up changes as needed (but I honestly wouldn't worry about the batching before you've shown an actual need for this in some benchmark/workload, even piglit shouldn't have this many things running concurrently I think, you don't have that many cpu cores). Between drm/sched and the lru you will have an emergent scheduler that cycles through all runnable gpu jobs.

- If you want to go fancy, have eviction tricks like skipping currently still active gpu context with higher priority than the one that you need to find a slot for.

- You don't need time slicing in this, not even for compute. compute is done with preempt context fences, if you give them a minimum scheduling quanta you'll have a very basic round robin scheduler as an emergent thing.

Quoted 5 lines

Any workload were it matters will be scheduled by the fw directly, with drm/sched only being the dma_fence dependcy sorter. My take is that if you spend more than a hundred or so lines with slot allocation logic (excluding the hw code to load/unload a slot) you're probably doing some serious overengineering.

Let me see if I got this right:

Quoted 13 lines
Quoted 5 lines

- we still keep a 1:1 drm_gpu_scheduler:drm_sched_entity approach, where hw_submission_limit == available_slots_in_ring_buf
- when ->run_job() is called, we write the RUN_JOB() instruction sequence to the next available ringbuf slot and queue the entity to the FW-slot queue

Quoted 8 lines

* if a slot is directly available, we program the slot directly
* if no slots are available, but some slots are done with the jobs they were given (last job fence signaled), we evict the LRU entity (possibly taking priority into account) and use this slot for the new entity
* if no slots are available and all currently assigned slots contain busy entities, we queue the entity to a pending list (possibly one list per prio)

You could also handle this in ->prepare_job, which is called after all the default fences have signalled. That allows you to put the "wait for a previous job to finnish/unload" behind a dma_fence, which is how (I think at least) you can get the round-robin emergent behaviour: If there's no idle slot, you just pick all the fences from the currently busy job you want to steal the slot from (with priority and lru taken into account), let the scheduler wait for that to finnish, and then it'll call your run_job when the slot is already available.

Also if you do the allocation in ->prepare_job with dma_fence and not run_job, then I think can sort out fairness issues (if they do pop up) in the drm/sched code instead of having to think about this in each driver. Few fw sched slots essentially just make fw scheduling unfairness more prominent than with others, but I don't think it's fundamentally something else really.

If every ctx does that and the lru isn't too busted, they should then form a nice orderly queue and cycle through the fw scheduler, while still being able to get some work done. It's essentially the exact same thing that happens with ttm vram eviction, when you have a total working set where each process fits in vram individually, but in total they're too big and you need to cycle things through.

Quoted 11 lines
Quoted 4 lines

I'll need to make sure this still works with the concept of group (it's not a single queue we schedule, it's a group of queues, meaning that we have N fences to watch to determine if the slot is busy or not, but that should be okay).

Oh, there's one other thing I forgot to mention: the FW scheduler is not entirely fair, it does take the slot priority (which has to be unique across all currently assigned slots) into account when scheduling groups. So, ideally, we'd want to rotate group priorities when they share the same drm_sched_priority (probably based on the position in the LRU).

Hm that will make things a bit more fun I guess, especially with your constraint to not update this too often. How strict is that priority difference? If it's a lot, we might need to treat this more like execlist and less like a real fw scheduler ...
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#50 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 293 lines · On Thu, 12 Jan 2023 11:42:57 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 278 lines · Boris Brezillon
Quoted 275 lines · On Thu, 12 Jan 2023 11:11:03 +0100 Boris Brezillon…
Quoted 238 lines · On Thu, 12 Jan 2023 10:32:18 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 221 lines · Boris Brezillon

Hi Daniel,

Quoted 155 lines · On Wed, 11 Jan 2023 22:47:02 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 153 lines · Boris Brezillon

Hi Daniel,

Quoted 146 lines · On Mon, 9 Jan 2023 21:40:21 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines

Hello Matthew,

On Thu, 22 Dec 2022 14:21:11 -0800
Matthew Brost <[email protected]> wrote:

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1
mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not
guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of
issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does
the scheduling between those N command streams, the kernel driver
does timeslice scheduling to update the command streams passed to the
FW). I must admit I gave up on using drm_sched at some point, mostly
because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a
timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines

help for a few things (didn't think it through yet), but I feel it's
coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it
instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I
have a better understanding of how you get away with using drm_sched
while still controlling how scheduling is really done. Here
drm_gpu_scheduler is just a dummy abstract that let's you use the
drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Quoted 8 lines
Quoted 3 lines

Would you mind pointing me to the amdgpu code you're mentioning here? Still have a hard time seeing what TTM has to do with scheduling, but I also don't know much about TTM, so I'll keep digging.

ttm is about moving stuff in&out of a limited space and gives you some nice tooling for pipelining it all. It doesn't care whether that space is vram or some limited id space. vmwgfx used ttm as an id manager iirc.

Ok.

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.

Honestly, I don't think there's much difference between those two cases already. There's just a bunch of additional code to schedule queues on FW slots for the limited-number-of-FW-slots case, which, right now, is driver specific. The job queuing front-end pretty much achieves what drm_sched does already: queuing job to entities, checking deps, submitting job to HW (in our case, writing to the command stream ring buffer). Things start to differ after that point: once a scheduling entity has pending jobs, we add it to one of the runnable queues (one queue per prio) and kick the kernel-side timeslice-based scheduler to re-evaluate, if needed.

Quoted 7 lines
Quoted 5 lines

I'm all for using generic code when it makes sense, even if that means adding this common code when it doesn't exists, but I don't want to be dragged into some major refactoring that might take years to land. Especially if pancsf is the first FW-assisted-scheduler-with-few-FW-slot driver.

I don't see where there's a major refactoring that you're getting dragged into?

Oh, no, I'm not saying this is the case just yet, just wanted to make sure we're on the same page :-).

Quoted 3 lines

Yes there's a huge sprawling discussion right now, but I think that's just largely people getting confused.

I definitely am :-).

Quoted 4 lines

Wrt the actual id assignment stuff, in amdgpu at least it's few lines of code. See the amdgpu_vmid_grab stuff for the simplest starting point.

Ok, thanks for the pointers. I'll have a look and see how I could use that. I guess that's about getting access to the FW slots with some sort of priority+FIFO ordering guarantees given by TTM. If that's the case, I'll have to think about it, because that's a major shift from what we're doing now, and I'm afraid this could lead to starving non-resident entities if all resident entities keep receiving new jobs to execute. Unless we put some sort of barrier when giving access to a slot, so we evict the entity when it's done executing the stuff it had when it was given access to this slot. But then, again, there are other constraints to take into account for the Arm Mali CSF case:

Quoted 8 lines

- it's more efficient to update all FW slots at once, because each update of a slot might require updating priorities of the other slots (FW mandates unique slot priorities, and those priorities depend on the entity priority/queue-ordering)
- context/FW slot switches have a non-negligible cost (FW needs to suspend the context and save the state every time there such a switch), so, limiting the number of FW slot updates might prove important

I frankly think you're overworrying. When you have 31+ contexts running at the same time, you have bigger problems. At that point there's two use-cases:
1. system is overloaded, the user will reach for reset button anyway
2. temporary situation, all you have to do is be roughly fair enough to get through it before case 1 happens.

Quoted 6 lines

Trying to write a perfect scheduler for this before we have actual benchmarks that justify the effort seems like pretty serious overkill. That's why I think the simplest solution is the one we should have:

- drm/sched frontend. If you get into slot exhaustion that alone will ensure enough fairness

We're talking about the CS ring buffer slots here, right?

Quoted 17 lines

- LRU list of slots, with dma_fence so you can pipeline/batch up changes as needed (but I honestly wouldn't worry about the batching before you've shown an actual need for this in some benchmark/workload, even piglit shouldn't have this many things running concurrently I think, you don't have that many cpu cores). Between drm/sched and the lru you will have an emergent scheduler that cycles through all runnable gpu jobs.

- If you want to go fancy, have eviction tricks like skipping currently still active gpu context with higher priority than the one that you need to find a slot for.

- You don't need time slicing in this, not even for compute. compute is done with preempt context fences, if you give them a minimum scheduling quanta you'll have a very basic round robin scheduler as an emergent thing.

Quoted 5 lines

Any workload were it matters will be scheduled by the fw directly, with drm/sched only being the dma_fence dependcy sorter. My take is that if you spend more than a hundred or so lines with slot allocation logic (excluding the hw code to load/unload a slot) you're probably doing some serious overengineering.

Let me see if I got this right:

Quoted 13 lines
Quoted 5 lines

- we still keep a 1:1 drm_gpu_scheduler:drm_sched_entity approach, where hw_submission_limit == available_slots_in_ring_buf
- when ->run_job() is called, we write the RUN_JOB() instruction sequence to the next available ringbuf slot and queue the entity to the FW-slot queue

Quoted 8 lines

* if a slot is directly available, we program the slot directly
* if no slots are available, but some slots are done with the jobs they were given (last job fence signaled), we evict the LRU entity (possibly taking priority into account) and use this slot for the new entity
* if no slots are available and all currently assigned slots contain busy entities, we queue the entity to a pending list (possibly one list per prio)

Quoted 8 lines

You could also handle this in ->prepare_job, which is called after all the default fences have signalled. That allows you to put the "wait for a previous job to finnish/unload" behind a dma_fence, which is how (I think at least) you can get the round-robin emergent behaviour: If there's no idle slot, you just pick all the fences from the currently busy job you want to steal the slot from (with priority and lru taken into account), let the scheduler wait for that to finnish, and then it'll call your run_job when the slot is already available.

Ah, nice! It would also avoid queuing new jobs to a resident entity when others are waiting for a FW slot, even if, in practice, I'm not sure we should do that: context will be suspended when the group is evicted anyway, and things could keep running in the meantime. I'll give it a try, thanks for the suggestion!

Quoted 4 lines

Also if you do the allocation in ->prepare_job with dma_fence and not run_job, then I think can sort out fairness issues (if they do pop up) in the drm/sched code instead of having to think about this in each driver.

By allocation, you mean assigning a FW slot ID? If we do this allocation in ->prepare_job(), couldn't we mess up ordering? Like, lower-prio/later-queuing entity being scheduled before its pairs, because there's no guarantee on the job completion order (and thus the queue idleness order). I mean, completion order depends on the kind of job being executed by the queues, the time the FW actually lets the queue execute things and probably other factors. You can use metrics like the position in the LRU list + the amount of jobs currently queued to a group to guess which one will be idle first, but that's just a guess. And I'm not sure I see what doing this slot selection in ->prepare_job() would bring us compared to doing it in ->run_job(), where we can just pick the least recently used slot.

Quoted 10 lines

Few fw sched slots essentially just make fw scheduling unfairness more prominent than with others, but I don't think it's fundamentally something else really.

If every ctx does that and the lru isn't too busted, they should then form a nice orderly queue and cycle through the fw scheduler, while still being able to get some work done. It's essentially the exact same thing that happens with ttm vram eviction, when you have a total working set where each process fits in vram individually, but in total they're too big and you need to cycle things through.

I see.

Quoted 13 lines
Quoted 6 lines
Quoted 5 lines

I'll need to make sure this still works with the concept of group (it's not a single queue we schedule, it's a group of queues, meaning that we have N fences to watch to determine if the slot is busy or not, but that should be okay).

Quoted 6 lines

Oh, there's one other thing I forgot to mention: the FW scheduler is not entirely fair, it does take the slot priority (which has to be unique across all currently assigned slots) into account when scheduling groups. So, ideally, we'd want to rotate group priorities when they share the same drm_sched_priority (probably based on the position in the LRU).

Quoted 4 lines

Hm that will make things a bit more fun I guess, especially with your constraint to not update this too often. How strict is that priority difference? If it's a lot, we might need to treat this more like execlist and less like a real fw scheduler ...

Strict as in, if two groups with same priority try to request an overlapping set of resources (cores or tilers), it can deadlock, so pretty strict I would say :-).

#51 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 316 lines · Boris Brezillon
Quoted 293 lines · On Thu, 12 Jan 2023 11:42:57 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 278 lines · Boris Brezillon
Quoted 275 lines · On Thu, 12 Jan 2023 11:11:03 +0100 Boris Brezillon…
Quoted 238 lines · On Thu, 12 Jan 2023 10:32:18 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 221 lines · Boris Brezillon

Hi Daniel,

Quoted 155 lines · On Wed, 11 Jan 2023 22:47:02 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 153 lines · Boris Brezillon

Hi Daniel,

Quoted 146 lines · On Mon, 9 Jan 2023 21:40:21 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 13 lines · Boris Brezillon

On Fri, 30 Dec 2022 12:55:08 +0100
Boris Brezillon <[email protected]> wrote:

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Hello Matthew,

On Thu, 22 Dec 2022 14:21:11 -0800
Matthew Brost <[email protected]> wrote:

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1
mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not
guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of
issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does
the scheduling between those N command streams, the kernel driver
does timeslice scheduling to update the command streams passed to the
FW). I must admit I gave up on using drm_sched at some point, mostly
because the integration with drm_sched was painful, but also because

I

Quoted 2 lines

felt trying to bend drm_sched to make it interact with a
timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines

help for a few things (didn't think it through yet), but I feel it's
coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it
instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I
have a better understanding of how you get away with using drm_sched
while still controlling how scheduling is really done. Here
drm_gpu_scheduler is just a dummy abstract that let's you use the
drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Quoted 8 lines
Quoted 3 lines

Would you mind pointing me to the amdgpu code you're mentioning here? Still have a hard time seeing what TTM has to do with scheduling, but I also don't know much about TTM, so I'll keep digging.

ttm is about moving stuff in&out of a limited space and gives you some nice tooling for pipelining it all. It doesn't care whether that space is vram or some limited id space. vmwgfx used ttm as an id manager iirc.

Ok.

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.

Honestly, I don't think there's much difference between those two cases already. There's just a bunch of additional code to schedule queues on FW slots for the limited-number-of-FW-slots case, which, right now, is driver specific. The job queuing front-end pretty much achieves what drm_sched does already: queuing job to entities, checking deps, submitting job to HW (in our case, writing to the command stream ring buffer). Things start to differ after that point: once a scheduling entity has pending jobs, we add it to one of the runnable queues (one queue per prio) and kick the kernel-side timeslice-based scheduler to re-evaluate, if needed.

Quoted 7 lines
Quoted 5 lines

I'm all for using generic code when it makes sense, even if that means adding this common code when it doesn't exists, but I don't want to be dragged into some major refactoring that might take years to land. Especially if pancsf is the first FW-assisted-scheduler-with-few-FW-slot driver.

I don't see where there's a major refactoring that you're getting dragged into?

Oh, no, I'm not saying this is the case just yet, just wanted to make sure we're on the same page :-).

Quoted 3 lines

Yes there's a huge sprawling discussion right now, but I think that's just largely people getting confused.

I definitely am :-).

Quoted 4 lines

Wrt the actual id assignment stuff, in amdgpu at least it's few lines of code. See the amdgpu_vmid_grab stuff for the simplest starting point.

Ok, thanks for the pointers. I'll have a look and see how I could use that. I guess that's about getting access to the FW slots with some sort of priority+FIFO ordering guarantees given by TTM. If that's the case, I'll have to think about it, because that's a major shift from what we're doing now, and I'm afraid this could lead to starving non-resident entities if all resident entities keep receiving new jobs to execute. Unless we put some sort of barrier when giving access to a slot, so we evict the entity when it's done executing the stuff it had when it was given access to this slot. But then, again, there are other constraints to take into account for the Arm Mali CSF case:

Quoted 8 lines

- it's more efficient to update all FW slots at once, because each update of a slot might require updating priorities of the other slots (FW mandates unique slot priorities, and those priorities depend on the entity priority/queue-ordering)
- context/FW slot switches have a non-negligible cost (FW needs to suspend the context and save the state every time there such a switch), so, limiting the number of FW slot updates might prove important

I frankly think you're overworrying. When you have 31+ contexts running at the same time, you have bigger problems. At that point there's two use-cases:
1. system is overloaded, the user will reach for reset button anyway
2. temporary situation, all you have to do is be roughly fair enough to get through it before case 1 happens.

Quoted 6 lines

Trying to write a perfect scheduler for this before we have actual benchmarks that justify the effort seems like pretty serious overkill. That's why I think the simplest solution is the one we should have:

- drm/sched frontend. If you get into slot exhaustion that alone will ensure enough fairness

We're talking about the CS ring buffer slots here, right?

Quoted 17 lines

- LRU list of slots, with dma_fence so you can pipeline/batch up changes as needed (but I honestly wouldn't worry about the batching before you've shown an actual need for this in some benchmark/workload, even piglit shouldn't have this many things running concurrently I think, you don't have that many cpu cores). Between drm/sched and the lru you will have an emergent scheduler that cycles through all runnable gpu jobs.

- If you want to go fancy, have eviction tricks like skipping currently still active gpu context with higher priority than the one that you need to find a slot for.

- You don't need time slicing in this, not even for compute. compute is done with preempt context fences, if you give them a minimum scheduling quanta you'll have a very basic round robin scheduler as an emergent thing.

Quoted 5 lines

Any workload were it matters will be scheduled by the fw directly, with drm/sched only being the dma_fence dependcy sorter. My take is that if you spend more than a hundred or so lines with slot allocation logic (excluding the hw code to load/unload a slot) you're probably doing some serious overengineering.

Let me see if I got this right:

Quoted 13 lines
Quoted 5 lines

- we still keep a 1:1 drm_gpu_scheduler:drm_sched_entity approach, where hw_submission_limit == available_slots_in_ring_buf
- when ->run_job() is called, we write the RUN_JOB() instruction sequence to the next available ringbuf slot and queue the entity to the FW-slot queue

Quoted 8 lines

* if a slot is directly available, we program the slot directly
* if no slots are available, but some slots are done with the jobs they were given (last job fence signaled), we evict the LRU entity (possibly taking priority into account) and use this slot for the new entity
* if no slots are available and all currently assigned slots contain busy entities, we queue the entity to a pending list (possibly one list per prio)

Quoted 8 lines

You could also handle this in ->prepare_job, which is called after all the default fences have signalled. That allows you to put the "wait for a previous job to finnish/unload" behind a dma_fence, which is how (I think at least) you can get the round-robin emergent behaviour: If there's no idle slot, you just pick all the fences from the currently busy job you want to steal the slot from (with priority and lru taken into account), let the scheduler wait for that to finnish, and then it'll call your run_job when the slot is already available.

Ah, nice! It would also avoid queuing new jobs to a resident entity when others are waiting for a FW slot, even if, in practice, I'm not sure we should do that: context will be suspended when the group is evicted anyway, and things could keep running in the meantime. I'll give it a try, thanks for the suggestion!

Quoted 4 lines

Also if you do the allocation in ->prepare_job with dma_fence and not run_job, then I think can sort out fairness issues (if they do pop up) in the drm/sched code instead of having to think about this in each driver.

Quoted 12 lines

By allocation, you mean assigning a FW slot ID? If we do this allocation in ->prepare_job(), couldn't we mess up ordering? Like, lower-prio/later-queuing entity being scheduled before its pairs, because there's no guarantee on the job completion order (and thus the queue idleness order). I mean, completion order depends on the kind of job being executed by the queues, the time the FW actually lets the queue execute things and probably other factors. You can use metrics like the position in the LRU list + the amount of jobs currently queued to a group to guess which one will be idle first, but that's just a guess. And I'm not sure I see what doing this slot selection in ->prepare_job() would bring us compared to doing it in ->run_job(), where we can just pick the least recently used slot.

In ->prepare_job you can let the scheduler code do the stalling (and ensure fairness), in ->run_job it's your job. The current RFC doesn't really bother much with getting this very right, but if the scheduler code tries to make sure it pushes higher-prio stuff in first before others, you should get the right outcome.

The more important functional issue is that you must only allocate the fw slot after all dependencies have signalled. Otherwise you might get a nice deadlock, where job A is waiting for the fw slot of B to become free, and B is waiting for A to finish.

Quoted 26 lines
Quoted 10 lines

Few fw sched slots essentially just make fw scheduling unfairness more prominent than with others, but I don't think it's fundamentally something else really.

If every ctx does that and the lru isn't too busted, they should then form a nice orderly queue and cycle through the fw scheduler, while still being able to get some work done. It's essentially the exact same thing that happens with ttm vram eviction, when you have a total working set where each process fits in vram individually, but in total they're too big and you need to cycle things through.

I see.

Quoted 13 lines
Quoted 12 lines
Quoted 5 lines

I'll need to make sure this still works with the concept of group (it's not a single queue we schedule, it's a group of queues, meaning that we have N fences to watch to determine if the slot is busy or not, but that should be okay).

Oh, there's one other thing I forgot to mention: the FW scheduler is not entirely fair, it does take the slot priority (which has to be unique across all currently assigned slots) into account when scheduling groups. So, ideally, we'd want to rotate group priorities when they share the same drm_sched_priority (probably based on the position in the LRU).

Quoted 8 lines
Quoted 4 lines

Hm that will make things a bit more fun I guess, especially with your constraint to not update this too often. How strict is that priority difference? If it's a lot, we might need to treat this more like execlist and less like a real fw scheduler ...

Strict as in, if two groups with same priority try to request an overlapping set of resources (cores or tilers), it can deadlock, so pretty strict I would say :-).

So it first finishes all the higher priority tasks and only then it runs the next one, so no round-robin? Or am I just confused what this all is about. Or is it more that the order in the group determines how it tries to schedule on the hw, and if the earlier job needs hw that also the later one needs, then the earlier one has to finish first? Which would still mean that for these overlapping cases there's just no round-robin in the fw scheduler at all.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#52 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

Quoted 18 lines · On Thu, 12 Jan 2023 16:38:18 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 5 lines
Quoted 4 lines

Also if you do the allocation in ->prepare_job with dma_fence and not run_job, then I think can sort out fairness issues (if they do pop up) in the drm/sched code instead of having to think about this in each driver.

Quoted 12 lines

By allocation, you mean assigning a FW slot ID? If we do this allocation in ->prepare_job(), couldn't we mess up ordering? Like, lower-prio/later-queuing entity being scheduled before its pairs, because there's no guarantee on the job completion order (and thus the queue idleness order). I mean, completion order depends on the kind of job being executed by the queues, the time the FW actually lets the queue execute things and probably other factors. You can use metrics like the position in the LRU list + the amount of jobs currently queued to a group to guess which one will be idle first, but that's just a guess. And I'm not sure I see what doing this slot selection in ->prepare_job() would bring us compared to doing it in ->run_job(), where we can just pick the least recently used slot.

Quoted 2 lines

In ->prepare_job you can let the scheduler code do the stalling (and ensure fairness), in ->run_job it's your job.

Yeah returning a fence in ->prepare_job() to wait for a FW slot to become idle sounds good. This fence would be signaled when one of the slots becomes idle. But I'm wondering why we'd want to select the slot so early. Can't we just do the selection in ->run_job()? After all, if the fence has been signaled, that means we'll find at least one slot that's ready when we hit ->run_job(), and we can select it at that point.

Quoted 4 lines

The current RFC doesn't
really bother much with getting this very right, but if the scheduler code tries to make sure it pushes higher-prio stuff in first before others, you should get the right outcome.

Okay, so I'm confused again. We said we had a 1:1 drm_gpu_scheduler:drm_sched_entity mapping, meaning that entities are isolated from each other. I can see how I could place the dma_fence returned by ->prepare_job() in a driver-specific per-priority list, so the driver can pick the highest-prio/first-inserted entry and signal the associated fence when a slot becomes idle. But I have a hard time seeing how common code could do that if it doesn't see the other entities. Right now, drm_gpu_scheduler only selects the best entity among the registered ones, and there's only one entity per drm_gpu_scheduler in this case.

Quoted 3 lines

The more important functional issue is that you must only allocate the fw slot after all dependencies have signalled.

Sure, but it doesn't have to be a specific FW slot, it can be any FW slot, as long as we don't signal more fences than we have slots available, right?

Quoted 3 lines

Otherwise you might get
a nice deadlock, where job A is waiting for the fw slot of B to become free, and B is waiting for A to finish.

Got that part, and that's ensured by the fact we wait for all regular deps before returning the FW-slot-available dma_fence in ->prepare_job(). This exact same fence will be signaled when a slot becomes idle.

Quoted 36 lines
Quoted 27 lines
Quoted 11 lines

Few fw sched slots essentially just make fw scheduling unfairness more prominent than with others, but I don't think it's fundamentally something else really.

If every ctx does that and the lru isn't too busted, they should then form a nice orderly queue and cycle through the fw scheduler, while still being able to get some work done. It's essentially the exact same thing that happens with ttm vram eviction, when you have a total working set where each process fits in vram individually, but in total they're too big and you need to cycle things through.

I see.

Quoted 13 lines
Quoted 12 lines
Quoted 5 lines

I'll need to make sure this still works with the concept of group (it's not a single queue we schedule, it's a group of queues, meaning that we have N fences to watch to determine if the slot is busy or not, but that should be okay).

Oh, there's one other thing I forgot to mention: the FW scheduler is not entirely fair, it does take the slot priority (which has to be unique across all currently assigned slots) into account when scheduling groups. So, ideally, we'd want to rotate group priorities when they share the same drm_sched_priority (probably based on the position in the LRU).

Quoted 8 lines
Quoted 4 lines

Hm that will make things a bit more fun I guess, especially with your constraint to not update this too often. How strict is that priority difference? If it's a lot, we might need to treat this more like execlist and less like a real fw scheduler ...

Strict as in, if two groups with same priority try to request an overlapping set of resources (cores or tilers), it can deadlock, so pretty strict I would say :-).

Quoted 7 lines

So it first finishes all the higher priority tasks and only then it runs the next one, so no round-robin? Or am I just confused what this all is about. Or is it more that the order in the group determines how it tries to schedule on the hw, and if the earlier job needs hw that also the later one needs, then the earlier one has to finish first? Which would still mean that for these overlapping cases there's just no round-robin in the fw scheduler at all.

Okay, so my understanding is: FW scheduler always takes the highest priority when selecting between X groups requesting access to a resource, but if 2 groups want the same resource and have the same priority, there's no ordering guarantee. The deadlock happens when both group A and B claim resources X and Y. Group A might get resource X and group B might get resource Y, both waiting for the other resource they claimed. If they have different priorities one of them would back off and let the other run, if they have the same priority, none of them would, and that's where the deadlock comes from. Note that we don't control the order resources get acquired from the CS, so there's no way to avoid this deadlock without assigning different priorities.

And you're right, if you pick different priorities, the only time lower priority groups get to run is when the highest priority group is waiting on an asynchronous operation to complete (can be a compute/frag/tiler job completion, some inter queue synchronization, waiting for an already acquired resource, ...), or when it's idle. I suspect queues from different groups can run concurrently if there's enough command-stream processing slots available, and those groups request resources that don't overlap, but I'm speculating here. So, no round-robin if slots are assigned unique priorities. Not even sure scheduling is time-slice based to be honest, it could be some cooperative scheduling where groups with the same priorities get to wait for the currently running group to be blocked to get access to the HW. In any case, there's no easy way to prevent deadlocks if we don't assign unique priorities.

#53 lore raw

Re: [Intel-gfx] [RFC PATCH 04/20] drm/sched: Convert drm scheduler to use a work queue rather than kthread

In reply to Boris Brezillon
Quoted 275 lines · On Thu, 12 Jan 2023 11:11:03 +0100 Boris Brezillon…
Quoted 238 lines · On Thu, 12 Jan 2023 10:32:18 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 221 lines · Boris Brezillon

Hi Daniel,

Quoted 155 lines · On Wed, 11 Jan 2023 22:47:02 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 153 lines · Boris Brezillon

Hi Daniel,

Quoted 146 lines · On Mon, 9 Jan 2023 21:40:21 +0100 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Boris Brezillon

Hi Jason,

Quoted 98 lines · On Mon, 9 Jan 2023 09:45:09 -0600 Jason Ekstrand <[email protected]> wrote:
Quoted 55 lines · Matthew Brost
Quoted 6 lines · Boris Brezillon
Quoted 3 lines · On Fri, 30 Dec 2022 12:55:08 +0100 Boris Brezillon…

On Fri, 30 Dec 2022 11:20:42 +0100
Boris Brezillon <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hello Matthew,

Quoted 2 lines · On Thu, 22 Dec 2022 14:21:11 -0800 Matthew Brost <[email protected]>…

In XE, the new Intel GPU driver, a choice has made to have a 1 to 1
mapping between a drm_gpu_scheduler and drm_sched_entity. At first

this

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

seems a bit odd but let us explain the reasoning below.

1. In XE the submission order from multiple drm_sched_entity is not
guaranteed to be the same completion even if targeting the same

hardware

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

engine. This is because in XE we have a firmware scheduler, the

GuC,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

which allowed to reorder, timeslice, and preempt submissions. If a

using

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR

falls

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

apart as the TDR expects submission order == completion order.

Using a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dedicated drm_gpu_scheduler per drm_sched_entity solve this

problem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Oh, that's interesting. I've been trying to solve the same sort of issues to support Arm's new Mali GPU which is relying on a

FW-assisted

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

scheduling scheme (you give the FW N streams to execute, and it does the scheduling between those N command streams, the kernel driver does timeslice scheduling to update the command streams passed to the FW). I must admit I gave up on using drm_sched at some point, mostly because the integration with drm_sched was painful, but also because

I

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

felt trying to bend drm_sched to make it interact with a timeslice-oriented scheduling model wasn't really future proof.

Giving

Quoted 1 line
Quoted 1 line
Quoted 1 line

drm_sched_entity exlusive access to a drm_gpu_scheduler probably

might

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

help for a few things (didn't think it through yet), but I feel it's coming short on other aspects we have to deal with on Arm GPUs.

Ok, so I just had a quick look at the Xe driver and how it instantiates the drm_sched_entity and drm_gpu_scheduler, and I think I have a better understanding of how you get away with using drm_sched while still controlling how scheduling is really done. Here drm_gpu_scheduler is just a dummy abstract that let's you use the drm_sched job queuing/dep/tracking mechanism. The whole run-queue

Quoted 8 lines

You nailed it here, we use the DRM scheduler for queuing jobs, dependency tracking and releasing jobs to be scheduled when dependencies are met, and lastly a tracking mechanism of inflights jobs that need to be cleaned up if an error occurs. It doesn't actually do any scheduling aside from the most basic level of not overflowing the submission ring buffer. In this sense, a 1 to 1 relationship between entity and scheduler fits quite well.

Yeah, I think there's an annoying difference between what AMD/NVIDIA/Intel want here and what you need for Arm thanks to the number of FW queues available. I don't remember the exact number of GuC queues but it's at least 1k. This puts it in an entirely different class from what you have on Mali. Roughly, there's about three categories here:

1. Hardware where the kernel is placing jobs on actual HW rings. This is
old Mali, Intel Haswell and earlier, and probably a bunch of others. (Intel BDW+ with execlists is a weird case that doesn't fit in this categorization.)

2. Hardware (or firmware) with a very limited number of queues where
you're going to have to juggle in the kernel in order to run desktop Linux.

3. Firmware scheduling with a high queue count. In this case, you don't
want the kernel scheduling anything. Just throw it at the firmware and let it go brrrrr. If we ever run out of queues (unlikely), the kernel can temporarily pause some low-priority contexts and do some juggling or, frankly, just fail userspace queue creation and tell the user to close some windows.

The existence of this 2nd class is a bit annoying but it's where we are. I think it's worth recognizing that Xe and panfrost are in different places here and will require different designs. For Xe, we really are just using drm/scheduler as a front-end and the firmware does all the real scheduling.

How do we deal with class 2? That's an interesting question. We may eventually want to break that off into a separate discussion and not litter the Xe thread but let's keep going here for a bit. I think there are some pretty reasonable solutions but they're going to look a bit different.

Quoted 12 lines

The way I did this for Xe with execlists was to keep the 1:1:1 mapping between drm_gpu_scheduler, drm_sched_entity, and userspace xe_engine. Instead of feeding a GuC ring, though, it would feed a fixed-size execlist ring and then there was a tiny kernel which operated entirely in IRQ handlers which juggled those execlists by smashing HW registers. For Panfrost, I think we want something slightly different but can borrow some ideas here. In particular, have the schedulers feed kernel-side SW queues (they can even be fixed-size if that helps) and then have a kthread which juggles those feeds the limited FW queues. In the case where you have few enough active contexts to fit them all in FW, I do think it's best to have them all active in FW and let it schedule. But with only 31, you need to be able to juggle if you run out.

Quoted 11 lines

That's more or less what I do right now, except I don't use the drm_sched front-end to handle deps or queue jobs (at least not yet). The kernel-side timeslice-based scheduler juggling with runnable queues (queues with pending jobs that are not yet resident on a FW slot) uses a dedicated ordered-workqueue instead of a thread, with scheduler ticks being handled with a delayed-work (tick happening every X milliseconds when queues are waiting for a slot). It all seems very HW/FW-specific though, and I think it's a bit premature to try to generalize that part, but the dep-tracking logic implemented by drm_sched looked like something I could easily re-use, hence my interest in Xe's approach.

So another option for these few fw queue slots schedulers would be to treat them as vram and enlist ttm.

Well maybe more enlist ttm and less treat them like vram, but ttm can handle idr (or xarray or whatever you want) and then help you with all the pipelining (and the drm_sched then with sorting out dependencies). If you then also preferentially "evict" low-priority queus you pretty much have the perfect thing.

Note that GuC with sriov splits up the id space and together with some restrictions due to multi-engine contexts media needs might also need this all.

If you're balking at the idea of enlisting ttm just for fw queue management, amdgpu has a shoddy version of id allocation for their vm/tlb index allocation. Might be worth it to instead lift that into some sched helper code.

Quoted 8 lines
Quoted 3 lines

Would you mind pointing me to the amdgpu code you're mentioning here? Still have a hard time seeing what TTM has to do with scheduling, but I also don't know much about TTM, so I'll keep digging.

ttm is about moving stuff in&out of a limited space and gives you some nice tooling for pipelining it all. It doesn't care whether that space is vram or some limited id space. vmwgfx used ttm as an id manager iirc.

Ok.

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines

Either way there's two imo rather solid approaches available to sort this out. And once you have that, then there shouldn't be any big difference in driver design between fw with defacto unlimited queue ids, and those with severe restrictions in number of queues.

Honestly, I don't think there's much difference between those two cases already. There's just a bunch of additional code to schedule queues on FW slots for the limited-number-of-FW-slots case, which, right now, is driver specific. The job queuing front-end pretty much achieves what drm_sched does already: queuing job to entities, checking deps, submitting job to HW (in our case, writing to the command stream ring buffer). Things start to differ after that point: once a scheduling entity has pending jobs, we add it to one of the runnable queues (one queue per prio) and kick the kernel-side timeslice-based scheduler to re-evaluate, if needed.

Quoted 7 lines
Quoted 5 lines

I'm all for using generic code when it makes sense, even if that means adding this common code when it doesn't exists, but I don't want to be dragged into some major refactoring that might take years to land. Especially if pancsf is the first FW-assisted-scheduler-with-few-FW-slot driver.

I don't see where there's a major refactoring that you're getting dragged into?

Oh, no, I'm not saying this is the case just yet, just wanted to make sure we're on the same page :-).

Quoted 3 lines

Yes there's a huge sprawling discussion right now, but I think that's just largely people getting confused.

I definitely am :-).

Quoted 4 lines

Wrt the actual id assignment stuff, in amdgpu at least it's few lines of code. See the amdgpu_vmid_grab stuff for the simplest starting point.

Ok, thanks for the pointers. I'll have a look and see how I could use that. I guess that's about getting access to the FW slots with some sort of priority+FIFO ordering guarantees given by TTM. If that's the case, I'll have to think about it, because that's a major shift from what we're doing now, and I'm afraid this could lead to starving non-resident entities if all resident entities keep receiving new jobs to execute. Unless we put some sort of barrier when giving access to a slot, so we evict the entity when it's done executing the stuff it had when it was given access to this slot. But then, again, there are other constraints to take into account for the Arm Mali CSF case:

Quoted 8 lines

- it's more efficient to update all FW slots at once, because each update of a slot might require updating priorities of the other slots (FW mandates unique slot priorities, and those priorities depend on the entity priority/queue-ordering)
- context/FW slot switches have a non-negligible cost (FW needs to suspend the context and save the state every time there such a switch), so, limiting the number of FW slot updates might prove important

I frankly think you're overworrying. When you have 31+ contexts running at the same time, you have bigger problems. At that point there's two use-cases:
1. system is overloaded, the user will reach for reset button anyway
2. temporary situation, all you have to do is be roughly fair enough to get through it before case 1 happens.

Quoted 6 lines

Trying to write a perfect scheduler for this before we have actual benchmarks that justify the effort seems like pretty serious overkill. That's why I think the simplest solution is the one we should have:

- drm/sched frontend. If you get into slot exhaustion that alone will ensure enough fairness

We're talking about the CS ring buffer slots here, right?

Quoted 17 lines

- LRU list of slots, with dma_fence so you can pipeline/batch up changes as needed (but I honestly wouldn't worry about the batching before you've shown an actual need for this in some benchmark/workload, even piglit shouldn't have this many things running concurrently I think, you don't have that many cpu cores). Between drm/sched and the lru you will have an emergent scheduler that cycles through all runnable gpu jobs.

- If you want to go fancy, have eviction tricks like skipping currently still active gpu context with higher priority than the one that you need to find a slot for.

- You don't need time slicing in this, not even for compute. compute is done with preempt context fences, if you give them a minimum scheduling quanta you'll have a very basic round robin scheduler as an emergent thing.

Quoted 5 lines

Any workload were it matters will be scheduled by the fw directly, with drm/sched only being the dma_fence dependcy sorter. My take is that if you spend more than a hundred or so lines with slot allocation logic (excluding the hw code to load/unload a slot) you're probably doing some serious overengineering.

Let me see if I got this right:

Quoted 13 lines

- we still keep a 1:1 drm_gpu_scheduler:drm_sched_entity approach, where hw_submission_limit == available_slots_in_ring_buf
- when ->run_job() is called, we write the RUN_JOB() instruction sequence to the next available ringbuf slot and queue the entity to the FW-slot queue
* if a slot is directly available, we program the slot directly
* if no slots are available, but some slots are done with the jobs they were given (last job fence signaled), we evict the LRU entity (possibly taking priority into account) and use this slot for the new entity
* if no slots are available and all currently assigned slots contain busy entities, we queue the entity to a pending list (possibly one list per prio)

Forgot:

* if the group is already resident, we just move the slot to the LRU list head.

Quoted 5 lines

I'll need to make sure this still works with the concept of group (it's not a single queue we schedule, it's a group of queues, meaning that we have N fences to watch to determine if the slot is busy or not, but that should be okay).

author maintainer
#54 lore raw

RFC PATCH 5/20 drm/sched: Add generic scheduler message interface

In reply to Matthew Brost

Add generic schedule message interface which sends messages to backend from the drm_gpu_scheduler main submission thread. The idea is some of these messages modify some state in drm_sched_entity which is also modified during submission. By scheduling these messages and submission in the same thread their is not race changing states in drm_sched_entity.

This interface will be used in XE, new Intel GPU driver, to cleanup, suspend, resume, and change scheduling properties of a drm_sched_entity.

The interface is designed to be generic and extendable with only the backend understanding the messages.

---
 drivers/gpu/drm/scheduler/sched_main.c | 58 +++++++++++++++++++++++++-
 include/drm/gpu_scheduler.h            | 29 ++++++++++++-
 2 files changed, 84 insertions(+), 3 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +56 -2
index 8c64045d0692..8e688c2fc482 100644
@@ -987,6 +987,54 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list,
 }
 EXPORT_SYMBOL(drm_sched_pick_best);
 
+/**
+ * drm_sched_add_msg - add scheduler message
+ *
+ * @sched: scheduler instance
+ * @msg: message to be added
+ *
+ * Can and will pass an jobs waiting on dependencies or in a runnable queue.
+ * Messages processing will stop if schedule run wq is stopped and resume when
+ * run wq is started.
+ */
+void drm_sched_add_msg(struct drm_gpu_scheduler *sched,
+		       struct drm_sched_msg *msg)
+{
+	spin_lock(&sched->job_list_lock);
+	list_add_tail(&msg->link, &sched->msgs);
+	spin_unlock(&sched->job_list_lock);
+
+	/*
+	 * Same as above in drm_sched_run_wq_queue, try to kick worker if
+	 * paused, harmless if this races
+	 */
+	if (!sched->pause_run_wq)
+		queue_work(sched->run_wq, &sched->work_run);
+}
+EXPORT_SYMBOL(drm_sched_add_msg);
+
+/**
+ * drm_sched_get_msg - get scheduler message
+ *
+ * @sched: scheduler instance
+ *
+ * Returns NULL or message
+ */
+static struct drm_sched_msg *
+drm_sched_get_msg(struct drm_gpu_scheduler *sched)
+{
+	struct drm_sched_msg *msg;
+
+	spin_lock(&sched->job_list_lock);
+	msg = list_first_entry_or_null(&sched->msgs,
+				       struct drm_sched_msg, link);
+	if (msg)
+		list_del(&msg->link);
+	spin_unlock(&sched->job_list_lock);
+
+	return msg;
+}
+
 /**
  * drm_sched_main - main scheduler thread
  *
@@ -1000,6 +1048,7 @@ static void drm_sched_main(struct work_struct *w)
 
 	while (!READ_ONCE(sched->pause_run_wq)) {
 		struct drm_sched_entity *entity;
+		struct drm_sched_msg *msg;
 		struct drm_sched_fence *s_fence;
 		struct drm_sched_job *sched_job;
 		struct dma_fence *fence;
@@ -1007,12 +1056,16 @@ static void drm_sched_main(struct work_struct *w)
 
 		cleanup_job = drm_sched_get_cleanup_job(sched);
 		entity = drm_sched_select_entity(sched);
+		msg = drm_sched_get_msg(sched);
 
 		if (cleanup_job)
 			sched->ops->free_job(cleanup_job);
 
+		if (msg)
+			sched->ops->process_msg(msg);
+
 		if (!entity) {
-			if (!cleanup_job)
+			if (!cleanup_job && !msg)
 				break;
 			continue;
 		}
@@ -1021,7 +1074,7 @@ static void drm_sched_main(struct work_struct *w)
 
 		if (!sched_job) {
 			complete_all(&entity->entity_idle);
-			if (!cleanup_job)
+			if (!cleanup_job && !msg)
 				break;
 			continue;
 		}
@@ -1097,6 +1150,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,
 
 	init_waitqueue_head(&sched->job_scheduled);
 	INIT_LIST_HEAD(&sched->pending_list);
+	INIT_LIST_HEAD(&sched->msgs);
 	spin_lock_init(&sched->job_list_lock);
 	atomic_set(&sched->hw_rq_count, 0);
 	INIT_DELAYED_WORK(&sched->work_tdr, drm_sched_job_timedout);
include/drm/gpu_scheduler.h +28 -1
index ff50f3c289cd..31448deb9412 100644
@@ -369,6 +369,23 @@ enum drm_gpu_sched_stat {
 	DRM_GPU_SCHED_STAT_ENODEV,
 };
 
+/**
+ * struct drm_sched_msg - an in-band (relative to GPU scheduler run queue)
+ * message
+ *
+ * Generic enough for backend defined messages, backend can expand if needed.
+ */
+struct drm_sched_msg {
+	/** @link: list link into the gpu scheduler list of messages */
+	struct list_head		link;
+	/**
+	 * @private_data: opaque pointer to message private data (backend defined)
+	 */
+	void				*private_data;
+	/** @opcode: opcode of message (backend defined) */
+	unsigned int			opcode;
+};
+
 /**
  * struct drm_sched_backend_ops - Define the backend operations
  *	called by the scheduler
@@ -446,6 +463,12 @@ struct drm_sched_backend_ops {
          * and it's time to clean it up.
 	 */
 	void (*free_job)(struct drm_sched_job *sched_job);
+
+	/**
+	 * @process_msg: Process a message. Allowed to block, it is this
+	 * function's responsibility to free message if dynamically allocated.
+	 */
+	void (*process_msg)(struct drm_sched_msg *msg);
 };
 
 /**
@@ -456,6 +479,7 @@ struct drm_sched_backend_ops {
  * @timeout: the time after which a job is removed from the scheduler.
  * @name: name of the ring for which this scheduler is being used.
  * @sched_rq: priority wise array of run queues.
+ * @msgs: list of messages to be processed in @work_run
  * @job_scheduled: once @drm_sched_entity_do_release is called the scheduler
  *                 waits on this wait queue until all the scheduled jobs are
  *                 finished.
@@ -463,7 +487,7 @@ struct drm_sched_backend_ops {
  * @job_id_count: used to assign unique id to the each job.
  * @run_wq: workqueue used to queue @work_run
  * @timeout_wq: workqueue used to queue @work_tdr
- * @work_run: schedules jobs and cleans up entities
+ * @work_run: schedules jobs, cleans up jobs, and processes messages
  * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the
  *            timeout interval is over.
  * @pending_list: the list of jobs which are currently in the job queue.
@@ -485,6 +509,7 @@ struct drm_gpu_scheduler {
 	long				timeout;
 	const char			*name;
 	struct drm_sched_rq		sched_rq[DRM_SCHED_PRIORITY_COUNT];
+	struct list_head		msgs;
 	wait_queue_head_t		job_scheduled;
 	atomic_t			hw_rq_count;
 	atomic64_t			job_id_count;
@@ -530,6 +555,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity,
 
 void drm_sched_job_cleanup(struct drm_sched_job *job);
 void drm_sched_wakeup(struct drm_gpu_scheduler *sched);
+void drm_sched_add_msg(struct drm_gpu_scheduler *sched,
+		       struct drm_sched_msg *msg);
 void drm_sched_run_wq_stop(struct drm_gpu_scheduler *sched);
 void drm_sched_run_wq_start(struct drm_gpu_scheduler *sched);
 void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad);
Signature · 1 line
2.37.3
author maintainer
#55 lore raw

RFC PATCH 6/20 drm/sched: Start run wq before TDR in drm_sched_start

In reply to Matthew Brost

If the TDR is set to a very small value it can fire before the run wq is started in the function drm_sched_start. The run wq is expected to running when the TDR fires, fix this ordering so this expectation is always met.

---
 drivers/gpu/drm/scheduler/sched_main.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +2 -2
index 8e688c2fc482..f39fdc01c37b 100644
@@ -586,13 +586,13 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery)
 			drm_sched_job_done(s_job);
 	}
 
+	drm_sched_run_wq_start(sched);
+
 	if (full_recovery) {
 		spin_lock(&sched->job_list_lock);
 		drm_sched_start_timeout(sched);
 		spin_unlock(&sched->job_list_lock);
 	}
-
-	drm_sched_run_wq_start(sched);
 }
 EXPORT_SYMBOL(drm_sched_start);
 
Signature · 1 line
2.37.3
author maintainer
#56 lore raw

RFC PATCH 7/20 drm/sched: Submit job before starting TDR

In reply to Matthew Brost

If the TDR is set to a value, it can fire before a job is submitted in drm_sched_main. The job should be always be submitted before the TDR fires, fix this ordering.

---
 drivers/gpu/drm/scheduler/sched_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/scheduler/sched_main.c +1 -1
index f39fdc01c37b..fa25541bb477 100644
@@ -1082,10 +1082,10 @@ static void drm_sched_main(struct work_struct *w)
 		s_fence = sched_job->s_fence;
 
 		atomic_inc(&sched->hw_rq_count);
-		drm_sched_job_begin(sched_job);
 
 		trace_drm_run_job(sched_job, entity);
 		fence = sched->ops->run_job(sched_job);
+		drm_sched_job_begin(sched_job);
 		complete_all(&entity->entity_idle);
 		drm_sched_fence_scheduled(s_fence);
 
Signature · 1 line
2.37.3
author maintainer
#57 lore raw

RFC PATCH 8/20 drm/sched: Add helper to set TDR timeout

In reply to Matthew Brost

Add helper to set TDR timeout and restart the TDR with new timeout value. This will be used in XE, new Intel GPU driver, to trigger the TDR to cleanup drm_sched_entity that encounter errors.

---
 drivers/gpu/drm/scheduler/sched_main.c | 18 ++++++++++++++++++
 include/drm/gpu_scheduler.h            |  1 +
 2 files changed, 19 insertions(+)
drivers/gpu/drm/scheduler/sched_main.c +18 -0
index fa25541bb477..bdf0541ad818 100644
@@ -345,6 +345,24 @@ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched)
 		queue_delayed_work(sched->timeout_wq, &sched->work_tdr, sched->timeout);
 }
 
+/**
+ * drm_sched_set_timeout - set timeout for reset worker
+ *
+ * @sched: scheduler instance to set and (re)-start the worker for
+ * @timeout: timeout period
+ *
+ * Set and (re)-start the timeout for the given scheduler.
+ */
+void drm_sched_set_timeout(struct drm_gpu_scheduler *sched, long timeout)
+{
+	spin_lock(&sched->job_list_lock);
+	sched->timeout = timeout;
+	cancel_delayed_work(&sched->work_tdr);
+	drm_sched_start_timeout(sched);
+	spin_unlock(&sched->job_list_lock);
+}
+EXPORT_SYMBOL(drm_sched_set_timeout);
+
 /**
  * drm_sched_fault - immediately start timeout handler
  *
include/drm/gpu_scheduler.h +1 -0
index 31448deb9412..b9967af1788b 100644
@@ -553,6 +553,7 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity,
 				    struct drm_gpu_scheduler **sched_list,
                                    unsigned int num_sched_list);
 
+void drm_sched_set_timeout(struct drm_gpu_scheduler *sched, long timeout);
 void drm_sched_job_cleanup(struct drm_sched_job *job);
 void drm_sched_wakeup(struct drm_gpu_scheduler *sched);
 void drm_sched_add_msg(struct drm_gpu_scheduler *sched,
Signature · 1 line
2.37.3
author maintainer
#58 lore raw

RFC PATCH 9/20 drm: Add a gpu page-table walker helper

In reply to Matthew Brost

From: Thomas Hellström <[email protected]>

Add a gpu page table walker similar in functionality to the cpu page-table walker in mm/pagewalk.c. This is made a drm helper in the hope that it might prove useful to other drivers, but we could of course make it single-driver only and rename the functions initially.

Also if remaining a DRM helper, we should consider making it a helper kernel module of its own.

---
 drivers/gpu/drm/Makefile      |   1 +
 drivers/gpu/drm/drm_pt_walk.c | 159 +++++++++++++++++++++++++++++++++
 include/drm/drm_pt_walk.h     | 161 ++++++++++++++++++++++++++++++++++
 3 files changed, 321 insertions(+)
 create mode 100644 drivers/gpu/drm/drm_pt_walk.c
 create mode 100644 include/drm/drm_pt_walk.h
drivers/gpu/drm/Makefile +1 -0
index 23ad760884b2..d030c2885dd8 100644
@@ -39,6 +39,7 @@ drm-y := \
 	drm_prime.o \
 	drm_print.o \
 	drm_property.o \
+	drm_pt_walk.o \
 	drm_syncobj.o \
 	drm_sysfs.o \
 	drm_trace_points.o \
drivers/gpu/drm/drm_pt_walk.c new file +159 -0

159 lines not on this page · raw message

include/drm/drm_pt_walk.h new file +161 -0

161 lines not on this page · raw message

Signature · 1 line
2.37.3
author maintainer
#59 lore raw

RFC PATCH 10/20 drm/ttm: Don't print error message if eviction was interrupted

In reply to Matthew Brost

From: Thomas Hellström <[email protected]>

Avoid printing an error message if eviction was interrupted by, for example, the user pressing CTRL-C. That may happen if eviction is waiting for something, like for example a free batch-buffer.

---
 drivers/gpu/drm/ttm/ttm_bo.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_bo.c +2 -1
index cd266a067773..e60aaa3299e7 100644
@@ -465,7 +465,8 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
 	if (ret == -EMULTIHOP) {
 		ret = ttm_bo_bounce_temp_buffer(bo, &evict_mem, ctx, &hop);
 		if (ret) {
-			pr_err("Buffer eviction failed\n");
+			if (ret != -ERESTARTSYS && ret != -EINTR)
+				pr_err("Buffer eviction failed\n");
 			ttm_resource_free(bo, &evict_mem);
 			goto out;
 		}
Signature · 1 line
2.37.3
author maintainer
#60 lore raw

RFC PATCH 11/20 drm/i915: Remove gem and overlay frontbuffer tracking

In reply to Matthew Brost

From: Maarten Lankhorst <[email protected]>

Frontbuffer update handling should be done explicitly by using dirtyfb calls only.

---
 drivers/gpu/drm/i915/display/i9xx_plane.c     |  1 +
 drivers/gpu/drm/i915/display/intel_drrs.c     |  1 +
 drivers/gpu/drm/i915/display/intel_fb.c       |  1 +
 drivers/gpu/drm/i915/display/intel_overlay.c  | 14 -----------
 .../drm/i915/display/intel_plane_initial.c    |  1 +
 drivers/gpu/drm/i915/display/intel_psr.c      |  1 +
 .../drm/i915/display/skl_universal_plane.c    |  1 +
 drivers/gpu/drm/i915/gem/i915_gem_clflush.c   |  4 ---
 drivers/gpu/drm/i915/gem/i915_gem_domain.c    |  7 ------
 .../gpu/drm/i915/gem/i915_gem_execbuffer.c    |  2 --
 drivers/gpu/drm/i915/gem/i915_gem_object.c    | 25 -------------------
 drivers/gpu/drm/i915/gem/i915_gem_object.h    | 22 ----------------
 drivers/gpu/drm/i915/gem/i915_gem_phys.c      |  4 ---
 drivers/gpu/drm/i915/i915_driver.c            |  1 +
 drivers/gpu/drm/i915/i915_gem.c               |  8 ------
 drivers/gpu/drm/i915/i915_gem_gtt.c           |  1 -
 drivers/gpu/drm/i915/i915_vma.c               | 12 ---------
 17 files changed, 7 insertions(+), 99 deletions(-)
drivers/gpu/drm/i915/display/i9xx_plane.c +1 -0
index ecaeb7dc196b..633e462d96a0 100644
@@ -17,6 +17,7 @@
 #include "intel_display_types.h"
 #include "intel_fb.h"
 #include "intel_fbc.h"
+#include "intel_frontbuffer.h"
 #include "intel_sprite.h"
 
 /* Primary plane formats for gen <= 3 */
drivers/gpu/drm/i915/display/intel_drrs.c +1 -0
index 5b9e44443814..3503d112387d 100644
@@ -9,6 +9,7 @@
 #include "intel_de.h"
 #include "intel_display_types.h"
 #include "intel_drrs.h"
+#include "intel_frontbuffer.h"
 #include "intel_panel.h"
 
 /**
drivers/gpu/drm/i915/display/intel_fb.c +1 -0
index 63137ae5ab21..7cf31c87884c 100644
@@ -12,6 +12,7 @@
 #include "intel_display_types.h"
 #include "intel_dpt.h"
 #include "intel_fb.h"
+#include "intel_frontbuffer.h"
 
 #define check_array_bounds(i915, a, i) drm_WARN_ON(&(i915)->drm, (i) >= ARRAY_SIZE(a))
 
drivers/gpu/drm/i915/display/intel_overlay.c +0 -14
index c12bdca8da9b..5b86563ce577 100644
@@ -186,7 +186,6 @@ struct intel_overlay {
 	struct intel_crtc *crtc;
 	struct i915_vma *vma;
 	struct i915_vma *old_vma;
-	struct intel_frontbuffer *frontbuffer;
 	bool active;
 	bool pfit_active;
 	u32 pfit_vscale_ratio; /* shifted-point number, (1<<12) == 1.0 */
@@ -287,20 +286,9 @@ static void intel_overlay_flip_prepare(struct intel_overlay *overlay,
 				       struct i915_vma *vma)
 {
 	enum pipe pipe = overlay->crtc->pipe;
-	struct intel_frontbuffer *frontbuffer = NULL;
 
 	drm_WARN_ON(&overlay->i915->drm, overlay->old_vma);
 
-	if (vma)
-		frontbuffer = intel_frontbuffer_get(vma->obj);
-
-	intel_frontbuffer_track(overlay->frontbuffer, frontbuffer,
-				INTEL_FRONTBUFFER_OVERLAY(pipe));
-
-	if (overlay->frontbuffer)
-		intel_frontbuffer_put(overlay->frontbuffer);
-	overlay->frontbuffer = frontbuffer;
-
 	intel_frontbuffer_flip_prepare(overlay->i915,
 				       INTEL_FRONTBUFFER_OVERLAY(pipe));
 
@@ -810,8 +798,6 @@ static int intel_overlay_do_put_image(struct intel_overlay *overlay,
 		goto out_pin_section;
 	}
 
-	i915_gem_object_flush_frontbuffer(new_bo, ORIGIN_DIRTYFB);
-
 	if (!overlay->active) {
 		const struct intel_crtc_state *crtc_state =
 			overlay->crtc->config;
drivers/gpu/drm/i915/display/intel_plane_initial.c +1 -0
index 76be796df255..cad9c8884af3 100644
@@ -9,6 +9,7 @@
 #include "intel_display.h"
 #include "intel_display_types.h"
 #include "intel_fb.h"
+#include "intel_frontbuffer.h"
 #include "intel_plane_initial.h"
 
 static bool
drivers/gpu/drm/i915/display/intel_psr.c +1 -0
index 9820e5fdd087..bc998b526d88 100644
@@ -33,6 +33,7 @@
 #include "intel_de.h"
 #include "intel_display_types.h"
 #include "intel_dp_aux.h"
+#include "intel_frontbuffer.h"
 #include "intel_hdmi.h"
 #include "intel_psr.h"
 #include "intel_snps_phy.h"
drivers/gpu/drm/i915/display/skl_universal_plane.c +1 -0
index 4b79c2d2d617..2f5524f380b0 100644
@@ -16,6 +16,7 @@
 #include "intel_display_types.h"
 #include "intel_fb.h"
 #include "intel_fbc.h"
+#include "intel_frontbuffer.h"
 #include "intel_psr.h"
 #include "intel_sprite.h"
 #include "skl_scaler.h"
drivers/gpu/drm/i915/gem/i915_gem_clflush.c +0 -4
index b3b398fe689c..df2db78b10ca 100644
@@ -6,8 +6,6 @@
 
 #include <drm/drm_cache.h>
 
-#include "display/intel_frontbuffer.h"
-
 #include "i915_drv.h"
 #include "i915_gem_clflush.h"
 #include "i915_sw_fence_work.h"
@@ -22,8 +20,6 @@ static void __do_clflush(struct drm_i915_gem_object *obj)
 {
 	GEM_BUG_ON(!i915_gem_object_has_pages(obj));
 	drm_clflush_sg(obj->mm.pages);
-
-	i915_gem_object_flush_frontbuffer(obj, ORIGIN_CPU);
 }
 
 static void clflush_work(struct dma_fence_work *base)
drivers/gpu/drm/i915/gem/i915_gem_domain.c +0 -7
index 9969e687ad85..cd5505da4884 100644
@@ -4,7 +4,6 @@
  * Copyright © 2014-2016 Intel Corporation
  */
 
-#include "display/intel_frontbuffer.h"
 #include "gt/intel_gt.h"
 
 #include "i915_drv.h"
@@ -65,8 +64,6 @@ flush_write_domain(struct drm_i915_gem_object *obj, unsigned int flush_domains)
 				intel_gt_flush_ggtt_writes(vma->vm->gt);
 		}
 		spin_unlock(&obj->vma.lock);
-
-		i915_gem_object_flush_frontbuffer(obj, ORIGIN_CPU);
 		break;
 
 	case I915_GEM_DOMAIN_WC:
@@ -629,9 +626,6 @@ i915_gem_set_domain_ioctl(struct drm_device *dev, void *data,
 out_unlock:
 	i915_gem_object_unlock(obj);
 
-	if (!err && write_domain)
-		i915_gem_object_invalidate_frontbuffer(obj, ORIGIN_CPU);
-
 out:
 	i915_gem_object_put(obj);
 	return err;
@@ -742,7 +736,6 @@ int i915_gem_object_prepare_write(struct drm_i915_gem_object *obj,
 	}
 
 out:
-	i915_gem_object_invalidate_frontbuffer(obj, ORIGIN_CPU);
 	obj->mm.dirty = true;
 	/* return with the pages pinned */
 	return 0;
drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c +0 -2
index f98600ca7557..08f84d4f4f92 100644
@@ -11,8 +11,6 @@
 
 #include <drm/drm_syncobj.h>
 
-#include "display/intel_frontbuffer.h"
-
 #include "gem/i915_gem_ioctls.h"
 #include "gt/intel_context.h"
 #include "gt/intel_gpu_commands.h"
drivers/gpu/drm/i915/gem/i915_gem_object.c +0 -25
index 1a0886b8aaa1..d2fef38cd12e 100644
@@ -27,7 +27,6 @@
 
 #include <drm/drm_cache.h>
 
-#include "display/intel_frontbuffer.h"
 #include "pxp/intel_pxp.h"
 
 #include "i915_drv.h"
@@ -400,30 +399,6 @@ static void i915_gem_free_object(struct drm_gem_object *gem_obj)
 		queue_work(i915->wq, &i915->mm.free_work);
 }
 
-void __i915_gem_object_flush_frontbuffer(struct drm_i915_gem_object *obj,
-					 enum fb_op_origin origin)
-{
-	struct intel_frontbuffer *front;
-
-	front = __intel_frontbuffer_get(obj);
-	if (front) {
-		intel_frontbuffer_flush(front, origin);
-		intel_frontbuffer_put(front);
-	}
-}
-
-void __i915_gem_object_invalidate_frontbuffer(struct drm_i915_gem_object *obj,
-					      enum fb_op_origin origin)
-{
-	struct intel_frontbuffer *front;
-
-	front = __intel_frontbuffer_get(obj);
-	if (front) {
-		intel_frontbuffer_invalidate(front, origin);
-		intel_frontbuffer_put(front);
-	}
-}
-
 static void
 i915_gem_object_read_from_page_kmap(struct drm_i915_gem_object *obj, u64 offset, void *dst, int size)
 {
drivers/gpu/drm/i915/gem/i915_gem_object.h +0 -22
index 3db53769864c..90dba761889c 100644
@@ -11,7 +11,6 @@
 #include <drm/drm_file.h>
 #include <drm/drm_device.h>
 
-#include "display/intel_frontbuffer.h"
 #include "intel_memory_region.h"
 #include "i915_gem_object_types.h"
 #include "i915_gem_gtt.h"
@@ -573,27 +572,6 @@ int i915_gem_object_wait_priority(struct drm_i915_gem_object *obj,
 				  unsigned int flags,
 				  const struct i915_sched_attr *attr);
 
-void __i915_gem_object_flush_frontbuffer(struct drm_i915_gem_object *obj,
-					 enum fb_op_origin origin);
-void __i915_gem_object_invalidate_frontbuffer(struct drm_i915_gem_object *obj,
-					      enum fb_op_origin origin);
-
-static inline void
-i915_gem_object_flush_frontbuffer(struct drm_i915_gem_object *obj,
-				  enum fb_op_origin origin)
-{
-	if (unlikely(rcu_access_pointer(obj->frontbuffer)))
-		__i915_gem_object_flush_frontbuffer(obj, origin);
-}
-
-static inline void
-i915_gem_object_invalidate_frontbuffer(struct drm_i915_gem_object *obj,
-				       enum fb_op_origin origin)
-{
-	if (unlikely(rcu_access_pointer(obj->frontbuffer)))
-		__i915_gem_object_invalidate_frontbuffer(obj, origin);
-}
-
 int i915_gem_object_read_from_page(struct drm_i915_gem_object *obj, u64 offset, void *dst, int size);
 
 bool i915_gem_object_is_shmem(const struct drm_i915_gem_object *obj);
drivers/gpu/drm/i915/gem/i915_gem_phys.c +0 -4
index 68453572275b..4cf57676e180 100644
@@ -156,15 +156,11 @@ int i915_gem_object_pwrite_phys(struct drm_i915_gem_object *obj,
 	 * We manually control the domain here and pretend that it
 	 * remains coherent i.e. in the GTT domain, like shmem_pwrite.
 	 */
-	i915_gem_object_invalidate_frontbuffer(obj, ORIGIN_CPU);
-
 	if (copy_from_user(vaddr, user_data, args->size))
 		return -EFAULT;
 
 	drm_clflush_virt_range(vaddr, args->size);
 	intel_gt_chipset_flush(to_gt(i915));
-
-	i915_gem_object_flush_frontbuffer(obj, ORIGIN_CPU);
 	return 0;
 }
 
drivers/gpu/drm/i915/i915_driver.c +1 -0
index c1e427ba57ae..f4201f9c5f84 100644
@@ -346,6 +346,7 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
 
 	spin_lock_init(&dev_priv->irq_lock);
 	spin_lock_init(&dev_priv->gpu_error.lock);
+	spin_lock_init(&dev_priv->display.fb_tracking.lock);
 	mutex_init(&dev_priv->display.backlight.lock);
 
 	mutex_init(&dev_priv->sb_lock);
drivers/gpu/drm/i915/i915_gem.c +0 -8
index 969581e7106f..594891291735 100644
@@ -40,7 +40,6 @@
 #include <drm/drm_vma_manager.h>
 
 #include "display/intel_display.h"
-#include "display/intel_frontbuffer.h"
 
 #include "gem/i915_gem_clflush.h"
 #include "gem/i915_gem_context.h"
@@ -569,8 +568,6 @@ i915_gem_gtt_pwrite_fast(struct drm_i915_gem_object *obj,
 		goto out_rpm;
 	}
 
-	i915_gem_object_invalidate_frontbuffer(obj, ORIGIN_CPU);
-
 	user_data = u64_to_user_ptr(args->data_ptr);
 	offset = args->offset;
 	remain = args->size;
@@ -613,7 +610,6 @@ i915_gem_gtt_pwrite_fast(struct drm_i915_gem_object *obj,
 	}
 
 	intel_gt_flush_ggtt_writes(ggtt->vm.gt);
-	i915_gem_object_flush_frontbuffer(obj, ORIGIN_CPU);
 
 	i915_gem_gtt_cleanup(obj, &node, vma);
 out_rpm:
@@ -700,8 +696,6 @@ i915_gem_shmem_pwrite(struct drm_i915_gem_object *obj,
 		offset = 0;
 	}
 
-	i915_gem_object_flush_frontbuffer(obj, ORIGIN_CPU);
-
 	i915_gem_object_unpin_pages(obj);
 	return ret;
 
@@ -1272,8 +1266,6 @@ void i915_gem_init_early(struct drm_i915_private *dev_priv)
 {
 	i915_gem_init__mm(dev_priv);
 	i915_gem_init__contexts(dev_priv);
-
-	spin_lock_init(&dev_priv->display.fb_tracking.lock);
 }
 
 void i915_gem_cleanup_early(struct drm_i915_private *dev_priv)
drivers/gpu/drm/i915/i915_gem_gtt.c +0 -1
index 7bd1861ddbdf..a9662cc6ed1e 100644
@@ -15,7 +15,6 @@
 #include <asm/set_memory.h>
 #include <asm/smp.h>
 
-#include "display/intel_frontbuffer.h"
 #include "gt/intel_gt.h"
 #include "gt/intel_gt_requests.h"
 
drivers/gpu/drm/i915/i915_vma.c +0 -12
index 7d044888ac33..e3b73175805b 100644
@@ -26,7 +26,6 @@
 #include <linux/dma-fence-array.h>
 #include <drm/drm_gem.h>
 
-#include "display/intel_frontbuffer.h"
 #include "gem/i915_gem_lmem.h"
 #include "gem/i915_gem_tiling.h"
 #include "gt/intel_engine.h"
@@ -1901,17 +1900,6 @@ int _i915_vma_move_to_active(struct i915_vma *vma,
 			return err;
 	}
 
-	if (flags & EXEC_OBJECT_WRITE) {
-		struct intel_frontbuffer *front;
-
-		front = __intel_frontbuffer_get(obj);
-		if (unlikely(front)) {
-			if (intel_frontbuffer_invalidate(front, ORIGIN_CS))
-				i915_active_add_request(&front->write, rq);
-			intel_frontbuffer_put(front);
-		}
-	}
-
 	if (fence) {
 		struct dma_fence *curr;
 		enum dma_resv_usage usage;
Signature · 1 line
2.37.3