Skip to contentSkip to conversation

cgroup private data and DRM/i915 integration

PATCH In review Matt Roper 6 participants 26 messages Mar 17, 2018 – Apr 5, 2018

Subsystems (4)
lore
Share & tools
lore

Matt Roper

author
#0 lore raw

v4 cover cgroup private data and DRM/i915 integration

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

v4 of this series brings large changes to the cgroup core half of the series and moderate changes to the i915 side.

cgroup core changes:

* The cgroup private data interface and implementation has changed again. The internal implementation is ida+radixtree based to allow much faster lookups than the previous hashtable approach. Private data is now registered and looked up via kref, which should allow the same private data to be set on multiple cgroups at the same time. The interface now looks like:

- cgroup_priv_getkey()
Obtain an integer key that will be used to store/lookup cgroup private data. This only needs to be called once at startup, driver init, etc.; after that the same key is used to store private data against any cgroup

- cgroup_priv_destroykey(key)
Release a private data key and drop references to any private data associated with the key on all cgroups. This function is very heavy, but will generally only be called by module-based users of the interface when the module is unloaded.

- cgroup_priv_install(cgrp, key, ref)
Store private data for a cgroup under the given key and increment the reference count.

- cgroup_priv_get(cgrp, key)
Take and return a reference to a cgroup's private data associated with the given key.

- cgroup_priv_get_current(key)
Same as cgroup_priv_get, but operates on the current task's cgroup.

- cgroup_priv_release(cgrp, key)
Remove private data from a cgroup and decrement its reference count.

* Dropped the cgroup_permission() function. My i915 usage of this functionality will take a different approach for determining access control.

i915 cgroup-based priority changes:

* cgroup priority offset is now bounded such that (context+cgroup) adjustments fall within the range [-0x7fffff,0x7fffff]. This only takes 24 bits, leaving several effective priority bits for future flags or bookkeeping.

* Display boost is added as a second cgroup parameter; each cgroup's processes can get differing boosts if a display operation is waiting on their completion. If we have non-overlapping cgroup priority ranges, this allows a system administrator to decide whether workloads from the lower priority cgroup(s) should still jump past the workloads in some/all of the higher priority cgroups.

* Access control for cgroup settings has been changed. Instead of following cgroup filesystem permissions, we now restrict the access to either the DRM master or capable(CAP_SYS_RESOURCE).

Cc Tejun Heo <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Roman Gushchin <[email protected]>
Cc Chris Wilson <[email protected]>

Matt Roper (8):
cgroup: Allow registration and lookup of cgroup private data (v2)
cgroup: Introduce task_get_dfl_cgroup() (v2)
cgroup: Introduce cgroup_priv_get_current
drm/i915: Adjust internal priority definitions drm/i915: cgroup integration (v3)
drm/i915: Introduce 'priority offset' for GPU contexts (v3) drm/i915: Introduce per-cgroup display boost setting drm/i915: Add context priority & priority offset to debugfs (v2)

 drivers/gpu/drm/i915/Makefile           |   1 +
 drivers/gpu/drm/i915/i915_cgroup.c      | 205 +++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_debugfs.c     |   3 +
 drivers/gpu/drm/i915/i915_drv.c         |   7 ++
 drivers/gpu/drm/i915/i915_drv.h         |  33 ++++-
 drivers/gpu/drm/i915/i915_gem_context.c |  11 +-
 drivers/gpu/drm/i915/i915_gem_context.h |   9 ++
 drivers/gpu/drm/i915/i915_request.h     |  18 ++-
 drivers/gpu/drm/i915/intel_display.c    |   5 +-
 include/linux/cgroup-defs.h             |   8 ++
 include/linux/cgroup.h                  |  37 ++++++
 include/uapi/drm/i915_drm.h             |  14 +++
 kernel/cgroup/cgroup.c                  | 208 +++++++++++++++++++++++++++++++-
 13 files changed, 545 insertions(+), 14 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_cgroup.c
Signature · 6 lines
2.14.3

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
author
#1 lore raw

PATCH v4 1/8 cgroup: Allow registration and lookup of cgroup private data (v2)

There are cases where other parts of the kernel may wish to store data associated with individual cgroups without building a full cgroup controller. Let's add interfaces to allow them to register and lookup this private data for individual cgroups.

A kernel system (e.g., a driver) that wishes to register private data for a cgroup should start by obtaining a unique private data key by calling cgroup_priv_getkey(). It may then associate private data with a cgroup by calling cgroup_priv_install(cgrp, key, ref) where 'ref' is a pointer to a kref field inside the private data structure. The private data may later be looked up by calling cgroup_priv_get(cgrp, key) to obtain a new reference to the private data. Private data may be unregistered via cgroup_priv_release(cgrp, key).

If a cgroup is removed, the reference count for all private data objects will be decremented.

v2: Significant overhaul suggested by Tejun, Alexei, and Roman
- Rework interface to make consumers obtain an ida-based key rather than supplying their own arbitrary void*
- Internal implementation now uses per-cgroup radixtrees which should allow much faster lookup than the previous hashtable approach
- Private data is registered via kref, allowing a single private data structure to potentially be assigned to multiple cgroups.

Cc Tejun Heo <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Roman Gushchin <[email protected]>
---
 include/linux/cgroup-defs.h |   8 ++
 include/linux/cgroup.h      |   7 ++
 kernel/cgroup/cgroup.c      | 185 +++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 197 insertions(+), 3 deletions(-)
include/linux/cgroup-defs.h +8 -0
index 9f242b876fde..465006274a84 100644
@@ -427,6 +427,14 @@ struct cgroup {
 	/* used to store eBPF programs */
 	struct cgroup_bpf bpf;
 
+	/*
+	 * cgroup private data registered by other non-controller parts of the
+	 * kernel.  Insertions are protected by privdata_lock, lookups by
+	 * rcu_read_lock().
+	 */
+	struct radix_tree_root privdata;
+	spinlock_t privdata_lock;
+
 	/* ids of the ancestors at each level including self */
 	int ancestor_ids[];
 };
include/linux/cgroup.h +7 -0
index 473e0c0abb86..63d22dfa00bd 100644
@@ -833,4 +833,11 @@ static inline void put_cgroup_ns(struct cgroup_namespace *ns)
 		free_cgroup_ns(ns);
 }
 
+/* cgroup private data handling */
+int cgroup_priv_getkey(void (*free)(struct kref *));
+void cgroup_priv_destroykey(int key);
+int cgroup_priv_install(struct cgroup *cgrp, int key, struct kref *ref);
+struct kref *cgroup_priv_get(struct cgroup *cgrp, int key);
+void cgroup_priv_release(struct cgroup *cgrp, int key);
+
 #endif /* _LINUX_CGROUP_H */
kernel/cgroup/cgroup.c +182 -3

213 lines not on this page · raw message

Signature · 6 lines
2.14.3

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#2 lore raw

[RFC PATCH] cgroup: cgroup_idr_lock can be static

Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Fixes d6de4e7e9e60 ("cgroup: Allow registration and lookup of cgroup private data (v2)")
---
 cgroup.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/cgroup/cgroup.c +1 -1
index c778830..9af6086 100644
@@ -91,7 +91,7 @@ static DEFINE_IDR(cgroup_priv_idr);
  * Protects cgroup_idr, css_idr, and cgroup_priv_idr so that IDs can be
  * released without grabbing cgroup_mutex.
  */
-DEFINE_SPINLOCK(cgroup_idr_lock);
+static DEFINE_SPINLOCK(cgroup_idr_lock);
 
 /*
  * Protects cgroup_file->kn for !self csses.  It synchronizes notifications

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#3 lore raw

Re: [Intel-gfx] [PATCH v4 1/8] cgroup: Allow registration and lookup of cgroup private data (v2)

In reply to Matt Roper
Thank you for the patch! Perhaps something to improve:

Hi Matt,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on next-20180309]
[also build test WARNING on v4.16-rc6]
[cannot apply to v4.16-rc4 v4.16-rc3 v4.16-rc2]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Matt-Roper/cgroup-private-data-and-DRM-i915-integration/20180319-071752
reproduce:

        # apt-get install sparse
        make ARCH=x86_64 allmodconfig
        make C=1 CF=-D__CHECK_ENDIAN__

sparse warnings: (new ones prefixed by >>)

>> kernel/cgroup/cgroup.c:94:1: sparse: symbol 'cgroup_idr_lock' was not declared. Should it be static?
>> kernel/cgroup/cgroup.c:4655:17: sparse: incorrect type in assignment (different address spaces) @@    expected void **slot @@    got void [noderef] <avoid **slot @@
   kernel/cgroup/cgroup.c:4655:17:    expected void **slot
   kernel/cgroup/cgroup.c:4655:17:    got void [noderef] <asn:4>**
>> kernel/cgroup/cgroup.c:4655:17: sparse: incorrect type in assignment (different address spaces) @@    expected void **slot @@    got void [noderef] <avoid **slot @@
   kernel/cgroup/cgroup.c:4655:17:    expected void **slot
   kernel/cgroup/cgroup.c:4655:17:    got void [noderef] <asn:4>**
>> kernel/cgroup/cgroup.c:4655:17: sparse: incorrect type in argument 1 (different address spaces) @@    expected void [noderef] <asn:4>**slot @@    got sn:4>**slot @@
   kernel/cgroup/cgroup.c:4655:17:    expected void [noderef] <asn:4>**slot
   kernel/cgroup/cgroup.c:4655:17:    got void **slot
>> kernel/cgroup/cgroup.c:4655:17: sparse: incorrect type in assignment (different address spaces) @@    expected void **slot @@    got void [noderef] <avoid **slot @@

kernel/cgroup/cgroup.c:4655:17: expected void **slot
kernel/cgroup/cgroup.c:4655:17: got void [noderef] <asn:4>**
kernel/cgroup/cgroup.c:2648:20: sparse: context imbalance in 'cgroup_procs_write_start' - wrong count at exit kernel/cgroup/cgroup.c:2704:9: sparse: context imbalance in 'cgroup_procs_write_finish' - wrong count at exit kernel/cgroup/cgroup.c:2815:9: sparse: context imbalance in 'cgroup_lock_and_drain_offline' - wrong count at exit kernel/cgroup/cgroup.c:4375:16: sparse: context imbalance in 'cgroup_procs_write' - wrong count at exit kernel/cgroup/cgroup.c:4416:16: sparse: context imbalance in 'cgroup_threads_write' - wrong count at exit

Please review and possibly fold the followup patch.

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

author
#4 lore raw

PATCH v4 2/8 cgroup: Introduce task_get_dfl_cgroup() (v2)

In reply to Matt Roper

Wraps task_dfl_cgroup() to also take a reference to the cgroup.

v2:
- Eliminate cgroup_mutex and make lighter-weight (Tejun)

---
 include/linux/cgroup.h | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
include/linux/cgroup.h +29 -0
index 63d22dfa00bd..74b435f913c1 100644
@@ -527,6 +527,35 @@ static inline struct cgroup *task_dfl_cgroup(struct task_struct *task)
 	return task_css_set(task)->dfl_cgrp;
 }
 
+/**
+ * task_get_dfl_cgroup() - find and get the cgroup for a task
+ * @task: the target task
+ *
+ * Find the cgroup in the v2 hierarchy that a task belongs to, increment its
+ * reference count, and return it.
+ *
+ * Returns:
+ * The appropriate cgroup from the default hierarchy.
+ */
+static inline struct cgroup *
+task_get_dfl_cgroup(struct task_struct *task)
+{
+	struct cgroup *cgrp;
+
+	rcu_read_lock();
+	while (true) {
+		cgrp = task_dfl_cgroup(task);
+
+		if (likely(cgroup_tryget(cgrp)))
+			break;
+
+		cpu_relax();
+	}
+	rcu_read_unlock();
+
+	return cgrp;
+}
+
 static inline struct cgroup *cgroup_parent(struct cgroup *cgrp)
 {
 	struct cgroup_subsys_state *parent_css = cgrp->self.parent;
Signature · 6 lines
2.14.3

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#5 lore raw

PATCH v4 3/8 cgroup: Introduce cgroup_priv_get_current

In reply to Matt Roper

Getting cgroup private data for the current process' cgroup is such a common pattern that we should add a convenience wrapper for it.

---
 include/linux/cgroup.h |  1 +
 kernel/cgroup/cgroup.c | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+)
include/linux/cgroup.h +1 -0
index 74b435f913c1..64d3dc45efd0 100644
@@ -867,6 +867,7 @@ int cgroup_priv_getkey(void (*free)(struct kref *));
 void cgroup_priv_destroykey(int key);
 int cgroup_priv_install(struct cgroup *cgrp, int key, struct kref *ref);
 struct kref *cgroup_priv_get(struct cgroup *cgrp, int key);
+struct kref *cgroup_priv_get_current(int key);
 void cgroup_priv_release(struct cgroup *cgrp, int key);
 
 #endif /* _LINUX_CGROUP_H */
kernel/cgroup/cgroup.c +23 -0
index a5e2017c9a94..56ed910beb8a 100644
@@ -6079,6 +6079,29 @@ cgroup_priv_get(struct cgroup *cgrp, int key)
 }
 EXPORT_SYMBOL_GPL(cgroup_priv_get);
 
+/**
+ * cgroup_priv_get_current - looks up cgroup private data for current task
+ * @key: key uniquely identifying owner of private data to lookup
+ *
+ * Convenience function that performs cgroup_priv_get() on the cgroup that owns
+ * %current.
+ *
+ * Returns:
+ * A pointer to the private data's kref field, or NULL if no private data has
+ * been registered.
+ */
+struct kref *
+cgroup_priv_get_current(int key)
+{
+	struct cgroup *cgrp = task_get_dfl_cgroup(current);
+	struct kref *ref = cgroup_priv_get(cgrp, key);
+
+	cgroup_put(cgrp);
+
+	return ref;
+}
+EXPORT_SYMBOL_GPL(cgroup_priv_get_current);
+
 /**
  * cgroup_priv_free - free cgroup private data
  * @cgrp: cgroup to release private data for
Signature · 6 lines
2.14.3

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
author
#6 lore raw

PATCH v4 4/8 drm/i915: Adjust internal priority definitions

In reply to Matt Roper

In preparation for adding cgroup-based priority adjustments, let's define the driver's priority values a little more clearly.

---
 drivers/gpu/drm/i915/i915_drv.h         |  1 -
 drivers/gpu/drm/i915/i915_gem_context.c |  4 ++--
 drivers/gpu/drm/i915/i915_request.h     | 13 ++++++++++---
 drivers/gpu/drm/i915/intel_display.c    |  2 +-
 4 files changed, 13 insertions(+), 7 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +0 -1
index e27ba8fb64e6..6ce7d0662bb6 100644
@@ -3150,7 +3150,6 @@ int i915_gem_object_wait(struct drm_i915_gem_object *obj,
 int i915_gem_object_wait_priority(struct drm_i915_gem_object *obj,
 				  unsigned int flags,
 				  int priority);
-#define I915_PRIORITY_DISPLAY I915_PRIORITY_MAX
 
 int __must_check
 i915_gem_object_set_to_wc_domain(struct drm_i915_gem_object *obj, bool write);
drivers/gpu/drm/i915/i915_gem_context.c +2 -2
index 5cfac0255758..551c1d75fe24 100644
@@ -474,7 +474,7 @@ int i915_gem_contexts_init(struct drm_i915_private *dev_priv)
 	ida_init(&dev_priv->contexts.hw_ida);
 
 	/* lowest priority; idle task */
-	ctx = i915_gem_context_create_kernel(dev_priv, I915_PRIORITY_MIN);
+	ctx = i915_gem_context_create_kernel(dev_priv, I915_PRIORITY_IDLE);
 	if (IS_ERR(ctx)) {
 		DRM_ERROR("Failed to create default global context\n");
 		return PTR_ERR(ctx);
@@ -488,7 +488,7 @@ int i915_gem_contexts_init(struct drm_i915_private *dev_priv)
 
 	/* highest priority; preempting task */
 	if (needs_preempt_context(dev_priv)) {
-		ctx = i915_gem_context_create_kernel(dev_priv, INT_MAX);
+		ctx = i915_gem_context_create_kernel(dev_priv, I915_PRIORITY_PREEMPT);
 		if (!IS_ERR(ctx))
 			dev_priv->preempt_context = ctx;
 		else
drivers/gpu/drm/i915/i915_request.h +10 -3
index 7d6eb82eeb91..72b13fc2b72b 100644
@@ -78,12 +78,19 @@ struct i915_priotree {
 	int priority;
 };
 
+/*
+ * Userspace can only assign priority values of [-1023,1023] via context param,
+ * but the effective priority value can fall in a larger range once we add in
+ * a cgroup-provided offset.
+ */
 enum {
-	I915_PRIORITY_MIN = I915_CONTEXT_MIN_USER_PRIORITY - 1,
 	I915_PRIORITY_NORMAL = I915_CONTEXT_DEFAULT_PRIORITY,
-	I915_PRIORITY_MAX = I915_CONTEXT_MAX_USER_PRIORITY + 1,
+	I915_PRIORITY_DEFAULT_DISPBOOST = I915_CONTEXT_MAX_USER_PRIORITY + 1,
 
-	I915_PRIORITY_INVALID = INT_MIN
+	/* Special case priority values */
+	I915_PRIORITY_INVALID = INT_MIN,
+	I915_PRIORITY_IDLE = INT_MIN + 1,
+	I915_PRIORITY_PREEMPT = INT_MAX,
 };
 
 struct i915_capture_list {
drivers/gpu/drm/i915/intel_display.c +1 -1
index 3e7ab75e1b41..b053a21f682b 100644
@@ -12783,7 +12783,7 @@ intel_prepare_plane_fb(struct drm_plane *plane,
 
 	ret = intel_plane_pin_fb(to_intel_plane_state(new_state));
 
-	i915_gem_object_wait_priority(obj, 0, I915_PRIORITY_DISPLAY);
+	i915_gem_object_wait_priority(obj, 0, I915_PRIORITY_DEFAULT_DISPBOOST);
 
 	mutex_unlock(&dev_priv->drm.struct_mutex);
 	i915_gem_object_unpin_pages(obj);
Signature · 6 lines
2.14.3

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#7 lore raw

PATCH v4 5/8 drm/i915: cgroup integration (v3)

In reply to Matt Roper

Introduce a new DRM_IOCTL_I915_CGROUP_SETPARAM ioctl that will allow userspace to set i915-specific parameters for individual cgroups. i915 cgroup data will be registered and later looked up via the new cgroup_priv infrastructure.

v2:
- Large rebase/rewrite for new cgroup_priv interface
v3:
- Another rebase/rewrite for ida-based keys and kref-based storage
- Access control no longer follows cgroup filesystem permissions; instead we restrict access to either DRM master or capable(CAP_SYS_RESOURCE).

---
 drivers/gpu/drm/i915/Makefile      |   1 +
 drivers/gpu/drm/i915/i915_cgroup.c | 140 +++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_drv.c    |   7 ++
 drivers/gpu/drm/i915/i915_drv.h    |  25 +++++++
 include/uapi/drm/i915_drm.h        |  12 ++++
 5 files changed, 185 insertions(+)
 create mode 100644 drivers/gpu/drm/i915/i915_cgroup.c
drivers/gpu/drm/i915/Makefile +1 -0
index 552e43e9663f..26031185cf0e 100644
@@ -48,6 +48,7 @@ i915-y := i915_drv.o \
 i915-$(CONFIG_COMPAT)   += i915_ioc32.o
 i915-$(CONFIG_DEBUG_FS) += i915_debugfs.o intel_pipe_crc.o
 i915-$(CONFIG_PERF_EVENTS) += i915_pmu.o
+i915-$(CONFIG_CGROUPS) += i915_cgroup.o
 
 # GEM code
 i915-y += i915_cmd_parser.o \
drivers/gpu/drm/i915/i915_cgroup.c new file +140 -0

140 lines not on this page · raw message

drivers/gpu/drm/i915/i915_drv.c +7 -0
index 3df5193487f3..99b571cb7ef8 100644
@@ -1407,6 +1407,10 @@ int i915_driver_load(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	intel_runtime_pm_put(dev_priv);
 
+	ret = i915_cgroup_init(dev_priv);
+	if (ret < 0)
+		goto out_cleanup_hw;
+
 	i915_welcome_messages(dev_priv);
 
 	return 0;
@@ -1433,6 +1437,8 @@ void i915_driver_unload(struct drm_device *dev)
 	struct drm_i915_private *dev_priv = to_i915(dev);
 	struct pci_dev *pdev = dev_priv->drm.pdev;
 
+	i915_cgroup_shutdown(dev_priv);
+
 	i915_driver_unregister(dev_priv);
 
 	if (i915_gem_suspend(dev_priv))
@@ -2834,6 +2840,7 @@ static const struct drm_ioctl_desc i915_ioctls[] = {
 	DRM_IOCTL_DEF_DRV(I915_PERF_ADD_CONFIG, i915_perf_add_config_ioctl, DRM_UNLOCKED|DRM_RENDER_ALLOW),
 	DRM_IOCTL_DEF_DRV(I915_PERF_REMOVE_CONFIG, i915_perf_remove_config_ioctl, DRM_UNLOCKED|DRM_RENDER_ALLOW),
 	DRM_IOCTL_DEF_DRV(I915_QUERY, i915_query_ioctl, DRM_UNLOCKED|DRM_RENDER_ALLOW),
+	DRM_IOCTL_DEF_DRV(I915_CGROUP_SETPARAM, i915_cgroup_setparam_ioctl, DRM_UNLOCKED|DRM_RENDER_ALLOW),
 };
 
 static struct drm_driver driver = {
drivers/gpu/drm/i915/i915_drv.h +25 -0
index 6ce7d0662bb6..b1862fe5e1a6 100644
@@ -1738,6 +1738,10 @@ struct drm_i915_private {
 
 	struct intel_ppat ppat;
 
+	/* cgroup private data */
+	int cgroup_priv_key;
+	struct mutex cgroup_lock;
+
 	/* Kernel Modesetting */
 
 	struct intel_crtc *plane_to_crtc_mapping[I915_MAX_PIPES];
@@ -2678,6 +2682,27 @@ intel_ggtt_update_needs_vtd_wa(struct drm_i915_private *dev_priv)
 int intel_sanitize_enable_ppgtt(struct drm_i915_private *dev_priv,
 				int enable_ppgtt);
 
+/* i915_cgroup.c */
+#ifdef CONFIG_CGROUPS
+int i915_cgroup_init(struct drm_i915_private *dev_priv);
+int i915_cgroup_setparam_ioctl(struct drm_device *dev, void *data,
+			       struct drm_file *file);
+void i915_cgroup_shutdown(struct drm_i915_private *dev_priv);
+#else
+static inline int
+i915_cgroup_init(struct drm_i915_private *dev_priv)
+{
+	return 0;
+}
+static inline void i915_cgroup_shutdown(struct drm_i915_private *dev_priv) {}
+static inline int
+i915_cgroup_setparam_ioctl(struct drm_device *dev, void *data,
+			   struct drm_file *file)
+{
+	return -EINVAL;
+}
+#endif
+
 /* i915_drv.c */
 void __printf(3, 4)
 __i915_printk(struct drm_i915_private *dev_priv, const char *level,
include/uapi/drm/i915_drm.h +12 -0
index 7f5634ce8e88..735128fa61de 100644
@@ -319,6 +319,7 @@ typedef struct _drm_i915_sarea {
 #define DRM_I915_PERF_ADD_CONFIG	0x37
 #define DRM_I915_PERF_REMOVE_CONFIG	0x38
 #define DRM_I915_QUERY			0x39
+#define DRM_I915_CGROUP_SETPARAM	0x3a
 
 #define DRM_IOCTL_I915_INIT		DRM_IOW( DRM_COMMAND_BASE + DRM_I915_INIT, drm_i915_init_t)
 #define DRM_IOCTL_I915_FLUSH		DRM_IO ( DRM_COMMAND_BASE + DRM_I915_FLUSH)
@@ -377,6 +378,7 @@ typedef struct _drm_i915_sarea {
 #define DRM_IOCTL_I915_PERF_ADD_CONFIG	DRM_IOW(DRM_COMMAND_BASE + DRM_I915_PERF_ADD_CONFIG, struct drm_i915_perf_oa_config)
 #define DRM_IOCTL_I915_PERF_REMOVE_CONFIG	DRM_IOW(DRM_COMMAND_BASE + DRM_I915_PERF_REMOVE_CONFIG, __u64)
 #define DRM_IOCTL_I915_QUERY			DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_QUERY, struct drm_i915_query)
+#define DRM_IOCTL_I915_CGROUP_SETPARAM		DRM_IOW(DRM_COMMAND_BASE + DRM_I915_CGROUP_SETPARAM, struct drm_i915_cgroup_param)
 
 /* Allow drivers to submit batchbuffers directly to hardware, relying
  * on the security mechanisms provided by hardware.
@@ -1717,6 +1719,16 @@ struct drm_i915_query_topology_info {
 	__u8 data[];
 };
 
+/**
+ * Structure to set i915 parameter on a cgroup.
+ */
+struct drm_i915_cgroup_param {
+	__s32 cgroup_fd;
+	__u32 flags;
+	__u64 param;
+	__s64 value;
+};
+
 #if defined(__cplusplus)
 }
 #endif
Signature · 6 lines
2.14.3

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#8 lore raw

PATCH v4 6/8 drm/i915: Introduce 'priority offset' for GPU contexts (v3)

In reply to Matt Roper

There are cases where a system integrator may wish to raise/lower the priority of GPU workloads being submitted by specific OS process(es), independently of how the software self-classifies its own priority. Exposing "priority offset" as an i915-specific cgroup parameter will enable such system-level configuration.

Normally GPU contexts start with a priority value of 0 (I915_CONTEXT_DEFAULT_PRIORITY) and then may be adjusted up/down from there via other mechanisms. We'd like to provide a system-level input to the priority decision that will be taken into consideration, even when userspace later attempts to set an absolute priority value via I915_CONTEXT_PARAM_PRIORITY. The priority offset introduced here provides a base value that will always be added to (or subtracted from) the software's self-assigned priority value.

This patch makes priority offset a cgroup-specific value; contexts will be created with a priority offset based on the cgroup membership of the process creating the context at the time the context is created. Note that priority offset is assigned at context creation time; migrating a process to a different cgroup or changing the offset associated with a cgroup will only affect new context creation and will not alter the behavior of existing contexts previously created by the process.

v2:
- Rebase onto new cgroup_priv API
- Use current instead of drm_file->pid to determine which process to lookup priority for. (Chris)
- Don't forget to subtract priority offset in context_getparam ioctl to make it match setparam behavior. (Chris)

v3:
- Rebase again onto new idr/kref-based cgroup_priv API
- Bound priority offset such that effective priority from settings on context + cgroup fall within [-0x7fffff, 0x7fffff]. (Chris)

---
 drivers/gpu/drm/i915/i915_cgroup.c      | 52 +++++++++++++++++++++++++++++++--
 drivers/gpu/drm/i915/i915_drv.h         | 15 +++++-----
 drivers/gpu/drm/i915/i915_gem_context.c |  7 +++--
 drivers/gpu/drm/i915/i915_gem_context.h |  9 ++++++
 drivers/gpu/drm/i915/i915_request.h     |  4 +++
 include/uapi/drm/i915_drm.h             |  1 +
 6 files changed, 76 insertions(+), 12 deletions(-)
drivers/gpu/drm/i915/i915_cgroup.c +50 -2
index eaa540b65efd..b58b243aef20 100644
@@ -10,6 +10,8 @@
 #include "i915_drv.h"
 
 struct i915_cgroup_data {
+	int priority_offset;
+
 	struct kref ref;
 };
 
@@ -54,7 +56,6 @@ i915_cgroup_shutdown(struct drm_i915_private *dev_priv)
  * Return i915 cgroup private data, creating and registering it if one doesn't
  * already exist for this cgroup.
  */
-__maybe_unused
 static struct i915_cgroup_data *
 get_or_create_cgroup_data(struct drm_i915_private *dev_priv,
 			  struct cgroup *cgrp)
@@ -98,9 +99,11 @@ i915_cgroup_setparam_ioctl(struct drm_device *dev,
 			   void *data,
 			   struct drm_file *file)
 {
+	struct drm_i915_private *dev_priv = to_i915(dev);
 	struct drm_i915_cgroup_param *req = data;
 	struct cgroup *cgrp;
-	int ret;
+	struct i915_cgroup_data *cgrpdata;
+	int ret = 0;
 
 	/* We don't actually support any flags yet. */
 	if (req->flags) {
@@ -127,14 +130,59 @@ i915_cgroup_setparam_ioctl(struct drm_device *dev,
 		goto out;
 	}
 
+	cgrpdata = get_or_create_cgroup_data(dev_priv, cgrp);
+	if (IS_ERR(cgrpdata)) {
+		ret = PTR_ERR(cgrpdata);
+		goto out;
+	}
+
 	switch (req->param) {
+	case I915_CGROUP_PARAM_PRIORITY_OFFSET:
+		if (req->value + I915_CONTEXT_MAX_USER_PRIORITY <= I915_PRIORITY_MAX &&
+		    req->value - I915_CONTEXT_MIN_USER_PRIORITY >= I915_PRIORITY_MIN) {
+			DRM_DEBUG_DRIVER("Setting cgroup priority offset to %lld\n",
+					 req->value);
+			cgrpdata->priority_offset = req->value;
+		} else {
+			DRM_DEBUG_DRIVER("Invalid cgroup priority offset %lld\n",
+					 req->value);
+			ret = -EINVAL;
+		}
+		break;
+
 	default:
 		DRM_DEBUG_DRIVER("Invalid cgroup parameter %lld\n", req->param);
 		ret = -EINVAL;
 	}
 
+	kref_put(&cgrpdata->ref, i915_cgroup_free);
+
 out:
 	cgroup_put(cgrp);
 
 	return ret;
 }
+
+/*
+ * Generator for simple getter functions that look up a cgroup private data
+ * field for the current task's cgroup.  It's safe to call these before
+ * a cgroup private data key has been registered; they'll just return the
+ * default value in that case.
+ */
+#define CGROUP_GET(name, field, def) \
+int i915_cgroup_get_current_##name(struct drm_i915_private *dev_priv)	\
+{									\
+	struct kref *ref;						\
+	int val = def;							\
+	if (!dev_priv->cgroup_priv_key) return def;			\
+	ref = cgroup_priv_get_current(dev_priv->cgroup_priv_key);	\
+	if (ref) {							\
+		val = cgrp_ref_to_i915(ref)->field;			\
+		kref_put(ref, i915_cgroup_free);			\
+	}								\
+	return val;							\
+}
+
+CGROUP_GET(prio_offset, priority_offset, 0)
+
+#undef CGROUP_GET
drivers/gpu/drm/i915/i915_drv.h +8 -7
index b1862fe5e1a6..fd5629593f4b 100644
@@ -2688,18 +2688,19 @@ int i915_cgroup_init(struct drm_i915_private *dev_priv);
 int i915_cgroup_setparam_ioctl(struct drm_device *dev, void *data,
 			       struct drm_file *file);
 void i915_cgroup_shutdown(struct drm_i915_private *dev_priv);
+int i915_cgroup_get_current_prio_offset(struct drm_i915_private *dev_priv);
 #else
-static inline int
-i915_cgroup_init(struct drm_i915_private *dev_priv)
+static inline void i915_cgroup_init(struct drm_i915_private *dev_priv) {}
+static inline void i915_cgroup_shutdown(struct drm_i915_private *dev_priv) {}
+static inline int i915_cgroup_setparam_ioctl(struct drm_device *dev, void *data,
+					     struct drm_file *file)
 {
-	return 0;
+	return -EINVAL;
 }
-static inline void i915_cgroup_shutdown(struct drm_i915_private *dev_priv) {}
 static inline int
-i915_cgroup_setparam_ioctl(struct drm_device *dev, void *data,
-			   struct drm_file *file)
+i915_cgroup_get_current_prio_offset(struct drm_i915_private *dev_priv)
 {
-	return -EINVAL;
+	return 0;
 }
 #endif
 
drivers/gpu/drm/i915/i915_gem_context.c +4 -3
index 551c1d75fe24..306886c1add8 100644
@@ -280,7 +280,8 @@ __create_hw_context(struct drm_i915_private *dev_priv,
 	kref_init(&ctx->ref);
 	list_add_tail(&ctx->link, &dev_priv->contexts.list);
 	ctx->i915 = dev_priv;
-	ctx->priority = I915_PRIORITY_NORMAL;
+	ctx->priority_offset = i915_cgroup_get_current_prio_offset(dev_priv);
+	ctx->priority = I915_PRIORITY_NORMAL + ctx->priority_offset;
 
 	INIT_RADIX_TREE(&ctx->handles_vma, GFP_KERNEL);
 	INIT_LIST_HEAD(&ctx->handles_list);
@@ -747,7 +748,7 @@ int i915_gem_context_getparam_ioctl(struct drm_device *dev, void *data,
 		args->value = i915_gem_context_is_bannable(ctx);
 		break;
 	case I915_CONTEXT_PARAM_PRIORITY:
-		args->value = ctx->priority;
+		args->value = ctx->priority - ctx->priority_offset;
 		break;
 	default:
 		ret = -EINVAL;
@@ -820,7 +821,7 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,
 				 !capable(CAP_SYS_NICE))
 				ret = -EPERM;
 			else
-				ctx->priority = priority;
+				ctx->priority = priority + ctx->priority_offset;
 		}
 		break;
 
drivers/gpu/drm/i915/i915_gem_context.h +9 -0
index 7854262ddfd9..c3b4fb54fbb6 100644
@@ -150,6 +150,15 @@ struct i915_gem_context {
 	 */
 	int priority;
 
+	/**
+	 * @priority_offset: priority offset
+	 *
+	 * A value, configured via cgroup, that sets the starting priority
+	 * of the context.  Any priority set explicitly via context parameter
+	 * will be added to the priority offset.
+	 */
+	int priority_offset;
+
 	/** ggtt_offset_bias: placement restriction for context objects */
 	u32 ggtt_offset_bias;
 
drivers/gpu/drm/i915/i915_request.h +4 -0
index 72b13fc2b72b..cf7a7147daf3 100644
@@ -87,6 +87,10 @@ enum {
 	I915_PRIORITY_NORMAL = I915_CONTEXT_DEFAULT_PRIORITY,
 	I915_PRIORITY_DEFAULT_DISPBOOST = I915_CONTEXT_MAX_USER_PRIORITY + 1,
 
+	/* Range reachable by combining user priority + cgroup offset */
+	I915_PRIORITY_MAX = 0x7fffff,
+	I915_PRIORITY_MIN = -I915_PRIORITY_MAX,
+
 	/* Special case priority values */
 	I915_PRIORITY_INVALID = INT_MIN,
 	I915_PRIORITY_IDLE = INT_MIN + 1,
include/uapi/drm/i915_drm.h +1 -0
index 735128fa61de..6b70f46d224e 100644
@@ -1726,6 +1726,7 @@ struct drm_i915_cgroup_param {
 	__s32 cgroup_fd;
 	__u32 flags;
 	__u64 param;
+#define I915_CGROUP_PARAM_PRIORITY_OFFSET	0x1
 	__s64 value;
 };
 
Signature · 6 lines
2.14.3

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#9 lore raw

PATCH v4 7/8 drm/i915: Introduce per-cgroup display boost setting

In reply to Matt Roper

Usually display-boosted contexts get treated as I915_CONTEXT_MAX_USER_PRIORITY+1, which prioritizes them above regular GPU contexts. Now that we allow a much larger range of effective priority values via per-cgroup priority offsets, a system administrator may want more detailed control over how much a process should be boosted by display operations (i.e., can a context from a cgroup with a low priority offset still be display boosted above contexts from a cgroup with a much higher priority offset? or are come cgroups more important than maintaining display rate?).

---
 drivers/gpu/drm/i915/i915_cgroup.c   | 17 +++++++++++++++++
 drivers/gpu/drm/i915/i915_drv.h      |  6 ++++++
 drivers/gpu/drm/i915/i915_request.h  |  1 +
 drivers/gpu/drm/i915/intel_display.c |  5 +++--
 include/uapi/drm/i915_drm.h          |  1 +
 5 files changed, 28 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_cgroup.c +17 -0
index b58b243aef20..868d455ffeb6 100644
@@ -11,6 +11,7 @@
 
 struct i915_cgroup_data {
 	int priority_offset;
+	int display_boost;
 
 	struct kref ref;
 };
@@ -75,6 +76,8 @@ get_or_create_cgroup_data(struct drm_i915_private *dev_priv,
 			goto out;
 		}
 
+		priv->display_boost = I915_PRIORITY_DEFAULT_DISPBOOST;
+
 		kref_init(&priv->ref);
 		cgroup_priv_install(cgrp, dev_priv->cgroup_priv_key,
 				    &priv->ref);
@@ -150,6 +153,19 @@ i915_cgroup_setparam_ioctl(struct drm_device *dev,
 		}
 		break;
 
+	case I915_CGROUP_PARAM_DISPBOOST_PRIORITY:
+		if (req->value < I915_PRIORITY_MAX_DISPBOOST &&
+		    req->value > I915_PRIORITY_MIN) {
+			DRM_DEBUG_DRIVER("Setting cgroup display boost priority to %lld\n",
+					 req->value);
+			cgrpdata->display_boost = req->value;
+		} else {
+			DRM_DEBUG_DRIVER("Invalid cgroup display boost priority %lld\n",
+					 req->value);
+			ret = -EINVAL;
+		}
+		break;
+
 	default:
 		DRM_DEBUG_DRIVER("Invalid cgroup parameter %lld\n", req->param);
 		ret = -EINVAL;
@@ -184,5 +200,6 @@ int i915_cgroup_get_current_##name(struct drm_i915_private *dev_priv)	\
 }
 
 CGROUP_GET(prio_offset, priority_offset, 0)
+CGROUP_GET(dispboost, display_boost, I915_PRIORITY_DEFAULT_DISPBOOST);
 
 #undef CGROUP_GET
drivers/gpu/drm/i915/i915_drv.h +6 -0
index fd5629593f4b..fe787cfcef72 100644
@@ -2689,6 +2689,7 @@ int i915_cgroup_setparam_ioctl(struct drm_device *dev, void *data,
 			       struct drm_file *file);
 void i915_cgroup_shutdown(struct drm_i915_private *dev_priv);
 int i915_cgroup_get_current_prio_offset(struct drm_i915_private *dev_priv);
+int i915_cgroup_get_current_dispboost(struct drm_i915_private *dev_priv);
 #else
 static inline void i915_cgroup_init(struct drm_i915_private *dev_priv) {}
 static inline void i915_cgroup_shutdown(struct drm_i915_private *dev_priv) {}
@@ -2702,6 +2703,11 @@ i915_cgroup_get_current_prio_offset(struct drm_i915_private *dev_priv)
 {
 	return 0;
 }
+static inline int
+i915_cgroup_get_current_dispboost(struct drm_i915_private *dev_priv)
+{
+	return I915_PRIORITY_DEFAULT_DISPBOOST;
+}
 #endif
 
 /* i915_drv.c */
drivers/gpu/drm/i915/i915_request.h +1 -0
index cf7a7147daf3..db300d93fd08 100644
@@ -90,6 +90,7 @@ enum {
 	/* Range reachable by combining user priority + cgroup offset */
 	I915_PRIORITY_MAX = 0x7fffff,
 	I915_PRIORITY_MIN = -I915_PRIORITY_MAX,
+	I915_PRIORITY_MAX_DISPBOOST = I915_PRIORITY_MAX + 1,
 
 	/* Special case priority values */
 	I915_PRIORITY_INVALID = INT_MIN,
drivers/gpu/drm/i915/intel_display.c +3 -2
index b053a21f682b..1d0245e2fd75 100644
@@ -12731,7 +12731,7 @@ intel_prepare_plane_fb(struct drm_plane *plane,
 	struct drm_framebuffer *fb = new_state->fb;
 	struct drm_i915_gem_object *obj = intel_fb_obj(fb);
 	struct drm_i915_gem_object *old_obj = intel_fb_obj(plane->state->fb);
-	int ret;
+	int boost, ret;
 
 	if (old_obj) {
 		struct drm_crtc_state *crtc_state =
@@ -12783,7 +12783,8 @@ intel_prepare_plane_fb(struct drm_plane *plane,
 
 	ret = intel_plane_pin_fb(to_intel_plane_state(new_state));
 
-	i915_gem_object_wait_priority(obj, 0, I915_PRIORITY_DEFAULT_DISPBOOST);
+	boost = i915_cgroup_get_current_dispboost(dev_priv);
+	i915_gem_object_wait_priority(obj, 0, boost);
 
 	mutex_unlock(&dev_priv->drm.struct_mutex);
 	i915_gem_object_unpin_pages(obj);
include/uapi/drm/i915_drm.h +1 -0
index 6b70f46d224e..ad454f121884 100644
@@ -1727,6 +1727,7 @@ struct drm_i915_cgroup_param {
 	__u32 flags;
 	__u64 param;
 #define I915_CGROUP_PARAM_PRIORITY_OFFSET	0x1
+#define I915_CGROUP_PARAM_DISPBOOST_PRIORITY    0x2
 	__s64 value;
 };
 
Signature · 6 lines
2.14.3

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#10 lore raw

PATCH v4 8/8 drm/i915: Add context priority & priority offset to debugfs (v2)

In reply to Matt Roper

Update i915_context_status to include priority information.

v2:
- Clarify that the offset is based on cgroup (Chris)

---
 drivers/gpu/drm/i915/i915_debugfs.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/i915/i915_debugfs.c +3 -0
index 5378863e3238..02d1983b8581 100644
@@ -1959,6 +1959,9 @@ static int i915_context_status(struct seq_file *m, void *unused)
 		seq_putc(m, ctx->remap_slice ? 'R' : 'r');
 		seq_putc(m, '\n');
 
+		seq_printf(m, "Priority %d (cgroup offset = %d)\n",
+			   ctx->priority, ctx->priority_offset);
+
 		for_each_engine(engine, dev_priv, id) {
 			struct intel_context *ce = &ctx->engine[engine->id];
 
Signature · 6 lines
2.14.3

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
author
#11 lore raw

PATCH 1/1 tests: Introduce drv_cgroup (v2)

In reply to Matt Roper

drv_cgroup exercises both valid and invalid usage of the i915 cgroup parameter ioctl.

v2:
- Add support for display boost
- Drop cgroup permissions test (the kernel no longer bases access control on fs permissions)

---
 tests/Makefile.sources |   1 +
 tests/drv_cgroup.c     | 262 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 263 insertions(+)
 create mode 100644 tests/drv_cgroup.c
tests/Makefile.sources +1 -0
index 4e6f5319..ba4b00bb 100644
@@ -37,6 +37,7 @@ TESTS_progs = \
 	drm_vma_limiter_cached \
 	drm_vma_limiter_cpu \
 	drm_vma_limiter_gtt \
+	drv_cgroup \
 	drv_getparams_basic \
 	drv_hangman \
 	drv_missed_irq \
tests/drv_cgroup.c new file +262 -0

262 lines not on this page · raw message

Signature · 6 lines
2.14.3

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#12 lore raw

✗ Fi.CI.CHECKPATCH: warning for cgroup private data and DRM/i915 integration

In reply to Matt Roper
== Series Details ==

== Series Details ==

Series: cgroup private data and DRM/i915 integration
URL : https://patchwork.freedesktop.org/series/40142/
State : warning

== Summary ==

$ dim checkpatch origin/drm-tip
ed1f8853712d cgroup: Allow registration and lookup of cgroup private data (v2)
-:52: CHECK:UNCOMMENTED_DEFINITION: spinlock_t definition without comment
#52: FILE: include/linux/cgroup-defs.h:436:
+	spinlock_t privdata_lock;

-:276: WARNING:FUNCTION_ARGUMENTS: function definition argument 'struct kref *' should also have an identifier name #276: FILE: kernel/cgroup/cgroup.c:6095:
+ void (*free)(struct kref *);

total: 0 errors, 1 warnings, 1 checks, 238 lines checked
a2cf2e58be10 cgroup: Introduce task_get_dfl_cgroup() (v2)
7837b0190516 cgroup: Introduce cgroup_priv_get_current
cd9b81bd5505 drm/i915: Adjust internal priority definitions
d4e5c99d4496 drm/i915: cgroup integration (v3)
-:34: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #34:
new file mode 100644

-:109: WARNING:SIZEOF_PARENTHESIS: sizeof *priv should be sizeof(*priv) #109: FILE: drivers/gpu/drm/i915/i915_cgroup.c:71:
+ priv = kzalloc(sizeof *priv, GFP_KERNEL);

-:207: WARNING:LONG_LINE: line over 100 characters #207: FILE: drivers/gpu/drm/i915/i915_drv.c:2843:
+ DRM_IOCTL_DEF_DRV(I915_CGROUP_SETPARAM, i915_cgroup_setparam_ioctl, DRM_UNLOCKED|DRM_RENDER_ALLOW),

-:207: CHECK:SPACING: spaces preferred around that '|' (ctx:VxV)
#207: FILE: drivers/gpu/drm/i915/i915_drv.c:2843:
+ DRM_IOCTL_DEF_DRV(I915_CGROUP_SETPARAM, i915_cgroup_setparam_ioctl, DRM_UNLOCKED|DRM_RENDER_ALLOW),
^

-:221: CHECK:UNCOMMENTED_DEFINITION: struct mutex definition without comment #221: FILE: drivers/gpu/drm/i915/i915_drv.h:1743:
+ struct mutex cgroup_lock;

-:242: CHECK:LINE_SPACING: Please use a blank line after function/struct/union/enum declarations #242: FILE: drivers/gpu/drm/i915/i915_drv.h:2697:
+}
+static inline void i915_cgroup_shutdown(struct drm_i915_private *dev_priv) {}

-:270: WARNING:LONG_LINE: line over 100 characters #270: FILE: include/uapi/drm/i915_drm.h:381:
+#define DRM_IOCTL_I915_CGROUP_SETPARAM DRM_IOW(DRM_COMMAND_BASE + DRM_I915_CGROUP_SETPARAM, struct drm_i915_cgroup_param)

total: 0 errors, 4 warnings, 3 checks, 239 lines checked 6d7ce25f32de drm/i915: Introduce 'priority offset' for GPU contexts (v3) -:121: CHECK:MACRO_ARG_REUSE: Macro argument reuse 'def' - possible side-effects? #121: FILE: drivers/gpu/drm/i915/i915_cgroup.c:172:
+#define CGROUP_GET(name, field, def) \
+int i915_cgroup_get_current_##name(struct drm_i915_private *dev_priv) \
+{ \
+ struct kref *ref; \
+ int val = def; \
+ if (!dev_priv->cgroup_priv_key) return def; \
+ ref = cgroup_priv_get_current(dev_priv->cgroup_priv_key); \
+ if (ref) { \
+ val = cgrp_ref_to_i915(ref)->field; \
+ kref_put(ref, i915_cgroup_free); \
+ } \
+ return val; \
+}

-:126: ERROR:TRAILING_STATEMENTS: trailing statements should be on next line #126: FILE: drivers/gpu/drm/i915/i915_cgroup.c:177:
+ if (!dev_priv->cgroup_priv_key) return def; \

total: 1 errors, 0 warnings, 1 checks, 169 lines checked
dac3bdd93cb9 drm/i915: Introduce per-cgroup display boost setting
-:83: CHECK:LINE_SPACING: Please use a blank line after function/struct/union/enum declarations #83: FILE: drivers/gpu/drm/i915/i915_drv.h:2706:
}
+static inline int

total: 0 errors, 0 warnings, 1 checks, 89 lines checked fa175d947839 drm/i915: Add context priority & priority offset to debugfs (v2)

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

maintainer
#13 lore raw

Re: ✗ Fi.CI.CHECKPATCH: warning for cgroup private data and DRM/i915 integration

Quoted 7 lines · Patchwork

== Series Details ==

Series: cgroup private data and DRM/i915 integration
URL : https://patchwork.freedesktop.org/series/40142/
State : warning

== Summary ==

Matt, please do look into these, I think most of these are valid.

Thanks,
Jani.

Quoted 82 lines
$ dim checkpatch origin/drm-tip
ed1f8853712d cgroup: Allow registration and lookup of cgroup private data (v2)
-:52: CHECK:UNCOMMENTED_DEFINITION: spinlock_t definition without comment
#52: FILE: include/linux/cgroup-defs.h:436:
(quoted diff) +1 -1
+	spinlock_t privdata_lock;
 
-:276: WARNING:FUNCTION_ARGUMENTS: function definition argument 'struct kref *' should also have an identifier name

#276: FILE: kernel/cgroup/cgroup.c:6095:

(quoted diff) +1 -0
+	void (*free)(struct kref *);

total: 0 errors, 1 warnings, 1 checks, 238 lines checked
a2cf2e58be10 cgroup: Introduce task_get_dfl_cgroup() (v2)
7837b0190516 cgroup: Introduce cgroup_priv_get_current
cd9b81bd5505 drm/i915: Adjust internal priority definitions
d4e5c99d4496 drm/i915: cgroup integration (v3)
-:34: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #34:
new file mode 100644

-:109: WARNING:SIZEOF_PARENTHESIS: sizeof *priv should be sizeof(*priv) #109: FILE: drivers/gpu/drm/i915/i915_cgroup.c:71:

(quoted diff) +1 -1
+		priv = kzalloc(sizeof *priv, GFP_KERNEL);
 
-:207: WARNING:LONG_LINE: line over 100 characters

#207: FILE: drivers/gpu/drm/i915/i915_drv.c:2843:

(quoted diff) +1 -1
+	DRM_IOCTL_DEF_DRV(I915_CGROUP_SETPARAM, i915_cgroup_setparam_ioctl, DRM_UNLOCKED|DRM_RENDER_ALLOW),
 
-:207: CHECK:SPACING: spaces preferred around that '|' (ctx:VxV)

#207: FILE: drivers/gpu/drm/i915/i915_drv.c:2843:

(quoted diff) +1 -1
+	DRM_IOCTL_DEF_DRV(I915_CGROUP_SETPARAM, i915_cgroup_setparam_ioctl, DRM_UNLOCKED|DRM_RENDER_ALLOW),
 	                                                                                ^
 
-:221: CHECK:UNCOMMENTED_DEFINITION: struct mutex definition without comment

#221: FILE: drivers/gpu/drm/i915/i915_drv.h:1743:

(quoted diff) +1 -1
+	struct mutex cgroup_lock;
 
-:242: CHECK:LINE_SPACING: Please use a blank line after function/struct/union/enum declarations

#242: FILE: drivers/gpu/drm/i915/i915_drv.h:2697:

(quoted diff) +2 -1
+}
+static inline void i915_cgroup_shutdown(struct drm_i915_private *dev_priv) {}
 
-:270: WARNING:LONG_LINE: line over 100 characters

#270: FILE: include/uapi/drm/i915_drm.h:381:
+#define DRM_IOCTL_I915_CGROUP_SETPARAM DRM_IOW(DRM_COMMAND_BASE + DRM_I915_CGROUP_SETPARAM, struct drm_i915_cgroup_param)

total: 0 errors, 4 warnings, 3 checks, 239 lines checked 6d7ce25f32de drm/i915: Introduce 'priority offset' for GPU contexts (v3) -:121: CHECK:MACRO_ARG_REUSE: Macro argument reuse 'def' - possible side-effects? #121: FILE: drivers/gpu/drm/i915/i915_cgroup.c:172:

(quoted diff) +13 -1
+#define CGROUP_GET(name, field, def) \
+int i915_cgroup_get_current_##name(struct drm_i915_private *dev_priv)	\
+{									\
+	struct kref *ref;						\
+	int val = def;							\
+	if (!dev_priv->cgroup_priv_key) return def;			\
+	ref = cgroup_priv_get_current(dev_priv->cgroup_priv_key);	\
+	if (ref) {							\
+		val = cgrp_ref_to_i915(ref)->field;			\
+		kref_put(ref, i915_cgroup_free);			\
+	}								\
+	return val;							\
+}
 
-:126: ERROR:TRAILING_STATEMENTS: trailing statements should be on next line

#126: FILE: drivers/gpu/drm/i915/i915_cgroup.c:177:

(quoted diff) +1 -0
+	if (!dev_priv->cgroup_priv_key) return def;			\

total: 1 errors, 0 warnings, 1 checks, 169 lines checked
dac3bdd93cb9 drm/i915: Introduce per-cgroup display boost setting
-:83: CHECK:LINE_SPACING: Please use a blank line after function/struct/union/enum declarations #83: FILE: drivers/gpu/drm/i915/i915_drv.h:2706:
}
+static inline int

total: 0 errors, 0 warnings, 1 checks, 89 lines checked fa175d947839 drm/i915: Add context priority & priority offset to debugfs (v2)

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Signature · 5 lines
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#14 lore raw

✓ Fi.CI.BAT: success for cgroup private data and DRM/i915 integration

In reply to Matt Roper
== Series Details ==

== Series Details ==

Series: cgroup private data and DRM/i915 integration
URL : https://patchwork.freedesktop.org/series/40142/
State : success

== Summary ==

Series 40142v1 cgroup private data and DRM/i915 integration
https://patchwork.freedesktop.org/api/1.0/series/40142/revisions/1/mbox/

---- Known issues:

Test kms_pipe_crc_basic:

        Subgroup suspend-read-crc-pipe-b:
                incomplete -> PASS       (fi-snb-2520m) fdo#103713

fdo#103713 https://bugs.freedesktop.org/show_bug.cgi?id=103713

fi-bdw-5557u     total:285  pass:264  dwarn:0   dfail:0   fail:0   skip:21  time:432s
fi-bdw-gvtdvm    total:285  pass:261  dwarn:0   dfail:0   fail:0   skip:24  time:442s
fi-blb-e6850     total:285  pass:220  dwarn:1   dfail:0   fail:0   skip:64  time:383s
fi-bsw-n3050     total:285  pass:239  dwarn:0   dfail:0   fail:0   skip:46  time:538s
fi-bwr-2160      total:285  pass:180  dwarn:0   dfail:0   fail:0   skip:105 time:297s
fi-bxt-dsi       total:285  pass:255  dwarn:0   dfail:0   fail:0   skip:30  time:516s
fi-bxt-j4205     total:285  pass:256  dwarn:0   dfail:0   fail:0   skip:29  time:513s
fi-byt-j1900     total:285  pass:250  dwarn:0   dfail:0   fail:0   skip:35  time:517s
fi-byt-n2820     total:285  pass:246  dwarn:0   dfail:0   fail:0   skip:39  time:515s
fi-cfl-8700k     total:285  pass:257  dwarn:0   dfail:0   fail:0   skip:28  time:417s
fi-cfl-s2        total:285  pass:259  dwarn:0   dfail:0   fail:0   skip:26  time:590s
fi-cfl-u         total:285  pass:259  dwarn:0   dfail:0   fail:0   skip:26  time:511s
fi-cnl-drrs      total:285  pass:254  dwarn:3   dfail:0   fail:0   skip:28  time:527s
fi-cnl-y3        total:285  pass:259  dwarn:0   dfail:0   fail:0   skip:26  time:587s
fi-elk-e7500     total:285  pass:225  dwarn:1   dfail:0   fail:0   skip:59  time:429s
fi-gdg-551       total:285  pass:176  dwarn:0   dfail:0   fail:1   skip:108 time:318s
fi-glk-1         total:285  pass:257  dwarn:0   dfail:0   fail:0   skip:28  time:542s
fi-hsw-4770      total:285  pass:258  dwarn:0   dfail:0   fail:0   skip:27  time:408s
fi-ilk-650       total:285  pass:225  dwarn:0   dfail:0   fail:0   skip:60  time:421s
fi-ivb-3520m     total:285  pass:256  dwarn:0   dfail:0   fail:0   skip:29  time:474s
fi-ivb-3770      total:285  pass:252  dwarn:0   dfail:0   fail:0   skip:33  time:428s
fi-kbl-7500u     total:285  pass:260  dwarn:1   dfail:0   fail:0   skip:24  time:471s
fi-kbl-7567u     total:285  pass:265  dwarn:0   dfail:0   fail:0   skip:20  time:466s
fi-kbl-r         total:285  pass:258  dwarn:0   dfail:0   fail:0   skip:27  time:518s
fi-pnv-d510      total:285  pass:219  dwarn:1   dfail:0   fail:0   skip:65  time:657s
fi-skl-6260u     total:285  pass:265  dwarn:0   dfail:0   fail:0   skip:20  time:452s
fi-skl-6600u     total:285  pass:258  dwarn:0   dfail:0   fail:0   skip:27  time:542s
fi-skl-6700hq    total:285  pass:259  dwarn:0   dfail:0   fail:0   skip:26  time:541s
fi-skl-6700k2    total:285  pass:261  dwarn:0   dfail:0   fail:0   skip:24  time:504s
fi-skl-6770hq    total:285  pass:265  dwarn:0   dfail:0   fail:0   skip:20  time:492s
fi-skl-guc       total:285  pass:257  dwarn:0   dfail:0   fail:0   skip:28  time:429s
fi-skl-gvtdvm    total:285  pass:262  dwarn:0   dfail:0   fail:0   skip:23  time:446s
fi-snb-2520m     total:285  pass:245  dwarn:0   dfail:0   fail:0   skip:40  time:587s
fi-snb-2600      total:285  pass:245  dwarn:0   dfail:0   fail:0   skip:40  time:399s

639c78d2805bec2022435a8184db27a3b9767fe7 drm-tip: 2018y-03m-16d-23h-05m-00s UTC integration manifest fa175d947839 drm/i915: Add context priority & priority offset to debugfs (v2) dac3bdd93cb9 drm/i915: Introduce per-cgroup display boost setting
6d7ce25f32de drm/i915: Introduce 'priority offset' for GPU contexts (v3) d4e5c99d4496 drm/i915: cgroup integration (v3)
cd9b81bd5505 drm/i915: Adjust internal priority definitions
7837b0190516 cgroup: Introduce cgroup_priv_get_current
a2cf2e58be10 cgroup: Introduce task_get_dfl_cgroup() (v2)
ed1f8853712d cgroup: Allow registration and lookup of cgroup private data (v2)

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_8386/issues.html
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#15 lore raw

✓ Fi.CI.BAT: success for tests: Introduce drv_cgroup (v2)

In reply to Matt Roper
== Series Details ==

== Series Details ==

Series: tests: Introduce drv_cgroup (v2)
URL : https://patchwork.freedesktop.org/series/40143/
State : success

== Summary ==

IGT patchset tested on top of latest successful build 98f7614bd725afaae48f7b70d18329149075661b lib: Parse plane IN_FORMATS blobifiers into a nicer form

with latest DRM-Tip kernel build CI_DRM_3944
639c78d2805b drm-tip: 2018y-03m-16d-23h-05m-00s UTC integration manifest

No testlist changes.

---- Known issues:

Test gem_mmap_gtt:

        Subgroup basic-small-bo-tiledx:
                fail       -> PASS       (fi-gdg-551) fdo#102575
Test kms_pipe_crc_basic:
        Subgroup suspend-read-crc-pipe-a:
                pass       -> INCOMPLETE (fi-bdw-5557u) fdo#104944
        Subgroup suspend-read-crc-pipe-b:
                incomplete -> PASS       (fi-snb-2520m) fdo#103713

fdo#102575 https://bugs.freedesktop.org/show_bug.cgi?id=102575
fdo#104944 https://bugs.freedesktop.org/show_bug.cgi?id=104944
fdo#103713 https://bugs.freedesktop.org/show_bug.cgi?id=103713

fi-bdw-5557u     total:241  pass:223  dwarn:0   dfail:0   fail:0   skip:17 
fi-bdw-gvtdvm    total:285  pass:261  dwarn:0   dfail:0   fail:0   skip:24  time:444s
fi-blb-e6850     total:285  pass:220  dwarn:1   dfail:0   fail:0   skip:64  time:383s
fi-bsw-n3050     total:285  pass:239  dwarn:0   dfail:0   fail:0   skip:46  time:545s
fi-bwr-2160      total:285  pass:180  dwarn:0   dfail:0   fail:0   skip:105 time:296s
fi-bxt-dsi       total:285  pass:255  dwarn:0   dfail:0   fail:0   skip:30  time:512s
fi-bxt-j4205     total:285  pass:256  dwarn:0   dfail:0   fail:0   skip:29  time:513s
fi-byt-j1900     total:285  pass:250  dwarn:0   dfail:0   fail:0   skip:35  time:518s
fi-byt-n2820     total:285  pass:246  dwarn:0   dfail:0   fail:0   skip:39  time:502s
fi-cfl-8700k     total:285  pass:257  dwarn:0   dfail:0   fail:0   skip:28  time:410s
fi-cfl-s2        total:285  pass:259  dwarn:0   dfail:0   fail:0   skip:26  time:582s
fi-cfl-u         total:285  pass:259  dwarn:0   dfail:0   fail:0   skip:26  time:511s
fi-cnl-drrs      total:285  pass:254  dwarn:3   dfail:0   fail:0   skip:28  time:521s
fi-cnl-y3        total:285  pass:259  dwarn:0   dfail:0   fail:0   skip:26  time:584s
fi-elk-e7500     total:285  pass:225  dwarn:1   dfail:0   fail:0   skip:59  time:429s
fi-gdg-551       total:285  pass:177  dwarn:0   dfail:0   fail:0   skip:108 time:318s
fi-glk-1         total:285  pass:257  dwarn:0   dfail:0   fail:0   skip:28  time:536s
fi-hsw-4770      total:285  pass:258  dwarn:0   dfail:0   fail:0   skip:27  time:406s
fi-ilk-650       total:285  pass:225  dwarn:0   dfail:0   fail:0   skip:60  time:426s
fi-ivb-3520m     total:285  pass:256  dwarn:0   dfail:0   fail:0   skip:29  time:474s
fi-ivb-3770      total:285  pass:252  dwarn:0   dfail:0   fail:0   skip:33  time:434s
fi-kbl-7500u     total:285  pass:260  dwarn:1   dfail:0   fail:0   skip:24  time:476s
fi-kbl-7567u     total:285  pass:265  dwarn:0   dfail:0   fail:0   skip:20  time:469s
fi-kbl-r         total:285  pass:258  dwarn:0   dfail:0   fail:0   skip:27  time:512s
fi-pnv-d510      total:285  pass:219  dwarn:1   dfail:0   fail:0   skip:65  time:660s
fi-skl-6260u     total:285  pass:265  dwarn:0   dfail:0   fail:0   skip:20  time:444s
fi-skl-6600u     total:285  pass:258  dwarn:0   dfail:0   fail:0   skip:27  time:563s
fi-skl-6700hq    total:285  pass:259  dwarn:0   dfail:0   fail:0   skip:26  time:539s
fi-skl-6700k2    total:285  pass:261  dwarn:0   dfail:0   fail:0   skip:24  time:509s
fi-skl-6770hq    total:285  pass:265  dwarn:0   dfail:0   fail:0   skip:20  time:489s
fi-skl-guc       total:285  pass:257  dwarn:0   dfail:0   fail:0   skip:28  time:427s
fi-skl-gvtdvm    total:285  pass:262  dwarn:0   dfail:0   fail:0   skip:23  time:446s
fi-snb-2520m     total:285  pass:245  dwarn:0   dfail:0   fail:0   skip:40  time:595s
fi-snb-2600      total:285  pass:245  dwarn:0   dfail:0   fail:0   skip:40  time:400s

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1158/issues.html
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

maintainer
#16 lore raw
In reply to Matt Roper

Quoting Matt Roper (2018-03-17 02:08:57)

Quoted 11 lines

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

Hi Matt,

For cross-subsystem changes such as this, it makes sense to Cc all relevant maintainers, especially if there have been previous comments to earlier revisions.

Please, do include and keep a reference to the userspace portion of the changes when you suggest new uAPI to be added. At least I have some trouble trying to track down the relevant interface consumer here.

I'm unsure how much sense it makes to commence with detailed i915 review if we will be blocked by lack of userspace after that? I'm assuming you've read through [1] already.

Regards, Joonas

[1] https://01.org/linuxgraphics/gfx-docs/drm/gpu/drm-uapi.html#open-source-userspace-requirements
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#17 lore raw
Quoted 19 lines · Joonas Lahtinen

Quoting Matt Roper (2018-03-17 02:08:57)

Quoted 11 lines

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

Hi Matt,

For cross-subsystem changes such as this, it makes sense to Cc all relevant maintainers, especially if there have been previous comments to earlier revisions.

Quoted 7 lines

Please, do include and keep a reference to the userspace portion of the changes when you suggest new uAPI to be added. At least I have some trouble trying to track down the relevant interface consumer here.

I'm unsure how much sense it makes to commence with detailed i915 review if we will be blocked by lack of userspace after that? I'm assuming you've read through [1] already.

Hi Joonas,

I've sent the userspace code out a few times, but it looks like I forgot to include a copy with v4/v4.5. Here's the version I provided with v3:
https://lists.freedesktop.org/archives/intel-gfx/2018-March/157935.html

Usually we don't consider things like i-g-t to be sufficient userspace consumers because we need a real-world consumer rather than a "toy" userspace. However in this case, the i-g-t tool, although very simple, is really the only userspace consumer I expect there to ever be. Ultimately the true consumer of this cgroups work are bash scripts, sysv init scripts, systemd recipes, etc. that just need a very simple tool to assign the specific values that make sense on a given system. There's no expectation that graphics clients or display servers would ever need to make use of these interfaces.

Matt

Quoted 4 lines

Regards, Joonas

[1] https://01.org/linuxgraphics/gfx-docs/drm/gpu/drm-uapi.html#open-source-userspace-requirements

Signature · 9 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
maintainer
#18 lore raw

Quoting Matt Roper (2018-03-23 17:46:16)

Quoted 28 lines · Joonas Lahtinen
Quoted 19 lines

Quoting Matt Roper (2018-03-17 02:08:57)

Quoted 11 lines

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

Hi Matt,

For cross-subsystem changes such as this, it makes sense to Cc all relevant maintainers, especially if there have been previous comments to earlier revisions.

Quoted 7 lines

Please, do include and keep a reference to the userspace portion of the changes when you suggest new uAPI to be added. At least I have some trouble trying to track down the relevant interface consumer here.

I'm unsure how much sense it makes to commence with detailed i915 review if we will be blocked by lack of userspace after that? I'm assuming you've read through [1] already.

Quoted 5 lines

Hi Joonas,

I've sent the userspace code out a few times, but it looks like I forgot to include a copy with v4/v4.5. Here's the version I provided with v3:
https://lists.freedesktop.org/archives/intel-gfx/2018-March/157935.html

Thanks. Keeping that in the relevant commit message of the patch that introduces the new uAPI will make it harder to forget and easiest for git blame, too.

Quoted 10 lines

Usually we don't consider things like i-g-t to be sufficient userspace consumers because we need a real-world consumer rather than a "toy" userspace. However in this case, the i-g-t tool, although very simple, is really the only userspace consumer I expect there to ever be. Ultimately the true consumer of this cgroups work are bash scripts, sysv init scripts, systemd recipes, etc. that just need a very simple tool to assign the specific values that make sense on a given system. There's no expectation that graphics clients or display servers would ever need to make use of these interfaces.

I was under the impression that a bit more generic GPU cgroups support was receiving a lot of support in the early discussion? A dedicated intel_cgroup sounds underwhelming, when comparing to idea of "gpu_nice", for user adoption :)

Also, I might not be up-to-date about all things cgroups, but the way intel_cgroup works, feels bit forced. We create a userspace context just to communicate with the driver and the IOCTL will still have global effects. I can't but think that i915 reading from the cgroups subsystem for the current process would feel more intuitive to me.

Does the implementation mimic some existing cgroups tool or de-facto way of doing things in cgroups world?

Regards, Joonas
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

author
#19 lore raw
Quoted 50 lines · Joonas Lahtinen

Quoting Matt Roper (2018-03-23 17:46:16)

Quoted 28 lines · Joonas Lahtinen
Quoted 19 lines

Quoting Matt Roper (2018-03-17 02:08:57)

Quoted 11 lines

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

Hi Matt,

For cross-subsystem changes such as this, it makes sense to Cc all relevant maintainers, especially if there have been previous comments to earlier revisions.

Quoted 7 lines

Please, do include and keep a reference to the userspace portion of the changes when you suggest new uAPI to be added. At least I have some trouble trying to track down the relevant interface consumer here.

I'm unsure how much sense it makes to commence with detailed i915 review if we will be blocked by lack of userspace after that? I'm assuming you've read through [1] already.

Quoted 5 lines

Hi Joonas,

I've sent the userspace code out a few times, but it looks like I forgot to include a copy with v4/v4.5. Here's the version I provided with v3:
https://lists.freedesktop.org/archives/intel-gfx/2018-March/157935.html

Thanks. Keeping that in the relevant commit message of the patch that introduces the new uAPI will make it harder to forget and easiest for git blame, too.

Quoted 10 lines

Usually we don't consider things like i-g-t to be sufficient userspace consumers because we need a real-world consumer rather than a "toy" userspace. However in this case, the i-g-t tool, although very simple, is really the only userspace consumer I expect there to ever be. Ultimately the true consumer of this cgroups work are bash scripts, sysv init scripts, systemd recipes, etc. that just need a very simple tool to assign the specific values that make sense on a given system. There's no expectation that graphics clients or display servers would ever need to make use of these interfaces.

Quoted 4 lines

I was under the impression that a bit more generic GPU cgroups support was receiving a lot of support in the early discussion? A dedicated intel_cgroup sounds underwhelming, when comparing to idea of "gpu_nice", for user adoption :)

I'm open to moving the cgroup_priv registration/lookup to the DRM core if other drivers are interested in using this mechanism and if we can come to an agreement on a standard priority offset range to support, how display boost should work for all drivers, etc. There might be some challenges mapping a DRM-defined priority range down to a different range that makes sense for individual driver schedulers, especially since some drivers already expose a different priority scheme to userspace via other interfaces like i915 does with GEM context priority.

So far I haven't really heard any interest outside the Intel camp, but hopefully other driver teams can speak up if they're for/against this. I don't want to try to artificially standardize this if other drivers want to go a different direction with priority/scheduling that's too different from the current Intel-driven design.

Quoted 5 lines

Also, I might not be up-to-date about all things cgroups, but the way intel_cgroup works, feels bit forced. We create a userspace context just to communicate with the driver and the IOCTL will still have global effects. I can't but think that i915 reading from the cgroups subsystem for the current process would feel more intuitive to me.

I think you're referring to the earlier discussion about exposing priority directly via the cgroups filesystem? That would certainly be simpler from a userspace perspective, but it's not the direction that the cgroups maintainer wants to see things go. Adding files directly to the cgroups filesystem is supposed to be something that's reserved for official cgroups controllers. The GPU priority concept we're trying to add here doesn't align with the requirements for creating a controller, so the preferred approach is to create a custom interface (syscall or ioctl) that simply takes a cgroup as a parameter. There's precendent with similar interfaces in areas like BPF (where the bpf() system call can accept a cgroup as a parameter and then perform its own private policy changes as it sees fit).

Using a true cgroups controller and exposing settings via the filesystem is likely still the way we'll want to go for some other types of cgroups-based policy in the future (e.g., limiting GPU memory usage); it just isn't the appropriate direction for priority.

Quoted 2 lines

Does the implementation mimic some existing cgroups tool or de-facto way of doing things in cgroups world?

The ioctl approach I took is similar to syscall approach that the BPF guys use to attach BPF programs to a cgroup. I'm not very familiar with BPF or how it gets used from userspace, so I'm not sure whether the interface is intended for one specific tool (like ours is), or whether there's more variety for userspace consumers.

Matt

Quoted 6 lines

Regards, Joonas
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 9 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
maintainer
#20 lore raw

DRM cgroups integration (Was: Re: [PATCH v4 0/8] cgroup private data and DRM/i915 integration)

+ Dave for commenting from DRM subsystem perspective. I strongly believe there would be benefit from agreeing on some foundation of DRM subsystem level program GPU niceness [-20,19] and memory limit [0,N] pages.

Quoting Matt Roper (2018-03-30 03:43:13)

Quoted 65 lines · Joonas Lahtinen
Quoted 50 lines

Quoting Matt Roper (2018-03-23 17:46:16)

Quoted 28 lines · Joonas Lahtinen
Quoted 19 lines

Quoting Matt Roper (2018-03-17 02:08:57)

Quoted 11 lines

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

Hi Matt,

For cross-subsystem changes such as this, it makes sense to Cc all relevant maintainers, especially if there have been previous comments to earlier revisions.

Quoted 7 lines

Please, do include and keep a reference to the userspace portion of the changes when you suggest new uAPI to be added. At least I have some trouble trying to track down the relevant interface consumer here.

I'm unsure how much sense it makes to commence with detailed i915 review if we will be blocked by lack of userspace after that? I'm assuming you've read through [1] already.

Quoted 5 lines

Hi Joonas,

I've sent the userspace code out a few times, but it looks like I forgot to include a copy with v4/v4.5. Here's the version I provided with v3:
https://lists.freedesktop.org/archives/intel-gfx/2018-March/157935.html

Thanks. Keeping that in the relevant commit message of the patch that introduces the new uAPI will make it harder to forget and easiest for git blame, too.

Quoted 10 lines

Usually we don't consider things like i-g-t to be sufficient userspace consumers because we need a real-world consumer rather than a "toy" userspace. However in this case, the i-g-t tool, although very simple, is really the only userspace consumer I expect there to ever be. Ultimately the true consumer of this cgroups work are bash scripts, sysv init scripts, systemd recipes, etc. that just need a very simple tool to assign the specific values that make sense on a given system. There's no expectation that graphics clients or display servers would ever need to make use of these interfaces.

Quoted 4 lines

I was under the impression that a bit more generic GPU cgroups support was receiving a lot of support in the early discussion? A dedicated intel_cgroup sounds underwhelming, when comparing to idea of "gpu_nice", for user adoption :)

I'm open to moving the cgroup_priv registration/lookup to the DRM core if other drivers are interested in using this mechanism and if we can come to an agreement on a standard priority offset range to support, how display boost should work for all drivers, etc. There might be some challenges mapping a DRM-defined priority range down to a different range that makes sense for individual driver schedulers, especially since some drivers already expose a different priority scheme to userspace via other interfaces like i915 does with GEM context priority.

Quoted 5 lines

So far I haven't really heard any interest outside the Intel camp, but hopefully other driver teams can speak up if they're for/against this. I don't want to try to artificially standardize this if other drivers want to go a different direction with priority/scheduling that's too different from the current Intel-driven design.

I don't think there are that many directions to go about GPU context priority, considering we have the EGL_IMG_context_priority extension, so it'll only be about granularity of the scale.

I would suggest to go with the nice like scale for easy user adoption, then just apply that as the N most significant bits.

The contexts could then of course further adjust their priority from what is set by the "gpu_nice" application with the remaining bits.

I'm strongly feeling this should be a DRM level "gpu_nice". And the binding to cgroups should come through DRM core. If it doesn't, limiting the amount of memory used becomes awkward as the allocation is centralized to DRM core.

Quoted 19 lines
Quoted 5 lines

Also, I might not be up-to-date about all things cgroups, but the way intel_cgroup works, feels bit forced. We create a userspace context just to communicate with the driver and the IOCTL will still have global effects. I can't but think that i915 reading from the cgroups subsystem for the current process would feel more intuitive to me.

I think you're referring to the earlier discussion about exposing priority directly via the cgroups filesystem? That would certainly be simpler from a userspace perspective, but it's not the direction that the cgroups maintainer wants to see things go. Adding files directly to the cgroups filesystem is supposed to be something that's reserved for official cgroups controllers. The GPU priority concept we're trying to add here doesn't align with the requirements for creating a controller, so the preferred approach is to create a custom interface (syscall or ioctl) that simply takes a cgroup as a parameter. There's precendent with similar interfaces in areas like BPF (where the bpf() system call can accept a cgroup as a parameter and then perform its own private policy changes as it sees fit).

Quoted 4 lines

Using a true cgroups controller and exposing settings via the filesystem is likely still the way we'll want to go for some other types of cgroups-based policy in the future (e.g., limiting GPU memory usage); it just isn't the appropriate direction for priority.

Might be just me but feels bit crazy to be setting GPU memory usage through another interface and then doing i915 specific IOCTLs to control the priority of that same cgroup.

I don't feel comfortable adding custom cgroups dependent IOCTLs to i915 where cgroups is only working as the variable carrier in background. We should really just be consuming a variable from cgroups and it should be set outside of of the i915 IOCTL interface.

I'm still seeing that we should have a DRM cgroups controller and a DRM subsystem wide application to control the priority and memory usage to be fed to the drivers.

If we end up just supporting i915 apps, we could as well use LD_PRELOAD wrapper and alter the context priority at creation time for exactly the same effect and no extra interfaces to maintain.

Quoted 8 lines
Quoted 2 lines

Does the implementation mimic some existing cgroups tool or de-facto way of doing things in cgroups world?

The ioctl approach I took is similar to syscall approach that the BPF guys use to attach BPF programs to a cgroup. I'm not very familiar with BPF or how it gets used from userspace, so I'm not sure whether the interface is intended for one specific tool (like ours is), or whether there's more variety for userspace consumers.

Is the proposal to set the memory usage from similar interface, or is that still not implemented?

I'm seeing a very close relation between time-slicing GPU time and allowed GPU buffer allocations, so having two completely different interfaces does just feel very hackish way of implementing this.

Regards, Joonas

Quoted 17 lines

Matt

Quoted 6 lines

Regards, Joonas
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

maintainer
#21 lore raw

Re: DRM cgroups integration (Was: Re: [PATCH v4 0/8] cgroup private data and DRM/i915 integration)

+ Some more Cc's based on IRC discussion

Quoting Joonas Lahtinen (2018-04-05 16:46:51)

Quoted 141 lines

+ Dave for commenting from DRM subsystem perspective. I strongly believe there would be benefit from agreeing on some foundation of DRM subsystem level program GPU niceness [-20,19] and memory limit [0,N] pages.

Quoting Matt Roper (2018-03-30 03:43:13)

Quoted 65 lines · Joonas Lahtinen
Quoted 50 lines

Quoting Matt Roper (2018-03-23 17:46:16)

Quoted 28 lines · Joonas Lahtinen
Quoted 19 lines

Quoting Matt Roper (2018-03-17 02:08:57)

Quoted 11 lines

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

Hi Matt,

For cross-subsystem changes such as this, it makes sense to Cc all relevant maintainers, especially if there have been previous comments to earlier revisions.

Quoted 7 lines

Please, do include and keep a reference to the userspace portion of the changes when you suggest new uAPI to be added. At least I have some trouble trying to track down the relevant interface consumer here.

I'm unsure how much sense it makes to commence with detailed i915 review if we will be blocked by lack of userspace after that? I'm assuming you've read through [1] already.

Quoted 5 lines

Hi Joonas,

I've sent the userspace code out a few times, but it looks like I forgot to include a copy with v4/v4.5. Here's the version I provided with v3:
https://lists.freedesktop.org/archives/intel-gfx/2018-March/157935.html

Thanks. Keeping that in the relevant commit message of the patch that introduces the new uAPI will make it harder to forget and easiest for git blame, too.

Quoted 10 lines

Usually we don't consider things like i-g-t to be sufficient userspace consumers because we need a real-world consumer rather than a "toy" userspace. However in this case, the i-g-t tool, although very simple, is really the only userspace consumer I expect there to ever be. Ultimately the true consumer of this cgroups work are bash scripts, sysv init scripts, systemd recipes, etc. that just need a very simple tool to assign the specific values that make sense on a given system. There's no expectation that graphics clients or display servers would ever need to make use of these interfaces.

Quoted 4 lines

I was under the impression that a bit more generic GPU cgroups support was receiving a lot of support in the early discussion? A dedicated intel_cgroup sounds underwhelming, when comparing to idea of "gpu_nice", for user adoption :)

I'm open to moving the cgroup_priv registration/lookup to the DRM core if other drivers are interested in using this mechanism and if we can come to an agreement on a standard priority offset range to support, how display boost should work for all drivers, etc. There might be some challenges mapping a DRM-defined priority range down to a different range that makes sense for individual driver schedulers, especially since some drivers already expose a different priority scheme to userspace via other interfaces like i915 does with GEM context priority.

Quoted 5 lines

So far I haven't really heard any interest outside the Intel camp, but hopefully other driver teams can speak up if they're for/against this. I don't want to try to artificially standardize this if other drivers want to go a different direction with priority/scheduling that's too different from the current Intel-driven design.

I don't think there are that many directions to go about GPU context priority, considering we have the EGL_IMG_context_priority extension, so it'll only be about granularity of the scale.

I would suggest to go with the nice like scale for easy user adoption, then just apply that as the N most significant bits.

The contexts could then of course further adjust their priority from what is set by the "gpu_nice" application with the remaining bits.

I'm strongly feeling this should be a DRM level "gpu_nice". And the binding to cgroups should come through DRM core. If it doesn't, limiting the amount of memory used becomes awkward as the allocation is centralized to DRM core.

Quoted 19 lines
Quoted 5 lines

Also, I might not be up-to-date about all things cgroups, but the way intel_cgroup works, feels bit forced. We create a userspace context just to communicate with the driver and the IOCTL will still have global effects. I can't but think that i915 reading from the cgroups subsystem for the current process would feel more intuitive to me.

I think you're referring to the earlier discussion about exposing priority directly via the cgroups filesystem? That would certainly be simpler from a userspace perspective, but it's not the direction that the cgroups maintainer wants to see things go. Adding files directly to the cgroups filesystem is supposed to be something that's reserved for official cgroups controllers. The GPU priority concept we're trying to add here doesn't align with the requirements for creating a controller, so the preferred approach is to create a custom interface (syscall or ioctl) that simply takes a cgroup as a parameter. There's precendent with similar interfaces in areas like BPF (where the bpf() system call can accept a cgroup as a parameter and then perform its own private policy changes as it sees fit).

Quoted 4 lines

Using a true cgroups controller and exposing settings via the filesystem is likely still the way we'll want to go for some other types of cgroups-based policy in the future (e.g., limiting GPU memory usage); it just isn't the appropriate direction for priority.

Might be just me but feels bit crazy to be setting GPU memory usage through another interface and then doing i915 specific IOCTLs to control the priority of that same cgroup.

I don't feel comfortable adding custom cgroups dependent IOCTLs to i915 where cgroups is only working as the variable carrier in background. We should really just be consuming a variable from cgroups and it should be set outside of of the i915 IOCTL interface.

I'm still seeing that we should have a DRM cgroups controller and a DRM subsystem wide application to control the priority and memory usage to be fed to the drivers.

If we end up just supporting i915 apps, we could as well use LD_PRELOAD wrapper and alter the context priority at creation time for exactly the same effect and no extra interfaces to maintain.

Quoted 8 lines
Quoted 2 lines

Does the implementation mimic some existing cgroups tool or de-facto way of doing things in cgroups world?

The ioctl approach I took is similar to syscall approach that the BPF guys use to attach BPF programs to a cgroup. I'm not very familiar with BPF or how it gets used from userspace, so I'm not sure whether the interface is intended for one specific tool (like ours is), or whether there's more variety for userspace consumers.

Quoted 26 lines

Is the proposal to set the memory usage from similar interface, or is that still not implemented?

I'm seeing a very close relation between time-slicing GPU time and allowed GPU buffer allocations, so having two completely different interfaces does just feel very hackish way of implementing this.

Regards, Joonas

Quoted 17 lines

Matt

Quoted 6 lines

Regards, Joonas
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

author
#22 lore raw

Re: DRM cgroups integration (Was: Re: [PATCH v4 0/8] cgroup private data and DRM/i915 integration)

Quoted 6 lines · Joonas Lahtinen

+ Some more Cc's based on IRC discussion

Quoting Joonas Lahtinen (2018-04-05 16:46:51)

Quoted 3 lines

+ Dave for commenting from DRM subsystem perspective. I strongly believe there would be benefit from agreeing on some foundation of DRM subsystem level program GPU niceness [-20,19] and memory limit [0,N] pages.

+Chris Wilson

If we ignore backward compatibility and ABI issues for now and assume all drivers can move to [-20, 19] for application-accessible priority ranges (i.e., self opt-in via existing context parameter ioctls and such), I think we still need a much larger range for the priority offset assigned via cgroup. One of the goals with the cgroup work is to give the system integrator the ability to define classes of applications with non-overlapping ranges (i.e., in some systems an app in the "very important" cgroup that self-lowers itself as far as possible should still be prioritized higher than an app in the "best effort" cgroup that self-boosts itself as far as possible). Chris Wilson and I discussed that a bit on this thread if you want to see the context:

https://lists.freedesktop.org/archives/intel-gfx/2018-March/158204.html

That discussion was based on i915's current application-accessible range of [-1023,1023]. If we shift to a much smaller [-20,19] range for applications to use directly, then we might want to allow cgroup offset to be something like [-1000,1000] to ensure the system integrator has enough flexibility?

Also note that "display boost" (i.e., a priority boost for contexts that are blocking a flip) may vary depending on system setup, so setting being able to define the display boost's effective priority via cgroup is important as well.

Matt

Quoted 168 lines
Quoted 138 lines

Quoting Matt Roper (2018-03-30 03:43:13)

Quoted 65 lines · Joonas Lahtinen
Quoted 50 lines

Quoting Matt Roper (2018-03-23 17:46:16)

Quoted 28 lines · Joonas Lahtinen
Quoted 19 lines

Quoting Matt Roper (2018-03-17 02:08:57)

Quoted 11 lines

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

Hi Matt,

For cross-subsystem changes such as this, it makes sense to Cc all relevant maintainers, especially if there have been previous comments to earlier revisions.

Quoted 7 lines

Please, do include and keep a reference to the userspace portion of the changes when you suggest new uAPI to be added. At least I have some trouble trying to track down the relevant interface consumer here.

I'm unsure how much sense it makes to commence with detailed i915 review if we will be blocked by lack of userspace after that? I'm assuming you've read through [1] already.

Quoted 5 lines

Hi Joonas,

I've sent the userspace code out a few times, but it looks like I forgot to include a copy with v4/v4.5. Here's the version I provided with v3:
https://lists.freedesktop.org/archives/intel-gfx/2018-March/157935.html

Thanks. Keeping that in the relevant commit message of the patch that introduces the new uAPI will make it harder to forget and easiest for git blame, too.

Quoted 10 lines

Usually we don't consider things like i-g-t to be sufficient userspace consumers because we need a real-world consumer rather than a "toy" userspace. However in this case, the i-g-t tool, although very simple, is really the only userspace consumer I expect there to ever be. Ultimately the true consumer of this cgroups work are bash scripts, sysv init scripts, systemd recipes, etc. that just need a very simple tool to assign the specific values that make sense on a given system. There's no expectation that graphics clients or display servers would ever need to make use of these interfaces.

Quoted 4 lines

I was under the impression that a bit more generic GPU cgroups support was receiving a lot of support in the early discussion? A dedicated intel_cgroup sounds underwhelming, when comparing to idea of "gpu_nice", for user adoption :)

I'm open to moving the cgroup_priv registration/lookup to the DRM core if other drivers are interested in using this mechanism and if we can come to an agreement on a standard priority offset range to support, how display boost should work for all drivers, etc. There might be some challenges mapping a DRM-defined priority range down to a different range that makes sense for individual driver schedulers, especially since some drivers already expose a different priority scheme to userspace via other interfaces like i915 does with GEM context priority.

Quoted 5 lines

So far I haven't really heard any interest outside the Intel camp, but hopefully other driver teams can speak up if they're for/against this. I don't want to try to artificially standardize this if other drivers want to go a different direction with priority/scheduling that's too different from the current Intel-driven design.

I don't think there are that many directions to go about GPU context priority, considering we have the EGL_IMG_context_priority extension, so it'll only be about granularity of the scale.

I would suggest to go with the nice like scale for easy user adoption, then just apply that as the N most significant bits.

The contexts could then of course further adjust their priority from what is set by the "gpu_nice" application with the remaining bits.

I'm strongly feeling this should be a DRM level "gpu_nice". And the binding to cgroups should come through DRM core. If it doesn't, limiting the amount of memory used becomes awkward as the allocation is centralized to DRM core.

Quoted 19 lines
Quoted 5 lines

Also, I might not be up-to-date about all things cgroups, but the way intel_cgroup works, feels bit forced. We create a userspace context just to communicate with the driver and the IOCTL will still have global effects. I can't but think that i915 reading from the cgroups subsystem for the current process would feel more intuitive to me.

I think you're referring to the earlier discussion about exposing priority directly via the cgroups filesystem? That would certainly be simpler from a userspace perspective, but it's not the direction that the cgroups maintainer wants to see things go. Adding files directly to the cgroups filesystem is supposed to be something that's reserved for official cgroups controllers. The GPU priority concept we're trying to add here doesn't align with the requirements for creating a controller, so the preferred approach is to create a custom interface (syscall or ioctl) that simply takes a cgroup as a parameter. There's precendent with similar interfaces in areas like BPF (where the bpf() system call can accept a cgroup as a parameter and then perform its own private policy changes as it sees fit).

Quoted 4 lines

Using a true cgroups controller and exposing settings via the filesystem is likely still the way we'll want to go for some other types of cgroups-based policy in the future (e.g., limiting GPU memory usage); it just isn't the appropriate direction for priority.

Might be just me but feels bit crazy to be setting GPU memory usage through another interface and then doing i915 specific IOCTLs to control the priority of that same cgroup.

I don't feel comfortable adding custom cgroups dependent IOCTLs to i915 where cgroups is only working as the variable carrier in background. We should really just be consuming a variable from cgroups and it should be set outside of of the i915 IOCTL interface.

I'm still seeing that we should have a DRM cgroups controller and a DRM subsystem wide application to control the priority and memory usage to be fed to the drivers.

If we end up just supporting i915 apps, we could as well use LD_PRELOAD wrapper and alter the context priority at creation time for exactly the same effect and no extra interfaces to maintain.

Quoted 8 lines
Quoted 2 lines

Does the implementation mimic some existing cgroups tool or de-facto way of doing things in cgroups world?

The ioctl approach I took is similar to syscall approach that the BPF guys use to attach BPF programs to a cgroup. I'm not very familiar with BPF or how it gets used from userspace, so I'm not sure whether the interface is intended for one specific tool (like ours is), or whether there's more variety for userspace consumers.

Quoted 26 lines

Is the proposal to set the memory usage from similar interface, or is that still not implemented?

I'm seeing a very close relation between time-slicing GPU time and allowed GPU buffer allocations, so having two completely different interfaces does just feel very hackish way of implementing this.

Regards, Joonas

Quoted 17 lines

Matt

Quoted 6 lines

Regards, Joonas
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795

--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 9 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
author
#23 lore raw

Re: DRM cgroups integration (Was: Re: [PATCH v4 0/8] cgroup private data and DRM/i915 integration)

Quoted 9 lines · Matt Roper
Quoted 6 lines · Joonas Lahtinen

+ Some more Cc's based on IRC discussion

Quoting Joonas Lahtinen (2018-04-05 16:46:51)

Quoted 3 lines

+ Dave for commenting from DRM subsystem perspective. I strongly believe there would be benefit from agreeing on some foundation of DRM subsystem level program GPU niceness [-20,19] and memory limit [0,N] pages.

+Chris Wilson

+more Cc's based on IRC discussion.

Matt

Quoted 206 lines

If we ignore backward compatibility and ABI issues for now and assume all drivers can move to [-20, 19] for application-accessible priority ranges (i.e., self opt-in via existing context parameter ioctls and such), I think we still need a much larger range for the priority offset assigned via cgroup. One of the goals with the cgroup work is to give the system integrator the ability to define classes of applications with non-overlapping ranges (i.e., in some systems an app in the "very important" cgroup that self-lowers itself as far as possible should still be prioritized higher than an app in the "best effort" cgroup that self-boosts itself as far as possible). Chris Wilson and I discussed that a bit on this thread if you want to see the context:

https://lists.freedesktop.org/archives/intel-gfx/2018-March/158204.html

That discussion was based on i915's current application-accessible range of [-1023,1023]. If we shift to a much smaller [-20,19] range for applications to use directly, then we might want to allow cgroup offset to be something like [-1000,1000] to ensure the system integrator has enough flexibility?

Also note that "display boost" (i.e., a priority boost for contexts that are blocking a flip) may vary depending on system setup, so setting being able to define the display boost's effective priority via cgroup is important as well.

Matt

Quoted 168 lines
Quoted 138 lines

Quoting Matt Roper (2018-03-30 03:43:13)

Quoted 65 lines · Joonas Lahtinen
Quoted 50 lines

Quoting Matt Roper (2018-03-23 17:46:16)

Quoted 28 lines · Joonas Lahtinen
Quoted 19 lines

Quoting Matt Roper (2018-03-17 02:08:57)

Quoted 11 lines

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

Hi Matt,

For cross-subsystem changes such as this, it makes sense to Cc all relevant maintainers, especially if there have been previous comments to earlier revisions.

Quoted 7 lines

Please, do include and keep a reference to the userspace portion of the changes when you suggest new uAPI to be added. At least I have some trouble trying to track down the relevant interface consumer here.

I'm unsure how much sense it makes to commence with detailed i915 review if we will be blocked by lack of userspace after that? I'm assuming you've read through [1] already.

Quoted 5 lines

Hi Joonas,

I've sent the userspace code out a few times, but it looks like I forgot to include a copy with v4/v4.5. Here's the version I provided with v3:
https://lists.freedesktop.org/archives/intel-gfx/2018-March/157935.html

Thanks. Keeping that in the relevant commit message of the patch that introduces the new uAPI will make it harder to forget and easiest for git blame, too.

Quoted 10 lines

Usually we don't consider things like i-g-t to be sufficient userspace consumers because we need a real-world consumer rather than a "toy" userspace. However in this case, the i-g-t tool, although very simple, is really the only userspace consumer I expect there to ever be. Ultimately the true consumer of this cgroups work are bash scripts, sysv init scripts, systemd recipes, etc. that just need a very simple tool to assign the specific values that make sense on a given system. There's no expectation that graphics clients or display servers would ever need to make use of these interfaces.

Quoted 4 lines

I was under the impression that a bit more generic GPU cgroups support was receiving a lot of support in the early discussion? A dedicated intel_cgroup sounds underwhelming, when comparing to idea of "gpu_nice", for user adoption :)

I'm open to moving the cgroup_priv registration/lookup to the DRM core if other drivers are interested in using this mechanism and if we can come to an agreement on a standard priority offset range to support, how display boost should work for all drivers, etc. There might be some challenges mapping a DRM-defined priority range down to a different range that makes sense for individual driver schedulers, especially since some drivers already expose a different priority scheme to userspace via other interfaces like i915 does with GEM context priority.

Quoted 5 lines

So far I haven't really heard any interest outside the Intel camp, but hopefully other driver teams can speak up if they're for/against this. I don't want to try to artificially standardize this if other drivers want to go a different direction with priority/scheduling that's too different from the current Intel-driven design.

I don't think there are that many directions to go about GPU context priority, considering we have the EGL_IMG_context_priority extension, so it'll only be about granularity of the scale.

I would suggest to go with the nice like scale for easy user adoption, then just apply that as the N most significant bits.

The contexts could then of course further adjust their priority from what is set by the "gpu_nice" application with the remaining bits.

I'm strongly feeling this should be a DRM level "gpu_nice". And the binding to cgroups should come through DRM core. If it doesn't, limiting the amount of memory used becomes awkward as the allocation is centralized to DRM core.

Quoted 19 lines
Quoted 5 lines

Also, I might not be up-to-date about all things cgroups, but the way intel_cgroup works, feels bit forced. We create a userspace context just to communicate with the driver and the IOCTL will still have global effects. I can't but think that i915 reading from the cgroups subsystem for the current process would feel more intuitive to me.

I think you're referring to the earlier discussion about exposing priority directly via the cgroups filesystem? That would certainly be simpler from a userspace perspective, but it's not the direction that the cgroups maintainer wants to see things go. Adding files directly to the cgroups filesystem is supposed to be something that's reserved for official cgroups controllers. The GPU priority concept we're trying to add here doesn't align with the requirements for creating a controller, so the preferred approach is to create a custom interface (syscall or ioctl) that simply takes a cgroup as a parameter. There's precendent with similar interfaces in areas like BPF (where the bpf() system call can accept a cgroup as a parameter and then perform its own private policy changes as it sees fit).

Quoted 4 lines

Using a true cgroups controller and exposing settings via the filesystem is likely still the way we'll want to go for some other types of cgroups-based policy in the future (e.g., limiting GPU memory usage); it just isn't the appropriate direction for priority.

Might be just me but feels bit crazy to be setting GPU memory usage through another interface and then doing i915 specific IOCTLs to control the priority of that same cgroup.

I don't feel comfortable adding custom cgroups dependent IOCTLs to i915 where cgroups is only working as the variable carrier in background. We should really just be consuming a variable from cgroups and it should be set outside of of the i915 IOCTL interface.

I'm still seeing that we should have a DRM cgroups controller and a DRM subsystem wide application to control the priority and memory usage to be fed to the drivers.

If we end up just supporting i915 apps, we could as well use LD_PRELOAD wrapper and alter the context priority at creation time for exactly the same effect and no extra interfaces to maintain.

Quoted 8 lines
Quoted 2 lines

Does the implementation mimic some existing cgroups tool or de-facto way of doing things in cgroups world?

The ioctl approach I took is similar to syscall approach that the BPF guys use to attach BPF programs to a cgroup. I'm not very familiar with BPF or how it gets used from userspace, so I'm not sure whether the interface is intended for one specific tool (like ours is), or whether there's more variety for userspace consumers.

Quoted 26 lines

Is the proposal to set the memory usage from similar interface, or is that still not implemented?

I'm seeing a very close relation between time-slicing GPU time and allowed GPU buffer allocations, so having two completely different interfaces does just feel very hackish way of implementing this.

Regards, Joonas

Quoted 17 lines

Matt

Quoted 6 lines

Regards, Joonas
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795

--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795
Signature · 9 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#24 lore raw

Re: DRM cgroups integration (Was: Re: [PATCH v4 0/8] cgroup private data and DRM/i915 integration)

Just realized Joonas had some other comments down at the bottom that I missed originally...

Quoted 9 lines · Matt Roper
Quoted 8 lines · Matt Roper
Quoted 6 lines · Joonas Lahtinen

+ Some more Cc's based on IRC discussion

Quoting Joonas Lahtinen (2018-04-05 16:46:51)

Quoted 3 lines

+ Dave for commenting from DRM subsystem perspective. I strongly believe there would be benefit from agreeing on some foundation of DRM subsystem level program GPU niceness [-20,19] and memory limit [0,N] pages.

Quoted 125 lines
Quoted 1 line

+Chris Wilson

+more Cc's based on IRC discussion.

Matt

Quoted 118 lines

If we ignore backward compatibility and ABI issues for now and assume all drivers can move to [-20, 19] for application-accessible priority ranges (i.e., self opt-in via existing context parameter ioctls and such), I think we still need a much larger range for the priority offset assigned via cgroup. One of the goals with the cgroup work is to give the system integrator the ability to define classes of applications with non-overlapping ranges (i.e., in some systems an app in the "very important" cgroup that self-lowers itself as far as possible should still be prioritized higher than an app in the "best effort" cgroup that self-boosts itself as far as possible). Chris Wilson and I discussed that a bit on this thread if you want to see the context:

https://lists.freedesktop.org/archives/intel-gfx/2018-March/158204.html

That discussion was based on i915's current application-accessible range of [-1023,1023]. If we shift to a much smaller [-20,19] range for applications to use directly, then we might want to allow cgroup offset to be something like [-1000,1000] to ensure the system integrator has enough flexibility?

Also note that "display boost" (i.e., a priority boost for contexts that are blocking a flip) may vary depending on system setup, so setting being able to define the display boost's effective priority via cgroup is important as well.

Matt

Quoted 87 lines
Quoted 87 lines

Quoting Matt Roper (2018-03-30 03:43:13)

Quoted 65 lines · Joonas Lahtinen
Quoted 50 lines

Quoting Matt Roper (2018-03-23 17:46:16)

Quoted 28 lines · Joonas Lahtinen
Quoted 19 lines

Quoting Matt Roper (2018-03-17 02:08:57)

Quoted 11 lines

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

Hi Matt,

For cross-subsystem changes such as this, it makes sense to Cc all relevant maintainers, especially if there have been previous comments to earlier revisions.

Quoted 7 lines

Please, do include and keep a reference to the userspace portion of the changes when you suggest new uAPI to be added. At least I have some trouble trying to track down the relevant interface consumer here.

I'm unsure how much sense it makes to commence with detailed i915 review if we will be blocked by lack of userspace after that? I'm assuming you've read through [1] already.

Quoted 5 lines

Hi Joonas,

I've sent the userspace code out a few times, but it looks like I forgot to include a copy with v4/v4.5. Here's the version I provided with v3:
https://lists.freedesktop.org/archives/intel-gfx/2018-March/157935.html

Thanks. Keeping that in the relevant commit message of the patch that introduces the new uAPI will make it harder to forget and easiest for git blame, too.

Quoted 10 lines

Usually we don't consider things like i-g-t to be sufficient userspace consumers because we need a real-world consumer rather than a "toy" userspace. However in this case, the i-g-t tool, although very simple, is really the only userspace consumer I expect there to ever be. Ultimately the true consumer of this cgroups work are bash scripts, sysv init scripts, systemd recipes, etc. that just need a very simple tool to assign the specific values that make sense on a given system. There's no expectation that graphics clients or display servers would ever need to make use of these interfaces.

Quoted 4 lines

I was under the impression that a bit more generic GPU cgroups support was receiving a lot of support in the early discussion? A dedicated intel_cgroup sounds underwhelming, when comparing to idea of "gpu_nice", for user adoption :)

I'm open to moving the cgroup_priv registration/lookup to the DRM core if other drivers are interested in using this mechanism and if we can come to an agreement on a standard priority offset range to support, how display boost should work for all drivers, etc. There might be some challenges mapping a DRM-defined priority range down to a different range that makes sense for individual driver schedulers, especially since some drivers already expose a different priority scheme to userspace via other interfaces like i915 does with GEM context priority.

Quoted 5 lines

So far I haven't really heard any interest outside the Intel camp, but hopefully other driver teams can speak up if they're for/against this. I don't want to try to artificially standardize this if other drivers want to go a different direction with priority/scheduling that's too different from the current Intel-driven design.

I don't think there are that many directions to go about GPU context priority, considering we have the EGL_IMG_context_priority extension, so it'll only be about granularity of the scale.

I would suggest to go with the nice like scale for easy user adoption, then just apply that as the N most significant bits.

The contexts could then of course further adjust their priority from what is set by the "gpu_nice" application with the remaining bits.

I'm strongly feeling this should be a DRM level "gpu_nice". And the binding to cgroups should come through DRM core. If it doesn't, limiting the amount of memory used becomes awkward as the allocation is centralized to DRM core.

My earliest iterations of this work did add the interfaces to set cgroup-based GPU priority at the DRM core. I didn't really get the sense at the time that anyone outside of Intel was interested in the cgroups work, so I moved the interfaces back into i915 for subequent updates, but I'm happy to move it back. Note that even in that case this will still end up as a DRM core ioctl rather than a true cgroups controller; see below.

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 20 lines
Quoted 6 lines

Also, I might not be up-to-date about all things cgroups, but the way intel_cgroup works, feels bit forced. We create a userspace context just to communicate with the driver and the IOCTL will still have global effects. I can't but think that i915 reading from the cgroups subsystem for the current process would feel more intuitive to me.

I think you're referring to the earlier discussion about exposing priority directly via the cgroups filesystem? That would certainly be simpler from a userspace perspective, but it's not the direction that the cgroups maintainer wants to see things go. Adding files directly to the cgroups filesystem is supposed to be something that's reserved for official cgroups controllers. The GPU priority concept we're trying to add here doesn't align with the requirements for creating a controller, so the preferred approach is to create a custom interface (syscall or ioctl) that simply takes a cgroup as a parameter. There's precendent with similar interfaces in areas like BPF (where the bpf() system call can accept a cgroup as a parameter and then perform its own private policy changes as it sees fit).

Quoted 4 lines

Using a true cgroups controller and exposing settings via the filesystem is likely still the way we'll want to go for some other types of cgroups-based policy in the future (e.g., limiting GPU memory usage); it just isn't the appropriate direction for priority.

Might be just me but feels bit crazy to be setting GPU memory usage through another interface and then doing i915 specific IOCTLs to control the priority of that same cgroup.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I don't feel comfortable adding custom cgroups dependent IOCTLs to i915 where cgroups is only working as the variable carrier in background. We should really just be consuming a variable from cgroups and it should be set outside of of the i915 IOCTL interface.

I'm still seeing that we should have a DRM cgroups controller and a DRM subsystem wide application to control the priority and memory usage to be fed to the drivers.

A cgroups controller could be used for GPU memory management, but not for priority adjustment the way we're doing it right now. Although there's some preexisting legacy stuff that doesn't quite follow the rules, the cgroups subsystem will only accept new cgroup controllers if they take the entire structure of the cgroups hierarchy into account (rather than just looking at values at a leaf node) and are based on one of the four resource distribution models described in the Documentation/cgroup-v2.txt (weights, limits, protections, allocations). GPU priority is somewhat close to a "weight" but not quite --- a true weight wouldn't allow unfair scheduling or intentional starvation which are very important to the use cases I'm focusing on.

The cgroups maintainer has suggested that a separate interface (syscall, ioctl, etc.) be used for items like this that don't meet the criteria for a full cgroups controller, and there are other examples of this kind of design elsewhere in the kernel too.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

If we end up just supporting i915 apps, we could as well use LD_PRELOAD wrapper and alter the context priority at creation time for exactly the same effect and no extra interfaces to maintain.

I think this is missing the point of using cgroups. cgroups allow policy to be set independently by a privileged system integrator or system administrator, in a way that applications themselves don't have access to. LD_PRELOAD would only have access to the interfaces available to the app itself and the app could simply undo whatever setting the LD_PRELOAD tried to use.

The goal with cgroups isn't to provide a "gpu nice" where you self-classify yourself, but rather the ability for the person defining how the overall system works to classify and control the various types of applications running in a transparent and global way.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Does the implementation mimic some existing cgroups tool or de-facto way of doing things in cgroups world?

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

The ioctl approach I took is similar to syscall approach that the BPF guys use to attach BPF programs to a cgroup. I'm not very familiar with BPF or how it gets used from userspace, so I'm not sure whether the interface is intended for one specific tool (like ours is), or whether there's more variety for userspace consumers.

Is the proposal to set the memory usage from similar interface, or is that still not implemented?

GPU memory management is a very different resource model, so I'd expect memory to be managed by a real cgroups controller. I.e., the interface would be different. I haven't personally done any work on cgroups-based memory management; I agree that would be nice to have, but none of the use cases my team is focused on have that as a requirement.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I'm seeing a very close relation between time-slicing GPU time and allowed GPU buffer allocations, so having two completely different interfaces does just feel very hackish way of implementing this.

If we were time slicing GPU time and giving out fractional shares of time to each client, that would be a very different situation than we have today; in that case I agree that a cgroups controller would be the way to go. While there probably are a lot of cases where time slicing and fair scheduling would be beneficial (especially stuff like desktop or mobile settings), I'm mostly focused on embedded use cases that depend on today's unfair, strict priority scheduling.

Matt

Quoted 38 lines
Quoted 31 lines
Quoted 24 lines
Quoted 20 lines

Regards, Joonas

Quoted 17 lines

Matt

Quoted 6 lines

Regards, Joonas
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795

--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795
Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795
Signature · 9 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#25 lore raw

Re: DRM cgroups integration (Was: Re: [PATCH v4 0/8] cgroup private data and DRM/i915 integration)

Adding Kenny, who recently started looking into cgroup support from the AMD GPU computing side.

Regards,
  Felix

Quoted 228 lines · Matt Roper

Just realized Joonas had some other comments down at the bottom that I missed originally...

Quoted 5 lines · Matt Roper
Quoted 4 lines · Matt Roper
Quoted 3 lines · Joonas Lahtinen

+ Some more Cc's based on IRC discussion

Quoting Joonas Lahtinen (2018-04-05 16:46:51)

Quoted 118 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

+ Dave for commenting from DRM subsystem perspective. I strongly believe there would be benefit from agreeing on some foundation of DRM subsystem level program GPU niceness [-20,19] and memory limit [0,N] pages.

+Chris Wilson

+more Cc's based on IRC discussion.

Matt

Quoted 109 lines

If we ignore backward compatibility and ABI issues for now and assume all drivers can move to [-20, 19] for application-accessible priority ranges (i.e., self opt-in via existing context parameter ioctls and such), I think we still need a much larger range for the priority offset assigned via cgroup. One of the goals with the cgroup work is to give the system integrator the ability to define classes of applications with non-overlapping ranges (i.e., in some systems an app in the "very important" cgroup that self-lowers itself as far as possible should still be prioritized higher than an app in the "best effort" cgroup that self-boosts itself as far as possible). Chris Wilson and I discussed that a bit on this thread if you want to see the context:

https://lists.freedesktop.org/archives/intel-gfx/2018-March/158204.html

That discussion was based on i915's current application-accessible range of [-1023,1023]. If we shift to a much smaller [-20,19] range for applications to use directly, then we might want to allow cgroup offset to be something like [-1000,1000] to ensure the system integrator has enough flexibility?

Also note that "display boost" (i.e., a priority boost for contexts that are blocking a flip) may vary depending on system setup, so setting being able to define the display boost's effective priority via cgroup is important as well.

Matt

Quoted 79 lines
Quoted 79 lines

Quoting Matt Roper (2018-03-30 03:43:13)

Quoted 59 lines · Joonas Lahtinen
Quoted 45 lines

Quoting Matt Roper (2018-03-23 17:46:16)

Quoted 23 lines · Joonas Lahtinen
Quoted 22 lines

Quoting Matt Roper (2018-03-17 02:08:57)

Quoted 11 lines

This is the fourth iteration of the work previously posted here:
(v1) https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
(v2) https://www.mail-archive.com/[email protected]/msg208170.html
(v3) https://lists.freedesktop.org/archives/intel-gfx/2018-March/157928.html

The high level goal of this work is to allow non-cgroup-controller parts of the kernel (e.g., device drivers) to register their own private policy data for specific cgroups. That mechanism is then made use of in the i915 graphics driver to allow GPU priority to be assigned according to the cgroup membership of the owning process. Please see the v1 cover letter linked above for a more in-depth explanation and justification.

Hi Matt,

For cross-subsystem changes such as this, it makes sense to Cc all relevant maintainers, especially if there have been previous comments to earlier revisions.

Please, do include and keep a reference to the userspace portion of the changes when you suggest new uAPI to be added. At least I have some trouble trying to track down the relevant interface consumer here.

Quoted 8 lines
Quoted 3 lines

I'm unsure how much sense it makes to commence with detailed i915 review if we will be blocked by lack of userspace after that? I'm assuming you've read through [1] already.

Hi Joonas,

I've sent the userspace code out a few times, but it looks like I forgot to include a copy with v4/v4.5. Here's the version I provided with v3:
https://lists.freedesktop.org/archives/intel-gfx/2018-March/157935.html

Thanks. Keeping that in the relevant commit message of the patch that introduces the new uAPI will make it harder to forget and easiest for git blame, too.

Quoted 9 lines

Usually we don't consider things like i-g-t to be sufficient userspace consumers because we need a real-world consumer rather than a "toy" userspace. However in this case, the i-g-t tool, although very simple, is really the only userspace consumer I expect there to ever be. Ultimately the true consumer of this cgroups work are bash scripts, sysv init scripts, systemd recipes, etc. that just need a very simple tool to assign the specific values that make sense on a given system. There's no expectation that graphics clients or display servers would ever need to make use of these interfaces.

Quoted 4 lines

I was under the impression that a bit more generic GPU cgroups support was receiving a lot of support in the early discussion? A dedicated intel_cgroup sounds underwhelming, when comparing to idea of "gpu_nice", for user adoption :)

I'm open to moving the cgroup_priv registration/lookup to the DRM core if other drivers are interested in using this mechanism and if we can come to an agreement on a standard priority offset range to support, how display boost should work for all drivers, etc. There might be some challenges mapping a DRM-defined priority range down to a different range that makes sense for individual driver schedulers, especially since some drivers already expose a different priority scheme to userspace via other interfaces like i915 does with GEM context priority.

Quoted 5 lines

So far I haven't really heard any interest outside the Intel camp, but hopefully other driver teams can speak up if they're for/against this. I don't want to try to artificially standardize this if other drivers want to go a different direction with priority/scheduling that's too different from the current Intel-driven design.

I don't think there are that many directions to go about GPU context priority, considering we have the EGL_IMG_context_priority extension, so it'll only be about granularity of the scale.

I would suggest to go with the nice like scale for easy user adoption, then just apply that as the N most significant bits.

The contexts could then of course further adjust their priority from what is set by the "gpu_nice" application with the remaining bits.

I'm strongly feeling this should be a DRM level "gpu_nice". And the binding to cgroups should come through DRM core. If it doesn't, limiting the amount of memory used becomes awkward as the allocation is centralized to DRM core.

My earliest iterations of this work did add the interfaces to set cgroup-based GPU priority at the DRM core. I didn't really get the sense at the time that anyone outside of Intel was interested in the cgroups work, so I moved the interfaces back into i915 for subequent updates, but I'm happy to move it back. Note that even in that case this will still end up as a DRM core ioctl rather than a true cgroups controller; see below.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 18 lines
Quoted 5 lines

Also, I might not be up-to-date about all things cgroups, but the way intel_cgroup works, feels bit forced. We create a userspace context just to communicate with the driver and the IOCTL will still have global effects. I can't but think that i915 reading from the cgroups subsystem for the current process would feel more intuitive to me.

I think you're referring to the earlier discussion about exposing priority directly via the cgroups filesystem? That would certainly be simpler from a userspace perspective, but it's not the direction that the cgroups maintainer wants to see things go. Adding files directly to the cgroups filesystem is supposed to be something that's reserved for official cgroups controllers. The GPU priority concept we're trying to add here doesn't align with the requirements for creating a controller, so the preferred approach is to create a custom interface (syscall or ioctl) that simply takes a cgroup as a parameter. There's precendent with similar interfaces in areas like BPF (where the bpf() system call can accept a cgroup as a parameter and then perform its own private policy changes as it sees fit).

Quoted 4 lines

Using a true cgroups controller and exposing settings via the filesystem is likely still the way we'll want to go for some other types of cgroups-based policy in the future (e.g., limiting GPU memory usage); it just isn't the appropriate direction for priority.

Might be just me but feels bit crazy to be setting GPU memory usage through another interface and then doing i915 specific IOCTLs to control the priority of that same cgroup.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I don't feel comfortable adding custom cgroups dependent IOCTLs to i915 where cgroups is only working as the variable carrier in background. We should really just be consuming a variable from cgroups and it should be set outside of of the i915 IOCTL interface.

I'm still seeing that we should have a DRM cgroups controller and a DRM subsystem wide application to control the priority and memory usage to be fed to the drivers.

A cgroups controller could be used for GPU memory management, but not for priority adjustment the way we're doing it right now. Although there's some preexisting legacy stuff that doesn't quite follow the rules, the cgroups subsystem will only accept new cgroup controllers if they take the entire structure of the cgroups hierarchy into account (rather than just looking at values at a leaf node) and are based on one of the four resource distribution models described in the Documentation/cgroup-v2.txt (weights, limits, protections, allocations). GPU priority is somewhat close to a "weight" but not quite --- a true weight wouldn't allow unfair scheduling or intentional starvation which are very important to the use cases I'm focusing on.

The cgroups maintainer has suggested that a separate interface (syscall, ioctl, etc.) be used for items like this that don't meet the criteria for a full cgroups controller, and there are other examples of this kind of design elsewhere in the kernel too.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

If we end up just supporting i915 apps, we could as well use LD_PRELOAD wrapper and alter the context priority at creation time for exactly the same effect and no extra interfaces to maintain.

I think this is missing the point of using cgroups. cgroups allow policy to be set independently by a privileged system integrator or system administrator, in a way that applications themselves don't have access to. LD_PRELOAD would only have access to the interfaces available to the app itself and the app could simply undo whatever setting the LD_PRELOAD tried to use.

The goal with cgroups isn't to provide a "gpu nice" where you self-classify yourself, but rather the ability for the person defining how the overall system works to classify and control the various types of applications running in a transparent and global way.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

Does the implementation mimic some existing cgroups tool or de-facto way of doing things in cgroups world?

The ioctl approach I took is similar to syscall approach that the BPF guys use to attach BPF programs to a cgroup. I'm not very familiar with BPF or how it gets used from userspace, so I'm not sure whether the interface is intended for one specific tool (like ours is), or whether there's more variety for userspace consumers.

Is the proposal to set the memory usage from similar interface, or is that still not implemented?

GPU memory management is a very different resource model, so I'd expect memory to be managed by a real cgroups controller. I.e., the interface would be different. I haven't personally done any work on cgroups-based memory management; I agree that would be nice to have, but none of the use cases my team is focused on have that as a requirement.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I'm seeing a very close relation between time-slicing GPU time and allowed GPU buffer allocations, so having two completely different interfaces does just feel very hackish way of implementing this.

If we were time slicing GPU time and giving out fractional shares of time to each client, that would be a very different situation than we have today; in that case I agree that a cgroups controller would be the way to go. While there probably are a lot of cases where time slicing and fair scheduling would be beneficial (especially stuff like desktop or mobile settings), I'm mostly focused on embedded use cases that depend on today's unfair, strict priority scheduling.

Quoted 34 lines

Matt

Quoted 32 lines
Quoted 26 lines
Quoted 20 lines
Quoted 16 lines

Regards, Joonas

Quoted 14 lines

Matt

Quoted 5 lines

Regards, Joonas
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795

--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795
Signature · 5 lines
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx