Skip to contentSkip to conversation

cgroup support for GPU devices

RFC PATCH In review Brian Welty 6 participants 19 messages May 1, 2019 – May 9, 2019

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover cgroup support for GPU devices

In containerized or virtualized environments, there is desire to have controls in place for resources that can be consumed by users of a GPU device. This RFC patch series proposes a framework for integrating use of existing cgroup controllers into device drivers.
The i915 driver is updated in this series as our primary use case to leverage this framework and to serve as an example for discussion.

The patch series enables device drivers to use cgroups to control the following resources within a GPU (or other accelerator device):
* control allocation of device memory (reuse of memcg) and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

Per-device controls would be exposed in cgroup filesystem as:

    mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/<subsys_files>

such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.weight

The drm/i915 patch in this series is based on top of other RFC work [1] for i915 device memory support.

AMD [2] and Intel [3] have proposed related work in this area within the last few years, listed below as reference. This new RFC reuses existing cgroup controllers and takes a different approach than prior work.

Finally, some potential discussion points for this series:
* merge proposed <subsys_name>.devices into a single devices directory?
* allow devices to have multiple registrations for subsets of resources?
* document a 'common charging policy' for device drivers to follow?

[1] https://patchwork.freedesktop.org/series/56683/
[2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html
[3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html

Brian Welty (5):
cgroup: Add cgroup_subsys per-device registration framework
cgroup: Change kernfs_node for directories to store
cgroup_subsys_state
memcg: Add per-device support to memory cgroup subsystem
drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit drm/i915: Use memory cgroup for enforcing device memory limit

 drivers/gpu/drm/drm_drv.c                  |  12 +
 drivers/gpu/drm/drm_gem.c                  |   7 +
 drivers/gpu/drm/i915/i915_drv.c            |   2 +-
 drivers/gpu/drm/i915/intel_memory_region.c |  24 +-
 include/drm/drm_device.h                   |   3 +
 include/drm/drm_drv.h                      |   8 +
 include/drm/drm_gem.h                      |  11 +
 include/linux/cgroup-defs.h                |  28 ++
 include/linux/cgroup.h                     |   3 +
 include/linux/memcontrol.h                 |  10 +
 kernel/cgroup/cgroup-v1.c                  |  10 +-
 kernel/cgroup/cgroup.c                     | 310 ++++++++++++++++++---
 mm/memcontrol.c                            | 183 +++++++++++-
 13 files changed, 552 insertions(+), 59 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

RFC PATCH 1/5 cgroup: Add cgroup_subsys per-device registration framework

In containerized or virtualized environments, there is desire to have controls in place for resources that can be consumed by users of a GPU device. For this purpose, we extend control groups with a mechanism for device drivers to register with cgroup subsystems. Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

A new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.

It is exposed in cgroup filesystem as:
mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/
such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat

The creation of above files is implemented in css_populate_dir() for cgroup subsystems that have enabled per-device support. Above files are created either at time of cgroup creation (for known registered devices) or at the time of device driver registration of the device, during cgroup_register_device. cgroup_device_unregister will remove files from all current cgroups.

---
 include/linux/cgroup-defs.h |  28 ++++
 include/linux/cgroup.h      |   3 +
 kernel/cgroup/cgroup.c      | 270 ++++++++++++++++++++++++++++++++++--
 3 files changed, 289 insertions(+), 12 deletions(-)
include/linux/cgroup-defs.h +28 -0
index 1c70803e9f77..aeaab420e349 100644
@@ -162,6 +162,17 @@ struct cgroup_subsys_state {
 	struct work_struct destroy_work;
 	struct rcu_work destroy_rwork;
 
+	/*
+	 * Per-device state for devices registered with our subsys.
+	 * @device_css_idr stores pointer to per-device cgroup_subsys_state,
+	 * created when devices are associated with this css.
+	 * @device_kn is for creating .devices sub-directory within this cgroup
+	 * or for the per-device sub-directory (subsys.devices/<dev_name>).
+	 */
+	struct device *device;
+	struct idr device_css_idr;
+	struct kernfs_node *device_kn;
+
 	/*
 	 * PI: the parent css.	Placed here for cache proximity to following
 	 * fields of the containing structure.
@@ -589,6 +600,9 @@ struct cftype {
  */
 struct cgroup_subsys {
 	struct cgroup_subsys_state *(*css_alloc)(struct cgroup_subsys_state *parent_css);
+	struct cgroup_subsys_state *(*device_css_alloc)(struct device *device,
+							struct cgroup_subsys_state *cgroup_css,
+							struct cgroup_subsys_state *parent_device_css);
 	int (*css_online)(struct cgroup_subsys_state *css);
 	void (*css_offline)(struct cgroup_subsys_state *css);
 	void (*css_released)(struct cgroup_subsys_state *css);
@@ -636,6 +650,13 @@ struct cgroup_subsys {
 	 */
 	bool threaded:1;
 
+	/*
+	 * If %true, the controller supports device drivers to register
+	 * with this controller for cloning the cgroup functionality
+	 * into per-device cgroup state under <cgroup-name>.dev/<dev_name>/.
+	 */
+	bool allow_devices:1;
+
 	/*
 	 * If %false, this subsystem is properly hierarchical -
 	 * configuration, resource accounting and restriction on a parent
@@ -664,6 +685,13 @@ struct cgroup_subsys {
 	/* idr for css->id */
 	struct idr css_idr;
 
+	/*
+	 * IDR of registered devices, allows subsys_state to have state
+	 * for each device. Exposed as per-device entries in filesystem,
+	 * under <subsys_name>.device/<dev_name>/.
+	 */
+	struct idr device_idr;
+
 	/*
 	 * List of cftypes.  Each entry is the first entry of an array
 	 * terminated by zero length name.
include/linux/cgroup.h +3 -0
index 81f58b4a5418..3531bf948703 100644
@@ -116,6 +116,9 @@ int cgroupstats_build(struct cgroupstats *stats, struct dentry *dentry);
 int proc_cgroup_show(struct seq_file *m, struct pid_namespace *ns,
 		     struct pid *pid, struct task_struct *tsk);
 
+int cgroup_device_register(struct cgroup_subsys *ss, struct device *dev,
+			   unsigned long *dev_id);
+void cgroup_device_unregister(struct cgroup_subsys *ss, unsigned long dev_id);
 void cgroup_fork(struct task_struct *p);
 extern int cgroup_can_fork(struct task_struct *p);
 extern void cgroup_cancel_fork(struct task_struct *p);
kernel/cgroup/cgroup.c +258 -12

386 lines not on this page · raw message

Signature · 1 line
2.21.0
#2 lore raw

RFC PATCH 2/5 cgroup: Change kernfs_node for directories to store cgroup_subsys_state

In reply to Brian Welty

Change the kernfs_node.priv to store the cgroup_subsys_state (CSS) pointer for directories, instead of storing cgroup pointer. This is done in order to support files within the cgroup associated with devices. We require of_css() to return the device-specific CSS pointer for these files.

---
 kernel/cgroup/cgroup-v1.c | 10 ++++----
 kernel/cgroup/cgroup.c    | 48 +++++++++++++++++----------------------
 2 files changed, 27 insertions(+), 31 deletions(-)
kernel/cgroup/cgroup-v1.c +6 -4
index c126b34fd4ff..4fa56cc2b99c 100644
@@ -723,6 +723,7 @@ int proc_cgroupstats_show(struct seq_file *m, void *v)
 int cgroupstats_build(struct cgroupstats *stats, struct dentry *dentry)
 {
 	struct kernfs_node *kn = kernfs_node_from_dentry(dentry);
+	struct cgroup_subsys_state *css;
 	struct cgroup *cgrp;
 	struct css_task_iter it;
 	struct task_struct *tsk;
@@ -740,12 +741,13 @@ int cgroupstats_build(struct cgroupstats *stats, struct dentry *dentry)
 	 * @kn->priv is RCU safe.  Let's do the RCU dancing.
 	 */
 	rcu_read_lock();
-	cgrp = rcu_dereference(*(void __rcu __force **)&kn->priv);
-	if (!cgrp || cgroup_is_dead(cgrp)) {
+	css = rcu_dereference(*(void __rcu __force **)&kn->priv);
+	if (!css || cgroup_is_dead(css->cgroup)) {
 		rcu_read_unlock();
 		mutex_unlock(&cgroup_mutex);
 		return -ENOENT;
 	}
+	cgrp = css->cgroup;
 	rcu_read_unlock();
 
 	css_task_iter_start(&cgrp->self, 0, &it);
@@ -851,7 +853,7 @@ void cgroup1_release_agent(struct work_struct *work)
 static int cgroup1_rename(struct kernfs_node *kn, struct kernfs_node *new_parent,
 			  const char *new_name_str)
 {
-	struct cgroup *cgrp = kn->priv;
+	struct cgroup_subsys_state *css = kn->priv;
 	int ret;
 
 	if (kernfs_type(kn) != KERNFS_DIR)
@@ -871,7 +873,7 @@ static int cgroup1_rename(struct kernfs_node *kn, struct kernfs_node *new_parent
 
 	ret = kernfs_rename(kn, new_parent, new_name_str);
 	if (!ret)
-		TRACE_CGROUP_PATH(rename, cgrp);
+		TRACE_CGROUP_PATH(rename, css->cgroup);
 
 	mutex_unlock(&cgroup_mutex);
 
kernel/cgroup/cgroup.c +21 -27

120 lines not on this page · raw message

Signature · 1 line
2.21.0
#3 lore raw

RFC PATCH 3/5 memcg: Add per-device support to memory cgroup subsystem

In reply to Brian Welty

Here we update memory cgroup to enable the newly introduced per-device framework. As mentioned in the prior patch, the intent here is to allow drivers to have their own private cgroup controls (such as memory limit) to be applied to device resources instead of host system resources.

In summary, to enable device registration for memory cgroup subsystem:
* set .allow_devices to true
* add new exported device register and device unregister functions to register a device with the cgroup subsystem
* implement the .device_css_alloc callback to create device specific cgroups_subsys_state within a cgroup

As cgroup is created and for current registered devices, one will see in the cgroup filesystem these additional files:
mount/<cgroup_name>/memory.devices/<dev_name>/<mem_cgrp attributes>

Registration of a new device is performed in device drivers using new mem_cgroup_device_register(). This will create above files in existing cgroups.

And for runtime charging to the cgroup, we add the following:
* add new routine to lookup the device-specific cgroup_subsys_state which is within the task's cgroup (mem_cgroup_device_from_task)
* add new functions for device specific 'direct' charging

The last point above involves adding new mem_cgroup_try_charge_direct and mem_cgroup_uncharge_direct functions. The 'direct' name is to say that we are charging the specified cgroup state directly and not using any associated page or mm_struct. We are called within device specific memory management routines, where the device driver will track which cgroup to charge within its own private data structures.

With this initial submission, support for memory accounting and charging is functional. Nested cgroups will correctly maintain the parent for device-specific state as well, such that hierarchial charging to device files is supported.

---
 include/linux/memcontrol.h |  10 ++
 mm/memcontrol.c            | 183 ++++++++++++++++++++++++++++++++++---
 2 files changed, 178 insertions(+), 15 deletions(-)
include/linux/memcontrol.h +10 -0
index dbb6118370c1..711669b613dc 100644
@@ -348,6 +348,11 @@ void mem_cgroup_cancel_charge(struct page *page, struct mem_cgroup *memcg,
 		bool compound);
 void mem_cgroup_uncharge(struct page *page);
 void mem_cgroup_uncharge_list(struct list_head *page_list);
+/* direct charging to mem_cgroup is primarily for device driver usage */
+int mem_cgroup_try_charge_direct(struct mem_cgroup *memcg,
+				 unsigned long nr_pages);
+void mem_cgroup_uncharge_direct(struct mem_cgroup *memcg,
+				unsigned long nr_pages);
 
 void mem_cgroup_migrate(struct page *oldpage, struct page *newpage);
 
@@ -395,6 +400,11 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *, struct pglist_data *);
 bool task_in_mem_cgroup(struct task_struct *task, struct mem_cgroup *memcg);
 struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p);
 
+struct mem_cgroup *mem_cgroup_device_from_task(unsigned long id,
+					       struct task_struct *p);
+int mem_cgroup_device_register(struct device *dev, unsigned long *dev_id);
+void mem_cgroup_device_unregister(unsigned long dev_id);
+
 struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm);
 
 struct mem_cgroup *get_mem_cgroup_from_page(struct page *page);
mm/memcontrol.c +168 -15

264 lines not on this page · raw message

Signature · 1 line
2.21.0
#4 lore raw

RFC PATCH 4/5 drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit

In reply to Brian Welty

With new cgroups per-device framework, registration with memory cgroup subsystem can allow us to enforce limit for allocation of device memory against process cgroups.

This patch adds new driver feature bit, DRIVER_CGROUPS, such that DRM will register the device with cgroups. Doing so allows device drivers to charge memory allocations to device-specific state within the cgroup.

Note, this is only for GEM objects allocated from device memory. Memory charging for GEM objects using system memory is already handled by the mm subsystem charing the normal (non-device) memory cgroup.

To charge device memory allocations, we need to (1) identify appropriate cgroup to charge (currently decided at object creation time), and (2) make the charging call at the time that memory pages are being allocated. Above is one policy, and this is open for debate if this is the right choice.

For (1), we associate the current task's cgroup with GEM objects as they are created. That cgroup will be charged/uncharged for all paging activity against the GEM object. Note, if the process is not part of a memory cgroup, then this returns NULL and no charging will occur. For shared objects, this may make the charge against a cgroup that is potentially not the same cgroup as the process using the memory. Based on the memory cgroup's discussion of "memory ownership", this seems acceptable [1]. For (2), this is for device drivers to implement within appropriate page allocation logic.

[1] https://www.kernel.org/doc/Documentation/cgroup-v2.txt, "Memory Ownership"

---
 drivers/gpu/drm/drm_drv.c | 12 ++++++++++++
 drivers/gpu/drm/drm_gem.c |  7 +++++++
 include/drm/drm_device.h  |  3 +++
 include/drm/drm_drv.h     |  8 ++++++++
 include/drm/drm_gem.h     | 11 +++++++++++
 5 files changed, 41 insertions(+)
drivers/gpu/drm/drm_drv.c +12 -0
index 862621494a93..890bd3c0e63e 100644
@@ -28,6 +28,7 @@
 
 #include <linux/debugfs.h>
 #include <linux/fs.h>
+#include <linux/memcontrol.h>
 #include <linux/module.h>
 #include <linux/moduleparam.h>
 #include <linux/mount.h>
@@ -987,6 +988,12 @@ int drm_dev_register(struct drm_device *dev, unsigned long flags)
 	if (ret)
 		goto err_minors;
 
+	if (dev->dev && drm_core_check_feature(dev, DRIVER_CGROUPS)) {
+		ret = mem_cgroup_device_register(dev->dev, &dev->memcg_id);
+		if (ret)
+			goto err_minors;
+	}
+
 	dev->registered = true;
 
 	if (dev->driver->load) {
@@ -1009,6 +1016,8 @@ int drm_dev_register(struct drm_device *dev, unsigned long flags)
 	goto out_unlock;
 
 err_minors:
+	if (dev->memcg_id)
+		mem_cgroup_device_unregister(dev->memcg_id);
 	remove_compat_control_link(dev);
 	drm_minor_unregister(dev, DRM_MINOR_PRIMARY);
 	drm_minor_unregister(dev, DRM_MINOR_RENDER);
@@ -1052,6 +1061,9 @@ void drm_dev_unregister(struct drm_device *dev)
 
 	drm_legacy_rmmaps(dev);
 
+	if (dev->memcg_id)
+		mem_cgroup_device_unregister(dev->memcg_id);
+
 	remove_compat_control_link(dev);
 	drm_minor_unregister(dev, DRM_MINOR_PRIMARY);
 	drm_minor_unregister(dev, DRM_MINOR_RENDER);
drivers/gpu/drm/drm_gem.c +7 -0
index 50de138c89e0..966fbd701deb 100644
@@ -38,6 +38,7 @@
 #include <linux/dma-buf.h>
 #include <linux/mem_encrypt.h>
 #include <linux/pagevec.h>
+#include <linux/memcontrol.h>
 #include <drm/drmP.h>
 #include <drm/drm_vma_manager.h>
 #include <drm/drm_gem.h>
@@ -281,6 +282,9 @@ drm_gem_handle_delete(struct drm_file *filp, u32 handle)
 	if (IS_ERR_OR_NULL(obj))
 		return -EINVAL;
 
+	/* Release reference on cgroup used with GEM object charging */
+	mem_cgroup_put(obj->memcg);
+
 	/* Release driver's reference and decrement refcount. */
 	drm_gem_object_release_handle(handle, obj, filp);
 
@@ -410,6 +414,9 @@ drm_gem_handle_create_tail(struct drm_file *file_priv,
 			goto err_revoke;
 	}
 
+	/* Acquire reference on cgroup for charging GEM memory allocations */
+	obj->memcg = mem_cgroup_device_from_task(dev->memcg_id, current);
+
 	*handlep = handle;
 	return 0;
 
include/drm/drm_device.h +3 -0
index 7f9ef709b2b6..9859f2289066 100644
@@ -190,6 +190,9 @@ struct drm_device {
 	 */
 	int irq;
 
+	/* @memcg_id: cgroup subsys (memcg) index for our device state */
+	unsigned long memcg_id;
+
 	/**
 	 * @vblank_disable_immediate:
 	 *
include/drm/drm_drv.h +8 -0
index 5cc7f728ec73..13b0e0b9527f 100644
@@ -92,6 +92,14 @@ enum drm_driver_feature {
 	 */
 	DRIVER_SYNCOBJ                  = BIT(5),
 
+	/**
+	 * @DRIVER_CGROUPS:
+	 *
+	 * Driver supports and requests DRM to register with cgroups during
+	 * drm_dev_register().
+	 */
+	DRIVER_CGROUPS			= BIT(6),
+
 	/* IMPORTANT: Below are all the legacy flags, add new ones above. */
 
 	/**
include/drm/drm_gem.h +11 -0
index 5047c7ee25f5..ca90ea512e45 100644
@@ -34,6 +34,7 @@
  * OTHER DEALINGS IN THE SOFTWARE.
  */
 
+#include <linux/memcontrol.h>
 #include <linux/kref.h>
 #include <linux/reservation.h>
 
@@ -202,6 +203,16 @@ struct drm_gem_object {
 	 */
 	struct file *filp;
 
+	/**
+	 * @memcg:
+	 *
+	 * cgroup used for charging GEM object page allocations against. This
+	 * is set to the current cgroup during GEM object creation.
+	 * Charging policy is up to each DRM driver to decide, but intent is to
+	 * charge during page allocation and use for device memory only.
+	 */
+	struct mem_cgroup *memcg;
+
 	/**
 	 * @vma_node:
 	 *
Signature · 1 line
2.21.0
#5 lore raw

RFC PATCH 5/5 drm/i915: Use memory cgroup for enforcing device memory limit

In reply to Brian Welty

i915 driver now includes DRIVER_CGROUPS in feature bits.

To charge device memory allocations, we need to (1) identify appropriate cgroup to charge (currently decided at object creation time), and (2) make the charging call at the time that memory pages are being allocated.

For (1), see prior DRM patch which associates current task's cgroup with GEM objects as they are created. That cgroup will be charged/uncharged for all paging activity against the GEM object.

For (2), we call mem_cgroup_try_charge_direct() in .get_pages callback for the GEM object type. Uncharging is done in .put_pages when the memory is marked such that it can be evicted. The try_charge() call will fail with -ENOMEM if the current memory allocation will exceed the cgroup device memory maximum, and allow for driver to perform memory reclaim.

---
 drivers/gpu/drm/i915/i915_drv.c            |  2 +-
 drivers/gpu/drm/i915/intel_memory_region.c | 24 ++++++++++++++++++----
 2 files changed, 21 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/i915_drv.c +1 -1
index 5a0a59922cb4..4d496c3c3681 100644
@@ -3469,7 +3469,7 @@ static struct drm_driver driver = {
 	 * deal with them for Intel hardware.
 	 */
 	.driver_features =
-	    DRIVER_GEM | DRIVER_PRIME |
+	    DRIVER_GEM | DRIVER_PRIME | DRIVER_CGROUPS |
 	    DRIVER_RENDER | DRIVER_MODESET | DRIVER_ATOMIC | DRIVER_SYNCOBJ,
 	.release = i915_driver_release,
 	.open = i915_driver_open,
drivers/gpu/drm/i915/intel_memory_region.c +20 -4
index 813ff83c132b..e4ac5e4d4857 100644
@@ -53,6 +53,8 @@ i915_memory_region_put_pages_buddy(struct drm_i915_gem_object *obj,
 	mutex_unlock(&obj->memory_region->mm_lock);
 
 	obj->mm.dirty = false;
+	mem_cgroup_uncharge_direct(obj->base.memcg,
+				   obj->base.size >> PAGE_SHIFT);
 }
 
 int
@@ -65,19 +67,29 @@ i915_memory_region_get_pages_buddy(struct drm_i915_gem_object *obj)
 	struct scatterlist *sg;
 	unsigned int sg_page_sizes;
 	unsigned long n_pages;
+	int err;
 
 	GEM_BUG_ON(!IS_ALIGNED(size, mem->mm.min_size));
 	GEM_BUG_ON(!list_empty(&obj->blocks));
 
+	err = mem_cgroup_try_charge_direct(obj->base.memcg, size >> PAGE_SHIFT);
+	if (err) {
+		DRM_DEBUG("MEMCG: try_charge failed for %lld\n", size);
+		return err;
+	}
+
 	st = kmalloc(sizeof(*st), GFP_KERNEL);
-	if (!st)
-		return -ENOMEM;
+	if (!st) {
+		err = -ENOMEM;
+		goto err_uncharge;
+	}
 
 	n_pages = div64_u64(size, mem->mm.min_size);
 
 	if (sg_alloc_table(st, n_pages, GFP_KERNEL)) {
 		kfree(st);
-		return -ENOMEM;
+		err = -ENOMEM;
+		goto err_uncharge;
 	}
 
 	sg = st->sgl;
@@ -161,7 +173,11 @@ i915_memory_region_get_pages_buddy(struct drm_i915_gem_object *obj)
 err_free_blocks:
 	memory_region_free_pages(obj, st);
 	mutex_unlock(&mem->mm_lock);
-	return -ENXIO;
+	err = -ENXIO;
+err_uncharge:
+	mem_cgroup_uncharge_direct(obj->base.memcg,
+				   obj->base.size >> PAGE_SHIFT);
+	return err;
 }
 
 int i915_memory_region_init_buddy(struct intel_memory_region *mem)
Signature · 1 line
2.21.0
#6 lore raw
In reply to Brian Welty
Quoted 10 lines · Brian Welty

In containerized or virtualized environments, there is desire to have controls in place for resources that can be consumed by users of a GPU device. This RFC patch series proposes a framework for integrating use of existing cgroup controllers into device drivers.
The i915 driver is updated in this series as our primary use case to leverage this framework and to serve as an example for discussion.

The patch series enables device drivers to use cgroups to control the following resources within a GPU (or other accelerator device):
* control allocation of device memory (reuse of memcg)

Count us (Mellanox) too, our RDMA devices are exposing special and limited in size device memory to the users and we would like to provide an option to use cgroup to control its exposure.

Quoted 61 lines

and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

Per-device controls would be exposed in cgroup filesystem as:

    mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/<subsys_files>

such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.weight

The drm/i915 patch in this series is based on top of other RFC work [1] for i915 device memory support.

AMD [2] and Intel [3] have proposed related work in this area within the last few years, listed below as reference. This new RFC reuses existing cgroup controllers and takes a different approach than prior work.

Finally, some potential discussion points for this series:
* merge proposed <subsys_name>.devices into a single devices directory?
* allow devices to have multiple registrations for subsets of resources?
* document a 'common charging policy' for device drivers to follow?

[1] https://patchwork.freedesktop.org/series/56683/
[2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html
[3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html

Brian Welty (5):
cgroup: Add cgroup_subsys per-device registration framework
cgroup: Change kernfs_node for directories to store
cgroup_subsys_state
memcg: Add per-device support to memory cgroup subsystem
drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit drm/i915: Use memory cgroup for enforcing device memory limit

 drivers/gpu/drm/drm_drv.c                  |  12 +
 drivers/gpu/drm/drm_gem.c                  |   7 +
 drivers/gpu/drm/i915/i915_drv.c            |   2 +-
 drivers/gpu/drm/i915/intel_memory_region.c |  24 +-
 include/drm/drm_device.h                   |   3 +
 include/drm/drm_drv.h                      |   8 +
 include/drm/drm_gem.h                      |  11 +
 include/linux/cgroup-defs.h                |  28 ++
 include/linux/cgroup.h                     |   3 +
 include/linux/memcontrol.h                 |  10 +
 kernel/cgroup/cgroup-v1.c                  |  10 +-
 kernel/cgroup/cgroup.c                     | 310 ++++++++++++++++++---
 mm/memcontrol.c                            | 183 +++++++++++-
 13 files changed, 552 insertions(+), 59 deletions(-)
Quoted 3 lines

--
2.21.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#7 lore raw
Quoted 3 lines

Count us (Mellanox) too, our RDMA devices are exposing special and limited in size device memory to the users and we would like to provide an option to use cgroup to control its exposure.

Doesn't RDMA already has a separate cgroup? Why not implement it there?

Quoted 61 lines
Quoted 61 lines

and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

Per-device controls would be exposed in cgroup filesystem as:

    mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/<subsys_files>

such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.weight

The drm/i915 patch in this series is based on top of other RFC work [1] for i915 device memory support.

AMD [2] and Intel [3] have proposed related work in this area within the last few years, listed below as reference. This new RFC reuses existing cgroup controllers and takes a different approach than prior work.

Finally, some potential discussion points for this series:
* merge proposed <subsys_name>.devices into a single devices directory?
* allow devices to have multiple registrations for subsets of resources?
* document a 'common charging policy' for device drivers to follow?

[1] https://patchwork.freedesktop.org/series/56683/
[2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html
[3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html

Brian Welty (5):
cgroup: Add cgroup_subsys per-device registration framework
cgroup: Change kernfs_node for directories to store
cgroup_subsys_state
memcg: Add per-device support to memory cgroup subsystem
drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit drm/i915: Use memory cgroup for enforcing device memory limit

 drivers/gpu/drm/drm_drv.c                  |  12 +
 drivers/gpu/drm/drm_gem.c                  |   7 +
 drivers/gpu/drm/i915/i915_drv.c            |   2 +-
 drivers/gpu/drm/i915/intel_memory_region.c |  24 +-
 include/drm/drm_device.h                   |   3 +
 include/drm/drm_drv.h                      |   8 +
 include/drm/drm_gem.h                      |  11 +
 include/linux/cgroup-defs.h                |  28 ++
 include/linux/cgroup.h                     |   3 +
 include/linux/memcontrol.h                 |  10 +
 kernel/cgroup/cgroup-v1.c                  |  10 +-
 kernel/cgroup/cgroup.c                     | 310 ++++++++++++++++++---
 mm/memcontrol.c                            | 183 +++++++++++-
 13 files changed, 552 insertions(+), 59 deletions(-)
Quoted 7 lines
Quoted 3 lines

--
2.21.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#8 lore raw
Quoted 4 lines · Kenny Ho
Quoted 3 lines · Leon Romanovsky

Count us (Mellanox) too, our RDMA devices are exposing special and limited in size device memory to the users and we would like to provide an option to use cgroup to control its exposure.

Hi Leon, great to hear and happy to work with you and RDMA community to shape this framework for use by RDMA devices as well. The intent was to support more than GPU devices.

Incidentally, I also wanted to ask about the rdma cgroup controller and if there is interest in updating the device registration implemented in that controller. It could use the cgroup_device_register() that is
proposed here. But this is perhaps future work, so can discuss separately.

Quoted 2 lines

Doesn't RDMA already has a separate cgroup? Why not implement it there?

Hi Kenny, I can't answer for Leon, but I'm hopeful he agrees with rationale I gave in the cover letter. Namely, to implement in rdma controller, would mean duplicating existing memcg controls there.

Is AMD interested in collaborating to help shape this framework? It is intended to be device-neutral, so could be leveraged by various types of devices.
If you have an alternative solution well underway, then maybe we can work together to merge our efforts into one.
In the end, the DRM community is best served with common solution.

Quoted 62 lines
Quoted 62 lines
Quoted 62 lines

and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

Per-device controls would be exposed in cgroup filesystem as:

    mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/<subsys_files>

such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.weight

The drm/i915 patch in this series is based on top of other RFC work [1] for i915 device memory support.

AMD [2] and Intel [3] have proposed related work in this area within the last few years, listed below as reference. This new RFC reuses existing cgroup controllers and takes a different approach than prior work.

Finally, some potential discussion points for this series:
* merge proposed <subsys_name>.devices into a single devices directory?
* allow devices to have multiple registrations for subsets of resources?
* document a 'common charging policy' for device drivers to follow?

[1] https://patchwork.freedesktop.org/series/56683/
[2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html
[3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html

Brian Welty (5):
cgroup: Add cgroup_subsys per-device registration framework
cgroup: Change kernfs_node for directories to store
cgroup_subsys_state
memcg: Add per-device support to memory cgroup subsystem
drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit drm/i915: Use memory cgroup for enforcing device memory limit

 drivers/gpu/drm/drm_drv.c                  |  12 +
 drivers/gpu/drm/drm_gem.c                  |   7 +
 drivers/gpu/drm/i915/i915_drv.c            |   2 +-
 drivers/gpu/drm/i915/intel_memory_region.c |  24 +-
 include/drm/drm_device.h                   |   3 +
 include/drm/drm_drv.h                      |   8 +
 include/drm/drm_gem.h                      |  11 +
 include/linux/cgroup-defs.h                |  28 ++
 include/linux/cgroup.h                     |   3 +
 include/linux/memcontrol.h                 |  10 +
 kernel/cgroup/cgroup-v1.c                  |  10 +-
 kernel/cgroup/cgroup.c                     | 310 ++++++++++++++++++---
 mm/memcontrol.c                            | 183 +++++++++++-
 13 files changed, 552 insertions(+), 59 deletions(-)
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

--
2.21.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#9 lore raw
Quoted 7 lines · Brian
Quoted 4 lines · Kenny Ho
Quoted 3 lines · Leon Romanovsky

Count us (Mellanox) too, our RDMA devices are exposing special and limited in size device memory to the users and we would like to provide an option to use cgroup to control its exposure.

Quoted 8 lines

Hi Leon, great to hear and happy to work with you and RDMA community to shape this framework for use by RDMA devices as well. The intent was to support more than GPU devices.

Incidentally, I also wanted to ask about the rdma cgroup controller and if there is interest in updating the device registration implemented in that controller. It could use the cgroup_device_register() that is
proposed here. But this is perhaps future work, so can discuss separately.

I'll try to take a look later this week.

Quoted 8 lines
Quoted 4 lines

Doesn't RDMA already has a separate cgroup? Why not implement it there?

Hi Kenny, I can't answer for Leon, but I'm hopeful he agrees with rationale I gave in the cover letter. Namely, to implement in rdma controller, would mean duplicating existing memcg controls there.

Exactly, I didn't feel comfortable to add notion of "device memory" to RDMA cgroup and postponed that decision to later point of time. RDMA operates with verbs objects and all our user space API is based around that concept. At the end, system administrator will have hard time to understand the differences between memcg and RDMA memory.

Quoted 78 lines

Is AMD interested in collaborating to help shape this framework? It is intended to be device-neutral, so could be leveraged by various types of devices.
If you have an alternative solution well underway, then maybe we can work together to merge our efforts into one.
In the end, the DRM community is best served with common solution.

Quoted 69 lines
Quoted 69 lines
Quoted 62 lines

and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

Per-device controls would be exposed in cgroup filesystem as:

    mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/<subsys_files>

such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.weight

The drm/i915 patch in this series is based on top of other RFC work [1] for i915 device memory support.

AMD [2] and Intel [3] have proposed related work in this area within the last few years, listed below as reference. This new RFC reuses existing cgroup controllers and takes a different approach than prior work.

Finally, some potential discussion points for this series:
* merge proposed <subsys_name>.devices into a single devices directory?
* allow devices to have multiple registrations for subsets of resources?
* document a 'common charging policy' for device drivers to follow?

[1] https://patchwork.freedesktop.org/series/56683/
[2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html
[3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html

Brian Welty (5):
cgroup: Add cgroup_subsys per-device registration framework
cgroup: Change kernfs_node for directories to store
cgroup_subsys_state
memcg: Add per-device support to memory cgroup subsystem
drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit drm/i915: Use memory cgroup for enforcing device memory limit

 drivers/gpu/drm/drm_drv.c                  |  12 +
 drivers/gpu/drm/drm_gem.c                  |   7 +
 drivers/gpu/drm/i915/i915_drv.c            |   2 +-
 drivers/gpu/drm/i915/intel_memory_region.c |  24 +-
 include/drm/drm_device.h                   |   3 +
 include/drm/drm_drv.h                      |   8 +
 include/drm/drm_gem.h                      |  11 +
 include/linux/cgroup-defs.h                |  28 ++
 include/linux/cgroup.h                     |   3 +
 include/linux/memcontrol.h                 |  10 +
 kernel/cgroup/cgroup-v1.c                  |  10 +-
 kernel/cgroup/cgroup.c                     | 310 ++++++++++++++++++---
 mm/memcontrol.c                            | 183 +++++++++++-
 13 files changed, 552 insertions(+), 59 deletions(-)
Quoted 3 lines

--
2.21.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#10 lore raw
Quoted 7 lines · Leon Romanovsky
Quoted 6 lines
Quoted 2 lines

Doesn't RDMA already has a separate cgroup? Why not implement it there?

Hi Kenny, I can't answer for Leon, but I'm hopeful he agrees with rationale I gave in the cover letter. Namely, to implement in rdma controller, would mean duplicating existing memcg controls there.

Quoted 5 lines

Exactly, I didn't feel comfortable to add notion of "device memory" to RDMA cgroup and postponed that decision to later point of time. RDMA operates with verbs objects and all our user space API is based around that concept. At the end, system administrator will have hard time to understand the differences between memcg and RDMA memory.

Interesting. I actually don't understand this part (I worked in devops/sysadmin side of things but never with rdma.) Don't applications that use rdma require some awareness of rdma (I mean, you mentioned verbs and objects... or do they just use regular malloc for buffer allocation and then send it through some function?) As a user, I would have this question: why do I need to configure some part of rdma resources under rdma cgroup while other part of rdma resources in a different, seemingly unrelated cgroups.

I think we need to be careful about drawing the line between duplication and over couplings between subsystems. I have other thoughts and concerns and I will try to organize them into a response in the next few days.

Regards,
Kenny

Quoted 71 lines
Quoted 71 lines

Is AMD interested in collaborating to help shape this framework? It is intended to be device-neutral, so could be leveraged by various types of devices.
If you have an alternative solution well underway, then maybe we can work together to merge our efforts into one.
In the end, the DRM community is best served with common solution.

Quoted 62 lines
Quoted 62 lines
Quoted 62 lines

and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

Per-device controls would be exposed in cgroup filesystem as:

    mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/<subsys_files>

such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.weight

The drm/i915 patch in this series is based on top of other RFC work [1] for i915 device memory support.

AMD [2] and Intel [3] have proposed related work in this area within the last few years, listed below as reference. This new RFC reuses existing cgroup controllers and takes a different approach than prior work.

Finally, some potential discussion points for this series:
* merge proposed <subsys_name>.devices into a single devices directory?
* allow devices to have multiple registrations for subsets of resources?
* document a 'common charging policy' for device drivers to follow?

[1] https://patchwork.freedesktop.org/series/56683/
[2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html
[3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html

Brian Welty (5):
cgroup: Add cgroup_subsys per-device registration framework
cgroup: Change kernfs_node for directories to store
cgroup_subsys_state
memcg: Add per-device support to memory cgroup subsystem
drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit drm/i915: Use memory cgroup for enforcing device memory limit

 drivers/gpu/drm/drm_drv.c                  |  12 +
 drivers/gpu/drm/drm_gem.c                  |   7 +
 drivers/gpu/drm/i915/i915_drv.c            |   2 +-
 drivers/gpu/drm/i915/intel_memory_region.c |  24 +-
 include/drm/drm_device.h                   |   3 +
 include/drm/drm_drv.h                      |   8 +
 include/drm/drm_gem.h                      |  11 +
 include/linux/cgroup-defs.h                |  28 ++
 include/linux/cgroup.h                     |   3 +
 include/linux/memcontrol.h                 |  10 +
 kernel/cgroup/cgroup-v1.c                  |  10 +-
 kernel/cgroup/cgroup.c                     | 310 ++++++++++++++++++---
 mm/memcontrol.c                            | 183 +++++++++++-
 13 files changed, 552 insertions(+), 59 deletions(-)
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

--
2.21.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#11 lore raw
Quoted 13 lines · Kenny Ho
Quoted 7 lines · Leon Romanovsky
Quoted 6 lines
Quoted 2 lines

Doesn't RDMA already has a separate cgroup? Why not implement it there?

Hi Kenny, I can't answer for Leon, but I'm hopeful he agrees with rationale I gave in the cover letter. Namely, to implement in rdma controller, would mean duplicating existing memcg controls there.

Quoted 5 lines

Exactly, I didn't feel comfortable to add notion of "device memory" to RDMA cgroup and postponed that decision to later point of time. RDMA operates with verbs objects and all our user space API is based around that concept. At the end, system administrator will have hard time to understand the differences between memcg and RDMA memory.

Quoted 8 lines

Interesting. I actually don't understand this part (I worked in devops/sysadmin side of things but never with rdma.) Don't applications that use rdma require some awareness of rdma (I mean, you mentioned verbs and objects... or do they just use regular malloc for buffer allocation and then send it through some function?) As a user, I would have this question: why do I need to configure some part of rdma resources under rdma cgroup while other part of rdma resources in a different, seemingly unrelated cgroups.

We are talking about two different access patterns for this device memory (DM). One is to use this device memory (DM) and second to configure/limit. Usually those actions will be performed by different groups.

First group (programmers) is using special API [1] through libibverbs [2] without any notion of cgroups or any limitations. Second group (sysadmins) is less interested in application specifics and for them "device memory" means "memory" and not "rdma, nic specific, internal memory".

[1] ibv_alloc_dm()
http://man7.org/linux/man-pages/man3/ibv_alloc_dm.3.html
https://www.openfabrics.org/images/2018workshop/presentations/304_LLiss_OnDeviceMemory.pdf
[2] https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/

Quoted 88 lines

I think we need to be careful about drawing the line between duplication and over couplings between subsystems. I have other thoughts and concerns and I will try to organize them into a response in the next few days.

Regards,
Kenny

Quoted 78 lines
Quoted 78 lines

Is AMD interested in collaborating to help shape this framework? It is intended to be device-neutral, so could be leveraged by various types of devices.
If you have an alternative solution well underway, then maybe we can work together to merge our efforts into one.
In the end, the DRM community is best served with common solution.

Quoted 69 lines
Quoted 69 lines
Quoted 62 lines

and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

Per-device controls would be exposed in cgroup filesystem as:

    mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/<subsys_files>

such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.weight

The drm/i915 patch in this series is based on top of other RFC work [1] for i915 device memory support.

AMD [2] and Intel [3] have proposed related work in this area within the last few years, listed below as reference. This new RFC reuses existing cgroup controllers and takes a different approach than prior work.

Finally, some potential discussion points for this series:
* merge proposed <subsys_name>.devices into a single devices directory?
* allow devices to have multiple registrations for subsets of resources?
* document a 'common charging policy' for device drivers to follow?

[1] https://patchwork.freedesktop.org/series/56683/
[2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html
[3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html

Brian Welty (5):
cgroup: Add cgroup_subsys per-device registration framework
cgroup: Change kernfs_node for directories to store
cgroup_subsys_state
memcg: Add per-device support to memory cgroup subsystem
drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit drm/i915: Use memory cgroup for enforcing device memory limit

 drivers/gpu/drm/drm_drv.c                  |  12 +
 drivers/gpu/drm/drm_gem.c                  |   7 +
 drivers/gpu/drm/i915/i915_drv.c            |   2 +-
 drivers/gpu/drm/i915/intel_memory_region.c |  24 +-
 include/drm/drm_device.h                   |   3 +
 include/drm/drm_drv.h                      |   8 +
 include/drm/drm_gem.h                      |  11 +
 include/linux/cgroup-defs.h                |  28 ++
 include/linux/cgroup.h                     |   3 +
 include/linux/memcontrol.h                 |  10 +
 kernel/cgroup/cgroup-v1.c                  |  10 +-
 kernel/cgroup/cgroup.c                     | 310 ++++++++++++++++++---
 mm/memcontrol.c                            | 183 +++++++++++-
 13 files changed, 552 insertions(+), 59 deletions(-)
Quoted 3 lines

--
2.21.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#12 lore raw

(sent again. Not sure why my previous email was just a reply instead of reply-all.)

Quoted 8 lines · Leon Romanovsky

We are talking about two different access patterns for this device memory (DM). One is to use this device memory (DM) and second to configure/limit. Usually those actions will be performed by different groups.

First group (programmers) is using special API [1] through libibverbs [2] without any notion of cgroups or any limitations. Second group (sysadmins) is less interested in application specifics and for them "device memory" means "memory" and not "rdma, nic specific, internal memory".

Um... I am not sure that answered it, especially in the context of cgroup (this is just for my curiosity btw, I don't know much about rdma.) You said sysadmins are less interested in application specifics but then how would they make the judgement call on how much "device memory" is provisioned to one application/container over another (let say you have 5 cgroup sharing an rdma device)? What are the consequences of under provisioning "device memory" to an application? And if they are all just memory, can a sysadmin provision more system memory in place of device memory (like, are they interchangeable)? I guess I am confused because if device memory is just memory (not rdma, nic specific) to sysadmins how would they know to set the right amount?

Regards,
Kenny

Quoted 93 lines

[1] ibv_alloc_dm()
http://man7.org/linux/man-pages/man3/ibv_alloc_dm.3.html
https://www.openfabrics.org/images/2018workshop/presentations/304_LLiss_OnDeviceMemory.pdf
[2] https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/

Quoted 88 lines

I think we need to be careful about drawing the line between duplication and over couplings between subsystems. I have other thoughts and concerns and I will try to organize them into a response in the next few days.

Regards,
Kenny

Quoted 78 lines
Quoted 78 lines

Is AMD interested in collaborating to help shape this framework? It is intended to be device-neutral, so could be leveraged by various types of devices.
If you have an alternative solution well underway, then maybe we can work together to merge our efforts into one.
In the end, the DRM community is best served with common solution.

Quoted 69 lines
Quoted 69 lines
Quoted 62 lines

and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

Per-device controls would be exposed in cgroup filesystem as:

    mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/<subsys_files>

such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.weight

The drm/i915 patch in this series is based on top of other RFC work [1] for i915 device memory support.

AMD [2] and Intel [3] have proposed related work in this area within the last few years, listed below as reference. This new RFC reuses existing cgroup controllers and takes a different approach than prior work.

Finally, some potential discussion points for this series:
* merge proposed <subsys_name>.devices into a single devices directory?
* allow devices to have multiple registrations for subsets of resources?
* document a 'common charging policy' for device drivers to follow?

[1] https://patchwork.freedesktop.org/series/56683/
[2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html
[3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html

Brian Welty (5):
cgroup: Add cgroup_subsys per-device registration framework
cgroup: Change kernfs_node for directories to store
cgroup_subsys_state
memcg: Add per-device support to memory cgroup subsystem
drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit drm/i915: Use memory cgroup for enforcing device memory limit

 drivers/gpu/drm/drm_drv.c                  |  12 +
 drivers/gpu/drm/drm_gem.c                  |   7 +
 drivers/gpu/drm/i915/i915_drv.c            |   2 +-
 drivers/gpu/drm/i915/intel_memory_region.c |  24 +-
 include/drm/drm_device.h                   |   3 +
 include/drm/drm_drv.h                      |   8 +
 include/drm/drm_gem.h                      |  11 +
 include/linux/cgroup-defs.h                |  28 ++
 include/linux/cgroup.h                     |   3 +
 include/linux/memcontrol.h                 |  10 +
 kernel/cgroup/cgroup-v1.c                  |  10 +-
 kernel/cgroup/cgroup.c                     | 310 ++++++++++++++++++---
 mm/memcontrol.c                            | 183 +++++++++++-
 13 files changed, 552 insertions(+), 59 deletions(-)
Quoted 3 lines

--
2.21.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#13 lore raw
Quoted 12 lines · Kenny Ho

(sent again. Not sure why my previous email was just a reply instead of reply-all.)

Quoted 8 lines · Leon Romanovsky

We are talking about two different access patterns for this device memory (DM). One is to use this device memory (DM) and second to configure/limit. Usually those actions will be performed by different groups.

First group (programmers) is using special API [1] through libibverbs [2] without any notion of cgroups or any limitations. Second group (sysadmins) is less interested in application specifics and for them "device memory" means "memory" and not "rdma, nic specific, internal memory".

Quoted 12 lines

Um... I am not sure that answered it, especially in the context of cgroup (this is just for my curiosity btw, I don't know much about rdma.) You said sysadmins are less interested in application specifics but then how would they make the judgement call on how much "device memory" is provisioned to one application/container over another (let say you have 5 cgroup sharing an rdma device)? What are the consequences of under provisioning "device memory" to an application? And if they are all just memory, can a sysadmin provision more system memory in place of device memory (like, are they interchangeable)? I guess I am confused because if device memory is just memory (not rdma, nic specific) to sysadmins how would they know to set the right amount?

One of the immediate usages of this DM that come to my mind is very fast spinlocks for MPI applications. In such case, the amount of DM will be property of network topology in given MPI cluster.

In this scenario, precise amount of memory will ensure that all jobs will continue to give maximal performance despite any programmer's error in DM allocation.

For under provisioning scenario and if application is written correctly, users will experience more latency and less performance, due to the PCI accesses.

Slide 3 in Liran's presentation gives brief overview about motivation.

Thanks

Quoted 97 lines

Regards,
Kenny

Quoted 93 lines

[1] ibv_alloc_dm()
http://man7.org/linux/man-pages/man3/ibv_alloc_dm.3.html
https://www.openfabrics.org/images/2018workshop/presentations/304_LLiss_OnDeviceMemory.pdf
[2] https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/

Quoted 88 lines

I think we need to be careful about drawing the line between duplication and over couplings between subsystems. I have other thoughts and concerns and I will try to organize them into a response in the next few days.

Regards,
Kenny

Quoted 78 lines
Quoted 78 lines

Is AMD interested in collaborating to help shape this framework? It is intended to be device-neutral, so could be leveraged by various types of devices.
If you have an alternative solution well underway, then maybe we can work together to merge our efforts into one.
In the end, the DRM community is best served with common solution.

Quoted 69 lines
Quoted 69 lines
Quoted 62 lines

and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

Per-device controls would be exposed in cgroup filesystem as:

    mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/<subsys_files>

such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.weight

The drm/i915 patch in this series is based on top of other RFC work [1] for i915 device memory support.

AMD [2] and Intel [3] have proposed related work in this area within the last few years, listed below as reference. This new RFC reuses existing cgroup controllers and takes a different approach than prior work.

Finally, some potential discussion points for this series:
* merge proposed <subsys_name>.devices into a single devices directory?
* allow devices to have multiple registrations for subsets of resources?
* document a 'common charging policy' for device drivers to follow?

[1] https://patchwork.freedesktop.org/series/56683/
[2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html
[3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html

Brian Welty (5):
cgroup: Add cgroup_subsys per-device registration framework
cgroup: Change kernfs_node for directories to store
cgroup_subsys_state
memcg: Add per-device support to memory cgroup subsystem
drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit drm/i915: Use memory cgroup for enforcing device memory limit

 drivers/gpu/drm/drm_drv.c                  |  12 +
 drivers/gpu/drm/drm_gem.c                  |   7 +
 drivers/gpu/drm/i915/i915_drv.c            |   2 +-
 drivers/gpu/drm/i915/intel_memory_region.c |  24 +-
 include/drm/drm_device.h                   |   3 +
 include/drm/drm_drv.h                      |   8 +
 include/drm/drm_gem.h                      |  11 +
 include/linux/cgroup-defs.h                |  28 ++
 include/linux/cgroup.h                     |   3 +
 include/linux/memcontrol.h                 |  10 +
 kernel/cgroup/cgroup-v1.c                  |  10 +-
 kernel/cgroup/cgroup.c                     | 310 ++++++++++++++++++---
 mm/memcontrol.c                            | 183 +++++++++++-
 13 files changed, 552 insertions(+), 59 deletions(-)
Quoted 3 lines

--
2.21.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#14 lore raw
In reply to Leon Romanovsky
Quoted 4 lines · Leon Romanovsky

First group (programmers) is using special API [1] through libibverbs [2] without any notion of cgroups or any limitations. Second group (sysadmins) is less interested in application specifics and for them "device memory" means "memory" and not "rdma, nic specific, internal memory".

I'd suggest otherwise, based on historic precedent -- sysadmins are typically very opinionated about operation of the memory subsystem (hence the endless discussions about swap, caching behaviour, etc).

Especially in this case, these types of memory operate fundamentally differently and have significantly different performance and availability characteristics. That's not something that can be trivially abstracted over without non-trivial drawbacks.
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

maintainer
#15 lore raw
In reply to Brian Welty
Quoted 7 lines · Brian Welty

In containerized or virtualized environments, there is desire to have controls in place for resources that can be consumed by users of a GPU device. This RFC patch series proposes a framework for integrating use of existing cgroup controllers into device drivers.
The i915 driver is updated in this series as our primary use case to leverage this framework and to serve as an example for discussion.

Quoted 6 lines

The patch series enables device drivers to use cgroups to control the following resources within a GPU (or other accelerator device):
* control allocation of device memory (reuse of memcg) and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Please create a separate controller for your purposes.

The memory controller is for traditional RAM. I don't see it having much in common with what you're trying to do, and it's barely reusing any of the memcg code. You can use the page_counter API directly.

Quoted 9 lines

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

Quoted 7 lines

Per-device controls would be exposed in cgroup filesystem as:

    mount/<cgroup_name>/<subsys_name>.devices/<dev_name>/<subsys_files>

such as (for example):
mount/<cgroup_name>/memory.devices/<dev_name>/memory.max mount/<cgroup_name>/memory.devices/<dev_name>/memory.current mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.stat mount/<cgroup_name>/cpu.devices/<dev_name>/cpu.weight

Subdirectories for anything other than actual cgroups are a no-go. If you need a hierarchy, use dotted filenames:

gpu.memory.max
gpu.cycles.max

etc. and look at Documentation/admin-guide/cgroup-v2.rst's 'Format' and 'Conventions', as well as how the io controller works, to see how multi-key / multi-device control files are implemented in cgroup2.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

maintainer
#16 lore raw
In reply to Brian Welty

Hello,

Quoted 7 lines · Brian Welty

The patch series enables device drivers to use cgroups to control the following resources within a GPU (or other accelerator device):
* control allocation of device memory (reuse of memcg) and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Quoted 8 lines

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

I'm really skeptical about this approach. When creating resource controllers, I think what's the most important and challenging is establishing resource model - what resources are and how they can be distributed. This patchset is going the other way around - building out core infrastructure for bolierplates at a significant risk of mixing up resource models across different types of resources.

IO controllers already implement per-device controls. I'd suggest following the same interface conventions and implementing a dedicated controller for the subsystem.

Thanks.

Signature · 5 lines
tejun
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#17 lore raw
Quoted 26 lines · Tejun Heo

Hello,

Quoted 7 lines · Brian Welty

The patch series enables device drivers to use cgroups to control the following resources within a GPU (or other accelerator device):
* control allocation of device memory (reuse of memcg) and with future work, we could extend to:
* track and control share of GPU time (reuse of cpu/cpuacct)
* apply mask of allowed execution engines (reuse of cpusets)

Quoted 8 lines

Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources.
Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones.

I'm really skeptical about this approach. When creating resource controllers, I think what's the most important and challenging is establishing resource model - what resources are and how they can be distributed. This patchset is going the other way around - building out core infrastructure for bolierplates at a significant risk of mixing up resource models across different types of resources.

Quoted 4 lines

IO controllers already implement per-device controls. I'd suggest following the same interface conventions and implementing a dedicated controller for the subsystem.

Okay, thanks for feedback. Preference is clear to see this done as dedicated cgroup controller.

Part of my proposal was an attempt for devices with "mem like" and "cpu
like" attributes to be managed by common controller. We can ignore this
idea for cpu attributes, as those can just go in a GPU controller.

There might still be merit in having a 'device mem' cgroup controller. The resource model at least is then no longer mixed up with host memory. RDMA community seemed to have some interest in a common controller at least for device memory aspects.
Thoughts on this? I believe could still reuse the 'struct mem_cgroup' data
structure. There should be some opportunity to reuse charging APIs and have some nice integration with HMM for charging to device memory, depending on backing store.

-Brian
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

maintainer
#18 lore raw

Hello,

Quoted 8 lines · Brian

There might still be merit in having a 'device mem' cgroup controller. The resource model at least is then no longer mixed up with host memory. RDMA community seemed to have some interest in a common controller at least for device memory aspects.
Thoughts on this? I believe could still reuse the 'struct mem_cgroup' data
structure. There should be some opportunity to reuse charging APIs and have some nice integration with HMM for charging to device memory, depending on backing store.

Library-ish sharing is fine but in terms of interface, I think it'd be better to keep them separate at least for now. Down the line maybe these resources will interact with each other in a more integrated way but I don't think it's a good idea to try to design and implement resource models for something like that preemptively.

Thanks.

Signature · 5 lines
tejun
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel