Skip to contentSkip to conversation

inotify, memcg: account inotify instances to kmemcg

PATCH Superseded Shakeel Butt 2 participants 8 messages Dec 18, 2020 – Dec 20, 2020

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 inotify, memcg: account inotify instances to kmemcg

Currently the fs sysctl inotify/max_user_instances is used to limit the number of inotify instances on the system. For systems running multiple workloads, the per-user namespace sysctl max_inotify_instances can be used to further partition inotify instances. However there is no easy way to set a sensible system level max limit on inotify instances and further partition it between the workloads. It is much easier to charge the underlying resource (i.e. memory) behind the inotify instances to the memcg of the workload and let their memory limits limit the number of inotify instances they can create.

---
 fs/notify/group.c                | 14 ++++++++++++--
 fs/notify/inotify/inotify_user.c |  5 +++--
 include/linux/fsnotify_backend.h |  2 ++
 3 files changed, 17 insertions(+), 4 deletions(-)
fs/notify/group.c +12 -2
index a4a4b1c64d32..fab3cfdb4d9e 100644
@@ -114,11 +114,12 @@ EXPORT_SYMBOL_GPL(fsnotify_put_group);
 /*
  * Create a new fsnotify_group and hold a reference for the group returned.
  */
-struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
+struct fsnotify_group *fsnotify_alloc_group_gfp(const struct fsnotify_ops *ops,
+						gfp_t gfp)
 {
 	struct fsnotify_group *group;
 
-	group = kzalloc(sizeof(struct fsnotify_group), GFP_KERNEL);
+	group = kzalloc(sizeof(struct fsnotify_group), gfp);
 	if (!group)
 		return ERR_PTR(-ENOMEM);
 
@@ -139,6 +140,15 @@ struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
 
 	return group;
 }
+EXPORT_SYMBOL_GPL(fsnotify_alloc_group_gfp);
+
+/*
+ * Create a new fsnotify_group and hold a reference for the group returned.
+ */
+struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
+{
+	return fsnotify_alloc_group_gfp(ops, GFP_KERNEL);
+}
 EXPORT_SYMBOL_GPL(fsnotify_alloc_group);
 
 int fsnotify_fasync(int fd, struct file *file, int on)
fs/notify/inotify/inotify_user.c +3 -2
index 59c177011a0f..7cb528c6154c 100644
@@ -632,11 +632,12 @@ static struct fsnotify_group *inotify_new_group(unsigned int max_events)
 	struct fsnotify_group *group;
 	struct inotify_event_info *oevent;
 
-	group = fsnotify_alloc_group(&inotify_fsnotify_ops);
+	group = fsnotify_alloc_group_gfp(&inotify_fsnotify_ops,
+					 GFP_KERNEL_ACCOUNT);
 	if (IS_ERR(group))
 		return group;
 
-	oevent = kmalloc(sizeof(struct inotify_event_info), GFP_KERNEL);
+	oevent = kmalloc(sizeof(struct inotify_event_info), GFP_KERNEL_ACCOUNT);
 	if (unlikely(!oevent)) {
 		fsnotify_destroy_group(group);
 		return ERR_PTR(-ENOMEM);
include/linux/fsnotify_backend.h +2 -0
index a2e42d3cd87c..a0138267f547 100644
@@ -470,6 +470,8 @@ static inline void fsnotify_update_flags(struct dentry *dentry)
 
 /* create a new group */
 extern struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops);
+extern struct fsnotify_group *fsnotify_alloc_group_gfp(const struct fsnotify_ops *ops,
+						       gfp_t gfp);
 /* get reference to a group */
 extern void fsnotify_get_group(struct fsnotify_group *group);
 /* drop reference on a group from fsnotify_alloc_group */
Signature · 1 line
2.29.2.684.gfbc64c5ab5-goog
#1 lore raw
Quoted 10 lines · Shakeel Butt

Currently the fs sysctl inotify/max_user_instances is used to limit the number of inotify instances on the system. For systems running multiple workloads, the per-user namespace sysctl max_inotify_instances can be used to further partition inotify instances. However there is no easy way to set a sensible system level max limit on inotify instances and further partition it between the workloads. It is much easier to charge the underlying resource (i.e. memory) behind the inotify instances to the memcg of the workload and let their memory limits limit the number of inotify instances they can create.

Not that I have a problem with this patch, but what problem does it try to solve? Are you concerned of users depleting system memory by creating userns's and allocating 128 * (struct fsnotify_group) at a time?

IMO, that is not what max_user_instances was meant to protect against. There are two reasons I can think of to limit user instances:
1. Pre-memgc, user allocation of events is limited to <max_user_instances>*<max_queued_events>
2. Performance penalty. User can place <max_user_instances> watches on the same "hot" directory, that will cause any access to that directory by any task on the system to pay the penalty of traversing <max_user_instances> marks and attempt to queue <max_user_instances> events. That cost, including <max_user_instances> inotify_merge() loops could be significant

#1 is not a problem anymore, since you already took care of accounting events to the user's memcg.
#2 is not addressed by your patch.

Quoted 52 lines
---
 fs/notify/group.c                | 14 ++++++++++++--
 fs/notify/inotify/inotify_user.c |  5 +++--
 include/linux/fsnotify_backend.h |  2 ++
 3 files changed, 17 insertions(+), 4 deletions(-)
fs/notify/group.c +12 -2
index a4a4b1c64d32..fab3cfdb4d9e 100644
@@ -114,11 +114,12 @@ EXPORT_SYMBOL_GPL(fsnotify_put_group);
 /*
  * Create a new fsnotify_group and hold a reference for the group returned.
  */
-struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
+struct fsnotify_group *fsnotify_alloc_group_gfp(const struct fsnotify_ops *ops,
+                                               gfp_t gfp)
 {
        struct fsnotify_group *group;
 
-       group = kzalloc(sizeof(struct fsnotify_group), GFP_KERNEL);
+       group = kzalloc(sizeof(struct fsnotify_group), gfp);
        if (!group)
                return ERR_PTR(-ENOMEM);
 
@@ -139,6 +140,15 @@ struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
 
        return group;
 }
+EXPORT_SYMBOL_GPL(fsnotify_alloc_group_gfp);
+
+/*
+ * Create a new fsnotify_group and hold a reference for the group returned.
+ */
+struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
+{
+       return fsnotify_alloc_group_gfp(ops, GFP_KERNEL);
+}
 EXPORT_SYMBOL_GPL(fsnotify_alloc_group);
 
 int fsnotify_fasync(int fd, struct file *file, int on)
fs/notify/inotify/inotify_user.c +0 -1
index 59c177011a0f..7cb528c6154c 100644
@@ -632,11 +632,12 @@ static struct fsnotify_group *inotify_new_group(unsigned int max_events)
        struct fsnotify_group *group;
        struct inotify_event_info *oevent;
 
-       group = fsnotify_alloc_group(&inotify_fsnotify_ops);

fs/notify/inotify/inotify_user.c static struct fsnotify_group *inotify_new_group(unsigned int max_events) · line 643

Quoted 10 lines
fs/notify/inotify/inotify_user.c +3 -1
+       group = fsnotify_alloc_group_gfp(&inotify_fsnotify_ops,
+                                        GFP_KERNEL_ACCOUNT);
        if (IS_ERR(group))
                return group;
 
-       oevent = kmalloc(sizeof(struct inotify_event_info), GFP_KERNEL);
+       oevent = kmalloc(sizeof(struct inotify_event_info), GFP_KERNEL_ACCOUNT);
        if (unlikely(!oevent)) {
                fsnotify_destroy_group(group);
                return ERR_PTR(-ENOMEM);

Any reason why you did not include fanotify in this patch?

Thanks,
Amir.

#2 lore raw
Quoted 13 lines · Amir Goldstein
Quoted 10 lines · Shakeel Butt

Currently the fs sysctl inotify/max_user_instances is used to limit the number of inotify instances on the system. For systems running multiple workloads, the per-user namespace sysctl max_inotify_instances can be used to further partition inotify instances. However there is no easy way to set a sensible system level max limit on inotify instances and further partition it between the workloads. It is much easier to charge the underlying resource (i.e. memory) behind the inotify instances to the memcg of the workload and let their memory limits limit the number of inotify instances they can create.

Quoted 2 lines

Not that I have a problem with this patch, but what problem does it try to solve?

I am aiming for the simplicity to not set another limit which can indirectly be limited by memcg limits. I just want to set the memcg limit on our production environment which runs multiple workloads on a system and not think about setting a sensible value to max_user_instances in production. I would prefer to set max_user_instances to max int and let the memcg limits of the workloads limit their inotify usage.

Quoted 14 lines

Are you concerned of users depleting system memory by creating userns's and allocating 128 * (struct fsnotify_group) at a time?

IMO, that is not what max_user_instances was meant to protect against. There are two reasons I can think of to limit user instances:
1. Pre-memgc, user allocation of events is limited to <max_user_instances>*<max_queued_events>
2. Performance penalty. User can place <max_user_instances> watches on the same "hot" directory, that will cause any access to that directory by any task on the system to pay the penalty of traversing <max_user_instances> marks and attempt to queue <max_user_instances> events. That cost, including <max_user_instances> inotify_merge() loops could be significant

Quoted 3 lines

#1 is not a problem anymore, since you already took care of accounting events to the user's memcg.
#2 is not addressed by your patch.

Yes, I am not addressing #2. Our workloads in prod have their own private filesystems, so this is not an issue we observed.

Quoted 58 lines
Quoted 58 lines
---
 fs/notify/group.c                | 14 ++++++++++++--
 fs/notify/inotify/inotify_user.c |  5 +++--
 include/linux/fsnotify_backend.h |  2 ++
 3 files changed, 17 insertions(+), 4 deletions(-)
fs/notify/group.c +12 -2
index a4a4b1c64d32..fab3cfdb4d9e 100644
@@ -114,11 +114,12 @@ EXPORT_SYMBOL_GPL(fsnotify_put_group);
 /*
  * Create a new fsnotify_group and hold a reference for the group returned.
  */
-struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
+struct fsnotify_group *fsnotify_alloc_group_gfp(const struct fsnotify_ops *ops,
+                                               gfp_t gfp)
 {
        struct fsnotify_group *group;
 
-       group = kzalloc(sizeof(struct fsnotify_group), GFP_KERNEL);
+       group = kzalloc(sizeof(struct fsnotify_group), gfp);
        if (!group)
                return ERR_PTR(-ENOMEM);
 
@@ -139,6 +140,15 @@ struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
 
        return group;
 }
+EXPORT_SYMBOL_GPL(fsnotify_alloc_group_gfp);
+
+/*
+ * Create a new fsnotify_group and hold a reference for the group returned.
+ */
+struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
+{
+       return fsnotify_alloc_group_gfp(ops, GFP_KERNEL);
+}
 EXPORT_SYMBOL_GPL(fsnotify_alloc_group);
 
 int fsnotify_fasync(int fd, struct file *file, int on)
fs/notify/inotify/inotify_user.c +2 -1
index 59c177011a0f..7cb528c6154c 100644
@@ -632,11 +632,12 @@ static struct fsnotify_group *inotify_new_group(unsigned int max_events)
        struct fsnotify_group *group;
        struct inotify_event_info *oevent;
 
-       group = fsnotify_alloc_group(&inotify_fsnotify_ops);
+       group = fsnotify_alloc_group_gfp(&inotify_fsnotify_ops,
+                                        GFP_KERNEL_ACCOUNT);
        if (IS_ERR(group))
                return group;
 
Quoted 7 lines

fs/notify/inotify/inotify_user.c static struct fsnotify_group *inotify_new_group(unsigned int max_events) · line 643

Quoted 5 lines
fs/notify/inotify/inotify_user.c +1 -1
-       oevent = kmalloc(sizeof(struct inotify_event_info), GFP_KERNEL);
+       oevent = kmalloc(sizeof(struct inotify_event_info), GFP_KERNEL_ACCOUNT);
        if (unlikely(!oevent)) {
                fsnotify_destroy_group(group);
                return ERR_PTR(-ENOMEM);

Any reason why you did not include fanotify in this patch?

The motivation was inotify's max_user_instances but we can charge fsnotify_group for fanotify as well. Though I would prefer that to be a separate patch. Let me know what you prefer?

Quoted 3 lines

Thanks,
Amir.

Thanks for the review. I really appreciate your time.

Shakeel

#3 lore raw
Quoted 18 lines · Shakeel Butt
Quoted 13 lines · Amir Goldstein
Quoted 10 lines · Shakeel Butt

Currently the fs sysctl inotify/max_user_instances is used to limit the number of inotify instances on the system. For systems running multiple workloads, the per-user namespace sysctl max_inotify_instances can be used to further partition inotify instances. However there is no easy way to set a sensible system level max limit on inotify instances and further partition it between the workloads. It is much easier to charge the underlying resource (i.e. memory) behind the inotify instances to the memcg of the workload and let their memory limits limit the number of inotify instances they can create.

Quoted 2 lines

Not that I have a problem with this patch, but what problem does it try to solve?

Quoted 8 lines

I am aiming for the simplicity to not set another limit which can indirectly be limited by memcg limits. I just want to set the memcg limit on our production environment which runs multiple workloads on a system and not think about setting a sensible value to max_user_instances in production. I would prefer to set max_user_instances to max int and let the memcg limits of the workloads limit their inotify usage.

understood.
and I guess the multiple workloads cannot run each in their own userns? because then you wouldn't need to change max_user_instances limit.

Quoted 87 lines
Quoted 14 lines

Are you concerned of users depleting system memory by creating userns's and allocating 128 * (struct fsnotify_group) at a time?

IMO, that is not what max_user_instances was meant to protect against. There are two reasons I can think of to limit user instances:
1. Pre-memgc, user allocation of events is limited to <max_user_instances>*<max_queued_events>
2. Performance penalty. User can place <max_user_instances> watches on the same "hot" directory, that will cause any access to that directory by any task on the system to pay the penalty of traversing <max_user_instances> marks and attempt to queue <max_user_instances> events. That cost, including <max_user_instances> inotify_merge() loops could be significant

Quoted 3 lines

#1 is not a problem anymore, since you already took care of accounting events to the user's memcg.
#2 is not addressed by your patch.

Yes, I am not addressing #2. Our workloads in prod have their own private filesystems, so this is not an issue we observed.

Quoted 58 lines
Quoted 58 lines
---
 fs/notify/group.c                | 14 ++++++++++++--
 fs/notify/inotify/inotify_user.c |  5 +++--
 include/linux/fsnotify_backend.h |  2 ++
 3 files changed, 17 insertions(+), 4 deletions(-)
fs/notify/group.c +12 -2
index a4a4b1c64d32..fab3cfdb4d9e 100644
@@ -114,11 +114,12 @@ EXPORT_SYMBOL_GPL(fsnotify_put_group);
 /*
  * Create a new fsnotify_group and hold a reference for the group returned.
  */
-struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
+struct fsnotify_group *fsnotify_alloc_group_gfp(const struct fsnotify_ops *ops,
+                                               gfp_t gfp)
 {
        struct fsnotify_group *group;
 
-       group = kzalloc(sizeof(struct fsnotify_group), GFP_KERNEL);
+       group = kzalloc(sizeof(struct fsnotify_group), gfp);
        if (!group)
                return ERR_PTR(-ENOMEM);
 
@@ -139,6 +140,15 @@ struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
 
        return group;
 }
+EXPORT_SYMBOL_GPL(fsnotify_alloc_group_gfp);
+
+/*
+ * Create a new fsnotify_group and hold a reference for the group returned.
+ */
+struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)
+{
+       return fsnotify_alloc_group_gfp(ops, GFP_KERNEL);
+}
 EXPORT_SYMBOL_GPL(fsnotify_alloc_group);
 
 int fsnotify_fasync(int fd, struct file *file, int on)
fs/notify/inotify/inotify_user.c +2 -1
index 59c177011a0f..7cb528c6154c 100644
@@ -632,11 +632,12 @@ static struct fsnotify_group *inotify_new_group(unsigned int max_events)
        struct fsnotify_group *group;
        struct inotify_event_info *oevent;
 
-       group = fsnotify_alloc_group(&inotify_fsnotify_ops);
+       group = fsnotify_alloc_group_gfp(&inotify_fsnotify_ops,
+                                        GFP_KERNEL_ACCOUNT);
        if (IS_ERR(group))
                return group;
 
Quoted 7 lines

fs/notify/inotify/inotify_user.c static struct fsnotify_group *inotify_new_group(unsigned int max_events) · line 643

Quoted 5 lines
fs/notify/inotify/inotify_user.c +1 -1
-       oevent = kmalloc(sizeof(struct inotify_event_info), GFP_KERNEL);
+       oevent = kmalloc(sizeof(struct inotify_event_info), GFP_KERNEL_ACCOUNT);
        if (unlikely(!oevent)) {
                fsnotify_destroy_group(group);
                return ERR_PTR(-ENOMEM);

Any reason why you did not include fanotify in this patch?

Quoted 4 lines

The motivation was inotify's max_user_instances but we can charge fsnotify_group for fanotify as well. Though I would prefer that to be a separate patch. Let me know what you prefer?

I would prefer to add the helper fsnotify_alloc_user_group() that will use the GFP_KERNEL_ACCOUNT allocation flags internally.

fsnotify_alloc_group() is used by all backends that initialize a single group instance for internal use and fsnotify_alloc_user_group() will be used by inotify/fanotify when users create instances.
I see no reason to separate that to two patches.

Thanks,
Amir.

#4 lore raw
Quoted 29 lines · Amir Goldstein
Quoted 18 lines · Shakeel Butt
Quoted 13 lines · Amir Goldstein
Quoted 10 lines · Shakeel Butt

Currently the fs sysctl inotify/max_user_instances is used to limit the number of inotify instances on the system. For systems running multiple workloads, the per-user namespace sysctl max_inotify_instances can be used to further partition inotify instances. However there is no easy way to set a sensible system level max limit on inotify instances and further partition it between the workloads. It is much easier to charge the underlying resource (i.e. memory) behind the inotify instances to the memcg of the workload and let their memory limits limit the number of inotify instances they can create.

Quoted 2 lines

Not that I have a problem with this patch, but what problem does it try to solve?

Quoted 8 lines

I am aiming for the simplicity to not set another limit which can indirectly be limited by memcg limits. I just want to set the memcg limit on our production environment which runs multiple workloads on a system and not think about setting a sensible value to max_user_instances in production. I would prefer to set max_user_instances to max int and let the memcg limits of the workloads limit their inotify usage.

Quoted 4 lines

understood.
and I guess the multiple workloads cannot run each in their own userns? because then you wouldn't need to change max_user_instances limit.

No workloads can run in their own user namespace but please note that max_user_instances is shared between all the user namespaces.

Quoted 1 line

[snip]

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Any reason why you did not include fanotify in this patch?

The motivation was inotify's max_user_instances but we can charge fsnotify_group for fanotify as well. Though I would prefer that to be a separate patch. Let me know what you prefer?

Quoted 9 lines

I would prefer to add the helper fsnotify_alloc_user_group() that will use the GFP_KERNEL_ACCOUNT allocation flags internally.

fsnotify_alloc_group() is used by all backends that initialize a single group instance for internal use and fsnotify_alloc_user_group() will be used by inotify/fanotify when users create instances.
I see no reason to separate that to two patches.

SGTM.

#5 lore raw
Quoted 31 lines · Shakeel Butt
Quoted 29 lines · Amir Goldstein
Quoted 26 lines · Shakeel Butt
Quoted 13 lines · Amir Goldstein
Quoted 10 lines · Shakeel Butt

Currently the fs sysctl inotify/max_user_instances is used to limit the number of inotify instances on the system. For systems running multiple workloads, the per-user namespace sysctl max_inotify_instances can be used to further partition inotify instances. However there is no easy way to set a sensible system level max limit on inotify instances and further partition it between the workloads. It is much easier to charge the underlying resource (i.e. memory) behind the inotify instances to the memcg of the workload and let their memory limits limit the number of inotify instances they can create.

Quoted 2 lines

Not that I have a problem with this patch, but what problem does it try to solve?

I am aiming for the simplicity to not set another limit which can indirectly be limited by memcg limits. I just want to set the memcg limit on our production environment which runs multiple workloads on a system and not think about setting a sensible value to max_user_instances in production. I would prefer to set max_user_instances to max int and let the memcg limits of the workloads limit their inotify usage.

Quoted 7 lines
Quoted 4 lines

understood.
and I guess the multiple workloads cannot run each in their own userns? because then you wouldn't need to change max_user_instances limit.

No workloads can run in their own user namespace but please note that max_user_instances is shared between all the user namespaces.

/proc/sys/fs/inotify/max_user_instances is shared between all the user namespaces, but it only controls the init_user_ns limits. /proc/sys/user/max_inotify_instances is per user ns and it is the one that actually controls the inotify limits in non init_user_ns.

That said, I see that it is always initialized to MAX_INT on non init user ns, which is exactly the setup that you are aiming at:

$ unshare -U
$ cat /proc/sys/user/max_inotify_instances
2147483647
$ cat /proc/sys/fs/inotify/max_user_instances
128

Thanks,
Amir.

#6 lore raw
Quoted 46 lines · Amir Goldstein
Quoted 31 lines · Shakeel Butt
Quoted 29 lines · Amir Goldstein
Quoted 26 lines · Shakeel Butt
Quoted 13 lines · Amir Goldstein
Quoted 10 lines · Shakeel Butt

Currently the fs sysctl inotify/max_user_instances is used to limit the number of inotify instances on the system. For systems running multiple workloads, the per-user namespace sysctl max_inotify_instances can be used to further partition inotify instances. However there is no easy way to set a sensible system level max limit on inotify instances and further partition it between the workloads. It is much easier to charge the underlying resource (i.e. memory) behind the inotify instances to the memcg of the workload and let their memory limits limit the number of inotify instances they can create.

Quoted 2 lines

Not that I have a problem with this patch, but what problem does it try to solve?

I am aiming for the simplicity to not set another limit which can indirectly be limited by memcg limits. I just want to set the memcg limit on our production environment which runs multiple workloads on a system and not think about setting a sensible value to max_user_instances in production. I would prefer to set max_user_instances to max int and let the memcg limits of the workloads limit their inotify usage.

Quoted 7 lines
Quoted 4 lines

understood.
and I guess the multiple workloads cannot run each in their own userns? because then you wouldn't need to change max_user_instances limit.

No workloads can run in their own user namespace but please note that max_user_instances is shared between all the user namespaces.

/proc/sys/fs/inotify/max_user_instances is shared between all the user namespaces, but it only controls the init_user_ns limits. /proc/sys/user/max_inotify_instances is per user ns and it is the one that actually controls the inotify limits in non init_user_ns.

Quoted 8 lines

That said, I see that it is always initialized to MAX_INT on non init user ns, which is exactly the setup that you are aiming at:

$ unshare -U
$ cat /proc/sys/user/max_inotify_instances
2147483647
$ cat /proc/sys/fs/inotify/max_user_instances
128

From what I understand, namespace-based limits are enforced hierarchically. More specifically in the example above, the application running in a user namespace with /proc/sys/user/max_inotify_instances = 2147483647 and /proc/sys/fs/inotify/max_user_instances = 128 will not be able to create more than 128 inotify instances. I actually tested this with a simple program which calls inotify_init() in a loop and it starts failing before the 128th iteration.

#7 lore raw
Quoted 57 lines · Shakeel Butt
Quoted 46 lines · Amir Goldstein
Quoted 31 lines · Shakeel Butt
Quoted 29 lines · Amir Goldstein
Quoted 26 lines · Shakeel Butt
Quoted 13 lines · Amir Goldstein
Quoted 10 lines · Shakeel Butt

Currently the fs sysctl inotify/max_user_instances is used to limit the number of inotify instances on the system. For systems running multiple workloads, the per-user namespace sysctl max_inotify_instances can be used to further partition inotify instances. However there is no easy way to set a sensible system level max limit on inotify instances and further partition it between the workloads. It is much easier to charge the underlying resource (i.e. memory) behind the inotify instances to the memcg of the workload and let their memory limits limit the number of inotify instances they can create.

Quoted 2 lines

Not that I have a problem with this patch, but what problem does it try to solve?

I am aiming for the simplicity to not set another limit which can indirectly be limited by memcg limits. I just want to set the memcg limit on our production environment which runs multiple workloads on a system and not think about setting a sensible value to max_user_instances in production. I would prefer to set max_user_instances to max int and let the memcg limits of the workloads limit their inotify usage.

Quoted 7 lines
Quoted 4 lines

understood.
and I guess the multiple workloads cannot run each in their own userns? because then you wouldn't need to change max_user_instances limit.

No workloads can run in their own user namespace but please note that max_user_instances is shared between all the user namespaces.

/proc/sys/fs/inotify/max_user_instances is shared between all the user namespaces, but it only controls the init_user_ns limits. /proc/sys/user/max_inotify_instances is per user ns and it is the one that actually controls the inotify limits in non init_user_ns.

Quoted 8 lines

That said, I see that it is always initialized to MAX_INT on non init user ns, which is exactly the setup that you are aiming at:

$ unshare -U
$ cat /proc/sys/user/max_inotify_instances
2147483647
$ cat /proc/sys/fs/inotify/max_user_instances
128
Quoted 8 lines

From what I understand, namespace-based limits are enforced hierarchically. More specifically in the example above, the application running in a user namespace with /proc/sys/user/max_inotify_instances = 2147483647 and /proc/sys/fs/inotify/max_user_instances = 128 will not be able to create more than 128 inotify instances. I actually tested this with a simple program which calls inotify_init() in a loop and it starts failing before the 128th iteration.

Right, it is.
Thanks for the clarification.

Thanks,
Amir.