Skip to contentSkip to conversation

net: Reduce rcu_barrier() contentions from 'unshare(CLONE_NEWNET)'

PATCH Superseded SeongJae Park 2 participants 6 messages Dec 11, 2020

Subsystems (4)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover net: Reduce rcu_barrier() contentions from 'unshare(CLONE_NEWNET)'

From: SeongJae Park <[email protected]>

On a few of our systems, I found frequent 'unshare(CLONE_NEWNET)' calls make the number of active slab objects including 'sock_inode_cache' type rapidly and continuously increase. As a result, memory pressure occurs.

In more detail, I made an artificial reproducer that resembles the workload that we found the problem and reproduce the problem faster. It merely repeats 'unshare(CLONE_NEWNET)' 50,000 times in a loop. It takes about 2 minutes. On 40 CPU cores, 70GB DRAM machine, the available memory continuously reduced in a fast speed (about 120MB per second, 15GB in total within the 2 minutes). Note that the issue don't reproduce on every machine. On my 6 CPU cores machine, the problem didn't reproduce.

'cleanup_net()' and 'fqdir_work_fn()' are functions that deallocate the relevant memory objects. They are asynchronously invoked by the work queues and internally use 'rcu_barrier()' to ensure safe destructions. 'cleanup_net()' works in a batched maneer in a single thread worker, while 'fqdir_work_fn()' works for each 'fqdir_exit()' call in the 'system_wq'.

Therefore, 'fqdir_work_fn()' called frequently under the workload and made the contention for 'rcu_barrier()' high. In more detail, the global mutex, 'rcu_state.barrier_mutex' became the bottleneck.

I tried making 'rcu_barrier()' and subsequent lightweight works in 'fqdir_work_fn()' to be processed by a dedicated singlethread worker in batch and confirmed it works. After the change, No continuous memory reduction but some fluctuation observed. Nevertheless, the available memory reduction was only up to about 400MB. The following patch is for the change. I think this is the right solution for point fix of this issue, but someone might blame different parts.

1. User: Frequent 'unshare()' calls
From some point of view, such frequent 'unshare()' calls might seem only insane.

2. Global mutex in 'rcu_barrier()'
Because of the global mutex, 'rcu_barrier()' callers could wait long even after the callbacks started before the call finished. Therefore, similar issues could happen in another 'rcu_barrier()' usages. Maybe we can use some wait queue like mechanism to notify the waiters when the desired time came.

I personally believe applying the point fix for now and making 'rcu_barrier()' improvement in longterm make sense. If I'm missing something or you have different opinion, please feel free to let me know.

Patch History
-------------

Changes from v2
(https://lore.kernel.org/lkml/[email protected]/)
- Add numbers after the patch (Eric Dumazet)
- Make only 'rcu_barrier()' and subsequent lightweight works serialized (Eric Dumazet)

Changes from v1
(https://lore.kernel.org/netdev/[email protected]/)
- Keep xmas tree variable ordering (Jakub Kicinski)
- Add more numbers (Eric Dumazet)
- Use 'llist_for_each_entry_safe()' (Eric Dumazet)

SeongJae Park (1):
net/ipv4/inet_fragment: Batch fqdir destroy works

 include/net/inet_frag.h  |  1 +
 net/ipv4/inet_fragment.c | 45 +++++++++++++++++++++++++++++++++-------
 2 files changed, 39 insertions(+), 7 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH v3 1/1 net/ipv4/inet_fragment: Batch fqdir destroy works

From: SeongJae Park <[email protected]>

For each 'fqdir_exit()' call, a work for destroy of the 'fqdir' is enqueued. The work function, 'fqdir_work_fn()', internally calls 'rcu_barrier()'. In case of intensive 'fqdir_exit()' (e.g., frequent 'unshare()' systemcalls), this increased contention could result in unacceptably high latency of 'rcu_barrier()'. This commit avoids such contention by doing the 'rcu_barrier()' and subsequent lightweight works in a batched manner using a dedicated singlethread worker, as similar to that of 'cleanup_net()'.

---
 include/net/inet_frag.h  |  1 +
 net/ipv4/inet_fragment.c | 45 +++++++++++++++++++++++++++++++++-------
 2 files changed, 39 insertions(+), 7 deletions(-)
include/net/inet_frag.h +1 -0
index bac79e817776..48cc5795ceda 100644
@@ -21,6 +21,7 @@ struct fqdir {
 	/* Keep atomic mem on separate cachelines in structs that include it */
 	atomic_long_t		mem ____cacheline_aligned_in_smp;
 	struct work_struct	destroy_work;
+	struct llist_node	free_list;
 };
 
 /**
net/ipv4/inet_fragment.c +38 -7

62 lines not on this page · raw message

Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH v3 1/1] net/ipv4/inet_fragment: Batch fqdir destroy works

Quoted 11 lines · SeongJae Park

From: SeongJae Park <[email protected]>

For each 'fqdir_exit()' call, a work for destroy of the 'fqdir' is enqueued. The work function, 'fqdir_work_fn()', internally calls 'rcu_barrier()'. In case of intensive 'fqdir_exit()' (e.g., frequent 'unshare()' systemcalls), this increased contention could result in unacceptably high latency of 'rcu_barrier()'. This commit avoids such contention by doing the 'rcu_barrier()' and subsequent lightweight works in a batched manner using a dedicated singlethread worker, as similar to that of 'cleanup_net()'.

Not sure why you submit a patch series with a single patch.

Your cover letter contains interesting info that would better be captured in this changelog IMO

Quoted 61 lines
---
 include/net/inet_frag.h  |  1 +
 net/ipv4/inet_fragment.c | 45 +++++++++++++++++++++++++++++++++-------
 2 files changed, 39 insertions(+), 7 deletions(-)
include/net/inet_frag.h +1 -0
index bac79e817776..48cc5795ceda 100644
@@ -21,6 +21,7 @@ struct fqdir {
        /* Keep atomic mem on separate cachelines in structs that include it */
        atomic_long_t           mem ____cacheline_aligned_in_smp;
        struct work_struct      destroy_work;
+       struct llist_node       free_list;
 };
 
 /**
net/ipv4/inet_fragment.c +15 -7
index 10d31733297d..a6fc4afcc323 100644
@@ -145,12 +145,17 @@ static void inet_frags_free_cb(void *ptr, void *arg)
                inet_frag_destroy(fq);
 }
 
-static void fqdir_work_fn(struct work_struct *work)
+static struct workqueue_struct *fqdir_wq;
+static LLIST_HEAD(free_list);
+
+static void fqdir_free_fn(struct work_struct *work)
 {
-       struct fqdir *fqdir = container_of(work, struct fqdir, destroy_work);
-       struct inet_frags *f = fqdir->f;
+       struct llist_node *kill_list;
+       struct fqdir *fqdir, *tmp;
+       struct inet_frags *f;
 
-       rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
+       /* Atomically snapshot the list of fqdirs to free */
+       kill_list = llist_del_all(&free_list);
 
        /* We need to make sure all ongoing call_rcu(..., inet_frag_destroy_rcu)
         * have completed, since they need to dereference fqdir.
@@ -158,12 +163,38 @@ static void fqdir_work_fn(struct work_struct *work)
         */
        rcu_barrier();
 
-       if (refcount_dec_and_test(&f->refcnt))
-               complete(&f->completion);
+       llist_for_each_entry_safe(fqdir, tmp, kill_list, free_list) {
+               f = fqdir->f;
+               if (refcount_dec_and_test(&f->refcnt))
+                       complete(&f->completion);
 
-       kfree(fqdir);
+               kfree(fqdir);
+       }
 }
 

net/ipv4/inet_fragment.c static void fqdir_work_fn(struct work_struct *work) · line 184

Quoted 10 lines
net/ipv4/inet_fragment.c +10 -0
+static DECLARE_WORK(fqdir_free_work, fqdir_free_fn);
+
+static void fqdir_work_fn(struct work_struct *work)
+{
+       struct fqdir *fqdir = container_of(work, struct fqdir, destroy_work);
+
+       rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
+
+       if (llist_add(&fqdir->free_list, &free_list))
+               queue_work(fqdir_wq, &fqdir_free_work);

I think you misunderstood me.

Since this fqdir_free_work will have at most one instance, you can use system_wq here, there is no risk of abuse.

My suggestion was to not use system_wq for fqdir_exit(), to better control the number
of threads in rhashtable cleanups.

void fqdir_exit(struct fqdir *fqdir)

{
        INIT_WORK(&fqdir->destroy_work, fqdir_work_fn);

- queue_work(system_wq, &fqdir->destroy_work);
+ queue_work(fqdir_wq, &fqdir->destroy_work);
}

net/ipv4/inet_fragment.c static void fqdir_work_fn(struct work_struct *work) · line 189

Quoted 5 lines
net/ipv4/inet_fragment.c +5 -0
+}
+
+static int __init fqdir_wq_init(void)
+{
+       fqdir_wq = create_singlethread_workqueue("fqdir");

And here, I suggest to use a non ordered work queue, allowing one thread per cpu, to allow concurrent rhashtable cleanups

Also "fqdir" name is rather vague, this is an implementation detail ?

fqdir_wq =create_workqueue("inet_frag_wq");

Quoted 14 lines
net/ipv4/inet_fragment.c +8 -1
+       if (!fqdir_wq)
+               panic("Could not create fqdir workq");
+       return 0;
+}
+
+pure_initcall(fqdir_wq_init);
+
+
 int fqdir_init(struct fqdir **fqdirp, struct inet_frags *f, struct net *net)
 {
        struct fqdir *fqdir = kzalloc(sizeof(*fqdir), GFP_KERNEL);
--

2.17.1

#3 lore raw

Re: [PATCH v3 1/1] net/ipv4/inet_fragment: Batch fqdir destroy works

Quoted 14 lines · On Fri, 11 Dec 2020 09:43:41 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 11 lines · SeongJae Park

From: SeongJae Park <[email protected]>

For each 'fqdir_exit()' call, a work for destroy of the 'fqdir' is enqueued. The work function, 'fqdir_work_fn()', internally calls 'rcu_barrier()'. In case of intensive 'fqdir_exit()' (e.g., frequent 'unshare()' systemcalls), this increased contention could result in unacceptably high latency of 'rcu_barrier()'. This commit avoids such contention by doing the 'rcu_barrier()' and subsequent lightweight works in a batched manner using a dedicated singlethread worker, as similar to that of 'cleanup_net()'.

Quoted 4 lines

Not sure why you submit a patch series with a single patch.

Your cover letter contains interesting info that would better be captured in this changelog IMO

I thought someone might think this is not a kernel issue but the reproducer is insane or 'rcu_barrier()' needs modification. I wanted to do such discussion on the coverletter. Seems I misjudged. I will make this single patch and move the detailed information here from the next version.

Quoted 87 lines
Quoted 62 lines
---
 include/net/inet_frag.h  |  1 +
 net/ipv4/inet_fragment.c | 45 +++++++++++++++++++++++++++++++++-------
 2 files changed, 39 insertions(+), 7 deletions(-)
include/net/inet_frag.h +1 -0
index bac79e817776..48cc5795ceda 100644
@@ -21,6 +21,7 @@ struct fqdir {
        /* Keep atomic mem on separate cachelines in structs that include it */
        atomic_long_t           mem ____cacheline_aligned_in_smp;
        struct work_struct      destroy_work;
+       struct llist_node       free_list;
 };
 
 /**
net/ipv4/inet_fragment.c +15 -7
index 10d31733297d..a6fc4afcc323 100644
@@ -145,12 +145,17 @@ static void inet_frags_free_cb(void *ptr, void *arg)
                inet_frag_destroy(fq);
 }
 
-static void fqdir_work_fn(struct work_struct *work)
+static struct workqueue_struct *fqdir_wq;
+static LLIST_HEAD(free_list);
+
+static void fqdir_free_fn(struct work_struct *work)
 {
-       struct fqdir *fqdir = container_of(work, struct fqdir, destroy_work);
-       struct inet_frags *f = fqdir->f;
+       struct llist_node *kill_list;
+       struct fqdir *fqdir, *tmp;
+       struct inet_frags *f;
 
-       rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
+       /* Atomically snapshot the list of fqdirs to free */
+       kill_list = llist_del_all(&free_list);
 
        /* We need to make sure all ongoing call_rcu(..., inet_frag_destroy_rcu)
         * have completed, since they need to dereference fqdir.
@@ -158,12 +163,38 @@ static void fqdir_work_fn(struct work_struct *work)
         */
        rcu_barrier();
 
-       if (refcount_dec_and_test(&f->refcnt))
-               complete(&f->completion);
+       llist_for_each_entry_safe(fqdir, tmp, kill_list, free_list) {
+               f = fqdir->f;
+               if (refcount_dec_and_test(&f->refcnt))
+                       complete(&f->completion);
 
-       kfree(fqdir);
+               kfree(fqdir);
+       }
 }
 

net/ipv4/inet_fragment.c static void fqdir_work_fn(struct work_struct *work) · line 184

Quoted 10 lines
net/ipv4/inet_fragment.c +10 -0
+static DECLARE_WORK(fqdir_free_work, fqdir_free_fn);
+
+static void fqdir_work_fn(struct work_struct *work)
+{
+       struct fqdir *fqdir = container_of(work, struct fqdir, destroy_work);
+
+       rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
+
+       if (llist_add(&fqdir->free_list, &free_list))
+               queue_work(fqdir_wq, &fqdir_free_work);

I think you misunderstood me.

Since this fqdir_free_work will have at most one instance, you can use system_wq here, there is no risk of abuse.

My suggestion was to not use system_wq for fqdir_exit(), to better control the number
of threads in rhashtable cleanups.

void fqdir_exit(struct fqdir *fqdir)

{
        INIT_WORK(&fqdir->destroy_work, fqdir_work_fn);
(quoted diff) +1 -1
-       queue_work(system_wq, &fqdir->destroy_work);
+      queue_work(fqdir_wq, &fqdir->destroy_work);
Quoted 1 line

}

Oh, got it. I definitely misunderstood. My fault, sorry.

Quoted 13 lines

net/ipv4/inet_fragment.c static void fqdir_work_fn(struct work_struct *work) · line 189

Quoted 8 lines
net/ipv4/inet_fragment.c +5 -0
+}
+
+static int __init fqdir_wq_init(void)
+{
+       fqdir_wq = create_singlethread_workqueue("fqdir");

And here, I suggest to use a non ordered work queue, allowing one thread per cpu, to allow concurrent rhashtable cleanups

Quoted 3 lines

Also "fqdir" name is rather vague, this is an implementation detail ?

fqdir_wq =create_workqueue("inet_frag_wq");

So, what you are suggesting is to use a dedicated non-ordered work queue (fqdir_wq) for rhashtable cleanup and do the remaining works with system_wq in the batched manner, right? IOW, doing below change on top of this patch.

net/ipv4/inet_fragment.c +6 -6
@@ -145,7 +145,7 @@ static void inet_frags_free_cb(void *ptr, void *arg)
                inet_frag_destroy(fq);
 }
 
-static struct workqueue_struct *fqdir_wq;
+static struct workqueue_struct *inet_frag_wq;
 static LLIST_HEAD(free_list);
 
 static void fqdir_free_fn(struct work_struct *work)
@@ -181,14 +181,14 @@ static void fqdir_work_fn(struct work_struct *work)
        rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
 
        if (llist_add(&fqdir->free_list, &free_list))
-               queue_work(fqdir_wq, &fqdir_free_work);
+               queue_work(system_wq, &fqdir_free_work);
 }
 
 static int __init fqdir_wq_init(void)
 {
-       fqdir_wq = create_singlethread_workqueue("fqdir");
-       if (!fqdir_wq)
-               panic("Could not create fqdir workq");
+       inet_frag_wq = create_workqueue("inet_frag_wq");
+       if (!inet_frag_wq)
+               panic("Could not create inet frag workq");
        return 0;
 }
 
@@ -218,7 +218,7 @@ EXPORT_SYMBOL(fqdir_init);
 void fqdir_exit(struct fqdir *fqdir)
 {
        INIT_WORK(&fqdir->destroy_work, fqdir_work_fn);
-       queue_work(system_wq, &fqdir->destroy_work);
+       queue_work(inet_frag_wq, &fqdir->destroy_work);
 }
 EXPORT_SYMBOL(fqdir_exit);
 

If I'm still misunderstanding, please let me know.

Thanks,
SeongJae Park

#4 lore raw

Re: [PATCH v3 1/1] net/ipv4/inet_fragment: Batch fqdir destroy works

Quoted 142 lines · SeongJae Park
Quoted 14 lines · On Fri, 11 Dec 2020 09:43:41 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 11 lines · SeongJae Park

From: SeongJae Park <[email protected]>

For each 'fqdir_exit()' call, a work for destroy of the 'fqdir' is enqueued. The work function, 'fqdir_work_fn()', internally calls 'rcu_barrier()'. In case of intensive 'fqdir_exit()' (e.g., frequent 'unshare()' systemcalls), this increased contention could result in unacceptably high latency of 'rcu_barrier()'. This commit avoids such contention by doing the 'rcu_barrier()' and subsequent lightweight works in a batched manner using a dedicated singlethread worker, as similar to that of 'cleanup_net()'.

Quoted 4 lines

Not sure why you submit a patch series with a single patch.

Your cover letter contains interesting info that would better be captured in this changelog IMO

I thought someone might think this is not a kernel issue but the reproducer is insane or 'rcu_barrier()' needs modification. I wanted to do such discussion on the coverletter. Seems I misjudged. I will make this single patch and move the detailed information here from the next version.

Quoted 87 lines
Quoted 62 lines
---
 include/net/inet_frag.h  |  1 +
 net/ipv4/inet_fragment.c | 45 +++++++++++++++++++++++++++++++++-------
 2 files changed, 39 insertions(+), 7 deletions(-)
include/net/inet_frag.h +1 -0
index bac79e817776..48cc5795ceda 100644
@@ -21,6 +21,7 @@ struct fqdir {
        /* Keep atomic mem on separate cachelines in structs that include it */
        atomic_long_t           mem ____cacheline_aligned_in_smp;
        struct work_struct      destroy_work;
+       struct llist_node       free_list;
 };
 
 /**
net/ipv4/inet_fragment.c +15 -7
index 10d31733297d..a6fc4afcc323 100644
@@ -145,12 +145,17 @@ static void inet_frags_free_cb(void *ptr, void *arg)
                inet_frag_destroy(fq);
 }
 
-static void fqdir_work_fn(struct work_struct *work)
+static struct workqueue_struct *fqdir_wq;
+static LLIST_HEAD(free_list);
+
+static void fqdir_free_fn(struct work_struct *work)
 {
-       struct fqdir *fqdir = container_of(work, struct fqdir, destroy_work);
-       struct inet_frags *f = fqdir->f;
+       struct llist_node *kill_list;
+       struct fqdir *fqdir, *tmp;
+       struct inet_frags *f;
 
-       rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
+       /* Atomically snapshot the list of fqdirs to free */
+       kill_list = llist_del_all(&free_list);
 
        /* We need to make sure all ongoing call_rcu(..., inet_frag_destroy_rcu)
         * have completed, since they need to dereference fqdir.
@@ -158,12 +163,38 @@ static void fqdir_work_fn(struct work_struct *work)
         */
        rcu_barrier();
 
-       if (refcount_dec_and_test(&f->refcnt))
-               complete(&f->completion);
+       llist_for_each_entry_safe(fqdir, tmp, kill_list, free_list) {
+               f = fqdir->f;
+               if (refcount_dec_and_test(&f->refcnt))
+                       complete(&f->completion);
 
-       kfree(fqdir);
+               kfree(fqdir);
+       }
 }
 

net/ipv4/inet_fragment.c static void fqdir_work_fn(struct work_struct *work) · line 184

Quoted 10 lines
net/ipv4/inet_fragment.c +10 -0
+static DECLARE_WORK(fqdir_free_work, fqdir_free_fn);
+
+static void fqdir_work_fn(struct work_struct *work)
+{
+       struct fqdir *fqdir = container_of(work, struct fqdir, destroy_work);
+
+       rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
+
+       if (llist_add(&fqdir->free_list, &free_list))
+               queue_work(fqdir_wq, &fqdir_free_work);

I think you misunderstood me.

Since this fqdir_free_work will have at most one instance, you can use system_wq here, there is no risk of abuse.

My suggestion was to not use system_wq for fqdir_exit(), to better control the number
of threads in rhashtable cleanups.

void fqdir_exit(struct fqdir *fqdir)

{
        INIT_WORK(&fqdir->destroy_work, fqdir_work_fn);
(quoted diff) +1 -1
-       queue_work(system_wq, &fqdir->destroy_work);
+      queue_work(fqdir_wq, &fqdir->destroy_work);
Quoted 1 line

}

Oh, got it. I definitely misunderstood. My fault, sorry.

Quoted 13 lines

net/ipv4/inet_fragment.c static void fqdir_work_fn(struct work_struct *work) · line 189

Quoted 8 lines
net/ipv4/inet_fragment.c +5 -0
+}
+
+static int __init fqdir_wq_init(void)
+{
+       fqdir_wq = create_singlethread_workqueue("fqdir");

And here, I suggest to use a non ordered work queue, allowing one thread per cpu, to allow concurrent rhashtable cleanups

Quoted 3 lines

Also "fqdir" name is rather vague, this is an implementation detail ?

fqdir_wq =create_workqueue("inet_frag_wq");

So, what you are suggesting is to use a dedicated non-ordered work queue (fqdir_wq) for rhashtable cleanup and do the remaining works with system_wq in the batched manner, right? IOW, doing below change on top of this patch.

net/ipv4/inet_fragment.c static void inet_frags_free_cb(void *ptr, void *arg) · line 149

Quoted 6 lines
net/ipv4/inet_fragment.c +1 -1
@@ -145,7 +145,7 @@ static void inet_frags_free_cb(void *ptr, void *arg)
                inet_frag_destroy(fq);
 }
 
-static struct workqueue_struct *fqdir_wq;
+static struct workqueue_struct *inet_frag_wq;
 static LLIST_HEAD(free_list);

Nit : Please prefix this free_list , like fqdir_free_list to avoid namespace pollution.

Quoted 30 lines
net/ipv4/inet_fragment.c +5 -5
 static void fqdir_free_fn(struct work_struct *work)
@@ -181,14 +181,14 @@ static void fqdir_work_fn(struct work_struct *work)
        rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
 
        if (llist_add(&fqdir->free_list, &free_list))
-               queue_work(fqdir_wq, &fqdir_free_work);
+               queue_work(system_wq, &fqdir_free_work);
 }
 
 static int __init fqdir_wq_init(void)
 {
-       fqdir_wq = create_singlethread_workqueue("fqdir");
-       if (!fqdir_wq)
-               panic("Could not create fqdir workq");
+       inet_frag_wq = create_workqueue("inet_frag_wq");
+       if (!inet_frag_wq)
+               panic("Could not create inet frag workq");
        return 0;
 }
 
@@ -218,7 +218,7 @@ EXPORT_SYMBOL(fqdir_init);
 void fqdir_exit(struct fqdir *fqdir)
 {
        INIT_WORK(&fqdir->destroy_work, fqdir_work_fn);
-       queue_work(system_wq, &fqdir->destroy_work);
+       queue_work(inet_frag_wq, &fqdir->destroy_work);
 }
 EXPORT_SYMBOL(fqdir_exit);
 
Quoted 2 lines

If I'm still misunderstanding, please let me know.

I think that with the above changes, we should be good ;)

Thanks !

#5 lore raw

Re: [PATCH v3 1/1] net/ipv4/inet_fragment: Batch fqdir destroy works

Quoted 146 lines · On Fri, 11 Dec 2020 11:41:36 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 145 lines · SeongJae Park
Quoted 14 lines · On Fri, 11 Dec 2020 09:43:41 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 11 lines · SeongJae Park

From: SeongJae Park <[email protected]>

For each 'fqdir_exit()' call, a work for destroy of the 'fqdir' is enqueued. The work function, 'fqdir_work_fn()', internally calls 'rcu_barrier()'. In case of intensive 'fqdir_exit()' (e.g., frequent 'unshare()' systemcalls), this increased contention could result in unacceptably high latency of 'rcu_barrier()'. This commit avoids such contention by doing the 'rcu_barrier()' and subsequent lightweight works in a batched manner using a dedicated singlethread worker, as similar to that of 'cleanup_net()'.

Quoted 4 lines

Not sure why you submit a patch series with a single patch.

Your cover letter contains interesting info that would better be captured in this changelog IMO

I thought someone might think this is not a kernel issue but the reproducer is insane or 'rcu_barrier()' needs modification. I wanted to do such discussion on the coverletter. Seems I misjudged. I will make this single patch and move the detailed information here from the next version.

Quoted 87 lines
Quoted 62 lines
---
 include/net/inet_frag.h  |  1 +
 net/ipv4/inet_fragment.c | 45 +++++++++++++++++++++++++++++++++-------
 2 files changed, 39 insertions(+), 7 deletions(-)
include/net/inet_frag.h +1 -0
index bac79e817776..48cc5795ceda 100644
@@ -21,6 +21,7 @@ struct fqdir {
        /* Keep atomic mem on separate cachelines in structs that include it */
        atomic_long_t           mem ____cacheline_aligned_in_smp;
        struct work_struct      destroy_work;
+       struct llist_node       free_list;
 };
 
 /**
net/ipv4/inet_fragment.c +15 -7
index 10d31733297d..a6fc4afcc323 100644
@@ -145,12 +145,17 @@ static void inet_frags_free_cb(void *ptr, void *arg)
                inet_frag_destroy(fq);
 }
 
-static void fqdir_work_fn(struct work_struct *work)
+static struct workqueue_struct *fqdir_wq;
+static LLIST_HEAD(free_list);
+
+static void fqdir_free_fn(struct work_struct *work)
 {
-       struct fqdir *fqdir = container_of(work, struct fqdir, destroy_work);
-       struct inet_frags *f = fqdir->f;
+       struct llist_node *kill_list;
+       struct fqdir *fqdir, *tmp;
+       struct inet_frags *f;
 
-       rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
+       /* Atomically snapshot the list of fqdirs to free */
+       kill_list = llist_del_all(&free_list);
 
        /* We need to make sure all ongoing call_rcu(..., inet_frag_destroy_rcu)
         * have completed, since they need to dereference fqdir.
@@ -158,12 +163,38 @@ static void fqdir_work_fn(struct work_struct *work)
         */
        rcu_barrier();
 
-       if (refcount_dec_and_test(&f->refcnt))
-               complete(&f->completion);
+       llist_for_each_entry_safe(fqdir, tmp, kill_list, free_list) {
+               f = fqdir->f;
+               if (refcount_dec_and_test(&f->refcnt))
+                       complete(&f->completion);
 
-       kfree(fqdir);
+               kfree(fqdir);
+       }
 }
 

net/ipv4/inet_fragment.c static void fqdir_work_fn(struct work_struct *work) · line 184

Quoted 10 lines
net/ipv4/inet_fragment.c +10 -0
+static DECLARE_WORK(fqdir_free_work, fqdir_free_fn);
+
+static void fqdir_work_fn(struct work_struct *work)
+{
+       struct fqdir *fqdir = container_of(work, struct fqdir, destroy_work);
+
+       rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
+
+       if (llist_add(&fqdir->free_list, &free_list))
+               queue_work(fqdir_wq, &fqdir_free_work);

I think you misunderstood me.

Since this fqdir_free_work will have at most one instance, you can use system_wq here, there is no risk of abuse.

My suggestion was to not use system_wq for fqdir_exit(), to better control the number
of threads in rhashtable cleanups.

void fqdir_exit(struct fqdir *fqdir)

{
        INIT_WORK(&fqdir->destroy_work, fqdir_work_fn);
(quoted diff) +1 -1
-       queue_work(system_wq, &fqdir->destroy_work);
+      queue_work(fqdir_wq, &fqdir->destroy_work);
Quoted 1 line

}

Oh, got it. I definitely misunderstood. My fault, sorry.

Quoted 13 lines

net/ipv4/inet_fragment.c static void fqdir_work_fn(struct work_struct *work) · line 189

Quoted 8 lines
net/ipv4/inet_fragment.c +5 -0
+}
+
+static int __init fqdir_wq_init(void)
+{
+       fqdir_wq = create_singlethread_workqueue("fqdir");

And here, I suggest to use a non ordered work queue, allowing one thread per cpu, to allow concurrent rhashtable cleanups

Quoted 3 lines

Also "fqdir" name is rather vague, this is an implementation detail ?

fqdir_wq =create_workqueue("inet_frag_wq");

So, what you are suggesting is to use a dedicated non-ordered work queue (fqdir_wq) for rhashtable cleanup and do the remaining works with system_wq in the batched manner, right? IOW, doing below change on top of this patch.

net/ipv4/inet_fragment.c +0 -0
@@ -145,7 +145,7 @@ static void inet_frags_free_cb(void *ptr, void *arg)
                inet_frag_destroy(fq);
 }
 
Quoted 6 lines

net/ipv4/inet_fragment.c static void inet_frags_free_cb(void *ptr, void *arg) · line 149

Quoted 3 lines
net/ipv4/inet_fragment.c +1 -1
-static struct workqueue_struct *fqdir_wq;
+static struct workqueue_struct *inet_frag_wq;
 static LLIST_HEAD(free_list);

Nit : Please prefix this free_list , like fqdir_free_list to avoid namespace pollution.

Sure!

Quoted 32 lines
Quoted 32 lines
net/ipv4/inet_fragment.c +5 -5
 static void fqdir_free_fn(struct work_struct *work)
@@ -181,14 +181,14 @@ static void fqdir_work_fn(struct work_struct *work)
        rhashtable_free_and_destroy(&fqdir->rhashtable, inet_frags_free_cb, NULL);
 
        if (llist_add(&fqdir->free_list, &free_list))
-               queue_work(fqdir_wq, &fqdir_free_work);
+               queue_work(system_wq, &fqdir_free_work);
 }
 
 static int __init fqdir_wq_init(void)
 {
-       fqdir_wq = create_singlethread_workqueue("fqdir");
-       if (!fqdir_wq)
-               panic("Could not create fqdir workq");
+       inet_frag_wq = create_workqueue("inet_frag_wq");
+       if (!inet_frag_wq)
+               panic("Could not create inet frag workq");
        return 0;
 }
 
@@ -218,7 +218,7 @@ EXPORT_SYMBOL(fqdir_init);
 void fqdir_exit(struct fqdir *fqdir)
 {
        INIT_WORK(&fqdir->destroy_work, fqdir_work_fn);
-       queue_work(system_wq, &fqdir->destroy_work);
+       queue_work(inet_frag_wq, &fqdir->destroy_work);
 }
 EXPORT_SYMBOL(fqdir_exit);
 
Quoted 4 lines
Quoted 2 lines

If I'm still misunderstanding, please let me know.

I think that with the above changes, we should be good ;)

Thank you for your quick and nice reply. I will send the next version soon!

Thanks,
SeongJae Park