Skip to contentSkip to conversation

mm: move pcp and lru-pcp drainging into vmstat_wq

RFC PATCH In review Michal Hocko 4 participants 9 messages Feb 7, 2017 – Feb 8, 2017

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 mm: move pcp and lru-pcp drainging into vmstat_wq

From: Michal Hocko <[email protected]>

We currently have 2 specific WQ_RECLAIM workqueues. One for updating pcp stats vmstat_wq and one dedicated to drain per cpu lru caches. This seems more than necessary because both can run on a single WQ. Both do not block on locks requiring a memory allocation nor perform any allocations themselves. We will save one rescuer thread this way.

On the other hand drain_all_pages queues work on the system wq which doesn't have rescuer and so this depend on memory allocation (when all workers are stuck allocating and new ones cannot be created). This is not critical as there should be somebody invoking the OOM killer (e.g. the forking worker) and get the situation unstuck and eventually performs the draining. Quite annoying though. This worker should be using WQ_RECLAIM as well. We can reuse the same one as for lru draining and vmstat.

---

Hi,
Tetsuo has noted that drain_all_pages doesn't use WQ_RECLAIM [1] and asked whether we can move the worker to the vmstat_wq which is WQ_RECLAIM. I think the deadlock he has described shouldn't happen but it would be really better to have the rescuer. I also think that we do not really need 2 or more workqueues and also pull lru draining in.

What do you think? Please note I haven't tested it yet.

[1] http://lkml.kernel.org/r/[email protected]

 mm/internal.h   |  6 ++++++
 mm/page_alloc.c |  2 +-
 mm/swap.c       | 20 +-------------------
 mm/vmstat.c     | 11 ++++++-----
 4 files changed, 14 insertions(+), 25 deletions(-)
mm/internal.h +6 -0
index ccfc2a2969f4..9ecafefe33ba 100644
@@ -498,4 +498,10 @@ extern const struct trace_print_flags pageflag_names[];
 extern const struct trace_print_flags vmaflag_names[];
 extern const struct trace_print_flags gfpflag_names[];
 
+/*
+ * only for MM internal work items which do not depend on
+ * any allocations or locks which might depend on allocations
+ */
+extern struct workqueue_struct *vmstat_wq;
+
 #endif	/* __MM_INTERNAL_H */
mm/page_alloc.c +1 -1
index 6c48053bcd81..0c0a7c38cd91 100644
@@ -2419,7 +2419,7 @@ void drain_all_pages(struct zone *zone)
 	for_each_cpu(cpu, &cpus_with_pcps) {
 		struct work_struct *work = per_cpu_ptr(&pcpu_drain, cpu);
 		INIT_WORK(work, drain_local_pages_wq);
-		schedule_work_on(cpu, work);
+		queue_work_on(cpu, vmstat_wq, work);
 	}
 	for_each_cpu(cpu, &cpus_with_pcps)
 		flush_work(per_cpu_ptr(&pcpu_drain, cpu));
mm/swap.c +1 -19
index c4910f14f957..23f09d6dd212 100644
@@ -670,24 +670,6 @@ static void lru_add_drain_per_cpu(struct work_struct *dummy)
 
 static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work);
 
-/*
- * lru_add_drain_wq is used to do lru_add_drain_all() from a WQ_MEM_RECLAIM
- * workqueue, aiding in getting memory freed.
- */
-static struct workqueue_struct *lru_add_drain_wq;
-
-static int __init lru_init(void)
-{
-	lru_add_drain_wq = alloc_workqueue("lru-add-drain", WQ_MEM_RECLAIM, 0);
-
-	if (WARN(!lru_add_drain_wq,
-		"Failed to create workqueue lru_add_drain_wq"))
-		return -ENOMEM;
-
-	return 0;
-}
-early_initcall(lru_init);
-
 void lru_add_drain_all(void)
 {
 	static DEFINE_MUTEX(lock);
@@ -707,7 +689,7 @@ void lru_add_drain_all(void)
 		    pagevec_count(&per_cpu(lru_deactivate_pvecs, cpu)) ||
 		    need_activate_page_drain(cpu)) {
 			INIT_WORK(work, lru_add_drain_per_cpu);
-			queue_work_on(cpu, lru_add_drain_wq, work);
+			queue_work_on(cpu, vmstat_wq, work);
 			cpumask_set_cpu(cpu, &has_work);
 		}
 	}
mm/vmstat.c +6 -5
index 69f9aff39a2e..fc9c2d9f014b 100644
@@ -1548,8 +1548,8 @@ static const struct file_operations proc_vmstat_file_operations = {
 };
 #endif /* CONFIG_PROC_FS */
 
+struct workqueue_struct *vmstat_wq;
 #ifdef CONFIG_SMP
-static struct workqueue_struct *vmstat_wq;
 static DEFINE_PER_CPU(struct delayed_work, vmstat_work);
 int sysctl_stat_interval __read_mostly = HZ;
 
@@ -1715,7 +1715,6 @@ static void __init start_shepherd_timer(void)
 		INIT_DEFERRABLE_WORK(per_cpu_ptr(&vmstat_work, cpu),
 			vmstat_update);
 
-	vmstat_wq = alloc_workqueue("vmstat", WQ_FREEZABLE|WQ_MEM_RECLAIM, 0);
 	schedule_delayed_work(&shepherd,
 		round_jiffies_relative(sysctl_stat_interval));
 }
@@ -1763,9 +1762,11 @@ static int vmstat_cpu_dead(unsigned int cpu)
 
 static int __init setup_vmstat(void)
 {
-#ifdef CONFIG_SMP
-	int ret;
+	int ret = 0;
+
+	vmstat_wq = alloc_workqueue("vmstat", WQ_FREEZABLE|WQ_MEM_RECLAIM, 0);
 
+#ifdef CONFIG_SMP
 	ret = cpuhp_setup_state_nocalls(CPUHP_MM_VMSTAT_DEAD, "mm/vmstat:dead",
 					NULL, vmstat_cpu_dead);
 	if (ret < 0)
@@ -1789,7 +1790,7 @@ static int __init setup_vmstat(void)
 	proc_create("vmstat", S_IRUGO, NULL, &proc_vmstat_file_operations);
 	proc_create("zoneinfo", S_IRUGO, NULL, &proc_zoneinfo_file_operations);
 #endif
-	return 0;
+	return ret;
 }
 module_init(setup_vmstat)
 
Signature · 7 lines
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw
Quoted 21 lines · Michal Hocko

From: Michal Hocko <[email protected]>

We currently have 2 specific WQ_RECLAIM workqueues. One for updating pcp stats vmstat_wq and one dedicated to drain per cpu lru caches. This seems more than necessary because both can run on a single WQ. Both do not block on locks requiring a memory allocation nor perform any allocations themselves. We will save one rescuer thread this way.

On the other hand drain_all_pages queues work on the system wq which doesn't have rescuer and so this depend on memory allocation (when all workers are stuck allocating and new ones cannot be created). This is not critical as there should be somebody invoking the OOM killer (e.g. the forking worker) and get the situation unstuck and eventually performs the draining. Quite annoying though. This worker should be using WQ_RECLAIM as well. We can reuse the same one as for lru draining and vmstat.

---

Quoted 8 lines

Hi,
Tetsuo has noted that drain_all_pages doesn't use WQ_RECLAIM [1] and asked whether we can move the worker to the vmstat_wq which is WQ_RECLAIM. I think the deadlock he has described shouldn't happen but it would be really better to have the rescuer. I also think that we do not really need 2 or more workqueues and also pull lru draining in.

What do you think? Please note I haven't tested it yet.

Why not, I guess, of course I may be overlooking some subtlety. You could have CC'd Tejun and Christoph. Watch out for the init order though, maybe? Is there no caller of the lru/pcp drain before module_init(setup_vmstat) happens?

Also one nit below.

Quoted 97 lines

[1] http://lkml.kernel.org/r/[email protected]

 mm/internal.h   |  6 ++++++
 mm/page_alloc.c |  2 +-
 mm/swap.c       | 20 +-------------------
 mm/vmstat.c     | 11 ++++++-----
 4 files changed, 14 insertions(+), 25 deletions(-)
mm/internal.h +6 -0
index ccfc2a2969f4..9ecafefe33ba 100644
@@ -498,4 +498,10 @@ extern const struct trace_print_flags pageflag_names[];
 extern const struct trace_print_flags vmaflag_names[];
 extern const struct trace_print_flags gfpflag_names[];
 
+/*
+ * only for MM internal work items which do not depend on
+ * any allocations or locks which might depend on allocations
+ */
+extern struct workqueue_struct *vmstat_wq;
+
 #endif	/* __MM_INTERNAL_H */
mm/page_alloc.c +1 -1
index 6c48053bcd81..0c0a7c38cd91 100644
@@ -2419,7 +2419,7 @@ void drain_all_pages(struct zone *zone)
 	for_each_cpu(cpu, &cpus_with_pcps) {
 		struct work_struct *work = per_cpu_ptr(&pcpu_drain, cpu);
 		INIT_WORK(work, drain_local_pages_wq);
-		schedule_work_on(cpu, work);
+		queue_work_on(cpu, vmstat_wq, work);
 	}
 	for_each_cpu(cpu, &cpus_with_pcps)
 		flush_work(per_cpu_ptr(&pcpu_drain, cpu));
mm/swap.c +1 -19
index c4910f14f957..23f09d6dd212 100644
@@ -670,24 +670,6 @@ static void lru_add_drain_per_cpu(struct work_struct *dummy)
 
 static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work);
 
-/*
- * lru_add_drain_wq is used to do lru_add_drain_all() from a WQ_MEM_RECLAIM
- * workqueue, aiding in getting memory freed.
- */
-static struct workqueue_struct *lru_add_drain_wq;
-
-static int __init lru_init(void)
-{
-	lru_add_drain_wq = alloc_workqueue("lru-add-drain", WQ_MEM_RECLAIM, 0);
-
-	if (WARN(!lru_add_drain_wq,
-		"Failed to create workqueue lru_add_drain_wq"))
-		return -ENOMEM;
-
-	return 0;
-}
-early_initcall(lru_init);
-
 void lru_add_drain_all(void)
 {
 	static DEFINE_MUTEX(lock);
@@ -707,7 +689,7 @@ void lru_add_drain_all(void)
 		    pagevec_count(&per_cpu(lru_deactivate_pvecs, cpu)) ||
 		    need_activate_page_drain(cpu)) {
 			INIT_WORK(work, lru_add_drain_per_cpu);
-			queue_work_on(cpu, lru_add_drain_wq, work);
+			queue_work_on(cpu, vmstat_wq, work);
 			cpumask_set_cpu(cpu, &has_work);
 		}
 	}
mm/vmstat.c +1 -2
index 69f9aff39a2e..fc9c2d9f014b 100644
@@ -1548,8 +1548,8 @@ static const struct file_operations proc_vmstat_file_operations = {
 };
 #endif /* CONFIG_PROC_FS */
 
+struct workqueue_struct *vmstat_wq;
 #ifdef CONFIG_SMP
-static struct workqueue_struct *vmstat_wq;
 static DEFINE_PER_CPU(struct delayed_work, vmstat_work);
 int sysctl_stat_interval __read_mostly = HZ;
 
@@ -1715,7 +1715,6 @@ static void __init start_shepherd_timer(void)
 		INIT_DEFERRABLE_WORK(per_cpu_ptr(&vmstat_work, cpu),
 			vmstat_update);
 
-	vmstat_wq = alloc_workqueue("vmstat", WQ_FREEZABLE|WQ_MEM_RECLAIM, 0);
 	schedule_delayed_work(&shepherd,
 		round_jiffies_relative(sysctl_stat_interval));
 }

mm/vmstat.c static int vmstat_cpu_dead(unsigned int cpu) · line 1767

Quoted 8 lines
mm/vmstat.c +3 -2
@@ -1763,9 +1762,11 @@ static int vmstat_cpu_dead(unsigned int cpu)
 
 static int __init setup_vmstat(void)
 {
-#ifdef CONFIG_SMP
-	int ret;
+	int ret = 0;
+
+	vmstat_wq = alloc_workqueue("vmstat", WQ_FREEZABLE|WQ_MEM_RECLAIM, 0);

Did you want to set ret to -ENOMEM if the alloc fails, or something? Otherwise I don't see why the changes.

Quoted 7 lines
mm/vmstat.c +1 -0
+#ifdef CONFIG_SMP
 	ret = cpuhp_setup_state_nocalls(CPUHP_MM_VMSTAT_DEAD, "mm/vmstat:dead",
 					NULL, vmstat_cpu_dead);
 	if (ret < 0)

mm/vmstat.c static int __init setup_vmstat(void) · line 1796

Quoted 8 lines
mm/vmstat.c +1 -1
@@ -1789,7 +1790,7 @@ static int __init setup_vmstat(void)
 	proc_create("vmstat", S_IRUGO, NULL, &proc_vmstat_file_operations);
 	proc_create("zoneinfo", S_IRUGO, NULL, &proc_zoneinfo_file_operations);
 #endif
-	return 0;
+	return ret;
 }
 module_init(setup_vmstat)
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
Quoted 29 lines · Vlastimil Babka
Quoted 28 lines · Michal Hocko

From: Michal Hocko <[email protected]>

We currently have 2 specific WQ_RECLAIM workqueues. One for updating pcp stats vmstat_wq and one dedicated to drain per cpu lru caches. This seems more than necessary because both can run on a single WQ. Both do not block on locks requiring a memory allocation nor perform any allocations themselves. We will save one rescuer thread this way.

On the other hand drain_all_pages queues work on the system wq which doesn't have rescuer and so this depend on memory allocation (when all workers are stuck allocating and new ones cannot be created). This is not critical as there should be somebody invoking the OOM killer (e.g. the forking worker) and get the situation unstuck and eventually performs the draining. Quite annoying though. This worker should be using WQ_RECLAIM as well. We can reuse the same one as for lru draining and vmstat.

---

Hi,
Tetsuo has noted that drain_all_pages doesn't use WQ_RECLAIM [1] and asked whether we can move the worker to the vmstat_wq which is WQ_RECLAIM. I think the deadlock he has described shouldn't happen but it would be really better to have the rescuer. I also think that we do not really need 2 or more workqueues and also pull lru draining in.

Quoted 4 lines
Quoted 1 line

What do you think? Please note I haven't tested it yet.

Why not, I guess, of course I may be overlooking some subtlety. You could have CC'd Tejun and Christoph.

will do on the next submission

Quoted 2 lines

Watch out for the init order though, maybe? Is there no caller of the lru/pcp drain before module_init(setup_vmstat) happens?

Hard to tell. I expect that there shouldn't be but I will add

mm/page_alloc.c +7 -0
index 0c0a7c38cd91..73018f07bcc9 100644
@@ -2370,6 +2370,13 @@ void drain_all_pages(struct zone *zone)
 	 */
 	static cpumask_t cpus_with_pcps;
 
+	/*
+	 * Make sure nobody triggers this path before vmstat_wq is fully
+	 * initialized.
+	 */
+	if (WARN_ON(!vmstat_wq))
+		return;
+
 	/* Workqueues cannot recurse */
 	if (current->flags & PF_WQ_WORKER)
 		return;
mm/swap.c +7 -0
index 23f09d6dd212..39c240fc9d48 100644
@@ -676,6 +676,13 @@ void lru_add_drain_all(void)
 	static struct cpumask has_work;
 	int cpu;
 
+	/*
+	 * Make sure nobody triggers this path before vmstat_wq is fully
+	 * initialized.
+	 */
+	if (WARN_ON(!vmstat_wq))
+		return;
+
 	mutex_lock(&lock);
 	get_online_cpus();
 	cpumask_clear(&has_work);

to be sure.
[...]

Quoted 12 lines

Review static int vmstat_cpu_dead(unsigned int cpu) · line 1767

Quoted 9 lines
(quoted diff) +3 -2
@@ -1763,9 +1762,11 @@ static int vmstat_cpu_dead(unsigned int cpu)
 
 static int __init setup_vmstat(void)
 {
-#ifdef CONFIG_SMP
-	int ret;
+	int ret = 0;
+
+	vmstat_wq = alloc_workqueue("vmstat", WQ_FREEZABLE|WQ_MEM_RECLAIM, 0);

Did you want to set ret to -ENOMEM if the alloc fails, or something? Otherwise I don't see why the changes.

no I just didn't want to get defined but not used warning for CONFIG_SMP=n

Quoted 16 lines
Quoted 16 lines
(quoted diff) +2 -1
+#ifdef CONFIG_SMP
 	ret = cpuhp_setup_state_nocalls(CPUHP_MM_VMSTAT_DEAD, "mm/vmstat:dead",
 					NULL, vmstat_cpu_dead);
 	if (ret < 0)
@@ -1789,7 +1790,7 @@ static int __init setup_vmstat(void)
 	proc_create("vmstat", S_IRUGO, NULL, &proc_vmstat_file_operations);
 	proc_create("zoneinfo", S_IRUGO, NULL, &proc_zoneinfo_file_operations);
 #endif
-	return 0;
+	return ret;
 }
 module_init(setup_vmstat)
 
Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#3 lore raw
In reply to Michal Hocko
Quoted 8 lines · Michal Hocko

From: Michal Hocko <[email protected]>

We currently have 2 specific WQ_RECLAIM workqueues. One for updating pcp stats vmstat_wq and one dedicated to drain per cpu lru caches. This seems more than necessary because both can run on a single WQ. Both do not block on locks requiring a memory allocation nor perform any allocations themselves. We will save one rescuer thread this way.

True.

Quoted 9 lines

On the other hand drain_all_pages queues work on the system wq which doesn't have rescuer and so this depend on memory allocation (when all workers are stuck allocating and new ones cannot be created). This is not critical as there should be somebody invoking the OOM killer (e.g. the forking worker) and get the situation unstuck and eventually performs the draining. Quite annoying though. This worker should be using WQ_RECLAIM as well. We can reuse the same one as for lru draining and vmstat.

That was still debatable which is why I didn't go that route. The drain itself is unlikely to fix anything with the possible exception of high-order pages. There are just too many reasons why direct reclaim can return 0 reclaimed pages making the drain is redundant but I couldn't decide what a better alternative would be and more importantly, how to measure it. The fact it allocates in that path is currently unfortunate but I couldn't convince myself it deserved a dedicated rescuer.

I don't object to it being actually moved. I have a slight concern that it could somehow starve a vmstat update while frequent drains happen during reclaim though which potentially compounds the problem. It could be offset by a variety of other factors but if it ever is an issue, it'll show up and the paths that really matter check the vmstats directly instead of waiting for an update.

Quoted 4 lines

---

Quoted 9 lines

Hi,
Tetsuo has noted that drain_all_pages doesn't use WQ_RECLAIM [1] and asked whether we can move the worker to the vmstat_wq which is WQ_RECLAIM. I think the deadlock he has described shouldn't happen but it would be really better to have the rescuer. I also think that we do not really need 2 or more workqueues and also pull lru draining in.

What do you think? Please note I haven't tested it yet.

As an aside, the LRU drain could also avoid a get_online_cpus() which is surprisingly heavy handed for an operation that can happen quite frequently during compaction or migration. Maybe not enough to make a big deal of but it's relatively low hanging fruit.

The altering of the return value in setup_vmstat was mildly surprising as it increases the severity of registering the vmstat callback for memory hotplug so maybe split that out and appears unrelated.

It also feels like vmstat is now a misleading name for something that handles vmstat, lru drains and per-cpu drains but that's cosmetic.

Fundamentally I have nothing against the patch.

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#4 lore raw
Quoted 22 lines · Mel Gorman
Quoted 8 lines · Michal Hocko

From: Michal Hocko <[email protected]>

We currently have 2 specific WQ_RECLAIM workqueues. One for updating pcp stats vmstat_wq and one dedicated to drain per cpu lru caches. This seems more than necessary because both can run on a single WQ. Both do not block on locks requiring a memory allocation nor perform any allocations themselves. We will save one rescuer thread this way.

True.

Quoted 9 lines

On the other hand drain_all_pages queues work on the system wq which doesn't have rescuer and so this depend on memory allocation (when all workers are stuck allocating and new ones cannot be created). This is not critical as there should be somebody invoking the OOM killer (e.g. the forking worker) and get the situation unstuck and eventually performs the draining. Quite annoying though. This worker should be using WQ_RECLAIM as well. We can reuse the same one as for lru draining and vmstat.

Quoted 7 lines

That was still debatable which is why I didn't go that route. The drain itself is unlikely to fix anything with the possible exception of high-order pages. There are just too many reasons why direct reclaim can return 0 reclaimed pages making the drain is redundant but I couldn't decide what a better alternative would be and more importantly, how to measure it. The fact it allocates in that path is currently unfortunate but I couldn't convince myself it deserved a dedicated rescuer.

agreed

Quoted 6 lines

I don't object to it being actually moved. I have a slight concern that it could somehow starve a vmstat update while frequent drains happen during reclaim though which potentially compounds the problem. It could be offset by a variety of other factors but if it ever is an issue, it'll show up and the paths that really matter check the vmstats directly instead of waiting for an update.

vmstat updates can tolared delays, that's we we are using deferable scheduling in the first place so I am not really worried about that. Any user which needs a better precision should use *_snapshot API.

Quoted 11 lines
Quoted 11 lines

---

Hi,
Tetsuo has noted that drain_all_pages doesn't use WQ_RECLAIM [1] and asked whether we can move the worker to the vmstat_wq which is WQ_RECLAIM. I think the deadlock he has described shouldn't happen but it would be really better to have the rescuer. I also think that we do not really need 2 or more workqueues and also pull lru draining in.

Quoted 7 lines
Quoted 2 lines

What do you think? Please note I haven't tested it yet.

As an aside, the LRU drain could also avoid a get_online_cpus() which is surprisingly heavy handed for an operation that can happen quite frequently during compaction or migration. Maybe not enough to make a big deal of but it's relatively low hanging fruit.

Yeah, this is sitting on my todo list already, I just didn't give it a priority.

Quoted 3 lines

The altering of the return value in setup_vmstat was mildly surprising as it increases the severity of registering the vmstat callback for memory hotplug so maybe split that out and appears unrelated.

not sure I understand. What do you mean?

Quoted 2 lines

It also feels like vmstat is now a misleading name for something that handles vmstat, lru drains and per-cpu drains but that's cosmetic.

yeah a better name sounds like a good thing. mm_nonblock_wq?

Quoted 1 line

Fundamentally I have nothing against the patch.

Thanks for the review. I will sit on it and give it some testing to see how it behaves and post after I get back from vacation (after 20th)

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#5 lore raw
Quoted 11 lines · Michal Hocko
Quoted 6 lines

I don't object to it being actually moved. I have a slight concern that it could somehow starve a vmstat update while frequent drains happen during reclaim though which potentially compounds the problem. It could be offset by a variety of other factors but if it ever is an issue, it'll show up and the paths that really matter check the vmstats directly instead of waiting for an update.

vmstat updates can tolared delays, that's we we are using deferable scheduling in the first place so I am not really worried about that. Any user which needs a better precision should use *_snapshot API.

Agreed, we already had cases where deferred vmstat updates had problems and were resolved by using _snapshot. It's a slight concern only and I'd be surprised if the _snapshot usage didn't cover it.

Quoted 6 lines
Quoted 3 lines

The altering of the return value in setup_vmstat was mildly surprising as it increases the severity of registering the vmstat callback for memory hotplug so maybe split that out and appears unrelated.

not sure I understand. What do you mean?

This hunk

@@ -1763,9 +1762,11 @@ static int vmstat_cpu_dead(unsigned int cpu)

static int __init setup_vmstat(void)
{
-#ifdef CONFIG_SMP
- int ret;
+ int ret = 0;
+
+ vmstat_wq = alloc_workqueue("vmstat", WQ_FREEZABLE|WQ_MEM_RECLAIM, 0);

+#ifdef CONFIG_SMP

        ret = cpuhp_setup_state_nocalls(CPUHP_MM_VMSTAT_DEAD, "mm/vmstat:dead",
                                        NULL, vmstat_cpu_dead);
        if (ret < 0)
@@ -1789,7 +1790,7 @@ static int __init setup_vmstat(void)
        proc_create("vmstat", S_IRUGO, NULL, &proc_vmstat_file_operations);
        proc_create("zoneinfo", S_IRUGO, NULL, &proc_zoneinfo_file_operations);

#endif
- return 0;
+ return ret;

A failed register of vmstat_cpu_dead is returning the failure code in an init function now. Chances are it'll never hit but it didn't seem related to the patches general intent.

Quoted 5 lines
Quoted 2 lines

It also feels like vmstat is now a misleading name for something that handles vmstat, lru drains and per-cpu drains but that's cosmetic.

yeah a better name sounds like a good thing. mm_nonblock_wq?

it's not always non-blocking. Maybe mm_percpu_wq to describev a workqueue that handles a variety of MM-related per-cpu updates?

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#6 lore raw
Quoted 9 lines · Mel Gorman
Quoted 5 lines
Quoted 2 lines

It also feels like vmstat is now a misleading name for something that handles vmstat, lru drains and per-cpu drains but that's cosmetic.

yeah a better name sounds like a good thing. mm_nonblock_wq?

it's not always non-blocking. Maybe mm_percpu_wq to describev a workqueue that handles a variety of MM-related per-cpu updates?

Why not make it global like ones created by workqueue_init_early() ?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw
Quoted 4 lines · Tetsuo Handa
Quoted 3 lines · Mel Gorman
Quoted 3 lines
Quoted 2 lines

It also feels like vmstat is now a misleading name for something that handles vmstat, lru drains and per-cpu drains but that's cosmetic.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

yeah a better name sounds like a good thing. mm_nonblock_wq?

it's not always non-blocking. Maybe mm_percpu_wq to describev a workqueue that handles a variety of MM-related per-cpu updates?

Why not make it global like ones created by workqueue_init_early() ?

I can see alloc_workqueue_attrs in that path so we can hit the page allocator and if unlucky try to drain_all_pages. We might have more even before this. So I think we still need a check for the WQ being initialized already. I do not have a strong preference to when to allocate it. Moving the initialization out to workqueue_init_early wold mean that the WQ would have to be visible outside of the mm proper which is not ideal. We can live with that, though, so I will move it there if this is a prevalent opinion.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#8 lore raw
In reply to Mel Gorman
Quoted 48 lines · Mel Gorman
Quoted 11 lines · Michal Hocko
Quoted 6 lines

I don't object to it being actually moved. I have a slight concern that it could somehow starve a vmstat update while frequent drains happen during reclaim though which potentially compounds the problem. It could be offset by a variety of other factors but if it ever is an issue, it'll show up and the paths that really matter check the vmstats directly instead of waiting for an update.

vmstat updates can tolared delays, that's we we are using deferable scheduling in the first place so I am not really worried about that. Any user which needs a better precision should use *_snapshot API.

Agreed, we already had cases where deferred vmstat updates had problems and were resolved by using _snapshot. It's a slight concern only and I'd be surprised if the _snapshot usage didn't cover it.

Quoted 6 lines
Quoted 3 lines

The altering of the return value in setup_vmstat was mildly surprising as it increases the severity of registering the vmstat callback for memory hotplug so maybe split that out and appears unrelated.

not sure I understand. What do you mean?

This hunk

mm/vmstat.c +5 -3
@@ -1763,9 +1762,11 @@ static int vmstat_cpu_dead(unsigned int cpu)
 
 static int __init setup_vmstat(void)
 {
-#ifdef CONFIG_SMP
-       int ret;
+       int ret = 0;
+
+       vmstat_wq = alloc_workqueue("vmstat", WQ_FREEZABLE|WQ_MEM_RECLAIM, 0);
 
+#ifdef CONFIG_SMP
        ret = cpuhp_setup_state_nocalls(CPUHP_MM_VMSTAT_DEAD, "mm/vmstat:dead",
                                        NULL, vmstat_cpu_dead);
        if (ret < 0)
@@ -1789,7 +1790,7 @@ static int __init setup_vmstat(void)
        proc_create("vmstat", S_IRUGO, NULL, &proc_vmstat_file_operations);
        proc_create("zoneinfo", S_IRUGO, NULL, &proc_zoneinfo_file_operations);
 #endif
-       return 0;
+       return ret;
 
 
Quoted 3 lines

A failed register of vmstat_cpu_dead is returning the failure code in an init function now. Chances are it'll never hit but it didn't seem related to the patches general intent.

Ohh, I see now. I will keep the original behavior.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

It also feels like vmstat is now a misleading name for something that handles vmstat, lru drains and per-cpu drains but that's cosmetic.

yeah a better name sounds like a good thing. mm_nonblock_wq?

it's not always non-blocking. Maybe mm_percpu_wq to describev a workqueue that handles a variety of MM-related per-cpu updates?

Why not, I do not have a strong preference. The WQ is already documented for its requirements on workers so the name doesn't really have to be explicit about blocking on allocations.

Thanks!

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>