mm: hugetlb: fix softlockup when a large number of hugepages are freed.

PATCH Superseded Mizuma, Masayoshi 3 participants 5 messages Mar 31, 2014 – Apr 1, 2014 1 message not in the archive

Subsystems (2)

Memory management HugeTLB

lore

Share & tools

lore

Mizuma, Masayoshi

author

2014-03-31 10:44 UTC #0 lore raw

PATCH 1/1 mm: hugetlb: fix softlockup when a large number of hugepages are freed.

Hi,

When I decrease the value of nr_hugepage in procfs a lot, softlockup happens. It is because there is no chance of context switch during this process.

On the other hand, when I allocate a large number of hugepages, there is some chance of context switch. Hence softlockup doesn't happen during this process. So it's necessary to add the context switch in the freeing process as same as allocating process to avoid softlockup.

When I freed 12 TB hugapages with kernel-2.6.32-358.el6, the freeing process occupied a CPU over 150 seconds and following softlockup message appeared twice or more.

$ echo 6000000 > /proc/sys/vm/nr_hugepages
$ cat /proc/sys/vm/nr_hugepages
6000000
$ grep ^Huge /proc/meminfo
HugePages_Total:   6000000
HugePages_Free:    6000000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
$ echo 0 > /proc/sys/vm/nr_hugepages

BUG: soft lockup - CPU#16 stuck for 67s! [sh:12883] ...
Pid: 12883, comm: sh Not tainted 2.6.32-358.el6.x86_64 #1
Call Trace:
 [<ffffffff8115a438>] ? free_pool_huge_page+0xb8/0xd0
 [<ffffffff8115a578>] ? set_max_huge_pages+0x128/0x190
 [<ffffffff8115c663>] ? hugetlb_sysctl_handler_common+0x113/0x140
 [<ffffffff8115c6de>] ? hugetlb_sysctl_handler+0x1e/0x20
 [<ffffffff811f3097>] ? proc_sys_call_handler+0x97/0xd0
 [<ffffffff811f30e4>] ? proc_sys_write+0x14/0x20
 [<ffffffff81180f98>] ? vfs_write+0xb8/0x1a0
 [<ffffffff81181891>] ? sys_write+0x51/0x90
 [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

--
I have not confirmed this problem with upstream kernels because I am not able to prepare the machine equipped with 12TB memory now. However I confirmed that the amount of decreasing hugepages was directly proportional to the amount of required time.

I measured required times on a smaller machine. It showed 130-145 hugepages decreased in a millisecond.

Amount of decreasing Required time Decreasing rate
hugepages (msec) (pages/msec)
------------------------------------------------------------
10,000 pages == 20GB 70 - 74 135-142
30,000 pages == 60GB 208 - 229 131-144

It means decrement of 6TB hugepages will trigger softlockup with the default threshold 20sec, in this decreasing rate.

Signed-off-by Masayoshi Mizuma <[email protected]>

Cc Andrew Morton <[email protected]>

Cc Joonsoo Kim <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Wanpeng Li <[email protected]>

Cc Aneesh Kumar <[email protected]>

Cc KOSAKI Motohiro <[email protected]>

---
 mm/hugetlb.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

mm/hugetlb.c +1 -0

index 7d57af2..fe67f2c 100644
@@ -1535,6 +1535,7 @@ static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count,
 	while (min_count < persistent_huge_pages(h)) {
 		if (!free_pool_huge_page(h, nodes_allowed, 0))
 			break;
+		cond_resched_lock(&hugetlb_lock);
 	}
 	while (count < persistent_huge_pages(h)) {
 		if (!adjust_pool_surplus(h, nodes_allowed, 1))

Signature · 10 lines

1.7.1

Thanks,
Masayoshi Mizuma

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Naoya Horiguchi

2014-03-31 15:02 UTC #1 lore raw

Quoted 73 lines · Masayoshi

Hi,
When I decrease the value of nr_hugepage in procfs a lot, softlockup happens. It is because there is no chance of context switch during this process.
On the other hand, when I allocate a large number of hugepages, there is some chance of context switch. Hence softlockup doesn't happen during this process. So it's necessary to add the context switch in the freeing process as same as allocating process to avoid softlockup.
When I freed 12 TB hugapages with kernel-2.6.32-358.el6, the freeing process occupied a CPU over 150 seconds and following softlockup message appeared twice or more.
--
$ echo 6000000 > /proc/sys/vm/nr_hugepages
$ cat /proc/sys/vm/nr_hugepages
6000000
$ grep ^Huge /proc/meminfo
HugePages_Total:   6000000
HugePages_Free:    6000000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
$ echo 0 > /proc/sys/vm/nr_hugepages
BUG: soft lockup - CPU#16 stuck for 67s! [sh:12883] ...
Pid: 12883, comm: sh Not tainted 2.6.32-358.el6.x86_64 #1
Call Trace:
 [<ffffffff8115a438>] ? free_pool_huge_page+0xb8/0xd0
 [<ffffffff8115a578>] ? set_max_huge_pages+0x128/0x190
 [<ffffffff8115c663>] ? hugetlb_sysctl_handler_common+0x113/0x140
 [<ffffffff8115c6de>] ? hugetlb_sysctl_handler+0x1e/0x20
 [<ffffffff811f3097>] ? proc_sys_call_handler+0x97/0xd0
 [<ffffffff811f30e4>] ? proc_sys_write+0x14/0x20
 [<ffffffff81180f98>] ? vfs_write+0xb8/0x1a0
 [<ffffffff81181891>] ? sys_write+0x51/0x90
 [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
--
I have not confirmed this problem with upstream kernels because I am not able to prepare the machine equipped with 12TB memory now. However I confirmed that the amount of decreasing hugepages was directly proportional to the amount of required time.
I measured required times on a smaller machine. It showed 130-145 hugepages decreased in a millisecond.
Amount of decreasing Required time Decreasing rate
hugepages (msec) (pages/msec)
------------------------------------------------------------
10,000 pages == 20GB 70 - 74 135-142
30,000 pages == 60GB 208 - 229 131-144
It means decrement of 6TB hugepages will trigger softlockup with the default threshold 20sec, in this decreasing rate.
Signed-off-by Masayoshi Mizuma <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Aneesh Kumar <[email protected]>
Cc KOSAKI Motohiro <[email protected]>
---
 mm/hugetlb.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
mm/hugetlb.c +0 -0
index 7d57af2..fe67f2c 100644

mm/hugetlb.c static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count, · line 1541

Quoted 7 lines

mm/hugetlb.c +1 -0

@@ -1535,6 +1535,7 @@ static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count,
 	while (min_count < persistent_huge_pages(h)) {
 		if (!free_pool_huge_page(h, nodes_allowed, 0))
 			break;
+		cond_resched_lock(&hugetlb_lock);
 	}
 	while (count < persistent_huge_pages(h)) {
 		if (!adjust_pool_surplus(h, nodes_allowed, 1))

It seems that the same thing could happen when freeing a number of surplus pages, so how about adding cond_resched_lock() also in return_unused_surplus_pages()?

Thanks,
Naoya Horiguchi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andrew Morton

maintainer

2014-03-31 19:30 UTC #2 lore raw

In reply to Mizuma, Masayoshi

Quoted 15 lines · Masayoshi

Hi,
When I decrease the value of nr_hugepage in procfs a lot, softlockup happens. It is because there is no chance of context switch during this process.
On the other hand, when I allocate a large number of hugepages, there is some chance of context switch. Hence softlockup doesn't happen during this process. So it's necessary to add the context switch in the freeing process as same as allocating process to avoid softlockup.
...

mm/hugetlb.c static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count, · line 1541

Quoted 7 lines

mm/hugetlb.c +1 -0

@@ -1535,6 +1535,7 @@ static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count,
 	while (min_count < persistent_huge_pages(h)) {
 		if (!free_pool_huge_page(h, nodes_allowed, 0))
 			break;
+		cond_resched_lock(&hugetlb_lock);
 	}
 	while (count < persistent_huge_pages(h)) {
 		if (!adjust_pool_surplus(h, nodes_allowed, 1))

Are you sure we don't need a cond_resched_lock() in this second loop as well?

Let's bear in mind the objective here: it is to avoid long scheduling stalls, not to prevent softlockup-detector warnings. A piece of code which doesn't trip the lockup detector can still be a problem.

Masayoshi Mizuma

author

2014-04-01 07:04 UTC #3 lore raw

Quoted 14 lines · On Mon, 31 Mar 2014 12:30:28 -0700 Andrew Morton wrote:

Quoted 12 lines · Masayoshi
Hi,
When I decrease the value of nr_hugepage in procfs a lot, softlockup happens. It is because there is no chance of context switch during this process.
On the other hand, when I allocate a large number of hugepages, there is some chance of context switch. Hence softlockup doesn't happen during this process. So it's necessary to add the context switch in the freeing process as same as allocating process to avoid softlockup.
...

Quoted 13 lines

mm/hugetlb.c static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count, · line 1541
Quoted 10 lines
mm/hugetlb.c +1 -0
@@ -1535,6 +1535,7 @@ static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count,
  	while (min_count < persistent_huge_pages(h)) {
  		if (!free_pool_huge_page(h, nodes_allowed, 0))
  			break;
+		cond_resched_lock(&hugetlb_lock);
  	}
  	while (count < persistent_huge_pages(h)) {
  		if (!adjust_pool_surplus(h, nodes_allowed, 1))
Are you sure we don't need a cond_resched_lock() in this second loop as well?

We don't need a cond_resched_lock() in the second loop. Because, the long scheduling stalls is occured by freeing hugepage at free_pool_huge_page() and the freeing is heavy. adjust_pool_surplus(), which is called at the second loop, is not heavy, I believe.

Quoted 4 lines

Let's bear in mind the objective here: it is to avoid long scheduling stalls, not to prevent softlockup-detector warnings. A piece of code which doesn't trip the lockup detector can still be a problem.

I see, thank you!

Thanks,
Masayoshi Mizuma

Quoted 2 lines

A message that is not in the archive (sent off-list or not yet received): 1396278140-k1hmxq77@[email protected]

Masayoshi Mizuma

author

2014-04-01 06:55 UTC #5 lore raw

Hi,

Quoted 69 lines · On Mon, 31 Mar 2014 11:02:20 -0400 Naoya Horiguchi wrote:

Quoted 68 lines · Masayoshi
Hi,
When I decrease the value of nr_hugepage in procfs a lot, softlockup happens. It is because there is no chance of context switch during this process.
On the other hand, when I allocate a large number of hugepages, there is some chance of context switch. Hence softlockup doesn't happen during this process. So it's necessary to add the context switch in the freeing process as same as allocating process to avoid softlockup.
When I freed 12 TB hugapages with kernel-2.6.32-358.el6, the freeing process occupied a CPU over 150 seconds and following softlockup message appeared twice or more.
--
$ echo 6000000 > /proc/sys/vm/nr_hugepages
$ cat /proc/sys/vm/nr_hugepages
6000000
$ grep ^Huge /proc/meminfo
HugePages_Total:   6000000
HugePages_Free:    6000000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
$ echo 0 > /proc/sys/vm/nr_hugepages
BUG: soft lockup - CPU#16 stuck for 67s! [sh:12883] ...
Pid: 12883, comm: sh Not tainted 2.6.32-358.el6.x86_64 #1
Call Trace:
  [<ffffffff8115a438>] ? free_pool_huge_page+0xb8/0xd0
  [<ffffffff8115a578>] ? set_max_huge_pages+0x128/0x190
  [<ffffffff8115c663>] ? hugetlb_sysctl_handler_common+0x113/0x140
  [<ffffffff8115c6de>] ? hugetlb_sysctl_handler+0x1e/0x20
  [<ffffffff811f3097>] ? proc_sys_call_handler+0x97/0xd0
  [<ffffffff811f30e4>] ? proc_sys_write+0x14/0x20
  [<ffffffff81180f98>] ? vfs_write+0xb8/0x1a0
  [<ffffffff81181891>] ? sys_write+0x51/0x90
  [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
  [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
--
I have not confirmed this problem with upstream kernels because I am not able to prepare the machine equipped with 12TB memory now. However I confirmed that the amount of decreasing hugepages was directly proportional to the amount of required time.
I measured required times on a smaller machine. It showed 130-145 hugepages decreased in a millisecond.
Amount of decreasing Required time Decreasing rate
hugepages (msec) (pages/msec)
------------------------------------------------------------
10,000 pages == 20GB 70 - 74 135-142
30,000 pages == 60GB 208 - 229 131-144
It means decrement of 6TB hugepages will trigger softlockup with the default threshold 20sec, in this decreasing rate.
Signed-off-by Masayoshi Mizuma <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Aneesh Kumar <[email protected]>
Cc KOSAKI Motohiro <[email protected]>
---
  mm/hugetlb.c |    1 +
  1 files changed, 1 insertions(+), 0 deletions(-)

Quoted 15 lines

mm/hugetlb.c static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count, · line 1541
Quoted 12 lines
mm/hugetlb.c +1 -0
index 7d57af2..fe67f2c 100644
@@ -1535,6 +1535,7 @@ static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count,
  	while (min_count < persistent_huge_pages(h)) {
  		if (!free_pool_huge_page(h, nodes_allowed, 0))
  			break;
+		cond_resched_lock(&hugetlb_lock);
  	}
  	while (count < persistent_huge_pages(h)) {
  		if (!adjust_pool_surplus(h, nodes_allowed, 1))
It seems that the same thing could happen when freeing a number of surplus pages, so how about adding cond_resched_lock() also in return_unused_surplus_pages()?

Thank you for pointing that out!
I will also add cond_resched_lock() in the following loop at return_unused_surplus_pages().

static void return_unused_surplus_pages(struct hstate *h,
unsigned long unused_resv_pages)
{
<cut>

        while (nr_pages--) {
                if (!free_pool_huge_page(h, &node_states[N_MEMORY], 1))
                        break;
        }
}

Thanks,
Masayoshi Mizuma

Quoted 4 lines

Thanks,
Naoya Horiguchi