Skip to contentSkip to conversation

mm: hugetlb: fix stalling when a large number of hugepages are freed

PATCH In review Masayoshi Mizuma 3 participants 3 messages Apr 7, 2014 1 message not in the archive

Subsystems (2)
Memory management HugeTLB
lore
Share & tools
lore

Masayoshi Mizuma

author
#0 lore raw

PATCH v2 1/1 mm: hugetlb: fix stalling when a large number of hugepages are freed

When I decrease the value of nr_hugepage in procfs a lot, a long stalling happens. It is because there is no chance of context switch during this process.

On the other hand, when I allocate a large number of hugepages, there is some chance of context switch. Hence the long stalling doesn't happen during this process. So it's necessary to add the context switch
in the freeing process as same as allocating process to avoid the long stalling.

When I freed 12 TB hugapages with kernel-2.6.32-358.el6, the freeing process occupied a CPU over 150 seconds and following softlockup message appeared twice or more.

--

$ echo 6000000 > /proc/sys/vm/nr_hugepages
$ cat /proc/sys/vm/nr_hugepages
6000000
$ grep ^Huge /proc/meminfo
HugePages_Total:   6000000
HugePages_Free:    6000000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
$ echo 0 > /proc/sys/vm/nr_hugepages
BUG: soft lockup - CPU#16 stuck for 67s! [sh:12883] ...
Pid: 12883, comm: sh Not tainted 2.6.32-358.el6.x86_64 #1
Call Trace:
 [<ffffffff8115a438>] ? free_pool_huge_page+0xb8/0xd0
 [<ffffffff8115a578>] ? set_max_huge_pages+0x128/0x190
 [<ffffffff8115c663>] ? hugetlb_sysctl_handler_common+0x113/0x140
 [<ffffffff8115c6de>] ? hugetlb_sysctl_handler+0x1e/0x20
 [<ffffffff811f3097>] ? proc_sys_call_handler+0x97/0xd0
 [<ffffffff811f30e4>] ? proc_sys_write+0x14/0x20
 [<ffffffff81180f98>] ? vfs_write+0xb8/0x1a0
 [<ffffffff81181891>] ? sys_write+0x51/0x90
 [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

--
I have not confirmed this problem with upstream kernels because I am not able to prepare the machine equipped with 12TB memory now. However I confirmed that the amount of decreasing hugepages was directly proportional to the amount of required time.

I measured required times on a smaller machine. It showed 130-145 hugepages decreased in a millisecond.

Amount of decreasing Required time Decreasing rate
hugepages (msec) (pages/msec)
------------------------------------------------------------
10,000 pages == 20GB 70 - 74 135-142
30,000 pages == 60GB 208 - 229 131-144

It means decrement of 6TB hugepages will trigger a long stalling (about 20sec), in this decreasing rate.

* Changes in v2
- Adding cond_resched_lock() in return_unused_surplus_pages() Because when freeing a number of surplus pages, same problems happen.

Signed-off-by Masayoshi Mizuma <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Aneesh Kumar <[email protected]>
Cc KOSAKI Motohiro <[email protected]>
Cc Naoya Horiguchi <[email protected]>
---
 mm/hugetlb.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
mm/hugetlb.c +2 -0
index 7d57af2..761ef5b 100644
@@ -1160,6 +1160,7 @@ static void return_unused_surplus_pages(struct hstate *h,
 	while (nr_pages--) {
 		if (!free_pool_huge_page(h, &node_states[N_MEMORY], 1))
 			break;
+		cond_resched_lock(&hugetlb_lock);
 	}
 }
 

@@ -1535,6 +1536,7 @@ static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count,
 	while (min_count < persistent_huge_pages(h)) {
 		if (!free_pool_huge_page(h, nodes_allowed, 0))
 			break;
+		cond_resched_lock(&hugetlb_lock);
 	}
 	while (count < persistent_huge_pages(h)) {
 		if (!adjust_pool_surplus(h, nodes_allowed, 1))
Signature · 7 lines
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Reviewed-by on v2 1/1
#1 lore raw
Quoted 59 lines · Masayoshi Mizuma

When I decrease the value of nr_hugepage in procfs a lot, a long stalling happens. It is because there is no chance of context switch during this process.

On the other hand, when I allocate a large number of hugepages, there is some chance of context switch. Hence the long stalling doesn't happen during this process. So it's necessary to add the context switch
in the freeing process as same as allocating process to avoid the long stalling.

When I freed 12 TB hugapages with kernel-2.6.32-358.el6, the freeing process occupied a CPU over 150 seconds and following softlockup message appeared twice or more.

--

$ echo 6000000 > /proc/sys/vm/nr_hugepages
$ cat /proc/sys/vm/nr_hugepages
6000000
$ grep ^Huge /proc/meminfo
HugePages_Total:   6000000
HugePages_Free:    6000000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
$ echo 0 > /proc/sys/vm/nr_hugepages
BUG: soft lockup - CPU#16 stuck for 67s! [sh:12883] ...
Pid: 12883, comm: sh Not tainted 2.6.32-358.el6.x86_64 #1
Call Trace:
 [<ffffffff8115a438>] ? free_pool_huge_page+0xb8/0xd0
 [<ffffffff8115a578>] ? set_max_huge_pages+0x128/0x190
 [<ffffffff8115c663>] ? hugetlb_sysctl_handler_common+0x113/0x140
 [<ffffffff8115c6de>] ? hugetlb_sysctl_handler+0x1e/0x20
 [<ffffffff811f3097>] ? proc_sys_call_handler+0x97/0xd0
 [<ffffffff811f30e4>] ? proc_sys_write+0x14/0x20
 [<ffffffff81180f98>] ? vfs_write+0xb8/0x1a0
 [<ffffffff81181891>] ? sys_write+0x51/0x90
 [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

--
I have not confirmed this problem with upstream kernels because I am not able to prepare the machine equipped with 12TB memory now. However I confirmed that the amount of decreasing hugepages was directly proportional to the amount of required time.

I measured required times on a smaller machine. It showed 130-145 hugepages decreased in a millisecond.

Amount of decreasing Required time Decreasing rate
hugepages (msec) (pages/msec)
------------------------------------------------------------
10,000 pages == 20GB 70 - 74 135-142
30,000 pages == 60GB 208 - 229 131-144

It means decrement of 6TB hugepages will trigger a long stalling (about 20sec), in this decreasing rate.

* Changes in v2
- Adding cond_resched_lock() in return_unused_surplus_pages() Because when freeing a number of surplus pages, same problems happen.

Quoted 8 lines
Signed-off-by Masayoshi Mizuma <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Aneesh Kumar <[email protected]>
Cc KOSAKI Motohiro <[email protected]>
Cc Naoya Horiguchi <[email protected]>
Reviewed-by Naoya Horiguchi <[email protected]>

Thanks,
Naoya Horiguchi

Quoted 27 lines
---
 mm/hugetlb.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
mm/hugetlb.c +2 -0
index 7d57af2..761ef5b 100644
@@ -1160,6 +1160,7 @@ static void return_unused_surplus_pages(struct hstate *h,
 	while (nr_pages--) {
 		if (!free_pool_huge_page(h, &node_states[N_MEMORY], 1))
 			break;
+		cond_resched_lock(&hugetlb_lock);
 	}
 }
 

@@ -1535,6 +1536,7 @@ static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count,
 	while (min_count < persistent_huge_pages(h)) {
 		if (!free_pool_huge_page(h, nodes_allowed, 0))
 			break;
+		cond_resched_lock(&hugetlb_lock);
 	}
 	while (count < persistent_huge_pages(h)) {
 		if (!adjust_pool_surplus(h, nodes_allowed, 1))
Signature · 1 line
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

A message that is not in the archive (sent off-list or not yet received): 1396876864-vnrouoxp@[email protected]

Acked-by KOSAKI Motohiro
#3 lore raw
Quoted 77 lines

-----Original Message-----
From: Naoya Horiguchi [mailto:[email protected]]
Sent: Monday, April 07, 2014 9:21 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Motohiro Kosaki JP
Subject: Re: [PATCH v2 1/1] mm: hugetlb: fix stalling when a large number of hugepages are freed

Quoted 57 lines · Masayoshi Mizuma

When I decrease the value of nr_hugepage in procfs a lot, a long
stalling happens. It is because there is no chance of context switch during this process.

On the other hand, when I allocate a large number of hugepages, there
is some chance of context switch. Hence the long stalling doesn't
happen during this process. So it's necessary to add the context
switch in the freeing process as same as allocating process to avoid the long stalling.

When I freed 12 TB hugapages with kernel-2.6.32-358.el6, the freeing process occupied a CPU over 150 seconds and following softlockup message appeared twice or more.

--

$ echo 6000000 > /proc/sys/vm/nr_hugepages $ cat
/proc/sys/vm/nr_hugepages
6000000
$ grep ^Huge /proc/meminfo
HugePages_Total:   6000000
HugePages_Free:    6000000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
$ echo 0 > /proc/sys/vm/nr_hugepages
BUG: soft lockup - CPU#16 stuck for 67s! [sh:12883] ...

Pid: 12883, comm: sh Not tainted 2.6.32-358.el6.x86_64 #1 Call Trace:
[<ffffffff8115a438>] ? free_pool_huge_page+0xb8/0xd0
[<ffffffff8115a578>] ? set_max_huge_pages+0x128/0x190 [<ffffffff8115c663>] ? hugetlb_sysctl_handler_common+0x113/0x140
[<ffffffff8115c6de>] ? hugetlb_sysctl_handler+0x1e/0x20
[<ffffffff811f3097>] ? proc_sys_call_handler+0x97/0xd0 [<ffffffff811f30e4>] ? proc_sys_write+0x14/0x20 [<ffffffff81180f98>]

? vfs_write+0xb8/0x1a0  [<ffffffff81181891>] ? sys_write+0x51/0x90
[<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

--
I have not confirmed this problem with upstream kernels because I am not able to prepare the machine equipped with 12TB memory now. However I confirmed that the amount of decreasing hugepages was directly proportional to the amount of required time.

I measured required times on a smaller machine. It showed 130-145 hugepages decreased in a millisecond.

Amount of decreasing Required time Decreasing rate
hugepages (msec) (pages/msec)
------------------------------------------------------------
10,000 pages == 20GB 70 - 74 135-142
30,000 pages == 60GB 208 - 229 131-144

It means decrement of 6TB hugepages will trigger a long stalling (about 20sec), in this decreasing rate.

* Changes in v2
- Adding cond_resched_lock() in return_unused_surplus_pages() Because when freeing a number of surplus pages, same problems happen.

Quoted 8 lines
Signed-off-by Masayoshi Mizuma <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Aneesh Kumar <[email protected]>
Cc KOSAKI Motohiro <[email protected]>
Cc Naoya Horiguchi <[email protected]>
Quoted 4 lines
Reviewed-by Naoya Horiguchi <[email protected]>

Thanks,
Naoya Horiguchi

Acked-by KOSAKI Motohiro <[email protected]>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>