mm: minute-long livelocks in memory reclaim
Subsystems (2)
Folks,
I've been testing parallel create workloads over the weekend, and I've seen this a couple of times now under 8 thread parallel creates with XFS. I'm running on an 8p VM with 4GB RAM and a fast disk subsystem. Basically I am seeing the create rate drop to zero with all 8 CPUs stuck spinning for up to 2 minutes. 'echo t > /proc/sysrq-trigger' while this is occurring gives the following trace for all the fs-mark processes:
[49506.624018] fs_mark R running task 0 8376 7917 0x00000008 [49506.624018] 0000000000000000 ffffffff81b94590 00000000000008fc 0000000000000002 [49506.624018] 0000000000000000 0000000000000286 0000000000000297 ffffffffffffff10 [49506.624018] ffffffff810b3d02 0000000000000010 0000000000000202 ffff88011df777a8 [49506.624018] Call Trace: [49506.624018] [<ffffffff810b3d02>] ? smp_call_function_many+0x1a2/0x210 [49506.624018] [<ffffffff810b3ce5>] ? smp_call_function_many+0x185/0x210 [49506.624018] [<ffffffff81109170>] ? drain_local_pages+0x0/0x20 [49506.624018] [<ffffffff810b3d92>] ? smp_call_function+0x22/0x30 [49506.624018] [<ffffffff810849a4>] ? on_each_cpu+0x24/0x50 [49506.624018] [<ffffffff81107bec>] ? drain_all_pages+0x1c/0x20 [49506.624018] [<ffffffff8110825a>] ? __alloc_pages_nodemask+0x57a/0x730 [49506.624018] [<ffffffff8113c6d2>] ? kmem_getpages+0x62/0x160 [49506.624018] [<ffffffff8113d2b2>] ? fallback_alloc+0x192/0x240 [49506.624018] [<ffffffff8113cce1>] ? cache_grow+0x2d1/0x300 [49506.624018] [<ffffffff8113d04a>] ? ____cache_alloc_node+0x9a/0x170 [49506.624018] [<ffffffff8113cf6c>] ? cache_alloc_refill+0x25c/0x2a0 [49506.624018] [<ffffffff8113ddb3>] ? __kmalloc+0x193/0x230 [49506.624018] [<ffffffff812f59af>] ? kmem_alloc+0x8f/0xe0 [49506.624018] [<ffffffff812f59af>] ? kmem_alloc+0x8f/0xe0 [49506.624018] [<ffffffff812f5a9e>] ? kmem_zalloc+0x1e/0x50 [49506.624018] [<ffffffff812e2f4d>] ? xfs_log_commit_cil+0x9d/0x440 [49506.624018] [<ffffffff812eeec6>] ? _xfs_trans_commit+0x1e6/0x2b0 [49506.624018] [<ffffffff812f2b6f>] ? xfs_create+0x51f/0x690 [49506.624018] [<ffffffff812ffdb7>] ? xfs_vn_mknod+0xa7/0x1c0 [49506.624018] [<ffffffff812fff00>] ? xfs_vn_create+0x10/0x20 [49506.624018] [<ffffffff811510b8>] ? vfs_create+0xb8/0xf0 [49506.624018] [<ffffffff81151d2c>] ? do_last+0x4dc/0x5d0 [49506.624018] [<ffffffff81153bd7>] ? do_filp_open+0x207/0x5e0 [49506.624018] [<ffffffff8105fc58>] ? pvclock_clocksource_read+0x58/0xd0 [49506.624018] [<ffffffff8115eaca>] ? alloc_fd+0x10a/0x150 [49506.624018] [<ffffffff81144005>] ? do_sys_open+0x65/0x130 [49506.624018] [<ffffffff81144110>] ? sys_open+0x20/0x30 [49506.624018] [<ffffffff81036072>] ? system_call_fastpath+0x16/0x1b
Eventually the problem goes away, and the system goes back to performing at the normal rate. Any ideas on how to avoid this problem? I'm using CONFIG_SLAB=y is that is relevant....
Cheers,
Dave.
Signature · 8 lines
Dave Chinner [email protected] -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 45 lines · Dave Chinner
Folks,
I've been testing parallel create workloads over the weekend, and I've seen this a couple of times now under 8 thread parallel creates with XFS. I'm running on an 8p VM with 4GB RAM and a fast disk subsystem. Basically I am seeing the create rate drop to zero with all 8 CPUs stuck spinning for up to 2 minutes. 'echo t >
/proc/sysrq-trigger' while this is occurring gives the following trace for all the fs-mark processes:[49506.624018] fs_mark R running task 0 8376 7917 0x00000008 [49506.624018] 0000000000000000 ffffffff81b94590 00000000000008fc 0000000000000002 [49506.624018] 0000000000000000 0000000000000286 0000000000000297 ffffffffffffff10 [49506.624018] ffffffff810b3d02 0000000000000010 0000000000000202 ffff88011df777a8 [49506.624018] Call Trace: [49506.624018] [<ffffffff810b3d02>] ? smp_call_function_many+0x1a2/0x210 [49506.624018] [<ffffffff810b3ce5>] ? smp_call_function_many+0x185/0x210 [49506.624018] [<ffffffff81109170>] ? drain_local_pages+0x0/0x20 [49506.624018] [<ffffffff810b3d92>] ? smp_call_function+0x22/0x30 [49506.624018] [<ffffffff810849a4>] ? on_each_cpu+0x24/0x50 [49506.624018] [<ffffffff81107bec>] ? drain_all_pages+0x1c/0x20 [49506.624018] [<ffffffff8110825a>] ? __alloc_pages_nodemask+0x57a/0x730 [49506.624018] [<ffffffff8113c6d2>] ? kmem_getpages+0x62/0x160 [49506.624018] [<ffffffff8113d2b2>] ? fallback_alloc+0x192/0x240 [49506.624018] [<ffffffff8113cce1>] ? cache_grow+0x2d1/0x300 [49506.624018] [<ffffffff8113d04a>] ? ____cache_alloc_node+0x9a/0x170 [49506.624018] [<ffffffff8113cf6c>] ? cache_alloc_refill+0x25c/0x2a0 [49506.624018] [<ffffffff8113ddb3>] ? __kmalloc+0x193/0x230 [49506.624018] [<ffffffff812f59af>] ? kmem_alloc+0x8f/0xe0 [49506.624018] [<ffffffff812f59af>] ? kmem_alloc+0x8f/0xe0 [49506.624018] [<ffffffff812f5a9e>] ? kmem_zalloc+0x1e/0x50 [49506.624018] [<ffffffff812e2f4d>] ? xfs_log_commit_cil+0x9d/0x440 [49506.624018] [<ffffffff812eeec6>] ? _xfs_trans_commit+0x1e6/0x2b0 [49506.624018] [<ffffffff812f2b6f>] ? xfs_create+0x51f/0x690 [49506.624018] [<ffffffff812ffdb7>] ? xfs_vn_mknod+0xa7/0x1c0 [49506.624018] [<ffffffff812fff00>] ? xfs_vn_create+0x10/0x20 [49506.624018] [<ffffffff811510b8>] ? vfs_create+0xb8/0xf0 [49506.624018] [<ffffffff81151d2c>] ? do_last+0x4dc/0x5d0 [49506.624018] [<ffffffff81153bd7>] ? do_filp_open+0x207/0x5e0 [49506.624018] [<ffffffff8105fc58>] ? pvclock_clocksource_read+0x58/0xd0 [49506.624018] [<ffffffff8115eaca>] ? alloc_fd+0x10a/0x150 [49506.624018] [<ffffffff81144005>] ? do_sys_open+0x65/0x130 [49506.624018] [<ffffffff81144110>] ? sys_open+0x20/0x30 [49506.624018] [<ffffffff81036072>] ? system_call_fastpath+0x16/0x1b
Quoted 3 lines
Eventually the problem goes away, and the system goes back to performing at the normal rate. Any ideas on how to avoid this problem? I'm using
CONFIG_SLAB=yis that is relevant....
zone->lock contention? Try rip the following two lines. The change might be a bit aggressive though :)
Thanks,
Fengguang
mm/page_alloc.c +0 -3
@@ -1864,9 +1864,6 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,cond_resched (); -if (order !=0 ) -drain_all_pages (); -if (likely (*did_some_progress)) page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist, high_zoneidx,
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 58 lines · Wu Fengguang
Quoted 43 lines
I've been testing parallel create workloads over the weekend, and I've seen this a couple of times now under 8 thread parallel creates with XFS. I'm running on an 8p VM with 4GB RAM and a fast disk subsystem. Basically I am seeing the create rate drop to zero with all 8 CPUs stuck spinning for up to 2 minutes. 'echo t >
/proc/sysrq-trigger' while this is occurring gives the following trace for all the fs-mark processes:[49506.624018] fs_mark R running task 0 8376 7917 0x00000008 [49506.624018] 0000000000000000 ffffffff81b94590 00000000000008fc 0000000000000002 [49506.624018] 0000000000000000 0000000000000286 0000000000000297 ffffffffffffff10 [49506.624018] ffffffff810b3d02 0000000000000010 0000000000000202 ffff88011df777a8 [49506.624018] Call Trace: [49506.624018] [<ffffffff810b3d02>] ? smp_call_function_many+0x1a2/0x210 [49506.624018] [<ffffffff810b3ce5>] ? smp_call_function_many+0x185/0x210 [49506.624018] [<ffffffff81109170>] ? drain_local_pages+0x0/0x20 [49506.624018] [<ffffffff810b3d92>] ? smp_call_function+0x22/0x30 [49506.624018] [<ffffffff810849a4>] ? on_each_cpu+0x24/0x50 [49506.624018] [<ffffffff81107bec>] ? drain_all_pages+0x1c/0x20 [49506.624018] [<ffffffff8110825a>] ? __alloc_pages_nodemask+0x57a/0x730 [49506.624018] [<ffffffff8113c6d2>] ? kmem_getpages+0x62/0x160 [49506.624018] [<ffffffff8113d2b2>] ? fallback_alloc+0x192/0x240 [49506.624018] [<ffffffff8113cce1>] ? cache_grow+0x2d1/0x300 [49506.624018] [<ffffffff8113d04a>] ? ____cache_alloc_node+0x9a/0x170 [49506.624018] [<ffffffff8113cf6c>] ? cache_alloc_refill+0x25c/0x2a0 [49506.624018] [<ffffffff8113ddb3>] ? __kmalloc+0x193/0x230 [49506.624018] [<ffffffff812f59af>] ? kmem_alloc+0x8f/0xe0 [49506.624018] [<ffffffff812f59af>] ? kmem_alloc+0x8f/0xe0 [49506.624018] [<ffffffff812f5a9e>] ? kmem_zalloc+0x1e/0x50 [49506.624018] [<ffffffff812e2f4d>] ? xfs_log_commit_cil+0x9d/0x440 [49506.624018] [<ffffffff812eeec6>] ? _xfs_trans_commit+0x1e6/0x2b0 [49506.624018] [<ffffffff812f2b6f>] ? xfs_create+0x51f/0x690 [49506.624018] [<ffffffff812ffdb7>] ? xfs_vn_mknod+0xa7/0x1c0 [49506.624018] [<ffffffff812fff00>] ? xfs_vn_create+0x10/0x20 [49506.624018] [<ffffffff811510b8>] ? vfs_create+0xb8/0xf0 [49506.624018] [<ffffffff81151d2c>] ? do_last+0x4dc/0x5d0 [49506.624018] [<ffffffff81153bd7>] ? do_filp_open+0x207/0x5e0 [49506.624018] [<ffffffff8105fc58>] ? pvclock_clocksource_read+0x58/0xd0 [49506.624018] [<ffffffff8115eaca>] ? alloc_fd+0x10a/0x150 [49506.624018] [<ffffffff81144005>] ? do_sys_open+0x65/0x130 [49506.624018] [<ffffffff81144110>] ? sys_open+0x20/0x30 [49506.624018] [<ffffffff81036072>] ? system_call_fastpath+0x16/0x1bQuoted 3 lines
Eventually the problem goes away, and the system goes back to performing at the normal rate. Any ideas on how to avoid this problem? I'm using
CONFIG_SLAB=yis that is relevant....
zone->lockcontention? Try rip the following two lines. The change might be a bit aggressive though :)Thanks,
Fengguangmm/page_alloc.c +0
-0
mm/page_alloc.c __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, · line 1869
Quoted 9 lines
mm/page_alloc.c +0
-3@@ -1864,9 +1864,6 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,cond_resched (); -if (order !=0 ) -drain_all_pages (); -if (likely (*did_some_progress)) page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist, high_zoneidx,
You may be interested in Mel's patchset that he just proposed for -mm which identifies watermark variations on machines with high cpu counts (perhaps even eight, as in this report). The last patch actually reworks this hunk of the code as well.
http://marc.info/?l=linux-mm&m=128255044912938 http://marc.info/?l=linux-mm&m=128255045312950 http://marc.info/?l=linux-mm&m=128255045012942 http://marc.info/?l=linux-mm&m=128255045612954
Dave, it would be interesting to see if this fixes your problem.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · David Rientjes
Quoted 7 lines · Wu Fengguang
Quoted 7 lines
I've been testing parallel create workloads over the weekend, and I've seen this a couple of times now under 8 thread parallel creates with XFS. I'm running on an 8p VM with 4GB RAM and a fast disk subsystem. Basically I am seeing the create rate drop to zero with all 8 CPUs stuck spinning for up to 2 minutes. 'echo t >
/proc/sysrq-trigger' while this is occurring gives the following trace for all the fs-mark processes:
.....
Quoted 6 lines
You may be interested in Mel's patchset that he just proposed for -mm which identifies watermark variations on machines with high cpu counts (perhaps even eight, as in this report). The last patch actually reworks this hunk of the code as well.
Quoted 6 lines
http://marc.info/?l=linux-mm&m=128255044912938 http://marc.info/?l=linux-mm&m=128255045312950 http://marc.info/?l=linux-mm&m=128255045012942 http://marc.info/?l=linux-mm&m=128255045612954Dave, it would be interesting to see if this fixes your problem.
That looks promising - I'll give it a shot, though my test case is not really what you'd call reproducable(*) so it might take a couple of days before I can say whether the issue has gone away or not.
Cheers,
Dave.
(*) create 100 million inodes in parallel using fsmark, collect and watch behavioural metrics via PCP/pmchart for stuff out of the ordinary, and dump stack traces, etc when somthing strange occurs.
Signature · 8 lines
Dave Chinner [email protected] -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>