Reclaim in the face of really fast I/O

LSF/MM TOPIC Dave Hansen 2 participants 3 messages Jan 15, 2015 – Jan 16, 2015 Memory management

Share & tools

Dave Hansen

author

2015-01-15 21:00 UTC #0 lore raw

I/O devices are only getting faster. In fact, they're getting closer and closer to memory in latency and bandwidth. But the VM is still designed to do very orderly and costly procedures to reclaim memory, and the existing algorithms don't parallelize particularly well. They hit contention on mmap_sem or the lru locks well before all of the CPU horsepower that we have can be brought to bear on reclaim.

Once the latency to bring pages in and out of storage becomes low enough, reclaiming the _right_ pages becomes much less important than doing something useful with the CPU horsepower that we have.

We need to talk about ways to do reclaim with lower CPU overhead and to parallelize more effectively.

There has been some research in this area by some folks at Intel and we could quickly summarize what has been learned so far to help kick off a discussion.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sasha Levin

2015-01-16 01:21 UTC #1 lore raw

Quoted 11 lines · Dave Hansen

I/O devices are only getting faster. In fact, they're getting closer and closer to memory in latency and bandwidth. But the VM is still designed to do very orderly and costly procedures to reclaim memory, and the existing algorithms don't parallelize particularly well. They hit contention on mmap_sem or the lru locks well before all of the CPU horsepower that we have can be brought to bear on reclaim.
Once the latency to bring pages in and out of storage becomes low enough, reclaiming the _right_ pages becomes much less important than doing something useful with the CPU horsepower that we have.

Quoted 6 lines

We need to talk about ways to do reclaim with lower CPU overhead and to parallelize more effectively.
There has been some research in this area by some folks at Intel and we could quickly summarize what has been learned so far to help kick off a discussion.

I was actually planning to bring that up. Trinity can cause enough stress to a system that the hang watchdog triggers (with a 10 minute timeout!) inside reclaim code.

Thanks,
Sasha

Sasha Levin

2015-01-16 01:32 UTC #2 lore raw

Quoted 15 lines · Sasha Levin

Quoted 14 lines · Dave Hansen
I/O devices are only getting faster. In fact, they're getting closer and closer to memory in latency and bandwidth. But the VM is still designed to do very orderly and costly procedures to reclaim memory, and the existing algorithms don't parallelize particularly well. They hit contention on mmap_sem or the lru locks well before all of the CPU horsepower that we have can be brought to bear on reclaim.
Once the latency to bring pages in and out of storage becomes low enough, reclaiming the _right_ pages becomes much less important than doing something useful with the CPU horsepower that we have.
We need to talk about ways to do reclaim with lower CPU overhead and to parallelize more effectively.

Quoted 7 lines

Quoted 3 lines
There has been some research in this area by some folks at Intel and we could quickly summarize what has been learned so far to help kick off a discussion.
I was actually planning to bring that up. Trinity can cause enough stress to a system that the hang watchdog triggers (with a 10 minute timeout!) inside reclaim code.

Something like this:

[ 5412.398971] trinity-c23     R  running task    10768 29063  30295 0x10000006
[ 5412.400111]  ffff88046127b178 0000000000000ec3 0000000000000000 0000000000000000
[ 5412.401404]  ffffe8fff263cd6e 0000000000000000 0000000000000000 0000000000000000
[ 5412.402695]  0000000000000000 ffff88076bfff4c8 0000000000001099 0000000000000000
[ 5412.404084] Call Trace:
[ 5412.404495]  [<ffffffff916124f2>] ? _raw_spin_unlock_irqrestore+0xa2/0xf0
[ 5412.405611]  [<ffffffff915fbe25>] schedule+0x55/0x280
[ 5412.406435]  [<ffffffff81903e72>] throttle_direct_reclaim+0x432/0x660
[ 5412.407530]  [<ffffffff81552160>] ? __init_waitqueue_head+0xe0/0xe0
[ 5412.408565]  [<ffffffff81916007>] try_to_free_pages+0xf7/0x460
[ 5412.409528]  [<ffffffff818e25e1>] __alloc_pages_nodemask+0xc01/0x1940
[ 5412.410589]  [<ffffffff81a05646>] alloc_pages_vma+0x216/0x6f0
[ 5412.411572]  [<ffffffff8191a62a>] ? shmem_alloc_page+0x9a/0x170
[ 5412.412556]  [<ffffffff8191a62a>] shmem_alloc_page+0x9a/0x170
[ 5412.413507]  [<ffffffff818bc241>] ? find_get_entry+0x191/0x2c0
[ 5412.414475]  [<ffffffff818bc0b5>] ? find_get_entry+0x5/0x2c0
[ 5412.415513]  [<ffffffff818bd02b>] ? find_lock_entry+0x2b/0x140
[ 5412.416474]  [<ffffffff81924f16>] shmem_getpage_gfp+0xde6/0x1710
[ 5412.417461]  [<ffffffff81926eb1>] shmem_fault+0x1a1/0x7d0
[ 5412.418353]  [<ffffffff819745dd>] __do_fault+0xad/0x2a0
[ 5412.419285]  [<ffffffff8197e6c1>] handle_mm_fault+0x1331/0x5440
[ 5412.420121]  [<ffffffff812f59d3>] __do_page_fault+0x2d3/0xfb0
[ 5412.420877]  [<ffffffff81579bc7>] ? mark_held_locks+0x117/0x2b0
[ 5412.421688]  [<ffffffff81571fcd>] ? trace_hardirqs_off+0xd/0x10
[ 5412.422491]  [<ffffffff812f6828>] trace_do_page_fault+0xc8/0x420
[ 5412.423348]  [<ffffffff812da7d3>] do_async_page_fault+0x83/0x120
[ 5412.424175]  [<ffffffff91614d68>] async_page_fault+0x28/0x30
[ 5412.424920]  [<ffffffff81960e5e>] ? iov_iter_fault_in_readable+0x17e/0x280
[ 5412.425851]  [<ffffffff814a1045>] ? ___might_sleep+0x2a5/0x420
[ 5412.426645]  [<ffffffff818baa49>] generic_perform_write+0x179/0x5b0
[ 5412.427573]  [<ffffffff81b2b267>] ? __mnt_drop_write+0x57/0xa0
[ 5412.428365]  [<ffffffff818c23fc>] __generic_file_write_iter+0x59c/0x13e0
[ 5412.429292]  [<ffffffff81aa1c7c>] ? rw_copy_check_uvector+0x5c/0x470
[ 5412.430153]  [<ffffffff81a8f964>] ? kasan_poison_shadow+0x34/0x40
[ 5412.431062]  [<ffffffff818c3319>] generic_file_write_iter+0xd9/0x510
[ 5412.431892]  [<ffffffff81a9bec0>] ? new_sync_read+0x220/0x220
[ 5412.432675]  [<ffffffff81a9c17d>] do_iter_readv_writev+0x9d/0x190
[ 5412.433504]  [<ffffffff81aa22c9>] do_readv_writev+0x239/0xe10
[ 5412.434294]  [<ffffffff818c3240>] ? __generic_file_write_iter+0x13e0/0x13e0
[ 5412.435300]  [<ffffffff8174cb7e>] ? acct_account_cputime+0x6e/0xa0
[ 5412.435942]  [<ffffffff818c3240>] ? __generic_file_write_iter+0x13e0/0x13e0
[ 5412.436678]  [<ffffffff818b5fa7>] ? context_tracking_user_exit+0xc7/0x330
[ 5412.437395]  [<ffffffff8157a281>] ? trace_hardirqs_on_caller+0x521/0x850
[ 5412.438097]  [<ffffffff81aa3033>] vfs_writev+0x93/0x100
[ 5412.438632]  [<ffffffff81aa39ba>] SyS_pwritev+0x11a/0x200

Thanks,
Sasha