High paging activity as soon as the swap is touched (with steps and code to reproduce it)
On Friday, July 20, 2018, 2:03:48 AM GMT+2, [email protected] <[email protected]> wrote: >
Quoted 9 lines
https://bugzilla.kernel.org/show_bug.cgi?id=200105
--- Comment #42 from Andrew Morton ([email protected]) --- Sorry, but nobody reads bugzilla. I tried to switch this discussion to an email thread for a reason!
Please resend all this (useful) info in reply to the email thread which I created for this purpose.
I'll resend the last message and attachments. Anyone interested on the previous "episodes" go read
https://bugzilla.kernel.org/show_bug.cgi?id=200105
Three attachments:
1) the read_vmstat output (started right before the first memeater was executed)
2) the perf report (as requested, perf was started right before executing the last memeater, the one that caused the swapping)
3) start and finish allocation times of the memeaters (except the last memeater which has only the starting time because I've ctrl-Z it before it completed all its allocations).
40,243 lines not shown · raw message
Part of this message is not shown here. View the raw message.
Quoted 9 lines · john terragon
Quoted 7 lines · On Friday, July 20, 2018, 2:03:48 AM GMT+2…
https://bugzilla.kernel.org/show_bug.cgi?id=200105 <https://bugzilla.kernel.org/show_bug.cgi?id=200105>
--- Comment #42 from Andrew Morton ([email protected] <mailto:[email protected]>) --- Sorry, but nobody reads bugzilla.A I tried to switch this discussion to an
email thread for a reason!
Quoted 5 lines
Quoted 2 lines
Please resend all this (useful) info in reply to the email thread which I created for this purpose.
I'll resend the last message and attachments. Anyone interested on the previous "episodes" go read
https://bugzilla.kernel.org/show_bug.cgi?id=200105
The summary is that John has put together a reliable reproducer for a problem he's seeing where on high memory usage any of his desktop systems with SSDs hang for around a minute, completely unresponsive, and swaps out 2-3x more memory than the system is allocating.
John's issue only happens using a LUKS encrypted swap partition, unencrypted swap or swap encrypted without LUKS works fine.
In one test (out5.txt) where most system memory is taken by anon pages beforehand, the heavy direct reclaim that Michal noticed lasts for 24 seconds, during which on average if I've crunched my numbers right, John's test program was allocating at 4MiB/s, the system overall (pgalloc_normal) was allocating at 235MiB/s, and the system was swapping out (pswpout) at 673MiB/s. pgalloc_normal and pswpout stay roughly the same each second, no big swings.
Is the disparity between allocation and swapout rate expected?
John ran perf during another test right before the last test program was started (this doesn't include the initial large allocation bringing the system close to swapping). The top five allocators (kmem:mm_page_alloc):
# Overhead Pid:Command
# ........ .......................
#
48.45% 2005:memeater # the test program
32.08% 73:kswapd0
3.16% 1957:perf_4.17
1.41% 1748:watch
1.16% 2043:freeSo it seems to be just reclaim activity, but why so much when the test program only allocates at 4MiB/s?
John, adding -g to perf record would show call stacks.
I'll be away for 2.5 weeks so won't be able to get back to this until then.
Quoted 31 lines · Daniel Jordan
Quoted 12 lines · john terragon
On Friday, July 20, 2018, 2:03:48 AM GMT+2, [email protected] <[email protected]> wrote:
A >
A >https://bugzilla.kernel.org/show_bug.cgi?id=200105 <https://bugzilla.kernel.org/show_bug.cgi?id=200105>
A >
A >--- Comment #42 from Andrew Morton ([email protected] <mailto:[email protected]>) --- A >Sorry, but nobody reads bugzilla.A I tried to switch this discussion to an
A >email thread for a reason!
A >
A >Please resend all this (useful) info in reply to the email thread which I
A >created for this purpose.Quoted 2 lines
I'll resend the last message and attachments. Anyone interested on the previous "episodes" go read
https://bugzilla.kernel.org/show_bug.cgi?id=200105The summary is that John has put together a reliable reproducer for a problem he's seeing where on high memory usage any of his desktop systems with SSDs hang for around a minute, completely unresponsive, and swaps out 2-3x more memory than the system is allocating.
John's issue only happens using a LUKS encrypted swap partition, unencrypted swap or swap encrypted without LUKS works fine.
In one test (out5.txt) where most system memory is taken by anon pages beforehand, the heavy direct reclaim that Michal noticed lasts for 24 seconds, during which on average if I've crunched my numbers right, John's test program was allocating at 4MiB/s, the system overall (
pgalloc_normal) was allocating at 235MiB/s, and the system was swapping out (pswpout) at 673MiB/s.Apgalloc_normaland pswpout stay roughly the same each second, no big swings.Is the disparity between allocation and swapout rate expected?
John ran perf during another test right before the last test program was started (this doesn't include the initial large allocation bringing the system close to swapping).A The top five allocators (kmem:mm_page_alloc):
# OverheadA A A A A Pid:Command
# ........A .......................
#
A A A 48.45%A A A A 2005:memeaterA A A A # the test program A A A 32.08%A A A A A A 73:kswapd0
Quoted 5 lines
A A A A 3.16%A A A A 1957:perf_4.17 A A A A 1.41%A A A A 1748:watch A A A A 1.16%A A A A 2043:freeSo it seems to be just reclaim activity, but why so much when the test program only allocates at 4MiB/s?
Should add that during the 24 seconds, reclaim efficiency for both kswapd and direct (pgsteal/pgscan) hovered around 1%, which seems low.
The 24 seconds cover =S 1530092789 to =S 1530092812 in out5.txt from bugzilla.
I'll add a couple of points to Daniel's summary:
1) initially I performed the test with the memeaters within a gui (KDE, but I tried gnome too). The system freezes occur when the test is performed within a gui.
2) Then, to isolate what seemed to be more symptoms (the freezes) than the problem itself, I started to perform the test using just text consoles. In these conditions there are no big, visible, system freezes. Although one of the tools used to snapshot vmstat (Michal's read_vmstat) complains with messages like "it took 28s to snapshot!" or something like that, so maybe there are still slowdowns.
3) Anyways, both in gui and text "mode", the test always causes a huge swap activity which seems to be disproportionate given the low pressure the vm is put under (tried as low as 4Mb/s allocation rate).
4) These are the swap configurations in which the huge swap activity occurs:
-luks encrypted swap partition
-dmcrypt plain swap partition
-unencrypted swapfile located inside an encrypted, luks or dmcrypt plain, filesystem
5) The test does not cause any huge swap activity under these swap configs *when performed without a gui*:
a) unencrypted swap partition
b) encrypted swap file located inside and unencryted ext4 filesystem (that is ext4 directly on the partition and the directory in which the swapfile resides is encrypted with fs-level encryption, thus no dm-crypt involved).
6) I was particularly happy about b) above because it seemed to be a viable workaround, but sadly I have to report that, although the test does not causes large swap activity with b), minute-long system freezes are occurring during normal usage even with b). These seem to be the same kind of freezes that prompted me to find a way to reliably reproduce them under controlled conditions. I haven't yet performed the test with b)+gui, though, only b)+text console as I wrote in 5.
[I am really sorry to be slow on responding]
Quoted 18 lines · Daniel Jordan
John's issue only happens using a LUKS encrypted swap partition, unencrypted swap or swap encrypted without LUKS works fine.
In one test (out5.txt) where most system memory is taken by anon pages beforehand, the heavy direct reclaim that Michal noticed lasts for 24 seconds, during which on average if I've crunched my numbers right, John's test program was allocating at 4MiB/s, the system overall (
pgalloc_normal) was allocating at 235MiB/s, and the system was swapping out (pswpout) at 673MiB/s.pgalloc_normaland pswpout stay roughly the same each second, no big swings.Is the disparity between allocation and swapout rate expected?
John ran perf during another test right before the last test program was started (this doesn't include the initial large allocation bringing the system close to swapping). The top five allocators (kmem:mm_page_alloc):
Quoted 8 lines
# Overhead Pid:Command
# ........ ....................... # 48.45% 2005:memeater # the test program 32.08% 73:kswapd0 3.16% 1957:perf_4.17 1.41% 1748:watch 1.16% 2043:free
Huh, kswapd allocating memory sounds really wrong here. Is it possible that the swap device driver is double buffering and allocating a new page for each one to swap out?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 34 lines · Michal Hocko
[I am really sorry to be slow on responding]
Quoted 18 lines · Daniel Jordan
John's issue only happens using a LUKS encrypted swap partition, unencrypted swap or swap encrypted without LUKS works fine.
In one test (out5.txt) where most system memory is taken by anon pages beforehand, the heavy direct reclaim that Michal noticed lasts for 24 seconds, during which on average if I've crunched my numbers right, John's test program was allocating at 4MiB/s, the system overall (
pgalloc_normal) was allocating at 235MiB/s, and the system was swapping out (pswpout) at 673MiB/s.pgalloc_normaland pswpout stay roughly the same each second, no big swings.Is the disparity between allocation and swapout rate expected?
John ran perf during another test right before the last test program was started (this doesn't include the initial large allocation bringing the system close to swapping). The top five allocators (kmem:mm_page_alloc):
Quoted 8 lines
# Overhead Pid:Command
# ........ ....................... # 48.45% 2005:memeater # the test program 32.08% 73:kswapd0 3.16% 1957:perf_4.17 1.41% 1748:watch 1.16% 2043:freeHuh, kswapd allocating memory sounds really wrong here. Is it possible that the swap device driver is double buffering and allocating a new page for each one to swap out?