mm: boosted kswapd reclaim b0rks system cache balance
Subsystems (3)
PATCH 1/1 mm: boosted kswapd reclaim b0rks system cache balance
From: Dave Chinner <[email protected]>
When running a simple, steady state 4kB file creation test to simulate extracting tarballs larger than memory full of small files into the filesystem, I noticed that once memory fills up the cache balance goes to hell.
The workload is creating one dirty cached inode for every dirty page, both of which should require a single IO each to clean and reclaim, and creation of inodes is throttled by the rate at which dirty writeback runs at (via balance dirty pages). Hence the ingest rate of new cached inodes and page cache pages is identical and steady. As a result, memory reclaim should quickly find a steady balance between page cache and inode caches.
It doesn't.
The moment memory fills, the page cache is reclaimed at a much faster rate than the inode cache, and evidence suggests taht the inode cache shrinker is not being called when large batches of pages are being reclaimed. In roughly the same time period that it takes to fill memory with 50% pages and 50% slab caches, memory reclaim reduces the page cache down to just dirty pages and slab caches fill the entirity of memory.
At the point where the page cache is reduced to just the dirty pages, there is a clear change in write IO patterns. Up to this point it has been running at a steady 1500 write IOPS for ~200MB/s of write throughtput (data, journal and metadata). Once the page cache is trimmed to just dirty pages, the write IOPS immediately start spiking to 5-10,000 IOPS and there is a noticable change in IO sizes and completion times. The SSD is fast enough to soak these up, so the measured performance is only slightly affected (numbers below). It results in > ~50% throughput slowdowns on a spinning disk with a NVRAM RAID cache in front of it, though. I didn't capture the numbers at the time, and it takes far to long for me to care to run it again and get them.
SSD perf degradation as the LRU empties to just dirty pages:
FSUse% Count Size Files/sec App Overhead
......
0 4320000 4096 51349.6 1370049
0 4480000 4096 48492.9 1362823
0 4640000 4096 48473.8 1435623
0 4800000 4096 46846.6 1370959
0 4960000 4096 47086.6 1567401
0 5120000 4096 46288.8 1368056
0 5280000 4096 46003.2 1391604
0 5440000 4096 46033.4 1458440
0 5600000 4096 45035.1 1484132
0 5760000 4096 43757.6 1492594
0 5920000 4096 40739.4 1552104
0 6080000 4096 37309.4 1627355
0 6240000 4096 42003.3 1517357
.....
real 3m28.093s
user 0m57.852s
sys 14m28.193sAverage rate: 51724.6+/-2.4e+04 files/sec.
At first memory full point:
MemFree: 432676 kB Active(file): 89820 kB Inactive(file): 7330892 kB Dirty: 1603576 kB Writeback: 2908 kB Slab: 6579384 kB SReclaimable: 3727612 kB SUnreclaim: 2851772 kB
A few seconds later at about half the page cache reclaimed:
MemFree: 1880588 kB Active(file): 89948 kB Inactive(file): 3021796 kB Dirty: 1097072 kB Writeback: 2600 kB Slab: 8900912 kB SReclaimable: 5060104 kB SUnreclaim: 3840808 kB
And at about the 6080000 count point in the results above, right to the end of the test:
MemFree: 574900 kB Active(file): 89856 kB Inactive(file): 483120 kB Dirty: 372436 kB Writeback: 324 kB KReclaimable: 6506496 kB Slab: 11898956 kB SReclaimable: 6506496 kB SUnreclaim: 5392460 kB
So the LRU is largely full of dirty pages, and we're getting spikes of random writeback from memory reclaim so it's all going to shit. Behaviour never recovers, the page cache remains pinned at just dirty pages, and nothing I could tune would make any difference. vfs_cache_pressure makes no difference - I would it up so high it should trim the entire inode caches in a singel pass, yet it didn't do anything. It was clear from tracing and live telemetry that the shrinkers were pretty much not running except when there was absolutely no memory free at all, and then they did the minimum necessary to free memory to make progress.
So I went looking at the code, trying to find places where pages got reclaimed and the shrinkers weren't called. There's only one - kswapd doing boosted reclaim as per commit 1c30844d2dfe ("mm: reclaim small amounts of memory when an external fragmentation event occurs"). I'm not even using THP or allocating huge pages, so this code should not be active or having any effect onmemory reclaim at all, yet the majority of reclaim is being done with "boost" and so it's not reclaiming slab caches at all. It will only free clean pages from the LRU.
And so when we do run out memory, it switches to normal reclaim, which hits dirty pages on the LRU and does some shrinker work, too, but then appears to switch back to boosted reclaim one watermarks are reached.
The patch below restores page cache vs inode cache balance for this steady state workload. It balances out at about 40% page cache, 60% slab cache, and sustained performance is 10-15% higher than without this patch because the IO patterns remain in control of dirty writeback and the filesystem, not kswapd.
Performance with boosted reclaim also running shrinkers over the same steady state portion of the test as above.
FSUse% Count Size Files/sec App Overhead
......
0 4320000 4096 51341.9 1409983
0 4480000 4096 51157.5 1486421
0 4640000 4096 52041.5 1421837
0 4800000 4096 52124.2 1442169
0 4960000 4096 56266.6 1388865
0 5120000 4096 52892.2 1357650
0 5280000 4096 51879.5 1326590
0 5440000 4096 52178.7 1362889
0 5600000 4096 53863.0 1345956
0 5760000 4096 52538.7 1321422
0 5920000 4096 53144.5 1336966
0 6080000 4096 53647.7 1372146
0 6240000 4096 52434.7 1362534.....
real 3m11.543s user 0m57.506s sys 14m20.913s
Average rate: 57634.2+/-2.8e+04 files/sec.
So it completed ~10% faster (both wall time and create rate) and had far better IO patterns so the differences would be even more pronounced on slow storage.
This patch is not a fix, just a demonstration of the fact that the heuristics this "boosted reclaim for compaction" are based on are flawed, will have nasty side effects for users that don't use THP and so needs revisiting.
--- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/vmscan.c +2 -2
@@ -2748,10 +2748,10 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc)shrink_node_memcg (pgdat, memcg, sc, &lru_pages); node_lru_pages += lru_pages; -if (sc->may_shrinkslab) { +//if (sc->may_shrinkslab) { shrink_slab (sc->gfp_mask, pgdat->node_id, memcg, sc->priority); - } +//} /* Record the group's reclaim efficiency */ vmpressure(sc->gfp_mask, memcg, false,
Signature · 1 line
2.23.0.rc1
Michal Hocko
reviewer[Cc Mel and Vlastimil as it seems like fallout from 1c30844d2dfe2]
Quoted 189 lines · Dave Chinner
From: Dave Chinner <[email protected]>
When running a simple, steady state 4kB file creation test to simulate extracting tarballs larger than memory full of small files into the filesystem, I noticed that once memory fills up the cache balance goes to hell.
The workload is creating one dirty cached inode for every dirty page, both of which should require a single IO each to clean and reclaim, and creation of inodes is throttled by the rate at which dirty writeback runs at (via balance dirty pages). Hence the ingest rate of new cached inodes and page cache pages is identical and steady. As a result, memory reclaim should quickly find a steady balance between page cache and inode caches.
It doesn't.
The moment memory fills, the page cache is reclaimed at a much faster rate than the inode cache, and evidence suggests taht the inode cache shrinker is not being called when large batches of pages are being reclaimed. In roughly the same time period that it takes to fill memory with 50% pages and 50% slab caches, memory reclaim reduces the page cache down to just dirty pages and slab caches fill the entirity of memory.
At the point where the page cache is reduced to just the dirty pages, there is a clear change in write IO patterns. Up to this point it has been running at a steady 1500 write IOPS for ~200MB/s of write throughtput (data, journal and metadata). Once the page cache is trimmed to just dirty pages, the write IOPS immediately start spiking to 5-10,000 IOPS and there is a noticable change in IO sizes and completion times. The SSD is fast enough to soak these up, so the measured performance is only slightly affected (numbers below). It results in > ~50% throughput slowdowns on a spinning disk with a NVRAM RAID cache in front of it, though. I didn't capture the numbers at the time, and it takes far to long for me to care to run it again and get them.
SSD perf degradation as the LRU empties to just dirty pages:
FSUse% Count Size Files/sec App Overhead
......0 4320000 4096 51349.6 1370049 0 4480000 4096 48492.9 1362823 0 4640000 4096 48473.8 1435623 0 4800000 4096 46846.6 1370959 0 4960000 4096 47086.6 1567401 0 5120000 4096 46288.8 1368056 0 5280000 4096 46003.2 1391604 0 5440000 4096 46033.4 1458440 0 5600000 4096 45035.1 1484132 0 5760000 4096 43757.6 1492594 0 5920000 4096 40739.4 1552104 0 6080000 4096 37309.4 1627355 0 6240000 4096 42003.3 1517357 ..... real 3m28.093s user 0m57.852s sys 14m28.193sAverage rate: 51724.6+/-2.4e+04 files/sec.
At first memory full point:
MemFree: 432676 kB Active(file): 89820 kB Inactive(file): 7330892 kB Dirty: 1603576 kB Writeback: 2908 kB Slab: 6579384 kB SReclaimable: 3727612 kB SUnreclaim: 2851772 kBA few seconds later at about half the page cache reclaimed:
MemFree: 1880588 kB Active(file): 89948 kB Inactive(file): 3021796 kB Dirty: 1097072 kB Writeback: 2600 kB Slab: 8900912 kB SReclaimable: 5060104 kB SUnreclaim: 3840808 kBAnd at about the 6080000 count point in the results above, right to the end of the test:
MemFree: 574900 kB Active(file): 89856 kB Inactive(file): 483120 kB Dirty: 372436 kB Writeback: 324 kB KReclaimable: 6506496 kB Slab: 11898956 kB SReclaimable: 6506496 kB SUnreclaim: 5392460 kBSo the LRU is largely full of dirty pages, and we're getting spikes of random writeback from memory reclaim so it's all going to shit. Behaviour never recovers, the page cache remains pinned at just dirty pages, and nothing I could tune would make any difference.
vfs_cache_pressuremakes no difference - I would it up so high it should trim the entire inode caches in a singel pass, yet it didn't do anything. It was clear from tracing and live telemetry that the shrinkers were pretty much not running except when there was absolutely no memory free at all, and then they did the minimum necessary to free memory to make progress.So I went looking at the code, trying to find places where pages got reclaimed and the shrinkers weren't called. There's only one - kswapd doing boosted reclaim as per commit
1c30844d2dfe("mm: reclaim small amounts of memory when an external fragmentation event occurs"). I'm not even using THP or allocating huge pages, so this code should not be active or having any effect onmemory reclaim at all, yet the majority of reclaim is being done with "boost" and so it's not reclaiming slab caches at all. It will only free clean pages from the LRU.And so when we do run out memory, it switches to normal reclaim, which hits dirty pages on the LRU and does some shrinker work, too, but then appears to switch back to boosted reclaim one watermarks are reached.
The patch below restores page cache vs inode cache balance for this steady state workload. It balances out at about 40% page cache, 60% slab cache, and sustained performance is 10-15% higher than without this patch because the IO patterns remain in control of dirty writeback and the filesystem, not kswapd.
Performance with boosted reclaim also running shrinkers over the same steady state portion of the test as above.
FSUse% Count Size Files/sec App Overhead
......0 4320000 4096 51341.9 1409983 0 4480000 4096 51157.5 1486421 0 4640000 4096 52041.5 1421837 0 4800000 4096 52124.2 1442169 0 4960000 4096 56266.6 1388865 0 5120000 4096 52892.2 1357650 0 5280000 4096 51879.5 1326590 0 5440000 4096 52178.7 1362889 0 5600000 4096 53863.0 1345956 0 5760000 4096 52538.7 1321422 0 5920000 4096 53144.5 1336966 0 6080000 4096 53647.7 1372146 0 6240000 4096 52434.7 1362534.....
real 3m11.543s user 0m57.506s sys 14m20.913sAverage rate: 57634.2+/-2.8e+04 files/sec.
So it completed ~10% faster (both wall time and create rate) and had far better IO patterns so the differences would be even more pronounced on slow storage.
This patch is not a fix, just a demonstration of the fact that the heuristics this "boosted reclaim for compaction" are based on are flawed, will have nasty side effects for users that don't use THP and so needs revisiting.
Signed-off-by Dave Chinner <[email protected]>--- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)mm/vmscan.c +2
-2@@ -2748,10 +2748,10 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc)shrink_node_memcg (pgdat, memcg, sc, &lru_pages); node_lru_pages += lru_pages; -if (sc->may_shrinkslab) { +//if (sc->may_shrinkslab) { shrink_slab (sc->gfp_mask, pgdat->node_id, memcg, sc->priority); - } +//} /* Record the group's reclaim efficiency */ vmpressure(sc->gfp_mask, memcg, false,Signature · 1 line
2.23.0.rc1
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 2 lines · Michal Hocko
[Cc Mel and Vlastimil as it seems like fallout from
1c30844d2dfe2]
More than likely.
Quoted 8 lines · Dave Chinner
Quoted 7 lines
From: Dave Chinner <[email protected]>
When running a simple, steady state 4kB file creation test to simulate extracting tarballs larger than memory full of small files into the filesystem, I noticed that once memory fills up the cache balance goes to hell.
Ok, I'm assuming you are using fsmark with -k to keep files around, and -S0 to leave cleaning to the background flush, a number of files per iteration to get regular reporting and a total number of iterations to fill memory to hit what you're seeing. I've created a configuration that should do this but it'll take a long time to run on a local test machine.
I'm not 100% certain I guessed right as to get fsmark reports while memory fills, it would have to be fewer files so each iteration would have to preserve files. If the number of files per iteration is large enough to fill memory then the drop in files/sec is not visible from the fs_mark output (or we are using different versions). I guess you could just be calculating average files/sec over the entire run based on elapsed time.
Quoted 18 lines
Quoted 18 lines
The workload is creating one dirty cached inode for every dirty page, both of which should require a single IO each to clean and reclaim, and creation of inodes is throttled by the rate at which dirty writeback runs at (via balance dirty pages). Hence the ingest rate of new cached inodes and page cache pages is identical and steady. As a result, memory reclaim should quickly find a steady balance between page cache and inode caches.
It doesn't.
The moment memory fills, the page cache is reclaimed at a much faster rate than the inode cache, and evidence suggests taht the inode cache shrinker is not being called when large batches of pages are being reclaimed. In roughly the same time period that it takes to fill memory with 50% pages and 50% slab caches, memory reclaim reduces the page cache down to just dirty pages and slab caches fill the entirity of memory.
Quoted 4 lines
Quoted 4 lines
At the point where the page cache is reduced to just the dirty pages, there is a clear change in write IO patterns. Up to this point it has been running at a steady 1500 write IOPS for ~200MB/s of write throughtput (data, journal and metadata).
As observed by iostat -x or something else? Sum of r/s and w/s would approximate iops but not the breakdown of whether it is data, journal or metadata writes. The rest can be inferred from a blktrace but I would prefer to replicate your setup as close as possible. If you're not using fs_mark to report Files/sec, are you simply monitoring df -i over time?
Quoted 3 lines
Quoted 3 lines
<SNIP additional detail on
fs_markoutput> <SNIP additional detail on monitoring meminfo over time> <SNIP observations on dirty handling>
All understood.
Quoted 21 lines
Quoted 21 lines
So I went looking at the code, trying to find places where pages got reclaimed and the shrinkers weren't called. There's only one - kswapd doing boosted reclaim as per commit
1c30844d2dfe("mm: reclaim small amounts of memory when an external fragmentation event occurs"). I'm not even using THP or allocating huge pages, so this code should not be active or having any effect onmemory reclaim at all, yet the majority of reclaim is being done with "boost" and so it's not reclaiming slab caches at all. It will only free clean pages from the LRU.And so when we do run out memory, it switches to normal reclaim, which hits dirty pages on the LRU and does some shrinker work, too, but then appears to switch back to boosted reclaim one watermarks are reached.
The patch below restores page cache vs inode cache balance for this steady state workload. It balances out at about 40% page cache, 60% slab cache, and sustained performance is 10-15% higher than without this patch because the IO patterns remain in control of dirty writeback and the filesystem, not kswapd.
Quoted 3 lines
Quoted 3 lines
Performance with boosted reclaim also running shrinkers over the same steady state portion of the test as above.
The boosting was not intended to target THP specifically -- it was meant to help recover early from any fragmentation-related event for any user that might need it. Hence, it's not tied to THP but even with THP disabled, the boosting will still take effect.
One band-aid would be to disable watermark boosting entirely when THP is disabled but that feels wrong. However, I would be interested in hearing if sysctl vm.watermark_boost_factor=0 has the same effect as your patch.
The intention behind avoiding slab reclaim from kswapd context is that the boost is due to a fragmentation-causing event. Compaction cannot move slab pages so reclaiming them in response to fragmentation does not directly help. However, it can indirectly help by avoiding fragmentation-causing events due to slab allocation like inodes and also mean that reclaim behaviour is not special cased.
On that basis, it may justify ripping out the may_shrinkslab logic everywhere. The downside is that some microbenchmarks will notice. Specifically IO benchmarks that fill memory and reread (particularly rereading the metadata via any inode operation) may show reduced results. Such benchmarks can be strongly affected by whether the inode information is still memory resident and watermark boosting reduces the changes the data is still resident in memory. Technically still a regression but a tunable one.
Hence the following "it builds" patch that has zero supporting data on whether it's a good idea or not.
mm/vmscan.c +2 -11
@@ -88,9 +88,6 @@ struct scan_control {/* Can pages be swapped as part of reclaim? */ unsigned int may_swap:1; -/* e.g. boosted watermark reclaim leaves slabs alone */ - unsigned int may_shrinkslab:1; -/* * Cgroups are not reclaimed below their configured memory.low, * unless we threaten to OOM. If any cgroups are skipped due to
@@ -2714,10 +2711,8 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc)shrink_node_memcg (pgdat, memcg, sc, &lru_pages); node_lru_pages += lru_pages; -if (sc->may_shrinkslab) { -shrink_slab (sc->gfp_mask, pgdat->node_id, - memcg, sc->priority); - } +shrink_slab (sc->gfp_mask, pgdat->node_id, + memcg, sc->priority);/* Record the group's reclaim efficiency */ vmpressure(sc->gfp_mask, memcg, false,
@@ -3194,7 +3189,6 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order, .may_writepage = !laptop_mode, .may_unmap = 1, .may_swap = 1, - .may_shrinkslab = 1, };/*
@@ -3238,7 +3232,6 @@ unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg, .may_unmap = 1, .reclaim_idx = MAX_NR_ZONES - 1, .may_swap = !noswap, - .may_shrinkslab = 1, };unsigned long lru_pages;
@@ -3286,7 +3279,6 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, .may_writepage = !laptop_mode, .may_unmap = 1, .may_swap = may_swap, - .may_shrinkslab = 1, };set_task_reclaim_state (current, &sc.reclaim_state);
@@ -3598,7 +3590,6 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)*/ sc.may_writepage = !laptop_mode && !nr_boost_reclaim; sc.may_swap = !nr_boost_reclaim; - sc.may_shrinkslab = !nr_boost_reclaim;/* * Do some background aging of the anon list, to give
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 11 lines · Mel Gorman
<SNIP>
On that basis, it may justify ripping out the
may_shrinkslablogic everywhere. The downside is that some microbenchmarks will notice. Specifically IO benchmarks that fill memory and reread (particularly rereading the metadata via any inode operation) may show reduced results. Such benchmarks can be strongly affected by whether the inode information is still memory resident and watermark boosting reduces the changes the data is still resident in memory. Technically still a regression but a tunable one.
Quoted 3 lines
Hence the following "it builds" patch that has zero supporting data on whether it's a good idea or not.
This is a more complete version of the same patch that summaries the problem and includes data from my own testing
---8<---
mm, vmscan: Do not special-case slab reclaim when watermarks are boosted
Dave Chinner reported a problem pointing a finger at commit 1c30844d2dfe ("mm: reclaim small amounts of memory when an external fragmentation event occurs"). The report is extensive (see
https://lore.kernel.org/linux-mm/[email protected]/)
and it's worth recording the most relevant parts (colorful language and typos included).
When running a simple, steady state 4kB file creation test to simulate extracting tarballs larger than memory full of small files into the filesystem, I noticed that once memory fills up the cache balance goes to hell.
The workload is creating one dirty cached inode for every dirty page, both of which should require a single IO each to clean and reclaim, and creation of inodes is throttled by the rate at which dirty writeback runs at (via balance dirty pages). Hence the ingest rate of new cached inodes and page cache pages is identical and steady. As a result, memory reclaim should quickly find a steady balance between page cache and inode caches.
The moment memory fills, the page cache is reclaimed at a much faster rate than the inode cache, and evidence suggests taht the inode cache shrinker is not being called when large batches of pages are being reclaimed. In roughly the same time period that it takes to fill memory with 50% pages and 50% slab caches, memory reclaim reduces the page cache down to just dirty pages and slab caches fill the entirity of memory.
The LRU is largely full of dirty pages, and we're getting spikes of random writeback from memory reclaim so it's all going to shit. Behaviour never recovers, the page cache remains pinned at just dirty pages, and nothing I could tune would make any difference. vfs_cache_pressure makes no difference - I would it up so high it should trim the entire inode caches in a singel pass, yet it didn't do anything. It was clear from tracing and live telemetry that the shrinkers were pretty much not running except when there was absolutely no memory free at all, and then they did the minimum necessary to free memory to make progress.
So I went looking at the code, trying to find places where pages got reclaimed and the shrinkers weren't called. There's only one
- kswapd doing boosted reclaim as per commit 1c30844d2dfe ("mm:
reclaim small amounts of memory when an external fragmentation event occurs").
The watermark boosting introduced by the commit is triggered in response to an allocation "fragmentation event". The boosting was not intended to target THP specifically. However, with Dave's perfectly reasonable workload, fragmentation events can be very common given the ratio of slab to page cache allocations so boosting remains active for long periods of time.
As high-order allocations typically use compaction and compaction cannot move slab pages the decision was made in the commit to special-case kswapd when watermarks are boosted -- kswapd avoids reclaiming slab as reclaiming slab does not directly help compaction.
As Dave notes, this decision means that slab can be artificially protected for long periods of time and messes up the balance with slab and page caches.
Removing the special casing can still indirectly help fragmentation by avoiding fragmentation-causing events due to slab allocation as pages from a slab pageblock will have some slab objects freed. Furthermore, with the special casing, reclaim behaviour is unpredictable as kswapd sometimes examines slab and sometimes does not in a manner that is tricky to tune against and tricky to analyse.
This patch removes the special casing. The downside is that this is not a universal performance win. Some benchmarks that depend on the residency of data when rereading metadata may see a regression when slab reclaim is restored to its original behaviour. Similarly, some benchmarks that only read-once or write-once may perform better when page reclaim is too aggressive. The primary upside is that slab shrinker is less surprising (arguably more sane but that's a matter of opinion) and behaves consistently regardless of the fragmentation state of the system.
A fsmark benchmark configuration was constructed similar to what Dave reported and is codified by the mmtest configuration config-io-fsmark-small-file-stream. It was evaluated on a 1-socket machine to avoid dealing with NUMA-related issues and the timing of reclaim. The storage was an SSD Samsung Evo and a fresh XFS filesystem was used for the test data.
It is likely that the test configuration is not a proper match for Dave's test as the results are different in terms of performance. However, my configuration reports fsmark performance every 10% of memory worth of files and I suspect Dave's configuration reported Files/sec when memory was already full. THP was enabled for mine, disabled for Dave's and probably a whole load of other methodology differences that rarely get recorded properly.
fsmark
5.3.0-rc3 5.3.0-rc3
vanilla shrinker-v1r1
Min 1-files/sec 5181.70 ( 0.00%) 3204.20 ( -38.16%)
1st-qrtle 1-files/sec 14877.10 ( 0.00%) 6596.90 ( -55.66%)
2nd-qrtle 1-files/sec 6521.30 ( 0.00%) 5707.80 ( -12.47%)
3rd-qrtle 1-files/sec 5614.30 ( 0.00%) 5363.80 ( -4.46%)
Max-1 1-files/sec 18463.00 ( 0.00%) 18479.90 ( 0.09%)
Max-5 1-files/sec 18028.40 ( 0.00%) 17829.00 ( -1.11%)
Max-10 1-files/sec 17502.70 ( 0.00%) 17080.90 ( -2.41%)
Max-90 1-files/sec 5438.80 ( 0.00%) 5106.60 ( -6.11%)
Max-95 1-files/sec 5390.30 ( 0.00%) 5020.40 ( -6.86%)
Max-99 1-files/sec 5271.20 ( 0.00%) 3376.20 ( -35.95%)
Max 1-files/sec 18463.00 ( 0.00%) 18479.90 ( 0.09%)
Hmean 1-files/sec 7459.11 ( 0.00%) 6249.49 ( -16.22%)
Stddev 1-files/sec 4733.16 ( 0.00%) 4362.10 ( 7.84%)
CoeffVar 1-files/sec 51.66 ( 0.00%) 57.49 ( -11.29%)
BHmean-99 1-files/sec 7515.09 ( 0.00%) 6351.81 ( -15.48%)
BHmean-95 1-files/sec 7625.39 ( 0.00%) 6486.09 ( -14.94%)
BHmean-90 1-files/sec 7803.19 ( 0.00%) 6588.61 ( -15.57%)
BHmean-75 1-files/sec 8518.74 ( 0.00%) 6954.25 ( -18.37%)
BHmean-50 1-files/sec 10953.31 ( 0.00%) 8017.89 ( -26.80%)
BHmean-25 1-files/sec 16732.38 ( 0.00%) 11739.65 ( -29.84%) 5.3.0-rc3 5.3.0-rc3
vanillashrinker-v1r1
Duration User 77.29 89.09
Duration System 1097.13 1332.86
Duration Elapsed 2014.14 2596.39This is showing that fsmark runs slower as a result of this patch but there are other important observations that justify the patch.
1. With the vanilla kernel, the number of dirty pages in the system is very low for much of the test. With this patch, dirty pages is generally kept at 10% which matches vm.dirty_background_ratio which is normal expected historical behaviour.
2. With the vanilla kernel, the ratio of Slab/Pagecache is close to 0.95 for much of the test i.e. Slab is being left alone and dominating memory consumption. With the patch applied, the ratio varies between 0.35 and 0.45 with the bulk of the measured ratios roughly half way between those values. This is a different balance to what Dave reported but it was at least consistent.
3. Slabs are scanned throughout the entire test with the patch applied. The vanille kernel has long periods with no scan activity and then relatively massive spikes.
4. Overall vmstats are closer to normal expectations
5.3.0-rc3 5.3.0-rc3
vanilla shrinker-v1r1
Direct pages scanned 60308.00 5226.00
Kswapd pages scanned 18316110.00 12295574.00
Kswapd pages reclaimed 13121037.00 7280152.00
Direct pages reclaimed 11817.00 5226.00
Kswapd efficiency % 71.64 59.21
Kswapd velocity 9093.76 4735.64
Direct efficiency % 19.59 100.00
Direct velocity 29.94 2.01
Page reclaim immediate 247921.00 0.00
Slabs scanned 16602344.00 29369536.00
Direct inode steals 1574.00 800.00
Kswapd inode steals 130033.00 3968788.00
Kswapd skipped wait 0.00 0.00
o Vanilla kernel is hitting direct reclaim more frequently, not very much in absolute terms but the fact the patch reduces it is interesting
o "Page reclaim immediate" in the vanilla kernel indicates dirty pages are being encountered at the tail of the LRU. This is generally bad and means in this case that the LRU is not long enough for dirty pages to be cleaned by the background flush in time. This is eliminated by the patch
o With the patch, kswapd is reclaiming 30 times more slab pages than with the vanilla kernel. This is indicative of the watermark boosting over-protecting slab
A more complete set of tests is being run but the bottom line is that special casing kswapd to avoid slab behaviour is unpredictable and can lead to abnormal results for normal workloads. This patch restores the expected behaviour that slab and page cache is balanced consistently for a workload with a steady allocation ratio of slab/pagecache pages.
--- mm/vmscan.c | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-)
mm/vmscan.c +2 -11
@@ -88,9 +88,6 @@ struct scan_control {/* Can pages be swapped as part of reclaim? */ unsigned int may_swap:1; -/* e.g. boosted watermark reclaim leaves slabs alone */ - unsigned int may_shrinkslab:1; -/* * Cgroups are not reclaimed below their configured memory.low, * unless we threaten to OOM. If any cgroups are skipped due to
@@ -2714,10 +2711,8 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc)shrink_node_memcg (pgdat, memcg, sc, &lru_pages); node_lru_pages += lru_pages; -if (sc->may_shrinkslab) { -shrink_slab (sc->gfp_mask, pgdat->node_id, - memcg, sc->priority); - } +shrink_slab (sc->gfp_mask, pgdat->node_id, + memcg, sc->priority);/* Record the group's reclaim efficiency */ vmpressure(sc->gfp_mask, memcg, false,
@@ -3194,7 +3189,6 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order, .may_writepage = !laptop_mode, .may_unmap = 1, .may_swap = 1, - .may_shrinkslab = 1, };/*
@@ -3238,7 +3232,6 @@ unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg, .may_unmap = 1, .reclaim_idx = MAX_NR_ZONES - 1, .may_swap = !noswap, - .may_shrinkslab = 1, };unsigned long lru_pages;
@@ -3286,7 +3279,6 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, .may_writepage = !laptop_mode, .may_unmap = 1, .may_swap = may_swap, - .may_shrinkslab = 1, };set_task_reclaim_state (current, &sc.reclaim_state);
@@ -3598,7 +3590,6 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)*/ sc.may_writepage = !laptop_mode && !nr_boost_reclaim; sc.may_swap = !nr_boost_reclaim; - sc.may_shrinkslab = !nr_boost_reclaim;/* * Do some background aging of the anon list, to give
Quoted 18 lines · Mel Gorman
Quoted 11 lines · Mel Gorman
<SNIP>
On that basis, it may justify ripping out the
may_shrinkslablogic everywhere. The downside is that some microbenchmarks will notice. Specifically IO benchmarks that fill memory and reread (particularly rereading the metadata via any inode operation) may show reduced results. Such benchmarks can be strongly affected by whether the inode information is still memory resident and watermark boosting reduces the changes the data is still resident in memory. Technically still a regression but a tunable one.Quoted 3 lines
Hence the following "it builds" patch that has zero supporting data on whether it's a good idea or not.
This is a more complete version of the same patch that summaries the problem and includes data from my own testing
....
Quoted 6 lines
A fsmark benchmark configuration was constructed similar to what Dave reported and is codified by the mmtest configuration config-io-fsmark-small-file-stream. It was evaluated on a 1-socket machine to avoid dealing with NUMA-related issues and the timing of reclaim. The storage was an SSD Samsung Evo and a fresh XFS filesystem was used for the test data.
Have you run fstrim on that drive recently? I'm running these tests on a 960 EVO ssd, and when I started looking at shrinkers 3 weeks ago I had all sorts of whacky performance problems and inconsistent results. Turned out there were all sorts of random long IO latencies occurring (in the hundreds of milliseconds) because the drive was constantly running garbage collection to free up space. As a result it was both blocking on GC and thermal throttling under these fsmark workloads.
I made a new XFS filesystem on it (lazy man's rm -rf *), then ran fstrim on it to tell the drive all the space is free. Drive temps dropped 30C immediately, and all of the whacky performance anomolies went away. I now fstrim the drive in my vm startup scripts before each test run, and it's giving consistent results again.
Quoted 7 lines
It is likely that the test configuration is not a proper match for Dave's test as the results are different in terms of performance. However, my configuration reports fsmark performance every 10% of memory worth of files and I suspect Dave's configuration reported Files/sec when memory was already full. THP was enabled for mine, disabled for Dave's and probably a whole load of other methodology differences that rarely get recorded properly.
Yup, like I forgot to mention that my test system is using a 4-node fakenuma setup (i.e. 4 nodes, 4GB RAM and 4 CPUs per node, so there are 4 separate kswapd's doing concurrent reclaim). That changes reclaim patterns as well.
Quoted 24 lines
fsmark
5.3.0-rc3 5.3.0-rc3 vanilla shrinker-v1r1 Min 1-files/sec 5181.70 ( 0.00%) 3204.20 ( -38.16%) 1st-qrtle 1-files/sec 14877.10 ( 0.00%) 6596.90 ( -55.66%) 2nd-qrtle 1-files/sec 6521.30 ( 0.00%) 5707.80 ( -12.47%) 3rd-qrtle 1-files/sec 5614.30 ( 0.00%) 5363.80 ( -4.46%) Max-1 1-files/sec 18463.00 ( 0.00%) 18479.90 ( 0.09%) Max-5 1-files/sec 18028.40 ( 0.00%) 17829.00 ( -1.11%) Max-10 1-files/sec 17502.70 ( 0.00%) 17080.90 ( -2.41%) Max-90 1-files/sec 5438.80 ( 0.00%) 5106.60 ( -6.11%) Max-95 1-files/sec 5390.30 ( 0.00%) 5020.40 ( -6.86%) Max-99 1-files/sec 5271.20 ( 0.00%) 3376.20 ( -35.95%) Max 1-files/sec 18463.00 ( 0.00%) 18479.90 ( 0.09%) Hmean 1-files/sec 7459.11 ( 0.00%) 6249.49 ( -16.22%) Stddev 1-files/sec 4733.16 ( 0.00%) 4362.10 ( 7.84%) CoeffVar 1-files/sec 51.66 ( 0.00%) 57.49 ( -11.29%) BHmean-99 1-files/sec 7515.09 ( 0.00%) 6351.81 ( -15.48%) BHmean-95 1-files/sec 7625.39 ( 0.00%) 6486.09 ( -14.94%) BHmean-90 1-files/sec 7803.19 ( 0.00%) 6588.61 ( -15.57%) BHmean-75 1-files/sec 8518.74 ( 0.00%) 6954.25 ( -18.37%) BHmean-50 1-files/sec 10953.31 ( 0.00%) 8017.89 ( -26.80%) BHmean-25 1-files/sec 16732.38 ( 0.00%) 11739.65 ( -29.84%)
Quoted 5 lines
5.3.0-rc3 5.3.0-rc3
vanillashrinker-v1r1
Duration User 77.29 89.09
Duration System 1097.13 1332.86
Duration Elapsed 2014.14 2596.39I'm not sure we are testing or measuring exactly the same things :)
Quoted 8 lines
This is showing that fsmark runs slower as a result of this patch but there are other important observations that justify the patch.
1. With the vanilla kernel, the number of dirty pages in the system is very low for much of the test. With this patch, dirty pages is generally kept at 10% which matches vm.dirty_background_ratio which is normal expected historical behaviour.
Quoted 6 lines
2. With the vanilla kernel, the ratio of Slab/Pagecache is close to 0.95 for much of the test i.e. Slab is being left alone and dominating memory consumption. With the patch applied, the ratio varies between 0.35 and 0.45 with the bulk of the measured ratios roughly half way between those values. This is a different balance to what Dave reported but it was at least consistent.
Yeah, the balance is typically a bit different for different configs and storage. The trick is getting the balance to be roughly consistent across a range of different configs. The fakenuma setup also has a significant impact on where the balance is found. And I can't remember if the "fixed" memory usage numbers I quoted came from a run with my "make XFS inode reclaim nonblocking" patchset or not.
Quoted 6 lines
3. Slabs are scanned throughout the entire test with the patch applied. The vanille kernel has long periods with no scan activity and then relatively massive spikes.
4. Overall vmstats are closer to normal expectations
Quoted 15 lines
5.3.0-rc3 5.3.0-rc3 vanilla shrinker-v1r1 Direct pages scanned 60308.00 5226.00 Kswapd pages scanned 18316110.00 12295574.00 Kswapd pages reclaimed 13121037.00 7280152.00 Direct pages reclaimed 11817.00 5226.00 Kswapd efficiency % 71.64 59.21 Kswapd velocity 9093.76 4735.64 Direct efficiency % 19.59 100.00 Direct velocity 29.94 2.01 Page reclaim immediate 247921.00 0.00 Slabs scanned 16602344.00 29369536.00 Direct inode steals 1574.00 800.00 Kswapd inode steals 130033.00 3968788.00 Kswapd skipped wait 0.00 0.00
That looks a lot better. Patch looks reasonable, though I'm interested to know what impact it has on tests you ran in the original commit for the boosting.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 27 lines · Dave Chinner
Quoted 18 lines · Mel Gorman
Quoted 11 lines · Mel Gorman
<SNIP>
On that basis, it may justify ripping out the
may_shrinkslablogic everywhere. The downside is that some microbenchmarks will notice. Specifically IO benchmarks that fill memory and reread (particularly rereading the metadata via any inode operation) may show reduced results. Such benchmarks can be strongly affected by whether the inode information is still memory resident and watermark boosting reduces the changes the data is still resident in memory. Technically still a regression but a tunable one.Quoted 3 lines
Hence the following "it builds" patch that has zero supporting data on whether it's a good idea or not.
This is a more complete version of the same patch that summaries the problem and includes data from my own testing
....
Quoted 6 lines
A fsmark benchmark configuration was constructed similar to what Dave reported and is codified by the mmtest configuration config-io-fsmark-small-file-stream. It was evaluated on a 1-socket machine to avoid dealing with NUMA-related issues and the timing of reclaim. The storage was an SSD Samsung Evo and a fresh XFS filesystem was used for the test data.
Quoted 9 lines
Have you run fstrim on that drive recently? I'm running these tests on a 960 EVO ssd, and when I started looking at shrinkers 3 weeks ago I had all sorts of whacky performance problems and inconsistent results. Turned out there were all sorts of random long IO latencies occurring (in the hundreds of milliseconds) because the drive was constantly running garbage collection to free up space. As a result it was both blocking on GC and thermal throttling under these fsmark workloads.
No, I was under the impression that making a new filesystem typically trimmed it as well. Maybe that's just some filesystems (e.g. ext4) or just completely wrong.
Quoted 1 line
I made a new XFS filesystem on it (lazy man's rm -rf *),
Ah, all IO tests I do make a new filesystem. I know there is the whole problem of filesystem aging but I've yet to come across a methodology that two people can agree on that is a sensible, reproducible method.
Quoted 6 lines
then ran
fstrim on it to tell the drive all the space is free. Drive temps dropped 30C immediately, and all of the whacky performance anomolies went away. I now fstrim the drive in my vm startup scripts before each test run, and it's giving consistent results again.
I'll replicate that if making a new filesystem is not guaranteed to trim. It'll muck up historical data but that happens to me every so often anyway.
Quoted 8 lines
Quoted 7 lines
It is likely that the test configuration is not a proper match for Dave's test as the results are different in terms of performance. However, my configuration reports fsmark performance every 10% of memory worth of files and I suspect Dave's configuration reported Files/sec when memory was already full. THP was enabled for mine, disabled for Dave's and probably a whole load of other methodology differences that rarely get recorded properly.
Quoted 5 lines
Yup, like I forgot to mention that my test system is using a 4-node fakenuma setup (i.e. 4 nodes, 4GB RAM and 4 CPUs per node, so there are 4 separate kswapd's doing concurrent reclaim). That changes reclaim patterns as well.
Good to know. In this particular case, I don't think I need to exactly replicate what you have given that the slam reclaim behaviour is definitely more consistent and the ratios of slab/pagecache are predictable.
Quoted 25 lines
Quoted 25 lines
fsmark
5.3.0-rc3 5.3.0-rc3 vanilla shrinker-v1r1 Min 1-files/sec 5181.70 ( 0.00%) 3204.20 ( -38.16%) 1st-qrtle 1-files/sec 14877.10 ( 0.00%) 6596.90 ( -55.66%) 2nd-qrtle 1-files/sec 6521.30 ( 0.00%) 5707.80 ( -12.47%) 3rd-qrtle 1-files/sec 5614.30 ( 0.00%) 5363.80 ( -4.46%) Max-1 1-files/sec 18463.00 ( 0.00%) 18479.90 ( 0.09%) Max-5 1-files/sec 18028.40 ( 0.00%) 17829.00 ( -1.11%) Max-10 1-files/sec 17502.70 ( 0.00%) 17080.90 ( -2.41%) Max-90 1-files/sec 5438.80 ( 0.00%) 5106.60 ( -6.11%) Max-95 1-files/sec 5390.30 ( 0.00%) 5020.40 ( -6.86%) Max-99 1-files/sec 5271.20 ( 0.00%) 3376.20 ( -35.95%) Max 1-files/sec 18463.00 ( 0.00%) 18479.90 ( 0.09%) Hmean 1-files/sec 7459.11 ( 0.00%) 6249.49 ( -16.22%) Stddev 1-files/sec 4733.16 ( 0.00%) 4362.10 ( 7.84%) CoeffVar 1-files/sec 51.66 ( 0.00%) 57.49 ( -11.29%) BHmean-99 1-files/sec 7515.09 ( 0.00%) 6351.81 ( -15.48%) BHmean-95 1-files/sec 7625.39 ( 0.00%) 6486.09 ( -14.94%) BHmean-90 1-files/sec 7803.19 ( 0.00%) 6588.61 ( -15.57%) BHmean-75 1-files/sec 8518.74 ( 0.00%) 6954.25 ( -18.37%) BHmean-50 1-files/sec 10953.31 ( 0.00%) 8017.89 ( -26.80%) BHmean-25 1-files/sec 16732.38 ( 0.00%) 11739.65 ( -29.84%)
Quoted 8 lines
Quoted 5 lines
5.3.0-rc3 5.3.0-rc3 vanillashrinker-v1r1 Duration User 77.29 89.09 Duration System 1097.13 1332.86 Duration Elapsed 2014.14 2596.39I'm not sure we are testing or measuring exactly the same things :)
Probably not.
Quoted 15 lines
Quoted 8 lines
This is showing that fsmark runs slower as a result of this patch but there are other important observations that justify the patch.
1. With the vanilla kernel, the number of dirty pages in the system is very low for much of the test. With this patch, dirty pages is generally kept at 10% which matches vm.dirty_background_ratio which is normal expected historical behaviour.
Quoted 6 lines
2. With the vanilla kernel, the ratio of Slab/Pagecache is close to 0.95 for much of the test i.e. Slab is being left alone and dominating memory consumption. With the patch applied, the ratio varies between 0.35 and 0.45 with the bulk of the measured ratios roughly half way between those values. This is a different balance to what Dave reported but it was at least consistent.
Quoted 8 lines
Yeah, the balance is typically a bit different for different configs and storage. The trick is getting the balance to be roughly consistent across a range of different configs. The fakenuma setup also has a significant impact on where the balance is found. And I can't remember if the "fixed" memory usage numbers I quoted came from a run with my "make XFS inode reclaim nonblocking" patchset or not.
Again, I wouldn't sweat too much about it. The generated graphs definitely showed more consistent behaviour even if the headline performance was not improved.
Quoted 22 lines
Quoted 6 lines
3. Slabs are scanned throughout the entire test with the patch applied. The vanille kernel has long periods with no scan activity and then relatively massive spikes.
4. Overall vmstats are closer to normal expectations
Quoted 15 lines
5.3.0-rc3 5.3.0-rc3 vanilla shrinker-v1r1 Direct pages scanned 60308.00 5226.00 Kswapd pages scanned 18316110.00 12295574.00 Kswapd pages reclaimed 13121037.00 7280152.00 Direct pages reclaimed 11817.00 5226.00 Kswapd efficiency % 71.64 59.21 Kswapd velocity 9093.76 4735.64 Direct efficiency % 19.59 100.00 Direct velocity 29.94 2.01 Page reclaim immediate 247921.00 0.00 Slabs scanned 16602344.00 29369536.00 Direct inode steals 1574.00 800.00 Kswapd inode steals 130033.00 3968788.00 Kswapd skipped wait 0.00 0.00
Quoted 4 lines
That looks a lot better. Patch looks reasonable, though I'm interested to know what impact it has on tests you ran in the original commit for the boosting.
I'll find out soon enough but I'm leaning on the side that kswapd reclaim should be predictable and that even if there are some performance problems as a result of it, there will be others that see a gain. It'll be a case of "no matter what way you jump, someone shouts" but kswapd having spiky unpredictable behaviour is a recipe for "sometimes my machine is crap and I've no idea why".
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 38 lines · Mel Gorman
Quoted 27 lines · Dave Chinner
Quoted 18 lines · Mel Gorman
Quoted 11 lines · Mel Gorman
<SNIP>
On that basis, it may justify ripping out the
may_shrinkslablogic everywhere. The downside is that some microbenchmarks will notice. Specifically IO benchmarks that fill memory and reread (particularly rereading the metadata via any inode operation) may show reduced results. Such benchmarks can be strongly affected by whether the inode information is still memory resident and watermark boosting reduces the changes the data is still resident in memory. Technically still a regression but a tunable one.Quoted 3 lines
Hence the following "it builds" patch that has zero supporting data on whether it's a good idea or not.
This is a more complete version of the same patch that summaries the problem and includes data from my own testing
....
Quoted 6 lines
A fsmark benchmark configuration was constructed similar to what Dave reported and is codified by the mmtest configuration config-io-fsmark-small-file-stream. It was evaluated on a 1-socket machine to avoid dealing with NUMA-related issues and the timing of reclaim. The storage was an SSD Samsung Evo and a fresh XFS filesystem was used for the test data.
Quoted 9 lines
Have you run fstrim on that drive recently? I'm running these tests on a 960 EVO ssd, and when I started looking at shrinkers 3 weeks ago I had all sorts of whacky performance problems and inconsistent results. Turned out there were all sorts of random long IO latencies occurring (in the hundreds of milliseconds) because the drive was constantly running garbage collection to free up space. As a result it was both blocking on GC and thermal throttling under these fsmark workloads.
Quoted 3 lines
No, I was under the impression that making a new filesystem typically trimmed it as well. Maybe that's just some filesystems (e.g. ext4) or just completely wrong.
Depends. IIRC, some have turned that off by default because of the amount of poor implementations that take minutes to trim a whole device. XFS discards by default, but that doesn't mean it actually gets done. e.g. it might be on a block device that does not support or pass down discard requests.
FWIW, I run these in a VM on a sparse filesystem image (500TB) held in a file on the host XFS filesystem and:
$ cat /sys/block/vdc/queue/discard_max_bytes 0
Discard requests don't pass down through the virtio block device (nor do I really want them to). Hence I have to punch the image file and fstrim on the host side before launching the VM that runs the tests...
Quoted 10 lines
Quoted 6 lines
then ran
fstrim on it to tell the drive all the space is free. Drive temps dropped 30C immediately, and all of the whacky performance anomolies went away. I now fstrim the drive in my vm startup scripts before each test run, and it's giving consistent results again.I'll replicate that if making a new filesystem is not guaranteed to trim. It'll muck up historical data but that happens to me every so often anyway.
mkfs.xfs should be doing it if you're directly on top of the SSD. Just wanted to check seeing as I've recently been bitten by this.
Quoted 11 lines
Quoted 4 lines
That looks a lot better. Patch looks reasonable, though I'm interested to know what impact it has on tests you ran in the original commit for the boosting.
I'll find out soon enough but I'm leaning on the side that kswapd reclaim should be predictable and that even if there are some performance problems as a result of it, there will be others that see a gain. It'll be a case of "no matter what way you jump, someone shouts" but kswapd having spiky unpredictable behaviour is a recipe for "sometimes my machine is crap and I've no idea why".
Yeah, and that's precisely the motiviation for getting XFS inode reclaim to avoid blocking altogether and relying on memory reclaim to back off when appropriate. I expect there will be other problems I find with reclaim backoff and blance as a kick the tyres more...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
mm/vmscan.c static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) · line 2715
Quoted 6 lines
mm/vmscan.c +2
-4-if (sc->may_shrinkslab) { -shrink_slab (sc->gfp_mask, pgdat->node_id, - memcg, sc->priority); - } +shrink_slab (sc->gfp_mask, pgdat->node_id, + memcg, sc->priority);
Not the most useful comment, but the indentation for the continuing line is weird (already in the original code). This should be something like:
shrink_slab(sc->gfp_mask, pgdat->node_id, memcg, sc->priority);
Quoted 10 lines · Christoph Hellwig
mm/vmscan.c static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) · line 2715
Quoted 6 lines
mm/vmscan.c +2
-4-if (sc->may_shrinkslab) { -shrink_slab (sc->gfp_mask, pgdat->node_id, - memcg, sc->priority); - } +shrink_slab (sc->gfp_mask, pgdat->node_id, + memcg, sc->priority);Not the most useful comment, but the indentation for the continuing line is weird (already in the original code). This should be something like:
Quoted 2 lines
shrink_slab(sc->gfp_mask, pgdat->node_id, memcg, sc->priority);
If that's the worst you found then I take it as good news. I have not sent a version with an updated changelog so I can fix it up.
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 15 lines · Mel Gorman
Quoted 2 lines · Michal Hocko
[Cc Mel and Vlastimil as it seems like fallout from
1c30844d2dfe2]More than likely.
Quoted 8 lines · Dave Chinner
Quoted 7 lines
From: Dave Chinner <[email protected]>
When running a simple, steady state 4kB file creation test to simulate extracting tarballs larger than memory full of small files into the filesystem, I noticed that once memory fills up the cache balance goes to hell.
Quoted 5 lines
Ok, I'm assuming you are using fsmark with -k to keep files around, and -S0 to leave cleaning to the background flush, a number of files per iteration to get regular reporting and a total number of iterations to fill memory to hit what you're seeing. I've created a configuration that should do this but it'll take a long time to run on a local test machine.
./fs_mark -D 10000 -S0 -n 10000 -s 4096 -L 60 \
-d /mnt/scratch/0 -d /mnt/scratch/1 -d /mnt/scratch/2 \ -d /mnt/scratch/3 -d /mnt/scratch/4 -d /mnt/scratch/5 \ -d /mnt/scratch/6 -d /mnt/scratch/7 -d /mnt/scratch/8 \ -d /mnt/scratch/9 -d /mnt/scratch/10 -d /mnt/scratch/11 \
-d /mnt/scratch/12 -d /mnt/scratch/13 -d /mnt/scratch/14 \
-d /mnt/scratch/15
This doesn't delete files at all, creates 160,000 files per iteration in directories of 10,000 files at a time, and runs 60 iterations. It leaves all writeback (data and metadata) to background kernel mechanisms.
Quoted 6 lines
I'm not 100% certain I guessed right as to get fsmark reports while memory fills, it would have to be fewer files so each iteration would have to preserve files. If the number of files per iteration is large enough to fill memory then the drop in files/sec is not visible from the
fs_markoutput (or we are using different versions). I guess you could just be calculating average files/sec over the entire run based on elapsed time.
The files/s average is the average of the fsmark iteration output. (i.e. the rate at which it creates each group of 160k files).
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
The workload is creating one dirty cached inode for every dirty page, both of which should require a single IO each to clean and reclaim, and creation of inodes is throttled by the rate at which dirty writeback runs at (via balance dirty pages). Hence the ingest rate of new cached inodes and page cache pages is identical and steady. As a result, memory reclaim should quickly find a steady balance between page cache and inode caches.
It doesn't.
The moment memory fills, the page cache is reclaimed at a much faster rate than the inode cache, and evidence suggests taht the inode cache shrinker is not being called when large batches of pages are being reclaimed. In roughly the same time period that it takes to fill memory with 50% pages and 50% slab caches, memory reclaim reduces the page cache down to just dirty pages and slab caches fill the entirity of memory.
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
At the point where the page cache is reduced to just the dirty pages, there is a clear change in write IO patterns. Up to this point it has been running at a steady 1500 write IOPS for ~200MB/s of write throughtput (data, journal and metadata).
As observed by iostat -x or something else? Sum of r/s and w/s would
PCP + live pmcharts. Same as I've done for 15+ years :)
I could look at iostat, but it's much easier to watch graphs run and then be able to double click on any point and get the actual value.
I've attached a screen shot of the test machine overview while the vanilla kernel runs the fsmark test (cpu, iops, IO bandwidth, XFS create/remove/lookup ops, context switch rate and memory usage) at a 1 second sample rate. You can see the IO patterns change, the context switch rate go nuts and the CPU usage pattern change when the page cache hits empty.
Quoted 2 lines
approximate iops but not the breakdown of whether it is data, journal or metadata writes.
I have that in other charts - the log chart tells me how many log IOs are being done (constant 30MB/s in ~150 IOs/sec). And the >1GB/s IO spike every 30s is the metadata writeback being aggregated into large IOs by metadata writeback. That doesn't change, either...
Quoted 3 lines
The rest can be inferred from a blktrace but I would
prefer to replicate your setup as close as possible. If you're not usingfs_markto report Files/sec, are you simply monitoring df -i over time?
The way I run fsmark is iterative - the count field tells you how many inodes it has created...
Quoted 25 lines
Quoted 21 lines
Quoted 21 lines
So I went looking at the code, trying to find places where pages got reclaimed and the shrinkers weren't called. There's only one - kswapd doing boosted reclaim as per commit
1c30844d2dfe("mm: reclaim small amounts of memory when an external fragmentation event occurs"). I'm not even using THP or allocating huge pages, so this code should not be active or having any effect onmemory reclaim at all, yet the majority of reclaim is being done with "boost" and so it's not reclaiming slab caches at all. It will only free clean pages from the LRU.And so when we do run out memory, it switches to normal reclaim, which hits dirty pages on the LRU and does some shrinker work, too, but then appears to switch back to boosted reclaim one watermarks are reached.
The patch below restores page cache vs inode cache balance for this steady state workload. It balances out at about 40% page cache, 60% slab cache, and sustained performance is 10-15% higher than without this patch because the IO patterns remain in control of dirty writeback and the filesystem, not kswapd.
Quoted 3 lines
Quoted 3 lines
Performance with boosted reclaim also running shrinkers over the same steady state portion of the test as above.
Quoted 8 lines
The boosting was not intended to target THP specifically -- it was meant to help recover early from any fragmentation-related event for any user that might need it. Hence, it's not tied to THP but even with THP disabled, the boosting will still take effect.
One band-aid would be to disable watermark boosting entirely when THP is disabled but that feels wrong. However, I would be interested in hearing if sysctl vm.watermark_boost_factor=0 has the same effect as your patch.
<runs test>
Ok, it still runs it out of page cache, but it doesn't drive page cache reclaim as hard once there's none left. The IO patterns are less peaky, context switch rates are increased from ~3k/s to 15k/s but remain pretty steady.
Test ran 5s faster and file rate improved by ~2%. So it's better once the page cache is largerly fully reclaimed, but it doesn't prevent the page cache from being reclaimed instead of inodes....
Quoted 5 lines
On that basis, it may justify ripping out the
may_shrinkslablogic everywhere. The downside is that some microbenchmarks will notice. Specifically IO benchmarks that fill memory and reread (particularly rereading the metadata via any inode operation) may show reduced results.
Sure, but right now benchmarks that rely on page cache being retained are being screwed over :)
Quoted 4 lines
Such benchmarks can be strongly affected by whether the inode information is still memory resident and watermark boosting reduces the changes the data is still resident in memory. Technically still a regression but a tunable one.
/proc/sys/vm/vfs_cache_pressure is for tuning page cache/inode cache balance. It should not occur as a side effect of watermark boosting. Everyone knows about vfs_cache_pressure. Lots of people complain when it doesn't work, and that's something watermark boosting to change cache balance does.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 20 lines · Dave Chinner
Quoted 6 lines · Mel Gorman
Quoted 6 lines
Quoted 6 lines
inode cache shrinker is not being called when large batches of pages are being reclaimed. In roughly the same time period that it takes to fill memory with 50% pages and 50% slab caches, memory reclaim reduces the page cache down to just dirty pages and slab caches fill the entirity of memory.
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
At the point where the page cache is reduced to just the dirty pages, there is a clear change in write IO patterns. Up to this point it has been running at a steady 1500 write IOPS for ~200MB/s of write throughtput (data, journal and metadata).
As observed by iostat -x or something else? Sum of r/s and w/s would
PCP + live pmcharts. Same as I've done for 15+ years :)
I could look at iostat, but it's much easier to watch graphs run and then be able to double click on any point and get the actual value.
Quoted 6 lines
I've attached a screen shot of the test machine overview while the vanilla kernel runs the fsmark test (cpu, iops, IO bandwidth, XFS create/remove/lookup ops, context switch rate and memory usage) at a 1 second sample rate. You can see the IO patterns change, the context switch rate go nuts and the CPU usage pattern change when the page cache hits empty.
And now attached. :)
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 24 lines · Dave Chinner
Quoted 15 lines · Mel Gorman
Quoted 2 lines · Michal Hocko
[Cc Mel and Vlastimil as it seems like fallout from
1c30844d2dfe2]More than likely.
Quoted 8 lines · Dave Chinner
Quoted 7 lines
From: Dave Chinner <[email protected]>
When running a simple, steady state 4kB file creation test to simulate extracting tarballs larger than memory full of small files into the filesystem, I noticed that once memory fills up the cache balance goes to hell.
Quoted 5 lines
Ok, I'm assuming you are using fsmark with -k to keep files around, and -S0 to leave cleaning to the background flush, a number of files per iteration to get regular reporting and a total number of iterations to fill memory to hit what you're seeing. I've created a configuration that should do this but it'll take a long time to run on a local test machine.
./fs_mark -D 10000 -S0 -n 10000 -s 4096 -L 60 \
Review
Quoted 6 lines
(quoted diff) +0
-6-d /mnt/scratch/0 -d /mnt/scratch/1 -d /mnt/scratch/2 \ -d /mnt/scratch/3 -d /mnt/scratch/4 -d /mnt/scratch/5 \ -d /mnt/scratch/6 -d /mnt/scratch/7 -d /mnt/scratch/8 \ -d /mnt/scratch/9 -d /mnt/scratch/10 -d /mnt/scratch/11 \ -d /mnt/scratch/12 -d /mnt/scratch/13 -d /mnt/scratch/14 \ -d /mnt/scratch/15
Ok, I can replicate that. From a previous discussion my fs_mark setup is able to replicate this sort of thing relatively easily. I'll tweak the configuration.
Quoted 5 lines
This doesn't delete files at all, creates 160,000 files per iteration in directories of 10,000 files at a time, and runs 60 iterations. It leaves all writeback (data and metadata) to background kernel mechanisms.
Both our configurations left writeback to the background but I took a different approach to get more reported results from fsmark.
Quoted 10 lines
Quoted 6 lines
I'm not 100% certain I guessed right as to get fsmark reports while memory fills, it would have to be fewer files so each iteration would have to preserve files. If the number of files per iteration is large enough to fill memory then the drop in files/sec is not visible from the
fs_markoutput (or we are using different versions). I guess you could just be calculating average files/sec over the entire run based on elapsed time.The files/s average is the average of the fsmark iteration output. (i.e. the rate at which it creates each group of 160k files).
Good to confirm for sure.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
The workload is creating one dirty cached inode for every dirty page, both of which should require a single IO each to clean and reclaim, and creation of inodes is throttled by the rate at which dirty writeback runs at (via balance dirty pages). Hence the ingest rate of new cached inodes and page cache pages is identical and steady. As a result, memory reclaim should quickly find a steady balance between page cache and inode caches.
It doesn't.
The moment memory fills, the page cache is reclaimed at a much faster rate than the inode cache, and evidence suggests taht the inode cache shrinker is not being called when large batches of pages are being reclaimed. In roughly the same time period that it takes to fill memory with 50% pages and 50% slab caches, memory reclaim reduces the page cache down to just dirty pages and slab caches fill the entirity of memory.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
At the point where the page cache is reduced to just the dirty pages, there is a clear change in write IO patterns. Up to this point it has been running at a steady 1500 write IOPS for ~200MB/s of write throughtput (data, journal and metadata).
As observed by iostat -x or something else? Sum of r/s and w/s would
PCP + live pmcharts. Same as I've done for 15+ years :)
Yep. This is not the first time I thought I should wire PCP into mmtests. There are historical reasons why it didn't happen but they are getting less and less relevant.
Quoted 4 lines
I could look at iostat, but it's much easier to watch graphs run and then be able to double click on any point and get the actual value.
Can't argue with the logic.
Quoted 7 lines
I've attached a screen shot of the test machine overview while the vanilla kernel runs the fsmark test (cpu, iops, IO bandwidth, XFS create/remove/lookup ops, context switch rate and memory usage) at a 1 second sample rate. You can see the IO patterns change, the context switch rate go nuts and the CPU usage pattern change when the page cache hits empty.
Indeed.
Quoted 8 lines
Quoted 2 lines
approximate iops but not the breakdown of whether it is data, journal or metadata writes.
I have that in other charts - the log chart tells me how many log IOs are being done (constant 30MB/s in ~150 IOs/sec). And the >1GB/s IO spike every 30s is the metadata writeback being aggregated into large IOs by metadata writeback. That doesn't change, either...
Understood.
Quoted 7 lines
Quoted 3 lines
The rest can be inferred from a blktrace but I would
prefer to replicate your setup as close as possible. If you're not usingfs_markto report Files/sec, are you simply monitoring df -i over time?The way I run fsmark is iterative - the count field tells you how many inodes it has created...
Understood, I'll update tests accordingly.
Quoted 36 lines
Quoted 25 lines
Quoted 21 lines
Quoted 21 lines
So I went looking at the code, trying to find places where pages got reclaimed and the shrinkers weren't called. There's only one - kswapd doing boosted reclaim as per commit
1c30844d2dfe("mm: reclaim small amounts of memory when an external fragmentation event occurs"). I'm not even using THP or allocating huge pages, so this code should not be active or having any effect onmemory reclaim at all, yet the majority of reclaim is being done with "boost" and so it's not reclaiming slab caches at all. It will only free clean pages from the LRU.And so when we do run out memory, it switches to normal reclaim, which hits dirty pages on the LRU and does some shrinker work, too, but then appears to switch back to boosted reclaim one watermarks are reached.
The patch below restores page cache vs inode cache balance for this steady state workload. It balances out at about 40% page cache, 60% slab cache, and sustained performance is 10-15% higher than without this patch because the IO patterns remain in control of dirty writeback and the filesystem, not kswapd.
Quoted 3 lines
Quoted 3 lines
Performance with boosted reclaim also running shrinkers over the same steady state portion of the test as above.
Quoted 8 lines
The boosting was not intended to target THP specifically -- it was meant to help recover early from any fragmentation-related event for any user that might need it. Hence, it's not tied to THP but even with THP disabled, the boosting will still take effect.
One band-aid would be to disable watermark boosting entirely when THP is disabled but that feels wrong. However, I would be interested in hearing if sysctl vm.watermark_boost_factor=0 has the same effect as your patch.
<runs test>
Quoted 9 lines
Ok, it still runs it out of page cache, but it doesn't drive page cache reclaim as hard once there's none left. The IO patterns are less peaky, context switch rates are increased from ~3k/s to 15k/s but remain pretty steady.
Test ran 5s faster and file rate improved by ~2%. So it's better once the page cache is largerly fully reclaimed, but it doesn't prevent the page cache from being reclaimed instead of inodes....
Ok. Ideally you would also confirm the patch itself works as you want. It *should* but an actual confirmation would be nice.
Quoted 10 lines
Quoted 6 lines
On that basis, it may justify ripping out the
may_shrinkslablogic everywhere. The downside is that some microbenchmarks will notice. Specifically IO benchmarks that fill memory and reread (particularly rereading the metadata via any inode operation) may show reduced results.Sure, but right now benchmarks that rely on page cache being retained are being screwed over :)
Yeah. This is a perfect example of a change that will help some workloads and hurt others. However, as mentioned elsewhere, predictable behaviour is far easier to work with and it's closer to the historical behaviour.
Quoted 11 lines
Quoted 4 lines
Such benchmarks can be strongly affected by whether the inode information is still memory resident and watermark boosting reduces the changes the data is still resident in memory. Technically still a regression but a tunable one.
/proc/sys/vm/vfs_cache_pressureis for tuning page cache/inode cache balance. It should not occur as a side effect of watermark boosting. Everyone knows aboutvfs_cache_pressure. Lots of people complain when it doesn't work, and that's something watermark boosting to change cache balance does.
And this -- the fact that there is actually a tuning knob to deal with the tradeoffs is better than the current vanilla boost behaviour which has side-effects that cannot be tuned against. I'll update the changelog and resend the patch tomorrow assuming that it actually fixes your problem.
Thanks Dave (for both the report and the analysis that made this a no-brainer)
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 19 lines · Mel Gorman
Quoted 18 lines · Dave Chinner
Quoted 9 lines · Mel Gorman
On Wed, Aug 07, 2019 at 11:30:56AM +0200, Michal Hocko wrote:
The boosting was not intended to target THP specifically -- it was meant to help recover early from any fragmentation-related event for any user that might need it. Hence, it's not tied to THP but even with THP disabled, the boosting will still take effect.One band-aid would be to disable watermark boosting entirely when THP is disabled but that feels wrong. However, I would be interested in hearing if sysctl vm.watermark_boost_factor=0 has the same effect as your patch.
<runs test>
Ok, it still runs it out of page cache, but it doesn't drive page cache reclaim as hard once there's none left. The IO patterns are less peaky, context switch rates are increased from ~3k/s to 15k/s but remain pretty steady.
Quoted 7 lines
Quoted 4 lines
Test ran 5s faster and file rate improved by ~2%. So it's better once the page cache is largerly fully reclaimed, but it doesn't prevent the page cache from being reclaimed instead of inodes....
Ok. Ideally you would also confirm the patch itself works as you want. It *should* but an actual confirmation would be nice.
Yup, I'll get to that later today.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Dave Chinner
Reviewed-byQuoted 25 lines · Dave Chinner
Quoted 24 lines · Mel Gorman
Quoted 22 lines · Dave Chinner
Quoted 9 lines · Mel Gorman
On Wed, Aug 07, 2019 at 11:30:56AM +0200, Michal Hocko wrote:
The boosting was not intended to target THP specifically -- it was meant to help recover early from any fragmentation-related event for any user that might need it. Hence, it's not tied to THP but even with THP disabled, the boosting will still take effect.One band-aid would be to disable watermark boosting entirely when THP is disabled but that feels wrong. However, I would be interested in hearing if sysctl vm.watermark_boost_factor=0 has the same effect as your patch.
<runs test>
Ok, it still runs it out of page cache, but it doesn't drive page cache reclaim as hard once there's none left. The IO patterns are less peaky, context switch rates are increased from ~3k/s to 15k/s but remain pretty steady.
Test ran 5s faster and file rate improved by ~2%. So it's better once the page cache is largerly fully reclaimed, but it doesn't prevent the page cache from being reclaimed instead of inodes....
Quoted 4 lines
Quoted 2 lines
Ok. Ideally you would also confirm the patch itself works as you want. It *should* but an actual confirmation would be nice.
Yup, I'll get to that later today.
Looks good, does what it says on the tin.
Signature · 2 lines
Dave Chinner [email protected]