mm: vmscan: Reduce throttling due to a failure to make progress
Subsystems (7)
PATCH 1/1 mm: vmscan: Reduce throttling due to a failure to make progress
Mike Galbraith, Alexey Avramov and Darrick Wong all reported similar problems due to reclaim throttling for excessive lengths of time. In Alexey's case, a memory hog that should go OOM quickly stalls for several minutes before stalling. In Mike and Darrick's cases, a small memcg environment stalled excessively even though the system had enough memory overall.
Commit 69392a403f49 ("mm/vmscan: throttle reclaim when no progress is being made") introduced the problem although commit a19594ca4a8b ("mm/vmscan:
increase the timeout if page reclaim is not making progress") made it worse. Systems at or near an OOM state that cannot be recovered must reach OOM quickly and memcg should kill tasks if a memcg is near OOM.
To address this, only stall for the first zone in the zonelist, reduce the timeout to 1 tick for VMSCAN_THROTTLE_NOPROGRESS and only stall if the scan control nr_reclaimed is 0, kswapd is still active and there were excessive pages pending for writeback. If kswapd has stopped reclaiming due to excessive failures, do not stall at all so that OOM triggers relatively quickly. Similarly, if an LRU is simply congested, only lightly throttle similar to NOPROGRESS.
Alexey's original case was the most straight forward
for i in {1..3}; do tail /dev/zero; done
On vanilla 5.16-rc1, this test stalled heavily, after the patch the test completes in a few seconds similar to 5.15.
Alexey's second test case added watching a youtube video while tail runs 10 times. On 5.15, playback only jitters slightly, 5.16-rc1 stalls a lot with lots of frames missing and numerous audio glitches. With this patch applies, the video plays similarly to 5.15.
--- include/linux/mmzone.h | 1 + include/trace/events/vmscan.h | 4 ++- mm/vmscan.c | 64 ++++++++++++++++++++++++++++++----- 3 files changed, 59 insertions(+), 10 deletions(-)
include/linux/mmzone.h +1 -0
@@ -277,6 +277,7 @@ enum vmscan_throttle_state {VMSCAN_THROTTLE_WRITEBACK ,VMSCAN_THROTTLE_ISOLATED ,VMSCAN_THROTTLE_NOPROGRESS , +VMSCAN_THROTTLE_CONGESTED , NR_VMSCAN_THROTTLE, };
include/trace/events/vmscan.h +3 -1
@@ -30,12 +30,14 @@#define _VMSCAN_THROTTLE_WRITEBACK (1 << VMSCAN_THROTTLE_WRITEBACK)#define _VMSCAN_THROTTLE_ISOLATED (1 << VMSCAN_THROTTLE_ISOLATED)#define _VMSCAN_THROTTLE_NOPROGRESS (1 << VMSCAN_THROTTLE_NOPROGRESS) +#define _VMSCAN_THROTTLE_CONGESTED (1 << VMSCAN_THROTTLE_CONGESTED)#define show_throttle_flags (flags) \ (flags) ? __print_flags(flags, "|", \ {_VMSCAN_THROTTLE_WRITEBACK, "VMSCAN_THROTTLE_WRITEBACK"}, \ {_VMSCAN_THROTTLE_ISOLATED, "VMSCAN_THROTTLE_ISOLATED"}, \ - {_VMSCAN_THROTTLE_NOPROGRESS, "VMSCAN_THROTTLE_NOPROGRESS"} \ + {_VMSCAN_THROTTLE_NOPROGRESS, "VMSCAN_THROTTLE_NOPROGRESS"}, \ + {_VMSCAN_THROTTLE_CONGESTED, "VMSCAN_THROTTLE_CONGESTED"} \ ) : "VMSCAN_THROTTLE_NONE"
mm/vmscan.c +55 -9
112 lines not on this page · raw message
Signature · 1 line
2.31.1
kernel test robot
build botI love your patch! Perhaps something to improve:
Hi Mel,
I love your patch! Perhaps something to improve:
[auto build test WARNING on hnaz-mm/master]
url: https://github.com/0day-ci/linux/commits/Mel-Gorman/mm-vmscan-Reduce-throttling-due-to-a-failure-to-make-progress/20211202-212004 base: https://github.com/hnaz/linux-mm master config: um-i386_defconfig (https://download.01.org/0day-ci/archive/20211202/202112022232.gARZWe0c-lkp(a)intel.com/config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
# https://github.com/0day-ci/linux/commit/01dada07590ae9c69a9415ba9af96d5ae184d861
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Mel-Gorman/mm-vmscan-Reduce-throttling-due-to-a-failure-to-make-progress/20211202-212004 git checkout 01dada07590ae9c69a9415ba9af96d5ae184d861
# save the config file to linux build tree
mkdir build_dir
make W=1 O=build_dir ARCH=um SUBARCH=i386 SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
>> mm/vmscan.c:1024:6: warning: no previous prototype for 'skip_throttle_noprogress' [-Wmissing-prototypes]
1024 | bool skip_throttle_noprogress(pg_data_t *pgdat)
| ^~~~~~~~~~~~~~~~~~~~~~~~
vim +/skip_throttle_noprogress +1024 mm/vmscan.c
1023
> 1024 bool skip_throttle_noprogress(pg_data_t *pgdat)
1025 {
1026 int reclaimable = 0, write_pending = 0;
1027 int i;
1028
1029 /*
1030 * If kswapd is disabled, reschedule if necessary but do not
1031 * throttle as the system is likely near OOM.
1032 */
1033 if (pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES)
1034 return true;
1035
1036 /*
1037 * If there are a lot of dirty/writeback pages then do not
1038 * throttle as throttling will occur when the pages cycle
1039 * towards the end of the LRU if still under writeback.
1040 */
1041 for (i = 0; i < MAX_NR_ZONES; i++) {
1042 struct zone *zone = pgdat->node_zones + i;
1043
1044 if (!populated_zone(zone))
1045 continue;
1046
1047 reclaimable += zone_reclaimable_pages(zone);
1048 write_pending += zone_page_state_snapshot(zone,
1049 NR_ZONE_WRITE_PENDING);
1050 }
1051 if (2 * write_pending <= reclaimable)
1052 return true;
1053
1054 return false;
1055 }
1056
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
kernel test robot
build botI love your patch! Perhaps something to improve:
Hi Mel,
I love your patch! Perhaps something to improve:
[auto build test WARNING on hnaz-mm/master]
url: https://github.com/0day-ci/linux/commits/Mel-Gorman/mm-vmscan-Reduce-throttling-due-to-a-failure-to-make-progress/20211202-212004 base: https://github.com/hnaz/linux-mm master config: x86_64-buildonly-randconfig-r001-20211202 (https://download.01.org/0day-ci/archive/20211203/202112030001.HUiErCyK-lkp(a)intel.com/config) compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 4b553297ef3ee4dc2119d5429adf3072e90fac38)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/01dada07590ae9c69a9415ba9af96d5ae184d861
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Mel-Gorman/mm-vmscan-Reduce-throttling-due-to-a-failure-to-make-progress/20211202-212004 git checkout 01dada07590ae9c69a9415ba9af96d5ae184d861
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
>> mm/vmscan.c:1024:6: warning: no previous prototype for function 'skip_throttle_noprogress' [-Wmissing-prototypes]
bool skip_throttle_noprogress(pg_data_t *pgdat)
^ mm/vmscan.c:1024:1: note: declare 'static' if the function is not intended to be used outside of this translation unit bool skip_throttle_noprogress(pg_data_t *pgdat)
^
static
1 warning generated.
vim +/skip_throttle_noprogress +1024 mm/vmscan.c
1023
> 1024 bool skip_throttle_noprogress(pg_data_t *pgdat)
1025 {
1026 int reclaimable = 0, write_pending = 0;
1027 int i;
1028
1029 /*
1030 * If kswapd is disabled, reschedule if necessary but do not
1031 * throttle as the system is likely near OOM.
1032 */
1033 if (pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES)
1034 return true;
1035
1036 /*
1037 * If there are a lot of dirty/writeback pages then do not
1038 * throttle as throttling will occur when the pages cycle
1039 * towards the end of the LRU if still under writeback.
1040 */
1041 for (i = 0; i < MAX_NR_ZONES; i++) {
1042 struct zone *zone = pgdat->node_zones + i;
1043
1044 if (!populated_zone(zone))
1045 continue;
1046
1047 reclaimable += zone_reclaimable_pages(zone);
1048 write_pending += zone_page_state_snapshot(zone,
1049 NR_ZONE_WRITE_PENDING);
1050 }
1051 if (2 * write_pending <= reclaimable)
1052 return true;
1053
1054 return false;
1055 }
1056
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]