selftests/damon: test DAMOS tried regions and {min,max}_nr_regions
Subsystems (3)
SeongJae Park
maintainercover selftests/damon: test DAMOS tried regions and {min,max}_nr_regions
This patch series fix a minor issue in a program for DAMON selftest, and implement new functionality selftests for DAMOS tried regions and {min,max}_nr_regions. The test for max_nr_regions also test the recovery from online tuning-caused limit violation, which was fixed by a previous patch [1] titled "mm/damon/core: merge regions aggressively when max_nr_regions is unmet".
The first patch fixes a minor problem in the articial memory access pattern generator for tests. Following 3 patches (2-4) implement schemes tried regions test. Then a couple of patches (5-6) implementing static setup based {min,max}_nr_regions functionality test follows. Final two patches (7-8) implement dynamic max_nr_regions update test.
[1] https://lore.kernel.org/[email protected]
SeongJae Park (8):
selftests/damon/access_memory: use user-defined region size selftests/damon/_damon_sysfs: support schemes_update_tried_regions selftests/damon: implement a program for even-numbered memory regions
access
selftests/damon: implement DAMOS tried regions test selftests/damon/_damon_sysfs: implement kdamonds stop function selftests/damon: implement test for min/max_nr_regions _damon_sysfs: implement commit() for online parameters update selftests/damon/damon_nr_regions: test online-tuned max_nr_regions
tools/testing/selftests/damon/Makefile | 3 +- tools/testing/selftests/damon/_damon_sysfs.py | 65 +++++++- tools/testing/selftests/damon/access_memory.c | 2 +- .../selftests/damon/access_memory_even.c | 42 +++++ .../selftests/damon/damon_nr_regions.py | 145 ++++++++++++++++++ .../selftests/damon/damos_tried_regions.py | 65 ++++++++ 6 files changed, 319 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/damon/access_memory_even.c create mode 100644 tools/testing/selftests/damon/damon_nr_regions.py create mode 100644 tools/testing/selftests/damon/damos_tried_regions.py
base-commit: 99348045d11f3bac71146b381f90b0aa39855ee7
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 1/8 selftests/damon/access_memory: use user-defined region size
'access_memory' is an artificial memory access pattern generator for DAMON tests. It creates and accesses memory regions that the user specified the number and size via the command line. However, real access part of the program ignores the user-specified size of each region. Instead, it uses a hard-coded value, 10 MiB. Fix it to use user-defined size.
Note that all existing 'access_memory' users are setting the region size as 10 MiB. Hence no real problem has happened so far.
--- tools/testing/selftests/damon/access_memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/damon/access_memory.c +1 -1
@@ -35,7 +35,7 @@ int main(int argc, char *argv[]) start_clock =clock ();while ((clock () - start_clock) *1000 /CLOCKS_PER_SEC < access_time_ms) -memset (regions[i], i,1024 *1024 *10 ); +memset (regions[i], i, sz_region); }return 0 ; }
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 2/8 selftests/damon/_damon_sysfs: support schemes_update_tried_regions
In reply to SeongJae ParkImplement schemes_update_tried_regions DAMON sysfs command on _damon_sysfs.py, to use on implementations of future tests for the feature.
--- tools/testing/selftests/damon/_damon_sysfs.py | 35 ++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-)
tools/testing/selftests/damon/_damon_sysfs.py +34 -1
@@ -175,16 +175,24 @@ class DamosStats: self.sz_applied = sz_applied self.qt_exceeds = qt_exceeds +class DamosTriedRegion: + def __init__(self, start, end, nr_accesses, age): + self.start = start + self.end = end + self.nr_accesses = nr_accesses + self.age = age + class Damos: action = None access_pattern = None quota = None apply_interval_us = None - # todo: Support watermarks, stats, tried_regions + # todo: Support watermarks, stats idx = None context = None tried_bytes = None stats = None + tried_regions = None def __init__(self, action='stat', access_pattern=DamosAccessPattern(), quota=DamosQuota(), apply_interval_us=0):
@@ -398,6 +406,31 @@ class Kdamond: err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'on') return err + def update_schemes_tried_regions(self): + err = write_file(os.path.join(self.sysfs_dir(), 'state'), + 'update_schemes_tried_regions') + if err is not None: + return err + for context in self.contexts: + for scheme in context.schemes: + tried_regions = [] + tried_regions_dir = os.path.join( + scheme.sysfs_dir(), 'tried_regions') + for filename in os.listdir( + os.path.join(scheme.sysfs_dir(), 'tried_regions')): + tried_region_dir = os.path.join(tried_regions_dir, filename) + if not os.path.isdir(tried_region_dir): + continue + region_values = [] + for f in ['start', 'end', 'nr_accesses', 'age']: + content, err = read_file( + os.path.join(tried_region_dir, f)) + if err is not None: + return err + region_values.append(int(content)) + tried_regions.append(DamosTriedRegion(*region_values)) + scheme.tried_regions = tried_regions + def update_schemes_tried_bytes(self): err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'update_schemes_tried_bytes')
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 3/8 selftests/damon: implement a program for even-numbered memory regions access
In reply to SeongJae ParkTo test schemes_tried_regions feature, we need to have a program having specific number of regions that having different access pattern. Existing artificial access pattern generator, 'access_memory', cannot be used for the purpose, since it accesses only one region at a given time. Extending it could be an option, but since the purpose and the implementation are pretty simple, implementing another one from the scratch is better.
Implement such another artificial memory access program that alloctes user-defined number/size regions and accesses even-numbered regions.
--- tools/testing/selftests/damon/Makefile | 2 +- .../selftests/damon/access_memory_even.c | 42 +++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/damon/access_memory_even.c
tools/testing/selftests/damon/Makefile +1 -1
@@ -4,7 +4,7 @@ TEST_GEN_FILES += huge_count_read_write TEST_GEN_FILES += debugfs_target_ids_read_before_terminate_race TEST_GEN_FILES += debugfs_target_ids_pid_leak -TEST_GEN_FILES += access_memory +TEST_GEN_FILES += access_memory access_memory_even TEST_FILES = _chk_dependency.sh _debugfs_common.sh
tools/testing/selftests/damon/access_memory_even.c new file +42 -0
42 lines not on this page · raw message
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 4/8 selftests/damon: implement DAMOS tried regions test
In reply to SeongJae ParkImplement a test for DAMOS tried regions command of DAMON sysfs interface. It ensures the expected number of monitoring regions are created using an artificial memory access pattern generator program.
--- tools/testing/selftests/damon/Makefile | 1 + .../selftests/damon/damos_tried_regions.py | 65 +++++++++++++++++++ 2 files changed, 66 insertions(+) create mode 100644 tools/testing/selftests/damon/damos_tried_regions.py
tools/testing/selftests/damon/Makefile +1 -0
@@ -13,6 +13,7 @@ TEST_PROGS = debugfs_attrs.sh debugfs_schemes.sh debugfs_target_ids.sh TEST_PROGS += sysfs.sh TEST_PROGS += sysfs_update_schemes_tried_regions_wss_estimation.py TEST_PROGS += damos_quota.py damos_quota_goal.py damos_apply_interval.py +TEST_PROGS += damos_tried_regions.py TEST_PROGS += reclaim.sh lru_sort.sh # regression tests (reproducers of previously found bugs)
tools/testing/selftests/damon/damos_tried_regions.py new file +65 -0
65 lines not on this page · raw message
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 5/8 selftests/damon/_damon_sysfs: implement kdamonds stop function
In reply to SeongJae ParkImplement DAMON stop function on the test-purpose DAMON sysfs interface wrapper Python module, _damon_sysfs.py. This feature will be used by future DAMON tests that need to start/stop DAMON multiple times.
--- tools/testing/selftests/damon/_damon_sysfs.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
tools/testing/selftests/damon/_damon_sysfs.py +11 -0
@@ -406,6 +406,10 @@ class Kdamond: err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'on') return err + def stop(self): + err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'off') + return err + def update_schemes_tried_regions(self): err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'update_schemes_tried_regions')
@@ -511,3 +515,10 @@ class Kdamonds: if err is not None: return err return None + + def stop(self): + for kdamond in self.kdamonds: + err = kdamond.stop() + if err is not None: + return err + return None
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 6/8 selftests/damon: implement test for min/max_nr_regions
In reply to SeongJae ParkImplement a kselftest for DAMON's {min,max}_nr_regions' parameters. The test ensures both the minimum and the maximum number of regions limit is respected even if the workload's real number of regions is less than the minimum or larger than the maximum limits.
--- tools/testing/selftests/damon/Makefile | 2 +- .../selftests/damon/damon_nr_regions.py | 85 +++++++++++++++++++ 2 files changed, 86 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/damon/damon_nr_regions.py
tools/testing/selftests/damon/Makefile +1 -1
@@ -13,7 +13,7 @@ TEST_PROGS = debugfs_attrs.sh debugfs_schemes.sh debugfs_target_ids.sh TEST_PROGS += sysfs.sh TEST_PROGS += sysfs_update_schemes_tried_regions_wss_estimation.py TEST_PROGS += damos_quota.py damos_quota_goal.py damos_apply_interval.py -TEST_PROGS += damos_tried_regions.py +TEST_PROGS += damos_tried_regions.py damon_nr_regions.py TEST_PROGS += reclaim.sh lru_sort.sh # regression tests (reproducers of previously found bugs)
tools/testing/selftests/damon/damon_nr_regions.py new file +85 -0
85 lines not on this page · raw message
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 7/8 _damon_sysfs: implement commit() for online parameters update
In reply to SeongJae ParkUsers can update DAMON parameters while it is running, using 'commit' DAMON sysfs interface command. For testing the feature in future tests, implement a function for doing that on the test-purpose DAMON sysfs interface wrapper Python module.
--- tools/testing/selftests/damon/_damon_sysfs.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
tools/testing/selftests/damon/_damon_sysfs.py +19 -0
@@ -481,6 +481,25 @@ class Kdamond: goal.effective_bytes = int(content) return None + def commit(self): + nr_contexts_file = os.path.join(self.sysfs_dir(), + 'contexts', 'nr_contexts') + content, err = read_file(nr_contexts_file) + if err is not None: + return err + if int(content) != len(self.contexts): + err = write_file(nr_contexts_file, '%d' % len(self.contexts)) + if err is not None: + return err + + for context in self.contexts: + err = context.stage() + if err is not None: + return err + err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'commit') + return err + + def commit_schemes_quota_goals(self): for context in self.contexts: for scheme in context.schemes:
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 8/8 selftests/damon/damon_nr_regions: test online-tuned max_nr_regions
In reply to SeongJae ParkUser could update max_nr_regions parameter while DAMON is running to a value that smaller than the current number of regions that DAMON is seeing. Such update could be done for reducing the monitoring overhead. In the case, DAMON should merge regions aggressively more than normal situation to ensure the new limit is successfully applied. Implement a kselftest to ensure that.
--- .../selftests/damon/damon_nr_regions.py | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+)
tools/testing/selftests/damon/damon_nr_regions.py +60 -0
@@ -81,5 +81,65 @@ def main(): # test max_nr_regions smaller than real nr regions test_nr_regions(15, 3, 10) + # test online-tuned max_nr_regions that smaller than real nr regions + sz_region = 10 * 1024 * 1024 + proc = subprocess.Popen(['./access_memory_even', '14', '%d' % sz_region]) + + # stat every monitored regions + kdamonds = _damon_sysfs.Kdamonds([_damon_sysfs.Kdamond( + contexts=[_damon_sysfs.DamonCtx( + monitoring_attrs=_damon_sysfs.DamonAttrs( + min_nr_regions=10, max_nr_regions=1000), + ops='vaddr', + targets=[_damon_sysfs.DamonTarget(pid=proc.pid)], + schemes=[_damon_sysfs.Damos(action='stat', + )] # schemes + )] # contexts + )]) # kdamonds + + err = kdamonds.start() + if err is not None: + proc.terminate() + print('kdamond start failed: %s' % err) + exit(1) + + # wait until the real regions are found + time.sleep(3) + + attrs = kdamonds.kdamonds[0].contexts[0].monitoring_attrs + attrs.min_nr_regions = 3 + attrs.max_nr_regions = 7 + err = kdamonds.kdamonds[0].commit() + if err is not None: + proc.terminate() + print('commit failed: %s' % err) + exit(1) + # wait for next merge operation is executed + time.sleep(0.3) + + err = kdamonds.kdamonds[0].update_schemes_tried_regions() + if err is not None: + proc.terminate() + print('tried regions update failed: %s' % err) + exit(1) + + scheme = kdamonds.kdamonds[0].contexts[0].schemes[0] + if scheme.tried_regions is None: + proc.terminate() + print('tried regions is not collected') + exit(1) + + nr_tried_regions = len(scheme.tried_regions) + if nr_tried_regions <= 0: + proc.terminate() + print('tried regions is not created') + exit(1) + proc.terminate() + + if nr_tried_regions > 7: + print('fail online-tuned max_nr_regions: %d > 7' % nr_tried_regions) + exit(1) + print('pass online-tuned max_nr_regions') + if __name__ == '__main__': main()
Signature · 1 line
2.39.2