Skip to contentSkip to conversation

selftests/damon: test DAMOS tried regions and {min,max}_nr_regions

PATCH Applied SeongJae Park 1 participant 9 messages Jun 25, 2024

Subsystems (3)
lore
Share & tools
lore

SeongJae Park

author maintainer
#0 lore raw

cover selftests/damon: test DAMOS tried regions and {min,max}_nr_regions

This patch series fix a minor issue in a program for DAMON selftest, and implement new functionality selftests for DAMOS tried regions and {min,max}_nr_regions. The test for max_nr_regions also test the recovery from online tuning-caused limit violation, which was fixed by a previous patch [1] titled "mm/damon/core: merge regions aggressively when max_nr_regions is unmet".

The first patch fixes a minor problem in the articial memory access pattern generator for tests. Following 3 patches (2-4) implement schemes tried regions test. Then a couple of patches (5-6) implementing static setup based {min,max}_nr_regions functionality test follows. Final two patches (7-8) implement dynamic max_nr_regions update test.

[1] https://lore.kernel.org/[email protected]

SeongJae Park (8):
selftests/damon/access_memory: use user-defined region size selftests/damon/_damon_sysfs: support schemes_update_tried_regions selftests/damon: implement a program for even-numbered memory regions
access
selftests/damon: implement DAMOS tried regions test selftests/damon/_damon_sysfs: implement kdamonds stop function selftests/damon: implement test for min/max_nr_regions _damon_sysfs: implement commit() for online parameters update selftests/damon/damon_nr_regions: test online-tuned max_nr_regions

 tools/testing/selftests/damon/Makefile        |   3 +-
 tools/testing/selftests/damon/_damon_sysfs.py |  65 +++++++-
 tools/testing/selftests/damon/access_memory.c |   2 +-
 .../selftests/damon/access_memory_even.c      |  42 +++++
 .../selftests/damon/damon_nr_regions.py       | 145 ++++++++++++++++++
 .../selftests/damon/damos_tried_regions.py    |  65 ++++++++
 6 files changed, 319 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/damon/access_memory_even.c
 create mode 100644 tools/testing/selftests/damon/damon_nr_regions.py
 create mode 100644 tools/testing/selftests/damon/damos_tried_regions.py

base-commit: 99348045d11f3bac71146b381f90b0aa39855ee7

Signature · 1 line
2.39.2
author maintainer
#1 lore raw

PATCH 1/8 selftests/damon/access_memory: use user-defined region size

'access_memory' is an artificial memory access pattern generator for DAMON tests. It creates and accesses memory regions that the user specified the number and size via the command line. However, real access part of the program ignores the user-specified size of each region. Instead, it uses a hard-coded value, 10 MiB. Fix it to use user-defined size.

Note that all existing 'access_memory' users are setting the region size as 10 MiB. Hence no real problem has happened so far.

Fixes b5906f5f7359 ("selftests/damon: add a test for update_schemes_tried_regions sysfs command")
---
 tools/testing/selftests/damon/access_memory.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/damon/access_memory.c +1 -1
index 585a2fa54329..56b17e8fe1be 100644
@@ -35,7 +35,7 @@ int main(int argc, char *argv[])
 		start_clock = clock();
 		while ((clock() - start_clock) * 1000 / CLOCKS_PER_SEC <
 				access_time_ms)
-			memset(regions[i], i, 1024 * 1024 * 10);
+			memset(regions[i], i, sz_region);
 	}
 	return 0;
 }
Signature · 1 line
2.39.2
author maintainer
#2 lore raw

PATCH 2/8 selftests/damon/_damon_sysfs: support schemes_update_tried_regions

In reply to SeongJae Park

Implement schemes_update_tried_regions DAMON sysfs command on _damon_sysfs.py, to use on implementations of future tests for the feature.

---
 tools/testing/selftests/damon/_damon_sysfs.py | 35 ++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)
tools/testing/selftests/damon/_damon_sysfs.py +34 -1
index 2bd44c32be1b..f975742f29b3 100644
@@ -175,16 +175,24 @@ class DamosStats:
         self.sz_applied = sz_applied
         self.qt_exceeds = qt_exceeds
 
+class DamosTriedRegion:
+    def __init__(self, start, end, nr_accesses, age):
+        self.start = start
+        self.end = end
+        self.nr_accesses = nr_accesses
+        self.age = age
+
 class Damos:
     action = None
     access_pattern = None
     quota = None
     apply_interval_us = None
-    # todo: Support watermarks, stats, tried_regions
+    # todo: Support watermarks, stats
     idx = None
     context = None
     tried_bytes = None
     stats = None
+    tried_regions = None
 
     def __init__(self, action='stat', access_pattern=DamosAccessPattern(),
                  quota=DamosQuota(), apply_interval_us=0):
@@ -398,6 +406,31 @@ class Kdamond:
         err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'on')
         return err
 
+    def update_schemes_tried_regions(self):
+        err = write_file(os.path.join(self.sysfs_dir(), 'state'),
+                         'update_schemes_tried_regions')
+        if err is not None:
+            return err
+        for context in self.contexts:
+            for scheme in context.schemes:
+                tried_regions = []
+                tried_regions_dir = os.path.join(
+                        scheme.sysfs_dir(), 'tried_regions')
+                for filename in os.listdir(
+                        os.path.join(scheme.sysfs_dir(), 'tried_regions')):
+                    tried_region_dir = os.path.join(tried_regions_dir, filename)
+                    if not os.path.isdir(tried_region_dir):
+                        continue
+                    region_values = []
+                    for f in ['start', 'end', 'nr_accesses', 'age']:
+                        content, err = read_file(
+                                os.path.join(tried_region_dir, f))
+                        if err is not None:
+                            return err
+                        region_values.append(int(content))
+                    tried_regions.append(DamosTriedRegion(*region_values))
+                scheme.tried_regions = tried_regions
+
     def update_schemes_tried_bytes(self):
         err = write_file(os.path.join(self.sysfs_dir(), 'state'),
                 'update_schemes_tried_bytes')
Signature · 1 line
2.39.2
author maintainer
#3 lore raw

PATCH 3/8 selftests/damon: implement a program for even-numbered memory regions access

In reply to SeongJae Park

To test schemes_tried_regions feature, we need to have a program having specific number of regions that having different access pattern. Existing artificial access pattern generator, 'access_memory', cannot be used for the purpose, since it accesses only one region at a given time. Extending it could be an option, but since the purpose and the implementation are pretty simple, implementing another one from the scratch is better.

Implement such another artificial memory access program that alloctes user-defined number/size regions and accesses even-numbered regions.

---
 tools/testing/selftests/damon/Makefile        |  2 +-
 .../selftests/damon/access_memory_even.c      | 42 +++++++++++++++++++
 2 files changed, 43 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/damon/access_memory_even.c
tools/testing/selftests/damon/Makefile +1 -1
index 29a22f50e762..7b972b5cf487 100644
@@ -4,7 +4,7 @@
 TEST_GEN_FILES += huge_count_read_write
 TEST_GEN_FILES += debugfs_target_ids_read_before_terminate_race
 TEST_GEN_FILES += debugfs_target_ids_pid_leak
-TEST_GEN_FILES += access_memory
+TEST_GEN_FILES += access_memory access_memory_even
 
 TEST_FILES = _chk_dependency.sh _debugfs_common.sh
 
tools/testing/selftests/damon/access_memory_even.c new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
2.39.2
author maintainer
#4 lore raw

PATCH 4/8 selftests/damon: implement DAMOS tried regions test

In reply to SeongJae Park

Implement a test for DAMOS tried regions command of DAMON sysfs interface. It ensures the expected number of monitoring regions are created using an artificial memory access pattern generator program.

---
 tools/testing/selftests/damon/Makefile        |  1 +
 .../selftests/damon/damos_tried_regions.py    | 65 +++++++++++++++++++
 2 files changed, 66 insertions(+)
 create mode 100644 tools/testing/selftests/damon/damos_tried_regions.py
tools/testing/selftests/damon/Makefile +1 -0
index 7b972b5cf487..356b4e9a515d 100644
@@ -13,6 +13,7 @@ TEST_PROGS = debugfs_attrs.sh debugfs_schemes.sh debugfs_target_ids.sh
 TEST_PROGS += sysfs.sh
 TEST_PROGS += sysfs_update_schemes_tried_regions_wss_estimation.py
 TEST_PROGS += damos_quota.py damos_quota_goal.py damos_apply_interval.py
+TEST_PROGS += damos_tried_regions.py
 TEST_PROGS += reclaim.sh lru_sort.sh
 
 # regression tests (reproducers of previously found bugs)
tools/testing/selftests/damon/damos_tried_regions.py new file +65 -0

65 lines not on this page · raw message

Signature · 1 line
2.39.2
author maintainer
#5 lore raw

PATCH 5/8 selftests/damon/_damon_sysfs: implement kdamonds stop function

In reply to SeongJae Park

Implement DAMON stop function on the test-purpose DAMON sysfs interface wrapper Python module, _damon_sysfs.py. This feature will be used by future DAMON tests that need to start/stop DAMON multiple times.

---
 tools/testing/selftests/damon/_damon_sysfs.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
tools/testing/selftests/damon/_damon_sysfs.py +11 -0
index f975742f29b3..17352b9d204d 100644
@@ -406,6 +406,10 @@ class Kdamond:
         err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'on')
         return err
 
+    def stop(self):
+        err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'off')
+        return err
+
     def update_schemes_tried_regions(self):
         err = write_file(os.path.join(self.sysfs_dir(), 'state'),
                          'update_schemes_tried_regions')
@@ -511,3 +515,10 @@ class Kdamonds:
             if err is not None:
                 return err
         return None
+
+    def stop(self):
+        for kdamond in self.kdamonds:
+            err = kdamond.stop()
+            if err is not None:
+                return err
+        return None
Signature · 1 line
2.39.2
author maintainer
#6 lore raw

PATCH 6/8 selftests/damon: implement test for min/max_nr_regions

In reply to SeongJae Park

Implement a kselftest for DAMON's {min,max}_nr_regions' parameters. The test ensures both the minimum and the maximum number of regions limit is respected even if the workload's real number of regions is less than the minimum or larger than the maximum limits.

---
 tools/testing/selftests/damon/Makefile        |  2 +-
 .../selftests/damon/damon_nr_regions.py       | 85 +++++++++++++++++++
 2 files changed, 86 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/damon/damon_nr_regions.py
tools/testing/selftests/damon/Makefile +1 -1
index 356b4e9a515d..1e2e98cc809d 100644
@@ -13,7 +13,7 @@ TEST_PROGS = debugfs_attrs.sh debugfs_schemes.sh debugfs_target_ids.sh
 TEST_PROGS += sysfs.sh
 TEST_PROGS += sysfs_update_schemes_tried_regions_wss_estimation.py
 TEST_PROGS += damos_quota.py damos_quota_goal.py damos_apply_interval.py
-TEST_PROGS += damos_tried_regions.py
+TEST_PROGS += damos_tried_regions.py damon_nr_regions.py
 TEST_PROGS += reclaim.sh lru_sort.sh
 
 # regression tests (reproducers of previously found bugs)
tools/testing/selftests/damon/damon_nr_regions.py new file +85 -0

85 lines not on this page · raw message

Signature · 1 line
2.39.2
author maintainer
#7 lore raw

PATCH 7/8 _damon_sysfs: implement commit() for online parameters update

In reply to SeongJae Park

Users can update DAMON parameters while it is running, using 'commit' DAMON sysfs interface command. For testing the feature in future tests, implement a function for doing that on the test-purpose DAMON sysfs interface wrapper Python module.

---
 tools/testing/selftests/damon/_damon_sysfs.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
tools/testing/selftests/damon/_damon_sysfs.py +19 -0
index 17352b9d204d..6e136dc3df19 100644
@@ -481,6 +481,25 @@ class Kdamond:
                     goal.effective_bytes = int(content)
         return None
 
+    def commit(self):
+        nr_contexts_file = os.path.join(self.sysfs_dir(),
+                'contexts', 'nr_contexts')
+        content, err = read_file(nr_contexts_file)
+        if err is not None:
+            return err
+        if int(content) != len(self.contexts):
+            err = write_file(nr_contexts_file, '%d' % len(self.contexts))
+            if err is not None:
+                return err
+
+        for context in self.contexts:
+            err = context.stage()
+            if err is not None:
+                return err
+        err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'commit')
+        return err
+
+
     def commit_schemes_quota_goals(self):
         for context in self.contexts:
             for scheme in context.schemes:
Signature · 1 line
2.39.2
author maintainer
#8 lore raw

PATCH 8/8 selftests/damon/damon_nr_regions: test online-tuned max_nr_regions

In reply to SeongJae Park

User could update max_nr_regions parameter while DAMON is running to a value that smaller than the current number of regions that DAMON is seeing. Such update could be done for reducing the monitoring overhead. In the case, DAMON should merge regions aggressively more than normal situation to ensure the new limit is successfully applied. Implement a kselftest to ensure that.

---
 .../selftests/damon/damon_nr_regions.py       | 60 +++++++++++++++++++
 1 file changed, 60 insertions(+)
tools/testing/selftests/damon/damon_nr_regions.py +60 -0
index dd2735923c59..2e8a74aff543 100644
@@ -81,5 +81,65 @@ def main():
     # test max_nr_regions smaller than real nr regions
     test_nr_regions(15, 3, 10)
 
+    # test online-tuned max_nr_regions that smaller than real nr regions
+    sz_region = 10 * 1024 * 1024
+    proc = subprocess.Popen(['./access_memory_even', '14', '%d' % sz_region])
+
+    # stat every monitored regions
+    kdamonds = _damon_sysfs.Kdamonds([_damon_sysfs.Kdamond(
+            contexts=[_damon_sysfs.DamonCtx(
+                monitoring_attrs=_damon_sysfs.DamonAttrs(
+                    min_nr_regions=10, max_nr_regions=1000),
+                ops='vaddr',
+                targets=[_damon_sysfs.DamonTarget(pid=proc.pid)],
+                schemes=[_damon_sysfs.Damos(action='stat',
+                    )] # schemes
+                )] # contexts
+            )]) # kdamonds
+
+    err = kdamonds.start()
+    if err is not None:
+        proc.terminate()
+        print('kdamond start failed: %s' % err)
+        exit(1)
+
+    # wait until the real regions are found
+    time.sleep(3)
+
+    attrs = kdamonds.kdamonds[0].contexts[0].monitoring_attrs
+    attrs.min_nr_regions = 3
+    attrs.max_nr_regions = 7
+    err = kdamonds.kdamonds[0].commit()
+    if err is not None:
+        proc.terminate()
+        print('commit failed: %s' % err)
+        exit(1)
+    # wait for next merge operation is executed
+    time.sleep(0.3)
+
+    err = kdamonds.kdamonds[0].update_schemes_tried_regions()
+    if err is not None:
+        proc.terminate()
+        print('tried regions update failed: %s' % err)
+        exit(1)
+
+    scheme = kdamonds.kdamonds[0].contexts[0].schemes[0]
+    if scheme.tried_regions is None:
+        proc.terminate()
+        print('tried regions is not collected')
+        exit(1)
+
+    nr_tried_regions = len(scheme.tried_regions)
+    if nr_tried_regions <= 0:
+        proc.terminate()
+        print('tried regions is not created')
+        exit(1)
+    proc.terminate()
+
+    if nr_tried_regions > 7:
+        print('fail online-tuned max_nr_regions: %d > 7' % nr_tried_regions)
+        exit(1)
+    print('pass online-tuned max_nr_regions')
+
 if __name__ == '__main__':
     main()
Signature · 1 line
2.39.2