mm/damon: misc fixes and improvements
Subsystems (6)
SeongJae Park
maintainercover mm/damon: misc fixes and improvements
Add miscelleneous and non-urgent fixes and improvements for DAMON code, selftests, and documents.
SeongJae Park (10):
mm/damon/core: initialize ->esz_bp from damos_quota_init_priv() selftests/damon/_damon_sysfs: check errors from nr_schemes file reads selftests/damon/_damon_sysfs: find sysfs mount point from /proc/mounts selftests/damon/_damon_sysfs: use 'is' instead of '==' for 'None' selftests/damon: classify tests for functionalities and regressions Docs/admin-guide/mm/damon/usage: fix wrong example of DAMOS filter
matching sysfs file
Docs/admin-guide/mm/damon/usage: fix wrong schemes effective quota
update command
Docs/mm/damon/design: use a list for supported filters Docs/mm/damon/maintainer-profile: change the maintainer's timezone
from PST to PT
Docs/mm/damon/maintainer-profile: allow posting patches based on
damon/next tree
Documentation/admin-guide/mm/damon/usage.rst | 6 +- Documentation/mm/damon/design.rst | 46 +++++---- Documentation/mm/damon/maintainer-profile.rst | 13 +-- mm/damon/core.c | 1 + tools/testing/selftests/damon/Makefile | 13 ++- tools/testing/selftests/damon/_damon_sysfs.py | 95 +++++++++++-------- 6 files changed, 100 insertions(+), 74 deletions(-)
base-commit: fc7314cb6b750187a1366e0bf9da4c3ca8cfd064
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 1/10 mm/damon/core: initialize ->esz_bp from damos_quota_init_priv()
damos_quota_init_priv() function should initialize all private fields of struct damos_quota. However, it is not initializing ->esz_bp field. This could result in use of uninitialized variable from damon_feed_loop_next_input() function. There is no such issue at the moment because every caller of the function is passing damos_quota object that already having the field zero value. But we cannot guarantee the future, and the function is not doing what it is promising. A bug is a bug. This fix is for preventing possible future issues.
--- mm/damon/core.c | 1 + 1 file changed, 1 insertion(+)
mm/damon/core.c +1 -0
@@ -346,6 +346,7 @@ static struct damos_quota *damos_quota_init(struct damos_quota *quota) quota->charged_from =0 ; quota->charge_target_from =NULL ; quota->charge_addr_from =0 ; + quota->esz_bp =0 ;return quota; }
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 2/10 selftests/damon/_damon_sysfs: check errors from nr_schemes file reads
In reply to SeongJae ParkDAMON context staging method in _damon_sysfs.py is not checking the returned error from nr_schemes file read. Check it.
--- tools/testing/selftests/damon/_damon_sysfs.py | 2 ++ 1 file changed, 2 insertions(+)
tools/testing/selftests/damon/_damon_sysfs.py +2 -0
@@ -341,6 +341,8 @@ class DamonCtx: nr_schemes_file = os.path.join( self.sysfs_dir(), 'schemes', 'nr_schemes') content, err = read_file(nr_schemes_file) + if err is not None: + return err if int(content) != len(self.schemes): err = write_file(nr_schemes_file, '%d' % len(self.schemes)) if err != None:
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 3/10 selftests/damon/_damon_sysfs: find sysfs mount point from /proc/mounts
In reply to SeongJae Park_damon_sysfs.py assumes sysfs is mounted at /sys. In some systems, that might not be true. Find the mount point from /proc/mounts file content.
--- tools/testing/selftests/damon/_damon_sysfs.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
tools/testing/selftests/damon/_damon_sysfs.py +12 -1
@@ -2,7 +2,18 @@ import os -sysfs_root = '/sys/kernel/mm/damon/admin' +ksft_skip=4 + +sysfs_root = None +with open('/proc/mounts', 'r') as f: + for line in f: + dev_name, mount_point, dev_fs = line.split()[:3] + if dev_fs == 'sysfs': + sysfs_root = '%s/kernel/mm/damon/admin' % mount_point + break +if sysfs_root is None: + print('Seems sysfs not mounted?') + exit(ksft_skip) def write_file(path, string): "Returns error string if failed, or None otherwise"
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 4/10 selftests/damon/_damon_sysfs: use 'is' instead of '==' for 'None'
In reply to SeongJae Park_damon_sysfs.py is using '==' or '!=' for 'None'. Since 'None' is a singleton, using 'is' or 'is not' is more efficient. Use the more efficient one.
--- tools/testing/selftests/damon/_damon_sysfs.py | 80 +++++++++---------- 1 file changed, 40 insertions(+), 40 deletions(-)
tools/testing/selftests/damon/_damon_sysfs.py +40 -40
@@ -45,11 +45,11 @@ class DamosAccessPattern: self.nr_accesses = nr_accesses self.age = age - if self.size == None: + if self.size is None: self.size = [0, 2**64 - 1] - if self.nr_accesses == None: + if self.nr_accesses is None: self.nr_accesses = [0, 2**64 - 1] - if self.age == None: + if self.age is None: self.age = [0, 2**64 - 1] def sysfs_dir(self):
@@ -58,27 +58,27 @@ class DamosAccessPattern: def stage(self): err = write_file( os.path.join(self.sysfs_dir(), 'sz', 'min'), self.size[0]) - if err != None: + if err is not None: return err err = write_file( os.path.join(self.sysfs_dir(), 'sz', 'max'), self.size[1]) - if err != None: + if err is not None: return err err = write_file(os.path.join(self.sysfs_dir(), 'nr_accesses', 'min'), self.nr_accesses[0]) - if err != None: + if err is not None: return err err = write_file(os.path.join(self.sysfs_dir(), 'nr_accesses', 'max'), self.nr_accesses[1]) - if err != None: + if err is not None: return err err = write_file( os.path.join(self.sysfs_dir(), 'age', 'min'), self.age[0]) - if err != None: + if err is not None: return err err = write_file( os.path.join(self.sysfs_dir(), 'age', 'max'), self.age[1]) - if err != None: + if err is not None: return err qgoal_metric_user_input = 'user_input'
@@ -137,14 +137,14 @@ class DamosQuota: def stage(self): err = write_file(os.path.join(self.sysfs_dir(), 'bytes'), self.sz) - if err != None: + if err is not None: return err err = write_file(os.path.join(self.sysfs_dir(), 'ms'), self.ms) - if err != None: + if err is not None: return err err = write_file(os.path.join(self.sysfs_dir(), 'reset_interval_ms'), self.reset_interval_ms) - if err != None: + if err is not None: return err nr_goals_file = os.path.join(self.sysfs_dir(), 'goals', 'nr_goals')
@@ -201,30 +201,30 @@ class Damos: def stage(self): err = write_file(os.path.join(self.sysfs_dir(), 'action'), self.action) - if err != None: + if err is not None: return err err = self.access_pattern.stage() - if err != None: + if err is not None: return err err = write_file(os.path.join(self.sysfs_dir(), 'apply_interval_us'), '%d' % self.apply_interval_us) - if err != None: + if err is not None: return err err = self.quota.stage() - if err != None: + if err is not None: return err # disable watermarks err = write_file( os.path.join(self.sysfs_dir(), 'watermarks', 'metric'), 'none') - if err != None: + if err is not None: return err # disable filters err = write_file( os.path.join(self.sysfs_dir(), 'filters', 'nr_filters'), '0') - if err != None: + if err is not None: return err class DamonTarget:
@@ -243,7 +243,7 @@ class DamonTarget: def stage(self): err = write_file( os.path.join(self.sysfs_dir(), 'regions', 'nr_regions'), '0') - if err != None: + if err is not None: return err return write_file( os.path.join(self.sysfs_dir(), 'pid_target'), self.pid)
@@ -275,27 +275,27 @@ class DamonAttrs: def stage(self): err = write_file(os.path.join(self.interval_sysfs_dir(), 'sample_us'), self.sample_us) - if err != None: + if err is not None: return err err = write_file(os.path.join(self.interval_sysfs_dir(), 'aggr_us'), self.aggr_us) - if err != None: + if err is not None: return err err = write_file(os.path.join(self.interval_sysfs_dir(), 'update_us'), self.update_us) - if err != None: + if err is not None: return err err = write_file( os.path.join(self.nr_regions_range_sysfs_dir(), 'min'), self.min_nr_regions) - if err != None: + if err is not None: return err err = write_file( os.path.join(self.nr_regions_range_sysfs_dir(), 'max'), self.max_nr_regions) - if err != None: + if err is not None: return err class DamonCtx:
@@ -329,24 +329,24 @@ class DamonCtx: def stage(self): err = write_file( os.path.join(self.sysfs_dir(), 'operations'), self.ops) - if err != None: + if err is not None: return err err = self.monitoring_attrs.stage() - if err != None: + if err is not None: return err nr_targets_file = os.path.join( self.sysfs_dir(), 'targets', 'nr_targets') content, err = read_file(nr_targets_file) - if err != None: + if err is not None: return err if int(content) != len(self.targets): err = write_file(nr_targets_file, '%d' % len(self.targets)) - if err != None: + if err is not None: return err for target in self.targets: err = target.stage() - if err != None: + if err is not None: return err nr_schemes_file = os.path.join(
@@ -356,11 +356,11 @@ class DamonCtx: return err if int(content) != len(self.schemes): err = write_file(nr_schemes_file, '%d' % len(self.schemes)) - if err != None: + if err is not None: return err for scheme in self.schemes: err = scheme.stage() - if err != None: + if err is not None: return err return None
@@ -384,16 +384,16 @@ class Kdamond: nr_contexts_file = os.path.join(self.sysfs_dir(), 'contexts', 'nr_contexts') content, err = read_file(nr_contexts_file) - if err != None: + if err is not None: return err if int(content) != len(self.contexts): err = write_file(nr_contexts_file, '%d' % len(self.contexts)) - if err != None: + if err is not None: return err for context in self.contexts: err = context.stage() - if err != None: + if err is not None: return err err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'on') return err
@@ -401,20 +401,20 @@ class Kdamond: def update_schemes_tried_bytes(self): err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'update_schemes_tried_bytes') - if err != None: + if err is not None: return err for context in self.contexts: for scheme in context.schemes: content, err = read_file(os.path.join(scheme.sysfs_dir(), 'tried_regions', 'total_bytes')) - if err != None: + if err is not None: return err scheme.tried_bytes = int(content) def update_schemes_stats(self): err = write_file(os.path.join(self.sysfs_dir(), 'state'), 'update_schemes_stats') - if err != None: + if err is not None: return err for context in self.contexts: for scheme in context.schemes:
@@ -423,7 +423,7 @@ class Kdamond: 'sz_applied', 'qt_exceeds']: content, err = read_file( os.path.join(scheme.sysfs_dir(), 'stats', stat)) - if err != None: + if err is not None: return err stat_values.append(int(content)) scheme.stats = DamosStats(*stat_values)
@@ -471,10 +471,10 @@ class Kdamonds: def start(self): err = write_file(os.path.join(self.sysfs_dir(), 'nr_kdamonds'), '%s' % len(self.kdamonds)) - if err != None: + if err is not None: return err for kdamond in self.kdamonds: err = kdamond.start() - if err != None: + if err is not None: return err return None
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 5/10 selftests/damon: classify tests for functionalities and regressions
In reply to SeongJae ParkDAMON selftests can be classified into two categories: functionalities and regressions. Functionality tests are for checking if the function is working as specified, while the regression tests are basically reproducers of previously reported and fixed bugs. The tests of the categories are mixed in the selftests Makefile. Separate those for easier understanding of the types of tests.
--- tools/testing/selftests/damon/Makefile | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
tools/testing/selftests/damon/Makefile +9 -4
@@ -7,16 +7,21 @@ TEST_GEN_FILES += debugfs_target_ids_pid_leak TEST_GEN_FILES += access_memory TEST_FILES = _chk_dependency.sh _debugfs_common.sh + +# functionality tests TEST_PROGS = debugfs_attrs.sh debugfs_schemes.sh debugfs_target_ids.sh +TEST_PROGS += sysfs.sh +TEST_PROGS += sysfs_update_schemes_tried_regions_wss_estimation.py +TEST_PROGS += damos_quota.py damos_quota_goal.py damos_apply_interval.py +TEST_PROGS += reclaim.sh lru_sort.sh + +# regression tests (reproducers of previously found bugs) TEST_PROGS += debugfs_empty_targets.sh debugfs_huge_count_read_write.sh TEST_PROGS += debugfs_duplicate_context_creation.sh TEST_PROGS += debugfs_rm_non_contexts.sh TEST_PROGS += debugfs_target_ids_read_before_terminate_race.sh TEST_PROGS += debugfs_target_ids_pid_leak.sh -TEST_PROGS += sysfs.sh sysfs_update_removed_scheme_dir.sh +TEST_PROGS += sysfs_update_removed_scheme_dir.sh TEST_PROGS += sysfs_update_schemes_tried_regions_hang.py -TEST_PROGS += sysfs_update_schemes_tried_regions_wss_estimation.py -TEST_PROGS += damos_quota.py damos_quota_goal.py damos_apply_interval.py -TEST_PROGS += reclaim.sh lru_sort.sh include ../lib.mk
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 6/10 Docs/admin-guide/mm/damon/usage: fix wrong example of DAMOS filter matching sysfs file
In reply to SeongJae ParkThe example usage of DAMOS filter sysfs files, specifically the part of 'matching' file writing for memcg type filter, is wrong. The intention is to exclude pages of a memcg that already getting enough care from a given scheme, but the example is setting the filter to apply the scheme to only the pages of the memcg. Fix it.
--- Documentation/admin-guide/mm/damon/usage.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/admin-guide/mm/damon/usage.rst +1 -1
@@ -434,7 +434,7 @@ pages of all memory cgroups except ``/having_care_already``.:: # # further filter out all cgroups except one at '/having_care_already' echo memcg > 1/type echo /having_care_already > 1/memcg_path - echo N > 1/matching + echo Y > 1/matching Note that ``anon`` and ``memcg`` filters are currently supported only when ``paddr`` :ref:`implementation <sysfs_context>` is being used.
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 7/10 Docs/admin-guide/mm/damon/usage: fix wrong schemes effective quota update command
In reply to SeongJae ParkTo update effective size quota of DAMOS schemes on DAMON sysfs file interface, user should write 'update_schemes_effective_quotas' to the kdamond 'state' file. But the document is mistakenly saying the input string as 'update_schemes_effective_bytes'. Fix it (s/bytes/quotas/).
--- Documentation/admin-guide/mm/damon/usage.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/admin-guide/mm/damon/usage.rst +2 -2
@@ -153,7 +153,7 @@ Users can write below commands for the kdamond to the ``state`` file. - ``clear_schemes_tried_regions``: Clear the DAMON-based operating scheme action tried regions directory for each DAMON-based operation scheme of the kdamond. -- ``update_schemes_effective_bytes``: Update the contents of +- ``update_schemes_effective_quotas``: Update the contents of ``effective_bytes`` files for each DAMON-based operation scheme of the kdamond. For more details, refer to :ref:`quotas directory <sysfs_quotas>`.
@@ -342,7 +342,7 @@ Based on the user-specified :ref:`goal <sysfs_schemes_quota_goals>`, the effective size quota is further adjusted. Reading ``effective_bytes`` returns the current effective size quota. The file is not updated in real time, so users should ask DAMON sysfs interface to update the content of the file for -the stats by writing a special keyword, ``update_schemes_effective_bytes`` to +the stats by writing a special keyword, ``update_schemes_effective_quotas`` to the relevant ``kdamonds/<N>/state`` file. Under ``weights`` directory, three files (``sz_permil``,
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 8/10 Docs/mm/damon/design: use a list for supported filters
In reply to SeongJae ParkFilters section is listing currently supported filter types in a normal paragraph. Since the number of types are higher than four, it is not easy to read for only specific types. Use a list for easier finding of specific types.
--- Documentation/mm/damon/design.rst | 46 +++++++++++++++++-------------- 1 file changed, 26 insertions(+), 20 deletions(-)
Documentation/mm/damon/design.rst +26 -20
@@ -461,26 +461,32 @@ number of filters for each scheme. Each filter specifies the type of target memory, and whether it should exclude the memory of the type (filter-out), or all except the memory of the type (filter-in). -Currently, anonymous page, memory cgroup, young page, address range, and DAMON -monitoring target type filters are supported by the feature. Some filter -target types require additional arguments. The memory cgroup filter type asks -users to specify the file path of the memory cgroup for the filter. The -address range type asks the start and end addresses of the range. The DAMON -monitoring target type asks the index of the target from the context's -monitoring targets list. Hence, users can apply specific schemes to only -anonymous pages, non-anonymous pages, pages of specific cgroups, all pages -excluding those of specific cgroups, pages that not accessed after the last -access check from the scheme, pages that accessed after the last access check -from the scheme, pages in specific address range, pages in specific DAMON -monitoring targets, and any combination of those. - -To handle filters efficiently, the address range and DAMON monitoring target -type filters are handled by the core layer, while others are handled by -operations set. If a memory region is filtered by a core layer-handled filter, -it is not counted as the scheme has tried to the region. In contrast, if a -memory regions is filtered by an operations set layer-handled filter, it is -counted as the scheme has tried. The difference in accounting leads to changes -in the statistics. +For efficient handling of filters, some types of filters are handled by the +core layer, while others are handled by operations set. In the latter case, +hence, support of the filter types depends on the DAMON operations set. In +case of the core layer-handled filters, the memory regions that excluded by the +filter are not counted as the scheme has tried to the region. In contrast, if +a memory regions is filtered by an operations set layer-handled filter, it is +counted as the scheme has tried. This difference affects the statistics. + +Below types of filters are currently supported. + +- anonymous page + - Applied to pages that containing data that not stored in files. + - Handled by operations set layer. Supported by only ``paddr`` set. +- memory cgroup + - Applied to pages that belonging to a given cgroup. + - Handled by operations set layer. Supported by only ``paddr`` set. +- young page + - Applied to pages that are accessed after the last access check from the + scheme. + - Handled by operations set layer. Supported by only ``paddr`` set. +- address range + - Applied to pages that belonging to a given address range. + - Handled by the core logic. +- DAMON monitoring target + - Applied to pages that belonging to a given DAMON monitoring target. + - Handled by the core logic. Application Programming Interface
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 9/10 Docs/mm/damon/maintainer-profile: change the maintainer's timezone from PST to PT
In reply to SeongJae ParkThe document says the maintainer is working on only PST. The maintainer respects daylight saving system, though. Update the time zone to PT.
--- Documentation/mm/damon/maintainer-profile.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Documentation/mm/damon/maintainer-profile.rst +3 -3
@@ -48,9 +48,9 @@ Review cadence -------------- The DAMON maintainer does the work on the usual work hour (09:00 to 17:00, -Mon-Fri) in PST. The response to patches will occasionally be slow. Do not -hesitate to send a ping if you have not heard back within a week of sending a -patch. +Mon-Fri) in PT (Pacific Time). The response to patches will occasionally be +slow. Do not hesitate to send a ping if you have not heard back within a week +of sending a patch. .. [1] https://git.kernel.org/akpm/mm/h/mm-unstable
Signature · 1 line
2.39.2
SeongJae Park
maintainerPATCH 10/10 Docs/mm/damon/maintainer-profile: allow posting patches based on damon/next tree
In reply to SeongJae ParkThe document mentions any patches for review should based on mm-unstable instead of damon/next. It should be the recommended process, but sometimes patches based on damon/next could be posted for some reasons. Actually, the DAMON-based tiered memory management patchset[1] was written on top of 'young page' DAMOS filter patchset, which was in damon/next tree as of the writing.
Allow such case and just ask such things to be clearly specified.
[1] https://lore.kernel.org/[email protected]
--- Documentation/mm/damon/maintainer-profile.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
Documentation/mm/damon/maintainer-profile.rst +4 -3
@@ -20,9 +20,10 @@ management subsystem maintainer. After more sufficient tests, the patches will be queued in mm-stable [3]_ , and finally pull-requested to the mainline by the memory management subsystem maintainer. -Note again the patches for review should be made against the mm-unstable -tree [1]_ whenever possible. damon/next is only for preview of others' works -in progress. +Note again the patches for mm-unstable tree [1]_ are queued by the memory +management subsystem maintainer. If the patches requires some patches in +damon/next tree [2]_ which not yet merged in mm-unstable, please make sure the +requirement is clearly specified. Submit checklist addendum -------------------------
Signature · 1 line
2.39.2
SeongJae Park
maintainerAndrew, please add DAMON selftests patchset[1] that I posted yesterday before this patchset. Otherwise, patches would get conflicts.
[1] https://lore.kernel.org/[email protected]
Thanks,
SJ
Quoted 31 lines · On Fri, 3 May 2024 11:03:08 -0700 SeongJae Park <[email protected]> wrote:
Add miscelleneous and non-urgent fixes and improvements for DAMON code, selftests, and documents.
SeongJae Park (10):
mm/damon/core: initialize ->esz_bp fromdamos_quota_init_priv()selftests/damon/_damon_sysfs: check errors fromnr_schemesfile reads selftests/damon/_damon_sysfs: find sysfs mount point from/proc/mountsselftests/damon/_damon_sysfs: use 'is' instead of '==' for 'None' selftests/damon: classify tests for functionalities and regressions Docs/admin-guide/mm/damon/usage: fix wrong example of DAMOS filter
matching sysfs file
Docs/admin-guide/mm/damon/usage: fix wrong schemes effective quota
update command
Docs/mm/damon/design: use a list for supported filters Docs/mm/damon/maintainer-profile: change the maintainer's timezone
from PST to PT
Docs/mm/damon/maintainer-profile: allow posting patches based on
damon/next treeDocumentation/admin-guide/mm/damon/usage.rst | 6 +- Documentation/mm/damon/design.rst | 46 +++++---- Documentation/mm/damon/maintainer-profile.rst | 13 +-- mm/damon/core.c | 1 + tools/testing/selftests/damon/Makefile | 13 ++- tools/testing/selftests/damon/_damon_sysfs.py | 95 +++++++++++-------- 6 files changed, 100 insertions(+), 74 deletions(-)base-commit:
fc7314cb6b750187a1366e0bf9da4c3ca8cfd064Signature · 1 line
2.39.2