Skip to contentSkip to conversation

KSM: support smart-scan feature

PATCH Superseded Stefan Roesch 2 participants 10 messages Dec 1, 2023 – Dec 4, 2023 Memory management

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover KSM: support smart-scan feature

This patch series makes two changes:
- Disable the smart-scan feature for ksm01 - ksm04
This is necessary to make sure that the volatile metrics have the expected counts.
- Add a new test for the smart-scan feature
The new test verifies that the smart-scan feature skips pages when it has been enabled for a VMA.

Versions:
- V2:
- Disable smart_scan in the test setup structure
- Remove the changes in create_same_memory()
- Add the new testcase ksm07 for the smart scan test

Stefan Roesch (2):
mem: disable KSM smart scan for ksm tests
add ksm test for smart-scan feature

 testcases/kernel/mem/.gitignore    |  1 +
 testcases/kernel/mem/include/mem.h |  1 +
 testcases/kernel/mem/ksm/ksm01.c   |  2 +
 testcases/kernel/mem/ksm/ksm02.c   |  2 +
 testcases/kernel/mem/ksm/ksm03.c   |  2 +
 testcases/kernel/mem/ksm/ksm04.c   |  2 +
 testcases/kernel/mem/ksm/ksm07.c   | 69 +++++++++++++++++++++++++++++
 testcases/kernel/mem/lib/mem.c     | 70 ++++++++++++++++++++++++++++++
 8 files changed, 149 insertions(+)
 create mode 100644 testcases/kernel/mem/ksm/ksm07.c

base-commit: 8c89ef3d451087ed6e18750bd5eedd10e5ab3d2e

Signature · 1 line
2.39.3
#1 lore raw

PATCH v2 1/2 mem: disable KSM smart scan for ksm tests

This disables the "smart scan" KSM feature to make sure that the volatile count remains at 0.

---
 testcases/kernel/mem/ksm/ksm01.c | 2 ++
 testcases/kernel/mem/ksm/ksm02.c | 2 ++
 testcases/kernel/mem/ksm/ksm03.c | 2 ++
 testcases/kernel/mem/ksm/ksm04.c | 2 ++
 4 files changed, 8 insertions(+)
testcases/kernel/mem/ksm/ksm01.c +2 -0
index bcd095865..e2d3d9e00 100644
@@ -86,6 +86,8 @@ static struct tst_test test = {
 			TST_SR_SKIP_MISSING | TST_SR_TCONF_RO},
 		{"/sys/kernel/mm/ksm/merge_across_nodes", "1",
 			TST_SR_SKIP_MISSING | TST_SR_TCONF_RO},
+		{"/sys/kernel/mm/ksm/smart_scan", "0",
+			TST_SR_SKIP_MISSING | TST_SR_TBROK_RO},
 		{}
 	},
 	.needs_kconfigs = (const char *const[]){
testcases/kernel/mem/ksm/ksm02.c +2 -0
index bce639dce..3707de95d 100644
@@ -107,6 +107,8 @@ static struct tst_test test = {
 			TST_SR_SKIP_MISSING | TST_SR_TCONF_RO},
 		{"/sys/kernel/mm/ksm/merge_across_nodes", "1",
 			TST_SR_SKIP_MISSING | TST_SR_TCONF_RO},
+		{"/sys/kernel/mm/ksm/smart_scan", "0",
+			TST_SR_SKIP_MISSING | TST_SR_TBROK_RO},
 		{}
 	},
 	.needs_kconfigs = (const char *const[]){
testcases/kernel/mem/ksm/ksm03.c +2 -0
index 4a733269f..cff74700d 100644
@@ -89,6 +89,8 @@ static struct tst_test test = {
 			TST_SR_SKIP_MISSING | TST_SR_TCONF_RO},
 		{"/sys/kernel/mm/ksm/merge_across_nodes", "1",
 			TST_SR_SKIP_MISSING | TST_SR_TCONF_RO},
+		{"/sys/kernel/mm/ksm/smart_scan", "0",
+			TST_SR_SKIP_MISSING | TST_SR_TBROK_RO},
 		{}
 	},
 	.needs_kconfigs = (const char *const[]){
testcases/kernel/mem/ksm/ksm04.c +2 -0
index 4f1f2f721..9935e32d7 100644
@@ -109,6 +109,8 @@ static struct tst_test test = {
 			TST_SR_SKIP_MISSING | TST_SR_TCONF_RO},
 		{"/sys/kernel/mm/ksm/merge_across_nodes", "1",
 			TST_SR_SKIP_MISSING | TST_SR_TCONF_RO},
+		{"/sys/kernel/mm/ksm/smart_scan", "0",
+			TST_SR_SKIP_MISSING | TST_SR_TBROK_RO},
 		{}
 	},
 	.needs_kconfigs = (const char *const[]){
Signature · 1 line
2.39.3
Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] mem: disable KSM smart scan for ksm tests

Hi Stefan,

Quoted 5 lines

This disables the "smart scan" KSM feature to make sure that the volatile count remains at 0.

Quoted 7 lines
---
 testcases/kernel/mem/ksm/ksm01.c | 2 ++
 testcases/kernel/mem/ksm/ksm02.c | 2 ++
 testcases/kernel/mem/ksm/ksm03.c | 2 ++
 testcases/kernel/mem/ksm/ksm04.c | 2 ++

Li suggested in v1, that also ksm0[56].c should disable smart_scan (ksm05.c is questionable, but since you prepared ksm07 it can IMHO be disable in both ksm0[56].c).

https://lore.kernel.org/ltp/CAEemH2fqamX720diM1N+iN9a8HM30_5sHg8V0EMHgHdrh3iZPw@mail.gmail.com/

With that, you can add:

Kind regards,
Petr

#3 lore raw

Re: [PATCH v2 1/2] mem: disable KSM smart scan for ksm tests

Quoted 17 lines · Petr Vorel

Hi Stefan,

Quoted 5 lines

This disables the "smart scan" KSM feature to make sure that the volatile count remains at 0.

Quoted 7 lines
---
 testcases/kernel/mem/ksm/ksm01.c | 2 ++
 testcases/kernel/mem/ksm/ksm02.c | 2 ++
 testcases/kernel/mem/ksm/ksm03.c | 2 ++
 testcases/kernel/mem/ksm/ksm04.c | 2 ++

Li suggested in v1, that also ksm0[56].c should disable smart_scan (ksm05.c is questionable, but since you prepared ksm07 it can IMHO be disable in both ksm0[56].c).

Quoted 7 lines

https://lore.kernel.org/ltp/CAEemH2fqamX720diM1N+iN9a8HM30_5sHg8V0EMHgHdrh3iZPw@mail.gmail.com/

With that, you can add:

Kind regards,
Petr

The next version will also disable smart scan for ksm05 and ksm06

#4 lore raw

Re: [PATCH v2 1/2] mem: disable KSM smart scan for ksm tests

In reply to Petr Vorel
Quoted 17 lines · Petr Vorel

Hi Stefan,

Quoted 5 lines

This disables the "smart scan" KSM feature to make sure that the volatile count remains at 0.

Quoted 7 lines
---
 testcases/kernel/mem/ksm/ksm01.c | 2 ++
 testcases/kernel/mem/ksm/ksm02.c | 2 ++
 testcases/kernel/mem/ksm/ksm03.c | 2 ++
 testcases/kernel/mem/ksm/ksm04.c | 2 ++

Li suggested in v1, that also ksm0[56].c should disable smart_scan (ksm05.c is questionable, but since you prepared ksm07 it can IMHO be disable in both ksm0[56].c).

Quoted 7 lines

https://lore.kernel.org/ltp/CAEemH2fqamX720diM1N+iN9a8HM30_5sHg8V0EMHgHdrh3iZPw@mail.gmail.com/

With that, you can add:

Kind regards,
Petr

The next version will also disable smart scan for ksm05 and ksm06

#5 lore raw

PATCH v2 2/2 add ksm test for smart-scan feature

In reply to Stefan Roesch

This adds a new ksm (kernel samepage merging) test to evaluate the new smart scan feature. It allocates a page and fills it with 'a' characters. It captures the pages_skipped counter, waits for a few iterations and captures the pages_skipped counter again. The expectation is that over 50% of the page scans are skipped (There is only one page that has KSM enabled and it gets scanned during each iteration and it cannot be de-duplicated).

---
 testcases/kernel/mem/.gitignore    |  1 +
 testcases/kernel/mem/include/mem.h |  1 +
 testcases/kernel/mem/ksm/ksm07.c   | 69 +++++++++++++++++++++++++++++
 testcases/kernel/mem/lib/mem.c     | 70 ++++++++++++++++++++++++++++++
 4 files changed, 141 insertions(+)
 create mode 100644 testcases/kernel/mem/ksm/ksm07.c
testcases/kernel/mem/.gitignore +1 -0
index 7258489ed..c96fe8bfc 100644
@@ -53,6 +53,7 @@
 /ksm/ksm04
 /ksm/ksm05
 /ksm/ksm06
+/ksm/ksm07
 /mem/mem02
 /mmapstress/mmap-corruption01
 /mmapstress/mmapstress01
testcases/kernel/mem/include/mem.h +1 -0
index cdc3ca146..dbc3eb9ec 100644
@@ -48,6 +48,7 @@ void testoom(int mempolicy, int lite, int retcode, int allow_sigkill);
 /* KSM */
 
 void create_same_memory(int size, int num, int unit);
+void create_memory_for_smartscan(void);
 void test_ksm_merge_across_nodes(unsigned long nr_pages);
 void ksm_group_check(int run, int pg_shared, int pg_sharing, int pg_volatile,
                      int pg_unshared, int sleep_msecs, int pages_to_scan);
testcases/kernel/mem/ksm/ksm07.c new file +69 -0

69 lines not on this page · raw message

testcases/kernel/mem/lib/mem.c +70 -0

76 lines not on this page · raw message

Signature · 1 line
2.39.3
#6 lore raw

Re: [PATCH v2 2/2] add ksm test for smart-scan feature

Hi Stefan,

Quoted 4 lines
 testcases/kernel/mem/.gitignore    |  1 +
 testcases/kernel/mem/include/mem.h |  1 +
 testcases/kernel/mem/ksm/ksm07.c   | 69 +++++++++++++++++++++++++++++
 testcases/kernel/mem/lib/mem.c     | 70 ++++++++++++++++++++++++++++++

You need to add 'ksm07 ksm07' line to runtest/mm. If it's the only error, we can fix this before merge.

Kind regards,
Petr

#7 lore raw

Re: [PATCH v2 2/2] add ksm test for smart-scan feature

Quoted 6 lines · Petr Vorel

Hi Stefan,

Quoted 4 lines
 testcases/kernel/mem/.gitignore    |  1 +
 testcases/kernel/mem/include/mem.h |  1 +
 testcases/kernel/mem/ksm/ksm07.c   | 69 +++++++++++++++++++++++++++++
 testcases/kernel/mem/lib/mem.c     | 70 ++++++++++++++++++++++++++++++
Quoted 6 lines

You need to add 'ksm07 ksm07' line to runtest/mm. If it's the only error, we can fix this before merge.

Kind regards,
Petr

The next version will also update runtest/mm.

#8 lore raw

Re: [PATCH v2 2/2] add ksm test for smart-scan feature

In reply to Stefan Roesch

Hi Stefan,

...

Quoted 5 lines

+++ b/testcases/kernel/mem/ksm/ksm07.c

testcases/kernel/mem/ksm/ksm07.c new file +3 -0
@@ -0,0 +1,69 @@
+/*
+ * Copyright (C) 2010-2017  Red Hat, Inc.
+ *

testcases/kernel/mem/ksm/ksm07.c line 12

Quoted 10 lines
testcases/kernel/mem/ksm/ksm07.c new file +9 -0
 
+ * This program is free software;  you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY;  without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+ * the GNU General Public License for more details.

NOTE: we use SPDX instead of verbose GPL (see ksm06.c).

Quoted 1 line

+ *

NOTE: we have special doc format which starts like this (see ksm06.c):
/*\
* [Description]
*
* ...

Quoted 7 lines

+ * Kernel Samepage Merging (KSM)
+ *
+ * This adds a new ksm (kernel samepage merging) test to evaluate the new
+ * smart scan feature. It allocates a page and fills it with 'a'
+ * characters. It captures the pages_skipped counter, waits for a few
+ * iterations and captures the pages_skipped counter again. The expectation
+ * is that over 50% of the page scans are skipped (There is only one page

Quoted 8 lines

+ * that has KSM enabled and it gets scanned during each iteration and it
+ * cannot be de-duplicated).
+ *
+ * Prerequisites:
+ *
+ * 1) ksm and ksmtuned daemons need to be disabled. Otherwise, it could
+ * distrub the testing as they also change some ksm tunables depends
+ * on current workloads.

Hm, we don't have to any helper in LTP API to detect this, so at least we document this.

Quoted 2 lines

+ *
+ */

The result is then uploaded:
https://github.com/linux-test-project/ltp/releases/download/20230929/metadata.20230929.html

Therefore please use:

// SPDX-License-Identifier: GPL-2.0-or-later
/*
* Copyright (C) 2010-2017 Red Hat, Inc.
*/
/*\
* [Description]
*
* Kernel Samepage Merging (KSM)
*
* This adds a new ksm (kernel samepage merging) test to evaluate the new
* smart scan feature. It allocates a page and fills it with 'a'
* characters. It captures the pages_skipped counter, waits for a few
* iterations and captures the pages_skipped counter again. The expectation
* is that over 50% of the page scans are skipped (There is only one page
* that has KSM enabled and it gets scanned during each iteration and it
* cannot be de-duplicated).
*
* Prerequisites:
*
* 1) ksm and ksmtuned daemons need to be disabled. Otherwise, it could
* distrub the testing as they also change some ksm tunables depends
* on current workloads.
*/

Quoted 8 lines
testcases/kernel/mem/ksm/ksm07.c new file +8 -0
+
+#include <sys/types.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <signal.h>

testcases/kernel/mem/ksm/ksm07.c line 48

Quoted 10 lines
testcases/kernel/mem/ksm/ksm07.c new file +10 -0
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include "../include/mem.h"
+#include "ksm_common.h"
+
+static void verify_ksm(void)
+{
+	create_memory_for_smartscan();

I wonder, if we reusable create_memory_for_smartscan() later. Maybe it should be put into ksm07 for the start.

Also, the test needs to run on older kernel - exit with TCONF (which is not considered as a failure) instead of TBROK which does now:
mem.c:488: TBROK: Failed to open FILE '/sys/kernel/mm/ksm/pages_skipped' for reading: ENOENT (2)

If the testing code stays in testcases/kernel/mem/lib/mem.c, you will have to stat() it. But if it's really just this test specific and you move it to ksm07.c, then you can simply add the handling via .save_restore.

Quoted 11 lines
testcases/kernel/mem/ksm/ksm07.c new file +11 -0
+}
+
+static struct tst_test test = {
+	.needs_root = 1,
+	.forks_child = 1,
+	.options = (struct tst_option[]) {
+		{}
+	},
+	.save_restore = (const struct tst_path_val[]) {
+		{"/sys/kernel/mm/ksm/run", NULL, TST_SR_TBROK},
+		{"/sys/kernel/mm/ksm/sleep_millisecs", NULL, TST_SR_TBROK},

testcases/kernel/mem/ksm/ksm07.c line 69

Quoted 10 lines
testcases/kernel/mem/ksm/ksm07.c new file +10 -0
+		{"/sys/kernel/mm/ksm/smart_scan", "1",
+			TST_SR_SKIP_MISSING | TST_SR_TBROK_RO},
+		{}
+	},
+	.needs_kconfigs = (const char *const[]){
+		"CONFIG_KSM=y",
+		NULL
+	},
+	.test_all = verify_ksm,
+};

Also, there are missing static:

$ cd testcases/kernel/mem/ksm/; make check-ksm07
CHECK testcases/kernel/mem/ksm/ksm07.c
ksm07.c:1: WARNING: Missing or malformed SPDX-License-Identifier tag in line 1
make: [../../../../include/mk/rules.mk:56: check-ksm07] Error 1 (ignored)
ksm07.c: note: in included file:
ksm_common.h:14:5: warning: Symbol 'size' has no prototype or library ('tst_') prefix. Should it be static?
ksm_common.h:14:29: warning: Symbol 'num' has no prototype or library ('tst_') prefix. Should it be static?
ksm_common.h:14:38: warning: Symbol 'unit' has no prototype or library ('tst_') prefix. Should it be static?
ksm_common.h:15:6: warning: Symbol 'opt_sizestr' has no prototype or library ('tst_') prefix. Should it be static?
ksm_common.h:15:20: warning: Symbol 'opt_numstr' has no prototype or library ('tst_') prefix. Should it be static?
ksm_common.h:15:33: warning: Symbol 'opt_unitstr' has no prototype or library ('tst_') prefix. Should it be static?

Kind regards,
Petr

#9 lore raw

Re: [PATCH v2 2/2] add ksm test for smart-scan feature

Quoted 66 lines · Petr Vorel

Hi Stefan,

...

Quoted 5 lines

+++ b/testcases/kernel/mem/ksm/ksm07.c

testcases/kernel/mem/ksm/ksm07.c new file +3 -0
@@ -0,0 +1,69 @@
+/*
+ * Copyright (C) 2010-2017  Red Hat, Inc.
+ *

testcases/kernel/mem/ksm/ksm07.c line 12

Quoted 10 lines
testcases/kernel/mem/ksm/ksm07.c new file +9 -0
 
+ * This program is free software;  you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY;  without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+ * the GNU General Public License for more details.

NOTE: we use SPDX instead of verbose GPL (see ksm06.c).

Quoted 1 line

+ *

NOTE: we have special doc format which starts like this (see ksm06.c):
/*\
* [Description]
*
* ...

Quoted 7 lines

+ * Kernel Samepage Merging (KSM)
+ *
+ * This adds a new ksm (kernel samepage merging) test to evaluate the new
+ * smart scan feature. It allocates a page and fills it with 'a'
+ * characters. It captures the pages_skipped counter, waits for a few
+ * iterations and captures the pages_skipped counter again. The expectation
+ * is that over 50% of the page scans are skipped (There is only one page

Quoted 8 lines

+ * that has KSM enabled and it gets scanned during each iteration and it
+ * cannot be de-duplicated).
+ *
+ * Prerequisites:
+ *
+ * 1) ksm and ksmtuned daemons need to be disabled. Otherwise, it could
+ * distrub the testing as they also change some ksm tunables depends
+ * on current workloads.

Hm, we don't have to any helper in LTP API to detect this, so at least we document this.

Quoted 2 lines

+ *
+ */

The result is then uploaded:
https://github.com/linux-test-project/ltp/releases/download/20230929/metadata.20230929.html

Therefore please use:

// SPDX-License-Identifier: GPL-2.0-or-later
/*
* Copyright (C) 2010-2017 Red Hat, Inc.
*/
/*\
* [Description]
*
* Kernel Samepage Merging (KSM)
*
* This adds a new ksm (kernel samepage merging) test to evaluate the new
* smart scan feature. It allocates a page and fills it with 'a'
* characters. It captures the pages_skipped counter, waits for a few
* iterations and captures the pages_skipped counter again. The expectation
* is that over 50% of the page scans are skipped (There is only one page
* that has KSM enabled and it gets scanned during each iteration and it

Quoted 9 lines

* cannot be de-duplicated).
*
* Prerequisites:
*
* 1) ksm and ksmtuned daemons need to be disabled. Otherwise, it could
* distrub the testing as they also change some ksm tunables depends
* on current workloads.
*/

The next version will use the above comment as documentation.

Quoted 12 lines
Quoted 12 lines
testcases/kernel/mem/ksm/ksm07.c new file +12 -0
+
+#include <sys/types.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
Quoted 9 lines

testcases/kernel/mem/ksm/ksm07.c line 48

Quoted 6 lines
testcases/kernel/mem/ksm/ksm07.c new file +6 -0
+#include "../include/mem.h"
+#include "ksm_common.h"
+
+static void verify_ksm(void)
+{
+	create_memory_for_smartscan();

I wonder, if we reusable create_memory_for_smartscan() later. Maybe it should be put into ksm07 for the start.

I moved it over to ksm07.c and renamed it to create_memory.

Quoted 4 lines

Also, the test needs to run on older kernel - exit with TCONF (which is not considered as a failure) instead of TBROK which does now:
mem.c:488: TBROK: Failed to open FILE '/sys/kernel/mm/ksm/pages_skipped' for reading: ENOENT (2)

Changed it to TCONF.

Quoted 19 lines

If the testing code stays in testcases/kernel/mem/lib/mem.c, you will have to stat() it. But if it's really just this test specific and you move it to ksm07.c, then you can simply add the handling via .save_restore.

Quoted 15 lines
testcases/kernel/mem/ksm/ksm07.c new file +15 -0
+}
+
+static struct tst_test test = {
+	.needs_root = 1,
+	.forks_child = 1,
+	.options = (struct tst_option[]) {
+		{}
+	},
+	.save_restore = (const struct tst_path_val[]) {
+		{"/sys/kernel/mm/ksm/run", NULL, TST_SR_TBROK},
+		{"/sys/kernel/mm/ksm/sleep_millisecs", NULL, TST_SR_TBROK},
+		{"/sys/kernel/mm/ksm/smart_scan", "1",
+			TST_SR_SKIP_MISSING | TST_SR_TBROK_RO},
+		{}
+	},
Quoted 9 lines

testcases/kernel/mem/ksm/ksm07.c line 69

Quoted 6 lines
testcases/kernel/mem/ksm/ksm07.c new file +6 -0
+	.needs_kconfigs = (const char *const[]){
+		"CONFIG_KSM=y",
+		NULL
+	},
+	.test_all = verify_ksm,
+};

Also, there are missing static:

These are declared as non-static in ksm_common.h.

Quoted 17 lines
$ cd testcases/kernel/mem/ksm/; make check-ksm07
CHECK testcases/kernel/mem/ksm/ksm07.c
ksm07.c:1: WARNING: Missing or malformed SPDX-License-Identifier tag in line 1
make: [../../../../include/mk/rules.mk:56: check-ksm07] Error 1 (ignored)
ksm07.c: note: in included file:
ksm_common.h:14:5: warning: Symbol 'size' has no prototype or library ('tst_') prefix. Should it be static?
ksm_common.h:14:29: warning: Symbol 'num' has no prototype or library ('tst_') prefix. Should it be static?
ksm_common.h:14:38: warning: Symbol 'unit' has no prototype or library ('tst_') prefix. Should it be static?
ksm_common.h:15:6: warning: Symbol 'opt_sizestr' has no prototype or library
('tst_') prefix. Should it be static?
ksm_common.h:15:20: warning: Symbol 'opt_numstr' has no prototype or library
('tst_') prefix. Should it be static?
ksm_common.h:15:33: warning: Symbol 'opt_unitstr' has no prototype or library
('tst_') prefix. Should it be static?

Kind regards,
Petr