Skip to contentSkip to conversation

Hugetlb:Migrating the libhugetlbfs tests

PATCH Superseded Tarun Sahu 3 participants 16 messages Oct 29, 2022 – Nov 2, 2022 Library code

v3 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Tarun Sahu

author
#0 lore raw

v3 cover Hugetlb:Migrating the libhugetlbfs tests

Hi,
Libhugetlbfs is not being maintained actively, and some distro is dropping support for it. There are some tests that are good for testing hugetlb functionality in kernel.

As per previous dicussion in RFC[1], Here, this patch series consists of hugetlb tests taken from libhugetlbfs modified to work in ltp environment.

Based on suggestion[2], I am reposting the patches again. Also, I added suggested changes at [3].

ref:
1:https://lore.kernel.org/all/[email protected]/ 2:https://lore.kernel.org/all/[email protected]/ 3:https://lore.kernel.org/all/[email protected]/

v1 -> v2
1. In (brk near huge) test [PATCH 1/3] removed unused library function test_addr_huge() and read_maps().
v2 -> v3
1. Added a new patch commit for two new flags "needs_hugetlbfs" & "needs_unlinked_hugetlb_file" in tst_test and modified tests to use these flags.
2. Added taint check in test [PATCH 1/3].
3. Removed redundant Hopt and nr_opt option.
4. Corrected pre-processor ARCH based conditional flags in test [PATCH 1/3]

Tarun Sahu (4):
Hugetlb: Add new tst_test options for hugeltb test support
Hugetlb: Migrating libhugetlbfs brk_near_huge
Hugetlb: Migrating libhugetlbfs chunk-overcommit
Hugetlb: Migrating libhugetlbfs corrupt-by-cow-opt

 include/tst_test.h                            |  20 ++-
 lib/tst_test.c                                |  51 ++++++-
 runtest/hugetlb                               |   3 +
 testcases/kernel/mem/.gitignore               |   3 +
 .../kernel/mem/hugetlb/hugemmap/Makefile      |   5 +
 .../kernel/mem/hugetlb/hugemmap/hugemmap07.c  | 129 ++++++++++++++++
 .../kernel/mem/hugetlb/hugemmap/hugemmap08.c  | 144 ++++++++++++++++++
 .../kernel/mem/hugetlb/hugemmap/hugemmap09.c  |  71 +++++++++
 testcases/kernel/mem/hugetlb/lib/hugetlb.h    |   3 +
 9 files changed, 425 insertions(+), 4 deletions(-)
 create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c
 create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c
 create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c
Signature · 5 lines
2.31.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#1 lore raw

PATCH v3 1/4 Hugetlb: Add new tst_test options for hugeltb test support

Most of libhugetlbfs test require mounted hugetlbfs and random opened unlinked file for follow-up test actions.

Here, this patch adds two new field in tst_test struct(include/tst_test.h) which user can set if the test requires mounted hugetlbfs and other one for if test requires opened unlinked file.

---
 include/tst_test.h | 20 +++++++++++++++++-
 lib/tst_test.c     | 51 +++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 67 insertions(+), 4 deletions(-)
include/tst_test.h +19 -1
index a69965b95..f36382ae9 100644
@@ -131,7 +131,7 @@ struct tst_tag {
 };
 
 extern unsigned int tst_variant;
-
+extern int tst_hugetlb_fd;
 #define TST_NO_HUGEPAGES ((unsigned long)-1)
 
 #define TST_UNLIMITED_RUNTIME (-1)
@@ -176,6 +176,18 @@ struct tst_test {
 	int all_filesystems:1;
 	int skip_in_lockdown:1;
 	int skip_in_compat:1;
+	/*
+	 * If set, the test function will create a hugetlbfs mount point
+	 * at /tmp/xxxxxx, where xxxxxx is a random string.
+	 */
+	int needs_hugetlbfs:1;
+	/*
+	 * If set, the test function will create and open a random file
+	 * under mounted hugetlbfs. To use this option, needs_hugetlbfs must
+	 * be set. The file descriptior will be set in tst_hugetlb_fd.
+	 * The close(tst_hugetlb_fd) will be called on test exit(cleanup).
+	 */
+	int needs_unlinked_hugetlb_file:1;
 
 	/*
 	 * The skip_filesystems is a NULL terminated list of filesystems the
@@ -357,6 +369,12 @@ unsigned int tst_remaining_runtime(void);
  */
 void tst_set_max_runtime(int max_runtime);
 
+/*
+ * Create and open a random file inside the given dir path.
+ * It unlinks the file after opening and return file descriptor.
+ */
+int tst_create_unlinked_file(const char *path);
+
 /*
  * Returns path to the test temporary directory in a newly allocated buffer.
  */
lib/tst_test.c +48 -3

81 lines not on this page · raw message

Signature · 5 lines
2.31.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw

Re: [LTP] [PATCH v3 1/4] Hugetlb: Add new tst_test options for hugeltb test support

Hi Tarun,

This version is much better, comments are inline below.

Quoted 40 lines · Tarun Sahu

Most of libhugetlbfs test require mounted hugetlbfs and random opened unlinked file for follow-up test actions.

Here, this patch adds two new field in tst_test struct(include/tst_test.h) which user can set if the test requires mounted hugetlbfs and other one for if test requires opened unlinked file.

---
 include/tst_test.h | 20 +++++++++++++++++-
 lib/tst_test.c     | 51 +++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 67 insertions(+), 4 deletions(-)
include/tst_test.h +9 -1
index a69965b95..f36382ae9 100644
@@ -131,7 +131,7 @@ struct tst_tag {
 };
 
 extern unsigned int tst_variant;
-
+extern int tst_hugetlb_fd;
 #define TST_NO_HUGEPAGES ((unsigned long)-1)
 
 #define TST_UNLIMITED_RUNTIME (-1)
@@ -176,6 +176,18 @@ struct tst_test {
        int all_filesystems:1;
        int skip_in_lockdown:1;
        int skip_in_compat:1;
+       /*
+        * If set, the test function will create a hugetlbfs mount point
+        * at /tmp/xxxxxx, where xxxxxx is a random string.
+        */
+       int needs_hugetlbfs:1;
+       /*
+        * If set, the test function will create and open a random file
+        * under mounted hugetlbfs. To use this option, needs_hugetlbfs

must

include/tst_test.h struct tst_test { · line 190

Quoted 4 lines
include/tst_test.h +4 -0
+        * be set. The file descriptior will be set in tst_hugetlb_fd.
+        * The close(tst_hugetlb_fd) will be called on test exit(cleanup).
+        */
+       int needs_unlinked_hugetlb_file:1;

Why not consider encapsulating these two new fields in 'struct tst_hugepage' ?

Then the tst_test in the case can simply initialize to:

....
static struct tst_test test = {
.needs_root = 1,
.taint_check = TST_TAINT_D | TST_TAINT_W,
.setup = setup,
.test_all = run_test,
.hugepages = {1, TST_NEEDS, 1, 1},
};

Quoted 5 lines

/*
* The skip_filesystems is a NULL terminated list of filesystems
the

include/tst_test.h unsigned int tst_remaining_runtime(void); · line 377

Quoted 9 lines
include/tst_test.h +5 -0
@@ -357,6 +369,12 @@ unsigned int tst_remaining_runtime(void);
  */
 void tst_set_max_runtime(int max_runtime);
 
+/*
+ * Create and open a random file inside the given dir path.
+ * It unlinks the file after opening and return file descriptor.
+ */
+int tst_create_unlinked_file(const char *path);
 

what about renaming this function to tst_'get|create'_unlinked_file_fd? I guess the 'fd' part should be emphasized here.

Quoted 76 lines
include/tst_test.h +1 -0
+
 /*
  * Returns path to the test temporary directory in a newly allocated

buffer.
*/

lib/tst_test.c +26 -1
index 8ccde1629..43cba1004 100644
@@ -925,7 +925,8 @@ static int needs_tmpdir(void)
               tst_test->needs_device ||
               tst_test->mntpoint ||
               tst_test->resource_files ||
-              tst_test->needs_checkpoints;
+              tst_test->needs_checkpoints ||
+                  tst_test->needs_hugetlbfs;
 }
 
 static void copy_resources(void)
@@ -1021,6 +1022,30 @@ static void prepare_and_mount_dev_fs(const char
 *mntpoint)
        }
 }
 
+static void prepare_and_mount_hugetlb_fs(void)
+{
+       tst_test->mntpoint = tst_get_tmpdir();
+       SAFE_MOUNT("none", tst_test->mntpoint, "hugetlbfs", 0, NULL);
+       mntpoint_mounted = 1;
+}
+
+int tst_create_unlinked_file(const char *path)
+{
+       char template[PATH_MAX];
+       int fd;
+
+       snprintf(template, PATH_MAX, "%s/ltp_%.3sXXXXXX",
+                       path, TCID);
+
+       fd = mkstemp(template);
+       if (fd < 0)
+               tst_brk(TBROK | TERRNO,
+                        "%s: mkstemp(%s) failed", __func__, template);
+
+       SAFE_UNLINK(template);
+       return fd;
+}
+
 static const char *limit_tmpfs_mount_size(const char *mnt_data,
                char *buf, size_t buf_size, const char *fs_type)

{

lib/tst_test.c +8 -0
@@ -1094,6 +1119,8 @@ static void do_cgroup_requires(void)
        tst_cg_init();
 }
 
+int tst_hugetlb_fd = -1;
+
 static void do_setup(int argc, char *argv[])
 {
        if (!tst_test)
@@ -1217,6 +1244,17 @@ static void do_setup(int argc, char *argv[])
                }
        }
 
+       if (tst_test->needs_hugetlbfs)
+               prepare_and_mount_hugetlb_fs();
+
+       if (tst_test->needs_unlinked_hugetlb_file) {
+               if (!(tst_test->needs_hugetlbfs)) {
+                       tst_brk(TBROK, "Option needs_unlinked_hugetlb_file

"

lib/tst_test.c +3 -0
+                                       "requires option needs_hugetlbfs");
+               }
+               tst_hugetlb_fd =

tst_create_unlinked_file(tst_test->mntpoint);

lib/tst_test.c

Quoted 2 lines
lib/tst_test.c +2 -0
+       }
+

Seems we have to add a confliction check[1] to avoid multiple mounting at 'tst_test->mntpoint'. Or maybe go another method to move all the hugetlbfs
operations into tst_hugetlb.c to isolate details from the tst_test library, but
this will require more changes for all preexisting hugetlb tests.

[1] something like this:

@@ -1224,9 +1224,9 @@ static void do_setup(int argc, char *argv[])

        }

        if (!!tst_test->needs_rofs + !!tst_test->needs_devfs +

- !!tst_test->needs_device > 1) {
+ !!tst_test->needs_device + !!tst_test->needs_hugetlbfs > 1) {
tst_brk(TBROK,
- "Two or more of needs_{rofs, devfs, device} are
set");
+ "Two or more of needs_{rofs, devfs, device,

hugetlb} are set");
        }


        if (tst_test->needs_device && !mntpoint_mounted) {
Quoted 30 lines

tdev.dev = tst_acquire_device_(NULL,
tst_test->dev_min_size);

lib/tst_test.c +9 -3
@@ -1299,8 +1337,15 @@ static void do_cleanup(void)
        if (ovl_mounted)
                SAFE_UMOUNT(OVL_MNT);
 
-       if (mntpoint_mounted)
-               tst_umount(tst_test->mntpoint);
+       if (tst_hugetlb_fd >= 0)
+               SAFE_CLOSE(tst_hugetlb_fd);
+
+       if (mntpoint_mounted) {
+               if (tst_test->needs_hugetlbfs)
+                       SAFE_UMOUNT(tst_test->mntpoint);
+               else
+                       tst_umount(tst_test->mntpoint);
+       }
 
        if (tst_test->needs_device && tdev.dev)
                tst_release_device(tdev.dev);
--

2.31.1

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#3 lore raw

Re: [LTP] [PATCH v3 1/4] Hugetlb: Add new tst_test options for hugeltb test support

Hello,
Thanks for reviewing patch Li,

Quoted 56 lines · Li Wang

Hi Tarun,

This version is much better, comments are inline below.

Quoted 40 lines · Tarun Sahu

Most of libhugetlbfs test require mounted hugetlbfs and random opened unlinked file for follow-up test actions.

Here, this patch adds two new field in tst_test struct(include/tst_test.h) which user can set if the test requires mounted hugetlbfs and other one for if test requires opened unlinked file.

---
 include/tst_test.h | 20 +++++++++++++++++-
 lib/tst_test.c     | 51 +++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 67 insertions(+), 4 deletions(-)
include/tst_test.h +9 -1
index a69965b95..f36382ae9 100644
@@ -131,7 +131,7 @@ struct tst_tag {
 };
 
 extern unsigned int tst_variant;
-
+extern int tst_hugetlb_fd;
 #define TST_NO_HUGEPAGES ((unsigned long)-1)
 
 #define TST_UNLIMITED_RUNTIME (-1)
@@ -176,6 +176,18 @@ struct tst_test {
        int all_filesystems:1;
        int skip_in_lockdown:1;
        int skip_in_compat:1;
+       /*
+        * If set, the test function will create a hugetlbfs mount point
+        * at /tmp/xxxxxx, where xxxxxx is a random string.
+        */
+       int needs_hugetlbfs:1;
+       /*
+        * If set, the test function will create and open a random file
+        * under mounted hugetlbfs. To use this option, needs_hugetlbfs

must

include/tst_test.h struct tst_test { · line 190

Quoted 4 lines
include/tst_test.h +4 -0
+        * be set. The file descriptior will be set in tst_hugetlb_fd.
+        * The close(tst_hugetlb_fd) will be called on test exit(cleanup).
+        */
+       int needs_unlinked_hugetlb_file:1;

Why not consider encapsulating these two new fields in 'struct tst_hugepage' ?

Then the tst_test in the case can simply initialize to:

Quoted 9 lines

....
static struct tst_test test = {
.needs_root = 1,
.taint_check = TST_TAINT_D | TST_TAINT_W,
.setup = setup,
.test_all = run_test,
.hugepages = {1, TST_NEEDS, 1, 1},
};

Ok, I will move these fields in tst_hugepages struct.

Quoted 10 lines
Quoted 10 lines

/*
* The skip_filesystems is a NULL terminated list of filesystems
the

include/tst_test.h +0 -0
@@ -357,6 +369,12 @@ unsigned int tst_remaining_runtime(void);
  */
 void tst_set_max_runtime(int max_runtime);
 
Quoted 10 lines

include/tst_test.h unsigned int tst_remaining_runtime(void);

Quoted 6 lines
include/tst_test.h +5 -0
+/*
+ * Create and open a random file inside the given dir path.
+ * It unlinks the file after opening and return file descriptor.
+ */
+int tst_create_unlinked_file(const char *path);
 

what about renaming this function to tst_'get|create'_unlinked_file_fd? I guess the 'fd' part should be emphasized here.

OK. Will update it.

Quoted 94 lines
Quoted 78 lines
include/tst_test.h +1 -0
+
 /*
  * Returns path to the test temporary directory in a newly allocated

buffer.
*/

lib/tst_test.c +26 -1
index 8ccde1629..43cba1004 100644
@@ -925,7 +925,8 @@ static int needs_tmpdir(void)
               tst_test->needs_device ||
               tst_test->mntpoint ||
               tst_test->resource_files ||
-              tst_test->needs_checkpoints;
+              tst_test->needs_checkpoints ||
+                  tst_test->needs_hugetlbfs;
 }
 
 static void copy_resources(void)
@@ -1021,6 +1022,30 @@ static void prepare_and_mount_dev_fs(const char
 *mntpoint)
        }
 }
 
+static void prepare_and_mount_hugetlb_fs(void)
+{
+       tst_test->mntpoint = tst_get_tmpdir();
+       SAFE_MOUNT("none", tst_test->mntpoint, "hugetlbfs", 0, NULL);
+       mntpoint_mounted = 1;
+}
+
+int tst_create_unlinked_file(const char *path)
+{
+       char template[PATH_MAX];
+       int fd;
+
+       snprintf(template, PATH_MAX, "%s/ltp_%.3sXXXXXX",
+                       path, TCID);
+
+       fd = mkstemp(template);
+       if (fd < 0)
+               tst_brk(TBROK | TERRNO,
+                        "%s: mkstemp(%s) failed", __func__, template);
+
+       SAFE_UNLINK(template);
+       return fd;
+}
+
 static const char *limit_tmpfs_mount_size(const char *mnt_data,
                char *buf, size_t buf_size, const char *fs_type)

{

lib/tst_test.c +8 -0
@@ -1094,6 +1119,8 @@ static void do_cgroup_requires(void)
        tst_cg_init();
 }
 
+int tst_hugetlb_fd = -1;
+
 static void do_setup(int argc, char *argv[])
 {
        if (!tst_test)
@@ -1217,6 +1244,17 @@ static void do_setup(int argc, char *argv[])
                }
        }
 
+       if (tst_test->needs_hugetlbfs)
+               prepare_and_mount_hugetlb_fs();
+
+       if (tst_test->needs_unlinked_hugetlb_file) {
+               if (!(tst_test->needs_hugetlbfs)) {
+                       tst_brk(TBROK, "Option needs_unlinked_hugetlb_file

"

lib/tst_test.c +3 -0
+                                       "requires option needs_hugetlbfs");
+               }
+               tst_hugetlb_fd =

tst_create_unlinked_file(tst_test->mntpoint);

lib/tst_test.c

Quoted 2 lines
lib/tst_test.c +2 -0
+       }
+

Seems we have to add a confliction check[1] to avoid multiple mounting at 'tst_test->mntpoint'. Or maybe go another method to move all the hugetlbfs
operations into tst_hugetlb.c to isolate details from the tst_test library, but
this will require more changes for all preexisting hugetlb tests.

[1] something like this:

(quoted diff) +0 -0
@@ -1224,9 +1224,9 @@ static void do_setup(int argc, char *argv[])
        }

Review static void do_setup(int argc, char *argv[]) · line 1232

Quoted 10 lines
(quoted diff) +2 -2
 
        if (!!tst_test->needs_rofs + !!tst_test->needs_devfs +
-           !!tst_test->needs_device > 1) {
+           !!tst_test->needs_device + !!tst_test->needs_hugetlbfs > 1) {
                tst_brk(TBROK,
-                       "Two or more of needs_{rofs, devfs, device} are
 set");
+                       "Two or more of needs_{rofs, devfs, device,
 hugetlb} are set");
        }

OK. Will update it.

Quoted 40 lines
        if (tst_test->needs_device && !mntpoint_mounted) {
Quoted 30 lines

tdev.dev = tst_acquire_device_(NULL,
tst_test->dev_min_size);

lib/tst_test.c +9 -3
@@ -1299,8 +1337,15 @@ static void do_cleanup(void)
        if (ovl_mounted)
                SAFE_UMOUNT(OVL_MNT);
 
-       if (mntpoint_mounted)
-               tst_umount(tst_test->mntpoint);
+       if (tst_hugetlb_fd >= 0)
+               SAFE_CLOSE(tst_hugetlb_fd);
+
+       if (mntpoint_mounted) {
+               if (tst_test->needs_hugetlbfs)
+                       SAFE_UMOUNT(tst_test->mntpoint);
+               else
+                       tst_umount(tst_test->mntpoint);
+       }
 
        if (tst_test->needs_device && tdev.dev)
                tst_release_device(tdev.dev);
--

2.31.1

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 6 lines
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw

Re: [LTP] [PATCH v3 1/4] Hugetlb: Add new tst_test options for hugeltb test support

In reply to Li Wang

Hi!

Quoted 4 lines
Quoted 4 lines
include/tst_test.h +0 -0
@@ -357,6 +369,12 @@ unsigned int tst_remaining_runtime(void);
  */
 void tst_set_max_runtime(int max_runtime);
 
Quoted 9 lines

include/tst_test.h unsigned int tst_remaining_runtime(void);

Quoted 6 lines
include/tst_test.h +5 -0
+/*
+ * Create and open a random file inside the given dir path.
+ * It unlinks the file after opening and return file descriptor.
+ */
+int tst_create_unlinked_file(const char *path);
 

what about renaming this function to tst_'get|create'_unlinked_file_fd? I guess the 'fd' part should be emphasized here.

It has create in name and in UNIX creat() creates file and returns a file descriptor so I think it's fine. Maybe we can be even shorted with something as tst_creat_unlinked(const char *path).

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw

Re: [LTP] [PATCH v3 1/4] Hugetlb: Add new tst_test options for hugeltb test support

In reply to Li Wang

Hi!

Quoted 5 lines

Why not consider encapsulating these two new fields in 'struct tst_hugepage' ?

Then the tst_test in the case can simply initialize to:

Quoted 8 lines

....
static struct tst_test test = {
.needs_root = 1,
.taint_check = TST_TAINT_D | TST_TAINT_W,
.setup = setup,
.test_all = run_test,
.hugepages = {1, TST_NEEDS, 1, 1},
};

I do not like that we have magic constants in the .hugepages that are not self describing. I would treat the hugetltbfs just as we treat devfs, that would be:

#define MNTPOINT "hugetlbfs/"
#define HUGEFILE MNTPOINT "hugefile"

static int huge_fd;

static void setup(void)

{
	huge_fd = tst_creat_unlinked(HUGEFILE);
	...
}

static void cleanup(void)

{
	if (huge_fd > 0)
		SAFE_CLOSE(huge_fd);
}

static struct tst_test test = {
...
.mntpoint = MNTPOINT,
.needs_hugetlbfs = 1,
.setup = setup,
.cleanup = cleanup,
...
}

What do you think?

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#6 lore raw

Re: [LTP] [PATCH v3 1/4] Hugetlb: Add new tst_test options for hugeltb test support

Hi Cyril,
Please find my comments inline.

Quoted 46 lines · Cyril Hrubis

Hi!

Quoted 5 lines

Why not consider encapsulating these two new fields in 'struct tst_hugepage' ?

Then the tst_test in the case can simply initialize to:

Quoted 8 lines

....
static struct tst_test test = {
.needs_root = 1,
.taint_check = TST_TAINT_D | TST_TAINT_W,
.setup = setup,
.test_all = run_test,
.hugepages = {1, TST_NEEDS, 1, 1},
};

I do not like that we have magic constants in the .hugepages that are not self describing. I would treat the hugetltbfs just as we treat devfs, that would be:

#define MNTPOINT "hugetlbfs/"
#define HUGEFILE MNTPOINT "hugefile"

static int huge_fd;

static void setup(void)

{
	huge_fd = tst_creat_unlinked(HUGEFILE);
	...
}

static void cleanup(void)

{
	if (huge_fd > 0)
		SAFE_CLOSE(huge_fd);
}

static struct tst_test test = {
...
.mntpoint = MNTPOINT,
.needs_hugetlbfs = 1,
.setup = setup,
.cleanup = cleanup,
...
}

Quoted 2 lines

What do you think?

My original idea behind putting it in tst_test struct instead of tst_hugepages was based on below two reasoning-

1. tst_hugepages seems to have only hugepages related info. It would be better to rename it to tst_hugetlb and rename <hugepages> field in tst_test struct to <hugetlb>. If we rename it which will require changes in already existing tests. and Moreover, there were similar field like needs_tmpdir in tst_test so I put it(needs_unlinked_hugetlb_file) in tst_test.
2. There was already related fields in tst_test for mounting fs, like needs_devfs, needs_rofs, So keeping all the needs_ABCfs at same structure.

Quoted 6 lines
Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#7 lore raw

Re: [LTP] [PATCH v3 1/4] Hugetlb: Add new tst_test options for hugeltb test support

In reply to Cyril Hrubis
Quoted 46 lines · Cyril Hrubis

Hi!

Quoted 5 lines

Why not consider encapsulating these two new fields in 'struct tst_hugepage' ?

Then the tst_test in the case can simply initialize to:

Quoted 8 lines

....
static struct tst_test test = {
.needs_root = 1,
.taint_check = TST_TAINT_D | TST_TAINT_W,
.setup = setup,
.test_all = run_test,
.hugepages = {1, TST_NEEDS, 1, 1},
};

I do not like that we have magic constants in the .hugepages that are not self describing. I would treat the hugetltbfs just as we treat devfs, that would be:

#define MNTPOINT "hugetlbfs/"
#define HUGEFILE MNTPOINT "hugefile"

static int huge_fd;

static void setup(void)

{
        huge_fd = tst_creat_unlinked(HUGEFILE);
        ...
}

static void cleanup(void)

{
        if (huge_fd > 0)
                SAFE_CLOSE(huge_fd);
}

static struct tst_test test = {
...
.mntpoint = MNTPOINT,
.needs_hugetlbfs = 1,
.setup = setup,
.cleanup = cleanup,
...
}

Quoted 2 lines

What do you think?

+1 Looks good, this treats it as an FS and is separated from the hugepage usage.

Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#8 lore raw

Re: [LTP] [PATCH v3 1/4] Hugetlb: Add new tst_test options for hugeltb test support

In reply to Tarun Sahu

Hi!

Quoted 96 lines

extern unsigned int tst_variant;

include/tst_test.h +19 -1
-
+extern int tst_hugetlb_fd;
 #define TST_NO_HUGEPAGES ((unsigned long)-1)
 
 #define TST_UNLIMITED_RUNTIME (-1)
@@ -176,6 +176,18 @@ struct tst_test {
 	int all_filesystems:1;
 	int skip_in_lockdown:1;
 	int skip_in_compat:1;
+	/*
+	 * If set, the test function will create a hugetlbfs mount point
+	 * at /tmp/xxxxxx, where xxxxxx is a random string.
+	 */
+	int needs_hugetlbfs:1;
+	/*
+	 * If set, the test function will create and open a random file
+	 * under mounted hugetlbfs. To use this option, needs_hugetlbfs must
+	 * be set. The file descriptior will be set in tst_hugetlb_fd.
+	 * The close(tst_hugetlb_fd) will be called on test exit(cleanup).
+	 */
+	int needs_unlinked_hugetlb_file:1;
 
 	/*
 	 * The skip_filesystems is a NULL terminated list of filesystems the
@@ -357,6 +369,12 @@ unsigned int tst_remaining_runtime(void);
  */
 void tst_set_max_runtime(int max_runtime);
 
+/*
+ * Create and open a random file inside the given dir path.
+ * It unlinks the file after opening and return file descriptor.
+ */
+int tst_create_unlinked_file(const char *path);
+
 /*
  * Returns path to the test temporary directory in a newly allocated buffer.
  */
lib/tst_test.c +28 -1
index 8ccde1629..43cba1004 100644
@@ -925,7 +925,8 @@ static int needs_tmpdir(void)
 	       tst_test->needs_device ||
 	       tst_test->mntpoint ||
 	       tst_test->resource_files ||
-	       tst_test->needs_checkpoints;
+	       tst_test->needs_checkpoints ||
+		   tst_test->needs_hugetlbfs;
 }
 
 static void copy_resources(void)
@@ -1021,6 +1022,30 @@ static void prepare_and_mount_dev_fs(const char *mntpoint)
 	}
 }
 
+static void prepare_and_mount_hugetlb_fs(void)
+{
+	tst_test->mntpoint = tst_get_tmpdir();
+	SAFE_MOUNT("none", tst_test->mntpoint, "hugetlbfs", 0, NULL);
+	mntpoint_mounted = 1;
+}
+
+int tst_create_unlinked_file(const char *path)
+{
+	char template[PATH_MAX];
+	int fd;
+
+	snprintf(template, PATH_MAX, "%s/ltp_%.3sXXXXXX",
+			path, TCID);
+
+	fd = mkstemp(template);
+	if (fd < 0)
+		tst_brk(TBROK | TERRNO,
+			 "%s: mkstemp(%s) failed", __func__, template);
+
+	SAFE_UNLINK(template);
+	return fd;
+}
+
 static const char *limit_tmpfs_mount_size(const char *mnt_data,
 		char *buf, size_t buf_size, const char *fs_type)
 {
@@ -1094,6 +1119,8 @@ static void do_cgroup_requires(void)
 	tst_cg_init();
 }
 
+int tst_hugetlb_fd = -1;
+
 static void do_setup(int argc, char *argv[])
 {
 	if (!tst_test)
@@ -1217,6 +1244,17 @@ static void do_setup(int argc, char *argv[])
 		}
 	}
 

lib/tst_test.c static void do_setup(int argc, char *argv[]) · line 1256

Quoted 10 lines
lib/tst_test.c +10 -0
+	if (tst_test->needs_hugetlbfs)
+		prepare_and_mount_hugetlb_fs();
+
+	if (tst_test->needs_unlinked_hugetlb_file) {
+		if (!(tst_test->needs_hugetlbfs)) {
+			tst_brk(TBROK, "Option needs_unlinked_hugetlb_file "
+					"requires option needs_hugetlbfs");
+		}
+		tst_hugetlb_fd = tst_create_unlinked_file(tst_test->mntpoint);
+	}

The function tst_create_unlinked_file() looks useful, but I do not think that adding the needs_unlinked_hugetlb_file flag simplifies things that much. Also this will not scale well when we would need two filedescripors like that. Maybe we it would be cleaner to add only the mount/umount functionality to the test library and call the tst_create_unlinked_file() in the test setup in the testcases.

Quoted 8 lines
lib/tst_test.c +0 -1
 	if (tst_test->needs_device && !mntpoint_mounted) {
 		tdev.dev = tst_acquire_device_(NULL, tst_test->dev_min_size);
 
@@ -1299,8 +1337,15 @@ static void do_cleanup(void)
 	if (ovl_mounted)
 		SAFE_UMOUNT(OVL_MNT);
 
-	if (mntpoint_mounted)

lib/tst_test.c static void do_cleanup(void) · line 1348

Quoted 10 lines
lib/tst_test.c +9 -1
-		tst_umount(tst_test->mntpoint);
+	if (tst_hugetlb_fd >= 0)
+		SAFE_CLOSE(tst_hugetlb_fd);
+
+	if (mntpoint_mounted) {
+		if (tst_test->needs_hugetlbfs)
+			SAFE_UMOUNT(tst_test->mntpoint);
+		else
+			tst_umount(tst_test->mntpoint);
+	}

Is there a good reason for this, why can't we call tst_umount() for hugetlbfs?

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#9 lore raw

Re: [LTP] [PATCH v3 1/4] Hugetlb: Add new tst_test options for hugeltb test support

Quoted 106 lines · Cyril Hrubis

Hi!

Quoted 105 lines

extern unsigned int tst_variant;

include/tst_test.h +19 -1
-
+extern int tst_hugetlb_fd;
 #define TST_NO_HUGEPAGES ((unsigned long)-1)
 
 #define TST_UNLIMITED_RUNTIME (-1)
@@ -176,6 +176,18 @@ struct tst_test {
 	int all_filesystems:1;
 	int skip_in_lockdown:1;
 	int skip_in_compat:1;
+	/*
+	 * If set, the test function will create a hugetlbfs mount point
+	 * at /tmp/xxxxxx, where xxxxxx is a random string.
+	 */
+	int needs_hugetlbfs:1;
+	/*
+	 * If set, the test function will create and open a random file
+	 * under mounted hugetlbfs. To use this option, needs_hugetlbfs must
+	 * be set. The file descriptior will be set in tst_hugetlb_fd.
+	 * The close(tst_hugetlb_fd) will be called on test exit(cleanup).
+	 */
+	int needs_unlinked_hugetlb_file:1;
 
 	/*
 	 * The skip_filesystems is a NULL terminated list of filesystems the
@@ -357,6 +369,12 @@ unsigned int tst_remaining_runtime(void);
  */
 void tst_set_max_runtime(int max_runtime);
 
+/*
+ * Create and open a random file inside the given dir path.
+ * It unlinks the file after opening and return file descriptor.
+ */
+int tst_create_unlinked_file(const char *path);
+
 /*
  * Returns path to the test temporary directory in a newly allocated buffer.
  */
lib/tst_test.c +37 -1
index 8ccde1629..43cba1004 100644
@@ -925,7 +925,8 @@ static int needs_tmpdir(void)
 	       tst_test->needs_device ||
 	       tst_test->mntpoint ||
 	       tst_test->resource_files ||
-	       tst_test->needs_checkpoints;
+	       tst_test->needs_checkpoints ||
+		   tst_test->needs_hugetlbfs;
 }
 
 static void copy_resources(void)
@@ -1021,6 +1022,30 @@ static void prepare_and_mount_dev_fs(const char *mntpoint)
 	}
 }
 
+static void prepare_and_mount_hugetlb_fs(void)
+{
+	tst_test->mntpoint = tst_get_tmpdir();
+	SAFE_MOUNT("none", tst_test->mntpoint, "hugetlbfs", 0, NULL);
+	mntpoint_mounted = 1;
+}
+
+int tst_create_unlinked_file(const char *path)
+{
+	char template[PATH_MAX];
+	int fd;
+
+	snprintf(template, PATH_MAX, "%s/ltp_%.3sXXXXXX",
+			path, TCID);
+
+	fd = mkstemp(template);
+	if (fd < 0)
+		tst_brk(TBROK | TERRNO,
+			 "%s: mkstemp(%s) failed", __func__, template);
+
+	SAFE_UNLINK(template);
+	return fd;
+}
+
 static const char *limit_tmpfs_mount_size(const char *mnt_data,
 		char *buf, size_t buf_size, const char *fs_type)
 {
@@ -1094,6 +1119,8 @@ static void do_cgroup_requires(void)
 	tst_cg_init();
 }
 
+int tst_hugetlb_fd = -1;
+
 static void do_setup(int argc, char *argv[])
 {
 	if (!tst_test)
@@ -1217,6 +1244,17 @@ static void do_setup(int argc, char *argv[])
 		}
 	}
 
+	if (tst_test->needs_hugetlbfs)
+		prepare_and_mount_hugetlb_fs();
+
+	if (tst_test->needs_unlinked_hugetlb_file) {
+		if (!(tst_test->needs_hugetlbfs)) {
+			tst_brk(TBROK, "Option needs_unlinked_hugetlb_file "
+					"requires option needs_hugetlbfs");
+		}
+		tst_hugetlb_fd = tst_create_unlinked_file(tst_test->mntpoint);
Quoted 9 lines

lib/tst_test.c static void do_setup(int argc, char *argv[]) · line 1256

Quoted 1 line
lib/tst_test.c +1 -0
+	}

The function tst_create_unlinked_file() looks useful, but I do not think that adding the needs_unlinked_hugetlb_file flag simplifies things that much. Also this will not scale well when we would need two filedescripors like that. Maybe we it would be cleaner to add only the mount/umount functionality to the test library and call the tst_create_unlinked_file() in the test setup in the testcases.

yes I agree, There is a test that requires two such unlinked file, there is one that requires one hugetlb and one normal such unlinked file.

Quoted 7 lines
Quoted 7 lines
lib/tst_test.c +0 -0
 	if (tst_test->needs_device && !mntpoint_mounted) {
 		tdev.dev = tst_acquire_device_(NULL, tst_test->dev_min_size);
 
@@ -1299,8 +1337,15 @@ static void do_cleanup(void)
 	if (ovl_mounted)
 		SAFE_UMOUNT(OVL_MNT);
 
Quoted 15 lines

lib/tst_test.c static void do_cleanup(void) · line 1348

Quoted 11 lines
lib/tst_test.c +9 -2
-	if (mntpoint_mounted)
-		tst_umount(tst_test->mntpoint);
+	if (tst_hugetlb_fd >= 0)
+		SAFE_CLOSE(tst_hugetlb_fd);
+
+	if (mntpoint_mounted) {
+		if (tst_test->needs_hugetlbfs)
+			SAFE_UMOUNT(tst_test->mntpoint);
+		else
+			tst_umount(tst_test->mntpoint);
+	}

Is there a good reason for this, why can't we call tst_umount() for hugetlbfs?

My reason behind this is,
tst_umount doesnt break the test, if umount fails. One of the cause of failure can be that test left the mounted fs busy, which is not expected.

Quoted 6 lines
Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#10 lore raw

Re: [LTP] [PATCH v3 1/4] Hugetlb: Add new tst_test options for hugeltb test support

Hi!

Quoted 8 lines
Quoted 3 lines

Is there a good reason for this, why can't we call tst_umount() for hugetlbfs?

My reason behind this is,
tst_umount doesnt break the test, if umount fails. One of the cause of failure can be that test left the mounted fs busy, which is not expected.

This does not matter, because the do_cleanup() is called only once at the end of the test. The test exists once the function does finish anyways.

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#11 lore raw

PATCH v3 2/4 Hugetlb: Migrating libhugetlbfs brk_near_huge

In reply to Tarun Sahu

Migrating the libhugetlbfs/testcases/brk_near_huge.c test

Test Description:
Certain kernels have a bug where brk() does not perform the same checks that a MAP_FIXED mmap() will, allowing brk() to create a normal page VMA in a hugepage only address region. This can lead to oopses or other badness.

---
 runtest/hugetlb                               |   1 +
 testcases/kernel/mem/.gitignore               |   1 +
 .../kernel/mem/hugetlb/hugemmap/Makefile      |   5 +
 .../kernel/mem/hugetlb/hugemmap/hugemmap07.c  | 129 ++++++++++++++++++
 testcases/kernel/mem/hugetlb/lib/hugetlb.h    |   3 +
 5 files changed, 139 insertions(+)
 create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c
runtest/hugetlb +1 -0
index f719217ab..f7ff81cb3 100644
@@ -3,6 +3,7 @@ hugemmap02 hugemmap02
 hugemmap04 hugemmap04
 hugemmap05 hugemmap05
 hugemmap06 hugemmap06
+hugemmap07 hugemmap07
 hugemmap05_1 hugemmap05 -m
 hugemmap05_2 hugemmap05 -s
 hugemmap05_3 hugemmap05 -s -m
testcases/kernel/mem/.gitignore +1 -0
index ff2910533..df5256ec8 100644
@@ -4,6 +4,7 @@
 /hugetlb/hugemmap/hugemmap04
 /hugetlb/hugemmap/hugemmap05
 /hugetlb/hugemmap/hugemmap06
+/hugetlb/hugemmap/hugemmap07
 /hugetlb/hugeshmat/hugeshmat01
 /hugetlb/hugeshmat/hugeshmat02
 /hugetlb/hugeshmat/hugeshmat03
testcases/kernel/mem/hugetlb/hugemmap/Makefile +5 -0
index 2d651b4aa..a368db2b6 100644
@@ -9,3 +9,8 @@ include $(abs_srcdir)/../Makefile.inc
 include $(top_srcdir)/include/mk/generic_leaf_target.mk
 
 hugemmap06: CFLAGS+=-pthread
+
+hugemmap07:
+ifeq ($(ARCH),ppc)
+	CPPFLAGS += -DPPC_NO_SEGMENTS
+endif
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +129 -0

129 lines not on this page · raw message

testcases/kernel/mem/hugetlb/lib/hugetlb.h +3 -0
index f75109f3e..1cfeca95a 100644
@@ -20,6 +20,9 @@
 #include "old_tmpdir.h"
 #include "mem.h"
 
+#define ALIGN(x, a)	(((x) + (a) - 1) & ~((a) - 1))
+#define PALIGN(p, a) ((void *)ALIGN((unsigned long)(p), (a)))
+
 #define SHM_RD	0400
 #define SHM_WR	0200
 #define SHM_RW	(SHM_RD|SHM_WR)
Signature · 5 lines
2.31.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#12 lore raw

PATCH v3 3/4 Hugetlb: Migrating libhugetlbfs chunk-overcommit

In reply to Tarun Sahu

Migrating the libhugetlbfs/testcases/chunk-overcommit.c test

Test Description: Some kernel versions after hugepage demand allocation was added used a dubious heuristic to check if there was enough hugepage space available for a given mapping. The number of not-already-instantiated pages in the mapping was compared against the total hugepage free pool. It was very easy to confuse this heuristic into overcommitting by allocating hugepage memory in chunks, each less than the total available pool size but together more than available. This would generally lead to OOM SIGKILLs of one process or another when it tried to instantiate pages beyond the available pool.

---
 runtest/hugetlb                               |   1 +
 testcases/kernel/mem/.gitignore               |   1 +
 .../kernel/mem/hugetlb/hugemmap/hugemmap08.c  | 144 ++++++++++++++++++
 3 files changed, 146 insertions(+)
 create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c
runtest/hugetlb +1 -0
index f7ff81cb3..664f18827 100644
@@ -4,6 +4,7 @@ hugemmap04 hugemmap04
 hugemmap05 hugemmap05
 hugemmap06 hugemmap06
 hugemmap07 hugemmap07
+hugemmap08 hugemmap08
 hugemmap05_1 hugemmap05 -m
 hugemmap05_2 hugemmap05 -s
 hugemmap05_3 hugemmap05 -s -m
testcases/kernel/mem/.gitignore +1 -0
index df5256ec8..003ce422b 100644
@@ -5,6 +5,7 @@
 /hugetlb/hugemmap/hugemmap05
 /hugetlb/hugemmap/hugemmap06
 /hugetlb/hugemmap/hugemmap07
+/hugetlb/hugemmap/hugemmap08
 /hugetlb/hugeshmat/hugeshmat01
 /hugetlb/hugeshmat/hugeshmat02
 /hugetlb/hugeshmat/hugeshmat03
testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +144 -0

144 lines not on this page · raw message

Signature · 5 lines
2.31.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#13 lore raw

Re: [LTP] [PATCH v3 3/4] Hugetlb: Migrating libhugetlbfs chunk-overcommit

Quoted 154 lines · Tarun Sahu

Migrating the libhugetlbfs/testcases/chunk-overcommit.c test

Test Description: Some kernel versions after hugepage demand allocation was added used a dubious heuristic to check if there was enough hugepage space available for a given mapping. The number of not-already-instantiated pages in the mapping was compared against the total hugepage free pool. It was very easy to confuse this heuristic into overcommitting by allocating hugepage memory in chunks, each less than the total available pool size but together more than available. This would generally lead to OOM SIGKILLs of one process or another when it tried to instantiate pages beyond the available pool.

---
 runtest/hugetlb                               |   1 +
 testcases/kernel/mem/.gitignore               |   1 +
 .../kernel/mem/hugetlb/hugemmap/hugemmap08.c  | 144 ++++++++++++++++++
 3 files changed, 146 insertions(+)
 create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c
runtest/hugetlb +1 -0
index f7ff81cb3..664f18827 100644
@@ -4,6 +4,7 @@ hugemmap04 hugemmap04
 hugemmap05 hugemmap05
 hugemmap06 hugemmap06
 hugemmap07 hugemmap07
+hugemmap08 hugemmap08
 hugemmap05_1 hugemmap05 -m
 hugemmap05_2 hugemmap05 -s
 hugemmap05_3 hugemmap05 -s -m
testcases/kernel/mem/.gitignore +1 -0
index df5256ec8..003ce422b 100644
@@ -5,6 +5,7 @@
 /hugetlb/hugemmap/hugemmap05
 /hugetlb/hugemmap/hugemmap06
 /hugetlb/hugemmap/hugemmap07
+/hugetlb/hugemmap/hugemmap08
 /hugetlb/hugeshmat/hugeshmat01
 /hugetlb/hugeshmat/hugeshmat02
 /hugetlb/hugeshmat/hugeshmat03
testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +58 -0
new file mode 100644
index 000000000..61db030d5
@@ -0,0 +1,144 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2005-2006 David Gibson & Adam Litke, IBM Corporation.
+ * Author: David Gibson & Adam Litke
+ */
+
+/*\
+ * [Description]
+ *
+ * Chunk Overcommit:
+ * Some kernel versions after hugepage demand allocation was added used a
+ * dubious heuristic to check if there was enough hugepage space available
+ * for a given mapping.  The number of not-already-instantiated pages in
+ * the mapping was compared against the total hugepage free pool. It was
+ * very easy to confuse this heuristic into overcommitting by allocating
+ * hugepage memory in chunks, each less than the total available pool size
+ * but together more than available.  This would generally lead to OOM
+ * SIGKILLs of one process or another when it tried to instantiate pages
+ * beyond the available pool.
+ *
+ * HISTORY
+ *
+ */
+
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mount.h>
+#include <limits.h>
+#include <sys/param.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <signal.h>
+
+#include "hugetlb.h"
+
+#define PROC_OVERCOMMIT "/proc/sys/vm/nr_overcommit_hugepages"
+#define WITH_OVERCOMMIT 0
+#define WITHOUT_OVERCOMMIT 1
+
+static long hpage_size;
+
+static void test_chunk_overcommit(void)
+{
+       unsigned long totpages, chunk1, chunk2;
+       void *p, *q;
+       pid_t child;
+       int status;
+
+       totpages = SAFE_READ_MEMINFO("HugePages_Free:");
+
+       chunk1 = (totpages / 2) + 1;
+       chunk2 = totpages - chunk1 + 1;
+
+       tst_res(TINFO, "Free: %ld hugepages available: "
+              "chunk1=%ld chunk2=%ld", totpages, chunk1, chunk2);
+
+       p = SAFE_MMAP(NULL, chunk1*hpage_size, PROT_READ|PROT_WRITE,

MAP_SHARED,

testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +10 -0
+                tst_hugetlb_fd, 0);
+
+       q = mmap(NULL, chunk2*hpage_size, PROT_READ|PROT_WRITE, MAP_SHARED,
+                tst_hugetlb_fd, chunk1*hpage_size);
+       if (q == MAP_FAILED) {
+               if (errno != ENOMEM) {
+                       tst_res(TFAIL | TERRNO, "mmap() chunk2");
+                       goto cleanup1;
+               } else {
+                       tst_res(TPASS, "Successful without overcommit

pages");

testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +31 -0
+                       goto cleanup1;
+               }
+       }
+
+       tst_res(TINFO, "Looks like we've overcommitted, testing...");
+       /* Looks like we're overcommited, but we need to confirm that
+        * this is bad.  We touch it all in a child process because an
+        * overcommit will generally lead to a SIGKILL which we can't
+        * handle, of course.
+        */
+       child = SAFE_FORK();
+
+       if (child == 0) {
+               memset(p, 0, chunk1*hpage_size);
+               memset(q, 0, chunk2*hpage_size);
+               exit(0);
+       }
+
+       SAFE_WAITPID(child, &status, 0);
+
+       if (WIFSIGNALED(status)) {
+               tst_res(TFAIL, "Killed by signal '%s' due to overcommit",
+                    tst_strsig(WTERMSIG(status)));
+               goto cleanup2;
+       }
+
+       tst_res(TPASS, "Successful with overcommit pages");
+
+cleanup2:
+       SAFE_MUNMAP(q, chunk2*hpage_size);
+

testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c line 109

Quoted 10 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +10 -0
+cleanup1:
+       SAFE_MUNMAP(p, chunk1*hpage_size);
+       SAFE_FTRUNCATE(tst_hugetlb_fd, 0);
+}
+
+static void run_test(unsigned int test_type)
+{
+       unsigned long saved_oc_hugepages;
+
+       SAFE_FILE_SCANF(PROC_OVERCOMMIT, "%ld", &saved_oc_hugepages);

There is unnecessary to read PROC_OVERCOMMIT value again,
we already save/restore it in struct tst_path_val[], so here we can set it directly to what we expected no matter if the original is 0 or 2.

static void run_test(unsigned int test_type)

{
        switch (test_type) {
        case WITHOUT_OVERCOMMIT:
                tst_res(TINFO, "Without overcommit testing...");
                SAFE_FILE_PRINTF(PROC_OVERCOMMIT, "%d", 0);
                break;
        case WITH_OVERCOMMIT:
                tst_res(TINFO, "With overcommit testing...");
                SAFE_FILE_PRINTF(PROC_OVERCOMMIT, "%d", 2);
                break;
    }
    test_chunk_overcommit();
}
Quoted 43 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +35 -1
+       switch (test_type) {
+       case WITHOUT_OVERCOMMIT:
+               tst_res(TINFO, "Without overcommit testing...");
+               if (saved_oc_hugepages > 0)
+                       SAFE_FILE_PRINTF(PROC_OVERCOMMIT, "%d", 0);
+               break;
+       case WITH_OVERCOMMIT:
+               tst_res(TINFO, "With overcommit testing...");
+               if (saved_oc_hugepages == 0)
+                       SAFE_FILE_PRINTF(PROC_OVERCOMMIT, "%d", 2);
+               break;
+       }
+       test_chunk_overcommit();
+}
+
+static void setup(void)
+{
+       hpage_size = SAFE_READ_MEMINFO("Hugepagesize:")*1024;
+}
+
+static struct tst_test test = {
+       .needs_root = 1,
+       .needs_hugetlbfs = 1,
+       .needs_unlinked_hugetlb_file = 1,
+       .forks_child = 1,
+       .save_restore = (const struct tst_path_val[]) {
+               {PROC_OVERCOMMIT, NULL},
+               {}
+       },
+       .tcnt = 2,
+       .setup = setup,
+       .test = run_test,
+       .hugepages = {3, TST_NEEDS},
+};
+
--

2.31.1

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#14 lore raw

Re: [LTP] [PATCH v3 3/4] Hugetlb: Migrating libhugetlbfs chunk-overcommit

Quoted 172 lines · Li Wang
Quoted 154 lines · Tarun Sahu

Migrating the libhugetlbfs/testcases/chunk-overcommit.c test

Test Description: Some kernel versions after hugepage demand allocation was added used a dubious heuristic to check if there was enough hugepage space available for a given mapping. The number of not-already-instantiated pages in the mapping was compared against the total hugepage free pool. It was very easy to confuse this heuristic into overcommitting by allocating hugepage memory in chunks, each less than the total available pool size but together more than available. This would generally lead to OOM SIGKILLs of one process or another when it tried to instantiate pages beyond the available pool.

---
 runtest/hugetlb                               |   1 +
 testcases/kernel/mem/.gitignore               |   1 +
 .../kernel/mem/hugetlb/hugemmap/hugemmap08.c  | 144 ++++++++++++++++++
 3 files changed, 146 insertions(+)
 create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c
runtest/hugetlb +1 -0
index f7ff81cb3..664f18827 100644
@@ -4,6 +4,7 @@ hugemmap04 hugemmap04
 hugemmap05 hugemmap05
 hugemmap06 hugemmap06
 hugemmap07 hugemmap07
+hugemmap08 hugemmap08
 hugemmap05_1 hugemmap05 -m
 hugemmap05_2 hugemmap05 -s
 hugemmap05_3 hugemmap05 -s -m
testcases/kernel/mem/.gitignore +1 -0
index df5256ec8..003ce422b 100644
@@ -5,6 +5,7 @@
 /hugetlb/hugemmap/hugemmap05
 /hugetlb/hugemmap/hugemmap06
 /hugetlb/hugemmap/hugemmap07
+/hugetlb/hugemmap/hugemmap08
 /hugetlb/hugeshmat/hugeshmat01
 /hugetlb/hugeshmat/hugeshmat02
 /hugetlb/hugeshmat/hugeshmat03
testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +58 -0
new file mode 100644
index 000000000..61db030d5
@@ -0,0 +1,144 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2005-2006 David Gibson & Adam Litke, IBM Corporation.
+ * Author: David Gibson & Adam Litke
+ */
+
+/*\
+ * [Description]
+ *
+ * Chunk Overcommit:
+ * Some kernel versions after hugepage demand allocation was added used a
+ * dubious heuristic to check if there was enough hugepage space available
+ * for a given mapping.  The number of not-already-instantiated pages in
+ * the mapping was compared against the total hugepage free pool. It was
+ * very easy to confuse this heuristic into overcommitting by allocating
+ * hugepage memory in chunks, each less than the total available pool size
+ * but together more than available.  This would generally lead to OOM
+ * SIGKILLs of one process or another when it tried to instantiate pages
+ * beyond the available pool.
+ *
+ * HISTORY
+ *
+ */
+
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mount.h>
+#include <limits.h>
+#include <sys/param.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <signal.h>
+
+#include "hugetlb.h"
+
+#define PROC_OVERCOMMIT "/proc/sys/vm/nr_overcommit_hugepages"
+#define WITH_OVERCOMMIT 0
+#define WITHOUT_OVERCOMMIT 1
+
+static long hpage_size;
+
+static void test_chunk_overcommit(void)
+{
+       unsigned long totpages, chunk1, chunk2;
+       void *p, *q;
+       pid_t child;
+       int status;
+
+       totpages = SAFE_READ_MEMINFO("HugePages_Free:");
+
+       chunk1 = (totpages / 2) + 1;
+       chunk2 = totpages - chunk1 + 1;
+
+       tst_res(TINFO, "Free: %ld hugepages available: "
+              "chunk1=%ld chunk2=%ld", totpages, chunk1, chunk2);
+
+       p = SAFE_MMAP(NULL, chunk1*hpage_size, PROT_READ|PROT_WRITE,

MAP_SHARED,

testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +10 -0
+                tst_hugetlb_fd, 0);
+
+       q = mmap(NULL, chunk2*hpage_size, PROT_READ|PROT_WRITE, MAP_SHARED,
+                tst_hugetlb_fd, chunk1*hpage_size);
+       if (q == MAP_FAILED) {
+               if (errno != ENOMEM) {
+                       tst_res(TFAIL | TERRNO, "mmap() chunk2");
+                       goto cleanup1;
+               } else {
+                       tst_res(TPASS, "Successful without overcommit

pages");

testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +31 -0
+                       goto cleanup1;
+               }
+       }
+
+       tst_res(TINFO, "Looks like we've overcommitted, testing...");
+       /* Looks like we're overcommited, but we need to confirm that
+        * this is bad.  We touch it all in a child process because an
+        * overcommit will generally lead to a SIGKILL which we can't
+        * handle, of course.
+        */
+       child = SAFE_FORK();
+
+       if (child == 0) {
+               memset(p, 0, chunk1*hpage_size);
+               memset(q, 0, chunk2*hpage_size);
+               exit(0);
+       }
+
+       SAFE_WAITPID(child, &status, 0);
+
+       if (WIFSIGNALED(status)) {
+               tst_res(TFAIL, "Killed by signal '%s' due to overcommit",
+                    tst_strsig(WTERMSIG(status)));
+               goto cleanup2;
+       }
+
+       tst_res(TPASS, "Successful with overcommit pages");
+
+cleanup2:
+       SAFE_MUNMAP(q, chunk2*hpage_size);
+

testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c line 109

Quoted 10 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +10 -0
+cleanup1:
+       SAFE_MUNMAP(p, chunk1*hpage_size);
+       SAFE_FTRUNCATE(tst_hugetlb_fd, 0);
+}
+
+static void run_test(unsigned int test_type)
+{
+       unsigned long saved_oc_hugepages;
+
+       SAFE_FILE_SCANF(PROC_OVERCOMMIT, "%ld", &saved_oc_hugepages);

There is unnecessary to read PROC_OVERCOMMIT value again,
we already save/restore it in struct tst_path_val[], so here we can set it directly to what we expected no matter if the original is 0 or 2.

static void run_test(unsigned int test_type)

Quoted 14 lines
{
        switch (test_type) {
        case WITHOUT_OVERCOMMIT:
                tst_res(TINFO, "Without overcommit testing...");
                SAFE_FILE_PRINTF(PROC_OVERCOMMIT, "%d", 0);
                break;
        case WITH_OVERCOMMIT:
                tst_res(TINFO, "With overcommit testing...");
                SAFE_FILE_PRINTF(PROC_OVERCOMMIT, "%d", 2);
                break;
    }
    test_chunk_overcommit();
}

Yeah, Missed it. Will update it.

Quoted 53 lines
Quoted 45 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +35 -1
+       switch (test_type) {
+       case WITHOUT_OVERCOMMIT:
+               tst_res(TINFO, "Without overcommit testing...");
+               if (saved_oc_hugepages > 0)
+                       SAFE_FILE_PRINTF(PROC_OVERCOMMIT, "%d", 0);
+               break;
+       case WITH_OVERCOMMIT:
+               tst_res(TINFO, "With overcommit testing...");
+               if (saved_oc_hugepages == 0)
+                       SAFE_FILE_PRINTF(PROC_OVERCOMMIT, "%d", 2);
+               break;
+       }
+       test_chunk_overcommit();
+}
+
+static void setup(void)
+{
+       hpage_size = SAFE_READ_MEMINFO("Hugepagesize:")*1024;
+}
+
+static struct tst_test test = {
+       .needs_root = 1,
+       .needs_hugetlbfs = 1,
+       .needs_unlinked_hugetlb_file = 1,
+       .forks_child = 1,
+       .save_restore = (const struct tst_path_val[]) {
+               {PROC_OVERCOMMIT, NULL},
+               {}
+       },
+       .tcnt = 2,
+       .setup = setup,
+       .test = run_test,
+       .hugepages = {3, TST_NEEDS},
+};
+
--

2.31.1

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 6 lines
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#15 lore raw

PATCH v3 4/4 Hugetlb: Migrating libhugetlbfs corrupt-by-cow-opt

In reply to Tarun Sahu

Migrating the libhugetlbfs/testcases/corrupt-by-cow-opt.c test

Test Description: Test sanity of cow optimization on page cache. If a page in page cache has only 1 ref count, it is mapped for a private mapping directly and is overwritten freely, so next time we access the page, we can see corrupt data.

---
 runtest/hugetlb                               |  1 +
 testcases/kernel/mem/.gitignore               |  1 +
 .../kernel/mem/hugetlb/hugemmap/hugemmap09.c  | 71 +++++++++++++++++++
 3 files changed, 73 insertions(+)
 create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c
runtest/hugetlb +1 -0
index 664f18827..e2ada7a97 100644
@@ -5,6 +5,7 @@ hugemmap05 hugemmap05
 hugemmap06 hugemmap06
 hugemmap07 hugemmap07
 hugemmap08 hugemmap08
+hugemmap09 hugemmap09
 hugemmap05_1 hugemmap05 -m
 hugemmap05_2 hugemmap05 -s
 hugemmap05_3 hugemmap05 -s -m
testcases/kernel/mem/.gitignore +1 -0
index 003ce422b..1a242ffe0 100644
@@ -6,6 +6,7 @@
 /hugetlb/hugemmap/hugemmap06
 /hugetlb/hugemmap/hugemmap07
 /hugetlb/hugemmap/hugemmap08
+/hugetlb/hugemmap/hugemmap09
 /hugetlb/hugeshmat/hugeshmat01
 /hugetlb/hugeshmat/hugeshmat02
 /hugetlb/hugeshmat/hugeshmat03
testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c new file +71 -0

71 lines not on this page · raw message

Signature · 5 lines
2.31.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp