Hugetlb:Migrating the libhugetlbfs tests
v2 cover Hugetlb:Migrating the libhugetlbfs tests
Hi,
Libhugetlbfs is not being maintained actively, and some distro is dropping support for it. There are some tests that are good for testing hugetlb functionality in kernel.
As per previous dicussion in RFC[1], Here, this patch series consists of hugetlb tests taken from libhugetlbfs modified to work in ltp environment.
Based on suggestion[2], I am reposting the patches again. Also, I added suggested changes by Cyril Hrubis and Li Wang[3].
ref:
1:https://lore.kernel.org/all/[email protected]/ 2:https://lore.kernel.org/all/[email protected]/ 3:https://lore.kernel.org/all/[email protected]/
v1 -> v2
1. In (brk near huge) test [PATCH 1/3] removed unused library function test_addr_huge() and read_maps().
Tarun Sahu (3):
Hugetlb: Migrating libhugetlbfs brk_near_huge
Hugetlb: Migrating libhugetlbfs chunk-overcommit
Hugetlb: Migrating libhugetlbfs corrupt-by-cow-opt
runtest/hugetlb | 3 + testcases/kernel/mem/.gitignore | 3 + .../kernel/mem/hugetlb/hugemmap/hugemmap07.c | 156 ++++++++++++++++ .../kernel/mem/hugetlb/hugemmap/hugemmap08.c | 166 ++++++++++++++++++ .../kernel/mem/hugetlb/hugemmap/hugemmap09.c | 90 ++++++++++ testcases/kernel/mem/hugetlb/lib/hugetlb.h | 3 + 6 files changed, 421 insertions(+) create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c
Signature · 5 lines
2.31.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH v2 1/3 Hugetlb: Migrating libhugetlbfs brk_near_huge
Migrating the libhugetlbfs/testcases/brk_near_huge.c test
Test Description:
Certain kernels have a bug where brk() does not perform the same checks that a MAP_FIXED mmap() will, allowing brk() to create a normal page VMA in a hugepage only address region. This can lead to oopses or other badness.
--- runtest/hugetlb | 1 + testcases/kernel/mem/.gitignore | 1 + .../kernel/mem/hugetlb/hugemmap/hugemmap07.c | 156 ++++++++++++++++++ testcases/kernel/mem/hugetlb/lib/hugetlb.h | 3 + 4 files changed, 161 insertions(+) create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c
runtest/hugetlb +1 -0
@@ -3,6 +3,7 @@ hugemmap02 hugemmap02 hugemmap04 hugemmap04 hugemmap05 hugemmap05 hugemmap06 hugemmap06 +hugemmap07 hugemmap07 hugemmap05_1 hugemmap05 -m hugemmap05_2 hugemmap05 -s hugemmap05_3 hugemmap05 -s -m
testcases/kernel/mem/.gitignore +1 -0
@@ -4,6 +4,7 @@ /hugetlb/hugemmap/hugemmap04 /hugetlb/hugemmap/hugemmap05 /hugetlb/hugemmap/hugemmap06 +/hugetlb/hugemmap/hugemmap07 /hugetlb/hugeshmat/hugeshmat01 /hugetlb/hugeshmat/hugeshmat02 /hugetlb/hugeshmat/hugeshmat03
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +156 -0
156 lines not on this page · raw message
testcases/kernel/mem/hugetlb/lib/hugetlb.h +3 -0
@@ -20,6 +20,9 @@#include "old_tmpdir.h" #include "mem.h" +#define ALIGN (x, a) (((x) + (a) - 1) & ~((a) - 1)) +#define PALIGN (p, a) ((void *)ALIGN((unsigned long)(p), (a))) +#define SHM_RD 0400#define SHM_WR 0200#define SHM_RW (SHM_RD|SHM_WR)
Signature · 5 lines
2.31.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v2 1/3] Hugetlb: Migrating libhugetlbfs brk_near_huge
Hello,
Quoted 7 lines · Tarun Sahu
Migrating the
libhugetlbfs/testcases/brk_near_huge.ctestTest Description:
Certain kernels have a bug wherebrk()does not perform the same checks that aMAP_FIXEDmmap()will, allowingbrk()to create a normal page VMA in a hugepage only address region. This can lead to oopses or other badness.
The actual test looks good, but it seems that the existing LTP library functions which you are using need some work. It looks like this test could be much smaller.
Quoted 62 lines
Signed-off-by Tarun Sahu <[email protected]>--- runtest/hugetlb | 1 + testcases/kernel/mem/.gitignore | 1 + .../kernel/mem/hugetlb/hugemmap/hugemmap07.c | 156 ++++++++++++++++++ testcases/kernel/mem/hugetlb/lib/hugetlb.h | 3 + 4 files changed, 161 insertions(+) create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.cruntest/hugetlb +1
-0@@ -3,6 +3,7 @@ hugemmap02 hugemmap02 hugemmap04 hugemmap04 hugemmap05 hugemmap05 hugemmap06 hugemmap06 +hugemmap07 hugemmap07 hugemmap05_1 hugemmap05 -m hugemmap05_2 hugemmap05 -s hugemmap05_3 hugemmap05 -s -mtestcases/kernel/mem/.gitignore +1
-0@@ -4,6 +4,7 @@ /hugetlb/hugemmap/hugemmap04 /hugetlb/hugemmap/hugemmap05 /hugetlb/hugemmap/hugemmap06 +/hugetlb/hugemmap/hugemmap07 /hugetlb/hugeshmat/hugeshmat01 /hugetlb/hugeshmat/hugeshmat02 /hugetlb/hugeshmat/hugeshmat03testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +22
-0@@ -0,0 +1,156 @@ +// SPDX-License-Identifier: LGPL-2.1-or-later +/* +* Copyright (C) 2005-2006 David Gibson & Adam Litke, IBM Corporation. +* Author: David Gibson & Adam Litke +*/ + +/*\ +* +* [Description] +* +* brk() near hugepage: +* Certain kernels have a bug where brk() does not perform the same +* checks that a MAP_FIXED mmap() will, allowing brk() to create a +* normal page VMA in a hugepage only address region. This can lead +* to oopses or other badness. +*/ + +#define _GNU_SOURCE +#include <stdio.h> +#include <sys/mount.h> +#include <limits.h> +#include <sys/param.h>
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 32
Quoted 10 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +10
-0+#include <sys/types.h> + +#include "hugetlb.h" +#include "tst_safe_stdio.h" + +static int fd =-1 ; +static char hfile[MAXPATHLEN ]; +static long hpage_size; + +static int arch_has_slice_support (void )
This only appears to be used on __powerpc64__ in next_chunk. So it can be defined in the ifdef for next_chunk.
Otherwise we could get unused function warnings and IMO it is more readable.
Quoted 5 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +5
-0+{ +#ifdef __powerpc64__ +char mmu_type[16 ]; +FILE *fp; +int ret;
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 47
Quoted 10 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +10
-0+ + fp =SAFE_POPEN ("cat /proc/cpuinfo | grep MMU | awk '{ print $3}'" ,"r" ); + ret =fscanf (fp,"%s" , mmu_type); +pclose (fp); + +if (ret <0 ) +tst_brk (TBROK ,"Failed to determine MMU type" ); + +return strcmp (mmu_type,"Hash" ) ==0 ; +#elif defined(__powerpc__) && !defined(PPC_NO_SEGMENTS )
This elif doesn't appear to be applicable.
Quoted 11 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +11
-0+return 1 ; +#else +return 0 ; +#endif +} + +#ifdef __powerpc64__ +static void *next_chunk (void *addr) +{ +if (!arch_has_slice_support ()) + return PALIGN(addr,
Quoted 1 line
SAFE_READ_MEMINFO("Hugepagesize:")*1024);
In setup we set hpage_size, but then keep reading it.
Quoted 13 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +13
-0+ +if ((unsigned long )addr <0x100000000UL ) +/* 256M segments below 4G */ +return PALIGN (addr,0x10000000UL ); +/* 1TB segments above */ + return PALIGN(addr, 0x10000000000UL); +} +#elif defined(__powerpc__) && !defined(PPC_NO_SEGMENTS) +static void *next_chunk (void *addr) +{ +return PALIGN (addr,0x10000000UL ); +} +#elif defined(__ia64__)
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 81
Quoted 10 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +10
-0+static void *next_chunk (void *addr) +{ +return PALIGN (addr,0x8000000000000000UL ); +} +#else +static void *next_chunk (void *addr) +{ +return PALIGN (addr,SAFE_READ_MEMINFO ("Hugepagesize:" )*1024 ); +} +#endif
If these functions are used in later tests I guess they should go in tst_hugepages.h
Quoted 43 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +43
-0+ +static void run_test (void ) +{ +void *brk0, *hugemap_addr, *newbrk; +char *p; +int err; + + brk0 =sbrk (0 ); +tst_res (TINFO ,"Initial break at %p" , brk0); + + hugemap_addr =next_chunk (brk0) + hpage_size; + + p =SAFE_MMAP (hugemap_addr, hpage_size,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_FIXED , fd,0 ); +if (p != hugemap_addr) { +tst_res (TFAIL ,"mmap() at unexpected address %p instead of %p\n" , p, + hugemap_addr); + goto cleanup; + } + + newbrk =next_chunk (brk0) +getpagesize (); + err =brk ((void *)newbrk); +if (err ==-1 ) { +/* Failing the brk() is an acceptable kernel response */ +tst_res (TPASS ,"Failing the brk is an acceptable response" ); + }else { +/* Suceeding the brk() is acceptable iff the new memory is +* properly accesible and we don't have a kernel blow up when +* we touch it. +*/ +tst_res (TINFO ,"New break at %p" , newbrk); +memset (brk0,0 , newbrk-brk0); +tst_res (TPASS ,"memory is accessible, hence successful brk() is " +"an acceptable response" ); + } +cleanup: +SAFE_MUNMAP (p, hpage_size); + err =brk (brk0); +if (err ==-1 ) +tst_brk (TBROK ,"Failed to set break at the original position" ); +} + +static void setup (void )
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 134
Quoted 10 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +10
-0+{ +if (!Hopt) + Hopt =tst_get_tmpdir (); +SAFE_MOUNT ("none" , Hopt,"hugetlbfs" ,0 ,NULL ); + +snprintf (hfile,sizeof (hfile),"%s/ltp_hugetlbfile%d" , Hopt,getpid ()); + hpage_size =SAFE_READ_MEMINFO ("Hugepagesize:" )*1024 ; + + fd =SAFE_OPEN (hfile,O_RDWR |O_CREAT ,0600 ); +SAFE_UNLINK (hfile);
I'm guessing opening this file and using it with mmap is a very common pattern. If so, it should be encapsulated in tst_hugepage.c.
Quoted 4 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +4
-0+} + +static void cleanup (void ) +{
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 148
Quoted 10 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +10
-0+if (fd >=0 ) +SAFE_CLOSE (fd); +SAFE_UMOUNT (Hopt); +} + +static struct tst_test test = { + .needs_root =1 , + .needs_tmpdir =1 , + .options = (struct tst_option []) { + {"H:" , &Hopt,"Location of hugetlbfs, i.e. -H /var/hugetlbfs" },
This is a source of confusion. This description suggests we pass in an existing hugetlb mount. However it's actually where it will be mounted.
Perhaps instead we could use set/getmntent to find an existing hugetlb mount? Then if it is not there, try mounting it. This is what we do for CGroups.
I'm not sure what difference it makes where the FS is mounted anyway. Why is it even an option?
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 155
Quoted 7 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +7
-0+ {"s:", &nr_opt, "Set the number of the been allocated hugepages"}, + {} + }, + .setup = setup, + .cleanup = cleanup, + .test_all = run_test, + .hugepages = {1, TST_NEEDS},
When we set this tst_hugepages.c could fill Hopt (which should be something like tst_hugepages_mount) with the location of the mount point. It could also open the hfile fd and store it in a static variable like tst_hugepage_fd.
Also .taint_check should be added here to check for the type of Ooops that are caused by this test. This makes debugging easier if the kernel doesn't kill the test process or freeze immediately.
Quoted 17 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +1
-0+};testcases/kernel/mem/hugetlb/lib/hugetlb.h +3
-0@@ -20,6 +20,9 @@#include "old_tmpdir.h" #include "mem.h" +#define ALIGN (x, a) (((x) + (a) - 1) & ~((a) - 1)) +#define PALIGN (p, a) ((void *)ALIGN((unsigned long)(p), (a))) +#define SHM_RD 0400#define SHM_WR 0200#define SHM_RW (SHM_RD|SHM_WR)Signature · 1 line
2.31.1
Signature · 5 lines
Thank you, Richard. -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v2 1/3] Hugetlb: Migrating libhugetlbfs brk_near_huge
Hi Richard,
Thanks for reviewing the patch.
-- skip
Quoted 8 lines
Quoted 1 line
+static int
arch_has_slice_support(void)This only appears to be used on
__powerpc64__innext_chunk. So it can
be defined in the ifdef fornext_chunk.Otherwise we could get unused function warnings and IMO it is more readable.
Yeah, I will update it in next version.
Quoted 11 lines
Quoted 11 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +7
-0+{ +#ifdef __powerpc64__ +char mmu_type[16 ]; +FILE *fp; +int ret; + + fp = SAFE_POPEN("cat /proc/cpuinfo | grep MMU | awk '{ print$3}'", "r");
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +2
-0+ ret =fscanf (fp,"%s" , mmu_type); +pclose (fp);
Quoted 8 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 47
Quoted 6 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +6
-0+ +if (ret <0 ) +tst_brk (TBROK ,"Failed to determine MMU type" ); + +return strcmp (mmu_type,"Hash" ) ==0 ; +#elif defined(__powerpc__) && !defined(PPC_NO_SEGMENTS )This elif doesn't appear to be applicable.
I missed adding PPC_NO_SEGMENTS cppflags in Makefile Will update it in next version.
Quoted 15 lines
Quoted 12 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +11
-0+return 1 ; +#else +return 0 ; +#endif +} + +#ifdef __powerpc64__ +static void *next_chunk (void *addr) +{ +if (!arch_has_slice_support ()) + return PALIGN(addr,Quoted 1 line
SAFE_READ_MEMINFO("Hugepagesize:")*1024);
In setup we set
hpage_size, but then keep reading it.
Yeah, I missed it, Will update it.
Quoted 19 lines
Quoted 19 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +18
-0+ +if ((unsigned long )addr <0x100000000UL ) +/* 256M segments below 4G */ +return PALIGN (addr,0x10000000UL ); +/* 1TB segments above */ + return PALIGN(addr, 0x10000000000UL); +} +#elif defined(__powerpc__) && !defined(PPC_NO_SEGMENTS) +static void *next_chunk (void *addr) +{ +return PALIGN (addr,0x10000000UL ); +} +#elif defined(__ia64__) +static void *next_chunk (void *addr) +{ +return PALIGN (addr,0x8000000000000000UL ); +} +#else
Quoted 8 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 81
Quoted 5 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +5
-0+static void *next_chunk (void *addr) +{ +return PALIGN (addr,SAFE_READ_MEMINFO ("Hugepagesize:" )*1024 ); +} +#endif If these functions are used in later tests I guess they should go in
tst_hugepages.h
No, These functions are only used in this test.
Quoted 12 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 134
Quoted 7 lines
(quoted diff) +1
-0+ snprintf(hfile, sizeof(hfile), "%s/ltp_hugetlbfile%d", Hopt,
getpid());testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +4
-0+ hpage_size =SAFE_READ_MEMINFO ("Hugepagesize:" )*1024 ; + + fd =SAFE_OPEN (hfile,O_RDWR |O_CREAT ,0600 ); +SAFE_UNLINK (hfile);I'm guessing opening this file and using it with mmap is a very common
pattern. If so, it should be encapsulated intst_hugepage.c.
Yeah I agree. But the change in tst_hugepage.c will require change in prexisting hugetlb tests.
Quoted 21 lines
Quoted 14 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +14
-0+} + +static void cleanup (void ) +{ +if (fd >=0 ) +SAFE_CLOSE (fd); +SAFE_UMOUNT (Hopt); +} + +static struct tst_test test = { + .needs_root =1 , + .needs_tmpdir =1 , + .options = (struct tst_option []) { + {"H:", &Hopt, "Location of hugetlbfs, i.e. -HQuoted 1 line
/var/hugetlbfs"},This is a source of confusion. This description suggests we pass in an
existing hugetlb mount. However it's actually where it will be mounted.
Quoted 8 lines
Perhaps instead we could use set/getmntent to find an existing hugetlb
mount? Then if it is not there, try mounting it. This is what we do for
CGroups.I'm not sure what difference it makes where the FS is mounted anyway. Why is it even an option?
Not sure, If user is ok for using premounted fs without their permission. So instead of searching, it will mount where user will explicitly asked for. Or otherwise if not provided, it will mount at temp location under /tmp.
I had taken this option from hugemmap01 test. Thinking, it might be to provide user the flexibility incase user doesnt want test to mount fs at random location by default.
I will change the description to "Provide the location to mount the hugetlbfs or default '/tmp/xxxxxxx'"
Quoted 14 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 155
Quoted 9 lines
(quoted diff) +1
-0+ {"s:", &nr_opt, "Set the number of the been allocatedhugepages"},
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +6
-0+ {} + }, + .setup = setup, + .cleanup = cleanup, + .test_all = run_test, + .hugepages = {1, TST_NEEDS},When we set this
tst_hugepages.ccould fill Hopt (which should be something liketst_hugepages_mount) with the location of the mount point. It could also open the hfile fd and store it in a static variable liketst_hugepage_fd.
Yeah, It will move the repeated actions to single location.
This will
require change in lib/tst_hugepage.c and which will require change in already existing test under hugetlb/ . Which will be like a new separate change patch series.
Quoted 5 lines
Also .taint_check should be added here to check for the type of Ooops that are caused by this test. This makes debugging easier if the kernel
doesn't kill the test process or freeze immediately.
Ok, Will update it in next version.
Quoted 21 lines
Quoted 19 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +1
-0+};testcases/kernel/mem/hugetlb/lib/hugetlb.h +3
-0@@ -20,6 +20,9 @@#include "old_tmpdir.h" #include "mem.h" +#define ALIGN (x, a) (((x) + (a) - 1) & ~((a) - 1)) +#define PALIGN (p, a) ((void *)ALIGN((unsigned long)(p), (a))) +#define SHM_RD 0400#define SHM_WR 0200#define SHM_RW (SHM_RD|SHM_WR)Signature · 1 line
2.31.1
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v2 1/3] Hugetlb: Migrating libhugetlbfs brk_near_huge
Hello,
Quoted 13 lines · Tarun Sahu
Quoted 13 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 134
Quoted 8 lines
(quoted diff) +1
-0+ snprintf(hfile, sizeof(hfile), "%s/ltp_hugetlbfile%d", Hopt,
getpid());testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +4
-0+ hpage_size =SAFE_READ_MEMINFO ("Hugepagesize:" )*1024 ; + + fd =SAFE_OPEN (hfile,O_RDWR |O_CREAT ,0600 ); +SAFE_UNLINK (hfile);I'm guessing opening this file and using it with mmap is a very common
pattern. If so, it should be encapsulated intst_hugepage.c.
Quoted 2 lines
Yeah I agree. But the change in
tst_hugepage.cwill require change in prexisting hugetlb tests.
You don't need to update existing LTP tests if you add a new flag to .tst_test I guess.
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +14
-0+} + +static void cleanup (void ) +{ +if (fd >=0 ) +SAFE_CLOSE (fd); +SAFE_UMOUNT (Hopt); +} + +static struct tst_test test = { + .needs_root =1 , + .needs_tmpdir =1 , + .options = (struct tst_option []) { + {"H:", &Hopt, "Location of hugetlbfs, i.e. -HQuoted 1 line
/var/hugetlbfs"},This is a source of confusion. This description suggests we pass in an
existing hugetlb mount. However it's actually where it will be mounted.Perhaps instead we could use set/getmntent to find an existing hugetlb
mount? Then if it is not there, try mounting it. This is what we do for
CGroups.
Quoted 7 lines
Quoted 2 lines
I'm not sure what difference it makes where the FS is mounted anyway. Why is it even an option?
Not sure, If user is ok for using premounted fs without their permission. So instead of searching, it will mount where user will explicitly asked for. Or otherwise if not provided, it will mount at temp location under /tmp.
Does it actually create a new FS or is it just remounting the existing hugetlb interface in a new place?
Also it requires privileges to mount an FS. It seems unlikely that some database wanting to use hugepages would mount it themselves.
From the kernel docs:
"If the user applications are going to request huge pages using mmap system call, then it is required that system administrator mount a file system of type hugetlbfs"
Quoted 22 lines
I had taken this option from hugemmap01 test. Thinking, it might be to provide user the flexibility incase user doesnt want test to mount fs at random location by default.
I will change the description to "Provide the location to mount the hugetlbfs or default '
/tmp/xxxxxxx'"Quoted 14 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 155
Quoted 9 lines
(quoted diff) +1
-0+ {"s:", &nr_opt, "Set the number of the been allocatedhugepages"},
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +6
-0+ {} + }, + .setup = setup, + .cleanup = cleanup, + .test_all = run_test, + .hugepages = {1, TST_NEEDS},When we set this
tst_hugepages.ccould fill Hopt (which should be something liketst_hugepages_mount) with the location of the mount point. It could also open the hfile fd and store it in a static variable liketst_hugepage_fd.
Quoted 6 lines
Yeah, It will move the repeated actions to single location.
This will
require change inlib/tst_hugepage.cand which will require change in already existing test under hugetlb/ . Which will be like a new separate change patch series.
If it results in too many complicated changes, then converting all the existing tests could be defered to a later time by creating new variables.
Signature · 5 lines
Thank you, Richard. -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v2 1/3] Hugetlb: Migrating libhugetlbfs brk_near_huge
Hello,
Quoted 15 lines
Quoted 12 lines · Tarun Sahu
Quoted 12 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 134
Quoted 7 lines
(quoted diff) +1
-0+ snprintf(hfile, sizeof(hfile), "%s/ltp_hugetlbfile%d",Hopt,
getpid());testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +4
-0+ hpage_size =SAFE_READ_MEMINFO ("Hugepagesize:" )*1024 ; + + fd =SAFE_OPEN (hfile,O_RDWR |O_CREAT ,0600 ); +SAFE_UNLINK (hfile);I'm guessing opening this file and using it with mmap is a very common
pattern. If so, it should be encapsulated intst_hugepage.c.
Quoted 6 lines
Quoted 3 lines
Yeah I agree. But the change in
tst_hugepage.cwill require change in
prexisting hugetlb tests.You don't need to update existing LTP tests if you add a new flag to .tst_test I guess.
yes.
I will add new tst_test flag option for Mount and open file setup in next version.
Quoted 48 lines
Quoted 31 lines
Quoted 31 lines
Quoted 15 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +14
-0+} + +static void cleanup (void ) +{ +if (fd >=0 ) +SAFE_CLOSE (fd); +SAFE_UMOUNT (Hopt); +} + +static struct tst_test test = { + .needs_root =1 , + .needs_tmpdir =1 , + .options = (struct tst_option []) { + {"H:", &Hopt, "Location of hugetlbfs, i.e. -Quoted 2 lines
H
/var/hugetlbfs"},This is a source of confusion. This description suggests we pass in
an
existing hugetlb mount. However it's actually where it will be mounted.Perhaps instead we could use set/getmntent to find an existing hugetlb
mount? Then if it is not there, try mounting it. This is what we do
for
CGroups.Quoted 8 lines
Quoted 2 lines
I'm not sure what difference it makes where the FS is mounted anyway. Why is it even an option?
Not sure, If user is ok for using premounted fs without their permission. So instead of searching, it will mount where user will explicitly asked for. Or otherwise if not provided, it will mount at
temp location under /tmp.Does it actually create a new FS or is it just remounting the existing
hugetlb interface in a new place?Also it requires privileges to mount an FS. It seems unlikely that some
database wanting to use hugepages would mount it themselves.
Quoted 7 lines
From the kernel docs:
"If the user applications are going to request huge pages using mmap system
call, then it is required that system administrator mount a file system of
type hugetlbfs"
I agree.
But this will make Hopt option compulsory for user to provide. and during batch running of tests (/opt/ltp/runltp -f hugetlb), user will have to modify each test line in /opt/ltp/runtest to provide the particular mounted option to get the tests running.
Instead, I propose, to follow the same mechanism as the older hugetlb test follow, create a random temp location under /tmp (/tmp/xxxxxx), and mount and umount hugetlbfs as needed. For this way, I will also remove the unnecessary Hopt option. Though need_root will become pre-requisite to run the test.
Quoted 32 lines
Quoted 24 lines
I had taken this option from hugemmap01 test. Thinking, it might be to provide user the flexibility incase user doesnt want test to mount
fs at random location by default.I will change the description to "Provide the location to mount the hugetlbfs or default '
/tmp/xxxxxxx'"Quoted 7 lines
Quoted 7 lines
(quoted diff) +1
-0+ {"s:", &nr_opt, "Set the number of the beenallocated
hugepages"},testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +4
-0+ {} + }, + .setup = setup, + .cleanup = cleanup,Quoted 8 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c line 155
Quoted 2 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap07.c new file +2
-0+ .test_all = run_test, + .hugepages = {1, TST_NEEDS},When we set this
tst_hugepages.ccould fill Hopt (which should be something liketst_hugepages_mount) with the location of the mount
point. It could also open the hfile fd and store it in a static variable liketst_hugepage_fd.Quoted 7 lines
Yeah, It will move the repeated actions to single location.
This will
require change inlib/tst_hugepage.cand which will require change in already existing test under hugetlb/ . Which will be like a
new separate change patch series.
Quoted 4 lines
If it results in too many complicated changes, then converting all the
existing tests could be defered to a later time by creating new variables.
Ok.
Quoted 1 line
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH v2 2/3 Hugetlb: Migrating libhugetlbfs chunk-overcommit
In reply to Tarun SahuMigrating the libhugetlbfs/testcases/chunk-overcommit.c test
Test Description: Some kernel versions after hugepage demand allocation was added used a dubious heuristic to check if there was enough hugepage space available for a given mapping. The number of not-already-instantiated pages in the mapping was compared against the total hugepage free pool. It was very easy to confuse this heuristic into overcommitting by allocating hugepage memory in chunks, each less than the total available pool size but together more than available. This would generally lead to OOM SIGKILLs of one process or another when it tried to instantiate pages beyond the available pool.
--- runtest/hugetlb | 1 + testcases/kernel/mem/.gitignore | 1 + .../kernel/mem/hugetlb/hugemmap/hugemmap08.c | 166 ++++++++++++++++++ 3 files changed, 168 insertions(+) create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c
runtest/hugetlb +1 -0
@@ -4,6 +4,7 @@ hugemmap04 hugemmap04 hugemmap05 hugemmap05 hugemmap06 hugemmap06 hugemmap07 hugemmap07 +hugemmap08 hugemmap08 hugemmap05_1 hugemmap05 -m hugemmap05_2 hugemmap05 -s hugemmap05_3 hugemmap05 -s -m
testcases/kernel/mem/.gitignore +1 -0
@@ -5,6 +5,7 @@ /hugetlb/hugemmap/hugemmap05 /hugetlb/hugemmap/hugemmap06 /hugetlb/hugemmap/hugemmap07 +/hugetlb/hugemmap/hugemmap08 /hugetlb/hugeshmat/hugeshmat01 /hugetlb/hugeshmat/hugeshmat02 /hugetlb/hugeshmat/hugeshmat03
testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file +166 -0
166 lines not on this page · raw message
Signature · 5 lines
2.31.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH v2 3/3 Hugetlb: Migrating libhugetlbfs corrupt-by-cow-opt
In reply to Tarun SahuMigrating the libhugetlbfs/testcases/corrupt-by-cow-opt.c test
Test Description: Test sanity of cow optimization on page cache. If a page in page cache has only 1 ref count, it is mapped for a private mapping directly and is overwritten freely, so next time we access the page, we can see corrupt data.
--- runtest/hugetlb | 1 + testcases/kernel/mem/.gitignore | 1 + .../kernel/mem/hugetlb/hugemmap/hugemmap09.c | 90 +++++++++++++++++++ 3 files changed, 92 insertions(+) create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c
runtest/hugetlb +1 -0
@@ -5,6 +5,7 @@ hugemmap05 hugemmap05 hugemmap06 hugemmap06 hugemmap07 hugemmap07 hugemmap08 hugemmap08 +hugemmap09 hugemmap09 hugemmap05_1 hugemmap05 -m hugemmap05_2 hugemmap05 -s hugemmap05_3 hugemmap05 -s -m
testcases/kernel/mem/.gitignore +1 -0
@@ -6,6 +6,7 @@ /hugetlb/hugemmap/hugemmap06 /hugetlb/hugemmap/hugemmap07 /hugetlb/hugemmap/hugemmap08 +/hugetlb/hugemmap/hugemmap09 /hugetlb/hugeshmat/hugeshmat01 /hugetlb/hugeshmat/hugeshmat02 /hugetlb/hugeshmat/hugeshmat03
testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c new file +90 -0
90 lines not on this page · raw message
Signature · 5 lines
2.31.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v2 3/3] Hugetlb: Migrating libhugetlbfs corrupt-by-cow-opt
Hello,
Quoted 6 lines · Tarun Sahu
Migrating the
libhugetlbfs/testcases/corrupt-by-cow-opt.ctestTest Description: Test sanity of cow optimization on page cache. If a page in page cache has only 1 ref count, it is mapped for a private mapping directly and is overwritten freely, so next time we access the page, we can see corrupt data.
Seems like this and 2/3 follow the same pattern. The setups are reasonably similar and could be encapsulated in tst_hugepage.
Quoted 8 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c new file +7
-0+ +static struct tst_test test = { + .needs_root =1 , + .needs_tmpdir =1 , + .options = (struct tst_option []) { + {"H:" , &Hopt,"Location of hugetlbfs, i.e. -H /var/hugetlbfs" }, + {"s:", &nr_opt, "Set the number of the been allocatedhugepages"},
nr_opt also seems suspicious. The test only ever allocates one page at a time regardless of what this is set to. Changing it will just change how much free memory we check for unless I am mistaken.
Quoted 9 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c new file +7
-0+ {} + }, + .setup = setup, + .cleanup = cleanup, + .test_all = run_test, + .hugepages = {2, TST_NEEDS}, +};Signature · 1 line
2.31.1
Signature · 5 lines
Thank you, Richard. -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v2 3/3] Hugetlb: Migrating libhugetlbfs corrupt-by-cow-opt
Quoted 14 lines · Richard Palethorpe
Hello,
Quoted 9 lines · Tarun Sahu
Migrating the
libhugetlbfs/testcases/corrupt-by-cow-opt.ctestTest Description: Test sanity of cow optimization on page cache. If a page
in page cache has only 1 ref count, it is mapped for a private mapping
directly and is overwritten freely, so next time we access the page, we
can see corrupt data.
Quoted 2 lines
Seems like this and 2/3 follow the same pattern. The setups are reasonably similar and could be encapsulated in
tst_hugepage.
Do you mean by a encapsulating in a function. and call it from setup. becuase it will anyway require explicit cleanup.
Or by defining a new field in struct tst_hugepage, if that is true, that setup will automatically be done in do_setup in tst_test file. which means it will require change in tst_test.c too. also change in do_cleanup in tst_test.c will also be required.
Quoted 16 lines
Quoted 10 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c new file +6
-0+ +static struct tst_test test = { + .needs_root =1 , + .needs_tmpdir =1 , + .options = (struct tst_option []) { + {"H:", &Hopt, "Location of hugetlbfs, i.e. -H
/var/hugetlbfs"},(quoted diff) +1
-0+ {"s:", &nr_opt, "Set the number of the been allocatedhugepages"},
nr_optalso seems suspicious. The test only ever allocates one page at a
time regardless of what this is set to. Changing it will just change how
much free memory we check for unless I am mistaken.
yes, Will update it and also will check for other test cases if not required.
Quoted 12 lines
Quoted 10 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c new file +7
-0+ {} + }, + .setup = setup, + .cleanup = cleanup, + .test_all = run_test, + .hugepages = {2, TST_NEEDS}, +};Signature · 1 line
2.31.1
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v2 3/3] Hugetlb: Migrating libhugetlbfs corrupt-by-cow-opt
Hello,
Quoted 17 lines · Tarun Sahu
Quoted 14 lines · Richard Palethorpe
Hello,
Quoted 9 lines · Tarun Sahu
Migrating the
libhugetlbfs/testcases/corrupt-by-cow-opt.ctestTest Description: Test sanity of cow optimization on page cache. If a page
in page cache has only 1 ref count, it is mapped for a private mapping
directly and is overwritten freely, so next time we access the page, we
can see corrupt data.Quoted 2 lines
Seems like this and 2/3 follow the same pattern. The setups are reasonably similar and could be encapsulated in
tst_hugepage.
Quoted 7 lines
Do you mean by a encapsulating in a function. and call it from setup. becuase it will anyway require explicit cleanup.
Or by defining a new field in struct
tst_hugepage, if that is true, that setup will automatically be done indo_setupintst_testfile. which means it will require change intst_test.ctoo. also change indo_cleanupintst_test.cwill also be required.
Yes, it's a very common pattern, so will probably save a lot of boilerplate.
Quoted 32 lines
Quoted 17 lines
Quoted 11 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c new file +6
-0+ +static struct tst_test test = { + .needs_root =1 , + .needs_tmpdir =1 , + .options = (struct tst_option []) { + {"H:", &Hopt, "Location of hugetlbfs, i.e. -H
/var/hugetlbfs"},(quoted diff) +1
-0+ {"s:", &nr_opt, "Set the number of the been allocatedhugepages"},
nr_optalso seems suspicious. The test only ever allocates one page at a
time regardless of what this is set to. Changing it will just change how
much free memory we check for unless I am mistaken.yes, Will update it and also will check for other test cases if not required.
Quoted 12 lines
Quoted 10 lines
testcases/kernel/mem/hugetlb/hugemmap/hugemmap09.c new file +7
-0+ {} + }, + .setup = setup, + .cleanup = cleanup, + .test_all = run_test, + .hugepages = {2, TST_NEEDS}, +};Signature · 1 line
2.31.1
Signature · 5 lines
Thank you, Richard. -- Mailing list info: https://lists.linux.it/listinfo/ltp