eal: allow hugetlbfs sub-directories
PATCH 1/1 eal: allow hugetlbfs sub-directories
get_hugepage_dir() was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified.
--- lib/eal/linux/eal_hugepage_info.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-)
lib/eal/linux/eal_hugepage_info.c +19 -6
@@ -226,16 +226,29 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len) default_size =get_default_hp_size ();while (fgets (buf,sizeof (buf), fd)){ +const char *dir; +if (rte_strsplit (buf,sizeof (buf), splitstr, _FIELDNAME_MAX, split_tok) != _FIELDNAME_MAX) {RTE_LOG (ERR ,EAL ,"Error parsing %s\n" , proc_mounts);break ;/* return NULL */ } -/* we have a specified --huge-dir option, only examine that dir */ -if (internal_conf->hugepage_dir !=NULL && -strcmp (splitstr[MOUNTPT ], internal_conf->hugepage_dir) !=0 ) -continue ; + dir = splitstr[MOUNTPT ]; + +/* +* If a --huge-dir option has been specified, only examine +* mounts that contain that directory, and make sure to return +* the directory, not the mount. +*/ +if (internal_conf->hugepage_dir !=NULL ) { +if (strncmp (internal_conf->hugepage_dir, + splitstr[MOUNTPT ], +strlen (splitstr[MOUNTPT ])) !=0 ) +continue ; + + dir = internal_conf->hugepage_dir; + }if (strncmp (splitstr[FSTYPE ], hugetlbfs_str, htlbfs_str_len) ==0 ){const char *pagesz_str =strstr (splitstr[OPTIONS ], pagesize_opt);
@@ -243,7 +256,7 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len)/* if no explicit page size, the default page size is compared */ if (pagesz_str ==NULL ){if (hugepage_sz == default_size){ -strlcpy (hugedir, splitstr[MOUNTPT ], len); +strlcpy (hugedir, dir, len); retval =0 ;break ; }
@@ -252,7 +265,7 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len)else {uint64_t pagesz =rte_str_to_size (&pagesz_str[pagesize_opt_len]);if (pagesz == hugepage_sz) { -strlcpy (hugedir, splitstr[MOUNTPT ], len); +strlcpy (hugedir, dir, len); retval =0 ;break ; }
Signature · 1 line
2.25.1
PATCH RESEND 1/1 eal: allow hugetlbfs sub-directories
get_hugepage_dir() was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified.
--- lib/eal/linux/eal_hugepage_info.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-)
lib/eal/linux/eal_hugepage_info.c +19 -6
@@ -226,16 +226,29 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len) default_size =get_default_hp_size ();while (fgets (buf,sizeof (buf), fd)){ +const char *dir; +if (rte_strsplit (buf,sizeof (buf), splitstr, _FIELDNAME_MAX, split_tok) != _FIELDNAME_MAX) {RTE_LOG (ERR ,EAL ,"Error parsing %s\n" , proc_mounts);break ;/* return NULL */ } -/* we have a specified --huge-dir option, only examine that dir */ -if (internal_conf->hugepage_dir !=NULL && -strcmp (splitstr[MOUNTPT ], internal_conf->hugepage_dir) !=0 ) -continue ; + dir = splitstr[MOUNTPT ]; + +/* +* If a --huge-dir option has been specified, only examine +* mounts that contain that directory, and make sure to return +* the directory, not the mount. +*/ +if (internal_conf->hugepage_dir !=NULL ) { +if (strncmp (internal_conf->hugepage_dir, + splitstr[MOUNTPT ], +strlen (splitstr[MOUNTPT ])) !=0 ) +continue ; + + dir = internal_conf->hugepage_dir; + }if (strncmp (splitstr[FSTYPE ], hugetlbfs_str, htlbfs_str_len) ==0 ){const char *pagesz_str =strstr (splitstr[OPTIONS ], pagesize_opt);
@@ -243,7 +256,7 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len)/* if no explicit page size, the default page size is compared */ if (pagesz_str ==NULL ){if (hugepage_sz == default_size){ -strlcpy (hugedir, splitstr[MOUNTPT ], len); +strlcpy (hugedir, dir, len); retval =0 ;break ; }
@@ -252,7 +265,7 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len)else {uint64_t pagesz =rte_str_to_size (&pagesz_str[pagesize_opt_len]);if (pagesz == hugepage_sz) { -strlcpy (hugedir, splitstr[MOUNTPT ], len); +strlcpy (hugedir, dir, len); retval =0 ;break ; }
Signature · 1 line
2.25.1
Hello Anatoly,
Quoted 5 lines · John Levon
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified.
Review please.
Quoted 2 lines
Signed-off-by John Levon <[email protected]>
Signature · 1 line
David Marchand
2021-06-25 11:44 (UTC+0100), John Levon:
Quoted 35 lines
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified.Signed-off-by John Levon <[email protected]>--- lib/eal/linux/eal_hugepage_info.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-)lib/eal/linux/eal_hugepage_info.c +7
-4@@ -226,16 +226,29 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len) default_size =get_default_hp_size ();while (fgets (buf,sizeof (buf), fd)){ +const char *dir; +if (rte_strsplit (buf,sizeof (buf), splitstr, _FIELDNAME_MAX, split_tok) != _FIELDNAME_MAX) {RTE_LOG (ERR ,EAL ,"Error parsing %s\n" , proc_mounts);break ;/* return NULL */ } -/* we have a specified --huge-dir option, only examine that dir */ -if (internal_conf->hugepage_dir !=NULL && -strcmp (splitstr[MOUNTPT ], internal_conf->hugepage_dir) !=0 ) -continue ; + dir = splitstr[MOUNTPT ]; + +/* +* If a --huge-dir option has been specified, only examine +* mounts that contain that directory, and make sure to return
lib/eal/linux/eal_hugepage_info.c get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len) · line 251
Quoted 10 lines
lib/eal/linux/eal_hugepage_info.c +10
-0+* the directory, not the mount. +*/ +if (internal_conf->hugepage_dir !=NULL ) { +if (strncmp (internal_conf->hugepage_dir, + splitstr[MOUNTPT ], +strlen (splitstr[MOUNTPT ])) !=0 ) +continue ; + + dir = internal_conf->hugepage_dir; + }
Suppose there are /mnt/huge and /mnt/huge-2M mounted with -o pagesize=2M. If --huge-dir=/mnt/huge-2M is specified, this code will select /mnt/huge. The code would be shorter with splitstr[MOUNTPT] -> dir.
Quoted 40 lines · Dmitry Kozlyuk
Quoted 40 lines
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified.Signed-off-by John Levon <[email protected]>--- lib/eal/linux/eal_hugepage_info.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-)lib/eal/linux/eal_hugepage_info.c +12
-4@@ -226,16 +226,29 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len) default_size =get_default_hp_size ();while (fgets (buf,sizeof (buf), fd)){ +const char *dir; +if (rte_strsplit (buf,sizeof (buf), splitstr, _FIELDNAME_MAX, split_tok) != _FIELDNAME_MAX) {RTE_LOG (ERR ,EAL ,"Error parsing %s\n" , proc_mounts);break ;/* return NULL */ } -/* we have a specified --huge-dir option, only examine that dir */ -if (internal_conf->hugepage_dir !=NULL && -strcmp (splitstr[MOUNTPT ], internal_conf->hugepage_dir) !=0 ) -continue ; + dir = splitstr[MOUNTPT ]; + +/* +* If a --huge-dir option has been specified, only examine +* mounts that contain that directory, and make sure to return +* the directory, not the mount. +*/ +if (internal_conf->hugepage_dir !=NULL ) { + if (strncmp(internal_conf->hugepage_dir, + splitstr[MOUNTPT],
Quoted 8 lines
lib/eal/linux/eal_hugepage_info.c get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len) · line 251
Quoted 5 lines
lib/eal/linux/eal_hugepage_info.c +5
-0+ strlen(splitstr[MOUNTPT])) != 0) +continue ; + + dir = internal_conf->hugepage_dir; + }Suppose there are
/mnt/hugeand/mnt/huge-2Mmounted with -o pagesize=2M. If --huge-dir=/mnt/huge-2M is specified, this code will select/mnt/huge.
Yeah, good spot. I'll change it to prefer closer matches or something.
Quoted 1 line
The code would be shorter with
splitstr[MOUNTPT]->dir.
Sure.
regards
john
PATCH 1/1 eal: allow hugetlbfs sub-directories
In reply to Dmitry Kozlyukv2: fix to prefer closest match
PATCH 1/1 eal: allow hugetlbfs sub-directories
get_hugepage_dir() was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.
--- lib/eal/linux/eal_hugepage_info.c | 74 +++++++++++++++++++++---------- 1 file changed, 51 insertions(+), 23 deletions(-)
lib/eal/linux/eal_hugepage_info.c +51 -23
@@ -213,8 +213,8 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len)const size_t pagesize_opt_len =sizeof (pagesize_opt) -1 ;const char split_tok =' ' ;char *splitstr[_FIELDNAME_MAX]; +char found[PATH_MAX ] ="" ;char buf[BUFSIZ ]; -int retval =-1 ;const struct internal_config *internal_conf =eal_get_internal_configuration ();
@@ -226,42 +226,70 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len) default_size =get_default_hp_size ();while (fgets (buf,sizeof (buf), fd)){ +const char *pagesz_str; +if (rte_strsplit (buf,sizeof (buf), splitstr, _FIELDNAME_MAX, split_tok) != _FIELDNAME_MAX) {RTE_LOG (ERR ,EAL ,"Error parsing %s\n" , proc_mounts);break ;/* return NULL */ } -/* we have a specified --huge-dir option, only examine that dir */ -if (internal_conf->hugepage_dir !=NULL && -strcmp (splitstr[MOUNTPT ], internal_conf->hugepage_dir) !=0 ) +if (strncmp (splitstr[FSTYPE ], hugetlbfs_str, htlbfs_str_len) !=0 ) {continue ; + } -if (strncmp (splitstr[FSTYPE ], hugetlbfs_str, htlbfs_str_len) ==0 ){ -const char *pagesz_str =strstr (splitstr[OPTIONS ], pagesize_opt); + pagesz_str =strstr (splitstr[OPTIONS ], pagesize_opt); -/* if no explicit page size, the default page size is compared */ -if (pagesz_str ==NULL ){ -if (hugepage_sz == default_size){ -strlcpy (hugedir, splitstr[MOUNTPT ], len); - retval =0 ; -break ; - } +/* if no explicit page size, the default page size is compared */ +if (pagesz_str ==NULL ) { +if (hugepage_sz != default_size) { +continue ; } -/* there is an explicit page size, so check it */ -else { -uint64_t pagesz =rte_str_to_size (&pagesz_str[pagesize_opt_len]); -if (pagesz == hugepage_sz) { -strlcpy (hugedir, splitstr[MOUNTPT ], len); - retval =0 ; -break ; - } + } +/* there is an explicit page size, so check it */ +else { +uint64_t pagesz =rte_str_to_size (&pagesz_str[pagesize_opt_len]); +if (pagesz != hugepage_sz) { +continue ; } - }/* end if strncmp hugetlbfs */ + } + +/* +* If no --huge-dir option has been given, we're done. +*/ +if (internal_conf->hugepage_dir ==NULL ) { +strlcpy (found, splitstr[MOUNTPT ], len); +break ; + } + +/* +* Ignore any mount that doesn't contain the --huge-dir +* directory. +*/ +if (strncmp (internal_conf->hugepage_dir, splitstr[MOUNTPT ], +strlen (splitstr[MOUNTPT ])) !=0 ) { +continue ; + } + +/* +* We found a match, but only prefer it if it's a longer match +* (so /mnt/1 is preferred over /mnt for matching /mnt/1/2)). +*/ +if (strlen (splitstr[MOUNTPT ]) >strlen (found)) { +strlcpy (found, splitstr[MOUNTPT ], len); + } }/* end while fgets */ fclose (fd); -return retval; + +if (found[0 ] !='\0' ) { +/* If needed, return the requested dir, not the mount point. */ +strlcpy (hugedir, internal_conf->hugepage_dir !=NULL ? + internal_conf->hugepage_dir : found, len); +return 0 ; + } + +return -1 ; }/*
Signature · 1 line
2.25.1
PATCH v3 1/1 eal: allow hugetlbfs sub-directories
In reply to Dmitry Kozlyukget_hugepage_dir() was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.
---
v2: prefer closer matches
v3: checkpatch fixes
lib/eal/linux/eal_hugepage_info.c | 74 ++++++++++++++++++++----------- 1 file changed, 49 insertions(+), 25 deletions(-)
lib/eal/linux/eal_hugepage_info.c +49 -25
@@ -213,8 +213,8 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len)const size_t pagesize_opt_len =sizeof (pagesize_opt) -1 ;const char split_tok =' ' ;char *splitstr[_FIELDNAME_MAX]; +char found[PATH_MAX ] ="" ;char buf[BUFSIZ ]; -int retval =-1 ;const struct internal_config *internal_conf =eal_get_internal_configuration ();
@@ -226,42 +226,66 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len) default_size =get_default_hp_size ();while (fgets (buf,sizeof (buf), fd)){ +const char *pagesz_str; +if (rte_strsplit (buf,sizeof (buf), splitstr, _FIELDNAME_MAX, split_tok) != _FIELDNAME_MAX) {RTE_LOG (ERR ,EAL ,"Error parsing %s\n" , proc_mounts);break ;/* return NULL */ } -/* we have a specified --huge-dir option, only examine that dir */ -if (internal_conf->hugepage_dir !=NULL && -strcmp (splitstr[MOUNTPT ], internal_conf->hugepage_dir) !=0 ) +if (strncmp (splitstr[FSTYPE ], hugetlbfs_str, htlbfs_str_len) !=0 )continue ; -if (strncmp (splitstr[FSTYPE ], hugetlbfs_str, htlbfs_str_len) ==0 ){ -const char *pagesz_str =strstr (splitstr[OPTIONS ], pagesize_opt); + pagesz_str =strstr (splitstr[OPTIONS ], pagesize_opt); -/* if no explicit page size, the default page size is compared */ -if (pagesz_str ==NULL ){ -if (hugepage_sz == default_size){ -strlcpy (hugedir, splitstr[MOUNTPT ], len); - retval =0 ; -break ; - } - } -/* there is an explicit page size, so check it */ -else { -uint64_t pagesz =rte_str_to_size (&pagesz_str[pagesize_opt_len]); -if (pagesz == hugepage_sz) { -strlcpy (hugedir, splitstr[MOUNTPT ], len); - retval =0 ; -break ; - } - } - }/* end if strncmp hugetlbfs */ +/* if no explicit page size, the default page size is compared */ +if (pagesz_str ==NULL ) { +if (hugepage_sz != default_size) +continue ; + } +/* there is an explicit page size, so check it */ +else { +uint64_t pagesz =rte_str_to_size (&pagesz_str[pagesize_opt_len]); +if (pagesz != hugepage_sz) +continue ; + } + +/* +* If no --huge-dir option has been given, we're done. +*/ +if (internal_conf->hugepage_dir ==NULL ) { +strlcpy (found, splitstr[MOUNTPT ], len); +break ; + } + +/* +* Ignore any mount that doesn't contain the --huge-dir +* directory. +*/ +if (strncmp (internal_conf->hugepage_dir, splitstr[MOUNTPT ], +strlen (splitstr[MOUNTPT ])) !=0 ) { +continue ; + } + +/* +* We found a match, but only prefer it if it's a longer match +* (so /mnt/1 is preferred over /mnt for matching /mnt/1/2)). +*/ +if (strlen (splitstr[MOUNTPT ]) >strlen (found)) +strlcpy (found, splitstr[MOUNTPT ], len); }/* end while fgets */ fclose (fd); -return retval; + +if (found[0 ] !='\0' ) { +/* If needed, return the requested dir, not the mount point. */ +strlcpy (hugedir, internal_conf->hugepage_dir !=NULL ? + internal_conf->hugepage_dir : found, len); +return 0 ; + } + +return -1 ; }/*
Signature · 1 line
2.25.1
2021-07-08 11:59 (UTC+0100), John Levon:
Quoted 7 lines
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.Signed-off-by John Levon <[email protected]>
Quoted 8 lines · John Levon
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.Signed-off-by John Levon <[email protected]>
This change in EAL hugetlbfs discovery is too dangerous to be taken after -rc1.
Could you give some usecases/examples on why this change is needed? Updating the documentation and the unit test also seem necessary.
Signature · 1 line
David Marchand
Quoted 11 lines · David Marchand
Quoted 8 lines · John Levon
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.Signed-off-by John Levon <[email protected]>This change in EAL hugetlbfs discovery is too dangerous to be taken after -rc1.
Sure.
Quoted 1 line
Could you give some usecases/examples on why this change is needed?
Would you like me to expand the commit message? I had hoped it was clear enough, but I suppose not. Simply put, DPDK above is assuming its the only user of hugepages on the system - including clear_hugedir(). That is certainly not the case for our use cases.
Quoted 1 line
Updating the documentation
https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html
"""
--huge-dir <path to hugetlbfs directory>
Use specified hugetlbfs directory instead of autodetected ones. """
That is, it already says "directory", not "mount". You'd like something additional saying it can be below a mount point?
Quoted 1 line
and the unit test also seem necessary.
You're talking about app/test/test_eal_flags.c or something else?
thanks,
john
22/07/2021 23:06, John Levon:
Quoted 12 lines · David Marchand
Quoted 11 lines · John Levon
Quoted 8 lines
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction. Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.Signed-off-by John Levon <[email protected]>
Quoted 7 lines
Quoted 1 line
This change in EAL hugetlbfs discovery is too dangerous to be taken after -rc1.
Sure.
Quoted 1 line
Could you give some usecases/examples on why this change is needed?
Would you like me to expand the commit message?
Yes please add some examples of directories explaining the issue you hit.
Quoted 9 lines
I had hoped it was clear enough,
but I suppose not. Simply put, DPDK above is assuming its the only user of hugepages on the system - includingclear_hugedir(). That is certainly not the case for our use cases.Quoted 1 line
Updating the documentation
https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html
Quoted 8 lines
"""
--huge-dir <path to hugetlbfs directory>Use specified hugetlbfs directory instead of autodetected ones. """
That is, it already says "directory", not "mount". You'd like something additional saying it can be below a mount point?
Yes
Quoted 3 lines
Quoted 1 line
and the unit test also seem necessary.
You're talking about
app/test/test_eal_flags.cor something else?
Yes
Thanks
PATCH 1/1 eal: allow hugetlbfs sub-directories
get_hugepage_dir() was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction: DPDK might not be the only user of hugepages, and shouldn't assume it owns an entire mountpoint. For example, if I have /dev/hugepages/myapp, and /dev/hugepages/dpdk, I should be able to specify:
--huge-dir=/dev/hugepages/dpdk/
and have DPDK only use that sub-directory.
Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.
---
v2: prefer closer matches
v3: checkpatch fixes
v4: fix docs, added tests
app/test/test_eal_flags.c | 116 ++++++++++++------ doc/guides/linux_gsg/linux_eal_parameters.rst | 3 +- lib/eal/linux/eal_hugepage_info.c | 83 +++++++++---- 3 files changed, 138 insertions(+), 64 deletions(-)
app/test/test_eal_flags.c +78 -38
@@ -14,8 +14,9 @@#include <errno.h> #include <unistd.h> #include <dirent.h> -#include <sys/wait.h> #include <sys/file.h> +#include <sys/stat.h> +#include <sys/wait.h> #include <limits.h> #include <fcntl.h>
@@ -800,6 +801,9 @@ static inttest_misc_flags (void) {char hugepath[PATH_MAX ] = {0 }; +char hugepath_dir[PATH_MAX ] = {0 }; +char hugepath_dir2[PATH_MAX ] = {0 }; +char hugepath_dir3[PATH_MAX ] = {0 };#ifdef RTE_EXEC_ENV_FREEBSD /* BSD target doesn't support prefixes at this point */ const char * prefix ="" ;
@@ -810,6 +814,7 @@ test_misc_flags(void)FILE * hugedir_handle =NULL ;char line[PATH_MAX ] = {0 };unsigned i, isempty =1 ; +if (get_current_prefix (tmp,sizeof (tmp)) ==NULL ) {printf ("Error - unable to get current prefix!\n" );return -1 ;
@@ -849,6 +854,20 @@ test_misc_flags(void) }#endif +snprintf (hugepath_dir,sizeof (hugepath_dir),"%s/dpdk.missing" , hugepath); +snprintf (hugepath_dir2,sizeof (hugepath_dir2),"%s/dpdk.dir" , hugepath); + +if (mkdir (hugepath_dir2,0700 ) !=0 && errno !=EEXIST ) { +printf ("Error - failed to mkdir(%s)\n" , hugepath_dir2); +return -1 ; + } + +snprintf (hugepath_dir3,sizeof (hugepath_dir3),"%s/dpdk.dir/sub" , hugepath); + +if (mkdir (hugepath_dir3,0700 ) !=0 && errno !=EEXIST ) { +printf ("Error - failed to mkdir(%s)\n" , hugepath_dir3); + goto fail; + }/* check that some general flags don't prevent things from working. * All cases, apart from the first, app should run.
@@ -881,60 +900,66 @@ test_misc_flags(void)/* With invalid --huge-dir */ const char *argv9[] = {prgname,"-m" ,DEFAULT_MEM_SIZE ,"--file-prefix=hugedir" ,"--huge-dir" ,"invalid" }; +/* With invalid --huge-dir sub-directory */ +const char *argv10[] = {prgname,"-m" ,DEFAULT_MEM_SIZE , +"--file-prefix=hugedir" ,"--huge-dir" , hugepath_dir}; +/* With valid --huge-dir sub-directory */ +const char *argv11[] = {prgname,"-m" ,DEFAULT_MEM_SIZE , +"--file-prefix=hugedir" ,"--huge-dir" , hugepath_dir2};/* Secondary process with invalid --huge-dir (should run as flag has no * effect on secondary processes) */ -const char *argv10[] = {prgname, prefix, mp_flag, +const char *argv12[] = {prgname, prefix, mp_flag,"--huge-dir" ,"invalid" };/* try running with base-virtaddr param */ -const char *argv11[] = {prgname,"--file-prefix=virtaddr" , +const char *argv13[] = {prgname,"--file-prefix=virtaddr" ,"--base-virtaddr=0x12345678" };/* try running with --vfio-intr INTx flag */ -const char *argv12[] = {prgname,"--file-prefix=intr" , +const char *argv14[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=legacy" };/* try running with --vfio-intr MSI flag */ -const char *argv13[] = {prgname,"--file-prefix=intr" , +const char *argv15[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=msi" };/* try running with --vfio-intr MSI-X flag */ -const char *argv14[] = {prgname,"--file-prefix=intr" , +const char *argv16[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=msix" };/* try running with --vfio-intr invalid flag */ -const char *argv15[] = {prgname,"--file-prefix=intr" , +const char *argv17[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=invalid" };/* With process type as auto-detect */ -const char *const argv16[] = {prgname,"--file-prefix=auto" , +const char *const argv18[] = {prgname,"--file-prefix=auto" ,"--proc-type=auto" };/* With process type as auto-detect with no-shconf */ -const char *const argv17[] = {prgname,"--proc-type=auto" , +const char *const argv19[] = {prgname,"--proc-type=auto" , no_shconf, nosh_prefix, no_huge};/* With process type as --create-uio-dev flag */ -const char *const argv18[] = {prgname,"--file-prefix=uiodev" , +const char *const argv20[] = {prgname,"--file-prefix=uiodev" ,"--create-uio-dev" };/* run all tests also applicable to FreeBSD first */ if (launch_proc (argv0) ==0 ) {printf ("Error - process ran ok with invalid flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv1) !=0 ) {printf ("Error - process did not run ok with --no-pci flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv2) !=0 ) {printf ("Error - process did not run ok with -v flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv6) !=0 ) {printf ("Error - process did not run ok with --no-shconf flag\n" ); -return -1 ; + goto fail; }#ifdef RTE_EXEC_ENV_FREEBSD
@@ -944,73 +969,88 @@ test_misc_flags(void)if (launch_proc (argv3) !=0 ) {printf ("Error - process did not run ok with --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv4) ==0 ) {printf ("Error - process run ok with empty --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv5) ==0 ) {printf ("Error - process run ok with invalid --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv7) !=0 ) {printf ("Error - process did not run ok with --huge-dir flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv8) ==0 ) {printf ("Error - process run ok with empty --huge-dir flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv9) ==0 ) {printf ("Error - process run ok with invalid --huge-dir flag\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv10) !=0 ) { -printf ("Error - secondary process did not run ok with invalid --huge-dir flag\n" ); -return -1 ; +if (launch_proc (argv10) ==0 ) { +printf ("Error - process run ok with invalid --huge-dir sub-dir flag\n" ); + goto fail; }if (launch_proc (argv11) !=0 ) { -printf ("Error - process did not run ok with --base-virtaddr parameter\n" ); -return -1 ; +printf ("Error - process did not run ok with --huge-dir subdir flag\n" ); + goto fail; }if (launch_proc (argv12) !=0 ) { +printf ("Error - secondary process did not run ok with invalid --huge-dir flag\n" ); + goto fail; + } +if (launch_proc (argv13) !=0 ) { +printf ("Error - process did not run ok with --base-virtaddr parameter\n" ); + goto fail; + } +if (launch_proc (argv14) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr INTx parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv13) !=0 ) { +if (launch_proc (argv15) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr MSI parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv14) !=0 ) { +if (launch_proc (argv16) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr MSI-X parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv15) ==0 ) { +if (launch_proc (argv17) ==0 ) {printf ("Error - process run ok with " "--vfio-intr invalid parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv16) !=0 ) { +if (launch_proc (argv18) !=0 ) {printf ("Error - process did not run ok with " "--proc-type as auto parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv17) !=0 ) { +if (launch_proc (argv19) !=0 ) {printf ("Error - process did not run ok with " "--proc-type and --no-shconf parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv18) !=0 ) { +if (launch_proc (argv20) !=0 ) {printf ("Error - process did not run ok with " "--create-uio-dev parameter\n" ); -return -1 ; + goto fail; } +rmdir (hugepath_dir3); +rmdir (hugepath_dir2);return 0 ; + +fail: +rmdir (hugepath_dir3); +rmdir (hugepath_dir2); +return -1 ; } static int
doc/guides/linux_gsg/linux_eal_parameters.rst +2 -1
@@ -81,7 +81,8 @@ Memory-related options * ``--huge-dir <path to hugetlbfs directory>`` - Use specified hugetlbfs directory instead of autodetected ones. + Use specified hugetlbfs directory instead of autodetected ones. This can be + a sub-directory within a hugetlbfs mountpoint. * ``--huge-unlink``
lib/eal/linux/eal_hugepage_info.c +58 -25
110 lines not on this page · raw message
Signature · 1 line
2.30.2
Quoted 8 lines · John Levon
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction: DPDK might not be the only user of hugepages, and shouldn't assume it owns an entire mountpoint. For example, if I have/dev/hugepages/myapp, and/dev/hugepages/dpdk, I should be able to specify:
Quoted 8 lines
--huge-dir=/dev/hugepages/dpdk/
and have DPDK only use that sub-directory.
Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.
Signed-off-by John Levon <[email protected]>
Cool feature.
Could you add to the release note as well when this patch gets merged
https://patches.dpdk.org/project/dpdk/patch/[email protected]/
Quoted 390 lines
---
v2: prefer closer matches
v3: checkpatch fixes
v4: fix docs, added testsapp/test/test_eal_flags.c | 116 ++++++++++++------ doc/guides/linux_gsg/linux_eal_parameters.rst | 3 +- lib/eal/linux/eal_hugepage_info.c | 83 +++++++++---- 3 files changed, 138 insertions(+), 64 deletions(-)app/test/test_eal_flags.c +78
-38@@ -14,8 +14,9 @@#include <errno.h> #include <unistd.h> #include <dirent.h> -#include <sys/wait.h> #include <sys/file.h> +#include <sys/stat.h> +#include <sys/wait.h> #include <limits.h> #include <fcntl.h> @@ -800,6 +801,9 @@ static inttest_misc_flags (void) {char hugepath[PATH_MAX ] = {0 }; +char hugepath_dir[PATH_MAX ] = {0 }; +char hugepath_dir2[PATH_MAX ] = {0 }; +char hugepath_dir3[PATH_MAX ] = {0 };#ifdef RTE_EXEC_ENV_FREEBSD /* BSD target doesn't support prefixes at this point */ const char * prefix ="" ;@@ -810,6 +814,7 @@ test_misc_flags(void)FILE * hugedir_handle =NULL ;char line[PATH_MAX ] = {0 };unsigned i, isempty =1 ; +if (get_current_prefix (tmp,sizeof (tmp)) ==NULL ) {printf ("Error - unable to get current prefix!\n" );return -1 ;@@ -849,6 +854,20 @@ test_misc_flags(void) }#endif +snprintf (hugepath_dir,sizeof (hugepath_dir),"%s/dpdk.missing" , hugepath); +snprintf (hugepath_dir2,sizeof (hugepath_dir2),"%s/dpdk.dir" , hugepath); + +if (mkdir (hugepath_dir2,0700 ) !=0 && errno !=EEXIST ) { +printf ("Error - failed to mkdir(%s)\n" , hugepath_dir2); +return -1 ; + } + +snprintf (hugepath_dir3,sizeof (hugepath_dir3),"%s/dpdk.dir/sub" , hugepath); + +if (mkdir (hugepath_dir3,0700 ) !=0 && errno !=EEXIST ) { +printf ("Error - failed to mkdir(%s)\n" , hugepath_dir3); + goto fail; + }/* check that some general flags don't prevent things from working. * All cases, apart from the first, app should run. @@ -881,60 +900,66 @@ test_misc_flags(void)/* With invalid --huge-dir */ const char *argv9[] = {prgname,"-m" ,DEFAULT_MEM_SIZE ,"--file-prefix=hugedir" ,"--huge-dir" ,"invalid" }; +/* With invalid --huge-dir sub-directory */ +const char *argv10[] = {prgname,"-m" ,DEFAULT_MEM_SIZE , +"--file-prefix=hugedir" ,"--huge-dir" , hugepath_dir}; +/* With valid --huge-dir sub-directory */ +const char *argv11[] = {prgname,"-m" ,DEFAULT_MEM_SIZE , +"--file-prefix=hugedir" ,"--huge-dir" , hugepath_dir2};/* Secondary process with invalid --huge-dir (should run as flag has no * effect on secondary processes) */ -const char *argv10[] = {prgname, prefix, mp_flag, +const char *argv12[] = {prgname, prefix, mp_flag,"--huge-dir" ,"invalid" };/* try running with base-virtaddr param */ -const char *argv11[] = {prgname,"--file-prefix=virtaddr" , +const char *argv13[] = {prgname,"--file-prefix=virtaddr" ,"--base-virtaddr=0x12345678" };/* try running with --vfio-intr INTx flag */ -const char *argv12[] = {prgname,"--file-prefix=intr" , +const char *argv14[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=legacy" };/* try running with --vfio-intr MSI flag */ -const char *argv13[] = {prgname,"--file-prefix=intr" , +const char *argv15[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=msi" };/* try running with --vfio-intr MSI-X flag */ -const char *argv14[] = {prgname,"--file-prefix=intr" , +const char *argv16[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=msix" };/* try running with --vfio-intr invalid flag */ -const char *argv15[] = {prgname,"--file-prefix=intr" , +const char *argv17[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=invalid" };/* With process type as auto-detect */ -const char *const argv16[] = {prgname,"--file-prefix=auto" , +const char *const argv18[] = {prgname,"--file-prefix=auto" ,"--proc-type=auto" };/* With process type as auto-detect with no-shconf */ -const char *const argv17[] = {prgname,"--proc-type=auto" , +const char *const argv19[] = {prgname,"--proc-type=auto" , no_shconf, nosh_prefix, no_huge};/* With process type as --create-uio-dev flag */ -const char *const argv18[] = {prgname,"--file-prefix=uiodev" , +const char *const argv20[] = {prgname,"--file-prefix=uiodev" ,"--create-uio-dev" };/* run all tests also applicable to FreeBSD first */ if (launch_proc (argv0) ==0 ) {printf ("Error - process ran ok with invalid flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv1) !=0 ) {printf ("Error - process did not run ok with --no-pci flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv2) !=0 ) {printf ("Error - process did not run ok with -v flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv6) !=0 ) {printf ("Error - process did not run ok with --no-shconf flag\n" ); -return -1 ; + goto fail; }#ifdef RTE_EXEC_ENV_FREEBSD @@ -944,73 +969,88 @@ test_misc_flags(void)if (launch_proc (argv3) !=0 ) {printf ("Error - process did not run ok with --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv4) ==0 ) {printf ("Error - process run ok with empty --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv5) ==0 ) {printf ("Error - process run ok with invalid --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv7) !=0 ) {printf ("Error - process did not run ok with --huge-dir flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv8) ==0 ) {printf ("Error - process run ok with empty --huge-dir flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv9) ==0 ) {printf ("Error - process run ok with invalid --huge-dir flag\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv10) !=0 ) { -printf ("Error - secondary process did not run ok with invalid --huge-dir flag\n" ); -return -1 ; +if (launch_proc (argv10) ==0 ) { +printf ("Error - process run ok with invalid --huge-dir sub-dir flag\n" ); + goto fail; }if (launch_proc (argv11) !=0 ) { -printf ("Error - process did not run ok with --base-virtaddr parameter\n" ); -return -1 ; +printf ("Error - process did not run ok with --huge-dir subdir flag\n" ); + goto fail; }if (launch_proc (argv12) !=0 ) { +printf ("Error - secondary process did not run ok with invalid --huge-dir flag\n" ); + goto fail; + } +if (launch_proc (argv13) !=0 ) { +printf ("Error - process did not run ok with --base-virtaddr parameter\n" ); + goto fail; + } +if (launch_proc (argv14) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr INTx parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv13) !=0 ) { +if (launch_proc (argv15) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr MSI parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv14) !=0 ) { +if (launch_proc (argv16) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr MSI-X parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv15) ==0 ) { +if (launch_proc (argv17) ==0 ) {printf ("Error - process run ok with " "--vfio-intr invalid parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv16) !=0 ) { +if (launch_proc (argv18) !=0 ) {printf ("Error - process did not run ok with " "--proc-type as auto parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv17) !=0 ) { +if (launch_proc (argv19) !=0 ) {printf ("Error - process did not run ok with " "--proc-type and --no-shconf parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv18) !=0 ) { +if (launch_proc (argv20) !=0 ) {printf ("Error - process did not run ok with " "--create-uio-dev parameter\n" ); -return -1 ; + goto fail; } +rmdir (hugepath_dir3); +rmdir (hugepath_dir2);return 0 ; + +fail: +rmdir (hugepath_dir3); +rmdir (hugepath_dir2); +return -1 ; } static intdoc/guides/linux_gsg/linux_eal_parameters.rst +2
-1@@ -81,7 +81,8 @@ Memory-related options * ``--huge-dir <path to hugetlbfs directory>`` - Use specified hugetlbfs directory instead of autodetected ones. + Use specified hugetlbfs directory instead of autodetected ones. This can be + a sub-directory within a hugetlbfs mountpoint. * ``--huge-unlink``lib/eal/linux/eal_hugepage_info.c +58
-26@@ -213,10 +213,19 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len)const size_t pagesize_opt_len =sizeof (pagesize_opt) -1 ;const char split_tok =' ' ;char *splitstr[_FIELDNAME_MAX]; +char found[PATH_MAX ] ="" ;char buf[BUFSIZ ]; -int retval =-1 ;const struct internal_config *internal_conf =eal_get_internal_configuration (); +struct stat st; + +/* +* If the specified dir doesn't exist, we can't match it. +*/ +if (internal_conf->hugepage_dir !=NULL && +stat (internal_conf->hugepage_dir, &st) !=0 ) { +return -1 ; + }FILE *fd =fopen (proc_mounts,"r" );if (fd ==NULL )@@ -226,42 +235,66 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len) default_size =get_default_hp_size ();while (fgets (buf,sizeof (buf), fd)){ +const char *pagesz_str; +if (rte_strsplit (buf,sizeof (buf), splitstr, _FIELDNAME_MAX, split_tok) != _FIELDNAME_MAX) {RTE_LOG (ERR ,EAL ,"Error parsing %s\n" , proc_mounts);break ;/* return NULL */ } -/* we have a specified --huge-dir option, only examine that dir */ -if (internal_conf->hugepage_dir !=NULL && -strcmp (splitstr[MOUNTPT ], internal_conf->hugepage_dir) !=0 ) +if (strncmp (splitstr[FSTYPE ], hugetlbfs_str, htlbfs_str_len) !=0 )continue ; -if (strncmp (splitstr[FSTYPE ], hugetlbfs_str, htlbfs_str_len) ==0 ){ -const char *pagesz_str =strstr (splitstr[OPTIONS ], pagesize_opt); + pagesz_str =strstr (splitstr[OPTIONS ], pagesize_opt); -/* if no explicit page size, the default page size is compared */ -if (pagesz_str ==NULL ){ -if (hugepage_sz == default_size){ -strlcpy (hugedir, splitstr[MOUNTPT ], len); - retval =0 ; -break ; - } - } -/* there is an explicit page size, so check it */ -else { -uint64_t pagesz =rte_str_to_size (&pagesz_str[pagesize_opt_len]); -if (pagesz == hugepage_sz) { -strlcpy (hugedir, splitstr[MOUNTPT ], len); - retval =0 ; -break ; - } - } - }/* end if strncmp hugetlbfs */ +/* if no explicit page size, the default page size is compared */ +if (pagesz_str ==NULL ) { +if (hugepage_sz != default_size) +continue ; + } +/* there is an explicit page size, so check it */ +else { +uint64_t pagesz =rte_str_to_size (&pagesz_str[pagesize_opt_len]); +if (pagesz != hugepage_sz) +continue ; + } + +/* +* If no --huge-dir option has been given, we're done. +*/ +if (internal_conf->hugepage_dir ==NULL ) { +strlcpy (found, splitstr[MOUNTPT ], len); +break ; + } + +/* +* Ignore any mount that doesn't contain the --huge-dir +* directory. +*/ +if (strncmp (internal_conf->hugepage_dir, splitstr[MOUNTPT ], +strlen (splitstr[MOUNTPT ])) !=0 ) { +continue ; + } + +/* +* We found a match, but only prefer it if it's a longer match +* (so /mnt/1 is preferred over /mnt for matching /mnt/1/2)). +*/ +if (strlen (splitstr[MOUNTPT ]) >strlen (found)) +strlcpy (found, splitstr[MOUNTPT ], len); }/* end while fgets */ fclose (fd); -return retval; + +if (found[0 ] !='\0' ) { +/* If needed, return the requested dir, not the mount point. */ +strlcpy (hugedir, internal_conf->hugepage_dir !=NULL ? + internal_conf->hugepage_dir : found, len); +return 0 ; + } + +return -1 ; }/* -- 2.30.2
Quoted 11 lines · John Levon
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction: DPDK might not be the only user of hugepages, and shouldn't assume it owns an entire mountpoint. For example, if I have/dev/hugepages/myapp, and/dev/hugepages/dpdk, I should be able to specify:--huge-dir=/dev/hugepages/dpdk/
and have DPDK only use that sub-directory.
Quoted 8 lines
Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.
Signed-off-by John Levon <[email protected]>---
v2: prefer closer matches
v3: checkpatch fixes
v4: fix docs, added tests
Anyone like to (re) review?
thanks
john
Quoted 383 lines
app/test/test_eal_flags.c | 116 ++++++++++++------ doc/guides/linux_gsg/linux_eal_parameters.rst | 3 +- lib/eal/linux/eal_hugepage_info.c | 83 +++++++++---- 3 files changed, 138 insertions(+), 64 deletions(-)app/test/test_eal_flags.c +78
-38@@ -14,8 +14,9 @@#include <errno.h> #include <unistd.h> #include <dirent.h> -#include <sys/wait.h> #include <sys/file.h> +#include <sys/stat.h> +#include <sys/wait.h> #include <limits.h> #include <fcntl.h> @@ -800,6 +801,9 @@ static inttest_misc_flags (void) {char hugepath[PATH_MAX ] = {0 }; +char hugepath_dir[PATH_MAX ] = {0 }; +char hugepath_dir2[PATH_MAX ] = {0 }; +char hugepath_dir3[PATH_MAX ] = {0 };#ifdef RTE_EXEC_ENV_FREEBSD /* BSD target doesn't support prefixes at this point */ const char * prefix ="" ;@@ -810,6 +814,7 @@ test_misc_flags(void)FILE * hugedir_handle =NULL ;char line[PATH_MAX ] = {0 };unsigned i, isempty =1 ; +if (get_current_prefix (tmp,sizeof (tmp)) ==NULL ) {printf ("Error - unable to get current prefix!\n" );return -1 ;@@ -849,6 +854,20 @@ test_misc_flags(void) }#endif +snprintf (hugepath_dir,sizeof (hugepath_dir),"%s/dpdk.missing" , hugepath); +snprintf (hugepath_dir2,sizeof (hugepath_dir2),"%s/dpdk.dir" , hugepath); + +if (mkdir (hugepath_dir2,0700 ) !=0 && errno !=EEXIST ) { +printf ("Error - failed to mkdir(%s)\n" , hugepath_dir2); +return -1 ; + } + +snprintf (hugepath_dir3,sizeof (hugepath_dir3),"%s/dpdk.dir/sub" , hugepath); + +if (mkdir (hugepath_dir3,0700 ) !=0 && errno !=EEXIST ) { +printf ("Error - failed to mkdir(%s)\n" , hugepath_dir3); + goto fail; + }/* check that some general flags don't prevent things from working. * All cases, apart from the first, app should run. @@ -881,60 +900,66 @@ test_misc_flags(void)/* With invalid --huge-dir */ const char *argv9[] = {prgname,"-m" ,DEFAULT_MEM_SIZE ,"--file-prefix=hugedir" ,"--huge-dir" ,"invalid" }; +/* With invalid --huge-dir sub-directory */ +const char *argv10[] = {prgname,"-m" ,DEFAULT_MEM_SIZE , +"--file-prefix=hugedir" ,"--huge-dir" , hugepath_dir}; +/* With valid --huge-dir sub-directory */ +const char *argv11[] = {prgname,"-m" ,DEFAULT_MEM_SIZE , +"--file-prefix=hugedir" ,"--huge-dir" , hugepath_dir2};/* Secondary process with invalid --huge-dir (should run as flag has no * effect on secondary processes) */ -const char *argv10[] = {prgname, prefix, mp_flag, +const char *argv12[] = {prgname, prefix, mp_flag,"--huge-dir" ,"invalid" };/* try running with base-virtaddr param */ -const char *argv11[] = {prgname,"--file-prefix=virtaddr" , +const char *argv13[] = {prgname,"--file-prefix=virtaddr" ,"--base-virtaddr=0x12345678" };/* try running with --vfio-intr INTx flag */ -const char *argv12[] = {prgname,"--file-prefix=intr" , +const char *argv14[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=legacy" };/* try running with --vfio-intr MSI flag */ -const char *argv13[] = {prgname,"--file-prefix=intr" , +const char *argv15[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=msi" };/* try running with --vfio-intr MSI-X flag */ -const char *argv14[] = {prgname,"--file-prefix=intr" , +const char *argv16[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=msix" };/* try running with --vfio-intr invalid flag */ -const char *argv15[] = {prgname,"--file-prefix=intr" , +const char *argv17[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=invalid" };/* With process type as auto-detect */ -const char *const argv16[] = {prgname,"--file-prefix=auto" , +const char *const argv18[] = {prgname,"--file-prefix=auto" ,"--proc-type=auto" };/* With process type as auto-detect with no-shconf */ -const char *const argv17[] = {prgname,"--proc-type=auto" , +const char *const argv19[] = {prgname,"--proc-type=auto" , no_shconf, nosh_prefix, no_huge};/* With process type as --create-uio-dev flag */ -const char *const argv18[] = {prgname,"--file-prefix=uiodev" , +const char *const argv20[] = {prgname,"--file-prefix=uiodev" ,"--create-uio-dev" };/* run all tests also applicable to FreeBSD first */ if (launch_proc (argv0) ==0 ) {printf ("Error - process ran ok with invalid flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv1) !=0 ) {printf ("Error - process did not run ok with --no-pci flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv2) !=0 ) {printf ("Error - process did not run ok with -v flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv6) !=0 ) {printf ("Error - process did not run ok with --no-shconf flag\n" ); -return -1 ; + goto fail; }#ifdef RTE_EXEC_ENV_FREEBSD @@ -944,73 +969,88 @@ test_misc_flags(void)if (launch_proc (argv3) !=0 ) {printf ("Error - process did not run ok with --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv4) ==0 ) {printf ("Error - process run ok with empty --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv5) ==0 ) {printf ("Error - process run ok with invalid --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv7) !=0 ) {printf ("Error - process did not run ok with --huge-dir flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv8) ==0 ) {printf ("Error - process run ok with empty --huge-dir flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv9) ==0 ) {printf ("Error - process run ok with invalid --huge-dir flag\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv10) !=0 ) { -printf ("Error - secondary process did not run ok with invalid --huge-dir flag\n" ); -return -1 ; +if (launch_proc (argv10) ==0 ) { +printf ("Error - process run ok with invalid --huge-dir sub-dir flag\n" ); + goto fail; }if (launch_proc (argv11) !=0 ) { -printf ("Error - process did not run ok with --base-virtaddr parameter\n" ); -return -1 ; +printf ("Error - process did not run ok with --huge-dir subdir flag\n" ); + goto fail; }if (launch_proc (argv12) !=0 ) { +printf ("Error - secondary process did not run ok with invalid --huge-dir flag\n" ); + goto fail; + } +if (launch_proc (argv13) !=0 ) { +printf ("Error - process did not run ok with --base-virtaddr parameter\n" ); + goto fail; + } +if (launch_proc (argv14) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr INTx parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv13) !=0 ) { +if (launch_proc (argv15) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr MSI parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv14) !=0 ) { +if (launch_proc (argv16) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr MSI-X parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv15) ==0 ) { +if (launch_proc (argv17) ==0 ) {printf ("Error - process run ok with " "--vfio-intr invalid parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv16) !=0 ) { +if (launch_proc (argv18) !=0 ) {printf ("Error - process did not run ok with " "--proc-type as auto parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv17) !=0 ) { +if (launch_proc (argv19) !=0 ) {printf ("Error - process did not run ok with " "--proc-type and --no-shconf parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv18) !=0 ) { +if (launch_proc (argv20) !=0 ) {printf ("Error - process did not run ok with " "--create-uio-dev parameter\n" ); -return -1 ; + goto fail; } +rmdir (hugepath_dir3); +rmdir (hugepath_dir2);return 0 ; + +fail: +rmdir (hugepath_dir3); +rmdir (hugepath_dir2); +return -1 ; } static intdoc/guides/linux_gsg/linux_eal_parameters.rst +2
-1@@ -81,7 +81,8 @@ Memory-related options * ``--huge-dir <path to hugetlbfs directory>`` - Use specified hugetlbfs directory instead of autodetected ones. + Use specified hugetlbfs directory instead of autodetected ones. This can be + a sub-directory within a hugetlbfs mountpoint. * ``--huge-unlink``lib/eal/linux/eal_hugepage_info.c +58
-25@@ -213,10 +213,19 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len)const size_t pagesize_opt_len =sizeof (pagesize_opt) -1 ;const char split_tok =' ' ;char *splitstr[_FIELDNAME_MAX]; +char found[PATH_MAX ] ="" ;char buf[BUFSIZ ]; -int retval =-1 ;const struct internal_config *internal_conf =eal_get_internal_configuration (); +struct stat st; + +/* +* If the specified dir doesn't exist, we can't match it. +*/ +if (internal_conf->hugepage_dir !=NULL && +stat (internal_conf->hugepage_dir, &st) !=0 ) { +return -1 ; + }FILE *fd =fopen (proc_mounts,"r" );if (fd ==NULL )@@ -226,42 +235,66 @@ get_hugepage_dir(uint64_t hugepage_sz, char *hugedir, int len) default_size =get_default_hp_size ();while (fgets (buf,sizeof (buf), fd)){ +const char *pagesz_str; +if (rte_strsplit (buf,sizeof (buf), splitstr, _FIELDNAME_MAX, split_tok) != _FIELDNAME_MAX) {RTE_LOG (ERR ,EAL ,"Error parsing %s\n" , proc_mounts);break ;/* return NULL */ } -/* we have a specified --huge-dir option, only examine that dir */ -if (internal_conf->hugepage_dir !=NULL && -strcmp (splitstr[MOUNTPT ], internal_conf->hugepage_dir) !=0 ) +if (strncmp (splitstr[FSTYPE ], hugetlbfs_str, htlbfs_str_len) !=0 )continue ; -if (strncmp (splitstr[FSTYPE ], hugetlbfs_str, htlbfs_str_len) ==0 ){ -const char *pagesz_str =strstr (splitstr[OPTIONS ], pagesize_opt); + pagesz_str =strstr (splitstr[OPTIONS ], pagesize_opt); -/* if no explicit page size, the default page size is compared */ -if (pagesz_str ==NULL ){ -if (hugepage_sz == default_size){ -strlcpy (hugedir, splitstr[MOUNTPT ], len); - retval =0 ; -break ; - } - } -/* there is an explicit page size, so check it */ -else { -uint64_t pagesz =rte_str_to_size (&pagesz_str[pagesize_opt_len]); -if (pagesz == hugepage_sz) { -strlcpy (hugedir, splitstr[MOUNTPT ], len); - retval =0 ; -break ; - } - } - }/* end if strncmp hugetlbfs */ +/* if no explicit page size, the default page size is compared */ +if (pagesz_str ==NULL ) { +if (hugepage_sz != default_size) +continue ; + } +/* there is an explicit page size, so check it */ +else { +uint64_t pagesz =rte_str_to_size (&pagesz_str[pagesize_opt_len]); +if (pagesz != hugepage_sz) +continue ; + } + +/* +* If no --huge-dir option has been given, we're done. +*/ +if (internal_conf->hugepage_dir ==NULL ) { +strlcpy (found, splitstr[MOUNTPT ], len); +break ; + } + +/* +* Ignore any mount that doesn't contain the --huge-dir +* directory. +*/ +if (strncmp (internal_conf->hugepage_dir, splitstr[MOUNTPT ], +strlen (splitstr[MOUNTPT ])) !=0 ) { +continue ; + } + +/* +* We found a match, but only prefer it if it's a longer match +* (so /mnt/1 is preferred over /mnt for matching /mnt/1/2)). +*/ +if (strlen (splitstr[MOUNTPT ]) >strlen (found)) +strlcpy (found, splitstr[MOUNTPT ], len); }/* end while fgets */ fclose (fd); -return retval; + +if (found[0 ] !='\0' ) { +/* If needed, return the requested dir, not the mount point. */ +strlcpy (hugedir, internal_conf->hugepage_dir !=NULL ? + internal_conf->hugepage_dir : found, len); +return 0 ; + } + +return -1 ; }/*
Quoted 16 lines · John Levon
Quoted 14 lines · John Levon
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction: DPDK might not be the only user of hugepages, and shouldn't assume it owns an entire mountpoint. For example, if I have/dev/hugepages/myapp, and/dev/hugepages/dpdk, I should be able to specify:--huge-dir=/dev/hugepages/dpdk/
and have DPDK only use that sub-directory.
Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.
Quoted 7 lines
Quoted 5 lines
Signed-off-by John Levon <[email protected]>---
v2: prefer closer matches
v3: checkpatch fixes
v4: fix docs, added testsAnyone like to (re) review?
Your patch lgtm, I tested a few scenarios I had in mind and it looks ok. Dmitry offered to rebase after your patch, can you send a new revision with following updates?
- This patch only concerns Linux, so the title prefix should be eal/linux,
- Jerin asked for a release notes update, a quick description is fine,
- Dmitry mentionned he acked your v3, so you can add it.
Thanks.
Signature · 1 line
David Marchand
PATCH 1/1 eal/linux: allow hugetlbfs sub-directories
get_hugepage_dir() was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction: DPDK might not be the only user of hugepages, and shouldn't assume it owns an entire mountpoint. For example, if I have /dev/hugepages/myapp, and /dev/hugepages/dpdk, I should be able to specify:
--huge-dir=/dev/hugepages/dpdk/
and have DPDK only use that sub-directory.
Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.
---
v2: prefer closer matches
v3: checkpatch fixes
v4: fix docs, added tests
v5: added release note
app/test/test_eal_flags.c | 116 ++++++++++++------ doc/guides/linux_gsg/linux_eal_parameters.rst | 3 +- doc/guides/rel_notes/release_21_11.rst | 6 +- lib/eal/linux/eal_hugepage_info.c | 83 +++++++++---- 4 files changed, 143 insertions(+), 65 deletions(-)
app/test/test_eal_flags.c +78 -38
@@ -14,8 +14,9 @@#include <errno.h> #include <unistd.h> #include <dirent.h> -#include <sys/wait.h> #include <sys/file.h> +#include <sys/stat.h> +#include <sys/wait.h> #include <limits.h> #include <fcntl.h>
@@ -800,6 +801,9 @@ static inttest_misc_flags (void) {char hugepath[PATH_MAX ] = {0 }; +char hugepath_dir[PATH_MAX ] = {0 }; +char hugepath_dir2[PATH_MAX ] = {0 }; +char hugepath_dir3[PATH_MAX ] = {0 };#ifdef RTE_EXEC_ENV_FREEBSD /* BSD target doesn't support prefixes at this point */ const char * prefix ="" ;
@@ -810,6 +814,7 @@ test_misc_flags(void)FILE * hugedir_handle =NULL ;char line[PATH_MAX ] = {0 };unsigned i, isempty =1 ; +if (get_current_prefix (tmp,sizeof (tmp)) ==NULL ) {printf ("Error - unable to get current prefix!\n" );return -1 ;
@@ -849,6 +854,20 @@ test_misc_flags(void) }#endif +snprintf (hugepath_dir,sizeof (hugepath_dir),"%s/dpdk.missing" , hugepath); +snprintf (hugepath_dir2,sizeof (hugepath_dir2),"%s/dpdk.dir" , hugepath); + +if (mkdir (hugepath_dir2,0700 ) !=0 && errno !=EEXIST ) { +printf ("Error - failed to mkdir(%s)\n" , hugepath_dir2); +return -1 ; + } + +snprintf (hugepath_dir3,sizeof (hugepath_dir3),"%s/dpdk.dir/sub" , hugepath); + +if (mkdir (hugepath_dir3,0700 ) !=0 && errno !=EEXIST ) { +printf ("Error - failed to mkdir(%s)\n" , hugepath_dir3); + goto fail; + }/* check that some general flags don't prevent things from working. * All cases, apart from the first, app should run.
@@ -881,60 +900,66 @@ test_misc_flags(void)/* With invalid --huge-dir */ const char *argv9[] = {prgname,"-m" ,DEFAULT_MEM_SIZE ,"--file-prefix=hugedir" ,"--huge-dir" ,"invalid" }; +/* With invalid --huge-dir sub-directory */ +const char *argv10[] = {prgname,"-m" ,DEFAULT_MEM_SIZE , +"--file-prefix=hugedir" ,"--huge-dir" , hugepath_dir}; +/* With valid --huge-dir sub-directory */ +const char *argv11[] = {prgname,"-m" ,DEFAULT_MEM_SIZE , +"--file-prefix=hugedir" ,"--huge-dir" , hugepath_dir2};/* Secondary process with invalid --huge-dir (should run as flag has no * effect on secondary processes) */ -const char *argv10[] = {prgname, prefix, mp_flag, +const char *argv12[] = {prgname, prefix, mp_flag,"--huge-dir" ,"invalid" };/* try running with base-virtaddr param */ -const char *argv11[] = {prgname,"--file-prefix=virtaddr" , +const char *argv13[] = {prgname,"--file-prefix=virtaddr" ,"--base-virtaddr=0x12345678" };/* try running with --vfio-intr INTx flag */ -const char *argv12[] = {prgname,"--file-prefix=intr" , +const char *argv14[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=legacy" };/* try running with --vfio-intr MSI flag */ -const char *argv13[] = {prgname,"--file-prefix=intr" , +const char *argv15[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=msi" };/* try running with --vfio-intr MSI-X flag */ -const char *argv14[] = {prgname,"--file-prefix=intr" , +const char *argv16[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=msix" };/* try running with --vfio-intr invalid flag */ -const char *argv15[] = {prgname,"--file-prefix=intr" , +const char *argv17[] = {prgname,"--file-prefix=intr" ,"--vfio-intr=invalid" };/* With process type as auto-detect */ -const char *const argv16[] = {prgname,"--file-prefix=auto" , +const char *const argv18[] = {prgname,"--file-prefix=auto" ,"--proc-type=auto" };/* With process type as auto-detect with no-shconf */ -const char *const argv17[] = {prgname,"--proc-type=auto" , +const char *const argv19[] = {prgname,"--proc-type=auto" , no_shconf, nosh_prefix, no_huge};/* With process type as --create-uio-dev flag */ -const char *const argv18[] = {prgname,"--file-prefix=uiodev" , +const char *const argv20[] = {prgname,"--file-prefix=uiodev" ,"--create-uio-dev" };/* run all tests also applicable to FreeBSD first */ if (launch_proc (argv0) ==0 ) {printf ("Error - process ran ok with invalid flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv1) !=0 ) {printf ("Error - process did not run ok with --no-pci flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv2) !=0 ) {printf ("Error - process did not run ok with -v flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv6) !=0 ) {printf ("Error - process did not run ok with --no-shconf flag\n" ); -return -1 ; + goto fail; }#ifdef RTE_EXEC_ENV_FREEBSD
@@ -944,73 +969,88 @@ test_misc_flags(void)if (launch_proc (argv3) !=0 ) {printf ("Error - process did not run ok with --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv4) ==0 ) {printf ("Error - process run ok with empty --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv5) ==0 ) {printf ("Error - process run ok with invalid --syslog flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv7) !=0 ) {printf ("Error - process did not run ok with --huge-dir flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv8) ==0 ) {printf ("Error - process run ok with empty --huge-dir flag\n" ); -return -1 ; + goto fail; }if (launch_proc (argv9) ==0 ) {printf ("Error - process run ok with invalid --huge-dir flag\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv10) !=0 ) { -printf ("Error - secondary process did not run ok with invalid --huge-dir flag\n" ); -return -1 ; +if (launch_proc (argv10) ==0 ) { +printf ("Error - process run ok with invalid --huge-dir sub-dir flag\n" ); + goto fail; }if (launch_proc (argv11) !=0 ) { -printf ("Error - process did not run ok with --base-virtaddr parameter\n" ); -return -1 ; +printf ("Error - process did not run ok with --huge-dir subdir flag\n" ); + goto fail; }if (launch_proc (argv12) !=0 ) { +printf ("Error - secondary process did not run ok with invalid --huge-dir flag\n" ); + goto fail; + } +if (launch_proc (argv13) !=0 ) { +printf ("Error - process did not run ok with --base-virtaddr parameter\n" ); + goto fail; + } +if (launch_proc (argv14) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr INTx parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv13) !=0 ) { +if (launch_proc (argv15) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr MSI parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv14) !=0 ) { +if (launch_proc (argv16) !=0 ) {printf ("Error - process did not run ok with " "--vfio-intr MSI-X parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv15) ==0 ) { +if (launch_proc (argv17) ==0 ) {printf ("Error - process run ok with " "--vfio-intr invalid parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv16) !=0 ) { +if (launch_proc (argv18) !=0 ) {printf ("Error - process did not run ok with " "--proc-type as auto parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv17) !=0 ) { +if (launch_proc (argv19) !=0 ) {printf ("Error - process did not run ok with " "--proc-type and --no-shconf parameter\n" ); -return -1 ; + goto fail; } -if (launch_proc (argv18) !=0 ) { +if (launch_proc (argv20) !=0 ) {printf ("Error - process did not run ok with " "--create-uio-dev parameter\n" ); -return -1 ; + goto fail; } +rmdir (hugepath_dir3); +rmdir (hugepath_dir2);return 0 ; + +fail: +rmdir (hugepath_dir3); +rmdir (hugepath_dir2); +return -1 ; } static int
doc/guides/linux_gsg/linux_eal_parameters.rst +2 -1
@@ -81,7 +81,8 @@ Memory-related options * ``--huge-dir <path to hugetlbfs directory>`` - Use specified hugetlbfs directory instead of autodetected ones. + Use specified hugetlbfs directory instead of autodetected ones. This can be + a sub-directory within a hugetlbfs mountpoint. * ``--huge-unlink``
doc/guides/rel_notes/release_21_11.rst +5 -1
@@ -62,6 +62,11 @@ New Features * Added bus-level parsing of the devargs syntax. * Kept compatibility with the legacy syntax as parsing fallback. +* **Updated EAL hugetlbfs mount handling.** + + * Modified to allow ``--huge-dir`` option to specify a sub-directory + within a hugetlbfs mountpoint. + * **Added new RSS offload types for IPv4/L4 checksum in RSS flow.** Added macros ETH_RSS_IPV4_CHKSUM and ETH_RSS_L4_CHKSUM, now IPv4 and
@@ -154,7 +159,6 @@ New Features * Added tests to verify tunnel header verification in IPsec inbound. * Added tests to verify inner checksum. - Removed Items -------------
lib/eal/linux/eal_hugepage_info.c +58 -25
110 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [dpdk-dev] [PATCH] eal/linux: allow hugetlbfs sub-directories
Quoted 10 lines · John Levon
get_hugepage_dir()was implemented in such a way that a --huge-dir option had to exactly match the mountpoint, but there's no reason for this restriction: DPDK might not be the only user of hugepages, and shouldn't assume it owns an entire mountpoint. For example, if I have/dev/hugepages/myapp, and/dev/hugepages/dpdk, I should be able to specify:--huge-dir=/dev/hugepages/dpdk/
Quoted 7 lines
and have DPDK only use that sub-directory.
Fix the implementation to allow a sub-directory within a suitable hugetlbfs mountpoint to be specified, preferring the closest match.
Signed-off-by John Levon <[email protected]>Acked-by Dmitry Kozlyuk <[email protected]>
Applied, thanks.
doc/guides/rel_notes/release_21_11.rst New Features · line 157
Quoted 9 lines
doc/guides/rel_notes/release_21_11.rst +0
-1@@ -154,7 +159,6 @@ New Features * Added tests to verify tunnel header verification in IPsec inbound. * Added tests to verify inner checksum. -
Restored this empty line: sections in doc are separated with two empty lines.
Quoted 3 lines
doc/guides/rel_notes/release_21_11.rst +0
-0Removed Items -------------
Signature · 1 line
David Marchand
Re: [dpdk-dev] [PATCH] eal/linux: allow hugetlbfs sub-directories
Quoted 11 lines · David Marchand
doc/guides/rel_notes/release_21_11.rst New Features · line 157
Quoted 9 lines
doc/guides/rel_notes/release_21_11.rst +0
-1@@ -154,7 +159,6 @@ New Features * Added tests to verify tunnel header verification in IPsec inbound. * Added tests to verify inner checksum. -Restored this empty line: sections in doc are separated with two empty lines.
Apologies, and thanks for fixing.
regards
john