Skip to contentSkip to conversation

ndctl, create-namespace: Allow 64K and 16M alignments

PATCH In review Oliver O'Halloran 2 participants 8 messages Apr 21, 2017 – Apr 24, 2017 libnvdimm: non-volatile memory device

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 ndctl, create-namespace: Allow 64K and 16M alignments

These are needed on powerpc since 64K is the default page size and 16MB is the PMD size when using the hash MMU.

---
 ndctl/builtin-xaction-namespace.c | 2 ++
 util/size.h                       | 1 +
 2 files changed, 3 insertions(+)
ndctl/builtin-xaction-namespace.c +2 -0
index 46d651e86153..d6c38dc15984 100644
@@ -494,7 +494,9 @@ static int validate_namespace_options(struct ndctl_region *region,
 
 		switch (p->align) {
 		case SZ_4K:
+		case SZ_64K:
 		case SZ_2M:
+		case SZ_16M:
 		case SZ_1G:
 			break;
 		default:
util/size.h +1 -0
index 4af14eb7d150..f1bfd1a30438 100644
@@ -3,6 +3,7 @@
 
 #define SZ_1K     0x00000400
 #define SZ_4K     0x00001000
+#define SZ_64K    0x00010000
 #define SZ_1M     0x00100000
 #define SZ_2M     0x00200000
 #define SZ_4M     0x00400000
Signature · 6 lines
2.9.3

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#1 lore raw

PATCH 2/3 ndctl, create-namespace: read default alignment from sysfs

Read the default alignment from the hpage_pmd_size in sysfs. On PPC the PMD size depends on the MMU being used. When the traditional hash MMU is used (P9 and earlier) the PMD size is 16MB while the newer radix MMU uses a 2MB PMD size. The choice of MMU is done at runtime depending on what the hardware supports so we need to detect this at runtime rather than hardcoding it.

---
 ndctl/Makefile.am                 |  3 ++-
 ndctl/builtin-xaction-namespace.c | 41 +++++++++++++++++++++++++++++----------
 2 files changed, 33 insertions(+), 11 deletions(-)
ndctl/Makefile.am +2 -1
index c563e9411cc3..6d565c643efd 100644
@@ -10,7 +10,8 @@ ndctl_SOURCES = ndctl.c \
 		 ../util/log.c \
 		builtin-list.c \
 		builtin-test.c \
-		../util/json.c
+		../util/json.c \
+		../util/sysfs.c
 
 if ENABLE_SMART
 ndctl_SOURCES += util/json-smart.c
ndctl/builtin-xaction-namespace.c +31 -10

113 lines not on this page · raw message

Signature · 6 lines
2.9.3

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#2 lore raw

Re: [PATCH 2/3] ndctl, create-namespace: read default alignment from sysfs

Quoted 58 lines · Oliver O'Halloran

Read the default alignment from the hpage_pmd_size in sysfs. On PPC the PMD size depends on the MMU being used. When the traditional hash MMU is used (P9 and earlier) the PMD size is 16MB while the newer radix MMU uses a 2MB PMD size. The choice of MMU is done at runtime depending on what the hardware supports so we need to detect this at runtime rather than hardcoding it.

---
 ndctl/Makefile.am                 |  3 ++-
 ndctl/builtin-xaction-namespace.c | 41 +++++++++++++++++++++++++++++----------
 2 files changed, 33 insertions(+), 11 deletions(-)
ndctl/Makefile.am +2 -1
index c563e9411cc3..6d565c643efd 100644
@@ -10,7 +10,8 @@ ndctl_SOURCES = ndctl.c \
                 ../util/log.c \
                builtin-list.c \
                builtin-test.c \
-               ../util/json.c
+               ../util/json.c \
+               ../util/sysfs.c
 
 if ENABLE_SMART
 ndctl_SOURCES += util/json-smart.c
ndctl/builtin-xaction-namespace.c +8 -1
index d6c38dc15984..713a95987d91 100644
@@ -22,6 +22,7 @@
 #include <sys/types.h>
 #include <util/size.h>
 #include <util/json.h>
+#include <util/sysfs.h>
 #include <json-c/json.h>
 #include <util/filter.h>
 #include <ndctl/libndctl.h>
@@ -54,6 +55,8 @@ static struct parameters {
        const char *align;
 } param;
 
+char default_align_buf[SYSFS_ATTR_SIZE];
+
 void builtin_xaction_namespace_reset(void)
 {
        /*
@@ -137,7 +140,24 @@ enum namespace_action {
        ACTION_DESTROY,
 };
 
-static int set_defaults(enum namespace_action mode)
+const char *sysfs_read_default_align(struct ndctl_ctx *ctx, const char *def,
+               const char *path)
+{
+       /*
+        * HACK: The command handlers aren't supposed to write into

ndctl/builtin-xaction-namespace.c enum namespace_action { · line 157

Quoted 10 lines
ndctl/builtin-xaction-namespace.c +10 -0
+        *       the ndctl command context, but we want the debug
+        *       output to go somewhere sensible.
+        */
+       if (__sysfs_read_attr((struct log_ctx *)ctx, path, default_align_buf))
+               return strdup(def);
+
+       if (!strlen(default_align_buf))
+               return def;
+
+       return default_align_buf;

I chatted with Dave Hansen about this and we're thinking we should go ahead and add a new attribute to the device-dax sysfs with the list of supported alignments, similar to what we have in the btt case for supported sector sizes.

The reason is that the sensitivity to page sizes is a device-dax internal requirement. Theoretically device-dax could support any alignment and handle it with a mix of page sizes. However, since device-dax wants to be strict and predictable about the tlb size backing a given device-dax mapping then it should list the possible options.

Looking at the transparent_hugepage sysfs is a bit of a layering violation. There is no strict guarantee that device-dax is tied to thp in the longterm. The thp sysfs is also awkward because it does not tell us the pud page size.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#3 lore raw

Re: [PATCH 2/3] ndctl, create-namespace: read default alignment from sysfs

Quoted 75 lines · Dan Williams
Quoted 58 lines · Oliver O'Halloran

Read the default alignment from the hpage_pmd_size in sysfs. On PPC the PMD size depends on the MMU being used. When the traditional hash MMU is used (P9 and earlier) the PMD size is 16MB while the newer radix MMU uses a 2MB PMD size. The choice of MMU is done at runtime depending on what the hardware supports so we need to detect this at runtime rather than hardcoding it.

---
 ndctl/Makefile.am                 |  3 ++-
 ndctl/builtin-xaction-namespace.c | 41 +++++++++++++++++++++++++++++----------
 2 files changed, 33 insertions(+), 11 deletions(-)
ndctl/Makefile.am +2 -1
index c563e9411cc3..6d565c643efd 100644
@@ -10,7 +10,8 @@ ndctl_SOURCES = ndctl.c \
                 ../util/log.c \
                builtin-list.c \
                builtin-test.c \
-               ../util/json.c
+               ../util/json.c \
+               ../util/sysfs.c
 
 if ENABLE_SMART
 ndctl_SOURCES += util/json-smart.c
ndctl/builtin-xaction-namespace.c +8 -1
index d6c38dc15984..713a95987d91 100644
@@ -22,6 +22,7 @@
 #include <sys/types.h>
 #include <util/size.h>
 #include <util/json.h>
+#include <util/sysfs.h>
 #include <json-c/json.h>
 #include <util/filter.h>
 #include <ndctl/libndctl.h>
@@ -54,6 +55,8 @@ static struct parameters {
        const char *align;
 } param;
 
+char default_align_buf[SYSFS_ATTR_SIZE];
+
 void builtin_xaction_namespace_reset(void)
 {
        /*
@@ -137,7 +140,24 @@ enum namespace_action {
        ACTION_DESTROY,
 };
 
-static int set_defaults(enum namespace_action mode)
+const char *sysfs_read_default_align(struct ndctl_ctx *ctx, const char *def,
+               const char *path)
+{
+       /*
+        * HACK: The command handlers aren't supposed to write into

ndctl/builtin-xaction-namespace.c enum namespace_action { · line 157

Quoted 10 lines
ndctl/builtin-xaction-namespace.c +10 -0
+        *       the ndctl command context, but we want the debug
+        *       output to go somewhere sensible.
+        */
+       if (__sysfs_read_attr((struct log_ctx *)ctx, path, default_align_buf))
+               return strdup(def);
+
+       if (!strlen(default_align_buf))
+               return def;
+
+       return default_align_buf;

I chatted with Dave Hansen about this and we're thinking we should go ahead and add a new attribute to the device-dax sysfs with the list of supported alignments, similar to what we have in the btt case for supported sector sizes.

Quoted 6 lines

The reason is that the sensitivity to page sizes is a device-dax internal requirement. Theoretically device-dax could support any alignment and handle it with a mix of page sizes. However, since device-dax wants to be strict and predictable about the tlb size backing a given device-dax mapping then it should list the possible options.

Sounds good to me. Using the thp sysfs entries was always going to be a bit of hack job, but I couldn't find anything better to use as a sane default. I'll post a patch Monday (unless you already wrote one?)

Quoted 4 lines

Looking at the transparent_hugepage sysfs is a bit of a layering violation. There is no strict guarantee that device-dax is tied to thp in the longterm. The thp sysfs is also awkward because it does not tell us the pud page size.

Are there any concrete plans on moving device-dax away from THP? THP under hash has some horrible quirks and although the hardware supports up to 16G pages they're impossible to use outside of hugetlbfs. It would be nice to make use of the larger page sizes with dax devs, but I'm not sure I want to jump down that rabbit hole...

Oliver
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#4 lore raw

Re: [PATCH 2/3] ndctl, create-namespace: read default alignment from sysfs

Quoted 92 lines · Oliver O'Halloran
Quoted 75 lines · Dan Williams
Quoted 58 lines · Oliver O'Halloran

Read the default alignment from the hpage_pmd_size in sysfs. On PPC the PMD size depends on the MMU being used. When the traditional hash MMU is used (P9 and earlier) the PMD size is 16MB while the newer radix MMU uses a 2MB PMD size. The choice of MMU is done at runtime depending on what the hardware supports so we need to detect this at runtime rather than hardcoding it.

---
 ndctl/Makefile.am                 |  3 ++-
 ndctl/builtin-xaction-namespace.c | 41 +++++++++++++++++++++++++++++----------
 2 files changed, 33 insertions(+), 11 deletions(-)
ndctl/Makefile.am +2 -1
index c563e9411cc3..6d565c643efd 100644
@@ -10,7 +10,8 @@ ndctl_SOURCES = ndctl.c \
                 ../util/log.c \
                builtin-list.c \
                builtin-test.c \
-               ../util/json.c
+               ../util/json.c \
+               ../util/sysfs.c
 
 if ENABLE_SMART
 ndctl_SOURCES += util/json-smart.c
ndctl/builtin-xaction-namespace.c +8 -1
index d6c38dc15984..713a95987d91 100644
@@ -22,6 +22,7 @@
 #include <sys/types.h>
 #include <util/size.h>
 #include <util/json.h>
+#include <util/sysfs.h>
 #include <json-c/json.h>
 #include <util/filter.h>
 #include <ndctl/libndctl.h>
@@ -54,6 +55,8 @@ static struct parameters {
        const char *align;
 } param;
 
+char default_align_buf[SYSFS_ATTR_SIZE];
+
 void builtin_xaction_namespace_reset(void)
 {
        /*
@@ -137,7 +140,24 @@ enum namespace_action {
        ACTION_DESTROY,
 };
 
-static int set_defaults(enum namespace_action mode)
+const char *sysfs_read_default_align(struct ndctl_ctx *ctx, const char *def,
+               const char *path)
+{
+       /*
+        * HACK: The command handlers aren't supposed to write into

ndctl/builtin-xaction-namespace.c enum namespace_action { · line 157

Quoted 10 lines
ndctl/builtin-xaction-namespace.c +10 -0
+        *       the ndctl command context, but we want the debug
+        *       output to go somewhere sensible.
+        */
+       if (__sysfs_read_attr((struct log_ctx *)ctx, path, default_align_buf))
+               return strdup(def);
+
+       if (!strlen(default_align_buf))
+               return def;
+
+       return default_align_buf;

I chatted with Dave Hansen about this and we're thinking we should go ahead and add a new attribute to the device-dax sysfs with the list of supported alignments, similar to what we have in the btt case for supported sector sizes.

Quoted 6 lines

The reason is that the sensitivity to page sizes is a device-dax internal requirement. Theoretically device-dax could support any alignment and handle it with a mix of page sizes. However, since device-dax wants to be strict and predictable about the tlb size backing a given device-dax mapping then it should list the possible options.

Sounds good to me. Using the thp sysfs entries was always going to be a bit of hack job, but I couldn't find anything better to use as a sane default. I'll post a patch Monday (unless you already wrote one?)

Quoted 4 lines

Looking at the transparent_hugepage sysfs is a bit of a layering violation. There is no strict guarantee that device-dax is tied to thp in the longterm. The thp sysfs is also awkward because it does not tell us the pud page size.

Quoted 5 lines

Are there any concrete plans on moving device-dax away from THP? THP under hash has some horrible quirks and although the hardware supports up to 16G pages they're impossible to use outside of hugetlbfs. It would be nice to make use of the larger page sizes with dax devs, but I'm not sure I want to jump down that rabbit hole...

Yes, we have done this for the 1GB / pud case on x86_64. We added the base THP primitives for pud support [1] even though the rest of the THP implementation only understands pmds.

[1]: commit a00cc7d9dd93 mm, x86: add support for PUD-sized transparent hugepages
_______________________________________________
Linux-nvdimm mailing list [email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#5 lore raw

PATCH 3/3 ndctl: print alignment of dax namespaces

In reply to Oliver O'Halloran

Print the alignment for dax namespaces creation. It's good to be explicit about it since about the selected region alignment dictates the type of faults we allow on the namespace.

---
 util/json.c | 5 +++++
 1 file changed, 5 insertions(+)
util/json.c +5 -0
index d6a8d4c899ed..720d22559df0 100644
@@ -299,6 +299,11 @@ struct json_object *util_namespace_to_json(struct ndctl_namespace *ndns,
 		if (!jobj)
 			goto err;
 		json_object_object_add(jndns, "uuid", jobj);
+		jobj = json_object_new_int64(ndctl_dax_get_align(dax));
+		if (!jobj)
+			goto err;
+		json_object_object_add(jndns, "align", jobj);
+
 		if (include_dax) {
 			dax_region = ndctl_dax_get_daxctl_region(dax);
 			jobj = util_daxctl_region_to_json(dax_region,
Signature · 6 lines
2.9.3

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#6 lore raw

Re: [PATCH 3/3] ndctl: print alignment of dax namespaces

Quoted 14 lines · Oliver O'Halloran

Print the alignment for dax namespaces creation. It's good to be explicit about it since about the selected region alignment dictates the type of faults we allow on the namespace.

---
 util/json.c | 5 +++++
 1 file changed, 5 insertions(+)
util/json.c +0 -0
index d6a8d4c899ed..720d22559df0 100644

util/json.c struct json_object *util_namespace_to_json(struct ndctl_namespace *ndns, · line 306

Quoted 8 lines
util/json.c +5 -0
@@ -299,6 +299,11 @@ struct json_object *util_namespace_to_json(struct ndctl_namespace *ndns,
                if (!jobj)
                        goto err;
                json_object_object_add(jndns, "uuid", jobj);
+               jobj = json_object_new_int64(ndctl_dax_get_align(dax));
+               if (!jobj)
+                       goto err;
+               json_object_object_add(jndns, "align", jobj);
+

Yes, we need this, but lets hide this attribute when the 'align' is zero. That helps with ndctl running on older kernels that do not have an align attribute available.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#7 lore raw
In reply to Oliver O'Halloran
Quoted 14 lines · Oliver O'Halloran

These are needed on powerpc since 64K is the default page size and 16MB is the PMD size when using the hash MMU.

---
 ndctl/builtin-xaction-namespace.c | 2 ++
 util/size.h                       | 1 +
 2 files changed, 3 insertions(+)
ndctl/builtin-xaction-namespace.c +0 -0
index 46d651e86153..d6c38dc15984 100644

ndctl/builtin-xaction-namespace.c static int validate_namespace_options(struct ndctl_region *region, · line 502

Quoted 9 lines
ndctl/builtin-xaction-namespace.c +2 -0
@@ -494,7 +494,9 @@ static int validate_namespace_options(struct ndctl_region *region,
 
                switch (p->align) {
                case SZ_4K:
+               case SZ_64K:
                case SZ_2M:
+               case SZ_16M:
                case SZ_1G:
                        break;
                default:

Following the comment on patch2, this is another location where it would be useful to just ask the kernel for the supported alignments. See how we do this for btt with the calls to ndctl_btt_get_supported_sector_size() later in the function.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm