Skip to contentSkip to conversation

libnvdimm: Update the meaning for persistence_domain values

RFC PATCH In review Aneesh Kumar K.V 3 participants 10 messages Jan 8, 2020 – Jan 16, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 libnvdimm: Update the meaning for persistence_domain values

Currently, kernel shows the below values
"persistence_domain":"cpu_cache"
"persistence_domain":"memory_controller"
"persistence_domain":"unknown"

This patch updates the meaning of these values such that

"cpu_cache" indicates no extra instructions is needed to ensure the persistence of data in the pmem media on power failure.

"memory_controller" indicates platform provided instructions need to be issued as per documented sequence to make sure data flushed is guaranteed to be on pmem media in case of system power loss.

---
 arch/powerpc/platforms/pseries/papr_scm.c | 7 ++++++-
 include/linux/libnvdimm.h                 | 6 +++---
 2 files changed, 9 insertions(+), 4 deletions(-)
arch/powerpc/platforms/pseries/papr_scm.c +6 -1
index c2ef320ba1bf..26a5ef263758 100644
@@ -360,8 +360,13 @@ static int papr_scm_nvdimm_init(struct papr_scm_priv *p)
 
 	if (p->is_volatile)
 		p->region = nvdimm_volatile_region_create(p->bus, &ndr_desc);
-	else
+	else {
+		/*
+		 * We need to flush things correctly to guarantee persistance
+		 */
+		set_bit(ND_REGION_PERSIST_MEMCTRL, &ndr_desc.flags);
 		p->region = nvdimm_pmem_region_create(p->bus, &ndr_desc);
+	}
 	if (!p->region) {
 		dev_err(dev, "Error registering region %pR from %pOF\n",
 				ndr_desc.res, p->dn);
include/linux/libnvdimm.h +3 -3
index f2a33f2e3ba8..9126737377e1 100644
@@ -52,9 +52,9 @@ enum {
 	 */
 	ND_REGION_PERSIST_CACHE = 1,
 	/*
-	 * Platform provides mechanisms to automatically flush outstanding
-	 * write data from memory controler to pmem on system power loss.
-	 * (ADR)
+	 * Platform provides instructions to flush data such that on completion
+	 * of the instructions, data flushed is guaranteed to be on pmem even
+	 * in case of a system power loss.
 	 */
 	ND_REGION_PERSIST_MEMCTRL = 2,
 
Signature · 4 lines
2.24.1
_______________________________________________
Linux-nvdimm mailing list -- [email protected]
To unsubscribe send an email to [email protected]
#1 lore raw
Quoted 29 lines · Aneesh Kumar K.V

Currently, kernel shows the below values
"persistence_domain":"cpu_cache"
"persistence_domain":"memory_controller"
"persistence_domain":"unknown"

This patch updates the meaning of these values such that

"cpu_cache" indicates no extra instructions is needed to ensure the persistence of data in the pmem media on power failure.

"memory_controller" indicates platform provided instructions need to be issued as per documented sequence to make sure data flushed is guaranteed to be on pmem media in case of system power loss.

---
 arch/powerpc/platforms/pseries/papr_scm.c | 7 ++++++-
 include/linux/libnvdimm.h                 | 6 +++---
 2 files changed, 9 insertions(+), 4 deletions(-)
arch/powerpc/platforms/pseries/papr_scm.c +0 -1
index c2ef320ba1bf..26a5ef263758 100644
@@ -360,8 +360,13 @@ static int papr_scm_nvdimm_init(struct papr_scm_priv *p)
 
 	if (p->is_volatile)
 		p->region = nvdimm_volatile_region_create(p->bus, &ndr_desc);
-	else

arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_nvdimm_init(struct papr_scm_priv *p) · line 372

Quoted 10 lines
arch/powerpc/platforms/pseries/papr_scm.c +6 -0
+	else {
+		/*
+		 * We need to flush things correctly to guarantee persistance
+		 */
+		set_bit(ND_REGION_PERSIST_MEMCTRL, &ndr_desc.flags);
 		p->region = nvdimm_pmem_region_create(p->bus, &ndr_desc);
+	}
 	if (!p->region) {
 		dev_err(dev, "Error registering region %pR from %pOF\n",
 				ndr_desc.res, p->dn);

Would you also update of_pmem to indicate the persistence domain, please?

Quoted 5 lines
include/linux/libnvdimm.h +0 -0
index f2a33f2e3ba8..9126737377e1 100644

include/linux/libnvdimm.h enum { · line 57

Quoted 9 lines
include/linux/libnvdimm.h +3 -3
@@ -52,9 +52,9 @@ enum {
 	 */
 	ND_REGION_PERSIST_CACHE = 1,
 	/*
-	 * Platform provides mechanisms to automatically flush outstanding
-	 * write data from memory controler to pmem on system power loss.
-	 * (ADR)
+	 * Platform provides instructions to flush data such that on completion
+	 * of the instructions, data flushed is guaranteed to be on pmem even
+	 * in case of a system power loss.

I find the prior description easier to understand.

-Jeff
_______________________________________________
Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]

#2 lore raw
Quoted 28 lines · Jeff Moyer
Quoted 26 lines · Aneesh Kumar K.V

Currently, kernel shows the below values
"persistence_domain":"cpu_cache"
"persistence_domain":"memory_controller"
"persistence_domain":"unknown"

This patch updates the meaning of these values such that

"cpu_cache" indicates no extra instructions is needed to ensure the persistence of data in the pmem media on power failure.

"memory_controller" indicates platform provided instructions need to be issued as per documented sequence to make sure data flushed is guaranteed to be on pmem media in case of system power loss.

---
  arch/powerpc/platforms/pseries/papr_scm.c | 7 ++++++-
  include/linux/libnvdimm.h                 | 6 +++---
  2 files changed, 9 insertions(+), 4 deletions(-)
arch/powerpc/platforms/pseries/papr_scm.c +0 -0
index c2ef320ba1bf..26a5ef263758 100644
@@ -360,8 +360,13 @@ static int papr_scm_nvdimm_init(struct papr_scm_priv *p)
 
Quoted 17 lines

arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_nvdimm_init(struct papr_scm_priv *p) · line 372

Quoted 13 lines
arch/powerpc/platforms/pseries/papr_scm.c +6 -1
  	if (p->is_volatile)
  		p->region = nvdimm_volatile_region_create(p->bus, &ndr_desc);
-	else
+	else {
+		/*
+		 * We need to flush things correctly to guarantee persistance
+		 */
+		set_bit(ND_REGION_PERSIST_MEMCTRL, &ndr_desc.flags);
  		p->region = nvdimm_pmem_region_create(p->bus, &ndr_desc);
+	}
  	if (!p->region) {
  		dev_err(dev, "Error registering region %pR from %pOF\n",
  				ndr_desc.res, p->dn);

Would you also update of_pmem to indicate the persistence domain, please?

sure.

Quoted 8 lines
Quoted 8 lines
include/linux/libnvdimm.h +0 -0
index f2a33f2e3ba8..9126737377e1 100644
@@ -52,9 +52,9 @@ enum {
  	 */
  	ND_REGION_PERSIST_CACHE = 1,
  	/*
Quoted 8 lines

include/linux/libnvdimm.h enum { · line 57

Quoted 6 lines
include/linux/libnvdimm.h +3 -3
-	 * Platform provides mechanisms to automatically flush outstanding
-	 * write data from memory controler to pmem on system power loss.
-	 * (ADR)
+	 * Platform provides instructions to flush data such that on completion
+	 * of the instructions, data flushed is guaranteed to be on pmem even
+	 * in case of a system power loss.

I find the prior description easier to understand.

I was trying to avoid the term 'automatically, 'memory controler' and ADR. Can I update the above as

/*
* Platform provides mechanisms to flush outstanding write data
* to pmem on system power loss.
*/

-aneesh
_______________________________________________
Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]

#3 lore raw
Quoted 70 lines · Aneesh Kumar K.V
Quoted 36 lines · Jeff Moyer
Quoted 34 lines · Aneesh Kumar K.V

Currently, kernel shows the below values
    "persistence_domain":"cpu_cache"
    "persistence_domain":"memory_controller"
    "persistence_domain":"unknown"

This patch updates the meaning of these values such that

"cpu_cache" indicates no extra instructions is needed to ensure the persistence of data in the pmem media on power failure.

"memory_controller" indicates platform provided instructions need to be issued as per documented sequence to make sure data flushed is guaranteed to be on pmem media in case of system power loss.

---
  arch/powerpc/platforms/pseries/papr_scm.c | 7 ++++++- include/linux/libnvdimm.h                 | 6 +++---

  2 files changed, 9 insertions(+), 4 deletions(-)
arch/powerpc/platforms/pseries/papr_scm.c +5 -1
index c2ef320ba1bf..26a5ef263758 100644
@@ -360,8 +360,13 @@ static int papr_scm_nvdimm_init(struct 
 papr_scm_priv *p)
       if (p->is_volatile)
           p->region = nvdimm_volatile_region_create(p->bus, &ndr_desc);
-    else
+    else {
+        /*
+         * We need to flush things correctly to guarantee persistance
+         */
+        set_bit(ND_REGION_PERSIST_MEMCTRL, &ndr_desc.flags);
Quoted 9 lines

arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_nvdimm_init(struct papr_scm_priv *p) · line 372

Quoted 5 lines
arch/powerpc/platforms/pseries/papr_scm.c +1 -0
           p->region = nvdimm_pmem_region_create(p->bus, &ndr_desc);
+    }
       if (!p->region) {
           dev_err(dev, "Error registering region %pR from %pOF\n",
                   ndr_desc.res, p->dn);

Would you also update of_pmem to indicate the persistence domain, please?

sure.

Quoted 10 lines
Quoted 10 lines
include/linux/libnvdimm.h +0 -2
index f2a33f2e3ba8..9126737377e1 100644
@@ -52,9 +52,9 @@ enum {
        */
       ND_REGION_PERSIST_CACHE = 1,
       /*
-     * Platform provides mechanisms to automatically flush outstanding
-     * write data from memory controler to pmem on system power loss.
Quoted 8 lines
Quoted 6 lines
include/linux/libnvdimm.h +1 -1
-     * (ADR)
+     * Platform provides instructions to flush data such that on 

completion
+     * of the instructions, data flushed is guaranteed to be on pmem even
+     * in case of a system power loss.

I find the prior description easier to understand.

Quoted 7 lines

I was trying to avoid the term 'automatically, 'memory controler' and ADR. Can I update the above as

/*
 * Platform provides mechanisms to flush outstanding write data
 * to pmem on system power loss.
 */

Wanted to add more details. So with the above interpretation, if the persistence_domain is found to be 'cpu_cache', application can expect a store instruction to guarantee persistence. If it is 'none' there is no persistence ( I am not sure how that is the difference from 'volatile' pmem region). If it is 'memory_controller' ( I am not sure whether that is the right term), application needs to follow the recommended mechanism to flush write data to pmem.

-aneesh
_______________________________________________
Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]

#4 lore raw
Quoted 6 lines · Aneesh Kumar K.V
Quoted 5 lines

/*
 * Platform provides mechanisms to flush outstanding write data
 * to pmem on system power loss.
 */

Quoted 7 lines

Wanted to add more details. So with the above interpretation, if the persistence_domain is found to be 'cpu_cache', application can expect a store instruction to guarantee persistence. If it is 'none' there is no persistence ( I am not sure how that is the difference from 'volatile' pmem region). If it is 'memory_controller' ( I am not sure whether that is the right term), application needs to follow the recommended mechanism to flush write data to pmem.

There is no enum for "NONE". If there is no persistence domain specified, then it is undefined/unknown, which results in the message:

nd_pmem namespace0.0: unable to guarantee persistence of writes

Other than that, yes, that's right.

-Jeff
_______________________________________________
Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]

#5 lore raw
In reply to Aneesh Kumar K.V
Quoted 85 lines · Aneesh Kumar K.V
Quoted 74 lines · Aneesh Kumar K.V
Quoted 40 lines · Jeff Moyer
Quoted 38 lines · Aneesh Kumar K.V

Currently, kernel shows the below values
"persistence_domain":"cpu_cache"
"persistence_domain":"memory_controller"
"persistence_domain":"unknown"

This patch updates the meaning of these values such that

"cpu_cache" indicates no extra instructions is needed to ensure the persistence
of data in the pmem media on power failure.

"memory_controller" indicates platform provided instructions need to be issued
as per documented sequence to make sure data flushed is guaranteed to be on pmem
media in case of system power loss.

---
  arch/powerpc/platforms/pseries/papr_scm.c | 7 ++++++-
  include/linux/libnvdimm.h                 | 6 +++---
  2 files changed, 9 insertions(+), 4 deletions(-)
arch/powerpc/platforms/pseries/papr_scm.c +5 -1
index c2ef320ba1bf..26a5ef263758 100644
@@ -360,8 +360,13 @@ static int papr_scm_nvdimm_init(struct
 papr_scm_priv *p)
      if (p->is_volatile)
          p->region = nvdimm_volatile_region_create(p->bus, &ndr_desc);
-    else
+    else {
+        /*
+         * We need to flush things correctly to guarantee persistance
+         */
+        set_bit(ND_REGION_PERSIST_MEMCTRL, &ndr_desc.flags);
Quoted 9 lines

arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_nvdimm_init(struct papr_scm_priv *p) · line 372

Quoted 5 lines
arch/powerpc/platforms/pseries/papr_scm.c +1 -0
          p->region = nvdimm_pmem_region_create(p->bus, &ndr_desc);
+    }
      if (!p->region) {
          dev_err(dev, "Error registering region %pR from %pOF\n",
                  ndr_desc.res, p->dn);

Would you also update of_pmem to indicate the persistence domain, please?

sure.

Quoted 10 lines
Quoted 10 lines
include/linux/libnvdimm.h +0 -2
index f2a33f2e3ba8..9126737377e1 100644
@@ -52,9 +52,9 @@ enum {
       */
      ND_REGION_PERSIST_CACHE = 1,
      /*
-     * Platform provides mechanisms to automatically flush outstanding
-     * write data from memory controler to pmem on system power loss.
Quoted 8 lines
Quoted 6 lines
include/linux/libnvdimm.h +1 -1
-     * (ADR)
+     * Platform provides instructions to flush data such that on

completion
+ * of the instructions, data flushed is guaranteed to be on pmem
even
+ * in case of a system power loss.

I find the prior description easier to understand.

Quoted 8 lines

I was trying to avoid the term 'automatically, 'memory controler' and ADR. Can I update the above as

/*
* Platform provides mechanisms to flush outstanding write data
* to pmem on system power loss.
*/

Quoted 3 lines

Wanted to add more details. So with the above interpretation, if the persistence_domain is found to be 'cpu_cache', application can expect a store instruction to guarantee persistence.

The expectation is globally visible stores are persisted

Quoted 3 lines

If it is 'none' there is no
persistence ( I am not sure how that is the difference from 'volatile' pmem region).

"None" means the "platform does not enumerate a persistence domain". That's not necessarily "volatile" because the user may know that they have battery backup, or some other private/out-of-band capability not exported by the platform. In which case they would manually need to manipulate the pmemX/write_cache property manually.

Quoted 3 lines

If it is 'memory_controller' ( I am not sure whether that is the right term), application needs to follow the recommended mechanism to flush write data to pmem.

If it is memory_controller the expectation is that flushing data out of the cpu caches and making those writebacks to memory globally visible is sufficient for the platform to persist flushed data on a power loss.
_______________________________________________
Linux-nvdimm mailing list -- [email protected]
To unsubscribe send an email to [email protected]

#6 lore raw
In reply to Aneesh Kumar K.V
Quoted 5 lines · Aneesh Kumar K.V
Quoted 3 lines

Would you also update of_pmem to indicate the persistence domain, please?

sure.

Thanks!

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
include/linux/libnvdimm.h +0 -0
index f2a33f2e3ba8..9126737377e1 100644
@@ -52,9 +52,9 @@ enum {
  	 */
  	ND_REGION_PERSIST_CACHE = 1,
  	/*
Quoted 8 lines

include/linux/libnvdimm.h enum { · line 57

Quoted 6 lines
include/linux/libnvdimm.h +3 -3
-	 * Platform provides mechanisms to automatically flush outstanding
-	 * write data from memory controler to pmem on system power loss.
-	 * (ADR)
+	 * Platform provides instructions to flush data such that on completion
+	 * of the instructions, data flushed is guaranteed to be on pmem even
+	 * in case of a system power loss.

I find the prior description easier to understand.

Quoted 2 lines

I was trying to avoid the term 'automatically, 'memory controler' and ADR. Can I update the above as

I can understand avoiding the very x86-specific "ADR," but is memory controller not accurate for your platform?

Quoted 4 lines

/*
* Platform provides mechanisms to flush outstanding write data
* to pmem on system power loss.
*/

That's way too broad. :) The comments are describing the persistence domain. i.e. if you get data to $HERE, it is guaranteed to make it out to stable media.

-Jeff
_______________________________________________
Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]

#7 lore raw
Quoted 26 lines · Jeff Moyer
Quoted 17 lines · Aneesh Kumar K.V
Quoted 8 lines
Quoted 8 lines
include/linux/libnvdimm.h +0 -0
index f2a33f2e3ba8..9126737377e1 100644
@@ -52,9 +52,9 @@ enum {
   	 */
   	ND_REGION_PERSIST_CACHE = 1,
   	/*
Quoted 8 lines

include/linux/libnvdimm.h enum { · line 57

Quoted 6 lines
include/linux/libnvdimm.h +3 -3
-	 * Platform provides mechanisms to automatically flush outstanding
-	 * write data from memory controler to pmem on system power loss.
-	 * (ADR)
+	 * Platform provides instructions to flush data such that on completion
+	 * of the instructions, data flushed is guaranteed to be on pmem even
+	 * in case of a system power loss.

I find the prior description easier to understand.

Quoted 2 lines

I was trying to avoid the term 'automatically, 'memory controler' and ADR. Can I update the above as

I can understand avoiding the very x86-specific "ADR," but is memory controller not accurate for your platform?

Quoted 8 lines
Quoted 4 lines

/*
* Platform provides mechanisms to flush outstanding write data
* to pmem on system power loss.
*/

That's way too broad. :) The comments are describing the persistence domain. i.e. if you get data to $HERE, it is guaranteed to make it out to stable media.

With technologies like OpenCAPI, we possibly may not want to call them "memory controller"? In a way platform mechanism will flush them such that on power failure, it is guaranteed to be on the pmem media. But should we call these boundary "memory_controller"? May be we can consider "memory controller" an overloaded term there. Considering we are calling this as memory_controller for application to parse via sysfs, may be the documentation can also carry the same term.

-aneesh
_______________________________________________
Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]

#8 lore raw
Quoted 37 lines · Aneesh Kumar K.V
Quoted 26 lines · Jeff Moyer
Quoted 17 lines · Aneesh Kumar K.V
Quoted 8 lines
Quoted 8 lines
include/linux/libnvdimm.h +0 -0
index f2a33f2e3ba8..9126737377e1 100644
@@ -52,9 +52,9 @@ enum {
            */
           ND_REGION_PERSIST_CACHE = 1,
           /*
Quoted 8 lines

include/linux/libnvdimm.h enum { · line 57

Quoted 6 lines
include/linux/libnvdimm.h +3 -3
-   * Platform provides mechanisms to automatically flush outstanding
-   * write data from memory controler to pmem on system power loss.
-   * (ADR)
+   * Platform provides instructions to flush data such that on completion
+   * of the instructions, data flushed is guaranteed to be on pmem even
+   * in case of a system power loss.

I find the prior description easier to understand.

Quoted 2 lines

I was trying to avoid the term 'automatically, 'memory controler' and ADR. Can I update the above as

I can understand avoiding the very x86-specific "ADR," but is memory controller not accurate for your platform?

Quoted 8 lines
Quoted 4 lines

/*
* Platform provides mechanisms to flush outstanding write data
* to pmem on system power loss.
*/

That's way too broad. :) The comments are describing the persistence domain. i.e. if you get data to $HERE, it is guaranteed to make it out to stable media.

Quoted 7 lines

With technologies like OpenCAPI, we possibly may not want to call them "memory controller"? In a way platform mechanism will flush them such that on power failure, it is guaranteed to be on the pmem media. But should we call these boundary "memory_controller"? May be we can consider "memory controller" an overloaded term there. Considering we are calling this as memory_controller for application to parse via sysfs, may be the documentation can also carry the same term.

I don't see how OpenCAPI or any other transport has any bearing on the "memory_controller" term. It's still a controller of persistent memory and it needs to have the write data received at its buffers / queue to ensure that the data gets persisted, or, as in the cpu_cache case, some other agent takes responsibility for shuttling pending writes that have hit the cache out over the transport to be persisted.
_______________________________________________
Linux-nvdimm mailing list -- [email protected]
To unsubscribe send an email to [email protected]

#9 lore raw
Quoted 47 lines · Dan Williams
Quoted 37 lines · Aneesh Kumar K.V
Quoted 26 lines · Jeff Moyer
Quoted 17 lines · Aneesh Kumar K.V
Quoted 8 lines
Quoted 8 lines
include/linux/libnvdimm.h +0 -0
index f2a33f2e3ba8..9126737377e1 100644
@@ -52,9 +52,9 @@ enum {
             */
            ND_REGION_PERSIST_CACHE = 1,
            /*
Quoted 8 lines

include/linux/libnvdimm.h enum { · line 57

Quoted 6 lines
include/linux/libnvdimm.h +3 -3
-   * Platform provides mechanisms to automatically flush outstanding
-   * write data from memory controler to pmem on system power loss.
-   * (ADR)
+   * Platform provides instructions to flush data such that on completion
+   * of the instructions, data flushed is guaranteed to be on pmem even
+   * in case of a system power loss.

I find the prior description easier to understand.

Quoted 2 lines

I was trying to avoid the term 'automatically, 'memory controler' and ADR. Can I update the above as

I can understand avoiding the very x86-specific "ADR," but is memory controller not accurate for your platform?

Quoted 8 lines
Quoted 4 lines

/*
* Platform provides mechanisms to flush outstanding write data
* to pmem on system power loss.
*/

That's way too broad. :) The comments are describing the persistence domain. i.e. if you get data to $HERE, it is guaranteed to make it out to stable media.

Quoted 7 lines

With technologies like OpenCAPI, we possibly may not want to call them "memory controller"? In a way platform mechanism will flush them such that on power failure, it is guaranteed to be on the pmem media. But should we call these boundary "memory_controller"? May be we can consider "memory controller" an overloaded term there. Considering we are calling this as memory_controller for application to parse via sysfs, may be the documentation can also carry the same term.

Quoted 7 lines

I don't see how OpenCAPI or any other transport has any bearing on the "memory_controller" term. It's still a controller of persistent memory and it needs to have the write data received at its buffers / queue to ensure that the data gets persisted, or, as in the cpu_cache case, some other agent takes responsibility for shuttling pending writes that have hit the cache out over the transport to be persisted.

Agreed. I want to make sure we document that details correctly. It is a controller of persistent memory and in some cases, there is no reserve power available to keep things in self-refresh mode and to flush things automatically. The platform provided mechanism will ensure the write data is in the pmem media.

Should the later have a persistence_domain value "pmem media" ? Even then I am not sure how applications are supposed to use this information.

IMHO what is important for application is to differentiate between whether a platform specific flush mechanism is needed or not. Hence was trying to keep this as two value property. IS there any other detail application is supposed to infer from this property?

-aneesh
_______________________________________________
Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]