powerpc/papr_scm: Workaround for failure of drc bind after kexec
v2 cover powerpc/papr_scm: Workaround for failure of drc bind after kexec
Presently an error is returned in response to hcall H_SCM_BIND_MEM when a new kernel boots on lpar via kexec. This prevents papr_scm from registering drc memory regions with nvdimm. The error reported is of the form below:
"papr_scm ibm,persistent-memory:ibm,pmemory@44100002: bind err: -68"
On investigation it was revealed that phyp returns this error as previous kernel did not completely release bindings for drc scm-memory blocks and hence phyp rejected request for re-binding these block to lpar with error H_OVERLAP. Also support for a new H_SCM_UNBIND_ALL is recently added which is better suited for releasing all the bound scm-memory block from an lpar.
So leveraging new hcall H_SCM_UNBIND_ALL, we can workaround H_OVERLAP issue during kexec by forcing an unbind of all drm scm-memory blocks and issuing H_SCM_BIND_MEM to re-bind the drc scm-memory blocks to lpar. This sequence will also be needed when a new kernel boot on lpar after previous kernel panicked and it never got an opportunity to call H_SCM_UNBIND_MEM/ALL.
Hence this patch-set implements following changes to papr_scm module:
* Update hvcall.h to include opcodes for new hcall H_SCM_UNBIND_ALL.
* Update it to use H_SCM_UNBIND_ALL instead of H_SCM_UNBIND_MEM
* In case hcall H_SCM_BIND_MEM fails with error H_OVERLAP, force H_SCM_UNBIND_ALL and retry the bind operation again.
With the patch-set applied re-bind of drc scm-memory to lpar succeeds after a kexec to new kernel as illustrated below:
# Old kernel
$ sudo ndctl list -R
[
{
"dev":"region0",
<snip>
....
}
]
# kexec to new kernel
$ sudo kexec --initrd=... vmlinux
...
...
I'm in purgatory
...
papr_scm ibm,persistent-memory:ibm,pmemory@44100002: Un-binding and retrying
...
# New kernel
$ sudo ndctl list -R
[
{
"dev":"region0",
<snip>
....
}
]---
Change-log:
v2:
* Addressed review comments from Oliver on v1 patchset.
Vaibhav Jain (3):
powerpc/pseries: Update SCM hcall op-codes in hvcall.h powerpc/papr_scm: Update drc_pmem_unbind() to use H_SCM_UNBIND_ALL powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails
arch/powerpc/include/asm/hvcall.h | 9 +++-- arch/powerpc/platforms/pseries/papr_scm.c | 44 ++++++++++++++++++----- 2 files changed, 43 insertions(+), 10 deletions(-)
Signature · 1 line
2.21.0
PATCH v2 1/3 powerpc/pseries: Update SCM hcall op-codes in hvcall.h
Update the hvcalls.h to include op-codes for new hcalls introduce to manage SCM memory. Also update existing hcall definitions to reflect current papr specification for SCM.
---
Change-log:
v2:
* None new patch in this series.
--- arch/powerpc/include/asm/hvcall.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/hvcall.h +7 -2
@@ -302,9 +302,14 @@#define H_SCM_UNBIND_MEM 0x3F0#define H_SCM_QUERY_BLOCK_MEM_BINDING 0x3F4#define H_SCM_QUERY_LOGICAL_MEM_BINDING 0x3F8 -#define H_SCM_MEM_QUERY 0x3FC +#define H_SCM_UNBIND_ALL 0x3FC#define H_SCM_BLOCK_CLEAR 0x400 -#define MAX_HCALL_OPCODE H_SCM_BLOCK_CLEAR +#define H_SCM_PERFORMANCE_STATS 0x418 +#define MAX_HCALL_OPCODE H_SCM_PERFORMANCE_STATS + +/* Scope args for H_SCM_UNBIND_ALL */ +#define H_UNBIND_SCOPE_ALL (0x1) +#define H_UNBIND_SCOPE_DRC (0x2)/* H_VIOCTL functions */ #define H_GET_VIOA_DUMP_SIZE 0x01
Signature · 1 line
2.21.0
Re: [PATCH v2 1/3] powerpc/pseries: Update SCM hcall op-codes in hvcall.h
Quoted 20 lines · Vaibhav Jain
Update the
hvcalls.hto include op-codes for new hcalls introduce to manage SCM memory. Also update existing hcall definitions to reflect current papr specification for SCM.Signed-off-by Vaibhav Jain <[email protected]>---
Change-log:v2:
* None new patch in this series.--- arch/powerpc/include/asm/hvcall.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)arch/powerpc/include/asm/hvcall.h +0
-0
arch/powerpc/include/asm/hvcall.h line 306
Quoted 6 lines
arch/powerpc/include/asm/hvcall.h +1
-1@@ -302,9 +302,14 @@#define H_SCM_UNBIND_MEM 0x3F0#define H_SCM_QUERY_BLOCK_MEM_BINDING 0x3F4#define H_SCM_QUERY_LOGICAL_MEM_BINDING 0x3F8 -#define H_SCM_MEM_QUERY 0x3FC +#define H_SCM_UNBIND_ALL 0x3FC#define H_SCM_BLOCK_CLEAR 0x400
The BLOCK_CLEAR hcall was removed from the spec and the 0x400 hcall number is now used by H_SCM_HEALTH.
Quoted 13 lines
arch/powerpc/include/asm/hvcall.h +6
-2-#define MAX_HCALL_OPCODE H_SCM_BLOCK_CLEAR +#define H_SCM_PERFORMANCE_STATS 0x418 +#define MAX_HCALL_OPCODE H_SCM_PERFORMANCE_STATS + +/* Scope args for H_SCM_UNBIND_ALL */ +#define H_UNBIND_SCOPE_ALL (0x1) +#define H_UNBIND_SCOPE_DRC (0x2)/* H_VIOCTL functions */ #define H_GET_VIOA_DUMP_SIZE 0x01 --2.21.0
PATCH v2 2/3 powerpc/papr_scm: Update drc_pmem_unbind() to use H_SCM_UNBIND_ALL
In reply to Vaibhav JainThe new hcall named H_SCM_UNBIND_ALL has been introduce that can unbind all the memory drc memory-blocks assigned to an lpar. This is more efficient than using H_SCM_UNBIND_MEM as currently we don't support partial unbind of drc memory-blocks.
Hence this patch proposes following changes to drc_pmem_unbind():
* Update drc_pmem_unbind() to replace hcall H_SCM_UNBIND_MEM to H_SCM_UNBIND_ALL.
* Update drc_pmem_unbind() to handles cases when PHYP asks the guest kernel to wait for specific amount of time before retrying the hcall via the 'LONG_BUSY' return value.
* Ensure appropriate error code is returned back from the function in case of an error.
---
Change-log:
v2:
* Added a dev_dbg when unbind operation succeeds [Oliver]
* Changed type of variable 'rc' to int64_t [Oliver]
* Removed the code that was logging a warning in case bind operation takes >1-seconds [Oliver]
* Spinned off changes to hvcall.h as a separate patch. [Oliver]
--- arch/powerpc/platforms/pseries/papr_scm.c | 29 +++++++++++++++++------ 1 file changed, 22 insertions(+), 7 deletions(-)
arch/powerpc/platforms/pseries/papr_scm.c +22 -7
@@ -11,6 +11,7 @@#include <linux/sched.h> #include <linux/libnvdimm.h> #include <linux/platform_device.h> +#include <linux/delay.h> #include <asm/plpar_wrappers.h>
@@ -77,22 +78,36 @@ static int drc_pmem_bind(struct papr_scm_priv *p)static int drc_pmem_unbind (struct papr_scm_priv *p) {unsigned long ret[PLPAR_HCALL_BUFSIZE ]; -uint64_t rc, token; +uint64_t token; +int64_t rc; - token =0 ; +dev_dbg (&p->pdev->dev,"unbind drc %x\n" , p->drc_index); -/* NB: unbind has the same retry requirements mentioned above */ +/* NB: unbind has the same retry requirements as drc_pmem_bind() */ do { - rc =plpar_hcall (H_SCM_UNBIND_MEM , ret, p->drc_index, - p->bound_addr, p->blocks, token); + +/* Unbind of all SCM resources associated with drcIndex */ + rc =plpar_hcall (H_SCM_UNBIND_ALL , ret,H_UNBIND_SCOPE_DRC , + p->drc_index, token); token = ret[0 ]; -cond_resched (); + +/* Check if we are stalled for some time */ +if (H_IS_LONG_BUSY (rc)) { +msleep (get_longbusy_msecs (rc)); + rc =H_BUSY ; + }else if (rc ==H_BUSY ) { +cond_resched (); + } + }while (rc ==H_BUSY );if (rc)dev_err (&p->pdev->dev,"unbind error: %lld\n" , rc); +else +dev_dbg (&p->pdev->dev,"unbind drc %x complete\n" , + p->drc_index); -return !!rc; +return rc ==H_SUCCESS ?0 : -ENXIO ; } static int papr_scm_meta_get(struct papr_scm_priv *p,
Signature · 1 line
2.21.0
kbuild test robot
build botRe: [PATCH v2 2/3] powerpc/papr_scm: Update drc_pmem_unbind() to use H_SCM_UNBIND_ALL
Thank you for the patch! Perhaps something to improve:
Hi Vaibhav,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on powerpc/next]
[also build test WARNING on v5.2-rc6 next-20190625]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Vaibhav-Jain/powerpc-papr_scm-Workaround-for-failure-of-drc-bind-after-kexec/20190626-034335 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next config: powerpc-allyesconfig (attached as .config) compiler: powerpc64-linux-gcc (GCC) 7.4.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.4.0 make.cross ARCH=powerpc
If you fix the issue, kindly add following tag
Note: it may well be a FALSE warning. FWIW you are at least aware of it now.
http://gcc.gnu.org/wiki/Better_Uninitialized_Warnings
All warnings (new ones prefixed by >>):
arch/powerpc/platforms/pseries/papr_scm.c: In function 'drc_pmem_unbind.isra.2':
>> arch/powerpc/platforms/pseries/papr_scm.c:90:6: warning: 'token' may be used uninitialized in this function [-Wmaybe-uninitialized]
rc = plpar_hcall(H_SCM_UNBIND_ALL, ret, H_UNBIND_SCOPE_DRC,
~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
p->drc_index, token);
~~~~~~~~~~~~~~~~~~~~vim +/token +90 arch/powerpc/platforms/pseries/papr_scm.c
77
78 static int drc_pmem_unbind(struct papr_scm_priv *p)
79 {
80 unsigned long ret[PLPAR_HCALL_BUFSIZE];
81 uint64_t token;
82 int64_t rc;
83
84 dev_dbg(&p->pdev->dev, "unbind drc %x\n", p->drc_index);
85
86 /* NB: unbind has the same retry requirements as drc_pmem_bind() */
87 do {
88
89 /* Unbind of all SCM resources associated with drcIndex */
> 90 rc = plpar_hcall(H_SCM_UNBIND_ALL, ret, H_UNBIND_SCOPE_DRC,
91 p->drc_index, token);
92 token = ret[0];
93
94 /* Check if we are stalled for some time */
95 if (H_IS_LONG_BUSY(rc)) {
96 msleep(get_longbusy_msecs(rc));
97 rc = H_BUSY;
98 } else if (rc == H_BUSY) {
99 cond_resched(); 100 }
101
102 } while (rc == H_BUSY);
103
104 if (rc)
105 dev_err(&p->pdev->dev, "unbind error: %lld\n", rc);
106 else
107 dev_dbg(&p->pdev->dev, "unbind drc %x complete\n",
108 p->drc_index);
109
110 return rc == H_SUCCESS ? 0 : -ENXIO;
111 }
112
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Re: [PATCH v2 2/3] powerpc/papr_scm: Update drc_pmem_unbind() to use H_SCM_UNBIND_ALL
In reply to Vaibhav JainQuoted 3 lines · Vaibhav Jain
The new hcall named
H_SCM_UNBIND_ALLhas been introduce that can unbind all the memory drc memory-blocks assigned to an lpar.
This is a little muddy. For normal memory each block has a DRC so you can add/remove individual blocks by configuring or deconfiguring the DRC for that block. For storage class memory the DRC refers to the SCM volume as a whole rather than an individual block so a "memory DRC" isn't really what you're taking about here.
This is probably being a little pedantic, but I find the whole DRC system confusing enough as-is so it's good to be clear about these things.
Quoted 19 lines
more efficient than using
H_SCM_UNBIND_MEMas currently we don't support partial unbind of drc memory-blocks.Hence this patch proposes following changes to
drc_pmem_unbind():* Update
drc_pmem_unbind()to replace hcallH_SCM_UNBIND_MEMtoH_SCM_UNBIND_ALL.* Update
drc_pmem_unbind()to handles cases when PHYP asks the guest kernel to wait for specific amount of time before retrying the hcall via the 'LONG_BUSY' return value.* Ensure appropriate error code is returned back from the function in case of an error.
Signed-off-by Vaibhav Jain <[email protected]>---
Change-log:
Quoted 6 lines
v2:
* Added adev_dbgwhen unbind operation succeeds [Oliver]
* Changed type of variable 'rc' toint64_t[Oliver]
* Removed the code that was logging a warning in case bind operation takes >1-seconds [Oliver]
* Spinned off changes tohvcall.has a separate patch. [Oliver]
Looks good otherwise, when you respin feel free to add:
PATCH v2 3/3 powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails
In reply to Vaibhav JainIn some cases initial bind of scm memory for an lpar can fail if previously it wasn't released using a scm-unbind hcall. This situation can arise due to panic of the previous kernel or forced lpar fadump. In such cases the H_SCM_BIND_MEM return a H_OVERLAP error.
To mitigate such cases the patch updates papr_scm_probe() to force a call to drc_pmem_unbind() in case the initial bind of scm memory fails with EBUSY error. In case scm-bind operation again fails after the forced scm-unbind then we follow the existing error path. We also update drc_pmem_bind() to handle the H_OVERLAP error returned by phyp and indicate it as a EBUSY error back to the caller.
---
Change-log:
v2:
* Moved the retry code from drc_pmem_bind() to papr_scm_probe() [Oliver]
* Changed the type of variable 'rc' in drc_pmem_bind() to int64_t. [Oliver]
--- arch/powerpc/platforms/pseries/papr_scm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
arch/powerpc/platforms/pseries/papr_scm.c +14 -1
@@ -43,8 +43,9 @@ struct papr_scm_priv {static int drc_pmem_bind (struct papr_scm_priv *p) {unsigned long ret[PLPAR_HCALL_BUFSIZE ]; -uint64_t rc, token;uint64_t saved =0 ; +uint64_t token; +int64_t rc;/* * When the hypervisor cannot map all the requested memory in a single
@@ -64,6 +65,10 @@ static int drc_pmem_bind(struct papr_scm_priv *p) }while (rc ==H_BUSY );if (rc) { +/* H_OVERLAP needs a separate error path */ +if (rc ==H_OVERLAP ) +return -EBUSY ; +dev_err (&p->pdev->dev,"bind err: %lld\n" , rc);return -ENXIO ; }
@@ -331,6 +336,14 @@ static int papr_scm_probe(struct platform_device *pdev)/* request the hypervisor to bind this region to somewhere in memory */ rc =drc_pmem_bind (p); + +/* If phyp reports drc memory still bound the force unbound and retry */ +if (rc == -EBUSY ) { +dev_warn (&pdev->dev,"Retrying bind after unbinding\n" ); +drc_pmem_unbind (p); + rc =drc_pmem_bind (p); + } +if (rc) goto err;
Signature · 1 line
2.21.0
Re: [PATCH v2 3/3] powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails
Quoted 54 lines · Vaibhav Jain
In some cases initial bind of scm memory for an lpar can fail if previously it wasn't released using a scm-unbind hcall. This situation can arise due to panic of the previous kernel or forced lpar fadump. In such cases the
H_SCM_BIND_MEMreturn aH_OVERLAPerror.To mitigate such cases the patch updates
papr_scm_probe()to force a call todrc_pmem_unbind()in case the initial bind of scm memory fails with EBUSY error. In case scm-bind operation again fails after the forced scm-unbind then we follow the existing error path. We also updatedrc_pmem_bind()to handle theH_OVERLAPerror returned by phyp and indicate it as a EBUSY error back to the caller.Suggested-by "Oliver O'Halloran" <[email protected]>Signed-off-by Vaibhav Jain <[email protected]>---
Change-log:v2:
* Moved the retry code fromdrc_pmem_bind()topapr_scm_probe()[Oliver]
* Changed the type of variable 'rc' indrc_pmem_bind()toint64_t. [Oliver]--- arch/powerpc/platforms/pseries/papr_scm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)arch/powerpc/platforms/pseries/papr_scm.c +6
-1@@ -43,8 +43,9 @@ struct papr_scm_priv {static int drc_pmem_bind (struct papr_scm_priv *p) {unsigned long ret[PLPAR_HCALL_BUFSIZE ]; -uint64_t rc, token;uint64_t saved =0 ; +uint64_t token; +int64_t rc;/* * When the hypervisor cannot map all the requested memory in a single @@ -64,6 +65,10 @@ static int drc_pmem_bind(struct papr_scm_priv *p) }while (rc ==H_BUSY );if (rc) { +/* H_OVERLAP needs a separate error path */ +if (rc ==H_OVERLAP ) +return -EBUSY ; +dev_err (&p->pdev->dev,"bind err: %lld\n" , rc);return -ENXIO ; }
arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_probe(struct platform_device *pdev) · line 340
Quoted 5 lines
arch/powerpc/platforms/pseries/papr_scm.c +2
-0@@ -331,6 +336,14 @@ static int papr_scm_probe(struct platform_device *pdev)/* request the hypervisor to bind this region to somewhere in memory */ rc =drc_pmem_bind (p); + +/* If phyp reports drc memory still bound the force unbound and retry */
"the force" should be "then force"?
arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_probe(struct platform_device *pdev) · line 342
Quoted 2 lines
arch/powerpc/platforms/pseries/papr_scm.c +2
-0+if (rc == -EBUSY ) { +dev_warn (&pdev->dev,"Retrying bind after unbinding\n" );
Looks good otherwise,