powerpc/papr_scm: Workaround for failure of drc bind after kexec
Subsystems (2)
v5 cover powerpc/papr_scm: Workaround for failure of drc bind after kexec
Presently an error is returned in response to hcall H_SCM_BIND_MEM when a new kernel boots on lpar via kexec. This prevents papr_scm from registering drc memory regions with nvdimm. The error reported is of the form below:
"papr_scm ibm,persistent-memory:ibm,pmemory@44100002: bind err: -68"
On investigation it was revealed that phyp returns this error as previous kernel did not completely release bindings for drc scm-memory blocks and hence phyp rejected request for re-binding these block to lpar with error H_OVERLAP. Also support for a new H_SCM_UNBIND_ALL is recently added which is better suited for releasing all the bound scm-memory block from an lpar.
So leveraging new hcall H_SCM_UNBIND_ALL, we can workaround H_OVERLAP issue during kexec by forcing an unbind of all drm scm-memory blocks and issuing H_SCM_BIND_MEM to re-bind the drc scm-memory blocks to lpar. This sequence will also be needed when a new kernel boot on lpar after previous kernel panicked and it never got an opportunity to call H_SCM_UNBIND_MEM/ALL.
Hence this patch-set implements following changes to papr_scm module:
* Update hvcall.h to include opcodes for new hcall H_SCM_UNBIND_ALL.
* Update it to use H_SCM_UNBIND_ALL instead of H_SCM_UNBIND_MEM
* In case hcall H_SCM_BIND_MEM fails with error H_OVERLAP, force H_SCM_UNBIND_ALL and retry the bind operation again.
With the patch-set applied re-bind of drc scm-memory to lpar succeeds after a kexec to new kernel as illustrated below:
# Old kernel
$ sudo ndctl list -R
[
{
"dev":"region0",
<snip>
....
}
]
# kexec to new kernel
$ sudo kexec --initrd=... vmlinux
...
...
I'm in purgatory
...
papr_scm ibm,persistent-memory:ibm,pmemory@44100002: Un-binding and retrying
...
# New kernel
$ sudo ndctl list -R
[
{
"dev":"region0",
<snip>
....
}
]---
Change-log:
v5:
* Added a new doc-patch describing the HCALL interface between a guest kernel and PAPR compliant hyper-visor like PowerVM/KVM.
v4:
* Updated the patch description of first patch in the series as suggested by Mpe.
v3:
* Fixed a build warning reported by kbuild test robot.
* Updated the hcall opcode from latest papr-scm specification.
* Fixed a minor code comment & patch description as pointed out by Oliver.
v2:
* Addressed review comments from Oliver on v1 patchset.
Vaibhav Jain (4):
powerpc: Document some HCalls for Storage Class Memory powerpc/pseries: Update SCM hcall op-codes in hvcall.h powerpc/papr_scm: Update drc_pmem_unbind() to use H_SCM_UNBIND_ALL powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails
Documentation/powerpc/hcalls.txt | 140 ++++++++++++++++++++++ arch/powerpc/include/asm/hvcall.h | 11 +- arch/powerpc/platforms/pseries/papr_scm.c | 44 +++++-- 3 files changed, 184 insertions(+), 11 deletions(-) create mode 100644 Documentation/powerpc/hcalls.txt
Signature · 1 line
2.21.0
PATCH v5 1/4 powerpc: Document some HCalls for Storage Class Memory
This doc patch provides an initial description of the HCall op-codes that are used by Linux kernel running as a guest operating system (LPAR) on top of PowerVM or any other sPAPR compliant hyper-visor (e.g qemu).
Apart from documenting the HCalls the doc-patch also provides a rudimentary overview of how Hcalls are implemented inside the Linux kernel and how information flows between kernel and PowerVM/KVM.
---
Change-log:
v5
* First patch in this patchset.
--- Documentation/powerpc/hcalls.txt | 140 +++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 Documentation/powerpc/hcalls.txt
Documentation/powerpc/hcalls.txt new file +140 -0
@@ -0,0 +1,140 @@ +Hyper-visor Call Op-codes (HCALLS) +==================================== + +Overview +========= + +Virtualization on PPC64 arch is based on the PAPR specification[1] which +describes run-time environment for a guest operating system and how it should +interact with the hyper-visor for privileged operations. Currently there are two +PAPR compliant hypervisors (PHYP): + +IBM PowerVM: IBM's proprietary hyper-visor that supports AIX, IBM-i and Linux as + supported guests (termed as Logical Partitions or LPARS). + +Qemu/KVM: Supports PPC64 linux guests running on a PPC64 linux host. + +On PPC64 arch a virtualized guest kernel runs in a non-privileged mode (HV=0). +Hence to perform a privileged operations the guest issues a Hyper-visor +Call (HCALL) with necessary input operands. PHYP after performing the privilege +operation returns a status code and output operands back to the guest. + +HCALL ABI +========= +The ABI specification for a HCall between guest os kernel and PHYP is +described in [1]. The Opcode for Hcall is set in R3 and subsequent in-arguments +for the Hcall are provided in registers R4-R12. On return from 'HVCS' +instruction the status code of HCall is available in R3 an the output parameters +are returned in registers R4-R12. + +Powerpc arch code provides convenient wrappers named plpar_hcall_xxx defined in +header 'hvcall.h' to issue HCalls from the linux kernel running as guest. + + +DRC & DRC Indexes +================= + + PAPR Guest + DR1 Hypervisor OS + +--+ +----------+ +---------+ + | |<------>| | | User | + +--+ DRC1 | | DRC | Space | + | | Index +---------+ + DR2 | | | | + +--+ | |<------->| Kernel | + | |<----- >| | HCall | | + +--+ DRC2 +----------+ +---------+ + +PHYP terms shared hardware resources like PCI devices, NVDimms etc available for +use by LPARs as Dynamic Resource (DR). When a DR is allocated to an LPAR, PHYP +creates a data-structure called Dynamic Resource Connector (DRC) to manage LPAR +access. An LPAR refers to a DRC via an opaque 32-bit number called DRC-Index. +The DRC-index value is provided to the LPAR via device-tree where its present +as an attribute in the device tree node associated with the DR. + +HCALL Op-codes +============== + +Below is a partial of of HCALLs that are supported by PHYP. For the +corresponding opcode values please look into the header +'arch/powerpc/include/asm/hvcall.h' : + +* H_SCM_READ_METADATA: + Input: drcIndex, offset, buffer-address, numBytesToRead + Out: None + Description: + Given a DRC Index of an NVDimm, read N-bytes from the the meta data area + associated with it, at a specified offset and copy it to provided buffer. + The metadata area stores configuration information such as label information, + bad-blocks etc. The metadata area is located out-of-band of NVDimm storage + area hence a separate access semantics is provided. + +* H_SCM_WRITE_METADATA: + Input: drcIndex, offset, data, numBytesToWrite + Out: None + Description: + Given a DRC Index of an NVDimm, write N-bytes from provided buffer at the + given offset to the the meta data area associated with the NVDimm. + + +* H_SCM_BIND_MEM: + Input: drcIndex, startingScmBlockIndex, numScmBlocksToBind, targetAddress + Out: guestMappedAddress, numScmBlockBound + Description: + Given a DRC-Index of an NVDimm, maps the SCM (Storage Class Memory) blocks to + continuous logical addresses in guest physical address space. The HCALL + arguments can be used to map partial range of SCM blocks instead of entire + NVDimm range to the LPAR. + +* H_SCM_UNBIND_MEM: + Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind + Out: numScmBlocksUnbound + Description: + Given a DRC-Index of an NVDimm, unmap one or more the SCM blocks from guest + physical address space. The HCALL can fail if the Guest has an active PTE + entry to the SCM block being unbinded. + +* H_SCM_QUERY_BLOCK_MEM_BINDING: + Input: drcIndex, scmBlockIndex + Out: Guest-Physical-Address + Description: + Given a DRC-Index and an SCM Block index return the guest physical address to + which the SCM block is mapped to. + +* H_SCM_QUERY_LOGICAL_MEM_BINDING: + Input: Guest-Physical-Address + Out: drcIndex, scmBlockIndex + Description: + Given a guest physical address return which DRC Index and SCM block is mapped + to that address. + +* H_SCM_UNBIND_ALL: + Input: scmTargetScope, drcIndex + Out: None + Description: + Depending on the Target scope unmap all scm blocks belonging to all NVDimms + or all scm blocks belonging to a single NVDimm identified by its drcIndex + from the LPAR memory. + +* H_SCM_HEALTH: + Input: drcIndex + Output: health-bitmap, health-bit-valid-bitmap + Description: + Given a DRC Index return the info on predictive failure and over all health of + the NVDimm. The asserted bits in the health-bitmap indicate a single predictive + failure and health-bit-valid-bitmap indicate which bits in health-bitmap are + valid. + + +* H_SCM_PERFORMANCE_STATS: + Input: drcIndex, resultBuffer Addr + Out: None + Description: + Given a DRC Index collect the performance statistics for NVDimm and copy them + to the resultBuffer. + + +References +========== +[1]: "Linux on Power Architecture Platform Reference" + https://members.openpowerfoundation.org/document/dl/469
Signature · 1 line
2.21.0
Re: [DOC][PATCH v5 1/4] powerpc: Document some HCalls for Storage Class Memory
Le 23/07/2019 à 18:13, Vaibhav Jain a écrit :
Quoted 8 lines
This doc patch provides an initial description of the HCall op-codes that are used by Linux kernel running as a guest operating system (LPAR) on top of PowerVM or any other sPAPR compliant hyper-visor (e.g qemu).
Apart from documenting the HCalls the doc-patch also provides a rudimentary overview of how Hcalls are implemented inside the Linux kernel and how information flows between kernel and PowerVM/KVM.
Hi Vaibhav,
That's a good idea to introduce such a documentation.
Quoted 35 lines
Signed-off-by Vaibhav Jain <[email protected]>---
Change-log:v5
* First patch in this patchset.--- Documentation/powerpc/hcalls.txt | 140 +++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+)create mode 100644
Documentation/powerpc/hcalls.txtDocumentation/powerpc/hcalls.txt new file +18
-0@@ -0,0 +1,140 @@ +Hyper-visor Call Op-codes (HCALLS) +==================================== + +Overview +========= + +Virtualization on PPC64 arch is based on the PAPR specification[1] which +describes run-time environment for a guest operating system and how it should +interact with the hyper-visor for privileged operations. Currently there are two +PAPR compliant hypervisors (PHYP): + +IBM PowerVM: IBM's proprietary hyper-visor that supports AIX, IBM-i and Linux as + supported guests (termed as Logical Partitions or LPARS). + +Qemu/KVM: Supports PPC64 linux guests running on a PPC64 linux host. + +On PPC64 arch a virtualized guest kernel runs in a non-privileged mode (HV=0). +Hence to perform a privileged operations the guest issues a Hyper-visor
Documentation/powerpc/hcalls.txt line 28
Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10
-0+Call (HCALL) with necessary input operands. PHYP after performing the privilege +operation returns a status code and output operands back to the guest. + +HCALL ABI +========= +The ABI specification for a HCall between guest os kernel and PHYP is +described in [1]. The Opcode for Hcall is set in R3 and subsequent in-arguments +for the Hcall are provided in registers R4-R12. On return from 'HVCS' +instruction the status code of HCall is available in R3 an the output parameters +are returned in registers R4-R12.
Would it be good to mention that values passed through the memory must be stored in Big Endian format ?
Quoted 14 lines
Documentation/powerpc/hcalls.txt new file +14
-0+Powerpc arch code provides convenient wrappers named plpar_hcall_xxx defined in +header 'hvcall.h' to issue HCalls from the linux kernel running as guest. + + +DRC & DRC Indexes +================= + + PAPR Guest + DR1 Hypervisor OS + +--+ +----------+ +---------+ + | |<------>| | | User | + +--+ DRC1 | | DRC | Space | + | | Index +---------+ + DR2 | | | |
Documentation/powerpc/hcalls.txt line 53
Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10
-0+ +--+ | |<------->| Kernel | + | |<----- >| | HCall | | + +--+ DRC2 +----------+ +---------+ + +PHYP terms shared hardware resources like PCI devices, NVDimms etc available for +use by LPARs as Dynamic Resource (DR). When a DR is allocated to an LPAR, PHYP +creates a data-structure called Dynamic Resource Connector (DRC) to manage LPAR +access. An LPAR refers to a DRC via an opaque 32-bit number called DRC-Index. +The DRC-index value is provided to the LPAR via device-tree where its present +as an attribute in the device tree node associated with the DR.
Should you use the term 'Hypervisor' instead of 'PHYP' which is not usually designing only the proprietary one ?
Thanks,
Laurent.
Quoted 88 lines
Documentation/powerpc/hcalls.txt new file +87
-0+ +HCALL Op-codes +============== + +Below is a partial of of HCALLs that are supported by PHYP. For the +corresponding opcode values please look into the header +'arch/powerpc/include/asm/hvcall.h' : + +* H_SCM_READ_METADATA: + Input: drcIndex, offset, buffer-address, numBytesToRead + Out: None + Description: + Given a DRC Index of an NVDimm, read N-bytes from the the meta data area + associated with it, at a specified offset and copy it to provided buffer. + The metadata area stores configuration information such as label information, + bad-blocks etc. The metadata area is located out-of-band of NVDimm storage + area hence a separate access semantics is provided. + +* H_SCM_WRITE_METADATA: + Input: drcIndex, offset, data, numBytesToWrite + Out: None + Description: + Given a DRC Index of an NVDimm, write N-bytes from provided buffer at the + given offset to the the meta data area associated with the NVDimm. + + +* H_SCM_BIND_MEM: + Input: drcIndex, startingScmBlockIndex, numScmBlocksToBind, targetAddress + Out: guestMappedAddress, numScmBlockBound + Description: + Given a DRC-Index of an NVDimm, maps the SCM (Storage Class Memory) blocks to + continuous logical addresses in guest physical address space. The HCALL + arguments can be used to map partial range of SCM blocks instead of entire + NVDimm range to the LPAR. + +* H_SCM_UNBIND_MEM: + Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind + Out: numScmBlocksUnbound + Description: + Given a DRC-Index of an NVDimm, unmap one or more the SCM blocks from guest + physical address space. The HCALL can fail if the Guest has an active PTE + entry to the SCM block being unbinded. + +* H_SCM_QUERY_BLOCK_MEM_BINDING: + Input: drcIndex, scmBlockIndex + Out: Guest-Physical-Address + Description: + Given a DRC-Index and an SCM Block index return the guest physical address to + which the SCM block is mapped to. + +* H_SCM_QUERY_LOGICAL_MEM_BINDING: + Input: Guest-Physical-Address + Out: drcIndex, scmBlockIndex + Description: + Given a guest physical address return which DRC Index and SCM block is mapped + to that address. + +* H_SCM_UNBIND_ALL: + Input: scmTargetScope, drcIndex + Out: None + Description: + Depending on the Target scope unmap all scm blocks belonging to all NVDimms + or all scm blocks belonging to a single NVDimm identified by its drcIndex + from the LPAR memory. + +* H_SCM_HEALTH: + Input: drcIndex + Output: health-bitmap, health-bit-valid-bitmap + Description: + Given a DRC Index return the info on predictive failure and over all health of + the NVDimm. The asserted bits in the health-bitmap indicate a single predictive + failure and health-bit-valid-bitmap indicate which bits in health-bitmap are + valid. + + +* H_SCM_PERFORMANCE_STATS: + Input: drcIndex, resultBuffer Addr + Out: None + Description: + Given a DRC Index collect the performance statistics for NVDimm and copy them + to the resultBuffer. + + +References +========== +[1]: "Linux on Power Architecture Platform Reference" + https://members.openpowerfoundation.org/document/dl/469
Re: [DOC][PATCH v5 1/4] powerpc: Document some HCalls for Storage Class Memory
Quoted 91 lines · On Wed, 24 Jul 2019 11:08:58 +0200 Laurent Dufour <[email protected]>…
Le 23/07/2019 à 18:13, Vaibhav Jain a écrit :
Quoted 8 lines
This doc patch provides an initial description of the HCall op-codes that are used by Linux kernel running as a guest operating system (LPAR) on top of PowerVM or any other sPAPR compliant hyper-visor (e.g qemu).
Apart from documenting the HCalls the doc-patch also provides a rudimentary overview of how Hcalls are implemented inside the Linux kernel and how information flows between kernel and PowerVM/KVM.
Hi Vaibhav,
That's a good idea to introduce such a documentation.
Quoted 35 lines
Signed-off-by Vaibhav Jain <[email protected]>---
Change-log:v5
* First patch in this patchset.--- Documentation/powerpc/hcalls.txt | 140 +++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+)create mode 100644
Documentation/powerpc/hcalls.txtDocumentation/powerpc/hcalls.txt new file +18
-0@@ -0,0 +1,140 @@ +Hyper-visor Call Op-codes (HCALLS) +==================================== + +Overview +========= + +Virtualization on PPC64 arch is based on the PAPR specification[1] which +describes run-time environment for a guest operating system and how it should +interact with the hyper-visor for privileged operations. Currently there are two +PAPR compliant hypervisors (PHYP): + +IBM PowerVM: IBM's proprietary hyper-visor that supports AIX, IBM-i and Linux as + supported guests (termed as Logical Partitions or LPARS). + +Qemu/KVM: Supports PPC64 linux guests running on a PPC64 linux host. + +On PPC64 arch a virtualized guest kernel runs in a non-privileged mode (HV=0). +Hence to perform a privileged operations the guest issues a Hyper-visorDocumentation/powerpc/hcalls.txt line 28
Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10
-0+Call (HCALL) with necessary input operands. PHYP after performing the privilege +operation returns a status code and output operands back to the guest. + +HCALL ABI +========= +The ABI specification for a HCall between guest os kernel and PHYP is +described in [1]. The Opcode for Hcall is set in R3 and subsequent in-arguments +for the Hcall are provided in registers R4-R12. On return from 'HVCS' +instruction the status code of HCall is available in R3 an the output parameters +are returned in registers R4-R12.Would it be good to mention that values passed through the memory must be stored in Big Endian format ?
Quoted 14 lines
Documentation/powerpc/hcalls.txt new file +14
-0+Powerpc arch code provides convenient wrappers named plpar_hcall_xxx defined in +header 'hvcall.h' to issue HCalls from the linux kernel running as guest. + + +DRC & DRC Indexes +================= + + PAPR Guest + DR1 Hypervisor OS + +--+ +----------+ +---------+ + | |<------>| | | User | + +--+ DRC1 | | DRC | Space | + | | Index +---------+ + DR2 | | | |Documentation/powerpc/hcalls.txt line 53
Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10
-0+ +--+ | |<------->| Kernel | + | |<----- >| | HCall | | + +--+ DRC2 +----------+ +---------+ + +PHYP terms shared hardware resources like PCI devices, NVDimms etc available for +use by LPARs as Dynamic Resource (DR). When a DR is allocated to an LPAR, PHYP +creates a data-structure called Dynamic Resource Connector (DRC) to manage LPAR +access. An LPAR refers to a DRC via an opaque 32-bit number called DRC-Index. +The DRC-index value is provided to the LPAR via device-tree where its present +as an attribute in the device tree node associated with the DR.Should you use the term 'Hypervisor' instead of 'PHYP' which is not usually designing only the proprietary one ?
Quoted 8 lines
Thanks,
Laurent.Documentation/powerpc/hcalls.txt line 58
Quoted 5 lines
Documentation/powerpc/hcalls.txt new file +5
-0+ +HCALL Op-codes +============== + +Below is a partial of of HCALLs that are supported by PHYP. For the
^^ list?
Thanks
Michal
Quoted 84 lines
Quoted 83 lines
Documentation/powerpc/hcalls.txt new file +82
-0+corresponding opcode values please look into the header +'arch/powerpc/include/asm/hvcall.h' : + +* H_SCM_READ_METADATA: + Input: drcIndex, offset, buffer-address, numBytesToRead + Out: None + Description: + Given a DRC Index of an NVDimm, read N-bytes from the the meta data area + associated with it, at a specified offset and copy it to provided buffer. + The metadata area stores configuration information such as label information, + bad-blocks etc. The metadata area is located out-of-band of NVDimm storage + area hence a separate access semantics is provided. + +* H_SCM_WRITE_METADATA: + Input: drcIndex, offset, data, numBytesToWrite + Out: None + Description: + Given a DRC Index of an NVDimm, write N-bytes from provided buffer at the + given offset to the the meta data area associated with the NVDimm. + + +* H_SCM_BIND_MEM: + Input: drcIndex, startingScmBlockIndex, numScmBlocksToBind, targetAddress + Out: guestMappedAddress, numScmBlockBound + Description: + Given a DRC-Index of an NVDimm, maps the SCM (Storage Class Memory) blocks to + continuous logical addresses in guest physical address space. The HCALL + arguments can be used to map partial range of SCM blocks instead of entire + NVDimm range to the LPAR. + +* H_SCM_UNBIND_MEM: + Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind + Out: numScmBlocksUnbound + Description: + Given a DRC-Index of an NVDimm, unmap one or more the SCM blocks from guest + physical address space. The HCALL can fail if the Guest has an active PTE + entry to the SCM block being unbinded. + +* H_SCM_QUERY_BLOCK_MEM_BINDING: + Input: drcIndex, scmBlockIndex + Out: Guest-Physical-Address + Description: + Given a DRC-Index and an SCM Block index return the guest physical address to + which the SCM block is mapped to. + +* H_SCM_QUERY_LOGICAL_MEM_BINDING: + Input: Guest-Physical-Address + Out: drcIndex, scmBlockIndex + Description: + Given a guest physical address return which DRC Index and SCM block is mapped + to that address. + +* H_SCM_UNBIND_ALL: + Input: scmTargetScope, drcIndex + Out: None + Description: + Depending on the Target scope unmap all scm blocks belonging to all NVDimms + or all scm blocks belonging to a single NVDimm identified by its drcIndex + from the LPAR memory. + +* H_SCM_HEALTH: + Input: drcIndex + Output: health-bitmap, health-bit-valid-bitmap + Description: + Given a DRC Index return the info on predictive failure and over all health of + the NVDimm. The asserted bits in the health-bitmap indicate a single predictive + failure and health-bit-valid-bitmap indicate which bits in health-bitmap are + valid. + + +* H_SCM_PERFORMANCE_STATS: + Input: drcIndex, resultBuffer Addr + Out: None + Description: + Given a DRC Index collect the performance statistics for NVDimm and copy them + to the resultBuffer. + + +References +========== +[1]: "Linux on Power Architecture Platform Reference" + https://members.openpowerfoundation.org/document/dl/469
Re: [DOC][PATCH v5 1/4] powerpc: Document some HCalls for Storage Class Memory
Thanks everyone for reviewing this patch-set. The V4 got merged upstream which didn't have this DOC-patch. So, I will re-spin a separate and independent doc-patch incorporating your review comments.
Cheers,
Signature · 2 lines
Vaibhav Jain <[email protected]> Linux Technology Center, IBM India Pvt. Ltd.
Michael Ellerman
reviewerRe: [DOC][PATCH v5 1/4] powerpc: Document some HCalls for Storage Class Memory
In reply to Vaibhav JainHi Vaibhav,
Thanks for writing this documentation.
Quoted 7 lines · Vaibhav Jain
This doc patch provides an initial description of the HCall op-codes that are used by Linux kernel running as a guest operating system (LPAR) on top of PowerVM or any other sPAPR compliant hyper-visor (e.g qemu).
Apart from documenting the HCalls the doc-patch also provides a rudimentary overview of how Hcalls are implemented inside the Linux
I prefer "hcall" rather than "HCall", "Hcall" or "HCALL".
Quoted 6 lines
kernel and how information flows between kernel and PowerVM/KVM.
Signed-off-by Vaibhav Jain <[email protected]>---
Change-log:
Quoted 6 lines
v5
* First patch in this patchset.--- Documentation/powerpc/hcalls.txt | 140 +++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 Documentation/powerpc/hcalls.txt
During this merge window all the existing documentation in there has been converted to rst format. I suspect Jon will be annoyed at us if we immediately start adding new plain text docs.
So we need to make this an rst file and do at least a minimal job of making it valid ReST.
A while back I worked out how to actually build the rst docs, I'll try and find my notes on that.
Documentation/powerpc/hcalls.txt
Quoted 5 lines
Documentation/powerpc/hcalls.txt new file +0
-0
Can we call it papr_hcalls please.
Documentation/powerpc/hcalls.txt line 2
Quoted 3 lines
Documentation/powerpc/hcalls.txt new file +2
-0@@ -0,0 +1,140 @@ +Hyper-visor Call Op-codes (HCALLS) +====================================
I'm not sure if "hyper-visor" is correct, but regardless that's not what we commonly use, so please just spell it "hypervisor". And similarly for "opcode".
Documentation/powerpc/hcalls.txt line 7
Quoted 4 lines
Documentation/powerpc/hcalls.txt new file +4
-0+Overview +========= + +Virtualization on PPC64 arch is based on the PAPR specification[1] which
We should probably say on "64-bit Power Book3S platforms".
Also the link you provide is to the "LoPAPR" specification. Which is not quite the same as "PAPR", which used to be released via power.org, or "PAPR+" which is the IBM internal version.
Documentation/powerpc/hcalls.txt line 10
Quoted 3 lines
Documentation/powerpc/hcalls.txt new file +3
-0+describes run-time environment for a guest operating system and how it should +interact with the hyper-visor for privileged operations. Currently there are two +PAPR compliant hypervisors (PHYP):
"PHYP" is only used as a another name for PowerVM (or part of PowerVM), not as a generic term for a PAPR hypervisor.
I also don't think it's accurate to say Qemu/KVM is a "PAPR compliant" hypervisor, it just implements (some of) the relevant parts of LoPAPR to support Linux guests.
Documentation/powerpc/hcalls.txt line 15
Quoted 4 lines
Documentation/powerpc/hcalls.txt new file +4
-0+IBM PowerVM: IBM's proprietary hyper-visor that supports AIX, IBM-i and Linux as + supported guests (termed as Logical Partitions or LPARS). + +Qemu/KVM: Supports PPC64 linux guests running on a PPC64 linux host.
And actually other hosts via Qemu TCG but that's a bit of a tangent.
Documentation/powerpc/hcalls.txt line 17
Quoted 1 line
Documentation/powerpc/hcalls.txt new file +1
-0+On PPC64 arch a virtualized guest kernel runs in a non-privileged mode (HV=0).
"virtualized guest" is redundant, just "guest" is sufficient IMHO.
I know what you're trying to say there, but it actually contradicts the language that's used in the ISA.
If you look at Chapter 1 of Book III it defines these terms:
* hypervisor privileged
A term used to describe an instruction or facility that is available only when the thread is in hypervisor state.
* privileged state and supervisor mode
Used interchangeably to refer to a state in which privileged facilities are available.
* problem state and user mode
Used interchangeably to refer to a state in which privileged facilities are not available.
So you might want to write it more like:
When running under a PAPR hypervisor the guest kernel runs in supervisor mode (HV=0), and must issue hypercalls to the hypervisor whenever it needs to perform an action that is hypervisor privileged or for other services managed by the hypervisor.
Documentation/powerpc/hcalls.txt line 24
Quoted 7 lines
Documentation/powerpc/hcalls.txt new file +7
-0+Hence to perform a privileged operations the guest issues a Hyper-visor +Call (HCALL) with necessary input operands. PHYP after performing the privilege +operation returns a status code and output operands back to the guest. + +HCALL ABI +========= +The ABI specification for a HCall between guest os kernel and PHYP is
^
"OS", but probably just "guest kernel" is fine.
Documentation/powerpc/hcalls.txt line 25
Quoted 1 line
Documentation/powerpc/hcalls.txt new file +1
-0+described in [1]. The Opcode for Hcall is set in R3 and subsequent in-arguments
Where in [1] ?
It's more common to spell the GPRs with lower case 'r', eg. 'r3'.
Documentation/powerpc/hcalls.txt line 26
Quoted 1 line
Documentation/powerpc/hcalls.txt new file +1
-0+for the Hcall are provided in registers R4-R12. On return from 'HVCS'
^the 'HVSC'
Documentation/powerpc/hcalls.txt line 27
Quoted 1 line
Documentation/powerpc/hcalls.txt new file +1
-0+instruction the status code of HCall is available in R3 an the output parameters
^ ^ ^
return value the andQuoted 11 lines
Documentation/powerpc/hcalls.txt new file +11
-0+are returned in registers R4-R12. + +Powerpc arch code provides convenient wrappers named plpar_hcall_xxx defined in +header 'hvcall.h' to issue HCalls from the linux kernel running as guest. + + +DRC & DRC Indexes +================= + + PAPR Guest + DR1 Hypervisor OS
Documentation/powerpc/hcalls.txt line 48
Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10
-0+ +--+ +----------+ +---------+ + | |<------>| | | User | + +--+ DRC1 | | DRC | Space | + | | Index +---------+ + DR2 | | | | + +--+ | |<------->| Kernel | + | |<----- >| | HCall | | + +--+ DRC2 +----------+ +---------+ + +PHYP terms shared hardware resources like PCI devices, NVDimms etc available for
DIMM is an acronym too, so it should be either nvdimm or NVDIMM IMHO.
Quoted 12 lines
Documentation/powerpc/hcalls.txt new file +12
-0+use by LPARs as Dynamic Resource (DR). When a DR is allocated to an LPAR, PHYP +creates a data-structure called Dynamic Resource Connector (DRC) to manage LPAR +access. An LPAR refers to a DRC via an opaque 32-bit number called DRC-Index. +The DRC-index value is provided to the LPAR via device-tree where its present +as an attribute in the device tree node associated with the DR. + +HCALL Op-codes +============== + +Below is a partial of of HCALLs that are supported by PHYP. For the +corresponding opcode values please look into the header +'arch/powerpc/include/asm/hvcall.h' :
Documentation/powerpc/hcalls.txt line 70
Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10
-0+ +* H_SCM_READ_METADATA: + Input: drcIndex, offset, buffer-address, numBytesToRead + Out: None + Description: + Given a DRC Index of an NVDimm, read N-bytes from the the meta data area + associated with it, at a specified offset and copy it to provided buffer. + The metadata area stores configuration information such as label information, + bad-blocks etc. The metadata area is located out-of-band of NVDimm storage + area hence a separate access semantics is provided.
Can you also document the possible return values? (If they are defined).
Quoted 6 lines
Documentation/powerpc/hcalls.txt new file +6
-0+* H_SCM_WRITE_METADATA: + Input: drcIndex, offset, data, numBytesToWrite + Out: None + Description: + Given a DRC Index of an NVDimm, write N-bytes from provided buffer at the + given offset to the the meta data area associated with the NVDimm.
Documentation/powerpc/hcalls.txt line 87
Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10
-0+ + +* H_SCM_BIND_MEM: + Input: drcIndex, startingScmBlockIndex, numScmBlocksToBind, targetAddress + Out: guestMappedAddress, numScmBlockBound + Description: + Given a DRC-Index of an NVDimm, maps the SCM (Storage Class Memory) blocks to + continuous logical addresses in guest physical address space. The HCALL + arguments can be used to map partial range of SCM blocks instead of entire + NVDimm range to the LPAR.
What address space do targetAddress and guestMappedAddress exist in?
What are these blocks and how do we find out about them? ie. how do I know what a valid block index is.
Documentation/powerpc/hcalls.txt line 90
Quoted 2 lines
Documentation/powerpc/hcalls.txt new file +2
-0+* H_SCM_UNBIND_MEM: + Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind
Similar question for startingScmLogicalMemoryAddress.
Documentation/powerpc/hcalls.txt line 93
Quoted 3 lines
Documentation/powerpc/hcalls.txt new file +3
-0+ Out: numScmBlocksUnbound + Description: + Given a DRC-Index of an NVDimm, unmap one or more the SCM blocks from guest
^
ofDocumentation/powerpc/hcalls.txt line 95
Quoted 2 lines
Documentation/powerpc/hcalls.txt new file +2
-0+ physical address space. The HCALL can fail if the Guest has an active PTE + entry to the SCM block being unbinded.
unbound
Quoted 9 lines
Documentation/powerpc/hcalls.txt new file +9
-0+* H_SCM_QUERY_BLOCK_MEM_BINDING: + Input: drcIndex, scmBlockIndex + Out: Guest-Physical-Address + Description: + Given a DRC-Index and an SCM Block index return the guest physical address to + which the SCM block is mapped to. + +* H_SCM_QUERY_LOGICAL_MEM_BINDING: + Input: Guest-Physical-Address
Documentation/powerpc/hcalls.txt line 115
Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10
-0+ Out: drcIndex, scmBlockIndex + Description: + Given a guest physical address return which DRC Index and SCM block is mapped + to that address. + +* H_SCM_UNBIND_ALL: + Input: scmTargetScope, drcIndex + Out: None + Description: + Depending on the Target scope unmap all scm blocks belonging to all NVDimms
^
SCM for consistency.
Documentation/powerpc/hcalls.txt line 123
Quoted 8 lines
Documentation/powerpc/hcalls.txt new file +8
-0+ or all scm blocks belonging to a single NVDimm identified by its drcIndex + from the LPAR memory. + +* H_SCM_HEALTH: + Input: drcIndex + Output: health-bitmap, health-bit-valid-bitmap + Description: + Given a DRC Index return the info on predictive failure and over all health of
overall
Documentation/powerpc/hcalls.txt line 126
Quoted 3 lines
Documentation/powerpc/hcalls.txt new file +3
-0+ the NVDimm. The asserted bits in the health-bitmap indicate a single predictive + failure and health-bit-valid-bitmap indicate which bits in health-bitmap are + valid.
Presumably using bit endian bit ordering?
Documentation/powerpc/hcalls.txt line 140
Quoted 12 lines
Documentation/powerpc/hcalls.txt new file +12
-0+* H_SCM_PERFORMANCE_STATS: + Input: drcIndex, resultBuffer Addr + Out: None + Description: + Given a DRC Index collect the performance statistics for NVDimm and copy them + to the resultBuffer. + + +References +========== +[1]: "Linux on Power Architecture Platform Reference" + https://members.openpowerfoundation.org/document/dl/469
cheers
PATCH v5 2/4 powerpc/pseries: Update SCM hcall op-codes in hvcall.h
In reply to Vaibhav JainUpdate the hvcalls.h to include op-codes for new hcalls introduce to manage SCM memory. Also update existing hcall definitions to reflect current papr specification for SCM.
The removed hcall op-codes H_SCM_MEM_QUERY, H_SCM_BLOCK_CLEAR were transient proposals and there support was never implemented by Power-VM nor they were used anywhere in Linux kernel. Hence we don't expect anyone to be impacted by this change.
---
Change-log:
v5:
* None. Re-spinning the patchset.
v4:
* Updated the patch description mentioned current status of removed hcall opcodes. [Mpe]
v3:
* Added updated opcode for H_SCM_HEALTH [Oliver]
v2:
* None new patch in this series.
--- arch/powerpc/include/asm/hvcall.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
arch/powerpc/include/asm/hvcall.h +8 -3
@@ -302,9 +302,14 @@#define H_SCM_UNBIND_MEM 0x3F0#define H_SCM_QUERY_BLOCK_MEM_BINDING 0x3F4#define H_SCM_QUERY_LOGICAL_MEM_BINDING 0x3F8 -#define H_SCM_MEM_QUERY 0x3FC -#define H_SCM_BLOCK_CLEAR 0x400 -#define MAX_HCALL_OPCODE H_SCM_BLOCK_CLEAR +#define H_SCM_UNBIND_ALL 0x3FC +#define H_SCM_HEALTH 0x400 +#define H_SCM_PERFORMANCE_STATS 0x418 +#define MAX_HCALL_OPCODE H_SCM_PERFORMANCE_STATS + +/* Scope args for H_SCM_UNBIND_ALL */ +#define H_UNBIND_SCOPE_ALL (0x1) +#define H_UNBIND_SCOPE_DRC (0x2)/* H_VIOCTL functions */ #define H_GET_VIOA_DUMP_SIZE 0x01
Signature · 1 line
2.21.0
Re: [PATCH v5 2/4] powerpc/pseries: Update SCM hcall op-codes in hvcall.h
Quoted 10 lines · Vaibhav Jain
Update the
hvcalls.hto include op-codes for new hcalls introduce to manage SCM memory. Also update existing hcall definitions to reflect current papr specification for SCM.The removed hcall op-codes
H_SCM_MEM_QUERY,H_SCM_BLOCK_CLEARwere transient proposals and there support was never implemented by Power-VM nor they were used anywhere in Linux kernel. Hence we don't expect anyone to be impacted by this change.Signed-off-by Vaibhav Jain <[email protected]>
They really should not have been merged while only interim proposals. But since they have changed, better to update them than not, obviously.
Quoted 41 lines
---
Change-log:v5:
* None. Re-spinning the patchset.v4:
* Updated the patch description mentioned current status of removed hcall opcodes. [Mpe]v3:
* Added updated opcode forH_SCM_HEALTH[Oliver]v2:
* None new patch in this series.--- arch/powerpc/include/asm/hvcall.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)arch/powerpc/include/asm/hvcall.h +8
-3@@ -302,9 +302,14 @@#define H_SCM_UNBIND_MEM 0x3F0#define H_SCM_QUERY_BLOCK_MEM_BINDING 0x3F4#define H_SCM_QUERY_LOGICAL_MEM_BINDING 0x3F8 -#define H_SCM_MEM_QUERY 0x3FC -#define H_SCM_BLOCK_CLEAR 0x400 -#define MAX_HCALL_OPCODE H_SCM_BLOCK_CLEAR +#define H_SCM_UNBIND_ALL 0x3FC +#define H_SCM_HEALTH 0x400 +#define H_SCM_PERFORMANCE_STATS 0x418 +#define MAX_HCALL_OPCODE H_SCM_PERFORMANCE_STATS + +/* Scope args for H_SCM_UNBIND_ALL */ +#define H_UNBIND_SCOPE_ALL (0x1) +#define H_UNBIND_SCOPE_DRC (0x2)/* H_VIOCTL functions */ #define H_GET_VIOA_DUMP_SIZE 0x01
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Michael Ellerman
reviewerRe: [PATCH v5 2/4] powerpc/pseries: Update SCM hcall op-codes in hvcall.h
Quoted 10 lines · David Gibson
Quoted 9 lines · Vaibhav Jain
Update the
hvcalls.hto include op-codes for new hcalls introduce to manage SCM memory. Also update existing hcall definitions to reflect current papr specification for SCM.The removed hcall op-codes
H_SCM_MEM_QUERY,H_SCM_BLOCK_CLEARwere transient proposals and there support was never implemented by Power-VM nor they were used anywhere in Linux kernel. Hence we don't expect anyone to be impacted by this change.
Quoted 4 lines
Quoted 1 line
Signed-off-by Vaibhav Jain <[email protected]>They really should not have been merged while only interim proposals. But since they have changed, better to update them than not, obviously.
Yes, absolutely agree. It wasn't clear when I merged them that they were *unpublished* and *unstable*.
Unfortunately we can't realistically wait for these APIs to be in a published PAPR spec because it seems to take about 5 years for a PAPR spec to escape the IBM firewall.
cheers
PATCH v5 3/4 powerpc/papr_scm: Update drc_pmem_unbind() to use H_SCM_UNBIND_ALL
In reply to Vaibhav JainThe new hcall named H_SCM_UNBIND_ALL has been introduce that can unbind all or specific scm memory assigned to an lpar. This is more efficient than using H_SCM_UNBIND_MEM as currently we don't support partial unbind of scm memory.
Hence this patch proposes following changes to drc_pmem_unbind():
* Update drc_pmem_unbind() to replace hcall H_SCM_UNBIND_MEM to H_SCM_UNBIND_ALL.
* Update drc_pmem_unbind() to handles cases when PHYP asks the guest kernel to wait for specific amount of time before retrying the hcall via the 'LONG_BUSY' return value.
* Ensure appropriate error code is returned back from the function in case of an error.
---
Change-log:
v5:
* None. Re-spinning the patchset.
v4:
* None. Re-spinning the patchset.
v3:
* Fixed a build warning reported by kbuild-robot.
* Updated patch description to put emphasis on 'scm memory' instead of 'scm drc memory blocks' as for phyp there is a stark difference between how drc are managed for scm memory v/s regular memory. [Oliver]
v2:
* Added a dev_dbg when unbind operation succeeds [Oliver]
* Changed type of variable 'rc' to int64_t [Oliver]
* Removed the code that was logging a warning in case bind operation takes >1-seconds [Oliver]
* Spinned off changes to hvcall.h as a separate patch. [Oliver]
--- arch/powerpc/platforms/pseries/papr_scm.c | 29 +++++++++++++++++------ 1 file changed, 22 insertions(+), 7 deletions(-)
arch/powerpc/platforms/pseries/papr_scm.c +22 -7
@@ -11,6 +11,7 @@#include <linux/sched.h> #include <linux/libnvdimm.h> #include <linux/platform_device.h> +#include <linux/delay.h> #include <asm/plpar_wrappers.h>
@@ -78,22 +79,36 @@ static int drc_pmem_bind(struct papr_scm_priv *p)static int drc_pmem_unbind (struct papr_scm_priv *p) {unsigned long ret[PLPAR_HCALL_BUFSIZE ]; -uint64_t rc, token; +uint64_t token =0 ; +int64_t rc; - token =0 ; +dev_dbg (&p->pdev->dev,"unbind drc %x\n" , p->drc_index); -/* NB: unbind has the same retry requirements mentioned above */ +/* NB: unbind has the same retry requirements as drc_pmem_bind() */ do { - rc =plpar_hcall (H_SCM_UNBIND_MEM , ret, p->drc_index, - p->bound_addr, p->blocks, token); + +/* Unbind of all SCM resources associated with drcIndex */ + rc =plpar_hcall (H_SCM_UNBIND_ALL , ret,H_UNBIND_SCOPE_DRC , + p->drc_index, token); token = ret[0 ]; -cond_resched (); + +/* Check if we are stalled for some time */ +if (H_IS_LONG_BUSY (rc)) { +msleep (get_longbusy_msecs (rc)); + rc =H_BUSY ; + }else if (rc ==H_BUSY ) { +cond_resched (); + } + }while (rc ==H_BUSY );if (rc)dev_err (&p->pdev->dev,"unbind error: %lld\n" , rc); +else +dev_dbg (&p->pdev->dev,"unbind drc %x complete\n" , + p->drc_index); -return !!rc; +return rc ==H_SUCCESS ?0 : -ENXIO ; } static int papr_scm_meta_get(struct papr_scm_priv *p,
Signature · 1 line
2.21.0
PATCH v5 4/4 powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails
In reply to Vaibhav JainIn some cases initial bind of scm memory for an lpar can fail if previously it wasn't released using a scm-unbind hcall. This situation can arise due to panic of the previous kernel or forced lpar fadump. In such cases the H_SCM_BIND_MEM return a H_OVERLAP error.
To mitigate such cases the patch updates papr_scm_probe() to force a call to drc_pmem_unbind() in case the initial bind of scm memory fails with EBUSY error. In case scm-bind operation again fails after the forced scm-unbind then we follow the existing error path. We also update drc_pmem_bind() to handle the H_OVERLAP error returned by phyp and indicate it as a EBUSY error back to the caller.
---
Change-log:
v5:
* None. Re-spinning the patchset.
v4:
* None. Re-spinning the patchset.
v3:
* Minor update to a code comment. [Oliver]
v2:
* Moved the retry code from drc_pmem_bind() to papr_scm_probe() [Oliver]
* Changed the type of variable 'rc' in drc_pmem_bind() to int64_t. [Oliver]
--- arch/powerpc/platforms/pseries/papr_scm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
arch/powerpc/platforms/pseries/papr_scm.c +14 -1
@@ -44,8 +44,9 @@ struct papr_scm_priv {static int drc_pmem_bind (struct papr_scm_priv *p) {unsigned long ret[PLPAR_HCALL_BUFSIZE ]; -uint64_t rc, token;uint64_t saved =0 ; +uint64_t token; +int64_t rc;/* * When the hypervisor cannot map all the requested memory in a single
@@ -65,6 +66,10 @@ static int drc_pmem_bind(struct papr_scm_priv *p) }while (rc ==H_BUSY );if (rc) { +/* H_OVERLAP needs a separate error path */ +if (rc ==H_OVERLAP ) +return -EBUSY ; +dev_err (&p->pdev->dev,"bind err: %lld\n" , rc);return -ENXIO ; }
@@ -404,6 +409,14 @@ static int papr_scm_probe(struct platform_device *pdev)/* request the hypervisor to bind this region to somewhere in memory */ rc =drc_pmem_bind (p); + +/* If phyp says drc memory still bound then force unbound and retry */ +if (rc == -EBUSY ) { +dev_warn (&pdev->dev,"Retrying bind after unbinding\n" ); +drc_pmem_unbind (p); + rc =drc_pmem_bind (p); + } +if (rc) goto err;
Signature · 1 line
2.21.0
Re: [PATCH v5 4/4] powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails
Le 23/07/2019 à 18:13, Vaibhav Jain a écrit :
Quoted 63 lines
In some cases initial bind of scm memory for an lpar can fail if previously it wasn't released using a scm-unbind hcall. This situation can arise due to panic of the previous kernel or forced lpar fadump. In such cases the
H_SCM_BIND_MEMreturn aH_OVERLAPerror.To mitigate such cases the patch updates
papr_scm_probe()to force a call todrc_pmem_unbind()in case the initial bind of scm memory fails with EBUSY error. In case scm-bind operation again fails after the forced scm-unbind then we follow the existing error path. We also updatedrc_pmem_bind()to handle theH_OVERLAPerror returned by phyp and indicate it as a EBUSY error back to the caller.Suggested-by "Oliver O'Halloran" <[email protected]>Signed-off-by Vaibhav Jain <[email protected]>Reviewed-by Oliver O'Halloran <[email protected]>---
Change-log:v5:
* None. Re-spinning the patchset.v4:
* None. Re-spinning the patchset.v3:
* Minor update to a code comment. [Oliver]v2:
* Moved the retry code fromdrc_pmem_bind()topapr_scm_probe()[Oliver]
* Changed the type of variable 'rc' indrc_pmem_bind()toint64_t. [Oliver]--- arch/powerpc/platforms/pseries/papr_scm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)arch/powerpc/platforms/pseries/papr_scm.c +6
-1@@ -44,8 +44,9 @@ struct papr_scm_priv {static int drc_pmem_bind (struct papr_scm_priv *p) {unsigned long ret[PLPAR_HCALL_BUFSIZE ]; -uint64_t rc, token;uint64_t saved =0 ; +uint64_t token; +int64_t rc;/* * When the hypervisor cannot map all the requested memory in a single @@ -65,6 +66,10 @@ static int drc_pmem_bind(struct papr_scm_priv *p) }while (rc ==H_BUSY );if (rc) { +/* H_OVERLAP needs a separate error path */ +if (rc ==H_OVERLAP ) +return -EBUSY ; +dev_err (&p->pdev->dev,"bind err: %lld\n" , rc);return -ENXIO ; }
arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_probe(struct platform_device *pdev) · line 417
Quoted 9 lines
arch/powerpc/platforms/pseries/papr_scm.c +6
-0@@ -404,6 +409,14 @@ static int papr_scm_probe(struct platform_device *pdev)/* request the hypervisor to bind this region to somewhere in memory */ rc =drc_pmem_bind (p); + +/* If phyp says drc memory still bound then force unbound and retry */ +if (rc == -EBUSY ) { +dev_warn (&pdev->dev,"Retrying bind after unbinding\n" ); +drc_pmem_unbind (p); + rc =drc_pmem_bind (p);
In the unlikely case where H_SCM_BIND_MEM is returning H_OVERLAP once the unbinding has been done, the error would be silently processed. That sounds really unlikely, but should an error message be displayed in this particular case ?
Quoted 5 lines
arch/powerpc/platforms/pseries/papr_scm.c +2
-0+ } +if (rc) goto err;
Re: [PATCH v5 4/4] powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails
Quoted 5 lines · Laurent Dufour
Quoted 3 lines · Vaibhav Jain
*snip*
(quoted diff) +0
-0@@ -404,6 +409,14 @@ static int papr_scm_probe(struct platform_device *pdev)
Quoted 13 lines
arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_probe(struct platform_device *pdev) · line 417
Quoted 8 lines
arch/powerpc/platforms/pseries/papr_scm.c +6
-0/* request the hypervisor to bind this region to somewhere in memory */ rc =drc_pmem_bind (p); + +/* If phyp says drc memory still bound then force unbound and retry */ +if (rc == -EBUSY ) { +dev_warn (&pdev->dev,"Retrying bind after unbinding\n" ); +drc_pmem_unbind (p); + rc =drc_pmem_bind (p);In the unlikely case where
H_SCM_BIND_MEMis returningH_OVERLAPonce the unbinding has been done, the error would be silently processed. That sounds really unlikely, but should an error message be displayed in this particular case ?
drc_pmem_bind() prints the h-call error code if we get one, so it's not silent
Re: [PATCH v5 4/4] powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails
Le 24/07/2019 à 11:24, Oliver O'Halloran a écrit :
Quoted 21 lines
Quoted 5 lines · Laurent Dufour
Quoted 3 lines · Vaibhav Jain
*snip*
(quoted diff) +0
-0@@ -404,6 +409,14 @@ static int papr_scm_probe(struct platform_device *pdev)Quoted 13 lines
arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_probe(struct platform_device *pdev) · line 417
Quoted 8 lines
arch/powerpc/platforms/pseries/papr_scm.c +6
-0/* request the hypervisor to bind this region to somewhere in memory */ rc =drc_pmem_bind (p); + +/* If phyp says drc memory still bound then force unbound and retry */ +if (rc == -EBUSY ) { +dev_warn (&pdev->dev,"Retrying bind after unbinding\n" ); +drc_pmem_unbind (p); + rc =drc_pmem_bind (p);In the unlikely case where
H_SCM_BIND_MEMis returningH_OVERLAPonce the unbinding has been done, the error would be silently processed. That sounds really unlikely, but should an error message be displayed in this particular case ?
Quoted 1 line
drc_pmem_bind()prints the h-call error code if we get one, so it's not silent
That's no more the case whith this patch, H_OVERLAP is handled before writing the error message, which would make sense for the first try.
For the record, the patch introduces:
@@ -65,6 +66,10 @@ static int drc_pmem_bind(struct papr_scm_priv *p)
} while (rc == H_BUSY);
if (rc) {
+ /* H_OVERLAP needs a separate error path */
+ if (rc == H_OVERLAP)
+ return -EBUSY;
+
dev_err(&p->pdev->dev, "bind err: %lld\n", rc); return -ENXIO;
}
Re: [PATCH v5 4/4] powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails
Quoted 31 lines · Laurent Dufour
Quoted 21 lines · Oliver O'Halloran
Quoted 5 lines · Laurent Dufour
Quoted 3 lines · Vaibhav Jain
*snip*
(quoted diff) +0
-0@@ -404,6 +409,14 @@ static int papr_scm_probe(struct platform_device *pdev)Quoted 13 lines
arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_probe(struct platform_device *pdev) · line 417
Quoted 8 lines
arch/powerpc/platforms/pseries/papr_scm.c +6
-0/* request the hypervisor to bind this region to somewhere in memory */ rc =drc_pmem_bind (p); + +/* If phyp says drc memory still bound then force unbound and retry */ +if (rc == -EBUSY ) { +dev_warn (&pdev->dev,"Retrying bind after unbinding\n" ); +drc_pmem_unbind (p); + rc =drc_pmem_bind (p);In the unlikely case where
H_SCM_BIND_MEMis returningH_OVERLAPonce the unbinding has been done, the error would be silently processed. That sounds really unlikely, but should an error message be displayed in this particular case ?Quoted 1 line
drc_pmem_bind()prints the h-call error code if we get one, so it's not silentThat's no more the case whith this patch,
H_OVERLAPis handled before writing the error message, which would make sense for the first try.For the record, the patch introduces:
arch/powerpc/platforms/pseries/papr_scm.c static int drc_pmem_bind(struct papr_scm_priv *p) · line 75
Quoted 10 lines
arch/powerpc/platforms/pseries/papr_scm.c +4
-0@@ -65,6 +66,10 @@ static int drc_pmem_bind(struct papr_scm_priv *p) }while (rc ==H_BUSY );if (rc) { +/* H_OVERLAP needs a separate error path */ +if (rc ==H_OVERLAP ) +return -EBUSY ; +dev_err (&p->pdev->dev,"bind err: %lld\n" , rc);return -ENXIO ; }
Ah, good point. Getting H_OVERLAP is still an error case so I think it's reasonable to still print the message in that case.
Quoted 18 lines · Vaibhav Jain
Presently an error is returned in response to hcall
H_SCM_BIND_MEMwhen a new kernel boots on lpar via kexec. This preventspapr_scmfrom registering drc memory regions with nvdimm. The error reported is of the form below:"
papr_scmibm,persistent-memory:ibm,pmemory@44100002: bind err: -68"On investigation it was revealed that phyp returns this error as previous kernel did not completely release bindings for drc scm-memory blocks and hence phyp rejected request for re-binding these block to lpar with error
H_OVERLAP. Also support for a newH_SCM_UNBIND_ALLis recently added which is better suited for releasing all the bound scm-memory block from an lpar.So leveraging new hcall
H_SCM_UNBIND_ALL, we can workaroundH_OVERLAPissue during kexec by forcing an unbind of all drm scm-memory blocks and issuingH_SCM_BIND_MEMto re-bind the drc scm-memory blocks to lpar. This sequence will also be needed when a new kernel boot on lpar after previous kernel panicked and it never got an opportunity to call H_SCM_UNBIND_MEM/ALL.
Quoted 9 lines
Hence this patch-set implements following changes to
papr_scmmodule:* Update
hvcall.hto include opcodes for new hcallH_SCM_UNBIND_ALL.* Update it to use
H_SCM_UNBIND_ALLinstead ofH_SCM_UNBIND_MEM* In case hcall
H_SCM_BIND_MEMfails with errorH_OVERLAP, forceH_SCM_UNBIND_ALLand retry the bind operation again.
You can add for the series.