arm64: Stolen time support
Subsystems (8)
v6 cover arm64: Stolen time support
This series add support for paravirtualized time for arm64 guests and KVM hosts following the specification in Arm's document DEN 0057A:
https://developer.arm.com/docs/den0057/a
It implements support for stolen time, allowing the guest to identify time when it is forcibly not executing.
Note that Live Physical Time (LPT) which was previously part of the above specification has now been removed.
Also available as a git tree:git://linux-arm.org/linux-sp.git stolen_time/v6
Changes from v5:
https://lore.kernel.org/kvm/[email protected]/
* Convert document to RST format
* Rename PV_FEATURES to PV_TIME_FEATURES to match spec
* Correct SMC number of PV_TIME_ST
Changes from v4:
https://lore.kernel.org/kvm/[email protected]/
* Rebased to v5.4-rc1
* Renamed KVM_ARM_VCPU_PVTIME_SET_IPA to remove _SET as it is used for both set/get operations
* Added kvm/arm_hypercalls.h to header-test-$(CONFIG_ARM{,64}) as it is only buildable on arm/arm64
* Documented no-steal-acc kernel parameter
Changes from v3:
https://lore.kernel.org/lkml/[email protected]/
* There's no longer a PV_TIME device, instead there are attributes on the VCPU. This allows the stolen time structures to be places arbitrarily by user space (subject to 64 byte alignment).
* Split documentation between information on the hypercalls and the attributes on the VCPU
* Fixed the type of SMCCC functions to return long not int
Changes from v2:
https://lore.kernel.org/lkml/[email protected]/
* Switched from using gfn_to_hva_cache to a new macro kvm_put_guest() that can provide the single-copy atomicity required (on arm64). This macro is added in patch 4.
* Tidied up the locking for kvm_update_stolen_time(). pagefault_disable() was unnecessary and the caller didn't need to take kvm->srcu as the function does it itself.
* Removed struct kvm_arch_pvtime from the arm implementation, replaced instead with inline static functions which are empty for arm.
* Fixed a few checkpatch --strict warnings.
Changes from v1:
https://lore.kernel.org/lkml/[email protected]/
* Host kernel no longer allocates the stolen time structure, instead it is allocated by user space. This means the save/restore functionality can be removed.
* Refactored the code so arm has stub implementations and to avoid initcall
* Rebased to pick up Documentation/{virt->virtual} change
* Bunch of typo fixes
Christoffer Dall (1):
KVM: arm/arm64: Factor out hypercall handling from PSCI code
Steven Price (9):
KVM: arm64: Document PV-time interface
KVM: arm64: Implement PV_TIME_FEATURES call
KVM: Implement kvm_put_guest()
KVM: arm64: Support stolen time reporting via shared structure
KVM: Allow kvm_device_ops to be const
KVM: arm64: Provide VCPU attributes for stolen time arm/arm64: Provide a wrapper for SMCCC 1.1 calls arm/arm64: Make use of the SMCCC 1.1 wrapper
arm64: Retrieve stolen time as paravirtualized guest
.../admin-guide/kernel-parameters.txt | 6 +- Documentation/virt/kvm/arm/pvtime.rst | 77 +++++++++ Documentation/virt/kvm/devices/vcpu.txt | 14 ++ arch/arm/include/asm/kvm_host.h | 26 +++ arch/arm/kvm/Makefile | 2 +- arch/arm/kvm/handle_exit.c | 2 +- arch/arm/mm/proc-v7-bugs.c | 13 +- arch/arm64/include/asm/kvm_host.h | 30 +++- arch/arm64/include/asm/paravirt.h | 9 +- arch/arm64/include/asm/pvclock-abi.h | 17 ++ arch/arm64/include/uapi/asm/kvm.h | 2 + arch/arm64/kernel/cpu_errata.c | 80 ++++------ arch/arm64/kernel/paravirt.c | 148 ++++++++++++++++++ arch/arm64/kernel/time.c | 3 + arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/Makefile | 2 + arch/arm64/kvm/guest.c | 9 ++ arch/arm64/kvm/handle_exit.c | 4 +- include/Kbuild | 2 + include/kvm/arm_hypercalls.h | 43 +++++ include/kvm/arm_psci.h | 2 +- include/linux/arm-smccc.h | 58 +++++++ include/linux/cpuhotplug.h | 1 + include/linux/kvm_host.h | 26 ++- include/linux/kvm_types.h | 2 + include/uapi/linux/kvm.h | 2 + virt/kvm/arm/arm.c | 11 ++ virt/kvm/arm/hypercalls.c | 68 ++++++++ virt/kvm/arm/psci.c | 84 +--------- virt/kvm/arm/pvtime.c | 124 +++++++++++++++ virt/kvm/kvm_main.c | 6 +- 31 files changed, 717 insertions(+), 157 deletions(-) create mode 100644 Documentation/virt/kvm/arm/pvtime.rst create mode 100644 arch/arm64/include/asm/pvclock-abi.h create mode 100644 include/kvm/arm_hypercalls.h create mode 100644 virt/kvm/arm/hypercalls.c create mode 100644 virt/kvm/arm/pvtime.c
Signature · 1 line
2.20.1
PATCH v6 1/10 KVM: arm64: Document PV-time interface
Introduce a paravirtualization interface for KVM/arm64 based on the "Arm Paravirtualized Time for Arm-Base Systems" specification DEN 0057A.
This only adds the details about "Stolen Time" as the details of "Live Physical Time" have not been fully agreed.
User space can specify a reserved area of memory for the guest and inform KVM to populate the memory with information on time that the host kernel has stolen from the guest.
A hypercall interface is provided for the guest to interrogate the hypervisor's support for this interface and the location of the shared memory structures.
--- Documentation/virt/kvm/arm/pvtime.rst | 77 +++++++++++++++++++++++++ Documentation/virt/kvm/devices/vcpu.txt | 14 +++++ 2 files changed, 91 insertions(+) create mode 100644 Documentation/virt/kvm/arm/pvtime.rst
Documentation/virt/kvm/arm/pvtime.rst new file +77 -0
@@ -0,0 +1,77 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Paravirtualized time support for arm64 +====================================== + +Arm specification DEN0057/A defines a standard for paravirtualised time +support for AArch64 guests: + +https://developer.arm.com/docs/den0057/a + +KVM/arm64 implements the stolen time part of this specification by providing +some hypervisor service calls to support a paravirtualized guest obtaining a +view of the amount of time stolen from its execution. + +Two new SMCCC compatible hypercalls are defined: + +* PV_TIME_FEATURES: 0xC5000020 +* PV_TIME_ST: 0xC5000021 + +These are only available in the SMC64/HVC64 calling convention as +paravirtualized time is not available to 32 bit Arm guests. The existence of +the PV_FEATURES hypercall should be probed using the SMCCC 1.1 ARCH_FEATURES +mechanism before calling it. + +PV_TIME_FEATURES + ============= ======== ========== + Function ID: (uint32) 0xC5000020 + PV_call_id: (uint32) The function to query for support. + Currently only PV_TIME_ST is supported. + Return value: (int64) NOT_SUPPORTED (-1) or SUCCESS (0) if the relevant + PV-time feature is supported by the hypervisor. + ============= ======== ========== + +PV_TIME_ST + ============= ======== ========== + Function ID: (uint32) 0xC5000021 + Return value: (int64) IPA of the stolen time data structure for this + VCPU. On failure: + NOT_SUPPORTED (-1) + ============= ======== ========== + +The IPA returned by PV_TIME_ST should be mapped by the guest as normal memory +with inner and outer write back caching attributes, in the inner shareable +domain. A total of 16 bytes from the IPA returned are guaranteed to be +meaningfully filled by the hypervisor (see structure below). + +PV_TIME_ST returns the structure for the calling VCPU. + +Stolen Time +----------- + +The structure pointed to by the PV_TIME_ST hypercall is as follows: + ++-------------+-------------+-------------+----------------------------+ +| Field | Byte Length | Byte Offset | Description | ++=============+=============+=============+============================+ +| Revision | 4 | 0 | Must be 0 for version 1.0 | ++-------------+-------------+-------------+----------------------------+ +| Attributes | 4 | 4 | Must be 0 | ++-------------+-------------+-------------+----------------------------+ +| Stolen time | 8 | 8 | Stolen time in unsigned | +| | | | nanoseconds indicating how | +| | | | much time this VCPU thread | +| | | | was involuntarily not | +| | | | running on a physical CPU. | ++-------------+-------------+-------------+----------------------------+ + +All values in the structure are stored little-endian. + +The structure will be updated by the hypervisor prior to scheduling a VCPU. It +will be present within a reserved region of the normal memory given to the +guest. The guest should not attempt to write into this memory. There is a +structure per VCPU of the guest. + +For the user space interface see Documentation/virt/kvm/devices/vcpu.txt +section "3. GROUP: KVM_ARM_VCPU_PVTIME_CTRL". +
Documentation/virt/kvm/devices/vcpu.txt +14 -0
@@ -60,3 +60,17 @@ time to use the number provided for a given timer, overwriting any previously configured values on other VCPUs. Userspace should configure the interrupt numbers on at least one VCPU after creating all VCPUs and before running any VCPUs. + +3. GROUP: KVM_ARM_VCPU_PVTIME_CTRL +Architectures: ARM64 + +3.1 ATTRIBUTE: KVM_ARM_VCPU_PVTIME_IPA +Parameters: 64-bit base address +Returns: -ENXIO: Stolen time not implemented + -EEXIST: Base address already set for this VCPU + -EINVAL: Base address not 64 byte aligned + +Specifies the base address of the stolen time structure for this VCPU. The +base address must be 64 byte aligned and exist within a valid guest memory +region. See Documentation/virt/kvm/arm/pvtime.txt for more information +including the layout of the stolen time structure.
Signature · 1 line
2.20.1
Mark Rutland
maintainerRe: [PATCH v6 01/10] KVM: arm64: Document PV-time interface
Hi Steven,
Quoted 2 lines · Steven Price
Introduce a paravirtualization interface for KVM/arm64 based on the "Arm Paravirtualized Time for Arm-Base Systems" specification DEN 0057A.
I notice that as published, this is a BETA Draft, with the explicit note:
| This document is for review purposes only and should not be used | for any implementation as changes are likely.
... what's the plan for getting a finalised version published?
Quoted 2 lines
This only adds the details about "Stolen Time" as the details of "Live Physical Time" have not been fully agreed.
... and what do we expect to happen on this front?
AFAICT, the spec hasn't changed since I called out issues in that area:
https://lore.kernel.org/r/[email protected]
... and I'd feel much happier about supporting this if that were dropped from the finalised spec.
Quoted 56 lines
User space can specify a reserved area of memory for the guest and inform KVM to populate the memory with information on time that the host kernel has stolen from the guest.
A hypercall interface is provided for the guest to interrogate the hypervisor's support for this interface and the location of the shared memory structures.
Signed-off-by Steven Price <[email protected]>--- Documentation/virt/kvm/arm/pvtime.rst | 77 +++++++++++++++++++++++++ Documentation/virt/kvm/devices/vcpu.txt | 14 +++++ 2 files changed, 91 insertions(+) create mode 100644 Documentation/virt/kvm/arm/pvtime.rstDocumentation/virt/kvm/arm/pvtime.rst new file +35
-0@@ -0,0 +1,77 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Paravirtualized time support for arm64 +====================================== + +Arm specification DEN0057/A defines a standard for paravirtualised time +support for AArch64 guests: + +https://developer.arm.com/docs/den0057/a + +KVM/arm64 implements the stolen time part of this specification by providing +some hypervisor service calls to support a paravirtualized guest obtaining a +view of the amount of time stolen from its execution. + +Two new SMCCC compatible hypercalls are defined: + +* PV_TIME_FEATURES: 0xC5000020 +* PV_TIME_ST: 0xC5000021 + +These are only available in the SMC64/HVC64 calling convention as +paravirtualized time is not available to 32 bit Arm guests. The existence of +the PV_FEATURES hypercall should be probed using the SMCCC 1.1 ARCH_FEATURES +mechanism before calling it. + +PV_TIME_FEATURES + ============= ======== ========== + Function ID: (uint32) 0xC5000020 + PV_call_id: (uint32) The function to query for support. + Currently only PV_TIME_ST is supported. + Return value: (int64) NOT_SUPPORTED (-1) or SUCCESS (0) if the relevant + PV-time feature is supported by the hypervisor. + ============= ======== ========== + +PV_TIME_ST + ============= ======== ==========
Documentation/virt/kvm/arm/pvtime.rst line 45
Quoted 10 lines
Documentation/virt/kvm/arm/pvtime.rst new file +10
-0+ Function ID: (uint32) 0xC5000021 + Return value: (int64) IPA of the stolen time data structure for this + VCPU. On failure: + NOT_SUPPORTED (-1) + ============= ======== ========== + +The IPA returned by PV_TIME_ST should be mapped by the guest as normal memory +with inner and outer write back caching attributes, in the inner shareable +domain. A total of 16 bytes from the IPA returned are guaranteed to be +meaningfully filled by the hypervisor (see structure below).
At what granularity is this allowed to share IPA space with other mappings? The spec doesn't provide any guidance here, and I strongly suspect that it should.
To support a 64K guest, we must ensure that this doesn't share a 64K IPA granule with any MMIO, and it probably only makes sense for an instance of this structure to share that granule with another vCPU's structure.
We probably _also_ want to ensure that this doesn't share a 64K granule with memory the guest sees as regular system RAM. Otherwise we're liable to force it into having mismatched attributes for any of that RAM it happens to map as part of mapping the PV_TIME_ST structure.
Quoted 13 lines
Documentation/virt/kvm/arm/pvtime.rst new file +13
-0+ +PV_TIME_ST returns the structure for the calling VCPU. + +Stolen Time +----------- + +The structure pointed to by the PV_TIME_ST hypercall is as follows: + ++-------------+-------------+-------------+----------------------------+ +| Field | Byte Length | Byte Offset | Description | ++=============+=============+=============+============================+ +| Revision | 4 | 0 | Must be 0 for version 1.0 | ++-------------+-------------+-------------+----------------------------+
Documentation/virt/kvm/arm/pvtime.rst line 68
Quoted 10 lines
Documentation/virt/kvm/arm/pvtime.rst new file +10
-0+| Attributes | 4 | 4 | Must be 0 | ++-------------+-------------+-------------+----------------------------+ +| Stolen time | 8 | 8 | Stolen time in unsigned | +| | | | nanoseconds indicating how | +| | | | much time this VCPU thread | +| | | | was involuntarily not | +| | | | running on a physical CPU. | ++-------------+-------------+-------------+----------------------------+ + +All values in the structure are stored little-endian.
Looking at the published DEN 0057A, endianness is never stated. Is this going to be corrected in the next release?
Thanks,
Mark.
Re: [PATCH v6 01/10] KVM: arm64: Document PV-time interface
Quoted 6 lines · Mark Rutland
Hi Steven,
Quoted 2 lines · Steven Price
Introduce a paravirtualization interface for KVM/arm64 based on the "Arm Paravirtualized Time for Arm-Base Systems" specification DEN 0057A.
Quoted 7 lines
I notice that as published, this is a BETA Draft, with the explicit note:
| This document is for review purposes only and should not be used | for any implementation as changes are likely.
... what's the plan for getting a finalised version published?
Sadly this wasn't handled very well, there's actually *two* documents called DEN/0057A:
* The draft[1] which contains the text above and previous postings were
based on
* The final[2] document which this latest series is updated to.
At least for me if you visit the short form link[3] you can only get to the released version - the version drop down gives you two choices ('a' or 'a') but both get you to the same place.
[1]
https://static.docs.arm.com/den0057/a/den0057a_paravirtualized_time_for_arm-based_systems_beta-2.pdf
[2]
https://static.docs.arm.com/den0057/a/DEN0057A_Paravirtualized_Time_for_Arm-based_Systems_v1.0.pdf
[3] https://developer.arm.com/docs/den0057/a
Quoted 11 lines
Quoted 2 lines
This only adds the details about "Stolen Time" as the details of "Live Physical Time" have not been fully agreed.
... and what do we expect to happen on this front?
AFAICT, the spec hasn't changed since I called out issues in that area:
https://lore.kernel.org/r/[email protected]
... and I'd feel much happier about supporting this if that were dropped from the finalised spec.
LPT has been dropped from the final spec (and annoyingly the SMC values re-assigned). I can't say for sure what the long-term future of LPT is going to be, but for now it's gone.
Quoted 71 lines
Quoted 56 lines
User space can specify a reserved area of memory for the guest and inform KVM to populate the memory with information on time that the host kernel has stolen from the guest.
A hypercall interface is provided for the guest to interrogate the hypervisor's support for this interface and the location of the shared memory structures.
Signed-off-by Steven Price <[email protected]>--- Documentation/virt/kvm/arm/pvtime.rst | 77 +++++++++++++++++++++++++ Documentation/virt/kvm/devices/vcpu.txt | 14 +++++ 2 files changed, 91 insertions(+) create mode 100644 Documentation/virt/kvm/arm/pvtime.rstDocumentation/virt/kvm/arm/pvtime.rst new file +35
-0@@ -0,0 +1,77 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Paravirtualized time support for arm64 +====================================== + +Arm specification DEN0057/A defines a standard for paravirtualised time +support for AArch64 guests: + +https://developer.arm.com/docs/den0057/a + +KVM/arm64 implements the stolen time part of this specification by providing +some hypervisor service calls to support a paravirtualized guest obtaining a +view of the amount of time stolen from its execution. + +Two new SMCCC compatible hypercalls are defined: + +* PV_TIME_FEATURES: 0xC5000020 +* PV_TIME_ST: 0xC5000021 + +These are only available in the SMC64/HVC64 calling convention as +paravirtualized time is not available to 32 bit Arm guests. The existence of +the PV_FEATURES hypercall should be probed using the SMCCC 1.1 ARCH_FEATURES +mechanism before calling it. + +PV_TIME_FEATURES + ============= ======== ========== + Function ID: (uint32) 0xC5000020 + PV_call_id: (uint32) The function to query for support. + Currently only PV_TIME_ST is supported. + Return value: (int64) NOT_SUPPORTED (-1) or SUCCESS (0) if the relevant + PV-time feature is supported by the hypervisor. + ============= ======== ========== + +PV_TIME_ST + ============= ======== ==========Documentation/virt/kvm/arm/pvtime.rst line 45
Quoted 10 lines
Documentation/virt/kvm/arm/pvtime.rst new file +10
-0+ Function ID: (uint32) 0xC5000021 + Return value: (int64) IPA of the stolen time data structure for this + VCPU. On failure: + NOT_SUPPORTED (-1) + ============= ======== ========== + +The IPA returned by PV_TIME_ST should be mapped by the guest as normal memory +with inner and outer write back caching attributes, in the inner shareable +domain. A total of 16 bytes from the IPA returned are guaranteed to be +meaningfully filled by the hypervisor (see structure below).At what granularity is this allowed to share IPA space with other mappings? The spec doesn't provide any guidance here, and I strongly suspect that it should.
Quoted 8 lines
To support a 64K guest, we must ensure that this doesn't share a 64K IPA granule with any MMIO, and it probably only makes sense for an instance of this structure to share that granule with another vCPU's structure.
We probably _also_ want to ensure that this doesn't share a 64K granule with memory the guest sees as regular system RAM. Otherwise we're liable to force it into having mismatched attributes for any of that RAM it happens to map as part of mapping the
PV_TIME_STstructure.
Good points, but this is no different from any other 'device' that is mapped into the guest space (e.g. you don't want a serial port sharing the same 64K page with memory for the same reasons). Ultimately user space must arrange the memory layout appropriately for the guest. Of course we can request this guidance to be added in the next release of the spec.
Quoted 18 lines
Quoted 18 lines
Documentation/virt/kvm/arm/pvtime.rst new file +18
-0+ +PV_TIME_ST returns the structure for the calling VCPU. + +Stolen Time +----------- + +The structure pointed to by the PV_TIME_ST hypercall is as follows: + ++-------------+-------------+-------------+----------------------------+ +| Field | Byte Length | Byte Offset | Description | ++=============+=============+=============+============================+ +| Revision | 4 | 0 | Must be 0 for version 1.0 | ++-------------+-------------+-------------+----------------------------+ +| Attributes | 4 | 4 | Must be 0 | ++-------------+-------------+-------------+----------------------------+ +| Stolen time | 8 | 8 | Stolen time in unsigned | +| | | | nanoseconds indicating how | +| | | | much time this VCPU thread |
Quoted 8 lines
Documentation/virt/kvm/arm/pvtime.rst line 68
Quoted 5 lines
Documentation/virt/kvm/arm/pvtime.rst new file +5
-0+| | | | was involuntarily not | +| | | | running on a physical CPU. | ++-------------+-------------+-------------+----------------------------+ + +All values in the structure are stored little-endian.Looking at the published DEN 0057A, endianness is never stated. Is this going to be corrected in the next release?
I've provided feedback that this should be explicit in the next revision of the spec. So yes, I expect this to be stated. For now though, Linux has to make a decision so I've documented it here.
Steve
Mark Rutland
maintainerRe: [PATCH v6 01/10] KVM: arm64: Document PV-time interface
In reply to Mark RutlandQuoted 6 lines · Mark Rutland
Hi Steven,
Quoted 2 lines · Steven Price
Introduce a paravirtualization interface for KVM/arm64 based on the "Arm Paravirtualized Time for Arm-Base Systems" specification DEN 0057A.
Quoted 7 lines
I notice that as published, this is a BETA Draft, with the explicit note:
| This document is for review purposes only and should not be used | for any implementation as changes are likely.
... what's the plan for getting a finalised version published?
Hmmm. Not sure how I manged to find the old version of the spec given your cover letter links to the finalised version.
Sorry for the noise!
Quoted 4 lines
Quoted 2 lines
This only adds the details about "Stolen Time" as the details of "Live Physical Time" have not been fully agreed.
... and what do we expect to happen on this front?
... and I see this has been removed from the spec for now.
[...]
Quoted 17 lines
Documentation/virt/kvm/arm/pvtime.rst line 45
Quoted 12 lines
Documentation/virt/kvm/arm/pvtime.rst new file +12
-0+PV_TIME_ST + ============= ======== ========== + Function ID: (uint32) 0xC5000021 + Return value: (int64) IPA of the stolen time data structure for this + VCPU. On failure: + NOT_SUPPORTED (-1) + ============= ======== ========== + +The IPA returned by PV_TIME_ST should be mapped by the guest as normal memory +with inner and outer write back caching attributes, in the inner shareable +domain. A total of 16 bytes from the IPA returned are guaranteed to be +meaningfully filled by the hypervisor (see structure below).At what granularity is this allowed to share IPA space with other mappings? The spec doesn't provide any guidance here, and I strongly suspect that it should.
Quoted 8 lines
To support a 64K guest, we must ensure that this doesn't share a 64K IPA granule with any MMIO, and it probably only makes sense for an instance of this structure to share that granule with another vCPU's structure.
We probably _also_ want to ensure that this doesn't share a 64K granule with memory the guest sees as regular system RAM. Otherwise we're liable to force it into having mismatched attributes for any of that RAM it happens to map as part of mapping the
PV_TIME_STstructure.
I guess we can say that it's userspace's responsibiltiy to set this up with sufficient alignment, but I do think we want to make a recommendataion here.
[...]
Quoted 17 lines
Quoted 17 lines
Documentation/virt/kvm/arm/pvtime.rst new file +17
-0+PV_TIME_ST returns the structure for the calling VCPU. + +Stolen Time +----------- + +The structure pointed to by the PV_TIME_ST hypercall is as follows: + ++-------------+-------------+-------------+----------------------------+ +| Field | Byte Length | Byte Offset | Description | ++=============+=============+=============+============================+ +| Revision | 4 | 0 | Must be 0 for version 1.0 | ++-------------+-------------+-------------+----------------------------+ +| Attributes | 4 | 4 | Must be 0 | ++-------------+-------------+-------------+----------------------------+ +| Stolen time | 8 | 8 | Stolen time in unsigned | +| | | | nanoseconds indicating how | +| | | | much time this VCPU thread |
Quoted 8 lines
Review
Quoted 5 lines
+| | | | was involuntarily not | +| | | | running on a physical CPU. |++-------------+-------------+-------------+----------------------------+
(quoted diff) +2
-0+ +All values in the structure are stored little-endian.Looking at the published DEN 0057A, endianness is never stated. Is this going to be corrected in the next release?
I'm assuming that this has been communicated internally, and we can assume the next rev of the spec will state so.
Assuming so, this looks good to me.
Thanks,
Mark.
Re: [PATCH v6 01/10] KVM: arm64: Document PV-time interface
Quoted 1 line · Mark Rutland
On Tue, Oct 15, 2019 at 06:56:51PM +0100, Mark Rutland wrote:
[...]
Quoted 21 lines
Quoted 21 lines
Documentation/virt/kvm/arm/pvtime.rst line 45
Quoted 12 lines
Documentation/virt/kvm/arm/pvtime.rst new file +12
-0+PV_TIME_ST + ============= ======== ========== + Function ID: (uint32) 0xC5000021 + Return value: (int64) IPA of the stolen time data structure for this + VCPU. On failure: + NOT_SUPPORTED (-1) + ============= ======== ========== + +The IPA returned by PV_TIME_ST should be mapped by the guest as normal memory +with inner and outer write back caching attributes, in the inner shareable +domain. A total of 16 bytes from the IPA returned are guaranteed to be +meaningfully filled by the hypervisor (see structure below).At what granularity is this allowed to share IPA space with other mappings? The spec doesn't provide any guidance here, and I strongly suspect that it should.
To support a 64K guest, we must ensure that this doesn't share a 64K IPA granule with any MMIO, and it probably only makes sense for an instance of this structure to share that granule with another vCPU's structure.
Quoted 8 lines
Quoted 4 lines
We probably _also_ want to ensure that this doesn't share a 64K granule with memory the guest sees as regular system RAM. Otherwise we're liable to force it into having mismatched attributes for any of that RAM it happens to map as part of mapping the
PV_TIME_STstructure.I guess we can say that it's userspace's responsibiltiy to set this up with sufficient alignment, but I do think we want to make a recommendataion here.
I can add something like this to the kernel's documentation:
It is advisable that one or more 64k pages are set aside for the purpose of these structures and not used for other purposes, this enables the guest to map the region using 64k pages and avoids conflicting attributes with other memory.
Quoted 28 lines
[...]
Quoted 17 lines
Quoted 17 lines
Documentation/virt/kvm/arm/pvtime.rst new file +17
-0+PV_TIME_ST returns the structure for the calling VCPU. + +Stolen Time +----------- + +The structure pointed to by the PV_TIME_ST hypercall is as follows: + ++-------------+-------------+-------------+----------------------------+ +| Field | Byte Length | Byte Offset | Description | ++=============+=============+=============+============================+ +| Revision | 4 | 0 | Must be 0 for version 1.0 | ++-------------+-------------+-------------+----------------------------+ +| Attributes | 4 | 4 | Must be 0 | ++-------------+-------------+-------------+----------------------------+ +| Stolen time | 8 | 8 | Stolen time in unsigned | +| | | | nanoseconds indicating how | +| | | | much time this VCPU thread |Quoted 8 lines
Review
Quoted 5 lines
+| | | | was involuntarily not | +| | | | running on a physical CPU. |++-------------+-------------+-------------+----------------------------+
(quoted diff) +2
-0+ +All values in the structure are stored little-endian.Looking at the published DEN 0057A, endianness is never stated. Is this going to be corrected in the next release?
Quoted 2 lines
I'm assuming that this has been communicated internally, and we can assume the next rev of the spec will state so.
Yes I've fed that back, so hopefully it should be in the next rev of the spec.
Quoted 1 line
Assuming so, this looks good to me.
Great, thanks for the review.
Steve
Mark Rutland
maintainerRe: [PATCH v6 01/10] KVM: arm64: Document PV-time interface
Quoted 33 lines · Steven Price
Quoted 1 line · Mark Rutland
On Tue, Oct 15, 2019 at 06:56:51PM +0100, Mark Rutland wrote:
[...]
Quoted 21 lines
Quoted 21 lines
Documentation/virt/kvm/arm/pvtime.rst line 45
Quoted 12 lines
Documentation/virt/kvm/arm/pvtime.rst new file +12
-0+PV_TIME_ST + ============= ======== ========== + Function ID: (uint32) 0xC5000021 + Return value: (int64) IPA of the stolen time data structure for this + VCPU. On failure: + NOT_SUPPORTED (-1) + ============= ======== ========== + +The IPA returned by PV_TIME_ST should be mapped by the guest as normal memory +with inner and outer write back caching attributes, in the inner shareable +domain. A total of 16 bytes from the IPA returned are guaranteed to be +meaningfully filled by the hypervisor (see structure below).At what granularity is this allowed to share IPA space with other mappings? The spec doesn't provide any guidance here, and I strongly suspect that it should.
To support a 64K guest, we must ensure that this doesn't share a 64K IPA granule with any MMIO, and it probably only makes sense for an instance of this structure to share that granule with another vCPU's structure.
Quoted 8 lines
Quoted 4 lines
We probably _also_ want to ensure that this doesn't share a 64K granule with memory the guest sees as regular system RAM. Otherwise we're liable to force it into having mismatched attributes for any of that RAM it happens to map as part of mapping the
PV_TIME_STstructure.I guess we can say that it's userspace's responsibiltiy to set this up with sufficient alignment, but I do think we want to make a recommendataion here.
Quoted 6 lines
I can add something like this to the kernel's documentation:
It is advisable that one or more 64k pages are set aside for the purpose of these structures and not used for other purposes, this enables the guest to map the region using 64k pages and avoids conflicting attributes with other memory.
Sounds good to me!
Thanks,
Mark.
PATCH v6 2/10 KVM: arm/arm64: Factor out hypercall handling from PSCI code
In reply to Steven PriceFrom: Christoffer Dall <[email protected]>
We currently intertwine the KVM PSCI implementation with the general dispatch of hypercall handling, which makes perfect sense because PSCI is the only category of hypercalls we support.
However, as we are about to support additional hypercalls, factor out this functionality into a separate hypercall handler file.
--- arch/arm/kvm/Makefile | 2 +- arch/arm/kvm/handle_exit.c | 2 +- arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/handle_exit.c | 4 +- include/Kbuild | 2 + include/kvm/arm_hypercalls.h | 43 ++++++++++++++++++ include/kvm/arm_psci.h | 2 +- virt/kvm/arm/hypercalls.c | 59 +++++++++++++++++++++++++ virt/kvm/arm/psci.c | 84 +----------------------------------- 9 files changed, 112 insertions(+), 87 deletions(-) create mode 100644 include/kvm/arm_hypercalls.h create mode 100644 virt/kvm/arm/hypercalls.c
arch/arm/kvm/Makefile +1 -1
@@ -24,7 +24,7 @@ obj-y += kvm-arm.o init.o interrupts.o obj-y += handle_exit.o guest.o emulate.o reset.o obj-y += coproc.o coproc_a15.o coproc_a7.o vgic-v3-coproc.o obj-y += $(KVM)/arm/arm.o $(KVM)/arm/mmu.o $(KVM)/arm/mmio.o -obj-y += $(KVM)/arm/psci.o $(KVM)/arm/perf.o +obj-y += $(KVM)/arm/psci.o $(KVM)/arm/perf.o $(KVM)/arm/hypercalls.o obj-y += $(KVM)/arm/aarch32.o obj-y += $(KVM)/arm/vgic/vgic.o
arch/arm/kvm/handle_exit.c +1 -1
@@ -9,7 +9,7 @@#include <asm/kvm_emulate.h> #include <asm/kvm_coproc.h> #include <asm/kvm_mmu.h> -#include <kvm/arm_psci.h> +#include <kvm/arm_hypercalls.h> #include <trace/events/kvm.h> #include "trace.h"
arch/arm64/kvm/Makefile +1 -0
@@ -13,6 +13,7 @@ obj-$(CONFIG_KVM_ARM_HOST) += hyp/ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arm.o $(KVM)/arm/mmu.o $(KVM)/arm/mmio.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/psci.o $(KVM)/arm/perf.o +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hypercalls.o kvm-$(CONFIG_KVM_ARM_HOST) += inject_fault.o regmap.o va_layout.o kvm-$(CONFIG_KVM_ARM_HOST) += hyp.o hyp-init.o handle_exit.o
arch/arm64/kvm/handle_exit.c +2 -2
@@ -11,8 +11,6 @@#include <linux/kvm.h> #include <linux/kvm_host.h> -#include <kvm/arm_psci.h> -#include <asm/esr.h> #include <asm/exception.h> #include <asm/kvm_asm.h>
@@ -22,6 +20,8 @@#include <asm/debug-monitors.h> #include <asm/traps.h> +#include <kvm/arm_hypercalls.h> +#define CREATE_TRACE_POINTS #include "trace.h"
include/Kbuild +2 -0
@@ -67,6 +67,8 @@ header-test- += keys/big_key-type.h header-test- += keys/request_key_auth-type.h header-test- += keys/trusted.h header-test- += kvm/arm_arch_timer.h +header-test-$(CONFIG_ARM) += kvm/arm_hypercalls.h +header-test-$(CONFIG_ARM64) += kvm/arm_hypercalls.h header-test- += kvm/arm_pmu.h header-test-$(CONFIG_ARM) += kvm/arm_psci.h header-test-$(CONFIG_ARM64) += kvm/arm_psci.h
include/kvm/arm_hypercalls.h new file +43 -0
43 lines not on this page · raw message
include/kvm/arm_psci.h +1 -1
@@ -40,7 +40,7 @@ static inline int kvm_psci_version(struct kvm_vcpu *vcpu, struct kvm *kvm) } -int kvm_hvc_call_handler (struct kvm_vcpu *vcpu); +int kvm_psci_call (struct kvm_vcpu *vcpu);struct kvm_one_reg ;
virt/kvm/arm/hypercalls.c new file +59 -0
59 lines not on this page · raw message
virt/kvm/arm/psci.c +2 -82
108 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v6 3/10 KVM: arm64: Implement PV_TIME_FEATURES call
In reply to Steven PriceThis provides a mechanism for querying which paravirtualized time features are available in this hypervisor.
Also add the header file which defines the ABI for the paravirtualized time features we're about to add.
--- arch/arm/include/asm/kvm_host.h | 6 ++++++ arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/include/asm/pvclock-abi.h | 17 +++++++++++++++++ arch/arm64/kvm/Makefile | 1 + include/linux/arm-smccc.h | 14 ++++++++++++++ virt/kvm/arm/hypercalls.c | 8 +++++++- virt/kvm/arm/pvtime.c | 21 +++++++++++++++++++++ 7 files changed, 68 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/pvclock-abi.h create mode 100644 virt/kvm/arm/pvtime.c
arch/arm/include/asm/kvm_host.h +6 -0
@@ -7,6 +7,7 @@#ifndef __ARM_KVM_HOST_H__#define __ARM_KVM_HOST_H__ +#include <linux/arm-smccc.h> #include <linux/errno.h> #include <linux/types.h> #include <linux/kvm_types.h>
@@ -323,6 +324,11 @@ static inline int kvm_arch_vm_ioctl_check_extension(struct kvm *kvm, long ext)int kvm_perf_init (void );int kvm_perf_teardown (void ); +static inline long kvm_hypercall_pv_features (struct kvm_vcpu *vcpu) +{ +return SMCCC_RET_NOT_SUPPORTED ; +} +void kvm_mmu_wp_memory_region (struct kvm *kvm,int slot);struct kvm_vcpu *kvm_mpidr_to_vcpu (struct kvm *kvm,unsigned long mpidr);
arch/arm64/include/asm/kvm_host.h +2 -0
@@ -478,6 +478,8 @@ void handle_exit_early(struct kvm_vcpu *vcpu, struct kvm_run *run,int kvm_perf_init (void );int kvm_perf_teardown (void ); +long kvm_hypercall_pv_features (struct kvm_vcpu *vcpu); +void kvm_set_sei_esr (struct kvm_vcpu *vcpu,u64 syndrome);struct kvm_vcpu *kvm_mpidr_to_vcpu (struct kvm *kvm,unsigned long mpidr);
arch/arm64/include/asm/pvclock-abi.h new file +17 -0
@@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (C) 2019 Arm Ltd. */ + +#ifndef __ASM_PVCLOCK_ABI_H +#define __ASM_PVCLOCK_ABI_H + +/* The below structure is defined in ARM DEN0057A */ + +struct pvclock_vcpu_stolen_time { +__le32 revision; +__le32 attributes; +__le64 stolen_time; +/* Structure must be 64 byte aligned, pad to that size */ +u8 padding[48 ]; +} __packed; + +#endif
arch/arm64/kvm/Makefile +1 -0
@@ -14,6 +14,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/e kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arm.o $(KVM)/arm/mmu.o $(KVM)/arm/mmio.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/psci.o $(KVM)/arm/perf.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hypercalls.o +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/pvtime.o kvm-$(CONFIG_KVM_ARM_HOST) += inject_fault.o regmap.o va_layout.o kvm-$(CONFIG_KVM_ARM_HOST) += hyp.o hyp-init.o handle_exit.o
include/linux/arm-smccc.h +14 -0
@@ -45,6 +45,7 @@#define ARM_SMCCC_OWNER_SIP 2#define ARM_SMCCC_OWNER_OEM 3#define ARM_SMCCC_OWNER_STANDARD 4 +#define ARM_SMCCC_OWNER_STANDARD_HYP 5#define ARM_SMCCC_OWNER_TRUSTED_APP 48#define ARM_SMCCC_OWNER_TRUSTED_APP_END 49#define ARM_SMCCC_OWNER_TRUSTED_OS 50
@@ -302,5 +303,18 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,#define SMCCC_RET_NOT_SUPPORTED -1#define SMCCC_RET_NOT_REQUIRED -2 +/* Paravirtualised time calls (defined by ARM DEN0057A) */ +#define ARM_SMCCC_HV_PV_TIME_FEATURES \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_64, \ + ARM_SMCCC_OWNER_STANDARD_HYP, \ + 0x20) + +#define ARM_SMCCC_HV_PV_TIME_ST \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_64, \ + ARM_SMCCC_OWNER_STANDARD_HYP, \ + 0x21) +#endif /*__ASSEMBLY__*/ #endif /*__LINUX_ARM_SMCCC_H*/
virt/kvm/arm/hypercalls.c +7 -1
@@ -12,7 +12,7 @@int kvm_hvc_call_handler (struct kvm_vcpu *vcpu) {u32 func_id =smccc_get_function (vcpu); -u32 val =SMCCC_RET_NOT_SUPPORTED ; +long val =SMCCC_RET_NOT_SUPPORTED ;u32 feature;switch (func_id) {
@@ -48,8 +48,14 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)break ; }break ; +case ARM_SMCCC_HV_PV_TIME_FEATURES : + val =SMCCC_RET_SUCCESS ; +break ; }break ; +case ARM_SMCCC_HV_PV_TIME_FEATURES : + val =kvm_hypercall_pv_features (vcpu); +break ;default :return kvm_psci_call (vcpu); }
virt/kvm/arm/pvtime.c new file +21 -0
@@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2019 Arm Ltd. + +#include <linux/arm-smccc.h> + +#include <kvm/arm_hypercalls.h> + +long kvm_hypercall_pv_features (struct kvm_vcpu *vcpu) +{ +u32 feature =smccc_get_arg1 (vcpu); +long val =SMCCC_RET_NOT_SUPPORTED ; + +switch (feature) { +case ARM_SMCCC_HV_PV_TIME_FEATURES : + val =SMCCC_RET_SUCCESS ; +break ; + } + +return val; +} +
Signature · 1 line
2.20.1
PATCH v6 4/10 KVM: Implement kvm_put_guest()
In reply to Steven Pricekvm_put_guest() is analogous to put_user() - it writes a single value to the guest physical address. The implementation is built upon put_user() and so it has the same single copy atomic properties.
--- include/linux/kvm_host.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
include/linux/kvm_host.h +22 -0
@@ -746,6 +746,28 @@ int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc, unsigned long len);int kvm_gfn_to_hva_cache_init (struct kvm *kvm,struct gfn_to_hva_cache *ghc,gpa_t gpa,unsigned long len); + +#define __kvm_put_guest (kvm, gfn, offset, value, type) \ +({ \ + unsigned long __addr = gfn_to_hva(kvm, gfn); \ + type __user *__uaddr = (type __user *)(__addr + offset); \ + int __ret = -EFAULT; \ + \ + if (!kvm_is_error_hva(__addr)) \ + __ret = put_user(value, __uaddr); \ + if (!__ret) \ + mark_page_dirty(kvm, gfn); \ + __ret; \ +}) + +#define kvm_put_guest (kvm, gpa, value, type) \ +({ \ + gpa_t __gpa = gpa; \ + struct kvm *__kvm = kvm; \ + __kvm_put_guest(__kvm, __gpa >> PAGE_SHIFT, \ + offset_in_page(__gpa), (value), type); \ +}) +int kvm_clear_guest_page (struct kvm *kvm,gfn_t gfn,int offset,int len);int kvm_clear_guest (struct kvm *kvm,gpa_t gpa,unsigned long len);struct kvm_memory_slot *gfn_to_memslot (struct kvm *kvm,gfn_t gfn);
Signature · 6 lines
2.20.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v6 5/10 KVM: arm64: Support stolen time reporting via shared structure
In reply to Steven PriceImplement the service call for configuring a shared structure between a VCPU and the hypervisor in which the hypervisor can write the time stolen from the VCPU's execution time by other tasks on the host.
User space allocates memory which is placed at an IPA also chosen by user space. The hypervisor then updates the shared structure using kvm_put_guest() to ensure single copy atomicity of the 64-bit value reporting the stolen time in nanoseconds.
Whenever stolen time is enabled by the guest, the stolen time counter is reset.
The stolen time itself is retrieved from the sched_info structure maintained by the Linux scheduler code. We enable SCHEDSTATS when selecting KVM Kconfig to ensure this value is meaningful.
--- arch/arm/include/asm/kvm_host.h | 20 +++++++++++ arch/arm64/include/asm/kvm_host.h | 21 +++++++++++- arch/arm64/kvm/Kconfig | 1 + include/linux/kvm_types.h | 2 ++ virt/kvm/arm/arm.c | 11 ++++++ virt/kvm/arm/hypercalls.c | 3 ++ virt/kvm/arm/pvtime.c | 56 +++++++++++++++++++++++++++++++ 7 files changed, 113 insertions(+), 1 deletion(-)
arch/arm/include/asm/kvm_host.h +20 -0
@@ -39,6 +39,7 @@ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1)#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)DECLARE_STATIC_KEY_FALSE (userspace_irqchip_in_use);
@@ -329,6 +330,25 @@ static inline long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu)return SMCCC_RET_NOT_SUPPORTED ; } +static inline long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu) +{ +return SMCCC_RET_NOT_SUPPORTED ; +} + +static inline int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init) +{ +return -ENOTSUPP ; +} + +static inline void kvm_arm_pvtime_vcpu_init (struct kvm_vcpu_arch *vcpu_arch) +{ +} + +static inline bool kvm_arm_is_pvtime_enabled (struct kvm_vcpu_arch *vcpu_arch) +{ +return false; +} +void kvm_mmu_wp_memory_region (struct kvm *kvm,int slot);struct kvm_vcpu *kvm_mpidr_to_vcpu (struct kvm *kvm,unsigned long mpidr);
arch/arm64/include/asm/kvm_host.h +20 -1
@@ -44,6 +44,7 @@ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1)#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)DECLARE_STATIC_KEY_FALSE (userspace_irqchip_in_use);
@@ -338,8 +339,14 @@ struct kvm_vcpu_arch {/* True when deferrable sysregs are loaded on the physical CPU, * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */ bool sysregs_loaded_on_cpu; -}; +/* Guest PV state */ +struct { +u64 steal; +u64 last_steal; +gpa_t base; + } steal; +};/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ #define vcpu_sve_pffr (vcpu) ((void *)((char *)((vcpu)->arch.sve_state) + \ sve_ffr_offset((vcpu)->arch.sve_max_vl)))
@@ -479,6 +486,18 @@ int kvm_perf_init(void);int kvm_perf_teardown (void );long kvm_hypercall_pv_features (struct kvm_vcpu *vcpu); +long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu); +int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init); + +static inline void kvm_arm_pvtime_vcpu_init (struct kvm_vcpu_arch *vcpu_arch) +{ + vcpu_arch->steal.base =GPA_INVALID ; +} + +static inline bool kvm_arm_is_pvtime_enabled (struct kvm_vcpu_arch *vcpu_arch) +{ +return (vcpu_arch->steal.base !=GPA_INVALID ); +}void kvm_set_sei_esr (struct kvm_vcpu *vcpu,u64 syndrome);
arch/arm64/kvm/Kconfig +1 -0
@@ -39,6 +39,7 @@ config KVM select IRQ_BYPASS_MANAGER select HAVE_KVM_IRQ_BYPASS select HAVE_KVM_VCPU_RUN_PID_CHANGE + select SCHEDSTATS ---help--- Support hosting virtualized guest machines. We don't support KVM with 16K page tables yet, due to the multiple
include/linux/kvm_types.h +2 -0
@@ -35,6 +35,8 @@ typedef unsigned long gva_t;typedef u64 gpa_t ;typedef u64 gfn_t ; +#define GPA_INVALID (~(gpa_t)0) +typedef unsigned long hva_t ;typedef u64 hpa_t ;typedef u64 hfn_t ;
virt/kvm/arm/arm.c +11 -0
@@ -40,6 +40,10 @@#include <asm/kvm_coproc.h> #include <asm/sections.h> +#include <kvm/arm_hypercalls.h> +#include <kvm/arm_pmu.h> +#include <kvm/arm_psci.h> +#ifdef REQUIRES_VIRT __asm__(".arch_extension virt" );#endif
@@ -351,6 +355,8 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)kvm_arm_reset_debug_ptr (vcpu); +kvm_arm_pvtime_vcpu_init (&vcpu->arch); +return kvm_vgic_vcpu_init (vcpu); }
@@ -380,6 +386,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)kvm_vcpu_load_sysregs (vcpu);kvm_arch_vcpu_load_fp (vcpu);kvm_vcpu_pmu_restore_guest (vcpu); +if (kvm_arm_is_pvtime_enabled (&vcpu->arch)) +kvm_make_request (KVM_REQ_RECORD_STEAL , vcpu);if (single_task_running ())vcpu_clear_wfe_traps (vcpu);
@@ -645,6 +653,9 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)* that a VCPU sees new virtual interrupts. */ kvm_check_request (KVM_REQ_IRQ_PENDING , vcpu); + +if (kvm_check_request (KVM_REQ_RECORD_STEAL , vcpu)) +kvm_update_stolen_time (vcpu, false); } }
virt/kvm/arm/hypercalls.c +3 -0
@@ -56,6 +56,9 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case ARM_SMCCC_HV_PV_TIME_FEATURES : val =kvm_hypercall_pv_features (vcpu);break ; +case ARM_SMCCC_HV_PV_TIME_ST : + val =kvm_hypercall_stolen_time (vcpu); +break ;default :return kvm_psci_call (vcpu); }
virt/kvm/arm/pvtime.c +56 -0
73 lines not on this page · raw message
Signature · 1 line
2.20.1
Marc Zyngier
maintainerRe: [PATCH v6 05/10] KVM: arm64: Support stolen time reporting via shared structure
Quoted 81 lines · Steven Price
Implement the service call for configuring a shared structure between a VCPU and the hypervisor in which the hypervisor can write the time stolen from the VCPU's execution time by other tasks on the host.
User space allocates memory which is placed at an IPA also chosen by user space. The hypervisor then updates the shared structure using
kvm_put_guest()to ensure single copy atomicity of the 64-bit value reporting the stolen time in nanoseconds.Whenever stolen time is enabled by the guest, the stolen time counter is reset.
The stolen time itself is retrieved from the
sched_infostructure maintained by the Linux scheduler code. We enable SCHEDSTATS when selecting KVMKconfigto ensure this value is meaningful.Signed-off-by Steven Price <[email protected]>--- arch/arm/include/asm/kvm_host.h | 20 +++++++++++ arch/arm64/include/asm/kvm_host.h | 21 +++++++++++- arch/arm64/kvm/Kconfig | 1 + include/linux/kvm_types.h | 2 ++ virt/kvm/arm/arm.c | 11 ++++++ virt/kvm/arm/hypercalls.c | 3 ++ virt/kvm/arm/pvtime.c | 56 +++++++++++++++++++++++++++++++ 7 files changed, 113 insertions(+), 1 deletion(-)arch/arm/include/asm/kvm_host.h +20
-0@@ -39,6 +39,7 @@ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1)#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)DECLARE_STATIC_KEY_FALSE (userspace_irqchip_in_use);@@ -329,6 +330,25 @@ static inline long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu)return SMCCC_RET_NOT_SUPPORTED ; } +static inline long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu) +{ +return SMCCC_RET_NOT_SUPPORTED ; +} + +static inline int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init) +{ +return -ENOTSUPP ; +} + +static inline void kvm_arm_pvtime_vcpu_init (struct kvm_vcpu_arch *vcpu_arch) +{ +} + +static inline bool kvm_arm_is_pvtime_enabled (struct kvm_vcpu_arch *vcpu_arch) +{ +return false; +} +void kvm_mmu_wp_memory_region (struct kvm *kvm,int slot);struct kvm_vcpu *kvm_mpidr_to_vcpu (struct kvm *kvm,unsigned long mpidr);arch/arm64/include/asm/kvm_host.h +1
-0@@ -44,6 +44,7 @@ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1)#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)DECLARE_STATIC_KEY_FALSE (userspace_irqchip_in_use);@@ -338,8 +339,14 @@ struct kvm_vcpu_arch {/* True when deferrable sysregs are loaded on the physical CPU, * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 349
Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +7
-1bool sysregs_loaded_on_cpu; -}; +/* Guest PV state */ +struct { +u64 steal; +u64 last_steal; +gpa_t base; + } steal; +};
nit: Please keep an empty line at the end of the structure.
Quoted 111 lines
arch/arm64/include/asm/kvm_host.h +12
-0/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ #define vcpu_sve_pffr (vcpu) ((void *)((char *)((vcpu)->arch.sve_state) + \ sve_ffr_offset((vcpu)->arch.sve_max_vl)))@@ -479,6 +486,18 @@ int kvm_perf_init(void);int kvm_perf_teardown (void );long kvm_hypercall_pv_features (struct kvm_vcpu *vcpu); +long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu); +int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init); + +static inline void kvm_arm_pvtime_vcpu_init (struct kvm_vcpu_arch *vcpu_arch) +{ + vcpu_arch->steal.base =GPA_INVALID ; +} + +static inline bool kvm_arm_is_pvtime_enabled (struct kvm_vcpu_arch *vcpu_arch) +{ +return (vcpu_arch->steal.base !=GPA_INVALID ); +}void kvm_set_sei_esr (struct kvm_vcpu *vcpu,u64 syndrome);arch/arm64/kvm/Kconfig +1
-0@@ -39,6 +39,7 @@ config KVM select IRQ_BYPASS_MANAGER select HAVE_KVM_IRQ_BYPASS select HAVE_KVM_VCPU_RUN_PID_CHANGE + select SCHEDSTATS ---help--- Support hosting virtualized guest machines. We don't support KVM with 16K page tables yet, due to the multipleinclude/linux/kvm_types.h +2
-0@@ -35,6 +35,8 @@ typedef unsigned long gva_t;typedef u64 gpa_t ;typedef u64 gfn_t ; +#define GPA_INVALID (~(gpa_t)0) +typedef unsigned long hva_t ;typedef u64 hpa_t ;typedef u64 hfn_t ;virt/kvm/arm/arm.c +11
-0@@ -40,6 +40,10 @@#include <asm/kvm_coproc.h> #include <asm/sections.h> +#include <kvm/arm_hypercalls.h> +#include <kvm/arm_pmu.h> +#include <kvm/arm_psci.h> +#ifdef REQUIRES_VIRT __asm__(".arch_extension virt" );#endif @@ -351,6 +355,8 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)kvm_arm_reset_debug_ptr (vcpu); +kvm_arm_pvtime_vcpu_init (&vcpu->arch); +return kvm_vgic_vcpu_init (vcpu); }@@ -380,6 +386,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)kvm_vcpu_load_sysregs (vcpu);kvm_arch_vcpu_load_fp (vcpu);kvm_vcpu_pmu_restore_guest (vcpu); +if (kvm_arm_is_pvtime_enabled (&vcpu->arch)) +kvm_make_request (KVM_REQ_RECORD_STEAL , vcpu);if (single_task_running ())vcpu_clear_wfe_traps (vcpu);@@ -645,6 +653,9 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)* that a VCPU sees new virtual interrupts. */ kvm_check_request (KVM_REQ_IRQ_PENDING , vcpu); + +if (kvm_check_request (KVM_REQ_RECORD_STEAL , vcpu)) +kvm_update_stolen_time (vcpu, false); } }virt/kvm/arm/hypercalls.c +3
-0@@ -56,6 +56,9 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case ARM_SMCCC_HV_PV_TIME_FEATURES : val =kvm_hypercall_pv_features (vcpu);break ; +case ARM_SMCCC_HV_PV_TIME_ST : + val =kvm_hypercall_stolen_time (vcpu); +break ;default :return kvm_psci_call (vcpu); }virt/kvm/arm/pvtime.c +0
-0@@ -3,8 +3,45 @@#include <linux/arm-smccc.h>
virt/kvm/arm/pvtime.c line 14
Quoted 10 lines
virt/kvm/arm/pvtime.c +7
-0+#include <asm/pvclock-abi.h> +#include <kvm/arm_hypercalls.h> +int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init) +{ +struct kvm *kvm = vcpu->kvm; +u64 steal; +u64 steal_le;
This should be __le64.
Quoted 9 lines
virt/kvm/arm/pvtime.c +9
-0+u64 offset; +int idx; +u64 base = vcpu->arch.steal.base; + +if (base ==GPA_INVALID ) +return -ENOTSUPP ; + +/* Let's do the local bookkeeping */ + steal = vcpu->arch.steal.steal;
virt/kvm/arm/pvtime.c line 33
Quoted 10 lines
virt/kvm/arm/pvtime.c +10
-0+ steal += current->sched_info.run_delay - vcpu->arch.steal.last_steal; + vcpu->arch.steal.last_steal = current->sched_info.run_delay; + vcpu->arch.steal.steal = steal; + + steal_le =cpu_to_le64 (steal); + idx =srcu_read_lock (&kvm->srcu); +if (init) { +struct pvclock_vcpu_stolen_time init_values = { + .revision =0 , + .attributes =0
nit: 0 is the default initialiser.
virt/kvm/arm/pvtime.c line 37
Quoted 4 lines
virt/kvm/arm/pvtime.c +4
-0+ }; +kvm_write_guest (kvm, base, &init_values, +sizeof (init_values)); + }
I'm not convinced by this init phase right in the middle of the normal path. It looks ugly, and it'd be better if moved out of line. I'd suggest:
static void kvm_init_stolen_time(struct kvm_vcpu *vcpu)
{
struct pvclock_vcpu_stolen_time init_values = { };
vcpu->arch.steal.steal = 0;
vcpu->arch.steal.last_steal = current->sched_info.run_delay;
idx = srcu_read_lock(&kvm->srcu);
kvm_write_guest(kvm, base, &init_values, sizeof(init_values));
srcu_read_unlock(&kvm->srcu, idx);
}and change the two callers accordingly. Or even better, move this code to the hypercall handling function, because that's where it actually belongs.
Quoted 19 lines
virt/kvm/arm/pvtime.c +8
-0+ offset = offsetof(struct pvclock_vcpu_stolen_time , stolen_time); +kvm_put_guest (kvm, base + offset, steal_le, u64); +srcu_read_unlock (&kvm->srcu, idx); + +return 0 ; +} +long kvm_hypercall_pv_features (struct kvm_vcpu *vcpu) {u32 feature =smccc_get_arg1 (vcpu);@@ -12,6 +49,7 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu)switch (feature) {case ARM_SMCCC_HV_PV_TIME_FEATURES : +case ARM_SMCCC_HV_PV_TIME_ST : val =SMCCC_RET_SUCCESS ;break ; }
virt/kvm/arm/pvtime.c long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu) · line 60
Quoted 4 lines
virt/kvm/arm/pvtime.c +1
-0@@ -19,3 +57,21 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu)return val; } +long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu)
Why long? If that's a base address, then it is either a phys_addr_t or a gpa_t. I'd suggest you move the error check to the caller.
Quoted 21 lines
virt/kvm/arm/pvtime.c +17
-0+{ +int err; + +/* +* Start counting stolen time from the time the guest requests +* the feature enabled. +*/ + vcpu->arch.steal.steal =0 ; + vcpu->arch.steal.last_steal = current->sched_info.run_delay; + + err =kvm_update_stolen_time (vcpu, true); + +if (err) +return SMCCC_RET_NOT_SUPPORTED ; + +return vcpu->arch.steal.base; +}Signature · 1 line
2.20.1
Thanks,
M.
Signature · 5 lines
Jazz is not dead, it just smells funny. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v6 05/10] KVM: arm64: Support stolen time reporting via shared structure
Quoted 263 lines · Marc Zyngier
Quoted 81 lines · Steven Price
Implement the service call for configuring a shared structure between a VCPU and the hypervisor in which the hypervisor can write the time stolen from the VCPU's execution time by other tasks on the host.
User space allocates memory which is placed at an IPA also chosen by user space. The hypervisor then updates the shared structure using
kvm_put_guest()to ensure single copy atomicity of the 64-bit value reporting the stolen time in nanoseconds.Whenever stolen time is enabled by the guest, the stolen time counter is reset.
The stolen time itself is retrieved from the
sched_infostructure maintained by the Linux scheduler code. We enable SCHEDSTATS when selecting KVMKconfigto ensure this value is meaningful.Signed-off-by Steven Price <[email protected]>--- arch/arm/include/asm/kvm_host.h | 20 +++++++++++ arch/arm64/include/asm/kvm_host.h | 21 +++++++++++- arch/arm64/kvm/Kconfig | 1 + include/linux/kvm_types.h | 2 ++ virt/kvm/arm/arm.c | 11 ++++++ virt/kvm/arm/hypercalls.c | 3 ++ virt/kvm/arm/pvtime.c | 56 +++++++++++++++++++++++++++++++ 7 files changed, 113 insertions(+), 1 deletion(-)arch/arm/include/asm/kvm_host.h +20
-0@@ -39,6 +39,7 @@ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1)#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)DECLARE_STATIC_KEY_FALSE (userspace_irqchip_in_use);@@ -329,6 +330,25 @@ static inline long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu)return SMCCC_RET_NOT_SUPPORTED ; } +static inline long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu) +{ +return SMCCC_RET_NOT_SUPPORTED ; +} + +static inline int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init) +{ +return -ENOTSUPP ; +} + +static inline void kvm_arm_pvtime_vcpu_init (struct kvm_vcpu_arch *vcpu_arch) +{ +} + +static inline bool kvm_arm_is_pvtime_enabled (struct kvm_vcpu_arch *vcpu_arch) +{ +return false; +} +void kvm_mmu_wp_memory_region (struct kvm *kvm,int slot);struct kvm_vcpu *kvm_mpidr_to_vcpu (struct kvm *kvm,unsigned long mpidr);arch/arm64/include/asm/kvm_host.h +1
-0@@ -44,6 +44,7 @@ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1)#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)DECLARE_STATIC_KEY_FALSE (userspace_irqchip_in_use);@@ -338,8 +339,14 @@ struct kvm_vcpu_arch {/* True when deferrable sysregs are loaded on the physical CPU, * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */ arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 349
Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +7
-1bool sysregs_loaded_on_cpu; -}; +/* Guest PV state */ +struct { +u64 steal; +u64 last_steal; +gpa_t base; + } steal; +};nit: Please keep an empty line at the end of the structure.
Quoted 111 lines
arch/arm64/include/asm/kvm_host.h +12
-0/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ #define vcpu_sve_pffr (vcpu) ((void *)((char *)((vcpu)->arch.sve_state) + \ sve_ffr_offset((vcpu)->arch.sve_max_vl)))@@ -479,6 +486,18 @@ int kvm_perf_init(void);int kvm_perf_teardown (void );long kvm_hypercall_pv_features (struct kvm_vcpu *vcpu); +long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu); +int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init); + +static inline void kvm_arm_pvtime_vcpu_init (struct kvm_vcpu_arch *vcpu_arch) +{ + vcpu_arch->steal.base =GPA_INVALID ; +} + +static inline bool kvm_arm_is_pvtime_enabled (struct kvm_vcpu_arch *vcpu_arch) +{ +return (vcpu_arch->steal.base !=GPA_INVALID ); +}void kvm_set_sei_esr (struct kvm_vcpu *vcpu,u64 syndrome);arch/arm64/kvm/Kconfig +1
-0@@ -39,6 +39,7 @@ config KVM select IRQ_BYPASS_MANAGER select HAVE_KVM_IRQ_BYPASS select HAVE_KVM_VCPU_RUN_PID_CHANGE + select SCHEDSTATS ---help--- Support hosting virtualized guest machines. We don't support KVM with 16K page tables yet, due to the multipleinclude/linux/kvm_types.h +2
-0@@ -35,6 +35,8 @@ typedef unsigned long gva_t;typedef u64 gpa_t ;typedef u64 gfn_t ; +#define GPA_INVALID (~(gpa_t)0) +typedef unsigned long hva_t ;typedef u64 hpa_t ;typedef u64 hfn_t ;virt/kvm/arm/arm.c +11
-0@@ -40,6 +40,10 @@#include <asm/kvm_coproc.h> #include <asm/sections.h> +#include <kvm/arm_hypercalls.h> +#include <kvm/arm_pmu.h> +#include <kvm/arm_psci.h> +#ifdef REQUIRES_VIRT __asm__(".arch_extension virt" );#endif @@ -351,6 +355,8 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)kvm_arm_reset_debug_ptr (vcpu); +kvm_arm_pvtime_vcpu_init (&vcpu->arch); +return kvm_vgic_vcpu_init (vcpu); }@@ -380,6 +386,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)kvm_vcpu_load_sysregs (vcpu);kvm_arch_vcpu_load_fp (vcpu);kvm_vcpu_pmu_restore_guest (vcpu); +if (kvm_arm_is_pvtime_enabled (&vcpu->arch)) +kvm_make_request (KVM_REQ_RECORD_STEAL , vcpu);if (single_task_running ())vcpu_clear_wfe_traps (vcpu);@@ -645,6 +653,9 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)* that a VCPU sees new virtual interrupts. */ kvm_check_request (KVM_REQ_IRQ_PENDING , vcpu); + +if (kvm_check_request (KVM_REQ_RECORD_STEAL , vcpu)) +kvm_update_stolen_time (vcpu, false); } }virt/kvm/arm/hypercalls.c +3
-0@@ -56,6 +56,9 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case ARM_SMCCC_HV_PV_TIME_FEATURES : val =kvm_hypercall_pv_features (vcpu);break ; +case ARM_SMCCC_HV_PV_TIME_ST : + val =kvm_hypercall_stolen_time (vcpu); +break ;default :return kvm_psci_call (vcpu); }virt/kvm/arm/pvtime.c +0
-0@@ -3,8 +3,45 @@#include <linux/arm-smccc.h> virt/kvm/arm/pvtime.c line 14
Quoted 10 lines
virt/kvm/arm/pvtime.c +7
-0+#include <asm/pvclock-abi.h> +#include <kvm/arm_hypercalls.h> +int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init) +{ +struct kvm *kvm = vcpu->kvm; +u64 steal; +u64 steal_le;This should be
__le64.Quoted 9 lines
virt/kvm/arm/pvtime.c +9
-0+u64 offset; +int idx; +u64 base = vcpu->arch.steal.base; + +if (base ==GPA_INVALID ) +return -ENOTSUPP ; + +/* Let's do the local bookkeeping */ + steal = vcpu->arch.steal.steal;virt/kvm/arm/pvtime.c line 33
Quoted 10 lines
virt/kvm/arm/pvtime.c +10
-0+ steal += current->sched_info.run_delay - vcpu->arch.steal.last_steal; + vcpu->arch.steal.last_steal = current->sched_info.run_delay; + vcpu->arch.steal.steal = steal; + + steal_le =cpu_to_le64 (steal); + idx =srcu_read_lock (&kvm->srcu); +if (init) { +struct pvclock_vcpu_stolen_time init_values = { + .revision =0 , + .attributes =0 nit: 0 is the default initialiser.
virt/kvm/arm/pvtime.c line 37
Quoted 4 lines
virt/kvm/arm/pvtime.c +4
-0+ }; +kvm_write_guest (kvm, base, &init_values, +sizeof (init_values)); + }I'm not convinced by this init phase right in the middle of the normal path. It looks ugly, and it'd be better if moved out of line. I'd suggest:
static void
kvm_init_stolen_time(struct kvm_vcpu *vcpu){ struct pvclock_vcpu_stolen_time init_values = { }; vcpu->arch.steal.steal = 0; vcpu->arch.steal.last_steal = current->sched_info.run_delay; idx = srcu_read_lock(&kvm->srcu); kvm_write_guest(kvm, base, &init_values, sizeof(init_values)); srcu_read_unlock(&kvm->srcu, idx); }
Quoted 3 lines
and change the two callers accordingly. Or even better, move this code to the hypercall handling function, because that's where it actually belongs.
Ok, it does add a little bit of duplicated code. But it also gets rid of the boolean argument that I never liked. As you suggest I might as well move this into kvm_hypercall_stolen_time().
Quoted 22 lines
Quoted 22 lines
virt/kvm/arm/pvtime.c +8
-0+ offset = offsetof(struct pvclock_vcpu_stolen_time , stolen_time); +kvm_put_guest (kvm, base + offset, steal_le, u64); +srcu_read_unlock (&kvm->srcu, idx); + +return 0 ; +} +long kvm_hypercall_pv_features (struct kvm_vcpu *vcpu) {u32 feature =smccc_get_arg1 (vcpu);@@ -12,6 +49,7 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu)switch (feature) {case ARM_SMCCC_HV_PV_TIME_FEATURES : +case ARM_SMCCC_HV_PV_TIME_ST : val =SMCCC_RET_SUCCESS ;break ; }@@ -19,3 +57,21 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu)return val; }
Quoted 4 lines
virt/kvm/arm/pvtime.c long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu) · line 60
Quoted 1 line
virt/kvm/arm/pvtime.c +1
-0+long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu)Why long? If that's a base address, then it is either a
phys_addr_tor agpa_t. I'd suggest you move the error check to the caller.
This is a bit more tricky. It's a long because that's the declared type of the SMCCC return in kvm_hvc_call_handler(). I can't (easily) move the code into kvm_hvc_call_handler() because that is compiled for arm (as well as arm64) and we don't have the definitions for stolen time there. The best option I could come up with is to have a dummy stub for arm and use generic types for this function.
This means we need a type which can contain both a gpa_t and the SMCCC_RET_NOT_SUPPORTED error code.
I'm open to alternative suggestions on how to make this work.
Thanks,
Steve
Quoted 24 lines
Quoted 21 lines
virt/kvm/arm/pvtime.c +17
-0+{ +int err; + +/* +* Start counting stolen time from the time the guest requests +* the feature enabled. +*/ + vcpu->arch.steal.steal =0 ; + vcpu->arch.steal.last_steal = current->sched_info.run_delay; + + err =kvm_update_stolen_time (vcpu, true); + +if (err) +return SMCCC_RET_NOT_SUPPORTED ; + +return vcpu->arch.steal.base; +}Signature · 1 line
2.20.1Thanks,
Quoted 2 lines
M.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Marc Zyngier
maintainerRe: [PATCH v6 05/10] KVM: arm64: Support stolen time reporting via shared structure
Quoted 31 lines · Steven Price
Quoted 30 lines · Marc Zyngier
Quoted 28 lines · Steven Price
Implement the service call for configuring a shared structure between a VCPU and the hypervisor in which the hypervisor can write the time stolen from the VCPU's execution time by other tasks on the host.
User space allocates memory which is placed at an IPA also chosen by user space. The hypervisor then updates the shared structure using
kvm_put_guest()to ensure single copy atomicity of the 64-bit value reporting the stolen time in nanoseconds.Whenever stolen time is enabled by the guest, the stolen time counter is reset.
The stolen time itself is retrieved from the
sched_infostructure maintained by the Linux scheduler code. We enable SCHEDSTATS when selecting KVMKconfigto ensure this value is meaningful.Signed-off-by Steven Price <[email protected]>--- arch/arm/include/asm/kvm_host.h | 20 +++++++++++ arch/arm64/include/asm/kvm_host.h | 21 +++++++++++- arch/arm64/kvm/Kconfig | 1 + include/linux/kvm_types.h | 2 ++ virt/kvm/arm/arm.c | 11 ++++++ virt/kvm/arm/hypercalls.c | 3 ++ virt/kvm/arm/pvtime.c | 56 +++++++++++++++++++++++++++++++ 7 files changed, 113 insertions(+), 1 deletion(-)
[...]
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
+long
kvm_hypercall_stolen_time(struct kvm_vcpu *vcpu)Why long? If that's a base address, then it is either a
phys_addr_tor agpa_t. I'd suggest you move the error check to the caller.This is a bit more tricky. It's a long because that's the declared type of the SMCCC return in
kvm_hvc_call_handler(). I can't (easily) move the code intokvm_hvc_call_handler()because that is compiled for arm (as well as arm64) and we don't have the definitions for stolen time there. The best option I could come up with is to have a dummy stub for arm and use generic types for this function.
Quoted 4 lines
This means we need a type which can contain both a
gpa_tand theSMCCC_RET_NOT_SUPPORTEDerror code.I'm open to alternative suggestions on how to make this work.
My suggestion would be to always return a gpa_t from this function, and change the 32bit stub for kvm_hypercall_stolen_time() to always return GPA_INVALID.
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v6 05/10] KVM: arm64: Support stolen time reporting via shared structure
Quoted 48 lines · Marc Zyngier
Quoted 32 lines · Steven Price
Quoted 31 lines · Marc Zyngier
Quoted 29 lines · Steven Price
Implement the service call for configuring a shared structure between a VCPU and the hypervisor in which the hypervisor can write the time stolen from the VCPU's execution time by other tasks on the host.
User space allocates memory which is placed at an IPA also chosen by user
space. The hypervisor then updates the shared structure usingkvm_put_guest()to ensure single copy atomicity of the 64-bit value reporting the stolen time in nanoseconds.Whenever stolen time is enabled by the guest, the stolen time counter is
reset.The stolen time itself is retrieved from the
sched_infostructure maintained by the Linux scheduler code. We enable SCHEDSTATS when selecting KVMKconfigto ensure this value is meaningful.Signed-off-by Steven Price <[email protected]>---
arch/arm/include/asm/kvm_host.h | 20 +++++++++++arch/arm64/include/asm/kvm_host.h| 21 +++++++++++- arch/arm64/kvm/Kconfig | 1 + include/linux/kvm_types.h | 2 ++ virt/kvm/arm/arm.c | 11 ++++++ virt/kvm/arm/hypercalls.c | 3 ++ virt/kvm/arm/pvtime.c | 56 +++++++++++++++++++++++++++++++7 files changed, 113 insertions(+), 1 deletion(-)[...]
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
+long
kvm_hypercall_stolen_time(struct kvm_vcpu *vcpu)Why long? If that's a base address, then it is either a
phys_addr_tor agpa_t. I'd suggest you move the error check to the caller.This is a bit more tricky. It's a long because that's the declared type of the SMCCC return in
kvm_hvc_call_handler(). I can't (easily) move the code intokvm_hvc_call_handler()because that is compiled for arm (as well as arm64) and we don't have the definitions for stolen time there. The best option I could come up with is to have a dummy stub for arm and use generic types for this function.
Quoted 8 lines
Quoted 4 lines
This means we need a type which can contain both a
gpa_tand theSMCCC_RET_NOT_SUPPORTEDerror code.I'm open to alternative suggestions on how to make this work.
My suggestion would be to always return a
gpa_tfrom this function, and change the 32bit stub forkvm_hypercall_stolen_time()to always returnGPA_INVALID.
Ok, fair enough. Although it ends up with this strange looking fragment in kvm_hvc_call_handler():
case ARM_SMCCC_HV_PV_TIME_ST: gpa = kvm_init_stolen_time(vcpu); if (gpa != GPA_INVALID) val = gpa; break;
But I agree the gpa_t return type is clearer.
Thanks,
Steve
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 6/10 KVM: Allow kvm_device_ops to be const
In reply to Steven PriceCurrently a kvm_device_ops structure cannot be const without triggering compiler warnings. However the structure doesn't need to be written to and, by marking it const, it can be read-only in memory. Add some more const keywords to allow this.
--- include/linux/kvm_host.h | 4 ++-- virt/kvm/kvm_main.c | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-)
include/linux/kvm_host.h +2 -2
@@ -1260,7 +1260,7 @@ extern unsigned int halt_poll_ns_grow_start;extern unsigned int halt_poll_ns_shrink;struct kvm_device { -struct kvm_device_ops *ops; +const struct kvm_device_ops *ops;struct kvm *kvm;void *private;struct list_head vm_node;
@@ -1313,7 +1313,7 @@ struct kvm_device_ops {void kvm_device_get (struct kvm_device *dev);void kvm_device_put (struct kvm_device *dev);struct kvm_device *kvm_device_from_filp (struct file *filp); -int kvm_register_device_ops (struct kvm_device_ops *ops,u32 type); +int kvm_register_device_ops (const struct kvm_device_ops *ops,u32 type);void kvm_unregister_device_ops (u32 type);extern struct kvm_device_ops kvm_mpic_ops;
virt/kvm/kvm_main.c +3 -3
@@ -3045,14 +3045,14 @@ struct kvm_device *kvm_device_from_filp(struct file *filp)return filp->private_data; } -static struct kvm_device_ops *kvm_device_ops_table[KVM_DEV_TYPE_MAX ] = { +static const struct kvm_device_ops *kvm_device_ops_table[KVM_DEV_TYPE_MAX ] = { #ifdef CONFIG_KVM_MPIC [KVM_DEV_TYPE_FSL_MPIC_20 ] = &kvm_mpic_ops, [KVM_DEV_TYPE_FSL_MPIC_42 ] = &kvm_mpic_ops, #endif }; -int kvm_register_device_ops (struct kvm_device_ops *ops,u32 type) +int kvm_register_device_ops (const struct kvm_device_ops *ops,u32 type) {if (type >=ARRAY_SIZE (kvm_device_ops_table))return -ENOSPC ;
@@ -3073,7 +3073,7 @@ void kvm_unregister_device_ops(u32 type)static int kvm_ioctl_create_device (struct kvm *kvm,struct kvm_create_device *cd) { -struct kvm_device_ops *ops =NULL ; +const struct kvm_device_ops *ops =NULL ;struct kvm_device *dev;bool test = cd->flags &KVM_CREATE_DEVICE_TEST ;int type;
Signature · 1 line
2.20.1
PATCH v6 7/10 KVM: arm64: Provide VCPU attributes for stolen time
In reply to Steven PriceAllow user space to inform the KVM host where in the physical memory map the paravirtualized time structures should be located.
User space can set an attribute on the VCPU providing the IPA base address of the stolen time structure for that VCPU. This must be repeated for every VCPU in the VM.
The address is given in terms of the physical address visible to the guest and must be 64 byte aligned. The guest will discover the address via a hypercall.
--- arch/arm64/include/asm/kvm_host.h | 7 +++++ arch/arm64/include/uapi/asm/kvm.h | 2 ++ arch/arm64/kvm/guest.c | 9 ++++++ include/uapi/linux/kvm.h | 2 ++ virt/kvm/arm/pvtime.c | 47 +++++++++++++++++++++++++++++++ 5 files changed, 67 insertions(+)
arch/arm64/include/asm/kvm_host.h +7 -0
@@ -489,6 +489,13 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu);long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu);int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init); +int kvm_arm_pvtime_set_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_pvtime_get_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_pvtime_has_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +static inline void kvm_arm_pvtime_vcpu_init (struct kvm_vcpu_arch *vcpu_arch) { vcpu_arch->steal.base =GPA_INVALID ;
arch/arm64/include/uapi/asm/kvm.h +2 -0
@@ -323,6 +323,8 @@ struct kvm_vcpu_events {#define KVM_ARM_VCPU_TIMER_CTRL 1#define KVM_ARM_VCPU_TIMER_IRQ_VTIMER 0#define KVM_ARM_VCPU_TIMER_IRQ_PTIMER 1 +#define KVM_ARM_VCPU_PVTIME_CTRL 2 +#define KVM_ARM_VCPU_PVTIME_IPA 0/* KVM_IRQ_LINE irq field index values */ #define KVM_ARM_IRQ_VCPU2_SHIFT 28
arch/arm64/kvm/guest.c +9 -0
@@ -858,6 +858,9 @@ int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu,case KVM_ARM_VCPU_TIMER_CTRL : ret =kvm_arm_timer_set_attr (vcpu, attr);break ; +case KVM_ARM_VCPU_PVTIME_CTRL : + ret =kvm_arm_pvtime_set_attr (vcpu, attr); +break ;default : ret = -ENXIO ;break ;
@@ -878,6 +881,9 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,case KVM_ARM_VCPU_TIMER_CTRL : ret =kvm_arm_timer_get_attr (vcpu, attr);break ; +case KVM_ARM_VCPU_PVTIME_CTRL : + ret =kvm_arm_pvtime_get_attr (vcpu, attr); +break ;default : ret = -ENXIO ;break ;
@@ -898,6 +904,9 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,case KVM_ARM_VCPU_TIMER_CTRL : ret =kvm_arm_timer_has_attr (vcpu, attr);break ; +case KVM_ARM_VCPU_PVTIME_CTRL : + ret =kvm_arm_pvtime_has_attr (vcpu, attr); +break ;default : ret = -ENXIO ;break ;
include/uapi/linux/kvm.h +2 -0
@@ -1227,6 +1227,8 @@ enum kvm_device_type {#define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_XIVE,#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE + KVM_DEV_TYPE_ARM_PV_TIME, +#define KVM_DEV_TYPE_ARM_PV_TIME KVM_DEV_TYPE_ARM_PV_TIME KVM_DEV_TYPE_MAX, };
virt/kvm/arm/pvtime.c +47 -0
57 lines not on this page · raw message
Signature · 1 line
2.20.1
Marc Zyngier
maintainerRe: [PATCH v6 07/10] KVM: arm64: Provide VCPU attributes for stolen time
Quoted 124 lines · Steven Price
Allow user space to inform the KVM host where in the physical memory map the paravirtualized time structures should be located.
User space can set an attribute on the VCPU providing the IPA base address of the stolen time structure for that VCPU. This must be repeated for every VCPU in the VM.
The address is given in terms of the physical address visible to the guest and must be 64 byte aligned. The guest will discover the address via a hypercall.
Signed-off-by Steven Price <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 7 +++++ arch/arm64/include/uapi/asm/kvm.h | 2 ++ arch/arm64/kvm/guest.c | 9 ++++++ include/uapi/linux/kvm.h | 2 ++ virt/kvm/arm/pvtime.c | 47 +++++++++++++++++++++++++++++++ 5 files changed, 67 insertions(+)arch/arm64/include/asm/kvm_host.h +7
-0@@ -489,6 +489,13 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu);long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu);int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init); +int kvm_arm_pvtime_set_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_pvtime_get_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_pvtime_has_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +static inline void kvm_arm_pvtime_vcpu_init (struct kvm_vcpu_arch *vcpu_arch) { vcpu_arch->steal.base =GPA_INVALID ;arch/arm64/include/uapi/asm/kvm.h +2
-0@@ -323,6 +323,8 @@ struct kvm_vcpu_events {#define KVM_ARM_VCPU_TIMER_CTRL 1#define KVM_ARM_VCPU_TIMER_IRQ_VTIMER 0#define KVM_ARM_VCPU_TIMER_IRQ_PTIMER 1 +#define KVM_ARM_VCPU_PVTIME_CTRL 2 +#define KVM_ARM_VCPU_PVTIME_IPA 0/* KVM_IRQ_LINE irq field index values */ #define KVM_ARM_IRQ_VCPU2_SHIFT 28arch/arm64/kvm/guest.c +9
-0@@ -858,6 +858,9 @@ int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu,case KVM_ARM_VCPU_TIMER_CTRL : ret =kvm_arm_timer_set_attr (vcpu, attr);break ; +case KVM_ARM_VCPU_PVTIME_CTRL : + ret =kvm_arm_pvtime_set_attr (vcpu, attr); +break ;default : ret = -ENXIO ;break ;@@ -878,6 +881,9 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,case KVM_ARM_VCPU_TIMER_CTRL : ret =kvm_arm_timer_get_attr (vcpu, attr);break ; +case KVM_ARM_VCPU_PVTIME_CTRL : + ret =kvm_arm_pvtime_get_attr (vcpu, attr); +break ;default : ret = -ENXIO ;break ;@@ -898,6 +904,9 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,case KVM_ARM_VCPU_TIMER_CTRL : ret =kvm_arm_timer_has_attr (vcpu, attr);break ; +case KVM_ARM_VCPU_PVTIME_CTRL : + ret =kvm_arm_pvtime_has_attr (vcpu, attr); +break ;default : ret = -ENXIO ;break ;include/uapi/linux/kvm.h +2
-0@@ -1227,6 +1227,8 @@ enum kvm_device_type {#define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_XIVE,#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE + KVM_DEV_TYPE_ARM_PV_TIME, +#define KVM_DEV_TYPE_ARM_PV_TIME KVM_DEV_TYPE_ARM_PV_TIME KVM_DEV_TYPE_MAX, };virt/kvm/arm/pvtime.c +9
-0@@ -2,7 +2,9 @@// Copyright (C) 2019 Arm Ltd. #include <linux/arm-smccc.h> +#include <linux/kvm_host.h> +#include <asm/kvm_mmu.h> #include <asm/pvclock-abi.h> #include <kvm/arm_hypercalls.h> @@ -75,3 +77,48 @@ long kvm_hypercall_stolen_time(struct kvm_vcpu *vcpu)return vcpu->arch.steal.base; } + +int kvm_arm_pvtime_set_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr) +{ +u64 __user *user = (u64 __user *)attr->addr; +u64 ipa; +
virt/kvm/arm/pvtime.c long kvm_hypercall_stolen_time(struct kvm_vcpu *vcpu) · line 96
Quoted 10 lines
virt/kvm/arm/pvtime.c +10
-0+if (attr->attr !=KVM_ARM_VCPU_PVTIME_IPA ) +return -ENXIO ; + +if (get_user (ipa, user)) +return -EFAULT ; +if (!IS_ALIGNED (ipa,64 )) +return -EINVAL ; +if (vcpu->arch.steal.base !=GPA_INVALID ) +return -EEXIST ; + vcpu->arch.steal.base = ipa;
And what if this IPA doesn't point to any memslot? I understand that everything will still work (kvm_put_user()) will handle the mishap, but it makes it hard for userspace to know that something is wrong.
Is there any problem in mandating that the corresponding memslot already has been created, and enforcing this check?
Thanks,
M.
Signature · 5 lines
Jazz is not dead, it just smells funny. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v6 07/10] KVM: arm64: Provide VCPU attributes for stolen time
Quoted 141 lines · Marc Zyngier
Quoted 124 lines · Steven Price
Allow user space to inform the KVM host where in the physical memory map the paravirtualized time structures should be located.
User space can set an attribute on the VCPU providing the IPA base address of the stolen time structure for that VCPU. This must be repeated for every VCPU in the VM.
The address is given in terms of the physical address visible to the guest and must be 64 byte aligned. The guest will discover the address via a hypercall.
Signed-off-by Steven Price <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 7 +++++ arch/arm64/include/uapi/asm/kvm.h | 2 ++ arch/arm64/kvm/guest.c | 9 ++++++ include/uapi/linux/kvm.h | 2 ++ virt/kvm/arm/pvtime.c | 47 +++++++++++++++++++++++++++++++ 5 files changed, 67 insertions(+)arch/arm64/include/asm/kvm_host.h +7
-0@@ -489,6 +489,13 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu);long kvm_hypercall_stolen_time (struct kvm_vcpu *vcpu);int kvm_update_stolen_time (struct kvm_vcpu *vcpu,bool init); +int kvm_arm_pvtime_set_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_pvtime_get_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +int kvm_arm_pvtime_has_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr); +static inline void kvm_arm_pvtime_vcpu_init (struct kvm_vcpu_arch *vcpu_arch) { vcpu_arch->steal.base =GPA_INVALID ;arch/arm64/include/uapi/asm/kvm.h +2
-0@@ -323,6 +323,8 @@ struct kvm_vcpu_events {#define KVM_ARM_VCPU_TIMER_CTRL 1#define KVM_ARM_VCPU_TIMER_IRQ_VTIMER 0#define KVM_ARM_VCPU_TIMER_IRQ_PTIMER 1 +#define KVM_ARM_VCPU_PVTIME_CTRL 2 +#define KVM_ARM_VCPU_PVTIME_IPA 0/* KVM_IRQ_LINE irq field index values */ #define KVM_ARM_IRQ_VCPU2_SHIFT 28arch/arm64/kvm/guest.c +9
-0@@ -858,6 +858,9 @@ int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu,case KVM_ARM_VCPU_TIMER_CTRL : ret =kvm_arm_timer_set_attr (vcpu, attr);break ; +case KVM_ARM_VCPU_PVTIME_CTRL : + ret =kvm_arm_pvtime_set_attr (vcpu, attr); +break ;default : ret = -ENXIO ;break ;@@ -878,6 +881,9 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,case KVM_ARM_VCPU_TIMER_CTRL : ret =kvm_arm_timer_get_attr (vcpu, attr);break ; +case KVM_ARM_VCPU_PVTIME_CTRL : + ret =kvm_arm_pvtime_get_attr (vcpu, attr); +break ;default : ret = -ENXIO ;break ;@@ -898,6 +904,9 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,case KVM_ARM_VCPU_TIMER_CTRL : ret =kvm_arm_timer_has_attr (vcpu, attr);break ; +case KVM_ARM_VCPU_PVTIME_CTRL : + ret =kvm_arm_pvtime_has_attr (vcpu, attr); +break ;default : ret = -ENXIO ;break ;include/uapi/linux/kvm.h +2
-0@@ -1227,6 +1227,8 @@ enum kvm_device_type {#define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_XIVE,#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE + KVM_DEV_TYPE_ARM_PV_TIME, +#define KVM_DEV_TYPE_ARM_PV_TIME KVM_DEV_TYPE_ARM_PV_TIME KVM_DEV_TYPE_MAX, };virt/kvm/arm/pvtime.c +9
-0@@ -2,7 +2,9 @@// Copyright (C) 2019 Arm Ltd. #include <linux/arm-smccc.h> +#include <linux/kvm_host.h> +#include <asm/kvm_mmu.h> #include <asm/pvclock-abi.h> #include <kvm/arm_hypercalls.h> @@ -75,3 +77,48 @@ long kvm_hypercall_stolen_time(struct kvm_vcpu *vcpu)return vcpu->arch.steal.base; } + +int kvm_arm_pvtime_set_attr (struct kvm_vcpu *vcpu, +struct kvm_device_attr *attr) +{ +u64 __user *user = (u64 __user *)attr->addr; +u64 ipa; +virt/kvm/arm/pvtime.c long kvm_hypercall_stolen_time(struct kvm_vcpu *vcpu) · line 96
Quoted 10 lines
virt/kvm/arm/pvtime.c +10
-0+if (attr->attr !=KVM_ARM_VCPU_PVTIME_IPA ) +return -ENXIO ; + +if (get_user (ipa, user)) +return -EFAULT ; +if (!IS_ALIGNED (ipa,64 )) +return -EINVAL ; +if (vcpu->arch.steal.base !=GPA_INVALID ) +return -EEXIST ; + vcpu->arch.steal.base = ipa;And what if this IPA doesn't point to any memslot? I understand that everything will still work (
kvm_put_user()) will handle the mishap, but it makes it hard for userspace to know that something is wrong.
Quoted 2 lines
Is there any problem in mandating that the corresponding memslot already has been created, and enforcing this check?
No that could be done. As you mentioned nothing bad will happen (to the host) if this is wrong, so I didn't see the need to enforce that the memory is setup first. And the check will be pretty weak because nothing stop the memslot vanishing after the check. But I guess this might make it easier to figure out what has gone wrong in user space, and we can always remove this ordering restriction in future if necessary. So I'll add a check for now.
Thanks,
Steve
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v6 8/10 arm/arm64: Provide a wrapper for SMCCC 1.1 calls
In reply to Steven PriceSMCCC 1.1 calls may use either HVC or SMC depending on the PSCI conduit. Rather than coding this in every call site provide a macro which uses the correct instruction. The macro also handles the case where no PSCI conduit is configured returning a not supported error in res, along with returning the conduit used for the call.
This allow us to remove some duplicated code and will be useful later when adding paravirtualized time hypervisor calls.
--- include/linux/arm-smccc.h | 44 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+)
include/linux/arm-smccc.h +44 -0
@@ -303,6 +303,50 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,#define SMCCC_RET_NOT_SUPPORTED -1#define SMCCC_RET_NOT_REQUIRED -2 +/* Like arm_smccc_1_1* but always returns SMCCC_RET_NOT_SUPPORTED. +* Used when the PSCI conduit is not defined. The empty asm statement +* avoids compiler warnings about unused variables. +*/ +#define __fail_smccc_1_1 (...) \ + do { \ + __declare_args(__count_args(__VA_ARGS__), __VA_ARGS__); \ + asm ("" __constraints(__count_args(__VA_ARGS__))); \ + if (___res) \ + ___res->a0 = SMCCC_RET_NOT_SUPPORTED; \ + } while (0) + +/* +* arm_smccc_1_1_invoke() - make an SMCCC v1.1 compliant call +* +* This is a variadic macro taking one to eight source arguments, and +* an optional return structure. +* +* @a0-a7: arguments passed in registers 0 to 7 +* @res: result values from registers 0 to 3 +* +* This macro will make either an HVC call or an SMC call depending on the +* current PSCI conduit. If no valid conduit is available then -1 +* (SMCCC_RET_NOT_SUPPORTED) is returned in @res.a0 (if supplied). +* +* The return value also provides the conduit that was used. +*/ +#define arm_smccc_1_1_invoke (...) ({ \ + int method = psci_ops.conduit; \ + switch (method) { \ + case PSCI_CONDUIT_HVC: \ + arm_smccc_1_1_hvc(__VA_ARGS__); \ + break; \ + case PSCI_CONDUIT_SMC: \ + arm_smccc_1_1_smc(__VA_ARGS__); \ + break; \ + default: \ + __fail_smccc_1_1(__VA_ARGS__); \ + method = PSCI_CONDUIT_NONE; \ + break; \ + } \ + method; \ + }) +/* Paravirtualised time calls (defined by ARM DEN0057A) */ #define ARM_SMCCC_HV_PV_TIME_FEATURES \ ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \
Signature · 6 lines
2.20.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v6 08/10] arm/arm64: Provide a wrapper for SMCCC 1.1 calls
Quoted 27 lines · Steven Price
SMCCC 1.1 calls may use either HVC or SMC depending on the PSCI conduit. Rather than coding this in every call site provide a macro which uses the correct instruction. The macro also handles the case where no PSCI conduit is configured returning a not supported error in res, along with returning the conduit used for the call.
This allow us to remove some duplicated code and will be useful later when adding paravirtualized time hypervisor calls.
Signed-off-by Steven Price <[email protected]>Acked-by Will Deacon <[email protected]>--- include/linux/arm-smccc.h | 44 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+)include/linux/arm-smccc.h +2
-0@@ -303,6 +303,50 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,#define SMCCC_RET_NOT_SUPPORTED -1#define SMCCC_RET_NOT_REQUIRED -2 +/* Like arm_smccc_1_1* but always returns SMCCC_RET_NOT_SUPPORTED. +* Used when the PSCI conduit is not defined. The empty asm
Quoted 2 lines
statement
+ * avoids compiler warnings about unused variables.
nit: comment style.
Quoted 36 lines
+ */
include/linux/arm-smccc.h +12
-0+#define __fail_smccc_1_1 (...) \ + do { \ + __declare_args(__count_args(__VA_ARGS__), __VA_ARGS__); \ + asm ("" __constraints(__count_args(__VA_ARGS__))); \ + if (___res) \ + ___res->a0 = SMCCC_RET_NOT_SUPPORTED; \ + } while (0) + +/* +* arm_smccc_1_1_invoke() - make an SMCCC v1.1 compliant call +* +* This is a variadic macro taking one to eight source arguments, and
+ * an optional return structure.
+ *
+ * @a0-a7: arguments passed in registers 0 to 7
+ * @res: result values from registers 0 to 3
+ *
+ * This macro will make either an HVC call or an SMC call depending on the
+ * current PSCI conduit. If no valid conduit is available then -1
+ * (SMCCC_RET_NOT_SUPPORTED) is returned in @res.a0 (if supplied).
+ *
+ * The return value also provides the conduit that was used.
+ */include/linux/arm-smccc.h +10
-0+#define arm_smccc_1_1_invoke (...) ({ \ + int method = psci_ops.conduit; \ + switch (method) { \ + case PSCI_CONDUIT_HVC: \ + arm_smccc_1_1_hvc(__VA_ARGS__); \ + break; \ + case PSCI_CONDUIT_SMC: \ + arm_smccc_1_1_smc(__VA_ARGS__); \ + break; \ + default: \
include/linux/arm-smccc.h asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1, · line 352
Quoted 10 lines
include/linux/arm-smccc.h +7
-0+__fail_smccc_1_1 (__VA_ARGS__); \ + method =PSCI_CONDUIT_NONE ; \ +break ; \ + } \ + method; \ + }) +/* Paravirtualised time calls (defined by ARM DEN0057A) */ #define ARM_SMCCC_HV_PV_TIME_FEATURES \ ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \
All this should most probably go on top of the SMCCC conduit cleanup that
has already been already queued in the arm64 tree (see arm64/for-next/smccc-conduit-cleanup).
Thanks,
M.
Signature · 6 lines
Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 08/10] arm/arm64: Provide a wrapper for SMCCC 1.1 calls
Quoted 1 line · Marc Zyngier
On 2019-10-11 13:59, Steven Price wrote:
[...]
Quoted 3 lines
All this should most probably go on top of the SMCCC conduit cleanup that has already been already queued in the arm64 tree (see arm64/for-next/smccc-conduit-cleanup).
Good point, I'll rebase. Are you happy for the entire series to be based on top of that? i.e. based on commit e6ea46511b1a ("firmware: arm_sdei:
use common SMCCC_CONDUIT_*")
Thanks,
Steve
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Marc Zyngier
maintainerRe: [PATCH v6 08/10] arm/arm64: Provide a wrapper for SMCCC 1.1 calls
Quoted 10 lines · Steven Price
Quoted 1 line · Marc Zyngier
On 2019-10-11 13:59, Steven Price wrote:
[...]
Quoted 3 lines
All this should most probably go on top of the SMCCC conduit cleanup that has already been already queued in the arm64 tree (see arm64/for-next/smccc-conduit-cleanup).
Good point, I'll rebase. Are you happy for the entire series to be based on top of that? i.e. based on commit
e6ea46511b1a("firmware:arm_sdei:
use common SMCCC_CONDUIT_*")
Absolutely. I'll sync with Will and Catalin to get a stable branch that includes these commits.
Thanks,
M.
Signature · 6 lines
Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 9/10 arm/arm64: Make use of the SMCCC 1.1 wrapper
In reply to Steven PriceRather than directly choosing which function to use based on psci_ops.conduit, use the new arm_smccc_1_1 wrapper instead.
In some cases we still need to do some operations based on the conduit, but the code duplication is removed.
No functional change.
--- arch/arm/mm/proc-v7-bugs.c | 13 +++--- arch/arm64/kernel/cpu_errata.c | 80 ++++++++++++---------------------- 2 files changed, 33 insertions(+), 60 deletions(-)
arch/arm/mm/proc-v7-bugs.c +5 -8
@@ -78,12 +78,13 @@ static void cpu_v7_spectre_init(void)if (psci_ops.smccc_version ==SMCCC_VERSION_1_0 )break ; +arm_smccc_1_1_invoke (ARM_SMCCC_ARCH_FEATURES_FUNC_ID , +ARM_SMCCC_ARCH_WORKAROUND_1 , &res); +if ((int )res.a0 !=0 ) +return ; +switch (psci_ops.conduit) {case PSCI_CONDUIT_HVC : -arm_smccc_1_1_hvc (ARM_SMCCC_ARCH_FEATURES_FUNC_ID , -ARM_SMCCC_ARCH_WORKAROUND_1 , &res); -if ((int )res.a0 !=0 ) -break ;per_cpu (harden_branch_predictor_fn, cpu) = call_hvc_arch_workaround_1; cpu_do_switch_mm = cpu_v7_hvc_switch_mm;
@@ -91,10 +92,6 @@ static void cpu_v7_spectre_init(void)break ;case PSCI_CONDUIT_SMC : -arm_smccc_1_1_smc (ARM_SMCCC_ARCH_FEATURES_FUNC_ID , -ARM_SMCCC_ARCH_WORKAROUND_1 , &res); -if ((int )res.a0 !=0 ) -break ;per_cpu (harden_branch_predictor_fn, cpu) = call_smc_arch_workaround_1; cpu_do_switch_mm = cpu_v7_smc_switch_mm;
arch/arm64/kernel/cpu_errata.c +28 -52
117 lines not on this page · raw message
Signature · 6 lines
2.20.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v6 10/10 arm64: Retrieve stolen time as paravirtualized guest
In reply to Steven PriceEnable paravirtualization features when running under a hypervisor supporting the PV_TIME_ST hypercall.
For each (v)CPU, we ask the hypervisor for the location of a shared page which the hypervisor will use to report stolen time to us. We set pv_time_ops to the stolen time function which simply reads the stolen value from the shared page for a VCPU. We guarantee single-copy atomicity using READ_ONCE which means we can also read the stolen time for another VCPU than the currently running one while it is potentially being updated by the hypervisor.
--- .../admin-guide/kernel-parameters.txt | 6 +- arch/arm64/include/asm/paravirt.h | 9 +- arch/arm64/kernel/paravirt.c | 148 ++++++++++++++++++ arch/arm64/kernel/time.c | 3 + include/linux/cpuhotplug.h | 1 + 5 files changed, 163 insertions(+), 4 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -3
@@ -3083,9 +3083,9 @@ [X86,PV_OPS] Disable paravirtualized VMware scheduler clock and use the default one. - no-steal-acc [X86,KVM] Disable paravirtualized steal time accounting. - steal time is computed, but won't influence scheduler - behaviour + no-steal-acc [X86,KVM,ARM64] Disable paravirtualized steal time + accounting. steal time is computed, but won't + influence scheduler behaviour nolapic [X86-32,APIC] Do not enable or use the local APIC.
arch/arm64/include/asm/paravirt.h +8 -1
@@ -21,6 +21,13 @@ static inline u64 paravirt_steal_clock(int cpu) {return pv_ops.time.steal_clock (cpu); } -#endif + +int __init kvm_guest_init (void ); + +#else + +#define kvm_guest_init () + +#endif // CONFIG_PARAVIRT #endif
arch/arm64/kernel/paravirt.c +148 -0
161 lines not on this page · raw message
arch/arm64/kernel/time.c +3 -0
@@ -30,6 +30,7 @@#include <asm/thread_info.h> #include <asm/stacktrace.h> +#include <asm/paravirt.h> unsigned long profile_pc (struct pt_regs *regs) {
@@ -65,4 +66,6 @@ void __init time_init(void)/* Calibrate the delay loop directly */ lpj_fine = arch_timer_rate /HZ ; + +kvm_guest_init (); }
include/linux/cpuhotplug.h +1 -0
@@ -136,6 +136,7 @@ enum cpuhp_state {/* Must be the last timer callback */ CPUHP_AP_DUMMY_TIMER_STARTING ,CPUHP_AP_ARM_XEN_STARTING , +CPUHP_AP_ARM_KVMPV_STARTING ,CPUHP_AP_ARM_CORESIGHT_STARTING ,CPUHP_AP_ARM64_ISNDEP_STARTING , CPUHP_AP_SMPCFD_DYING,
Signature · 1 line
2.20.1
Marc Zyngier
maintainerRe: [PATCH v6 10/10] arm64: Retrieve stolen time as paravirtualized guest
Quoted 95 lines · Steven Price
Enable paravirtualization features when running under a hypervisor supporting the
PV_TIME_SThypercall.For each (v)CPU, we ask the hypervisor for the location of a shared page which the hypervisor will use to report stolen time to us. We set
pv_time_opsto the stolen time function which simply reads the stolen value from the shared page for a VCPU. We guarantee single-copy atomicity usingREAD_ONCEwhich means we can also read the stolen time for another VCPU than the currently running one while it is potentially being updated by the hypervisor.Signed-off-by Steven Price <[email protected]>--- .../admin-guide/kernel-parameters.txt | 6 +- arch/arm64/include/asm/paravirt.h | 9 +- arch/arm64/kernel/paravirt.c | 148 ++++++++++++++++++ arch/arm64/kernel/time.c | 3 + include/linux/cpuhotplug.h | 1 + 5 files changed, 163 insertions(+), 4 deletions(-)Documentation/admin-guide/kernel-parameters.txt +3
-3@@ -3083,9 +3083,9 @@ [X86,PV_OPS] Disable paravirtualized VMware scheduler clock and use the default one. - no-steal-acc [X86,KVM] Disable paravirtualized steal time accounting. - steal time is computed, but won't influence scheduler - behaviour + no-steal-acc [X86,KVM,ARM64] Disable paravirtualized steal time + accounting. steal time is computed, but won't + influence scheduler behaviour nolapic [X86-32,APIC] Do not enable or use the local APIC.arch/arm64/include/asm/paravirt.h +8
-1@@ -21,6 +21,13 @@ static inline u64 paravirt_steal_clock(int cpu) {return pv_ops.time.steal_clock (cpu); } -#endif + +int __init kvm_guest_init (void ); + +#else + +#define kvm_guest_init () + +#endif // CONFIG_PARAVIRT #endif arch/arm64/kernel/paravirt.c +20
-0@@ -6,13 +6,161 @@* Author: Stefano Stabellini <[email protected]> */ +#define pr_fmt (fmt) "kvmarm-pv: " fmt + +#include <linux/arm-smccc.h> +#include <linux/cpuhotplug.h> #include <linux/export.h> +#include <linux/io.h> #include <linux/jump_label.h> +#include <linux/printk.h> +#include <linux/psci.h> +#include <linux/reboot.h> +#include <linux/slab.h> #include <linux/types.h> +#include <asm/paravirt.h> +#include <asm/pvclock-abi.h> +#include <asm/smp_plat.h> struct static_key paravirt_steal_enabled;struct static_key paravirt_steal_rq_enabled;struct paravirt_patch_template pv_ops;EXPORT_SYMBOL_GPL (pv_ops); + +struct kvmarm_stolen_time_region { +struct pvclock_vcpu_stolen_time *kaddr; +}; + +static DEFINE_PER_CPU(struct kvmarm_stolen_time_region, stolen_time_region); + +static bool steal_acc = true;
arch/arm64/kernel/paravirt.c line 48
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+static int __init parse_no_stealacc (char *arg) +{ + steal_acc = false; +return 0 ; +} + +early_param ("no-steal-acc" , parse_no_stealacc); + +/* return stolen time in ns by asking the hypervisor */ +static u64 kvm_steal_clock (int cpu)
This isn't KVM specific.
Quoted 21 lines
arch/arm64/kernel/paravirt.c +21
-0+{ +struct kvmarm_stolen_time_region *reg; + + reg =per_cpu_ptr (&stolen_time_region, cpu); +if (!reg->kaddr) { +pr_warn_once ("stolen time enabled but not configured for cpu %d\n" , + cpu); +return 0 ; + } + +return le64_to_cpu (READ_ONCE (reg->kaddr->stolen_time)); +} + +static int disable_stolen_time_current_cpu (void ) +{ +struct kvmarm_stolen_time_region *reg; + + reg =this_cpu_ptr (&stolen_time_region); +if (!reg->kaddr) +return 0 ; +
arch/arm64/kernel/paravirt.c line 79
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+memunmap (reg->kaddr); +memset (reg,0 ,sizeof (*reg)); + +return 0 ; +} + +static int stolen_time_dying_cpu (unsigned int cpu) +{ +return disable_stolen_time_current_cpu (); +}
You can merge these two functions, as there is no other caller.
arch/arm64/kernel/paravirt.c line 91
Quoted 12 lines
arch/arm64/kernel/paravirt.c +12
-0+ +static int init_stolen_time_cpu (unsigned int cpu) +{ +struct kvmarm_stolen_time_region *reg; +struct arm_smccc_res res; + + reg =this_cpu_ptr (&stolen_time_region); + +arm_smccc_1_1_invoke (ARM_SMCCC_HV_PV_TIME_ST , &res); + +if ((long )res.a0 <0 ) +return -EINVAL ;
I'd rather you check an actual error code, just in case the memory map starts growing to a point where we have 64bit (I)PAs...
arch/arm64/kernel/paravirt.c line 104
Quoted 13 lines
arch/arm64/kernel/paravirt.c +13
-0+ + reg->kaddr =memremap (res.a0, +sizeof (struct pvclock_vcpu_stolen_time ), +MEMREMAP_WB ); + +if (!reg->kaddr) { +pr_warn ("Failed to map stolen time data structure\n" ); +return -ENOMEM ; + } + +if (le32_to_cpu (reg->kaddr->revision) !=0 || +le32_to_cpu (reg->kaddr->attributes) !=0 ) { +pr_warn ("Unexpected revision or attributes in stolen time data\n" );
WARN_ONCE instead? You probably don't want to scream for each and every CPU that boots...
Quoted 9 lines
arch/arm64/kernel/paravirt.c +9
-0+return -ENXIO ; + } + +return 0 ; +} + +static int kvm_arm_init_stolen_time (void ) +{ +int ret;
arch/arm64/kernel/paravirt.c line 123
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+ + ret =cpuhp_setup_state (CPUHP_AP_ARM_KVMPV_STARTING , +"hypervisor/kvmarm/pv:starting" , + init_stolen_time_cpu, stolen_time_dying_cpu); +if (ret <0 ) +return ret; +return 0 ; +} + +static bool has_kvm_steal_clock (void )
This is not KVM specific either.
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+{ +struct arm_smccc_res res; + +/* To detect the presence of PV time support we require SMCCC 1.1+ */ +if (psci_ops.smccc_version <SMCCC_VERSION_1_1 ) +return false; + +arm_smccc_1_1_invoke (ARM_SMCCC_ARCH_FEATURES_FUNC_ID , +ARM_SMCCC_HV_PV_TIME_FEATURES , &res); +
arch/arm64/kernel/paravirt.c line 143
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+if (res.a0 !=SMCCC_RET_SUCCESS ) +return false; + +arm_smccc_1_1_invoke (ARM_SMCCC_HV_PV_TIME_FEATURES , +ARM_SMCCC_HV_PV_TIME_ST , &res); + +if (res.a0 !=SMCCC_RET_SUCCESS ) +return false; + +return true;
return (res.a0 == SMCCC_RET_SUCCESS);
arch/arm64/kernel/paravirt.c line 146
Quoted 3 lines
arch/arm64/kernel/paravirt.c +3
-0+} + +int __init kvm_guest_init (void )
How about something like pv_time_init() instead? In the guest, this is no way KVM specific, and I still hope for this to work on things like Xen/HyperV/VMware (yeah, I'm foolishly optimistic). All the references to KVM should go, and be replaced by something more generic (after all, you're only implementing the spec, so feel free to call it den0057_* if you really want).
Quoted 55 lines
arch/arm64/kernel/paravirt.c +20
-0+{ +int ret; + +if (!has_kvm_steal_clock ()) +return 0 ; + + ret =kvm_arm_init_stolen_time (); +if (ret) +return ret; + + pv_ops.time.steal_clock = kvm_steal_clock; + +static_key_slow_inc (¶virt_steal_enabled); +if (steal_acc) +static_key_slow_inc (¶virt_steal_rq_enabled); + +pr_info ("using stolen time PV\n" ); + +return 0 ; +}arch/arm64/kernel/time.c +3
-0@@ -30,6 +30,7 @@#include <asm/thread_info.h> #include <asm/stacktrace.h> +#include <asm/paravirt.h> unsigned long profile_pc (struct pt_regs *regs) {@@ -65,4 +66,6 @@ void __init time_init(void)/* Calibrate the delay loop directly */ lpj_fine = arch_timer_rate /HZ ; + +kvm_guest_init (); }include/linux/cpuhotplug.h +1
-0@@ -136,6 +136,7 @@ enum cpuhp_state {/* Must be the last timer callback */ CPUHP_AP_DUMMY_TIMER_STARTING ,CPUHP_AP_ARM_XEN_STARTING , +CPUHP_AP_ARM_KVMPV_STARTING ,CPUHP_AP_ARM_CORESIGHT_STARTING ,CPUHP_AP_ARM64_ISNDEP_STARTING , CPUHP_AP_SMPCFD_DYING,Signature · 1 line
2.20.1
Thanks,
M.
Signature · 5 lines
Jazz is not dead, it just smells funny. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Jürgen Groß
maintainerRe: [PATCH v6 10/10] arm64: Retrieve stolen time as paravirtualized guest
Quoted 222 lines · Marc Zyngier
Quoted 95 lines · Steven Price
Enable paravirtualization features when running under a hypervisor supporting the
PV_TIME_SThypercall.For each (v)CPU, we ask the hypervisor for the location of a shared page which the hypervisor will use to report stolen time to us. We set
pv_time_opsto the stolen time function which simply reads the stolen value from the shared page for a VCPU. We guarantee single-copy atomicity usingREAD_ONCEwhich means we can also read the stolen time for another VCPU than the currently running one while it is potentially being updated by the hypervisor.Signed-off-by Steven Price <[email protected]>--- .../admin-guide/kernel-parameters.txt | 6 +- arch/arm64/include/asm/paravirt.h | 9 +- arch/arm64/kernel/paravirt.c | 148 ++++++++++++++++++ arch/arm64/kernel/time.c | 3 + include/linux/cpuhotplug.h | 1 + 5 files changed, 163 insertions(+), 4 deletions(-)Documentation/admin-guide/kernel-parameters.txt +3
-3@@ -3083,9 +3083,9 @@ [X86,PV_OPS] Disable paravirtualized VMware scheduler clock and use the default one. - no-steal-acc [X86,KVM] Disable paravirtualized steal time accounting. - steal time is computed, but won't influence scheduler - behaviour + no-steal-acc [X86,KVM,ARM64] Disable paravirtualized steal time + accounting. steal time is computed, but won't + influence scheduler behaviour nolapic [X86-32,APIC] Do not enable or use the local APIC.arch/arm64/include/asm/paravirt.h +8
-1@@ -21,6 +21,13 @@ static inline u64 paravirt_steal_clock(int cpu) {return pv_ops.time.steal_clock (cpu); } -#endif + +int __init kvm_guest_init (void ); + +#else + +#define kvm_guest_init () + +#endif // CONFIG_PARAVIRT #endif arch/arm64/kernel/paravirt.c +20
-0@@ -6,13 +6,161 @@* Author: Stefano Stabellini <[email protected]> */ +#define pr_fmt (fmt) "kvmarm-pv: " fmt + +#include <linux/arm-smccc.h> +#include <linux/cpuhotplug.h> #include <linux/export.h> +#include <linux/io.h> #include <linux/jump_label.h> +#include <linux/printk.h> +#include <linux/psci.h> +#include <linux/reboot.h> +#include <linux/slab.h> #include <linux/types.h> +#include <asm/paravirt.h> +#include <asm/pvclock-abi.h> +#include <asm/smp_plat.h> struct static_key paravirt_steal_enabled;struct static_key paravirt_steal_rq_enabled;struct paravirt_patch_template pv_ops;EXPORT_SYMBOL_GPL (pv_ops); + +struct kvmarm_stolen_time_region { +struct pvclock_vcpu_stolen_time *kaddr; +}; + +static DEFINE_PER_CPU(struct kvmarm_stolen_time_region, stolen_time_region); + +static bool steal_acc = true;arch/arm64/kernel/paravirt.c line 48
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+static int __init parse_no_stealacc (char *arg) +{ + steal_acc = false; +return 0 ; +} + +early_param ("no-steal-acc" , parse_no_stealacc); + +/* return stolen time in ns by asking the hypervisor */ +static u64 kvm_steal_clock (int cpu)This isn't KVM specific.
Quoted 21 lines
arch/arm64/kernel/paravirt.c +21
-0+{ +struct kvmarm_stolen_time_region *reg; + + reg =per_cpu_ptr (&stolen_time_region, cpu); +if (!reg->kaddr) { +pr_warn_once ("stolen time enabled but not configured for cpu %d\n" , + cpu); +return 0 ; + } + +return le64_to_cpu (READ_ONCE (reg->kaddr->stolen_time)); +} + +static int disable_stolen_time_current_cpu (void ) +{ +struct kvmarm_stolen_time_region *reg; + + reg =this_cpu_ptr (&stolen_time_region); +if (!reg->kaddr) +return 0 ; +arch/arm64/kernel/paravirt.c line 79
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+memunmap (reg->kaddr); +memset (reg,0 ,sizeof (*reg)); + +return 0 ; +} + +static int stolen_time_dying_cpu (unsigned int cpu) +{ +return disable_stolen_time_current_cpu (); +}You can merge these two functions, as there is no other caller.
arch/arm64/kernel/paravirt.c line 91
Quoted 12 lines
arch/arm64/kernel/paravirt.c +12
-0+ +static int init_stolen_time_cpu (unsigned int cpu) +{ +struct kvmarm_stolen_time_region *reg; +struct arm_smccc_res res; + + reg =this_cpu_ptr (&stolen_time_region); + +arm_smccc_1_1_invoke (ARM_SMCCC_HV_PV_TIME_ST , &res); + +if ((long )res.a0 <0 ) +return -EINVAL ;I'd rather you check an actual error code, just in case the memory map starts growing to a point where we have 64bit (I)PAs...
arch/arm64/kernel/paravirt.c line 104
Quoted 13 lines
arch/arm64/kernel/paravirt.c +13
-0+ + reg->kaddr =memremap (res.a0, +sizeof (struct pvclock_vcpu_stolen_time ), +MEMREMAP_WB ); + +if (!reg->kaddr) { +pr_warn ("Failed to map stolen time data structure\n" ); +return -ENOMEM ; + } + +if (le32_to_cpu (reg->kaddr->revision) !=0 || +le32_to_cpu (reg->kaddr->attributes) !=0 ) { +pr_warn ("Unexpected revision or attributes in stolen time data\n" );
WARN_ONCEinstead? You probably don't want to scream for each and every CPU that boots...Quoted 9 lines
arch/arm64/kernel/paravirt.c +9
-0+return -ENXIO ; + } + +return 0 ; +} + +static int kvm_arm_init_stolen_time (void ) +{ +int ret;arch/arm64/kernel/paravirt.c line 123
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+ + ret =cpuhp_setup_state (CPUHP_AP_ARM_KVMPV_STARTING , +"hypervisor/kvmarm/pv:starting" , + init_stolen_time_cpu, stolen_time_dying_cpu); +if (ret <0 ) +return ret; +return 0 ; +} + +static bool has_kvm_steal_clock (void )This is not KVM specific either.
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+{ +struct arm_smccc_res res; + +/* To detect the presence of PV time support we require SMCCC 1.1+ */ +if (psci_ops.smccc_version <SMCCC_VERSION_1_1 ) +return false; + +arm_smccc_1_1_invoke (ARM_SMCCC_ARCH_FEATURES_FUNC_ID , +ARM_SMCCC_HV_PV_TIME_FEATURES , &res); +arch/arm64/kernel/paravirt.c line 143
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+if (res.a0 !=SMCCC_RET_SUCCESS ) +return false; + +arm_smccc_1_1_invoke (ARM_SMCCC_HV_PV_TIME_FEATURES , +ARM_SMCCC_HV_PV_TIME_ST , &res); + +if (res.a0 !=SMCCC_RET_SUCCESS ) +return false; + +return true;return (res.a0 == SMCCC_RET_SUCCESS);
Quoted 10 lines
arch/arm64/kernel/paravirt.c line 146
Quoted 3 lines
arch/arm64/kernel/paravirt.c +3
-0+} + +int __init kvm_guest_init (void )How about something like
pv_time_init()instead? In the guest, this is no way KVM specific, and I still hope for this to work on things like Xen/HyperV/VMware (yeah, I'm foolishly optimistic). All the references to KVM should go, and be replaced by something more generic (after all, you're only implementing the spec, so feel free to call it den0057_* if you really want).
Xen guests already have the needed functionality. On ARM this just needs to be hooked up.
Juergen
Marc Zyngier
maintainerRe: [PATCH v6 10/10] arm64: Retrieve stolen time as paravirtualized guest
Quoted 11 lines · Jürgen Groß
Quoted 6 lines · Marc Zyngier
How about something like
pv_time_init()instead? In the guest, this is no way KVM specific, and I still hope for this to work on things like Xen/HyperV/VMware (yeah, I'm foolishly optimistic). All the references to KVM should go, and be replaced by something more generic (after all, you're only implementing the spec, so feel free to call it den0057_* if you really want).Xen guests already have the needed functionality. On ARM this just needs to be hooked up.
Yes, Xen offers its own PV interface for that. But this code is about implementing support for a cross hypervisor functionnality (which AFAICT is not implemented by Xen).
M.
Signature · 6 lines
Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 10/10] arm64: Retrieve stolen time as paravirtualized guest
In reply to Marc ZyngierQuoted 101 lines · Marc Zyngier
Quoted 99 lines · Steven Price
Enable paravirtualization features when running under a hypervisor supporting the
PV_TIME_SThypercall.For each (v)CPU, we ask the hypervisor for the location of a shared page which the hypervisor will use to report stolen time to us. We set
pv_time_opsto the stolen time function which simply reads the stolen value from the shared page for a VCPU. We guarantee single-copy atomicity usingREAD_ONCEwhich means we can also read the stolen time for another VCPU than the currently running one while it is potentially being updated by the hypervisor.Signed-off-by Steven Price <[email protected]>--- .../admin-guide/kernel-parameters.txt | 6 +- arch/arm64/include/asm/paravirt.h | 9 +- arch/arm64/kernel/paravirt.c | 148 ++++++++++++++++++ arch/arm64/kernel/time.c | 3 + include/linux/cpuhotplug.h | 1 + 5 files changed, 163 insertions(+), 4 deletions(-)Documentation/admin-guide/kernel-parameters.txt +3
-3@@ -3083,9 +3083,9 @@ [X86,PV_OPS] Disable paravirtualized VMware scheduler clock and use the default one. - no-steal-acc [X86,KVM] Disable paravirtualized steal time accounting. - steal time is computed, but won't influence scheduler - behaviour + no-steal-acc [X86,KVM,ARM64] Disable paravirtualized steal time + accounting. steal time is computed, but won't + influence scheduler behaviour nolapic [X86-32,APIC] Do not enable or use the local APIC.arch/arm64/include/asm/paravirt.h +8
-1@@ -21,6 +21,13 @@ static inline u64 paravirt_steal_clock(int cpu) {return pv_ops.time.steal_clock (cpu); } -#endif + +int __init kvm_guest_init (void ); + +#else + +#define kvm_guest_init () + +#endif // CONFIG_PARAVIRT #endif arch/arm64/kernel/paravirt.c +24
-0@@ -6,13 +6,161 @@* Author: Stefano Stabellini <[email protected]> */ +#define pr_fmt (fmt) "kvmarm-pv: " fmt + +#include <linux/arm-smccc.h> +#include <linux/cpuhotplug.h> #include <linux/export.h> +#include <linux/io.h> #include <linux/jump_label.h> +#include <linux/printk.h> +#include <linux/psci.h> +#include <linux/reboot.h> +#include <linux/slab.h> #include <linux/types.h> +#include <asm/paravirt.h> +#include <asm/pvclock-abi.h> +#include <asm/smp_plat.h> struct static_key paravirt_steal_enabled;struct static_key paravirt_steal_rq_enabled;struct paravirt_patch_template pv_ops;EXPORT_SYMBOL_GPL (pv_ops); + +struct kvmarm_stolen_time_region { +struct pvclock_vcpu_stolen_time *kaddr; +}; + +static DEFINE_PER_CPU(struct kvmarm_stolen_time_region, stolen_time_region); + +static bool steal_acc = true; +static int __init parse_no_stealacc (char *arg) +{ + steal_acc = false; +return 0 ;
Quoted 8 lines
arch/arm64/kernel/paravirt.c line 48
Quoted 6 lines
arch/arm64/kernel/paravirt.c +6
-0+} + +early_param ("no-steal-acc" , parse_no_stealacc); + +/* return stolen time in ns by asking the hypervisor */ +static u64 kvm_steal_clock (int cpu)This isn't KVM specific.
Good point - these changes have grown from an initial KVM-only implementation, but this is no longer KVM specific in any way. So I'll rename the KVM part away.
Quoted 25 lines
Quoted 25 lines
arch/arm64/kernel/paravirt.c +25
-0+{ +struct kvmarm_stolen_time_region *reg; + + reg =per_cpu_ptr (&stolen_time_region, cpu); +if (!reg->kaddr) { +pr_warn_once ("stolen time enabled but not configured for cpu %d\n" , + cpu); +return 0 ; + } + +return le64_to_cpu (READ_ONCE (reg->kaddr->stolen_time)); +} + +static int disable_stolen_time_current_cpu (void ) +{ +struct kvmarm_stolen_time_region *reg; + + reg =this_cpu_ptr (&stolen_time_region); +if (!reg->kaddr) +return 0 ; + +memunmap (reg->kaddr); +memset (reg,0 ,sizeof (*reg)); + +return 0 ;
Quoted 8 lines
arch/arm64/kernel/paravirt.c line 79
Quoted 6 lines
arch/arm64/kernel/paravirt.c +6
-0+} + +static int stolen_time_dying_cpu (unsigned int cpu) +{ +return disable_stolen_time_current_cpu (); +}You can merge these two functions, as there is no other caller.
Will do
Quoted 7 lines
Quoted 7 lines
arch/arm64/kernel/paravirt.c +7
-0+ +static int init_stolen_time_cpu (unsigned int cpu) +{ +struct kvmarm_stolen_time_region *reg; +struct arm_smccc_res res; + + reg =this_cpu_ptr (&stolen_time_region);
Quoted 8 lines
arch/arm64/kernel/paravirt.c line 91
Quoted 5 lines
arch/arm64/kernel/paravirt.c +5
-0+ +arm_smccc_1_1_invoke (ARM_SMCCC_HV_PV_TIME_ST , &res); + +if ((long )res.a0 <0 ) +return -EINVAL ;I'd rather you check an actual error code, just in case the memory map starts growing to a point where we have 64bit (I)PAs...
I suspect we'd have a whole lot of other problems with 64 bit IPAs, but fair enough; we know the one error code that can be returned.
Quoted 8 lines
Quoted 8 lines
arch/arm64/kernel/paravirt.c +8
-0+ + reg->kaddr =memremap (res.a0, +sizeof (struct pvclock_vcpu_stolen_time ), +MEMREMAP_WB ); + +if (!reg->kaddr) { +pr_warn ("Failed to map stolen time data structure\n" ); +return -ENOMEM ;
Quoted 8 lines
arch/arm64/kernel/paravirt.c line 104
Quoted 5 lines
arch/arm64/kernel/paravirt.c +5
-0+ } + +if (le32_to_cpu (reg->kaddr->revision) !=0 || +le32_to_cpu (reg->kaddr->attributes) !=0 ) { +pr_warn ("Unexpected revision or attributes in stolen time data\n" );
WARN_ONCEinstead? You probably don't want to scream for each and every CPU that boots...
Good point.
Quoted 45 lines
Quoted 9 lines
arch/arm64/kernel/paravirt.c +9
-0+return -ENXIO ; + } + +return 0 ; +} + +static int kvm_arm_init_stolen_time (void ) +{ +int ret;arch/arm64/kernel/paravirt.c line 123
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+ + ret =cpuhp_setup_state (CPUHP_AP_ARM_KVMPV_STARTING , +"hypervisor/kvmarm/pv:starting" , + init_stolen_time_cpu, stolen_time_dying_cpu); +if (ret <0 ) +return ret; +return 0 ; +} + +static bool has_kvm_steal_clock (void )This is not KVM specific either.
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+{ +struct arm_smccc_res res; + +/* To detect the presence of PV time support we require SMCCC 1.1+ */ +if (psci_ops.smccc_version <SMCCC_VERSION_1_1 ) +return false; + +arm_smccc_1_1_invoke (ARM_SMCCC_ARCH_FEATURES_FUNC_ID , +ARM_SMCCC_HV_PV_TIME_FEATURES , &res); +arch/arm64/kernel/paravirt.c line 143
Quoted 10 lines
arch/arm64/kernel/paravirt.c +10
-0+if (res.a0 !=SMCCC_RET_SUCCESS ) +return false; + +arm_smccc_1_1_invoke (ARM_SMCCC_HV_PV_TIME_FEATURES , +ARM_SMCCC_HV_PV_TIME_ST , &res); + +if (res.a0 !=SMCCC_RET_SUCCESS ) +return false; + +return true;return (res.a0 == SMCCC_RET_SUCCESS);
Quoted 10 lines
arch/arm64/kernel/paravirt.c line 146
Quoted 3 lines
arch/arm64/kernel/paravirt.c +3
-0+} + +int __init kvm_guest_init (void )How about something like
pv_time_init()instead? In the guest, this is no way KVM specific, and I still hope for this to work on things like Xen/HyperV/VMware (yeah, I'm foolishly optimistic). All the references to KVM should go, and be replaced by something more generic (after all, you're only implementing the spec, so feel free to call it den0057_* if you really want).
pv_time_init() seems like a reasonable name, it rolls off the tongue better than den0057 :)
Thanks,
Steve
Quoted 58 lines
Quoted 55 lines
arch/arm64/kernel/paravirt.c +20
-0+{ +int ret; + +if (!has_kvm_steal_clock ()) +return 0 ; + + ret =kvm_arm_init_stolen_time (); +if (ret) +return ret; + + pv_ops.time.steal_clock = kvm_steal_clock; + +static_key_slow_inc (¶virt_steal_enabled); +if (steal_acc) +static_key_slow_inc (¶virt_steal_rq_enabled); + +pr_info ("using stolen time PV\n" ); + +return 0 ; +}arch/arm64/kernel/time.c +3
-0@@ -30,6 +30,7 @@#include <asm/thread_info.h> #include <asm/stacktrace.h> +#include <asm/paravirt.h> unsigned long profile_pc (struct pt_regs *regs) {@@ -65,4 +66,6 @@ void __init time_init(void)/* Calibrate the delay loop directly */ lpj_fine = arch_timer_rate /HZ ; + +kvm_guest_init (); }include/linux/cpuhotplug.h +1
-0@@ -136,6 +136,7 @@ enum cpuhp_state {/* Must be the last timer callback */ CPUHP_AP_DUMMY_TIMER_STARTING ,CPUHP_AP_ARM_XEN_STARTING , +CPUHP_AP_ARM_KVMPV_STARTING ,CPUHP_AP_ARM_CORESIGHT_STARTING ,CPUHP_AP_ARM64_ISNDEP_STARTING , CPUHP_AP_SMPCFD_DYING,Signature · 1 line
2.20.1Thanks,
Quoted 2 lines
M.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel