Allow userspace to manage MSRs
Subsystems (6)
v3 cover Allow userspace to manage MSRs
This series makes it possible for userspace to manage MSRs by having KVM forward select MSRs to it when rdmsr and wrmsr are executed in the guest. Userspace can set this up by calling the ioctl KVM_SET_EXIT_MSRS with a list of MSRs it wants to manage. When KVM encounters any of these MSRs they are forwarded to userspace for processing. Userspace can then read from or write to the MSR, or it can also throw a #GP if needed.
This series includes the kernel changes needed to implement this feature and a test that exercises this behavior. Also, included is an implementation of expection handling in selftests, which allows the test to excercise throwing a #GP.
v1 -> v2:
- Added support for generic instruction emulator bouncing to userspace when rdmsr or wrmsr are called, and userspace has asked to manage the MSR. These changes are committed in patch 3, and are based on changes made by Alexander Graf <[email protected]>.
- Added tests to excercise the code paths for em_{rdmsr,wrmsr} and emulator_{get,set}_msr. These changes are committed in patch 8.
v2 -> v3:
- Added the series by Alexander Graf <[email protected]> to the beginning of This series (patches 1-3). The two have a lot of overlap, so it made sense to combine them to simplify merging them both upstream. Alex's changes account for the first 3 commits in this series. As a result of incorporating those changes, commit 05/12 required some refactoring.
- Split exception handling in selftests into its own commit (patch 09/12).
- Split the changes to ucall_get() into it's own commit based on Andrew Jones
suggestion, and added support for aarch64 and s390x.
Aaron Lewis (12):
KVM: x86: Deflect unknown MSR accesses to user space
KVM: x86: Introduce allow list for MSR emulation
KVM: selftests: Add test for user space MSR handling
KVM: x86: Add ioctl for accepting a userspace provided MSR list
KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
KVM: x86: Prepare MSR bitmaps for userspace tracked MSRs
KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
selftests: kvm: Fix the segment descriptor layout to match the actual
layout
selftests: kvm: Clear uc so UCALL_NONE is being properly reported
selftests: kvm: Add exception handling to selftests
selftests: kvm: Add a test to exercise the userspace MSR list
selftests: kvm: Add emulated rdmsr, wrmsr tests
Documentation/virt/kvm/api.rst | 181 +++++++- arch/x86/include/asm/kvm_host.h | 18 + arch/x86/include/uapi/asm/kvm.h | 15 + arch/x86/kvm/emulate.c | 18 +- arch/x86/kvm/svm/svm.c | 93 ++-- arch/x86/kvm/trace.h | 24 + arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/vmx.c | 94 ++-- arch/x86/kvm/vmx/vmx.h | 2 +- arch/x86/kvm/x86.c | 379 +++++++++++++++- include/trace/events/kvm.h | 2 +- include/uapi/linux/kvm.h | 17 + tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 21 +- .../selftests/kvm/include/x86_64/processor.h | 29 +- .../testing/selftests/kvm/lib/aarch64/ucall.c | 3 + tools/testing/selftests/kvm/lib/kvm_util.c | 17 + .../selftests/kvm/lib/kvm_util_internal.h | 2 + tools/testing/selftests/kvm/lib/s390x/ucall.c | 3 + .../selftests/kvm/lib/x86_64/handlers.S | 81 ++++ .../selftests/kvm/lib/x86_64/processor.c | 168 ++++++- .../testing/selftests/kvm/lib/x86_64/ucall.c | 3 + .../selftests/kvm/x86_64/user_msr_test.c | 221 +++++++++ .../selftests/kvm/x86_64/userspace_msr_exit.c | 421 ++++++++++++++++++ 24 files changed, 1719 insertions(+), 96 deletions(-) create mode 100644 tools/testing/selftests/kvm/lib/x86_64/handlers.S create mode 100644 tools/testing/selftests/kvm/x86_64/user_msr_test.c create mode 100644 tools/testing/selftests/kvm/x86_64/userspace_msr_exit.c
Signature · 1 line
2.28.0.220.ged08abb693-goog
PATCH v3 1/12 KVM: x86: Deflect unknown MSR accesses to user space
MSRs are weird. Some of them are normal control registers, such as EFER. Some however are registers that really are model specific, not very interesting to virtualization workloads, and not performance critical. Others again are really just windows into package configuration.
Out of these MSRs, only the first category is necessary to implement in kernel space. Rarely accessed MSRs, MSRs that should be fine tunes against certain CPU models and MSRs that contain information on the package level are much better suited for user space to process. However, over time we have accumulated a lot of MSRs that are not the first category, but still handled by in-kernel KVM code.
This patch adds a generic interface to handle WRMSR and RDMSR from user space. With this, any future MSR that is part of the latter categories can be handled in user space.
Furthermore, it allows us to replace the existing "ignore_msrs" logic with something that applies per-VM rather than on the full system. That way you can run productive VMs in parallel to experimental ones where you don't care about proper MSR handling.
---
v1 -> v2:
- s/ETRAP_TO_USER_SPACE/ENOENT/g
- deflect all #GP injection events to user space, not just unknown MSRs. That was we can also deflect allowlist errors later
- fix emulator case
v2 -> v3:
- return r if r == X86EMUL_IO_NEEDED
- s/KVM_EXIT_RDMSR/KVM_EXIT_X86_RDMSR/g
- s/KVM_EXIT_WRMSR/KVM_EXIT_X86_WRMSR/g
- Use complete_userspace_io logic instead of reply field
- Simplify trapping code
v3 -> v4:
- Mention exit reasons in re-inter mandatory section of API documentation
- Clear padding bytes
- Generalize get/set deflect functions
- Remove redundant pending_user_msr field
--- Documentation/virt/kvm/api.rst | 66 +++++++++++++++++++- arch/x86/include/asm/kvm_host.h | 3 + arch/x86/kvm/emulate.c | 18 +++++- arch/x86/kvm/x86.c | 106 ++++++++++++++++++++++++++++++-- include/trace/events/kvm.h | 2 +- include/uapi/linux/kvm.h | 10 +++ 6 files changed, 196 insertions(+), 9 deletions(-)
Documentation/virt/kvm/api.rst +64 -2
@@ -4869,8 +4869,8 @@ to the byte array. .. note:: - For KVM_EXIT_IO, KVM_EXIT_MMIO, KVM_EXIT_OSI, KVM_EXIT_PAPR and - KVM_EXIT_EPR the corresponding + For KVM_EXIT_IO, KVM_EXIT_MMIO, KVM_EXIT_OSI, KVM_EXIT_PAPR, + KVM_EXIT_EPR, KVM_EXIT_X86_RDMSR and KVM_EXIT_X86_WRMSR the corresponding operations are complete (and guest state is consistent) only after userspace has re-entered the kernel with KVM_RUN. The kernel side will first finish
@@ -5163,6 +5163,35 @@ Note that KVM does not skip the faulting instruction as it does for KVM_EXIT_MMIO, but userspace has to emulate any change to the processing state if it decides to decode and emulate the instruction. +:: + + /* KVM_EXIT_X86_RDMSR / KVM_EXIT_X86_WRMSR */ + struct { + __u8 error; + __u8 pad[3]; + __u32 index; + __u64 data; + } msr; + +Used on x86 systems. When the VM capability KVM_CAP_X86_USER_SPACE_MSR is +enabled, MSR accesses to registers that would invoke a #GP by KVM kernel code +will instead trigger a KVM_EXIT_X86_RDMSR exit for reads and KVM_EXIT_X86_WRMSR +exit for writes. + +For KVM_EXIT_X86_RDMSR, the "index" field tells user space which MSR the guest +wants to read. To respond to this request with a successful read, user space +writes the respective data into the "data" field and must continue guest +execution to ensure the read data is transferred into guest register state. + +If the RDMSR request was unsuccessful, user space indicates that with a "1" in +the "error" field. This will inject a #GP into the guest when the VCPU is +executed again. + +For KVM_EXIT_X86_WRMSR, the "index" field tells user space which MSR the guest +wants to write. Once finished processing the event, user space must continue +vCPU execution. If the MSR write was unsuccessful, user space also sets the +"error" field to "1". + :: /* Fix the size of the union. */
@@ -5852,6 +5881,28 @@ controlled by the kvm module parameter halt_poll_ns. This capability allows the maximum halt time to specified on a per-VM basis, effectively overriding the module parameter for the target VM. +7.21 KVM_CAP_X86_USER_SPACE_MSR +------------------------------- + +:Architectures: x86 +:Target: VM +:Parameters: args[0] is 1 if user space MSR handling is enabled, 0 otherwise +:Returns: 0 on success; -1 on error + +This capability enables trapping of #GP invoking RDMSR and WRMSR instructions +into user space. + +When a guest requests to read or write an MSR, KVM may not implement all MSRs +that are relevant to a respective system. It also does not differentiate by +CPU type. + +To allow more fine grained control over MSR handling, user space may enable +this capability. With it enabled, MSR accesses that would usually trigger +a #GP event inside the guest by KVM will instead trigger KVM_EXIT_X86_RDMSR +and KVM_EXIT_X86_WRMSR exit notifications which user space can then handle to +implement model specific MSR handling and/or user notifications to inform +a user that an MSR was not handled. + 8. Other capabilities. ======================
@@ -6159,3 +6210,14 @@ KVM can therefore start protected VMs. This capability governs the KVM_S390_PV_COMMAND ioctl and the KVM_MP_STATE_LOAD MP_STATE. KVM_SET_MP_STATE can fail for protected guests when the state change is invalid. + +8.24 KVM_CAP_X86_USER_SPACE_MSR +---------------------------- + +:Architectures: x86 + +This capability indicates that KVM supports deflection of MSR reads and +writes to user space. It can be enabled on a VM level. If enabled, MSR +accesses that would usually trigger a #GP by KVM into the guest will +instead get bounced to user space through the KVM_EXIT_X86_RDMSR and +KVM_EXIT_X86_WRMSR exit notifications.
arch/x86/include/asm/kvm_host.h +3 -0
@@ -961,6 +961,9 @@ struct kvm_arch {bool guest_can_read_msr_platform_info;bool exception_payload_enabled; +/* Deflect RDMSR and WRMSR to user space when they trigger a #GP */ +bool user_space_msr_enabled; +struct kvm_pmu_event_filter *pmu_event_filter;struct task_struct *nx_lpage_recovery_thread; };
arch/x86/kvm/emulate.c +16 -2
@@ -3689,11 +3689,18 @@ static int em_dr_write(struct x86_emulate_ctxt *ctxt)static int em_wrmsr (struct x86_emulate_ctxt *ctxt) { +u64 msr_index =reg_read (ctxt,VCPU_REGS_RCX );u64 msr_data; +int r; msr_data = (u32 )reg_read (ctxt,VCPU_REGS_RAX ) | ((u64 )reg_read (ctxt,VCPU_REGS_RDX ) <<32 ); -if (ctxt->ops->set_msr (ctxt,reg_read (ctxt,VCPU_REGS_RCX ), msr_data)) + r = ctxt->ops->set_msr (ctxt, msr_index, msr_data); + +if (r ==X86EMUL_IO_NEEDED ) +return r; + +if (r)return emulate_gp (ctxt,0 );return X86EMUL_CONTINUE ;
@@ -3701,9 +3708,16 @@ static int em_wrmsr(struct x86_emulate_ctxt *ctxt)static int em_rdmsr (struct x86_emulate_ctxt *ctxt) { +u64 msr_index =reg_read (ctxt,VCPU_REGS_RCX );u64 msr_data; +int r; + + r = ctxt->ops->get_msr (ctxt, msr_index, &msr_data); + +if (r ==X86EMUL_IO_NEEDED ) +return r; -if (ctxt->ops->get_msr (ctxt,reg_read (ctxt,VCPU_REGS_RCX ), &msr_data)) +if (r)return emulate_gp (ctxt,0 ); *reg_write (ctxt,VCPU_REGS_RAX ) = (u32 )msr_data;
arch/x86/kvm/x86.c +102 -4
147 lines not on this page · raw message
include/trace/events/kvm.h +1 -1
@@ -17,7 +17,7 @@ERSN (NMI ),ERSN (INTERNAL_ERROR ),ERSN (OSI ),ERSN (PAPR_HCALL ), \ERSN (S390_UCONTROL ),ERSN (WATCHDOG ),ERSN (S390_TSCH ),ERSN (EPR ),\ERSN (SYSTEM_EVENT ),ERSN (S390_STSI ),ERSN (IOAPIC_EOI ), \ - ERSN(HYPERV), ERSN(ARM_NISV) + ERSN(HYPERV), ERSN(ARM_NISV), ERSN(X86_RDMSR), ERSN(X86_WRMSR) TRACE_EVENT(kvm_userspace_exit, TP_PROTO(__u32 reason, int errno),
include/uapi/linux/kvm.h +10 -0
@@ -248,6 +248,8 @@ struct kvm_hyperv_exit {#define KVM_EXIT_IOAPIC_EOI 26#define KVM_EXIT_HYPERV 27#define KVM_EXIT_ARM_NISV 28 +#define KVM_EXIT_X86_RDMSR 29 +#define KVM_EXIT_X86_WRMSR 30/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */
@@ -413,6 +415,13 @@ struct kvm_run {__u64 esr_iss;__u64 fault_ipa; } arm_nisv; +/* KVM_EXIT_X86_RDMSR / KVM_EXIT_X86_WRMSR */ +struct { +__u8 error; +__u8 pad[3 ]; +__u32 index; +__u64 data; + } msr;/* Fix the size of the union. */ char padding[256 ]; };
@@ -1035,6 +1044,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_LAST_CPU 184#define KVM_CAP_SMALLER_MAXPHYADDR 185#define KVM_CAP_S390_DIAG318 186 +#define KVM_CAP_X86_USER_SPACE_MSR 187#ifdef KVM_CAP_IRQ_ROUTING
Signature · 1 line
2.28.0.220.ged08abb693-goog
Re: [PATCH v3 01/12] KVM: x86: Deflect unknown MSR accesses to user space
Quoted 17 lines · Aaron Lewis
MSRs are weird. Some of them are normal control registers, such as EFER. Some however are registers that really are model specific, not very interesting to virtualization workloads, and not performance critical. Others again are really just windows into package configuration.
Out of these MSRs, only the first category is necessary to implement in kernel space. Rarely accessed MSRs, MSRs that should be fine tunes against certain CPU models and MSRs that contain information on the package level are much better suited for user space to process. However, over time we have accumulated a lot of MSRs that are not the first category, but still handled by in-kernel KVM code.
This patch adds a generic interface to handle WRMSR and RDMSR from user space. With this, any future MSR that is part of the latter categories can be handled in user space.
Quoted 7 lines
Furthermore, it allows us to replace the existing "
ignore_msrs" logic with something that applies per-VM rather than on the full system. That way you can run productive VMs in parallel to experimental ones where you don't care about proper MSR handling.Signed-off-by Alexander Graf <[email protected]>Reviewed-by Jim Mattson <[email protected]>
You need to add your Signed-off-by line here as well :).
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
PATCH v3 2/12 KVM: x86: Introduce allow list for MSR emulation
In reply to Aaron LewisIt's not desireable to have all MSRs always handled by KVM kernel space. Some MSRs would be useful to handle in user space to either emulate behavior (like uCode updates) or differentiate whether they are valid based on the CPU model.
To allow user space to specify which MSRs it wants to see handled by KVM, this patch introduces a new ioctl to push allow lists of bitmaps into KVM. Based on these bitmaps, KVM can then decide whether to reject MSR access. With the addition of KVM_CAP_X86_USER_SPACE_MSR it can also deflect the denied MSR events to user space to operate on.
If no allowlist is populated, MSR handling stays identical to before.
---
v2 -> v3:
- document flags for KVM_X86_ADD_MSR_ALLOWLIST
- generalize exit path, always unlock when returning
- s/KVM_CAP_ADD_MSR_ALLOWLIST/KVM_CAP_X86_MSR_ALLOWLIST/g
- Add KVM_X86_CLEAR_MSR_ALLOWLIST
v3 -> v4:
- lock allow check and clearing
- free bitmaps on clear
v4 -> v5:
- use srcu
--- Documentation/virt/kvm/api.rst | 91 ++++++++++++++++++ arch/x86/include/asm/kvm_host.h | 10 ++ arch/x86/include/uapi/asm/kvm.h | 15 +++ arch/x86/kvm/x86.c | 160 ++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 5 + 5 files changed, 281 insertions(+)
Documentation/virt/kvm/api.rst +91 -0
@@ -4704,6 +4704,82 @@ KVM_PV_VM_VERIFY Verify the integrity of the unpacked image. Only if this succeeds, KVM is allowed to start protected VCPUs. +4.126 KVM_X86_ADD_MSR_ALLOWLIST +------------------------------- + +:Capability: KVM_CAP_X86_MSR_ALLOWLIST +:Architectures: x86 +:Type: vm ioctl +:Parameters: struct kvm_msr_allowlist +:Returns: 0 on success, < 0 on error + +:: + + struct kvm_msr_allowlist { + __u32 flags; + __u32 nmsrs; /* number of msrs in bitmap */ + __u32 base; /* base address for the MSRs bitmap */ + __u32 pad; + + __u8 bitmap[0]; /* a set bit allows that the operation set in flags */ + }; + +flags values: + +KVM_MSR_ALLOW_READ + + Filter read accesses to MSRs using the given bitmap. A 0 in the bitmap + indicates that a read should immediately fail, while a 1 indicates that + a read should be handled by the normal KVM MSR emulation logic. + +KVM_MSR_ALLOW_WRITE + + Filter write accesses to MSRs using the given bitmap. A 0 in the bitmap + indicates that a write should immediately fail, while a 1 indicates that + a write should be handled by the normal KVM MSR emulation logic. + +KVM_MSR_ALLOW_READ | KVM_MSR_ALLOW_WRITE + + Filter booth read and write accesses to MSRs using the given bitmap. A 0 + in the bitmap indicates that both reads and writes should immediately fail, + while a 1 indicates that reads and writes should be handled by the normal + KVM MSR emulation logic. + +This ioctl allows user space to define a set of bitmaps of MSR ranges to +specify whether a certain MSR access is allowed or not. + +If this ioctl has never been invoked, MSR accesses are not guarded and the +old KVM in-kernel emulation behavior is fully preserved. + +As soon as the first allow list was specified, only allowed MSR accesses +are permitted inside of KVM's MSR code. + +Each allowlist specifies a range of MSRs to potentially allow access on. +The range goes from MSR index [base .. base+nmsrs]. The flags field +indicates whether reads, writes or both reads and writes are permitted +by setting a 1 bit in the bitmap for the corresponding MSR index. + +If an MSR access is not permitted through the allow list, it generates a +#GP inside the guest. When combined with KVM_CAP_X86_USER_SPACE_MSR, that +allows user space to deflect and potentially handle various MSR accesses +into user space. + +4.124 KVM_X86_CLEAR_MSR_ALLOWLIST +--------------------------------- + +:Capability: KVM_CAP_X86_MSR_ALLOWLIST +:Architectures: x86 +:Type: vcpu ioctl +:Parameters: none +:Returns: 0 + +This ioctl resets all internal MSR allow lists. After this call, no allow +list is present and the guest would execute as if no allow lists were set, +so all MSRs are considered allowed and thus handled by the in-kernel MSR +emulation logic. + +No vCPU may be in running state when calling this ioctl. + 5. The kvm_run structure ========================
@@ -6221,3 +6297,18 @@ writes to user space. It can be enabled on a VM level. If enabled, MSR accesses that would usually trigger a #GP by KVM into the guest will instead get bounced to user space through the KVM_EXIT_X86_RDMSR and KVM_EXIT_X86_WRMSR exit notifications. + +8.25 KVM_CAP_X86_MSR_ALLOWLIST +------------------------------ + +:Architectures: x86 + +This capability indicates that KVM supports emulation of only select MSR +registers. With this capability exposed, KVM exports two new VM ioctls: +KVM_X86_ADD_MSR_ALLOWLIST which user space can call to specify bitmaps of MSR +ranges that KVM should emulate in kernel space and KVM_X86_CLEAR_MSR_ALLOWLIST +which user space can call to remove all MSR allow lists from the VM context. + +In combination with KVM_CAP_X86_USER_SPACE_MSR, this allows user space to +trap and emulate MSRs that are outside of the scope of KVM as well as +limit the attack surface on KVM's MSR emulation code.
arch/x86/include/asm/kvm_host.h +10 -0
@@ -860,6 +860,13 @@ struct kvm_hv {struct kvm_hv_syndbg hv_syndbg; }; +struct msr_bitmap_range { +u32 flags; +u32 nmsrs; +u32 base; +unsigned long *bitmap; +}; +enum kvm_irqchip_mode {KVM_IRQCHIP_NONE ,KVM_IRQCHIP_KERNEL ,/* created with KVM_CREATE_IRQCHIP */
@@ -964,6 +971,9 @@ struct kvm_arch {/* Deflect RDMSR and WRMSR to user space when they trigger a #GP */ bool user_space_msr_enabled; +struct msr_bitmap_range msr_allowlist_ranges[10 ]; +int msr_allowlist_ranges_count; +struct kvm_pmu_event_filter *pmu_event_filter;struct task_struct *nx_lpage_recovery_thread; };
arch/x86/include/uapi/asm/kvm.h +15 -0
@@ -192,6 +192,21 @@ struct kvm_msr_list {__u32 indices[0 ]; }; +#define KVM_MSR_ALLOW_READ (1 << 0) +#define KVM_MSR_ALLOW_WRITE (1 << 1) + +/* Maximum size of the of the bitmap in bytes */ +#define KVM_MSR_ALLOWLIST_MAX_LEN 0x600 + +/* for KVM_X86_ADD_MSR_ALLOWLIST */ +struct kvm_msr_allowlist { +__u32 flags; +__u32 nmsrs;/* number of msrs in bitmap */ +__u32 base;/* base address for the MSRs bitmap */ +__u32 pad; + +__u8 bitmap[0 ];/* a set bit allows that the operation set in flags */ +};struct kvm_cpuid_entry {__u32 function;
arch/x86/kvm/x86.c +160 -0
208 lines not on this page · raw message
include/uapi/linux/kvm.h +5 -0
@@ -1045,6 +1045,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_SMALLER_MAXPHYADDR 185#define KVM_CAP_S390_DIAG318 186#define KVM_CAP_X86_USER_SPACE_MSR 187 +#define KVM_CAP_X86_MSR_ALLOWLIST 188#ifdef KVM_CAP_IRQ_ROUTING
@@ -1546,6 +1547,10 @@ struct kvm_pv_cmd {/* Available with KVM_CAP_S390_PROTECTED */ #define KVM_S390_PV_COMMAND _IOWR(KVMIO, 0xc5, struct kvm_pv_cmd) +/* Available with KVM_CAP_X86_MSR_ALLOWLIST */ +#define KVM_X86_ADD_MSR_ALLOWLIST _IOW(KVMIO, 0xc6, struct kvm_msr_allowlist) +#define KVM_X86_CLEAR_MSR_ALLOWLIST _IO(KVMIO, 0xc7) +/* Secure Encrypted Virtualization command */ enum sev_cmd_id {/* Guest initialization commands */
Signature · 1 line
2.28.0.220.ged08abb693-goog
Re: [PATCH v3 02/12] KVM: x86: Introduce allow list for MSR emulation
Quoted 11 lines · Aaron Lewis
It's not desireable to have all MSRs always handled by KVM kernel space. Some MSRs would be useful to handle in user space to either emulate behavior (like uCode updates) or differentiate whether they are valid based on the CPU model.
To allow user space to specify which MSRs it wants to see handled by KVM, this patch introduces a new ioctl to push allow lists of bitmaps into KVM. Based on these bitmaps, KVM can then decide whether to reject MSR access. With the addition of
KVM_CAP_X86_USER_SPACE_MSRit can also deflect the denied MSR events to user space to operate on.
Quoted 4 lines
If no allowlist is populated, MSR handling stays identical to before.
Signed-off-by KarimAllah Ahmed <[email protected]>Signed-off-by Alexander Graf <[email protected]>
Same here, SoB line is missing.
I also see that you didn't address the nits you had on this patch:
[...]
Quoted 6 lines
Documentation/virt/kvm/api.rst KVM_PV_VM_VERIFY · line 4746
Quoted 4 lines
Documentation/virt/kvm/api.rst +4
-0+ Filter booth read and write accesses to MSRs using the given bitmap. A 0 + in the bitmap indicates that both reads and writes should immediately fail, + while a 1 indicates that reads and writes should be handled by the normal + KVM MSR emulation logic.nit: Filter both
[...]
Quoted 3 lines
Quoted 1 line
+/* Maximum size of the of the bitmap in bytes */
nit: "of the" is repeated twice
Feel free to change them in your patch setand add a note between the SoB lines:
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 02/12] KVM: x86: Introduce allow list for MSR emulation
In reply to Aaron LewisHi Aaron,
url: https://github.com/0day-ci/linux/commits/Aaron-Lewis/Allow-userspace-to-manage-MSRs/20200819-051903 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git linux-next config: x86_64-randconfig-m001-20200827 (attached as .config) compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
If you fix the issue, kindly add following tag as appropriate
smatch warnings:arch/x86/kvm/x86.c:5248 kvm_vm_ioctl_add_msr_allowlist() error: 'bitmap' dereferencing possible ERR_PTR()
# https://github.com/0day-ci/linux/commit/107c87325cf461b7b1bd07bb6ddbaf808a8d8a2a
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Aaron-Lewis/Allow-userspace-to-manage-MSRs/20200819-051903 git checkout 107c87325cf461b7b1bd07bb6ddbaf808a8d8a2a
vim +/bitmap +5248 arch/x86/kvm/x86.c
107c87325cf461 Aaron Lewis 2020-08-18 5181 static int kvm_vm_ioctl_add_msr_allowlist(struct kvm *kvm, void __user *argp)
107c87325cf461 Aaron Lewis 2020-08-18 5182 {
107c87325cf461 Aaron Lewis 2020-08-18 5183 struct msr_bitmap_range *ranges = kvm->arch.msr_allowlist_ranges;
107c87325cf461 Aaron Lewis 2020-08-18 5184 struct kvm_msr_allowlist __user *user_msr_allowlist = argp;
107c87325cf461 Aaron Lewis 2020-08-18 5185 struct msr_bitmap_range range;
107c87325cf461 Aaron Lewis 2020-08-18 5186 struct kvm_msr_allowlist kernel_msr_allowlist;
107c87325cf461 Aaron Lewis 2020-08-18 5187 unsigned long *bitmap = NULL;
107c87325cf461 Aaron Lewis 2020-08-18 5188 size_t bitmap_size;
107c87325cf461 Aaron Lewis 2020-08-18 5189 int r = 0;
107c87325cf461 Aaron Lewis 2020-08-18 5190
107c87325cf461 Aaron Lewis 2020-08-18 5191 if (copy_from_user(&kernel_msr_allowlist, user_msr_allowlist,
107c87325cf461 Aaron Lewis 2020-08-18 5192 sizeof(kernel_msr_allowlist))) {
107c87325cf461 Aaron Lewis 2020-08-18 5193 r = -EFAULT;
107c87325cf461 Aaron Lewis 2020-08-18 5194 goto out;
107c87325cf461 Aaron Lewis 2020-08-18 5195 }
107c87325cf461 Aaron Lewis 2020-08-18 5196
107c87325cf461 Aaron Lewis 2020-08-18 5197 bitmap_size = BITS_TO_LONGS(kernel_msr_allowlist.nmsrs) * sizeof(long);
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^nOn 32 bit systems the BITS_TO_LONGS() can integer overflow if kernel_msr_allowlist.nmsrs is larger than ULONG_MAX - bits_per_long. In that case bitmap_size is zero.
107c87325cf461 Aaron Lewis 2020-08-18 5198 if (bitmap_size > KVM_MSR_ALLOWLIST_MAX_LEN) {
107c87325cf461 Aaron Lewis 2020-08-18 5199 r = -EINVAL;
107c87325cf461 Aaron Lewis 2020-08-18 5200 goto out;
107c87325cf461 Aaron Lewis 2020-08-18 5201 }
107c87325cf461 Aaron Lewis 2020-08-18 5202
107c87325cf461 Aaron Lewis 2020-08-18 5203 bitmap = memdup_user(user_msr_allowlist->bitmap, bitmap_size);
107c87325cf461 Aaron Lewis 2020-08-18 5204 if (IS_ERR(bitmap)) {
107c87325cf461 Aaron Lewis 2020-08-18 5205 r = PTR_ERR(bitmap);
107c87325cf461 Aaron Lewis 2020-08-18 5206 goto out;
^^^^^^^^"out" is always a vague label name. It's better style to return directly instead of doing a complicated no-op.
if (IS_ERR(bitmap)) return PTR_ERR(bitmap);
107c87325cf461 Aaron Lewis 2020-08-18 5207 }
107c87325cf461 Aaron Lewis 2020-08-18 5208
107c87325cf461 Aaron Lewis 2020-08-18 5209 range = (struct msr_bitmap_range) {
107c87325cf461 Aaron Lewis 2020-08-18 5210 .flags = kernel_msr_allowlist.flags,
107c87325cf461 Aaron Lewis 2020-08-18 5211 .base = kernel_msr_allowlist.base,
107c87325cf461 Aaron Lewis 2020-08-18 5212 .nmsrs = kernel_msr_allowlist.nmsrs,
107c87325cf461 Aaron Lewis 2020-08-18 5213 .bitmap = bitmap,In case of overflow then "bitmap" is 0x16 and .nmsrs is a very high number.
107c87325cf461 Aaron Lewis 2020-08-18 5214 };
107c87325cf461 Aaron Lewis 2020-08-18 5215
107c87325cf461 Aaron Lewis 2020-08-18 5216 if (range.flags & ~(KVM_MSR_ALLOW_READ | KVM_MSR_ALLOW_WRITE)) {
107c87325cf461 Aaron Lewis 2020-08-18 5217 r = -EINVAL;
107c87325cf461 Aaron Lewis 2020-08-18 5218 goto out;
107c87325cf461 Aaron Lewis 2020-08-18 5219 }
107c87325cf461 Aaron Lewis 2020-08-18 5220
107c87325cf461 Aaron Lewis 2020-08-18 5221 /*
107c87325cf461 Aaron Lewis 2020-08-18 5222 * Protect from concurrent calls to this function that could trigger
107c87325cf461 Aaron Lewis 2020-08-18 5223 * a TOCTOU violation on kvm->arch.msr_allowlist_ranges_count.
107c87325cf461 Aaron Lewis 2020-08-18 5224 */
107c87325cf461 Aaron Lewis 2020-08-18 5225 mutex_lock(&kvm->lock);
107c87325cf461 Aaron Lewis 2020-08-18 5226
107c87325cf461 Aaron Lewis 2020-08-18 5227 if (kvm->arch.msr_allowlist_ranges_count >=
107c87325cf461 Aaron Lewis 2020-08-18 5228 ARRAY_SIZE(kvm->arch.msr_allowlist_ranges)) {
107c87325cf461 Aaron Lewis 2020-08-18 5229 r = -E2BIG;
107c87325cf461 Aaron Lewis 2020-08-18 5230 goto out_locked;
107c87325cf461 Aaron Lewis 2020-08-18 5231 }
107c87325cf461 Aaron Lewis 2020-08-18 5232
107c87325cf461 Aaron Lewis 2020-08-18 5233 if (msr_range_overlaps(kvm, &range)) {
107c87325cf461 Aaron Lewis 2020-08-18 5234 r = -EINVAL;
107c87325cf461 Aaron Lewis 2020-08-18 5235 goto out_locked;
107c87325cf461 Aaron Lewis 2020-08-18 5236 }
107c87325cf461 Aaron Lewis 2020-08-18 5237
107c87325cf461 Aaron Lewis 2020-08-18 5238 /* Everything ok, add this range identifier to our global pool */
107c87325cf461 Aaron Lewis 2020-08-18 5239 ranges[kvm->arch.msr_allowlist_ranges_count] = range;
107c87325cf461 Aaron Lewis 2020-08-18 5240 /* Make sure we filled the array before we tell anyone to walk it */
107c87325cf461 Aaron Lewis 2020-08-18 5241 smp_wmb();
107c87325cf461 Aaron Lewis 2020-08-18 5242 kvm->arch.msr_allowlist_ranges_count++;
107c87325cf461 Aaron Lewis 2020-08-18 5243
107c87325cf461 Aaron Lewis 2020-08-18 5244 out_locked:
107c87325cf461 Aaron Lewis 2020-08-18 5245 mutex_unlock(&kvm->lock);
107c87325cf461 Aaron Lewis 2020-08-18 5246 out:
107c87325cf461 Aaron Lewis 2020-08-18 5247 if (r)
107c87325cf461 Aaron Lewis 2020-08-18 @5248 kfree(bitmap);
107c87325cf461 Aaron Lewis 2020-08-18 5249
107c87325cf461 Aaron Lewis 2020-08-18 5250 return r;
107c87325cf461 Aaron Lewis 2020-08-18 5251 }---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Re: [PATCH v3 02/12] KVM: x86: Introduce allow list for MSR emulation
Quoted 11 lines · Dan Carpenter
Hi Aaron,
url: https://github.com/0day-ci/linux/commits/Aaron-Lewis/Allow-userspace-to-manage-MSRs/20200819-051903 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git linux-next config: x86_64-randconfig-m001-20200827 (attached as .config) compiler: gcc-9 (Debian 9.3.0-15) 9.3.0If you fix the issue, kindly add following tag as appropriate
Reported-by kernel test robot <[email protected]>Reported-by Dan Carpenter <[email protected]>
Thanks a bunch for looking at this! I'd squash in the change with the actual patch as it's tiny, so I'm not sure how attribution would work in that case.
Quoted 28 lines
smatch warnings:
arch/x86/kvm/x86.c:5248kvm_vm_ioctl_add_msr_allowlist()error: 'bitmap' dereferencing possibleERR_PTR()# https://github.com/0day-ci/linux/commit/107c87325cf461b7b1bd07bb6ddbaf808a8d8a2a
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Aaron-Lewis/Allow-userspace-to-manage-MSRs/20200819-051903 git checkout107c87325cf461b7b1bd07bb6ddbaf808a8d8a2a
vim +/bitmap +5248arch/x86/kvm/x86.c107c87325cf461 Aaron Lewis 2020-08-18 5181 static int kvm_vm_ioctl_add_msr_allowlist(struct kvm *kvm, void __user *argp) 107c87325cf461 Aaron Lewis 2020-08-18 5182 { 107c87325cf461 Aaron Lewis 2020-08-18 5183 struct msr_bitmap_range *ranges = kvm->arch.msr_allowlist_ranges; 107c87325cf461 Aaron Lewis 2020-08-18 5184 struct kvm_msr_allowlist __user *user_msr_allowlist = argp; 107c87325cf461 Aaron Lewis 2020-08-18 5185 struct msr_bitmap_range range; 107c87325cf461 Aaron Lewis 2020-08-18 5186 struct kvm_msr_allowlist kernel_msr_allowlist; 107c87325cf461 Aaron Lewis 2020-08-18 5187 unsigned long *bitmap = NULL; 107c87325cf461 Aaron Lewis 2020-08-18 5188 size_t bitmap_size; 107c87325cf461 Aaron Lewis 2020-08-18 5189 int r = 0; 107c87325cf461 Aaron Lewis 2020-08-18 5190 107c87325cf461 Aaron Lewis 2020-08-18 5191 if (copy_from_user(&kernel_msr_allowlist, user_msr_allowlist, 107c87325cf461 Aaron Lewis 2020-08-18 5192 sizeof(kernel_msr_allowlist))) { 107c87325cf461 Aaron Lewis 2020-08-18 5193 r = -EFAULT; 107c87325cf461 Aaron Lewis 2020-08-18 5194 goto out; 107c87325cf461 Aaron Lewis 2020-08-18 5195 } 107c87325cf461 Aaron Lewis 2020-08-18 5196 107c87325cf461 Aaron Lewis 2020-08-18 5197 bitmap_size = BITS_TO_LONGS(kernel_msr_allowlist.nmsrs) * sizeof(long); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^n
Quoted 3 lines
On 32 bit systems the
BITS_TO_LONGS()can integer overflow if kernel_msr_allowlist.nmsrs is larger thanULONG_MAX-bits_per_long. In that casebitmap_sizeis zero.
Nice catch! It should be enough to ...
Quoted 2 lines
107c87325cf461Aaron Lewis 2020-08-18 5198 if (bitmap_size>KVM_MSR_ALLOWLIST_MAX_LEN) {
... add a check for !bitmap_size here as well then, right?
Quoted 9 lines
107c87325cf461 Aaron Lewis 2020-08-18 5199 r = -EINVAL;
107c87325cf461 Aaron Lewis 2020-08-18 5200 goto out;
107c87325cf461 Aaron Lewis 2020-08-18 5201 }
107c87325cf461 Aaron Lewis 2020-08-18 5202
107c87325cf461 Aaron Lewis 2020-08-18 5203 bitmap = memdup_user(user_msr_allowlist->bitmap, bitmap_size);
107c87325cf461 Aaron Lewis 2020-08-18 5204 if (IS_ERR(bitmap)) {
107c87325cf461 Aaron Lewis 2020-08-18 5205 r = PTR_ERR(bitmap);
107c87325cf461 Aaron Lewis 2020-08-18 5206 goto out;
^^^^^^^^Quoted 5 lines
"out" is always a vague label name. It's better style to return directly instead of doing a complicated no-op.
if (IS_ERR(bitmap)) return PTR_ERR(bitmap);
I agree 100% :). In fact, I agree so much that I already did change it for v6 last week, just did not send it out yet.
Quoted 11 lines
107c87325cf461 Aaron Lewis 2020-08-18 5207 } 107c87325cf461 Aaron Lewis 2020-08-18 5208 107c87325cf461 Aaron Lewis 2020-08-18 5209 range = (struct msr_bitmap_range) { 107c87325cf461 Aaron Lewis 2020-08-18 5210 .flags = kernel_msr_allowlist.flags, 107c87325cf461 Aaron Lewis 2020-08-18 5211 .base = kernel_msr_allowlist.base, 107c87325cf461 Aaron Lewis 2020-08-18 5212 .nmsrs = kernel_msr_allowlist.nmsrs, 107c87325cf461 Aaron Lewis 2020-08-18 5213 .bitmap = bitmap,In case of overflow then "bitmap" is
0x16and .nmsrs is a very high number.
The overflow case should disappear with the additional check above, right?
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 02/12] KVM: x86: Introduce allow list for MSR emulation
Quoted 11 lines · Alexander Graf
Quoted 8 lines · Dan Carpenter
Hi Aaron,
url: https://github.com/0day-ci/linux/commits/Aaron-Lewis/Allow-userspace-to-manage-MSRs/20200819-051903 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git linux-next config: x86_64-randconfig-m001-20200827 (attached as .config) compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
Quoted 6 lines
Quoted 3 lines
If you fix the issue, kindly add following tag as appropriate
Reported-by kernel test robot <[email protected]>Reported-by Dan Carpenter <[email protected]>Thanks a bunch for looking at this! I'd squash in the change with the actual patch as it's tiny, so I'm not sure how attribution would work in that case.
Yep. No problem. These are just a template that gets sent to everyone.
Quoted 34 lines
Quoted 30 lines
smatch warnings:
arch/x86/kvm/x86.c:5248kvm_vm_ioctl_add_msr_allowlist()error: 'bitmap' dereferencing possibleERR_PTR()# https://github.com/0day-ci/linux/commit/107c87325cf461b7b1bd07bb6ddbaf808a8d8a2a
git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Aaron-Lewis/Allow-userspace-to-manage-MSRs/20200819-051903
git checkout107c87325cf461b7b1bd07bb6ddbaf808a8d8a2a
vim +/bitmap +5248arch/x86/kvm/x86.c107c87325cf461 Aaron Lewis 2020-08-18 5181 static int kvm_vm_ioctl_add_msr_allowlist(struct kvm *kvm, void __user *argp) 107c87325cf461 Aaron Lewis 2020-08-18 5182 { 107c87325cf461 Aaron Lewis 2020-08-18 5183 struct msr_bitmap_range *ranges = kvm->arch.msr_allowlist_ranges; 107c87325cf461 Aaron Lewis 2020-08-18 5184 struct kvm_msr_allowlist __user *user_msr_allowlist = argp; 107c87325cf461 Aaron Lewis 2020-08-18 5185 struct msr_bitmap_range range; 107c87325cf461 Aaron Lewis 2020-08-18 5186 struct kvm_msr_allowlist kernel_msr_allowlist; 107c87325cf461 Aaron Lewis 2020-08-18 5187 unsigned long *bitmap = NULL; 107c87325cf461 Aaron Lewis 2020-08-18 5188 size_t bitmap_size; 107c87325cf461 Aaron Lewis 2020-08-18 5189 int r = 0; 107c87325cf461 Aaron Lewis 2020-08-18 5190 107c87325cf461 Aaron Lewis 2020-08-18 5191 if (copy_from_user(&kernel_msr_allowlist, user_msr_allowlist, 107c87325cf461 Aaron Lewis 2020-08-18 5192 sizeof(kernel_msr_allowlist))) { 107c87325cf461 Aaron Lewis 2020-08-18 5193 r = -EFAULT; 107c87325cf461 Aaron Lewis 2020-08-18 5194 goto out; 107c87325cf461 Aaron Lewis 2020-08-18 5195 } 107c87325cf461 Aaron Lewis 2020-08-18 5196 107c87325cf461 Aaron Lewis 2020-08-18 5197 bitmap_size = BITS_TO_LONGS(kernel_msr_allowlist.nmsrs) * sizeof(long); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^nQuoted 3 lines
On 32 bit systems the
BITS_TO_LONGS()can integer overflow if kernel_msr_allowlist.nmsrs is larger thanULONG_MAX-bits_per_long. In that casebitmap_sizeis zero.
Quoted 6 lines
Nice catch! It should be enough to ...
Quoted 2 lines
107c87325cf461Aaron Lewis 2020-08-18 5198 if (bitmap_size>KVM_MSR_ALLOWLIST_MAX_LEN) {... add a check for !bitmap_size here as well then, right?
Yup.
regards,
dan carpenter
PATCH v3 3/12 KVM: selftests: Add test for user space MSR handling
In reply to Aaron LewisNow that we have the ability to handle MSRs from user space and also to select which ones we do want to prevent in-kernel KVM code from handling, let's add a selftest to show case and verify the API.
---
v2 -> v3:
- s/KVM_CAP_ADD_MSR_ALLOWLIST/KVM_CAP_X86_MSR_ALLOWLIST/g
- Add test to clear whitelist
- Adjust to reply-less API
- Fix asserts
- Actually trap on MSR_IA32_POWER_CTL writes
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/x86_64/user_msr_test.c | 221 ++++++++++++++++++ 2 files changed, 222 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86_64/user_msr_test.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -55,6 +55,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/vmx_set_nested_state_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_tsc_adjust_test TEST_GEN_PROGS_x86_64 += x86_64/xss_msr_test TEST_GEN_PROGS_x86_64 += x86_64/debug_regs +TEST_GEN_PROGS_x86_64 += x86_64/user_msr_test TEST_GEN_PROGS_x86_64 += clear_dirty_log_test TEST_GEN_PROGS_x86_64 += demand_paging_test TEST_GEN_PROGS_x86_64 += dirty_log_test
tools/testing/selftests/kvm/x86_64/user_msr_test.c new file +221 -0
221 lines not on this page · raw message
Signature · 1 line
2.28.0.220.ged08abb693-goog
PATCH v3 4/12 KVM: x86: Add ioctl for accepting a userspace provided MSR list
In reply to Aaron LewisAdd KVM_SET_EXIT_MSRS ioctl to allow userspace to pass in a list of MSRs that force an exit to userspace when rdmsr or wrmsr are used by the guest.
KVM_SET_EXIT_MSRS will need to be called before any vCPUs are created to protect the 'user_exit_msrs' list from being mutated while vCPUs are running.
Add KVM_CAP_SET_MSR_EXITS to identify the feature exists.
--- Documentation/virt/kvm/api.rst | 24 +++++++++++++++++++ arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/x86.c | 41 +++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 2 ++ 4 files changed, 69 insertions(+)
Documentation/virt/kvm/api.rst +24 -0
@@ -1010,6 +1010,30 @@ such as migration. :Parameters: struct kvm_vcpu_event (out) :Returns: 0 on success, -1 on error +4.32 KVM_SET_EXIT_MSRS +------------------ + +:Capability: KVM_CAP_SET_MSR_EXITS +:Architectures: x86 +:Type: vm ioctl +:Parameters: struct kvm_msr_list (in) +:Returns: 0 on success, -1 on error + +Sets the userspace MSR list which is used to track which MSRs KVM should send +to userspace to be serviced when the guest executes rdmsr or wrmsr. + +This ioctl needs to be called before vCPUs are setup otherwise the list of MSRs +will not be accepted and an EINVAL error will be returned. Also, if a list of +MSRs has already been supplied, and this ioctl is called again an EEXIST error +will be returned. + +:: + + struct kvm_msr_list { + __u32 nmsrs; + __u32 indices[0]; +}; + X86: ^^^^
arch/x86/include/asm/kvm_host.h +2 -0
@@ -976,6 +976,8 @@ struct kvm_arch {struct kvm_pmu_event_filter *pmu_event_filter;struct task_struct *nx_lpage_recovery_thread; + +struct kvm_msr_list *user_exit_msrs; };struct kvm_vm_stat {
arch/x86/kvm/x86.c +41 -0
59 lines not on this page · raw message
include/uapi/linux/kvm.h +2 -0
@@ -1046,6 +1046,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_S390_DIAG318 186#define KVM_CAP_X86_USER_SPACE_MSR 187#define KVM_CAP_X86_MSR_ALLOWLIST 188 +#define KVM_CAP_SET_MSR_EXITS 189#ifdef KVM_CAP_IRQ_ROUTING
@@ -1386,6 +1387,7 @@ struct kvm_s390_ucas_mapping {/* Available with KVM_CAP_PMU_EVENT_FILTER */ #define KVM_SET_PMU_EVENT_FILTER _IOW(KVMIO, 0xb2, struct kvm_pmu_event_filter)#define KVM_PPC_SVM_OFF _IO(KVMIO, 0xb3) +#define KVM_SET_EXIT_MSRS _IOW(KVMIO, 0xb4, struct kvm_msr_list)/* ioctl for vm fd */ #define KVM_CREATE_DEVICE _IOWR(KVMIO, 0xe0, struct kvm_create_device)
Signature · 1 line
2.28.0.220.ged08abb693-goog
Re: [PATCH v3 04/12] KVM: x86: Add ioctl for accepting a userspace provided MSR list
Quoted 5 lines · Aaron Lewis
Add
KVM_SET_EXIT_MSRSioctl to allow userspace to pass in a list of MSRs that force an exit to userspace when rdmsr or wrmsr are used by the guest.
Quoted 8 lines
KVM_SET_EXIT_MSRSwill need to be called before any vCPUs are created to protect the 'user_exit_msrs' list from being mutated while vCPUs are running.Add
KVM_CAP_SET_MSR_EXITSto identify the feature exists.Signed-off-by Aaron Lewis <[email protected]>Reviewed-by Oliver Upton <[email protected]>
Why would we still need this with the allow list and user space #GP deflection logic in place?
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 04/12] KVM: x86: Add ioctl for accepting a userspace provided MSR list
Quoted 2 lines · Alexander Graf
Why would we still need this with the allow list and user space #GP deflection logic in place?
Conversion to an allow list is cumbersome when you have a short deny list. Suppose that I want to implement the following deny list:
{IA32_ARCH_CAPABILITIES, HV_X64_MSR_REFERENCE_TSC, MSR_GOOGLE_TRUE_TIME, MSR_GOOGLE_FDR_TRACE, MSR_GOOGLE_HBI}. What would the corresponding deny list look like? Given your current implementation, I don't think the corresponding allow list can actually be constructed. I want to allow 2^32-5 MSRs, but I can allow at most 122880, if I've done the math correctly. (10 ranges, each spanning at most 0x600 bytes worth of bitmap.)
Perhaps we should adopt allow/deny rules similar to those accepted by most firewalls. Instead of ports, we have MSR indices. Instead of protocols, we have READ, WRITE, or READ/WRITE. Suppose that we supported up to <n> rules of the form: {start index, end index, access modes, allow or deny}? Rules would be processed in the order given, and the first rule that matched a given access would take precedence. Finally, userspace could specify the default behavior (either allow or deny) for any MSR access that didn't match any of the rules.
Thoughts?
Re: [PATCH v3 04/12] KVM: x86: Add ioctl for accepting a userspace provided MSR list
Hi Jim,
Quoted 7 lines · Jim Mattson
Quoted 2 lines · Alexander Graf
Why would we still need this with the allow list and user space #GP deflection logic in place?
Quoted 9 lines
Conversion to an allow list is cumbersome when you have a short deny list. Suppose that I want to implement the following deny list:
{IA32_ARCH_CAPABILITIES,HV_X64_MSR_REFERENCE_TSC,MSR_GOOGLE_TRUE_TIME,MSR_GOOGLE_FDR_TRACE,MSR_GOOGLE_HBI}. What would the corresponding deny list look like? Given your current implementation, I don't think the corresponding allow list can actually be constructed. I want to allow 2^32-5 MSRs, but I can allow at most 122880, if I've done the math correctly. (10 ranges, each spanning at most0x600bytes worth of bitmap.)
There are only very few MSR ranges that actually data. So in your case, to allow all MSRs that Linux knows about in msr-index.h, you would need
[0x00000000 - 0x00002000]
[0x40000000 - 0x40000200]
[0x4b564d00 - 0x4b564e00]
[0x80868000 - 0x80868020]
[0xc0000000 - 0xc0000200]
[0xc0010000 - 0xc0012000]
[0xc0020000 - 0xc0020010]
which are 7 regions. For good measure, you can probably pad every one of them to the full 0x3000 MSRs they can span.
For MSRs that KVM actually handles in-kernel (others don't need to be allowed), the list shrinks to 5:
[0x00000000 - 0x00001000]
[0x40000000 - 0x40000200]
[0x4b564d00 - 0x4b564e00]
[0xc0000000 - 0xc0000200]
[0xc0010000 - 0xc0012000]
Let's extend them a bit to make reasoning easier:
[0x00000000 - 0x00003000]
[0x40000000 - 0x40003000]
[0x4b564d00 - 0x4b567000]
[0xc0000000 - 0xc0003000]
[0xc0010000 - 0xc0013000]
What are the odds that you will want to implicitly (without a new CAP, that would need user space adjustments anyway) have a random new MSR handled in-kernel with an identifier that is outside of those ranges?
I'm fairly confident that trends towards 0.
The only real downside I can see is that we just wasted ~8kb of RAM. Nothing I would really get hung up on though.
Quoted 10 lines
Perhaps we should adopt allow/deny rules similar to those accepted by most firewalls. Instead of ports, we have MSR indices. Instead of protocols, we have READ, WRITE, or READ/WRITE. Suppose that we supported up to <n> rules of the form: {start index, end index, access modes, allow or deny}? Rules would be processed in the order given, and the first rule that matched a given access would take precedence. Finally, userspace could specify the default behavior (either allow or deny) for any MSR access that didn't match any of the rules.
Thoughts?
That wouldn't scale well if you want to allow all architecturally useful MSRs in a purely allow list fashion. You'd have to create hundreds of rules - or at least a few dozen if you combine contiguous ranges.
If you really desperately believe a deny list is a better fit for your use case, we could redesign the interface differently:
struct msr_set_accesslist {
#define MSR_ACCESSLIST_DEFAULT_ALLOW 0
#define MSR_ACCESSLIST_DEFAULT_DENY 1
u32 flags;
struct {
u32 flags;
u32 nmsrs; /* MSRs in bitmap */
u32 base; /* first MSR address to bitmap */
void *bitmap; /* pointer to bitmap, 1 means allow, 0 deny */
} lists[10];
};which means in your use case, you can do
u64 deny = 0;
struct msr_set_accesslist access = {
.flags = MSR_ACCESSLIST_DEFAULT_ALLOW,
.lists = {
{
.nmsrs = 1,
.base = IA32_ARCH_CAPABILITIES,
.bitmap = &deny,
}, {
{
.nmsrs = 1,
.base = HV_X64_MSR_REFERENCE_TSC,
.bitmap = &deny,
}, {
{
.nmsrs = 1,
/* can probably be combined with the ones below? */
.base = MSR_GOOGLE_TRUE_TIME,
.bitmap = &deny,
}, {
{
.nmsrs = 1,
.base = MSR_GOOGLE_FDR_TRACE,
.bitmap = &deny,
}, {
{
.nmsrs = 1,
.base = MSR_GOOGLE_HBI,
.bitmap = &deny,
},
}
};msr_set_accesslist(kvm_fd, &access);
while I can do the same dance as before, but with a single call rather than multiple ones.
What do you think?
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 04/12] KVM: x86: Add ioctl for accepting a userspace provided MSR list
Quoted 2 lines · Alexander Graf
The only real downside I can see is that we just wasted ~8kb of RAM. Nothing I would really get hung up on though.
I also suspect that the MSR permission bitmap modifications are going to be a bit more expensive with 4kb (6kb on AMD) of pertinent allow-bitmaps than they would be with a few bytes of pertinent deny-bitmaps.
Quoted 50 lines
If you really desperately believe a deny list is a better fit for your use case, we could redesign the interface differently:
struct
msr_set_accesslist{
#defineMSR_ACCESSLIST_DEFAULT_ALLOW0#define MSR_ACCESSLIST_DEFAULT_DENY 1 u32 flags; struct { u32 flags; u32 nmsrs; /* MSRs in bitmap */ u32 base; /* first MSR address to bitmap */ void *bitmap; /* pointer to bitmap, 1 means allow, 0 deny */ } lists[10]; };which means in your use case, you can do
u64 deny = 0;
struct msr_set_accesslist access = { .flags = MSR_ACCESSLIST_DEFAULT_ALLOW, .lists = { { .nmsrs = 1, .base = IA32_ARCH_CAPABILITIES, .bitmap = &deny, }, { { .nmsrs = 1, .base = HV_X64_MSR_REFERENCE_TSC, .bitmap = &deny, }, { { .nmsrs = 1, /* can probably be combined with the ones below? */ .base = MSR_GOOGLE_TRUE_TIME, .bitmap = &deny, }, { { .nmsrs = 1, .base = MSR_GOOGLE_FDR_TRACE, .bitmap = &deny, }, { { .nmsrs = 1, .base = MSR_GOOGLE_HBI, .bitmap = &deny, }, } };
Quoted 6 lines
msr_set_accesslist(kvm_fd, &access);while I can do the same dance as before, but with a single call rather than multiple ones.
What do you think?
I like it. I think this suits our use case well.
PATCH v3 5/12 KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
In reply to Aaron LewisAdd support for exiting to userspace on a rdmsr or wrmsr instruction if the MSR being read from or written to is in the user_exit_msrs list.
---
v2 -> v3
- Refactored commit based on Alexander Graf's changes in the first commit in this series. Changes made were:
- Updated member 'inject_gp' to 'error' based on struct msr in kvm_run.
- Move flag 'vcpu->kvm->arch.user_space_msr_enabled' out of kvm_msr_user_space() to allow it to work with both methods that bounce to userspace (msr list and #GP fallback). Updated caller functions to account for this change.
- trace_kvm_msr has been moved up and combine with a previous call in complete_emulated_msr() based on the suggestion made by Alexander Graf <[email protected]>.
--- arch/x86/kvm/trace.h | 24 ++++++++++++++ arch/x86/kvm/x86.c | 76 ++++++++++++++++++++++++++++++++++++++------ 2 files changed, 90 insertions(+), 10 deletions(-)
arch/x86/kvm/trace.h +24 -0
@@ -367,6 +367,30 @@ TRACE_EVENT(kvm_msr,#define trace_kvm_msr_read_ex (ecx) trace_kvm_msr(0, ecx, 0, true)#define trace_kvm_msr_write_ex (ecx, data) trace_kvm_msr(1, ecx, data, true) +TRACE_EVENT (kvm_userspace_msr, + TP_PROTO(bool is_write, u8 error, u32 index, u64 data), +TP_ARGS (is_write, error, index, data), + +TP_STRUCT__entry ( + __field(bool, is_write) + __field(u8, error) + __field(u32, index) +__field (u64, data) + ), + +TP_fast_assign ( + __entry->is_write = is_write; + __entry->error = error; + __entry->index = index; + __entry->data = data; + ), + +TP_printk ("userspace %s %x = 0x%llx, %s" , + __entry->is_write ?"wrmsr" :"rdmsr" , + __entry->index, __entry->data, + __entry->error ?"error" :"no_error" ) +); +/* * Tracepoint for guest CR access. */
arch/x86/kvm/x86.c +66 -10
139 lines not on this page · raw message
Signature · 1 line
2.28.0.220.ged08abb693-goog
Re: [PATCH v3 05/12] KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
Quoted 5 lines · Aaron Lewis
Add support for exiting to userspace on a rdmsr or wrmsr instruction if the MSR being read from or written to is in the
user_exit_msrslist.Signed-off-by Aaron Lewis <[email protected]>
Again, this patch should be redundant with the allow list?
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 05/12] KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
In reply to Aaron LewisQuoted 42 lines · Aaron Lewis
Add support for exiting to userspace on a rdmsr or wrmsr instruction if the MSR being read from or written to is in the
user_exit_msrslist.Signed-off-by Aaron Lewis <[email protected]>---
v2 -> v3
- Refactored commit based on Alexander Graf's changes in the first commit in this series. Changes made were:
- Updated member 'inject_gp' to 'error' based on struct msr inkvm_run.
- Move flag 'vcpu->kvm->arch.user_space_msr_enabled' out ofkvm_msr_user_space()to allow it to work with both methods that bounce to userspace (msr list and #GP fallback). Updated caller functions to account for this change.
-trace_kvm_msrhas been moved up and combine with a previous call incomplete_emulated_msr()based on the suggestion made by Alexander Graf <[email protected]>.---
arch/x86/kvm/x86.c +6
-3@@ -1653,9 +1663,6 @@ static int kvm_msr_user_space(struct kvm_vcpu *vcpu, u32 index, u32 exit_reason, u64 data, int (*completion)(struct kvm_vcpu *vcpu)) { - if (!vcpu->kvm->arch.user_space_msr_enabled) - return 0; - vcpu->run->exit_reason = exit_reason; vcpu->run->msr.error =0 ; vcpu->run->msr.pad[0 ] =0 ;@@ -1686,10 +1693,18 @@ int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu)u64 data;int r; +if (kvm_msr_user_exit (vcpu->kvm, ecx)) { +kvm_get_msr_user_space (vcpu, ecx); +/* Bounce to user space */ +return 0 ; + } +
arch/x86/kvm/x86.c int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu) · line 1710
Quoted 10 lines
arch/x86/kvm/x86.c +3
-1+ r =kvm_get_msr (vcpu, ecx, &data);/* MSR read failed? See if we should ask user space */ -if (r &&kvm_get_msr_user_space (vcpu, ecx)) { +if (r && vcpu->kvm->arch.user_space_msr_enabled) { +kvm_get_msr_user_space (vcpu, ecx);/* Bounce to user space */ return 0 ; }
The before and after bounce to userspace is unfortunate. If we can consolidate the allow/deny list checks at the top of kvm_get_msr(), and we can tell why kvm_get_msr() failed (e.g. -EPERM=disallowed, -ENOENT=unknown MSR, -EINVAL=illegal access), then we can eliminate the first bounce to userspace above. -EPERM would always go to userspace. -ENOENT would go to userspace if userspace asked to handle unknown MSRs. -EINVAL would go to userspace if userspace asked to handle all #GPs. (Yes; I'd still like to be able to distinguish between "unknown MSR" and "illegal value." Otherwise, it seems impossible for userspace to know how to proceed.)
(You may ask, "why would you get -EINVAL on a RDMSR?" This would be the case if you tried to read a write-only MSR, like IA32_FLUSH_CMD.)
Quoted 9 lines
arch/x86/kvm/x86.c +5
-0@@ -1715,10 +1730,17 @@ int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu)u64 data =kvm_read_edx_eax (vcpu);int r; +if (kvm_msr_user_exit (vcpu->kvm, ecx)) { +kvm_set_msr_user_space (vcpu, ecx, data); +/* Bounce to user space */ +return 0 ; + }
arch/x86/kvm/x86.c int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu) · line 1746
Quoted 10 lines
arch/x86/kvm/x86.c +3
-1+ r =kvm_set_msr (vcpu, ecx, data);/* MSR write failed? See if we should ask user space */ -if (r &&kvm_set_msr_user_space (vcpu, ecx, data)) { +if (r && vcpu->kvm->arch.user_space_msr_enabled) { +kvm_set_msr_user_space (vcpu, ecx, data);/* Bounce to user space */ return 0 ; }
Same idea as above.
Quoted 12 lines
arch/x86/kvm/x86.c +8
-0@@ -3606,6 +3628,25 @@ static int kvm_vm_ioctl_set_exit_msrs(struct kvm *kvm,return 0 ; } +bool kvm_msr_user_exit (struct kvm *kvm,u32 index) +{ +struct kvm_msr_list *exit_msrs; +int i; + + exit_msrs = kvm->arch.user_exit_msrs; + +if (!exit_msrs)
arch/x86/kvm/x86.c static int kvm_vm_ioctl_set_exit_msrs(struct kvm *kvm, · line 3648
Quoted 10 lines
arch/x86/kvm/x86.c +10
-0+return false; + +for (i =0 ; i < exit_msrs->nmsrs; ++i) { +if (exit_msrs->indices[i] == index) +return true; + } + +return false; +} +EXPORT_SYMBOL_GPL (kvm_msr_user_exit);
I think this should probably be scrapped, along with Alexander's allow-list check, in favor of a rule-based allow/deny list approach as I outlined in an earlier message today.
Re: [PATCH v3 05/12] KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
Quoted 66 lines · Jim Mattson
Quoted 42 lines · Aaron Lewis
Add support for exiting to userspace on a rdmsr or wrmsr instruction if the MSR being read from or written to is in the
user_exit_msrslist.Signed-off-by Aaron Lewis <[email protected]>---
v2 -> v3
- Refactored commit based on Alexander Graf's changes in the first commit in this series. Changes made were:
- Updated member 'inject_gp' to 'error' based on struct msr inkvm_run.
- Move flag 'vcpu->kvm->arch.user_space_msr_enabled' out ofkvm_msr_user_space()to allow it to work with both methods that bounce to userspace (msr list and #GP fallback). Updated caller functions to account for this change.
-trace_kvm_msrhas been moved up and combine with a previous call incomplete_emulated_msr()based on the suggestion made by Alexander Graf <[email protected]>.---
arch/x86/kvm/x86.c +6
-3@@ -1653,9 +1663,6 @@ static int kvm_msr_user_space(struct kvm_vcpu *vcpu, u32 index, u32 exit_reason, u64 data, int (*completion)(struct kvm_vcpu *vcpu)) { - if (!vcpu->kvm->arch.user_space_msr_enabled) - return 0; - vcpu->run->exit_reason = exit_reason; vcpu->run->msr.error =0 ; vcpu->run->msr.pad[0 ] =0 ;@@ -1686,10 +1693,18 @@ int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu)u64 data;int r; +if (kvm_msr_user_exit (vcpu->kvm, ecx)) { +kvm_get_msr_user_space (vcpu, ecx); +/* Bounce to user space */ +return 0 ; + } +arch/x86/kvm/x86.c int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu) · line 1710
Quoted 10 lines
arch/x86/kvm/x86.c +3
-1+ r =kvm_get_msr (vcpu, ecx, &data);/* MSR read failed? See if we should ask user space */ -if (r &&kvm_get_msr_user_space (vcpu, ecx)) { +if (r && vcpu->kvm->arch.user_space_msr_enabled) { +kvm_get_msr_user_space (vcpu, ecx);/* Bounce to user space */ return 0 ; }The before and after bounce to userspace is unfortunate. If we can consolidate the allow/deny list checks at the top of
kvm_get_msr(), and we can tell whykvm_get_msr()failed (e.g. -EPERM=disallowed, -ENOENT=unknown MSR, -EINVAL=illegal access), then we can eliminate the first bounce to userspace above.-EPERMwould always go to userspace.-ENOENTwould go to userspace if userspace asked to handle unknown MSRs.-EINVALwould go to userspace if userspace asked to handle all #GPs. (Yes; I'd still like to be able to distinguish between "unknown MSR" and "illegal value." Otherwise, it seems impossible for userspace to know how to proceed.)
Quoted 2 lines
(You may ask, "why would you get
-EINVALon a RDMSR?" This would be the case if you tried to read a write-only MSR, likeIA32_FLUSH_CMD.)
Do we really need to do all of this dance of differentiating in kernel space between an exit that's there because user space asked for the exit and an MSR access that would just generate a #GP?
At the end of the day, user space *knows* which MSRs it asked to receive. It can filter for them super easily.
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 05/12] KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
Quoted 6 lines · Alexander Graf
Do we really need to do all of this dance of differentiating in kernel space between an exit that's there because user space asked for the exit and an MSR access that would just generate a #GP?
At the end of the day, user space *knows* which MSRs it asked to receive. It can filter for them super easily.
If no one else has an opinion, I can let this go. :-)
However, to make the right decision in kvm_emulate_{rdmsr,wrmsr} (without the unfortunate before and after checks that Aaron added), kvm_{get,set}_msr should at least distinguish between "permission denied" and "raise #GP," so I can provide a deny list without asking for userspace exits on #GP.
Re: [PATCH v3 05/12] KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
Quoted 10 lines · Jim Mattson
Quoted 6 lines · Alexander Graf
Do we really need to do all of this dance of differentiating in kernel space between an exit that's there because user space asked for the exit and an MSR access that would just generate a #GP?
At the end of the day, user space *knows* which MSRs it asked to receive. It can filter for them super easily.
Quoted 7 lines
If no one else has an opinion, I can let this go. :-)
However, to make the right decision in kvm_emulate_{rdmsr,wrmsr} (without the unfortunate before and after checks that Aaron added), kvm_{get,set}_msr should at least distinguish between "permission denied" and "raise #GP," so I can provide a deny list without asking for userspace exits on #GP.
Actually, I think this whole discussion is moot. You no longer need the first ioctl (ask for a userspace exit on #GP). The allow/deny list is sufficient. Moreover, the allow/deny list checks can be in kvm_emulate_{rdmsr,wrmsr} before the call to kvm_{get,set}_msr, so we needn't be concerned with distinguishable error values either.
Re: [PATCH v3 05/12] KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
Quoted 20 lines · Jim Mattson
Quoted 10 lines · Jim Mattson
Quoted 6 lines · Alexander Graf
Do we really need to do all of this dance of differentiating in kernel space between an exit that's there because user space asked for the exit and an MSR access that would just generate a #GP?
At the end of the day, user space *knows* which MSRs it asked to receive. It can filter for them super easily.
Quoted 7 lines
If no one else has an opinion, I can let this go. :-)
However, to make the right decision in kvm_emulate_{rdmsr,wrmsr} (without the unfortunate before and after checks that Aaron added), kvm_{get,set}_msr should at least distinguish between "permission denied" and "raise #GP," so I can provide a deny list without asking for userspace exits on #GP.
Quoted 6 lines
Actually, I think this whole discussion is moot. You no longer need the first ioctl (ask for a userspace exit on #GP). The allow/deny list is sufficient. Moreover, the allow/deny list checks can be in kvm_emulate_{rdmsr,wrmsr} before the call to kvm_{get,set}_msr, so we needn't be concerned with distinguishable error values either.
I also care about cases where I allow in-kernel handling, but for whatever reason there still would be a #GP injected into the guest. I want to record those events and be able to later have data that tell me why something went wrong.
So yes, for your use case you do not care about the distinction between "deny MSR access" and "report invalid MSR access". However, I do care :).
My stance on this is again that it's trivial to handle a few invalid MSR #GPs from user space and just not report anything. It should come at almost negligible performance cost, no?
As for your argumentation above, we have a second call chain into kvm_{get,set}_msr from the x86 emulator which you'd also need to cover.
One thing we could do I guess is to add a parameter to ENABLE_CAP on KVM_CAP_X86_USER_SPACE_MSR so that it only bounces on certain return values, such as -ENOENT. I still fail to see cases where that's genuinely beneficial though.
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 05/12] KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
Quoted 31 lines · Alexander Graf
Quoted 20 lines · Jim Mattson
Quoted 10 lines · Jim Mattson
Quoted 6 lines · Alexander Graf
Do we really need to do all of this dance of differentiating in kernel space between an exit that's there because user space asked for the exit and an MSR access that would just generate a #GP?
At the end of the day, user space *knows* which MSRs it asked to receive. It can filter for them super easily.
Quoted 7 lines
If no one else has an opinion, I can let this go. :-)
However, to make the right decision in kvm_emulate_{rdmsr,wrmsr} (without the unfortunate before and after checks that Aaron added), kvm_{get,set}_msr should at least distinguish between "permission denied" and "raise #GP," so I can provide a deny list without asking for userspace exits on #GP.
Quoted 6 lines
Actually, I think this whole discussion is moot. You no longer need the first ioctl (ask for a userspace exit on #GP). The allow/deny list is sufficient. Moreover, the allow/deny list checks can be in kvm_emulate_{rdmsr,wrmsr} before the call to kvm_{get,set}_msr, so we needn't be concerned with distinguishable error values either.
Quoted 7 lines
I also care about cases where I allow in-kernel handling, but for whatever reason there still would be a #GP injected into the guest. I want to record those events and be able to later have data that tell me why something went wrong.
So yes, for your use case you do not care about the distinction between "deny MSR access" and "report invalid MSR access". However, I do care :).
In that case, I'm going to continue to hold a hard line on the distinction between a #GP for an invalid MSR access and the #GP for an unknown MSR. If, for instance, you wanted to implement ignore_msrs in userspace, as you've proposed in the past, this would be extremely helpful. Without it, userspace gets an exit because (1) the MSR access isn't in the allow list, (2) the MSR access is invalid, or (3) the MSR is unknown to kvm. As you've pointed out, it is easy for userspace to distinguish (1) from the others, since it provided the allow/deny list in the first place. But how do you distinguish (2) from (3) without replicating the logic in the kernel?
Quoted 3 lines
My stance on this is again that it's trivial to handle a few invalid MSR #GPs from user space and just not report anything. It should come at almost negligible performance cost, no?
Yes, the performance cost should be negligible, but what is the point? We're trying to design a good API here, aren't we?
Quoted 7 lines
As for your argumentation above, we have a second call chain into kvm_{get,set}_msr from the x86 emulator which you'd also need to cover.
One thing we could do I guess is to add a parameter to
ENABLE_CAPonKVM_CAP_X86_USER_SPACE_MSRso that it only bounces on certain return values, such as-ENOENT. I still fail to see cases where that's genuinely beneficial though.
I'd like to see two completely independent APIs, so that I can just request a bounce on -EPERM through a deny list. I think it's useful to distinguish between -ENOENT and -EINVAL, but I have no issues wih both causing an exit to userspace, if userspace has requested exits on MSR #GPs.
Re: [PATCH v3 05/12] KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
Quoted 62 lines · Jim Mattson
Quoted 31 lines · Alexander Graf
Quoted 20 lines · Jim Mattson
Quoted 10 lines · Jim Mattson
Quoted 6 lines · Alexander Graf
Do we really need to do all of this dance of differentiating in kernel space between an exit that's there because user space asked for the exit and an MSR access that would just generate a #GP?
At the end of the day, user space *knows* which MSRs it asked to receive. It can filter for them super easily.
Quoted 7 lines
If no one else has an opinion, I can let this go. :-)
However, to make the right decision in kvm_emulate_{rdmsr,wrmsr} (without the unfortunate before and after checks that Aaron added), kvm_{get,set}_msr should at least distinguish between "permission denied" and "raise #GP," so I can provide a deny list without asking for userspace exits on #GP.
Quoted 6 lines
Actually, I think this whole discussion is moot. You no longer need the first ioctl (ask for a userspace exit on #GP). The allow/deny list is sufficient. Moreover, the allow/deny list checks can be in kvm_emulate_{rdmsr,wrmsr} before the call to kvm_{get,set}_msr, so we needn't be concerned with distinguishable error values either.
Quoted 7 lines
I also care about cases where I allow in-kernel handling, but for whatever reason there still would be a #GP injected into the guest. I want to record those events and be able to later have data that tell me why something went wrong.
So yes, for your use case you do not care about the distinction between "deny MSR access" and "report invalid MSR access". However, I do care :).
In that case, I'm going to continue to hold a hard line on the distinction between a #GP for an invalid MSR access and the #GP for an unknown MSR. If, for instance, you wanted to implement
ignore_msrsin userspace, as you've proposed in the past, this would be extremely helpful. Without it, userspace gets an exit because (1) the MSR access isn't in the allow list, (2) the MSR access is invalid, or (3) the MSR is unknown to kvm. As you've pointed out, it is easy for userspace to distinguish (1) from the others, since it provided the allow/deny list in the first place. But how do you distinguish (2) from (3) without replicating the logic in the kernel?Quoted 3 lines
My stance on this is again that it's trivial to handle a few invalid MSR #GPs from user space and just not report anything. It should come at almost negligible performance cost, no?
Yes, the performance cost should be negligible, but what is the point? We're trying to design a good API here, aren't we?
Quoted 3 lines
As for your argumentation above, we have a second call chain into kvm_{get,set}_msr from the x86 emulator which you'd also need to cover.
Quoted 7 lines
Quoted 4 lines
One thing we could do I guess is to add a parameter to
ENABLE_CAPonKVM_CAP_X86_USER_SPACE_MSRso that it only bounces on certain return values, such as-ENOENT. I still fail to see cases where that's genuinely beneficial though.I'd like to see two completely independent APIs, so that I can just request a bounce on
-EPERMthrough a deny list. I think it's useful
Where would that bounce to? Which user space event does that trigger? Yet another one? Wouldn't 4 exit reasons just for MSR traps be a bit much? :)
Quoted 3 lines
to distinguish between
-ENOENTand-EINVAL, but I have no issues wih both causing an exit to userspace, if userspace has requested exits on MSR #GPs.
So imagine we took the first argument to ENABLE_CAP as filter:
(1<<0) REPORT_ENOENT
(1<<1) REPORT_EINVAL
(1<<2) REPORT_EPERM
(1<<31) REPORT_ANY
Then we also add the reason to the kvm_run exit response and user space can differentiate easily between the different events.
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 05/12] KVM: x86: Add support for exiting to userspace on rdmsr or wrmsr
Quoted 71 lines · Alexander Graf
Quoted 67 lines · Jim Mattson
Quoted 31 lines · Alexander Graf
Quoted 20 lines · Jim Mattson
Quoted 10 lines · Jim Mattson
Quoted 6 lines · Alexander Graf
Do we really need to do all of this dance of differentiating in kernel space between an exit that's there because user space asked for the exit and an MSR access that would just generate a #GP?
At the end of the day, user space *knows* which MSRs it asked to receive. It can filter for them super easily.
Quoted 7 lines
If no one else has an opinion, I can let this go. :-)
However, to make the right decision in kvm_emulate_{rdmsr,wrmsr} (without the unfortunate before and after checks that Aaron added), kvm_{get,set}_msr should at least distinguish between "permission denied" and "raise #GP," so I can provide a deny list without asking for userspace exits on #GP.
Quoted 6 lines
Actually, I think this whole discussion is moot. You no longer need the first ioctl (ask for a userspace exit on #GP). The allow/deny list is sufficient. Moreover, the allow/deny list checks can be in kvm_emulate_{rdmsr,wrmsr} before the call to kvm_{get,set}_msr, so we needn't be concerned with distinguishable error values either.
Quoted 7 lines
I also care about cases where I allow in-kernel handling, but for whatever reason there still would be a #GP injected into the guest. I want to record those events and be able to later have data that tell me why something went wrong.
So yes, for your use case you do not care about the distinction between "deny MSR access" and "report invalid MSR access". However, I do care :).
In that case, I'm going to continue to hold a hard line on the distinction between a #GP for an invalid MSR access and the #GP for an unknown MSR. If, for instance, you wanted to implement
ignore_msrsin userspace, as you've proposed in the past, this would be extremely helpful. Without it, userspace gets an exit because (1) the MSR access isn't in the allow list, (2) the MSR access is invalid, or (3) the MSR is unknown to kvm. As you've pointed out, it is easy for userspace to distinguish (1) from the others, since it provided the allow/deny list in the first place. But how do you distinguish (2) from (3) without replicating the logic in the kernel?Quoted 3 lines
My stance on this is again that it's trivial to handle a few invalid MSR #GPs from user space and just not report anything. It should come at almost negligible performance cost, no?
Yes, the performance cost should be negligible, but what is the point? We're trying to design a good API here, aren't we?
Quoted 7 lines
As for your argumentation above, we have a second call chain into kvm_{get,set}_msr from the x86 emulator which you'd also need to cover.
One thing we could do I guess is to add a parameter to
ENABLE_CAPonKVM_CAP_X86_USER_SPACE_MSRso that it only bounces on certain return values, such as-ENOENT. I still fail to see cases where that's genuinely beneficial though.
Quoted 6 lines
Quoted 2 lines
I'd like to see two completely independent APIs, so that I can just request a bounce on
-EPERMthrough a deny list. I think it's usefulWhere would that bounce to? Which user space event does that trigger? Yet another one? Wouldn't 4 exit reasons just for MSR traps be a bit much? :)
All of the exits are either KVM_EXIT_X86_RDMSR or KVM_EXIT_X86_WRMSR. Or, we could put the direction in the msr struct and just have one exit reason.
Quoted 6 lines
Quoted 3 lines
to distinguish between
-ENOENTand-EINVAL, but I have no issues wih both causing an exit to userspace, if userspace has requested exits on MSR #GPs.So imagine we took the first argument to
ENABLE_CAPas filter:
Quoted 7 lines
(1<<0) REPORT_ENOENT (1<<1) REPORT_EINVAL (1<<2) REPORT_EPERM (1<<31) REPORT_ANYThen we also add the reason to the
kvm_runexit response and user space can differentiate easily between the different events.
I think this works well. I still have to call both APIs to satisfy my use case, but I'm willing to cave on that request. (I just realized that there is a very good use case for an allow/deny list *without* exits to userspace: prohibiting kvm from doing cross-vendor MSR emulation.)
PATCH v3 6/12 KVM: x86: Prepare MSR bitmaps for userspace tracked MSRs
In reply to Aaron LewisPrepare vmx and svm for a subsequent change that ensures the MSR permission bitmap is set to allow an MSR that userspace is tracking to force a vmx_vmexit in the guest.
--- arch/x86/kvm/svm/svm.c | 48 +++++++++++----------- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/vmx.c | 83 +++++++++++++++++++-------------------- arch/x86/kvm/vmx/vmx.h | 2 +- 4 files changed, 67 insertions(+), 68 deletions(-)
arch/x86/kvm/svm/svm.c +25 -23
@@ -564,7 +564,7 @@ static bool valid_msr_intercept(u32 index)return false; } -static bool msr_write_intercepted (struct kvm_vcpu *vcpu,unsigned msr) +static bool msr_write_intercepted (struct kvm_vcpu *vcpu,u32 msr) {u8 bit_write;unsigned long tmp;
@@ -583,9 +583,11 @@ static bool msr_write_intercepted(struct kvm_vcpu *vcpu, unsigned msr)return !!test_bit (bit_write, &tmp); } -static void set_msr_interception (u32 *msrpm,unsigned msr, -int read,int write) +static void set_msr_interception (struct kvm_vcpu *vcpu,u32 msr,int read, +int write) { +struct vcpu_svm *svm =to_svm (vcpu); +u32 *msrpm = svm->msrpm;u8 bit_read, bit_write;unsigned long tmp;u32 offset;
@@ -609,7 +611,7 @@ static void set_msr_interception(u32 *msrpm, unsigned msr, msrpm[offset] = tmp; } -static void svm_vcpu_init_msrpm (u32 *msrpm) +static void svm_vcpu_init_msrpm (struct kvm_vcpu *vcpu,u32 *msrpm) {int i;
@@ -619,7 +621,7 @@ static void svm_vcpu_init_msrpm(u32 *msrpm)if (!direct_access_msrs[i].always)continue ; -set_msr_interception (msrpm, direct_access_msrs[i].index,1 ,1 ); +set_msr_interception (vcpu, direct_access_msrs[i].index,1 ,1 ); } }
@@ -666,26 +668,26 @@ static void init_msrpm_offsets(void) } } -static void svm_enable_lbrv (struct vcpu_svm *svm) +static void svm_enable_lbrv (struct kvm_vcpu *vcpu) { -u32 *msrpm = svm->msrpm; +struct vcpu_svm *svm =to_svm (vcpu); svm->vmcb->control.virt_ext |=LBR_CTL_ENABLE_MASK ; -set_msr_interception (msrpm,MSR_IA32_LASTBRANCHFROMIP ,1 ,1 ); -set_msr_interception (msrpm,MSR_IA32_LASTBRANCHTOIP ,1 ,1 ); -set_msr_interception (msrpm,MSR_IA32_LASTINTFROMIP ,1 ,1 ); -set_msr_interception (msrpm,MSR_IA32_LASTINTTOIP ,1 ,1 ); +set_msr_interception (vcpu,MSR_IA32_LASTBRANCHFROMIP ,1 ,1 ); +set_msr_interception (vcpu,MSR_IA32_LASTBRANCHTOIP ,1 ,1 ); +set_msr_interception (vcpu,MSR_IA32_LASTINTFROMIP ,1 ,1 ); +set_msr_interception (vcpu,MSR_IA32_LASTINTTOIP ,1 ,1 ); } -static void svm_disable_lbrv (struct vcpu_svm *svm) +static void svm_disable_lbrv (struct kvm_vcpu *vcpu) { -u32 *msrpm = svm->msrpm; +struct vcpu_svm *svm =to_svm (vcpu); svm->vmcb->control.virt_ext &= ~LBR_CTL_ENABLE_MASK ; -set_msr_interception (msrpm,MSR_IA32_LASTBRANCHFROMIP ,0 ,0 ); -set_msr_interception (msrpm,MSR_IA32_LASTBRANCHTOIP ,0 ,0 ); -set_msr_interception (msrpm,MSR_IA32_LASTINTFROMIP ,0 ,0 ); -set_msr_interception (msrpm,MSR_IA32_LASTINTTOIP ,0 ,0 ); +set_msr_interception (vcpu,MSR_IA32_LASTBRANCHFROMIP ,0 ,0 ); +set_msr_interception (vcpu,MSR_IA32_LASTBRANCHTOIP ,0 ,0 ); +set_msr_interception (vcpu,MSR_IA32_LASTINTFROMIP ,0 ,0 ); +set_msr_interception (vcpu,MSR_IA32_LASTINTTOIP ,0 ,0 ); }void disable_nmi_singlestep (struct vcpu_svm *svm)
@@ -1211,10 +1213,10 @@ static int svm_create_vcpu(struct kvm_vcpu *vcpu)clear_page (svm->nested.hsave); svm->msrpm =page_address (msrpm_pages); -svm_vcpu_init_msrpm (svm->msrpm); +svm_vcpu_init_msrpm (vcpu, svm->msrpm); svm->nested.msrpm =page_address (nested_msrpm_pages); -svm_vcpu_init_msrpm (svm->nested.msrpm); +svm_vcpu_init_msrpm (vcpu, svm->nested.msrpm); svm->vmcb =page_address (page);clear_page (svm->vmcb);
@@ -2556,7 +2558,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)* We update the L1 MSR bit as well since it will end up * touching the MSR anyway now. */ -set_msr_interception (svm->msrpm,MSR_IA32_SPEC_CTRL ,1 ,1 ); +set_msr_interception (vcpu,MSR_IA32_SPEC_CTRL ,1 ,1 );break ;case MSR_IA32_PRED_CMD : if (!msr->host_initiated &&
@@ -2571,7 +2573,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)break ;wrmsrl (MSR_IA32_PRED_CMD ,PRED_CMD_IBPB ); -set_msr_interception (svm->msrpm,MSR_IA32_PRED_CMD ,0 ,1 ); +set_msr_interception (vcpu,MSR_IA32_PRED_CMD ,0 ,1 );break ;case MSR_AMD64_VIRT_SPEC_CTRL : if (!msr->host_initiated &&
@@ -2635,9 +2637,9 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) svm->vmcb->save.dbgctl = data;vmcb_mark_dirty (svm->vmcb,VMCB_LBR );if (data & (1ULL <<0 )) -svm_enable_lbrv (svm); +svm_enable_lbrv (vcpu);else -svm_disable_lbrv (svm); +svm_disable_lbrv (vcpu);break ;case MSR_VM_HSAVE_PA : svm->nested.hsave_msr = data;
arch/x86/kvm/vmx/nested.c +1 -1
@@ -4752,7 +4752,7 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu)if (vmx_pt_mode_is_host_guest ()) { vmx->pt_desc.guest.ctl =0 ; -pt_update_intercept_for_msr (vmx); +pt_update_intercept_for_msr (vcpu); }return 0 ;
arch/x86/kvm/vmx/vmx.c +40 -43
167 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.h +1 -1
@@ -351,7 +351,7 @@ bool vmx_get_nmi_mask(struct kvm_vcpu *vcpu);void vmx_set_nmi_mask (struct kvm_vcpu *vcpu,bool masked);void vmx_set_virtual_apic_mode (struct kvm_vcpu *vcpu);struct shared_msr_entry *find_msr_entry (struct vcpu_vmx *vmx,u32 msr); -void pt_update_intercept_for_msr (struct vcpu_vmx *vmx); +void pt_update_intercept_for_msr (struct kvm_vcpu *vcpu);void vmx_update_host_rsp (struct vcpu_vmx *vmx,unsigned long host_rsp);int vmx_find_msr_index (struct vmx_msrs *m,u32 msr); int vmx_handle_memory_failure(struct kvm_vcpu *vcpu, int r,
Signature · 1 line
2.28.0.220.ged08abb693-goog
PATCH v3 7/12 KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
In reply to Aaron LewisSDM volume 3: 24.6.9 "MSR-Bitmap Address" and APM volume 2: 15.11 "MS intercepts" describe MSR permission bitmaps. Permission bitmaps are used to control whether an execution of rdmsr or wrmsr will cause a vm exit. For userspace tracked MSRs it is required they cause a vm exit, so the host is able to forward the MSR to userspace. This change adds vmx/svm support to ensure the permission bitmap is properly set to cause a vm_exit to the host when rdmsr or wrmsr is used by one of the userspace tracked MSRs. Also, to avoid repeatedly setting them, kvm_make_request() is used to coalesce these into a single call.
--- arch/x86/include/asm/kvm_host.h | 3 ++ arch/x86/kvm/svm/svm.c | 49 ++++++++++++++++++++++++++------- arch/x86/kvm/vmx/vmx.c | 13 ++++++++- arch/x86/kvm/x86.c | 16 +++++++++++ 4 files changed, 70 insertions(+), 11 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -87,6 +87,7 @@#define KVM_REQ_HV_TLB_FLUSH \ KVM_ARCH_REQ_FLAGS(27, KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_APF_READY KVM_ARCH_REQ(28) +#define KVM_REQ_USER_MSR_UPDATE KVM_ARCH_REQ(29)#define CR0_RESERVED_BITS \ (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
@@ -1242,6 +1243,8 @@ struct kvm_x86_ops {int (*enable_direct_tlbflush)(struct kvm_vcpu *vcpu);void (*migrate_timers)(struct kvm_vcpu *vcpu); + +void (*set_user_msr_intercept)(struct kvm_vcpu *vcpu,u32 msr); };struct kvm_x86_nested_ops {
arch/x86/kvm/svm/svm.c +39 -10
75 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +12 -1
@@ -3725,6 +3725,10 @@ static __always_inline void vmx_disable_intercept_for_msr(struct kvm_vcpu *vcpu,__clear_bit (msr, msr_bitmap +0xc00 / f); } + +if (type &MSR_TYPE_R || type &MSR_TYPE_W ) { +kvm_make_request (KVM_REQ_USER_MSR_UPDATE , vcpu); + } } static __always_inline void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu,
@@ -3792,7 +3796,7 @@ static u8 vmx_msr_bitmap_mode(struct kvm_vcpu *vcpu) }static void vmx_update_msr_bitmap_x2apic (struct kvm_vcpu *vcpu, -unsigned long *msr_bitmap,u8 mode) +unsigned long *msr_bitmap,u8 mode) {int msr;
@@ -3816,6 +3820,11 @@ static void vmx_update_msr_bitmap_x2apic(struct kvm_vcpu *vcpu, } } +void vmx_set_user_msr_intercept (struct kvm_vcpu *vcpu,u32 msr) +{ +vmx_enable_intercept_for_msr (vcpu, msr,MSR_TYPE_RW ); +} +void vmx_update_msr_bitmap (struct kvm_vcpu *vcpu) {struct vcpu_vmx *vmx =to_vmx (vcpu);
@@ -8002,6 +8011,8 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .need_emulation_on_page_fault = vmx_need_emulation_on_page_fault, .apic_init_signal_blocked = vmx_apic_init_signal_blocked, .migrate_timers = vmx_migrate_timers, + + .set_user_msr_intercept = vmx_set_user_msr_intercept, };static __init int hardware_setup (void )
arch/x86/kvm/x86.c +16 -0
@@ -3647,6 +3647,19 @@ bool kvm_msr_user_exit(struct kvm *kvm, u32 index) }EXPORT_SYMBOL_GPL (kvm_msr_user_exit); +static void kvm_set_user_msr_intercepts (struct kvm_vcpu *vcpu) +{ +struct kvm_msr_list *msr_list = vcpu->kvm->arch.user_exit_msrs; +u32 i, msr; + +if (msr_list) { +for (i =0 ; i < msr_list->nmsrs; i++) { + msr = msr_list->indices[i]; + kvm_x86_ops.set_user_msr_intercept (vcpu, msr); + } + } +} +int kvm_vm_ioctl_check_extension (struct kvm *kvm,long ext) {int r =0 ;
@@ -8823,6 +8836,9 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)kvm_vcpu_update_apicv (vcpu);if (kvm_check_request (KVM_REQ_APF_READY , vcpu))kvm_check_async_pf_completion (vcpu); + +if (kvm_check_request (KVM_REQ_USER_MSR_UPDATE , vcpu)) +kvm_set_user_msr_intercepts (vcpu); }if (kvm_check_request (KVM_REQ_EVENT , vcpu) || req_int_win) {
Signature · 1 line
2.28.0.220.ged08abb693-goog
kernel test robot
build botRe: [PATCH v3 07/12] KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
Thank you for the patch! Perhaps something to improve:
Hi Aaron,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on kvm/linux-next]
[also build test WARNING on v5.9-rc1 next-20200818]
[cannot apply to kvms390/next vhost/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Aaron-Lewis/Allow-userspace-to-manage-MSRs/20200819-051903 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git linux-next config: i386-randconfig-s001-20200818 (attached as .config) compiler: gcc-9 (Debian 9.3.0-15) 9.3.0 reproduce: # apt-get install sparse # sparse version: v0.6.2-183-gaa6ede3b-dirty # save the attached .config to linux build tree make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=i386
If you fix the issue, kindly add following tag as appropriate
sparse warnings: (new ones prefixed by >>)
>> arch/x86/kvm/vmx/vmx.c:3823:6: sparse: sparse: symbol 'vmx_set_user_msr_intercept' was not declared. Should it be static?
arch/x86/kvm/vmx/vmx.c: note: in included file:
arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (110011 becomes 11) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (110011 becomes 11) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100110 becomes 110) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100490 becomes 490) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100310 becomes 310) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100510 becomes 510) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100410 becomes 410) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100490 becomes 490) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100310 becomes 310) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100510 becomes 510) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100410 becomes 410) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (30203 becomes 203) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (30203 becomes 203) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (30283 becomes 283) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (30283 becomes 283) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1b019b becomes 19b) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1b021b becomes 21b) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1b029b becomes 29b) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1b031b becomes 31b) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1b041b becomes 41b) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (80c88 becomes c88) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a081a becomes 81a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a081a becomes 81a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a081a becomes 81a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120912 becomes 912) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120912 becomes 912) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120912 becomes 912) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (110311 becomes 311) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120992 becomes 992) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120992 becomes 992) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100610 becomes 610) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100690 becomes 690) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100590 becomes 590) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (80408 becomes 408) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120a92 becomes a92) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a099a becomes 99a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a091a becomes 91a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a048a becomes 48a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120a92 becomes a92) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a099a becomes 99a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a091a becomes 91a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a048a becomes 48a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a010a becomes 10a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a050a becomes 50a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a071a becomes 71a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a079a becomes 79a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a001a becomes 1a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a009a becomes 9a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a011a becomes 11a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a081a becomes 81a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a081a becomes 81a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a011a becomes 11a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (180198 becomes 198) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a011a becomes 11a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a051a becomes 51a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120392 becomes 392) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120892 becomes 892) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a081a becomes 81a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a081a becomes 81a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a011a becomes 11a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a011a becomes 11a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100490 becomes 490) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100490 becomes 490) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a028a becomes 28a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a030a becomes 30a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a038a becomes 38a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a040a becomes 40a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a028a becomes 28a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a030a becomes 30a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a038a becomes 38a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (a040a becomes 40a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100090 becomes 90) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100090 becomes 90) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (180118 becomes 118) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a001a becomes 1a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (80688 becomes 688) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a009a becomes 9a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100790 becomes 790) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (100790 becomes 790) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (180198 becomes 198) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a011a becomes 11a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120492 becomes 492) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a061a becomes 61a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120492 becomes 492) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a061a becomes 61a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120412 becomes 412) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a059a becomes 59a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (120412 becomes 412) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1a059a becomes 59a) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (20402 becomes 402) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1b001b becomes 1b) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1b009b becomes 9b) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (1b011b becomes 11b) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (30083 becomes 83) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (30183 becomes 183) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (30003 becomes 3) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (30103 becomes 103) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: cast truncates bits from constant value (30303 becomes 303) arch/x86/kvm/vmx/evmcs.h:81:30: sparse: sparse: too many warnings
--
>> arch/x86/kvm/svm/svm.c:636:6: sparse: sparse: symbol 'svm_set_user_msr_intercept' was not declared. Should it be static?
arch/x86/kvm/svm/svm.c:471:17: sparse: sparse: cast truncates bits from constant value (100000000 becomes 0) arch/x86/kvm/svm/svm.c:471:17: sparse: sparse: cast truncates bits from constant value (100000000 becomes 0) arch/x86/kvm/svm/svm.c:471:17: sparse: sparse: cast truncates bits from constant value (100000000 becomes 0)
Please review and possibly fold the followup patch.
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build bot[RFC PATCH] KVM: x86: vmx_set_user_msr_intercept() can be static
In reply to Aaron LewisSigned-off-by: kernel test robot <[email protected]>
--- svm/svm.c | 2 +- vmx/vmx.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/svm.c +1 -1
@@ -633,7 +633,7 @@ static void set_user_msr_interception(struct kvm_vcpu *vcpu, u32 msr, int read,__set_msr_interception (msrpm, msr, read, write, offset); } -void svm_set_user_msr_intercept (struct kvm_vcpu *vcpu,u32 msr) +static void svm_set_user_msr_intercept (struct kvm_vcpu *vcpu,u32 msr) {set_user_msr_interception (vcpu, msr,0 ,0 ); }
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -3820,7 +3820,7 @@ static void vmx_update_msr_bitmap_x2apic(struct kvm_vcpu *vcpu, } } -void vmx_set_user_msr_intercept (struct kvm_vcpu *vcpu,u32 msr) +static void vmx_set_user_msr_intercept (struct kvm_vcpu *vcpu,u32 msr) {vmx_enable_intercept_for_msr (vcpu, msr,MSR_TYPE_RW ); }
Re: [PATCH v3 07/12] KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
In reply to Aaron LewisQuoted 11 lines · Aaron Lewis
SDM volume 3: 24.6.9 "MSR-Bitmap Address" and APM volume 2: 15.11 "MS intercepts" describe MSR permission bitmaps. Permission bitmaps are used to control whether an execution of rdmsr or wrmsr will cause a vm exit. For userspace tracked MSRs it is required they cause a vm exit, so the host is able to forward the MSR to userspace. This change adds vmx/svm support to ensure the permission bitmap is properly set to cause a
vm_exitto the host when rdmsr or wrmsr is used by one of the userspace tracked MSRs. Also, to avoid repeatedly setting them,kvm_make_request()is used to coalesce these into a single call.
Quoted 2 lines
Signed-off-by Aaron Lewis <[email protected]>Reviewed-by Oliver Upton <[email protected]>
This is incomplete, as it doesn't cover all of the x2apic registers. There are also a few MSRs that IIRC are handled differently from this logic, such as EFER.
I'm really curious if this is worth the effort? I would be inclined to say that MSRs that KVM has direct access for need special handling one way or another.
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 07/12] KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
Quoted 18 lines · Alexander Graf
Quoted 11 lines · Aaron Lewis
SDM volume 3: 24.6.9 "MSR-Bitmap Address" and APM volume 2: 15.11 "MS intercepts" describe MSR permission bitmaps. Permission bitmaps are used to control whether an execution of rdmsr or wrmsr will cause a vm exit. For userspace tracked MSRs it is required they cause a vm exit, so the host is able to forward the MSR to userspace. This change adds vmx/svm support to ensure the permission bitmap is properly set to cause a
vm_exitto the host when rdmsr or wrmsr is used by one of the userspace tracked MSRs. Also, to avoid repeatedly setting them,kvm_make_request()is used to coalesce these into a single call.Quoted 2 lines
Signed-off-by Aaron Lewis <[email protected]>Reviewed-by Oliver Upton <[email protected]>
Quoted 8 lines
This is incomplete, as it doesn't cover all of the x2apic registers. There are also a few MSRs that IIRC are handled differently from this logic, such as EFER.
I'm really curious if this is worth the effort? I would be inclined to say that MSRs that KVM has direct access for need special handling one way or another.
Can you please elaborate on this? It was my understanding that the permission bitmap covers the x2apic registers. Also, I’m not sure how EFER is handled differently, but I see there is a separate conversation on that.
This effort does seem worthwhile as it ensures userspace is able to manage the MSRs it is requesting, and will remain that way in the future.
Quoted 14 lines
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 07/12] KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
Quoted 24 lines · Aaron Lewis
Quoted 22 lines · Alexander Graf
Quoted 11 lines · Aaron Lewis
SDM volume 3: 24.6.9 "MSR-Bitmap Address" and APM volume 2: 15.11 "MS intercepts" describe MSR permission bitmaps. Permission bitmaps are used to control whether an execution of rdmsr or wrmsr will cause a vm exit. For userspace tracked MSRs it is required they cause a vm exit, so the host is able to forward the MSR to userspace. This change adds vmx/svm support to ensure the permission bitmap is properly set to cause a
vm_exitto the host when rdmsr or wrmsr is used by one of the userspace tracked MSRs. Also, to avoid repeatedly setting them,kvm_make_request()is used to coalesce these into a single call.Quoted 2 lines
Signed-off-by Aaron Lewis <[email protected]>Reviewed-by Oliver Upton <[email protected]>This is incomplete, as it doesn't cover all of the x2apic registers. There are also a few MSRs that IIRC are handled differently from this logic, such as EFER.
Quoted 7 lines
Quoted 4 lines
I'm really curious if this is worth the effort? I would be inclined to say that MSRs that KVM has direct access for need special handling one way or another.
Can you please elaborate on this? It was my understanding that the permission bitmap covers the x2apic registers. Also, I’m not sure how
So x2apic MSR passthrough is configured specially:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/vmx/vmx.c#n3796
and I think not handled by this patch?
Quoted 2 lines
EFER is handled differently, but I see there is a separate conversation on that.
EFER is a really special beast in VT.
Quoted 3 lines
This effort does seem worthwhile as it ensures userspace is able to manage the MSRs it is requesting, and will remain that way in the future.
I couldn't see why any of the passthrough MSRs are relevant to user space, but I tend to agree that it makes everything more consistent.
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 07/12] KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
Quoted 36 lines · Alexander Graf
Quoted 24 lines · Aaron Lewis
Quoted 22 lines · Alexander Graf
Quoted 11 lines · Aaron Lewis
SDM volume 3: 24.6.9 "MSR-Bitmap Address" and APM volume 2: 15.11 "MS intercepts" describe MSR permission bitmaps. Permission bitmaps are used to control whether an execution of rdmsr or wrmsr will cause a vm exit. For userspace tracked MSRs it is required they cause a vm exit, so the host is able to forward the MSR to userspace. This change adds vmx/svm support to ensure the permission bitmap is properly set to cause a
vm_exitto the host when rdmsr or wrmsr is used by one of the userspace tracked MSRs. Also, to avoid repeatedly setting them,kvm_make_request()is used to coalesce these into a single call.Quoted 2 lines
Signed-off-by Aaron Lewis <[email protected]>Reviewed-by Oliver Upton <[email protected]>This is incomplete, as it doesn't cover all of the x2apic registers. There are also a few MSRs that IIRC are handled differently from this logic, such as EFER.
Quoted 7 lines
Quoted 4 lines
I'm really curious if this is worth the effort? I would be inclined to say that MSRs that KVM has direct access for need special handling one way or another.
Can you please elaborate on this? It was my understanding that the permission bitmap covers the x2apic registers. Also, I’m not sure how
Quoted 6 lines
So x2apic MSR passthrough is configured specially:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/vmx/vmx.c#n3796
and I think not handled by this patch?
By happenstance only, I think, since there is also a call there to vmx_disable_intercept_for_msr() for the TPR when x2APIC is enabled.
Re: [PATCH v3 07/12] KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
Quoted 41 lines · Jim Mattson
Quoted 39 lines · Alexander Graf
Quoted 24 lines · Aaron Lewis
Quoted 22 lines · Alexander Graf
Quoted 11 lines · Aaron Lewis
SDM volume 3: 24.6.9 "MSR-Bitmap Address" and APM volume 2: 15.11 "MS intercepts" describe MSR permission bitmaps. Permission bitmaps are used to control whether an execution of rdmsr or wrmsr will cause a vm exit. For userspace tracked MSRs it is required they cause a vm exit, so the host is able to forward the MSR to userspace. This change adds vmx/svm support to ensure the permission bitmap is properly set to cause a
vm_exitto the host when rdmsr or wrmsr is used by one of the userspace tracked MSRs. Also, to avoid repeatedly setting them,kvm_make_request()is used to coalesce these into a single call.Quoted 2 lines
Signed-off-by Aaron Lewis <[email protected]>Reviewed-by Oliver Upton <[email protected]>This is incomplete, as it doesn't cover all of the x2apic registers. There are also a few MSRs that IIRC are handled differently from this logic, such as EFER.
Quoted 7 lines
Quoted 4 lines
I'm really curious if this is worth the effort? I would be inclined to say that MSRs that KVM has direct access for need special handling one way or another.
Can you please elaborate on this? It was my understanding that the permission bitmap covers the x2apic registers. Also, I’m not sure how
So x2apic MSR passthrough is configured specially:
Quoted 6 lines
Quoted 3 lines
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/vmx/vmx.c#n3796
and I think not handled by this patch?
By happenstance only, I think, since there is also a call there to
vmx_disable_intercept_for_msr()for the TPR when x2APIC is enabled.
If we want to be more explicit about it we could add kvm_make_request(KVM_REQ_USER_MSR_UPDATE, vcpu) After the bitmap is modified, but that doesn't seem to be necessary as Jim pointed out as there are calls to vmx_disable_intercept_for_msr() there which will set the update request for us. And we only have to worry about that if the bitmap is cleared which means MSR_BITMAP_MODE_X2APIC_APICV is set, and that flag can only be set if MSR_BITMAP_MODE_X2APIC is set. So, AFAICT that is covered by my changes.
Re: [PATCH v3 07/12] KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
Quoted 50 lines · Aaron Lewis
Quoted 41 lines · Jim Mattson
Quoted 39 lines · Alexander Graf
Quoted 24 lines · Aaron Lewis
Quoted 22 lines · Alexander Graf
Quoted 11 lines · Aaron Lewis
SDM volume 3: 24.6.9 "MSR-Bitmap Address" and APM volume 2: 15.11 "MS intercepts" describe MSR permission bitmaps. Permission bitmaps are used to control whether an execution of rdmsr or wrmsr will cause a vm exit. For userspace tracked MSRs it is required they cause a vm exit, so the host is able to forward the MSR to userspace. This change adds vmx/svm support to ensure the permission bitmap is properly set to cause a
vm_exitto the host when rdmsr or wrmsr is used by one of the userspace tracked MSRs. Also, to avoid repeatedly setting them,kvm_make_request()is used to coalesce these into a single call.Quoted 2 lines
Signed-off-by Aaron Lewis <[email protected]>Reviewed-by Oliver Upton <[email protected]>This is incomplete, as it doesn't cover all of the x2apic registers. There are also a few MSRs that IIRC are handled differently from this logic, such as EFER.
Quoted 7 lines
Quoted 4 lines
I'm really curious if this is worth the effort? I would be inclined to say that MSRs that KVM has direct access for need special handling one way or another.
Can you please elaborate on this? It was my understanding that the permission bitmap covers the x2apic registers. Also, I’m not sure how
So x2apic MSR passthrough is configured specially:
Quoted 6 lines
Quoted 3 lines
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/vmx/vmx.c#n3796
and I think not handled by this patch?
By happenstance only, I think, since there is also a call there to
vmx_disable_intercept_for_msr()for the TPR when x2APIC is enabled.
Quoted 9 lines
If we want to be more explicit about it we could add
kvm_make_request(KVM_REQ_USER_MSR_UPDATE, vcpu)After the bitmap is modified, but that doesn't seem to be necessary as Jim pointed out as there are calls tovmx_disable_intercept_for_msr()there which will set the update request for us. And we only have to worry about that if the bitmap is cleared which meansMSR_BITMAP_MODE_X2APIC_APICVis set, and that flag can only be set ifMSR_BITMAP_MODE_X2APICis set. So, AFAICT that is covered by my changes.
I don't understand - for most x2APIC MSRs, vmx_{en,dis}able_intercept_for_msr() never gets called, no?
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [PATCH v3 07/12] KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
Quoted 64 lines · Alexander Graf
Quoted 50 lines · Aaron Lewis
Quoted 41 lines · Jim Mattson
Quoted 39 lines · Alexander Graf
Quoted 24 lines · Aaron Lewis
Quoted 22 lines · Alexander Graf
Quoted 11 lines · Aaron Lewis
SDM volume 3: 24.6.9 "MSR-Bitmap Address" and APM volume 2: 15.11 "MS intercepts" describe MSR permission bitmaps. Permission bitmaps are used to control whether an execution of rdmsr or wrmsr will cause a vm exit. For userspace tracked MSRs it is required they cause a vm exit, so the host is able to forward the MSR to userspace. This change adds vmx/svm support to ensure the permission bitmap is properly set to cause a
vm_exitto the host when rdmsr or wrmsr is used by one of the userspace tracked MSRs. Also, to avoid repeatedly setting them,kvm_make_request()is used to coalesce these into a single call.Quoted 2 lines
Signed-off-by Aaron Lewis <[email protected]>Reviewed-by Oliver Upton <[email protected]>This is incomplete, as it doesn't cover all of the x2apic registers. There are also a few MSRs that IIRC are handled differently from this logic, such as EFER.
Quoted 7 lines
Quoted 4 lines
I'm really curious if this is worth the effort? I would be inclined to say that MSRs that KVM has direct access for need special handling one way or another.
Can you please elaborate on this? It was my understanding that the permission bitmap covers the x2apic registers. Also, I’m not sure how
So x2apic MSR passthrough is configured specially:
Quoted 6 lines
Quoted 3 lines
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/vmx/vmx.c#n3796
and I think not handled by this patch?
By happenstance only, I think, since there is also a call there to
vmx_disable_intercept_for_msr()for the TPR when x2APIC is enabled.Quoted 9 lines
If we want to be more explicit about it we could add
kvm_make_request(KVM_REQ_USER_MSR_UPDATE, vcpu)After the bitmap is modified, but that doesn't seem to be necessary as Jim pointed out as there are calls tovmx_disable_intercept_for_msr()there which will set the update request for us. And we only have to worry about that if the bitmap is cleared which meansMSR_BITMAP_MODE_X2APIC_APICVis set, and that flag can only be set ifMSR_BITMAP_MODE_X2APICis set. So, AFAICT that is covered by my changes.
Quoted 3 lines
I don't understand - for most x2APIC MSRs, vmx_{en,dis}able_intercept_for_msr() never gets called, no?
Sorry, to be clear. We just need it to be called once for us to be covered. When it's invoked we set a flag, so the next time vm enter is called we run over the list of MSRs userspace cares about and ensure they are all set correctly. So, if the x2APIC permission bitmaps are modified directly and cleared, we just need for vmx_disable_intercept_for_msr() to be called at least once to tell us to run over the list and set the bits for all MSRs userspace is tracking.
Quoted 14 lines
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
kernel test robot
build botRe: [PATCH v3 07/12] KVM: x86: Ensure the MSR bitmap never clears userspace tracked MSRs
In reply to Aaron LewisThank you for the patch! Perhaps something to improve:
Hi Aaron,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on kvm/linux-next]
[also build test WARNING on v5.9-rc2 next-20200826]
[cannot apply to kvms390/next vhost/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Aaron-Lewis/Allow-userspace-to-manage-MSRs/20200819-051903 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git linux-next config: x86_64-randconfig-a001-20200826 (attached as .config) compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project 7cfcecece0e0430937cf529ce74d3a071a4dedc6)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
>> arch/x86/kvm/vmx/vmx.c:3823:6: warning: no previous prototype for function 'vmx_set_user_msr_intercept' [-Wmissing-prototypes]
void vmx_set_user_msr_intercept(struct kvm_vcpu *vcpu, u32 msr)
^ arch/x86/kvm/vmx/vmx.c:3823:1: note: declare 'static' if the function is not intended to be used outside of this translation unit void vmx_set_user_msr_intercept(struct kvm_vcpu *vcpu, u32 msr)
^
static
1 warning generated.
# https://github.com/0day-ci/linux/commit/e78e2c7f2ae3e9e6be9768f1616b043406ae24dd
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Aaron-Lewis/Allow-userspace-to-manage-MSRs/20200819-051903 git checkout e78e2c7f2ae3e9e6be9768f1616b043406ae24dd
vim +/vmx_set_user_msr_intercept +3823 arch/x86/kvm/vmx/vmx.c
3822
> 3823 void vmx_set_user_msr_intercept(struct kvm_vcpu *vcpu, u32 msr)
3824 {
3825 vmx_enable_intercept_for_msr(vcpu, msr, MSR_TYPE_RW);
3826 }
3827
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
PATCH v3 8/12 selftests: kvm: Fix the segment descriptor layout to match the actual layout
In reply to Aaron LewisFix the layout of 'struct desc64' to match the layout described in the SDM Vol 3, 3.4.5 Segment Descriptors, Figure 3-8. The test added later in this series relies on this and crashes if this layout is not correct.
---
v2 -> v3
- Pulled changes to kvm_seg_fill_gdt_64bit() from a subsequent commit to here.
--- tools/testing/selftests/kvm/include/x86_64/processor.h | 2 +- tools/testing/selftests/kvm/lib/x86_64/processor.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/include/x86_64/processor.h +1 -1
@@ -59,7 +59,7 @@ struct gpr64_regs {struct desc64 {uint16_t limit0;uint16_t base0; -unsigned base1:8 , s:1 , type:4 , dpl:2 , p:1 ; +unsigned base1:8 , type:4 , s:1 , dpl:2 , p:1 ;unsigned limit1:4 , avl:1 , l:1 , db:1 , g:1 , base2:8 ;uint32_t base3;uint32_t zero1;
tools/testing/selftests/kvm/lib/x86_64/processor.c +2 -1
@@ -392,11 +392,12 @@ static void kvm_seg_fill_gdt_64bit(struct kvm_vm *vm, struct kvm_segment *segp) desc->limit0 = segp->limit &0xFFFF ; desc->base0 = segp->base &0xFFFF ; desc->base1 = segp->base >>16 ; - desc->s = segp->s; desc->type = segp->type; + desc->s = segp->s; desc->dpl = segp->dpl; desc->p = segp->present; desc->limit1 = segp->limit >>16 ; + desc->avl = segp->avl; desc->l = segp->l; desc->db = segp->db; desc->g = segp->g;
Signature · 1 line
2.28.0.220.ged08abb693-goog
PATCH v3 9/12 selftests: kvm: Clear uc so UCALL_NONE is being properly reported
In reply to Aaron LewisEnsure the out value 'uc' in get_ucall() is properly reporting UCALL_NONE if the call fails. The return value will be correctly reported, however, the out parameter 'uc' will not be. Clear the struct to ensure the correct value is being reported in the out parameter.
---
v2 -> v3
- This commit is new to the series. This was added to have the ucall changes separate from the exception handling changes and the addition of the test.
- Added support on aarch64 and s390x as well.
--- tools/testing/selftests/kvm/lib/aarch64/ucall.c | 3 +++ tools/testing/selftests/kvm/lib/s390x/ucall.c | 3 +++ tools/testing/selftests/kvm/lib/x86_64/ucall.c | 3 +++ 3 files changed, 9 insertions(+)
tools/testing/selftests/kvm/lib/aarch64/ucall.c +3 -0
@@ -94,6 +94,9 @@ uint64_t get_ucall(struct kvm_vm *vm, uint32_t vcpu_id, struct ucall *uc)struct kvm_run *run =vcpu_state (vm, vcpu_id);struct ucall ucall = {}; +if (uc) +memset (uc,0 ,sizeof (*uc)); +if (run->exit_reason ==KVM_EXIT_MMIO && run->mmio.phys_addr == (uint64_t )ucall_exit_mmio_addr) {vm_vaddr_t gva;
tools/testing/selftests/kvm/lib/s390x/ucall.c +3 -0
@@ -38,6 +38,9 @@ uint64_t get_ucall(struct kvm_vm *vm, uint32_t vcpu_id, struct ucall *uc)struct kvm_run *run =vcpu_state (vm, vcpu_id);struct ucall ucall = {}; +if (uc) +memset (uc,0 ,sizeof (*uc)); + if (run->exit_reason == KVM_EXIT_S390_SIEIC && run->s390_sieic.icptcode == 4 && (run->s390_sieic.ipa >> 8) == 0x83 && /* 0x83 means DIAGNOSE */
tools/testing/selftests/kvm/lib/x86_64/ucall.c +3 -0
@@ -40,6 +40,9 @@ uint64_t get_ucall(struct kvm_vm *vm, uint32_t vcpu_id, struct ucall *uc)struct kvm_run *run =vcpu_state (vm, vcpu_id);struct ucall ucall = {}; +if (uc) +memset (uc,0 ,sizeof (*uc)); +if (run->exit_reason ==KVM_EXIT_IO && run->io.port ==UCALL_PIO_PORT ) {struct kvm_regs regs;
Signature · 1 line
2.28.0.220.ged08abb693-goog
Re: [PATCH v3 09/12] selftests: kvm: Clear uc so UCALL_NONE is being properly reported
Quoted 65 lines · Aaron Lewis
Ensure the out value 'uc' in
get_ucall()is properly reportingUCALL_NONEif the call fails. The return value will be correctly reported, however, the out parameter 'uc' will not be. Clear the struct to ensure the correct value is being reported in the out parameter.Signed-off-by Aaron Lewis <[email protected]>---
v2 -> v3
- This commit is new to the series. This was added to have the ucall changes separate from the exception handling changes and the addition of the test.
- Added support on aarch64 and s390x as well.--- tools/testing/selftests/kvm/lib/aarch64/ucall.c | 3 +++ tools/testing/selftests/kvm/lib/s390x/ucall.c | 3 +++ tools/testing/selftests/kvm/lib/x86_64/ucall.c | 3 +++ 3 files changed, 9 insertions(+)tools/testing/selftests/kvm/lib/aarch64/ucall.c +3
-0@@ -94,6 +94,9 @@ uint64_t get_ucall(struct kvm_vm *vm, uint32_t vcpu_id, struct ucall *uc)struct kvm_run *run =vcpu_state (vm, vcpu_id);struct ucall ucall = {}; +if (uc) +memset (uc,0 ,sizeof (*uc)); +if (run->exit_reason ==KVM_EXIT_MMIO && run->mmio.phys_addr == (uint64_t )ucall_exit_mmio_addr) {vm_vaddr_t gva;tools/testing/selftests/kvm/lib/s390x/ucall.c +3
-0@@ -38,6 +38,9 @@ uint64_t get_ucall(struct kvm_vm *vm, uint32_t vcpu_id, struct ucall *uc)struct kvm_run *run =vcpu_state (vm, vcpu_id);struct ucall ucall = {}; +if (uc) +memset (uc,0 ,sizeof (*uc)); + if (run->exit_reason == KVM_EXIT_S390_SIEIC && run->s390_sieic.icptcode == 4 && (run->s390_sieic.ipa >> 8) == 0x83 && /* 0x83 means DIAGNOSE */tools/testing/selftests/kvm/lib/x86_64/ucall.c +3
-0@@ -40,6 +40,9 @@ uint64_t get_ucall(struct kvm_vm *vm, uint32_t vcpu_id, struct ucall *uc)struct kvm_run *run =vcpu_state (vm, vcpu_id);struct ucall ucall = {}; +if (uc) +memset (uc,0 ,sizeof (*uc)); +if (run->exit_reason ==KVM_EXIT_IO && run->io.port ==UCALL_PIO_PORT ) {struct kvm_regs regs;Signature · 1 line
2.28.0.220.ged08abb693-goog
PATCH v3 10/12 selftests: kvm: Add exception handling to selftests
In reply to Aaron LewisAdd the infrastructure needed to enable exception handling in selftests. This allows any of the exception and interrupt vectors to be overridden in the guest.
---
v2 -> v3
- This commit is new to the series. It was added to have all the infrastructure changes needed to support excpetion handling alone in one commit. The selftest that was included with this change is now in another commit.
- Removed 'dummy' variable. This was added to match other regs structs, but wasn't needed. Removed stack adjustment for this in handlers.S as well.
--- tools/testing/selftests/kvm/Makefile | 19 ++-- .../selftests/kvm/include/x86_64/processor.h | 24 +++++ tools/testing/selftests/kvm/lib/kvm_util.c | 15 +++ .../selftests/kvm/lib/kvm_util_internal.h | 2 + .../selftests/kvm/lib/x86_64/handlers.S | 81 ++++++++++++++ .../selftests/kvm/lib/x86_64/processor.c | 100 +++++++++++++++++- 6 files changed, 232 insertions(+), 9 deletions(-) create mode 100644 tools/testing/selftests/kvm/lib/x86_64/handlers.S
tools/testing/selftests/kvm/Makefile +13 -6
@@ -34,7 +34,7 @@ ifeq ($(ARCH),s390) endif LIBKVM = lib/assert.c lib/elf.c lib/io.c lib/kvm_util.c lib/sparsebit.c lib/test_util.c -LIBKVM_x86_64 = lib/x86_64/processor.c lib/x86_64/vmx.c lib/x86_64/svm.c lib/x86_64/ucall.c +LIBKVM_x86_64 = lib/x86_64/processor.c lib/x86_64/vmx.c lib/x86_64/svm.c lib/x86_64/ucall.c lib/x86_64/handlers.S LIBKVM_aarch64 = lib/aarch64/processor.c lib/aarch64/ucall.c LIBKVM_s390x = lib/s390x/processor.c lib/s390x/ucall.c
@@ -109,14 +109,21 @@ LDFLAGS += -pthread $(no-pie-option) $(pgste-option) include ../lib.mk STATIC_LIBS := $(OUTPUT)/libkvm.a -LIBKVM_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM)) -EXTRA_CLEAN += $(LIBKVM_OBJ) $(STATIC_LIBS) cscope.* +LIBKVM_C := $(filter %.c,$(LIBKVM)) +LIBKVM_S := $(filter %.S,$(LIBKVM)) +LIBKVM_C_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_C)) +LIBKVM_S_OBJ := $(patsubst %.S, $(OUTPUT)/%.o, $(LIBKVM_S)) +EXTRA_CLEAN += $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(STATIC_LIBS) cscope.* + +x := $(shell mkdir -p $(sort $(dir $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ)))) +$(LIBKVM_C_OBJ): $(OUTPUT)/%.o: %.c + $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $< -o $@ -x := $(shell mkdir -p $(sort $(dir $(LIBKVM_OBJ)))) -$(LIBKVM_OBJ): $(OUTPUT)/%.o: %.c +$(LIBKVM_S_OBJ): $(OUTPUT)/%.o: %.S $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $< -o $@ -$(OUTPUT)/libkvm.a: $(LIBKVM_OBJ) +LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) +$(OUTPUT)/libkvm.a: $(LIBKVM_OBJS) $(AR) crs $@ $^ x := $(shell mkdir -p $(sort $(dir $(TEST_GEN_PROGS))))
tools/testing/selftests/kvm/include/x86_64/processor.h +24 -0
@@ -36,6 +36,8 @@#define X86_CR4_SMAP (1ul << 21)#define X86_CR4_PKE (1ul << 22) +#define UNEXPECTED_VECTOR_PORT 0xfff0u +/* General Registers in 64-Bit Mode */ struct gpr64_regs {u64 rax;
@@ -239,6 +241,11 @@ static inline struct desc_ptr get_idt(void)return idt; } +static inline void outl (uint16_t port,uint32_t value) +{ + __asm__ __volatile__("outl %%eax, %%dx" : :"d" (port),"a" (value)); +} +#define SET_XMM (__var, __xmm) \ asm volatile("movq %0, %%"#__xmm : : "r"(__var) : #__xmm)
@@ -338,6 +345,23 @@ uint32_t kvm_get_cpuid_max_basic(void);uint32_t kvm_get_cpuid_max_extended (void );void kvm_get_cpu_address_width (unsigned int *pa_bits,unsigned int *va_bits); +struct ex_regs { +uint64_t rax, rcx, rdx, rbx; +uint64_t rbp, rsi, rdi; +uint64_t r8, r9, r10, r11; +uint64_t r12, r13, r14, r15; +uint64_t vector; +uint64_t error_code; +uint64_t rip; +uint64_t cs; +uint64_t rflags; +}; + +void vm_init_descriptor_tables (struct kvm_vm *vm); +void vcpu_init_descriptor_tables (struct kvm_vm *vm,uint32_t vcpuid); +void vm_handle_exception (struct kvm_vm *vm,int vector, +void (*handler)(struct ex_regs *)); +/* * Basic CPU control in CR0 */
tools/testing/selftests/kvm/lib/kvm_util.c +15 -0
@@ -1195,6 +1195,21 @@ int _vcpu_run(struct kvm_vm *vm, uint32_t vcpuid)do { rc =ioctl (vcpu->fd,KVM_RUN ,NULL ); }while (rc ==-1 && errno ==EINTR ); + +#ifdef __x86_64__ +if (vcpu_state (vm, vcpuid)->exit_reason ==KVM_EXIT_IO + &&vcpu_state (vm, vcpuid)->io.port ==UNEXPECTED_VECTOR_PORT + &&vcpu_state (vm, vcpuid)->io.size ==4 ) { +/* Grab pointer to io data */ +uint32_t *data = (void *)vcpu_state (vm, vcpuid) + +vcpu_state (vm, vcpuid)->io.data_offset; + +TEST_ASSERT (false, +"Unexpected vectored event in guest (vector:0x%x)" , + *data); + } +#endif +return rc; }
tools/testing/selftests/kvm/lib/kvm_util_internal.h +2 -0
@@ -50,6 +50,8 @@ struct kvm_vm {vm_paddr_t pgd;vm_vaddr_t gdt;vm_vaddr_t tss; +vm_vaddr_t idt; +vm_vaddr_t handlers; };struct vcpu *vcpu_find (struct kvm_vm *vm,uint32_t vcpuid);
tools/testing/selftests/kvm/lib/x86_64/handlers.S new file +81 -0
81 lines not on this page · raw message
tools/testing/selftests/kvm/lib/x86_64/processor.c +97 -3
115 lines not on this page · raw message
Signature · 1 line
2.28.0.220.ged08abb693-goog
PATCH v3 11/12 selftests: kvm: Add a test to exercise the userspace MSR list
In reply to Aaron LewisAdd a selftest to test that when ioctl KVM_SET_EXIT_MSRS is called with an MSR list the guest exits to the host and then to userspace when an MSR in that list is read from or written to.
This test uses 3 MSRs to test these new features:
1. MSR_IA32_XSS, an MSR the kernel knows about.
2. MSR_IA32_FLUSH_CMD, an MSR the kernel does not know about.
3. MSR_NON_EXISTENT, an MSR invented in this test for the purposes of passing a fake MSR from the guest to userspace and having the guest be able to read from and write to it, with userspace handling it. KVM just acts as a pass through.
Userspace is also able to inject a #GP. This is demonstrated when MSR_IA32_XSS and MSR_IA32_FLUSH_CMD are misused in the test. When this happens a #GP is initiated in userspace to be thrown in the guest. The #GP exception has been overridden in the test so it can be handled gracefully.
---
v2 -> v3
- Simplified this change by removing exception handling support from it. This commit now just implements the test needed to verify the changes made in this series.
--- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/processor.h | 3 + tools/testing/selftests/kvm/lib/kvm_util.c | 2 + .../selftests/kvm/lib/x86_64/processor.c | 65 ++++ .../selftests/kvm/x86_64/userspace_msr_exit.c | 279 ++++++++++++++++++ 6 files changed, 351 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86_64/userspace_msr_exit.c
tools/testing/selftests/kvm/.gitignore +1 -0
@@ -14,6 +14,7 @@ /x86_64/vmx_preemption_timer_test /x86_64/svm_vmcall_test /x86_64/sync_regs_test +/x86_64/userspace_msr_exit /x86_64/vmx_close_while_nested_test /x86_64/vmx_dirty_log_test /x86_64/vmx_set_nested_state_test
tools/testing/selftests/kvm/Makefile +1 -0
@@ -49,6 +49,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/state_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_preemption_timer_test TEST_GEN_PROGS_x86_64 += x86_64/svm_vmcall_test TEST_GEN_PROGS_x86_64 += x86_64/sync_regs_test +TEST_GEN_PROGS_x86_64 += x86_64/userspace_msr_exit TEST_GEN_PROGS_x86_64 += x86_64/vmx_close_while_nested_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_dirty_log_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_set_nested_state_test
tools/testing/selftests/kvm/include/x86_64/processor.h +3 -0
@@ -341,6 +341,9 @@ int _vcpu_set_msr(struct kvm_vm *vm, uint32_t vcpuid, uint64_t msr_index,void vcpu_set_msr (struct kvm_vm *vm,uint32_t vcpuid,uint64_t msr_index,uint64_t msr_value); +void kvm_set_exit_msrs (struct kvm_vm *vm,uint32_t nmsrs, +uint32_t msr_indices[]); +uint32_t kvm_get_cpuid_max_basic (void );uint32_t kvm_get_cpuid_max_extended (void );void kvm_get_cpu_address_width (unsigned int *pa_bits,unsigned int *va_bits);
tools/testing/selftests/kvm/lib/kvm_util.c +2 -0
@@ -1605,6 +1605,8 @@ static struct exit_reason { {KVM_EXIT_INTERNAL_ERROR, "INTERNAL_ERROR"}, {KVM_EXIT_OSI, "OSI"}, {KVM_EXIT_PAPR_HCALL, "PAPR_HCALL"}, + {KVM_EXIT_X86_RDMSR, "RDMSR"}, + {KVM_EXIT_X86_WRMSR, "WRMSR"}, #ifdef KVM_EXIT_MEMORY_NOT_PRESENT {KVM_EXIT_MEMORY_NOT_PRESENT, "MEMORY_NOT_PRESENT"}, #endif
tools/testing/selftests/kvm/lib/x86_64/processor.c +65 -0
71 lines not on this page · raw message
tools/testing/selftests/kvm/x86_64/userspace_msr_exit.c new file +279 -0
279 lines not on this page · raw message
Signature · 1 line
2.28.0.220.ged08abb693-goog
PATCH v3 12/12 selftests: kvm: Add emulated rdmsr, wrmsr tests
In reply to Aaron LewisAdd tests to exercise the code paths for em_{rdmsr,wrmsr} and emulator_{get,set}_msr. For the generic instruction emulator to work the module parameter kvm.force_emulation_prefix=1 has to be enabled. If it isn't the tests will be skipped.
---
v1 -> v2
- This commit was added to test the changes made to rdmsr and wrmsr when going through the generic instruction emulator which were made by Alexander Graf <[email protected]>.
--- .../selftests/kvm/x86_64/userspace_msr_exit.c | 158 +++++++++++++++++- 1 file changed, 150 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/x86_64/userspace_msr_exit.c +150 -8
@@ -12,8 +12,12 @@#include "kvm_util.h" #include "vmx.h" -#define VCPU_ID 1 +/* Forced emulation prefix, used to invoke the emulator unconditionally. */ +#define KVM_FEP "ud2; .byte 'k', 'v', 'm';" +#define KVM_FEP_LENGTH 5 +static int fep_available =1 ; +#define VCPU_ID 1#define MSR_NON_EXISTENT 0x474f4f00uint32_t msrs[] = {
@@ -63,6 +67,39 @@ static noinline void test_wrmsr(uint32_t msr, uint64_t value)extern char rdmsr_start, rdmsr_end;extern char wrmsr_start, wrmsr_end; +/* +* Note: Force test_em_rdmsr() to not be inlined to prevent the labels, +* rdmsr_start and rdmsr_end, from being defined multiple times. +*/ +static noinline uint64_t test_em_rdmsr(uint32_t msr) +{ +uint32_t a, d; + + guest_exception_count =0 ; + + __asm__ __volatile__(KVM_FEP "em_rdmsr_start: rdmsr; em_rdmsr_end:" : +"=a" (a),"=d" (d) :"c" (msr) :"memory" ); + +return a | ((uint64_t ) d <<32 ); +} + +/* +* Note: Force test_em_wrmsr() to not be inlined to prevent the labels, +* wrmsr_start and wrmsr_end, from being defined multiple times. +*/ +static noinline void test_em_wrmsr(uint32_t msr, uint64_t value) +{ +uint32_t a = value; +uint32_t d = value >>32 ; + + guest_exception_count =0 ; + + __asm__ __volatile__(KVM_FEP "em_wrmsr_start: wrmsr; em_wrmsr_end:" :: +"a" (a),"d" (d),"c" (msr) :"memory" ); +} + +extern char em_rdmsr_start, em_rdmsr_end; +extern char em_wrmsr_start, em_wrmsr_end;static void guest_code (void ) {
@@ -112,17 +149,55 @@ static void guest_code(void)GUEST_ASSERT (data ==2 );GUEST_ASSERT (guest_exception_count ==0 ); +/* +* Test to see if the instruction emulator is available (ie: the module +* parameter 'kvm.force_emulation_prefix=1' is set). This instruction +* will #UD if it isn't available. +*/ + __asm__ __volatile__(KVM_FEP "nop" ); + +if (fep_available) { +/* Let userspace know we aren't done. */ +GUEST_SYNC (0 ); + +/* +* Now run the same tests with the instruction emulator. +*/ + data =test_em_rdmsr (MSR_IA32_XSS ); +GUEST_ASSERT (data ==0 ); +GUEST_ASSERT (guest_exception_count ==0 ); +test_em_wrmsr (MSR_IA32_XSS ,0 ); +GUEST_ASSERT (guest_exception_count ==0 ); +test_em_wrmsr (MSR_IA32_XSS ,1 ); +GUEST_ASSERT (guest_exception_count ==1 ); + +test_em_rdmsr (MSR_IA32_FLUSH_CMD ); +GUEST_ASSERT (guest_exception_count ==1 ); +test_em_wrmsr (MSR_IA32_FLUSH_CMD ,0 ); +GUEST_ASSERT (guest_exception_count ==1 ); +test_em_wrmsr (MSR_IA32_FLUSH_CMD ,1 ); +GUEST_ASSERT (guest_exception_count ==0 ); + +test_em_wrmsr (MSR_NON_EXISTENT ,2 ); +GUEST_ASSERT (guest_exception_count ==0 ); + data =test_em_rdmsr (MSR_NON_EXISTENT ); +GUEST_ASSERT (data ==2 ); +GUEST_ASSERT (guest_exception_count ==0 ); + } +GUEST_DONE (); } -static void guest_gp_handler (struct ex_regs *regs) +static void __guest_gp_handler (struct ex_regs *regs, +char *r_start,char *r_end, +char *w_start,char *w_end) { -if (regs->rip == (uintptr_t )&rdmsr_start) { - regs->rip = (uintptr_t )&rdmsr_end; +if (regs->rip == (uintptr_t )r_start) { + regs->rip = (uintptr_t )r_end; regs->rax =0 ; regs->rdx =0 ; - }else if (regs->rip == (uintptr_t )&wrmsr_start) { - regs->rip = (uintptr_t )&wrmsr_end; + }else if (regs->rip == (uintptr_t )w_start) { + regs->rip = (uintptr_t )w_end; }else {GUEST_ASSERT (!"RIP is at an unknown location!" ); }
@@ -130,6 +205,24 @@ static void guest_gp_handler(struct ex_regs *regs) ++guest_exception_count; } +static void guest_gp_handler (struct ex_regs *regs) +{ +__guest_gp_handler (regs, &rdmsr_start, &rdmsr_end, + &wrmsr_start, &wrmsr_end); +} + +static void guest_fep_gp_handler (struct ex_regs *regs) +{ +__guest_gp_handler (regs, &em_rdmsr_start, &em_rdmsr_end, + &em_wrmsr_start, &em_wrmsr_end); +} + +static void guest_ud_handler (struct ex_regs *regs) +{ + fep_available =0 ; + regs->rip +=KVM_FEP_LENGTH ; +} +static void run_guest (struct kvm_vm *vm) {int rc;
@@ -225,6 +318,32 @@ static void process_ucall_done(struct kvm_vm *vm) uc.cmd, UCALL_DONE); } +static uint64_t process_ucall (struct kvm_vm *vm) +{ +struct kvm_run *run =vcpu_state (vm,VCPU_ID ); +struct ucall uc = {}; + +TEST_ASSERT (run->exit_reason ==KVM_EXIT_IO , +"Unexpected exit reason: %u (%s)" , + run->exit_reason, +exit_reason_str (run->exit_reason)); + +switch (get_ucall (vm,VCPU_ID , &uc)) { +case UCALL_SYNC : +break ; +case UCALL_ABORT : +check_for_guest_assert (vm); +break ; +case UCALL_DONE : +process_ucall_done (vm); +break ; +default : +TEST_ASSERT (false,"Unexpected ucall" ); + } + +return uc.cmd; +} +static void run_guest_then_process_rdmsr (struct kvm_vm *vm,uint32_t msr_index) {run_guest (vm);
@@ -260,7 +379,7 @@ int main(int argc, char *argv[])vm_handle_exception (vm,GP_VECTOR , guest_gp_handler); -/* Process guest code userspace exits */ +/* Process guest code userspace exits. */ run_guest_then_process_rdmsr (vm,MSR_IA32_XSS );run_guest_then_process_wrmsr (vm,MSR_IA32_XSS );run_guest_then_process_wrmsr (vm,MSR_IA32_XSS );
@@ -272,7 +391,30 @@ int main(int argc, char *argv[])run_guest_then_process_wrmsr (vm,MSR_NON_EXISTENT );run_guest_then_process_rdmsr (vm,MSR_NON_EXISTENT ); -run_guest_then_process_ucall_done (vm); +vm_handle_exception (vm,UD_VECTOR , guest_ud_handler); +run_guest (vm); +vm_handle_exception (vm,UD_VECTOR ,NULL ); + +if (process_ucall (vm) !=UCALL_DONE ) { +vm_handle_exception (vm,GP_VECTOR , guest_fep_gp_handler); + +/* Process emulated rdmsr and wrmsr instructions. */ +run_guest_then_process_rdmsr (vm,MSR_IA32_XSS ); +run_guest_then_process_wrmsr (vm,MSR_IA32_XSS ); +run_guest_then_process_wrmsr (vm,MSR_IA32_XSS ); + +run_guest_then_process_rdmsr (vm,MSR_IA32_FLUSH_CMD ); +run_guest_then_process_wrmsr (vm,MSR_IA32_FLUSH_CMD ); +run_guest_then_process_wrmsr (vm,MSR_IA32_FLUSH_CMD ); + +run_guest_then_process_wrmsr (vm,MSR_NON_EXISTENT ); +run_guest_then_process_rdmsr (vm,MSR_NON_EXISTENT ); + +/* Confirm the guest completed without issues. */ +run_guest_then_process_ucall_done (vm); + }else { +printf ("To run the instruction emulated tests set the module parameter 'kvm.force_emulation_prefix=1'\n" ); + }kvm_vm_free (vm);return 0 ;
Signature · 1 line
2.28.0.220.ged08abb693-goog