x86: SEV-ES Guest Support
Subsystems (10)
v7 cover x86: SEV-ES Guest Support
From: Joerg Roedel <[email protected]>
Hi,
here is a new version of the SEV-ES Guest Support patches for x86. The previous versions can be found as a linked list starting here:
https://lore.kernel.org/lkml/[email protected]/
I updated the patch-set based on ther review comments I got and the discussions around it.
Another important change is that the early IDT setup code is now completly moved to arch/x86/kernel/head64.c. This makes the whole early exception handling setup code more robust for kernels that have KASAN and/or Tracing enabled.
A side effect of this change is that secondary CPU now don't use the idt_table at early boot, which means that on the secondary CPUs the early handler does not use IST or paranoid_entry() anymore. This allowed to remove a couple of patches from this series which were only needed to setup relevant processor starte early enough for IST exceptions. In particular this means that the early FSGSBASE and TSS setup is gone now. Also the patch which moved the idt_table to the data segement is now removed.
A related change was necessary in the boot path of secondary CPUs, because those loaded the runtime IDT before setting up the TSS and the getcpu GDT entry. This is now in proper order so that IST exceptions will work when the runtime IDT is loaded for the first time. This setup is added in patch 67.
The cpu_init() function is untouched so that it still act as the intended cpu-state barrier, regardless of what happens before.
The code survived the usual load test of running one x86-selftest loop on each core of the guest in parallel with perf (for NMI load). This runs for several (4+) hours now without any issues.
Please review.
Thanks,
Joerg
Borislav Petkov (1):
KVM: SVM: Use __packed shorthand
Doug Covelli (1):
x86/vmware: Add VMware specific handling for VMMCALL under SEV-ES
Joerg Roedel (50):
KVM: SVM: nested: Don't allocate VMCB structures on stack
KVM: SVM: Add GHCB Accessor functions
x86/traps: Move pf error codes to <asm/trap_pf.h> x86/insn: Make inat-tables.c suitable for pre-decompression code x86/umip: Factor out instruction fetch
x86/umip: Factor out instruction decoding
x86/insn: Add insn_get_modrm_reg_off()
x86/insn: Add insn_has_rep_prefix() helper x86/boot/compressed/64: Disable red-zone usage x86/boot/compressed/64: Add IDT Infrastructure x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c x86/boot/compressed/64: Add page-fault handler x86/boot/compressed/64: Always switch to own page-table x86/boot/compressed/64: Don't pre-map memory in KASLR code x86/boot/compressed/64: Change add_identity_map() to take start and
end
x86/boot/compressed/64: Add stage1 #VC handler x86/boot/compressed/64: Call set_sev_encryption_mask() earlier x86/boot/compressed/64: Check return value of
kernel_ident_mapping_init()
x86/boot/compressed/64: Add set_page_en/decrypted() helpers x86/boot/compressed/64: Setup GHCB Based VC Exception handler x86/boot/compressed/64: Unmap GHCB page before booting the kernel x86/fpu: Move xgetbv()/xsetbv() into separate header x86/idt: Split idt_data setup out of set_intr_gate() x86/head/64: Install startup GDT
x86/head/64: Load GDT after switch to virtual addresses x86/head/64: Load segment registers earlier
x86/head/64: Switch to initial stack earlier
x86/head/64: Install a CPU bringup IDT
x86/idt: Move two function from k/idt.c to i/a/desc.h x86/head/64: Move early exception dispatch to C code x86/sev-es: Add SEV-ES Feature Detection
x86/sev-es: Print SEV-ES info into kernel log
x86/sev-es: Compile early handler code into kernel image x86/sev-es: Setup early #VC handler
x86/sev-es: Setup GHCB based boot #VC handler
x86/sev-es: Allocate and Map IST stack for #VC handler x86/sev-es: Adjust #VC IST Stack on entering NMI handler x86/dumpstack/64: Add noinstr version of get_stack_info() x86/entry/64: Add entry code for #VC handler
x86/sev-es: Wire up existing #VC exit-code handlers x86/sev-es: Handle instruction fetches from user-space x86/sev-es: Handle MMIO String Instructions
x86/sev-es: Handle #AC Events
x86/sev-es: Handle #DB Events
x86/paravirt: Allow hypervisor specific VMMCALL handling under SEV-ES x86/realmode: Add SEV-ES specific trampoline entry point x86/smpboot: Load TSS and getcpu GDT entry before loading IDT x86/head/64: Don't call verify_cpu() on starting APs x86/sev-es: Support CPU offline/online
x86/sev-es: Handle NMI State
Martin Radev (1):
x86/sev-es: Check required CPU features for SEV-ES
Tom Lendacky (19):
KVM: SVM: Add GHCB definitions x86/cpufeatures: Add SEV-ES CPU feature x86/sev-es: Add support for handling IOIO exceptions x86/sev-es: Add CPUID handling to #VC handler x86/sev-es: Setup per-cpu GHCBs for the runtime handler x86/sev-es: Add Runtime #VC Exception Handler x86/sev-es: Handle MMIO events
x86/sev-es: Handle MSR events
x86/sev-es: Handle DR7 read/write events x86/sev-es: Handle WBINVD Events
x86/sev-es: Handle RDTSC(P) Events
x86/sev-es: Handle RDPMC Events
x86/sev-es: Handle INVD Events
x86/sev-es: Handle MONITOR/MONITORX Events x86/sev-es: Handle MWAIT/MWAITX Events x86/sev-es: Handle VMMCALL Events
x86/kvm: Add KVM specific VMMCALL handling under SEV-ES x86/realmode: Setup AP jump table
x86/efi: Add GHCB mappings when SEV-ES is active
arch/x86/Kconfig | 1 + arch/x86/boot/compressed/Makefile | 11 +- arch/x86/boot/compressed/cpuflags.c | 4 - arch/x86/boot/compressed/head_64.S | 33 +- arch/x86/boot/compressed/ident_map_64.c | 349 +++++ arch/x86/boot/compressed/idt_64.c | 54 + arch/x86/boot/compressed/idt_handlers_64.S | 77 ++ arch/x86/boot/compressed/kaslr.c | 36 +- arch/x86/boot/compressed/kaslr_64.c | 153 --- arch/x86/boot/compressed/misc.c | 7 + arch/x86/boot/compressed/misc.h | 50 +- arch/x86/boot/compressed/sev-es.c | 214 +++ arch/x86/entry/entry_64.S | 80 ++ arch/x86/include/asm/cpu_entry_area.h | 33 +- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/desc.h | 27 + arch/x86/include/asm/desc_defs.h | 10 + arch/x86/include/asm/fpu/internal.h | 30 +- arch/x86/include/asm/fpu/xcr.h | 34 + arch/x86/include/asm/idtentry.h | 50 + arch/x86/include/asm/insn-eval.h | 6 + arch/x86/include/asm/mem_encrypt.h | 5 + arch/x86/include/asm/msr-index.h | 3 + arch/x86/include/asm/page_64_types.h | 1 + arch/x86/include/asm/pgtable.h | 2 +- arch/x86/include/asm/processor.h | 1 + arch/x86/include/asm/proto.h | 1 + arch/x86/include/asm/realmode.h | 7 + arch/x86/include/asm/segment.h | 2 +- arch/x86/include/asm/setup.h | 6 +- arch/x86/include/asm/sev-es.h | 114 ++ arch/x86/include/asm/stacktrace.h | 2 + arch/x86/include/asm/svm.h | 106 +- arch/x86/include/asm/trap_pf.h | 24 + arch/x86/include/asm/trapnr.h | 1 + arch/x86/include/asm/traps.h | 20 +- arch/x86/include/asm/x86_init.h | 16 +- arch/x86/include/uapi/asm/svm.h | 11 + arch/x86/kernel/Makefile | 3 + arch/x86/kernel/cpu/amd.c | 3 +- arch/x86/kernel/cpu/common.c | 25 + arch/x86/kernel/cpu/scattered.c | 1 + arch/x86/kernel/cpu/vmware.c | 50 +- arch/x86/kernel/dumpstack.c | 7 +- arch/x86/kernel/dumpstack_64.c | 46 +- arch/x86/kernel/head64.c | 122 +- arch/x86/kernel/head_64.S | 160 ++- arch/x86/kernel/idt.c | 41 +- arch/x86/kernel/kvm.c | 35 +- arch/x86/kernel/nmi.c | 15 + arch/x86/kernel/sev-es-shared.c | 507 +++++++ arch/x86/kernel/sev-es.c | 1404 ++++++++++++++++++++ arch/x86/kernel/smpboot.c | 2 +- arch/x86/kernel/traps.c | 48 + arch/x86/kernel/umip.c | 49 +- arch/x86/kvm/svm/nested.c | 47 +- arch/x86/kvm/svm/svm.c | 2 + arch/x86/lib/insn-eval.c | 130 ++ arch/x86/mm/cpu_entry_area.c | 3 +- arch/x86/mm/extable.c | 1 + arch/x86/mm/mem_encrypt.c | 38 +- arch/x86/mm/mem_encrypt_identity.c | 3 + arch/x86/platform/efi/efi_64.c | 10 + arch/x86/realmode/init.c | 24 +- arch/x86/realmode/rm/header.S | 3 + arch/x86/realmode/rm/trampoline_64.S | 20 + arch/x86/tools/gen-insn-attr-x86.awk | 50 +- tools/arch/x86/tools/gen-insn-attr-x86.awk | 50 +- 68 files changed, 4030 insertions(+), 451 deletions(-) create mode 100644 arch/x86/boot/compressed/ident_map_64.c create mode 100644 arch/x86/boot/compressed/idt_64.c create mode 100644 arch/x86/boot/compressed/idt_handlers_64.S delete mode 100644 arch/x86/boot/compressed/kaslr_64.c create mode 100644 arch/x86/boot/compressed/sev-es.c create mode 100644 arch/x86/include/asm/fpu/xcr.h create mode 100644 arch/x86/include/asm/sev-es.h create mode 100644 arch/x86/include/asm/trap_pf.h create mode 100644 arch/x86/kernel/sev-es-shared.c create mode 100644 arch/x86/kernel/sev-es.c
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v7 1/72 KVM: SVM: nested: Don't allocate VMCB structures on stack
From: Joerg Roedel <[email protected]>
Do not allocate a vmcb_control_area and a vmcb_save_area on the stack, as these structures will become larger with future extenstions of SVM and thus the svm_set_nested_state() function will become a too large stack frame.
--- arch/x86/kvm/svm/nested.c | 47 +++++++++++++++++++++++++++------------ 1 file changed, 33 insertions(+), 14 deletions(-)
arch/x86/kvm/svm/nested.c +33 -14
@@ -1060,10 +1060,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,struct vmcb *hsave = svm->nested.hsave;struct vmcb __user *user_vmcb = (struct vmcb __user *) &user_kvm_nested_state->data.svm[0 ]; -struct vmcb_control_area ctl; -struct vmcb_save_area save; +struct vmcb_control_area *ctl; +struct vmcb_save_area *save; +int ret;u32 cr0; +BUILD_BUG_ON (sizeof (struct vmcb_control_area ) +sizeof (struct vmcb_save_area ) > +KVM_STATE_NESTED_SVM_VMCB_SIZE ); +if (kvm_state->format !=KVM_STATE_NESTED_FORMAT_SVM )return -EINVAL ;
@@ -1095,13 +1099,22 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,return -EINVAL ;if (kvm_state->size <sizeof (*kvm_state) +KVM_STATE_NESTED_SVM_VMCB_SIZE )return -EINVAL ; -if (copy_from_user (&ctl, &user_vmcb->control,sizeof (ctl))) -return -EFAULT ; -if (copy_from_user (&save, &user_vmcb->save,sizeof (save))) -return -EFAULT ; -if (!nested_vmcb_check_controls (&ctl)) -return -EINVAL ; + ret = -ENOMEM ; + ctl =kzalloc (sizeof (*ctl),GFP_KERNEL ); + save =kzalloc (sizeof (*save),GFP_KERNEL ); +if (!ctl || !save) + goto out_free; + + ret = -EFAULT ; +if (copy_from_user (ctl, &user_vmcb->control,sizeof (*ctl))) + goto out_free; +if (copy_from_user (save, &user_vmcb->save,sizeof (*save))) + goto out_free; + + ret = -EINVAL ; +if (!nested_vmcb_check_controls (ctl)) + goto out_free;/* * Processor state contains L2 state. Check that it is
@@ -1109,15 +1122,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,*/ cr0 =kvm_read_cr0 (vcpu);if (((cr0 &X86_CR0_CD ) ==0 ) && (cr0 &X86_CR0_NW )) -return -EINVAL ; + goto out_free;/* * Validate host state saved from before VMRUN (see * nested_svm_check_permissions). * TODO: validate reserved bits for all saved state. */ -if (!(save.cr0 &X86_CR0_PG )) -return -EINVAL ; +if (!(save->cr0 &X86_CR0_PG )) + goto out_free;/* * All checks done, we can enter guest mode. L1 control fields
@@ -1126,15 +1139,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,* contains saved L1 state. */ copy_vmcb_control_area (&hsave->control, &svm->vmcb->control); - hsave->save = save; + hsave->save = *save; svm->nested.vmcb = kvm_state->hdr.svm.vmcb_pa; -load_nested_vmcb_control (svm, &ctl); +load_nested_vmcb_control (svm, ctl);nested_prepare_vmcb_control (svm); out_set_gif:svm_set_gif (svm, !!(kvm_state->flags &KVM_STATE_NESTED_GIF_SET )); -return 0 ; + + ret =0 ; +out_free: +kfree (save); +kfree (ctl); + +return ret; }struct kvm_x86_nested_ops svm_nested_ops = {
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] KVM: SVM: nested: Don't allocate VMCB structures on stack
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 6ccbd29ade0d159ee1be398dc9defaae567c253d Gitweb: https://git.kernel.org/tip/6ccbd29ade0d159ee1be398dc9defaae567c253d Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:02 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
KVM: SVM: nested: Don't allocate VMCB structures on stack
Do not allocate a vmcb_control_area and a vmcb_save_area on the stack, as these structures will become larger with future extenstions of SVM and thus the svm_set_nested_state() function will become a too large stack frame.
--- arch/x86/kvm/svm/nested.c | 47 ++++++++++++++++++++++++++------------ 1 file changed, 33 insertions(+), 14 deletions(-)
arch/x86/kvm/svm/nested.c +33 -14
@@ -1060,10 +1060,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,struct vmcb *hsave = svm->nested.hsave;struct vmcb __user *user_vmcb = (struct vmcb __user *) &user_kvm_nested_state->data.svm[0 ]; -struct vmcb_control_area ctl; -struct vmcb_save_area save; +struct vmcb_control_area *ctl; +struct vmcb_save_area *save; +int ret;u32 cr0; +BUILD_BUG_ON (sizeof (struct vmcb_control_area ) +sizeof (struct vmcb_save_area ) > +KVM_STATE_NESTED_SVM_VMCB_SIZE ); +if (kvm_state->format !=KVM_STATE_NESTED_FORMAT_SVM )return -EINVAL ;
@@ -1095,13 +1099,22 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,return -EINVAL ;if (kvm_state->size <sizeof (*kvm_state) +KVM_STATE_NESTED_SVM_VMCB_SIZE )return -EINVAL ; -if (copy_from_user (&ctl, &user_vmcb->control,sizeof (ctl))) -return -EFAULT ; -if (copy_from_user (&save, &user_vmcb->save,sizeof (save))) -return -EFAULT ; -if (!nested_vmcb_check_controls (&ctl)) -return -EINVAL ; + ret = -ENOMEM ; + ctl =kzalloc (sizeof (*ctl),GFP_KERNEL ); + save =kzalloc (sizeof (*save),GFP_KERNEL ); +if (!ctl || !save) + goto out_free; + + ret = -EFAULT ; +if (copy_from_user (ctl, &user_vmcb->control,sizeof (*ctl))) + goto out_free; +if (copy_from_user (save, &user_vmcb->save,sizeof (*save))) + goto out_free; + + ret = -EINVAL ; +if (!nested_vmcb_check_controls (ctl)) + goto out_free;/* * Processor state contains L2 state. Check that it is
@@ -1109,15 +1122,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,*/ cr0 =kvm_read_cr0 (vcpu);if (((cr0 &X86_CR0_CD ) ==0 ) && (cr0 &X86_CR0_NW )) -return -EINVAL ; + goto out_free;/* * Validate host state saved from before VMRUN (see * nested_svm_check_permissions). * TODO: validate reserved bits for all saved state. */ -if (!(save.cr0 &X86_CR0_PG )) -return -EINVAL ; +if (!(save->cr0 &X86_CR0_PG )) + goto out_free;/* * All checks done, we can enter guest mode. L1 control fields
@@ -1126,15 +1139,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,* contains saved L1 state. */ copy_vmcb_control_area (&hsave->control, &svm->vmcb->control); - hsave->save = save; + hsave->save = *save; svm->nested.vmcb = kvm_state->hdr.svm.vmcb_pa; -load_nested_vmcb_control (svm, &ctl); +load_nested_vmcb_control (svm, ctl);nested_prepare_vmcb_control (svm); out_set_gif:svm_set_gif (svm, !!(kvm_state->flags &KVM_STATE_NESTED_GIF_SET )); -return 0 ; + + ret =0 ; +out_free: +kfree (save); +kfree (ctl); + +return ret; }struct kvm_x86_nested_ops svm_nested_ops = {
Re: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
In reply to Joerg RoedelQuoting Joerg Roedel (2020-09-07 08:15:02)
Quoted 91 lines
From: Joerg Roedel <[email protected]>
Do not allocate a
vmcb_control_areaand avmcb_save_areaon the stack, as these structures will become larger with future extenstions of SVM and thus thesvm_set_nested_state()function will become a too large stack frame.Signed-off-by Joerg Roedel <[email protected]>--- arch/x86/kvm/svm/nested.c | 47 +++++++++++++++++++++++++++------------ 1 file changed, 33 insertions(+), 14 deletions(-)arch/x86/kvm/svm/nested.c +26
-13@@ -1060,10 +1060,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,struct vmcb *hsave = svm->nested.hsave;struct vmcb __user *user_vmcb = (struct vmcb __user *) &user_kvm_nested_state->data.svm[0 ]; -struct vmcb_control_area ctl; -struct vmcb_save_area save; +struct vmcb_control_area *ctl; +struct vmcb_save_area *save; +int ret;u32 cr0; +BUILD_BUG_ON (sizeof (struct vmcb_control_area ) +sizeof (struct vmcb_save_area ) > +KVM_STATE_NESTED_SVM_VMCB_SIZE ); +if (kvm_state->format !=KVM_STATE_NESTED_FORMAT_SVM )return -EINVAL ;@@ -1095,13 +1099,22 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,return -EINVAL ;if (kvm_state->size <sizeof (*kvm_state) +KVM_STATE_NESTED_SVM_VMCB_SIZE )return -EINVAL ; -if (copy_from_user (&ctl, &user_vmcb->control,sizeof (ctl))) -return -EFAULT ; -if (copy_from_user (&save, &user_vmcb->save,sizeof (save))) -return -EFAULT ; -if (!nested_vmcb_check_controls (&ctl)) -return -EINVAL ; + ret = -ENOMEM ; + ctl =kzalloc (sizeof (*ctl),GFP_KERNEL ); + save =kzalloc (sizeof (*save),GFP_KERNEL ); +if (!ctl || !save) + goto out_free; + + ret = -EFAULT ; +if (copy_from_user (ctl, &user_vmcb->control,sizeof (*ctl))) + goto out_free; +if (copy_from_user (save, &user_vmcb->save,sizeof (*save))) + goto out_free; + + ret = -EINVAL ; +if (!nested_vmcb_check_controls (ctl)) + goto out_free;/* * Processor state contains L2 state. Check that it is @@ -1109,15 +1122,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,*/ cr0 =kvm_read_cr0 (vcpu);if (((cr0 &X86_CR0_CD ) ==0 ) && (cr0 &X86_CR0_NW )) -return -EINVAL ; + goto out_free;/* * Validate host state saved from before VMRUN (see * nested_svm_check_permissions). * TODO: validate reserved bits for all saved state. */ -if (!(save.cr0 &X86_CR0_PG )) -return -EINVAL ; +if (!(save->cr0 &X86_CR0_PG )) + goto out_free;/* * All checks done, we can enter guest mode. L1 control fields @@ -1126,15 +1139,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,* contains saved L1 state. */ copy_vmcb_control_area (&hsave->control, &svm->vmcb->control); - hsave->save = save; + hsave->save = *save; svm->nested.vmcb = kvm_state->hdr.svm.vmcb_pa; -load_nested_vmcb_control (svm, &ctl); +load_nested_vmcb_control (svm, ctl);
arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 1154
Quoted 10 lines
arch/x86/kvm/svm/nested.c +5
-1nested_prepare_vmcb_control (svm); out_set_gif:svm_set_gif (svm, !!(kvm_state->flags &KVM_STATE_NESTED_GIF_SET )); -return 0 ; + + ret =0 ; +out_free: +kfree (save); +kfree (ctl);
This change seems to trigger a crash via smm-test.c (and state-test.c) KVM selftest when we call vcpu_load_state->KVM_SET_NESTED_STATE. I think what's happening is we are hitting the 'goto out_set_gif;' and then attempting to free save and ctl, which are still uninitialized at that point:
[ 1999.801176] APIC base relocation is unsupported by KVM [ 1999.828562] BUG: unable to handle page fault for address: fffff12379020288 [ 1999.841968] #PF: supervisor read access in kernel mode [ 1999.847693] #PF: error_code(0x0000) - not-present page [ 1999.853426] PGD 0 P4D 0 [ 1999.856252] Oops: 0000 [#1] SMP NOPTI [ 1999.861366] CPU: 112 PID: 10162 Comm: smm_test Tainted: G E 5.9.1-amdsos-build31t0+ #1 [ 1999.871655] Hardware name: AMD Corporation ETHANOL_X/ETHANOL_X, BIOS RXM0092B 10/27/2020 [ 1999.880694] RIP: 0010:kfree+0x5b/0x3c0 [ 1999.884876] Code: 80 49 01 dc 0f 82 70 03 00 00 48 c7 c0 00 00 00 80 48 2b 05 97 4a 1c 01 49 01 c4 49 c1 ec 0c 49 c1 e4 06 4c 03 25 75 4a 1c 01 <49> 8b 44 24 08 48 8d 50 ff a8 01 4c 0f 45 e2 49 8b 54 24 08 48 8d [ 1999.906674] RSP: 0018:ffffb7004d9d3cf8 EFLAGS: 00010282 [ 1999.912502] RAX: 0000723d80000000 RBX: 000000004080aebf RCX: 0000000002000000 [ 1999.920464] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000004080aebf [ 1999.929335] RBP: ffffb7004d9d3d28 R08: ffff8de1c2a43628 R09: 0000000000000000 [ 1999.937298] R10: 0000000000000000 R11: 0000000000000000 R12: fffff12379020280 [ 1999.945258] R13: ffffffffc0f6626a R14: 0000000000000000 R15: ffffb7004d9d3db0 [ 1999.953221] FS: 00007f231b4c1740(0000) GS:ffff8de20e800000(0000) knlGS:0000000000000000 [ 1999.963255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1999.969667] CR2: fffff12379020288 CR3: 0000001faf5aa006 CR4: 0000000000770ee0 [ 1999.977627] PKRU: 55555554 [ 1999.980644] Call Trace: [ 1999.983384] svm_leave_nested+0xea/0x2f0 [kvm_amd] [ 1999.988743] kvm_arch_vcpu_ioctl+0x6fd/0x1260 [kvm] [ 1999.995026] ? avic_vcpu_load+0x20/0x130 [kvm_amd] [ 2000.000372] kvm_vcpu_kick+0x705/0xae0 [kvm] [ 2000.005216] __x64_sys_ioctl+0x91/0xc0 [ 2000.009470] do_syscall_64+0x38/0x90 [ 2000.013461] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 2000.019090] RIP: 0033:0x7f231b5db50b [ 2000.024052] Code: 0f 1e fa 48 8b 05 85 39 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 55 39 0d 00 f7 d8 64 89 01 48 [ 2000.045005] RSP: 002b:00007ffc01de6918 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 2000.053453] RAX: ffffffffffffffda RBX: 0000000002147880 RCX: 00007f231b5db50b [ 2000.062375] RDX: 0000000002148c98 RSI: 000000004080aebf RDI: 0000000000000009 [ 2000.070335] RBP: 000000000214d0c0 R08: 00000000004103f8 R09: 0000000000000000 [ 2000.078296] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc01de6960 [ 2000.087278] R13: 0000000000000002 R14: 00007f231b711000 R15: 0000000002147880 [ 2000.095244] Modules linked in: nls_iso8859_1(E) dm_multipath(E)
scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E) intel_rapl_msr(E) intel_rapl_common(E) amd64_edac_mod(E) kvm_amd(E) kvm(E) joydev(E) input_leds(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) aesni_intel(E) crypto_simd(E) cryptd(E) glue_helper(E) hid_generic(E) efi_pstore(E) rapl(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) usbhid(E) hid(E) ast(E) drm_vram_helper(E) drm_ttm_helper(E) ttm(E) drm_kms_helper(E) cec(E) i2c_algo_bit(E) fb_sys_fops(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) wmi_bmof(E) e1000e(E) i2c_piix4(E) ccp(E) wmi(E) mac_hid(E) sch_fq_codel(E) drm(E) ip_tables(E) x_tables(E) autofs4(E)
[ 2000.164824] CR2: fffff12379020288 [ 2000.168522] ---[ end trace c5975ced3c660340 ]---
Quoted 12 lines
arch/x86/kvm/svm/nested.c +2
-0+ +return ret; }struct kvm_x86_nested_ops svm_nested_ops = {Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Borislav Petkov
maintainerRe: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
Quoted 4 lines · Michael Roth
Quoted 4 lines
out_set_gif:svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));(quoted diff) +1
-1- return 0; +
Quoted 8 lines
arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 1154
Quoted 4 lines
arch/x86/kvm/svm/nested.c +4
-0+ ret =0 ; +out_free: +kfree (save); +kfree (ctl);This change seems to trigger a crash via
smm-test.c(andstate-test.c) KVM selftest when we callvcpu_load_state->KVM_SET_NESTED_STATE. I think what's happening is we are hitting the 'gotoout_set_gif;'
No out_set_gif upstream anymore after
d5cd6f340145 ("KVM: nSVM: Avoid freeing uninitialized pointers in svm_set_nested_state()")
and it looks like you hit the issue this patch is fixing.
Can you test with the above commit cherrypicked ontop of your what looks like 5.9.1-ish tree?
If that fixes it, we should route this patch to stable.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
Quoted 16 lines · Borislav Petkov
Quoted 4 lines · Michael Roth
Quoted 4 lines
out_set_gif:svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));(quoted diff) +1
-1- return 0; +Quoted 8 lines
arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 1154
Quoted 4 lines
arch/x86/kvm/svm/nested.c +4
-0+ ret =0 ; +out_free: +kfree (save); +kfree (ctl);This change seems to trigger a crash via
smm-test.c(andstate-test.c) KVM selftest when we callvcpu_load_state->KVM_SET_NESTED_STATE. I think what's happening is we are hitting the 'gotoout_set_gif;'No
out_set_gifupstream anymore after
Quoted 8 lines
d5cd6f340145("KVM: nSVM: Avoid freeing uninitialized pointers insvm_set_nested_state()")and it looks like you hit the issue this patch is fixing.
Can you test with the above commit cherrypicked ontop of your what looks like 5.9.1-ish tree?
If that fixes it, we should route this patch to stable.
Hi Boris,
I can confirm that patch fixes the issue. It is indeed a 5.9.1 tree, but looks like the SEV-ES patches didn't go in until v5.10-rc1 (this tree had a backport of them), so stable trees shouldn't be affected.
Thanks!
-Mike
Quoted 8 lines
Thx.
Signature · 4 lines
Regards/Gruss, Boris. https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpeople.kernel.org%2Ftglx%2Fnotes-about-netiquette&data=04%7C01%7Cmichael.roth%40amd.com%7Cfdf6f0dd23ed48449e1e08d881a93909%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637401911116171669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yvG6Gtb%2FVJnjMTXsOBApU83DCuKx3%2FRAID6f3TpEy7w%3D&reserved=0
Borislav Petkov
maintainerRe: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
Quoted 2 lines · Michael Roth
I can confirm that patch fixes the issue. It is indeed a 5.9.1 tree, but looks like the SEV-ES patches didn't go in until v5.10-rc1
Yes, they went into 5.10-rc1 during the merge window.
Quoted 2 lines
(this tree had a backport of them), so stable trees shouldn't be affected.
Ah, ok, that makes sense.
Thanks for checking!
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
In reply to Michael RothQuoting Michael Roth (2020-11-05 10:24:37)
Quoted 94 lines
Quoting Joerg Roedel (2020-09-07 08:15:02)
Quoted 93 lines
From: Joerg Roedel <[email protected]>
Do not allocate a
vmcb_control_areaand avmcb_save_areaon the stack, as these structures will become larger with future extenstions of SVM and thus thesvm_set_nested_state()function will become a too large stack frame.Signed-off-by Joerg Roedel <[email protected]>--- arch/x86/kvm/svm/nested.c | 47 +++++++++++++++++++++++++++------------ 1 file changed, 33 insertions(+), 14 deletions(-)arch/x86/kvm/svm/nested.c +26
-13@@ -1060,10 +1060,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,struct vmcb *hsave = svm->nested.hsave;struct vmcb __user *user_vmcb = (struct vmcb __user *) &user_kvm_nested_state->data.svm[0 ]; -struct vmcb_control_area ctl; -struct vmcb_save_area save; +struct vmcb_control_area *ctl; +struct vmcb_save_area *save; +int ret;u32 cr0; +BUILD_BUG_ON (sizeof (struct vmcb_control_area ) +sizeof (struct vmcb_save_area ) > +KVM_STATE_NESTED_SVM_VMCB_SIZE ); +if (kvm_state->format !=KVM_STATE_NESTED_FORMAT_SVM )return -EINVAL ;@@ -1095,13 +1099,22 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,return -EINVAL ;if (kvm_state->size <sizeof (*kvm_state) +KVM_STATE_NESTED_SVM_VMCB_SIZE )return -EINVAL ; -if (copy_from_user (&ctl, &user_vmcb->control,sizeof (ctl))) -return -EFAULT ; -if (copy_from_user (&save, &user_vmcb->save,sizeof (save))) -return -EFAULT ; -if (!nested_vmcb_check_controls (&ctl)) -return -EINVAL ; + ret = -ENOMEM ; + ctl =kzalloc (sizeof (*ctl),GFP_KERNEL ); + save =kzalloc (sizeof (*save),GFP_KERNEL ); +if (!ctl || !save) + goto out_free; + + ret = -EFAULT ; +if (copy_from_user (ctl, &user_vmcb->control,sizeof (*ctl))) + goto out_free; +if (copy_from_user (save, &user_vmcb->save,sizeof (*save))) + goto out_free; + + ret = -EINVAL ; +if (!nested_vmcb_check_controls (ctl)) + goto out_free;/* * Processor state contains L2 state. Check that it is @@ -1109,15 +1122,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,*/ cr0 =kvm_read_cr0 (vcpu);if (((cr0 &X86_CR0_CD ) ==0 ) && (cr0 &X86_CR0_NW )) -return -EINVAL ; + goto out_free;/* * Validate host state saved from before VMRUN (see * nested_svm_check_permissions). * TODO: validate reserved bits for all saved state. */ -if (!(save.cr0 &X86_CR0_PG )) -return -EINVAL ; +if (!(save->cr0 &X86_CR0_PG )) + goto out_free;/* * All checks done, we can enter guest mode. L1 control fields @@ -1126,15 +1139,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,* contains saved L1 state. */ copy_vmcb_control_area (&hsave->control, &svm->vmcb->control); - hsave->save = save; + hsave->save = *save; svm->nested.vmcb = kvm_state->hdr.svm.vmcb_pa; -load_nested_vmcb_control (svm, &ctl); +load_nested_vmcb_control (svm, ctl);nested_prepare_vmcb_control (svm);
Quoted 13 lines
arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 1154
Quoted 8 lines
arch/x86/kvm/svm/nested.c +5
-1out_set_gif:svm_set_gif (svm, !!(kvm_state->flags &KVM_STATE_NESTED_GIF_SET )); -return 0 ; + + ret =0 ; +out_free: +kfree (save); +kfree (ctl);This change seems to trigger a crash via
smm-test.c(andstate-test.c) KVM selftest when we callvcpu_load_state->KVM_SET_NESTED_STATE. I think what's happening is we are hitting the 'gotoout_set_gif;' and then attempting to free save and ctl, which are still uninitialized at that point:
Sorry, looks like this one was already fixed upstream by
d5cd6f34014592a232ce79dc25e295778bd43c22
Please ignore.
Quoted 83 lines
[ 1999.801176] APIC base relocation is unsupported by KVM [ 1999.828562] BUG: unable to handle page fault for address: fffff12379020288 [ 1999.841968] #PF: supervisor read access in kernel mode [ 1999.847693] #PF: error_code(0x0000) - not-present page [ 1999.853426] PGD 0 P4D 0 [ 1999.856252] Oops: 0000 [#1] SMP NOPTI [ 1999.861366] CPU: 112 PID: 10162 Comm: smm_test Tainted: G E 5.9.1-amdsos-build31t0+ #1 [ 1999.871655] Hardware name: AMD Corporation ETHANOL_X/ETHANOL_X, BIOS RXM0092B 10/27/2020 [ 1999.880694] RIP: 0010:kfree+0x5b/0x3c0 [ 1999.884876] Code: 80 49 01 dc 0f 82 70 03 00 00 48 c7 c0 00 00 00 80 48 2b 05 97 4a 1c 01 49 01 c4 49 c1 ec 0c 49 c1 e4 06 4c 03 25 75 4a 1c 01 <49> 8b 44 24 08 48 8d 50 ff a8 01 4c 0f 45 e2 49 8b 54 24 08 48 8d [ 1999.906674] RSP: 0018:ffffb7004d9d3cf8 EFLAGS: 00010282 [ 1999.912502] RAX: 0000723d80000000 RBX: 000000004080aebf RCX: 0000000002000000 [ 1999.920464] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000004080aebf [ 1999.929335] RBP: ffffb7004d9d3d28 R08: ffff8de1c2a43628 R09: 0000000000000000 [ 1999.937298] R10: 0000000000000000 R11: 0000000000000000 R12: fffff12379020280 [ 1999.945258] R13: ffffffffc0f6626a R14: 0000000000000000 R15: ffffb7004d9d3db0 [ 1999.953221] FS: 00007f231b4c1740(0000) GS:ffff8de20e800000(0000) knlGS:0000000000000000 [ 1999.963255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1999.969667] CR2: fffff12379020288 CR3: 0000001faf5aa006 CR4: 0000000000770ee0 [ 1999.977627] PKRU: 55555554 [ 1999.980644] Call Trace: [ 1999.983384] svm_leave_nested+0xea/0x2f0 [kvm_amd] [ 1999.988743] kvm_arch_vcpu_ioctl+0x6fd/0x1260 [kvm] [ 1999.995026] ? avic_vcpu_load+0x20/0x130 [kvm_amd] [ 2000.000372] kvm_vcpu_kick+0x705/0xae0 [kvm] [ 2000.005216] __x64_sys_ioctl+0x91/0xc0 [ 2000.009470] do_syscall_64+0x38/0x90 [ 2000.013461] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 2000.019090] RIP: 0033:0x7f231b5db50b [ 2000.024052] Code: 0f 1e fa 48 8b 05 85 39 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 55 39 0d 00 f7 d8 64 89 01 48 [ 2000.045005] RSP: 002b:00007ffc01de6918 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 2000.053453] RAX: ffffffffffffffda RBX: 0000000002147880 RCX: 00007f231b5db50b [ 2000.062375] RDX: 0000000002148c98 RSI: 000000004080aebf RDI: 0000000000000009 [ 2000.070335] RBP: 000000000214d0c0 R08: 00000000004103f8 R09: 0000000000000000 [ 2000.078296] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc01de6960 [ 2000.087278] R13: 0000000000000002 R14: 00007f231b711000 R15: 0000000002147880 [ 2000.095244] Modules linked in: nls_iso8859_1(E) dm_multipath(E)
scsi_dh_rdac(E)scsi_dh_emc(E)scsi_dh_alua(E)intel_rapl_msr(E)intel_rapl_common(E)amd64_edac_mod(E)kvm_amd(E)kvm(E) joydev(E)input_leds(E)crct10dif_pclmul(E)crc32_pclmul(E)ghash_clmulni_intel(E)aesni_intel(E)crypto_simd(E)cryptd(E)glue_helper(E)hid_generic(E)efi_pstore(E)rapl(E)ipmi_si(E)ipmi_devintf(E)ipmi_msghandler(E)usbhid(E) hid(E) ast(E)drm_vram_helper(E)drm_ttm_helper(E)ttm(E)drm_kms_helper(E)cec(E)i2c_algo_bit(E)fb_sys_fops(E)syscopyarea(E) sysfillrect(E) sysimgblt(E)wmi_bmof(E)e1000e(E)i2c_piix4(E)ccp(E) wmi(E)mac_hid(E)sch_fq_codel(E)drm(E)ip_tables(E)x_tables(E)autofs4(E)[ 2000.164824] CR2: fffff12379020288 [ 2000.168522] ---[ end trace c5975ced3c660340 ]---Quoted 12 lines
arch/x86/kvm/svm/nested.c +2
-0+ +return ret; }struct kvm_x86_nested_ops svm_nested_ops = {Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v7 2/72 KVM: SVM: Add GHCB definitions
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
Extend the vmcb_safe_area with SEV-ES fields and add a new 'struct ghcb' which will be used for guest-hypervisor communication.
--- arch/x86/include/asm/svm.h | 51 ++++++++++++++++++++++++++++++++++++-- arch/x86/kvm/svm/svm.c | 2 ++ 2 files changed, 51 insertions(+), 2 deletions(-)
arch/x86/include/asm/svm.h +49 -2
@@ -200,13 +200,60 @@ struct __attribute__ ((__packed__)) vmcb_save_area {u64 br_to;u64 last_excp_from;u64 last_excp_to; + +/* +* The following part of the save area is valid only for +* SEV-ES guests when referenced through the GHCB. +*/ +u8 reserved_7[104 ]; +u64 reserved_8;/* rax already available at 0x01f8 */ +u64 rcx; +u64 rdx; +u64 rbx; +u64 reserved_9;/* rsp already available at 0x01d8 */ +u64 rbp; +u64 rsi; +u64 rdi; +u64 r8; +u64 r9; +u64 r10; +u64 r11; +u64 r12; +u64 r13; +u64 r14; +u64 r15; +u8 reserved_10[16 ]; +u64 sw_exit_code; +u64 sw_exit_info_1; +u64 sw_exit_info_2; +u64 sw_scratch; +u8 reserved_11[56 ]; +u64 xcr0; +u8 valid_bitmap[16 ]; +u64 x87_state_gpa; }; +struct ghcb { +struct vmcb_save_area save; +u8 reserved_save[2048 -sizeof (struct vmcb_save_area )]; + +u8 shared_buffer[2032 ]; + +u8 reserved_1[10 ]; +u16 protocol_version;/* negotiated SEV-ES/GHCB protocol version */ +u32 ghcb_usage; +} __packed; + + +#define EXPECTED_VMCB_SAVE_AREA_SIZE 1032 +#define EXPECTED_VMCB_CONTROL_AREA_SIZE 256 +#define EXPECTED_GHCB_SIZE PAGE_SIZEstatic inline void __unused_size_checks (void ) { -BUILD_BUG_ON (sizeof (struct vmcb_save_area ) !=0x298 ); -BUILD_BUG_ON (sizeof (struct vmcb_control_area ) !=256 ); +BUILD_BUG_ON (sizeof (struct vmcb_save_area ) !=EXPECTED_VMCB_SAVE_AREA_SIZE ); +BUILD_BUG_ON (sizeof (struct vmcb_control_area ) !=EXPECTED_VMCB_CONTROL_AREA_SIZE ); +BUILD_BUG_ON (sizeof (struct ghcb ) !=EXPECTED_GHCB_SIZE ); }struct __attribute__ ((__packed__ ))vmcb {
arch/x86/kvm/svm/svm.c +2 -0
@@ -4164,6 +4164,8 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = {static int __init svm_init (void ) { +__unused_size_checks (); +return kvm_init (&svm_init_ops,sizeof (struct vcpu_svm ), __alignof__(struct vcpu_svm ),THIS_MODULE ); }
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] KVM: SVM: Add GHCB definitions
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: d07f46f9f51afc7fc9f021eae19eba3c2e7870ac Gitweb: https://git.kernel.org/tip/d07f46f9f51afc7fc9f021eae19eba3c2e7870ac Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:03 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
KVM: SVM: Add GHCB definitions
Extend the vmcb_safe_area with SEV-ES fields and add a new 'struct ghcb' which will be used for guest-hypervisor communication.
--- arch/x86/include/asm/svm.h | 51 +++++++++++++++++++++++++++++++++++-- arch/x86/kvm/svm/svm.c | 2 +- 2 files changed, 51 insertions(+), 2 deletions(-)
arch/x86/include/asm/svm.h +49 -2
@@ -200,13 +200,60 @@ struct __attribute__ ((__packed__)) vmcb_save_area {u64 br_to;u64 last_excp_from;u64 last_excp_to; + +/* +* The following part of the save area is valid only for +* SEV-ES guests when referenced through the GHCB. +*/ +u8 reserved_7[104 ]; +u64 reserved_8;/* rax already available at 0x01f8 */ +u64 rcx; +u64 rdx; +u64 rbx; +u64 reserved_9;/* rsp already available at 0x01d8 */ +u64 rbp; +u64 rsi; +u64 rdi; +u64 r8; +u64 r9; +u64 r10; +u64 r11; +u64 r12; +u64 r13; +u64 r14; +u64 r15; +u8 reserved_10[16 ]; +u64 sw_exit_code; +u64 sw_exit_info_1; +u64 sw_exit_info_2; +u64 sw_scratch; +u8 reserved_11[56 ]; +u64 xcr0; +u8 valid_bitmap[16 ]; +u64 x87_state_gpa; }; +struct ghcb { +struct vmcb_save_area save; +u8 reserved_save[2048 -sizeof (struct vmcb_save_area )]; + +u8 shared_buffer[2032 ]; + +u8 reserved_1[10 ]; +u16 protocol_version;/* negotiated SEV-ES/GHCB protocol version */ +u32 ghcb_usage; +} __packed; + + +#define EXPECTED_VMCB_SAVE_AREA_SIZE 1032 +#define EXPECTED_VMCB_CONTROL_AREA_SIZE 256 +#define EXPECTED_GHCB_SIZE PAGE_SIZEstatic inline void __unused_size_checks (void ) { -BUILD_BUG_ON (sizeof (struct vmcb_save_area ) !=0x298 ); -BUILD_BUG_ON (sizeof (struct vmcb_control_area ) !=256 ); +BUILD_BUG_ON (sizeof (struct vmcb_save_area ) !=EXPECTED_VMCB_SAVE_AREA_SIZE ); +BUILD_BUG_ON (sizeof (struct vmcb_control_area ) !=EXPECTED_VMCB_CONTROL_AREA_SIZE ); +BUILD_BUG_ON (sizeof (struct ghcb ) !=EXPECTED_GHCB_SIZE ); }struct __attribute__ ((__packed__ ))vmcb {
arch/x86/kvm/svm/svm.c +2 -0
@@ -4164,6 +4164,8 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = {static int __init svm_init (void ) { +__unused_size_checks (); +return kvm_init (&svm_init_ops,sizeof (struct vcpu_svm ), __alignof__(struct vcpu_svm ),THIS_MODULE ); }
PATCH v7 3/72 KVM: SVM: Add GHCB Accessor functions
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Building a correct GHCB for the hypervisor requires setting valid bits in the GHCB. Simplify that process by providing accessor functions to set values and to update the valid bitmap and to check the valid bitmap in KVM.
--- arch/x86/include/asm/svm.h | 43 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
arch/x86/include/asm/svm.h +43 -0
@@ -345,4 +345,47 @@ struct __attribute__ ((__packed__)) vmcb {#define SVM_CR0_SELECTIVE_MASK (X86_CR0_TS | X86_CR0_MP) +/* GHCB Accessor functions */ + +#define GHCB_BITMAP_IDX (field) \ + (offsetof(struct vmcb_save_area, field) / sizeof(u64)) + +#define DEFINE_GHCB_ACCESSORS (field) \ + static inline bool ghcb_##field##_is_valid(const struct ghcb *ghcb) \ + { \ + return test_bit(GHCB_BITMAP_IDX(field), \ + (unsigned long *)&ghcb->save.valid_bitmap); \ + } \ + \ + static inline void ghcb_set_##field(struct ghcb *ghcb, u64 value) \ + { \ + __set_bit(GHCB_BITMAP_IDX(field), \ + (unsigned long *)&ghcb->save.valid_bitmap); \ + ghcb->save.field = value; \ + } + +DEFINE_GHCB_ACCESSORS(cpl) +DEFINE_GHCB_ACCESSORS(rip) +DEFINE_GHCB_ACCESSORS(rsp) +DEFINE_GHCB_ACCESSORS(rax) +DEFINE_GHCB_ACCESSORS(rcx) +DEFINE_GHCB_ACCESSORS(rdx) +DEFINE_GHCB_ACCESSORS(rbx) +DEFINE_GHCB_ACCESSORS(rbp) +DEFINE_GHCB_ACCESSORS(rsi) +DEFINE_GHCB_ACCESSORS(rdi) +DEFINE_GHCB_ACCESSORS(r8) +DEFINE_GHCB_ACCESSORS(r9) +DEFINE_GHCB_ACCESSORS(r10) +DEFINE_GHCB_ACCESSORS(r11) +DEFINE_GHCB_ACCESSORS(r12) +DEFINE_GHCB_ACCESSORS(r13) +DEFINE_GHCB_ACCESSORS(r14) +DEFINE_GHCB_ACCESSORS(r15) +DEFINE_GHCB_ACCESSORS(sw_exit_code) +DEFINE_GHCB_ACCESSORS(sw_exit_info_1) +DEFINE_GHCB_ACCESSORS(sw_exit_info_2) +DEFINE_GHCB_ACCESSORS(sw_scratch) +DEFINE_GHCB_ACCESSORS(xcr0) +#endif
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] KVM: SVM: Add GHCB Accessor functions
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 3702c2f4eed2188440f65ecdfc89165106fe565d Gitweb: https://git.kernel.org/tip/3702c2f4eed2188440f65ecdfc89165106fe565d Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:04 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
KVM: SVM: Add GHCB Accessor functions
Building a correct GHCB for the hypervisor requires setting valid bits in the GHCB. Simplify that process by providing accessor functions to set values and to update the valid bitmap and to check the valid bitmap in KVM.
--- arch/x86/include/asm/svm.h | 43 +++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+)
arch/x86/include/asm/svm.h +43 -0
@@ -345,4 +345,47 @@ struct __attribute__ ((__packed__)) vmcb {#define SVM_CR0_SELECTIVE_MASK (X86_CR0_TS | X86_CR0_MP) +/* GHCB Accessor functions */ + +#define GHCB_BITMAP_IDX (field) \ + (offsetof(struct vmcb_save_area, field) / sizeof(u64)) + +#define DEFINE_GHCB_ACCESSORS (field) \ + static inline bool ghcb_##field##_is_valid(const struct ghcb *ghcb) \ + { \ + return test_bit(GHCB_BITMAP_IDX(field), \ + (unsigned long *)&ghcb->save.valid_bitmap); \ + } \ + \ + static inline void ghcb_set_##field(struct ghcb *ghcb, u64 value) \ + { \ + __set_bit(GHCB_BITMAP_IDX(field), \ + (unsigned long *)&ghcb->save.valid_bitmap); \ + ghcb->save.field = value; \ + } + +DEFINE_GHCB_ACCESSORS(cpl) +DEFINE_GHCB_ACCESSORS(rip) +DEFINE_GHCB_ACCESSORS(rsp) +DEFINE_GHCB_ACCESSORS(rax) +DEFINE_GHCB_ACCESSORS(rcx) +DEFINE_GHCB_ACCESSORS(rdx) +DEFINE_GHCB_ACCESSORS(rbx) +DEFINE_GHCB_ACCESSORS(rbp) +DEFINE_GHCB_ACCESSORS(rsi) +DEFINE_GHCB_ACCESSORS(rdi) +DEFINE_GHCB_ACCESSORS(r8) +DEFINE_GHCB_ACCESSORS(r9) +DEFINE_GHCB_ACCESSORS(r10) +DEFINE_GHCB_ACCESSORS(r11) +DEFINE_GHCB_ACCESSORS(r12) +DEFINE_GHCB_ACCESSORS(r13) +DEFINE_GHCB_ACCESSORS(r14) +DEFINE_GHCB_ACCESSORS(r15) +DEFINE_GHCB_ACCESSORS(sw_exit_code) +DEFINE_GHCB_ACCESSORS(sw_exit_info_1) +DEFINE_GHCB_ACCESSORS(sw_exit_info_2) +DEFINE_GHCB_ACCESSORS(sw_scratch) +DEFINE_GHCB_ACCESSORS(xcr0) +#endif
PATCH v7 4/72 KVM: SVM: Use __packed shorthand
In reply to Joerg RoedelFrom: Borislav Petkov <[email protected]>
Use the shorthand to make it more readable.
No functional changes.
--- arch/x86/include/asm/svm.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
arch/x86/include/asm/svm.h +6 -6
@@ -150,14 +150,14 @@ struct __attribute__ ((__packed__)) vmcb_control_area {#define SVM_NESTED_CTL_NP_ENABLE BIT(0)#define SVM_NESTED_CTL_SEV_ENABLE BIT(1) -struct __attribute__ ((__packed__ ))vmcb_seg { +struct vmcb_seg {u16 selector;u16 attrib;u32 limit;u64 base; -}; +} __packed; -struct __attribute__ ((__packed__ ))vmcb_save_area { +struct vmcb_save_area {struct vmcb_seg es;struct vmcb_seg cs;struct vmcb_seg ss;
@@ -231,7 +231,7 @@ struct __attribute__ ((__packed__)) vmcb_save_area {u64 xcr0;u8 valid_bitmap[16 ];u64 x87_state_gpa; -}; +} __packed;struct ghcb {struct vmcb_save_area save;
@@ -256,11 +256,11 @@ static inline void __unused_size_checks(void)BUILD_BUG_ON (sizeof (struct ghcb ) !=EXPECTED_GHCB_SIZE ); } -struct __attribute__ ((__packed__ ))vmcb { +struct vmcb {struct vmcb_control_area control;u8 reserved_control[1024 -sizeof (struct vmcb_control_area )];struct vmcb_save_area save; -}; +} __packed;#define SVM_CPUID_FUNC 0x8000000a
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] KVM: SVM: Use __packed shorthand
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 976bc5e2aceedef13e0ba1f0e6e372a22164aa0c Gitweb: https://git.kernel.org/tip/976bc5e2aceedef13e0ba1f0e6e372a22164aa0c Author: Borislav Petkov <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:05 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
KVM: SVM: Use __packed shorthand
Use the shorthand to make it more readable.
No functional changes.
--- arch/x86/include/asm/svm.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
arch/x86/include/asm/svm.h +6 -6
@@ -150,14 +150,14 @@ struct __attribute__ ((__packed__)) vmcb_control_area {#define SVM_NESTED_CTL_NP_ENABLE BIT(0)#define SVM_NESTED_CTL_SEV_ENABLE BIT(1) -struct __attribute__ ((__packed__ ))vmcb_seg { +struct vmcb_seg {u16 selector;u16 attrib;u32 limit;u64 base; -}; +} __packed; -struct __attribute__ ((__packed__ ))vmcb_save_area { +struct vmcb_save_area {struct vmcb_seg es;struct vmcb_seg cs;struct vmcb_seg ss;
@@ -231,7 +231,7 @@ struct __attribute__ ((__packed__)) vmcb_save_area {u64 xcr0;u8 valid_bitmap[16 ];u64 x87_state_gpa; -}; +} __packed;struct ghcb {struct vmcb_save_area save;
@@ -256,11 +256,11 @@ static inline void __unused_size_checks(void)BUILD_BUG_ON (sizeof (struct ghcb ) !=EXPECTED_GHCB_SIZE ); } -struct __attribute__ ((__packed__ ))vmcb { +struct vmcb {struct vmcb_control_area control;u8 reserved_control[1024 -sizeof (struct vmcb_control_area )];struct vmcb_save_area save; -}; +} __packed;#define SVM_CPUID_FUNC 0x8000000a
PATCH v7 5/72 x86/cpufeatures: Add SEV-ES CPU feature
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
Add CPU feature detection for Secure Encrypted Virtualization with Encrypted State. This feature enhances SEV by also encrypting the guest register state, making it in-accessible to the hypervisor.
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/amd.c | 3 ++- arch/x86/kernel/cpu/scattered.c | 1 + 3 files changed, 4 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -236,6 +236,7 @@#define X86_FEATURE_EPT_AD ( 8*32+17)/* Intel Extended Page Table access-dirty bit */ #define X86_FEATURE_VMCALL ( 8*32+18)/* "" Hypervisor supports the VMCALL instruction */ #define X86_FEATURE_VMW_VMMCALL ( 8*32+19)/* "" VMware prefers VMMCALL hypercall instruction */ +#define X86_FEATURE_SEV_ES ( 8*32+20)/* AMD Secure Encrypted Virtualization - Encrypted State */ /* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */ #define X86_FEATURE_FSGSBASE ( 9*32+ 0)/* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
arch/x86/kernel/cpu/amd.c +2 -1
@@ -614,7 +614,7 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)* If BIOS has not enabled SME then don't advertise the * SME feature (set in scattered.c). * For SEV: If BIOS has not enabled SEV then don't advertise the -* SEV feature (set in scattered.c). +* SEV and SEV_ES feature (set in scattered.c). * * In all cases, since support for SME and SEV requires long mode, * don't advertise the feature under CONFIG_X86_32.
@@ -645,6 +645,7 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)setup_clear_cpu_cap (X86_FEATURE_SME ); clear_sev:setup_clear_cpu_cap (X86_FEATURE_SEV ); +setup_clear_cpu_cap (X86_FEATURE_SEV_ES ); } }
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -42,6 +42,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 }, { X86_FEATURE_SME, CPUID_EAX, 0, 0x8000001f, 0 }, { X86_FEATURE_SEV, CPUID_EAX, 1, 0x8000001f, 0 }, + { X86_FEATURE_SEV_ES, CPUID_EAX, 3, 0x8000001f, 0 }, { 0, 0, 0, 0, 0 } };
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/cpufeatures: Add SEV-ES CPU feature
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 360e7c5c4ca4fd8e627781ed42f95d58bc3bb732 Gitweb: https://git.kernel.org/tip/360e7c5c4ca4fd8e627781ed42f95d58bc3bb732 Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:06 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/cpufeatures: Add SEV-ES CPU feature
Add CPU feature detection for Secure Encrypted Virtualization with Encrypted State. This feature enhances SEV by also encrypting the guest register state, making it in-accessible to the hypervisor.
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/amd.c | 3 ++- arch/x86/kernel/cpu/scattered.c | 1 + 3 files changed, 4 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -236,6 +236,7 @@#define X86_FEATURE_EPT_AD ( 8*32+17)/* Intel Extended Page Table access-dirty bit */ #define X86_FEATURE_VMCALL ( 8*32+18)/* "" Hypervisor supports the VMCALL instruction */ #define X86_FEATURE_VMW_VMMCALL ( 8*32+19)/* "" VMware prefers VMMCALL hypercall instruction */ +#define X86_FEATURE_SEV_ES ( 8*32+20)/* AMD Secure Encrypted Virtualization - Encrypted State */ /* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */ #define X86_FEATURE_FSGSBASE ( 9*32+ 0)/* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
arch/x86/kernel/cpu/amd.c +2 -1
@@ -614,7 +614,7 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)* If BIOS has not enabled SME then don't advertise the * SME feature (set in scattered.c). * For SEV: If BIOS has not enabled SEV then don't advertise the -* SEV feature (set in scattered.c). +* SEV and SEV_ES feature (set in scattered.c). * * In all cases, since support for SME and SEV requires long mode, * don't advertise the feature under CONFIG_X86_32.
@@ -645,6 +645,7 @@ clear_all:setup_clear_cpu_cap (X86_FEATURE_SME ); clear_sev:setup_clear_cpu_cap (X86_FEATURE_SEV ); +setup_clear_cpu_cap (X86_FEATURE_SEV_ES ); } }
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -41,6 +41,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 }, { X86_FEATURE_SME, CPUID_EAX, 0, 0x8000001f, 0 }, { X86_FEATURE_SEV, CPUID_EAX, 1, 0x8000001f, 0 }, + { X86_FEATURE_SEV_ES, CPUID_EAX, 3, 0x8000001f, 0 }, { 0, 0, 0, 0, 0 } };
PATCH v7 6/72 x86/traps: Move pf error codes to <asm/trap_pf.h>
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Move the definition of the x86 page-fault error code bits to the new header file asm/trap_pf.h. This makes it easier to include them into pre-decompression boot code. No functional changes.
--- arch/x86/include/asm/trap_pf.h | 24 ++++++++++++++++++++++++ arch/x86/include/asm/traps.h | 19 +------------------ 2 files changed, 25 insertions(+), 18 deletions(-) create mode 100644 arch/x86/include/asm/trap_pf.h
arch/x86/include/asm/trap_pf.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_TRAP_PF_H +#define _ASM_X86_TRAP_PF_H + +/* +* Page fault error code bits: +* +* bit 0 == 0: no page found 1: protection fault +* bit 1 == 0: read access 1: write access +* bit 2 == 0: kernel-mode access 1: user-mode access +* bit 3 == 1: use of reserved bit detected +* bit 4 == 1: fault was an instruction fetch +* bit 5 == 1: protection keys block access +*/ +enum x86_pf_error_code { +X86_PF_PROT =1 <<0 , +X86_PF_WRITE =1 <<1 , +X86_PF_USER =1 <<2 , +X86_PF_RSVD =1 <<3 , +X86_PF_INSTR =1 <<4 , +X86_PF_PK =1 <<5 , +}; + +#endif /* _ASM_X86_TRAP_PF_H */
arch/x86/include/asm/traps.h +1 -18
@@ -8,6 +8,7 @@#include <asm/debugreg.h> #include <asm/idtentry.h> #include <asm/siginfo.h> /* TRAP_TRACE, ... */ +#include <asm/trap_pf.h> #ifdef CONFIG_X86_64 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
@@ -41,22 +42,4 @@ void __noreturn handle_stack_overflow(const char *message, unsigned long fault_address);#endif -/* -* Page fault error code bits: -* -* bit 0 == 0: no page found 1: protection fault -* bit 1 == 0: read access 1: write access -* bit 2 == 0: kernel-mode access 1: user-mode access -* bit 3 == 1: use of reserved bit detected -* bit 4 == 1: fault was an instruction fetch -* bit 5 == 1: protection keys block access -*/ -enum x86_pf_error_code { -X86_PF_PROT =1 <<0 , -X86_PF_WRITE =1 <<1 , -X86_PF_USER =1 <<2 , -X86_PF_RSVD =1 <<3 , -X86_PF_INSTR =1 <<4 , -X86_PF_PK =1 <<5 , -};#endif /* _ASM_X86_TRAPS_H */
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/traps: Move pf error codes to <asm/trap_pf.h>
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 05a2fdf3230306daee1def019b8f52cd06bd2e48 Gitweb: https://git.kernel.org/tip/05a2fdf3230306daee1def019b8f52cd06bd2e48 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:07 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/traps: Move pf error codes to <asm/trap_pf.h>
Move the definition of the x86 page-fault error code bits to a new header file asm/trap_pf.h. This makes it easier to include them into pre-decompression boot code. No functional changes.
--- arch/x86/include/asm/trap_pf.h | 24 ++++++++++++++++++++++++ arch/x86/include/asm/traps.h | 19 +------------------ 2 files changed, 25 insertions(+), 18 deletions(-) create mode 100644 arch/x86/include/asm/trap_pf.h
arch/x86/include/asm/trap_pf.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_TRAP_PF_H +#define _ASM_X86_TRAP_PF_H + +/* +* Page fault error code bits: +* +* bit 0 == 0: no page found 1: protection fault +* bit 1 == 0: read access 1: write access +* bit 2 == 0: kernel-mode access 1: user-mode access +* bit 3 == 1: use of reserved bit detected +* bit 4 == 1: fault was an instruction fetch +* bit 5 == 1: protection keys block access +*/ +enum x86_pf_error_code { +X86_PF_PROT =1 <<0 , +X86_PF_WRITE =1 <<1 , +X86_PF_USER =1 <<2 , +X86_PF_RSVD =1 <<3 , +X86_PF_INSTR =1 <<4 , +X86_PF_PK =1 <<5 , +}; + +#endif /* _ASM_X86_TRAP_PF_H */
arch/x86/include/asm/traps.h +1 -18
@@ -8,6 +8,7 @@#include <asm/debugreg.h> #include <asm/idtentry.h> #include <asm/siginfo.h> /* TRAP_TRACE, ... */ +#include <asm/trap_pf.h> #ifdef CONFIG_X86_64 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
@@ -41,22 +42,4 @@ void __noreturn handle_stack_overflow(const char *message, unsigned long fault_address);#endif -/* -* Page fault error code bits: -* -* bit 0 == 0: no page found 1: protection fault -* bit 1 == 0: read access 1: write access -* bit 2 == 0: kernel-mode access 1: user-mode access -* bit 3 == 1: use of reserved bit detected -* bit 4 == 1: fault was an instruction fetch -* bit 5 == 1: protection keys block access -*/ -enum x86_pf_error_code { -X86_PF_PROT =1 <<0 , -X86_PF_WRITE =1 <<1 , -X86_PF_USER =1 <<2 , -X86_PF_RSVD =1 <<3 , -X86_PF_INSTR =1 <<4 , -X86_PF_PK =1 <<5 , -};#endif /* _ASM_X86_TRAPS_H */
PATCH v7 7/72 x86/insn: Make inat-tables.c suitable for pre-decompression code
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The inat-tables.c file has some arrays in it that contain pointers to other arrays. These pointers need to be relocated when the kernel image is moved to a different location.
The pre-decompression boot-code has no support for applying ELF relocations, so initialize these arrays at runtime in the pre-decompression code to make sure all pointers are correctly initialized.
--- arch/x86/tools/gen-insn-attr-x86.awk | 50 +++++++++++++++++++++- tools/arch/x86/tools/gen-insn-attr-x86.awk | 50 +++++++++++++++++++++- 2 files changed, 98 insertions(+), 2 deletions(-)
arch/x86/tools/gen-insn-attr-x86.awk +49 -1
@@ -362,6 +362,9 @@ function convert_operands(count,opnd, i,j,imm,mod) END { if (awkchecked != "") exit 1 + + print "#ifndef __BOOT_COMPRESSED\n" + # print escape opcode map's array print "/* Escape opcode map array */" print "const insn_attr_t * const inat_escape_tables[INAT_ESC_MAX + 1]" \
@@ -388,6 +391,51 @@ END { for (j = 0; j < max_lprefix; j++) if (atable[i,j]) print " ["i"]["j"] = "atable[i,j]"," - print "};" + print "};\n" + + print "#else /* !__BOOT_COMPRESSED */\n" + + print "/* Escape opcode map array */" + print "static const insn_attr_t *inat_escape_tables[INAT_ESC_MAX + 1]" \ + "[INAT_LSTPFX_MAX + 1];" + print "" + + print "/* Group opcode map array */" + print "static const insn_attr_t *inat_group_tables[INAT_GRP_MAX + 1]"\ + "[INAT_LSTPFX_MAX + 1];" + print "" + + print "/* AVX opcode map array */" + print "static const insn_attr_t *inat_avx_tables[X86_VEX_M_MAX + 1]"\ + "[INAT_LSTPFX_MAX + 1];" + print "" + + print "static void inat_init_tables(void)" + print "{" + + # print escape opcode map's array + print "\t/* Print Escape opcode map array */" + for (i = 0; i < geid; i++) + for (j = 0; j < max_lprefix; j++) + if (etable[i,j]) + print "\tinat_escape_tables["i"]["j"] = "etable[i,j]";" + print "" + + # print group opcode map's array + print "\t/* Print Group opcode map array */" + for (i = 0; i < ggid; i++) + for (j = 0; j < max_lprefix; j++) + if (gtable[i,j]) + print "\tinat_group_tables["i"]["j"] = "gtable[i,j]";" + print "" + # print AVX opcode map's array + print "\t/* Print AVX opcode map array */" + for (i = 0; i < gaid; i++) + for (j = 0; j < max_lprefix; j++) + if (atable[i,j]) + print "\tinat_avx_tables["i"]["j"] = "atable[i,j]";" + + print "}" + print "#endif" }
tools/arch/x86/tools/gen-insn-attr-x86.awk +49 -1
61 lines not on this page · raw message
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/insn: Make inat-tables.c suitable for pre-decompression code
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 05a2ae7c033ee30f25fbed3ceed549a5cac398a9 Gitweb: https://git.kernel.org/tip/05a2ae7c033ee30f25fbed3ceed549a5cac398a9 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:08 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/insn: Make inat-tables.c suitable for pre-decompression code
The inat-tables.c file has some arrays in it that contain pointers to other arrays. These pointers need to be relocated when the kernel image is moved to a different location.
The pre-decompression boot-code has no support for applying ELF relocations, so initialize these arrays at runtime in the pre-decompression code to make sure all pointers are correctly initialized.
--- arch/x86/tools/gen-insn-attr-x86.awk | 50 ++++++++++++++++++++- tools/arch/x86/tools/gen-insn-attr-x86.awk | 50 ++++++++++++++++++++- 2 files changed, 98 insertions(+), 2 deletions(-)
arch/x86/tools/gen-insn-attr-x86.awk +49 -1
@@ -362,6 +362,9 @@ function convert_operands(count,opnd, i,j,imm,mod) END { if (awkchecked != "") exit 1 + + print "#ifndef __BOOT_COMPRESSED\n" + # print escape opcode map's array print "/* Escape opcode map array */" print "const insn_attr_t * const inat_escape_tables[INAT_ESC_MAX + 1]" \
@@ -388,6 +391,51 @@ END { for (j = 0; j < max_lprefix; j++) if (atable[i,j]) print " ["i"]["j"] = "atable[i,j]"," - print "};" + print "};\n" + + print "#else /* !__BOOT_COMPRESSED */\n" + + print "/* Escape opcode map array */" + print "static const insn_attr_t *inat_escape_tables[INAT_ESC_MAX + 1]" \ + "[INAT_LSTPFX_MAX + 1];" + print "" + + print "/* Group opcode map array */" + print "static const insn_attr_t *inat_group_tables[INAT_GRP_MAX + 1]"\ + "[INAT_LSTPFX_MAX + 1];" + print "" + + print "/* AVX opcode map array */" + print "static const insn_attr_t *inat_avx_tables[X86_VEX_M_MAX + 1]"\ + "[INAT_LSTPFX_MAX + 1];" + print "" + + print "static void inat_init_tables(void)" + print "{" + + # print escape opcode map's array + print "\t/* Print Escape opcode map array */" + for (i = 0; i < geid; i++) + for (j = 0; j < max_lprefix; j++) + if (etable[i,j]) + print "\tinat_escape_tables["i"]["j"] = "etable[i,j]";" + print "" + + # print group opcode map's array + print "\t/* Print Group opcode map array */" + for (i = 0; i < ggid; i++) + for (j = 0; j < max_lprefix; j++) + if (gtable[i,j]) + print "\tinat_group_tables["i"]["j"] = "gtable[i,j]";" + print "" + # print AVX opcode map's array + print "\t/* Print AVX opcode map array */" + for (i = 0; i < gaid; i++) + for (j = 0; j < max_lprefix; j++) + if (atable[i,j]) + print "\tinat_avx_tables["i"]["j"] = "atable[i,j]";" + + print "}" + print "#endif" }
tools/arch/x86/tools/gen-insn-attr-x86.awk +49 -1
61 lines not on this page · raw message
PATCH v7 8/72 x86/umip: Factor out instruction fetch
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Factor out the code to fetch the instruction from user-space to a helper function.
No functional changes.
--- arch/x86/include/asm/insn-eval.h | 2 ++ arch/x86/kernel/umip.c | 26 +++++----------------- arch/x86/lib/insn-eval.c | 38 ++++++++++++++++++++++++++++++++ 3 files changed, 46 insertions(+), 20 deletions(-)
arch/x86/include/asm/insn-eval.h +2 -0
@@ -19,5 +19,7 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);int insn_get_modrm_rm_off (struct insn *insn,struct pt_regs *regs);unsigned long insn_get_seg_base (struct pt_regs *regs,int seg_reg_idx);int insn_get_code_seg_params (struct pt_regs *regs); +int insn_fetch_from_user (struct pt_regs *regs, +unsigned char buf[MAX_INSN_SIZE ]);#endif /* _ASM_X86_INSN_EVAL_H */
arch/x86/kernel/umip.c +6 -20
@@ -335,11 +335,11 @@ static void force_sig_info_umip_fault(void __user *addr, struct pt_regs *regs)*/ bool fixup_umip_exception (struct pt_regs *regs) { -int not_copied, nr_copied, reg_offset, dummy_data_size, umip_inst; -unsigned long seg_base =0 , *reg_addr; +int nr_copied, reg_offset, dummy_data_size, umip_inst;/* 10 bytes is the maximum size of the result of UMIP instructions */ unsigned char dummy_data[10 ] = {0 };unsigned char buf[MAX_INSN_SIZE ]; +unsigned long *reg_addr;void __user *uaddr;struct insn insn;int seg_defs;
@@ -347,26 +347,12 @@ bool fixup_umip_exception(struct pt_regs *regs)if (!regs)return false; -/* -* If not in user-space long mode, a custom code segment could be in -* use. This is true in protected mode (if the process defined a local -* descriptor table), or virtual-8086 mode. In most of the cases -* seg_base will be zero as in USER_CS. -*/ -if (!user_64bit_mode (regs)) - seg_base =insn_get_seg_base (regs,INAT_SEG_REG_CS ); - -if (seg_base ==-1L ) -return false; - - not_copied =copy_from_user (buf, (void __user *)(seg_base + regs->ip), -sizeof (buf)); - nr_copied =sizeof (buf) - not_copied; + nr_copied =insn_fetch_from_user (regs, buf);/* -* The copy_from_user above could have failed if user code is protected -* by a memory protection key. Give up on emulation in such a case. -* Should we issue a page fault? +* The insn_fetch_from_user above could have failed if user code +* is protected by a memory protection key. Give up on emulation +* in such a case. Should we issue a page fault? */ if (!nr_copied)return false;
arch/x86/lib/insn-eval.c +38 -0
@@ -1367,3 +1367,41 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs)return (void __user *)-1L ; } } + +/** +* insn_fetch_from_user() - Copy instruction bytes from user-space memory +* @regs: Structure with register values as seen when entering kernel mode +* @buf: Array to store the fetched instruction +* +* Gets the linear address of the instruction and copies the instruction bytes +* to the buf. +* +* Returns: +* +* Number of instruction bytes copied. +* +* 0 if nothing was copied. +*/ +int insn_fetch_from_user (struct pt_regs *regs,unsigned char buf[MAX_INSN_SIZE ]) +{ +unsigned long seg_base =0 ; +int not_copied; + +/* +* If not in user-space long mode, a custom code segment could be in +* use. This is true in protected mode (if the process defined a local +* descriptor table), or virtual-8086 mode. In most of the cases +* seg_base will be zero as in USER_CS. +*/ +if (!user_64bit_mode (regs)) { + seg_base =insn_get_seg_base (regs,INAT_SEG_REG_CS ); +if (seg_base ==-1L ) +return 0 ; + } + + + not_copied =copy_from_user (buf, (void __user *)(seg_base + regs->ip), +MAX_INSN_SIZE ); + +return MAX_INSN_SIZE - not_copied; +}
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/umip: Factor out instruction fetch
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 172b75e56b08846e6fb07a88e5685ce4e24f4620 Gitweb: https://git.kernel.org/tip/172b75e56b08846e6fb07a88e5685ce4e24f4620 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:09 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/umip: Factor out instruction fetch
Factor out the code to fetch the instruction from user-space to a helper function.
No functional changes.
--- arch/x86/include/asm/insn-eval.h | 2 ++- arch/x86/kernel/umip.c | 26 ++++----------------- arch/x86/lib/insn-eval.c | 38 +++++++++++++++++++++++++++++++- 3 files changed, 46 insertions(+), 20 deletions(-)
arch/x86/include/asm/insn-eval.h +2 -0
@@ -19,5 +19,7 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);int insn_get_modrm_rm_off (struct insn *insn,struct pt_regs *regs);unsigned long insn_get_seg_base (struct pt_regs *regs,int seg_reg_idx);int insn_get_code_seg_params (struct pt_regs *regs); +int insn_fetch_from_user (struct pt_regs *regs, +unsigned char buf[MAX_INSN_SIZE ]);#endif /* _ASM_X86_INSN_EVAL_H */
arch/x86/kernel/umip.c +6 -20
@@ -335,11 +335,11 @@ static void force_sig_info_umip_fault(void __user *addr, struct pt_regs *regs)*/ bool fixup_umip_exception (struct pt_regs *regs) { -int not_copied, nr_copied, reg_offset, dummy_data_size, umip_inst; -unsigned long seg_base =0 , *reg_addr; +int nr_copied, reg_offset, dummy_data_size, umip_inst;/* 10 bytes is the maximum size of the result of UMIP instructions */ unsigned char dummy_data[10 ] = {0 };unsigned char buf[MAX_INSN_SIZE ]; +unsigned long *reg_addr;void __user *uaddr;struct insn insn;int seg_defs;
@@ -347,26 +347,12 @@ bool fixup_umip_exception(struct pt_regs *regs)if (!regs)return false; -/* -* If not in user-space long mode, a custom code segment could be in -* use. This is true in protected mode (if the process defined a local -* descriptor table), or virtual-8086 mode. In most of the cases -* seg_base will be zero as in USER_CS. -*/ -if (!user_64bit_mode (regs)) - seg_base =insn_get_seg_base (regs,INAT_SEG_REG_CS ); - -if (seg_base ==-1L ) -return false; - - not_copied =copy_from_user (buf, (void __user *)(seg_base + regs->ip), -sizeof (buf)); - nr_copied =sizeof (buf) - not_copied; + nr_copied =insn_fetch_from_user (regs, buf);/* -* The copy_from_user above could have failed if user code is protected -* by a memory protection key. Give up on emulation in such a case. -* Should we issue a page fault? +* The insn_fetch_from_user above could have failed if user code +* is protected by a memory protection key. Give up on emulation +* in such a case. Should we issue a page fault? */ if (!nr_copied)return false;
arch/x86/lib/insn-eval.c +38 -0
@@ -1367,3 +1367,41 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs)return (void __user *)-1L ; } } + +/** +* insn_fetch_from_user() - Copy instruction bytes from user-space memory +* @regs: Structure with register values as seen when entering kernel mode +* @buf: Array to store the fetched instruction +* +* Gets the linear address of the instruction and copies the instruction bytes +* to the buf. +* +* Returns: +* +* Number of instruction bytes copied. +* +* 0 if nothing was copied. +*/ +int insn_fetch_from_user (struct pt_regs *regs,unsigned char buf[MAX_INSN_SIZE ]) +{ +unsigned long seg_base =0 ; +int not_copied; + +/* +* If not in user-space long mode, a custom code segment could be in +* use. This is true in protected mode (if the process defined a local +* descriptor table), or virtual-8086 mode. In most of the cases +* seg_base will be zero as in USER_CS. +*/ +if (!user_64bit_mode (regs)) { + seg_base =insn_get_seg_base (regs,INAT_SEG_REG_CS ); +if (seg_base ==-1L ) +return 0 ; + } + + + not_copied =copy_from_user (buf, (void __user *)(seg_base + regs->ip), +MAX_INSN_SIZE ); + +return MAX_INSN_SIZE - not_copied; +}
PATCH v7 9/72 x86/umip: Factor out instruction decoding
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Factor out the code used to decode an instruction with the correct address and operand sizes to a helper function.
No functional changes.
--- arch/x86/include/asm/insn-eval.h | 2 ++ arch/x86/kernel/umip.c | 23 +--------------- arch/x86/lib/insn-eval.c | 45 ++++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+), 22 deletions(-)
arch/x86/include/asm/insn-eval.h +2 -0
@@ -21,5 +21,7 @@ unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);int insn_get_code_seg_params (struct pt_regs *regs);int insn_fetch_from_user (struct pt_regs *regs,unsigned char buf[MAX_INSN_SIZE ]); +bool insn_decode (struct insn *insn,struct pt_regs *regs, +unsigned char buf[MAX_INSN_SIZE ],int buf_size);#endif /* _ASM_X86_INSN_EVAL_H */
arch/x86/kernel/umip.c +1 -22
@@ -342,7 +342,6 @@ bool fixup_umip_exception(struct pt_regs *regs)unsigned long *reg_addr;void __user *uaddr;struct insn insn; -int seg_defs;if (!regs)return false;
@@ -357,27 +356,7 @@ bool fixup_umip_exception(struct pt_regs *regs)if (!nr_copied)return false; -insn_init (&insn, buf, nr_copied,user_64bit_mode (regs)); - -/* -* Override the default operand and address sizes with what is specified -* in the code segment descriptor. The instruction decoder only sets -* the address size it to either 4 or 8 address bytes and does nothing -* for the operand bytes. This OK for most of the cases, but we could -* have special cases where, for instance, a 16-bit code segment -* descriptor is used. -* If there is an address override prefix, the instruction decoder -* correctly updates these values, even for 16-bit defaults. -*/ - seg_defs =insn_get_code_seg_params (regs); -if (seg_defs == -EINVAL ) -return false; - - insn.addr_bytes =INSN_CODE_SEG_ADDR_SZ (seg_defs); - insn.opnd_bytes =INSN_CODE_SEG_OPND_SZ (seg_defs); - -insn_get_length (&insn); -if (nr_copied < insn.length) +if (!insn_decode (&insn, regs, buf, nr_copied))return false; umip_inst =identify_insn (&insn);
arch/x86/lib/insn-eval.c +45 -0
48 lines not on this page · raw message
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/umip: Factor out instruction decoding
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 172639d79977ca7b5ce6f84f6606262f4081718f Gitweb: https://git.kernel.org/tip/172639d79977ca7b5ce6f84f6606262f4081718f Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:10 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/umip: Factor out instruction decoding
Factor out the code used to decode an instruction with the correct address and operand sizes to a helper function.
No functional changes.
--- arch/x86/include/asm/insn-eval.h | 2 +- arch/x86/kernel/umip.c | 23 +---------------- arch/x86/lib/insn-eval.c | 45 +++++++++++++++++++++++++++++++- 3 files changed, 48 insertions(+), 22 deletions(-)
arch/x86/include/asm/insn-eval.h +2 -0
@@ -21,5 +21,7 @@ unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);int insn_get_code_seg_params (struct pt_regs *regs);int insn_fetch_from_user (struct pt_regs *regs,unsigned char buf[MAX_INSN_SIZE ]); +bool insn_decode (struct insn *insn,struct pt_regs *regs, +unsigned char buf[MAX_INSN_SIZE ],int buf_size);#endif /* _ASM_X86_INSN_EVAL_H */
arch/x86/kernel/umip.c +1 -22
@@ -342,7 +342,6 @@ bool fixup_umip_exception(struct pt_regs *regs)unsigned long *reg_addr;void __user *uaddr;struct insn insn; -int seg_defs;if (!regs)return false;
@@ -357,27 +356,7 @@ bool fixup_umip_exception(struct pt_regs *regs)if (!nr_copied)return false; -insn_init (&insn, buf, nr_copied,user_64bit_mode (regs)); - -/* -* Override the default operand and address sizes with what is specified -* in the code segment descriptor. The instruction decoder only sets -* the address size it to either 4 or 8 address bytes and does nothing -* for the operand bytes. This OK for most of the cases, but we could -* have special cases where, for instance, a 16-bit code segment -* descriptor is used. -* If there is an address override prefix, the instruction decoder -* correctly updates these values, even for 16-bit defaults. -*/ - seg_defs =insn_get_code_seg_params (regs); -if (seg_defs == -EINVAL ) -return false; - - insn.addr_bytes =INSN_CODE_SEG_ADDR_SZ (seg_defs); - insn.opnd_bytes =INSN_CODE_SEG_OPND_SZ (seg_defs); - -insn_get_length (&insn); -if (nr_copied < insn.length) +if (!insn_decode (&insn, regs, buf, nr_copied))return false; umip_inst =identify_insn (&insn);
arch/x86/lib/insn-eval.c +45 -0
48 lines not on this page · raw message
PATCH v7 10/72 x86/insn: Add insn_get_modrm_reg_off()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Add a function to the instruction decoder which returns the pt_regs offset of the register specified in the reg field of the modrm byte.
--- arch/x86/include/asm/insn-eval.h | 1 + arch/x86/lib/insn-eval.c | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+)
arch/x86/include/asm/insn-eval.h +1 -0
@@ -17,6 +17,7 @@void __user *insn_get_addr_ref (struct insn *insn,struct pt_regs *regs);int insn_get_modrm_rm_off (struct insn *insn,struct pt_regs *regs); +int insn_get_modrm_reg_off (struct insn *insn,struct pt_regs *regs);unsigned long insn_get_seg_base (struct pt_regs *regs,int seg_reg_idx);int insn_get_code_seg_params (struct pt_regs *regs); int insn_fetch_from_user(struct pt_regs *regs,
arch/x86/lib/insn-eval.c +23 -0
@@ -20,6 +20,7 @@enum reg_type {REG_TYPE_RM =0 , +REG_TYPE_REG ,REG_TYPE_INDEX ,REG_TYPE_BASE , };
@@ -439,6 +440,13 @@ static int get_reg_offset(struct insn *insn, struct pt_regs *regs, regno +=8 ;break ; +case REG_TYPE_REG : + regno =X86_MODRM_REG (insn->modrm.value); + +if (X86_REX_R (insn->rex_prefix.value)) + regno +=8 ; +break ; +case REG_TYPE_INDEX : regno =X86_SIB_INDEX (insn->sib.value);if (X86_REX_X (insn->rex_prefix.value))
@@ -807,6 +815,21 @@ int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs)return get_reg_offset (insn, regs,REG_TYPE_RM ); } +/** +* insn_get_modrm_reg_off() - Obtain register in reg part of the ModRM byte +* @insn: Instruction containing the ModRM byte +* @regs: Register values as seen when entering kernel mode +* +* Returns: +* +* The register indicated by the reg part of the ModRM byte. The +* register is obtained as an offset from the base of pt_regs. +*/ +int insn_get_modrm_reg_off (struct insn *insn,struct pt_regs *regs) +{ +return get_reg_offset (insn, regs,REG_TYPE_REG ); +} +/** * get_seg_base_limit() - obtain base address and limit of a segment * @insn: Instruction. Must be valid.
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/insn: Add insn_get_modrm_reg_off()
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 7af1bd822dd45a669fc178a35cc8183922333d56 Gitweb: https://git.kernel.org/tip/7af1bd822dd45a669fc178a35cc8183922333d56 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:11 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/insn: Add insn_get_modrm_reg_off()
Add a function to the instruction decoder which returns the pt_regs offset of the register specified in the reg field of the modrm byte.
--- arch/x86/include/asm/insn-eval.h | 1 + arch/x86/lib/insn-eval.c | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+)
arch/x86/include/asm/insn-eval.h +1 -0
@@ -17,6 +17,7 @@void __user *insn_get_addr_ref (struct insn *insn,struct pt_regs *regs);int insn_get_modrm_rm_off (struct insn *insn,struct pt_regs *regs); +int insn_get_modrm_reg_off (struct insn *insn,struct pt_regs *regs);unsigned long insn_get_seg_base (struct pt_regs *regs,int seg_reg_idx);int insn_get_code_seg_params (struct pt_regs *regs); int insn_fetch_from_user(struct pt_regs *regs,
arch/x86/lib/insn-eval.c +23 -0
@@ -20,6 +20,7 @@enum reg_type {REG_TYPE_RM =0 , +REG_TYPE_REG ,REG_TYPE_INDEX ,REG_TYPE_BASE , };
@@ -439,6 +440,13 @@ static int get_reg_offset(struct insn *insn, struct pt_regs *regs, regno +=8 ;break ; +case REG_TYPE_REG : + regno =X86_MODRM_REG (insn->modrm.value); + +if (X86_REX_R (insn->rex_prefix.value)) + regno +=8 ; +break ; +case REG_TYPE_INDEX : regno =X86_SIB_INDEX (insn->sib.value);if (X86_REX_X (insn->rex_prefix.value))
@@ -808,6 +816,21 @@ int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs) }/** +* insn_get_modrm_reg_off() - Obtain register in reg part of the ModRM byte +* @insn: Instruction containing the ModRM byte +* @regs: Register values as seen when entering kernel mode +* +* Returns: +* +* The register indicated by the reg part of the ModRM byte. The +* register is obtained as an offset from the base of pt_regs. +*/ +int insn_get_modrm_reg_off (struct insn *insn,struct pt_regs *regs) +{ +return get_reg_offset (insn, regs,REG_TYPE_REG ); +} + +/** * get_seg_base_limit() - obtain base address and limit of a segment * @insn: Instruction. Must be valid. * @regs: Register values as seen when entering kernel mode
PATCH v7 11/72 x86/insn: Add insn_has_rep_prefix() helper
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Add a function to check whether an instruction has a REP prefix.
--- arch/x86/include/asm/insn-eval.h | 1 + arch/x86/lib/insn-eval.c | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+)
arch/x86/include/asm/insn-eval.h +1 -0
@@ -15,6 +15,7 @@#define INSN_CODE_SEG_OPND_SZ (params) (params & 0xf)#define INSN_CODE_SEG_PARAMS (oper_sz, addr_sz) (oper_sz | (addr_sz << 4)) +bool insn_has_rep_prefix (struct insn *insn);void __user *insn_get_addr_ref (struct insn *insn,struct pt_regs *regs);int insn_get_modrm_rm_off (struct insn *insn,struct pt_regs *regs);int insn_get_modrm_reg_off (struct insn *insn,struct pt_regs *regs);
arch/x86/lib/insn-eval.c +24 -0
@@ -53,6 +53,30 @@ static bool is_string_insn(struct insn *insn) } } +/** +* insn_has_rep_prefix() - Determine if instruction has a REP prefix +* @insn: Instruction containing the prefix to inspect +* +* Returns: +* +* true if the instruction has a REP prefix, false if not. +*/ +bool insn_has_rep_prefix (struct insn *insn) +{ +int i; + +insn_get_prefixes (insn); + +for (i =0 ; i < insn->prefixes.nbytes; i++) { +insn_byte_t p = insn->prefixes.bytes[i]; + +if (p ==0xf2 || p ==0xf3 ) +return true; + } + +return false; +} +/** * get_seg_reg_override_idx() - obtain segment register override index * @insn: Valid instruction with segment override prefixes
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/insn: Add insn_has_rep_prefix() helper
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 5901781a11175a5e5ee91746ec8627f18d47eebd Gitweb: https://git.kernel.org/tip/5901781a11175a5e5ee91746ec8627f18d47eebd Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:12 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/insn: Add insn_has_rep_prefix() helper
Add a function to check whether an instruction has a REP prefix.
--- arch/x86/include/asm/insn-eval.h | 1 + arch/x86/lib/insn-eval.c | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+)
arch/x86/include/asm/insn-eval.h +1 -0
@@ -15,6 +15,7 @@#define INSN_CODE_SEG_OPND_SZ (params) (params & 0xf)#define INSN_CODE_SEG_PARAMS (oper_sz, addr_sz) (oper_sz | (addr_sz << 4)) +bool insn_has_rep_prefix (struct insn *insn);void __user *insn_get_addr_ref (struct insn *insn,struct pt_regs *regs);int insn_get_modrm_rm_off (struct insn *insn,struct pt_regs *regs);int insn_get_modrm_reg_off (struct insn *insn,struct pt_regs *regs);
arch/x86/lib/insn-eval.c +24 -0
@@ -54,6 +54,30 @@ static bool is_string_insn(struct insn *insn) }/** +* insn_has_rep_prefix() - Determine if instruction has a REP prefix +* @insn: Instruction containing the prefix to inspect +* +* Returns: +* +* true if the instruction has a REP prefix, false if not. +*/ +bool insn_has_rep_prefix (struct insn *insn) +{ +int i; + +insn_get_prefixes (insn); + +for (i =0 ; i < insn->prefixes.nbytes; i++) { +insn_byte_t p = insn->prefixes.bytes[i]; + +if (p ==0xf2 || p ==0xf3 ) +return true; + } + +return false; +} + +/** * get_seg_reg_override_idx() - obtain segment register override index * @insn: Valid instruction with segment override prefixes *
PATCH v7 12/72 x86/boot/compressed/64: Disable red-zone usage
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The x86-64 ABI defines a red-zone on the stack:
The 128-byte area beyond the location pointed to by %rsp is considered to be reserved and shall not be modified by signal or interrupt handlers. Therefore, functions may use this area for temporary data that is not needed across function calls. In particular, leaf functions may use this area for their entire stack frame, rather than adjusting the stack pointer in the prologue and epilogue. This area is known as the red zone.
This is not compatible with exception handling, because the IRET frame written by the hardware at the stack pointer and the functions to handle the exception will overwrite the temporary variables of the interrupted function, causing undefined behavior. So disable red-zones for the pre-decompression boot code.
--- arch/x86/boot/compressed/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/boot/compressed/Makefile +1 -1
@@ -32,7 +32,7 @@ KBUILD_CFLAGS := -m$(BITS) -O2 KBUILD_CFLAGS += -fno-strict-aliasing -fPIE KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING cflags-$(CONFIG_X86_32) := -march=i386 -cflags-$(CONFIG_X86_64) := -mcmodel=small +cflags-$(CONFIG_X86_64) := -mcmodel=small -mno-red-zone KBUILD_CFLAGS += $(cflags-y) KBUILD_CFLAGS += -mno-mmx -mno-sse KBUILD_CFLAGS += -ffreestanding
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Disable red-zone usage
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 6ba0efa46047936afa81460489cfd24bc95dd863 Gitweb: https://git.kernel.org/tip/6ba0efa46047936afa81460489cfd24bc95dd863 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:13 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Disable red-zone usage
The x86-64 ABI defines a red-zone on the stack:
The 128-byte area beyond the location pointed to by %rsp is considered to be reserved and shall not be modified by signal or interrupt handlers. Therefore, functions may use this area for temporary data that is not needed across function calls. In particular, leaf functions may use this area for their entire stack frame, rather than adjusting the stack pointer in the prologue and epilogue. This area is known as the red zone.
This is not compatible with exception handling, because the IRET frame written by the hardware at the stack pointer and the functions to handle the exception will overwrite the temporary variables of the interrupted function, causing undefined behavior. So disable red-zones for the pre-decompression boot code.
--- arch/x86/boot/compressed/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/boot/compressed/Makefile +1 -1
@@ -32,7 +32,7 @@ KBUILD_CFLAGS := -m$(BITS) -O2 KBUILD_CFLAGS += -fno-strict-aliasing $(call cc-option, -fPIE, -fPIC) KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING cflags-$(CONFIG_X86_32) := -march=i386 -cflags-$(CONFIG_X86_64) := -mcmodel=small +cflags-$(CONFIG_X86_64) := -mcmodel=small -mno-red-zone KBUILD_CFLAGS += $(cflags-y) KBUILD_CFLAGS += -mno-mmx -mno-sse KBUILD_CFLAGS += -ffreestanding
PATCH v7 13/72 x86/boot/compressed/64: Add IDT Infrastructure
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Add code needed to setup an IDT in the early pre-decompression boot-code. The IDT is loaded first in startup_64, which is after EfiExitBootServices() has been called, and later reloaded when the kernel image has been relocated to the end of the decompression area.
This allows to setup different IDT handlers before and after the relocation.
--- arch/x86/boot/compressed/Makefile | 1 + arch/x86/boot/compressed/head_64.S | 25 +++++++- arch/x86/boot/compressed/idt_64.c | 44 ++++++++++++++ arch/x86/boot/compressed/idt_handlers_64.S | 70 ++++++++++++++++++++++ arch/x86/boot/compressed/misc.h | 5 ++ arch/x86/include/asm/desc_defs.h | 3 + 6 files changed, 147 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/idt_64.c create mode 100644 arch/x86/boot/compressed/idt_handlers_64.S
arch/x86/boot/compressed/Makefile +1 -0
@@ -80,6 +80,7 @@ vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o ifdef CONFIG_X86_64 vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o + vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o vmlinux-objs-y += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o endif
arch/x86/boot/compressed/head_64.S +24 -1
@@ -33,6 +33,7 @@ #include <asm/processor-flags.h> #include <asm/asm-offsets.h> #include <asm/bootparam.h> +#include <asm/desc_defs.h> #include "pgtable.h" /*
@@ -415,6 +416,10 @@ SYM_CODE_START(startup_64) .Lon_kernel_cs: + pushq %rsi + call load_stage1_idt + popq %rsi + /* * paging_prepare() sets up the trampoline and checks if we need to * enable 5-level paging.
@@ -527,6 +532,13 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) shrq $3, %rcx rep stosq +/* + * Load stage2 IDT + */ + pushq %rsi + call load_stage2_idt + popq %rsi + /* * Do the extraction, and jump to the new kernel.. */
@@ -659,10 +671,21 @@ SYM_DATA_START_LOCAL(gdt) .quad 0x0000000000000000 /* TS continued */ SYM_DATA_END_LABEL(gdt, SYM_L_LOCAL, gdt_end) +SYM_DATA_START(boot_idt_desc) + .word boot_idt_end - boot_idt - 1 + .quad 0 +SYM_DATA_END(boot_idt_desc) + .balign 8 +SYM_DATA_START(boot_idt) + .rept BOOT_IDT_ENTRIES + .quad 0 + .quad 0 + .endr +SYM_DATA_END_LABEL(boot_idt, SYM_L_GLOBAL, boot_idt_end) + #ifdef CONFIG_EFI_STUB SYM_DATA(image_offset, .long 0) #endif - #ifdef CONFIG_EFI_MIXED SYM_DATA_LOCAL(efi32_boot_args, .long 0, 0, 0) SYM_DATA(efi_is64, .byte 1)
arch/x86/boot/compressed/idt_64.c new file +44 -0
44 lines not on this page · raw message
arch/x86/boot/compressed/idt_handlers_64.S new file +70 -0
70 lines not on this page · raw message
arch/x86/boot/compressed/misc.h +5 -0
@@ -23,6 +23,7 @@#include <asm/page.h> #include <asm/boot.h> #include <asm/bootparam.h> +#include <asm/desc_defs.h> #define BOOT_CTYPE_H #include <linux/acpi.h>
@@ -133,4 +134,8 @@ int count_immovable_mem_regions(void);static inline int count_immovable_mem_regions (void ) {return 0 ; }#endif +/* idt_64.c */ +extern gate_desc boot_idt[BOOT_IDT_ENTRIES ]; +extern struct desc_ptr boot_idt_desc; +#endif /* BOOT_COMPRESSED_MISC_H */
arch/x86/include/asm/desc_defs.h +3 -0
@@ -109,6 +109,9 @@ struct desc_ptr {#endif /* !__ASSEMBLY__ */ +/* Boot IDT definitions */ +#define BOOT_IDT_ENTRIES 32 +/* Access rights as returned by LAR */ #define AR_TYPE_RODATA (0 * (1 << 9))#define AR_TYPE_RWDATA (1 * (1 << 9))
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Add IDT Infrastructure
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 64e682638eb51070ba6044535b250aad43c5564e Gitweb: https://git.kernel.org/tip/64e682638eb51070ba6044535b250aad43c5564e Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:14 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Add IDT Infrastructure
Add code needed to setup an IDT in the early pre-decompression boot-code. The IDT is loaded first in startup_64, which is after EfiExitBootServices() has been called, and later reloaded when the kernel image has been relocated to the end of the decompression area.
This allows to setup different IDT handlers before and after the relocation.
--- arch/x86/boot/compressed/Makefile | 1 +- arch/x86/boot/compressed/head_64.S | 25 +++++++- arch/x86/boot/compressed/idt_64.c | 44 +++++++++++++- arch/x86/boot/compressed/idt_handlers_64.S | 70 +++++++++++++++++++++- arch/x86/boot/compressed/misc.h | 5 ++- arch/x86/include/asm/desc_defs.h | 3 +- 6 files changed, 147 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/idt_64.c create mode 100644 arch/x86/boot/compressed/idt_handlers_64.S
arch/x86/boot/compressed/Makefile +1 -0
@@ -85,6 +85,7 @@ vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o ifdef CONFIG_X86_64 vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o + vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o vmlinux-objs-y += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o endif
arch/x86/boot/compressed/head_64.S +24 -1
@@ -33,6 +33,7 @@ #include <asm/processor-flags.h> #include <asm/asm-offsets.h> #include <asm/bootparam.h> +#include <asm/desc_defs.h> #include "pgtable.h" /*
@@ -410,6 +411,10 @@ SYM_CODE_START(startup_64) .Lon_kernel_cs: + pushq %rsi + call load_stage1_idt + popq %rsi + /* * paging_prepare() sets up the trampoline and checks if we need to * enable 5-level paging.
@@ -538,6 +543,13 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) rep stosq /* + * Load stage2 IDT + */ + pushq %rsi + call load_stage2_idt + popq %rsi + +/* * Do the extraction, and jump to the new kernel.. */ pushq %rsi /* Save the real mode argument */
@@ -690,10 +702,21 @@ SYM_DATA_START_LOCAL(gdt) .quad 0x0000000000000000 /* TS continued */ SYM_DATA_END_LABEL(gdt, SYM_L_LOCAL, gdt_end) +SYM_DATA_START(boot_idt_desc) + .word boot_idt_end - boot_idt - 1 + .quad 0 +SYM_DATA_END(boot_idt_desc) + .balign 8 +SYM_DATA_START(boot_idt) + .rept BOOT_IDT_ENTRIES + .quad 0 + .quad 0 + .endr +SYM_DATA_END_LABEL(boot_idt, SYM_L_GLOBAL, boot_idt_end) + #ifdef CONFIG_EFI_STUB SYM_DATA(image_offset, .long 0) #endif - #ifdef CONFIG_EFI_MIXED SYM_DATA_LOCAL(efi32_boot_args, .long 0, 0, 0) SYM_DATA(efi_is64, .byte 1)
arch/x86/boot/compressed/idt_64.c new file +44 -0
44 lines not on this page · raw message
arch/x86/boot/compressed/idt_handlers_64.S new file +70 -0
70 lines not on this page · raw message
arch/x86/boot/compressed/misc.h +5 -0
@@ -23,6 +23,7 @@#include <asm/page.h> #include <asm/boot.h> #include <asm/bootparam.h> +#include <asm/desc_defs.h> #define BOOT_CTYPE_H #include <linux/acpi.h>
@@ -133,4 +134,8 @@ int count_immovable_mem_regions(void);static inline int count_immovable_mem_regions (void ) {return 0 ; }#endif +/* idt_64.c */ +extern gate_desc boot_idt[BOOT_IDT_ENTRIES ]; +extern struct desc_ptr boot_idt_desc; +#endif /* BOOT_COMPRESSED_MISC_H */
arch/x86/include/asm/desc_defs.h +3 -0
@@ -109,6 +109,9 @@ struct desc_ptr {#endif /* !__ASSEMBLY__ */ +/* Boot IDT definitions */ +#define BOOT_IDT_ENTRIES 32 +/* Access rights as returned by LAR */ #define AR_TYPE_RODATA (0 * (1 << 9))#define AR_TYPE_RWDATA (1 * (1 << 9))
PATCH v7 14/72 x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The file contains only code related to identity mapped page-tables. Rename the file and compile it always in.
---
arch/x86/boot/compressed/Makefile | 2 +-
arch/x86/boot/compressed/{kaslr_64.c => ident_map_64.c} | 9 +++++++++
arch/x86/boot/compressed/kaslr.c | 9 ---------
arch/x86/boot/compressed/misc.h | 8 ++++++++
4 files changed, 18 insertions(+), 10 deletions(-)
rename arch/x86/boot/compressed/{kaslr_64.c => ident_map_64.c} (95%)arch/x86/boot/compressed/Makefile +1 -1
@@ -79,7 +79,7 @@ vmlinux-objs-y := $(obj)/vmlinux.lds $(obj)/kernel_info.o $(obj)/head_$(BITS).o vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o ifdef CONFIG_X86_64 - vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o + vmlinux-objs-y += $(obj)/ident_map_64.o vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o vmlinux-objs-y += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o
arch/x86/boot/compressed/ident_map_64.c renamed from arch/x86/boot/compressed/kaslr_64.c +9 -0
@@ -29,6 +29,15 @@#define __PAGE_OFFSET __PAGE_OFFSET_BASE#include "../../mm/ident_map.c" +#ifdef CONFIG_X86_5LEVEL +unsigned int __pgtable_l5_enabled; +unsigned int pgdir_shift =39 ; +unsigned int ptrs_per_p4d =1 ; +#endif + +/* Used by PAGE_KERN* macros: */ +pteval_t __default_kernel_pte_mask__read_mostly = ~0 ; +/* Used to track our page table allocation area. */ struct alloc_pgt_data {unsigned char *pgt_buf;
arch/x86/boot/compressed/kaslr.c +0 -9
@@ -40,17 +40,8 @@#include <asm/setup.h> /* For COMMAND_LINE_SIZE */ #undef _SETUP -#ifdef CONFIG_X86_5LEVEL -unsigned int __pgtable_l5_enabled; -unsigned int pgdir_shift__ro_after_init =39 ; -unsigned int ptrs_per_p4d__ro_after_init =1 ; -#endif -extern unsigned long get_cmd_line_ptr (void ); -/* Used by PAGE_KERN* macros: */ -pteval_t __default_kernel_pte_mask__read_mostly = ~0 ; -/* Simplified build-specific string for starting entropy. */ static const char build_str[] =UTS_RELEASE " (" LINUX_COMPILE_BY "@" LINUX_COMPILE_HOST ") (" LINUX_COMPILER ") " UTS_VERSION ;
arch/x86/boot/compressed/misc.h +8 -0
@@ -134,6 +134,14 @@ int count_immovable_mem_regions(void);static inline int count_immovable_mem_regions (void ) {return 0 ; }#endif +/* ident_map_64.c */ +#ifdef CONFIG_X86_5LEVEL +extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d; +#endif + +/* Used by PAGE_KERN* macros: */ +extern pteval_t __default_kernel_pte_mask; +/* idt_64.c */ extern gate_desc boot_idt[BOOT_IDT_ENTRIES ];extern struct desc_ptr boot_idt_desc;
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 5f2bb01682b7b067783207994c7b8a3dbeb1cd83 Gitweb: https://git.kernel.org/tip/5f2bb01682b7b067783207994c7b8a3dbeb1cd83 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:15 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c
The file contains only code related to identity-mapped page tables. Rename the file and compile it always in.
--- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/ident_map_64.c | 162 +++++++++++++++++++++++- arch/x86/boot/compressed/kaslr.c | 9 +- arch/x86/boot/compressed/kaslr_64.c | 153 +---------------------- arch/x86/boot/compressed/misc.h | 8 +- 5 files changed, 171 insertions(+), 163 deletions(-) create mode 100644 arch/x86/boot/compressed/ident_map_64.c delete mode 100644 arch/x86/boot/compressed/kaslr_64.c
arch/x86/boot/compressed/Makefile +1 -1
@@ -84,7 +84,7 @@ vmlinux-objs-y := $(obj)/vmlinux.lds $(obj)/kernel_info.o $(obj)/head_$(BITS).o vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o ifdef CONFIG_X86_64 - vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o + vmlinux-objs-y += $(obj)/ident_map_64.o vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o vmlinux-objs-y += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o
arch/x86/boot/compressed/ident_map_64.c new file +162 -0
162 lines not on this page · raw message
arch/x86/boot/compressed/kaslr.c +0 -9
@@ -40,17 +40,8 @@#include <asm/setup.h> /* For COMMAND_LINE_SIZE */ #undef _SETUP -#ifdef CONFIG_X86_5LEVEL -unsigned int __pgtable_l5_enabled; -unsigned int pgdir_shift__ro_after_init =39 ; -unsigned int ptrs_per_p4d__ro_after_init =1 ; -#endif -extern unsigned long get_cmd_line_ptr (void ); -/* Used by PAGE_KERN* macros: */ -pteval_t __default_kernel_pte_mask__read_mostly = ~0 ; -/* Simplified build-specific string for starting entropy. */ static const char build_str[] =UTS_RELEASE " (" LINUX_COMPILE_BY "@" LINUX_COMPILE_HOST ") (" LINUX_COMPILER ") " UTS_VERSION ;
arch/x86/boot/compressed/kaslr_64.c deleted +0 -153
153 lines not on this page · raw message
arch/x86/boot/compressed/misc.h +8 -0
@@ -134,6 +134,14 @@ int count_immovable_mem_regions(void);static inline int count_immovable_mem_regions (void ) {return 0 ; }#endif +/* ident_map_64.c */ +#ifdef CONFIG_X86_5LEVEL +extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d; +#endif + +/* Used by PAGE_KERN* macros: */ +extern pteval_t __default_kernel_pte_mask; +/* idt_64.c */ extern gate_desc boot_idt[BOOT_IDT_ENTRIES ];extern struct desc_ptr boot_idt_desc;
PATCH v7 15/72 x86/boot/compressed/64: Add page-fault handler
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Install a page-fault handler to add an identity mapping to addresses not yet mapped. Also do some checking whether the error code is sane.
This makes non SEV-ES machines use the exception handling infrastructure in the pre-decompressions boot code too, making it less likely to break in the future.
--- arch/x86/boot/compressed/ident_map_64.c | 39 ++++++++++++++++++++++ arch/x86/boot/compressed/idt_64.c | 2 ++ arch/x86/boot/compressed/idt_handlers_64.S | 2 ++ arch/x86/boot/compressed/misc.h | 6 ++++ 4 files changed, 49 insertions(+)
arch/x86/boot/compressed/ident_map_64.c +39 -0
@@ -19,10 +19,13 @@/* No PAGE_TABLE_ISOLATION support needed either: */ #undef CONFIG_PAGE_TABLE_ISOLATION +#include "error.h" #include "misc.h" /* These actually do the work of building the kernel identity maps. */ #include <linux/pgtable.h> +#include <asm/trap_pf.h> +#include <asm/trapnr.h> #include <asm/init.h> /* Use the static base for this part of the boot process */ #undef __PAGE_OFFSET
@@ -160,3 +163,39 @@ void finalize_identity_maps(void) {write_cr3 (top_level_pgt); } + +static void do_pf_error (const char *msg,unsigned long error_code, +unsigned long address,unsigned long ip) +{ +error_putstr (msg); + +error_putstr ("\nError Code: " ); +error_puthex (error_code); +error_putstr ("\nCR2: 0x" ); +error_puthex (address); +error_putstr ("\nRIP relative to _head: 0x" ); +error_puthex (ip - (unsigned long )_head); +error_putstr ("\n" ); + +error ("Stopping.\n" ); +} + +void do_boot_page_fault (struct pt_regs *regs,unsigned long error_code) +{ +unsigned long address =native_read_cr2 (); + +/* +* Check for unexpected error codes. Unexpected are: +* - Faults on present pages +* - User faults +* - Reserved bits set +*/ +if (error_code & (X86_PF_PROT |X86_PF_USER |X86_PF_RSVD )) +do_pf_error ("Unexpected page-fault:" , error_code, address, regs->ip); + +/* +* Error code is sane - now identity map the 2M region around +* the faulting address. +*/ +add_identity_map (address &PMD_MASK ,PMD_SIZE ); +}
arch/x86/boot/compressed/idt_64.c +2 -0
@@ -40,5 +40,7 @@ void load_stage2_idt(void) { boot_idt_desc.address = (unsigned long )boot_idt; +set_idt_entry (X86_TRAP_PF , boot_page_fault); +load_boot_idt (&boot_idt_desc); }
arch/x86/boot/compressed/idt_handlers_64.S +2 -0
@@ -68,3 +68,5 @@ SYM_FUNC_END(\name) .text .code64 + +EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
arch/x86/boot/compressed/misc.h +6 -0
@@ -37,6 +37,9 @@#define memptr unsigned#endif +/* boot/compressed/vmlinux start and end markers */ +extern char _head[], _end[]; +/* misc.c */ extern memptr free_mem_ptr;extern memptr free_mem_end_ptr;
@@ -146,4 +149,7 @@ extern pteval_t __default_kernel_pte_mask;extern gate_desc boot_idt[BOOT_IDT_ENTRIES ];extern struct desc_ptr boot_idt_desc; +/* IDT Entry Points */ +void boot_page_fault (void ); +#endif /* BOOT_COMPRESSED_MISC_H */
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Add page-fault handler
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 8b0d3b3b41ab6f14f1ce6d4a6b1c5f60b825123f Gitweb: https://git.kernel.org/tip/8b0d3b3b41ab6f14f1ce6d4a6b1c5f60b825123f Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:16 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Add page-fault handler
Install a page-fault handler to add an identity mapping to addresses not yet mapped. Also do some checking whether the error code is sane.
This makes non SEV-ES machines use the exception handling infrastructure in the pre-decompressions boot code too, making it less likely to break in the future.
--- arch/x86/boot/compressed/ident_map_64.c | 39 +++++++++++++++++++++- arch/x86/boot/compressed/idt_64.c | 2 +- arch/x86/boot/compressed/idt_handlers_64.S | 2 +- arch/x86/boot/compressed/misc.h | 6 +++- 4 files changed, 49 insertions(+)
arch/x86/boot/compressed/ident_map_64.c +39 -0
@@ -19,10 +19,13 @@/* No PAGE_TABLE_ISOLATION support needed either: */ #undef CONFIG_PAGE_TABLE_ISOLATION +#include "error.h" #include "misc.h" /* These actually do the work of building the kernel identity maps. */ #include <linux/pgtable.h> +#include <asm/trap_pf.h> +#include <asm/trapnr.h> #include <asm/init.h> /* Use the static base for this part of the boot process */ #undef __PAGE_OFFSET
@@ -160,3 +163,39 @@ void finalize_identity_maps(void) {write_cr3 (top_level_pgt); } + +static void do_pf_error (const char *msg,unsigned long error_code, +unsigned long address,unsigned long ip) +{ +error_putstr (msg); + +error_putstr ("\nError Code: " ); +error_puthex (error_code); +error_putstr ("\nCR2: 0x" ); +error_puthex (address); +error_putstr ("\nRIP relative to _head: 0x" ); +error_puthex (ip - (unsigned long )_head); +error_putstr ("\n" ); + +error ("Stopping.\n" ); +} + +void do_boot_page_fault (struct pt_regs *regs,unsigned long error_code) +{ +unsigned long address =native_read_cr2 (); + +/* +* Check for unexpected error codes. Unexpected are: +* - Faults on present pages +* - User faults +* - Reserved bits set +*/ +if (error_code & (X86_PF_PROT |X86_PF_USER |X86_PF_RSVD )) +do_pf_error ("Unexpected page-fault:" , error_code, address, regs->ip); + +/* +* Error code is sane - now identity map the 2M region around +* the faulting address. +*/ +add_identity_map (address &PMD_MASK ,PMD_SIZE ); +}
arch/x86/boot/compressed/idt_64.c +2 -0
@@ -40,5 +40,7 @@ void load_stage2_idt(void) { boot_idt_desc.address = (unsigned long )boot_idt; +set_idt_entry (X86_TRAP_PF , boot_page_fault); +load_boot_idt (&boot_idt_desc); }
arch/x86/boot/compressed/idt_handlers_64.S +2 -0
@@ -68,3 +68,5 @@ SYM_FUNC_END(\name) .text .code64 + +EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
arch/x86/boot/compressed/misc.h +6 -0
@@ -37,6 +37,9 @@#define memptr unsigned#endif +/* boot/compressed/vmlinux start and end markers */ +extern char _head[], _end[]; +/* misc.c */ extern memptr free_mem_ptr;extern memptr free_mem_end_ptr;
@@ -146,4 +149,7 @@ extern pteval_t __default_kernel_pte_mask;extern gate_desc boot_idt[BOOT_IDT_ENTRIES ];extern struct desc_ptr boot_idt_desc; +/* IDT Entry Points */ +void boot_page_fault (void ); +#endif /* BOOT_COMPRESSED_MISC_H */
PATCH v7 16/72 x86/boot/compressed/64: Always switch to own page-table
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
When booted through startup_64 the kernel keeps running on the EFI page-table until the KASLR code sets up its own page-table. Without KASLR the pre-decompression boot code never switches off the EFI page-table. Change that by unconditionally switching to a kernel controlled page-table after relocation.
This makes sure we can make changes to the mapping when necessary, for example map pages unencrypted in SEV and SEV-ES guests.
Also remove the debug_putstr() calls in initialize_identity_maps() because the function now runs before console_init() is called.
--- arch/x86/boot/compressed/head_64.S | 3 +- arch/x86/boot/compressed/ident_map_64.c | 51 +++++++++++++++---------- arch/x86/boot/compressed/kaslr.c | 3 -- 3 files changed, 32 insertions(+), 25 deletions(-)
arch/x86/boot/compressed/head_64.S +2 -1
@@ -533,10 +533,11 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) rep stosq /* - * Load stage2 IDT + * Load stage2 IDT and switch to our own page-table */ pushq %rsi call load_stage2_idt + call initialize_identity_maps popq %rsi /*
arch/x86/boot/compressed/ident_map_64.c +30 -21
76 lines not on this page · raw message
arch/x86/boot/compressed/kaslr.c +0 -3
@@ -861,9 +861,6 @@ void choose_random_location(unsigned long input, boot_params->hdr.loadflags |=KASLR_FLAG ; -/* Prepare to add new identity pagetables on demand. */ -initialize_identity_maps (); -if (IS_ENABLED (CONFIG_X86_32 )) mem_limit =KERNEL_IMAGE_SIZE ;else
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Always switch to own page table
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: ca0e22d4f011a56e974fa3a712d76e86a791559d Gitweb: https://git.kernel.org/tip/ca0e22d4f011a56e974fa3a712d76e86a791559d Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:17 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Always switch to own page table
When booted through startup_64(), the kernel keeps running on the EFI page table until the KASLR code sets up its own page table. Without KASLR, the pre-decompression boot code never switches off the EFI page table. Change that by unconditionally switching to a kernel-controlled page table after relocation.
This makes sure the kernel can make changes to the mapping when necessary, for example map pages unencrypted in SEV and SEV-ES guests.
Also, remove the debug_putstr() calls in initialize_identity_maps() because the function now runs before console_init() is called.
[ bp: Massage commit message. ]
--- arch/x86/boot/compressed/head_64.S | 3 +- arch/x86/boot/compressed/ident_map_64.c | 51 ++++++++++++++---------- arch/x86/boot/compressed/kaslr.c | 3 +- 3 files changed, 32 insertions(+), 25 deletions(-)
arch/x86/boot/compressed/head_64.S +2 -1
@@ -543,10 +543,11 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) rep stosq /* - * Load stage2 IDT + * Load stage2 IDT and switch to our own page-table */ pushq %rsi call load_stage2_idt + call initialize_identity_maps popq %rsi /*
arch/x86/boot/compressed/ident_map_64.c +30 -21
76 lines not on this page · raw message
arch/x86/boot/compressed/kaslr.c +0 -3
@@ -861,9 +861,6 @@ void choose_random_location(unsigned long input, boot_params->hdr.loadflags |=KASLR_FLAG ; -/* Prepare to add new identity pagetables on demand. */ -initialize_identity_maps (); -if (IS_ENABLED (CONFIG_X86_32 )) mem_limit =KERNEL_IMAGE_SIZE ;else
PATCH v7 17/72 x86/boot/compressed/64: Don't pre-map memory in KASLR code
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
With the page-fault handler in place the identity mapping can be built on-demand. So remove the code which manually creates the mappings and unexport/remove the functions used for it.
--- arch/x86/boot/compressed/ident_map_64.c | 6 ++---- arch/x86/boot/compressed/kaslr.c | 24 +----------------------- arch/x86/boot/compressed/misc.h | 10 ---------- 3 files changed, 3 insertions(+), 37 deletions(-)
arch/x86/boot/compressed/ident_map_64.c +2 -4
@@ -87,11 +87,9 @@ phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1;static struct x86_mapping_info mapping_info;/* -* Adds the specified range to what will become the new identity mappings. -* Once all ranges have been added, the new mapping is activated by calling -* finalize_identity_maps() below. +* Adds the specified range to the identity mappings. */ -void add_identity_map (unsigned long start,unsigned long size) +static void add_identity_map (unsigned long start,unsigned long size) {unsigned long end = start + size;
arch/x86/boot/compressed/kaslr.c +1 -23
@@ -397,8 +397,6 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,*/ mem_avoid[MEM_AVOID_ZO_RANGE ].start = input; mem_avoid[MEM_AVOID_ZO_RANGE ].size = (output + init_size) - input; -add_identity_map (mem_avoid[MEM_AVOID_ZO_RANGE ].start, - mem_avoid[MEM_AVOID_ZO_RANGE ].size);/* Avoid initrd. */ initrd_start = (u64 )boot_params->ext_ramdisk_image <<32 ;
@@ -416,15 +414,11 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size, cmd_line_size =strnlen ((char *)cmd_line,COMMAND_LINE_SIZE -1 ) +1 ; mem_avoid[MEM_AVOID_CMDLINE ].start = cmd_line; mem_avoid[MEM_AVOID_CMDLINE ].size = cmd_line_size; -add_identity_map (mem_avoid[MEM_AVOID_CMDLINE ].start, - mem_avoid[MEM_AVOID_CMDLINE ].size); }/* Avoid boot parameters. */ mem_avoid[MEM_AVOID_BOOTPARAMS ].start = (unsigned long )boot_params; mem_avoid[MEM_AVOID_BOOTPARAMS ].size =sizeof (*boot_params); -add_identity_map (mem_avoid[MEM_AVOID_BOOTPARAMS ].start, - mem_avoid[MEM_AVOID_BOOTPARAMS ].size);/* We don't need to set a mapping for setup_data. */
@@ -433,11 +427,6 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,/* Enumerate the immovable memory regions */ num_immovable_mem =count_immovable_mem_regions (); - -#ifdef CONFIG_X86_VERBOSE_BOOTUP -/* Make sure video RAM can be used. */ -add_identity_map (0 ,PMD_SIZE ); -#endif }/*
@@ -884,19 +873,8 @@ void choose_random_location(unsigned long input,warn ("Physical KASLR disabled: no suitable memory region!" ); }else {/* Update the new physical address location. */ -if (*output != random_addr) { -add_identity_map (random_addr, output_size); +if (*output != random_addr) *output = random_addr; - } - -/* -* This loads the identity mapping page table. -* This should only be done if a new physical address -* is found for the kernel, otherwise we should keep -* the old page table to make it be like the "nokaslr" -* case. -*/ -finalize_identity_maps (); }
arch/x86/boot/compressed/misc.h +0 -10
@@ -98,17 +98,7 @@ static inline void choose_random_location(unsigned long input,#endif #ifdef CONFIG_X86_64 -void initialize_identity_maps (void ); -void add_identity_map (unsigned long start,unsigned long size); -void finalize_identity_maps (void );extern unsigned char _pgtable[]; -#else -static inline void initialize_identity_maps (void ) -{ } -static inline void add_identity_map (unsigned long start,unsigned long size) -{ } -static inline void finalize_identity_maps (void ) -{ }#endif #ifdef CONFIG_EARLY_PRINTK
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Don't pre-map memory in KASLR code
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 8570978ea030757839747aa9944ea576708be3d4 Gitweb: https://git.kernel.org/tip/8570978ea030757839747aa9944ea576708be3d4 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:18 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Don't pre-map memory in KASLR code
With the page-fault handler in place, he identity mapping can be built on-demand. So remove the code which manually creates the mappings and unexport/remove the functions used for it.
--- arch/x86/boot/compressed/ident_map_64.c | 6 ++---- arch/x86/boot/compressed/kaslr.c | 24 +----------------------- arch/x86/boot/compressed/misc.h | 10 +---------- 3 files changed, 3 insertions(+), 37 deletions(-)
arch/x86/boot/compressed/ident_map_64.c +2 -4
@@ -87,11 +87,9 @@ phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1;static struct x86_mapping_info mapping_info;/* -* Adds the specified range to what will become the new identity mappings. -* Once all ranges have been added, the new mapping is activated by calling -* finalize_identity_maps() below. +* Adds the specified range to the identity mappings. */ -void add_identity_map (unsigned long start,unsigned long size) +static void add_identity_map (unsigned long start,unsigned long size) {unsigned long end = start + size;
arch/x86/boot/compressed/kaslr.c +1 -23
@@ -397,8 +397,6 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,*/ mem_avoid[MEM_AVOID_ZO_RANGE ].start = input; mem_avoid[MEM_AVOID_ZO_RANGE ].size = (output + init_size) - input; -add_identity_map (mem_avoid[MEM_AVOID_ZO_RANGE ].start, - mem_avoid[MEM_AVOID_ZO_RANGE ].size);/* Avoid initrd. */ initrd_start = (u64 )boot_params->ext_ramdisk_image <<32 ;
@@ -416,15 +414,11 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size, cmd_line_size =strnlen ((char *)cmd_line,COMMAND_LINE_SIZE -1 ) +1 ; mem_avoid[MEM_AVOID_CMDLINE ].start = cmd_line; mem_avoid[MEM_AVOID_CMDLINE ].size = cmd_line_size; -add_identity_map (mem_avoid[MEM_AVOID_CMDLINE ].start, - mem_avoid[MEM_AVOID_CMDLINE ].size); }/* Avoid boot parameters. */ mem_avoid[MEM_AVOID_BOOTPARAMS ].start = (unsigned long )boot_params; mem_avoid[MEM_AVOID_BOOTPARAMS ].size =sizeof (*boot_params); -add_identity_map (mem_avoid[MEM_AVOID_BOOTPARAMS ].start, - mem_avoid[MEM_AVOID_BOOTPARAMS ].size);/* We don't need to set a mapping for setup_data. */
@@ -433,11 +427,6 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,/* Enumerate the immovable memory regions */ num_immovable_mem =count_immovable_mem_regions (); - -#ifdef CONFIG_X86_VERBOSE_BOOTUP -/* Make sure video RAM can be used. */ -add_identity_map (0 ,PMD_SIZE ); -#endif }/*
@@ -884,19 +873,8 @@ void choose_random_location(unsigned long input,warn ("Physical KASLR disabled: no suitable memory region!" ); }else {/* Update the new physical address location. */ -if (*output != random_addr) { -add_identity_map (random_addr, output_size); +if (*output != random_addr) *output = random_addr; - } - -/* -* This loads the identity mapping page table. -* This should only be done if a new physical address -* is found for the kernel, otherwise we should keep -* the old page table to make it be like the "nokaslr" -* case. -*/ -finalize_identity_maps (); }
arch/x86/boot/compressed/misc.h +0 -10
@@ -98,17 +98,7 @@ static inline void choose_random_location(unsigned long input,#endif #ifdef CONFIG_X86_64 -void initialize_identity_maps (void ); -void add_identity_map (unsigned long start,unsigned long size); -void finalize_identity_maps (void );extern unsigned char _pgtable[]; -#else -static inline void initialize_identity_maps (void ) -{ } -static inline void add_identity_map (unsigned long start,unsigned long size) -{ } -static inline void finalize_identity_maps (void ) -{ }#endif #ifdef CONFIG_EARLY_PRINTK
PATCH v7 18/72 x86/boot/compressed/64: Change add_identity_map() to take start and end
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Changing the function to take start and end as parameters instead of start and size simplifies the callers, which don't need to calculate the size if they already have start and end.
--- arch/x86/boot/compressed/ident_map_64.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-)
arch/x86/boot/compressed/ident_map_64.c +5 -10
@@ -89,10 +89,8 @@ static struct x86_mapping_info mapping_info;/* * Adds the specified range to the identity mappings. */ -static void add_identity_map (unsigned long start,unsigned long size) +static void add_identity_map (unsigned long start,unsigned long end) { -unsigned long end = start + size; -/* Align boundary to 2M. */ start =round_down (start,PMD_SIZE ); end =round_up (end,PMD_SIZE );
@@ -107,8 +105,6 @@ static void add_identity_map(unsigned long start, unsigned long size)/* Locates and clears a region for a new top level page table. */ void initialize_identity_maps (void ) { -unsigned long start, size; -/* If running as an SEV guest, the encryption mask is required. */ set_sev_encryption_mask ();
@@ -155,9 +151,7 @@ void initialize_identity_maps(void)* New page-table is set up - map the kernel image and load it * into cr3. */ - start = (unsigned long )_head; - size = _end - _head; -add_identity_map (start, size); +add_identity_map ((unsigned long )_head, (unsigned long )_end);write_cr3 (top_level_pgt); }
@@ -189,7 +183,8 @@ static void do_pf_error(const char *msg, unsigned long error_code,void do_boot_page_fault (struct pt_regs *regs,unsigned long error_code) { -unsigned long address =native_read_cr2 (); +unsigned long address =native_read_cr2 () &PMD_MASK ; +unsigned long end = address +PMD_SIZE ;/* * Check for unexpected error codes. Unexpected are:
@@ -204,5 +199,5 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)* Error code is sane - now identity map the 2M region around * the faulting address. */ -add_identity_map (address &PMD_MASK ,PMD_SIZE ); +add_identity_map (address, end); }
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Change add_identity_map() to take start and end
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 21cf2372618ef167d8c4ae04880fb873b55b2daa Gitweb: https://git.kernel.org/tip/21cf2372618ef167d8c4ae04880fb873b55b2daa Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:19 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Change add_identity_map() to take start and end
Changing the function to take start and end as parameters instead of start and size simplifies the callers which don't need to calculate the size if they already have start and end.
--- arch/x86/boot/compressed/ident_map_64.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-)
arch/x86/boot/compressed/ident_map_64.c +5 -10
@@ -89,10 +89,8 @@ static struct x86_mapping_info mapping_info;/* * Adds the specified range to the identity mappings. */ -static void add_identity_map (unsigned long start,unsigned long size) +static void add_identity_map (unsigned long start,unsigned long end) { -unsigned long end = start + size; -/* Align boundary to 2M. */ start =round_down (start,PMD_SIZE ); end =round_up (end,PMD_SIZE );
@@ -107,8 +105,6 @@ static void add_identity_map(unsigned long start, unsigned long size)/* Locates and clears a region for a new top level page table. */ void initialize_identity_maps (void ) { -unsigned long start, size; -/* If running as an SEV guest, the encryption mask is required. */ set_sev_encryption_mask ();
@@ -155,9 +151,7 @@ void initialize_identity_maps(void)* New page-table is set up - map the kernel image and load it * into cr3. */ - start = (unsigned long )_head; - size = _end - _head; -add_identity_map (start, size); +add_identity_map ((unsigned long )_head, (unsigned long )_end);write_cr3 (top_level_pgt); }
@@ -189,7 +183,8 @@ static void do_pf_error(const char *msg, unsigned long error_code,void do_boot_page_fault (struct pt_regs *regs,unsigned long error_code) { -unsigned long address =native_read_cr2 (); +unsigned long address =native_read_cr2 () &PMD_MASK ; +unsigned long end = address +PMD_SIZE ;/* * Check for unexpected error codes. Unexpected are:
@@ -204,5 +199,5 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)* Error code is sane - now identity map the 2M region around * the faulting address. */ -add_identity_map (address &PMD_MASK ,PMD_SIZE ); +add_identity_map (address, end); }
PATCH v7 19/72 x86/boot/compressed/64: Add stage1 #VC handler
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Add the first handler for #VC exceptions. At stage 1 there is no GHCB yet becaue the kernel might still be running on the EFI page table.
The stage 1 handler is limited to the MSR based protocol to talk to the hypervisor and can only support CPUID exit-codes, but that is enough to get to stage 2.
--- arch/x86/boot/compressed/Makefile | 1 + arch/x86/boot/compressed/idt_64.c | 4 ++ arch/x86/boot/compressed/idt_handlers_64.S | 4 ++ arch/x86/boot/compressed/misc.h | 1 + arch/x86/boot/compressed/sev-es.c | 45 +++++++++++++++ arch/x86/include/asm/msr-index.h | 1 + arch/x86/include/asm/sev-es.h | 37 ++++++++++++ arch/x86/include/asm/trapnr.h | 1 + arch/x86/kernel/sev-es-shared.c | 66 ++++++++++++++++++++++ 9 files changed, 160 insertions(+) create mode 100644 arch/x86/boot/compressed/sev-es.c create mode 100644 arch/x86/include/asm/sev-es.h create mode 100644 arch/x86/kernel/sev-es-shared.c
arch/x86/boot/compressed/Makefile +1 -0
@@ -83,6 +83,7 @@ ifdef CONFIG_X86_64 vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o vmlinux-objs-y += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o + vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev-es.o endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o
arch/x86/boot/compressed/idt_64.c +4 -0
@@ -32,6 +32,10 @@ void load_stage1_idt(void) { boot_idt_desc.address = (unsigned long )boot_idt; + +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) +set_idt_entry (X86_TRAP_VC , boot_stage1_vc); +load_boot_idt (&boot_idt_desc); }
arch/x86/boot/compressed/idt_handlers_64.S +4 -0
@@ -70,3 +70,7 @@ SYM_FUNC_END(\name) .code64 EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1 + +#ifdef CONFIG_AMD_MEM_ENCRYPT +EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1 +#endif
arch/x86/boot/compressed/misc.h +1 -0
@@ -141,5 +141,6 @@ extern struct desc_ptr boot_idt_desc;/* IDT Entry Points */ void boot_page_fault (void ); +void boot_stage1_vc (void );#endif /* BOOT_COMPRESSED_MISC_H */
arch/x86/boot/compressed/sev-es.c new file +45 -0
45 lines not on this page · raw message
arch/x86/include/asm/msr-index.h +1 -0
@@ -466,6 +466,7 @@#define MSR_AMD64_IBSBRTARGET 0xc001103b#define MSR_AMD64_IBSOPDATA4 0xc001103d#define MSR_AMD64_IBS_REG_COUNT_MAX 8/* includes MSR_AMD64_IBSBRTARGET */ +#define MSR_AMD64_SEV_ES_GHCB 0xc0010130#define MSR_AMD64_SEV 0xc0010131#define MSR_AMD64_SEV_ENABLED_BIT 0#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
arch/x86/include/asm/sev-es.h new file +37 -0
@@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* AMD Encrypted Register State Support +* +* Author: Joerg Roedel <[email protected]> +*/ + +#ifndef __ASM_ENCRYPTED_STATE_H +#define __ASM_ENCRYPTED_STATE_H + +#include <linux/types.h> + +#define GHCB_SEV_CPUID_REQ 0x004UL +#define GHCB_CPUID_REQ_EAX 0 +#define GHCB_CPUID_REQ_EBX 1 +#define GHCB_CPUID_REQ_ECX 2 +#define GHCB_CPUID_REQ_EDX 3 +#define GHCB_CPUID_REQ (fn, reg) (GHCB_SEV_CPUID_REQ | \ + (((unsigned long)reg & 3) << 30) | \ + (((unsigned long)fn) << 32)) + +#define GHCB_SEV_CPUID_RESP 0x005UL +#define GHCB_SEV_TERMINATE 0x100UL + +#define GHCB_SEV_GHCB_RESP_CODE (v) ((v) & 0xfff) +#define VMGEXIT () { asm volatile("rep; vmmcall\n\r"); } + +void do_vc_no_ghcb (struct pt_regs *regs,unsigned long exit_code); + +static inline u64 lower_bits (u64 val,unsigned int bits) +{ +u64 mask = (1ULL << bits) -1 ; + +return (val & mask); +} + +#endif
arch/x86/include/asm/trapnr.h +1 -0
@@ -26,6 +26,7 @@#define X86_TRAP_XF 19/* SIMD Floating-Point Exception */ #define X86_TRAP_VE 20/* Virtualization Exception */ #define X86_TRAP_CP 21/* Control Protection Exception */ +#define X86_TRAP_VC 29/* VMM Communication Exception */ #define X86_TRAP_IRET 32/* IRET Exception */ #endif
arch/x86/kernel/sev-es-shared.c new file +66 -0
66 lines not on this page · raw message
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Borislav Petkov
maintainerRe: [PATCH v7 19/72] x86/boot/compressed/64: Add stage1 #VC handler
Quoted 8 lines · Joerg Roedel
arch/x86/boot/compressed/sev-es.c new file +8
-0+static inline u64 sev_es_rd_ghcb_msr (void ) +{ +unsigned long low, high; + + asmvolatile ("rdmsr\n" :"=a" (low),"=d" (high) : +"c" (MSR_AMD64_SEV_ES_GHCB )); + +return ((high <<32 ) | low);
arch/x86/boot/compressed/sev-es.c line 37
Quoted 10 lines
arch/x86/boot/compressed/sev-es.c new file +10
-0+} + +static inline void sev_es_wr_ghcb_msr (u64 val) +{ +u32 low, high; + + low = val &0xffffffffUL ; + high = val >>32 ; + + asm volatile("wrmsr\n" : : "c" (MSR_AMD64_SEV_ES_GHCB),
^^
No need for that newline and the one above. I've zapped it while applying.
Signature · 8 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization[tip: x86/seves] x86/boot/compressed/64: Add stage1 #VC handler
In reply to Joerg RoedelThe following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 29dcc60f6a19fb0aaee97bd1ae2ed8a7dc6f0cfe Gitweb: https://git.kernel.org/tip/29dcc60f6a19fb0aaee97bd1ae2ed8a7dc6f0cfe Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:20 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Add stage1 #VC handler
Add the first handler for #VC exceptions. At stage 1 there is no GHCB yet because the kernel might still be running on the EFI page table.
The stage 1 handler is limited to the MSR-based protocol to talk to the hypervisor and can only support CPUID exit-codes, but that is enough to get to stage 2.
[ bp: Zap superfluous newlines after rd/wrmsr instruction mnemonics. ]
--- arch/x86/boot/compressed/Makefile | 1 +- arch/x86/boot/compressed/idt_64.c | 4 +- arch/x86/boot/compressed/idt_handlers_64.S | 4 +- arch/x86/boot/compressed/misc.h | 1 +- arch/x86/boot/compressed/sev-es.c | 45 ++++++++++++++- arch/x86/include/asm/msr-index.h | 1 +- arch/x86/include/asm/sev-es.h | 37 ++++++++++++- arch/x86/include/asm/trapnr.h | 1 +- arch/x86/kernel/sev-es-shared.c | 66 +++++++++++++++++++++- 9 files changed, 160 insertions(+) create mode 100644 arch/x86/boot/compressed/sev-es.c create mode 100644 arch/x86/include/asm/sev-es.h create mode 100644 arch/x86/kernel/sev-es-shared.c
arch/x86/boot/compressed/Makefile +1 -0
@@ -88,6 +88,7 @@ ifdef CONFIG_X86_64 vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o vmlinux-objs-y += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o + vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev-es.o endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o
arch/x86/boot/compressed/idt_64.c +4 -0
@@ -32,6 +32,10 @@ void load_stage1_idt(void) { boot_idt_desc.address = (unsigned long )boot_idt; + +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) +set_idt_entry (X86_TRAP_VC , boot_stage1_vc); +load_boot_idt (&boot_idt_desc); }
arch/x86/boot/compressed/idt_handlers_64.S +4 -0
@@ -70,3 +70,7 @@ SYM_FUNC_END(\name) .code64 EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1 + +#ifdef CONFIG_AMD_MEM_ENCRYPT +EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1 +#endif
arch/x86/boot/compressed/misc.h +1 -0
@@ -141,5 +141,6 @@ extern struct desc_ptr boot_idt_desc;/* IDT Entry Points */ void boot_page_fault (void ); +void boot_stage1_vc (void );#endif /* BOOT_COMPRESSED_MISC_H */
arch/x86/boot/compressed/sev-es.c new file +45 -0
45 lines not on this page · raw message
arch/x86/include/asm/msr-index.h +1 -0
@@ -466,6 +466,7 @@#define MSR_AMD64_IBSBRTARGET 0xc001103b#define MSR_AMD64_IBSOPDATA4 0xc001103d#define MSR_AMD64_IBS_REG_COUNT_MAX 8/* includes MSR_AMD64_IBSBRTARGET */ +#define MSR_AMD64_SEV_ES_GHCB 0xc0010130#define MSR_AMD64_SEV 0xc0010131#define MSR_AMD64_SEV_ENABLED_BIT 0#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
arch/x86/include/asm/sev-es.h new file +37 -0
@@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* AMD Encrypted Register State Support +* +* Author: Joerg Roedel <[email protected]> +*/ + +#ifndef __ASM_ENCRYPTED_STATE_H +#define __ASM_ENCRYPTED_STATE_H + +#include <linux/types.h> + +#define GHCB_SEV_CPUID_REQ 0x004UL +#define GHCB_CPUID_REQ_EAX 0 +#define GHCB_CPUID_REQ_EBX 1 +#define GHCB_CPUID_REQ_ECX 2 +#define GHCB_CPUID_REQ_EDX 3 +#define GHCB_CPUID_REQ (fn, reg) (GHCB_SEV_CPUID_REQ | \ + (((unsigned long)reg & 3) << 30) | \ + (((unsigned long)fn) << 32)) + +#define GHCB_SEV_CPUID_RESP 0x005UL +#define GHCB_SEV_TERMINATE 0x100UL + +#define GHCB_SEV_GHCB_RESP_CODE (v) ((v) & 0xfff) +#define VMGEXIT () { asm volatile("rep; vmmcall\n\r"); } + +void do_vc_no_ghcb (struct pt_regs *regs,unsigned long exit_code); + +static inline u64 lower_bits (u64 val,unsigned int bits) +{ +u64 mask = (1ULL << bits) -1 ; + +return (val & mask); +} + +#endif
arch/x86/include/asm/trapnr.h +1 -0
@@ -26,6 +26,7 @@#define X86_TRAP_XF 19/* SIMD Floating-Point Exception */ #define X86_TRAP_VE 20/* Virtualization Exception */ #define X86_TRAP_CP 21/* Control Protection Exception */ +#define X86_TRAP_VC 29/* VMM Communication Exception */ #define X86_TRAP_IRET 32/* IRET Exception */ #endif
arch/x86/kernel/sev-es-shared.c new file +66 -0
66 lines not on this page · raw message
PATCH v7 20/72 x86/boot/compressed/64: Call set_sev_encryption_mask() earlier
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Call set_sev_encryption_mask() while still on the stage 1 #VC-handler, because the stage 2 handler needs the kernel's own page-tables to be set up, to which calling set_sev_encryption_mask() is a prerequisite.
--- arch/x86/boot/compressed/head_64.S | 9 ++++++++- arch/x86/boot/compressed/ident_map_64.c | 3 --- 2 files changed, 8 insertions(+), 4 deletions(-)
arch/x86/boot/compressed/head_64.S +8 -1
@@ -533,9 +533,16 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) rep stosq /* - * Load stage2 IDT and switch to our own page-table + * If running as an SEV guest, the encryption mask is required in the + * page-table setup code below. When the guest also has SEV-ES enabled + * set_sev_encryption_mask() will cause #VC exceptions, but the stage2 + * handler can't map its GHCB because the page-table is not set up yet. + * So set up the encryption mask here while still on the stage1 #VC + * handler. Then load stage2 IDT and switch to the kernel's own + * page-table. */ pushq %rsi + call set_sev_encryption_mask call load_stage2_idt call initialize_identity_maps popq %rsi
arch/x86/boot/compressed/ident_map_64.c +0 -3
@@ -105,9 +105,6 @@ static void add_identity_map(unsigned long start, unsigned long end)/* Locates and clears a region for a new top level page table. */ void initialize_identity_maps (void ) { -/* If running as an SEV guest, the encryption mask is required. */ -set_sev_encryption_mask (); -/* Exclude the encryption mask from __PHYSICAL_MASK */ physical_mask &= ~sme_me_mask;
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Call set_sev_encryption_mask() earlier
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: c2a0304a286f386e45cea3f4b0617f0813de67fd Gitweb: https://git.kernel.org/tip/c2a0304a286f386e45cea3f4b0617f0813de67fd Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:21 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Call set_sev_encryption_mask() earlier
Call set_sev_encryption_mask() while still on the stage 1 #VC-handler because the stage 2 handler needs the kernel's own page tables to be set up, to which calling set_sev_encryption_mask() is a prerequisite.
--- arch/x86/boot/compressed/head_64.S | 9 ++++++++- arch/x86/boot/compressed/ident_map_64.c | 3 --- 2 files changed, 8 insertions(+), 4 deletions(-)
arch/x86/boot/compressed/head_64.S +8 -1
@@ -543,9 +543,16 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) rep stosq /* - * Load stage2 IDT and switch to our own page-table + * If running as an SEV guest, the encryption mask is required in the + * page-table setup code below. When the guest also has SEV-ES enabled + * set_sev_encryption_mask() will cause #VC exceptions, but the stage2 + * handler can't map its GHCB because the page-table is not set up yet. + * So set up the encryption mask here while still on the stage1 #VC + * handler. Then load stage2 IDT and switch to the kernel's own + * page-table. */ pushq %rsi + call set_sev_encryption_mask call load_stage2_idt call initialize_identity_maps popq %rsi
arch/x86/boot/compressed/ident_map_64.c +0 -3
@@ -105,9 +105,6 @@ static void add_identity_map(unsigned long start, unsigned long end)/* Locates and clears a region for a new top level page table. */ void initialize_identity_maps (void ) { -/* If running as an SEV guest, the encryption mask is required. */ -set_sev_encryption_mask (); -/* Exclude the encryption mask from __PHYSICAL_MASK */ physical_mask &= ~sme_me_mask;
PATCH v7 21/72 x86/boot/compressed/64: Check return value of kernel_ident_mapping_init()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The function can fail to create an identity mapping, check for that and bail out if it happens.
--- arch/x86/boot/compressed/ident_map_64.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
arch/x86/boot/compressed/ident_map_64.c +5 -2
@@ -91,6 +91,8 @@ static struct x86_mapping_info mapping_info;*/ static void add_identity_map (unsigned long start,unsigned long end) { +int ret; +/* Align boundary to 2M. */ start =round_down (start,PMD_SIZE ); end =round_up (end,PMD_SIZE );
@@ -98,8 +100,9 @@ static void add_identity_map(unsigned long start, unsigned long end)return ;/* Build the mapping. */ -kernel_ident_mapping_init (&mapping_info, (pgd_t *)top_level_pgt, - start, end); + ret =kernel_ident_mapping_init (&mapping_info, (pgd_t *)top_level_pgt, start, end); +if (ret) +error ("Error: kernel_ident_mapping_init() failed\n" ); }/* Locates and clears a region for a new top level page table. */
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Check return value of kernel_ident_mapping_init()
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 4b3fdca64a7e8ad90c87cad1fbc6991471f48dc7 Gitweb: https://git.kernel.org/tip/4b3fdca64a7e8ad90c87cad1fbc6991471f48dc7 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:22 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Check return value of kernel_ident_mapping_init()
The function can fail to create an identity mapping, check for that and bail out if it happens.
--- arch/x86/boot/compressed/ident_map_64.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
arch/x86/boot/compressed/ident_map_64.c +5 -2
@@ -91,6 +91,8 @@ static struct x86_mapping_info mapping_info;*/ static void add_identity_map (unsigned long start,unsigned long end) { +int ret; +/* Align boundary to 2M. */ start =round_down (start,PMD_SIZE ); end =round_up (end,PMD_SIZE );
@@ -98,8 +100,9 @@ static void add_identity_map(unsigned long start, unsigned long end)return ;/* Build the mapping. */ -kernel_ident_mapping_init (&mapping_info, (pgd_t *)top_level_pgt, - start, end); + ret =kernel_ident_mapping_init (&mapping_info, (pgd_t *)top_level_pgt, start, end); +if (ret) +error ("Error: kernel_ident_mapping_init() failed\n" ); }/* Locates and clears a region for a new top level page table. */
PATCH v7 22/72 x86/boot/compressed/64: Add set_page_en/decrypted() helpers
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The functions are needed to map the GHCB for SEV-ES guests. The GHCB is used for communication with the hypervisor, so its content must not be encrypted. After the GHCB is not needed anymore it must be mapped encrypted again so that the running kernel image can safely re-use the memory.
--- arch/x86/boot/compressed/ident_map_64.c | 133 ++++++++++++++++++++++++ arch/x86/boot/compressed/misc.h | 2 + 2 files changed, 135 insertions(+)
arch/x86/boot/compressed/ident_map_64.c +133 -0
@@ -24,6 +24,7 @@/* These actually do the work of building the kernel identity maps. */ #include <linux/pgtable.h> +#include <asm/cmpxchg.h> #include <asm/trap_pf.h> #include <asm/trapnr.h> #include <asm/init.h>
@@ -165,6 +166,138 @@ void finalize_identity_maps(void)write_cr3 (top_level_pgt); } +static pte_t *split_large_pmd (struct x86_mapping_info *info, +pmd_t *pmdp,unsigned long __address) +{ +unsigned long page_flags; +unsigned long address; +pte_t *pte; +pmd_t pmd; +int i; + + pte = (pte_t *)info->alloc_pgt_page (info->context); +if (!pte) +return NULL ; + + address = __address &PMD_MASK ; +/* No large page - clear PSE flag */ + page_flags = info->page_flag & ~_PAGE_PSE; + +/* Populate the PTEs */ +for (i =0 ; i <PTRS_PER_PMD ; i++) { +set_pte (&pte[i],__pte (address | page_flags)); + address +=PAGE_SIZE ; + } + +/* +* Ideally we need to clear the large PMD first and do a TLB +* flush before we write the new PMD. But the 2M range of the +* PMD might contain the code we execute and/or the stack +* we are on, so we can't do that. But that should be safe here +* because we are going from large to small mappings and we are +* also the only user of the page-table, so there is no chance +* of a TLB multihit. +*/ + pmd =__pmd ((unsigned long )pte | info->kernpg_flag); +set_pmd (pmdp, pmd); +/* Flush TLB to establish the new PMD */ +write_cr3 (top_level_pgt); + +return pte +pte_index (__address); +} + +static void clflush_page (unsigned long address) +{ +unsigned int flush_size; +char *cl, *start, *end; + +/* +* Hardcode cl-size to 64 - CPUID can't be used here because that might +* cause another #VC exception and the GHCB is not ready to use yet. +*/ + flush_size =64 ; + start = (char *)(address &PAGE_MASK ); + end = start +PAGE_SIZE ; + +/* +* First make sure there are no pending writes on the cache-lines to +* flush. +*/ + asmvolatile ("mfence" : : :"memory" ); + +for (cl = start; cl != end; cl += flush_size) +clflush (cl); +} + +static int set_clr_page_flags (struct x86_mapping_info *info, +unsigned long address, +pteval_t set,pteval_t clr) +{ +pgd_t *pgdp = (pgd_t *)top_level_pgt; +p4d_t *p4dp; +pud_t *pudp; +pmd_t *pmdp; +pte_t *ptep, pte; + +/* +* First make sure there is a PMD mapping for 'address'. +* It should already exist, but keep things generic. +* +* To map the page just read from it and fault it in if there is no +* mapping yet. add_identity_map() can't be called here because that +* would unconditionally map the address on PMD level, destroying any +* PTE-level mappings that might already exist. Use assembly here so +* the access won't be optimized away. +*/ + asmvolatile ("mov %[address], %%r9" + :: [address]"g" (*(unsigned long *)address) + :"r9" ,"memory" ); + +/* +* The page is mapped at least with PMD size - so skip checks and walk +* directly to the PMD. +*/ + p4dp =p4d_offset (pgdp, address); + pudp =pud_offset (p4dp, address); + pmdp =pmd_offset (pudp, address); + +if (pmd_large (*pmdp)) + ptep =split_large_pmd (info, pmdp, address); +else + ptep =pte_offset_kernel (pmdp, address); + +if (!ptep) +return -ENOMEM ; + +/* +* Changing encryption attributes of a page requires to flush it from +* the caches. +*/ +if ((set | clr) & _PAGE_ENC) +clflush_page (address); + +/* Update PTE */ + pte = *ptep; + pte =pte_set_flags (pte, set); + pte =pte_clear_flags (pte, clr); +set_pte (ptep, pte); + +/* Flush TLB after changing encryption attribute */ +write_cr3 (top_level_pgt); + +return 0 ; +} + +int set_page_decrypted (unsigned long address) +{ +return set_clr_page_flags (&mapping_info, address,0 , _PAGE_ENC); +} + +int set_page_encrypted (unsigned long address) +{ +return set_clr_page_flags (&mapping_info, address, _PAGE_ENC,0 ); +} +static void do_pf_error (const char *msg,unsigned long error_code,unsigned long address,unsigned long ip) {
arch/x86/boot/compressed/misc.h +2 -0
@@ -98,6 +98,8 @@ static inline void choose_random_location(unsigned long input,#endif #ifdef CONFIG_X86_64 +extern int set_page_decrypted (unsigned long address); +extern int set_page_encrypted (unsigned long address);extern unsigned char _pgtable[];#endif
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Add set_page_en/decrypted() helpers
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: c81d60029a1393183d2125fcb4b64831629b8864 Gitweb: https://git.kernel.org/tip/c81d60029a1393183d2125fcb4b64831629b8864 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:23 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Add set_page_en/decrypted() helpers
The functions are needed to map the GHCB for SEV-ES guests. The GHCB is used for communication with the hypervisor, so its content must not be encrypted. After the GHCB is not needed anymore it must be mapped encrypted again so that the running kernel image can safely re-use the memory.
--- arch/x86/boot/compressed/ident_map_64.c | 133 +++++++++++++++++++++++- arch/x86/boot/compressed/misc.h | 2 +- 2 files changed, 135 insertions(+)
arch/x86/boot/compressed/ident_map_64.c +133 -0
@@ -24,6 +24,7 @@/* These actually do the work of building the kernel identity maps. */ #include <linux/pgtable.h> +#include <asm/cmpxchg.h> #include <asm/trap_pf.h> #include <asm/trapnr.h> #include <asm/init.h>
@@ -165,6 +166,138 @@ void finalize_identity_maps(void)write_cr3 (top_level_pgt); } +static pte_t *split_large_pmd (struct x86_mapping_info *info, +pmd_t *pmdp,unsigned long __address) +{ +unsigned long page_flags; +unsigned long address; +pte_t *pte; +pmd_t pmd; +int i; + + pte = (pte_t *)info->alloc_pgt_page (info->context); +if (!pte) +return NULL ; + + address = __address &PMD_MASK ; +/* No large page - clear PSE flag */ + page_flags = info->page_flag & ~_PAGE_PSE; + +/* Populate the PTEs */ +for (i =0 ; i <PTRS_PER_PMD ; i++) { +set_pte (&pte[i],__pte (address | page_flags)); + address +=PAGE_SIZE ; + } + +/* +* Ideally we need to clear the large PMD first and do a TLB +* flush before we write the new PMD. But the 2M range of the +* PMD might contain the code we execute and/or the stack +* we are on, so we can't do that. But that should be safe here +* because we are going from large to small mappings and we are +* also the only user of the page-table, so there is no chance +* of a TLB multihit. +*/ + pmd =__pmd ((unsigned long )pte | info->kernpg_flag); +set_pmd (pmdp, pmd); +/* Flush TLB to establish the new PMD */ +write_cr3 (top_level_pgt); + +return pte +pte_index (__address); +} + +static void clflush_page (unsigned long address) +{ +unsigned int flush_size; +char *cl, *start, *end; + +/* +* Hardcode cl-size to 64 - CPUID can't be used here because that might +* cause another #VC exception and the GHCB is not ready to use yet. +*/ + flush_size =64 ; + start = (char *)(address &PAGE_MASK ); + end = start +PAGE_SIZE ; + +/* +* First make sure there are no pending writes on the cache-lines to +* flush. +*/ + asmvolatile ("mfence" : : :"memory" ); + +for (cl = start; cl != end; cl += flush_size) +clflush (cl); +} + +static int set_clr_page_flags (struct x86_mapping_info *info, +unsigned long address, +pteval_t set,pteval_t clr) +{ +pgd_t *pgdp = (pgd_t *)top_level_pgt; +p4d_t *p4dp; +pud_t *pudp; +pmd_t *pmdp; +pte_t *ptep, pte; + +/* +* First make sure there is a PMD mapping for 'address'. +* It should already exist, but keep things generic. +* +* To map the page just read from it and fault it in if there is no +* mapping yet. add_identity_map() can't be called here because that +* would unconditionally map the address on PMD level, destroying any +* PTE-level mappings that might already exist. Use assembly here so +* the access won't be optimized away. +*/ + asmvolatile ("mov %[address], %%r9" + :: [address]"g" (*(unsigned long *)address) + :"r9" ,"memory" ); + +/* +* The page is mapped at least with PMD size - so skip checks and walk +* directly to the PMD. +*/ + p4dp =p4d_offset (pgdp, address); + pudp =pud_offset (p4dp, address); + pmdp =pmd_offset (pudp, address); + +if (pmd_large (*pmdp)) + ptep =split_large_pmd (info, pmdp, address); +else + ptep =pte_offset_kernel (pmdp, address); + +if (!ptep) +return -ENOMEM ; + +/* +* Changing encryption attributes of a page requires to flush it from +* the caches. +*/ +if ((set | clr) & _PAGE_ENC) +clflush_page (address); + +/* Update PTE */ + pte = *ptep; + pte =pte_set_flags (pte, set); + pte =pte_clear_flags (pte, clr); +set_pte (ptep, pte); + +/* Flush TLB after changing encryption attribute */ +write_cr3 (top_level_pgt); + +return 0 ; +} + +int set_page_decrypted (unsigned long address) +{ +return set_clr_page_flags (&mapping_info, address,0 , _PAGE_ENC); +} + +int set_page_encrypted (unsigned long address) +{ +return set_clr_page_flags (&mapping_info, address, _PAGE_ENC,0 ); +} +static void do_pf_error (const char *msg,unsigned long error_code,unsigned long address,unsigned long ip) {
arch/x86/boot/compressed/misc.h +2 -0
@@ -98,6 +98,8 @@ static inline void choose_random_location(unsigned long input,#endif #ifdef CONFIG_X86_64 +extern int set_page_decrypted (unsigned long address); +extern int set_page_encrypted (unsigned long address);extern unsigned char _pgtable[];#endif
PATCH v7 23/72 x86/boot/compressed/64: Setup GHCB Based VC Exception handler
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Install an exception handler for #VC exception that uses a GHCB. Also add the infrastructure for handling different exit-codes by decoding the instruction that caused the exception and error handling.
--- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/Makefile | 5 + arch/x86/boot/compressed/idt_64.c | 4 + arch/x86/boot/compressed/idt_handlers_64.S | 3 +- arch/x86/boot/compressed/misc.c | 7 + arch/x86/boot/compressed/misc.h | 7 + arch/x86/boot/compressed/sev-es.c | 111 +++++++++++++++ arch/x86/include/asm/sev-es.h | 39 ++++++ arch/x86/include/uapi/asm/svm.h | 1 + arch/x86/kernel/sev-es-shared.c | 154 +++++++++++++++++++++ 10 files changed, 331 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
@@ -1523,6 +1523,7 @@ config AMD_MEM_ENCRYPT select DYNAMIC_PHYSICAL_MASK select ARCH_USE_MEMREMAP_PROT select ARCH_HAS_FORCE_DMA_UNENCRYPTED + select INSTRUCTION_DECODER help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory
arch/x86/boot/compressed/Makefile +5 -0
@@ -45,6 +45,11 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables KBUILD_CFLAGS += -D__DISABLE_EXPORTS KBUILD_CFLAGS += -include $(srctree)/include/linux/hidden.h +# sev-es.c indirectly inludes inat-table.h which is generated during +# compilation and stored in $(objtree). Add the directory to the includes so +# that the compiler finds it even with out-of-tree builds (make O=/some/path). +CFLAGS_sev-es.o += -I$(objtree)/arch/x86/lib/ + KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__ GCOV_PROFILE := n UBSAN_SANITIZE :=n
arch/x86/boot/compressed/idt_64.c +4 -0
@@ -46,5 +46,9 @@ void load_stage2_idt(void)set_idt_entry (X86_TRAP_PF , boot_page_fault); +#ifdef CONFIG_AMD_MEM_ENCRYPT +set_idt_entry (X86_TRAP_VC , boot_stage2_vc); +#endif +load_boot_idt (&boot_idt_desc); }
arch/x86/boot/compressed/idt_handlers_64.S +2 -1
@@ -72,5 +72,6 @@ SYM_FUNC_END(\name) EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1 #ifdef CONFIG_AMD_MEM_ENCRYPT -EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1 +EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1 +EXCEPTION_HANDLER boot_stage2_vc do_boot_stage2_vc error_code=1 #endif
arch/x86/boot/compressed/misc.c +7 -0
@@ -442,6 +442,13 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap,parse_elf (output);handle_relocations (output, output_len, virt_addr);debug_putstr ("done.\nBooting the kernel.\n" ); + +/* +* Flush GHCB from cache and map it encrypted again when running as +* SEV-ES guest. +*/ +sev_es_shutdown_ghcb (); +return output; }
arch/x86/boot/compressed/misc.h +7 -0
@@ -115,6 +115,12 @@ static inline void console_init(void)void set_sev_encryption_mask (void ); +#ifdef CONFIG_AMD_MEM_ENCRYPT +void sev_es_shutdown_ghcb (void ); +#else +static inline void sev_es_shutdown_ghcb (void ) { } +#endif +/* acpi.c */ #ifdef CONFIG_ACPI acpi_physical_address get_rsdp_addr (void );
@@ -144,5 +150,6 @@ extern struct desc_ptr boot_idt_desc;/* IDT Entry Points */ void boot_page_fault (void );void boot_stage1_vc (void ); +void boot_stage2_vc (void );#endif /* BOOT_COMPRESSED_MISC_H */
arch/x86/boot/compressed/sev-es.c +111 -0
129 lines not on this page · raw message
arch/x86/include/asm/sev-es.h +39 -0
@@ -9,7 +9,14 @@#define __ASM_ENCRYPTED_STATE_H#include <linux/types.h> +#include <asm/insn.h> +#define GHCB_SEV_INFO 0x001UL +#define GHCB_SEV_INFO_REQ 0x002UL +#define GHCB_INFO (v) ((v) & 0xfffUL) +#define GHCB_PROTO_MAX (v) (((v) >> 48) & 0xffffUL) +#define GHCB_PROTO_MIN (v) (((v) >> 32) & 0xffffUL) +#define GHCB_PROTO_OUR 0x0001UL#define GHCB_SEV_CPUID_REQ 0x004UL#define GHCB_CPUID_REQ_EAX 0#define GHCB_CPUID_REQ_EBX 1
@@ -19,12 +26,44 @@ (((unsigned long)reg & 3) << 30) | \ (((unsigned long)fn) << 32)) +#define GHCB_PROTOCOL_MAX 0x0001UL +#define GHCB_DEFAULT_USAGE 0x0000UL +#define GHCB_SEV_CPUID_RESP 0x005UL#define GHCB_SEV_TERMINATE 0x100UL +#define GHCB_SEV_TERMINATE_REASON (reason_set, reason_val) \ + (((((u64)reason_set) & 0x7) << 12) | \ + ((((u64)reason_val) & 0xff) << 16)) +#define GHCB_SEV_ES_REASON_GENERAL_REQUEST 0 +#define GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED 1#define GHCB_SEV_GHCB_RESP_CODE (v) ((v) & 0xfff)#define VMGEXIT () { asm volatile("rep; vmmcall\n\r"); } +enum es_result { +ES_OK ,/* All good */ +ES_UNSUPPORTED ,/* Requested operation not supported */ +ES_VMM_ERROR ,/* Unexpected state from the VMM */ +ES_DECODE_FAILED ,/* Instruction decoding failed */ +ES_EXCEPTION ,/* Instruction caused exception */ +ES_RETRY ,/* Retry instruction emulation */ +}; + +struct es_fault_info { +unsigned long vector; +unsigned long error_code; +unsigned long cr2; +}; + +struct pt_regs ; + +/* ES instruction emulation context */ +struct es_em_ctxt { +struct pt_regs *regs; +struct insn insn; +struct es_fault_info fi; +}; +void do_vc_no_ghcb (struct pt_regs *regs,unsigned long exit_code);static inline u64 lower_bits (u64 val,unsigned int bits)
arch/x86/include/uapi/asm/svm.h +1 -0
@@ -29,6 +29,7 @@#define SVM_EXIT_WRITE_DR6 0x036#define SVM_EXIT_WRITE_DR7 0x037#define SVM_EXIT_EXCP_BASE 0x040 +#define SVM_EXIT_LAST_EXCP 0x05f#define SVM_EXIT_INTR 0x060#define SVM_EXIT_NMI 0x061#define SVM_EXIT_SMI 0x062
arch/x86/kernel/sev-es-shared.c +154 -0
163 lines not on this page · raw message
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Setup a GHCB-based VC Exception handler
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 597cfe48212a3f110ab0f918bf59791f453e65b7 Gitweb: https://git.kernel.org/tip/597cfe48212a3f110ab0f918bf59791f453e65b7 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:24 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Setup a GHCB-based VC Exception handler
Install an exception handler for #VC exception that uses a GHCB. Also add the infrastructure for handling different exit-codes by decoding the instruction that caused the exception and error handling.
--- arch/x86/Kconfig | 1 +- arch/x86/boot/compressed/Makefile | 5 +- arch/x86/boot/compressed/idt_64.c | 4 +- arch/x86/boot/compressed/idt_handlers_64.S | 3 +- arch/x86/boot/compressed/misc.c | 7 +- arch/x86/boot/compressed/misc.h | 7 +- arch/x86/boot/compressed/sev-es.c | 111 ++++++++++++++- arch/x86/include/asm/sev-es.h | 39 +++++- arch/x86/include/uapi/asm/svm.h | 1 +- arch/x86/kernel/sev-es-shared.c | 154 ++++++++++++++++++++- 10 files changed, 331 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
@@ -1521,6 +1521,7 @@ config AMD_MEM_ENCRYPT select DYNAMIC_PHYSICAL_MASK select ARCH_USE_MEMREMAP_PROT select ARCH_HAS_FORCE_DMA_UNENCRYPTED + select INSTRUCTION_DECODER help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory
arch/x86/boot/compressed/Makefile +5 -0
@@ -44,6 +44,11 @@ KBUILD_CFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables KBUILD_CFLAGS += -D__DISABLE_EXPORTS +# sev-es.c indirectly inludes inat-table.h which is generated during +# compilation and stored in $(objtree). Add the directory to the includes so +# that the compiler finds it even with out-of-tree builds (make O=/some/path). +CFLAGS_sev-es.o += -I$(objtree)/arch/x86/lib/ + KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__ GCOV_PROFILE := n UBSAN_SANITIZE :=n
arch/x86/boot/compressed/idt_64.c +4 -0
@@ -46,5 +46,9 @@ void load_stage2_idt(void)set_idt_entry (X86_TRAP_PF , boot_page_fault); +#ifdef CONFIG_AMD_MEM_ENCRYPT +set_idt_entry (X86_TRAP_VC , boot_stage2_vc); +#endif +load_boot_idt (&boot_idt_desc); }
arch/x86/boot/compressed/idt_handlers_64.S +2 -1
@@ -72,5 +72,6 @@ SYM_FUNC_END(\name) EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1 #ifdef CONFIG_AMD_MEM_ENCRYPT -EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1 +EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1 +EXCEPTION_HANDLER boot_stage2_vc do_boot_stage2_vc error_code=1 #endif
arch/x86/boot/compressed/misc.c +7 -0
@@ -442,6 +442,13 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap,parse_elf (output);handle_relocations (output, output_len, virt_addr);debug_putstr ("done.\nBooting the kernel.\n" ); + +/* +* Flush GHCB from cache and map it encrypted again when running as +* SEV-ES guest. +*/ +sev_es_shutdown_ghcb (); +return output; }
arch/x86/boot/compressed/misc.h +7 -0
@@ -115,6 +115,12 @@ static inline void console_init(void)void set_sev_encryption_mask (void ); +#ifdef CONFIG_AMD_MEM_ENCRYPT +void sev_es_shutdown_ghcb (void ); +#else +static inline void sev_es_shutdown_ghcb (void ) { } +#endif +/* acpi.c */ #ifdef CONFIG_ACPI acpi_physical_address get_rsdp_addr (void );
@@ -144,5 +150,6 @@ extern struct desc_ptr boot_idt_desc;/* IDT Entry Points */ void boot_page_fault (void );void boot_stage1_vc (void ); +void boot_stage2_vc (void );#endif /* BOOT_COMPRESSED_MISC_H */
arch/x86/boot/compressed/sev-es.c +111 -0
129 lines not on this page · raw message
arch/x86/include/asm/sev-es.h +39 -0
@@ -9,7 +9,14 @@#define __ASM_ENCRYPTED_STATE_H#include <linux/types.h> +#include <asm/insn.h> +#define GHCB_SEV_INFO 0x001UL +#define GHCB_SEV_INFO_REQ 0x002UL +#define GHCB_INFO (v) ((v) & 0xfffUL) +#define GHCB_PROTO_MAX (v) (((v) >> 48) & 0xffffUL) +#define GHCB_PROTO_MIN (v) (((v) >> 32) & 0xffffUL) +#define GHCB_PROTO_OUR 0x0001UL#define GHCB_SEV_CPUID_REQ 0x004UL#define GHCB_CPUID_REQ_EAX 0#define GHCB_CPUID_REQ_EBX 1
@@ -19,12 +26,44 @@ (((unsigned long)reg & 3) << 30) | \ (((unsigned long)fn) << 32)) +#define GHCB_PROTOCOL_MAX 0x0001UL +#define GHCB_DEFAULT_USAGE 0x0000UL +#define GHCB_SEV_CPUID_RESP 0x005UL#define GHCB_SEV_TERMINATE 0x100UL +#define GHCB_SEV_TERMINATE_REASON (reason_set, reason_val) \ + (((((u64)reason_set) & 0x7) << 12) | \ + ((((u64)reason_val) & 0xff) << 16)) +#define GHCB_SEV_ES_REASON_GENERAL_REQUEST 0 +#define GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED 1#define GHCB_SEV_GHCB_RESP_CODE (v) ((v) & 0xfff)#define VMGEXIT () { asm volatile("rep; vmmcall\n\r"); } +enum es_result { +ES_OK ,/* All good */ +ES_UNSUPPORTED ,/* Requested operation not supported */ +ES_VMM_ERROR ,/* Unexpected state from the VMM */ +ES_DECODE_FAILED ,/* Instruction decoding failed */ +ES_EXCEPTION ,/* Instruction caused exception */ +ES_RETRY ,/* Retry instruction emulation */ +}; + +struct es_fault_info { +unsigned long vector; +unsigned long error_code; +unsigned long cr2; +}; + +struct pt_regs ; + +/* ES instruction emulation context */ +struct es_em_ctxt { +struct pt_regs *regs; +struct insn insn; +struct es_fault_info fi; +}; +void do_vc_no_ghcb (struct pt_regs *regs,unsigned long exit_code);static inline u64 lower_bits (u64 val,unsigned int bits)
arch/x86/include/uapi/asm/svm.h +1 -0
@@ -29,6 +29,7 @@#define SVM_EXIT_WRITE_DR6 0x036#define SVM_EXIT_WRITE_DR7 0x037#define SVM_EXIT_EXCP_BASE 0x040 +#define SVM_EXIT_LAST_EXCP 0x05f#define SVM_EXIT_INTR 0x060#define SVM_EXIT_NMI 0x061#define SVM_EXIT_SMI 0x062
arch/x86/kernel/sev-es-shared.c +154 -0
163 lines not on this page · raw message
PATCH v7 24/72 x86/boot/compressed/64: Unmap GHCB page before booting the kernel
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Force a page-fault on any further accesses to the GHCB page when they shouldn't happen anymore. This will catch the bugs where a #VC exception is raised when no one is expected anymore.
--- arch/x86/boot/compressed/ident_map_64.c | 17 +++++++++++++++-- arch/x86/boot/compressed/misc.h | 6 ++++++ arch/x86/boot/compressed/sev-es.c | 14 ++++++++++++++ 3 files changed, 35 insertions(+), 2 deletions(-)
arch/x86/boot/compressed/ident_map_64.c +15 -2
@@ -298,6 +298,11 @@ int set_page_encrypted(unsigned long address)return set_clr_page_flags (&mapping_info, address, _PAGE_ENC,0 ); } +int set_page_non_present (unsigned long address) +{ +return set_clr_page_flags (&mapping_info, address,0 , _PAGE_PRESENT); +} +static void do_pf_error (const char *msg,unsigned long error_code,unsigned long address,unsigned long ip) {
@@ -316,8 +321,14 @@ static void do_pf_error(const char *msg, unsigned long error_code,void do_boot_page_fault (struct pt_regs *regs,unsigned long error_code) { -unsigned long address =native_read_cr2 () &PMD_MASK ; -unsigned long end = address +PMD_SIZE ; +unsigned long address =native_read_cr2 (); +unsigned long end; +bool ghcb_fault; + + ghcb_fault =sev_es_check_ghcb_fault (address); + + address &=PMD_MASK ; + end = address +PMD_SIZE ;/* * Check for unexpected error codes. Unexpected are:
@@ -327,6 +338,8 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)*/ if (error_code & (X86_PF_PROT |X86_PF_USER |X86_PF_RSVD ))do_pf_error ("Unexpected page-fault:" , error_code, address, regs->ip); +else if (ghcb_fault) +do_pf_error ("Page-fault on GHCB page:" , error_code, address, regs->ip);/* * Error code is sane - now identity map the 2M region around
arch/x86/boot/compressed/misc.h +6 -0
@@ -100,6 +100,7 @@ static inline void choose_random_location(unsigned long input,#ifdef CONFIG_X86_64 extern int set_page_decrypted (unsigned long address);extern int set_page_encrypted (unsigned long address); +extern int set_page_non_present (unsigned long address);extern unsigned char _pgtable[];#endif
@@ -117,8 +118,13 @@ void set_sev_encryption_mask(void);#ifdef CONFIG_AMD_MEM_ENCRYPT void sev_es_shutdown_ghcb (void ); +extern bool sev_es_check_ghcb_fault (unsigned long address);#else static inline void sev_es_shutdown_ghcb (void ) { } +static inline bool sev_es_check_ghcb_fault (unsigned long address) +{ +return false; +}#endif /* acpi.c */
arch/x86/boot/compressed/sev-es.c +14 -0
@@ -121,6 +121,20 @@ void sev_es_shutdown_ghcb(void)*/ if (set_page_encrypted ((unsigned long )&boot_ghcb_page))error ("Can't map GHCB page encrypted" ); + +/* +* GHCB page is mapped encrypted again and flushed from the cache. +* Mark it non-present now to catch bugs when #VC exceptions trigger +* after this point. +*/ +if (set_page_non_present ((unsigned long )&boot_ghcb_page)) +error ("Can't unmap GHCB page" ); +} + +bool sev_es_check_ghcb_fault (unsigned long address) +{ +/* Check whether the fault was on the GHCB page */ +return ((address &PAGE_MASK ) == (unsigned long )&boot_ghcb_page); }void do_boot_stage2_vc (struct pt_regs *regs,unsigned long exit_code)
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/boot/compressed/64: Unmap GHCB page before booting the kernel
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 69add17a7c1992593a7cf775a66e0256ad4b3ef8 Gitweb: https://git.kernel.org/tip/69add17a7c1992593a7cf775a66e0256ad4b3ef8 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:25 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:26 +02:00
x86/boot/compressed/64: Unmap GHCB page before booting the kernel
Force a page-fault on any further accesses to the GHCB page when they shouldn't happen anymore. This will catch any bugs where a #VC exception is raised even though none is expected anymore.
--- arch/x86/boot/compressed/ident_map_64.c | 17 +++++++++++++++-- arch/x86/boot/compressed/misc.h | 6 ++++++ arch/x86/boot/compressed/sev-es.c | 14 ++++++++++++++ 3 files changed, 35 insertions(+), 2 deletions(-)
arch/x86/boot/compressed/ident_map_64.c +15 -2
@@ -298,6 +298,11 @@ int set_page_encrypted(unsigned long address)return set_clr_page_flags (&mapping_info, address, _PAGE_ENC,0 ); } +int set_page_non_present (unsigned long address) +{ +return set_clr_page_flags (&mapping_info, address,0 , _PAGE_PRESENT); +} +static void do_pf_error (const char *msg,unsigned long error_code,unsigned long address,unsigned long ip) {
@@ -316,8 +321,14 @@ static void do_pf_error(const char *msg, unsigned long error_code,void do_boot_page_fault (struct pt_regs *regs,unsigned long error_code) { -unsigned long address =native_read_cr2 () &PMD_MASK ; -unsigned long end = address +PMD_SIZE ; +unsigned long address =native_read_cr2 (); +unsigned long end; +bool ghcb_fault; + + ghcb_fault =sev_es_check_ghcb_fault (address); + + address &=PMD_MASK ; + end = address +PMD_SIZE ;/* * Check for unexpected error codes. Unexpected are:
@@ -327,6 +338,8 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)*/ if (error_code & (X86_PF_PROT |X86_PF_USER |X86_PF_RSVD ))do_pf_error ("Unexpected page-fault:" , error_code, address, regs->ip); +else if (ghcb_fault) +do_pf_error ("Page-fault on GHCB page:" , error_code, address, regs->ip);/* * Error code is sane - now identity map the 2M region around
arch/x86/boot/compressed/misc.h +6 -0
@@ -100,6 +100,7 @@ static inline void choose_random_location(unsigned long input,#ifdef CONFIG_X86_64 extern int set_page_decrypted (unsigned long address);extern int set_page_encrypted (unsigned long address); +extern int set_page_non_present (unsigned long address);extern unsigned char _pgtable[];#endif
@@ -117,8 +118,13 @@ void set_sev_encryption_mask(void);#ifdef CONFIG_AMD_MEM_ENCRYPT void sev_es_shutdown_ghcb (void ); +extern bool sev_es_check_ghcb_fault (unsigned long address);#else static inline void sev_es_shutdown_ghcb (void ) { } +static inline bool sev_es_check_ghcb_fault (unsigned long address) +{ +return false; +}#endif /* acpi.c */
arch/x86/boot/compressed/sev-es.c +14 -0
@@ -121,6 +121,20 @@ void sev_es_shutdown_ghcb(void)*/ if (set_page_encrypted ((unsigned long )&boot_ghcb_page))error ("Can't map GHCB page encrypted" ); + +/* +* GHCB page is mapped encrypted again and flushed from the cache. +* Mark it non-present now to catch bugs when #VC exceptions trigger +* after this point. +*/ +if (set_page_non_present ((unsigned long )&boot_ghcb_page)) +error ("Can't unmap GHCB page" ); +} + +bool sev_es_check_ghcb_fault (unsigned long address) +{ +/* Check whether the fault was on the GHCB page */ +return ((address &PAGE_MASK ) == (unsigned long )&boot_ghcb_page); }void do_boot_stage2_vc (struct pt_regs *regs,unsigned long exit_code)
PATCH v7 25/72 x86/sev-es: Add support for handling IOIO exceptions
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
Add support for decoding and handling #VC exceptions for IOIO events.
--- arch/x86/boot/compressed/sev-es.c | 32 +++++ arch/x86/kernel/sev-es-shared.c | 214 ++++++++++++++++++++++++++++++ 2 files changed, 246 insertions(+)
arch/x86/boot/compressed/sev-es.c +32 -0
@@ -24,6 +24,35 @@struct ghcb boot_ghcb_page__aligned(PAGE_SIZE) ;struct ghcb *boot_ghcb; +/* +* Copy a version of this function here - insn-eval.c can't be used in +* pre-decompression code. +*/ +static bool insn_has_rep_prefix (struct insn *insn) +{ +int i; + +insn_get_prefixes (insn); + +for (i =0 ; i < insn->prefixes.nbytes; i++) { +insn_byte_t p = insn->prefixes.bytes[i]; + +if (p ==0xf2 || p ==0xf3 ) +return true; + } + +return false; +} + +/* +* Only a dummy for insn_get_seg_base() - Early boot-code is 64bit only and +* doesn't use segments. +*/ +static unsigned long insn_get_seg_base (struct pt_regs *regs,int seg_reg_idx) +{ +return 0UL ; +} +static inline u64 sev_es_rd_ghcb_msr (void ) {unsigned long low, high;
@@ -151,6 +180,9 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code) goto finish;switch (exit_code) { +case SVM_EXIT_IOIO : + result =vc_handle_ioio (boot_ghcb, &ctxt); +break ;default : result =ES_UNSUPPORTED ;break ;
arch/x86/kernel/sev-es-shared.c +214 -0
217 lines not on this page · raw message
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Add support for handling IOIO exceptions
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 25189d08e5168c098c307a0eaae5b30c13a331ef Gitweb: https://git.kernel.org/tip/25189d08e5168c098c307a0eaae5b30c13a331ef Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:26 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:45:26 +02:00
x86/sev-es: Add support for handling IOIO exceptions
Add support for decoding and handling #VC exceptions for IOIO events.
--- arch/x86/boot/compressed/sev-es.c | 32 ++++- arch/x86/kernel/sev-es-shared.c | 214 +++++++++++++++++++++++++++++- 2 files changed, 246 insertions(+)
arch/x86/boot/compressed/sev-es.c +32 -0
@@ -24,6 +24,35 @@struct ghcb boot_ghcb_page__aligned(PAGE_SIZE) ;struct ghcb *boot_ghcb; +/* +* Copy a version of this function here - insn-eval.c can't be used in +* pre-decompression code. +*/ +static bool insn_has_rep_prefix (struct insn *insn) +{ +int i; + +insn_get_prefixes (insn); + +for (i =0 ; i < insn->prefixes.nbytes; i++) { +insn_byte_t p = insn->prefixes.bytes[i]; + +if (p ==0xf2 || p ==0xf3 ) +return true; + } + +return false; +} + +/* +* Only a dummy for insn_get_seg_base() - Early boot-code is 64bit only and +* doesn't use segments. +*/ +static unsigned long insn_get_seg_base (struct pt_regs *regs,int seg_reg_idx) +{ +return 0UL ; +} +static inline u64 sev_es_rd_ghcb_msr (void ) {unsigned long low, high;
@@ -151,6 +180,9 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code) goto finish;switch (exit_code) { +case SVM_EXIT_IOIO : + result =vc_handle_ioio (boot_ghcb, &ctxt); +break ;default : result =ES_UNSUPPORTED ;break ;
arch/x86/kernel/sev-es-shared.c +214 -0
217 lines not on this page · raw message
PATCH v7 26/72 x86/fpu: Move xgetbv()/xsetbv() into separate header
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The xgetbv() function is needed in pre-decompression boot code, but asm/fpu/internal.h can't be included there directly. Doing so opens the door to include-hell due to various include-magic in boot/compressed/misc.h.
Avoid that by moving xgetbv()/xsetbv() to a separate header file and include this instead.
--- arch/x86/include/asm/fpu/internal.h | 30 +------------------------ arch/x86/include/asm/fpu/xcr.h | 34 +++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 29 deletions(-) create mode 100644 arch/x86/include/asm/fpu/xcr.h
arch/x86/include/asm/fpu/internal.h +1 -29
@@ -19,6 +19,7 @@#include <asm/user.h> #include <asm/fpu/api.h> #include <asm/fpu/xstate.h> +#include <asm/fpu/xcr.h> #include <asm/cpufeature.h> #include <asm/trace/fpu.h>
@@ -585,33 +586,4 @@ static inline void switch_fpu_finish(struct fpu *new_fpu) __write_pkru(pkru_val); } -/* -* MXCSR and XCR definitions: -*/ - -static inline void ldmxcsr (u32 mxcsr) -{ - asmvolatile ("ldmxcsr %0" ::"m" (mxcsr)); -} - -extern unsigned int mxcsr_feature_mask; - -#define XCR_XFEATURE_ENABLED_MASK 0x00000000 - -static inline u64 xgetbv (u32 index) -{ -u32 eax, edx; - - asmvolatile ("xgetbv" :"=a" (eax),"=d" (edx) :"c" (index)); -return eax + ((u64 )edx <<32 ); -} - -static inline void xsetbv (u32 index,u64 value) -{ -u32 eax = value; -u32 edx = value >>32 ; - - asmvolatile ("xsetbv" ::"a" (eax),"d" (edx),"c" (index)); -} -#endif /* _ASM_X86_FPU_INTERNAL_H */
arch/x86/include/asm/fpu/xcr.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_FPU_XCR_H +#define _ASM_X86_FPU_XCR_H + +/* +* MXCSR and XCR definitions: +*/ + +static inline void ldmxcsr (u32 mxcsr) +{ + asmvolatile ("ldmxcsr %0" ::"m" (mxcsr)); +} + +extern unsigned int mxcsr_feature_mask; + +#define XCR_XFEATURE_ENABLED_MASK 0x00000000 + +static inline u64 xgetbv (u32 index) +{ +u32 eax, edx; + + asmvolatile ("xgetbv" :"=a" (eax),"=d" (edx) :"c" (index)); +return eax + ((u64 )edx <<32 ); +} + +static inline void xsetbv (u32 index,u64 value) +{ +u32 eax = value; +u32 edx = value >>32 ; + + asmvolatile ("xsetbv" ::"a" (eax),"d" (edx),"c" (index)); +} + +#endif /* _ASM_X86_FPU_XCR_H */
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/fpu: Move xgetbv()/xsetbv() into a separate header
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 1b4fb8545f2b00f2844c4b7619d64d98440a477c Gitweb: https://git.kernel.org/tip/1b4fb8545f2b00f2844c4b7619d64d98440a477c Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:27 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 19:54:20 +02:00
x86/fpu: Move xgetbv()/xsetbv() into a separate header
The xgetbv() function is needed in the pre-decompression boot code, but asm/fpu/internal.h can't be included there directly. Doing so opens the door to include-hell due to various include-magic in boot/compressed/misc.h.
Avoid that by moving xgetbv()/xsetbv() to a separate header file and include it instead.
--- arch/x86/include/asm/fpu/internal.h | 30 +------------------------- arch/x86/include/asm/fpu/xcr.h | 34 ++++++++++++++++++++++++++++- 2 files changed, 35 insertions(+), 29 deletions(-) create mode 100644 arch/x86/include/asm/fpu/xcr.h
arch/x86/include/asm/fpu/internal.h +1 -29
@@ -19,6 +19,7 @@#include <asm/user.h> #include <asm/fpu/api.h> #include <asm/fpu/xstate.h> +#include <asm/fpu/xcr.h> #include <asm/cpufeature.h> #include <asm/trace/fpu.h>
@@ -585,33 +586,4 @@ static inline void switch_fpu_finish(struct fpu *new_fpu) __write_pkru(pkru_val); } -/* -* MXCSR and XCR definitions: -*/ - -static inline void ldmxcsr (u32 mxcsr) -{ - asmvolatile ("ldmxcsr %0" ::"m" (mxcsr)); -} - -extern unsigned int mxcsr_feature_mask; - -#define XCR_XFEATURE_ENABLED_MASK 0x00000000 - -static inline u64 xgetbv (u32 index) -{ -u32 eax, edx; - - asmvolatile ("xgetbv" :"=a" (eax),"=d" (edx) :"c" (index)); -return eax + ((u64 )edx <<32 ); -} - -static inline void xsetbv (u32 index,u64 value) -{ -u32 eax = value; -u32 edx = value >>32 ; - - asmvolatile ("xsetbv" ::"a" (eax),"d" (edx),"c" (index)); -} -#endif /* _ASM_X86_FPU_INTERNAL_H */
arch/x86/include/asm/fpu/xcr.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_FPU_XCR_H +#define _ASM_X86_FPU_XCR_H + +/* +* MXCSR and XCR definitions: +*/ + +static inline void ldmxcsr (u32 mxcsr) +{ + asmvolatile ("ldmxcsr %0" ::"m" (mxcsr)); +} + +extern unsigned int mxcsr_feature_mask; + +#define XCR_XFEATURE_ENABLED_MASK 0x00000000 + +static inline u64 xgetbv (u32 index) +{ +u32 eax, edx; + + asmvolatile ("xgetbv" :"=a" (eax),"=d" (edx) :"c" (index)); +return eax + ((u64 )edx <<32 ); +} + +static inline void xsetbv (u32 index,u64 value) +{ +u32 eax = value; +u32 edx = value >>32 ; + + asmvolatile ("xsetbv" ::"a" (eax),"d" (edx),"c" (index)); +} + +#endif /* _ASM_X86_FPU_XCR_H */
PATCH v7 27/72 x86/sev-es: Add CPUID handling to #VC handler
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
Handle #VC exceptions caused by CPUID instructions. These happen in early boot code when the KASLR code checks for RDTSC.
--- arch/x86/boot/compressed/sev-es.c | 4 ++++ arch/x86/kernel/sev-es-shared.c | 35 +++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
arch/x86/boot/compressed/sev-es.c +4 -0
@@ -16,6 +16,7 @@#include <asm/trapnr.h> #include <asm/trap_pf.h> #include <asm/msr-index.h> +#include <asm/fpu/xcr.h> #include <asm/ptrace.h> #include <asm/svm.h>
@@ -183,6 +184,9 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)case SVM_EXIT_IOIO : result =vc_handle_ioio (boot_ghcb, &ctxt);break ; +case SVM_EXIT_CPUID : + result =vc_handle_cpuid (boot_ghcb, &ctxt); +break ;default : result =ES_UNSUPPORTED ;break ;
arch/x86/kernel/sev-es-shared.c +35 -0
@@ -432,3 +432,38 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)return ret; } + +static enum es_result vc_handle_cpuid (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) +{ +struct pt_regs *regs = ctxt->regs; +u32 cr4 =native_read_cr4 (); +enum es_result ret; + +ghcb_set_rax (ghcb, regs->ax); +ghcb_set_rcx (ghcb, regs->cx); + +if (cr4 &X86_CR4_OSXSAVE ) +/* Safe to read xcr0 */ +ghcb_set_xcr0 (ghcb,xgetbv (XCR_XFEATURE_ENABLED_MASK )); +else +/* xgetbv will cause #GP - use reset value for xcr0 */ +ghcb_set_xcr0 (ghcb,1 ); + + ret =sev_es_ghcb_hv_call (ghcb, ctxt,SVM_EXIT_CPUID ,0 ,0 ); +if (ret !=ES_OK ) +return ret; + +if (!(ghcb_rax_is_valid (ghcb) && +ghcb_rbx_is_valid (ghcb) && +ghcb_rcx_is_valid (ghcb) && +ghcb_rdx_is_valid (ghcb))) +return ES_VMM_ERROR ; + + regs->ax = ghcb->save.rax; + regs->bx = ghcb->save.rbx; + regs->cx = ghcb->save.rcx; + regs->dx = ghcb->save.rdx; + +return ES_OK ; +}
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Add CPUID handling to #VC handler
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: a7de15d489d956217b47671705ac2218ca50eaae Gitweb: https://git.kernel.org/tip/a7de15d489d956217b47671705ac2218ca50eaae Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:28 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 20:15:51 +02:00
x86/sev-es: Add CPUID handling to #VC handler
Handle #VC exceptions caused by CPUID instructions. These happen in early boot code when the KASLR code checks for RDTSC.
--- arch/x86/boot/compressed/sev-es.c | 4 +++- arch/x86/kernel/sev-es-shared.c | 35 ++++++++++++++++++++++++++++++- 2 files changed, 39 insertions(+)
arch/x86/boot/compressed/sev-es.c +4 -0
@@ -16,6 +16,7 @@#include <asm/trapnr.h> #include <asm/trap_pf.h> #include <asm/msr-index.h> +#include <asm/fpu/xcr.h> #include <asm/ptrace.h> #include <asm/svm.h>
@@ -183,6 +184,9 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)case SVM_EXIT_IOIO : result =vc_handle_ioio (boot_ghcb, &ctxt);break ; +case SVM_EXIT_CPUID : + result =vc_handle_cpuid (boot_ghcb, &ctxt); +break ;default : result =ES_UNSUPPORTED ;break ;
arch/x86/kernel/sev-es-shared.c +35 -0
@@ -432,3 +432,38 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)return ret; } + +static enum es_result vc_handle_cpuid (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) +{ +struct pt_regs *regs = ctxt->regs; +u32 cr4 =native_read_cr4 (); +enum es_result ret; + +ghcb_set_rax (ghcb, regs->ax); +ghcb_set_rcx (ghcb, regs->cx); + +if (cr4 &X86_CR4_OSXSAVE ) +/* Safe to read xcr0 */ +ghcb_set_xcr0 (ghcb,xgetbv (XCR_XFEATURE_ENABLED_MASK )); +else +/* xgetbv will cause #GP - use reset value for xcr0 */ +ghcb_set_xcr0 (ghcb,1 ); + + ret =sev_es_ghcb_hv_call (ghcb, ctxt,SVM_EXIT_CPUID ,0 ,0 ); +if (ret !=ES_OK ) +return ret; + +if (!(ghcb_rax_is_valid (ghcb) && +ghcb_rbx_is_valid (ghcb) && +ghcb_rcx_is_valid (ghcb) && +ghcb_rdx_is_valid (ghcb))) +return ES_VMM_ERROR ; + + regs->ax = ghcb->save.rax; + regs->bx = ghcb->save.rbx; + regs->cx = ghcb->save.rcx; + regs->dx = ghcb->save.rdx; + +return ES_OK ; +}
PATCH v7 28/72 x86/idt: Split idt_data setup out of set_intr_gate()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The code to setup idt_data is needed for early exception handling, but set_intr_gate() can't be used that early because it has pv-ops in its code path, which don't work that early.
Split out the idt_data initialization part from set_intr_gate() so that it can be used separately.
--- arch/x86/kernel/idt.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
arch/x86/kernel/idt.c +14 -8
@@ -205,18 +205,24 @@ idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sy } } +static void init_idt_data (struct idt_data *data,unsigned int n, +const void *addr) +{ +BUG_ON (n >0xFF ); + +memset (data,0 ,sizeof (*data)); + data->vector = n; + data->addr = addr; + data->segment = __KERNEL_CS; + data->bits.type =GATE_INTERRUPT ; + data->bits.p =1 ; +} +static __init void set_intr_gate (unsigned int n,const void *addr) {struct idt_data data; -BUG_ON (n >0xFF ); - -memset (&data,0 ,sizeof (data)); - data.vector = n; - data.addr = addr; - data.segment = __KERNEL_CS; - data.bits.type =GATE_INTERRUPT ; - data.bits.p =1 ; +init_idt_data (&data, n, addr);idt_setup_from_table (idt_table, &data,1 , false); }
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/idt: Split idt_data setup out of set_intr_gate()
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 4bed2266cc6f9c3f6cd91378ea4fc76edde674cf Gitweb: https://git.kernel.org/tip/4bed2266cc6f9c3f6cd91378ea4fc76edde674cf Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:29 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 21:30:38 +02:00
x86/idt: Split idt_data setup out of set_intr_gate()
The code to setup idt_data is needed for early exception handling, but set_intr_gate() can't be used that early because it has pv-ops in its code path which don't work that early.
Split out the idt_data initialization part from set_intr_gate() so that it can be used separately.
--- arch/x86/kernel/idt.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
arch/x86/kernel/idt.c +14 -8
@@ -205,18 +205,24 @@ idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sy } } +static void init_idt_data (struct idt_data *data,unsigned int n, +const void *addr) +{ +BUG_ON (n >0xFF ); + +memset (data,0 ,sizeof (*data)); + data->vector = n; + data->addr = addr; + data->segment = __KERNEL_CS; + data->bits.type =GATE_INTERRUPT ; + data->bits.p =1 ; +} +static __init void set_intr_gate (unsigned int n,const void *addr) {struct idt_data data; -BUG_ON (n >0xFF ); - -memset (&data,0 ,sizeof (data)); - data.vector = n; - data.addr = addr; - data.segment = __KERNEL_CS; - data.bits.type =GATE_INTERRUPT ; - data.bits.p =1 ; +init_idt_data (&data, n, addr);idt_setup_from_table (idt_table, &data,1 , false); }
PATCH v7 29/72 x86/head/64: Install startup GDT
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Handling exceptions during boot requires a working GDT. The kernel GDT can't be used on the direct mapping, so load a startup GDT and setup segments.
--- arch/x86/include/asm/setup.h | 1 + arch/x86/kernel/head64.c | 33 +++++++++++++++++++++++++++++++++ arch/x86/kernel/head_64.S | 14 ++++++++++++++ 3 files changed, 48 insertions(+)
arch/x86/include/asm/setup.h +1 -0
@@ -48,6 +48,7 @@ extern void reserve_standard_io_resources(void);extern void i386_reserve_resources (void );extern unsigned long __startup_64 (unsigned long physaddr,struct boot_params *bp);extern unsigned long __startup_secondary_64 (void ); +extern void startup_64_setup_env (unsigned long physbase);extern int early_make_pgtable (unsigned long address);#ifdef CONFIG_X86_INTEL_MID
arch/x86/kernel/head64.c +33 -0
@@ -61,6 +61,24 @@ unsigned long vmemmap_base __ro_after_init = __VMEMMAP_BASE_L4;EXPORT_SYMBOL (vmemmap_base);#endif +/* +* GDT used on the boot CPU before switching to virtual addresses. +*/ +static struct desc_struct startup_gdt[GDT_ENTRIES ] = { + [GDT_ENTRY_KERNEL32_CS ] =GDT_ENTRY_INIT (0xc09b ,0 ,0xfffff ), + [GDT_ENTRY_KERNEL_CS ] =GDT_ENTRY_INIT (0xa09b ,0 ,0xfffff ), + [GDT_ENTRY_KERNEL_DS ] =GDT_ENTRY_INIT (0xc093 ,0 ,0xfffff ), +}; + +/* +* Address needs to be set at runtime because it references the startup_gdt +* while the kernel still uses a direct mapping. +*/ +static struct desc_ptr startup_gdt_descr = { + .size =sizeof (startup_gdt), + .address =0 , +}; +#define __head __section(.head.text) static void __head *fixup_pointer(void *ptr, unsigned long physaddr)
@@ -489,3 +507,18 @@ void __init x86_64_start_reservations(char *real_mode_data)start_kernel (); } + +/* +* Setup boot CPU state needed before kernel switches to virtual addresses. +*/ +void __head startup_64_setup_env(unsigned long physbase) +{ +/* Load GDT */ + startup_gdt_descr.address = (unsigned long )fixup_pointer (startup_gdt, physbase); +native_load_gdt (&startup_gdt_descr); + +/* New GDT is live - reload data segment registers */ + asmvolatile ("movl %%eax, %%ds\n" +"movl %%eax, %%ss\n" +"movl %%eax, %%es\n" : :"a" (__KERNEL_DS) :"memory" ); +}
arch/x86/kernel/head_64.S +14 -0
@@ -73,6 +73,20 @@ SYM_CODE_START_NOALIGN(startup_64) /* Set up the stack for verify_cpu(), similar to initial_stack below */ leaq (__end_init_task - SIZEOF_PTREGS)(%rip), %rsp + leaq _text(%rip), %rdi + pushq %rsi + call startup_64_setup_env + popq %rsi + + /* Now switch to __KERNEL_CS so IRET works reliably */ + pushq $__KERNEL_CS + leaq .Lon_kernel_cs(%rip), %rax + pushq %rax + lretq + +.Lon_kernel_cs: + UNWIND_HINT_EMPTY + /* Sanitize CPU configuration */ call verify_cpu
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/head/64: Install startup GDT
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 866b556efa1295934ed0bc20c2f208c93a873fb0 Gitweb: https://git.kernel.org/tip/866b556efa1295934ed0bc20c2f208c93a873fb0 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:30 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 21:33:17 +02:00
x86/head/64: Install startup GDT
Handling exceptions during boot requires a working GDT. The kernel GDT can't be used on the direct mapping, so load a startup GDT and setup segments.
--- arch/x86/include/asm/setup.h | 1 + arch/x86/kernel/head64.c | 33 +++++++++++++++++++++++++++++++++ arch/x86/kernel/head_64.S | 14 ++++++++++++++ 3 files changed, 48 insertions(+)
arch/x86/include/asm/setup.h +1 -0
@@ -48,6 +48,7 @@ extern void reserve_standard_io_resources(void);extern void i386_reserve_resources (void );extern unsigned long __startup_64 (unsigned long physaddr,struct boot_params *bp);extern unsigned long __startup_secondary_64 (void ); +extern void startup_64_setup_env (unsigned long physbase);extern int early_make_pgtable (unsigned long address);#ifdef CONFIG_X86_INTEL_MID
arch/x86/kernel/head64.c +33 -0
@@ -61,6 +61,24 @@ unsigned long vmemmap_base __ro_after_init = __VMEMMAP_BASE_L4;EXPORT_SYMBOL (vmemmap_base);#endif +/* +* GDT used on the boot CPU before switching to virtual addresses. +*/ +static struct desc_struct startup_gdt[GDT_ENTRIES ] = { + [GDT_ENTRY_KERNEL32_CS ] =GDT_ENTRY_INIT (0xc09b ,0 ,0xfffff ), + [GDT_ENTRY_KERNEL_CS ] =GDT_ENTRY_INIT (0xa09b ,0 ,0xfffff ), + [GDT_ENTRY_KERNEL_DS ] =GDT_ENTRY_INIT (0xc093 ,0 ,0xfffff ), +}; + +/* +* Address needs to be set at runtime because it references the startup_gdt +* while the kernel still uses a direct mapping. +*/ +static struct desc_ptr startup_gdt_descr = { + .size =sizeof (startup_gdt), + .address =0 , +}; +#define __head __section(.head.text) static void __head *fixup_pointer(void *ptr, unsigned long physaddr)
@@ -489,3 +507,18 @@ void __init x86_64_start_reservations(char *real_mode_data)start_kernel (); } + +/* +* Setup boot CPU state needed before kernel switches to virtual addresses. +*/ +void __head startup_64_setup_env(unsigned long physbase) +{ +/* Load GDT */ + startup_gdt_descr.address = (unsigned long )fixup_pointer (startup_gdt, physbase); +native_load_gdt (&startup_gdt_descr); + +/* New GDT is live - reload data segment registers */ + asmvolatile ("movl %%eax, %%ds\n" +"movl %%eax, %%ss\n" +"movl %%eax, %%es\n" : :"a" (__KERNEL_DS) :"memory" ); +}
arch/x86/kernel/head_64.S +14 -0
@@ -73,6 +73,20 @@ SYM_CODE_START_NOALIGN(startup_64) /* Set up the stack for verify_cpu(), similar to initial_stack below */ leaq (__end_init_task - SIZEOF_PTREGS)(%rip), %rsp + leaq _text(%rip), %rdi + pushq %rsi + call startup_64_setup_env + popq %rsi + + /* Now switch to __KERNEL_CS so IRET works reliably */ + pushq $__KERNEL_CS + leaq .Lon_kernel_cs(%rip), %rax + pushq %rax + lretq + +.Lon_kernel_cs: + UNWIND_HINT_EMPTY + /* Sanitize CPU configuration */ call verify_cpu
PATCH v7 30/72 x86/head/64: Load GDT after switch to virtual addresses
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Load the GDT right after switching to virtual addresses to make sure there is a defined GDT for exception handling.
--- arch/x86/kernel/head_64.S | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
arch/x86/kernel/head_64.S +8 -8
@@ -158,6 +158,14 @@ SYM_CODE_START(secondary_startup_64) 1: UNWIND_HINT_EMPTY + /* + * We must switch to a new descriptor in kernel space for the GDT + * because soon the kernel won't have access anymore to the userspace + * addresses where we're currently running on. We have to do that here + * because in 32bit we couldn't load a 64bit linear address. + */ + lgdt early_gdt_descr(%rip) + /* Check if nx is implemented */ movl $0x80000001, %eax cpuid
@@ -185,14 +193,6 @@ SYM_CODE_START(secondary_startup_64) pushq $0 popfq - /* - * We must switch to a new descriptor in kernel space for the GDT - * because soon the kernel won't have access anymore to the userspace - * addresses where we're currently running on. We have to do that here - * because in 32bit we couldn't load a 64bit linear address. - */ - lgdt early_gdt_descr(%rip) - /* set up data segments */ xorl %eax,%eax movl %eax,%ds
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/head/64: Load GDT after switch to virtual addresses
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: e04b88336360e101329add0c05e5cb1cebae64fd Gitweb: https://git.kernel.org/tip/e04b88336360e101329add0c05e5cb1cebae64fd Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:31 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 21:35:54 +02:00
x86/head/64: Load GDT after switch to virtual addresses
Load the GDT right after switching to virtual addresses to make sure there is a defined GDT for exception handling.
--- arch/x86/kernel/head_64.S | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
arch/x86/kernel/head_64.S +8 -8
@@ -158,6 +158,14 @@ SYM_CODE_START(secondary_startup_64) 1: UNWIND_HINT_EMPTY + /* + * We must switch to a new descriptor in kernel space for the GDT + * because soon the kernel won't have access anymore to the userspace + * addresses where we're currently running on. We have to do that here + * because in 32bit we couldn't load a 64bit linear address. + */ + lgdt early_gdt_descr(%rip) + /* Check if nx is implemented */ movl $0x80000001, %eax cpuid
@@ -185,14 +193,6 @@ SYM_CODE_START(secondary_startup_64) pushq $0 popfq - /* - * We must switch to a new descriptor in kernel space for the GDT - * because soon the kernel won't have access anymore to the userspace - * addresses where we're currently running on. We have to do that here - * because in 32bit we couldn't load a 64bit linear address. - */ - lgdt early_gdt_descr(%rip) - /* set up data segments */ xorl %eax,%eax movl %eax,%ds
PATCH v7 31/72 x86/head/64: Load segment registers earlier
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Make sure segments are properly set up before setting up an IDT and doing anything that might cause a #VC exception. This is later needed for early exception handling.
--- arch/x86/kernel/head_64.S | 52 +++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
arch/x86/kernel/head_64.S +26 -26
@@ -166,6 +166,32 @@ SYM_CODE_START(secondary_startup_64) */ lgdt early_gdt_descr(%rip) + /* set up data segments */ + xorl %eax,%eax + movl %eax,%ds + movl %eax,%ss + movl %eax,%es + + /* + * We don't really need to load %fs or %gs, but load them anyway + * to kill any stale realmode selectors. This allows execution + * under VT hardware. + */ + movl %eax,%fs + movl %eax,%gs + + /* Set up %gs. + * + * The base of %gs always points to fixed_percpu_data. If the + * stack protector canary is enabled, it is located at %gs:40. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up. + */ + movl $MSR_GS_BASE,%ecx + movl initial_gs(%rip),%eax + movl initial_gs+4(%rip),%edx + wrmsr + /* Check if nx is implemented */ movl $0x80000001, %eax cpuid
@@ -193,32 +219,6 @@ SYM_CODE_START(secondary_startup_64) pushq $0 popfq - /* set up data segments */ - xorl %eax,%eax - movl %eax,%ds - movl %eax,%ss - movl %eax,%es - - /* - * We don't really need to load %fs or %gs, but load them anyway - * to kill any stale realmode selectors. This allows execution - * under VT hardware. - */ - movl %eax,%fs - movl %eax,%gs - - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. - * Note that, on SMP, the boot cpu uses init data section until - * the per cpu areas are set up. - */ - movl $MSR_GS_BASE,%ecx - movl initial_gs(%rip),%eax - movl initial_gs+4(%rip),%edx - wrmsr - /* rsi is pointer to real mode structure with interesting info. pass it to C */ movq %rsi, %rdi
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/head/64: Load segment registers earlier
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 7b99819dfb60268cc1c75f83c949bc4a09221bea Gitweb: https://git.kernel.org/tip/7b99819dfb60268cc1c75f83c949bc4a09221bea Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:32 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 21:40:53 +02:00
x86/head/64: Load segment registers earlier
Make sure segments are properly set up before setting up an IDT and doing anything that might cause a #VC exception. This is later needed for early exception handling.
--- arch/x86/kernel/head_64.S | 52 +++++++++++++++++++------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
arch/x86/kernel/head_64.S +26 -26
@@ -166,6 +166,32 @@ SYM_CODE_START(secondary_startup_64) */ lgdt early_gdt_descr(%rip) + /* set up data segments */ + xorl %eax,%eax + movl %eax,%ds + movl %eax,%ss + movl %eax,%es + + /* + * We don't really need to load %fs or %gs, but load them anyway + * to kill any stale realmode selectors. This allows execution + * under VT hardware. + */ + movl %eax,%fs + movl %eax,%gs + + /* Set up %gs. + * + * The base of %gs always points to fixed_percpu_data. If the + * stack protector canary is enabled, it is located at %gs:40. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up. + */ + movl $MSR_GS_BASE,%ecx + movl initial_gs(%rip),%eax + movl initial_gs+4(%rip),%edx + wrmsr + /* Check if nx is implemented */ movl $0x80000001, %eax cpuid
@@ -193,32 +219,6 @@ SYM_CODE_START(secondary_startup_64) pushq $0 popfq - /* set up data segments */ - xorl %eax,%eax - movl %eax,%ds - movl %eax,%ss - movl %eax,%es - - /* - * We don't really need to load %fs or %gs, but load them anyway - * to kill any stale realmode selectors. This allows execution - * under VT hardware. - */ - movl %eax,%fs - movl %eax,%gs - - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. - * Note that, on SMP, the boot cpu uses init data section until - * the per cpu areas are set up. - */ - movl $MSR_GS_BASE,%ecx - movl initial_gs(%rip),%eax - movl initial_gs+4(%rip),%edx - wrmsr - /* rsi is pointer to real mode structure with interesting info. pass it to C */ movq %rsi, %rdi
PATCH v7 32/72 x86/head/64: Switch to initial stack earlier
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Make sure there is a stack once the kernel runs from virual addresses. At this stage any secondary CPU which boots will have lost its stack because the kernel switched to a new page-table which does not map the real-mode stack anymore.
This is needed for handling early #VC exceptions caused by instructions like CPUID.
--- arch/x86/kernel/head_64.S | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
arch/x86/kernel/head_64.S +6 -3
@@ -192,6 +192,12 @@ SYM_CODE_START(secondary_startup_64) movl initial_gs+4(%rip),%edx wrmsr + /* + * Setup a boot time stack - Any secondary CPU will have lost its stack + * by now because the cr3-switch above unmaps the real-mode stack + */ + movq initial_stack(%rip), %rsp + /* Check if nx is implemented */ movl $0x80000001, %eax cpuid
@@ -212,9 +218,6 @@ SYM_CODE_START(secondary_startup_64) /* Make changes effective */ movq %rax, %cr0 - /* Setup a boot time stack */ - movq initial_stack(%rip), %rsp - /* zero EFLAGS after setting rsp */ pushq $0 popfq
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/head/64: Switch to initial stack earlier
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 3add38cb96a1ae7d152db69ab4329e809c2af2d4 Gitweb: https://git.kernel.org/tip/3add38cb96a1ae7d152db69ab4329e809c2af2d4 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:33 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 21:44:01 +02:00
x86/head/64: Switch to initial stack earlier
Make sure there is a stack once the kernel runs from virtual addresses. At this stage any secondary CPU which boots will have lost its stack because the kernel switched to a new page-table which does not map the real-mode stack anymore.
This is needed for handling early #VC exceptions caused by instructions like CPUID.
--- arch/x86/kernel/head_64.S | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
arch/x86/kernel/head_64.S +6 -3
@@ -192,6 +192,12 @@ SYM_CODE_START(secondary_startup_64) movl initial_gs+4(%rip),%edx wrmsr + /* + * Setup a boot time stack - Any secondary CPU will have lost its stack + * by now because the cr3-switch above unmaps the real-mode stack + */ + movq initial_stack(%rip), %rsp + /* Check if nx is implemented */ movl $0x80000001, %eax cpuid
@@ -212,9 +218,6 @@ SYM_CODE_START(secondary_startup_64) /* Make changes effective */ movq %rax, %cr0 - /* Setup a boot time stack */ - movq initial_stack(%rip), %rsp - /* zero EFLAGS after setting rsp */ pushq $0 popfq
PATCH v7 33/72 x86/head/64: Install a CPU bringup IDT
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Add a separate bringup IDT used by CPU bringup code that will be used until the kernel switches to the idt_table. There are two reasons for a separate IDT:
1) When the idt_table is set up and the secondary CPUs are booted, it contains entries (e.g. IST entries) which require certain CPU state to be set up. This includes a working TSS (for IST), MSR_GS_BASE (for stack protector) or CR4.FSGSBASE (for paranoid_entry) path. By using a dedicated IDT for early boot this state need not to be set up early.
2) The idt_table is static to idt.c, so any function using/modifying must be in idt.c too. That means that all compiler driven instrumentation like tracing or KASAN is also active in this code. But during early CPU bringup the environment is not set up for this instrumentation to work correctly.
To avoid all of these hassles and make early exception handling robust, use a dedicated bringup IDT.
The IDT is loaded two times, first on the boot CPU while the kernel is still running on direct mapped addresses, and again later when the switch to kernel addresses happened. The second IDT load happens on the boot and secondary CPUs.
--- arch/x86/include/asm/setup.h | 1 + arch/x86/kernel/head64.c | 39 ++++++++++++++++++++++++++++++++++++ arch/x86/kernel/head_64.S | 5 +++++ 3 files changed, 45 insertions(+)
arch/x86/include/asm/setup.h +1 -0
@@ -50,6 +50,7 @@ extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bpextern unsigned long __startup_secondary_64 (void );extern void startup_64_setup_env (unsigned long physbase);extern int early_make_pgtable (unsigned long address); +extern void early_setup_idt (void );#ifdef CONFIG_X86_INTEL_MID extern void x86_intel_mid_early_setup (void );
arch/x86/kernel/head64.c +39 -0
@@ -36,6 +36,8 @@#include <asm/microcode.h> #include <asm/kasan.h> #include <asm/fixmap.h> +#include <asm/realmode.h> +#include <asm/desc.h> /* * Manage page tables very early on.
@@ -508,6 +510,41 @@ void __init x86_64_start_reservations(char *real_mode_data)start_kernel (); } +/* +* Data structures and code used for IDT setup in head_64.S. The bringup-IDT is +* used until the idt_table takes over. On the boot CPU this happens in +* x86_64_start_kernel(), on secondary CPUs in start_secondary(). In both cases +* this happens in the functions called from head_64.S. +* +* The idt_table can't be used that early because all the code modifying it is +* in idt.c and can be instrumented by tracing or KASAN, which both don't work +* during early CPU bringup. Also the idt_table has the runtime vectors +* configured which require certain CPU state to be setup already (like TSS), +* which also hasn't happened yet in early CPU bringup. +*/ +static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data; + +static struct desc_ptr bringup_idt_descr = { + .size = (NUM_EXCEPTION_VECTORS *sizeof (gate_desc)) -1 , + .address =0 ,/* Set at runtime */ +}; + +/* This runs while still in the direct mapping */ +static void startup_64_load_idt (unsigned long physbase) +{ +struct desc_ptr *desc =fixup_pointer (&bringup_idt_descr, physbase); + + desc->address = (unsigned long )fixup_pointer (bringup_idt_table, physbase); +native_load_idt (desc); +} + +/* This is used when running on kernel addresses */ +void early_setup_idt (void ) +{ + bringup_idt_descr.address = (unsigned long )bringup_idt_table; +native_load_idt (&bringup_idt_descr); +} +/* * Setup boot CPU state needed before kernel switches to virtual addresses. */
@@ -521,4 +558,6 @@ void __head startup_64_setup_env(unsigned long physbase) asmvolatile ("movl %%eax, %%ds\n" "movl %%eax, %%ss\n" "movl %%eax, %%es\n" : :"a" (__KERNEL_DS) :"memory" ); + +startup_64_load_idt (physbase); }
arch/x86/kernel/head_64.S +5 -0
@@ -198,6 +198,11 @@ SYM_CODE_START(secondary_startup_64) */ movq initial_stack(%rip), %rsp + /* Setup and Load IDT */ + pushq %rsi + call early_setup_idt + popq %rsi + /* Check if nx is implemented */ movl $0x80000001, %eax cpuid
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/head/64: Install a CPU bringup IDT
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: f5963ba7a45fc6ff298a34976064354be437e1d8 Gitweb: https://git.kernel.org/tip/f5963ba7a45fc6ff298a34976064354be437e1d8 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:34 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 22:18:38 +02:00
x86/head/64: Install a CPU bringup IDT
Add a separate bringup IDT for the CPU bringup code that will be used until the kernel switches to the idt_table. There are two reasons for a separate IDT:
1) When the idt_table is set up and the secondary CPUs are booted, it contains entries (e.g. IST entries) which require certain CPU state to be set up. This includes a working TSS (for IST), MSR_GS_BASE (for stack protector) or CR4.FSGSBASE (for paranoid_entry) path. By using a dedicated IDT for early boot this state need not to be set up early.
2) The idt_table is static to idt.c, so any function using/modifying must be in idt.c too. That means that all compiler driven instrumentation like tracing or KASAN is also active in this code. But during early CPU bringup the environment is not set up for this instrumentation to work correctly.
To avoid all of these hassles and make early exception handling robust, use a dedicated bringup IDT.
The IDT is loaded two times, first on the boot CPU while the kernel is still running on direct mapped addresses, and again later after the switch to kernel addresses has happened. The second IDT load happens on the boot and secondary CPUs.
--- arch/x86/include/asm/setup.h | 1 +- arch/x86/kernel/head64.c | 39 +++++++++++++++++++++++++++++++++++- arch/x86/kernel/head_64.S | 5 ++++- 3 files changed, 45 insertions(+)
arch/x86/include/asm/setup.h +1 -0
@@ -50,6 +50,7 @@ extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bpextern unsigned long __startup_secondary_64 (void );extern void startup_64_setup_env (unsigned long physbase);extern int early_make_pgtable (unsigned long address); +extern void early_setup_idt (void );#ifdef CONFIG_X86_INTEL_MID extern void x86_intel_mid_early_setup (void );
arch/x86/kernel/head64.c +39 -0
@@ -36,6 +36,8 @@#include <asm/microcode.h> #include <asm/kasan.h> #include <asm/fixmap.h> +#include <asm/realmode.h> +#include <asm/desc.h> /* * Manage page tables very early on.
@@ -509,6 +511,41 @@ void __init x86_64_start_reservations(char *real_mode_data) }/* +* Data structures and code used for IDT setup in head_64.S. The bringup-IDT is +* used until the idt_table takes over. On the boot CPU this happens in +* x86_64_start_kernel(), on secondary CPUs in start_secondary(). In both cases +* this happens in the functions called from head_64.S. +* +* The idt_table can't be used that early because all the code modifying it is +* in idt.c and can be instrumented by tracing or KASAN, which both don't work +* during early CPU bringup. Also the idt_table has the runtime vectors +* configured which require certain CPU state to be setup already (like TSS), +* which also hasn't happened yet in early CPU bringup. +*/ +static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data; + +static struct desc_ptr bringup_idt_descr = { + .size = (NUM_EXCEPTION_VECTORS *sizeof (gate_desc)) -1 , + .address =0 ,/* Set at runtime */ +}; + +/* This runs while still in the direct mapping */ +static void startup_64_load_idt (unsigned long physbase) +{ +struct desc_ptr *desc =fixup_pointer (&bringup_idt_descr, physbase); + + desc->address = (unsigned long )fixup_pointer (bringup_idt_table, physbase); +native_load_idt (desc); +} + +/* This is used when running on kernel addresses */ +void early_setup_idt (void ) +{ + bringup_idt_descr.address = (unsigned long )bringup_idt_table; +native_load_idt (&bringup_idt_descr); +} + +/* * Setup boot CPU state needed before kernel switches to virtual addresses. */ void __head startup_64_setup_env(unsigned long physbase)
@@ -521,4 +558,6 @@ void __head startup_64_setup_env(unsigned long physbase) asmvolatile ("movl %%eax, %%ds\n" "movl %%eax, %%ss\n" "movl %%eax, %%es\n" : :"a" (__KERNEL_DS) :"memory" ); + +startup_64_load_idt (physbase); }
arch/x86/kernel/head_64.S +5 -0
@@ -198,6 +198,11 @@ SYM_CODE_START(secondary_startup_64) */ movq initial_stack(%rip), %rsp + /* Setup and Load IDT */ + pushq %rsi + call early_setup_idt + popq %rsi + /* Check if nx is implemented */ movl $0x80000001, %eax cpuid
PATCH v7 34/72 x86/idt: Move two function from k/idt.c to i/a/desc.h
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Move these two functions from kernel/idt.c to include/asm/desc.h:
* init_idt_data()
* idt_init_desc()
These functions are needed to setup IDT entries very early and need to be called from head64.c. To be usable this early these functions need to be compiled without instrumentation and the stack-protector feature. These features need to be kept enabled for kernel/idt.c, so head64.c must use its own versions.
--- arch/x86/include/asm/desc.h | 27 +++++++++++++++++++++++++ arch/x86/include/asm/desc_defs.h | 7 +++++++ arch/x86/kernel/idt.c | 34 -------------------------------- 3 files changed, 34 insertions(+), 34 deletions(-)
arch/x86/include/asm/desc.h +27 -0
@@ -383,6 +383,33 @@ static inline void set_desc_limit(struct desc_struct *desc, unsigned long limit)void alloc_intr_gate (unsigned int n,const void *addr); +static inline void init_idt_data (struct idt_data *data,unsigned int n, +const void *addr) +{ +BUG_ON (n >0xFF ); + +memset (data,0 ,sizeof (*data)); + data->vector = n; + data->addr = addr; + data->segment = __KERNEL_CS; + data->bits.type =GATE_INTERRUPT ; + data->bits.p =1 ; +} + +static inline void idt_init_desc (gate_desc *gate,const struct idt_data *d) +{ +unsigned long addr = (unsigned long ) d->addr; + + gate->offset_low = (u16 ) addr; + gate->segment = (u16 ) d->segment; + gate->bits = d->bits; + gate->offset_middle = (u16) (addr >>16 ); +#ifdef CONFIG_X86_64 + gate->offset_high = (u32) (addr >>32 ); + gate->reserved =0 ; +#endif +} +extern unsigned long system_vectors[];extern void load_current_idt (void );
arch/x86/include/asm/desc_defs.h +7 -0
@@ -74,6 +74,13 @@ struct idt_bits { p :1 ; } __attribute__((packed)); +struct idt_data { +unsigned int vector; +unsigned int segment; +struct idt_bits bits; +const void *addr; +}; +struct gate_struct {u16 offset_low;u16 segment;
arch/x86/kernel/idt.c +0 -34
@@ -11,13 +11,6 @@#include <asm/desc.h> #include <asm/hw_irq.h> -struct idt_data { -unsigned int vector; -unsigned int segment; -struct idt_bits bits; -const void *addr; -}; -#define DPL0 0x0#define DPL3 0x3
@@ -178,20 +171,6 @@ bool idt_is_f00f_address(unsigned long address) }#endif -static inline void idt_init_desc (gate_desc *gate,const struct idt_data *d) -{ -unsigned long addr = (unsigned long ) d->addr; - - gate->offset_low = (u16 ) addr; - gate->segment = (u16 ) d->segment; - gate->bits = d->bits; - gate->offset_middle = (u16) (addr >>16 ); -#ifdef CONFIG_X86_64 - gate->offset_high = (u32) (addr >>32 ); - gate->reserved =0 ; -#endif -} -static __init void idt_setup_from_table (gate_desc *idt,const struct idt_data *t,int size,bool sys) {
@@ -205,19 +184,6 @@ idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sy } } -static void init_idt_data (struct idt_data *data,unsigned int n, -const void *addr) -{ -BUG_ON (n >0xFF ); - -memset (data,0 ,sizeof (*data)); - data->vector = n; - data->addr = addr; - data->segment = __KERNEL_CS; - data->bits.type =GATE_INTERRUPT ; - data->bits.p =1 ; -} -static __init void set_intr_gate (unsigned int n,const void *addr) {struct idt_data data;
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/idt: Make IDT init functions static inlines
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 097ee5b778b8970e1c2ed3ca1631b297d90acd61 Gitweb: https://git.kernel.org/tip/097ee5b778b8970e1c2ed3ca1631b297d90acd61 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:35 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 22:44:43 +02:00
x86/idt: Make IDT init functions static inlines
Move these two functions from kernel/idt.c to include/asm/desc.h:
* init_idt_data()
* idt_init_desc()
These functions are needed to setup IDT entries very early and need to be called from head64.c. To be usable this early, these functions need to be compiled without instrumentation and the stack-protector feature.
These features need to be kept enabled for kernel/idt.c, so head64.c must use its own versions.
[ bp: Take Kees' suggested patch title and add his Rev-by. ]
--- arch/x86/include/asm/desc.h | 27 +++++++++++++++++++++++++- arch/x86/include/asm/desc_defs.h | 7 ++++++- arch/x86/kernel/idt.c | 34 +------------------------------- 3 files changed, 34 insertions(+), 34 deletions(-)
arch/x86/include/asm/desc.h +27 -0
@@ -383,6 +383,33 @@ static inline void set_desc_limit(struct desc_struct *desc, unsigned long limit)void alloc_intr_gate (unsigned int n,const void *addr); +static inline void init_idt_data (struct idt_data *data,unsigned int n, +const void *addr) +{ +BUG_ON (n >0xFF ); + +memset (data,0 ,sizeof (*data)); + data->vector = n; + data->addr = addr; + data->segment = __KERNEL_CS; + data->bits.type =GATE_INTERRUPT ; + data->bits.p =1 ; +} + +static inline void idt_init_desc (gate_desc *gate,const struct idt_data *d) +{ +unsigned long addr = (unsigned long ) d->addr; + + gate->offset_low = (u16 ) addr; + gate->segment = (u16 ) d->segment; + gate->bits = d->bits; + gate->offset_middle = (u16) (addr >>16 ); +#ifdef CONFIG_X86_64 + gate->offset_high = (u32) (addr >>32 ); + gate->reserved =0 ; +#endif +} +extern unsigned long system_vectors[];extern void load_current_idt (void );
arch/x86/include/asm/desc_defs.h +7 -0
@@ -74,6 +74,13 @@ struct idt_bits { p :1 ; } __attribute__((packed)); +struct idt_data { +unsigned int vector; +unsigned int segment; +struct idt_bits bits; +const void *addr; +}; +struct gate_struct {u16 offset_low;u16 segment;
arch/x86/kernel/idt.c +0 -34
@@ -11,13 +11,6 @@#include <asm/desc.h> #include <asm/hw_irq.h> -struct idt_data { -unsigned int vector; -unsigned int segment; -struct idt_bits bits; -const void *addr; -}; -#define DPL0 0x0#define DPL3 0x3
@@ -178,20 +171,6 @@ bool idt_is_f00f_address(unsigned long address) }#endif -static inline void idt_init_desc (gate_desc *gate,const struct idt_data *d) -{ -unsigned long addr = (unsigned long ) d->addr; - - gate->offset_low = (u16 ) addr; - gate->segment = (u16 ) d->segment; - gate->bits = d->bits; - gate->offset_middle = (u16) (addr >>16 ); -#ifdef CONFIG_X86_64 - gate->offset_high = (u32) (addr >>32 ); - gate->reserved =0 ; -#endif -} -static __init void idt_setup_from_table (gate_desc *idt,const struct idt_data *t,int size,bool sys) {
@@ -205,19 +184,6 @@ idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sy } } -static void init_idt_data (struct idt_data *data,unsigned int n, -const void *addr) -{ -BUG_ON (n >0xFF ); - -memset (data,0 ,sizeof (*data)); - data->vector = n; - data->addr = addr; - data->segment = __KERNEL_CS; - data->bits.type =GATE_INTERRUPT ; - data->bits.p =1 ; -} -static __init void set_intr_gate (unsigned int n,const void *addr) {struct idt_data data;
PATCH v7 35/72 x86/head/64: Move early exception dispatch to C code
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Move the assembly coded dispatch between page-faults and all other exceptions to C code to make it easier to maintain and extend.
Also change the return-type of early_make_pgtable() to bool and make it static.
--- arch/x86/include/asm/pgtable.h | 2 +- arch/x86/include/asm/setup.h | 4 +++- arch/x86/kernel/head64.c | 19 +++++++++++++++---- arch/x86/kernel/head_64.S | 11 +---------- 4 files changed, 20 insertions(+), 16 deletions(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -28,7 +28,7 @@#include <asm-generic/pgtable_uffd.h> extern pgd_t early_top_pgt[PTRS_PER_PGD ]; -int __init __early_make_pgtable (unsigned long address,pmdval_t pmd); +bool __init __early_make_pgtable (unsigned long address,pmdval_t pmd);void ptdump_walk_pgd_level (struct seq_file *m,struct mm_struct *mm); void ptdump_walk_pgd_level_debugfs(struct seq_file *m, struct mm_struct *mm,
arch/x86/include/asm/setup.h +3 -1
@@ -39,6 +39,8 @@ void vsmp_init(void);static inline void vsmp_init (void ) { }#endif +struct pt_regs ; +void setup_bios_corruption_check (void );void early_platform_quirks (void );
@@ -49,8 +51,8 @@ extern void i386_reserve_resources(void);extern unsigned long __startup_64 (unsigned long physaddr,struct boot_params *bp);extern unsigned long __startup_secondary_64 (void );extern void startup_64_setup_env (unsigned long physbase); -extern int early_make_pgtable (unsigned long address);extern void early_setup_idt (void ); +extern void __init do_early_exception (struct pt_regs *regs,int trapnr);#ifdef CONFIG_X86_INTEL_MID extern void x86_intel_mid_early_setup (void );
arch/x86/kernel/head64.c +15 -4
@@ -38,6 +38,8 @@#include <asm/fixmap.h> #include <asm/realmode.h> #include <asm/desc.h> +#include <asm/extable.h> +#include <asm/trapnr.h> /* * Manage page tables very early on.
@@ -317,7 +319,7 @@ static void __init reset_early_page_tables(void) }/* Create a new PMD entry */ -int __init __early_make_pgtable (unsigned long address,pmdval_t pmd) +bool __init __early_make_pgtable (unsigned long address,pmdval_t pmd) {unsigned long physaddr = address - __PAGE_OFFSET;pgdval_t pgd, *pgd_p;
@@ -327,7 +329,7 @@ int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)/* Invalid address or early pgt is done ? */ if (physaddr >=MAXMEM ||read_cr3_pa () !=__pa_nodebug (early_top_pgt)) -return -1 ; +return false; again: pgd_p = &early_top_pgt[pgd_index (address)].pgd;
@@ -384,10 +386,10 @@ int __init __early_make_pgtable(unsigned long address, pmdval_t pmd) } pmd_p[pmd_index (address)] = pmd; -return 0 ; +return true; } -int __init early_make_pgtable (unsigned long address) +static bool __init early_make_pgtable (unsigned long address) {unsigned long physaddr = address - __PAGE_OFFSET;pmdval_t pmd;
@@ -397,6 +399,15 @@ int __init early_make_pgtable(unsigned long address)return __early_make_pgtable (address, pmd); } +void __init do_early_exception (struct pt_regs *regs,int trapnr) +{ +if (trapnr ==X86_TRAP_PF && +early_make_pgtable (native_read_cr2 ())) +return ; + +early_fixup_exception (regs, trapnr); +} +/* Don't add a printk in there. printk relies on the PDA which is not initialized yet. */ static void __init clear_bss (void )
arch/x86/kernel/head_64.S +1 -10
@@ -341,18 +341,9 @@ SYM_CODE_START_LOCAL(early_idt_handler_common) pushq %r15 /* pt_regs->r15 */ UNWIND_HINT_REGS - cmpq $14,%rsi /* Page fault? */ - jnz 10f - GET_CR2_INTO(%rdi) /* can clobber %rax if pv */ - call early_make_pgtable - andl %eax,%eax - jz 20f /* All good */ - -10: movq %rsp,%rdi /* RDI = pt_regs; RSI is already trapnr */ - call early_fixup_exception + call do_early_exception -20: decl early_recursion_flag(%rip) jmp restore_regs_and_return_to_kernel SYM_CODE_END(early_idt_handler_common)
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/head/64: Move early exception dispatch to C code
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 4b47cdbda6f1ad73b08dc7d497bac12b8f26ae0d Gitweb: https://git.kernel.org/tip/4b47cdbda6f1ad73b08dc7d497bac12b8f26ae0d Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:36 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 22:49:18 +02:00
x86/head/64: Move early exception dispatch to C code
Move the assembly coded dispatch between page-faults and all other exceptions to C code to make it easier to maintain and extend.
Also change the return-type of early_make_pgtable() to bool and make it static.
--- arch/x86/include/asm/pgtable.h | 2 +- arch/x86/include/asm/setup.h | 4 +++- arch/x86/kernel/head64.c | 19 +++++++++++++++---- arch/x86/kernel/head_64.S | 11 +---------- 4 files changed, 20 insertions(+), 16 deletions(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -28,7 +28,7 @@#include <asm-generic/pgtable_uffd.h> extern pgd_t early_top_pgt[PTRS_PER_PGD ]; -int __init __early_make_pgtable (unsigned long address,pmdval_t pmd); +bool __init __early_make_pgtable (unsigned long address,pmdval_t pmd);void ptdump_walk_pgd_level (struct seq_file *m,struct mm_struct *mm); void ptdump_walk_pgd_level_debugfs(struct seq_file *m, struct mm_struct *mm,
arch/x86/include/asm/setup.h +3 -1
@@ -39,6 +39,8 @@ void vsmp_init(void);static inline void vsmp_init (void ) { }#endif +struct pt_regs ; +void setup_bios_corruption_check (void );void early_platform_quirks (void );
@@ -49,8 +51,8 @@ extern void i386_reserve_resources(void);extern unsigned long __startup_64 (unsigned long physaddr,struct boot_params *bp);extern unsigned long __startup_secondary_64 (void );extern void startup_64_setup_env (unsigned long physbase); -extern int early_make_pgtable (unsigned long address);extern void early_setup_idt (void ); +extern void __init do_early_exception (struct pt_regs *regs,int trapnr);#ifdef CONFIG_X86_INTEL_MID extern void x86_intel_mid_early_setup (void );
arch/x86/kernel/head64.c +15 -4
@@ -38,6 +38,8 @@#include <asm/fixmap.h> #include <asm/realmode.h> #include <asm/desc.h> +#include <asm/extable.h> +#include <asm/trapnr.h> /* * Manage page tables very early on.
@@ -317,7 +319,7 @@ static void __init reset_early_page_tables(void) }/* Create a new PMD entry */ -int __init __early_make_pgtable (unsigned long address,pmdval_t pmd) +bool __init __early_make_pgtable (unsigned long address,pmdval_t pmd) {unsigned long physaddr = address - __PAGE_OFFSET;pgdval_t pgd, *pgd_p;
@@ -327,7 +329,7 @@ int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)/* Invalid address or early pgt is done ? */ if (physaddr >=MAXMEM ||read_cr3_pa () !=__pa_nodebug (early_top_pgt)) -return -1 ; +return false; again: pgd_p = &early_top_pgt[pgd_index (address)].pgd;
@@ -384,10 +386,10 @@ again: } pmd_p[pmd_index (address)] = pmd; -return 0 ; +return true; } -int __init early_make_pgtable (unsigned long address) +static bool __init early_make_pgtable (unsigned long address) {unsigned long physaddr = address - __PAGE_OFFSET;pmdval_t pmd;
@@ -397,6 +399,15 @@ int __init early_make_pgtable(unsigned long address)return __early_make_pgtable (address, pmd); } +void __init do_early_exception (struct pt_regs *regs,int trapnr) +{ +if (trapnr ==X86_TRAP_PF && +early_make_pgtable (native_read_cr2 ())) +return ; + +early_fixup_exception (regs, trapnr); +} +/* Don't add a printk in there. printk relies on the PDA which is not initialized yet. */ static void __init clear_bss (void )
arch/x86/kernel/head_64.S +1 -10
@@ -341,18 +341,9 @@ SYM_CODE_START_LOCAL(early_idt_handler_common) pushq %r15 /* pt_regs->r15 */ UNWIND_HINT_REGS - cmpq $14,%rsi /* Page fault? */ - jnz 10f - GET_CR2_INTO(%rdi) /* can clobber %rax if pv */ - call early_make_pgtable - andl %eax,%eax - jz 20f /* All good */ - -10: movq %rsp,%rdi /* RDI = pt_regs; RSI is already trapnr */ - call early_fixup_exception + call do_early_exception -20: decl early_recursion_flag(%rip) jmp restore_regs_and_return_to_kernel SYM_CODE_END(early_idt_handler_common)
PATCH v7 36/72 x86/sev-es: Add SEV-ES Feature Detection
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Add the sev_es_active function for checking whether SEV-ES is enabled. Also cache the value of MSR_AMD64_SEV at boot to speed up the feature checking in the running code.
--- arch/x86/include/asm/mem_encrypt.h | 3 +++ arch/x86/include/asm/msr-index.h | 2 ++ arch/x86/mm/mem_encrypt.c | 9 ++++++++- arch/x86/mm/mem_encrypt_identity.c | 3 +++ 4 files changed, 16 insertions(+), 1 deletion(-)
arch/x86/include/asm/mem_encrypt.h +3 -0
@@ -19,6 +19,7 @@#ifdef CONFIG_AMD_MEM_ENCRYPT extern u64 sme_me_mask; +extern u64 sev_status;extern bool sev_enabled; void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr,
@@ -50,6 +51,7 @@ void __init mem_encrypt_init(void);bool sme_active (void );bool sev_active (void ); +bool sev_es_active (void );#define __bss_decrypted __attribute__((__section__(".bss..decrypted")))
@@ -72,6 +74,7 @@ static inline void __init sme_enable(struct boot_params *bp) { }static inline bool sme_active (void ) {return false; }static inline bool sev_active (void ) {return false; } +static inline bool sev_es_active (void ) {return false; }static inline int __init early_set_memory_decrypted (unsigned long vaddr,unsigned long size) {return 0 ; }
arch/x86/include/asm/msr-index.h +2 -0
@@ -469,7 +469,9 @@#define MSR_AMD64_SEV_ES_GHCB 0xc0010130#define MSR_AMD64_SEV 0xc0010131#define MSR_AMD64_SEV_ENABLED_BIT 0 +#define MSR_AMD64_SEV_ES_ENABLED_BIT 1#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT) +#define MSR_AMD64_SEV_ES_ENABLED BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)#define MSR_AMD64_VIRT_SPEC_CTRL 0xc001011f
arch/x86/mm/mem_encrypt.c +8 -1
@@ -38,6 +38,7 @@* section is later cleared. */ u64 sme_me_mask __section(.data) = 0; +u64 sev_status __section(.data) = 0;EXPORT_SYMBOL (sme_me_mask);DEFINE_STATIC_KEY_FALSE (sev_enable_key);EXPORT_SYMBOL_GPL (sev_enable_key);
@@ -347,7 +348,13 @@ bool sme_active(void)bool sev_active (void ) { -return sme_me_mask && sev_enabled; +return !!(sev_status &MSR_AMD64_SEV_ENABLED ); +} + +/* Needs to be called from non-instrumentable code */ +bool noinstr sev_es_active(void) +{ +return sev_status &MSR_AMD64_SEV_ES_ENABLED ; }/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
arch/x86/mm/mem_encrypt_identity.c +3 -0
@@ -540,6 +540,9 @@ void __init sme_enable(struct boot_params *bp)if (!(msr &MSR_AMD64_SEV_ENABLED ))return ; +/* Save SEV_STATUS to avoid reading MSR again */ + sev_status = msr; +/* SEV state cannot be controlled by a command line option */ sme_me_mask = me_mask; sev_enabled = true;
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Borislav Petkov
maintainerRe: [PATCH v7 36/72] x86/sev-es: Add SEV-ES Feature Detection
arch/x86/mm/mem_encrypt.c bool sme_active(void) · line 351
Quoted 6 lines · Joerg Roedel
arch/x86/mm/mem_encrypt.c +1
-1@@ -347,7 +348,13 @@ bool sme_active(void)bool sev_active (void ) { -return sme_me_mask && sev_enabled; +return !!(sev_status &MSR_AMD64_SEV_ENABLED );
Dropped those "!!" here too while applying.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette[tip: x86/seves] x86/sev-es: Add SEV-ES Feature Detection
In reply to Joerg RoedelThe following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: b57de6cd16395be1ebdaa9b489ffbf462bb585c4 Gitweb: https://git.kernel.org/tip/b57de6cd16395be1ebdaa9b489ffbf462bb585c4 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:37 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 07 Sep 2020 23:00:20 +02:00
x86/sev-es: Add SEV-ES Feature Detection
Add a sev_es_active() function for checking whether SEV-ES is enabled. Also cache the value of MSR_AMD64_SEV at boot to speed up the feature checking in the running code.
[ bp: Remove "!!" in sev_active() too. ]
--- arch/x86/include/asm/mem_encrypt.h | 3 +++ arch/x86/include/asm/msr-index.h | 2 ++ arch/x86/mm/mem_encrypt.c | 9 ++++++++- arch/x86/mm/mem_encrypt_identity.c | 3 +++ 4 files changed, 16 insertions(+), 1 deletion(-)
arch/x86/include/asm/mem_encrypt.h +3 -0
@@ -19,6 +19,7 @@#ifdef CONFIG_AMD_MEM_ENCRYPT extern u64 sme_me_mask; +extern u64 sev_status;extern bool sev_enabled; void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr,
@@ -50,6 +51,7 @@ void __init mem_encrypt_init(void);bool sme_active (void );bool sev_active (void ); +bool sev_es_active (void );#define __bss_decrypted __attribute__((__section__(".bss..decrypted")))
@@ -72,6 +74,7 @@ static inline void __init sme_enable(struct boot_params *bp) { }static inline bool sme_active (void ) {return false; }static inline bool sev_active (void ) {return false; } +static inline bool sev_es_active (void ) {return false; }static inline int __init early_set_memory_decrypted (unsigned long vaddr,unsigned long size) {return 0 ; }
arch/x86/include/asm/msr-index.h +2 -0
@@ -469,7 +469,9 @@#define MSR_AMD64_SEV_ES_GHCB 0xc0010130#define MSR_AMD64_SEV 0xc0010131#define MSR_AMD64_SEV_ENABLED_BIT 0 +#define MSR_AMD64_SEV_ES_ENABLED_BIT 1#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT) +#define MSR_AMD64_SEV_ES_ENABLED BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)#define MSR_AMD64_VIRT_SPEC_CTRL 0xc001011f
arch/x86/mm/mem_encrypt.c +8 -1
@@ -38,6 +38,7 @@* section is later cleared. */ u64 sme_me_mask __section(.data) = 0; +u64 sev_status __section(.data) = 0;EXPORT_SYMBOL (sme_me_mask);DEFINE_STATIC_KEY_FALSE (sev_enable_key);EXPORT_SYMBOL_GPL (sev_enable_key);
@@ -347,7 +348,13 @@ bool sme_active(void)bool sev_active (void ) { -return sme_me_mask && sev_enabled; +return sev_status &MSR_AMD64_SEV_ENABLED ; +} + +/* Needs to be called from non-instrumentable code */ +bool noinstr sev_es_active(void) +{ +return sev_status &MSR_AMD64_SEV_ES_ENABLED ; }/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
arch/x86/mm/mem_encrypt_identity.c +3 -0
@@ -540,6 +540,9 @@ void __init sme_enable(struct boot_params *bp)if (!(msr &MSR_AMD64_SEV_ENABLED ))return ; +/* Save SEV_STATUS to avoid reading MSR again */ + sev_status = msr; +/* SEV state cannot be controlled by a command line option */ sme_me_mask = me_mask; sev_enabled = true;
PATCH v7 37/72 x86/sev-es: Print SEV-ES info into kernel log
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Refactor the message printed to the kernel log which indicates whether SEV or SME is active to print a list of enabled encryption features. This will scale better in the future when more memory encryption features might be added. Also add SEV-ES to the list of features.
--- arch/x86/mm/mem_encrypt.c | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-)
arch/x86/mm/mem_encrypt.c +26 -3
@@ -407,6 +407,31 @@ void __init mem_encrypt_free_decrypted_mem(void)free_init_pages ("unused decrypted" , vaddr, vaddr_end); } +static void print_mem_encrypt_feature_info (void ) +{ +pr_info ("AMD Memory Encryption Features active:" ); + +/* Secure Memory Encryption */ +if (sme_active ()) { +/* +* SME is mutually exclusive with any of the SEV +* features below. +*/ +pr_cont (" SME\n" ); +return ; + } + +/* Secure Encrypted Virtualization */ +if (sev_active ()) +pr_cont (" SEV" ); + +/* Encrypted Register State */ +if (sev_es_active ()) +pr_cont (" SEV-ES" ); + +pr_cont ("\n" ); +} +/* Architecture __weak replacement functions */ void __init mem_encrypt_init (void ) {
@@ -422,8 +447,6 @@ void __init mem_encrypt_init(void)if (sev_active ())static_branch_enable (&sev_enable_key); -pr_info ("AMD %s active\n" , -sev_active () ?"Secure Encrypted Virtualization (SEV)" - :"Secure Memory Encryption (SME)" ); +print_mem_encrypt_feature_info (); }
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Print SEV-ES info into the kernel log
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: c685eb0c12b4d4816d22ee734e91f4005b152fcd Gitweb: https://git.kernel.org/tip/c685eb0c12b4d4816d22ee734e91f4005b152fcd Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:38 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Tue, 08 Sep 2020 00:38:01 +02:00
x86/sev-es: Print SEV-ES info into the kernel log
Refactor the message printed to the kernel log which indicates whether SEV or SME, etc is active. This will scale better in the future when more memory encryption features might be added. Also add SEV-ES to the list of features.
[ bp: Massage. ]
--- arch/x86/mm/mem_encrypt.c | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-)
arch/x86/mm/mem_encrypt.c +26 -3
@@ -407,6 +407,31 @@ void __init mem_encrypt_free_decrypted_mem(void)free_init_pages ("unused decrypted" , vaddr, vaddr_end); } +static void print_mem_encrypt_feature_info (void ) +{ +pr_info ("AMD Memory Encryption Features active:" ); + +/* Secure Memory Encryption */ +if (sme_active ()) { +/* +* SME is mutually exclusive with any of the SEV +* features below. +*/ +pr_cont (" SME\n" ); +return ; + } + +/* Secure Encrypted Virtualization */ +if (sev_active ()) +pr_cont (" SEV" ); + +/* Encrypted Register State */ +if (sev_es_active ()) +pr_cont (" SEV-ES" ); + +pr_cont ("\n" ); +} +/* Architecture __weak replacement functions */ void __init mem_encrypt_init (void ) {
@@ -422,8 +447,6 @@ void __init mem_encrypt_init(void)if (sev_active ())static_branch_enable (&sev_enable_key); -pr_info ("AMD %s active\n" , -sev_active () ?"Secure Encrypted Virtualization (SEV)" - :"Secure Memory Encryption (SME)" ); +print_mem_encrypt_feature_info (); }
PATCH v7 38/72 x86/sev-es: Compile early handler code into kernel image
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Setup sev-es.c and include the code from the pre-decompression stage to also build it into the image of the running kernel. Temporarily add __maybe_unused annotations to avoid build warnings until the functions get used.
--- arch/x86/kernel/Makefile | 1 + arch/x86/kernel/sev-es-shared.c | 21 ++-- arch/x86/kernel/sev-es.c | 163 ++++++++++++++++++++++++++++++++ 3 files changed, 175 insertions(+), 10 deletions(-) create mode 100644 arch/x86/kernel/sev-es.c
arch/x86/kernel/Makefile +1 -0
@@ -146,6 +146,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o +obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev-es.o ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/sev-es-shared.c +11 -10
@@ -9,7 +9,7 @@* and is included directly into both code-bases. */ -static void sev_es_terminate (unsigned int reason) +static void __maybe_unused sev_es_terminate (unsigned int reason) {u64 val =GHCB_SEV_TERMINATE ;
@@ -27,7 +27,7 @@ static void sev_es_terminate(unsigned int reason) asmvolatile ("hlt\n" : : :"memory" ); } -static bool sev_es_negotiate_protocol (void ) +static bool __maybe_unused sev_es_negotiate_protocol (void ) {u64 val;
@@ -46,7 +46,7 @@ static bool sev_es_negotiate_protocol(void)return true; } -static void vc_ghcb_invalidate (struct ghcb *ghcb) +static void __maybe_unused vc_ghcb_invalidate (struct ghcb *ghcb) {memset (ghcb->save.valid_bitmap,0 ,sizeof (ghcb->save.valid_bitmap)); }
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code) exit_code <= SVM_EXIT_LAST_EXCP); } -static enum es_result vc_init_em_ctxt (struct es_em_ctxt *ctxt, -struct pt_regs *regs, -unsigned long exit_code) +static enum es_result __maybe_unused vc_init_em_ctxt (struct es_em_ctxt *ctxt, +struct pt_regs *regs, +unsigned long exit_code) {enum es_result ret =ES_OK ;
@@ -73,7 +73,7 @@ static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,return ret; } -static void vc_finish_insn (struct es_em_ctxt *ctxt) +static void __maybe_unused vc_finish_insn (struct es_em_ctxt *ctxt) { ctxt->regs->ip += ctxt->insn.length; }
@@ -325,7 +325,8 @@ static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)return ES_OK ; } -static enum es_result vc_handle_ioio (struct ghcb *ghcb,struct es_em_ctxt *ctxt) +static enum es_result __maybe_unused +vc_handle_ioio (struct ghcb *ghcb,struct es_em_ctxt *ctxt) {struct pt_regs *regs = ctxt->regs;u64 exit_info_1, exit_info_2;
@@ -433,8 +434,8 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)return ret; } -static enum es_result vc_handle_cpuid (struct ghcb *ghcb, -struct es_em_ctxt *ctxt) +static enum es_result __maybe_unused vc_handle_cpuid (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) {struct pt_regs *regs = ctxt->regs;u32 cr4 =native_read_cr4 ();
arch/x86/kernel/sev-es.c new file +163 -0
163 lines not on this page · raw message
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Compile early handler code into kernel image
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: f980f9c31a923e9040dee0bc679a5f5b09e61f40 Gitweb: https://git.kernel.org/tip/f980f9c31a923e9040dee0bc679a5f5b09e61f40 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:39 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 10:44:46 +02:00
x86/sev-es: Compile early handler code into kernel image
Setup sev-es.c and include the code from the pre-decompression stage to also build it into the image of the running kernel. Temporarily add __maybe_unused annotations to avoid build warnings until the functions get used.
[ bp: Use the non-tracing rd/wrmsr variants because:
vmlinux.o: warning: objtool: __sev_es_nmi_complete()+0x11f: \
call to do_trace_write_msr() leaves .noinstr.text section
as __sev_es_nmi_complete() is noinstr due to being called from the NMI handler exc_nmi(). ]
--- arch/x86/kernel/Makefile | 1 +- arch/x86/kernel/sev-es-shared.c | 21 ++-- arch/x86/kernel/sev-es.c | 163 +++++++++++++++++++++++++++++++- 3 files changed, 175 insertions(+), 10 deletions(-) create mode 100644 arch/x86/kernel/sev-es.c
arch/x86/kernel/Makefile +1 -0
@@ -145,6 +145,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o +obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev-es.o ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/sev-es-shared.c +11 -10
@@ -9,7 +9,7 @@* and is included directly into both code-bases. */ -static void sev_es_terminate (unsigned int reason) +static void __maybe_unused sev_es_terminate (unsigned int reason) {u64 val =GHCB_SEV_TERMINATE ;
@@ -27,7 +27,7 @@ static void sev_es_terminate(unsigned int reason) asmvolatile ("hlt\n" : : :"memory" ); } -static bool sev_es_negotiate_protocol (void ) +static bool __maybe_unused sev_es_negotiate_protocol (void ) {u64 val;
@@ -46,7 +46,7 @@ static bool sev_es_negotiate_protocol(void)return true; } -static void vc_ghcb_invalidate (struct ghcb *ghcb) +static void __maybe_unused vc_ghcb_invalidate (struct ghcb *ghcb) {memset (ghcb->save.valid_bitmap,0 ,sizeof (ghcb->save.valid_bitmap)); }
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code) exit_code <= SVM_EXIT_LAST_EXCP); } -static enum es_result vc_init_em_ctxt (struct es_em_ctxt *ctxt, -struct pt_regs *regs, -unsigned long exit_code) +static enum es_result __maybe_unused vc_init_em_ctxt (struct es_em_ctxt *ctxt, +struct pt_regs *regs, +unsigned long exit_code) {enum es_result ret =ES_OK ;
@@ -73,7 +73,7 @@ static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,return ret; } -static void vc_finish_insn (struct es_em_ctxt *ctxt) +static void __maybe_unused vc_finish_insn (struct es_em_ctxt *ctxt) { ctxt->regs->ip += ctxt->insn.length; }
@@ -325,7 +325,8 @@ static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)return ES_OK ; } -static enum es_result vc_handle_ioio (struct ghcb *ghcb,struct es_em_ctxt *ctxt) +static enum es_result __maybe_unused +vc_handle_ioio (struct ghcb *ghcb,struct es_em_ctxt *ctxt) {struct pt_regs *regs = ctxt->regs;u64 exit_info_1, exit_info_2;
@@ -433,8 +434,8 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)return ret; } -static enum es_result vc_handle_cpuid (struct ghcb *ghcb, -struct es_em_ctxt *ctxt) +static enum es_result __maybe_unused vc_handle_cpuid (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) {struct pt_regs *regs = ctxt->regs;u32 cr4 =native_read_cr4 ();
arch/x86/kernel/sev-es.c new file +163 -0
163 lines not on this page · raw message
PATCH v7 39/72 x86/sev-es: Setup early #VC handler
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Setup an early handler for #VC exceptions. There is no GHCB mapped yet, so just re-use the vc_no_ghcb_handler. It can only handle CPUID exit-codes, but that should be enough to get the kernel through verify_cpu() and __startup_64() until it runs on virtual addresses.
--- arch/x86/include/asm/sev-es.h | 3 +++ arch/x86/kernel/head64.c | 20 ++++++++++++++++++++ arch/x86/kernel/head_64.S | 27 +++++++++++++++++++++++++++ 3 files changed, 50 insertions(+)
arch/x86/include/asm/sev-es.h +3 -0
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits) return (val & mask); } +/* Early IDT entry points for #VC handler */ +extern void vc_no_ghcb (void ); +#endif
arch/x86/kernel/head64.c +20 -0
@@ -40,6 +40,7 @@#include <asm/desc.h> #include <asm/extable.h> #include <asm/trapnr.h> +#include <asm/sev-es.h> /* * Manage page tables very early on.
@@ -540,10 +541,29 @@ static struct desc_ptr bringup_idt_descr = { .address = 0, /* Set at runtime */ }; +#ifdef CONFIG_AMD_MEM_ENCRYPT +static void set_bringup_idt_handler (int n,void *handler) +{ +struct idt_data data; +gate_desc desc; + +init_idt_data (&data, n, handler); +idt_init_desc (&desc, &data); +native_write_idt_entry (bringup_idt_table, n, &desc); +} +#endif +/* This runs while still in the direct mapping */ static void startup_64_load_idt (unsigned long physbase) {struct desc_ptr *desc =fixup_pointer (&bringup_idt_descr, physbase); +void __maybe_unused *handler; + +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* VMM Communication Exception */ + handler =fixup_pointer (vc_no_ghcb, physbase); +set_bringup_idt_handler (X86_TRAP_VC , handler); +#endif desc->address = (unsigned long )fixup_pointer (bringup_idt_table, physbase);native_load_idt (desc);
arch/x86/kernel/head_64.S +27 -0
@@ -348,6 +348,33 @@ SYM_CODE_START_LOCAL(early_idt_handler_common) jmp restore_regs_and_return_to_kernel SYM_CODE_END(early_idt_handler_common) +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * VC Exception handler used during very early boot. The + * early_idt_handler_array can't be used because it returns via the + * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early. + */ +SYM_CODE_START_NOALIGN(vc_no_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + call do_vc_no_ghcb + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pure iret required here - don't use INTERRUPT_RETURN */ + iretq +SYM_CODE_END(vc_no_ghcb) +#endif #define SYM_DATA_START_PAGE_ALIGNED(name) \ SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v7 39/72 x86/sev-es: Setup early #VC handler
From: Joerg Roedel <[email protected]>
Setup an early handler for #VC exceptions. There is no GHCB mapped yet, so just re-use the vc_no_ghcb_handler. It can only handle CPUID exit-codes, but that should be enough to get the kernel through verify_cpu() and __startup_64() until it runs on virtual addresses.
--- arch/x86/include/asm/sev-es.h | 3 +++ arch/x86/kernel/head64.c | 21 +++++++++++++++++++++ arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++++++++ 3 files changed, 54 insertions(+)
arch/x86/include/asm/sev-es.h +3 -0
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits) return (val & mask); } +/* Early IDT entry points for #VC handler */ +extern void vc_no_ghcb (void ); +#endif
arch/x86/kernel/head64.c +21 -0
@@ -40,6 +40,7 @@#include <asm/desc.h> #include <asm/extable.h> #include <asm/trapnr.h> +#include <asm/sev-es.h> /* * Manage page tables very early on.
@@ -540,11 +541,31 @@ static struct desc_ptr bringup_idt_descr = { .address = 0, /* Set at runtime */ }; +static void set_bringup_idt_handler (int n,void *handler) +{ +#ifdef CONFIG_AMD_MEM_ENCRYPT +struct idt_data data; +gate_desc desc; + +init_idt_data (&data, n, handler); +idt_init_desc (&desc, &data); +native_write_idt_entry (bringup_idt_table, n, &desc); +#endif +} +/* This runs while still in the direct mapping */ static void startup_64_load_idt (unsigned long physbase) {struct desc_ptr *desc =fixup_pointer (&bringup_idt_descr, physbase); +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) { +void *handler; + +/* VMM Communication Exception */ + handler =fixup_pointer (vc_no_ghcb, physbase); +set_bringup_idt_handler (X86_TRAP_VC , handler); + } + desc->address = (unsigned long )fixup_pointer (bringup_idt_table, physbase);native_load_idt (desc); }
arch/x86/kernel/head_64.S +30 -0
@@ -348,6 +348,36 @@ SYM_CODE_START_LOCAL(early_idt_handler_common) jmp restore_regs_and_return_to_kernel SYM_CODE_END(early_idt_handler_common) +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * VC Exception handler used during very early boot. The + * early_idt_handler_array can't be used because it returns via the + * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early. + * + * This handler will end up in the .init.text section and not be + * available to boot secondary CPUs. + */ +SYM_CODE_START_NOALIGN(vc_no_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + call do_vc_no_ghcb + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pure iret required here - don't use INTERRUPT_RETURN */ + iretq +SYM_CODE_END(vc_no_ghcb) +#endif #define SYM_DATA_START_PAGE_ALIGNED(name) \ SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
Signature · 1 line
2.28.0
PATCH 39/74 x86/sev-es: Setup early #VC handler
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Setup an early handler for #VC exceptions. There is no GHCB mapped yet, so just re-use the vc_no_ghcb_handler. It can only handle CPUID exit-codes, but that should be enough to get the kernel through verify_cpu() and __startup_64() until it runs on virtual addresses.
--- arch/x86/include/asm/sev-es.h | 3 +++ arch/x86/kernel/head64.c | 25 ++++++++++++++++++++++++- arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++++++++ 3 files changed, 57 insertions(+), 1 deletion(-)
arch/x86/include/asm/sev-es.h +3 -0
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits) return (val & mask); } +/* Early IDT entry points for #VC handler */ +extern void vc_no_ghcb (void ); +#endif
arch/x86/kernel/head64.c +24 -1
@@ -40,6 +40,7 @@#include <asm/desc.h> #include <asm/extable.h> #include <asm/trapnr.h> +#include <asm/sev-es.h> /* * Manage page tables very early on.
@@ -540,12 +541,34 @@ static struct desc_ptr bringup_idt_descr = { .address = 0, /* Set at runtime */ }; +static void set_bringup_idt_handler (gate_desc *idt,int n,void *handler) +{ +#ifdef CONFIG_AMD_MEM_ENCRYPT +struct idt_data data; +gate_desc desc; + +init_idt_data (&data, n, handler); +idt_init_desc (&desc, &data); +native_write_idt_entry (idt, n, &desc); +#endif +} +/* This runs while still in the direct mapping */ static void startup_64_load_idt (unsigned long physbase) {struct desc_ptr *desc =fixup_pointer (&bringup_idt_descr, physbase); +gate_desc *idt =fixup_pointer (bringup_idt_table, physbase); + + +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) { +void *handler; + +/* VMM Communication Exception */ + handler =fixup_pointer (vc_no_ghcb, physbase); +set_bringup_idt_handler (idt,X86_TRAP_VC , handler); + } - desc->address = (unsigned long )fixup_pointer (bringup_idt_table, physbase); + desc->address = (unsigned long )idt;native_load_idt (desc); }
arch/x86/kernel/head_64.S +30 -0
@@ -348,6 +348,36 @@ SYM_CODE_START_LOCAL(early_idt_handler_common) jmp restore_regs_and_return_to_kernel SYM_CODE_END(early_idt_handler_common) +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * VC Exception handler used during very early boot. The + * early_idt_handler_array can't be used because it returns via the + * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early. + * + * This handler will end up in the .init.text section and not be + * available to boot secondary CPUs. + */ +SYM_CODE_START_NOALIGN(vc_no_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + call do_vc_no_ghcb + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pure iret required here - don't use INTERRUPT_RETURN */ + iretq +SYM_CODE_END(vc_no_ghcb) +#endif #define SYM_DATA_START_PAGE_ALIGNED(name) \ SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Setup an early #VC handler
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 74d8d9d531b4cc945a9f75aa2fc21d99ca5a9fe3 Gitweb: https://git.kernel.org/tip/74d8d9d531b4cc945a9f75aa2fc21d99ca5a9fe3 Author: Joerg Roedel <[email protected]> AuthorDate: Tue, 08 Sep 2020 14:35:17 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 10:45:24 +02:00
x86/sev-es: Setup an early #VC handler
Setup an early handler for #VC exceptions. There is no GHCB mapped yet, so just re-use the vc_no_ghcb_handler(). It can only handle CPUID exit-codes, but that should be enough to get the kernel through verify_cpu() and __startup_64() until it runs on virtual addresses.
--- arch/x86/include/asm/sev-es.h | 3 +++ arch/x86/kernel/head64.c | 25 ++++++++++++++++++++++++- arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++++++++ 3 files changed, 57 insertions(+), 1 deletion(-)
arch/x86/include/asm/sev-es.h +3 -0
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits) return (val & mask); } +/* Early IDT entry points for #VC handler */ +extern void vc_no_ghcb (void ); +#endif
arch/x86/kernel/head64.c +24 -1
@@ -40,6 +40,7 @@#include <asm/desc.h> #include <asm/extable.h> #include <asm/trapnr.h> +#include <asm/sev-es.h> /* * Manage page tables very early on.
@@ -540,12 +541,34 @@ static struct desc_ptr bringup_idt_descr = { .address = 0, /* Set at runtime */ }; +static void set_bringup_idt_handler (gate_desc *idt,int n,void *handler) +{ +#ifdef CONFIG_AMD_MEM_ENCRYPT +struct idt_data data; +gate_desc desc; + +init_idt_data (&data, n, handler); +idt_init_desc (&desc, &data); +native_write_idt_entry (idt, n, &desc); +#endif +} +/* This runs while still in the direct mapping */ static void startup_64_load_idt (unsigned long physbase) {struct desc_ptr *desc =fixup_pointer (&bringup_idt_descr, physbase); +gate_desc *idt =fixup_pointer (bringup_idt_table, physbase); + + +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) { +void *handler; + +/* VMM Communication Exception */ + handler =fixup_pointer (vc_no_ghcb, physbase); +set_bringup_idt_handler (idt,X86_TRAP_VC , handler); + } - desc->address = (unsigned long )fixup_pointer (bringup_idt_table, physbase); + desc->address = (unsigned long )idt;native_load_idt (desc); }
arch/x86/kernel/head_64.S +30 -0
@@ -348,6 +348,36 @@ SYM_CODE_START_LOCAL(early_idt_handler_common) jmp restore_regs_and_return_to_kernel SYM_CODE_END(early_idt_handler_common) +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * VC Exception handler used during very early boot. The + * early_idt_handler_array can't be used because it returns via the + * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early. + * + * This handler will end up in the .init.text section and not be + * available to boot secondary CPUs. + */ +SYM_CODE_START_NOALIGN(vc_no_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + call do_vc_no_ghcb + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pure iret required here - don't use INTERRUPT_RETURN */ + iretq +SYM_CODE_END(vc_no_ghcb) +#endif #define SYM_DATA_START_PAGE_ALIGNED(name) \ SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
PATCH v7 40/72 x86/sev-es: Setup GHCB based boot #VC handler
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Add the infrastructure to handle #VC exceptions when the kernel runs on virtual addresses and has a GHCB mapped. This handler will be used until the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
--- arch/x86/include/asm/realmode.h | 3 + arch/x86/include/asm/segment.h | 2 +- arch/x86/include/asm/sev-es.h | 2 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/head64.c | 11 +++ arch/x86/kernel/head_64.S | 34 ++++++++++ arch/x86/kernel/sev-es-shared.c | 14 ++-- arch/x86/kernel/sev-es.c | 116 ++++++++++++++++++++++++++++++++ arch/x86/mm/extable.c | 1 + 9 files changed, 177 insertions(+), 8 deletions(-)
arch/x86/include/asm/realmode.h +3 -0
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];extern unsigned long initial_code;extern unsigned long initial_gs;extern unsigned long initial_stack; +#ifdef CONFIG_AMD_MEM_ENCRYPT +extern unsigned long initial_vc_handler; +#endif extern unsigned char real_mode_blob[];extern unsigned char real_mode_relocs[];
arch/x86/include/asm/segment.h +1 -1
@@ -226,7 +226,7 @@#define NUM_EXCEPTION_VECTORS 32/* Bitmask of exception vectors which push an error code on the stack: */ -#define EXCEPTION_ERRCODE_MASK 0x00027d00 +#define EXCEPTION_ERRCODE_MASK 0x20027d00#define GDT_SIZE (GDT_ENTRIES*8)#define GDT_ENTRY_TLS_ENTRIES 3
arch/x86/include/asm/sev-es.h +2 -0
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)/* Early IDT entry points for #VC handler */ extern void vc_no_ghcb (void ); +extern void vc_boot_ghcb (void ); +extern bool handle_vc_boot_ghcb (struct pt_regs *regs);#endif
arch/x86/kernel/Makefile +2 -0
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg CFLAGS_REMOVE_ftrace.o = -pg CFLAGS_REMOVE_early_printk.o = -pg CFLAGS_REMOVE_head64.o = -pg +CFLAGS_REMOVE_sev-es.o = -pg endif KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n KASAN_SANITIZE_dumpstack_$(BITS).o := n KASAN_SANITIZE_stacktrace.o := n KASAN_SANITIZE_paravirt.o := n +KASAN_SANITIZE_sev-es.o := n # With some compiler versions the generated code results in boot hangs, caused # by several compilation units. To be safe, disable all instrumentation.
arch/x86/kernel/head64.c +11 -0
@@ -406,6 +406,12 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr) early_make_pgtable(native_read_cr2()))return ; +#ifdef CONFIG_AMD_MEM_ENCRYPT +if (trapnr ==X86_TRAP_VC && +handle_vc_boot_ghcb (regs)) +return ; +#endif +early_fixup_exception (regs, trapnr); }
@@ -572,6 +578,11 @@ static void startup_64_load_idt(unsigned long physbase)/* This is used when running on kernel addresses */ void early_setup_idt (void ) { +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* VMM Communication Exception */ +set_bringup_idt_handler (X86_TRAP_VC , vc_boot_ghcb); +#endif + bringup_idt_descr.address = (unsigned long )bringup_idt_table;native_load_idt (&bringup_idt_descr); }
arch/x86/kernel/head_64.S +34 -0
@@ -279,6 +279,37 @@ SYM_CODE_START(start_cpu0) movq initial_stack(%rip), %rsp jmp .Ljump_to_C_code SYM_CODE_END(start_cpu0) +#endif + +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * VC Exception handler used during early boot when running on kernel + * addresses, but before the switch to the idt_table can be mase. The + * early_idt_handler_array can't be used here because it calls into a lot of + * __init code and this handler is also used during CPU offlining/onlining. + */ +SYM_CODE_START_NOALIGN(vc_boot_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + movq initial_vc_handler(%rip), %rax + ANNOTATE_RETPOLINE_SAFE + call *%rax + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pure iret required here - don't use INTERRUPT_RETURN */ + iretq +SYM_CODE_END(vc_boot_ghcb) #endif /* Both SMP bootup and ACPI suspend change these variables */
@@ -286,6 +317,9 @@ SYM_CODE_END(start_cpu0) .balign 8 SYM_DATA(initial_code, .quad x86_64_start_kernel) SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data)) +#ifdef CONFIG_AMD_MEM_ENCRYPT +SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) +#endif /* * The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
arch/x86/kernel/sev-es-shared.c +7 -7
@@ -9,7 +9,7 @@* and is included directly into both code-bases. */ -static void __maybe_unused sev_es_terminate (unsigned int reason) +static void sev_es_terminate (unsigned int reason) {u64 val =GHCB_SEV_TERMINATE ;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason) asmvolatile ("hlt\n" : : :"memory" ); } -static bool __maybe_unused sev_es_negotiate_protocol (void ) +static bool sev_es_negotiate_protocol (void ) {u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)return true; } -static void __maybe_unused vc_ghcb_invalidate (struct ghcb *ghcb) +static void vc_ghcb_invalidate (struct ghcb *ghcb) {memset (ghcb->save.valid_bitmap,0 ,sizeof (ghcb->save.valid_bitmap)); }
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code) exit_code <= SVM_EXIT_LAST_EXCP); } -static enum es_result __maybe_unused vc_init_em_ctxt (struct es_em_ctxt *ctxt, -struct pt_regs *regs, -unsigned long exit_code) +static enum es_result vc_init_em_ctxt (struct es_em_ctxt *ctxt, +struct pt_regs *regs, +unsigned long exit_code) {enum es_result ret =ES_OK ;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,return ret; } -static void __maybe_unused vc_finish_insn (struct es_em_ctxt *ctxt) +static void vc_finish_insn (struct es_em_ctxt *ctxt) { ctxt->regs->ip += ctxt->insn.length; }
arch/x86/kernel/sev-es.c +116 -0
132 lines not on this page · raw message
arch/x86/mm/extable.c +1 -0
@@ -5,6 +5,7 @@#include <xen/xen.h> #include <asm/fpu/internal.h> +#include <asm/sev-es.h> #include <asm/traps.h> #include <asm/kdebug.h>
Signature · 1 line
2.28.0
PATCH v7 40/72 x86/sev-es: Setup GHCB based boot #VC handler
From: Joerg Roedel <[email protected]>
Add the infrastructure to handle #VC exceptions when the kernel runs on virtual addresses and has a GHCB mapped. This handler will be used until the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
--- arch/x86/include/asm/realmode.h | 3 + arch/x86/include/asm/segment.h | 2 +- arch/x86/include/asm/sev-es.h | 2 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/head64.c | 8 +++ arch/x86/kernel/head_64.S | 36 ++++++++++ arch/x86/kernel/sev-es-shared.c | 14 ++-- arch/x86/kernel/sev-es.c | 116 ++++++++++++++++++++++++++++++++ arch/x86/mm/extable.c | 1 + 9 files changed, 176 insertions(+), 8 deletions(-)
arch/x86/include/asm/realmode.h +3 -0
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];extern unsigned long initial_code;extern unsigned long initial_gs;extern unsigned long initial_stack; +#ifdef CONFIG_AMD_MEM_ENCRYPT +extern unsigned long initial_vc_handler; +#endif extern unsigned char real_mode_blob[];extern unsigned char real_mode_relocs[];
arch/x86/include/asm/segment.h +1 -1
@@ -226,7 +226,7 @@#define NUM_EXCEPTION_VECTORS 32/* Bitmask of exception vectors which push an error code on the stack: */ -#define EXCEPTION_ERRCODE_MASK 0x00027d00 +#define EXCEPTION_ERRCODE_MASK 0x20027d00#define GDT_SIZE (GDT_ENTRIES*8)#define GDT_ENTRY_TLS_ENTRIES 3
arch/x86/include/asm/sev-es.h +2 -0
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)/* Early IDT entry points for #VC handler */ extern void vc_no_ghcb (void ); +extern void vc_boot_ghcb (void ); +extern bool handle_vc_boot_ghcb (struct pt_regs *regs);#endif
arch/x86/kernel/Makefile +2 -0
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg CFLAGS_REMOVE_ftrace.o = -pg CFLAGS_REMOVE_early_printk.o = -pg CFLAGS_REMOVE_head64.o = -pg +CFLAGS_REMOVE_sev-es.o = -pg endif KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n KASAN_SANITIZE_dumpstack_$(BITS).o := n KASAN_SANITIZE_stacktrace.o := n KASAN_SANITIZE_paravirt.o := n +KASAN_SANITIZE_sev-es.o := n # With some compiler versions the generated code results in boot hangs, caused # by several compilation units. To be safe, disable all instrumentation.
arch/x86/kernel/head64.c +8 -0
@@ -406,6 +406,10 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr) early_make_pgtable(native_read_cr2()))return ; +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT ) && + trapnr ==X86_TRAP_VC &&handle_vc_boot_ghcb (regs)) +return ; +early_fixup_exception (regs, trapnr); }
@@ -573,6 +577,10 @@ static void startup_64_load_idt(unsigned long physbase)/* This is used when running on kernel addresses */ void early_setup_idt (void ) { +/* VMM Communication Exception */ +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) +set_bringup_idt_handler (X86_TRAP_VC , vc_boot_ghcb); + bringup_idt_descr.address = (unsigned long )bringup_idt_table;native_load_idt (&bringup_idt_descr); }
arch/x86/kernel/head_64.S +36 -0
@@ -279,6 +279,39 @@ SYM_CODE_START(start_cpu0) movq initial_stack(%rip), %rsp jmp .Ljump_to_C_code SYM_CODE_END(start_cpu0) +#endif + +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * VC Exception handler used during early boot when running on kernel + * addresses, but before the switch to the idt_table can be made. + * The early_idt_handler_array can't be used here because it calls into a lot + * of __init code and this handler is also used during CPU offlining/onlining. + * Therefore this handler ends up in the .text section so that it stays around + * when .init.text is freed. + */ +SYM_CODE_START_NOALIGN(vc_boot_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + movq initial_vc_handler(%rip), %rax + ANNOTATE_RETPOLINE_SAFE + call *%rax + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pure iret required here - don't use INTERRUPT_RETURN */ + iretq +SYM_CODE_END(vc_boot_ghcb) #endif /* Both SMP bootup and ACPI suspend change these variables */
@@ -286,6 +319,9 @@ SYM_CODE_END(start_cpu0) .balign 8 SYM_DATA(initial_code, .quad x86_64_start_kernel) SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data)) +#ifdef CONFIG_AMD_MEM_ENCRYPT +SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) +#endif /* * The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
arch/x86/kernel/sev-es-shared.c +7 -7
@@ -9,7 +9,7 @@* and is included directly into both code-bases. */ -static void __maybe_unused sev_es_terminate (unsigned int reason) +static void sev_es_terminate (unsigned int reason) {u64 val =GHCB_SEV_TERMINATE ;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason) asmvolatile ("hlt\n" : : :"memory" ); } -static bool __maybe_unused sev_es_negotiate_protocol (void ) +static bool sev_es_negotiate_protocol (void ) {u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)return true; } -static void __maybe_unused vc_ghcb_invalidate (struct ghcb *ghcb) +static void vc_ghcb_invalidate (struct ghcb *ghcb) {memset (ghcb->save.valid_bitmap,0 ,sizeof (ghcb->save.valid_bitmap)); }
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code) exit_code <= SVM_EXIT_LAST_EXCP); } -static enum es_result __maybe_unused vc_init_em_ctxt (struct es_em_ctxt *ctxt, -struct pt_regs *regs, -unsigned long exit_code) +static enum es_result vc_init_em_ctxt (struct es_em_ctxt *ctxt, +struct pt_regs *regs, +unsigned long exit_code) {enum es_result ret =ES_OK ;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,return ret; } -static void __maybe_unused vc_finish_insn (struct es_em_ctxt *ctxt) +static void vc_finish_insn (struct es_em_ctxt *ctxt) { ctxt->regs->ip += ctxt->insn.length; }
arch/x86/kernel/sev-es.c +116 -0
132 lines not on this page · raw message
arch/x86/mm/extable.c +1 -0
@@ -5,6 +5,7 @@#include <xen/xen.h> #include <asm/fpu/internal.h> +#include <asm/sev-es.h> #include <asm/traps.h> #include <asm/kdebug.h>
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH 40/74 x86/sev-es: Setup GHCB based boot #VC handler
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Add the infrastructure to handle #VC exceptions when the kernel runs on virtual addresses and has a GHCB mapped. This handler will be used until the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
--- arch/x86/include/asm/realmode.h | 3 + arch/x86/include/asm/segment.h | 2 +- arch/x86/include/asm/sev-es.h | 2 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/head64.c | 8 +++ arch/x86/kernel/head_64.S | 36 ++++++++++ arch/x86/kernel/sev-es-shared.c | 14 ++-- arch/x86/kernel/sev-es.c | 116 ++++++++++++++++++++++++++++++++ arch/x86/mm/extable.c | 1 + 9 files changed, 176 insertions(+), 8 deletions(-)
arch/x86/include/asm/realmode.h +3 -0
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];extern unsigned long initial_code;extern unsigned long initial_gs;extern unsigned long initial_stack; +#ifdef CONFIG_AMD_MEM_ENCRYPT +extern unsigned long initial_vc_handler; +#endif extern unsigned char real_mode_blob[];extern unsigned char real_mode_relocs[];
arch/x86/include/asm/segment.h +1 -1
@@ -226,7 +226,7 @@#define NUM_EXCEPTION_VECTORS 32/* Bitmask of exception vectors which push an error code on the stack: */ -#define EXCEPTION_ERRCODE_MASK 0x00027d00 +#define EXCEPTION_ERRCODE_MASK 0x20027d00#define GDT_SIZE (GDT_ENTRIES*8)#define GDT_ENTRY_TLS_ENTRIES 3
arch/x86/include/asm/sev-es.h +2 -0
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)/* Early IDT entry points for #VC handler */ extern void vc_no_ghcb (void ); +extern void vc_boot_ghcb (void ); +extern bool handle_vc_boot_ghcb (struct pt_regs *regs);#endif
arch/x86/kernel/Makefile +2 -0
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg CFLAGS_REMOVE_ftrace.o = -pg CFLAGS_REMOVE_early_printk.o = -pg CFLAGS_REMOVE_head64.o = -pg +CFLAGS_REMOVE_sev-es.o = -pg endif KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n KASAN_SANITIZE_dumpstack_$(BITS).o := n KASAN_SANITIZE_stacktrace.o := n KASAN_SANITIZE_paravirt.o := n +KASAN_SANITIZE_sev-es.o := n # With some compiler versions the generated code results in boot hangs, caused # by several compilation units. To be safe, disable all instrumentation.
arch/x86/kernel/head64.c +8 -0
@@ -406,6 +406,10 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr) early_make_pgtable(native_read_cr2()))return ; +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT ) && + trapnr ==X86_TRAP_VC &&handle_vc_boot_ghcb (regs)) +return ; +early_fixup_exception (regs, trapnr); }
@@ -575,6 +579,10 @@ static void startup_64_load_idt(unsigned long physbase)/* This is used when running on kernel addresses */ void early_setup_idt (void ) { +/* VMM Communication Exception */ +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) +set_bringup_idt_handler (bringup_idt_table,X86_TRAP_VC , vc_boot_ghcb); + bringup_idt_descr.address = (unsigned long )bringup_idt_table;native_load_idt (&bringup_idt_descr); }
arch/x86/kernel/head_64.S +36 -0
@@ -279,6 +279,39 @@ SYM_CODE_START(start_cpu0) movq initial_stack(%rip), %rsp jmp .Ljump_to_C_code SYM_CODE_END(start_cpu0) +#endif + +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * VC Exception handler used during early boot when running on kernel + * addresses, but before the switch to the idt_table can be made. + * The early_idt_handler_array can't be used here because it calls into a lot + * of __init code and this handler is also used during CPU offlining/onlining. + * Therefore this handler ends up in the .text section so that it stays around + * when .init.text is freed. + */ +SYM_CODE_START_NOALIGN(vc_boot_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + movq initial_vc_handler(%rip), %rax + ANNOTATE_RETPOLINE_SAFE + call *%rax + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pure iret required here - don't use INTERRUPT_RETURN */ + iretq +SYM_CODE_END(vc_boot_ghcb) #endif /* Both SMP bootup and ACPI suspend change these variables */
@@ -286,6 +319,9 @@ SYM_CODE_END(start_cpu0) .balign 8 SYM_DATA(initial_code, .quad x86_64_start_kernel) SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data)) +#ifdef CONFIG_AMD_MEM_ENCRYPT +SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) +#endif /* * The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
arch/x86/kernel/sev-es-shared.c +7 -7
@@ -9,7 +9,7 @@* and is included directly into both code-bases. */ -static void __maybe_unused sev_es_terminate (unsigned int reason) +static void sev_es_terminate (unsigned int reason) {u64 val =GHCB_SEV_TERMINATE ;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason) asmvolatile ("hlt\n" : : :"memory" ); } -static bool __maybe_unused sev_es_negotiate_protocol (void ) +static bool sev_es_negotiate_protocol (void ) {u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)return true; } -static void __maybe_unused vc_ghcb_invalidate (struct ghcb *ghcb) +static void vc_ghcb_invalidate (struct ghcb *ghcb) {memset (ghcb->save.valid_bitmap,0 ,sizeof (ghcb->save.valid_bitmap)); }
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code) exit_code <= SVM_EXIT_LAST_EXCP); } -static enum es_result __maybe_unused vc_init_em_ctxt (struct es_em_ctxt *ctxt, -struct pt_regs *regs, -unsigned long exit_code) +static enum es_result vc_init_em_ctxt (struct es_em_ctxt *ctxt, +struct pt_regs *regs, +unsigned long exit_code) {enum es_result ret =ES_OK ;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,return ret; } -static void __maybe_unused vc_finish_insn (struct es_em_ctxt *ctxt) +static void vc_finish_insn (struct es_em_ctxt *ctxt) { ctxt->regs->ip += ctxt->insn.length; }
arch/x86/kernel/sev-es.c +116 -0
132 lines not on this page · raw message
arch/x86/mm/extable.c +1 -0
@@ -5,6 +5,7 @@#include <xen/xen.h> #include <asm/fpu/internal.h> +#include <asm/sev-es.h> #include <asm/traps.h> #include <asm/kdebug.h>
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Setup GHCB-based boot #VC handler
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 1aa9aa8ee517e0443b06e816a4fd2d15f2113615 Gitweb: https://git.kernel.org/tip/1aa9aa8ee517e0443b06e816a4fd2d15f2113615 Author: Joerg Roedel <[email protected]> AuthorDate: Tue, 08 Sep 2020 14:38:16 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:32:27 +02:00
x86/sev-es: Setup GHCB-based boot #VC handler
Add the infrastructure to handle #VC exceptions when the kernel runs on virtual addresses and has mapped a GHCB. This handler will be used until the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
[ bp: Make vc_ghcb_invalidate() __always_inline so that it can be
inlined in noinstr functions like __sev_es_nmi_complete(). ]
--- arch/x86/include/asm/realmode.h | 3 +- arch/x86/include/asm/segment.h | 2 +- arch/x86/include/asm/sev-es.h | 2 +- arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/head64.c | 8 ++- arch/x86/kernel/head_64.S | 36 ++++++++++- arch/x86/kernel/sev-es-shared.c | 14 ++-- arch/x86/kernel/sev-es.c | 116 +++++++++++++++++++++++++++++++- arch/x86/mm/extable.c | 1 +- 9 files changed, 176 insertions(+), 8 deletions(-)
arch/x86/include/asm/realmode.h +3 -0
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];extern unsigned long initial_code;extern unsigned long initial_gs;extern unsigned long initial_stack; +#ifdef CONFIG_AMD_MEM_ENCRYPT +extern unsigned long initial_vc_handler; +#endif extern unsigned char real_mode_blob[];extern unsigned char real_mode_relocs[];
arch/x86/include/asm/segment.h +1 -1
@@ -230,7 +230,7 @@#define NUM_EXCEPTION_VECTORS 32/* Bitmask of exception vectors which push an error code on the stack: */ -#define EXCEPTION_ERRCODE_MASK 0x00027d00 +#define EXCEPTION_ERRCODE_MASK 0x20027d00#define GDT_SIZE (GDT_ENTRIES*8)#define GDT_ENTRY_TLS_ENTRIES 3
arch/x86/include/asm/sev-es.h +2 -0
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)/* Early IDT entry points for #VC handler */ extern void vc_no_ghcb (void ); +extern void vc_boot_ghcb (void ); +extern bool handle_vc_boot_ghcb (struct pt_regs *regs);#endif
arch/x86/kernel/Makefile +2 -0
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg CFLAGS_REMOVE_ftrace.o = -pg CFLAGS_REMOVE_early_printk.o = -pg CFLAGS_REMOVE_head64.o = -pg +CFLAGS_REMOVE_sev-es.o = -pg endif KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n KASAN_SANITIZE_dumpstack_$(BITS).o := n KASAN_SANITIZE_stacktrace.o := n KASAN_SANITIZE_paravirt.o := n +KASAN_SANITIZE_sev-es.o := n # With some compiler versions the generated code results in boot hangs, caused # by several compilation units. To be safe, disable all instrumentation.
arch/x86/kernel/head64.c +8 -0
@@ -406,6 +406,10 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr) early_make_pgtable(native_read_cr2()))return ; +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT ) && + trapnr ==X86_TRAP_VC &&handle_vc_boot_ghcb (regs)) +return ; +early_fixup_exception (regs, trapnr); }
@@ -575,6 +579,10 @@ static void startup_64_load_idt(unsigned long physbase)/* This is used when running on kernel addresses */ void early_setup_idt (void ) { +/* VMM Communication Exception */ +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) +set_bringup_idt_handler (bringup_idt_table,X86_TRAP_VC , vc_boot_ghcb); + bringup_idt_descr.address = (unsigned long )bringup_idt_table;native_load_idt (&bringup_idt_descr); }
arch/x86/kernel/head_64.S +36 -0
@@ -281,11 +281,47 @@ SYM_CODE_START(start_cpu0) SYM_CODE_END(start_cpu0) #endif +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * VC Exception handler used during early boot when running on kernel + * addresses, but before the switch to the idt_table can be made. + * The early_idt_handler_array can't be used here because it calls into a lot + * of __init code and this handler is also used during CPU offlining/onlining. + * Therefore this handler ends up in the .text section so that it stays around + * when .init.text is freed. + */ +SYM_CODE_START_NOALIGN(vc_boot_ghcb) + UNWIND_HINT_IRET_REGS offset=8 + + /* Build pt_regs */ + PUSH_AND_CLEAR_REGS + + /* Call C handler */ + movq %rsp, %rdi + movq ORIG_RAX(%rsp), %rsi + movq initial_vc_handler(%rip), %rax + ANNOTATE_RETPOLINE_SAFE + call *%rax + + /* Unwind pt_regs */ + POP_REGS + + /* Remove Error Code */ + addq $8, %rsp + + /* Pure iret required here - don't use INTERRUPT_RETURN */ + iretq +SYM_CODE_END(vc_boot_ghcb) +#endif + /* Both SMP bootup and ACPI suspend change these variables */ __REFDATA .balign 8 SYM_DATA(initial_code, .quad x86_64_start_kernel) SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data)) +#ifdef CONFIG_AMD_MEM_ENCRYPT +SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) +#endif /* * The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
arch/x86/kernel/sev-es-shared.c +7 -7
@@ -9,7 +9,7 @@* and is included directly into both code-bases. */ -static void __maybe_unused sev_es_terminate (unsigned int reason) +static void sev_es_terminate (unsigned int reason) {u64 val =GHCB_SEV_TERMINATE ;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason) asmvolatile ("hlt\n" : : :"memory" ); } -static bool __maybe_unused sev_es_negotiate_protocol (void ) +static bool sev_es_negotiate_protocol (void ) {u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)return true; } -static void __maybe_unused vc_ghcb_invalidate (struct ghcb *ghcb) +static __always_inline void vc_ghcb_invalidate(struct ghcb *ghcb) {memset (ghcb->save.valid_bitmap,0 ,sizeof (ghcb->save.valid_bitmap)); }
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code) exit_code <= SVM_EXIT_LAST_EXCP); } -static enum es_result __maybe_unused vc_init_em_ctxt (struct es_em_ctxt *ctxt, -struct pt_regs *regs, -unsigned long exit_code) +static enum es_result vc_init_em_ctxt (struct es_em_ctxt *ctxt, +struct pt_regs *regs, +unsigned long exit_code) {enum es_result ret =ES_OK ;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,return ret; } -static void __maybe_unused vc_finish_insn (struct es_em_ctxt *ctxt) +static void vc_finish_insn (struct es_em_ctxt *ctxt) { ctxt->regs->ip += ctxt->insn.length; }
arch/x86/kernel/sev-es.c +116 -0
132 lines not on this page · raw message
arch/x86/mm/extable.c +1 -0
@@ -5,6 +5,7 @@#include <xen/xen.h> #include <asm/fpu/internal.h> +#include <asm/sev-es.h> #include <asm/traps.h> #include <asm/kdebug.h>
PATCH v7 41/72 x86/sev-es: Setup per-cpu GHCBs for the runtime handler
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
The runtime handler needs a GHCB per CPU. Set them up and map them unencrypted.
--- arch/x86/include/asm/mem_encrypt.h | 2 ++ arch/x86/kernel/sev-es.c | 56 +++++++++++++++++++++++++++++- arch/x86/kernel/traps.c | 3 ++ 3 files changed, 60 insertions(+), 1 deletion(-)
arch/x86/include/asm/mem_encrypt.h +2 -0
@@ -49,6 +49,7 @@ void __init mem_encrypt_free_decrypted_mem(void);/* Architecture __weak replacement functions */ void __init mem_encrypt_init (void ); +void __init sev_es_init_vc_handling (void );bool sme_active (void );bool sev_active (void );bool sev_es_active (void );
@@ -72,6 +73,7 @@ static inline void __init sme_early_init(void) { }static inline void __init sme_encrypt_kernel (struct boot_params *bp) { }static inline void __init sme_enable (struct boot_params *bp) { } +static inline void sev_es_init_vc_handling (void ) { }static inline bool sme_active (void ) {return false; }static inline bool sev_active (void ) {return false; }static inline bool sev_es_active (void ) {return false; }
arch/x86/kernel/sev-es.c +55 -1
75 lines not on this page · raw message
arch/x86/kernel/traps.c +3 -0
@@ -1082,6 +1082,9 @@ void __init trap_init(void)/* Init cpu_entry_area before IST entries are set up */ setup_cpu_entry_areas (); +/* Init GHCB memory pages when running as an SEV-ES guest */ +sev_es_init_vc_handling (); +idt_setup_traps ();/*
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Borislav Petkov
maintainerRe: [PATCH v7 41/72] x86/sev-es: Setup per-cpu GHCBs for the runtime handler
arch/x86/kernel/sev-es.c static bool __init sev_es_setup_ghcb(void) · line 243
Quoted 5 lines · Joerg Roedel
arch/x86/kernel/sev-es.c +5
-0+void __init sev_es_init_vc_handling (void ) +{ +int cpu; + +BUILD_BUG_ON ((offsetof(struct sev_es_runtime_data , ghcb_page) %PAGE_SIZE ) !=0 );
Simplified that to:
BUILD_BUG_ON(offsetof(struct sev_es_runtime_data, ghcb_page) % PAGE_SIZE);
Signature · 8 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization[tip: x86/seves] x86/sev-es: Setup per-CPU GHCBs for the runtime handler
In reply to Joerg RoedelThe following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 885689e47dfa1499b756a07237eb645234d93cf9 Gitweb: https://git.kernel.org/tip/885689e47dfa1499b756a07237eb645234d93cf9 Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:42 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Setup per-CPU GHCBs for the runtime handler
The runtime handler needs one GHCB per-CPU. Set them up and map them unencrypted.
[ bp: Touchups and simplification. ]
--- arch/x86/include/asm/mem_encrypt.h | 2 +- arch/x86/kernel/sev-es.c | 56 ++++++++++++++++++++++++++++- arch/x86/kernel/traps.c | 3 ++- 3 files changed, 60 insertions(+), 1 deletion(-)
arch/x86/include/asm/mem_encrypt.h +2 -0
@@ -49,6 +49,7 @@ void __init mem_encrypt_free_decrypted_mem(void);/* Architecture __weak replacement functions */ void __init mem_encrypt_init (void ); +void __init sev_es_init_vc_handling (void );bool sme_active (void );bool sev_active (void );bool sev_es_active (void );
@@ -72,6 +73,7 @@ static inline void __init sme_early_init(void) { }static inline void __init sme_encrypt_kernel (struct boot_params *bp) { }static inline void __init sme_enable (struct boot_params *bp) { } +static inline void sev_es_init_vc_handling (void ) { }static inline bool sme_active (void ) {return false; }static inline bool sev_active (void ) {return false; }static inline bool sev_es_active (void ) {return false; }
arch/x86/kernel/sev-es.c +55 -1
75 lines not on this page · raw message
arch/x86/kernel/traps.c +3 -0
@@ -1074,6 +1074,9 @@ void __init trap_init(void)/* Init cpu_entry_area before IST entries are set up */ setup_cpu_entry_areas (); +/* Init GHCB memory pages when running as an SEV-ES guest */ +sev_es_init_vc_handling (); +idt_setup_traps ();/*
PATCH v7 42/72 x86/sev-es: Allocate and Map IST stack for #VC handler
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Allocate and map an IST stack and an additional fall-back stack for the #VC handler. The memory for the stacks is allocated only when SEV-ES is active.
The #VC handler needs to use an IST stack because it could be raised from kernel space with unsafe stack, e.g. in the SYSCALL entry path.
Since the #VC exception can be nested, the #VC handler switches back to the interrupted stack when entered from kernel space. If switching back is not possible the fall-back stack is used.
--- arch/x86/include/asm/cpu_entry_area.h | 33 +++++++++++++++++---------- arch/x86/include/asm/page_64_types.h | 1 + arch/x86/kernel/cpu/common.c | 2 ++ arch/x86/kernel/dumpstack_64.c | 8 +++++-- arch/x86/kernel/sev-es.c | 33 +++++++++++++++++++++++++++ 5 files changed, 63 insertions(+), 14 deletions(-)
arch/x86/include/asm/cpu_entry_area.h +21 -12
@@ -11,25 +11,29 @@#ifdef CONFIG_X86_64 /* Macro to enforce the same ordering and stack sizes */ -#define ESTACKS_MEMBERS (guardsize) \ - char DF_stack_guard[guardsize]; \ - char DF_stack[EXCEPTION_STKSZ]; \ - char NMI_stack_guard[guardsize]; \ - char NMI_stack[EXCEPTION_STKSZ]; \ - char DB_stack_guard[guardsize]; \ - char DB_stack[EXCEPTION_STKSZ]; \ - char MCE_stack_guard[guardsize]; \ - char MCE_stack[EXCEPTION_STKSZ]; \ - char IST_top_guard[guardsize]; \ +#define ESTACKS_MEMBERS (guardsize, optional_stack_size) \ + char DF_stack_guard[guardsize]; \ + char DF_stack[EXCEPTION_STKSZ]; \ + char NMI_stack_guard[guardsize]; \ + char NMI_stack[EXCEPTION_STKSZ]; \ + char DB_stack_guard[guardsize]; \ + char DB_stack[EXCEPTION_STKSZ]; \ + char MCE_stack_guard[guardsize]; \ + char MCE_stack[EXCEPTION_STKSZ]; \ + char VC_stack_guard[guardsize]; \ + char VC_stack[optional_stack_size]; \ + char VC2_stack_guard[guardsize]; \ + char VC2_stack[optional_stack_size]; \ + char IST_top_guard[guardsize]; \/* The exception stacks' physical storage. No guard pages required */ struct exception_stacks { - ESTACKS_MEMBERS(0) + ESTACKS_MEMBERS(0, 0) };/* The effective cpu entry area mapping with guard pages. */ struct cea_exception_stacks { - ESTACKS_MEMBERS(PAGE_SIZE) + ESTACKS_MEMBERS(PAGE_SIZE, EXCEPTION_STKSZ) };/*
@@ -40,6 +44,8 @@ enum exception_stack_ordering {ESTACK_NMI ,ESTACK_DB ,ESTACK_MCE , +ESTACK_VC , +ESTACK_VC2 , N_EXCEPTION_STACKS };
@@ -139,4 +145,7 @@ static inline struct entry_stack *cpu_entry_stack(int cpu)#define __this_cpu_ist_top_va (name) \ CEA_ESTACK_TOP(__this_cpu_read(cea_exception_stacks), name) +#define __this_cpu_ist_bottom_va (name) \ + CEA_ESTACK_BOT(__this_cpu_read(cea_exception_stacks), name) +#endif
arch/x86/include/asm/page_64_types.h +1 -0
@@ -28,6 +28,7 @@#define IST_INDEX_NMI 1#define IST_INDEX_DB 2#define IST_INDEX_MCE 3 +#define IST_INDEX_VC 4/* * Set __PAGE_OFFSET to the most negative possible address +
arch/x86/kernel/cpu/common.c +2 -0
@@ -1821,6 +1821,8 @@ static inline void tss_setup_ist(struct tss_struct *tss) tss->x86_tss.ist[IST_INDEX_NMI ] =__this_cpu_ist_top_va (NMI ); tss->x86_tss.ist[IST_INDEX_DB ] =__this_cpu_ist_top_va (DB ); tss->x86_tss.ist[IST_INDEX_MCE ] =__this_cpu_ist_top_va (MCE ); +/* Only mapped when SEV-ES is active */ + tss->x86_tss.ist[IST_INDEX_VC] = __this_cpu_ist_top_va(VC); }#else /* CONFIG_X86_64 */
arch/x86/kernel/dumpstack_64.c +6 -2
@@ -24,11 +24,13 @@ static const char * const exception_stack_names[] = { [ ESTACK_NMI ] = "NMI", [ ESTACK_DB ] = "#DB", [ ESTACK_MCE ] = "#MC", + [ ESTACK_VC ] = "#VC", + [ ESTACK_VC2 ] = "#VC2", };const char *stack_type_name (enum stack_type type) { -BUILD_BUG_ON (N_EXCEPTION_STACKS !=4 ); +BUILD_BUG_ON (N_EXCEPTION_STACKS !=6 );if (type ==STACK_TYPE_IRQ )return "IRQ" ;
@@ -79,6 +81,8 @@ struct estack_pages estack_pages[CEA_ESTACK_PAGES] ____cacheline_aligned = {EPAGERANGE (NMI ),EPAGERANGE (DB ),EPAGERANGE (MCE ), +EPAGERANGE (VC ), + EPAGERANGE(VC2), };static bool in_exception_stack (unsigned long *stack,struct stack_info *info)
@@ -88,7 +92,7 @@ static bool in_exception_stack(unsigned long *stack, struct stack_info *info)struct pt_regs *regs;unsigned int k; -BUILD_BUG_ON (N_EXCEPTION_STACKS !=4 ); +BUILD_BUG_ON (N_EXCEPTION_STACKS !=6 ); begin = (unsigned long )__this_cpu_read (cea_exception_stacks);/*
arch/x86/kernel/sev-es.c +33 -0
@@ -17,6 +17,7 @@#include <linux/kernel.h> #include <linux/mm.h> +#include <asm/cpu_entry_area.h> #include <asm/sev-es.h> #include <asm/insn-eval.h> #include <asm/fpu/internal.h>
@@ -37,10 +38,41 @@ static struct ghcb __initdata *boot_ghcb;/* #VC handler runtime per-cpu data */ struct sev_es_runtime_data {struct ghcb ghcb_page; + +/* Physical storage for the per-cpu IST stack of the #VC handler */ +char ist_stack[EXCEPTION_STKSZ ]__aligned(PAGE_SIZE) ; + +/* +* Physical storage for the per-cpu fall-back stack of the #VC handler. +* The fall-back stack is used when it is not safe to switch back to the +* interrupted stack in the #VC entry code. +*/ +char fallback_stack[EXCEPTION_STKSZ ]__aligned(PAGE_SIZE) ; }; static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data); +static void __init setup_vc_stacks (int cpu) +{ +struct sev_es_runtime_data *data; +struct cpu_entry_area *cea; +unsigned long vaddr; +phys_addr_t pa; + + data =per_cpu (runtime_data, cpu); + cea =get_cpu_entry_area (cpu); + +/* Map #VC IST stack */ + vaddr =CEA_ESTACK_BOT (&cea->estacks,VC ); + pa =__pa (data->ist_stack); +cea_set_pte ((void *)vaddr, pa,PAGE_KERNEL ); + +/* Map VC fall-back stack */ + vaddr =CEA_ESTACK_BOT (&cea->estacks,VC2 ); + pa =__pa (data->fallback_stack); +cea_set_pte ((void *)vaddr, pa,PAGE_KERNEL ); +} +/* Needed in vc_early_forward_exception */ void do_early_exception (struct pt_regs *regs,int trapnr);
@@ -249,6 +281,7 @@ void __init sev_es_init_vc_handling(void)for_each_possible_cpu (cpu) {alloc_runtime_data (cpu);init_ghcb (cpu); +setup_vc_stacks (cpu); } }
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Allocate and map an IST stack for #VC handler
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 02772fb9b68e6a72a5e17f994048df832fe2b15e Gitweb: https://git.kernel.org/tip/02772fb9b68e6a72a5e17f994048df832fe2b15e Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:43 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Allocate and map an IST stack for #VC handler
Allocate and map an IST stack and an additional fall-back stack for the #VC handler. The memory for the stacks is allocated only when SEV-ES is active.
The #VC handler needs to use an IST stack because a #VC exception can be raised from kernel space with unsafe stack, e.g. in the SYSCALL entry path.
Since the #VC exception can be nested, the #VC handler switches back to the interrupted stack when entered from kernel space. If switching back is not possible, the fall-back stack is used.
--- arch/x86/include/asm/cpu_entry_area.h | 33 ++++++++++++++++---------- arch/x86/include/asm/page_64_types.h | 1 +- arch/x86/kernel/cpu/common.c | 2 ++- arch/x86/kernel/dumpstack_64.c | 8 ++++-- arch/x86/kernel/sev-es.c | 33 ++++++++++++++++++++++++++- 5 files changed, 63 insertions(+), 14 deletions(-)
arch/x86/include/asm/cpu_entry_area.h +21 -12
@@ -11,25 +11,29 @@#ifdef CONFIG_X86_64 /* Macro to enforce the same ordering and stack sizes */ -#define ESTACKS_MEMBERS (guardsize) \ - char DF_stack_guard[guardsize]; \ - char DF_stack[EXCEPTION_STKSZ]; \ - char NMI_stack_guard[guardsize]; \ - char NMI_stack[EXCEPTION_STKSZ]; \ - char DB_stack_guard[guardsize]; \ - char DB_stack[EXCEPTION_STKSZ]; \ - char MCE_stack_guard[guardsize]; \ - char MCE_stack[EXCEPTION_STKSZ]; \ - char IST_top_guard[guardsize]; \ +#define ESTACKS_MEMBERS (guardsize, optional_stack_size) \ + char DF_stack_guard[guardsize]; \ + char DF_stack[EXCEPTION_STKSZ]; \ + char NMI_stack_guard[guardsize]; \ + char NMI_stack[EXCEPTION_STKSZ]; \ + char DB_stack_guard[guardsize]; \ + char DB_stack[EXCEPTION_STKSZ]; \ + char MCE_stack_guard[guardsize]; \ + char MCE_stack[EXCEPTION_STKSZ]; \ + char VC_stack_guard[guardsize]; \ + char VC_stack[optional_stack_size]; \ + char VC2_stack_guard[guardsize]; \ + char VC2_stack[optional_stack_size]; \ + char IST_top_guard[guardsize]; \/* The exception stacks' physical storage. No guard pages required */ struct exception_stacks { - ESTACKS_MEMBERS(0) + ESTACKS_MEMBERS(0, 0) };/* The effective cpu entry area mapping with guard pages. */ struct cea_exception_stacks { - ESTACKS_MEMBERS(PAGE_SIZE) + ESTACKS_MEMBERS(PAGE_SIZE, EXCEPTION_STKSZ) };/*
@@ -40,6 +44,8 @@ enum exception_stack_ordering {ESTACK_NMI ,ESTACK_DB ,ESTACK_MCE , +ESTACK_VC , +ESTACK_VC2 , N_EXCEPTION_STACKS };
@@ -139,4 +145,7 @@ static inline struct entry_stack *cpu_entry_stack(int cpu)#define __this_cpu_ist_top_va (name) \ CEA_ESTACK_TOP(__this_cpu_read(cea_exception_stacks), name) +#define __this_cpu_ist_bottom_va (name) \ + CEA_ESTACK_BOT(__this_cpu_read(cea_exception_stacks), name) +#endif
arch/x86/include/asm/page_64_types.h +1 -0
@@ -28,6 +28,7 @@#define IST_INDEX_NMI 1#define IST_INDEX_DB 2#define IST_INDEX_MCE 3 +#define IST_INDEX_VC 4/* * Set __PAGE_OFFSET to the most negative possible address +
arch/x86/kernel/cpu/common.c +2 -0
@@ -1829,6 +1829,8 @@ static inline void tss_setup_ist(struct tss_struct *tss) tss->x86_tss.ist[IST_INDEX_NMI ] =__this_cpu_ist_top_va (NMI ); tss->x86_tss.ist[IST_INDEX_DB ] =__this_cpu_ist_top_va (DB ); tss->x86_tss.ist[IST_INDEX_MCE ] =__this_cpu_ist_top_va (MCE ); +/* Only mapped when SEV-ES is active */ + tss->x86_tss.ist[IST_INDEX_VC] = __this_cpu_ist_top_va(VC); }#else /* CONFIG_X86_64 */
arch/x86/kernel/dumpstack_64.c +6 -2
@@ -24,11 +24,13 @@ static const char * const exception_stack_names[] = { [ ESTACK_NMI ] = "NMI", [ ESTACK_DB ] = "#DB", [ ESTACK_MCE ] = "#MC", + [ ESTACK_VC ] = "#VC", + [ ESTACK_VC2 ] = "#VC2", };const char *stack_type_name (enum stack_type type) { -BUILD_BUG_ON (N_EXCEPTION_STACKS !=4 ); +BUILD_BUG_ON (N_EXCEPTION_STACKS !=6 );if (type ==STACK_TYPE_IRQ )return "IRQ" ;
@@ -79,6 +81,8 @@ struct estack_pages estack_pages[CEA_ESTACK_PAGES] ____cacheline_aligned = {EPAGERANGE (NMI ),EPAGERANGE (DB ),EPAGERANGE (MCE ), +EPAGERANGE (VC ), + EPAGERANGE(VC2), };static bool in_exception_stack (unsigned long *stack,struct stack_info *info)
@@ -88,7 +92,7 @@ static bool in_exception_stack(unsigned long *stack, struct stack_info *info)struct pt_regs *regs;unsigned int k; -BUILD_BUG_ON (N_EXCEPTION_STACKS !=4 ); +BUILD_BUG_ON (N_EXCEPTION_STACKS !=6 ); begin = (unsigned long )__this_cpu_read (cea_exception_stacks);/*
arch/x86/kernel/sev-es.c +33 -0
@@ -17,6 +17,7 @@#include <linux/kernel.h> #include <linux/mm.h> +#include <asm/cpu_entry_area.h> #include <asm/sev-es.h> #include <asm/insn-eval.h> #include <asm/fpu/internal.h>
@@ -37,10 +38,41 @@ static struct ghcb __initdata *boot_ghcb;/* #VC handler runtime per-CPU data */ struct sev_es_runtime_data {struct ghcb ghcb_page; + +/* Physical storage for the per-CPU IST stack of the #VC handler */ +char ist_stack[EXCEPTION_STKSZ ]__aligned(PAGE_SIZE) ; + +/* +* Physical storage for the per-CPU fall-back stack of the #VC handler. +* The fall-back stack is used when it is not safe to switch back to the +* interrupted stack in the #VC entry code. +*/ +char fallback_stack[EXCEPTION_STKSZ ]__aligned(PAGE_SIZE) ; }; static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data); +static void __init setup_vc_stacks (int cpu) +{ +struct sev_es_runtime_data *data; +struct cpu_entry_area *cea; +unsigned long vaddr; +phys_addr_t pa; + + data =per_cpu (runtime_data, cpu); + cea =get_cpu_entry_area (cpu); + +/* Map #VC IST stack */ + vaddr =CEA_ESTACK_BOT (&cea->estacks,VC ); + pa =__pa (data->ist_stack); +cea_set_pte ((void *)vaddr, pa,PAGE_KERNEL ); + +/* Map VC fall-back stack */ + vaddr =CEA_ESTACK_BOT (&cea->estacks,VC2 ); + pa =__pa (data->fallback_stack); +cea_set_pte ((void *)vaddr, pa,PAGE_KERNEL ); +} +/* Needed in vc_early_forward_exception */ void do_early_exception (struct pt_regs *regs,int trapnr);
@@ -249,6 +281,7 @@ void __init sev_es_init_vc_handling(void)for_each_possible_cpu (cpu) {alloc_runtime_data (cpu);init_ghcb (cpu); +setup_vc_stacks (cpu); } }
PATCH v7 43/72 x86/sev-es: Adjust #VC IST Stack on entering NMI handler
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
When an NMI hits in the #VC handler entry code before it switched to another stack, any subsequent #VC exception in the NMI code-path will overwrite the interrupted #VC handlers stack.
Make sure this doesn't happen by explicitly adjusting the #VC IST entry in the NMI handler for the time in can cause #VC exceptions.
--- arch/x86/include/asm/sev-es.h | 19 +++++++++++++ arch/x86/kernel/nmi.c | 9 ++++++ arch/x86/kernel/sev-es.c | 53 +++++++++++++++++++++++++++++++++++ 3 files changed, 81 insertions(+)
arch/x86/include/asm/sev-es.h +19 -0
@@ -78,4 +78,23 @@ extern void vc_no_ghcb(void);extern void vc_boot_ghcb (void );extern bool handle_vc_boot_ghcb (struct pt_regs *regs); +#ifdef CONFIG_AMD_MEM_ENCRYPT +extern struct static_key_false sev_es_enable_key; +extern void __sev_es_ist_enter (struct pt_regs *regs); +extern void __sev_es_ist_exit (void ); +static __always_inline void sev_es_ist_enter(struct pt_regs *regs) +{ +if (static_branch_unlikely (&sev_es_enable_key)) +__sev_es_ist_enter (regs); +} +static __always_inline void sev_es_ist_exit(void) +{ +if (static_branch_unlikely (&sev_es_enable_key)) +__sev_es_ist_exit (); +} +#else +static inline void sev_es_ist_enter (struct pt_regs *regs) { } +static inline void sev_es_ist_exit (void ) { } +#endif +#endif
arch/x86/kernel/nmi.c +9 -0
@@ -33,6 +33,7 @@#include <asm/reboot.h> #include <asm/cache.h> #include <asm/nospec-branch.h> +#include <asm/sev-es.h> #define CREATE_TRACE_POINTS #include <trace/events/nmi.h>
@@ -488,6 +489,12 @@ DEFINE_IDTENTRY_RAW(exc_nmi)this_cpu_write (nmi_cr2,read_cr2 ()); nmi_restart: +/* +* Needs to happen before DR7 is accessed, because the hypervisor can +* intercept DR7 reads/writes, turings those into #VC exceptions. +*/ +sev_es_ist_enter (regs); +this_cpu_write (nmi_dr7,local_db_save ()); irq_state =idtentry_enter_nmi (regs);
@@ -501,6 +508,8 @@ DEFINE_IDTENTRY_RAW(exc_nmi)local_db_restore (this_cpu_read (nmi_dr7)); +sev_es_ist_exit (); +if (unlikely (this_cpu_read (nmi_cr2) !=read_cr2 ()))write_cr2 (this_cpu_read (nmi_cr2));if (this_cpu_dec_return (nmi_state))
arch/x86/kernel/sev-es.c +53 -0
71 lines not on this page · raw message
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Adjust #VC IST Stack on entering NMI handler
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 315562c9af3d583502b35c4b223a08d95ce69864 Gitweb: https://git.kernel.org/tip/315562c9af3d583502b35c4b223a08d95ce69864 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:44 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Adjust #VC IST Stack on entering NMI handler
When an NMI hits in the #VC handler entry code before it has switched to another stack, any subsequent #VC exception in the NMI code-path will overwrite the interrupted #VC handler's stack.
Make sure this doesn't happen by explicitly adjusting the #VC IST entry in the NMI handler for the time it can cause #VC exceptions.
[ bp: Touchups, spelling fixes. ]
--- arch/x86/include/asm/sev-es.h | 19 ++++++++++++- arch/x86/kernel/nmi.c | 9 ++++++- arch/x86/kernel/sev-es.c | 53 ++++++++++++++++++++++++++++++++++- 3 files changed, 81 insertions(+)
arch/x86/include/asm/sev-es.h +19 -0
@@ -78,4 +78,23 @@ extern void vc_no_ghcb(void);extern void vc_boot_ghcb (void );extern bool handle_vc_boot_ghcb (struct pt_regs *regs); +#ifdef CONFIG_AMD_MEM_ENCRYPT +extern struct static_key_false sev_es_enable_key; +extern void __sev_es_ist_enter (struct pt_regs *regs); +extern void __sev_es_ist_exit (void ); +static __always_inline void sev_es_ist_enter(struct pt_regs *regs) +{ +if (static_branch_unlikely (&sev_es_enable_key)) +__sev_es_ist_enter (regs); +} +static __always_inline void sev_es_ist_exit(void) +{ +if (static_branch_unlikely (&sev_es_enable_key)) +__sev_es_ist_exit (); +} +#else +static inline void sev_es_ist_enter (struct pt_regs *regs) { } +static inline void sev_es_ist_exit (void ) { } +#endif +#endif
arch/x86/kernel/nmi.c +9 -0
@@ -33,6 +33,7 @@#include <asm/reboot.h> #include <asm/cache.h> #include <asm/nospec-branch.h> +#include <asm/sev-es.h> #define CREATE_TRACE_POINTS #include <trace/events/nmi.h>
@@ -488,6 +489,12 @@ DEFINE_IDTENTRY_RAW(exc_nmi)this_cpu_write (nmi_cr2,read_cr2 ()); nmi_restart: +/* +* Needs to happen before DR7 is accessed, because the hypervisor can +* intercept DR7 reads/writes, turning those into #VC exceptions. +*/ +sev_es_ist_enter (regs); +this_cpu_write (nmi_dr7,local_db_save ()); irq_state =idtentry_enter_nmi (regs);
@@ -501,6 +508,8 @@ nmi_restart:local_db_restore (this_cpu_read (nmi_dr7)); +sev_es_ist_exit (); +if (unlikely (this_cpu_read (nmi_cr2) !=read_cr2 ()))write_cr2 (this_cpu_read (nmi_cr2));if (this_cpu_dec_return (nmi_state))
arch/x86/kernel/sev-es.c +53 -0
71 lines not on this page · raw message
PATCH v7 44/72 x86/dumpstack/64: Add noinstr version of get_stack_info()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The get_stack_info functionality is needed in the entry code for the #VC exception handler. Provide a version of it in the .text.noinstr section which can be called safely from there.
--- arch/x86/include/asm/stacktrace.h | 2 ++ arch/x86/kernel/dumpstack.c | 7 +++--- arch/x86/kernel/dumpstack_64.c | 38 ++++++++++++++++++------------- arch/x86/mm/cpu_entry_area.c | 3 ++- 4 files changed, 30 insertions(+), 20 deletions(-)
arch/x86/include/asm/stacktrace.h +2 -0
@@ -35,6 +35,8 @@ bool in_entry_stack(unsigned long *stack, struct stack_info *info);int get_stack_info (unsigned long *stack,struct task_struct *task,struct stack_info *info,unsigned long *visit_mask); +bool get_stack_info_noinstr (unsigned long *stack,struct task_struct *task, +struct stack_info *info);const char *stack_type_name (enum stack_type type);
arch/x86/kernel/dumpstack.c +4 -3
@@ -29,8 +29,8 @@ static int die_counter;static struct pt_regs exec_summary_regs; -bool in_task_stack (unsigned long *stack,struct task_struct *task, -struct stack_info *info) +bool noinstr in_task_stack(unsigned long *stack, struct task_struct *task, +struct stack_info *info) {unsigned long *begin =task_stack_page (task);unsigned long *end =task_stack_page (task) +THREAD_SIZE ;
@@ -46,7 +46,8 @@ bool in_task_stack(unsigned long *stack, struct task_struct *task,return true; } -bool in_entry_stack (unsigned long *stack,struct stack_info *info) +/* Called from get_stack_info_noinstr - so must be noinstr too */ +bool noinstr in_entry_stack(unsigned long *stack, struct stack_info *info) {struct entry_stack *ss =cpu_entry_stack (smp_processor_id ());
arch/x86/kernel/dumpstack_64.c +22 -16
@@ -85,7 +85,7 @@ struct estack_pages estack_pages[CEA_ESTACK_PAGES] ____cacheline_aligned = { EPAGERANGE(VC2), }; -static bool in_exception_stack (unsigned long *stack,struct stack_info *info) +static __always_inline bool in_exception_stack(unsigned long *stack, struct stack_info *info) {unsigned long begin, end, stk = (unsigned long )stack;const struct estack_pages *ep;
@@ -126,7 +126,7 @@ static bool in_exception_stack(unsigned long *stack, struct stack_info *info)return true; } -static bool in_irq_stack (unsigned long *stack,struct stack_info *info) +static __always_inline bool in_irq_stack(unsigned long *stack, struct stack_info *info) {unsigned long *end = (unsigned long *)this_cpu_read (hardirq_stack_ptr);unsigned long *begin = end - (IRQ_STACK_SIZE /sizeof (long ));
@@ -151,32 +151,38 @@ static bool in_irq_stack(unsigned long *stack, struct stack_info *info)return true; } -int get_stack_info (unsigned long *stack,struct task_struct *task, -struct stack_info *info,unsigned long *visit_mask) +bool noinstr get_stack_info_noinstr(unsigned long *stack, struct task_struct *task, +struct stack_info *info) { -if (!stack) - goto unknown; - - task = task ? : current; -if (in_task_stack (stack, task, info)) - goto recursion_check; +return true;if (task != current) - goto unknown; +return false;if (in_exception_stack (stack, info)) - goto recursion_check; +return true;if (in_irq_stack (stack, info)) - goto recursion_check; +return true;if (in_entry_stack (stack, info)) - goto recursion_check; +return true; + +return false; +} + +int get_stack_info (unsigned long *stack,struct task_struct *task, +struct stack_info *info,unsigned long *visit_mask) +{ + task = task ? : current; - goto unknown; +if (!stack) + goto unknown; + +if (!get_stack_info_noinstr (stack, task, info)) + goto unknown; -recursion_check:/* * Make sure we don't iterate through any given stack more than once. * If it comes up a second time then there's something wrong going on:
arch/x86/mm/cpu_entry_area.c +2 -1
@@ -21,7 +21,8 @@ DEFINE_PER_CPU(struct cea_exception_stacks*, cea_exception_stacks); DECLARE_PER_CPU_PAGE_ALIGNED(struct doublefault_stack, doublefault_stack);#endif -struct cpu_entry_area *get_cpu_entry_area (int cpu) +/* Is called from entry code, so must be noinstr */ +noinstr struct cpu_entry_area *get_cpu_entry_area(int cpu) {unsigned long va =CPU_ENTRY_AREA_PER_CPU + cpu *CPU_ENTRY_AREA_SIZE ;BUILD_BUG_ON (sizeof (struct cpu_entry_area ) %PAGE_SIZE !=0 );
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/dumpstack/64: Add noinstr version of get_stack_info()
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 6b27edd74a5e9669120f7bd0ae1f475d124c1042 Gitweb: https://git.kernel.org/tip/6b27edd74a5e9669120f7bd0ae1f475d124c1042 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:45 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/dumpstack/64: Add noinstr version of get_stack_info()
The get_stack_info() functionality is needed in the entry code for the #VC exception handler. Provide a version of it in the .text.noinstr section which can be called safely from there.
--- arch/x86/include/asm/stacktrace.h | 2 ++- arch/x86/kernel/dumpstack.c | 7 +++--- arch/x86/kernel/dumpstack_64.c | 38 +++++++++++++++++------------- arch/x86/mm/cpu_entry_area.c | 3 +- 4 files changed, 30 insertions(+), 20 deletions(-)
arch/x86/include/asm/stacktrace.h +2 -0
@@ -35,6 +35,8 @@ bool in_entry_stack(unsigned long *stack, struct stack_info *info);int get_stack_info (unsigned long *stack,struct task_struct *task,struct stack_info *info,unsigned long *visit_mask); +bool get_stack_info_noinstr (unsigned long *stack,struct task_struct *task, +struct stack_info *info);const char *stack_type_name (enum stack_type type);
arch/x86/kernel/dumpstack.c +4 -3
@@ -29,8 +29,8 @@ static int die_counter;static struct pt_regs exec_summary_regs; -bool in_task_stack (unsigned long *stack,struct task_struct *task, -struct stack_info *info) +bool noinstr in_task_stack(unsigned long *stack, struct task_struct *task, +struct stack_info *info) {unsigned long *begin =task_stack_page (task);unsigned long *end =task_stack_page (task) +THREAD_SIZE ;
@@ -46,7 +46,8 @@ bool in_task_stack(unsigned long *stack, struct task_struct *task,return true; } -bool in_entry_stack (unsigned long *stack,struct stack_info *info) +/* Called from get_stack_info_noinstr - so must be noinstr too */ +bool noinstr in_entry_stack(unsigned long *stack, struct stack_info *info) {struct entry_stack *ss =cpu_entry_stack (smp_processor_id ());
arch/x86/kernel/dumpstack_64.c +22 -16
@@ -85,7 +85,7 @@ struct estack_pages estack_pages[CEA_ESTACK_PAGES] ____cacheline_aligned = { EPAGERANGE(VC2), }; -static bool in_exception_stack (unsigned long *stack,struct stack_info *info) +static __always_inline bool in_exception_stack(unsigned long *stack, struct stack_info *info) {unsigned long begin, end, stk = (unsigned long )stack;const struct estack_pages *ep;
@@ -126,7 +126,7 @@ static bool in_exception_stack(unsigned long *stack, struct stack_info *info)return true; } -static bool in_irq_stack (unsigned long *stack,struct stack_info *info) +static __always_inline bool in_irq_stack(unsigned long *stack, struct stack_info *info) {unsigned long *end = (unsigned long *)this_cpu_read (hardirq_stack_ptr);unsigned long *begin = end - (IRQ_STACK_SIZE /sizeof (long ));
@@ -151,32 +151,38 @@ static bool in_irq_stack(unsigned long *stack, struct stack_info *info)return true; } -int get_stack_info (unsigned long *stack,struct task_struct *task, -struct stack_info *info,unsigned long *visit_mask) +bool noinstr get_stack_info_noinstr(unsigned long *stack, struct task_struct *task, +struct stack_info *info) { -if (!stack) - goto unknown; - - task = task ? : current; -if (in_task_stack (stack, task, info)) - goto recursion_check; +return true;if (task != current) - goto unknown; +return false;if (in_exception_stack (stack, info)) - goto recursion_check; +return true;if (in_irq_stack (stack, info)) - goto recursion_check; +return true;if (in_entry_stack (stack, info)) - goto recursion_check; +return true; + +return false; +} + +int get_stack_info (unsigned long *stack,struct task_struct *task, +struct stack_info *info,unsigned long *visit_mask) +{ + task = task ? : current; - goto unknown; +if (!stack) + goto unknown; + +if (!get_stack_info_noinstr (stack, task, info)) + goto unknown; -recursion_check:/* * Make sure we don't iterate through any given stack more than once. * If it comes up a second time then there's something wrong going on:
arch/x86/mm/cpu_entry_area.c +2 -1
@@ -21,7 +21,8 @@ DEFINE_PER_CPU(struct cea_exception_stacks*, cea_exception_stacks); DECLARE_PER_CPU_PAGE_ALIGNED(struct doublefault_stack, doublefault_stack);#endif -struct cpu_entry_area *get_cpu_entry_area (int cpu) +/* Is called from entry code, so must be noinstr */ +noinstr struct cpu_entry_area *get_cpu_entry_area(int cpu) {unsigned long va =CPU_ENTRY_AREA_PER_CPU + cpu *CPU_ENTRY_AREA_SIZE ;BUILD_BUG_ON (sizeof (struct cpu_entry_area ) %PAGE_SIZE !=0 );
PATCH v7 45/72 x86/entry/64: Add entry code for #VC handler
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The #VC handler needs special entry code because:
1. It runs on an IST stack
2. It needs to be able to handle nested #VC exceptions
To make this work the entry code is implemented to pretend it doesn't use an IST stack. When entered from user-mode or early SYSCALL entry path it switches to the task stack, if entered from kernel-mode it tries to switch back to the previous stack in the IRET frame.
The stack found in the IRET frame is validated first, and if it is not safe to use it for the #VC handler, the code will switch to a fall-back stack (the #VC2 IST stack). From there it can cause nested exceptions again.
--- arch/x86/entry/entry_64.S | 80 +++++++++++++++++++++++++++++++++ arch/x86/include/asm/idtentry.h | 44 ++++++++++++++++++ arch/x86/include/asm/proto.h | 1 + arch/x86/include/asm/traps.h | 1 + arch/x86/kernel/traps.c | 45 +++++++++++++++++++ 5 files changed, 171 insertions(+)
arch/x86/entry/entry_64.S +80 -0
@@ -101,6 +101,8 @@ SYM_CODE_START(entry_SYSCALL_64) SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp +SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL) + /* Construct struct pt_regs on stack */ pushq $__USER_DS /* pt_regs->ss */ pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2) /* pt_regs->sp */
@@ -446,6 +448,84 @@ _ASM_NOKPROBE(\asmsym) SYM_CODE_END(\asmsym) .endm +#ifdef CONFIG_AMD_MEM_ENCRYPT +/** + * idtentry_vc - Macro to generate entry stub for #VC + * @vector: Vector number + * @asmsym: ASM symbol for the entry point + * @cfunc: C function to be called + * + * The macro emits code to set up the kernel context for #VC. The #VC handler + * runs on an IST stack and needs to be able to cause nested #VC exceptions. + * + * To make this work the #VC entry code tries its best to pretend it doesn't use + * an IST stack by switching to the task stack if coming from user-space (which + * includes early SYSCALL entry path) or back to the stack in the IRET frame if + * entered from kernel-mode. + * + * If entered from kernel-mode the return stack is validated first, and if it is + * not safe to use (e.g. because it points to the entry stack) the #VC handler + * will switch to a fall-back stack (VC2) and call a special handler function. + * + * The macro is only used for one vector, but it is planned to be extended in + * the future for the #HV exception. + */ +.macro idtentry_vc vector asmsym cfunc +SYM_CODE_START(\asmsym) + UNWIND_HINT_IRET_REGS + ASM_CLAC + + /* + * If the entry is from userspace, switch stacks and treat it as + * a normal entry. + */ + testb $3, CS-ORIG_RAX(%rsp) + jnz .Lfrom_usermode_switch_stack_\@ + + /* + * paranoid_entry returns SWAPGS flag for paranoid_exit in EBX. + * EBX == 0 -> SWAPGS, EBX == 1 -> no SWAPGS + */ + call paranoid_entry + + UNWIND_HINT_REGS + + /* + * Switch off the IST stack to make it free for nested exceptions. The + * vc_switch_off_ist() function will switch back to the interrupted + * stack if it is safe to do so. If not it switches to the VC fall-back + * stack. + */ + movq %rsp, %rdi /* pt_regs pointer */ + call vc_switch_off_ist + movq %rax, %rsp /* Switch to new stack */ + + UNWIND_HINT_REGS + + /* Update pt_regs */ + movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/ + movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ + + movq %rsp, %rdi /* pt_regs pointer */ + + call \cfunc + + /* + * No need to switch back to the IST stack. The current stack is either + * identical to the stack in the IRET frame or the VC fall-back stack, + * so it is definitly mapped even with PTI enabled. + */ + jmp paranoid_exit + + /* Switch to the regular task stack */ +.Lfrom_usermode_switch_stack_\@: + idtentry_body safe_stack_\cfunc, has_error_code=1 + +_ASM_NOKPROBE(\asmsym) +SYM_CODE_END(\asmsym) +.endm +#endif + /* * Double fault entry. Straight paranoid. No checks from which context * this comes because for the espfix induced #DF this would do the wrong
arch/x86/include/asm/idtentry.h +44 -0
62 lines not on this page · raw message
arch/x86/include/asm/proto.h +1 -0
@@ -10,6 +10,7 @@ void syscall_init(void);#ifdef CONFIG_X86_64 void entry_SYSCALL_64 (void ); +void entry_SYSCALL_64_safe_stack (void );long do_arch_prctl_64 (struct task_struct *task,int option,unsigned long arg2);#endif
arch/x86/include/asm/traps.h +1 -0
@@ -15,6 +15,7 @@ asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs); asmlinkage __visible notracestruct bad_iret_stack *fixup_bad_iret (struct bad_iret_stack *s);void __init trap_init (void ); +asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *eregs);#endif #ifdef CONFIG_X86_F00F_BUG
arch/x86/kernel/traps.c +45 -0
57 lines not on this page · raw message
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/entry/64: Add entry code for #VC handler
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: a13644f3a53de4e95a7bce6459f834e832ea44c5 Gitweb: https://git.kernel.org/tip/a13644f3a53de4e95a7bce6459f834e832ea44c5 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:46 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/entry/64: Add entry code for #VC handler
The #VC handler needs special entry code because:
1. It runs on an IST stack
2. It needs to be able to handle nested #VC exceptions
To make this work, the entry code is implemented to pretend it doesn't use an IST stack. When entered from user-mode or early SYSCALL entry path it switches to the task stack. If entered from kernel-mode it tries to switch back to the previous stack in the IRET frame.
The stack found in the IRET frame is validated first, and if it is not safe to use it for the #VC handler, the code will switch to a fall-back stack (the #VC2 IST stack). From there, it can cause nested exceptions again.
--- arch/x86/entry/entry_64.S | 80 ++++++++++++++++++++++++++++++++- arch/x86/include/asm/idtentry.h | 44 ++++++++++++++++++- arch/x86/include/asm/proto.h | 1 +- arch/x86/include/asm/traps.h | 1 +- arch/x86/kernel/traps.c | 45 ++++++++++++++++++- 5 files changed, 171 insertions(+)
arch/x86/entry/entry_64.S +80 -0
@@ -101,6 +101,8 @@ SYM_CODE_START(entry_SYSCALL_64) SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp +SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL) + /* Construct struct pt_regs on stack */ pushq $__USER_DS /* pt_regs->ss */ pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2) /* pt_regs->sp */
@@ -446,6 +448,84 @@ _ASM_NOKPROBE(\asmsym) SYM_CODE_END(\asmsym) .endm +#ifdef CONFIG_AMD_MEM_ENCRYPT +/** + * idtentry_vc - Macro to generate entry stub for #VC + * @vector: Vector number + * @asmsym: ASM symbol for the entry point + * @cfunc: C function to be called + * + * The macro emits code to set up the kernel context for #VC. The #VC handler + * runs on an IST stack and needs to be able to cause nested #VC exceptions. + * + * To make this work the #VC entry code tries its best to pretend it doesn't use + * an IST stack by switching to the task stack if coming from user-space (which + * includes early SYSCALL entry path) or back to the stack in the IRET frame if + * entered from kernel-mode. + * + * If entered from kernel-mode the return stack is validated first, and if it is + * not safe to use (e.g. because it points to the entry stack) the #VC handler + * will switch to a fall-back stack (VC2) and call a special handler function. + * + * The macro is only used for one vector, but it is planned to be extended in + * the future for the #HV exception. + */ +.macro idtentry_vc vector asmsym cfunc +SYM_CODE_START(\asmsym) + UNWIND_HINT_IRET_REGS + ASM_CLAC + + /* + * If the entry is from userspace, switch stacks and treat it as + * a normal entry. + */ + testb $3, CS-ORIG_RAX(%rsp) + jnz .Lfrom_usermode_switch_stack_\@ + + /* + * paranoid_entry returns SWAPGS flag for paranoid_exit in EBX. + * EBX == 0 -> SWAPGS, EBX == 1 -> no SWAPGS + */ + call paranoid_entry + + UNWIND_HINT_REGS + + /* + * Switch off the IST stack to make it free for nested exceptions. The + * vc_switch_off_ist() function will switch back to the interrupted + * stack if it is safe to do so. If not it switches to the VC fall-back + * stack. + */ + movq %rsp, %rdi /* pt_regs pointer */ + call vc_switch_off_ist + movq %rax, %rsp /* Switch to new stack */ + + UNWIND_HINT_REGS + + /* Update pt_regs */ + movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/ + movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ + + movq %rsp, %rdi /* pt_regs pointer */ + + call \cfunc + + /* + * No need to switch back to the IST stack. The current stack is either + * identical to the stack in the IRET frame or the VC fall-back stack, + * so it is definitly mapped even with PTI enabled. + */ + jmp paranoid_exit + + /* Switch to the regular task stack */ +.Lfrom_usermode_switch_stack_\@: + idtentry_body safe_stack_\cfunc, has_error_code=1 + +_ASM_NOKPROBE(\asmsym) +SYM_CODE_END(\asmsym) +.endm +#endif + /* * Double fault entry. Straight paranoid. No checks from which context * this comes because for the espfix induced #DF this would do the wrong
arch/x86/include/asm/idtentry.h +44 -0
62 lines not on this page · raw message
arch/x86/include/asm/proto.h +1 -0
@@ -10,6 +10,7 @@ void syscall_init(void);#ifdef CONFIG_X86_64 void entry_SYSCALL_64 (void ); +void entry_SYSCALL_64_safe_stack (void );long do_arch_prctl_64 (struct task_struct *task,int option,unsigned long arg2);#endif
arch/x86/include/asm/traps.h +1 -0
@@ -15,6 +15,7 @@ asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs); asmlinkage __visible notracestruct bad_iret_stack *fixup_bad_iret (struct bad_iret_stack *s);void __init trap_init (void ); +asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *eregs);#endif #ifdef CONFIG_X86_F00F_BUG
arch/x86/kernel/traps.c +45 -0
57 lines not on this page · raw message
Re: [PATCH v7 45/72] x86/entry/64: Add entry code for #VC handler
In reply to Joerg Roedelarch/x86/entry/entry_64.S _ASM_NOKPROBE(\asmsym) · line 519
Quoted 9 lines
arch/x86/entry/entry_64.S +8
-0+ + /* + * No need to switch back to the IST stack. The current stack is either + * identical to the stack in the IRET frame or the VC fall-back stack, + * so it is definitly mapped even with PTI enabled. + */ + jmp paranoid_exit +
Hello
I know we don't enable PTI on AMD, but the above comment doesn't align to the next code.
We assume PTI is enabled as the comments said "even with PTI enabled".
When #VC happens after entry_SYSCALL_64 but before it switches to the kernel CR3. vc_switch_off_ist() will switch the stack to the kernel stack and paranoid_exit can't work when it switches to user CR3 on the kernel stack.
The comment above lost information that the current stack is possible to be the kernel stack which is mapped not user CR3.
Maybe I missed something.
Thanks
Lai
Quoted 16 lines
arch/x86/kernel/traps.c +16
-0+#ifdef CONFIG_AMD_MEM_ENCRYPT +asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *regs) +{ +unsigned long sp, *stack; +struct stack_info info; +struct pt_regs *regs_ret; + +/* +* In the SYSCALL entry path the RSP value comes from user-space - don't +* trust it and switch to the current kernel stack +*/ +if (regs->ip >= (unsigned long )entry_SYSCALL_64 && + regs->ip < (unsigned long )entry_SYSCALL_64_safe_stack) { + sp =this_cpu_read (cpu_current_top_of_stack); + goto sync; + }
Re: [PATCH v7 45/72] x86/entry/64: Add entry code for #VC handler
Hello Lai,
Quoted 17 lines · Lai Jiangshan
arch/x86/entry/entry_64.S _ASM_NOKPROBE(\asmsym) · line 519
Quoted 9 lines
arch/x86/entry/entry_64.S +8
-0+ + /* + * No need to switch back to the IST stack. The current stack is either + * identical to the stack in the IRET frame or the VC fall-back stack, + * so it is definitly mapped even with PTI enabled. + */ + jmp paranoid_exit +Hello
I know we don't enable PTI on AMD, but the above comment doesn't align to the next code.
We assume PTI is enabled as the comments said "even with PTI enabled".
Quoted 8 lines
When #VC happens after
entry_SYSCALL_64but before it switches to the kernel CR3.vc_switch_off_ist()will switch the stack to the kernel stack andparanoid_exitcan't work when it switches to user CR3 on the kernel stack.The comment above lost information that the current stack is possible to be the kernel stack which is mapped not user CR3.
Maybe I missed something.
You are right, the scenario above would cause problems for the current #VC entry code. With SEV-ES an #VC exception can't happen in the early syscall entry code, so I think its the best to update the comment reflecting this.
In the future this might change and then the #VC entry code needs to take care of this case too. Thanks for pointing it out.
Regards,
Joerg
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v7 46/72 x86/sev-es: Add Runtime #VC Exception Handler
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
Add the handlers for #VC exceptions invoked at runtime.
--- arch/x86/include/asm/idtentry.h | 6 + arch/x86/kernel/idt.c | 11 +- arch/x86/kernel/sev-es.c | 246 +++++++++++++++++++++++++++++++- 3 files changed, 255 insertions(+), 8 deletions(-)
arch/x86/include/asm/idtentry.h +6 -0
@@ -318,6 +318,7 @@ static __always_inline void __##func(struct pt_regs *regs)*/ #define DECLARE_IDTENTRY_VC (vector, func) \ DECLARE_IDTENTRY_RAW_ERRORCODE(vector, func); \ + __visible noinstr void ist_##func(struct pt_regs *regs, unsigned long error_code); \ __visible noinstr void safe_stack_##func(struct pt_regs *regs, unsigned long error_code)/**
@@ -608,6 +609,11 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_DB, xenpv_exc_debug);/* #DF */ DECLARE_IDTENTRY_DF (X86_TRAP_DF , exc_double_fault); +/* #VC */ +#ifdef CONFIG_AMD_MEM_ENCRYPT +DECLARE_IDTENTRY_VC (X86_TRAP_VC , exc_vmm_communication); +#endif +#ifdef CONFIG_XEN_PV DECLARE_IDTENTRY_XENCB (X86_TRAP_OTHER , exc_xen_hypervisor_callback);#endif
arch/x86/kernel/idt.c +7 -4
@@ -229,11 +229,14 @@ static const __initconst struct idt_data early_pf_idts[] = {* cpu_init() when the TSS has been initialized. */ static const __initconst struct idt_data ist_idts[] = { -ISTG (X86_TRAP_DB , asm_exc_debug,IST_INDEX_DB ), -ISTG (X86_TRAP_NMI , asm_exc_nmi,IST_INDEX_NMI ), -ISTG (X86_TRAP_DF , asm_exc_double_fault,IST_INDEX_DF ), +ISTG (X86_TRAP_DB , asm_exc_debug,IST_INDEX_DB ), +ISTG (X86_TRAP_NMI , asm_exc_nmi,IST_INDEX_NMI ), +ISTG (X86_TRAP_DF , asm_exc_double_fault,IST_INDEX_DF ), #ifdef CONFIG_X86_MCE -ISTG (X86_TRAP_MC , asm_exc_machine_check,IST_INDEX_MCE ), +ISTG (X86_TRAP_MC , asm_exc_machine_check,IST_INDEX_MCE ), +#endif +#ifdef CONFIG_AMD_MEM_ENCRYPT +ISTG (X86_TRAP_VC , asm_exc_vmm_communication,IST_INDEX_VC ), #endif };
arch/x86/kernel/sev-es.c +242 -4
296 lines not on this page · raw message
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Add a Runtime #VC Exception Handler
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 0786138c78e79343c7b015d77507cbf9d5f15d00 Gitweb: https://git.kernel.org/tip/0786138c78e79343c7b015d77507cbf9d5f15d00 Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:47 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Add a Runtime #VC Exception Handler
Add the handlers for #VC exceptions invoked at runtime.
--- arch/x86/include/asm/idtentry.h | 6 +- arch/x86/kernel/idt.c | 11 +- arch/x86/kernel/sev-es.c | 246 ++++++++++++++++++++++++++++++- 3 files changed, 255 insertions(+), 8 deletions(-)
arch/x86/include/asm/idtentry.h +6 -0
@@ -318,6 +318,7 @@ static __always_inline void __##func(struct pt_regs *regs)*/ #define DECLARE_IDTENTRY_VC (vector, func) \ DECLARE_IDTENTRY_RAW_ERRORCODE(vector, func); \ + __visible noinstr void ist_##func(struct pt_regs *regs, unsigned long error_code); \ __visible noinstr void safe_stack_##func(struct pt_regs *regs, unsigned long error_code)/**
@@ -608,6 +609,11 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_DB, xenpv_exc_debug);/* #DF */ DECLARE_IDTENTRY_DF (X86_TRAP_DF , exc_double_fault); +/* #VC */ +#ifdef CONFIG_AMD_MEM_ENCRYPT +DECLARE_IDTENTRY_VC (X86_TRAP_VC , exc_vmm_communication); +#endif +#ifdef CONFIG_XEN_PV DECLARE_IDTENTRY_XENCB (X86_TRAP_OTHER , exc_xen_hypervisor_callback);#endif
arch/x86/kernel/idt.c +7 -4
@@ -229,11 +229,14 @@ static const __initconst struct idt_data early_pf_idts[] = {* cpu_init() when the TSS has been initialized. */ static const __initconst struct idt_data ist_idts[] = { -ISTG (X86_TRAP_DB , asm_exc_debug,IST_INDEX_DB ), -ISTG (X86_TRAP_NMI , asm_exc_nmi,IST_INDEX_NMI ), -ISTG (X86_TRAP_DF , asm_exc_double_fault,IST_INDEX_DF ), +ISTG (X86_TRAP_DB , asm_exc_debug,IST_INDEX_DB ), +ISTG (X86_TRAP_NMI , asm_exc_nmi,IST_INDEX_NMI ), +ISTG (X86_TRAP_DF , asm_exc_double_fault,IST_INDEX_DF ), #ifdef CONFIG_X86_MCE -ISTG (X86_TRAP_MC , asm_exc_machine_check,IST_INDEX_MCE ), +ISTG (X86_TRAP_MC , asm_exc_machine_check,IST_INDEX_MCE ), +#endif +#ifdef CONFIG_AMD_MEM_ENCRYPT +ISTG (X86_TRAP_VC , asm_exc_vmm_communication,IST_INDEX_VC ), #endif };
arch/x86/kernel/sev-es.c +242 -4
296 lines not on this page · raw message
PATCH v7 47/72 x86/sev-es: Wire up existing #VC exit-code handlers
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Re-use the handlers for CPUID and IOIO caused #VC exceptions in the early boot handler.
--- arch/x86/kernel/sev-es-shared.c | 7 +++---- arch/x86/kernel/sev-es.c | 6 ++++++ 2 files changed, 9 insertions(+), 4 deletions(-)
arch/x86/kernel/sev-es-shared.c +3 -4
@@ -325,8 +325,7 @@ static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)return ES_OK ; } -static enum es_result __maybe_unused -vc_handle_ioio (struct ghcb *ghcb,struct es_em_ctxt *ctxt) +static enum es_result vc_handle_ioio (struct ghcb *ghcb,struct es_em_ctxt *ctxt) {struct pt_regs *regs = ctxt->regs;u64 exit_info_1, exit_info_2;
@@ -434,8 +433,8 @@ vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)return ret; } -static enum es_result __maybe_unused vc_handle_cpuid (struct ghcb *ghcb, -struct es_em_ctxt *ctxt) +static enum es_result vc_handle_cpuid (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) {struct pt_regs *regs = ctxt->regs;u32 cr4 =native_read_cr4 ();
arch/x86/kernel/sev-es.c +6 -0
@@ -441,6 +441,12 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,enum es_result result;switch (exit_code) { +case SVM_EXIT_CPUID : + result =vc_handle_cpuid (ghcb, ctxt); +break ; +case SVM_EXIT_IOIO : + result =vc_handle_ioio (ghcb, ctxt); +break ;default :/* * Unexpected #VC exception
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Wire up existing #VC exit-code handlers
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: d3529bb73f76d0ec8aafaca505226fa0971c1dc9 Gitweb: https://git.kernel.org/tip/d3529bb73f76d0ec8aafaca505226fa0971c1dc9 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:48 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Wire up existing #VC exit-code handlers
Re-use the handlers for CPUID- and IOIO-caused #VC exceptions in the early boot handler.
--- arch/x86/kernel/sev-es-shared.c | 7 +++---- arch/x86/kernel/sev-es.c | 6 ++++++ 2 files changed, 9 insertions(+), 4 deletions(-)
arch/x86/kernel/sev-es-shared.c +3 -4
@@ -325,8 +325,7 @@ static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)return ES_OK ; } -static enum es_result __maybe_unused -vc_handle_ioio (struct ghcb *ghcb,struct es_em_ctxt *ctxt) +static enum es_result vc_handle_ioio (struct ghcb *ghcb,struct es_em_ctxt *ctxt) {struct pt_regs *regs = ctxt->regs;u64 exit_info_1, exit_info_2;
@@ -434,8 +433,8 @@ vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)return ret; } -static enum es_result __maybe_unused vc_handle_cpuid (struct ghcb *ghcb, -struct es_em_ctxt *ctxt) +static enum es_result vc_handle_cpuid (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) {struct pt_regs *regs = ctxt->regs;u32 cr4 =native_read_cr4 ();
arch/x86/kernel/sev-es.c +6 -0
@@ -441,6 +441,12 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,enum es_result result;switch (exit_code) { +case SVM_EXIT_CPUID : + result =vc_handle_cpuid (ghcb, ctxt); +break ; +case SVM_EXIT_IOIO : + result =vc_handle_ioio (ghcb, ctxt); +break ;default :/* * Unexpected #VC exception
PATCH v7 48/72 x86/sev-es: Handle instruction fetches from user-space
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
When a #VC exception is triggered by user-space the instruction decoder needs to read the instruction bytes from user addresses. Enhance vc_decode_insn() to safely fetch kernel and user instructions.
--- arch/x86/kernel/sev-es.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)
arch/x86/kernel/sev-es.c +22 -9
@@ -232,17 +232,30 @@ static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)enum es_result ret;int res; - res =vc_fetch_insn_kernel (ctxt, buffer); -if (unlikely (res == -EFAULT )) { - ctxt->fi.vector =X86_TRAP_PF ; - ctxt->fi.error_code =0 ; - ctxt->fi.cr2 = ctxt->regs->ip; -return ES_EXCEPTION ; +if (user_mode (ctxt->regs)) { + res =insn_fetch_from_user (ctxt->regs, buffer); +if (!res) { + ctxt->fi.vector =X86_TRAP_PF ; + ctxt->fi.error_code =X86_PF_INSTR |X86_PF_USER ; + ctxt->fi.cr2 = ctxt->regs->ip; +return ES_EXCEPTION ; + } + +if (!insn_decode (&ctxt->insn, ctxt->regs, buffer, res)) +return ES_DECODE_FAILED ; + }else { + res =vc_fetch_insn_kernel (ctxt, buffer); +if (res) { + ctxt->fi.vector =X86_TRAP_PF ; + ctxt->fi.error_code =X86_PF_INSTR ; + ctxt->fi.cr2 = ctxt->regs->ip; +return ES_EXCEPTION ; + } + +insn_init (&ctxt->insn, buffer,MAX_INSN_SIZE - res,1 ); +insn_get_length (&ctxt->insn); } -insn_init (&ctxt->insn, buffer,MAX_INSN_SIZE - res,1 ); -insn_get_length (&ctxt->insn); - ret = ctxt->insn.immediate.got ?ES_OK :ES_DECODE_FAILED ;return ret;
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Handle instruction fetches from user-space
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 5e3427a7bc432ed2e5de394ac30f160cc6c37a1f Gitweb: https://git.kernel.org/tip/5e3427a7bc432ed2e5de394ac30f160cc6c37a1f Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:49 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Handle instruction fetches from user-space
When a #VC exception is triggered by user-space, the instruction decoder needs to read the instruction bytes from user addresses. Enhance vc_decode_insn() to safely fetch kernel and user instructions.
--- arch/x86/kernel/sev-es.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)
arch/x86/kernel/sev-es.c +22 -9
@@ -232,17 +232,30 @@ static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)enum es_result ret;int res; - res =vc_fetch_insn_kernel (ctxt, buffer); -if (unlikely (res == -EFAULT )) { - ctxt->fi.vector =X86_TRAP_PF ; - ctxt->fi.error_code =0 ; - ctxt->fi.cr2 = ctxt->regs->ip; -return ES_EXCEPTION ; +if (user_mode (ctxt->regs)) { + res =insn_fetch_from_user (ctxt->regs, buffer); +if (!res) { + ctxt->fi.vector =X86_TRAP_PF ; + ctxt->fi.error_code =X86_PF_INSTR |X86_PF_USER ; + ctxt->fi.cr2 = ctxt->regs->ip; +return ES_EXCEPTION ; + } + +if (!insn_decode (&ctxt->insn, ctxt->regs, buffer, res)) +return ES_DECODE_FAILED ; + }else { + res =vc_fetch_insn_kernel (ctxt, buffer); +if (res) { + ctxt->fi.vector =X86_TRAP_PF ; + ctxt->fi.error_code =X86_PF_INSTR ; + ctxt->fi.cr2 = ctxt->regs->ip; +return ES_EXCEPTION ; + } + +insn_init (&ctxt->insn, buffer,MAX_INSN_SIZE - res,1 ); +insn_get_length (&ctxt->insn); } -insn_init (&ctxt->insn, buffer,MAX_INSN_SIZE - res,1 ); -insn_get_length (&ctxt->insn); - ret = ctxt->insn.immediate.got ?ES_OK :ES_DECODE_FAILED ;return ret;
PATCH v7 49/72 x86/sev-es: Handle MMIO events
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
Add handler for VC exceptions caused by MMIO intercepts. These intercepts come along as nested page faults on pages with reserved bits set.
--- arch/x86/include/uapi/asm/svm.h | 5 + arch/x86/kernel/sev-es.c | 222 ++++++++++++++++++++++++++++++++ 2 files changed, 227 insertions(+)
arch/x86/include/uapi/asm/svm.h +5 -0
@@ -81,6 +81,11 @@#define SVM_EXIT_AVIC_INCOMPLETE_IPI 0x401#define SVM_EXIT_AVIC_UNACCELERATED_ACCESS 0x402 +/* SEV-ES software-defined VMGEXIT events */ +#define SVM_VMGEXIT_MMIO_READ 0x80000001 +#define SVM_VMGEXIT_MMIO_WRITE 0x80000002 +#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff +#define SVM_EXIT_ERR -1#define SVM_EXIT_REASONS \
arch/x86/kernel/sev-es.c +222 -0
240 lines not on this page · raw message
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Handle MMIO events
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 51ee7d6e3d2b70a3f232cceffab5084a2abd6719 Gitweb: https://git.kernel.org/tip/51ee7d6e3d2b70a3f232cceffab5084a2abd6719 Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:50 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Handle MMIO events
Add a handler for #VC exceptions caused by MMIO intercepts. These intercepts come along as nested page faults on pages with reserved bits set.
--- arch/x86/include/uapi/asm/svm.h | 5 +- arch/x86/kernel/sev-es.c | 222 +++++++++++++++++++++++++++++++- 2 files changed, 227 insertions(+)
arch/x86/include/uapi/asm/svm.h +5 -0
@@ -81,6 +81,11 @@#define SVM_EXIT_AVIC_INCOMPLETE_IPI 0x401#define SVM_EXIT_AVIC_UNACCELERATED_ACCESS 0x402 +/* SEV-ES software-defined VMGEXIT events */ +#define SVM_VMGEXIT_MMIO_READ 0x80000001 +#define SVM_VMGEXIT_MMIO_WRITE 0x80000002 +#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff +#define SVM_EXIT_ERR -1#define SVM_EXIT_REASONS \
arch/x86/kernel/sev-es.c +222 -0
240 lines not on this page · raw message
PATCH v7 50/72 x86/sev-es: Handle MMIO String Instructions
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Add handling for emulation the MOVS instruction on MMIO regions, as done by the memcpy_toio() and memcpy_fromio() functions.
--- arch/x86/kernel/sev-es.c | 77 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+)
arch/x86/kernel/sev-es.c +77 -0
@@ -594,6 +594,73 @@ static enum es_result vc_handle_mmio_twobyte_ops(struct ghcb *ghcb,return ret; } +/* +* The MOVS instruction has two memory operands, which raises the +* problem that it is not known whether the access to the source or the +* destination caused the #VC exception (and hence whether an MMIO read +* or write operation needs to be emulated). +* +* Instead of playing games with walking page-tables and trying to guess +* whether the source or destination is an MMIO range, split the move +* into two operations, a read and a write with only one memory operand. +* This will cause a nested #VC exception on the MMIO address which can +* then be handled. +* +* This implementation has the benefit that it also supports MOVS where +* source _and_ destination are MMIO regions. +* +* It will slow MOVS on MMIO down a lot, but in SEV-ES guests it is a +* rare operation. If it turns out to be a performance problem the split +* operations can be moved to memcpy_fromio() and memcpy_toio(). +*/ +static enum es_result vc_handle_mmio_movs (struct es_em_ctxt *ctxt, +unsigned int bytes) +{ +unsigned long ds_base, es_base; +unsigned char *src, *dst; +unsigned char buffer[8 ]; +enum es_result ret; +bool rep; +int off; + + ds_base =insn_get_seg_base (ctxt->regs,INAT_SEG_REG_DS ); + es_base =insn_get_seg_base (ctxt->regs,INAT_SEG_REG_ES ); + +if (ds_base ==-1L || es_base ==-1L ) { + ctxt->fi.vector =X86_TRAP_GP ; + ctxt->fi.error_code =0 ; +return ES_EXCEPTION ; + } + + src = ds_base + (unsigned char *)ctxt->regs->si; + dst = es_base + (unsigned char *)ctxt->regs->di; + + ret =vc_read_mem (ctxt, src, buffer, bytes); +if (ret !=ES_OK ) +return ret; + + ret =vc_write_mem (ctxt, dst, buffer, bytes); +if (ret !=ES_OK ) +return ret; + +if (ctxt->regs->flags &X86_EFLAGS_DF ) + off = -bytes; +else + off = bytes; + + ctxt->regs->si += off; + ctxt->regs->di += off; + + rep =insn_has_rep_prefix (&ctxt->insn); +if (rep) + ctxt->regs->cx -=1 ; + +if (!rep || ctxt->regs->cx ==0 ) +return ES_OK ; +else +return ES_RETRY ; +} +static enum es_result vc_handle_mmio (struct ghcb *ghcb,struct es_em_ctxt *ctxt) {
@@ -655,6 +722,16 @@ static enum es_result vc_handle_mmio(struct ghcb *ghcb,memcpy (reg_data, ghcb->shared_buffer, bytes);break ; +/* MOVS instruction */ +case 0xa4 : + bytes =1 ; + fallthrough; +case 0xa5 : +if (!bytes) + bytes = insn->opnd_bytes; + + ret =vc_handle_mmio_movs (ctxt, bytes); +break ;/* Two-Byte Opcodes */ case 0x0f : ret =vc_handle_mmio_twobyte_ops (ghcb, ctxt);
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Handle MMIO String Instructions
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 0118b604c2c94c6e34982015cfa7891af4764786 Gitweb: https://git.kernel.org/tip/0118b604c2c94c6e34982015cfa7891af4764786 Author: Joerg Roedel <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:51 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Handle MMIO String Instructions
Add handling for emulation of the MOVS instruction on MMIO regions, as done by the memcpy_toio() and memcpy_fromio() functions.
--- arch/x86/kernel/sev-es.c | 77 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 77 insertions(+)
arch/x86/kernel/sev-es.c +77 -0
@@ -594,6 +594,73 @@ static enum es_result vc_handle_mmio_twobyte_ops(struct ghcb *ghcb,return ret; } +/* +* The MOVS instruction has two memory operands, which raises the +* problem that it is not known whether the access to the source or the +* destination caused the #VC exception (and hence whether an MMIO read +* or write operation needs to be emulated). +* +* Instead of playing games with walking page-tables and trying to guess +* whether the source or destination is an MMIO range, split the move +* into two operations, a read and a write with only one memory operand. +* This will cause a nested #VC exception on the MMIO address which can +* then be handled. +* +* This implementation has the benefit that it also supports MOVS where +* source _and_ destination are MMIO regions. +* +* It will slow MOVS on MMIO down a lot, but in SEV-ES guests it is a +* rare operation. If it turns out to be a performance problem the split +* operations can be moved to memcpy_fromio() and memcpy_toio(). +*/ +static enum es_result vc_handle_mmio_movs (struct es_em_ctxt *ctxt, +unsigned int bytes) +{ +unsigned long ds_base, es_base; +unsigned char *src, *dst; +unsigned char buffer[8 ]; +enum es_result ret; +bool rep; +int off; + + ds_base =insn_get_seg_base (ctxt->regs,INAT_SEG_REG_DS ); + es_base =insn_get_seg_base (ctxt->regs,INAT_SEG_REG_ES ); + +if (ds_base ==-1L || es_base ==-1L ) { + ctxt->fi.vector =X86_TRAP_GP ; + ctxt->fi.error_code =0 ; +return ES_EXCEPTION ; + } + + src = ds_base + (unsigned char *)ctxt->regs->si; + dst = es_base + (unsigned char *)ctxt->regs->di; + + ret =vc_read_mem (ctxt, src, buffer, bytes); +if (ret !=ES_OK ) +return ret; + + ret =vc_write_mem (ctxt, dst, buffer, bytes); +if (ret !=ES_OK ) +return ret; + +if (ctxt->regs->flags &X86_EFLAGS_DF ) + off = -bytes; +else + off = bytes; + + ctxt->regs->si += off; + ctxt->regs->di += off; + + rep =insn_has_rep_prefix (&ctxt->insn); +if (rep) + ctxt->regs->cx -=1 ; + +if (!rep || ctxt->regs->cx ==0 ) +return ES_OK ; +else +return ES_RETRY ; +} +static enum es_result vc_handle_mmio (struct ghcb *ghcb,struct es_em_ctxt *ctxt) {
@@ -655,6 +722,16 @@ static enum es_result vc_handle_mmio(struct ghcb *ghcb,memcpy (reg_data, ghcb->shared_buffer, bytes);break ; +/* MOVS instruction */ +case 0xa4 : + bytes =1 ; + fallthrough; +case 0xa5 : +if (!bytes) + bytes = insn->opnd_bytes; + + ret =vc_handle_mmio_movs (ctxt, bytes); +break ;/* Two-Byte Opcodes */ case 0x0f : ret =vc_handle_mmio_twobyte_ops (ghcb, ctxt);
PATCH v7 51/72 x86/sev-es: Handle MSR events
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
Implement a handler for #VC exceptions caused by RDMSR/WRMSR instructions.
--- arch/x86/kernel/sev-es.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
arch/x86/kernel/sev-es.c +28 -0
@@ -393,6 +393,31 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,/* Include code shared with pre-decompression boot stage */ #include "sev-es-shared.c" +static enum es_result vc_handle_msr (struct ghcb *ghcb,struct es_em_ctxt *ctxt) +{ +struct pt_regs *regs = ctxt->regs; +enum es_result ret; +u64 exit_info_1; + +/* Is it a WRMSR? */ + exit_info_1 = (ctxt->insn.opcode.bytes[1 ] ==0x30 ) ?1 :0 ; + +ghcb_set_rcx (ghcb, regs->cx); +if (exit_info_1) { +ghcb_set_rax (ghcb, regs->ax); +ghcb_set_rdx (ghcb, regs->dx); + } + + ret =sev_es_ghcb_hv_call (ghcb, ctxt,SVM_EXIT_MSR , exit_info_1,0 ); + +if ((ret ==ES_OK ) && (!exit_info_1)) { + regs->ax = ghcb->save.rax; + regs->dx = ghcb->save.rdx; + } + +return ret; +} +/* * This function runs on the first #VC exception after the kernel * switched to virtual addresses.
@@ -756,6 +781,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,case SVM_EXIT_IOIO : result =vc_handle_ioio (ghcb, ctxt);break ; +case SVM_EXIT_MSR : + result =vc_handle_msr (ghcb, ctxt); +break ;case SVM_EXIT_NPF : result =vc_handle_mmio (ghcb, ctxt);break ;
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Handle MSR events
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: a4afa6081c88701635e1e20090f953a25f9444e0 Gitweb: https://git.kernel.org/tip/a4afa6081c88701635e1e20090f953a25f9444e0 Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:52 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Handle MSR events
Implement a handler for #VC exceptions caused by RDMSR/WRMSR instructions.
--- arch/x86/kernel/sev-es.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
arch/x86/kernel/sev-es.c +28 -0
@@ -393,6 +393,31 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,/* Include code shared with pre-decompression boot stage */ #include "sev-es-shared.c" +static enum es_result vc_handle_msr (struct ghcb *ghcb,struct es_em_ctxt *ctxt) +{ +struct pt_regs *regs = ctxt->regs; +enum es_result ret; +u64 exit_info_1; + +/* Is it a WRMSR? */ + exit_info_1 = (ctxt->insn.opcode.bytes[1 ] ==0x30 ) ?1 :0 ; + +ghcb_set_rcx (ghcb, regs->cx); +if (exit_info_1) { +ghcb_set_rax (ghcb, regs->ax); +ghcb_set_rdx (ghcb, regs->dx); + } + + ret =sev_es_ghcb_hv_call (ghcb, ctxt,SVM_EXIT_MSR , exit_info_1,0 ); + +if ((ret ==ES_OK ) && (!exit_info_1)) { + regs->ax = ghcb->save.rax; + regs->dx = ghcb->save.rdx; + } + +return ret; +} +/* * This function runs on the first #VC exception after the kernel * switched to virtual addresses.
@@ -756,6 +781,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,case SVM_EXIT_IOIO : result =vc_handle_ioio (ghcb, ctxt);break ; +case SVM_EXIT_MSR : + result =vc_handle_msr (ghcb, ctxt); +break ;case SVM_EXIT_NPF : result =vc_handle_mmio (ghcb, ctxt);break ;
PATCH v7 52/72 x86/sev-es: Handle DR7 read/write events
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
Add code to handle #VC exceptions on DR7 register reads and writes. This is needed early because show_regs() reads DR7 to print it out.
Under SEV-ES there is currently no support for saving/restoring the DRx registers, but software expects to be able to write to the DR7 register. For now, cache the value written to DR7 and return it on read attempts, but do not touch the real hardware DR7.
[ [email protected]: - Adapt to #VC handling framework
- Support early usage ]
--- arch/x86/kernel/sev-es.c | 85 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+)
arch/x86/kernel/sev-es.c +85 -0
@@ -29,6 +29,8 @@#include <asm/traps.h> #include <asm/svm.h> +#define DR7_RESET_VALUE 0x400 +/* For early boot hypervisor communication in SEV-ES enabled guests */ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
@@ -76,6 +78,13 @@ struct sev_es_runtime_data {*/ bool ghcb_active;bool backup_ghcb_active; + +/* +* Cached DR7 value - write it on DR7 writes and return it on reads. +* That value will never make it to the real hardware DR7 as debugging +* is currently unsupported in SEV-ES guests. +*/ +unsigned long dr7; };struct ghcb_state {
@@ -519,6 +528,21 @@ static long *vc_insn_get_reg(struct es_em_ctxt *ctxt)return reg_array + offset; } +static long *vc_insn_get_rm (struct es_em_ctxt *ctxt) +{ +long *reg_array; +int offset; + + reg_array = (long *)ctxt->regs; + offset =insn_get_modrm_rm_off (&ctxt->insn, ctxt->regs); + +if (offset <0 ) +return NULL ; + + offset /=sizeof (long ); + +return reg_array + offset; +}static enum es_result vc_do_mmio (struct ghcb *ghcb,struct es_em_ctxt *ctxt,unsigned int bytes,bool read) {
@@ -768,6 +792,61 @@ static enum es_result vc_handle_mmio(struct ghcb *ghcb,return ret; } +static enum es_result vc_handle_dr7_write (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) +{ +struct sev_es_runtime_data *data =this_cpu_read (runtime_data); +long val, *reg =vc_insn_get_rm (ctxt); +enum es_result ret; + +if (!reg) +return ES_DECODE_FAILED ; + + val = *reg; + +/* Upper 32 bits must be written as zeroes */ +if (val >>32 ) { + ctxt->fi.vector =X86_TRAP_GP ; + ctxt->fi.error_code =0 ; +return ES_EXCEPTION ; + } + +/* Clear out other reserved bits and set bit 10 */ + val = (val &0xffff23ffL ) |BIT (10 ); + +/* Early non-zero writes to DR7 are not supported */ +if (!data && (val & ~DR7_RESET_VALUE )) +return ES_UNSUPPORTED ; + +/* Using a value of 0 for ExitInfo1 means RAX holds the value */ +ghcb_set_rax (ghcb, val); + ret =sev_es_ghcb_hv_call (ghcb, ctxt,SVM_EXIT_WRITE_DR7 ,0 ,0 ); +if (ret !=ES_OK ) +return ret; + +if (data) + data->dr7 = val; + +return ES_OK ; +} + +static enum es_result vc_handle_dr7_read (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) +{ +struct sev_es_runtime_data *data =this_cpu_read (runtime_data); +long *reg =vc_insn_get_rm (ctxt); + +if (!reg) +return ES_DECODE_FAILED ; + +if (data) + *reg = data->dr7; +else + *reg =DR7_RESET_VALUE ; + +return ES_OK ; +} +static enum es_result vc_handle_exitcode (struct es_em_ctxt *ctxt,struct ghcb *ghcb,unsigned long exit_code)
@@ -775,6 +854,12 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,enum es_result result;switch (exit_code) { +case SVM_EXIT_READ_DR7 : + result =vc_handle_dr7_read (ghcb, ctxt); +break ; +case SVM_EXIT_WRITE_DR7 : + result =vc_handle_dr7_write (ghcb, ctxt); +break ;case SVM_EXIT_CPUID : result =vc_handle_cpuid (ghcb, ctxt);break ;
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Handle DR7 read/write events
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 479a7bf5c1f7f14eb37c6d6aeb401562efc1fcad Gitweb: https://git.kernel.org/tip/479a7bf5c1f7f14eb37c6d6aeb401562efc1fcad Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:53 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle DR7 read/write events
Add code to handle #VC exceptions on DR7 register reads and writes. This is needed early because show_regs() reads DR7 to print it out.
Under SEV-ES, there is currently no support for saving/restoring the DRx registers but software expects to be able to write to the DR7 register. For now, cache the value written to DR7 and return it on read attempts, but do not touch the real hardware DR7.
[ [email protected]: - Adapt to #VC handling framework
- Support early usage ]
--- arch/x86/kernel/sev-es.c | 85 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 85 insertions(+)
arch/x86/kernel/sev-es.c +85 -0
@@ -29,6 +29,8 @@#include <asm/traps.h> #include <asm/svm.h> +#define DR7_RESET_VALUE 0x400 +/* For early boot hypervisor communication in SEV-ES enabled guests */ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
@@ -76,6 +78,13 @@ struct sev_es_runtime_data {*/ bool ghcb_active;bool backup_ghcb_active; + +/* +* Cached DR7 value - write it on DR7 writes and return it on reads. +* That value will never make it to the real hardware DR7 as debugging +* is currently unsupported in SEV-ES guests. +*/ +unsigned long dr7; };struct ghcb_state {
@@ -519,6 +528,21 @@ static long *vc_insn_get_reg(struct es_em_ctxt *ctxt)return reg_array + offset; } +static long *vc_insn_get_rm (struct es_em_ctxt *ctxt) +{ +long *reg_array; +int offset; + + reg_array = (long *)ctxt->regs; + offset =insn_get_modrm_rm_off (&ctxt->insn, ctxt->regs); + +if (offset <0 ) +return NULL ; + + offset /=sizeof (long ); + +return reg_array + offset; +}static enum es_result vc_do_mmio (struct ghcb *ghcb,struct es_em_ctxt *ctxt,unsigned int bytes,bool read) {
@@ -768,6 +792,61 @@ static enum es_result vc_handle_mmio(struct ghcb *ghcb,return ret; } +static enum es_result vc_handle_dr7_write (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) +{ +struct sev_es_runtime_data *data =this_cpu_read (runtime_data); +long val, *reg =vc_insn_get_rm (ctxt); +enum es_result ret; + +if (!reg) +return ES_DECODE_FAILED ; + + val = *reg; + +/* Upper 32 bits must be written as zeroes */ +if (val >>32 ) { + ctxt->fi.vector =X86_TRAP_GP ; + ctxt->fi.error_code =0 ; +return ES_EXCEPTION ; + } + +/* Clear out other reserved bits and set bit 10 */ + val = (val &0xffff23ffL ) |BIT (10 ); + +/* Early non-zero writes to DR7 are not supported */ +if (!data && (val & ~DR7_RESET_VALUE )) +return ES_UNSUPPORTED ; + +/* Using a value of 0 for ExitInfo1 means RAX holds the value */ +ghcb_set_rax (ghcb, val); + ret =sev_es_ghcb_hv_call (ghcb, ctxt,SVM_EXIT_WRITE_DR7 ,0 ,0 ); +if (ret !=ES_OK ) +return ret; + +if (data) + data->dr7 = val; + +return ES_OK ; +} + +static enum es_result vc_handle_dr7_read (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) +{ +struct sev_es_runtime_data *data =this_cpu_read (runtime_data); +long *reg =vc_insn_get_rm (ctxt); + +if (!reg) +return ES_DECODE_FAILED ; + +if (data) + *reg = data->dr7; +else + *reg =DR7_RESET_VALUE ; + +return ES_OK ; +} +static enum es_result vc_handle_exitcode (struct es_em_ctxt *ctxt,struct ghcb *ghcb,unsigned long exit_code)
@@ -775,6 +854,12 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,enum es_result result;switch (exit_code) { +case SVM_EXIT_READ_DR7 : + result =vc_handle_dr7_read (ghcb, ctxt); +break ; +case SVM_EXIT_WRITE_DR7 : + result =vc_handle_dr7_write (ghcb, ctxt); +break ;case SVM_EXIT_CPUID : result =vc_handle_cpuid (ghcb, ctxt);break ;
PATCH v7 53/72 x86/sev-es: Handle WBINVD Events
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
Implement a handler for #VC exceptions caused by WBINVD instructions.
--- arch/x86/kernel/sev-es.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/x86/kernel/sev-es.c +9 -0
@@ -847,6 +847,12 @@ static enum es_result vc_handle_dr7_read(struct ghcb *ghcb,return ES_OK ; } +static enum es_result vc_handle_wbinvd (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) +{ +return sev_es_ghcb_hv_call (ghcb, ctxt,SVM_EXIT_WBINVD ,0 ,0 ); +} +static enum es_result vc_handle_exitcode (struct es_em_ctxt *ctxt,struct ghcb *ghcb,unsigned long exit_code)
@@ -869,6 +875,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,case SVM_EXIT_MSR : result =vc_handle_msr (ghcb, ctxt);break ; +case SVM_EXIT_WBINVD : + result =vc_handle_wbinvd (ghcb, ctxt); +break ;case SVM_EXIT_NPF : result =vc_handle_mmio (ghcb, ctxt);break ;
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Handle WBINVD Events
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: a14a92fc4b420ae6713800a8ae8c3686c1a1fa20 Gitweb: https://git.kernel.org/tip/a14a92fc4b420ae6713800a8ae8c3686c1a1fa20 Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:54 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle WBINVD Events
Implement a handler for #VC exceptions caused by WBINVD instructions.
--- arch/x86/kernel/sev-es.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/x86/kernel/sev-es.c +9 -0
@@ -847,6 +847,12 @@ static enum es_result vc_handle_dr7_read(struct ghcb *ghcb,return ES_OK ; } +static enum es_result vc_handle_wbinvd (struct ghcb *ghcb, +struct es_em_ctxt *ctxt) +{ +return sev_es_ghcb_hv_call (ghcb, ctxt,SVM_EXIT_WBINVD ,0 ,0 ); +} +static enum es_result vc_handle_exitcode (struct es_em_ctxt *ctxt,struct ghcb *ghcb,unsigned long exit_code)
@@ -869,6 +875,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,case SVM_EXIT_MSR : result =vc_handle_msr (ghcb, ctxt);break ; +case SVM_EXIT_WBINVD : + result =vc_handle_wbinvd (ghcb, ctxt); +break ;case SVM_EXIT_NPF : result =vc_handle_mmio (ghcb, ctxt);break ;
PATCH v7 54/72 x86/sev-es: Handle RDTSC(P) Events
In reply to Joerg RoedelFrom: Tom Lendacky <[email protected]>
Implement a handler for #VC exceptions caused by RDTSC and RDTSCP instructions. Also make it available in the pre-decompression stage because the KASLR code used RDTSC/RDTSCP to gather entropy and some hypervisors intercept these instructions.
[ [email protected]: - Adapt to #VC handling infrastructure
- Make it available early ]
--- arch/x86/boot/compressed/sev-es.c | 4 ++++ arch/x86/kernel/sev-es-shared.c | 23 +++++++++++++++++++++++ arch/x86/kernel/sev-es.c | 4 ++++ 3 files changed, 31 insertions(+)
arch/x86/boot/compressed/sev-es.c +4 -0
@@ -181,6 +181,10 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code) goto finish;switch (exit_code) { +case SVM_EXIT_RDTSC : +case SVM_EXIT_RDTSCP : + result =vc_handle_rdtsc (boot_ghcb, &ctxt, exit_code); +break ;case SVM_EXIT_IOIO : result =vc_handle_ioio (boot_ghcb, &ctxt);break ;
arch/x86/kernel/sev-es-shared.c +23 -0
@@ -467,3 +467,26 @@ static enum es_result vc_handle_cpuid(struct ghcb *ghcb,return ES_OK ; } + +static enum es_result vc_handle_rdtsc (struct ghcb *ghcb, +struct es_em_ctxt *ctxt, +unsigned long exit_code) +{ +bool rdtscp = (exit_code ==SVM_EXIT_RDTSCP ); +enum es_result ret; + + ret =sev_es_ghcb_hv_call (ghcb, ctxt, exit_code,0 ,0 ); +if (ret !=ES_OK ) +return ret; + +if (!(ghcb_rax_is_valid (ghcb) &&ghcb_rdx_is_valid (ghcb) && + (!rdtscp ||ghcb_rcx_is_valid (ghcb)))) +return ES_VMM_ERROR ; + + ctxt->regs->ax = ghcb->save.rax; + ctxt->regs->dx = ghcb->save.rdx; +if (rdtscp) + ctxt->regs->cx = ghcb->save.rcx; + +return ES_OK ; +}
arch/x86/kernel/sev-es.c +4 -0
@@ -866,6 +866,10 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,case SVM_EXIT_WRITE_DR7 : result =vc_handle_dr7_write (ghcb, ctxt);break ; +case SVM_EXIT_RDTSC : +case SVM_EXIT_RDTSCP : + result =vc_handle_rdtsc (ghcb, ctxt, exit_code); +break ;case SVM_EXIT_CPUID : result =vc_handle_cpuid (ghcb, ctxt);break ;
Signature · 6 lines
2.28.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[tip: x86/seves] x86/sev-es: Handle RDTSC(P) Events
The following commit has been merged into the x86/seves branch of tip:
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 4711e7acaa125d8cc242f06e1f4d6c74e177454b Gitweb: https://git.kernel.org/tip/4711e7acaa125d8cc242f06e1f4d6c74e177454b Author: Tom Lendacky <[email protected]> AuthorDate: Mon, 07 Sep 2020 15:15:55 +02:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle RDTSC(P) Events
Implement a handler for #VC exceptions caused by RDTSC and RDTSCP instructions. Also make it available in the pre-decompression stage because the KASLR code uses RDTSC/RDTSCP to gather entropy and some hypervisors intercept these instructions.
[ [email protected]: - Adapt to #VC handling infrastructure
- Make it available early ]
--- arch/x86/boot/compressed/sev-es.c | 4 ++++ arch/x86/kernel/sev-es-shared.c | 23 +++++++++++++++++++++++ arch/x86/kernel/sev-es.c | 4 ++++ 3 files changed, 31 insertions(+)
arch/x86/boot/compressed/sev-es.c +4 -0
@@ -181,6 +181,10 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code) goto finish;switch (exit_code) { +case SVM_EXIT_RDTSC : +case SVM_EXIT_RDTSCP : + result =vc_handle_rdtsc (boot_ghcb, &ctxt, exit_code); +break ;case SVM_EXIT_IOIO : result =vc_handle_ioio (boot_ghcb, &ctxt);break ;
arch/x86/kernel/sev-es-shared.c +23 -0
@@ -467,3 +467,26 @@ static enum es_result vc_handle_cpuid(struct ghcb *ghcb,return ES_OK ; } + +static enum es_result vc_handle_rdtsc (struct ghcb *ghcb, +struct es_em_ctxt *ctxt, +unsigned long exit_code) +{ +bool rdtscp = (exit_code ==SVM_EXIT_RDTSCP ); +enum es_result ret; + + ret =sev_es_ghcb_hv_call (ghcb, ctxt, exit_code,0 ,0 ); +if (ret !=ES_OK ) +return ret; + +if (!(ghcb_rax_is_valid (ghcb) &&ghcb_rdx_is_valid (ghcb) && + (!rdtscp ||ghcb_rcx_is_valid (ghcb)))) +return ES_VMM_ERROR ; + + ctxt->regs->ax = ghcb->save.rax; + ctxt->regs->dx = ghcb->save.rdx; +if (rdtscp) + ctxt->regs->cx = ghcb->save.rcx; + +return ES_OK ; +}
arch/x86/kernel/sev-es.c +4 -0
@@ -866,6 +866,10 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,case SVM_EXIT_WRITE_DR7 : result =vc_handle_dr7_write (ghcb, ctxt);break ; +case SVM_EXIT_RDTSC : +case SVM_EXIT_RDTSCP : + result =vc_handle_rdtsc (ghcb, ctxt, exit_code); +break ;case SVM_EXIT_CPUID : result =vc_handle_cpuid (ghcb, ctxt);break ;