Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00A6EC433E7 for ; Thu, 3 Sep 2020 09:18:05 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id A36602071B for ; Thu, 3 Sep 2020 09:18:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="Z3AwNlaf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A36602071B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 4A8794B268; Thu, 3 Sep 2020 05:18:04 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T9wPlK-rKIRB; Thu, 3 Sep 2020 05:18:03 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0271F4B214; Thu, 3 Sep 2020 05:18:03 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id F260A4B1C9 for ; Thu, 3 Sep 2020 05:18:00 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PieLnISrldQG for ; Thu, 3 Sep 2020 05:17:59 -0400 (EDT) Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id AC3A34B183 for ; Thu, 3 Sep 2020 05:17:59 -0400 (EDT) Received: by mail-ej1-f54.google.com with SMTP id a15so2780071ejf.11 for ; Thu, 03 Sep 2020 02:17:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kA/M5uo1pgexmorks7VPWASAyNAWS3vx6/WPRm8af9o=; b=Z3AwNlaf6J5hW5xgtHRnIRpOFoT8tF/HLwZ8GJ0e5cvG+dJo2ZV8Td+XYGpSz0PDPU yZ/Eba/KurKSLu2xTfCSNv8w3CSqYugtsRgbJhG+wLL490SlO28pS2CRnJpJnDP/DwCo +E/F0+gL8wXjD44TOvVWyXQD/GZW/uPbFgKF9ERR7IXDZMkzS0sEOUJoNgA1H6/Byx4y NSwW2yoSQkQeLoA3uSwlSJJ/rckcuNL/6FNJHtZa16XVwYykjh/5UAxiCobCQ1amihKK R0W7G3p7XxI5OMHuVyUdK3LZNxzxpOJDnETrkKBJ1UwfzsUT0SGPh94z2Picpt7x6ZPk rD2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kA/M5uo1pgexmorks7VPWASAyNAWS3vx6/WPRm8af9o=; b=EjtA8TQVdNuXm4Prdf48rS4dYYhc59O949Ex/v321qpczHfyHDPAm5UqJgAZCj1WBc ifB2k2lAj2BOZfg1hHumAVVetw+fUfk63th5eLMIVZpw/TdKscZjbruEG1/ZMry3G2Pa VzK1rH7kw0X2HbDA2AqjXx4OC+yo7G+TJa2c5ROWo6tmaPivKvxO236CuXMJOV7YOnJl kEUyfEmNdjFnBKE4D/qmPOAD8267bVGKBTES6sp1yYvtRhdJtvThe5slROxkcyfxDiwy Zs7i9L0wPKx1L0UlE0xBJQ0529FNnv5a+iLYO8NpYKJB468fMCHuXPhyRst9EMfpd7w2 Utjg== X-Gm-Message-State: AOAM5302bXgKX0BdbK+5HSdM3pMvPj4XrQ6eDuPKA6XkIUkoO7obredl /XATOj/X9gPE0E/bOIEESqVGsg== X-Google-Smtp-Source: ABdhPJxmXDw453SzS5U6zfFul53RqVj1EsU1USDwBl8Q/g0xulot4lLcRQNNUqR52/MOFV2rbWsbIQ== X-Received: by 2002:a17:906:2f02:: with SMTP id v2mr1041974eji.465.1599124678549; Thu, 03 Sep 2020 02:17:58 -0700 (PDT) Received: from localhost (dynamic-2a00-1028-919a-a06e-64ac-0036-822c-68d3.ipv6.broadband.iol.cz. [2a00:1028:919a:a06e:64ac:36:822c:68d3]) by smtp.gmail.com with ESMTPSA id z23sm2614737eja.29.2020.09.03.02.17.56 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 03 Sep 2020 02:17:57 -0700 (PDT) From: David Brazdil To: Marc Zyngier , Catalin Marinas , Will Deacon , Dennis Zhou , Tejun Heo , Christoph Lameter , Arnd Bergmann Subject: [PATCH v2 09/10] kvm: arm64: Set up hyp percpu data for nVHE Date: Thu, 3 Sep 2020 11:17:11 +0200 Message-Id: <20200903091712.46456-10-dbrazdil@google.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200903091712.46456-1-dbrazdil@google.com> References: <20200903091712.46456-1-dbrazdil@google.com> MIME-Version: 1.0 Cc: linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Add hyp percpu section to linker script and rename the corresponding ELF sections of hyp/nvhe object files. This moves all nVHE-specific percpu variables to the new hyp percpu section. Allocate sufficient amount of memory for all percpu hyp regions at global KVM init time and create corresponding hyp mappings. The base addresses of hyp percpu regions are kept in a dynamically allocated array in the kernel. Add NULL checks in PMU event-reset code as it may run before KVM memory is initialized. Signed-off-by: David Brazdil --- arch/arm64/include/asm/kvm_asm.h | 19 +++++++++-- arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/vmlinux.lds.S | 10 ++++++ arch/arm64/kvm/arm.c | 56 +++++++++++++++++++++++++++++-- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 5 +++ arch/arm64/kvm/pmu.c | 5 ++- 6 files changed, 90 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 2b89817cdb01..c87111c25d9e 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -72,8 +72,23 @@ #define CHOOSE_VHE_SYM(sym) sym #define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym) -#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym)) -#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu) +/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */ +extern unsigned long *kvm_arm_hyp_percpu_base; + +/* + * Compute pointer to a symbol defined in nVHE percpu region. + * Returns NULL if percpu memory has not been allocated yet. + */ +#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id()) +#define per_cpu_ptr_nvhe(sym, cpu) \ + ({ \ + unsigned long base, off; \ + base = kvm_arm_hyp_percpu_base \ + ? kvm_arm_hyp_percpu_base[cpu] : 0; \ + off = (unsigned long)&kvm_nvhe_sym(sym) - \ + (unsigned long)&kvm_nvhe_sym(__per_cpu_start); \ + base ? (typeof(kvm_nvhe_sym(sym))*)(base + off) : NULL; \ + }) #ifndef __KVM_NVHE_HYPERVISOR__ /* diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 3994169985ef..5062553a6847 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -18,5 +18,6 @@ extern char __exittext_begin[], __exittext_end[]; extern char __irqentry_text_start[], __irqentry_text_end[]; extern char __mmuoff_data_start[], __mmuoff_data_end[]; extern char __entry_tramp_text_start[], __entry_tramp_text_end[]; +extern char __kvm_nvhe___per_cpu_start[], __kvm_nvhe___per_cpu_end[]; #endif /* __ASM_SECTIONS_H */ diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 7cba7623fcec..5904a4de9f40 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -15,6 +15,9 @@ #include "image.h" +#define __CONCAT3(x, y, z) x ## y ## z +#define CONCAT3(x, y, z) __CONCAT3(x, y, z) + OUTPUT_ARCH(aarch64) ENTRY(_text) @@ -191,6 +194,13 @@ SECTIONS PERCPU_SECTION(L1_CACHE_BYTES) + /* KVM nVHE per-cpu section */ + #undef PERCPU_SECTION_NAME + #undef PERCPU_SYMBOL_NAME + #define PERCPU_SECTION_NAME(suffix) CONCAT3(.hyp, PERCPU_SECTION_BASE_NAME, suffix) + #define PERCPU_SYMBOL_NAME(name) __kvm_nvhe_ ## name + PERCPU_SECTION(L1_CACHE_BYTES) + .rela.dyn : ALIGN(8) { *(.rela .rela*) } diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 8a1fcf4dffca..df7d133056ce 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -47,6 +47,7 @@ __asm__(".arch_extension virt"); #endif static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); +unsigned long *kvm_arm_hyp_percpu_base; /* The VMID used in the VTTBR */ static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1); @@ -1255,6 +1256,15 @@ long kvm_arch_vm_ioctl(struct file *filp, } } +#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu)) +#define kvm_hyp_percpu_array_size (nr_cpu_ids * sizeof(*kvm_arm_hyp_percpu_base)) +#define kvm_hyp_percpu_array_order (get_order(kvm_hyp_percpu_array_size)) +#define kvm_hyp_percpu_start (CHOOSE_NVHE_SYM(__per_cpu_start)) +#define kvm_hyp_percpu_end (CHOOSE_NVHE_SYM(__per_cpu_end)) +#define kvm_hyp_percpu_size ((unsigned long)kvm_hyp_percpu_end - \ + (unsigned long)kvm_hyp_percpu_start) +#define kvm_hyp_percpu_order (kvm_hyp_percpu_size ? get_order(kvm_hyp_percpu_size) : 0) + static void cpu_init_hyp_mode(void) { phys_addr_t pgd_ptr; @@ -1270,8 +1280,8 @@ static void cpu_init_hyp_mode(void) * kernel's mapping to the linear mapping, and store it in tpidr_el2 * so that we can use adr_l to access per-cpu variables in EL2. */ - tpidr_el2 = ((unsigned long)this_cpu_ptr(&kvm_host_data) - - (unsigned long)kvm_ksym_ref(&kvm_host_data)); + tpidr_el2 = (unsigned long)this_cpu_ptr_nvhe(__per_cpu_start) - + (unsigned long)kvm_ksym_ref(kvm_hyp_percpu_start); pgd_ptr = kvm_mmu_get_httbr(); hyp_stack_ptr = __this_cpu_read(kvm_arm_hyp_stack_page) + PAGE_SIZE; @@ -1503,8 +1513,11 @@ static void teardown_hyp_mode(void) int cpu; free_hyp_pgds(); - for_each_possible_cpu(cpu) + for_each_possible_cpu(cpu) { free_hyp_page(per_cpu(kvm_arm_hyp_stack_page, cpu)); + free_hyp_pages(kvm_hyp_percpu_base(cpu), kvm_hyp_percpu_order); + } + free_hyp_pages((unsigned long)kvm_arm_hyp_percpu_base, kvm_hyp_percpu_array_order); } /** @@ -1537,6 +1550,28 @@ static int init_hyp_mode(void) per_cpu(kvm_arm_hyp_stack_page, cpu) = stack_page; } + /* + * Allocate and initialize pages for Hypervisor-mode percpu regions. + */ + kvm_arm_hyp_percpu_base = (unsigned long *)alloc_hyp_pages( + GFP_KERNEL | __GFP_ZERO, kvm_hyp_percpu_array_order); + if (!kvm_arm_hyp_percpu_base) { + err = -ENOMEM; + goto out_err; + } + for_each_possible_cpu(cpu) { + unsigned long percpu_base; + + percpu_base = alloc_hyp_pages(GFP_KERNEL, kvm_hyp_percpu_order); + if (!percpu_base) { + err = -ENOMEM; + goto out_err; + } + + memcpy((void *)percpu_base, kvm_hyp_percpu_start, kvm_hyp_percpu_size); + kvm_arm_hyp_percpu_base[cpu] = percpu_base; + } + /* * Map the Hyp-code called directly from the host */ @@ -1581,6 +1616,21 @@ static int init_hyp_mode(void) } } + /* + * Map Hyp percpu pages + */ + for_each_possible_cpu(cpu) { + char *percpu_begin = (char *)kvm_arm_hyp_percpu_base[cpu]; + char *percpu_end = percpu_begin + PAGE_ALIGN(kvm_hyp_percpu_size); + + err = create_hyp_mappings(percpu_begin, percpu_end, PAGE_HYP); + + if (err) { + kvm_err("Cannot map hyp percpu region\n"); + goto out_err; + } + } + for_each_possible_cpu(cpu) { kvm_host_data_t *cpu_data; diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S index aaa0ce133a32..7d8c3fa004f4 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S @@ -11,4 +11,9 @@ SECTIONS { HYP_SECTION(.text) + HYP_SECTION(.data..percpu) + HYP_SECTION(.data..percpu..first) + HYP_SECTION(.data..percpu..page_aligned) + HYP_SECTION(.data..percpu..read_mostly) + HYP_SECTION(.data..percpu..shared_aligned) } diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c index 6d80ffe1ebfc..cd653091f213 100644 --- a/arch/arm64/kvm/pmu.c +++ b/arch/arm64/kvm/pmu.c @@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) { struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data); - if (!kvm_pmu_switch_needed(attr)) + if (!ctx || !kvm_pmu_switch_needed(attr)) return; if (!attr->exclude_host) @@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr) { struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data); + if (!ctx) + return; + ctx->pmu_events.events_host &= ~clr; ctx->pmu_events.events_guest &= ~clr; } -- 2.28.0.402.g5ffc5be6b7-goog _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm