Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C400CC433E7 for ; Thu, 8 Oct 2020 08:43:37 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 42F84204EC for ; Thu, 8 Oct 2020 08:43:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="Lcgljvm0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 42F84204EC Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.4361.11553 (Exim 4.92) (envelope-from ) id 1kQRWA-0006YX-Ar; Thu, 08 Oct 2020 08:43:18 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 4361.11553; Thu, 08 Oct 2020 08:43:18 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kQRWA-0006YQ-7Y; Thu, 08 Oct 2020 08:43:18 +0000 Received: by outflank-mailman (input) for mailman id 4361; Thu, 08 Oct 2020 08:43:17 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kQRW9-0006YK-E6 for xen-devel@lists.xenproject.org; Thu, 08 Oct 2020 08:43:17 +0000 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id aa018b75-b839-4962-849f-ef07e73e739a; Thu, 08 Oct 2020 08:43:14 +0000 (UTC) Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kQRW9-0006YK-E6 for xen-devel@lists.xenproject.org; Thu, 08 Oct 2020 08:43:17 +0000 X-Inumbo-ID: aa018b75-b839-4962-849f-ef07e73e739a Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id aa018b75-b839-4962-849f-ef07e73e739a; Thu, 08 Oct 2020 08:43:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1602146595; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=hmfaHDqcevzm6TqwFYb1N7ROXvDqhIsFeaUBR+owTgQ=; b=Lcgljvm0EehTjS+UgyJWGwCcCnhe3Qk/Am/VjWDXJWkE/Q1UMYJXDHUj 20atrpLqqeS1BU7k5hiSBA7CGkdKOab0zfrFaAOHTspZJ2sFwWRBWg2WQ Sx78JPrFryxDj9FgOMdaeBURgurHcj4h9su949yMF3Hmax+6WfIYkNv+N k=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: T00na4NxHkPyH2ebQCtGS8s6trKNpm+AH5Ip44klzrTCr1jYCODX9BhpN80rTcAPeE8rVKFtJd LGoOPZzk7VoFShWFTNYzMWGpYjvdnJXIei2/+jHjU36PCI5GiNnWfFbCOA1rkzHwraBlMqXMkh 2nKCVqmjLV2YGuy479FWHuMNqfS0slh0ydCepFFLXHQI+34ZGJOi5ubUYwej2D6vCFkHjxZFH9 4LnPPTVUeAK3BEr3Cv3FreH8hjIzwOb1qBLU79cGUg3GC+uJPV5Yhii0875hN0Noc1fMhGqqlZ gHw= X-SBRS: None X-MesageID: 28888532 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.77,350,1596513600"; d="scan'208";a="28888532" Date: Thu, 8 Oct 2020 10:43:06 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Juergen Gross CC: , Jan Beulich , "Andrew Cooper" , Wei Liu , George Dunlap , Ian Jackson , Julien Grall , Stefano Stabellini Subject: Re: [PATCH v2 1/2] xen/x86: add nmi continuation framework Message-ID: <20201008084306.GJ19254@Air-de-Roger> References: <20201007133011.18871-1-jgross@suse.com> <20201007133011.18871-2-jgross@suse.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline In-Reply-To: <20201007133011.18871-2-jgross@suse.com> X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To FTLPEX02CL06.citrite.net (10.13.108.179) On Wed, Oct 07, 2020 at 03:30:10PM +0200, Juergen Gross wrote: > Actions in NMI context are rather limited as e.g. locking is rather > fragile. > > Add a generic framework to continue processing in softirq context after > leaving NMI processing. This is working for NMIs happening in guest > context as NMI exit handling will issue an IPI to itself in case a > softirq is pending, resulting in the continuation running before the > guest gets control again. There's already kind of a nmi callback framework using nmi_callback. I assume that moving this existing callback code to be executed in softirq context is not suitable because that would be past the execution of an iret instruction? Might be worth mentioning in the commit message. > Signed-off-by: Juergen Gross > --- > xen/arch/x86/traps.c | 37 +++++++++++++++++++++++++++++++++++++ > xen/include/asm-x86/nmi.h | 8 +++++++- > xen/include/xen/softirq.h | 5 ++++- > 3 files changed, 48 insertions(+), 2 deletions(-) > > diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c > index bc5b8f8ea3..f433fe5acb 100644 > --- a/xen/arch/x86/traps.c > +++ b/xen/arch/x86/traps.c > @@ -1799,6 +1799,42 @@ void unset_nmi_callback(void) > nmi_callback = dummy_nmi_callback; > } > > +static DEFINE_PER_CPU(void (*)(void *), nmi_cont_func); > +static DEFINE_PER_CPU(void *, nmi_cont_par); > + > +static void nmi_cont_softirq(void) > +{ > + unsigned int cpu = smp_processor_id(); > + void (*func)(void *par) = per_cpu(nmi_cont_func, cpu); > + void *par = per_cpu(nmi_cont_par, cpu); I think you can use this_cpu here and below, and avoid having to worry about the processor id at all? Also less likely to mess up and assign a NMI callback to a wrong CPU. > + > + /* Reads must be done before following write (local cpu ordering only). */ > + barrier(); Is this added because of the usage of RELOC_HIDE by per_cpu? > + > + per_cpu(nmi_cont_func, cpu) = NULL; Since we are using RELOC_HIDE, doesn't it imply that the compiler cannot reorder this, since it has no idea what variable we are accessing? > + > + if ( func ) > + func(par); > +} > + > +int set_nmi_continuation(void (*func)(void *par), void *par) > +{ > + unsigned int cpu = smp_processor_id(); > + > + if ( per_cpu(nmi_cont_func, cpu) ) > + { > + printk("Trying to set NMI continuation while still one active!\n"); > + return -EBUSY; > + } > + > + per_cpu(nmi_cont_func, cpu) = func; > + per_cpu(nmi_cont_par, cpu) = par; > + > + raise_softirq(NMI_CONT_SOFTIRQ); > + > + return 0; > +} > + > void do_device_not_available(struct cpu_user_regs *regs) > { > #ifdef CONFIG_PV > @@ -2132,6 +2168,7 @@ void __init trap_init(void) > > cpu_init(); > > + open_softirq(NMI_CONT_SOFTIRQ, nmi_cont_softirq); > open_softirq(PCI_SERR_SOFTIRQ, pci_serr_softirq); > } > > diff --git a/xen/include/asm-x86/nmi.h b/xen/include/asm-x86/nmi.h > index a288f02a50..da40fb6599 100644 > --- a/xen/include/asm-x86/nmi.h > +++ b/xen/include/asm-x86/nmi.h > @@ -33,5 +33,11 @@ nmi_callback_t *set_nmi_callback(nmi_callback_t *callback); > void unset_nmi_callback(void); > > DECLARE_PER_CPU(unsigned int, nmi_count); > - > + > +/** > + * set_nmi_continuation > + * > + * Schedule a function to be started in softirq context after NMI handling. > + */ > +int set_nmi_continuation(void (*func)(void *par), void *par); I would introduce a type for the nmi callback, as I think it's easier than having to retype the prototype everywhere: typedef void nmi_continuation_t(void *); Thanks, Roger.