Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DEE2C43463 for ; Mon, 21 Sep 2020 13:05:10 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CFED621789 for ; Mon, 21 Sep 2020 13:05:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="DPCEVABo" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CFED621789 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kKLUy-0007v5-BH; Mon, 21 Sep 2020 13:04:52 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kKLUw-0007v0-P8 for xen-devel@lists.xenproject.org; Mon, 21 Sep 2020 13:04:50 +0000 X-Inumbo-ID: 242a9cae-0e4e-4801-91da-c9e27fbee8ff Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 242a9cae-0e4e-4801-91da-c9e27fbee8ff; Mon, 21 Sep 2020 13:04:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1600693490; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=vSFMGs5TuuPLoqwLBeEOXLbo/pehHspz/Z8ArxMPTXU=; b=DPCEVABo8z+zh/OAk/rcL4zNGDh/z224UCOj2iUjfOqk13UyH/J7tmpD O2uJkqH1NqfttsDhirjjilrhwYMzcoXcoA8cNmooZOyBMggzMbkPSnL9n lvP+prDaF3d50Sh4r0jjDnNbecKS3M5JBBGDP1WSXTEHmRu4BOacuX0G1 4=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: OCnyWsl6IA+ZBlRDJDGwkJYYVR7K0r9uZjeSbxLVuvpj6sMuJB5CZsSVGV+rdo6aH8aBrBwEy8 twtkLrFhjVEhtVBTh82Es4DRTTCzcAf6s5a2zCUL5dXsAXNBihtsV+1aL6be7b+DhoPleSVykY Y1KxwzuNAVM3usbdtYrjdpRZf54mT8EDrTMZjgHNcf7YQQB0gusQDK3B5uMVVDPj8fSowY6hBJ vPjQ/pkpfUKR3i9ZUXdENTC2lrXZCRqgXGFmuqzWAetHgoEoZ2SHaRtcTHJLzGkM6CgyCg8V0G jLs= X-SBRS: 2.7 X-MesageID: 27131129 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.77,286,1596513600"; d="scan'208";a="27131129" From: Andrew Cooper To: Xen-devel CC: Andrew Cooper , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Wei Liu , Ian Jackson Subject: [PATCH] x86: Use LOCK ADD instead of MFENCE for smp_mb() Date: Mon, 21 Sep 2020 14:04:23 +0100 Message-ID: <20200921130423.8035-1-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" MFENCE is overly heavyweight for SMP semantics on WB memory, because it also orders weaker cached writes, and flushes the WC buffers. This technique was used as an optimisation in Java[1], and later adopted by Linux[2] where it was measured to have a 60% performance improvement in VirtIO benchmarks. The stack is used because it is hot in the L1 cache, and a -4 offset is used to avoid creating a false data dependency on live data. (For 64bit userspace, the offset needs to be under the red zone to avoid false dependences). Fix up the 32 bit definitions in HVMLoader and libxc to avoid a false data dependency. [1] https://shipilev.net/blog/2014/on-the-fence-with-dependencies/ [2] https://git.kernel.org/torvalds/c/450cbdd0125cfa5d7bbf9e2a6b6961cc48d29730 Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Roger Pau Monné CC: Wei Liu CC: Ian Jackson --- tools/firmware/hvmloader/util.h | 2 +- tools/libs/ctrl/include/xenctrl.h | 4 ++-- xen/include/asm-x86/system.h | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tools/firmware/hvmloader/util.h b/tools/firmware/hvmloader/util.h index 31889de634..4f0baade0e 100644 --- a/tools/firmware/hvmloader/util.h +++ b/tools/firmware/hvmloader/util.h @@ -133,7 +133,7 @@ static inline void cpu_relax(void) #define barrier() asm volatile ( "" : : : "memory" ) #define rmb() barrier() #define wmb() barrier() -#define mb() asm volatile ( "lock; addl $0,0(%%esp)" : : : "memory" ) +#define mb() asm volatile ( "lock addl $0, -4(%%esp)" ::: "memory" ) /* * Divide a 64-bit dividend by a 32-bit divisor. diff --git a/tools/libs/ctrl/include/xenctrl.h b/tools/libs/ctrl/include/xenctrl.h index 73e9535fc8..1d9f514302 100644 --- a/tools/libs/ctrl/include/xenctrl.h +++ b/tools/libs/ctrl/include/xenctrl.h @@ -68,11 +68,11 @@ #define xen_barrier() asm volatile ( "" : : : "memory") #if defined(__i386__) -#define xen_mb() asm volatile ( "lock; addl $0,0(%%esp)" : : : "memory" ) +#define xen_mb() asm volatile ( "lock addl $0, -4(%%esp)" ::: "memory" ) #define xen_rmb() xen_barrier() #define xen_wmb() xen_barrier() #elif defined(__x86_64__) -#define xen_mb() asm volatile ( "mfence" : : : "memory") +#define xen_mb() asm volatile ( "lock addl $0, -128(%%rsp)" ::: "memory" ) #define xen_rmb() xen_barrier() #define xen_wmb() xen_barrier() #elif defined(__arm__) diff --git a/xen/include/asm-x86/system.h b/xen/include/asm-x86/system.h index 7e5891f3df..6474dd1243 100644 --- a/xen/include/asm-x86/system.h +++ b/xen/include/asm-x86/system.h @@ -226,7 +226,7 @@ static always_inline unsigned long __xadd( * * Refer to the vendor system programming manuals for further details. */ -#define smp_mb() mb() +#define smp_mb() asm volatile ( "lock addl $0, -4(%%rsp)" ::: "memory" ) #define smp_rmb() barrier() #define smp_wmb() barrier() -- 2.11.0