Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B643C433EF for ; Mon, 25 Apr 2022 16:34:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79FDE6B0078; Mon, 25 Apr 2022 12:34:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74EDC6B007B; Mon, 25 Apr 2022 12:34:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 616336B007D; Mon, 25 Apr 2022 12:34:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 51C476B0078 for ; Mon, 25 Apr 2022 12:34:56 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 203F027E4A for ; Mon, 25 Apr 2022 16:34:56 +0000 (UTC) X-FDA: 79395950592.09.E3D35BA Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf23.hostedemail.com (Postfix) with ESMTP id EA757140044 for ; Mon, 25 Apr 2022 16:34:50 +0000 (UTC) Received: by mail-pg1-f202.google.com with SMTP id h14-20020a63530e000000b0039daafb0a84so9609724pgb.7 for ; Mon, 25 Apr 2022 09:34:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc :content-transfer-encoding; bh=OBSlIm3CsQonY97RSbwbgst09NYzSikOBggGs+V9f94=; b=TYRTW7DIanUt2X8HgTIqIDraBRR5R7KYqDOJrnvevffogbmiQ6jZo2E5iOR3iZOeQz F08mM+vpxMjBNKI9T6vOVmmsrdoo50C6GGtEfRpLhYV3DpeFx/1oeKqPEomSzMllgqfI 8TQDmC7SEu2lknd2RBm14XOQDU3+0q1HFREPi7qSMUAPfWakm7b2KVFa6UF5x7nA1KJ2 11ovMDMIfmLS+dx5OWhVcmk0sebT3rN0RcqQssxubHAvXxCOFpiEwdZgDHgJERu9BQOr hy4uV176MzQQ1bfpZhU+H+eKhQJ9v/BE0TXLfNbQpuTIA4XdUIcNSJiuU2uAaI1hczQ1 Km/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc :content-transfer-encoding; bh=OBSlIm3CsQonY97RSbwbgst09NYzSikOBggGs+V9f94=; b=yFZC4+1UKeBIgJzh1GxUdzWFeFUgsEWSTb50X7eOzDqmxZgn7PrnsV6pxCR7Ley8E/ V7q7c7OG47S9zv+adBW2d0e/uEvyF7a/zkV2o8+vwbUGMRf+08iBLrDmscYEQs7IXCMs hYm2wnkNJIm6JXVWk+Y+5KFqmYSRiMdID1P9b4Dp/xiyM1Rw1da3RyVZxHaQjtuP7Xs+ umZTjnjWR6ewil+GX91OBVR/hArT83ZUlZDbAbIZ19irpEHmE1khsTz4tNGS0p/BGn1o GwwYD92oT8MnXJqVWp9/wJt6QLb1JRXRl35j6/qvdlJitq7y1yJAU77bK3xPsHC+phF/ w22g== X-Gm-Message-State: AOAM530hfu/WxbdJfn31ZtNw2ji9PZcTrRmgZpM21lvGGtidEOEwfMYU o2LYMPFe2dd1RKAoFyha2QLCW44Q X-Google-Smtp-Source: ABdhPJyiuPZ8cYIWnL0U7bLrrEHhaNdhi5mhqdxeyHNjUF7VRrgtymq7PkDFO7DJr3KkpLD05Puxdc9Z X-Received: from juew-desktop.sea.corp.google.com ([2620:15c:100:202:3272:9a24:5a9f:52c9]) (user=juew job=sendgmr) by 2002:a62:2504:0:b0:505:7693:b48e with SMTP id l4-20020a622504000000b005057693b48emr19618060pfl.4.1650904494529; Mon, 25 Apr 2022 09:34:54 -0700 (PDT) Date: Mon, 25 Apr 2022 09:34:51 -0700 Message-Id: <20220425163451.3818838-1-juew@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [RFC] Expose a memory poison detector ioctl to user space. From: Jue Wang To: Naoya Horiguchi , Tony Luck , Dave Hansen Cc: Jiaqi Yan , Greg Thelen , Mina Almasry , linux-mm@kvack.org, Jue Wang Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: 19p5nbgtmg7yujra6u5baehhkbheg7pe Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TYRTW7DI; spf=pass (imf23.hostedemail.com: domain of 3rs1mYgQKCP0ozj1lttlqj.htrqnsz2-rrp0fhp.twl@flex--juew.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3rs1mYgQKCP0ozj1lttlqj.htrqnsz2-rrp0fhp.twl@flex--juew.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: EA757140044 X-HE-Tag: 1650904490-87274 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The ever increasing server size and cost in DRAM has brought the memory subsystem reliability to the forefront of large fleet owners=E2=80=99 conce= rn. Memory-error-caused server and workload crashes are ranked #1 among all hardware failures by a large margin. Deploying extra reliable DRAM adds significant cost at a fleet size, e.g, 10% extra cost on DRAM can amount to hundreds of millions worth of dollars spending. =E2=80=9CReactive=E2=80=9D memory poison recovery [3], i.e., recover from M= CEs raised during an execution context (the kernel mechanisms are MCE handler + CONFIG_MEMORY_FAILURE + SIGBUS to the user space process), has been found effective in keeping systems resilient from memory errors. However, it has several major drawbacks: 1. It requires software systems that access poisoned memory to be specifically designed and implemented to recover from memory errors: . Uncorrectable errors are random, which may happen outside of the enlightened address spaces or execution contexts. . The added error recovery capability comes at a cost of added complexity and often not possible to enlighten in 3rd party software. 2. In a virtualized environment, the injected MCEs introduce the same challenge to the guest. 3. Because of random execution contexts, CPU erratum that are vulnerable to speculative execution, split cache line accesses, hyperthread buddy scheduling etc (e.g., [1]) can often turn a recoverable UC error into an unrecoverable MCE and a system crash. 4. Aside from CPU accesses, NIC or other PCIe devices accessing poisoned memory cause host crashes in the production system regularly. 5. In a multi-tenant environment, the =E2=80=9Creactive=E2=80=9D poison rec= overy is less effective to the largest workloads compared to the smaller workloads in that the smaller workloads have a much higher chance to get saved cleanly as a victim=E2=80=99s neighbor rather than the victim itself. The goal is to minimize the probability that any software / hardware component actually gets a chance to consume an error. A possible solution is to =E2=80=9Cproactively=E2=80=9D look for and detect memory errors befor= e consumption. Here we assume system software is enlightened to drain the affected host and migrate the running jobs off to another healthy host as soon as an error is detected; and is able to recover from, contain, and emulate the errors that surface up during the migration process. The main benefits (free memory ratios come from a large production fleet): 1. Memory errors in free memory (~50%) can be completely contained without impacting software / hardware systems later on. 2. Inside a VM guest, memory errors in free memory (~50%) can be completely contained with the UCNA injection via CMCI capability being added to KVM in [2]. 3. It=E2=80=99s required to detect memory errors on allocated pages without impacting the execution or performance of the page owners. For instance, in the cloud world, the majority of the host memory is pre-allocated as a guest memory pool and memory errors can emerge well after the guest memory pool allocation. 4. Early detection and ensured containment (e.g., unmapping and PG_HWPOISON) can effectively prevent most if not all the crashes due to CPU erratum ([3], section 3.5.2 - 3.5.5) that =E2=80=9Creactive=E2=80=9D po= ison recovery cannot avoid and these crashes represent >40% of all host crashes in a production fleet. The hardware patrol scrubber [4] is evaluated, and the type of performance (i.e., latency between error emergence and detection are in days to hours) and error loss rate due to downgrading (or otherwise system instability due to SRAO MCE broadcast and overflow) do not meet the requirements (e.g., ~30 min emergence to consumption based on simulation). A possible solution is to have some specially purposed poison detector userspace agent that proactively looks for memory errors by invoking some ioctl specifically implemented to avoid the CPU erratum, minimize performance interferences (cache pollution etc) and avoid leaking the memory content into the registers. The detector agent runs with minimal configurable resource consumption (e.g., 0.1 core / socket, <0.5% membw consumption etc) and pauses itself when the host system is under heavy load (e.g., CPU>90% or membw>75%). The kernel ioctl may take the following form and a potential point of discussion is whether Unmapping guest Private Memory will require zapping the kernel direct map or not. This ioctl can be compiled off in case incompatible with other use cases like UPM. /* Could stop and return after the 1st poison is detected */ #define MCESCAN_IOCTL_SCAN 0 struct SysramRegion { /* input */ uint64_t first_byte; /* first page-aligned physical address to scan */ uint64_t length; /* page-aligned length of memory region to scan */ /* output */ uint32_t poisoned; /* 1 - a poisoned page is found, 0 - otherwise */ uint32_t poisoned_pfn; /* PFN of the 1st detected poisoned page */ } 1. https://lore.kernel.org/lkml/164529415398.16921.8042682039148828519.tip-= bot2@tip-bot2/ 2. https://lore.kernel.org/kvm/20220412223134.1736547-1-juew@google.com/ 3. https://www.intel.com/content/dam/www/public/us/en/documents/white-paper= s/reduce-server-crash-rate-tencent-paper.pdf --=20 2.36.0.rc2.479.g8af0fa9b8e-goog