Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64F5EC43461 for ; Wed, 9 Sep 2020 10:59:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A846521D7B for ; Wed, 9 Sep 2020 10:59:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A846521D7B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CE5996B0037; Wed, 9 Sep 2020 06:59:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C963D6B0055; Wed, 9 Sep 2020 06:59:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD2D96B005A; Wed, 9 Sep 2020 06:59:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0254.hostedemail.com [216.40.44.254]) by kanga.kvack.org (Postfix) with ESMTP id A45336B0037 for ; Wed, 9 Sep 2020 06:59:18 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 64CA03620 for ; Wed, 9 Sep 2020 10:59:18 +0000 (UTC) X-FDA: 77243226396.11.map79_0d0e008270dc Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 39B1D180F8B80 for ; Wed, 9 Sep 2020 10:59:18 +0000 (UTC) X-HE-Tag: map79_0d0e008270dc X-Filterd-Recvd-Size: 9317 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Wed, 9 Sep 2020 10:59:17 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 61DE9AC8C; Wed, 9 Sep 2020 10:59:17 +0000 (UTC) Date: Wed, 9 Sep 2020 12:59:14 +0200 From: Michal Hocko To: Laurent Dufour Cc: akpm@linux-foundation.org, David Hildenbrand , Oscar Salvador , rafael@kernel.org, nathanl@linux.ibm.com, cheloha@linux.ibm.com, stable@vger.kernel.org, Greg Kroah-Hartman , linux-mm@kvack.org, LKML Subject: Re: [PATCH] mm: don't rely on system state to detect hot-plug operations Message-ID: <20200909105914.GF7348@dhcp22.suse.cz> References: <5cbd92e1-c00a-4253-0119-c872bfa0f2bc@redhat.com> <20200908170835.85440-1-ldufour@linux.ibm.com> <20200909074011.GD7348@dhcp22.suse.cz> <9faac1ce-c02d-7dbc-f79a-4aaaa5a73d28@linux.ibm.com> <20200909090953.GE7348@dhcp22.suse.cz> <4cdb54be-1a92-4ba4-6fee-3b415f3468a9@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <4cdb54be-1a92-4ba4-6fee-3b415f3468a9@linux.ibm.com> X-Rspamd-Queue-Id: 39B1D180F8B80 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 09-09-20 11:21:58, Laurent Dufour wrote: > Le 09/09/2020 =E0 11:09, Michal Hocko a =E9crit=A0: > > On Wed 09-09-20 09:48:59, Laurent Dufour wrote: > > > Le 09/09/2020 =E0 09:40, Michal Hocko a =E9crit=A0: [...] > > > > > In > > > > > that case, the system is able to boot but later hot-plug operat= ion may lead > > > > > to this panic because the node's links are correctly broken: > > > >=20 > > > > Correctly broken? Could you provide more details on the inconsist= ency > > > > please? > > >=20 > > > laurent@ltczep3-lp4:~$ ls -l /sys/devices/system/memory/memory21 > > > total 0 > > > lrwxrwxrwx 1 root root 0 Aug 24 05:27 node1 -> ../../node/node1 > > > lrwxrwxrwx 1 root root 0 Aug 24 05:27 node2 -> ../../node/node2 > > > -rw-r--r-- 1 root root 65536 Aug 24 05:27 online > > > -r--r--r-- 1 root root 65536 Aug 24 05:27 phys_device > > > -r--r--r-- 1 root root 65536 Aug 24 05:27 phys_index > > > drwxr-xr-x 2 root root 0 Aug 24 05:27 power > > > -r--r--r-- 1 root root 65536 Aug 24 05:27 removable > > > -rw-r--r-- 1 root root 65536 Aug 24 05:27 state > > > lrwxrwxrwx 1 root root 0 Aug 24 05:25 subsystem -> ../../../../= bus/memory > > > -rw-r--r-- 1 root root 65536 Aug 24 05:25 uevent > > > -r--r--r-- 1 root root 65536 Aug 24 05:27 valid_zones > >=20 > > OK, so there are two nodes referenced here. Not terrible from the use= r > > point of view. Such a memory block will refuse to offline or online > > IIRC. >=20 > No the memory block is still owned by one node, only the sysfs > representation is wrong. So the memory block can be hot unplugged, but = only > one node's link will be cleaned, and a '/syss/devices/system/node#/memo= ry21' > link will remain and that will be detected later when that memory block= is > hot plugged again. OK, so you need to hotremove first and hotadd again to trigger the problem. It is not like you would be a hot adding something new. This is a useful information to have in the changelog. > > > > Which physical memory range you are trying to add here and what i= s the > > > > node affinity? > > >=20 > > > None is added, the root cause of the issue is happening at boot tim= e. > >=20 > > Let me clarify my question. The crash has clearly happened during the > > hotplug add_memory_resource - which is clearly not a boot time path. > > I was askin for more information about why this has failed. It is qui= te > > clear that sysfs machinery has failed and that led to BUG_ON but we a= re > > mising an information on why. What was the physical memory range to b= e > > added and why sysfs failed? >=20 > The BUG_ON is detecting a bad state generated earlier, at boot time bec= ause > register_mem_sect_under_node() didn't check for the block's node id. >=20 > > > > > ------------[ cut here ]------------ > > > > > kernel BUG at /Users/laurent/src/linux-ppc/mm/memory_hotplug.c:= 1084! > > > > > Oops: Exception in kernel mode, sig: 5 [#1] > > > > > LE PAGE_SIZE=3D64K MMU=3DHash SMP NR_CPUS=3D2048 NUMA pSeries > > > > > Modules linked in: rpadlpar_io rpaphp pseries_rng rng_core vmx_= crypto gf128mul binfmt_misc ip_tables x_tables xfs libcrc32c crc32c_vpmsu= m autofs4 > > > > > CPU: 8 PID: 10256 Comm: drmgr Not tainted 5.9.0-rc1+ #25 > > > > > NIP: c000000000403f34 LR: c000000000403f2c CTR: 00000000000000= 00 > > > > > REGS: c0000004876e3660 TRAP: 0700 Not tainted (5.9.0-rc1+) > > > > > MSR: 800000000282b033 CR: 2= 4000448 XER: 20040000 > > > > > CFAR: c000000000846d20 IRQMASK: 0 > > > > > GPR00: c000000000403f2c c0000004876e38f0 c0000000012f6f00 fffff= fffffffffef > > > > > GPR04: 0000000000000227 c0000004805ae680 0000000000000000 00000= 004886f0000 > > > > > GPR08: 0000000000000226 0000000000000003 0000000000000002 fffff= ffffffffffd > > > > > GPR12: 0000000088000484 c00000001ec96280 0000000000000000 00000= 00000000000 > > > > > GPR16: 0000000000000000 0000000000000000 0000000000000004 00000= 00000000003 > > > > > GPR20: c00000047814ffe0 c0000007ffff7c08 0000000000000010 c0000= 000013332c8 > > > > > GPR24: 0000000000000000 c0000000011f6cc0 0000000000000000 00000= 00000000000 > > > > > GPR28: ffffffffffffffef 0000000000000001 0000000150000000 00000= 00010000000 > > > > > NIP [c000000000403f34] add_memory_resource+0x244/0x340 > > > > > LR [c000000000403f2c] add_memory_resource+0x23c/0x340 > > > > > Call Trace: > > > > > [c0000004876e38f0] [c000000000403f2c] add_memory_resource+0x23c= /0x340 (unreliable) > > > > > [c0000004876e39c0] [c00000000040408c] __add_memory+0x5c/0xf0 > > > > > [c0000004876e39f0] [c0000000000e2b94] dlpar_add_lmb+0x1b4/0x500 > > > > > [c0000004876e3ad0] [c0000000000e3888] dlpar_memory+0x1f8/0xb80 > > > > > [c0000004876e3b60] [c0000000000dc0d0] handle_dlpar_errorlog+0xc= 0/0x190 > > > > > [c0000004876e3bd0] [c0000000000dc398] dlpar_store+0x198/0x4a0 > > > > > [c0000004876e3c90] [c00000000072e630] kobj_attr_store+0x30/0x50 > > > > > [c0000004876e3cb0] [c00000000051f954] sysfs_kf_write+0x64/0x90 > > > > > [c0000004876e3cd0] [c00000000051ee40] kernfs_fop_write+0x1b0/0x= 290 > > > > > [c0000004876e3d20] [c000000000438dd8] vfs_write+0xe8/0x290 > > > > > [c0000004876e3d70] [c0000000004391ac] ksys_write+0xdc/0x130 > > > > > [c0000004876e3dc0] [c000000000034e40] system_call_exception+0x1= 60/0x270 > > > > > [c0000004876e3e20] [c00000000000d740] system_call_common+0xf0/0= x27c > > > > > Instruction dump: > > > > > 48442e35 60000000 0b030000 3cbe0001 7fa3eb78 7bc48402 38a5fffe = 7ca5fa14 > > > > > 78a58402 48442db1 60000000 7c7c1b78 <0b030000> 7f23cb78 4bda371= d 60000000 > > > > > ---[ end trace 562fd6c109cd0fb2 ]--- > > > >=20 > > > > The BUG_ON on failure is absolutely horrendous. There must be a b= etter > > > > way to handle a failure like that. The failure means that > > > > sysfs_create_link_nowarn has failed. Please describe why that is = the > > > > case. > > > >=20 > > > > > This patch addresses the root cause by not relying on the syste= m_state > > > > > value to detect whether the call is due to a hot-plug operation= or not. An > > > > > additional parameter is added to link_mem_sections() to tell th= e context of > > > > > the call and this parameter is propagated to register_mem_sect_= under_node() > > > > > throuugh the walk_memory_blocks()'s call. > > > >=20 > > > > This looks like a hack to me and it deserves a better explanation= . The > > > > existing code is a hack on its own and it is inconsistent with ot= her > > > > boot time detection. We are using (system_state < SYSTEM_RUNNING)= at other > > > > places IIRC. Would it help to use the same here as well? Maybe we= want to > > > > wrap that inside a helper (early_memory_init()) and use it at all > > > > places. > > >=20 > > > I agree, this looks like a hack to check for the system_state value= . > > > I'll follow the David's proposal and introduce an enum detailing wh= en the > > > node id check has to be done or not. > >=20 > > I am not sure an enum is going to make the existing situation less > > messy. Sure we somehow have to distinguish boot init and runtime hotp= lug > > because they have different constrains. I am arguing that a) we shoul= d > > have a consistent way to check for those and b) we shouldn't blow up > > easily just because sysfs infrastructure has failed to initialize. >=20 > For the point a, using the enum allows to know in > register_mem_sect_under_node() if the link operation is due to a hotplu= g > operation or done at boot time. Yes, but let me repeat. We have a mess here and different paths check for the very same condition by different ways. We need to unify those. > For the point b, one option would be ignore the link error in the case = the > link is already existing, but that BUG_ON() had the benefit to highligh= t the > root issue. Yes BUG_ON is obviously an over-reaction. The system is not in a state to die anytime soon. --=20 Michal Hocko SUSE Labs