Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1B65C3A5A7 for ; Tue, 6 Dec 2022 09:00:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 216498E0002; Tue, 6 Dec 2022 04:00:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C5F28E0001; Tue, 6 Dec 2022 04:00:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 067A58E0002; Tue, 6 Dec 2022 04:00:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id ECAAC8E0001 for ; Tue, 6 Dec 2022 04:00:29 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9A0A8C0CDB for ; Tue, 6 Dec 2022 09:00:29 +0000 (UTC) X-FDA: 80211285378.04.034B5BD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 120AA40015 for ; Tue, 6 Dec 2022 09:00:28 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WyN9SVGN; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670317229; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HGTXIY8L+DJ21AIcNg1CmE9ujShwBnVaqo1xss4HQhk=; b=yIFIMJo34XKnv0J893Eu6jJFepdphkEAuyWkHxNFW2UOqSOyUBb5gNGnU7WA1gymYLshiR 1NFnCQdwdeOA59pVpzCPuo+Fgnr0ajM+YeUQyGr0GGF+cN/aIYaC65omyzNBOPzk6Gacn1 aiC8QBAGABCmzKh4Xu21Dva0XfQHMz8= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WyN9SVGN; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670317229; a=rsa-sha256; cv=none; b=b1jkwnwUBWKpn+udeVoS2aANgxvecq6kzIAAaUarbnr9ekx2mF8M5jyFmuEkIgF4uL1iiB W+u2mkPNhGoZolZpck3WEI/OgGQcq/0ti4P5KnMbH9XrZdvTfV4pET+EhuEhifyXlVd4uJ R1T/W8c1PwD5WehU1qQ4yYBNLL6PKjo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670317228; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HGTXIY8L+DJ21AIcNg1CmE9ujShwBnVaqo1xss4HQhk=; b=WyN9SVGNNTr3a6PTA511s3vCzkO88wMUOa//WilYBvUAdjXQr8SLByPM/isX+bZlHXycMq YhZ4o6XGYy88BCmWhu+ZBnu3Yyu+Z4hWEAPuvXPUlHyKscsjZJpzyUGWbW0XpC1SUnUUBn rkuVPb3SNZC5zN21B2n0kR2QeUK28iE= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-588-Fd_AhlhKNwivMv6Gr5FRfA-1; Tue, 06 Dec 2022 04:00:27 -0500 X-MC-Unique: Fd_AhlhKNwivMv6Gr5FRfA-1 Received: by mail-wm1-f69.google.com with SMTP id f1-20020a1cc901000000b003cf703a4f08so5394429wmb.2 for ; Tue, 06 Dec 2022 01:00:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HGTXIY8L+DJ21AIcNg1CmE9ujShwBnVaqo1xss4HQhk=; b=28XSj3xzrc6XlS/SuFqydduL3J4NjjsoIFa8+GtiQ8mYbxK4dQ5PJmFRdsdY7VPTtg B52fJYG5Ga3egp7hFE17L4OFtTxYWFxngnnsZEM36U03JNHHjMf9OTTZCdu4VWa9Hr0G Hd4XKj/lK8xkPsrwupoyn2gQ+xtwHmkrnxuZ+kM9uC63rMYQYzku6DfjpbeuOM5syNYz N1E6KL1a96S9rTaxxG2ZOwzguYlr+8Bh6s0kh8vGG/qylvmGbpDga52HsArTjrtpF3Ow yoOmV4p3e0FHv06T4ttEYD2yrL5FqFk1WWcP32Hy3inN5+e/92f2KfgEWJBwPLxnVsDY bqNg== X-Gm-Message-State: ANoB5plKsPgxhPluFmtUYYJMOr5/kdNjFeRbqC1qfeY8NL35PMpGt3AD d5TFDk4xTe/RrkZGmiO2c7KMmedILZ/VzIOInJo/0V+btHGFhxpbogF9LCb0rvQiCNxK8eIBrf2 BGV0Kb2Jdk4Y= X-Received: by 2002:a05:600c:1d12:b0:3cf:8441:4a7c with SMTP id l18-20020a05600c1d1200b003cf84414a7cmr67229931wms.0.1670317225521; Tue, 06 Dec 2022 01:00:25 -0800 (PST) X-Google-Smtp-Source: AA0mqf7NgvpsWx9d1Dl44P/n2vQSchPjDT2NSctvIV6yaloKyLs2Kdv8I7uNNMdFNyXKC0sabj4l2Q== X-Received: by 2002:a05:600c:1d12:b0:3cf:8441:4a7c with SMTP id l18-20020a05600c1d1200b003cf84414a7cmr67229906wms.0.1670317225063; Tue, 06 Dec 2022 01:00:25 -0800 (PST) Received: from ?IPV6:2003:cb:c705:4f00:41f1:185d:4f9f:d1c2? (p200300cbc7054f0041f1185d4f9fd1c2.dip0.t-ipconnect.de. [2003:cb:c705:4f00:41f1:185d:4f9f:d1c2]) by smtp.gmail.com with ESMTPSA id m41-20020a05600c3b2900b003d1e3b1624dsm3781426wms.2.2022.12.06.01.00.24 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 06 Dec 2022 01:00:24 -0800 (PST) Message-ID: <24acd798-4e54-ca82-5ff9-e84a4795742e@redhat.com> Date: Tue, 6 Dec 2022 10:00:23 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.5.0 Subject: Re: kernel BUG at include/linux/swapops.h:497! To: Yang Shi Cc: "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Hugh Dickins , Peter Xu References: <0efeec33-8030-df1e-b055-2e0748fab4e0@redhat.com> <082e64a7-7340-b1fb-b819-43bc4df0584f@redhat.com> <3b75765f-d4d7-fd0e-366d-59e18ab42766@redhat.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Stat-Signature: qn3x1s1chyjwjop3rq5skhysm5i66o6m X-Rspam-User: X-Spamd-Result: default: False [-2.90 / 9.00]; BAYES_HAM(-6.00)[100.00%]; SORBS_IRL_BL(3.00)[209.85.128.69:received]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[redhat.com:+]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; SUBJECT_ENDS_EXCLAIM(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; ARC_NA(0.00)[]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:170.10.133.0/24]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; TO_DN_EQ_ADDR_SOME(0.00)[] X-Rspamd-Queue-Id: 120AA40015 X-Rspamd-Server: rspam06 X-HE-Tag: 1670317228-449447 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 06.12.22 00:14, Yang Shi wrote: > On Mon, Dec 5, 2022 at 7:14 AM David Hildenbrand wrote: >> >> On 02.12.22 13:36, David Hildenbrand wrote: >>> On 01.12.22 19:14, Yang Shi wrote: >>>> On Thu, Dec 1, 2022 at 9:48 AM Yang Shi wrote: >>>>> >>>>> On Thu, Dec 1, 2022 at 8:58 AM David Hildenbrand wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> running COW tests (in mm-unstable) on x86-pae with 8GiB, I am able to trigger the >>>>>> following BUG on latest upstream: >>>>>> >>>>>> root@debian:/mnt/scratch/linux/tools/testing/selftests/vm# ./cow >>>>>> # [INFO] detected THP size: 2048 KiB >>>>>> # [INFO] detected hugetlb size: 2048 KiB >>>>>> # [INFO] huge zeropage is enabled >>>>>> TAP version 13 >>>>>> 1..147 >>>>>> # [INFO] Anonymous memory tests in private mappings >>>>>> # [RUN] Basic COW after fork() ... with base page >>>>>> ok 1 No leak from parent into child >>>>>> # [RUN] Basic COW after fork() ... with swapped out base page >>>>>> ok 2 No leak from parent into child >>>>>> # [RUN] Basic COW after fork() ... with THP >>>>>> ok 3 No leak from parent into child >>>>>> # [RUN] Basic COW after fork() ... with swapped-out THP >>>>>> Segmentation fault >>>>>> >>>>>> >>>>>> [ 879.314600] kernel BUG at include/linux/swapops.h:497! >>>>>> [ 879.314615] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI >>>>>> [ 879.314624] CPU: 7 PID: 746 Comm: cow Tainted: G E 6.1.0-rc7+ #5 >>>>>> [ 879.314631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-1.fc36 04/01/2014 >>>>>> [ 879.314634] EIP: pagemap_pmd_range+0x644/0x650 >>>>>> [ 879.314645] Code: 00 00 00 00 66 90 89 ce b9 00 f0 ff ff e9 ff fb ff ff 89 d8 31 db e8 1b c2 52 00 e9 23 fb ff ff e8 51 80 56 00 e9 b6 fe ff ff <0f> 0b bf 00 f0 ff ff e9 38 fa ff ff 3e 8d 74 26 00 55 89 e5 57 31 >>>>>> [ 879.314651] EAX: ee2bd000 EBX: 00000002 ECX: ee2bd000 EDX: 00000000 >>>>>> [ 879.314656] ESI: f54b9ed4 EDI: 0001f400 EBP: f54b9db4 ESP: f54b9d68 >>>>>> [ 879.314660] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 EFLAGS: 00010246 >>>>>> [ 879.314670] CR0: 80050033 CR2: b7a00000 CR3: 357452a0 CR4: 00350ef0 >>>>>> [ 879.314675] Call Trace: >>>>>> [ 879.314681] ? madvise_free_pte_range+0x720/0x720 >>>>>> [ 879.314689] ? smaps_pte_range+0x4b0/0x4b0 >>>>>> [ 879.314694] walk_pgd_range+0x325/0x720 >>>>>> [ 879.314701] ? mt_find+0x1d6/0x3a0 >>>>>> [ 879.314710] __walk_page_range+0x164/0x170 >>>>>> [ 879.314716] walk_page_range+0xf9/0x170 >>>>>> [ 879.314720] ? __kmem_cache_alloc_node+0x2a8/0x340 >>>>>> [ 879.314729] pagemap_read+0x124/0x280 >>>>>> [ 879.314738] ? default_llseek+0xf1/0x160 >>>>>> [ 879.314747] ? smaps_account+0x1d0/0x1d0 >>>>>> [ 879.314754] vfs_read+0x90/0x290 >>>>>> [ 879.314760] ? do_madvise.part.0+0x24b/0x390 >>>>>> [ 879.314765] ? debug_smp_processor_id+0x12/0x20 >>>>>> [ 879.314773] ksys_pread64+0x58/0x90 >>>>>> [ 879.314778] __ia32_sys_ia32_pread64+0x1b/0x20 >>>>>> [ 879.314787] __do_fast_syscall_32+0x4c/0xc0 >>>>>> [ 879.314796] do_fast_syscall_32+0x29/0x60 >>>>>> [ 879.314803] do_SYSENTER_32+0x15/0x20 >>>>>> [ 879.314809] entry_SYSENTER_32+0x98/0xf1 >>>>>> [ 879.314815] EIP: 0xb7f36559 >>>>>> [ 879.314820] Code: 03 74 c0 01 10 05 03 74 b8 01 10 06 03 74 b4 01 10 07 03 74 b0 01 10 08 03 74 d8 01 00 00 00 00 00 51 52 55 89 e5 0f 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d 76 00 58 b8 77 00 00 00 cd 80 90 8d 76 >>>>>> [ 879.314825] EAX: ffffffda EBX: 00000003 ECX: bff00a50 EDX: 00000008 >>>>>> [ 879.314829] ESI: 005bd000 EDI: 00000000 EBP: b7f1c000 ESP: bff00a00 >>>>>> [ 879.314833] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000246 >>>>>> [ 879.314840] Modules linked in: intel_rapl_msr(E) intel_rapl_common(E) intel_pmc_core(E) kvm_intel(E) kvm(E) irqbypass(E) aesni_intel(E) libaes(E) crypto_simd(E) cryptd(E) rfkill(E) snd_pcm(E) snd_timer(E) joydev(E) snd(E) soundcore(E) sg(E) evdev(E) pcspkr(E) serio_raw(E) qemu_fw_cfg(E) parport_pc(E) ppdev(E) lp(E) parport(E) fuse(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc32c_generic(E) crc16(E) mbcache(E) jbd2(E) sd_mod(E) t10_pi(E) crc64_rocksoft(E) crc64(E) crc_t10dif(E) sr_mod(E) crct10dif_generic(E) cdrom(E) crct10dif_common(E) bochs(E) drm_vram_helper(E) drm_ttm_helper(E) ttm(E) drm_kms_helper(E) ata_generic(E) ata_piix(E) crc32_pclmul(E) libata(E) crc32c_intel(E) drm(E) e1000(E) scsi_mod(E) psmouse(E) i2c_piix4(E) scsi_common(E) floppy(E) button(E) >>>>>> [ 879.314936] ---[ end trace 0000000000000000 ]--- >>>>>> [ 879.314940] EIP: pagemap_pmd_range+0x644/0x650 >>>>>> [ 879.314944] Code: 00 00 00 00 66 90 89 ce b9 00 f0 ff ff e9 ff fb ff ff 89 d8 31 db e8 1b c2 52 00 e9 23 fb ff ff e8 51 80 56 00 e9 b6 fe ff ff <0f> 0b bf 00 f0 ff ff e9 38 fa ff ff 3e 8d 74 26 00 55 89 e5 57 31 >>>>>> [ 879.314949] EAX: ee2bd000 EBX: 00000002 ECX: ee2bd000 EDX: 00000000 >>>>>> [ 879.314953] ESI: f54b9ed4 EDI: 0001f400 EBP: f54b9db4 ESP: f54b9d68 >>>>>> [ 879.314957] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 EFLAGS: 00010246 >>>>>> [ 879.314961] CR0: 80050033 CR2: b7a00000 CR3: 357452a0 CR4: 00350ef0 >>>>>> >>>>>> >>>>>> Reading /proc/self/pagemap in THP test case seems to trigger the >>>>>> BUG_ON(is_migration_entry(entry) && !PageLocked(p)); >>>>>> in pfn_swap_entry_to_page(). >>>>>> >>>>>> I did not have time to cherry pick (slow machine) or look into details. >>>>>> And I don't remember seeing that BUG 64bit yet during my tests. >>>>>> >>>>>> Having a migration entry in the swap testcase is kind-of weird. But maybe it's >>>>>> related to THP splitting (which would, however, also be weird). I'd have expected >>>>>> a swap entry ... hopefully our swap type doesn't get corrupted. >>>>> >>>>> I'm on a slow machine too... anyway some hints off the top of my head. >>>>> >>>>> First of all, I don't think we will see a real swap PMD entry since >>>>> even though THP swap is supported the transhuge PMD is split by >>>>> try_to_unmap() if I remember correctly. So we should just be able to >>>>> see a regular PMD, a transhuge PMD, a migration PMD or a PROT_NONE PMD >>>>> (if autonuma is on). >>> >>> Yes. >>> >>>>> >>>>> Secondly, THP splitting doesn't convert transhuge PMD to migration PMD >>>>> either, it just splits transhuge PMD then converts every single PTEs >>>>> to migration PTEs. >>> >>> Right. >>> >>>>> >>>>> Thirdly, before pfn_swap_entry_to_page() is called, it does check >>>>> whether the swap PMD is migration PMD or not, if it is not a VM_BUG is >>>>> triggered. >>>>> >>>>> So it seems like a migration PMD is fine. The problem seems like the >>>>> page is not locked when doing migration IIUC. >>>> >>>> A quick look at the migration code, I don't see the page is unlocked >>>> if I don't miss something. So it may be helpful to dump the page. >>> >>> It is highly unlikely that we have migration happening here, because >>> >>> 1) This triggers 100% on the first try >>> 2) The machine is essentially idle with 7 GiB of free memory. >>> >>> I'll try digging a bit what exactly is happening here, dumping the PMD >>> entry first. >> >> Turns out that 32bit x86 doesn't even support PMD migration. We're >> stumbling over a PTE holding a migration entry and the underlying page >> was indeed unlocked. Turns out we fail to remove the migration entries >> we temporarily installed while splitting the THP. Splitting code >> doesn't/cannot notice that and unlocks the now-split page(s). > > Thanks for debugging this. IIUC the real call trace should be: > > pagemap_pmd_range -> > pte_to_pagemap_entry -> > pfn_swap_entry_to_page <--- BUG > > I thought it was due to the migration PMD entry in the first place. Yeah, me too. Unfortunately, the system swallowed that part of the call trace. -- Thanks, David / dhildenb