Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.4 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5541C4320A for ; Tue, 24 Aug 2021 20:14:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4322E61373 for ; Tue, 24 Aug 2021 20:14:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4322E61373 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 7E4548D0001; Tue, 24 Aug 2021 16:14:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 794B56B0071; Tue, 24 Aug 2021 16:14:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65C168D0001; Tue, 24 Aug 2021 16:14:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0128.hostedemail.com [216.40.44.128]) by kanga.kvack.org (Postfix) with ESMTP id 49AB96B006C for ; Tue, 24 Aug 2021 16:14:04 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DEF39248A1 for ; Tue, 24 Aug 2021 20:14:03 +0000 (UTC) X-FDA: 78511075566.04.5930751 Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by imf11.hostedemail.com (Postfix) with ESMTP id 8646BF0000BA for ; Tue, 24 Aug 2021 20:14:03 +0000 (UTC) Received: by mail-wr1-f53.google.com with SMTP id k8so33044567wrn.3 for ; Tue, 24 Aug 2021 13:14:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=wkAZmoc+zV9Oaok23kSwGMmh0db7JSUVeM2+ROd3syg=; b=ikmhZVza7ykOvV0hJTp8LyLD5gB8e2Og3cVaib2EIPoUeSNoaqp0DZJpCrQy+CJmt7 3enmuS/P+d7LfwIfsjKo0RRZ/v8AOx0ZsLI1+mv9AYUaNBl8Kdcu6zvax6KS/PtuO3Dp YlxWbqAoq+6Om1ItswPxMmWzG/ubzTmkFA8LTL9EBfJ7+fBMy9es8n10wAmPOB9sHDM4 tTEQbDVQ1TBV4XzVa8pw0fD3WU+Eg3C/k03flYID0r6jinc2IkiW/GQwGROhftY4eoLS La6Jm84IlItb86uHMGe2BEK+ssLWJ59mnOSIPISXc2HeaVIABB08GueVhT8fhQUfb94o ae/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=wkAZmoc+zV9Oaok23kSwGMmh0db7JSUVeM2+ROd3syg=; b=UJ2EXGZTgPhwhcJw3tKWN7wLu4Yzjh21RI0G4lzAc7cR7Ry+lipPBbkgRNwWGVGsu4 V8cU+GwOw9fmPBqgNlPZGwJu0FikJhWQYMsK3t8xTBN+aMK4Z8pWhXrzCPsZwOm/KEvy l2VNbTAMarapSbg58tbPpP2nc0pyzIZABneB090dlXE3ubCMmNyJJSxUU+xqoftZxR83 j2aFvZHCjLZLXUmZwyvL9hnk4uDpT8SUWNpIxohJHYe1Hjefsmm43sfu/90iqVatDAa2 EIKNvNPKjbCDDIdov1NvrksF2xpVwOSCL2FuRckddraD80AZT4cexoWuGAVTvIbYj812 a0fw== X-Gm-Message-State: AOAM531Kg84PEdRH6PdURa4JPvpysDsnNgTnBOnSAlrlrRZ35/GFId5f WRme8iv8Nb4xKQXwQpyuJQ== X-Google-Smtp-Source: ABdhPJyVYvyYPiXcnQc3VTSF6kCktfMU8iLC1IK7JKxQNLuRQMGoZgx5mJQyLINb7OdCbzATY5vEOA== X-Received: by 2002:a5d:4605:: with SMTP id t5mr20716776wrq.276.1629836042306; Tue, 24 Aug 2021 13:14:02 -0700 (PDT) Received: from ?IPv6:2a02:810b:f40:4200:65a1:1952:1d23:2e04? ([2a02:810b:f40:4200:65a1:1952:1d23:2e04]) by smtp.gmail.com with ESMTPSA id f20sm3147445wml.38.2021.08.24.13.14.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 24 Aug 2021 13:14:01 -0700 (PDT) Subject: Re: [BUG 5.14] arm64/mm: dma memory mapping fails (in some cases) To: Mike Rapoport , Catalin Marinas Cc: Will Deacon , Andrew Morton , Anshuman Khandual , Linux Kernel Mailing List , linux-mm@kvack.org, Linux ARM , Robin Murphy References: <20210824173741.GC623@arm.com> From: Alex Bee Message-ID: Date: Tue, 24 Aug 2021 22:14:01 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=ikmhZVza; spf=pass (imf11.hostedemail.com: domain of knaerzche@gmail.com designates 209.85.221.53 as permitted sender) smtp.mailfrom=knaerzche@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8646BF0000BA X-Stat-Signature: ifgsw3uuxtx4r7eox44spnmnfsan14kg X-HE-Tag: 1629836043-2960 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Mike, thanks for your reply Am 24.08.21 um 20:28 schrieb Mike Rapoport: > On Tue, Aug 24, 2021 at 06:37:41PM +0100, Catalin Marinas wrote: >> Hi Alex, >> >> Thanks for the report. >> >> On Tue, Aug 24, 2021 at 03:40:47PM +0200, Alex Bee wrote: >>> it seems there is a regression in arm64 memory mapping in 5.14, since= it >>> fails on Rockchip RK3328 when the pl330 dmac tries to map with: >>> >>> [=C2=A0=C2=A0=C2=A0 8.921909] ------------[ cut here ]------------ >>> [=C2=A0=C2=A0=C2=A0 8.921940] WARNING: CPU: 2 PID: 373 at kernel/dma/= mapping.c:235 dma_map_resource+0x68/0xc0 >>> [=C2=A0=C2=A0=C2=A0 8.921973] Modules linked in: spi_rockchip(+) fuse >>> [=C2=A0=C2=A0=C2=A0 8.921996] CPU: 2 PID: 373 Comm: systemd-udevd Not= tainted 5.14.0-rc7 #1 >>> [=C2=A0=C2=A0=C2=A0 8.922004] Hardware name: Pine64 Rock64 (DT) >>> [=C2=A0=C2=A0=C2=A0 8.922011] pstate: 80000005 (Nzcv daif -PAN -UAO -= TCO BTYPE=3D--) >>> [=C2=A0=C2=A0=C2=A0 8.922018] pc : dma_map_resource+0x68/0xc0 >>> [=C2=A0=C2=A0=C2=A0 8.922026] lr : pl330_prep_slave_fifo+0x78/0xd0 >>> [=C2=A0=C2=A0=C2=A0 8.922040] sp : ffff800012102ae0 >>> [=C2=A0=C2=A0=C2=A0 8.922043] x29: ffff800012102ae0 x28: ffff000005c9= 4800 x27: 0000000000000000 >>> [=C2=A0=C2=A0=C2=A0 8.922056] x26: ffff000000566bd0 x25: 000000000000= 0001 x24: 0000000000000001 >>> [=C2=A0=C2=A0=C2=A0 8.922067] x23: 0000000000000002 x22: ffff00000062= 8c00 x21: 0000000000000001 >>> [=C2=A0=C2=A0=C2=A0 8.922078] x20: ffff000000566bd0 x19: 000000000000= 0001 x18: 0000000000000000 >>> [=C2=A0=C2=A0=C2=A0 8.922089] x17: 0000000000000000 x16: 000000000000= 0000 x15: 0000000000000000 >>> [=C2=A0=C2=A0=C2=A0 8.922100] x14: 0000000000000277 x13: 000000000000= 0001 x12: 0000000000000000 >>> [=C2=A0=C2=A0=C2=A0 8.922111] x11: 0000000000000001 x10: 000000000000= 08e0 x9 : ffff800012102a80 >>> [=C2=A0=C2=A0=C2=A0 8.922123] x8 : ffff000000d14b80 x7 : ffff0000fe7b= 12f0 x6 : ffff0000fe7b1100 >>> [=C2=A0=C2=A0=C2=A0 8.922134] x5 : fffffc000000000f x4 : 000000000000= 0000 x3 : 0000000000000001 >>> [=C2=A0=C2=A0=C2=A0 8.922145] x2 : 0000000000000001 x1 : 00000000ff19= 0800 x0 : ffff000000628c00 >>> [=C2=A0=C2=A0=C2=A0 8.922158] Call trace: >>> [=C2=A0=C2=A0=C2=A0 8.922163]=C2=A0 dma_map_resource+0x68/0xc0 >>> [=C2=A0=C2=A0=C2=A0 8.922173]=C2=A0 pl330_prep_slave_sg+0x58/0x220 >>> [=C2=A0=C2=A0=C2=A0 8.922181]=C2=A0 rockchip_spi_prepare_dma+0xd8/0x2= c0 [spi_rockchip] >>> [=C2=A0=C2=A0=C2=A0 8.922208]=C2=A0 rockchip_spi_transfer_one+0x294/0= x3d8 [spi_rockchip] >> [...] >>> Note: This does not relate to the spi driver - when disabling this de= vice in >>> the device tree it fails for any other (i2s, for instance) which uses= dma. >>> Commenting out the failing check at [1], however, helps and the mappi= ng >>> works again. >> Do you know which address dma_map_resource() is trying to map (maybe >> add some printk())? It's not supposed to map RAM, hence the warning. >> Random guess, the address is 0xff190800 (based on the x1 above but the >> regs might as well be mangled). > 0xff190800 will cause this warning for sure. It has a memory map, but i= t is > not RAM so old version of pfn_valid() would return 0 and the new one > returns 1. > =20 >>> I tried to follow the recent changes for arm64 mm which could relate = to the >>> check failing at [1] and reverting >>> =C2=A0 commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_VALID") >>> helps and makes it work again, but I'm 100% uncertain if that commit = is >>> really the culprit. >>> >>> Note, that the firmware (legacy u-boot) injects memory configuration = in the >>> device tree as follows: >>> >>> /memreserve/=C2=A0=C2=A0=C2=A0 0x00000000fcefc000 0x000000000000d000; >>> / { >>> .. >>> =C2=A0=C2=A0=C2=A0 compatible =3D "pine64,rock64\0rockchip,rk3328"; >>> .. >>> =C2=A0=C2=A0=C2=A0 memory { >>> =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 reg =3D <0x00 0x200000 0x00 0x= fee00000 0x00 0x00 0x00 0x00>; >>> =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 device_type =3D "memory"; >>> =C2=A0=C2=A0=C2=A0 }; >>> >>> .. >>> } >> Either pfn_valid() gets confused in 5.14 or something is wrong with th= e >> DT. I have a suspicion it's the former since reverting the above commi= t >> makes it disappear. > I think pfn_valid() actually behaves as expected but the caller is wron= g > because pfn_valid !=3D RAM (this applies btw to !arm64 as well). > > /* Don't allow RAM to be mapped */ > if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) > return DMA_MAPPING_ERROR; > > Alex, can you please try this patch: > > diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c > index 2b06a809d0b9..4715e9641a29 100644 > --- a/kernel/dma/mapping.c > +++ b/kernel/dma/mapping.c > @@ -232,7 +232,7 @@ dma_addr_t dma_map_resource(struct device *dev, phy= s_addr_t phys_addr, > return DMA_MAPPING_ERROR; > =20 > /* Don't allow RAM to be mapped */ > - if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) > + if (WARN_ON_ONCE(!memblock_is_memory(phys_addr))) > return DMA_MAPPING_ERROR; > =20 > if (dma_map_direct(dev, ops)) > =20 Nope, doesn't help: [=C2=A0=C2=A0=C2=A0 8.353879] dma_map_resource Failed to map address 0xff= 190800 [=C2=A0=C2=A0=C2=A0 8.353886] dma_map_resource pfn_valid(PHYS_PFN(0xff190= 800)): 1 [=C2=A0=C2=A0=C2=A0 8.353892] dma_map_resource memblock_is_memory(0xff190= 800): 0 If understand the comment for that check correct, that we _don't_ want=20 RAM to be mapped - shoudn't that be: + if (WARN_ON_ONCE(memblock_is_memory(phys_addr))) ? Alex >>> So: there is a "hole" in the mappable memory and reading the commit m= essage >>> of >>> =C2=A0 commit a7d9f306ba70 ("arm64: drop pfn_valid_within() and simp= lify >>> pfn_valid()") >>> suggests, there was a change for that case recently. >> I think the change from the arm64 pfn_valid() to the generic one is >> avoiding the call to memblock_is_memory(). I wonder whether pfn_valid(= ) >> returns true just because we have a struct page available but the memo= ry >> may have been reserved. >> >> Cc'ing Mike R. >> >>> I also noticed there is a diff in the kernel log regarding memory ini= t up >>> until 5.13.12 it says >>> >>> [=C2=A0=C2=A0=C2=A0 0.000000] Zone ranges: >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 [mem 0x0000000000200000-0x00000000feffffff] >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA32=C2=A0=C2=A0=C2=A0 emp= ty >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 Normal=C2=A0=C2=A0 empty >>> [=C2=A0=C2=A0=C2=A0 0.000000] Movable zone start for each node >>> [=C2=A0=C2=A0=C2=A0 0.000000] Early memory node ranges >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 node=C2=A0=C2=A0 0: [mem 0x= 0000000000200000-0x00000000feffffff] >>> [=C2=A0=C2=A0=C2=A0 0.000000] Initmem setup node 0 [mem 0x00000000002= 00000-0x00000000feffffff] >>> [=C2=A0=C2=A0=C2=A0 0.000000] On node 0 totalpages: 1043968 >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA zone: 16312 pages used = for memmap >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA zone: 0 pages reserved >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA zone: 1043968 pages, LI= FO batch:63 >>> >>> In contrary in 5.14-rc7 it says: >>> >>> [=C2=A0=C2=A0=C2=A0 0.000000] Zone ranges: >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 [mem 0x0000000000200000-0x00000000feffffff] >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA32=C2=A0=C2=A0=C2=A0 emp= ty >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 Normal=C2=A0=C2=A0 empty >>> [=C2=A0=C2=A0=C2=A0 0.000000] Movable zone start for each node >>> [=C2=A0=C2=A0=C2=A0 0.000000] Early memory node ranges >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 node=C2=A0=C2=A0 0: [mem 0x= 0000000000200000-0x00000000feffffff] >>> [=C2=A0=C2=A0=C2=A0 0.000000] Initmem setup node 0 [mem 0x00000000002= 00000-0x00000000feffffff] >>> [=C2=A0=C2=A0=C2=A0 0.000000] On node 0, zone DMA: 512 pages in unava= ilable ranges >>> [=C2=A0=C2=A0=C2=A0 0.000000] On node 0, zone DMA: 4096 pages in unav= ailable ranges >>> >>> (note the "unavailable ranges") >>> I'm uncertain again here, if that diff is expected behavior because o= f those >>> recent mm changes for arm64. >>> >>> After reverting >>> =C2=A0 commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_VALID") >>> the log changes to >>> >>> [=C2=A0=C2=A0=C2=A0 0.000000] Zone ranges: >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 [mem 0x0000000000200000-0x00000000feffffff] >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 DMA32=C2=A0=C2=A0=C2=A0 emp= ty >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 Normal=C2=A0=C2=A0 empty >>> [=C2=A0=C2=A0=C2=A0 0.000000] Movable zone start for each node >>> [=C2=A0=C2=A0=C2=A0 0.000000] Early memory node ranges >>> [=C2=A0=C2=A0=C2=A0 0.000000]=C2=A0=C2=A0 node=C2=A0=C2=A0 0: [mem 0x= 0000000000200000-0x00000000feffffff] >>> [=C2=A0=C2=A0=C2=A0 0.000000] Initmem setup node 0 [mem >>> 0x0000000000200000-0x00000000feffffff] >>> >>> (no DMA zones here) >>> >>> As you might have noticed I have _zero_ clue about memory mapping and= dma >>> subsystem - so let me know if there is any more information needed fo= r that >>> and thanks for your help. >> Adding Robin as well, he has a better clue than us on DMA ;). >> >>> Alex >>> >>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.gi= t/tree/kernel/dma/mapping.c?id=3De22ce8eb631bdc47a4a4ea7ecf4e4ba499db4f93= #n235 >> --=20 >> Catalin