From: Andre Przywara Date: Tue, 7 Jan 2020 18:14:22 +0000 Subject: [RFC PATCH] Revert "arm: Show cache warnings in U-Boot proper only" In-Reply-To: <20200106160121.GG4866@bill-the-cat> References: <20191219005211.25327-1-andre.przywara@arm.com> <487d6382-7445-af7f-4c30-36673e837e61@denx.de> <16cefb13-820d-49b2-8ba6-8d1726df7660@arm.com> <74491a93-9cf5-5540-6611-71b365a54b7d@denx.de> <20191219113604.0d031e62@donnerap.cambridge.arm.com> <70ecd325-00e9-6229-b057-c0aa2b8285c4@denx.de> <20191219123209.763dbc5a@donnerap.cambridge.arm.com> <20200106160121.GG4866@bill-the-cat> Message-ID: <20200107181422.65e105ef@donnerap.cambridge.arm.com> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable To: u-boot@lists.denx.de On Mon, 6 Jan 2020 11:01:21 -0500 Tom Rini wrote: Hi Tom, thanks for chiming in on this! > On Thu, Dec 19, 2019 at 12:32:09PM +0000, Andre Przywara wrote: > > On Thu, 19 Dec 2019 12:43:35 +0100 > > Marek Vasut wrote: > >=20 > > Hi Marek, > > =20 > > > On 12/19/19 12:36 PM, Andre Przywara wrote: =20 > > > > On Thu, 19 Dec 2019 09:04:32 +0100 > > > > Marek Vasut wrote: =20 > > > >> On 12/19/19 2:23 AM, Andr=C3=A9 Przywara wrote: =20 > > > >>> On 19/12/2019 00:55, Marek Vasut wrote: =20 > > > >>>> On 12/19/19 1:52 AM, Andre Przywara wrote: =20 > > > >>>>> According to commit 11aa6a32eb5f ("arm: cache: Implement cache = range > > > >>>>> check for v7"), which introduced check_cache_range(), this was = meant > > > >>>>> as a pure debugging feature, only to be compiled in when a deve= loper > > > >>>>> explicitly #defined DEBUG in cache.c. Presumably the intention = was to > > > >>>>> help with finding *certain* alignment issues with DMA buffers. > > > >>>>> > > > >>>>> Commit bcc53bf09589 ("arm: Show cache warnings in U-Boot proper= only") > > > >>>>> compiled this in *unconditionally* into U-Boot proper. > > > >>>>> > > > >>>>> This has the annoying side effect of producing tons of somewhat > > > >>>>> pointless warnings about non-aligned clean&invalidate operation= s, which > > > >>>>> tend to be appeased by even more pointless rounding operations = in many > > > >>>>> drivers (mostly those used on ARM boards). > > > >>>>> > > > >>>>> Bring back the old behaviour, of only compiling this in for DEB= UG > > > >>>>> situations, but staying silent otherwise. > > > >>>>> > > > >>>>> This reverts commit bcc53bf095893fbdae531a9a7b5d4ef4a125a7fc. > > > >>>>> > > > >>>>> Signed-off-by: Andre Przywara > > > >>>>> --- > > > >>>>> Hi, > > > >>>>> > > > >>>>> if the intention was indeed to always force cache maintenance r= ange > > > >>>>> alignments, I would like to open a discussion on this, because = I believe > > > >>>>> it is not useful, especially in the clean&invalidate case. = =20 > > > >>>> > > > >>>> Why don't you rather fix the cache op alignment bugs ? =20 > > > >>> > > > >>> Which bugs do you mean? =20 > > > >> > > > >> Well, the ones reported by this warning. =20 > > > >=20 > > > > As mentioned, most of them are just silenced in drivers, though not= actually fixed. =20 > > >=20 > > > Do you have specific details here ? They should be fixed. > > > =20 > > > > This patch here was actually triggered by me cleaning up sun8i-emac= .c, which then made those warnings appear again. > > > > I will try to send out this series, which might make it more clear = what I am after. > > > > =20 > > > >>> Those that would currently trigger those warnings? > > > >>> I don't think they are actual bugs, besides I don't think they ar= e any > > > >>> cases left for 32-bit ARM boards (leave alone the new RPi4 Ethern= et > > > >>> driver in the rpi-4-32b_defconfig). =20 > > > >> > > > >> If your cache isn't flushed or invalidated properly, it's a bug. > > > >> On ARM32, this is (was, before this warning) a real problem. =20 > > > >=20 > > > > The actual cache maintenance is fine in the drivers, it's the over-= strict alignment requirements that bothers me and makes things actually wor= se: > > > >=20 > > > > Architecturally (ARM ARM) there is no alignment requirement for VA = operations. Walking over a range of addresses does get easier if you start = on a cache line boundary, so this is what we do in the implementations for = v7 and v8. =20 > > >=20 > > > U-Boot is not only V7 and V8, but also all the earlier ones (back to = V4). =20 > >=20 > > I checked v6 as well (same), and couldn't find any evidence that this w= ould be different for v5 or v4, although CP15 does not seem to be architect= urally defined there. Do you have an example of an older core failing that? > > But anyway this particular check and patch is affecting v7 only. > > =20 > > > > Alignment gets critical when it comes to a pure invalidate call. If= you have other dirty data in this same cache line, you might lose the late= st stores, that's why it is important to align those *buffers*. So the stri= ct check we have in v7_dcache_inval_range() is the right thing (at least fo= r the base address). What is not right is to deliberately align just the *p= arameters* of the invalidate call, as seens in net/sun8i_emac.c or net/dwc_= eth_qos.c, for instance. This could actually hide bugs. Drivers should alwa= ys pass on the actual buffer address to invalidate(). =20 > > >=20 > > > Sure, but that's a driver bug. That's a separate topic from this patc= h. > > > =20 > > > > Then there is no real need to align the range for flush (clean&inva= lidate) calls. Overshooting here is not critical, and again we do align the= m anyway in the implementation. =20 > > >=20 > > > Wrong, you may interfere with some DMA engine which might be accessing > > > the area which you just overshot. =20 > >=20 > > Can you give an example on that? Are you thinking about cleaning dirty = cache lines to a DRAM region which DMA is just writing data into? > > After all a clean could happen at any time, when a line gets evicted fr= om the cache, without an explicit cache maintenance instruction. So if you = have such a case, that's a separate bug. > > =20 > > > > This > > > > forced check only leads to drivers aligning the *parameters* of the= se calls, to no actual effect, probably given a wrong impression to driver = authors and readers. =20 > > >=20 > > > Again, driver bug, that's unrelated and must be fixed. =20 > >=20 > > If you fix them, they will trigger tons of warnings, mostly bogus, abou= t end addresses not being aligned (see below). Been there, done that. That'= s why this patch as a first step to allow those fixes without spamming the = user. > > =20 > > > > So in preparation for fixing the drivers, I ask for those checks to= become pure DEBUG warnings. Another alternative would be to relax those ch= ecks, to just require the start address of invalidate to be aligned - but t= his is what we check separately already. =20 > > >=20 > > > See above. > > > =20 > > > >>> Or those that are currently hidden because we *force* an alignmen= t on > > > >>> the *arguments* passed to invalidate_dcache_range, for instance? > > > >>> These are quite numerous, so I would rather get some input first = before > > > >>> spending a lot of time on this. =20 > > > >> > > > >> Make sure your buffers are properly aligned and you have no cache > > > >> issues. It's really that easy. If you see numerous, then your plat= form > > > >> has a real problem which must be fixed ; I didn't see any for quit= e a > > > >> while on platforms I have access to. =20 > > > >=20 > > > > Yes, I think the buffers are correctly aligned in the drivers, as l= east the ones I checked. But then we should always pass this address to inv= alidate, and not round it just before calling invalidate. Same goes to some= degree for the end address, although is a somewhat separate topic. =20 > > >=20 > > > Of course the address passed to cache ops should not be rounded up. > > > That's a bug (unless the buffer actually covers that rounded address, > > > which sometimes is the case, and might or might not thus be harmless. > > > That's up to the driver author to determine) =20 > >=20 > > Yes, and that's why the strict unconditional check we have right now is= not helping. We should have them in, and developers could check the warnin= gs for being legit or not. > > =20 > > > > And that's why I think having check_cache_range() is useful for dev= elopment and debugging, but only then, because not all warnings are legit (= end addresses). =20 > > >=20 > > > This implication makes no sense. How do you infer from "drivers have > > > bugs" that a warning about such bugs should be turned into a DEBUG()?= =20 > >=20 > > Most (if not all) of those warnings come from the end address not being= aligned. And this is something not easily being checked outside of the dri= ver. What we want to check for invalidate is that we only invalidate a well= known region, so: > > 1) The start address of the buffer is aligned. > > 2) The end address, rounded up to the next cache line, is still within = our buffer (so nothing else is using the same cacheline). > >=20 > > We can check 1) easily, but not 2), that would require driver specific = code, which knows about the size of the buffer. We could provide something = like: > > bool check_cache_buffer(void *start_address, size_t length, size_t bu= ffer_size); > > which drivers would call explicitly. Or we could include the buffer siz= e into the invalidate call, but that would change the interface. > >=20 > > That would at least cover the buffers that the driver provides itself (= RX buffers for network drivers). Network TX buffers are handed in from the = generic network code, and the driver does not know what is there after "len= gth". So the best thing we can do is to call "flush_dcache_range(packet, pa= cket + length);", and then should not receive warnings if the end address i= s unaligned. That's why I think checks for flush are not useful. =20 >=20 > Thanks for having this discussion. I think it is reasonable at this > point to investigate how we can evolve what we about cache flushing to > be better than what we have now. Looking at the cases above, it seems > like it might be reasonable to have always enabled checks #1 (that can > easily be disabled) and if we can't easily migrate our callers to handle > #2 (and maybe not have that check be default enabled?) at least provide > the helper for it, for debugging issues. Thanks all! OK, I prototyped #2 in my sun8i-emac fixes series already. Will try to send= this out later as a discussion base. But still I get (false positive) warnings about unaligned flush() calls whe= n fixing the driver. So shall we apply this patch to get this out of the wa= y first? People can still bring back the checks by #define DEBUG. For see the impact of this problem I attach my analysis of the network driv= ers. This is best effort and probably not 100% correct in every aspect, so = bear with me if I misjudged your driver. But I think the general picture is evident: - Most (if not all) drivers used on ARM(32) boards do some kind of (mislead= ing) rounding. We don't see this in MIPS drivers, for instance. This tells = me that this is probably just to appease these unconditional checks. - There are far more length roundings than base roundings. - The ARMv7 implementation of flush/invalidate_dcache_range() is the only o= ne doing these checks (unconditionally). Another reason having this outside= of debug is a mistake and it should be reverted? Fixing those drivers is possible, but tedious. And would rely on those chec= ks being fixed first. Also it sounds hard to get this all properly tested. So shall we just fix t= hose drivers that people actually care about? Cheers, Andre. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cache_maint_net.txt URL: