From: Andre Przywara Date: Thu, 19 Dec 2019 12:32:09 +0000 Subject: [RFC PATCH] Revert "arm: Show cache warnings in U-Boot proper only" In-Reply-To: <70ecd325-00e9-6229-b057-c0aa2b8285c4@denx.de> References: <20191219005211.25327-1-andre.przywara@arm.com> <487d6382-7445-af7f-4c30-36673e837e61@denx.de> <16cefb13-820d-49b2-8ba6-8d1726df7660@arm.com> <74491a93-9cf5-5540-6611-71b365a54b7d@denx.de> <20191219113604.0d031e62@donnerap.cambridge.arm.com> <70ecd325-00e9-6229-b057-c0aa2b8285c4@denx.de> Message-ID: <20191219123209.763dbc5a@donnerap.cambridge.arm.com> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable To: u-boot@lists.denx.de On Thu, 19 Dec 2019 12:43:35 +0100 Marek Vasut wrote: Hi Marek, > On 12/19/19 12:36 PM, Andre Przywara wrote: > > On Thu, 19 Dec 2019 09:04:32 +0100 > > Marek Vasut wrote: > >> On 12/19/19 2:23 AM, Andr=C3=A9 Przywara wrote: =20 > >>> On 19/12/2019 00:55, Marek Vasut wrote: > >>>> On 12/19/19 1:52 AM, Andre Przywara wrote: =20 > >>>>> According to commit 11aa6a32eb5f ("arm: cache: Implement cache range > >>>>> check for v7"), which introduced check_cache_range(), this was meant > >>>>> as a pure debugging feature, only to be compiled in when a developer > >>>>> explicitly #defined DEBUG in cache.c. Presumably the intention was = to > >>>>> help with finding *certain* alignment issues with DMA buffers. > >>>>> > >>>>> Commit bcc53bf09589 ("arm: Show cache warnings in U-Boot proper onl= y") > >>>>> compiled this in *unconditionally* into U-Boot proper. > >>>>> > >>>>> This has the annoying side effect of producing tons of somewhat > >>>>> pointless warnings about non-aligned clean&invalidate operations, w= hich > >>>>> tend to be appeased by even more pointless rounding operations in m= any > >>>>> drivers (mostly those used on ARM boards). > >>>>> > >>>>> Bring back the old behaviour, of only compiling this in for DEBUG > >>>>> situations, but staying silent otherwise. > >>>>> > >>>>> This reverts commit bcc53bf095893fbdae531a9a7b5d4ef4a125a7fc. > >>>>> > >>>>> Signed-off-by: Andre Przywara > >>>>> --- > >>>>> Hi, > >>>>> > >>>>> if the intention was indeed to always force cache maintenance range > >>>>> alignments, I would like to open a discussion on this, because I be= lieve > >>>>> it is not useful, especially in the clean&invalidate case. =20 > >>>> > >>>> Why don't you rather fix the cache op alignment bugs ? =20 > >>> > >>> Which bugs do you mean? =20 > >> > >> Well, the ones reported by this warning. =20 > >=20 > > As mentioned, most of them are just silenced in drivers, though not act= ually fixed. =20 >=20 > Do you have specific details here ? They should be fixed. >=20 > > This patch here was actually triggered by me cleaning up sun8i-emac.c, = which then made those warnings appear again. > > I will try to send out this series, which might make it more clear what= I am after. > > =20 > >>> Those that would currently trigger those warnings? > >>> I don't think they are actual bugs, besides I don't think they are any > >>> cases left for 32-bit ARM boards (leave alone the new RPi4 Ethernet > >>> driver in the rpi-4-32b_defconfig). =20 > >> > >> If your cache isn't flushed or invalidated properly, it's a bug. > >> On ARM32, this is (was, before this warning) a real problem. =20 > >=20 > > The actual cache maintenance is fine in the drivers, it's the over-stri= ct alignment requirements that bothers me and makes things actually worse: > >=20 > > Architecturally (ARM ARM) there is no alignment requirement for VA oper= ations. Walking over a range of addresses does get easier if you start on a= cache line boundary, so this is what we do in the implementations for v7 a= nd v8. =20 >=20 > U-Boot is not only V7 and V8, but also all the earlier ones (back to V4). I checked v6 as well (same), and couldn't find any evidence that this would= be different for v5 or v4, although CP15 does not seem to be architectural= ly defined there. Do you have an example of an older core failing that? But anyway this particular check and patch is affecting v7 only. > > Alignment gets critical when it comes to a pure invalidate call. If you= have other dirty data in this same cache line, you might lose the latest s= tores, that's why it is important to align those *buffers*. So the strict c= heck we have in v7_dcache_inval_range() is the right thing (at least for th= e base address). What is not right is to deliberately align just the *param= eters* of the invalidate call, as seens in net/sun8i_emac.c or net/dwc_eth_= qos.c, for instance. This could actually hide bugs. Drivers should always p= ass on the actual buffer address to invalidate(). =20 >=20 > Sure, but that's a driver bug. That's a separate topic from this patch. >=20 > > Then there is no real need to align the range for flush (clean&invalida= te) calls. Overshooting here is not critical, and again we do align them an= yway in the implementation. =20 >=20 > Wrong, you may interfere with some DMA engine which might be accessing > the area which you just overshot. Can you give an example on that? Are you thinking about cleaning dirty cach= e lines to a DRAM region which DMA is just writing data into? After all a clean could happen at any time, when a line gets evicted from t= he cache, without an explicit cache maintenance instruction. So if you have= such a case, that's a separate bug. > > This > > forced check only leads to drivers aligning the *parameters* of these c= alls, to no actual effect, probably given a wrong impression to driver auth= ors and readers. >=20 > Again, driver bug, that's unrelated and must be fixed. If you fix them, they will trigger tons of warnings, mostly bogus, about en= d addresses not being aligned (see below). Been there, done that. That's wh= y this patch as a first step to allow those fixes without spamming the user. > > So in preparation for fixing the drivers, I ask for those checks to bec= ome pure DEBUG warnings. Another alternative would be to relax those checks= , to just require the start address of invalidate to be aligned - but this = is what we check separately already. =20 >=20 > See above. >=20 > >>> Or those that are currently hidden because we *force* an alignment on > >>> the *arguments* passed to invalidate_dcache_range, for instance? > >>> These are quite numerous, so I would rather get some input first befo= re > >>> spending a lot of time on this. =20 > >> > >> Make sure your buffers are properly aligned and you have no cache > >> issues. It's really that easy. If you see numerous, then your platform > >> has a real problem which must be fixed ; I didn't see any for quite a > >> while on platforms I have access to. =20 > >=20 > > Yes, I think the buffers are correctly aligned in the drivers, as least= the ones I checked. But then we should always pass this address to invalid= ate, and not round it just before calling invalidate. Same goes to some deg= ree for the end address, although is a somewhat separate topic. =20 >=20 > Of course the address passed to cache ops should not be rounded up. > That's a bug (unless the buffer actually covers that rounded address, > which sometimes is the case, and might or might not thus be harmless. > That's up to the driver author to determine) Yes, and that's why the strict unconditional check we have right now is not= helping. We should have them in, and developers could check the warnings f= or being legit or not. =20 > > And that's why I think having check_cache_range() is useful for develop= ment and debugging, but only then, because not all warnings are legit (end = addresses). =20 >=20 > This implication makes no sense. How do you infer from "drivers have > bugs" that a warning about such bugs should be turned into a DEBUG()? Most (if not all) of those warnings come from the end address not being ali= gned. And this is something not easily being checked outside of the driver.= What we want to check for invalidate is that we only invalidate a well kno= wn region, so: 1) The start address of the buffer is aligned. 2) The end address, rounded up to the next cache line, is still within our = buffer (so nothing else is using the same cacheline). We can check 1) easily, but not 2), that would require driver specific code= , which knows about the size of the buffer. We could provide something like: bool check_cache_buffer(void *start_address, size_t length, size_t buffer= _size); which drivers would call explicitly. Or we could include the buffer size in= to the invalidate call, but that would change the interface. That would at least cover the buffers that the driver provides itself (RX b= uffers for network drivers). Network TX buffers are handed in from the gene= ric network code, and the driver does not know what is there after "length"= . So the best thing we can do is to call "flush_dcache_range(packet, packet= + length);", and then should not receive warnings if the end address is un= aligned. That's why I think checks for flush are not useful. Cheers, Andre