Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DE355C4332F for ; Fri, 14 Oct 2022 16:44:50 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 2988C84BDB; Fri, 14 Oct 2022 18:44:48 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=konsulko.com header.i=@konsulko.com header.b="n3w8KMIk"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 2EA1A84DFC; Fri, 14 Oct 2022 18:44:46 +0200 (CEST) Received: from mail-qv1-xf2b.google.com (mail-qv1-xf2b.google.com [IPv6:2607:f8b0:4864:20::f2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id D283A84B42 for ; Fri, 14 Oct 2022 18:44:42 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=trini@konsulko.com Received: by mail-qv1-xf2b.google.com with SMTP id o67so3563157qvo.13 for ; Fri, 14 Oct 2022 09:44:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=OlgAmN/QU/s02ummXD6R3aLwfWrvGQ9ghPQRaQlVEdA=; b=n3w8KMIkG/o3jUQh96+rTBQdmK7fSIcBO58muM7Y2bmnSqAVOSgeN7TwGic6grRfcx gB9Wj7uN2136Dqzk9EK1Oz/YM4tuHBwrK3v4Aq9gI5lvqNESDhDCcK8EJMvDybkwZti7 Dofp1vw0Z+6mMUPuBUvh52RmUs3qxegAprs6A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=OlgAmN/QU/s02ummXD6R3aLwfWrvGQ9ghPQRaQlVEdA=; b=1VR6YTksu0VuuBCtXCD0hnZxoGixhH2k84lz0SW5PT4P0a2BrTc5vJyYXoasyb+M6x C0L1GNB+6ZTxZhAI8cZQSDjFZDrqYSXVlDMtJafn67IKepnWwP1YYeGNNmQIVXQdGvrS L01MB/nnhwb4VDJWksEdD5/NTaH8ttSip50lb3GSCzsU0NR4GTfm41hE7gVctPZHSZ2x 4AAdrPis+m2nE7QMOJAIfBoT2MQrEKMQuTDyp7IiAfRvWyY+f5ps3QPy4AZVBu2qQqZp qpzEWqMOmekxf6vHY2vZPMApDOD1mXhgBoJybL0yGWkTVD7zeukimLIKr0gc7Sn9jpTB IB4w== X-Gm-Message-State: ACrzQf3AldRiKhFBoQQyKrK+wz0gzwm95ChRI8GiOOjKAv0KhXXJ/mFk WsoaYWthNW84Ips5tZJ7vAkeRM9PHLAjwA== X-Google-Smtp-Source: AMsMyM5Og5D1n/xv11vW6Gg9dHZO80OG4/1e+GWudeyCxwBVYJBPeHORY8Np9dbOj2KtW1cH2UDdow== X-Received: by 2002:a05:6214:c66:b0:4b2:5ca6:8a84 with SMTP id t6-20020a0562140c6600b004b25ca68a84mr4686705qvj.110.1665765881614; Fri, 14 Oct 2022 09:44:41 -0700 (PDT) Received: from bill-the-cat (2603-6081-7b00-6400-b049-0e25-0576-9c40.res6.spectrum.com. [2603:6081:7b00:6400:b049:e25:576:9c40]) by smtp.gmail.com with ESMTPSA id fw11-20020a05622a4a8b00b0039c72bb51f3sm2316535qtb.86.2022.10.14.09.44.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 09:44:40 -0700 (PDT) Date: Fri, 14 Oct 2022 12:44:38 -0400 From: Tom Rini To: Simon Glass Cc: u-boot@lists.denx.de, Rasmus Villemoes Subject: Re: [PATCH 2/2] buildman: Add --allow-missing-binaries flag to build with BINMAN_ALLOW_MISSING=1 Message-ID: <20221014164438.GZ2020586@bill-the-cat> References: <20221010151831.3376759-1-trini@konsulko.com> <20221010151831.3376759-2-trini@konsulko.com> <20221011162235.GF2020586@bill-the-cat> <20221012145205.GX2020586@bill-the-cat> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="rQj5ySmgPRE3FKMj" Content-Disposition: inline In-Reply-To: X-Clacks-Overhead: GNU Terry Pratchett X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean --rQj5ySmgPRE3FKMj Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Oct 14, 2022 at 09:56:40AM -0600, Simon Glass wrote: > Hi Tom, >=20 > On Wed, 12 Oct 2022 at 08:52, Tom Rini wrote: > > > > On Wed, Oct 12, 2022 at 06:59:35AM -0600, Simon Glass wrote: > > > Hi Tom, > > > > > > On Tue, 11 Oct 2022 at 10:22, Tom Rini wrote: > > > > > > > > On Mon, Oct 10, 2022 at 05:48:55PM -0600, Simon Glass wrote: > > > > > Hi Tom, > > > > > > > > > > On Mon, 10 Oct 2022 at 09:18, Tom Rini wrote: > > > > > > > > > > > > Add a new flag to buildman so that we will in turn pass > > > > > > BINMAN_ALLOW_MISSING=3D1 to 'make'. Make use of this flag in CI. > > > > > > > > > > > > Cc: Rasmus Villemoes > > > > > > Cc: Simon Glass > > > > > > Signed-off-by: Tom Rini > > > > > > --- > > > > > > .azure-pipelines.yml | 2 +- > > > > > > .gitlab-ci.yml | 6 +++--- > > > > > > tools/buildman/builder.py | 5 ++++- > > > > > > tools/buildman/builderthread.py | 2 ++ > > > > > > tools/buildman/cmdline.py | 3 +++ > > > > > > tools/buildman/control.py | 3 ++- > > > > > > 6 files changed, 15 insertions(+), 6 deletions(-) > > > > > > > > > > > > diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml > > > > > > index f200b40dbb24..c932c2b3c619 100644 > > > > > > --- a/.azure-pipelines.yml > > > > > > +++ b/.azure-pipelines.yml > > > > > > @@ -553,7 +553,7 @@ stages: > > > > > > cat << "EOF" >> build.sh > > > > > > if [[ "${BUILDMAN}" !=3D "" ]]; then > > > > > > ret=3D0; > > > > > > - tools/buildman/buildman -o /tmp -P -E -W ${BUILD= MAN} ${OVERRIDE} || ret=3D$?; > > > > > > + tools/buildman/buildman -o /tmp -P -E -W --allow= -missing-binaries ${BUILDMAN} ${OVERRIDE} || ret=3D$?; > > > > > > > > > > This is fine for CI. > > > > > > > > I think one of the issues here is we need to agree on the common use > > > > cases. I don't think most people use buildman to build, they use ma= ke. > > > > > > OK. With the new -I/--ide flag I use buildman directly for single > > > builds on some machines at least. > > > > Of the people that do use buildman, how many aren't already using a > > > > wrapper script? I know I always do. > > > > > > I don't know, but I always use buildman directly for build-testing > > > multiple boards. > > > > I've been building for single machines with buildman for years, with > > --keep being passed to my wrapper script. And always use it for multiple > > boards. But this is you and I, to borrow a meme, we're builders georg > > and are outliers who should perhaps not be counted. >=20 > Yes, but we need to preserve that workflow. Right. > > > > Can we also not just deal with > > > > setting this flag in ~/.buildman ? Would it Just Work as: > > > > [builder] > > > > allow-missing-binaries =3D True > > > > > > > > Or is more code needed? > > > > > > Yes I think that would work, although I think we would need two flags, > > > one for building current source (where people might want it off) and > > > another for building multiple boards (when presumably you want it on). > > > > I'm not sure. It's not "always use fake binaries" it's allow them. I'm > > honestly not sure how to tell buildman to use external binaries > > correctly, that's one of the cases where I use a different script that > > just manages output directories and known-good additional binaries. >=20 > To me this is a policy thing. I just don't want to get 50 boards in > and find that the builds start failing due to annoying blobs, or have > it fail on the first blob when there is a binman error beyond just the > first missing blob that is now masked. Yeah, there's some policy here. But managing policy is what config files are for too, right? Heck, I could probably drop some of my wrapper script and move more of it to the config file if I was confident in the syntax / sections. Maybe that needs to be documented? > > > ...although I don't see the advantage of this approach over the series > > > I sent. Basically, the build should fail if there are missing blobs. > > > The only question is whether we want to handle missing blobs by: > > > > > > 1. failing immediately when we see the first missing blob (your appro= ach) > > > 2. failing at the end after all binman processing is done (my approac= h) > > > > > > The nice thing about 2 is that the build does complete and the exit > > > code lets buildman decide what to do afterwards (i.e. we can make > > > missing blobs be an error or a warning). > > > > > > Option 1 has the benefit that we don't do any of the blob handling, so > > > it just dies right away. Perhaps this is a philosophical question, but > > > it is a little subtle and I'm not actually sure people would notice > > > the difference so long as they get the errors as expected. > > > > The way I'm thinking of it is there's two cases. The first case is > > someone who is testing on the hardware that requires these files. Stop > > ASAP because they either will know they forgot to pass X/Y/Z files or > > they'll re-read the board instructions page and see oh, they need to > > grab binaries X/Y/Z too. Waiting to collect up missing file errors > > doesn't save time. It's also still hugely likely I believe that they're > > using make and not buildman. > > > > The second case is someone building multiple boards at once to > > build-check (but not run-time check) something in multiple places. This > > is where passing a specific failure exit code can be helpful, yes. >=20 > Yes that makes sense. >=20 > > > > > BTW we need a one-character flag if we do this as it will be a very > > > common requirement. > > > > Sadly both -a and -A are taken. We could use -M for Missing ? >=20 > SGTM. >=20 > If we apply some of the patches in my series then I think I could > enable that always, since the build will return an exit code if a blob > is missing, which I can disable with -W. I think so? > > > > > But having to provide a flag to do build testing seems like the t= ail > > > > > is wagging the dog. Boards should be discouraged to use blobs and= we > > > > > don't want to make it hard for everyone else (who doesn't have the > > > > > blobs) to test whether their patch breaks a build. > > > > > > > > I'm not sure this ends up being a common case. If someone is build > > > > testing for something they can't run test, the error is going to be > > > > after whatever they compile-tested was compiled/linked, and if they= 're > > > > testing everything it's via CI. > > > > > > I don't think people will be able to tell that. The 'make' fails with > > > an error so it looks like a failure. It happens to be at the end in > > > the binman step, but it is still a build failure. > > > > > > My overall concern is constantly having to rerun a build because I > > > forgot to add a flag, when I don't have the blobs for the boards > > > anyway. Like the LTO thing that cost 4 seconds on every build, that > > > could get really annoying. Blobs should not get in the way of > > > development. > > > > The counter problem is this isn't the first time someone has come up and > > noted how much time they've wasted because we defaulted to fake > > binaries. I think we've optimized too much for the people that build a > > thousand configs all the time (us) instead of the people that build for > > 1 or two configs at a time (most people?). > > > > To that end, I am really curious what Rasmus has to say here, or anyone > > else that has a different workflow from you and I. >=20 > The main problem I think is that 'binman' should have emitted an exit > code if there are missing blobs. That is [1] and I think was just an > oversight on my part, or not thinking through things clearly enough. OK, can you take / use as inspiration what you think makes sense of my series as well and v2 your series? I want to grab that "N:" patch asap since that unblocks a bunch of other platforms that are ready to come in, otherwise. --=20 Tom --rQj5ySmgPRE3FKMj Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmNJkesACgkQFHw5/5Y0 tyx2Pgv+ON6EUBXXl3EE+XFBo0uMTUfmYcO+CKIVMgBLJrpxtUghNkFOgDyi6bg1 jgqQLGQP388n8MfiqZbjjQltM38gg9XRkoz2FHahj3IC2WR7ybOmpdG/m/Gaf03Y O5/52QUdTyxSc41hwJspqJRxkSe3VhVW48amW0ClXKBoHK4ld5emYaomPIkwXbWl xbKUBYKQg2oV6HtTEhNkD0KjFdLQYA+yV9bhWbuRk+N5jPDO06xRCqLUiWfUf9br 3GYG/lK+7AFpSghnnu6XqDJA6ZheEA5MtbiTDbxDeZJX+AHgiVnLD4MFRC7w71// rouiZtOZaP/YBGLR5e6GYeejC200h8iPbFg+bdCbU60fxsS7Fz5YnBxmbfwKnkNa 60Sv38/ghcPN2EbsXcCsB7qnDa2uWJXk58d2hwYliraq1laGJ5vfI0d9o00JMSfG WgYQeziYF8efyNPleEKdreYLsPnVEX8zRncnFYc6rQ+4Jn5hD+f/4rTQGHZvscj7 GGDQZmqj =ZkVU -----END PGP SIGNATURE----- --rQj5ySmgPRE3FKMj--