Skip to contentSkip to conversation

sg-report-flight: Rename "sub" to more descriptive "relevant_flights"

PATCH New Ian Jackson 1 participant 3 messages Nov 19, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 sg-report-flight: Rename "sub" to more descriptive "relevant_flights"

No functional change. We're going to add another WITH...

---
 sg-report-flight | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
sg-report-flight +3 -2
index fd266586..40a3cc92 100755
@@ -316,7 +316,8 @@ END
     # best option.
 
     my $flightsq= <<END;
-      WITH sub AS (
+      WITH
+      relevant_flights AS (
         SELECT DISTINCT flight, blessing
              FROM flights
 $runvars_joins
@@ -328,7 +329,7 @@ $runvars_conds
             LIMIT 1000
       )
       SELECT flight, jobs.status
-        FROM sub
+        FROM relevant_flights
 $flightsq_jobs_join
        WHERE (1=1)
 $flightsq_jobcond
Signature · 1 line
2.20.1
#1 lore raw

PATCH 2/3 sg-report-flight: Actually look at retest flights (part 1)

The existing approach does not find retest flights. This is because it starts by looking for flights whose runvars say they built the version in question, and then looks to see if they contain the relevant job.

Retest flights don't contain build jobs; they reuse the builds from the primary flight.

Rather than making a fully general recursion scheme (which would involve adding another index so we could quickly find all flights which refer to this one), we add a one-level recursion.

This recursion is restricted to the flights named on the command line. This means it takes nearly no time (as opposed to searching the whole db for things that might be relevant - see above re need for an index).

We filter the command line flights, looking ones which refer to the only the primarily found flights as build jobs.

---
 sg-report-flight | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
sg-report-flight +24 -1
index 40a3cc92..70cad230 100755
@@ -315,6 +315,11 @@ END
     # Sadly this trick is not formally documented but it is our
     # best option.
 
+    my $cmdline_flight_cond =
+      join ' OR ',
+      (defined $specflight ? "f1.flight=$specflight" : "FALSE"),
+      map { "f1.flight=$_->{Flight}" } @refer_to_flights;
+
     my $flightsq= <<END;
       WITH
       relevant_flights AS (
@@ -327,9 +332,27 @@ $runvars_conds
             ORDER BY flight DESC
             OFFSET 0
             LIMIT 1000
+      ),
+      retest_flights AS (
+        SELECT DISTINCT f1.flight flight, f1.blessing blessing
+             FROM flights f1
+             JOIN jobs j USING (flight) 
+       CROSS JOIN relevant_flights f0
+            WHERE ($cmdline_flight_cond)
+              AND (
+                SELECT bool_and( val LIKE f0.flight || '.%' )
+                       IS TRUE
+                 FROM runvars r
+                WHERE r.flight=j.flight
+                  AND r.job=j.job 
+                  AND r.name LIKE '%buildjob'
+                  )
       )
       SELECT flight, jobs.status
-        FROM relevant_flights
+        FROM (
+           SELECT * FROM relevant_flights
+     UNION SELECT * FROM retest_flights
+             ) all_relevant_flights
 $flightsq_jobs_join
        WHERE (1=1)
 $flightsq_jobcond
Signature · 1 line
2.20.1
#2 lore raw

PATCH 3/3 sg-report-flight: Actually look at retest flights (part 2)

In reply to Ian Jackson

To avoid going down ratholes (especially for jobs which reuse outputs from their previous selves), the primary flight/job finder in sg-report-flight does not recurse indefinitely through build jobs. Instead, it restricts the build jobs investigated to those within the same flight as the job which might be of interest.

As a result, retest jobs are, unfortunately, discarded at this stage because we insist that the build jobs we find did use the tree revision we are investigating.

Fix this by recursing into the corresponding primary flight too. In the $flightsq->fetchrow loop that's $xflight.

For the primary flight, ie the first half of the UNION, that's just the fligth itself. So there this has no change.

For the retest flights, it is the flight that all the build jobs refer to. Despite the CROSS JOIN, this will be unique for any particular "retest flight", because the query on the runvars insists that all of the (at least some) buildjob runvars for f1 point to f0. Ie, f1 has no build jobs and refers to f0 for build jobs; so it can't refer to any other f0' in the cross join.

With this change, a -retest flight can now actually be used to justify a push.

---
 sg-report-flight | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)
sg-report-flight +15 -7
index 70cad230..7f0543ff 100755
@@ -334,9 +334,12 @@ $runvars_conds
             LIMIT 1000
       ),
       retest_flights AS (
-        SELECT DISTINCT f1.flight flight, f1.blessing blessing
+        SELECT DISTINCT 
+                  f1.flight flight,
+                  f0.flight xflight,
+                  f1.blessing blessing
              FROM flights f1
-             JOIN jobs j USING (flight) 
+             JOIN jobs j USING (flight)
        CROSS JOIN relevant_flights f0
             WHERE ($cmdline_flight_cond)
               AND (
@@ -348,10 +351,10 @@ $runvars_conds
                   AND r.name LIKE '%buildjob'
                   )
       )
-      SELECT flight, jobs.status
+      SELECT flight, xflight, jobs.status
         FROM (
-           SELECT * FROM relevant_flights
-     UNION SELECT * FROM retest_flights
+ SELECT        flight, flight xflight, blessing FROM relevant_flights
+ UNION SELECT  flight,        xflight, blessing FROM retest_flights
              ) all_relevant_flights
 $flightsq_jobs_join
        WHERE (1=1)
@@ -392,7 +395,7 @@ END
                 WHERE flight=?
 END
 
-    while (my ($tflight, $tjstatus) = $flightsq->fetchrow_array) {
+    while (my ($tflight, $xflight, $tjstatus) = $flightsq->fetchrow_array) {
 	# Recurse from the starting flight looking for relevant build
 	# jobs.  We start with all jobs in $tflight, and for each job
 	# we also process any other jobs it refers to in *buildjob runvars.
@@ -408,6 +411,11 @@ END
 	# actually interested in (ii) the kind of continuous update
 	# thing seen with freebsdbuildjob.
 	#
+	# However, when we are looking at a retest flight (offered to
+	# us with --refer-to-flights, or found because it was
+	# specified on the command line) we look at its build flight;
+	# that's what $xflight is;
+	#
 	# (This is rather different to cs-bisection-step, which is
 	# less focused on changes in a particular set of trees.)
 	#
@@ -435,7 +443,7 @@ END
 	while (@binfos_todo) {
 	    my ($why,$bflight,$bjob) = @{ shift @binfos_todo };
 	    next if $binfos{$bflight}{$bjob};
-            next unless $bflight == $tflight;
+            next unless $bflight == $tflight || $bflight == $xflight;
 	    $binfos{$bflight}{$bjob} = $why;
 	    $buildjobsq->execute($bflight,$bjob);
 	    $binfos_queue->($bflight,$buildjobsq,$why);
Signature · 1 line
2.20.1