Skip to contentSkip to conversation

progress: verify progress counters in the test suite

PATCH In review SZEDER Gábor 12 participants 197 messages Jun 20, 2021 – Feb 3, 2022 Page 2 of 2

lore
Share & tools
lore
#153 lore raw

Re: [PATCH v3 10/10] progress.c: add & assert a "global_progress" variable

In reply to SZEDER Gábor
Quoted 24 lines · SZEDER Gábor
Quoted 4 lines · Ævar Arnfjörð Bjarmason

The progress.c code makes a hard assumption that only one progress bar be active at a time (see [1] for a bug where this wasn't the case). Add a BUG() that'll trigger if we ever regress on that promise and have two progress bars active at the same time.

I still very much dislike the idea of a BUG() in the progress code that can trigger outside of the test suite, because the progress line is only a UI gimmick and not a crucial part of any Git operation, and even though a progress line might be buggy, the underlying Git operation is not affected by it and would still finish successfully, as was the case with the dozen of so progress line bugs in the past.

Quoted 7 lines

There was an alternative test-only approach to doing the same thing[2], but by doing this outside of a GIT_TEST_* mode we'll know we've put a hard stop to this particular API misuse.

It will also establish scaffolding to address current fundamental limitations in the progress output: The current output must be "driven" by calls to the likes of display_progress().

Please elaborate why that is a "fundamental limitation"; I don't see any drawback of the current approach.

Quoted 7 lines
Quoted 3 lines

Once we have a
global current progress object we'll be able to update that object via SIGALRM.

What are the supposed benefits of doing that? I do see its drawbacks, considering that we have progress lines that are updated from multiple threads.

I've updated the commit messages in a re-roll I have incoming to hopefully clear this up.

Quoted 75 lines
Quoted 55 lines

See [3] for early code to do that.

It's conceivable that this change will hit the BUG() condition in some scenario that we don't currently have tests for, this would be very bad. If that happened we'd die just because we couldn't emit some pretty output.

See [4] for a discussion of why our test coverage is lacking; our progress display is hidden behind isatty(2) checks in many cases, so the test suite doesn't cover it unless individual tests are run in "--verbose" mode, we might also have multi-threaded use of the API, so two progress bars stopping and starting would only be visible due to a race condition.

Despite that, I think that this change won't introduce such regressions, because:

1. I've read all the code using the progress API (and have modified a large part of it in some WIP code I have). Almost all of it is really simple, the parts that aren't[5] are complex in the display_progress() part, not in starting or stopping the progress bar.

2. The entire test suite passes when instrumented with an ad-hoc Linux-specific mode (it uses gettid()) to die if progress bars are ever started or stopped on anything but the main thread[6].

Extending that to die if display_progress() is called in a thread reveals that we have exactly two users of the progress bar under threaded conditions, "git index-pack" and "git pack-objects". Both uses are straightforward, and they don't start/stop the progress bar when threads are active.

3. I've likewise done an ad-hoc test to force progress bars to be displayed with:

perl -pi -e 's[isatty\(2\)][1]g' $(git grep -l -F 'isatty(2)')

I.e. to replace all checks (not just for progress) of checking whether STDERR is connected to a TTY, and then monkeypatching is_foreground_fd() in progress.c to always "return 1". Running the tests with those applied, interactively and under -V reveals via:

$ grep -e set_progress_signal -e clear_progress_signal test-results/*out

That nothing our tests cover hits the BUG conditions added here, except the expected "BUG: start two concurrent progress bars" test being added here.

That isn't entirely true since we won't be getting 100% coverage due to cascading failures from tests that expected no progress output on stderr. To make sure I covered 100% I also tried making the display() function in progress.c a NOOP on top of that (it's the calls to start_progress_delay() and stop_progress()) that matter.

Quoted 7 lines

That doesn't hit the BUG() either. Some tests fail in that mode due to a combination of the overzealous isatty(2) munging noted above, and the tests that are testing that the progress output itself is present (but for testing I'd made display() a NOOP).

Between those three points I think it's safe to go ahead with this change.

The above analysis only considers _our_ _current_ codebase.

However, even though this might be safe now, it doesn't mean that it will remain safe in the future, as we might add new progress lines that lack test coverage (though hopefully won't), and would hit that BUG() at a user.

Furthermore, even though this might be safe in our codebase, it doesn't mean that it is safe in some 20+k forks of Git that exist on GitHub alone (I for one have a git command or two with in my fork which output progress lines but, sadly, have zero test coverage).

Quoted 5 lines

But more importantly, even though it might be safe to do so, that doesn't mean that it's a good idea to do so. The commit message does little to justify why it is conceptually a good idea to add this BUG() to the progress code in a way that it can trigger outside of the test suite.

I think partially I've addressed this above (i.e. in the incoming re-roll's update commit message), but I might not for the question of whether this is worth it overall. I'll update the commit message to address this specific case, which was missing from it.

Quoted 58 lines
Quoted 30 lines

1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)
2. https://lore.kernel.org/git/[email protected]
3. https://lore.kernel.org/git/[email protected]/
4. https://lore.kernel.org/git/[email protected]/
5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into
next, 2021-09-10)
6. https://lore.kernel.org/git/[email protected]/

---
 progress.c                  | 18 ++++++++++++++----
 t/t0500-progress-display.sh | 11 +++++++++++
 2 files changed, 25 insertions(+), 4 deletions(-)
progress.c +1 -0
index b9369e9a264..a31500f8b2b 100644
@@ -46,6 +46,7 @@ struct progress {
 };
 
 static volatile sig_atomic_t progress_update;
+static struct progress *global_progress;
 
 /*
  * These are only intended for testing the progress output, i.e. exclusively
@@ -219,11 +220,16 @@ void progress_test_force_update(void)
 	progress_interval(SIGALRM);
 }
 

progress.c void progress_test_force_update(void) · line 231

Quoted 10 lines
progress.c +5 -1
-static void set_progress_signal(void)
+static void set_progress_signal(struct progress *progress)
 {
 	struct sigaction sa;
 	struct itimerval v;
 
+	if (global_progress)
+		BUG("'%s' progress still active when trying to start '%s'",
+		    global_progress->title, progress->title);
+	global_progress = progress;

This function is called set_progress_signal(), so checking and setting 'global_progress' feels out of place here; it would be better to do that in start_progress_delay().

Quoted 13 lines
progress.c +2 -1
+
 	if (progress_testing)
 		return;
 
@@ -241,10 +247,14 @@ static void set_progress_signal(void)
 	setitimer(ITIMER_REAL, &v, NULL);
 }
 
-static void clear_progress_signal(void)
+static void clear_progress_signal(struct progress *progress)
 {
 	struct itimerval v = {{0,},};
 
Quoted 5 lines

progress.c static void set_progress_signal(void) · line 256

Quoted 3 lines
progress.c +3 -0
+	if (!global_progress)
+		BUG("should have active global_progress when cleaning up");
+	global_progress = NULL;

Likewise.

*Nod* cleaned up this whole part, which became much simpler overall as a result, thank you.

#154 lore raw

Re: [PATCH v3 00/10] progress: assert "global_progress" + test fixes / cleanup

In reply to Ævar Arnfjörð Bjarmason
Quoted 18 lines · Ævar Arnfjörð Bjarmason

This series fixes various issues in and related to progress.c, and adds a BUG() assertion for us not starting two progress bars at the same time. Those changes are needed for subsequent changes that do more interesting things with this new global progress bar.

This v3 hopefully addresses all the feedback on the v2, thanks all. Changes:

* Fix a memory leak in 1/10, and make the progress tests use the SANITIZE=leak test mode.

* Simplified some of the test-progress.c code (no more "start" handling, the "total" count is mandatory now.

* Split out a formatting change into 2/10 to make 3/10 easier to read.

Quoted 6 lines

* A new 9/10 makes an ad-hoc test recipie in 10/10 easier to explain (in response to Emily's comment).

* The BUG() assertion in 10/10 now has a much better message, we dump the title of the two progress bars in play if we have a bug where we started two at the same time.

One thing I noticed (and so did SZEDER) was the addition of string_list to the helper; I made a comment accordingly in the patch where it was added. It kind of seemed like an oversight - but I remember you were wanting to head in that direction (multiple progress bar support) so maybe it was paving the way for that?

Anyways, with or without it the series looks OK to me in this round too.

- Emily

#155 lore raw

Re: [PATCH v3 00/10] progress: assert "global_progress" + test fixes / cleanup

Quoted 26 lines · Emily Shaffer
Quoted 18 lines · Ævar Arnfjörð Bjarmason

This series fixes various issues in and related to progress.c, and adds a BUG() assertion for us not starting two progress bars at the same time. Those changes are needed for subsequent changes that do more interesting things with this new global progress bar.

This v3 hopefully addresses all the feedback on the v2, thanks all. Changes:

* Fix a memory leak in 1/10, and make the progress tests use the SANITIZE=leak test mode.

* Simplified some of the test-progress.c code (no more "start" handling, the "total" count is mandatory now.

* Split out a formatting change into 2/10 to make 3/10 easier to read.

Quoted 6 lines

* A new 9/10 makes an ad-hoc test recipie in 10/10 easier to explain (in response to Emily's comment).

* The BUG() assertion in 10/10 now has a much better message, we dump the title of the two progress bars in play if we have a bug where we started two at the same time.

Quoted 7 lines

One thing I noticed (and so did SZEDER) was the addition of string_list to the helper; I made a comment accordingly in the patch where it was added. It kind of seemed like an oversight - but I remember you were wanting to head in that direction (multiple progress bar support) so maybe it was paving the way for that?

Anyways, with or without it the series looks OK to me in this round too.

Replied to in the thread with SZEDER at:
https://lore.kernel.org/git/[email protected]/

I.e. it's not for multi-progress-bar support, but because the API doesn't xstrdup() the title. The loop handing it the title needs to keep the string it gave to it alive.

#156 lore raw

v4 cover progress: assert "global_progress" + test fixes / cleanup

In reply to Ævar Arnfjörð Bjarmason

This series fixes various issues in and related to progress.c, and adds a BUG() assertion for us not starting two progress bars at the same time. Those changes are needed for subsequent changes that do more interesting things with this new global progress bar.

I think this should address all the feedback on v3[1]. Changes:

* Clarified the whole string_list_insert() in test-progress.c. I added a wrapper function with a comment to make that code flow clearer, and as a bonus could drop braces and a temporary variable from the main function as a result.

* I elaborated on the "limitations in the progress output" part of the commit message, and tried to address the safety concerns SZEDER expressed in [2]. I think we might still disagree on whether 8/8 is worth it overall, but hopefully this addreses any outstanding questions about the approach & why it's being taken.

* SZEDER pointed out that I should move the BUG() assert out of the signal handlers, there's now a couple of small helpers to do that, which also allowed for ejecting a couple of earlier refactoring commits (code moves as they were "static").

1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/

Ævar Arnfjörð Bjarmason (8):
leak tests: fix a memory leaks in "test-progress" helper progress.c test helper: add missing braces
progress.c tests: make start/stop commands on stdin progress.c tests: test some invalid usage
progress.c: add temporary variable from progress struct pack-bitmap-write.c: don't return without stop_progress() various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO) progress.c: add & assert a "global_progress" variable

 builtin/bisect--helper.c    |   2 +-
 builtin/bundle.c            |   2 +-
 compat/mingw.c              |   2 +-
 pack-bitmap-write.c         |   6 +--
 progress.c                  |  23 +++++++-
 t/helper/test-progress.c    |  52 +++++++++++++-----
 t/t0500-progress-display.sh | 105 ++++++++++++++++++++++++++++--------
 7 files changed, 150 insertions(+), 42 deletions(-)

Range-diff against v3:

 1:  40f7c438a1e =  1:  a3bd032d1eb leak tests: fix a memory leaks in "test-progress" helper
 2:  ee177d253a8 =  2:  e441cfea7c5 progress.c test helper: add missing braces
 3:  045d58d8201 !  3:  1c5f9bdfe6d progress.c tests: make start/stop verbs on stdin
    @@ Metadata
     Author: Ævar Arnfjörð Bjarmason <[email protected]>
     
      ## Commit message ##
    -    progress.c tests: make start/stop verbs on stdin
    +    progress.c tests: make start/stop commands on stdin
     
         Change the usage of the "test-tool progress" introduced in
         2bb74b53a49 (Test the progress display, 2019-09-16) to take command
    @@ t/helper/test-progress.c
      #include "progress.h"
      #include "strbuf.h"
     +#include "string-list.h"
    ++
    ++/*
    ++ * We can't use "end + 1" as an argument to start_progress() below, it
    ++ * doesn't xstrdup() its "title" argument. We need to hold onto a
    ++ * valid "char *" for it until the end.
    ++ */
    ++static char *dup_title(struct string_list *titles, const char *title)
    ++{
    ++	return string_list_insert(titles, title)->string;
    ++}
      
      int cmd__progress(int argc, const char **argv)
      {
     -	int total = 0;
     -	const char *title;
     +	const char *const default_title = "Working hard";
    -+	struct string_list list = STRING_LIST_INIT_DUP;
    -+	const struct string_list_item *item;
    ++	struct string_list titles = STRING_LIST_INIT_DUP;
      	struct strbuf line = STRBUF_INIT;
     -	struct progress *progress;
     +	struct progress *progress = NULL;
    @@ t/helper/test-progress.c
     -		if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
     +		if (skip_prefix(line.buf, "start ", (const char **) &end)) {
     +			uint64_t total = strtoull(end, &end, 10);
    -+			if (*end == '\0') {
    ++			if (*end == '\0')
     +				progress = start_progress(default_title, total);
    -+			} else if (*end == ' ') {
    -+				item = string_list_insert(&list, end + 1);
    -+				progress = start_progress(item->string, total);
    -+			} else {
    ++			else if (*end == ' ')
    ++				progress = start_progress(dup_title(&titles,
    ++								    end + 1),
    ++							  total);
    ++			else
     +				die("invalid input: '%s'\n", line.buf);
    -+			}
     +		} else if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
      			uint64_t item_count = strtoull(end, &end, 10);
      			if (*end != '\0')
    @@ t/helper/test-progress.c: int cmd__progress(int argc, const char **argv)
      	}
     -	stop_progress(&progress);
      	strbuf_release(&line);
    -+	string_list_clear(&list, 0);
    ++	string_list_clear(&titles, 0);
      
      	return 0;
      }
 4:  efc0ec360cc =  4:  474ce31f9d2 progress.c tests: test some invalid usage
 5:  9e36f03de46 <  -:  ----------- progress.c: move signal handler functions lower
 6:  c7c3843564e <  -:  ----------- progress.c: call progress_interval() from progress_test_force_update()
 7:  cd2d27b1626 =  5:  ff039742148 progress.c: add temporary variable from progress struct
 8:  e0a3510dd88 =  6:  3dfe31decff pack-bitmap-write.c: don't return without stop_progress()
 9:  2cf14881ecf =  7:  8a18eb40fae various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)
10:  01d5bbfce76 !  8:  06124e8ac5e progress.c: add & assert a "global_progress" variable

@@ Commit message
limitations in the progress output: The current output must be "driven" by calls to the likes of display_progress(). Once we have a global current progress object we'll be able to update that object via
- SIGALRM. See [3] for early code to do that.
+ SIGALRM, this will cover cases where we're busy, but either haven't
+ invoked our first display_progress() yet, or the time between
+ display_progress() is too long. See [3] for early code to do that.

It's conceivable that this change will hit the BUG() condition in some scenario that we don't currently have tests for, this would be very
@@ Commit message
above, and the tests that are testing that the progress output itself is present (but for testing I'd made display() a NOOP).

- Between those three points I think it's safe to go ahead with this
+ This doesn't address any currently out-of-tree user of progress.c,
+ i.e. WIP patches, or progress output that's a part of forks of
+ git.git. Those hopefully have test coverage that would expose the
+ BUG().
+
+ If they don't they'll either run into it in code that displays more
+ than one progress bar for the lifetime of the progress, or which calls
+ stop_progress() with a non-NULL "progress" without a corresponding
+ start_progress(). Both of those cases are less likely than the general
+ cases of progress.c API misuse.
+
+ Between those three points above and the discussion of how this could
+ impact out-of-tree users I think it's safe to go ahead with this
change.

1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)
@@ progress.c: struct progress {

/*
* These are only intended for testing the progress output, i.e. exclusively
-@@ progress.c: void progress_test_force_update(void)
- progress_interval(SIGALRM);
+@@ progress.c: void display_progress(struct progress *progress, uint64_t n)
+ display(progress, n, NULL);
}

--static void set_progress_signal(void)
-+static void set_progress_signal(struct progress *progress)
- {
- struct sigaction sa;
- struct itimerval v;
-
++static void set_global_progress(struct progress *progress)
++{
+ if (global_progress)
+ BUG("'%s' progress still active when trying to start '%s'",
+ global_progress->title, progress->title);
+ global_progress = progress;
++}
+
- if (progress_testing)
- return;
-
-@@ progress.c: static void set_progress_signal(void)
- setitimer(ITIMER_REAL, &v, NULL);
- }
-
--static void clear_progress_signal(void)
-+static void clear_progress_signal(struct progress *progress)
+ static struct progress *start_progress_delay(const char *title, uint64_t total,
+ unsigned delay, unsigned sparse)
{
- struct itimerval v = {{0,},};
-
-+ if (!global_progress)
-+ BUG("should have active global_progress when cleaning up");
-+ global_progress = NULL;
-+
- if (progress_testing)
- return;
-
@@ progress.c: static struct progress *start_progress_delay(const char *title, uint64_t total,
strbuf_init(&progress->counters_sb, 0);
progress->title_len = utf8_strwidth(title);
progress->split = 0;
-- set_progress_signal();
-+ set_progress_signal(progress);
++ set_global_progress(progress);
+ set_progress_signal();
trace2_region_enter("progress", title, the_repository);
return progress;
+@@ progress.c: void stop_progress(struct progress **p_progress)
+ stop_progress_msg(p_progress, _("done"));
}
+
++static void unset_global_progress(void)
++{
++ if (!global_progress)
++ BUG("should have active global_progress when cleaning up");
++ global_progress = NULL;
++}
++
+ void stop_progress_msg(struct progress **p_progress, const char *msg)
+ {
+ struct progress *progress;
@@ progress.c: void stop_progress_msg(struct progress **p_progress, const char *msg)
- display(progress, progress->last_value, buf);
free(buf);
}
-- clear_progress_signal();
-+ clear_progress_signal(progress);
+ clear_progress_signal();
++ unset_global_progress();
strbuf_release(&progress->counters_sb);
if (progress->throughput)
strbuf_release(&progress->throughput->display);

Signature · 1 line
2.33.1.1510.ge5c82eefb93
#157 lore raw

PATCH v4 1/8 leak tests: fix a memory leaks in "test-progress" helper

Fix a memory leak in the test-progress helper, and mark the corresponding "t0500-progress-display.sh" test as being leak-free under SANITIZE=leak. This fixes a leak added in 2bb74b53a4 (Test the progress display, 2019-09-16).

My 48f68715b14 (tr2: stop leaking "thread_name" memory, 2021-08-27) had fixed another memory leak in this test (as it did some trace2 testing).

---
 t/helper/test-progress.c    | 1 +
 t/t0500-progress-display.sh | 1 +
 2 files changed, 2 insertions(+)
t/helper/test-progress.c +1 -0
index 5d05cbe7894..9265e6ab7cf 100644
@@ -69,6 +69,7 @@ int cmd__progress(int argc, const char **argv)
 			die("invalid input: '%s'\n", line.buf);
 	}
 	stop_progress(&progress);
+	strbuf_release(&line);
 
 	return 0;
 }
t/t0500-progress-display.sh +1 -0
index 22058b503ac..f37cf2eb9c9 100755
@@ -2,6 +2,7 @@
 
 test_description='progress display'
 
+TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 show_cr () {
Signature · 1 line
2.33.1.1510.ge5c82eefb93
#158 lore raw

PATCH v4 2/8 progress.c test helper: add missing braces

In reply to Ævar Arnfjörð Bjarmason

If we have braces on one arm of an if/else all of them should have it, per the CodingGuidelines's "When there are multiple arms to a conditional[...]" advice. This formatting change makes a subsequent commit smaller.

---
 t/helper/test-progress.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
t/helper/test-progress.c +3 -2
index 9265e6ab7cf..50fd3be3dad 100644
@@ -63,10 +63,11 @@ int cmd__progress(int argc, const char **argv)
 				die("invalid input: '%s'\n", line.buf);
 			progress_test_ns = test_ms * 1000 * 1000;
 			display_throughput(progress, byte_count);
-		} else if (!strcmp(line.buf, "update"))
+		} else if (!strcmp(line.buf, "update")) {
 			progress_test_force_update();
-		else
+		} else {
 			die("invalid input: '%s'\n", line.buf);
+		}
 	}
 	stop_progress(&progress);
 	strbuf_release(&line);
Signature · 1 line
2.33.1.1510.ge5c82eefb93
#159 lore raw

PATCH v4 3/8 progress.c tests: make start/stop commands on stdin

In reply to Ævar Arnfjörð Bjarmason

Change the usage of the "test-tool progress" introduced in 2bb74b53a49 (Test the progress display, 2019-09-16) to take command like "start" and "stop" on stdin, instead of running them implicitly.

This makes for tests that are easier to read, since the recipe will mirror the API usage, and allows for easily testing invalid usage that would yield (or should yield) a BUG(), e.g. providing two "start" calls in a row. A subsequent commit will add such tests.

---
 t/helper/test-progress.c    | 46 ++++++++++++++++++++++-------
 t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------
 2 files changed, 72 insertions(+), 32 deletions(-)
t/helper/test-progress.c +35 -11
index 50fd3be3dad..1435c28e950 100644
@@ -3,6 +3,9 @@
  *
  * Reads instructions from standard input, one instruction per line:
  *
+ *   "start <total>[ <title>]" - Call start_progress(title, total),
+ *                               Uses the default title of "Working hard"
+ *                               if the " <title>" is omitted.
  *   "progress <items>" - Call display_progress() with the given item count
  *                        as parameter.
  *   "throughput <bytes> <millis> - Call display_throughput() with the given
@@ -10,6 +13,7 @@
  *                                  specify the time elapsed since the
  *                                  start_progress() call.
  *   "update" - Set the 'progress_update' flag.
+ *   "stop" - Call stop_progress().
  *
  * See 't0500-progress-display.sh' for examples.
  */
@@ -19,34 +23,52 @@
 #include "parse-options.h"
 #include "progress.h"
 #include "strbuf.h"
+#include "string-list.h"
+
+/*
+ * We can't use "end + 1" as an argument to start_progress() below, it
+ * doesn't xstrdup() its "title" argument. We need to hold onto a
+ * valid "char *" for it until the end.
+ */
+static char *dup_title(struct string_list *titles, const char *title)
+{
+	return string_list_insert(titles, title)->string;
+}
 
 int cmd__progress(int argc, const char **argv)
 {
-	int total = 0;
-	const char *title;
+	const char *const default_title = "Working hard";
+	struct string_list titles = STRING_LIST_INIT_DUP;
 	struct strbuf line = STRBUF_INIT;
-	struct progress *progress;
+	struct progress *progress = NULL;
 
 	const char *usage[] = {
-		"test-tool progress [--total=<n>] <progress-title>",
+		"test-tool progress <stdin",
 		NULL
 	};
 	struct option options[] = {
-		OPT_INTEGER(0, "total", &total, "total number of items"),
 		OPT_END(),
 	};
 
 	argc = parse_options(argc, argv, NULL, options, usage, 0);
-	if (argc != 1)
-		die("need a title for the progress output");
-	title = argv[0];
+	if (argc)
+		usage_with_options(usage, options);
 
 	progress_testing = 1;
-	progress = start_progress(title, total);
 	while (strbuf_getline(&line, stdin) != EOF) {
 		char *end;
 
-		if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
+		if (skip_prefix(line.buf, "start ", (const char **) &end)) {
+			uint64_t total = strtoull(end, &end, 10);
+			if (*end == '\0')
+				progress = start_progress(default_title, total);
+			else if (*end == ' ')
+				progress = start_progress(dup_title(&titles,
+								    end + 1),
+							  total);
+			else
+				die("invalid input: '%s'\n", line.buf);
+		} else if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
 			uint64_t item_count = strtoull(end, &end, 10);
 			if (*end != '\0')
 				die("invalid input: '%s'\n", line.buf);
@@ -65,12 +87,14 @@ int cmd__progress(int argc, const char **argv)
 			display_throughput(progress, byte_count);
 		} else if (!strcmp(line.buf, "update")) {
 			progress_test_force_update();
+		} else if (!strcmp(line.buf, "stop")) {
+			stop_progress(&progress);
 		} else {
 			die("invalid input: '%s'\n", line.buf);
 		}
 	}
-	stop_progress(&progress);
 	strbuf_release(&line);
+	string_list_clear(&titles, 0);
 
 	return 0;
 }
t/t0500-progress-display.sh +37 -21

204 lines not on this page · raw message

Signature · 1 line
2.33.1.1510.ge5c82eefb93
#160 lore raw

PATCH v4 4/8 progress.c tests: test some invalid usage

In reply to Ævar Arnfjörð Bjarmason

Test what happens when we "stop" without a "start", omit the "stop" after a "start", or try to start two concurrent progress bars. This extends the trace2 tests added in 98a13647408 (trace2: log progress time and throughput, 2020-05-12).

These tests are not merely testing the helper, but invalid API usage that can happen if the progress.c API is misused.

The "without stop" test will leak under SANITIZE=leak, since this buggy use of the API will leak memory. But let's not skip it entirely, or use the "!SANITIZE_LEAK" prerequisite check as we'd do with tests that we're skipping due to leaks we haven't fixed yet. Instead annotate the specific command that should skip leak checking with custom $LSAN_OPTIONS[1].

1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer

---
 t/t0500-progress-display.sh | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
t/t0500-progress-display.sh +35 -0
index 27ab4218b01..59e9f226ea4 100755
@@ -325,4 +325,39 @@ test_expect_success 'progress generates traces' '
 	grep "\"key\":\"total_bytes\",\"value\":\"409600\"" trace.event
 '
 
+test_expect_success 'progress generates traces: stop / start' '
+	cat >in <<-\EOF &&
+	start 0
+	stop
+	EOF
+
+	GIT_TRACE2_EVENT="$(pwd)/trace-startstop.event" test-tool progress \
+		<in 2>stderr &&
+	test_region progress "Working hard" trace-startstop.event
+'
+
+test_expect_success 'progress generates traces: start without stop' '
+	cat >in <<-\EOF &&
+	start 0
+	EOF
+
+	GIT_TRACE2_EVENT="$(pwd)/trace-start.event" \
+	LSAN_OPTIONS=detect_leaks=0 \
+	test-tool progress \
+		<in 2>stderr &&
+	grep region_enter.*progress trace-start.event &&
+	! grep region_leave.*progress trace-start.event
+'
+
+test_expect_success 'progress generates traces: stop without start' '
+	cat >in <<-\EOF &&
+	stop
+	EOF
+
+	GIT_TRACE2_EVENT="$(pwd)/trace-stop.event" test-tool progress \
+		<in 2>stderr &&
+	! grep region_enter.*progress trace-stop.event &&
+	! grep region_leave.*progress trace-stop.event
+'
+
 test_done
Signature · 1 line
2.33.1.1510.ge5c82eefb93
#161 lore raw

PATCH v4 5/8 progress.c: add temporary variable from progress struct

In reply to Ævar Arnfjörð Bjarmason

Add a temporary "progress" variable for the dereferenced p_progress pointer to a "struct progress *". Before 98a13647408 (trace2: log progress time and throughput, 2020-05-12) we didn't dereference "p_progress" in this function, now that we do it's easier to read the code if we work with a "progress" struct pointer like everywhere else, instead of a pointer to a pointer.

---
 progress.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
progress.c +3 -2
index 680c6a8bf93..76a95cb7322 100644
@@ -325,15 +325,16 @@ void stop_progress(struct progress **p_progress)
 	finish_if_sparse(*p_progress);
 
 	if (*p_progress) {
+		struct progress *progress = *p_progress;
 		trace2_data_intmax("progress", the_repository, "total_objects",
 				   (*p_progress)->total);
 
 		if ((*p_progress)->throughput)
 			trace2_data_intmax("progress", the_repository,
 					   "total_bytes",
-					   (*p_progress)->throughput->curr_total);
+					   progress->throughput->curr_total);
 
-		trace2_region_leave("progress", (*p_progress)->title, the_repository);
+		trace2_region_leave("progress", progress->title, the_repository);
 	}
 
 	stop_progress_msg(p_progress, _("done"));
Signature · 1 line
2.33.1.1510.ge5c82eefb93
#162 lore raw

PATCH v4 6/8 pack-bitmap-write.c: don't return without stop_progress()

In reply to Ævar Arnfjörð Bjarmason

Fix a bug that's been here since 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21), we did not call stop_progress() if we reached the early exit in this function.

We could call stop_progress() before we return, but better yet is to defer calling start_progress() until we need it.

This will matter in a subsequent commit where we BUG(...) out if this happens, and matters now e.g. because we don't have a corresponding "region_end" for the progress trace2 event.

---
 pack-bitmap-write.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
pack-bitmap-write.c +3 -3
index 9c55c1531e1..cab3eaa2acd 100644
@@ -575,15 +575,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,
 
 	QSORT(indexed_commits, indexed_commits_nr, date_compare);
 
-	if (writer.show_progress)
-		writer.progress = start_progress("Selecting bitmap commits", 0);
-
 	if (indexed_commits_nr < 100) {
 		for (i = 0; i < indexed_commits_nr; ++i)
 			push_bitmapped_commit(indexed_commits[i]);
 		return;
 	}
 
+	if (writer.show_progress)
+		writer.progress = start_progress("Selecting bitmap commits", 0);
+
 	for (;;) {
 		struct commit *chosen = NULL;
 
Signature · 1 line
2.33.1.1510.ge5c82eefb93
#163 lore raw

PATCH v4 7/8 various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)

In reply to Ævar Arnfjörð Bjarmason

We have over 50 uses of "isatty(1)" and "isatty(2)" in the codebase, only these two used the stdlib.h macros to refer to them.

Let's change these for consistency, and because a subsequent commit's commit message outlines a recipe to change all of these for ad-hoc testing, not needing to match these with that ad-hoc regex will make things easier to explain.

---
 builtin/bisect--helper.c | 2 +-
 builtin/bundle.c         | 2 +-
 compat/mingw.c           | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
builtin/bisect--helper.c +1 -1
index 28a2e6a5750..1727cb051fb 100644
@@ -830,7 +830,7 @@ static int bisect_autostart(struct bisect_terms *terms)
 	fprintf_ln(stderr, _("You need to start by \"git bisect "
 			  "start\"\n"));
 
-	if (!isatty(STDIN_FILENO))
+	if (!isatty(1))
 		return -1;
 
 	/*
builtin/bundle.c +1 -1
index 5a85d7cd0fe..df69c651753 100644
@@ -56,7 +56,7 @@ static int parse_options_cmd_bundle(int argc,
 
 static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {
 	int all_progress_implied = 0;
-	int progress = isatty(STDERR_FILENO);
+	int progress = isatty(2);
 	struct strvec pack_opts;
 	int version = -1;
 	int ret;
compat/mingw.c +1 -1
index 9e0cd1e097f..0f545c1a7d1 100644
@@ -2374,7 +2374,7 @@ int mingw_raise(int sig)
 	switch (sig) {
 	case SIGALRM:
 		if (timer_fn == SIG_DFL) {
-			if (isatty(STDERR_FILENO))
+			if (isatty(2))
 				fputs("Alarm clock\n", stderr);
 			exit(128 + SIGALRM);
 		} else if (timer_fn != SIG_IGN)
Signature · 1 line
2.33.1.1510.ge5c82eefb93
#164 lore raw

PATCH v4 8/8 progress.c: add & assert a "global_progress" variable

In reply to Ævar Arnfjörð Bjarmason

The progress.c code makes a hard assumption that only one progress bar be active at a time (see [1] for a bug where this wasn't the case). Add a BUG() that'll trigger if we ever regress on that promise and have two progress bars active at the same time.

There was an alternative test-only approach to doing the same thing[2], but by doing this outside of a GIT_TEST_* mode we'll know we've put a hard stop to this particular API misuse.

It will also establish scaffolding to address current fundamental limitations in the progress output: The current output must be "driven" by calls to the likes of display_progress(). Once we have a global current progress object we'll be able to update that object via SIGALRM, this will cover cases where we're busy, but either haven't invoked our first display_progress() yet, or the time between display_progress() is too long. See [3] for early code to do that.

It's conceivable that this change will hit the BUG() condition in some scenario that we don't currently have tests for, this would be very bad. If that happened we'd die just because we couldn't emit some pretty output.

See [4] for a discussion of why our test coverage is lacking; our progress display is hidden behind isatty(2) checks in many cases, so the test suite doesn't cover it unless individual tests are run in "--verbose" mode, we might also have multi-threaded use of the API, so two progress bars stopping and starting would only be visible due to a race condition.

Despite that, I think that this change won't introduce such regressions, because:

1. I've read all the code using the progress API (and have modified a large part of it in some WIP code I have). Almost all of it is really simple, the parts that aren't[5] are complex in the display_progress() part, not in starting or stopping the progress bar.

2. The entire test suite passes when instrumented with an ad-hoc Linux-specific mode (it uses gettid()) to die if progress bars are ever started or stopped on anything but the main thread[6].

Extending that to die if display_progress() is called in a thread reveals that we have exactly two users of the progress bar under threaded conditions, "git index-pack" and "git pack-objects". Both uses are straightforward, and they don't start/stop the progress bar when threads are active.

3. I've likewise done an ad-hoc test to force progress bars to be displayed with:

perl -pi -e 's[isatty\(2\)][1]g' $(git grep -l -F 'isatty(2)')

I.e. to replace all checks (not just for progress) of checking whether STDERR is connected to a TTY, and then monkeypatching is_foreground_fd() in progress.c to always "return 1". Running the tests with those applied, interactively and under -V reveals via:

$ grep -e set_progress_signal -e clear_progress_signal test-results/*out

That nothing our tests cover hits the BUG conditions added here, except the expected "BUG: start two concurrent progress bars" test being added here.

That isn't entirely true since we won't be getting 100% coverage due to cascading failures from tests that expected no progress output on stderr. To make sure I covered 100% I also tried making the display() function in progress.c a NOOP on top of that (it's the calls to start_progress_delay() and stop_progress()) that matter.

That doesn't hit the BUG() either. Some tests fail in that mode due to a combination of the overzealous isatty(2) munging noted above, and the tests that are testing that the progress output itself is present (but for testing I'd made display() a NOOP).

This doesn't address any currently out-of-tree user of progress.c, i.e. WIP patches, or progress output that's a part of forks of git.git. Those hopefully have test coverage that would expose the BUG().

If they don't they'll either run into it in code that displays more than one progress bar for the lifetime of the progress, or which calls stop_progress() with a non-NULL "progress" without a corresponding start_progress(). Both of those cases are less likely than the general cases of progress.c API misuse.

Between those three points above and the discussion of how this could impact out-of-tree users I think it's safe to go ahead with this change.

1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)
2. https://lore.kernel.org/git/[email protected]
3. https://lore.kernel.org/git/[email protected]/
4. https://lore.kernel.org/git/[email protected]/
5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into
next, 2021-09-10)
6. https://lore.kernel.org/git/[email protected]/

---
 progress.c                  | 18 ++++++++++++++++++
 t/t0500-progress-display.sh | 11 +++++++++++
 2 files changed, 29 insertions(+)
progress.c +18 -0
index 76a95cb7322..7483aec2e2a 100644
@@ -46,6 +46,7 @@ struct progress {
 };
 
 static volatile sig_atomic_t progress_update;
+static struct progress *global_progress;
 
 /*
  * These are only intended for testing the progress output, i.e. exclusively
@@ -249,6 +250,14 @@ void display_progress(struct progress *progress, uint64_t n)
 		display(progress, n, NULL);
 }
 
+static void set_global_progress(struct progress *progress)
+{
+	if (global_progress)
+		BUG("'%s' progress still active when trying to start '%s'",
+		    global_progress->title, progress->title);
+	global_progress = progress;
+}
+
 static struct progress *start_progress_delay(const char *title, uint64_t total,
 					     unsigned delay, unsigned sparse)
 {
@@ -264,6 +273,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,
 	strbuf_init(&progress->counters_sb, 0);
 	progress->title_len = utf8_strwidth(title);
 	progress->split = 0;
+	set_global_progress(progress);
 	set_progress_signal();
 	trace2_region_enter("progress", title, the_repository);
 	return progress;
@@ -340,6 +350,13 @@ void stop_progress(struct progress **p_progress)
 	stop_progress_msg(p_progress, _("done"));
 }
 
+static void unset_global_progress(void)
+{
+	if (!global_progress)
+		BUG("should have active global_progress when cleaning up");
+	global_progress = NULL;
+}
+
 void stop_progress_msg(struct progress **p_progress, const char *msg)
 {
 	struct progress *progress;
@@ -369,6 +386,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)
 		free(buf);
 	}
 	clear_progress_signal();
+	unset_global_progress();
 	strbuf_release(&progress->counters_sb);
 	if (progress->throughput)
 		strbuf_release(&progress->throughput->display);
t/t0500-progress-display.sh +11 -0
index 59e9f226ea4..867fdace3f2 100755
@@ -298,6 +298,17 @@ test_expect_success 'cover up after throughput shortens a lot' '
 	test_cmp expect out
 '
 
+test_expect_success 'BUG: start two concurrent progress bars' '
+	cat >in <<-\EOF &&
+	start 0 one
+	start 0 two
+	EOF
+
+	test_must_fail test-tool progress \
+		<in 2>stderr &&
+	grep "^BUG: .*'\''one'\'' progress still active when trying to start '\''two'\''$" stderr
+'
+
 test_expect_success 'progress generates traces' '
 	cat >in <<-\EOF &&
 	start 40
Signature · 1 line
2.33.1.1510.ge5c82eefb93
#165 lore raw

Re: [PATCH v4 8/8] progress.c: add & assert a "global_progress" variable

Am 25.10.21 um 13:25 schrieb Ævar Arnfjörð Bjarmason:

Quoted 5 lines

Once we have a
global current progress object we'll be able to update that object via SIGALRM, this will cover cases where we're busy, but either haven't invoked our first display_progress() yet, or the time between display_progress() is too long. See [3] for early code to do that.

This can't be a goal. You cannot do serious stuff in a signal handler. Don't proliferate complicated signal handlers.

On top of that, our SIGALRM handler on Windows runs in its own thread and can't do more than set a volatile sigatomic_t (and even that is undefined behavior, strictly speaking).

-- Hannes

#166 lore raw

Re: [PATCH v4 8/8] progress.c: add & assert a "global_progress" variable

Quoted 9 lines · Johannes Sixt

Am 25.10.21 um 13:25 schrieb Ævar Arnfjörð Bjarmason:

Quoted 5 lines

Once we have a
global current progress object we'll be able to update that object via SIGALRM, this will cover cases where we're busy, but either haven't invoked our first display_progress() yet, or the time between display_progress() is too long. See [3] for early code to do that.

This can't be a goal. You cannot do serious stuff in a signal handler. Don't proliferate complicated signal handlers.

I'd missed some of the nuances with signal safety when I hacked that up initially. For the purposes of this patch the commit message is discussing a POC that suggests an interesting direction to go in.

My current implementation of this which I've run with for some time relies on undefined behavior and glibc being forgiving, but for any future submission I'll make sure it's signal-safe.

In particular to format a progress line etc. we we need sprintf() and the like, but to have a "stalled" message we can pre-format that at start_progress() time, and inside the signal handler all we'll need is a write(), which should be safe in most places...

Quoted 3 lines

On top of that, our SIGALRM handler on Windows runs in its own thread and can't do more than set a volatile sigatomic_t (and even that is undefined behavior, strictly speaking).

... but we may need to disable it entirely on some platforms, I didn't know about that Windows edge case.

Anyway, for this patch having the BUG() is useful even if we never went for that.

#167 lore raw

v5 cover progress: assert "global_progress" + test fixes / cleanup

In reply to Ævar Arnfjörð Bjarmason

This is a commit-message only update of v4[1]. It was pointed out on v4 that linked-to POC code discussed in the commit message wasn't safe for running inside a signal handler.

That's correct, but achieving the same goals would be possible while retaining signal safety on some platforms, so update the commit message discussion to note that.

In any case that part of the commit message is discussing future side-benefits of the change, the change as it is isn't doing anything novel with signal handlers in the progress.c code.

1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/

Ævar Arnfjörð Bjarmason (8):
leak tests: fix a memory leaks in "test-progress" helper progress.c test helper: add missing braces
progress.c tests: make start/stop commands on stdin progress.c tests: test some invalid usage
progress.c: add temporary variable from progress struct pack-bitmap-write.c: don't return without stop_progress() various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO) progress.c: add & assert a "global_progress" variable

 builtin/bisect--helper.c    |   2 +-
 builtin/bundle.c            |   2 +-
 compat/mingw.c              |   2 +-
 pack-bitmap-write.c         |   6 +--
 progress.c                  |  23 +++++++-
 t/helper/test-progress.c    |  52 +++++++++++++-----
 t/t0500-progress-display.sh | 105 ++++++++++++++++++++++++++++--------
 7 files changed, 150 insertions(+), 42 deletions(-)

Range-diff against v4:

1:  a3bd032d1eb = 1:  0c0e5bfc656 leak tests: fix a memory leaks in "test-progress" helper
2:  e441cfea7c5 = 2:  0ae86e40e16 progress.c test helper: add missing braces
3:  1c5f9bdfe6d = 3:  767aa010026 progress.c tests: make start/stop commands on stdin
4:  474ce31f9d2 = 4:  9bcc8cfc9d2 progress.c tests: test some invalid usage
5:  ff039742148 = 5:  3a95613cd04 progress.c: add temporary variable from progress struct
6:  3dfe31decff = 6:  0300b802dcb pack-bitmap-write.c: don't return without stop_progress()
7:  8a18eb40fae = 7:  4795d4835b0 various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)
8:  06124e8ac5e ! 8:  e58f4a0c1b9 progress.c: add & assert a "global_progress" variable

@@ Commit message

It will also establish scaffolding to address current fundamental limitations in the progress output: The current output must be
- "driven" by calls to the likes of display_progress(). Once we have a
- global current progress object we'll be able to update that object via
- SIGALRM, this will cover cases where we're busy, but either haven't
- invoked our first display_progress() yet, or the time between
- display_progress() is too long. See [3] for early code to do that.
+ "driven" by calls to the likes of display_progress().
+
+ Once we have a global current progress object we'll be able to update
+ that object via SIGALRM, this will cover cases where we're busy, but
+ either haven't invoked our first display_progress() yet, or the time
+ between display_progress() is too long. See [3] for early code to do
+ that.
+
+ The linked code in [3] is WIP and not signal-safe since among other
+ things it calls sprintf() from within a signal handler, see e.g. "man
+ 7 signal-safety". But on some platforms a real implementation of it
+ would be able to write() out a prepared-formatted progress update from
+ within a signal handler. That would be sufficient to e.g. show that
+ we're "stalled", or to display something like a simple pre-formatted
+ "spinner".

It's conceivable that this change will hit the BUG() condition in some scenario that we don't currently have tests for, this would be very

Signature · 1 line
2.33.1.1570.g069344fdd45
#168 lore raw

PATCH v5 1/8 leak tests: fix a memory leaks in "test-progress" helper

Fix a memory leak in the test-progress helper, and mark the corresponding "t0500-progress-display.sh" test as being leak-free under SANITIZE=leak. This fixes a leak added in 2bb74b53a4 (Test the progress display, 2019-09-16).

My 48f68715b14 (tr2: stop leaking "thread_name" memory, 2021-08-27) had fixed another memory leak in this test (as it did some trace2 testing).

---
 t/helper/test-progress.c    | 1 +
 t/t0500-progress-display.sh | 1 +
 2 files changed, 2 insertions(+)
t/helper/test-progress.c +1 -0
index 5d05cbe7894..9265e6ab7cf 100644
@@ -69,6 +69,7 @@ int cmd__progress(int argc, const char **argv)
 			die("invalid input: '%s'\n", line.buf);
 	}
 	stop_progress(&progress);
+	strbuf_release(&line);
 
 	return 0;
 }
t/t0500-progress-display.sh +1 -0
index 22058b503ac..f37cf2eb9c9 100755
@@ -2,6 +2,7 @@
 
 test_description='progress display'
 
+TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 show_cr () {
Signature · 1 line
2.33.1.1570.g069344fdd45
#169 lore raw

PATCH v5 2/8 progress.c test helper: add missing braces

In reply to Ævar Arnfjörð Bjarmason

If we have braces on one arm of an if/else all of them should have it, per the CodingGuidelines's "When there are multiple arms to a conditional[...]" advice. This formatting change makes a subsequent commit smaller.

---
 t/helper/test-progress.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
t/helper/test-progress.c +3 -2
index 9265e6ab7cf..50fd3be3dad 100644
@@ -63,10 +63,11 @@ int cmd__progress(int argc, const char **argv)
 				die("invalid input: '%s'\n", line.buf);
 			progress_test_ns = test_ms * 1000 * 1000;
 			display_throughput(progress, byte_count);
-		} else if (!strcmp(line.buf, "update"))
+		} else if (!strcmp(line.buf, "update")) {
 			progress_test_force_update();
-		else
+		} else {
 			die("invalid input: '%s'\n", line.buf);
+		}
 	}
 	stop_progress(&progress);
 	strbuf_release(&line);
Signature · 1 line
2.33.1.1570.g069344fdd45
#170 lore raw

PATCH v5 3/8 progress.c tests: make start/stop commands on stdin

In reply to Ævar Arnfjörð Bjarmason

Change the usage of the "test-tool progress" introduced in 2bb74b53a49 (Test the progress display, 2019-09-16) to take command like "start" and "stop" on stdin, instead of running them implicitly.

This makes for tests that are easier to read, since the recipe will mirror the API usage, and allows for easily testing invalid usage that would yield (or should yield) a BUG(), e.g. providing two "start" calls in a row. A subsequent commit will add such tests.

---
 t/helper/test-progress.c    | 46 ++++++++++++++++++++++-------
 t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------
 2 files changed, 72 insertions(+), 32 deletions(-)
t/helper/test-progress.c +35 -11
index 50fd3be3dad..1435c28e950 100644
@@ -3,6 +3,9 @@
  *
  * Reads instructions from standard input, one instruction per line:
  *
+ *   "start <total>[ <title>]" - Call start_progress(title, total),
+ *                               Uses the default title of "Working hard"
+ *                               if the " <title>" is omitted.
  *   "progress <items>" - Call display_progress() with the given item count
  *                        as parameter.
  *   "throughput <bytes> <millis> - Call display_throughput() with the given
@@ -10,6 +13,7 @@
  *                                  specify the time elapsed since the
  *                                  start_progress() call.
  *   "update" - Set the 'progress_update' flag.
+ *   "stop" - Call stop_progress().
  *
  * See 't0500-progress-display.sh' for examples.
  */
@@ -19,34 +23,52 @@
 #include "parse-options.h"
 #include "progress.h"
 #include "strbuf.h"
+#include "string-list.h"
+
+/*
+ * We can't use "end + 1" as an argument to start_progress() below, it
+ * doesn't xstrdup() its "title" argument. We need to hold onto a
+ * valid "char *" for it until the end.
+ */
+static char *dup_title(struct string_list *titles, const char *title)
+{
+	return string_list_insert(titles, title)->string;
+}
 
 int cmd__progress(int argc, const char **argv)
 {
-	int total = 0;
-	const char *title;
+	const char *const default_title = "Working hard";
+	struct string_list titles = STRING_LIST_INIT_DUP;
 	struct strbuf line = STRBUF_INIT;
-	struct progress *progress;
+	struct progress *progress = NULL;
 
 	const char *usage[] = {
-		"test-tool progress [--total=<n>] <progress-title>",
+		"test-tool progress <stdin",
 		NULL
 	};
 	struct option options[] = {
-		OPT_INTEGER(0, "total", &total, "total number of items"),
 		OPT_END(),
 	};
 
 	argc = parse_options(argc, argv, NULL, options, usage, 0);
-	if (argc != 1)
-		die("need a title for the progress output");
-	title = argv[0];
+	if (argc)
+		usage_with_options(usage, options);
 
 	progress_testing = 1;
-	progress = start_progress(title, total);
 	while (strbuf_getline(&line, stdin) != EOF) {
 		char *end;
 
-		if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
+		if (skip_prefix(line.buf, "start ", (const char **) &end)) {
+			uint64_t total = strtoull(end, &end, 10);
+			if (*end == '\0')
+				progress = start_progress(default_title, total);
+			else if (*end == ' ')
+				progress = start_progress(dup_title(&titles,
+								    end + 1),
+							  total);
+			else
+				die("invalid input: '%s'\n", line.buf);
+		} else if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
 			uint64_t item_count = strtoull(end, &end, 10);
 			if (*end != '\0')
 				die("invalid input: '%s'\n", line.buf);
@@ -65,12 +87,14 @@ int cmd__progress(int argc, const char **argv)
 			display_throughput(progress, byte_count);
 		} else if (!strcmp(line.buf, "update")) {
 			progress_test_force_update();
+		} else if (!strcmp(line.buf, "stop")) {
+			stop_progress(&progress);
 		} else {
 			die("invalid input: '%s'\n", line.buf);
 		}
 	}
-	stop_progress(&progress);
 	strbuf_release(&line);
+	string_list_clear(&titles, 0);
 
 	return 0;
 }
t/t0500-progress-display.sh +37 -21

204 lines not on this page · raw message

Signature · 1 line
2.33.1.1570.g069344fdd45
#171 lore raw

PATCH v5 4/8 progress.c tests: test some invalid usage

In reply to Ævar Arnfjörð Bjarmason

Test what happens when we "stop" without a "start", omit the "stop" after a "start", or try to start two concurrent progress bars. This extends the trace2 tests added in 98a13647408 (trace2: log progress time and throughput, 2020-05-12).

These tests are not merely testing the helper, but invalid API usage that can happen if the progress.c API is misused.

The "without stop" test will leak under SANITIZE=leak, since this buggy use of the API will leak memory. But let's not skip it entirely, or use the "!SANITIZE_LEAK" prerequisite check as we'd do with tests that we're skipping due to leaks we haven't fixed yet. Instead annotate the specific command that should skip leak checking with custom $LSAN_OPTIONS[1].

1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer

---
 t/t0500-progress-display.sh | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
t/t0500-progress-display.sh +35 -0
index 27ab4218b01..59e9f226ea4 100755
@@ -325,4 +325,39 @@ test_expect_success 'progress generates traces' '
 	grep "\"key\":\"total_bytes\",\"value\":\"409600\"" trace.event
 '
 
+test_expect_success 'progress generates traces: stop / start' '
+	cat >in <<-\EOF &&
+	start 0
+	stop
+	EOF
+
+	GIT_TRACE2_EVENT="$(pwd)/trace-startstop.event" test-tool progress \
+		<in 2>stderr &&
+	test_region progress "Working hard" trace-startstop.event
+'
+
+test_expect_success 'progress generates traces: start without stop' '
+	cat >in <<-\EOF &&
+	start 0
+	EOF
+
+	GIT_TRACE2_EVENT="$(pwd)/trace-start.event" \
+	LSAN_OPTIONS=detect_leaks=0 \
+	test-tool progress \
+		<in 2>stderr &&
+	grep region_enter.*progress trace-start.event &&
+	! grep region_leave.*progress trace-start.event
+'
+
+test_expect_success 'progress generates traces: stop without start' '
+	cat >in <<-\EOF &&
+	stop
+	EOF
+
+	GIT_TRACE2_EVENT="$(pwd)/trace-stop.event" test-tool progress \
+		<in 2>stderr &&
+	! grep region_enter.*progress trace-stop.event &&
+	! grep region_leave.*progress trace-stop.event
+'
+
 test_done
Signature · 1 line
2.33.1.1570.g069344fdd45
#172 lore raw

PATCH v5 5/8 progress.c: add temporary variable from progress struct

In reply to Ævar Arnfjörð Bjarmason

Add a temporary "progress" variable for the dereferenced p_progress pointer to a "struct progress *". Before 98a13647408 (trace2: log progress time and throughput, 2020-05-12) we didn't dereference "p_progress" in this function, now that we do it's easier to read the code if we work with a "progress" struct pointer like everywhere else, instead of a pointer to a pointer.

---
 progress.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
progress.c +3 -2
index 680c6a8bf93..76a95cb7322 100644
@@ -325,15 +325,16 @@ void stop_progress(struct progress **p_progress)
 	finish_if_sparse(*p_progress);
 
 	if (*p_progress) {
+		struct progress *progress = *p_progress;
 		trace2_data_intmax("progress", the_repository, "total_objects",
 				   (*p_progress)->total);
 
 		if ((*p_progress)->throughput)
 			trace2_data_intmax("progress", the_repository,
 					   "total_bytes",
-					   (*p_progress)->throughput->curr_total);
+					   progress->throughput->curr_total);
 
-		trace2_region_leave("progress", (*p_progress)->title, the_repository);
+		trace2_region_leave("progress", progress->title, the_repository);
 	}
 
 	stop_progress_msg(p_progress, _("done"));
Signature · 1 line
2.33.1.1570.g069344fdd45
#173 lore raw

PATCH v5 6/8 pack-bitmap-write.c: don't return without stop_progress()

In reply to Ævar Arnfjörð Bjarmason

Fix a bug that's been here since 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21), we did not call stop_progress() if we reached the early exit in this function.

We could call stop_progress() before we return, but better yet is to defer calling start_progress() until we need it.

This will matter in a subsequent commit where we BUG(...) out if this happens, and matters now e.g. because we don't have a corresponding "region_end" for the progress trace2 event.

---
 pack-bitmap-write.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
pack-bitmap-write.c +3 -3
index 9c55c1531e1..cab3eaa2acd 100644
@@ -575,15 +575,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,
 
 	QSORT(indexed_commits, indexed_commits_nr, date_compare);
 
-	if (writer.show_progress)
-		writer.progress = start_progress("Selecting bitmap commits", 0);
-
 	if (indexed_commits_nr < 100) {
 		for (i = 0; i < indexed_commits_nr; ++i)
 			push_bitmapped_commit(indexed_commits[i]);
 		return;
 	}
 
+	if (writer.show_progress)
+		writer.progress = start_progress("Selecting bitmap commits", 0);
+
 	for (;;) {
 		struct commit *chosen = NULL;
 
Signature · 1 line
2.33.1.1570.g069344fdd45
#174 lore raw

PATCH v5 7/8 various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)

In reply to Ævar Arnfjörð Bjarmason

We have over 50 uses of "isatty(1)" and "isatty(2)" in the codebase, only these two used the stdlib.h macros to refer to them.

Let's change these for consistency, and because a subsequent commit's commit message outlines a recipe to change all of these for ad-hoc testing, not needing to match these with that ad-hoc regex will make things easier to explain.

---
 builtin/bisect--helper.c | 2 +-
 builtin/bundle.c         | 2 +-
 compat/mingw.c           | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
builtin/bisect--helper.c +1 -1
index 28a2e6a5750..1727cb051fb 100644
@@ -830,7 +830,7 @@ static int bisect_autostart(struct bisect_terms *terms)
 	fprintf_ln(stderr, _("You need to start by \"git bisect "
 			  "start\"\n"));
 
-	if (!isatty(STDIN_FILENO))
+	if (!isatty(1))
 		return -1;
 
 	/*
builtin/bundle.c +1 -1
index 5a85d7cd0fe..df69c651753 100644
@@ -56,7 +56,7 @@ static int parse_options_cmd_bundle(int argc,
 
 static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {
 	int all_progress_implied = 0;
-	int progress = isatty(STDERR_FILENO);
+	int progress = isatty(2);
 	struct strvec pack_opts;
 	int version = -1;
 	int ret;
compat/mingw.c +1 -1
index 9e0cd1e097f..0f545c1a7d1 100644
@@ -2374,7 +2374,7 @@ int mingw_raise(int sig)
 	switch (sig) {
 	case SIGALRM:
 		if (timer_fn == SIG_DFL) {
-			if (isatty(STDERR_FILENO))
+			if (isatty(2))
 				fputs("Alarm clock\n", stderr);
 			exit(128 + SIGALRM);
 		} else if (timer_fn != SIG_IGN)
Signature · 1 line
2.33.1.1570.g069344fdd45
#175 lore raw

Re: [PATCH v5 7/8] various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)

Quoted 21 lines · Ævar Arnfjörð Bjarmason

We have over 50 uses of "isatty(1)" and "isatty(2)" in the codebase, only these two used the stdlib.h macros to refer to them.

Let's change these for consistency, and because a subsequent commit's commit message outlines a recipe to change all of these for ad-hoc testing, not needing to match these with that ad-hoc regex will make things easier to explain.

---
 builtin/bisect--helper.c | 2 +-
 builtin/bundle.c         | 2 +-
 compat/mingw.c           | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
builtin/bisect--helper.c +0 -0
index 28a2e6a5750..1727cb051fb 100644

builtin/bisect--helper.c static int bisect_autostart(struct bisect_terms *terms) · line 833

Quoted 5 lines
builtin/bisect--helper.c +1 -1
@@ -830,7 +830,7 @@ static int bisect_autostart(struct bisect_terms *terms)
        fprintf_ln(stderr, _("You need to start by \"git bisect "
                          "start\"\n"));
 
-       if (!isatty(STDIN_FILENO))
+       if (!isatty(1))

STDIN_FILENO -> 0

Carlo

#176 lore raw

PATCH v5 8/8 progress.c: add & assert a "global_progress" variable

In reply to Ævar Arnfjörð Bjarmason

The progress.c code makes a hard assumption that only one progress bar be active at a time (see [1] for a bug where this wasn't the case). Add a BUG() that'll trigger if we ever regress on that promise and have two progress bars active at the same time.

There was an alternative test-only approach to doing the same thing[2], but by doing this outside of a GIT_TEST_* mode we'll know we've put a hard stop to this particular API misuse.

It will also establish scaffolding to address current fundamental limitations in the progress output: The current output must be "driven" by calls to the likes of display_progress().

Once we have a global current progress object we'll be able to update that object via SIGALRM, this will cover cases where we're busy, but either haven't invoked our first display_progress() yet, or the time between display_progress() is too long. See [3] for early code to do that.

The linked code in [3] is WIP and not signal-safe since among other things it calls sprintf() from within a signal handler, see e.g. "man 7 signal-safety". But on some platforms a real implementation of it would be able to write() out a prepared-formatted progress update from within a signal handler. That would be sufficient to e.g. show that we're "stalled", or to display something like a simple pre-formatted "spinner".

It's conceivable that this change will hit the BUG() condition in some scenario that we don't currently have tests for, this would be very bad. If that happened we'd die just because we couldn't emit some pretty output.

See [4] for a discussion of why our test coverage is lacking; our progress display is hidden behind isatty(2) checks in many cases, so the test suite doesn't cover it unless individual tests are run in "--verbose" mode, we might also have multi-threaded use of the API, so two progress bars stopping and starting would only be visible due to a race condition.

Despite that, I think that this change won't introduce such regressions, because:

1. I've read all the code using the progress API (and have modified a large part of it in some WIP code I have). Almost all of it is really simple, the parts that aren't[5] are complex in the display_progress() part, not in starting or stopping the progress bar.

2. The entire test suite passes when instrumented with an ad-hoc Linux-specific mode (it uses gettid()) to die if progress bars are ever started or stopped on anything but the main thread[6].

Extending that to die if display_progress() is called in a thread reveals that we have exactly two users of the progress bar under threaded conditions, "git index-pack" and "git pack-objects". Both uses are straightforward, and they don't start/stop the progress bar when threads are active.

3. I've likewise done an ad-hoc test to force progress bars to be displayed with:

perl -pi -e 's[isatty\(2\)][1]g' $(git grep -l -F 'isatty(2)')

I.e. to replace all checks (not just for progress) of checking whether STDERR is connected to a TTY, and then monkeypatching is_foreground_fd() in progress.c to always "return 1". Running the tests with those applied, interactively and under -V reveals via:

$ grep -e set_progress_signal -e clear_progress_signal test-results/*out

That nothing our tests cover hits the BUG conditions added here, except the expected "BUG: start two concurrent progress bars" test being added here.

That isn't entirely true since we won't be getting 100% coverage due to cascading failures from tests that expected no progress output on stderr. To make sure I covered 100% I also tried making the display() function in progress.c a NOOP on top of that (it's the calls to start_progress_delay() and stop_progress()) that matter.

That doesn't hit the BUG() either. Some tests fail in that mode due to a combination of the overzealous isatty(2) munging noted above, and the tests that are testing that the progress output itself is present (but for testing I'd made display() a NOOP).

This doesn't address any currently out-of-tree user of progress.c, i.e. WIP patches, or progress output that's a part of forks of git.git. Those hopefully have test coverage that would expose the BUG().

If they don't they'll either run into it in code that displays more than one progress bar for the lifetime of the progress, or which calls stop_progress() with a non-NULL "progress" without a corresponding start_progress(). Both of those cases are less likely than the general cases of progress.c API misuse.

Between those three points above and the discussion of how this could impact out-of-tree users I think it's safe to go ahead with this change.

1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)
2. https://lore.kernel.org/git/[email protected]
3. https://lore.kernel.org/git/[email protected]/
4. https://lore.kernel.org/git/[email protected]/
5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into
next, 2021-09-10)
6. https://lore.kernel.org/git/[email protected]/

---
 progress.c                  | 18 ++++++++++++++++++
 t/t0500-progress-display.sh | 11 +++++++++++
 2 files changed, 29 insertions(+)
progress.c +18 -0
index 76a95cb7322..7483aec2e2a 100644
@@ -46,6 +46,7 @@ struct progress {
 };
 
 static volatile sig_atomic_t progress_update;
+static struct progress *global_progress;
 
 /*
  * These are only intended for testing the progress output, i.e. exclusively
@@ -249,6 +250,14 @@ void display_progress(struct progress *progress, uint64_t n)
 		display(progress, n, NULL);
 }
 
+static void set_global_progress(struct progress *progress)
+{
+	if (global_progress)
+		BUG("'%s' progress still active when trying to start '%s'",
+		    global_progress->title, progress->title);
+	global_progress = progress;
+}
+
 static struct progress *start_progress_delay(const char *title, uint64_t total,
 					     unsigned delay, unsigned sparse)
 {
@@ -264,6 +273,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,
 	strbuf_init(&progress->counters_sb, 0);
 	progress->title_len = utf8_strwidth(title);
 	progress->split = 0;
+	set_global_progress(progress);
 	set_progress_signal();
 	trace2_region_enter("progress", title, the_repository);
 	return progress;
@@ -340,6 +350,13 @@ void stop_progress(struct progress **p_progress)
 	stop_progress_msg(p_progress, _("done"));
 }
 
+static void unset_global_progress(void)
+{
+	if (!global_progress)
+		BUG("should have active global_progress when cleaning up");
+	global_progress = NULL;
+}
+
 void stop_progress_msg(struct progress **p_progress, const char *msg)
 {
 	struct progress *progress;
@@ -369,6 +386,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)
 		free(buf);
 	}
 	clear_progress_signal();
+	unset_global_progress();
 	strbuf_release(&progress->counters_sb);
 	if (progress->throughput)
 		strbuf_release(&progress->throughput->display);
t/t0500-progress-display.sh +11 -0
index 59e9f226ea4..867fdace3f2 100755
@@ -298,6 +298,17 @@ test_expect_success 'cover up after throughput shortens a lot' '
 	test_cmp expect out
 '
 
+test_expect_success 'BUG: start two concurrent progress bars' '
+	cat >in <<-\EOF &&
+	start 0 one
+	start 0 two
+	EOF
+
+	test_must_fail test-tool progress \
+		<in 2>stderr &&
+	grep "^BUG: .*'\''one'\'' progress still active when trying to start '\''two'\''$" stderr
+'
+
 test_expect_success 'progress generates traces' '
 	cat >in <<-\EOF &&
 	start 40
Signature · 1 line
2.33.1.1570.g069344fdd45
#177 lore raw

v6 cover progress: assert "global_progress" + test fixes / cleanup

In reply to Ævar Arnfjörð Bjarmason

A minor but important fixup of v5[1], as pointed out by Carlo Arenas the STDIN_FILENO I added in response to feedback in an earlier re-roll had a pretty bad off-by-one bug.

As an aside that we have no tests for this "git bisect" code at all is pretty bad, I'll try to address that separately from this series.

https://lore.kernel.org/git/[email protected]/

Ævar Arnfjörð Bjarmason (8):
leak tests: fix a memory leaks in "test-progress" helper progress.c test helper: add missing braces
progress.c tests: make start/stop commands on stdin progress.c tests: test some invalid usage
progress.c: add temporary variable from progress struct pack-bitmap-write.c: don't return without stop_progress() various *.c: use isatty(0|2), not isatty(STDIN_FILENO|STDERR_FILENO) progress.c: add & assert a "global_progress" variable

 builtin/bisect--helper.c    |   2 +-
 builtin/bundle.c            |   2 +-
 compat/mingw.c              |   2 +-
 pack-bitmap-write.c         |   6 +--
 progress.c                  |  23 +++++++-
 t/helper/test-progress.c    |  52 +++++++++++++-----
 t/t0500-progress-display.sh | 105 ++++++++++++++++++++++++++++--------
 7 files changed, 150 insertions(+), 42 deletions(-)

Range-diff against v5:

1:  0c0e5bfc656 = 1:  88d89652831 leak tests: fix a memory leaks in "test-progress" helper
2:  0ae86e40e16 = 2:  2fa92e7db83 progress.c test helper: add missing braces
3:  767aa010026 = 3:  400b491449e progress.c tests: make start/stop commands on stdin
4:  9bcc8cfc9d2 = 4:  7557975a122 progress.c tests: test some invalid usage
5:  3a95613cd04 = 5:  61c8da31aeb progress.c: add temporary variable from progress struct
6:  0300b802dcb = 6:  f6a76b80e91 pack-bitmap-write.c: don't return without stop_progress()
7:  4795d4835b0 ! 7:  1a2eadf28d0 various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)

@@ Metadata
Author: Ævar Arnfjörð Bjarmason <[email protected]>

## Commit message ##
- various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)
+ various *.c: use isatty(0|2), not isatty(STDIN_FILENO|STDERR_FILENO)

We have over 50 uses of "isatty(1)" and "isatty(2)" in the codebase,
- only these two used the stdlib.h macros to refer to them.
+ and around 10 "isatty(0)", but these used the
+ {STDIN_FILENO,STD{OUT,ERR}_FILENO} macros in "stdlib.h" to refer to
+ them.

Let's change these for consistency, and because a subsequent commit's commit message outlines a recipe to change all of these for ad-hoc
@@ builtin/bisect--helper.c: static int bisect_autostart(struct bisect_terms *terms
"start\"\n"));

- if (!isatty(STDIN_FILENO))
-+ if (!isatty(1))
++ if (!isatty(0))
return -1;

/*
8: e58f4a0c1b9 = 8: bff919994b5 progress.c: add & assert a "global_progress" variable

Signature · 1 line
2.33.1.1570.g069344fdd45
#178 lore raw

PATCH v6 1/8 leak tests: fix a memory leaks in "test-progress" helper

Fix a memory leak in the test-progress helper, and mark the corresponding "t0500-progress-display.sh" test as being leak-free under SANITIZE=leak. This fixes a leak added in 2bb74b53a4 (Test the progress display, 2019-09-16).

My 48f68715b14 (tr2: stop leaking "thread_name" memory, 2021-08-27) had fixed another memory leak in this test (as it did some trace2 testing).

---
 t/helper/test-progress.c    | 1 +
 t/t0500-progress-display.sh | 1 +
 2 files changed, 2 insertions(+)
t/helper/test-progress.c +1 -0
index 5d05cbe7894..9265e6ab7cf 100644
@@ -69,6 +69,7 @@ int cmd__progress(int argc, const char **argv)
 			die("invalid input: '%s'\n", line.buf);
 	}
 	stop_progress(&progress);
+	strbuf_release(&line);
 
 	return 0;
 }
t/t0500-progress-display.sh +1 -0
index 22058b503ac..f37cf2eb9c9 100755
@@ -2,6 +2,7 @@
 
 test_description='progress display'
 
+TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 show_cr () {
Signature · 1 line
2.33.1.1570.g069344fdd45
#179 lore raw

PATCH v6 2/8 progress.c test helper: add missing braces

In reply to Ævar Arnfjörð Bjarmason

If we have braces on one arm of an if/else all of them should have it, per the CodingGuidelines's "When there are multiple arms to a conditional[...]" advice. This formatting change makes a subsequent commit smaller.

---
 t/helper/test-progress.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
t/helper/test-progress.c +3 -2
index 9265e6ab7cf..50fd3be3dad 100644
@@ -63,10 +63,11 @@ int cmd__progress(int argc, const char **argv)
 				die("invalid input: '%s'\n", line.buf);
 			progress_test_ns = test_ms * 1000 * 1000;
 			display_throughput(progress, byte_count);
-		} else if (!strcmp(line.buf, "update"))
+		} else if (!strcmp(line.buf, "update")) {
 			progress_test_force_update();
-		else
+		} else {
 			die("invalid input: '%s'\n", line.buf);
+		}
 	}
 	stop_progress(&progress);
 	strbuf_release(&line);
Signature · 1 line
2.33.1.1570.g069344fdd45
#180 lore raw

PATCH v6 3/8 progress.c tests: make start/stop commands on stdin

In reply to Ævar Arnfjörð Bjarmason

Change the usage of the "test-tool progress" introduced in 2bb74b53a49 (Test the progress display, 2019-09-16) to take command like "start" and "stop" on stdin, instead of running them implicitly.

This makes for tests that are easier to read, since the recipe will mirror the API usage, and allows for easily testing invalid usage that would yield (or should yield) a BUG(), e.g. providing two "start" calls in a row. A subsequent commit will add such tests.

---
 t/helper/test-progress.c    | 46 ++++++++++++++++++++++-------
 t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------
 2 files changed, 72 insertions(+), 32 deletions(-)
t/helper/test-progress.c +35 -11
index 50fd3be3dad..1435c28e950 100644
@@ -3,6 +3,9 @@
  *
  * Reads instructions from standard input, one instruction per line:
  *
+ *   "start <total>[ <title>]" - Call start_progress(title, total),
+ *                               Uses the default title of "Working hard"
+ *                               if the " <title>" is omitted.
  *   "progress <items>" - Call display_progress() with the given item count
  *                        as parameter.
  *   "throughput <bytes> <millis> - Call display_throughput() with the given
@@ -10,6 +13,7 @@
  *                                  specify the time elapsed since the
  *                                  start_progress() call.
  *   "update" - Set the 'progress_update' flag.
+ *   "stop" - Call stop_progress().
  *
  * See 't0500-progress-display.sh' for examples.
  */
@@ -19,34 +23,52 @@
 #include "parse-options.h"
 #include "progress.h"
 #include "strbuf.h"
+#include "string-list.h"
+
+/*
+ * We can't use "end + 1" as an argument to start_progress() below, it
+ * doesn't xstrdup() its "title" argument. We need to hold onto a
+ * valid "char *" for it until the end.
+ */
+static char *dup_title(struct string_list *titles, const char *title)
+{
+	return string_list_insert(titles, title)->string;
+}
 
 int cmd__progress(int argc, const char **argv)
 {
-	int total = 0;
-	const char *title;
+	const char *const default_title = "Working hard";
+	struct string_list titles = STRING_LIST_INIT_DUP;
 	struct strbuf line = STRBUF_INIT;
-	struct progress *progress;
+	struct progress *progress = NULL;
 
 	const char *usage[] = {
-		"test-tool progress [--total=<n>] <progress-title>",
+		"test-tool progress <stdin",
 		NULL
 	};
 	struct option options[] = {
-		OPT_INTEGER(0, "total", &total, "total number of items"),
 		OPT_END(),
 	};
 
 	argc = parse_options(argc, argv, NULL, options, usage, 0);
-	if (argc != 1)
-		die("need a title for the progress output");
-	title = argv[0];
+	if (argc)
+		usage_with_options(usage, options);
 
 	progress_testing = 1;
-	progress = start_progress(title, total);
 	while (strbuf_getline(&line, stdin) != EOF) {
 		char *end;
 
-		if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
+		if (skip_prefix(line.buf, "start ", (const char **) &end)) {
+			uint64_t total = strtoull(end, &end, 10);
+			if (*end == '\0')
+				progress = start_progress(default_title, total);
+			else if (*end == ' ')
+				progress = start_progress(dup_title(&titles,
+								    end + 1),
+							  total);
+			else
+				die("invalid input: '%s'\n", line.buf);
+		} else if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
 			uint64_t item_count = strtoull(end, &end, 10);
 			if (*end != '\0')
 				die("invalid input: '%s'\n", line.buf);
@@ -65,12 +87,14 @@ int cmd__progress(int argc, const char **argv)
 			display_throughput(progress, byte_count);
 		} else if (!strcmp(line.buf, "update")) {
 			progress_test_force_update();
+		} else if (!strcmp(line.buf, "stop")) {
+			stop_progress(&progress);
 		} else {
 			die("invalid input: '%s'\n", line.buf);
 		}
 	}
-	stop_progress(&progress);
 	strbuf_release(&line);
+	string_list_clear(&titles, 0);
 
 	return 0;
 }
t/t0500-progress-display.sh +37 -21

204 lines not on this page · raw message

Signature · 1 line
2.33.1.1570.g069344fdd45
#181 lore raw

PATCH v6 4/8 progress.c tests: test some invalid usage

In reply to Ævar Arnfjörð Bjarmason

Test what happens when we "stop" without a "start", omit the "stop" after a "start", or try to start two concurrent progress bars. This extends the trace2 tests added in 98a13647408 (trace2: log progress time and throughput, 2020-05-12).

These tests are not merely testing the helper, but invalid API usage that can happen if the progress.c API is misused.

The "without stop" test will leak under SANITIZE=leak, since this buggy use of the API will leak memory. But let's not skip it entirely, or use the "!SANITIZE_LEAK" prerequisite check as we'd do with tests that we're skipping due to leaks we haven't fixed yet. Instead annotate the specific command that should skip leak checking with custom $LSAN_OPTIONS[1].

1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer

---
 t/t0500-progress-display.sh | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
t/t0500-progress-display.sh +35 -0
index 27ab4218b01..59e9f226ea4 100755
@@ -325,4 +325,39 @@ test_expect_success 'progress generates traces' '
 	grep "\"key\":\"total_bytes\",\"value\":\"409600\"" trace.event
 '
 
+test_expect_success 'progress generates traces: stop / start' '
+	cat >in <<-\EOF &&
+	start 0
+	stop
+	EOF
+
+	GIT_TRACE2_EVENT="$(pwd)/trace-startstop.event" test-tool progress \
+		<in 2>stderr &&
+	test_region progress "Working hard" trace-startstop.event
+'
+
+test_expect_success 'progress generates traces: start without stop' '
+	cat >in <<-\EOF &&
+	start 0
+	EOF
+
+	GIT_TRACE2_EVENT="$(pwd)/trace-start.event" \
+	LSAN_OPTIONS=detect_leaks=0 \
+	test-tool progress \
+		<in 2>stderr &&
+	grep region_enter.*progress trace-start.event &&
+	! grep region_leave.*progress trace-start.event
+'
+
+test_expect_success 'progress generates traces: stop without start' '
+	cat >in <<-\EOF &&
+	stop
+	EOF
+
+	GIT_TRACE2_EVENT="$(pwd)/trace-stop.event" test-tool progress \
+		<in 2>stderr &&
+	! grep region_enter.*progress trace-stop.event &&
+	! grep region_leave.*progress trace-stop.event
+'
+
 test_done
Signature · 1 line
2.33.1.1570.g069344fdd45
#182 lore raw

PATCH v6 5/8 progress.c: add temporary variable from progress struct

In reply to Ævar Arnfjörð Bjarmason

Add a temporary "progress" variable for the dereferenced p_progress pointer to a "struct progress *". Before 98a13647408 (trace2: log progress time and throughput, 2020-05-12) we didn't dereference "p_progress" in this function, now that we do it's easier to read the code if we work with a "progress" struct pointer like everywhere else, instead of a pointer to a pointer.

---
 progress.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
progress.c +3 -2
index 680c6a8bf93..76a95cb7322 100644
@@ -325,15 +325,16 @@ void stop_progress(struct progress **p_progress)
 	finish_if_sparse(*p_progress);
 
 	if (*p_progress) {
+		struct progress *progress = *p_progress;
 		trace2_data_intmax("progress", the_repository, "total_objects",
 				   (*p_progress)->total);
 
 		if ((*p_progress)->throughput)
 			trace2_data_intmax("progress", the_repository,
 					   "total_bytes",
-					   (*p_progress)->throughput->curr_total);
+					   progress->throughput->curr_total);
 
-		trace2_region_leave("progress", (*p_progress)->title, the_repository);
+		trace2_region_leave("progress", progress->title, the_repository);
 	}
 
 	stop_progress_msg(p_progress, _("done"));
Signature · 1 line
2.33.1.1570.g069344fdd45
#183 lore raw

PATCH v6 6/8 pack-bitmap-write.c: don't return without stop_progress()

In reply to Ævar Arnfjörð Bjarmason

Fix a bug that's been here since 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21), we did not call stop_progress() if we reached the early exit in this function.

We could call stop_progress() before we return, but better yet is to defer calling start_progress() until we need it.

This will matter in a subsequent commit where we BUG(...) out if this happens, and matters now e.g. because we don't have a corresponding "region_end" for the progress trace2 event.

---
 pack-bitmap-write.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
pack-bitmap-write.c +3 -3
index 9c55c1531e1..cab3eaa2acd 100644
@@ -575,15 +575,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,
 
 	QSORT(indexed_commits, indexed_commits_nr, date_compare);
 
-	if (writer.show_progress)
-		writer.progress = start_progress("Selecting bitmap commits", 0);
-
 	if (indexed_commits_nr < 100) {
 		for (i = 0; i < indexed_commits_nr; ++i)
 			push_bitmapped_commit(indexed_commits[i]);
 		return;
 	}
 
+	if (writer.show_progress)
+		writer.progress = start_progress("Selecting bitmap commits", 0);
+
 	for (;;) {
 		struct commit *chosen = NULL;
 
Signature · 1 line
2.33.1.1570.g069344fdd45
#184 lore raw

PATCH v6 7/8 various *.c: use isatty(0|2), not isatty(STDIN_FILENO|STDERR_FILENO)

In reply to Ævar Arnfjörð Bjarmason

We have over 50 uses of "isatty(1)" and "isatty(2)" in the codebase, and around 10 "isatty(0)", but these used the {STDIN_FILENO,STD{OUT,ERR}_FILENO} macros in "stdlib.h" to refer to them.

Let's change these for consistency, and because a subsequent commit's commit message outlines a recipe to change all of these for ad-hoc testing, not needing to match these with that ad-hoc regex will make things easier to explain.

---
 builtin/bisect--helper.c | 2 +-
 builtin/bundle.c         | 2 +-
 compat/mingw.c           | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
builtin/bisect--helper.c +1 -1
index 28a2e6a5750..21360a4e70b 100644
@@ -830,7 +830,7 @@ static int bisect_autostart(struct bisect_terms *terms)
 	fprintf_ln(stderr, _("You need to start by \"git bisect "
 			  "start\"\n"));
 
-	if (!isatty(STDIN_FILENO))
+	if (!isatty(0))
 		return -1;
 
 	/*
builtin/bundle.c +1 -1
index 5a85d7cd0fe..df69c651753 100644
@@ -56,7 +56,7 @@ static int parse_options_cmd_bundle(int argc,
 
 static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {
 	int all_progress_implied = 0;
-	int progress = isatty(STDERR_FILENO);
+	int progress = isatty(2);
 	struct strvec pack_opts;
 	int version = -1;
 	int ret;
compat/mingw.c +1 -1
index 9e0cd1e097f..0f545c1a7d1 100644
@@ -2374,7 +2374,7 @@ int mingw_raise(int sig)
 	switch (sig) {
 	case SIGALRM:
 		if (timer_fn == SIG_DFL) {
-			if (isatty(STDERR_FILENO))
+			if (isatty(2))
 				fputs("Alarm clock\n", stderr);
 			exit(128 + SIGALRM);
 		} else if (timer_fn != SIG_IGN)
Signature · 1 line
2.33.1.1570.g069344fdd45
#185 lore raw

PATCH v6 8/8 progress.c: add & assert a "global_progress" variable

In reply to Ævar Arnfjörð Bjarmason

The progress.c code makes a hard assumption that only one progress bar be active at a time (see [1] for a bug where this wasn't the case). Add a BUG() that'll trigger if we ever regress on that promise and have two progress bars active at the same time.

There was an alternative test-only approach to doing the same thing[2], but by doing this outside of a GIT_TEST_* mode we'll know we've put a hard stop to this particular API misuse.

It will also establish scaffolding to address current fundamental limitations in the progress output: The current output must be "driven" by calls to the likes of display_progress().

Once we have a global current progress object we'll be able to update that object via SIGALRM, this will cover cases where we're busy, but either haven't invoked our first display_progress() yet, or the time between display_progress() is too long. See [3] for early code to do that.

The linked code in [3] is WIP and not signal-safe since among other things it calls sprintf() from within a signal handler, see e.g. "man 7 signal-safety". But on some platforms a real implementation of it would be able to write() out a prepared-formatted progress update from within a signal handler. That would be sufficient to e.g. show that we're "stalled", or to display something like a simple pre-formatted "spinner".

It's conceivable that this change will hit the BUG() condition in some scenario that we don't currently have tests for, this would be very bad. If that happened we'd die just because we couldn't emit some pretty output.

See [4] for a discussion of why our test coverage is lacking; our progress display is hidden behind isatty(2) checks in many cases, so the test suite doesn't cover it unless individual tests are run in "--verbose" mode, we might also have multi-threaded use of the API, so two progress bars stopping and starting would only be visible due to a race condition.

Despite that, I think that this change won't introduce such regressions, because:

1. I've read all the code using the progress API (and have modified a large part of it in some WIP code I have). Almost all of it is really simple, the parts that aren't[5] are complex in the display_progress() part, not in starting or stopping the progress bar.

2. The entire test suite passes when instrumented with an ad-hoc Linux-specific mode (it uses gettid()) to die if progress bars are ever started or stopped on anything but the main thread[6].

Extending that to die if display_progress() is called in a thread reveals that we have exactly two users of the progress bar under threaded conditions, "git index-pack" and "git pack-objects". Both uses are straightforward, and they don't start/stop the progress bar when threads are active.

3. I've likewise done an ad-hoc test to force progress bars to be displayed with:

perl -pi -e 's[isatty\(2\)][1]g' $(git grep -l -F 'isatty(2)')

I.e. to replace all checks (not just for progress) of checking whether STDERR is connected to a TTY, and then monkeypatching is_foreground_fd() in progress.c to always "return 1". Running the tests with those applied, interactively and under -V reveals via:

$ grep -e set_progress_signal -e clear_progress_signal test-results/*out

That nothing our tests cover hits the BUG conditions added here, except the expected "BUG: start two concurrent progress bars" test being added here.

That isn't entirely true since we won't be getting 100% coverage due to cascading failures from tests that expected no progress output on stderr. To make sure I covered 100% I also tried making the display() function in progress.c a NOOP on top of that (it's the calls to start_progress_delay() and stop_progress()) that matter.

That doesn't hit the BUG() either. Some tests fail in that mode due to a combination of the overzealous isatty(2) munging noted above, and the tests that are testing that the progress output itself is present (but for testing I'd made display() a NOOP).

This doesn't address any currently out-of-tree user of progress.c, i.e. WIP patches, or progress output that's a part of forks of git.git. Those hopefully have test coverage that would expose the BUG().

If they don't they'll either run into it in code that displays more than one progress bar for the lifetime of the progress, or which calls stop_progress() with a non-NULL "progress" without a corresponding start_progress(). Both of those cases are less likely than the general cases of progress.c API misuse.

Between those three points above and the discussion of how this could impact out-of-tree users I think it's safe to go ahead with this change.

1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)
2. https://lore.kernel.org/git/[email protected]
3. https://lore.kernel.org/git/[email protected]/
4. https://lore.kernel.org/git/[email protected]/
5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into
next, 2021-09-10)
6. https://lore.kernel.org/git/[email protected]/

---
 progress.c                  | 18 ++++++++++++++++++
 t/t0500-progress-display.sh | 11 +++++++++++
 2 files changed, 29 insertions(+)
progress.c +18 -0
index 76a95cb7322..7483aec2e2a 100644
@@ -46,6 +46,7 @@ struct progress {
 };
 
 static volatile sig_atomic_t progress_update;
+static struct progress *global_progress;
 
 /*
  * These are only intended for testing the progress output, i.e. exclusively
@@ -249,6 +250,14 @@ void display_progress(struct progress *progress, uint64_t n)
 		display(progress, n, NULL);
 }
 
+static void set_global_progress(struct progress *progress)
+{
+	if (global_progress)
+		BUG("'%s' progress still active when trying to start '%s'",
+		    global_progress->title, progress->title);
+	global_progress = progress;
+}
+
 static struct progress *start_progress_delay(const char *title, uint64_t total,
 					     unsigned delay, unsigned sparse)
 {
@@ -264,6 +273,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,
 	strbuf_init(&progress->counters_sb, 0);
 	progress->title_len = utf8_strwidth(title);
 	progress->split = 0;
+	set_global_progress(progress);
 	set_progress_signal();
 	trace2_region_enter("progress", title, the_repository);
 	return progress;
@@ -340,6 +350,13 @@ void stop_progress(struct progress **p_progress)
 	stop_progress_msg(p_progress, _("done"));
 }
 
+static void unset_global_progress(void)
+{
+	if (!global_progress)
+		BUG("should have active global_progress when cleaning up");
+	global_progress = NULL;
+}
+
 void stop_progress_msg(struct progress **p_progress, const char *msg)
 {
 	struct progress *progress;
@@ -369,6 +386,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)
 		free(buf);
 	}
 	clear_progress_signal();
+	unset_global_progress();
 	strbuf_release(&progress->counters_sb);
 	if (progress->throughput)
 		strbuf_release(&progress->throughput->display);
t/t0500-progress-display.sh +11 -0
index 59e9f226ea4..867fdace3f2 100755
@@ -298,6 +298,17 @@ test_expect_success 'cover up after throughput shortens a lot' '
 	test_cmp expect out
 '
 
+test_expect_success 'BUG: start two concurrent progress bars' '
+	cat >in <<-\EOF &&
+	start 0 one
+	start 0 two
+	EOF
+
+	test_must_fail test-tool progress \
+		<in 2>stderr &&
+	grep "^BUG: .*'\''one'\'' progress still active when trying to start '\''two'\''$" stderr
+'
+
 test_expect_success 'progress generates traces' '
 	cat >in <<-\EOF &&
 	start 40
Signature · 1 line
2.33.1.1570.g069344fdd45
#186 lore raw

v9 cover progress: test fixes / cleanup

In reply to Ævar Arnfjörð Bjarmason

A re-roll of this series of various basic fixes to the progress API.

This addresses all the comments brought up on v8, thanks a lot to René and Johannes Altmanninger. It just took me about to a month to get around to re-rolling this.

There's various small fixes early in the series. I amended some commit messages, added a mising promised test etc.

But most significantly as René pointed out stop_progress() and stop_progress_msg() should really be unified. This series does that, which fixes a long-standing bug with the trace2 region not being ended if the stop_progress_msg() variant was called.

This is a reply to the v6, which is the thread this was originally in[1]. In v7[2] I didn't add In-Reply-T, and in v8[3] I ended up replying to the wrong thread! Sorry about that.

I then ejected the isatty() change. That was done for a final addition of a BUG() assertion that's now gone, but which I plan to submit after this series. I can re-visit that if and when I re-roll that.

1. https://lore.kernel.org/git/[email protected]
2. https://lore.kernel.org/git/[email protected]
3. https://lore.kernel.org/git/[email protected]

Ævar Arnfjörð Bjarmason (9):
leak tests: fix a memory leak in "test-progress" helper progress.c test helper: add missing braces
progress.c tests: make start/stop commands on stdin progress.c tests: test some invalid usage
progress.h: format and be consistent with progress.c naming progress.c: use dereferenced "progress" variable, not "(*p_progress)" progress.c: refactor stop_progress{,_msg}() to use helpers progress API: unify stop_progress{,_msg}(), fix trace2 bug pack-bitmap-write.c: don't return without stop_progress()

 pack-bitmap-write.c         |   6 +-
 progress.c                  |  66 ++++++++++------------
 progress.h                  |   9 ++-
 t/helper/test-progress.c    |  50 ++++++++++++-----
 t/t0500-progress-display.sh | 109 +++++++++++++++++++++++++++++-------
 t/t5316-pack-delta-depth.sh |   6 +-
 6 files changed, 170 insertions(+), 76 deletions(-)

Range-diff against v8:

 1:  aa08dab654d =  1:  dc304df9468 leak tests: fix a memory leak in "test-progress" helper
 2:  3ecdab074b6 =  2:  6951c059aa9 progress.c test helper: add missing braces
 3:  271f6d7ec3b !  3:  4548144a0b4 progress.c tests: make start/stop commands on stdin
    @@ t/helper/test-progress.c
     +		if (skip_prefix(line.buf, "start ", (const char **) &end)) {
     +			uint64_t total = strtoull(end, &end, 10);
     +			const char *title;
    -+			const char *str;
     +
     +			/*
     +			 * We can't use "end + 1" as an argument to
    @@ t/helper/test-progress.c
     +			else
     +				die("invalid input: '%s'\n", line.buf);
     +
    -+			str = title ? title : default_title;
    -+			progress = start_progress(str, total);
    ++			progress = start_progress(title, total);
     +		} else if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
      			uint64_t item_count = strtoull(end, &end, 10);
      			if (*end != '\0')
 4:  7c1b8b287c5 !  4:  965d4ba7b54 progress.c tests: test some invalid usage
    @@ Commit message
         progress.c tests: test some invalid usage
     
         Test what happens when we "stop" without a "start", omit the "stop"
    -    after a "start", or try to start two concurrent progress bars. This
    +    after a "start", or start two concurrent progress bars. This
         extends the trace2 tests added in 98a13647408 (trace2: log progress
         time and throughput, 2020-05-12).
     
    @@ t/t0500-progress-display.sh: test_expect_success 'progress generates traces' '
     +	stop
     +	EOF
     +
    -+	GIT_TRACE2_EVENT="$(pwd)/trace-startstop.event" test-tool progress \
    ++	GIT_TRACE2_EVENT="$PWD/trace-startstop.event" test-tool progress \
     +		<in 2>stderr &&
     +	test_region progress "Working hard" trace-startstop.event
     +'
    @@ t/t0500-progress-display.sh: test_expect_success 'progress generates traces' '
     +	start 0
     +	EOF
     +
    -+	GIT_TRACE2_EVENT="$(pwd)/trace-start.event" \
    ++	GIT_TRACE2_EVENT="$PWD/trace-start.event" \
     +	LSAN_OPTIONS=detect_leaks=0 \
     +	test-tool progress \
     +		<in 2>stderr &&
    @@ t/t0500-progress-display.sh: test_expect_success 'progress generates traces' '
     +	stop
     +	EOF
     +
    -+	GIT_TRACE2_EVENT="$(pwd)/trace-stop.event" test-tool progress \
    ++	GIT_TRACE2_EVENT="$PWD/trace-stop.event" test-tool progress \
     +		<in 2>stderr &&
     +	! grep region_enter.*progress trace-stop.event &&
     +	! grep region_leave.*progress trace-stop.event
     +'
    ++
    ++test_expect_success 'progress generates traces: start with active progress bar (no stops)' '
    ++	cat >in <<-\EOF &&
    ++	start 0 One
    ++	start 0 Two
    ++	EOF
    ++
    ++	GIT_TRACE2_EVENT="$PWD/trace-2start.event" \
    ++	LSAN_OPTIONS=detect_leaks=0 \
    ++	test-tool progress \
    ++		<in 2>stderr &&
    ++	grep region_enter.*progress.*One trace-2start.event &&
    ++	grep region_enter.*progress.*Two trace-2start.event &&
    ++	! grep region_leave trace-2start.event
    ++'
     +
      test_done
 -:  ----------- >  5:  4ab4eb3a20a progress.h: format and be consistent with progress.c naming
 5:  72a31bd7191 !  6:  ab24cb78d73 progress.c: add temporary variable from progress struct

@@ Metadata
Author: Ævar Arnfjörð Bjarmason <[email protected]>

## Commit message ##
- progress.c: add temporary variable from progress struct
+ progress.c: use dereferenced "progress" variable, not "(*p_progress)"

Since 98a13647408 (trace2: log progress time and throughput, 2020-05-12) stop_progress() dereferences a "struct progress **"
- parameter in several places. Extract a dereferenced variable (like in
- stop_progress_msg()) to reduce clutter and make it clearer who needs
- to write to this parameter.
+ parameter in several places. Extract a dereferenced variable to reduce
+ clutter and make it clearer who needs to write to this parameter.

Now instead of using "*p_progress" several times in stop_progress() we check it once for NULL and then use a dereferenced "progress" variable
- thereafter. This continues the same pattern used in the above
- stop_progress() function, see ac900fddb7f (progress: don't dereference
- before checking for NULL, 2020-08-10).
+ thereafter. This uses the same pattern as the adjacent
+ stop_progress_msg() function, see ac900fddb7f (progress: don't
+ dereference before checking for NULL, 2020-08-10).

@@ progress.c: static void finish_if_sparse(struct progress *progress)
void stop_progress(struct progress **p_progress)
{
+ struct progress *progress;
++
if (!p_progress)
BUG("don't provide NULL to stop_progress");
+ progress = *p_progress;

 -:  ----------- >  7:  3406d71b499 progress.c: refactor stop_progress{,_msg}() to use helpers
 -:  ----------- >  8:  62a93bb98b0 progress API: unify stop_progress{,_msg}(), fix trace2 bug
 6:  0bd08e1b018 =  9:  0b2248a2d74 pack-bitmap-write.c: don't return without stop_progress()
 7:  060483fb5ce <  -:  ----------- *.c: use isatty(0|2), not isatty(STDIN_FILENO|STDERR_FILENO)
Signature · 1 line
2.35.1.939.g42bf83caa3d
#187 lore raw

PATCH v9 1/9 leak tests: fix a memory leak in "test-progress" helper

Fix a memory leak in the test-progress helper, and mark the corresponding "t0500-progress-display.sh" test as being leak-free under SANITIZE=leak. This fixes a leak added in 2bb74b53a4 (Test the progress display, 2019-09-16).

My 48f68715b14 (tr2: stop leaking "thread_name" memory, 2021-08-27) had fixed another memory leak in this test (as it did some trace2 testing).

---
 t/helper/test-progress.c    | 1 +
 t/t0500-progress-display.sh | 1 +
 2 files changed, 2 insertions(+)
t/helper/test-progress.c +1 -0
index 5d05cbe7894..9265e6ab7cf 100644
@@ -69,6 +69,7 @@ int cmd__progress(int argc, const char **argv)
 			die("invalid input: '%s'\n", line.buf);
 	}
 	stop_progress(&progress);
+	strbuf_release(&line);
 
 	return 0;
 }
t/t0500-progress-display.sh +1 -0
index 22058b503ac..f37cf2eb9c9 100755
@@ -2,6 +2,7 @@
 
 test_description='progress display'
 
+TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 show_cr () {
Signature · 1 line
2.35.1.939.g42bf83caa3d
#188 lore raw

PATCH v9 2/9 progress.c test helper: add missing braces

In reply to Ævar Arnfjörð Bjarmason

If we have braces on one arm of an if/else all of them should have it, per the CodingGuidelines's "When there are multiple arms to a conditional[...]" advice. This formatting change makes a subsequent commit smaller.

---
 t/helper/test-progress.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
t/helper/test-progress.c +3 -2
index 9265e6ab7cf..50fd3be3dad 100644
@@ -63,10 +63,11 @@ int cmd__progress(int argc, const char **argv)
 				die("invalid input: '%s'\n", line.buf);
 			progress_test_ns = test_ms * 1000 * 1000;
 			display_throughput(progress, byte_count);
-		} else if (!strcmp(line.buf, "update"))
+		} else if (!strcmp(line.buf, "update")) {
 			progress_test_force_update();
-		else
+		} else {
 			die("invalid input: '%s'\n", line.buf);
+		}
 	}
 	stop_progress(&progress);
 	strbuf_release(&line);
Signature · 1 line
2.35.1.939.g42bf83caa3d
#189 lore raw

PATCH v9 3/9 progress.c tests: make start/stop commands on stdin

In reply to Ævar Arnfjörð Bjarmason

Change the usage of the "test-tool progress" introduced in 2bb74b53a49 (Test the progress display, 2019-09-16) to take command like "start" and "stop" on stdin, instead of running them implicitly.

This makes for tests that are easier to read, since the recipe will mirror the API usage, and allows for easily testing invalid usage that would yield (or should yield) a BUG(), e.g. providing two "start" calls in a row. A subsequent commit will add such tests.

---
 t/helper/test-progress.c    | 44 +++++++++++++++++++++-------
 t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------
 2 files changed, 70 insertions(+), 32 deletions(-)
t/helper/test-progress.c +33 -11
index 50fd3be3dad..6cc9735b601 100644
@@ -3,6 +3,9 @@
  *
  * Reads instructions from standard input, one instruction per line:
  *
+ *   "start <total>[ <title>]" - Call start_progress(title, total),
+ *                               Uses the default title of "Working hard"
+ *                               if the " <title>" is omitted.
  *   "progress <items>" - Call display_progress() with the given item count
  *                        as parameter.
  *   "throughput <bytes> <millis> - Call display_throughput() with the given
@@ -10,6 +13,7 @@
  *                                  specify the time elapsed since the
  *                                  start_progress() call.
  *   "update" - Set the 'progress_update' flag.
+ *   "stop" - Call stop_progress().
  *
  * See 't0500-progress-display.sh' for examples.
  */
@@ -19,34 +23,50 @@
 #include "parse-options.h"
 #include "progress.h"
 #include "strbuf.h"
+#include "string-list.h"
 
 int cmd__progress(int argc, const char **argv)
 {
-	int total = 0;
-	const char *title;
+	const char *const default_title = "Working hard";
+	struct string_list titles = STRING_LIST_INIT_DUP;
 	struct strbuf line = STRBUF_INIT;
-	struct progress *progress;
+	struct progress *progress = NULL;
 
 	const char *usage[] = {
-		"test-tool progress [--total=<n>] <progress-title>",
+		"test-tool progress <stdin",
 		NULL
 	};
 	struct option options[] = {
-		OPT_INTEGER(0, "total", &total, "total number of items"),
 		OPT_END(),
 	};
 
 	argc = parse_options(argc, argv, NULL, options, usage, 0);
-	if (argc != 1)
-		die("need a title for the progress output");
-	title = argv[0];
+	if (argc)
+		usage_with_options(usage, options);
 
 	progress_testing = 1;
-	progress = start_progress(title, total);
 	while (strbuf_getline(&line, stdin) != EOF) {
 		char *end;
 
-		if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
+		if (skip_prefix(line.buf, "start ", (const char **) &end)) {
+			uint64_t total = strtoull(end, &end, 10);
+			const char *title;
+
+			/*
+			 * We can't use "end + 1" as an argument to
+			 * start_progress(), it doesn't xstrdup() its
+			 * "title" argument. We need to hold onto a
+			 * valid "char *" for it until the end.
+			 */
+			if (!*end)
+				title = default_title;
+			else if (*end == ' ')
+				title = string_list_insert(&titles, end + 1)->string;
+			else
+				die("invalid input: '%s'\n", line.buf);
+
+			progress = start_progress(title, total);
+		} else if (skip_prefix(line.buf, "progress ", (const char **) &end)) {
 			uint64_t item_count = strtoull(end, &end, 10);
 			if (*end != '\0')
 				die("invalid input: '%s'\n", line.buf);
@@ -65,12 +85,14 @@ int cmd__progress(int argc, const char **argv)
 			display_throughput(progress, byte_count);
 		} else if (!strcmp(line.buf, "update")) {
 			progress_test_force_update();
+		} else if (!strcmp(line.buf, "stop")) {
+			stop_progress(&progress);
 		} else {
 			die("invalid input: '%s'\n", line.buf);
 		}
 	}
-	stop_progress(&progress);
 	strbuf_release(&line);
+	string_list_clear(&titles, 0);
 
 	return 0;
 }
t/t0500-progress-display.sh +37 -21

204 lines not on this page · raw message

Signature · 1 line
2.35.1.939.g42bf83caa3d
#190 lore raw

PATCH v9 4/9 progress.c tests: test some invalid usage

In reply to Ævar Arnfjörð Bjarmason

Test what happens when we "stop" without a "start", omit the "stop" after a "start", or start two concurrent progress bars. This extends the trace2 tests added in 98a13647408 (trace2: log progress time and throughput, 2020-05-12).

These tests are not merely testing the helper, but invalid API usage that can happen if the progress.c API is misused.

The "without stop" test will leak under SANITIZE=leak, since this buggy use of the API will leak memory. But let's not skip it entirely, or use the "!SANITIZE_LEAK" prerequisite check as we'd do with tests that we're skipping due to leaks we haven't fixed yet. Instead annotate the specific command that should skip leak checking with custom $LSAN_OPTIONS[1].

1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer

---
 t/t0500-progress-display.sh | 50 +++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
t/t0500-progress-display.sh +50 -0
index 27ab4218b01..1eb3a8306ba 100755
@@ -325,4 +325,54 @@ test_expect_success 'progress generates traces' '
 	grep "\"key\":\"total_bytes\",\"value\":\"409600\"" trace.event
 '
 
+test_expect_success 'progress generates traces: stop / start' '
+	cat >in <<-\EOF &&
+	start 0
+	stop
+	EOF
+
+	GIT_TRACE2_EVENT="$PWD/trace-startstop.event" test-tool progress \
+		<in 2>stderr &&
+	test_region progress "Working hard" trace-startstop.event
+'
+
+test_expect_success 'progress generates traces: start without stop' '
+	cat >in <<-\EOF &&
+	start 0
+	EOF
+
+	GIT_TRACE2_EVENT="$PWD/trace-start.event" \
+	LSAN_OPTIONS=detect_leaks=0 \
+	test-tool progress \
+		<in 2>stderr &&
+	grep region_enter.*progress trace-start.event &&
+	! grep region_leave.*progress trace-start.event
+'
+
+test_expect_success 'progress generates traces: stop without start' '
+	cat >in <<-\EOF &&
+	stop
+	EOF
+
+	GIT_TRACE2_EVENT="$PWD/trace-stop.event" test-tool progress \
+		<in 2>stderr &&
+	! grep region_enter.*progress trace-stop.event &&
+	! grep region_leave.*progress trace-stop.event
+'
+
+test_expect_success 'progress generates traces: start with active progress bar (no stops)' '
+	cat >in <<-\EOF &&
+	start 0 One
+	start 0 Two
+	EOF
+
+	GIT_TRACE2_EVENT="$PWD/trace-2start.event" \
+	LSAN_OPTIONS=detect_leaks=0 \
+	test-tool progress \
+		<in 2>stderr &&
+	grep region_enter.*progress.*One trace-2start.event &&
+	grep region_enter.*progress.*Two trace-2start.event &&
+	! grep region_leave trace-2start.event
+'
+
 test_done
Signature · 1 line
2.35.1.939.g42bf83caa3d
#191 lore raw

PATCH v9 5/9 progress.h: format and be consistent with progress.c naming

In reply to Ævar Arnfjörð Bjarmason

Fix an inconsistency introduced in dc6a0757c4f (make struct progress an opaque type, 2007-10-30) and rename the "progress" parameters to stop_progress{,_msg}() to "p_progress". Now these match the corresponding parameters in the *.c code.

While we're at it let's move the definition of the former below the latter, a subsequent change will start defining stop_progress() in terms of stop_progress_msg(). Let's also remove the excess whitespace at the end of the file.

---
 progress.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
progress.h +2 -3
index f1913acf73f..4f6806904a8 100644
@@ -18,7 +18,6 @@ struct progress *start_sparse_progress(const char *title, uint64_t total);
 struct progress *start_delayed_progress(const char *title, uint64_t total);
 struct progress *start_delayed_sparse_progress(const char *title,
 					       uint64_t total);
-void stop_progress(struct progress **progress);
-void stop_progress_msg(struct progress **progress, const char *msg);
-
+void stop_progress_msg(struct progress **p_progress, const char *msg);
+void stop_progress(struct progress **p_progress);
 #endif
Signature · 1 line
2.35.1.939.g42bf83caa3d
#192 lore raw

PATCH v9 6/9 progress.c: use dereferenced "progress" variable, not "(*p_progress)"

In reply to Ævar Arnfjörð Bjarmason

Since 98a13647408 (trace2: log progress time and throughput, 2020-05-12) stop_progress() dereferences a "struct progress **" parameter in several places. Extract a dereferenced variable to reduce clutter and make it clearer who needs to write to this parameter.

Now instead of using "*p_progress" several times in stop_progress() we check it once for NULL and then use a dereferenced "progress" variable thereafter. This uses the same pattern as the adjacent stop_progress_msg() function, see ac900fddb7f (progress: don't dereference before checking for NULL, 2020-08-10).

---
 progress.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
progress.c +9 -6
index 680c6a8bf93..6e7daa3f8a2 100644
@@ -319,21 +319,24 @@ static void finish_if_sparse(struct progress *progress)
 
 void stop_progress(struct progress **p_progress)
 {
+	struct progress *progress;
+
 	if (!p_progress)
 		BUG("don't provide NULL to stop_progress");
+	progress = *p_progress;
 
-	finish_if_sparse(*p_progress);
+	finish_if_sparse(progress);
 
-	if (*p_progress) {
+	if (progress) {
 		trace2_data_intmax("progress", the_repository, "total_objects",
-				   (*p_progress)->total);
+				   progress->total);
 
-		if ((*p_progress)->throughput)
+		if (progress->throughput)
 			trace2_data_intmax("progress", the_repository,
 					   "total_bytes",
-					   (*p_progress)->throughput->curr_total);
+					   progress->throughput->curr_total);
 
-		trace2_region_leave("progress", (*p_progress)->title, the_repository);
+		trace2_region_leave("progress", progress->title, the_repository);
 	}
 
 	stop_progress_msg(p_progress, _("done"));
Signature · 1 line
2.35.1.939.g42bf83caa3d
#193 lore raw

PATCH v9 7/9 progress.c: refactor stop_progress{,_msg}() to use helpers

In reply to Ævar Arnfjörð Bjarmason

Create two new static helpers for the stop_progress() and stop_progress_msg() functions.

As we'll see in the subsequent commit having those two split up doesn't make much sense, and results in a bug in how we log to trace2. This narrow preparatory change makes the diff for that subsequent change smaller.

---
 progress.c | 64 ++++++++++++++++++++++++++++++------------------------
 1 file changed, 36 insertions(+), 28 deletions(-)
progress.c +36 -28
index 6e7daa3f8a2..6cc7f902f5e 100644
@@ -317,6 +317,36 @@ static void finish_if_sparse(struct progress *progress)
 		display_progress(progress, progress->total);
 }
 
+static void force_last_update(struct progress *progress, const char *msg)
+{
+	char *buf;
+	struct throughput *tp = progress->throughput;
+
+	if (tp) {
+		uint64_t now_ns = progress_getnanotime(progress);
+		unsigned int misecs, rate;
+		misecs = ((now_ns - progress->start_ns) * 4398) >> 32;
+		rate = tp->curr_total / (misecs ? misecs : 1);
+		throughput_string(&tp->display, tp->curr_total, rate);
+	}
+	progress_update = 1;
+	buf = xstrfmt(", %s.\n", msg);
+	display(progress, progress->last_value, buf);
+	free(buf);
+}
+
+static void log_trace2(struct progress *progress)
+{
+	trace2_data_intmax("progress", the_repository, "total_objects",
+			   progress->total);
+
+	if (progress->throughput)
+		trace2_data_intmax("progress", the_repository, "total_bytes",
+				   progress->throughput->curr_total);
+
+	trace2_region_leave("progress", progress->title, the_repository);
+}
+
 void stop_progress(struct progress **p_progress)
 {
 	struct progress *progress;
@@ -327,17 +357,8 @@ void stop_progress(struct progress **p_progress)
 
 	finish_if_sparse(progress);
 
-	if (progress) {
-		trace2_data_intmax("progress", the_repository, "total_objects",
-				   progress->total);
-
-		if (progress->throughput)
-			trace2_data_intmax("progress", the_repository,
-					   "total_bytes",
-					   progress->throughput->curr_total);
-
-		trace2_region_leave("progress", progress->title, the_repository);
-	}
+	if (progress)
+		log_trace2(*p_progress);
 
 	stop_progress_msg(p_progress, _("done"));
 }
@@ -353,23 +374,10 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)
 	if (!progress)
 		return;
 	*p_progress = NULL;
-	if (progress->last_value != -1) {
-		/* Force the last update */
-		char *buf;
-		struct throughput *tp = progress->throughput;
-
-		if (tp) {
-			uint64_t now_ns = progress_getnanotime(progress);
-			unsigned int misecs, rate;
-			misecs = ((now_ns - progress->start_ns) * 4398) >> 32;
-			rate = tp->curr_total / (misecs ? misecs : 1);
-			throughput_string(&tp->display, tp->curr_total, rate);
-		}
-		progress_update = 1;
-		buf = xstrfmt(", %s.\n", msg);
-		display(progress, progress->last_value, buf);
-		free(buf);
-	}
+
+	if (progress->last_value != -1)
+		force_last_update(progress, msg);
+
 	clear_progress_signal();
 	strbuf_release(&progress->counters_sb);
 	if (progress->throughput)
Signature · 1 line
2.35.1.939.g42bf83caa3d
#194 lore raw

PATCH v9 8/9 progress API: unify stop_progress{,_msg}(), fix trace2 bug

In reply to Ævar Arnfjörð Bjarmason

Fix a bug that's been with us ever since 98a13647408 (trace2: log progress time and throughput, 2020-05-12), when the stop_progress_msg() API was used we didn't log a "region_leave" for the "region_enter" we start in "start_progress_delay()".

The only user of the "stop_progress_msg()" function is "index-pack". Let's add a previously failing test to check that we have the same number of "region_enter" and "region_leave" events, with "-v" we'll log progress even in the test environment.

In addition to that we've had a submarine bug here introduced with 9d81ecb52b5 (progress: add sparse mode to force 100% complete message, 2019-03-21). The "start_sparse_progress()" API would only do the right thing if the progress was ended with "stop_progress()", not "stop_progress_msg()".

The only user of that API uses "stop_progress()", but let's still fix that along with the trace2 issue by making "stop_progress()" a trivial wrapper for "stop_progress_msg()".

We can also drop the "if (progress)" test from "finish_if_sparse()". It's now a helper for the small "stop_progress_msg()" function. We'll already have returned from it if "progress" is "NULL".

---
 progress.c                  | 21 +++------------------
 progress.h                  |  6 +++++-
 t/t5316-pack-delta-depth.sh |  6 +++++-
 3 files changed, 13 insertions(+), 20 deletions(-)
progress.c +3 -18
index 6cc7f902f5e..0cdd875d37f 100644
@@ -311,8 +311,7 @@ struct progress *start_delayed_sparse_progress(const char *title,
 
 static void finish_if_sparse(struct progress *progress)
 {
-	if (progress &&
-	    progress->sparse &&
+	if (progress->sparse &&
 	    progress->last_value != progress->total)
 		display_progress(progress, progress->total);
 }
@@ -347,22 +346,6 @@ static void log_trace2(struct progress *progress)
 	trace2_region_leave("progress", progress->title, the_repository);
 }
 
-void stop_progress(struct progress **p_progress)
-{
-	struct progress *progress;
-
-	if (!p_progress)
-		BUG("don't provide NULL to stop_progress");
-	progress = *p_progress;
-
-	finish_if_sparse(progress);
-
-	if (progress)
-		log_trace2(*p_progress);
-
-	stop_progress_msg(p_progress, _("done"));
-}
-
 void stop_progress_msg(struct progress **p_progress, const char *msg)
 {
 	struct progress *progress;
@@ -375,8 +358,10 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)
 		return;
 	*p_progress = NULL;
 
+	finish_if_sparse(progress);
 	if (progress->last_value != -1)
 		force_last_update(progress, msg);
+	log_trace2(progress);
 
 	clear_progress_signal();
 	strbuf_release(&progress->counters_sb);
progress.h +5 -1
index 4f6806904a8..3a945637c81 100644
@@ -1,5 +1,6 @@
 #ifndef PROGRESS_H
 #define PROGRESS_H
+#include "gettext.h"
 
 struct progress;
 
@@ -19,5 +20,8 @@ struct progress *start_delayed_progress(const char *title, uint64_t total);
 struct progress *start_delayed_sparse_progress(const char *title,
 					       uint64_t total);
 void stop_progress_msg(struct progress **p_progress, const char *msg);
-void stop_progress(struct progress **p_progress);
+static inline void stop_progress(struct progress **p_progress)
+{
+	stop_progress_msg(p_progress, _("done"));
+}
 #endif
t/t5316-pack-delta-depth.sh +5 -1
index df524f7b6dd..e9045009a11 100755
@@ -64,7 +64,11 @@ test_expect_success 'create series of packs' '
 			echo $cur &&
 			echo "$(git rev-parse :file) file"
 		} | git pack-objects --stdout >tmp &&
-		git index-pack --stdin --fix-thin <tmp || return 1
+		GIT_TRACE2_EVENT=$PWD/trace \
+		git index-pack -v --stdin --fix-thin <tmp || return 1 &&
+		grep -c region_enter.*progress trace >enter &&
+		grep -c region_leave.*progress trace >leave &&
+		test_cmp enter leave &&
 		prev=$cur
 	done
 '
Signature · 1 line
2.35.1.939.g42bf83caa3d
#195 lore raw

PATCH v9 9/9 pack-bitmap-write.c: don't return without stop_progress()

In reply to Ævar Arnfjörð Bjarmason

Fix a bug that's been here since 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21), we did not call stop_progress() if we reached the early exit in this function.

We could call stop_progress() before we return, but better yet is to defer calling start_progress() until we need it. For now this only matters in practice because we'd previously omit the "region_leave" for the progress trace2 event.

---
 pack-bitmap-write.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
pack-bitmap-write.c +3 -3
index 9c55c1531e1..cab3eaa2acd 100644
@@ -575,15 +575,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,
 
 	QSORT(indexed_commits, indexed_commits_nr, date_compare);
 
-	if (writer.show_progress)
-		writer.progress = start_progress("Selecting bitmap commits", 0);
-
 	if (indexed_commits_nr < 100) {
 		for (i = 0; i < indexed_commits_nr; ++i)
 			push_bitmapped_commit(indexed_commits[i]);
 		return;
 	}
 
+	if (writer.show_progress)
+		writer.progress = start_progress("Selecting bitmap commits", 0);
+
 	for (;;) {
 		struct commit *chosen = NULL;
 
Signature · 1 line
2.35.1.939.g42bf83caa3d
#196 lore raw

Re: [PATCH v9 0/9] progress: test fixes / cleanup

In reply to Ævar Arnfjörð Bjarmason
Quoted 4 lines · Ævar Arnfjörð Bjarmason

But most significantly as René pointed out stop_progress() and stop_progress_msg() should really be unified. This series does that, which fixes a long-standing bug with the trace2 region not being ended if the stop_progress_msg() variant was called.

;-)

Yes, it is very pleasing to see it fixed.

Will take a look and queue but after finishing the week's "What's cooking" report.

Thanks.