Skip to contentSkip to conversation

http: exponential backoff, configurable status codes, and post_rpc() retries

PATCH Superseded Ravi Mistry via GitGitGadget 5 participants 49 messages Oct 31 – Nov 19 Documentation

lore
Share & tools
lore
#0 lore raw

cover http: exponential backoff, configurable status codes, and post_rpc() retries

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) added HTTP 429 retry support using http.maxRetries, http.retryAfter, and http.maxRetryTime.

In practice, when running Git against hosting backends and reverse proxies under high concurrency, four gaps remain:

1. Off-by-one retry count and Retry-After Trace2 timing: Because http_request_recoverable() pre-decrements the retry counter (--rate_limit_retries) in its loop condition, setting http.maxRetries=1 performs 0 retries (and N configured retries performs N-1 retries). In addition, the Retry-After header is extracted after Trace2 logging has already run, so the http/429-retry-after Trace2 event always logs -1.

2. No exponential backoff when Retry-After is absent: When a server does not send a Retry-After header, Git waits the same fixed http.retryAfter delay on every retry. If that delay is short (e.g., 1s), Git burns through all of its retries in just a few seconds before a busy server can recover. If that delay is long, even a brief glitch has to wait a long time on the first retry. Doubling the delay on each attempt (1s, 2s, 4s, ...) keeps the first retry fast while giving the server more time to recover on later attempts, matching curl's own --retry behavior.

3. Transient 5xx errors cannot be retried: Reverse proxies, load balancers, and Git backends frequently surface transient overload or connection resets as 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout. Because 5xx errors are not universally transient across all environments, 429 should remain the default, with an opt-in config to specify additional status codes.

4. HTTP POST requests are not retried: Git's existing 429 retry logic only covers HTTP GET requests (such as the initial /info/refs handshake). However, the actual data transfer during fetch and clone, as well as on-demand blob downloads in partial clones (during checkout, diff, or blame), are sent as HTTP POST requests (/git-upload-pack) via post_rpc(), which only retries authentication and immediately aborts on 429 or 5xx.

This 4-patch series addresses all four gaps:

* Patch 1/4 fixes the off-by-one retry count bug and extracts Retry-After in finish_active_slot() before Trace2 logging runs.
* Patch 2/4 doubles http.retryAfter on each subsequent retry attempt (1s, 2s, 4s, capped at http.maxRetryTime) when Retry-After is absent.
* Patch 3/4 adds http.retryStatusCodes (and GIT_HTTP_RETRY_STATUS_CODES), defaulting to "429", so environments can opt into retrying codes like "429,502,503,504".
* Patch 4/4 extracts a shared http_retry_wait() helper and extends retry support to HTTP POST requests in probe_rpc() and post_rpc() (while fixing a missing Content-Encoding: gzip header on retry).

Prior discussions:

* https://lore.kernel.org/git/[email protected]/
* https://lore.kernel.org/git/[email protected]/

Ravi Mistry (4):
http: fix retry count and Retry-After timing
http: add exponential backoff without Retry-After
http: add http.retryStatusCodes config option
remote-curl: retry transient HTTP errors in post_rpc()

 Documentation/config/http.adoc |  42 ++++++---
 http.c                         | 122 ++++++++++++++++++--------
 http.h                         |  12 ++-
 remote-curl.c                  |  75 ++++++++--------
 t/lib-httpd/http-429.sh        |  63 ++++++++++++--
 t/t5584-http-429-retry.sh      | 155 +++++++++++++++++++++++++++++++++
 6 files changed, 373 insertions(+), 96 deletions(-)

base-commit: 8d1b215c01512b9658d4525c54ff9b9245f9a7e8
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2287%2Frmistry%2Fhttp-exponential-retries-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2287/rmistry/http-exponential-retries-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2287

Signature · 1 line
gitgitgadget
#1 lore raw

PATCH 1/4 http: fix retry count and Retry-After timing

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) introduced two small bugs in HTTP 429 retry handling:

1. In http_request_recoverable(), rate_limit_retries is initialized to http_max_retries (the number of retries to perform after the initial failed request) and decremented with pre-decrement (--rate_limit_retries) in the while condition. Because of the pre-decrement, setting http.maxRetries=1 performs 0 retries, and more generally N configured retries only retries N-1 times. In addition, if the first request returns 401 (HTTP_REAUTH) and the re-authenticated request returns 429 when http.maxRetries is 0, --rate_limit_retries wraps from 0 to -1 (truthy in C).

Fix this by checking http_max_retries > 0 && rate_limit_retries-- > 0 in the loop condition and returning HTTP_ERROR when http_max_retries <= 0.

2. In http_request(), CURLINFO_RETRY_AFTER is queried after run_one_slot() has already called handle_curl_result(), which logs results->retry_after to Trace2 ("http/429-retry-after") while it is still -1. Move the CURLINFO_RETRY_AFTER extraction into finish_active_slot() (initializing slot->results->retry_after = -1 first) so that both handle_curl_result() and callers of run_one_slot() observe the parsed Retry-After value.

---
 http.c                    | 27 +++++++++++++--------------
 t/lib-httpd/http-429.sh   | 31 ++++++++++++++++++++++++++-----
 t/t5584-http-429-retry.sh | 21 +++++++++++++++++++++
 3 files changed, 60 insertions(+), 19 deletions(-)
http.c +13 -14
index d79678d01f..0308f973a6 100644
@@ -345,11 +345,20 @@ static void finish_active_slot(struct active_request_slot *slot)
 	if (slot->results) {
 		slot->results->curl_result = slot->curl_result;
 		slot->results->http_code = slot->http_code;
+		slot->results->retry_after = -1;
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTPAUTH_AVAIL,
 				  &slot->results->auth_avail);
 
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CONNECTCODE,
 			&slot->results->http_connectcode);
+#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
+		{
+			curl_off_t retry_after;
+			if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
+					      &retry_after) == CURLE_OK && retry_after > 0)
+				slot->results->retry_after = (long)retry_after;
+		}
+#endif
 	}
 
 	/* Run callback if appropriate */
@@ -2286,15 +2295,6 @@ static int http_request(const char *url,
 
 	ret = run_one_slot(slot, &results);
 
-#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
-	if (ret == HTTP_RATE_LIMITED) {
-		curl_off_t retry_after;
-		if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
-				      &retry_after) == CURLE_OK && retry_after > 0)
-			results.retry_after = (long)retry_after;
-	}
-#endif
-
 	options->retry_after = results.retry_after;
 
 	if (options->content_type) {
@@ -2461,10 +2461,6 @@ static int http_request_recoverable(const char *url,
 	if (ret != HTTP_OK && ret != HTTP_REAUTH && ret != HTTP_RATE_LIMITED)
 		return ret;
 
-	/* If retries are disabled and we got a 429, fail immediately */
-	if (ret == HTTP_RATE_LIMITED && !http_max_retries)
-		return HTTP_ERROR;
-
 	if (options->effective_url && options->base_url) {
 		if (update_url_from_redirect(options->base_url,
 					     url, options->effective_url)) {
@@ -2474,7 +2470,8 @@ static int http_request_recoverable(const char *url,
 	}
 
 	while ((ret == HTTP_REAUTH && --i) ||
-	       (ret == HTTP_RATE_LIMITED && --rate_limit_retries)) {
+	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
+		rate_limit_retries-- > 0)) {
 		long retry_delay = -1;
 		/*
 		 * The previous request may have put cruft into our output stream; we
@@ -2518,6 +2515,8 @@ static int http_request_recoverable(const char *url,
 		ret = http_request(url, result, target, options);
 	}
 	if (ret == HTTP_RATE_LIMITED) {
+		if (http_max_retries <= 0)
+			return HTTP_ERROR;
 		trace2_data_string("http", the_repository,
 				   "http/429-error", "retries-exhausted");
 		return HTTP_RATE_LIMITED;
t/lib-httpd/http-429.sh +26 -5
index 1a5d7987db..5537fb7ac4 100644
@@ -32,12 +32,33 @@ repo_name="${repo_path%%/*}"
 safe_name=$(echo "${test_context}-${retry_after}-${repo_name}" | tr '/' '_' | tr -cd 'a-zA-Z0-9_-')
 state="http-429-state-${safe_name}"
 
-# Check if this is the first call (no state directory exists), or if
-# the retry-after-value is "permanent", which indicates a 429 must be
-# returned for every request (even if the state directory exists).
-if test "$retry_after" != permanent && ! mkdir "$state" 2>/dev/null
+max_fails=1
+case "$test_context" in
+	*-fail-2*) max_fails=2 ;;
+	*-fail-3*) max_fails=3 ;;
+esac
+
+should_fail=0
+if test "$retry_after" = permanent
+then
+	should_fail=1
+else
+	mkdir -p "$state" 2>/dev/null
+	i=1
+	while test $i -le $max_fails
+	do
+		if mkdir "$state/$i" 2>/dev/null
+		then
+			should_fail=1
+			break
+		fi
+		i=$((i + 1))
+	done
+fi
+
+if test $should_fail -eq 0
 then
-	# Already returned 429 once, forward to git-http-backend
+	# Already returned 429 max_fails times, forward to git-http-backend
 	# Set PATH_INFO to just the repo path (without retry-after value)
 	# Set GIT_PROJECT_ROOT so git-http-backend can find the repository
 	# Use exec to replace this process so git-http-backend gets the updated environment
t/t5584-http-429-retry.sh +21 -0
index a22007b2cf..d92a26893d 100755
@@ -51,6 +51,27 @@ test_expect_success 'HTTP 429 with Retry-After is retried and succeeds' '
 	test_grep "refs/heads/" output
 '
 
+test_expect_success 'http.maxRetries=1 retries once and succeeds' '
+	test_config http.maxRetries 1 &&
+	git ls-remote "$HTTPD_URL/http_429/max-retries-one/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output
+'
+
+test_expect_success 'http.maxRetries=2 fails after 3 failures and succeeds on 2 failures' '
+	test_config http.maxRetries 2 &&
+	test_must_fail git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-3/none/repo.git" 2>err &&
+	test_grep "rate limited" err &&
+	git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-2/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output
+'
+
+test_expect_success 'Trace2 logs actual CURLINFO_RETRY_AFTER value instead of -1' '
+	test_config http.maxRetries 1 &&
+	GIT_TRACE2_EVENT="$(pwd)/trace-retry-after.event" \
+		git ls-remote "$HTTPD_URL/http_429/trace2-retry-after/1/repo.git" >output 2>err &&
+	test_grep "\"key\":\"http/429-retry-after\",\"value\":\"1\"" trace-retry-after.event
+'
+
 test_expect_success 'HTTP 429 without Retry-After uses configured default' '
 	# Enable retries and configure default delay
 	test_config http.maxRetries 3 &&
Signature · 1 line
gitgitgadget
#2 lore raw

Re: [PATCH 1/4] http: fix retry count and Retry-After timing

Quoted 11 lines · Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) introduced two small bugs in HTTP 429 retry handling:

1. In http_request_recoverable(), rate_limit_retries is initialized to http_max_retries (the number of retries to perform after the initial failed request) and decremented with pre-decrement (--rate_limit_retries) in the while condition. Because of the pre-decrement, setting http.maxRetries=1 performs 0 retries, and more generally N configured retries only retries N-1 times. In

The new test exactly demonstrates it, which is fun.

Quoted 4 lines

In
addition, if the first request returns 401 (HTTP_REAUTH) and the re-authenticated request returns 429 when http.maxRetries is 0, --rate_limit_retries wraps from 0 to -1 (truthy in C).

Hmph, which would mean we would retry essentially forever until the integer wraps around again? That sounds, eh, nasty.

Quoted 3 lines
   Fix this by checking http_max_retries > 0 && rate_limit_retries-- > 0
   in the loop condition and returning HTTP_ERROR when
   http_max_retries <= 0.

OK.

Quoted 43 lines

2. In http_request(), CURLINFO_RETRY_AFTER is queried after run_one_slot() has already called handle_curl_result(), which logs results->retry_after to Trace2 ("http/429-retry-after") while it is still -1. Move the CURLINFO_RETRY_AFTER extraction into finish_active_slot() (initializing slot->results->retry_after = -1 first) so that both handle_curl_result() and callers of run_one_slot() observe the parsed Retry-After value.

---
 http.c                    | 27 +++++++++++++--------------
 t/lib-httpd/http-429.sh   | 31 ++++++++++++++++++++++++++-----
 t/t5584-http-429-retry.sh | 21 +++++++++++++++++++++
 3 files changed, 60 insertions(+), 19 deletions(-)
http.c +9 -0
index d79678d01f..0308f973a6 100644
@@ -345,11 +345,20 @@ static void finish_active_slot(struct active_request_slot *slot)
 	if (slot->results) {
 		slot->results->curl_result = slot->curl_result;
 		slot->results->http_code = slot->http_code;
+		slot->results->retry_after = -1;
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTPAUTH_AVAIL,
 				  &slot->results->auth_avail);
 
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CONNECTCODE,
 			&slot->results->http_connectcode);
+#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
+		{
+			curl_off_t retry_after;
+			if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
+					      &retry_after) == CURLE_OK && retry_after > 0)
+				slot->results->retry_after = (long)retry_after;
+		}
+#endif
 	}
 
 	/* Run callback if appropriate */
@@ -2286,15 +2295,6 @@ static int http_request(const char *url,
 
 	ret = run_one_slot(slot, &results);

http.c static int http_request(const char *url, · line 2297

Quoted 10 lines
http.c +0 -9
 
-#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
-	if (ret == HTTP_RATE_LIMITED) {
-		curl_off_t retry_after;
-		if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
-				      &retry_after) == CURLE_OK && retry_after > 0)
-			results.retry_after = (long)retry_after;
-	}
-#endif
-

OK, this is the fix #2 explained in the log message.

http.c static int http_request_recoverable(const char *url, · line 2467

Quoted 8 lines
http.c +0 -4
@@ -2461,10 +2461,6 @@ static int http_request_recoverable(const char *url,
 	if (ret != HTTP_OK && ret != HTTP_REAUTH && ret != HTTP_RATE_LIMITED)
 		return ret;
 
-	/* If retries are disabled and we got a 429, fail immediately */
-	if (ret == HTTP_RATE_LIMITED && !http_max_retries)
-		return HTTP_ERROR;
-

I am not sure I get this removal. If it is because the change to the next hunk (the condition to loop with the while statement) already covers it, I guess it makes sense, because we never enter the while() loop and we will return HTTP_ERROR more or less the same way as the above if http_max_retries is false.

Quoted 16 lines
http.c +4 -1
 	while ((ret == HTTP_REAUTH && --i) ||
-	       (ret == HTTP_RATE_LIMITED && --rate_limit_retries)) {
+	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
+		rate_limit_retries-- > 0)) {
 		long retry_delay = -1;
 		/*
 		 * The previous request may have put cruft into our output stream; we
@@ -2518,6 +2515,8 @@ static int http_request_recoverable(const char *url,
 		ret = http_request(url, result, target, options);
 	}
 	if (ret == HTTP_RATE_LIMITED) {
+		if (http_max_retries <= 0)
+			return HTTP_ERROR;
 		trace2_data_string("http", the_repository,
 				   "http/429-error", "retries-exhausted");
 		return HTTP_RATE_LIMITED;
#3 lore raw

Re: [PATCH 1/4] http: fix retry count and Retry-After timing

Quoted 10 lines · Junio C Hamano

http.c static int http_request_recoverable(const char *url, · line 2467

Quoted 4 lines
http.c +0 -4
-	/* If retries are disabled and we got a 429, fail immediately */
-	if (ret == HTTP_RATE_LIMITED && !http_max_retries)
-		return HTTP_ERROR;
-

I am not sure I get this removal. If it is because the change to the next hunk (the condition to loop with the while statement) already covers it, I guess it makes sense, because we never enter the while() loop and we will return HTTP_ERROR more or less the same way as the above if http_max_retries is false.

Yes, and moving that check to after the while() loop also handles the case where the initial request returns 401 and the re-authenticated request inside the loop returns 429 when http.maxRetries is 0, without falsely logging "retries-exhausted". I clarified this in the commit message in v2.

Thanks!

#4 lore raw

PATCH 2/4 http: add exponential backoff without Retry-After

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

When a server returns HTTP 429 without a Retry-After header, Git waits the same fixed http.retryAfter delay on every retry. If that delay is short (e.g., 1s), Git burns through all of its retries in just a few seconds before a busy server can recover. If that delay is long, even a brief glitch has to wait a long time on the first retry.

Pass the 1-based retry attempt count to handle_rate_limit_retry() and double http.retryAfter on each subsequent retry (1s, 2s, 4s, ..., via http.retryAfter * 2^(attempt - 1)), clamping the delay at http.maxRetryTime. This keeps the first retry fast while giving the server more time to recover on later attempts, matching curl's own --retry behavior. When the server provides an explicit Retry-After header, continue to honor the server-provided delay (or fail fast if it exceeds http.maxRetryTime).

---
 Documentation/config/http.adoc | 11 ++++++++---
 http.c                         | 16 +++++++++++++---
 t/t5584-http-429-retry.sh      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 6 deletions(-)
Documentation/config/http.adoc +8 -3
index 792a71b413..b387b0ad6a 100644
@@ -327,9 +327,12 @@ http.keepAliveCount::
 	`GIT_HTTP_KEEPALIVE_COUNT` environment variable.
 
 http.retryAfter::
-	Default wait time in seconds before retrying when a server returns
+	Initial wait time in seconds before retrying when a server returns
 	HTTP 429 (Too Many Requests) without a Retry-After header.
-	Defaults to 0 (retry immediately). When a Retry-After header is
+	Defaults to 0 (retry immediately). When set to a positive value and
+	no Retry-After header is present, the delay doubles on each
+	subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
+	capped at `http.maxRetryTime`. When a Retry-After header is
 	present, its value takes precedence over this setting; however,
 	automatic use of the server-provided `Retry-After` header requires
 	libcurl 7.66.0 or later. On older versions, configure this setting
@@ -348,7 +351,9 @@ http.maxRetryTime::
 	handling HTTP 429 (Too Many Requests) responses. If the server
 	requests a delay (via Retry-After header) or if `http.retryAfter`
 	is configured with a value that exceeds this maximum, Git will fail
-	immediately rather than waiting. Default is 300 seconds (5 minutes).
+	immediately rather than waiting. Exponentially backed-off delays
+	computed from `http.retryAfter` are capped at this maximum. Default
+	is 300 seconds (5 minutes).
 	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
 	variable. See also `http.retryAfter` and `http.maxRetries`.
 
http.c +13 -3
index 0308f973a6..0dec6377dd 100644
@@ -2411,7 +2411,7 @@ static int update_url_from_redirect(struct strbuf *base,
  * Returns a negative value if configuration is invalid (delay exceeds
  * http.maxRetryTime), otherwise returns the delay in seconds (>= 0).
  */
-static long handle_rate_limit_retry(long slot_retry_after)
+static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 {
 	/* Use the slot-specific retry_after value or configured default */
 	if (slot_retry_after >= 0) {
@@ -2427,6 +2427,8 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		return slot_retry_after;
 	} else {
+		long delay = http_retry_after;
+
 		/* No Retry-After header provided, use configured default */
 		if (http_retry_after > http_max_retry_time) {
 			error(_("configured http.retryAfter exceeds http.maxRetryTime (%ld > %ld seconds)"),
@@ -2437,7 +2439,14 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		trace2_data_string("http", the_repository,
 			"http/429-retry-source", "config-default");
-		return http_retry_after;
+		for (int a = 1; a < attempt && delay > 0; a++) {
+			if (delay >= http_max_retry_time - delay) {
+				delay = http_max_retry_time;
+				break;
+			}
+			delay *= 2;
+		}
+		return delay;
 	}
 }
 
@@ -2449,6 +2458,7 @@ static int http_request_recoverable(const char *url,
 	int i = 3;
 	int ret;
 	int rate_limit_retries = http_max_retries;
+	int attempt = 0;
 
 	if (!options)
 		options = &empty_opts;
@@ -2498,7 +2508,7 @@ static int http_request_recoverable(const char *url,
 			BUG("Unknown http_request target");
 		}
 		if (ret == HTTP_RATE_LIMITED) {
-			retry_delay = handle_rate_limit_retry(options->retry_after);
+			retry_delay = handle_rate_limit_retry(options->retry_after, ++attempt);
 			if (retry_delay < 0)
 				return HTTP_ERROR;
 
t/t5584-http-429-retry.sh +33 -0
index d92a26893d..f0c3cc3b61 100755
@@ -82,6 +82,39 @@ test_expect_success 'HTTP 429 without Retry-After uses configured default' '
 	test_grep "refs/heads/" output
 '
 
+test_expect_success 'exponential backoff doubles http.retryAfter without Retry-After header' '
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
+	git ls-remote "$HTTPD_URL/http_429/exp-backoff-fail-3/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep "waiting 2 seconds before retry" err &&
+	test_grep "waiting 4 seconds before retry" err
+'
+
+test_expect_success 'exponential backoff clamps delay at http.maxRetryTime' '
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
+	test_config http.maxRetryTime 2 &&
+	GIT_TRACE2_EVENT="$(pwd)/trace-clamp.event" \
+		git ls-remote "$HTTPD_URL/http_429/exp-clamp-fail-3/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep "waiting 2 seconds before retry" err &&
+	test_grep ! "waiting 4 seconds before retry" err &&
+	grep "\"key\":\"http/retry-sleep-seconds\",\"value\":\"2\"" trace-clamp.event >clamped &&
+	test_line_count = 2 clamped
+'
+
+test_expect_success 'server Retry-After header overrides exponential backoff' '
+	test_config http.maxRetries 2 &&
+	test_config http.retryAfter 1 &&
+	git ls-remote "$HTTPD_URL/http_429/header-override-fail-2/1/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep ! "waiting 2 seconds before retry" err
+'
+
 test_expect_success 'HTTP 429 retry delays are respected' '
 	# Enable retries
 	test_config http.maxRetries 3 &&
Signature · 1 line
gitgitgadget
#5 lore raw

Re: [PATCH 2/4] http: add exponential backoff without Retry-After

Quoted 8 lines · Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

When a server returns HTTP 429 without a Retry-After header, Git waits the same fixed http.retryAfter delay on every retry. If that delay is short (e.g., 1s), Git burns through all of its retries in just a few seconds before a busy server can recover. If that delay is long, even a brief glitch has to wait a long time on the first retry.

Quoted 8 lines

Pass the 1-based retry attempt count to handle_rate_limit_retry() and double http.retryAfter on each subsequent retry (1s, 2s, 4s, ..., via http.retryAfter * 2^(attempt - 1)), clamping the delay at http.maxRetryTime. This keeps the first retry fast while giving the server more time to recover on later attempts, matching curl's own --retry behavior. When the server provides an explicit Retry-After header, continue to honor the server-provided delay (or fail fast if it exceeds http.maxRetryTime).

The logic makes perfect sense (I am taking the above description at face value without checking if this really matches cURL's behavior myself, though).

Documentation/config/http.adoc http.keepAliveCount:: · line 335

Quoted 7 lines

http.retryAfter::

Documentation/config/http.adoc +5 -0
+	Initial wait time in seconds before retrying when a server returns
 	HTTP 429 (Too Many Requests) without a Retry-After header.
+	Defaults to 0 (retry immediately). When set to a positive value and
+	no Retry-After header is present, the delay doubles on each
+	subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
+	capped at `http.maxRetryTime`. When a Retry-After header is

It somehow feels that the default value of 0 (immediate retry) does not mesh well with exponential backoff. Will we keep retrying immediately after failing until we run out of retries?

Also, it feels somewhat inconsistent with how maxretrytime is currently treated. Specifically, when the other side tells us via Retry-After, or the configuration specifies via http.retryAfter, to wait longer than maxretrytime, instead of capping the wait, we immediately give up without retrying. But when retryAfter grows with exponential backoff, the wait is capped at maxretrytime instead of resulting in an immediate failure.

The handling of maxretrytime in this definition, where it caps the waiting time before retrying, is definitely more useful than the traditional "if we are told to wait longer than this, we fail instead of capping", I would think.

Quoted 28 lines
Documentation/config/http.adoc +4 -2
@@ -348,7 +351,9 @@ http.maxRetryTime::
 	handling HTTP 429 (Too Many Requests) responses. If the server
 	requests a delay (via Retry-After header) or if `http.retryAfter`
 	is configured with a value that exceeds this maximum, Git will fail
-	immediately rather than waiting. Default is 300 seconds (5 minutes).
+	immediately rather than waiting. Exponentially backed-off delays
+	computed from `http.retryAfter` are capped at this maximum. Default
+	is 300 seconds (5 minutes).
 
-static long handle_rate_limit_retry(long slot_retry_after)
+static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 {
http.c +2 -1
 	/* Use the slot-specific retry_after value or configured default */
 	if (slot_retry_after >= 0) {
@@ -2427,6 +2427,8 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		return slot_retry_after;
 	} else {
+		long delay = http_retry_after;
+
 		/* No Retry-After header provided, use configured default */
 		if (http_retry_after > http_max_retry_time) {
 			error(_("configured http.retryAfter exceeds http.maxRetryTime (%ld > %ld seconds)"),
@@ -2437,7 +2439,14 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		trace2_data_string("http", the_repository,
 			"http/429-retry-source", "config-default");
-		return http_retry_after;

http.c static long handle_rate_limit_retry(long slot_retry_after) · line 2451

Quoted 10 lines
http.c +8 -0
+		for (int a = 1; a < attempt && delay > 0; a++) {
+			if (delay >= http_max_retry_time - delay) {
+				delay = http_max_retry_time;
+				break;
+			}
+			delay *= 2;
+		}
+		return delay;
 	}
 }

I guess what is in the loop is a tricky arithmetic to compute

	delay = ((delay * 2) > http_max_retry_time)
		? http_max_retry_time
		: (delay * 2);

without integer overflow? Very effective.

Quoted 9 lines
http.c +1 -0
@@ -2449,6 +2458,7 @@ static int http_request_recoverable(const char *url,
 	int i = 3;
 	int ret;
 	int rate_limit_retries = http_max_retries;
+	int attempt = 0;
 
 	if (!options)
 		options = &empty_opts;

http.c static int http_request_recoverable(const char *url, · line 2513

Quoted 7 lines
http.c +1 -1
@@ -2498,7 +2508,7 @@ static int http_request_recoverable(const char *url,
 			BUG("Unknown http_request target");
 		}
 		if (ret == HTTP_RATE_LIMITED) {
-			retry_delay = handle_rate_limit_retry(options->retry_after);
+			retry_delay = handle_rate_limit_retry(options->retry_after, ++attempt);
 			if (retry_delay < 0)
 				return HTTP_ERROR;

Interesting.

We already have to deal with the maximum number of retries to honor http_max_retries, so we must be counting how many times we have retried so far. But we still need to introduce another variable to count the number of attempts.

Or would it suffice to pass (http_max_retries - rate_limit_retries) in place of the 'attempt' parameter?

Not complaining. Just making an observation.

#6 lore raw

Re: [PATCH 2/4] http: add exponential backoff without Retry-After

Quoted 3 lines · Junio C Hamano

It somehow feels that the default value of 0 (immediate retry) does not mesh well with exponential backoff. Will we keep retrying immediately after failing until we run out of retries?

Right, when http.retryAfter is 0 (and no Retry-After header is sent), every retry happens immediately without waiting. I kept 0 as the default to preserve the existing behavior from 640657ffd0 (and its t5584 tests). I made this explicit in the http.retryAfter and http.maxRetryTime documentation in v2.

Quoted 4 lines

Or would it suffice to pass (http_max_retries - rate_limit_retries) in place of the 'attempt' parameter?

Not complaining. Just making an observation.

Passing (http_max_retries - rate_limit_retries) here and only introducing 'attempt' in Patch 4/4 is much cleaner, so I updated v2 to do that.

Thanks!

#7 lore raw

PATCH 3/4 http: add http.retryStatusCodes config option

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) hardcoded HTTP 429 as the only status code eligible for retry. In practice, reverse proxies, load balancers, and Git hosting backends also return transient 5xx responses such as 502 Bad Gateway, 503 Service Unavailable (which RFC 9110 section 10.2.3 defines to support Retry-After), and 504 Gateway Timeout during load shedding or upstream connection resets.

Because 5xx responses are not universally transient across all environments, keep 429 as the default and introduce a new http.retryStatusCodes configuration option (and corresponding GIT_HTTP_RETRY_STATUS_CODES environment variable) accepting a comma- or whitespace-separated list of HTTP status codes in the 400..599 range (for example, "429,502,503,504").

Keep the existing "rate limited" warning and error messages for HTTP 429, and report the numeric status code (e.g., "HTTP 502, waiting <N> seconds before retry") when retrying other status codes.

---
 Documentation/config/http.adoc | 31 ++++++++++++++++------
 http.c                         | 39 ++++++++++++++++++++++++---
 http.h                         |  6 +++--
 t/lib-httpd/http-429.sh        | 10 +++++--
 t/t5584-http-429-retry.sh      | 48 ++++++++++++++++++++++++++++++++++
 5 files changed, 118 insertions(+), 16 deletions(-)
Documentation/config/http.adoc +23 -8
index b387b0ad6a..8e02a1d754 100644
@@ -328,7 +328,8 @@ http.keepAliveCount::
 
 http.retryAfter::
 	Initial wait time in seconds before retrying when a server returns
-	HTTP 429 (Too Many Requests) without a Retry-After header.
+	a retryable HTTP status code (HTTP 429 by default) without a
+	Retry-After header.
 	Defaults to 0 (retry immediately). When set to a positive value and
 	no Retry-After header is present, the delay doubles on each
 	subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
@@ -338,24 +339,38 @@ http.retryAfter::
 	libcurl 7.66.0 or later. On older versions, configure this setting
 	manually to control the retry delay. Can be overridden by the
 	`GIT_HTTP_RETRY_AFTER` environment variable.
-	See also `http.maxRetries` and `http.maxRetryTime`.
+	See also `http.maxRetries`, `http.maxRetryTime`, and
+	`http.retryStatusCodes`.
 
 http.maxRetries::
-	Maximum number of times to retry after receiving HTTP 429 (Too Many
-	Requests) responses. Set to 0 (the default) to disable retries.
+	Maximum number of times to retry after receiving a retryable HTTP
+	status code (HTTP 429 by default). Set to 0 (the default) to disable
+	retries.
 	Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
-	See also `http.retryAfter` and `http.maxRetryTime`.
+	See also `http.retryAfter`, `http.maxRetryTime`, and
+	`http.retryStatusCodes`.
 
 http.maxRetryTime::
 	Maximum time in seconds to wait for a single retry attempt when
-	handling HTTP 429 (Too Many Requests) responses. If the server
-	requests a delay (via Retry-After header) or if `http.retryAfter`
+	handling a retryable HTTP status code (HTTP 429 by default). If the
+	server requests a delay (via Retry-After header) or if `http.retryAfter`
 	is configured with a value that exceeds this maximum, Git will fail
 	immediately rather than waiting. Exponentially backed-off delays
 	computed from `http.retryAfter` are capped at this maximum. Default
 	is 300 seconds (5 minutes).
 	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
-	variable. See also `http.retryAfter` and `http.maxRetries`.
+	variable. See also `http.retryAfter`, `http.maxRetries`, and
+	`http.retryStatusCodes`.
+
+http.retryStatusCodes::
+	Comma- or whitespace-separated list of HTTP status codes (in the
+	400..599 range) that should be retried up to `http.maxRetries`
+	times. Defaults to `429`. For example, setting this to
+	`429,502,503,504` also retries transient `5xx` gateway and
+	service-unavailable errors. Can be overridden by the
+	`GIT_HTTP_RETRY_STATUS_CODES` environment variable.
+	See also `http.maxRetries`, `http.retryAfter`, and
+	`http.maxRetryTime`.
 
 http.noEPSV::
 	A boolean which disables using of EPSV ftp command by curl.
http.c +35 -4
index 0dec6377dd..950e2bbffa 100644
@@ -156,6 +156,7 @@ static int http_schannel_check_revoke = 1;
 static long http_retry_after = 0;
 static long http_max_retries = 0;
 static long http_max_retry_time = 300;
+static char *http_retry_status_codes;
 
 /*
  * With the backend being set to `schannel`, setting sslCAinfo would override
@@ -607,6 +608,11 @@ static int http_options(const char *var, const char *value,
 		return 0;
 	}
 
+	if (!strcmp("http.retrystatuscodes", var)) {
+		FREE_AND_NULL(http_retry_status_codes);
+		return git_config_string(&http_retry_status_codes, var, value);
+	}
+
 	/* Fall back on the default ones */
 	return git_default_config(var, value, ctx, data);
 }
@@ -1518,6 +1524,7 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
 	set_long_from_env(&http_retry_after, "GIT_HTTP_RETRY_AFTER");
 	set_long_from_env(&http_max_retries, "GIT_HTTP_MAX_RETRIES");
 	set_long_from_env(&http_max_retry_time, "GIT_HTTP_MAX_RETRY_TIME");
+	set_from_env(&http_retry_status_codes, "GIT_HTTP_RETRY_STATUS_CODES");
 
 	curl_default = get_curl_handle();
 }
@@ -1579,6 +1586,7 @@ void http_cleanup(void)
 	proxy_ssl_cert_password_required = 0;
 
 	FREE_AND_NULL(cached_accept_language);
+	FREE_AND_NULL(http_retry_status_codes);
 }
 
 struct active_request_slot *get_active_slot(void)
@@ -1928,6 +1936,23 @@ void normalize_curl_result(CURLcode *result, long http_code,
 	}
 }
 
+static int is_retryable_http_code(long code)
+{
+	const char *p = http_retry_status_codes ? http_retry_status_codes : "429";
+	char *end;
+
+	for (; code >= 400 && code < 600 && *p; p = end) {
+		while (*p == ',' || isspace(*p))
+			p++;
+		if (strtol(p, &end, 10) == code && end > p &&
+		    (!*end || *end == ',' || isspace(*end)))
+			return 1;
+		while (*end && *end != ',' && !isspace(*end))
+			end++;
+	}
+	return 0;
+}
+
 static int handle_curl_result(struct slot_results *results)
 {
 	normalize_curl_result(&results->curl_result, results->http_code,
@@ -1981,7 +2006,7 @@ static int handle_curl_result(struct slot_results *results)
 			}
 			return HTTP_REAUTH;
 		}
-	} else if (results->http_code == 429) {
+	} else if (is_retryable_http_code(results->http_code)) {
 		trace2_data_intmax("http", the_repository, "http/429-retry-after",
 			results->retry_after);
 		return HTTP_RATE_LIMITED;
@@ -2296,6 +2321,7 @@ static int http_request(const char *url,
 	ret = run_one_slot(slot, &results);
 
 	options->retry_after = results.retry_after;
+	options->http_code = results.http_code;
 
 	if (options->content_type) {
 		struct strbuf raw = STRBUF_INIT;
@@ -2407,7 +2433,7 @@ static int update_url_from_redirect(struct strbuf *base,
 }
 
 /*
- * Compute the retry delay for an HTTP 429 response.
+ * Compute the retry delay for a retryable HTTP response.
  * Returns a negative value if configuration is invalid (delay exceeds
  * http.maxRetryTime), otherwise returns the delay in seconds (>= 0).
  */
@@ -2513,7 +2539,12 @@ static int http_request_recoverable(const char *url,
 				return HTTP_ERROR;
 
 			if (retry_delay > 0) {
-				warning(_("rate limited, waiting %ld seconds before retry"), retry_delay);
+				if (options->http_code == 429)
+					warning(_("rate limited, waiting %ld seconds before retry"),
+						retry_delay);
+				else
+					warning(_("HTTP %ld, waiting %ld seconds before retry"),
+						options->http_code, retry_delay);
 				trace2_data_intmax("http", the_repository,
 						   "http/retry-sleep-seconds", retry_delay);
 				sleep(retry_delay);
@@ -2529,7 +2560,7 @@ static int http_request_recoverable(const char *url,
 			return HTTP_ERROR;
 		trace2_data_string("http", the_repository,
 				   "http/429-error", "retries-exhausted");
-		return HTTP_RATE_LIMITED;
+		return options->http_code == 429 ? HTTP_RATE_LIMITED : HTTP_ERROR;
 	}
 	return ret;
 }
http.h +4 -2
index 729c51904d..5985a0f71f 100644
@@ -167,10 +167,12 @@ struct http_get_options {
 
 	/*
 	 * After a request completes, contains the Retry-After delay in seconds
-	 * if the server returned HTTP 429 with a Retry-After header (requires
-	 * libcurl 7.66.0 or later), or -1 if no such header was present.
+	 * if the server returned a retryable status code with a Retry-After
+	 * header (requires libcurl 7.66.0 or later), or -1 if no such header
+	 * was present.
 	 */
 	long retry_after;
+	long http_code;
 };
 
 /* Return values for http_get_*() */
t/lib-httpd/http-429.sh +8 -2
index 5537fb7ac4..5c7d58a75f 100644
@@ -75,8 +75,14 @@ then
 	exec "$GIT_EXEC_PATH/git-http-backend"
 fi
 
-# Output HTTP 429 response
-printf "Status: 429 Too Many Requests\r\n"
+# Output HTTP 429 (or configured status) response
+status_line="429 Too Many Requests"
+case "$test_context" in
+	*-status-502*) status_line="502 Bad Gateway" ;;
+	*-status-503*) status_line="503 Service Unavailable" ;;
+	*-status-504*) status_line="504 Gateway Timeout" ;;
+esac
+printf "Status: %s\r\n" "$status_line"
 
 # Set Retry-After header based on retry_after value
 case "$retry_after" in
t/t5584-http-429-retry.sh +48 -0

52 lines not on this page · raw message

Signature · 1 line
gitgitgadget
#8 lore raw

Re: [PATCH 3/4] http: add http.retryStatusCodes config option

Documentation/config/http.adoc http.keepAliveCount:: · line 332

Quoted 5 lines · Ravi Mistry via GitGitGadget

http.retryAfter::

Documentation/config/http.adoc +2 -1
 	Initial wait time in seconds before retrying when a server returns
-	HTTP 429 (Too Many Requests) without a Retry-After header.
+	a retryable HTTP status code (HTTP 429 by default) without a
+	Retry-After header.

This, and ...

Documentation/config/http.adoc http.retryAfter:: · line 348

Quoted 6 lines

http.maxRetries::

Documentation/config/http.adoc +3 -2
-	Maximum number of times to retry after receiving HTTP 429 (Too Many
-	Requests) responses. Set to 0 (the default) to disable retries.
+	Maximum number of times to retry after receiving a retryable HTTP
+	status code (HTTP 429 by default). Set to 0 (the default) to disable
+	retries.

this, and ...

Documentation/config/http.adoc http.retryAfter:: · line 356

Quoted 11 lines
Documentation/config/http.adoc +4 -3
 	Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
-	See also `http.retryAfter` and `http.maxRetryTime`.
+	See also `http.retryAfter`, `http.maxRetryTime`, and
+	`http.retryStatusCodes`.
 
 http.maxRetryTime::
 	Maximum time in seconds to wait for a single retry attempt when
-	handling HTTP 429 (Too Many Requests) responses. If the server
-	requests a delay (via Retry-After header) or if `http.retryAfter`
+	handling a retryable HTTP status code (HTTP 429 by default). If the
+	server requests a delay (via Retry-After header) or if `http.retryAfter`

this all share two issues.

- "Too Many Requests", which is a helpful comment to highlight what 429 is, is lost, and

- What we want to say is that there is a customizable set of HTTP response codes that we treat as "the other side invites us to retry if we want to", and that HTTP 429 is the ONLY member of that set by default. But "HTTP 429 by default" alone in the above description does not quite say so.

It is unlikely that the reader realizes that we are talking about the default membership in a set until ...

Documentation/config/http.adoc http.retryAfter:: · line 373

Quoted 9 lines
Documentation/config/http.adoc +9 -0
+http.retryStatusCodes::
+	Comma- or whitespace-separated list of HTTP status codes (in the
+	400..599 range) that should be retried up to `http.maxRetries`
+	times. Defaults to `429`. For example, setting this to
+	`429,502,503,504` also retries transient `5xx` gateway and
+	service-unavailable errors. Can be overridden by the
+	`GIT_HTTP_RETRY_STATUS_CODES` environment variable.
+	See also `http.maxRetries`, `http.retryAfter`, and
+	`http.maxRetryTime`.

... they read this, which is very clearly written.

Quoted 13 lines
http.c +1 -0
index 0dec6377dd..950e2bbffa 100644
@@ -156,6 +156,7 @@ static int http_schannel_check_revoke = 1;
 static long http_retry_after = 0;
 static long http_max_retries = 0;
 static long http_max_retry_time = 300;
+static char *http_retry_status_codes;
 
 /*
  * With the backend being set to `schannel`, setting sslCAinfo would override

http.c static int http_options(const char *var, const char *value, · line 614

Quoted 7 lines
http.c +4 -0
@@ -607,6 +608,11 @@ static int http_options(const char *var, const char *value,
 		return 0;
 	}
 
+	if (!strcmp("http.retrystatuscodes", var)) {
+		FREE_AND_NULL(http_retry_status_codes);
+		return git_config_string(&http_retry_status_codes, var, value);
+	}

OK, the input is a simple string.

Quoted 6 lines
http.c +6 -0
+static int is_retryable_http_code(long code)
+{
+	const char *p = http_retry_status_codes ? http_retry_status_codes : "429";
+	char *end;
+
+	for (; code >= 400 && code < 600 && *p; p = end) {

http.c void normalize_curl_result(CURLcode *result, long http_code, · line 1954

Quoted 10 lines
http.c +10 -0
+		while (*p == ',' || isspace(*p))
+			p++;
+		if (strtol(p, &end, 10) == code && end > p &&
+		    (!*end || *end == ',' || isspace(*end)))
+			return 1;
+		while (*end && *end != ',' && !isspace(*end))
+			end++;
+	}
+	return 0;
+}

Ahh, we parse the string every time?

I guess it is not performance-critical (after all, we are talking about waiting and then retrying here), but I somehow imagined we would allocate an array of ints and pre-parse the string upon reading the configuration variable.

Also 400 <= code < 600 is a constant throughout the function and it is a tad strange to see it in the loop control part of a for() statement.

In any case, the strtol() based parsing looks cleanly done.

Thanks.

#9 lore raw

Re: [PATCH 3/4] http: add http.retryStatusCodes config option

Quoted 8 lines · Junio C Hamano

- "Too Many Requests", which is a helpful comment to highlight what 429 is, is lost, and

- What we want to say is that there is a customizable set of HTTP response codes that we treat as "the other side invites us to retry if we want to", and that HTTP 429 is the ONLY member of that set by default. But "HTTP 429 by default" alone in the above description does not quite say so.

Makes sense. In v2 I restored "Too Many Requests" and stated explicitly in all three entries that the set of retryable status codes is configured by http.retryStatusCodes, which includes only HTTP 429 by default.

Quoted 3 lines

Also 400 <= code < 600 is a constant throughout the function and it is a tad strange to see it in the loop control part of a for() statement.

I moved the 400 <= code < 600 check to an early return at the top of is_retryable_http_code() in v2.

Thanks!

#10 lore raw

PATCH 4/4 remote-curl: retry transient HTTP errors in post_rpc()

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Git's existing retry logic only covers HTTP GET requests. However, the actual data transfer during fetch and clone, as well as on-demand blob downloads in partial clones, are sent as HTTP POST requests via post_rpc(), which only retries authentication and immediately aborts on 429 or 5xx.

Extract the retry wait logic from http_request_recoverable() into a shared http_retry_wait() helper and use it to retry transient HTTP errors in probe_rpc() and post_rpc(). Also defer reporting RPC errors until retries are exhausted, and re-attach the "Content-Encoding: gzip" header when retrying a compressed request.

---
 http.c                    | 64 ++++++++++++++++++---------------
 http.h                    |  6 ++++
 remote-curl.c             | 75 +++++++++++++++++++--------------------
 t/lib-httpd/http-429.sh   | 50 ++++++++++++++++++--------
 t/t5584-http-429-retry.sh | 53 +++++++++++++++++++++++++++
 5 files changed, 167 insertions(+), 81 deletions(-)
http.c +35 -29
index 950e2bbffa..36b559e596 100644
@@ -2476,6 +2476,34 @@ static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 	}
 }
 
+int http_retry_wait(long http_code, long slot_retry_after, int attempt)
+{
+	long retry_delay;
+
+	if (http_max_retries <= 0)
+		return -1;
+	if (attempt > http_max_retries) {
+		trace2_data_string("http", the_repository,
+				   "http/429-error", "retries-exhausted");
+		return -1;
+	}
+	retry_delay = handle_rate_limit_retry(slot_retry_after, attempt);
+	if (retry_delay < 0)
+		return -1;
+	if (retry_delay > 0) {
+		if (http_code == 429)
+			warning(_("rate limited, waiting %ld seconds before retry"),
+				retry_delay);
+		else
+			warning(_("HTTP %ld, waiting %ld seconds before retry"),
+				http_code, retry_delay);
+		trace2_data_intmax("http", the_repository,
+				   "http/retry-sleep-seconds", retry_delay);
+		sleep(retry_delay);
+	}
+	return 0;
+}
+
 static int http_request_recoverable(const char *url,
 			       void *result, int target,
 			       struct http_get_options *options)
@@ -2483,7 +2511,6 @@ static int http_request_recoverable(const char *url,
 	static struct http_get_options empty_opts;
 	int i = 3;
 	int ret;
-	int rate_limit_retries = http_max_retries;
 	int attempt = 0;
 
 	if (!options)
@@ -2506,9 +2533,8 @@ static int http_request_recoverable(const char *url,
 	}
 
 	while ((ret == HTTP_REAUTH && --i) ||
-	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
-		rate_limit_retries-- > 0)) {
-		long retry_delay = -1;
+	       (ret == HTTP_RATE_LIMITED &&
+		!http_retry_wait(options->http_code, options->retry_after, ++attempt))) {
 		/*
 		 * The previous request may have put cruft into our output stream; we
 		 * should clear it out before making our next request.
@@ -2533,35 +2559,15 @@ static int http_request_recoverable(const char *url,
 		default:
 			BUG("Unknown http_request target");
 		}
-		if (ret == HTTP_RATE_LIMITED) {
-			retry_delay = handle_rate_limit_retry(options->retry_after, ++attempt);
-			if (retry_delay < 0)
-				return HTTP_ERROR;
-
-			if (retry_delay > 0) {
-				if (options->http_code == 429)
-					warning(_("rate limited, waiting %ld seconds before retry"),
-						retry_delay);
-				else
-					warning(_("HTTP %ld, waiting %ld seconds before retry"),
-						options->http_code, retry_delay);
-				trace2_data_intmax("http", the_repository,
-						   "http/retry-sleep-seconds", retry_delay);
-				sleep(retry_delay);
-			}
-		} else if (ret == HTTP_REAUTH) {
+		if (ret == HTTP_REAUTH)
 			http_reauth_prepare(1);
-		}
 
 		ret = http_request(url, result, target, options);
 	}
-	if (ret == HTTP_RATE_LIMITED) {
-		if (http_max_retries <= 0)
-			return HTTP_ERROR;
-		trace2_data_string("http", the_repository,
-				   "http/429-error", "retries-exhausted");
-		return options->http_code == 429 ? HTTP_RATE_LIMITED : HTTP_ERROR;
-	}
+	if (ret == HTTP_RATE_LIMITED &&
+	    (http_max_retries <= 0 || attempt <= http_max_retries ||
+	     options->http_code != 429))
+		return HTTP_ERROR;
 	return ret;
 }
 
http.h +6 -0
index 5985a0f71f..fa9f46a11b 100644
@@ -201,6 +201,12 @@ int http_get_strbuf(const char *url, struct strbuf *result, struct http_get_opti
 int http_get_file(const char *url, const char *filename,
 		  struct http_get_options *options);
 
+/*
+ * Wait before retrying a transient HTTP error (1-based attempt count).
+ * Returns 0 if the caller should retry, or -1 to stop retrying.
+ */
+int http_retry_wait(long http_code, long slot_retry_after, int attempt);
+
 int http_fetch_ref(const char *base, struct ref *ref);
 
 struct curl_slist *http_append_auth_header(const struct credential *c,
remote-curl.c +37 -38

130 lines not on this page · raw message

t/lib-httpd/http-429.sh +36 -14

63 lines not on this page · raw message

t/t5584-http-429-retry.sh +53 -0

57 lines not on this page · raw message

Signature · 1 line
gitgitgadget
#11 lore raw

Re: [PATCH 4/4] remote-curl: retry transient HTTP errors in post_rpc()

Quoted 25 lines · Ravi Mistry via GitGitGadget
http.c +17 -0
index 950e2bbffa..36b559e596 100644
@@ -2476,6 +2476,34 @@ static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 	}
 }
 
+int http_retry_wait(long http_code, long slot_retry_after, int attempt)
+{
+	long retry_delay;
+
+	if (http_max_retries <= 0)
+		return -1;
+	if (attempt > http_max_retries) {
+		trace2_data_string("http", the_repository,
+				   "http/429-error", "retries-exhausted");
+		return -1;
+	}
+	retry_delay = handle_rate_limit_retry(slot_retry_after, attempt);
+	if (retry_delay < 0)
+		return -1;
+	if (retry_delay > 0) {
+		if (http_code == 429)
+			warning(_("rate limited, waiting %ld seconds before retry"),

http.c static long handle_rate_limit_retry(long slot_retry_after, int attempt) · line 2505

Quoted 10 lines
http.c +10 -0
+				retry_delay);
+		else
+			warning(_("HTTP %ld, waiting %ld seconds before retry"),
+				http_code, retry_delay);
+		trace2_data_intmax("http", the_repository,
+				   "http/retry-sleep-seconds", retry_delay);
+		sleep(retry_delay);
+	}
+	return 0;
+}

This code looks familiar. Beautifully extracted from ...

Quoted 37 lines
http.c +2 -14
 static int http_request_recoverable(const char *url,
 			       void *result, int target,
 			       struct http_get_options *options)
@@ -2483,7 +2511,6 @@ static int http_request_recoverable(const char *url,
 	static struct http_get_options empty_opts;
 	int i = 3;
 	int ret;
-	int rate_limit_retries = http_max_retries;
 	int attempt = 0;
 
 	if (!options)
@@ -2506,9 +2533,8 @@ static int http_request_recoverable(const char *url,
 	}
 
 	while ((ret == HTTP_REAUTH && --i) ||
-	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
-		rate_limit_retries-- > 0)) {
-		long retry_delay = -1;
+	       (ret == HTTP_RATE_LIMITED &&
+		!http_retry_wait(options->http_code, options->retry_after, ++attempt))) {
 		/*
 		 * The previous request may have put cruft into our output stream; we
 		 * should clear it out before making our next request.
@@ -2533,35 +2559,15 @@ static int http_request_recoverable(const char *url,
 		default:
 			BUG("Unknown http_request target");
 		}
-		if (ret == HTTP_RATE_LIMITED) {
-			retry_delay = handle_rate_limit_retry(options->retry_after, ++attempt);
-			if (retry_delay < 0)
-				return HTTP_ERROR;
-
-			if (retry_delay > 0) {
-				if (options->http_code == 429)
-					warning(_("rate limited, waiting %ld seconds before retry"),
-						retry_delay);
-				else

http.c static int http_request_recoverable(const char *url, · line 2554

Quoted 10 lines
http.c +1 -8
-					warning(_("HTTP %ld, waiting %ld seconds before retry"),
-						options->http_code, retry_delay);
-				trace2_data_intmax("http", the_repository,
-						   "http/retry-sleep-seconds", retry_delay);
-				sleep(retry_delay);
-			}
-		} else if (ret == HTTP_REAUTH) {
+		if (ret == HTTP_REAUTH)
 			http_reauth_prepare(1);
-		}

... existing code around here. It always feels good to see that an introduction of a new function is accompanied by replacing many lines of existsing code with a call to the new function.

Thanks.

#12 lore raw

v2 cover http: exponential backoff, configurable status codes, and post_rpc() retries

In reply to Ravi Mistry via GitGitGadget

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) added HTTP 429 retry support using http.maxRetries, http.retryAfter, and http.maxRetryTime.

In practice, when running Git against hosting backends and reverse proxies under high concurrency, four gaps remain:

1. Off-by-one retry count and Retry-After Trace2 timing: Because http_request_recoverable() pre-decrements the retry counter (--rate_limit_retries) in its loop condition, setting http.maxRetries=1 performs 0 retries (and N configured retries performs N-1 retries). In addition, the Retry-After header is extracted after Trace2 logging has already run, so the http/429-retry-after Trace2 event always logs -1.

2. No exponential backoff when Retry-After is absent: When a server does not send a Retry-After header, Git waits the same fixed http.retryAfter delay on every retry. If that delay is short (e.g., 1s), Git burns through all of its retries in just a few seconds before a busy server can recover. If that delay is long, even a brief glitch has to wait a long time on the first retry. Doubling the delay on each attempt (1s, 2s, 4s, ...) keeps the first retry fast while giving the server more time to recover on later attempts, matching curl's own --retry behavior.

3. Transient 5xx errors cannot be retried: Reverse proxies, load balancers, and Git backends frequently surface transient overload or connection resets as 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout. Because 5xx errors are not universally transient across all environments, 429 should remain the default, with an opt-in config to specify additional status codes.

4. HTTP POST requests are not retried: Git's existing 429 retry logic only covers HTTP GET requests (such as the initial /info/refs handshake). However, the actual data transfer during fetch and clone, as well as on-demand blob downloads in partial clones (during checkout, diff, or blame), are sent as HTTP POST requests (/git-upload-pack) via post_rpc(), which only retries authentication and immediately aborts on 429 or 5xx.

This 4-patch series addresses all four gaps:

* Patch 1/4 fixes the off-by-one retry count bug and extracts Retry-After in finish_active_slot() before Trace2 logging runs.
* Patch 2/4 doubles http.retryAfter on each subsequent retry attempt (1s, 2s, 4s, capped at http.maxRetryTime) when Retry-After is absent.
* Patch 3/4 adds http.retryStatusCodes (and GIT_HTTP_RETRY_STATUS_CODES), defaulting to "429", so environments can opt into retrying codes like "429,502,503,504".
* Patch 4/4 extracts a shared http_retry_wait() helper and extends retry support to HTTP POST requests in probe_rpc() and post_rpc() (while fixing a missing Content-Encoding: gzip header on retry).

Prior discussions:

* https://lore.kernel.org/git/[email protected]/
* https://lore.kernel.org/git/[email protected]/

Changes since v1:

* Patch 1/4: Clarified in the commit message why the http_max_retries <= 0 check was moved after the retry loop.
* Patch 2/4: Passed http_max_retries - rate_limit_retries instead of adding a separate attempt counter, and clarified the http.retryAfter and http.maxRetryTime docs.
* Patch 3/4: Restored (Too Many Requests) and clarified http.retryStatusCodes in the config docs, and moved the 400..599 range check to the top of is_retryable_http_code().
* Patch 4/4: Added the attempt counter here when replacing rate_limit_retries with http_retry_wait().

Ravi Mistry (4):
http: fix retry count and Retry-After timing
http: add exponential backoff without Retry-After
http: add http.retryStatusCodes config option
remote-curl: retry transient HTTP errors in post_rpc()

 Documentation/config/http.adoc |  52 ++++++++---
 http.c                         | 125 ++++++++++++++++++--------
 http.h                         |  12 ++-
 remote-curl.c                  |  75 ++++++++--------
 t/lib-httpd/http-429.sh        |  63 ++++++++++++--
 t/t5584-http-429-retry.sh      | 155 +++++++++++++++++++++++++++++++++
 6 files changed, 385 insertions(+), 97 deletions(-)

base-commit: 8d1b215c01512b9658d4525c54ff9b9245f9a7e8
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2287%2Frmistry%2Fhttp-exponential-retries-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2287/rmistry/http-exponential-retries-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/2287

Range-diff vs v1:

1: 80b7f207f4 ! 1: d53778dd46 http: fix retry count and Retry-After timing
@@ Commit message
--rate_limit_retries wraps from 0 to -1 (truthy in C).

Fix this by checking http_max_retries > 0 && rate_limit_retries-- > 0
- in the loop condition and returning HTTP_ERROR when
- http_max_retries <= 0.
+ in the loop condition and checking http_max_retries <= 0 after the
+ loop (which handles both an initial 429 and a 429 after 401 re-auth
+ without logging "retries-exhausted", making the pre-loop check
+ redundant).

2. In http_request(), CURLINFO_RETRY_AFTER is queried after run_one_slot() has already called handle_curl_result(), which logs
2: 36cef34668 ! 2: fe1e847ded http: add exponential backoff without Retry-After
@@ Commit message
http.retryAfter * 2^(attempt - 1)), clamping the delay at http.maxRetryTime. This keeps the first retry fast while giving the server more time to recover on later attempts, matching curl's own
- --retry behavior. When the server provides an explicit Retry-After
- header, continue to honor the server-provided delay (or fail fast if it
- exceeds http.maxRetryTime).
+ --retry behavior (while keeping http.retryAfter's existing default of 0
+ for immediate retries when unconfigured). When the server provides an
+ explicit Retry-After header, continue to honor the server-provided delay
+ (or fail fast if it exceeds http.maxRetryTime).

@@ Documentation/config/http.adoc: http.keepAliveCount::
+ Initial wait time in seconds before retrying when a server returns
HTTP 429 (Too Many Requests) without a Retry-After header.
- Defaults to 0 (retry immediately). When a Retry-After header is
-+ Defaults to 0 (retry immediately). When set to a positive value and
-+ no Retry-After header is present, the delay doubles on each
-+ subsequent retry attempt (http.retryAfter * 2^(attempt - 1)),
++ Defaults to 0, which retries immediately without waiting on every
++ attempt (unless a Retry-After header is present). When set to a
++ positive value and no Retry-After header is present, the first
++ retry waits http.retryAfter seconds and the delay doubles on
++ each subsequent retry attempt (http.retryAfter * 2^(attempt - 1)),
+ capped at http.maxRetryTime. When a Retry-After header is
present, its value takes precedence over this setting; however, automatic use of the server-provided Retry-After header requires libcurl 7.66.0 or later. On older versions, configure this setting
-@@ Documentation/config/http.adoc: http.maxRetryTime::
- handling HTTP 429 (Too Many Requests) responses. If the server
- requests a delay (via Retry-After header) or if http.retryAfter
- is configured with a value that exceeds this maximum, Git will fail
+@@ Documentation/config/http.adoc: http.maxRetries::
+
+ http.maxRetryTime::
+ Maximum time in seconds to wait for a single retry attempt when
+- handling HTTP 429 (Too Many Requests) responses. If the server
+- requests a delay (via Retry-After header) or if http.retryAfter
+- is configured with a value that exceeds this maximum, Git will fail
- immediately rather than waiting. Default is 300 seconds (5 minutes).
-+ immediately rather than waiting. Exponentially backed-off delays
-+ computed from http.retryAfter are capped at this maximum. Default
-+ is 300 seconds (5 minutes).
++ handling HTTP 429 (Too Many Requests) responses.
++ If the server requests a delay (via Retry-After header) or if the
++ configured http.retryAfter value itself exceeds this maximum,
++ Git will fail immediately rather than waiting. Exponentially
++ backed-off delays computed from http.retryAfter, however, are
++ capped at this maximum rather than causing an immediate failure.
++ Default is 300 seconds (5 minutes).
Can be overridden by the GIT_HTTP_MAX_RETRY_TIME environment variable. See also http.retryAfter and http.maxRetries.

@@ http.c: static long handle_rate_limit_retry(long slot_retry_after)
}
}

-@@ http.c: static int http_request_recoverable(const char *url,
- int i = 3;
- int ret;
- int rate_limit_retries = http_max_retries;
-+ int attempt = 0;
-
- if (!options)
- options = &empty_opts;
@@ http.c: static int http_request_recoverable(const char *url,
BUG("Unknown http_request target");
}
if (ret == HTTP_RATE_LIMITED) {
- retry_delay = handle_rate_limit_retry(options->retry_after);
-+ retry_delay = handle_rate_limit_retry(options->retry_after, ++attempt);
++ retry_delay = handle_rate_limit_retry(
++ options->retry_after,
++ http_max_retries - rate_limit_retries);
if (retry_delay < 0)
return HTTP_ERROR;

3: f56ab2afaa ! 3: 6510dddc39 http: add http.retryStatusCodes config option
@@ Documentation/config/http.adoc: http.keepAliveCount::
http.retryAfter::
Initial wait time in seconds before retrying when a server returns
- HTTP 429 (Too Many Requests) without a Retry-After header.
-+ a retryable HTTP status code (HTTP 429 by default) without a
-+ Retry-After header.
- Defaults to 0 (retry immediately). When set to a positive value and
- no Retry-After header is present, the delay doubles on each
- subsequent retry attempt (http.retryAfter * 2^(attempt - 1)),
++ a retryable HTTP status code without a Retry-After header. The
++ set of retryable status codes is configured by
++ http.retryStatusCodes, which includes only HTTP 429 (Too Many
++ Requests) by default.
+ Defaults to 0, which retries immediately without waiting on every
+ attempt (unless a Retry-After header is present). When set to a
+ positive value and no Retry-After header is present, the first
@@ Documentation/config/http.adoc: http.retryAfter::
libcurl 7.66.0 or later. On older versions, configure this setting manually to control the retry delay. Can be overridden by the
@@ Documentation/config/http.adoc: http.retryAfter::
- Maximum number of times to retry after receiving HTTP 429 (Too Many
- Requests) responses. Set to 0 (the default) to disable retries.
+ Maximum number of times to retry after receiving a retryable HTTP
-+ status code (HTTP 429 by default). Set to 0 (the default) to disable
-+ retries.
++ status code. The set of retryable status codes is configured by
++ http.retryStatusCodes, which includes only HTTP 429 (Too Many
++ Requests) by default. Set to 0 (the default) to disable retries.
Can be overridden by the GIT_HTTP_MAX_RETRIES environment variable.
- See also http.retryAfter and http.maxRetryTime.
+ See also http.retryAfter, http.maxRetryTime, and
@@ Documentation/config/http.adoc: http.retryAfter::

http.maxRetryTime::
Maximum time in seconds to wait for a single retry attempt when
-- handling HTTP 429 (Too Many Requests) responses. If the server
-- requests a delay (via Retry-After header) or if http.retryAfter
-+ handling a retryable HTTP status code (HTTP 429 by default). If the
-+ server requests a delay (via Retry-After header) or if http.retryAfter
- is configured with a value that exceeds this maximum, Git will fail
- immediately rather than waiting. Exponentially backed-off delays
- computed from http.retryAfter are capped at this maximum. Default
- is 300 seconds (5 minutes).
+- handling HTTP 429 (Too Many Requests) responses.
++ handling a retryable HTTP status code. The set of retryable
++ status codes is configured by http.retryStatusCodes, which
++ includes only HTTP 429 (Too Many Requests) by default.
+ If the server requests a delay (via Retry-After header) or if the
+ configured http.retryAfter value itself exceeds this maximum,
+ Git will fail immediately rather than waiting. Exponentially
+@@ Documentation/config/http.adoc: http.maxRetryTime::
+ capped at this maximum rather than causing an immediate failure.
+ Default is 300 seconds (5 minutes).
Can be overridden by the GIT_HTTP_MAX_RETRY_TIME environment
- variable. See also http.retryAfter and http.maxRetries.
+ variable. See also http.retryAfter, http.maxRetries, and
@@ Documentation/config/http.adoc: http.retryAfter::
+http.retryStatusCodes::
+ Comma- or whitespace-separated list of HTTP status codes (in the
+ 400..599 range) that should be retried up to http.maxRetries
-+ times. Defaults to 429. For example, setting this to
-+ 429,502,503,504 also retries transient 5xx gateway and
++ times. Defaults to 429 (Too Many Requests). For example, setting
++ this to 429,502,503,504 also retries transient 5xx gateway and
+ service-unavailable errors. Can be overridden by the
+ GIT_HTTP_RETRY_STATUS_CODES environment variable.
+ See also http.maxRetries, http.retryAfter, and
@@ http.c: void normalize_curl_result(CURLcode *result, long http_code,
+ const char *p = http_retry_status_codes ? http_retry_status_codes : "429";
+ char *end;
+
-+ for (; code >= 400 && code < 600 && *p; p = end) {
++ if (code < 400 || code >= 600)
++ return 0;
++
++ for (; *p; p = end) {
+ while (*p == ',' || isspace(*p))
+ p++;
+ if (strtol(p, &end, 10) == code && end > p &&
4: 52819accc8 ! 4: 786964ff00 remote-curl: retry transient HTTP errors in post_rpc()

     @@ http.c: static int http_request_recoverable(const char *url,
       	int i = 3;
       	int ret;
      -	int rate_limit_retries = http_max_retries;
     - 	int attempt = 0;
     ++	int attempt = 0;
       
       	if (!options)
     + 		options = &empty_opts;
      @@ http.c: static int http_request_recoverable(const char *url,
       	}
       
     @@ http.c: static int http_request_recoverable(const char *url,
       			BUG("Unknown http_request target");
       		}
      -		if (ret == HTTP_RATE_LIMITED) {
     --			retry_delay = handle_rate_limit_retry(options->retry_after, ++attempt);
     +-			retry_delay = handle_rate_limit_retry(
     +-				options->retry_after,
     +-				http_max_retries - rate_limit_retries);
      -			if (retry_delay < 0)
      -				return HTTP_ERROR;
      -
Signature · 1 line
gitgitgadget
#13 lore raw

PATCH v2 1/4 http: fix retry count and Retry-After timing

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) introduced two small bugs in HTTP 429 retry handling:

1. In http_request_recoverable(), rate_limit_retries is initialized to http_max_retries (the number of retries to perform after the initial failed request) and decremented with pre-decrement (--rate_limit_retries) in the while condition. Because of the pre-decrement, setting http.maxRetries=1 performs 0 retries, and more generally N configured retries only retries N-1 times. In addition, if the first request returns 401 (HTTP_REAUTH) and the re-authenticated request returns 429 when http.maxRetries is 0, --rate_limit_retries wraps from 0 to -1 (truthy in C).

Fix this by checking http_max_retries > 0 && rate_limit_retries-- > 0 in the loop condition and checking http_max_retries <= 0 after the loop (which handles both an initial 429 and a 429 after 401 re-auth without logging "retries-exhausted", making the pre-loop check redundant).

2. In http_request(), CURLINFO_RETRY_AFTER is queried after run_one_slot() has already called handle_curl_result(), which logs results->retry_after to Trace2 ("http/429-retry-after") while it is still -1. Move the CURLINFO_RETRY_AFTER extraction into finish_active_slot() (initializing slot->results->retry_after = -1 first) so that both handle_curl_result() and callers of run_one_slot() observe the parsed Retry-After value.

---
 http.c                    | 27 +++++++++++++--------------
 t/lib-httpd/http-429.sh   | 31 ++++++++++++++++++++++++++-----
 t/t5584-http-429-retry.sh | 21 +++++++++++++++++++++
 3 files changed, 60 insertions(+), 19 deletions(-)
http.c +13 -14
index d79678d01f..0308f973a6 100644
@@ -345,11 +345,20 @@ static void finish_active_slot(struct active_request_slot *slot)
 	if (slot->results) {
 		slot->results->curl_result = slot->curl_result;
 		slot->results->http_code = slot->http_code;
+		slot->results->retry_after = -1;
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTPAUTH_AVAIL,
 				  &slot->results->auth_avail);
 
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CONNECTCODE,
 			&slot->results->http_connectcode);
+#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
+		{
+			curl_off_t retry_after;
+			if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
+					      &retry_after) == CURLE_OK && retry_after > 0)
+				slot->results->retry_after = (long)retry_after;
+		}
+#endif
 	}
 
 	/* Run callback if appropriate */
@@ -2286,15 +2295,6 @@ static int http_request(const char *url,
 
 	ret = run_one_slot(slot, &results);
 
-#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
-	if (ret == HTTP_RATE_LIMITED) {
-		curl_off_t retry_after;
-		if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
-				      &retry_after) == CURLE_OK && retry_after > 0)
-			results.retry_after = (long)retry_after;
-	}
-#endif
-
 	options->retry_after = results.retry_after;
 
 	if (options->content_type) {
@@ -2461,10 +2461,6 @@ static int http_request_recoverable(const char *url,
 	if (ret != HTTP_OK && ret != HTTP_REAUTH && ret != HTTP_RATE_LIMITED)
 		return ret;
 
-	/* If retries are disabled and we got a 429, fail immediately */
-	if (ret == HTTP_RATE_LIMITED && !http_max_retries)
-		return HTTP_ERROR;
-
 	if (options->effective_url && options->base_url) {
 		if (update_url_from_redirect(options->base_url,
 					     url, options->effective_url)) {
@@ -2474,7 +2470,8 @@ static int http_request_recoverable(const char *url,
 	}
 
 	while ((ret == HTTP_REAUTH && --i) ||
-	       (ret == HTTP_RATE_LIMITED && --rate_limit_retries)) {
+	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
+		rate_limit_retries-- > 0)) {
 		long retry_delay = -1;
 		/*
 		 * The previous request may have put cruft into our output stream; we
@@ -2518,6 +2515,8 @@ static int http_request_recoverable(const char *url,
 		ret = http_request(url, result, target, options);
 	}
 	if (ret == HTTP_RATE_LIMITED) {
+		if (http_max_retries <= 0)
+			return HTTP_ERROR;
 		trace2_data_string("http", the_repository,
 				   "http/429-error", "retries-exhausted");
 		return HTTP_RATE_LIMITED;
t/lib-httpd/http-429.sh +26 -5
index 1a5d7987db..5537fb7ac4 100644
@@ -32,12 +32,33 @@ repo_name="${repo_path%%/*}"
 safe_name=$(echo "${test_context}-${retry_after}-${repo_name}" | tr '/' '_' | tr -cd 'a-zA-Z0-9_-')
 state="http-429-state-${safe_name}"
 
-# Check if this is the first call (no state directory exists), or if
-# the retry-after-value is "permanent", which indicates a 429 must be
-# returned for every request (even if the state directory exists).
-if test "$retry_after" != permanent && ! mkdir "$state" 2>/dev/null
+max_fails=1
+case "$test_context" in
+	*-fail-2*) max_fails=2 ;;
+	*-fail-3*) max_fails=3 ;;
+esac
+
+should_fail=0
+if test "$retry_after" = permanent
+then
+	should_fail=1
+else
+	mkdir -p "$state" 2>/dev/null
+	i=1
+	while test $i -le $max_fails
+	do
+		if mkdir "$state/$i" 2>/dev/null
+		then
+			should_fail=1
+			break
+		fi
+		i=$((i + 1))
+	done
+fi
+
+if test $should_fail -eq 0
 then
-	# Already returned 429 once, forward to git-http-backend
+	# Already returned 429 max_fails times, forward to git-http-backend
 	# Set PATH_INFO to just the repo path (without retry-after value)
 	# Set GIT_PROJECT_ROOT so git-http-backend can find the repository
 	# Use exec to replace this process so git-http-backend gets the updated environment
t/t5584-http-429-retry.sh +21 -0
index a22007b2cf..d92a26893d 100755
@@ -51,6 +51,27 @@ test_expect_success 'HTTP 429 with Retry-After is retried and succeeds' '
 	test_grep "refs/heads/" output
 '
 
+test_expect_success 'http.maxRetries=1 retries once and succeeds' '
+	test_config http.maxRetries 1 &&
+	git ls-remote "$HTTPD_URL/http_429/max-retries-one/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output
+'
+
+test_expect_success 'http.maxRetries=2 fails after 3 failures and succeeds on 2 failures' '
+	test_config http.maxRetries 2 &&
+	test_must_fail git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-3/none/repo.git" 2>err &&
+	test_grep "rate limited" err &&
+	git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-2/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output
+'
+
+test_expect_success 'Trace2 logs actual CURLINFO_RETRY_AFTER value instead of -1' '
+	test_config http.maxRetries 1 &&
+	GIT_TRACE2_EVENT="$(pwd)/trace-retry-after.event" \
+		git ls-remote "$HTTPD_URL/http_429/trace2-retry-after/1/repo.git" >output 2>err &&
+	test_grep "\"key\":\"http/429-retry-after\",\"value\":\"1\"" trace-retry-after.event
+'
+
 test_expect_success 'HTTP 429 without Retry-After uses configured default' '
 	# Enable retries and configure default delay
 	test_config http.maxRetries 3 &&
Signature · 1 line
gitgitgadget
#14 lore raw

PATCH v2 2/4 http: add exponential backoff without Retry-After

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

When a server returns HTTP 429 without a Retry-After header, Git waits the same fixed http.retryAfter delay on every retry. If that delay is short (e.g., 1s), Git burns through all of its retries in just a few seconds before a busy server can recover. If that delay is long, even a brief glitch has to wait a long time on the first retry.

Pass the 1-based retry attempt count to handle_rate_limit_retry() and double http.retryAfter on each subsequent retry (1s, 2s, 4s, ..., via http.retryAfter * 2^(attempt - 1)), clamping the delay at http.maxRetryTime. This keeps the first retry fast while giving the server more time to recover on later attempts, matching curl's own --retry behavior (while keeping http.retryAfter's existing default of 0 for immediate retries when unconfigured). When the server provides an explicit Retry-After header, continue to honor the server-provided delay (or fail fast if it exceeds http.maxRetryTime).

---
 Documentation/config/http.adoc | 20 ++++++++++++++------
 http.c                         | 17 ++++++++++++++---
 t/t5584-http-429-retry.sh      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 9 deletions(-)
Documentation/config/http.adoc +14 -6
index 792a71b413..8e7538ab76 100644
@@ -327,9 +327,14 @@ http.keepAliveCount::
 	`GIT_HTTP_KEEPALIVE_COUNT` environment variable.
 
 http.retryAfter::
-	Default wait time in seconds before retrying when a server returns
+	Initial wait time in seconds before retrying when a server returns
 	HTTP 429 (Too Many Requests) without a Retry-After header.
-	Defaults to 0 (retry immediately). When a Retry-After header is
+	Defaults to 0, which retries immediately without waiting on every
+	attempt (unless a Retry-After header is present). When set to a
+	positive value and no Retry-After header is present, the first
+	retry waits `http.retryAfter` seconds and the delay doubles on
+	each subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
+	capped at `http.maxRetryTime`. When a Retry-After header is
 	present, its value takes precedence over this setting; however,
 	automatic use of the server-provided `Retry-After` header requires
 	libcurl 7.66.0 or later. On older versions, configure this setting
@@ -345,10 +350,13 @@ http.maxRetries::
 
 http.maxRetryTime::
 	Maximum time in seconds to wait for a single retry attempt when
-	handling HTTP 429 (Too Many Requests) responses. If the server
-	requests a delay (via Retry-After header) or if `http.retryAfter`
-	is configured with a value that exceeds this maximum, Git will fail
-	immediately rather than waiting. Default is 300 seconds (5 minutes).
+	handling HTTP 429 (Too Many Requests) responses.
+	If the server requests a delay (via Retry-After header) or if the
+	configured `http.retryAfter` value itself exceeds this maximum,
+	Git will fail immediately rather than waiting. Exponentially
+	backed-off delays computed from `http.retryAfter`, however, are
+	capped at this maximum rather than causing an immediate failure.
+	Default is 300 seconds (5 minutes).
 	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
 	variable. See also `http.retryAfter` and `http.maxRetries`.
 
http.c +14 -3
index 0308f973a6..8a28cd0fb3 100644
@@ -2411,7 +2411,7 @@ static int update_url_from_redirect(struct strbuf *base,
  * Returns a negative value if configuration is invalid (delay exceeds
  * http.maxRetryTime), otherwise returns the delay in seconds (>= 0).
  */
-static long handle_rate_limit_retry(long slot_retry_after)
+static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 {
 	/* Use the slot-specific retry_after value or configured default */
 	if (slot_retry_after >= 0) {
@@ -2427,6 +2427,8 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		return slot_retry_after;
 	} else {
+		long delay = http_retry_after;
+
 		/* No Retry-After header provided, use configured default */
 		if (http_retry_after > http_max_retry_time) {
 			error(_("configured http.retryAfter exceeds http.maxRetryTime (%ld > %ld seconds)"),
@@ -2437,7 +2439,14 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		trace2_data_string("http", the_repository,
 			"http/429-retry-source", "config-default");
-		return http_retry_after;
+		for (int a = 1; a < attempt && delay > 0; a++) {
+			if (delay >= http_max_retry_time - delay) {
+				delay = http_max_retry_time;
+				break;
+			}
+			delay *= 2;
+		}
+		return delay;
 	}
 }
 
@@ -2498,7 +2507,9 @@ static int http_request_recoverable(const char *url,
 			BUG("Unknown http_request target");
 		}
 		if (ret == HTTP_RATE_LIMITED) {
-			retry_delay = handle_rate_limit_retry(options->retry_after);
+			retry_delay = handle_rate_limit_retry(
+				options->retry_after,
+				http_max_retries - rate_limit_retries);
 			if (retry_delay < 0)
 				return HTTP_ERROR;
 
t/t5584-http-429-retry.sh +33 -0
index d92a26893d..f0c3cc3b61 100755
@@ -82,6 +82,39 @@ test_expect_success 'HTTP 429 without Retry-After uses configured default' '
 	test_grep "refs/heads/" output
 '
 
+test_expect_success 'exponential backoff doubles http.retryAfter without Retry-After header' '
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
+	git ls-remote "$HTTPD_URL/http_429/exp-backoff-fail-3/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep "waiting 2 seconds before retry" err &&
+	test_grep "waiting 4 seconds before retry" err
+'
+
+test_expect_success 'exponential backoff clamps delay at http.maxRetryTime' '
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
+	test_config http.maxRetryTime 2 &&
+	GIT_TRACE2_EVENT="$(pwd)/trace-clamp.event" \
+		git ls-remote "$HTTPD_URL/http_429/exp-clamp-fail-3/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep "waiting 2 seconds before retry" err &&
+	test_grep ! "waiting 4 seconds before retry" err &&
+	grep "\"key\":\"http/retry-sleep-seconds\",\"value\":\"2\"" trace-clamp.event >clamped &&
+	test_line_count = 2 clamped
+'
+
+test_expect_success 'server Retry-After header overrides exponential backoff' '
+	test_config http.maxRetries 2 &&
+	test_config http.retryAfter 1 &&
+	git ls-remote "$HTTPD_URL/http_429/header-override-fail-2/1/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep ! "waiting 2 seconds before retry" err
+'
+
 test_expect_success 'HTTP 429 retry delays are respected' '
 	# Enable retries
 	test_config http.maxRetries 3 &&
Signature · 1 line
gitgitgadget
#15 lore raw

Re: [PATCH v2 2/4] http: add exponential backoff without Retry-After

Quoted 8 lines · Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

When a server returns HTTP 429 without a Retry-After header, Git waits the same fixed http.retryAfter delay on every retry. If that delay is short (e.g., 1s), Git burns through all of its retries in just a few seconds before a busy server can recover. If that delay is long, even a brief glitch has to wait a long time on the first retry.

Quoted 9 lines

Pass the 1-based retry attempt count to handle_rate_limit_retry() and double http.retryAfter on each subsequent retry (1s, 2s, 4s, ..., via http.retryAfter * 2^(attempt - 1)), clamping the delay at http.maxRetryTime. This keeps the first retry fast while giving the server more time to recover on later attempts, matching curl's own --retry behavior (while keeping http.retryAfter's existing default of 0 for immediate retries when unconfigured). When the server provides an explicit Retry-After header, continue to honor the server-provided delay (or fail fast if it exceeds http.maxRetryTime).

We should not immediately retry on a 429 response, even if that's the behaviour we have now. The server told us we were being too aggressive, so the exact thing we should not do is to continue making requests. As a server operator, I definitely don't want this behaviour; instead, we should pick something sensible if no Retry-After is given, such as 30 seconds.

Signature · 3 lines
brian m. carlson (they/them)
Toronto, Ontario, CA
Speaking for myself only
#16 lore raw

Re: [PATCH v2 2/4] http: add exponential backoff without Retry-After

Quoted 6 lines · brian m. carlson

We should not immediately retry on a 429 response, even if that's the behaviour we have now. The server told us we were being too aggressive, so the exact thing we should not do is to continue making requests. As a server operator, I definitely don't want this behaviour; instead, we should pick something sensible if no Retry-After is given, such as 30 seconds.

Makes sense. Junio noted on v1 as well that defaulting http.retryAfter to 0 does not mesh well with exponential backoff.

What do you think about defaulting http.retryAfter to 1 second, matching curl's --retry behavior (1s, 2s, 4s, ...)? Because the delay doubles on each attempt (and Patch 3/4 also reuses http.retryAfter for opt-in transient 5xx retries), starting at 30s would turn 3 retries into 30s, 60s, and 120s. Starting at 1s avoids hammering the server immediately while ramping up on repeated failures.

If 1s sounds reasonable, I can send a v3 with that change.

Thanks!

#17 lore raw

Re: [PATCH v2 2/4] http: add exponential backoff without Retry-After

Quoted 12 lines · Ravi Mistry
Quoted 6 lines · brian m. carlson

We should not immediately retry on a 429 response, even if that's the behaviour we have now. The server told us we were being too aggressive, so the exact thing we should not do is to continue making requests. As a server operator, I definitely don't want this behaviour; instead, we should pick something sensible if no Retry-After is given, such as 30 seconds.

Makes sense. Junio noted on v1 as well that defaulting http.retryAfter to 0 does not mesh well with exponential backoff.

Quoted 8 lines

What do you think about defaulting http.retryAfter to 1 second, matching curl's --retry behavior (1s, 2s, 4s, ...)? Because the delay doubles on each attempt (and Patch 3/4 also reuses http.retryAfter for opt-in transient 5xx retries), starting at 30s would turn 3 retries into 30s, 60s, and 120s. Starting at 1s avoids hammering the server immediately while ramping up on repeated failures.

If 1s sounds reasonable, I can send a v3 with that change.

I am not brian and I cannot tell why he suggested "such as 30 seconds", but matching what somebody else like curl does sounds sensible, if you ask me.

Thanks.

#18 lore raw

Re: [PATCH v2 2/4] http: add exponential backoff without Retry-After

In reply to Ravi Mistry
Quoted 12 lines · Ravi Mistry
Quoted 6 lines · brian m. carlson

We should not immediately retry on a 429 response, even if that's the behaviour we have now. The server told us we were being too aggressive, so the exact thing we should not do is to continue making requests. As a server operator, I definitely don't want this behaviour; instead, we should pick something sensible if no Retry-After is given, such as 30 seconds.

Makes sense. Junio noted on v1 as well that defaulting http.retryAfter to 0 does not mesh well with exponential backoff.

Quoted 6 lines

What do you think about defaulting http.retryAfter to 1 second, matching curl's --retry behavior (1s, 2s, 4s, ...)? Because the delay doubles on each attempt (and Patch 3/4 also reuses http.retryAfter for opt-in transient 5xx retries), starting at 30s would turn 3 retries into 30s, 60s, and 120s. Starting at 1s avoids hammering the server immediately while ramping up on repeated failures.

I can't imagine a situation where 1s is a reasonable backoff for a 429 response since I can't imagine how the load could reasonably decrease in that amount of time. I suggested 30 seconds because that's a reasonable amount of time to back off and allow load to go down (and I think that 30, 60, and 120 seconds is a reasonable set of timeframes), but I'd say it should be at the very least 10 seconds. I've seen cases where users retry very aggressively and it increases the load on the server so they just get rate-limited way, way more than if they'd been more gentle and waited a little bit longer.

Note that the curl binary often makes very cheap requests, such as static pages or small files, but we are making what are almost certainly very expensive requests requiring a substantial amount of server computation, so the curl defaults are almost certainly not right here.

Signature · 3 lines
brian m. carlson (they/them)
Toronto, Ontario, CA
Speaking for myself only
#19 lore raw

Re: [PATCH v2 2/4] http: add exponential backoff without Retry-After

Quoted 11 lines · brian m. carlson

I can't imagine a situation where 1s is a reasonable backoff for a 429 response since I can't imagine how the load could reasonably decrease in that amount of time. I suggested 30 seconds because that's a reasonable amount of time to back off and allow load to go down (and I think that 30, 60, and 120 seconds is a reasonable set of timeframes), but I'd say it should be at the very least 10 seconds.
...
Note that the curl binary often makes very cheap requests, such as static pages or small files, but we are making what are almost certainly very expensive requests requiring a substantial amount of server computation, so the curl defaults are almost certainly not right here.

ACK. In that case, lets go with 10 seconds (10s, 20s, 40s, ...). Waiting 30s, 60s, and 120s feels too long for interactive commands, whereas starting at 10s still gives the server breathing room.

I will send a v3 with http.retryAfter defaulting to 10.

Thanks!

#20 lore raw

PATCH v2 3/4 http: add http.retryStatusCodes config option

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) hardcoded HTTP 429 as the only status code eligible for retry. In practice, reverse proxies, load balancers, and Git hosting backends also return transient 5xx responses such as 502 Bad Gateway, 503 Service Unavailable (which RFC 9110 section 10.2.3 defines to support Retry-After), and 504 Gateway Timeout during load shedding or upstream connection resets.

Because 5xx responses are not universally transient across all environments, keep 429 as the default and introduce a new http.retryStatusCodes configuration option (and corresponding GIT_HTTP_RETRY_STATUS_CODES environment variable) accepting a comma- or whitespace-separated list of HTTP status codes in the 400..599 range (for example, "429,502,503,504").

Keep the existing "rate limited" warning and error messages for HTTP 429, and report the numeric status code (e.g., "HTTP 502, waiting <N> seconds before retry") when retrying other status codes.

---
 Documentation/config/http.adoc | 34 +++++++++++++++++++-----
 http.c                         | 42 ++++++++++++++++++++++++++---
 http.h                         |  6 +++--
 t/lib-httpd/http-429.sh        | 10 +++++--
 t/t5584-http-429-retry.sh      | 48 ++++++++++++++++++++++++++++++++++
 5 files changed, 125 insertions(+), 15 deletions(-)
Documentation/config/http.adoc +27 -7
index 8e7538ab76..e7fa251b65 100644
@@ -328,7 +328,10 @@ http.keepAliveCount::
 
 http.retryAfter::
 	Initial wait time in seconds before retrying when a server returns
-	HTTP 429 (Too Many Requests) without a Retry-After header.
+	a retryable HTTP status code without a Retry-After header. The
+	set of retryable status codes is configured by
+	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
+	Requests) by default.
 	Defaults to 0, which retries immediately without waiting on every
 	attempt (unless a Retry-After header is present). When set to a
 	positive value and no Retry-After header is present, the first
@@ -340,17 +343,23 @@ http.retryAfter::
 	libcurl 7.66.0 or later. On older versions, configure this setting
 	manually to control the retry delay. Can be overridden by the
 	`GIT_HTTP_RETRY_AFTER` environment variable.
-	See also `http.maxRetries` and `http.maxRetryTime`.
+	See also `http.maxRetries`, `http.maxRetryTime`, and
+	`http.retryStatusCodes`.
 
 http.maxRetries::
-	Maximum number of times to retry after receiving HTTP 429 (Too Many
-	Requests) responses. Set to 0 (the default) to disable retries.
+	Maximum number of times to retry after receiving a retryable HTTP
+	status code. The set of retryable status codes is configured by
+	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
+	Requests) by default. Set to 0 (the default) to disable retries.
 	Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
-	See also `http.retryAfter` and `http.maxRetryTime`.
+	See also `http.retryAfter`, `http.maxRetryTime`, and
+	`http.retryStatusCodes`.
 
 http.maxRetryTime::
 	Maximum time in seconds to wait for a single retry attempt when
-	handling HTTP 429 (Too Many Requests) responses.
+	handling a retryable HTTP status code. The set of retryable
+	status codes is configured by `http.retryStatusCodes`, which
+	includes only HTTP 429 (Too Many Requests) by default.
 	If the server requests a delay (via Retry-After header) or if the
 	configured `http.retryAfter` value itself exceeds this maximum,
 	Git will fail immediately rather than waiting. Exponentially
@@ -358,7 +367,18 @@ http.maxRetryTime::
 	capped at this maximum rather than causing an immediate failure.
 	Default is 300 seconds (5 minutes).
 	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
-	variable. See also `http.retryAfter` and `http.maxRetries`.
+	variable. See also `http.retryAfter`, `http.maxRetries`, and
+	`http.retryStatusCodes`.
+
+http.retryStatusCodes::
+	Comma- or whitespace-separated list of HTTP status codes (in the
+	400..599 range) that should be retried up to `http.maxRetries`
+	times. Defaults to `429` (Too Many Requests). For example, setting
+	this to `429,502,503,504` also retries transient `5xx` gateway and
+	service-unavailable errors. Can be overridden by the
+	`GIT_HTTP_RETRY_STATUS_CODES` environment variable.
+	See also `http.maxRetries`, `http.retryAfter`, and
+	`http.maxRetryTime`.
 
 http.noEPSV::
 	A boolean which disables using of EPSV ftp command by curl.
http.c +38 -4

102 lines not on this page · raw message

http.h +4 -2
index 729c51904d..5985a0f71f 100644
@@ -167,10 +167,12 @@ struct http_get_options {
 
 	/*
 	 * After a request completes, contains the Retry-After delay in seconds
-	 * if the server returned HTTP 429 with a Retry-After header (requires
-	 * libcurl 7.66.0 or later), or -1 if no such header was present.
+	 * if the server returned a retryable status code with a Retry-After
+	 * header (requires libcurl 7.66.0 or later), or -1 if no such header
+	 * was present.
 	 */
 	long retry_after;
+	long http_code;
 };
 
 /* Return values for http_get_*() */
t/lib-httpd/http-429.sh +8 -2
index 5537fb7ac4..5c7d58a75f 100644
@@ -75,8 +75,14 @@ then
 	exec "$GIT_EXEC_PATH/git-http-backend"
 fi
 
-# Output HTTP 429 response
-printf "Status: 429 Too Many Requests\r\n"
+# Output HTTP 429 (or configured status) response
+status_line="429 Too Many Requests"
+case "$test_context" in
+	*-status-502*) status_line="502 Bad Gateway" ;;
+	*-status-503*) status_line="503 Service Unavailable" ;;
+	*-status-504*) status_line="504 Gateway Timeout" ;;
+esac
+printf "Status: %s\r\n" "$status_line"
 
 # Set Retry-After header based on retry_after value
 case "$retry_after" in
t/t5584-http-429-retry.sh +48 -0

52 lines not on this page · raw message

Signature · 1 line
gitgitgadget
#21 lore raw

Re: [PATCH v2 3/4] http: add http.retryStatusCodes config option

Documentation/config/http.adoc http.keepAliveCount:: · line 334

Quoted 7 lines · Ravi Mistry via GitGitGadget

http.retryAfter::

Documentation/config/http.adoc +4 -1
 	Initial wait time in seconds before retrying when a server returns
-	HTTP 429 (Too Many Requests) without a Retry-After header.
+	a retryable HTTP status code without a Retry-After header. The
+	set of retryable status codes is configured by
+	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
+	Requests) by default.

I as a lazy person would probably have written

... when a server returns a retryable HTTP status code (see http.retryStatusCodes) without a Retry-After header.

and omitted everything else, but this is better by being more self contained ;-).

Looking good.

By the way, after the dust settles, we should fix the formatting of these variable name entries wholesale. Looking at a few files in the same directory, like log.adoc and grep.adoc:

    $ git grep -h '^[^ ]*::$' Documentation/config/{log,grep}.adoc
    `grep.lineNumber`::
    `grep.column`::
    `grep.patternType`::
    `grep.extendedRegexp`::
    `grep.threads`::
    `grep.fullName`::
    `grep.fallbackToNoIndex`::
    `log.abbrevCommit`::
    `log.date`::
    `log.decorate`::
    `log.initialDecorationSet`::
    `log.excludeDecoration`::
    `log.diffMerges`::
    `log.follow`::
    `log.graphColors`::
    `log.graphIndent`::
    `log.showRoot`::
    `log.showSignature`::
    `log.mailmap`::

these names are supposed to be marked up in certain ways, but none in this file is.

Anyway, the whole series was a pleasant read. Thanks.

#22 lore raw

PATCH v2 4/4 remote-curl: retry transient HTTP errors in post_rpc()

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Git's existing retry logic only covers HTTP GET requests. However, the actual data transfer during fetch and clone, as well as on-demand blob downloads in partial clones, are sent as HTTP POST requests via post_rpc(), which only retries authentication and immediately aborts on 429 or 5xx.

Extract the retry wait logic from http_request_recoverable() into a shared http_retry_wait() helper and use it to retry transient HTTP errors in probe_rpc() and post_rpc(). Also defer reporting RPC errors until retries are exhausted, and re-attach the "Content-Encoding: gzip" header when retrying a compressed request.

---
 http.c                    | 67 ++++++++++++++++++----------------
 http.h                    |  6 ++++
 remote-curl.c             | 75 +++++++++++++++++++--------------------
 t/lib-httpd/http-429.sh   | 50 ++++++++++++++++++--------
 t/t5584-http-429-retry.sh | 53 +++++++++++++++++++++++++++
 5 files changed, 168 insertions(+), 83 deletions(-)
http.c +36 -31
index fc9384353c..ea4f9245c4 100644
@@ -2479,6 +2479,34 @@ static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 	}
 }
 
+int http_retry_wait(long http_code, long slot_retry_after, int attempt)
+{
+	long retry_delay;
+
+	if (http_max_retries <= 0)
+		return -1;
+	if (attempt > http_max_retries) {
+		trace2_data_string("http", the_repository,
+				   "http/429-error", "retries-exhausted");
+		return -1;
+	}
+	retry_delay = handle_rate_limit_retry(slot_retry_after, attempt);
+	if (retry_delay < 0)
+		return -1;
+	if (retry_delay > 0) {
+		if (http_code == 429)
+			warning(_("rate limited, waiting %ld seconds before retry"),
+				retry_delay);
+		else
+			warning(_("HTTP %ld, waiting %ld seconds before retry"),
+				http_code, retry_delay);
+		trace2_data_intmax("http", the_repository,
+				   "http/retry-sleep-seconds", retry_delay);
+		sleep(retry_delay);
+	}
+	return 0;
+}
+
 static int http_request_recoverable(const char *url,
 			       void *result, int target,
 			       struct http_get_options *options)
@@ -2486,7 +2514,7 @@ static int http_request_recoverable(const char *url,
 	static struct http_get_options empty_opts;
 	int i = 3;
 	int ret;
-	int rate_limit_retries = http_max_retries;
+	int attempt = 0;
 
 	if (!options)
 		options = &empty_opts;
@@ -2508,9 +2536,8 @@ static int http_request_recoverable(const char *url,
 	}
 
 	while ((ret == HTTP_REAUTH && --i) ||
-	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
-		rate_limit_retries-- > 0)) {
-		long retry_delay = -1;
+	       (ret == HTTP_RATE_LIMITED &&
+		!http_retry_wait(options->http_code, options->retry_after, ++attempt))) {
 		/*
 		 * The previous request may have put cruft into our output stream; we
 		 * should clear it out before making our next request.
@@ -2535,37 +2562,15 @@ static int http_request_recoverable(const char *url,
 		default:
 			BUG("Unknown http_request target");
 		}
-		if (ret == HTTP_RATE_LIMITED) {
-			retry_delay = handle_rate_limit_retry(
-				options->retry_after,
-				http_max_retries - rate_limit_retries);
-			if (retry_delay < 0)
-				return HTTP_ERROR;
-
-			if (retry_delay > 0) {
-				if (options->http_code == 429)
-					warning(_("rate limited, waiting %ld seconds before retry"),
-						retry_delay);
-				else
-					warning(_("HTTP %ld, waiting %ld seconds before retry"),
-						options->http_code, retry_delay);
-				trace2_data_intmax("http", the_repository,
-						   "http/retry-sleep-seconds", retry_delay);
-				sleep(retry_delay);
-			}
-		} else if (ret == HTTP_REAUTH) {
+		if (ret == HTTP_REAUTH)
 			http_reauth_prepare(1);
-		}
 
 		ret = http_request(url, result, target, options);
 	}
-	if (ret == HTTP_RATE_LIMITED) {
-		if (http_max_retries <= 0)
-			return HTTP_ERROR;
-		trace2_data_string("http", the_repository,
-				   "http/429-error", "retries-exhausted");
-		return options->http_code == 429 ? HTTP_RATE_LIMITED : HTTP_ERROR;
-	}
+	if (ret == HTTP_RATE_LIMITED &&
+	    (http_max_retries <= 0 || attempt <= http_max_retries ||
+	     options->http_code != 429))
+		return HTTP_ERROR;
 	return ret;
 }
 
http.h +6 -0
index 5985a0f71f..fa9f46a11b 100644
@@ -201,6 +201,12 @@ int http_get_strbuf(const char *url, struct strbuf *result, struct http_get_opti
 int http_get_file(const char *url, const char *filename,
 		  struct http_get_options *options);
 
+/*
+ * Wait before retrying a transient HTTP error (1-based attempt count).
+ * Returns 0 if the caller should retry, or -1 to stop retrying.
+ */
+int http_retry_wait(long http_code, long slot_retry_after, int attempt);
+
 int http_fetch_ref(const char *base, struct ref *ref);
 
 struct curl_slist *http_append_auth_header(const struct credential *c,
remote-curl.c +37 -38

130 lines not on this page · raw message

t/lib-httpd/http-429.sh +36 -14

63 lines not on this page · raw message

t/t5584-http-429-retry.sh +53 -0

57 lines not on this page · raw message

Signature · 1 line
gitgitgadget
#23 lore raw

v3 cover http: exponential backoff, configurable status codes, and post_rpc() retries

In reply to Ravi Mistry via GitGitGadget

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) added HTTP 429 retry support using http.maxRetries, http.retryAfter, and http.maxRetryTime.

In practice, when running Git against hosting backends and reverse proxies under high concurrency, four gaps remain:

1. Off-by-one retry count and Retry-After Trace2 timing: Because http_request_recoverable() pre-decrements the retry counter (--rate_limit_retries) in its loop condition, setting http.maxRetries=1 performs 0 retries (and N configured retries performs N-1 retries). In addition, the Retry-After header is extracted after Trace2 logging has already run, so the http/429-retry-after Trace2 event always logs -1.

2. No exponential backoff (and 0s default) when Retry-After is absent: When a server does not send a Retry-After header, Git defaults http.retryAfter to 0 (retrying immediately in a tight loop) and waits the same fixed delay on every retry when configured. Doubling the delay on each attempt and defaulting http.retryAfter to 10 seconds (10s, 20s, 40s, ...) gives overloaded servers breathing room while keeping interactive commands reasonably responsive.

3. Transient 5xx errors cannot be retried: Reverse proxies, load balancers, and Git backends frequently surface transient overload or connection resets as 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout. Because 5xx errors are not universally transient across all environments, 429 should remain the default, with an opt-in config to specify additional status codes.

4. HTTP POST requests are not retried: Git's existing 429 retry logic only covers HTTP GET requests (such as the initial /info/refs handshake). However, the actual data transfer during fetch and clone, as well as on-demand blob downloads in partial clones (during checkout, diff, or blame), are sent as HTTP POST requests (/git-upload-pack) via post_rpc(), which only retries authentication and immediately aborts on 429 or 5xx.

This 5-patch series addresses all four gaps:

* Patch 1/5 fixes the off-by-one retry count bug and extracts Retry-After in finish_active_slot() before Trace2 logging runs.
* Patch 2/5 doubles http.retryAfter on each subsequent retry attempt (capped at http.maxRetryTime) when Retry-After is absent.
* Patch 3/5 adds http.retryStatusCodes (and GIT_HTTP_RETRY_STATUS_CODES), defaulting to "429", so environments can opt into retrying codes like "429,502,503,504".
* Patch 4/5 extracts a shared http_retry_wait() helper and extends retry support to HTTP POST requests in probe_rpc() and post_rpc() (while fixing a missing Content-Encoding: gzip header on retry).
* Patch 5/5 changes the default of http.retryAfter from 0 to 10 seconds so Git does not immediately hammer a rate-limiting server when no Retry-After header is provided.

Prior discussions:

* https://lore.kernel.org/git/[email protected]/
* https://lore.kernel.org/git/[email protected]/

Changes since v2:

* Patches 1/5..4/5: Unchanged from v2.
* Added Patch 5/5 to default http.retryAfter to 10 seconds instead of 0 when Retry-After is absent (per feedback from Junio C Hamano and brian m. carlson).

Changes since v1:

* Patch 1/4: Clarified in the commit message why the http_max_retries <= 0 check was moved after the retry loop.
* Patch 2/4: Passed http_max_retries - rate_limit_retries instead of adding a separate attempt counter, and clarified the http.retryAfter and http.maxRetryTime docs.
* Patch 3/4: Restored (Too Many Requests) and clarified http.retryStatusCodes in the config docs, and moved the 400..599 range check to the top of is_retryable_http_code().
* Patch 4/4: Added the attempt counter here when replacing rate_limit_retries with http_retry_wait().

Ravi Mistry (5):
http: fix retry count and Retry-After timing
http: add exponential backoff without Retry-After
http: add http.retryStatusCodes config option
remote-curl: retry transient HTTP errors in post_rpc()
http: default http.retryAfter to 10 seconds

 Documentation/config/http.adoc |  61 ++++++++---
 http.c                         | 129 +++++++++++++++--------
 http.h                         |  12 ++-
 remote-curl.c                  |  75 +++++++-------
 t/lib-httpd/http-429.sh        |  63 ++++++++++--
 t/t5584-http-429-retry.sh      | 181 +++++++++++++++++++++++++++++++--
 6 files changed, 412 insertions(+), 109 deletions(-)

base-commit: 8d1b215c01512b9658d4525c54ff9b9245f9a7e8
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2287%2Frmistry%2Fhttp-exponential-retries-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2287/rmistry/http-exponential-retries-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/2287

Range-diff vs v2:

 1:  d53778dd46 = 1:  d53778dd46 http: fix retry count and Retry-After timing
 2:  fe1e847ded = 2:  fe1e847ded http: add exponential backoff without Retry-After
 3:  6510dddc39 = 3:  6510dddc39 http: add http.retryStatusCodes config option
 4:  786964ff00 = 4:  786964ff00 remote-curl: retry transient HTTP errors in post_rpc()
 -:  ---------- > 5:  61e3ffe9ab http: default http.retryAfter to 10 seconds
Signature · 1 line
gitgitgadget
#24 lore raw

PATCH v3 1/5 http: fix retry count and Retry-After timing

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) introduced two small bugs in HTTP 429 retry handling:

1. In http_request_recoverable(), rate_limit_retries is initialized to http_max_retries (the number of retries to perform after the initial failed request) and decremented with pre-decrement (--rate_limit_retries) in the while condition. Because of the pre-decrement, setting http.maxRetries=1 performs 0 retries, and more generally N configured retries only retries N-1 times. In addition, if the first request returns 401 (HTTP_REAUTH) and the re-authenticated request returns 429 when http.maxRetries is 0, --rate_limit_retries wraps from 0 to -1 (truthy in C).

Fix this by checking http_max_retries > 0 && rate_limit_retries-- > 0 in the loop condition and checking http_max_retries <= 0 after the loop (which handles both an initial 429 and a 429 after 401 re-auth without logging "retries-exhausted", making the pre-loop check redundant).

2. In http_request(), CURLINFO_RETRY_AFTER is queried after run_one_slot() has already called handle_curl_result(), which logs results->retry_after to Trace2 ("http/429-retry-after") while it is still -1. Move the CURLINFO_RETRY_AFTER extraction into finish_active_slot() (initializing slot->results->retry_after = -1 first) so that both handle_curl_result() and callers of run_one_slot() observe the parsed Retry-After value.

---
 http.c                    | 27 +++++++++++++--------------
 t/lib-httpd/http-429.sh   | 31 ++++++++++++++++++++++++++-----
 t/t5584-http-429-retry.sh | 21 +++++++++++++++++++++
 3 files changed, 60 insertions(+), 19 deletions(-)
http.c +13 -14
index d79678d01f..0308f973a6 100644
@@ -345,11 +345,20 @@ static void finish_active_slot(struct active_request_slot *slot)
 	if (slot->results) {
 		slot->results->curl_result = slot->curl_result;
 		slot->results->http_code = slot->http_code;
+		slot->results->retry_after = -1;
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTPAUTH_AVAIL,
 				  &slot->results->auth_avail);
 
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CONNECTCODE,
 			&slot->results->http_connectcode);
+#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
+		{
+			curl_off_t retry_after;
+			if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
+					      &retry_after) == CURLE_OK && retry_after > 0)
+				slot->results->retry_after = (long)retry_after;
+		}
+#endif
 	}
 
 	/* Run callback if appropriate */
@@ -2286,15 +2295,6 @@ static int http_request(const char *url,
 
 	ret = run_one_slot(slot, &results);
 
-#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
-	if (ret == HTTP_RATE_LIMITED) {
-		curl_off_t retry_after;
-		if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
-				      &retry_after) == CURLE_OK && retry_after > 0)
-			results.retry_after = (long)retry_after;
-	}
-#endif
-
 	options->retry_after = results.retry_after;
 
 	if (options->content_type) {
@@ -2461,10 +2461,6 @@ static int http_request_recoverable(const char *url,
 	if (ret != HTTP_OK && ret != HTTP_REAUTH && ret != HTTP_RATE_LIMITED)
 		return ret;
 
-	/* If retries are disabled and we got a 429, fail immediately */
-	if (ret == HTTP_RATE_LIMITED && !http_max_retries)
-		return HTTP_ERROR;
-
 	if (options->effective_url && options->base_url) {
 		if (update_url_from_redirect(options->base_url,
 					     url, options->effective_url)) {
@@ -2474,7 +2470,8 @@ static int http_request_recoverable(const char *url,
 	}
 
 	while ((ret == HTTP_REAUTH && --i) ||
-	       (ret == HTTP_RATE_LIMITED && --rate_limit_retries)) {
+	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
+		rate_limit_retries-- > 0)) {
 		long retry_delay = -1;
 		/*
 		 * The previous request may have put cruft into our output stream; we
@@ -2518,6 +2515,8 @@ static int http_request_recoverable(const char *url,
 		ret = http_request(url, result, target, options);
 	}
 	if (ret == HTTP_RATE_LIMITED) {
+		if (http_max_retries <= 0)
+			return HTTP_ERROR;
 		trace2_data_string("http", the_repository,
 				   "http/429-error", "retries-exhausted");
 		return HTTP_RATE_LIMITED;
t/lib-httpd/http-429.sh +26 -5
index 1a5d7987db..5537fb7ac4 100644
@@ -32,12 +32,33 @@ repo_name="${repo_path%%/*}"
 safe_name=$(echo "${test_context}-${retry_after}-${repo_name}" | tr '/' '_' | tr -cd 'a-zA-Z0-9_-')
 state="http-429-state-${safe_name}"
 
-# Check if this is the first call (no state directory exists), or if
-# the retry-after-value is "permanent", which indicates a 429 must be
-# returned for every request (even if the state directory exists).
-if test "$retry_after" != permanent && ! mkdir "$state" 2>/dev/null
+max_fails=1
+case "$test_context" in
+	*-fail-2*) max_fails=2 ;;
+	*-fail-3*) max_fails=3 ;;
+esac
+
+should_fail=0
+if test "$retry_after" = permanent
+then
+	should_fail=1
+else
+	mkdir -p "$state" 2>/dev/null
+	i=1
+	while test $i -le $max_fails
+	do
+		if mkdir "$state/$i" 2>/dev/null
+		then
+			should_fail=1
+			break
+		fi
+		i=$((i + 1))
+	done
+fi
+
+if test $should_fail -eq 0
 then
-	# Already returned 429 once, forward to git-http-backend
+	# Already returned 429 max_fails times, forward to git-http-backend
 	# Set PATH_INFO to just the repo path (without retry-after value)
 	# Set GIT_PROJECT_ROOT so git-http-backend can find the repository
 	# Use exec to replace this process so git-http-backend gets the updated environment
t/t5584-http-429-retry.sh +21 -0
index a22007b2cf..d92a26893d 100755
@@ -51,6 +51,27 @@ test_expect_success 'HTTP 429 with Retry-After is retried and succeeds' '
 	test_grep "refs/heads/" output
 '
 
+test_expect_success 'http.maxRetries=1 retries once and succeeds' '
+	test_config http.maxRetries 1 &&
+	git ls-remote "$HTTPD_URL/http_429/max-retries-one/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output
+'
+
+test_expect_success 'http.maxRetries=2 fails after 3 failures and succeeds on 2 failures' '
+	test_config http.maxRetries 2 &&
+	test_must_fail git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-3/none/repo.git" 2>err &&
+	test_grep "rate limited" err &&
+	git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-2/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output
+'
+
+test_expect_success 'Trace2 logs actual CURLINFO_RETRY_AFTER value instead of -1' '
+	test_config http.maxRetries 1 &&
+	GIT_TRACE2_EVENT="$(pwd)/trace-retry-after.event" \
+		git ls-remote "$HTTPD_URL/http_429/trace2-retry-after/1/repo.git" >output 2>err &&
+	test_grep "\"key\":\"http/429-retry-after\",\"value\":\"1\"" trace-retry-after.event
+'
+
 test_expect_success 'HTTP 429 without Retry-After uses configured default' '
 	# Enable retries and configure default delay
 	test_config http.maxRetries 3 &&
Signature · 1 line
gitgitgadget
#25 lore raw

Re: [PATCH v3 1/5] http: fix retry count and Retry-After timing

Quoted 21 lines · Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) introduced two small bugs in HTTP 429 retry handling:

1. In http_request_recoverable(), rate_limit_retries is initialized to http_max_retries (the number of retries to perform after the initial failed request) and decremented with pre-decrement (--rate_limit_retries) in the while condition. Because of the pre-decrement, setting http.maxRetries=1 performs 0 retries, and more generally N configured retries only retries N-1 times. In addition, if the first request returns 401 (HTTP_REAUTH) and the re-authenticated request returns 429 when http.maxRetries is 0, --rate_limit_retries wraps from 0 to -1 (truthy in C).

Fix this by checking http_max_retries > 0 && rate_limit_retries-- > 0 in the loop condition and checking http_max_retries <= 0 after the loop (which handles both an initial 429 and a 429 after 401 re-auth without logging "retries-exhausted", making the pre-loop check redundant).

Quoted 7 lines

2. In http_request(), CURLINFO_RETRY_AFTER is queried after run_one_slot() has already called handle_curl_result(), which logs results->retry_after to Trace2 ("http/429-retry-after") while it is still -1. Move the CURLINFO_RETRY_AFTER extraction into finish_active_slot() (initializing slot->results->retry_after = -1 first) so that both handle_curl_result() and callers of run_one_slot() observe the parsed Retry-After value.

Nit: feels like two separate bugs that should have been fixed in two separate commits.

Quoted 11 lines
http.c +1 -0
index d79678d01f..0308f973a6 100644
@@ -345,11 +345,20 @@ static void finish_active_slot(struct active_request_slot *slot)
 	if (slot->results) {
 		slot->results->curl_result = slot->curl_result;
 		slot->results->http_code = slot->http_code;
+		slot->results->retry_after = -1;
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTPAUTH_AVAIL,
 				  &slot->results->auth_avail);

http.c static void finish_active_slot(struct active_request_slot *slot) · line 360

Quoted 10 lines
http.c +7 -0
 
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CONNECTCODE,
 			&slot->results->http_connectcode);
+#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
+		{
+			curl_off_t retry_after;
+			if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
+					      &retry_after) == CURLE_OK && retry_after > 0)
+				slot->results->retry_after = (long)retry_after;
+		}

Makes me wonder whether results->retry_after should be declared as curl_off_t. But that's not a problem introduced by this patch series, as you simply move code around anyway.

http.c static int http_request_recoverable(const char *url, · line 2477

Quoted 10 lines
http.c +2 -1
@@ -2474,7 +2470,8 @@ static int http_request_recoverable(const char *url,
 	}
 
 	while ((ret == HTTP_REAUTH && --i) ||
-	       (ret == HTTP_RATE_LIMITED && --rate_limit_retries)) {
+	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
+		rate_limit_retries-- > 0)) {
 		long retry_delay = -1;
 		/*
 		 * The previous request may have put cruft into our output stream; we

Right, this here fixes the pre- vs post-decrement issue. Makes sense.

Honestly, every time I see a pre-decrement I'm already wondering whether it is correct in the first place. It always feels a bit too neat in my opinion.

http.c static int http_request_recoverable(const char *url, · line 2522

Quoted 9 lines
http.c +2 -0
@@ -2518,6 +2515,8 @@ static int http_request_recoverable(const char *url,
 		ret = http_request(url, result, target, options);
 	}
 	if (ret == HTTP_RATE_LIMITED) {
+		if (http_max_retries <= 0)
+			return HTTP_ERROR;
 		trace2_data_string("http", the_repository,
 				   "http/429-error", "retries-exhausted");
 		return HTTP_RATE_LIMITED;

Right. If we never had retries configured in the first place then it doesn't make sense to log "retries-exhausted", because there weren't any.

It's a bit curious that we return HTTP_ERROR instead of HTTP_RATE_LIMITED, as we're essentially still rate-limited. I would have expected us to only skip loggint the "retries-exhausted" message.

Quoted 7 lines
t/lib-httpd/http-429.sh +0 -0
index 1a5d7987db..5537fb7ac4 100644
@@ -32,12 +32,33 @@ repo_name="${repo_path%%/*}"
 safe_name=$(echo "${test_context}-${retry_after}-${repo_name}" | tr '/' '_' | tr -cd 'a-zA-Z0-9_-')
 state="http-429-state-${safe_name}"

t/lib-httpd/http-429.sh repo_name="${repo_path%%/*}" · line 39

Quoted 10 lines
t/lib-httpd/http-429.sh +5 -4
 
-# Check if this is the first call (no state directory exists), or if
-# the retry-after-value is "permanent", which indicates a 429 must be
-# returned for every request (even if the state directory exists).
-if test "$retry_after" != permanent && ! mkdir "$state" 2>/dev/null
+max_fails=1
+case "$test_context" in
+	*-fail-2*) max_fails=2 ;;
+	*-fail-3*) max_fails=3 ;;
+esac

Nit: coding style, arms of a case statement are typically on the same level as case.

Patrick

#26 lore raw

Re: [PATCH v3 1/5] http: fix retry count and Retry-After timing

Quoted 2 lines · Patrick Steinhardt

Nit: feels like two separate bugs that should have been fixed in two separate commits.

Since both are small fixes to 640657ffd0, would prefer to keep them together here to avoid churning the series layout.

Quoted 3 lines

It's a bit curious that we return HTTP_ERROR instead of HTTP_RATE_LIMITED, as we're essentially still rate-limited. I would have expected us to only skip loggint the "retries-exhausted" message.

That preserves existing behavior from 640657ffd0: when http.maxRetries is 0, callers like discover_refs() keep printing the standard curl 429 error instead of switching to the custom "rate limited" message.

Quoted 2 lines

Nit: coding style, arms of a case statement are typically on the same level as case.

ACK. Will fix in v4.

Thanks!

#27 lore raw

PATCH v3 2/5 http: add exponential backoff without Retry-After

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

When a server returns HTTP 429 without a Retry-After header, Git waits the same fixed http.retryAfter delay on every retry. If that delay is short (e.g., 1s), Git burns through all of its retries in just a few seconds before a busy server can recover. If that delay is long, even a brief glitch has to wait a long time on the first retry.

Pass the 1-based retry attempt count to handle_rate_limit_retry() and double http.retryAfter on each subsequent retry (1s, 2s, 4s, ..., via http.retryAfter * 2^(attempt - 1)), clamping the delay at http.maxRetryTime. This keeps the first retry fast while giving the server more time to recover on later attempts, matching curl's own --retry behavior (while keeping http.retryAfter's existing default of 0 for immediate retries when unconfigured). When the server provides an explicit Retry-After header, continue to honor the server-provided delay (or fail fast if it exceeds http.maxRetryTime).

---
 Documentation/config/http.adoc | 20 ++++++++++++++------
 http.c                         | 17 ++++++++++++++---
 t/t5584-http-429-retry.sh      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 9 deletions(-)
Documentation/config/http.adoc +14 -6
index 792a71b413..8e7538ab76 100644
@@ -327,9 +327,14 @@ http.keepAliveCount::
 	`GIT_HTTP_KEEPALIVE_COUNT` environment variable.
 
 http.retryAfter::
-	Default wait time in seconds before retrying when a server returns
+	Initial wait time in seconds before retrying when a server returns
 	HTTP 429 (Too Many Requests) without a Retry-After header.
-	Defaults to 0 (retry immediately). When a Retry-After header is
+	Defaults to 0, which retries immediately without waiting on every
+	attempt (unless a Retry-After header is present). When set to a
+	positive value and no Retry-After header is present, the first
+	retry waits `http.retryAfter` seconds and the delay doubles on
+	each subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
+	capped at `http.maxRetryTime`. When a Retry-After header is
 	present, its value takes precedence over this setting; however,
 	automatic use of the server-provided `Retry-After` header requires
 	libcurl 7.66.0 or later. On older versions, configure this setting
@@ -345,10 +350,13 @@ http.maxRetries::
 
 http.maxRetryTime::
 	Maximum time in seconds to wait for a single retry attempt when
-	handling HTTP 429 (Too Many Requests) responses. If the server
-	requests a delay (via Retry-After header) or if `http.retryAfter`
-	is configured with a value that exceeds this maximum, Git will fail
-	immediately rather than waiting. Default is 300 seconds (5 minutes).
+	handling HTTP 429 (Too Many Requests) responses.
+	If the server requests a delay (via Retry-After header) or if the
+	configured `http.retryAfter` value itself exceeds this maximum,
+	Git will fail immediately rather than waiting. Exponentially
+	backed-off delays computed from `http.retryAfter`, however, are
+	capped at this maximum rather than causing an immediate failure.
+	Default is 300 seconds (5 minutes).
 	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
 	variable. See also `http.retryAfter` and `http.maxRetries`.
 
http.c +14 -3
index 0308f973a6..8a28cd0fb3 100644
@@ -2411,7 +2411,7 @@ static int update_url_from_redirect(struct strbuf *base,
  * Returns a negative value if configuration is invalid (delay exceeds
  * http.maxRetryTime), otherwise returns the delay in seconds (>= 0).
  */
-static long handle_rate_limit_retry(long slot_retry_after)
+static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 {
 	/* Use the slot-specific retry_after value or configured default */
 	if (slot_retry_after >= 0) {
@@ -2427,6 +2427,8 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		return slot_retry_after;
 	} else {
+		long delay = http_retry_after;
+
 		/* No Retry-After header provided, use configured default */
 		if (http_retry_after > http_max_retry_time) {
 			error(_("configured http.retryAfter exceeds http.maxRetryTime (%ld > %ld seconds)"),
@@ -2437,7 +2439,14 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		trace2_data_string("http", the_repository,
 			"http/429-retry-source", "config-default");
-		return http_retry_after;
+		for (int a = 1; a < attempt && delay > 0; a++) {
+			if (delay >= http_max_retry_time - delay) {
+				delay = http_max_retry_time;
+				break;
+			}
+			delay *= 2;
+		}
+		return delay;
 	}
 }
 
@@ -2498,7 +2507,9 @@ static int http_request_recoverable(const char *url,
 			BUG("Unknown http_request target");
 		}
 		if (ret == HTTP_RATE_LIMITED) {
-			retry_delay = handle_rate_limit_retry(options->retry_after);
+			retry_delay = handle_rate_limit_retry(
+				options->retry_after,
+				http_max_retries - rate_limit_retries);
 			if (retry_delay < 0)
 				return HTTP_ERROR;
 
t/t5584-http-429-retry.sh +33 -0
index d92a26893d..f0c3cc3b61 100755
@@ -82,6 +82,39 @@ test_expect_success 'HTTP 429 without Retry-After uses configured default' '
 	test_grep "refs/heads/" output
 '
 
+test_expect_success 'exponential backoff doubles http.retryAfter without Retry-After header' '
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
+	git ls-remote "$HTTPD_URL/http_429/exp-backoff-fail-3/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep "waiting 2 seconds before retry" err &&
+	test_grep "waiting 4 seconds before retry" err
+'
+
+test_expect_success 'exponential backoff clamps delay at http.maxRetryTime' '
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
+	test_config http.maxRetryTime 2 &&
+	GIT_TRACE2_EVENT="$(pwd)/trace-clamp.event" \
+		git ls-remote "$HTTPD_URL/http_429/exp-clamp-fail-3/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep "waiting 2 seconds before retry" err &&
+	test_grep ! "waiting 4 seconds before retry" err &&
+	grep "\"key\":\"http/retry-sleep-seconds\",\"value\":\"2\"" trace-clamp.event >clamped &&
+	test_line_count = 2 clamped
+'
+
+test_expect_success 'server Retry-After header overrides exponential backoff' '
+	test_config http.maxRetries 2 &&
+	test_config http.retryAfter 1 &&
+	git ls-remote "$HTTPD_URL/http_429/header-override-fail-2/1/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep ! "waiting 2 seconds before retry" err
+'
+
 test_expect_success 'HTTP 429 retry delays are respected' '
 	# Enable retries
 	test_config http.maxRetries 3 &&
Signature · 1 line
gitgitgadget
#28 lore raw

Re: [PATCH v3 2/5] http: add exponential backoff without Retry-After

Quoted 11 lines · Ravi Mistry via GitGitGadget
Documentation/config/http.adoc +1 -1
index 792a71b413..8e7538ab76 100644
@@ -327,9 +327,14 @@ http.keepAliveCount::
 	`GIT_HTTP_KEEPALIVE_COUNT` environment variable.
 
 http.retryAfter::
-	Default wait time in seconds before retrying when a server returns
+	Initial wait time in seconds before retrying when a server returns
 	HTTP 429 (Too Many Requests) without a Retry-After header.

Documentation/config/http.adoc http.keepAliveCount:: · line 340

Quoted 10 lines
Documentation/config/http.adoc +6 -1
-	Defaults to 0 (retry immediately). When a Retry-After header is
+	Defaults to 0, which retries immediately without waiting on every
+	attempt (unless a Retry-After header is present). When set to a
+	positive value and no Retry-After header is present, the first
+	retry waits `http.retryAfter` seconds and the delay doubles on
+	each subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
+	capped at `http.maxRetryTime`. When a Retry-After header is
 	present, its value takes precedence over this setting; however,
 	automatic use of the server-provided `Retry-After` header requires
 	libcurl 7.66.0 or later. On older versions, configure this setting

Okay, having exponential backoff makes sense to me, and we do limit the maximum backoff via http.maxRetryTime. What I wonder is whether it would also make sense to introduce some jitter, but that of course does not have to be part of thi spatch series.

http.c static int update_url_from_redirect(struct strbuf *base, · line 2417

Quoted 13 lines
http.c +1 -1
index 0308f973a6..8a28cd0fb3 100644
@@ -2411,7 +2411,7 @@ static int update_url_from_redirect(struct strbuf *base,
  * Returns a negative value if configuration is invalid (delay exceeds
  * http.maxRetryTime), otherwise returns the delay in seconds (>= 0).
  */
-static long handle_rate_limit_retry(long slot_retry_after)
+static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 {
 	/* Use the slot-specific retry_after value or configured default */
 	if (slot_retry_after >= 0) {

Nit: a bit weird that the number of attempts is signed.

http.c static long handle_rate_limit_retry(long slot_retry_after) · line 2447

Quoted 11 lines
http.c +6 -1
@@ -2437,7 +2439,14 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		trace2_data_string("http", the_repository,
 			"http/429-retry-source", "config-default");
-		return http_retry_after;
+		for (int a = 1; a < attempt && delay > 0; a++) {
+			if (delay >= http_max_retry_time - delay) {
+				delay = http_max_retry_time;
+				break;
+			}
+			delay *= 2;

Isn't this the same as MIN(http_max_retry_time, delay * (2 << attempts - 1)? Feels a bit convoluted to have a whole loop for this.

Patrick

#29 lore raw

Re: [PATCH v3 2/5] http: add exponential backoff without Retry-After

Quoted 7 lines · Patrick Steinhardt

http.c static int update_url_from_redirect(struct strbuf *base, · line 2417

Quoted 5 lines
http.c +1 -1
-static long handle_rate_limit_retry(long slot_retry_after)
+static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 {
 	/* Use the slot-specific retry_after value or configured default */
 	if (slot_retry_after >= 0) {

Nit: a bit weird that the number of attempts is signed.

Good point. I will make attempt size_t in v4.

Quoted 9 lines

http.c static long handle_rate_limit_retry(long slot_retry_after) · line 2447

Quoted 6 lines
http.c +6 -0
+		for (int a = 1; a < attempt && delay > 0; a++) {
+			if (delay >= http_max_retry_time - delay) {
+				delay = http_max_retry_time;
+				break;
+			}
+			delay *= 2;

Isn't this the same as MIN(http_max_retry_time, delay * (2 << attempts - 1)? Feels a bit convoluted to have a whole loop for this.

I used the loop to avoid undefined bit-shifts and multiplication overflow before MIN() runs if someone configures a large http.maxRetries (for example, 100).

Thanks!

#30 lore raw

PATCH v3 3/5 http: add http.retryStatusCodes config option

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) hardcoded HTTP 429 as the only status code eligible for retry. In practice, reverse proxies, load balancers, and Git hosting backends also return transient 5xx responses such as 502 Bad Gateway, 503 Service Unavailable (which RFC 9110 section 10.2.3 defines to support Retry-After), and 504 Gateway Timeout during load shedding or upstream connection resets.

Because 5xx responses are not universally transient across all environments, keep 429 as the default and introduce a new http.retryStatusCodes configuration option (and corresponding GIT_HTTP_RETRY_STATUS_CODES environment variable) accepting a comma- or whitespace-separated list of HTTP status codes in the 400..599 range (for example, "429,502,503,504").

Keep the existing "rate limited" warning and error messages for HTTP 429, and report the numeric status code (e.g., "HTTP 502, waiting <N> seconds before retry") when retrying other status codes.

---
 Documentation/config/http.adoc | 34 +++++++++++++++++++-----
 http.c                         | 42 ++++++++++++++++++++++++++---
 http.h                         |  6 +++--
 t/lib-httpd/http-429.sh        | 10 +++++--
 t/t5584-http-429-retry.sh      | 48 ++++++++++++++++++++++++++++++++++
 5 files changed, 125 insertions(+), 15 deletions(-)
Documentation/config/http.adoc +27 -7
index 8e7538ab76..e7fa251b65 100644
@@ -328,7 +328,10 @@ http.keepAliveCount::
 
 http.retryAfter::
 	Initial wait time in seconds before retrying when a server returns
-	HTTP 429 (Too Many Requests) without a Retry-After header.
+	a retryable HTTP status code without a Retry-After header. The
+	set of retryable status codes is configured by
+	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
+	Requests) by default.
 	Defaults to 0, which retries immediately without waiting on every
 	attempt (unless a Retry-After header is present). When set to a
 	positive value and no Retry-After header is present, the first
@@ -340,17 +343,23 @@ http.retryAfter::
 	libcurl 7.66.0 or later. On older versions, configure this setting
 	manually to control the retry delay. Can be overridden by the
 	`GIT_HTTP_RETRY_AFTER` environment variable.
-	See also `http.maxRetries` and `http.maxRetryTime`.
+	See also `http.maxRetries`, `http.maxRetryTime`, and
+	`http.retryStatusCodes`.
 
 http.maxRetries::
-	Maximum number of times to retry after receiving HTTP 429 (Too Many
-	Requests) responses. Set to 0 (the default) to disable retries.
+	Maximum number of times to retry after receiving a retryable HTTP
+	status code. The set of retryable status codes is configured by
+	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
+	Requests) by default. Set to 0 (the default) to disable retries.
 	Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
-	See also `http.retryAfter` and `http.maxRetryTime`.
+	See also `http.retryAfter`, `http.maxRetryTime`, and
+	`http.retryStatusCodes`.
 
 http.maxRetryTime::
 	Maximum time in seconds to wait for a single retry attempt when
-	handling HTTP 429 (Too Many Requests) responses.
+	handling a retryable HTTP status code. The set of retryable
+	status codes is configured by `http.retryStatusCodes`, which
+	includes only HTTP 429 (Too Many Requests) by default.
 	If the server requests a delay (via Retry-After header) or if the
 	configured `http.retryAfter` value itself exceeds this maximum,
 	Git will fail immediately rather than waiting. Exponentially
@@ -358,7 +367,18 @@ http.maxRetryTime::
 	capped at this maximum rather than causing an immediate failure.
 	Default is 300 seconds (5 minutes).
 	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
-	variable. See also `http.retryAfter` and `http.maxRetries`.
+	variable. See also `http.retryAfter`, `http.maxRetries`, and
+	`http.retryStatusCodes`.
+
+http.retryStatusCodes::
+	Comma- or whitespace-separated list of HTTP status codes (in the
+	400..599 range) that should be retried up to `http.maxRetries`
+	times. Defaults to `429` (Too Many Requests). For example, setting
+	this to `429,502,503,504` also retries transient `5xx` gateway and
+	service-unavailable errors. Can be overridden by the
+	`GIT_HTTP_RETRY_STATUS_CODES` environment variable.
+	See also `http.maxRetries`, `http.retryAfter`, and
+	`http.maxRetryTime`.
 
 http.noEPSV::
 	A boolean which disables using of EPSV ftp command by curl.
http.c +38 -4

102 lines not on this page · raw message

http.h +4 -2
index 729c51904d..5985a0f71f 100644
@@ -167,10 +167,12 @@ struct http_get_options {
 
 	/*
 	 * After a request completes, contains the Retry-After delay in seconds
-	 * if the server returned HTTP 429 with a Retry-After header (requires
-	 * libcurl 7.66.0 or later), or -1 if no such header was present.
+	 * if the server returned a retryable status code with a Retry-After
+	 * header (requires libcurl 7.66.0 or later), or -1 if no such header
+	 * was present.
 	 */
 	long retry_after;
+	long http_code;
 };
 
 /* Return values for http_get_*() */
t/lib-httpd/http-429.sh +8 -2
index 5537fb7ac4..5c7d58a75f 100644
@@ -75,8 +75,14 @@ then
 	exec "$GIT_EXEC_PATH/git-http-backend"
 fi
 
-# Output HTTP 429 response
-printf "Status: 429 Too Many Requests\r\n"
+# Output HTTP 429 (or configured status) response
+status_line="429 Too Many Requests"
+case "$test_context" in
+	*-status-502*) status_line="502 Bad Gateway" ;;
+	*-status-503*) status_line="503 Service Unavailable" ;;
+	*-status-504*) status_line="504 Gateway Timeout" ;;
+esac
+printf "Status: %s\r\n" "$status_line"
 
 # Set Retry-After header based on retry_after value
 case "$retry_after" in
t/t5584-http-429-retry.sh +48 -0

52 lines not on this page · raw message

Signature · 1 line
gitgitgadget
#31 lore raw

Re: [PATCH v3 3/5] http: add http.retryStatusCodes config option

Quoted 10 lines · Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) hardcoded HTTP 429 as the only status code eligible for retry. In practice, reverse proxies, load balancers, and Git hosting backends also return transient 5xx responses such as 502 Bad Gateway, 503 Service Unavailable (which RFC 9110 section 10.2.3 defines to support Retry-After), and 504 Gateway Timeout during load shedding or upstream connection resets.

Quoted 6 lines

Because 5xx responses are not universally transient across all environments, keep 429 as the default and introduce a new http.retryStatusCodes configuration option (and corresponding GIT_HTTP_RETRY_STATUS_CODES environment variable) accepting a comma- or whitespace-separated list of HTTP status codes in the 400..599 range (for example, "429,502,503,504").

Hmm, okay. It feels a bit like we're capitulating and push the issue onto the user instead of fixing the inconsistencies on the server side. I'm not really much of a fan of that, as now every single user will have to potentially reconfigure their Git client so that it knows to speak with whatever server they want to.

I'd really much rather prefer servers to get their act together and fix this properly. Because what I think would be a bad outcome here is that the fragmentation continues and that providers tell their users that they are holding it wrong, and that they need to just configure their Git client properly.

Maybe the fact that Git now handles retries is a motivation for providers to fix their infrastructure and instead align on one specific error code here. So instead of making this configurable, can we instead nudge the ecosystem in the right direction and make providers fix their infra, so that it just works and users don't have to configure anything?

Patrick

#32 lore raw

Re: [PATCH v3 3/5] http: add http.retryStatusCodes config option

Quoted 5 lines · Patrick Steinhardt

Maybe the fact that Git now handles retries is a motivation for providers to fix their infrastructure and instead align on one specific error code here. So instead of making this configurable, can we instead nudge the ecosystem in the right direction and make providers fix their infra, so that it just works and users don't have to configure anything?

For rate limiting, I agree that providers should align on 429. 502/503/504 cover different failures: reverse proxies, CDNs, and client-side proxies return 502/504 when a connection drops or times out, and servers return 503 during temporary overload or maintenance.

We also should not retry them by default because in some setups they could point to a permanently broken proxy or server. Keeping 429 as the default lets CI systems and system administrators opt in centrally via http.retryStatusCodes where 5xx errors are transient.

Thanks!

#33 lore raw

Re: [PATCH v3 3/5] http: add http.retryStatusCodes config option

Quoted 13 lines · Ravi Mistry
Quoted 5 lines · Patrick Steinhardt

Maybe the fact that Git now handles retries is a motivation for providers to fix their infrastructure and instead align on one specific error code here. So instead of making this configurable, can we instead nudge the ecosystem in the right direction and make providers fix their infra, so that it just works and users don't have to configure anything?

For rate limiting, I agree that providers should align on 429. 502/503/504 cover different failures: reverse proxies, CDNs, and client-side proxies return 502/504 when a connection drops or times out, and servers return 503 during temporary overload or maintenance.

Quoted 4 lines

We also should not retry them by default because in some setups they could point to a permanently broken proxy or server. Keeping 429 as the default lets CI systems and system administrators opt in centrally via http.retryStatusCodes where 5xx errors are transient.

That part I understand, but I'm still not particularly happy about the fact that we make this the problem of the user.

At the very least, we should document that the expected behaviour of a service provider is to return 429 in case they are overloaded, and that all other behaviour in this case is considered to be buggy. The config should thus make it explicit that it is a workaround for buggy providers to entice them to fix their infra.

That would hopefully nudge them into the right direction.

Patrick

#34 lore raw

Re: [PATCH v3 3/5] http: add http.retryStatusCodes config option

Quoted 7 lines · Patrick Steinhardt

At the very least, we should document that the expected behaviour of a service provider is to return 429 in case they are overloaded, and that all other behaviour in this case is considered to be buggy. The config should thus make it explicit that it is a workaround for buggy providers to entice them to fix their infra.

That would hopefully nudge them into the right direction.

Documenting this makes sense. I will update the http.retryStatusCodes doc in v4 to make it explicit that providers are expected to return 429 when overloaded, and that configuring other status codes should only be a temporary workaround while the underlying infrastructure is fixed.

Thanks!

#35 lore raw

PATCH v3 4/5 remote-curl: retry transient HTTP errors in post_rpc()

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Git's existing retry logic only covers HTTP GET requests. However, the actual data transfer during fetch and clone, as well as on-demand blob downloads in partial clones, are sent as HTTP POST requests via post_rpc(), which only retries authentication and immediately aborts on 429 or 5xx.

Extract the retry wait logic from http_request_recoverable() into a shared http_retry_wait() helper and use it to retry transient HTTP errors in probe_rpc() and post_rpc(). Also defer reporting RPC errors until retries are exhausted, and re-attach the "Content-Encoding: gzip" header when retrying a compressed request.

---
 http.c                    | 67 ++++++++++++++++++----------------
 http.h                    |  6 ++++
 remote-curl.c             | 75 +++++++++++++++++++--------------------
 t/lib-httpd/http-429.sh   | 50 ++++++++++++++++++--------
 t/t5584-http-429-retry.sh | 53 +++++++++++++++++++++++++++
 5 files changed, 168 insertions(+), 83 deletions(-)
http.c +36 -31
index fc9384353c..ea4f9245c4 100644
@@ -2479,6 +2479,34 @@ static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 	}
 }
 
+int http_retry_wait(long http_code, long slot_retry_after, int attempt)
+{
+	long retry_delay;
+
+	if (http_max_retries <= 0)
+		return -1;
+	if (attempt > http_max_retries) {
+		trace2_data_string("http", the_repository,
+				   "http/429-error", "retries-exhausted");
+		return -1;
+	}
+	retry_delay = handle_rate_limit_retry(slot_retry_after, attempt);
+	if (retry_delay < 0)
+		return -1;
+	if (retry_delay > 0) {
+		if (http_code == 429)
+			warning(_("rate limited, waiting %ld seconds before retry"),
+				retry_delay);
+		else
+			warning(_("HTTP %ld, waiting %ld seconds before retry"),
+				http_code, retry_delay);
+		trace2_data_intmax("http", the_repository,
+				   "http/retry-sleep-seconds", retry_delay);
+		sleep(retry_delay);
+	}
+	return 0;
+}
+
 static int http_request_recoverable(const char *url,
 			       void *result, int target,
 			       struct http_get_options *options)
@@ -2486,7 +2514,7 @@ static int http_request_recoverable(const char *url,
 	static struct http_get_options empty_opts;
 	int i = 3;
 	int ret;
-	int rate_limit_retries = http_max_retries;
+	int attempt = 0;
 
 	if (!options)
 		options = &empty_opts;
@@ -2508,9 +2536,8 @@ static int http_request_recoverable(const char *url,
 	}
 
 	while ((ret == HTTP_REAUTH && --i) ||
-	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
-		rate_limit_retries-- > 0)) {
-		long retry_delay = -1;
+	       (ret == HTTP_RATE_LIMITED &&
+		!http_retry_wait(options->http_code, options->retry_after, ++attempt))) {
 		/*
 		 * The previous request may have put cruft into our output stream; we
 		 * should clear it out before making our next request.
@@ -2535,37 +2562,15 @@ static int http_request_recoverable(const char *url,
 		default:
 			BUG("Unknown http_request target");
 		}
-		if (ret == HTTP_RATE_LIMITED) {
-			retry_delay = handle_rate_limit_retry(
-				options->retry_after,
-				http_max_retries - rate_limit_retries);
-			if (retry_delay < 0)
-				return HTTP_ERROR;
-
-			if (retry_delay > 0) {
-				if (options->http_code == 429)
-					warning(_("rate limited, waiting %ld seconds before retry"),
-						retry_delay);
-				else
-					warning(_("HTTP %ld, waiting %ld seconds before retry"),
-						options->http_code, retry_delay);
-				trace2_data_intmax("http", the_repository,
-						   "http/retry-sleep-seconds", retry_delay);
-				sleep(retry_delay);
-			}
-		} else if (ret == HTTP_REAUTH) {
+		if (ret == HTTP_REAUTH)
 			http_reauth_prepare(1);
-		}
 
 		ret = http_request(url, result, target, options);
 	}
-	if (ret == HTTP_RATE_LIMITED) {
-		if (http_max_retries <= 0)
-			return HTTP_ERROR;
-		trace2_data_string("http", the_repository,
-				   "http/429-error", "retries-exhausted");
-		return options->http_code == 429 ? HTTP_RATE_LIMITED : HTTP_ERROR;
-	}
+	if (ret == HTTP_RATE_LIMITED &&
+	    (http_max_retries <= 0 || attempt <= http_max_retries ||
+	     options->http_code != 429))
+		return HTTP_ERROR;
 	return ret;
 }
 
http.h +6 -0
index 5985a0f71f..fa9f46a11b 100644
@@ -201,6 +201,12 @@ int http_get_strbuf(const char *url, struct strbuf *result, struct http_get_opti
 int http_get_file(const char *url, const char *filename,
 		  struct http_get_options *options);
 
+/*
+ * Wait before retrying a transient HTTP error (1-based attempt count).
+ * Returns 0 if the caller should retry, or -1 to stop retrying.
+ */
+int http_retry_wait(long http_code, long slot_retry_after, int attempt);
+
 int http_fetch_ref(const char *base, struct ref *ref);
 
 struct curl_slist *http_append_auth_header(const struct credential *c,
remote-curl.c +37 -38

130 lines not on this page · raw message

t/lib-httpd/http-429.sh +36 -14

63 lines not on this page · raw message

t/t5584-http-429-retry.sh +53 -0

57 lines not on this page · raw message

Signature · 1 line
gitgitgadget
#36 lore raw

Re: [PATCH v3 4/5] remote-curl: retry transient HTTP errors in post_rpc()

Quoted 8 lines · Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Git's existing retry logic only covers HTTP GET requests. However, the actual data transfer during fetch and clone, as well as on-demand blob downloads in partial clones, are sent as HTTP POST requests via post_rpc(), which only retries authentication and immediately aborts on 429 or 5xx.

Quoted 5 lines

Extract the retry wait logic from http_request_recoverable() into a shared http_retry_wait() helper and use it to retry transient HTTP errors in probe_rpc() and post_rpc(). Also defer reporting RPC errors until retries are exhausted, and re-attach the "Content-Encoding: gzip" header when retrying a compressed request.

But the POST requests aren't necessarily idempotent, are they? We for example also use post_rpc() in push_git() to POST data when reaching out to git-receive-pack(1). I don't think we can assume that it's safe to retry requests there.

So I think we'll have to adapt this so that we only do this for some request types.

Patrick

#37 lore raw

Re: [PATCH v3 4/5] remote-curl: retry transient HTTP errors in post_rpc()

Quoted 7 lines · Patrick Steinhardt

But the POST requests aren't necessarily idempotent, are they? We for example also use post_rpc() in push_git() to POST data when reaching out to git-receive-pack(1). I don't think we can assume that it's safe to retry requests there.

So I think we'll have to adapt this so that we only do this for some request types.

This is a very good point. Retrying a git-receive-pack POST after a 502/504 is not safe if the remote already processed the push.

In v4 I will restrict retries in post_rpc() to read-only services ("git-upload-pack" and "git-upload-archive") so git-receive-pack is not retried, and add a test covering push.

Thanks!

#38 lore raw

PATCH v3 5/5 http: default http.retryAfter to 10 seconds

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

When http.maxRetries is enabled and a server returns HTTP 429 (or an opt-in status code via http.retryStatusCodes) without a Retry-After header, http.retryAfter currently defaults to 0. That causes Git to retry immediately in a tight loop on every attempt, defeating exponential backoff and continuing to hammer a rate-limiting server right after it asked the client to back off.

Default http.retryAfter to 10 seconds (backing off as 10s, 20s, 40s, ..., capped at http.maxRetryTime) so that overloaded Git servers get meaningful breathing room while keeping interactive commands reasonably responsive. Callers or tests that want immediate retries can still set http.retryAfter=0 explicitly.

Because http.retryAfter can now exceed a lowered http.maxRetryTime even when http.retryAfter itself was not explicitly configured, drop the word "configured" from the error message and config documentation. Also update the past HTTP-date test in t5584: CURLINFO_RETRY_AFTER only yields a positive delay for future timestamps, so past dates fall back to http.retryAfter (which previously waited 0s only because the default was 0).

---
 Documentation/config/http.adoc | 25 +++++++++++++------------
 http.c                         |  4 ++--
 t/t5584-http-429-retry.sh      | 26 ++++++++++++++++++++------
 3 files changed, 35 insertions(+), 20 deletions(-)
Documentation/config/http.adoc +13 -12
index e7fa251b65..aa86bc68d5 100644
@@ -332,16 +332,17 @@ http.retryAfter::
 	set of retryable status codes is configured by
 	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
 	Requests) by default.
-	Defaults to 0, which retries immediately without waiting on every
-	attempt (unless a Retry-After header is present). When set to a
-	positive value and no Retry-After header is present, the first
-	retry waits `http.retryAfter` seconds and the delay doubles on
-	each subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
-	capped at `http.maxRetryTime`. When a Retry-After header is
-	present, its value takes precedence over this setting; however,
-	automatic use of the server-provided `Retry-After` header requires
-	libcurl 7.66.0 or later. On older versions, configure this setting
-	manually to control the retry delay. Can be overridden by the
+	Defaults to 10. When positive and no Retry-After header is
+	present, the first retry waits `http.retryAfter` seconds and the
+	delay doubles on each subsequent retry attempt
+	(`http.retryAfter * 2^(attempt - 1)`), capped at
+	`http.maxRetryTime`. Set to 0 to retry immediately without waiting
+	on every attempt (unless a Retry-After header is present). When a
+	Retry-After header with a positive delay is present, its value
+	takes precedence over this setting; however, automatic use of the
+	server-provided `Retry-After` header requires libcurl 7.66.0 or
+	later. On older versions, `Retry-After` headers are ignored and
+	`http.retryAfter` is always used. Can be overridden by the
 	`GIT_HTTP_RETRY_AFTER` environment variable.
 	See also `http.maxRetries`, `http.maxRetryTime`, and
 	`http.retryStatusCodes`.
@@ -360,8 +361,8 @@ http.maxRetryTime::
 	handling a retryable HTTP status code. The set of retryable
 	status codes is configured by `http.retryStatusCodes`, which
 	includes only HTTP 429 (Too Many Requests) by default.
-	If the server requests a delay (via Retry-After header) or if the
-	configured `http.retryAfter` value itself exceeds this maximum,
+	If the server requests a delay (via Retry-After header) or if
+	`http.retryAfter` itself exceeds this maximum,
 	Git will fail immediately rather than waiting. Exponentially
 	backed-off delays computed from `http.retryAfter`, however, are
 	capped at this maximum rather than causing an immediate failure.
http.c +2 -2
index ea4f9245c4..4ce662ffe4 100644
@@ -153,7 +153,7 @@ static char *http_ssl_backend;
 
 static int http_schannel_check_revoke = 1;
 
-static long http_retry_after = 0;
+static long http_retry_after = 10;
 static long http_max_retries = 0;
 static long http_max_retry_time = 300;
 static char *http_retry_status_codes;
@@ -2460,7 +2460,7 @@ static long handle_rate_limit_retry(long slot_retry_after, int attempt)
 
 		/* No Retry-After header provided, use configured default */
 		if (http_retry_after > http_max_retry_time) {
-			error(_("configured http.retryAfter exceeds http.maxRetryTime (%ld > %ld seconds)"),
+			error(_("http.retryAfter exceeds http.maxRetryTime (%ld > %ld seconds)"),
 			      http_retry_after, http_max_retry_time);
 			trace2_data_string("http", the_repository,
 					"http/429-error", "config-exceeds-max-retry-time");
t/t5584-http-429-retry.sh +20 -6
index e627aa2b63..388c5db1b6 100755
@@ -53,12 +53,14 @@ test_expect_success 'HTTP 429 with Retry-After is retried and succeeds' '
 
 test_expect_success 'http.maxRetries=1 retries once and succeeds' '
 	test_config http.maxRetries 1 &&
+	test_config http.retryAfter 0 &&
 	git ls-remote "$HTTPD_URL/http_429/max-retries-one/none/repo.git" >output 2>err &&
 	test_grep "refs/heads/" output
 '
 
 test_expect_success 'http.maxRetries=2 fails after 3 failures and succeeds on 2 failures' '
 	test_config http.maxRetries 2 &&
+	test_config http.retryAfter 0 &&
 	test_must_fail git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-3/none/repo.git" 2>err &&
 	test_grep "rate limited" err &&
 	git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-2/none/repo.git" >output 2>err &&
@@ -72,6 +74,13 @@ test_expect_success 'Trace2 logs actual CURLINFO_RETRY_AFTER value instead of -1
 	test_grep "\"key\":\"http/429-retry-after\",\"value\":\"1\"" trace-retry-after.event
 '
 
+test_expect_success 'http.retryAfter defaults to 10 seconds when unset' '
+	test_config http.maxRetries 1 &&
+	test_config http.maxRetryTime 9 &&
+	test_must_fail git ls-remote "$HTTPD_URL/http_429/default-retry-after-10/none/repo.git" 2>err &&
+	test_grep "http.retryAfter exceeds http.maxRetryTime (10 > 9 seconds)" err
+'
+
 test_expect_success 'HTTP 429 without Retry-After uses configured default' '
 	# Enable retries and configure default delay
 	test_config http.maxRetries 3 &&
@@ -161,7 +170,7 @@ test_expect_success 'HTTP 429 fails if configured http.retryAfter exceeds http.m
 	# Should fail quickly (no 100 second wait)
 	duration_int=${duration%.*} &&
 	test "$duration_int" -lt 99 &&
-	test_grep "configured http.retryAfter.*exceeds.*http.maxRetryTime" err
+	test_grep "http.retryAfter.*exceeds.*http.maxRetryTime" err
 '
 
 test_expect_success 'HTTP 429 with Retry-After HTTP-date format' '
@@ -210,7 +219,7 @@ test_expect_success 'HTTP 429 with HTTP-date exceeding maxRetryTime fails immedi
 	test_grep "http.maxRetryTime" err
 '
 
-test_expect_success 'HTTP 429 with past HTTP-date should not wait' '
+test_expect_success 'HTTP 429 with past HTTP-date falls back to http.retryAfter' '
 	raw=$(test-tool date timestamp now) &&
 	now="${raw#* -> }" &&
 	past_time=$((now - 10)) &&
@@ -218,18 +227,21 @@ test_expect_success 'HTTP 429 with past HTTP-date should not wait' '
 	past_date="${raw#* -> }" &&
 	past_date_encoded=$(echo "$past_date" | sed "s/ /%20/g") &&
 
-	# Enable retries
+	# Enable retries with a short fallback delay
 	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
 
-	# Git should retry immediately without waiting
+	# CURLINFO_RETRY_AFTER does not report a positive delay for past dates,
+	# so Git falls back to http.retryAfter rather than retrying immediately
 	start=$(test-tool date getnanos) &&
 	git ls-remote "$HTTPD_URL/http_429/past-http-date/$past_date_encoded/repo.git" >output 2>err &&
 	duration=$(test-tool date getnanos $start) &&
 
-	# Should complete quickly (no wait for a past-date Retry-After)
 	duration_int=${duration%.*} &&
+	test "$duration_int" -ge 1 &&
 	test "$duration_int" -lt 5 &&
-	test_grep "refs/heads/" output
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err
 '
 
 test_expect_success 'HTTP 429 with invalid Retry-After format uses configured default' '
@@ -349,6 +361,7 @@ test_expect_success 'HTTP 503 honors Retry-After header when present' '
 
 test_expect_success 'exhausted 502 retries report HTTP 502 error, not rate limited' '
 	test_config http.maxRetries 2 &&
+	test_config http.retryAfter 0 &&
 	test_config http.retryStatusCodes "429,502" &&
 	test_must_fail git ls-remote "$HTTPD_URL/http_429/exhausted-fail-3-status-502/none/repo.git" 2>err &&
 	test_grep "The requested URL returned error: 502" err &&
@@ -357,6 +370,7 @@ test_expect_success 'exhausted 502 retries report HTTP 502 error, not rate limit
 
 test_expect_success 'GIT_HTTP_RETRY_STATUS_CODES and urlmatch http.<url>.retryStatusCodes' '
 	test_config http.maxRetries 2 &&
+	test_config http.retryAfter 0 &&
 	GIT_HTTP_RETRY_STATUS_CODES="429,502" \
 		git ls-remote "$HTTPD_URL/http_429/env-status-502/none/repo.git" >output 2>err &&
 	test_grep "refs/heads/" output &&
Signature · 1 line
gitgitgadget
#39 lore raw

Re: [PATCH v3 5/5] http: default http.retryAfter to 10 seconds

Quoted 16 lines · Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Documentation/config/http.adoc +0 -7
index e7fa251b65..aa86bc68d5 100644
@@ -332,16 +332,17 @@ http.retryAfter::
 	set of retryable status codes is configured by
 	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
 	Requests) by default.
-	Defaults to 0, which retries immediately without waiting on every
-	attempt (unless a Retry-After header is present). When set to a
-	positive value and no Retry-After header is present, the first
-	retry waits `http.retryAfter` seconds and the delay doubles on
-	each subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
-	capped at `http.maxRetryTime`. When a Retry-After header is
-	present, its value takes precedence over this setting; however,

Documentation/config/http.adoc http.retryAfter:: · line 341

Quoted 10 lines
Documentation/config/http.adoc +7 -3
-	automatic use of the server-provided `Retry-After` header requires
-	libcurl 7.66.0 or later. On older versions, configure this setting
-	manually to control the retry delay. Can be overridden by the
+	Defaults to 10. When positive and no Retry-After header is
+	present, the first retry waits `http.retryAfter` seconds and the
+	delay doubles on each subsequent retry attempt
+	(`http.retryAfter * 2^(attempt - 1)`), capped at
+	`http.maxRetryTime`. Set to 0 to retry immediately without waiting
+	on every attempt (unless a Retry-After header is present). When a
+	Retry-After header with a positive delay is present, its value

So we ignore an explicit "Retry-After: 0"? Shouldn't we honor that in case the server explicitly tells us to retry immediately?

Patrick

#40 lore raw

Re: [PATCH v3 5/5] http: default http.retryAfter to 10 seconds

Quoted 2 lines · Patrick Steinhardt

So we ignore an explicit "Retry-After: 0"? Shouldn't we honor that in case the server explicitly tells us to retry immediately?

Yes, I agree, we should honor that. libcurl's CURLINFO_RETRY_AFTER returns 0 when Retry-After is absent or malformed, and <= 0 when the server sends "Retry-After: 0" or a past HTTP-date, so the existing retry_after > 0 check could not tell them apart.

I will inspect the Retry-After header in v4 when CURLINFO_RETRY_AFTER <= 0 so we retry immediately on an explicit "Retry-After: 0" or past HTTP-date, while absent or malformed headers still fall back to http.retryAfter.

Thanks!

#41 lore raw
In reply to Ravi Mistry via GitGitGadget
Quoted 6 lines · Ravi Mistry via GitGitGadget

Changes since v2:

* Patches 1/5..4/5: Unchanged from v2.
* Added Patch 5/5 to default http.retryAfter to 10 seconds instead of 0 when Retry-After is absent (per feedback from Junio C Hamano and brian m. carlson).

We usually do not structure our topics this way, to first implement one decision (i.e., "waiting 0 is the right thing to do") and then change the decision in the middle of the topic (i.e., "no, hammering the server immediately is a mistake, so let's give it the initial wait of 10 seconds").

This might make it easier to later switch it to 30 seconds if we wanted to, by identifying exact places that needs to be updated?

I dunno. brian, what do you think?

Thanks.

#42 lore raw
Quoted 10 lines · Junio C Hamano

We usually do not structure our topics this way, to first implement one decision (i.e., "waiting 0 is the right thing to do") and then change the decision in the middle of the topic (i.e., "no, hammering the server immediately is a mistake, so let's give it the initial wait of 10 seconds").

This might make it easier to later switch it to 30 seconds if we wanted to, by identifying exact places that needs to be updated?

I dunno. brian, what do you think?

I kept 5/5 separate to isolate changing the existing 0s default from 640657ffd0. Happy to go either way:

- Leave v3 as is (with 5/5 as a separate commit).
- Fold 5/5 into Patch 2 in a v4.

Thanks!

#43 lore raw

v4 cover http: exponential backoff, configurable status codes, and post_rpc() retries

In reply to Ravi Mistry via GitGitGadget

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) added HTTP 429 retry support using http.maxRetries, http.retryAfter, and http.maxRetryTime.

In practice, when running Git against hosting backends and reverse proxies under high concurrency, four gaps remain:

1. Off-by-one retry count and Retry-After Trace2 timing: Because http_request_recoverable() pre-decrements the retry counter (--rate_limit_retries) in its loop condition, setting http.maxRetries=1 performs 0 retries (and N configured retries performs N-1 retries). In addition, the Retry-After header is extracted after Trace2 logging has already run, so the http/429-retry-after Trace2 event always logs -1.

2. No exponential backoff (and 0s default) when Retry-After is absent: When a server does not send a Retry-After header, Git defaults http.retryAfter to 0 (retrying immediately in a tight loop) and waits the same fixed delay on every retry when configured. Doubling the delay on each attempt and defaulting http.retryAfter to 10 seconds (10s, 20s, 40s, ...) gives overloaded servers breathing room while keeping interactive commands reasonably responsive.

3. Transient 5xx errors cannot be retried: While service providers are expected to return 429 when rate-limiting or overloaded, reverse proxies, CDNs, client-side proxies, and servers also surface transient connection drops, timeouts, or load shedding as 502, 503, or 504. Because 5xx errors are not universally transient across all setups, 429 should remain the default, with an opt-in config to specify additional status codes as a temporary workaround while underlying infrastructure is fixed.

4. Read-only HTTP POST requests are not retried: Git's existing 429 retry logic only covers HTTP GET requests (such as the initial /info/refs handshake). However, the actual data transfer during fetch and clone, as well as on-demand blob downloads in partial clones (during checkout, diff, or blame), are sent as HTTP POST requests (/git-upload-pack) via post_rpc(), which only retries authentication and immediately aborts on 429 or 5xx.

This 5-patch series addresses all four gaps:

* Patch 1/5 fixes the off-by-one retry count bug and extracts Retry-After in finish_active_slot() before Trace2 logging runs.
* Patch 2/5 doubles http.retryAfter on each subsequent retry attempt (capped at http.maxRetryTime) when Retry-After is absent.
* Patch 3/5 adds http.retryStatusCodes (and GIT_HTTP_RETRY_STATUS_CODES), defaulting to "429", so environments can opt into retrying codes like "429,502,503,504".
* Patch 4/5 extracts a shared http_retry_wait() helper and extends retry support in post_rpc() to read-only services ("git-upload-pack" and "git-upload-archive") without retrying "git-receive-pack" (git push), while fixing a missing Content-Encoding: gzip header on retry.
* Patch 5/5 changes the default of http.retryAfter from 0 to 10 seconds so Git does not immediately hammer a rate-limiting server when no Retry-After header is provided.

Prior discussions:

* https://lore.kernel.org/git/[email protected]/
* https://lore.kernel.org/git/[email protected]/

Changes since v3 (thanks to Patrick Steinhardt for the review):

* Patches 1/5, 3/5, 4/5: Un-indented shell case arms in t/lib-httpd/http-429.sh to match CodingGuidelines.
* Patches 2/5, 4/5: Changed the retry attempt counter from int to size_t.
* Patch 3/5: Documented in http.retryStatusCodes that providers are expected to return 429 when rate-limiting or overloaded, and that configuring additional status codes is a temporary workaround while underlying provider or proxy infrastructure is fixed.
* Patch 4/5: Restricted retries in post_rpc() to read-only services ("git-upload-pack" and "git-upload-archive") so "git-receive-pack" (git push) is not retried, documented the retried request types in http.maxRetries, and added a test covering push.
* Patch 5/5: Inspected the Retry-After header via curl_easy_header() when CURLINFO_RETRY_AFTER <= 0 so an explicit "Retry-After: 0" or past HTTP-date retries immediately instead of falling back to the 10-second default, restored the past HTTP-date test in t5584, and added a test for "Retry-After: 0".

Changes since v2:

* Patches 1/5..4/5: Unchanged from v2.
* Added Patch 5/5 to default http.retryAfter to 10 seconds instead of 0 when Retry-After is absent (per feedback from Junio C Hamano and brian m. carlson).

Changes since v1:

* Patch 1/4: Clarified in the commit message why the http_max_retries <= 0 check was moved after the retry loop.
* Patch 2/4: Passed http_max_retries - rate_limit_retries instead of adding a separate attempt counter, and clarified the http.retryAfter and http.maxRetryTime docs.
* Patch 3/4: Restored (Too Many Requests) and clarified http.retryStatusCodes in the config docs, and moved the 400..599 range check to the top of is_retryable_http_code().
* Patch 4/4: Added the attempt counter here when replacing rate_limit_retries with http_retry_wait().

Ravi Mistry (5):
http: fix retry count and Retry-After timing
http: add exponential backoff without Retry-After
http: add http.retryStatusCodes config option
remote-curl: retry transient HTTP errors in post_rpc()
http: default http.retryAfter to 10 seconds

 Documentation/config/http.adoc |  67 ++++++++---
 git-curl-compat.h              |   8 ++
 http.c                         | 169 ++++++++++++++++++++-------
 http.h                         |  12 +-
 remote-curl.c                  |  82 +++++++------
 t/lib-httpd/http-429.sh        |  63 ++++++++--
 t/t5584-http-429-retry.sh      | 205 ++++++++++++++++++++++++++++++++-
 7 files changed, 502 insertions(+), 104 deletions(-)

base-commit: 8d1b215c01512b9658d4525c54ff9b9245f9a7e8
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2287%2Frmistry%2Fhttp-exponential-retries-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2287/rmistry/http-exponential-retries-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/2287

Range-diff vs v3:

1: d53778dd46 ! 1: d4d3407728 http: fix retry count and Retry-After timing

     @@ t/lib-httpd/http-429.sh: repo_name="${repo_path%%/*}"
      -if test "$retry_after" != permanent && ! mkdir "$state" 2>/dev/null
      +max_fails=1
      +case "$test_context" in
     -+	*-fail-2*) max_fails=2 ;;
     -+	*-fail-3*) max_fails=3 ;;
     ++*-fail-2*) max_fails=2 ;;
     ++*-fail-3*) max_fails=3 ;;
      +esac
      +
      +should_fail=0
 2:  fe1e847ded ! 2:  fc5dbf8aea http: add exponential backoff without Retry-After
     @@ http.c: static int update_url_from_redirect(struct strbuf *base,
        * http.maxRetryTime), otherwise returns the delay in seconds (>= 0).
        */
      -static long handle_rate_limit_retry(long slot_retry_after)
     -+static long handle_rate_limit_retry(long slot_retry_after, int attempt)
     ++static long handle_rate_limit_retry(long slot_retry_after, size_t attempt)
       {
       	/* Use the slot-specific retry_after value or configured default */
       	if (slot_retry_after >= 0) {
     @@ http.c: static long handle_rate_limit_retry(long slot_retry_after)
       		trace2_data_string("http", the_repository,
       			"http/429-retry-source", "config-default");
      -		return http_retry_after;
     -+		for (int a = 1; a < attempt && delay > 0; a++) {
     ++		for (size_t a = 1; a < attempt && delay > 0; a++) {
      +			if (delay >= http_max_retry_time - delay) {
      +				delay = http_max_retry_time;
      +				break;
     @@ http.c: static int http_request_recoverable(const char *url,
      -			retry_delay = handle_rate_limit_retry(options->retry_after);
      +			retry_delay = handle_rate_limit_retry(
      +				options->retry_after,
     -+				http_max_retries - rate_limit_retries);
     ++				(size_t)(http_max_retries - rate_limit_retries));
       			if (retry_delay < 0)
       				return HTTP_ERROR;

3: 6510dddc39 ! 3: 8dc66bfca4 http: add http.retryStatusCodes config option
@@ Commit message

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) hardcoded HTTP 429 as the only status code eligible for
- retry. In practice, reverse proxies, load balancers, and Git hosting
- backends also return transient 5xx responses such as 502 Bad Gateway,
- 503 Service Unavailable (which RFC 9110 section 10.2.3 defines to
- support Retry-After), and 504 Gateway Timeout during load shedding or
- upstream connection resets.
+ retry. While service providers are expected to return 429 when
+ rate-limiting or overloaded, in practice reverse proxies, CDNs,
+ client-side proxies, and servers also return transient 502, 503, or 504
+ errors when a connection drops, times out, or sheds load.

- Because 5xx responses are not universally transient across all
- environments, keep 429 as the default and introduce a new
- http.retryStatusCodes configuration option (and corresponding
- GIT_HTTP_RETRY_STATUS_CODES environment variable) accepting a comma- or
- whitespace-separated list of HTTP status codes in the 400..599 range
- (for example, "429,502,503,504").
+ We should not retry 5xx responses unconditionally by default because in
+ some setups they point to a permanently broken proxy or server. Keep 429
+ as the default and introduce an http.retryStatusCodes configuration
+ option (and corresponding GIT_HTTP_RETRY_STATUS_CODES environment
+ variable) accepting a comma- or whitespace-separated list of HTTP status
+ codes in the 400..599 range (for example, "429,502,503,504"), documented
+ as a temporary workaround while underlying provider or proxy
+ infrastructure is fixed.

Keep the existing "rate limited" warning and error messages for HTTP 429, and report the numeric status code (e.g., "HTTP 502, waiting <N>
@@ Documentation/config/http.adoc: http.maxRetryTime::
+http.retryStatusCodes::
+ Comma- or whitespace-separated list of HTTP status codes (in the
+ 400..599 range) that should be retried up to http.maxRetries
-+ times. Defaults to 429 (Too Many Requests). For example, setting
-+ this to 429,502,503,504 also retries transient 5xx gateway and
-+ service-unavailable errors. Can be overridden by the
-+ GIT_HTTP_RETRY_STATUS_CODES environment variable.
++ times. Defaults to 429 (Too Many Requests). Can be overridden by
++ the GIT_HTTP_RETRY_STATUS_CODES environment variable.
+ See also http.maxRetries, http.retryAfter, and
+ http.maxRetryTime.
+++
++Service providers are expected to return HTTP 429 when rate-limiting or ++overloaded. Configuring additional status codes (for example, ++429,502,503,504) is intended only as a temporary workaround while the ++underlying provider or proxy infrastructure is fixed.

http.noEPSV::
A boolean which disables using of EPSV ftp command by curl.
@@ t/lib-httpd/http-429.sh: then
+# Output HTTP 429 (or configured status) response +status_line="429 Too Many Requests"
+case "$test_context" in
-+ *-status-502*) status_line="502 Bad Gateway" ;;
-+ *-status-503*) status_line="503 Service Unavailable" ;;
-+ *-status-504*) status_line="504 Gateway Timeout" ;;
++*-status-502*) status_line="502 Bad Gateway" ;; ++*-status-503*) status_line="503 Service Unavailable" ;; ++*-status-504*) status_line="504 Gateway Timeout" ;;
+esac
+printf "Status: %s\r\n" "$status_line"

4: 786964ff00 ! 4: ca247f390c remote-curl: retry transient HTTP errors in post_rpc()

     @@ Commit message
          429 or 5xx.
      
          Extract the retry wait logic from http_request_recoverable() into a
     -    shared http_retry_wait() helper and use it to retry transient HTTP
     -    errors in probe_rpc() and post_rpc(). Also defer reporting RPC errors
     -    until retries are exhausted, and re-attach the "Content-Encoding: gzip"
     -    header when retrying a compressed request.
     +    shared http_retry_wait() helper and use it in post_rpc() to retry
     +    transient HTTP errors for read-only services ("git-upload-pack" and
     +    "git-upload-archive"). Do not retry "git-receive-pack" (git push), since
     +    retrying a push POST after a 502 or 504 is not safe if the remote
     +    already processed the request. Also defer reporting RPC errors until
     +    retries are exhausted, and re-attach the "Content-Encoding: gzip" header
     +    when retrying a compressed request.
      
          Signed-off-by: Ravi Mistry <[email protected]>
      
     + ## Documentation/config/http.adoc ##
     +@@ Documentation/config/http.adoc: http.maxRetries::
     + 	Maximum number of times to retry after receiving a retryable HTTP
     + 	status code. The set of retryable status codes is configured by
     + 	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
     +-	Requests) by default. Set to 0 (the default) to disable retries.
     ++	Requests) by default. Retries apply to HTTP GET requests and
     ++	read-only smart-HTTP POST requests (`git-upload-pack` and
     ++	`git-upload-archive`). Set to 0 (the default) to disable retries.
     + 	Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
     + 	See also `http.retryAfter`, `http.maxRetryTime`, and
     + 	`http.retryStatusCodes`.
     +
       ## http.c ##
     -@@ http.c: static long handle_rate_limit_retry(long slot_retry_after, int attempt)
     +@@ http.c: static long handle_rate_limit_retry(long slot_retry_after, size_t attempt)
       	}
       }
       
     -+int http_retry_wait(long http_code, long slot_retry_after, int attempt)
     ++int http_retry_wait(long http_code, long slot_retry_after, size_t attempt)
      +{
      +	long retry_delay;
      +
      +	if (http_max_retries <= 0)
      +		return -1;
     -+	if (attempt > http_max_retries) {
     ++	if (attempt > (size_t)http_max_retries) {
      +		trace2_data_string("http", the_repository,
      +				   "http/429-error", "retries-exhausted");
      +		return -1;
     @@ http.c: static int http_request_recoverable(const char *url,
       	int i = 3;
       	int ret;
      -	int rate_limit_retries = http_max_retries;
     -+	int attempt = 0;
     ++	size_t attempt = 0;
       
       	if (!options)
       		options = &empty_opts;
     @@ http.c: static int http_request_recoverable(const char *url,
      -		if (ret == HTTP_RATE_LIMITED) {
      -			retry_delay = handle_rate_limit_retry(
      -				options->retry_after,
     --				http_max_retries - rate_limit_retries);
     +-				(size_t)(http_max_retries - rate_limit_retries));
      -			if (retry_delay < 0)
      -				return HTTP_ERROR;
      -
     @@ http.c: static int http_request_recoverable(const char *url,
      -		return options->http_code == 429 ? HTTP_RATE_LIMITED : HTTP_ERROR;
      -	}
      +	if (ret == HTTP_RATE_LIMITED &&
     -+	    (http_max_retries <= 0 || attempt <= http_max_retries ||
     ++	    (http_max_retries <= 0 || attempt <= (size_t)http_max_retries ||
      +	     options->http_code != 429))
      +		return HTTP_ERROR;
       	return ret;
     @@ http.h: int http_get_strbuf(const char *url, struct strbuf *result, struct http_
      + * Wait before retrying a transient HTTP error (1-based attempt count).
      + * Returns 0 if the caller should retry, or -1 to stop retrying.
      + */
     -+int http_retry_wait(long http_code, long slot_retry_after, int attempt);
     ++int http_retry_wait(long http_code, long slot_retry_after, size_t attempt);
      +
       int http_fetch_ref(const char *base, struct ref *ref);
       
     @@ remote-curl.c: static int probe_rpc(struct rpc_state *rpc, struct slot_results *
       
       	curl_slist_free_all(headers);
       	strbuf_release(&buf);
     + 	return err;
     + }
     + 
     ++static int can_retry_rpc(const struct rpc_state *rpc)
     ++{
     ++	return !strcmp(rpc->service_name, "git-upload-pack") ||
     ++	       !strcmp(rpc->service_name, "git-upload-archive");
     ++}
     ++
     + /*
     +  * If flush_received is true, do not attempt to read any more; just use what's
     +  * in rpc->buf.
      @@ remote-curl.c: static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)
       static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_received)
       {
     @@ remote-curl.c: static int probe_rpc(struct rpc_state *rpc, struct slot_results *
       	size_t gzip_size = 0;
       	int err, large_request = 0;
       	int needs_100_continue = 0;
     -+	int attempt = 0;
     ++	int can_retry = can_retry_rpc(rpc);
     ++	size_t attempt = 0;
       	struct rpc_in_data rpc_in_data;
       
       	/* Try to load the entire request, if we can fit it into the
     @@ remote-curl.c: static int post_rpc(struct rpc_state *rpc, int stateless_connect,
      -		} while (err == HTTP_REAUTH);
      -		if (err != HTTP_OK)
      +		} while (err == HTTP_REAUTH ||
     -+			 (err == HTTP_RATE_LIMITED &&
     ++			 (can_retry && err == HTTP_RATE_LIMITED &&
      +			  !http_retry_wait(results.http_code,
      +					   results.retry_after, ++attempt)));
      +		if (err != HTTP_OK) {
     @@ remote-curl.c: retry:
      +	err = run_one_slot(slot, &results);
      +	if (!large_request && !rpc->any_written &&
      +	    (err == HTTP_REAUTH ||
     -+	     (err == HTTP_RATE_LIMITED &&
     ++	     (can_retry && err == HTTP_RATE_LIMITED &&
      +	      !http_retry_wait(results.http_code, results.retry_after, ++attempt)))) {
      +		if (err == HTTP_REAUTH)
      +			http_reauth_prepare(0);
     @@ t/lib-httpd/http-429.sh: case "$test_context" in
       should_fail=0
      -if test "$retry_after" = permanent
      +case "$test_context,$REQUEST_METHOD" in
     -+	*-post-only*,POST)
     -+		can_fail=1
     -+		;;
     -+	*-post-only*,*)
     -+		can_fail=0
     -+		;;
     -+	*)
     -+		can_fail=1
     -+		;;
     ++*-post-only*,POST)
     ++	can_fail=1
     ++	;;
     ++*-post-only*,*)
     ++	can_fail=0
     ++	;;
     ++*)
     ++	can_fail=1
     ++	;;
      +esac
      +
      +if test $can_fail -eq 1
     @@ t/t5584-http-429-retry.sh: test_expect_success 'GIT_HTTP_RETRY_STATUS_CODES and
      +	grep "Content-Encoding: gzip" curl.log >gzip.headers &&
      +	test_line_count = 2 gzip.headers
      +'
     ++
     ++test_expect_success 'POST /git-receive-pack (git push) is not retried on 429 or 502' '
     ++	test_config http.maxRetries 3 &&
     ++	test_config http.retryAfter 1 &&
     ++	test_config http.retryStatusCodes "429,502" &&
     ++	test_must_fail git push \
     ++		"$HTTPD_URL/http_429/push-post-only-429/1/repo.git" \
     ++		HEAD:refs/heads/push-429 2>err &&
     ++	test_grep ! -i "waiting.*retry" err &&
     ++	test_grep "RPC failed; HTTP 429" err &&
     ++	test_must_fail git push \
     ++		"$HTTPD_URL/http_429/push-post-only-status-502/1/repo.git" \
     ++		HEAD:refs/heads/push-502 2>err &&
     ++	test_grep ! -i "waiting.*retry" err &&
     ++	test_grep "RPC failed; HTTP 502" err
     ++'
      +
       test_done

5: 61e3ffe9ab ! 5: 7d4335839d http: default http.retryAfter to 10 seconds
@@ Commit message
responsive. Callers or tests that want immediate retries can still set http.retryAfter=0 explicitly.

- Because http.retryAfter can now exceed a lowered http.maxRetryTime even
- when http.retryAfter itself was not explicitly configured, drop the word
- "configured" from the error message and config documentation. Also
- update the past HTTP-date test in t5584: CURLINFO_RETRY_AFTER only
- yields a positive delay for future timestamps, so past dates fall back
- to http.retryAfter (which previously waited 0s only because the default
- was 0).
+ Because libcurl's CURLINFO_RETRY_AFTER returns 0 when Retry-After is
+ absent or malformed and <= 0 when the server sends "Retry-After: 0" or a
+ past HTTP-date, inspect the Retry-After response header via
+ curl_easy_header() (enabled by default since libcurl 7.84.0) when
+ CURLINFO_RETRY_AFTER <= 0 so that an explicit "Retry-After: 0" or past
+ HTTP-date continues to retry immediately rather than falling back to the
+ new 10-second default.
+
+ Also drop the word "configured" from the http.maxRetryTime error message
+ and config documentation, since http.retryAfter can now exceed a lowered
+ http.maxRetryTime even when http.retryAfter was not explicitly set.

@@ Documentation/config/http.adoc: http.retryAfter::
+ (http.retryAfter * 2^(attempt - 1)), capped at
+ http.maxRetryTime. Set to 0 to retry immediately without waiting
+ on every attempt (unless a Retry-After header is present). When a
-+ Retry-After header with a positive delay is present, its value
-+ takes precedence over this setting; however, automatic use of the
-+ server-provided Retry-After header requires libcurl 7.66.0 or
-+ later. On older versions, Retry-After headers are ignored and
-+ http.retryAfter is always used. Can be overridden by the
++ Retry-After header is present, its value takes precedence over
++ this setting; however, automatic use of the server-provided
++ Retry-After header requires libcurl 7.66.0 or later (7.84.0 or
++ later for Retry-After: 0). On versions older than 7.66.0,
++ Retry-After headers are ignored and http.retryAfter is always
++ used. Can be overridden by the
GIT_HTTP_RETRY_AFTER environment variable.
See also http.maxRetries, http.maxRetryTime, and http.retryStatusCodes.
@@ Documentation/config/http.adoc: http.maxRetryTime::
backed-off delays computed from http.retryAfter, however, are capped at this maximum rather than causing an immediate failure.

+ ## git-curl-compat.h ##
+@@
+ #define GIT_CURL_HAVE_CURLINFO_RETRY_AFTER 1
+ #endif
+
++/**
++ * curl_easy_header() was added in 7.83.0 and enabled by default in 7.84.0, ++ * released in June 2022.
++ */
++#if LIBCURL_VERSION_NUM >= 0x075400
++#define GIT_CURL_HAVE_CURL_EASY_HEADER 1
++#endif
++
+ /**
+ * CURLOPT_PROTOCOLS_STR and CURLOPT_REDIR_PROTOCOLS_STR were added in 7.85.0,
+ * released in August 2022.
+
## http.c ##
@@ http.c: static char *http_ssl_backend;

@@ http.c: static char *http_ssl_backend;
static long http_max_retries = 0;
static long http_max_retry_time = 300;
static char *http_retry_status_codes;
-@@ http.c: static long handle_rate_limit_retry(long slot_retry_after, int attempt) +@@ http.c: static void closedown_active_slot(struct active_request_slot *slot)
+ slot->in_use = 0;
+ }
+
++#ifdef GIT_CURL_HAVE_CURL_EASY_HEADER
++static int has_zero_or_past_retry_after(CURL *curl)
++{
++ struct curl_header *hdr;
++ const char *p;
++
++ if (curl_easy_header(curl, "Retry-After", 0, CURLH_HEADER, -1,
++ &hdr) != CURLHE_OK)
++ return 0;
++ if (hdr->amount > 1 &&
++ curl_easy_header(curl, "Retry-After", hdr->amount - 1,
++ CURLH_HEADER, -1, &hdr) != CURLHE_OK)
++ return 0;
++
++ p = hdr->value;
++ while (isspace(*p))
++ p++;
++ if (!*p)
++ return 0;
++ if (curl_getdate(p, NULL) != (time_t)-1)
++ return 1;
++ if (*p != '0')
++ return 0;
++ while (*p == '0')
++ p++;
++ while (isspace(*p))
++ p++;
++ return !*p;
++}
++#endif
++
+ static void finish_active_slot(struct active_request_slot *slot)
+ {
+ closedown_active_slot(slot);
+@@ http.c: static void finish_active_slot(struct active_request_slot *slot)
+ {
+ curl_off_t retry_after;
+ if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
+- &retry_after) == CURLE_OK && retry_after > 0)
+- slot->results->retry_after = (long)retry_after;
++ &retry_after) == CURLE_OK) {
++ if (retry_after > 0)
++ slot->results->retry_after = (long)retry_after;
++ else if (retry_after < 0)
++ slot->results->retry_after = 0;
++#ifdef GIT_CURL_HAVE_CURL_EASY_HEADER
++ else if (slot->http_code >= 400 &&
++ has_zero_or_past_retry_after(slot->curl))
++ slot->results->retry_after = 0;
++#endif
++ }
+ }
+ #endif
+ }
+@@ http.c: static long handle_rate_limit_retry(long slot_retry_after, size_t attempt)

/* No Retry-After header provided, use configured default */ if (http_retry_after > http_max_retry_time) {
@@ http.c: static long handle_rate_limit_retry(long slot_retry_after, int attempt)
"http/429-error", "config-exceeds-max-retry-time");

## t/t5584-http-429-retry.sh ##
+@@ t/t5584-http-429-retry.sh: test_description='test HTTP 429 Too Many Requests retry logic' +
+ start_httpd
+
++test_lazy_prereq CURL_EASY_HEADER - <<\EOT
++ build_option libcurl |
++ awk -F. '
++ $1 > 7 || ($1 == 7 && $2 >= 84) { ok = 1 }
++ END { exit !ok }
++ '
++EOT
++
+ test_expect_success 'setup test repository' '
+ test_commit initial &&
+ git clone --bare . "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
@@ t/t5584-http-429-retry.sh: test_expect_success 'HTTP 429 with Retry-After is retried and succeeds' '

test_expect_success 'http.maxRetries=1 retries once and succeeds' '
@@ t/t5584-http-429-retry.sh: test_expect_success 'Trace2 logs actual CURLINFO_RETR
+ test_must_fail git ls-remote "$HTTPD_URL/http_429/default-retry-after-10/none/repo.git" 2>err &&
+ test_grep "http.retryAfter exceeds http.maxRetryTime (10 > 9 seconds)" err
+'
++
++test_expect_success CURL_EASY_HEADER 'HTTP 429 with Retry-After: 0 retries immediately' '
++ test_config http.maxRetries 3 &&
++ start=$(test-tool date getnanos) &&
++ GIT_TRACE2_EVENT="$(pwd)/trace-retry-zero.event" \
++ git ls-remote "$HTTPD_URL/http_429/retry-after-zero/0/repo.git" >output 2>err &&
++ duration=$(test-tool date getnanos $start) &&
++ duration_int=${duration%.*} &&
++ test "$duration_int" -lt 5 &&
++ test_grep "refs/heads/" output &&
++ test_grep ! -i "waiting.*retry" err &&
++ test_grep "\"key\":\"http/429-retry-after\",\"value\":\"0\"" trace-retry-zero.event
++'
+
test_expect_success 'HTTP 429 without Retry-After uses configured default' '
# Enable retries and configure default delay
@@ t/t5584-http-429-retry.sh: test_expect_success 'HTTP 429 fails if configured htt
'

test_expect_success 'HTTP 429 with Retry-After HTTP-date format' '
-@@ t/t5584-http-429-retry.sh: test_expect_success 'HTTP 429 with HTTP-date exceeding maxRetryTime fails immedi
- test_grep "http.maxRetryTime" err
- '
-
--test_expect_success 'HTTP 429 with past HTTP-date should not wait' '
-+test_expect_success 'HTTP 429 with past HTTP-date falls back to http.retryAfter' '
- raw=$(test-tool date timestamp now) &&
- now="${raw#* -> }" &&
- past_time=$((now - 10)) &&
-@@ t/t5584-http-429-retry.sh: test_expect_success 'HTTP 429 with past HTTP-date should not wait' '
- past_date="${raw#* -> }" &&
- past_date_encoded=$(echo "$past_date" | sed "s/ /%20/g") &&
-
-- # Enable retries
-+ # Enable retries with a short fallback delay
- test_config http.maxRetries 3 &&
-+ test_config http.retryAfter 1 &&
-
-- # Git should retry immediately without waiting
-+ # CURLINFO_RETRY_AFTER does not report a positive delay for past dates,
-+ # so Git falls back to http.retryAfter rather than retrying immediately
- start=$(test-tool date getnanos) &&
- git ls-remote "$HTTPD_URL/http_429/past-http-date/$past_date_encoded/repo.git" >output 2>err &&
- duration=$(test-tool date getnanos $start) &&
-
-- # Should complete quickly (no wait for a past-date Retry-After)
- duration_int=${duration%.*} &&
-+ test "$duration_int" -ge 1 &&
- test "$duration_int" -lt 5 &&
-- test_grep "refs/heads/" output
-+ test_grep "refs/heads/" output &&
-+ test_grep "waiting 1 seconds before retry" err
- '
-
- test_expect_success 'HTTP 429 with invalid Retry-After format uses configured default' ' @@ t/t5584-http-429-retry.sh: test_expect_success 'HTTP 503 honors Retry-After header when present' '

test_expect_success 'exhausted 502 retries report HTTP 502 error, not rate limited' '

Signature · 1 line
gitgitgadget
#44 lore raw

PATCH v4 1/5 http: fix retry count and Retry-After timing

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) introduced two small bugs in HTTP 429 retry handling:

1. In http_request_recoverable(), rate_limit_retries is initialized to http_max_retries (the number of retries to perform after the initial failed request) and decremented with pre-decrement (--rate_limit_retries) in the while condition. Because of the pre-decrement, setting http.maxRetries=1 performs 0 retries, and more generally N configured retries only retries N-1 times. In addition, if the first request returns 401 (HTTP_REAUTH) and the re-authenticated request returns 429 when http.maxRetries is 0, --rate_limit_retries wraps from 0 to -1 (truthy in C).

Fix this by checking http_max_retries > 0 && rate_limit_retries-- > 0 in the loop condition and checking http_max_retries <= 0 after the loop (which handles both an initial 429 and a 429 after 401 re-auth without logging "retries-exhausted", making the pre-loop check redundant).

2. In http_request(), CURLINFO_RETRY_AFTER is queried after run_one_slot() has already called handle_curl_result(), which logs results->retry_after to Trace2 ("http/429-retry-after") while it is still -1. Move the CURLINFO_RETRY_AFTER extraction into finish_active_slot() (initializing slot->results->retry_after = -1 first) so that both handle_curl_result() and callers of run_one_slot() observe the parsed Retry-After value.

---
 http.c                    | 27 +++++++++++++--------------
 t/lib-httpd/http-429.sh   | 31 ++++++++++++++++++++++++++-----
 t/t5584-http-429-retry.sh | 21 +++++++++++++++++++++
 3 files changed, 60 insertions(+), 19 deletions(-)
http.c +13 -14
index d79678d01f..0308f973a6 100644
@@ -345,11 +345,20 @@ static void finish_active_slot(struct active_request_slot *slot)
 	if (slot->results) {
 		slot->results->curl_result = slot->curl_result;
 		slot->results->http_code = slot->http_code;
+		slot->results->retry_after = -1;
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTPAUTH_AVAIL,
 				  &slot->results->auth_avail);
 
 		curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CONNECTCODE,
 			&slot->results->http_connectcode);
+#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
+		{
+			curl_off_t retry_after;
+			if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
+					      &retry_after) == CURLE_OK && retry_after > 0)
+				slot->results->retry_after = (long)retry_after;
+		}
+#endif
 	}
 
 	/* Run callback if appropriate */
@@ -2286,15 +2295,6 @@ static int http_request(const char *url,
 
 	ret = run_one_slot(slot, &results);
 
-#ifdef GIT_CURL_HAVE_CURLINFO_RETRY_AFTER
-	if (ret == HTTP_RATE_LIMITED) {
-		curl_off_t retry_after;
-		if (curl_easy_getinfo(slot->curl, CURLINFO_RETRY_AFTER,
-				      &retry_after) == CURLE_OK && retry_after > 0)
-			results.retry_after = (long)retry_after;
-	}
-#endif
-
 	options->retry_after = results.retry_after;
 
 	if (options->content_type) {
@@ -2461,10 +2461,6 @@ static int http_request_recoverable(const char *url,
 	if (ret != HTTP_OK && ret != HTTP_REAUTH && ret != HTTP_RATE_LIMITED)
 		return ret;
 
-	/* If retries are disabled and we got a 429, fail immediately */
-	if (ret == HTTP_RATE_LIMITED && !http_max_retries)
-		return HTTP_ERROR;
-
 	if (options->effective_url && options->base_url) {
 		if (update_url_from_redirect(options->base_url,
 					     url, options->effective_url)) {
@@ -2474,7 +2470,8 @@ static int http_request_recoverable(const char *url,
 	}
 
 	while ((ret == HTTP_REAUTH && --i) ||
-	       (ret == HTTP_RATE_LIMITED && --rate_limit_retries)) {
+	       (ret == HTTP_RATE_LIMITED && http_max_retries > 0 &&
+		rate_limit_retries-- > 0)) {
 		long retry_delay = -1;
 		/*
 		 * The previous request may have put cruft into our output stream; we
@@ -2518,6 +2515,8 @@ static int http_request_recoverable(const char *url,
 		ret = http_request(url, result, target, options);
 	}
 	if (ret == HTTP_RATE_LIMITED) {
+		if (http_max_retries <= 0)
+			return HTTP_ERROR;
 		trace2_data_string("http", the_repository,
 				   "http/429-error", "retries-exhausted");
 		return HTTP_RATE_LIMITED;
t/lib-httpd/http-429.sh +26 -5
index 1a5d7987db..bd06d94c16 100644
@@ -32,12 +32,33 @@ repo_name="${repo_path%%/*}"
 safe_name=$(echo "${test_context}-${retry_after}-${repo_name}" | tr '/' '_' | tr -cd 'a-zA-Z0-9_-')
 state="http-429-state-${safe_name}"
 
-# Check if this is the first call (no state directory exists), or if
-# the retry-after-value is "permanent", which indicates a 429 must be
-# returned for every request (even if the state directory exists).
-if test "$retry_after" != permanent && ! mkdir "$state" 2>/dev/null
+max_fails=1
+case "$test_context" in
+*-fail-2*) max_fails=2 ;;
+*-fail-3*) max_fails=3 ;;
+esac
+
+should_fail=0
+if test "$retry_after" = permanent
+then
+	should_fail=1
+else
+	mkdir -p "$state" 2>/dev/null
+	i=1
+	while test $i -le $max_fails
+	do
+		if mkdir "$state/$i" 2>/dev/null
+		then
+			should_fail=1
+			break
+		fi
+		i=$((i + 1))
+	done
+fi
+
+if test $should_fail -eq 0
 then
-	# Already returned 429 once, forward to git-http-backend
+	# Already returned 429 max_fails times, forward to git-http-backend
 	# Set PATH_INFO to just the repo path (without retry-after value)
 	# Set GIT_PROJECT_ROOT so git-http-backend can find the repository
 	# Use exec to replace this process so git-http-backend gets the updated environment
t/t5584-http-429-retry.sh +21 -0
index a22007b2cf..d92a26893d 100755
@@ -51,6 +51,27 @@ test_expect_success 'HTTP 429 with Retry-After is retried and succeeds' '
 	test_grep "refs/heads/" output
 '
 
+test_expect_success 'http.maxRetries=1 retries once and succeeds' '
+	test_config http.maxRetries 1 &&
+	git ls-remote "$HTTPD_URL/http_429/max-retries-one/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output
+'
+
+test_expect_success 'http.maxRetries=2 fails after 3 failures and succeeds on 2 failures' '
+	test_config http.maxRetries 2 &&
+	test_must_fail git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-3/none/repo.git" 2>err &&
+	test_grep "rate limited" err &&
+	git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-2/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output
+'
+
+test_expect_success 'Trace2 logs actual CURLINFO_RETRY_AFTER value instead of -1' '
+	test_config http.maxRetries 1 &&
+	GIT_TRACE2_EVENT="$(pwd)/trace-retry-after.event" \
+		git ls-remote "$HTTPD_URL/http_429/trace2-retry-after/1/repo.git" >output 2>err &&
+	test_grep "\"key\":\"http/429-retry-after\",\"value\":\"1\"" trace-retry-after.event
+'
+
 test_expect_success 'HTTP 429 without Retry-After uses configured default' '
 	# Enable retries and configure default delay
 	test_config http.maxRetries 3 &&
Signature · 1 line
gitgitgadget
#45 lore raw

PATCH v4 2/5 http: add exponential backoff without Retry-After

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

When a server returns HTTP 429 without a Retry-After header, Git waits the same fixed http.retryAfter delay on every retry. If that delay is short (e.g., 1s), Git burns through all of its retries in just a few seconds before a busy server can recover. If that delay is long, even a brief glitch has to wait a long time on the first retry.

Pass the 1-based retry attempt count to handle_rate_limit_retry() and double http.retryAfter on each subsequent retry (1s, 2s, 4s, ..., via http.retryAfter * 2^(attempt - 1)), clamping the delay at http.maxRetryTime. This keeps the first retry fast while giving the server more time to recover on later attempts, matching curl's own --retry behavior (while keeping http.retryAfter's existing default of 0 for immediate retries when unconfigured). When the server provides an explicit Retry-After header, continue to honor the server-provided delay (or fail fast if it exceeds http.maxRetryTime).

---
 Documentation/config/http.adoc | 20 ++++++++++++++------
 http.c                         | 17 ++++++++++++++---
 t/t5584-http-429-retry.sh      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 9 deletions(-)
Documentation/config/http.adoc +14 -6
index 792a71b413..8e7538ab76 100644
@@ -327,9 +327,14 @@ http.keepAliveCount::
 	`GIT_HTTP_KEEPALIVE_COUNT` environment variable.
 
 http.retryAfter::
-	Default wait time in seconds before retrying when a server returns
+	Initial wait time in seconds before retrying when a server returns
 	HTTP 429 (Too Many Requests) without a Retry-After header.
-	Defaults to 0 (retry immediately). When a Retry-After header is
+	Defaults to 0, which retries immediately without waiting on every
+	attempt (unless a Retry-After header is present). When set to a
+	positive value and no Retry-After header is present, the first
+	retry waits `http.retryAfter` seconds and the delay doubles on
+	each subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
+	capped at `http.maxRetryTime`. When a Retry-After header is
 	present, its value takes precedence over this setting; however,
 	automatic use of the server-provided `Retry-After` header requires
 	libcurl 7.66.0 or later. On older versions, configure this setting
@@ -345,10 +350,13 @@ http.maxRetries::
 
 http.maxRetryTime::
 	Maximum time in seconds to wait for a single retry attempt when
-	handling HTTP 429 (Too Many Requests) responses. If the server
-	requests a delay (via Retry-After header) or if `http.retryAfter`
-	is configured with a value that exceeds this maximum, Git will fail
-	immediately rather than waiting. Default is 300 seconds (5 minutes).
+	handling HTTP 429 (Too Many Requests) responses.
+	If the server requests a delay (via Retry-After header) or if the
+	configured `http.retryAfter` value itself exceeds this maximum,
+	Git will fail immediately rather than waiting. Exponentially
+	backed-off delays computed from `http.retryAfter`, however, are
+	capped at this maximum rather than causing an immediate failure.
+	Default is 300 seconds (5 minutes).
 	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
 	variable. See also `http.retryAfter` and `http.maxRetries`.
 
http.c +14 -3
index 0308f973a6..b766bbf035 100644
@@ -2411,7 +2411,7 @@ static int update_url_from_redirect(struct strbuf *base,
  * Returns a negative value if configuration is invalid (delay exceeds
  * http.maxRetryTime), otherwise returns the delay in seconds (>= 0).
  */
-static long handle_rate_limit_retry(long slot_retry_after)
+static long handle_rate_limit_retry(long slot_retry_after, size_t attempt)
 {
 	/* Use the slot-specific retry_after value or configured default */
 	if (slot_retry_after >= 0) {
@@ -2427,6 +2427,8 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		return slot_retry_after;
 	} else {
+		long delay = http_retry_after;
+
 		/* No Retry-After header provided, use configured default */
 		if (http_retry_after > http_max_retry_time) {
 			error(_("configured http.retryAfter exceeds http.maxRetryTime (%ld > %ld seconds)"),
@@ -2437,7 +2439,14 @@ static long handle_rate_limit_retry(long slot_retry_after)
 		}
 		trace2_data_string("http", the_repository,
 			"http/429-retry-source", "config-default");
-		return http_retry_after;
+		for (size_t a = 1; a < attempt && delay > 0; a++) {
+			if (delay >= http_max_retry_time - delay) {
+				delay = http_max_retry_time;
+				break;
+			}
+			delay *= 2;
+		}
+		return delay;
 	}
 }
 
@@ -2498,7 +2507,9 @@ static int http_request_recoverable(const char *url,
 			BUG("Unknown http_request target");
 		}
 		if (ret == HTTP_RATE_LIMITED) {
-			retry_delay = handle_rate_limit_retry(options->retry_after);
+			retry_delay = handle_rate_limit_retry(
+				options->retry_after,
+				(size_t)(http_max_retries - rate_limit_retries));
 			if (retry_delay < 0)
 				return HTTP_ERROR;
 
t/t5584-http-429-retry.sh +33 -0
index d92a26893d..f0c3cc3b61 100755
@@ -82,6 +82,39 @@ test_expect_success 'HTTP 429 without Retry-After uses configured default' '
 	test_grep "refs/heads/" output
 '
 
+test_expect_success 'exponential backoff doubles http.retryAfter without Retry-After header' '
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
+	git ls-remote "$HTTPD_URL/http_429/exp-backoff-fail-3/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep "waiting 2 seconds before retry" err &&
+	test_grep "waiting 4 seconds before retry" err
+'
+
+test_expect_success 'exponential backoff clamps delay at http.maxRetryTime' '
+	test_config http.maxRetries 3 &&
+	test_config http.retryAfter 1 &&
+	test_config http.maxRetryTime 2 &&
+	GIT_TRACE2_EVENT="$(pwd)/trace-clamp.event" \
+		git ls-remote "$HTTPD_URL/http_429/exp-clamp-fail-3/none/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep "waiting 2 seconds before retry" err &&
+	test_grep ! "waiting 4 seconds before retry" err &&
+	grep "\"key\":\"http/retry-sleep-seconds\",\"value\":\"2\"" trace-clamp.event >clamped &&
+	test_line_count = 2 clamped
+'
+
+test_expect_success 'server Retry-After header overrides exponential backoff' '
+	test_config http.maxRetries 2 &&
+	test_config http.retryAfter 1 &&
+	git ls-remote "$HTTPD_URL/http_429/header-override-fail-2/1/repo.git" >output 2>err &&
+	test_grep "refs/heads/" output &&
+	test_grep "waiting 1 seconds before retry" err &&
+	test_grep ! "waiting 2 seconds before retry" err
+'
+
 test_expect_success 'HTTP 429 retry delays are respected' '
 	# Enable retries
 	test_config http.maxRetries 3 &&
Signature · 1 line
gitgitgadget
#46 lore raw

PATCH v4 3/5 http: add http.retryStatusCodes config option

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Commit 640657ffd0 (http: add support for HTTP 429 rate limit retries, 2026-03-17) hardcoded HTTP 429 as the only status code eligible for retry. While service providers are expected to return 429 when rate-limiting or overloaded, in practice reverse proxies, CDNs, client-side proxies, and servers also return transient 502, 503, or 504 errors when a connection drops, times out, or sheds load.

We should not retry 5xx responses unconditionally by default because in some setups they point to a permanently broken proxy or server. Keep 429 as the default and introduce an http.retryStatusCodes configuration option (and corresponding GIT_HTTP_RETRY_STATUS_CODES environment variable) accepting a comma- or whitespace-separated list of HTTP status codes in the 400..599 range (for example, "429,502,503,504"), documented as a temporary workaround while underlying provider or proxy infrastructure is fixed.

Keep the existing "rate limited" warning and error messages for HTTP 429, and report the numeric status code (e.g., "HTTP 502, waiting <N> seconds before retry") when retrying other status codes.

---
 Documentation/config/http.adoc | 37 +++++++++++++++++++++-----
 http.c                         | 42 ++++++++++++++++++++++++++---
 http.h                         |  6 +++--
 t/lib-httpd/http-429.sh        | 10 +++++--
 t/t5584-http-429-retry.sh      | 48 ++++++++++++++++++++++++++++++++++
 5 files changed, 128 insertions(+), 15 deletions(-)
Documentation/config/http.adoc +30 -7
index 8e7538ab76..726669e85c 100644
@@ -328,7 +328,10 @@ http.keepAliveCount::
 
 http.retryAfter::
 	Initial wait time in seconds before retrying when a server returns
-	HTTP 429 (Too Many Requests) without a Retry-After header.
+	a retryable HTTP status code without a Retry-After header. The
+	set of retryable status codes is configured by
+	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
+	Requests) by default.
 	Defaults to 0, which retries immediately without waiting on every
 	attempt (unless a Retry-After header is present). When set to a
 	positive value and no Retry-After header is present, the first
@@ -340,17 +343,23 @@ http.retryAfter::
 	libcurl 7.66.0 or later. On older versions, configure this setting
 	manually to control the retry delay. Can be overridden by the
 	`GIT_HTTP_RETRY_AFTER` environment variable.
-	See also `http.maxRetries` and `http.maxRetryTime`.
+	See also `http.maxRetries`, `http.maxRetryTime`, and
+	`http.retryStatusCodes`.
 
 http.maxRetries::
-	Maximum number of times to retry after receiving HTTP 429 (Too Many
-	Requests) responses. Set to 0 (the default) to disable retries.
+	Maximum number of times to retry after receiving a retryable HTTP
+	status code. The set of retryable status codes is configured by
+	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
+	Requests) by default. Set to 0 (the default) to disable retries.
 	Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
-	See also `http.retryAfter` and `http.maxRetryTime`.
+	See also `http.retryAfter`, `http.maxRetryTime`, and
+	`http.retryStatusCodes`.
 
 http.maxRetryTime::
 	Maximum time in seconds to wait for a single retry attempt when
-	handling HTTP 429 (Too Many Requests) responses.
+	handling a retryable HTTP status code. The set of retryable
+	status codes is configured by `http.retryStatusCodes`, which
+	includes only HTTP 429 (Too Many Requests) by default.
 	If the server requests a delay (via Retry-After header) or if the
 	configured `http.retryAfter` value itself exceeds this maximum,
 	Git will fail immediately rather than waiting. Exponentially
@@ -358,7 +367,21 @@ http.maxRetryTime::
 	capped at this maximum rather than causing an immediate failure.
 	Default is 300 seconds (5 minutes).
 	Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
-	variable. See also `http.retryAfter` and `http.maxRetries`.
+	variable. See also `http.retryAfter`, `http.maxRetries`, and
+	`http.retryStatusCodes`.
+
+http.retryStatusCodes::
+	Comma- or whitespace-separated list of HTTP status codes (in the
+	400..599 range) that should be retried up to `http.maxRetries`
+	times. Defaults to `429` (Too Many Requests). Can be overridden by
+	the `GIT_HTTP_RETRY_STATUS_CODES` environment variable.
+	See also `http.maxRetries`, `http.retryAfter`, and
+	`http.maxRetryTime`.
++
+Service providers are expected to return HTTP 429 when rate-limiting or
+overloaded. Configuring additional status codes (for example,
+`429,502,503,504`) is intended only as a temporary workaround while the
+underlying provider or proxy infrastructure is fixed.
 
 http.noEPSV::
 	A boolean which disables using of EPSV ftp command by curl.
http.c +38 -4

102 lines not on this page · raw message

http.h +4 -2
index 729c51904d..5985a0f71f 100644
@@ -167,10 +167,12 @@ struct http_get_options {
 
 	/*
 	 * After a request completes, contains the Retry-After delay in seconds
-	 * if the server returned HTTP 429 with a Retry-After header (requires
-	 * libcurl 7.66.0 or later), or -1 if no such header was present.
+	 * if the server returned a retryable status code with a Retry-After
+	 * header (requires libcurl 7.66.0 or later), or -1 if no such header
+	 * was present.
 	 */
 	long retry_after;
+	long http_code;
 };
 
 /* Return values for http_get_*() */
t/lib-httpd/http-429.sh +8 -2
index bd06d94c16..d42feae8ba 100644
@@ -75,8 +75,14 @@ then
 	exec "$GIT_EXEC_PATH/git-http-backend"
 fi
 
-# Output HTTP 429 response
-printf "Status: 429 Too Many Requests\r\n"
+# Output HTTP 429 (or configured status) response
+status_line="429 Too Many Requests"
+case "$test_context" in
+*-status-502*) status_line="502 Bad Gateway" ;;
+*-status-503*) status_line="503 Service Unavailable" ;;
+*-status-504*) status_line="504 Gateway Timeout" ;;
+esac
+printf "Status: %s\r\n" "$status_line"
 
 # Set Retry-After header based on retry_after value
 case "$retry_after" in
t/t5584-http-429-retry.sh +48 -0

52 lines not on this page · raw message

Signature · 1 line
gitgitgadget
#47 lore raw

PATCH v4 4/5 remote-curl: retry transient HTTP errors in post_rpc()

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

Git's existing retry logic only covers HTTP GET requests. However, the actual data transfer during fetch and clone, as well as on-demand blob downloads in partial clones, are sent as HTTP POST requests via post_rpc(), which only retries authentication and immediately aborts on 429 or 5xx.

Extract the retry wait logic from http_request_recoverable() into a shared http_retry_wait() helper and use it in post_rpc() to retry transient HTTP errors for read-only services ("git-upload-pack" and "git-upload-archive"). Do not retry "git-receive-pack" (git push), since retrying a push POST after a 502 or 504 is not safe if the remote already processed the request. Also defer reporting RPC errors until retries are exhausted, and re-attach the "Content-Encoding: gzip" header when retrying a compressed request.

---
 Documentation/config/http.adoc |  4 +-
 http.c                         | 67 ++++++++++++++-------------
 http.h                         |  6 +++
 remote-curl.c                  | 82 ++++++++++++++++++----------------
 t/lib-httpd/http-429.sh        | 50 +++++++++++++++------
 t/t5584-http-429-retry.sh      | 69 ++++++++++++++++++++++++++++
 6 files changed, 194 insertions(+), 84 deletions(-)
Documentation/config/http.adoc +3 -1
index 726669e85c..72e9bf011d 100644
@@ -350,7 +350,9 @@ http.maxRetries::
 	Maximum number of times to retry after receiving a retryable HTTP
 	status code. The set of retryable status codes is configured by
 	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
-	Requests) by default. Set to 0 (the default) to disable retries.
+	Requests) by default. Retries apply to HTTP GET requests and
+	read-only smart-HTTP POST requests (`git-upload-pack` and
+	`git-upload-archive`). Set to 0 (the default) to disable retries.
 	Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
 	See also `http.retryAfter`, `http.maxRetryTime`, and
 	`http.retryStatusCodes`.
http.c +36 -31

95 lines not on this page · raw message

http.h +6 -0
index 5985a0f71f..ec96b6d2c7 100644
@@ -201,6 +201,12 @@ int http_get_strbuf(const char *url, struct strbuf *result, struct http_get_opti
 int http_get_file(const char *url, const char *filename,
 		  struct http_get_options *options);
 
+/*
+ * Wait before retrying a transient HTTP error (1-based attempt count).
+ * Returns 0 if the caller should retry, or -1 to stop retrying.
+ */
+int http_retry_wait(long http_code, long slot_retry_after, size_t attempt);
+
 int http_fetch_ref(const char *base, struct ref *ref);
 
 struct curl_slist *http_append_auth_header(const struct credential *c,
remote-curl.c +44 -38

143 lines not on this page · raw message

t/lib-httpd/http-429.sh +36 -14

63 lines not on this page · raw message

t/t5584-http-429-retry.sh +69 -0

73 lines not on this page · raw message

Signature · 1 line
gitgitgadget
#48 lore raw

PATCH v4 5/5 http: default http.retryAfter to 10 seconds

In reply to Ravi Mistry via GitGitGadget

From: Ravi Mistry <[email protected]>

When http.maxRetries is enabled and a server returns HTTP 429 (or an opt-in status code via http.retryStatusCodes) without a Retry-After header, http.retryAfter currently defaults to 0. That causes Git to retry immediately in a tight loop on every attempt, defeating exponential backoff and continuing to hammer a rate-limiting server right after it asked the client to back off.

Default http.retryAfter to 10 seconds (backing off as 10s, 20s, 40s, ..., capped at http.maxRetryTime) so that overloaded Git servers get meaningful breathing room while keeping interactive commands reasonably responsive. Callers or tests that want immediate retries can still set http.retryAfter=0 explicitly.

Because libcurl's CURLINFO_RETRY_AFTER returns 0 when Retry-After is absent or malformed and <= 0 when the server sends "Retry-After: 0" or a past HTTP-date, inspect the Retry-After response header via curl_easy_header() (enabled by default since libcurl 7.84.0) when CURLINFO_RETRY_AFTER <= 0 so that an explicit "Retry-After: 0" or past HTTP-date continues to retry immediately rather than falling back to the new 10-second default.

Also drop the word "configured" from the http.maxRetryTime error message and config documentation, since http.retryAfter can now exceed a lowered http.maxRetryTime even when http.retryAfter was not explicitly set.

---
 Documentation/config/http.adoc | 26 +++++++++---------
 git-curl-compat.h              |  8 ++++++
 http.c                         | 48 +++++++++++++++++++++++++++++++---
 t/t5584-http-429-retry.sh      | 34 +++++++++++++++++++++++-
 4 files changed, 99 insertions(+), 17 deletions(-)
Documentation/config/http.adoc +14 -12
index 72e9bf011d..051c29d7c4 100644
@@ -332,16 +332,18 @@ http.retryAfter::
 	set of retryable status codes is configured by
 	`http.retryStatusCodes`, which includes only HTTP 429 (Too Many
 	Requests) by default.
-	Defaults to 0, which retries immediately without waiting on every
-	attempt (unless a Retry-After header is present). When set to a
-	positive value and no Retry-After header is present, the first
-	retry waits `http.retryAfter` seconds and the delay doubles on
-	each subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`),
-	capped at `http.maxRetryTime`. When a Retry-After header is
-	present, its value takes precedence over this setting; however,
-	automatic use of the server-provided `Retry-After` header requires
-	libcurl 7.66.0 or later. On older versions, configure this setting
-	manually to control the retry delay. Can be overridden by the
+	Defaults to 10. When positive and no Retry-After header is
+	present, the first retry waits `http.retryAfter` seconds and the
+	delay doubles on each subsequent retry attempt
+	(`http.retryAfter * 2^(attempt - 1)`), capped at
+	`http.maxRetryTime`. Set to 0 to retry immediately without waiting
+	on every attempt (unless a Retry-After header is present). When a
+	Retry-After header is present, its value takes precedence over
+	this setting; however, automatic use of the server-provided
+	`Retry-After` header requires libcurl 7.66.0 or later (7.84.0 or
+	later for `Retry-After: 0`). On versions older than 7.66.0,
+	`Retry-After` headers are ignored and `http.retryAfter` is always
+	used. Can be overridden by the
 	`GIT_HTTP_RETRY_AFTER` environment variable.
 	See also `http.maxRetries`, `http.maxRetryTime`, and
 	`http.retryStatusCodes`.
@@ -362,8 +364,8 @@ http.maxRetryTime::
 	handling a retryable HTTP status code. The set of retryable
 	status codes is configured by `http.retryStatusCodes`, which
 	includes only HTTP 429 (Too Many Requests) by default.
-	If the server requests a delay (via Retry-After header) or if the
-	configured `http.retryAfter` value itself exceeds this maximum,
+	If the server requests a delay (via Retry-After header) or if
+	`http.retryAfter` itself exceeds this maximum,
 	Git will fail immediately rather than waiting. Exponentially
 	backed-off delays computed from `http.retryAfter`, however, are
 	capped at this maximum rather than causing an immediate failure.
git-curl-compat.h +8 -0
index 25678b5dbd..c72a2a766c 100644
@@ -52,6 +52,14 @@
 #define GIT_CURL_HAVE_CURLINFO_RETRY_AFTER 1
 #endif
 
+/**
+ * curl_easy_header() was added in 7.83.0 and enabled by default in 7.84.0,
+ * released in June 2022.
+ */
+#if LIBCURL_VERSION_NUM >= 0x075400
+#define GIT_CURL_HAVE_CURL_EASY_HEADER 1
+#endif
+
 /**
  * CURLOPT_PROTOCOLS_STR and CURLOPT_REDIR_PROTOCOLS_STR were added in 7.85.0,
  * released in August 2022.
http.c +44 -4

72 lines not on this page · raw message

t/t5584-http-429-retry.sh +33 -1
index afdea04b4e..43ed9b0de1 100755
@@ -8,6 +8,14 @@ test_description='test HTTP 429 Too Many Requests retry logic'
 
 start_httpd
 
+test_lazy_prereq CURL_EASY_HEADER - <<\EOT
+	build_option libcurl |
+	awk -F. '
+		$1 > 7 || ($1 == 7 && $2 >= 84) { ok = 1 }
+		END { exit !ok }
+	'
+EOT
+
 test_expect_success 'setup test repository' '
 	test_commit initial &&
 	git clone --bare . "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
@@ -53,12 +61,14 @@ test_expect_success 'HTTP 429 with Retry-After is retried and succeeds' '
 
 test_expect_success 'http.maxRetries=1 retries once and succeeds' '
 	test_config http.maxRetries 1 &&
+	test_config http.retryAfter 0 &&
 	git ls-remote "$HTTPD_URL/http_429/max-retries-one/none/repo.git" >output 2>err &&
 	test_grep "refs/heads/" output
 '
 
 test_expect_success 'http.maxRetries=2 fails after 3 failures and succeeds on 2 failures' '
 	test_config http.maxRetries 2 &&
+	test_config http.retryAfter 0 &&
 	test_must_fail git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-3/none/repo.git" 2>err &&
 	test_grep "rate limited" err &&
 	git ls-remote "$HTTPD_URL/http_429/max-retries-two-fail-2/none/repo.git" >output 2>err &&
@@ -72,6 +82,26 @@ test_expect_success 'Trace2 logs actual CURLINFO_RETRY_AFTER value instead of -1
 	test_grep "\"key\":\"http/429-retry-after\",\"value\":\"1\"" trace-retry-after.event
 '
 
+test_expect_success 'http.retryAfter defaults to 10 seconds when unset' '
+	test_config http.maxRetries 1 &&
+	test_config http.maxRetryTime 9 &&
+	test_must_fail git ls-remote "$HTTPD_URL/http_429/default-retry-after-10/none/repo.git" 2>err &&
+	test_grep "http.retryAfter exceeds http.maxRetryTime (10 > 9 seconds)" err
+'
+
+test_expect_success CURL_EASY_HEADER 'HTTP 429 with Retry-After: 0 retries immediately' '
+	test_config http.maxRetries 3 &&
+	start=$(test-tool date getnanos) &&
+	GIT_TRACE2_EVENT="$(pwd)/trace-retry-zero.event" \
+		git ls-remote "$HTTPD_URL/http_429/retry-after-zero/0/repo.git" >output 2>err &&
+	duration=$(test-tool date getnanos $start) &&
+	duration_int=${duration%.*} &&
+	test "$duration_int" -lt 5 &&
+	test_grep "refs/heads/" output &&
+	test_grep ! -i "waiting.*retry" err &&
+	test_grep "\"key\":\"http/429-retry-after\",\"value\":\"0\"" trace-retry-zero.event
+'
+
 test_expect_success 'HTTP 429 without Retry-After uses configured default' '
 	# Enable retries and configure default delay
 	test_config http.maxRetries 3 &&
@@ -161,7 +191,7 @@ test_expect_success 'HTTP 429 fails if configured http.retryAfter exceeds http.m
 	# Should fail quickly (no 100 second wait)
 	duration_int=${duration%.*} &&
 	test "$duration_int" -lt 99 &&
-	test_grep "configured http.retryAfter.*exceeds.*http.maxRetryTime" err
+	test_grep "http.retryAfter.*exceeds.*http.maxRetryTime" err
 '
 
 test_expect_success 'HTTP 429 with Retry-After HTTP-date format' '
@@ -349,6 +379,7 @@ test_expect_success 'HTTP 503 honors Retry-After header when present' '
 
 test_expect_success 'exhausted 502 retries report HTTP 502 error, not rate limited' '
 	test_config http.maxRetries 2 &&
+	test_config http.retryAfter 0 &&
 	test_config http.retryStatusCodes "429,502" &&
 	test_must_fail git ls-remote "$HTTPD_URL/http_429/exhausted-fail-3-status-502/none/repo.git" 2>err &&
 	test_grep "The requested URL returned error: 502" err &&
@@ -357,6 +388,7 @@ test_expect_success 'exhausted 502 retries report HTTP 502 error, not rate limit
 
 test_expect_success 'GIT_HTTP_RETRY_STATUS_CODES and urlmatch http.<url>.retryStatusCodes' '
 	test_config http.maxRetries 2 &&
+	test_config http.retryAfter 0 &&
 	GIT_HTTP_RETRY_STATUS_CODES="429,502" \
 		git ls-remote "$HTTPD_URL/http_429/env-status-502/none/repo.git" >output 2>err &&
 	test_grep "refs/heads/" output &&
Signature · 1 line
gitgitgadget