packfile URIs: support concurrent downloads
cover packfile URIs: support concurrent downloads
Packfile URI downloads currently stage a pack at objects/pack/pack-<hash>.pack.temp. Two Git processes fetching the same pack into one object database can append to that file concurrently, which can corrupt the temporary pack or cause a resume request at EOF.
The first patch gives each direct packfile URI download a private temporary file. Ordinary dumb HTTP pack requests retain their existing resumable staging behavior. A later packfile URI retry starts a new download.
The second patch handles the related .keep race. When another process has already created the keep file, index-pack reports "pack<TAB><hash>" instead of "keep<TAB><hash>". Accept both successful forms and remove only keep files created by the current process.
Each patch adds a regression test for its respective race.
Ted Nyman (2):
http: use unique tempfiles for packfile URI downloads
fetch-pack: accept "pack" output for packfile URIs
Documentation/git-http-fetch.adoc | 5 +- fetch-pack.c | 36 ++++++++------- http.c | 77 +++++++++++++++++++++---------- http.h | 1 + t/t5550-http-fetch-dumb.sh | 72 ++++++++++++++++++++++++++++- t/t5702-protocol-v2.sh | 31 +++++++++++++ 6 files changed, 177 insertions(+), 45 deletions(-)
base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc
Signature · 1 line
2.55.0
Quoted 15 lines · Ted Nyman
Packfile URI downloads currently stage a pack at objects/pack/pack-<hash>.pack.temp. Two Git processes fetching the same pack into one object database can append to that file concurrently, which can corrupt the temporary pack or cause a resume request at EOF.
The first patch gives each direct packfile URI download a private temporary file. Ordinary dumb HTTP pack requests retain their existing resumable staging behavior. A later packfile URI retry starts a new download.
The second patch handles the related .keep race. When another process has already created the keep file, index-pack reports "pack<TAB><hash>" instead of "keep<TAB><hash>". Accept both successful forms and remove only keep files created by the current process.
Quoted 5 lines
Each patch adds a regression test for its respective race.
Ted Nyman (2):
http: use unique tempfiles for packfile URI downloads
fetch-pack: accept "pack" output for packfile URIs
This cover letter has
Message-ID: <alVn7UWvdWRAG-Vv@com-76773>
but in the header of [PATCH 1/2] has
Message-ID: <alVn-QmK3K91_tkH@com-76773>
References: <[email protected]>
In-Reply-To: <[email protected]>Similarly, [PATCH 2/2] has
Message-ID: <alVoA5-fDDPwKPZZ@com-76773>
References: <[email protected]>
In-Reply-To: <[email protected]>And "b4 am" seems to be having problem grabbing the patchset X-<.
Quoted 1 line
And "b4 am" seems to be having problem grabbing the patchset X-<.
Sorry for the noise -- Mutt rewrote the original cover-letter Message-ID. The patches reference the corrected cover:
https://lore.kernel.org/git/[email protected]/
I confirmed that this retrieves both patches:
b4 am [email protected]
Thanks,
Ted
Quoted 10 lines · Ted Nyman
Quoted 1 line
And "b4 am" seems to be having problem grabbing the patchset X-<.
Sorry for the noise -- Mutt rewrote the original cover-letter Message-ID. The patches reference the corrected cover:
https://lore.kernel.org/git/[email protected]/
I confirmed that this retrieves both patches:
b4 am [email protected]
This is a mistake on my end as I was porting over some of the scripts for sending patches to the mailing list to OpenAI's infrastructure.
The short version of this e-mail is that the issue is fixed. But the longer version is funny (at least to me), so I figured I would share.
As some background, my workflow for sending patches to the mailing list is to use a script called 'git mail' that effectively runs format-patch to build an *.mbox and then opens Mutt in that directory. I then review the patches one last time before sending, and then run a macro I have bound to 'B', which (effectively) runs <resend-message>.
For reasons that I cannot quite recall, I chose this workflow many years ago when it would likely have been more appropriate to use mutt -H, which does *not* rewrite Message-ID headers when resending.
To work around this, I wrote a patch that I applied to the version of Mutt I used both on my old work laptop as well as the Linux workstation where I did the majority of my work. The patch is fairly small, and is effectively:
--- 8< ---
postpone.c +1 -5
@@ -607,13 +607,9 @@ int mutt_prepare_template (FILE *fp, CONTEXT *ctx, HEADER *newhdr, HEADER *hdr, newhdr->content->length = hdr->content->length;mutt_parse_part (fp, newhdr->content); -/* If resending a message, don't keep message_id or mail_followup_to. -* Otherwise, we are resuming a postponed message, and want to keep those -* headers if they exist. -*/ +/* If resending a message, don't keep mail_followup_to. */ if (resend) { -FREE (&newhdr->env->message_id);FREE (&newhdr->env->mail_followup_to); }
--
2.26.0.106.g9fadedd637
--- >8 ---
(The Git version this patch was prepared with should give you some sense of how ancient this part of my workflow is ;-).)
When looking at this yesterday after sending the 'no-ref-delta' patches to the list, I could not figure out quite why my Mutt client was rewriting Message-ID headers until I remembered the aforementioned patch.
The fix is somewhat OpenAI-specific, and so not interesting to share with the list, but effectively relies on piping messages to 'mutt -H -' to send the message without dropping (and thus rewriting) the Message-ID header.
(As an alternative, I could have continued to carry that patch to 'postpone.c', but in retrospect it seems gross^W unnecessary, so I ditched it.)
Thanks,
Taylor
Quoted 9 lines · Taylor Blau
As some background, my workflow for sending patches to the mailing list is to use a script called 'git mail' that effectively runs format-patch to build an *.mbox and then opens Mutt in that directory. I then review the patches one last time before sending, and then run a macro I have bound to 'B', which (effectively) runs <resend-message>.
For reasons that I cannot quite recall, I chose this workflow many years ago when it would likely have been more appropriate to use
mutt -H, which does *not* rewrite Message-ID headers when resending.
You might have inherited the <resend-message> thing from me. I thought I used it exactly because "-H" insisted on rewriting the message-id, but it doesn't seem to now. So either I've completely forgotten the reason, or perhaps the behavior used to be different.
I also like that <resend-message> lets me open the whole mbox and send each message within a single session. But I never run into the issue you're mentioning because I write my cover letter separately as a normal email, and then generate the actual patches as in-reply-to (with some script magic to pull the message-id from my sent folder).
So possibly my fault for leading you in the wrong direction many years ago, or your fault for not following my sage advice to the letter. ;)
-Peff