Is there an recommended way to refer to bitkeepr commits?
(Apologies resent as I forgot to include linux-kernel when I sent this
the first time)
I am in the process of digging through the intersection of threads, signals, ptrace, and exec and I am encountering bugs that predate 2.6.12-rc2 the first version in our current git tree. I am trying to figure out what to put in the fixes tag so that someone else can find the commits.
I have an old tree that appears to be no longer available on kernel.org that has all of the commits from the bit keeper era as well as the annotations indicating the bit keeper revisions those commits came from.
Thomas Gleixner appears to have a tree with all of those same commits except with the BKrev tags stripped out.
While reading through the post 2.6.12-rc2 commits I ran into a commit from I think Linus that references what I believe was a pre 2.6.12-rc2 with what looked like a truncated sha1 hash. There title of the commit/patch was no included and the sha1 was not in any tree that I have so I didn't know how to follow that reference. I wish I had saved off which commit that was so I could make this description more concrete.
I was thinking of referring to the old commit that was broken as:
Fixes: BKrev: 3e8e57a1JvR25MkFRNzoz85l2Gzccg ("[PATCH] linux-2.5.66-signal-cleanup.patch")
With no trees that have a BKrev in them publicly available it doesn't look like anyone will be able to find that code. It use a git sha1 hash we would need a standard tree that we can reference and we don't appear to have that either.
Is there any plan or standard recommendation on how handle bitkeeper era commits? Was there something sane and it was not properly restored after the kernel.org break-in?
Eric
Quoted 3 lines · Eric W. Biederman
Thomas Gleixner appears to have a tree with all of those same commits except with the BKrev tags stripped out.
That's the best import - so use that tree by Thomas, and just use the git revision numbers in it (and say "tglx's linux-history tree" or something).
Linus
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Eric W. Biederman
Thomas Gleixner appears to have a tree with all of those same commits except with the BKrev tags stripped out.
That's the best import - so use that tree by Thomas, and just use the git revision numbers in it (and say "tglx's linux-history tree" or something).
I've been using this one by Rob Landley which seems good:
https://landley.net/kdocs/fullhist/
It's grafted into the modern history so you can search seamlessly between the two which is pretty nice. I don't see any Bitkeeper tags though.
cheers
Quoted 12 lines · Michael Ellerman
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Eric W. Biederman
Thomas Gleixner appears to have a tree with all of those same commits except with the BKrev tags stripped out.
That's the best import - so use that tree by Thomas, and just use the git revision numbers in it (and say "tglx's linux-history tree" or something).
Quoted 7 lines
I've been using this one by Rob Landley which seems good:
https://landley.net/kdocs/fullhist/
It's grafted into the modern history so you can search seamlessly between the two which is pretty nice. I don't see any Bitkeeper tags though.
I went through and found/tagged the major old releases, did I forget to upload a new tarball after that?
v0.0.1 cff5a6fb66765e90470f4d9ca2398da0ca3c75d5
v1.0.0 a068026b4a060e822892a64d5107fb58c45743ef
v1.2.0 8610c92442d125f165dc84e4a96f5cbc9b240484
v2.0.0 a374953c636bd91ea40b2d1e31af5405b90e8bf8
v2.2.0 bf330b5e3c471d0b67737c4822b0174ef4f89bed
v2.4.0 13a80dffb74939e292b6e90e5d79dd26d577489f
v2.6.0 4e9b4bc7a660962ae5f04f939469263b91cf95c2
Rob
Quoted 15 lines · Rob Landley
Quoted 14 lines · Michael Ellerman
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Eric W. Biederman
Thomas Gleixner appears to have a tree with all of those same commits except with the BKrev tags stripped out.
That's the best import - so use that tree by Thomas, and just use the git revision numbers in it (and say "tglx's linux-history tree" or something).
I've been using this one by Rob Landley which seems good:
Quoted 8 lines
Quoted 5 lines
https://landley.net/kdocs/fullhist/
It's grafted into the modern history so you can search seamlessly between the two which is pretty nice. I don't see any Bitkeeper tags though.
I went through and found/tagged the major old releases, did I forget to upload a new tarball after that?
No you didn't forget, I see several:
refs/tags/v1.2.0
refs/tags/v2.6.0
refs/tags/v4.5-rc3
refs/tags/v2.4.0
refs/tags/v1.0.0
refs/tags/v2.2.0
refs/tags/v2.0.0
refs/tags/v4.5-rc4
refs/tags/v4.5-rc5
refs/tags/v4.5-rc2
refs/tags/v0.0.1
refs/tags/v4.5-rc1
But in the original mail Eric was wanting to refer to a specific Bitkeeper revision using the BK revision id, eg:
In your tree that is c3c107051660 ("[PATCH] linux-2.5.66-signal-cleanup.patch"), but you don't have the 3e8e57a1JvR25MkFRNzoz85l2Gzccg revision recorded anywhere that I can see.
cheers
Quoted 5 lines · Michael Ellerman
Fixes BKrev: 3e8e57a1JvR25MkFRNzoz85l2Gzccg ("[PATCH] linux-2.5.66-signal-cleanup.patch")In your tree that is
c3c107051660("[PATCH] linux-2.5.66-signal-cleanup.patch"), but you don't have the 3e8e57a1JvR25MkFRNzoz85l2Gzccg revision recorded anywhere that I can see.
That's correct. I did not include the BK revisions when I imported the commits into the history git. I did not see any reason to do so. I still have no idea what the value would have been or why anyone wants to reference them at all.
Thanks,
tglx
Quoted 11 lines · Thomas Gleixner
Quoted 5 lines · Michael Ellerman
Fixes BKrev: 3e8e57a1JvR25MkFRNzoz85l2Gzccg ("[PATCH] linux-2.5.66-signal-cleanup.patch")In your tree that is
c3c107051660("[PATCH] linux-2.5.66-signal-cleanup.patch"), but you don't have the 3e8e57a1JvR25MkFRNzoz85l2Gzccg revision recorded anywhere that I can see.That's correct. I did not include the BK revisions when I imported the commits into the history git. I did not see any reason to do so. I still have no idea what the value would have been or why anyone wants to reference them at all.
Thomas your import seems to be significantly better than the one I got my hands on years ago.
I just know that if were to do something similar today we would really want to preserve the existing git sha1 hashes somewhere because we refer to commits everywhere in the code. So I was imagining that bitkeeper would be similar. Especially since the copy of the bitkeeper import into git had appened to each commit a BKrev which I presume tacked back to the original source.
If everyone who had imported the bitkeepr tree had done that it would not have mattered which bitkeeper import you were using they would all share a common identifier for commits. With that absent the robustness we have to allow looking things up in an alternate tree lies solely with the one line patch description.
Compare the quotes lines above with what I have below. Every tree appears to have a different identifier.
Below is what I wound up doing, and have queued for the next merge window. Comments?
Eric
From: "Eric W. Biederman" <[email protected]>
Date: Tue, 9 May 2017 19:54:27 -0500
Subject: [PATCH] signal: Do not perform permission checks when sending pdeath_signal
This fixes and old old regression. When Roland switched from sending pdeath_signal with send_sig() to send_group_sig_info it gained a permission check, and started taking the tasklist lock. Roland earlier fixed the double taking of the tasklist lock in 3f2a0d1df938 ("[PATCH] fix pdeath_signal SMP locking") but pdeath_signal still performs an unnecessary permission check.
Ordinarily I would be hesitant at fixing an ancient regression but a permission check for our parent sending to us is almost never likely to fail (so it is unlikely anyone has noticed), and it is stupid. It makes absolutely no sense to see if our parent has permission to send us a signal we requested be sent to us.
As this is more permisssive there is no chance anything will break. The information of if our parent is living is available elsewhere getppid, tkill, and proc with no special permissions so this should not be an information leak.
See https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/
for the bitkeeper era history that I refer to.
--- kernel/exit.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/exit.c +2 -2
@@ -693,8 +693,8 @@ static void forget_original_parent(struct task_struct *father,if (likely (!t->ptrace)) t->parent = t->real_parent;if (t->pdeath_signal) -group_send_sig_info (t->pdeath_signal, -SEND_SIG_NOINFO , t); +do_send_sig_info (t->pdeath_signal, +SEND_SIG_NOINFO , t, true); }/* * If this is a threaded reparent there is no need to
Signature · 1 line
2.10.1
Quoted 14 lines · Eric W. Biederman
Quoted 11 lines · Thomas Gleixner
Quoted 5 lines · Michael Ellerman
Fixes BKrev: 3e8e57a1JvR25MkFRNzoz85l2Gzccg ("[PATCH] linux-2.5.66-signal-cleanup.patch")In your tree that is
c3c107051660("[PATCH] linux-2.5.66-signal-cleanup.patch"), but you don't have the 3e8e57a1JvR25MkFRNzoz85l2Gzccg revision recorded anywhere that I can see.That's correct. I did not include the BK revisions when I imported the commits into the history git. I did not see any reason to do so. I still have no idea what the value would have been or why anyone wants to reference them at all.
Quoted 6 lines
Thomas your import seems to be significantly better than the one I got my hands on years ago.
I just know that if were to do something similar today we would really want to preserve the existing git sha1 hashes somewhere because we refer to commits everywhere in the code.
Which is why the https://landley.net/kdocs/fullhist tree uses "git graft", so the git commit numbers are the same.
Quoted 4 lines · As Yoann Padioleau
It's built from 3 other git repositories:
- the one from Dave Jones from 0.01 to 2.4.0,
- the one from tglx from 2.4.0 to 2.6.12,
- the one from Linus Torvalds from 2.6.12 to now.
And the hashes in his tree were the same as in each of those trees, all three of which are on git.kernel.org. If you "git pull" the fullhist tree to current, it still uses the same hashes today. (I think you can still reproduce it localy using his scripts, which I mirrored. You'll have to manually re-tag those old commits from last message, and reset the "upstream" to pull current from.)
Last I checked I couldn't just "git push" the fullhist tree to git.kernel.org because git graft didn't propagate right. I had to start people from a tarball. Local clones doing the hardlink thing worked fine though. (Maybe that's changed?)
Quoted 1 line
So I was imagining that bitkeeper would be similar.
When Larry flounced and people lost access to the bitkeeper tool they lost access to read the old data, so what the bitkeeper numbers were became irrelevant. That's why nobody's cared before now.
You're looking for a consistent way to refer to old commits, even using bitkeeper numbers wouldn't fully solve that problem (it only goes back to 2.4). Between the dave jones and tglx trees, there's complete coverage back to 0.0.1. Yoann stitched them together, and I've kept a current version. I used to host it on kernel.org/doc until
http://lkml.iu.edu/hypermail/linux/kernel/1411.3/04693.html happened, they've since deleted it but it's GPL so anybody who wants to host a mirror... :)
I'm traveling and not downloading a gigabyte through my phone tether (darn tmobile 4 gig monthly tethering limit) but the date on the
https://landley.net/kdocs/local/linux-fullhist.tar.bz2
tarball is February 2016 so I'm pretty sure that's 4.0 with the old major releases tagged (ala last email). Anybody who wants to mirror it somewhere more official (and presumably .xz instead of .bz2) is welcome to.
(I would if I still had rsync access to kernel.org/doc, but alas I can't even get them to link kernel.org/doc/Documentation from the page above it. It used to, they accidentally deleted it, and nobody maintains the page anymore...)
Quoted 4 lines
Especially since the copy of the bitkeeper
import into git had appened to each commit a BKrev which I presume tacked back to the original source.
Quoted 8 lines
If everyone who had imported the bitkeepr tree had done that it would not have mattered which bitkeeper import you were using they would all share a common identifier for commits. With that absent the robustness we have to allow looking things up in an alternate tree lies solely with the one line patch description.
Compare the quotes lines above with what I have below. Every tree appears to have a different identifier.
The commits in the fullhist tree have been stable since at least
https://lwn.net/Articles/285366/ which was June 6, 2008. It's derived from earlier trees with the same commits, and kept those commit hashes.
Quoted 2 lines
Below is what I wound up doing, and have queued for the next merge window. Comments?
I've bisected or used the "git annotate... git annotate HASH^1... git annotate NEXTHASH^1..." peeling trick back to some really old commits over the years, which I've then referred to by submitter and date if it wasn't in the current git tree.
I'll happily give people hashes out of the fullhist tree if they ask, but haven't assumed they're using it. But if you're looking for an existing standard, this exists and predates my use of it.
Quoted 1 line
Eric
Rob
Quoted 2 lines · Rob Landley
Last I checked I couldn't just "git push" the fullhist tree to git.kernel.org because git graft didn't propagate right.
Perhaps you could recreate them with git replace --graft. That creates replace objects that can be pushed and fetched. (They are stored in refs/replace, and must be pushed/fetched explicitly.)
Andreas.
Signature · 3 lines
Andreas Schwab, [email protected] GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Quoted 8 lines · Andreas Schwab
Quoted 2 lines · Rob Landley
Last I checked I couldn't just "git push" the fullhist tree to git.kernel.org because git graft didn't propagate right.
Perhaps you could recreate them with git replace --graft. That creates replace objects that can be pushed and fetched. (They are stored in refs/replace, and must be pushed/fetched explicitly.)
It's the "must be pushed/fetched explicitly" part that I couldn't figure out back when I tried it.
I inherited this tree from somebody who made it. I noticed its existence because lwn.net covered it, and then 6 months later it had vanished without trace (as so many things do). I reproduced it from the build script (if you can't reproduce the experiment from initial starting conditions, it's not science), went "look, cool thing", and hosted a copy with an occasional repaint.
I would be _thrilled_ to hand it off to somebody who knows what they're doing with git. I'm just unusually interested in computer history and the preservation thereof. (https://landley.net/history/mirror).
Rob
Quoted 2 lines · Rob Landley
It's the "must be pushed/fetched explicitly" part that I couldn't figure out back when I tried it.
You have to use refs/replace/*:refs/replace/* as the refspec for push and fetch. By default, push and fetch only look at refs/heads/*.
Andreas.
Signature · 3 lines
Andreas Schwab, [email protected] GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Quoted 32 lines · Rob Landley
Quoted 14 lines · Eric W. Biederman
Quoted 11 lines · Thomas Gleixner
Quoted 5 lines · Michael Ellerman
Fixes BKrev: 3e8e57a1JvR25MkFRNzoz85l2Gzccg ("[PATCH] linux-2.5.66-signal-cleanup.patch")In your tree that is
c3c107051660("[PATCH] linux-2.5.66-signal-cleanup.patch"), but you don't have the 3e8e57a1JvR25MkFRNzoz85l2Gzccg revision recorded anywhere that I can see.That's correct. I did not include the BK revisions when I imported the commits into the history git. I did not see any reason to do so. I still have no idea what the value would have been or why anyone wants to reference them at all.
Quoted 6 lines
Thomas your import seems to be significantly better than the one I got my hands on years ago.
I just know that if were to do something similar today we would really want to preserve the existing git sha1 hashes somewhere because we refer to commits everywhere in the code.
Which is why the https://landley.net/kdocs/fullhist tree uses "git graft", so the git commit numbers are the same.
Quoted 4 lines · As Yoann Padioleau
It's built from 3 other git repositories:
- the one from Dave Jones from 0.01 to 2.4.0,
- the one from tglx from 2.4.0 to 2.6.12,
- the one from Linus Torvalds from 2.6.12 to now.
Quoted 6 lines
And the hashes in his tree were the same as in each of those trees, all three of which are on git.kernel.org. If you "git pull" the fullhist tree to current, it still uses the same hashes today. (I think you can still reproduce it localy using his scripts, which I mirrored. You'll have to manually re-tag those old commits from last message, and reset the "upstream" to pull current from.)
Which leaves me perplexed. The hashes from tglx's current tree:
https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
on kernel.org and the hashes in your full history tree differ. Given that they are in theory the same tree this distrubs me.
Case in point in the commit connected to:
"[PATCH] linux-2.5.66-signal-cleanup.patch"
in tglx's tree is: da334d91ff7001d234863fc7692de1ff90bed57a
in the fullhist tree is: c3c107051660455a3e1ea3bde0278a75a0dc11e7
Quoted 7 lines
Quoted 2 lines
Compare the quotes lines above with what I have below. Every tree appears to have a different identifier.
The commits in the fullhist tree have been stable since at least
https://lwn.net/Articles/285366/ which was June 6, 2008. It's derived from earlier trees with the same commits, and kept those commit hashes.
*scratches my head*
Something appears to have changed somewhere.
Eric
Quoted 8 lines · Eric W. Biederman
Which leaves me perplexed. The hashes from tglx's current tree:
https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
on kernel.org and the hashes in your full history tree differ. Given that they are in theory the same tree this distrubs me.Case in point in the commit connected to:
"[PATCH] linux-2.5.66-signal-cleanup.patch"
in tglx's tree is:da334d91ff7001d234863fc7692de1ff90bed57a
That's the proper sha1 for my tree. I jsut verified it against the original tree which I still have in my archive.
Quoted 3 lines
*scratches my head*
Something appears to have changed somewhere.
Correct. That full history git rewrote the commits in my bitkeeper import.
history.git:
commit 7a2deb32924142696b8174cdf9b38cd72a11fc96
Author: Linus Torvalds <[email protected]>
Date: Mon Feb 4 17:40:40 2002 -0800
Import changeset
full-history:
commit 26245c315da55330cb25dbfdd80be62db41dedb2
Author: linus1 <[email protected]>
Date: Thu Jan 4 12:00:00 2001 -0600
Import changeset
and as a consequence all other commits have different shas as well.
Sigh....
Thanks,
tglx
Quoted 5 lines · Thomas Gleixner
Quoted 4 lines · Eric W. Biederman
Which leaves me perplexed. The hashes from tglx's current tree:
https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
on kernel.org and the hashes in your full history tree differ. Given that they are in theory the same tree this distrubs me.
The original build script used to make fullhist is at:
http://landley.net/kdocs/fullhist/make-full-linux-history.tgz
And his original description of what he did and why is at:
https://lwn.net/Articles/285366/
He mentioned something about rewriting dates?
I used the "graft" feature of git (thanks to Junio and people on #git for the tip) to link them together. I also modified (via a git-filter-branch) the dates of some commits as for instance all commits from the Dave Jones's repo had the same date (23 Nov 2007). For this I mainly used the timestamp info of files on kernel.org. The script and info I used are also available on my website[2].
(I tried to read his conversion plumbing but it's in ocaml.)
Apparently he only considered the git commits in Linus's tree to be worth preserving. I'd forgotten that part. (It was 9 years ago. I remembered the pre-bitkeeper tree got edited but I forgot the other one did too.)
Quoted 4 lines
Quoted 3 lines
Case in point in the commit connected to:
"[PATCH] linux-2.5.66-signal-cleanup.patch"
in tglx's tree is:da334d91ff7001d234863fc7692de1ff90bed57a
Quoted 8 lines
That's the proper sha1 for my tree. I jsut verified it against the original tree which I still have in my archive.
Quoted 3 lines
*scratches my head*
Something appears to have changed somewhere.
Correct. That full history git rewrote the commits in my bitkeeper import.
I only checked that the current ones in Linus's tree were the same. Nobody'd ever pointed me at a file hash in your conversion of bitkeeper to git, so over the years I forgot that the date editing extended into bitkeeper for some reason.
Quoted 7 lines
history.git:
commit
7a2deb32924142696b8174cdf9b38cd72a11fc96
Author: Linus Torvalds <[email protected]>
Date: Mon Feb 4 17:40:40 2002 -0800Import changeset
February 4, 2002.
Quoted 7 lines
full-history:
commit
26245c315da55330cb25dbfdd80be62db41dedb2
Author: linus1 <[email protected]>
Date: Thu Jan 4 12:00:00 2001 -0600Import changeset
January 4, 2001.
According to https://www.kernel.org/pub/linux/kernel/v2.4/ January 4 2001 is when 2.4.0 was released. So yes, it looks like he rewrote these dates to be correct.
I see what he did. Linus started his bitkeeper tree by importing 2.4.0 and then applying a year's worth of release diffs from 2.4.0 as individual commits. That year+ worth of work was all dated February 4, 2002 in the repo, so the fullhist script went through and changed the dates on those commits to match the release tarballs for those kernel versions, and that changed the hashes in the rest of the history tree.
Upside, there's no longer a year+ hole in the commit dates (which makes looking up associated mailing list posts a lot easier). Downside: this changed the history.git commit hashes for the rest of that era. (I'd missed that.)
Quoted 1 line
and as a consequence all other commits have different shas as well.
The most embarassing part is that the ocaml plumbing appears to occasionally leak host context when doing the conversion, specifically from "git log 26245c315da5" (checking to make sure the fullhist tree's dates make sense in context) I get:
commit 26245c315da55330cb25dbfdd80be62db41dedb2
Author: linus1 <[email protected]> Date: Thu Jan 4 12:00:00 2001 -0600
Import changeset
commit 13a80dffb74939e292b6e90e5d79dd26d577489f
Author: linus1 <landley@driftwood.(none)> Date: Thu Jan 4 12:00:00 2001 -0600
add prerelease patch to get a 2.4.0
commit 4c5b4d50bb08753433f5962bd926198fe2b7105d
Author: linus1 <[email protected]> Date: Sun Dec 31 12:00:00 2000 -0600
That landley@driftood should not be there. Sigh.
I guess the question is which is more broken? I linked the build scripts above if somebody else wants to modify or rerun them, but... lithp. Do you prefer a year gap in the archive dates, or do you prefer to call the history.git hashes cannonical?
Rob
Quoted 5 lines · Thomas Gleixner
That's correct. I did not include the BK revisions when I imported the commits into the history git. I did not see any reason to do so. I still have no idea what the value would have been or why anyone wants to reference them at all.
The bitkeeper hashed revision numbers were almost entirely useless, to the point where most BK users never really knew about them.
The revision numbers that people *saw* were the nasty horribly sequential-tree ones that are fundamentally broken, and that change when you merge (eg things like "1.56.3.2"). That's what people actually *used*, and saw in the logs, and were aware of. People used them too much, in fact, exactly because people who came from other environments thought that they were the revision numbers, and couldn't understand that they weren't stable.
Nobody ever used the "true" hashes, except in scripts.
So exposing them would have been ridiculous. Even most BK users would just have been confused by the line noise.
Linus