Skip to contentSkip to conversation

BK->CVS (real time mirror)

ANNOUNCE Larry McVoy 19 participants 37 messages Mar 12, 2003 – Mar 17, 2003

lore
Share & tools
lore
#0 lore raw

We've been working on a gateway between BitKeeper and CVS to provide the revision history in a form which makes the !BK people happy (or happier).

We have the first pass of this completed and have a linux 2.5 tree on kernel.bkbits.net and you can check out the tree as follows (please don't do this unless you are a programmer and will be using this. Penguin Computing provided the hardware and the bandwidth for that machine and if you all melt down the network they could get annoyed. By all means go for it if you actually write code, though, that's why it is there.)

mkdir ws
cd ws
cvs -d:pserver:[email protected]:/home/cvs co linux-2.5

Each of the releases are tagged, they are of the form v2_5_64 etc.

Linus had said in the past that someone other than us should do this but as it turns out, to do a reasonable job you need BK source. So we did it. What do we mean by a reasonable job? BitKeeper has an automatic branch feature which captures all parallel development. It's cool but a bit pedantic and it makes exporting to a different system almost impossible if you try and match what BK does exactly. So we didn't. What we (actually Wayne Scott) did was to write a graph traversal alg which finds the longest path through the revision history which includes all tags. For the 2.5 tree, that is currently 8298 distinct points. Each of those points has been captured in CVS as a commit. If we did our job correctly, each of these commits has the same timestamp across all files. So you should be able to get any changeset out of the CVS tree with the appropriate CVS command based on dates.

We also created a ChangeSet file in the CVS tree. It has no contents, it serves as a place to capture the BK changeset comments. Each file which is part of a changeset has an extra comment which is of the form

(Logical change 1.%d)

where the "1.%d" matches the changeset rev. So you can look for all files that have (Logical change 1.300) in their comments to reconstruct the changeset. NOTE! That information is actually redundant, the timestamps are supposed to do the same thing, let us know if that is not working, we'll redo it. I expect we'll find bugs, please be patient, it takes 4 hours of CPU time on a 2.1Ghz Athlon to do the conversion, that's a big part of why this has taken so long. That's after a week's worth of optimizations.

Each ChangeSet delta has a BK rev associated with it in the comments. We'll be giving you a small shell script which you can use to send Linus patches that include the rev and we'll modify BK so that it can take those patches with no patch rejects if you used that script.

We have a first pass of a real time gateway between BK and this CVS tree done. Right now it is done by hand (by me) but as soon as it is debugged you will see this tree being updated about 1-3 minutes after Linus pushes to bkbits.

Once you guys look this over and decide you like it, we'll do the same thing for the 2.4 tree.

We're also talking to an unnamed (in case it doesn't work out) Linux company who may host bkbits.net for us. If they do that, we'll turn the GNU patch exporter feature in BKD. That means that you'll be able to wget any changeset as a GNU patch, complete with checkin comments. I'm working with Alan on the format, I think we're close though I have to run the latest version past him.

If all of this sounds nice, it is. It was a lot of work for us to do this and you might be wondering why we bothered. Well, for a couple of reasons. First of all, it was only recently that I realized that because BK is not free software some people won't run BK to get data out of BK. It may be dense on my part, but I simply did not anticipate that people would be that extreme, it never occurred to me. We did a ton of work to make sure anyone could get their data out of BK but you do have to run BK to get the data. I never thought of people not being willing to run BK to get at the data. Second, we have maintained SCCS compatible file formats so that there would be another way to get the data out of BK. This has held us back in terms of functionality and performance. I had thought there was some value in the SCCS format but recent discussions on this list have convinced me that without the changeset information the file format doesn't have much value.

Our goal is to provide the data in a way that you can get at it without being dependent on us or BK in any way. As soon as we have this debugged, I'd like to move the CVS repositories to kernel.org (if I can get HPA to agree) and then you'll have the revision history and can live without the fear of the "don't piss Larry off license". Quite frankly, we don't like the current situation any better than many of you, so if this addresses your concerns that will take some pressure off of us.

Another goal is to have the freedom to evolve our file formats to be better, better performance and more features. SCCS is holding us back. So you should look hard at what we are providing and figure out if it is enough. If you come back with "well, it's not BitKeeper so it's not enough" we'll just ignore that. CVS isn't BitKeeper. On the other hand, we believe we have gone as far as is possible to provide all of the information, checkin comments, data, timestamps, user names, everything. The graph traversal alg captures information at an extremely fine granularity, absolutely as fine is possible. We have 8298 distinct points over the 2.5.0 .. 2.5.64 set of changes, so it is 130 times finer than the official releases. If you think something is missing, tell us, we'll try and fix it.

The payoff for you is that you have the data in a format that is not locked into some tool which could be taken away. The payoff for us is that we can evolve our tool as we see fit. We have that right today, we can do whatever we want, but it would be anywhere from annoying to unethical to do so if that meant that you couldn't get at the data except through BitKeeper. So the "deal" here is that you get the data in CVS (and/or patches + comments) and we get to hack the heck out of the file format. Our changes are going to move far faster than CSSC or anyone else could keep up without a lot of effort. On the other hand, our changes are going to make cold cache performance be much closer to hot cache performance, use a lot less disk space, a lot less memory, and a lot less CPU.

So take a look and tell me what you think.

Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#1 lore raw
Quoted 4 lines

The payoff for you is that you have the data in a format that is not locked into some tool which could be taken away. The payoff for us is that we can evolve our tool as we see fit. We have that right today, we can do whatever we want, but it would be anywhere from annoying

Quoted 8 lines

to unethical to do so if that meant that you couldn't get at the data except through BitKeeper. So the "deal" here is that you get the data in CVS (and/or patches + comments) and we get to hack the heck out of the file format. Our changes are going to move far faster than CSSC or anyone else could keep up without a lot of effort. On the other hand, our changes are going to make cold cache performance be much closer to hot cache performance, use a lot less disk space, a lot less memory, and a lot less CPU.

Larry, I don't mean to start yet another anti-bitmover, anti-bitkeeper or anti-larry flame-fest, but I have to be honest that I am a little bit worried.

You are giving us approximately 90% of our data in exchange for the one thing that made using bitkeeper not a total sellout; the fact that the revision history of the repo was still accessible without proprietary software.

I honestly appreciate the work that you and BitMover do for the kernel, but not giving us access to 100% of _our_ data is unacceptable to me. Quite honestly, I think your move is to restrict the possible alternatives to the BK client (the CSSC based ones like I and others had done), which were able to extract 100% of the data, even if they couldn't make use of it in the same way as bitkeeper. Atleast it was there.

You've made quite a marketing move. It's obvious to me, maybe not to others. By providing this CVS gateway, you make it almost pointless to work on an alternative client. Also by providing it, you make it easier to get away with locking the revision history into a proprietary format.

Signature · 4 lines
Debian     - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
Deqo       - http://www.deqo.com/
#30 lore raw
In reply to Larry McVoy
Followup to:  <[email protected]>
By author:    Larry McVoy <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 16 lines

If all of this sounds nice, it is. It was a lot of work for us to do this and you might be wondering why we bothered. Well, for a couple of reasons. First of all, it was only recently that I realized that because BK is not free software some people won't run BK to get data out of BK. It may be dense on my part, but I simply did not anticipate that people would be that extreme, it never occurred to me. We did a ton of work to make sure anyone could get their data out of BK but you do have to run BK to get the data. I never thought of people not being willing to run BK to get at the data. Second, we have maintained SCCS compatible file formats so that there would be another way to get the data out of BK. This has held us back in terms of functionality and performance. I had thought there was some value in the SCCS format but recent discussions on this list have convinced me that without the changeset information the file format doesn't have much value.

I can only speak for myself, but I didn't mind until the license ended up having the "unless you hack on other tools" exception in it. Personally, I value my freedom to hack on whatever I want a lot more than the convenience of BK. This is a personal choice on my part and may sound "extreme" to you, and other people have made other tradeoffs, but for me freedom was the reason I started hacking Linux instead of becoming a Win32 geek.

Having this capability available will certainly make life better for everyone involved. Besides, "we won't hold your data hostage" is actually a pretty nice selling argument.

Quoted 9 lines

Our goal is to provide the data in a way that you can get at it without being dependent on us or BK in any way. As soon as we have this debugged, I'd like to move the CVS repositories to kernel.org (if I can get HPA to agree) and then you'll have the revision history and can live without the fear of the "don't piss Larry off license". Quite frankly, we don't like the current situation any better than many of you, so if this addresses your concerns that will take some pressure off of us.

I'm sure we can work something out. However, at the moment zeus.kernel.org, our main server with lots and lots of bandwidth, is starting to run into its limits, so I can't promise *when* that will happen. Just putting in another server

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
#31 lore raw
Quoted 4 lines · H. Peter Anvin

I can only speak for myself, but I didn't mind until the license ended up having the "unless you hack on other tools" exception in it. Personally, I value my freedom to hack on whatever I want a lot more than the convenience of BK.

Yeah, that's cool, I don't blame you, it's a pretty extreme clause. We may well drop it in the future if we feel we have pulled far enough ahead that everyone else is just playing catchup. I do apologize for that clause, I know it caused a lot of concern, but try and remember that I'm "you" in that I'm just an engineer figuring this stuff out as I go. We try and fix it as we go as well so there is hope.

Quoted 3 lines

Having this capability available will certainly make life better for everyone involved. Besides, "we won't hold your data hostage" is actually a pretty nice selling argument.

Yup. I really thought that all the export stuff was !hostage but I didn't factor in the license issues.

Quoted 8 lines
Quoted 3 lines

As soon as we have this
debugged, I'd like to move the CVS repositories to kernel.org (if I can get HPA to agree)

I'm sure we can work something out. However, at the moment zeus.kernel.org, our main server with lots and lots of bandwidth, is starting to run into its limits, so I can't promise *when* that will happen. Just putting in another server

We can certainly pay for a server, a server is not that much money and is not an ongoing cost. So if that's the problem, I can get you a server in less than a week, ping me off line and we'll work it out.

The main thing is that the CVS server and the tarball of the CVS repository are *not* under our control. That's the only way some people are going to believe that we're not out to screw them and it would oh-so-nice to have people think that, it really would.

Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#2 lore raw
In reply to Ben Collins
Quoted 4 lines · Ben Collins

You've made quite a marketing move. It's obvious to me, maybe not to others. By providing this CVS gateway, you make it almost pointless to work on an alternative client. Also by providing it, you make it easier to get away with locking the revision history into a proprietary format.

This is a really good point, deserves high lighting imho...

The BK candy is getting increasingly bitter to swallow here, I may just have to drop it soon. A shame.

Signature · 1 line
Jens Axboe
#3 lore raw
Quoted 10 lines · Jens Axboe
Quoted 4 lines · Ben Collins

You've made quite a marketing move. It's obvious to me, maybe not to others. By providing this CVS gateway, you make it almost pointless to work on an alternative client. Also by providing it, you make it easier to get away with locking the revision history into a proprietary format.

This is a really good point, deserves high lighting imho...

The BK candy is getting increasingly bitter to swallow here, I may just have to drop it soon. A shame.

Sadly, some people see the dark side of everything. I don't see how making a CVS repository available with comments and an as-good-as-you-can-do-with-CVS equivalent of a BK changeset equals "locking the revision history into a proprietary format". Yes, Larry said that this would allow him to change the BK file format to break compatibility with CSSC, but it is no more "locked away" now than before for those people who refuse to use BK.

Ironically, SCCS was a former "evil proprietary format" that was reverse engineered to get CSSC, AFAIK. People are still free to update CSSC to track BK if they so choose.

Some people will just never be happy no matter what you give them.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#4 lore raw
Quoted 1 line · Andreas Dilger

Some people will just never be happy no matter what you give them.

I've been very happy with BK, been using it shortly after Linus started doing so. Mostly out of curiosity at first, later because it was actually quite useful. I even see myself as a fairly pragmatic individual, but even so I do find it increasingly difficult to defend my BK usage.

So please stop thinking you can judge that easily by pushing me into your nice little 'some people will never be happy bla bla' category.

Signature · 1 line
Jens Axboe
#5 lore raw
Quoted 5 lines · Jens Axboe

I've been very happy with BK, been using it shortly after Linus started doing so. Mostly out of curiosity at first, later because it was actually quite useful. I even see myself as a fairly pragmatic individual, but even so I do find it increasingly difficult to defend my BK usage.

Interesting. I _had_ lumped you into the "unhappy with BK" camp that has become so vocal on l-k these days. My apologies. I do find it sort of sad that you (or anyone) actually have to defend your BK usage to others.

I'm personally a "do what you want and let others do what they want as long as it doesn't interfere with me" kind of person, but it seems that lots of people here have the opinion that they know what is better for everyone else, and have no problem telling the list over an over about it. Probably time to fork a linux-code-repository mailing list and have everyone spend their time over there instead of rehashing BK flamewars and/or BK replacement here every week.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#6 lore raw
Quoted 10 lines · Andreas Dilger
Quoted 5 lines · Jens Axboe

I've been very happy with BK, been using it shortly after Linus started doing so. Mostly out of curiosity at first, later because it was actually quite useful. I even see myself as a fairly pragmatic individual, but even so I do find it increasingly difficult to defend my BK usage.

Interesting. I _had_ lumped you into the "unhappy with BK" camp that has become so vocal on l-k these days. My apologies. I do find it sort of sad that you (or anyone) actually have to defend your BK usage to others.

No offense taken, and I personally don't have any sort of political agenda that I care to voice on lkml :). That's part of where the pragmatism comes in, I just don't care enough.

About every patch I sent here on lkml has been generated by bk for the past year. I typically don't do commits, just have trees with pending deltas and bk -r diffs -u does the job for me.

Quoted 7 lines

I'm personally a "do what you want and let others do what they want as long as it doesn't interfere with me" kind of person, but it seems that lots of people here have the opinion that they know what is better for everyone else, and have no problem telling the list over an over about it. Probably time to fork a linux-code-repository mailing list and have everyone spend their time over there instead of rehashing BK flamewars and/or BK replacement here every week.

For me, I think Andrew's patch management scripts will do the job. And yeah, the non-stop bk threads make me sick as well and are rarely read here. So I better make this my last mail on the subject...

Signature · 1 line
Jens Axboe
#7 lore raw
In reply to Andreas Dilger
Quoted 3 lines · Andreas Dilger

Ironically, SCCS was a former "evil proprietary format" that was reverse engineered to get CSSC, AFAIK. People are still free to update CSSC to track BK if they so choose.

Actually the SCCS format is documented in a manual page (although there are a few annoying ambiguities in it).

-- Jamie

#8 lore raw
In reply to Andreas Dilger
Followup to:  <[email protected]>
By author:    Andreas Dilger <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 8 lines

Sadly, some people see the dark side of everything. I don't see how making a CVS repository available with comments and an as-good-as-you-can-do-with-CVS equivalent of a BK changeset equals "locking the revision history into a proprietary format". Yes, Larry said that this would allow him to change the BK file format to break compatibility with CSSC, but it is no more "locked away" now than before for those people who refuse to use BK.

Quoted 8 lines

Ironically, SCCS was a former "evil proprietary format" that was reverse engineered to get CSSC, AFAIK. People are still free to update CSSC to track BK if they so choose.

Some people will just never be happy no matter what you give them.

From what I can gather, the question is very simple:

"Can we get our data out of BK into some kind of open format?"

It's an important question. If the answer is "yes, but only the stuff that can be mapped onto CVS" then that's a significant data loss, and if BitMover changes the data format without documentation, then there is no longer a way to get all the data out.

Presumably the CVS exporter could get augmented with some kind of metadata export... perhaps an XML schema that describes how the various points are to be linked or whatnot... it won't turn CVS into BK overnight (so Larry can still sleep at night), but it would give BitMover the freedom to change their data format.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
#21 lore raw
In reply to Andreas Dilger
Quoted 19 lines · Andreas Dilger
Quoted 10 lines · Jens Axboe
Quoted 4 lines · Ben Collins

You've made quite a marketing move. It's obvious to me, maybe not to others. By providing this CVS gateway, you make it almost pointless to work on an alternative client. Also by providing it, you make it easier to get away with locking the revision history into a proprietary format.

This is a really good point, deserves high lighting imho...

The BK candy is getting increasingly bitter to swallow here, I may just have to drop it soon. A shame.

Sadly, some people see the dark side of everything. I don't see how making a CVS repository available with comments and an as-good-as-you-can-do-with-CVS equivalent of a BK changeset equals "locking the revision history into a proprietary format". Yes, Larry said that this would allow him to change the BK file format to break compatibility with CSSC, but it is no more "locked away" now than before for those people who refuse to use BK.

Quoted 3 lines

Ironically, SCCS was a former "evil proprietary format" that was reverse engineered to get CSSC, AFAIK. People are still free to update CSSC to track BK if they so choose.

Atleast SCCS is mostly ascii. Larry is talking about binary. Who knows, maybe even encrypted and using some unknown compression method (I'm sure if it's encrypted, it will be called "compression").

Signature · 4 lines
Debian     - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
Deqo       - http://www.deqo.com/
#9 lore raw
In reply to H. Peter Anvin
Quoted 6 lines · H. Peter Anvin

"Can we get our data out of BK into some kind of open format?"

It's an important question. If the answer is "yes, but only the stuff that can be mapped onto CVS" then that's a significant data loss, and if BitMover changes the data format without documentation, then there is no longer a way to get all the data out.

This sounds like the old GPL argument.

The GPL'd redistributor has to supply the source, they don't have to supply it in the format that's best for you, being an 80mm tape drive cuz you're stuck in the punch card age.

Seriously, if CVS loses all that data, is that BK's fault? BK's so powerful because it has more information than anyone else, but it's not their fault (and it's not proprietary data) that no-one else can deal with the data when it's exported, now is it????

It's not a significant data loss when you try to view a 24bpp image on an 8bpp display, so it's not a significant data loss that CVS can't handle the BK. If it could, Linus would've switched to CVS instead....

I'm not saying Larry's a God or anything, I'm just hoping you guys can give it up already. Linus uses BK, nobody else needs to, so move on!

Dana Lacoste
Ottawa, Canada

#10 lore raw
Quoted 1 line

Linus uses BK, nobody else needs to, so move on!

Seconded.

John.

#22 lore raw
In reply to Ben Collins
Quoted 3 lines · Ben Collins

Atleast SCCS is mostly ascii. Larry is talking about binary. Who knows, maybe even encrypted and using some unknown compression method (I'm sure if it's encrypted, it will be called "compression").

*sigh* However, all Larry _could_ be talking about is that he wants to replace the SCCS format by something more powerful. Nowhere did he say that the format would not be documented.

Granted, he also did not say that it _would_ be, but you all are jumping so hard on him based on the assumption that it would not be without knowing that either, so maybe you could have just written to Larry and asked?

I'm rather agnostic to the BK debate: I think it is an awesome tool, and if it gets the job done, I am all for it. If you don't want to use it for whatever reason, that's fine too. And asking for the Linux Kernel data to be fully available without using a proprietary tool also makes lots of sense. I also agree with Larry that duplicating the work done in BK in an Open Source tool is going to take quite a while and effort: I'm not going as far as saying that it cannot be done, because Linux itself is the best example that it _can_ be done if people really want too. But if you want, more power to you, too.

But the 'BK and Larry are evil to the bone because it is not GPL!' crap is annoying the hell out of me. Shut up, will you all, pretty please? And could you please first ask/clarify, then flame?

Sincerely,
Lars Marowsky-Brée <[email protected]>

Signature · 4 lines
SuSE Labs - Research & Development, SuSE Linux AG
  
"If anything can go wrong, it will." "Chance favors the prepared (mind)."
  -- Capt. Edward A. Murphy            -- Louis Pasteur
#11 lore raw
In reply to Dana Lacoste

Hi,

Quoted 4 lines · Dana Lacoste

Seriously, if CVS loses all that data, is that BK's fault? BK's so powerful because it has more information than anyone else, but it's not their fault (and it's not proprietary data) that no-one else can deal with the data when it's exported, now is it????

That's not the point. Larry does not own the data in the Linux repository, this was one of the conditions for the bk usage, so Larry cannot say, that you only get all the data if you use bk. If cvs can't represent all the information, we have to find another solution.

bye, Roman

#14 lore raw
In reply to Dana Lacoste
Quoted 17 lines · Dana Lacoste
Quoted 7 lines · H. Peter Anvin

"Can we get our data out of BK into some kind of open format?"

It's an important question. If the answer is "yes, but only the stuff that can be mapped onto CVS" then that's a significant data loss, and if BitMover changes the data format without documentation, then there is no longer a way to get all the data out.

This sounds like the old GPL argument.

The GPL'd redistributor has to supply the source, they don't have to supply it in the format that's best for you, being an 80mm tape drive cuz you're stuck in the punch card age.

Quoted 9 lines

Seriously, if CVS loses all that data, is that BK's fault? BK's so powerful because it has more information than anyone else, but it's not their fault (and it's not proprietary data) that no-one else can deal with the data when it's exported, now is it????

It's not a significant data loss when you try to view a 24bpp image on an 8bpp display, so it's not a significant data loss that CVS can't handle the BK. If it could, Linus would've switched to CVS instead....

You're missing the point completely.

Of course it's not BK's fault that CVS can't represent the data. However, one of the (valid!) selling points of BK was "we won't hold your data hostage." That requires that you can export both the data and the metadata into some kind of open format. Since CVS clearly can't be that open format (CVS being insufficiently powerful), the additional metadata needs to be available in some kind of auxilliary form. It's then, of course, not BK's fault that CVS can't possibly make use of that auxilliary metadata.

-hpa

#23 lore raw
In reply to Lars Marowsky-Bree
Quoted 7 lines · Lars Marowsky-Bree
Quoted 3 lines · Ben Collins

Atleast SCCS is mostly ascii. Larry is talking about binary. Who knows, maybe even encrypted and using some unknown compression method (I'm sure if it's encrypted, it will be called "compression").

Quoted 7 lines

*sigh* However, all Larry _could_ be talking about is that he wants to replace the SCCS format by something more powerful. Nowhere did he say that the format would not be documented.

Granted, he also did not say that it _would_ be, but you all are jumping so hard on him based on the assumption that it would not be without knowing that either, so maybe you could have just written to Larry and asked?

This is, I think, an underlying assumption that people are making.

Larry, given that I *think* you've said that the algorithms are the important parts, not the file format (at least in the past), would you consider publicly documenting the file format?

Thanks for the CVS gateway, anyway, though.

Signature · 2 lines
Ryan Anderson
  sometimes Pug Majere
#15 lore raw
In reply to H. Peter Anvin
Quoted 18 lines
Quoted 6 lines

This sounds like the old GPL argument.

The GPL'd redistributor has to supply the source, they don't have to supply it in the format that's best for you, being an 80mm tape drive cuz you're stuck in the punch card age.

Quoted 9 lines

Seriously, if CVS loses all that data, is that BK's fault? BK's so powerful because it has more information than anyone else, but it's not their fault (and it's not proprietary data) that no-one else can deal with the data when it's exported, now is it????

It's not a significant data loss when you try to view a 24bpp image on an 8bpp display, so it's not a significant data loss that CVS can't handle the BK. If it could, Linus would've switched to CVS instead....

You're missing the point completely.

Quoted 8 lines

Of course it's not BK's fault that CVS can't represent the data. However, one of the (valid!) selling points of BK was "we won't hold your data hostage." That requires that you can export both the data and the metadata into some kind of open format. Since CVS clearly can't be that open format (CVS being insufficiently powerful), the additional metadata needs to be available in some kind of auxilliary form. It's then, of course, not BK's fault that CVS can't possibly make use of that auxilliary metadata.

I thought that BK has been able to export everything to a text file since the first version.

(Ah, but of course, unless that text file is available in EBCDIC, we still have a problem...)

John.

#16 lore raw
Quoted 2 lines

I thought that BK has been able to export everything to a text file since the first version.

bk export -tpatch -r1.900 > patch.1.900
bk changes -v -r1.900 > comments.1.900

Been there forever. So has ways to get all the metadata from the command line without having to reverse engineer the file format. See

http://www.bitkeeper.com/manpages/bk-prs-1.html

it's all there. Always has been.

Wayne wanted me to point that it is easy to write the BK to CVS exporter completely from the command line, we prototyped it that way, the only reason we rewrote part of it in C was for performance. The point being that you guys could have done this yourself without help from us because all the metadata is right there. Ditto for anyone else worried about getting their data out of BK now or in the future. The whole point of prs is to be able to have a will-always-work way to get at the data or the metadata, it makes the file format a non-issue.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#24 lore raw
In reply to Ben Collins
Quoted 3 lines · On Tue, 11 Mar 2003 23:16:21 -0500 Ben Collins <[email protected]> wrote:

You've made quite a marketing move. It's obvious to me, maybe not to others. By providing this CVS gateway, you make it almost pointless to work on an alternative client. Also by providing it, you make it easier

I don't think so. This also bits Larry. If he does well enought, there'll be some people here that won't use bitkeeper just because they can use the cvs gateway and they don't need/miss the features they could get with bk.

And i don't think it avoids creating a free bk clone. I guess that there's a lot of people out there interested in such tool, and not only for kernel development; this won't stop them.

As far as i can see; Larry is just wasting time (money) to help the kernel development and people who doesn't use BK just because it isn't free. And he's not charging me, so i find this a good movement for everybody. I only can say thanks.

Diego Calleja

#25 lore raw
Quoted 11 lines · Arador
Quoted 3 lines · On Tue, 11 Mar 2003 23:16:21 -0500 Ben Collins <[email protected]> wrote:

You've made quite a marketing move. It's obvious to me, maybe not to others. By providing this CVS gateway, you make it almost pointless to work on an alternative client. Also by providing it, you make it easier

I don't think so. This also bits Larry. If he does well enought, there'll be some people here that won't use bitkeeper just because they can use the cvs gateway and they don't need/miss the features they could get with bk.

Quoted 8 lines

And i don't think it avoids creating a free bk clone. I guess that there's a lot of people out there interested in such tool, and not only for kernel development; this won't stop them.

As far as i can see; Larry is just wasting time (money) to help the kernel development and people who doesn't use BK just because it isn't free. And he's not charging me, so i find this a good movement for everybody. I only can say thanks.

You're missing the point. I am not against the CVS->BK gateway. I'm all for it. But it's kind of sour given that he now wants to change the disk format of the repo to make it harder to get the data from it.

If all he announced was "you now have a CVS->BK repo", I wouldn't be complaining, I'd be patting him on the back.

Signature · 4 lines
Debian     - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
Deqo       - http://www.deqo.com/
#26 lore raw
Quoted 3 lines · Ben Collins

You're missing the point. I am not against the CVS->BK gateway. I'm all for it. But it's kind of sour given that he now wants to change the disk format of the repo to make it harder to get the data from it.

Actually, that is purely YOU reading something into what he wrote. He didn't say "now I'm going to make the repo harder to get data from it". What he said was "now I'm free to change the format from SCCS to something that is more efficient for BK to use". Who knows, maybe the new format will be _easier_ to reverse engineer/parse using 3rd party tools?

Also, it's not like he can change things overnight, because there are lots of customers/users who have repos in the old SCCS format, and he doesn't want to completely throw away his current code just to piss off some whiny l-k users. At worst, if it bothers you so much, you can take up the now seemingly forgotten Linux trait of "taking things into your own hands and fixing it to your own needs" and write bk_evil_format_2_CVS conversion tool instead of bitching on l-k about it.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#33 lore raw
In reply to Larry McVoy

Well I'm a nobody in linux kernel land, but I'm a long time lurker on LKML and have been doing patchsets for the Gentoo Linux kernel for a while now.

I'd like to say that before Linus started using BK, close to 50% of the revision data that is now saved was completely lost in the process of him merging patches by hand into his repository. I mean be realistic, do you think that Linus kept perfect track of EVERY single ' ' ';' ')' that he changed when merging a patch with minor rejects with his repo? Do you think that every single time that he made a 1 line change or merged a 1 line change that was sent to this list it was documented and recorded? I doubt it. So now we are able to get a publicly available CVS repository with close to two times the data that was ever available before, and infinitely more than was ever available to anyone outside of Linus' own head.

I personally think that Larry has done an amazing job supporting this project and it's goals, and I will give him a big "Thanks for all your support and hard work" at this time. I think that those of you complaining about this as bitmover clearing the road to steal our data should take a long hard look at what you are really saying and consider what BK has given us that we never had before because nothing before was ever usable by Linus.

Thanks for reading (those who do),

--Brandon Low
Gentoo Linux Senior Developer

#12 lore raw
In reply to Roman Zippel
Quoted 3 lines · Roman Zippel

this was one of the conditions for the bk usage, so Larry cannot say, that you only get all the data if you use bk. If cvs can't represent all the information, we have to find another solution.

CVS can't represent it all because CVS isnt up to the job. If the rest exists as comments then its your problem to write a VCS that can extract the comment data and represent it in full

#17 lore raw
In reply to Larry McVoy
Quoted 14 lines · Larry McVoy
Quoted 2 lines

I thought that BK has been able to export everything to a text file since the first version.

bk export -tpatch -r1.900 > patch.1.900
bk changes -v -r1.900 > comments.1.900

Been there forever. So has ways to get all the metadata from the command line without having to reverse engineer the file format. See

http://www.bitkeeper.com/manpages/bk-prs-1.html

it's all there. Always has been.

Quoted 9 lines

Wayne wanted me to point that it is easy to write the BK to CVS exporter completely from the command line, we prototyped it that way, the only reason we rewrote part of it in C was for performance. The point being that you guys could have done this yourself without help from us because all the metadata is right there. Ditto for anyone else worried about getting their data out of BK now or in the future. The whole point of prs is to be able to have a will-always-work way to get at the data or the metadata, it makes the file format a non-issue.

This is a Good Thing[TM] for a whole bunch of reasons.

Maybe this output could be made available automatically in addition to the CVS tree? If bandwidth is a concern then I reiterate what I said offline yesterday, if you can give me a ballpark idea of what the requirements seem to be I'll start hunting for a place to park a .kernel.org server dedicated to this task.

-hpa

#27 lore raw
In reply to Ben Collins
Quoted 6 lines

You're missing the point. I am not against the CVS->BK gateway. I'm all for it. But it's kind of sour given that he now wants to change the disk format of the repo to make it harder to get the data from it.

If all he announced was "you now have a CVS->BK repo", I wouldn't be complaining, I'd be patting him on the back.

As long as we continue to get all the data in an open format, I'm not sure this really matters, personally. If there's some data loss, let's focus on that issue ... but it seems there isn't at the moment.

I'd rather we *didn't* go trying to clone BK and make it file-format compatible underneath ... that seems more incendiary than useful. Cloning other products is always a loosing game, the best you can do is catch them. Personally, I'd prefer we spent the effort making a usable simple SCM that 95% of us can use that does merges and stuff, and not bother trying to follow someone else in file format.

Of course, I'm in no position to dictate to others what they should implement, do what you like ... just my personal opinion. But there's always the possiblity we can make something that fits kernel development *better*, rather than playing catchup to BK all the time ;-)

M.

#28 lore raw
Quoted 4 lines

Of course, I'm in no position to dictate to others what they should implement, do what you like ... just my personal opinion. But there's always the possiblity we can make something that fits kernel development *better*, rather than playing catchup to BK all the time ;-)

I like it when I agree with people, especially you since we've bumped heads. It's much more fun to agree...

My personal opinion is that BK maps only so so well onto the kernel development effort. It's not horrible, it's closer than any other SCM, but it could be better. The kernel guys tend to be "more loose" than commercial guys, i.e., stuff is tried, it sits in Alan's tree for a while or DaveJ's tree and then is rejected if it is found to be bad. You really need a sort of "lossy" SCM system, one which is willing to throw data away. BK is absolutely not about losing information, we view everything as valuable, even bad ideas. That matches the commercial world better than the Linux world.

I _think_ that Arch is closer. You will definitely give up some stuff if you move to Arch but you will also gain some stuff. Arch is willing to pick and choose, we aren't, we're sort of an all or nothing answer. Pavel is all hot and bothered about PRCS but PRCS is sort of BK without the distribution, gui tools, and scripting. It's a step backwards as far as I can tell (don't get me wrong, we've acknowledged the coolness of PRCS on our website for years and I tried to team up with Josh, I'm a fan). You should really look at Arch, it may be a better fit. And these days, if you could find a better fit, none of us at BitMover would shed a tear if you moved off BK. This has *not* been a pleasant experience for us.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#19 lore raw
In reply to Larry McVoy
Quoted 7 lines · Larry McVoy
Quoted 2 lines

I thought that BK has been able to export everything to a text file since the first version.

bk export -tpatch -r1.900 > patch.1.900
bk changes -v -r1.900 > comments.1.900

Been there forever.

More importantly, even if someone isn't allowed to use the BK command line tool because once upon time, a long time ago, they submitted a patch to arch or subversion, they can still find someone is allowed to set up a bk daemon under the terms of the FUL, connect to the BK daemon using a http client, and extract the full diff of any changeset that way. This doesn't have to be the bkd on bkbits.net; anyone who is authorized to use BK under the terms of the FUL can set up a bk daemon to be listening on a port of any machine for which they have shell access (it doesn't even require root privs). And every last changeset can always be made available using this path.

So to the people are complaining that they won't be able to get out their data if a future version of BK uses a more powerful representation than SCCS files ---- would you like some more whine with your cheese?

- Ted

#20 lore raw

Hi,

Quoted 4 lines · Theodore Ts'o

So to the people are complaining that they won't be able to get out their data if a future version of BK uses a more powerful representation than SCCS files ---- would you like some more whine with your cheese?

Oh, thank you, but I'd like some of the stuff that all the bk fan boys must be taking.

bye, Roman

#13 lore raw
In reply to Alan Cox

Hi,

Quoted 7 lines · Alan Cox
Quoted 3 lines

this was one of the conditions for the bk usage, so Larry cannot say, that you only get all the data if you use bk. If cvs can't represent all the information, we have to find another solution.

CVS can't represent it all because CVS isnt up to the job. If the rest exists as comments then its your problem to write a VCS that can extract the comment data and represent it in full

This would require the full data, I looked at it and neither the CVS tree nor the Web interface has everything.
- the changeset ids are missing, if bk does its renumbering thing ids in comments become useless
- complete changesets are missing, RCS is quite capable of branches, so there is no need of merging patches into a single patch.
- merge information is missing, e.g. what branch was merged into which changeset.

I attached an example RCS file, how that could look like. I had to guess the merging information and tags are missing.
It would be really useful to have this information as well.

bye, Roman

#18 lore raw

BK->CVS (2.4 + 2.5 updates)

In reply to H. Peter Anvin

Updates from the latest 2.4 and 2.5 BK trees have been applied to the kernel.bkbits.net tree. If you did a checkout, try an update and let me know how it goes.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#34 lore raw
In reply to Larry McVoy

Hi!

Quoted 4 lines

(actually Wayne Scott) did was to write a graph traversal alg which finds the longest path through the revision history which includes all tags. For the 2.5 tree, that is currently 8298 distinct points. Each of those points has been captured in CVS as a commit. If we did

As far as I can see, linux-2.5 repository has over 17000 ChangeSets, that means half the granularity. Would it be possible to use cvs branches to capture tree structure and have special form of commit comment "this is merge of changeset 1.2.3.4"?
That way BK->CVS conversion could
preserve all the data...
Pavel

Signature · 2 lines
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...
#29 lore raw
In reply to Ben Collins
Quoted 16 lines · Ben Collins
Quoted 15 lines · Arador
Quoted 3 lines · On Tue, 11 Mar 2003 23:16:21 -0500 Ben Collins <[email protected]> wrote:

You've made quite a marketing move. It's obvious to me, maybe not to others. By providing this CVS gateway, you make it almost pointless to work on an alternative client. Also by providing it, you make it easier

I don't think so. This also bits Larry. If he does well enought, there'll be some people here that won't use bitkeeper just because they can use the cvs gateway and they don't need/miss the features they could get with bk.

And i don't think it avoids creating a free bk clone. I guess that there's a lot of people out there interested in such tool, and not only for kernel development; this won't stop them.

Quoted 8 lines
Quoted 4 lines

As far as i can see; Larry is just wasting time (money) to help the kernel development and people who doesn't use BK just because it isn't free. And he's not charging me, so i find this a good movement for everybody. I only can say thanks.

You're missing the point. I am not against the CVS->BK gateway. I'm all for it. But it's kind of sour given that he now wants to change the disk format of the repo to make it harder to get the data from it.

Ben,

You shouldn't care less of the disk format. You *can't* run bk in the first place to reach those files, it's by pure luck that somebody is been fine to give away his right to write free software (oh and proprietary software too but we don't care :) in the SCM arena and to provide this info to you via rsync or whatever proxy or open protocol that tytso mentioned is doable.

Before you can remotely care about the disk format you've to reverse engeneer the network protocol first, having more proprietary stuff there won't make differences for us. And of course it makes perfect sense for Larry to hide the stuff better, but even if he encrypts it, the secret key has to be in the bk binary, I mean, it's all in open source assembly anyways, if you figured out the network protcol, you shouldn't have an order of magnitude more of troubles to figure out the new file format too. NOTE: I don't want to discuss the legal details of reading the open source assembly, this was only an example ;).

really, what we care is the data, and what I discussed in the last weeks with Larry about the kernel CVS at first sigh seems enough for kernel developers, what matters is the _mainline_ evolution. All other trees matters much less (and NOTE: all important non mainline trees don't use bitkeeper anyways). If getting the changesets with dates will be too hard I assume Larry could help on it. Some script should do it pretty well thanks to the logical tag in the log. I know it's not the most useful format for export but this is reliable, documented and open and it makes it trivial to checkout and search the file logs. which makes it very usable immediatly w/o the need of new software which is good for us kernel developers in the short term. This is a good short/mid term solution.

IMHO cloning bitkeeper would be an option if Larry would be supporting it, but that is obviously not the case.

There is no point to complain about the change of format of files in the Larry has all the rights to change the file format even after you reverse engeneered it the first second third fourth time, so all your effort will break in seconds. You can spend the rest of your life to keep up with Larry and he'll always be ahead of you. We have to do this with the SMB protocol because there's no open ""exporter"", but here Larry provided the data, and the data belongs to the community in the first place so there's no need to slowdown innovation here trying to catch up with closed proprietary protocols. And note: if you don't like that linux is developed with bk you should speak with Linus not with Larry. That is Linus's choice, Larry couldn't make that change.

If you complain about the file format change, it means you realized right now you did a mistake in depending on bk in the first place.

I think we reached a point of balance here that will solve all the collisions. The CVS is a "stability" point. The lack of data-availability with CVS or similar open protocol would force us to reverse engeneer bk to access the data, and the availability of CVS immediatly make us wasting time reverse engeneering bk. Cloning bitkeeper is a waste of time if the CVS just exports the data correctly.

Please focus on this: the only thing we miss is the visibility of the jfs tree and similar other bits that aren't even guaranteed to be merged in mainline. But that doesn't worry me at all, in one year from now if the jfs tree didn't merge correctly it won't matter what was in such dead tree.

If you want to contribute, stop these threads, and start importing CVS into a more powerful SCM and let us know an URL where we can access the data from there. I will only answer to a working URL, either that or live with CVS. The SCM can be evolved over tiem. If this new underground domain will be better than bitkeeper than jfs and Linus as well could join us in the future. In the meantime CVS will do fine and it guarantees the openess of the linux info. As you probably know I don't have much time in helping with SCM developement by I can t try give my $.02 (or at the very least I want to be still allowed to give my $.02! ;).

I won't answer further emails about this issue to avoid hurting the l-k traffic too much (the last bk threads even made me overlook the BK->CVS announcement after a one day and half of email backlog go figure ;).

And of course many thanks to Larry for the BK->CVS effort! While I think it was due, it certainly takes some relevant effort to do it.

Andrea

#35 lore raw
In reply to Pavel Machek

From: Pavel Machek <[email protected]>

Quoted 2 lines

As far as I can see, linux-2.5 repository has over 17000 ChangeSets, that means half the granularity.

I assume this has already been answered since this is Monday morning and I haven't finished my mountain of email (I try not to read it on weekends), but I will answer this anyway.

The ChangeSet file has many csets and we only capture around 1/2 of them in CVS ChangeSet file. The extra ChangeSets are grouped together with the merge cset where they were added to the path we are recording. That is correct, but it is not the whole story.

What happens is that most csets modifiy a non overlapping set of files. So while we didn't get every delta to the ChangeSet file, we did capture >90% of the actual changes to the source files in the tree.

Perhaps that will help explain things.

-Wayne

#36 lore raw

Hi!

Quoted 12 lines
Quoted 2 lines

As far as I can see, linux-2.5 repository has over 17000 ChangeSets, that means half the granularity.

I assume this has already been answered since this is Monday morning and I haven't finished my mountain of email (I try not to read it on weekends), but I will answer this anyway.

The ChangeSet file has many csets and we only capture around 1/2 of them in CVS ChangeSet file. The extra ChangeSets are grouped together with the merge cset where they were added to the path we are recording. That is correct, but it is not the whole story.

Quoted 4 lines

What happens is that most csets modifiy a non overlapping set of files. So while we didn't get every delta to the ChangeSet file, we did capture >90% of the actual changes to the source files in the tree.

Oh, so there's extra magic.

Question, through: why is it impossible / infeasible to use CVS branches to capture *full* information? Merge would then say "(changeset 1.2345, merge from 1.23.4.5)" or similar...

Pavel

Signature · 2 lines
Horseback riding is like software...
...vgf orggre jura vgf serr.