Is the BitKeeper network protocol documented?

Dear Larry,

Please don't take this as a contentious question. I have an honest request, and would like a technical answer without politics if that's possible.

Can't use BitKeeper
-------------------

I'll explain where I'm coming from.

Recently, I was debating with Linus about the new vsyscall code. To keep track of Linus' changes, so that I might make better comments and produce a patch or so, I had a need to track the head of the kernel's bitkeeper repository at bkbits.net. (The experimental code was not yet available as a normal patch from Linus).

Although I am unfortable using closed source software, it seemed pragmatic to fetch and install BitKeeper. I went to bitmover.com, and read the free license before downloading:

http://www.bitkeeper.com/Sales.Licensing.Free.html

That looked ok. I am allowed to use it. Great!

So I downloaded version 3.0, and typed "bk help bkl". I found that the license with the software is _different_ to the licence on the web page.

[Note to Larry, you may wish to update the above URL to the current version].

Unfortunately, the license that comes with the download adds a new clause 3(d): that's the clause which tells me that actually I'm not allowed to use BitKeeper, because of other software I occasionally work on. (No, I do not work on Subversion, but I do occasionally dabble with sophisticated version management scripts).

So, being conscientious and obedient, I removed BitKeeper from my system.

As a result I had great difficulty having a meaningful debate with Linus - as I had no easy way to look at the code Linus was checking in, that we were talking about! (And submitting a patch to illustrate my thoughts was out of the question).

Real-time kernel tree only available over BitKeeper protocol?
-------------------------------------------------------------

To synchronise with the kernel repository, in order to communicate effectively with Linus about changes as they are checked in, I think that it's necessary to use the BitKeeper network protocol (or the over-HTTP equivalent).

I know that Rik van Riel keeps a mirror of the repository in various formats over at nl.linux.org:

	http://ftp.nl.linux.org/linux/bk2patch/

Thus far, the best solution I have for tracking checkins is to rsync the SCCS files from Rik's mirror, and use a Perl script to extract the head version from each SCCS file. (I could use GNU CSSC, but for this purpose a simple Perl script is enough; the SCCS file format is quite simple).

However, as is the nature of mirrors, I'd rather not have to wait for the time delay getting updates to Rik's mirror. Not to mention the lack of tree-wide atomicity, if I rsync at the wrong moment (I am not sure if this is a problem in practice).

Anyway, the point is I would like to be able to access the "official" kernel development tree, in real time like everyone else, which I understand is only available at bkbits.net.

As far as I know, the only way to follow updates to the offical tree is using the BitKeeper network protocol.

And I have not been able to find any documentation of that protocol. (I hope it is not necessary to reverse engineer it!)

My question
-----------

Larry, or anyone else, can you direct me to the information I need to track the kernel development tree in real time? A document describing the BitKeeper network protocol would be ideal.

I don't require to use the bk protocol - but if it that is as efficient as you say on the bitmover.com web site, that would be nice.

Please note that I am _not_ writing a bk clone, or any other significant VC project. However I do wish to use my own software to analyse changes to the Linux kernel as they are checked in, and I think that is a reasonable request.

Thanks in advance,
-- Jamie Lokier

I'm starting to think that one cannot legally use BitKeeper as the
preferred means of developing a GPLed program. The problem is, the GPL defines the source as the preferred base to modify the software from and requires you to be able to distribute the source without any additional licensing requirements.

If BitKeeper is the version management tool, then BitKeeper is part
of the source by this definition. Providing the source in BK form without BK is as useless as providing it encrypted. Providing it in any other form does not satisfy the GPL (assuming that BK form is in fact the preferred way of modifying it).

DS

It doesn't require that you distribute the tools for editing the source, though. For example I believe it is fine to distribute a program for Microsoft Visual Studio, in the form of the files you would actually use with Visual Studio, even though the format of some of those files is not documented.

I disagree, because the BK file format is actually quite well documented - it is SCCS with some annotations that do not seem essential if you are using a different tool.

The data is easily extracted using an SCCS-compatible tool. It is certainly not encrypted, and I had no difficulty writing a Perl script to extract any version of the source, although I have yet to look if changesets are so easy as individual files.

Credit to Larry, for choosing an easily read file format.

(Although not perfectly - see the CSSC documentation for some things that they are not sure how to decode in an SCCS file - and yes, those do appear in BK-generated SCCS files from time to time).

Linus and other people have said repeatedly that BitKeeper is _not_ essential to working with them on the kernel.

That said, it does seem that if you can't read bkbits.net, then you are at a disadvantage sometimes.

-- Jamie

So then suppose the tool I use for modifying the source code
unpacks/decrypts it, allows changes, and then packs/encrypts it again. Suppose further that this tool is proprietary and not available without onerous licensing requirements. Would you say releasing the source code thus packed/encrypted meets the GPL?

If not, then what would? The decrypted/unpacked form of the source
is not the preferred form for making modifications.

It seems to me that if you can't distribute the source in its
preferred form for modification such that it can actually be used and modified without complying with some other more restrictive license, you cannot comply with the GPL. The alternative is to say that you can distribute utterly useless "source" and still comply with the GPL.

Anyway, this has veered off-topic for this list. I apologize for
that.

DS

I think that if you distribute a program in Emacs-Lisp, but you don't provide the Emacs-Lisp interpreter, that is considered ok. If you distribute a program in Visual Basic under the GPL, that is considered ok too. Similarly if it's a GPL'd Excel spreadsheet macro, or a program written in Jonny's own version of Prolog.

However if you distribute obfuscated or encrypted files, then clearly that's not the preferred form for making changes. If it's encrypted, the preferred form obviously includes the decryption key. (And if the code has to be signed to run, it might include the signing key - ooh).

I don't know where the line in the sand stops. It's not something GNU people seem to worry much about, and neither do I as it is usually quite clear cut one way or the other.

         -----

About BitKeeper: if it were actually essential, I think you'd have a point. But it isn't.

However, this begs another question: the kernel source is GPL'd. But is the _repository_ at bkbits.net GPL'd? And if so, do I have the right to demand a copy of the whole repository whenever I receive a binary kernel, or is that right restricted to the checked out files used to compile that kernel?

cheers,
-- Jamie

It is not the preferred form for editing the source code, so it is not the real source code as defined by the GPL.

However, this begs another question: the kernel source is GPL'd. But is the _repository_ at bkbits.net GPL'd?

I believe the contents are all under the GPL.

And if so, do I have the
right to demand a copy of the whole repository whenever I receive a binary kernel, or is that right restricted to the checked out files used to compile that kernel?

Whoever distributes a binary kernel to you has the obligation to offer you the complete source code corresponding to the binary. Source code not used in producing that binary need not be included.

Do you not use

	http://www.kernel.org/pub/linux/kernel/v2.5/testing/cset/

?

It always has the latest diff against the last-released kernel.

I snarf it hourly, so I have decent granularity for doing the binary-search-to-see-where-it-broke trick.

#!/bin/sh

URL=ftp://ftp.kernel.org/pub/linux/kernel/v2.5/testing/cset/

cd /opt/downloads/bk
rm -f index.html
wget --quiet $URL/index.html

VERSION=$(grep 'patches since' index.html | \
		head -1 | \
		sed -e 's/.*since \([^:]*\).*/\1/')

mkdir -p $VERSION
cd $VERSION
wget --quiet --timestamping --recursive $URL 2>&1

Oh, thanks. I didn't know about that.

It is updated in real time then?
(Assuming yes) that reduces the need to talk bk protocol quite a lot :)

(I'd still like to, though).

Good idea.

-- Jamie

As far as I can tell your complaint is that you can't have access to the up to minute source view without using something which violates your politics.

The fact that you can get almost real time views via one of many BK to tarball/patch mirrors seems to not be good enough.

I guess I don't know how to help you. As far as I can tell, if Linus wasn't using BK he'd still be doing what he was doing up until he started using BK which means you wouldn't have the option of the up to date snapshots you can currently get.

The basis of your complaint seems to be "BK makes some things possible which weren't possible before, my politics don't let me use BK but I want the advantages which would come from using BK". I'm sorry, but I don't know what to do to help you. The part of BK you'd like me to disclose is something that we consider quite valuable and unique to our product and we have no intention of disclosing how it works.

I fail to see why this is such a big deal, you now have up to the hour snapshots in the form you want where before you had to wait weeks between releases. That's a dramatic improvement over what you had a year ago and complaining that you can't have up to the minute views of the source when the only reason is your politics, well, is it going to seem really unreasonable if I think that maybe your politics are getting in the way of your technical goals?

---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

I agree with your sentiment, but you mis-characterize one thing. Jamie was stating that his interpretation of the BitKeeper license forbade him to use the free version of BitKeeper because of some of his non-kernel related activities. This does seem to be a fair interpretation of the license, according to clause 3-d, and has nothing to do with his politics. Jamie is stating that it would be illegal for him to use BitKeeper. Whether or not you agree with the use of BitKeeper for linux kernel maintenance, it would seem like this is an unnecessarily onerous clause that prevents some individuals from participating on an equal footing with everybody else.

-Tupshin

Clause 3-d:
Notwithstanding any other terms in this License, this License is not available to You if You and/or your employer develop, produce, sell, and/or resell a product which con- tains substantially similar capabilities of the BitKeeper Software, or, in the reasonable opinion of BitMover, com- petes with the BitKeeper Software.

No, without something that violates your license. Nice of him to actually respect it :-)

Yes, a huge thank-you for making this possible... it's easy to forget that the max wait time is now an hour, and it used to be weeks... linus's brain is a much harder protocol to mirror than bk :-)

Well, I would point out that it's not politics, but rather respect for your licensing terms that prevents him from using bk. (this part got snipped relatively early, maybe you missed it)

So, as you said you would consider case by case license grants if this clause became a problem when it was last discussed (IIRC anyway, I don't mean to put words in your mouth if I'm remembering that thread wrong), maybe this would be a good time for one. Or he can use the hourly changeset mirror :-)

I've sent Jamie mail asking him why he things 3(d) is a problem for him, we'll see what he says. If he's working on things that compete with BK then the answer is no, if he's not, then there is no problem.

We want to help the kernel team, that should be obvious. I draw the line where helping the kernel team hurts BitMover, as would any of you in my position. Fortunately, it's pretty rare that anyone talented enough to work on the kernel also wants to work on source management.

We could change the license to have the standard legalese which says you can't reverse engineer, etc. If the community at large would prefer that, we could discuss it. I suspect that when you realize the implications of
that legalese, the BKL will seem a lot nicer by comparison. Would you
rather have something like:

You may not yourself and may not permit or enable anyone to: (i) modify or translate the Software; (ii) reverse engineer, decompile, or disassemble the Software or otherwise reduce the Software to a form understandable by humans, except to the extent this restriction is expressly prohibited by applicable law notwithstanding this limitation; (iii) rent, lease, loan, resell or cre- ate derivative works based on the Software; (iv) merge the Software with another product; (v) separate the Software into its component parts; (vi)

 copy the Software, except (A) as expressly provided herein and (B) as reason-

ably necessary for back up and recovery purposes; or (vii) remove or obscure any proprietary rights notices, labels, copyrights, trademarks, servicemarks, confidentiality notices and/or restricted rights notices on or in the Soft- ware.

---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Oh, I agree that BK makes things possible that didn't happen before. The only downside is that now Linus _assumes_ you can see what he's doing - because that makes less work for him - which is fair enough really. That's the best part of decent version management!

Anyway, I am mostly satisfied with the hourly patch link that Andrew Morton pointed me to.

Thanks,
-- Jamie

Well. Many source management tools would give us the things we are now enjoying. The main difference is that Linus is actually using one now.

Hi,

IOW "You should be thankful for what I offer, if you don't like it, piss off!"
Might not be what you've intended, but that's what I arrived here and I'm sure I'm not the only one.

bye, Roman

That's what he intended, and rightfully so. You now have things you didn't have before (i.e. hourly snapshots of Linus' tree) and you still aren't happy. I guess some people will never be happy with anything, so there is no point in trying to appease them.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Hey, I only asked, and I made it clear what I would be happy with.

-- Jamie

Hi,

I just wanted to make sure I understood correctly, I have an appropriate answer, but I can't word it as nicely as Larry.

If you don't see the problem, maybe you should read /usr/src/linux/COPYING again for a change.

bye, Roman

There is nothing in the GPL which requires anyone to make their changes available to you the minute they make them. The fact that you have access to the changes within an hour of when they are made far exceeds the requirements in the GPL, which only require that the source code be made available if you distribute the OBJECT CODE OR EXECUTABLE.

If Linus uses BK to make pre-releases available to some people, that does not appear to even invoke the "executable distribution" clause, any more than him emailing patches to other developers privately. If Linus started making kernel patches available via a Lotus Notes database (heaven forbid, I think even the IBM folks agree on that one ;-) doesn't mean that IBM suddenly has to make all the details of Lotus Notes available, or that Linus is forbidden to use tools as he wants. There are still lots of other ways to get the kernel source.

In fact (think about this for a while 8-o), EVERY SINGLE CHANGE that has gone into the "official Linus kernel" had Linus doing the merge (i.e. acting as editor of the combined work which is the kernel), which may imply that Linus even owns a complete copyright over the entire kernel source tree (i.e. compiled work). Since he has never (or not in the last decade, AFAIK) distributed a binary or object version of the kernel, it may be that he isn't under any obligation to do anything related to distribution under the GPL. If you think that being the editor of a combined work is not enough to give him copyright over the combined work, then you need to learn your copyright law a bit more. If it wasn't for Linus acting as a "gatekeeper", the kernel would be full of the crap that makes up 99% of the sourceforge projects out there.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

I think you're ignoring the way the GPL defines the "source code".
The GPL defines the "source code" as the preferred form for modifying the program. If the preferred form of a work for purposes of modifying it is live access to a BK repository, then that's the "source code" for GPL purposes.

You are using the conventional meaning of "source code", which is
roughly, "whatever you compile to get the executable". However, this is not the "source" for GPL purposes. For GPL purposes, the source is the preferred form of a work for purposes of modifying it.

This means you can't remove meta information that's useful for
modifying because that is not the preferred form. Such meta information includes whatever is useful for modifying it, such as revision history and chain of custody.

You can't have two "source"s, one a private repository that you
prefer to use for making changes and the other an "obfuscated" public version you distribute for GPL compliance which is missing all the other useful information.

Checking source out of a repository, separating away the revision
history, is an obfuscatory act. The GPL prohibits such source obfuscation and requires you to distribute the source in whatever is the actual preferred form for modifying it. Really. Sorry.

DS

I think you are ignoring the fact that this clause (#3) in the GPL only relates only if "you copy or distribute the Program (or a work based on it, under Section 2) in object code or executable form".

Again you are ignoring the fact that there are other methods by which the source code is available in the "preferred form", just not quite as timely as directly from the BK repository (which is itself in a form, SCCS, which does not require BK to access), and there is nothing in the GPL which requires that the source be made avaible instantly.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Here's the problem. RedHat ships the code in "object code or
executable form". This requires them to distribute the source code in the "preferred form for modifications". The problem is, the preferred form for modifications might well be Linus' BK tree, which RedHat might not even have!

If you assume that live access to Linus' BK tree is the preferred
form for modifying the Linux kernel, then RedHat can't ship a compiled kernel if they can't give people access to Linus' repository.

The GPL is nonsense. Lots of people have suffered absurdities
similar to this one in a crazy attempt to comply with it. I think if the people who *chose* it had to suffer its insanities a little, they'd think twice the next time they choose a license for their open source projects.

DS

You are a lawyer working in this area, and so can cite chapter and verse where this definition was made (the GPL text is rather vague)?

Anyway, Andreas Dilger is (curiously) right AFAIU: Linus has _never_ distributed a binary to anybody AFAIK, so he is under no obligation by the GPL do give out any form of source. Furthermore, as he is (in the editor sense at least) copyright holder for the whole source, he isn't bound by the GPL in any case. ;-)

IANAL, etc.

Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Nobody knows, that's definitely part of the problem. If you
genuinely want to make a good faith effort to comply with the GPL, I'm not sure what you can do other than guess.

The problem then occurs with companies like RedHat. They distribute
binaries, so they must distribute the source in the preferred form for making modifications to it. *If* metainformation in Linus' BK tree is part of the preferred version of the work for the purposes of making modifications to it, then RedHat *cannot* comply with the GPL.

Checking source code out of a repository is an obfuscatory act that
separates the raw source code from the rationale for that source code. It's equivalent to stripping comments. The GPL does not allow you to obfuscate the source, so if all *you* have is obfuscated source, *you* cannot ship binaries.

I don't think this is currently an issue for the Linux kernel.
However, it may well be an issue for projects using things like sourceforge or using proprietary file formats to hold portions of their source.

DS

David Schwartz <[email protected]>

Well, as a license is in escence a promise not to sue you for using my property as long as you comply with certain conditions, it will then be up to the licensors. If Linus is OK with distributing just tar.bz2's, its OK for the kernel. Also, RMS specifically said using bk doesn't make the repository source in the GPL sense (this is presumably the intention the FSF will put forward, and which most other GPL-licensing parties will agree).

IANAL (and happy for it ;-)

Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Nobody knows, that's definitely part of the problem. If you
genuinely want to make a good faith effort to comply with the GPL, I'm not sure what you can do other than guess.

The problem then occurs with companies like RedHat. They distribute
binaries, so they must distribute the source in the preferred form for making modifications to it. *If* metainformation in Linus' BK tree is part of the preferred version of the work for the purposes of making modifications to it, then RedHat *cannot* comply with the GPL.

Checking source code out of a repository is an obfuscatory act that
separates the raw source code from the rationale for that source code. It's equivalent to stripping comments. The GPL does not allow you to obfuscate the source, so if all *you* have is obfuscated source, *you* cannot ship binaries.

I don't think this is currently an issue for the Linux kernel.
However, it may well be an issue for projects using things like sourceforge or using proprietary file formats to hold portions of their source.

DS

Hi,

I knew I should have been more specific. It would have been enough to read and understand the preamble.
"By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. [..] When we speak of free software, we are referring to freedom, not price."
The GPL is intended to protect our freedom. How does BK fit in here? BK is not free and even worse not everyone is allowed to use it. You don't see a small discrepancy here?
The few who are allowed to use it have to use considerable extra effort to make the source available to people who can't or don't want to use BK. How does this help to promote freedom? Is the convenience of a few really helping here?
The actual license is more for lawyers, but for the users it's a lot more important to at least understand the preamble. It's a real pity how easily users forget about this and only think of their own short term advantage. Only because they can use something for free, they believe they gained some kind of freedom, but what is this "freedom" worth if it depends on the mercy of others or it can't be shared with others? In the end it's the decision of every user what they do, but at least they shouldn't fool themselves.

bye, Roman

Not his politics. These no-competition clauses quickly extend to where you don't expect them, unintentionally.

I'd translate it to the old "have a GP/BSD-licensed tool to just check out the stuff" (and leave modifications to some party that is entitled to the full BK version by whatever means, no competition + openlogging for instance or something). You mentioned CSSC in the past, but AFAIR, that's for local uncompressed repositories only.