Linux Kernel Subversion Howto
Quoted 10 lines · Nicolas Pitre
Quoted 3 lines · Larry McVoy
You know, you could change all this. Instead of complaining that we are somehow hurting you, which virtually 100% of the readers know is nonsense, you could be producing an alternative answer which is better.
IMHO something is flawed in this whole argument. Why would someone be interested into any alternative answer for working on the Linux kernel tree if the whole thing can't be imported into it with the same granularity as can be found in BK? IOW what's the point to alternatives if you can't retrieve the entire workset?
Please explain to me where the data is being lost. 100% of the patches are available on bkbits.net with no license agreement required. They always have been.
The problem is that you want us to tell you how BK manages those patches. That was never part of the agreement, in fact we made it clear in the license that that information was considered to be IP and could not be distributed. How BK does that is our business, not yours. If you want to know how BK does it you must go figure it out without the benefit of BK itself or metadata managed by BK.
While I understand that you don't like it, is there no sense of fairness left? We did the hard work to create BK. Some of us worked for *years* without pay to create this product. Some of us put our life savings into the product. It's our IP, we paid heavily to create this, is it so unreasonable of us to want to protect our work? I believe we are within our legal rights, or so our legal team tells us, but that should be beside the point. It's our work. We paid for it. We certainly don't have any obligation to tell you how we did it and to us it seems pretty unreasonable that you don't just go off and do the work yourself. And pretty unadmirable as well, don't you have any faith in your own abilities?
Signature · 2 lines
--- Larry McVoy lm at bitmover.com http://www.bitkeeper.com
Quoted 12 lines · Larry McVoy
Quoted 10 lines · Nicolas Pitre
Quoted 3 lines · Larry McVoy
You know, you could change all this. Instead of complaining that we are somehow hurting you, which virtually 100% of the readers know is nonsense, you could be producing an alternative answer which is better.
IMHO something is flawed in this whole argument. Why would someone be interested into any alternative answer for working on the Linux kernel tree if the whole thing can't be imported into it with the same granularity as can be found in BK? IOW what's the point to alternatives if you can't retrieve the entire workset?
Quoted 3 lines
Please explain to me where the data is being lost. 100% of the patches are available on bkbits.net with no license agreement required. They always have been.
Are you saying that it is now OK to write scripts that would bit bang on the bkbits http interface to fetch patches/comments with the purpose of populating an alternate scm? Andreas tried that a while ago but you threatened to shut the service down entirely if he was to continue.
Quoted 4 lines
The problem is that you want us to tell you how BK manages those patches. That was never part of the agreement, in fact we made it clear in the license that that information was considered to be IP and could not be distributed. How BK does that is our business, not yours.
I completely agree here. But just to be on the same page, let me quote Stelian's proposal here again:
On Tue, 8 Feb 2005, Stelian Pop wrote:
| What you could make available in the bkcvs export would be, for each | changeset (both for the changesets and for the merged changesets), | include three BKRevs: the changeset's one, the changeset's parent one, | and the changeset's merge parent one. | | That information could be used to reconstruct the entire tree, | using either bk-commit-head (preferred) or bkbits, provided you | put the BKRev: tag into the bk-commit-head posts too. | | Technicaly speaking this should be very easy for you to implement.
Is the above actually part of the protected BK IP? Can anyone run with such information and clone BK in a smooth breeze with that? I don't see how such info can tell how BK manages it.
Quoted 3 lines
If you want
to know how BK does it you must go figure it out without the benefit of BK itself or metadata managed by BK.
I think what people want here is the tree structure representation in whatever form, not necessarily in the BK format. One example of that was provided above. You can't deny others access to the whole of the Linux kernel development history even if their purpose is to import it into a competing system (more on that below).
Quoted 9 lines
While I understand that you don't like it, is there no sense of fairness left? We did the hard work to create BK. Some of us worked for *years* without pay to create this product. Some of us put our life savings into the product. It's our IP, we paid heavily to create this, is it so unreasonable of us to want to protect our work? I believe we are within our legal rights, or so our legal team tells us, but that should be beside the point. It's our work. We paid for it. We certainly don't have any obligation to tell you how we did it and to us it seems pretty unreasonable that you don't just go off and do the work yourself.
Again I wholeheartedly agree with you above. But that's not exactly the point. You certainly have the right to protect your work. But please admit that the Linux kernel developers own the right to move (100% not 96%) of the development tree with all its branches _they_ produced. In other words, the product of the current Linux BK repository is the result of those Linux developers not yours. Of course BK is the indispensable tool that made it possible, but BK could have been developed even without the Linux BK repository.
So what people are asking for again and again is a way to represent in alternate form to BK internal metadata the Linux development tree structure but without you to give away the BK IP, be that in XML, plain text, or just with BK refs attached to patches like suggested above. Unless I'm completely dumb, this won't reveal anything about BK itself and how you did it?
Now obviously enough some people will run away with that raw data and try to import the Linux kernel source tree in their own scm of choice. That's why I'm asking you "and so what?" Granted all the efforts you put into BK as you say above, do you really expect the alternative scm to suddenly be as functional and scalable as BK in a single swoop just because they can import the Linux BK tree with the same patch granularity as BK does? If you worked so hard on BK for many years it means the competition should be far behind. Therefore what do you fear?
Note that if someone actually needs a big tree to test bench an alternate scm there are alternatives to the kernel -- gcc is a good example. So allowing the Linux kernel tree to be imported into another scm is not really a requirement for developing on said scm.
Quoted 2 lines
And pretty unadmirable as well, don't you have any faith in your own abilities?
Do you have faith in your own work? If so allowing others to import the complete Linux development tree into alternative scm shouldn't scare you so much. You probably know already that they won't be able to scale as well as BK, and we're not even talking about the ability to perform push and pull type operations here. So what do you actually fear? I'll be fully on your side once I grasp the actual threat to BK.
Note I'm trying to understand what the actual problem is here. I'm not concerned by the non-free nature of BK at all. It is the best tool for distributed development available, and it's been applied to Linux development with no $ cost. I'm just wondering why providing some additional info which would allow for rebuilding the tree with all changeset relationships (into whatever alternate inferior scm that's not the point) would uncover your IP. It's just a graph with patches attached to it in the end. No need to tell us how BK manages it.
Nicolas
Quoted 6 lines · Nicolas Pitre
Are you saying that it is now OK to write scripts that would bit bang on
the bkbits http interface to fetch patches/comments with the purpose of
populating an alternate scm? Andreas tried that a while ago but you threatened to shut the service down entirely if he was to continue.
Go for it, if it becomes a problem we'll rate limit it. Our first concern is that the BK users can get at the trees so if you are eating up the bandwidth too much we'll slow down the web interface.
Quoted 7 lines
| What you could make available in the bkcvs export would be, for each | changeset (both for the changesets and for the merged changesets), | include three BKRevs: the changeset's one, the changeset's parentone,
| and the changeset's merge parent one.Is the above actually part of the protected BK IP?
Yes.
Quoted 2 lines
I think what people want here is the tree structure representation in whatever form, not necessarily in the BK format.
That's fine, they can do that. Get the patches and figure out how to put them back together. These people do know how to use patch, right? OK, then they are welcome to patch things in, when they don't work, find a place they do work and create a branch, patch them, repeat. Haven't they ever dealt with a patch reject before? It's not that hard.
Quoted 3 lines
You can't deny others access to the whole of the
Linux kernel development history even if their purpose is to import it into a competing system (more on that below).
I'm not denying anyone that. The history consists of the diffs and the checkin comments, you have that.
Quoted 4 lines
Again I wholeheartedly agree with you above. But that's not exactly the point. You certainly have the right to protect your work. But please admit that the Linux kernel developers own the right to move (100% not 96%) of the development tree with all its branches _they_ produced.
_They_ didn't produce the branching structure, BitKeeper did that. Go look, there isn't a way to type a command which produces a branch in BK. So claiming that metadata is property of the developers is nonsense, they didn't produce, it isn't physically possible for them to produce it.
That's part of BK's design and power. 100% of what any developer produced is already available on the web, in the form that has been used for more than 10 years as the preferred form, a unified diff. And we added comments because those are useful and you typed them in. You guys have been importing patches for more than a decade, since when did that become a problem?
Quoted 3 lines
Now obviously enough some people will run away with that raw data and try to import the Linux kernel source tree in their own scm of choice. That's why I'm asking you "and so what?"
That's fine if they want to do that, they have the patches. What they don't get is the tree structure that BK has because that gives them the ability to go back and forth and say "well, BK did it this way and it worked, why doesn't it work in our system?".
Quoted 4 lines
Note that if someone actually needs a big tree to test bench an alternate scm there are alternatives to the kernel -- gcc is a good example. So allowing the Linux kernel tree to be imported into another scm is not really a requirement for developing on said scm.
Indeed. I don't suppose there is any chance you could get people to go play with the gcc tree?
Quoted 3 lines
I'm just wondering why providing some additional info which would allow for rebuilding the tree with all changeset relationships (into whatever alternate inferior scm that's not the point) would uncover your IP.
I think you are fishing for BK internal information and I'm not going to bite. The bottom line is that we laid out some rules, the BK users agreed to them, that's the deal. If you don't like the deal then go build an alternative. You can use all the patches you want from BK but you don't get to use BK's metadata.
--lm
Hi,
On Wed, 9 Feb 2005, Larry McVoy wrote:
(I just sent a similiar mail in private and didn't immediately realize it didn't went to lkml, so sorry, who gets it twice.)
Quoted 9 lines · Nicolas Pitre
Quoted 2 lines
I think what people want here is the tree structure representation in whatever form, not necessarily in the BK format.
That's fine, they can do that. Get the patches and figure out how to put them back together. These people do know how to use patch, right? OK, then they are welcome to patch things in, when they don't work, find a place they do work and create a branch, patch them, repeat. Haven't they ever dealt with a patch reject before? It's not that hard.
No, just impossible.
As I mentioned already previously, the main problem is restoring the changeset information, so one actually knows how the kernel tree looked at a specific point. I also mentioned that it's not really possible to find for all changesets their parent changesets, e.g. changeset 1.889 has 416 branches which include 3560 changesets, that means some of the branches have over 3000 potential parents. Due to parallel changes to a file on multiple branches one can reduce that number, but it's likely still greater than one.
Now for a while I hoped to at least find the end of branch, that means where two branches get merged again. The gnupatches contain some information of what they merge, so that one could use the log text to find the changeset. The problem is that even these gnupatches don't contain information to reliable find its parents. First, they don't include other merge logs, so there are again multiple potential parents if there are merge changeset near the changeset identified by the log text. Second, there are completely empty changesets with no log text and so no indication what they are merging, I currently have 171 of them and of these 13 are at a start of the branch (and therefore have no usable information at all of either parent).
So why is this parent information so important? If the patches are not applied in the correct order, one simply gets the wrong kernel snapshot. What makes this more difficult are the merge changesets, as they don't contain the complete information of what has changed compared to one of the parents, they just contain the file conflicts. If one had the correct parent information this would not be a problem, repeating the merge is one of the smaller problems. But with fuzzy parent information one also only gets fuzzy merge results and if the parents have n potential parents that results in n^2 possible merge results in the worst case. Since the merge changeset only contains conflict information, that makes it rather likely one detects a problem (e.g. that a changeset doesn't apply) very late. So if one detects such problem after m merges, we have a worst case of n^(m+1). IOW the complexity of a bkweb export grows exponentially with the size of the repository! And there is still no guarantee to get a correct (that means only one) result.
So doing the work is one thing, getting a result within my lifetime would be nice too.
bye, Roman
Quoted 3 lines · Roman Zippel
[long explanation which is summarized as "it's hard"]
So doing the work is one thing, getting a result within my lifetime would be nice too.
I understand the complexity you are facing. This may be hard for you to believe but we have solved problems that are quite a bit more complex than what you are trying to do and then gave you the solution for free.
This problem is nowhere near as hard as you are making it out to be but it is hard. But it's not that bad, we do this every time we do a CVS import, we have to intuit the changeset boundaries themselves, which is actually harder than what you are trying to do. Think about taking revision history without any changeset grouping and recreating the changesets (aka patches). We do that all the time, automatically. If we can do that then you can do this.
Yeah, it's hard, though, I admit that. This is a bit of a cheap shot but you did say that this stuff was easy, remember? Yeah, you may hate me for making you realize it is hard but maybe you'll start to respect what it is we have given you. That would be cool.
Anyway, you can do this. It will probably take you a couple of months of 80 hour weeks. Does that suck? Maybe. Bear in mind that we have given you *years* of 80 hour weeks. Years. Many of them unpaid. So suck it up and get to work, we did.
You might have fun doing this, you know. These problems, while hard, have some very satisfying mathematical qualities and that's really fun coming from the kernel background where things are far less deterministic. You can actually write proofs about how things work for a change.
Signature · 2 lines
--- Larry McVoy lm at bitmover.com http://www.bitkeeper.com
Hi,
Quoted 7 lines · Larry McVoy
This problem is nowhere near as hard as you are making it out to be but it is hard. But it's not that bad, we do this every time we do a CVS import, we have to intuit the changeset boundaries themselves, which is actually harder than what you are trying to do. Think about taking revision history without any changeset grouping and recreating the changesets (aka patches). We do that all the time, automatically. If we can do that then you can do this.
That's simply not true and you know that.
I repeat, the bkweb information is not enough to recreate the history, incomplete data will only produce incomplete results no matter how hard you try and I delivered the _proof_ for that.
bye, Roman
I really got bored of this thread.Can you all question your self on thing? If someone starts reading right now the sources of the linux kernel will be able to understand every aspect and part of the code???
Do you understand every aspect?
Is it still "opensource" or starts to be a "closedsource" software "product " despite the fact that is still free to the community.
DONT say again the source is there ,you dont have but to read it. Someone in this list ( very popular ) has said some years ago that even now Micro$oft gives out the source noone will be able to do some changes and some of us wont be able to understand this product ever never. So in the name of this idea is still linux an "opensource" idea or a "closed" one delivered from those who have the information already to maintain it and give it to us freely. I tend to develop in linux only 4 years now and i tried to make some changes in the kernel so that they can conform according to my needs... i have to say that it was very very time consuming.....no doc...no comments...no explanation....I think that a lot of us have the same question....
"Opensource" || "Closedsource" ???????????
Roman, please give up on importing 100% of the history. There's no point arguing something if you already know what the other person's answer will be. Larry will not change his mind under any currently foreseeable circumstances. Yes, there is "meta-data lockin" whether anyone at BitMover will admit it or not, but no that will not change.
Linux survived in the past without much history, and if a replacement arrives, people can make the switch even with a degraded history. In very little time that switchover would seem as remote as the pre-BK times are now.
Right now I don't see why its necessary to track the Linux repo in 100% detail for SCM development; There are plenty of other big trees to test on if you need every detail. Time spent tracking Linux are probably better spent improving an alternative SCM, most of which have plenty of wishlist items awaiting developers. For kernel development, yes it's painful for SCM developers or purists, but you can still work just fine with patches. Maintainers certainly benefit from BK, but for developers on the leaves of the hierarchy there's not that much difference.
- Jim Bruce
Quoted 10 lines · James Bruce
Roman, please give up on importing 100% of the history. There's no point arguing something if you already know what the other person's answer will be. Larry will not change his mind under any currently foreseeable circumstances. Yes, there is "meta-data lockin" whether anyone at BitMover will admit it or not, but no that will not change.
Linux survived in the past without much history, and if a replacement arrives, people can make the switch even with a degraded history. In very little time that switchover would seem as remote as the pre-BK times are now.
Nice conclusion to this thread.
Stelian.
Signature · 1 line
Stelian Pop <[email protected]>