Skip to contentSkip to conversation

Request for discussion on when to merge drivers

DISCUSSION James Bottomley 7 participants 11 messages Jun 18, 2008 – Jun 19, 2008

lore
Share & tools
lore
#0 lore raw

There have been a large number of emails on lkml (too many for me to dredge up and quote) plus a fair few private ones on the subject of when to merge drivers. The opinions seem to vary from "immediately they show up on the mailing list" to " after we're sure they're demonstrated to be working and maintainable". The fact that the arguments have been going round and round on the various mailing lists without generating any consensus would seem to indicate the topic would benefit from some face to face discussion time.

I think the Kernel Summit would be a good place to have a discussion of what the criteria are for merging a driver (even if, in the end, it's at the discretion of the subsystem maintainers).

I think everyone agrees that we can't put just anything that appears on the mailing list in the tree, since they all have to be at least code inspected and be checked for the usual errors, omissions and root holes. However, disagreements seem to set in after this.

For the record, my own view is that when a new driver does appear we have a limited time to get the author to make any necessary changes, so I try to get it reviewed and most of the major issues elucidated as soon as possible. However, since the only leverage I have is inclusion, I tend to hold it out of tree until the problems are sorted out. Experience has shown that for most SCSI drivers, the authors tend to be the people producing the hardware and without documentation, no-one else can fix up anything other than obvious coding errors, so we can't put it in the tree and hope someone else will fix it if they have a problem.

One possible way of doing this is certainly to put the drivers in the staging tree:

http://marc.info/?l=linux-kernel&m=121312896923196

The only slight wrinkle (at least for me) is that often the process of cleaning up a driver is fairly intensive for a maintainer and turn around is a lot faster if you're doing it in a tree you control. (All the scsi drivers we've done like this have lived in temporary branches while they were being worked on). So perhaps in addition we should be encouraging maintainers to run staging branches under similar rules in the staging tree, but allowing inclusion into linux-next?

James

#1 lore raw
Quoted 4 lines

Experience has shown that for most SCSI drivers, the authors tend to be the people producing the hardware and without documentation, no-one else can fix up anything other than obvious coding errors, so we can't put it in the tree and hope someone else will fix it if they have a problem.

Is there an opportunity for a carrot here ... a driver submitted *with documentation* gets a faster path into the tree?

-Tony

#2 lore raw
Quoted 7 lines · Tony
Quoted 4 lines

Experience has shown that for most SCSI drivers, the authors tend to be the people producing the hardware and without documentation, no-one else can fix up anything other than obvious coding errors, so we can't put it in the tree and hope someone else will fix it if they have a problem.

Is there an opportunity for a carrot here ... a driver submitted *with documentation* gets a faster path into the tree?

You mean with actual how to program it information? er dunno ... never seen one of those ...

But I suppose, theoretically, since anyone could access the documentation to maintain it, it would be much more likely to be fixed up in tree, yes.

James

#3 lore raw
In reply to James Bottomley
Quoted 33 lines · James Bottomley

There have been a large number of emails on lkml (too many for me to dredge up and quote) plus a fair few private ones on the subject of when to merge drivers. The opinions seem to vary from "immediately they show up on the mailing list" to " after we're sure they're demonstrated to be working and maintainable". The fact that the arguments have been going round and round on the various mailing lists without generating any consensus would seem to indicate the topic would benefit from some face to face discussion time.

I think the Kernel Summit would be a good place to have a discussion of what the criteria are for merging a driver (even if, in the end, it's at the discretion of the subsystem maintainers).

I think everyone agrees that we can't put just anything that appears on the mailing list in the tree, since they all have to be at least code inspected and be checked for the usual errors, omissions and root holes. However, disagreements seem to set in after this.

For the record, my own view is that when a new driver does appear we have a limited time to get the author to make any necessary changes, so I try to get it reviewed and most of the major issues elucidated as soon as possible. However, since the only leverage I have is inclusion, I tend to hold it out of tree until the problems are sorted out. Experience has shown that for most SCSI drivers, the authors tend to be the people producing the hardware and without documentation, no-one else can fix up anything other than obvious coding errors, so we can't put it in the tree and hope someone else will fix it if they have a problem.

One possible way of doing this is certainly to put the drivers in the staging tree:

http://marc.info/?l=linux-kernel&m=121312896923196

Quoted 7 lines

The only slight wrinkle (at least for me) is that often the process of cleaning up a driver is fairly intensive for a maintainer and turn around is a lot faster if you're doing it in a tree you control. (All the scsi drivers we've done like this have lived in temporary branches while they were being worked on). So perhaps in addition we should be encouraging maintainers to run staging branches under similar rules in the staging tree, but allowing inclusion into linux-next?

This is a very good idea.

Exposing the not-yet-ready-to-be-released code to linux-next will expose conflicts earlier, and hopefully in smaller, more manageable deltas.

However, some projects that need staging may change kernel APIs to such extent that including them in linux-next will require committing to the API changes in the -next time frame. If that's a problem, staging them in the maintainer's tree may still be valuable.

Benny

Quoted 9 lines

James

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#4 lore raw
Quoted 1 line · Benny Halevy

Exposing the not-yet-ready-to-be-released code to linux-next will expose conflicts earlier, and hopefully in smaller, more manageable deltas.

I like the way linux-next works now, i.e. it should reflect what is going into the next major kernel release. Nothing more, nothing less.

Also, when talking about entirely new drivers for hardware that hasn't been supported by the kernel at all so far (this is the situation we are talking about, right?), there shouldn't be a lot of conflicts to be dealt with, right? Ideally, drivers should be pretty isolated standalone pieces of code that don't have any business changing any code that has been already there.

Signature · 2 lines
Jiri Kosina
SUSE Labs
#5 lore raw
Quoted 8 lines · Jiri Kosina
Quoted 2 lines · Benny Halevy

Exposing the not-yet-ready-to-be-released code to linux-next will expose conflicts earlier, and hopefully in smaller, more manageable deltas.

I like the way linux-next works now, i.e. it should reflect what is going into the next major kernel release. Nothing more, nothing less.

Quoted 7 lines

Also, when talking about entirely new drivers for hardware that hasn't been supported by the kernel at all so far (this is the situation we are talking about, right?), there shouldn't be a lot of conflicts to be dealt with, right? Ideally, drivers should be pretty isolated standalone pieces of code that don't have any business changing any code that has been already there.

The showcase example I have in mind is our OSD initiator for which we pushed all dependencies already upstream, so currently it has no conflicts with the existing code. However, it is not ready for the next release since it should be reviewed, documented, and accepted by the sub-system maintainer first.

I would like it to be included in linux-next so to be alerted of any arising conflict so I can fix it in "real time" rather than waiting for the next release to come out. Initially I thought it'd be a good candidate for linux-staging, but Greg doesn't quite agree with that
(http://lkml.org/lkml/2008/6/11/225) since it's destined for linux-scsi. Hence, a -staging branch/tree based off of James' tree makes sense.

Benny

#6 lore raw
In reply to James Bottomley
Quoted 3 lines · On Wed, 18 Jun 2008 10:27:46 -0500 James Bottomley…

I think the Kernel Summit would be a good place to have a discussion of what the criteria are for merging a driver (even if, in the end, it's at the discretion of the subsystem maintainers).

I agree.

Quoted 3 lines

So perhaps in addition we should be
encouraging maintainers to run staging branches under similar rules in the staging tree, but allowing inclusion into linux-next?

This seems a good idea for drivers.

One potencial issue that we may discuss is some sort of policy on how to deal with drivers at the "generic" staging tree versus a subsystem maintainer staging tree. I can foresee a few troubles if a maintainer have such tree, like:

- After the fixes at -staging, the patch may be kept for a while at
maintainer's staging tree. So, the workflow will be longer than having just one tree;

- Two different patches for the same device can be sent to the
"generic" staging and to the maintainer's staging tree. So, a conflict will rise (hopefully detected at linux-next, but may require additional checks for conflicts with PCI/USB ID's);

On the other hand, without the maintainer's staging tree, this would mean that the maintainer would have to send their cooking drivers to the generic tree, with doesn't seem to be very productive.

A possible solution would be if each maintainer with such staging tree would keep the wiki at linux driver project updated, but this adds some additional tasks to the maintainer's arms.

Cheers,
Mauro

#7 lore raw
In reply to James Bottomley
Quoted 8 lines

The only slight wrinkle (at least for me) is that often the process of cleaning up a driver is fairly intensive for a maintainer and turn around is a lot faster if you're doing it in a tree you control. (All the scsi drivers we've done like this have lived in temporary branches while they were being worked on). So perhaps in addition we should be encouraging maintainers to run staging branches under similar rules in the staging tree, but allowing inclusion into linux-next?

.../...

linux-next should, imho, exclusively be for things we are pretty much commited to merge in the next release. ie, a staging place to fixup things like build breakages, patch conflicts, etc...

Or else, it will just be another -mm ....

So while what you say makes sense, I think we should only put drivers in once we have pretty much decided that the drivers in question would be merged... in which case, why not straight upstream ?

Cheers,
Ben.

#8 lore raw
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 7 lines

The only slight wrinkle (at least for me) is that often the process of cleaning up a driver is fairly intensive for a maintainer and turn around is a lot faster if you're doing it in a tree you control. (All the scsi drivers we've done like this have lived in temporary branches while they were being worked on). So perhaps in addition we should be encouraging maintainers to run staging branches under similar rules in the staging tree, but allowing inclusion into linux-next?

Quoted 7 lines

.../...

linux-next should, imho, exclusively be for things we are pretty much commited to merge in the next release. ie, a staging place to fixup things like build breakages, patch conflicts, etc...

Or else, it will just be another -mm ....

Well, as Greg said for his driver staging tree, I think we can elide this requirement for new drivers. The good thing about drivers is that there's nothing we're really doing to break anything: before the driver the hardware was just plane unusable with Linux after the driver well, we're hoping it might be ...

Quoted 3 lines

So while what you say makes sense, I think we should only put drivers in once we have pretty much decided that the drivers in question would be merged... in which case, why not straight upstream ?

The reason for staging early in linux-next is twofold

1. We want to encourage the driver submitter that something is being done
2. we want to make the driver available to a pool of testers who might have the hardware but might not otherwise find it so they can report errors that aren't picked up via the normal code inspection and analysis methods.

Arguably, I can do this by putting it into my upstream tree (which feeds into linux-next) the reason for not doing so is that Linus likes us to preserve history in there when we can. If I need to pull the driver for a reroll it really screws the git history (plus makes it hard for me). If it's in its own branch, I can toy with it as much as I need to without affecting my upstream tree. Plus the commitment is that anything which goes into my upstream tree should be for the next merge window. A driver with substantive code issues (or which shows problems under test) might not be such a candidate until the issues are sorted out.

James

#9 lore raw
Quoted 3 lines · James Bottomley

Arguably, I can do this by putting it into my upstream tree (which feeds into linux-next) the reason for not doing so is that Linus likes us to preserve history in there when we can.

This of course applies only for branches/trees for which there are downstream git users.

If the only reason you merge the driver into your tree is to have it propagated into linux-next, you could very well create a separate branch for it, and pull this branch into the one linux-next pulls from you (I guess you have a dedicated branch for linux-next which is safe to be rebased by definition, right?).

Quoted 1 line

If I need to pull the driver for a reroll it really screws the git history

But not in any of the branches that have upstream git users.

Signature · 2 lines
Jiri Kosina
SUSE Labs
#10 lore raw
In reply to James Bottomley
Quoted 5 lines · James Bottomley
Quoted 4 lines · Benjamin Herrenschmidt

linux-next should, imho, exclusively be for things we are pretty much commited to merge in the next release. ie, a staging place to fixup things like build breakages, patch conflicts, etc...

Quoted 7 lines
Quoted 1 line

Or else, it will just be another -mm ....

Well, as Greg said for his driver staging tree, I think we can elide this requirement for new drivers. The good thing about drivers is that there's nothing we're really doing to break anything: before the driver the hardware was just plane unusable with Linux after the driver well, we're hoping it might be ...

This requires that the staged driver does not contain build bugs on any architecture.

Besides, the definition of what we published in -next so far was:

Everything in -next is *merge-ready* now (from a subsystem maintainer's POV). The only reason that it is not merged yet is that Linus doesn't have a merge window open right now.

And the goal of -next was to check for potential integration problems that cannot be detected in a maintainer's tree.

So IMO the question is not whether to put stuff which is not ready to be merged into -next. It shouldn't, IMO. The questions rather are:
- What are criteria for merge-readiness of drivers?
(When do we have to / are we able to address issues like CamelCase names or use of obsolete APIs --- before or after merge?)
- How to publish drivers which are not yet merge-ready but should get into the hands of testers?

IMO the answer to the latter question is still not -next, because the testing that -next gets is (from what I understood) also with the specific goal to detect potential integration problems. Remember, you publish code in -next of which you can say: This is ready for mainline, except that there is little experience yet WRT potential integration issues.

Signature · 3 lines
Stefan Richter
-=====-==--- -==- =--==
http://arcgraph.de/sr/