Skip to contentSkip to conversation

Distribution kernel bugzillas considered harmful

Ksummit-discuss James Bottomley 15 participants 33 messages Sep 5, 2018 – Sep 19, 2018

lore
Share & tools
lore
#0 lore raw

I'm seeing a lot of wasted effort by our customers on kernel bugs and trying to engage the distribution to fix them. As a caveat, I'm working in the cloud, so the distributions in question are usually community ones not enterprise ones. However, we do have a fair few customers on LTS kernels from Distributions.

Mostly they find a cloud performance regression, they try to engage the distro, spend ages working on it or submitting bugs and usually end up with an unsatisfactory result. By the time they call my team in, we've likely only got a week to fix the issue. However, step one is always confirming whether upstream works (95% of the time it does) and then finding the fix by bisection (usually assisted by knowledge of where the bug is). To do the bisection we usually have to build a kernel package with our guesses and get them to try it, so it can be a bit slow. Once we have the backport, we send it to stable and notify the distribution to include it in their next kernel release.

Here's the rub: community distributions (even LTS ones) don't have the resources even to triage cloud bugs in environments they likely can't reproduce, so we really need to develop assistive tools for customers to perform bisections to identify what caused the bug or (in the 95% case) what fixed it. Having a bugzilla and using it as first line of support implies a service expectation (usually coming from Enterprise) that simply isn't met, so distributions need to fix this at the point of interaction: bugzilla.

The first suggestion is that kernel builds are pretty much automated and we try to make every commit buildable, so could we automate the machinery that allows a customer to do bisection simply by installing a kernel package? (we here, obviously means the distro, but going from git bisect to kernel package would be the useful link).

Second suggestion is that the bugzillas need to say much more strongly that the reporter really needs to confirm the fix in upstream and do the bisection themselves (and ideally request the backport to stable themselves).

James

#1 lore raw
Quoted 5 lines · James Bottomley

The first suggestion is that kernel builds are pretty much automated and we try to make every commit buildable, so could we automate the machinery that allows a customer to do bisection simply by installing a kernel package? (we here, obviously means the distro, but going from git bisect to kernel package would be the useful link).

Improving bisectability would obviously help with other testing efforts too - we have existing users, Guillaume Tucker implemented automated bisection support in KernelCI which is incredibly useful providing one can actually bisect. Right now it works pretty well a lot of the time but there are cases where it gets messy, especially when you add boot issues onto the buildability ones.

#24 lore raw
In reply to James Bottomley
Quoted 6 lines · James Bottomley

The first suggestion is that kernel builds are pretty much automated and we try to make every commit buildable, so could we automate the machinery that allows a customer to do bisection simply by installing a kernel package? (we here, obviously means the distro, but going from git bisect to kernel package would be the useful link).

Oh yeah, that's a thing I've been dreaming for years! :)

A part of problems is the build power: it still takes an hour or longer (depending on arch) to build a full distro kernel package, for example, on OBS. That's too painful for bisection.

So some semi-automatic way to reduce the config would be required for usable bisection. I guess automatic make localmodconfig, prep the repo and kick off. Also some garbage collections for stale repos, etc.

Quoted 4 lines

Second suggestion is that the bugzillas need to say much more strongly that the reporter really needs to confirm the fix in upstream and do the bisection themselves (and ideally request the backport to stable themselves).

OK, distros definitely need to try hard not to annoy upstream devs.

In the case of SUSE Kernel, we usually ask testing the latest (more-or-less) vanilla kernel at first. If it's an upstream problem, then it's often tossed to the upstream. If it's already addressed in the upstream kernel, we take the responsibility for backports. Asking bisection by reporter is usually the last resort.

It'd be helpful if we get any suggestion to improve the process.

thanks,

Takashi

#25 lore raw
Quoted 3 lines · Takashi Iwai

Asking bisection by reporter is usually the last resort.

It'd be helpful if we get any suggestion to improve the process.

IIRC, years ago someone started on "web-based bisection tool", but that was never finished.

The idea was that the user would just click "Good" and "Bad" buttons, and would be getting kernel RPMs to test.

If this would have been picked up and finished (GSoC perhaps?), it might help a lot with this aspect.

Signature · 2 lines
Jiri Kosina
SUSE Labs
#26 lore raw
In reply to Takashi Iwai
Quoted 7 lines · Takashi Iwai
Quoted 4 lines

Second suggestion is that the bugzillas need to say much more strongly that the reporter really needs to confirm the fix in upstream and do the bisection themselves (and ideally request the backport to stable themselves).

OK, distros definitely need to try hard not to annoy upstream devs.

Quoted 7 lines

In the case of SUSE Kernel, we usually ask testing the latest (more-or-less) vanilla kernel at first. If it's an upstream problem, then it's often tossed to the upstream. If it's already addressed in the upstream kernel, we take the responsibility for backports. Asking bisection by reporter is usually the last resort.

It'd be helpful if we get any suggestion to improve the process.

It would be awesome to have a "bisect@home" type of thing with a similar idea like seti@home and folding@home. Have a central queue where developers can submit upstream commits and testcases, and a swarm of volunteer drones would grab and bisect-build them until the bug-introducing commit is identified and reported back.

I'll totally host the hell out of this.

-K

#2 lore raw
In reply to Mark Brown
Quoted 8 lines · Mark Brown
Quoted 5 lines · James Bottomley

The first suggestion is that kernel builds are pretty much automated and we try to make every commit buildable, so could we automate the machinery that allows a customer to do bisection simply by installing a kernel package? (we here, obviously means the distro, but going from git bisect to kernel package would be the useful link).

Quoted 6 lines

Improving bisectability would obviously help with other testing efforts too - we have existing users, Guillaume Tucker implemented automated bisection support in KernelCI which is incredibly useful providing one can actually bisect. Right now it works pretty well a lot of the time but there are cases where it gets messy, especially when you add boot issues onto the buildability ones.

I am one of those strange people who rebase in order to improve bisectability. But one reason I can do that is that I have relatively few patches, and it gets harder the more patches I am carrying. I suppose that someone (not me!) could rebase -stable to make it more bisectable, but that sounds difficult, painful, and error-prone. Could added tooling make bisection work better? Sounds valuable, but non-trivial.

In some of my bisection efforts, I have had to apply fix patches to fix various unrelated bugs. I suppose that this could be automated, for example by tracking which fix-patches are needed at which potential bisection points, though this sounds like a large effort. Of course, automated backporting of patches would make it easier, and much else easier as well. ;-)

Thanx, Paul

#27 lore raw
In reply to Konstantin Ryabitsev
Quoted 16 lines · Konstantin Ryabitsev
Quoted 7 lines · Takashi Iwai
Quoted 4 lines

Second suggestion is that the bugzillas need to say much more strongly that the reporter really needs to confirm the fix in upstream and do the bisection themselves (and ideally request the backport to stable themselves).

OK, distros definitely need to try hard not to annoy upstream devs.

Quoted 7 lines

In the case of SUSE Kernel, we usually ask testing the latest (more-or-less) vanilla kernel at first. If it's an upstream problem, then it's often tossed to the upstream. If it's already addressed in the upstream kernel, we take the responsibility for backports. Asking bisection by reporter is usually the last resort.

It'd be helpful if we get any suggestion to improve the process.

Quoted 7 lines

It would be awesome to have a "bisect@home" type of thing with a similar idea like seti@home and folding@home. Have a central queue where developers can submit upstream commits and testcases, and a swarm of volunteer drones would grab and bisect-build them until the bug-introducing commit is identified and reported back.

I'll totally host the hell out of this.

I don't think KernelCI are too far on this front. They're already working on auto bisection, so if there's an ability to add (simple) testcases you'll be all set.

--
Thanks,
Sasha

#3 lore raw
In reply to Paul E. McKenney
Quoted 4 lines · On Wed, 5 Sep 2018 08:03:15 -0700 "Paul E. McKenney"…

I am one of those strange people who rebase in order to improve bisectability. But one reason I can do that is that I have relatively few patches, and it gets harder the more patches I am carrying. I suppose that someone (not me!) could rebase -stable to make it more bisectable,

How would rebasing it make stable more bisectable? Once you rebase, you don't have a tree that use to work? Although I guess you may find the commit that caused the problem better. But rebasing creates a lot of other issues, I would not recommend rebasing stable, as that would totally break the RT stable tree work flow.

-- Steve

Quoted 2 lines

but that sounds difficult, painful, and error-prone. Could added tooling make bisection work better? Sounds valuable, but non-trivial.

#4 lore raw
Quoted 8 lines · Steven Rostedt
Quoted 4 lines · On Wed, 5 Sep 2018 08:03:15 -0700 "Paul E. McKenney"…

I am one of those strange people who rebase in order to improve bisectability. But one reason I can do that is that I have relatively few patches, and it gets harder the more patches I am carrying. I suppose that someone (not me!) could rebase -stable to make it more bisectable,

Quoted 5 lines

How would rebasing it make stable more bisectable? Once you rebase, you don't have a tree that use to work? Although I guess you may find the commit that caused the problem better. But rebasing creates a lot of other issues, I would not recommend rebasing stable, as that would totally break the RT stable tree work flow.

Instead of leaving the buggy commit and the span where the bug exists, you rebase the fix into the original buggy fix.

And I bet that rebasing -stable would cause no end of broken glass in a great many projects. ;-)

Thanx, Paul

#28 lore raw
In reply to Konstantin Ryabitsev
Quoted 16 lines · Konstantin Ryabitsev
Quoted 7 lines · Takashi Iwai
Quoted 4 lines

Second suggestion is that the bugzillas need to say much more strongly that the reporter really needs to confirm the fix in upstream and do the bisection themselves (and ideally request the backport to stable themselves).

OK, distros definitely need to try hard not to annoy upstream devs.

Quoted 7 lines

In the case of SUSE Kernel, we usually ask testing the latest (more-or-less) vanilla kernel at first. If it's an upstream problem, then it's often tossed to the upstream. If it's already addressed in the upstream kernel, we take the responsibility for backports. Asking bisection by reporter is usually the last resort.

It'd be helpful if we get any suggestion to improve the process.

Quoted 8 lines

It would be awesome to have a "bisect@home" type of thing with a similar idea like seti@home and folding@home. Have a central queue where developers can submit upstream commits and testcases, and a swarm of volunteer drones would grab and bisect-build them until the bug-introducing commit is identified and reported back.

I'll totally host the hell out of this.

I played around with bisect scripts for Fedora a while ago but mostly abandoned them due to lack of interest and fragility. I was attempting to use the full Fedora patch set and spec file but making that work for arbitrary kernel commits was a pain.

Developers usually have no problem building and bisecting kernels, it's non-kernel developers who often struggle with bisection. One idea that I haven't followed up on was to extend the existing targets for building distro packages to just build the source side of things and then take advantage of existing environments (e.g. COPR) to build the package binaries. I'd love a web interface that would handle some of this automatically but, again, lack of resources and knowledge of web frameworks.

Thanks,
Laura

#5 lore raw
In reply to Paul E. McKenney
Quoted 12 lines · Paul E. McKenney
Quoted 11 lines · Steven Rostedt
Quoted 7 lines · On Wed, 5 Sep 2018 08:03:15 -0700 "Paul E. McKenney"…

I am one of those strange people who rebase in order to improve bisectability.  But one reason I can do that is that I have relatively
few patches, and it gets harder the more patches I am carrying.  I suppose
that someone (not me!) could rebase -stable to make it more bisectable,

Quoted 8 lines
Quoted 5 lines

How would rebasing it make stable more bisectable? Once you rebase, you don't have a tree that use to work? Although I guess you may find the commit that caused the problem better. But rebasing creates a lot of other issues, I would not recommend rebasing stable, as that would totally break the RT stable tree work flow.

Instead of leaving the buggy commit and the span where the bug exists, you rebase the fix into the original buggy fix.

We do this in SCSI as well, but only if the tree hasn't yet been submitted to Linus. The technical term is folding. It's obviously better to fix buggy commits that haven't gone upstream because it improves bisectability.

Quoted 2 lines

And I bet that rebasing -stable would cause no end of broken glass in a great many projects.  ;-)

If others rely on your tree, rebasing is harder and must be done more carefully and with co-ordination, but it's not impossible assuming you have a problem big enough. Again, it's an expediency based trade off.

James

#6 lore raw
Quoted 19 lines · James Bottomley
Quoted 18 lines · Paul E. McKenney
Quoted 16 lines · Steven Rostedt
Quoted 7 lines · On Wed, 5 Sep 2018 08:03:15 -0700 "Paul E. McKenney"…

I am one of those strange people who rebase in order to improve bisectability.  But one reason I can do that is that I have relatively
few patches, and it gets harder the more patches I am carrying.  I suppose
that someone (not me!) could rebase -stable to make it more bisectable,

How would rebasing it make stable more bisectable? Once you rebase, you don't have a tree that use to work? Although I guess you may find the commit that caused the problem better. But rebasing creates a lot of other issues, I would not recommend rebasing stable, as that would totally break the RT stable tree work flow.

Quoted 7 lines
Quoted 2 lines

Instead of leaving the buggy commit and the span where the bug exists, you rebase the fix into the original buggy fix.

We do this in SCSI as well, but only if the tree hasn't yet been submitted to Linus. The technical term is folding. It's obviously better to fix buggy commits that haven't gone upstream because it improves bisectability.

So I am not the only one doing this. ;-)

Quoted 6 lines
Quoted 2 lines

And I bet that rebasing -stable would cause no end of broken glass in a great many projects.  ;-)

If others rely on your tree, rebasing is harder and must be done more carefully and with co-ordination, but it's not impossible assuming you have a problem big enough. Again, it's an expediency based trade off.

I use date-coded branches to keep the old commits around. I delete them periodically, but keep them around for at least six months, on the theory that if you are developing against a two-year old -rcu branch, you have bigger problems. And you should have instead developed against a formal release anyway.

Thanx, Paul

#32 lore raw
In reply to James Bottomley
Quoted 26 lines · James Bottomley

I'm seeing a lot of wasted effort by our customers on kernel bugs and trying to engage the distribution to fix them. As a caveat, I'm working in the cloud, so the distributions in question are usually community ones not enterprise ones. However, we do have a fair few customers on LTS kernels from Distributions.

Mostly they find a cloud performance regression, they try to engage the distro, spend ages working on it or submitting bugs and usually end up with an unsatisfactory result. By the time they call my team in, we've likely only got a week to fix the issue. However, step one is always confirming whether upstream works (95% of the time it does) and then finding the fix by bisection (usually assisted by knowledge of where the bug is). To do the bisection we usually have to build a kernel package with our guesses and get them to try it, so it can be a bit slow. Once we have the backport, we send it to stable and notify the distribution to include it in their next kernel release.

Here's the rub: community distributions (even LTS ones) don't have the resources even to triage cloud bugs in environments they likely can't reproduce, so we really need to develop assistive tools for customers to perform bisections to identify what caused the bug or (in the 95% case) what fixed it. Having a bugzilla and using it as first line of support implies a service expectation (usually coming from Enterprise) that simply isn't met, so distributions need to fix this at the point of interaction: bugzilla.

Quoted 6 lines

The first suggestion is that kernel builds are pretty much automated and we try to make every commit buildable, so could we automate the machinery that allows a customer to do bisection simply by installing a kernel package? (we here, obviously means the distro, but going from git bisect to kernel package would be the useful link).

As mentioned further down, I did try having scripts to do this but it was ultimately fairly fragile. I think a better approach is leveraging the existing in tree support for building a distro package and using that with regular git bisect.

Quoted 5 lines

Second suggestion is that the bugzillas need to say much more strongly that the reporter really needs to confirm the fix in upstream and do the bisection themselves (and ideally request the backport to stable themselves).

At least in Fedora this is something we hit fairly frequently. We do strongly encourage people to report bugs and bisect. This runs into a number of problems:

- Bisecting on local machines is slow and people often don't want to give up their machine resources.
- If people give me a test case I can reproduce, I'm usually okay to run a bisect myself but it's pretty rare to get a test case.
- People are hesitant to run bisections and build kernels. There's a lot of steps involved. We try and point people to wiki pages with instructions but many times we end up having to go back and forth explaining how to do the setup.
- People are hesitant to report bugs to the upstream. I end up having to explain where to report bugs or run get_maintainer.pl for people otherwise they just file it against kernel.org bugzilla or just e-mail LKML. I started a skeleton of a web project to make a web interface for get_maintainer.pl but it never got very far.

Tooling can certainly help with a lot of this but some of this may also just be more documentation and needing to guide people.

Thanks,
Laura

#7 lore raw
In reply to James Bottomley
Quoted 4 lines · James Bottomley

We do this in SCSI as well, but only if the tree hasn't yet been submitted to Linus. The technical term is folding. It's obviously better to fix buggy commits that haven't gone upstream because it improves bisectability.

We are drifting away a bit here, but now that you have mentioned it, let me add a datapoint to this -- it's actually causing issues to our workflow, as we have scsi.git as one of the upstreams [1], and when you rebase, it blows up our git workflow and we have to fixup things manually.

So if you are aware of your tree having downstreams, and care about not breaking them and want to be nice to them, you shouldn't rebase that tree [2].

[1] there are some funny technical details, but in basic principle it's exactly like that

[2] https://yarchive.net/comp/linux/git_rebase.html

Thanks,

Signature · 2 lines
Jiri Kosina
SUSE Labs
#8 lore raw
Quoted 7 lines · Jiri Kosina
Quoted 4 lines · James Bottomley

We do this in SCSI as well, but only if the tree hasn't yet been submitted to Linus.  The technical term is folding.  It's obviously better to fix buggy commits that haven't gone upstream because it improves bisectability.

Quoted 5 lines

We are drifting away a bit here, but now that you have mentioned it, let me add a datapoint to this -- it's actually causing issues to our workflow, as we have scsi.git as one of the upstreams [1], and when you rebase, it blows up our git workflow and we have to fixup things manually.

Describe the issues and we'll try to come up with a fix, but on the whole you should regard the scsi trees somewhat similarly to linux-
next: it's our proposal for a patch set but we may update it.

Quoted 3 lines

So if you are aware of your tree having downstreams, and care about not breaking them and want to be nice to them, you shouldn't rebase that tree [2].

Well, I wasn't aware of this one, but I'm sure we can come up with a functional workflow once I understand what's happening with the downstream tree.

James

Quoted 7 lines

[1] there are some funny technical details, but in basic principle it's exactly like that

[2] https://yarchive.net/comp/linux/git_rebase.html

Thanks,

#29 lore raw
In reply to Laura Abbott
Quoted 8 lines · Laura Abbott
Quoted 8 lines

It would be awesome to have a "bisect@home" type of thing with a similar idea like seti@home and folding@home. Have a central queue where developers can submit upstream commits and testcases, and a swarm of volunteer drones would grab and bisect-build them until the bug-introducing commit is identified and reported back.

I'll totally host the hell out of this.

Quoted 8 lines

Developers usually have no problem building and bisecting kernels, it's non-kernel developers who often struggle with bisection. One idea that I haven't followed up on was to extend the existing targets for building distro packages to just build the source side of things and then take advantage of existing environments (e.g. COPR) to build the package binaries. I'd love a web interface that would handle some of this automatically but, again, lack of resources and knowledge of web frameworks.

I'm excited that kernelci.org is coming on board as a full-fledged Linux Foundation project, since I'm hoping that their charter would include funding this kind of development. [1] I was already chatting with Kevin about some of the cool things we could do to make various CI/fuzzing/bug-reporting tools more streamlined, so I'll add "bisecting as a service" to my list of suggestions for the glorious kernel CI tool of the future. :)

-K

[1] I don't bring this up too often, but kernel.org's official organizational purpose is "to distribute the Linux kernel and other Open Source software to the public without charge" -- so drifting too far away from such activity could potentially jeopardize its legal standing as far as non-profit filing paperwork is concerned. This is why we can organize funding for things like public-inbox and lore.kernel.org, but not other things that wouldn't be overall related to "distributing the Linux kernel free of charge."

#30 lore raw
Quoted 20 lines · Konstantin Ryabitsev
Quoted 9 lines · Laura Abbott
Quoted 9 lines

It would be awesome to have a "bisect@home" type of thing with a similar
idea like seti@home and folding@home. Have a central queue where developers can submit upstream commits and testcases, and a swarm of volunteer drones would grab and bisect-build them until the bug-introducing commit is identified and reported back.

I'll totally host the hell out of this.

Quoted 8 lines

Developers usually have no problem building and bisecting kernels, it's non-kernel developers who often struggle with bisection. One idea that I haven't followed up on was to extend the existing targets for building distro packages to just build the source side of things and then take advantage of existing environments (e.g. COPR) to build the package binaries. I'd love a web interface that would handle some of this automatically but, again, lack of resources and knowledge of web frameworks.

Quoted 7 lines

I'm excited that kernelci.org is coming on board as a full-fledged Linux Foundation project, since I'm hoping that their charter would include funding this kind of development. [1] I was already chatting with Kevin about some of the cool things we could do to make various CI/fuzzing/bug-reporting tools more streamlined, so I'll add "bisecting as a service" to my list of suggestions for the glorious kernel CI tool of the future. :)

That sounds promising, thanks!

Takashi

#9 lore raw
In reply to James Bottomley
Quoted 9 lines · James Bottomley
Quoted 5 lines

We are drifting away a bit here, but now that you have mentioned it, let me add a datapoint to this -- it's actually causing issues to our workflow, as we have scsi.git as one of the upstreams [1], and when you rebase, it blows up our git workflow and we have to fixup things manually.

Describe the issues and we'll try to come up with a fix, but on the whole you should regard the scsi trees somewhat similarly to linux-
next: it's our proposal for a patch set but we may update it.

We order patches in our trees in the same git-topological-ordering as they are upstream. It has a lot of benefits, most importantly: it doesn't introduce artificial conflicts that don't exist in reality.

In order to achieve that, we of course need 1:1 mapping between our patches and upstream commits.
Rebases destroy that mapping.

And in some areas (scsi is one, but not the only one), we basically had no other choice than considering maintainer's tree to be already "upstream enough", without waiting for Linus' tree merge.

Quoted 7 lines
Quoted 3 lines

So if you are aware of your tree having downstreams, and care about not breaking them and want to be nice to them, you shouldn't rebase that tree [2].

Well, I wasn't aware of this one, but I'm sure we can come up with a functional workflow once I understand what's happening with the downstream tree.

Thanks,

Signature · 2 lines
Jiri Kosina
SUSE Labs
#31 lore raw
In reply to Konstantin Ryabitsev

Em Wed, 5 Sep 2018 16:15:33 -0400
Konstantin Ryabitsev <[email protected]> escreveu:

Quoted 7 lines

I'm excited that kernelci.org is coming on board as a full-fledged Linux Foundation project, since I'm hoping that their charter would include funding this kind of development. [1] I was already chatting with Kevin about some of the cool things we could do to make various CI/fuzzing/bug-reporting tools more streamlined, so I'll add "bisecting as a service" to my list of suggestions for the glorious kernel CI tool of the future. :)

Kernel-ci seems interesting. Yet, 1/3 of the Kernel patches are drivers. At least for media stuff, IMO, CI is too far away from our needs, as one would need some real hardware to do some testing stuff (like for example, some webcams and some ways to check if the webcam support is actually working). To make it worse, there are a wide variety of different hardware with different features and their quirk needs. I suspect that similar troubles would affect sound devices as well.

It could be worth to have some discussions with CI people with regards to how it can better cope with driver's subsystem needs.

Thanks,
Mauro

#10 lore raw
In reply to Jiri Kosina

Hi Jiri,

Sorry I'm a little late to the game here. Been out on vacation.

Quoted 10 lines

We order patches in our trees in the same git-topological-ordering as they are upstream. It has a lot of benefits, most importantly: it doesn't introduce artificial conflicts that don't exist in reality.

In order to achieve that, we of course need 1:1 mapping between our patches and upstream commits. Rebases destroy that mapping.

And in some areas (scsi is one, but not the only one), we basically had no other choice than considering maintainer's tree to be already "upstream enough", without waiting for Linus' tree merge.

When I discussed this with Johannes a little while ago, I suggested you guys used git patch-id to track patches instead of commit ids. That's how we track patches applied across many different trees internally. Works much better than using the upstream sha.

I would like to understand your "upstream enough" requirement. Why do you need a tree that's stable before Linus pulls the changes?

Note that I am generally only rebasing as a last resort and typically only very early in the rc cycle. It usually happens when I need to drop a patch series that turned out to be unfixable in its current state.

And before everyone screams because I'm not supposed to be pushing stuff that breaks, please realize that it is impossible to test all the different types of hardware I have to merge drivers for. There is no regression test suite or lab setup with anything resembling comprehensive coverage. I test changes to the SCSI core code and do some rudimentary testing on SAS and FC on x86_64. But that's really the best I can do.

Even though most patches posted to linux-scsi get picked up by 0day, more often than not they only get x86_64 build coverage. Whereas 0day build failures on arm, mips, sparc32, whatever typically only get reported after patches have been simmering in linux-next for a while. Depends how busy 0day is.

Also, actual driver failures on platforms not officially supported and tested by the controller vendor are only found after the fact. And most of the time it's not a matter of reverting a single patch but effectively dropping all of the patches in the series until they can be reworked. Sometimes a workaround takes a week or two to deliver, and people don't appreciate not being able to boot their systems in the meantime. So that's why I generally drop the series instead.

I would love for every patch sent to linux-scsi to be bug free and instantly build tested by 0day on every architecture. And I would love for hardware vendors to be more cognizant about architectures they don't commercially support. But reality is that things break frequently when I merge big, complex driver update patch series.

As a result, the preference has been to have the flexibility to amend or drop patches early in every cycle. It hasn't really been a problem because there have been no downstream users of SCSI at all. I only recently found out about your use case.

I'm pretty flexible about how to address this, there are a couple of ways to go about it.

1. I could just always revert instead of dropping the patches. The downside is that we end up with a pretty messy history because, as I pointed out above, it's usually a matter of dropping tens of patches at a time and not reverting a single offending commit. In addition, having a messy history makes it harder on distro kernel people to track driver updates.

2. The other option is that I set up a scsi-staging tree where stuff can simmer for a bit and hopefully get some 0day coverage before getting shuffled over to scsi-queue. However, I do question how much actual real hardware coverage we'll get by having a SCSI tree that people would explicitly have to pull to test. As opposed to linux-next which at least gets some coverage by test farms and users.

3. The third option is that we pick a number like rc4 and I will promise not to rebase after that. We can see how that works out.

I'm open to ideas. The most important thing to me is that 0day and linux-next are indispensable tools in my workflow given that I have no way to personally test most of the code I merge. So it is imperative that I have the ability to push code early because that's where I get the bulk of my (non-SCSI-core) build and test coverage.

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#11 lore raw
Quoted 73 lines · Martin K. Petersen
Quoted 10 lines

We order patches in our trees in the same git-topological-ordering as they are upstream. It has a lot of benefits, most importantly: it doesn't introduce artificial conflicts that don't exist in reality.

In order to achieve that, we of course need 1:1 mapping between our patches and upstream commits. Rebases destroy that mapping.

And in some areas (scsi is one, but not the only one), we basically had no other choice than considering maintainer's tree to be already "upstream enough", without waiting for Linus' tree merge.

When I discussed this with Johannes a little while ago, I suggested you guys used git patch-id to track patches instead of commit ids. That's how we track patches applied across many different trees internally. Works much better than using the upstream sha.

I would like to understand your "upstream enough" requirement. Why do you need a tree that's stable before Linus pulls the changes?

Note that I am generally only rebasing as a last resort and typically only very early in the rc cycle. It usually happens when I need to drop a patch series that turned out to be unfixable in its current state.

And before everyone screams because I'm not supposed to be pushing stuff that breaks, please realize that it is impossible to test all the different types of hardware I have to merge drivers for. There is no regression test suite or lab setup with anything resembling comprehensive coverage. I test changes to the SCSI core code and do some rudimentary testing on SAS and FC on x86_64. But that's really the best I can do.

Even though most patches posted to linux-scsi get picked up by 0day, more often than not they only get x86_64 build coverage. Whereas 0day build failures on arm, mips, sparc32, whatever typically only get reported after patches have been simmering in linux-next for a while. Depends how busy 0day is.

Also, actual driver failures on platforms not officially supported and tested by the controller vendor are only found after the fact. And most of the time it's not a matter of reverting a single patch but effectively dropping all of the patches in the series until they can be reworked. Sometimes a workaround takes a week or two to deliver, and people don't appreciate not being able to boot their systems in the meantime. So that's why I generally drop the series instead.

I would love for every patch sent to linux-scsi to be bug free and instantly build tested by 0day on every architecture. And I would love for hardware vendors to be more cognizant about architectures they don't commercially support. But reality is that things break frequently when I merge big, complex driver update patch series.

As a result, the preference has been to have the flexibility to amend or drop patches early in every cycle. It hasn't really been a problem because there have been no downstream users of SCSI at all. I only recently found out about your use case.

I'm pretty flexible about how to address this, there are a couple of ways to go about it.

1. I could just always revert instead of dropping the patches. The downside is that we end up with a pretty messy history because, as I pointed out above, it's usually a matter of dropping tens of patches at a time and not reverting a single offending commit. In addition, having a messy history makes it harder on distro kernel people to track driver updates.

2. The other option is that I set up a scsi-staging tree where stuff can simmer for a bit and hopefully get some 0day coverage before getting shuffled over to scsi-queue. However, I do question how much actual real hardware coverage we'll get by having a SCSI tree that people would explicitly have to pull to test. As opposed to linux-next which at least gets some coverage by test farms and users.

Quoted 8 lines

3. The third option is that we pick a number like rc4 and I will promise not to rebase after that. We can see how that works out.

I'm open to ideas. The most important thing to me is that 0day and linux-next are indispensable tools in my workflow given that I have no way to personally test most of the code I merge. So it is imperative that I have the ability to push code early because that's where I get the bulk of my (non-SCSI-core) build and test coverage.

You could start using topic branches (for core, and per driver/vendor), and recreate your for-next branch as a merge of all topic branches on a daily basis. If one patch series turns out to be bad, at least the commit IDs in the other topic branches will remain stable.

I believe Mark Brown used to have such a system (except for dropping patches, he never rebases his topic branches), unless he got bashed by Linus for his cephalopodic merges having more branches than an octopus(?).

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#12 lore raw

Geert,

Quoted 4 lines

You could start using topic branches (for core, and per driver/vendor), and recreate your for-next branch as a merge of all topic branches on a daily basis. If one patch series turns out to be bad, at least the commit IDs in the other topic branches will remain stable.

Yeah. I thought about doing what Jens does, splitting core and drivers into separate branches. However, compared to block I have so few SCSI core updates that I am not convinced it's worth it. Core changes only amount to a couple percent of the total number of commits every release.

For topic branches to be truly effective I'd need a separate branch per driver. Not impossible, but it seems like a pretty big hammer...

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#13 lore raw
Quoted 4 lines · Martin K. Petersen

Yeah. I thought about doing what Jens does, splitting core and drivers into separate branches. However, compared to block I have so few SCSI core updates that I am not convinced it's worth it. Core changes only amount to a couple percent of the total number of commits every release.

I did that for the SCSI tree a long time ago, and it did not work out at all as the few core changes are usually required by driver changes.

Note that I've not really seen the split recently in the block tree either, probably for the same reason.

#14 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 4 lines · Martin K. Petersen

Yeah. I thought about doing what Jens does, splitting core and drivers into separate branches. However, compared to block I have so few SCSI core updates that I am not convinced it's worth it. Core changes only amount to a couple percent of the total number of commits every release.

I did that for the SCSI tree a long time ago, and it did not work out at all as the few core changes are usually required by driver changes.

Note that I've not really seen the split recently in the block tree either, probably for the same reason.

Yeah exactly, I basically stopped doing that as the most active driver was NVMe, and for quite a while that also helped drive core changes. Since they went hand in hand, there wasn't much point in doing the split branches anymore, in fact it was harmful.

I'd love to start doing it again, and I may do that, since we've now got less feature additions that end up depending on core changes.

Signature · 1 line
Jens Axboe
#16 lore raw
In reply to Martin K. Petersen
Quoted 6 lines · Martin K. Petersen

1. I could just always revert instead of dropping the patches. The downside is that we end up with a pretty messy history because, as I pointed out above, it's usually a matter of dropping tens of patches at a time and not reverting a single offending commit. In addition, having a messy history makes it harder on distro kernel people to track driver updates.

I used to deal with this by using topic branches heavily and making my -next be an automated merge of those branches, if something went badly I could just throw away the branch. I stopped for a while because Linus didn't like the number of branches I was creating, though it wasn't a problem with the approach in general.

#15 lore raw
In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig

I did that for the SCSI tree a long time ago, and it did not work out at all as the few core changes are usually required by driver changes.

Note that I've not really seen the split recently in the block tree either, probably for the same reason.

Most of my topic branches end up being per-driver ones, with embedded stuff that often winds up being useful for cross tree merges that weren't anticipated. The core changes that affect or are used by drivers either get the relevant driver changes stacked onto the same branch or merged into the driver branches depending on taste.

#17 lore raw
In reply to Mark Brown
Quoted 10 lines · Mark Brown
Quoted 6 lines · Martin K. Petersen

1. I could just always revert instead of dropping the patches. The downside is that we end up with a pretty messy history because, as I pointed out above, it's usually a matter of dropping tens of patches at a time and not reverting a single offending commit. In addition, having a messy history makes it harder on distro kernel people to track driver updates.

Quoted 5 lines

I used to deal with this by using topic branches heavily and making my -next be an automated merge of those branches, if something went badly I could just throw away the branch. I stopped for a while because Linus didn't like the number of branches I was creating, though it wasn't a problem with the approach in general.

I think that the topic branch approach would work well if you merge topic branches back to the main branch more often. That is, each topic branch lives only for a relatively short time (e.g. a few weeks), and not merging the whole branches in a shot at the end of the development cycle.

Basically the merge to the main branch "fixates" the developments of the given topic, hence other people can start working on the commits safely. Meanwhile the not-yet-merged branches can be still thrown away if they are really bad.

Takashi

#19 lore raw
In reply to Martin K. Petersen
Quoted 13 lines · Martin K. Petersen

Hi Jiri,

Sorry I'm a little late to the game here. Been out on vacation.

Quoted 7 lines

We order patches in our trees in the same git-topological-ordering as they are upstream. It has a lot of benefits, most importantly: it doesn't introduce artificial conflicts that don't exist in reality.

In order to achieve that, we of course need 1:1 mapping between our patches and upstream commits. Rebases destroy that mapping.

Quoted 8 lines
Quoted 3 lines

And in some areas (scsi is one, but not the only one), we basically had no other choice than considering maintainer's tree to be already "upstream enough", without waiting for Linus' tree merge.

When I discussed this with Johannes a little while ago, I suggested you guys used git patch-id to track patches instead of commit ids. That's how we track patches applied across many different trees internally. Works much better than using the upstream sha.

What is git patch id? Is there any internal patch tag?

Quoted 2 lines

I would like to understand your "upstream enough" requirement. Why do you need a tree that's stable before Linus pulls the changes?

A few reasons:
- We'd like to sort also the multiple would-be-upstream patches in the git topo order, which allows us to keep the patches closer to the original shape.

- Once when such patches are merged to upstream, they are moved to the upstream section, again ordered. For moving them, of course it'd be easy if the commit ids are kept.

And here the worst case is that the old commit ID disappears after the rebase. In that sense, at least, it'd be helpful if you can preserve the old branches via tag or another named branch in the repo; then the commits are not pruned, and we can still track the order of patches.

Quoted 3 lines

Note that I am generally only rebasing as a last resort and typically only very early in the rc cycle. It usually happens when I need to drop a patch series that turned out to be unfixable in its current state.

... and we shouldn't take such a patch series always as "would-be-upstream" thing, either.

So it sounds like a sort of lack of communication to me; you might be interested in that a certain patch series are already pushed to distro kernels, and we are definitely interested in whether the patches are merged or dropped. Hmm...

Takashi

Quoted 66 lines

And before everyone screams because I'm not supposed to be pushing stuff that breaks, please realize that it is impossible to test all the different types of hardware I have to merge drivers for. There is no regression test suite or lab setup with anything resembling comprehensive coverage. I test changes to the SCSI core code and do some rudimentary testing on SAS and FC on x86_64. But that's really the best I can do.

Even though most patches posted to linux-scsi get picked up by 0day, more often than not they only get x86_64 build coverage. Whereas 0day build failures on arm, mips, sparc32, whatever typically only get reported after patches have been simmering in linux-next for a while. Depends how busy 0day is.

Also, actual driver failures on platforms not officially supported and tested by the controller vendor are only found after the fact. And most of the time it's not a matter of reverting a single patch but effectively dropping all of the patches in the series until they can be reworked. Sometimes a workaround takes a week or two to deliver, and people don't appreciate not being able to boot their systems in the meantime. So that's why I generally drop the series instead.

I would love for every patch sent to linux-scsi to be bug free and instantly build tested by 0day on every architecture. And I would love for hardware vendors to be more cognizant about architectures they don't commercially support. But reality is that things break frequently when I merge big, complex driver update patch series.

As a result, the preference has been to have the flexibility to amend or drop patches early in every cycle. It hasn't really been a problem because there have been no downstream users of SCSI at all. I only recently found out about your use case.

I'm pretty flexible about how to address this, there are a couple of ways to go about it.

1. I could just always revert instead of dropping the patches. The downside is that we end up with a pretty messy history because, as I pointed out above, it's usually a matter of dropping tens of patches at a time and not reverting a single offending commit. In addition, having a messy history makes it harder on distro kernel people to track driver updates.

2. The other option is that I set up a scsi-staging tree where stuff can simmer for a bit and hopefully get some 0day coverage before getting shuffled over to scsi-queue. However, I do question how much actual real hardware coverage we'll get by having a SCSI tree that people would explicitly have to pull to test. As opposed to linux-next which at least gets some coverage by test farms and users.

3. The third option is that we pick a number like rc4 and I will promise not to rebase after that. We can see how that works out.

I'm open to ideas. The most important thing to me is that 0day and linux-next are indispensable tools in my workflow given that I have no way to personally test most of the code I merge. So it is imperative that I have the ability to push code early because that's where I get the bulk of my (non-SCSI-core) build and test coverage.

Signature · 5 lines
Martin K. Petersen	Oracle Linux Engineering
_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
#18 lore raw
In reply to Takashi Iwai
Quoted 8 lines · Takashi Iwai
Quoted 5 lines · Mark Brown

I used to deal with this by using topic branches heavily and making my -next be an automated merge of those branches, if something went badly I could just throw away the branch. I stopped for a while because Linus didn't like the number of branches I was creating, though it wasn't a problem with the approach in general.

Quoted 5 lines

I think that the topic branch approach would work well if you merge topic branches back to the main branch more often. That is, each topic branch lives only for a relatively short time (e.g. a few weeks), and not merging the whole branches in a shot at the end of the development cycle.

That approach tends to lead to more merges (as you end up with more of the small topic branches) and if you don't have any commits actually on the trunk then the long series of merges is also seen as an antipattern by some.

Quoted 4 lines

Basically the merge to the main branch "fixates" the developments of the given topic, hence other people can start working on the commits safely. Meanwhile the not-yet-merged branches can be still thrown away if they are really bad.

For less widely used stuff it can be unfortunately difficult to get the confidence that things are done :(

#20 lore raw
In reply to Takashi Iwai

Hi Iwai-san,

Quoted 15 lines · Takashi Iwai
Quoted 13 lines · Martin K. Petersen

Sorry I'm a little late to the game here. Been out on vacation.

Quoted 10 lines

We order patches in our trees in the same git-topological-ordering as they are upstream. It has a lot of benefits, most importantly: it doesn't introduce artificial conflicts that don't exist in reality.

In order to achieve that, we of course need 1:1 mapping between our patches and upstream commits. Rebases destroy that mapping.

And in some areas (scsi is one, but not the only one), we basically had no other choice than considering maintainer's tree to be already "upstream enough", without waiting for Linus' tree merge.

Quoted 6 lines
Quoted 4 lines

When I discussed this with Johannes a little while ago, I suggested you guys used git patch-id to track patches instead of commit ids. That's how we track patches applied across many different trees internally. Works much better than using the upstream sha.

What is git patch id? Is there any internal patch tag?

It's the output of "git patch-id", which is a hash of the patch, without including its dependencies. This allows to match identical commits in different branches, and is what "git cherry" and "git rebase" use to detect already applied patches.

Example (from v4.18.y):

$ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
Author: Robert Munteanu <[email protected]>
Date:   Tue Jun 19 11:20:40 2018 +0300

HID: redragon: fix num lock and caps lock LEDs

commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream.

The redragon asura keyboard registers two input devices. The initial commit 85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura
Keyboard")
considered this an error and prevented one of the devices from registering. However, once this is done the num lock and caps lock leds no
longer toggle on
and off, although the key functionality is not affected.

This commit removes the code that prevents the input device registration and restores the num lock and caps lock LEDs.

Fixes 85455dd906d5 ("HID: redragon: Fix modifier keys for

Redragon Asura Keyboard")

$ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id
9d4546a92b459de7249748c0a38b50c989f13bd1
e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
$ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id
9d4546a92b459de7249748c0a38b50c989f13bd1
dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e

So both commits are identical, as they have the same patch ID.

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#21 lore raw
Quoted 59 lines · Geert Uytterhoeven

Hi Iwai-san,

Quoted 15 lines · Takashi Iwai
Quoted 13 lines · Martin K. Petersen

Sorry I'm a little late to the game here. Been out on vacation.

Quoted 10 lines

We order patches in our trees in the same git-topological-ordering as they are upstream. It has a lot of benefits, most importantly: it doesn't introduce artificial conflicts that don't exist in reality.

In order to achieve that, we of course need 1:1 mapping between our patches and upstream commits. Rebases destroy that mapping.

And in some areas (scsi is one, but not the only one), we basically had no other choice than considering maintainer's tree to be already "upstream enough", without waiting for Linus' tree merge.

Quoted 6 lines
Quoted 4 lines

When I discussed this with Johannes a little while ago, I suggested you guys used git patch-id to track patches instead of commit ids. That's how we track patches applied across many different trees internally. Works much better than using the upstream sha.

What is git patch id? Is there any internal patch tag?

It's the output of "git patch-id", which is a hash of the patch, without including its dependencies. This allows to match identical commits in different branches, and is what "git cherry" and "git rebase" use to detect already applied patches.

Example (from v4.18.y):

$ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
Author: Robert Munteanu <[email protected]>
Date:   Tue Jun 19 11:20:40 2018 +0300

HID: redragon: fix num lock and caps lock LEDs

commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream.

The redragon asura keyboard registers two input devices. The initial commit 85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura
Keyboard")
considered this an error and prevented one of the devices from registering. However, once this is done the num lock and caps lock leds no
longer toggle on
and off, although the key functionality is not affected.

This commit removes the code that prevents the input device registration and restores the num lock and caps lock LEDs.

Fixes 85455dd906d5 ("HID: redragon: Fix modifier keys for

Redragon Asura Keyboard")

Quoted 8 lines
$ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id
9d4546a92b459de7249748c0a38b50c989f13bd1
e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
$ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id
9d4546a92b459de7249748c0a38b50c989f13bd1
dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e

So both commits are identical, as they have the same patch ID.

Thanks for the explanation!

This would help for identifying the rebased patch, yes. But it still doesn't help for sorting as is. You cannot get the topology information from the patch id, so you don't know the patch order, either. (You can deduce the actual commit from the patch id, but it's too heavy if you have 20,000 patches :)

Takashi

#22 lore raw
Quoted 70 lines · Takashi Iwai
Quoted 59 lines · Geert Uytterhoeven

Hi Iwai-san,

Quoted 15 lines · Takashi Iwai
Quoted 13 lines · Martin K. Petersen

Sorry I'm a little late to the game here. Been out on vacation.

Quoted 10 lines

We order patches in our trees in the same git-topological-ordering as they are upstream. It has a lot of benefits, most importantly: it doesn't introduce artificial conflicts that don't exist in reality.

In order to achieve that, we of course need 1:1 mapping between our patches and upstream commits. Rebases destroy that mapping.

And in some areas (scsi is one, but not the only one), we basically had no other choice than considering maintainer's tree to be already "upstream enough", without waiting for Linus' tree merge.

Quoted 6 lines
Quoted 4 lines

When I discussed this with Johannes a little while ago, I suggested you guys used git patch-id to track patches instead of commit ids. That's how we track patches applied across many different trees internally. Works much better than using the upstream sha.

What is git patch id? Is there any internal patch tag?

It's the output of "git patch-id", which is a hash of the patch, without including its dependencies. This allows to match identical commits in different branches, and is what "git cherry" and "git rebase" use to detect already applied patches.

Example (from v4.18.y):

$ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
Author: Robert Munteanu <[email protected]>
Date:   Tue Jun 19 11:20:40 2018 +0300

HID: redragon: fix num lock and caps lock LEDs

commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream.

The redragon asura keyboard registers two input devices. The initial commit 85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura
Keyboard")
considered this an error and prevented one of the devices from registering. However, once this is done the num lock and caps lock leds no
longer toggle on
and off, although the key functionality is not affected.

This commit removes the code that prevents the input device registration and restores the num lock and caps lock LEDs.

Fixes 85455dd906d5 ("HID: redragon: Fix modifier keys for

Redragon Asura Keyboard")

Quoted 8 lines
$ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id
9d4546a92b459de7249748c0a38b50c989f13bd1
e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
$ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id
9d4546a92b459de7249748c0a38b50c989f13bd1
dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e

So both commits are identical, as they have the same patch ID.

Quoted 7 lines

Thanks for the explanation!

This would help for identifying the rebased patch, yes. But it still doesn't help for sorting as is. You cannot get the topology information from the patch id, so you don't know the patch order, either. (You can deduce the actual commit from the patch id, but it's too heavy if you have 20,000 patches :)

Actually we could use the patch id to identify new commit id of a rebased patch and infer proper ordering once we have the new commit id. So I think using patch id (additionally to commit id) would really help us and solve 95% of rebases of e.g. SCSI tree by our automation without human interaction.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#23 lore raw
Quoted 80 lines · Jan Kara
Quoted 70 lines · Takashi Iwai
Quoted 59 lines · Geert Uytterhoeven

Hi Iwai-san,

Quoted 15 lines · Takashi Iwai
Quoted 13 lines · Martin K. Petersen

Sorry I'm a little late to the game here. Been out on vacation.

Quoted 10 lines

We order patches in our trees in the same git-topological-ordering as they are upstream. It has a lot of benefits, most importantly: it doesn't introduce artificial conflicts that don't exist in reality.

In order to achieve that, we of course need 1:1 mapping between our patches and upstream commits. Rebases destroy that mapping.

And in some areas (scsi is one, but not the only one), we basically had no other choice than considering maintainer's tree to be already "upstream enough", without waiting for Linus' tree merge.

Quoted 6 lines
Quoted 4 lines

When I discussed this with Johannes a little while ago, I suggested you guys used git patch-id to track patches instead of commit ids. That's how we track patches applied across many different trees internally. Works much better than using the upstream sha.

What is git patch id? Is there any internal patch tag?

It's the output of "git patch-id", which is a hash of the patch, without including its dependencies. This allows to match identical commits in different branches, and is what "git cherry" and "git rebase" use to detect already applied patches.

Example (from v4.18.y):

$ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
Author: Robert Munteanu <[email protected]>
Date:   Tue Jun 19 11:20:40 2018 +0300

HID: redragon: fix num lock and caps lock LEDs

commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream.

The redragon asura keyboard registers two input devices. The initial commit 85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura
Keyboard")
considered this an error and prevented one of the devices from registering. However, once this is done the num lock and caps lock leds no
longer toggle on
and off, although the key functionality is not affected.

This commit removes the code that prevents the input device registration and restores the num lock and caps lock LEDs.

Fixes 85455dd906d5 ("HID: redragon: Fix modifier keys for

Redragon Asura Keyboard")

Quoted 8 lines
$ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id
9d4546a92b459de7249748c0a38b50c989f13bd1
e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
$ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id
9d4546a92b459de7249748c0a38b50c989f13bd1
dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e

So both commits are identical, as they have the same patch ID.

Quoted 7 lines

Thanks for the explanation!

This would help for identifying the rebased patch, yes. But it still doesn't help for sorting as is. You cannot get the topology information from the patch id, so you don't know the patch order, either. (You can deduce the actual commit from the patch id, but it's too heavy if you have 20,000 patches :)

Quoted 5 lines

Actually we could use the patch id to identify new commit id of a rebased patch and infer proper ordering once we have the new commit id. So I think using patch id (additionally to commit id) would really help us and solve 95% of rebases of e.g. SCSI tree by our automation without human interaction.

Yes, we can use the patch id in addition to the commit id as complementary information for tracking the upstream. I chatted with people here now, and it seems feasible.

Thanks for the nice suggestion!

Takashi