New list for people to share maintainer workflows
At the Maintainer's Summit yesterday, we created a new mailing list:
[email protected], where various Maintainers can share their workflows for handling patch review, collection, testing, and submission.
We will also be discussing what requirements should be for infrastructure that will be best suited for common use. That is, if we can find rough agreement on what it is needed to make kernel development more scalable, efficient, and fun, what would we hand as a set of requirements to a development team that could be funded by our various sponsors to turn into reality.
Please note that this is a discussion about *requirements* not implementation. So for example, if one of the requirements were:
"it should be like patchwork, in that it is fully compatible with e-mail review, except it should work off-line and use something like git as its transport layer"
it doesn't mean that patchwork would be used as its implementation layer.
Or, if there is a requirement such as,
"it should have a web interface, and it should be easy to pull down patch series via a git pull, and we should be able to easily view diffs between the v49 and v50 version of the patch series"
... it also doesn't follow that Gerrit should be the basis of the implentation. (In fact, both of these requirements were expressed as requirements at the Maintainer's Summit, and there was general agreement that both of these would be highly desirable as requirements.)
Of course, if the Gerrit and patchwork teams would like to participate in the discussion, and work with a smaller working group we will be forming to refine the requirements and to work with the LF and other companies to find said funding to implement what these requirements should look like, that would be excellent. I think it's fair to say that in their current forms, *neither* Gerrit nor patchwork completely meets all of the needs of the kernel development community as a whole, and as Dmitry stated in his "Reflections on Kernel Development Process" talk, spreading out our efforts towards improving engineering productivity may not be the best path to succeess.
Let's continue the discussion on [email protected].
Cheers,
- Ted
_______________________________________________
Ksummit-discuss mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
Quoted 1 line · Theodore Y. Ts'o
Let's continue the discussion on [email protected].
Hi,
Will this be archived on lore.kernel.org?
thnx.
Signature · 1 line
~Randy
Quoted 6 lines · Randy Dunlap
Quoted 1 line · Theodore Y. Ts'o
Let's continue the discussion on [email protected].
Hi,
Will this be archived on lore.kernel.org?
It is already there:
https://lore.kernel.org/workflows/
Hi Ted! Long email incoming.
I'm very interested in solving these kinds of problems. By way of introduction, I'm the maintainer of two relevant pieces of free (as in freedom) software:
<begin introduction>
https://sourcehut.org/
https://aerc-mail.org/
Source code:
https://git.sr.ht/~sircmpwn?search=sr.ht
https://git.sr.ht/~sircmpwn/aerc
AGPL and MIT licensed, respectively, and the former is also available as a hosted service. Sourcehut is a software hosting platform like Github and Gitlab, but it prefers the mailing list workflow. We host git repos, mailing lists, and continuous integration using KVM. Each of the services like git repos and bug tracking are standalone and can be used ala-carte and integrated with external services.
Here's an example archive for a development mailing list:
https://lists.sr.ht/~sircmpwn/sr.ht-dev
Note that patches are recognized like patchworks, and feedback written in the usual style of mailing list patch review is interpreted and shown inline to make a comfortable web interface. Here's an example of a patch with some feedback:
https://lists.sr.ht/~sircmpwn/ctools/patches/8134
Over time this will evolve into an interface similar to Gerrit. Feeding emails into automated testing and sending the results back to the list is another feature I have planned before October. Automated testing of kernels seems feasible to me given that the CIs run on KVM rather than containers or something similar. Here's an example build of a Wayland compositor on Alpine Linux: https://builds.sr.ht/~sircmpwn/job/89260
Click "view manifest" to see the YAML which triggered this, though it has been programmatically modified and the pretty YAML formatting didn't survive. The needs of Linux in particular are probably not addressed by the upstream software, but it might make for a good jumping off point for a custom solution.
Things like tracking bugs can also be done over email and doesn't require an account. With an account, though, you can use web-based tools to do the same tasks that can be done with email. For example, you'll soon be able to prepare patches on the web as well, so you needn't configure git send-email locally. I have written a nice tutorial for those who want to, though: https://git-send-email.io
This plays into a general theme with all of this software, which is building on top of what people are already using (email), rather than trying to build a new system from the ground up. We can improve the user experience without making people drop workflows that they're already productive with.
aerc is a new mail client I'm working on that aims to replace mutt and have first-class git integration. The git-specific features haven't materialized yet so there's not much to introduce, it's just a better mutt right now.
With introductions out of the way...
<end introduction>
Problems I've faced with trying to advanace email-driven development into 2019 include:
# Identifying different versions of the same patchset.
Like you mentioned, no one likes Change Ids, including me. So far the best option I've considered is checking the date - it gets perserved even through rebases and is /basically/ unique. Enough rebasing and it *will* get lost, though, and then manual intervention is required.
I've considered a few ways to address this. One route might be to make git send-email smarter, and utilize heuristics on the reflog to try and identify that a patch is a -v2. Then we can add an email header like Superceeds: <commit or message id or something new>.
Less desirable options would affect people's workflows, which is something I'd like to avoid. One example is adding new commands which explicitly track the evolution of a feature in your local git repository.
# Pulling down patches to use locally
On the web archives I have a copy-pasteable command which curls down a patchset and pipes it to git am -3. This seems like a fairly comfortable solution to me, but this does seem attractive:
Quoted 3 lines
"it should have a web interface, and it should be easy to pull down patch series via a git pull, and we should be able to easily view diffs between the v49 and v50 version of the patch series"
But, I think this could end up being problematic. We have a similar example already in GitHub: every pull request is available for fetching from the remote. For one of my bigger projects, however, the result is this:
~/s/wlroots > git ls-remote | wc -l From [email protected]:swaywm/wlroots 1255
At the scale of the Linux kernel I imagine this would get seriously out of hand. Another problem would be mapping mailing lists to repos. What git repository do patches on LKML belong to?
Solvable problems, I think, but something to consider.
# Storage medium
Quoted 3 lines
"it should be like patchwork, in that it is fully compatible with e-mail review, except it should work off-line and use something like git as its transport layer"
I'm not sure how much this was example idea vs actual idea, but let's assume actual. I just wanted to mention that mail spools work offline and are already pretty distributed. There's no doubt thousands of copies of LKML spread about the internet. Maybe this one doesn't need solving.
Quoted 11 lines · Drew DeVault
# Identifying different versions of the same patchset.
Like you mentioned, no one likes Change Ids, including me. So far the best option I've considered is checking the date - it gets perserved even through rebases and is /basically/ unique. Enough rebasing and it *will* get lost, though, and then manual intervention is required.
I've considered a few ways to address this. One route might be to make git send-email smarter, and utilize heuristics on the reflog to try and identify that a patch is a -v2. Then we can add an email header like
Superceeds: <commit or message id or something new>.
The date however is lost across git-send-email. However, you can match series, rather than patches; a Superseeds header would be nice for that, but often times it is enough to match on the subject to get a decent result.
It's then possible to run a minimal matching algorithm to identify a patch that "moves" across revisions of the series. This is how "git range-diff" works.
By the way, let me introduce Patchew! [1] Patchew was started when patchwork seemed to be mostly dead, so it is somewhat similar to Patchwork 2.0. But it has some nice functionality such as "git range-diff"-like version comparison, and especially the patch importer pushes each submitted series to git, complete with Reviewed-by tags and the like. The result is then used to run test scripts. It also has a REST API and a simple plugin API, so it should also be quite easy to write new plugins to automatically parse syzbot or 0day emails and turn them into test failures.
Paolo
[1] I have mentioned Patchew before on the kernel.org users mailing list. The largest project using Patchew is QEMU, whose "patch list" page on patchew.org is at https://patchew.org/QEMU/.
Quoted 4 lines
Less desirable options would affect people's workflows, which is something I'd like to avoid. One example is adding new commands which explicitly track the evolution of a feature in your local git repository.