Skip to contentSkip to conversation

Services needed from kernel.org

Ksummit-discuss H. Peter Anvin 7 participants 10 messages May 20, 2014 – May 21, 2014

lore
Share & tools
lore
#0 lore raw

I have been discussing with the Linux Foundation about if there are any new IT services needed or desired for kernel developers (maybe or maybe not under the kernel.org.) The kernel.org cleanup and inevitable changes meant we couldn't service some of the needs that we had in the past, some of which has moved to privately owned equipment and some of which has migrated to cloud services or discontinued.

However, the big question is: what IT services do kernel developers need, and how can we best provide them? One idea bandied about would be to provide either managed or unmanaged VMs to kernel developers. Would that reduce the workload on the upper tier of kernel developers so that we can focus more on our time on what actually provides value to our users and employers, and which we hopefully find more interesting/fun?

-hpa

#1 lore raw
Quoted 7 lines · H. Peter Anvin

I have been discussing with the Linux Foundation about if there are any new IT services needed or desired for kernel developers (maybe or maybe not under the kernel.org.) The kernel.org cleanup and inevitable changes meant we couldn't service some of the needs that we had in the past, some of which has moved to privately owned equipment and some of which has migrated to cloud services or discontinued.

Quoted 6 lines

However, the big question is: what IT services do kernel developers need, and how can we best provide them? One idea bandied about would be to provide either managed or unmanaged VMs to kernel developers. Would that reduce the workload on the upper tier of kernel developers so that we can focus more on our time on what actually provides value to our users and employers, and which we hopefully find more interesting/fun?

How feasible is it to support git hooks that want to construct and send (significant volumes of) email, while retaining the security and sandboxing currently being applied to git repositories?

- Josh Triplett

#2 lore raw
Quoted 15 lines · On Tue, 2014-05-20 at 15:53 -0700, [email protected] wrote:
Quoted 7 lines · H. Peter Anvin

I have been discussing with the Linux Foundation about if there are any new IT services needed or desired for kernel developers (maybe or maybe not under the kernel.org.) The kernel.org cleanup and inevitable changes meant we couldn't service some of the needs that we had in the past, some of which has moved to privately owned equipment and some of which has migrated to cloud services or discontinued.

Quoted 6 lines

However, the big question is: what IT services do kernel developers need, and how can we best provide them? One idea bandied about would be to provide either managed or unmanaged VMs to kernel developers. Would that reduce the workload on the upper tier of kernel developers so that we can focus more on our time on what actually provides value to our users and employers, and which we hopefully find more interesting/fun?

Quoted 3 lines

How feasible is it to support git hooks that want to construct and send (significant volumes of) email, while retaining the security and sandboxing currently being applied to git repositories?

Seconded; it's a pain having to run my commit hooks on a different system, particularly as the email is set up to come from kernel.org

James

#3 lore raw
Quoted 6 lines · James Bottomley
Quoted 3 lines

How feasible is it to support git hooks that want to construct and send (significant volumes of) email, while retaining the security and sandboxing currently being applied to git repositories?

Seconded; it's a pain having to run my commit hooks on a different system, particularly as the email is set up to come from kernel.org

I hate e-mails which contain just '+1' and nothing else, so I'd also add that I find this very nice-to-have feature as well, as discussed at [1] already :)

[1] http://lists.linuxfoundation.org/pipermail/ksummit-discuss/2014-May/000489.html

Thanks,

Signature · 2 lines
Jiri Kosina
SUSE Labs
#4 lore raw
In reply to [email protected]
Quoted 3 lines · On 20/05/14 06:53 PM, [email protected] wrote:

How feasible is it to support git hooks that want to construct and send (significant volumes of) email, while retaining the security and sandboxing currently being applied to git repositories?

It's feasible, but I can't make this a free-for-all, for obvious reasons. :) With gitolite, hooks execute as the same user "git" and therefore any code running inside a hook has unfettered access to all git repositories regardless of in-gitolite repository permissions.

What we can do is have a peer-reviewed collection of "blessed" hooks available to developers. Not being a kernel dev myself, I'm not the one to put such a collection together, though -- I'm not even sure if such a cookie-cutter approach would be suitable.

I guess what I'm really trying to say is that I'd rather avoid having to continuously review code written by people to whom "you write perl like it's C" is a compliment. ;) If we can get by with a handful of standard hooks, I'm for it, though.

Best,

Signature · 4 lines
Konstantin Ryabitsev
Senior Systems Administrator
Linux Foundation Collab Projects
Montréal, Québec
#5 lore raw
Quoted 17 lines · Konstantin Ryabitsev
Quoted 4 lines · On 20/05/14 06:53 PM, [email protected] wrote:

How feasible is it to support git hooks that want to construct and send (significant volumes of) email, while retaining the security and sandboxing currently being applied to git repositories?

It's feasible, but I can't make this a free-for-all, for obvious reasons. :) With gitolite, hooks execute as the same user "git" and therefore any code running inside a hook has unfettered access to all git repositories regardless of in-gitolite repository permissions.

What we can do is have a peer-reviewed collection of "blessed" hooks available to developers. Not being a kernel dev myself, I'm not the one to put such a collection together, though -- I'm not even sure if such a cookie-cutter approach would be suitable.

Quoted 5 lines

I guess what I'm really trying to say is that I'd rather avoid having to continuously review code written by people to whom "you write perl like it's C" is a compliment. ;) If we can get by with a handful of standard hooks, I'm for it, though.

Hmm... I resemble that remark.

That being said, there is no fundamental reason that email generator have to be run on the master copy. It can be run on an entirely different box, which doesn't even need any kind of privileged access to the machinery; this is how the tip-bot runs: it is just a "regular puller" of the tip tree, using the publicly accessible git port. This does add some latency, but has the advantage that it sends out the emails once the commits are actually visible to general users upstream.

Generalizing the tip-bot is probably not that hard. I already use variants of it for multiple projects. It does have a *lot* of configuration options, though, some of which are in the form of scripts. This pretty much means that in its current form it is equivalent to a shell account and would have to be sandboxed accordingly.

-hpa

#6 lore raw
Quoted 34 lines · H. Peter Anvin
Quoted 17 lines · Konstantin Ryabitsev
Quoted 4 lines · On 20/05/14 06:53 PM, [email protected] wrote:

How feasible is it to support git hooks that want to construct and send (significant volumes of) email, while retaining the security and sandboxing currently being applied to git repositories?

It's feasible, but I can't make this a free-for-all, for obvious reasons. :) With gitolite, hooks execute as the same user "git" and therefore any code running inside a hook has unfettered access to all git repositories regardless of in-gitolite repository permissions.

What we can do is have a peer-reviewed collection of "blessed" hooks available to developers. Not being a kernel dev myself, I'm not the one to put such a collection together, though -- I'm not even sure if such a cookie-cutter approach would be suitable.

Quoted 5 lines

I guess what I'm really trying to say is that I'd rather avoid having to continuously review code written by people to whom "you write perl like it's C" is a compliment. ;) If we can get by with a handful of standard hooks, I'm for it, though.

Hmm... I resemble that remark.

That being said, there is no fundamental reason that email generator have to be run on the master copy. It can be run on an entirely different box, which doesn't even need any kind of privileged access to the machinery; this is how the tip-bot runs: it is just a "regular puller" of the tip tree, using the publicly accessible git port. This does add some latency, but has the advantage that it sends out the emails once the commits are actually visible to general users upstream.

Quoted 6 lines

Generalizing the tip-bot is probably not that hard. I already use variants of it for multiple projects. It does have a *lot* of configuration options, though, some of which are in the form of scripts. This pretty much means that in its current form it is equivalent to a shell account and would have to be sandboxed accordingly.

There *must* be a sensible way to construct a sandbox these days. Some combination of user namespaces, seccomp, and maybe even a VM should be quite secure and even fairly easy to use. Not to mention that various scripting languages should have viable sandbox modes.

I'm sort of working on a userns-based general-purpose sandbox. Maybe I'll have something usable to play with at the kernel summit.

--Andy

#7 lore raw
In reply to H. Peter Anvin
Quoted 34 lines · H. Peter Anvin
Quoted 17 lines · Konstantin Ryabitsev
Quoted 4 lines · On 20/05/14 06:53 PM, [email protected] wrote:

How feasible is it to support git hooks that want to construct and send (significant volumes of) email, while retaining the security and sandboxing currently being applied to git repositories?

It's feasible, but I can't make this a free-for-all, for obvious reasons. :) With gitolite, hooks execute as the same user "git" and therefore any code running inside a hook has unfettered access to all git repositories regardless of in-gitolite repository permissions.

What we can do is have a peer-reviewed collection of "blessed" hooks available to developers. Not being a kernel dev myself, I'm not the one to put such a collection together, though -- I'm not even sure if such a cookie-cutter approach would be suitable.

Quoted 5 lines

I guess what I'm really trying to say is that I'd rather avoid having to continuously review code written by people to whom "you write perl like it's C" is a compliment. ;) If we can get by with a handful of standard hooks, I'm for it, though.

Hmm... I resemble that remark.

That being said, there is no fundamental reason that email generator have to be run on the master copy. It can be run on an entirely different box, which doesn't even need any kind of privileged access to the machinery; this is how the tip-bot runs: it is just a "regular puller" of the tip tree, using the publicly accessible git port. This does add some latency, but has the advantage that it sends out the emails once the commits are actually visible to general users upstream.

Quoted 5 lines

Generalizing the tip-bot is probably not that hard. I already use variants of it for multiple projects. It does have a *lot* of configuration options, though, some of which are in the form of scripts. This pretty much means that in its current form it is equivalent to a shell account and would have to be sandboxed accordingly.

That's exactly the kind of approach I had in mind: let's not run this on git.kernel.org or gitolite.kernel.org, but rather in a separate sandbox.

However, I don't think we need the full generality of tip-bot's shell configuration; I'd personally be fine with Konstantin's proposal of a curated set of hook scripts, perhaps enabled/disabled and configured via a git branch in each repository.

- Josh Triplett

#8 lore raw
In reply to H. Peter Anvin

I posted a list of niggles about the cgit interface over the gitweb interface a while back (see below). As far as I can tell, none of them got addressed.

David
---
There some things in gitweb that are missing from the cgit interface on git.kernel.org that would be nice to have:

(1) Could the patch view screen be made to show tabs expanded? (Both in the diff and in the description).

(2) If an additional remote was pushed into a kernel tree, IIRC with gitweb its branches used to show up in the tree in pink boxes - see:

http://git.infradead.org/users/dhowells/linux-headers.git/shortlog/refs/tags/disintegrate-fbdev-20121220

where you can see:

linus/master

in a pink box. Could this be brought back please?

(3) Tag entries in the tag list on the front page had a "shortlog" button associated with them in gitweb that would show the log starting from that tag - but that's no longer so easy to do.

(4) Author names are no longer clickable to get a list of patches by that author.

#9 lore raw
Quoted 2 lines · David Howells

I posted a list of niggles about the cgit interface over the gitweb interface a while back (see below). As far as I can tell, none of them got addressed.

Cgit is an upstream project, so best I can do is pass these feature requests upstream -- which I did.

Quoted 10 lines

(2) If an additional remote was pushed into a kernel tree, IIRC with gitweb its branches used to show up in the tree in pink boxes - see:

http://git.infradead.org/users/dhowells/linux-headers.git/shortlog/refs/tags/disintegrate-fbdev-20121220

where you can see:

linus/master

in a pink box. Could this be brought back please?

I'm not sure we can do that, as git.kernel.org is a "git clone --mirror" of the actual repositories on gitolite.kernel.org, so are a) bare, and
b) wouldn't have any remotes other than origin. Unless I'm misunderstanding the functionality.

Quoted 2 lines

(4) Author names are no longer clickable to get a list of patches by that author.

I can add this easily via a filter. Currently, to do the same, you just need to use the search box and copy-paste the author name.

Regards,

Signature · 4 lines
Konstantin Ryabitsev
Senior Systems Administrator
Linux Foundation Collab Projects
Montréal, Québec