Add eBPF hooks for cgroups

This is v5 of the patch set to allow eBPF programs for network filtering and accounting to be attached to cgroups, so that they apply to all sockets of all tasks placed in that cgroup. The logic also allows to be extendeded for other cgroup based eBPF logic.

After chatting with Daniel Borkmann and Alexei off-list, we concluded that __dev_queue_xmit() is the place where the egress hooks should live when eBPF programs need access to the L2 bits of the skb.

Quoted 4 lines · Daniel Mack

This is v5 of the patch set to allow eBPF programs for network filtering and accounting to be attached to cgroups, so that they apply to all sockets of all tasks placed in that cgroup. The logic also allows to be extendeded for other cgroup based eBPF logic.

1) This infrastructure can only be useful to systemd, or any similar orchestration daemon. Look, you can only apply filtering policies to processes that are launched by systemd, so this only works for server processes.

For client processes this infrastructure is
*racy*, you have to add new processes in runtime to the cgroup, thus there will be time some little time where no filtering policy will be applied. For quality of service, this may be an acceptable race, but this is aiming to deploy a filtering policy.

2) This aproach looks uninfrastructured to me. This provides a hook to push a bpf blob at a place in the stack that deploys a filtering policy that is not visible to others.

We have interfaces that allows
us to dump the filtering policy that is being applied, report events to enable cooperation between several processes with similar capabilities and so on.

Quoted 3 lines

After chatting with Daniel Borkmann and Alexei off-list, we concluded that __dev_queue_xmit() is the place where the egress hooks should live when eBPF programs need access to the L2 bits of the skb.

3) This egress hook is coming very late, the only reason I find to place it at __dev_queue_xmit() is that bpf naturally works with layer 2 information in place. But this new hook is placed in _everyone's output ath_ that only works for the very specific usecase I exposed above.

The main concern during the workshop was that a hook only for cgroups is too specific, but this is actually even more specific than this.

I have nothing against systemd or the needs for more programmability/flexibility in the stack, but I think this needs to fulfill some requirements to fit into the infrastructure that we have in the right way.

Quoted 10 lines · Pablo Neira Ayuso

Quoted 4 lines · Daniel Mack

This is v5 of the patch set to allow eBPF programs for network filtering and accounting to be attached to cgroups, so that they apply to all sockets of all tasks placed in that cgroup. The logic also allows to be extendeded for other cgroup based eBPF logic.

1) This infrastructure can only be useful to systemd, or any similar orchestration daemon. Look, you can only apply filtering policies to processes that are launched by systemd, so this only works for server processes.

Sorry, but both statements aren't true. The eBPF policies apply to every process that is placed in a cgroup, and my example program in 6/6 shows how that can be done from the command line. Also, systemd is able to control userspace processes just fine, and it not limited to 'server processes'.

Quoted 5 lines

For client processes this infrastructure is
*racy*, you have to add new processes in runtime to the cgroup, thus there will be time some little time where no filtering policy will be applied. For quality of service, this may be an acceptable race, but this is aiming to deploy a filtering policy.

That's a limitation that applies to many more control mechanisms in the kernel, and it's something that can easily be solved with fork+exec.

Quoted 3 lines

2) This aproach looks uninfrastructured to me. This provides a hook to push a bpf blob at a place in the stack that deploys a filtering policy that is not visible to others.

That's just as transparent as SO_ATTACH_FILTER. What kind of introspection mechanism do you have in mind?

Quoted 4 lines

We have interfaces that allows
us to dump the filtering policy that is being applied, report events to enable cooperation between several processes with similar capabilities and so on.

Well, in practice, for netfilter, there can only be one instance in the system that acts as central authoritative, otherwise you'll end up with orphaned entries or with situation where some client deletes rules behind the back of the one that originally installed it. So I really think there is nothing wrong with demanding a single, privileged controller to manage things.

Quoted 9 lines

Quoted 3 lines

After chatting with Daniel Borkmann and Alexei off-list, we concluded that __dev_queue_xmit() is the place where the egress hooks should live when eBPF programs need access to the L2 bits of the skb.

3) This egress hook is coming very late, the only reason I find to place it at __dev_queue_xmit() is that bpf naturally works with layer 2 information in place. But this new hook is placed in _everyone's output ath_ that only works for the very specific usecase I exposed above.

It's about filtering outgoing network packets of applications, and providing them with L2 information for filtering purposes. I don't think that's a very specific use-case.

When the feature is not used at all, the added costs on the output path are close to zero, due to the use of static branches. If used somewhere in the system but not for the packet in flight, costs are slightly higher but acceptable. In fact, it's not even measurable in my tests here. How is that different from the netfilter OUTPUT hook, btw?

That said, limiting it to L3 is still an option. It's just that we need ingress and egress to be in sync, so both would be L3 then. So far, the possible advantages for future use-cases having access to L2 outweighed the concerns of putting the hook to dev_queue_xmit(), but I'm open to discussing that.

Quoted 2 lines

The main concern during the workshop was that a hook only for cgroups is too specific, but this is actually even more specific than this.

This patch set merely implements an infrastructure that can accommodate many more things as well in the future. We could, in theory, even add hooks for forwarded packets specifically, or other eBPF programs, not even for network filtering etc.

Quoted 4 lines

I have nothing against systemd or the needs for more programmability/flexibility in the stack, but I think this needs to fulfill some requirements to fit into the infrastructure that we have in the right way.

Well, as I explained already, this patch set results from endless discussions that went nowhere, about how such a thing can be achieved with netfilter.

Thanks,
Daniel

Hi,

Quoted 6 lines · Pablo Neira Ayuso

Quoted 4 lines · Daniel Mack

This is v5 of the patch set to allow eBPF programs for network filtering and accounting to be attached to cgroups, so that they apply to all sockets of all tasks placed in that cgroup. The logic also allows to be extendeded for other cgroup based eBPF logic.

Quoted 4 lines

1) This infrastructure can only be useful to systemd, or any similar orchestration daemon. Look, you can only apply filtering policies to processes that are launched by systemd, so this only works for server processes.

Sorry, but both statements aren't true. The eBPF policies apply to every process that is placed in a cgroup, and my example program in 6/6 shows how that can be done from the command line.

Also, systemd is able to control userspace processes just fine, and it not limited to 'server processes'.

Quoted 5 lines

For client processes this infrastructure is
*racy*, you have to add new processes in runtime to the cgroup, thus there will be time some little time where no filtering policy will be applied. For quality of service, this may be an acceptable race, but this is aiming to deploy a filtering policy.

That's a limitation that applies to many more control mechanisms in the kernel, and it's something that can easily be solved with fork+exec.

Quoted 3 lines

2) This aproach looks uninfrastructured to me. This provides a hook to push a bpf blob at a place in the stack that deploys a filtering policy that is not visible to others.

That's just as transparent as SO_ATTACH_FILTER. What kind of introspection mechanism do you have in mind?

Quoted 9 lines

Quoted 3 lines

After chatting with Daniel Borkmann and Alexei off-list, we concluded that __dev_queue_xmit() is the place where the egress hooks should live when eBPF programs need access to the L2 bits of the skb.

3) This egress hook is coming very late, the only reason I find to place it at __dev_queue_xmit() is that bpf naturally works with layer 2 information in place. But this new hook is placed in _everyone's output ath_ that only works for the very specific usecase I exposed above.

It's about filtering outgoing network packets of applications, and providing them with L2 information for filtering purposes. I don't think that's a very specific use-case.

When the feature is not used at all, the added costs on the output path are close to zero, due to the use of static branches.

Quoted 4 lines

I have nothing against systemd or the needs for more programmability/flexibility in the stack, but I think this needs to fulfill some requirements to fit into the infrastructure that we have in the right way.

Well, as I explained already, this patch set results from endless discussions that went nowhere, about how such a thing can be achieved with netfilter.

Quoted 14 lines · Daniel Mack

Hi,

Quoted 10 lines · Pablo Neira Ayuso

Quoted 4 lines · Daniel Mack

This is v5 of the patch set to allow eBPF programs for network filtering and accounting to be attached to cgroups, so that they apply to all sockets of all tasks placed in that cgroup. The logic also allows to be extendeded for other cgroup based eBPF logic.

1) This infrastructure can only be useful to systemd, or any similar orchestration daemon. Look, you can only apply filtering policies to processes that are launched by systemd, so this only works for server processes.

Quoted 3 lines

Sorry, but both statements aren't true. The eBPF policies apply to every process that is placed in a cgroup, and my example program in 6/6 shows how that can be done from the command line.

Then you have to explain me how can anyone else than systemd use this infrastructure?

Quoted 2 lines

Also, systemd is able to control userspace processes just fine, and it not limited to 'server processes'.

My main point is that those processes *need* to be launched by the orchestrator, which is was refering as 'server processes'.

Quoted 6 lines

Quoted 5 lines

For client processes this infrastructure is
*racy*, you have to add new processes in runtime to the cgroup, thus there will be time some little time where no filtering policy will be applied. For quality of service, this may be an acceptable race, but this is aiming to deploy a filtering policy.

Quoted 2 lines

That's a limitation that applies to many more control mechanisms in the kernel, and it's something that can easily be solved with fork+exec.

As long as you have control to launch the processes yes, but this will not work in other scenarios. Just like cgroup net_cls and friends are broken for filtering for things that you have no control to fork+exec.

To use this infrastructure from a non-launcher process, you'll have to rely on the proc connection to subscribe to new process events, then echo that pid to the cgroup, and that interface is asynchronous so *adding new processes to the cgroup is subject to races*.

*You're proposing a socket filtering facility that hooks layer 2 output path*!

Quoted 2 lines · Pablo Neira Ayuso

Then you have to explain me how can anyone else than systemd use this infrastructure?

Jokes aside. I'm puzzled why systemd is even being mentioned here. Here we use tupperware (our internal container management system) that is heavily using cgroups and has nothing to do with systemd.

For us this cgroup+bpf is _not_ for filterting and _not_ for security. We run a ton of tasks in cgroups that launch all sorts of
things on their own. We need to monitor what they do from networking point of view. Therefore bpf programs need to monitor the traffic in particular part of cgroup hierarchy. Not globally and no pass/drop decisions.

For us this cgroup+bpf is _not_ for filterting and _not_ for security.

I'd really love to have an alternative to bpf for such tasks, but you seem to spend all the energy arguing against bpf whereas nft still has a lot to be desired.

On Tue, Sep 13, 2016 at 09:42:19PM -0700, Alexei Starovoitov wrote:
[...]

Quoted 1 line

For us this cgroup+bpf is _not_ for filterting and _not_ for security.

If your goal is monitoring, then convert these hooks not to allow to issue a verdict on the packet, so this becomes inoquous in the same fashion as the tracing infrastructure.

On Tue, Sep 13, 2016 at 09:42:19PM -0700, Alexei Starovoitov wrote:
[...]

Quoted 1 line

For us this cgroup+bpf is _not_ for filterting and _not_ for security.

If your goal is monitoring, then convert these hooks not to allow to issue a verdict on the packet, so this becomes inoquous in the same fashion as the tracing infrastructure.

[...]

Quoted 3 lines

I'd really love to have an alternative to bpf for such tasks, but you seem to spend all the energy arguing against bpf whereas nft still has a lot to be desired.

Please Alexei, stop that FUD. Anyone that has spent just one day using the bpf tooling and infrastructure knows you have problems to resolve...

Quoted 12 lines · Daniel Mack

Quoted 10 lines · Pablo Neira Ayuso

Quoted 4 lines · Daniel Mack

This is v5 of the patch set to allow eBPF programs for network filtering and accounting to be attached to cgroups, so that they apply to all sockets of all tasks placed in that cgroup. The logic also allows to be extendeded for other cgroup based eBPF logic.

1) This infrastructure can only be useful to systemd, or any similar orchestration daemon. Look, you can only apply filtering policies to processes that are launched by systemd, so this only works for server processes.

Quoted 3 lines

Sorry, but both statements aren't true. The eBPF policies apply to every process that is placed in a cgroup, and my example program in 6/6 shows how that can be done from the command line.

Then you have to explain me how can anyone else than systemd use this infrastructure?

My main point is that those processes *need* to be launched by the orchestrator, which is was refering as 'server processes'.

Quoted 2 lines

That's a limitation that applies to many more control mechanisms in the kernel, and it's something that can easily be solved with fork+exec.

As long as you have control to launch the processes yes, but this will not work in other scenarios. Just like cgroup net_cls and friends are broken for filtering for things that you have no control to fork+exec.

Quoted 2 lines

That's just as transparent as SO_ATTACH_FILTER. What kind of introspection mechanism do you have in mind?

SO_ATTACH_FILTER is called from the process itself, so this is a local filtering policy that you apply to your own process.

Quoted 6 lines

It's about filtering outgoing network packets of applications, and providing them with L2 information for filtering purposes. I don't think that's a very specific use-case.

When the feature is not used at all, the added costs on the output path are close to zero, due to the use of static branches.

*You're proposing a socket filtering facility that hooks layer 2 output path*!

That is only a rough ~30 lines kernel patchset to support this in netfilter and only one extra input hook, with potential access to conntrack and better integration with other existing subsystems.

Quoted 13 lines · Pablo Neira Ayuso

Quoted 12 lines · Daniel Mack

Quoted 10 lines · Pablo Neira Ayuso

Quoted 4 lines · Daniel Mack

This is v5 of the patch set to allow eBPF programs for network filtering and accounting to be attached to cgroups, so that they apply to all sockets of all tasks placed in that cgroup. The logic also allows to be extendeded for other cgroup based eBPF logic.

1) This infrastructure can only be useful to systemd, or any similar orchestration daemon. Look, you can only apply filtering policies to processes that are launched by systemd, so this only works for server processes.

Quoted 6 lines

Quoted 3 lines

Sorry, but both statements aren't true. The eBPF policies apply to every process that is placed in a cgroup, and my example program in 6/6 shows how that can be done from the command line.

Then you have to explain me how can anyone else than systemd use this infrastructure?

I have no idea what makes you think this is limited to systemd. As I said, I provided an example for userspace that works from the command line. The same limitation apply as for all other users of cgroups.

Quoted 2 lines

My main point is that those processes *need* to be launched by the orchestrator, which is was refering as 'server processes'.

Yes, that's right. But as I said, this rule applies to many other kernel concepts, so I don't see any real issue.

Quoted 7 lines

Quoted 2 lines

That's a limitation that applies to many more control mechanisms in the kernel, and it's something that can easily be solved with fork+exec.

As long as you have control to launch the processes yes, but this will not work in other scenarios. Just like cgroup net_cls and friends are broken for filtering for things that you have no control to fork+exec.

Probably, but that's only solvable with rules that store the full cgroup path then, and do a string comparison (!) for each packet flying by.

Quoted 5 lines

Quoted 2 lines

That's just as transparent as SO_ATTACH_FILTER. What kind of introspection mechanism do you have in mind?

SO_ATTACH_FILTER is called from the process itself, so this is a local filtering policy that you apply to your own process.

Not necessarily. You can as well do it the inetd way, and pass the socket to a process that is launched on demand, but do SO_ATTACH_FILTER
+ SO_LOCK_FILTER in the middle. What happens with payload on the socket is not transparent to the launched binary at all. The proposed cgroup eBPF solution implements a very similar behavior in that regard.

Quoted 7 lines

Quoted 6 lines

It's about filtering outgoing network packets of applications, and providing them with L2 information for filtering purposes. I don't think that's a very specific use-case.

When the feature is not used at all, the added costs on the output path are close to zero, due to the use of static branches.

Quoted 2 lines

*You're proposing a socket filtering facility that hooks layer 2 output path*!

As I said, I'm open to discussing that. In order to make it work for L3, the LL_OFF issues need to be solved, as Daniel explained. Daniel, Alexei, any idea how much work that would be?

Quoted 3 lines

That is only a rough ~30 lines kernel patchset to support this in netfilter and only one extra input hook, with potential access to conntrack and better integration with other existing subsystems.

Care to share the patches for that? I'd really like to have a look.

And FWIW, I agree with Thomas - there is nothing wrong with having multiple options to use for such use-cases.

Thanks,
Daniel

Quoted 3 lines

As I said, I'm open to discussing that. In order to make it work for L3, the LL_OFF issues need to be solved, as Daniel explained. Daniel, Alexei, any idea how much work that would be?

Not much. You simply need to declare your own struct bpf_verifier_ops with a get_func_proto() handler that handles BPF_FUNC_skb_load_bytes, and verifier in do_check() loop would need to handle that these ld_abs/ ld_ind are rejected for BPF_PROG_TYPE_CGROUP_SOCKET.

Hi Pablo,

Quoted 17 lines · Pablo Neira Ayuso

Quoted 15 lines · Daniel Mack

Quoted 10 lines · Pablo Neira Ayuso

Quoted 4 lines · Daniel Mack

This is v5 of the patch set to allow eBPF programs for network filtering and accounting to be attached to cgroups, so that they apply to all sockets of all tasks placed in that cgroup. The logic also allows to be extendeded for other cgroup based eBPF logic.

1) This infrastructure can only be useful to systemd, or any similar orchestration daemon. Look, you can only apply filtering policies to processes that are launched by systemd, so this only works for server processes.

Sorry, but both statements aren't true. The eBPF policies apply to every process that is placed in a cgroup, and my example program in 6/6 shows how that can be done from the command line.

Quoted 2 lines

Then you have to explain me how can anyone else than systemd use this infrastructure?

I have no idea what makes you think this is limited to systemd. As I said, I provided an example for userspace that works from the command line. The same limitation apply as for all other users of cgroups.

Quoted 2 lines

My main point is that those processes *need* to be launched by the orchestrator, which is was refering as 'server processes'.

Yes, that's right. But as I said, this rule applies to many other kernel concepts, so I don't see any real issue.

Quoted 7 lines

Quoted 2 lines

That's a limitation that applies to many more control mechanisms in the kernel, and it's something that can easily be solved with fork+exec.

As long as you have control to launch the processes yes, but this will not work in other scenarios. Just like cgroup net_cls and friends are broken for filtering for things that you have no control to fork+exec.

Probably, but that's only solvable with rules that store the full cgroup path then, and do a string comparison (!) for each packet flying by.

Quoted 3 lines

Quoted 2 lines

That's just as transparent as SO_ATTACH_FILTER. What kind of introspection mechanism do you have in mind?

Quoted 2 lines

SO_ATTACH_FILTER is called from the process itself, so this is a local filtering policy that you apply to your own process.

Not necessarily. You can as well do it the inetd way, and pass the socket to a process that is launched on demand, but do SO_ATTACH_FILTER
+ SO_LOCK_FILTER in the middle. What happens with payload on the socket is not transparent to the launched binary at all. The proposed cgroup eBPF solution implements a very similar behavior in that regard.

Quoted 9 lines

Quoted 6 lines

It's about filtering outgoing network packets of applications, and providing them with L2 information for filtering purposes. I don't think that's a very specific use-case.

When the feature is not used at all, the added costs on the output path are close to zero, due to the use of static branches.

*You're proposing a socket filtering facility that hooks layer 2 output path*!

As I said, I'm open to discussing that. In order to make it work for L3, the LL_OFF issues need to be solved, as Daniel explained. Daniel, Alexei, any idea how much work that would be?

Quoted 3 lines

That is only a rough ~30 lines kernel patchset to support this in netfilter and only one extra input hook, with potential access to conntrack and better integration with other existing subsystems.

Care to share the patches for that? I'd really like to have a look.

And FWIW, I agree with Thomas - there is nothing wrong with having multiple options to use for such use-cases.

Thanks,
Daniel

Quoted 20 lines · Daniel Mack

Hi Pablo,

Quoted 17 lines · Pablo Neira Ayuso

Quoted 15 lines · Daniel Mack

Quoted 10 lines · Pablo Neira Ayuso

Quoted 4 lines · Daniel Mack

This is v5 of the patch set to allow eBPF programs for network filtering and accounting to be attached to cgroups, so that they apply to all sockets of all tasks placed in that cgroup. The logic also allows to be extendeded for other cgroup based eBPF logic.

1) This infrastructure can only be useful to systemd, or any similar orchestration daemon. Look, you can only apply filtering policies to processes that are launched by systemd, so this only works for server processes.

Sorry, but both statements aren't true. The eBPF policies apply to every process that is placed in a cgroup, and my example program in 6/6 shows how that can be done from the command line.

Quoted 7 lines

Quoted 2 lines

Then you have to explain me how can anyone else than systemd use this infrastructure?

I have no idea what makes you think this is limited to systemd. As I said, I provided an example for userspace that works from the command line. The same limitation apply as for all other users of cgroups.

So, at least in my work, we have Mesos, but on nearly every machine that Mesos runs, people also have systemd. Now, there's recently become a bit of a battle of ownership of things like cgroups on these machines. We can usually solve it by nesting under systemd cgroups, and thus so far we've avoided making too many systemd-specific concessions.

The reason this works (mostly), is because everything we touch has a sense of nesting, where we can apply policy at a place lower in the hierarchy, and yet systemd's monitoring and policy still stays in place.

Now, with this patch, we don't have that, but I think we can reasonably add some flag like "no override" when applying policies, or alternatively something like "no new privileges", to prevent children from applying policies that override top-level policy. I realize there is a speed concern as well, but I think for people who want nested policy, we're willing to make the tradeoff. The cost of traversing a few extra pointers still outweighs the overhead of network namespaces, iptables, etc.. for many of us.

What do you think Daniel?

Quoted 6 lines

Quoted 2 lines

My main point is that those processes *need* to be launched by the orchestrator, which is was refering as 'server processes'.

Yes, that's right. But as I said, this rule applies to many other kernel concepts, so I don't see any real issue.

Also, cgroups have become such a big part of how applications are managed that many of us have solved this problem.

Quoted 14 lines

Quoted 7 lines

Quoted 2 lines

That's a limitation that applies to many more control mechanisms in the kernel, and it's something that can easily be solved with fork+exec.

As long as you have control to launch the processes yes, but this will not work in other scenarios. Just like cgroup net_cls and friends are broken for filtering for things that you have no control to fork+exec.

Probably, but that's only solvable with rules that store the full cgroup path then, and do a string comparison (!) for each packet flying by.

Quoted 3 lines

Quoted 2 lines

That's just as transparent as SO_ATTACH_FILTER. What kind of introspection mechanism do you have in mind?

Quoted 9 lines

Quoted 2 lines

SO_ATTACH_FILTER is called from the process itself, so this is a local filtering policy that you apply to your own process.

Not necessarily. You can as well do it the inetd way, and pass the socket to a process that is launched on demand, but do SO_ATTACH_FILTER
+ SO_LOCK_FILTER in the middle. What happens with payload on the socket is not transparent to the launched binary at all. The proposed cgroup eBPF solution implements a very similar behavior in that regard.

It would be nice to be able to see whether or not a filter is attached to a cgroup, but given this is going through syscalls, at least introspection is possible as opposed to something like netlink.

Quoted 14 lines

Quoted 9 lines

Quoted 6 lines

It's about filtering outgoing network packets of applications, and providing them with L2 information for filtering purposes. I don't think that's a very specific use-case.

When the feature is not used at all, the added costs on the output path are close to zero, due to the use of static branches.

*You're proposing a socket filtering facility that hooks layer 2 output path*!

As I said, I'm open to discussing that. In order to make it work for L3, the LL_OFF issues need to be solved, as Daniel explained. Daniel, Alexei, any idea how much work that would be?

Quoted 8 lines

Quoted 3 lines

That is only a rough ~30 lines kernel patchset to support this in netfilter and only one extra input hook, with potential access to conntrack and better integration with other existing subsystems.

Care to share the patches for that? I'd really like to have a look.

And FWIW, I agree with Thomas - there is nothing wrong with having multiple options to use for such use-cases.

Right now, for containers, we have netfilter and network namespaces. There's a lot of performance overhead that comes with this. Not only that, but iptables doesn't really have a simple way of usage by automated infrastructure. We (firewalld, systemd, dockerd, mesos) end up fighting with one another for ownership over firewall rules.

Although, I have problems with this approach, I think that it's a good baseline where we can have top level owned by systemd, docker underneath that, and Mesos underneath that. We can add additional hooks for things like Checmate and Landlock, and with a little more work, we can do compositition, solving all of our problems.

Quoted 5 lines

Thanks,
Daniel

Quoted 4 lines · Daniel Mack

I have no idea what makes you think this is limited to systemd. As I said, I provided an example for userspace that works from the command line. The same limitation apply as for all other users of cgroups.

So, at least in my work, we have Mesos, but on nearly every machine that Mesos runs, people also have systemd. Now, there's recently become a bit of a battle of ownership of things like cgroups on these machines. We can usually solve it by nesting under systemd cgroups, and thus so far we've avoided making too many systemd-specific concessions.

The reason this works (mostly), is because everything we touch has a sense of nesting, where we can apply policy at a place lower in the hierarchy, and yet systemd's monitoring and policy still stays in place.

Now, with this patch, we don't have that, but I think we can reasonably add some flag like "no override" when applying policies, or alternatively something like "no new privileges", to prevent children from applying policies that override top-level policy.

I realize there is a speed concern as well, but I think for people who want nested policy, we're willing to make the tradeoff. The cost of traversing a few extra pointers still outweighs the overhead of network namespaces, iptables, etc.. for many of us.

What do you think Daniel?

Quoted 5 lines

Not necessarily. You can as well do it the inetd way, and pass the socket to a process that is launched on demand, but do SO_ATTACH_FILTER
+ SO_LOCK_FILTER in the middle. What happens with payload on the socket is not transparent to the launched binary at all. The proposed cgroup eBPF solution implements a very similar behavior in that regard.

It would be nice to be able to see whether or not a filter is attached to a cgroup, but given this is going through syscalls, at least introspection is possible as opposed to something like netlink.

Quoted 2 lines

And FWIW, I agree with Thomas - there is nothing wrong with having multiple options to use for such use-cases.

Right now, for containers, we have netfilter and network namespaces. There's a lot of performance overhead that comes with this.

Not only
that, but iptables doesn't really have a simple way of usage by automated infrastructure. We (firewalld, systemd, dockerd, mesos) end up fighting with one another for ownership over firewall rules.

Although, I have problems with this approach, I think that it's a good baseline where we can have top level owned by systemd, docker underneath that, and Mesos underneath that. We can add additional hooks for things like Checmate and Landlock, and with a little more work, we can do compositition, solving all of our problems.

Hi,

Quoted 16 lines · Sargun Dhillon

Quoted 4 lines · Daniel Mack

I have no idea what makes you think this is limited to systemd. As I said, I provided an example for userspace that works from the command line. The same limitation apply as for all other users of cgroups.

So, at least in my work, we have Mesos, but on nearly every machine that Mesos runs, people also have systemd. Now, there's recently become a bit of a battle of ownership of things like cgroups on these machines. We can usually solve it by nesting under systemd cgroups, and thus so far we've avoided making too many systemd-specific concessions.

The reason this works (mostly), is because everything we touch has a sense of nesting, where we can apply policy at a place lower in the hierarchy, and yet systemd's monitoring and policy still stays in place.

Quoted 4 lines

Now, with this patch, we don't have that, but I think we can reasonably add some flag like "no override" when applying policies, or alternatively something like "no new privileges", to prevent children from applying policies that override top-level policy.

Yes, but the API is already guarded by CAP_NET_ADMIN. Take that capability away from your children, and they can't tamper with the policy. Does that work for you?

Quoted 4 lines

I realize there is a speed concern as well, but I think for people who want nested policy, we're willing to make the tradeoff. The cost of traversing a few extra pointers still outweighs the overhead of network namespaces, iptables, etc.. for many of us.

Not sure. Have you tried it?

Quoted 1 line

What do you think Daniel?

I think we should look at an implementation once we really need it, and then revisit the performance impact. In any case, this can be changed under the hood, without touching the userspace API (except for adding flags if we need them).

Quoted 9 lines

Quoted 5 lines

Not necessarily. You can as well do it the inetd way, and pass the socket to a process that is launched on demand, but do SO_ATTACH_FILTER
+ SO_LOCK_FILTER in the middle. What happens with payload on the socket is not transparent to the launched binary at all. The proposed cgroup eBPF solution implements a very similar behavior in that regard.

It would be nice to be able to see whether or not a filter is attached to a cgroup, but given this is going through syscalls, at least introspection is possible as opposed to something like netlink.

Sure, there are many ways. I implemented the bpf cgroup logic using an own cgroup controller once, which made it possible to read out the status. But as we agreed on attaching programs through the bpf(2) system call, I moved back to the implementation that directly stores the pointers in the cgroup.

First enabling the controller through the fs-backed cgroup interface, then come back through the bpf(2) syscall and then go back to the fs interface to read out status values is a bit weird.

Quoted 5 lines

Quoted 2 lines

And FWIW, I agree with Thomas - there is nothing wrong with having multiple options to use for such use-cases.

Right now, for containers, we have netfilter and network namespaces. There's a lot of performance overhead that comes with this.

Out of curiosity: Could you express that in numbers? And how exactly are you testing?

Quoted 4 lines

Not only
that, but iptables doesn't really have a simple way of usage by automated infrastructure. We (firewalld, systemd, dockerd, mesos) end up fighting with one another for ownership over firewall rules.

Yes, that's a common problem.

Quoted 6 lines

Although, I have problems with this approach, I think that it's a good baseline where we can have top level owned by systemd, docker underneath that, and Mesos underneath that. We can add additional hooks for things like Checmate and Landlock, and with a little more work, we can do compositition, solving all of our problems.

It is supposed to be just a baseline, yes.

Thanks for your feedback,
Daniel

Quoted 10 lines · Daniel Mack

Quoted 4 lines · Sargun Dhillon

Now, with this patch, we don't have that, but I think we can reasonably add some flag like "no override" when applying policies, or alternatively something like "no new privileges", to prevent children from applying policies that override top-level policy.

Yes, but the API is already guarded by CAP_NET_ADMIN. Take that capability away from your children, and they can't tamper with the policy. Does that work for you?

No. This can be addressed in a follow-on patch, but the use-case is that I have a container orchestrator (Docker, or Mesos), and systemd. The sysadmin controls systemd, and Docker is controlled by devs. Typically, the system owner wants some system level statistics, and filtering, and then we want to do per-container filtering.

We really want to be able to do nesting with userspace tools that are oblivious, and we want to delegate a level of the cgroup hierarchy to the tool that created it. I do not see Docker integrating with systemd any time soon, and that's really the only other alternative.

Quoted 10 lines

Quoted 3 lines

It would be nice to be able to see whether or not a filter is attached to a cgroup, but given this is going through syscalls, at least introspection is possible as opposed to something like netlink.

Sure, there are many ways. I implemented the bpf cgroup logic using an own cgroup controller once, which made it possible to read out the status. But as we agreed on attaching programs through the bpf(2) system call, I moved back to the implementation that directly stores the pointers in the cgroup.

Quoted 4 lines

First enabling the controller through the fs-backed cgroup interface, then come back through the bpf(2) syscall and then go back to the fs interface to read out status values is a bit weird.

Hrm, that makes sense. with the BPF syscall, would there be a way to get file descriptor of the currently attached BPF program?

* The sample program learned to support both ingress and egress, and can now optionally make the eBPF program drop packets by making it return 0.

❦ 12 septembre 2016 18:12 CEST, Daniel Mack <[email protected]> :

Quoted 3 lines

* The sample program learned to support both ingress and egress, and can now optionally make the eBPF program drop packets by making it return 0.

Ability to lock the eBPF program to avoid modification from a later program or in a subcgroup would be pretty interesting from a security perspective.