Accept passed in socket 'fd' open from outside for unix socket

Quoted 7 lines · On 05/31/2016 10:30 AM, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Any reason this wasn't sent to the list?

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

This sounds like a bug in libvirt, or selinux, or a mistaken configuration of the guest. It is entirely possible for QEMU to create a unix socket - not least because that is exactly what QEMU uses for its QMP monitor backend. Looking at your example command line, I think the issue is simply that you should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to create sockets already.

Alternatively you could enhance the SELinux policy to grant svirt_t the permission to create sockets under /var/run/openvswitch too.

Quoted 13 lines

I finished a RFC patch for Unix socket after a glance of the code, and not sure if this is right or there maybe other side-effects, please point me out.

I tested it for both server and client mode 'PF_UNIX' socket with a VM running vhost-user.

Old command line:
-chardev socket,id=char0,path=/var/run/openvswitch/vhost-user1,server

New command line:
-chardev socket,id=char0,path=/var/run/openvswitch/vhost-user1,server,sockfd=$n

Quoted 2 lines

because unix socket is bundled with a path, so it should be kept even with the 'fd' is indicated, this looks odd, any comments?

Yes, this syntax doesn't really make sense. The passed in sockfd may not even be a UNIX socket - it could be a TCP socket. As such, the 'sockfd' option should be mutually exclusive with the 'path' and 'host' options. ie you can only supply one out of 'sockfd', 'path', or 'host'.

FWIW, I think the ability to pass a pre-opened socket FD with the -chardev socket backend is a useful enhancement, even though I don't think you need this in order to fix the problem you have.

Regards,
Daniel

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 11 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend. Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

Quoted 4 lines

Alternatively you could enhance the SELinux policy to grant svirt_t the permission to create sockets under /var/run/openvswitch too.

Quoted 15 lines

Quoted 15 lines

I finished a RFC patch for Unix socket after a glance of the code, and not sure if this is right or there maybe other side-effects, please point me out.

I tested it for both server and client mode 'PF_UNIX' socket with a VM running vhost-user.

Old command line:
-chardev socket,id=char0,path=/var/run/openvswitch/vhost-user1,server

New command line:
-chardev socket,id=char0,path=/var/run/openvswitch/vhost-user1,server,sockfd=$n

Quoted 8 lines

Quoted 3 lines

because unix socket is bundled with a path, so it should be kept even with the
'fd' is indicated, this looks odd, any comments?

Yes, this syntax doesn't really make sense. The passed in sockfd may not even be a UNIX socket - it could be a TCP socket. As such, the 'sockfd' option should be mutually exclusive with the 'path' and 'host' options. ie you can only supply one out of 'sockfd', 'path', or 'host'.

Currently i just add it for unix socket, and the connect/listen syscall must have a path name, an inet socket doesn't need this param at all, should it be supported also?

Quoted 5 lines

FWIW, I think the ability to pass a pre-opened socket FD with the -chardev socket backend is a useful enhancement, even though I don't think you need this in order to fix the problem you have.

Quoted 21 lines · Wei Xu

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 11 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend. Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

Quoted 6 lines

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

    <interface type='vhostuser'>
      <mac address='52:54:00:ee:96:6c'/>
      <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
      <model type='virtio'/>
    </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

Quoted 2 lines

Quoted 2 lines

Alternatively you could enhance the SELinux policy to grant svirt_t the permission to create sockets under /var/run/openvswitch too.

Nah, the point of using libvirt is that you don't have to configure anything (or just bare minimum) and libvirt makes sure your domains have all the permissions needed - that's why we relabel domain's disks when starting it up.

Quoted 45 lines · Michal Privoznik

Quoted 21 lines · Wei Xu

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 11 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend. Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

Quoted 6 lines

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

    <interface type='vhostuser'>
      <mac address='52:54:00:ee:96:6c'/>
      <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
      <model type='virtio'/>
    </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Quoted 46 lines · Daniel P. Berrange

Quoted 45 lines · Michal Privoznik

Quoted 21 lines · Wei Xu

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 11 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend. Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

Quoted 6 lines

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

     <interface type='vhostuser'>
       <mac address='52:54:00:ee:96:6c'/>
       <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
       <model type='virtio'/>
     </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 10 lines

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

Quoted 6 lines

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

Quoted 5 lines

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Michal

Quoted 47 lines · Michal Privoznik

Quoted 46 lines · Daniel P. Berrange

Quoted 45 lines · Michal Privoznik

Quoted 21 lines · Wei Xu

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 11 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend. Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

Quoted 6 lines

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

     <interface type='vhostuser'>
       <mac address='52:54:00:ee:96:6c'/>
       <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
       <model type='virtio'/>
     </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 16 lines

Quoted 10 lines

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

Quoted 6 lines

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

AFAIK, Vhost user is an interface for third party implementations, and ovs/dpdk is one of the most popular choices, if it limits the socket location of a fixed and unprivileged directory to qemu, actually this should be the default and only one option, this maybe also a security consideration, so we'll have no other choice but ask sys admin to manipulate the permission, looks accepting a safe passed in 'fd' from libvirt is more rigorous and convenient, i'm not sure if this is a typical or a corner scenario.

From: "Michal Privoznik" <[email protected]>
To: "Daniel P. Berrange" <[email protected]>
Cc: [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 2, 2016 2:38:53 PM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 64 lines · Daniel P. Berrange

Quoted 46 lines · Michal Privoznik

Quoted 22 lines · Wei Xu

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 12 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend.
Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

Quoted 6 lines

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

    <interface type='vhostuser'>
      <mac address='52:54:00:ee:96:6c'/>
      <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
      <model type='virtio'/>
    </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

Quoted 4 lines

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Michal

From: "Aaron Conole" <[email protected]>
To: "Flavio Leitner" <[email protected]>
Cc: "Amnon Ilan" <[email protected]>, "Michal Privoznik" <[email protected]>, "Daniel P. Berrange" <[email protected]>, [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 9, 2016 12:48:57 AM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 110 lines · Flavio Leitner

Adding Aaron who is fixing exactly that on the OVS side.

Aaron, please see the last question in the bottom of this email.

Quoted 105 lines · Amnon Ilan

----- Original Message -----

Quoted 95 lines

From: "Michal Privoznik" <[email protected]>
To: "Daniel P. Berrange" <[email protected]>
Cc: [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 2, 2016 2:38:53 PM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 78 lines · Daniel P. Berrange

Quoted 54 lines · Michal Privoznik

Quoted 34 lines · Wei Xu

Quoted 11 lines · Daniel P. Berrange

Quoted 9 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to
create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely
passing it to qemu.

Quoted 14 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend.
Looking at your example command line, I think the issue is simply that
you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

ah.. adjusting permission or file location can solve this problem, i'm
guessing maybe this is a more security concern, the socket is used as a
network interface for a vm, similar as the qcow image file, thus should
prevent it to be arbitrarily accessed.

Quoted 1 line

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

    <interface type='vhostuser'>
      <mac address='52:54:00:ee:96:6c'/>
      <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
      <model type='virtio'/>
    </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 13 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this
and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is
then able to create the socket. One possible fix would be to allow qemu
create sockets just anywhere in the system. This, however, brings huge
security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory
for the sockets. We certainly do not want to allow QEMU to create sockets
anywhere.

Quoted 6 lines

I don't think we want to grant QEMU svirtt permission to create sockets
in the /var/run/openvswitch directory either really.IMHO, users of vhost
user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Quoted 4 lines

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Does OVS has some limit for it's sockets to be only in /var/run/openvswitch ?

As of a recent commit, it can only be in /var/run/openvswitch or a subdirectory therein (found in the openvswitch database).

Quoted 2 lines

Quoted 2 lines

Flavio, do you know?
If not, we are good as it is today with /var/lib/libvirt/qemu, right?

Probably not. There are other issues as well. From a DAC perspective (so forgetting selinux at the moment), qemu and ovs run as different users. This will mean that when ovs creates the unix domain socket file, qemu will not be allowed to actually open it properly. I have a commit I'm trying to get upstream for that particular issue (see bz:1281911 and upstream list discussion:
http://openvswitch.org/pipermail/dev/2016-May/071453.html
and
http://openvswitch.org/pipermail/dev/2016-June/071978.html)

Ansis is (I think) attacking this from the selinux/MAC side. It would be great to hear from users, libvirt folks, or anyone else in that thread to help push it to a conclusion one way or another (even if the approach that I've proposed is crap and wrong - then say it there :).

Hope this helps,
-Aaron

Quoted 93 lines · Flavio Leitner

Adding Aaron who is fixing exactly that on the OVS side.

Aaron, please see the last question in the bottom of this email.

Quoted 88 lines · Amnon Ilan

----- Original Message -----

Quoted 79 lines

From: "Michal Privoznik" <[email protected]>
To: "Daniel P. Berrange" <[email protected]>
Cc: [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 2, 2016 2:38:53 PM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 64 lines · Daniel P. Berrange

Quoted 46 lines · Michal Privoznik

Quoted 22 lines · Wei Xu

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 12 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend.
Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

Quoted 6 lines

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

    <interface type='vhostuser'>
      <mac address='52:54:00:ee:96:6c'/>
      <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
      <model type='virtio'/>
    </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

Quoted 4 lines

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Quoted 4 lines

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Does OVS has some limit for it's sockets to be only in /var/run/openvswitch ?

As of a recent commit, it can only be in /var/run/openvswitch or a subdirectory therein (found in the openvswitch database).

Quoted 2 lines

Quoted 2 lines

Flavio, do you know?
If not, we are good as it is today with /var/lib/libvirt/qemu, right?

Probably not. There are other issues as well. From a DAC perspective (so forgetting selinux at the moment), qemu and ovs run as different users. This will mean that when ovs creates the unix domain socket file, qemu will not be allowed to actually open it properly. I have a commit I'm trying to get upstream for that particular issue (see bz:1281911 and upstream list discussion:
http://openvswitch.org/pipermail/dev/2016-May/071453.html
and
http://openvswitch.org/pipermail/dev/2016-June/071978.html)

Ansis is (I think) attacking this from the selinux/MAC side. It would be great to hear from users, libvirt folks, or anyone else in that thread to help push it to a conclusion one way or another (even if the approach that I've proposed is crap and wrong - then say it there :).

Quoted 94 lines · Aaron Conole

Quoted 92 lines · Flavio Leitner

Adding Aaron who is fixing exactly that on the OVS side.

Aaron, please see the last question in the bottom of this email.

Quoted 87 lines · Amnon Ilan

----- Original Message -----

Quoted 83 lines

From: "Michal Privoznik" <[email protected]>
To: "Daniel P. Berrange" <[email protected]>
Cc: [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 2, 2016 2:38:53 PM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 64 lines · Daniel P. Berrange

Quoted 46 lines · Michal Privoznik

Quoted 22 lines · Wei Xu

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 12 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend.
Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

Quoted 6 lines

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

    <interface type='vhostuser'>
      <mac address='52:54:00:ee:96:6c'/>
      <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
      <model type='virtio'/>
    </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

Quoted 4 lines

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Quoted 4 lines

Quoted 1 line

Quoted 1 line

Does OVS has some limit for it's sockets to be only in /var/run/openvswitch ?

As of a recent commit, it can only be in /var/run/openvswitch or a subdirectory therein (found in the openvswitch database).

Quoted 92 lines · Flavio Leitner

Adding Aaron who is fixing exactly that on the OVS side.

Aaron, please see the last question in the bottom of this email.

Quoted 87 lines · Amnon Ilan

----- Original Message -----

Quoted 83 lines

From: "Michal Privoznik" <[email protected]>
To: "Daniel P. Berrange" <[email protected]>
Cc: [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 2, 2016 2:38:53 PM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 64 lines · Daniel P. Berrange

Quoted 46 lines · Michal Privoznik

Quoted 22 lines · Wei Xu

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 12 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend.
Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

Quoted 6 lines

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

     <interface type='vhostuser'>
       <mac address='52:54:00:ee:96:6c'/>
       <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
       <model type='virtio'/>
     </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

Quoted 4 lines

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Quoted 1 line

Quoted 1 line

Does OVS has some limit for it's sockets to be only in /var/run/openvswitch ?

As of a recent commit, it can only be in /var/run/openvswitch or a subdirectory therein (found in the openvswitch database).

Quoted 3 lines

Quoted 3 lines

Flavio, do you know?
If not, we are good as it is today with /var/lib/libvirt/qemu, right?

Probably not. There are other issues as well. From a DAC perspective (so forgetting selinux at the moment), qemu and ovs run as different users. This will mean that when ovs creates the unix domain socket file, qemu will not be allowed to actually open it properly. I have a commit I'm trying to get upstream for that particular issue (see bz:1281911 and upstream list discussion:

   http://openvswitch.org/pipermail/dev/2016-May/071453.html
   and
   http://openvswitch.org/pipermail/dev/2016-June/071978.html)

Ansis is (I think) attacking this from the selinux/MAC side. It would be great to hear from users, libvirt folks, or anyone else in that thread to help push it to a conclusion one way or another (even if the approach that I've proposed is crap and wrong - then say it there :).

Hope this helps,
-Aaron

Quoted 94 lines · Aaron Conole

Quoted 92 lines · Flavio Leitner

Adding Aaron who is fixing exactly that on the OVS side.

Aaron, please see the last question in the bottom of this email.

Quoted 87 lines · Amnon Ilan

----- Original Message -----

Quoted 83 lines

From: "Michal Privoznik" <[email protected]>
To: "Daniel P. Berrange" <[email protected]>
Cc: [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 2, 2016 2:38:53 PM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 64 lines · Daniel P. Berrange

Quoted 46 lines · Michal Privoznik

Quoted 22 lines · Wei Xu

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 12 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend.
Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

Quoted 6 lines

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

     <interface type='vhostuser'>
       <mac address='52:54:00:ee:96:6c'/>
       <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
       <model type='virtio'/>
     </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

Quoted 4 lines

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Quoted 4 lines

Quoted 1 line

Quoted 1 line

Does OVS has some limit for it's sockets to be only in /var/run/openvswitch ?

As of a recent commit, it can only be in /var/run/openvswitch or a subdirectory therein (found in the openvswitch database).

Aaron, thanks for your reply.

Just a question about the usage of openvswitch, in this user case when launching a vhostuser/dpdk via libvirt, qemu works as server mode for socket under /var/run/openvswitch, but per my previous test, ovs/dpdk always works as server mode, which means ovs will creates the socket and listening for connection, thus qemu works as client mode, does ovs/dpdk support working in client mode? which means it's qemu's duty to create the socket? and ovs will connect to it on demanding?

Quoted 114 lines · Wei Xu

Quoted 106 lines · Aaron Conole

Quoted 104 lines · Flavio Leitner

Adding Aaron who is fixing exactly that on the OVS side.

Aaron, please see the last question in the bottom of this email.

Quoted 99 lines · Amnon Ilan

----- Original Message -----

Quoted 95 lines

From: "Michal Privoznik" <[email protected]>
To: "Daniel P. Berrange" <[email protected]>
Cc: [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 2, 2016 2:38:53 PM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 75 lines · Daniel P. Berrange

Quoted 55 lines · Michal Privoznik

Quoted 36 lines · Wei Xu

Quoted 13 lines · Daniel P. Berrange

Quoted 11 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue,
selinux forbids qemu to
create a unix socket for a chardev when managing
VMs with libvirt,
because qemu don't have sufficient permissions in
this case, and
proposal from libvirt team is opening the 'fd' in
libvirt and merely
passing it to qemu.

Quoted 14 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend.
Looking at your example command line, I think the issue is simply that
you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

ah.. adjusting permission or file location can solve this problem, i'm
guessing maybe this is a more security concern, the socket is used as a
network interface for a vm, similar as the qcow image file, thus should
prevent it to be arbitrarily accessed.

Quoted 1 line

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

     <interface type='vhostuser'>
       <mac address='52:54:00:ee:96:6c'/>
       <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
       <model type='virtio'/>
     </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu
as the directory
for the sockets. We certainly do not want to allow QEMU to create sockets
anywhere.

Quoted 5 lines

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost
user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Quoted 5 lines

Quoted 2 lines

Quoted 2 lines

Does OVS has some limit for it's sockets to be only in /var/run/openvswitch ?

As of a recent commit, it can only be in /var/run/openvswitch or a subdirectory therein (found in the openvswitch database).

Aaron, thanks for your reply.

Quoted 7 lines

Just a question about the usage of openvswitch, in this user case when launching a vhostuser/dpdk via libvirt, qemu works as server mode for socket under /var/run/openvswitch, but per my previous test, ovs/dpdk always works as server mode, which means ovs will creates the socket and listening for connection, thus qemu works as client mode, does ovs/dpdk support working in client mode? which means it's qemu's duty to create the socket? and ovs will connect to it on demanding?

Oh, I was assuming that QEMU would be working in server mode - no wonder we have somewhat different views :-)

If OVS is running the UNIX socket server, and QEMU is purely the client, then that means the solution would be slightly different. In particular libvirt would *not* do any SELinux relabelling. Instead you would have to get an addition to the SELinux policy, to allow svirt_t type to connect to the SELinux type associated with the openvswitch socket.

For file permissions, if the OVS socket is owned by a particular UNIX group, you could potentially add the 'qemu' user account to that group to grant access.

Regards,
Daniel

Quoted 126 lines · Daniel P. Berrange

Quoted 114 lines · Wei Xu

Quoted 106 lines · Aaron Conole

Quoted 104 lines · Flavio Leitner

Adding Aaron who is fixing exactly that on the OVS side.

Aaron, please see the last question in the bottom of this email.

Quoted 99 lines · Amnon Ilan

----- Original Message -----

Quoted 95 lines

From: "Michal Privoznik" <[email protected]>
To: "Daniel P. Berrange" <[email protected]>
Cc: [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 2, 2016 2:38:53 PM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 75 lines · Daniel P. Berrange

Quoted 55 lines · Michal Privoznik

Quoted 36 lines · Wei Xu

Quoted 13 lines · Daniel P. Berrange

Quoted 11 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue,
selinux forbids qemu to
create a unix socket for a chardev when managing
VMs with libvirt,
because qemu don't have sufficient permissions in
this case, and
proposal from libvirt team is opening the 'fd' in
libvirt and merely
passing it to qemu.

Quoted 14 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend.
Looking at your example command line, I think the issue is simply that
you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

ah.. adjusting permission or file location can solve this problem, i'm
guessing maybe this is a more security concern, the socket is used as a
network interface for a vm, similar as the qcow image file, thus should
prevent it to be arbitrarily accessed.

Quoted 1 line

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

      <interface type='vhostuser'>
        <mac address='52:54:00:ee:96:6c'/>
        <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
        <model type='virtio'/>
      </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu
as the directory
for the sockets. We certainly do not want to allow QEMU to create sockets
anywhere.

Quoted 5 lines

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost
user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Quoted 5 lines

Quoted 2 lines

Quoted 2 lines

Does OVS has some limit for it's sockets to be only in /var/run/openvswitch ?

As of a recent commit, it can only be in /var/run/openvswitch or a subdirectory therein (found in the openvswitch database).

Aaron, thanks for your reply.

Quoted 7 lines

Just a question about the usage of openvswitch, in this user case when launching a vhostuser/dpdk via libvirt, qemu works as server mode for socket under /var/run/openvswitch, but per my previous test, ovs/dpdk always works as server mode, which means ovs will creates the socket and listening for connection, thus qemu works as client mode, does ovs/dpdk support working in client mode? which means it's qemu's duty to create the socket? and ovs will connect to it on demanding?

Oh, I was assuming that QEMU would be working in server mode - no wonder we have somewhat different views :-)

Quoted 5 lines

If OVS is running the UNIX socket server, and QEMU is purely the client, then that means the solution would be slightly different. In particular libvirt would *not* do any SELinux relabelling. Instead you would have to get an addition to the SELinux policy, to allow svirt_t type to connect to the SELinux type associated with the openvswitch socket.

I agree, this is a good MAC solution.

Quoted 4 lines

For file permissions, if the OVS socket is owned by a particular UNIX group, you could potentially add the 'qemu' user account to that group to grant access.

I actually would propose making a new vhost group, and adding the qemu user account and openvswitch user accounts to that group. That way reduces pollution into other aspects of each process' function.

-Aaron

Quoted 3 lines

Regards,
Daniel

From: "Wei Xu" <[email protected]>
To: "Aaron Conole" <[email protected]>, "Flavio Leitner" <[email protected]>
Cc: "Michal Privoznik" <[email protected]>, [email protected], [email protected], [email protected], "amit shah" <[email protected]>, "Amnon Ilan" <[email protected]>
Sent: Tuesday, June 14, 2016 11:03:43 AM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 107 lines · Aaron Conole

Quoted 105 lines · Flavio Leitner

Adding Aaron who is fixing exactly that on the OVS side.

Aaron, please see the last question in the bottom of this email.

Quoted 100 lines · Amnon Ilan

----- Original Message -----

Quoted 96 lines

From: "Michal Privoznik" <[email protected]>
To: "Daniel P. Berrange" <[email protected]>
Cc: [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 2, 2016 2:38:53 PM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 76 lines · Daniel P. Berrange

Quoted 53 lines · Michal Privoznik

Quoted 33 lines · Wei Xu

Quoted 10 lines · Daniel P. Berrange

Quoted 8 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely
passing it to qemu.

Quoted 14 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend.
Looking at your example command line, I think the issue is simply that
you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

ah.. adjusting permission or file location can solve this problem, i'm
guessing maybe this is a more security concern, the socket is used as a
network interface for a vm, similar as the qcow image file, thus should
prevent it to be arbitrarily accessed.

Quoted 1 line

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

     <interface type='vhostuser'>
       <mac address='52:54:00:ee:96:6c'/>
       <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
       <model type='virtio'/>
     </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 12 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this
and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is
then able to create the socket. One possible fix would be to allow qemu
create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory
for the sockets. We certainly do not want to allow QEMU to create sockets
anywhere.

Quoted 5 lines

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost
user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Quoted 5 lines

Quoted 2 lines

Quoted 2 lines

Does OVS has some limit for it's sockets to be only in /var/run/openvswitch ?

As of a recent commit, it can only be in /var/run/openvswitch or a subdirectory therein (found in the openvswitch database).

Aaron, thanks for your reply.

Just a question about the usage of openvswitch, in this user case when launching a vhostuser/dpdk via libvirt, qemu works as server mode for socket under /var/run/openvswitch, but per my previous test, ovs/dpdk always works as server mode, which means ovs will creates the socket and listening for connection, thus qemu works as client mode, does ovs/dpdk support working in client mode? which means it's qemu's duty to create the socket? and ovs will connect to it on demanding?

Quoted 23 lines

Quoted 4 lines

Quoted 4 lines

Flavio, do you know?
If not, we are good as it is today with /var/lib/libvirt/qemu, right?

Probably not. There are other issues as well. From a DAC perspective (so forgetting selinux at the moment), qemu and ovs run as different users. This will mean that when ovs creates the unix domain socket file, qemu will not be allowed to actually open it properly. I have a commit I'm trying to get upstream for that particular issue (see bz:1281911 and upstream list discussion:

   http://openvswitch.org/pipermail/dev/2016-May/071453.html
   and
   http://openvswitch.org/pipermail/dev/2016-June/071978.html)

Ansis is (I think) attacking this from the selinux/MAC side. It would be great to hear from users, libvirt folks, or anyone else in that thread to help push it to a conclusion one way or another (even if the approach that I've proposed is crap and wrong - then say it there :).

Hope this helps,
-Aaron

Quoted 94 lines · Aaron Conole

Quoted 92 lines · Flavio Leitner

Adding Aaron who is fixing exactly that on the OVS side.

Aaron, please see the last question in the bottom of this email.

Quoted 87 lines · Amnon Ilan

----- Original Message -----

Quoted 83 lines

From: "Michal Privoznik" <[email protected]>
To: "Daniel P. Berrange" <[email protected]>
Cc: [email protected], "amit shah" <[email protected]>, [email protected], "Wei Xu" <[email protected]>, [email protected]
Sent: Thursday, June 2, 2016 2:38:53 PM
Subject: Re: [Qemu-devel] [RFC Patch 0/3] Accept passed in socket 'fd' open from outside for unix socket

Quoted 64 lines · Daniel P. Berrange

Quoted 46 lines · Michal Privoznik

Quoted 22 lines · Wei Xu

Quoted 9 lines · Daniel P. Berrange

Quoted 7 lines · On Wed, Jun 01, 2016 at 12:30:44AM +0800, [email protected] wrote:

From: Wei Xu <[email protected]>

Recently I'm working on a fd passing issue, selinux forbids qemu to create a unix socket for a chardev when managing VMs with libvirt, because qemu don't have sufficient permissions in this case, and proposal from libvirt team is opening the 'fd' in libvirt and merely passing it to qemu.

Quoted 12 lines

This sounds like a bug in libvirt, or selinux, or a mistaken configuration
of the guest. It is entirely possible for QEMU to create a unix socket
- not
least because that is exactly what QEMU uses for its QMP monitor backend.
Looking at your example command line, I think the issue is simply that you
should be putting the sockets in a different location. ie at /var/lib/libvirt/qemu/$guest-vhost-user1.sock where QEMU has permission to
create sockets already.

Quoted 6 lines

ah.. adjusting permission or file location can solve this problem, i'm guessing maybe this is a more security concern, the socket is used as a network interface for a vm, similar as the qcow image file, thus should prevent it to be arbitrarily accessed.

Michael, do you have any comment on this?

I haven't seen the patches. But in libvirt we allow users to create a vhostuser interface and even specify where the socket should be placed:

     <interface type='vhostuser'>
       <mac address='52:54:00:ee:96:6c'/>
       <source type='unix' path='/tmp/vhost1.sock' mode='server'/>
       <model type='virtio'/>
     </interface>

The following cmd line is generated by libvirt then:

(quoted diff) +0 -3

-chardev socket,id=charnet1,path=/tmp/vhost1.sock,server \
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 \
-device

virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ee:96:6c,bus=pci.0,\

Quoted 9 lines

Now, if we accept only /var/run/openvwitch path in /interface/source/@path (or whatever path to OVS is), we don't need this and have users manually label the dir (unless already labeled). But since we accept just any path in there, we should make sure that qemu is then able to create the socket. One possible fix would be to allow qemu create sockets just anywhere in the system. This, however, brings huge security risks and it's not acceptable IMO. The other option would be that libvirt would create the socket, and pass its FD to qemu (since libvirt already is allowed to create sockets anywhere).

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

Quoted 4 lines

I don't think we want to grant QEMU svirtt permission to create sockets in the /var/run/openvswitch directory either really.IMHO, users of vhost user should really be using /var/lib/libvirt/qemu, as is used for all other UNIX sockets we create wrt other devices.

Okay. I can live with that; but in that case we should document it somewhere, that we guarantee only paths under /var/lib/libvirt/ to be accessible and for the rest we do our best but maybe require sys admin intervention (e.g. to label the whole tree for a non-standard location).

Quoted 4 lines

Quoted 1 line

Quoted 1 line

Does OVS has some limit for it's sockets to be only in /var/run/openvswitch ?

As of a recent commit, it can only be in /var/run/openvswitch or a subdirectory therein (found in the openvswitch database).

Aaron, thanks for your reply.

Just a question about the usage of openvswitch, in this user case when launching a vhostuser/dpdk via libvirt, qemu works as server mode for socket under /var/run/openvswitch, but per my previous test, ovs/dpdk always works as server mode, which means ovs will creates the socket and listening for connection, thus qemu works as client mode, does ovs/dpdk support working in client mode? which means it's qemu's duty to create the socket? and ovs will connect to it on demanding?

Quoted 22 lines

Quoted 3 lines

Quoted 3 lines

Flavio, do you know?
If not, we are good as it is today with /var/lib/libvirt/qemu, right?

Probably not. There are other issues as well. From a DAC perspective (so forgetting selinux at the moment), qemu and ovs run as different users. This will mean that when ovs creates the unix domain socket file, qemu will not be allowed to actually open it properly. I have a commit I'm trying to get upstream for that particular issue (see bz:1281911 and upstream list discussion:

   http://openvswitch.org/pipermail/dev/2016-May/071453.html
   and
   http://openvswitch.org/pipermail/dev/2016-June/071978.html)

Ansis is (I think) attacking this from the selinux/MAC side. It would be great to hear from users, libvirt folks, or anyone else in that thread to help push it to a conclusion one way or another (even if the approach that I've proposed is crap and wrong - then say it there :).

Hope this helps,
-Aaron

Quoted 2 lines · Michal Privoznik

Quoted 1 line · Wei Xu

On 2016年06月01日 00:44, Daniel P. Berrange wrote:

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

Dear Daniel,

Quoted 9 lines · Daniel P. Berrange

Quoted 2 lines · Michal Privoznik

Quoted 1 line · Wei Xu

On 2016年06月01日 00:44, Daniel P. Berrange wrote:

There are plenty of other places where we allow arbitrary paths in the XML, but which have restrictions imposed by the security drivers. Not least the <channel> devices which have the exact same scenario as this network device, and require use of /var/lib/libvirt/qemu as the directory for the sockets. We certainly do not want to allow QEMU to create sockets anywhere.

Umm, how exactly is an application supposed to use (i.e. open) the channel devices defined in XML?

Previously I deliberately left out the path in the XML to let libvirt choose the path and extracted it from the XML after defining the domain. This ensured qemu could access the path, plus it was the responsibility of libvirt to clean it up once the domain was undefined. Easy and simple.

Since 71408079 [qemu: Don't bother user with libvirt-internal paths], the path chosen by libvirt isn't included in the domain XML anymore. So now I need to choose the path inside the application. The only safe way to do that is by using something in an application-managed namespace (e.g. /var/lib/myapp/foo or /tmp/myapp/foo); I certainly wouldn't want to second guess what paths would be safe inside the libvirt namespace (/var/lib/libvirt/qemu). Except now I hear that anything outside /var/lib/libvirt/qemu is not guaranteed to work due to e.g. the SELinux policy configured by libvirt...

Quoted 9 lines · Sascha Silbe

Since 71408079 [qemu: Don't bother user with libvirt-internal paths], the path chosen by libvirt isn't included in the domain XML anymore. So now I need to choose the path inside the application. The only safe way to do that is by using something in an application-managed namespace (e.g. /var/lib/myapp/foo or /tmp/myapp/foo); I certainly wouldn't want to second guess what paths would be safe inside the libvirt namespace (/var/lib/libvirt/qemu). Except now I hear that anything outside /var/lib/libvirt/qemu is not guaranteed to work due to e.g. the SELinux policy configured by libvirt...

IIUC that change 71408079 ought to only apply to the persistent XML configuration on disk. When the guest is running the live XML ought to still report the path libvirt chose, so you should still be able to see it while running. If that isn't the case then please raise a bug, because we certainly expect apps to be able to discover the path libvirt picked for exactly the reason you describe

Quoted 26 lines · On 05/31/2016 10:30 AM, [email protected] wrote:

From: Wei Xu <[email protected]>

Save the 'fd' paramter as unix socket 'sockfd' member.

Signed-off-by Wei Xu <[email protected]>

---
  qemu-char.c | 7 +++++++
  1 file changed, 7 insertions(+)

qemu-char.c +2 -0

index ea9c02e..8d20494 100644
@@ -3664,6 +3664,7 @@ static void qemu_chr_parse_socket(QemuOpts *opts, ChardevBackend *backend,
      bool is_telnet      = qemu_opt_get_bool(opts, "telnet", false);
      bool do_nodelay     = !qemu_opt_get_bool(opts, "delay", true);
      int64_t reconnect   = qemu_opt_get_number(opts, "reconnect", 0);
+    const int32_t fd = (int32_t)qemu_opt_get_number(opts, "sockfd", 0);
      const char *path = qemu_opt_get(opts, "path");
      const char *host = qemu_opt_get(opts, "host");
      const char *port = qemu_opt_get(opts, "port");

@@ -3708,6 +3709,12 @@ static void qemu_chr_parse_socket(QemuOpts *opts, ChardevBackend *backend,
          addr->type = SOCKET_ADDRESS_KIND_UNIX;
          q_unix = addr->u.q_unix.data = g_new0(UnixSocketAddress, 1);
          q_unix->path = g_strdup(path);
+

qemu-char.c static void qemu_chr_parse_socket(QemuOpts *opts, ChardevBackend *backend, · line 3716

Quoted 4 lines

qemu-char.c +4 -0

+        if (fd) {
+            q_unix->sockfd = fd;
+        } else {
+            q_unix->sockfd = 0;

0 is a valid fd number; this risks accidentally closing stdin later on.
Please use -1 for unset, if you must store an fd number. But given my
comments on patch 1, I'm not sure that you need this addition.