vlan packet handling in ixgbevf driver when in promiscuous mode
Hi,
I'm doing some work with ixgbevf in the 3.10 kernel, but the current linux kernel git version seems to have similar behaviour.
Basically I'm interested in doing packet sniffing in a VM guest using PCI passthrough of a VF of an 82599.
What I'm seeing is that if I register for a vlan in the guest (creating eth1.100, for example) and then run tcpdump on the underlying device (eth1, for example) then I see the packet with the vlan header in it.
If I remove that vlan, then I see the packet with no vlan header.
I think this is coming from ixgbevf_receive_skb() in 3.10 (or ixgbevf_process_skb_fields() in current upstream git) where it has code that looks like this:
if (test_bit(vid & VLAN_VID_MASK, active_vlans)) __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), vid);
While this makes sense for normal operation (where traffic from other vlans isn't coming in anyways) I don't think it makes sense when promiscuous mode is enabled.
In promiscuous mode I want to be able to see exactly what packets are coming on the (virtual) wire, so I should be able to see the vlan header even if I haven't actually registered for that vlan.
Basically I think that code above should add an "or device is in promiscuous mode" clause.
Thoughts?
Chris
Rose
82599 devices do not support promiscuous mode for the virtual functions - you should use the physical function for true promiscuous capture.
----------
Greg Rose
FreeBSD/NFV PAE
Network Division
Intel Corporation
Desk - 503-712-5048
Any man who afflicts the human race with ideas must be prepared to see them misunderstood. ?
- H. L. Mencken
Quoted 48 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Chris Friesen
Sent: Friday, May 29, 2015 2:00 PM
To: intel-wired-lan at lists.osuosl.org
Subject: [Intel-wired-lan] vlan packet handling in ixgbevf driver when in promiscuous modeHi,
I'm doing some work with ixgbevf in the 3.10 kernel, but the current linux kernel git version seems to have similar behaviour.
Basically I'm interested in doing packet sniffing in a VM guest using PCI passthrough of a VF of an 82599.
What I'm seeing is that if I register for a vlan in the guest (creating eth1.100, for example) and then run tcpdump on the underlying device (eth1, for
example) then I see the packet with the vlan header in it.If I remove that vlan, then I see the packet with no vlan header.
I think this is coming from
ixgbevf_receive_skb()in 3.10 (orixgbevf_process_skb_fields()in current upstream git) where it has code that looks like this:if (test_bit(vid & VLAN_VID_MASK, active_vlans)) __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), vid);While this makes sense for normal operation (where traffic from other vlans isn't coming in anyways) I don't think it makes sense when promiscuous mode is enabled.
In promiscuous mode I want to be able to see exactly what packets are coming on the (virtual) wire, so I should be able to see the vlan header even if I haven't actually registered for that vlan.
Basically I think that code above should add an "or device is in promiscuous mode" clause.
Thoughts?
Chris
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Could you elaborate?
The use-case here is for NFV/SDN type stuff, where we want to be able to run a router or something similar in the guest--so it needs to be able to receive packets from all VLANs.
From what I see if you enable promiscuous mode on the PF then the VFs get
packets from all VLANs.
With the stock ixgbevf driver in the guest this doesn't work well because any packets from VLANs that the guest hasn't registered for do not get the vlan tags added back in to the packets (which makes it useless for routing). With a minor change to the ixgbevf driver to put the vlan tags back in for all packets it seems to work okay. Maybe this change would be a reasonable thing to do in the stock ixgbevf driver when enabling promiscuous mode in the guest?
Also, it seems to work fine with the DPDK drivers in the guest when the host is in promiscuous mode.
Chris
Quoted 61 lines · Gregory V
82599 devices do not support promiscuous mode for the virtual functions - you should use the physical function for true promiscuous capture.
----------
Greg Rose
FreeBSD/NFV PAE
Network Division
Intel Corporation
Desk - 503-712-5048Any man who afflicts the human race with ideas must be prepared to see them misunderstood.
- H. L. Mencken
Quoted 48 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Chris Friesen
Sent: Friday, May 29, 2015 2:00 PM
To: intel-wired-lan at lists.osuosl.org
Subject: [Intel-wired-lan] vlan packet handling in ixgbevf driver when in promiscuous modeHi,
I'm doing some work with ixgbevf in the 3.10 kernel, but the current linux kernel git version seems to have similar behaviour.
Basically I'm interested in doing packet sniffing in a VM guest using PCI passthrough of a VF of an 82599.
What I'm seeing is that if I register for a vlan in the guest (creating eth1.100, for example) and then run tcpdump on the underlying device (eth1, for
example) then I see the packet with the vlan header in it.If I remove that vlan, then I see the packet with no vlan header.
I think this is coming from
ixgbevf_receive_skb()in 3.10 (orixgbevf_process_skb_fields()in current upstream git) where it has code that looks like this:if (test_bit(vid & VLAN_VID_MASK, active_vlans)) __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), vid);While this makes sense for normal operation (where traffic from other vlans isn't coming in anyways) I don't think it makes sense when promiscuous mode is enabled.
In promiscuous mode I want to be able to see exactly what packets are coming on the (virtual) wire, so I should be able to see the vlan header even if I haven't actually registered for that vlan.
Basically I think that code above should add an "or device is in promiscuous mode" clause.
Thoughts?
Chris
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Quoted 10 lines
Subject: Re: [Intel-wired-lan] vlan packet handling in ixgbevf driver when in promiscuous mode
Could you elaborate?
The use-case here is for NFV/SDN type stuff, where we want to be able to run a router or something similar in the guest--so it needs to be able to receive packets from all VLANs.
From what I see if you enable promiscuous mode on the PF then the VFs get
packets from all VLANs.
I'd like to have and am working to add a functionality to disable VLAN filter on PF to pass all VLANs to VFs in stock kernel ixgbe driver.
How did you do that?
Quoted 7 lines
With the stock ixgbevf driver in the guest this doesn't work well because any packets from VLANs that the guest hasn't registered for do not get the vlan tags added back in to the packets (which makes it useless for routing). With a minor change to the ixgbevf driver to put the vlan tags back in for all packets it seems to work okay. Maybe this change would be a reasonable thing to do in the stock ixgbevf driver when enabling promiscuous mode in the guest?
I just have confirmed that behavior during my work, and thought something to do to fix this issue.
Currently ixgbevf driver relies on VLAN filtering, that leads to this issue.
Dropping unregistered VLAN packet is expected behavior, I think. I'm not sure whether we should implement to capture unknown VLAN packet in VF.
What do you think?
thanks,
Hiroshi
Quoted 74 lines
Also, it seems to work fine with the DPDK drivers in the guest when the host is in promiscuous mode.
Chris
Quoted 1 line · Gregory V
82599 devices do not support promiscuous mode for the virtual functions - you should use the physical function for true
promiscuous capture.
Quoted 8 lines
----------
Greg Rose
FreeBSD/NFV PAE
Network Division
Intel Corporation
Desk - 503-712-5048Quoted 52 lines
Any man who afflicts the human race with ideas must be prepared to see them misunderstood.
- H. L. Mencken
Quoted 48 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Chris Friesen
Sent: Friday, May 29, 2015 2:00 PM
To: intel-wired-lan at lists.osuosl.org
Subject: [Intel-wired-lan] vlan packet handling in ixgbevf driver when in promiscuous modeHi,
I'm doing some work with ixgbevf in the 3.10 kernel, but the current linux kernel git version seems to have similar behaviour.
Basically I'm interested in doing packet sniffing in a VM guest using PCI passthrough of a VF of an 82599.
What I'm seeing is that if I register for a vlan in the guest (creating eth1.100, for example) and then run tcpdump on the underlying device (eth1, for
example) then I see the packet with the vlan header in it.If I remove that vlan, then I see the packet with no vlan header.
I think this is coming from
ixgbevf_receive_skb()in 3.10 (orixgbevf_process_skb_fields()in current upstream git) where it has code that looks like this:if (test_bit(vid & VLAN_VID_MASK, active_vlans)) __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), vid);While this makes sense for normal operation (where traffic from other vlans isn't coming in anyways) I don't think it makes sense when promiscuous mode is enabled.
In promiscuous mode I want to be able to see exactly what packets are coming on the (virtual) wire, so I should be able to see the vlan header even if I haven't actually registered for that vlan.
Basically I think that code above should add an "or device is in promiscuous mode" clause.
Thoughts?
Chris
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Quoted 25 lines · Hiroshi Shimamoto
Quoted 10 lines
Subject: Re: [Intel-wired-lan] vlan packet handling in ixgbevf driver when in promiscuous mode
Could you elaborate?
The use-case here is for NFV/SDN type stuff, where we want to be able to run a router or something similar in the guest--so it needs to be able to receive packets from all VLANs.
From what I see if you enable promiscuous mode on the PF then the VFs get
packets from all VLANs.I'd like to have and am working to add a functionality to disable VLAN filter on PF to pass all VLANs to VFs in stock kernel ixgbe driver.
How did you do that?Quoted 6 lines
With the stock ixgbevf driver in the guest this doesn't work well because any packets from VLANs that the guest hasn't registered for do not get the vlan tags added back in to the packets (which makes it useless for routing). With a minor change to the ixgbevf driver to put the vlan tags back in for all packets it seems to work okay. Maybe this change would be a reasonable thing to do in the stock ixgbevf driver when enabling promiscuous mode in the guest?
I just have confirmed that behavior during my work, and thought something to do to fix this issue.
Currently ixgbevf driver relies on VLAN filtering, that leads to this issue.
Quoted 7 lines
Dropping unregistered VLAN packet is expected behavior, I think. I'm not sure whether we should implement to capture unknown VLAN packet in VF.
What do you think?
thanks,
Hiroshi
Hiroshi,
What I would recommend doing is looking at something like fm10k_process_skb_fields and possibly working on updating ixgbevf to work similarly. Basically you need to be able to determine when the VF has a port VLAN enabled, and when it does it needs to know what the port VLAN ID is so that you can correctly strip that tag on incoming frames. This is why I said your change would break legacy drivers by enabling VLAN promiscuous mode.
What you would probably need to do is setup a new API version and find a way of passing a notification that a port VLAN has been assigned. Once you have that then you could look at rewriting the VLAN ID check so that you could pass all VLAN tagged frames that are not the port VLAN up to the stack as VLAN tagged.
- Alex
Rose
Quoted 24 lines
-----Original Message-----
From: Chris Friesen [mailto:chris.friesen at windriver.com]
Sent: Monday, June 01, 2015 5:02 PM
To: Rose, Gregory V; intel-wired-lan at lists.osuosl.org
Subject: Re: [Intel-wired-lan] vlan packet handling in ixgbevf driver when in promiscuous modeCould you elaborate?
The use-case here is for NFV/SDN type stuff, where we want to be able to run a router or something similar in the guest--so it needs to be able to receive packets from all VLANs.
From what I see if you enable promiscuous mode on the PF then the VFs get
packets from all VLANs.With the stock ixgbevf driver in the guest this doesn't work well because any packets from VLANs that the guest hasn't registered for do not get the vlan tags added back in to the packets (which makes it useless for routing). With a minor change to the ixgbevf driver to put the vlan tags back in for all packets it seems to work okay. Maybe this change would be a reasonable thing to do in the stock ixgbevf driver when enabling promiscuous mode in the guest?
Quoted 2 lines
Also, it seems to work fine with the DPDK drivers in the guest when the host is in promiscuous mode.
If you have a working configuration then that's fine. I was merely pointing out that promiscuous mode, as the semantics are commonly applied to word in a networking context, is not supported for virtual functions on the 82599 controller. There are configurations you can apply to suit your own purposes and we would like to support that for you if we can.
----------
Greg Rose
FreeBSD/NFV PAE
Network Division
Intel Corporation
Desk - 503-712-5048
Any man who afflicts the human race with ideas must be prepared to see them misunderstood. ?
- H. L. Mencken
Quoted 12 lines · Gregory V
Quoted 12 lines
-----Original Message----- From: Chris Friesen [mailto:chris.friesen at windriver.com] Sent: Monday, June 01, 2015 5:02 PM
To: Rose, Gregory V; intel-wired-lan at lists.osuosl.org Subject: Re:
[Intel-wired-lan] vlan packet handling in ixgbevf driver when in promiscuous modeCould you elaborate?
The use-case here is for NFV/SDN type stuff, where we want to be able to run a router or something similar in the guest--so it needs to be able to receive packets from all VLANs.
Quoted 8 lines
Quoted 2 lines
From what I see if you enable promiscuous mode on the PF then the VFs get packets from all VLANs.
If you have a working configuration then that's fine. I was merely pointing out that promiscuous mode, as the semantics are commonly applied to word in a networking context, is not supported for virtual functions on the 82599 controller. There are configurations you can apply to suit your own purposes and we would like to support that for you if we can.
Well, it's "working" but it means we have to patch the ixgbevf driver, which is a bit of a pain.
I'd like to see something like what Alexander Duyck suggested where the stock ixgbevf driver would only strip the VLAN tags off for packets where the VLAN matches the port VLAN ID and let all the other VLAN tags through. (If the guest interface is in promiscuous mode.)
Arguably if the guest interface is not in promiscuous mode then I think we should be dropping packets from all VLANs other than what the guest has explicitly registered for (or that the PF has assigned to the VF. It makes no sense to me that we would let them all through but strip off the VLAN header.
Chris
Quoted 37 lines · Chris Friesen
Quoted 19 lines · Gregory V
Quoted 15 lines
-----Original Message----- From: Chris Friesen [mailto:chris.friesen at windriver.com] Sent: Monday, June 01, 2015 5:02 PM
To: Rose, Gregory V; intel-wired-lan at lists.osuosl.org Subject: Re:
[Intel-wired-lan] vlan packet handling in ixgbevf driver when in promiscuous modeCould you elaborate?
The use-case here is for NFV/SDN type stuff, where we want to be able to
run a router or something similar in the guest--so it needs to be able to
receive packets from all VLANs.Quoted 3 lines
From what I see if you enable promiscuous mode on the PF then the VFs get
packets from all VLANs.Quoted 8 lines
If you have a working configuration then that's fine. I was merely pointing
out that promiscuous mode, as the semantics are commonly applied to word in a
networking context, is not supported for virtual functions on the 82599 controller. There are configurations you can apply to suit your own purposes
and we would like to support that for you if we can.Well, it's "working" but it means we have to patch the ixgbevf driver, which is a bit of a pain.
I'd like to see something like what Alexander Duyck suggested where the stock ixgbevf driver would only strip the VLAN tags off for packets where the VLAN matches the port VLAN ID and let all the other VLAN tags through. (If the guest interface is in promiscuous mode.)
Quoted 7 lines
Arguably if the guest interface is not in promiscuous mode then I think we should be dropping packets from all VLANs other than what the guest has explicitly registered for (or that the PF has assigned to the VF. It makes no sense to me that we would let them all through but strip off the VLAN header.
Chris
Chris,
Based on what is in current net-next I don't see how you can enable VLAN promiscuous mode on the PF without having a modified driver of some sort. That is the first issue. That is the one we have been discussing with Hiroshi as he had submitted patches to enable such a feature.
That is what leads us to your request. If you enable VLAN promiscous the VF drivers currently don't support it and generally make a mess of things since this misidentify the VLAN ID on the incoming frames. So you would need to do two things. The first would be to update the PF driver so that if VLAN promiscuous mode is enabled you turn off any legacy VFs that might be running. The second would be to generate a new API version for the VF driver and add the necessary bits so that the VFs can identify which VLAN they are assigned to, if any. We did something similar for Jumbo Frames and FCoE/DCB so that we could allow a PF to enable these features even though it caused some legacy VFs to crash if such a feature was enabled. If you are wanting to enable VLAN promiscuous mode then that is the route you would like need to take.
- Alex
Quoted 36 lines · Alexander Duyck
Quoted 35 lines · Chris Friesen
Quoted 12 lines · Gregory V
Quoted 12 lines
-----Original Message----- From: Chris Friesen [mailto:chris.friesen at windriver.com] Sent: Monday, June 01, 2015 5:02 PM
To: Rose, Gregory V; intel-wired-lan at lists.osuosl.org Subject: Re:
[Intel-wired-lan] vlan packet handling in ixgbevf driver when in promiscuous modeCould you elaborate?
The use-case here is for NFV/SDN type stuff, where we want to be able to run a router or something similar in the guest--so it needs to be able to receive packets from all VLANs.
Quoted 8 lines
Quoted 2 lines
From what I see if you enable promiscuous mode on the PF then the VFs get packets from all VLANs.
If you have a working configuration then that's fine. I was merely pointing out that promiscuous mode, as the semantics are commonly applied to word in a networking context, is not supported for virtual functions on the 82599 controller. There are configurations you can apply to suit your own purposes and we would like to support that for you if we can.
Well, it's "working" but it means we have to patch the ixgbevf driver, which is a bit of a pain.
I'd like to see something like what Alexander Duyck suggested where the stock ixgbevf driver would only strip the VLAN tags off for packets where the VLAN matches the port VLAN ID and let all the other VLAN tags through. (If the guest interface is in promiscuous mode.)
Arguably if the guest interface is not in promiscuous mode then I think we should be dropping packets from all VLANs other than what the guest has explicitly registered for (or that the PF has assigned to the VF. It makes no sense to me that we would let them all through but strip off the VLAN header.
Quoted 8 lines
Quoted 1 line
Chris
Chris,
Based on what is in current net-next I don't see how you can enable VLAN promiscuous mode on the PF without having a modified driver of some sort. That is the first issue. That is the one we have been discussing with Hiroshi as he had submitted patches to enable such a feature.
I'm using the in-kernel linux 3.10 ixgbe driver, I think. From looking at the code when I enable promisc mode it just turns off the VLAN filtering on the PF completely.
Quoted 11 lines
That is what leads us to your request. If you enable VLAN promiscous the VF drivers currently don't support it and generally make a mess of things since this misidentify the VLAN ID on the incoming frames. So you would need to do two things. The first would be to update the PF driver so that if VLAN promiscuous mode is enabled you turn off any legacy VFs that might be running. The second would be to generate a new API version for the VF driver and add the necessary bits so that the VFs can identify which VLAN they are assigned to, if any. We did something similar for Jumbo Frames and FCoE/DCB so that we could allow a PF to enable these features even though it caused some legacy VFs to crash if such a feature was enabled. If you are wanting to enable VLAN promiscuous mode then that is the route you would like need to take.
Right, that makes sense. So are you suggesting that we code this up and submit it for inclusion in the upstream driver? Or are you suggesting that this is the route we should take for patching it privately? My goal would be to have this be supported in the upstream ixgbevf driver, since otherwise we would have to get all of our users to patch their drivers separately.
Chris
Quoted 63 lines · Chris Friesen
Quoted 53 lines · Alexander Duyck
Quoted 50 lines · Chris Friesen
Quoted 19 lines · Gregory V
Quoted 15 lines
-----Original Message----- From: Chris Friesen [mailto:chris.friesen at windriver.com] Sent: Monday, June 01, 2015 5:02 PM
To: Rose, Gregory V; intel-wired-lan at lists.osuosl.org Subject: Re:
[Intel-wired-lan] vlan packet handling in ixgbevf driver when in promiscuous modeCould you elaborate?
The use-case here is for NFV/SDN type stuff, where we want to be able to
run a router or something similar in the guest--so it needs to be able to
receive packets from all VLANs.Quoted 3 lines
From what I see if you enable promiscuous mode on the PF then the VFs get
packets from all VLANs.Quoted 9 lines
If you have a working configuration then that's fine. I was merely pointing
out that promiscuous mode, as the semantics are commonly applied to word in a
networking context, is not supported for virtual functions on the 82599
controller. There are configurations you can apply to suit your own purposes
and we would like to support that for you if we can.Well, it's "working" but it means we have to patch the ixgbevf driver, which
is a bit of a pain.I'd like to see something like what Alexander Duyck suggested where the stock
ixgbevf driver would only strip the VLAN tags off for packets where the VLAN
matches the port VLAN ID and let all the other VLAN tags through. (If the
guest interface is in promiscuous mode.)Arguably if the guest interface is not in promiscuous mode then I think we
should be dropping packets from all VLANs other than what the guest has explicitly registered for (or that the PF has assigned to the VF. It makes no
sense to me that we would let them all through but strip off the VLAN header.Quoted 1 line
Chris
Quoted 8 lines
Chris,
Based on what is in current net-next I don't see how you can enable VLAN promiscuous mode on the PF without having a modified driver of some sort. That
is the first issue. That is the one we have been discussing with Hiroshi as he
had submitted patches to enable such a feature.
Quoted 4 lines
I'm using the in-kernel linux 3.10 ixgbe driver, I think. From looking at the code when I enable promisc mode it just turns off the VLAN filtering on the PF completely.
Ah, okay. I see it looks like this was changed by commit 670224f129299 ("ixgbe: Retain VLAN filtering in promiscuous + VT mode").
Quoted 20 lines
Quoted 11 lines
That is what leads us to your request. If you enable VLAN promiscous the VF
drivers currently don't support it and generally make a mess of things since
this misidentify the VLAN ID on the incoming frames. So you would need to do
two things. The first would be to update the PF driver so that if VLAN promiscuous mode is enabled you turn off any legacy VFs that might be running.
The second would be to generate a new API version for the VF driver and add theQuoted 8 lines
necessary bits so that the VFs can identify which VLAN they are assigned to, if
any. We did something similar for Jumbo Frames and FCoE/DCB so that we could
allow a PF to enable these features even though it caused some legacy VFs to
crash if such a feature was enabled. If you are wanting to enable VLAN promiscuous mode then that is the route you would like need to take.
Quoted 8 lines
Right, that makes sense. So are you suggesting that we code this up and submit it for inclusion in the upstream driver? Or are you suggesting that this is the route we should take for patching it privately? My goal would be to have this be supported in the upstream ixgbevf driver, since otherwise we would have to get all of our users to patch their drivers separately.
Chris
I cannot really talk for the Intel guys as I am more of an outside observer, but as a kernel contributor I would say yes, we would rather see this upstream than floating around as an out-of-tree change. You would need to patch the upstream PF and VF drivers to get it resolved since you wouldn't want to break any existing VFs so backwards compatibility on both sides is a requirement.
- Alex
- Alex