Deliver skbs to intermediate interfaces, when using cascaded rx-handlers
Subsystems (2)
I have a system with the following configuration:
* eth0 is a regular Ethernet MAC, connected to a hardware switch.
* port[1-4] are the in-kernel representation of the switch.
* team0 is a regular team-interface which bonds port1 and port2.
+-------+
| team0 |
+---+---+
|
+----+----+
| |
+---+---+ +---+---+ +-------+ +-------+
| port1 | | port2 | | port3 | | port4 |
+---+---+ +---+---+ +---+---+ +---+---+
| | | |
+---------+----+----+---------+
|
+---+---+
| eth0 |
+-------+Both the switch and the team driver attaches rx-handlers to their lower layers. The problem is that team expects LACP frames that are intercepted by team0, to also be delivered to the port interface by __netif_receive_skb_core (in the final iteration of the registered protocols). However in this case, when two rx-handlers are cascaded, the skb will be delivered to eth0, since that is the original device (orig_dev).
There are a few ways this can be solved as far as i can see:
1. Introduce a new rx-handler return code that specifies that skb->dev has been altered, like RX_HANDLER_ANOTHER, and that we also want to change orig_dev to this new device.
2. Change the switch driver to return RX_HANDLER_CONSUMED and then queue the skb all over again, though I am afraid that this will eat some cycles.
3. Keep track of all traversed interfaces and change the final protocol iteration to deliver the skb to all intermediate devices.
What would be your suggestion?
Signature · 2 lines
Thanks - wkz
Quoted 32 lines · Fri, Feb 21, 2014 at 11:01:58AM CET, [email protected] wrote:
I have a system with the following configuration:
* eth0 is a regular Ethernet MAC, connected to a hardware switch.
* port[1-4] are the in-kernel representation of the switch.
* team0 is a regular team-interface which bonds port1 and port2.+-------+ | team0 | +---+---+ | +----+----+ | | +---+---+ +---+---+ +-------+ +-------+ | port1 | | port2 | | port3 | | port4 | +---+---+ +---+---+ +---+---+ +---+---+ | | | | +---------+----+----+---------+ | +---+---+ | eth0 | +-------+Both the switch and the team driver attaches rx-handlers to their lower layers. The problem is that team expects LACP frames that are intercepted by team0, to also be delivered to the port interface by
__netif_receive_skb_core(in the final iteration of the registered protocols). However in this case, when two rx-handlers are cascaded, the skb will be delivered to eth0, since that is the original device (orig_dev).There are a few ways this can be solved as far as i can see:
Quoted 7 lines
1. Introduce a new rx-handler return code that specifies that
skb->devhas been altered, likeRX_HANDLER_ANOTHER, and that we also want to changeorig_devto this new device.2. Change the switch driver to return
RX_HANDLER_CONSUMEDand then queue the skb all over again, though I am afraid that this will eat some cycles.
What "switch driver" are you reffering to?
Quoted 14 lines
3. Keep track of all traversed interfaces and change the final protocol iteration to deliver the skb to all intermediate devices.
What would be your suggestion?
Signature · 7 lines
Thanks - wkz -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines
What "switch driver" are you reffering to?
The driver that is in charge of multiplexing packets from the switch chip out to the port interfaces. It registers an rx-handler on eth0 and forwards packets to port[1-4] by reading a custom tag (similar to 802.1Q) inserted by the switch.
Signature · 2 lines
Thanks - wkz
Quoted 7 lines · Fri, Feb 21, 2014 at 12:12:57PM CET, [email protected] wrote:
Quoted 2 lines
What "switch driver" are you reffering to?
The driver that is in charge of multiplexing packets from the switch chip out to the port interfaces. It registers an rx-handler on eth0 and forwards packets to port[1-4] by reading a custom tag (similar to 802.1Q) inserted by the switch.
Allright, but what's the name? I would like to find it and see the code.
Quoted 4 lines
Signature · 3 lines
-- Thanks - wkz
Quoted 10 lines · Jiri Pirko
Quoted 7 lines · Fri, Feb 21, 2014 at 12:12:57PM CET, [email protected] wrote:
Quoted 2 lines
What "switch driver" are you reffering to?
The driver that is in charge of multiplexing packets from the switch chip out to the port interfaces. It registers an rx-handler on eth0 and forwards packets to port[1-4] by reading a custom tag (similar to 802.1Q) inserted by the switch.
Allright, but what's the name? I would like to find it and see the code.
There's not really a name :), but I have made the source available on github:
https://github.com/wkz/switchcore
The relevant parts for this discussion should be in sc_main.c.
Quoted 5 lines
Quoted 5 lines
Signature · 3 lines
-- Thanks - wkz
Signature · 2 lines
Thanks - wkz
Quoted 10 lines · Fri, Feb 21, 2014 at 01:02:44PM CET, [email protected] wrote:
Quoted 9 lines · Jiri Pirko
Quoted 7 lines · Fri, Feb 21, 2014 at 12:12:57PM CET, [email protected] wrote:
Quoted 2 lines
What "switch driver" are you reffering to?
The driver that is in charge of multiplexing packets from the switch chip out to the port interfaces. It registers an rx-handler on eth0 and forwards packets to port[1-4] by reading a custom tag (similar to 802.1Q) inserted by the switch.
Quoted 5 lines
Quoted 1 line
Allright, but what's the name? I would like to find it and see the code.
There's not really a name :), but I have made the source available on github:
Are you aware of existing DSA infrastructure?
Quoted 12 lines
The relevant parts for this discussion should be in
sc_main.c.Quoted 5 lines
Quoted 5 lines
Signature · 3 lines
-- Thanks - wkzSignature · 2 lines
Thanks - wkz
Quoted 11 lines · Jiri Pirko
Quoted 10 lines · Fri, Feb 21, 2014 at 01:02:44PM CET, [email protected] wrote:
Quoted 9 lines · Jiri Pirko
Quoted 7 lines · Fri, Feb 21, 2014 at 12:12:57PM CET, [email protected] wrote:
Quoted 2 lines
What "switch driver" are you reffering to?
The driver that is in charge of multiplexing packets from the switch chip out to the port interfaces. It registers an rx-handler on eth0 and forwards packets to port[1-4] by reading a custom tag (similar to 802.1Q) inserted by the switch.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Allright, but what's the name? I would like to find it and see the code.
There's not really a name :), but I have made the source available on github:
Are you aware of existing DSA infrastructure?
Yes I am, we have our reasons for using our own driver. Most of it having to do with dynamic configuration, multiple hardware channel support and a variety of exotic hardware setups.
Do you have any input on the original question?
Quoted 12 lines
Quoted 12 lines
The relevant parts for this discussion should be in
sc_main.c.Quoted 5 lines
Quoted 5 lines
Signature · 3 lines
-- Thanks - wkzSignature · 2 lines
Thanks - wkz
Signature · 2 lines
Thanks - wkz
Quoted 16 lines · Fri, Feb 21, 2014 at 03:31:14PM CET, [email protected] wrote:
Quoted 15 lines · Jiri Pirko
Quoted 14 lines · Fri, Feb 21, 2014 at 01:02:44PM CET, [email protected] wrote:
Quoted 10 lines · Jiri Pirko
Quoted 7 lines · Fri, Feb 21, 2014 at 12:12:57PM CET, [email protected] wrote:
Quoted 2 lines
What "switch driver" are you reffering to?
The driver that is in charge of multiplexing packets from the switch chip out to the port interfaces. It registers an rx-handler on eth0 and forwards packets to port[1-4] by reading a custom tag (similar to 802.1Q) inserted by the switch.
Allright, but what's the name? I would like to find it and see the code.
There's not really a name :), but I have made the source available on github:
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Are you aware of existing DSA infrastructure?
Yes I am, we have our reasons for using our own driver. Most of it having to do with dynamic configuration, multiple hardware channel support and a variety of exotic hardware setups.
Would not it be better to make DSA better in order to achieve your goal?
Quoted 2 lines
Do you have any input on the original question?
I'm not really sure how to handle this. I fail to clearly understand the topology you are describing...
Quoted 17 lines
Quoted 13 lines
Quoted 13 lines
The relevant parts for this discussion should be in
sc_main.c.Quoted 5 lines
Quoted 5 lines
Signature · 3 lines
-- Thanks - wkzSignature · 2 lines
Thanks - wkzSignature · 2 lines
Thanks - wkz
Quoted 19 lines · Jiri Pirko
Quoted 18 lines · Fri, Feb 21, 2014 at 03:31:14PM CET, [email protected] wrote:
Quoted 17 lines · Jiri Pirko
Quoted 15 lines · Fri, Feb 21, 2014 at 01:02:44PM CET, [email protected] wrote:
Quoted 10 lines · Jiri Pirko
Quoted 7 lines · Fri, Feb 21, 2014 at 12:12:57PM CET, [email protected] wrote:
Quoted 2 lines
What "switch driver" are you reffering to?
The driver that is in charge of multiplexing packets from the switch chip out to the port interfaces. It registers an rx-handler on eth0 and forwards packets to port[1-4] by reading a custom tag (similar to 802.1Q) inserted by the switch.
Allright, but what's the name? I would like to find it and see the code.
There's not really a name :), but I have made the source available on github:
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Are you aware of existing DSA infrastructure?
Yes I am, we have our reasons for using our own driver. Most of it having to do with dynamic configuration, multiple hardware channel support and a variety of exotic hardware setups.
Would not it be better to make DSA better in order to achieve your goal?
Sure, that is something we are considering. But before we do that we need to make sure that our model works, end-to-end, for all important use-cases.
Quoted 23 lines
Quoted 2 lines
Do you have any input on the original question?
I'm not really sure how to handle this. I fail to clearly understand the topology you are describing...
Quoted 17 lines
Quoted 13 lines
Quoted 13 lines
The relevant parts for this discussion should be in
sc_main.c.Quoted 5 lines
Quoted 5 lines
Signature · 3 lines
-- Thanks - wkzSignature · 2 lines
Thanks - wkzSignature · 2 lines
Thanks - wkz
Signature · 2 lines
Thanks - wkz
David Miller
maintainerFrom: Jiri Pirko <[email protected]>
Date: Fri, 21 Feb 2014 15:48:27 +0100
Quoted 1 line
Would not it be better to make DSA better in order to achieve your goal?
+1
Quoted 6 lines · David Miller
From: Jiri Pirko <[email protected]>
Date: Fri, 21 Feb 2014 15:48:27 +0100Quoted 1 line
Would not it be better to make DSA better in order to achieve your goal?
+1
Ok, message received. I will try to merge the changes back into the mainline DSA infrastructure. I honestly didn't think it would be of much interest to anyone.
The original question is really separate from that though. In this case it is a DSA driver at the bottom, but the question applies to any case where there are multiple rx-handlers in a cascade configuration. Wouldn't you agree?
Signature · 2 lines
Thanks - wkz
Quoted 22 lines · Tobias Waldekranz
I have a system with the following configuration:
* eth0 is a regular Ethernet MAC, connected to a hardware switch. * port[1-4] are the in-kernel representation of the switch. * team0 is a regular team-interface which bonds port1 and port2. +-------+ | team0 | +---+---+ | +----+----+ | | +---+---+ +---+---+ +-------+ +-------+ | port1 | | port2 | | port3 | | port4 | +---+---+ +---+---+ +---+---+ +---+---+ | | | | +---------+----+----+---------+ | +---+---+ | eth0 | +-------+
Quoted 7 lines
Both the switch and the team driver attaches rx-handlers to their lower layers. The problem is that team expects LACP frames that are intercepted by team0, to also be delivered to the port interface by
__netif_receive_skb_core(in the final iteration of the registered protocols). However in this case, when two rx-handlers are cascaded, the skb will be delivered to eth0, since that is the original device (orig_dev).
Indeed. It is not possible to update orig_dev, so it is stuck on eth0 regardless of all the stacked interfaces.
Quoted 5 lines
There are a few ways this can be solved as far as i can see:
1. Introduce a new rx-handler return code that specifies that
skb->devhas been altered, likeRX_HANDLER_ANOTHER, and that we also want to changeorig_devto this new device.
But then you need to know which rx-handler will run next, because the orig_dev might still be implicit wanted.
Quoted 3 lines
2. Change the switch driver to return
RX_HANDLER_CONSUMEDand then queue the skb all over again, though I am afraid that this will eat some cycles.
Before queue again, you need to update skb->dev, right? This seems similar to (1).
Quoted 2 lines
3. Keep track of all traversed interfaces and change the final protocol iteration to deliver the skb to all intermediate devices.
That's what I thought too.
fbl
Quoted 34 lines · Flavio Leitner
Quoted 22 lines · Tobias Waldekranz
I have a system with the following configuration:
* eth0 is a regular Ethernet MAC, connected to a hardware switch. * port[1-4] are the in-kernel representation of the switch. * team0 is a regular team-interface which bonds port1 and port2. +-------+ | team0 | +---+---+ | +----+----+ | | +---+---+ +---+---+ +-------+ +-------+ | port1 | | port2 | | port3 | | port4 | +---+---+ +---+---+ +---+---+ +---+---+ | | | | +---------+----+----+---------+ | +---+---+ | eth0 | +-------+Quoted 7 lines
Both the switch and the team driver attaches rx-handlers to their lower layers. The problem is that team expects LACP frames that are intercepted by team0, to also be delivered to the port interface by
__netif_receive_skb_core(in the final iteration of the registered protocols). However in this case, when two rx-handlers are cascaded, the skb will be delivered to eth0, since that is the original device (orig_dev).Indeed. It is not possible to update
orig_dev, so it is stuck on eth0 regardless of all the stacked interfaces.
Quoted 8 lines
Quoted 5 lines
There are a few ways this can be solved as far as i can see:
1. Introduce a new rx-handler return code that specifies that
skb->devhas been altered, likeRX_HANDLER_ANOTHER, and that we also want to changeorig_devto this new device.But then you need to know which rx-handler will run next, because the
orig_devmight still be implicit wanted.
The next rx-handler is resolved by looking at skb->dev->rx_handler, so that shouldn't be a problem. Or am I missing something?
Quoted 7 lines
Quoted 4 lines
2. Change the switch driver to return
RX_HANDLER_CONSUMEDand then queue the skb all over again, though I am afraid that this will eat some cycles.Before queue again, you need to update
skb->dev, right? This seems similar to (1).
Exactly. It is very similar. The difference is that I would have to call netif_rx and go through a lot of code again, instead of taking the fast path via the another_round-loop in __netif_receive_skb_core.
Quoted 5 lines
Quoted 3 lines
3. Keep track of all traversed interfaces and change the final protocol iteration to deliver the skb to all intermediate devices.
That's what I thought too.
Ok, so if I provide such a patch, you think it would have a good chance of getting accepted?
What are the performance implications of deliver_skb-ing the packet to each intermediate layer? Considering that most of those packets will probably not be of interest to anyone.
Quoted 2 lines
fbl
Signature · 2 lines
Thanks - wkz
Quoted 41 lines · Tobias Waldekranz
Quoted 40 lines · Flavio Leitner
Quoted 22 lines · Tobias Waldekranz
I have a system with the following configuration:
* eth0 is a regular Ethernet MAC, connected to a hardware switch. * port[1-4] are the in-kernel representation of the switch. * team0 is a regular team-interface which bonds port1 and port2. +-------+ | team0 | +---+---+ | +----+----+ | | +---+---+ +---+---+ +-------+ +-------+ | port1 | | port2 | | port3 | | port4 | +---+---+ +---+---+ +---+---+ +---+---+ | | | | +---------+----+----+---------+ | +---+---+ | eth0 | +-------+Quoted 7 lines
Both the switch and the team driver attaches rx-handlers to their lower layers. The problem is that team expects LACP frames that are intercepted by team0, to also be delivered to the port interface by
__netif_receive_skb_core(in the final iteration of the registered protocols). However in this case, when two rx-handlers are cascaded, the skb will be delivered to eth0, since that is the original device (orig_dev).Indeed. It is not possible to update
orig_dev, so it is stuck on eth0 regardless of all the stacked interfaces.Quoted 5 lines
There are a few ways this can be solved as far as i can see:
1. Introduce a new rx-handler return code that specifies that
skb->devhas been altered, likeRX_HANDLER_ANOTHER, and that we also want to changeorig_devto this new device.
Quoted 5 lines
Quoted 2 lines
But then you need to know which rx-handler will run next, because the
orig_devmight still be implicit wanted.The next rx-handler is resolved by looking at
skb->dev->rx_handler, so that shouldn't be a problem. Or am I missing something?
Yes, but all rx_handlers assume that orig_dev is intact and will be delivered at the end. That will change with this solution and I don't know if it can break any rx-handler.
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
2. Change the switch driver to return
RX_HANDLER_CONSUMEDand then queue the skb all over again, though I am afraid that this will eat some cycles.Before queue again, you need to update
skb->dev, right? This seems similar to (1).Exactly. It is very similar. The difference is that I would have to call
netif_rxand go through a lot of code again, instead of taking the fast path via the another_round-loop in__netif_receive_skb_core.
Right.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
3. Keep track of all traversed interfaces and change the final protocol iteration to deliver the skb to all intermediate devices.
That's what I thought too.
Ok, so if I provide such a patch, you think it would have a good chance of getting accepted?
Well, the final word comes from David and certainly you will need to justify why fixing DSA isn't enough.
Quoted 3 lines
What are the performance implications of deliver_skb-ing the packet to each intermediate layer? Considering that most of those packets will probably not be of interest to anyone.
Good question.
fbl