PF_PACKET updates
Subsystems (4)
cover PF_PACKET updates
Patch descriptions in individual patches.
In order to avoid a possible merge conflict, we suggest to take the first patch through net tree, merge net into net-next and apply the remaining two patches on top of it. Dave, please let us know if you like to handle this differently than suggested.
For patch 3 we'll send a man-page update as a follow-up.
Thanks !
Daniel Borkmann (3):
packet: fix send path when running with proto == 0
net: dev: move inline skb_needs_linearize helper to header
packet: introduce PACKET_QDISC_BYPASS socket option
Documentation/networking/packet_mmap.txt | 31 ++++++ include/linux/skbuff.h | 18 ++++ include/uapi/linux/if_packet.h | 1 + net/core/dev.c | 15 --- net/packet/af_packet.c | 156 +++++++++++++++++++++++-------- net/packet/internal.h | 1 + 6 files changed, 170 insertions(+), 52 deletions(-)
Signature · 1 line
1.8.3.1
PATCH 1/3 packet: fix send path when running with proto == 0
Commit e40526cb20b5 introduced a cached dev pointer, that gets hooked into register_prot_hook(), __unregister_prot_hook() to update the device used for the send path.
We need to fix this up, as otherwise this will not work with sockets created with protocol = 0, plus with sll_protocol = 0 passed via sockaddr_ll when doing the bind.
So instead, assign the pointer directly. The compiler can inline these helper functions automagically.
While at it, also assume the cached dev fast-path as likely(), and document this variant of socket creation as it seems it is not widely used (seems not even the author of TX_RING was aware of that in his reference example [1]). Tested with reproducer from e40526cb20b5.
[1] http://wiki.ipxwarzone.com/index.php5?title=Linux_packet_mmap#Example
--- Documentation/networking/packet_mmap.txt | 10 +++++ net/packet/af_packet.c | 65 ++++++++++++++++++++------------ 2 files changed, 50 insertions(+), 25 deletions(-)
Documentation/networking/packet_mmap.txt +10 -0
@@ -123,6 +123,16 @@ Transmission process is similar to capture as shown below. [shutdown] close() --------> destruction of the transmission socket and deallocation of all associated resources. +Socket creation and destruction is also straight forward, and is done +the same way as in capturing described in the previous paragraph: + + int fd = socket(PF_PACKET, mode, 0); + +The protocol can optionally be 0 in case we only want to transmit +via this socket, which avoids an expensive call to packet_rcv(). +In this case, you also need to bind(2) the TX_RING with sll_protocol = 0 +set. Otherwise, htons(ETH_P_ALL) or any other protocol, for example. + Binding the socket to your network interface is mandatory (with zero copy) to know the header size of frames used in the circular buffer.
net/packet/af_packet.c +40 -25
136 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH 2/3 net: dev: move inline skb_needs_linearize helper to header
In reply to Daniel BorkmannAs we need it elsewhere, move the inline helper function of skb_needs_linearize() over to skbuff.h include file. While at it, also convert the return to 'bool' instead of 'int' and add a proper kernel doc.
--- include/linux/skbuff.h | 18 ++++++++++++++++++ net/core/dev.c | 15 --------------- 2 files changed, 18 insertions(+), 15 deletions(-)
include/linux/skbuff.h +18 -0
@@ -2395,6 +2395,24 @@ static inline void *skb_header_pointer(const struct sk_buff *skb, int offset,return buffer; } +/** +* skb_needs_linearize - check if we need to linearize a given skb +* depending on the given device features. +* @skb: socket buffer to check +* @features: net device features +* +* Returns true if either: +* 1. skb has frag_list and the device doesn't support FRAGLIST, or +* 2. skb is fragmented and the device does not support SG. +*/ +static inline bool skb_needs_linearize (struct sk_buff *skb, +netdev_features_t features) +{ +return skb_is_nonlinear (skb) && + ((skb_has_frag_list (skb) && !(features &NETIF_F_FRAGLIST )) || + (skb_shinfo (skb)->nr_frags && !(features &NETIF_F_SG ))); +} +static inline void skb_copy_from_linear_data (const struct sk_buff *skb,void *to,const unsigned int len)
net/core/dev.c +0 -15
@@ -2523,21 +2523,6 @@ netdev_features_t netif_skb_features(struct sk_buff *skb) }EXPORT_SYMBOL (netif_skb_features); -/* -* Returns true if either: -* 1. skb has frag_list and the device doesn't support FRAGLIST, or -* 2. skb is fragmented and the device does not support SG. -*/ -static inline int skb_needs_linearize (struct sk_buff *skb, -netdev_features_t features) -{ -return skb_is_nonlinear (skb) && - ((skb_has_frag_list (skb) && - !(features &NETIF_F_FRAGLIST )) || - (skb_shinfo (skb)->nr_frags && - !(features &NETIF_F_SG ))); -} -int dev_hard_start_xmit (struct sk_buff *skb,struct net_device *dev,struct netdev_queue *txq,void *accel_priv) {
Signature · 1 line
1.8.3.1
PATCH 3/3 packet: introduce PACKET_QDISC_BYPASS socket option
In reply to Daniel BorkmannThis patch introduces a PACKET_QDISC_BYPASS socket option, that allows for using a similar xmit() function as in pktgen instead of taking the dev_queue_xmit() path. This can be very useful when PF_PACKET applications are required to be used in a similar scenario as pktgen, but with full, flexible packet payload that needs to be provided, for example.
On default, nothing changes in behaviour for normal PF_PACKET TX users, so everything stays as is for applications. New users, however, can now set PACKET_QDISC_BYPASS if needed to prevent own packets from i) reentering packet_rcv() and ii) to directly push the frame to the driver.
In doing so we can increase pps (here 64 byte packets) for PF_PACKET a bit:
# CPUs -- QDISC_BYPASS -- qdisc path -- qdisc path[**]
1 CPU == 1,509,628 pps -- 1,208,708 -- 1,247,436 2 CPUs == 3,198,659 pps -- 2,536,012 -- 1,605,779 3 CPUs == 4,787,992 pps -- 3,788,740 -- 1,735,610 4 CPUs == 6,173,956 pps -- 4,907,799 -- 1,909,114 5 CPUs == 7,495,676 pps -- 5,956,499 -- 2,014,422 6 CPUs == 9,001,496 pps -- 7,145,064 -- 2,155,261 7 CPUs == 10,229,776 pps -- 8,190,596 -- 2,220,619 8 CPUs == 11,040,732 pps -- 9,188,544 -- 2,241,879 9 CPUs == 12,009,076 pps -- 10,275,936 -- 2,068,447 10 CPUs == 11,380,052 pps -- 11,265,337 -- 1,578,689 11 CPUs == 11,672,676 pps -- 11,845,344 -- 1,297,412
[...]
20 CPUs == 11,363,192 pps -- 11,014,933 -- 1,245,081
[**]: qdisc path with packet_rcv(), how probably most people
seem to use it (hopefully not anymore if not needed)
The test was done using a modified trafgen, sending a simple static 64 bytes packet, on all CPUs. The trick in the fast "qdisc path" case, is to avoid reentering packet_rcv() by setting the RAW socket protocol to zero, like:socket(PF_PACKET, SOCK_RAW, 0);
Tradeoffs are documented as well in this patch, clearly, if queues are busy, we will drop more packets, tc disciplines are ignored, and these packets are not visible to taps anymore. For a pktgen like scenario, we argue that this is acceptable.
The pointer to the xmit function has been placed in packet socket structure hole between cached_dev and prot_hook that is hot anyway as we're working on cached_dev in each send path.
Done in joint work together with Jesper Dangaard Brouer.
---
We will send a patch for the man-page after this is accepted.
Documentation/networking/packet_mmap.txt | 21 ++++++++ include/uapi/linux/if_packet.h | 1 + net/packet/af_packet.c | 91 +++++++++++++++++++++++++++----- net/packet/internal.h | 1 + 4 files changed, 102 insertions(+), 12 deletions(-)
Documentation/networking/packet_mmap.txt +21 -0
@@ -953,6 +953,27 @@ int main(int argc, char **argp) } ------------------------------------------------------------------------------- ++ PACKET_QDISC_BYPASS +------------------------------------------------------------------------------- + +If there is a requirement to load the network with many packets in a similar +fashion as pktgen does, you might set the following option after socket +creation: + + int one = 1; + setsockopt(fd, SOL_PACKET, PACKET_QDISC_BYPASS, &one, sizeof(one)); + +This has the side-effect, that packets sent through PF_PACKET will bypass the +kernel's qdisc layer and are forcedly pushed to the driver directly. Meaning, +packet are not buffered, tc disciplines are ignored, increased loss can occur +and such packets are also not visible to other PF_PACKET sockets anymore. So, +you have been warned; generally, this can be useful for stress testing various +components of a system. + +On default, PACKET_QDISC_BYPASS is disabled and needs to be explicitly enabled +on PF_PACKET sockets. + +------------------------------------------------------------------------------- + PACKET_TIMESTAMP -------------------------------------------------------------------------------
include/uapi/linux/if_packet.h +1 -0
@@ -51,6 +51,7 @@ struct sockaddr_ll {#define PACKET_TIMESTAMP 17#define PACKET_FANOUT 18#define PACKET_TX_HAS_OFF 19 +#define PACKET_QDISC_BYPASS 20#define PACKET_FANOUT_HASH 0#define PACKET_FANOUT_LB 1
net/packet/af_packet.c +79 -12
166 lines not on this page · raw message
net/packet/internal.h +1 -0
@@ -114,6 +114,7 @@ struct packet_sock { unsigned int tp_tx_has_off:1;unsigned int tp_tstamp;struct net_device __rcu *cached_dev; +int (*xmit)(struct sk_buff *skb);struct packet_type prot_hook____cacheline_aligned_in_smp ; };
Signature · 1 line
1.8.3.1
David Miller
maintainerFrom: Daniel Borkmann <[email protected]>
Date: Fri, 6 Dec 2013 11:36:14 +0100
Quoted 8 lines
Patch descriptions in individual patches.
In order to avoid a possible merge conflict, we suggest to take the first patch through net tree, merge net into net-next and apply the remaining two patches on top of it. Dave, please let us know if you like to handle this differently than suggested.
For patch 3 we'll send a man-page update as a follow-up.
Patch #1 applied to 'net' and queued up for -stable.
Patch #2 and #3 applied to 'net-next'.
Thanks!