Skip to contentSkip to conversation

netdev: show the number of tx-packets in device

RFC PATCH New Koki Sanagi 1 participant 3 messages May 24, 2010

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover netdev: show the number of tx-packets in device

This patch-set adds tracepoints to dev_hard_start_xmit, consume_skb and dev_kfree_skb_irq and perf script which calculates the time from entry of ndo_start_xmit to dev_kfree_skb_* and the number of tx-packets in device.

-Perf script description
This script calculates two metric.

metric1: lap time between start_xmit - free_skb
This script calculate the time a packet passes from entry of ndo_start_xmit to dev_kfree_skb_irq or consume_skb. It indicate a time driver/device owns that packet. This script outputs an average time of all packets and a longest of that.

metric2: number of packets in device

Quoted 1 line

From the above time, we can calculate the number of packets in device at a

certain time. This script outputs an average of the number of packets in device and a largest of that.

-Merit
These tracepoints and script have two merits.

1. Detecting a packed tx-ring of network device
2. Detecting a defect of transmit functionality of network device

merit1: Detecting a packed tx-ring of network device

Using attached scripts, we can get a maximum number of packets in a device. If it reaches to the number of packets a device can own, tx-ring of that device is full and causes loss of network transmit performance. Because the driver of the device drop packet or stops tx-ring and reject it until it keeps some space. So, to keep good network transmit performance, it is good to keep some space in tx-ring. To keep some space in tx-ring, these tracepoints and script are useful.

To check this merit, I did a test.

Before starting a test, I want you to know that a maximum number of tx-packets e1000e can own is (tx-ring size - 20) / 2 packets.
Because e1000e keeps 20 descriptors for frags and 1 packet needs 2 descriptors due to tx-checksum.
So, if tx-ring size is 256,
(256 - 20) / 2 = 118 packets
If tx-ring size is 512,
(512 - 20) / 2 = 246 packets

Environment:

Test NIC:     Intel 82571EB (InterruptThrottleRate=1000)
Opponent NIC: Broadcom BCM5703X

InterruptThrottleRate was set to 1000 to make tx-ring packed deliberately.

Test load tool:
netperf -H XXX.XXX.XXX.XXX -t UDP_STREAM -- -m 1

With this environment, I compared following 2 cases. 1.Tx-ring size=256 case
2.Tx-ring size=512 case

Result:
1.The case of Tx-ring=256:
eth0 TX packets=1137811

        lap time between start_xmit - free_skb:
            avg=0.795687msec
            max=0.985911msec
         number of packets in device:
            avg=  64.66
            max= 118

netperf's result:

Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec
112640       1   10.00     1179077      0       0.94
109568           10.00      544750              0.44

2.The case of Tx-ring=512:
eth0 TX packets=1531629

        lap time between start_xmit - free_skb:
            avg=0.370052msec
            max=0.982069msec
        number of packets in device:
            avg=  49.70
            max= 164

netperf's result:

Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec
112640       1   10.00     1577840      0       1.26
109568           10.00      871058              0.70

In the case of tx-ring size=256(default), maximum number of packets in device reaches to 118. So this tx-ring is full and becomes a cause of network performance loss.
On the other hand, in the case of tx-ring size=512, e1000e can own 246 packets, but maximum number of packets in device doesn't reach it. so tx-ring has always some space and there is no network performance loss caused by packed tx-ring. Actually, about transmit throughput, The case of Tx-ring size=512 is better than the case of tx-ring size=256.
Like this, the number of packets in device is available to tune tx-ring size or other parameters to avoid packed tx-ring and is related to network transmit performance.

merit2: Detecting a defect of transmit functionality of network device

When device can't transmit due to hardware fault or driver's bug(I've encountered this), we can detect it. Because in such case, dev_hard_start_xmit is passed, but dev_kfree_skb_* is not passed.

NOTE:
This script has some problem,

-The number of tx-packets of netperf and that of this script are not equal. -Sometimes The max number of packets in device larger than the device can own.

Thanks,
Koki Sanagi.

#1 lore raw

RFC PATCH 1/2 netdev: add tracepoint to dev_hard_start_xmit, consume_skb and dev_kfree_skb_irq

This patch adds tracepoint to dev_hard_start_xmit, consume_skb and dev_kfree_skb_irq to monitor tx-packets in device.

---
 include/trace/events/skb.h |   64 ++++++++++++++++++++++++++++++++++++++++++++
 net/core/dev.c             |    4 +++
 net/core/skbuff.c          |    1 +
 3 files changed, 69 insertions(+), 0 deletions(-)
include/trace/events/skb.h +64 -0
index 4b2be6d..ed97580 100644
@@ -9,6 +9,52 @@
 #include <linux/tracepoint.h>
 
 /*
+ *  netdev_start_xmit - invoked when skb is passed to the driver
+ *  @skb:		pointer to struct sk_buff
+ *  @dev:		pointer to struct net_device
+ */
+TRACE_EVENT(netdev_start_xmit,
+
+	TP_PROTO(struct sk_buff *skb,
+		 struct net_device *dev),
+
+	TP_ARGS(skb, dev),
+
+	TP_STRUCT__entry(
+		__field(	const void *,	skbaddr		)
+		__field(	unsigned int,	len		)
+		__string(	name,		dev->name	)
+       ),
+
+	TP_fast_assign(
+		__entry->skbaddr = skb;
+		__entry->len = skb->len;
+		__assign_str(name, dev->name);
+	),
+
+	TP_printk("dev=%s skbaddr=%p len=%u",
+		__get_str(name), __entry->skbaddr, __entry->len)
+);
+
+TRACE_EVENT(consume_skb,
+
+	TP_PROTO(struct sk_buff *skb),
+
+	TP_ARGS(skb),
+
+	TP_STRUCT__entry(
+		__field(	void *,		skbaddr		)
+	),
+
+	TP_fast_assign(
+		__entry->skbaddr = skb;
+	),
+
+	TP_printk("skbaddr=%p",
+		__entry->skbaddr)
+);
+
+/*
  * Tracepoint for free an sk_buff:
  */
 TRACE_EVENT(kfree_skb,
@@ -35,6 +81,24 @@ TRACE_EVENT(kfree_skb,
 		__entry->skbaddr, __entry->protocol, __entry->location)
 );
 
+TRACE_EVENT(dev_kfree_skb_irq,
+
+	TP_PROTO(struct sk_buff *skb),
+
+	TP_ARGS(skb),
+
+	TP_STRUCT__entry(
+		__field(	void *,		skbaddr		)
+	),
+
+	TP_fast_assign(
+		__entry->skbaddr = skb;
+	),
+
+	TP_printk("skbaddr=%p",
+		__entry->skbaddr)
+);
+
 TRACE_EVENT(skb_copy_datagram_iovec,
 
 	TP_PROTO(const struct sk_buff *skb, int len),
net/core/dev.c +4 -0
index 32611c8..647d812 100644
@@ -130,6 +130,7 @@
 #include <linux/jhash.h>
 #include <linux/random.h>
 #include <trace/events/napi.h>
+#include <trace/events/skb.h>
 #include <linux/pci.h>
 
 #include "net-sysfs.h"
@@ -1577,6 +1578,7 @@ void dev_kfree_skb_irq(struct sk_buff *skb)
 		struct softnet_data *sd;
 		unsigned long flags;
 
+		trace_dev_kfree_skb_irq(skb);
 		local_irq_save(flags);
 		sd = &__get_cpu_var(softnet_data);
 		skb->next = sd->completion_queue;
@@ -1919,6 +1921,7 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				goto gso;
 		}
 
+		trace_netdev_start_xmit(skb, dev);
 		rc = ops->ndo_start_xmit(skb, dev);
 		if (rc == NETDEV_TX_OK)
 			txq_trans_update(txq);
@@ -1939,6 +1942,7 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
+		trace_netdev_start_xmit(nskb, dev);
 		rc = ops->ndo_start_xmit(nskb, dev);
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
net/core/skbuff.c +1 -0
index a9b0e1f..b9c963c 100644
@@ -466,6 +466,7 @@ void consume_skb(struct sk_buff *skb)
 		smp_rmb();
 	else if (likely(!atomic_dec_and_test(&skb->users)))
 		return;
+	trace_consume_skb(skb);
 	__kfree_skb(skb);
 }
 EXPORT_SYMBOL(consume_skb);
#2 lore raw

RFC PATCH 2/2 netdev: perf script to show the number of tx-packets in device

In reply to Koki Sanagi

This patch adds perf script to calculate the time from entry of ndo_start_xmit to dev_kfree_skb_* and the number of tx-packets in device.

---
 .../scripts/python/bin/tx-packet-in-device-record  |    2 +
 .../scripts/python/bin/tx-packet-in-device-report  |    4 +
 tools/perf/scripts/python/tx-packet-in-device.py   |  109 ++++++++++++++++++++
 3 files changed, 115 insertions(+), 0 deletions(-)
tools/perf/scripts/python/bin/tx-packet-in-device-record new file +2 -0
new file mode 100644
index 0000000..18f2356
@@ -0,0 +1,2 @@
+#!/bin/bash
+perf record -c 1 -f -a -M -R -e skb:netdev_start_xmit -e skb:consume_skb -e skb:dev_kfree_skb_irq
tools/perf/scripts/python/bin/tx-packet-in-device-report new file +4 -0
new file mode 100644
index 0000000..8ef4cc2
@@ -0,0 +1,4 @@
+#!/bin/bash
+# description: netif_receive_skb counts per poll
+# args: [comm]
+perf trace -s ~/libexec/perf-core/scripts/python/tx-packet-in-device.py $1
tools/perf/scripts/python/tx-packet-in-device.py new file +109 -0

109 lines not on this page · raw message