net: add network device notifier trace points

There already are network trace points for transmit and receive but nothing for state changes. Add network tracepoints for before and after netlink callback is done. This is simple (without extack or other info) but that could be added if useful. Network namespace id would also be helpful but hard to get a string for it.

This is the result of a conversation about monitoring of link state changes with BPF. Parsing netlink is hard and unnecessary because the data exists (unserialized) already in the callbacks.

  #  cd /sys/kernel/debug/tracing
  #  echo 1 > events/net/net_dev_notifier_entry/enable
  #  echo 1 > events/net/net_dev_notifier/enable
  #  ip li set dev eno1 down
  #  ip li set dev eno1 up
  #  cat trace
	# tracer: nop
	#
	#                              _-----=> irqs-off

	#                             / _----=> need-resched
	#                            | / _---=> hardirq/softirq
	#                            || / _--=> preempt-depth
	#                            ||| /     delay
	#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
	#              | |       |   ||||       |         |
		      ip-3194  [011] ....    74.926831: net_dev_notifier_entry: dev=eno1 event=GOING_DOWN
		      ip-3194  [011] ....    74.926838: net_dev_notifier: dev=eno1 event=GOING_DOWN ret=1

This is the result of a conversation about monitoring of link state changes with BPF.

Signed-off-by Stephen Hemminger <[email protected]>

---
 include/trace/events/net.h | 115 +++++++++++++++++++++++++++++++++++++
 net/core/dev.c             |   9 ++-
 2 files changed, 123 insertions(+), 1 deletion(-)

include/trace/events/net.h +0 -0

index 1efd7d9b25fe..141310d24610 100644

include/trace/events/net.h +5 -0

+TRACE_EVENT(net_dev_notifier_entry,
+
+	TP_PROTO(const struct netdev_notifier_info *info, unsigned long val),
+
+	TP_ARGS(info, val),

include/trace/events/net.h +18 -0

+
+	TP_printk("dev=%s event=%s",
+		  __get_str(name), netdev_event_type(__entry->event))
+);
+
+TRACE_EVENT(net_dev_notifier,
+
+	TP_PROTO(const struct netdev_notifier_info *info, int rc, unsigned long val),
+
+	TP_ARGS(info, rc, val),
+
+	TP_STRUCT__entry(
+		__string(	name,		 info->dev->name   )
+		__field(	enum netdev_cmd, event	           )
+		__field(	int,		 rc		   )
+	),
+
+	TP_fast_assign(

Quoted 2 lines · On Tue, 18 Dec 2018 18:27:06 -0800 Stephen Hemminger…

This is the result of a conversation about monitoring of link state changes with BPF.

If you want to use this from BPF then you are in for a surprise. As tracepoints BPF cannot read these "__string" constructs, here the netdev name. I tried a lot of different tricks that didn't work, see [1], until Alexei explained that it simply isn't supported.

I instead recommend adding the ifindex to the tracepoint. The__string and __assign_str is also a performance concern as it does strcpy behind your back.

I have an year old TODO list item about improving this:
** TODO Make perf-script plugin for ifindex to name translation

    SCHEDULED: <2017-11-20 Mon>

Today, the existing network tracepoints using dev->name is not that usable by BPF, as BPF cannot identify the interface. Thus IMHO it would make sense to convert the existing network tracepoints dev->name into dev->ifindex, and then let perf-script convert this to the interface name. Either in userspace via if_indextoname(3), or (as ACME pointed out at the time) we might want to have a lookup table stored together with perf.data for later inspection (in-case ifindexes changed).

[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c#L34-L130

Quoted 2 lines · On Tue, 18 Dec 2018 18:27:06 -0800 Stephen Hemminger…

This is the result of a conversation about monitoring of link state changes with BPF.

If you want to use this from BPF then you are in for a surprise. As tracepoints BPF cannot read these "__string" constructs, here the netdev name. I tried a lot of different tricks that didn't work, see [1], until Alexei explained that it simply isn't supported.

I instead recommend adding the ifindex to the tracepoint. The__string and __assign_str is also a performance concern as it does strcpy behind your back.

I have an year old TODO list item about improving this:
** TODO Make perf-script plugin for ifindex to name translation

    SCHEDULED: <2017-11-20 Mon>

Today, the existing network tracepoints using dev->name is not that usable by BPF, as BPF cannot identify the interface. Thus IMHO it would make sense to convert the existing network tracepoints dev->name into dev->ifindex, and then let perf-script convert this to the interface name. Either in userspace via if_indextoname(3), or (as ACME pointed out at the time) we might want to have a lookup table stored together with perf.data for later inspection (in-case ifindexes changed).

[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c#L34-L130

Quoted 10 lines

Signed-off-by Stephen Hemminger <[email protected]>

---
 include/trace/events/net.h | 115 +++++++++++++++++++++++++++++++++++++
 net/core/dev.c             |   9 ++-
 2 files changed, 123 insertions(+), 1 deletion(-)

include/trace/events/net.h +0 -0

index 1efd7d9b25fe..141310d24610 100644

[...]

Quoted 5 lines

include/trace/events/net.h +5 -0

+TRACE_EVENT(net_dev_notifier_entry,
+
+	TP_PROTO(const struct netdev_notifier_info *info, unsigned long val),
+
+	TP_ARGS(info, val),

include/trace/events/net.h line 100

Quoted 10 lines

include/trace/events/net.h +10 -0

+
+	TP_STRUCT__entry(
+		__string(	name,		 info->dev->name )
+		__field(	enum netdev_cmd, event	         )
+	),
+
+	TP_fast_assign(
+		__assign_str(name, info->dev->name);
+		__entry->event = val;
+       ),

These __string and __assign_str are costly and behind the scenes does a strcpy.

Quoted 18 lines

include/trace/events/net.h +18 -0

+
+	TP_printk("dev=%s event=%s",
+		  __get_str(name), netdev_event_type(__entry->event))
+);
+
+TRACE_EVENT(net_dev_notifier,
+
+	TP_PROTO(const struct netdev_notifier_info *info, int rc, unsigned long val),
+
+	TP_ARGS(info, rc, val),
+
+	TP_STRUCT__entry(
+		__string(	name,		 info->dev->name   )
+		__field(	enum netdev_cmd, event	           )
+		__field(	int,		 rc		   )
+	),
+
+	TP_fast_assign(

include/trace/events/net.h line 128

Quoted 10 lines

include/trace/events/net.h +10 -0

+		__assign_str(name, info->dev->name);
+		__entry->event = val;
+		__entry->rc = rc;
+       ),
+
+	TP_printk("dev=%s event=%s ret=%d",
+		  __get_str(name), netdev_event_type(__entry->event),
+		  __entry->rc)
+);
+

The bare minimum change is to _also_ add (info->dev->)ifindex to the tracepoint, as this makes it usable from BPF.

Hmm, why not just doing something as in your example below with napi_poll() where you pass in the napi pointer, and then use bpf_probe_read_str() on ctx->dev for fetching the name? At least there this should work and should be okay given it's rather slow-path event.

Quoted 1 line

[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c#L34-L130

My napi_monitor use-case is not a slow-path event, even-though in optimal cases we should handle 64 packets per tracepoint invocation, but I'm using this for 100G NICs with >20Mpps. And I mostly use the tool when something looks wrong and I don't see 64 packet bulks, which is also why I detect when this gets invoked from idle task or from ksoftirqd.

Quoted 6 lines · On Wed, 19 Dec 2018 16:46:05 +0100 Daniel Borkmann <[email protected]>…

Hmm, why not just doing something as in your example below with napi_poll() where you pass in the napi pointer, and then use bpf_probe_read_str() on ctx->dev for fetching the name? At least there this should work and should be okay given it's rather slow-path event.

Quoted 1 line

[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c#L34-L130

I didn't try to use bpf_probe_read_str() in [1], but that is also not what I want in my use-case. I don't want the name, but the ifindex to filter on, as it will be faster. My use-case is allowing my napi_monitor program to filter on a specific net_device, inside the kernel via BPF.

E.g. this didn't work:
bpf_probe_read(&ifindex, 4, &ctx->napi->dev->ifindex);

Perhaps you know how I can do this deref correctly?

My napi_monitor use-case is not a slow-path event, even-though in optimal cases we should handle 64 packets per tracepoint invocation, but I'm using this for 100G NICs with >20Mpps. And I mostly use the tool when something looks wrong and I don't see 64 packet bulks, which is also why I detect when this gets invoked from idle task or from ksoftirqd.

Quoted 10 lines

include/trace/events/net.h +10 -0

+TRACE_EVENT(net_dev_notifier_entry,
+
+	TP_PROTO(const struct netdev_notifier_info *info, unsigned long val),
+
+	TP_ARGS(info, val),
+
+	TP_STRUCT__entry(
+		__string(	name,		 info->dev->name )
+		__field(	enum netdev_cmd, event	         )
+	),

include/trace/events/net.h line 100

Quoted 5 lines

include/trace/events/net.h +5 -0

+
+	TP_fast_assign(
+		__assign_str(name, info->dev->name);
+		__entry->event = val;
+       ),  

These __string and __assign_str are costly and behind the scenes does a strcpy.

Today, the existing network tracepoints using dev->name is not that usable by BPF, as BPF cannot identify the interface. Thus IMHO it would make sense to convert the existing network tracepoints dev->name into dev->ifindex, and then let perf-script convert this to the interface name. Either in userspace via if_indextoname(3), or (as ACME pointed out at the time) we might want to have a lookup table stored together with perf.data for later inspection (in-case ifindexes changed).