Skip to contentSkip to conversation

udp: Add tracepoint for udp_sendmsg()

PATCH In review Breno Leitao 6 participants 19 messages Apr 16, 2025 – May 27, 2025

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 udp: Add tracepoint for udp_sendmsg()

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

This implementation uses DECLARE_TRACE instead of TRACE_EVENT to avoid creating extensive trace event infrastructure and exporting to tracefs, keeping it minimal and efficient.

Since this patch creates a rawtracepoint, it can be accessed using standard tracing tools like bpftrace:

    rawtracepoint:udp_sendmsg_tp {
        ...
    }
---
 include/trace/events/udp.h | 5 +++++
 net/ipv4/udp.c             | 2 ++
 2 files changed, 7 insertions(+)
include/trace/events/udp.h +5 -0
index 6142be4068e29..38ab24053b6ff 100644
@@ -46,6 +46,11 @@ TRACE_EVENT(udp_fail_queue_rcv_skb,
 		  __entry->saddr, __entry->daddr)
 );
 
+DECLARE_TRACE(udp_sendmsg_tp,
+	TP_PROTO(const struct sock *sk, const struct msghdr *msg),
+	TP_ARGS(sk, msg)
+);
+
 #endif /* _TRACE_UDP_H */
 
 /* This part must be outside protection */
net/ipv4/udp.c +2 -0
index f9f5b92cf4b61..8c2902504a399 100644
@@ -1345,6 +1345,8 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		connected = 1;
 	}
 
+	trace_udp_sendmsg_tp(sk, msg);
+
 	ipcm_init_sk(&ipc, inet);
 	ipc.gso_size = READ_ONCE(up->gso_size);
 

---
base-commit: 1d6f4861027b451e064896f34dd0beada8871bfe
change-id: 20250416-udp_sendmsg-084a32657a56

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
maintainer
#1 lore raw
Quoted 9 lines · Breno Leitao

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

This implementation uses DECLARE_TRACE instead of TRACE_EVENT to avoid creating extensive trace event infrastructure and exporting to tracefs, keeping it minimal and efficient.

Quoted 6 lines

Since this patch creates a rawtracepoint, it can be accessed using standard tracing tools like bpftrace:

    rawtracepoint:udp_sendmsg_tp {
        ...
    }

What does this enable beyond kfunc:udp_sendmsg?

The arguments are the same, unlike the TCP tracepoint.

#2 lore raw

Hello Willem,

Quoted 10 lines · Willem de Bruijn
Quoted 9 lines · Breno Leitao

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

This implementation uses DECLARE_TRACE instead of TRACE_EVENT to avoid creating extensive trace event infrastructure and exporting to tracefs, keeping it minimal and efficient.

Quoted 8 lines
Quoted 6 lines

Since this patch creates a rawtracepoint, it can be accessed using standard tracing tools like bpftrace:

    rawtracepoint:udp_sendmsg_tp {
        ...
    }

What does this enable beyond kfunc:udp_sendmsg?

A few things come to mind when evaluating the use of tracepoints.

One significant advantage is that tracepoints provide a stable API where programs can hook into, making it easier for users to interact with key functions.

However, kfunc/kprobes has some notable disadvantages. For instance, partial or total inlining can cause hooks to fail, which is not ideal and can lead to issues (mainly when we have partial inlines, and the hook works _sometimes_).

In contrast, tracepoints create a more stable API for users to hook into, eliminating the need to patch the kernel with noinline function attributes. This solution may be ugly, but it is a common practice. (and this is my main goal with it, remove noinline from our internal kernel)

While tracepoints are not officially considered stable APIs, they tend to be "more stable" in practice due to their deliberate and strategic placement. As a result, they are less likely to get renamed or changed frequently.

Additionally, tracepoints offer performance benefits, being faster than both kfunc and kprobes.

For further discussion on this topic, please refer to same discussion in VFS:

https://lore.kernel.org/bpf/20250118033723.GV1977892@ZenIV/T/#m4c2fb2d904e839b34800daf8578dff0b9abd69a0

Thanks
--breno

maintainer
#3 lore raw
Quoted 24 lines · Breno Leitao

Hello Willem,

Quoted 10 lines · Willem de Bruijn
Quoted 9 lines · Breno Leitao

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

This implementation uses DECLARE_TRACE instead of TRACE_EVENT to avoid creating extensive trace event infrastructure and exporting to tracefs, keeping it minimal and efficient.

Quoted 8 lines
Quoted 6 lines

Since this patch creates a rawtracepoint, it can be accessed using standard tracing tools like bpftrace:

    rawtracepoint:udp_sendmsg_tp {
        ...
    }

What does this enable beyond kfunc:udp_sendmsg?

A few things come to mind when evaluating the use of tracepoints.

Quoted 8 lines

One significant advantage is that tracepoints provide a stable API where programs can hook into, making it easier for users to interact with key functions.

However, kfunc/kprobes has some notable disadvantages. For instance, partial or total inlining can cause hooks to fail, which is not ideal and can lead to issues (mainly when we have partial inlines, and the hook works _sometimes_).

As Paolo explained, that is unlikely to happen in this case, as this is a protocol specific callback function.

Quoted 6 lines

In contrast, tracepoints create a more stable API for users to hook into, eliminating the need to patch the kernel with noinline function attributes. This solution may be ugly, but it is a common practice. (and this is my main goal with it, remove noinline from our internal kernel)

Quoted 7 lines

While tracepoints are not officially considered stable APIs, they tend to be "more stable" in practice due to their deliberate and strategic placement. As a result, they are less likely to get renamed or changed frequently.

Additionally, tracepoints offer performance benefits, being faster than both kfunc and kprobes.

The performance argument is fair.

Perhaps we want to think this through more broadly for networking tracepoints vs more flexible kprobes/kfuncs, rather than on a case by case basis:

Where do we think the performance or functionality (if exposing additional info, as for tcp_sendmsg) warrants the tracepoint?

I suspect that the use is predominantly for on-demand debugging, where the performance cost (and latency impact) of measurement is minor.

Quoted 7 lines

For further discussion on this topic, please refer to same discussion in VFS:

https://lore.kernel.org/bpf/20250118033723.GV1977892@ZenIV/T/#m4c2fb2d904e839b34800daf8578dff0b9abd69a0

Thanks
--breno

#4 lore raw
In reply to Breno Leitao

net/ipv4/udp.c int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) · line 1348

Quoted 9 lines · Breno Leitao
net/ipv4/udp.c +1 -0
index f9f5b92cf4b61..8c2902504a399 100644
@@ -1345,6 +1345,8 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		connected = 1;
 	}
 
+	trace_udp_sendmsg_tp(sk, msg);

why _tp suffix? the usual naming is trace_${func}

#5 lore raw

Hello David,

Quoted 9 lines · David Ahern
Quoted 8 lines · Breno Leitao
net/ipv4/udp.c +0 -0
index f9f5b92cf4b61..8c2902504a399 100644
@@ -1345,6 +1345,8 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		connected = 1;
 	}
 
Quoted 3 lines

net/ipv4/udp.c int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) · line 1348

Quoted 1 line
net/ipv4/udp.c +1 -0
+	trace_udp_sendmsg_tp(sk, msg);

why _tp suffix? the usual naming is trace_${func}

I got the impression that DECLARE_TRACE() raw tracepoints names end up in _tp():

include/trace/events/tcp.h
DECLARE_TRACE(tcp_cwnd_reduction_tp,

include/trace/events/sched.h
DECLARE_TRACE(pelt_cfs_tp,
DECLARE_TRACE(pelt_rt_tp,
DECLARE_TRACE(pelt_dl_tp,
DECLARE_TRACE(pelt_hw_tp,
DECLARE_TRACE(pelt_irq_tp,
DECLARE_TRACE(pelt_se_tp,
DECLARE_TRACE(sched_cpu_capacity_tp,
DECLARE_TRACE(sched_overutilized_tp,
DECLARE_TRACE(sched_util_est_cfs_tp,
DECLARE_TRACE(sched_util_est_se_tp,
DECLARE_TRACE(sched_update_nr_running_tp,
DECLARE_TRACE(sched_compute_energy_tp,
DECLARE_TRACE(sched_entry_tp,
DECLARE_TRACE(sched_exit_tp,

But, I am happy to remove _tp if that is not correct.

Thanks,
--breno

#6 lore raw
Quoted 16 lines · Breno Leitao

Hello David,

Quoted 9 lines · David Ahern
Quoted 8 lines · Breno Leitao
net/ipv4/udp.c +0 -0
index f9f5b92cf4b61..8c2902504a399 100644
@@ -1345,6 +1345,8 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		connected = 1;
 	}
 
Quoted 3 lines

net/ipv4/udp.c int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) · line 1348

Quoted 1 line
net/ipv4/udp.c +1 -0
+	trace_udp_sendmsg_tp(sk, msg);

why _tp suffix? the usual naming is trace_${func}

Quoted 5 lines

I got the impression that DECLARE_TRACE() raw tracepoints names end up in _tp():

include/trace/events/tcp.h
DECLARE_TRACE(tcp_cwnd_reduction_tp,

that is the only networking one:

$ git grep trace_ net drivers/net | grep _tp
net/bpf/test_run.c:	trace_bpf_trigger_tp(nonce);
net/ipv4/tcp_input.c:	trace_tcp_cwnd_reduction_tp(sk,
newly_acked_sacked, newly_lost, flag);

The rest follow do not have the _tp suffix:

$ git grep trace_ net drivers/net | wc -l
    2727

2725 without; 2 with

#7 lore raw
Quoted 23 lines · David Ahern
Quoted 16 lines · Breno Leitao

Hello David,

Quoted 9 lines · David Ahern
Quoted 8 lines · Breno Leitao
net/ipv4/udp.c +0 -0
index f9f5b92cf4b61..8c2902504a399 100644
@@ -1345,6 +1345,8 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		connected = 1;
 	}
 
Quoted 3 lines

net/ipv4/udp.c int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) · line 1348

Quoted 1 line
net/ipv4/udp.c +1 -0
+	trace_udp_sendmsg_tp(sk, msg);

why _tp suffix? the usual naming is trace_${func}

Quoted 5 lines

I got the impression that DECLARE_TRACE() raw tracepoints names end up in _tp():

include/trace/events/tcp.h
DECLARE_TRACE(tcp_cwnd_reduction_tp,

Quoted 6 lines

that is the only networking one:

$ git grep trace_ net drivers/net | grep _tp
net/bpf/test_run.c:	trace_bpf_trigger_tp(nonce);
net/ipv4/tcp_input.c:	trace_tcp_cwnd_reduction_tp(sk,
newly_acked_sacked, newly_lost, flag);

Do we want to rename them and remove the _tp? I suppose it is OK given that tracepoints are not expected to be stable?

Also, if we have consensus about this patch, I will remove the _tp from it.

Thanks!
--breno

#8 lore raw
Quoted 11 lines · Breno Leitao
Quoted 4 lines
$ git grep trace_ net drivers/net | grep _tp
net/bpf/test_run.c:	trace_bpf_trigger_tp(nonce);
net/ipv4/tcp_input.c:	trace_tcp_cwnd_reduction_tp(sk,
newly_acked_sacked, newly_lost, flag);

Do we want to rename them and remove the _tp? I suppose it is OK given that tracepoints are not expected to be stable?

Also, if we have consensus about this patch, I will remove the _tp from it.

I am only asking for consistency. Based on existing networking instances, consistency is no _tp suffix.

maintainer
#9 lore raw
Quoted 9 lines · On Thu, 17 Apr 2025 21:57:56 -0700 David Ahern <[email protected]> wrote:
Quoted 8 lines · Breno Leitao
Quoted 4 lines
$ git grep trace_ net drivers/net | grep _tp
net/bpf/test_run.c:	trace_bpf_trigger_tp(nonce);
net/ipv4/tcp_input.c:	trace_tcp_cwnd_reduction_tp(sk,
newly_acked_sacked, newly_lost, flag);  

Do we want to rename them and remove the _tp? I suppose it is OK given that tracepoints are not expected to be stable?

Quoted 6 lines
Quoted 3 lines

Also, if we have consensus about this patch, I will remove the _tp from it.

I am only asking for consistency. Based on existing networking instances, consistency is no _tp suffix.

I was looking at what other tracepoints have "_tp" and found a few. What it appears to be is that the "_tp" tracepoints are defined by:

DECLARE_TRACEPOINT()

and have no corresponding trace event in tracefs (/sys/kernel/tracing/events).

I like that distinction because it lets the developer know that this tracepoint is in kernel only, and not exposed to user space.

Perhaps it should stay as "_tp()" if it's not exposed via tracefs.

In fact, if there is a clean up, it should be adding "_tp" to all tracepoints that do not have a corresponding trace event attached to them. As they are in kernel only, that change should not cause any ABI breakage.

-- Steve

maintainer
#10 lore raw
Quoted 3 lines · On Fri, 18 Apr 2025 08:33:51 -0400 Steven Rostedt <[email protected]> wrote:

In fact, if there is a clean up, it should be adding "_tp" to all tracepoints that do not have a corresponding trace event attached to them. As they are in kernel only, that change should not cause any ABI breakage.

Actually, I think I'll make it where tracepoints created via DECLARE_TRACE() will automatically get the "_tp" ending. That would help enforce this API.

-- Steve

maintainer
#11 lore raw
In reply to Breno Leitao
Quoted 4 lines · Breno Leitao

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

Why is it needed? what would add on top of a plain perf probe, which will be always available for such function with such argument, as the function can't be inlined?

Thanks,

Paolo

#12 lore raw

Hello Paolo,

Quoted 9 lines · Paolo Abeni
Quoted 4 lines · Breno Leitao

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

Why is it needed? what would add on top of a plain perf probe, which will be always available for such function with such argument, as the function can't be inlined?

Why this function can't be inlined? I got the impression that this funciton could be, at least, partially inlined. Mainly when generating ultra optimized kernels (i.e, kernels compiled with PGO and LTO features enabled).

Thanks,
--breno

maintainer
#13 lore raw
Quoted 7 lines · Breno Leitao
Quoted 6 lines · Paolo Abeni
Quoted 4 lines · Breno Leitao

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

Quoted 5 lines
Quoted 3 lines

Why is it needed? what would add on top of a plain perf probe, which will be always available for such function with such argument, as the function can't be inlined?

Why this function can't be inlined?

Because the kernel need to be able find a pointer to it:

.sendmsg = udp_sendmsg,

I'll be really curious to learn how the compiler could inline that.

/P

#14 lore raw

Hi Paolo,

Quoted 14 lines · Paolo Abeni
Quoted 11 lines · Breno Leitao
Quoted 9 lines · Paolo Abeni
Quoted 4 lines · Breno Leitao

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

Why is it needed? what would add on top of a plain perf probe, which will be always available for such function with such argument, as the function can't be inlined?

Quoted 7 lines
Quoted 1 line

Why this function can't be inlined?

Because the kernel need to be able find a pointer to it:

.sendmsg = udp_sendmsg,

I'll be really curious to learn how the compiler could inline that.

It is true that functions that are only used via function pointers will not be inlined by compilers (at least for those we have tested). For this reason, we do not worry about functions in various tcp_congestion_ops. However, udp_sendmsg is also called directly by udpv6_sendmsg, so it can still get inlined by LTO.

Thanks,
Song

maintainer
#15 lore raw
Quoted 24 lines · Song Liu

Hi Paolo,

Quoted 14 lines · Paolo Abeni
Quoted 11 lines · Breno Leitao
Quoted 9 lines · Paolo Abeni
Quoted 4 lines · Breno Leitao

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

Why is it needed? what would add on top of a plain perf probe, which will be always available for such function with such argument, as the function can't be inlined?

Quoted 7 lines
Quoted 1 line

Why this function can't be inlined?

Because the kernel need to be able find a pointer to it:

.sendmsg = udp_sendmsg,

I'll be really curious to learn how the compiler could inline that.

Quoted 9 lines

It is true that functions that are only used via function pointers will not be inlined by compilers (at least for those we have tested). For this reason, we do not worry about functions in various tcp_congestion_ops. However, udp_sendmsg is also called directly by udpv6_sendmsg, so it can still get inlined by LTO.

Thanks,
Song

I would think that hitting this tracepoint for ipv6_addr_v4mapped addresses is unintentional and surprising, as those would already hit udpv6_sendmsg.

On which note, any IPv4 change to UDP needs an equivalent IPv6 one.

#16 lore raw
Quoted 35 lines · Willem de Bruijn
Quoted 30 lines · Song Liu

Hi Paolo,

Quoted 14 lines · Paolo Abeni
Quoted 11 lines · Breno Leitao
Quoted 9 lines · Paolo Abeni
Quoted 4 lines · Breno Leitao

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

Why is it needed? what would add on top of a plain perf probe, which will be always available for such function with such argument, as the function can't be inlined?

Quoted 7 lines
Quoted 1 line

Why this function can't be inlined?

Because the kernel need to be able find a pointer to it:

.sendmsg = udp_sendmsg,

I'll be really curious to learn how the compiler could inline that.

It is true that functions that are only used via function pointers will not be inlined by compilers (at least for those we have tested). For this reason, we do not worry about functions in various tcp_congestion_ops. However, udp_sendmsg is also called directly by udpv6_sendmsg, so it can still get inlined by LTO.

Quoted 7 lines
Quoted 3 lines

Thanks,
Song

I would think that hitting this tracepoint for ipv6_addr_v4mapped addresses is unintentional and surprising, as those would already hit udpv6_sendmsg.

It is up to the user to decide how these tracepoints should be used. For example, the user may only be interested in udpv6_sendmsg => udp_sendmsg case. Without a tracepoint, the user has to understand whether the compiler inlined this function.

Quoted 2 lines

On which note, any IPv4 change to UDP needs an equivalent IPv6 one.

Do you mean we need to also add tracepoints for udpv6_sendmsg?

Thanks,
Song

maintainer
#17 lore raw
Quoted 49 lines · Song Liu
Quoted 35 lines · Willem de Bruijn
Quoted 30 lines · Song Liu

Hi Paolo,

Quoted 14 lines · Paolo Abeni
Quoted 11 lines · Breno Leitao
Quoted 9 lines · Paolo Abeni
Quoted 4 lines · Breno Leitao

Add a lightweight tracepoint to monitor UDP send message operations, similar to the recently introduced tcp_sendmsg_locked() trace event in commit 0f08335ade712 ("trace: tcp: Add tracepoint for tcp_sendmsg_locked()")

Why is it needed? what would add on top of a plain perf probe, which will be always available for such function with such argument, as the function can't be inlined?

Quoted 7 lines
Quoted 1 line

Why this function can't be inlined?

Because the kernel need to be able find a pointer to it:

.sendmsg = udp_sendmsg,

I'll be really curious to learn how the compiler could inline that.

It is true that functions that are only used via function pointers will not be inlined by compilers (at least for those we have tested). For this reason, we do not worry about functions in various tcp_congestion_ops. However, udp_sendmsg is also called directly by udpv6_sendmsg, so it can still get inlined by LTO.

Quoted 7 lines
Quoted 3 lines

Thanks,
Song

I would think that hitting this tracepoint for ipv6_addr_v4mapped addresses is unintentional and surprising, as those would already hit udpv6_sendmsg.

It is up to the user to decide how these tracepoints should be used. For example, the user may only be interested in udpv6_sendmsg => udp_sendmsg case. Without a tracepoint, the user has to understand whether the compiler inlined this function.

Quoted 1 line
Quoted 3 lines
Quoted 1 line

On which note, any IPv4 change to UDP needs an equivalent IPv6 one.

Do you mean we need to also add tracepoints for udpv6_sendmsg?

If there is consensus that a tracepoint at this point is valuable, then it should be supported equally for IPv4 and IPv6.

That holds true for all such hooks. No IPv4 only.

#18 lore raw

Hello,

Quoted 7 lines · Willem de Bruijn
Quoted 1 line · Song Liu

Do you mean we need to also add tracepoints for udpv6_sendmsg?

If there is consensus that a tracepoint at this point is valuable, then it should be supported equally for IPv4 and IPv6.

That holds true for all such hooks. No IPv4 only.

Revamping this thread to make sure we have a conclusion.

Any objection for not having the tracepoint in UDP sendmsg, both IPv4 and IPv6?

Thanks
--breno