io_uring cmd for tx timestamps

Add a helper function skb_get_tx_timestamp() that returns a tx timestamp associated with an skb from an queue queue.

Signed-off-by Pavel Begunkov <[email protected]>

---
 include/net/sock.h |  9 +++++++++
 net/socket.c       | 45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)

include/net/sock.h +9 -0

index 92e7c1aae3cc..0b96196d8a34 100644
@@ -2677,6 +2677,15 @@ void __sock_recv_timestamp(struct msghdr *msg, struct sock *sk,
 void __sock_recv_wifi_status(struct msghdr *msg, struct sock *sk,
 			     struct sk_buff *skb);
 
+enum {
+	NET_TIMESTAMP_ORIGIN_SW		= 0,
+	NET_TIMESTAMP_ORIGIN_HW		= 1,
+};
+
+bool skb_has_tx_timestamp(struct sk_buff *skb, const struct sock *sk);
+int skb_get_tx_timestamp(struct sk_buff *skb, struct sock *sk,
+			 struct timespec64 *ts);
+
 static inline void
 sock_recv_timestamp(struct msghdr *msg, struct sock *sk, struct sk_buff *skb)
 {

net/socket.c +32 -0

index 9a0e720f0859..9bb618c32d65 100644
@@ -843,6 +843,51 @@ static void put_ts_pktinfo(struct msghdr *msg, struct sk_buff *skb,
 		 sizeof(ts_pktinfo), &ts_pktinfo);
 }
 
+bool skb_has_tx_timestamp(struct sk_buff *skb, const struct sock *sk)
+{
+	const struct sock_exterr_skb *serr = SKB_EXT_ERR(skb);
+	u32 tsflags = READ_ONCE(sk->sk_tsflags);
+
+	if (serr->ee.ee_errno != ENOMSG ||
+	   serr->ee.ee_origin != SO_EE_ORIGIN_TIMESTAMPING)
+		return false;
+
+	/* software time stamp available and wanted */
+	if ((tsflags & SOF_TIMESTAMPING_SOFTWARE) && skb->tstamp)
+		return true;
+	/* hardware time stamps available and wanted */
+	return (tsflags & SOF_TIMESTAMPING_RAW_HARDWARE) &&
+		skb_hwtstamps(skb)->hwtstamp;
+}
+
+int skb_get_tx_timestamp(struct sk_buff *skb, struct sock *sk,
+			  struct timespec64 *ts)
+{
+	u32 tsflags = READ_ONCE(sk->sk_tsflags);
+	ktime_t hwtstamp;
+	int if_index = 0;
+
+	if ((tsflags & SOF_TIMESTAMPING_SOFTWARE) &&
+	    ktime_to_timespec64_cond(skb->tstamp, ts))
+		return NET_TIMESTAMP_ORIGIN_SW;
+
+	if (!(tsflags & SOF_TIMESTAMPING_RAW_HARDWARE) ||
+	    skb_is_swtx_tstamp(skb, false))
+		return -ENOENT;
+

net/socket.c +3 -0

+	return NET_TIMESTAMP_ORIGIN_HW;
+}
+
 /*
  * called from sock_recv_timestamp() if sock_flag(sk, SOCK_RCVTSTAMP)
  */

Signature · 1 line

2.49.0

Add a new socket command which returns tx time stamps to the user. It provide an alternative to the existing error queue recvmsg interface. The command works in a polled multishot mode, which means io_uring will poll the socket and keep posting timestamps until the request is cancelled or fails in any other way (e.g. with no space in the CQ). It reuses the net infra and grabs timestamps from the socket's error queue.

The command requires IORING_SETUP_CQE32. All non-final CQEs (marked with IORING_CQE_F_MORE) have cqe->res set to the tskey, and the upper 16 bits of cqe->flags keep tstype (i.e. offset by IORING_CQE_BUFFER_SHIFT). The timevalue is store in the upper part of the extended CQE. The final completion won't have IORING_CQR_F_MORE and will have cqe->res storing

io_uring/cmd_net.c +22 -0

index e99170c7d41a..bc2d33ea2db3 100644
@@ -1,5 +1,6 @@
 #include <asm/ioctls.h>
 #include <linux/io_uring/net.h>
+#include <linux/errqueue.h>
 #include <net/sock.h>
 
 #include "uring_cmd.h"

@@ -51,6 +52,85 @@ static inline int io_uring_cmd_setsockopt(struct socket *sock,
 				  optlen);
 }
 
+static bool io_process_timestamp_skb(struct io_uring_cmd *cmd, struct sock *sk,
+				     struct sk_buff *skb, unsigned issue_flags)
+{
+	struct sock_exterr_skb *serr = SKB_EXT_ERR(skb);
+	struct io_uring_cqe cqe[2];
+	struct io_timespec *iots;
+	struct timespec64 ts;
+	u32 tstype, tskey;
+	int ret;
+
+	BUILD_BUG_ON(sizeof(struct io_uring_cqe) != sizeof(struct io_timespec));
+
+	ret = skb_get_tx_timestamp(skb, sk, &ts);
+	if (ret < 0)
+		return false;
+
+	tskey = serr->ee.ee_data;
+	tstype = serr->ee.ee_info;
+
+	cqe->user_data = 0;
+	cqe->res = tskey;

Quoted 7 lines · Pavel Begunkov

Add a new socket command which returns tx time stamps to the user. It provide an alternative to the existing error queue recvmsg interface. The command works in a polled multishot mode, which means io_uring will poll the socket and keep posting timestamps until the request is cancelled or fails in any other way (e.g. with no space in the CQ). It reuses the net infra and grabs timestamps from the socket's error queue.

Quoted 5 lines

The command requires IORING_SETUP_CQE32. All non-final CQEs (marked with IORING_CQE_F_MORE) have cqe->res set to the tskey, and the upper 16 bits of cqe->flags keep tstype (i.e. offset by IORING_CQE_BUFFER_SHIFT). The timevalue is store in the upper part of the extended CQE. The final completion won't have IORING_CQR_F_MORE and will have cqe->res storing

                         ^^^^^^^^^^^^^^^^^

Pointed this out before, but this typo is still there.

Quoted 12 lines

include/uapi/linux/io_uring.h +3 -0

index cfd17e382082..5c89e6f6d624 100644
@@ -968,6 +968,15 @@ enum io_uring_socket_op {
  	SOCKET_URING_OP_SIOCOUTQ,
  	SOCKET_URING_OP_GETSOCKOPT,
  	SOCKET_URING_OP_SETSOCKOPT,
+	SOCKET_URING_OP_TX_TIMESTAMP,
+};
+

include/uapi/linux/io_uring.h enum io_uring_socket_op { · line 975

Quoted 2 lines

include/uapi/linux/io_uring.h +2 -0

+#define IORING_CQE_F_TIMESTAMP_HW	((__u32)1 << IORING_CQE_BUFFER_SHIFT)
+#define IORING_TIMESTAMP_TSTYPE_SHIFT	(IORING_CQE_BUFFER_SHIFT + 1)

Don't completely follow this, would at the very least need a comment. Whether it's a HW or SW timestamp is flagged in the upper 16 bits, just like a provided buffer ID. But since we don't use buffer IDs here, then it's up for grabs. Do we have other commands that use the upper flags space for command private flags?

The above makes sense, but then what is IORING_TIMESTAMP_TSTYPE_SHIFT?

Quoted 11 lines

Quoted 11 lines

include/uapi/linux/io_uring.h +3 -0

index cfd17e382082..5c89e6f6d624 100644
@@ -968,6 +968,15 @@ enum io_uring_socket_op {
      SOCKET_URING_OP_SIOCOUTQ,
      SOCKET_URING_OP_GETSOCKOPT,
      SOCKET_URING_OP_SETSOCKOPT,
+    SOCKET_URING_OP_TX_TIMESTAMP,
+};
+

Quoted 8 lines

include/uapi/linux/io_uring.h enum io_uring_socket_op { · line 975

Quoted 2 lines

include/uapi/linux/io_uring.h +2 -0

+#define IORING_CQE_F_TIMESTAMP_HW    ((__u32)1 << IORING_CQE_BUFFER_SHIFT)
+#define IORING_TIMESTAMP_TSTYPE_SHIFT    (IORING_CQE_BUFFER_SHIFT + 1)

Don't completely follow this, would at the very least need a comment. Whether it's a HW or SW timestamp is flagged in the upper 16 bits, just like a provided buffer ID. But since we don't use buffer IDs here, then it's up for grabs. Do we have other commands that use the upper flags space for command private flags?

Probably not, but the place is better than the lower half, which has common flags like F_MORE, especially since the patch is already using it to store the type.

Quoted 1 line

The above makes sense, but then what is IORING_TIMESTAMP_TSTYPE_SHIFT?

It's a shift for where the timestamp type is stored, HW vs SW is not a timestamp type. I don't get the question.

Quoted 20 lines · Pavel Begunkov

Quoted 19 lines

include/uapi/linux/io_uring.h enum io_uring_socket_op { · line 975

Quoted 13 lines

include/uapi/linux/io_uring.h +5 -0

index cfd17e382082..5c89e6f6d624 100644
@@ -968,6 +968,15 @@ enum io_uring_socket_op {
       SOCKET_URING_OP_SIOCOUTQ,
       SOCKET_URING_OP_GETSOCKOPT,
       SOCKET_URING_OP_SETSOCKOPT,
+    SOCKET_URING_OP_TX_TIMESTAMP,
+};
+
+#define IORING_CQE_F_TIMESTAMP_HW    ((__u32)1 << IORING_CQE_BUFFER_SHIFT)
+#define IORING_TIMESTAMP_TSTYPE_SHIFT    (IORING_CQE_BUFFER_SHIFT + 1)

Don't completely follow this, would at the very least need a comment. Whether it's a HW or SW timestamp is flagged in the upper 16 bits, just like a provided buffer ID. But since we don't use buffer IDs here, then it's up for grabs. Do we have other commands that use the upper flags space for command private flags?

Quoted 3 lines

Probably not, but the place is better than the lower half, which has common flags like F_MORE, especially since the patch is already using it to store the type.

Just pondering whether it should be formalized, but probably no point as each opcode should be free to use the space as it wants.

Quoted 4 lines

Quoted 1 line

The above makes sense, but then what is IORING_TIMESTAMP_TSTYPE_SHIFT?

It's a shift for where the timestamp type is stored, HW vs SW is not a timestamp type. I don't get the question.

Please add a spec like comment on top of it explaining the usage of the upper bits in the flags field, then. I try to keep the io_uring.h uapi header pretty well commented and documented.

Add a new socket command which returns tx time stamps to the user. It provide an alternative to the existing error queue recvmsg interface. The command works in a polled multishot mode, which means io_uring will poll the socket and keep posting timestamps until the request is cancelled or fails in any other way (e.g. with no space in the CQ). It reuses the net infra and grabs timestamps from the socket's error queue.

The command requires IORING_SETUP_CQE32. All non-final CQEs (marked with IORING_CQE_F_MORE) have cqe->res set to the tskey, and the upper 16 bits of cqe->flags keep tstype (i.e. offset by IORING_CQE_BUFFER_SHIFT). The timevalue is store in the upper part of the extended CQE. The final completion won't have IORING_CQR_F_MORE and will have cqe->res storing 0/error.

Suggested-by Vadim Fedorenko <[email protected]>

Signed-off-by Pavel Begunkov <[email protected]>

---
 include/uapi/linux/io_uring.h |  9 ++++
 io_uring/cmd_net.c            | 82 +++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+)

include/uapi/linux/io_uring.h +0 -0

index cfd17e382082..5c89e6f6d624 100644

include/uapi/linux/io_uring.h +3 -0

+struct io_timespec {
+	__u64		tv_sec;
+	__u64		tv_nsec;
 };
 
 /* Zero copy receive refill queue entry */

io_uring/cmd_net.c +82 -0

index e99170c7d41a..bc2d33ea2db3 100644
@@ -1,5 +1,6 @@
 #include <asm/ioctls.h>
 #include <linux/io_uring/net.h>
+#include <linux/errqueue.h>
 #include <net/sock.h>
 
 #include "uring_cmd.h"

@@ -51,6 +52,85 @@ static inline int io_uring_cmd_setsockopt(struct socket *sock,
 				  optlen);
 }
 
+static bool io_process_timestamp_skb(struct io_uring_cmd *cmd, struct sock *sk,
+				     struct sk_buff *skb, unsigned issue_flags)
+{
+	struct sock_exterr_skb *serr = SKB_EXT_ERR(skb);
+	struct io_uring_cqe cqe[2];
+	struct io_timespec *iots;
+	struct timespec64 ts;
+	u32 tstype, tskey;
+	int ret;
+
+	BUILD_BUG_ON(sizeof(struct io_uring_cqe) != sizeof(struct io_timespec));
+
+	ret = skb_get_tx_timestamp(skb, sk, &ts);
+	if (ret < 0)
+		return false;
+
+	tskey = serr->ee.ee_data;
+	tstype = serr->ee.ee_info;
+
+	cqe->user_data = 0;
+	cqe->res = tskey;
+	cqe->flags = IORING_CQE_F_MORE;
+	cqe->flags |= tstype << IORING_TIMESTAMP_TSTYPE_SHIFT;
+	if (ret == NET_TIMESTAMP_ORIGIN_HW)
+		cqe->flags |= IORING_CQE_F_TIMESTAMP_HW;
+
+	iots = (struct io_timespec *)&cqe[1];
+	iots->tv_sec = ts.tv_sec;
+	iots->tv_nsec = ts.tv_nsec;
+	return io_uring_cmd_post_mshot_cqe32(cmd, issue_flags, cqe);
+}
+
+static int io_uring_cmd_timestamp(struct socket *sock,
+				  struct io_uring_cmd *cmd,
+				  unsigned int issue_flags)
+{
+	struct sock *sk = sock->sk;
+	struct sk_buff_head *q = &sk->sk_error_queue;
+	struct sk_buff *skb, *tmp;
+	struct sk_buff_head list;
+	int ret;
+
+	if (!(issue_flags & IO_URING_F_CQE32))
+		return -EINVAL;
+	ret = io_cmd_poll_multishot(cmd, issue_flags, EPOLLERR);
+	if (unlikely(ret))
+		return ret;
+
+	if (skb_queue_empty_lockless(q))
+		return -EAGAIN;
+	__skb_queue_head_init(&list);
+
+	scoped_guard(spinlock_irq, &q->lock) {
+		skb_queue_walk_safe(q, skb, tmp) {
+			/* don't support skbs with payload */
+			if (!skb_has_tx_timestamp(skb, sk) || skb->len)
+				continue;
+			__skb_unlink(skb, q);
+			__skb_queue_tail(&list, skb);
+		}
+	}
+
+	while (1) {
+		skb = skb_peek(&list);
+		if (!skb)
+			break;
+		if (!io_process_timestamp_skb(cmd, sk, skb, issue_flags))
+			break;
+		__skb_dequeue(&list);
+		consume_skb(skb);
+	}
+
+	if (!unlikely(skb_queue_empty(&list))) {
+		scoped_guard(spinlock_irqsave, &q->lock)
+			skb_queue_splice(q, &list);
+	}
+	return -EAGAIN;
+}
+
 int io_uring_cmd_sock(struct io_uring_cmd *cmd, unsigned int issue_flags)
 {
 	struct socket *sock = cmd->file->private_data;

@@ -76,6 +156,8 @@ int io_uring_cmd_sock(struct io_uring_cmd *cmd, unsigned int issue_flags)
 		return io_uring_cmd_getsockopt(sock, cmd, issue_flags);
 	case SOCKET_URING_OP_SETSOCKOPT:
 		return io_uring_cmd_setsockopt(sock, cmd, issue_flags);
+	case SOCKET_URING_OP_TX_TIMESTAMP:
+		return io_uring_cmd_timestamp(sock, cmd, issue_flags);
 	default:
 		return -EOPNOTSUPP;
 	}

Signature · 1 line

2.49.0

Quoted 30 lines · Pavel Begunkov

Add a new socket command which returns tx time stamps to the user. It provide an alternative to the existing error queue recvmsg interface. The command works in a polled multishot mode, which means io_uring will poll the socket and keep posting timestamps until the request is cancelled or fails in any other way (e.g. with no space in the CQ). It reuses the net infra and grabs timestamps from the socket's error queue.

The command requires IORING_SETUP_CQE32. All non-final CQEs (marked with IORING_CQE_F_MORE) have cqe->res set to the tskey, and the upper 16 bits of cqe->flags keep tstype (i.e. offset by IORING_CQE_BUFFER_SHIFT). The timevalue is store in the upper part of the extended CQE. The final completion won't have IORING_CQR_F_MORE and will have cqe->res storing 0/error.

Suggested-by Vadim Fedorenko <[email protected]>

Signed-off-by Pavel Begunkov <[email protected]>

---
  include/uapi/linux/io_uring.h |  9 ++++
  io_uring/cmd_net.c            | 82 +++++++++++++++++++++++++++++++++++
  2 files changed, 91 insertions(+)

include/uapi/linux/io_uring.h +1 -0

index cfd17e382082..5c89e6f6d624 100644
@@ -968,6 +968,15 @@ enum io_uring_socket_op {
  	SOCKET_URING_OP_SIOCOUTQ,
  	SOCKET_URING_OP_GETSOCKOPT,
  	SOCKET_URING_OP_SETSOCKOPT,
+	SOCKET_URING_OP_TX_TIMESTAMP,

include/uapi/linux/io_uring.h enum io_uring_socket_op { · line 976

Quoted 5 lines

include/uapi/linux/io_uring.h +5 -0

+};
+
+#define IORING_CQE_F_TIMESTAMP_HW	((__u32)1 << IORING_CQE_BUFFER_SHIFT)
+#define IORING_TIMESTAMP_TSTYPE_SHIFT	(IORING_CQE_BUFFER_SHIFT + 1)
+

Perhaps instead of these shifts define an actual struct, e.g., io_uring_cqe_tstamp.

Vadim Fedorenko suggested to add an alternative API for receiving tx timestamps through io_uring. The series introduces io_uring socket cmd for fetching tx timestamps, which is a polled multishot request, i.e. internally polling the socket for POLLERR and posts timestamps when they're arrives. For the API description see Patch 5.

It reuses existing timestamp infra and takes them from the socket's error queue. For networking people the important parts are Patch 1, and io_uring_cmd_timestamp() from Patch 5 walking the error queue.

It should be reasonable to take it through the io_uring tree once we have consensus, but let me know if there are any concerns.

v3: Add a flag to distinguish sw vs hw timestamp. skb_get_tx_timestamp()
from Patch 1 now returns the indication of that, and in Patch 5 it's converted into a io_uring CQE bit flag.