Skip to contentSkip to conversation

Add `sendto(2)` and `recvfrom(2)` support

PATCH In review Ammar Faizi 2 participants 6 messages Jan 29, 2022

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover Add `sendto(2)` and `recvfrom(2)` support

Hello,

This patchset adds sendto(2) and recvfrom(2) support for io_uring. It also addresses an issue in the liburing GitHub repository [1].

## Motivations:

1) By using sendto() and recvfrom() we can make the submission simpler compared to always using sendmsg() and recvmsg() from the userspace. Especially for UDP socket.

2) There is a historical patch that tried to add the same functionality, but did not end up being applied. [2]

Quoted 6 lines · Jens Axboe

In a private conversation with the author, a good point was brought up that the sendto/recvfrom do not require an allocation of an async context, if we need to defer or go async with the request. I think that's a major win, to be honest. There are other benefits as well (like shorter path), but to me, the async less part is nice and will reduce overhead

## Changes summary:

1) Rename io_{send,recv} to io_{sendto,recvfrom}
_____________________________________________________
The following call

send(sockfd, buf, len, flags);

is equivalent to

sendto(sockfd, buf, len, flags, NULL, 0);
_____________________________________________________
The following call

recv(sockfd, buf, len, flags);

is equivalent to

recvfrom(sockfd, buf, len, flags, NULL, NULL);
_____________________________________________________

Currently, io_uring supports send() and recv() operation. Now, we are going to add sendto() and recvfrom() support. Since the latter is the superset of the former, change the function name to the latter.

2) Make move_addr_to_user() be a non static function. This is required for adding recvfrom() support for io_uring.

3) Add sendto(2) and recvfrom(2) support.

New opcodes:
- IORING_OP_SENDTO
- IORING_OP_RECVFROM

## How to apply

This work is based on Jens' tree, branch "io-uring-5.17".

The following changes since commit f6133fbd373811066c8441737e65f384c8f31974:

io_uring: remove unused argument from io_rsrc_node_alloc (2022-01-27 10:18:53 -0700)

are available in the Git repository at:

git://github.com/ammarfaizi2/linux-block.git tags/io_uring-sendto-recvfrom.v1

for you to fetch changes up to 68d110c39241b887ec388cd3316dbedb85b0cbcf:

io_uring: Add sendto(2) and recvfrom(2) support (2022-01-29 13:08:13 +0700)

```
----------------------------------------------------------------
io_uring-sendto-recvfrom.v1

----------------------------------------------------------------
```

## liburing support and test program:

git://github.com/ammarfaizi2/liburing.git tags/sendto-recvfrom.v1-2022-01-29

Run the test program from liburing:

```
    ./configure;
    make -j$(nproc);
    test/sendto_recvfrom;
    echo $?;
```

## Changelog:

v1:
- Rebase the work (sync with "io_uring-5.17" branch in Jens' tree).
- Add BUILD_BUG_SQE_ELEM(48, __u64, addr3); for compile time assertion.
- Reword the commit messages.
- Add Alviro Iskandar Setiawan to CC list (tester).

RFC v4:
- Rebase the work (sync with "for-next" branch in Jens' tree).
- Remove Tested-by tag from Nugra as this patch changes.
- (Address Praveen's comment) Zero sendto_addr_len and recvfrom_addr_len on prep when the req->opcode is not IORING_OP_{SENDTO,RECVFROM}.

RFC v3:
- Fix build error when CONFIG_NET is undefined for PATCH 1/3. I tried to fix it in PATCH 3/3, but it should be fixed in PATCH 1/3, otherwise it breaks the build in PATCH 1/3.
- Add io_uring_prep_{sendto,recvfrom} docs to the liburing.

RFC v2:
- Rebase the work, now this patchset is based on commit bb3294e22482db4b7ec ("Merge branch 'for-5.17/drivers' into for-next").
- In io_recvfrom(), mark the error check of move_addr_to_user() call as unlikely.
- Fix build error when CONFIG_NET is undefined.
- Update liburing test (the branch is still the same, just force pushed).
- Add Nugra to CC list (tester).

## Refs
[1]: https://github.com/axboe/liburing/issues/397
[2]: https://lore.kernel.org/io-uring/[email protected]/

---
RFC v4: https://lore.kernel.org/io-uring/[email protected]/
RFC v3: https://lore.kernel.org/io-uring/[email protected]/
RFC v2: https://lore.kernel.org/io-uring/[email protected]/
RFC v1: https://lore.kernel.org/io-uring/[email protected]/

---
Ammar Faizi (3):
io_uring: Rename io_{send,recv} to io_{sendto,recvfrom}
net: Make move_addr_to_user() be a non static function
io_uring: Add sendto(2) and recvfrom(2) support

 fs/io_uring.c                 | 95 +++++++++++++++++++++++++++++++----
 include/linux/socket.h        |  2 +
 include/uapi/linux/io_uring.h |  5 +-
 net/socket.c                  |  4 +-
 4 files changed, 93 insertions(+), 13 deletions(-)

base-commit: f6133fbd373811066c8441737e65f384c8f31974

Signature · 1 line
Ammar Faizi
#1 lore raw

PATCH 1/3 io_uring: Rename `io_{send,recv}` to `io_{sendto,recvfrom}`

This is a preparation to add sendto() and recvfrom() support for io_uring.
_____________________________________________________
The following call

    send(sockfd, buf, len, flags);

is equivalent to

    sendto(sockfd, buf, len, flags, NULL, 0);

_____________________________________________________
The following call

    recv(sockfd, buf, len, flags);

is equivalent to

    recvfrom(sockfd, buf, len, flags, NULL, NULL);

_____________________________________________________

Currently, io_uring supports send() and recv() operation. Now, we are going to add sendto() and recvfrom() support. Since the latter is the superset of the former, change the function name to the latter.

This renames:
- io_send() to io_sendto()
- io_recv() to io_recvfrom()

Cc Alviro Iskandar Setiawan <[email protected]>

---
v1:
- Rebase the work (sync with "io_uring-5.17" branch in Jens' tree).
- Reword the commit message.
- Add Alviro Iskandar Setiawan to CC list (tester).

RFC v4:
- Rebase the work (sync with "for-next" branch in Jens' tree).

RFC v3:
- Fix build error when CONFIG_NET is undefined for PATCH 1/3. I tried to fix it in PATCH 3/3, but it should be fixed in PATCH 1/3, otherwise it breaks the build in PATCH 1/3.

RFC v2:
- Add Nugra to CC list (tester).

---

---
 fs/io_uring.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
fs/io_uring.c +6 -6
index 2e04f718319d..742e252a052a 100644
@@ -4961,7 +4961,7 @@ static int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags)
 	return 0;
 }
 
-static int io_send(struct io_kiocb *req, unsigned int issue_flags)
+static int io_sendto(struct io_kiocb *req, unsigned int issue_flags)
 {
 	struct io_sr_msg *sr = &req->sr_msg;
 	struct msghdr msg;
@@ -5187,7 +5187,7 @@ static int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags)
 	return 0;
 }
 
-static int io_recv(struct io_kiocb *req, unsigned int issue_flags)
+static int io_recvfrom(struct io_kiocb *req, unsigned int issue_flags)
 {
 	struct io_buffer *kbuf;
 	struct io_sr_msg *sr = &req->sr_msg;
@@ -5395,8 +5395,8 @@ IO_NETOP_PREP_ASYNC(sendmsg);
 IO_NETOP_PREP_ASYNC(recvmsg);
 IO_NETOP_PREP_ASYNC(connect);
 IO_NETOP_PREP(accept);
-IO_NETOP_FN(send);
-IO_NETOP_FN(recv);
+IO_NETOP_FN(sendto);
+IO_NETOP_FN(recvfrom);
 #endif /* CONFIG_NET */
 
 struct io_poll_table {
@@ -6771,13 +6771,13 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
 		ret = io_sendmsg(req, issue_flags);
 		break;
 	case IORING_OP_SEND:
-		ret = io_send(req, issue_flags);
+		ret = io_sendto(req, issue_flags);
 		break;
 	case IORING_OP_RECVMSG:
 		ret = io_recvmsg(req, issue_flags);
 		break;
 	case IORING_OP_RECV:
-		ret = io_recv(req, issue_flags);
+		ret = io_recvfrom(req, issue_flags);
 		break;
 	case IORING_OP_TIMEOUT:
 		ret = io_timeout(req, issue_flags);
Signature · 1 line
2.32.0
#2 lore raw

PATCH 2/3 net: Make `move_addr_to_user()` be a non static function

In reply to Ammar Faizi

To add recvfrom() support for io_uring, we need to call move_addr_to_user() from fs/io_uring.c.

This makes move_addr_to_user() be a non static function so we can call it from io_uring.

Cc "David S. Miller" <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Alviro Iskandar Setiawan <[email protected]>

---
v1:
- Add Alviro Iskandar Setiawan to CC list (tester).

RFC v4:
* No changes *

RFC v3:
* No changes *

RFC v2:
- Added Nugra to CC list (tester).

---

---
 include/linux/socket.h | 2 ++
 net/socket.c           | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)
include/linux/socket.h +2 -0
index 8ef26d89ef49..0d0bc1ace50c 100644
@@ -371,6 +371,8 @@ struct ucred {
 #define IPX_TYPE	1
 
 extern int move_addr_to_kernel(void __user *uaddr, int ulen, struct sockaddr_storage *kaddr);
+extern int move_addr_to_user(struct sockaddr_storage *kaddr, int klen,
+			     void __user *uaddr, int __user *ulen);
 extern int put_cmsg(struct msghdr*, int level, int type, int len, void *data);
 
 struct timespec64;
net/socket.c +2 -2
index 50cf75730fd7..9bc586ab4e93 100644
@@ -268,8 +268,8 @@ int move_addr_to_kernel(void __user *uaddr, int ulen, struct sockaddr_storage *k
  *	specified. Zero is returned for a success.
  */
 
-static int move_addr_to_user(struct sockaddr_storage *kaddr, int klen,
-			     void __user *uaddr, int __user *ulen)
+int move_addr_to_user(struct sockaddr_storage *kaddr, int klen,
+		      void __user *uaddr, int __user *ulen)
 {
 	int err;
 	int len;
Signature · 1 line
2.32.0
#3 lore raw

PATCH 3/3 io_uring: Add `sendto(2)` and `recvfrom(2)` support

In reply to Ammar Faizi

This adds sendto(2) and recvfrom(2) support for io_uring.

New opcodes:
IORING_OP_SENDTO
IORING_OP_RECVFROM

---
v1:
- Add BUILD_BUG_SQE_ELEM(48, __u64, addr3); for compile time assertion.
- Add Alviro Iskandar Setiawan to CC list (tester).

RFC v4:
- Rebase the work (sync with "for-next" branch in Jens' tree).
- Remove Tested-by tag from Nugra as this patch changes.
- (Address Praveen's comment) Zero sendto_addr_len and

    `recvfrom_addr_len` on prep when the `req->opcode` is not
    `IORING_OP_{SENDTO,RECVFROM}`.

RFC v3:
- Fix build error when CONFIG_NET is undefined should be done in the first patch, not this patch.
- Add Tested-by tag from Nugra.

RFC v2:
- In io_recvfrom(), mark the error check of move_addr_to_user() call as unlikely.
- Fix build error when CONFIG_NET is undefined.
- Added Nugra to CC list (tester).
---

---
 fs/io_uring.c                 | 83 +++++++++++++++++++++++++++++++++--
 include/uapi/linux/io_uring.h |  5 ++-
 2 files changed, 83 insertions(+), 5 deletions(-)
fs/io_uring.c +79 -4
index 742e252a052a..0c322e89de84 100644
@@ -574,7 +574,15 @@ struct io_sr_msg {
 	union {
 		struct compat_msghdr __user	*umsg_compat;
 		struct user_msghdr __user	*umsg;
-		void __user			*buf;
+
+		struct {
+			void __user		*buf;
+			struct sockaddr __user	*addr;
+			union {
+				int		sendto_addr_len;
+				int __user	*recvfrom_addr_len;
+			};
+		};
 	};
 	int				msg_flags;
 	int				bgid;
@@ -1105,6 +1113,19 @@ static const struct io_op_def io_op_defs[] = {
 	[IORING_OP_MKDIRAT] = {},
 	[IORING_OP_SYMLINKAT] = {},
 	[IORING_OP_LINKAT] = {},
+	[IORING_OP_SENDTO] = {
+		.needs_file		= 1,
+		.unbound_nonreg_file	= 1,
+		.pollout		= 1,
+		.audit_skip		= 1,
+	},
+	[IORING_OP_RECVFROM] = {
+		.needs_file		= 1,
+		.unbound_nonreg_file	= 1,
+		.pollin			= 1,
+		.buffer_select		= 1,
+		.audit_skip		= 1,
+	},
 };
 
 /* requests with any of those set should undergo io_disarm_next() */
@@ -4904,12 +4925,25 @@ static int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 	if (unlikely(req->ctx->flags & IORING_SETUP_IOPOLL))
 		return -EINVAL;
 
+	/*
+	 * For IORING_OP_SEND{,TO}, the assignment to @sr->umsg
+	 * is equivalent to an assignment to @sr->buf.
+	 */
 	sr->umsg = u64_to_user_ptr(READ_ONCE(sqe->addr));
+
 	sr->len = READ_ONCE(sqe->len);
 	sr->msg_flags = READ_ONCE(sqe->msg_flags) | MSG_NOSIGNAL;
 	if (sr->msg_flags & MSG_DONTWAIT)
 		req->flags |= REQ_F_NOWAIT;
 
+	if (req->opcode == IORING_OP_SENDTO) {
+		sr->addr = u64_to_user_ptr(READ_ONCE(sqe->addr2));
+		sr->sendto_addr_len = READ_ONCE(sqe->addr3);
+	} else {
+		sr->addr = (struct sockaddr __user *) NULL;
+		sr->sendto_addr_len = 0;
+	}
+
 #ifdef CONFIG_COMPAT
 	if (req->ctx->compat)
 		sr->msg_flags |= MSG_CMSG_COMPAT;
@@ -4963,6 +4997,7 @@ static int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags)
 
 static int io_sendto(struct io_kiocb *req, unsigned int issue_flags)
 {
+	struct sockaddr_storage address;
 	struct io_sr_msg *sr = &req->sr_msg;
 	struct msghdr msg;
 	struct iovec iov;
@@ -4979,10 +5014,20 @@ static int io_sendto(struct io_kiocb *req, unsigned int issue_flags)
 	if (unlikely(ret))
 		return ret;
 
-	msg.msg_name = NULL;
+
 	msg.msg_control = NULL;
 	msg.msg_controllen = 0;
-	msg.msg_namelen = 0;
+	if (sr->addr) {
+		ret = move_addr_to_kernel(sr->addr, sr->sendto_addr_len,
+					  &address);
+		if (unlikely(ret < 0))
+			goto fail;
+		msg.msg_name = (struct sockaddr *) &address;
+		msg.msg_namelen = sr->sendto_addr_len;
+	} else {
+		msg.msg_name = NULL;
+		msg.msg_namelen = 0;
+	}
 
 	flags = req->sr_msg.msg_flags;
 	if (issue_flags & IO_URING_F_NONBLOCK)
@@ -4997,6 +5042,7 @@ static int io_sendto(struct io_kiocb *req, unsigned int issue_flags)
 			return -EAGAIN;
 		if (ret == -ERESTARTSYS)
 			ret = -EINTR;
+fail:
 		req_set_fail(req);
 	}
 	__io_req_complete(req, issue_flags, ret, 0);
@@ -5115,13 +5161,26 @@ static int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 	if (unlikely(req->ctx->flags & IORING_SETUP_IOPOLL))
 		return -EINVAL;
 
+	/*
+	 * For IORING_OP_RECV{,FROM}, the assignment to @sr->umsg
+	 * is equivalent to an assignment to @sr->buf.
+	 */
 	sr->umsg = u64_to_user_ptr(READ_ONCE(sqe->addr));
+
 	sr->len = READ_ONCE(sqe->len);
 	sr->bgid = READ_ONCE(sqe->buf_group);
 	sr->msg_flags = READ_ONCE(sqe->msg_flags) | MSG_NOSIGNAL;
 	if (sr->msg_flags & MSG_DONTWAIT)
 		req->flags |= REQ_F_NOWAIT;
 
+	if (req->opcode == IORING_OP_RECVFROM) {
+		sr->addr = u64_to_user_ptr(READ_ONCE(sqe->addr2));
+		sr->recvfrom_addr_len = u64_to_user_ptr(READ_ONCE(sqe->addr3));
+	} else {
+		sr->addr = (struct sockaddr __user *) NULL;
+		sr->recvfrom_addr_len = (int __user *) NULL;
+	}
+
 #ifdef CONFIG_COMPAT
 	if (req->ctx->compat)
 		sr->msg_flags |= MSG_CMSG_COMPAT;
@@ -5197,6 +5256,7 @@ static int io_recvfrom(struct io_kiocb *req, unsigned int issue_flags)
 	struct iovec iov;
 	unsigned flags;
 	int ret, min_ret = 0;
+	struct sockaddr_storage address;
 	bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK;
 
 	sock = sock_from_file(req->file);
@@ -5214,9 +5274,10 @@ static int io_recvfrom(struct io_kiocb *req, unsigned int issue_flags)
 	if (unlikely(ret))
 		goto out_free;
 
-	msg.msg_name = NULL;
+	msg.msg_name = sr->addr ? (struct sockaddr *) &address : NULL;
 	msg.msg_control = NULL;
 	msg.msg_controllen = 0;
+	/* We assume all kernel code knows the size of sockaddr_storage */
 	msg.msg_namelen = 0;
 	msg.msg_iocb = NULL;
 	msg.msg_flags = 0;
@@ -5228,6 +5289,15 @@ static int io_recvfrom(struct io_kiocb *req, unsigned int issue_flags)
 		min_ret = iov_iter_count(&msg.msg_iter);
 
 	ret = sock_recvmsg(sock, &msg, flags);
+	if (ret >= 0 && sr->addr != NULL) {
+		int tmp;
+
+		tmp = move_addr_to_user(&address, msg.msg_namelen, sr->addr,
+					sr->recvfrom_addr_len);
+		if (unlikely(tmp < 0))
+			ret = tmp;
+	}
+
 out_free:
 	if (ret < min_ret) {
 		if (ret == -EAGAIN && force_nonblock)
@@ -6513,9 +6583,11 @@ static int io_req_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 	case IORING_OP_SYNC_FILE_RANGE:
 		return io_sfr_prep(req, sqe);
 	case IORING_OP_SENDMSG:
+	case IORING_OP_SENDTO:
 	case IORING_OP_SEND:
 		return io_sendmsg_prep(req, sqe);
 	case IORING_OP_RECVMSG:
+	case IORING_OP_RECVFROM:
 	case IORING_OP_RECV:
 		return io_recvmsg_prep(req, sqe);
 	case IORING_OP_CONNECT:
@@ -6770,12 +6842,14 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
 	case IORING_OP_SENDMSG:
 		ret = io_sendmsg(req, issue_flags);
 		break;
+	case IORING_OP_SENDTO:
 	case IORING_OP_SEND:
 		ret = io_sendto(req, issue_flags);
 		break;
 	case IORING_OP_RECVMSG:
 		ret = io_recvmsg(req, issue_flags);
 		break;
+	case IORING_OP_RECVFROM:
 	case IORING_OP_RECV:
 		ret = io_recvfrom(req, issue_flags);
 		break;
@@ -11218,6 +11292,7 @@ static int __init io_uring_init(void)
 	BUILD_BUG_SQE_ELEM(42, __u16,  personality);
 	BUILD_BUG_SQE_ELEM(44, __s32,  splice_fd_in);
 	BUILD_BUG_SQE_ELEM(44, __u32,  file_index);
+	BUILD_BUG_SQE_ELEM(48, __u64,  addr3);
 
 	BUILD_BUG_ON(sizeof(struct io_uring_files_update) !=
 		     sizeof(struct io_uring_rsrc_update));
include/uapi/linux/io_uring.h +4 -1
index 787f491f0d2a..a58cde19b4d0 100644
@@ -60,7 +60,8 @@ struct io_uring_sqe {
 		__s32	splice_fd_in;
 		__u32	file_index;
 	};
-	__u64	__pad2[2];
+	__u64	addr3;
+	__u64	__pad2[1];
 };
 
 enum {
@@ -143,6 +144,8 @@ enum {
 	IORING_OP_MKDIRAT,
 	IORING_OP_SYMLINKAT,
 	IORING_OP_LINKAT,
+	IORING_OP_SENDTO,
+	IORING_OP_RECVFROM,
 
 	/* this goes last, obviously */
 	IORING_OP_LAST,
Signature · 1 line
2.32.0
maintainer
#4 lore raw
In reply to Ammar Faizi
Quoted 5 lines · Ammar Faizi

Hello,

This patchset adds sendto(2) and recvfrom(2) support for io_uring. It also addresses an issue in the liburing GitHub repository [1].

Quoted 8 lines

## Motivations:

1) By using sendto() and recvfrom() we can make the submission simpler compared to always using sendmsg() and recvmsg() from the userspace. Especially for UDP socket.

2) There is a historical patch that tried to add the same functionality, but did not end up being applied. [2]

As far as I can tell, the only win from sendto/recvfrom is that we can handle async offload a bit cheaper compared to sendmsg/recvmsg. Is this enough to warrant adding them separately? I don't know, which is why this has been somewhat stalled for a while.

Maybe you have done some testing and have numbers (or other reasons) to back up the submission? There's not a whole lot of justification in this patchset.

Signature · 1 line
Jens Axboe
#5 lore raw
Quoted 15 lines · Jens Axboe
Quoted 5 lines · Ammar Faizi

Hello,

This patchset adds sendto(2) and recvfrom(2) support for io_uring. It also addresses an issue in the liburing GitHub repository [1].

Quoted 8 lines

## Motivations:

1) By using sendto() and recvfrom() we can make the submission simpler compared to always using sendmsg() and recvmsg() from the userspace. Especially for UDP socket.

2) There is a historical patch that tried to add the same functionality, but did not end up being applied. [2]

Quoted 9 lines

As far as I can tell, the only win from sendto/recvfrom is that we can handle async offload a bit cheaper compared to sendmsg/recvmsg. Is this enough to warrant adding them separately? I don't know, which is why this has been somewhat stalled for a while.

Maybe you have done some testing and have numbers (or other reasons) to back up the submission? There's not a whole lot of justification in this patchset.

So far, I haven't done it. I only created a test that ensures the functionality is working properly.

I will play with this further. If I win, I will submit the v2 of this series for review. Thanks, Jens!

Signature · 1 line
Ammar Faizi