io_uring: add napi busy polling support

RFC PATCH Superseded Stefan Roesch 3 participants 10 messages Nov 21, 2022

Subsystems (3)

Networking Network drivers io_uring

lore

Share & tools

lore

Stefan Roesch

author

2022-11-21 17:30 UTC #0 lore raw

v4 cover io_uring: add napi busy polling support

io-uring allows specifying two parameters:
- busy poll timeout and
- prefer busy poll to call of io_napi_busy_loop()
This sets the above parameters for the ring. The settings are passed with a new structure io_uring_napi.

There is also a corresponding liburing patch series, which enables this feature. The name of the series is "liburing: add add api for napi busy poll timeout". It also contains two programs to test the this.

Testing has shown that the round-trip times are reduced to 38us from 55us by enabling napi busy polling with a busy poll timeout of 100us. More detailled results are part of the commit message of the first patch.

Changes:
- V4:
- Pass structure for napi config, instead of individual parameters
- V3:
- Refreshed to 6.1-rc5
- Added a new io-uring api for the prefer napi busy poll api and wire it to io_napi_busy_loop().
- Removed the unregister (implemented as register)
- Added more performance results to the first commit message.
- V2:
- Add missing defines if CONFIG_NET_RX_BUSY_POLL is not defined
- Changes signature of function io_napi_add_list to static inline if CONFIG_NET_RX_BUSY_POLL is not defined
- define some functions as static

Signed-off-by Stefan Roesch <[email protected]>

Stefan Roesch (3):
io_uring: add napi busy polling support
io_uring: add api to set / get napi configuration.
io_uring: add api to set napi prefer busy poll

 include/linux/io_uring_types.h |   8 +
 include/uapi/linux/io_uring.h  |  12 ++
 io_uring/io_uring.c            | 302 +++++++++++++++++++++++++++++++++
 io_uring/napi.h                |  22 +++
 io_uring/poll.c                |   3 +
 io_uring/sqpoll.c              |  10 ++
 6 files changed, 357 insertions(+)
 create mode 100644 io_uring/napi.h

base-commit: ab290eaddc4c41b237b9a366fa6a5527be890b84

Signature · 1 line

2.30.2

Stefan Roesch

author

2022-11-21 17:30 UTC #2 lore raw

RFC PATCH v4 2/3 io_uring: add api to set / get napi configuration.

This adds an api to register the busy poll timeout from liburing. To be able to use this functionality, the corresponding liburing patch is needed.

Signed-off-by Stefan Roesch <[email protected]>

---
 include/linux/io_uring_types.h |  2 +-
 include/uapi/linux/io_uring.h  | 11 +++++++
 io_uring/io_uring.c            | 54 ++++++++++++++++++++++++++++++++++
 3 files changed, 66 insertions(+), 1 deletion(-)

include/linux/io_uring_types.h +1 -1

index 23993b5d3186..67b861305d97 100644
@@ -274,8 +274,8 @@ struct io_ring_ctx {
 	struct list_head	napi_list;	/* track busy poll napi_id */
 	spinlock_t		napi_lock;	/* napi_list lock */
 
-	unsigned int		napi_busy_poll_to; /* napi busy poll default timeout */
 	bool			napi_prefer_busy_poll;
+	unsigned int		napi_busy_poll_to;
 #endif
 
 	struct {

include/uapi/linux/io_uring.h +11 -0

index 2df3225b562f..1a713bbafaee 100644
@@ -490,6 +490,10 @@ enum {
 	/* register a range of fixed file slots for automatic slot allocation */
 	IORING_REGISTER_FILE_ALLOC_RANGE	= 25,
 
+	/* set/clear busy poll settings */
+	IORING_REGISTER_NAPI			= 26,
+	IORING_UNREGISTER_NAPI			= 27,
+
 	/* this goes last */
 	IORING_REGISTER_LAST
 };

@@ -612,6 +616,13 @@ struct io_uring_buf_reg {
 	__u64	resv[3];
 };
 
+/* argument for IORING_(UN)REGISTER_NAPI */
+struct io_uring_napi {
+	__u32	busy_poll_to;
+	__u32	pad;
+	__u64	resv;
+};
+
 /*
  * io_uring_restriction->opcode values
  */

io_uring/io_uring.c +54 -0

66 lines not on this page · raw message

Signature · 1 line

2.30.2

Stefan Roesch

author

2022-11-21 17:30 UTC #9 lore raw

RFC PATCH v4 3/3 io_uring: add api to set napi prefer busy poll

In reply to Stefan Roesch

This adds an api to register and unregister the napi prefer busy poll setting from liburing. To be able to use this functionality, the corresponding liburing patch is needed.

Signed-off-by Stefan Roesch <[email protected]>

---
 include/uapi/linux/io_uring.h | 3 ++-
 io_uring/io_uring.c           | 5 ++++-
 2 files changed, 6 insertions(+), 2 deletions(-)

include/uapi/linux/io_uring.h +2 -1

index 1a713bbafaee..b74021454bf7 100644
@@ -619,7 +619,8 @@ struct io_uring_buf_reg {
 /* argument for IORING_(UN)REGISTER_NAPI */
 struct io_uring_napi {
 	__u32	busy_poll_to;
-	__u32	pad;
+	__u8    prefer_busy_poll;
+	__u8	pad[3];
 	__u64	resv;
 };

io_uring/io_uring.c +4 -1

index cf0e7cc8ad2e..dd903c71fa20 100644
@@ -4127,6 +4127,7 @@ static int io_register_napi(struct io_ring_ctx *ctx, void __user *arg)
 #ifdef CONFIG_NET_RX_BUSY_POLL
 	const struct io_uring_napi curr = {
 		.busy_poll_to = ctx->napi_busy_poll_to,
+		.prefer_busy_poll = ctx->napi_prefer_busy_poll
 	};
 	struct io_uring_napi *napi;

@@ -4135,7 +4136,7 @@ static int io_register_napi(struct io_ring_ctx *ctx, void __user *arg)
 		return PTR_ERR(napi);
 
 	WRITE_ONCE(ctx->napi_busy_poll_to, napi->busy_poll_to);
-
+	WRITE_ONCE(ctx->napi_prefer_busy_poll, !!napi->prefer_busy_poll);
 	kfree(napi);
 
 	if (copy_to_user(arg, &curr, sizeof(curr)))

@@ -4152,12 +4153,14 @@ static int io_unregister_napi(struct io_ring_ctx *ctx, void __user *arg)
 #ifdef CONFIG_NET_RX_BUSY_POLL
 	const struct io_uring_napi curr = {
 		.busy_poll_to = ctx->napi_busy_poll_to,
+		.prefer_busy_poll = ctx->napi_prefer_busy_poll
 	};
 
 	if (copy_to_user(arg, &curr, sizeof(curr)))
 		return -EFAULT;
 
 	WRITE_ONCE(ctx->napi_busy_poll_to, 0);
+	WRITE_ONCE(ctx->napi_prefer_busy_poll, false);
 	return 0;
 #else
 	return -EINVAL;

Signature · 1 line

2.30.2

Stefan Roesch

author

2022-11-21 17:30 UTC #1 lore raw

RFC PATCH v4 1/3 io_uring: add napi busy polling support

In reply to Stefan Roesch

This adds the napi busy polling support in io_uring.c. It adds a new napi_list to the io_ring_ctx structure. This list contains the list of napi_id's that are currently enabled for busy polling. The list is synchronized by the new napi_lock spin lock. The current default napi busy polling time is stored in napi_busy_poll_to. If napi busy polling is not enabled, the value is 0.

The NAPI_TIMEOUT is stored as a timeout to make sure that the time a napi entry is stored in the napi list is limited.

The busy poll timeout is also stored as part of the io_wait_queue. This is necessary as for sq polling the poll interval needs to be adjusted and the napi callback allows only to pass in one value.

This has been tested with two simple programs from the liburing library
repository: the napi client and the napi server program. The client sends a request, which has a timestamp in its payload and the server replies with the same payload. The client calculates the roundtrip time and stores it to calcualte the results.

The client is running on host1 and the server is running on host 2 (in the same rack). The measured times below are roundtrip times. They are average times over 5 runs each. Each run measures 1 million roundtrips.

no rx coal rx coal: frames=88,usecs=33
Default 57us 56us

client_poll=100us 47us 46us

server_poll=100us 51us 46us

client_poll=100us+ 40us 40us
server_poll=100us

client_poll=100us+ 41us 39us
server_poll=100us+
prefer napi busy poll on client

client_poll=100us+ 41us 39us
server_poll=100us+
prefer napi busy poll on server

client_poll=100us+ 41us 39us
server_poll=100us+
prefer napi busy poll on client + server

Signed-off-by Stefan Roesch <[email protected]>

---
 include/linux/io_uring_types.h |   8 ++
 io_uring/io_uring.c            | 245 +++++++++++++++++++++++++++++++++
 io_uring/napi.h                |  22 +++
 io_uring/poll.c                |   3 +
 io_uring/sqpoll.c              |  10 ++
 5 files changed, 288 insertions(+)
 create mode 100644 io_uring/napi.h

include/linux/io_uring_types.h +8 -0

index f5b687a787a3..23993b5d3186 100644
@@ -270,6 +270,14 @@ struct io_ring_ctx {
 	struct xarray		personalities;
 	u32			pers_next;
 
+#ifdef CONFIG_NET_RX_BUSY_POLL
+	struct list_head	napi_list;	/* track busy poll napi_id */
+	spinlock_t		napi_lock;	/* napi_list lock */
+
+	unsigned int		napi_busy_poll_to; /* napi busy poll default timeout */
+	bool			napi_prefer_busy_poll;
+#endif
+
 	struct {
 		/*
 		 * We cache a range of free CQEs we can use, once exhausted it

io_uring/io_uring.c +245 -0

299 lines not on this page · raw message

io_uring/napi.h new file +22 -0

new file mode 100644
index 000000000000..11602d679629
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef IOU_NAPI_H
+#define IOU_NAPI_H
+
+#include <linux/kernel.h>
+#include <linux/io_uring.h>
+#include <net/busy_poll.h>
+
+#ifdef CONFIG_NET_RX_BUSY_POLL
+
+void io_napi_add(struct file *file, struct io_ring_ctx *ctx);
+bool io_napi_busy_loop(struct list_head *napi_list, bool prefer_busy_poll);
+
+#else
+
+static inline void io_napi_add(struct file *file, struct io_ring_ctx *ctx)
+{
+}
+
+#endif
+#endif

io_uring/poll.c +3 -0

index f500506984ec..fafce408ee14 100644
@@ -15,6 +15,7 @@
 
 #include "io_uring.h"
 #include "refs.h"
+#include "napi.h"
 #include "opdef.h"
 #include "kbuf.h"
 #include "poll.h"

@@ -250,6 +251,7 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked)
 				io_req_set_res(req, mask, 0);
 				return IOU_POLL_REMOVE_POLL_USE_RES;
 			}
+			io_napi_add(req->file, ctx);
 		} else {
 			ret = io_poll_issue(req, locked);
 			if (ret == IOU_STOP_MULTISHOT)

@@ -571,6 +573,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req,
 		__io_poll_execute(req, mask);
 		return 0;
 	}
+	io_napi_add(req->file, req->ctx);
 
 	if (ipt->owning) {
 		/*

io_uring/sqpoll.c +10 -0

index 559652380672..6f50bc156038 100644
@@ -15,6 +15,7 @@
 #include <uapi/linux/io_uring.h>
 
 #include "io_uring.h"
+#include "napi.h"
 #include "sqpoll.h"
 
 #define IORING_SQPOLL_CAP_ENTRIES_VALUE 8

@@ -193,6 +194,15 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries)
 			ret = io_submit_sqes(ctx, to_submit);
 		mutex_unlock(&ctx->uring_lock);
 
+#ifdef CONFIG_NET_RX_BUSY_POLL
+		spin_lock(&ctx->napi_lock);
+		if (!list_empty(&ctx->napi_list) &&
+		    READ_ONCE(ctx->napi_busy_poll_to) > 0 &&
+		    io_napi_busy_loop(&ctx->napi_list, ctx->napi_prefer_busy_poll))
+			++ret;
+		spin_unlock(&ctx->napi_lock);
+#endif
+
 		if (to_submit && wq_has_sleeper(&ctx->sqo_sq_wait))
 			wake_up(&ctx->sqo_sq_wait);
 		if (creds)

Signature · 1 line

2.30.2

Jens Axboe

maintainer

2022-11-21 17:35 UTC #3 lore raw

Re: [RFC PATCH v4 2/3] io_uring: add api to set / get napi configuration.

In reply to Stefan Roesch

Quoted 15 lines · Stefan Roesch

This adds an api to register the busy poll timeout from liburing. To be able to use this functionality, the corresponding liburing patch is needed.
Signed-off-by Stefan Roesch <[email protected]>
---
 include/linux/io_uring_types.h |  2 +-
 include/uapi/linux/io_uring.h  | 11 +++++++
 io_uring/io_uring.c            | 54 ++++++++++++++++++++++++++++++++++
 3 files changed, 66 insertions(+), 1 deletion(-)
include/linux/io_uring_types.h +0 -0
index 23993b5d3186..67b861305d97 100644

include/linux/io_uring_types.h struct io_ring_ctx { · line 279

Quoted 7 lines

include/linux/io_uring_types.h +1 -1

@@ -274,8 +274,8 @@ struct io_ring_ctx {
 	struct list_head	napi_list;	/* track busy poll napi_id */
 	spinlock_t		napi_lock;	/* napi_list lock */
 
-	unsigned int		napi_busy_poll_to; /* napi busy poll default timeout */
 	bool			napi_prefer_busy_poll;
+	unsigned int		napi_busy_poll_to;
 #endif

Why is this being moved? Seems unrelated, and it actually creates another hole rather than filling one as it did before.

Signature · 1 line

Jens Axboe

Jens Axboe

maintainer

2022-11-21 17:37 UTC #5 lore raw

Re: [RFC PATCH v4 2/3] io_uring: add api to set / get napi configuration.

In reply to Stefan Roesch

Quoted 22 lines · Stefan Roesch

io_uring/io_uring.c +14 -0

index 4f432694cbed..cf0e7cc8ad2e 100644
@@ -4122,6 +4122,48 @@ static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx,
 	return ret;
 }
 
+static int io_register_napi(struct io_ring_ctx *ctx, void __user *arg)
+{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+	const struct io_uring_napi curr = {
+		.busy_poll_to = ctx->napi_busy_poll_to,
+	};
+	struct io_uring_napi *napi;
+
+	napi = memdup_user(arg, sizeof(*napi));
+	if (IS_ERR(napi))
+		return PTR_ERR(napi);
+
+	WRITE_ONCE(ctx->napi_busy_poll_to, napi->busy_poll_to);
+

io_uring/io_uring.c static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, · line 4148

Quoted 10 lines

io_uring/io_uring.c +10 -0

+	kfree(napi);
+
+	if (copy_to_user(arg, &curr, sizeof(curr)))
+		return -EFAULT;
+
+	return 0;
+#else
+	return -EINVAL;
+#endif
+}

This should return -EINVAL if any of the padding or reserved fields are non-zero. If you don't do that, then it's not expendable in the future.

Signature · 1 line

Jens Axboe

Ammar Faizi

2022-11-21 17:45 UTC #7 lore raw

Re: [RFC PATCH v4 2/3] io_uring: add api to set / get napi configuration.

In reply to Stefan Roesch

Quoted 10 lines · Stefan Roesch

io_uring/io_uring.c +10 -0

+static int io_register_napi(struct io_ring_ctx *ctx, void __user *arg)
+{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+	const struct io_uring_napi curr = {
+		.busy_poll_to = ctx->napi_busy_poll_to,
+	};
+	struct io_uring_napi *napi;
+
+	napi = memdup_user(arg, sizeof(*napi));
+	if (IS_ERR(napi))

io_uring/io_uring.c static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, · line 4144

Quoted 10 lines

io_uring/io_uring.c +10 -0

+		return PTR_ERR(napi);
+
+	WRITE_ONCE(ctx->napi_busy_poll_to, napi->busy_poll_to);
+
+	kfree(napi);
+
+	if (copy_to_user(arg, &curr, sizeof(curr)))
+		return -EFAULT;
+
+	return 0;

Considering:

1) struct io_uring_napi is 16 bytes in size.

2) The lifetime of struct io_uring_napi *napi; is brief.

There is no need to use memdup_user() and kfree(). You can place it on the stack and use copy_from_user() instead.

Signature · 1 line

Ammar Faizi

Stefan Roesch

author

2022-11-21 18:55 UTC #6 lore raw

Re: [RFC PATCH v4 2/3] io_uring: add api to set / get napi configuration.

In reply to Jens Axboe

Quoted 28 lines · Jens Axboe

Quoted 27 lines · Stefan Roesch

io_uring/io_uring.c +19 -0

index 4f432694cbed..cf0e7cc8ad2e 100644
@@ -4122,6 +4122,48 @@ static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx,
 	return ret;
 }
 
+static int io_register_napi(struct io_ring_ctx *ctx, void __user *arg)
+{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+	const struct io_uring_napi curr = {
+		.busy_poll_to = ctx->napi_busy_poll_to,
+	};
+	struct io_uring_napi *napi;
+
+	napi = memdup_user(arg, sizeof(*napi));
+	if (IS_ERR(napi))
+		return PTR_ERR(napi);
+
+	WRITE_ONCE(ctx->napi_busy_poll_to, napi->busy_poll_to);
+
+	kfree(napi);
+
+	if (copy_to_user(arg, &curr, sizeof(curr)))
+		return -EFAULT;
+

Quoted 8 lines

io_uring/io_uring.c static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, · line 4148
Quoted 5 lines
io_uring/io_uring.c +5 -0
+	return 0;
+#else
+	return -EINVAL;
+#endif
+}
This should return -EINVAL if any of the padding or reserved fields are non-zero. If you don't do that, then it's not expendable in the future.

The next version will have that change.

Stefan Roesch

author

2022-11-21 18:56 UTC #4 lore raw

Re: [RFC PATCH v4 2/3] io_uring: add api to set / get napi configuration.

In reply to Jens Axboe

Quoted 19 lines · Jens Axboe

Quoted 18 lines · Stefan Roesch
This adds an api to register the busy poll timeout from liburing. To be able to use this functionality, the corresponding liburing patch is needed.
Signed-off-by Stefan Roesch <[email protected]>
---
 include/linux/io_uring_types.h |  2 +-
 include/uapi/linux/io_uring.h  | 11 +++++++
 io_uring/io_uring.c            | 54 ++++++++++++++++++++++++++++++++++
 3 files changed, 66 insertions(+), 1 deletion(-)
include/linux/io_uring_types.h +0 -0
index 23993b5d3186..67b861305d97 100644
@@ -274,8 +274,8 @@ struct io_ring_ctx {
 	struct list_head	napi_list;	/* track busy poll napi_id */
 	spinlock_t		napi_lock;	/* napi_list lock */
 

Quoted 7 lines

include/linux/io_uring_types.h struct io_ring_ctx { · line 279
Quoted 4 lines
include/linux/io_uring_types.h +1 -1
-	unsigned int		napi_busy_poll_to; /* napi busy poll default timeout */
 	bool			napi_prefer_busy_poll;
+	unsigned int		napi_busy_poll_to;
 #endif
Why is this being moved? Seems unrelated, and it actually creates another hole rather than filling one as it did before.

That was not intended. The next version of the patch will restore the previous order.

Stefan Roesch

author

2022-11-21 18:57 UTC #8 lore raw

Re: [RFC PATCH v4 2/3] io_uring: add api to set / get napi configuration.

In reply to Ammar Faizi

Quoted 24 lines · Ammar Faizi

Quoted 10 lines · Stefan Roesch

io_uring/io_uring.c +10 -0

+static int io_register_napi(struct io_ring_ctx *ctx, void __user *arg)
+{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+	const struct io_uring_napi curr = {
+		.busy_poll_to = ctx->napi_busy_poll_to,
+	};
+	struct io_uring_napi *napi;
+
+	napi = memdup_user(arg, sizeof(*napi));
+	if (IS_ERR(napi))

io_uring/io_uring.c static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, · line 4144

Quoted 10 lines

io_uring/io_uring.c +10 -0

+		return PTR_ERR(napi);
+
+	WRITE_ONCE(ctx->napi_busy_poll_to, napi->busy_poll_to);
+
+	kfree(napi);
+
+	if (copy_to_user(arg, &curr, sizeof(curr)))
+		return -EFAULT;
+
+	return 0;

Considering:

Quoted 6 lines

1) struct io_uring_napi is 16 bytes in size.
2) The lifetime of struct io_uring_napi *napi; is brief.
There is no need to use memdup_user() and kfree(). You can place it on the stack and use copy_from_user() instead.

The next version of the patch will use copy_from_user.