Skip to contentSkip to conversation

Optimize get_current_ioprio() a bit

PATCH In review Jens Axboe 4 participants 9 messages Jan 8, 2024 – Jan 9, 2024 Block layer

lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

cover Optimize get_current_ioprio() a bit

Hi,

Came across this one in some recent profiling, and it's actually quite grim:

+ 2.71% io_uring [kernel.vmlinux] [k] __get_task_ioprio ▒

Just do the easy thing and move it into the header so we avoid a function call per IO for this. Patch 2 is just a general cleanup.

Signature · 1 line
Jens Axboe
author maintainer
#1 lore raw

PATCH 1/2 block: move __get_task_ioprio() into header file

We call this once per IO, which can be millions of times per second. Since nobody really uses io priorities, or at least it isn't very common, this is all wasted time and can amount to as much as 3% of the total kernel time.

---
 block/ioprio.c         | 26 --------------------------
 include/linux/ioprio.h | 25 ++++++++++++++++++++++++-
 2 files changed, 24 insertions(+), 27 deletions(-)
block/ioprio.c +0 -26
index b5a942519a79..73301a261429 100644
@@ -139,32 +139,6 @@ SYSCALL_DEFINE3(ioprio_set, int, which, int, who, int, ioprio)
 	return ret;
 }
 
-/*
- * If the task has set an I/O priority, use that. Otherwise, return
- * the default I/O priority.
- *
- * Expected to be called for current task or with task_lock() held to keep
- * io_context stable.
- */
-int __get_task_ioprio(struct task_struct *p)
-{
-	struct io_context *ioc = p->io_context;
-	int prio;
-
-	if (p != current)
-		lockdep_assert_held(&p->alloc_lock);
-	if (ioc)
-		prio = ioc->ioprio;
-	else
-		prio = IOPRIO_DEFAULT;
-
-	if (IOPRIO_PRIO_CLASS(prio) == IOPRIO_CLASS_NONE)
-		prio = IOPRIO_PRIO_VALUE(task_nice_ioclass(p),
-					 task_nice_ioprio(p));
-	return prio;
-}
-EXPORT_SYMBOL_GPL(__get_task_ioprio);
-
 static int get_task_ioprio(struct task_struct *p)
 {
 	int ret;
include/linux/ioprio.h +24 -1
index 7578d4f6a969..d6a9b5b7ed16 100644
@@ -47,7 +47,30 @@ static inline int task_nice_ioclass(struct task_struct *task)
 }
 
 #ifdef CONFIG_BLOCK
-int __get_task_ioprio(struct task_struct *p);
+/*
+ * If the task has set an I/O priority, use that. Otherwise, return
+ * the default I/O priority.
+ *
+ * Expected to be called for current task or with task_lock() held to keep
+ * io_context stable.
+ */
+static inline int __get_task_ioprio(struct task_struct *p)
+{
+	struct io_context *ioc = p->io_context;
+	int prio;
+
+	if (p != current)
+		lockdep_assert_held(&p->alloc_lock);
+	if (ioc)
+		prio = ioc->ioprio;
+	else
+		prio = IOPRIO_DEFAULT;
+
+	if (IOPRIO_PRIO_CLASS(prio) == IOPRIO_CLASS_NONE)
+		prio = IOPRIO_PRIO_VALUE(task_nice_ioclass(p),
+					 task_nice_ioprio(p));
+	return prio;
+}
 #else
 static inline int __get_task_ioprio(struct task_struct *p)
 {
Signature · 1 line
2.43.0
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] block: move __get_task_ioprio() into header file

Quoted 2 lines · Jens Axboe

We call this once per IO, which can be millions of times per second. Since nobody really uses io priorities, or at least it isn't very

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We have plans to set an I/O priority for most Android processes in the near future. According to what Damien wrote on linux-block, there is probably a significant number of hard disks for which I/O priority is configured.

Quoted 2 lines

common, this is all wasted time and can amount to as much as 3% of the total kernel time.

Anyway:

author maintainer
#3 lore raw

Re: [PATCH 1/2] block: move __get_task_ioprio() into header file

Quoted 9 lines · Bart Van Assche
Quoted 2 lines · Jens Axboe

We call this once per IO, which can be millions of times per second. Since nobody really uses io priorities, or at least it isn't very

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We have plans to set an I/O priority for most Android processes in the near future. According to what Damien wrote on linux-block, there is probably a significant number of hard disks for which I/O priority is configured.

The rates at which Android does IO means this won't really make any difference, should probably have qualified the statement with "nobody uses io priorities and does fast IO".

Quoted 6 lines
Quoted 2 lines

common, this is all wasted time and can amount to as much as 3% of the total kernel time.

Anyway:

Thanks!

Signature · 1 line
Jens Axboe
Reviewed-by on v1 1/2
#4 lore raw

Re: [PATCH 1/2] block: move __get_task_ioprio() into header file

In reply to Jens Axboe
Quoted 8 lines · Jens Axboe

We call this once per IO, which can be millions of times per second. Since nobody really uses io priorities, or at least it isn't very common, this is all wasted time and can amount to as much as 3% of the total kernel time.

---

Looks good.

-ck

author maintainer
#5 lore raw

PATCH 2/2 block: make __get_task_ioprio() easier to read

In reply to Jens Axboe

We don't need to do any gymnastics if we don't have an io_context assigned at all, so just return early with our default priority.

---
 include/linux/ioprio.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
include/linux/ioprio.h +4 -4
index d6a9b5b7ed16..db1249cd9692 100644
@@ -59,13 +59,13 @@ static inline int __get_task_ioprio(struct task_struct *p)
 	struct io_context *ioc = p->io_context;
 	int prio;
 
+	if (!ioc)
+		return IOPRIO_DEFAULT;
+
 	if (p != current)
 		lockdep_assert_held(&p->alloc_lock);
-	if (ioc)
-		prio = ioc->ioprio;
-	else
-		prio = IOPRIO_DEFAULT;
 
+	prio = ioc->ioprio;
 	if (IOPRIO_PRIO_CLASS(prio) == IOPRIO_CLASS_NONE)
 		prio = IOPRIO_PRIO_VALUE(task_nice_ioclass(p),
 					 task_nice_ioprio(p));
Signature · 1 line
2.43.0
Reviewed-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] block: make __get_task_ioprio() easier to read

Quoted 20 lines · Jens Axboe

We don't need to do any gymnastics if we don't have an io_context assigned at all, so just return early with our default priority.

---
  include/linux/ioprio.h | 8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)
include/linux/ioprio.h +3 -0
index d6a9b5b7ed16..db1249cd9692 100644
@@ -59,13 +59,13 @@ static inline int __get_task_ioprio(struct task_struct *p)
  	struct io_context *ioc = p->io_context;
  	int prio;
 
+	if (!ioc)
+		return IOPRIO_DEFAULT;
+
  	if (p != current)

include/linux/ioprio.h static inline int __get_task_ioprio(struct task_struct *p) · line 71

Quoted 10 lines
include/linux/ioprio.h +1 -4
  		lockdep_assert_held(&p->alloc_lock);
-	if (ioc)
-		prio = ioc->ioprio;
-	else
-		prio = IOPRIO_DEFAULT;
 
+	prio = ioc->ioprio;
  	if (IOPRIO_PRIO_CLASS(prio) == IOPRIO_CLASS_NONE)
  		prio = IOPRIO_PRIO_VALUE(task_nice_ioclass(p),
  					 task_nice_ioprio(p));

Shouldn't it be mentioned in the subject that this patch is a performance optimization? Anyway:

author maintainer
#7 lore raw

Re: [PATCH 2/2] block: make __get_task_ioprio() easier to read

Quoted 24 lines · Bart Van Assche
Quoted 23 lines · Jens Axboe

We don't need to do any gymnastics if we don't have an io_context assigned at all, so just return early with our default priority.

---
  include/linux/ioprio.h | 8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)
include/linux/ioprio.h +2 -3
index d6a9b5b7ed16..db1249cd9692 100644
@@ -59,13 +59,13 @@ static inline int __get_task_ioprio(struct task_struct *p)
      struct io_context *ioc = p->io_context;
      int prio;
  +    if (!ioc)
+        return IOPRIO_DEFAULT;
+
      if (p != current)
          lockdep_assert_held(&p->alloc_lock);
-    if (ioc)
-        prio = ioc->ioprio;
-    else
Quoted 8 lines

include/linux/ioprio.h static inline int __get_task_ioprio(struct task_struct *p) · line 71

Quoted 5 lines
include/linux/ioprio.h +0 -1
-        prio = IOPRIO_DEFAULT;
  +    prio = ioc->ioprio;
      if (IOPRIO_PRIO_CLASS(prio) == IOPRIO_CLASS_NONE)
          prio = IOPRIO_PRIO_VALUE(task_nice_ioclass(p),
                       task_nice_ioprio(p));

Shouldn't it be mentioned in the subject that this patch is a performance optimization? Anyway:

I doubt this matters for performance really, it's more of a readability thing for me.

Quoted 1 line

Thanks!

Signature · 1 line
Jens Axboe