Skip to contentSkip to conversation

sched, mm: Optimize current_gfp_context()

PATCH Applied Waiman Long 3 participants 6 messages Jun 18, 2020 – Sep 19, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 sched, mm: Optimize current_gfp_context()

The current_gfp_context() converts a number of PF_MEMALLOC_* per-process flags into the corresponding GFP_* flags for memory allocation. In that function, current->flags is accessed 3 times. That may lead to duplicated access of the same memory location.

This is not usually a problem with minimal debug config options on as the compiler can optimize away the duplicated memory accesses. With most of the debug config options on, however, that may not be the case. For example, the x86-64 object size of the __need_fs_reclaim() in a debug kernel that calls current_gfp_context() was 309 bytes. With this patch applied, the object size is reduced to 202 bytes. This is a saving of 107 bytes and will probably be slightly faster too.

---
 include/linux/sched/mm.h | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
include/linux/sched/mm.h +6 -4
index 480a4d1b7dd8..ff37cd8bbd82 100644
@@ -181,18 +181,20 @@ static inline bool in_vfork(struct task_struct *tsk)
  */
 static inline gfp_t current_gfp_context(gfp_t flags)
 {
-	if (unlikely(current->flags &
+	unsigned int pflags = READ_ONCE(current->flags);
+
+	if (unlikely(pflags &
 		     (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS | PF_MEMALLOC_NOCMA))) {
 		/*
 		 * NOIO implies both NOIO and NOFS and it is a weaker context
 		 * so always make sure it makes precedence
 		 */
-		if (current->flags & PF_MEMALLOC_NOIO)
+		if (pflags & PF_MEMALLOC_NOIO)
 			flags &= ~(__GFP_IO | __GFP_FS);
-		else if (current->flags & PF_MEMALLOC_NOFS)
+		else if (pflags & PF_MEMALLOC_NOFS)
 			flags &= ~__GFP_FS;
 #ifdef CONFIG_CMA
-		if (current->flags & PF_MEMALLOC_NOCMA)
+		if (pflags & PF_MEMALLOC_NOCMA)
 			flags &= ~__GFP_MOVABLE;
 #endif
 	}
Signature · 1 line
2.18.1
#1 lore raw
Quoted 18 lines · On Thu, 18 Jun 2020 17:29:36 -0400 Waiman Long <[email protected]> wrote:

The current_gfp_context() converts a number of PF_MEMALLOC_* per-process flags into the corresponding GFP_* flags for memory allocation. In that function, current->flags is accessed 3 times. That may lead to duplicated access of the same memory location.

This is not usually a problem with minimal debug config options on as the compiler can optimize away the duplicated memory accesses. With most of the debug config options on, however, that may not be the case. For example, the x86-64 object size of the __need_fs_reclaim() in a debug kernel that calls current_gfp_context() was 309 bytes. With this patch applied, the object size is reduced to 202 bytes. This is a saving of 107 bytes and will probably be slightly faster too.

...

include/linux/sched/mm.h static inline bool in_vfork(struct task_struct *tsk) · line 184

Quoted 5 lines
include/linux/sched/mm.h +1 -1
@@ -181,18 +181,20 @@ static inline bool in_vfork(struct task_struct *tsk)
  */
 static inline gfp_t current_gfp_context(gfp_t flags)
 {
-	if (unlikely(current->flags &
+	unsigned int pflags = READ_ONCE(current->flags);

Why use READ_ONCE() here?

Quoted 18 lines
include/linux/sched/mm.h +4 -3
+	if (unlikely(pflags &
 		     (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS | PF_MEMALLOC_NOCMA))) {
 		/*
 		 * NOIO implies both NOIO and NOFS and it is a weaker context
 		 * so always make sure it makes precedence
 		 */
-		if (current->flags & PF_MEMALLOC_NOIO)
+		if (pflags & PF_MEMALLOC_NOIO)
 			flags &= ~(__GFP_IO | __GFP_FS);
-		else if (current->flags & PF_MEMALLOC_NOFS)
+		else if (pflags & PF_MEMALLOC_NOFS)
 			flags &= ~__GFP_FS;
 #ifdef CONFIG_CMA
-		if (current->flags & PF_MEMALLOC_NOCMA)
+		if (pflags & PF_MEMALLOC_NOCMA)
 			flags &= ~__GFP_MOVABLE;
 #endif
 	}
#2 lore raw
Quoted 17 lines · Andrew Morton
Quoted 15 lines · On Thu, 18 Jun 2020 17:29:36 -0400 Waiman Long <[email protected]> wrote:

The current_gfp_context() converts a number of PF_MEMALLOC_* per-process flags into the corresponding GFP_* flags for memory allocation. In that function, current->flags is accessed 3 times. That may lead to duplicated access of the same memory location.

This is not usually a problem with minimal debug config options on as the compiler can optimize away the duplicated memory accesses. With most of the debug config options on, however, that may not be the case. For example, the x86-64 object size of the __need_fs_reclaim() in a debug kernel that calls current_gfp_context() was 309 bytes. With this patch applied, the object size is reduced to 202 bytes. This is a saving of 107 bytes and will probably be slightly faster too.

...

Quoted 9 lines

include/linux/sched/mm.h static inline bool in_vfork(struct task_struct *tsk) · line 184

Quoted 8 lines
include/linux/sched/mm.h +1 -1
@@ -181,18 +181,20 @@ static inline bool in_vfork(struct task_struct *tsk)
   */
  static inline gfp_t current_gfp_context(gfp_t flags)
  {
-	if (unlikely(current->flags &
+	unsigned int pflags = READ_ONCE(current->flags);

Why use READ_ONCE() here?

It was a change suggested by PeterZ. He said without the READ_ONCE(), the compiler may still choose to access current->flags multiple times instead of just reading it once. I think he is right and so I made the change.

Cheers,
Longman

#3 lore raw
In reply to Andrew Morton

Hi Waiman,

Quoted 7 lines · Andrew Morton
Quoted 5 lines · On Thu, 18 Jun 2020 17:29:36 -0400 Waiman Long <[email protected]> wrote:

The current_gfp_context() converts a number of PF_MEMALLOC_* per-process flags into the corresponding GFP_* flags for memory allocation. In that function, current->flags is accessed 3 times. That may lead to duplicated access of the same memory location.

I have a puzzle about this comment, what's the meaning about "That may lead to duplicated access of the same memory location". After using variable 'pflags', will it not duplicated access the same memory location?
Looking forward to your reply :)

Thanks,
Yanfei

Quoted 40 lines
Quoted 13 lines

This is not usually a problem with minimal debug config options on as the compiler can optimize away the duplicated memory accesses. With most of the debug config options on, however, that may not be the case. For example, the x86-64 object size of the __need_fs_reclaim() in a debug kernel that calls current_gfp_context() was 309 bytes. With this patch applied, the object size is reduced to 202 bytes. This is a saving of 107 bytes and will probably be slightly faster too.

...

include/linux/sched/mm.h static inline bool in_vfork(struct task_struct *tsk) · line 184

Quoted 5 lines
include/linux/sched/mm.h +1 -1
@@ -181,18 +181,20 @@ static inline bool in_vfork(struct task_struct *tsk)
   */
  static inline gfp_t current_gfp_context(gfp_t flags)
  {
-	if (unlikely(current->flags &
+	unsigned int pflags = READ_ONCE(current->flags);

Why use READ_ONCE() here?

Quoted 18 lines
include/linux/sched/mm.h +4 -3
+	if (unlikely(pflags &
  		     (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS | PF_MEMALLOC_NOCMA))) {
  		/*
  		 * NOIO implies both NOIO and NOFS and it is a weaker context
  		 * so always make sure it makes precedence
  		 */
-		if (current->flags & PF_MEMALLOC_NOIO)
+		if (pflags & PF_MEMALLOC_NOIO)
  			flags &= ~(__GFP_IO | __GFP_FS);
-		else if (current->flags & PF_MEMALLOC_NOFS)
+		else if (pflags & PF_MEMALLOC_NOFS)
  			flags &= ~__GFP_FS;
  #ifdef CONFIG_CMA
-		if (current->flags & PF_MEMALLOC_NOCMA)
+		if (pflags & PF_MEMALLOC_NOCMA)
  			flags &= ~__GFP_MOVABLE;
  #endif
  	}
#4 lore raw
Quoted 10 lines · Yanfei

Hi Waiman,

Quoted 7 lines · Andrew Morton
Quoted 5 lines · On Thu, 18 Jun 2020 17:29:36 -0400 Waiman Long <[email protected]> wrote:

The current_gfp_context() converts a number of PF_MEMALLOC_* per-process flags into the corresponding GFP_* flags for memory allocation. In that function, current->flags is accessed 3 times. That may lead to duplicated access of the same memory location.

Quoted 5 lines

I have a puzzle about this comment, what's the meaning about "That may lead to duplicated access of the same memory location". After using variable 'pflags', will it not duplicated access the same memory location?
Looking forward to your reply :)

That condition usually won't happen on a non-debug kernel. However, if certain debugging capability is turned on, access to current will be compiled into a bunch of checks and memory accesses. So if current is used multiple times, the same set of codes will be duplicated the same number of times slowing down the operation and increasing code size. By accessing current once, we avoid this overhead in a debug kernel.

Cheers,
Longman

#5 lore raw
Quoted 17 lines · Waiman Long
Quoted 10 lines · Yanfei

Hi Waiman,

Quoted 7 lines · Andrew Morton
Quoted 5 lines · On Thu, 18 Jun 2020 17:29:36 -0400 Waiman Long <[email protected]> wrote:

The current_gfp_context() converts a number of PF_MEMALLOC_* per-process flags into the corresponding GFP_* flags for memory allocation. In that function, current->flags is accessed 3 times. That may lead to duplicated access of the same memory location.

Quoted 5 lines

I have a puzzle about this comment, what's the meaning about "That may lead to duplicated access of the same memory location". After using variable 'pflags', will it not duplicated access the same memory location?
Looking forward to your reply :)

Quoted 5 lines

That condition usually won't happen on a non-debug kernel. However, if certain debugging capability is turned on, access to current will be compiled into a bunch of checks and memory accesses. So if current is used multiple times, the same set of codes will be duplicated the same number of times slowing down the operation and increasing code size. By accessing current once, we avoid this overhead in a debug kernel.

Cheers,
Longman

Hah, got it.
Thanks for your detailed explain!

cheers,
Yanfei