Skip to contentSkip to conversation

sched, mm: Optimize current_gfp_context()

PATCH Superseded Waiman Long 2 participants 3 messages Jun 18, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched, mm: Optimize current_gfp_context()

The current_gfp_context() converts a number of PF_MEMALLOC_* per-process flags into the corresponding GFP_* flags for memory allocation. In that function, current->flags is accessed 3 times. That may lead to duplicated access of the same memory location.

This is not usually a problem with minimal debug config options on as the compiler can optimize away the duplicated memory accesses. With most of the debug config options on, however, that may not be the case. For example, the x86-64 object size of the __need_fs_reclaim() in a debug kernel that calls current_gfp_context() was 309 bytes. With this patch applied, the object size is reduced to 202 bytes. This is a saving of 107 bytes and will probably be slightly faster too.

---
 include/linux/sched/mm.h | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
include/linux/sched/mm.h +6 -4
index 480a4d1b7dd8..09e01909d192 100644
@@ -181,18 +181,20 @@ static inline bool in_vfork(struct task_struct *tsk)
  */
 static inline gfp_t current_gfp_context(gfp_t flags)
 {
-	if (unlikely(current->flags &
+	unsigned int pflags = current->flags;
+
+	if (unlikely(pflags &
 		     (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS | PF_MEMALLOC_NOCMA))) {
 		/*
 		 * NOIO implies both NOIO and NOFS and it is a weaker context
 		 * so always make sure it makes precedence
 		 */
-		if (current->flags & PF_MEMALLOC_NOIO)
+		if (pflags & PF_MEMALLOC_NOIO)
 			flags &= ~(__GFP_IO | __GFP_FS);
-		else if (current->flags & PF_MEMALLOC_NOFS)
+		else if (pflags & PF_MEMALLOC_NOFS)
 			flags &= ~__GFP_FS;
 #ifdef CONFIG_CMA
-		if (current->flags & PF_MEMALLOC_NOCMA)
+		if (pflags & PF_MEMALLOC_NOCMA)
 			flags &= ~__GFP_MOVABLE;
 #endif
 	}
Signature · 1 line
2.18.1
#1 lore raw
Quoted 23 lines · Waiman Long

The current_gfp_context() converts a number of PF_MEMALLOC_* per-process flags into the corresponding GFP_* flags for memory allocation. In that function, current->flags is accessed 3 times. That may lead to duplicated access of the same memory location.

This is not usually a problem with minimal debug config options on as the compiler can optimize away the duplicated memory accesses. With most of the debug config options on, however, that may not be the case. For example, the x86-64 object size of the __need_fs_reclaim() in a debug kernel that calls current_gfp_context() was 309 bytes. With this patch applied, the object size is reduced to 202 bytes. This is a saving of 107 bytes and will probably be slightly faster too.

---
 include/linux/sched/mm.h | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
include/linux/sched/mm.h +0 -0
index 480a4d1b7dd8..09e01909d192 100644

include/linux/sched/mm.h static inline bool in_vfork(struct task_struct *tsk) · line 184

Quoted 5 lines
include/linux/sched/mm.h +1 -1
@@ -181,18 +181,20 @@ static inline bool in_vfork(struct task_struct *tsk)
  */
 static inline gfp_t current_gfp_context(gfp_t flags)
 {
-	if (unlikely(current->flags &
+	unsigned int pflags = current->flags;

Per the above, I would've expected a READ_ONCE() there, otherwise the compiler can still re-load.

#2 lore raw
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Waiman Long

The current_gfp_context() converts a number of PF_MEMALLOC_* per-process flags into the corresponding GFP_* flags for memory allocation. In that function, current->flags is accessed 3 times. That may lead to duplicated access of the same memory location.

This is not usually a problem with minimal debug config options on as the compiler can optimize away the duplicated memory accesses. With most of the debug config options on, however, that may not be the case. For example, the x86-64 object size of the __need_fs_reclaim() in a debug kernel that calls current_gfp_context() was 309 bytes. With this patch applied, the object size is reduced to 202 bytes. This is a saving of 107 bytes and will probably be slightly faster too.

---
  include/linux/sched/mm.h | 10 ++++++----
  1 file changed, 6 insertions(+), 4 deletions(-)
Quoted 13 lines

include/linux/sched/mm.h static inline bool in_vfork(struct task_struct *tsk) · line 184

Quoted 10 lines
include/linux/sched/mm.h +1 -1
index 480a4d1b7dd8..09e01909d192 100644
@@ -181,18 +181,20 @@ static inline bool in_vfork(struct task_struct *tsk)
   */
  static inline gfp_t current_gfp_context(gfp_t flags)
  {
-	if (unlikely(current->flags &
+	unsigned int pflags = current->flags;

Per the above, I would've expected a READ_ONCE() there, otherwise the compiler can still re-load.

Right. Will do that.

Thanks,
Longman