Skip to contentSkip to conversation

fs/xfs: Use atomic64_try_cmpxchg in xlog_grant_{add,sub}_space

PATCH In review Uros Bizjak 2 participants 5 messages Aug 9, 2022 – Aug 10, 2022 XFS filesystem

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fs/xfs: Use atomic64_try_cmpxchg in xlog_grant_{add,sub}_space

Use !atomic64_try_cmpxchg(ptr, &old, new) instead of atomic64_cmpxchg(ptr, old, new) != old in xlog_grant_{add,sub}_space. This has two benefits:

- The x86 cmpxchg instruction returns success in the ZF flag, so this change saves a compare after cmpxchg, as well as a related move instruction in the front of cmpxchg.

- atomic64_try_cmpxchg implicitly assigns the *ptr value to &old when cmpxchg fails, enabling further code simplifications.

This patch has no functional change.

Cc "Darrick J. Wong" <[email protected]>
---
 fs/xfs/xfs_log.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
fs/xfs/xfs_log.c +6 -8
index 4b1c0a9c6368..92e39873d09e 100644
@@ -148,7 +148,7 @@ xlog_grant_sub_space(
 	int			bytes)
 {
 	int64_t	head_val = atomic64_read(head);
-	int64_t new, old;
+	int64_t new;
 
 	do {
 		int	cycle, space;
@@ -161,10 +161,9 @@ xlog_grant_sub_space(
 			cycle--;
 		}
 
-		old = head_val;
 		new = xlog_assign_grant_head_val(cycle, space);
-		head_val = atomic64_cmpxchg(head, old, new);
-	} while (head_val != old);
+
+	} while (!atomic64_try_cmpxchg(head, &head_val, new));
 }
 
 static void
@@ -174,7 +173,7 @@ xlog_grant_add_space(
 	int			bytes)
 {
 	int64_t	head_val = atomic64_read(head);
-	int64_t new, old;
+	int64_t new;
 
 	do {
 		int		tmp;
@@ -190,10 +189,9 @@ xlog_grant_add_space(
 			cycle++;
 		}
 
-		old = head_val;
 		new = xlog_assign_grant_head_val(cycle, space);
-		head_val = atomic64_cmpxchg(head, old, new);
-	} while (head_val != old);
+
+	} while (!atomic64_try_cmpxchg(head, &head_val, new));
 }
 
 STATIC void
Signature · 1 line
2.37.1
#1 lore raw
Quoted 10 lines · Uros Bizjak

Use !atomic64_try_cmpxchg(ptr, &old, new) instead of atomic64_cmpxchg(ptr, old, new) != old in xlog_grant_{add,sub}_space. This has two benefits:

- The x86 cmpxchg instruction returns success in the ZF flag, so this change saves a compare after cmpxchg, as well as a related move instruction in the front of cmpxchg.

- atomic64_try_cmpxchg implicitly assigns the *ptr value to &old when cmpxchg fails, enabling further code simplifications.

Do the two cmpxchg operations have the same memory ordering semantics on failure?

Quoted 1 line

This patch has no functional change.

The patch looks ok, but ....

... I'm about 2 hours away from posting a patchset that completely removes the cmpxchg and the new grant head accounting has significantly lower fast path overhead. It also opens the door for tracking more than 2GB of log space in the grant heads.

So I don't think we need to be micro-optimising this code given that there are much bigger perf gains about to land...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw
Quoted 15 lines · Dave Chinner
Quoted 10 lines · Uros Bizjak

Use !atomic64_try_cmpxchg(ptr, &old, new) instead of atomic64_cmpxchg(ptr, old, new) != old in xlog_grant_{add,sub}_space. This has two benefits:

- The x86 cmpxchg instruction returns success in the ZF flag, so this change saves a compare after cmpxchg, as well as a related move instruction in the front of cmpxchg.

- atomic64_try_cmpxchg implicitly assigns the *ptr value to &old when cmpxchg fails, enabling further code simplifications.

Do the two cmpxchg operations have the same memory ordering semantics on failure?

Quoted 8 lines
Quoted 1 line

This patch has no functional change.

The patch looks ok, but ....

... I'm about 2 hours away from posting a patchset that completely removes the cmpxchg and the new grant head accounting has significantly lower fast path overhead. It also opens the door for tracking more than 2GB of log space in the grant heads.

FYI, the original RFC for this was posted a bit over a month ago:

https://lore.kernel.org/linux-xfs/[email protected]/

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#3 lore raw
Quoted 11 lines · Dave Chinner
Quoted 9 lines · Dave Chinner
Quoted 8 lines · Uros Bizjak

Use !atomic64_try_cmpxchg(ptr, &old, new) instead of atomic64_cmpxchg(ptr, old, new) != old in xlog_grant_{add,sub}_space. This has two benefits:

- The x86 cmpxchg instruction returns success in the ZF flag, so this change saves a compare after cmpxchg, as well as a related move instruction in the front of cmpxchg.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

- atomic64_try_cmpxchg implicitly assigns the *ptr value to &old when cmpxchg fails, enabling further code simplifications.

Do the two cmpxchg operations have the same memory ordering semantics on failure?

Yes. The API also provides _acquire, _release and _relaxed variants of both, atomic64_cmpxchg and atomic64_try_cmpxchg. On x86, these two functions actually compile to the same CMPXCHG instruction, the difference is only in how the comparison is handled:

      15:    48 09 c2                 or     %rax,%rdx
      18:    48 89 c8                 mov    %rcx,%rax
      1b:    f0 48 0f b1 16           lock cmpxchg %rdx,(%rsi)
      20:    48 39 c1                 cmp    %rax,%rcx
      23:    74 2a                    je     4f <xlog_grant_add_space+0x4f>

becomes:

     29c:    48 09 ca                 or     %rcx,%rdx
     29f:    f0 48 0f b1 16           lock cmpxchg %rdx,(%rsi)
     2a4:    75 d2                    jne    278 <xlog_grant_add_space+0x8>

And as demonstrated in [1], even the fallback code compiles to a better assembly.

[1] https://lore.kernel.org/lkml/CAFULd4bc54+_FmJ=f++zzz99mR8r5c11-Y49pz86Yb8G3dyJpA@mail.gmail.com/

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

This patch has no functional change.

The patch looks ok, but ....

Quoted 8 lines
Quoted 4 lines

... I'm about 2 hours away from posting a patchset that completely removes the cmpxchg and the new grant head accounting has significantly lower fast path overhead. It also opens the door for tracking more than 2GB of log space in the grant heads.

FYI, the original RFC for this was posted a bit over a month ago:

https://lore.kernel.org/linux-xfs/[email protected]/

-static void
+void
xlog_grant_sub_space(

[...]

- old = head_val;
- new = xlog_assign_grant_head_val(cycle, space);
- head_val = atomic64_cmpxchg(&head->grant, old, new);
- } while (head_val != old);
+ atomic64_sub(bytes, &head->grant);
}

I actually wondered why these two functions were not implemented as atomic64_{add,sub}. Of course, this is a much better solution that renders the proposed patch obsolete.

BR,
Uros.

#4 lore raw
Quoted 12 lines · Uros Bizjak
Quoted 11 lines · Dave Chinner
Quoted 9 lines · Dave Chinner
Quoted 8 lines · Uros Bizjak

Use !atomic64_try_cmpxchg(ptr, &old, new) instead of atomic64_cmpxchg(ptr, old, new) != old in xlog_grant_{add,sub}_space. This has two benefits:

- The x86 cmpxchg instruction returns success in the ZF flag, so this change saves a compare after cmpxchg, as well as a related move instruction in the front of cmpxchg.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

- atomic64_try_cmpxchg implicitly assigns the *ptr value to &old when cmpxchg fails, enabling further code simplifications.

Do the two cmpxchg operations have the same memory ordering semantics on failure?

Yes. The API also provides _acquire, _release and _relaxed variants of both, atomic64_cmpxchg and atomic64_try_cmpxchg.

Yes, I know this, which was why I was asking if the default behaviour is the same - many people get caught out by assuming that cmpxchg implies release semantics even if it fails....

Quoted 10 lines

On x86, these two
functions actually compile to the same CMPXCHG instruction, the difference is only in how the comparison is handled:

      15:    48 09 c2                 or     %rax,%rdx
      18:    48 89 c8                 mov    %rcx,%rax
      1b:    f0 48 0f b1 16           lock cmpxchg %rdx,(%rsi)
      20:    48 39 c1                 cmp    %rax,%rcx
      23:    74 2a                    je     4f <xlog_grant_add_space+0x4f>
Quoted 8 lines

becomes:

     29c:    48 09 ca                 or     %rcx,%rdx
     29f:    f0 48 0f b1 16           lock cmpxchg %rdx,(%rsi)
     2a4:    75 d2                    jne    278 <xlog_grant_add_space+0x8>

And as demonstrated in [1], even the fallback code compiles to a better assembly.o

FWIW, I mostly don't care about assembler level optimisations for the code I write. I'll try to write code efficiently, but I don't really care that much for micro-optimisation. Fundamentally, focussing on optimising code down to the instruction level means you are not looking for algorithmic optimisations, which is where all the big gains typically come from....

This thread demonstrates that - I get an improvement from roughly 1.4 million transactions/s to roughly 2 million transactions/s with the change to the grant head accounting algorithm, whilst the improvement from removing 2 instructions from the cmpxchg can't actually be measured on my tests. i.e. the improvement is lost within the noise floor of the benchmarks.

So, yeah, when it comes to making code faster, I focus on efficient algorithms rather than efficient code because algorithms are where the gains users will notice are....

Quoted 17 lines
Quoted 3 lines

FYI, the original RFC for this was posted a bit over a month ago:

https://lore.kernel.org/linux-xfs/[email protected]/

(quoted diff) +2 -5
-static void
+void
 xlog_grant_sub_space(
 
[...]
 
- old = head_val;
- new = xlog_assign_grant_head_val(cycle, space);
- head_val = atomic64_cmpxchg(&head->grant, old, new);
- } while (head_val != old);
+ atomic64_sub(bytes, &head->grant);
 }
Quoted 2 lines

I actually wondered why these two functions were not implemented as atomic64_{add,sub}.

Because the grant heads were not integer values that can be added and subtracted. Log sequence numbers (LSNs) are 64 bit objects made up of two discrete 32 bit values. Essentially the upper 32 bits counts the number of overflows of the fixed size space the lower 32 bits accounts.

The lower 32 bits matches the size of the journal, so will overflow at some boundary much lower than 2^32. Hence adding or subtracting to a LSN has to handle the space overflow/underflow itself to modify the overflow (cycle) counter top 32 bits appropriately.

These calculations cannot be done as a single atomic operation, hence the crack/calc/combine/cmpxchg loops to enable them to be done without requiring locks in the fast path.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]