CodeSamples/count: change full memory barrier to partial one
PATCH v2 1/1 CodeSamples/count: change full memory barrier to partial one
The original memory barriers in count_stat_eventual.c ensure writing to global_count happens before writing to stopflag and reading from stopflag happens before later reading from global_count.
Thus, smp_load_acquire and smp_store_release will suffice.
In count_lim_sig.c, there is only one ordering required, that is writing to counter happens before setting theft to THEFT_READY in add_count/sub_count's fast path. Therefore, partial memory barrier will suffice.
--- CodeSamples/count/count_lim_sig.c | 10 +++------- CodeSamples/count/count_stat_eventual.c | 6 ++---- count/count.tex | 9 ++++----- 3 files changed, 9 insertions(+), 16 deletions(-)
CodeSamples/count/count_lim_sig.c +3 -7
@@ -56,12 +56,10 @@ static void flush_local_count_sig(int unused) //\lnlbl{flush_sig:b} {if (READ_ONCE (theft) !=THEFT_REQ )//\lnlbl{flush_sig:check:REQ} return ;//\lnlbl{flush_sig:return:n} -smp_mb ();//\lnlbl{flush_sig:mb:1} WRITE_ONCE (theft,THEFT_ACK );//\lnlbl{flush_sig:set:ACK} if (!counting) {//\lnlbl{flush_sig:check:fast} -WRITE_ONCE (theft,THEFT_READY );//\lnlbl{flush_sig:set:READY} +smp_store_release (&theft,THEFT_READY );//\lnlbl{flush_sig:set:READY} } -smp_mb (); }//\lnlbl{flush_sig:e} static void flush_local_count (void )//\lnlbl{flush:b}
@@ -125,8 +123,7 @@ int add_count(unsigned long delta) //\lnlbl{b}WRITE_ONCE (counting,0 );//\lnlbl{clearcnt} barrier ();//\lnlbl{barrier:3} if (READ_ONCE (theft) ==THEFT_ACK ) {//\lnlbl{check:ACK} -smp_mb ();//\lnlbl{mb} -WRITE_ONCE (theft,THEFT_READY );//\lnlbl{READY} +smp_store_release (&theft,THEFT_READY );//\lnlbl{READY} }if (fastpath)return 1 ;//\lnlbl{return:fs}
@@ -164,8 +161,7 @@ int sub_count(unsigned long delta)WRITE_ONCE (counting,0 );barrier ();if (READ_ONCE (theft) ==THEFT_ACK ) { -smp_mb (); -WRITE_ONCE (theft,THEFT_READY ); +smp_store_release (&theft,THEFT_READY ); }if (fastpath)return 1 ;
CodeSamples/count/count_stat_eventual.c +2 -4
@@ -51,8 +51,7 @@ void *eventual(void *arg) //\lnlbl{eventual:b}WRITE_ONCE (global_count, sum);poll (NULL ,0 ,1 );if (READ_ONCE (stopflag)) { -smp_mb (); -WRITE_ONCE (stopflag, stopflag +1 ); +smp_store_release (&stopflag, stopflag +1 ); } }return NULL ;
@@ -73,9 +72,8 @@ void count_init(void) //\lnlbl{init:b}void count_cleanup (void )//\lnlbl{cleanup:b} {WRITE_ONCE (stopflag,1 ); -while (READ_ONCE (stopflag) <3 ) +while (smp_load_acquire (&stopflag) <3 )poll (NULL ,0 ,1 ); -smp_mb (); }//\lnlbl{cleanup:e} //\end{snippet}
count/count.tex +4 -5
@@ -2425,12 +2425,11 @@ handler used in the theft process. \Clnref{check:REQ,return:n} check to see if the \co{theft} state is REQ, and, if not returns without change. -\Clnref{mb:1} executes a \IX{memory barrier} to ensure that the sampling -of the theft variable happens before any change to that variable. \Clnref{set:ACK} sets the \co{theft} state to ACK, and, if \clnref{check:fast} sees that this thread's fastpaths are not running, \clnref{set:READY} sets the \co{theft} -state to READY\@. +state to READY, the release store here is to ensure that the change to counter +in fast path happens before changing the theft varibale to READY\@. \end{fcvref} \begin{listing}
@@ -2595,8 +2594,8 @@ handlers to undertake theft. \Clnref{barrier:3} again disables compiler reordering, and then \clnref{check:ACK} checks to see if the signal handler deferred the \co{theft} -state-change to READY, and, if so, \clnref{mb} executes a memory -barrier to ensure that any CPU that sees \clnref{READY} setting state to +state-change to READY, and, if so, the release store at \clnref{READY} +is to ensure that any CPU that sees \clnref{READY} setting state to READY also sees the effects of \clnref{add:f}. If the fastpath addition at \clnref{add:f} was executed, then \clnref{return:fs} returns
Signature · 1 line
2.34.1
Hi Alan,
Quoted 5 lines · Alan Huang
The original memory barriers in
count_stat_eventual.censure writing toglobal_counthappens before writing to stopflag and reading from stopflag happens before later reading fromglobal_count.Thus,
smp_load_acquireandsmp_store_releasewill suffice.
So far, looks good.
But those smp_mb()s are mentioned in Section 5.2.4. Please see inline comments below:
Quoted 4 lines
In
count_lim_sig.c, there is only one ordering required, that is writing to counter happens before setting theft toTHEFT_READYin add_count/sub_count's fast path.
I'm kind of not 100% sure of your changes in count_lim_sig.c. I'll comment in another mail. That might take a while.
Quoted 10 lines
Therefore, partial memory barrier
will suffice.Signed-off-by Alan Huang <[email protected]>--- CodeSamples/count/count_lim_sig.c | 10 +++------- CodeSamples/count/count_stat_eventual.c | 6 ++---- count/count.tex | 9 ++++----- 3 files changed, 9 insertions(+), 16 deletions(-)
[...]
Quoted 15 lines
CodeSamples/count/count_stat_eventual.c +1
-2@@ -51,8 +51,7 @@ void *eventual(void *arg) //\lnlbl{eventual:b}WRITE_ONCE (global_count, sum);poll (NULL ,0 ,1 );if (READ_ONCE (stopflag)) { -smp_mb (); -WRITE_ONCE (stopflag, stopflag +1 ); +smp_store_release (&stopflag, stopflag +1 ); } }return NULL ;
CodeSamples/count/count_stat_eventual.c void count_init(void) //\lnlbl{init:b} · line 78
Quoted 9 lines
CodeSamples/count/count_stat_eventual.c +1
-2@@ -73,9 +72,8 @@ void count_init(void) //\lnlbl{init:b}void count_cleanup (void )//\lnlbl{cleanup:b} {WRITE_ONCE (stopflag,1 ); -while (READ_ONCE (stopflag) <3 ) +while (smp_load_acquire (&stopflag) <3 )poll (NULL ,0 ,1 ); -smp_mb (); }//\lnlbl{cleanup:e} //\end{snippet}
So, you'd also need to update a paragraph in Section 5.2.4, which currently reads:
The count_cleanup() function on lines 48--54 coordinates termination. The calls to smp_mb() here and in eventual() ensure that all updates to global_count are visible to code following the call to count_cleanup().
And I think splitting the changes in count_stat_eventual.c and count_lim_sig.c into individual patches would ease the review on our side.
So, can you send a patch concerning count_stat_eventual.c first ?
I'll comment on count_lim_sig.c later, that is, if Paul won't beat me to it.
Thanks, Akira
Hi Akira,
Thanks for your review!
I’ll send a patch only concerning count_stat_eventual.c later.
Thanks,
Alan
Quoted 81 lines
2023年4月5日 下午10:08,Akira Yokosawa <[email protected]> 写道:
Hi Alan,
Quoted 5 lines · Alan Huang
The original memory barriers in
count_stat_eventual.censure writing toglobal_counthappens before writing to stopflag and reading from stopflag happens before later reading fromglobal_count.Thus,
smp_load_acquireandsmp_store_releasewill suffice.So far, looks good.
But those smp_mb()s are mentioned in Section 5.2.4. Please see inline comments below:Quoted 4 lines
In
count_lim_sig.c, there is only one ordering required, that is writing to counter happens before setting theft toTHEFT_READYin add_count/sub_count's fast path.I'm kind of not 100% sure of your changes in
count_lim_sig.c. I'll comment in another mail. That might take a while.Quoted 10 lines
Therefore, partial memory barrier
will suffice.Signed-off-by Alan Huang <[email protected]>---
CodeSamples/count/count_lim_sig.c | 10 +++------- CodeSamples/count/count_stat_eventual.c | 6 ++---- count/count.tex | 9 ++++----- 3 files changed, 9 insertions(+), 16 deletions(-)[...]
Quoted 15 lines
CodeSamples/count/count_stat_eventual.c +1
-2@@ -51,8 +51,7 @@ void *eventual(void *arg) //\lnlbl{eventual:b}WRITE_ONCE (global_count, sum);poll (NULL ,0 ,1 );if (READ_ONCE (stopflag)) { -smp_mb (); -WRITE_ONCE (stopflag, stopflag +1 ); +smp_store_release (&stopflag, stopflag +1 ); } }return NULL ;CodeSamples/count/count_stat_eventual.c void count_init(void) //\lnlbl{init:b} · line 78
Quoted 9 lines
CodeSamples/count/count_stat_eventual.c +1
-2@@ -73,9 +72,8 @@ void count_init(void) //\lnlbl{init:b}void count_cleanup (void )//\lnlbl{cleanup:b} {WRITE_ONCE (stopflag,1 ); -while (READ_ONCE (stopflag) <3 ) +while (smp_load_acquire (&stopflag) <3 )poll (NULL ,0 ,1 ); -smp_mb (); }//\lnlbl{cleanup:e} //\end{snippet} So, you'd also need to update a paragraph in Section 5.2.4, which currently reads:
The
count_cleanup()function on lines 48--54 coordinates termination. The calls tosmp_mb()here and ineventual()ensure that all updates toglobal_countare visible to code following the call tocount_cleanup().And I think splitting the changes in
count_stat_eventual.candcount_lim_sig.cinto individual patches would ease the review on our side.So, can you send a patch concerning
count_stat_eventual.cfirst ?I'll comment on
count_lim_sig.clater, that is, if Paul won't beat me to it.Thanks, Akira
Quoted 10 lines · Alan Huang
The original memory barriers in
count_stat_eventual.censure writing toglobal_counthappens before writing to stopflag and reading from stopflag happens before later reading fromglobal_count.Thus,
smp_load_acquireandsmp_store_releasewill suffice.In
count_lim_sig.c, there is only one ordering required, that is writing to counter happens before setting theft toTHEFT_READYin add_count/sub_count's fast path. Therefore, partial memory barrier will suffice.
Hi Alan,
My comment on count_lim_sig.c here. See below
Quoted 14 lines
Signed-off-by Alan Huang <[email protected]>--- CodeSamples/count/count_lim_sig.c | 10 +++------- CodeSamples/count/count_stat_eventual.c | 6 ++---- count/count.tex | 9 ++++----- 3 files changed, 9 insertions(+), 16 deletions(-)CodeSamples/count/count_lim_sig.c +0
-0@@ -56,12 +56,10 @@ static void flush_local_count_sig(int unused) //\lnlbl{flush_sig:b} {
CodeSamples/count/count_lim_sig.c static void flush_local_count_sig(int unused) //\lnlbl{flush_sig:b} · line 63
Quoted 10 lines
CodeSamples/count/count_lim_sig.c +1
-3if (READ_ONCE (theft) !=THEFT_REQ )//\lnlbl{flush_sig:check:REQ} return ;//\lnlbl{flush_sig:return:n} -smp_mb ();//\lnlbl{flush_sig:mb:1} WRITE_ONCE (theft,THEFT_ACK );//\lnlbl{flush_sig:set:ACK} if (!counting) {//\lnlbl{flush_sig:check:fast} -WRITE_ONCE (theft,THEFT_READY );//\lnlbl{flush_sig:set:READY} +smp_store_release (&theft,THEFT_READY );//\lnlbl{flush_sig:set:READY} } -smp_mb (); }//\lnlbl{flush_sig:e}
You see, this a signal handler.
It can interrupt anywhere in normal threads.
That is why you see a lot of barrier() calls in updater functions.
This code has no assumption on memory barriers around signal handlers.
So, I'd rather keep smp_mb()s in this function.
Do you have a good reasoning why the pair of smp_mb()s can be safely replaced with the single smp_store_release() ?
Quoted 12 lines
CodeSamples/count/count_lim_sig.c +1
-2static void flush_local_count (void )//\lnlbl{flush:b} @@ -125,8 +123,7 @@ int add_count(unsigned long delta) //\lnlbl{b}WRITE_ONCE (counting,0 );//\lnlbl{clearcnt} barrier ();//\lnlbl{barrier:3} if (READ_ONCE (theft) ==THEFT_ACK ) {//\lnlbl{check:ACK} -smp_mb ();//\lnlbl{mb} -WRITE_ONCE (theft,THEFT_READY );//\lnlbl{READY} +smp_store_release (&theft,THEFT_READY );//\lnlbl{READY} }> if (fastpath)return 1 ;//\lnlbl{return:fs}
CodeSamples/count/count_lim_sig.c int sub_count(unsigned long delta) · line 167
Quoted 9 lines
CodeSamples/count/count_lim_sig.c +1
-2@@ -164,8 +161,7 @@ int sub_count(unsigned long delta)WRITE_ONCE (counting,0 );barrier ();if (READ_ONCE (theft) ==THEFT_ACK ) { -smp_mb (); -WRITE_ONCE (theft,THEFT_READY ); +smp_store_release (&theft,THEFT_READY ); }if (fastpath)return 1 ;
These changes in updater functions look good to me.
I'm stopping here. Let's reach a consensus on code changes first.
Thanks, Akira
[...]
Hi Akira,
Quoted 2 lines
Do you have a good reasoning why the pair of smp_mb()s can be safely replaced with the single
smp_store_release()?
When the signal handler is running, there is only one thread waiting on the while loop in flush_local_count, which means we only need to synchronize with that thread, and that thread only cares the THEFT_READY status, we just need to make sure after that thread seeing theft equals THEFT_READY, it can also see the changes to counter.
And because the signal handler is running on the interrupted thread, we don’t need any memory barrier.
Please correct me if I'm wrong.
Thanks,
Alan
Quoted 84 lines
2023年4月6日 下午10:33,Akira Yokosawa <[email protected]> 写道:
Quoted 10 lines · Alan Huang
The original memory barriers in
count_stat_eventual.censure writing toglobal_counthappens before writing to stopflag and reading from stopflag happens before later reading fromglobal_count.Thus,
smp_load_acquireandsmp_store_releasewill suffice.In
count_lim_sig.c, there is only one ordering required, that is writing to counter happens before setting theft toTHEFT_READYin add_count/sub_count's fast path. Therefore, partial memory barrier will suffice.Hi Alan,
My comment on
count_lim_sig.chere. See belowQuoted 14 lines
Signed-off-by Alan Huang <[email protected]>---
CodeSamples/count/count_lim_sig.c | 10 +++------- CodeSamples/count/count_stat_eventual.c | 6 ++---- count/count.tex | 9 ++++----- 3 files changed, 9 insertions(+), 16 deletions(-)CodeSamples/count/count_lim_sig.c +0
-0@@ -56,12 +56,10 @@ static void flush_local_count_sig(int unused) //\lnlbl{flush_sig:b} {CodeSamples/count/count_lim_sig.c static void flush_local_count_sig(int unused) //\lnlbl{flush_sig:b} · line 63
Quoted 10 lines
CodeSamples/count/count_lim_sig.c +1
-3if (READ_ONCE (theft) !=THEFT_REQ )//\lnlbl{flush_sig:check:REQ} return ;//\lnlbl{flush_sig:return:n} -smp_mb ();//\lnlbl{flush_sig:mb:1} WRITE_ONCE (theft,THEFT_ACK );//\lnlbl{flush_sig:set:ACK} if (!counting) {//\lnlbl{flush_sig:check:fast} -WRITE_ONCE (theft,THEFT_READY );//\lnlbl{flush_sig:set:READY} +smp_store_release (&theft,THEFT_READY );//\lnlbl{flush_sig:set:READY} } -smp_mb (); }//\lnlbl{flush_sig:e} You see, this a signal handler.
It can interrupt anywhere in normal threads.That is why you see a lot of
barrier()calls in updater functions.This code has no assumption on memory barriers around signal handlers.
So, I'd rather keep smp_mb()s in this function.
Do you have a good reasoning why the pair of smp_mb()s can be safely replaced with the single
smp_store_release()?Quoted 12 lines
CodeSamples/count/count_lim_sig.c +1
-2static void flush_local_count (void )//\lnlbl{flush:b} @@ -125,8 +123,7 @@ int add_count(unsigned long delta) //\lnlbl{b}WRITE_ONCE (counting,0 );//\lnlbl{clearcnt} barrier ();//\lnlbl{barrier:3} if (READ_ONCE (theft) ==THEFT_ACK ) {//\lnlbl{check:ACK} -smp_mb ();//\lnlbl{mb} -WRITE_ONCE (theft,THEFT_READY );//\lnlbl{READY} +smp_store_release (&theft,THEFT_READY );//\lnlbl{READY} }> if (fastpath)return 1 ;//\lnlbl{return:fs} CodeSamples/count/count_lim_sig.c int sub_count(unsigned long delta) · line 167
Quoted 9 lines
CodeSamples/count/count_lim_sig.c +1
-2@@ -164,8 +161,7 @@ int sub_count(unsigned long delta)WRITE_ONCE (counting,0 );barrier ();if (READ_ONCE (theft) ==THEFT_ACK ) { -smp_mb (); -WRITE_ONCE (theft,THEFT_READY ); +smp_store_release (&theft,THEFT_READY ); }if (fastpath)return 1 ;These changes in updater functions look good to me.
I'm stopping here. Let's reach a consensus on code changes first.Thanks, Akira
[...]
Quoted 10 lines · Alan Huang
Hi Akira,
Quoted 2 lines
Do you have a good reasoning why the pair of smp_mb()s can be safely replaced with the single
smp_store_release()?When the signal handler is running, there is only one thread waiting on the while loop in
flush_local_count, which means we only need to synchronize with that thread, and that thread only cares theTHEFT_READYstatus, we just need to make sure after that thread seeing theft equalsTHEFT_READY, it can also see the changes to counter.And because the signal handler is running on the interrupted thread, we don’t need any memory barrier.
OK, now I'm convinced. You are right!
Let's see your changes in corresponding text:
Quoted 6 lines
count/count.tex +0
-0@@ -2425,12 +2425,11 @@ handler used in the theft process. \Clnref{check:REQ,return:n} check to see if
count/count.tex handler used in the theft process. · line 2432
Quoted 10 lines
count/count.tex +2
-3the \co{theft} state is REQ, and, if not returns without change. -\Clnref{mb:1} executes a \IX{memory barrier} to ensure that the sampling -of the theft variable happens before any change to that variable. \Clnref{set:ACK} sets the \co{theft} state to ACK, and, if \clnref{check:fast} sees that this thread's fastpaths are not running, \clnref{set:READY} sets the \co{theft} -state to READY\@. +state to READY, the release store here is to ensure that the change to counter +in fast path happens before changing the theft varibale to READY\@.
the fastpath variable
I can't parse this sentence grammatically.
This sentence is rendered as follows (with minor nits fixed):
Line 13 sets the theft state to ACK, and, if line 14 sees that this thread's fastpaths are not running, line 15 sets the theft state to READY, the release store here is to ensure that the change to counter in the fastpath happens before changing the theft variable to READY.
How about this?
Line 13 sets the theft state to ACK, and, if line 14 sees that this thread's fastpaths are not running, line 15 sets the theft state to READY, with the release store ensuring any change to counter in the fastpath happens before the change of theft to READY.
Also, I'd like to see your reasoning above mentioned somewhere around here. Might worth a Quick Quiz.
Question:
Doesn't flush_local_count_sig() need a full memory barrier?
count/count.tex handlers to undertake theft. · line 2599
Quoted 12 lines
count/count.tex +2
-2\end{fcvref} \begin{listing}@@ -2595,8 +2594,8 @@ handlers to undertake theft. \Clnref{barrier:3} again disables compiler reordering, and then \clnref{check:ACK} checks to see if the signal handler deferred the \co{theft} -state-change to READY, and, if so, \clnref{mb} executes a memory -barrier to ensure that any CPU that sees \clnref{READY} setting state to +state-change to READY, and, if so, the release store at \clnref{READY} +is to ensure that any CPU that sees \clnref{READY} setting state to READY also sees the effects of \clnref{add:f}.
Again, this looks to me incomplete in grammatical structure.
This sentence is rendered as follows:
Line 14 again disables compiler reordering, and then line 15 checks to see if the signal handler deferred the theft state-change to READY, and, if so, the release store at line 16 is to ensure that any CPU that sees line 16 setting state to READY also sees the effects of line 9.
How about something like this?
Line 14 again disables compiler reordering, and then line 15 checks to see if the signal handler deferred the theft state-change to READY, and, if so, line 16 changes theft to READY with the release store ensuring that any CPU that sees the READY state also sees the effects of line 9.
Paul might have better ideas. ;-)
count/count.tex handlers to undertake theft. · line 2601
Quoted 2 lines
count/count.tex +0
-0If the fastpath addition at \clnref{add:f} was executed, then \clnref{return:fs} returns
Can you post a v3 with whatever text-side updates you come up with?
Thanks, Akira
Quoted 6 lines
Please correct me if I'm wrong.
Thanks,
Alan
Thanks for the suggestion!
I’ll send patch v3 later.
Thanks,
Alan
Quoted 106 lines
2023年4月7日 10:14,Akira Yokosawa <[email protected]> 写道:
Quoted 10 lines · Alan Huang
Hi Akira,
Quoted 2 lines
Do you have a good reasoning why the pair of smp_mb()s can be safely replaced with the single
smp_store_release()?When the signal handler is running, there is only one thread waiting on the while loop in
flush_local_count, which means we only need to synchronize with that thread, and that thread only cares theTHEFT_READYstatus, we just need to make sure after that thread seeing theft equalsTHEFT_READY, it can also see the changes to counter.And because the signal handler is running on the interrupted thread, we don’t need any memory barrier.
OK, now I'm convinced. You are right!
Let's see your changes in corresponding text:
Quoted 13 lines
count/count.tex +0
-0@@ -2425,12 +2425,11 @@ handler used in the theft process.\Clnref{check:REQ,return:n} check to see if
the \co{theft} state is REQ, and, if not
returns without change.
-\Clnref{mb:1} executes a \IX{memory barrier} to ensure that the sampling -of the theft variable happens before any change to that variable. \Clnref{set:ACK} sets the \co{theft} state to ACK, and, if \clnref{check:fast} sees that
this thread's fastpaths are not running, \clnref{set:READY} sets the \co{theft}count/count.tex handler used in the theft process. · line 2432
Quoted 3 lines
count/count.tex +2
-1-state to READY\@. +state to READY, the release store here is to ensure that the change to counter +in fast path happens before changing the theft varibale to READY\@.the fastpath variable
I can't parse this sentence grammatically.
This sentence is rendered as follows (with minor nits fixed):
Line 13 sets the theft state to ACK, and, if line 14 sees that this thread's fastpaths are not running, line 15 sets the theft state to READY, the release store here is to ensure that the change to counter in the fastpath happens before changing the theft variable to READY.
How about this?
Line 13 sets the theft state to ACK, and, if line 14 sees that this thread's fastpaths are not running, line 15 sets the theft state to READY, with the release store ensuring any change to counter in the fastpath happens before the change of theft to READY.
Also, I'd like to see your reasoning above mentioned somewhere around here. Might worth a Quick Quiz.
Question:
Doesn'tflush_local_count_sig()need a full memory barrier?Quoted 11 lines
\end{fcvref}
\begin{listing}
count/count.tex +0
-0@@ -2595,8 +2594,8 @@ handlers to undertake theft.\Clnref{barrier:3} again disables compiler reordering, and then \clnref{check:ACK}
checks to see if the signal handler deferred the \co{theft}count/count.tex +2
-2-state-change to READY, and, if so, \clnref{mb} executes a memory -barrier to ensure that any CPU that sees \clnref{READY} setting state to +state-change to READY, and, if so, the release store at \clnref{READY} +is to ensure that any CPU that sees \clnref{READY} setting state toQuoted 1 line
READY also sees the effects of \clnref{add:f}.
Again, this looks to me incomplete in grammatical structure.
This sentence is rendered as follows:
Line 14 again disables compiler reordering, and then line 15 checks to see if the signal handler deferred the theft state-change to READY, and, if so, the release store at line 16 is to ensure that any CPU that sees line 16 setting state to READY also sees the effects of line 9.
How about something like this?
Line 14 again disables compiler reordering, and then line 15 checks to see if the signal handler deferred the theft state-change to READY, and, if so, line 16 changes theft to READY with the release store ensuring that any CPU that sees the READY state also sees the effects of line 9.
Paul might have better ideas. ;-)
Quoted 2 lines
If the fastpath addition at \clnref{add:f} was executed, then \clnref{return:fs} returns
Can you post a v3 with whatever text-side updates you come up with?
Thanks, Akira
Quoted 5 lines
Please correct me if I'm wrong.
Thanks,
Alan