x_tables: Properly close read section with read_seqcount_retry
Subsystems (2)
PATCH 1/1 x_tables: Properly close read section with read_seqcount_retry
xtables uses a modified version of a seqcount lock for synchronization between replacing private table information and the packet processing path (i.e. XX_do_table). The main modification is in the "write" semantics, where instead of adding 1 for each write, the write side will add 1 only if it finds no other writes ongoing, and adds 1 again (thereby clearing the LSB) when it finishes.
This allows the read side to avoid calling read_seqcount_begin() in a loop if a write is detected, since the value is guaranteed to only increment once all writes have completed. As such, it is only necessary to check if the initial value of the sequence count has changed to inform the reader that all writes are finished.
However, the current check for the changed value uses the wrong API; raw_seqcount_read() is protected by smp_rmb() in the same way as read_seqcount_begin(), making it appropriate for ENTERING read-side critical sections, but not exiting them. For that, read_seqcount_rety() must be used to ensure the proper barrier placement for synchronization with xt_write_recseq_end() (itself modeled after write_seqcount_end()).
--- net/netfilter/x_tables.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/netfilter/x_tables.c +1 -1
@@ -1404,7 +1404,7 @@ xt_replace_table(struct xt_table *table,do {cond_resched ();cpu_relax (); - }while (seq ==raw_read_seqcount (s)); + }while (!read_seqcount_retry (s, seq)); } }
Signature · 1 line
2.7.4
Florian Westphal
maintainerQuoted 19 lines · Sean Tranchetti
xtables uses a modified version of a seqcount lock for synchronization between replacing private table information and the packet processing path (i.e.
XX_do_table). The main modification is in the "write" semantics, where instead of adding 1 for each write, the write side will add 1 only if it finds no other writes ongoing, and adds 1 again (thereby clearing the LSB) when it finishes.This allows the read side to avoid calling
read_seqcount_begin()in a loop if a write is detected, since the value is guaranteed to only increment once all writes have completed. As such, it is only necessary to check if the initial value of the sequence count has changed to inform the reader that all writes are finished.However, the current check for the changed value uses the wrong API;
raw_seqcount_read()is protected bysmp_rmb()in the same way asread_seqcount_begin(), making it appropriate for ENTERING read-side critical sections, but not exiting them. For that,read_seqcount_rety()must be used to ensure the proper barrier placement for synchronization withxt_write_recseq_end()(itself modeled afterwrite_seqcount_end()).
[..]
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1407
Quoted 10 lines
net/netfilter/x_tables.c +1
-1@@ -1404,7 +1404,7 @@ xt_replace_table(struct xt_table *table,do {cond_resched ();cpu_relax (); - }while (seq ==raw_read_seqcount (s)); + }while (!read_seqcount_retry (s, seq));
I'm fine with this change but AFAIU this is just a cleanup since this part isn't a read-sequence as no 'shared state' is accessed/read between the seqcount begin and the do{}while. smb_rmb placement should not matter here.
Did I miss anything?
Thanks.
Quoted 7 lines
I'm fine with this change but AFAIU this is just a cleanup since this part isn't a read-sequence as no 'shared state' is accessed/read between the seqcount begin and the do{}while.
smb_rmbplacement should not matter here.Did I miss anything?
Thanks.
Hi Florian
To provide more background on this, we are seeing occasional crashes in a regression rack in the packet receive path where there appear to be some rules being modified concurrently.
Unable to handle kernel NULL pointer dereference at virtual address 000000000000008e pc : ip6t_do_table+0x5d0/0x89c lr : ip6t_do_table+0x5b8/0x89c ip6t_do_table+0x5d0/0x89c ip6table_filter_hook+0x24/0x30 nf_hook_slow+0x84/0x120 ip6_input+0x74/0xe0 ip6_rcv_finish+0x7c/0x128 ipv6_rcv+0xac/0xe4 __netif_receive_skb+0x84/0x17c process_backlog+0x15c/0x1b8 napi_poll+0x88/0x284 net_rx_action+0xbc/0x23c __do_softirq+0x20c/0x48c
We found that ip6t_do_table was reading stale values from the READ_ONCE leading to use after free when dereferencing per CPU jumpstack values.
addend = xt_write_recseq_begin(); private = READ_ONCE(table->private); /* Address dependency. */
We were able to confirm that the xt_replace_table & __do_replace
had already executed by logging the seqcount values. The value of seqcount at ip6t_do_table was 1 more than the value at xt_replace_table. The seqcount read at xt_replace_table also showed that it was an even value and hence meant that there was no conccurent writer instance (xt_write_recseq_begin) at that point.
u32 seq = raw_read_seqcount(s);
This means that table assignment at xt_replace_table did not take effect as expected.
smp_wmb(); table->private = newinfo; /* make sure all cpus see new ->private value */ smp_wmb();
We want to know if this barrier usage is as expected here. Alternatively, would changing this help here -
net/netfilter/x_tables.c +3 -3
@@ -1431,11 +1431,11 @@ xt_replace_table(struct xt_table *table,* Ensure contents of newinfo are visible before assigning to * private. */ -smp_wmb (); - table->private = newinfo; +smp_mb (); +WRITE_ONCE (table->private, newinfo);/* make sure all cpus see new ->private value */ -smp_wmb (); +smp_mb ();/* * Even though table entries have now been swapped, other CPU's
Florian Westphal
maintainerQuoted 35 lines · [email protected] <[email protected]> wrote:
Quoted 9 lines
I'm fine with this change but AFAIU this is just a cleanup since this part isn't a read-sequence as no 'shared state' is accessed/read between
the seqcount begin and the do{}while.smb_rmbplacement should not matter here.Did I miss anything?
Thanks.
Hi Florian
To provide more background on this, we are seeing occasional crashes in a regression rack in the packet receive path where there appear to be some rules being modified concurrently.
Unable to handle kernel NULL pointer dereference at virtual address 000000000000008e pc : ip6t_do_table+0x5d0/0x89c lr : ip6t_do_table+0x5b8/0x89c ip6t_do_table+0x5d0/0x89c ip6table_filter_hook+0x24/0x30 nf_hook_slow+0x84/0x120 ip6_input+0x74/0xe0 ip6_rcv_finish+0x7c/0x128 ipv6_rcv+0xac/0xe4 __netif_receive_skb+0x84/0x17c process_backlog+0x15c/0x1b8 napi_poll+0x88/0x284 net_rx_action+0xbc/0x23c __do_softirq+0x20c/0x48cWe found that
ip6t_do_tablewas reading stale values from theREAD_ONCEleading to use after free when dereferencing per CPU jumpstack values.
Quoted 7 lines
addend = xt_write_recseq_begin(); private = READ_ONCE(table->private); /* Address dependency. */We were able to confirm that the
xt_replace_table&__do_replacehad already executed by logging the seqcount values. The value of seqcount atip6t_do_tablewas 1 more than the value atxt_replace_table.
Can you elaborate? Was that before xt_write_recseq_begin() or after?
Quoted 4 lines
The seqcount read at
xt_replace_tablealso showed that it was an even value and hence meant that there was no conccurent writer instance (xt_write_recseq_begin)
at that point.
Ok, so either ip6_do_table was done or just starting (and it should not matter which one).
Quoted 5 lines
u32 seq = raw_read_seqcount(s);This means that table assignment at
xt_replace_tabledid not take effect as expected.
You mean
"private = READ_ONCE(table->private); /* Address dependency. */" in ip6_do_table did pick up the 'old' private pointer, AFTER xt_replace had updated it?
Quoted 15 lines
smp_wmb(); table->private = newinfo; /* make sure all cpus see new ->private value */ smp_wmb();We want to know if this barrier usage is as expected here. Alternatively, would changing this help here -
net/netfilter/x_tables.c +0
-0
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1435
Quoted 7 lines
net/netfilter/x_tables.c +2
-2@@ -1431,11 +1431,11 @@ xt_replace_table(struct xt_table *table,* Ensure contents of newinfo are visible before assigning to * private. */ -smp_wmb (); - table->private = newinfo; +smp_mb (); +WRITE_ONCE (table->private, newinfo);
The WRITE_ONCE looks correct.
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1438
Quoted 3 lines
net/netfilter/x_tables.c +1
-1/* make sure all cpus see new ->private value */ -smp_wmb (); +smp_mb ();
Is that to order wrt. seqcount_sequence?
Do you have a way to reproduce such crashes?
I tried to no avail but I guess thats just because amd64 is more forgiving.
Thanks!
Quoted 7 lines
Quoted 7 lines
We found that
ip6t_do_tablewas reading stale values from theREAD_ONCEleading to use after free when dereferencing per CPU jumpstack values.addend = xt_write_recseq_begin(); private = READ_ONCE(table->private); /* Address dependency. */
Quoted 5 lines
Quoted 3 lines
We were able to confirm that the
xt_replace_table&__do_replacehad already executed by logging the seqcount values. The value of seqcount atip6t_do_tablewas 1 more than the value atxt_replace_table.Can you elaborate? Was that before
xt_write_recseq_begin()or after?
The log in ip6t_do_table was after xt_write_recseq_begin().
addend = xt_write_recseq_begin();
The log in xt_replace_table was after raw_read_seqcount of per_cpu(xt_recseq)
seqcount_t *s = &per_cpu(xt_recseq, cpu); u32 seq = raw_read_seqcount(s);
In this specific run, we observed that the value in ip6t_do_table after xt_write_recseq_begin() was 75316887. The value in xt_replace_table was after raw_read_seqcount of per_cpu(xt_recseq) was 75316886. This confirms that the private pointer assignment in xt_replace_table happened before the start of ip6t_do_table.
Quoted 5 lines
You mean
"private =READ_ONCE(table->private); /* Address dependency. */" inip6_do_tabledid pick up the 'old' private pointer, AFTERxt_replacehad updated it?
Yes, that is what we have observed from the logging.
Quoted 15 lines
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1435
Quoted 12 lines
net/netfilter/x_tables.c +2
-2@@ -1431,11 +1431,11 @@ xt_replace_table(struct xt_table *table,* Ensure contents of newinfo are visible before assigning to * private. */ -smp_wmb (); - table->private = newinfo; +smp_mb (); +WRITE_ONCE (table->private, newinfo);The
WRITE_ONCElooks correct.
Quoted 5 lines
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1438
Quoted 3 lines
net/netfilter/x_tables.c +1
-1/* make sure all cpus see new ->private value */ -smp_wmb (); +smp_mb ();Is that to order wrt.
seqcount_sequence?
Correct, we want to ensure that the table->private is updated and is in sync on all CPUs beyond that point.
Quoted 6 lines
Do you have a way to reproduce such crashes?
I tried to no avail but I guess thats just because amd64 is more forgiving.
Thanks!
Unfortunately we are seeing it on ARM64 regression systems which runs a variety of usecases so the exact steps are not known.
Florian Westphal
maintainerQuoted 7 lines · [email protected] <[email protected]> wrote:
Quoted 1 line
Is that to order wrt.
seqcount_sequence?Correct, we want to ensure that the
table->privateis updated and is in sync on all CPUs beyond that point.Quoted 2 lines
Do you have a way to reproduce such crashes?
Quoted 8 lines
Quoted 4 lines
I tried to no avail but I guess thats just because amd64 is more forgiving.
Thanks!
Unfortunately we are seeing it on ARM64 regression systems which runs a variety of
usecases so the exact steps are not known.
Ok. Would you be willing to run some of those with your suggested change to see if that resolves the crashes or is that so rare that this isn't practical?
Quoted 7 lines
Quoted 3 lines
Unfortunately we are seeing it on ARM64 regression systems which runs a variety of
usecases so the exact steps are not known.Ok. Would you be willing to run some of those with your suggested change to see if that resolves the crashes or is that so rare that this isn't practical?
I can try that out. Let me know if you have any other suggestions as well and I can try that too.
I assume we cant add locks here as it would be in the packet processing path.
Florian Westphal
maintainerQuoted 9 lines · [email protected] <[email protected]> wrote:
Quoted 8 lines
Quoted 4 lines
Unfortunately we are seeing it on ARM64 regression systems which runs a
variety of
usecases so the exact steps are not known.Ok. Would you be willing to run some of those with your suggested change to see if that resolves the crashes or is that so rare that this isn't practical?
Quoted 5 lines
I can try that out. Let me know if you have any other suggestions as well and I can try that too.
I assume we cant add locks here as it would be in the packet processing path.
Yes. We can add a synchronize_net() in xt_replace_table if needed though, before starting to put the references on the old ruleset This would avoid the free of the jumpstack while skbs are still in-flight.
Hi all,
[+tglx and peterz for seqcount abuse]
Quoted 13 lines · Florian Westphal
Quoted 12 lines · [email protected] <[email protected]> wrote:
Quoted 8 lines
Quoted 4 lines
Unfortunately we are seeing it on ARM64 regression systems which runs a
variety of
usecases so the exact steps are not known.Ok. Would you be willing to run some of those with your suggested change to see if that resolves the crashes or is that so rare that this isn't practical?
I can try that out. Let me know if you have any other suggestions as well and I can try that too.
Quoted 7 lines
Quoted 2 lines
I assume we cant add locks here as it would be in the packet processing path.
Yes. We can add a
synchronize_net()inxt_replace_tableif needed though, before starting to put the references on the old ruleset This would avoid the free of the jumpstack while skbs are still in-flight.
I tried to make sense of what's going on here, and it looks to me like the interaction between xt_replace_table() and ip6t_do_table() relies on store->load order that is _not_ enforced in the code.
xt_replace_table() does:
table->private = newinfo;
/* make sure all cpus see new ->private value */
smp_wmb();
/*
* Even though table entries have now been swapped, other CPU's
* may still be using the old entries...
*/
local_bh_enable();
/* ... so wait for even xt_recseq on all cpus */
for_each_possible_cpu(cpu) {
seqcount_t *s = &per_cpu(xt_recseq, cpu);
u32 seq = raw_read_seqcount(s);
if (seq & 1) {
do {
cond_resched();
cpu_relax();
} while (seq == raw_read_seqcount(s));
}
}and I think the idea here is that we swizzle the table->private pointer to point to the new data, and then wait for all pre-existing readers to finish with the old table (isn't this what RCU is for?).
On the reader side, ip6t_do_table() does:
addend = xt_write_recseq_begin(); private = READ_ONCE(table->private); /* Address dependency. */
where xt_write_recseq_begin() is pretty creative:
unsigned int addend = (__this_cpu_read(xt_recseq.sequence) + 1) & 1; __this_cpu_add(xt_recseq.sequence, addend); smp_wmb(); return addend;
I think this can be boiled down to an instance of the "SB" litmus test:
CPU 0 CPU 1
Wtable = newinfo Rseq = 0smp_wmb() Wseq = 1 (i.e. seq += 1)
Rseq = 0 smp_wmb()
<free oldinfo> Rtable = oldinfo
Since smp_wmb() only orders store-store, then both CPUs can hoist the reads up and we get the undesirable outcome. This should be observable on x86 too.
Given that this appears to be going wrong in practice, I've included a quick hack below which should fix the ordering. However, I think it would be better if we could avoid abusing the seqcount code for this sort of thing.
Cheers,
Will
--->8
include/linux/netfilter/x_tables.h +1 -1
@@ -376,7 +376,7 @@ static inline unsigned int xt_write_recseq_begin(void)* since addend is most likely 1 */ __this_cpu_add (xt_recseq.sequence, addend); -smp_wmb (); +smp_mb ();return addend; }
net/netfilter/x_tables.c +6 -4
@@ -1384,10 +1384,12 @@ xt_replace_table(struct xt_table *table,* private. */ smp_wmb (); - table->private = newinfo; - -/* make sure all cpus see new ->private value */ -smp_wmb (); +WRITE_ONCE (table->private, newinfo); +/* +* make sure all cpus see new ->private value before we load the +* sequence counters +*/ +smp_mb ();/* * Even though table entries have now been swapped, other CPU's
Florian Westphal
maintainerQuoted 52 lines · Will Deacon
Quoted 13 lines · Florian Westphal
Quoted 12 lines · [email protected] <[email protected]> wrote:
Quoted 8 lines
Quoted 4 lines
Unfortunately we are seeing it on ARM64 regression systems which runs a
variety of
usecases so the exact steps are not known.Ok. Would you be willing to run some of those with your suggested change to see if that resolves the crashes or is that so rare that this isn't practical?
I can try that out. Let me know if you have any other suggestions as well and I can try that too.
Quoted 7 lines
Quoted 2 lines
I assume we cant add locks here as it would be in the packet processing path.
Yes. We can add a
synchronize_net()inxt_replace_tableif needed though, before starting to put the references on the old ruleset This would avoid the free of the jumpstack while skbs are still in-flight.I tried to make sense of what's going on here, and it looks to me like the interaction between
xt_replace_table()andip6t_do_table()relies onstore->loadorder that is _not_ enforced in the code.
xt_replace_table()does:table->private = newinfo; /* make sure all cpus see new ->private value */ smp_wmb(); /* * Even though table entries have now been swapped, other CPU's * may still be using the old entries... */ local_bh_enable(); /* ... so wait for even xt_recseq on all cpus */ for_each_possible_cpu(cpu) { seqcount_t *s = &per_cpu(xt_recseq, cpu); u32 seq = raw_read_seqcount(s); if (seq & 1) { do { cond_resched(); cpu_relax(); } while (seq == raw_read_seqcount(s)); } }
Quoted 3 lines
and I think the idea here is that we swizzle the
table->privatepointer to point to the new data, and then wait for all pre-existing readers to finish with the old table (isn't this what RCU is for?).
Yes and yes. Before the current 'for_each_possible_cpu() + seqcount
abuse the abuse was just implicit; xt_replace_table did NOT wait for readers to go away and relied on arp, eb and ip(6)tables to store the counters back to userspace after xt_replace_table().
This meant a read_seqcount_begin/retry for each counter & eventually gave complaits from k8s users that x_tables rule replacement was too slow.
[..]
Quoted 3 lines
Given that this appears to be going wrong in practice, I've included a quick hack below which should fix the ordering. However, I think it would be better if we could avoid abusing the seqcount code for this sort of thing.
I'd be ok with going via the simpler solution & wait if k8s users complain that its too slow. Those memory blobs can be huge so I would not use call_rcu here.
net/netfilter/x_tables.c +2 -14
@@ -1384,7 +1384,7 @@ xt_replace_table(struct xt_table *table,* private. */ smp_wmb (); - table->private = newinfo; +WRITE_ONCE (table->private, newinfo);/* make sure all cpus see new ->private value */ smp_wmb ();
@@ -1394,19 +1394,7 @@ xt_replace_table(struct xt_table *table,* may still be using the old entries... */ local_bh_enable (); - -/* ... so wait for even xt_recseq on all cpus */ -for_each_possible_cpu (cpu) { -seqcount_t *s = &per_cpu (xt_recseq, cpu); -u32 seq =raw_read_seqcount (s); - -if (seq &1 ) { -do { -cond_resched (); -cpu_relax (); - }while (seq ==raw_read_seqcount (s)); - } - } +synchronize_rcu (); audit_log_nfcfg(table->name, table->af, private->number, !private->number ? AUDIT_XT_OP_REGISTER :
Quoted 53 lines · Florian Westphal
Quoted 52 lines · Will Deacon
Quoted 13 lines · Florian Westphal
Quoted 12 lines · [email protected] <[email protected]> wrote:
Quoted 8 lines
Quoted 4 lines
Unfortunately we are seeing it on ARM64 regression systems which runs a
variety of
usecases so the exact steps are not known.Ok. Would you be willing to run some of those with your suggested change to see if that resolves the crashes or is that so rare that this isn't practical?
I can try that out. Let me know if you have any other suggestions as well and I can try that too.
Quoted 7 lines
Quoted 2 lines
I assume we cant add locks here as it would be in the packet processing path.
Yes. We can add a
synchronize_net()inxt_replace_tableif needed though, before starting to put the references on the old ruleset This would avoid the free of the jumpstack while skbs are still in-flight.I tried to make sense of what's going on here, and it looks to me like the interaction between
xt_replace_table()andip6t_do_table()relies onstore->loadorder that is _not_ enforced in the code.
xt_replace_table()does:table->private = newinfo; /* make sure all cpus see new ->private value */ smp_wmb(); /* * Even though table entries have now been swapped, other CPU's * may still be using the old entries... */ local_bh_enable(); /* ... so wait for even xt_recseq on all cpus */ for_each_possible_cpu(cpu) { seqcount_t *s = &per_cpu(xt_recseq, cpu); u32 seq = raw_read_seqcount(s); if (seq & 1) { do { cond_resched(); cpu_relax(); } while (seq == raw_read_seqcount(s)); } }
Quoted 8 lines
Quoted 3 lines
and I think the idea here is that we swizzle the
table->privatepointer to point to the new data, and then wait for all pre-existing readers to finish with the old table (isn't this what RCU is for?).Yes and yes. Before the current '
for_each_possible_cpu()+ seqcount
abuse the abuse was just implicit;xt_replace_tabledid NOT wait for readers to go away and relied on arp, eb and ip(6)tables to store the counters back to userspace afterxt_replace_table().
Thanks for the background.
Quoted 6 lines
This meant a read_seqcount_begin/retry for each counter & eventually gave complaits from k8s users that
x_tablesrule replacement was too slow.[..]
Quoted 7 lines
Quoted 3 lines
Given that this appears to be going wrong in practice, I've included a quick hack below which should fix the ordering. However, I think it would be better if we could avoid abusing the seqcount code for this sort of thing.
I'd be ok with going via the simpler solution & wait if k8s users complain that its too slow. Those memory blobs can be huge so I would not use
call_rcuhere.
If you can stomach the cost of synchronize_rcu() then this at least gets rid of that for_each_possible_cpu() loop! Also...
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1387
Quoted 10 lines
net/netfilter/x_tables.c +1
-1@@ -1384,7 +1384,7 @@ xt_replace_table(struct xt_table *table,* private. */ smp_wmb (); - table->private = newinfo; +WRITE_ONCE (table->private, newinfo);
... you could make this rcu_assign_pointer() and get rid of the preceding smp_wmb()...
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1390
Quoted 3 lines
net/netfilter/x_tables.c +0
-0/* make sure all cpus see new ->private value */ smp_wmb ();
... and this smp_wmb() is no longer needed because synchronize_rcu() takes care of the ordering.
Quoted 7 lines
net/netfilter/x_tables.c +0
-3@@ -1394,19 +1394,7 @@ xt_replace_table(struct xt_table *table,* may still be using the old entries... */ local_bh_enable (); - -/* ... so wait for even xt_recseq on all cpus */ -for_each_possible_cpu (cpu) {
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1409
Quoted 10 lines
net/netfilter/x_tables.c +0
-10-seqcount_t *s = &per_cpu (xt_recseq, cpu); -u32 seq =raw_read_seqcount (s); - -if (seq &1 ) { -do { -cond_resched (); -cpu_relax (); - }while (seq ==raw_read_seqcount (s)); - } - }
Do we still need xt_write_recseq_{begin,end}() with this removed?
Cheers,
Will
Florian Westphal
maintainerQuoted 5 lines · Will Deacon
Quoted 5 lines
net/netfilter/x_tables.c +0
-0@@ -1384,7 +1384,7 @@ xt_replace_table(struct xt_table *table,
Quoted 8 lines
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1387
Quoted 5 lines
net/netfilter/x_tables.c +1
-1* private. */ smp_wmb (); - table->private = newinfo; +WRITE_ONCE (table->private, newinfo);... you could make this
rcu_assign_pointer()and get rid of the precedingsmp_wmb()...
Yes, it would make sense to add proper rcu annotation as well.
Quoted 5 lines
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1390
Quoted 2 lines
net/netfilter/x_tables.c +0
-0/* make sure all cpus see new ->private value */ smp_wmb ();... and this
smp_wmb()is no longer needed becausesynchronize_rcu()takes care of the ordering.
Right, thanks.
Quoted 11 lines
Quoted 11 lines
net/netfilter/x_tables.c +0
-7@@ -1394,19 +1394,7 @@ xt_replace_table(struct xt_table *table,* may still be using the old entries... */ local_bh_enable (); - -/* ... so wait for even xt_recseq on all cpus */ -for_each_possible_cpu (cpu) { -seqcount_t *s = &per_cpu (xt_recseq, cpu); -u32 seq =raw_read_seqcount (s); - -if (seq &1 ) {
Quoted 8 lines
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1409
Quoted 6 lines
net/netfilter/x_tables.c +0
-6-do { -cond_resched (); -cpu_relax (); - }while (seq ==raw_read_seqcount (s)); - } - }Do we still need xt_write_recseq_{begin,end}() with this removed?
Yes, it enables userspace to get stable per rule packet and byte counters.
I have tried the following to ensure the instruction ordering of private assignment and I haven't seen the crash so far.
net/netfilter/x_tables.c +3 -0
@@ -1389,6 +1389,9 @@ xt_replace_table(struct xt_table *table,/* make sure all cpus see new ->private value */ smp_wmb (); +/* make sure the instructions above are actually executed */ +smp_mb (); +/* * Even though table entries have now been swapped, other CPU's * may still be using the old entries...
I had added a debug to store the values of the xt_recseq.sequence in ip6t_do_table after the increment so it probably forced a load in the code order rather than allowing CPU to defer it.
addend = xt_write_recseq_begin();
Otherwise, I would have needed the additional instruction barrier which Will noted in xt_write_recseq_begin() below.
Quoted 16 lines
include/linux/netfilter/x_tables.h +1
-1@@ -376,7 +376,7 @@ static inline unsigned intxt_write_recseq_begin(void) * since addend is most likely 1 */ __this_cpu_add (xt_recseq.sequence, addend); -smp_wmb (); +smp_mb ();return addend;}
Quoted 10 lines
Quoted 2 lines
... you could make this
rcu_assign_pointer()and get rid of the precedingsmp_wmb()...Yes, it would make sense to add proper rcu annotation as well.
Quoted 5 lines
Quoted 2 lines
/* make sure all cpus see new ->private value */ smp_wmb();... and this
smp_wmb()is no longer needed becausesynchronize_rcu()takes care of the ordering.
I assume we would need the following changes to address this.
include/linux/netfilter/x_tables.h +1 -1
@@ -227,7 +227,7 @@ struct xt_table {unsigned int valid_hooks;/* Man behind the curtain... */ -struct xt_table_info *private; +struct xt_table_info __rcu *private;/* Set this to THIS_MODULE if you are a module, otherwise NULL */ struct module *me;
net/ipv4/netfilter/arp_tables.c +4 -4
@@ -203,7 +203,7 @@ unsigned int arpt_do_table(struct sk_buff *skb,local_bh_disable (); addend =xt_write_recseq_begin (); - private =READ_ONCE (table->private);/* Address dependency. */ + private =rcu_access_pointer (table->private); cpu =smp_processor_id (); table_base = private->entries; jumpstack = (struct arpt_entry **)private->jumpstack[cpu];
@@ -649,7 +649,7 @@ static struct xt_counters *alloc_counters(const struct xt_table *table) {unsigned int countersize;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =rcu_access_pointer (table->private);/* We need atomic snapshot of counters: rest doesn't change * (other than comefrom, which userspace doesn't care
@@ -673,7 +673,7 @@ static int copy_entries_to_user(unsigned int total_size,unsigned int off, num;const struct arpt_entry *e;struct xt_counters *counters; -struct xt_table_info *private = table->private; +struct xt_table_info *private =rcu_access_pointer (table->private);int ret =0 ;void *loc_cpu_entry;
@@ -1330,7 +1330,7 @@ static int compat_copy_entries_to_user(unsigned int total_size, void __user *userptr) {struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =rcu_access_pointer (table->private);void __user *pos;unsigned int size;int ret =0 ;
net/ipv4/netfilter/ip_tables.c +4 -4
@@ -258,7 +258,7 @@ ipt_do_table(struct sk_buff *skb,WARN_ON (!(table->valid_hooks & (1 << hook)));local_bh_disable (); addend =xt_write_recseq_begin (); - private =READ_ONCE (table->private);/* Address dependency. */ + private =rcu_access_pointer (table->private); cpu =smp_processor_id (); table_base = private->entries; jumpstack = (struct ipt_entry **)private->jumpstack[cpu];
@@ -791,7 +791,7 @@ static struct xt_counters *alloc_counters(const struct xt_table *table) {unsigned int countersize;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =rcu_access_pointer (table->private);/* We need atomic snapshot of counters: rest doesn't change (other than comefrom, which userspace doesn't care
@@ -815,7 +815,7 @@ copy_entries_to_user(unsigned int total_size,unsigned int off, num;const struct ipt_entry *e;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =rcu_access_pointer (table->private);int ret =0 ;const void *loc_cpu_entry;
@@ -1543,7 +1543,7 @@ compat_copy_entries_to_user(unsigned int total_size, struct xt_table *table, void __user *userptr) {struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =rcu_access_pointer (table->private);void __user *pos;unsigned int size;int ret =0 ;
net/ipv6/netfilter/ip6_tables.c +4 -4
@@ -280,7 +280,7 @@ ip6t_do_table(struct sk_buff *skb,local_bh_disable (); addend =xt_write_recseq_begin (); - private =READ_ONCE (table->private);/* Address dependency. */ + private =rcu_access_pointer (table->private); cpu =smp_processor_id (); table_base = private->entries; jumpstack = (struct ip6t_entry **)private->jumpstack[cpu];
@@ -807,7 +807,7 @@ static struct xt_counters *alloc_counters(const struct xt_table *table) {unsigned int countersize;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =rcu_access_pointer (table->private);/* We need atomic snapshot of counters: rest doesn't change (other than comefrom, which userspace doesn't care
@@ -831,7 +831,7 @@ copy_entries_to_user(unsigned int total_size,unsigned int off, num;const struct ip6t_entry *e;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =rcu_access_pointer (table->private);int ret =0 ;const void *loc_cpu_entry;
@@ -1552,7 +1552,7 @@ compat_copy_entries_to_user(unsigned int total_size, struct xt_table *table, void __user *userptr) {struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =rcu_access_pointer (table->private);void __user *pos;unsigned int size;int ret =0 ;
net/netfilter/x_tables.c +6 -12
@@ -1367,7 +1367,7 @@ xt_replace_table(struct xt_table *table,/* Do the substitution. */ local_bh_disable (); - private = table->private; + private =rcu_access_pointer (table->private);/* Check inside lock: is the old number correct? */ if (num_counters != private->number) {
@@ -1379,15 +1379,8 @@ xt_replace_table(struct xt_table *table, } newinfo->initial_entries = private->initial_entries; -/* -* Ensure contents of newinfo are visible before assigning to -* private. -*/ -smp_wmb (); - table->private = newinfo; -/* make sure all cpus see new ->private value */ -smp_wmb (); +rcu_assign_pointer (table->private, newinfo);/* * Even though table entries have now been swapped, other CPU's
@@ -1442,12 +1435,12 @@ struct xt_table *xt_register_table(struct net *net, }/* Simplifies replace_table code. */ - table->private = bootstrap; +rcu_assign_pointer (table->private, bootstrap);if (!xt_replace_table (table,0 , newinfo, &ret)) goto unlock; - private = table->private; + private =rcu_access_pointer (table->private);pr_debug ("table->private->number = %u\n" , private->number);/* save number of initial entries */
@@ -1470,7 +1463,8 @@ void *xt_unregister_table(struct xt_table *table)struct xt_table_info *private;mutex_lock (&xt[table->af].mutex); - private = table->private; + private =rcu_access_pointer (table->private); +RCU_INIT_POINTER (table->private,NULL );list_del (&table->list);mutex_unlock (&xt[table->af].mutex); audit_log_nfcfg(table->name, table->af, private->number,
Florian Westphal
maintainerQuoted 8 lines · [email protected] <[email protected]> wrote:
I have tried the following to ensure the instruction ordering of private assignment and I haven't seen the crash so far.
net/netfilter/x_tables.c +0
-0
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1394
Quoted 6 lines
net/netfilter/x_tables.c +3
-0@@ -1389,6 +1389,9 @@ xt_replace_table(struct xt_table *table,/* make sure all cpus see new ->private value */ smp_wmb (); +/* make sure the instructions above are actually executed */ +smp_mb (); +
This looks funny, I'd rather have s/smp_wmb/smp_mb instead of yet another one.
Quoted 1 line
I assume we would need the following changes to address this.
Yes, something like this. More comments below.
Quoted 20 lines
include/linux/netfilter/x_tables.h +1
-1@@ -227,7 +227,7 @@ struct xt_table {unsigned int valid_hooks;/* Man behind the curtain... */ -struct xt_table_info *private; +struct xt_table_info __rcu *private;/* Set this to THIS_MODULE if you are a module, otherwise NULL */ struct module *me;net/ipv4/netfilter/arp_tables.c +0
-0
net/ipv4/netfilter/arp_tables.c unsigned int arpt_do_table(struct sk_buff *skb, · line 206
Quoted 5 lines
net/ipv4/netfilter/arp_tables.c +1
-1@@ -203,7 +203,7 @@ unsigned int arpt_do_table(struct sk_buff *skb,local_bh_disable (); addend =xt_write_recseq_begin (); - private =READ_ONCE (table->private);/* Address dependency. */ + private =rcu_access_pointer (table->private);
The three _do_table() functions need to use rcu_dereference().
net/ipv4/netfilter/arp_tables.c static struct xt_counters *alloc_counters(const struct · line 653
Quoted 10 lines
net/ipv4/netfilter/arp_tables.c +1
-1cpu =smp_processor_id (); table_base = private->entries; jumpstack = (struct arpt_entry **)private->jumpstack[cpu];@@ -649,7 +649,7 @@ static struct xt_counters *alloc_counters(const struct xt_table *table) {unsigned int countersize;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =rcu_access_pointer (table->private);
This looks wrong. I know its ok, but perhaps its better to add this:
struct xt_table_info *xt_table_get_private_protected(const struct xt_table *table) {
return rcu_dereference_protected(table->private, mutex_is_locked(&xt[table->af].mutex));
}EXPORT_SYMBOL(xt_table_get_private_protected);
to x_tables.c.
If you dislike this extra function, add
#define xt_table_get_private_protected(t) rcu_access_pointer((t)->private)
in include/linux/netfilter/x_tables.h, with a bit fat comment telling that the xt table mutex must be held.
But I'd rather have/use the helper function as it documents when its safe to do this (and there will be splats if misused).
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1370
Quoted 9 lines
index af22dbe..2e6c09c 100644
net/netfilter/x_tables.c +1
-1@@ -1367,7 +1367,7 @@ xt_replace_table(struct xt_table *table,/* Do the substitution. */ local_bh_disable (); - private = table->private; + private =rcu_access_pointer (table->private);
AFAICS the local_bh_disable/enable calls can be removed too after this, if we're interrupted by softirq calling any of the _do_table() functions changes to the xt seqcount do not matter anymore.
net/netfilter/x_tables.c xt_replace_table(struct xt_table *table, · line 1372
Quoted 2 lines
net/netfilter/x_tables.c +0
-0/* * Even though table entries have now been swapped, other CPU's
We need this additional hunk to switch to rcu for replacement/sync, no?
- local_bh_enable();
-
- /* ... so wait for even xt_recseq on all cpus */
- for_each_possible_cpu(cpu) {
- seqcount_t *s = &per_cpu(xt_recseq, cpu);
- u32 seq = raw_read_seqcount(s);
-
- if (seq & 1) {
- do {
- cond_resched();
- cpu_relax();
- } while (seq == raw_read_seqcount(s));
- }
- }
+ synchronize_rcu();
Quoted 35 lines
The three
_do_table()functions need to usercu_dereference().This looks wrong. I know its ok, but perhaps its better to add this:
struct
xt_table_info*xt_table_get_private_protected(const structxt_table*table)
{
return rcu_dereference_protected(table->private,mutex_is_locked(&xt[table->af].mutex));
}EXPORT_SYMBOL(xt_table_get_private_protected);to
x_tables.c.If you dislike this extra function, add
#define
xt_table_get_private_protected(t)rcu_access_pointer((t)->private)in
include/linux/netfilter/x_tables.h, with a bit fat comment telling that the xt table mutex must be held.But I'd rather have/use the helper function as it documents when its safe to do this (and there will be splats if misused). AFAICS the local_bh_disable/enable calls can be removed too after this, if we're interrupted by softirq calling any of the
_do_table()functions changes to the xt seqcount do not matter anymore.We need this additional hunk to switch to rcu for replacement/sync, no?
(quoted diff) +0
-5- local_bh_enable(); - - /* ... so wait for even xt_recseq on all cpus */ - for_each_possible_cpu(cpu) { - seqcount_t *s = &per_cpu(xt_recseq, cpu);
Review
Quoted 10 lines
(quoted diff) +1
-9- u32 seq = raw_read_seqcount(s); - - if (seq & 1) { - do { - cond_resched(); - cpu_relax(); - } while (seq == raw_read_seqcount(s)); - } - } + synchronize_rcu();
I've updated the patch with your comments.
Do you expect a performance impact either in datapath or perhaps more in the rule installation with the rcu changes.
As a I understand, the change in the barrier types would be sufficient to resolve the race.
include/linux/netfilter/x_tables.h +1 -1
@@ -227,7 +227,7 @@ struct xt_table {unsigned int valid_hooks;/* Man behind the curtain... */ -struct xt_table_info *private; +struct xt_table_info __rcu *private;/* Set this to THIS_MODULE if you are a module, otherwise NULL */ struct module *me;
@@ -448,6 +448,9 @@ xt_get_per_cpu_counter(struct xt_counters *cnt, unsigned int cpu) struct nf_hook_ops *xt_hook_ops_alloc(const struct xt_table *,
nf_hookfn *);
+struct xt_table_info +*xt_table_get_private_protected(const struct xt_table *table); +
#ifdef CONFIG_COMPAT
#include <net/compat.h>
net/ipv4/netfilter/arp_tables.c +4 -4
@@ -203,7 +203,7 @@ unsigned int arpt_do_table(struct sk_buff *skb,local_bh_disable (); addend =xt_write_recseq_begin (); - private =READ_ONCE (table->private);/* Address dependency. */ + private =xt_table_get_private_protected (table); cpu =smp_processor_id (); table_base = private->entries; jumpstack = (struct arpt_entry **)private->jumpstack[cpu];
@@ -649,7 +649,7 @@ static struct xt_counters *alloc_counters(const struct xt_table *table) {unsigned int countersize;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =xt_table_get_private_protected (table);/* We need atomic snapshot of counters: rest doesn't change * (other than comefrom, which userspace doesn't care
@@ -673,7 +673,7 @@ static int copy_entries_to_user(unsigned int total_size,unsigned int off, num;const struct arpt_entry *e;struct xt_counters *counters; -struct xt_table_info *private = table->private; +struct xt_table_info *private =xt_table_get_private_protected (table);int ret =0 ;void *loc_cpu_entry;
@@ -1330,7 +1330,7 @@ static int compat_copy_entries_to_user(unsigned int total_size, void __user *userptr) {struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =xt_table_get_private_protected (table);void __user *pos;unsigned int size;int ret =0 ;
net/ipv4/netfilter/ip_tables.c +4 -4
@@ -258,7 +258,7 @@ ipt_do_table(struct sk_buff *skb,WARN_ON (!(table->valid_hooks & (1 << hook)));local_bh_disable (); addend =xt_write_recseq_begin (); - private =READ_ONCE (table->private);/* Address dependency. */ + private =xt_table_get_private_protected (table); cpu =smp_processor_id (); table_base = private->entries; jumpstack = (struct ipt_entry **)private->jumpstack[cpu];
@@ -791,7 +791,7 @@ static struct xt_counters *alloc_counters(const struct xt_table *table) {unsigned int countersize;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =xt_table_get_private_protected (table);/* We need atomic snapshot of counters: rest doesn't change (other than comefrom, which userspace doesn't care
@@ -815,7 +815,7 @@ copy_entries_to_user(unsigned int total_size,unsigned int off, num;const struct ipt_entry *e;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =xt_table_get_private_protected (table);int ret =0 ;const void *loc_cpu_entry;
@@ -1543,7 +1543,7 @@ compat_copy_entries_to_user(unsigned int total_size, struct xt_table *table, void __user *userptr) {struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =xt_table_get_private_protected (table);void __user *pos;unsigned int size;int ret =0 ;
net/ipv6/netfilter/ip6_tables.c +4 -4
@@ -280,7 +280,7 @@ ip6t_do_table(struct sk_buff *skb,local_bh_disable (); addend =xt_write_recseq_begin (); - private =READ_ONCE (table->private);/* Address dependency. */ + private =xt_table_get_private_protected (table); cpu =smp_processor_id (); table_base = private->entries; jumpstack = (struct ip6t_entry **)private->jumpstack[cpu];
@@ -807,7 +807,7 @@ static struct xt_counters *alloc_counters(const struct xt_table *table) {unsigned int countersize;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =xt_table_get_private_protected (table);/* We need atomic snapshot of counters: rest doesn't change (other than comefrom, which userspace doesn't care
@@ -831,7 +831,7 @@ copy_entries_to_user(unsigned int total_size,unsigned int off, num;const struct ip6t_entry *e;struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =xt_table_get_private_protected (table);int ret =0 ;const void *loc_cpu_entry;
@@ -1552,7 +1552,7 @@ compat_copy_entries_to_user(unsigned int total_size, struct xt_table *table, void __user *userptr) {struct xt_counters *counters; -const struct xt_table_info *private = table->private; +const struct xt_table_info *private =xt_table_get_private_protected (table);void __user *pos;unsigned int size;int ret =0 ;
net/netfilter/x_tables.c +15 -33
89 lines not on this page · raw message
Florian Westphal
maintainerQuoted 3 lines · [email protected] <[email protected]> wrote:
I've updated the patch with your comments.
Do you expect a performance impact either in datapath or perhaps more in the rule installation with the rcu changes.
Rule installation. synchronize_rcu() can take several seconds on busy systems.
net/ipv4/netfilter/arp_tables.c unsigned int arpt_do_table(struct sk_buff *skb, · line 206
Quoted 11 lines
net/ipv4/netfilter/arp_tables.c +1
-1@@ -203,7 +203,7 @@ unsigned int arpt_do_table(struct sk_buff *skb,local_bh_disable (); addend =xt_write_recseq_begin (); - private =READ_ONCE (table->private);/* Address dependency. */ + private =xt_table_get_private_protected (table);
Err, no, this needs to be plain rcu_dereference(table->private). Same in the other _do_table() versions.
We do not hold the table mutex here.
net/netfilter/x_tables.c struct xt_counters *xt_counters_alloc(unsigned int counters) · line 1358
Quoted 7 lines · On Thu, Nov 19, 2020 at 10:53:13PM -0700, [email protected] wrote:
net/netfilter/x_tables.c +7
-0+struct xt_table_info +*xt_table_get_private_protected (const struct xt_table *table) +{ +return rcu_dereference_protected (table->private, +mutex_is_locked (&xt[table->af].mutex)); +} +EXPORT_SYMBOL (xt_table_get_private_protected);
In all debug builds this function compiles to a single memory dereference. Do you really want that out-of-line?
Quoted 10 lines · Peter Zijlstra
net/netfilter/x_tables.c struct xt_counters *xt_counters_alloc(unsigned int counters) · line 1358
Quoted 7 lines · On Thu, Nov 19, 2020 at 10:53:13PM -0700, [email protected] wrote:
net/netfilter/x_tables.c +7
-0+struct xt_table_info +*xt_table_get_private_protected (const struct xt_table *table) +{ +return rcu_dereference_protected (table->private, +mutex_is_locked (&xt[table->af].mutex)); +} +EXPORT_SYMBOL (xt_table_get_private_protected);In all debug builds this function compiles to a single memory
! went missing... :/
Quoted 1 line
dereference. Do you really want that out-of-line?
Florian Westphal
maintainerQuoted 12 lines · Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
net/netfilter/x_tables.c struct xt_counters *xt_counters_alloc(unsigned int counters) · line 1358
Quoted 7 lines · On Thu, Nov 19, 2020 at 10:53:13PM -0700, [email protected] wrote:
net/netfilter/x_tables.c +7
-0+struct xt_table_info +*xt_table_get_private_protected (const struct xt_table *table) +{ +return rcu_dereference_protected (table->private, +mutex_is_locked (&xt[table->af].mutex)); +} +EXPORT_SYMBOL (xt_table_get_private_protected);In all debug builds this function compiles to a single memory
Quoted 1 line
! went missing... :/
? Not sure about "!".
Quoted 1 line
Quoted 1 line
dereference. Do you really want that out-of-line?
Its the lesser of two evils, the other solution would be to export xt[] from x_tables.c; its static right now.
Or we just use a rcu primitive that does no checking but I'd prefer avoid that.
Quoted 13 lines · Florian Westphal
Quoted 12 lines · Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
net/netfilter/x_tables.c struct xt_counters *xt_counters_alloc(unsigned int counters) · line 1358
Quoted 7 lines · On Thu, Nov 19, 2020 at 10:53:13PM -0700, [email protected] wrote:
net/netfilter/x_tables.c +7
-0+struct xt_table_info +*xt_table_get_private_protected (const struct xt_table *table) +{ +return rcu_dereference_protected (table->private, +mutex_is_locked (&xt[table->af].mutex)); +} +EXPORT_SYMBOL (xt_table_get_private_protected);In all debug builds this function compiles to a single memory
Quoted 3 lines
Quoted 1 line
! went missing... :/
? Not sure about "!".
!debug gets you a single deref, for debug builds do we get extra code.
Quoted 4 lines
Quoted 1 line
Quoted 1 line
dereference. Do you really want that out-of-line?
Its the lesser of two evils, the other solution would be to export xt[] from
x_tables.c; its static right now.
Bah, missed that.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
net/netfilter/x_tables.c struct xt_counters *xt_counters_alloc(unsigned int counters) · line 1358
Quoted 7 lines
net/netfilter/x_tables.c +7
-0+struct xt_table_info +*xt_table_get_private_protected (const struct xt_table *table) +{ +return rcu_dereference_protected (table->private, +mutex_is_locked (&xt[table->af].mutex)); +} +EXPORT_SYMBOL (xt_table_get_private_protected);In all debug builds this function compiles to a single memory
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
! went missing... :/
? Not sure about "!".
!debug gets you a single deref, for debug builds do we get extra code.
Can you elaborate on this. I am not able to get the context here. Are you referring to the definition of RCU_LOCKDEP_WARN. RCU_LOCKDEP_WARN is a NOOP if CONFIG_PROVE_RCU is false.
#ifdef CONFIG_PROVE_RCU
#define RCU_LOCKDEP_WARN(c, s) \
do { \
static bool __section(.data.unlikely) __warned; \
if (debug_lockdep_rcu_enabled() && !__warned && (c)) { \
__warned = true; \
lockdep_rcu_suspicious(__FILE__, __LINE__, s); \
} \
} while (0)...
#else /* #ifdef CONFIG_PROVE_RCU */
#define RCU_LOCKDEP_WARN(c, s) do { } while (0)