netfilter: nf_conntrack: there maybe a bug in __nf_conntrack_confirm, when it race against get_next_corpse
Subsystems (3)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi, all:
sorry for sending this mail again, the last mail doesn't show text
clearly.
In function __nf_conntrack_confirm, we check the conntrack if it was
alreay dead, before insert it into hash-table.
we do this because if we insert an already 'dead' hash, it will
block further use of that particular connection.
but we don't do that right.
let's consider the following case:
cpu1 cpu2__nf_conntrack_confirm get_next_corpse
lock corresponding hash-list ....
check nf_ct_is_dying(ct)
for_each_possible_cpu(cpu) {
......
spin_lock_bh(&pcpu->lock);
......set_bit(IPS_DYING_BIT, &ct->status);
nf_ct_del_from_dying_or_unconfirmed_list(ct);spin_unlock_bh(&pcpu_lock);
add_timer(&ct->timeout); }
ct->status |= IPS_CONFIRMD;
__nf_conntrack_hash_insert(ct); /* the conntrack has been seted as
dying*/
The above case reveal two problems:
1. we may insert a dead conntrack to hash-table, it will block
further use of that particular connection.
2. operation on ct->status should be atomic, because it race aginstget_next_corpse.
due to this reason, the operation on ct->status in
nf_nat_setup_info should be atomic as well.
if we want to resolve the first problem, we must delete the
unconfirmed conntrack from unconfirmed-list first, then check if it is already dead.
Am I right to do this ?
Appreciate any comments and reply.
Florian Westphal
maintainerQuoted 5 lines · billbonaparte
In function
__nf_conntrack_confirm, we check the conntrack if it was
alreay dead, before insert it into hash-table.
we do this because if we insert an already 'dead' hash, it will
block further use of that particular connection.
but we don't do that right.
Correct. This is broken since the central spin lock removal, since nf_conntrack_lock no longer protects both get_next_corpse and conntrack_confirm.
Please send a patch, moving dying check after removal of conntrack from the percpu list, and add
tag to patch.
Quoted 3 lines
The above case reveal two problems:
1. we may insert a dead conntrack to hash-table, it will block
further use of that particular connection.
Yes.
Quoted 2 lines
2. operation on
ct->statusshould be atomic, because it race aginstget_next_corpse.
Alternatively we could also get rid of the unconfirmed list handling in get_next_corpse, it looks to me as if its simply not worth the trouble to also caring about unconfirmed lists.
Quoted 3 lines · On Tue, 28 Oct 2014 11:37:31 +0800 "billbonaparte" <[email protected]>…
Hi, all:
sorry for sending this mail again, the last mail doesn't show text clearly.
This one also mangles the text, so I cannot follow the race you are describing. I'll try to reconstruct...
Quoted 4 lines
In function
__nf_conntrack_confirm, we check the conntrack if it was already dead, before insert it into hash-table.
We do this because if we insert an already 'dead' hash, it will block further use of that particular connection.
Have you run into this problem in practice, or is this based on a theory?
Quoted 3 lines
but we don't do that right.
let's consider the following case:
[tried to reconstruct]
Quoted 9 lines
cpu1 cpu2
__nf_conntrack_confirmget_next_corpse
lock corresponding hash-list ....
checknf_ct_is_dying(ct)......... for_each_possible_cpu(cpu) { ..... (processing &pcpu->unconfirmed) ..... spin_lock_bh(&pcpu->lock); ..... set_bit(IPS_DYING_BIT, &ct->status); ..... spin_unlock_bh(&pcpu_lock);
Quoted 8 lines
spin_lock_bh(&pcpu->lock);
nf_ct_del_from_dying_or_unconfirmed_list(ct);
spin_unlock_bh(&pcpu_lock);
add_timer(&ct->timeout);
ct->status|=IPS_CONFIRMED;
__nf_conntrack_hash_insert(ct);
/* the conntrack has been seted as dying*/
Yes, I think you are correct. There is a race. As we are modifying the ct->status, without holding the hash bucket lock.
Quoted 11 lines
The above case reveal two problems:
1. we may insert a dead conntrack to hash-table, it will block
further use of that particular connection.
2. operation onct->statusshould be atomic, because it race aginstget_next_corpse.due to this reason, the operation on ct->status in
nf_nat_setup_infoshould be atomic as well.if we want to resolve the first problem, we must delete the
unconfirmed conntrack from unconfirmed-list first, then check if it is already dead.
Guess that would be one approach.
Quoted 2 lines
Am I right to do this ?
Appreciate any comments and reply.
Perhaps we could get rid of unconfirmed list handling in get_next_corpse?
Signature · 5 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer