performance issues (nat / conntrack)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines
From: "Jean-Michel Hemstedt" <[email protected]>
I suppose that due to the load, packets are dropped not because of conntrack but because they simply can't be processed, and thus conntrack misses packets of existing connections (such as FIN, RST) and can't thus recover due to its timeouts.
But I believe that conntrack is exactly what's causing the load. If the machine is busy creating connections then it's clearly going to lose packets. I imagine that the packets that are dropped are those arriving at full queues. So fin's and rst's could very well be dropped if they arrive on interfaces that are receiving lots of other stuff while the cpu is busy building conntrack records.
Quoted 9 lines
From: Jozsef Kadlecsik <[email protected]>
Quoted 4 lines
We don't even need to have a full table to get into troubles. If at one point, the vast majority of the conntrack entries are unused, but still in hash, then you get more and more collisions, which decreases the hash efficiency.
What kind of collisions? Do you mean, that we end up in the same hash entry and the linked list in the entry becomes too long? There is not much wizardy we can do about it:
Not true. See my proposed bucket size limit. I hope to find some formulae to post on this later.
Quoted 5 lines
- increase the hash size (i.e buy more RAM) if the hash is small
- create better hash function, if one can deliberately hit the same entry.By the way, so far nobody has ever proved that the hash function is not good enough.
I suspect the hash function is fine. I propose to insert code that does printk whenever a bucket size exceeds some threshold and then invite all the readers of this list to try it and report their results. Obviously an attacker can arrange to attack one bucket, but I think the solution is limiting the size of each bucket. He still has no way of attacking a particular client/server pair or a particular client or particular server.
Quoted 10 lines
Quoted 4 lines
There's another side effect: when the system get's loaded (because of hash exhaustion or hash collisions), it can't process all packets arriving which means that conntrack will not see some FIN or RST packets allowing it to recover... This is a kind of 'vicious circle', or point of failure.
This is not true. If those FIN/RST packets belong to already existing connections, then those are in the conntrack hash and data can be updated. If those packets do not belong to an existing connection, then either they can create a new entry and we are fine, or conntrack is full and the packets will be dropped - we are fine again.
I think what happens is that it's not hash collisions but conntrack record creation that takes a long time, and that pretty much all packets are likely to be lost when the cpu is saturated with that activity. In that case it's true that connections are not garbage collected as fast as they should be.
Quoted 3 lines
From: Jozsef Kadlecsik <[email protected]>
If conntrack missed packets such a way, then the destination would miss as well and the sender should resend them. No problem.
Other than the delay. And if there's a high probability of drop then you get multiple delays, with each delay being longer than the last.
Quoted 6 lines
But I believe that conntrack is exactly what's causing the load. If the machine is busy creating connections then it's clearly going to lose packets. I imagine that the packets that are dropped are those arriving at full queues. So fin's and rst's could very well be dropped if they arrive on interfaces that are receiving lots of other stuff while the cpu is busy building conntrack records.
agreed.
(strange thing is that ethernet irq's reported by procinfo are
decreasing when the machine is overloaded. It suppose that it means either that irq's are not even caught by the kernel/driver, which is quite worrying, or either that irq's counters refer to 'processessed' interrupts)
Quoted 2 lines
Not true. See my proposed bucket size limit. I hope to find some formulae to post on this later.
longing to see that.
Quoted 4 lines
I suspect the hash function is fine. I propose to insert code that does printk whenever a bucket size exceeds some threshold and then invite all the readers of this list to try it and report their results.
I would rather go for global stats reported periodically, so that we have a constant measure (counters update) overhead.
Quoted 5 lines
I think what happens is that it's not hash collisions but conntrack record creation that takes a long time, and that pretty much all packets are likely to be lost when the cpu is saturated with that activity. In that case it's true that connections are not garbage collected as fast as they should be.
Since the difference between entry creation and entry lookup is only a call to init_conntrack(), profiling will be welcome, because as far as the slab allocator is concerned, at constant speed, the system should not do any costly alloc() anymore and instead dig into the slab' freelist. A lock problem somewhere, or a lack of inlines?
same thing for nat I suppose
Harld: could you ask to your kernel specialist what is the weakpoint of kmem_cache_alloc()? (locks, allocs, ...), and how we could possibly improve it (batch alloc, but isn't it already the case?)
PS: Harald
Quoted 3 lines
I've recently did some testing which try to avoid the null binding, but as I'm not entirely sure they don't break something else I haven't been releasing them yet.
I would be glad to test it at the same time. I'll come back to you when ready for testings. But I've no SMP system.
Quoted 3 lines
I've been talking about this with a couple of people here at the kernel summit, and it looks like the per-packet del_timer/add_timer in
ip_ct_refreshshould be a severe performance hit on SMP boxes.
Any indications that it would not be the same on non SMP boxes?
Quoted 2 lines
Changing this to 'do not update timer if update would be < HZ different than current timer' is a two-line patch.
I've seen that discussion, but not the patch. (I'll come back to you)
--
-jmhe-
Quoted 7 lines · Jean-Michel Hemstedt
agreed.
(strange thing is that ethernet irq's reported by procinfo are
decreasing when the machine is overloaded. It suppose that it means either that irq's are not even caught by the kernel/driver, which is quite worrying, or either that irq's counters refer to 'processessed' interrupts)
are you using a driver which uses the irq mitigation interface of 2.4.x or the NAPI of 2.5.x ?
Quoted 3 lines
Harld: could you ask to your kernel specialist what is the weakpoint of
kmem_cache_alloc()? (locks, allocs, ...), and how we could possibly improve it (batch alloc, but isn't it already the case?)
I don't think that the slab allocator is the bottleneck. please show me profiling data pointing this out.
Quoted 5 lines
Quoted 3 lines
I've been talking about this with a couple of people here at the kernel summit, and it looks like the per-packet del_timer/add_timer in
ip_ct_refreshshould be a severe performance hit on SMP boxes.Any indications that it would not be the same on non SMP boxes?
well, it's still a performance problem, but on UP it doesn't involve disabling interrupts across all cpus.
Quoted 1 line
-jmhe-
Signature · 5 lines
Live long and prosper - Harald Welte / [email protected] http://www.gnumonks.org/ ============================================================================ GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
Quoted 8 lines
Quoted 5 lines
(strange thing is that ethernet irq's reported by procinfo are
decreasing when the machine is overloaded. It suppose that it means either that irq's are not even caught by the kernel/driver, which is quite worrying, or either that irq's counters refer to 'processessed' interrupts)are you using a driver which uses the irq mitigation interface of 2.4.x or the NAPI of 2.5.x ?
no. (is it available for non Gigabit ethernets?). I'm using 3c905-TX.
one explanation could be that, SYN packets are 'lost'/not processed, and there are thus no additional SYN/ACK,ACK (at least), and thus no extra flow (each connection was made of about 10 packets). Since packets don't get forwarded, the same applies for the other interface. It means also that most retransmissions get lost.
Quoted 8 lines
Quoted 4 lines
Harld: could you ask to your kernel specialist what is the weakpoint of
kmem_cache_alloc()? (locks, allocs, ...), and how we could possibly improve it (batch alloc, but isn't it already the case?)I don't think that the slab allocator is the bottleneck. please show me profiling data pointing this out.
yesss, facts... but in the mean time, grabbing info/experience/feelings from 'non involved' people could be instructive. what do they say?
Quoted 1 line
- Harald Welte
kr,
-jmhe-