Skip to contentSkip to conversation

performance

DISCUSSION [email protected] 3 participants 6 messages Jun 9, 2003 – Jun 12, 2003 netfilter

lore
Share & tools
lore
#0 lore raw

Hi,

I'm testing netfilter performance here on PIII 1.2GHz based systems. With default kernel configuration, netfilter is able to process 85,000 pps with 125 rules (all rules matching).

Note the application is just counting. There is no transmitting/forwarding.

Also note the nics are e100.

So my simple question are there any tips in increasing the performance? Hmm actually the performance seems optimal? is it only taking 9 instructions per match? 1.2*10^9/(85000*1500) = 9

thanks,
Pádraig.

#1 lore raw
Quoted 11 lines · [email protected] wrote:

Hi,

I'm testing netfilter performance here on PIII 1.2GHz based systems. With default kernel configuration, netfilter is able to process 85,000 pps with 125 rules (all rules matching).

Note the application is just counting. There is no transmitting/forwarding.

Quoted 7 lines

Also note the nics are e100.

So my simple question are there any tips in increasing the performance? Hmm actually the performance seems optimal? is it only taking 9 instructions per match? 1.2*10^9/(85000*1500) = 9

I knew that couldn't be right.
That was tested on a dual 1.2GHz,
so that should be approx:
2*10^9/(85000*125) = 188 instructions per match.

I guess that's pretty optimal? The best I could hope for after that would be to increase the rx packet buffer space so as to handle higher spikes than this.

cheers,
Pádraig.

#2 lore raw

Monday, June 9, 2003, 7:16:35 PM, you wrote:

Quoted 11 lines · Pdc> [email protected] wrote:
Quoted 11 lines

Hi,

I'm testing netfilter performance here on PIII 1.2GHz based systems. With default kernel configuration, netfilter is able to process 85,000 pps with 125 rules (all rules matching).

Note the application is just counting. There is no transmitting/forwarding.

Quoted 7 lines
Quoted 7 lines

Also note the nics are e100.

So my simple question are there any tips in increasing the performance? Hmm actually the performance seems optimal? is it only taking 9 instructions per match? 1.2*10^9/(85000*1500) = 9

Pdc> I knew that couldn't be right.
Pdc> That was tested on a dual 1.2GHz,
Pdc> so that should be approx:
Pdc> 2*10^9/(85000*125) = 188 instructions per match.

I am afraid that is not instructions per match but ticks per match? An instruction can take more than one tick (clock cycle).

P.Krumins

#3 lore raw
Quoted 24 lines · Peteris Krumins

Monday, June 9, 2003, 7:16:35 PM, you wrote:

Quoted 11 lines · Pdc> [email protected] wrote:
Quoted 11 lines

Hi,

I'm testing netfilter performance here on PIII 1.2GHz based systems. With default kernel configuration, netfilter is able to process 85,000 pps with 125 rules (all rules matching).

Note the application is just counting. There is no transmitting/forwarding.

Quoted 7 lines
Quoted 7 lines

Also note the nics are e100.

So my simple question are there any tips in increasing the performance? Hmm actually the performance seems optimal? is it only taking 9 instructions per match? 1.2*10^9/(85000*1500) = 9

Quoted 8 lines

Pdc> I knew that couldn't be right.
Pdc> That was tested on a dual 1.2GHz,
Pdc> so that should be approx:
Pdc> 2*10^9/(85000*125) = 188 instructions per match.

I am afraid that is not instructions per match but ticks per match? An instruction can take more than one tick (clock cycle).

yes true. I was making assumptions.
The following suggests the average cycles:instruction ratio is 1.68 I think:
http://www.cs.berkeley.edu/~pattrsn/252S01/Lec18-dynamic3.pdf

Looking again at the system it's actually 2 x 1.4GHz So the instructions per match are about:
((2.8*10^9)/1.68)/(85000*125) = 156

With the overhead associated with SMP/kernel/... this suggests a figure closer to 120? I'm very impressed.

So to scale I would have to organise the rules into chains that could be bypassed. I wonder is there any projects to do this automatically? hmm..

Pádraig.

#4 lore raw
Quoted 5 lines · On Tue, Jun 10, 2003 at 09:49:54AM +0100, [email protected] wrote:

yes true. I was making assumptions.
The following suggests the average cycles:instruction ratio is 1.68 I think:
http://www.cs.berkeley.edu/~pattrsn/252S01/Lec18-dynamic3.pdf

Quoted 7 lines

Looking again at the system it's actually 2 x 1.4GHz So the instructions per match are about:
((2.8*10^9)/1.68)/(85000*125) = 156

With the overhead associated with SMP/kernel/... this suggests a figure closer to 120? I'm very impressed.

I don't know what kind of weird calculation that would be. First of all, why the hack are you adding up the clock rates of your CPU's? Do you have any idea how SMP systems work?

And then, why do you multiply the pps rate with the size of the packets? do you think we process every byte of a packet individually?

Let's make a simple calculation for the UP case:

average clock ticks per processed packet:
1.4*10^9/85000 = 16470

In order to get any idea about how those clock ticks are distributed among the various pars of kernel networking, you need to use some means of profiling.

I advise you to dig into profiling, and exploit all general kernel networking optimizations (like NAPI, ...) before starting to think about optimizing iptables.

Quoted 3 lines

So to scale I would have to organise the rules into chains that could be bypassed. I wonder is there any projects to do this automatically? hmm..

no.

Quoted 1 line

Pádraig.

Signature · 5 lines
- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie
#5 lore raw
Quoted 3 lines · Harald Welte

In order to get any idea about how those clock ticks are distributed among the various pars of kernel networking, you need to use some means of profiling.

iptables is in modules so I can't profile it at the moment. But this is informative. This is with 160 match any rules in the mangle::PREROUTING chain and then the packets are just dropped after. Packet rate again is 85Kpps.

14754 total 0.0096
11421 default_idle 142.7625 (71% !!)

   537 handle_IRQ_event                           3.3563
   341 eth_type_trans                             1.7760
   298 ip_rcv                                     0.4233
   238 do_gettimeofday                            1.8594
   224 netif_rx                                   0.5185
   217 add_timer_randomness                       0.9042
   196 skb_release_data                           1.3611
   180 batch_entropy_store                        1.0227
   162 alloc_skb                                  0.3375
   157 process_backlog                            0.5164
   137 kfree                                      0.7784
   126 __kmem_cache_alloc                         0.3937
   109 netif_receive_skb                          0.2004
    85 nf_hook_slow                               0.1771
    66 kmalloc                                    0.8250
    61 kfree_skbmem                               0.4766
    50 __constant_c_and_count_memset              0.3125
    36 __kfree_skb                                0.0978
    28 nf_iterate                                 0.1750
    16 get_sample_stats                           0.1250
    15 add_entropy_words                          0.0852
    14 ip_promisc_rcv_finish                      0.2917
    10 kmem_cache_free                            0.0625
     8 add_interrupt_randomness                   0.1250
     6 __generic_copy_to_user                     0.0625
     4 schedule                                   0.0030
     2 net_rx_action                              0.0057
     2 do_softirq                                 0.0089

The main question is why all the idle time?
Note userspace is locked out at this packet rate.

system specs:
dual 1.4GHz PIII
e1000 nic
kernel 2.4.20

Pádraig.