32 core net-next stack/netfilter "scaling"

Folks -

Under:

ftp://ftp.netperf.org/iptable_scaling

can be found netperf results and Caliper profiles for three scenarios on a 32-core, 1.6 GHz 'Montecito' rx8640 system. An rx8640 is what HP call a "cell based" system in that it is comprised of "cell boards" on which reside CPU and memory resources. In this case there are four cell boards, each with 4, dual-core Montecito processors and 1/4 of the overall RAM. The system was configured with a mix of cell-local and global interleaved memory, where the global interleave is on a cacheline (128 byte) boundary (IIRC). Total RAM in the system is 256 GB. The cells are joined via cross-bar connections. (numactl --hardware output is available under the URL above)

There was an "I/O expander" connected to the system. This meant there were as many distinct PCI-X domains as there were cells, and every cell had a "local" set of PCI-X slots.

Into those slots I placed four HP AD385A PCI-X 10Gbit Ethernet NICs - aka Neterion XFrame IIs. These were then connected to an HP ProCurve 5806 switch, which was in turn connected to three, 4P/16C, 2.3 GHz HP DL585 G5s, each of which had a pair of HP AD386A PCIe 10Gbit Ethernet NICs (Aka Chelsio T3C-based). They were running RHEL 5.2 I think. Each NIC was in either a PCI-X 2.0 266 MHz slot (rx8640) or a PCIe 1.mumble x8 slot (DL585 G5)

The kernel is from DaveM's net-next tree ca last week, multiq enabled. The s2io driver is Neterion's out-of-tree version 2.0.36.15914 to get multiq support. It was loaded into the kernel via:

insmod ./s2io.ko tx_steering_type=3 tx_fifo_num=8

There were then 8 tx queues and 8 rx queues per interface in the rx8640. The "setaffinity.txt" script was used to set the IRQ affinities to cores "closest" to the physical NIC. In all three tests all 32 cores went to 100% utilization. At least for all incense and porpoises. (there was some occasional idle reported by top on the full_iptables run)

A set of 64, concurrent "burst mode" netperf omni RR tests (tcp) with a burst mode of 17 were run (ie 17 "transactions" outstanding on a connection at one time,) with TCP_NODELAY set and the results gathered, along with a set of Caliper profiles. The script used to launch these can be found in "runemomniagg2.sh.txt under the URL above.

I picked an "RR" test to maximize the trips up and down the stack while minimizing the bandwidth consumed.

I picked a burst size of 16 because that was sufficient to saturate a single core on the rx8640.

I picked 64 concurrent netperfs because I wanted to make sure I had enough concurrent connections to get spread across all the cores/queues by the algorithms in place.

I picked the combination of 64 and 16 rather than say 1024 and 0 (one tran at a time) because I didn't want to run a context switching benchmark :)

The rx8640 was picked because it was available and I was confident it was not going to have any hardware scaling issues getting in the way. I wanted to see SW issues, not HW issues. I am ass-u-me-ing the rx8640 is a reasonable analog for any "decent or better scaling" 32 core hardware and that while there are ia64-specific routines present in the profiles, they are there for platform-independent reasons.

The no_iptables/ data was run after a fresh boot, with no iptables commands run and so no iptables related modules loaded into the kernel.

The empty_iptables/ data was run after an "iptables --list" command which loaded one or two modules into the kernel.

The full_iptables/ data was run after an "iptables-restore" command pointed at full_iptables/iptables.txt which was created from what RH creates by default when one enables firewall via their installer, with a port range added by me to allow pretty much anything netperf would ask. As such, while it does excercise netfilter functionality, I cannot make any claims as to its "real world" applicability. (while the firewall settings came from an RH setup, FWIW, the base bits running on the rx8640 are Debian Lenny, with the net-next kernel on top)

The "cycles" profile is able to grab flat profile hits while interrupts are disabled so it can see stuff happening while interrupts are disabled. The "scgprof" profile is an attempt to get some call graphs - it does not have visibility into code running with interrupts disabled. The "cache" profile is a profile that looks to get some cache miss information.

So, having said all that, details can be found under the previously mentioned URL. Some quick highlights:

no_iptables - ~22000 transactions/s/netperf. Top of the cycles profile looks like:

Function Summary
-----------------------------------------------------------------------
% Total
IP Cumulat IP
Samples % of Samples
(ETB) Total (ETB) Function File
-----------------------------------------------------------------------

   5.70     5.70         37772   s2io.ko::tx_intr_handler
   5.14    10.84         34012   vmlinux::__ia64_readq
   4.88    15.72         32285   s2io.ko::s2io_msix_ring_handle
   4.63    20.34         30625   s2io.ko::rx_intr_handler
   4.60    24.94         30429   s2io.ko::s2io_xmit
   3.85    28.79         25488   s2io.ko::s2io_poll_msix
   2.87    31.65         18987   vmlinux::dev_queue_xmit
   2.51    34.16         16620   vmlinux::tcp_sendmsg
   2.51    36.67         16588   vmlinux::tcp_ack
   2.15    38.82         14221   vmlinux::__inet_lookup_established
   2.10    40.92         13937   vmlinux::ia64_spinlock_contention

empty_iptables - ~12000 transactions/s/netperf. Top of the cycles profile looks like:

Function Summary
-----------------------------------------------------------------------
% Total
IP Cumulat IP
Samples % of Samples
(ETB) Total (ETB) Function File
-----------------------------------------------------------------------

  26.38    26.38        137458   vmlinux::_read_lock_bh
  10.63    37.01         55388   vmlinux::local_bh_enable_ip
   3.42    40.43         17812   s2io.ko::tx_intr_handler
   3.01    43.44         15691   ip_tables.ko::ipt_do_table
   2.90    46.34         15100   vmlinux::__ia64_readq
   2.72    49.06         14179   s2io.ko::rx_intr_handler
   2.55    51.61         13288   s2io.ko::s2io_xmit
   1.98    53.59         10329   s2io.ko::s2io_msix_ring_handle
   1.75    55.34          9104   vmlinux::dev_queue_xmit
   1.64    56.98          8546   s2io.ko::s2io_poll_msix
   1.52    58.50          7943   vmlinux::sock_wfree
   1.40    59.91          7302   vmlinux::tcp_ack

full_iptables - some test instances didn't complete, I think they got starved. Of those which did complete, their performance ranged all the way from 330 to 3100 transactions/s/netperf. Top of the cycles profile looks like:

Function Summary
-----------------------------------------------------------------------
% Total
IP Cumulat IP
Samples % of Samples
(ETB) Total (ETB) Function File
-----------------------------------------------------------------------

  64.71    64.71        582171   vmlinux::_write_lock_bh
  18.43    83.14        165822   vmlinux::ia64_spinlock_contention
   2.86    85.99         25709   nf_conntrack.ko::init_module
   2.36    88.35         21194   nf_conntrack.ko::tcp_packet
   1.78    90.13         16009   vmlinux::_spin_lock_bh
   1.20    91.33         10810   nf_conntrack.ko::nf_conntrack_in
   1.20    92.52         10755   vmlinux::nf_iterate
   1.09    93.62          9833   vmlinux::default_idle
   0.26    93.88          2331   vmlinux::__ia64_readq
   0.25    94.12          2213   vmlinux::__interrupt
   0.24    94.37          2203   s2io.ko::tx_intr_handler

Suggestions as to things to look at/with and/or patches to try are welcome. I should have the HW available to me for at least a little while, but not indefinitely.

rick jones

Quoted 162 lines · Rick Jones

Folks -

Under:

ftp://ftp.netperf.org/iptable_scaling

can be found netperf results and Caliper profiles for three scenarios on a 32-core, 1.6 GHz 'Montecito' rx8640 system. An rx8640 is what HP call a "cell based" system in that it is comprised of "cell boards" on which reside CPU and memory resources. In this case there are four cell boards, each with 4, dual-core Montecito processors and 1/4 of the overall RAM. The system was configured with a mix of cell-local and global interleaved memory, where the global interleave is on a cacheline (128 byte) boundary (IIRC). Total RAM in the system is 256 GB. The cells are joined via cross-bar connections. (numactl --hardware output is available under the URL above)

There was an "I/O expander" connected to the system. This meant there were as many distinct PCI-X domains as there were cells, and every cell had a "local" set of PCI-X slots.

Into those slots I placed four HP AD385A PCI-X 10Gbit Ethernet NICs - aka Neterion XFrame IIs. These were then connected to an HP ProCurve 5806 switch, which was in turn connected to three, 4P/16C, 2.3 GHz HP DL585 G5s, each of which had a pair of HP AD386A PCIe 10Gbit Ethernet NICs (Aka Chelsio T3C-based). They were running RHEL 5.2 I think. Each NIC was in either a PCI-X 2.0 266 MHz slot (rx8640) or a PCIe 1.mumble x8 slot (DL585 G5)

The kernel is from DaveM's net-next tree ca last week, multiq enabled. The s2io driver is Neterion's out-of-tree version 2.0.36.15914 to get multiq support. It was loaded into the kernel via:

insmod ./s2io.ko tx_steering_type=3 tx_fifo_num=8

There were then 8 tx queues and 8 rx queues per interface in the rx8640. The "setaffinity.txt" script was used to set the IRQ affinities to cores "closest" to the physical NIC. In all three tests all 32 cores went to 100% utilization. At least for all incense and porpoises. (there was some occasional idle reported by top on the full_iptables run)

A set of 64, concurrent "burst mode" netperf omni RR tests (tcp) with a burst mode of 17 were run (ie 17 "transactions" outstanding on a connection at one time,) with TCP_NODELAY set and the results gathered, along with a set of Caliper profiles. The script used to launch these can be found in "runemomniagg2.sh.txt under the URL above.

I picked an "RR" test to maximize the trips up and down the stack while minimizing the bandwidth consumed.

I picked a burst size of 16 because that was sufficient to saturate a single core on the rx8640.

I picked 64 concurrent netperfs because I wanted to make sure I had enough concurrent connections to get spread across all the cores/queues by the algorithms in place.

I picked the combination of 64 and 16 rather than say 1024 and 0 (one tran at a time) because I didn't want to run a context switching benchmark :)

The rx8640 was picked because it was available and I was confident it was not going to have any hardware scaling issues getting in the way. I wanted to see SW issues, not HW issues. I am ass-u-me-ing the rx8640 is a reasonable analog for any "decent or better scaling" 32 core hardware and that while there are ia64-specific routines present in the profiles, they are there for platform-independent reasons.

The no_iptables/ data was run after a fresh boot, with no iptables commands run and so no iptables related modules loaded into the kernel.

The empty_iptables/ data was run after an "iptables --list" command which loaded one or two modules into the kernel.

The full_iptables/ data was run after an "iptables-restore" command pointed at full_iptables/iptables.txt which was created from what RH creates by default when one enables firewall via their installer, with a port range added by me to allow pretty much anything netperf would ask. As such, while it does excercise netfilter functionality, I cannot make any claims as to its "real world" applicability. (while the firewall settings came from an RH setup, FWIW, the base bits running on the rx8640 are Debian Lenny, with the net-next kernel on top)

The "cycles" profile is able to grab flat profile hits while interrupts are disabled so it can see stuff happening while interrupts are disabled. The "scgprof" profile is an attempt to get some call graphs - it does not have visibility into code running with interrupts disabled. The "cache" profile is a profile that looks to get some cache miss information.

So, having said all that, details can be found under the previously mentioned URL. Some quick highlights:

no_iptables - ~22000 transactions/s/netperf. Top of the cycles profile looks like:

Function Summary
-----------------------------------------------------------------------
% Total
IP Cumulat IP
Samples % of Samples
(ETB) Total (ETB) Function File
-----------------------------------------------------------------------

   5.70     5.70         37772   s2io.ko::tx_intr_handler
   5.14    10.84         34012   vmlinux::__ia64_readq
   4.88    15.72         32285   s2io.ko::s2io_msix_ring_handle
   4.63    20.34         30625   s2io.ko::rx_intr_handler
   4.60    24.94         30429   s2io.ko::s2io_xmit
   3.85    28.79         25488   s2io.ko::s2io_poll_msix
   2.87    31.65         18987   vmlinux::dev_queue_xmit
   2.51    34.16         16620   vmlinux::tcp_sendmsg
   2.51    36.67         16588   vmlinux::tcp_ack
   2.15    38.82         14221   vmlinux::__inet_lookup_established
   2.10    40.92         13937   vmlinux::ia64_spinlock_contention

empty_iptables - ~12000 transactions/s/netperf. Top of the cycles profile looks like:

Function Summary
-----------------------------------------------------------------------
% Total
IP Cumulat IP
Samples % of Samples
(ETB) Total (ETB) Function File
-----------------------------------------------------------------------

  26.38    26.38        137458   vmlinux::_read_lock_bh
  10.63    37.01         55388   vmlinux::local_bh_enable_ip
   3.42    40.43         17812   s2io.ko::tx_intr_handler
   3.01    43.44         15691   ip_tables.ko::ipt_do_table
   2.90    46.34         15100   vmlinux::__ia64_readq
   2.72    49.06         14179   s2io.ko::rx_intr_handler
   2.55    51.61         13288   s2io.ko::s2io_xmit
   1.98    53.59         10329   s2io.ko::s2io_msix_ring_handle
   1.75    55.34          9104   vmlinux::dev_queue_xmit
   1.64    56.98          8546   s2io.ko::s2io_poll_msix
   1.52    58.50          7943   vmlinux::sock_wfree
   1.40    59.91          7302   vmlinux::tcp_ack

full_iptables - some test instances didn't complete, I think they got starved. Of those which did complete, their performance ranged all the way from 330 to 3100 transactions/s/netperf. Top of the cycles profile looks like:

Function Summary
-----------------------------------------------------------------------
% Total
IP Cumulat IP
Samples % of Samples
(ETB) Total (ETB) Function File
-----------------------------------------------------------------------

  64.71    64.71        582171   vmlinux::_write_lock_bh
  18.43    83.14        165822   vmlinux::ia64_spinlock_contention
   2.86    85.99         25709   nf_conntrack.ko::init_module
   2.36    88.35         21194   nf_conntrack.ko::tcp_packet
   1.78    90.13         16009   vmlinux::_spin_lock_bh
   1.20    91.33         10810   nf_conntrack.ko::nf_conntrack_in
   1.20    92.52         10755   vmlinux::nf_iterate
   1.09    93.62          9833   vmlinux::default_idle
   0.26    93.88          2331   vmlinux::__ia64_readq
   0.25    94.12          2213   vmlinux::__interrupt
   0.24    94.37          2203   s2io.ko::tx_intr_handler

Quoted 5 lines

Suggestions as to things to look at/with and/or patches to try are welcome. I should have the HW available to me for at least a little while, but not indefinitely.

rick jones

Hi Rick, nice hardware you have :)

Stephen had a patch to nuke read_lock() from iptables, using RCU and seqlocks. I hit this contention point even with low cost hardware, and quite standard application.

I pinged him few days ago to try to finish the job with him, but it seems Stephen is busy at the moment.

Then conntrack (tcp sessions) is awfull, since it uses a single rwlock_t tcp_lock
that must be write_locked() for basically every handled tcp frame...

How long is "not indefinitely" ?

Hi Rick, nice hardware you have :)

Stephen had a patch to nuke read_lock() from iptables, using RCU and seqlocks. I hit this contention point even with low cost hardware, and quite standard application.

I pinged him few days ago to try to finish the job with him, but it seems Stephen is busy at the moment.

Then conntrack (tcp sessions) is awfull, since it uses a single rwlock_t tcp_lock
that must be write_locked() for basically every handled tcp frame...

How long is "not indefinitely" ?

Quoted 2 lines

Hi Rick, nice hardware you have :)

Every once in a while the cobbler's children get some shoes to wear :)

Quoted 10 lines

Stephen had a patch to nuke read_lock() from iptables, using RCU and seqlocks.
I hit this contention point even with low cost hardware, and quite standard application.

I pinged him few days ago to try to finish the job with him, but it seems Stephen
is busy at the moment.

Quoted 5 lines

Then conntrack (tcp sessions) is awfull, since it uses a single rwlock_t tcp_lock
that must be write_locked() for basically every handled tcp frame...

How long is "not indefinitely" ?

The system I am using is being borrowed under an open ended loan. However, my use of it can be thought of as being the "null process" in VMS - once anyone else wants it I have to get off of it.

That said, I would guess that the chances of someone else trying to get that system are pretty small for the next four+ weeks. I had a similar system (PCIe I/O rather than PCI-X) for quite a few weeks before it got pulled-out from under.

[PATCH] netfilter: Get rid of central rwlock in tcp conntracking

TCP connection tracking suffers of huge contention on a global rwlock, used to protect tcp conntracking state.
As each tcp conntrack state have no relations between each others, we can switch to fine grained lock, using a spinlock per "struct ip_ct_tcp"

tcp_print_conntrack() dont need to lock anything to read ct->proto.tcp.state, so speedup /proc/net/ip_conntrack as well.

Quoted 7 lines · Eric Dumazet

[PATCH] netfilter: Get rid of central rwlock in tcp conntracking

TCP connection tracking suffers of huge contention on a global rwlock, used to protect tcp conntracking state.
As each tcp conntrack state have no relations between each others, we can switch to fine grained lock, using a spinlock per "struct ip_ct_tcp"

Quoted 3 lines

tcp_print_conntrack() dont need to lock anything to read ct->proto.tcp.state,
so speedup /proc/net/ip_conntrack as well.

Thats an interesting test-case, but one lock per conntrack just for TCP tracking seems like overkill. We're trying to keep the conntrack stuctures as small as possible, so I'd prefer an array of spinlocks or something like that.

Quoted 5 lines · Patrick McHardy

Thats an interesting test-case, but one lock per conntrack just for TCP tracking seems like overkill. We're trying to keep the conntrack stuctures as small as possible, so I'd prefer an array of spinlocks or something like that.

Yes, this is wise. Current sizeof(struct nf_conn) is 220 (0xdc) on 32 bits, probably rounded to 0xE0 by SLAB/SLUB. I will provide a new patch using an array of say 512 spinlocks. (512 spinlocks use 2048 bytes if non debuging spinlocks, that spread to 32 x 64bytes cache lines)

However I wonder if for very large number of cpus we should at least ask conntrack to use hardware aligned "struct nf_conn" to avoid false sharing

Quoted 7 lines · Eric Dumazet

[PATCH] netfilter: Get rid of central rwlock in tcp conntracking

TCP connection tracking suffers of huge contention on a global rwlock, used to protect tcp conntracking state.
As each tcp conntrack state have no relations between each others, we can switch to fine grained lock, using a spinlock per "struct ip_ct_tcp"

Quoted 3 lines

tcp_print_conntrack() dont need to lock anything to read ct->proto.tcp.state,
so speedup /proc/net/ip_conntrack as well.

Thats an interesting test-case, but one lock per conntrack just for TCP tracking seems like overkill. We're trying to keep the conntrack stuctures as small as possible, so I'd prefer an array of spinlocks or something like that.

[PATCH] netfilter: Get rid of central rwlock in tcp conntracking

TCP connection tracking suffers of huge contention on a global rwlock, used to protect tcp conntracking state.

As each tcp conntrack state have no relations between each others, we can switch to fine grained lock. Using an array of spinlocks avoids enlarging size of connection tracking structures, yet giving reasonable fanout.

tcp_print_conntrack() doesnt need to lock anything to read ct->proto.tcp.state, so speedup /proc/net/ip_conntrack as well.

nf_conntrack_hash_rnd_initted & nf_conntrack_hash_rnd declared read_mostly

Quoted 13 lines · Eric Dumazet

[PATCH] netfilter: Get rid of central rwlock in tcp conntracking

TCP connection tracking suffers of huge contention on a global rwlock, used to protect tcp conntracking state.

As each tcp conntrack state have no relations between each others, we can switch to fine grained lock. Using an array of spinlocks avoids enlarging size of connection tracking structures, yet giving reasonable fanout.

tcp_print_conntrack() doesnt need to lock anything to read ct->proto.tcp.state, so speedup /proc/net/ip_conntrack as well.

Quoted 2 lines

nf_conntrack_hash_rnd_initted & nf_conntrack_hash_rnd declared read_mostly

This looks good to me. Rick, would you like to give it a try?

I'll convert the remaining conntrack protocols when applying it.

Quoted 20 lines · Patrick McHardy

Quoted 15 lines · Eric Dumazet

[PATCH] netfilter: Get rid of central rwlock in tcp conntracking

TCP connection tracking suffers of huge contention on a global rwlock, used to protect tcp conntracking state.

As each tcp conntrack state have no relations between each others, we can switch to fine grained lock. Using an array of spinlocks avoids enlarging size of connection tracking structures, yet giving reasonable fanout.

tcp_print_conntrack() doesnt need to lock anything to read ct->proto.tcp.state, so speedup /proc/net/ip_conntrack as well.

nf_conntrack_hash_rnd_initted & nf_conntrack_hash_rnd declared read_mostly

Quoted 3 lines

This looks good to me. Rick, would you like to give it a try?

I'll convert the remaining conntrack protocols when applying it.

I will give it a try and let folks know the results - unless told otherwise, I will ass-u-me I only need rerun the "full_iptables" test case.

Quoted 1 line

I will give it a try and let folks know the results - unless told otherwise, I will ass-u-me I only need rerun the "full_iptables" test case.

The runemomniagg2.sh script is still running, but the initial cycles profile suggests that the main change is converting the write_lock time into spinlock contention time with 78.39% of the cycles spent in ia64_spinlock_contention. When the script completes I'll upload the profiles and the netperf results to the same base URL as in the basenote under "contrack01/"

Quoted 4 lines

Quoted 1 line

I will give it a try and let folks know the results - unless told otherwise, I will ass-u-me I only need rerun the "full_iptables" test case.

The runemomniagg2.sh script is still running, but the initial cycles profile suggests that the main change is converting the write_lock time into spinlock contention time with 78.39% of the cycles spent in ia64_spinlock_contention. When the script completes I'll upload the profiles and the netperf results to the same base URL as in the basenote under "contrack01/"

The script completed - although at some point I hit an fd limit - I think I have an fd leak in netperf somewhere :( .

Anyhow, there are still some netperfs that end-up kicking the bucket during the run - I suspect starvation because where in the other configs (no iptables, and empty iptables) each netperf seems to consume about 50% of a CPU - stands to reason - 64 netperfs, 32 cores
- in the "full" case I see many netperfs consuming 100% of a CPU. My
gut is thinking that one or more netperf contexts gets stuck doing something on behalf of others. There is also ksoftirqd time for a few of those processes.

Anyhow, the spread on trans/s/netperf is now 600 to 500 or 6000, which does represent an improvement.

rick jones

PS - just to be certain that running-out of fd's didn't skew the results I'm rerunning the script with ulimit -n 10240 and will see if that changes the results any. And I suppose I need to go fd leak hunting in netperf omni code :(

Thanks for the report

If you have so much contention on spinlocks, maybe hash function is not good at all...

hash = (unsigned long)ct;
hash ^= hash >> 16;
hash ^= hash >> 8;

I ass-u-me you compiled your kernel with NR_CPUS=32 ?

Quoted 9 lines

Thanks for the report

If you have so much contention on spinlocks, maybe hash function is not good at all...

hash = (unsigned long)ct;
hash ^= hash >> 16;
hash ^= hash >> 8;

Quoted 1 line

I ass-u-me you compiled your kernel with NR_CPUS=32 ?

I believe so - CONFIG_NR_CPUS in .config is 64 anyway. If there is a more definitive place to check I'd be happy to look.

rick

By any chance, are you using SLAB instead of SLUB ?

While running your netperf session, issuing :

grep conntrack /proc/slabinfo

Would help to know hash dispertion on your machine.

Quoted 11 lines

Quoted 3 lines

I will give it a try and let folks know the results - unless told otherwise, I will ass-u-me I only need rerun the "full_iptables" test case.

The runemomniagg2.sh script is still running, but the initial cycles profile suggests that the main change is converting the write_lock time into spinlock contention time with 78.39% of the cycles spent in ia64_spinlock_contention. When the script completes I'll upload the profiles and the netperf results to the same base URL as in the basenote under "contrack01/"

The script completed - although at some point I hit an fd limit - I think I have an fd leak in netperf somewhere :( .

Anyhow, there are still some netperfs that end-up kicking the bucket during the run - I suspect starvation because where in the other configs (no iptables, and empty iptables) each netperf seems to consume about 50% of a CPU - stands to reason - 64 netperfs, 32 cores - in the "full" case I see many netperfs consuming 100% of a CPU. My gut is thinking that one or more netperf contexts gets stuck doing something on behalf of others. There is also ksoftirqd time for a few of those processes.

Anyhow, the spread on trans/s/netperf is now 600 to 500 or 6000, which does represent an improvement.

Quoted 3 lines · Rick Jones

Anyhow, the spread on trans/s/netperf is now 600 to 500 or 6000, which does represent an improvement.

Yes indeed you have a speedup, tcp conntracking is OK.

You now hit the nf_conntrack_lock spinlock we have in generic conntrack code (net/netfilter/nf_conntrack_core.c)

nf_ct_refresh_acct() for instance has to lock it.

We really want some finer locking here.

Quoted 12 lines · Eric Dumazet

Quoted 3 lines · Rick Jones

Anyhow, the spread on trans/s/netperf is now 600 to 500 or 6000, which does represent an improvement.

Yes indeed you have a speedup, tcp conntracking is OK.

You now hit the nf_conntrack_lock spinlock we have in generic conntrack code (net/netfilter/nf_conntrack_core.c)

nf_ct_refresh_acct() for instance has to lock it.

Quoted 1 line

We really want some finer locking here.

That looks more complicated since it requires to take multiple locks occasionally (f.i. hash insertion, potentially helper-related and expectation-related stuff), and there is the unconfirmed_list, where fine-grained locking can't really be used without changing it to a hash.

Quoted 10 lines · Patrick McHardy

Quoted 9 lines · Eric Dumazet

Quoted 3 lines · Rick Jones

Anyhow, the spread on trans/s/netperf is now 600 to 500 or 6000, which does represent an improvement.

Yes indeed you have a speedup, tcp conntracking is OK.

You now hit the nf_conntrack_lock spinlock we have in generic conntrack code (net/netfilter/nf_conntrack_core.c)

Quoted 9 lines

Quoted 3 lines

nf_ct_refresh_acct() for instance has to lock it.

We really want some finer locking here.

That looks more complicated since it requires to take multiple locks occasionally (f.i. hash insertion, potentially helper-related and expectation-related stuff), and there is the unconfirmed_list, where fine-grained locking can't really be used without changing it to a hash.

Yes its more complicated, but look what we did in 2.6.29 for tcp/udp
sockets, using RCU to have lockless lookups.
Yes, we still take a lock when doing an insert or delete at socket bind/unbind time.

We could keep a central nf_conntrack_lock to guard insertions/deletes from hash and unconfirmed_list.

But *normal* packets that only need to change state of one particular connection could use RCU (without spinlock) to locate the conntrack, then lock the found conntrack to perform all state changes.

While doing oprofile tests I noticed two loops are not properly unrolled by gcc

Quoted 1 line · Eric Dumazet

While doing oprofile tests I noticed two loops are not properly unrolled by gcc

That's because nobody passed -funroll-loops. Did you try that for that file? Likely will need -O2 too

Quoted 4 lines

(quoted diff) +3 -0

+static unsigned long ifname_compare(const void *_a, const void *_b, const void *_mask)
+{
+	const unsigned long *a = (const unsigned long *)_a;

Review

Quoted 10 lines

(quoted diff) +10 -0

+	const unsigned long *b = (const unsigned long *)_b;
+	const unsigned long *mask = (const unsigned long *)_mask;
+	unsigned long ret;
+
+	ret = (a[0] ^ b[0]) & mask[0];
+	ret |= (a[1] ^ b[1]) & mask[1];
+	if (IFNAMSIZ > 2 * sizeof(unsigned long))
+		ret |= (a[2] ^ b[2]) & mask[2];
+	if (IFNAMSIZ > 3 * sizeof(unsigned long))
+		ret |= (a[3] ^ b[3]) & mask[3];

That will silently break for IFNAMSIZ >= 4*sizeof(unsigned long) You should add a dummy loop for that or at least a BUILD_BUG_ON

Quoted 6 lines · Andi Kleen

Quoted 1 line · Eric Dumazet

While doing oprofile tests I noticed two loops are not properly unrolled by gcc

That's because nobody passed -funroll-loops. Did you try that for that file? Likely will need -O2 too

I dont want to unroll all loops, only those two :)

I wish gcc (4.3.2 here) was litle bit smarter :(

(By the way, I still use the patch on arch/x86/oprofile/op_model_ppro.c to have a working oprofile on my dev machine...)

Quoted 7 lines · Eric Dumazet

Quoted 5 lines · Andi Kleen

Quoted 1 line · Eric Dumazet

While doing oprofile tests I noticed two loops are not properly unrolled by gcc

That's because nobody passed -funroll-loops. Did you try that for that file? Likely will need -O2 too

I dont want to unroll all loops, only those two :)

gcc 4.4 will have a way to do that per function, but earlier you would need to move it to a separate file and specify the option only for that.

Doing so would be still a good idea compared to your patch because the code will be cleaner and might be more adaptable to future architectures (such manual tunings tend to outdate)

So... you suggest me to split file, and use a "-funroll-loops", that might doing strange things on some arches / compilers...

Quoted 7 lines · Eric Dumazet

Quoted 5 lines · Andi Kleen

Quoted 1 line · Eric Dumazet

While doing oprofile tests I noticed two loops are not properly unrolled by gcc

That's because nobody passed -funroll-loops. Did you try that for that file? Likely will need -O2 too

I dont want to unroll all loops, only those two :)

gcc 4.4 will have a way to do that per function, but earlier you would need to move it to a separate file and specify the option only for that.

Doing so would be still a good idea compared to your patch because the code will be cleaner and might be more adaptable to future architectures (such manual tunings tend to outdate)

The interface name matching has shown up in profiles forever though and we've actually already tried to optimize it IIRC.

Eric, I'm trying to keep all the *tables files synchronized, could you send me a patch updating the other ones as well please?

Quoted 6 lines · Patrick McHardy

The interface name matching has shown up in profiles forever though and we've actually already tried to optimize it IIRC.

Eric, I'm trying to keep all the *tables files synchronized, could you send me a patch updating the other ones as well please?

While doing this, I found arp_tables is still using loop using byte operations.

Also, I could not find how iniface_mask[], outiface_mask[], iniface[] and outiface[] were forced to long word alignment ...

(in struct ipt_ip, struct ip6t_ip6, struct arpt_arp)

Quoted 11 lines · Eric Dumazet

Quoted 6 lines · Patrick McHardy

The interface name matching has shown up in profiles forever though and we've actually already tried to optimize it IIRC.

Eric, I'm trying to keep all the *tables files synchronized, could you send me a patch updating the other ones as well please?

While doing this, I found arp_tables is still using loop using byte operations.

Quoted 3 lines

Also, I could not find how iniface_mask[], outiface_mask[], iniface[] and outiface[] were forced to long word alignment ...
(in struct ipt_ip, struct ip6t_ip6, struct arpt_arp)

In case of IPv4 and IPv6 they are already suitable aligned, it simply performing the comparison in unsigned long quantities. struct arpt_arp unfortunately doesn't properly align the interface names, so we need to continue to do byte-wise comparisons.

Quoted 5 lines · Patrick McHardy

In case of IPv4 and IPv6 they are already suitable aligned, it simply performing the comparison in unsigned long quantities. struct arpt_arp unfortunately doesn't properly align the interface names, so we need to continue to do byte-wise comparisons.

I see, but #ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS can help here ;)

ifname_compare() should be static in three files (ipv4_ip_tables, ipv6_ip_tables and arp_tables), since only arp_tables variant has the alignement problem.

[PATCH] netfilter: unfold two critical loops in arp_packet_match()

x86 and powerpc can perform long word accesses in an efficient maner. We can use this to unroll two loops in arp_packet_match(), to perform arithmetic on long words instead of bytes. This is a win on x86_64 for example.

Hi Patrick

Please find following patch to prepare next one (loop unrolling)

I am not sure xt_physdev_info is aligned, either on an int or a long, so please check my assertion before applying :)

[PATCH] netfilter: xt_physdev fixes

1) physdev_mt() incorrectly assumes nulldevname[] is aligned on an int

2) It also uses word comparisons, while it could use long word ones.

ip6_tables netfilter module can use an ifname_compare() helper so that two loops are unfolded.

xt_physdev netfilter module can use an ifname_compare() helper so that two loops are unfolded.

While doing oprofile tests I noticed two loops are not properly unrolled by gcc

Using a hand coded unrolled loop provides nice speedup : ipt_do_table credited of 2.52 % of cpu instead of 3.29 % in tbench.

Quoted 15 lines · Eric Dumazet

Quoted 6 lines · Patrick McHardy

That looks more complicated since it requires to take multiple locks occasionally (f.i. hash insertion, potentially helper-related and expectation-related stuff), and there is the unconfirmed_list, where fine-grained locking can't really be used without changing it to a hash.

Yes its more complicated, but look what we did in 2.6.29 for tcp/udp
sockets, using RCU to have lockless lookups.
Yes, we still take a lock when doing an insert or delete at socket bind/unbind time.

We could keep a central nf_conntrack_lock to guard insertions/deletes from hash and unconfirmed_list.

Quoted 3 lines

But *normal* packets that only need to change state of one particular connection could use RCU (without spinlock) to locate the conntrack, then lock the found conntrack to perform all state changes.

Well... RCU is already used by conntrack :)

Maybe only __nf_ct_refresh_acct() needs not taking nf_conntrack_lock

Quoted 162 lines · Rick Jones

Folks -

Under:

ftp://ftp.netperf.org/iptable_scaling

can be found netperf results and Caliper profiles for three scenarios on a 32-core, 1.6 GHz 'Montecito' rx8640 system. An rx8640 is what HP call a "cell based" system in that it is comprised of "cell boards" on which reside CPU and memory resources. In this case there are four cell boards, each with 4, dual-core Montecito processors and 1/4 of the overall RAM. The system was configured with a mix of cell-local and global interleaved memory, where the global interleave is on a cacheline (128 byte) boundary (IIRC). Total RAM in the system is 256 GB. The cells are joined via cross-bar connections. (numactl --hardware output is available under the URL above)

There was an "I/O expander" connected to the system. This meant there were as many distinct PCI-X domains as there were cells, and every cell had a "local" set of PCI-X slots.

Into those slots I placed four HP AD385A PCI-X 10Gbit Ethernet NICs - aka Neterion XFrame IIs. These were then connected to an HP ProCurve 5806 switch, which was in turn connected to three, 4P/16C, 2.3 GHz HP DL585 G5s, each of which had a pair of HP AD386A PCIe 10Gbit Ethernet NICs (Aka Chelsio T3C-based). They were running RHEL 5.2 I think. Each NIC was in either a PCI-X 2.0 266 MHz slot (rx8640) or a PCIe 1.mumble x8 slot (DL585 G5)

The kernel is from DaveM's net-next tree ca last week, multiq enabled. The s2io driver is Neterion's out-of-tree version 2.0.36.15914 to get multiq support. It was loaded into the kernel via:

insmod ./s2io.ko tx_steering_type=3 tx_fifo_num=8

There were then 8 tx queues and 8 rx queues per interface in the rx8640. The "setaffinity.txt" script was used to set the IRQ affinities to cores "closest" to the physical NIC. In all three tests all 32 cores went to 100% utilization. At least for all incense and porpoises. (there was some occasional idle reported by top on the full_iptables run)

A set of 64, concurrent "burst mode" netperf omni RR tests (tcp) with a burst mode of 17 were run (ie 17 "transactions" outstanding on a connection at one time,) with TCP_NODELAY set and the results gathered, along with a set of Caliper profiles. The script used to launch these can be found in "runemomniagg2.sh.txt under the URL above.

I picked an "RR" test to maximize the trips up and down the stack while minimizing the bandwidth consumed.

I picked a burst size of 16 because that was sufficient to saturate a single core on the rx8640.

I picked 64 concurrent netperfs because I wanted to make sure I had enough concurrent connections to get spread across all the cores/queues by the algorithms in place.

I picked the combination of 64 and 16 rather than say 1024 and 0 (one tran at a time) because I didn't want to run a context switching benchmark :)

The rx8640 was picked because it was available and I was confident it was not going to have any hardware scaling issues getting in the way. I wanted to see SW issues, not HW issues. I am ass-u-me-ing the rx8640 is a reasonable analog for any "decent or better scaling" 32 core hardware and that while there are ia64-specific routines present in the profiles, they are there for platform-independent reasons.

The no_iptables/ data was run after a fresh boot, with no iptables commands run and so no iptables related modules loaded into the kernel.

The empty_iptables/ data was run after an "iptables --list" command which loaded one or two modules into the kernel.

The full_iptables/ data was run after an "iptables-restore" command pointed at full_iptables/iptables.txt which was created from what RH creates by default when one enables firewall via their installer, with a port range added by me to allow pretty much anything netperf would ask. As such, while it does excercise netfilter functionality, I cannot make any claims as to its "real world" applicability. (while the firewall settings came from an RH setup, FWIW, the base bits running on the rx8640 are Debian Lenny, with the net-next kernel on top)

The "cycles" profile is able to grab flat profile hits while interrupts are disabled so it can see stuff happening while interrupts are disabled. The "scgprof" profile is an attempt to get some call graphs - it does not have visibility into code running with interrupts disabled. The "cache" profile is a profile that looks to get some cache miss information.

So, having said all that, details can be found under the previously mentioned URL. Some quick highlights:

no_iptables - ~22000 transactions/s/netperf. Top of the cycles profile looks like:

Function Summary
-----------------------------------------------------------------------
% Total
IP Cumulat IP
Samples % of Samples
(ETB) Total (ETB) Function File
-----------------------------------------------------------------------

   5.70     5.70         37772   s2io.ko::tx_intr_handler
   5.14    10.84         34012   vmlinux::__ia64_readq
   4.88    15.72         32285   s2io.ko::s2io_msix_ring_handle
   4.63    20.34         30625   s2io.ko::rx_intr_handler
   4.60    24.94         30429   s2io.ko::s2io_xmit
   3.85    28.79         25488   s2io.ko::s2io_poll_msix
   2.87    31.65         18987   vmlinux::dev_queue_xmit
   2.51    34.16         16620   vmlinux::tcp_sendmsg
   2.51    36.67         16588   vmlinux::tcp_ack
   2.15    38.82         14221   vmlinux::__inet_lookup_established
   2.10    40.92         13937   vmlinux::ia64_spinlock_contention

empty_iptables - ~12000 transactions/s/netperf. Top of the cycles profile looks like:

Function Summary
-----------------------------------------------------------------------
% Total
IP Cumulat IP
Samples % of Samples
(ETB) Total (ETB) Function File
-----------------------------------------------------------------------

  26.38    26.38        137458   vmlinux::_read_lock_bh
  10.63    37.01         55388   vmlinux::local_bh_enable_ip
   3.42    40.43         17812   s2io.ko::tx_intr_handler
   3.01    43.44         15691   ip_tables.ko::ipt_do_table
   2.90    46.34         15100   vmlinux::__ia64_readq
   2.72    49.06         14179   s2io.ko::rx_intr_handler
   2.55    51.61         13288   s2io.ko::s2io_xmit
   1.98    53.59         10329   s2io.ko::s2io_msix_ring_handle
   1.75    55.34          9104   vmlinux::dev_queue_xmit
   1.64    56.98          8546   s2io.ko::s2io_poll_msix
   1.52    58.50          7943   vmlinux::sock_wfree
   1.40    59.91          7302   vmlinux::tcp_ack

full_iptables - some test instances didn't complete, I think they got starved. Of those which did complete, their performance ranged all the way from 330 to 3100 transactions/s/netperf. Top of the cycles profile looks like:

Function Summary
-----------------------------------------------------------------------
% Total
IP Cumulat IP
Samples % of Samples
(ETB) Total (ETB) Function File
-----------------------------------------------------------------------

  64.71    64.71        582171   vmlinux::_write_lock_bh
  18.43    83.14        165822   vmlinux::ia64_spinlock_contention
   2.86    85.99         25709   nf_conntrack.ko::init_module
   2.36    88.35         21194   nf_conntrack.ko::tcp_packet
   1.78    90.13         16009   vmlinux::_spin_lock_bh
   1.20    91.33         10810   nf_conntrack.ko::nf_conntrack_in
   1.20    92.52         10755   vmlinux::nf_iterate
   1.09    93.62          9833   vmlinux::default_idle
   0.26    93.88          2331   vmlinux::__ia64_readq
   0.25    94.12          2213   vmlinux::__interrupt
   0.24    94.37          2203   s2io.ko::tx_intr_handler

Quoted 5 lines

Suggestions as to things to look at/with and/or patches to try are welcome. I should have the HW available to me for at least a little while, but not indefinitely.

rick jones

Hi Rick, nice hardware you have :)

Stephen had a patch to nuke read_lock() from iptables, using RCU and seqlocks. I hit this contention point even with low cost hardware, and quite standard application.

I pinged him few days ago to try to finish the job with him, but it seems Stephen is busy at the moment.

Then conntrack (tcp sessions) is awfull, since it uses a single rwlock_t tcp_lock
that must be write_locked() for basically every handled tcp frame...

How long is "not indefinitely" ?