PROBLEM: high system usage / poor SMP network performance

I am the server admin for several very busy IRC servers with an ever increasing user load but I've hit a severe bottle neck which after numerous system tweaks and driver configuration changes I can only assume is related to a performance problem with the Linux Kernel.

The server configurations are all identical:
Compaq Proliant 330R's with Dual Pentium III -800 MHz's & 384MB RAM Intel NIC's using the Intel e100 driver
2.4.17 kernel
2 ircd processes per box

Here is a snapshot from 'top' :

      9:51pm  up 11 days, 10:13,  1 user,  load average: 0.95, 1.24, 1.21
    44 processes: 40 sleeping, 4 running, 0 zombie, 0 stopped
    CPU0 states: 27.2% user, 62.4% system,  0.0% nice,  9.2% idle
    CPU1 states: 28.4% user, 62.3% system,  0.0% nice,  8.1% idle
    Mem:   385096K av,  376896K used,    8200K free,       0K shrd,    3588K
buff
    Swap:  379416K av,   12744K used,  366672K free                   58980K

cached

      PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
     7825 ircd      18   0 84504  82M  5604 R    89.5 21.9  6929m ircd
    31010 ircd      20   0 86352  84M  5676 R    85.0 22.3  7218m ircd

When this snapshot was taken each ircd had 2000 users connect each. As you can see I am using more than a single cpu's worth of processer power just on system cpu and the ircd processes are using just over 50% of a single cpu! Now in comparison, another server admin who runs a ircd on a single P3-500 Linux 2.4.x system with 3000 users reaches about 60% *total* cpu usage. Likewise admins who run *BSD or Solaris can run with similar user connections and barely break a sweat. I have tried setting all the network performance tweaks mentioned on numerous sites and also using the cpu saver option on the e100 driver but at best I have only seen a 1-2% cpu saving.

Naturally I would really like to know where / what is using up all this system cpu so I would like to try profiling the kernel as I'm sure this is a pure kernel network layer performance issue but I have no idea where to start so does anyone have some advice / tips on where I should start?

Vince.

The important bit here is ^^^^^^^^ that one. Something is causing
horrendous lock contention it appears. Is the e100 driver optimised for SMP yet ? Do you get better numbers if you use the eepro100 driver ?

there's WAY too many busy waits (upto 500 msec with irq's disabled) in e100 to call it smp optimized.... also in most tests I've seen eepro100 won outright

----- Original Message -----
From: "Alan Cox" <[email protected]>
To: "Vincent Sweeney" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, January 27, 2002 10:54 PM
Subject: Re: PROBLEM: high system usage / poor SMP network performance

SMP

It's been a while since I tested with the eepro100 drivers (I switch to e100 about 2.4.4 due to some unrelated problems) so I cannot give a comparision just at present. I will test the eepro100 driver tomorrow on one of the servers and post results then.

I will also try Andrew Morton's profiling tips on another box with the e100 driver.

Vince.

----- Original Message -----
From: "Alan Cox" <[email protected]>
To: "Vincent Sweeney" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, January 27, 2002 10:54 PM
Subject: Re: PROBLEM: high system usage / poor SMP network performance

SMP

I've switched a server over to the default eepro100 driver as supplied in 2.4.17 (compiled as a module). This is tonights snapshot with about 10% higher user count than above (2200 connections per ircd)

7:25pm up 5:44, 2 users, load average: 0.85, 1.01, 1.09
38 processes: 33 sleeping, 5 running, 0 zombie, 0 stopped CPU0 states: 27.3% user, 69.3% system, 0.0% nice, 2.2% idle CPU1 states: 26.1% user, 71.2% system, 0.0% nice, 2.0% idle
Mem: 385096K av, 232960K used, 152136K free, 0K shrd, 4724K
buff
Swap: 379416K av, 0K used, 379416K free 21780K
cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
  659 ircd      15   0 74976  73M   660 R    96.7 19.4 263:21 ircd
  666 ircd      14   0 75004  73M   656 R    95.5 19.4 253:10 ircd

So as you can see the numbers are almost the same, though they were worse at lower users than the e100 driver (~45% system per cpu at 1000 users per ircd with eepro100, ~30% with e100).

I will try the profiling tomorrow with the eepro100 driver compiled into the kernel, I was unable to do the same for the Intel e100 driver today as I discovered that the Intel driver can currenty only be compiled as a module.

Vince.

Hummm ... poll() / select() ? ;)

readprofile | sort -n | tail -20

kind regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

----- Original Message -----
From: "Rik van Riel" <[email protected]>
To: "Vincent Sweeney" <[email protected]>
Cc: "Alan Cox" <[email protected]>; <[email protected]>
Sent: Monday, January 28, 2002 7:40 PM
Subject: Re: PROBLEM: high system usage / poor SMP network performance

Right then, here is the results from today so far (snapshot taken with 2000 users per ircd). Kernel profiling enabled with the eepro100 driver compiled statically.

---

    11 tcp_rcv_established                        0.0055
    12 do_softirq                                 0.0536
    12 nf_hook_slow                               0.0291
    13 __free_pages_ok                            0.0256
    13 kmalloc                                    0.0378
    13 rmqueue                                    0.0301
    13 tcp_ack                                    0.0159
    14 __kfree_skb                                0.0455
    14 tcp_v4_rcv                                 0.0084
    15 __ip_conntrack_find                        0.0441
    16 handle_IRQ_event                           0.1290
    16 tcp_packet                                 0.0351
    17 speedo_rx                                  0.0227
    17 speedo_start_xmit                          0.0346
    18 ip_route_input                             0.0484
    23 speedo_interrupt                           0.0301
    30 ipt_do_table                               0.0284
    30 tcp_sendmsg                                0.0065
   116 __pollwait                                 0.7838
   140 poll_freewait                              1.7500
   170 sys_poll                                   0.1897
   269 do_pollfd                                  1.4944
   462 remove_wait_queue                         12.8333
   474 add_wait_queue                             9.1154
   782 fput                                       3.3707
  1216 default_idle                              23.3846
  1334 fget                                      16.6750
  1347 sock_poll                                 33.6750

2408 tcp_poll 6.9195
9366 total 0.0094

4:30pm up 2:57, 2 users, load average: 0.76, 0.85, 0.82
36 processes: 33 sleeping, 3 running, 0 zombie, 0 stopped CPU0 states: 21.4% user, 68.1% system, 0.0% nice, 9.3% idle CPU1 states: 23.4% user, 67.1% system, 0.0% nice, 8.3% idle
Mem: 382916K av, 191276K used, 191640K free, 0K shrd, 1444K
buff
Swap: 379416K av, 0K used, 379416K free 23188K
cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
  613 ircd      16   0 67140  65M   660 R    89.6 17.5 102:00 ircd
  607 ircd      16   0 64868  63M   656 S    88.7 16.9  98:50 ircd

---

So with my little knowledge of what this means I would say this is purely down to poll(), but surely even with 4000 connections to the box that shouldn't stretch a dual P3-800 box as much as it does?

Vince.

Yes, profiling the kernel is clearly the next step. And it's really easy.

1: If possible, rebuild your kernel with as few modules as possible.
Current profiler doesn't cope with code which is in modules.

2: If you're on uniprocessor, enable the "Local APIC support on
Uniprocessors" option. This allows higher-resolution profiling.

3: Arrange for the kernel to be provided the `profile=1' boot
option. I use

	append="profile=1"

in /etc/lilo.conf

After a reboot, the kernel is profiling itself. The overhead is really low.

4: Bring the server online and wait until it starts to play up.

Now we can profile it. I use this script:

mnm:/home/akpm> cat $(which akpm-prof)
#!/bin/sh
TIMEFILE=/tmp/$(basename $1).time
sudo readprofile -r
sudo readprofile -M10
time "$@"
readprofile -v -m /boot/System.map | sort -n +2 | tail -40 | tee $TIMEFILE echo created $TIMEFILE

Let's go through it:

readprofile -r

This clears out the kernel's current profiling info

readprofile -M10

This attempts to set the profiling interval to 10*HZ (1000 Hz). This requires a local APIC, and a recent util-linux package. Not very important if this fails. This command also cleans out the kernel's current profiling info (it's a superset of -r).

time "$@"

Runs the command which we wish to profile

readprofile ...

Emits the profile info, sorted in useful order. You must make sure that /boot/System.map is the correct one for the currently-running kernel!

So in your situation, the command which you want to profile isn't important - you want to profile kernel activity arising from *existing* processes. So you can use:

akpm-prof sleep 30

Please send the results!

-