Skip to contentSkip to conversation

Costly Context Switches

DISCUSSION Ralf Hildebrandt 3 participants 8 messages Dec 12, 2010 – Dec 13, 2010

lore
Share & tools
lore
#0 lore raw

I recently made a parallel installation of dovecot-2.0 on my mailbox server, which is running dovecot-1.2 without any problems whatsoever.

Using dovecot-2.0 on the same hardware, same kernel, with the same users and same mailboxes and usage behaviour results in an immense increase in the load numbers.

Switching back to 1.2 results in a immediate decrease of the load back to "normal" numbers.

This is mainly due to a 10-20 fold increase of the number of context switches. The same problem has been reported independently by Cor Bosman of XS4All, on different hardware (64bit instead of 32bit, real hardware instead of virtual hardware).

So, now the kernel related question: How can I find out WHY the context switches are happening? Are there any "in kernel" statistics I could look at?

I'm running an Ubuntu kernel: 2.6.32-27-generic-pae #49-Ubuntu SMP

Signature · 7 lines
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  [email protected] | http://www.charite.de
#1 lore raw
Quoted 10 lines · On Sunday 12 December 2010 16:11:12 Ralf Hildebrandt wrote:

I recently made a parallel installation of dovecot-2.0 on my mailbox server, which is running dovecot-1.2 without any problems whatsoever.

Using dovecot-2.0 on the same hardware, same kernel, with the same users and same mailboxes and usage behaviour results in an immense increase in the load numbers.

Switching back to 1.2 results in a immediate decrease of the load back to "normal" numbers.

Quoted 8 lines

This is mainly due to a 10-20 fold increase of the number of context switches. The same problem has been reported independently by Cor Bosman of XS4All, on different hardware (64bit instead of 32bit, real hardware instead of virtual hardware).

So, now the kernel related question: How can I find out WHY the context switches are happening? Are there any "in kernel" statistics I could look at?

"strace" or "perf trace syscall-counts" would be a good start.

Andres

#2 lore raw

Em Sun, Dec 12, 2010 at 08:07:07PM +0100, Andres Freund escreveu:

Quoted 16 lines · On Sunday 12 December 2010 16:11:12 Ralf Hildebrandt wrote:
Quoted 14 lines

I recently made a parallel installation of dovecot-2.0 on my mailbox server, which is running dovecot-1.2 without any problems whatsoever.

Using dovecot-2.0 on the same hardware, same kernel, with the same users and same mailboxes and usage behaviour results in an immense increase in the load numbers.

Switching back to 1.2 results in a immediate decrease of the load back to "normal" numbers.

This is mainly due to a 10-20 fold increase of the number of context switches. The same problem has been reported independently by Cor Bosman of XS4All, on different hardware (64bit instead of 32bit, real hardware instead of virtual hardware).

Quoted 5 lines
Quoted 3 lines

So, now the kernel related question: How can I find out WHY the context switches are happening? Are there any "in kernel" statistics I could look at?

"strace" or "perf trace syscall-counts" would be a good start.

Better to record just "cs" (Context Switches) events and also to collect callchains when those events take place:

[acme@felicio ~]$ perf record -e cs -g chromium-browser ^C
[acme@felicio ~]$ perf report
# Overhead Command Shared Object Symbol

# ........  ...............  .................  ......
#
    91.32%  chromium-browse  [kernel.kallsyms]  [k] perf_event_task_sched_out
            |
            --- perf_event_task_sched_out
               |          
               |--37.80%-- sysret_careful
               |          |          
               |          |--85.11%-- 0x365300e42d
               |          |          
               |           --14.89%-- 0x365300e24a
               |          
               |--30.29%-- retint_careful
               |          |          
               |          |--29.20%-- 0x3652809658
               |          |          
               |          |--28.32%-- 0x7fcb90463603
               |          |          
               |          |--15.04%-- 0x7fcb8bca2e20
               |          |          
               |          |--14.16%-- 0x7fcb903ff36e
               |          |          
               |           --13.27%-- 0x3652809d2f
               |          
               |--23.86%-- schedule_timeout
               |          |          
               |          |--83.15%-- sys_epoll_wait
               |          |          system_call_fastpath
               |          |          0x3652ce5013
               |          |          
               |           --16.85%-- __skb_recv_datagram
               |                     skb_recv_datagram
               |                     unix_dgram_recvmsg
               |                     __sock_recvmsg
               |                     sock_recvmsg
               |                     __sys_recvmsg
               |                     sys_recvmsg
               |                     system_call_fastpath
               |                     __recvmsg
               |          
               |--4.02%-- __cond_resched
               |          _cond_resched
               |          might_fault
               |          memcpy_toiovec
               |          unix_stream_recvmsg
               |          __sock_recvmsg
               |          sock_aio_read
               |          do_sync_read
               |          vfs_read
               |          sys_read
               |          system_call_fastpath
               |          0x365300e48d
               |          (nil)
               |          
                --4.02%-- futex_wait_queue_me
                          futex_wait
                          do_futex
                          sys_futex
                          system_call_fastpath
                          __pthread_cond_timedwait

     7.05%   chrome-sandbox  [kernel.kallsyms]  [k] perf_event_task_sched_out
             |
             --- perf_event_task_sched_out
                 __cond_resched
                 _cond_resched
                 might_fault
                 filldir
                 proc_fill_cache
                 proc_readfd_common
                 proc_readfd
                 vfs_readdir
                 sys_getdents
                 system_call_fastpath
                 __getdents64

     1.34%      gconftool-2  [kernel.kallsyms]  [k] perf_event_task_sched_out
                |
                --- perf_event_task_sched_out
                    sysret_careful
                   |          
                   |--51.61%-- __recv
                   |          
                    --48.39%-- __recvmsg

- Arnaldo

#3 lore raw

* Arnaldo Carvalho de Melo <[email protected]>:

Quoted 4 lines

Better to record just "cs" (Context Switches) events and also to collect callchains when those events take place:

[acme@felicio ~]$ perf record -e cs -g chromium-browser

OK, I will try this. The application is dovecot-2.0, so I need to perf record it and all it's children (imap, pop3, auth, etc).

Is this done by default?

Signature · 7 lines
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  [email protected] | http://www.charite.de
#4 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 17 lines · On Monday 13 December 2010 14:51:04 Arnaldo Carvalho de Melo wrote:

Em Sun, Dec 12, 2010 at 08:07:07PM +0100, Andres Freund escreveu:

Quoted 16 lines · On Sunday 12 December 2010 16:11:12 Ralf Hildebrandt wrote:
Quoted 15 lines

I recently made a parallel installation of dovecot-2.0 on my mailbox server, which is running dovecot-1.2 without any problems whatsoever.

Using dovecot-2.0 on the same hardware, same kernel, with the same users and same mailboxes and usage behaviour results in an immense increase in the load numbers.

Switching back to 1.2 results in a immediate decrease of the load back to "normal" numbers.

This is mainly due to a 10-20 fold increase of the number of context switches. The same problem has been reported independently by Cor Bosman of XS4All, on different hardware (64bit instead of 32bit, real hardware instead of virtual hardware).

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

So, now the kernel related question: How can I find out WHY the context switches are happening? Are there any "in kernel" statistics I could look at?

"strace" or "perf trace syscall-counts" would be a good start.

Better to record just "cs" (Context Switches) events and also to collect callchains when those events take place:

Hm. Its also a good starting point but it may be harder to see the differences between dovecot-2.0 and dovecot-1.2 that way because its harder to see if its the usage being different causing the problem (i.e. calling in a different order, trashing caches) or if its the amount of syscalls that changed. But I agree that both are very usefull analyze problems like that.

Andres

#5 lore raw

Em Mon, Dec 13, 2010 at 03:25:54PM +0100, Andres Freund escreveu:

Quoted 22 lines · On Monday 13 December 2010 14:51:04 Arnaldo Carvalho de Melo wrote:
Quoted 21 lines

Em Sun, Dec 12, 2010 at 08:07:07PM +0100, Andres Freund escreveu:

Quoted 20 lines · On Sunday 12 December 2010 16:11:12 Ralf Hildebrandt wrote:
Quoted 18 lines

I recently made a parallel installation of dovecot-2.0 on my mailbox server, which is running dovecot-1.2 without any problems whatsoever.

Using dovecot-2.0 on the same hardware, same kernel, with the same users and same mailboxes and usage behaviour results in an immense increase in the load numbers.

Switching back to 1.2 results in a immediate decrease of the load back to "normal" numbers.

This is mainly due to a 10-20 fold increase of the number of context switches. The same problem has been reported independently by Cor Bosman of XS4All, on different hardware (64bit instead of 32bit, real hardware instead of virtual hardware).

So, now the kernel related question: How can I find out WHY the context switches are happening? Are there any "in kernel" statistics I could look at?

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

"strace" or "perf trace syscall-counts" would be a good start.

Better to record just "cs" (Context Switches) events and also to collect callchains when those events take place:

Hm. Its also a good starting point but it may be harder to see the differences between dovecot-2.0 and dovecot-1.2 that way because its harder to see if its

Well, for that he can try 'perf diff', i.e.:

1. run perf record on dovecot-1.2
2. run perf record on dovecot-2.0
3. perf diff

Quoted 3 lines

the usage being different causing the problem (i.e. calling in a different order, trashing caches) or if its the amount of syscalls that changed. But I agree that both are very usefull analyze problems like that.

Well, I tried to answer his question:

"How can I find out WHY the context switches are happening?"

:-)

- Arnaldo

#6 lore raw
Quoted 34 lines · On Monday 13 December 2010 15:29:11 Arnaldo Carvalho de Melo wrote:

Em Mon, Dec 13, 2010 at 03:25:54PM +0100, Andres Freund escreveu:

Quoted 25 lines · On Monday 13 December 2010 14:51:04 Arnaldo Carvalho de Melo wrote:
Quoted 24 lines

Em Sun, Dec 12, 2010 at 08:07:07PM +0100, Andres Freund escreveu:

Quoted 22 lines · On Sunday 12 December 2010 16:11:12 Ralf Hildebrandt wrote:
Quoted 11 lines

I recently made a parallel installation of dovecot-2.0 on my mailbox server, which is running dovecot-1.2 without any problems whatsoever.

Using dovecot-2.0 on the same hardware, same kernel, with the same users and same mailboxes and usage behaviour results in an immense increase in the load numbers.

Switching back to 1.2 results in a immediate decrease of the load back to "normal" numbers.

Quoted 8 lines

This is mainly due to a 10-20 fold increase of the number of context switches. The same problem has been reported independently by Cor Bosman of XS4All, on different hardware (64bit instead of 32bit, real hardware instead of virtual hardware).

So, now the kernel related question: How can I find out WHY the context switches are happening? Are there any "in kernel" statistics I could look at?

"strace" or "perf trace syscall-counts" would be a good start.

Quoted 7 lines
Quoted 4 lines

Better to record just "cs" (Context Switches) events and also to collect

callchains when those events take place:

Hm. Its also a good starting point but it may be harder to see the differences between dovecot-2.0 and dovecot-1.2 that way because its harder to see if its

Quoted 5 lines

Well, for that he can try 'perf diff', i.e.:

1. run perf record on dovecot-1.2
2. run perf record on dovecot-2.0
3. perf diff

Wow. That one I didnt know about yet.

Too bad that perf docs/examples are so spread out...

Greetings,

Andres

#7 lore raw

Em Mon, Dec 13, 2010 at 03:30:46PM +0100, Andres Freund escreveu:

Quoted 6 lines · On Monday 13 December 2010 15:29:11 Arnaldo Carvalho de Melo wrote:
Quoted 4 lines

Em Mon, Dec 13, 2010 at 03:25:54PM +0100, Andres Freund escreveu:

Quoted 3 lines

Hm. Its also a good starting point but it may be harder to see the differences between dovecot-2.0 and dovecot-1.2 that way because its harder to see if its

Quoted 8 lines
Quoted 5 lines

Well, for that he can try 'perf diff', i.e.:

1. run perf record on dovecot-1.2
2. run perf record on dovecot-2.0
3. perf diff

Wow. That one I didnt know about yet.

Too bad that perf docs/examples are so spread out...

There is a bit of info on these places:

http://vger.kernel.org/~acme/perf/lk2010-perf-paper.pdf
http://vger.kernel.org/~acme/perf/perf-plumbers2010.pdf

Also please consider helping organizing the existing documentation in a way that gets them more easily accessible :-)

- Arnaldo