Skip to contentSkip to conversation

epoll,threading

DISCUSSION Arunachalam 6 participants 12 messages May 26, 2007 – May 30, 2007

lore
Share & tools
lore
#0 lore raw

Hello all,

I want to know in detail about , what the events (epoll or /dev/poll or select ) achieve in contrast to thread per client.

i can have a thread per client and use send and recv system call directly right? Why do i go for these event mechanisms?

Please help me to understand this.

Thanks in advance,
Arunachalam

Signature · 2 lines
View this message in context: http://www.nabble.com/epoll%2Cthreading-tf3820388.html#a10815949
Sent from the linux-kernel mailing list archive at Nabble.com.
#1 lore raw

Hi Arunachalam,

Quoted 5 lines · Arunachalam

I want to know in detail about , what the events (epoll or /dev/poll or select ) achieve in contrast to thread per client.

i can have a thread per client and use send and recv system call directly right? Why do i go for these event mechanisms?

Try 30.000 clients or more on a x86 32bit box. That will show you the difference quite nicely :-)

More seriously: Thread per client scales only to a certain amount of clients
per RAM. If you like to scale beyond that to like to minimize your state per client. If you have a thread then you have a task structure as unswappable memory in kernel, a per-thread stack, which is reducing your virtual memory per process (you have only around 3GB of virtual memory per process in Linux x86 32bit).

So one uses a process or thread pool to scale beyond that. Pool size is typically related to the amount of CPU cores in the system.

Regards

Ingo Oeser

#2 lore raw
In reply to Arunachalam
Quoted 10 lines

Hello all,

I want to know in detail about , what the events (epoll or /dev/poll or select ) achieve in contrast to thread per client.

i can have a thread per client and use send and recv system call directly right? Why do i go for these event mechanisms?

Please help me to understand this.

Aside from the obvious, consider a server that needs to do a little bit of work on each of 1,000 clients on a single CPU system. With a thread-per-client approach, 1,000 context switches will be needed. With an epoll thread pool approach, none are needed and five or six are typical.

Both get you the major advantages of threading. You can take full advantage of multiple CPUs. You can write code that blocks occasionally without bringing the whole server down. A page fault doesn't stall your whole server.

DS

#3 lore raw
Quoted 13 lines · David Schwartz
Quoted 7 lines

I want to know in detail about , what the events (epoll or /dev/poll or select ) achieve in contrast to thread per client.

i can have a thread per client and use send and recv system call directly right? Why do i go for these event mechanisms?

Please help me to understand this.

Aside from the obvious, consider a server that needs to do a little bit of work on each of 1,000 clients on a single CPU system. With a thread-per-client approach, 1,000 context switches will be needed. With an epoll thread pool approach, none are needed and five or six are typical.

Quoted 4 lines

Both get you the major advantages of threading. You can take full advantage of multiple CPUs. You can write code that blocks occasionally without bringing the whole server down. A page fault doesn't stall your whole server.

It all depends on the workload but thread switching overhead can be negligible - after all all it does is entering kernel, schedule, swap processor context and return. Maybe using separate stack has minor impact on cache hit ratio. You need benchmark numbers to claim one way or the other.

In my experience with web caches, epoll or similar for idle clients and thread per active client scaled and performed pretty well - it needed more memory but the performance wasn't worse than asynchronous design and doing complex server in async model is a lot of pain.

Signature · 1 line
tejun
#4 lore raw
Quoted 19 lines · Tejun Heo
Quoted 13 lines · David Schwartz
Quoted 7 lines

I want to know in detail about , what the events (epoll or /dev/poll or select ) achieve in contrast to thread per client.

i can have a thread per client and use send and recv system call directly right? Why do i go for these event mechanisms?

Please help me to understand this.

Aside from the obvious, consider a server that needs to do a little bit of work on each of 1,000 clients on a single CPU system. With a thread-per-client approach, 1,000 context switches will be needed. With an epoll thread pool approach, none are needed and five or six are typical.

Quoted 4 lines

Both get you the major advantages of threading. You can take full advantage of multiple CPUs. You can write code that blocks occasionally without bringing the whole server down. A page fault doesn't stall your whole server.

Quoted 5 lines

It all depends on the workload but thread switching overhead can be negligible - after all all it does is entering kernel, schedule, swap processor context and return. Maybe using separate stack has minor impact on cache hit ratio. You need benchmark numbers to claim one way or the other.

In my experience, it's not much the context switch by itself which causes performance degradation, but the fact that with threads, you have to put mutexes everywhere. And frankly, walking a list with locks everywhere is quite slower than doing it in one run at a rate of 3 or 4 cycles per entry. Also, local storage in function returns is not possible anymore, and some functions even need to malloc() instead of returning statically allocated data. I believe this is the reason for openssl being twice as slow when compiled thread-safe than in native mode.

So in fact, converting a threaded program to a pure async model should not improve it much because of the initial architectural design. But a program written from scratch to be purely async should perform better simply because it has less operations to perform. And there's no magics here : less cycles spend synchronizing and locking = more cycles available for the real job.

Quoted 4 lines

In my experience with web caches, epoll or similar for idle clients and thread per active client scaled and performed pretty well - it needed more memory but the performance wasn't worse than asynchronous design and doing complex server in async model is a lot of pain.

It's true that an async model is a lot of pain. But it's always where I got the best performance. For instance, with epoll(), I can achieve 20000 HTTP reqs/s with 40000 concurrent sessions. The best performance I have observed from threaded competitors was an order of magnitude below on either value (sometimes both).

However, I agree that few uses really require to spend time writing and debugging async programs.

Regards,
Willy

#5 lore raw

I mostly agree with your comments, so I'm only responding to the points I disagree with.

Quoted 6 lines

So in fact, converting a threaded program to a pure async model should not improve it much because of the initial architectural design. But a program written from scratch to be purely async should perform better simply because it has less operations to perform. And there's no magics here : less cycles spend synchronizing and locking = more cycles available for the real job.

On the contrary, converting threaded programs to pure async is a disaster. For one thing, you can never, ever block under any circumstances on pain of total disaster. This means every single line of code is performance critical. For all but the most trivial applications, this alone is a deal killer.

The other problem is that no matter how careful you are, there is still going to be some unexpected blocking. For example, suppose one connection triggers some kind of error condition and the code to handle this error has never faulted in. Now your entire application stalls until the error handling code can fault in, which can be awhile if the disk is slow/busy.

Quoted 5 lines

It's true that an async model is a lot of pain. But it's always where I got the best performance. For instance, with epoll(), I can achieve 20000 HTTP reqs/s with 40000 concurrent sessions. The best performance I have observed from threaded competitors was an order of magnitude below on either value (sometimes both).

I can't see how adding a second thread to your program could possibly make it all that much worse. And when you do block unexpectedly, there's that second thread to keep on working. And, of course, you can take advantage of another CPU.

If adding a few more threads to your design makes it much worse, there's something wrong with it. Perhaps you are conflating "threaded competitors" with "thread-per-client competitors".

I also must be misunderstanding something you're saying, or you are very confused. At one point you say that threaded programs are an order of magnitude below pure async. At another point you say converting a threaded program to pure async should not improve it much. I don't see how both of these can be true unless you mean something different by "threaded" and "async" in the two statements.

Thread-per-client works reasonably well up to some number of clients. On most UNIXes, that number is around 300. On Windows, it's around 800. I personally would only recommend it for applications that plan to handle 100 clients or fewer, or one platforms where you know the threading library works well this way.

Pure async is just too painful. If you only have a single thread, the program becomes unmaintainable because you must never ever block anywhere. And you still can't avoid blocking sometimes, and this can cause a backup from which you can never recover. (If you always wondered why your servers were bursty, this is why.)

The best approach, in my experience, is to use multiple threads so that you can take advantage of multiple CPUs, not be screwed if you block unexpectedly, and even block intentionally in code that's not performance critical. On Linux, 'epoll' is the most efficient socket discovery technique, so you should use it in any design that's not thread-per-client.

DS

#6 lore raw
In reply to Willy Tarreau
Quoted 9 lines

In my experience, it's not much the context switch by itself which causes performance degradation, but the fact that with threads, you have to put mutexes everywhere. And frankly, walking a list with locks everywhere is quite slower than doing it in one run at a rate of 3 or 4 cycles per entry. Also, local storage in function returns is not possible anymore, and some functions even need to malloc() instead of returning statically allocated data. I believe this is the reason for openssl being twice as slow when compiled thread-safe than in native mode.

I don't know where you got this idea from, but several experiments of mine failed to confirm it. Here, for example, are two identical builds of OpenSSL, one with 'threads' and one with 'no-threads'. Same compiler, same other flags, same hardware (Linux 2.6.21.2, P3-1Ghz):

OpenSSL 0.9.8d 28 Sep 2006
built on: Wed May 30 00:55:25 PDT 2007
options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx)
compiler:
gcc -DDSO_DLFCN -DHAVE_DLFCN_H -mcpu=pentium -DL_ENDIAN -DTERMIO -O3 -fomit- frame-pointer -Wall -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DSHA1_A SM -DMD5_ASM -DRMD160_ASM -DAES_ASM
available timing options: TIMES TIMEB HZ=100 [sysconf value] timing function used: times
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192
bytes
md5 5295.13k 18837.99k 56614.66k 113037.65k
157575.85k
des cbc 21280.46k 22224.36k 22536.87k 22626.65k
22686.38k
sign verify sign/s verify/s
rsa 1024 bits 0.007653s 0.000381s 130.7 2626.1

OpenSSL 0.9.8d 28 Sep 2006
built on: Wed May 30 00:55:23 PDT 2007
options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx)
compiler:
gcc -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -mcpu=pentium -DL_ENDIAN -DTERMIO -O3 -fomit-frame-pointer -Wall -DOPENSSL_BN_ASM_PART_WOR DS -DOPENSSL_IA32_SSE2 -DSHA1_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM available timing options: TIMES TIMEB HZ=100 [sysconf value] timing function used: times
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192
bytes
md5 5355.29k 18880.64k 56515.93k 113145.51k
157908.99k
des cbc 21361.71k 22185.28k 22494.72k 22654.29k
22659.07k
sign verify sign/s verify/s
rsa 1024 bits 0.007612s 0.000380s 131.4 2629.5

DS

#7 lore raw

Change IP TOS inside ethernet driver

I want to change the IP TOS inside ioctl callback of ethernet driver, but don't know how to do this. Anybody has ideas?

Thanks in advance,
Gavin.

#9 lore raw
In reply to Willy Tarreau

Hello,

Quoted 10 lines · Willy Tarreau

In my experience, it's not much the context switch by itself which causes performance degradation, but the fact that with threads, you have to put mutexes everywhere. And frankly, walking a list with locks everywhere is quite slower than doing it in one run at a rate of 3 or 4 cycles per entry. Also, local storage in function returns is not possible anymore, and some functions even need to malloc() instead of returning statically allocated data. I believe this is the reason for openssl being twice as slow when compiled thread-safe than in native mode.

Quoted 6 lines

So in fact, converting a threaded program to a pure async model should not improve it much because of the initial architectural design. But a program written from scratch to be purely async should perform better simply because it has less operations to perform. And there's no magics here : less cycles spend synchronizing and locking = more cycles available for the real job.

The thing is that the synchronization overhead is something you'll have to pay anyway to support multiple processors. Actually, supporting multiple processors on an async program is beyond painful. Either you have to restrict all locking to busy locks or introduce new state for each possibly blocking synchronization point and what happens if they have to nest? You kind of end up with stackable state thingie - an extremely restricted stack.

If you're really serious about performance and scalability, you just have to support multiple processors and if you do it right the performance overhead shouldn't be too high. Common servers will soon have 8 cores on two physical processors - paying some overhead for synchronization is pretty good deal for scalability.

Quoted 11 lines
Quoted 5 lines

In my experience with web caches, epoll or similar for idle clients and thread per active client scaled and performed pretty well - it needed more memory but the performance wasn't worse than asynchronous design and doing complex server in async model is a lot of pain.

It's true that an async model is a lot of pain. But it's always where I got the best performance. For instance, with epoll(), I can achieve
20000 HTTP reqs/s with 40000 concurrent sessions. The best
performance I have observed from threaded competitors was an order of magnitude below on either value (sometimes both).

Well, it all depends on how you do it but an order of magnitude performance difference sounds too much to me. Memory-wise scalability can be worse by orders of magnitude. You need to restrict per-thread stack size and use epoll for idle threads, if you wanna scale. Workers
+ async monitoring of idle clients scale pretty well.

Quoted 2 lines

However, I agree that few uses really require to spend time writing and debugging async programs.

Yeap, also there are several things which just are too painful in async server - e.g. adding coordination with another server (virus scan, sharing cached data), implementing pluggable extension framwork for third parties (and what happens if they should be able to stack!), and maintaining the damn thing while trying to add a few features. :-)

IMHO, complex pure async server doesn't really make sense anymore.

Thanks.

Signature · 1 line
tejun
#8 lore raw
In reply to David Schwartz
Quoted 10 lines · David Schwartz
Quoted 9 lines

In my experience, it's not much the context switch by itself which causes performance degradation, but the fact that with threads, you have to put mutexes everywhere. And frankly, walking a list with locks everywhere is quite slower than doing it in one run at a rate of 3 or 4 cycles per entry. Also, local storage in function returns is not possible anymore, and some functions even need to malloc() instead of returning statically allocated data. I believe this is the reason for openssl being twice as slow when compiled thread-safe than in native mode.

Quoted 4 lines

I don't know where you got this idea from, but several experiments of mine failed to confirm it. Here, for example, are two identical builds of OpenSSL, one with 'threads' and one with 'no-threads'. Same compiler, same other flags, same hardware (Linux 2.6.21.2, P3-1Ghz):

I can see that your numbers are equal in both cases. You have openssl 0.9.8d, and doing functions benchmarks. I observed this on openssl 0.9.6 + apache 1.3
+ mod_ssl. The difference in terms of hits/s was 1 to 2, just with -DREENTRANT.

Most probably some water has flowed under the bridges since... Anyway, that makes 0.9.8 interesting to look at ;-)

Willy

Quoted 43 lines

OpenSSL 0.9.8d 28 Sep 2006
built on: Wed May 30 00:55:25 PDT 2007
options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx)
compiler:
gcc -DDSO_DLFCN -DHAVE_DLFCN_H -mcpu=pentium -DL_ENDIAN -DTERMIO -O3 -fomit- frame-pointer -Wall -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DSHA1_A SM -DMD5_ASM -DRMD160_ASM -DAES_ASM
available timing options: TIMES TIMEB HZ=100 [sysconf value] timing function used: times
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192
bytes
md5 5295.13k 18837.99k 56614.66k 113037.65k
157575.85k
des cbc 21280.46k 22224.36k 22536.87k 22626.65k
22686.38k
sign verify sign/s verify/s
rsa 1024 bits 0.007653s 0.000381s 130.7 2626.1

OpenSSL 0.9.8d 28 Sep 2006
built on: Wed May 30 00:55:23 PDT 2007
options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx)
compiler:
gcc -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -mcpu=pentium -DL_ENDIAN -DTERMIO -O3 -fomit-frame-pointer -Wall -DOPENSSL_BN_ASM_PART_WOR DS -DOPENSSL_IA32_SSE2 -DSHA1_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM available timing options: TIMES TIMEB HZ=100 [sysconf value] timing function used: times
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192
bytes
md5 5355.29k 18880.64k 56515.93k 113145.51k
157908.99k
des cbc 21361.71k 22185.28k 22494.72k 22654.29k
22659.07k
sign verify sign/s verify/s
rsa 1024 bits 0.007612s 0.000380s 131.4 2629.5

DS

#10 lore raw
In reply to Tejun Heo
Quoted 20 lines · Tejun Heo

Hello,

Quoted 10 lines · Willy Tarreau

In my experience, it's not much the context switch by itself which causes performance degradation, but the fact that with threads, you have to put mutexes everywhere. And frankly, walking a list with locks everywhere is quite slower than doing it in one run at a rate of 3 or 4 cycles per entry. Also, local storage in function returns is not possible anymore, and some functions even need to malloc() instead of returning statically allocated data. I believe this is the reason for openssl being twice as slow when compiled thread-safe than in native mode.

Quoted 6 lines

So in fact, converting a threaded program to a pure async model should not improve it much because of the initial architectural design. But a program written from scratch to be purely async should perform better simply because it has less operations to perform. And there's no magics here : less cycles spend synchronizing and locking = more cycles available for the real job.

Quoted 2 lines

The thing is that the synchronization overhead is something you'll have to pay anyway to support multiple processors.

But you don't need to sync *everything*. It is doable to have 1 thread per processor, each with their own data, and sync the minimum information (eg: statistics).

Quoted 6 lines

Actually, supporting
multiple processors on an async program is beyond painful. Either you have to restrict all locking to busy locks or introduce new state for each possibly blocking synchronization point and what happens if they have to nest? You kind of end up with stackable state thingie - an extremely restricted stack.

I have not said it is simple, I said that when it is justified, it is doable.

Quoted 18 lines

If you're really serious about performance and scalability, you just have to support multiple processors and if you do it right the performance overhead shouldn't be too high. Common servers will soon have 8 cores on two physical processors - paying some overhead for synchronization is pretty good deal for scalability.

Quoted 11 lines
Quoted 5 lines

In my experience with web caches, epoll or similar for idle clients and thread per active client scaled and performed pretty well - it needed more memory but the performance wasn't worse than asynchronous design and doing complex server in async model is a lot of pain.

It's true that an async model is a lot of pain. But it's always where I got the best performance. For instance, with epoll(), I can achieve
20000 HTTP reqs/s with 40000 concurrent sessions. The best
performance I have observed from threaded competitors was an order of magnitude below on either value (sometimes both).

Quoted 3 lines

Well, it all depends on how you do it but an order of magnitude performance difference sounds too much to me. Memory-wise scalability can be worse by orders of magnitude.

It is very often a problem because system limits have not evolved as fast as requirements.

Quoted 3 lines

You need to restrict per-thread
stack size and use epoll for idle threads, if you wanna scale. Workers
+ async monitoring of idle clients scale pretty well.

I agree with a small pool of workers. But they must be dedicated to CPU only, and perform no I/O. Then you can have 1 thread/CPU.

Quoted 10 lines
Quoted 2 lines

However, I agree that few uses really require to spend time writing and debugging async programs.

Yeap, also there are several things which just are too painful in async server - e.g. adding coordination with another server (virus scan, sharing cached data), implementing pluggable extension framwork for third parties (and what happens if they should be able to stack!), and maintaining the damn thing while trying to add a few features. :-)

IMHO, complex pure async server doesn't really make sense anymore.

That's clearly not my opinion, but I don't want to enter a flamewar on the subject, it's not interesting. As long as people like us will push the system to limits using either model, at least there will be references for comparisons :-)

Cheers
Willy

#11 lore raw

Hello,

Quoted 6 lines · Willy Tarreau
Quoted 2 lines

The thing is that the synchronization overhead is something you'll have to pay anyway to support multiple processors.

But you don't need to sync *everything*. It is doable to have 1 thread per processor, each with their own data, and sync the minimum information (eg: statistics).

Agreed it's doable but the list of things to synchronize tends to grow and it's really hurts when it does. Also, the list becomes much less different between async and threaded model with multi processor support.

[--snip--]

Quoted 6 lines
Quoted 3 lines

You need to restrict per-thread
stack size and use epoll for idle threads, if you wanna scale. Workers
+ async monitoring of idle clients scale pretty well.

I agree with a small pool of workers. But they must be dedicated to CPU only, and perform no I/O. Then you can have 1 thread/CPU.

Well, if you go that far, it's basically async model with multiprocessor support. Not my favorite one. :-)

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

However, I agree that few uses really require to spend time writing and debugging async programs.

Yeap, also there are several things which just are too painful in async server - e.g. adding coordination with another server (virus scan, sharing cached data), implementing pluggable extension framwork for third parties (and what happens if they should be able to stack!), and maintaining the damn thing while trying to add a few features. :-)

Quoted 6 lines
Quoted 1 line

IMHO, complex pure async server doesn't really make sense anymore.

That's clearly not my opinion, but I don't want to enter a flamewar on the subject, it's not interesting. As long as people like us will push the system to limits using either model, at least there will be references for comparisons :-)

Yeap, it's getting off-topic. Let's see what future brings. :-)

Thanks.

Signature · 1 line
tejun