Introduce to batch variants of accept() and epoll_ctl() syscall

Hi,

 We encounter a performance problem in a large scale computer cluster, which needs to handle a lot of incoming concurrent TCP connection requests.

 The top shows the kernel is most cpu hog, the testing is simple, just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to si% is about 2:5.

 I also asked some experienced webserver/proxy developers in my team for suggestions, it seem that behavior of many userland programs already called accept() multiple times after it is waked up by epoll_wait(). And the common action is adding the fd that accept() return into epoll interface by epoll_ctl() syscall then.

 Therefore, I think that we'd better to introduce to batch variants of accept() and epoll_ctl() syscall, just like sendmmsg() or recvmmsg().

 For accept(), we may need a new syscall, it may like this,

 struct accept_result {
     int fd;
     struct sockaddr addr;
     socklen_t addr_len;
 };

 int maccept4(int fd, int flags, int nr_accept_result, struct accept_result *results);

 For epoll_ctl(), there are two means to extend it, I prefer to extend current interface instead of introduce to new syscall. We may introduce to a new flag EPOLL_CTL_BATCH. If userland call epoll_ctl() with this flag set, the meaning of last two arguments of epoll_ctl() change, .e.g:

 struct batch_epoll_event batch_event[] = {
        {
             .fd = a_newsock_fd;
             .epoll_event = { ... };
        },
        ...
 };

 ret = epoll_ctl(fd, EPOLL_CTL_ADD|EPOLL_CTL_BATCH, nr_batch_events, batch_events);

Quoted 43 lines · Li Yu

Hi,

We encounter a performance problem in a large scale computer
cluster, which needs to handle a lot of incoming concurrent TCP connection requests.

The top shows the kernel is most cpu hog, the testing is simple,
just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to si% is about 2:5.

I also asked some experienced webserver/proxy developers in my team
for suggestions, it seem that behavior of many userland programs already called accept() multiple times after it is waked up by epoll_wait(). And the common action is adding the fd that accept() return into epoll interface by epoll_ctl() syscall then.

Therefore, I think that we'd better to introduce to batch variants of
accept() and epoll_ctl() syscall, just like sendmmsg() or recvmmsg().

For accept(), we may need a new syscall, it may like this,

  struct accept_result {
      int fd;
      struct sockaddr addr;
      socklen_t addr_len;
  };

int maccept4(int fd, int flags, int nr_accept_result, struct
accept_result *results);

For epoll_ctl(), there are two means to extend it, I prefer to extend
current interface instead of introduce to new syscall. We may introduce to a new flag EPOLL_CTL_BATCH. If userland call epoll_ctl() with this flag set, the meaning of last two arguments of epoll_ctl() change, .e.g:

  struct batch_epoll_event batch_event[] = {
         {
              .fd = a_newsock_fd;
              .epoll_event = { ... };
         },
         ...
  };

Quoted 3 lines

ret = epoll_ctl(fd, EPOLL_CTL_ADD|EPOLL_CTL_BATCH, nr_batch_events,
batch_events);

I think it is good idea. Would you please implement a prototype and give some numbers? This kind of data may help selling this idea. Thanks.

Of course, I think that implementing them should not be a hard work :)

Em. I really do not know whether it is necessary to introduce to a new syscall here. An alternative solution to add new socket option to handle such batch requirement, so applications also can detect if kernel has this extended ability with a easy getsockopt() call.

Any way, I am going to try to write a prototype first.

The top shows the kernel is most cpu hog, the testing is simple,
just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to si% is about 2:5.

I believe accept() is the problem here, because it contends with the softirq processing the tcp session handshake.

Quoted 7 lines · Li Yu

Of course, I think that implementing them should not be a hard work :)

Em. I really do not know whether it is necessary to introduce to a new syscall here. An alternative solution to add new socket option to handle such batch requirement, so applications also can detect if kernel has this extended ability with a easy getsockopt() call.

Quoted 1 line

Any way, I am going to try to write a prototype first.

Before that, could you post the result of "perf top", or "perf record ...;perf report"

Quoted 3 lines

The top shows the kernel is most cpu hog, the testing is simple,
just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to si% is about 2:5.

This ratio is not meaningful, if we dont know where time is spent.

I doubt epoll_ctl(ADD) is a problem here...

If it is, batching the fds wont speed the thing anyway...

I believe accept() is the problem here, because it contends with the softirq processing the tcp session handshake.

Quoted 9 lines · Eric Dumazet

Quoted 7 lines · Li Yu

Of course, I think that implementing them should not be a hard work :)

Em. I really do not know whether it is necessary to introduce to a new syscall here. An alternative solution to add new socket option to handle such batch requirement, so applications also can detect if kernel has this extended ability with a easy getsockopt() call.

Quoted 5 lines

Quoted 1 line

Any way, I am going to try to write a prototype first.

Before that, could you post the result of "perf top", or "perf record ...;perf report"

Sorry for I just have time to write a benchmark to reproduce this problem on my test bed, below are results of "perf record -g -C 0". kernel is 3.4.0:

Events: 7K cycles

(quoted diff) +1 -1

+  54.87%  swapper  [kernel.kallsyms]  [k] poll_idle
-   3.10%   :22984  [kernel.kallsyms]  [k] _raw_spin_lock
    - _raw_spin_lock
       - 64.62% sch_direct_xmit
            dev_queue_xmit
            ip_finish_output
            ip_output
          - ip_local_out
             + 49.48% ip_queue_xmit
             + 37.48% ip_build_and_send_pkt
             + 13.04% ip_send_skb

I can not reproduce complete same high CPU usage on my testing environment, but top show that it has similar ratio of sys% and si% on one CPU:

Tasks: 125 total, 2 running, 123 sleeping, 0 stopped, 0 zombie
Cpu0 : 1.0%us, 30.7%sy, 0.0%ni, 18.8%id, 0.0%wa, 0.0%hi, 49.5%si, 0.0%st

Well, it seem that I must acknowledge I was wrong here. however, I recall that I indeed ever encountered this in another benchmarking a small packets performance.

I guess, this is since TX softirq and syscall context contend same lock in sch_direct_xmit(), is this right?

thanks

Yu

Quoted 17 lines

Quoted 3 lines

The top shows the kernel is most cpu hog, the testing is simple,
just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to si% is about 2:5.

This ratio is not meaningful, if we dont know where time is spent.

I doubt epoll_ctl(ADD) is a problem here...

If it is, batching the fds wont speed the thing anyway...

I believe accept() is the problem here, because it contends with the softirq processing the tcp session handshake.

We encounter a performance problem in a large scale computer
cluster, which needs to handle a lot of incoming concurrent TCP connection requests.

The top shows the kernel is most cpu hog, the testing is simple,
just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to si% is about 2:5.

I also asked some experienced webserver/proxy developers in my team
for suggestions, it seem that behavior of many userland programs already
called accept() multiple times after it is waked up by epoll_wait(). And the common action is adding the fd that accept() return into epoll interface by epoll_ctl() syscall then.

Therefore, I think that we'd better to introduce to batch
variants of
accept() and epoll_ctl() syscall, just like sendmmsg() or recvmmsg().