Skip to contentSkip to conversation

Multicast packet loss

PATCH Applied Kenny Chang 10 participants 69 messages Jan 30, 2009 – Apr 8, 2009

Subsystems (9)
lore
Share & tools
lore
#0 lore raw

Hi all,

We've been having some issues with multicast packet loss, we were wondering if anyone knows anything about the behavior we're seeing.

Background: we use multicast messaging with lots of messages per sec for our work. We recently transitioned many of our systems from an Ubuntu Dapper Drake ia32 distribution to Ubuntu Hardy Heron x86_64. Since the transition, we've noticed much more multicast packet loss, and we think it's related to the transition. Our particular theory is that it's specifically a 32 vs 64-bit issue.

We narrowed the problem down to the attached program (mcasttest.cc). Run "mcasttest server" on one machine -- it'll send 500,000 messages small message to a multicast group, 50,000 messages per second. If we run "mcasttest client" on another machine, it'll receive all those messages and print a count at the end of how many messages it sees. It almost never loses any messages. However, if we run 4 copies of the client on the same machine, receiving the same data, then the programs usually sees fewer than 500,000 messages. We're running with:

for i in $(seq 1 4); do (./mcasttest client &); done

We know this because the program prints a count, but dropped packets also show up in ifconfig's "RX packets" section.

Things we're curious about: do other people see similar problems? The tests
we've done: we've tried this program on a bunch of different machines, all of
which are running either dapper ia32 or hardy x86_64. Uniformly, the dapper
machines have no problems but on certain machines, Hardy shows significant loss. We did some experiments on a troubled machine, varying the OS install, including mixed installations where the kernel was 64-bit and the userspace was 32-bit. This is what we found:

On machines that exhibit this problem, the ksoftirqd process seems to be pegged to 100% CPU when receiving packets.

Note: while we're on Ubuntu, we've tried this with other distros and have seen similar results, we just haven't tabulated them.

Quoted 13 lines

----------------------------------------------------------------------------
userland | userland arch | kernel | kernel arch | mode
----------------------------------------------------------------------------

Dapper   |            32 | 2.6.15-28-server |          32 | no packet loss
Dapper   |            32 | 2.6.22-generic   |          32 | no packet loss Dapper   |            32 | 2.6.22-server    |          32 | no packet loss Hardy    |            32 | 2.6.24-rt        |          32 | no packet loss
Hardy    |            32 | 2.6.24-generic   |          32 | ~5% packet loss
Hardy    |            32 | 2.6.24-server    |          32 | ~10% packet loss
Hardy    |            32 | 2.6.22-server    |          64 | no packet loss
Hardy    |            32 | 2.6.24-rt        |          64 | no packet loss
Hardy    |            32 | 2.6.24-generic   |          64 | 14% packet loss
Hardy    |            32 | 2.6.24-server    |          64 | 12% packet loss
Quoted 5 lines
Hardy    |            64 | 2.6.22-vanilla   |          64 | packet loss
Hardy    |            64 | 2.6.24-rt        |          64 | ~5% packet loss
Hardy    |            64 | 2.6.24-server    |          64 | ~30% packet loss
Hardy    |            64 | 2.6.24-generic   |          64 | ~5% packet loss

----------------------------------------------------------------------------

It's not exactly clear what exactly the problem is but dapper shows no issues regardless of what we try. For hardy, userspace seem to matter: 2.6.24-rt kernel shows no packet loss for 32&64bit kernels, as long as the userspace is 32-bit.

Kernel comments:
2.6.15-28-server: This is Ubuntu Dapper's stock kernel build. 2.6.24-*: This is Ubuntu Hardy's stock kernel. 2.6.22-{generic,server}: This is a custom, in-house kernel build, built for ia32. 2.6.22-vanilla: This is our custom, in-house kernel build, built for x86_64.

We don't think it's related to our custom kernels, because the same phenomena show up with the Ubuntu stock kernels.

Hardware:

The benchmark machine We've been using is an Intel Xeon E5440 @2.83GHz dual-cpu quad-core with Broadcom NetXtreme II BCM5708 bnx2 networking.

We've also tried AMD machines, as well as machines with Tigon3 partno(BCM95704A6) tg3 network cards, they all show consistent behavior.

Our hardy x86_64 server machines all appear to have this problem, new and old.

On the other hand, a desktop with Intel Q6600 quad core 2.4GHz and Intel 82566DC GigE seem to work fine.

All of the dapper ia32 machines have no trouble, even our older hardware.

Thanks,
Kenny Chang
Athena Capital Research

#1 lore raw
Quoted 112 lines · Kenny Chang

Hi all,

We've been having some issues with multicast packet loss, we were wondering if anyone knows anything about the behavior we're seeing.

Background: we use multicast messaging with lots of messages per sec for our
work. We recently transitioned many of our systems from an Ubuntu Dapper Drake
ia32 distribution to Ubuntu Hardy Heron x86_64. Since the transition, we've noticed much more multicast packet loss, and we think it's related to the transition. Our particular theory is that it's specifically a 32 vs 64-bit issue.

We narrowed the problem down to the attached program (mcasttest.cc). Run "mcasttest server" on one machine -- it'll send 500,000 messages small message
to a multicast group, 50,000 messages per second. If we run "mcasttest client"
on another machine, it'll receive all those messages and print a count at the
end of how many messages it sees. It almost never loses any messages. However,
if we run 4 copies of the client on the same machine, receiving the same data,
then the programs usually sees fewer than 500,000 messages. We're running with:

for i in $(seq 1 4); do (./mcasttest client &); done

We know this because the program prints a count, but dropped packets also show up in ifconfig's "RX packets" section.

Things we're curious about: do other people see similar problems? The tests
we've done: we've tried this program on a bunch of different machines, all of
which are running either dapper ia32 or hardy x86_64. Uniformly, the dapper machines have no problems but on certain machines, Hardy shows significant loss. We did some experiments on a troubled machine, varying the OS install, including mixed installations where the kernel was 64-bit and the userspace was
32-bit. This is what we found:

On machines that exhibit this problem, the ksoftirqd process seems to be pegged to 100% CPU when receiving packets.

Note: while we're on Ubuntu, we've tried this with other distros and have seen
similar results, we just haven't tabulated them.

Quoted 26 lines

----------------------------------------------------------------------------

userland | userland arch | kernel | kernel arch |
mode
----------------------------------------------------------------------------

Dapper | 32 | 2.6.15-28-server | 32 | no packet
loss
Dapper | 32 | 2.6.22-generic | 32 | no packet
loss Dapper | 32 | 2.6.22-server | 32 | no
packet loss Hardy | 32 | 2.6.24-rt | 32
| no packet loss
Hardy | 32 | 2.6.24-generic | 32 | ~5% packet
loss
Hardy | 32 | 2.6.24-server | 32 | ~10%
packet loss

Hardy | 32 | 2.6.22-server | 64 | no packet
loss
Hardy | 32 | 2.6.24-rt | 64 | no packet
loss
Hardy | 32 | 2.6.24-generic | 64 | 14% packet
loss
Hardy | 32 | 2.6.24-server | 64 | 12% packet
loss

Quoted 9 lines
Hardy    |            64 | 2.6.22-vanilla   |          64 | packet loss
Hardy    |            64 | 2.6.24-rt        |          64 | ~5% packet

loss
Hardy | 64 | 2.6.24-server | 64 | ~30%
packet loss
Hardy | 64 | 2.6.24-generic | 64 | ~5% packet
loss
----------------------------------------------------------------------------

It's not exactly clear what exactly the problem is but dapper shows no issues regardless of what we try. For hardy, userspace seem to matter:
2.6.24-rt kernel shows no packet loss for 32&64bit kernels, as long as the userspace is 32-bit.

Kernel comments:
2.6.15-28-server: This is Ubuntu Dapper's stock kernel build. 2.6.24-*: This is Ubuntu Hardy's stock kernel. 2.6.22-{generic,server}: This is a custom, in-house kernel build, built for ia32.
2.6.22-vanilla: This is our custom, in-house kernel build, built for x86_64.

We don't think it's related to our custom kernels, because the same phenomena
show up with the Ubuntu stock kernels.

Hardware:

The benchmark machine We've been using is an Intel Xeon E5440 @2.83GHz dual-cpu quad-core with Broadcom NetXtreme II BCM5708 bnx2 networking.

We've also tried AMD machines, as well as machines with Tigon3 partno(BCM95704A6) tg3 network cards, they all show consistent behavior.

Quoted 10 lines

Our hardy x86_64 server machines all appear to have this problem, new and old.

On the other hand, a desktop with Intel Q6600 quad core 2.4GHz and Intel 82566DC GigE
seem to work fine.

All of the dapper ia32 machines have no trouble, even our older hardware.

Hi Kenny

Interesting... You forgot the mcasttest.cc program

Any chance you try a recent kernel (2.6.29-rcX) ?

Could you post "cat /proc/interrupts" results (one for working
setup, another for non working/droping setup)

#2 lore raw
In reply to Kenny Chang
Quoted 4 lines · On Friday 30 January 2009 19:49:48 Kenny Chang wrote:

Hi all,

We've been having some issues with multicast packet loss, we were wondering if anyone knows anything about the behavior we're seeing.

I didn't work much on multicast, but i have heavy unicast udp streaming (PEP for satellite).

First thing to check
net.core.wmem_max = 131071
net.core.wmem_default = 124928
net.ipv4.udp_mem = 379008 505344 758016

Usually they are too small by default.

next
netstat -s

Important part
Udp:

    1263992126 packets received
    260196 packets to unknown port received.
    627001 packet receive errors
    74235906 packets sent
    RcvbufErrors: 56683
    SndbufErrors: 4295851

in your case SndbufErrors matter.

#3 lore raw
In reply to Kenny Chang
Quoted 85 lines · Kenny Chang

Hi all,

We've been having some issues with multicast packet loss, we were wondering if anyone knows anything about the behavior we're seeing.

Background: we use multicast messaging with lots of messages per sec for our work. We recently transitioned many of our systems from an Ubuntu Dapper Drake ia32 distribution to Ubuntu Hardy Heron x86_64. Since the transition, we've noticed much more multicast packet loss, and we think it's related to the transition. Our particular theory is that it's specifically a 32 vs 64-bit issue.

We narrowed the problem down to the attached program (mcasttest.cc). Run "mcasttest server" on one machine -- it'll send 500,000 messages small message to a multicast group, 50,000 messages per second. If we run "mcasttest client" on another machine, it'll receive all those messages and print a count at the end of how many messages it sees. It almost never loses any messages. However, if we run 4 copies of the client on the same machine, receiving the same data, then the programs usually sees fewer than 500,000 messages. We're running with:

for i in $(seq 1 4); do (./mcasttest client &); done

We know this because the program prints a count, but dropped packets also show up in ifconfig's "RX packets" section.

Things we're curious about: do other people see similar problems? The tests we've done: we've tried this program on a bunch of different machines, all of which are running either dapper ia32 or hardy x86_64. Uniformly, the dapper machines have no problems but on certain machines, Hardy shows significant loss. We did some experiments on a troubled machine, varying the OS install, including mixed installations where the kernel was 64-bit and the userspace was
32-bit. This is what we found:

On machines that exhibit this problem, the ksoftirqd process seems to be pegged to 100% CPU when receiving packets.

Note: while we're on Ubuntu, we've tried this with other distros and have seen similar results, we just haven't tabulated them.

Quoted 17 lines

----------------------------------------------------------------------------
userland | userland arch | kernel | kernel arch | mode

----------------------------------------------------------------------------

Dapper   |            32 | 2.6.15-28-server |          32 | no packet loss
Dapper   |            32 | 2.6.22-generic   |          32 | no packet 
loss Dapper   |            32 | 2.6.22-server    |          32 | no 
packet loss Hardy    |            32 | 2.6.24-rt        |          32 | 
no packet loss
Hardy    |            32 | 2.6.24-generic   |          32 | ~5% packet loss
Hardy    |            32 | 2.6.24-server    |          32 | ~10% packet loss
Hardy    |            32 | 2.6.22-server    |          64 | no packet loss
Hardy    |            32 | 2.6.24-rt        |          64 | no packet loss
Hardy    |            32 | 2.6.24-generic   |          64 | 14% packet loss
Hardy    |            32 | 2.6.24-server    |          64 | 12% packet loss
Quoted 5 lines
Hardy    |            64 | 2.6.22-vanilla   |          64 | packet loss
Hardy    |            64 | 2.6.24-rt        |          64 | ~5% packet loss
Hardy    |            64 | 2.6.24-server    |          64 | ~30% packet loss
Hardy    |            64 | 2.6.24-generic   |          64 | ~5% packet loss

----------------------------------------------------------------------------

It's not exactly clear what exactly the problem is but dapper shows no issues regardless of what we try. For hardy, userspace seem to matter:
2.6.24-rt kernel shows no packet loss for 32&64bit kernels, as long as the userspace is 32-bit.

Kernel comments:
2.6.15-28-server: This is Ubuntu Dapper's stock kernel build. 2.6.24-*: This is Ubuntu Hardy's stock kernel. 2.6.22-{generic,server}: This is a custom, in-house kernel build, built for ia32. 2.6.22-vanilla: This is our custom, in-house kernel build, built for x86_64.

We don't think it's related to our custom kernels, because the same phenomena show up with the Ubuntu stock kernels.

Hardware:

The benchmark machine We've been using is an Intel Xeon E5440 @2.83GHz dual-cpu quad-core with Broadcom NetXtreme II BCM5708 bnx2 networking.

We've also tried AMD machines, as well as machines with Tigon3 partno(BCM95704A6) tg3 network cards, they all show consistent behavior.

Quoted 7 lines

Our hardy x86_64 server machines all appear to have this problem, new and old.

On the other hand, a desktop with Intel Q6600 quad core 2.4GHz and Intel 82566DC GigE seem to work fine.

All of the dapper ia32 machines have no trouble, even our older hardware.

Like Eric mentioned, I'd start with a latest kernel if at all possible. If it doesn't happen there, you're work is half over, you just need to figure out what changed, and tell Canonical to backport it.

Quoted 1 line

From there, you can solve this like most packet loss issues are solved:

1) Determine if its a rx or tx packet loss. From your comments above it sounds like this is an rx side issue

2) Look at statistics from the hardware to the application. Use ethtool & /proc/net/dev to get hardware packet loss stats, /proc/net/snmp netstat -s to get core network loss stats

3) Use those stats to identify where and why packets are getting dropped. Posting some summary of that data here is something we can help with if need be

4) Determine how to reduce the loss (i.e. code change vs. tuning)

5) Lather, rinse repeat (given that eliminating a drop cause in one location will likely increase througput, potentially putting strain on another location in the code path, possibly leading to more drops elsewhere.

You had mentioned that ifconfig was showing rx drops, which indicates that your hardware rx buffer is likely overflowing. Usually the best way to fix that is to:

1) modify any available interrupt coalescing parameters on the driver such that interrupts have less latency between packet arrival and assertion

2) increase (if possible) the napi weight (I think thats still the right term) so that each napi poll interation receives more frames on the interface, draining that queue more quickly.

Neil

#4 lore raw

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

Neil Norman wrote:

1) Determine if its a rx or tx packet loss. From your comments above it sounds like this is an rx side issue

We're pretty sure it's an rx issue. Other machines receiving at the
same time will
get all the packets.

I'll gather the information mentioned and summarize in a subsequent email.

Thanks!
Kenny

Quoted 114 lines · Neil Horman
Quoted 76 lines · Kenny Chang

Hi all,

We've been having some issues with multicast packet loss, we were wondering if anyone knows anything about the behavior we're seeing.

Background: we use multicast messaging with lots of messages per sec for our work. We recently transitioned many of our systems from an Ubuntu Dapper Drake ia32 distribution to Ubuntu Hardy Heron x86_64. Since the transition, we've noticed much more multicast packet loss, and we think it's related to the transition. Our particular theory is that it's specifically a 32 vs 64-bit issue.

We narrowed the problem down to the attached program (mcasttest.cc). Run "mcasttest server" on one machine -- it'll send 500,000 messages small message to a multicast group, 50,000 messages per second. If we run "mcasttest client" on another machine, it'll receive all those messages and print a count at the end of how many messages it sees. It almost never loses any messages. However, if we run 4 copies of the client on the same machine, receiving the same data, then the programs usually sees fewer than 500,000 messages. We're running with:

for i in $(seq 1 4); do (./mcasttest client &); done

We know this because the program prints a count, but dropped packets also show up in ifconfig's "RX packets" section.

Things we're curious about: do other people see similar problems? The tests
we've done: we've tried this program on a bunch of different machines, all of
which are running either dapper ia32 or hardy x86_64. Uniformly, the dapper
machines have no problems but on certain machines, Hardy shows significant loss. We did some experiments on a troubled machine, varying the OS install, including mixed installations where the kernel was 64-bit and the userspace was 32-bit. This is what we found:

On machines that exhibit this problem, the ksoftirqd process seems to be pegged to 100% CPU when receiving packets.

Note: while we're on Ubuntu, we've tried this with other distros and have seen similar results, we just haven't tabulated them.

Quoted 14 lines

----------------------------------------------------------------------------
userland | userland arch | kernel | kernel arch | mode

----------------------------------------------------------------------------

Dapper   |            32 | 2.6.15-28-server |          32 | no packet loss
Dapper   |            32 | 2.6.22-generic   |          32 | no packet loss Dapper   |            32 | 2.6.22-server    |          32 | no packet loss Hardy    |            32 | 2.6.24-rt        |          32 | no packet loss
Hardy    |            32 | 2.6.24-generic   |          32 | ~5% packet loss
Hardy    |            32 | 2.6.24-server    |          32 | ~10% packet loss
Hardy    |            32 | 2.6.22-server    |          64 | no packet loss
Hardy    |            32 | 2.6.24-rt        |          64 | no packet loss
Hardy    |            32 | 2.6.24-generic   |          64 | 14% packet loss
Hardy    |            32 | 2.6.24-server    |          64 | 12% packet loss
Quoted 6 lines
Hardy    |            64 | 2.6.22-vanilla   |          64 | packet loss
Hardy    |            64 | 2.6.24-rt        |          64 | ~5% packet loss
Hardy    |            64 | 2.6.24-server    |          64 | ~30% packet loss
Hardy    |            64 | 2.6.24-generic   |          64 | ~5% packet loss

----------------------------------------------------------------------------

It's not exactly clear what exactly the problem is but dapper shows no issues regardless of what we try. For hardy, userspace seem to matter: 2.6.24-rt kernel shows no packet loss for 32&64bit kernels, as long as the userspace is 32-bit.

Kernel comments:
2.6.15-28-server: This is Ubuntu Dapper's stock kernel build. 2.6.24-*: This is Ubuntu Hardy's stock kernel. 2.6.22-{generic,server}: This is a custom, in-house kernel build, built for ia32. 2.6.22-vanilla: This is our custom, in-house kernel build, built for x86_64.

We don't think it's related to our custom kernels, because the same phenomena show up with the Ubuntu stock kernels.

Hardware:

The benchmark machine We've been using is an Intel Xeon E5440 @2.83GHz dual-cpu quad-core with Broadcom NetXtreme II BCM5708 bnx2 networking.

We've also tried AMD machines, as well as machines with Tigon3 partno(BCM95704A6) tg3 network cards, they all show consistent behavior.

Quoted 8 lines

Our hardy x86_64 server machines all appear to have this problem, new and old.

On the other hand, a desktop with Intel Q6600 quad core 2.4GHz and Intel 82566DC GigE seem to work fine.

All of the dapper ia32 machines have no trouble, even our older hardware.

Like Eric mentioned, I'd start with a latest kernel if at all possible. If it doesn't happen there, you're work is half over, you just need to figure out what changed, and tell Canonical to backport it.

From there, you can solve this like most packet loss issues are solved:

1) Determine if its a rx or tx packet loss. From your comments above it sounds like this is an rx side issue

2) Look at statistics from the hardware to the application. Use ethtool & /proc/net/dev to get hardware packet loss stats, /proc/net/snmp netstat -s to get core network loss stats

3) Use those stats to identify where and why packets are getting dropped. Posting some summary of that data here is something we can help with if need be

4) Determine how to reduce the loss (i.e. code change vs. tuning)

5) Lather, rinse repeat (given that eliminating a drop cause in one location will likely increase througput, potentially putting strain on another location in the code path, possibly leading to more drops elsewhere.

You had mentioned that ifconfig was showing rx drops, which indicates that your hardware rx buffer is likely overflowing. Usually the best way to fix that is to:

Quoted 10 lines

1) modify any available interrupt coalescing parameters on the driver such that interrupts have less latency between packet arrival and assertion

2) increase (if possible) the napi weight (I think thats still the right term) so that each napi poll interation receives more frames on the interface, draining that queue more quickly.

Neil

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <arpa/inet.h>
#include <sys/epoll.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/select.h>
#include <unistd.h>

void error(const char *s)

{
    fprintf(stderr, "%s\n", s);
    exit(1);
}

void check(int v)

{
    int myerr = errno;
    char *myerrstr = strerror(myerr);
    if(!v)
        error("bad return code");
}

const char *g_mcastaddr = "239.100.0.99";
int g_port = 10100;

int main(int argc, char **argv)

{
    if(argc != 2)
        error("usage: mcasttest (server|client)");
    if(strcmp(argv[1], "client") == 0)
    {
        // Client program: subscribes to a multicast group, receives messages
        // and prints a count of messages received once it's done.

        int s = socket(AF_INET, SOCK_DGRAM, 0);
        check(s > 0);
        int val = 1;
        check(setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val)) == 0);

        struct sockaddr_in addr;
        memset(&addr, 0, sizeof(addr));
        addr.sin_family = AF_INET;
        addr.sin_port = htons(g_port);
        addr.sin_addr.s_addr = htonl(INADDR_ANY);
        check(bind(s, (struct sockaddr *) &addr, sizeof(addr)) == 0);

        struct ip_mreqn mreq;
        memset(&mreq, 0, sizeof(mreq));
        check(inet_pton(AF_INET, g_mcastaddr, &mreq.imr_multiaddr));
        mreq.imr_address.s_addr = htonl(INADDR_ANY);
        mreq.imr_ifindex = 0;
        check(setsockopt(s, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) == 0);

        int bufSz;
        socklen_t len = sizeof(bufSz);
        getsockopt(s, SOL_SOCKET, SO_RCVBUF, (char*)(&bufSz), &len);
        printf("bufsz: %d\n", bufSz);

        int npackets = 0;
        char buf[1000];
        memset(buf, 0, sizeof(buf));
        while(1)
        {
            struct sockaddr_in from;
            socklen_t fromlen = sizeof(from);
            check(recvfrom(s, buf, 1000, 0, (struct sockaddr*)&from, &fromlen) == 100);
            ++npackets;
            if(buf[0] == 1) // exit message
                break;
        }
        printf("received %d packets\n", npackets);
    }
    else if(strcmp(argv[1], "server") == 0)
    {
        // Server program: sends 50,000 packets per second to a multicast address,
        // for 10 seconds.
        int s = socket(AF_INET, SOCK_DGRAM, 0);
        int val = 1;
        int i = 1;
        check(s > 0);

        struct sockaddr_in addr;
        memset(&addr, 0, sizeof(addr));
        addr.sin_family = AF_INET;
        addr.sin_port = htons(g_port);
        check(inet_pton(AF_INET, g_mcastaddr, &addr.sin_addr.s_addr));
        check(connect(s, (struct sockaddr *) &addr, sizeof(addr)) == 0);

        int npackets = 500000;
        char buf[100];
        memset(buf, 0, sizeof(buf));
        for(i = 1; i < npackets; ++i)
        {
            check(send(s, buf, sizeof(buf), 0) > 0);
            usleep(20); // 50,000 messages per second
        }

        buf[0] = 1;
        for(i = 1; i < 5; ++i)
        {
            check(send(s, buf, sizeof(buf), 0) > 0);
            sleep(1);
        }
    }
    else
        error("unknown mode");
    return 0;
}
#5 lore raw
Quoted 8 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)

I was asking you "cat /proc/interrupts" because I believe you might have a problem NIC interrupts being handled by one CPU only (when having problems)

#6 lore raw
Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

Quoted 8 lines

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)

I was asking you "cat /proc/interrupts" because I believe you might have a problem NIC interrupts being handled by one CPU only (when having problems)

That would be expected (if irqbalance is running), and desireable, since spreading high volume interrupts like NICS accross multiple cores (or more specifically multiple L2 caches), is going increase your cache line miss rate significantly and decrease rx throughput.

Although you do have a point here, if the system isn't running irqbalance, and the NICS irq affinity is spread accross multiple L2 caches, that would be a point of improvement performance-wise.

Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo and your stats that I asked about earlier, that would be a big help.

Regards
Neil

#7 lore raw
Quoted 31 lines · Neil Horman
Quoted 11 lines · Eric Dumazet
Quoted 9 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

Quoted 9 lines

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)

I was asking you "cat /proc/interrupts" because I believe you might have a problem NIC interrupts being handled by one CPU only (when having problems)

That would be expected (if irqbalance is running), and desireable, since spreading high volume interrupts like NICS accross multiple cores (or more specifically multiple L2 caches), is going increase your cache line miss rate significantly and decrease rx throughput.

Although you do have a point here, if the system isn't running irqbalance, and the NICS irq affinity is spread accross multiple L2 caches, that would be a point of improvement performance-wise.

Quoted 7 lines

Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo and your stats that I asked about earlier, that would be a big help.

Regards
Neil

Hi Neil,

Here's the information you requested.

Kenny

kchang@beast8:~$ uname -a
Linux beast8 2.6.24-19-server #1 SMP Wed Aug 20 18:43:06 UTC 2008 x86_64 GNU/Linux
kchang@beast8:~$ cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 0
siblings    : 4
core id        : 0
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5322.91
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU L5430 @ 2.66GHz
stepping : 10
cpu MHz : 2659.999
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 5320.03
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU L5430 @ 2.66GHz
stepping : 10
cpu MHz : 2659.999
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 5320.06
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU L5430 @ 2.66GHz
stepping : 10
cpu MHz : 2659.999
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 5320.06
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:

processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU L5430 @ 2.66GHz
stepping : 10
cpu MHz : 2659.999
cache size : 6144 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 5320.07
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:

processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU L5430 @ 2.66GHz
stepping : 10
cpu MHz : 2659.999
cache size : 6144 KB
physical id : 1
siblings : 4
core id : 1
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 5320.07
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:

processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU L5430 @ 2.66GHz
stepping : 10
cpu MHz : 2659.999
cache size : 6144 KB
physical id : 1
siblings : 4
core id : 2
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 5320.08
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:

processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU L5430 @ 2.66GHz
stepping : 10
cpu MHz : 2659.999
cache size : 6144 KB
physical id : 1
siblings : 4
core id : 3
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 5320.08
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:

kchang@beast8:~$ cat /proc/interrupts
          CPU0       CPU1       CPU2       CPU3       CPU4       
CPU5       CPU6       CPU7      
 0:         67          0          1          0          0          
0          0          0   IO-APIC-edge      timer
 1:          0          0          0          0          0          
0          0          0   IO-APIC-edge      i8042
 8:          0          0          0          1          0          
0          0          0   IO-APIC-edge      rtc
 9:          0          0          0          0          0          
0          0          0   IO-APIC-fasteoi   acpi
14:         12         13         13         13         12         10         13         13   IO-APIC-edge      libata
15:          0          0          0          0          0          0          0          0   IO-APIC-edge      libata
17:        294        295        293        294        294        296        293        288   IO-APIC-fasteoi   aacraid
22:          6          5          5          5          6          6          6          6   IO-APIC-fasteoi   uhci_hcd:usb3
23:          7          8          8          7          7          8          7          8   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb4
2294:         48         46         48         48         49         47         48         51   PCI-MSI-edge      eth0
NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC:       5088       3394       3129       2835       2561       2938       2576       2798   Local timer interrupts
RES:         59        119         58         36         34         71         50         17   Rescheduling interrupts
CAL:        132        128        149        141        138        140        152        140   function call interrupts
TLB:        285        178        278        183        297        191        295        157   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
ERR:          0
#8 lore raw
In reply to Neil Horman
Quoted 31 lines · Neil Horman
Quoted 11 lines · Eric Dumazet
Quoted 9 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

Quoted 9 lines

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)

I was asking you "cat /proc/interrupts" because I believe you might have a problem NIC interrupts being handled by one CPU only (when having problems)

That would be expected (if irqbalance is running), and desireable, since spreading high volume interrupts like NICS accross multiple cores (or more specifically multiple L2 caches), is going increase your cache line miss rate significantly and decrease rx throughput.

Although you do have a point here, if the system isn't running irqbalance, and the NICS irq affinity is spread accross multiple L2 caches, that would be a point of improvement performance-wise.

Quoted 7 lines

Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo and your stats that I asked about earlier, that would be a big help.

Regards
Neil

This is for a working setup.

-Kenny

kchang@fiji:~$ uname -a
Linux fiji 2.6.24-19-generic #1 SMP Wed Aug 20 17:53:40 UTC 2008 x86_64 GNU/Linux
kchang@fiji:~$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
stepping        : 11
cpu MHz         : 1600.000
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 10
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips    : 4791.31
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
stepping : 11
cpu MHz : 1600.000
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips : 4788.05
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
stepping : 11
cpu MHz : 1600.000
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips : 4788.08
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
stepping : 11
cpu MHz : 1600.000
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips : 4788.07
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

kchang@fiji:~$ cat /proc/interrupts
          CPU0       CPU1       CPU2       CPU3      
 0:        165          0          0          0   IO-APIC-edge      timer
 1:          2          0          0          0   IO-APIC-edge      i8042
 8:          1          0          0          0   IO-APIC-edge      rtc
 9:          0          0          0          0   IO-APIC-fasteoi   acpi
12:          4          0          0          0   IO-APIC-edge      i8042
16:   25614400          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3, HDA Intel, eth1, nvidia
17:     571932          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4, uhci_hcd:usb6
18:     102824          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb7
22:          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1
23:    1636819          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5
507:   12542966          0          0          0   PCI-MSI-edge      eth0
508:    1201118          0          0          0   PCI-MSI-edge      ahci
NMI:          0          0          0          0   Non-maskable interrupts
LOC:   29214662   20141857   21777347   14279251   Local timer interrupts
RES:     205758     173268     238058     123958   Rescheduling interrupts
CAL:       2623       3732       3814       2747   function call interrupts
TLB:      29961      56621      31440      55783   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
#9 lore raw
Quoted 32 lines · Kenny Chang
Quoted 31 lines · Neil Horman
Quoted 11 lines · Eric Dumazet
Quoted 9 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

Quoted 9 lines

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)

I was asking you "cat /proc/interrupts" because I believe you might have a problem NIC interrupts being handled by one CPU only (when having problems)

That would be expected (if irqbalance is running), and desireable, since spreading high volume interrupts like NICS accross multiple cores (or more specifically multiple L2 caches), is going increase your cache line miss rate significantly and decrease rx throughput.

Although you do have a point here, if the system isn't running irqbalance, and the NICS irq affinity is spread accross multiple L2 caches, that would be a point of improvement performance-wise.

Quoted 9 lines
Quoted 7 lines

Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo and your stats that I asked about earlier, that would be a big help.

Regards
Neil

This is for a working setup.

Are these quad core systems? Or dual core w/ hyperthreading? I ask because in your working setup you have 1/2 the number of cpus' and was not sure if you removed an entire package of if you just disabled hyperthreading.

Neil

#10 lore raw
Quoted 51 lines · Neil Horman
Quoted 40 lines · Kenny Chang
Quoted 38 lines · Neil Horman
Quoted 13 lines · Eric Dumazet
Quoted 10 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

Quoted 10 lines

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)

I was asking you "cat /proc/interrupts" because I believe you might have a problem NIC interrupts being handled by one CPU only (when having problems)

That would be expected (if irqbalance is running), and desireable, since spreading high volume interrupts like NICS accross multiple cores (or more specifically multiple L2 caches), is going increase your cache line miss rate significantly and decrease rx throughput.

Although you do have a point here, if the system isn't running irqbalance, and the NICS irq affinity is spread accross multiple L2 caches, that would be a point of improvement performance-wise.

Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo and your stats that I asked about earlier, that would be a big help.

Quoted 8 lines
Quoted 5 lines

Regards
Neil

This is for a working setup.

Quoted 8 lines

Are these quad core systems? Or dual core w/ hyperthreading? I ask because in your working setup you have 1/2 the number of cpus' and was not sure if you removed an entire package of if you just disabled hyperthreading.

Neil

Yeah, these are quad core systems. The 8 cpu system is a dual-processor quad-core. The other is my desktop, single cpu quad core.

Kenny

#11 lore raw
Quoted 52 lines · Kenny Chang
Quoted 51 lines · Neil Horman
Quoted 33 lines · Kenny Chang
Quoted 31 lines · Neil Horman
Quoted 11 lines · Eric Dumazet
Quoted 9 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

Quoted 9 lines

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)

I was asking you "cat /proc/interrupts" because I believe you might have a problem NIC interrupts being handled by one CPU only (when having problems)

That would be expected (if irqbalance is running), and desireable, since spreading high volume interrupts like NICS accross multiple cores (or more specifically multiple L2 caches), is going increase your cache line miss rate significantly and decrease rx throughput.

Although you do have a point here, if the system isn't running irqbalance, and the NICS irq affinity is spread accross multiple L2 caches, that would be a point of improvement performance-wise.

Quoted 10 lines
Quoted 7 lines

Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo and your stats that I asked about earlier, that would be a big help.

Regards
Neil

This is for a working setup.

Are these quad core systems? Or dual core w/ hyperthreading? I ask because in your working setup you have 1/2 the number of cpus' and was not sure if you removed an entire package of if you just disabled hyperthreading.

Quoted 6 lines
Quoted 3 lines

Neil

Yeah, these are quad core systems. The 8 cpu system is a dual-processor quad-core. The other is my desktop, single cpu quad core.

Ok, so their separate systms then. Did you actually experience drops on the 8-core system since the last reboot? I ask because even when its distributed across all 8 cores, you only have about 500 total interrupts from the NIC, and if you did get drops, something more than just affinity is wrong.

Regardless, spreading interrupts across cores is definately a problem. As eric says, quad core chips are actually 2x2 cores, so you'll want to either just run irqbalance to assign an apropriate affinity to the NIC, or manually look at each cores physical id and sibling id, to assign affininty to a core or cores that share an L2 cache. If you need to, as you've found, you may need to disable msi interrupt mode on your bnx2 driver. That kinda stinks, but bnx2 IIRC isn't multiqueue, so its not like msi provides you any real performance gain.

Neil

Quoted 7 lines

Kenny

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw
Quoted 82 lines · Neil Horman
Quoted 60 lines · Kenny Chang
Quoted 58 lines · Neil Horman
Quoted 41 lines · Kenny Chang
Quoted 38 lines · Neil Horman
Quoted 13 lines · Eric Dumazet
Quoted 10 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

Quoted 10 lines

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)

I was asking you "cat /proc/interrupts" because I believe you might have a problem NIC interrupts being handled by one CPU only (when having problems)

That would be expected (if irqbalance is running), and desireable, since spreading high volume interrupts like NICS accross multiple cores (or more specifically multiple L2 caches), is going increase your cache line miss rate significantly and decrease rx throughput.

Although you do have a point here, if the system isn't running irqbalance, and the NICS irq affinity is spread accross multiple L2 caches, that would be a point of improvement performance-wise.

Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo and your stats that I asked about earlier, that would be a big help.

Quoted 9 lines
Quoted 5 lines

Regards
Neil

This is for a working setup.

Are these quad core systems? Or dual core w/ hyperthreading? I ask because in your working setup you have 1/2 the number of cpus' and was not sure if you removed an entire package of if you just disabled hyperthreading.

Quoted 7 lines
Quoted 4 lines

Neil

Yeah, these are quad core systems. The 8 cpu system is a dual-processor quad-core. The other is my desktop, single cpu quad core.

Ok, so their separate systms then. Did you actually experience drops on the 8-core system since the last reboot? I ask because even when its distributed across all 8 cores, you only have about 500 total interrupts from the NIC, and if you did get drops, something more than just affinity is wrong.

Regardless, spreading interrupts across cores is definately a problem. As eric says, quad core chips are actually 2x2 cores, so you'll want to either just run irqbalance to assign an apropriate affinity to the NIC, or manually look at each cores physical id and sibling id, to assign affininty to a core or cores that share an L2 cache. If you need to, as you've found, you may need to disable msi interrupt mode on your bnx2 driver. That kinda stinks, but bnx2 IIRC isn't multiqueue, so its not like msi provides you any real performance gain.

Quoted 3 lines

Neil

Hi Neil,

Yeah, we've been rebooting this system left and right switch kernels. The results are fairly consistent. We were able to set the irq affinities, and as Wes had mentioned, what we see is that if we pin the softirq to 1 core, and pin the app to its sibling, we see really good performance, but as we load up other cores, the machine reaches a breaking point where all hell breaks loose and we drop a bunch. (we hadn't turned off msi btw..)

While we were able to tune and adjust performance like that, in the end, it doesn't really explain the difference between earlier and recent kernels, also it doesn't quite explain the difference between machines.

You mentioned it would be good to see the interrupts for each kernel, in light of the above information, would it still be useful for me to provide that?

Kenny

#13 lore raw
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Are these quad core systems? Or dual core w/ hyperthreading? I ask because in your working setup you have 1/2 the number of cpus' and was not sure if you removed an entire package of if you just disabled hyperthreading.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Neil

Yeah, these are quad core systems. The 8 cpu system is a dual-processor quad-core. The other is my desktop, single cpu quad core.

Just to be clear: on the 2 x quad core system, we can run with a 2.6.15 kernel and see no packet drops. In fact, we can run with 2.6.19, 2.6.20, and 2.6.21 just fine. 2.6.22 is the first kernel that shows problems.

Kenny posted results from a working setup on a different machine.

What I would really like to know is if whatever changed between 2.6.21 and 2.6.22 that broke things is confined just to bnx2. To make this a rigorous test, we would need to use the same machine with a different nic, which we don't have quite yet. An Intel Pro 1000 ethernet card is in the mail as I type this.

I also tried forward porting the bnx2 driver in 2.6.21 to 2.6.22 (unsuccessfully), and building the most recent driver from the Broadcom site to Ubuntu Hardy's 2.6.24. The most recent driver with hardy 2.6.24 showed similar packet dropping problems. Hm, perhaps I'll try to build the most recent broadcom driver against 2.6.21.

Wes

#14 lore raw
Quoted 31 lines · Wesley Chow
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

Are these quad core systems? Or dual core w/ hyperthreading? I ask because in
your working setup you have 1/2 the number of cpus' and was not sure if you
removed an entire package of if you just disabled hyperthreading.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Neil

Yeah, these are quad core systems. The 8 cpu system is a dual-processor quad-core. The other is my desktop, single cpu quad core.

Just to be clear: on the 2 x quad core system, we can run with a 2.6.15 kernel and see no packet drops. In fact, we can run with 2.6.19, 2.6.20, and 2.6.21 just fine. 2.6.22 is the first kernel that shows problems.

Kenny posted results from a working setup on a different machine.

What I would really like to know is if whatever changed between 2.6.21 and 2.6.22 that broke things is confined just to bnx2. To make this a rigorous test, we would need to use the same machine with a different nic, which we don't have quite yet. An Intel Pro 1000 ethernet card is in the mail as I type this.

Quoted 6 lines

I also tried forward porting the bnx2 driver in 2.6.21 to 2.6.22 (unsuccessfully), and building the most recent driver from the Broadcom site to Ubuntu Hardy's 2.6.24. The most recent driver with hardy 2.6.24 showed similar packet dropping problems. Hm, perhaps I'll try to build the most recent broadcom driver against 2.6.21.

Try oprofile session, you shall see a scheduler effect (dont want to call this a regression, no need for another flame war).

also give us "vmstat 1" results (number of context switches per second)

On recent kernels, scheduler might be faster than before: You get more wakeups per second and more work to do by softirq handler (it does more calls to scheduler, thus less cpu cycles available for draining NIC RX queue in time)

opcontrol --vmlinux=/path/vmlinux --start
<run benchmark>
opreport -l /path/vmlinux | head -n 50

Recent schedulers tend to be optimum for lower latencies (and thus, on a high level of wakeups, you get less bandwidth because of sofirq using a whole CPU)

For example, if you have one tread receiving data on 4 or 8 sockets, you'll probably notice better throughput (because it will sleep less often)

Multicast receiving on N sockets, with one thread waiting on each socket is basically a way to trigger a scheduler storm. (N wakeups per packet). So its more a benchmark to stress scheduler than stressing network stack...

Maybe its time to change user side, and not try to find an appropriate kernel :)

If you know you have to receive N frames per 20us units, then its better to :
Use non blocking sockets, and doing such loop :

{
usleep(20); // or try to compensate if this thread is slowed too much by following code

for (i = 0 ; i < N ; i++) {
	while (revfrom(socket[N], ....) != -1)
		receive_frame(...);
	}
}

That way, you are pretty sure network softirq handler wont have to spend time trying to wakeup 400.000 time per second one thread. All cpu cycles can be spent in NIC driver and network stack.

Your thread will do 50.000 calls to nanosleep() per second, that is not really expensive, then N recvfrom() per iteration. It should work on all past , current and future kernels.

#15 lore raw
Quoted 77 lines · Eric Dumazet
Quoted 31 lines · Wesley Chow
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

Are these quad core systems? Or dual core w/ hyperthreading? I ask because in
your working setup you have 1/2 the number of cpus' and was not sure if you
removed an entire package of if you just disabled hyperthreading.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Neil

Yeah, these are quad core systems. The 8 cpu system is a dual-processor quad-core. The other is my desktop, single cpu quad core.

Just to be clear: on the 2 x quad core system, we can run with a 2.6.15 kernel and see no packet drops. In fact, we can run with 2.6.19, 2.6.20, and 2.6.21 just fine. 2.6.22 is the first kernel that shows problems.

Kenny posted results from a working setup on a different machine.

What I would really like to know is if whatever changed between 2.6.21 and 2.6.22 that broke things is confined just to bnx2. To make this a rigorous test, we would need to use the same machine with a different nic, which we don't have quite yet. An Intel Pro 1000 ethernet card is in the mail as I type this.

Quoted 6 lines

I also tried forward porting the bnx2 driver in 2.6.21 to 2.6.22 (unsuccessfully), and building the most recent driver from the Broadcom site to Ubuntu Hardy's 2.6.24. The most recent driver with hardy 2.6.24 showed similar packet dropping problems. Hm, perhaps I'll try to build the most recent broadcom driver against 2.6.21.

Try oprofile session, you shall see a scheduler effect (dont want to call this a regression, no need for another flame war).

also give us "vmstat 1" results (number of context switches per second)

On recent kernels, scheduler might be faster than before: You get more wakeups per second and more work to do by softirq handler (it does more calls to scheduler, thus less cpu cycles available for draining NIC RX queue in time)

opcontrol --vmlinux=/path/vmlinux --start
<run benchmark>
opreport -l /path/vmlinux | head -n 50

Recent schedulers tend to be optimum for lower latencies (and thus, on a high level of wakeups, you get less bandwidth because of sofirq using a whole CPU)

For example, if you have one tread receiving data on 4 or 8 sockets, you'll probably notice better throughput (because it will sleep less often)

Multicast receiving on N sockets, with one thread waiting on each socket is basically a way to trigger a scheduler storm. (N wakeups per packet). So its more a benchmark to stress scheduler than stressing network stack...

Maybe its time to change user side, and not try to find an appropriate kernel :)

If you know you have to receive N frames per 20us units, then its better to :
Use non blocking sockets, and doing such loop :

{
usleep(20); // or try to compensate if this thread is slowed too much by following code

for (i = 0 ; i < N ; i++) {
	while (revfrom(socket[N], ....) != -1)
		receive_frame(...);
	}
}
Quoted 7 lines

That way, you are pretty sure network softirq handler wont have to spend time trying to wakeup 400.000 time per second one thread. All cpu cycles can be spent in NIC driver and network stack.

Your thread will do 50.000 calls to nanosleep() per second, that is not really expensive, then N recvfrom() per iteration. It should work on all past , current and future kernels.

+1 to this idea. Since the last oprofile traces showed significant variance in the time spent in schedule(), it might be worthwhile to investigate the affects of the application behavior on this. I might also be worth adding a systemtap probe to sys_recvmsg, to count how many times we receive frames on a working and non-working system. If the app is behaving differently on different kernels, and its affecting the number of times you go to get a frame out of the stack, that would affect your drop rates, and it would show up in sys_recvmsg

Neil

#16 lore raw
Quoted 23 lines
Quoted 16 lines

Maybe its time to change user side, and not try to find an appropriate kernel :)

If you know you have to receive N frames per 20us units, then its better to :
Use non blocking sockets, and doing such loop :

{
usleep(20); // or try to compensate if this thread is slowed too much by following code

for (i = 0 ; i < N ; i++) {
	while (revfrom(socket[N], ....) != -1)
		receive_frame(...);
	}
}
Quoted 7 lines

That way, you are pretty sure network softirq handler wont have to spend time trying to wakeup 400.000 time per second one thread. All cpu cycles can be spent in NIC driver and network stack.

Your thread will do 50.000 calls to nanosleep() per second, that is not really expensive, then N recvfrom() per iteration. It should work on all past , current and future kernels.

Quoted 9 lines

+1 to this idea. Since the last oprofile traces showed significant variance in
the time spent in schedule(), it might be worthwhile to investigate the affects of the application behavior on this. I might also be worth adding a systemtap probe to sys_recvmsg, to count how many times we receive frames on a working and non-working system. If the app is behaving differently on different kernels, and its affecting the number of times you go to get a frame out of the stack, that would affect your drop rates, and it would show up in sys_recvmsg

I did some work to our test program to spin on a non-blocking socket and it indeed seems to fix the problem, at least for 2.6.28.1, which was a kernel we had problems with. The number of context switches drastically drops -- from 200,000+ to less than 50!

I haven't done totally comprehensive tests yet, so I don't want to officially state any results. I'm also out today, but Kenny may get a chance to play with it. Spinning on the socket is looking like an interesting solution, but we're a bit nervous about seeing our processes constantly running at 100% CPU. Does C++ have a MachineOnFire exception?

Wes

#17 lore raw
In reply to Kenny Chang
Quoted 89 lines · Kenny Chang
Quoted 75 lines · Neil Horman
Quoted 52 lines · Kenny Chang
Quoted 50 lines · Neil Horman
Quoted 33 lines · Kenny Chang
Quoted 31 lines · Neil Horman
Quoted 11 lines · Eric Dumazet
Quoted 9 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

Quoted 9 lines

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)

I was asking you "cat /proc/interrupts" because I believe you might have a problem NIC interrupts being handled by one CPU only (when having problems)

That would be expected (if irqbalance is running), and desireable, since spreading high volume interrupts like NICS accross multiple cores (or more specifically multiple L2 caches), is going increase your cache line miss rate significantly and decrease rx throughput.

Although you do have a point here, if the system isn't running irqbalance, and the NICS irq affinity is spread accross multiple L2 caches, that would be a point of improvement performance-wise.

Quoted 10 lines
Quoted 7 lines

Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo and your stats that I asked about earlier, that would be a big help.

Regards
Neil

This is for a working setup.

Are these quad core systems? Or dual core w/ hyperthreading? I ask because in your working setup you have 1/2 the number of cpus' and was not sure if you removed an entire package of if you just disabled hyperthreading.

Quoted 8 lines
Quoted 3 lines

Neil

Yeah, these are quad core systems. The 8 cpu system is a dual-processor quad-core. The other is my desktop, single cpu quad core.

Ok, so their separate systms then. Did you actually experience drops on the 8-core system since the last reboot? I ask because even when its distributed across all 8 cores, you only have about 500 total interrupts from the NIC, and if you did get drops, something more than just affinity is wrong.

Regardless, spreading interrupts across cores is definately a problem. As eric says, quad core chips are actually 2x2 cores, so you'll want to either just run irqbalance to assign an apropriate affinity to the NIC, or manually look at each cores physical id and sibling id, to assign affininty to a core or cores that share an L2 cache. If you need to, as you've found, you may need to disable msi interrupt mode on your bnx2 driver. That kinda stinks, but bnx2 IIRC isn't multiqueue, so its not like msi provides you any real performance gain.

Quoted 3 lines

Neil

Hi Neil,

Yeah, we've been rebooting this system left and right switch kernels. The results are fairly consistent. We were able to set the irq affinities, and as Wes had mentioned, what we see is that if we pin the softirq to 1 core, and pin the app to its sibling, we see really good performance, but as we load up other cores, the machine reaches a breaking point where all hell breaks loose and we drop a bunch. (we hadn't turned off msi btw..)

Quoted 8 lines

While we were able to tune and adjust performance like that, in the end, it doesn't really explain the difference between earlier and recent kernels, also it doesn't quite explain the difference between machines.

You mentioned it would be good to see the interrupts for each kernel, in light of the above information, would it still be useful for me to provide that?

In light of what you said, I probably don't need to see it no, although if you go through testing on all these kernels again, I would suggest you take a look at the /proc/interrupt numbers yourself. Like you said, its odd that this behavior changed, since the fast receive path is fairly consistent. It may be that te nic driver your using (bnx2 I think?), had a change that either broke the ability to set affinity for msi interrupts, forcing an irq spread and killing performance, or perhaps some large inefficiency was introduced either in the interrupt handler itself, or in the napi poll method of the driver. Another good analysis technique would be to grab the latest kernel (which is 'broken' I think your chart indicated), and the nic driver from the last working kernel. Merge the driver into the latest kernel and see if the problem persists. If not, thats a pretty good indicator that a change in the driver has at least some culpability.

Best
Neil

Quoted 7 lines

Kenny

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#18 lore raw
In reply to Eric Dumazet
Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

I tried your program on my dev machines and 2.6.29 (each machine : two quad core cpus, 32bits kernel)

With 8 clients, about 10% packet loss,

Might be a scheduling problem, not sure... 50.000 packets per second, x 8 cpus = 400.000 wakeups per second... But at least UDP receive path seems OK.

Thing is the receiver (softirq that queues the packet) seems to fight on socket lock with readers...

I tried to setup IRQ affinities, but it doesnt work any more on bnx2 (unless using msi_disable=1)

I tried playing with ethtool -C|c G|g params...
And /proc/net/core/rmem_max (and setsockopt(RCVBUF) to set bigger receive buffers in your program)

I can have 0% packet loss if booting with msi_disable and

echo 1 >/proc/irq/16/smp_affinities

(16 being interrupt of eth0 NIC)

then, a second run gave me errors, about 2%, oh well...

oprofile numbers without playing IRQ affinities:

CPU: Core 2, speed 2999.89 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        symbol name
327928   10.1427  schedule
259625    8.0301  mwait_idle
187337    5.7943  __skb_recv_datagram
109854    3.3977  lock_sock_nested
104713    3.2387  tick_nohz_stop_sched_tick
98831     3.0568  select_nohz_load_balancer
88163     2.7268  skb_release_data
78552     2.4296  update_curr
75241     2.3272  getnstimeofday
71400     2.2084  set_next_entity
67629     2.0917  get_next_timer_interrupt
67375     2.0839  sched_clock_tick
58112     1.7974  enqueue_entity
56462     1.7463  udp_recvmsg
55049     1.7026  copy_to_user
54277     1.6788  sched_clock_cpu
54031     1.6712  __copy_skb_header
51859     1.6040  __slab_free
51786     1.6017  prepare_to_wait_exclusive
51776     1.6014  sock_def_readable
50062     1.5484  try_to_wake_up
42182     1.3047  __switch_to
41631     1.2876  read_tsc
38337     1.1857  tick_nohz_restart_sched_tick
34358     1.0627  cpu_idle
34194     1.0576  native_sched_clock
33812     1.0458  pick_next_task_fair
33685     1.0419  resched_task
33340     1.0312  sys_recvfrom
33287     1.0296  dst_release
32439     1.0033  kmem_cache_free
32131     0.9938  hrtimer_start_range_ns
29807     0.9219  udp_queue_rcv_skb
27815     0.8603  task_rq_lock
26875     0.8312  __update_sched_clock
23912     0.7396  sock_queue_rcv_skb
21583     0.6676  __wake_up_sync
21001     0.6496  effective_load
20531     0.6350  hrtick_start_fair

With IRQ affinities and msi_disable (no packet drops)

CPU: Core 2, speed 3000.13 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        symbol name
79788    10.3815  schedule
69422     9.0328  mwait_idle
44877     5.8391  __skb_recv_datagram
28629     3.7250  tick_nohz_stop_sched_tick
27252     3.5459  select_nohz_load_balancer
24320     3.1644  lock_sock_nested
20833     2.7107  getnstimeofday
20666     2.6889  skb_release_data
18612     2.4217  set_next_entity
17785     2.3141  get_next_timer_interrupt
17691     2.3018  udp_recvmsg
17271     2.2472  sched_clock_tick
16032     2.0860  copy_to_user
14785     1.9237  update_curr
12512     1.6280  prepare_to_wait_exclusive
12498     1.6262  __slab_free
11380     1.4807  read_tsc
11145     1.4501  sched_clock_cpu
10598     1.3789  __switch_to
9588      1.2475  pick_next_task_fair
9480      1.2335  cpu_idle
9218      1.1994  sys_recvfrom
9008      1.1721  tick_nohz_restart_sched_tick
8977      1.1680  dst_release
8930      1.1619  native_sched_clock
8392      1.0919  kmem_cache_free
8124      1.0570  hrtimer_start_range_ns
7274      0.9464  bnx2_interrupt
7175      0.9336  __copy_skb_header
7006      0.9116  try_to_wake_up
6949      0.9042  sock_def_readable
6787      0.8831  enqueue_entity
6772      0.8811  __update_sched_clock
6349      0.8261  finish_task_switch
6164      0.8020  copy_from_user
5096      0.6631  resched_task
5007      0.6515  sysenter_past_esp

I will try to investigate a litle bit more in following days if time permits.

#19 lore raw
Quoted 128 lines · Eric Dumazet
Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

I tried your program on my dev machines and 2.6.29 (each machine : two quad core cpus, 32bits kernel)

With 8 clients, about 10% packet loss,

Might be a scheduling problem, not sure... 50.000 packets per second, x 8 cpus = 400.000 wakeups per second... But at least UDP receive path seems OK.

Thing is the receiver (softirq that queues the packet) seems to fight on socket lock with readers...

I tried to setup IRQ affinities, but it doesnt work any more on bnx2 (unless using msi_disable=1)

I tried playing with ethtool -C|c G|g params...
And /proc/net/core/rmem_max (and setsockopt(RCVBUF) to set bigger receive buffers in your program)

I can have 0% packet loss if booting with msi_disable and

echo 1 >/proc/irq/16/smp_affinities

(16 being interrupt of eth0 NIC)

then, a second run gave me errors, about 2%, oh well...

oprofile numbers without playing IRQ affinities:

CPU: Core 2, speed 2999.89 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        symbol name
327928   10.1427  schedule
259625    8.0301  mwait_idle
187337    5.7943  __skb_recv_datagram
109854    3.3977  lock_sock_nested
104713    3.2387  tick_nohz_stop_sched_tick
98831     3.0568  select_nohz_load_balancer
88163     2.7268  skb_release_data
78552     2.4296  update_curr
75241     2.3272  getnstimeofday
71400     2.2084  set_next_entity
67629     2.0917  get_next_timer_interrupt
67375     2.0839  sched_clock_tick
58112     1.7974  enqueue_entity
56462     1.7463  udp_recvmsg
55049     1.7026  copy_to_user
54277     1.6788  sched_clock_cpu
54031     1.6712  __copy_skb_header
51859     1.6040  __slab_free
51786     1.6017  prepare_to_wait_exclusive
51776     1.6014  sock_def_readable
50062     1.5484  try_to_wake_up
42182     1.3047  __switch_to
41631     1.2876  read_tsc
38337     1.1857  tick_nohz_restart_sched_tick
34358     1.0627  cpu_idle
34194     1.0576  native_sched_clock
33812     1.0458  pick_next_task_fair
33685     1.0419  resched_task
33340     1.0312  sys_recvfrom
33287     1.0296  dst_release
32439     1.0033  kmem_cache_free
32131     0.9938  hrtimer_start_range_ns
29807     0.9219  udp_queue_rcv_skb
27815     0.8603  task_rq_lock
26875     0.8312  __update_sched_clock
23912     0.7396  sock_queue_rcv_skb
21583     0.6676  __wake_up_sync
21001     0.6496  effective_load
20531     0.6350  hrtick_start_fair

With IRQ affinities and msi_disable (no packet drops)

CPU: Core 2, speed 3000.13 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        symbol name
79788    10.3815  schedule
69422     9.0328  mwait_idle
44877     5.8391  __skb_recv_datagram
28629     3.7250  tick_nohz_stop_sched_tick
27252     3.5459  select_nohz_load_balancer
24320     3.1644  lock_sock_nested
20833     2.7107  getnstimeofday
20666     2.6889  skb_release_data
18612     2.4217  set_next_entity
17785     2.3141  get_next_timer_interrupt
17691     2.3018  udp_recvmsg
17271     2.2472  sched_clock_tick
16032     2.0860  copy_to_user
14785     1.9237  update_curr
12512     1.6280  prepare_to_wait_exclusive
12498     1.6262  __slab_free
11380     1.4807  read_tsc
11145     1.4501  sched_clock_cpu
10598     1.3789  __switch_to
9588      1.2475  pick_next_task_fair
9480      1.2335  cpu_idle
9218      1.1994  sys_recvfrom
9008      1.1721  tick_nohz_restart_sched_tick
8977      1.1680  dst_release
8930      1.1619  native_sched_clock
8392      1.0919  kmem_cache_free
8124      1.0570  hrtimer_start_range_ns
7274      0.9464  bnx2_interrupt
7175      0.9336  __copy_skb_header
7006      0.9116  try_to_wake_up
6949      0.9042  sock_def_readable
6787      0.8831  enqueue_entity
6772      0.8811  __update_sched_clock
6349      0.8261  finish_task_switch
6164      0.8020  copy_from_user
5096      0.6631  resched_task
5007      0.6515  sysenter_past_esp
Quoted 2 lines

I will try to investigate a litle bit more in following days if time permits.

I'm not 100% versed on this, but IIRC, some hardware simply can't set irq affinity when operating in msi interrupt mode. If this is the case with this particular bnx2 card, then I would expect some packet loss, simply due to the constant cache misses. It would be interesting to re-run your oprofile cases, counting L2 cache hits/misses (if your cpu supports that class of counter) for both bnx2 running in msi enabled mode and msi disabled mode. It would also be interesting to use a different card, that can set irq affinity, and compare loss with irqbalance on, and irqbalance off with irq afninty set to all cpus.

Neil

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#20 lore raw
Quoted 129 lines · Neil Horman
Quoted 126 lines · Eric Dumazet
Quoted 9 lines · Eric Dumazet
Quoted 8 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

I tried your program on my dev machines and 2.6.29 (each machine : two quad core cpus, 32bits kernel)

With 8 clients, about 10% packet loss,

Might be a scheduling problem, not sure... 50.000 packets per second, x 8 cpus = 400.000 wakeups per second... But at least UDP receive path seems OK.

Thing is the receiver (softirq that queues the packet) seems to fight on socket lock with readers...

I tried to setup IRQ affinities, but it doesnt work any more on bnx2 (unless using msi_disable=1)

I tried playing with ethtool -C|c G|g params...
And /proc/net/core/rmem_max (and setsockopt(RCVBUF) to set bigger receive buffers in your program)

I can have 0% packet loss if booting with msi_disable and

echo 1 >/proc/irq/16/smp_affinities

(16 being interrupt of eth0 NIC)

then, a second run gave me errors, about 2%, oh well...

oprofile numbers without playing IRQ affinities:

CPU: Core 2, speed 2999.89 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        symbol name
327928   10.1427  schedule
259625    8.0301  mwait_idle
187337    5.7943  __skb_recv_datagram
109854    3.3977  lock_sock_nested
104713    3.2387  tick_nohz_stop_sched_tick
98831     3.0568  select_nohz_load_balancer
88163     2.7268  skb_release_data
78552     2.4296  update_curr
75241     2.3272  getnstimeofday
71400     2.2084  set_next_entity
67629     2.0917  get_next_timer_interrupt
67375     2.0839  sched_clock_tick
58112     1.7974  enqueue_entity
56462     1.7463  udp_recvmsg
55049     1.7026  copy_to_user
54277     1.6788  sched_clock_cpu
54031     1.6712  __copy_skb_header
51859     1.6040  __slab_free
51786     1.6017  prepare_to_wait_exclusive
51776     1.6014  sock_def_readable
50062     1.5484  try_to_wake_up
42182     1.3047  __switch_to
41631     1.2876  read_tsc
38337     1.1857  tick_nohz_restart_sched_tick
34358     1.0627  cpu_idle
34194     1.0576  native_sched_clock
33812     1.0458  pick_next_task_fair
33685     1.0419  resched_task
33340     1.0312  sys_recvfrom
33287     1.0296  dst_release
32439     1.0033  kmem_cache_free
32131     0.9938  hrtimer_start_range_ns
29807     0.9219  udp_queue_rcv_skb
27815     0.8603  task_rq_lock
26875     0.8312  __update_sched_clock
23912     0.7396  sock_queue_rcv_skb
21583     0.6676  __wake_up_sync
21001     0.6496  effective_load
20531     0.6350  hrtick_start_fair

With IRQ affinities and msi_disable (no packet drops)

CPU: Core 2, speed 3000.13 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        symbol name
79788    10.3815  schedule
69422     9.0328  mwait_idle
44877     5.8391  __skb_recv_datagram
28629     3.7250  tick_nohz_stop_sched_tick
27252     3.5459  select_nohz_load_balancer
24320     3.1644  lock_sock_nested
20833     2.7107  getnstimeofday
20666     2.6889  skb_release_data
18612     2.4217  set_next_entity
17785     2.3141  get_next_timer_interrupt
17691     2.3018  udp_recvmsg
17271     2.2472  sched_clock_tick
16032     2.0860  copy_to_user
14785     1.9237  update_curr
12512     1.6280  prepare_to_wait_exclusive
12498     1.6262  __slab_free
11380     1.4807  read_tsc
11145     1.4501  sched_clock_cpu
10598     1.3789  __switch_to
9588      1.2475  pick_next_task_fair
9480      1.2335  cpu_idle
9218      1.1994  sys_recvfrom
9008      1.1721  tick_nohz_restart_sched_tick
8977      1.1680  dst_release
8930      1.1619  native_sched_clock
8392      1.0919  kmem_cache_free
8124      1.0570  hrtimer_start_range_ns
7274      0.9464  bnx2_interrupt
7175      0.9336  __copy_skb_header
7006      0.9116  try_to_wake_up
6949      0.9042  sock_def_readable
6787      0.8831  enqueue_entity
6772      0.8811  __update_sched_clock
6349      0.8261  finish_task_switch
6164      0.8020  copy_from_user
5096      0.6631  resched_task
5007      0.6515  sysenter_past_esp
Quoted 2 lines

I will try to investigate a litle bit more in following days if time permits.

Quoted 8 lines

I'm not 100% versed on this, but IIRC, some hardware simply can't set irq affinity when operating in msi interrupt mode. If this is the case with this particular bnx2 card, then I would expect some packet loss, simply due to the constant cache misses. It would be interesting to re-run your oprofile cases, counting L2 cache hits/misses (if your cpu supports that class of counter) for both bnx2 running in msi enabled mode and msi disabled mode. It would also be interesting to use a different card, that can set irq affinity, and compare loss with irqbalance on, and irqbalance off with irq afninty set to all cpus.

booted with msi_disable=1, IRQ of eth0 handled by CPU0 only, so that oprofile results sorted on CPU0 numbers.

We can see scheduler has hard time to cope with this workload with more of two CPUS

OK up to 30.000 (* 8 sockets) packets per second.

CPU0 is 100% handling softirq (ksoftirqd/0)

CPU: Core 2, speed 3000.31 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000 Samples on CPU 0
Samples on CPU 1
Samples on CPU 2
Samples on CPU 3
Samples on CPU 4
Samples on CPU 5
Samples on CPU 6
Samples on CPU 7

samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        symbol name
6152     12.5595  3         0.0098  3         0.0090  5         0.0156  1         0.0582  0              0  2         0.0065  3         0.0169  enqueue_entity
4453      9.0909  2         0.0065  3         0.0090  4         0.0125  5         0.2910  0              0  1         0.0033  2         0.0113  try_to_wake_up
3837      7.8333  3         0.0098  8         0.0241  0              0  0              0  0              0  0              0  0              0  sock_def_readable
3694      7.5414  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __copy_skb_header
2320      4.7363  1         0.0033  2         0.0060  2         0.0062  1         0.0582  1         0.0028  2         0.0065  0              0  resched_task
1818      3.7115  6         0.0196  32        0.0962  0              0  0              0  0              0  0              0  0              0  sock_queue_rcv_skb
1776      3.6257  0              0  0              0  0              0  0              0  0              0  0              0  0              0  udp_queue_rcv_skb
1677      3.4236  0              0  1         0.0030  0              0  1         0.0582  1         0.0028  0              0  0              0  __slab_alloc
1658      3.3848  260       0.8496  303       0.9109  289       0.9021  24        1.3970  418       1.1730  326       1.0626  173       0.9733  sched_clock_cpu
1614      3.2950  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __wake_up_sync
1600      3.2664  0              0  1         0.0030  0              0  1         0.0582  1         0.0028  0              0  0              0  select_task_rq_fair
1569      3.2032  1299      4.2447  1530      4.5996  1271      3.9675  6         0.3492  1677      4.7062  1275      4.1559  759       4.2703  update_curr
1532      3.1276  4         0.0131  4         0.0120  0              0  2         0.1164  1         0.0028  1         0.0033  1         0.0056  task_rq_lock
1325      2.7050  1         0.0033  7         0.0210  0              0  0              0  0              0  0              0  0              0  skb_queue_tail
1273      2.5989  1         0.0033  1         0.0030  1         0.0031  0              0  0              0  1         0.0033  0              0  enqueue_task_fair
1227      2.5050  0              0  0              0  0              0  0              0  0              0  0              0  0              0  effective_load
1071      2.1865  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __udp4_lib_rcv
1009      2.0599  0              0  0              0  0              0  2         0.1164  1         0.0028  0              0  0              0  activate_task
940       1.9190  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __wake_up_common
930       1.8986  0              0  2         0.0060  0              0  0              0  0              0  0              0  1         0.0056  account_scheduler_latency
859       1.7537  0              0  0              0  0              0  0              0  0              0  0              0  1         0.0056  __skb_clone
609       1.2433  0              0  0              0  0              0  1         0.0582  1         0.0028  1         0.0033  0              0  enqueue_task
588       1.2004  3         0.0098  2         0.0060  5         0.0156  8         0.4657  2         0.0056  3         0.0098  2         0.0113  kmem_cache_alloc
477       0.9738  307       1.0032  322       0.9680  358       1.1175  27        1.5716  338       0.9485  315       1.0268  203       1.1421  native_sched_clock
441       0.9003  0              0  0              0  0              0  0              0  0              0  0              0  0              0  skb_clone
408       0.8329  0              0  0              0  0              0  0              0  0              0  0              0  0              0  ip_route_input
375       0.7656  0              0  0              0  0              0  0              0  0              0  0              0  0              0  bnx2_poll_work
366       0.7472  248       0.8104  269       0.8087  293       0.9146  22        1.2806  289       0.8110  332       1.0822  157       0.8833  __update_sched_clock
327       0.6676  1         0.0033  0              0  0              0  0              0  0              0  2         0.0065  1         0.0056  place_entity
265       0.5410  54        0.1765  62        0.1864  39        0.1217  3         0.1746  84        0.2357  61        0.1988  12        0.0675  rb_insert_color
194       0.3961  2662      8.6985  3291      9.8936  3231     10.0858  372      21.6531  2994      8.4021  3299     10.7533  1719      9.6714  mwait_idle

This problem completely disapears if I launch all clients bounded to CPU1 (and NIC irq still on CPU0)

taskset -p2 ./mcasttest.sh

(No packet loss, while CPU1 has 0% idle time...)
We have less context switches. Once awakened, a task can read several packets in its socket.

CPU: Core 2, speed 3000.31 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000 Samples on CPU 0
Samples on CPU 1
Samples on CPU 2
Samples on CPU 3
Samples on CPU 4
Samples on CPU 5
Samples on CPU 6
Samples on CPU 7

samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        symbol name
25316    13.6664  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __copy_skb_header
14584     7.8729  13        0.0083  2         0.2670  0              0  0              0  1         0.0218  1         0.1577  1         0.2604  task_rq_lock
11624     6.2750  10657     6.7650  3         0.4005  2         0.5013  5         0.7278  36        0.7862  2         0.3155  1         0.2604  update_curr
10038     5.4188  318       0.2019  0              0  0              0  0              0  0              0  0              0  0              0  sock_def_readable
10021     5.4097  0              0  0              0  0              0  0              0  0              0  0              0  0              0  bnx2_interrupt
7777      4.1983  11        0.0070  1         0.1335  2         0.5013  2         0.2911  8         0.1747  3         0.4732  1         0.2604  try_to_wake_up
6559      3.5408  0              0  0              0  0              0  0              0  0              0  0              0  0              0  udp_queue_rcv_skb
6389      3.4490  257       0.1631  0              0  0              0  0              0  0              0  0              0  0              0  sock_queue_rcv_skb
6305      3.4036  6         0.0038  0              0  0              0  0              0  0              0  0              0  1         0.2604  __slab_alloc
5661      3.0560  44        0.0279  2         0.2670  1         0.2506  5         0.7278  0              0  1         0.1577  0              0  kmem_cache_alloc
5529      2.9847  5         0.0032  1         0.1335  4         1.0025  0              0  14        0.3057  3         0.4732  1         0.2604  enqueue_entity
4706      2.5404  64        0.0406  0              0  0              0  0              0  0              0  0              0  0              0  skb_queue_tail
4390      2.3699  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __udp4_lib_rcv
4043      2.1825  0              0  0              0  0              0  0              0  0              0  0              0  0              0  uhci_irq
3897      2.1037  0              0  0              0  0              0  0              0  0              0  0              0  0              0  bnx2_poll_work
3556      1.9196  0              0  296      39.5194  261      65.4135  258      37.5546  650      14.1952  263      41.4826  257      66.9271  mwait_idle
3449      1.8619  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __skb_clone
3348      1.8074  0              0  0              0  0              0  0              0  0              0  0              0  0              0  skb_clone
3243      1.7507  1653      1.0493  7         0.9346  3         0.7519  2         0.2911  63        1.3758  1         0.1577  1         0.2604  sched_clock_cpu
3068      1.6562  1        6.3e-04  0              0  0              0  0              0  0              0  0              0  0              0  __wake_up_sync
2923      1.5779  1        6.3e-04  0              0  0              0  0              0  0              0  0              0  0              0  check_preempt_wakeup
2588      1.3971  1        6.3e-04  4         0.5340  2         0.5013  1         0.1456  1         0.0218  2         0.3155  0              0  enqueue_task_fair
2399      1.2951  0              0  0              0  0              0  0              0  0              0  0              0  0              0  ip_route_input
1986      1.0721  5         0.0032  0              0  0              0  0              0  1         0.0218  0              0  0              0  __wake_up_common
1777      0.9593  132       0.0838  0              0  0              0  3         0.4367  8         0.1747  1         0.1577  1         0.2604  rb_insert_color
1754      0.9469  34        0.0216  0              0  0              0  0              0  0              0  0              0  0              0  __sk_mem_schedule
1550      0.8367  0              0  0              0  0              0  0              0  0              0  0              0  0              0  irq_entries_start
1527      0.8243  0              0  13        1.7356  2         0.5013  0              0  104       2.2712  6         0.9464  0              0  get_next_timer_interrupt
1398      0.7547  6         0.0038  0              0  0              0  1         0.1456  1         0.0218  0              0  0              0  select_task_rq_fair
1159      0.6257  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __alloc_skb

Exchanging CPU0/CPU1 to get oprofile number sorted on the CPU used by user application :

CPU: Core 2, speed 3000.31 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000 Samples on CPU 0
Samples on CPU 1
Samples on CPU 2
Samples on CPU 3
Samples on CPU 4
Samples on CPU 5
Samples on CPU 6
Samples on CPU 7

samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        symbol name
6040     10.1815  8         0.0134  4         0.3208  6         1.3699  1         0.1580  3         1.0909  5         0.6039  0              0  schedule
6014     10.1377  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __skb_recv_datagram
4730      7.9733  36        0.0604  0              0  0              0  0              0  0              0  0              0  0              0  skb_release_data
4014      6.7663  2         0.0034  33        2.6464  1         0.2283  6         0.9479  0              0  1         0.1208  0              0  copy_to_user
3732      6.2910  3708      6.2167  4         0.3208  1         0.2283  3         0.4739  3         1.0909  2         0.2415  0              0  update_curr
3446      5.8089  0              0  0              0  0              0  0              0  0              0  0              0  0              0  lock_sock_nested
2430      4.0962  0              0  0              0  0              0  0              0  0              0  0              0  0              0  udp_recvmsg
2028      3.4186  73        0.1224  0              0  0              0  2         0.3160  0              0  1         0.1208  1         0.3861  __slab_free
1898      3.1994  43        0.0721  0              0  0              0  0              0  0              0  0              0  0              0  dst_release
1645      2.7730  0              0  0              0  0              0  0              0  0              0  0              0  0              0  memcpy_toiovec
1635      2.7561  0              0  0              0  0              0  1         0.1580  0              0  0              0  0              0  copy_from_user
1407      2.3718  2         0.0034  0              0  2         0.4566  6         0.9479  0              0  5         0.6039  1         0.3861  sysenter_past_esp
1389      2.3414  58        0.0972  3         0.2406  3         0.6849  4         0.6319  2         0.7273  3         0.3623  0              0  kmem_cache_free
1135      1.9133  1         0.0017  0              0  0              0  0              0  0              0  0              0  0              0  release_sock
1069      1.8020  0              0  0              0  0              0  0              0  0              0  0              0  0              0  prepare_to_wait_exclusive
1031      1.7379  3         0.0050  0              0  0              0  0              0  0              0  1         0.1208  0              0  put_prev_task_fair
1007      1.6975  0              0  0              0  0              0  0              0  0              0  0              0  0              0  sock_rfree
926       1.5609  0              0  0              0  0              0  0              0  0              0  0              0  0              0  sys_recvfrom
838       1.4126  0              0  0              0  0              0  0              0  0              0  0              0  0              0  skb_copy_datagram_iovec
697       1.1749  27        0.0453  1         0.0802  0              0  0              0  1         0.3636  0              0  0              0  add_partial
697       1.1749  0              0  0              0  0              0  0              0  0              0  1         0.1208  0              0  dequeue_task
604       1.0182  0              0  0              0  0              0  0              0  0              0  0              0  0              0  sock_recvmsg
582       0.9811  933       1.5642  17        1.3633  1         0.2283  2         0.3160  1         0.3636  6         0.7246  0              0  sched_clock_cpu
525       0.8850  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __sk_mem_reclaim
512       0.8631  0              0  0              0  1         0.2283  0              0  0              0  1         0.1208  1         0.3861  __switch_to
489       0.8243  0              0  0              0  0              0  0              0  0              0  0              0  0              0  fget_light
450       0.7586  1         0.0017  2         0.1604  1         0.2283  0              0  0              0  4         0.4831  0              0  rb_erase
409       0.6894  95        0.1593  0              0  0              0  0              0  0              0  0              0  0              0  wakeup_preempt_entity
360       0.6068  0              0  0              0  0              0  0              0  0              0  0              0  0              0  move_addr_to_user
348       0.5866  0              0  0              0  0              0  0              0  0              0  0              0  0              0  sys_socketcall
347       0.5849  1         0.0017  0              0  0              0  0              0  0              0  0              0  1         0.3861  set_next_entity
#21 lore raw
Quoted 139 lines · Eric Dumazet
Quoted 129 lines · Neil Horman
Quoted 126 lines · Eric Dumazet
Quoted 9 lines · Eric Dumazet
Quoted 8 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

I tried your program on my dev machines and 2.6.29 (each machine : two quad core cpus, 32bits kernel)

With 8 clients, about 10% packet loss,

Might be a scheduling problem, not sure... 50.000 packets per second, x 8 cpus = 400.000 wakeups per second... But at least UDP receive path seems OK.

Thing is the receiver (softirq that queues the packet) seems to fight on socket lock with readers...

I tried to setup IRQ affinities, but it doesnt work any more on bnx2 (unless using msi_disable=1)

I tried playing with ethtool -C|c G|g params...
And /proc/net/core/rmem_max (and setsockopt(RCVBUF) to set bigger receive buffers in your program)

I can have 0% packet loss if booting with msi_disable and

echo 1 >/proc/irq/16/smp_affinities

(16 being interrupt of eth0 NIC)

then, a second run gave me errors, about 2%, oh well...

oprofile numbers without playing IRQ affinities:

CPU: Core 2, speed 2999.89 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        symbol name
327928   10.1427  schedule
259625    8.0301  mwait_idle
187337    5.7943  __skb_recv_datagram
109854    3.3977  lock_sock_nested
104713    3.2387  tick_nohz_stop_sched_tick
98831     3.0568  select_nohz_load_balancer
88163     2.7268  skb_release_data
78552     2.4296  update_curr
75241     2.3272  getnstimeofday
71400     2.2084  set_next_entity
67629     2.0917  get_next_timer_interrupt
67375     2.0839  sched_clock_tick
58112     1.7974  enqueue_entity
56462     1.7463  udp_recvmsg
55049     1.7026  copy_to_user
54277     1.6788  sched_clock_cpu
54031     1.6712  __copy_skb_header
51859     1.6040  __slab_free
51786     1.6017  prepare_to_wait_exclusive
51776     1.6014  sock_def_readable
50062     1.5484  try_to_wake_up
42182     1.3047  __switch_to
41631     1.2876  read_tsc
38337     1.1857  tick_nohz_restart_sched_tick
34358     1.0627  cpu_idle
34194     1.0576  native_sched_clock
33812     1.0458  pick_next_task_fair
33685     1.0419  resched_task
33340     1.0312  sys_recvfrom
33287     1.0296  dst_release
32439     1.0033  kmem_cache_free
32131     0.9938  hrtimer_start_range_ns
29807     0.9219  udp_queue_rcv_skb
27815     0.8603  task_rq_lock
26875     0.8312  __update_sched_clock
23912     0.7396  sock_queue_rcv_skb
21583     0.6676  __wake_up_sync
21001     0.6496  effective_load
20531     0.6350  hrtick_start_fair

With IRQ affinities and msi_disable (no packet drops)

CPU: Core 2, speed 3000.13 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        symbol name
79788    10.3815  schedule
69422     9.0328  mwait_idle
44877     5.8391  __skb_recv_datagram
28629     3.7250  tick_nohz_stop_sched_tick
27252     3.5459  select_nohz_load_balancer
24320     3.1644  lock_sock_nested
20833     2.7107  getnstimeofday
20666     2.6889  skb_release_data
18612     2.4217  set_next_entity
17785     2.3141  get_next_timer_interrupt
17691     2.3018  udp_recvmsg
17271     2.2472  sched_clock_tick
16032     2.0860  copy_to_user
14785     1.9237  update_curr
12512     1.6280  prepare_to_wait_exclusive
12498     1.6262  __slab_free
11380     1.4807  read_tsc
11145     1.4501  sched_clock_cpu
10598     1.3789  __switch_to
9588      1.2475  pick_next_task_fair
9480      1.2335  cpu_idle
9218      1.1994  sys_recvfrom
9008      1.1721  tick_nohz_restart_sched_tick
8977      1.1680  dst_release
8930      1.1619  native_sched_clock
8392      1.0919  kmem_cache_free
8124      1.0570  hrtimer_start_range_ns
7274      0.9464  bnx2_interrupt
7175      0.9336  __copy_skb_header
7006      0.9116  try_to_wake_up
6949      0.9042  sock_def_readable
6787      0.8831  enqueue_entity
6772      0.8811  __update_sched_clock
6349      0.8261  finish_task_switch
6164      0.8020  copy_from_user
5096      0.6631  resched_task
5007      0.6515  sysenter_past_esp
Quoted 2 lines

I will try to investigate a litle bit more in following days if time permits.

Quoted 8 lines

I'm not 100% versed on this, but IIRC, some hardware simply can't set irq affinity when operating in msi interrupt mode. If this is the case with this particular bnx2 card, then I would expect some packet loss, simply due to the constant cache misses. It would be interesting to re-run your oprofile cases, counting L2 cache hits/misses (if your cpu supports that class of counter) for both bnx2 running in msi enabled mode and msi disabled mode. It would also be interesting to use a different card, that can set irq affinity, and compare loss with irqbalance on, and irqbalance off with irq afninty set to all cpus.

Quoted 9 lines

booted with msi_disable=1, IRQ of eth0 handled by CPU0 only, so that oprofile results sorted on CPU0 numbers.

We can see scheduler has hard time to cope with this workload with more of two CPUS

OK up to 30.000 (* 8 sockets) packets per second.

CPU0 is 100% handling softirq (ksoftirqd/0)

This explains alot. if the application is scheduled to run on the same cpu that has the irq for the NIC bound to it, you get a perf boost by not having to warm up two caches (1 for the app cpu and one for the irq & softirq work), but you loose it and then some fighting for cpu time. If both the app and the irq are on the same cpu, and we spend so much time in softirq context, we will eventually overflow higher up the network stack, as the application doesn't have enough time to dequeue frames.

It may also speak to the need to make the bnx2 napi routine more efficient :)

Neil

#22 lore raw

(I'm Kenny's colleague, and I've been doing the kernel builds)

First I'd like to note that there were a lot of bnx2 NAPI changes between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss, whereas loss in 2.6.22 is significant.

Second, some CPU affinity info: if I do like Eric and pin all of the apps onto a single CPU, I see no packet loss. Also, I do *not* see ksoftirqd show up on top at all!

If I pin half the processes on one CPU and the other half on another CPU, one ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss in that case is significant (25%).

Now, the strange case: if I pin 3 processes to one CPU and 1 process to another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one of the apps takes significantly less CPU than the others, and all apps lose the *exact same number of packets*. In all other situations where we see packet loss, the actual number lost per application instance appears random.

We're about to plug in an Intel ethernet card into this machine to collect more rigorous testing data. Please note, though, that we have seen packet loss with a tg3 chipset as well. For now, though, I'm assuming that this is purely a bnx2 problem.

If I understand correctly, when the nic signals a hardware interrupt, the kernel grabs it and defers the meaty work to the softirq handler -- how does it decide which ksoftirqd gets the interrupts? Is this something determined by how the driver implements the NAPI?

Wes

#23 lore raw
Quoted 15 lines · Wes Chow

(I'm Kenny's colleague, and I've been doing the kernel builds)

First I'd like to note that there were a lot of bnx2 NAPI changes between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss, whereas loss in 2.6.22 is significant.

Second, some CPU affinity info: if I do like Eric and pin all of the apps onto a single CPU, I see no packet loss. Also, I do *not* see ksoftirqd show up on top at all!

If I pin half the processes on one CPU and the other half on another CPU, one ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss in that case is significant (25%).

Quoted 5 lines

Now, the strange case: if I pin 3 processes to one CPU and 1 process to another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one of the apps takes significantly less CPU than the others, and all apps lose the *exact same number of packets*. In all other situations where we see packet loss, the actual number lost per application instance appears random.

You see same number of packet lost because they are lost at NIC level

(check ifconfig eth0 for droped packets)

if softirq is too busy to process packets, we are not able to get them from hardware in time.

Quoted 10 lines

We're about to plug in an Intel ethernet card into this machine to collect more rigorous testing data. Please note, though, that we have seen packet loss with a tg3 chipset as well. For now, though, I'm assuming that this is purely a bnx2 problem.

If I understand correctly, when the nic signals a hardware interrupt, the kernel grabs it and defers the meaty work to the softirq handler -- how does it decide which ksoftirqd gets the interrupts? Is this something determined by how the driver implements the NAPI?

Normaly, softirq runs on same cpu (the one handling hard irq)

#24 lore raw
Quoted 13 lines · Eric Dumazet
Quoted 12 lines · Wes Chow

(I'm Kenny's colleague, and I've been doing the kernel builds)

First I'd like to note that there were a lot of bnx2 NAPI changes between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss, whereas loss in 2.6.22 is significant.

Second, some CPU affinity info: if I do like Eric and pin all of the apps onto a single CPU, I see no packet loss. Also, I do *not* see ksoftirqd show up on top at all!

If I pin half the processes on one CPU and the other half on another CPU, one ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss in that case is significant (25%).

Quoted 6 lines
Quoted 4 lines

Now, the strange case: if I pin 3 processes to one CPU and 1 process to another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one of the apps takes significantly less CPU than the others, and all apps lose the
*exact same number of packets*. In all other situations where we see packet
loss, the actual number lost per application instance appears random.

You see same number of packet lost because they are lost at NIC level

Understood.

I have a new observation: if I pin processes to just CPUs 0 and 1, I see no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and 3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any other combination appears to produce loss (though I have not tried all 28 combinations, this seems to be the case).

At first I thought maybe it had to do with processes pinned to the same CPU, but different cores. The machine is a dual quad core, which means that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3 produce packet loss.

I've also noticed that it does not matter which of the working pairs I pin to. For example, pinning 5 processes in any combination on either 0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no packet loss.

The failures are also sudden. In all of the working cases mentioned above, I don't see ksoftirqd on top at all. But when I run 6 processes on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of packets.

Quoted 2 lines

Normaly, softirq runs on same cpu (the one handling hard irq)

What determines which CPU the hard irq occurs on?

Wes

#25 lore raw
Quoted 39 lines · Wes Chow
Quoted 26 lines · Eric Dumazet
Quoted 16 lines · Wes Chow

(I'm Kenny's colleague, and I've been doing the kernel builds)

First I'd like to note that there were a lot of bnx2 NAPI changes between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss,
whereas loss in 2.6.22 is significant.

Second, some CPU affinity info: if I do like Eric and pin all of the apps onto a single CPU, I see no packet loss. Also, I do *not* see ksoftirqd show up on top at all!

If I pin half the processes on one CPU and the other half on another CPU, one ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss
in that case is significant (25%).

Quoted 8 lines

Now, the strange case: if I pin 3 processes to one CPU and 1 process to another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one
of the apps takes significantly less CPU than the others, and all apps lose the
*exact same number of packets*. In all other situations where we see packet
loss, the actual number lost per application instance appears random.

Quoted 1 line

You see same number of packet lost because they are lost at NIC level

Understood.

I have a new observation: if I pin processes to just CPUs 0 and 1, I see no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and 3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any other combination appears to produce loss (though I have not tried all 28 combinations, this seems to be the case).

Quoted 4 lines

At first I thought maybe it had to do with processes pinned to the same CPU, but different cores. The machine is a dual quad core, which means that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3 produce packet loss.

a quad core is really a 2 x 2 core

L2 cache is splited on two blocks, one block used by CPU0/1, other by CPU2/3

You are at the limit of the machine with such workload, so as soon as your CPUs have to transfert 64 bytes lines between those two L2 blocks, you loose.

Quoted 12 lines

I've also noticed that it does not matter which of the working pairs I pin to. For example, pinning 5 processes in any combination on either 0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no packet loss.

The failures are also sudden. In all of the working cases mentioned above, I don't see ksoftirqd on top at all. But when I run 6 processes on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of packets.

Quoted 1 line
Quoted 4 lines
Quoted 1 line

Normaly, softirq runs on same cpu (the one handling hard irq)

What determines which CPU the hard irq occurs on?

Check /proc/irq/{irqnumber}/smp_affinity

If you want IRQ16 only served by CPU0 :

echo 1 >/proc/irq/16/smp_affinity

#26 lore raw
Quoted 73 lines · Eric Dumazet
Quoted 39 lines · Wes Chow
Quoted 27 lines · Eric Dumazet
Quoted 16 lines · Wes Chow

(I'm Kenny's colleague, and I've been doing the kernel builds)

First I'd like to note that there were a lot of bnx2 NAPI changes between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss,
whereas loss in 2.6.22 is significant.

Second, some CPU affinity info: if I do like Eric and pin all of the apps onto a single CPU, I see no packet loss. Also, I do *not* see ksoftirqd show up on top at all!

If I pin half the processes on one CPU and the other half on another CPU, one ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss
in that case is significant (25%).

Quoted 9 lines

Now, the strange case: if I pin 3 processes to one CPU and 1 process to another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one
of the apps takes significantly less CPU than the others, and all apps lose the
*exact same number of packets*. In all other situations where we see packet
loss, the actual number lost per application instance appears random.

Quoted 2 lines

You see same number of packet lost because they are lost at NIC level

Understood.

I have a new observation: if I pin processes to just CPUs 0 and 1, I see no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and 3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any other combination appears to produce loss (though I have not tried all 28 combinations, this seems to be the case).

Quoted 5 lines

At first I thought maybe it had to do with processes pinned to the same CPU, but different cores. The machine is a dual quad core, which means that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3 produce packet loss.

a quad core is really a 2 x 2 core

L2 cache is splited on two blocks, one block used by CPU0/1, other by CPU2/3

You are at the limit of the machine with such workload, so as soon as your CPUs have to transfert 64 bytes lines between those two L2 blocks, you loose.

Quoted 11 lines

I've also noticed that it does not matter which of the working pairs I pin to. For example, pinning 5 processes in any combination on either 0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no packet loss.

The failures are also sudden. In all of the working cases mentioned above, I don't see ksoftirqd on top at all. But when I run 6 processes on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of packets.

Quoted 5 lines
Quoted 2 lines

Normaly, softirq runs on same cpu (the one handling hard irq)

What determines which CPU the hard irq occurs on?

Quoted 7 lines

Check /proc/irq/{irqnumber}/smp_affinity

If you want IRQ16 only served by CPU0 :

echo 1 >/proc/irq/16/smp_affinity

Hi everyone,

First, thanks for all the effort so far, I think we've learned so much more about the problem in the last couple of days than we had previously in a month.

Just to summarize where we are:

* pinning processes to specific cores/CPUs alleviate the problem
* issues exist from 2.6.22 up to 2.6.29-rc3
* issue does not appear to be isolated to 64-bit, 32-bits have problems too.
* I'm attaching an updated test program with the PR_SET_TIMERSTACK call added.
* on troubled machines, we are seeing high number of context switches and interrupts.
* we've ordered an Intel card to try in our machine to see if we can circumvent the issue with a different driver.

Kernel Version         Has Problem?     Notes
----------             ----------       ----------
2.6.15.x                N    2.6.16.x                -
2.6.17.x                -               Doesn't build on Hardy
2.6.18.x                -               Doesn't boot (kernel panic)
2.6.19.7                N               ksoftirqd is up there, but not pegging a CPU.

Takes roughly same amount of CPU
as the other
processes, all of which are from
20-40%
2.6.20.21 N
2.6.21.7 N sort of lopsided load, but no load from
ksoftirqd -- strange

2.6.22.19               Y               First broken kernel
2.6.23.x                -
2.6.24-19               Y               (from Hardy)
2.6.25.x                -
2.6.26.x                -
2.6.27.x                Y               (from Intrepid)
2.6.28.1                Y
2.6.29-rc               Y

Correct me if I'm wrong, from what we've seen, it looks like its pointing to some inefficiency in the softirq handling. The question is whether it's something in the driver or the kernel. If we can isolate that, maybe we can take some action to have it fixed.

Kenny

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <arpa/inet.h>
#include <sys/epoll.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/select.h>
#include <unistd.h>

#ifndef PR_SET_TIMERSLACK
#define PR_SET_TIMERSLACK 29
#endif

const char *g_mcastaddr = "239.100.0.99";
int g_port = 10100;

void error(const char *s)

{
    fprintf(stderr, "%s\n", s);
    exit(1);
}

void check(int v)

{
    int myerr = errno;
    char *myerrstr = strerror(myerr);
    if(!v)
        error("bad return code");
}

int main(int argc, char **argv)

{
    if(argc != 2)
        error("usage: mcasttest (server|client)");
    if(strcmp(argv[1], "client") == 0)
    {
        /*
         * Client program: subscribes to a multicast group, receives messages
         * and prints a count of messages received once it's done.
         */
        int s = socket(AF_INET, SOCK_DGRAM, 0);
        check(s > 0);
        int val = 1;
        check(setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val)) == 0);

        struct sockaddr_in addr;
        memset(&addr, 0, sizeof(addr));
        addr.sin_family = AF_INET;
        addr.sin_port = htons(g_port);
        addr.sin_addr.s_addr = htonl(INADDR_ANY);
        check(bind(s, (struct sockaddr *) &addr, sizeof(addr)) == 0);

        struct ip_mreqn mreq;
        memset(&mreq, 0, sizeof(mreq));
        check(inet_pton(AF_INET, g_mcastaddr, &mreq.imr_multiaddr));
        mreq.imr_address.s_addr = htonl(INADDR_ANY);
        mreq.imr_ifindex = 0;
        check(setsockopt(s, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) == 0);

        int bufSz;
        socklen_t len = sizeof(bufSz);
        getsockopt(s, SOL_SOCKET, SO_RCVBUF, (char*)(&bufSz), &len);
        printf("bufsz: %d\n", bufSz);

        int npackets = 0;
        char buf[1000];
        memset(buf, 0, sizeof(buf));
        while(1)
        {
            struct sockaddr_in from;
            socklen_t fromlen = sizeof(from);
            check(recvfrom(s, buf, 1000, 0, (struct sockaddr*)&from, &fromlen) == 100);
            ++npackets;
            if(buf[0] == 1) // exit message
                break;
        }
        printf("received %d packets\n", npackets);
    }
    else if(strcmp(argv[1], "server") == 0)
    {
        /*
         * Setup a timer resolution of 1000 ns : 1 us
         */
        prctl(PR_SET_TIMERSLACK, 1000); 

        /*
         * Server program: sends 50,000 packets per second to a multicast address,
         * for 10 seconds.
         */
        int s = socket(AF_INET, SOCK_DGRAM, 0);
        int val = 1;
        int i = 1;
        check(s > 0);

        struct sockaddr_in addr;
        memset(&addr, 0, sizeof(addr));
        addr.sin_family = AF_INET;
        addr.sin_port = htons(g_port);
        check(inet_pton(AF_INET, g_mcastaddr, &addr.sin_addr.s_addr));
        check(connect(s, (struct sockaddr *) &addr, sizeof(addr)) == 0);

        int npackets = 500000;
        char buf[100];
        memset(buf, 0, sizeof(buf));
        for(i = 1; i < npackets; ++i)
        {
            check(send(s, buf, sizeof(buf), 0) > 0);
            usleep(20); // 50,000 messages per second
        }

        buf[0] = 1;
        for(i = 1; i < 5; ++i)
        {
            check(send(s, buf, sizeof(buf), 0) > 0);
            sleep(1);
        }
    }
    else
        error("unknown mode");
    return 0;
}
#27 lore raw
Quoted 126 lines · Kenny Chang
Quoted 74 lines · Eric Dumazet
Quoted 39 lines · Wes Chow
Quoted 27 lines · Eric Dumazet
Quoted 16 lines · Wes Chow

(I'm Kenny's colleague, and I've been doing the kernel builds)

First I'd like to note that there were a lot of bnx2 NAPI changes between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss,
whereas loss in 2.6.22 is significant.

Second, some CPU affinity info: if I do like Eric and pin all of the apps onto a single CPU, I see no packet loss. Also, I do *not* see ksoftirqd show up on top at all!

If I pin half the processes on one CPU and the other half on another CPU, one ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss
in that case is significant (25%).

Quoted 9 lines

Now, the strange case: if I pin 3 processes to one CPU and 1 process to another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one
of the apps takes significantly less CPU than the others, and all apps lose the
*exact same number of packets*. In all other situations where we see packet
loss, the actual number lost per application instance appears random.

Quoted 2 lines

You see same number of packet lost because they are lost at NIC level

Understood.

I have a new observation: if I pin processes to just CPUs 0 and 1, I see no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and 3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any other combination appears to produce loss (though I have not tried all 28 combinations, this seems to be the case).

Quoted 5 lines

At first I thought maybe it had to do with processes pinned to the same CPU, but different cores. The machine is a dual quad core, which means that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3 produce packet loss.

a quad core is really a 2 x 2 core

L2 cache is splited on two blocks, one block used by CPU0/1, other by CPU2/3

You are at the limit of the machine with such workload, so as soon as your CPUs have to transfert 64 bytes lines between those two L2 blocks, you loose.

Quoted 11 lines

I've also noticed that it does not matter which of the working pairs I pin to. For example, pinning 5 processes in any combination on either 0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no packet loss.

The failures are also sudden. In all of the working cases mentioned above, I don't see ksoftirqd on top at all. But when I run 6 processes on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of packets.

Quoted 5 lines
Quoted 2 lines

Normaly, softirq runs on same cpu (the one handling hard irq)

What determines which CPU the hard irq occurs on?

Quoted 7 lines

Check /proc/irq/{irqnumber}/smp_affinity

If you want IRQ16 only served by CPU0 :

echo 1 >/proc/irq/16/smp_affinity

Hi everyone,

First, thanks for all the effort so far, I think we've learned so much more about the problem in the last couple of days than we had previously in a month.

Just to summarize where we are:

* pinning processes to specific cores/CPUs alleviate the problem
* issues exist from 2.6.22 up to 2.6.29-rc3
* issue does not appear to be isolated to 64-bit, 32-bits have problems too.
* I'm attaching an updated test program with the PR_SET_TIMERSTACK call added.
* on troubled machines, we are seeing high number of context switches and interrupts.
* we've ordered an Intel card to try in our machine to see if we can circumvent the issue with a different driver.

Kernel Version         Has Problem?     Notes
----------             ----------       ----------
2.6.15.x                N    2.6.16.x                -
2.6.17.x                -               Doesn't build on Hardy
2.6.18.x                -               Doesn't boot (kernel panic)
2.6.19.7                N               ksoftirqd is up there, but not  

pegging a CPU.
Takes roughly same amount of CPU
as the other
processes, all of which are from
20-40%
2.6.20.21 N
2.6.21.7 N sort of lopsided load, but no
load from
ksoftirqd -- strange

2.6.22.19               Y               First broken kernel
2.6.23.x                -
2.6.24-19               Y               (from Hardy)
2.6.25.x                -
2.6.26.x                -
2.6.27.x                Y               (from Intrepid)
2.6.28.1                Y
2.6.29-rc               Y
Quoted 5 lines

Correct me if I'm wrong, from what we've seen, it looks like its pointing to some inefficiency in the softirq handling. The question is whether it's something in the driver or the kernel. If we can isolate that, maybe we can take some action to have it fixed.

I don't think its sofirq ineffeciencies (oprofile would have shown that). I know I keep harping on this, but I still think irq affininty is your problem. I'd be interested in knowning what your /proc/interrupts file looked like on each of the above kenrels. Perhaps its not that the bnx2 card you have can't handle the setting of MSI interrupt affinities, but rather that something changeed to break irq affinity on this card.

Neil

Quoted 1 line
#28 lore raw
Quoted 85 lines · Neil Horman
Quoted 76 lines · Kenny Chang
Quoted 75 lines · Eric Dumazet
Quoted 39 lines · Wes Chow
Quoted 27 lines · Eric Dumazet
Quoted 16 lines · Wes Chow

(I'm Kenny's colleague, and I've been doing the kernel builds)

First I'd like to note that there were a lot of bnx2 NAPI changes between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss,
whereas loss in 2.6.22 is significant.

Second, some CPU affinity info: if I do like Eric and pin all of the apps onto a single CPU, I see no packet loss. Also, I do *not* see ksoftirqd show up on top at all!

If I pin half the processes on one CPU and the other half on another CPU, one ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss
in that case is significant (25%).

Quoted 9 lines

Now, the strange case: if I pin 3 processes to one CPU and 1 process to another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one
of the apps takes significantly less CPU than the others, and all apps lose the
*exact same number of packets*. In all other situations where we see packet
loss, the actual number lost per application instance appears random.

Quoted 2 lines

You see same number of packet lost because they are lost at NIC level

Understood.

I have a new observation: if I pin processes to just CPUs 0 and 1, I see no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and 3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any other combination appears to produce loss (though I have not tried all 28 combinations, this seems to be the case).

Quoted 5 lines

At first I thought maybe it had to do with processes pinned to the same CPU, but different cores. The machine is a dual quad core, which means that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3 produce packet loss.

a quad core is really a 2 x 2 core

L2 cache is splited on two blocks, one block used by CPU0/1, other by CPU2/3

You are at the limit of the machine with such workload, so as soon as your CPUs have to transfert 64 bytes lines between those two L2 blocks, you loose.

Quoted 11 lines

I've also noticed that it does not matter which of the working pairs I pin to. For example, pinning 5 processes in any combination on either 0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no packet loss.

The failures are also sudden. In all of the working cases mentioned above, I don't see ksoftirqd on top at all. But when I run 6 processes on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of packets.

Quoted 5 lines
Quoted 2 lines

Normaly, softirq runs on same cpu (the one handling hard irq)

What determines which CPU the hard irq occurs on?

Check /proc/irq/{irqnumber}/smp_affinity

If you want IRQ16 only served by CPU0 :

Quoted 8 lines
Quoted 3 lines

echo 1 >/proc/irq/16/smp_affinity

Hi everyone,

-snip-
Correct me if I'm wrong, from what we've seen, it looks like its pointing to some inefficiency in the softirq handling. The question is whether it's something in the driver or the kernel. If we can isolate that, maybe we can take some action to have it fixed.

Quoted 10 lines

I don't think its sofirq ineffeciencies (oprofile would have shown that). I know I keep harping on this, but I still think irq affininty is your problem. I'd be interested in knowning what your /proc/interrupts file looked like on each of the above kenrels. Perhaps its not that the bnx2 card you have can't handle the setting of MSI interrupt affinities, but rather that something changeed to break irq affinity on this card.

Neil

It's been a while since I updated this thread. We've been running through the different suggestions and tabulating their effects, as well as trying out an Intel card. The short story is that setting affinity and MSI works to some extent, and the Intel card doesn't seem to change things significantly. The results don't seem consistent enough for us to be able to point to a smoking gun.

It does look like the 2.6.29-rc4 kernel performs okay with the Intel card, but this is not a real-time build and it's not likely to be in a supported Ubuntu distribution real soon. We've reached the point where we'd like to look for an expert dedicated to work on this problem for a period of time. The final result being some sort of solution to produce a realtime configuration with a reasonably "aged" kernel (.24~.28) that has multicast performance greater than or equal to that of 2.6.15.

If anybody is interested in devoting some compensated time to this issue, we're offering up a bounty: http://www.athenacr.com/bounties/multicast-performance/

For completeness, here's the table of our experiment results:

====================== ================== ========= ========== =============== ============== ============== =================
Kernel flavor IRQ affinity *4x mcasttest* *5x mcasttest* *6x mcasttest* *Mtools2* [4]_ ====================== ================== ========= ========== =============== ============== ============== =================
Intel e1000e

-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
2.6.24.19 rt | any | OK Maybe X 2.6.24.19 rt | CPU0 | OK OK X 2.6.24.19 generic | any | X 2.6.24.19 generic | CPU0 | OK 2.6.29-rc3 vanilla-server | any | X 2.6.29-rc3 vanilla-server | CPU0 | OK 2.6.29-rc4 vanilla-generic | any | X OK 2.6.29-rc4 vanilla-generic | CPU0 | OK
OK OK [5]_ OK
-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
Broadcom BNX2

-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
2.6.24-19 rt | MSI any | OK OK X 2.6.24-19 rt | MSI CPU0 | OK Maybe X 2.6.24-19 rt | APIC any | OK OK X 2.6.24-19 rt | APIC CPU0 | OK Maybe X 2.6.24-19-bnx-latest rt | APIC CPU0 | OK X 2.6.24-19 server | MSI any | X 2.6.24-19 server | MSI CPU0 | OK 2.6.24-19 generic | APIC any | X 2.6.24-19 generic | APIC CPU0 | OK 2.6.27-11 generic | APIC any | X 2.6.27-11 generic | APIC CPU0 | OK 10% drop 2.6.28-8 generic | APIC any | OK X 2.6.28-8 generic | APIC CPU0 | OK OK 0.5% drop 2.6.29-rc3 vanilla-server | MSI any | X 2.6.29-rc3 vanilla-server | MSI CPU0 | X 2.6.29-rc3 vanilla-server | APIC any | OK X 2.6.29-rc3 vanilla-server | APIC CPU0 | OK OK 2.6.29-rc4 vanilla-generic | APIC any | X 2.6.29-rc4 vanilla-generic | APIC CPU0 | OK 3% drop 10% drop X ====================== ==================+=========+==========+===============+==============+==============+=================
* [4] MTools2 is a test from 29West: http://www.29west.com/docs/TestNet/
* [5] In 5 trials, 1 of the trials dropped 2%, 4 of the trials dropped nothing.

Kenny

#29 lore raw
Quoted 96 lines · Kenny Chang

It's been a while since I updated this thread. We've been running through the different suggestions and tabulating their effects, as well as trying out an Intel card. The short story is that setting affinity and MSI works to some extent, and the Intel card doesn't seem to change things significantly. The results don't seem consistent enough for us to be able to point to a smoking gun.

It does look like the 2.6.29-rc4 kernel performs okay with the Intel card, but this is not a real-time build and it's not likely to be in a supported Ubuntu distribution real soon. We've reached the point where we'd like to look for an expert dedicated to work on this problem for a period of time. The final result being some sort of solution to produce a realtime configuration with a reasonably "aged" kernel (.24~.28) that has multicast performance greater than or equal to that of 2.6.15.

If anybody is interested in devoting some compensated time to this issue, we're offering up a bounty:
http://www.athenacr.com/bounties/multicast-performance/

For completeness, here's the table of our experiment results:

====================== ================== ========= ========== =============== ============== ============== =================
Kernel flavor IRQ affinity *4x
mcasttest* *5x mcasttest* *6x mcasttest* *Mtools2* [4]_ ====================== ================== ========= ========== =============== ============== ============== ================= Intel
e1000e

-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------

2.6.24.19 rt | any |
OK Maybe X
2.6.24.19 rt | CPU0 |
OK OK X
2.6.24.19 generic | any |
X
2.6.24.19 generic | CPU0 |
OK
2.6.29-rc3 vanilla-server | any |
X
2.6.29-rc3 vanilla-server | CPU0 |
OK
2.6.29-rc4 vanilla-generic | any |
X OK
2.6.29-rc4 vanilla-generic | CPU0 | OK
OK OK [5]_ OK
-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------

Broadcom
BNX2

-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------

2.6.24-19 rt | MSI any |
OK OK X
2.6.24-19 rt | MSI CPU0 |
OK Maybe X
2.6.24-19 rt | APIC any |
OK OK X
2.6.24-19 rt | APIC CPU0 |
OK Maybe X
2.6.24-19-bnx-latest rt | APIC CPU0 |
OK X
2.6.24-19 server | MSI any |
X
2.6.24-19 server | MSI CPU0 |
OK
2.6.24-19 generic | APIC any |
X
2.6.24-19 generic | APIC CPU0 |
OK
2.6.27-11 generic | APIC any |
X
2.6.27-11 generic | APIC CPU0 |
OK 10% drop
2.6.28-8 generic | APIC any |
OK X
2.6.28-8 generic | APIC CPU0 |
OK OK 0.5% drop
2.6.29-rc3 vanilla-server | MSI any |
X
2.6.29-rc3 vanilla-server | MSI CPU0 |
X
2.6.29-rc3 vanilla-server | APIC any |
OK X
2.6.29-rc3 vanilla-server | APIC CPU0 |
OK OK
2.6.29-rc4 vanilla-generic | APIC any |
X
2.6.29-rc4 vanilla-generic | APIC CPU0 |
OK 3% drop 10% drop X
======================
==================+=========+==========+===============+==============+==============+=================

Quoted 6 lines

* [4] MTools2 is a test from 29West: http://www.29west.com/docs/TestNet/
* [5] In 5 trials, 1 of the trials dropped 2%, 4 of the trials dropped nothing.

Kenny

Hi Kenny

I am investigating how to reduce contention (and schedule() calls) on this workload.

Following patch already gave me less packet drops (but not yet *perfect*) (10% packet loss instead of 30%, if 8 receivers on my 8 cpus machine)

David, this is a preliminary work, not meant for inclusion as is, comments are welcome.

Thank you

[PATCH] net: sk_forward_alloc becomes an atomic_t

Commit 95766fff6b9a78d11fc2d3812dd035381690b55d
(UDP: Add memory accounting) introduced a regression for high rate UDP flows, because of extra lock_sock() in udp_recvmsg()

In order to reduce need for lock_sock() in UDP receive path, we might need to declare sk_forward_alloc as an atomic_t.

udp_recvmsg() can avoid a lock_sock()/release_sock() pair.

---
 include/net/sock.h   |   14 +++++++-------
 net/core/sock.c      |   31 +++++++++++++++++++------------
 net/core/stream.c    |    2 +-
 net/ipv4/af_inet.c   |    2 +-
 net/ipv4/inet_diag.c |    2 +-
 net/ipv4/tcp_input.c |    2 +-
 net/ipv4/udp.c       |    2 --
 net/ipv6/udp.c       |    2 --
 net/sched/em_meta.c  |    2 +-
 9 files changed, 31 insertions(+), 28 deletions(-)
include/net/sock.h +7 -7
index 4bb1ff9..c4befb9 100644
@@ -250,7 +250,7 @@ struct sock {
 	struct sk_buff_head	sk_async_wait_queue;
 #endif
 	int			sk_wmem_queued;
-	int			sk_forward_alloc;
+	atomic_t		sk_forward_alloc;
 	gfp_t			sk_allocation;
 	int			sk_route_caps;
 	int			sk_gso_type;
@@ -823,7 +823,7 @@ static inline int sk_wmem_schedule(struct sock *sk, int size)
 {
 	if (!sk_has_account(sk))
 		return 1;
-	return size <= sk->sk_forward_alloc ||
+	return size <= atomic_read(&sk->sk_forward_alloc) ||
 		__sk_mem_schedule(sk, size, SK_MEM_SEND);
 }
 
@@ -831,7 +831,7 @@ static inline int sk_rmem_schedule(struct sock *sk, int size)
 {
 	if (!sk_has_account(sk))
 		return 1;
-	return size <= sk->sk_forward_alloc ||
+	return size <= atomic_read(&sk->sk_forward_alloc) ||
 		__sk_mem_schedule(sk, size, SK_MEM_RECV);
 }
 
@@ -839,7 +839,7 @@ static inline void sk_mem_reclaim(struct sock *sk)
 {
 	if (!sk_has_account(sk))
 		return;
-	if (sk->sk_forward_alloc >= SK_MEM_QUANTUM)
+	if (atomic_read(&sk->sk_forward_alloc) >= SK_MEM_QUANTUM)
 		__sk_mem_reclaim(sk);
 }
 
@@ -847,7 +847,7 @@ static inline void sk_mem_reclaim_partial(struct sock *sk)
 {
 	if (!sk_has_account(sk))
 		return;
-	if (sk->sk_forward_alloc > SK_MEM_QUANTUM)
+	if (atomic_read(&sk->sk_forward_alloc) > SK_MEM_QUANTUM)
 		__sk_mem_reclaim(sk);
 }
 
@@ -855,14 +855,14 @@ static inline void sk_mem_charge(struct sock *sk, int size)
 {
 	if (!sk_has_account(sk))
 		return;
-	sk->sk_forward_alloc -= size;
+	atomic_sub(size, &sk->sk_forward_alloc);
 }
 
 static inline void sk_mem_uncharge(struct sock *sk, int size)
 {
 	if (!sk_has_account(sk))
 		return;
-	sk->sk_forward_alloc += size;
+	atomic_add(size, &sk->sk_forward_alloc);
 }
 
 static inline void sk_wmem_free_skb(struct sock *sk, struct sk_buff *skb)
net/core/sock.c +19 -12
index 0620046..8489105 100644
@@ -1081,7 +1081,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority)
 
 		newsk->sk_dst_cache	= NULL;
 		newsk->sk_wmem_queued	= 0;
-		newsk->sk_forward_alloc = 0;
+		atomic_set(&newsk->sk_forward_alloc, 0);
 		newsk->sk_send_head	= NULL;
 		newsk->sk_userlocks	= sk->sk_userlocks & ~SOCK_BINDPORT_LOCK;
 
@@ -1479,7 +1479,7 @@ int __sk_mem_schedule(struct sock *sk, int size, int kind)
 	int amt = sk_mem_pages(size);
 	int allocated;
 
-	sk->sk_forward_alloc += amt * SK_MEM_QUANTUM;
+	atomic_add(amt * SK_MEM_QUANTUM, &sk->sk_forward_alloc);
 	allocated = atomic_add_return(amt, prot->memory_allocated);
 
 	/* Under limit. */
@@ -1520,7 +1520,7 @@ int __sk_mem_schedule(struct sock *sk, int size, int kind)
 		if (prot->sysctl_mem[2] > alloc *
 		    sk_mem_pages(sk->sk_wmem_queued +
 				 atomic_read(&sk->sk_rmem_alloc) +
-				 sk->sk_forward_alloc))
+				 atomic_read(&sk->sk_forward_alloc)))
 			return 1;
 	}
 
@@ -1537,7 +1537,7 @@ suppress_allocation:
 	}
 
 	/* Alas. Undo changes. */
-	sk->sk_forward_alloc -= amt * SK_MEM_QUANTUM;
+	atomic_sub(amt * SK_MEM_QUANTUM, &sk->sk_forward_alloc);
 	atomic_sub(amt, prot->memory_allocated);
 	return 0;
 }
@@ -1551,14 +1551,21 @@ EXPORT_SYMBOL(__sk_mem_schedule);
 void __sk_mem_reclaim(struct sock *sk)
 {
 	struct proto *prot = sk->sk_prot;
-
-	atomic_sub(sk->sk_forward_alloc >> SK_MEM_QUANTUM_SHIFT,
-		   prot->memory_allocated);
-	sk->sk_forward_alloc &= SK_MEM_QUANTUM - 1;
-
-	if (prot->memory_pressure && *prot->memory_pressure &&
-	    (atomic_read(prot->memory_allocated) < prot->sysctl_mem[0]))
-		*prot->memory_pressure = 0;
+	int val = atomic_read(&sk->sk_forward_alloc);
+
+begin:
+	val = atomic_read(&sk->sk_forward_alloc);
+	if (val >= SK_MEM_QUANTUM) {
+		if (atomic_cmpxchg(&sk->sk_forward_alloc, val,
+				   val & (SK_MEM_QUANTUM - 1)) != val)
+			goto begin;
+		atomic_sub(val >> SK_MEM_QUANTUM_SHIFT,
+			   prot->memory_allocated);
+
+		if (prot->memory_pressure && *prot->memory_pressure &&
+		    (atomic_read(prot->memory_allocated) < prot->sysctl_mem[0]))
+			*prot->memory_pressure = 0;
+	}
 }
 
 EXPORT_SYMBOL(__sk_mem_reclaim);
net/core/stream.c +1 -1
index 8727cea..4d04d28 100644
@@ -198,7 +198,7 @@ void sk_stream_kill_queues(struct sock *sk)
 	sk_mem_reclaim(sk);
 
 	WARN_ON(sk->sk_wmem_queued);
-	WARN_ON(sk->sk_forward_alloc);
+	WARN_ON(atomic_read(&sk->sk_forward_alloc));
 
 	/* It is _impossible_ for the backlog to contain anything
 	 * when we get here.  All user references to this socket
net/ipv4/af_inet.c +1 -1
index 627be4d..7a1475c 100644
@@ -152,7 +152,7 @@ void inet_sock_destruct(struct sock *sk)
 	WARN_ON(atomic_read(&sk->sk_rmem_alloc));
 	WARN_ON(atomic_read(&sk->sk_wmem_alloc));
 	WARN_ON(sk->sk_wmem_queued);
-	WARN_ON(sk->sk_forward_alloc);
+	WARN_ON(atomic_read(&sk->sk_forward_alloc));
 
 	kfree(inet->opt);
 	dst_release(sk->sk_dst_cache);
net/ipv4/inet_diag.c +1 -1
index 588a779..903ad66 100644
@@ -158,7 +158,7 @@ static int inet_csk_diag_fill(struct sock *sk,
 	if (minfo) {
 		minfo->idiag_rmem = atomic_read(&sk->sk_rmem_alloc);
 		minfo->idiag_wmem = sk->sk_wmem_queued;
-		minfo->idiag_fmem = sk->sk_forward_alloc;
+		minfo->idiag_fmem = atomic_read(&sk->sk_forward_alloc);
 		minfo->idiag_tmem = atomic_read(&sk->sk_wmem_alloc);
 	}
 
net/ipv4/tcp_input.c +1 -1
index a6961d7..5e08f37 100644
@@ -5258,7 +5258,7 @@ int tcp_rcv_established(struct sock *sk, struct sk_buff *skb,
 
 				tcp_rcv_rtt_measure_ts(sk, skb);
 
-				if ((int)skb->truesize > sk->sk_forward_alloc)
+				if ((int)skb->truesize > atomic_read(&sk->sk_forward_alloc))
 					goto step5;
 
 				NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPHPHITS);
net/ipv4/udp.c +0 -2
index 4bd178a..dcc246a 100644
@@ -955,9 +955,7 @@ try_again:
 		err = ulen;
 
 out_free:
-	lock_sock(sk);
 	skb_free_datagram(sk, skb);
-	release_sock(sk);
 out:
 	return err;
 
net/ipv6/udp.c +0 -2
index 84b1a29..582b80a 100644
@@ -257,9 +257,7 @@ try_again:
 		err = ulen;
 
 out_free:
-	lock_sock(sk);
 	skb_free_datagram(sk, skb);
-	release_sock(sk);
 out:
 	return err;
 
net/sched/em_meta.c +1 -1
index 72cf86e..94d90b6 100644
@@ -383,7 +383,7 @@ META_COLLECTOR(int_sk_wmem_queued)
 META_COLLECTOR(int_sk_fwd_alloc)
 {
 	SKIP_NONLOCAL(skb);
-	dst->value = skb->sk->sk_forward_alloc;
+	dst->value = atomic_read(&skb->sk->sk_forward_alloc);
 }
 
 META_COLLECTOR(int_sk_sndbuf)
#30 lore raw
Quoted 101 lines · Eric Dumazet
Quoted 100 lines · Kenny Chang

It's been a while since I updated this thread. We've been running through the different suggestions and tabulating their effects, as well as trying out an Intel card. The short story is that setting affinity and MSI works to some extent, and the Intel card doesn't seem to change things significantly. The results don't seem consistent enough for us to be able to point to a smoking gun.

It does look like the 2.6.29-rc4 kernel performs okay with the Intel card, but this is not a real-time build and it's not likely to be in a supported Ubuntu distribution real soon. We've reached the point where we'd like to look for an expert dedicated to work on this problem for a period of time. The final result being some sort of solution to produce a realtime configuration with a reasonably "aged" kernel (.24~.28) that has multicast performance greater than or equal to that of 2.6.15.

If anybody is interested in devoting some compensated time to this issue, we're offering up a bounty:
http://www.athenacr.com/bounties/multicast-performance/

For completeness, here's the table of our experiment results:

====================== ================== ========= ========== =============== ============== ============== =================
Kernel flavor IRQ affinity *4x
mcasttest* *5x mcasttest* *6x mcasttest* *Mtools2* [4]_ ====================== ================== ========= ========== =============== ============== ============== ================= Intel
e1000e

-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------

2.6.24.19 rt | any |
OK Maybe X
2.6.24.19 rt | CPU0 |
OK OK X
2.6.24.19 generic | any |
X
2.6.24.19 generic | CPU0 |
OK
2.6.29-rc3 vanilla-server | any |
X
2.6.29-rc3 vanilla-server | CPU0 |
OK
2.6.29-rc4 vanilla-generic | any |
X OK
2.6.29-rc4 vanilla-generic | CPU0 | OK
OK OK [5]_ OK
-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------

Broadcom
BNX2

-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------

2.6.24-19 rt | MSI any |
OK OK X
2.6.24-19 rt | MSI CPU0 |
OK Maybe X
2.6.24-19 rt | APIC any |
OK OK X
2.6.24-19 rt | APIC CPU0 |
OK Maybe X
2.6.24-19-bnx-latest rt | APIC CPU0 |
OK X
2.6.24-19 server | MSI any |
X
2.6.24-19 server | MSI CPU0 |
OK
2.6.24-19 generic | APIC any |
X
2.6.24-19 generic | APIC CPU0 |
OK
2.6.27-11 generic | APIC any |
X
2.6.27-11 generic | APIC CPU0 |
OK 10% drop
2.6.28-8 generic | APIC any |
OK X
2.6.28-8 generic | APIC CPU0 |
OK OK 0.5% drop
2.6.29-rc3 vanilla-server | MSI any |
X
2.6.29-rc3 vanilla-server | MSI CPU0 |
X
2.6.29-rc3 vanilla-server | APIC any |
OK X
2.6.29-rc3 vanilla-server | APIC CPU0 |
OK OK
2.6.29-rc4 vanilla-generic | APIC any |
X
2.6.29-rc4 vanilla-generic | APIC CPU0 |
OK 3% drop 10% drop X
======================
==================+=========+==========+===============+==============+==============+=================

* [4] MTools2 is a test from 29West: http://www.29west.com/docs/TestNet/
* [5] In 5 trials, 1 of the trials dropped 2%, 4 of the trials dropped nothing.

Quoted 7 lines
Quoted 2 lines

Kenny

Hi Kenny

I am investigating how to reduce contention (and schedule() calls) on this workload.

I bound NIC (gigabit BNX2) irq to cpu 0, so that oprofile results on this cpu can show us where ksoftirqd is spending its time.

We can see scheduler at work :)

Also, one thing to note is __copy_skb_header() : 9.49 % of cpu0 time. The problem comes from dst_clone() (6.05 % total, so 2/3 of __copy_skb_header()), touching a highly contended cache line. (other cpus are doing the decrement of dst refcounter)

CPU: Core 2, speed 3000.05 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000 Samples on CPU 0
(samples for other cpus 1..7 omitted)

samples  cum. samples  %        cum. %     symbol name
23750    23750          9.8159   9.8159    try_to_wake_up
22972    46722          9.4944  19.3103    __copy_skb_header
20217    66939          8.3557  27.6660    enqueue_task_fair
14565    81504          6.0197  33.6857    sock_def_readable
13454    94958          5.5606  39.2463    task_rq_lock
13381    108339         5.5304  44.7767    resched_task
13090    121429         5.4101  50.1868    udp_queue_rcv_skb
11441    132870         4.7286  54.9154    skb_queue_tail
10109    142979         4.1781  59.0935    sock_queue_rcv_skb
10024    153003         4.1429  63.2364    __wake_up_sync
9952     162955         4.1132  67.3496    update_curr
8761     171716         3.6209  70.9705    sched_clock_cpu
7414     179130         3.0642  74.0347    rb_insert_color
7381     186511         3.0506  77.0853    select_task_rq_fair
6749     193260         2.7894  79.8747    __slab_alloc
5881     199141         2.4306  82.3053    __wake_up_common
5432     204573         2.2451  84.5504    __skb_clone
4306     208879         1.7797  86.3300    kmem_cache_alloc
3524     212403         1.4565  87.7865    place_entity
2783     215186         1.1502  88.9367    skb_clone
2576     217762         1.0647  90.0014    __udp4_lib_rcv
2430     220192         1.0043  91.0057    bnx2_poll_work
2184     222376         0.9027  91.9084    ipt_do_table
2090     224466         0.8638  92.7722    ip_route_input
1877     226343         0.7758  93.5479    __alloc_skb
1495     227838         0.6179  94.1658    native_sched_clock
1166     229004         0.4819  94.6477    __update_sched_clock
1083     230087         0.4476  95.0953    netif_receive_skb
1062     231149         0.4389  95.5343    activate_task
644      231793         0.2662  95.8004    __kmalloc_track_caller
638      232431         0.2637  96.0641    nf_iterate
549      232980         0.2269  96.2910    skb_put
maintainer
#31 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Sat, 28 Feb 2009 09:51:11 +0100

Quoted 9 lines

David, this is a preliminary work, not meant for inclusion as is, comments are welcome.

[PATCH] net: sk_forward_alloc becomes an atomic_t

Commit 95766fff6b9a78d11fc2d3812dd035381690b55d
(UDP: Add memory accounting) introduced a regression for high rate UDP flows, because of extra lock_sock() in udp_recvmsg()

Quoted 6 lines

In order to reduce need for lock_sock() in UDP receive path, we might need to declare sk_forward_alloc as an atomic_t.

udp_recvmsg() can avoid a lock_sock()/release_sock() pair.

This adds new overhead for TCP which has to hold the socket lock for other reasons in these paths.

I don't get how an atomic_t operation is cheaper than a lock_sock/release_sock. Is it the case that in many executions of these paths only atomic_read()'s are necessary?

I actually think this scheme is racy. There is a reason we have to hold the socket lock when doing memory scheduling. Two threads can get in there and say "hey I have enough space already" even though only enough space is allocated for one of their requests.

What did I miss? :)

#32 lore raw
Quoted 26 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Sat, 28 Feb 2009 09:51:11 +0100

Quoted 9 lines

David, this is a preliminary work, not meant for inclusion as is, comments are welcome.

[PATCH] net: sk_forward_alloc becomes an atomic_t

Commit 95766fff6b9a78d11fc2d3812dd035381690b55d
(UDP: Add memory accounting) introduced a regression for high rate UDP flows, because of extra lock_sock() in udp_recvmsg()

Quoted 6 lines

In order to reduce need for lock_sock() in UDP receive path, we might need to declare sk_forward_alloc as an atomic_t.

udp_recvmsg() can avoid a lock_sock()/release_sock() pair.

This adds new overhead for TCP which has to hold the socket lock for other reasons in these paths.

I don't get how an atomic_t operation is cheaper than a lock_sock/release_sock. Is it the case that in many executions of these paths only atomic_read()'s are necessary?

Quoted 8 lines

I actually think this scheme is racy. There is a reason we have to hold the socket lock when doing memory scheduling. Two threads can get in there and say "hey I have enough space already" even though only enough space is allocated for one of their requests.

What did I miss? :)

I believe you are right, and in fact was about to post a "dont look at this patch" since it doesnt help the multicast reception at all, I redone tests more carefuly and got nothing but noise.

We have a cache line ping pong mess here, and need more thinking.

I rewrote Kenny prog to use non blocking sockets.

Receivers are doing :

        int delay = 50;
	fcntl(s, F_SETFL, O_NDELAY);
        while(1)
        {
            struct sockaddr_in from;
            socklen_t fromlen = sizeof(from);
            res = recvfrom(s, buf, 1000, 0, (struct sockaddr*)&from, &fromlen);
            if (res == -1) {
                      delay++;
                      usleep(delay);
                      continue;
            }
            if (delay > 40)
                delay--;
            ++npackets;

With this litle user space change and 8 receivers on my dual quad core, softirqd only takes 8% of one cpu and no drops at all (instead of 100% cpu and 30% drops)

So this is definitly a problem mixing scheduler cache line ping pongs with network stack cache line ping pongs.

We could reorder fields so that fewer cache lines are touched by the softirq processing, I tried this but still got packet drops.

#33 lore raw
Quoted 61 lines · Eric Dumazet
Quoted 25 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Sat, 28 Feb 2009 09:51:11 +0100

Quoted 9 lines

David, this is a preliminary work, not meant for inclusion as is, comments are welcome.

[PATCH] net: sk_forward_alloc becomes an atomic_t

Commit 95766fff6b9a78d11fc2d3812dd035381690b55d
(UDP: Add memory accounting) introduced a regression for high rate UDP flows, because of extra lock_sock() in udp_recvmsg()

Quoted 6 lines

In order to reduce need for lock_sock() in UDP receive path, we might need to declare sk_forward_alloc as an atomic_t.

udp_recvmsg() can avoid a lock_sock()/release_sock() pair.

This adds new overhead for TCP which has to hold the socket lock for other reasons in these paths.

I don't get how an atomic_t operation is cheaper than a lock_sock/release_sock. Is it the case that in many executions of these paths only atomic_read()'s are necessary?

Quoted 8 lines

I actually think this scheme is racy. There is a reason we have to hold the socket lock when doing memory scheduling. Two threads can get in there and say "hey I have enough space already" even though only enough space is allocated for one of their requests.

What did I miss? :)

I believe you are right, and in fact was about to post a "dont look at this patch" since it doesnt help the multicast reception at all, I redone tests more carefuly and got nothing but noise.

We have a cache line ping pong mess here, and need more thinking.

I rewrote Kenny prog to use non blocking sockets.

Receivers are doing :

        int delay = 50;
	fcntl(s, F_SETFL, O_NDELAY);
        while(1)
        {
            struct sockaddr_in from;
            socklen_t fromlen = sizeof(from);
            res = recvfrom(s, buf, 1000, 0, (struct sockaddr*)&from, &fromlen);
            if (res == -1) {
                      delay++;
                      usleep(delay);
                      continue;
            }
            if (delay > 40)
                delay--;
            ++npackets;
Quoted 11 lines

With this litle user space change and 8 receivers on my dual quad core, softirqd only takes 8% of one cpu and no drops at all (instead of 100% cpu and 30% drops)

So this is definitly a problem mixing scheduler cache line ping pongs with network stack cache line ping pongs.

We could reorder fields so that fewer cache lines are touched by the softirq processing, I tried this but still got packet drops.

I have more questions :

What is the maximum latency you can afford on the delivery of the packet(s) ?

Are user apps using real time scheduling ?

I had an idea, that keep cpu handling NIC interrupts only delivering packets to socket queues, and not messing with scheduler : fast queueing, and wakeing up a workqueue (on another cpu) to perform the scheduler work. But that means some extra latency (in the order of 2 or 3 us I guess)

We could enter in this mode automatically, if the NIC rx handler *see* more than N packets are waiting in NIC queue : In case of moderate or light trafic, no extra latency would be necessary. This would mean some changes in NIC driver.

Hum, then, if NIC rx handler is run beside the ksoftirqd, we already know we are in a stress situation, so maybe no driver changes are necessary :
Just test if we run ksoftirqd...

#34 lore raw
Quoted 12 lines · Eric Dumazet

I have more questions :

What is the maximum latency you can afford on the delivery of the packet(s) ?

Are user apps using real time scheduling ?

I had an idea, that keep cpu handling NIC interrupts only delivering packets to socket queues, and not messing with scheduler : fast queueing, and wakeing up a workqueue (on another cpu) to perform the scheduler work. But that means some extra latency (in the order of 2 or 3 us I guess)

Quoted 8 lines

We could enter in this mode automatically, if the NIC rx handler *see* more than N packets are waiting in NIC queue : In case of moderate or light trafic, no extra latency would be necessary. This would mean some changes in NIC driver.

Hum, then, if NIC rx handler is run beside the ksoftirqd, we already know we are in a stress situation, so maybe no driver changes are necessary :
Just test if we run ksoftirqd...

Here is a patch that helps. It's still an RFC of course, since its somewhat ugly :)

I am now able to have 8 receivers on my 8 cpus machine, with one multicast packet every 10 us, without any loss. (standard setup, no affinity games)

oprofile results see that scheduler overhead vanished, we get back to pure network profile :)

(First offender being __copy_skb_header because of the atomic_inc on dst refcount)

CPU: Core 2, speed 3000.43 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  cum. samples  %        cum. %     symbol name
126329   126329        20.4296  20.4296    __copy_skb_header
31395    157724         5.0771  25.5067    udp_queue_rcv_skb
29191    186915         4.7207  30.2274    sock_def_readable
26284    213199         4.2506  34.4780    sock_queue_rcv_skb
26010    239209         4.2063  38.6842    kmem_cache_alloc
20040    259249         3.2408  41.9251    __udp4_lib_rcv
19570    278819         3.1648  45.0899    skb_queue_tail
17799    296618         2.8784  47.9683    bnx2_poll_work
17267    313885         2.7924  50.7606    skb_release_data
14663    328548         2.3713  53.1319    __skb_recv_datagram
14443    342991         2.3357  55.4676    __slab_alloc
13248    356239         2.1424  57.6100    copy_to_user
13138    369377         2.1246  59.7347    __sk_mem_schedule
12004    381381         1.9413  61.6759    __skb_clone
11924    393305         1.9283  63.6042    skb_clone
11077    404382         1.7913  65.3956    lock_sock_nested
10320    414702         1.6689  67.0645    ip_route_input
9622     424324         1.5560  68.6205    dst_release
8344     432668         1.3494  69.9699    __slab_free
8124     440792         1.3138  71.2837    mwait_idle
7066     447858         1.1427  72.4264    udp_recvmsg
6652     454510         1.0757  73.5021    netif_receive_skb
6386     460896         1.0327  74.5349    ipt_do_table
6010     466906         0.9719  75.5068    release_sock
6003     472909         0.9708  76.4776    memcpy_toiovec
5697     478606         0.9213  77.3989    __alloc_skb
5671     484277         0.9171  78.3160    copy_from_user
5031     489308         0.8136  79.1296    sysenter_past_esp
4753     494061         0.7686  79.8982    bnx2_interrupt
4429     498490         0.7162  80.6145    sock_rfree

[PATCH] softirq: Introduce mechanism to defer wakeups

Some network workloads need to call scheduler too many times. For example, each received multicast frame can wakeup many threads. ksoftirqd is then not able to drain NIC RX queues and we get frame losses and high latencies.

This patch adds an infrastructure to delay part of work done in sock_def_readable() at end of do_softirq()

---
 include/linux/interrupt.h |    9 +++++++++
 include/net/sock.h        |    1 +
 kernel/softirq.c          |   29 ++++++++++++++++++++++++++++-
 net/core/sock.c           |   21 +++++++++++++++++++--
 4 files changed, 57 insertions(+), 3 deletions(-)
include/linux/interrupt.h +9 -0
index 9127f6b..62caaae 100644
@@ -295,6 +295,15 @@ extern void send_remote_softirq(struct call_single_data *cp, int cpu, int softir
 extern void __send_remote_softirq(struct call_single_data *cp, int cpu,
 				  int this_cpu, int softirq);
 
+/*
+ * delayed works : should be delayed at do_softirq() end
+ */
+struct softirq_del {
+	struct softirq_del	*next;
+	void 			(*func)(struct softirq_del *);
+};
+int softirq_del(struct softirq_del *sdel, void (*func)(struct softirq_del *));
+
 /* Tasklets --- multithreaded analogue of BHs.
 
    Main feature differing them of generic softirqs: tasklet
include/net/sock.h +1 -0
index eefeeaf..95841de 100644
@@ -260,6 +260,7 @@ struct sock {
 	unsigned long	        sk_lingertime;
 	struct sk_buff_head	sk_error_queue;
 	struct proto		*sk_prot_creator;
+	struct softirq_del	sk_del;
 	rwlock_t		sk_callback_lock;
 	int			sk_err,
 				sk_err_soft;
kernel/softirq.c +28 -1
index bdbe9de..40fe527 100644
@@ -158,6 +158,33 @@ void local_bh_enable_ip(unsigned long ip)
 }
 EXPORT_SYMBOL(local_bh_enable_ip);
 
+
+static DEFINE_PER_CPU(struct softirq_del *, softirq_del_head);
+int softirq_del(struct softirq_del *sdel, void (*func)(struct softirq_del *))
+{
+	if (cmpxchg(&sdel->func, NULL, func) == NULL) {
+		sdel->next = __get_cpu_var(softirq_del_head);
+		__get_cpu_var(softirq_del_head) = sdel;
+		return 1;
+	}
+	return 0;
+}
+
+static void softirqdel_exec(void)
+{
+	struct softirq_del *sdel;
+	void (*func)(struct softirq_del *);
+
+	while ((sdel = __get_cpu_var(softirq_del_head)) != NULL) {
+		__get_cpu_var(softirq_del_head) = sdel->next;
+		func = sdel->func;
+		sdel->func = NULL;
+		(*func)(sdel);
+		}
+}
+
+
+
 /*
  * We restart softirq processing MAX_SOFTIRQ_RESTART times,
  * and we fall back to softirqd after that.
@@ -219,7 +246,7 @@ restart:
 
 	if (pending)
 		wakeup_softirqd();
-
+	softirqdel_exec();
 	trace_softirq_exit();
 
 	account_system_vtime(current);
net/core/sock.c +19 -2
index 5f97caa..f9ee8dd 100644
@@ -1026,6 +1026,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority)
 #endif
 
 		rwlock_init(&newsk->sk_dst_lock);
+		newsk->sk_del.func = NULL;
 		rwlock_init(&newsk->sk_callback_lock);
 		lockdep_set_class_and_name(&newsk->sk_callback_lock,
 				af_callback_keys + newsk->sk_family,
@@ -1634,12 +1635,27 @@ static void sock_def_error_report(struct sock *sk)
 	read_unlock(&sk->sk_callback_lock);
 }
 
+static void sock_readable_defer(struct softirq_del *sdel)
+{
+	struct sock *sk = container_of(sdel, struct sock, sk_del);
+
+	wake_up_interruptible_sync(sk->sk_sleep);
+	read_unlock(&sk->sk_callback_lock);
+}
+
 static void sock_def_readable(struct sock *sk, int len)
 {
 	read_lock(&sk->sk_callback_lock);
-	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep))
-		wake_up_interruptible_sync(sk->sk_sleep);
 	sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
+	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) {
+		if (in_softirq()) {
+			if (!softirq_del(&sk->sk_del, sock_readable_defer))
+				goto unlock;
+			return;
+		}
+		wake_up_interruptible_sync(sk->sk_sleep);
+	}
+unlock:
 	read_unlock(&sk->sk_callback_lock);
 }
 
@@ -1720,6 +1736,7 @@ void sock_init_data(struct socket *sock, struct sock *sk)
 		sk->sk_sleep	=	NULL;
 
 	rwlock_init(&sk->sk_dst_lock);
+	sk->sk_del.func		=	NULL;
 	rwlock_init(&sk->sk_callback_lock);
 	lockdep_set_class_and_name(&sk->sk_callback_lock,
 			af_callback_keys + sk->sk_family,
maintainer
#35 lore raw

From: Eric Dumazet <[email protected]>
Date: Sun, 08 Mar 2009 17:46:13 +0100

net/core/sock.c static void sock_def_error_report(struct sock *sk) · line 1655

Quoted 6 lines
net/core/sock.c +6 -0
+	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) {
+		if (in_softirq()) {
+			if (!softirq_del(&sk->sk_del, sock_readable_defer))
+				goto unlock;
+			return;
+		}

This is interesting.

I think you should make softirq_del() more flexible. Make it the socket's job to make sure it doesn't try to defer different functions, and put the onus on locking there as well.

The cmpxchg() and all of this checking is just wasted work.

I'd really like to get rid of that callback lock too, then we'd really be in business. :-)

#36 lore raw
Quoted 12 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Sun, 08 Mar 2009 17:46:13 +0100

net/core/sock.c static void sock_def_error_report(struct sock *sk) · line 1655

Quoted 6 lines
net/core/sock.c +6 -0
+	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) {
+		if (in_softirq()) {
+			if (!softirq_del(&sk->sk_del, sock_readable_defer))
+				goto unlock;
+			return;
+		}

This is interesting.

Quoted 8 lines

I think you should make softirq_del() more flexible. Make it the socket's job to make sure it doesn't try to defer different functions, and put the onus on locking there as well.

The cmpxchg() and all of this checking is just wasted work.

I'd really like to get rid of that callback lock too, then we'd really be in business. :-)

First thanks for your review David.

I chose cmpxchg() because I needed some form of exclusion here. I first added a spinlock inside "struct softirq_del" then I realize I could use cmpxchg() instead and keep the structure small. As the synchronization is only needed at queueing time, we could pass the address of a spinlock XXX to sofirq_del() call.

Also, when an event was queued for later invocation, I also needed to keep a reference on "struct socket" to make sure it doesnt disappear before the invocation. Not all sockets are RCU guarded (we added RCU only for some protocols (TCP, UDP ...). So I found keeping a read_lock on callback was the easyest thing to do. I now realize we might overflow preempt_count, so special care is needed.

About your first point, maybe we should make sofirq_del() (poor name I confess) only have one argument (pointer to struct softirq_del), and initialize the function pointer at socket init time. That would insure "struct softirq_del" is associated to one callback only. cmpxchg() test would have to be done on "next" field then (or use the spinlock XXX)

I am not sure output path needs such tricks, since threads are rarely blocking on output : We dont trigger 400.000 wakeups per second ?

Another point : I did a tbench test and got 2517 MB/s with the patch, instead of 2538 MB/s (using Linus 2.6 git tree), thats ~ 0.8 % regression for this workload.

maintainer
#37 lore raw

From: Eric Dumazet <[email protected]>
Date: Mon, 09 Mar 2009 07:36:57 +0100

Quoted 5 lines

I chose cmpxchg() because I needed some form of exclusion here. I first added a spinlock inside "struct softirq_del" then I realize I could use cmpxchg() instead and keep the structure small. As the synchronization is only needed at queueing time, we could pass the address of a spinlock XXX to sofirq_del() call.

I don't understand why you need the mutual exclusion in the first place. The function pointer always has the same value. And this locking isn't protecting the list insertion either, as that isn't even necessary.

It just looks like plain overhead to me.

Quoted 6 lines

Also, when an event was queued for later invocation, I also needed to keep a reference on "struct socket" to make sure it doesnt disappear before the invocation. Not all sockets are RCU guarded (we added RCU only for some protocols (TCP, UDP ...). So I found keeping a read_lock on callback was the easyest thing to do. I now realize we might overflow preempt_count, so special care is needed.

You're using this in UDP so... make the rule that you can't use this with a non-RCU-quiescent protocol.

Quoted 6 lines

About your first point, maybe we should make sofirq_del() (poor name I confess) only have one argument (pointer to struct softirq_del), and initialize the function pointer at socket init time. That would insure "struct softirq_del" is associated to one callback only. cmpxchg() test would have to be done on "next" field then (or use the spinlock XXX)

Why? You run this from softirq safe context, nothing can run other softirqs on this cpu and corrupt the list, therefore.

#38 lore raw
Quoted 9 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Mon, 09 Mar 2009 07:36:57 +0100

Quoted 5 lines

I chose cmpxchg() because I needed some form of exclusion here. I first added a spinlock inside "struct softirq_del" then I realize I could use cmpxchg() instead and keep the structure small. As the synchronization is only needed at queueing time, we could pass the address of a spinlock XXX to sofirq_del() call.

Quoted 6 lines

I don't understand why you need the mutual exclusion in the first place. The function pointer always has the same value. And this locking isn't protecting the list insertion either, as that isn't even necessary.

It just looks like plain overhead to me.

I was lazy to check all callers (all protocols) had a lock on sock, and prefered safety.

I was fooled by the read_lock(), and though several cpus could call this function in //

Quoted 10 lines
Quoted 7 lines

Also, when an event was queued for later invocation, I also needed to keep a reference on "struct socket" to make sure it doesnt disappear before the invocation. Not all sockets are RCU guarded (we added RCU only for some protocols (TCP, UDP ...). So I found keeping a read_lock on callback was the easyest thing to do. I now realize we might overflow preempt_count, so special care is needed.

You're using this in UDP so... make the rule that you can't use this with a non-RCU-quiescent protocol.

UDP/TCP only ? I though many other protocols (not all using RCU) were using sock_def_readable() too...

maintainer
#39 lore raw

From: Eric Dumazet <[email protected]>
Date: Fri, 13 Mar 2009 23:30:31 +0100

Quoted 8 lines
Quoted 7 lines · David Miller
Quoted 6 lines

Also, when an event was queued for later invocation, I also needed to keep a reference on "struct socket" to make sure it doesnt disappear before the invocation. Not all sockets are RCU guarded (we added RCU only for some protocols (TCP, UDP ...). So I found keeping a read_lock on callback was the easyest thing to do. I now realize we might overflow preempt_count, so special care is needed.

Quoted 5 lines
Quoted 2 lines

You're using this in UDP so... make the rule that you can't use this with a non-RCU-quiescent protocol.

UDP/TCP only ? I though many other protocols (not all using RCU) were using sock_def_readable() too...

Maybe create a inet_def_readable() just for this purpose :-)

#40 lore raw
Quoted 10 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Fri, 13 Mar 2009 23:30:31 +0100

Quoted 7 lines
Quoted 6 lines · David Miller
Quoted 6 lines

Also, when an event was queued for later invocation, I also needed to keep a reference on "struct socket" to make sure it doesnt disappear before the invocation. Not all sockets are RCU guarded (we added RCU only for some protocols (TCP, UDP ...). So I found keeping a read_lock on callback was the easyest thing to do. I now realize we might overflow preempt_count, so special care is needed.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

You're using this in UDP so... make the rule that you can't use this with a non-RCU-quiescent protocol.

UDP/TCP only ? I though many other protocols (not all using RCU) were using sock_def_readable() too...

Maybe create a inet_def_readable() just for this purpose :-)

I must be tired, I should had this idea before you :)

I post a new patch after some rest, I definitly should not be still awake !

#41 lore raw

PATCH 1/1 net: reorder fields of struct socket

On x86_64, its rather unfortunate that "wait_queue_head_t wait" field of "struct socket" spans two cache lines (assuming a 64 bytes cache line in current cpus)

offsetof(struct socket, wait)=0x30
sizeof(wait_queue_head_t)=0x18

This might explain why Kenny Chang noticed that his multicast workload was performing bad with 64 bit kernels, since more cache lines ping pongs were involved.

This litle patch moves "wait" field next "fasync_list" so that both fields share a single cache line, to speedup sock_def_readable()

include/linux/net.h +6 -2
index 4515efa..4fc2ffd 100644
@@ -129,11 +129,15 @@ struct socket {
 	socket_state		state;
 	short			type;
 	unsigned long		flags;
-	const struct proto_ops	*ops;
+	/*
+	 * Please keep fasync_list & wait fields in the same cache line
+	 */
 	struct fasync_struct	*fasync_list;
+	wait_queue_head_t	wait;
+
 	struct file		*file;
 	struct sock		*sk;
-	wait_queue_head_t	wait;
+	const struct proto_ops	*ops;
 };
 
 struct vm_area_struct;
maintainer Applied
#42 lore raw

Re: [PATCH] net: reorder fields of struct socket

From: Eric Dumazet <[email protected]>
Date: Sat, 14 Mar 2009 10:03:54 +0100

Quoted 7 lines

On x86_64, its rather unfortunate that "wait_queue_head_t wait" field of "struct socket" spans two cache lines (assuming a 64 bytes cache line in current cpus)

offsetof(struct socket, wait)=0x30
sizeof(wait_queue_head_t)=0x18

Quoted 8 lines

This might explain why Kenny Chang noticed that his multicast workload was performing bad with 64 bit kernels, since more cache lines ping pongs were involved.

This litle patch moves "wait" field next "fasync_list" so that both fields share a single cache line, to speedup sock_def_readable()

Applied, thanks a lot Eric.

#43 lore raw
In reply to David Miller
Quoted 10 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Fri, 13 Mar 2009 23:30:31 +0100

Quoted 7 lines
Quoted 6 lines · David Miller
Quoted 6 lines

Also, when an event was queued for later invocation, I also needed to keep a reference on "struct socket" to make sure it doesnt disappear before the invocation. Not all sockets are RCU guarded (we added RCU only for some protocols (TCP, UDP ...). So I found keeping a read_lock on callback was the easyest thing to do. I now realize we might overflow preempt_count, so special care is needed.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

You're using this in UDP so... make the rule that you can't use this with a non-RCU-quiescent protocol.

UDP/TCP only ? I though many other protocols (not all using RCU) were using sock_def_readable() too...

Maybe create a inet_def_readable() just for this purpose :-)

Here is the last incantation of the patch, that of course should be split in two parts and better Changelog for further discussion on lkml.

We need to take a reference on sock when queued on a softirq delay list. RCU wont help here because of SLAB_DESTROY_BY_RCU thing :
Another cpu could free/reuse the socket before we have a chance to call softirq_delay_exec()

UDP & UDPLite use this delayed wakeup feature.

Thank you

[PATCH] softirq: Introduce mechanism to defer wakeups

Some network workloads need to call scheduler too many times. For example, each received multicast frame can wakeup many threads. ksoftirqd is then not able to drain NIC RX queues in time and we get frame losses and high latencies.

This patch adds an infrastructure to delay work done in sock_def_readable() at end of do_softirq(). This needs to make available current->softirq_context even if !CONFIG_TRACE_IRQFLAGS

---
 include/linux/interrupt.h |   18 +++++++++++++++
 include/linux/irqflags.h  |   11 ++++-----
 include/linux/sched.h     |    2 -
 include/net/sock.h        |    2 +
 include/net/udplite.h     |    1
 kernel/lockdep.c          |    2 -
 kernel/softirq.c          |   42 ++++++++++++++++++++++++++++++++++--
 lib/locking-selftest.c    |    4 +--
 net/core/sock.c           |   41 +++++++++++++++++++++++++++++++++++
 net/ipv4/udp.c            |    7 ++++++
 net/ipv6/udp.c            |    7 ++++++
 11 files changed, 125 insertions(+), 12 deletions(-)
include/linux/interrupt.h +18 -0
index 9127f6b..a773d0c 100644
@@ -295,6 +295,24 @@ extern void send_remote_softirq(struct call_single_data *cp, int cpu, int softir
 extern void __send_remote_softirq(struct call_single_data *cp, int cpu,
 				  int this_cpu, int softirq);
 
+/*
+ * softirq delayed works : should be delayed at do_softirq() end
+ */
+struct softirq_delay {
+	struct softirq_delay	*next;
+	void 			(*func)(struct softirq_delay *);
+};
+
+int softirq_delay_queue(struct softirq_delay *sdel);
+
+static inline void softirq_delay_init(struct softirq_delay *sdel,
+				      void (*func)(struct softirq_delay *))
+{
+	sdel->next = NULL;
+	sdel->func = func;
+}
+
+
 /* Tasklets --- multithreaded analogue of BHs.
 
    Main feature differing them of generic softirqs: tasklet
include/linux/irqflags.h +5 -6
index 74bde13..30c1e01 100644
@@ -13,19 +13,21 @@
 
 #include <linux/typecheck.h>
 
+#define softirq_enter()	do { current->softirq_context++; } while (0)
+#define softirq_exit()	do { current->softirq_context--; } while (0)
+#define softirq_context(p)	((p)->softirq_context)
+#define running_from_softirq()  (softirq_context(current) > 0)
+
 #ifdef CONFIG_TRACE_IRQFLAGS
   extern void trace_softirqs_on(unsigned long ip);
   extern void trace_softirqs_off(unsigned long ip);
   extern void trace_hardirqs_on(void);
   extern void trace_hardirqs_off(void);
 # define trace_hardirq_context(p)	((p)->hardirq_context)
-# define trace_softirq_context(p)	((p)->softirq_context)
 # define trace_hardirqs_enabled(p)	((p)->hardirqs_enabled)
 # define trace_softirqs_enabled(p)	((p)->softirqs_enabled)
 # define trace_hardirq_enter()	do { current->hardirq_context++; } while (0)
 # define trace_hardirq_exit()	do { current->hardirq_context--; } while (0)
-# define trace_softirq_enter()	do { current->softirq_context++; } while (0)
-# define trace_softirq_exit()	do { current->softirq_context--; } while (0)
 # define INIT_TRACE_IRQFLAGS	.softirqs_enabled = 1,
 #else
 # define trace_hardirqs_on()		do { } while (0)
@@ -33,13 +35,10 @@
 # define trace_softirqs_on(ip)		do { } while (0)
 # define trace_softirqs_off(ip)		do { } while (0)
 # define trace_hardirq_context(p)	0
-# define trace_softirq_context(p)	0
 # define trace_hardirqs_enabled(p)	0
 # define trace_softirqs_enabled(p)	0
 # define trace_hardirq_enter()		do { } while (0)
 # define trace_hardirq_exit()		do { } while (0)
-# define trace_softirq_enter()		do { } while (0)
-# define trace_softirq_exit()		do { } while (0)
 # define INIT_TRACE_IRQFLAGS
 #endif
 
include/linux/sched.h +1 -1
index 8c216e0..5dd8487 100644
@@ -1320,8 +1320,8 @@ struct task_struct {
 	unsigned long softirq_enable_ip;
 	unsigned int softirq_enable_event;
 	int hardirq_context;
-	int softirq_context;
 #endif
+	int softirq_context;
 #ifdef CONFIG_LOCKDEP
 # define MAX_LOCK_DEPTH 48UL
 	u64 curr_chain_key;
include/net/sock.h +2 -0
index 4bb1ff9..0160a83 100644
@@ -260,6 +260,7 @@ struct sock {
 	unsigned long	        sk_lingertime;
 	struct sk_buff_head	sk_error_queue;
 	struct proto		*sk_prot_creator;
+	struct softirq_delay	sk_delay;
 	rwlock_t		sk_callback_lock;
 	int			sk_err,
 				sk_err_soft;
@@ -960,6 +961,7 @@ extern void *sock_kmalloc(struct sock *sk, int size,
 			  gfp_t priority);
 extern void sock_kfree_s(struct sock *sk, void *mem, int size);
 extern void sk_send_sigurg(struct sock *sk);
+extern void inet_def_readable(struct sock *sk, int len);
 
 /*
  * Functions to fill in entries in struct proto_ops when a protocol
include/net/udplite.h +1 -0
index afdffe6..7ce0ee0 100644
@@ -25,6 +25,7 @@ static __inline__ int udplite_getfrag(void *from, char *to, int  offset,
 /* Designate sk as UDP-Lite socket */
 static inline int udplite_sk_init(struct sock *sk)
 {
+	sk->sk_data_ready = inet_def_readable;
 	udp_sk(sk)->pcflag = UDPLITE_BIT;
 	return 0;
 }
kernel/lockdep.c +1 -1
index 06b0c35..9873b40 100644
@@ -1807,7 +1807,7 @@ print_usage_bug(struct task_struct *curr, struct held_lock *this,
 	printk("%s/%d [HC%u[%lu]:SC%u[%lu]:HE%u:SE%u] takes:\n",
 		curr->comm, task_pid_nr(curr),
 		trace_hardirq_context(curr), hardirq_count() >> HARDIRQ_SHIFT,
-		trace_softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
+		softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
 		trace_hardirqs_enabled(curr),
 		trace_softirqs_enabled(curr));
 	print_lock(this);
kernel/softirq.c +40 -2

66 lines not on this page · raw message

lib/locking-selftest.c +2 -2
index 280332c..1aa7351 100644
@@ -157,11 +157,11 @@ static void init_shared_classes(void)
 #define SOFTIRQ_ENTER()				\
 		local_bh_disable();		\
 		local_irq_disable();		\
-		trace_softirq_enter();		\
+		softirq_enter();		\
 		WARN_ON(!in_softirq());
 
 #define SOFTIRQ_EXIT()				\
-		trace_softirq_exit();		\
+		softirq_exit();		\
 		local_irq_enable();		\
 		local_bh_enable();
 
net/core/sock.c +41 -0

65 lines not on this page · raw message

net/ipv4/udp.c +7 -0
index 05b7abb..1cc0907 100644
@@ -1342,6 +1342,12 @@ void udp_destroy_sock(struct sock *sk)
 	release_sock(sk);
 }
 
+static int udp_init_sock(struct sock *sk)
+{
+	sk->sk_data_ready = inet_def_readable;
+	return 0;
+}
+
 /*
  *	Socket option code for UDP
  */
@@ -1559,6 +1565,7 @@ struct proto udp_prot = {
 	.connect	   = ip4_datagram_connect,
 	.disconnect	   = udp_disconnect,
 	.ioctl		   = udp_ioctl,
+	.init		   = udp_init_sock,
 	.destroy	   = udp_destroy_sock,
 	.setsockopt	   = udp_setsockopt,
 	.getsockopt	   = udp_getsockopt,
net/ipv6/udp.c +7 -0
index 84b1a29..1a9f8d4 100644
@@ -960,6 +960,12 @@ void udpv6_destroy_sock(struct sock *sk)
 	inet6_destroy_sock(sk);
 }
 
+static int udpv6_init_sock(struct sock *sk)
+{
+	sk->sk_data_ready = inet_def_readable;
+	return 0;
+}
+
 /*
  *	Socket option code for UDP
  */
@@ -1084,6 +1090,7 @@ struct proto udpv6_prot = {
 	.connect	   = ip6_datagram_connect,
 	.disconnect	   = udp_disconnect,
 	.ioctl		   = udp_ioctl,
+	.init 		   = udpv6_init_sock,
 	.destroy	   = udpv6_destroy_sock,
 	.setsockopt	   = udpv6_setsockopt,
 	.getsockopt	   = udpv6_getsockopt,
maintainer
#44 lore raw
Quoted 2 lines · Eric Dumazet

Here is the last incantation of the patch, that of course should be split in two parts and better Changelog for further discussion on lkml.

I read the entire thread up to now, and I still don't really understand the Changelog, sorry :(

Quoted 10 lines

[PATCH] softirq: Introduce mechanism to defer wakeups

Some network workloads need to call scheduler too many times. For example, each received multicast frame can wakeup many threads. ksoftirqd is then not able to drain NIC RX queues in time and we get frame losses and high latencies.

This patch adds an infrastructure to delay work done in sock_def_readable() at end of do_softirq(). This needs to make available current->softirq_context even if !CONFIG_TRACE_IRQFLAGS

How does that solve the wakeup issue?

Quoted 6 lines

---

kernel/softirq.c void local_bh_enable_ip(unsigned long ip) · line 165

Quoted 8 lines
kernel/softirq.c +5 -0
@@ -158,6 +158,42 @@ void local_bh_enable_ip(unsigned long ip)
 }
 EXPORT_SYMBOL(local_bh_enable_ip);
 
+
+#define SOFTIRQ_DELAY_END (struct softirq_delay *)1L
+static DEFINE_PER_CPU(struct softirq_delay *, softirq_delay_head) = {
+	SOFTIRQ_DELAY_END
+};

Why the magic termination value? Can't we NULL terminate the list

kernel/softirq.c void local_bh_enable_ip(unsigned long ip) · line 170

Quoted 5 lines
kernel/softirq.c +5 -0
+
+/*
+ * Caller must disable preemption, and take care of appropriate
+ * locking and refcounting
+ */

Shouldn't we call it __softirq_delay_queue() if the caller needs to disabled preemption?

Futhermore, don't we always require the caller to take care of lifetime issues when we queue something?

Quoted 13 lines
kernel/softirq.c +13 -0
+int softirq_delay_queue(struct softirq_delay *sdel)
+{
+	if (!sdel->next) {
+		sdel->next = __get_cpu_var(softirq_delay_head);
+		__get_cpu_var(softirq_delay_head) = sdel;
+		return 1;
+	}
+	return 0;
+}
+
+/*
+ * Because locking is provided by subsystem, please note
+ * that sdel->func(sdel) is responsible for setting sdel->next to NULL

kernel/softirq.c void local_bh_enable_ip(unsigned long ip) · line 193

Quoted 10 lines
kernel/softirq.c +10 -0
+ */
+static void softirq_delay_exec(void)
+{
+	struct softirq_delay *sdel;
+
+	while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
+		__get_cpu_var(softirq_delay_head) = sdel->next;
+		sdel->func(sdel);	/*	sdel->next = NULL;*/
+		}
+}

Why can't we write:

struct softirq_delay *sdel, *next;

  sdel = __get_cpu_var(softirq_delay_head);
  __get_cpu_var(softirq_delay_head) = NULL;

  while (sdel) {
    next = sdel->next;
    sdel->func(sdel);
    sdel = next;
  }

Why does it matter what happens to sdel->next? we've done the callback.

Aah, the crux is in the re-use policy.. that most certainly does deserve a comment.

How about we make sdel->next point to itself in the init case?

Then we can write:

  while (sdel) {
    next = sdel->next;
    sdel->next = sdel;
    sdel->func(sdel);
    sdel = next;
  }

and have the enqueue bit look like:

int __softirq_delay_queue(struct softirq_delay *sdel) {
struct softirq_delay **head;

if (sdel->next != sdel)

    return 0;

head = &__get_cpu_var(softirq_delay_head);
sdel->next = *head;
*head = sdel;
return 1;
}

Quoted 8 lines
kernel/softirq.c +4 -0
@@ -1691,6 +1694,43 @@ static void sock_def_readable(struct sock *sk, int len)
 	read_unlock(&sk->sk_callback_lock);
 }
 
+/*
+ * helper function called by softirq_delay_exec(),
+ * if inet_def_readable() queued us.
+ */

kernel/softirq.c static void sock_def_readable(struct sock *sk, int len) · line 1710

Quoted 10 lines
kernel/softirq.c +10 -0
+static void sock_readable_defer(struct softirq_delay *sdel)
+{
+	struct sock *sk = container_of(sdel, struct sock, sk_delay);
+
+	sdel->next = NULL;
+	/*
+	 * At this point, we dont own a lock on socket, only a reference.
+	 * We must commit above write, or another cpu could miss a wakeup
+	 */
+	smp_wmb();

Where's the matching barrier?

Quoted 12 lines
kernel/softirq.c +12 -0
+	sock_def_readable(sk, 0);
+	sock_put(sk);
+}
+
+/*
+ * Custom version of sock_def_readable()
+ * We want to defer scheduler processing at the end of do_softirq()
+ * Called with socket locked.
+ */
+void inet_def_readable(struct sock *sk, int len)
+{
+	if (running_from_softirq()) {

kernel/softirq.c static void sock_def_readable(struct sock *sk, int len) · line 1732

Quoted 10 lines
kernel/softirq.c +10 -0
+		if (softirq_delay_queue(&sk->sk_delay))
+			/*
+			 * If we queued this socket, take a reference on it
+			 * Caller owns socket lock, so write to sk_delay.next
+			 * will be committed before unlock.
+			 */
+			sock_hold(sk);
+	} else
+		sock_def_readable(sk, len);
+}

OK, so the idea is to handle a bunch of packets and instead of waking N threads for each packet, only wake them once at the end of the batch?

Sounds like a sensible idea..

#45 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Eric Dumazet

Here is the last incantation of the patch, that of course should be split in two parts and better Changelog for further discussion on lkml.

I read the entire thread up to now, and I still don't really understand the Changelog, sorry :(

Sure, I should have taken more time, will repost this in a couple of hours, with nice CHangelogs and split patches.

Quoted 8 lines
Quoted 8 lines

[PATCH] softirq: Introduce mechanism to defer wakeups

Some network workloads need to call scheduler too many times. For example, each received multicast frame can wakeup many threads. ksoftirqd is then not able to drain NIC RX queues in time and we get frame losses and high latencies.

Quoted 5 lines
Quoted 3 lines

This patch adds an infrastructure to delay work done in sock_def_readable() at end of do_softirq(). This needs to make available current->softirq_context even if !CONFIG_TRACE_IRQFLAGS

How does that solve the wakeup issue?

Apparently, on SMP machines this actually helps a lot, in case of multicast trafic handled by many subscribers. skb_cloning involves atomic ops on route cache entries, and if we wakeup threads as we currently do, they start to consume skb while the feeder is still doing skb clones for other sockets. Many cache line ping pongs are slowing down the softirq.

I will post the test program to reproduce the problem.

Quoted 10 lines
Quoted 10 lines

---

kernel/softirq.c +0 -0
@@ -158,6 +158,42 @@ void local_bh_enable_ip(unsigned long ip)
 }
 EXPORT_SYMBOL(local_bh_enable_ip);
 
Quoted 7 lines

kernel/softirq.c void local_bh_enable_ip(unsigned long ip) · line 165

Quoted 5 lines
kernel/softirq.c +5 -0
+
+#define SOFTIRQ_DELAY_END (struct softirq_delay *)1L
+static DEFINE_PER_CPU(struct softirq_delay *, softirq_delay_head) = {
+	SOFTIRQ_DELAY_END
+};

Why the magic termination value? Can't we NULL terminate the list

Yes we can, you are right.

Quoted 9 lines

kernel/softirq.c void local_bh_enable_ip(unsigned long ip) · line 170

Quoted 6 lines
kernel/softirq.c +5 -0
+
+/*
+ * Caller must disable preemption, and take care of appropriate
+ * locking and refcounting
+ */

Shouldn't we call it __softirq_delay_queue() if the caller needs to disabled preemption?

I was wondering if some BUG_ON() can be added to crash if preemption is enabled at this point. Could not find an existing check,
doing again the 'if (running_from_softirq())'" test might be overkill, should I document caller should do :

skeleton :

    lock_my_data(data); /* barrier here */
    sdel = &data->sdel;
    if (running_from_softirq()) {
	if (softirq_delay_queue(sdel)) {
		hold a refcount on data;
	} else {
		/* already queued, nothing to do */
	}
    } else {
	/* cannot queue the work , must do it right now */
	do_work(data);
    }
    release_my_data(data);
}
Quoted 3 lines

Futhermore, don't we always require the caller to take care of lifetime issues when we queue something?

You mean comment is too verbose... or

Quoted 38 lines
Quoted 14 lines
kernel/softirq.c +13 -0
+int softirq_delay_queue(struct softirq_delay *sdel)
+{
+	if (!sdel->next) {
+		sdel->next = __get_cpu_var(softirq_delay_head);
+		__get_cpu_var(softirq_delay_head) = sdel;
+		return 1;
+	}
+	return 0;
+}
+
+/*
+ * Because locking is provided by subsystem, please note
+ * that sdel->func(sdel) is responsible for setting sdel->next to NULL

kernel/softirq.c void local_bh_enable_ip(unsigned long ip) · line 193

Quoted 10 lines
kernel/softirq.c +10 -0
+ */
+static void softirq_delay_exec(void)
+{
+	struct softirq_delay *sdel;
+
+	while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
+		__get_cpu_var(softirq_delay_head) = sdel->next;
+		sdel->func(sdel);	/*	sdel->next = NULL;*/
+		}
+}

Why can't we write:

struct softirq_delay *sdel, *next;

  sdel = __get_cpu_var(softirq_delay_head);
  __get_cpu_var(softirq_delay_head) = NULL;

  while (sdel) {
    next = sdel->next;
    sdel->func(sdel);
    sdel = next;
  }
Quoted 4 lines

Why does it matter what happens to sdel->next? we've done the callback.

Aah, the crux is in the re-use policy.. that most certainly does deserve a comment.

Hum, so my comment was not verbose enough :)

Quoted 12 lines

How about we make sdel->next point to itself in the init case?

Then we can write:

  while (sdel) {
    next = sdel->next;
    sdel->next = sdel;
    sdel->func(sdel);
    sdel = next;
  }
Quoted 8 lines

and have the enqueue bit look like:

int __softirq_delay_queue(struct softirq_delay *sdel) {
struct softirq_delay **head;

if (sdel->next != sdel)

    return 0;

Yes we could do that

Quoted 11 lines

head = &__get_cpu_var(softirq_delay_head);
sdel->next = *head;
*head = sdel;
return 1;
}

Quoted 4 lines
kernel/softirq.c +0 -0
@@ -1691,6 +1694,43 @@ static void sock_def_readable(struct sock *sk, int len)
 	read_unlock(&sk->sk_callback_lock);
 }
 
Quoted 16 lines
Quoted 4 lines
kernel/softirq.c +4 -0
+/*
+ * helper function called by softirq_delay_exec(),
+ * if inet_def_readable() queued us.
+ */

kernel/softirq.c static void sock_def_readable(struct sock *sk, int len) · line 1710

Quoted 10 lines
kernel/softirq.c +10 -0
+static void sock_readable_defer(struct softirq_delay *sdel)
+{
+	struct sock *sk = container_of(sdel, struct sock, sk_delay);
+
+	sdel->next = NULL;
+	/*
+	 * At this point, we dont own a lock on socket, only a reference.
+	 * We must commit above write, or another cpu could miss a wakeup
+	 */
+	smp_wmb();

Where's the matching barrier?

Check softirq_delay_exec(void) comment, where I stated synchronization had to be done by the subsystem.

In this socket case, caller of softirq_delay_exec() has a lock on socket.

Problem is I dont want to get this lock again in sock_readable_defer() callback

if sdel->next is not committed, another cpu could call _softirq_delay_queue() and find sdel->next being not null (or != sdel with your suggestion). Then next->func() wont be called as it should (or called litle bit too soon)

So matching barrier is on "lock_my_data(data)" in previous skeleton ?

Quoted 27 lines
Quoted 13 lines
kernel/softirq.c +12 -0
+	sock_def_readable(sk, 0);
+	sock_put(sk);
+}
+
+/*
+ * Custom version of sock_def_readable()
+ * We want to defer scheduler processing at the end of do_softirq()
+ * Called with socket locked.
+ */
+void inet_def_readable(struct sock *sk, int len)
+{
+	if (running_from_softirq()) {

kernel/softirq.c static void sock_def_readable(struct sock *sk, int len) · line 1732

Quoted 10 lines
kernel/softirq.c +10 -0
+		if (softirq_delay_queue(&sk->sk_delay))
+			/*
+			 * If we queued this socket, take a reference on it
+			 * Caller owns socket lock, so write to sk_delay.next
+			 * will be committed before unlock.
+			 */
+			sock_hold(sk);
+	} else
+		sock_def_readable(sk, len);
+}

OK, so the idea is to handle a bunch of packets and instead of waking N threads for each packet, only wake them once at the end of the batch?

Quoted 1 line

Sounds like a sensible idea..

Idea is to batch wakeups() yes, and if we receive several packets for the same socket(s), we reduce number of wakeups to one. In the multicast stress situation of Athena CR, it really helps, no packets dropped instead of 30%

Thanks Peter

maintainer
#46 lore raw
Quoted 11 lines · Eric Dumazet
Quoted 8 lines

kernel/softirq.c void local_bh_enable_ip(unsigned long ip) · line 170

Quoted 5 lines
kernel/softirq.c +5 -0
+
+/*
+ * Caller must disable preemption, and take care of appropriate
+ * locking and refcounting
+ */

Shouldn't we call it __softirq_delay_queue() if the caller needs to disabled preemption?

I was wondering if some BUG_ON() can be added to crash if preemption is enabled at this point.

__get_cpu_var() has a preemption check and will generate BUGs when CONFIG_DEBUG_PREEMPT similar to smp_processor_id().

Quoted 9 lines

Could not find an existing check,
doing again the 'if (running_from_softirq())'" test might be overkill, should I document caller should do :

skeleton :

    lock_my_data(data); /* barrier here */
    sdel = &data->sdel;
    if (running_from_softirq()) {

Small nit: I don't particularly like the running_from_softirq() name, but in_softirq() is already taken, and sadly means something slightly different.

Quoted 13 lines
	if (softirq_delay_queue(sdel)) {
		hold a refcount on data;
	} else {
		/* already queued, nothing to do */
	}
    } else {
	/* cannot queue the work , must do it right now */
	do_work(data);
    }
    release_my_data(data);
}
Quoted 1 line
Quoted 4 lines
Quoted 2 lines

Futhermore, don't we always require the caller to take care of lifetime issues when we queue something?

You mean comment is too verbose... or

Yeah.

Quoted 4 lines
Quoted 2 lines

Aah, the crux is in the re-use policy.. that most certainly does deserve a comment.

Hum, so my comment was not verbose enough :)

That too :-)

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
net/core/sock.c +4 -0
+static void sock_readable_defer(struct softirq_delay *sdel)
+{
+	struct sock *sk = container_of(sdel, struct sock, sk_delay);
+
Quoted 8 lines

net/core/sock.c static void sock_def_readable(struct sock *sk, int len) · line 1710

Quoted 6 lines
net/core/sock.c +6 -0
+	sdel->next = NULL;
+	/*
+	 * At this point, we dont own a lock on socket, only a reference.
+	 * We must commit above write, or another cpu could miss a wakeup
+	 */
+	smp_wmb();

Where's the matching barrier?

Quoted 2 lines

Check softirq_delay_exec(void) comment, where I stated synchronization had to be done by the subsystem.

afaiu the memory barrier semantics you cannot pair a wmb with a lock barrier, it must either be a read, read_barrier_depends or full barrier.

Quoted 7 lines

In this socket case, caller of softirq_delay_exec() has a lock on socket.

Problem is I dont want to get this lock again in sock_readable_defer() callback

if sdel->next is not committed, another cpu could call _softirq_delay_queue() and find sdel->next being not null (or != sdel with your suggestion). Then next->func() wont be called as it should (or called litle bit too soon)

Right, what we can do is put the wmb in the callback and the rmb right before the __queue op, or simply integrate it into the framework.

Quoted 9 lines
Quoted 4 lines

OK, so the idea is to handle a bunch of packets and instead of waking N threads for each packet, only wake them once at the end of the batch?

Sounds like a sensible idea..

Idea is to batch wakeups() yes, and if we receive several packets for the same socket(s), we reduce number of wakeups to one. In the multicast stress situation of Athena CR, it really helps, no packets dropped instead of 30%

Yes I can see that helping tremendously.

#47 lore raw
In reply to Eric Dumazet
Quoted 2 lines · Eric Dumazet

Sure, I should have taken more time, will repost this in a couple of hours, with nice CHangelogs and split patches.

One small thing: with CONFIG_IPV6=m, inet_def_readable needs to be exported, right?

Thanks,
Brian Bloniarz

#48 lore raw
Quoted 9 lines · Brian Bloniarz
Quoted 3 lines · Eric Dumazet

Sure, I should have taken more time, will repost this in a couple of hours,
with nice CHangelogs and split patches.

One small thing: with CONFIG_IPV6=m, inet_def_readable needs to be exported,
right?

Absolutely, thank you !

#49 lore raw

I did some testing with this and see at least a 20% improvement without drop.

I agree with Peter's recommended changes (esp. sentinel vs null), and also the trivial brace indentation in softirq_delay_exec(), but otherwise looks good to me. Nice work.

                                        +-DLS
#50 lore raw
Quoted 10 lines · David Stevens

I did some testing with this and see at least a 20% improvement without drop.

I agree with Peter's recommended changes (esp. sentinel vs null), and also the trivial brace indentation in softirq_delay_exec(), but otherwise looks good to me. Nice work.

                                        +-DLS

Still I dont like very much all softirq.c changes. I feel very uncomfortable to justify one extra call in do_softirq(), and not very clean interface (stuff about locking, barriers...)

Easy way could be to add a SOFTIRQ but its not very wise.

I was wondering if we could use the infrastructure added in commit 54514a70adefe356afe854e2d3912d46668068e6
(softirq: Add support for triggering softirq work on softirqs.) But I dont understand how it can works... (softirq_work_list is feeded, but never processed)

Alternatively, we could use a framework dedicated to network use, with well defined semantic :

Calling softirq_delay_exec() from net_rx_action(), from this function, we know if time_squeeze was incremented, or all netdev_budget consumed, and in this stress case try to give the wakeups job to another cpu.

#51 lore raw
In reply to Eric Dumazet

Hi Eric,

We've been experimenting with this softirq-delay patch in production, and have seen some hard-to-reproduce crashes. We finally managed to capture a kexec crashdump this morning.

This is the dmesg:

[53417.592868] Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
[53417.598377]  [<ffffffff80243643>] __do_softirq+0xc3/0x150
[53417.606300] PGD 32abb8067 PUD 32faf5067 PMD 0
[53417.610829] Oops: 0000 [1] SMP
[53417.614032] CPU 2
[53417.616083] Modules linked in: nfs lockd nfs_acl sunrpc openafs(P) autofs4 ipv6 ac sbs sbshc video output dock battery container iptable_filter ip_tables x_tables parport_pc lp parport loop joydev iTCO_wdt iTCO_vendor_support evdev button i5000_edac psmouse serio_raw pcspkr shpchp pci_hotplug edac_core ext3 jbd mbcache sr_mod cdrom ata_generic usbhid hid ata_piix sg sd_mod ehci_hcd pata_acpi uhci_hcd libata bnx2 aacraid usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse
[53417.662067] Pid: 13039, comm: gball Tainted: P        2.6.24-19acr2-generic #1
[53417.669219] RIP: 0010:[<ffffffff80243643>]  [<ffffffff80243643>] __do_softirq+0xc3/0x150
[53417.677368] RSP: 0018:ffff8103314f3f20  EFLAGS: 00010297
[53417.682697] RAX: ffff810084a1b000 RBX: ffffffff805ba530 RCX: 0000000000000000
[53417.689843] RDX: ffff8103305811e0 RSI: 0000000000000282 RDI: ffff810332ada580
[53417.696993] RBP: 0000000000000000 R08: ffff81032fad9f08 R09: ffff810332382000
[53417.704144] R10: 0000000000000000 R11: ffffffff80316ec0 R12: ffffffff8062b3d8
[53417.711294] R13: ffffffff8062b480 R14: 0000000000000002 R15: 000000000000000a
[53417.718447] FS:  00007fab0d7b8750(0000) GS:ffff810334401b80(0000) knlGS:0000000000000000
[53417.726568] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[53417.732332] CR2: 0000000000000000 CR3: 0000000329e2d000 CR4: 00000000000006e0
[53417.739476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[53417.746637] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[53417.753787] Process gball (pid: 13039, threadinfo ffff81032adde000, task ffff810329ff77d0)
[53417.761991] Stack:  ffffffff8062b3d8 0000000000000046 ffff8103314f3f68 0000000000000000
[53417.770146]  00000000000000a0 ffff81032addfee8 0000000000000000 ffffffff8020d50c
[53417.777660]  ffff8103314f3f68 00000000000000c1 ffffffff8020ed25 ffffffff8062c870
[53417.784961] Call Trace:
[53417.787635]  <IRQ>  [<ffffffff8020d50c>] call_softirq+0x1c/0x30
[53417.793597]  [<ffffffff8020ed25>] do_softirq+0x35/0x90
[53417.798747]  [<ffffffff80243578>] irq_exit+0x88/0x90
[53417.803727]  [<ffffffff8020ef70>] do_IRQ+0x80/0x100
[53417.808624]  [<ffffffff8020c891>] ret_from_intr+0x0/0xa
[53417.813862]  <EOI>  [<ffffffff803e53c8>] skb_release_all+0x18/0x150
[53417.820164]  [<ffffffff803e4ad9>] __kfree_skb+0x9/0x90
[53417.825327]  [<ffffffff80437612>] udp_recvmsg+0x222/0x260
[53417.830744]  [<ffffffff80231264>] source_load+0x34/0x70
[53417.835984]  [<ffffffff80232a9a>] find_busiest_group+0x1fa/0x850
[53417.842019]  [<ffffffff803e0100>] sock_common_recvmsg+0x30/0x50
[53417.847958]  [<ffffffff803de1ca>] sock_recvmsg+0x14a/0x160
[53417.853462]  [<ffffffff80231c21>] update_curr+0x71/0x100
[53419.858789]  [<ffffffff802320fd>] __dequeue_entity+0x3d/0x50
[53417.864469]  [<ffffffff80253ab0>] autoremove_wake_function+0x0/0x30
[53417.870758]  [<ffffffff8046662f>] thread_return+0x3a/0x57b
[53417.876262]  [<ffffffff803df73e>] sys_recvfrom+0xfe/0x190
[53417.881680]  [<ffffffff802e2a95>] sys_epoll_wait+0x245/0x4e0
[53417.887358]  [<ffffffff80233e20>] default_wake_function+0x0/0x10
[53417.893384]  [<ffffffff8020c37e>] system_call+0x7e/0x83
[53417.898628]
[53417.900134]
[53417.900134] Code: 48 8b 11 48 89 cf 65 48 8b 04 25 08 00 00 00 4a 89 14 20 ff
[53417.909430] RIP  [<ffffffff80243643>] __do_softirq+0xc3/0x150
[53417.915210]  RSP <ffff8103314f3f20>

The disassembly where it crashed:
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:273
ffffffff8024361b: d1 ed shr %ebp
rcu_bh_qsctr_inc():
/local/home/bmb/doc/kernels/linux-hardy-eric/include/linux/rcupdate.h:130

ffffffff8024361d:       48 8b 40 08             mov    0x8(%rax),%rax
ffffffff80243621:       41 c7 44 05 08 01 00    movl   $0x1,0x8(%r13,%rax,1)
ffffffff80243628:       00 00

__do_softirq():
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:273
ffffffff8024362a: 75 d8 jne ffffffff80243604 <__do_softirq+0x84>
softirq_delay_exec():
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:225

ffffffff8024362c:       48 8b 14 24             mov    (%rsp),%rdx
ffffffff80243630:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
ffffffff80243637:       00 00
ffffffff80243639:       48 8b 0c 10             mov    (%rax,%rdx,1),%rcx
ffffffff8024363d:       48 83 f9 01             cmp    $0x1,%rcx
ffffffff80243641:       74 29                   je     ffffffff8024366c <__do_softirq+0xec>

/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:226
ffffffff80243643: 48 8b 11 mov (%rcx),%rdx
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:227
ffffffff80243646: 48 89 cf mov %rcx,%rdi
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:226

ffffffff80243649:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
ffffffff80243650:       00 00
ffffffff80243652:       4a 89 14 20             mov    %rdx,(%rax,%r12,1)
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:227
ffffffff80243656:       ff 51 08                callq  *0x8(%rcx)
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:225
ffffffff80243659:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
ffffffff80243660:       00 00
ffffffff80243662:       4a 8b 0c 20             mov    (%rax,%r12,1),%rcx
ffffffff80243666:       48 83 f9 01             cmp    $0x1,%rcx
ffffffff8024366a:       75 d7                   jne    ffffffff80243643 <__do_softirq+0xc3>

raw_local_irq_disable():
/local/home/bmb/doc/kernels/linux-hardy-eric/debian/build/build-generic/include2/asm/irqflags_64.h:76
ffffffff8024366c: fa cli

And softirq.c line numbers:

   218   * Because locking is provided by subsystem, please note
   219   * that sdel->func(sdel) is responsible for setting sdel->next to NULL
   220   */
   221  static void softirq_delay_exec(void)
   222  {
   223          struct softirq_delay *sdel;
   224
   225          while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
   226                  __get_cpu_var(softirq_delay_head) = sdel->next;
   227                  sdel->func(sdel);       /*      sdel->next = NULL;*/
   228                  }

229 }

So it's crashing because __get_cpu_var(softirq_delay_head)) is NULL somehow.

We aren't running a recent kernel -- we're running Ubuntu Hardy's 2.6.24-19, with a backported version of this patch. One more atypical thing is that we run openafs, 1.4.6.dfsg1-2.

Like I said, I have a full vmcore (3, actually) and would be happy to post any more information you'd like to know.

Thanks,
Brian Bloniarz

Quoted 403 lines · Eric Dumazet
Quoted 9 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Fri, 13 Mar 2009 23:30:31 +0100

Quoted 6 lines
Quoted 5 lines · David Miller
Quoted 5 lines

Also, when an event was queued for later invocation, I also needed to keep
a reference on "struct socket" to make sure it doesnt disappear before
the invocation. Not all sockets are RCU guarded (we added RCU only for some protocols (TCP, UDP ...). So I found keeping a read_lock on callback was the easyest thing to do. I now realize we might
overflow preempt_count, so special care is needed.

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

You're using this in UDP so... make the rule that you can't use this with a non-RCU-quiescent protocol.

UDP/TCP only ? I though many other protocols (not all using RCU) were using sock_def_readable() too...

Maybe create a inet_def_readable() just for this purpose :-)

Here is the last incantation of the patch, that of course should be split in two parts and better Changelog for further discussion on lkml.

We need to take a reference on sock when queued on a softirq delay list. RCU wont help here because of SLAB_DESTROY_BY_RCU thing :
Another cpu could free/reuse the socket before we have a chance to call softirq_delay_exec()

UDP & UDPLite use this delayed wakeup feature.

Thank you

[PATCH] softirq: Introduce mechanism to defer wakeups

Some network workloads need to call scheduler too many times. For example, each received multicast frame can wakeup many threads. ksoftirqd is then not able to drain NIC RX queues in time and we get frame losses and high latencies.

This patch adds an infrastructure to delay work done in sock_def_readable() at end of do_softirq(). This needs to make available current->softirq_context even if !CONFIG_TRACE_IRQFLAGS

---
 include/linux/interrupt.h |   18 +++++++++++++++
 include/linux/irqflags.h  |   11 ++++-----
 include/linux/sched.h     |    2 -
 include/net/sock.h        |    2 +
 include/net/udplite.h     |    1
 kernel/lockdep.c          |    2 -
 kernel/softirq.c          |   42 ++++++++++++++++++++++++++++++++++--
 lib/locking-selftest.c    |    4 +--
 net/core/sock.c           |   41 +++++++++++++++++++++++++++++++++++
 net/ipv4/udp.c            |    7 ++++++
 net/ipv6/udp.c            |    7 ++++++
 11 files changed, 125 insertions(+), 12 deletions(-)
include/linux/interrupt.h +18 -0
index 9127f6b..a773d0c 100644
@@ -295,6 +295,24 @@ extern void send_remote_softirq(struct call_single_data *cp, int cpu, int softir
 extern void __send_remote_softirq(struct call_single_data *cp, int cpu,
 				  int this_cpu, int softirq);
 
+/*
+ * softirq delayed works : should be delayed at do_softirq() end
+ */
+struct softirq_delay {
+	struct softirq_delay	*next;
+	void 			(*func)(struct softirq_delay *);
+};
+
+int softirq_delay_queue(struct softirq_delay *sdel);
+
+static inline void softirq_delay_init(struct softirq_delay *sdel,
+				      void (*func)(struct softirq_delay *))
+{
+	sdel->next = NULL;
+	sdel->func = func;
+}
+
+
 /* Tasklets --- multithreaded analogue of BHs.
 
    Main feature differing them of generic softirqs: tasklet
include/linux/irqflags.h +5 -6
index 74bde13..30c1e01 100644
@@ -13,19 +13,21 @@
 
 #include <linux/typecheck.h>
 
+#define softirq_enter()	do { current->softirq_context++; } while (0)
+#define softirq_exit()	do { current->softirq_context--; } while (0)
+#define softirq_context(p)	((p)->softirq_context)
+#define running_from_softirq()  (softirq_context(current) > 0)
+
 #ifdef CONFIG_TRACE_IRQFLAGS
   extern void trace_softirqs_on(unsigned long ip);
   extern void trace_softirqs_off(unsigned long ip);
   extern void trace_hardirqs_on(void);
   extern void trace_hardirqs_off(void);
 # define trace_hardirq_context(p)	((p)->hardirq_context)
-# define trace_softirq_context(p)	((p)->softirq_context)
 # define trace_hardirqs_enabled(p)	((p)->hardirqs_enabled)
 # define trace_softirqs_enabled(p)	((p)->softirqs_enabled)
 # define trace_hardirq_enter()	do { current->hardirq_context++; } while (0)
 # define trace_hardirq_exit()	do { current->hardirq_context--; } while (0)
-# define trace_softirq_enter()	do { current->softirq_context++; } while (0)
-# define trace_softirq_exit()	do { current->softirq_context--; } while (0)
 # define INIT_TRACE_IRQFLAGS	.softirqs_enabled = 1,
 #else
 # define trace_hardirqs_on()		do { } while (0)
@@ -33,13 +35,10 @@
 # define trace_softirqs_on(ip)		do { } while (0)
 # define trace_softirqs_off(ip)		do { } while (0)
 # define trace_hardirq_context(p)	0
-# define trace_softirq_context(p)	0
 # define trace_hardirqs_enabled(p)	0
 # define trace_softirqs_enabled(p)	0
 # define trace_hardirq_enter()		do { } while (0)
 # define trace_hardirq_exit()		do { } while (0)
-# define trace_softirq_enter()		do { } while (0)
-# define trace_softirq_exit()		do { } while (0)
 # define INIT_TRACE_IRQFLAGS
 #endif
 
include/linux/sched.h +1 -1
index 8c216e0..5dd8487 100644
@@ -1320,8 +1320,8 @@ struct task_struct {
 	unsigned long softirq_enable_ip;
 	unsigned int softirq_enable_event;
 	int hardirq_context;
-	int softirq_context;
 #endif
+	int softirq_context;
 #ifdef CONFIG_LOCKDEP
 # define MAX_LOCK_DEPTH 48UL
 	u64 curr_chain_key;
include/net/sock.h +2 -0
index 4bb1ff9..0160a83 100644
@@ -260,6 +260,7 @@ struct sock {
 	unsigned long	        sk_lingertime;
 	struct sk_buff_head	sk_error_queue;
 	struct proto		*sk_prot_creator;
+	struct softirq_delay	sk_delay;
 	rwlock_t		sk_callback_lock;
 	int			sk_err,
 				sk_err_soft;
@@ -960,6 +961,7 @@ extern void *sock_kmalloc(struct sock *sk, int size,
 			  gfp_t priority);
 extern void sock_kfree_s(struct sock *sk, void *mem, int size);
 extern void sk_send_sigurg(struct sock *sk);
+extern void inet_def_readable(struct sock *sk, int len);
 
 /*
  * Functions to fill in entries in struct proto_ops when a protocol
include/net/udplite.h +1 -0
index afdffe6..7ce0ee0 100644
@@ -25,6 +25,7 @@ static __inline__ int udplite_getfrag(void *from, char *to, int  offset,
 /* Designate sk as UDP-Lite socket */
 static inline int udplite_sk_init(struct sock *sk)
 {
+	sk->sk_data_ready = inet_def_readable;
 	udp_sk(sk)->pcflag = UDPLITE_BIT;
 	return 0;
 }
kernel/lockdep.c +1 -1
index 06b0c35..9873b40 100644
@@ -1807,7 +1807,7 @@ print_usage_bug(struct task_struct *curr, struct held_lock *this,
 	printk("%s/%d [HC%u[%lu]:SC%u[%lu]:HE%u:SE%u] takes:\n",
 		curr->comm, task_pid_nr(curr),
 		trace_hardirq_context(curr), hardirq_count() >> HARDIRQ_SHIFT,
-		trace_softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
+		softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
 		trace_hardirqs_enabled(curr),
 		trace_softirqs_enabled(curr));
 	print_lock(this);
kernel/softirq.c +40 -2
index bdbe9de..91a1714 100644
@@ -158,6 +158,42 @@ void local_bh_enable_ip(unsigned long ip)
 }
 EXPORT_SYMBOL(local_bh_enable_ip);
 
+
+#define SOFTIRQ_DELAY_END (struct softirq_delay *)1L
+static DEFINE_PER_CPU(struct softirq_delay *, softirq_delay_head) = {
+	SOFTIRQ_DELAY_END
+};
+
+/*
+ * Caller must disable preemption, and take care of appropriate
+ * locking and refcounting
+ */
+int softirq_delay_queue(struct softirq_delay *sdel)
+{
+	if (!sdel->next) {
+		sdel->next = __get_cpu_var(softirq_delay_head);
+		__get_cpu_var(softirq_delay_head) = sdel;
+		return 1;
+	}
+	return 0;
+}
+
+/*
+ * Because locking is provided by subsystem, please note
+ * that sdel->func(sdel) is responsible for setting sdel->next to NULL
+ */
+static void softirq_delay_exec(void)
+{
+	struct softirq_delay *sdel;
+
+	while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
+		__get_cpu_var(softirq_delay_head) = sdel->next;
+		sdel->func(sdel);	/*	sdel->next = NULL;*/
+		}
+}
+
+
+
 /*
  * We restart softirq processing MAX_SOFTIRQ_RESTART times,
  * and we fall back to softirqd after that.
@@ -180,7 +216,7 @@ asmlinkage void __do_softirq(void)
 	account_system_vtime(current);
 
 	__local_bh_disable((unsigned long)__builtin_return_address(0));
-	trace_softirq_enter();
+	softirq_enter();
 
 	cpu = smp_processor_id();
 restart:
@@ -211,6 +247,8 @@ restart:
 		pending >>= 1;
 	} while (pending);
 
+	softirq_delay_exec();
+
 	local_irq_disable();
 
 	pending = local_softirq_pending();
@@ -220,7 +258,7 @@ restart:
 	if (pending)
 		wakeup_softirqd();
 
-	trace_softirq_exit();
+	softirq_exit();
 
 	account_system_vtime(current);
 	_local_bh_enable();
lib/locking-selftest.c +2 -2
index 280332c..1aa7351 100644
@@ -157,11 +157,11 @@ static void init_shared_classes(void)
 #define SOFTIRQ_ENTER()				\
 		local_bh_disable();		\
 		local_irq_disable();		\
-		trace_softirq_enter();		\
+		softirq_enter();		\
 		WARN_ON(!in_softirq());
 
 #define SOFTIRQ_EXIT()				\
-		trace_softirq_exit();		\
+		softirq_exit();		\
 		local_irq_enable();		\
 		local_bh_enable();
 
net/core/sock.c +41 -0
index 0620046..c8745d1 100644
@@ -213,6 +213,8 @@ __u32 sysctl_rmem_default __read_mostly = SK_RMEM_MAX;
 /* Maximal space eaten by iovec or ancilliary data plus some space */
 int sysctl_optmem_max __read_mostly = sizeof(unsigned long)*(2*UIO_MAXIOV+512);
 
+static void sock_readable_defer(struct softirq_delay *sdel);
+
 static int sock_set_timeout(long *timeo_p, char __user *optval, int optlen)
 {
 	struct timeval tv;
@@ -1074,6 +1076,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority)
 #endif
 
 		rwlock_init(&newsk->sk_dst_lock);
+		softirq_delay_init(&newsk->sk_delay, sock_readable_defer);
 		rwlock_init(&newsk->sk_callback_lock);
 		lockdep_set_class_and_name(&newsk->sk_callback_lock,
 				af_callback_keys + newsk->sk_family,
@@ -1691,6 +1694,43 @@ static void sock_def_readable(struct sock *sk, int len)
 	read_unlock(&sk->sk_callback_lock);
 }
 
+/*
+ * helper function called by softirq_delay_exec(),
+ * if inet_def_readable() queued us.
+ */
+static void sock_readable_defer(struct softirq_delay *sdel)
+{
+	struct sock *sk = container_of(sdel, struct sock, sk_delay);
+
+	sdel->next = NULL;
+	/*
+	 * At this point, we dont own a lock on socket, only a reference.
+	 * We must commit above write, or another cpu could miss a wakeup
+	 */
+	smp_wmb();
+	sock_def_readable(sk, 0);
+	sock_put(sk);
+}
+
+/*
+ * Custom version of sock_def_readable()
+ * We want to defer scheduler processing at the end of do_softirq()
+ * Called with socket locked.
+ */
+void inet_def_readable(struct sock *sk, int len)
+{
+	if (running_from_softirq()) {
+		if (softirq_delay_queue(&sk->sk_delay))
+			/*
+			 * If we queued this socket, take a reference on it
+			 * Caller owns socket lock, so write to sk_delay.next
+			 * will be committed before unlock.
+			 */
+			sock_hold(sk);
+	} else
+		sock_def_readable(sk, len);
+}
+
 static void sock_def_write_space(struct sock *sk)
 {
 	read_lock(&sk->sk_callback_lock);
@@ -1768,6 +1808,7 @@ void sock_init_data(struct socket *sock, struct sock *sk)
 		sk->sk_sleep	=	NULL;
 
 	rwlock_init(&sk->sk_dst_lock);
+	softirq_delay_init(&sk->sk_delay, sock_readable_defer);
 	rwlock_init(&sk->sk_callback_lock);
 	lockdep_set_class_and_name(&sk->sk_callback_lock,
 			af_callback_keys + sk->sk_family,
net/ipv4/udp.c +7 -0
index 05b7abb..1cc0907 100644
@@ -1342,6 +1342,12 @@ void udp_destroy_sock(struct sock *sk)
 	release_sock(sk);
 }
 
+static int udp_init_sock(struct sock *sk)
+{
+	sk->sk_data_ready = inet_def_readable;
+	return 0;
+}
+
 /*
  *	Socket option code for UDP
  */
@@ -1559,6 +1565,7 @@ struct proto udp_prot = {
 	.connect	   = ip4_datagram_connect,
 	.disconnect	   = udp_disconnect,
 	.ioctl		   = udp_ioctl,
+	.init		   = udp_init_sock,
 	.destroy	   = udp_destroy_sock,
 	.setsockopt	   = udp_setsockopt,
 	.getsockopt	   = udp_getsockopt,
net/ipv6/udp.c +7 -0
index 84b1a29..1a9f8d4 100644
@@ -960,6 +960,12 @@ void udpv6_destroy_sock(struct sock *sk)
 	inet6_destroy_sock(sk);
 }
 
+static int udpv6_init_sock(struct sock *sk)
+{
+	sk->sk_data_ready = inet_def_readable;
+	return 0;
+}
+
 /*
  *	Socket option code for UDP
  */
@@ -1084,6 +1090,7 @@ struct proto udpv6_prot = {
 	.connect	   = ip6_datagram_connect,
 	.disconnect	   = udp_disconnect,
 	.ioctl		   = udp_ioctl,
+	.init 		   = udpv6_init_sock,
 	.destroy	   = udpv6_destroy_sock,
 	.setsockopt	   = udpv6_setsockopt,
 	.getsockopt	   = udpv6_getsockopt,
#52 lore raw
Quoted 151 lines · Brian Bloniarz

Hi Eric,

We've been experimenting with this softirq-delay patch in production, and have seen some hard-to-reproduce crashes. We finally managed to capture a kexec crashdump this morning.

This is the dmesg:

[53417.592868] Unable to handle kernel NULL pointer dereference at
0000000000000000 RIP:
[53417.598377]  [<ffffffff80243643>] __do_softirq+0xc3/0x150
[53417.606300] PGD 32abb8067 PUD 32faf5067 PMD 0
[53417.610829] Oops: 0000 [1] SMP
[53417.614032] CPU 2
[53417.616083] Modules linked in: nfs lockd nfs_acl sunrpc openafs(P)

autofs4 ipv6 ac sbs sbshc video output dock battery container iptable_filter ip_tables x_tables parport_pc lp parport loop joydev iTCO_wdt iTCO_vendor_support evdev button i5000_edac psmouse serio_raw pcspkr shpchp pci_hotplug edac_core ext3 jbd mbcache sr_mod cdrom ata_generic usbhid hid ata_piix sg sd_mod ehci_hcd pata_acpi uhci_hcd libata bnx2 aacraid usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse

[53417.662067] Pid: 13039, comm: gball Tainted: P       
2.6.24-19acr2-generic #1
[53417.669219] RIP: 0010:[<ffffffff80243643>]  [<ffffffff80243643>]
__do_softirq+0xc3/0x150
[53417.677368] RSP: 0018:ffff8103314f3f20  EFLAGS: 00010297
[53417.682697] RAX: ffff810084a1b000 RBX: ffffffff805ba530 RCX:
0000000000000000
[53417.689843] RDX: ffff8103305811e0 RSI: 0000000000000282 RDI:
ffff810332ada580
[53417.696993] RBP: 0000000000000000 R08: ffff81032fad9f08 R09:
ffff810332382000
[53417.704144] R10: 0000000000000000 R11: ffffffff80316ec0 R12:
ffffffff8062b3d8
[53417.711294] R13: ffffffff8062b480 R14: 0000000000000002 R15:
000000000000000a
[53417.718447] FS:  00007fab0d7b8750(0000) GS:ffff810334401b80(0000)
knlGS:0000000000000000
[53417.726568] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[53417.732332] CR2: 0000000000000000 CR3: 0000000329e2d000 CR4:
00000000000006e0
[53417.739476] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[53417.746637] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[53417.753787] Process gball (pid: 13039, threadinfo ffff81032adde000,
task ffff810329ff77d0)
[53417.761991] Stack:  ffffffff8062b3d8 0000000000000046
ffff8103314f3f68 0000000000000000
[53417.770146]  00000000000000a0 ffff81032addfee8 0000000000000000
ffffffff8020d50c
[53417.777660]  ffff8103314f3f68 00000000000000c1 ffffffff8020ed25
ffffffff8062c870
[53417.784961] Call Trace:
[53417.787635]  <IRQ>  [<ffffffff8020d50c>] call_softirq+0x1c/0x30
[53417.793597]  [<ffffffff8020ed25>] do_softirq+0x35/0x90
[53417.798747]  [<ffffffff80243578>] irq_exit+0x88/0x90
[53417.803727]  [<ffffffff8020ef70>] do_IRQ+0x80/0x100
[53417.808624]  [<ffffffff8020c891>] ret_from_intr+0x0/0xa
[53417.813862]  <EOI>  [<ffffffff803e53c8>] skb_release_all+0x18/0x150
[53417.820164]  [<ffffffff803e4ad9>] __kfree_skb+0x9/0x90
[53417.825327]  [<ffffffff80437612>] udp_recvmsg+0x222/0x260
[53417.830744]  [<ffffffff80231264>] source_load+0x34/0x70
[53417.835984]  [<ffffffff80232a9a>] find_busiest_group+0x1fa/0x850
[53417.842019]  [<ffffffff803e0100>] sock_common_recvmsg+0x30/0x50
[53417.847958]  [<ffffffff803de1ca>] sock_recvmsg+0x14a/0x160
[53417.853462]  [<ffffffff80231c21>] update_curr+0x71/0x100
[53419.858789]  [<ffffffff802320fd>] __dequeue_entity+0x3d/0x50
[53417.864469]  [<ffffffff80253ab0>] autoremove_wake_function+0x0/0x30
[53417.870758]  [<ffffffff8046662f>] thread_return+0x3a/0x57b
[53417.876262]  [<ffffffff803df73e>] sys_recvfrom+0xfe/0x190
[53417.881680]  [<ffffffff802e2a95>] sys_epoll_wait+0x245/0x4e0
[53417.887358]  [<ffffffff80233e20>] default_wake_function+0x0/0x10
[53417.893384]  [<ffffffff8020c37e>] system_call+0x7e/0x83
[53417.898628]
[53417.900134]
[53417.900134] Code: 48 8b 11 48 89 cf 65 48 8b 04 25 08 00 00 00 4a 89
14 20 ff
[53417.909430] RIP  [<ffffffff80243643>] __do_softirq+0xc3/0x150
[53417.915210]  RSP <ffff8103314f3f20>

The disassembly where it crashed:
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:273
ffffffff8024361b: d1 ed shr %ebp
rcu_bh_qsctr_inc():
/local/home/bmb/doc/kernels/linux-hardy-eric/include/linux/rcupdate.h:130

ffffffff8024361d:       48 8b 40 08             mov    0x8(%rax),%rax
ffffffff80243621:       41 c7 44 05 08 01 00    movl  

$0x1,0x8(%r13,%rax,1)
ffffffff80243628: 00 00
__do_softirq():
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:273
ffffffff8024362a: 75 d8 jne ffffffff80243604
<__do_softirq+0x84>
softirq_delay_exec():
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:225

ffffffff8024362c:       48 8b 14 24             mov    (%rsp),%rdx
ffffffff80243630:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
ffffffff80243637:       00 00
ffffffff80243639:       48 8b 0c 10             mov    (%rax,%rdx,1),%rcx
ffffffff8024363d:       48 83 f9 01             cmp    $0x1,%rcx
ffffffff80243641:       74 29                   je     ffffffff8024366c

<__do_softirq+0xec> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:226
ffffffff80243643: 48 8b 11 mov (%rcx),%rdx
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:227
ffffffff80243646: 48 89 cf mov %rcx,%rdi
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:226

ffffffff80243649:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
ffffffff80243650:       00 00
ffffffff80243652:       4a 89 14 20             mov    %rdx,(%rax,%r12,1)
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:227
ffffffff80243656:       ff 51 08                callq  *0x8(%rcx)
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:225
ffffffff80243659:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
ffffffff80243660:       00 00
ffffffff80243662:       4a 8b 0c 20             mov    (%rax,%r12,1),%rcx
ffffffff80243666:       48 83 f9 01             cmp    $0x1,%rcx
ffffffff8024366a:       75 d7                   jne    ffffffff80243643

<__do_softirq+0xc3>
raw_local_irq_disable():
/local/home/bmb/doc/kernels/linux-hardy-eric/debian/build/build-generic/include2/asm/irqflags_64.h:76

ffffffff8024366c: fa cli

And softirq.c line numbers:
218 * Because locking is provided by subsystem, please note
219 * that sdel->func(sdel) is responsible for setting sdel->next
to NULL
220 */
221 static void softirq_delay_exec(void)

   222  {
   223          struct softirq_delay *sdel;
   224
   225          while ((sdel = __get_cpu_var(softirq_delay_head)) !=
SOFTIRQ_DELAY_END) {
   226                  __get_cpu_var(softirq_delay_head) = sdel->next;
   227                  sdel->func(sdel);       /*      sdel->next =

NULL;*/
228 }
229 }

So it's crashing because __get_cpu_var(softirq_delay_head)) is NULL somehow.

We aren't running a recent kernel -- we're running Ubuntu Hardy's 2.6.24-19,
with a backported version of this patch. One more atypical thing is that we run openafs, 1.4.6.dfsg1-2.

Quoted 6 lines

Like I said, I have a full vmcore (3, actually) and would be happy to post any
more information you'd like to know.

Thanks,
Brian Bloniarz

Hi Brian

2.6.24-19 kernel... hmm...

Could you please send me the diff of your backport against this kernel ?

I take you use Ubuntu Hardys 8.04 LTS server edition ?

Pointer being null might tell us that we managed to call inet_def_readable() without socket lock hold...

#53 lore raw
Quoted 2 lines · Eric Dumazet

Pointer being null might tell us that we managed to call inet_def_readable() without socket lock hold...

Trying to track this down: I added:

	BUG_ON(!spin_is_locked(&sk->sk_lock.slock));

to the top of inet_def_readable. This gives me the following panic:

[ 2528.745311] kernel BUG at net/core/sock.c:1674!
[ 2528.745311] invalid opcode: 0000 [#1] PREEMPT SMP
[ 2528.745311] last sysfs file: /sys/devices/system/cpu/cpu7/crash_notes
[ 2528.745311] CPU 6
[ 2528.745311] Modules linked in: iptable_filter ip_tables x_tables parport_pc lp parport loop iTCO_wdt iTCO_vendor_support serio_raw psmouse pcspkr i5k_amb shpchp i5000_edac pci_hotplug button edac_core ipv6 ibmpex joydev ipmi_msghandler evdev ext3 jbd mbcache usbhid hid sr_mod cdrom pata_acpi ata_generic sg sd_mod ata_piix ehci_hcd uhci_hcd libata aacraid usbcore scsi_mod bnx2 thermal processor fan thermal_sys fuse
[ 2528.745311] Pid: 14507, comm: signalgen Not tainted 2.6.29.1-eric2-lowlat-lockdep #3 IBM System x3550 -[7978AC1]-
[ 2528.745311] RIP: 0010:[<ffffffff80444ec2>]  [<ffffffff80444ec2>] inet_def_readable+0x52/0x60
[ 2528.745311] RSP: 0018:ffff88043b985b58  EFLAGS: 00010246
[ 2528.745311] RAX: 0000000000000019 RBX: ffff88043b90c280 RCX: 0000000000000000
[ 2528.745311] RDX: 0000000000001919 RSI: 0000000000000068 RDI: ffff88043b90c280
[ 2528.745311] RBP: ffff88043b985b68 R08: 0000000000000000 R09: 0000000000000000
[ 2528.745311] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88043b811400
[ 2528.745311] R13: 0000000000000000 R14: 0000000000000068 R15: 0000000000000000
[ 2528.745311] FS:  00007f82f0742750(0000) GS:ffff88043dbc8280(0000) knlGS:0000000000000000
[ 2528.745311] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2528.745311] CR2: 000000000057f1a0 CR3: 000000043915e000 CR4: 00000000000406e0
[ 2528.745311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2528.745311] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 2528.745311] Process signalgen (pid: 14507, threadinfo ffff88043b984000, task ffff8804309a9ef0)
[ 2528.745311] Stack:
[ 2528.745311]  ffff88043b811400 ffff88043b90c280 ffff88043b985b98 ffffffff80444ff6
[ 2528.745311]  ffff88043b90c280 ffff88043b811400 0000000000000000 ffff88043b90c2c0
[ 2528.745311]  ffff88043b985bc8 ffffffff8049ee67 ffff88043b985bc8 ffff88043b811400
[ 2528.745311] Call Trace:
[ 2528.745311]  [<ffffffff80444ff6>] sock_queue_rcv_skb+0xd6/0x120
[ 2528.745311]  [<ffffffff8049ee67>] __udp_queue_rcv_skb+0x27/0xe0
[ 2528.745311]  [<ffffffff8044406a>] release_sock+0x7a/0xe0
[ 2528.745311]  [<ffffffff804a1d0d>] udp_recvmsg+0x1ed/0x330
[ 2528.745311]  [<ffffffff804437e2>] sock_common_recvmsg+0x32/0x50
[ 2528.745311]  [<ffffffff80441449>] sock_recvmsg+0x139/0x150
[ 2528.745311]  [<ffffffff8025a590>] ? autoremove_wake_function+0x0/0x40
[ 2528.745311]  [<ffffffff8026c4d9>] ? validate_chain+0x469/0x1270
[ 2528.745311]  [<ffffffff8026d60e>] ? __lock_acquire+0x32e/0xa40
[ 2528.745311]  [<ffffffff804429df>] sys_recvfrom+0xaf/0x110
[ 2528.745311]  [<ffffffff804e6109>] ? mutex_unlock+0x9/0x10
[ 2528.745311]  [<ffffffff80310041>] ? sys_epoll_wait+0x4a1/0x510
[ 2528.745311]  [<ffffffff8020c55b>] system_call_fastpath+0x16/0x1b
[ 2528.745311] Code: 85 c0 7e 1b 48 8d bf 98 02 00 00 e8 29 34 e0 ff 85 c0 74 04 f0 ff 43 28 48 83 c4 08 5b c9 c3 e8 15 f3 ff ff 48 83 c4 08 5b c9 c3 <0f> 0b eb fe 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec
[ 2528.745311] RIP  [<ffffffff80444ec2>] inet_def_readable+0x52/0x60
[ 2528.745311]  RSP <ffff88043b985b58>

Looks to me like __release_sock will call sk_backlog_rcv() with the socket unlocked -- does that help at all?

Thanks,
Brian Bloniarz

#54 lore raw
Quoted 1 line · Brian Bloniarz
    BUG_ON(!spin_is_locked(&sk->sk_lock.slock));

Oh, sorry, I think I'm just misunderstanding how the socket lock works. This doesn't actually check that the socket is locked, right?

-Brian

#55 lore raw
In reply to Eric Dumazet
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Brian Bloniarz

We've been experimenting with this softirq-delay patch in production, and have seen some hard-to-reproduce crashes. We finally managed to capture a kexec crashdump this morning.

Pointer being null might tell us that we managed to call inet_def_readable() without socket lock hold...

False alarm -- I think I did the backport to 2.6.24 incorrectly. 2.6.24 was before the UDP receive path started taking the socket lock, so inet_def_readable's assumption doesn't hold.

Sorry to waste everyone's time.

Thanks,
Brian Bloniarz

#56 lore raw
Quoted 9 lines · Brian Bloniarz
Quoted 2 lines · Eric Dumazet
Quoted 1 line · Brian Bloniarz

We've been experimenting with this softirq-delay patch in production,

and

Quoted 1 line
Quoted 1 line

have seen some hard-to-reproduce crashes. We finally managed to

capture a

Quoted 3 lines
Quoted 1 line

kexec crashdump this morning.

Pointer being null might tell us that we managed to call

Quoted 9 lines

inet_def_readable()

Quoted 1 line

without socket lock hold...

False alarm -- I think I did the backport to 2.6.24 incorrectly. 2.6.24 was before the UDP receive path started taking the socket lock, so inet_def_readable's assumption doesn't hold.

Sorry to waste everyone's time.

Thanks for doing this discovery work and analysis.

I am currently off-computers and could not do this until next week.

So, if you want to use 2.6.24, we need to back port other patches as well ?

#57 lore raw
In reply to Eric Dumazet
Quoted 1 line · Eric Dumazet

Here is a patch that helps. It's still an RFC of course, since its somewhat ugly :)

Hi Eric,

I did some experimenting with this patch today -- we're users, not kernel hackers, but the performance looks great. We see no loss with mcasttest, and no loss with our internal test programs (which do much more user-space work). We're very encouraged :)

One thing I'm curious about: previously, setting /proc/irq/<eth0>/smp_affinity to one CPU made things perform better, but with this patch, performance is better with smp_affinity == ff than with smp_affinity == 1. Do you know why that is? Our tests are all with bnx2 msi_disable=1. I can investigate with oprofile tomorrow.

Thank you for your continued help, we all deeply appreciate having someone looking at this workload.

Thanks,
Brian Bloniarz

#58 lore raw
Quoted 13 lines · Brian Bloniarz
Quoted 2 lines · Eric Dumazet

Here is a patch that helps. It's still an RFC of course, since its somewhat ugly :)

Hi Eric,

I did some experimenting with this patch today -- we're users, not kernel hackers,
but the performance looks great. We see no loss with mcasttest, and no loss with
our internal test programs (which do much more user-space work). We're very encouraged :)

Quoted 9 lines

One thing I'm curious about: previously, setting /proc/irq/<eth0>/smp_affinity
to one CPU made things perform better, but with this patch, performance is better
with smp_affinity == ff than with smp_affinity == 1. Do you know why that is? Our tests are all with bnx2 msi_disable=1. I can investigate with oprofile
tomorrow.

Well, smp_affinity could help in my opininon if you dedicate one cpu for the NIC, and others for user apps, if the average work done per packet is large. If load is light, its better to use the same cpu to perform all the work, since no expensive bus trafic is needed between cpu to exchange memory lines.

If you only change /proc/irq/<eth0>/smp_affinity, and let scheduler chose any cpu for your user-space work that can have long latencies, I would not expect better performances.

Try to cpu affine your taks to 0xFE to get better determinism.

#59 lore raw

Hi Eric,

FYI: with your patch applied and lockdep enabled, I see:

[   39.114628] ================================================
[   39.121964] [ BUG: lock held when returning to user space! ]
[   39.127704] ------------------------------------------------
[   39.133461] msgtest/5242 is leaving the kernel with locks still held!
[   39.140132] 1 lock held by msgtest/5242:
[   39.144287]  #0:  (clock-AF_INET){-.-?}, at: [<ffffffff8041f5b9>] sock_def_readable+0x19/0xb0

I can't reproduced this with the mcasttest program yet, it was with an internal test program which does some userspace processing on the messages. I'll let you know if I find a way to reproduce it with a simple program I can share.

Quoted 5 lines

Well, smp_affinity could help in my opininon if you dedicate one cpu for the NIC, and others for user apps, if the average work done per packet is large. If load is light, its better to use the same cpu to perform all the work, since no expensive bus trafic is needed between cpu to exchange memory lines.

I tried this setup as well: an 8-core box with 4 userspace processes, each affined to an individual CPU1-4. The IRQ was on CPU0. On most kernels, this setup loses fewer packets than the default affinity (though they both lose some). With your patch enabled, the default affinity loses 0 packets, and this setup loses some.

Thanks,
Brian Bloniarz

#60 lore raw
Quoted 10 lines · Brian Bloniarz

Hi Eric,

FYI: with your patch applied and lockdep enabled, I see:

[   39.114628] ================================================
[   39.121964] [ BUG: lock held when returning to user space! ]
[   39.127704] ------------------------------------------------
[   39.133461] msgtest/5242 is leaving the kernel with locks still held!
[   39.140132] 1 lock held by msgtest/5242:
[   39.144287]  #0:  (clock-AF_INET){-.-?}, at: [<ffffffff8041f5b9>]

sock_def_readable+0x19/0xb0

And you told me you were not a kernel hacker ;)

Quoted 5 lines

I can't reproduced this with the mcasttest program yet, it was with an internal test program which does some userspace processing on the messages. I'll let you know if I find a way to reproduce it with a simple program I can share.

I reproduced it as well here quite easily with a tcpdump of a tcp session, thanks for the report.

It seems "if (in_softirq())" doesnt do what I thought.

I wanted to test if we were called from __do_softirq() handler, since only this function is calling softirq_delay_exec() to dequeue events.

It appears I have to make current->softirq_context available even if !CONFIG_TRACE_IRQFLAGS

Here is an updated version of the patch.

I also made the call to softirq_delay_exec() is performed with interrupts enabled, and that preempt count wont overflow if many events are queued.

[PATCH] softirq: Introduce mechanism to defer wakeups

Some network workloads need to call scheduler too many times. For example, each received multicast frame can wakeup many threads. ksoftirqd is then not able to drain NIC RX queues and we get frame losses and high latencies.

This patch adds an infrastructure to delay part of work done in sock_def_readable() at end of do_softirq(). This need to make available current->softirq_context even if !CONFIG_TRACE_IRQFLAGS

---
 include/linux/interrupt.h |   18 +++++++++++++++++
 include/linux/irqflags.h  |    7 ++----
 include/linux/sched.h     |    2 -
 include/net/sock.h        |    1
 kernel/softirq.c          |   34 +++++++++++++++++++++++++++++++++
 net/core/sock.c           |   37 ++++++++++++++++++++++++++++++++++--
 6 files changed, 92 insertions(+), 7 deletions(-)
include/linux/interrupt.h +18 -0
index 9127f6b..a773d0c 100644
@@ -295,6 +295,24 @@ extern void send_remote_softirq(struct call_single_data *cp, int cpu, int softir
 extern void __send_remote_softirq(struct call_single_data *cp, int cpu,
 				  int this_cpu, int softirq);
 
+/*
+ * softirq delayed works : should be delayed at do_softirq() end
+ */
+struct softirq_delay {
+	struct softirq_delay	*next;
+	void 			(*func)(struct softirq_delay *);
+};
+
+int softirq_delay_queue(struct softirq_delay *sdel);
+
+static inline void softirq_delay_init(struct softirq_delay *sdel,
+				      void (*func)(struct softirq_delay *))
+{
+	sdel->next = NULL;
+	sdel->func = func;
+}
+
+
 /* Tasklets --- multithreaded analogue of BHs.
 
    Main feature differing them of generic softirqs: tasklet
include/linux/irqflags.h +3 -4
index 74bde13..fe55ec4 100644
@@ -13,6 +13,9 @@
 
 #include <linux/typecheck.h>
 
+#define trace_softirq_enter()	do { current->softirq_context++; } while (0)
+#define trace_softirq_exit()	do { current->softirq_context--; } while (0)
+
 #ifdef CONFIG_TRACE_IRQFLAGS
   extern void trace_softirqs_on(unsigned long ip);
   extern void trace_softirqs_off(unsigned long ip);
@@ -24,8 +27,6 @@
 # define trace_softirqs_enabled(p)	((p)->softirqs_enabled)
 # define trace_hardirq_enter()	do { current->hardirq_context++; } while (0)
 # define trace_hardirq_exit()	do { current->hardirq_context--; } while (0)
-# define trace_softirq_enter()	do { current->softirq_context++; } while (0)
-# define trace_softirq_exit()	do { current->softirq_context--; } while (0)
 # define INIT_TRACE_IRQFLAGS	.softirqs_enabled = 1,
 #else
 # define trace_hardirqs_on()		do { } while (0)
@@ -38,8 +39,6 @@
 # define trace_softirqs_enabled(p)	0
 # define trace_hardirq_enter()		do { } while (0)
 # define trace_hardirq_exit()		do { } while (0)
-# define trace_softirq_enter()		do { } while (0)
-# define trace_softirq_exit()		do { } while (0)
 # define INIT_TRACE_IRQFLAGS
 #endif
 
include/linux/sched.h +1 -1
index 8c216e0..5dd8487 100644
@@ -1320,8 +1320,8 @@ struct task_struct {
 	unsigned long softirq_enable_ip;
 	unsigned int softirq_enable_event;
 	int hardirq_context;
-	int softirq_context;
 #endif
+	int softirq_context;
 #ifdef CONFIG_LOCKDEP
 # define MAX_LOCK_DEPTH 48UL
 	u64 curr_chain_key;
include/net/sock.h +1 -0
index eefeeaf..1bfd9b8 100644
@@ -260,6 +260,7 @@ struct sock {
 	unsigned long	        sk_lingertime;
 	struct sk_buff_head	sk_error_queue;
 	struct proto		*sk_prot_creator;
+	struct softirq_delay	sk_delay;
 	rwlock_t		sk_callback_lock;
 	int			sk_err,
 				sk_err_soft;
kernel/softirq.c +34 -0
index 9041ea7..c601730 100644
@@ -158,6 +158,38 @@ void local_bh_enable_ip(unsigned long ip)
 }
 EXPORT_SYMBOL(local_bh_enable_ip);
 
+
+#define SOFTIRQ_DELAY_END (struct softirq_delay *)1L
+
+static DEFINE_PER_CPU(struct softirq_delay *, softirq_delay_head) = {
+	SOFTIRQ_DELAY_END
+};
+
+/*
+ * Preemption is disabled by caller
+ */
+int softirq_delay_queue(struct softirq_delay *sdel)
+{
+	if (cmpxchg(&sdel->next, NULL, __get_cpu_var(softirq_delay_head)) == NULL) {
+		__get_cpu_var(softirq_delay_head) = sdel;
+		return 1;
+	}
+	return 0;
+}
+
+static void softirq_delay_exec(void)
+{
+	struct softirq_delay *sdel;
+
+	while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
+		__get_cpu_var(softirq_delay_head) = sdel->next;
+		sdel->next = NULL;
+		sdel->func(sdel);
+		}
+}
+
+
+
 /*
  * We restart softirq processing MAX_SOFTIRQ_RESTART times,
  * and we fall back to softirqd after that.
@@ -211,6 +243,8 @@ restart:
 		pending >>= 1;
 	} while (pending);
 
+	softirq_delay_exec();
+
 	local_irq_disable();
 
 	pending = local_softirq_pending();
net/core/sock.c +35 -2
index 5f97caa..d51d57d 100644
@@ -212,6 +212,8 @@ __u32 sysctl_rmem_default __read_mostly = SK_RMEM_MAX;
 /* Maximal space eaten by iovec or ancilliary data plus some space */
 int sysctl_optmem_max __read_mostly = sizeof(unsigned long)*(2*UIO_MAXIOV+512);
 
+static void sock_readable_defer(struct softirq_delay *sdel);
+
 static int sock_set_timeout(long *timeo_p, char __user *optval, int optlen)
 {
 	struct timeval tv;
@@ -1026,6 +1028,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority)
 #endif
 
 		rwlock_init(&newsk->sk_dst_lock);
+		softirq_delay_init(&newsk->sk_delay, sock_readable_defer);
 		rwlock_init(&newsk->sk_callback_lock);
 		lockdep_set_class_and_name(&newsk->sk_callback_lock,
 				af_callback_keys + newsk->sk_family,
@@ -1634,12 +1637,41 @@ static void sock_def_error_report(struct sock *sk)
 	read_unlock(&sk->sk_callback_lock);
 }
 
+static void sock_readable_defer(struct softirq_delay *sdel)
+{
+	struct sock *sk = container_of(sdel, struct sock, sk_delay);
+
+	wake_up_interruptible_sync(sk->sk_sleep);
+	/*
+	 * Before unlocking, we increase preempt_count,
+	 * as it was decreased in sock_def_readable()
+	 */
+	preempt_disable();
+	read_unlock(&sk->sk_callback_lock);
+}
+
 static void sock_def_readable(struct sock *sk, int len)
 {
 	read_lock(&sk->sk_callback_lock);
-	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep))
-		wake_up_interruptible_sync(sk->sk_sleep);
 	sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
+	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) {
+		if (current->softirq_context) {
+			/*
+			 * If called from __do_softirq(), we want to delay
+			 * calls to wake_up_interruptible_sync()
+			 */
+			if (!softirq_delay_queue(&sk->sk_delay))
+				goto unlock;
+			/*
+			 * We keep sk->sk_callback_lock read locked,
+			 * but decrease preempt_count to avoid an overflow
+			 */
+			preempt_enable_no_resched();
+			return;
+		}
+		wake_up_interruptible_sync(sk->sk_sleep);
+	}
+unlock:
 	read_unlock(&sk->sk_callback_lock);
 }
 
@@ -1720,6 +1752,7 @@ void sock_init_data(struct socket *sock, struct sock *sk)
 		sk->sk_sleep	=	NULL;
 
 	rwlock_init(&sk->sk_dst_lock);
+	softirq_delay_init(&sk->sk_delay, sock_readable_defer);
 	rwlock_init(&sk->sk_callback_lock);
 	lockdep_set_class_and_name(&sk->sk_callback_lock,
 			af_callback_keys + sk->sk_family,
#61 lore raw
Quoted 1 line · Eric Dumazet

Here is an updated version of the patch.

This works great in all my tests so far.

Thanks,
Brian Bloniarz

#62 lore raw
Quoted 7 lines · Brian Bloniarz
Quoted 1 line · Eric Dumazet

Here is an updated version of the patch.

This works great in all my tests so far.

Thanks,
Brian Bloniarz

Cool

I am wondering if we should extend the mechanism and change softirq_delay_exec() to wakeup a workqueue instead of doing the loop from softirq handler, in case a given level of stress / load is hit.

This could help machines with several cpus, and one NIC (without multi RX queues) flooded by messages (not necessarly multicast trafic). Imagine a media/chat server receiving XXX.000 frames / second

One cpu could be dedicated to pure softirq/network handling, and other cpus could participate and handle the scheduler part if any.

Condition could be :

- We run __do_softirq() from ksoftirqd and
- We queued more than N 'struct softirq_delay' in softirq_delay_head
- We have more than one cpu online

#64 lore raw
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Eric Dumazet

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)

When will that happen?

When proven necessary :)

Kenny problem is about scheduling storm. The extra spin_lock() in UDP multicast receive is not a problem.

#65 lore raw
Quoted 9 lines · Eric Dumazet

Christoph Lameter a ?crit :

Quoted 4 lines · Eric Dumazet
Quoted 2 lines

2.6.29-rc contains UDP receive improvements (lockless) Problem is multicast handling was not yet updated, but could be :)

When will that happen?

When proven necessary :)

Kenny problem is about scheduling storm. The extra spin_lock() in UDP multicast receive is not a problem.

My tests here show that 2.6.29-rc5 still looses ~5usec vs. 2.6.22 via UDP. This would fix a regression.....

#66 lore raw
Quoted 6 lines · Christoph Lameter
Quoted 4 lines · Eric Dumazet

Christoph Lameter a ?crit :

Quoted 3 lines · Eric Dumazet
Quoted 2 lines

2.6.29-rc contains UDP receive improvements (lockless) Problem is multicast handling was not yet updated, but could be :)

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

When will that happen?

When proven necessary :)

Kenny problem is about scheduling storm. The extra spin_lock() in UDP multicast receive is not a problem.

My tests here show that 2.6.29-rc5 still looses ~5usec vs. 2.6.22 via UDP. This would fix a regression.....

Could you elaborate ?

I just retried Kenny test here. As one cpu is looping in ksoftirqd, only this cpu touches the spin_lock, so spin_lock()/spin_unlock() is free.

oprofile shows that udp stack is lightweight in this case. Problem is about wakeing up so many threads...

CPU: Core 2, speed 3000.16 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  cum. samples  %        cum. %     symbol name
356857   356857        15.1789  15.1789    schedule
274028   630885        11.6557  26.8346    mwait_idle
189218   820103         8.0484  34.8829    __skb_recv_datagram
116903   937006         4.9725  39.8554    skb_release_data
103152   1040158        4.3876  44.2430    lock_sock_nested
89600    1129758        3.8111  48.0541    udp_recvmsg
74171    1203929        3.1549  51.2089    copy_to_user
72299    1276228        3.0752  54.2842    set_next_entity
60392    1336620        2.5688  56.8529    sched_clock_cpu
54026    1390646        2.2980  59.1509    __slab_free
50212    1440858        2.1358  61.2867    prepare_to_wait_exclusive
38689    1479547        1.6456  62.9323    cpu_idle
38142    1517689        1.6224  64.5547    __switch_to
36701    1554390        1.5611  66.1157    hrtick_start_fair
36673    1591063        1.5599  67.6756    dst_release
36268    1627331        1.5427  69.2183    sys_recvfrom
35052    1662383        1.4909  70.7092    kmem_cache_free
32680    1695063        1.3900  72.0992    pick_next_task_fair
31209    1726272        1.3275  73.4267    try_to_wake_up
30382    1756654        1.2923  74.7190    dequeue_task_fair
29048    1785702        1.2356  75.9545    __copy_skb_header
28801    1814503        1.2250  77.1796    sock_def_readable
28655    1843158        1.2188  78.3984    enqueue_task_fair
27232    1870390        1.1583  79.5567    update_curr
21688    1892078        0.9225  80.4792    copy_from_user
18832    1910910        0.8010  81.2802    sysenter_past_esp
17732    1928642        0.7542  82.0345    finish_task_switch
17583    1946225        0.7479  82.7824    resched_task
17367    1963592        0.7387  83.5211    native_sched_clock
15691    1979283        0.6674  84.1885    task_rq_lock
15352    1994635        0.6530  84.8415    sock_queue_rcv_skb
15022    2009657        0.6390  85.4804    udp_queue_rcv_skb
13999    2023656        0.5954  86.0759    __update_sched_clock
12284    2035940        0.5225  86.5984    skb_copy_datagram_iovec
11869    2047809        0.5048  87.1032    release_sock
10986    2058795        0.4673  87.5705    __wake_up_sync
10488    2069283        0.4461  88.0166    sock_recvmsg
9686     2078969        0.4120  88.4286    skb_queue_tail
9425     2088394        0.4009  88.8295    sys_socketcall
#67 lore raw
Quoted 58 lines · Eric Dumazet
Quoted 6 lines · Christoph Lameter
Quoted 4 lines · Eric Dumazet

Christoph Lameter a ?crit :

Quoted 3 lines · Eric Dumazet
Quoted 2 lines

2.6.29-rc contains UDP receive improvements (lockless) Problem is multicast handling was not yet updated, but could be :)

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

When will that happen?

When proven necessary :)

Kenny problem is about scheduling storm. The extra spin_lock() in UDP multicast receive is not a problem.

My tests here show that 2.6.29-rc5 still looses ~5usec vs. 2.6.22 via UDP. This would fix a regression.....

Could you elaborate ?

I just retried Kenny test here. As one cpu is looping in ksoftirqd, only this cpu touches the spin_lock, so spin_lock()/spin_unlock() is free.

oprofile shows that udp stack is lightweight in this case. Problem is about wakeing up so many threads...

CPU: Core 2, speed 3000.16 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  cum. samples  %        cum. %     symbol name
356857   356857        15.1789  15.1789    schedule
274028   630885        11.6557  26.8346    mwait_idle
189218   820103         8.0484  34.8829    __skb_recv_datagram
116903   937006         4.9725  39.8554    skb_release_data
103152   1040158        4.3876  44.2430    lock_sock_nested
89600    1129758        3.8111  48.0541    udp_recvmsg
74171    1203929        3.1549  51.2089    copy_to_user
72299    1276228        3.0752  54.2842    set_next_entity
60392    1336620        2.5688  56.8529    sched_clock_cpu
54026    1390646        2.2980  59.1509    __slab_free
50212    1440858        2.1358  61.2867    prepare_to_wait_exclusive
38689    1479547        1.6456  62.9323    cpu_idle
38142    1517689        1.6224  64.5547    __switch_to
36701    1554390        1.5611  66.1157    hrtick_start_fair
36673    1591063        1.5599  67.6756    dst_release
36268    1627331        1.5427  69.2183    sys_recvfrom
35052    1662383        1.4909  70.7092    kmem_cache_free
32680    1695063        1.3900  72.0992    pick_next_task_fair
31209    1726272        1.3275  73.4267    try_to_wake_up
30382    1756654        1.2923  74.7190    dequeue_task_fair
29048    1785702        1.2356  75.9545    __copy_skb_header
28801    1814503        1.2250  77.1796    sock_def_readable
28655    1843158        1.2188  78.3984    enqueue_task_fair
27232    1870390        1.1583  79.5567    update_curr
21688    1892078        0.9225  80.4792    copy_from_user
18832    1910910        0.8010  81.2802    sysenter_past_esp
17732    1928642        0.7542  82.0345    finish_task_switch
17583    1946225        0.7479  82.7824    resched_task
17367    1963592        0.7387  83.5211    native_sched_clock
15691    1979283        0.6674  84.1885    task_rq_lock
15352    1994635        0.6530  84.8415    sock_queue_rcv_skb
Quoted 10 lines
15022    2009657        0.6390  85.4804    udp_queue_rcv_skb
13999    2023656        0.5954  86.0759    __update_sched_clock
12284    2035940        0.5225  86.5984    skb_copy_datagram_iovec
11869    2047809        0.5048  87.1032    release_sock
10986    2058795        0.4673  87.5705    __wake_up_sync
10488    2069283        0.4461  88.0166    sock_recvmsg
9686     2078969        0.4120  88.4286    skb_queue_tail
9425     2088394        0.4009  88.8295    sys_socketcall

My guess is commit 95766fff6b9a78d11fc2d3812dd035381690b55d (UDP: Add memory accounting)
Hideo Aoki [Mon, 31 Dec 2007 08:29:24 +0000 (00:29 -0800)]

and 3ab224be6d69de912ee21302745ea45a99274dbc [NET] CORE: Introducing new memory accounting interface.
Date: Mon Dec 31 00:11:19 2007 -0800

are responsible for slowdown, because they add some lock_sock()/release_sock() pairs.

function udp_recvmsg()

out_free:

+       lock_sock(sk);
        skb_free_datagram(sk, skb);

+ release_sock(sk);
out:

I wonder why we can call __sk_mem_reclaim() when dequeing *one* UDP frame in queue, while many others can still be in sk_receive_queue. This defeats memory accounting, no ?

We should avoid lock_sock() if possible, or risk delaying softirq RX in udp_queue_rcv_skb()

#68 lore raw
In reply to Kenny Chang
Quoted 10 lines · Kenny Chang

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we see where the performance drops significantly. We'll try 2.6.29-rc soon and post the result.

Neil Norman wrote:

On latest kernels, we have a "timer_slack_ns" default of 50.000 ns, aka 50us

So usleep(20) sleeps much more than expected.

You might add in your program a call to prcrl() to setup a smaller timer_slack :

#ifndef PR_SET_TIMERSLACK
#define PR_SET_TIMERSLACK 29
#endif
/*
* Setup a timer resolution of 1000 ns : 1 us
*/
prctl(PR_SET_TIMERSLACK, 1000);