PROBLEM: select() on TCP socket sleeps for 1 tick even if data available

DISCUSSION Michael Lindner 5 participants 8 messages Jan 19, 2001 – Jan 20, 2001

Share & tools

Michael Lindner

author

2001-01-19 20:42 UTC #0 lore raw

[1.] select() sleeps for 1 tick even if data available
[2.] Full description of the problem/report:
If select() is waiting for data to become available on a TCP socket FD,
and
data becomes available, it doesn't return until the next clock tick.
This
produces large latencies when passing data between processes several
times,
since each transaction (which requires microseconds) does not occur
until
the next clock tick, limiting the entire throughput of the system to
100
transactions/second.
[3.] Keywords: select, socket, networking
[4.] Kernel version (from /proc/version): 2.2.18
[5.] Output of Oops.. message (not applicable)
[6.] A small shell script or example program which triggers the

     problem (if possible)

	#include <sys/time.h>
	#include <sys/types.h>
	#include <unistd.h>

	/* this program should take 1 second to complete, but takes 10 */
	/* yes i know, it doesn't do any I/O, but the behavior is the
		same as if it did */
	main()
	{
	        for (int i = 0; i < 1000; i++) {
			struct timeval to;
			to.tv_sec = 0;
			to.tv_usec = 1000;
			select(0, 0, 0, 0, &to);
		}
		return 0;
	}

[7.] Environment
Red Hat 7.0 with kernel upgraded to 2.2.18

[7.1.] Software (add the output of the ver_linux script here)
ver_linux di dnot appear to work. However, here's its output.

Linux mlindner-ras.sonusnet.com 2.2.18 #8 Wed Jan 3 01:40:29 EST 2001 i586 unknown

Kernel modules         found
Gnu C                  2.96
Binutils               2.10.0.18
Linux C Library        ..
ldd: missing file arguments
Try `ldd --help' for more information.
ls: /usr/lib/libg++.so: No such file or directory
Procps                 2.0.7
Mount                  2.10m
Net-tools              (2000-05-21)
Kbd                    [option...]
Sh-utils               2.0
Sh-utils               Parker.
Sh-utils               
Sh-utils               Inc.
Sh-utils               NO
Sh-utils               PURPOSE.

[7.2.] Processor information (from /proc/cpuinfo):

processor : 0
vendor_id : AuthenticAMD
cpu family : 5
model : 8
model name : AMD-K6(tm) 3D processor
stepping : 12
cpu MHz : 451.033
cache size : 64 KB
fdiv_bug : no
hlt_bug : no
sep_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr mce cx8 sep mtrr pge mmx 3dnow
bogomips : 897.84

[7.3.] Module information (from /proc/modules): N/A [7.4.] SCSI information (from /proc/scsi/scsi) N/A [7.5.] Other information that might be relevant to the problem
(please look in /proc and include all information that you think to be relevant):
[X.] Other notes, patches, fixes, workarounds:

--
Mike Lindner
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

David Schwartz

2001-01-19 23:20 UTC #1 lore raw

Quoted 4 lines

If select() is waiting for data to become available on a TCP socket FD, and
data becomes available, it doesn't return until the next clock tick.

If your application has scheduling requirements, you need to communicate
them to the scheduler.

Quoted 9 lines

	#include <sys/time.h>
	#include <sys/types.h>
	#include <unistd.h>

	/* this program should take 1 second to complete, but takes 10 */
	/* yes i know, it doesn't do any I/O, but the behavior is the
		same as if it did */
	main()
	{

Quoted 8 lines

	        for (int i = 0; i < 1000; i++) {
			struct timeval to;
			to.tv_sec = 0;
			to.tv_usec = 1000;
			select(0, 0, 0, 0, &to);
		}
		return 0;
	}

This program doesn't demonstrate anything except that Linux's sleep time is
granular. This shouldn't be news to anyone. If you don't force a reschedule, everything works the way it's supposed to:

main()
{
int i, j, pipes[2];
struct timeval tv;
fd_set rd;
pipe(pipes);
write(pipes[1], "foo", 4);
for(i=0; i<1000; i++)
{
FD_ZERO(&rd);
FD_SET(pipes[0], &rd);
tv.tv_sec=0;
tv.tv_usec=1000;
j=select(10, &rd, NULL, NULL, &tv);
if(j!=1) printf("oops\n");
}
}

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Michael Lindner

author

2001-01-20 02:26 UTC #2 lore raw

Thanks CW and DS for the prompt replies. However, although each addressed the (flawed) example I included, neither addressed the problem described in the text.

Quoted 3 lines · I wrote:

Quoted 3 lines
If select() is waiting for data to become available on a TCP socket FD, and
data becomes available, it doesn't return until the next clock tick.

Quoted 3 lines · David Schwartz

This program doesn't demonstrate anything except that Linux's sleep time is
granular. This shouldn't be news to anyone. If you don't force a reschedule, everything works the way it's supposed to:

The sample program you included doesn't show anything other than that select() doesn't sleep at all if there's already data available when select() starts. That was not my claim either.

My problem is that if data is NOT available when select() starts, but becomes available immediately afterwards, select() doesn't wake up immediately, but sleeps for 1/100 second. In other words, select doesn't wake up immediately when data becomes available, but on the next clock tick. This is not the experience I've had with any other OS I've used, and is a source of great latency in my application. Since I am passing data from one process to another, and that data is generated as a result of data received via a select(), the next delivery occurs a clock tick later, with the machine mostly idle.

It can be argued that there's no law governing the latency of select() waking up, and that my application is expecting too much. Yet, it runs on other UNIXes and Windows, and I'd like to be able to get the same high performance out of Linux.

P.S. Chris Wedgwood writes:
"The time passed to slect is a _minimum_ "

but the man page for select says:
"timeout is an upper bound on the amount of time elapsed
before select returns."

who is right?

--
Mike Lindner
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

David Schwartz

2001-01-20 03:27 UTC #3 lore raw

Quoted 17 lines

Thanks CW and DS for the prompt replies. However, although each addressed the (flawed) example I included, neither addressed the problem described in the text.
Quoted 3 lines · I wrote:
Quoted 3 lines
If select() is waiting for data to become available on a TCP socket FD, and
data becomes available, it doesn't return until the next clock tick.
Quoted 5 lines · David Schwartz
This program doesn't demonstrate anything except that Linux's sleep time is
granular. This shouldn't be news to anyone. If you don't force a reschedule,
everything works the way it's supposed to:

Quoted 4 lines

The sample program you included doesn't show anything other than that select() doesn't sleep at all if there's already data available when select() starts. That was not my claim either.

Correct. Select doesn't sleep unless it has to.

Quoted 3 lines

My problem is that if data is NOT available when select() starts, but becomes available immediately afterwards, select() doesn't wake up immediately, but sleeps for 1/100 second.

How can you tell when select wakes up the process? What you are seeing has
nothing whatsoever to do with select and simply has to do with the fact that the kernel does not give the CPU to a process the second that process may want it.

Quoted 2 lines

In other words, select doesn't wake up immediately when data becomes available, but on the next clock tick.

Right. The process becomes eligible to run -- it's no longer blocked. The
scheduler then schedules it whenever it feels like it.

Quoted 7 lines

This is
not the experience I've had with any other OS I've used, and is a source of great latency in my application. Since I am passing data from one process to another, and that data is generated as a result of data received via a select(), the next delivery occurs a clock tick later, with the machine mostly idle.

If you have scheduling latency requirements, you MUST communicate them to
the scheduler. If your process had an altered scheduling class, then you would be right -- it should get the CPU immediately. Otherwise, there's no reason for the scheduler to give that process the CPU immediately.

Quoted 4 lines

It can be argued that there's no law governing the latency of select() waking up, and that my application is expecting too much. Yet, it runs on other UNIXes and Windows, and I'd like to be able to get the same high performance out of Linux.

Then tell that to the scheduler.

Quoted 8 lines

P.S. Chris Wedgwood writes:
"The time passed to slect is a _minimum_ "
but the man page for select says:
"timeout is an upper bound on the amount of time elapsed
before select returns."
who is right?

This should read "before select returns the process to the list of runnable
processes".

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Michael Lindner

author

2001-01-20 04:33 UTC #4 lore raw

Quoted 5 lines · David Schwartz

How can you tell when select wakes up the process? What you are seeing has
nothing whatsoever to do with select and simply has to do with the fact that the kernel does not give the CPU to a process the second that process may want it.

I guess I can't. But on an idle machine, I would expect a process that becomes runnable would be run immediately, not on the next clock tick. strace reports that each select() is taking 0.009xxx seconds of real time, and the system's CPU load (as reported by top) is under 1%.

...

Quoted 4 lines

If you have scheduling latency requirements, you MUST communicate them to
the scheduler. If your process had an altered scheduling class, then you would be right -- it should get the CPU immediately. Otherwise, there's no reason for the scheduler to give that process the CPU immediately.

OK, if this is the case, how do I alter the scheduling class?

--
Mike Lindner
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Martin MaD Douda

2001-01-20 12:25 UTC #5 lore raw

In reply to Michael Lindner

Quoted 3 lines · Michael Lindner

data is generated as a result of data received via a select(), the next delivery occurs a clock tick later, with the machine mostly idle.

^^^^^^^^^^^

The machine is in fact not idle - there is a task running - idle task. Could the problem be that scheduler does not preempt this task to run something more useful?

Symptoms seems to show this.

Martin

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Bjorn Wesen

2001-01-20 12:59 UTC #6 lore raw

Quoted 8 lines · Martin MaD Douda

Quoted 3 lines · Michael Lindner
data is generated as a result of data received via a select(), the next delivery occurs a clock tick later, with the machine mostly idle.
The machine is in fact not idle - there is a task running - idle task. Could the problem be that scheduler does not preempt this task to run something more useful?

Normally, the "idle task" (task[0]) does this pseudo-code:

   while(1) { 
      if(need_resched)
         schedule();
   }

to minimize latency out of idle so if that actually is running it should not be a problem (unless need_resched is not set by the wakeup calls)

Perhaps the kapm-idled kernel thread is killing your latency, you could try disabling APM and APM-making-idle-calls especially. Also check ps aux and see if anything else is taking your idle CPU %.

-BW

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Chris Wedgwood

2001-01-19 23:32 UTC #7 lore raw

In reply to Michael Lindner

    	main()
    	{
    	        for (int i = 0; i < 1000; i++) {
    			struct timeval to;
    			to.tv_sec = 0;
    			to.tv_usec = 1000;
    			select(0, 0, 0, 0, &to);
    		}
    		return 0;
    	}

ia32 with HZ=100 means sleep will sleep for 10ms each time, no less. The time passed to slect is a _minimum_ the kernel make no guarantee it will be less, but assures it will be more (10ms is the default timer interval for the intel platform, others (e.g. alpha) differ)

run your conde with strace -t to see what i mean

--cw
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/