Skip to contentSkip to conversation

2.6 vs 2.4, ssh terminal slowdown

DISCUSSION Grant Coady 8 participants 41 messages Feb 8, 2006 – Feb 15, 2006

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi there,

grant@deltree:~$ uname -r
2.6.15.3a
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut -c-95
...
2006-02-08 12:38:13 +1100: bugsplatter.mine.nu 193.196.182.215 "GET /test/linux-2.6/tosh/ HTTP/
real    0m8.537s
user    0m0.970s
sys     0m1.100s

--> reboot to 2.4.32-hf32.2

grant@deltree:~$ uname -r
2.4.32-hf32.2
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut -c-95
...
2006-02-08 12:38:13 +1100: bugsplatter.mine.nu 193.196.182.215 "GET /test/linux-2.6/tosh/ HTTP/
real    0m2.271s
user    0m0.730s
sys     0m0.540s

Still a 4:1 slowdown, machine .config and dmesg info:
http://bugsplatter.mine.nu/test/boxen/deltree/

While it is very nice to know I can run recent 2.6 kernels without a fuss, 2.4.latest feels better in this particular context.

This console sluggishness is noticeable enough on older hardware for me to forgo exercising 2.6.latest.stable bugs for much time on it ;)

For those suffering deja vu, yes, I reported this last month (or, recently).

Grant.

#1 lore raw
Quoted 4 lines · Grant Coady

This console sluggishness is noticeable enough on older hardware for me to forgo exercising 2.6.latest.stable bugs for much time on it ;)

For those suffering deja vu, yes, I reported this last month (or, recently).

This bug report is a bit vague in terms of what the problem is -- the test case hits 3 major subsystems (io, vm, net), all of which have changed rather substantially in the course of 2.6 development. Would it be possible to profile the system using oprofile to get an idea what the hotspots are? Have you compared basic hard disk throughput with hdparm, as well as ensuring DMA is enabled with 32 bit io? What about testing network performance with netperf (or a netcat of /dev/zero)? A few more data points would be quite helpful. Cheers,

-ben

Signature · 2 lines
"Ladies and gentlemen, I'm sorry to interrupt, but the police are here 
and they've asked us to stop the party."  Don't Email: <[email protected]>.
#2 lore raw
Quoted 9 lines · Benjamin LaHaise
Quoted 4 lines · Grant Coady

This console sluggishness is noticeable enough on older hardware for me to forgo exercising 2.6.latest.stable bugs for much time on it ;)

For those suffering deja vu, yes, I reported this last month (or, recently).

This bug report is a bit vague in terms of what the problem is -- the test case hits 3 major subsystems (io, vm, net), all of which have changed rather substantially in the course of 2.6 development.

Vague 'cos I do not know where the problem is. One might say slowdown is like a near a 1ms delay per line output, but slowdown does not correlate to kernel tick frequency. :(

Quoted 2 lines

Would it be possible
to profile the system using oprofile to get an idea what the hotspots are?

Perhaps, I've yet to try that.

Quoted 4 lines

Have you compared basic hard disk throughput with hdparm, as well as ensuring DMA is enabled with 32 bit io? What about testing network performance with netperf (or a netcat of /dev/zero)? A few more data points would be quite helpful.

Yes, the gross datapoints such as basic net i/o, disk i/o, say 2.6 is better/faster than 2.4 on this hardware, my problem is how to describe measured console slowdown in terms meaningful to you.

I'll take a look at oprofile, report back if I can make sense of it ;)

Grant.

#3 lore raw
Quoted 3 lines · Grant Coady

Vague 'cos I do not know where the problem is. One might say slowdown is like a near a 1ms delay per line output, but slowdown does not correlate to kernel tick frequency. :(

Two things come to mind: can you try doing a vmstat 1 while running the test and compare 2.4 vs 2.6? Also, does it make a difference if you switch from the e100 driver to eepro100?

Quoted 1 line

I'll take a look at oprofile, report back if I can make sense of it ;)

If the CPU is pegged that will guide fixing things quite nicely, but the fact that it's 1ms per line sounds like something more sinister.

-ben

Signature · 2 lines
"Ladies and gentlemen, I'm sorry to interrupt, but the police are here 
and they've asked us to stop the party."  Don't Email: <[email protected]>.
#4 lore raw
In reply to Grant Coady
Quoted 22 lines · Grant Coady

Hi there,

grant@deltree:~$ uname -r
2.6.15.3a
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut
-c-95 ...
2006-02-08 12:38:13 +1100: bugsplatter.mine.nu 193.196.182.215 "GET
/test/linux-2.6/tosh/ HTTP/
real    0m8.537s
user    0m0.970s
sys     0m1.100s

--> reboot to 2.4.32-hf32.2

grant@deltree:~$ uname -r
2.4.32-hf32.2
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut
-c-95 ...
2006-02-08 12:38:13 +1100: bugsplatter.mine.nu 193.196.182.215 "GET
/test/linux-2.6/tosh/ HTTP/
Quoted 6 lines
real    0m2.271s
user    0m0.730s
sys     0m0.540s

Still a 4:1 slowdown, machine .config and dmesg info:
http://bugsplatter.mine.nu/test/boxen/deltree/

What happens if you add "| cat" on the end of your command?

Cheers,
Con

#5 lore raw
Quoted 23 lines · Con Kolivas
Quoted 22 lines · Grant Coady

Hi there,

grant@deltree:~$ uname -r
2.6.15.3a
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut
-c-95 ...
2006-02-08 12:38:13 +1100: bugsplatter.mine.nu 193.196.182.215 "GET
/test/linux-2.6/tosh/ HTTP/
real    0m8.537s
user    0m0.970s
sys     0m1.100s

--> reboot to 2.4.32-hf32.2

grant@deltree:~$ uname -r
2.4.32-hf32.2
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut
-c-95 ...
2006-02-08 12:38:13 +1100: bugsplatter.mine.nu 193.196.182.215 "GET
/test/linux-2.6/tosh/ HTTP/
Quoted 8 lines
Quoted 6 lines
real    0m2.271s
user    0m0.730s
sys     0m0.540s

Still a 4:1 slowdown, machine .config and dmesg info:
http://bugsplatter.mine.nu/test/boxen/deltree/

What happens if you add "| cat" on the end of your command?

It gets faster with 2.4.32-hf32.2 by a little bit (I forgot to copy)

reboot to 2.6.15.3a, without...

real    0m8.737s
user    0m1.030s
sys     0m1.200s

with... oh shit / surprise!!

real    0m1.861s
user    0m0.560s
sys     0m0.370s

What is that telling me / you / us?

Thanks,
Grant

Quoted 3 lines

Cheers,
Con

#6 lore raw
Quoted 41 lines · Grant Coady
Quoted 23 lines · Con Kolivas
Quoted 22 lines · Grant Coady

Hi there,

grant@deltree:~$ uname -r
2.6.15.3a
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut
-c-95 ...
2006-02-08 12:38:13 +1100: bugsplatter.mine.nu 193.196.182.215 "GET
/test/linux-2.6/tosh/ HTTP/
real    0m8.537s
user    0m0.970s
sys     0m1.100s

--> reboot to 2.4.32-hf32.2

grant@deltree:~$ uname -r
2.4.32-hf32.2
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut
-c-95 ...
2006-02-08 12:38:13 +1100: bugsplatter.mine.nu 193.196.182.215 "GET
/test/linux-2.6/tosh/ HTTP/
Quoted 8 lines
Quoted 6 lines
real    0m2.271s
user    0m0.730s
sys     0m0.540s

Still a 4:1 slowdown, machine .config and dmesg info:
http://bugsplatter.mine.nu/test/boxen/deltree/

What happens if you add "| cat" on the end of your command?

It gets faster with 2.4.32-hf32.2 by a little bit (I forgot to copy)

reboot to 2.6.15.3a, without...

real    0m8.737s
user    0m1.030s
sys     0m1.200s
Quoted 7 lines

with... oh shit / surprise!!

real    0m1.861s
user    0m0.560s
sys     0m0.370s

What is that telling me / you / us?

Heh.

This is the terminal's fault. xterm et al use an algorithm to determine how fast your machine is and decide whether to jump scroll or smooth scroll. This algorithm is basically broken with the 2.6 scheduler and it decides to mostly smooth scroll.

Cheers,
Con

#7 lore raw
Quoted 4 lines · Con Kolivas

This is the terminal's fault. xterm et al use an algorithm to determine how fast your machine is and decide whether to jump scroll or smooth scroll. This algorithm is basically broken with the 2.6 scheduler and it decides to mostly smooth scroll.

Recent versions of xterm are supposed to fix this. (Skimming xterm's changelog, I think it might have been fixed in version 201, but I'm not completely sure.)
--
-Barry K. Nathan <[email protected]>

#8 lore raw
Quoted 9 lines · Barry K. Nathan
Quoted 4 lines · Con Kolivas

This is the terminal's fault. xterm et al use an algorithm to determine how fast your machine is and decide whether to jump scroll or smooth scroll. This algorithm is basically broken with the 2.6 scheduler and it decides to mostly smooth scroll.

Recent versions of xterm are supposed to fix this. (Skimming xterm's changelog, I think it might have been fixed in version 201, but I'm not completely sure.)

Yeah but I'm using ssh to PuTTY on windoze. No GUI on linux box.

Grant.

#9 lore raw
In reply to Con Kolivas
Quoted 4 lines · Con Kolivas

This is the terminal's fault. xterm et al use an algorithm to determine how fast your machine is and decide whether to jump scroll or smooth scroll. This algorithm is basically broken with the 2.6 scheduler and it decides to mostly smooth scroll.

Strange it does that over localnet to a PuTTY terminal on windoze.

Seems a strange thing to do in the kernel though, presentation buffering / management surely can be done in userspace?

Grant.

#10 lore raw

Hi Grant,

Quoted 11 lines · Grant Coady
Quoted 4 lines · Con Kolivas

This is the terminal's fault. xterm et al use an algorithm to determine how fast your machine is and decide whether to jump scroll or smooth scroll. This algorithm is basically broken with the 2.6 scheduler and it decides to mostly smooth scroll.

Strange it does that over localnet to a PuTTY terminal on windoze.

Seems a strange thing to do in the kernel though, presentation buffering / management surely can be done in userspace?

I suspect the sshd on the firewall gets woken up for each line and it behaves exactly like an xterm. After having done a lot of "ls -l|cat" on 2.6, I'm not surprized at all :-/

A good test would be to strace sshd under 2.4 and 2.6. You could even use strace -tt. Probably that you will see something like 1 ms between two reads on 2.6 and nearly nothing between them in 2.4.

Quoted 1 line

Grant.

Cheers,
Willy

#11 lore raw

Hi Willy,

Quoted 15 lines · Willy Tarreau

Hi Grant,

Quoted 11 lines · Grant Coady
Quoted 4 lines · Con Kolivas

This is the terminal's fault. xterm et al use an algorithm to determine how fast your machine is and decide whether to jump scroll or smooth scroll. This algorithm is basically broken with the 2.6 scheduler and it decides to mostly smooth scroll.

Strange it does that over localnet to a PuTTY terminal on windoze.

Seems a strange thing to do in the kernel though, presentation buffering / management surely can be done in userspace?

Quoted 7 lines

I suspect the sshd on the firewall gets woken up for each line and it behaves exactly like an xterm. After having done a lot of "ls -l|cat" on 2.6, I'm not surprized at all :-/

A good test would be to strace sshd under 2.4 and 2.6. You could even use strace -tt. Probably that you will see something like 1 ms between two reads on 2.6 and nearly nothing between them in 2.4.

Yes, it is nearly 1ms per line delay with 2.6, but 2.4 and 2.6 with the trailing '|cat' give similar times, didn't try that notion last time.

We know now it isn't the network cards, disk I/O, just an oddness in 2.6 ;)

Cheers,
Grant.

#12 lore raw
In reply to Con Kolivas
Quoted 5 lines · Con Kolivas

This is the terminal's fault. xterm et al use an algorithm to determine how fast your machine is and decide whether to jump scroll or smooth scroll. This algorithm is basically broken with the 2.6 scheduler and it decides to mostly smooth scroll.

Hmm, I've been having a similar problem for ages. If I just do "ls" in my home directory 10 or 20 times, approximately 20% of the time it's "fast":

real    0m0.177s
user    0m0.028s
sys     0m0.027s

And the rest of the times it's "slow":

real    0m1.240s
user    0m0.036s
sys     0m0.040s

I rarely get anything in between - it's either ~1.2s or ~0.2s.

"time ls | cat" is always fast - 0.18 - 0.35s.

real    0m0.188s
user    0m0.014s
sys     0m0.018s

It has been this way as long as I can remember and it never made sense...

Lee

#13 lore raw
In reply to Con Kolivas
Quoted 5 lines
Quoted 2 lines
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut
-c-95 ...

What happens if you add "| cat" on the end of your command?

Do you think it's the new pipe buffering thing? (Introduced 2.6.10-.12, don't remember exactly)

Jan Engelhardt

#14 lore raw
Quoted 7 lines · Jan Engelhardt
Quoted 5 lines
Quoted 2 lines
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut
-c-95 ...

What happens if you add "| cat" on the end of your command?

Do you think it's the new pipe buffering thing? (Introduced 2.6.10-.12, don't remember exactly)

If it's the same problem I've been seeing it goes back much farther than 2.6.10.

Lately I suspect the scheduler.

Lee

#15 lore raw
Quoted 6 lines · Lee Revell
Quoted 5 lines · Jan Engelhardt
Quoted 5 lines
Quoted 2 lines
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut
-c-95 ...

What happens if you add "| cat" on the end of your command?

Quoted 7 lines
Quoted 2 lines

Do you think it's the new pipe buffering thing? (Introduced 2.6.10-.12, don't remember exactly)

If it's the same problem I've been seeing it goes back much farther than 2.6.10.

Lately I suspect the scheduler.

Hmm. I ran into an oddity while testing a modified kernel, and see something in schedule() that I don't think is right...

Down where it does requeue_task(next, array) if a freshly awakened task is to possibly receive a priority boost for the time it sat on the runqueue, I see a potential problem. If the task didn't sit on the queue long enough to be promoted, and isn't at the very top, it is going to the back of the bus as soon it gets preempted by say xmms. For a task that possibly just sat through the full rotation of a busy queue waiting for a shot at the cpu, that has got to hurt. Speculating, that requeue looks like it's there to increase the queue rotation rate, ie to reduce latency, but it looks to me like it can also accomplish the opposite if the context switch rate for your queue isn't very high.

... I ended up sharing a queue with a few rampaging irman2 threads, and each keystroke took ages. [btw, i wonder how the heck next->array could not be rq->active there]

-Mike

#16 lore raw
Quoted 29 lines · MIke Galbraith
Quoted 6 lines · Lee Revell
Quoted 5 lines · Jan Engelhardt
Quoted 5 lines
Quoted 2 lines
grant@deltree:~$ time grep -v 192\.168\. /var/log/apache/access_log| cut
-c-95 ...

What happens if you add "| cat" on the end of your command?

Quoted 7 lines
Quoted 2 lines

Do you think it's the new pipe buffering thing? (Introduced 2.6.10-.12, don't remember exactly)

If it's the same problem I've been seeing it goes back much farther than 2.6.10.

Lately I suspect the scheduler.

Hmm. I ran into an oddity while testing a modified kernel, and see something in schedule() that I don't think is right...

Down where it does requeue_task(next, array) if a freshly awakened task is to possibly receive a priority boost for the time it sat on the runqueue, I see a potential problem. If the task didn't sit on the queue long enough to be promoted, and isn't at the very top, it is going to the back of the bus as soon it gets preempted by say xmms. For a task that possibly just sat through the full rotation of a busy queue waiting for a shot at the cpu, that has got to hurt. Speculating, that requeue looks like it's there to increase the queue rotation rate, ie to reduce latency, but it looks to me like it can also accomplish the opposite if the context switch rate for your queue isn't very high.

Quoted 3 lines

... I ended up sharing a queue with a few rampaging irman2 threads, and each keystroke took ages. [btw, i wonder how the heck next->array could not be rq->active there]

I guess you didn't try my pseudo suggestion. Since I happen to be actively tinkering in this very area, I'll be a bit more direct :)

If you think it's the scheduler, how about try the patch below. It's against 2.6.16-rc2-mm1, and should tell you if it is the interactivity logic in the scheduler or not. I don't see other candidates in there, not that that means there aren't any of course.

With this patch in place, running an irman2 in one window, a make -j4 over nfs in another, and multimedia_sim as a test application in another, I get these results...

[mikeg@Homer]:> ./multimedia_sim 0 60 nice_level = 0
duration = 60 seconds
[frames] received: 1784 dropped: 0
[latency] mean: 0.000627 max: 0.019647 stddev: 0.000786
score: 0.004240

.... test proggy attached for inspection.

To be extra sure, set both /proc/sys/kernel/sched_g1 and g2 to 0. That will (I mean should of course;) more or less restore the original O(1) scheduler behavior.

-Mike

Maybe not pretty, but effective counts too...

include/linux/sched.h renamed from include/linux/sched.h.org +2 -2
@@ -721,14 +721,14 @@
 
 	unsigned short ioprio;
 
-	unsigned long sleep_avg;
+	unsigned long sleep_avg, last_slice, throttle_stamp;
 	unsigned long long timestamp, last_ran;
 	unsigned long long sched_time; /* sched_clock time spent running */
 	enum sleep_type sleep_type;
 
 	unsigned long policy;
 	cpumask_t cpus_allowed;
-	unsigned int time_slice, first_time_slice;
+	unsigned int time_slice, slice_info;
 
 #ifdef CONFIG_SCHEDSTATS
 	struct sched_info sched_info;
include/linux/sysctl.h renamed from include/linux/sysctl.h.org +2 -0
@@ -147,6 +147,8 @@
 	KERN_SETUID_DUMPABLE=69, /* int: behaviour of dumps for setuid core */
 	KERN_SPIN_RETRY=70,	/* int: number of spinlock retries */
 	KERN_ACPI_VIDEO_FLAGS=71, /* int: flags for setting up video after ACPI sleep */
+	KERN_SCHED_THROTTLE1=72,  /* int: throttling grace period 1 in secs */
+	KERN_SCHED_THROTTLE2=73,  /* int: throttling grace period 2 in secs */
 };
 
 
kernel/sched.c renamed from kernel/sched.c.org +316 -26

443 lines not on this page · raw message

kernel/sysctl.c renamed from kernel/sysctl.c.org +27 -6
@@ -69,6 +69,8 @@
 extern int pid_max_min, pid_max_max;
 extern int sysctl_drop_caches;
 extern int percpu_pagelist_fraction;
+extern int sched_g1;
+extern int sched_g2;
 
 #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
 int unknown_nmi_panic;
@@ -224,6 +226,11 @@
 	{ .ctl_name = 0 }
 };
 
+/* Constants for minimum and maximum testing in vm_table and
+ * kern_table.  We use these as one-element integer vectors. */
+static int zero;
+static int one_hundred = 100;
+
 static ctl_table kern_table[] = {
 	{
 		.ctl_name	= KERN_OSTYPE,
@@ -666,15 +673,29 @@
 		.proc_handler	= &proc_dointvec,
 	},
 #endif
+	{
+		.ctl_name	= KERN_SCHED_THROTTLE1,
+		.procname	= "sched_g1",
+		.data		= &sched_g1,
+		.maxlen		= sizeof (int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+		.strategy	= &sysctl_intvec,
+		.extra1		= &zero,
+	},
+	{
+		.ctl_name	= KERN_SCHED_THROTTLE2,
+		.procname	= "sched_g2",
+		.data		= &sched_g2,
+		.maxlen		= sizeof (int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+		.strategy	= &sysctl_intvec,
+		.extra1		= &zero,
+	},
 	{ .ctl_name = 0 }
 };
 
-/* Constants for minimum and maximum testing in vm_table.
-   We use these as one-element integer vectors. */
-static int zero;
-static int one_hundred = 100;
-
-
 static ctl_table vm_table[] = {
 	{
 		.ctl_name	= VM_OVERCOMMIT_MEMORY,
fs/pipe.c renamed from fs/pipe.c.org +1 -5
@@ -39,11 +39,7 @@
 {
 	DEFINE_WAIT(wait);
 
-	/*
-	 * Pipes are system-local resources, so sleeping on them
-	 * is considered a noninteractive wait:
-	 */
-	prepare_to_wait(PIPE_WAIT(*inode), &wait, TASK_INTERRUPTIBLE|TASK_NONINTERACTIVE);
+	prepare_to_wait(PIPE_WAIT(*inode), &wait, TASK_INTERRUPTIBLE);
 	mutex_unlock(PIPE_MUTEX(*inode));
 	schedule();
 	finish_wait(PIPE_WAIT(*inode), &wait);

/* multimedia_sim.c v0.3
*
* Dec 2002 - Miguel Freitas
*
* this is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
*
* this program is meant to simulate a dummy multimedia application and
* measure how it would perform in a loaded system. it basicaly tries to
* identify frame skipping problems that would have affected the movie
* playback.
*
* although the model of threads is heavily based on xine's architecture,
* its results should be also comparable with any other player program
* like mplayer or avifile. the idea is to measure when the player isn't
* scheduled in time for sending images at full frame rate and if X server
* would also be scheduled in time for displaying.
*
* of course one might try some tricks to improve performance like decreasing
* nice values for both XFree86 and player. however some distros don't
* ship the X reniced and modifying desktop menu entries to add "nice" and
* "sudo" commands is beyond most of users who just want to play their dvds... *
* compile with: gcc -o multimedia_sim multimedia_sim.c -lpthread -lm
* run as: ./multimedia_sim [nice_level] [test_duration]
*
* note1: default CPU_BURNING value should simulate more or less a mpeg2-class
* decoding cpu usage. that will require, at least, a 300MHz processor. *
* note2: a better simulation of xine's backend/frontend architecture would
* also include another thread (frontend) to receive the xshm completion
* events. i have intentionally not implemented it here.
*/

#include <stdio.h>
#include <pthread.h>
#include <unistd.h>
#include <sys/time.h>
#include <math.h>
#include <inttypes.h>

#define FRAME_PERIOD     1000000/30    /* NTSC period in us */
#define FRAME_SIZE       720*480*3/2   /* std resolution in yv12 format */
#define PREBUFFER_FRAMES 15            /* how many frames to "decode" ahead */
#define CPU_BURNING      8             /* reduce if your cpu isn't fast enough */

int nice_level = 0;

int decoder_running;
int video_out_running;
int server_running;

pthread_mutex_t counters_lock;
pthread_cond_t wakeup_server;
pthread_mutex_t queue_lock;
pthread_cond_t enqueued;
pthread_cond_t dequeued;
int frames_sent = 0;
int frames_received = 0;
int frames_enqueued = 0;
double start_time;

pthread_mutex_t counters_lock;
pthread_cond_t wakeup_server;

/* statistics */
double total_latency = 0.0;
double total_square = 0.0;
double max_latency = 0.0;
int frames_dropped = 0;
int dropped_in_burst = 0;
int bursts = 0;

static void *alloc_frame( void ) {
void *frame;

frame = (void *)malloc(FRAME_SIZE);
memset(frame,0,FRAME_SIZE);
return frame;
}

static void dummy_enqueue_frame( void ) {
pthread_mutex_lock( &queue_lock );
while( frames_enqueued == PREBUFFER_FRAMES )

    pthread_cond_wait( &dequeued, &queue_lock );

frames_enqueued++;
pthread_cond_signal( &enqueued );
pthread_mutex_unlock( &queue_lock );
}

static void dummy_dequeue_frame( void ) {
pthread_mutex_lock( &queue_lock );
while( !frames_enqueued )

    pthread_cond_wait( &enqueued, &queue_lock );

frames_enqueued--;
pthread_cond_signal( &dequeued );
pthread_mutex_unlock( &queue_lock );
}

static double get_us_time( void ) {
struct timeval tv;
double us;

  gettimeofday(&tv, NULL);
  us = tv.tv_sec * 1e6;
  us += tv.tv_usec;

return us;
}

/* from libmpeg2, used to burn cpu cycles */
#define W1 2841 /* 2048*sqrt (2)*cos (1*pi/16) */
#define W2 2676 /* 2048*sqrt (2)*cos (2*pi/16) */
#define W3 2408 /* 2048*sqrt (2)*cos (3*pi/16) */
#define W5 1609 /* 2048*sqrt (2)*cos (5*pi/16) */
#define W6 1108 /* 2048*sqrt (2)*cos (6*pi/16) */
#define W7 565 /* 2048*sqrt (2)*cos (7*pi/16) */
static void idct_row (int16_t * block)

{
    int x0, x1, x2, x3, x4, x5, x6, x7, x8;

    x1 = block[4] << 11;
    x2 = block[6];
    x3 = block[2];
    x4 = block[1];
    x5 = block[7];
    x6 = block[5];
    x7 = block[3];

    x0 = (block[0] << 11) + 128; /* for proper rounding in the fourth stage */

    /* first stage */
    x8 = W7 * (x4 + x5);
    x4 = x8 + (W1 - W7) * x4;
    x5 = x8 - (W1 + W7) * x5;
    x8 = W3 * (x6 + x7);
    x6 = x8 - (W3 - W5) * x6;
    x7 = x8 - (W3 + W5) * x7;
 
    /* second stage */
    x8 = x0 + x1;
    x0 -= x1;
    x1 = W6 * (x3 + x2);
    x2 = x1 - (W2 + W6) * x2;
    x3 = x1 + (W2 - W6) * x3;
    x1 = x4 + x6;
    x4 -= x6;
    x6 = x5 + x7;
    x5 -= x7;
 
    /* third stage */
    x7 = x8 + x3;
    x8 -= x3;
    x3 = x0 + x2;
    x0 -= x2;
    x2 = (181 * (x4 + x5) + 128) >> 8;
    x4 = (181 * (x4 - x5) + 128) >> 8;
 
    /* fourth stage */
    block[0] = (x7 + x1) >> 8;
    block[1] = (x3 + x2) >> 8;
    block[2] = (x0 + x4) >> 8;
    block[3] = (x8 + x6) >> 8;
    block[4] = (x8 - x6) >> 8;
    block[5] = (x0 - x4) >> 8;
    block[6] = (x3 - x2) >> 8;
    block[7] = (x7 - x1) >> 8;
}

static void *decoder_loop (void *this_gen) {

int16_t *frame;
int i,j;

  frame = alloc_frame();
  /* dummy data */
  for( i = 0; i < FRAME_SIZE/sizeof(int16_t); i++ )
    frame[i] = i;

  while( decoder_running )
  {
    /* eat some cpu cycles */
    for( j = 0; j < CPU_BURNING; j++ )
     for( i = 0; i < FRAME_SIZE/8/sizeof(int16_t); i+=8 )
       idct_row( &frame[i] );
    dummy_enqueue_frame();

}

  free(frame);

pthread_exit(NULL);
}

static void *video_out_loop (void *this_gen) {

double ttf; /* time to frame */
void *frame1, *frame2;

  nice(nice_level);
  
  frame1 = alloc_frame();
  frame2 = alloc_frame();

  start_time = get_us_time() + (FRAME_PERIOD * PREBUFFER_FRAMES);
  
  while( video_out_running )
  {
    dummy_dequeue_frame();
    /* eat some cpu cycles */
    /*memcpy(frame1, frame2, FRAME_SIZE);*/
    
    ttf = start_time + (frames_sent * FRAME_PERIOD);
    ttf -= get_us_time();
                   
    if( ttf > 0 )
      usleep( ttf );
    
    pthread_mutex_lock( &counters_lock ); 
    frames_sent++;
    pthread_cond_signal( &wakeup_server );
    pthread_mutex_unlock( &counters_lock ); 

}

  free(frame1);
  free(frame2);

pthread_exit(NULL);
}

static void *server_loop (void *this_gen) {

double estimated;
double latency;
void *frame1, *frame2;

  nice(nice_level);

  frame1 = alloc_frame();
  frame2 = alloc_frame();
  
  while( server_running )
  {
    pthread_mutex_lock( &counters_lock ); 
    while( frames_sent <= frames_received )
      pthread_cond_wait( &wakeup_server, &counters_lock );
    pthread_mutex_unlock( &counters_lock ); 
    estimated = start_time + (frames_received * FRAME_PERIOD);
    latency = (get_us_time() - estimated)/1.0e6;
     
    if( latency > max_latency )
      max_latency = latency;
       
    frames_received++;
    if( latency > FRAME_PERIOD/1.0e6 ) {
      frames_dropped++;
      dropped_in_burst++;
    } else if (dropped_in_burst) {
      dropped_in_burst = 0;
      bursts++;
    }
    
    total_latency += latency;
    total_square += latency * latency;
    
    /* eat some cpu cycles */
    memcpy(frame1, frame2, FRAME_SIZE);

}

if (dropped_in_burst)

    bursts++;
  free(frame1);
  free(frame2);

pthread_exit(NULL);
}

int main(int argc, char *argv[])
{
pthread_t decoder_thread;
pthread_t video_thread;
pthread_t server_thread;
void *p;
double mean, var, stddev;
double burst_size;
double score;
int duration = 10;

  if( argc > 1 ) {
    nice_level = atoi(argv[1]);
    printf("nice_level = %d\n", nice_level );
    if( nice_level < 0 )
      printf("(make sure you are root for negative nice)\n");
    if( argc > 2 ) {
      duration = atoi(argv[2]);
      printf("duration = %d seconds\n", duration );
    }
  }
  
  pthread_mutex_init (&counters_lock, NULL);
  pthread_cond_init  (&wakeup_server, NULL);
  pthread_mutex_init (&queue_lock, NULL);
  pthread_cond_init  (&enqueued, NULL);
  pthread_cond_init  (&dequeued, NULL);

  server_running = 1;
  if ( pthread_create (&server_thread, NULL, server_loop, NULL) != 0) {
    printf("Error creating server thread.\n");
    return 1;
  }
    
  video_out_running = 1;
  if ( pthread_create (&video_thread, NULL, video_out_loop, NULL) != 0) {
    printf("Error creating video thread.\n");
    return 1;
  }

  decoder_running = 1;
  if ( pthread_create (&decoder_thread, NULL, decoder_loop, NULL) != 0) {
    printf("Error creating decoder thread.\n");
    return 1;
  }
  
  sleep(duration);
  
  server_running = 0;
  pthread_join(server_thread,&p);
  video_out_running = 0;
  pthread_join(video_thread, &p);
  
  printf("[frames] received: %d dropped: %d\n", frames_received, frames_dropped );
  if( bursts ) {
    burst_size = (double)frames_dropped / bursts;
    printf("[frames] mean dropped per burst: %lf\n", burst_size );
  } else
    burst_size = 1.0;
  
  mean = total_latency / frames_received;
  var = total_square / frames_received - mean*mean;
  stddev = sqrt(var);
  printf("[latency] mean: %lf max: %lf stddev: %lf\n", mean, max_latency, stddev);

/* score: lower is better. it tries to measure "how bad" the playback
* was to the user. it counts fraction of dropped frames, if the dropped
* frames were somewhat evenly distributed (instead of in bursts), and
* also the mean and standard deviation.
* note that the formula is actually arbitrary, i'm just trying to count
* all these factors and weight them.
*/

score = 0.0;
score += 0.90 * (double) frames_dropped / frames_received * sqrt(burst_size); score += 0.10 * (mean + stddev)/(FRAME_PERIOD/1.0e6); printf("score: %lf\n", score );
}

#17 lore raw
Quoted 4 lines · MIke Galbraith

If you think it's the scheduler, how about try the patch below. It's against 2.6.16-rc2-mm1, and should tell you if it is the interactivity logic in the scheduler or not. I don't see other candidates in there, not that that means there aren't any of course.

I'll try, but it's a serious pain for me to build an -mm kernel. A patch against 2.6.16-rc1 would be much easier.

Lee

#18 lore raw
Quoted 8 lines · Lee Revell
Quoted 4 lines · MIke Galbraith

If you think it's the scheduler, how about try the patch below. It's against 2.6.16-rc2-mm1, and should tell you if it is the interactivity logic in the scheduler or not. I don't see other candidates in there, not that that means there aren't any of course.

I'll try, but it's a serious pain for me to build an -mm kernel. A patch against 2.6.16-rc1 would be much easier.

Ok, here she comes. It's a bit too reluctant to release a task so it can reach interactive status at the moment, but for this test, that's a feature. In fact, for this test, it's probably best to jump straight to setting both g1 and g2 to zero.

-Mike

include/linux/sched.h renamed from include/linux/sched.h.org +10 -3
@@ -688,6 +688,13 @@
 struct audit_context;		/* See audit.c */
 struct mempolicy;
 
+enum sleep_type {
+	SLEEP_NORMAL,
+	SLEEP_NONINTERACTIVE,
+	SLEEP_INTERACTIVE,
+	SLEEP_INTERRUPTED,
+};
+
 struct task_struct {
 	volatile long state;	/* -1 unrunnable, 0 runnable, >0 stopped */
 	struct thread_info *thread_info;
@@ -709,14 +716,14 @@
 
 	unsigned short ioprio;
 
-	unsigned long sleep_avg;
+	unsigned long sleep_avg, last_slice, throttle_stamp;
 	unsigned long long timestamp, last_ran;
 	unsigned long long sched_time; /* sched_clock time spent running */
-	int activated;
+	enum sleep_type sleep_type;
 
 	unsigned long policy;
 	cpumask_t cpus_allowed;
-	unsigned int time_slice, first_time_slice;
+	unsigned int time_slice, slice_info;
 
 #ifdef CONFIG_SCHEDSTATS
 	struct sched_info sched_info;
include/linux/sysctl.h renamed from include/linux/sysctl.h.org +2 -0
@@ -146,6 +146,8 @@
 	KERN_RANDOMIZE=68, /* int: randomize virtual address space */
 	KERN_SETUID_DUMPABLE=69, /* int: behaviour of dumps for setuid core */
 	KERN_SPIN_RETRY=70,	/* int: number of spinlock retries */
+	KERN_SCHED_THROTTLE1=71,  /* int: throttling grace period 1 in secs */
+	KERN_SCHED_THROTTLE2=72,  /* int: throttling grace period 2 in secs */
 };
 
 
kernel/sched.c renamed from kernel/sched.c.org +348 -57

539 lines not on this page · raw message

kernel/sysctl.c renamed from kernel/sysctl.c.org +27 -6
@@ -71,6 +71,8 @@
 extern int pid_max_min, pid_max_max;
 extern int sysctl_drop_caches;
 extern int percpu_pagelist_fraction;
+extern int sched_g1;
+extern int sched_g2;
 
 #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
 int unknown_nmi_panic;
@@ -226,6 +228,11 @@
 	{ .ctl_name = 0 }
 };
 
+/* Constants for minimum and maximum testing in vm_table and
+ * kern_table.  We use these as one-element integer vectors. */
+static int zero;
+static int one_hundred = 100;
+
 static ctl_table kern_table[] = {
 	{
 		.ctl_name	= KERN_OSTYPE,
@@ -658,15 +665,29 @@
 		.proc_handler	= &proc_dointvec,
 	},
 #endif
+	{
+		.ctl_name	= KERN_SCHED_THROTTLE1,
+		.procname	= "sched_g1",
+		.data		= &sched_g1,
+		.maxlen		= sizeof (int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+		.strategy	= &sysctl_intvec,
+		.extra1		= &zero,
+	},
+	{
+		.ctl_name	= KERN_SCHED_THROTTLE2,
+		.procname	= "sched_g2",
+		.data		= &sched_g2,
+		.maxlen		= sizeof (int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+		.strategy	= &sysctl_intvec,
+		.extra1		= &zero,
+	},
 	{ .ctl_name = 0 }
 };
 
-/* Constants for minimum and maximum testing in vm_table.
-   We use these as one-element integer vectors. */
-static int zero;
-static int one_hundred = 100;
-
-
 static ctl_table vm_table[] = {
 	{
 		.ctl_name	= VM_OVERCOMMIT_MEMORY,
fs/pipe.c renamed from fs/pipe.c.org +1 -5
@@ -39,11 +39,7 @@
 {
 	DEFINE_WAIT(wait);
 
-	/*
-	 * Pipes are system-local resources, so sleeping on them
-	 * is considered a noninteractive wait:
-	 */
-	prepare_to_wait(PIPE_WAIT(*inode), &wait, TASK_INTERRUPTIBLE|TASK_NONINTERACTIVE);
+	prepare_to_wait(PIPE_WAIT(*inode), &wait, TASK_INTERRUPTIBLE);
 	mutex_unlock(PIPE_MUTEX(*inode));
 	schedule();
 	finish_wait(PIPE_WAIT(*inode), &wait);
#19 lore raw
Quoted 7 lines · MIke Galbraith
Quoted 6 lines · Lee Revell
Quoted 4 lines · MIke Galbraith

If you think it's the scheduler, how about try the patch below. It's against 2.6.16-rc2-mm1, and should tell you if it is the interactivity logic in the scheduler or not. I don't see other candidates in there, not that that means there aren't any of course.

Quoted 7 lines
Quoted 2 lines

I'll try, but it's a serious pain for me to build an -mm kernel. A patch against 2.6.16-rc1 would be much easier.

Ok, here she comes. It's a bit too reluctant to release a task so it can reach interactive status at the moment, but for this test, that's a feature. In fact, for this test, it's probably best to jump straight to setting both g1 and g2 to zero.

Thanks, this solves the "ls" problem I was having! The strange "oscillating" behavior is gone, now it consistently takes 0.19-0.45s.

It's still not as consistent as "time ls | cat", which takes 0.19-0.26s, but MUCH better.

Lee

#20 lore raw
In reply to MIke Galbraith
Quoted 7 lines · MIke Galbraith
Quoted 6 lines · Lee Revell
Quoted 4 lines · MIke Galbraith

If you think it's the scheduler, how about try the patch below. It's against 2.6.16-rc2-mm1, and should tell you if it is the interactivity logic in the scheduler or not. I don't see other candidates in there, not that that means there aren't any of course.

Quoted 7 lines
Quoted 2 lines

I'll try, but it's a serious pain for me to build an -mm kernel. A patch against 2.6.16-rc1 would be much easier.

Ok, here she comes. It's a bit too reluctant to release a task so it can reach interactive status at the moment, but for this test, that's a feature. In fact, for this test, it's probably best to jump straight to setting both g1 and g2 to zero.

Not only does this fix my "time ls" test case, it seems to drastically improve interactivity for my desktop apps. I was really being plagued by weird stalls, it's much smoother now.

Now to regression test it...

Lee

#21 lore raw
Quoted 11 lines · Lee Revell
Quoted 10 lines · MIke Galbraith
Quoted 8 lines · Lee Revell
Quoted 4 lines · MIke Galbraith

If you think it's the scheduler, how about try the patch below. It's against 2.6.16-rc2-mm1, and should tell you if it is the interactivity logic in the scheduler or not. I don't see other candidates in there, not that that means there aren't any of course.

I'll try, but it's a serious pain for me to build an -mm kernel. A patch against 2.6.16-rc1 would be much easier.

Quoted 8 lines
Quoted 4 lines

Ok, here she comes. It's a bit too reluctant to release a task so it can reach interactive status at the moment, but for this test, that's a feature. In fact, for this test, it's probably best to jump straight to setting both g1 and g2 to zero.

Not only does this fix my "time ls" test case, it seems to drastically improve interactivity for my desktop apps. I was really being plagued by weird stalls, it's much smoother now.

Yeah, but under load, that reluctance to release is fairly annoying...

Quoted 2 lines

Now to regression test it...

...and may cause test applications to not reach their proper priority before measurement begins.

-Mike

#22 lore raw
Quoted 17 lines · MIke Galbraith
Quoted 16 lines · Lee Revell
Quoted 14 lines · MIke Galbraith
Quoted 8 lines · Lee Revell
Quoted 4 lines · MIke Galbraith

If you think it's the scheduler, how about try the patch below. It's against 2.6.16-rc2-mm1, and should tell you if it is the interactivity logic in the scheduler or not. I don't see other candidates in there, not that that means there aren't any of course.

I'll try, but it's a serious pain for me to build an -mm kernel. A patch against 2.6.16-rc1 would be much easier.

Ok, here she comes. It's a bit too reluctant to release a task so it can reach interactive status at the moment, but for this test, that's a feature. In fact, for this test, it's probably best to jump straight to setting both g1 and g2 to zero.

Quoted 5 lines
Quoted 3 lines

Not only does this fix my "time ls" test case, it seems to drastically improve interactivity for my desktop apps. I was really being plagued by weird stalls, it's much smoother now.

Yeah, but under load, that reluctance to release is fairly annoying...

This seems to manifest on my system as the mouse getting jerky under load. Still, I don't mind - the overall feel is still smoother - as if the X server was getting too much CPU before.

Lee

#23 lore raw
Quoted 7 lines · Lee Revell
Quoted 6 lines · MIke Galbraith
Quoted 3 lines · Lee Revell

Not only does this fix my "time ls" test case, it seems to drastically improve interactivity for my desktop apps. I was really being plagued by weird stalls, it's much smoother now.

Quoted 5 lines
Quoted 1 line

Yeah, but under load, that reluctance to release is fairly annoying...

This seems to manifest on my system as the mouse getting jerky under load. Still, I don't mind - the overall feel is still smoother - as if the X server was getting too much CPU before.

Not only the X server, the interactivity boost in general is way too severe. It still needs work, but it's shaping up. Who knows, it may even get to the point of applying for inclusion.

Now, let's see if we can get your problem fixed with something that can possibly go into 2.6.16 as a bugfix. Can you please try the below?

This patch fixes two of what I would call thinkos in the interactivity logic. One, the requeuing of a freshly awakened task can lead to that task losing it's slice of cpu upon preempt if it isn't alone in it's queue. Two, tasks which sleep for > INTERACTIVE_SLEEP(p), approximately 700ms, are treated as being idle, and prevented from slamming straight to max dynamic priority for obvious reasons, yet a pure cpu hog that sleeps for 100ms will fail the test, and subsequently have it's sleep time multiplied by MAX_BONUS.

Just in case it does fix it for you, I'll even add a blame line.

kernel/sched.c renamed from kernel/sched.c.org +8 -6
@@ -150,8 +150,11 @@
 	((p)->prio <= (p)->static_prio - DELTA(p))
 
 #define INTERACTIVE_SLEEP(p) \
-	(JIFFIES_TO_NS(MAX_SLEEP_AVG * \
-		(MAX_BONUS / 2 + DELTA((p)) + 1) / MAX_BONUS - 1))
+	(min(JIFFIES_TO_NS(MAX_SLEEP_AVG * (MAX_BONUS / 2 + DELTA((p)) + 1) / \
+	MAX_BONUS - 1), NS_MAX_SLEEP_AVG))
+
+#define BONUS_MULTIPLIER(p) \
+	((MAX_BONUS - CURRENT_BONUS(p)) ? : 1)
 
 #define TASK_PREEMPTS_CURR(p, rq) \
 	((p)->prio < (rq)->curr->prio)
@@ -708,7 +711,7 @@
 		 * prevent them suddenly becoming cpu hogs and starving
 		 * other processes.
 		 */
-		if (p->mm && p->activated != -1 &&
+		if (p->mm && p->activated != -1 && BONUS_MULTIPLIER(p) *
 			sleep_time > INTERACTIVE_SLEEP(p)) {
 				p->sleep_avg = JIFFIES_TO_NS(MAX_SLEEP_AVG -
 						DEF_TIMESLICE);
@@ -717,7 +720,7 @@
 			 * The lower the sleep avg a task has the more
 			 * rapidly it will rise with sleep time.
 			 */
-			sleep_time *= (MAX_BONUS - CURRENT_BONUS(p)) ? : 1;
+			sleep_time *= BONUS_MULTIPLIER(p);
 
 			/*
 			 * Tasks waking from uninterruptible sleep are
@@ -3009,8 +3012,7 @@
 			dequeue_task(next, array);
 			next->prio = new_prio;
 			enqueue_task(next, array);
-		} else
-			requeue_task(next, array);
+		}
 	}
 	next->activated = 0;
 switch_tasks:
#24 lore raw
Quoted 2 lines · MIke Galbraith

Now, let's see if we can get your problem fixed with something that can possibly go into 2.6.16 as a bugfix. Can you please try the below?

These sorts of changes definitely need to pass through -mm first... and don't forget -mm looks quite different to mainline.

Cheers,
Con

#25 lore raw
Quoted 6 lines · Con Kolivas
Quoted 2 lines · MIke Galbraith

Now, let's see if we can get your problem fixed with something that can possibly go into 2.6.16 as a bugfix. Can you please try the below?

These sorts of changes definitely need to pass through -mm first... and don't forget -mm looks quite different to mainline.

I'll leave that up to Ingo of course, and certainly have no problem with them burning in mm. However, I must say that I personally classify these two changes as being trivial and obviously correct enough to be included in 2.6.16. I realize that mm is different in this area (we talked about it), but the problems addressed by this patch still remain. Anyway, let's wait and see if it fixes Lee's problem.

-Mike

#26 lore raw
Quoted 5 lines · MIke Galbraith
Quoted 4 lines · Con Kolivas
Quoted 2 lines · MIke Galbraith

Now, let's see if we can get your problem fixed with something that can possibly go into 2.6.16 as a bugfix. Can you please try the below?

Quoted 7 lines
Quoted 2 lines

These sorts of changes definitely need to pass through -mm first... and don't forget -mm looks quite different to mainline.

I'll leave that up to Ingo of course, and certainly have no problem with them burning in mm. However, I must say that I personally classify these two changes as being trivial and obviously correct enough to be included in 2.6.16.

This part I agree with:
- } else
- requeue_task(next, array);
+ }

The rest changes behaviour; it's not a "bug" so needs testing, should be a separate patch from this part, and modified to suit -mm.

Cheers,
Con

#27 lore raw
Quoted 19 lines · Con Kolivas
Quoted 5 lines · MIke Galbraith
Quoted 4 lines · Con Kolivas
Quoted 2 lines · MIke Galbraith

Now, let's see if we can get your problem fixed with something that can possibly go into 2.6.16 as a bugfix. Can you please try the below?

Quoted 7 lines
Quoted 2 lines

These sorts of changes definitely need to pass through -mm first... and don't forget -mm looks quite different to mainline.

I'll leave that up to Ingo of course, and certainly have no problem with them burning in mm. However, I must say that I personally classify these two changes as being trivial and obviously correct enough to be included in 2.6.16.

This part I agree with:

kernel/sched.c renamed from kernel/sched.c.org +1 -2
-               } else
-                       requeue_task(next, array);
+               }
Quoted 2 lines

The rest changes behaviour; it's not a "bug" so needs testing, should be a separate patch from this part, and modified to suit -mm.

Well, both change behavior, and I heartily disagree. Blocking a 700ms sleep while allowing a 100ms sleep to bypass the same checkpoint only to then be multiplied by 10 is a bug.

Actually, the point at which a task becomes interactive is the point at which scheduler semantics change. Ergo, as far as I'm concerned, this should be a boundary which must be crossed before proceeding further. That, I agree, would be a behavioral change which should be baked in mm.

-Mike

#28 lore raw
Quoted 21 lines · MIke Galbraith
Quoted 20 lines · Con Kolivas
Quoted 6 lines · MIke Galbraith
Quoted 5 lines · Con Kolivas
Quoted 3 lines · MIke Galbraith

Now, let's see if we can get your problem fixed with something that can possibly go into 2.6.16 as a bugfix. Can you please try the below?

Quoted 7 lines
Quoted 2 lines

These sorts of changes definitely need to pass through -mm first... and don't forget -mm looks quite different to mainline.

I'll leave that up to Ingo of course, and certainly have no problem with them burning in mm. However, I must say that I personally classify these two changes as being trivial and obviously correct enough to be included in 2.6.16.

This part I agree with:

kernel/sched.c renamed from kernel/sched.c.org +1 -2
-               } else
-                       requeue_task(next, array);
+               }
Quoted 4 lines
Quoted 2 lines

The rest changes behaviour; it's not a "bug" so needs testing, should be a separate patch from this part, and modified to suit -mm.

Well, both change behavior, and I heartily disagree.

The first change was the previous behaviour for some time. Your latter change while it makes sense has never been in the kernel. Either way I don't disagree with your reasoning but most things that change behaviour should go through -mm. The first as I said was the behaviour in mainline for some time till my silly requeue change.

Cheers,
Con

Quoted 8 lines

Blocking a 700ms
sleep while allowing a 100ms sleep to bypass the same checkpoint only to then be multiplied by 10 is a bug.

Actually, the point at which a task becomes interactive is the point at which scheduler semantics change. Ergo, as far as I'm concerned, this should be a boundary which must be crossed before proceeding further. That, I agree, would be a behavioral change which should be baked in mm.

#29 lore raw
Quoted 25 lines · Con Kolivas
Quoted 24 lines · MIke Galbraith
Quoted 22 lines · Con Kolivas
Quoted 6 lines · MIke Galbraith
Quoted 5 lines · Con Kolivas
Quoted 3 lines · MIke Galbraith

Now, let's see if we can get your problem fixed with something that can possibly go into 2.6.16 as a bugfix. Can you please try the below?

Quoted 7 lines
Quoted 2 lines

These sorts of changes definitely need to pass through -mm first... and don't forget -mm looks quite different to mainline.

I'll leave that up to Ingo of course, and certainly have no problem with them burning in mm. However, I must say that I personally classify these two changes as being trivial and obviously correct enough to be included in 2.6.16.

This part I agree with:

kernel/sched.c renamed from kernel/sched.c.org +1 -2
-               } else
-                       requeue_task(next, array);
+               }

The rest changes behaviour; it's not a "bug" so needs testing, should be a separate patch from this part, and modified to suit -mm.

Quoted 7 lines
Quoted 1 line

Well, both change behavior, and I heartily disagree.

The first change was the previous behaviour for some time. Your latter change while it makes sense has never been in the kernel. Either way I don't disagree with your reasoning but most things that change behaviour should go through -mm. The first as I said was the behaviour in mainline for some time till my silly requeue change.

Ok, we're basically in agreement on these changes, it's just a matter of when. As maintainer, Ingo has to weigh the benefit, danger, etc etc.

-Mike

#30 lore raw
Quoted 28 lines · MIke Galbraith
Quoted 27 lines · Con Kolivas
Quoted 25 lines · MIke Galbraith
Quoted 20 lines · Con Kolivas
Quoted 6 lines · MIke Galbraith
Quoted 5 lines · Con Kolivas
Quoted 3 lines · MIke Galbraith

Now, let's see if we can get your problem fixed with something that can possibly go into 2.6.16 as a bugfix. Can you please try the below?

Quoted 7 lines
Quoted 2 lines

These sorts of changes definitely need to pass through -mm first... and don't forget -mm looks quite different to mainline.

I'll leave that up to Ingo of course, and certainly have no problem with them burning in mm. However, I must say that I personally classify these two changes as being trivial and obviously correct enough to be included in 2.6.16.

This part I agree with:

(quoted diff) +1 -2
-               } else
-                       requeue_task(next, array);
+               }
Quoted 2 lines

The rest changes behaviour; it's not a "bug" so needs testing, should be a separate patch from this part, and modified to suit -mm.

Well, both change behavior, and I heartily disagree.

Quoted 8 lines
Quoted 5 lines

The first change was the previous behaviour for some time. Your latter change while it makes sense has never been in the kernel. Either way I don't disagree with your reasoning but most things that change behaviour should go through -mm. The first as I said was the behaviour in mainline for some time till my silly requeue change.

Ok, we're basically in agreement on these changes, it's just a matter of when. As maintainer, Ingo has to weigh the benefit, danger, etc etc.

Do you know which of those changes fixes the "ls" problem?

Lee

#31 lore raw
Quoted 1 line · Lee Revell

Do you know which of those changes fixes the "ls" problem?

No, it could be either, both, or neither. Heck, it _could_ be a combination of all of the things in my experimental tree for that matter. I put this patch out there because I know they're both bugs, and strongly suspect it'll cure the worst of the interactivity related delays.

I'm hoping you'll test it and confirm that it fixes yours.

-Mike

#32 lore raw
Quoted 10 lines · MIke Galbraith
Quoted 1 line · Lee Revell

Do you know which of those changes fixes the "ls" problem?

No, it could be either, both, or neither. Heck, it _could_ be a combination of all of the things in my experimental tree for that matter. I put this patch out there because I know they're both bugs, and strongly suspect it'll cure the worst of the interactivity related delays.

I'm hoping you'll test it and confirm that it fixes yours.

Nope, this does not fix it. "time ls" ping-pongs back and forth between ~0.1s and ~0.9s. Must have been something else in the first patch.

Lee

#33 lore raw
Quoted 10 lines · Lee Revell
Quoted 9 lines · MIke Galbraith
Quoted 1 line · Lee Revell

Do you know which of those changes fixes the "ls" problem?

No, it could be either, both, or neither. Heck, it _could_ be a combination of all of the things in my experimental tree for that matter. I put this patch out there because I know they're both bugs, and strongly suspect it'll cure the worst of the interactivity related delays.

Quoted 4 lines
Quoted 1 line

I'm hoping you'll test it and confirm that it fixes yours.

Nope, this does not fix it. "time ls" ping-pongs back and forth between ~0.1s and ~0.9s. Must have been something else in the first patch.

Oh well. Thanks for testing Lee. I was hoping this would be a case of instant gratification, 2.6.16 being near, but it's not to be.

-Mike

#34 lore raw
In reply to Lee Revell
Quoted 10 lines · Lee Revell
Quoted 9 lines · MIke Galbraith
Quoted 1 line · Lee Revell

Do you know which of those changes fixes the "ls" problem?

No, it could be either, both, or neither. Heck, it _could_ be a combination of all of the things in my experimental tree for that matter. I put this patch out there because I know they're both bugs, and strongly suspect it'll cure the worst of the interactivity related delays.

Quoted 4 lines
Quoted 1 line

I'm hoping you'll test it and confirm that it fixes yours.

Nope, this does not fix it. "time ls" ping-pongs back and forth between ~0.1s and ~0.9s. Must have been something else in the first patch.

Hmm. Thinking about it some more, it's probably more than this alone, but it could well be the boost qualifier I'm using...

Instead of declaring a task to be deserving of large quantities of boost based upon their present shortage of sleep_avg, I based it upon their not using their full slice. He who uses the least gets the most. This made a large contribution to mitigating the parallel compile over NFS problem the current scheduler has. The fact that (current) heuristics which mandate that any task which sleeps for 5% of it's slice may use 95% cpu practically forever can not only work, but work quite well in the general case, tells me that the vast majority of all tasks are, and will forever remain, cpu hogs.

The present qualifier creates positive feedback for cpu hogs by giving them the most reward for being the biggest hog by our own definition. If you'll pardon the pun, we gives pigs wings, and hope that they don't actually use them and fly directly over head. This is the root problem as I see it, that and the fact that even if sleep_avg acquisition and consumption were purely 1:1 as the original O(1) scheduler was, if you sleep 1 ns longer than you run, you'll eventually be up to you neck in sleep_avg. (a darn good reason to use something like slice_avg to help determine when to drain off the excess)

Changing that qualifier would also mean that he who is _getting_ the least cpu would get the most boost as well, so it should help with fairness, and things like the test case mentioned in comments in the patch where one task can end up starving it's own partner.

Is there any reason that "he who uses the least gets the most" would be inferior to "he who has the least for whatever reason gets the most"?

If I were to put a patch together that did only that (IMHO sensible) thing, would anyone be interested in trying it?

-Mike

#35 lore raw
Quoted 11 lines · MIke Galbraith
Quoted 10 lines · Lee Revell
Quoted 9 lines · MIke Galbraith
Quoted 1 line · Lee Revell

Do you know which of those changes fixes the "ls" problem?

No, it could be either, both, or neither. Heck, it _could_ be a combination of all of the things in my experimental tree for that matter. I put this patch out there because I know they're both bugs, and strongly suspect it'll cure the worst of the interactivity related delays.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

I'm hoping you'll test it and confirm that it fixes yours.

Nope, this does not fix it. "time ls" ping-pongs back and forth between ~0.1s and ~0.9s. Must have been something else in the first patch.

Hmm. Thinking about it some more, it's probably more than this alone, but it could well be the boost qualifier I'm using...

OK, with 2.6.16-rc2-mm1, "ls" bounces around between 0.15s and 0.50s. Better than mainline but the large seemingly random variance is still perceptible and annoying. And, "ls | cat" behaves about the same as "ls", while on mainline it was consistently faster (!).

Do you have an updated patch against -mm that I can test?

Lee

#36 lore raw
Quoted 17 lines · Lee Revell
Quoted 16 lines · MIke Galbraith
Quoted 14 lines · Lee Revell
Quoted 10 lines · MIke Galbraith
Quoted 1 line · Lee Revell

Do you know which of those changes fixes the "ls" problem?

No, it could be either, both, or neither. Heck, it _could_ be a combination of all of the things in my experimental tree for that matter. I put this patch out there because I know they're both bugs, and strongly suspect it'll cure the worst of the interactivity related delays.

I'm hoping you'll test it and confirm that it fixes yours.

Nope, this does not fix it. "time ls" ping-pongs back and forth between ~0.1s and ~0.9s. Must have been something else in the first patch.

Quoted 7 lines
Quoted 2 lines

Hmm. Thinking about it some more, it's probably more than this alone, but it could well be the boost qualifier I'm using...

OK, with 2.6.16-rc2-mm1, "ls" bounces around between 0.15s and 0.50s. Better than mainline but the large seemingly random variance is still perceptible and annoying. And, "ls | cat" behaves about the same as "ls", while on mainline it was consistently faster (!).

Ok. That means the reduction in fluctuation had nothing to do with my changes. It also suggests that there may be something of a regression in the changes that are in mm, which I also carried in my patch, since the timing for both kernels appear to be ~identical with or without my bits. That seems a little odd to me considering what those changes do.

Quoted 2 lines

Do you have an updated patch against -mm that I can test?

I will soon if you still want to try it. I've fixed the throttle release thing, and am fine tuning the interactivity bits. I have it working very well now, but want to try to squeeze some more from it.

Drop me a line if you're still interested from the interactivity side, but I think the ls delay reduction has turned out to be a red herring.

-Mike

#37 lore raw
Quoted 17 lines · MIke Galbraith
Quoted 1 line

OK, with 2.6.16-rc2-mm1, "ls" bounces around between 0.15s and

0.50s.

Quoted 1 line

Better than mainline but the large seemingly random variance is

still

Quoted 2 lines

perceptible and annoying. And, "ls | cat" behaves about the same as "ls", while on mainline it was consistently faster (!).

Ok. That means the reduction in fluctuation had nothing to do with my changes. It also suggests that there may be something of a regression in the changes that are in mm, which I also carried in my patch, since the timing for both kernels appear to be ~identical with or without my bits. That seems a little odd to me considering what those changes do.

Quoted 2 lines

Do you have an updated patch against -mm that I can test?

Quoted 8 lines

I will soon if you still want to try it. I've fixed the throttle release
thing, and am fine tuning the interactivity bits. I have it working very well now, but want to try to squeeze some more from it.

Drop me a line if you're still interested from the interactivity side, but I think the ls delay reduction has turned out to be a red herring.

Just to be clear - this is 2.6.16-rc2-mm1 *without* your patch that I am talking about.

Lee

#38 lore raw
Quoted 23 lines · Lee Revell
Quoted 22 lines · MIke Galbraith
Quoted 1 line

OK, with 2.6.16-rc2-mm1, "ls" bounces around between 0.15s and

0.50s.

Quoted 1 line

Better than mainline but the large seemingly random variance is

still

Quoted 2 lines

perceptible and annoying. And, "ls | cat" behaves about the same as "ls", while on mainline it was consistently faster (!).

Ok. That means the reduction in fluctuation had nothing to do with my changes. It also suggests that there may be something of a regression in the changes that are in mm, which I also carried in my patch, since the timing for both kernels appear to be ~identical with or without my bits. That seems a little odd to me considering what those changes do.

Quoted 2 lines

Do you have an updated patch against -mm that I can test?

I will soon if you still want to try it. I've fixed the throttle release
thing, and am fine tuning the interactivity bits. I have it working very well now, but want to try to squeeze some more from it.

Quoted 6 lines
Quoted 3 lines

Drop me a line if you're still interested from the interactivity side, but I think the ls delay reduction has turned out to be a red herring.

Just to be clear - this is 2.6.16-rc2-mm1 *without* your patch that I am talking about.

Exactly. 2.6.16-rc2-mm1 without my patch has a delay of .15 to .50s. 2.6.16-rc1 with my patch had a reported delay of from .19 to .45s. That's identical in my book. My patch to rc1 also contained Con's changes that are in mm, that's constant. Subtracting the variable, my patch, made no difference. Con's changes may be responsible for the behavior change, but mine are certainly not.

-Mike

#39 lore raw
In reply to MIke Galbraith
Quoted 28 lines · MIke Galbraith
Quoted 27 lines · Con Kolivas
Quoted 25 lines · MIke Galbraith
Quoted 20 lines · Con Kolivas
Quoted 6 lines · MIke Galbraith
Quoted 5 lines · Con Kolivas
Quoted 3 lines · MIke Galbraith

Now, let's see if we can get your problem fixed with something that can possibly go into 2.6.16 as a bugfix. Can you please try the below?

Quoted 7 lines
Quoted 2 lines

These sorts of changes definitely need to pass through -mm first... and don't forget -mm looks quite different to mainline.

I'll leave that up to Ingo of course, and certainly have no problem with them burning in mm. However, I must say that I personally classify these two changes as being trivial and obviously correct enough to be included in 2.6.16.

This part I agree with:

(quoted diff) +1 -2
-               } else
-                       requeue_task(next, array);
+               }
Quoted 2 lines

The rest changes behaviour; it's not a "bug" so needs testing, should be a separate patch from this part, and modified to suit -mm.

Well, both change behavior, and I heartily disagree.

Quoted 8 lines
Quoted 5 lines

The first change was the previous behaviour for some time. Your latter change while it makes sense has never been in the kernel. Either way I don't disagree with your reasoning but most things that change behaviour should go through -mm. The first as I said was the behaviour in mainline for some time till my silly requeue change.

Ok, we're basically in agreement on these changes, it's just a matter of when. As maintainer, Ingo has to weigh the benefit, danger, etc etc.

Aye and do note the change to the idle detection code currently in -mm will already make substantial difference there, possibly related to fluctuating behaviour.

Cheers,
Con

#40 lore raw
Quoted 8 lines · Con Kolivas
Quoted 3 lines · MIke Galbraith

Ok, we're basically in agreement on these changes, it's just a matter of when. As maintainer, Ingo has to weigh the benefit, danger, etc etc.

Aye and do note the change to the idle detection code currently in -mm will already make substantial difference there, possibly related to fluctuating behaviour.

Possibly... but there also lies a two edged sword. Previously, some bad boys were being truncated back to user_prio 17. In mm, that's no longer true.

-Mike