Skip to contentSkip to conversation

Poor desktop responsiveness with background I/O-operations

DISCUSSION Ulrich Lukas 10 participants 26 messages Sep 20, 2009 – Sep 22, 2009

lore
Share & tools
lore
#0 lore raw

Hi,

using a recent hard-/software setup, I observed that continuous read/write operations severely degrade the overall system responsiveness in typical desktop-PC use cases.

Merely doing write/read operations on a data volume leads to stuck text and mouse cursors, seconds-long delays for simple window-context switches in X11, dropouts in low-resolution video playback etc.

Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit

How to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile (or dd if=/home/john-doe/big-testfile of=/dev/null)

- a real use scenario would be a daily disk-backup or the simple extraction of a tarball containing slightly bigger files

Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.

- Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers) compounds the issue to a painful extent.

Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.

I realize that a single heavyweight transfer slows down I/O for the corresponding transactions/processes/volumes etc.

But there needs to be a fair distribution of I/O or CPU time which leaves enough for other basic operations. And this doesn't seem to be the case with recent Linux versions.

On a side note, I've tried the BFS-patches (bfs-221 on 2.6.31): This yields significantly higher throughput when using disk encryption (50% improvement with dm-crypt/LUKS, 512 bit aes-xts-plain cipher mode). But with these patches, the responsiveness was even worse during my quick test. Switching to a text-mode console: several /minutes/ delay...

I'm attaching my .config for linux 2.6.31 (grep ^C and bzip2-ed)

#1 lore raw
Quoted 49 lines · Ulrich Lukas

Hi,

using a recent hard-/software setup, I observed that continuous read/write operations severely degrade the overall system responsiveness in typical desktop-PC use cases.

Merely doing write/read operations on a data volume leads to stuck text and mouse cursors, seconds-long delays for simple window-context switches in X11, dropouts in low-resolution video playback etc.

Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit

How to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile (or dd if=/home/john-doe/big-testfile of=/dev/null)

- a real use scenario would be a daily disk-backup or the simple extraction of a tarball containing slightly bigger files

Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.

- Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers) compounds the issue to a painful extent.

Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.

I realize that a single heavyweight transfer slows down I/O for the corresponding transactions/processes/volumes etc.

But there needs to be a fair distribution of I/O or CPU time which leaves enough for other basic operations. And this doesn't seem to be the case with recent Linux versions.

Quoted 9 lines

On a side note, I've tried the BFS-patches (bfs-221 on 2.6.31): This yields significantly higher throughput when using disk encryption (50% improvement with dm-crypt/LUKS, 512 bit aes-xts-plain cipher mode). But with these patches, the responsiveness was even worse during my quick test. Switching to a text-mode console: several /minutes/ delay...

I'm attaching my .config for linux 2.6.31 (grep ^C and bzip2-ed)

I've seen similar things with my Intel p35 (q6600) system, especially under 64bit. Heavy i/o makes the rest of the system perform very badly. I have however not noticed nearly as much of a problem with my AMD (SB700&SB750) systems.

Signature · 2 lines
Thomas Fjellstrom
[email protected]
#2 lore raw
In reply to Ulrich Lukas
Quoted 11 lines · On Sun, 20 Sep 2009 05:08:43 +0200 Ulrich Lukas…

Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.

- Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers) compounds the issue to a painful extent.

Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.

this has nothing to do with the cpu scheduler....
can you try the following two things?

echo 4096 > /sys/block/sda/queue/nr_requests for i in pidof kjournald ; do ionice -c1 -p $i ; done

to see if they help?

they tweak the IO scheduler... I assume you're using CFQ.

#3 lore raw
Quoted 4 lines · Arjan van de Ven

can you try the following two things?

echo 4096 > /sys/block/sda/queue/nr_requests for i in pidof kjournald ; do ionice -c1 -p $i ; done

I tried this just now, but it doesn't change things much. I can do a more testing later today.

Quoted 1 line

I assume you're using CFQ.

Yes, it's CFQ. (see .config of my previous posting)

Filesystem is EXT4.

#4 lore raw

(Please don't strip the CC list when you reply. Use the "Reply to all" functionality of your mail client and keep all CCs.)

Quoted 10 lines · Ulrich Lukas
Quoted 4 lines · Arjan van de Ven

can you try the following two things?

echo 4096> /sys/block/sda/queue/nr_requests for i in pidof kjournald ; do ionice -c1 -p $i ; done

I tried this just now, but it doesn't change things much. I can do a more testing later today.

Quoted 5 lines
Quoted 1 line

I assume you're using CFQ.

Yes, it's CFQ. (see .config of my previous posting)

Filesystem is EXT4.

I can also confirm total lock-up when copying a big file (Intel Core 2 Duo, Intel X38 chipset). Everything locks-up for several seconds during the copying. I'm also on ext4. I've been told on various mailing lists that this is normal, expected and unavoidable so I pretty much think the only solution is to avoid heavy I/O :P

#5 lore raw
In reply to Ulrich Lukas
Quoted 8 lines · Ulrich Lukas
Quoted 4 lines · Arjan van de Ven

can you try the following two things?

echo 4096 > /sys/block/sda/queue/nr_requests for i in pidof kjournald ; do ionice -c1 -p $i ; done

I tried this just now, but it doesn't change things much. I can do a more testing later today.

Try ionice -c3 for your dd.

Quoted 5 lines
Quoted 1 line

I assume you're using CFQ.

Yes, it's CFQ. (see .config of my previous posting)

Filesystem is EXT4.

Mine is ext3.

I can easily reproduce pain. The attached cheezy test script, when run back to back with no args (stock IO), regularly produces beauties:

Performance counter stats for 'sh -c konsole -e exit':

     257.670601  task-clock-msecs         #      0.004 CPUs
           1036  context-switches         #      0.004 M/sec
             30  CPU-migrations           #      0.000 M/sec
           6317  page-faults              #      0.025 M/sec
      471734719  cycles                   #   1830.767 M/sec
      393365060  instructions             #      0.834 IPC
        7849464  cache-references         #     30.463 M/sec
        1038976  cache-misses             #      4.032 M/sec

65.766937233 seconds time elapsed

Yup, a full minute to get a binary loaded and fired up. Seems it and others get evicted by the VM. Getting whatever back into ram is painful to listen to (poor disk/me.. seek latency eating MUCH time).

Calling script back to back with args 1 (realtime foreground) 3 (idle background) does best:

Performance counter stats for 'sh -c konsole -e exit':

     169.915036  task-clock-msecs         #      0.077 CPUs
            439  context-switches         #      0.003 M/sec
              9  CPU-migrations           #      0.000 M/sec
           6314  page-faults              #      0.037 M/sec
      363128252  cycles                   #   2137.117 M/sec
      367410945  instructions             #      1.012 IPC
        5822302  cache-references         #     34.266 M/sec
         437189  cache-misses             #      2.573 M/sec
    2.213079979  seconds time elapsed

Args 2 (best effort foreground) and 3 (idle background);

Performance counter stats for 'sh -c konsole -e exit':

     176.689856  task-clock-msecs         #      0.015 CPUs
            357  context-switches         #      0.002 M/sec
             15  CPU-migrations           #      0.000 M/sec
           6314  page-faults              #      0.036 M/sec
      374861947  cycles                   #   2121.582 M/sec
      368929476  instructions             #      0.984 IPC
        5806286  cache-references         #     32.861 M/sec
         496651  cache-misses             #      2.811 M/sec

12.036165968 seconds time elapsed

-Mike

#6 lore raw
Quoted 74 lines · Mike Galbraith
Quoted 10 lines · Ulrich Lukas
Quoted 5 lines · Arjan van de Ven

can you try the following two things?

echo 4096> /sys/block/sda/queue/nr_requests for i in pidof kjournald ; do ionice -c1 -p $i ; done

I tried this just now, but it doesn't change things much. I can do a more testing later today.

Try ionice -c3 for your dd.

Quoted 6 lines
Quoted 2 lines

I assume you're using CFQ.

Yes, it's CFQ. (see .config of my previous posting)

Filesystem is EXT4.

Mine is ext3.

I can easily reproduce pain. The attached cheezy test script, when run back to back with no args (stock IO), regularly produces beauties:

Performance counter stats for 'sh -c konsole -e exit':

      257.670601  task-clock-msecs         #      0.004 CPUs
            1036  context-switches         #      0.004 M/sec
              30  CPU-migrations           #      0.000 M/sec
            6317  page-faults              #      0.025 M/sec
       471734719  cycles                   #   1830.767 M/sec
       393365060  instructions             #      0.834 IPC
         7849464  cache-references         #     30.463 M/sec
         1038976  cache-misses             #      4.032 M/sec
    65.766937233  seconds time elapsed

Yup, a full minute to get a binary loaded and fired up. Seems it and others get evicted by the VM. Getting whatever back into ram is painful to listen to (poor disk/me.. seek latency eating MUCH time).

Calling script back to back with args 1 (realtime foreground) 3 (idle background) does best:

Performance counter stats for 'sh -c konsole -e exit':

      169.915036  task-clock-msecs         #      0.077 CPUs
             439  context-switches         #      0.003 M/sec
               9  CPU-migrations           #      0.000 M/sec
            6314  page-faults              #      0.037 M/sec
       363128252  cycles                   #   2137.117 M/sec
       367410945  instructions             #      1.012 IPC
         5822302  cache-references         #     34.266 M/sec
          437189  cache-misses             #      2.573 M/sec
     2.213079979  seconds time elapsed

Args 2 (best effort foreground) and 3 (idle background);

Performance counter stats for 'sh -c konsole -e exit':

      176.689856  task-clock-msecs         #      0.015 CPUs
             357  context-switches         #      0.002 M/sec
              15  CPU-migrations           #      0.000 M/sec
            6314  page-faults              #      0.036 M/sec
       374861947  cycles                   #   2121.582 M/sec
       368929476  instructions             #      0.984 IPC
         5806286  cache-references         #     32.861 M/sec
          496651  cache-misses             #      2.811 M/sec
Quoted 4 lines
    12.036165968  seconds time elapsed

	-Mike

Wow, at first I did not notice anything from your *.sh script, but then a few moments later, the system became vary sluggish. firefox takes about 15 seconds to load. OS is mint linux with the latest HEAD on an imac9,1.

Justin P. Mattock

#7 lore raw
Quoted 5 lines · Justin P. Mattock

Wow, at first I did not notice anything from your *.sh script, but then a few moments later, the system became vary sluggish. firefox takes about 15 seconds to load. OS is mint linux with the latest HEAD on an imac9,1.

CFQ inherits IO priority from the parent on fork(), so nicing a shell or the dd should (and does) help a LOT. It also inherits the CPU scheduler class. I just checked, and IO does indeed inherit SCHED_IDLE as well as the RT classes (SCHED_BATCH does nothing). Setting the dd to SCHED_IDLE makes a big difference, as it should, so there's a bigger hammer than nice + ionice. SCHED_IDLE is CPU_adorable + IO_nicest.

Anyway, the problem seems to have to have at least two facets:
1. streaming IO causing pagecache reclaim. That problem doesn't seem to want to go away. Fixing that won't help 2 one bit though.

2. reads are sync, more heavily affected by seek latency than writes. If the disk your binaries live on is seeking a lot, you pay a LOT.

Hohum, IO + VM = haaaaard ;-)

-Mike

#8 lore raw

Hi and thanks for your reply!

Quoted 1 line · Mike Galbraith

nicing a shell or the dd should (and does) help a LOT.

If this is the only way to influence this, maybe the default settings for the niceness of interactive and non-interactive tasks are not the best choice. (Maybe a distribution problem in this case)

Quoted 1 line

reads are sync, more heavily affected by seek latency than writes.

But how does this explain the seconds-long delays?

If an interactive process causes a lot of seeks because of reads/writes which "are sync", I see how this can greatly slow down otherwise pipelined write operations, but the other way around?

#9 lore raw
Quoted 1 line · Ulrich Lukas

Hi and thanks for your reply!

Hi.

Quoted 6 lines
Quoted 1 line · Mike Galbraith

nicing a shell or the dd should (and does) help a LOT.

If this is the only way to influence this, maybe the default settings for the niceness of interactive and non-interactive tasks are not the best choice. (Maybe a distribution problem in this case)

There is no knowledge in the CPU nor IO scheduler wrt interactive vs non-interactive. I've tinkered many times with a SCHED_INTERACTIVE class, but it's not at all an easy problem, so keeps landing on the trash heap. I could ramble on _for ever_ about that subject, but it's thankfully irrelevant to this thread ;-)

Quoted 3 lines
Quoted 1 line

reads are sync, more heavily affected by seek latency than writes.

But how does this explain the seconds-long delays?

Seek latencies are cumulative is my (wild arsed) theory.

Quoted 3 lines

If an interactive process causes a lot of seeks because of reads/writes which "are sync", I see how this can greatly slow down otherwise pipelined write operations, but the other way around?

If you seek between tiny reads, the more little reads you do, the more seeks hurt. Readahead is supposed to help, but.. the pain is there.

-Mike

#10 lore raw

FWIW, I did not notice any problems with the last kernel I had compiled in the -- IIRC -- 29-rc timeframe. Or maybe it was the 30-rc timeframe.

I had that kernel up for soemthing like five weeks, on a busy, low-ram laptop, w/o and long pauses or other noticeable pain. For at least two years before that the best this laptop could manage before latency became horrific was a fortnight, and weekly reboots were desirable.

Unfortunately, before I could send out a congratulatory note (I wanted to first confirm that the result remained) I pulled up and started see- ing even worse latency then ever.

Turning ext4 barriers off helped avoid some of the new latency, and my current compile (master as of the 17th) is much better than .31 was.

But long pauses still occur whenever paging and backgound i/o-bound tasks interact.

There was a major regression in paging performance either .30 or .31 (see above). Interestingly, top(1) also shows less swap usage for the same usage patterns than with the older kernels. I don't know which is cause or which is effect.

A git pull on a clone of Linus' tree is a great way to trigger the problem. As is using Gentoo's portage to install/upgrade anything. (Bad python; no alligator!¹) (Yes, that is a burlesque of the no doughnut joke.)

[1] Pythons which have invaded Florida have reportedly been witnessed eating alligators, according to a recent wire service article.

-JimC

Signature · 1 line
James Cloos <[email protected]>         OpenPGP: 1024D/ED7DAEA6
#11 lore raw
Quoted 6 lines · James Cloos

[...]
I had that kernel up for soemthing like five weeks, on a busy, low-ram laptop, w/o and long pauses or other noticeable pain. For at least two years before that the best this laptop could manage before latency became horrific was a fortnight, and weekly reboots were desirable.

Quoted 7 lines

Unfortunately, before I could send out a congratulatory note (I wanted to first confirm that the result remained) I pulled up and started see- ing even worse latency then ever.
[...]
A git pull on a clone of Linus' tree is a great way to trigger the problem. As is using Gentoo's portage to install/upgrade anything. (Bad python; no alligator!¹)

Fortunately, this is completely (at least here) solved by putting those in make.conf:

PORTAGE_NICENESS=19
PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"

#12 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Nikos" == Nikos Chantziaras <[email protected]> writes:

Quoted 2 lines · James Cloos
Quoted 2 lines

is a great way to trigger the problem. As is using Gentoo's portage to install/upgrade anything.

Nikos> Fortunately, this is completely (at least here) solved by putting Nikos> those in make.conf:

Nikos> PORTAGE_NICENESS=19
Nikos> PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"

I had NICENESS set, but had never noticed that PORTAGE_IONICE_COMMAND was available. I'll give that a test. Thanks.

-JimC

Signature · 1 line
James Cloos <[email protected]>         OpenPGP: 1024D/ED7DAEA6
#13 lore raw
In reply to James Cloos
Quoted 15 lines · James Cloos

FWIW, I did not notice any problems with the last kernel I had compiled in the -- IIRC -- 29-rc timeframe. Or maybe it was the 30-rc timeframe.

I had that kernel up for soemthing like five weeks, on a busy, low-ram laptop, w/o and long pauses or other noticeable pain. For at least two years before that the best this laptop could manage before latency became horrific was a fortnight, and weekly reboots were desirable.

Unfortunately, before I could send out a congratulatory note (I wanted to first confirm that the result remained) I pulled up and started see- ing even worse latency then ever.

Turning ext4 barriers off helped avoid some of the new latency, and my current compile (master as of the 17th) is much better than .31 was.

Quoted 7 lines

But long pauses still occur whenever paging and backgound i/o-bound tasks interact.

There was a major regression in paging performance either .30 or .31 (see above). Interestingly, top(1) also shows less swap usage for the same usage patterns than with the older kernels. I don't know which is cause or which is effect.

If someone has a definite good/bad cutoff for the problem they're seeing, a git bisect (yeah, takes time) would be a good thing to try.

The "interactivity while saturating root disk" thing isn't a new problem though, I can recall that being a problem to varying degrees forever. (forever being defined as feb '93 -> today;)

Quoted 4 lines

A git pull on a clone of Linus' tree is a great way to trigger the problem. As is using Gentoo's portage to install/upgrade anything. (Bad python; no alligator!¹) (Yes, that is a burlesque of the no doughnut joke.)

Git doesn't cause me any woes whatsoever, but my box's primary drive isn't a slow laptop drive, ram isn't tight, and it has 4 cores.

Dunno.

If you're using the CFQ IO scheduler, IO bandwidth follows CPU nice level, so tweaking nice tweaks both CPU and IO in one go. ionice can also be used to tweak IO independently.

I don't know if distros provide a tool that supports SCHED_IDLE, if so, you can use that to set both CPU and IO usage to IDLE class in one go.

If you don't have such a tool, you can try the attached for background loads. Someone posted it to LKML in '98, and I've been updating/using it for experiments ever since. Just do schedctl -I background-job, and the job will be as polite as CPU and IO schedulers can make it.

-Mike

#include <unistd.h>
#include <sched.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

#ifndef SCHED_BATCH
#define SCHED_BATCH 3
#endif
#ifndef SCHED_ISO
#define SCHED_ISO 4
#endif
#ifndef SCHED_IDLE
#define SCHED_IDLE 5
#endif
#ifndef SCHED_SYNC
#define SCHED_SYNC 6
#endif
#ifndef SCHED_RESET_ON_FORK
#define SCHED_RESET_ON_FORK 0x40000000
#endif

void usage();
void version();
void info(int pid);
void setsched(int pid, int policy, int pri, char flags);

unsigned char *ourname;
static int reset_flag;

/* flags */
#define GOTPID 1
#define GOTPOLICY 2
#define GOTPRI 4
#define GOTCMD 8
#define GOTINFO 16

int main(argc, argv)
unsigned int argc;
unsigned char *argv[];

{
	int pid;
	int policy;
	int pri = 0;
	int i1;
	int i2;
	unsigned long l1;
	char *ptr;
	char flags = 0;

	ourname = argv[0];

	if (argc <= 1) {
		usage();
		exit(1);
	}

	for (i1 = 1; i1 < argc; i1++)
		if (argv[i1][0] == '-') {	/* switch */
			if (argv[i1][1] >= '0' && argv[i1][1] <= '9') {
				l1 = strtol(&argv[i1][1], &ptr, 10);
				if (*ptr || l1 < 0 || l1 > 0x7FFF) {
					fprintf(stderr,
						"invalid priority: %s\n",
						argv[i1]);
					exit(1);
				}
				pri = l1;
				flags |= GOTPRI;
				continue;
			}
			if (argv[i1][1] == '-') {	/* long opt. */
				if (!strcasecmp("help", &argv[i1][2])) {
					usage();
					exit(0);
				}
				if (!strcasecmp("version", &argv[i1][2])) {
					version();
					exit(0);
				}
				if (!strcasecmp("info", &argv[i1][2])) {
				      INFO:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
							}
							flags |= GOTPID;
						}
					}
					if (flags & GOTPID) {
						info(pid);
						exit(0);
					} else {
						flags |= GOTINFO;
//                                              fprintf(stderr,"no PID given\n");
//                                              exit(1);
					}
					continue;
				}	/* end of --info */
				if (!strcasecmp("pid", &argv[i1][2])) {
				      PID:
					i1++;
					if (i1 >= argc) {
						fprintf(stderr,
							"option requires an argument\n");
						exit(1);
					} else {
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr || l1 < 0
						    || l1 > 0x7FFF) {
							fprintf(stderr,
								"invalid PID: %s\n",
								argv[i1]);
							exit(1);
						}
						if (!l1)
							pid = getpid();
						else
							pid = l1;
						flags |= GOTPID;
					}
					continue;
				}	/* end of --pid */
				if (!strcasecmp("priority", &argv[i1][2])
				    || !strcasecmp("pri", &argv[i1][2])) {
				      PRIORITY:
					i1++;
					if (i1 >= argc) {
						fprintf(stderr,
							"option requires an argument\n");
						exit(1);
					} else {
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							fprintf(stderr,
								"invalid priority: %s\n",
								argv[i1]);
							exit(1);
						}
						pri = l1;
						flags |= GOTPRI;
					}
					continue;
				}	/* end of --priority */
				if (!strcasecmp("other", &argv[i1][2])
				    || !strcasecmp("normal", &argv[i1][2])) {
				      OTHER:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_OTHER;
					continue;
				}	/* end of --other */
				if (!strcasecmp("batch", &argv[i1][2])) {
				      BATCH:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_BATCH;
					continue;
				}	/* end of --batch */
				if (!strcasecmp("idle", &argv[i1][2])) {
				      IDLE:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_IDLE;
					continue;
				}	/* end of --idle */
				if (!strcasecmp("sync", &argv[i1][2])) {
				      SYNC:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_SYNC;
					continue;
				}	/* end of --sync */
				if (!strcasecmp("reset", &argv[i1][2])) {
				      RESET:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					reset_flag = SCHED_RESET_ON_FORK;
					continue;
				}	/* end of --reset */
				if (!strcasecmp("fifo", &argv[i1][2])) {
				      FIFO:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_FIFO;
					continue;
				}	/* end of --fifo */
				if (!strcasecmp("rr", &argv[i1][2])
				    || !strcasecmp("round-robin",
						   &argv[i1][2])) {
				      RR:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_RR;
					continue;
				}
				/* end of --rr */
				fprintf(stderr,
					"warning: unknown long switch: %s\n",
					&argv[i1][2]);
				continue;
			}
			for (i2 = 1; i2 < strlen(argv[i1]); i2++)
				switch (argv[i1][i2]) {
				case 'h':
				case 'H':
				case '?':
					usage();
					exit(0);
				case 'i':
					goto INFO;
				case 'p':
					goto PID;
				case 'P':
					goto PRIORITY;
				case 'r':
					goto RESET;
				case 'N':
				case 'O':
					goto OTHER;
				case 'B':
					goto BATCH;
				case 'I':
					goto IDLE;
				case 'S':
					goto SYNC;
				case 'F':
					goto FIFO;
				case 'R':
					goto RR;
				default:
					fprintf(stderr,
						"warning: unknown switch: %c\n",
						argv[i1][i2]);
					break;
				}
		} else {	/* command */
//                      printf("command: %s\n",argv[i1]);
			if (flags & GOTPID) {
				fprintf(stderr,
					"warning: both PID and command given\n");
			} else
				pid = getpid();
			if (!(flags & (GOTPOLICY | GOTPRI))) {
				fprintf(stderr,
					"warning: neither policy nor priority given\n");
			}

			flags |= GOTCMD;

			setsched(pid, policy, pri, flags);

			execvp(argv[i1], (char **)(&argv[i1]));
			if (errno) {
				printf("%s: error: %s\n", argv[i1],
				       strerror(errno));
				exit(1);
			}
			break;
		}

	if (reset_flag && !(flags & GOTPOLICY)) {
		policy = sched_getscheduler(pid);
		flags |= GOTPOLICY;
	}

	if (flags & GOTINFO) {
		if (!(flags & GOTPID)) {
			fprintf(stderr, "no PID given\n");
			exit(1);
		}
		info(pid);
	}

	if (!(flags & (GOTCMD | GOTINFO))) {
		if (!(flags & GOTPID)) {
			fprintf(stderr, "neither PID nor command given\n");
			exit(1);
		}
		if (!(flags & (GOTPOLICY | GOTPRI))) {
			fprintf(stderr,
				"warning: neither policy nor priority given\n");
		}
		setsched(pid, policy, pri, flags);
	}
	exit(0);
}

void usage()

{
	printf("Usage: %s [options] [command [params...]]\n\n", ourname);
	printf("  --pid, -p <n>		specify process id\n");
	printf("  --fifo, -F [pid]	set scheduling policy to FIFO\n");
	printf("  --rr, -R [pid]	set scheduling policy to RR\n");
	printf("  --batch, -B [pid]	set scheduling policy to BATCH\n");
	printf("  --idle, -I [pid]	set scheduling policy to IDLE\n");
	printf("  --sync, -S [pid]	set scheduling policy to SYNC\n");
	printf("  --other, --normal,\n");
	printf("      -N, -O [pid]	specify normal scheduling\n");
	printf("  --reset, -r 		set SCHED_RESET_ON_FORK flag\n");
	printf("  --priority, --pri,\n");
	printf("      -<n>		set priority\n");
	printf("  --info, -i [pid]	show scheduling data\n");
	printf("  --help, -h, -H, -?	display this text\n");
	printf("  --version		show version data\n\n");
}

void version()

{
	printf("schedctl, version 0.95 (beta), 8-May-1998\n");
	printf("report bugs to [email protected] or [email protected]\n");
}

void info(int pid)

{
	struct sched_param sp;
	int i1;

	i1 = sched_getscheduler(pid);
	if (errno == ESRCH) {
		fprintf(stderr, "process not found\n");
		return;
	}

	if (i1 & SCHED_RESET_ON_FORK) {
		i1 &= ~SCHED_RESET_ON_FORK;
		reset_flag = SCHED_RESET_ON_FORK;
	}

	printf("scheduling policy of process %d is ", pid);
	switch (i1) {
	case SCHED_FIFO:
		printf("FIFO");
		break;
	case SCHED_RR:
		printf("RR");
		break;
	case SCHED_OTHER:
		printf("OTHER (normal)");
		break;
	case SCHED_BATCH:
		printf("BATCH");
		break;
	case SCHED_ISO:
		printf("ISO");
		break;
	case SCHED_IDLE:
		printf("IDLE");
		break;
	case SCHED_SYNC:
		printf("SYNC");
		break;
	default:
		printf("**UNKNOWN**");
	}
	if (reset_flag)
		printf(" SCHED_RESET_ON_FORK");

	sched_getparam(pid, &sp);
	printf(", priority is %d\n", sp.sched_priority);
}

void setsched(int pid, int policy, int pri, char flags)

{
	struct sched_param sp;
	int i1;

	i1 = sched_getscheduler(pid);

	if (errno == ESRCH) {
		fprintf(stderr, "process not found\n");
		return;
	}

	if (!(flags & GOTPOLICY))
		policy = i1;
	sched_getparam(pid, &sp);
	if (!(flags & GOTPRI))
		pri = sp.sched_priority;
	else if (pri < sched_get_priority_min(policy)
		 || pri > sched_get_priority_max(policy)) {
		fprintf(stderr, "priority (%d) out of range\n", pri);
		return;
	}
	if (policy == SCHED_FIFO || policy == SCHED_RR) {
		if (!pri) {
			if (!sp.sched_priority)
				sp.sched_priority++;
		} else
			sp.sched_priority = pri;
	} else if (pri && (policy == SCHED_OTHER || policy == SCHED_BATCH ||
			   policy == SCHED_ISO || policy == SCHED_IDLE || policy == SCHED_SYNC))
		sp.sched_priority = 0;
	else
		sp.sched_priority = pri;
	if (sched_setscheduler(pid, policy|reset_flag, &sp)) {
		fprintf(stderr, "error #%d: %s\n", errno, strerror(errno));
		return;
	}

	i1 = sched_getscheduler(pid);
	if (i1 & SCHED_RESET_ON_FORK)
		i1 &= ~SCHED_RESET_ON_FORK;
	sched_getparam(pid, &sp);
	if (flags & GOTCMD)
		return;
	printf("scheduling policy of process %d set to ", pid);
	switch (i1) {
		case SCHED_FIFO:
			printf("FIFO");
			break;
		case SCHED_RR:
			printf("RR");
			break;
		case SCHED_OTHER:
			printf("OTHER");
			break;
		case SCHED_BATCH:
			printf("BATCH");
			break;
		case SCHED_ISO:
			printf("ISO");
			break;
		case SCHED_IDLE:
			printf("IDLE");
			break;
		case SCHED_SYNC:
			printf("SYNC");
			break;
	}

	if (reset_flag)
		printf(" SCHED_RESET_ON_FORK");
	printf("\n");
}
#14 lore raw
Quoted 2 lines · Mike Galbraith

I don't know if distros provide a tool that supports SCHED_IDLE, if so, you can use that to set both CPU and IO usage to IDLE class in one go.

P.S. I just remembered, don't use SCHED_IDLE with kernels < 2.6.28. There are two patches wrt this scheduling class which did not go to the stable trees less than 2.6.28.

-Mike

#15 lore raw
Quoted 7 lines · Mike Galbraith
Quoted 2 lines · Mike Galbraith

I don't know if distros provide a tool that supports SCHED_IDLE, if so, you can use that to set both CPU and IO usage to IDLE class in one go.

P.S. I just remembered, don't use SCHED_IDLE with kernels < 2.6.28. There are two patches wrt this scheduling class which did not go to the stable trees less than 2.6.28.

Since 2.6.27 is long-term, maybe they should be sent to 2.6.27? It is weird to have a long-term stable kernel for which the recommendation is "don't use SCHED_IDLE".

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
#16 lore raw
Quoted 5 lines · Henrique de Moraes Holschuh
Quoted 4 lines · Mike Galbraith
Quoted 2 lines · Mike Galbraith

I don't know if distros provide a tool that supports SCHED_IDLE, if so, you can use that to set both CPU and IO usage to IDLE class in one go.

Quoted 7 lines
Quoted 3 lines

P.S. I just remembered, don't use SCHED_IDLE with kernels < 2.6.28. There are two patches wrt this scheduling class which did not go to the stable trees less than 2.6.28.

Since 2.6.27 is long-term, maybe they should be sent to 2.6.27? It is weird to have a long-term stable kernel for which the recommendation is "don't use SCHED_IDLE".

Yeah, that's what I thought too, but apparently maintainers didn't think SCHED_IDLE was common/important enough.

-Mike

#17 lore raw
In reply to Mike Galbraith
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Mike" == Mike Galbraith <[email protected]> writes:

Mike> If someone has a definite good/bad cutoff for the problem they're Mike> seeing, a git bisect (yeah, takes time) would be a good thing to try.

Yes, I wanted to try that for this, but given how long it takes to compile a kernel on this box, and how long it takes to get everything up so that I can test my real-life load, added to the difference in performance after each day of uptime, I could do at best one kernel per day; it would take weeks to finish.

That said, I jsut took a look at the git log for my /boot/grub repo and can confirm that the last really good kernel -- the one I had up for more than a month for the first time in years -- was pulled between 2.6.30-rc6 and 2.6.30-rc7 and installed on 2009/05/17 which, since I keep .config in git in my compile repo, lets me narrow down the make oldconfig to 2009/05/17 18:37:24Z.

So, whatever Linus' tree looked like on that date is GOOD. His last commit before that was 0f6f49a8cd0163fdb1723ed29f01fc65177108dc.

My next make oldconfig was on 2009/06/25 04:32:03Z, when I started testing the radeon kms. (I'm back to non-KMS now.) There were too many issues directly caused by KMS itself for me to comment on non- KMS kernel issues from the kernels I ran with KMS enabled.

My return from KMS as on 2009/08/31; Linus' last commit before I did that was adda766193ea1cf3137484a9521972d080d0b7af.

To give an idea of how little that narrows things:

:; git log 0f6f49a8cd..adda766193|egrep -c ^commit 12094

For those who use the tars, that means 30-rc6 was fast and 31-rc8 slow.

Mike> Git doesn't cause me any woes whatsoever, but my box's primary Mike> drive isn't a slow laptop drive, ram isn't tight,

That does make a difference. Git is *great* when it fits into ram, but a pull or merge on a clone of Linus' tree takes a quarter gig of VM on x86-32, which guarantees paging, including to the swap partitions. (The laptop is old enough to support three platters — including the optical — so I have my primary swap partition on the second drive to reduce some of the load to the / /var /boot drive. The first disk's swap partition only gets hit when I boot w/o the second drive. It helps.)

The problem isn't that paging occurs, but rather that now, when paging occurs, said paging is much slower than it had been.

IOW, the priority of paging vs other disk I/O has changed for the worse, at some point after 0f6f49a8cd and before adda766193. But, as I wrote above, to actually test kernels between those to determine where would take weeks.

-JimC

Signature · 1 line
James Cloos <[email protected]>         OpenPGP: 1024D/ED7DAEA6
#18 lore raw
In reply to Arjan van de Ven
Quoted 4 lines · Arjan van de Ven
Quoted 4 lines

can you try the following two things?

echo 4096 > /sys/block/sda/queue/nr_requests for i in pidof kjournald ; do ionice -c1 -p $i ; done

No, just tested again. Only a marginal improvement if there is any.

#19 lore raw
In reply to Ulrich Lukas
Quoted 11 lines · Ulrich Lukas

Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit

How to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile (or dd if=/home/john-doe/big-testfile of=/dev/null)

Quoted 7 lines

- a real use scenario would be a daily disk-backup or the simple extraction of a tarball containing slightly bigger files

Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.

I guess that switching from CFQ to deadline I/O scheduler improves the situation, right?

Signature · 2 lines
Jiri Kosina
SUSE Labs, Novell Inc.
#20 lore raw
Quoted 17 lines · Jiri Kosina
Quoted 15 lines · Ulrich Lukas

Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit

How to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile (or dd if=/home/john-doe/big-testfile of=/dev/null)

- a real use scenario would be a daily disk-backup or the simple extraction of a tarball containing slightly bigger files

Quoted 6 lines
Quoted 3 lines

Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.

I guess that switching from CFQ to deadline I/O scheduler improves the situation, right?

For example on my desktop, switching to deadline slightly improves the
situation but the machine is still mostly unusable while dd is running... I'll try to debug it some more to see whether it can be somehow helped.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#21 lore raw
Quoted 22 lines · Jan Kara
Quoted 21 lines · Jiri Kosina
Quoted 18 lines · Ulrich Lukas

Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit

How to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile (or dd if=/home/john-doe/big-testfile of=/dev/null)

- a real use scenario would be a daily disk-backup or the simple extraction of a tarball containing slightly bigger files

Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.

Quoted 5 lines
Quoted 2 lines

I guess that switching from CFQ to deadline I/O scheduler improves the situation, right?

For example on my desktop, switching to deadline slightly improves the
situation but the machine is still mostly unusable while dd is running... I'll try to debug it some more to see whether it can be somehow helped.

Datapoint:

echo 1 > /sys/block/$disk/queue/iosched/quantum seems to make a huge difference. Stock is 4. (wisdom of setting it to 1? no idea)

This...
marge:/root/tmp # ./testo.sh
quantum is 4, setting to 1
dd if=3DMark2000.mkv of=/tmp/3DMark2000.mkv 204529+1 records in
204529+1 records out
104718895 bytes (105 MB) copied, 1.62672 s, 64.4 MB/s perf sched record -o /tmp/mplayer-c2-c2.data (tmpfs)& mplayer /tmp/3DMark2000.mkv&
dd if=/dev/zero of=crud-25859&

sleep 180
timing konsole -e exit

Performance counter stats for 'sh -c konsole -e exit':

     173.615837  task-clock-msecs         #      0.031 CPUs
            431  context-switches         #      0.002 M/sec
             10  CPU-migrations           #      0.000 M/sec
           6314  page-faults              #      0.036 M/sec
      368422027  cycles                   #   2122.053 M/sec
      376283291  instructions             #      1.021 IPC
        5883195  cache-references         #     33.886 M/sec
         444333  cache-misses             #      2.559 M/sec
    5.597628358  seconds time elapsed

...is pretty typical, and a LOT better than the 60-70 seconds I was seeing, or the minutes I sometimes saw when playing with evolution.

I still occasionally see some largish numbers though, and suspect that depends on how much cache was evicted.

-Mike

#22 lore raw
Quoted 2 lines · On Mon, 21 Sep 2009 09:25:08 +0200 Mike Galbraith <[email protected]> wrote:

I still occasionally see some largish numbers though, and suspect that depends on how much cache was evicted.

if you run latencytop it'll at least tell you what kind of operating was going on that hit the delay...

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#23 lore raw
Quoted 7 lines · Arjan van de Ven
Quoted 2 lines · On Mon, 21 Sep 2009 09:25:08 +0200 Mike Galbraith <[email protected]> wrote:

I still occasionally see some largish numbers though, and suspect that depends on how much cache was evicted.

if you run latencytop it'll at least tell you what kind of operating was going on that hit the delay...

One of the really bad ones for evolution is below. It was so bad that I had _plenty_ of time to get to a GUI free shell, and poke SysRq-W before killing the darn dd.

[ 5112.427012] SysRq : Show Blocked State
[ 5112.430995]   task                        PC stack   pid father
[ 5112.430995] kjournald     D 0000000000000002     0  1230      2 0x00000000
[ 5112.430995]  ffff8800378d7c50 0000000000000046 0000000000000000 ffff8800a5ed53b0
[ 5112.430995]  ffff8800bd2c0000 ffff8800378d7fd8 000000000000dd38 0000000000013480
[ 5112.430995]  0000000000004000 0000000000013480 ffff8800378d7bf0 00001000810e0c0a
[ 5112.430995] Call Trace:
[ 5112.430995]  [<ffffffff812cacb4>] io_schedule+0x80/0xc6
[ 5112.430995]  [<ffffffff8110df99>] sync_buffer+0x40/0x44
[ 5112.430995]  [<ffffffff812cb373>] __wait_on_bit+0x4c/0x7e
[ 5112.430995]  [<ffffffff8110df59>] ? sync_buffer+0x0/0x44
[ 5112.430995]  [<ffffffff8110df59>] ? sync_buffer+0x0/0x44
[ 5112.430995]  [<ffffffff812cb414>] out_of_line_wait_on_bit+0x6f/0x7c
[ 5112.430995]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.430995]  [<ffffffff8110deff>] __wait_on_buffer+0x24/0x26
[ 5112.430995]  [<ffffffffa0077e91>] journal_commit_transaction+0xca4/0x138b [jbd]
[ 5112.430995]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.430995]  [<ffffffffa007ba02>] kjournald+0x168/0x397 [jbd]
[ 5112.430995]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.430995]  [<ffffffffa007b89a>] ? kjournald+0x0/0x397 [jbd]
[ 5112.430995]  [<ffffffff81065acb>] kthread+0x82/0x8a
[ 5112.625071]  [<ffffffff8100ccca>] child_rip+0xa/0x20
[ 5112.625071]  [<ffffffff8100c669>] ? restore_args+0x0/0x30
[ 5112.625071]  [<ffffffff81065a49>] ? kthread+0x0/0x8a
[ 5112.625071]  [<ffffffff8100ccc0>] ? child_rip+0x0/0x20
[ 5112.625071] flush-8:16    D ffff8800bd2097f0     0  1236      2 0x00000000
[ 5112.625071]  ffff8800bd209790 0000000000000046 0000000000000000 ffffffff810beb58
[ 5112.625071]  ffff8800bd209740 ffff8800bd209fd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 0000000001080001 0000000000000080
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffff810beb58>] ? get_page_from_freelist+0x33d/0x4a6
[ 5112.625071]  [<ffffffff81174cb6>] ? cfq_may_queue+0x60/0xe1
[ 5112.625071]  [<ffffffff812cacb4>] io_schedule+0x80/0xc6
[ 5112.625071]  [<ffffffff81169184>] get_request_wait+0xf3/0x18f
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff8115f142>] ? elv_merge+0x37/0x19d
[ 5112.625071]  [<ffffffff8116954d>] __make_request+0x32d/0x45b
[ 5112.625071]  [<ffffffff811678ae>] generic_make_request+0x339/0x38b
[ 5112.625071]  [<ffffffff811679d4>] submit_bio+0xd4/0xdd
[ 5112.625071]  [<ffffffff8110ccfc>] submit_bh+0x106/0x129
[ 5112.625071]  [<ffffffff8110f7a4>] __block_write_full_page+0x1eb/0x2d1
[ 5112.625071]  [<ffffffff8110e197>] ? end_buffer_async_write+0x0/0x13f
[ 5112.625071]  [<ffffffff8110f94f>] block_write_full_page_endio+0xc5/0xca
[ 5112.625071]  [<ffffffff8110f969>] block_write_full_page+0x15/0x17
[ 5112.625071]  [<ffffffffa0095e1f>] ext3_writeback_writepage+0x71/0x142 [ext3]
[ 5112.625071]  [<ffffffff810bf647>] __writepage+0x17/0x34
[ 5112.625071]  [<ffffffff810bfd9f>] write_cache_pages+0x250/0x393
[ 5112.625071]  [<ffffffff810bf630>] ? __writepage+0x0/0x34
[ 5112.625071]  [<ffffffff810bff06>] generic_writepages+0x24/0x2a
[ 5112.625071]  [<ffffffff810bff34>] do_writepages+0x28/0x2a
[ 5112.625071]  [<ffffffff81106b94>] writeback_single_inode+0x181/0x395
[ 5112.625071]  [<ffffffff811077fc>] writeback_inodes_wb+0x341/0x42f
[ 5112.625071]  [<ffffffff81107a2b>] wb_writeback+0x141/0x194
[ 5112.625071]  [<ffffffff81095031>] ? call_rcu_sched+0x15/0x17
[ 5112.625071]  [<ffffffff81095041>] ? call_rcu+0xe/0x10
[ 5112.625071]  [<ffffffff81107bb9>] wb_do_writeback+0x7a/0x16b
[ 5112.625071]  [<ffffffff81107ce9>] bdi_writeback_task+0x3f/0xad
[ 5112.625071]  [<ffffffff810cca5d>] ? bdi_start_fn+0x0/0xd7
[ 5112.625071]  [<ffffffff810ccad3>] bdi_start_fn+0x76/0xd7
[ 5112.625071]  [<ffffffff810cca5d>] ? bdi_start_fn+0x0/0xd7
[ 5112.625071]  [<ffffffff81065acb>] kthread+0x82/0x8a
[ 5112.625071]  [<ffffffff8100ccca>] child_rip+0xa/0x20
[ 5112.625071]  [<ffffffff8100c669>] ? restore_args+0x0/0x30
[ 5112.625071]  [<ffffffff81065a49>] ? kthread+0x0/0x8a
[ 5112.625071]  [<ffffffff8100ccc0>] ? child_rip+0x0/0x20
[ 5112.625071] console-kit-d D ffff8800b788c480     0  2963      1 0x00000000
[ 5112.625071]  ffff8800bc59f6b8 0000000000000082 0000000000000000 ffffffffa0076655
[ 5112.625071]  ffff8800bc59f628 ffff8800bc59ffd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 ffff8800bd2f3c00 0000000000000000
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffffa0076655>] ? do_get_write_access+0x443/0x48b [jbd]
[ 5112.625071]  [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071]  [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071]  [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071]  [<ffffffffa0092901>] ext3_new_blocks+0x333/0x669 [ext3]
[ 5112.625071]  [<ffffffffa0096654>] ext3_get_blocks_handle+0x3ec/0x92d [ext3]
[ 5112.625071]  [<ffffffff8110c866>] ? alloc_buffer_head+0x1e/0x51
[ 5112.625071]  [<ffffffffa0096c3a>] ext3_get_block+0xa5/0xe3 [ext3]
[ 5112.625071]  [<ffffffff8110ec0a>] __block_prepare_write+0x1f0/0x3d2
[ 5112.625071]  [<ffffffffa0076e49>] ? start_this_handle+0x37d/0x474 [jbd]
[ 5112.625071]  [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071]  [<ffffffffa007709a>] ? journal_start+0x60/0xca [jbd]
[ 5112.625071]  [<ffffffff8110ef7c>] block_write_begin+0x85/0xd5
[ 5112.625071]  [<ffffffffa0098130>] ext3_write_begin+0xf0/0x1e7 [ext3]
[ 5112.625071]  [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071]  [<ffffffff810ba9e8>] generic_file_buffered_write+0x11f/0x277
[ 5112.625071]  [<ffffffff810bafe1>] __generic_file_aio_write+0x351/0x385
[ 5112.625071]  [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad
[ 5112.625071]  [<ffffffff810e75b4>] do_sync_write+0xec/0x132
[ 5112.625071]  [<ffffffff810c176a>] ? lru_cache_add_lru+0x2b/0x2d
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff8117c94f>] ? __up_read+0x9e/0xa7
[ 5112.625071]  [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071]  [<ffffffff810e8053>] vfs_write+0xb3/0x16f
[ 5112.625071]  [<ffffffff810e81dd>] sys_write+0x4c/0x74
[ 5112.625071]  [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] dd            D ffff8800b788c480     0  8587   6686 0x00000000
[ 5112.625071]  ffff8800a5e2faa8 0000000000000086 0000000000000000 ffffffff812cab8c
[ 5112.625071]  ffff8800a5e2fbe0 ffff8800a5e2ffd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 ffff8800a5e2fa28 ffffffff810b9b1c
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffff812cab8c>] ? thread_return+0x60/0x108
[ 5112.625071]  [<ffffffff810b9b1c>] ? find_get_page+0x28/0x89
[ 5112.625071]  [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071]  [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071]  [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071]  [<ffffffffa00953f5>] ext3_reserve_inode_write+0x44/0x80 [ext3]
[ 5112.625071]  [<ffffffffa0095460>] ext3_mark_inode_dirty+0x2f/0x4c [ext3]
[ 5112.625071]  [<ffffffffa0095605>] ext3_dirty_inode+0xab/0xc2 [ext3]
[ 5112.625071]  [<ffffffff81107392>] __mark_inode_dirty+0x34/0x15d
[ 5112.625071]  [<ffffffff810fc6af>] file_update_time+0xbe/0x102
[ 5112.625071]  [<ffffffff810baef7>] __generic_file_aio_write+0x267/0x385
[ 5112.625071]  [<ffffffff810b8427>] ? perf_swcounter_ctx_event+0x16c/0x1c6
[ 5112.625071]  [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad
[ 5112.625071]  [<ffffffff810e75b4>] do_sync_write+0xec/0x132
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff8117ffa0>] ? __clear_user+0x17/0x55
[ 5112.625071]  [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071]  [<ffffffff810e8053>] vfs_write+0xb3/0x16f
[ 5112.625071]  [<ffffffff810e81dd>] sys_write+0x4c/0x74
[ 5112.625071]  [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] perf          D ffff8800b788c480     0  8588   6686 0x00000000
[ 5112.625071]  ffff8800a5f115d8 0000000000000082 0000000000000000 000004a57bce3b78
[ 5112.625071]  ffff8800be035f40 ffff8800a5f11fd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 ffff8800a5f11558 ffffffff810b9b1c
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffff810b9b1c>] ? find_get_page+0x28/0x89
[ 5112.625071]  [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071]  [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071]  [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071]  [<ffffffffa00953f5>] ext3_reserve_inode_write+0x44/0x80 [ext3]
[ 5112.625071]  [<ffffffffa0095460>] ext3_mark_inode_dirty+0x2f/0x4c [ext3]
[ 5112.625071]  [<ffffffffa0095605>] ext3_dirty_inode+0xab/0xc2 [ext3]
[ 5112.625071]  [<ffffffff81107392>] __mark_inode_dirty+0x34/0x15d
[ 5112.625071]  [<ffffffffa0092672>] ext3_new_blocks+0xa4/0x669 [ext3]
[ 5112.625071]  [<ffffffff81110195>] ? __bread+0x13/0x92
[ 5112.625071]  [<ffffffffa00959c4>] ? ext3_get_branch+0x7b/0xee [ext3]
[ 5112.625071]  [<ffffffffa0096654>] ext3_get_blocks_handle+0x3ec/0x92d [ext3]
[ 5112.625071]  [<ffffffffa0096c3a>] ext3_get_block+0xa5/0xe3 [ext3]
[ 5112.625071]  [<ffffffff8110ec0a>] __block_prepare_write+0x1f0/0x3d2
[ 5112.625071]  [<ffffffffa0076e49>] ? start_this_handle+0x37d/0x474 [jbd]
[ 5112.625071]  [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff8110ef7c>] block_write_begin+0x85/0xd5
[ 5112.625071]  [<ffffffffa0098130>] ext3_write_begin+0xf0/0x1e7 [ext3]
[ 5112.625071]  [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071]  [<ffffffff810ba9e8>] generic_file_buffered_write+0x11f/0x277
[ 5112.625071]  [<ffffffff810bafe1>] __generic_file_aio_write+0x351/0x385
[ 5112.625071]  [<ffffffff8100c669>] ? restore_args+0x0/0x30
[ 5112.625071]  [<ffffffff812cb922>] ? mutex_lock+0x1c/0x3b
[ 5112.625071]  [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad
[ 5112.625071]  [<ffffffff810e75b4>] do_sync_write+0xec/0x132
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff812cab6a>] ? thread_return+0x3e/0x108
[ 5112.625071]  [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071]  [<ffffffff810e8053>] vfs_write+0xb3/0x16f
[ 5112.625071]  [<ffffffff810e81dd>] sys_write+0x4c/0x74
[ 5112.625071]  [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] evolution.bin D 0000000000000082     0  8638   6627 0x00000000
[ 5112.625071]  ffff880099339c08 0000000000000086 0000000000000000 ffffffff811157a6
[ 5112.625071]  ffffffffa0096b95 ffff880099339fd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 ffffea0001c5ae08 00000000009ff29b
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffff811157a6>] ? mpage_readpages+0x10a/0x120
[ 5112.625071]  [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071]  [<ffffffff812cacb4>] io_schedule+0x80/0xc6
[ 5112.625071]  [<ffffffff810b9d22>] sync_page+0x46/0x4a
[ 5112.625071]  [<ffffffff810b9d34>] sync_page_killable+0xe/0x35
[ 5112.625071]  [<ffffffff812cb263>] __wait_on_bit_lock+0x4a/0x92
[ 5112.625071]  [<ffffffff810b9d26>] ? sync_page_killable+0x0/0x35
[ 5112.625071]  [<ffffffff810b9c54>] __lock_page_killable+0x68/0x6f
[ 5112.625071]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071]  [<ffffffff810bb480>] generic_file_aio_read+0x3be/0x5bb
[ 5112.625071]  [<ffffffff810e76e6>] do_sync_read+0xec/0x132
[ 5112.625071]  [<ffffffff810c176a>] ? lru_cache_add_lru+0x2b/0x2d
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071]  [<ffffffff810e82b5>] vfs_read+0xb0/0x16c
[ 5112.625071]  [<ffffffff810e843f>] sys_read+0x4c/0x75
[ 5112.625071]  [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] hal-acl-tool  D ffff8800b788c480     0  8675   2964 0x00000000
[ 5112.625071]  ffff880037b3db78 0000000000000082 ffff88007af907e0 ffff880062b7f7e0
[ 5112.625071]  ffff880037b3dac8 ffff880037b3dfd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 ffff880037b3db18 ffffffff8110d414
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffff8110d414>] ? __getblk+0x2d/0x2ae
[ 5112.625071]  [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071]  [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071]  [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071]  [<ffffffffa0093ef8>] ext3_new_inode+0x4d4/0x96d [ext3]
[ 5112.625071]  [<ffffffffa009afca>] ext3_create+0x76/0xf3 [ext3]
[ 5112.625071]  [<ffffffff810f2be9>] vfs_create+0xa2/0x117
[ 5112.625071]  [<ffffffff810f50fb>] do_filp_open+0x327/0xa39
[ 5112.625071]  [<ffffffff810f2e69>] ? getname+0x31/0x1b3
[ 5112.625071]  [<ffffffff810e513e>] do_sys_open+0x62/0x109
[ 5112.625071]  [<ffffffff810e5218>] sys_open+0x20/0x22
[ 5112.625071]  [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
#24 lore raw
Quoted 6 lines · Mike Galbraith
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines · On Mon, 21 Sep 2009 09:25:08 +0200 Mike Galbraith <[email protected]> wrote:

I still occasionally see some largish numbers though, and suspect that depends on how much cache was evicted.

Quoted 6 lines
Quoted 2 lines

if you run latencytop it'll at least tell you what kind of operating was going on that hit the delay...

One of the really bad ones for evolution is below. It was so bad that I had _plenty_ of time to get to a GUI free shell, and poke SysRq-W before killing the darn dd.

P.S. evolution is always journal journal and more journal.

#25 lore raw
In reply to Jiri Kosina
Quoted 2 lines · Jiri Kosina

I guess that switching from CFQ to deadline I/O scheduler improves the situation, right?

Indeed, there is an improvement. But, especially in the case with an encrypted /home, copying a large file from a fast source means I better go get another cup of coffee rather than continue working on something else in the meantime.

BTW, I've tested this with a default install of OpenSUSE 11.1, and I get basically the same behaviour.