Poor desktop responsiveness with background I/O-operations
Hi,
using a recent hard-/software setup, I observed that continuous read/write operations severely degrade the overall system responsiveness in typical desktop-PC use cases.
Merely doing write/read operations on a data volume leads to stuck text and mouse cursors, seconds-long delays for simple window-context switches in X11, dropouts in low-resolution video playback etc.
Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit
How to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile (or dd if=/home/john-doe/big-testfile of=/dev/null)
- a real use scenario would be a daily disk-backup or the simple extraction of a tarball containing slightly bigger files
Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.
- Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers) compounds the issue to a painful extent.
Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.
I realize that a single heavyweight transfer slows down I/O for the corresponding transactions/processes/volumes etc.
But there needs to be a fair distribution of I/O or CPU time which leaves enough for other basic operations. And this doesn't seem to be the case with recent Linux versions.
On a side note, I've tried the BFS-patches (bfs-221 on 2.6.31): This yields significantly higher throughput when using disk encryption (50% improvement with dm-crypt/LUKS, 512 bit aes-xts-plain cipher mode). But with these patches, the responsiveness was even worse during my quick test. Switching to a text-mode console: several /minutes/ delay...
I'm attaching my .config for linux 2.6.31 (grep ^C and bzip2-ed)
Quoted 49 lines · Ulrich Lukas
Hi,
using a recent hard-/software setup, I observed that continuous read/write operations severely degrade the overall system responsiveness in typical desktop-PC use cases.
Merely doing write/read operations on a data volume leads to stuck text and mouse cursors, seconds-long delays for simple window-context switches in X11, dropouts in low-resolution video playback etc.
Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bitHow to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile (or dd if=/home/john-doe/big-testfile of=/dev/null)- a real use scenario would be a daily disk-backup or the simple extraction of a tarball containing slightly bigger files
Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.- Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers) compounds the issue to a painful extent.
Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.
I realize that a single heavyweight transfer slows down I/O for the corresponding transactions/processes/volumes etc.
But there needs to be a fair distribution of I/O or CPU time which leaves enough for other basic operations. And this doesn't seem to be the case with recent Linux versions.
Quoted 9 lines
On a side note, I've tried the BFS-patches (bfs-221 on 2.6.31): This yields significantly higher throughput when using disk encryption (50% improvement with dm-crypt/LUKS, 512 bit aes-xts-plain cipher mode). But with these patches, the responsiveness was even worse during my quick test. Switching to a text-mode console: several /minutes/ delay...
I'm attaching my .config for linux 2.6.31 (grep ^C and bzip2-ed)
I've seen similar things with my Intel p35 (q6600) system, especially under 64bit. Heavy i/o makes the rest of the system perform very badly. I have however not noticed nearly as much of a problem with my AMD (SB700&SB750) systems.
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 11 lines · On Sun, 20 Sep 2009 05:08:43 +0200 Ulrich Lukas…
Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.- Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers) compounds the issue to a painful extent.
Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.
this has nothing to do with the cpu scheduler....
can you try the following two things?
echo 4096 > /sys/block/sda/queue/nr_requests for i in pidof kjournald ; do ionice -c1 -p $i ; done
to see if they help?
they tweak the IO scheduler... I assume you're using CFQ.
Quoted 4 lines · Arjan van de Ven
can you try the following two things?
echo 4096 >
/sys/block/sda/queue/nr_requestsfor i inpidof kjournald; do ionice -c1 -p $i ; done
I tried this just now, but it doesn't change things much. I can do a more testing later today.
Quoted 1 line
I assume you're using CFQ.
Yes, it's CFQ. (see .config of my previous posting)
Filesystem is EXT4.
(Please don't strip the CC list when you reply. Use the "Reply to all" functionality of your mail client and keep all CCs.)
Quoted 10 lines · Ulrich Lukas
Quoted 4 lines · Arjan van de Ven
can you try the following two things?
echo 4096>
/sys/block/sda/queue/nr_requestsfor i inpidof kjournald; do ionice -c1 -p $i ; doneI tried this just now, but it doesn't change things much. I can do a more testing later today.
Quoted 5 lines
Quoted 1 line
I assume you're using CFQ.
Yes, it's CFQ. (see .config of my previous posting)
Filesystem is EXT4.
I can also confirm total lock-up when copying a big file (Intel Core 2 Duo, Intel X38 chipset). Everything locks-up for several seconds during the copying. I'm also on ext4. I've been told on various mailing lists that this is normal, expected and unavoidable so I pretty much think the only solution is to avoid heavy I/O :P
Quoted 8 lines · Ulrich Lukas
Quoted 4 lines · Arjan van de Ven
can you try the following two things?
echo 4096 >
/sys/block/sda/queue/nr_requestsfor i inpidof kjournald; do ionice -c1 -p $i ; doneI tried this just now, but it doesn't change things much. I can do a more testing later today.
Try ionice -c3 for your dd.
Quoted 5 lines
Quoted 1 line
I assume you're using CFQ.
Yes, it's CFQ. (see .config of my previous posting)
Filesystem is EXT4.
Mine is ext3.
I can easily reproduce pain. The attached cheezy test script, when run back to back with no args (stock IO), regularly produces beauties:
Performance counter stats for 'sh -c konsole -e exit':
257.670601 task-clock-msecs # 0.004 CPUs
1036 context-switches # 0.004 M/sec
30 CPU-migrations # 0.000 M/sec
6317 page-faults # 0.025 M/sec
471734719 cycles # 1830.767 M/sec
393365060 instructions # 0.834 IPC
7849464 cache-references # 30.463 M/sec
1038976 cache-misses # 4.032 M/sec65.766937233 seconds time elapsed
Yup, a full minute to get a binary loaded and fired up. Seems it and others get evicted by the VM. Getting whatever back into ram is painful to listen to (poor disk/me.. seek latency eating MUCH time).
Calling script back to back with args 1 (realtime foreground) 3 (idle background) does best:
Performance counter stats for 'sh -c konsole -e exit':
169.915036 task-clock-msecs # 0.077 CPUs
439 context-switches # 0.003 M/sec
9 CPU-migrations # 0.000 M/sec
6314 page-faults # 0.037 M/sec
363128252 cycles # 2137.117 M/sec
367410945 instructions # 1.012 IPC
5822302 cache-references # 34.266 M/sec
437189 cache-misses # 2.573 M/sec2.213079979 seconds time elapsed
Args 2 (best effort foreground) and 3 (idle background);
Performance counter stats for 'sh -c konsole -e exit':
176.689856 task-clock-msecs # 0.015 CPUs
357 context-switches # 0.002 M/sec
15 CPU-migrations # 0.000 M/sec
6314 page-faults # 0.036 M/sec
374861947 cycles # 2121.582 M/sec
368929476 instructions # 0.984 IPC
5806286 cache-references # 32.861 M/sec
496651 cache-misses # 2.811 M/sec12.036165968 seconds time elapsed
-Mike
Quoted 74 lines · Mike Galbraith
Quoted 10 lines · Ulrich Lukas
Quoted 5 lines · Arjan van de Ven
can you try the following two things?
echo 4096>
/sys/block/sda/queue/nr_requestsfor i inpidof kjournald; do ionice -c1 -p $i ; doneI tried this just now, but it doesn't change things much. I can do a more testing later today.
Try ionice -c3 for your dd.
Quoted 6 lines
Quoted 2 lines
I assume you're using CFQ.
Yes, it's CFQ. (see .config of my previous posting)
Filesystem is EXT4.
Mine is ext3.
I can easily reproduce pain. The attached cheezy test script, when run back to back with no args (stock IO), regularly produces beauties:
Performance counter stats for 'sh -c konsole -e exit':
257.670601 task-clock-msecs # 0.004 CPUs 1036 context-switches # 0.004 M/sec 30 CPU-migrations # 0.000 M/sec 6317 page-faults # 0.025 M/sec 471734719 cycles # 1830.767 M/sec 393365060 instructions # 0.834 IPC 7849464 cache-references # 30.463 M/sec 1038976 cache-misses # 4.032 M/sec65.766937233 seconds time elapsedYup, a full minute to get a binary loaded and fired up. Seems it and others get evicted by the VM. Getting whatever back into ram is painful to listen to (poor disk/me.. seek latency eating MUCH time).
Calling script back to back with args 1 (realtime foreground) 3 (idle background) does best:
Performance counter stats for 'sh -c konsole -e exit':
169.915036 task-clock-msecs # 0.077 CPUs 439 context-switches # 0.003 M/sec 9 CPU-migrations # 0.000 M/sec 6314 page-faults # 0.037 M/sec 363128252 cycles # 2137.117 M/sec 367410945 instructions # 1.012 IPC 5822302 cache-references # 34.266 M/sec 437189 cache-misses # 2.573 M/sec2.213079979 seconds time elapsedArgs 2 (best effort foreground) and 3 (idle background);
Performance counter stats for 'sh -c konsole -e exit':
176.689856 task-clock-msecs # 0.015 CPUs 357 context-switches # 0.002 M/sec 15 CPU-migrations # 0.000 M/sec 6314 page-faults # 0.036 M/sec 374861947 cycles # 2121.582 M/sec 368929476 instructions # 0.984 IPC 5806286 cache-references # 32.861 M/sec 496651 cache-misses # 2.811 M/sec
Quoted 4 lines
12.036165968 seconds time elapsed -Mike
Wow, at first I did not notice anything from your *.sh script, but then a few moments later, the system became vary sluggish. firefox takes about 15 seconds to load. OS is mint linux with the latest HEAD on an imac9,1.
Justin P. Mattock
Quoted 5 lines · Justin P. Mattock
Wow, at first I did not notice anything from your *.sh script, but then a few moments later, the system became vary sluggish. firefox takes about 15 seconds to load. OS is mint linux with the latest HEAD on an imac9,1.
CFQ inherits IO priority from the parent on fork(), so nicing a shell or the dd should (and does) help a LOT. It also inherits the CPU scheduler class. I just checked, and IO does indeed inherit SCHED_IDLE as well as the RT classes (SCHED_BATCH does nothing). Setting the dd to SCHED_IDLE makes a big difference, as it should, so there's a bigger hammer than nice + ionice. SCHED_IDLE is CPU_adorable + IO_nicest.
Anyway, the problem seems to have to have at least two facets:
1. streaming IO causing pagecache reclaim. That problem doesn't seem to want to go away. Fixing that won't help 2 one bit though.
2. reads are sync, more heavily affected by seek latency than writes. If the disk your binaries live on is seeking a lot, you pay a LOT.
Hohum, IO + VM = haaaaard ;-)
-Mike
Hi and thanks for your reply!
Quoted 1 line · Mike Galbraith
nicing a shell or the dd should (and does) help a LOT.
If this is the only way to influence this, maybe the default settings for the niceness of interactive and non-interactive tasks are not the best choice. (Maybe a distribution problem in this case)
Quoted 1 line
reads are sync, more heavily affected by seek latency than writes.
But how does this explain the seconds-long delays?
If an interactive process causes a lot of seeks because of reads/writes which "are sync", I see how this can greatly slow down otherwise pipelined write operations, but the other way around?
Quoted 1 line · Ulrich Lukas
Hi and thanks for your reply!
Hi.
Quoted 6 lines
Quoted 1 line · Mike Galbraith
nicing a shell or the dd should (and does) help a LOT.
If this is the only way to influence this, maybe the default settings for the niceness of interactive and non-interactive tasks are not the best choice. (Maybe a distribution problem in this case)
There is no knowledge in the CPU nor IO scheduler wrt interactive vs non-interactive. I've tinkered many times with a SCHED_INTERACTIVE class, but it's not at all an easy problem, so keeps landing on the trash heap. I could ramble on _for ever_ about that subject, but it's thankfully irrelevant to this thread ;-)
Quoted 3 lines
Quoted 1 line
reads are sync, more heavily affected by seek latency than writes.
But how does this explain the seconds-long delays?
Seek latencies are cumulative is my (wild arsed) theory.
Quoted 3 lines
If an interactive process causes a lot of seeks because of reads/writes which "are sync", I see how this can greatly slow down otherwise pipelined write operations, but the other way around?
If you seek between tiny reads, the more little reads you do, the more seeks hurt. Readahead is supposed to help, but.. the pain is there.
-Mike
FWIW, I did not notice any problems with the last kernel I had compiled in the -- IIRC -- 29-rc timeframe. Or maybe it was the 30-rc timeframe.
I had that kernel up for soemthing like five weeks, on a busy, low-ram laptop, w/o and long pauses or other noticeable pain. For at least two years before that the best this laptop could manage before latency became horrific was a fortnight, and weekly reboots were desirable.
Unfortunately, before I could send out a congratulatory note (I wanted to first confirm that the result remained) I pulled up and started see- ing even worse latency then ever.
Turning ext4 barriers off helped avoid some of the new latency, and my current compile (master as of the 17th) is much better than .31 was.
But long pauses still occur whenever paging and backgound i/o-bound tasks interact.
There was a major regression in paging performance either .30 or .31 (see above). Interestingly, top(1) also shows less swap usage for the same usage patterns than with the older kernels. I don't know which is cause or which is effect.
A git pull on a clone of Linus' tree is a great way to trigger the problem. As is using Gentoo's portage to install/upgrade anything. (Bad python; no alligator!¹) (Yes, that is a burlesque of the no doughnut joke.)
[1] Pythons which have invaded Florida have reportedly been witnessed eating alligators, according to a recent wire service article.
-JimC
Signature · 1 line
James Cloos <[email protected]> OpenPGP: 1024D/ED7DAEA6
Quoted 6 lines · James Cloos
[...]
I had that kernel up for soemthing like five weeks, on a busy, low-ram laptop, w/o and long pauses or other noticeable pain. For at least two years before that the best this laptop could manage before latency became horrific was a fortnight, and weekly reboots were desirable.
Quoted 7 lines
Unfortunately, before I could send out a congratulatory note (I wanted to first confirm that the result remained) I pulled up and started see- ing even worse latency then ever.
[...]
A git pull on a clone of Linus' tree is a great way to trigger the problem. As is using Gentoo's portage to install/upgrade anything. (Bad python; no alligator!¹)
Fortunately, this is completely (at least here) solved by putting those in make.conf:
PORTAGE_NICENESS=19
PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Nikos" == Nikos Chantziaras <[email protected]> writes:
Quoted 2 lines · James Cloos
Quoted 2 lines
is a great way to trigger the problem. As is using Gentoo's portage to install/upgrade anything.
Nikos> Fortunately, this is completely (at least here) solved by putting Nikos> those in make.conf:
Nikos> PORTAGE_NICENESS=19
Nikos> PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"
I had NICENESS set, but had never noticed that PORTAGE_IONICE_COMMAND was available. I'll give that a test. Thanks.
-JimC
Signature · 1 line
James Cloos <[email protected]> OpenPGP: 1024D/ED7DAEA6
Quoted 15 lines · James Cloos
FWIW, I did not notice any problems with the last kernel I had compiled in the -- IIRC -- 29-rc timeframe. Or maybe it was the 30-rc timeframe.
I had that kernel up for soemthing like five weeks, on a busy, low-ram laptop, w/o and long pauses or other noticeable pain. For at least two years before that the best this laptop could manage before latency became horrific was a fortnight, and weekly reboots were desirable.
Unfortunately, before I could send out a congratulatory note (I wanted to first confirm that the result remained) I pulled up and started see- ing even worse latency then ever.
Turning ext4 barriers off helped avoid some of the new latency, and my current compile (master as of the 17th) is much better than .31 was.
Quoted 7 lines
But long pauses still occur whenever paging and backgound i/o-bound tasks interact.
There was a major regression in paging performance either .30 or .31 (see above). Interestingly, top(1) also shows less swap usage for the same usage patterns than with the older kernels. I don't know which is cause or which is effect.
If someone has a definite good/bad cutoff for the problem they're seeing, a git bisect (yeah, takes time) would be a good thing to try.
The "interactivity while saturating root disk" thing isn't a new problem though, I can recall that being a problem to varying degrees forever. (forever being defined as feb '93 -> today;)
Quoted 4 lines
A git pull on a clone of Linus' tree is a great way to trigger the problem. As is using Gentoo's portage to install/upgrade anything. (Bad python; no alligator!¹) (Yes, that is a burlesque of the no doughnut joke.)
Git doesn't cause me any woes whatsoever, but my box's primary drive isn't a slow laptop drive, ram isn't tight, and it has 4 cores.
Dunno.
If you're using the CFQ IO scheduler, IO bandwidth follows CPU nice level, so tweaking nice tweaks both CPU and IO in one go. ionice can also be used to tweak IO independently.
I don't know if distros provide a tool that supports SCHED_IDLE, if so, you can use that to set both CPU and IO usage to IDLE class in one go.
If you don't have such a tool, you can try the attached for background loads. Someone posted it to LKML in '98, and I've been updating/using it for experiments ever since. Just do schedctl -I background-job, and the job will be as polite as CPU and IO schedulers can make it.
-Mike
#include <unistd.h>
#include <sched.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#ifndef SCHED_BATCH
#define SCHED_BATCH 3
#endif
#ifndef SCHED_ISO
#define SCHED_ISO 4
#endif
#ifndef SCHED_IDLE
#define SCHED_IDLE 5
#endif
#ifndef SCHED_SYNC
#define SCHED_SYNC 6
#endif
#ifndef SCHED_RESET_ON_FORK
#define SCHED_RESET_ON_FORK 0x40000000
#endif
void usage();
void version();
void info(int pid);
void setsched(int pid, int policy, int pri, char flags);
unsigned char *ourname;
static int reset_flag;
/* flags */
#define GOTPID 1
#define GOTPOLICY 2
#define GOTPRI 4
#define GOTCMD 8
#define GOTINFO 16
int main(argc, argv)
unsigned int argc;
unsigned char *argv[];
{
int pid;
int policy;
int pri = 0;
int i1;
int i2;
unsigned long l1;
char *ptr;
char flags = 0;
ourname = argv[0];
if (argc <= 1) {
usage();
exit(1);
}
for (i1 = 1; i1 < argc; i1++)
if (argv[i1][0] == '-') { /* switch */
if (argv[i1][1] >= '0' && argv[i1][1] <= '9') {
l1 = strtol(&argv[i1][1], &ptr, 10);
if (*ptr || l1 < 0 || l1 > 0x7FFF) {
fprintf(stderr,
"invalid priority: %s\n",
argv[i1]);
exit(1);
}
pri = l1;
flags |= GOTPRI;
continue;
}
if (argv[i1][1] == '-') { /* long opt. */
if (!strcasecmp("help", &argv[i1][2])) {
usage();
exit(0);
}
if (!strcasecmp("version", &argv[i1][2])) {
version();
exit(0);
}
if (!strcasecmp("info", &argv[i1][2])) {
INFO:
if (i1 < argc - 1) {
i1++;
l1 = strtol(argv[i1], &ptr, 0);
if (*ptr) {
i1--;
} else {
if (l1 < 0
|| l1 > 0x7FFF) {
fprintf(stderr,
"invalid PID: %s\n",
argv
[i1]);
exit(1);
}
if (!l1)
pid = getpid();
else {
pid = l1;
}
flags |= GOTPID;
}
}
if (flags & GOTPID) {
info(pid);
exit(0);
} else {
flags |= GOTINFO;
// fprintf(stderr,"no PID given\n");
// exit(1);
}
continue;
} /* end of --info */
if (!strcasecmp("pid", &argv[i1][2])) {
PID:
i1++;
if (i1 >= argc) {
fprintf(stderr,
"option requires an argument\n");
exit(1);
} else {
l1 = strtol(argv[i1], &ptr, 0);
if (*ptr || l1 < 0
|| l1 > 0x7FFF) {
fprintf(stderr,
"invalid PID: %s\n",
argv[i1]);
exit(1);
}
if (!l1)
pid = getpid();
else
pid = l1;
flags |= GOTPID;
}
continue;
} /* end of --pid */
if (!strcasecmp("priority", &argv[i1][2])
|| !strcasecmp("pri", &argv[i1][2])) {
PRIORITY:
i1++;
if (i1 >= argc) {
fprintf(stderr,
"option requires an argument\n");
exit(1);
} else {
l1 = strtol(argv[i1], &ptr, 0);
if (*ptr) {
fprintf(stderr,
"invalid priority: %s\n",
argv[i1]);
exit(1);
}
pri = l1;
flags |= GOTPRI;
}
continue;
} /* end of --priority */
if (!strcasecmp("other", &argv[i1][2])
|| !strcasecmp("normal", &argv[i1][2])) {
OTHER:
if (i1 < argc - 1) {
i1++;
l1 = strtol(argv[i1], &ptr, 0);
if (*ptr) {
i1--;
} else {
if (l1 < 0
|| l1 > 0x7FFF) {
fprintf(stderr,
"invalid PID: %s\n",
argv
[i1]);
exit(1);
}
if (!l1)
pid = getpid();
else {
pid = l1;
flags |= GOTPID;
}
}
}
flags |= GOTPOLICY;
policy = SCHED_OTHER;
continue;
} /* end of --other */
if (!strcasecmp("batch", &argv[i1][2])) {
BATCH:
if (i1 < argc - 1) {
i1++;
l1 = strtol(argv[i1], &ptr, 0);
if (*ptr) {
i1--;
} else {
if (l1 < 0
|| l1 > 0x7FFF) {
fprintf(stderr,
"invalid PID: %s\n",
argv
[i1]);
exit(1);
}
if (!l1)
pid = getpid();
else {
pid = l1;
flags |= GOTPID;
}
}
}
flags |= GOTPOLICY;
policy = SCHED_BATCH;
continue;
} /* end of --batch */
if (!strcasecmp("idle", &argv[i1][2])) {
IDLE:
if (i1 < argc - 1) {
i1++;
l1 = strtol(argv[i1], &ptr, 0);
if (*ptr) {
i1--;
} else {
if (l1 < 0
|| l1 > 0x7FFF) {
fprintf(stderr,
"invalid PID: %s\n",
argv
[i1]);
exit(1);
}
if (!l1)
pid = getpid();
else {
pid = l1;
flags |= GOTPID;
}
}
}
flags |= GOTPOLICY;
policy = SCHED_IDLE;
continue;
} /* end of --idle */
if (!strcasecmp("sync", &argv[i1][2])) {
SYNC:
if (i1 < argc - 1) {
i1++;
l1 = strtol(argv[i1], &ptr, 0);
if (*ptr) {
i1--;
} else {
if (l1 < 0
|| l1 > 0x7FFF) {
fprintf(stderr,
"invalid PID: %s\n",
argv
[i1]);
exit(1);
}
if (!l1)
pid = getpid();
else {
pid = l1;
flags |= GOTPID;
}
}
}
flags |= GOTPOLICY;
policy = SCHED_SYNC;
continue;
} /* end of --sync */
if (!strcasecmp("reset", &argv[i1][2])) {
RESET:
if (i1 < argc - 1) {
i1++;
l1 = strtol(argv[i1], &ptr, 0);
if (*ptr) {
i1--;
} else {
if (l1 < 0
|| l1 > 0x7FFF) {
fprintf(stderr,
"invalid PID: %s\n",
argv
[i1]);
exit(1);
}
if (!l1)
pid = getpid();
else {
pid = l1;
flags |= GOTPID;
}
}
}
reset_flag = SCHED_RESET_ON_FORK;
continue;
} /* end of --reset */
if (!strcasecmp("fifo", &argv[i1][2])) {
FIFO:
if (i1 < argc - 1) {
i1++;
l1 = strtol(argv[i1], &ptr, 0);
if (*ptr) {
i1--;
} else {
if (l1 < 0
|| l1 > 0x7FFF) {
fprintf(stderr,
"invalid PID: %s\n",
argv
[i1]);
exit(1);
}
if (!l1)
pid = getpid();
else {
pid = l1;
flags |= GOTPID;
}
}
}
flags |= GOTPOLICY;
policy = SCHED_FIFO;
continue;
} /* end of --fifo */
if (!strcasecmp("rr", &argv[i1][2])
|| !strcasecmp("round-robin",
&argv[i1][2])) {
RR:
if (i1 < argc - 1) {
i1++;
l1 = strtol(argv[i1], &ptr, 0);
if (*ptr) {
i1--;
} else {
if (l1 < 0
|| l1 > 0x7FFF) {
fprintf(stderr,
"invalid PID: %s\n",
argv
[i1]);
exit(1);
}
if (!l1)
pid = getpid();
else {
pid = l1;
flags |= GOTPID;
}
}
}
flags |= GOTPOLICY;
policy = SCHED_RR;
continue;
}
/* end of --rr */
fprintf(stderr,
"warning: unknown long switch: %s\n",
&argv[i1][2]);
continue;
}
for (i2 = 1; i2 < strlen(argv[i1]); i2++)
switch (argv[i1][i2]) {
case 'h':
case 'H':
case '?':
usage();
exit(0);
case 'i':
goto INFO;
case 'p':
goto PID;
case 'P':
goto PRIORITY;
case 'r':
goto RESET;
case 'N':
case 'O':
goto OTHER;
case 'B':
goto BATCH;
case 'I':
goto IDLE;
case 'S':
goto SYNC;
case 'F':
goto FIFO;
case 'R':
goto RR;
default:
fprintf(stderr,
"warning: unknown switch: %c\n",
argv[i1][i2]);
break;
}
} else { /* command */// printf("command: %s\n",argv[i1]);
if (flags & GOTPID) {
fprintf(stderr,
"warning: both PID and command given\n");
} else
pid = getpid();
if (!(flags & (GOTPOLICY | GOTPRI))) {
fprintf(stderr,
"warning: neither policy nor priority given\n");
}
flags |= GOTCMD;
setsched(pid, policy, pri, flags);
execvp(argv[i1], (char **)(&argv[i1]));
if (errno) {
printf("%s: error: %s\n", argv[i1],
strerror(errno));
exit(1);
}
break;
}
if (reset_flag && !(flags & GOTPOLICY)) {
policy = sched_getscheduler(pid);
flags |= GOTPOLICY;
}
if (flags & GOTINFO) {
if (!(flags & GOTPID)) {
fprintf(stderr, "no PID given\n");
exit(1);
}
info(pid);
}
if (!(flags & (GOTCMD | GOTINFO))) {
if (!(flags & GOTPID)) {
fprintf(stderr, "neither PID nor command given\n");
exit(1);
}
if (!(flags & (GOTPOLICY | GOTPRI))) {
fprintf(stderr,
"warning: neither policy nor priority given\n");
}
setsched(pid, policy, pri, flags);
}
exit(0);
}void usage()
{
printf("Usage: %s [options] [command [params...]]\n\n", ourname);
printf(" --pid, -p <n> specify process id\n");
printf(" --fifo, -F [pid] set scheduling policy to FIFO\n");
printf(" --rr, -R [pid] set scheduling policy to RR\n");
printf(" --batch, -B [pid] set scheduling policy to BATCH\n");
printf(" --idle, -I [pid] set scheduling policy to IDLE\n");
printf(" --sync, -S [pid] set scheduling policy to SYNC\n");
printf(" --other, --normal,\n");
printf(" -N, -O [pid] specify normal scheduling\n");
printf(" --reset, -r set SCHED_RESET_ON_FORK flag\n");
printf(" --priority, --pri,\n");
printf(" -<n> set priority\n");
printf(" --info, -i [pid] show scheduling data\n");
printf(" --help, -h, -H, -? display this text\n");
printf(" --version show version data\n\n");
}void version()
{
printf("schedctl, version 0.95 (beta), 8-May-1998\n");
printf("report bugs to [email protected] or [email protected]\n");
}void info(int pid)
{
struct sched_param sp;
int i1;
i1 = sched_getscheduler(pid);
if (errno == ESRCH) {
fprintf(stderr, "process not found\n");
return;
}
if (i1 & SCHED_RESET_ON_FORK) {
i1 &= ~SCHED_RESET_ON_FORK;
reset_flag = SCHED_RESET_ON_FORK;
}
printf("scheduling policy of process %d is ", pid);
switch (i1) {
case SCHED_FIFO:
printf("FIFO");
break;
case SCHED_RR:
printf("RR");
break;
case SCHED_OTHER:
printf("OTHER (normal)");
break;
case SCHED_BATCH:
printf("BATCH");
break;
case SCHED_ISO:
printf("ISO");
break;
case SCHED_IDLE:
printf("IDLE");
break;
case SCHED_SYNC:
printf("SYNC");
break;
default:
printf("**UNKNOWN**");
}
if (reset_flag)
printf(" SCHED_RESET_ON_FORK");
sched_getparam(pid, &sp);
printf(", priority is %d\n", sp.sched_priority);
}void setsched(int pid, int policy, int pri, char flags)
{
struct sched_param sp;
int i1;
i1 = sched_getscheduler(pid);
if (errno == ESRCH) {
fprintf(stderr, "process not found\n");
return;
}
if (!(flags & GOTPOLICY))
policy = i1;
sched_getparam(pid, &sp);
if (!(flags & GOTPRI))
pri = sp.sched_priority;
else if (pri < sched_get_priority_min(policy)
|| pri > sched_get_priority_max(policy)) {
fprintf(stderr, "priority (%d) out of range\n", pri);
return;
}
if (policy == SCHED_FIFO || policy == SCHED_RR) {
if (!pri) {
if (!sp.sched_priority)
sp.sched_priority++;
} else
sp.sched_priority = pri;
} else if (pri && (policy == SCHED_OTHER || policy == SCHED_BATCH ||
policy == SCHED_ISO || policy == SCHED_IDLE || policy == SCHED_SYNC))
sp.sched_priority = 0;
else
sp.sched_priority = pri;
if (sched_setscheduler(pid, policy|reset_flag, &sp)) {
fprintf(stderr, "error #%d: %s\n", errno, strerror(errno));
return;
}
i1 = sched_getscheduler(pid);
if (i1 & SCHED_RESET_ON_FORK)
i1 &= ~SCHED_RESET_ON_FORK;
sched_getparam(pid, &sp);
if (flags & GOTCMD)
return;
printf("scheduling policy of process %d set to ", pid);
switch (i1) {
case SCHED_FIFO:
printf("FIFO");
break;
case SCHED_RR:
printf("RR");
break;
case SCHED_OTHER:
printf("OTHER");
break;
case SCHED_BATCH:
printf("BATCH");
break;
case SCHED_ISO:
printf("ISO");
break;
case SCHED_IDLE:
printf("IDLE");
break;
case SCHED_SYNC:
printf("SYNC");
break;
}
if (reset_flag)
printf(" SCHED_RESET_ON_FORK");
printf("\n");
}Quoted 2 lines · Mike Galbraith
I don't know if distros provide a tool that supports
SCHED_IDLE, if so, you can use that to set both CPU and IO usage to IDLE class in one go.
P.S. I just remembered, don't use SCHED_IDLE with kernels < 2.6.28. There are two patches wrt this scheduling class which did not go to the stable trees less than 2.6.28.
-Mike
Quoted 7 lines · Mike Galbraith
Quoted 2 lines · Mike Galbraith
I don't know if distros provide a tool that supports
SCHED_IDLE, if so, you can use that to set both CPU and IO usage to IDLE class in one go.
P.S. I just remembered, don't useSCHED_IDLEwith kernels < 2.6.28. There are two patches wrt this scheduling class which did not go to the stable trees less than 2.6.28.
Since 2.6.27 is long-term, maybe they should be sent to 2.6.27? It is weird to have a long-term stable kernel for which the recommendation is "don't use SCHED_IDLE".
Signature · 4 lines
"One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh
Quoted 5 lines · Henrique de Moraes Holschuh
Quoted 4 lines · Mike Galbraith
Quoted 2 lines · Mike Galbraith
I don't know if distros provide a tool that supports
SCHED_IDLE, if so, you can use that to set both CPU and IO usage to IDLE class in one go.
Quoted 7 lines
Quoted 3 lines
P.S. I just remembered, don't useSCHED_IDLEwith kernels < 2.6.28. There are two patches wrt this scheduling class which did not go to the stable trees less than 2.6.28.Since 2.6.27 is long-term, maybe they should be sent to 2.6.27? It is weird to have a long-term stable kernel for which the recommendation is "don't use
SCHED_IDLE".
Yeah, that's what I thought too, but apparently maintainers didn't think SCHED_IDLE was common/important enough.
-Mike
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Mike" == Mike Galbraith <[email protected]> writes:
Mike> If someone has a definite good/bad cutoff for the problem they're Mike> seeing, a git bisect (yeah, takes time) would be a good thing to try.
Yes, I wanted to try that for this, but given how long it takes to compile a kernel on this box, and how long it takes to get everything up so that I can test my real-life load, added to the difference in performance after each day of uptime, I could do at best one kernel per day; it would take weeks to finish.
That said, I jsut took a look at the git log for my /boot/grub repo and can confirm that the last really good kernel -- the one I had up for more than a month for the first time in years -- was pulled between 2.6.30-rc6 and 2.6.30-rc7 and installed on 2009/05/17 which, since I keep .config in git in my compile repo, lets me narrow down the make oldconfig to 2009/05/17 18:37:24Z.
So, whatever Linus' tree looked like on that date is GOOD. His last commit before that was 0f6f49a8cd0163fdb1723ed29f01fc65177108dc.
My next make oldconfig was on 2009/06/25 04:32:03Z, when I started testing the radeon kms. (I'm back to non-KMS now.) There were too many issues directly caused by KMS itself for me to comment on non- KMS kernel issues from the kernels I ran with KMS enabled.
My return from KMS as on 2009/08/31; Linus' last commit before I did that was adda766193ea1cf3137484a9521972d080d0b7af.
To give an idea of how little that narrows things:
:; git log 0f6f49a8cd..adda766193|egrep -c ^commit 12094
For those who use the tars, that means 30-rc6 was fast and 31-rc8 slow.
Mike> Git doesn't cause me any woes whatsoever, but my box's primary Mike> drive isn't a slow laptop drive, ram isn't tight,
That does make a difference. Git is *great* when it fits into ram, but a pull or merge on a clone of Linus' tree takes a quarter gig of VM on x86-32, which guarantees paging, including to the swap partitions. (The laptop is old enough to support three platters — including the optical — so I have my primary swap partition on the second drive to reduce some of the load to the / /var /boot drive. The first disk's swap partition only gets hit when I boot w/o the second drive. It helps.)
The problem isn't that paging occurs, but rather that now, when paging occurs, said paging is much slower than it had been.
IOW, the priority of paging vs other disk I/O has changed for the worse, at some point after 0f6f49a8cd and before adda766193. But, as I wrote above, to actually test kernels between those to determine where would take weeks.
-JimC
Signature · 1 line
James Cloos <[email protected]> OpenPGP: 1024D/ED7DAEA6
Quoted 4 lines · Arjan van de Ven
Quoted 4 lines
can you try the following two things?
echo 4096 >
/sys/block/sda/queue/nr_requestsfor i inpidof kjournald; do ionice -c1 -p $i ; done
No, just tested again. Only a marginal improvement if there is any.
Quoted 11 lines · Ulrich Lukas
Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bitHow to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile (or dd if=/home/john-doe/big-testfile of=/dev/null)
Quoted 7 lines
- a real use scenario would be a daily disk-backup or the simple extraction of a tarball containing slightly bigger files
Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.
I guess that switching from CFQ to deadline I/O scheduler improves the situation, right?
Signature · 2 lines
Jiri Kosina SUSE Labs, Novell Inc.
Quoted 17 lines · Jiri Kosina
Quoted 15 lines · Ulrich Lukas
Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bitHow to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile (or dd if=/home/john-doe/big-testfile of=/dev/null)- a real use scenario would be a daily disk-backup or the simple extraction of a tarball containing slightly bigger files
Quoted 6 lines
Quoted 3 lines
Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.I guess that switching from CFQ to deadline I/O scheduler improves the situation, right?
For example on my desktop, switching to deadline slightly improves the
situation but the machine is still mostly unusable while dd is running... I'll try to debug it some more to see whether it can be somehow helped.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 22 lines · Jan Kara
Quoted 21 lines · Jiri Kosina
Quoted 18 lines · Ulrich Lukas
Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bitHow to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
dd if=/dev/zero of=/home/john-doe/testfile (or dd if=/home/john-doe/big-testfile of=/dev/null)- a real use scenario would be a daily disk-backup or the simple extraction of a tarball containing slightly bigger files
Observation:
- The system becomes _really_ slow as described above; unusable for any multimedia tasks.
Quoted 5 lines
Quoted 2 lines
I guess that switching from CFQ to deadline I/O scheduler improves the situation, right?
For example on my desktop, switching to deadline slightly improves the
situation but the machine is still mostly unusable while dd is running... I'll try to debug it some more to see whether it can be somehow helped.
Datapoint:
echo 1 > /sys/block/$disk/queue/iosched/quantum seems to make a huge difference. Stock is 4. (wisdom of setting it to 1? no idea)
This...
marge:/root/tmp # ./testo.sh
quantum is 4, setting to 1
dd if=3DMark2000.mkv of=/tmp/3DMark2000.mkv 204529+1 records in
204529+1 records out
104718895 bytes (105 MB) copied, 1.62672 s, 64.4 MB/s perf sched record -o /tmp/mplayer-c2-c2.data (tmpfs)& mplayer /tmp/3DMark2000.mkv&
dd if=/dev/zero of=crud-25859&
sleep 180
timing konsole -e exit
Performance counter stats for 'sh -c konsole -e exit':
173.615837 task-clock-msecs # 0.031 CPUs
431 context-switches # 0.002 M/sec
10 CPU-migrations # 0.000 M/sec
6314 page-faults # 0.036 M/sec
368422027 cycles # 2122.053 M/sec
376283291 instructions # 1.021 IPC
5883195 cache-references # 33.886 M/sec
444333 cache-misses # 2.559 M/sec5.597628358 seconds time elapsed
...is pretty typical, and a LOT better than the 60-70 seconds I was seeing, or the minutes I sometimes saw when playing with evolution.
I still occasionally see some largish numbers though, and suspect that depends on how much cache was evicted.
-Mike
Quoted 2 lines · On Mon, 21 Sep 2009 09:25:08 +0200 Mike Galbraith <[email protected]> wrote:
I still occasionally see some largish numbers though, and suspect that depends on how much cache was evicted.
if you run latencytop it'll at least tell you what kind of operating was going on that hit the delay...
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 7 lines · Arjan van de Ven
Quoted 2 lines · On Mon, 21 Sep 2009 09:25:08 +0200 Mike Galbraith <[email protected]> wrote:
I still occasionally see some largish numbers though, and suspect that depends on how much cache was evicted.
if you run latencytop it'll at least tell you what kind of operating was going on that hit the delay...
One of the really bad ones for evolution is below. It was so bad that I had _plenty_ of time to get to a GUI free shell, and poke SysRq-W before killing the darn dd.
[ 5112.427012] SysRq : Show Blocked State [ 5112.430995] task PC stack pid father [ 5112.430995] kjournald D 0000000000000002 0 1230 2 0x00000000 [ 5112.430995] ffff8800378d7c50 0000000000000046 0000000000000000 ffff8800a5ed53b0 [ 5112.430995] ffff8800bd2c0000 ffff8800378d7fd8 000000000000dd38 0000000000013480 [ 5112.430995] 0000000000004000 0000000000013480 ffff8800378d7bf0 00001000810e0c0a [ 5112.430995] Call Trace: [ 5112.430995] [<ffffffff812cacb4>] io_schedule+0x80/0xc6 [ 5112.430995] [<ffffffff8110df99>] sync_buffer+0x40/0x44 [ 5112.430995] [<ffffffff812cb373>] __wait_on_bit+0x4c/0x7e [ 5112.430995] [<ffffffff8110df59>] ? sync_buffer+0x0/0x44 [ 5112.430995] [<ffffffff8110df59>] ? sync_buffer+0x0/0x44 [ 5112.430995] [<ffffffff812cb414>] out_of_line_wait_on_bit+0x6f/0x7c [ 5112.430995] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f [ 5112.430995] [<ffffffff8110deff>] __wait_on_buffer+0x24/0x26 [ 5112.430995] [<ffffffffa0077e91>] journal_commit_transaction+0xca4/0x138b [jbd] [ 5112.430995] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d [ 5112.430995] [<ffffffffa007ba02>] kjournald+0x168/0x397 [jbd] [ 5112.430995] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d [ 5112.430995] [<ffffffffa007b89a>] ? kjournald+0x0/0x397 [jbd] [ 5112.430995] [<ffffffff81065acb>] kthread+0x82/0x8a [ 5112.625071] [<ffffffff8100ccca>] child_rip+0xa/0x20 [ 5112.625071] [<ffffffff8100c669>] ? restore_args+0x0/0x30 [ 5112.625071] [<ffffffff81065a49>] ? kthread+0x0/0x8a [ 5112.625071] [<ffffffff8100ccc0>] ? child_rip+0x0/0x20 [ 5112.625071] flush-8:16 D ffff8800bd2097f0 0 1236 2 0x00000000 [ 5112.625071] ffff8800bd209790 0000000000000046 0000000000000000 ffffffff810beb58 [ 5112.625071] ffff8800bd209740 ffff8800bd209fd8 000000000000dd38 0000000000013480 [ 5112.625071] 0000000000004000 0000000000013480 0000000001080001 0000000000000080 [ 5112.625071] Call Trace: [ 5112.625071] [<ffffffff810beb58>] ? get_page_from_freelist+0x33d/0x4a6 [ 5112.625071] [<ffffffff81174cb6>] ? cfq_may_queue+0x60/0xe1 [ 5112.625071] [<ffffffff812cacb4>] io_schedule+0x80/0xc6 [ 5112.625071] [<ffffffff81169184>] get_request_wait+0xf3/0x18f [ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d [ 5112.625071] [<ffffffff8115f142>] ? elv_merge+0x37/0x19d [ 5112.625071] [<ffffffff8116954d>] __make_request+0x32d/0x45b [ 5112.625071] [<ffffffff811678ae>] generic_make_request+0x339/0x38b [ 5112.625071] [<ffffffff811679d4>] submit_bio+0xd4/0xdd [ 5112.625071] [<ffffffff8110ccfc>] submit_bh+0x106/0x129 [ 5112.625071] [<ffffffff8110f7a4>] __block_write_full_page+0x1eb/0x2d1 [ 5112.625071] [<ffffffff8110e197>] ? end_buffer_async_write+0x0/0x13f [ 5112.625071] [<ffffffff8110f94f>] block_write_full_page_endio+0xc5/0xca [ 5112.625071] [<ffffffff8110f969>] block_write_full_page+0x15/0x17 [ 5112.625071] [<ffffffffa0095e1f>] ext3_writeback_writepage+0x71/0x142 [ext3] [ 5112.625071] [<ffffffff810bf647>] __writepage+0x17/0x34 [ 5112.625071] [<ffffffff810bfd9f>] write_cache_pages+0x250/0x393 [ 5112.625071] [<ffffffff810bf630>] ? __writepage+0x0/0x34 [ 5112.625071] [<ffffffff810bff06>] generic_writepages+0x24/0x2a [ 5112.625071] [<ffffffff810bff34>] do_writepages+0x28/0x2a [ 5112.625071] [<ffffffff81106b94>] writeback_single_inode+0x181/0x395 [ 5112.625071] [<ffffffff811077fc>] writeback_inodes_wb+0x341/0x42f [ 5112.625071] [<ffffffff81107a2b>] wb_writeback+0x141/0x194 [ 5112.625071] [<ffffffff81095031>] ? call_rcu_sched+0x15/0x17 [ 5112.625071] [<ffffffff81095041>] ? call_rcu+0xe/0x10 [ 5112.625071] [<ffffffff81107bb9>] wb_do_writeback+0x7a/0x16b [ 5112.625071] [<ffffffff81107ce9>] bdi_writeback_task+0x3f/0xad [ 5112.625071] [<ffffffff810cca5d>] ? bdi_start_fn+0x0/0xd7 [ 5112.625071] [<ffffffff810ccad3>] bdi_start_fn+0x76/0xd7 [ 5112.625071] [<ffffffff810cca5d>] ? bdi_start_fn+0x0/0xd7 [ 5112.625071] [<ffffffff81065acb>] kthread+0x82/0x8a [ 5112.625071] [<ffffffff8100ccca>] child_rip+0xa/0x20 [ 5112.625071] [<ffffffff8100c669>] ? restore_args+0x0/0x30 [ 5112.625071] [<ffffffff81065a49>] ? kthread+0x0/0x8a [ 5112.625071] [<ffffffff8100ccc0>] ? child_rip+0x0/0x20 [ 5112.625071] console-kit-d D ffff8800b788c480 0 2963 1 0x00000000 [ 5112.625071] ffff8800bc59f6b8 0000000000000082 0000000000000000 ffffffffa0076655 [ 5112.625071] ffff8800bc59f628 ffff8800bc59ffd8 000000000000dd38 0000000000013480 [ 5112.625071] 0000000000004000 0000000000013480 ffff8800bd2f3c00 0000000000000000 [ 5112.625071] Call Trace: [ 5112.625071] [<ffffffffa0076655>] ? do_get_write_access+0x443/0x48b [jbd] [ 5112.625071] [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd] [ 5112.625071] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f [ 5112.625071] [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd] [ 5112.625071] [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3] [ 5112.625071] [<ffffffffa0092901>] ext3_new_blocks+0x333/0x669 [ext3] [ 5112.625071] [<ffffffffa0096654>] ext3_get_blocks_handle+0x3ec/0x92d [ext3] [ 5112.625071] [<ffffffff8110c866>] ? alloc_buffer_head+0x1e/0x51 [ 5112.625071] [<ffffffffa0096c3a>] ext3_get_block+0xa5/0xe3 [ext3] [ 5112.625071] [<ffffffff8110ec0a>] __block_prepare_write+0x1f0/0x3d2 [ 5112.625071] [<ffffffffa0076e49>] ? start_this_handle+0x37d/0x474 [jbd] [ 5112.625071] [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3] [ 5112.625071] [<ffffffffa007709a>] ? journal_start+0x60/0xca [jbd] [ 5112.625071] [<ffffffff8110ef7c>] block_write_begin+0x85/0xd5 [ 5112.625071] [<ffffffffa0098130>] ext3_write_begin+0xf0/0x1e7 [ext3] [ 5112.625071] [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3] [ 5112.625071] [<ffffffff810ba9e8>] generic_file_buffered_write+0x11f/0x277 [ 5112.625071] [<ffffffff810bafe1>] __generic_file_aio_write+0x351/0x385 [ 5112.625071] [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad [ 5112.625071] [<ffffffff810e75b4>] do_sync_write+0xec/0x132 [ 5112.625071] [<ffffffff810c176a>] ? lru_cache_add_lru+0x2b/0x2d [ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d [ 5112.625071] [<ffffffff8117c94f>] ? __up_read+0x9e/0xa7 [ 5112.625071] [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18 [ 5112.625071] [<ffffffff810e8053>] vfs_write+0xb3/0x16f [ 5112.625071] [<ffffffff810e81dd>] sys_write+0x4c/0x74 [ 5112.625071] [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b [ 5112.625071] dd D ffff8800b788c480 0 8587 6686 0x00000000 [ 5112.625071] ffff8800a5e2faa8 0000000000000086 0000000000000000 ffffffff812cab8c [ 5112.625071] ffff8800a5e2fbe0 ffff8800a5e2ffd8 000000000000dd38 0000000000013480 [ 5112.625071] 0000000000004000 0000000000013480 ffff8800a5e2fa28 ffffffff810b9b1c [ 5112.625071] Call Trace: [ 5112.625071] [<ffffffff812cab8c>] ? thread_return+0x60/0x108 [ 5112.625071] [<ffffffff810b9b1c>] ? find_get_page+0x28/0x89 [ 5112.625071] [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd] [ 5112.625071] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f [ 5112.625071] [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd] [ 5112.625071] [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3] [ 5112.625071] [<ffffffffa00953f5>] ext3_reserve_inode_write+0x44/0x80 [ext3] [ 5112.625071] [<ffffffffa0095460>] ext3_mark_inode_dirty+0x2f/0x4c [ext3] [ 5112.625071] [<ffffffffa0095605>] ext3_dirty_inode+0xab/0xc2 [ext3] [ 5112.625071] [<ffffffff81107392>] __mark_inode_dirty+0x34/0x15d [ 5112.625071] [<ffffffff810fc6af>] file_update_time+0xbe/0x102 [ 5112.625071] [<ffffffff810baef7>] __generic_file_aio_write+0x267/0x385 [ 5112.625071] [<ffffffff810b8427>] ? perf_swcounter_ctx_event+0x16c/0x1c6 [ 5112.625071] [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad [ 5112.625071] [<ffffffff810e75b4>] do_sync_write+0xec/0x132 [ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d [ 5112.625071] [<ffffffff8117ffa0>] ? __clear_user+0x17/0x55 [ 5112.625071] [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18 [ 5112.625071] [<ffffffff810e8053>] vfs_write+0xb3/0x16f [ 5112.625071] [<ffffffff810e81dd>] sys_write+0x4c/0x74 [ 5112.625071] [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b [ 5112.625071] perf D ffff8800b788c480 0 8588 6686 0x00000000 [ 5112.625071] ffff8800a5f115d8 0000000000000082 0000000000000000 000004a57bce3b78 [ 5112.625071] ffff8800be035f40 ffff8800a5f11fd8 000000000000dd38 0000000000013480 [ 5112.625071] 0000000000004000 0000000000013480 ffff8800a5f11558 ffffffff810b9b1c [ 5112.625071] Call Trace: [ 5112.625071] [<ffffffff810b9b1c>] ? find_get_page+0x28/0x89 [ 5112.625071] [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd] [ 5112.625071] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f [ 5112.625071] [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd] [ 5112.625071] [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3] [ 5112.625071] [<ffffffffa00953f5>] ext3_reserve_inode_write+0x44/0x80 [ext3] [ 5112.625071] [<ffffffffa0095460>] ext3_mark_inode_dirty+0x2f/0x4c [ext3] [ 5112.625071] [<ffffffffa0095605>] ext3_dirty_inode+0xab/0xc2 [ext3] [ 5112.625071] [<ffffffff81107392>] __mark_inode_dirty+0x34/0x15d [ 5112.625071] [<ffffffffa0092672>] ext3_new_blocks+0xa4/0x669 [ext3] [ 5112.625071] [<ffffffff81110195>] ? __bread+0x13/0x92 [ 5112.625071] [<ffffffffa00959c4>] ? ext3_get_branch+0x7b/0xee [ext3] [ 5112.625071] [<ffffffffa0096654>] ext3_get_blocks_handle+0x3ec/0x92d [ext3] [ 5112.625071] [<ffffffffa0096c3a>] ext3_get_block+0xa5/0xe3 [ext3] [ 5112.625071] [<ffffffff8110ec0a>] __block_prepare_write+0x1f0/0x3d2 [ 5112.625071] [<ffffffffa0076e49>] ? start_this_handle+0x37d/0x474 [jbd] [ 5112.625071] [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3] [ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d [ 5112.625071] [<ffffffff8110ef7c>] block_write_begin+0x85/0xd5 [ 5112.625071] [<ffffffffa0098130>] ext3_write_begin+0xf0/0x1e7 [ext3] [ 5112.625071] [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3] [ 5112.625071] [<ffffffff810ba9e8>] generic_file_buffered_write+0x11f/0x277 [ 5112.625071] [<ffffffff810bafe1>] __generic_file_aio_write+0x351/0x385 [ 5112.625071] [<ffffffff8100c669>] ? restore_args+0x0/0x30 [ 5112.625071] [<ffffffff812cb922>] ? mutex_lock+0x1c/0x3b [ 5112.625071] [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad [ 5112.625071] [<ffffffff810e75b4>] do_sync_write+0xec/0x132 [ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d [ 5112.625071] [<ffffffff812cab6a>] ? thread_return+0x3e/0x108 [ 5112.625071] [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18 [ 5112.625071] [<ffffffff810e8053>] vfs_write+0xb3/0x16f [ 5112.625071] [<ffffffff810e81dd>] sys_write+0x4c/0x74 [ 5112.625071] [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b [ 5112.625071] evolution.bin D 0000000000000082 0 8638 6627 0x00000000 [ 5112.625071] ffff880099339c08 0000000000000086 0000000000000000 ffffffff811157a6 [ 5112.625071] ffffffffa0096b95 ffff880099339fd8 000000000000dd38 0000000000013480 [ 5112.625071] 0000000000004000 0000000000013480 ffffea0001c5ae08 00000000009ff29b [ 5112.625071] Call Trace: [ 5112.625071] [<ffffffff811157a6>] ? mpage_readpages+0x10a/0x120 [ 5112.625071] [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3] [ 5112.625071] [<ffffffff812cacb4>] io_schedule+0x80/0xc6 [ 5112.625071] [<ffffffff810b9d22>] sync_page+0x46/0x4a [ 5112.625071] [<ffffffff810b9d34>] sync_page_killable+0xe/0x35 [ 5112.625071] [<ffffffff812cb263>] __wait_on_bit_lock+0x4a/0x92 [ 5112.625071] [<ffffffff810b9d26>] ? sync_page_killable+0x0/0x35 [ 5112.625071] [<ffffffff810b9c54>] __lock_page_killable+0x68/0x6f [ 5112.625071] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f [ 5112.625071] [<ffffffff810bb480>] generic_file_aio_read+0x3be/0x5bb [ 5112.625071] [<ffffffff810e76e6>] do_sync_read+0xec/0x132 [ 5112.625071] [<ffffffff810c176a>] ? lru_cache_add_lru+0x2b/0x2d [ 5112.625071] [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d [ 5112.625071] [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18 [ 5112.625071] [<ffffffff810e82b5>] vfs_read+0xb0/0x16c [ 5112.625071] [<ffffffff810e843f>] sys_read+0x4c/0x75 [ 5112.625071] [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b [ 5112.625071] hal-acl-tool D ffff8800b788c480 0 8675 2964 0x00000000 [ 5112.625071] ffff880037b3db78 0000000000000082 ffff88007af907e0 ffff880062b7f7e0 [ 5112.625071] ffff880037b3dac8 ffff880037b3dfd8 000000000000dd38 0000000000013480 [ 5112.625071] 0000000000004000 0000000000013480 ffff880037b3db18 ffffffff8110d414 [ 5112.625071] Call Trace: [ 5112.625071] [<ffffffff8110d414>] ? __getblk+0x2d/0x2ae [ 5112.625071] [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd] [ 5112.625071] [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f [ 5112.625071] [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd] [ 5112.625071] [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3] [ 5112.625071] [<ffffffffa0093ef8>] ext3_new_inode+0x4d4/0x96d [ext3] [ 5112.625071] [<ffffffffa009afca>] ext3_create+0x76/0xf3 [ext3] [ 5112.625071] [<ffffffff810f2be9>] vfs_create+0xa2/0x117 [ 5112.625071] [<ffffffff810f50fb>] do_filp_open+0x327/0xa39 [ 5112.625071] [<ffffffff810f2e69>] ? getname+0x31/0x1b3 [ 5112.625071] [<ffffffff810e513e>] do_sys_open+0x62/0x109 [ 5112.625071] [<ffffffff810e5218>] sys_open+0x20/0x22 [ 5112.625071] [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
Quoted 6 lines · Mike Galbraith
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines · On Mon, 21 Sep 2009 09:25:08 +0200 Mike Galbraith <[email protected]> wrote:
I still occasionally see some largish numbers though, and suspect that depends on how much cache was evicted.
Quoted 6 lines
Quoted 2 lines
if you run latencytop it'll at least tell you what kind of operating was going on that hit the delay...
One of the really bad ones for evolution is below. It was so bad that I had _plenty_ of time to get to a GUI free shell, and poke SysRq-W before killing the darn dd.
P.S. evolution is always journal journal and more journal.
Quoted 2 lines · Jiri Kosina
I guess that switching from CFQ to deadline I/O scheduler improves the situation, right?
Indeed, there is an improvement. But, especially in the case with an encrypted /home, copying a large file from a fast source means I better go get another cup of coffee rather than continue working on something else in the meantime.
BTW, I've tested this with a default install of OpenSUSE 11.1, and I get basically the same behaviour.