Finding what is stuck...
Hi all...
I'm running 2.6.27-rc5-git2 on an Aspire One.
The system is in general pretty responsive, but sometimes it just gets totally stuck. Even the mouse stops.
It looks related to disk (SSD) access, but I'm not totally sure. Is there any way to find what is getting stuck ? I know that SSDs can be slow on write, I don't mind if the system is faster or slower (it's small :))), but if the speed is constant. That occasional pauses are strange, like if SSD flushing gets stuck on BKL (I know, no idea about what I talk...).
I'm using ext3 fs, noop iosched. But as I say, I'm not sure that the disk writes are the culprit.
Any idea about how to find this ?
TIA
Signature · 4 lines
J.A. Magallon \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux kernel 2.6.23-jam01, gcc version 4.2.2 20070909Quoted 13 lines · On Tue, 2 Sep 2008 02:04:47 +0200 "J.A. Magallón" <[email protected]> wrote:
Hi all...
I'm running 2.6.27-rc5-git2 on an Aspire One.
The system is in general pretty responsive, but sometimes it just gets totally stuck. Even the mouse stops.It looks related to disk (SSD) access, but I'm not totally sure. Is there any way to find what is getting stuck ? I know that SSDs can be slow on write, I don't mind if the system is faster or slower (it's small :))), but if the speed is constant. That occasional pauses are strange, like if SSD flushing gets stuck on BKL (I know, no idea about what I talk...).
Quoted 4 lines
I'm using ext3 fs, noop iosched. But as I say, I'm not sure that the disk writes are the culprit.
Any idea about how to find this ?
Have you tried to run "latencytop"?
(you need to enable this in the kernel config as well)
it tends to (for me at least) point out very well where stalls happen, or at least, what the system is doing when they happen.
(hint: make sure you do "make install" before running it)
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 21 lines · Arjan van de Ven
Quoted 13 lines · On Tue, 2 Sep 2008 02:04:47 +0200 "J.A. Magallón" <[email protected]> wrote:
Hi all...
I'm running 2.6.27-rc5-git2 on an Aspire One.
The system is in general pretty responsive, but sometimes it just gets totally stuck. Even the mouse stops.It looks related to disk (SSD) access, but I'm not totally sure. Is there any way to find what is getting stuck ? I know that SSDs can be slow on write, I don't mind if the system is faster or slower (it's small :))), but if the speed is constant. That occasional pauses are strange, like if SSD flushing gets stuck on BKL (I know, no idea about what I talk...).
Quoted 4 lines
I'm using ext3 fs, noop iosched. But as I say, I'm not sure that the disk writes are the culprit.
Any idea about how to find this ?
Quoted 8 lines
Have you tried to run "latencytop"?
(you need to enable this in the kernel config as well)it tends to (for me at least) point out very well where stalls happen, or at least, what the system is doing when they happen.
(hint: make sure you do "make install" before running it)
Thanks, I'm a bit limited to distro kernels 'cause I don't want to build my own kernel on the One, but I have just looked and Mandriva kernel is built with LATENCYTOP=y. I will try that and report...
Signature · 4 lines
J.A. Magallon \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux kernel 2.6.23-jam01, gcc version 4.2.2 20070909Quoted 21 lines · Arjan van de Ven
Quoted 13 lines · On Tue, 2 Sep 2008 02:04:47 +0200 "J.A. Magallón" <[email protected]> wrote:
Hi all...
I'm running 2.6.27-rc5-git2 on an Aspire One.
The system is in general pretty responsive, but sometimes it just gets totally stuck. Even the mouse stops.It looks related to disk (SSD) access, but I'm not totally sure. Is there any way to find what is getting stuck ? I know that SSDs can be slow on write, I don't mind if the system is faster or slower (it's small :))), but if the speed is constant. That occasional pauses are strange, like if SSD flushing gets stuck on BKL (I know, no idea about what I talk...).
Quoted 4 lines
I'm using ext3 fs, noop iosched. But as I say, I'm not sure that the disk writes are the culprit.
Any idea about how to find this ?
Quoted 8 lines
Have you tried to run "latencytop"?
(you need to enable this in the kernel config as well)it tends to (for me at least) point out very well where stalls happen, or at least, what the system is doing when they happen.
(hint: make sure you do "make install" before running it)
These are some shots of latencytop while working. I copied the screen when I saw any very high timing...
Cause Maximum Percentage
Walking directory tree 790.2 msec 10.6 %
Scheduler: waiting for cpu 40.5 msec 38.5 %
fsync() on a file 3191.6 msec 36.7 % Reading EXT3 directory 1553.3 msec 3.9 % EXT3: Waiting for journal access 1518.7 msec 3.8 % Deleting an inode 657.5 msec 1.7 % EXT3: Looking for file 576.3 msec 1.4 % Writing a page to disk 363.0 msec 6.1 %
call_usermodehelper_exec request_module __sock_cre 69.4 msec 0.2 %
Scheduler: waiting for cpull 28.0 msec 18.9 %
Page fault 7.9 msec 0.3 %
fsync() on a file 8843.7 msec 81.8 % Writing a page to disk 1548.6 msec 2.9 % EXT3: Waiting for journal access 356.5 msec 1.1 %
Scheduler: waiting for cpu 40.6 msec 6.0 %
EXT3: Waiting for journal access 6153.6 msec 17.2 % fsync() on a file 4902.7 msec 53.5 % Writing a page to disk 3487.2 msec 9.8 % Writing buffer to disk (synchronous) 2288.7 msec 4.6 % synchronous write 1320.6 msec 2.6 % Truncating file 302.6 msec 0.6 % Page fault 290.5 msec 0.6 %
Scheduler: waiting for cpu 13.4 msec 4.5 %
Waiting for event (select) 5.0 msec 5.0 %
Deleting an inode 5584.7 msec 27.5 % Writing a page to disk 829.2 msec 5.4 % fsync() on a file 265.1 msec 11.2 % Scheduler: waiting for cpu 37.2 msec 11.5 % stat() operation 5.6 msec 0.1 % Page fault 5.5 msec 0.2 % Waiting for event (select) 5.0 msec 39.8 % Waiting for event (poll) 5.0 msec 2.5 % synchronous write 4.2 msec 0.0 %
All look fs access:
one:~# df
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext3 6.9G 3.7G 3.2G 55% /
tmpfs tmpfs 246M 12K 246M 1% /tmp
/dev/mmcblk0p1
ext3 1.9G 35M 1.8G 2% /storeI will try mounting root as ext2, to rule out jorunaling...
TIA
Signature · 4 lines
J.A. Magallon \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux kernel 2.6.23-jam01, gcc version 4.2.2 20070909Quoted 27 lines · On Thu, 4 Sep 2008 01:00:43 +0200 "J.A. Magallón" <[email protected]> wrote:
Quoted 24 lines · Arjan van de Ven
Quoted 13 lines · On Tue, 2 Sep 2008 02:04:47 +0200 "J.A. Magallón" <[email protected]> wrote:
Hi all...
I'm running 2.6.27-rc5-git2 on an Aspire One.
The system is in general pretty responsive, but sometimes it just gets totally stuck. Even the mouse stops.It looks related to disk (SSD) access, but I'm not totally sure. Is there any way to find what is getting stuck ? I know that SSDs can be slow on write, I don't mind if the system is faster or slower (it's small :))), but if the speed is constant. That occasional pauses are strange, like if SSD flushing gets stuck on BKL (I know, no idea about what I talk...).
Quoted 4 lines
I'm using ext3 fs, noop iosched. But as I say, I'm not sure that the disk writes are the culprit.
Any idea about how to find this ?
Have you tried to run "latencytop"?
(you need to enable this in the kernel config as well)
Quoted 9 lines
Quoted 5 lines
it tends to (for me at least) point out very well where stalls happen, or at least, what the system is doing when they happen.
(hint: make sure you do "make install" before running it)
These are some shots of latencytop while working. I copied the screen when I saw any very high timing...
wow bad ones..
one thing to note.. you have *something* doing fsync() a lot it seems (latencytop is likely to tell you which one it is); fsync on ext3 is really expensive, especially on an ssd that is slow to write to.
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 5 lines · J.A. Magallón
I'm running 2.6.27-rc5-git2 on an Aspire One.
The system is in general pretty responsive, but sometimes it just gets totally stuck. Even the mouse stops.It looks related to disk (SSD) access, but I'm not totally sure.
Are you hitting swap at all? Does it have swap?
On my N810 tablet there can be a huge slowdown if it starts swapping. The first time it dips into swap is relatively painless when stuff gets written out sequentially, but after that when writes are mostly random the throughput drops to a few kilobytes per second, causing massive slowdown... The MMC/SD cards hate random write, and I expect SSDs are no better.
iostat -x -k 10
in a terminal is useful. Check the iowait and util% numbers after/during slowdown...