Sudden "hangs"....
Hi,
It seems my home workstation hangs on a "mkdir" once every morning for about half a minute. It'll freeze whatever I'm doing and continue happily 30 seconds later, but I can't figure out what's going on because it's frozen...
My server did it just now and I was able to grab some more debugging info.....
obelix:~> time mkdir /recover3/testdir
0.000u 0.004s 2:11.83 0.0% 0+0k 0+0io 5pf+0w
obelix:~>
Over two minutes to make a directory! (I have the impression that the wait is for something to finish, so if I had started the mkdir a minute earlier, it would've taken a minute longer).
The process:
28649 root 20 0 0 0 0 D 6.3 0.0 0:11.69 flush-9:0
was in "top", and as shown here in "disk wait" mode.
While this is going on, "vmstat 1" shows:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 5368 12764 324804 1616932 0 0 464 0 244 477 0 2 0 98 0 1 5368 12764 324556 1617168 0 0 448 0 255 474 0 1 0 99 0 1 5368 12764 324952 1616832 0 0 472 0 257 487 0 1 0 99
... about 400 kbytes of IO per second. This is similar to the single-disk machine at home.
The server runs:
Linux version 2.6.34 (wolff@abra2) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #5 SMP Tue Jun 1 14:44:01 CEST 2010
at home I run:
Linux version 2.6.28-19-generic (buildd@palmer) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #61-Ubuntu SMP Wed May 26 23:35:15 UTC 2010
Does anybody know what's going on?
Roger.
Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. Does it sit on the couch all day? Is it unemployed? Please be specific! Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
Hi,
Quoted 24 lines · Rogier Wolff
Hi,
It seems my home workstation hangs on a "mkdir" once every morning for about half a minute. It'll freeze whatever I'm doing and continue happily 30 seconds later, but I can't figure out what's going on because it's frozen...
My server did it just now and I was able to grab some more debugging info.....
obelix:~> time mkdir /recover3/testdir
0.000u 0.004s 2:11.83 0.0% 0+0k 0+0io 5pf+0w
obelix:~>Over two minutes to make a directory! (I have the impression that the wait is for something to finish, so if I had started the mkdir a minute earlier, it would've taken a minute longer).
The process:
28649 root 20 0 0 0 0 D 6.3 0.0 0:11.69 flush-9:0was in "top", and as shown here in "disk wait" mode.
Quoted 7 lines
While this is going on, "vmstat 1" shows:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 5368 12764 324804 1616932 0 0 464 0 244 477 0 2 0 98 0 1 5368 12764 324556 1617168 0 0 448 0 255 474 0 1 0 99 0 1 5368 12764 324952 1616832 0 0 472 0 257 487 0 1 0 99
It seems normal data. I suggest you to do the check dmesg and iostat to confirm whether something wrong is occuring on your disk which you issued mkdir.
Thanks,
Satoru
Quoted 17 lines
... about 400 kbytes of IO per second. This is similar to the single-disk machine at home.
The server runs:
Linux version 2.6.34 (wolff@abra2) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #5 SMP Tue Jun 1 14:44:01 CEST 2010
at home I run:
Linux version 2.6.28-19-generic (buildd@palmer) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #61-Ubuntu SMP Wed May 26 23:35:15 UTC 2010Does anybody know what's going on?
Roger.
Quoted 4 lines · Rogier Wolff
It seems my home workstation hangs on a "mkdir" once every morning for about half a minute. It'll freeze whatever I'm doing and continue happily 30 seconds later, but I can't figure out what's going on because it's frozen...
One of my volumes was doing this. As someone mentioned earlier, check your kernel log to see if you have a disk error. On mine, I had no errors. The activity led for that volume would be on solid until the mkdir completed. My fix was to create a new directory and move everything over and remove the old directory. Obviously this won't work if you're having problems with more than 1 directory or the offending directory is the root of the volume.
Just for information, this volume I have is on a single disk created on the whole disk (no lvm nor partition).
Signature · 2 lines
Microsoft has beaten Volkswagen's world record. Volkswagen only created 22 million bugs.
Quoted 8 lines · Wakko Warner
Quoted 4 lines · Rogier Wolff
It seems my home workstation hangs on a "mkdir" once every morning for about half a minute. It'll freeze whatever I'm doing and continue happily 30 seconds later, but I can't figure out what's going on because it's frozen...
One of my volumes was doing this. As someone mentioned earlier, check your kernel log to see if you have a disk error.
I work at "harddisk-recovery.nl". I know how to recognize bad disks. You can recognize a bad block in the vmstat output in that the bi and bo numbers go to zero for at least 10 seconds. This doesn't happen, as you can see in the original post.
Quoted 2 lines
On mine, I had no errors. The activity led for that volume would be on solid until the mkdir completed.
Besides that I can't see the activity led from my chair. But vmstat shows that it would be solid indeed.
Quoted 4 lines
My fix was to create a new
directory and move everything over and remove the old directory. Obviously this won't work if you're having problems with more than 1 directory or the offending directory is the root of the volume.
This helps for the "very big directory" case. Directories are never shrunk. So if a directory has become very large because of thousands of files at one point in time, you can reduce that with the move-all-files trick.
When it happened and I recognized the effect, I typed the "time mkdir" command. I just chose the root of the partition for this. The directory there is NOT large:
drwxrwsr-x 32 root hdr 4096 2010-09-17 22:05 .
In fact it is as small as possible.
IMHO, the hint is that the flush process for the device ends up in top. Now if I understand the word "flush" correctly I'd expect it to write dirty buffers to the device. In this case, vmstat however reports mostly data coming INTO the computer. So why would the flush process need to READ data from the device?
Roger.
Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. Does it sit on the couch all day? Is it unemployed? Please be specific! Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ