Skip to contentSkip to conversation

Sudden "hangs"....

DISCUSSION Rogier Wolff 3 participants 4 messages Sep 16, 2010 – Sep 18, 2010

lore
Share & tools
lore
#0 lore raw

Hi,

It seems my home workstation hangs on a "mkdir" once every morning for about half a minute. It'll freeze whatever I'm doing and continue happily 30 seconds later, but I can't figure out what's going on because it's frozen...

My server did it just now and I was able to grab some more debugging info.....

obelix:~> time mkdir /recover3/testdir
0.000u 0.004s 2:11.83 0.0% 0+0k 0+0io 5pf+0w
obelix:~>

Over two minutes to make a directory! (I have the impression that the wait is for something to finish, so if I had started the mkdir a minute earlier, it would've taken a minute longer).

The process:
28649 root 20 0 0 0 0 D 6.3 0.0 0:11.69 flush-9:0

was in "top", and as shown here in "disk wait" mode.

While this is going on, "vmstat 1" shows:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  1   5368  12764 324804 1616932    0    0   464     0  244  477  0  2  0 98
 0  1   5368  12764 324556 1617168    0    0   448     0  255  474  0  1  0 99
 0  1   5368  12764 324952 1616832    0    0   472     0  257  487  0  1  0 99

... about 400 kbytes of IO per second. This is similar to the single-disk machine at home.

The server runs:

Linux version 2.6.34 (wolff@abra2) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #5 SMP Tue Jun 1 14:44:01 CEST 2010

at home I run:
Linux version 2.6.28-19-generic (buildd@palmer) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #61-Ubuntu SMP Wed May 26 23:35:15 UTC 2010

Does anybody know what's going on?

Roger.

Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
#1 lore raw

Hi,

Quoted 24 lines · Rogier Wolff

Hi,

It seems my home workstation hangs on a "mkdir" once every morning for about half a minute. It'll freeze whatever I'm doing and continue happily 30 seconds later, but I can't figure out what's going on because it's frozen...

My server did it just now and I was able to grab some more debugging info.....

obelix:~> time mkdir /recover3/testdir
0.000u 0.004s 2:11.83 0.0% 0+0k 0+0io 5pf+0w
obelix:~>

Over two minutes to make a directory! (I have the impression that the wait is for something to finish, so if I had started the mkdir a minute earlier, it would've taken a minute longer).

The process:
28649 root 20 0 0 0 0 D 6.3 0.0 0:11.69 flush-9:0

was in "top", and as shown here in "disk wait" mode.

Quoted 7 lines

While this is going on, "vmstat 1" shows:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
  0  1   5368  12764 324804 1616932    0    0   464     0  244  477  0  2  0 98
  0  1   5368  12764 324556 1617168    0    0   448     0  255  474  0  1  0 99
  0  1   5368  12764 324952 1616832    0    0   472     0  257  487  0  1  0 99

It seems normal data. I suggest you to do the check dmesg and iostat to confirm whether something wrong is occuring on your disk which you issued mkdir.

Thanks,
Satoru

Quoted 17 lines

... about 400 kbytes of IO per second. This is similar to the single-disk machine at home.

The server runs:

Linux version 2.6.34 (wolff@abra2) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #5 SMP Tue Jun 1 14:44:01 CEST 2010

at home I run:
Linux version 2.6.28-19-generic (buildd@palmer) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #61-Ubuntu SMP Wed May 26 23:35:15 UTC 2010

Does anybody know what's going on?

Roger.

#2 lore raw
In reply to Rogier Wolff
Quoted 4 lines · Rogier Wolff

It seems my home workstation hangs on a "mkdir" once every morning for about half a minute. It'll freeze whatever I'm doing and continue happily 30 seconds later, but I can't figure out what's going on because it's frozen...

One of my volumes was doing this. As someone mentioned earlier, check your kernel log to see if you have a disk error. On mine, I had no errors. The activity led for that volume would be on solid until the mkdir completed. My fix was to create a new directory and move everything over and remove the old directory. Obviously this won't work if you're having problems with more than 1 directory or the offending directory is the root of the volume.

Just for information, this volume I have is on a single disk created on the whole disk (no lvm nor partition).

Signature · 2 lines
 Microsoft has beaten Volkswagen's world record.  Volkswagen only created 22
 million bugs.
#3 lore raw
Quoted 8 lines · Wakko Warner
Quoted 4 lines · Rogier Wolff

It seems my home workstation hangs on a "mkdir" once every morning for about half a minute. It'll freeze whatever I'm doing and continue happily 30 seconds later, but I can't figure out what's going on because it's frozen...

One of my volumes was doing this. As someone mentioned earlier, check your kernel log to see if you have a disk error.

I work at "harddisk-recovery.nl". I know how to recognize bad disks. You can recognize a bad block in the vmstat output in that the bi and bo numbers go to zero for at least 10 seconds. This doesn't happen, as you can see in the original post.

Quoted 2 lines

On mine, I had no errors. The activity led for that volume would be on solid until the mkdir completed.

Besides that I can't see the activity led from my chair. But vmstat shows that it would be solid indeed.

Quoted 4 lines

My fix was to create a new
directory and move everything over and remove the old directory. Obviously this won't work if you're having problems with more than 1 directory or the offending directory is the root of the volume.

This helps for the "very big directory" case. Directories are never shrunk. So if a directory has become very large because of thousands of files at one point in time, you can reduce that with the move-all-files trick.

When it happened and I recognized the effect, I typed the "time mkdir" command. I just chose the root of the partition for this. The directory there is NOT large:

drwxrwsr-x 32 root hdr 4096 2010-09-17 22:05 .

In fact it is as small as possible.

IMHO, the hint is that the flush process for the device ends up in top. Now if I understand the word "flush" correctly I'd expect it to write dirty buffers to the device. In this case, vmstat however reports mostly data coming INTO the computer. So why would the flush process need to READ data from the device?

Roger.

Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ