Skip to contentSkip to conversation

bdflush and postgres stuck in D state

DISCUSSION Jakob Østergaard 3 participants 5 messages Sep 18, 2001 – Sep 19, 2001

lore
Share & tools
lore
#0 lore raw

Hello,

I have a machine here with (RedHat 7.0 plus official updates plus kernel.org kernel):

[osprey:joe] $ uname -a
Linux osprey 2.4.7 #1 Sat Jul 21 21:50:21 CEST 2001 i686 unknown

[osprey:joe] $ gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-85)

/ is mounted on a four disk (software) RAID-5, the same four disks are used for a RAID-1 but the fs on there is mounted somewhere irrelevant for this post.

All fs are ext2. The machine is not heavily loaded at all. What made me wonder was, that the load on the machine was '1' some days ago, today it was '2'. The machine usually has 90%+ CPU idle, and doesn't use the disks very much. Looking at 'ps' shows:

[osprey:joe] $ ps ax|grep ' D'

    6 ?        DW     0:26 [bdflush]
10023 ?        D      0:00 /usr/bin/postmaster -D /var/lib/pgsql/data

But there is *NO* disk activity. The processes are just stuck. As far as
I can see, there's nothing even remotely suspicious in dmesg.

Any ideas ? I can dig further before rebooting and trying 2.4.9
if someone tells me where to dig...

Signature · 6 lines
................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
#1 lore raw

Sorry for following up on my own post, I have a little extra information.

I started a g++ job to try to force the machine to write out some dirty
buffers before I reboot. g++ now hangs along with two sync's, bdflush
and the postgres process.

This is from top:

  PID USER     PRI  NI  SIZE  RSS SHARE WCHAN     STAT %CPU %MEM   TIME COMMAN
    6 root       9   0     0    0     0 raid1_all DW    0.0  0.0   0:26 bdflush
 1140 joe        9   0 71564  39M     0 wait_on_b D     0.0 32.3   1:04 cc1plus
 1007 root       9   0    72    4     4 wait_on_b D     0.0  0.0   0:00 sync
10023 postgres   9   0   368    4     4 wait_on_b D     0.0  0.0   0:00 postmas

Seems like something's rotten with bdflush and raid1_all (-something).

There is one (software) RAID1 on four SCSI disks in the machine, perhaps RAID1 has a misfeature when more than just two disks are in the mirror ?

Anyway, this machine is going down now - I can't wait anymore, sorry. I wonder how many file writes didn't make it...

Signature · 6 lines
................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
#2 lore raw
In reply to Jakob Østergaard
Quoted 8 lines · Jakob Østergaard

Sorry for following up on my own post, I have a little extra information.

I started a g++ job to try to force the machine to write out some dirty
buffers before I reboot. g++ now hangs along with two sync's, bdflush
and the postgres process.

Since 2.4.7 several bugs have been fixed in RAID1 which would cause this, including a missing blockdevice unplug and failure to hang onto the supposedly-reserved RAID1 buffer-heads.

-

#3 lore raw
In reply to Jakob Østergaard
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Jakob Østergaard

Sorry for following up on my own post, I have a little extra information.

Quoted 8 lines
Quoted 4 lines

I started a g++ job to try to force the machine to write out some dirty
buffers before I reboot. g++ now hangs along with two sync's, bdflush
and the postgres process.

Since 2.4.7 several bugs have been fixed in RAID1 which would cause this, including a missing blockdevice unplug and failure to hang onto the supposedly-reserved RAID1 buffer-heads.

Even kernels as recent as 2.4.9 have this bug. See this thread for more info and a patch which fixes this bug.

The thread:
http://marc.theaimsgroup.com/?t=99911655500004&w=2&r=1

The patch:
http://marc.theaimsgroup.com/?l=linux-kernel&m=99913223508789&w=2

-Dave

#4 lore raw
In reply to Jakob Østergaard
Quoted 15 lines · David Rees
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Jakob Østergaard

Sorry for following up on my own post, I have a little extra information.

Quoted 8 lines
Quoted 4 lines

I started a g++ job to try to force the machine to write out some dirty
buffers before I reboot. g++ now hangs along with two sync's, bdflush
and the postgres process.

Since 2.4.7 several bugs have been fixed in RAID1 which would cause this, including a missing blockdevice unplug and failure to hang onto the supposedly-reserved RAID1 buffer-heads.

Quoted 8 lines

Even kernels as recent as 2.4.9 have this bug. See this thread for more info and a patch which fixes this bug.

The thread:
http://marc.theaimsgroup.com/?t=99911655500004&w=2&r=1

The patch:
http://marc.theaimsgroup.com/?l=linux-kernel&m=99913223508789&w=2

Thanks a lot !

Somehow I seem not have lost "most" linux-raid mails, dunno why... I hadn't seen that thread before, but it was indeed the problem I saw here too.

I didn't lose any data on the 2.4.7 that did this, but it seems the situation is more severe in 2.4.9, leading potentially to significant data loss.

/me prepares another boot (and a spare 32MB stick) for the raid-1 box

Signature · 6 lines
................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............: