bdflush and postgres stuck in D state
Hello,
I have a machine here with (RedHat 7.0 plus official updates plus kernel.org kernel):
[osprey:joe] $ uname -a
Linux osprey 2.4.7 #1 Sat Jul 21 21:50:21 CEST 2001 i686 unknown
[osprey:joe] $ gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-85)
/ is mounted on a four disk (software) RAID-5, the same four disks are used for a RAID-1 but the fs on there is mounted somewhere irrelevant for this post.
All fs are ext2. The machine is not heavily loaded at all. What made me wonder was, that the load on the machine was '1' some days ago, today it was '2'. The machine usually has 90%+ CPU idle, and doesn't use the disks very much. Looking at 'ps' shows:
[osprey:joe] $ ps ax|grep ' D'
6 ? DW 0:26 [bdflush] 10023 ? D 0:00 /usr/bin/postmaster -D /var/lib/pgsql/data
But there is *NO* disk activity. The processes are just stuck. As far as
I can see, there's nothing even remotely suspicious in dmesg.
Any ideas ? I can dig further before rebooting and trying 2.4.9
if someone tells me where to dig...
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............:
Sorry for following up on my own post, I have a little extra information.
I started a g++ job to try to force the machine to write out some dirty
buffers before I reboot. g++ now hangs along with two sync's, bdflush
and the postgres process.
This is from top:
PID USER PRI NI SIZE RSS SHARE WCHAN STAT %CPU %MEM TIME COMMAN
6 root 9 0 0 0 0 raid1_all DW 0.0 0.0 0:26 bdflush
1140 joe 9 0 71564 39M 0 wait_on_b D 0.0 32.3 1:04 cc1plus
1007 root 9 0 72 4 4 wait_on_b D 0.0 0.0 0:00 sync
10023 postgres 9 0 368 4 4 wait_on_b D 0.0 0.0 0:00 postmasSeems like something's rotten with bdflush and raid1_all (-something).
There is one (software) RAID1 on four SCSI disks in the machine, perhaps RAID1 has a misfeature when more than just two disks are in the mirror ?
Anyway, this machine is going down now - I can't wait anymore, sorry. I wonder how many file writes didn't make it...
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............:
Quoted 8 lines · Jakob Østergaard
Sorry for following up on my own post, I have a little extra information.
I started a g++ job to try to force the machine to write out some dirty
buffers before I reboot. g++ now hangs along with two sync's, bdflush
and the postgres process.
Since 2.4.7 several bugs have been fixed in RAID1 which would cause this, including a missing blockdevice unplug and failure to hang onto the supposedly-reserved RAID1 buffer-heads.
-
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Jakob Østergaard
Sorry for following up on my own post, I have a little extra information.
Quoted 8 lines
Quoted 4 lines
I started a g++ job to try to force the machine to write out some dirty
buffers before I reboot. g++ now hangs along with two sync's, bdflush
and the postgres process.Since 2.4.7 several bugs have been fixed in RAID1 which would cause this, including a missing blockdevice unplug and failure to hang onto the supposedly-reserved RAID1 buffer-heads.
Even kernels as recent as 2.4.9 have this bug. See this thread for more info and a patch which fixes this bug.
The thread:
http://marc.theaimsgroup.com/?t=99911655500004&w=2&r=1
The patch:
http://marc.theaimsgroup.com/?l=linux-kernel&m=99913223508789&w=2
-Dave
Quoted 15 lines · David Rees
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Jakob Østergaard
Sorry for following up on my own post, I have a little extra information.
Quoted 8 lines
Quoted 4 lines
I started a g++ job to try to force the machine to write out some dirty
buffers before I reboot. g++ now hangs along with two sync's, bdflush
and the postgres process.Since 2.4.7 several bugs have been fixed in RAID1 which would cause this, including a missing blockdevice unplug and failure to hang onto the supposedly-reserved RAID1 buffer-heads.
Quoted 8 lines
Even kernels as recent as 2.4.9 have this bug. See this thread for more info and a patch which fixes this bug.
The thread:
http://marc.theaimsgroup.com/?t=99911655500004&w=2&r=1The patch:
http://marc.theaimsgroup.com/?l=linux-kernel&m=99913223508789&w=2
Thanks a lot !
Somehow I seem not have lost "most" linux-raid mails, dunno why... I hadn't seen that thread before, but it was indeed the problem I saw here too.
I didn't lose any data on the 2.4.7 that did this, but it seems the situation is more severe in 2.4.9, leading potentially to significant data loss.
/me prepares another boot (and a spare 32MB stick) for the raid-1 box
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: