pvmove and 100% IO Wait
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Yohoo!
I've some issues with pvmove.
It is very likely to appear during a pvmove operation for my server to spike to 100% IO waits.
If this happens, no IO on the LVs is possible, even the rest of the server is accessible without any issues.
I can't predict when it happens, but it happens...
pvmove started. Some minutes or hours later the io wait on the server goes up to 100% (or 3 cores of the quad core are going to 100%).
Running a RedHat EL 5.x (CentOS) with all current updates. The storage consists of two 3Ware 9500 PCI-X, a USB-SATA drive and three disks connected to the local SATA controller.
I can't see anything relevant in the logfiles. When the CPU is up, I have to reset the server as any operation on the LV will never finish :-(
Any clues?
Christian
Any clues?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFJFfgN0XNIYlAXmzsRAih2AKCk2ItzDHX0s4n+pCHdnom9wh4WQgCeOlba PDfvqDf1WmgYDmPbB49Ltso=
=nvvi
-----END PGP SIGNATURE-----
Quoted 3 lines
The storage
consists of two 3Ware 9500 PCI-X, a USB-SATA drive and three disks connected to the local SATA controller.
is the USB-SATA at all involved in the pvmove?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Yohoo!
Quoted 3 lines
Quoted 2 lines
consists of two 3Ware 9500 PCI-X, a USB-SATA drive and three disks connected to the local SATA controller.
is the USB-SATA at all involved in the pvmove?
Hm...good question. I'd say no- but I can't say for sure where any temporary PEs have been allocated.
The last pvmove FROM the USB-SATA went without any issues (500GB!).
Is it relevant?
Cheers
Christian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFJFfuU0XNIYlAXmzsRAsTbAJ0daUInw5jMl4veCwctLdjK12h1EwCfSBQJ In7oAFNWQkKka1KfFU2OO14=
=HDDo
-----END PGP SIGNATURE-----
Quoted 3 lines
The last pvmove FROM the USB-SATA went without any issues (500GB!).
Is it relevant?
Might not be, but if an operation involves multiple disks it will probably go at the pace of the slowest disk, which for me has always been USB...
Are you 100% certainty the devices underlying all PVs are good, that you can dd 100% successfully?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Yohoo!
Quoted 3 lines
Might not be, but if an operation involves multiple disks it will probably go at the pace of the slowest disk, which for me has always been USB...
I agree about the speed- but not with the issue of hanging. When pvmove works the IO load is moderate (about 20% or less). But when it hangs it jumps up to 100% on one or more cores.
Quoted 2 lines
Are you 100% certainty the devices underlying all PVs are good, that you can dd 100% successfully?
Yes. I can do resize2fs, cp, mv and dd, too. Well, I haven't tried to dd the whole devices as they are aprox. 1.5TB in size. But I'd say this is fine!
GReetings
Christian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFJFmcM0XNIYlAXmzsRAiQOAJ9Fo6SA3+yBGlZ6Ky0bKDw1sStAGACgmprm pT+asXOQS/a0MMQ2jQoCeGo=
=eMR2
-----END PGP SIGNATURE-----
Quoted 9 lines
Quoted 4 lines
Might not be, but if an operation involves multiple disks it will probably go at the pace of the slowest disk, which for me has always been USB...
I agree about the speed- but not with the issue of hanging. When pvmove works the IO load is moderate (about 20% or less). But when it hangs it jumps up to 100% on one or more cores.
I ran into similar issues when attempting a pvmove with clustered VG's under CLVMD. I had to change the clustered VG to non-clustered and then I could do the pvmove. You didn't mention clusters, but thought I would share my experiences.
-Jeremy
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Yohoo!
Quoted 2 lines
I ran into similar issues when attempting a pvmove with clustered VG's under CLVMD.
What is CLVMD?
Anyway: It happened again and USB was not involved at all! All SATA disks or SATA RAIDs.
$%^$%^�!!!
Any clues?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFJG7h80XNIYlAXmzsRAtHXAKDf0EyUvZngOLzoMoIHsWj+VxtWUQCcCqR2 E1COMYVO2FaqxB88K4wBGgY=
=+riP
-----END PGP SIGNATURE-----