Sudden instabilty of mysql boxen
----- Original Message -----
From: Andreas Haumer <[email protected]>
Date: Thu, 12 Jun 2003 13:24:12 +0200
To: Christopher Chan <[email protected]>
Subject: Re: Sudden instabilty of mysql boxen
Quoted 15 lines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Hi!
Quoted 9 lines · Christopher Chan
Can somebody help out with pointers to why my boxes would suddenly become unstable?
I have two boxes running mysql servers. The first one is running RH 7.3 with a 2.4.18-26 kernel that has the 3ware no bounce buffer patch enabled (the src.rpm was downloaded, 3ware patch enabled and rpmbuild) and the second one is running Rh 7.0 with a 2.4.2-ac28 kernel. The 3ware box handles over 600 queries per second.
Quoted 6 lines
Quoted 4 lines
The first box runs on a linux software mirrored array but has the mysql data on a 3ware stripped mirrored array while the second one runs on a linux software mirrored array (got two disks only).
I assume you didn't change anything of the (previously working) setup?
Zippo
Quoted 19 lines
Quoted 10 lines
They both suddenly started vomiting messages like:
Code: 8b 42 04 83 f8 ff 0f 84 6f 01 00 00 83 f8 fc 77 07 c7 42 04 <1>Unable to handle kernel NULL pointer dereference at virtualaddress 00000004
Code: 8b 42 04 83 f8 ff 0f 84 6f 01 00 00 83 f8 fc 77 07 c7 42 04 Oops: 0000Quoted 8 lines
and any commands you run on shells (if you them in the first place) would exit with a segmentation fault. All attempts to ssh into the box would fail after successful authentication with the connection being closed (bash segfault?)
A reboot would cause them to run normally until they start acting up again minutes later.
Quoted 6 lines
Are both boxes located in the same room?
Maybe they're getting too hot now for some reason (ambient air temperature too high, cooler and/or fan not working right)? Also check your mains power supply. If it's unstable (e.g. voltage too low), it could cause such effects, too (but this is usually less likely than overheating).
Nope. The non-3ware box is working perfectly fine now.
The 3ware box was taken out of production and a backup made the master instead.
I have now put some load back on the 3ware (it is now a slave) by pointing some queries its way and the problem has not resurfaced.
These two boxes are located in a data centre along with lots of other boxes. Only these two boxes had problems and they work fine now.
I am lead to believe that something else is causing the weird behaviour for they were running perfectly fine, and especially the non-3ware one since that has been running for ages and we've not done anything on them prior to their acting up. The 3ware box was the first and it kept on for a couple hours before we took it out of production. A day later, the non-3ware box (they happen to be related in the system although performing different functions) also had one case of the same funny behaviour/problem.
Quoted 4 lines
HTH
- - andreas
Signature · 4 lines
_______________________________________________ Get your free email from http://www.graffiti.net Powered by Outblaze