2.6.0-test2-mm3 and mysql
Sergey,
does it always crash when you start mysqld?
It is page number 0 in the InnoDB tablespace. That is, the header page of the whole tablespace!
The checksums in the page are ok. That shows the page was not corrupted in the Linux file system.
InnoDB is trying to do an index search, but that of course crashes, because the header page is not any index page.
The reason for the crash is probably that a page number in a pointer record in the father node of the B-tree has been reset to zero. The corruption has happened in the mysqld process memory, not in the file system of Linux. Otherwise, InnoDB would have complained about page checksum errors.
No one else has reported this error. I have now added a check to a future version of InnoDB which will catch this particular error earlier and will hex dump the father page.
By the way, I noticed that a website http://www.linuxtestproject.org has made an extensive regression test suite for Linux. They have also successfully run big MySQL and DB2 stress tests on their computers, on 2.5.xx kernels. If there is something wrong with 2.5.xx or 2.6.0, it apparently does not concern all computers.
"
The Linux Test Project test suite, ltp-20030807, has been released. The latest version of the testsuite contains 2000+ tests for the Linux OS. "
The general picture about InnoDB corruption is that reports have almost stopped after I advised people on the mailing list to upgrade to Linux-2.4.20 kernels.
With apologies,
Heikki
Innobase Oy
http://www.innodb.com
"
030827 15:34:10 InnoDB: Page checksum 1165918361, prior-to-4.0.14-form checksum 4088416325
InnoDB: stored checksum 1165918361, prior-to-4.0.14-form stored checksum 4088416325
InnoDB: Page lsn 0 4080819655, low 4 bytes of lsn at page end 4080819655
InnoDB: Page directory corruption: supremum not pointed to 030827 15:34:10 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 457e809900000000000000000000000000000000f33c5dc7000000000000f356ce970000000100000000000000 0000040f0000040240000000000000006c00000002000400000 1b60004000001de0000000400028000144e00040000009e0000 00360000000001160002800015de0000000000000b410000000 20000000200260002b5e500260000000200027d300026000119 3a0026000000000000000000014000207e00018000009e00000 003aaaaaaaaaaaaaaaa
...
000000000000000f3b04845f33c5dc7
"
From: Sergey S. Kostyliov ([email protected])
Subject: Re: 2.6.0-test2-mm3 and mysql
View: Complete Thread (22 articles)
Original Format
Newsgroups: linux.kernel
Date: 2003-08-27 09:00:19 PST
Quoted 11 lines · Matt Mackall
Quoted 10 lines · Sergey S. Kostyliov
Hello Andrew,
Quoted 7 lines · Andrew Morton
Quoted 5 lines · Shane Shrybman
One last thing, I have started seeing mysql database corruption recently. I am not sure it is a kernel problem. And I don't know the exact steps to reproduce it, but I think I started seeing it with -test2-mm2. I haven't ever seen db corruption in the 8-12 months I have being playing with mysql/php.
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
hm, that's a worry. No additional info available?
I also suffer from this problem (I'm speaking about heavy InnoDB corruption here), but with vanilla 2.6.0-test2. I can't blame MySQL/InnoDB because there are a lot of MySQL boxes around of me with
the
Quoted 8 lines
Quoted 2 lines
same (in fact the box wich failed is replication slave) or allmost the same database setup. All other boxes (2.4 kernel) works fine up to now.
All Linux kernels prior to 2.6.0-test2-mm3-1 would silently fail to complete
fsync()andmsync()operations if they encountered an I/O error, resulting in corruption. If a particular disk subsystem was producing these errors, the symptoms would likely be:
Quoted 8 lines
- no error reported
- no messages in logs
- independent of kernel version, etc.
- suddenly appear at some point in drive life
- works flawlessly on other machinesIf you can reproduce this corruption, please try running against mm3-1 and seeing if it reports problems (both to fsync and in logs).
I've just got another one InnoDB crash with 2.6.0-test4. As in previous case there was no messages in kernel log. You can find mysql error log here.
http://sysadminday.org.ru/linux-2.6.0-test4_InnoDB_crash
It's a development server, so this isn't a big problem. I do understand that this can easily be a hardware problem, but the kernel silence is really sad in such case. Memory is fine (at least according to memtest 3.0).
Any hints will be appreciated.
Signature · 3 lines
Best regards,
Sergey S. Kostyliov <[email protected]>
Public PGP key: http://sysadminday.org.ru/rathamahata.ascHi Heikki,
Quoted 3 lines · Heikki Tuuri
Sergey,
does it always crash when you start mysqld?
Yes It was always crashing until I deleted all InnoDB files and restored InnoDB tables from backup.
Quoted 10 lines
It is page number 0 in the InnoDB tablespace. That is, the header page of the whole tablespace!
The checksums in the page are ok. That shows the page was not corrupted in the Linux file system.
InnoDB is trying to do an index search, but that of course crashes, because the header page is not any index page.
Quoted 8 lines
The reason for the crash is probably that a page number in a pointer record in the father node of the B-tree has been reset to zero. The corruption has happened in the mysqld process memory, not in the file system of Linux. Otherwise, InnoDB would have complained about page checksum errors.
No one else has reported this error. I have now added a check to a future version of InnoDB which will catch this particular error earlier and will hex dump the father page.
Yes, now it seems for me that this particular crash in not related to linux kernel at all.
The funny thing I've managed to get another InnoDB crash on the same box
http://sysadminday.org.ru/linux-2.6.0-test4_InnoDB_crash-20030828
which in turn was posted to linux-kernel over a two hours ago. This time the cheksums are different :(
Quoted 7 lines
By the way, I noticed that a website http://www.linuxtestproject.org has made an extensive regression test suite for Linux. They have also successfully run big MySQL and DB2 stress tests on their computers, on 2.5.xx kernels. If there is something wrong with 2.5.xx or 2.6.0, it apparently does not concern all computers.
Quoted 8 lines
"
The Linux Test Project test suite, ltp-20030807, has been released. The latest version of the testsuite contains 2000+ tests for the Linux OS. "The general picture about InnoDB corruption is that reports have almost stopped after I advised people on the mailing list to upgrade to Linux-2.4.20 kernels.
In fact I'm also a happy InnoDB user. It runs fine on 6 of my production servers. Thanks for a nice work btw!
It has worked fine also on our development server until I upgraded it to 2.6.0-testX. I don't know. It might be just a broken hardware which is better stressed with 2.6 than with 2.4...
Quoted 5 lines
With apologies,
Heikki
Innobase Oy
Sergey,
----- Original Message -----
From: "Sergey S. Kostyliov" <[email protected]>
To: "Heikki Tuuri" <[email protected]>; <[email protected]>
Sent: Thursday, August 28, 2003 10:01 PM
Subject: Re: 2.6.0-test2-mm3 and mysql
Quoted 12 lines
Hi Heikki,
Quoted 3 lines · Heikki Tuuri
Sergey,
does it always crash when you start mysqld?
Yes It was always crashing until I deleted all InnoDB files and restored InnoDB tables from backup.
Quoted 2 lines
It is page number 0 in the InnoDB tablespace. That is, the header page
of
Quoted 3 lines
Quoted 3 lines
the whole tablespace!
The checksums in the page are ok. That shows the page was not corrupted
in
Quoted 3 lines
Quoted 3 lines
the Linux file system.
InnoDB is trying to do an index search, but that of course crashes,
because
Quoted 3 lines
Quoted 3 lines
the header page is not any index page.
The reason for the crash is probably that a page number in a pointer
record
Quoted 1 line
Quoted 1 line
in the father node of the B-tree has been reset to zero. The corruption
has
Quoted 4 lines
Quoted 4 lines
happened in the mysqld process memory, not in the file system of Linux. Otherwise, InnoDB would have complained about page checksum errors.
No one else has reported this error. I have now added a check to a
future
Quoted 1 line
Quoted 1 line
version of InnoDB which will catch this particular error earlier and
will
Quoted 3 lines
Quoted 1 line
hex dump the father page.
Yes, now it seems for me that this particular crash in not related to
linux
Quoted 5 lines
kernel at all.
The funny thing I've managed to get another InnoDB crash on the same box
http://sysadminday.org.ru/linux-2.6.0-test4_InnoDB_crash-20030828
which in turn was posted to linux-kernel over a two hours ago. This time the cheksums are different :(
ok, this time the corruption probably happened in the file cache, or the file system of Linux, or in the hardware.
It is not at all surprising that you encounter memory corruption and file corruption in the same computer. That is a common pattern if these problems appear at all.
Do you have a swap partition? I do not know Linux well enough, but in theory, file or disk corruption could cause also memory corruption if pages of the process memory get swapped to disk.
Quoted 15 lines
Quoted 6 lines
By the way, I noticed that a website http://www.linuxtestproject.org has made an extensive regression test suite for Linux. They have also successfully run big MySQL and DB2 stress tests on their computers, on 2.5.xx kernels. If there is something wrong with 2.5.xx or 2.6.0, it apparently does not concern all computers.
Quoted 8 lines
"
The Linux Test Project test suite, ltp-20030807, has been released. The latest version of the testsuite contains 2000+ tests for the Linux OS. "The general picture about InnoDB corruption is that reports have almost stopped after I advised people on the mailing list to upgrade to Linux-2.4.20 kernels.
Quoted 12 lines
In fact I'm also a happy InnoDB user. It runs fine on 6 of my production servers. Thanks for a nice work btw!
It has worked fine also on our development server until I upgraded it to 2.6.0-testX. I don't know. It might be just a broken hardware which is better stressed with 2.6 than with 2.4...
Quoted 5 lines
With apologies,
Heikki
Innobase Oy
Best regards,
Heikki
Innobase Oy
http://www.innodb.com
Quoted 70 lines · Heikki Tuuri
Sergey,
----- Original Message -----
From: "Sergey S. Kostyliov" <[email protected]>
To: "Heikki Tuuri" <[email protected]>; <[email protected]> Sent: Thursday, August 28, 2003 10:01 PM
Subject: Re: 2.6.0-test2-mm3 and mysqlQuoted 11 lines
Hi Heikki,
Quoted 3 lines · Heikki Tuuri
Sergey,
does it always crash when you start mysqld?
Yes It was always crashing until I deleted all InnoDB files and restored InnoDB tables from backup.
Quoted 1 line
It is page number 0 in the InnoDB tablespace. That is, the header page
of
Quoted 3 lines
Quoted 3 lines
the whole tablespace!
The checksums in the page are ok. That shows the page was not corrupted
in
Quoted 3 lines
Quoted 3 lines
the Linux file system.
InnoDB is trying to do an index search, but that of course crashes,
because
Quoted 3 lines
Quoted 3 lines
the header page is not any index page.
The reason for the crash is probably that a page number in a pointer
record
Quoted 1 line
Quoted 1 line
in the father node of the B-tree has been reset to zero. The corruption
has
Quoted 4 lines
Quoted 4 lines
happened in the mysqld process memory, not in the file system of Linux. Otherwise, InnoDB would have complained about page checksum errors.
No one else has reported this error. I have now added a check to a
future
Quoted 1 line
Quoted 1 line
version of InnoDB which will catch this particular error earlier and
will
Quoted 3 lines
Quoted 1 line
hex dump the father page.
Yes, now it seems for me that this particular crash in not related to
linux
Quoted 5 lines
kernel at all.
The funny thing I've managed to get another InnoDB crash on the same box
http://sysadminday.org.ru/linux-2.6.0-test4_InnoDB_crash-20030828
which in turn was posted to linux-kernel over a two hours ago. This time the cheksums are different :(ok, this time the corruption probably happened in the file cache, or the file system of Linux, or in the hardware.
Quoted 7 lines
It is not at all surprising that you encounter memory corruption and file corruption in the same computer. That is a common pattern if these problems appear at all.
Do you have a swap partition? I do not know Linux well enough, but in theory, file or disk corruption could cause also memory corruption if pages of the process memory get swapped to disk.
Yes, the swap is used on this box.
rathamahata@dev rathamahata $ free -k
total used free shared buffers cached
Mem: 1035248 860740 174508 0 32676 581852 -/+ buffers/cache: 246212 789036 Swap: 8969484 195136 8774348