2.4.5 data corruption
Folks, I believe I have a reproducible test case which corrupts data in 2.4.5.
We do nightly, weekly, and monthly backups by copying our entire /home partition on the company file server:
Filesystem Size Used Avail Use% Mounted on /dev/hda1 1.9G 1.7G 123M 93% / /dev/hda6 1.9G 437M 1.4G 23% /tmp /dev/sda1 37G 26G 11G 71% /home /dev/sdc1 37G 26G 11G 70% /weekly /dev/sdd1 37G 24G 13G 65% /monthly /dev/sdb1 37G 26G 11G 71% /nightly
The sd? drives are actually ide drives on a 3ware escalade controller. I have reason to believe the drives are good, before I installed them I scrubbed them with varying data patterns and verified that that I got back what I put there. All tested cleanly overnight.
I recently added an integrity check to our backups - the integrity checker writes out the path, the gzip adler32 checksum, the size, and the mtime of each file. Each time I do a backup, the backup scripts look for the integrity listing in the other partitions and compares all files with the same path, size, and modtime.
This morning I had a pile of errors after things having gone smoothly for the last few weeks. I suspected that I had screwed something up, looked over the backup scripts, simplified them down to a simple cpio, and tried again. Another pile of errors, different set of files.
In both cases, the newly created files were corrupted, the ones on the live /home partition as well as the /weekly & /monthly partitions all compared cleanly.
I rebooted into 2.2.19, tried again, no errors. I was running 2.4.5, no patches. I power cycled the machine between each reboot, went through the bios memory check, and also went through my own memory check; memory does not seem to be an issue.
I think I can reproduce this, it takes a reboot and about 2 hours. I made it happen twice with 2.4.5, the first try on 2.2.19 did not work.
The data corruption looks like *extra* bytes added at the beginning of files. I only looked at a few, if we go down the path of debugging this I'll save them all next time. The extra byte counts were small, in one case there was the letter "1" added to the start of the file, other than that it was identical. That's really weird, as a file system guy, I'd expect to see blocks of data not small chunks of data. Very strange.
One thing I haven't done is to rule out the 3ware controller. I tend to doubt it is the problem but who knows.
There were no kernel messages complaining about anything during the backup, so the kernel doesn't seem to know there is a problem.
So, does anyone recognize these symptoms? Does anyone care?
Quoted 2 lines · Larry McVoy
Folks, I believe I have a reproducible test case which corrupts data in 2.4.5.
Why don't you send the test case to the list? I would love to try it out and it would be a good addition to LTP.
Signature · 3 lines
Nate Straz [email protected] sgi, inc http://www.sgi.com/ Linux Test Project http://ltp.sf.net/
Quoted 9 lines · On Tuesday, June 12, 2001 01:17:49 PM -0700 Larry McVoy <[email protected]> wrote:
Folks, I believe I have a reproducible test case which corrupts data in 2.4.5.
We do nightly, weekly, and monthly backups by copying our entire /home partition on the company file server:
Filesystem Size Used Avail Use% Mounted on /dev/hda1 1.9G 1.7G 123M 93% / /dev/hda6 1.9G 437M 1.4G 23% /tmp
What flavor of IDE controller? Where is swap?
-chris
Quoted 2 lines
Folks, I believe I have a reproducible test case which corrupts data in 2.4.5.
2.4.5 has an out of date 3ware driver that is short
+ 1.02.00.005 - Allocate bounce buffers and custom queue depth for raid5 for
+ 6000 and 5000 series controllers. + Reduce polling mdelays causing problems on some systems. + Fix use_sg = 1 calculation bug. + Check for scsi_register returning NULL. + Add aen count to /proc/scsi/3w-xxxx. + Remove aen code unit masking in tw_aen_complete(). + 1.02.00.006 - Remove unit from printk in tw_scsi_eh_abort(), causing + possible oops. + Fix possible null pointer dereference in tw_scsi_queue() + if done function pointer was invalid.
+ 1.02.00.007 - Fix possible null pointer dereferences in tw_ioctl().
+ Remove check for invalid done function pointer from
+ tw_scsi_queue().
That might be a first thing to check
In article <[email protected]>,
Quoted 4 lines · Alan Cox
Quoted 2 lines
Folks, I believe I have a reproducible test case which corrupts data in 2.4.5.
2.4.5 has an out of date 3ware driver that is short
These days I observed massive FS curruption on vanilla 2.4.5, SCSI disk on Sym53c8-something controller (UW). I did not notice any problems since 2.4.5 was published, they seem to have surfaced immediately after I created a rather big file capturing video with broadcast2000 (video card is bt848). Filesystem is ext2.
Eugene
Quoted 3 lines
any problems since 2.4.5 was published, they seem to have surfaced immediately after I created a rather big file capturing video with broadcast2000 (video card is bt848). Filesystem is ext2.
Thats something I've seen reported elsehwere. The high bandwidth capture card stuff seems to show up problems. It could be drivers could be hardware. On my AMD 751 pre release board I see that problem but on the 751 production board I dont
In article <[email protected]>,
Quoted 8 lines · Alan Cox
Quoted 3 lines
any problems since 2.4.5 was published, they seem to have surfaced immediately after I created a rather big file capturing video with broadcast2000 (video card is bt848). Filesystem is ext2.
Thats something I've seen reported elsehwere. The high bandwidth capture card stuff seems to show up problems. It could be drivers could be hardware. On my AMD 751 pre release board I see that problem but on the 751 production board I dont
You must be right, today I created another big file with the same program but without doing caputre and the filesystem was intact. OTOH, Russell Leighton reports curruption when creating a file with dd...
Eugene
Quoted 6 lines · Eugene Crosser
In article <[email protected]>,
Quoted 4 lines · Alan Cox
Quoted 3 lines
any problems since 2.4.5 was published, they seem to have surfaced immediately after I created a rather big file capturing video with broadcast2000 (video card is bt848). Filesystem is ext2.
Quoted 8 lines
Quoted 4 lines
Thats something I've seen reported elsehwere. The high bandwidth capture card stuff seems to show up problems. It could be drivers could be hardware. On my AMD 751 pre release board I see that problem but on the 751 production board I dont
You must be right, today I created another big file with the same program but without doing caputre and the filesystem was intact. OTOH, Russell Leighton reports curruption when creating a file with dd...
For what it is worth, after having three failures in a row, now it isn't happening. My test case is/was my nightly backup. If it happens again, I'll save the corrupted data so we can do more digging. I'm kicking myself for not having done it the first time around.
Signature · 2 lines
--- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Nuther anecdote:
I was creating a big swapfile on ext2 (because 2.4.5 needs too much swap) with dd (SCSI disk on Sym53c8-something controller) and corrupted the partition THEN fsck would cause the kernel to panic. I thought I had some bad hw ... the box sits on my office floor waiting resurrection.
Quoted 13 lines · Eugene Crosser
In article <[email protected]>,
Quoted 4 lines · Alan Cox
Quoted 2 lines
Folks, I believe I have a reproducible test case which corrupts data in 2.4.5.
2.4.5 has an out of date 3ware driver that is short
These days I observed massive FS curruption on vanilla 2.4.5, SCSI disk on Sym53c8-something controller (UW). I did not notice any problems since 2.4.5 was published, they seem to have surfaced immediately after I created a rather big file capturing video with broadcast2000 (video card is bt848). Filesystem is ext2.
Quoted 6 lines
Eugene
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
--
---------------------------------------------------
Russell Leighton [email protected]
---------------------------------------------------
Review
Quoted 8 lines · Alan Cox
Quoted 2 lines
Folks, I believe I have a reproducible test case which corrupts data in 2.4.5.
2.4.5 has an out of date 3ware driver that is short
(quoted diff) +3
-0+ 1.02.00.007 - Fix possible null pointer dereferences in tw_ioctl(). + Remove check for invalid done function pointer from + tw_scsi_queue().
hehe, this one keeps the 3dmd from running here at all.
Quoted 1 line
That might be a first thing to check
Well, I do not understand how the driver is distributed. The actual 3ware stuff won't compile on 2.4.x, and the stuff in kernel is always different from 3ware releases.
I use two 8-port cards (8 disks each) and I see different but fatal problems on both systems.
Is anyone here using an actual firmware and raid-5 ? Does it work up to some level on 6800 ?
Anyway, a useful proc-interface would be really cool (like DAC); I guess that many people would love to get rid of the - sorry - fucking closed sourced and totally broken 3dmd which makes an extremly nice product totally useless (you can't trust it; not only because it's closed source, it simply doesn't work (except that it wastes memory, that works fine. tested.))
Signature · 4 lines
ciao -
Stefan
" destroy-your-data-by-3dmd-no-need-for-hammer-anymore CNAME www.3ware.com. "Quoted 3 lines
Well, I do not understand how the driver is distributed. The actual 3ware stuff won't compile on 2.4.x, and the stuff in kernel is always different from 3ware releases.
The stuff in the -ac tree is directly from 3ware
Quoted 4 lines · Stefan Traby
Quoted 3 lines · Alan Cox
Quoted 2 lines
Folks, I believe I have a reproducible test case which corrupts data in 2.4.5.
Quoted 8 lines
Review
Quoted 6 lines
2.4.5 has an out of date 3ware driver that is short
(quoted diff) +4
-0+ 1.02.00.007 - Fix possible null pointer dereferences in + tw_ioctl(). + Remove check for invalid done function pointer + from tw_scsi_queue().hehe, this one keeps the 3dmd from running here at all.
Saw that one here too.
[...]
Quoted 7 lines
(like DAC); I guess that many people would love to get rid of the - sorry - fucking closed sourced and totally broken 3dmd which makes an extremly nice product totally useless (you can't trust it; not only because it's closed source, it simply doesn't work (except that it wastes memory, that works fine. tested.))
3dmd does have a lot of problems, but i thought it was just me. I
only made it work once in a machine, and not very well. Last week i installed the latest version in another of my machines and after half an hour wrestling with it - trying to make it change passwords and ask for one, among other things - i gave up.
Quoted 4 lines
ciao -
Stefan
Pedro