FS corruption on 2.4.0-ac8

Hi all,

I was running 2.4.0test10pre5 happily for months and wanted to see how things stand in the 'latest stuff'. Here's what i found:

I compiled 2.4.0-ac8 with nearly the same .config as test10pre5 (with latest gcc on rh7). Then i booted it and used X for some normal browsing and mp3s. Performance was poor, responsivness also, even the mouse stopped responding for a couple of seconds at a time, a lot of disk trashing & so on. I deceided to boot test10 back, and there was a nasty
suprise: fsck found filesystem with errors, and LOTS of them ... i had to hold down 'y' for almost 5 minutes ... :)

Then i examined the logs for what would be the cause for this ... and here's what 2.4.0-ac8 left in the logs:

Jan 14 16:26:47 open kernel: ee_blocks: Freeing blocks not in datazone -
block = 979727457, count = 1
Jan 14 16:26:47 open kernel: EXT2-fs error (device md(9,1)):
ext2_free_blocks: Freeing blocks not in datazone - block = 1769096736,
count = 1
Jan 14 16:26:47 open kernel: EXT2-fs error (device md(9,1)):
ext2_free_blocks: Freeing blocks not in datazone - block = 842080300,
count = 1
Jan 14 16:26:47 open kernel: EXT2-fs error (device md(9,1)):
ext2_free_blocks: Freeing blocks not in datazone - block = 1851869728,
count = 1
Jan 14 16:26:47 open kernel: EXT2-fs error (device md(9,1)):

ext2_free_blocks: Freeing blocks not in datazone - block = 808464928, count = 1
...
and so on for about 150 such lines in 3 seconds.

There is something not that usual about my setup: i run raid1 /boot and raid5 root with one disk disconnected (its simply too loud...), so the array is in degraded mode all the time. Other hardware is more or less standard, p200 classic, 430vx board, adaptec2940u, 64mb ram.

Is this a known problem? If it's not, please advise me on how to provide more usefull informations.

Jure Pecar
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Neil,

This is the second report of corruption with RAID5.

Do you know if any of your recent changes can be the reason?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi all,

I have a 100% reproducable bug in all of the 2.4.0 kernels including the
latest stable one. The issue is that if I compile the kernel to support 4GB RAM (I have 1 GB) and then try to access a samba mount I get an oops. This ALWAYS happens. Usually after this the system is frozen (although the magic SYSREQ still works). If the system isn't frozen then any commands that access the disk will freeze. Fortunately GPM worked and I was able to paste the oops to a file via telnet.

Attached is my oops.txt and the result sent through ksymoops. The results
don't look particularly useful to me so perhaps I'm doing something wrong. PLEASE tell me if I should parse this differently. Likewise, if there is anything else I can do to help debug this, please tell me.

--Rainer

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

It seems you forgot to attach oops.txt.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I knew that, I was just testing you all. ;-)

\e hides his head in shame

Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
f889e044
*pde = 00000000
Oops: 0002
CPU:    1
EIP:    0010:[<f889e044>]
EFLAGS: 00010246
eax: 00000000   ebx: d5762800   ecx: 00000400   edx: c19665fc
esi: d55be120   edi: 00000000   ebp: d5764260   esp: d5505f1c
ds: 0018   es: 0018   ss: 0018
Process ls (pid: 865, stackpage=d5505000)
Stack: d5762800 d55be120 d5764260 d5764260 d55be120 00000000 f889d966 d55be120
       d5762800 d5504000 d5764260 fffffffe fffffffb d5762800 d5764260 d55be120
       00000000 d5764260 bffffa40 00000006 c0140c10 d5764260 d5505fb0 c0140e7c
Call Trace: [<f889d966>] [<c0140c10>] [<c0140e7c>] [<c0140f9e>] [<c0140e7c>] [<c0108f4b>]

Code: f3 ab e9 8b 00 00 00 90 8d 74 26 00 8b 44 24 14 c7 00 00 00

Segmentation fault

Trace; f889d966 <END_OF_CODE+385bf756/????>
Trace; c0140c10 <vfs_readdir+90/ec>
Trace; c0140e7c <filldir+0/d8>
Trace; c0140f9e <sys_getdents+4a/98>
Trace; c0140e7c <filldir+0/d8>

It seems the oops is happening in a module's function.

You have to make ksymoops parse the oops output against a System.map which has all modules symbols. Load each module by hand with the insmod -m option ("insmod -m module.o") and _append_ the outputs to System.map.

After that you can run ksymoops against this new System.map.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Ok, now were making progress. I did as you said and have attached (really!) the new parsed output. Now we have some useful information (I hope). I still got lots of warnings on symbols (which I have edited out of the parsed file for the sake of briefness). What's the next step?

--Rainer

Wait for someone who has a clue about smbfs to find out the problem.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

No need, just create directory /var/log/ksymoops. insmod and rmmod will automatically save the list of modules and the symbol table on every module load or unload, neatly timestamped. When you get an oops, find the entries just before the oops and point ksymoops at those.

ksymoops -m /lib/modules/2.4.0/System.map \
	 -k /var/log/ksymoops/20010116093850.ksyms \
	 -l /var/log/ksymoops/20010116093850.modules < oops.txt

man insmod, section KSYMOOPS ASSISTANCE. Much easier than trying to reproduce the environment by hand.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I'll have a look tonight or so. It works for you on non-bigmem?

smb_rename suggests mv, but the process is ls ... er? What commands where you running on smbfs when it crashed?

Could this be a symbol mismatch? Keith Owens suggested a less manual way to get module symbol output. Do you get the same results using that?

/Urban

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Here is a newly parsed oops, this time using the /var/log/ksymoops method mentioned by Keith Owens. Does this look better?

--Rainer

Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
c01239a4
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c01239a4>]
EFLAGS: 00010202
eax: 00001001   ebx: 00000000   ecx: c0256730   edx: 0003f435
esi: c20cde24   edi: 00000000   ebp: 00000001   esp: ee5e3e30
ds: 0018   es: 0018   ss: 0018
Process ls (pid: 449, stackpage=ee5e3000)
Stack: c20cde24 ee5e3e64 f7e00004 00000001 c01262f5 c20cde24 00000000 00000001
       f7e00004 c1000010 fe2f0014 00000018 fe2f0000 c20cde24 f88982f8 00000000
       00000001 00000070 ee5e3ee8 f889e180 ee61a000 ee6ede9c 00000010 f8896e69
Call Trace: [<c01262f5>] [<fe2f0014>] [<fe2f0000>] [<f88982f8>] [<f889e180>] [<f8896e69>] [<f8896eaa>]
       [<fe2f0000>] [<fe2f0000>] [<f889e048>] [<f889e03c>] [<f8896f40>] [<fe2f0000>] [<f88983b0>] [<fe2f0000>]
       [<fe2f0000>] [<f889798b>] [<fe2f0000>] [<c0140c10>] [<c0140e7c>] [<c0140f9e>] [<c0140e7c>] [<c0108f4b>]

Code: 8b 07 ff 47 18 89 70 04 89 06 89 7e 04 89 37 89 7e 08 8b 44

Segmentation fault

Yes, and it sort of matches the other oops someone sent. Thanks.

I have a changed version now, based on the ncpfs directory cahce code. But it doesn't work at all right now. (and that would be the "based on" bit, the copy and paste bits haven't crashed yet :)

Assuming that all meetings have an end, and sometimes they don't seem to have one, there may be something for you to try tomorrow.

/Urban

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Jure, you write:

These block numbers read as "ate: Fri, 12 Jan 200" (i.e. part of an email header) if you convert to 32-bit hex, then ascii. There was another report of corruption like this under 2.4.0 as well.

Actually, I thought there were problems in the test11? through test-13 kernels of this sort. I'm not sure exactly when it started, but a search of l-k should locate it. Depending on when last you fsck'd your filesystem, you may have already had some of these disk problems. However, there was another report recently on 2.4.0 that looked the same, and that user had just e2fsck'd his filesystem before booting 2.4.0 for the first time after running only 2.2, so it looks like a 2.4.0 bug.

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/