ubifs panic with 2.6.39 stable - followup

[sorry for the duplicate post; I tried to post a followup with just
the url to the pastebin log but for some reason it got stuck waiting for the moderator]

Running and older 2.6.31 kernel with UBIFS I see a panic which appears to be during recovery of a bad block. The root fs (which is UBIFS) won't mount and the kernel panics.

I upgraded the kernel 2.6.39-stable, hoping that would fix the problem, as I noticed a lot of recovery fixes had gone in. It still panics; it appears the recovery fails.

I rebooted with "ignore_loglevel" and the output is here:

http://pastebin.com/ETJjP4uw

The board is essentially an Olimex SAM9-L9260, with Samsung NAND.

I'm curious if this looks familiar and if it might be fixed post 2.5.39

thanks for any insight.

-brad

UBIFS: recovery needed
UBI error: ubi_io_read: error -74 (ECC error) while reading 126976 bytes from PEB 2970:4096, read 126976 bytes UBIFS error (pid 1): ubifs_check_node: bad CRC: calculated 0xf510fb95, read 0x4f0a3196
UBIFS error (pid 1): ubifs_check_node: bad node at LEB 3589:45128

        magic          0x6101831
        crc            0x4f0a3196
        node_type      0 (inode node)
        group_type     2 (last of node group)
        sqnum          69874250
        len            160
        key            (23417, inode)
        creat_sqnum    230076
        size           608
        nlink          2
        atime          1306744746.0
        mtime          1316662493.0
        ctime          1316662493.0
        uid            0
        gid            0
        mode           16832
        flags          0x1
        xattr_cnt      2147483648
        xattr_size     0
        xattr_names    0
        compr_type     0x0
        data len       0

...
UBIFS error (pid 1): ubifs_recover_leb: bad node
UBIFS error (pid 1): ubifs_scanned_corruption: corruption at LEB 3589:45128 UBIFS error (pid 1): ubifs_scanned_corruption: first 8192 bytes from LEB 3589:45128
...
UBIFS error (pid 1): ubifs_recover_leb: LEB 3589 scanning failed

Would you please also enable scanning, mount, and recovery debugging messages and provide the log?

http://www.linux-mtd.infradead.org/faq/ubifs.html#L_how_debug

But in newer kernels this has been turned into dynamic debug, see Documentation/dynamic-debug-howto.txt

(Gosh, I have to update the web site!)

In older kernels you can do this via the configuration menu, in newer kernel this has been turned into Documentation/dynamic-debug-howto.txt

To use them at boot time, use ddebug_query=...

You need to use 3 format string to show "UBIFS DBG mnt" and the like.

I think, but did not verify, you need to add kernel boot 3 parameters:

ddebug_query="format UBIFS DBG rcvry" ddebug_query="format UBIFS DBG mnt" ddebug_query="format UBIFS DBG scan"

Best Regards,
Artem Bityutskiy

So there is a corrupted inode node, and UBIFS believes it has been corrupted not because of a power cut. I do not know why it is corrupted, but if you use MLC then this may be related to the paired pages problem.

Anyway, there is another issue I see from the dump. Even if you somehow make the node good again, UBIFS will still fail saying something like "corrupt empty space". Look at line 292 in your pastebin:

ffffffff ffffffff ffffffff ffffffff ffffffff ffdfffff ffffffff ffffffff

See that little "d"? It means that the empty space has a bit-flip. The question is why? Unstable bit I guess? Does your NAND driver / HW provides ECC protection for empty pages?

Anyway, currently UBIFS cannot handle these situation. Someone needs to do this - I can assist by reviewing and advising.

Best Regards,
Artem Bityutskiy