jffs2 Oops on mount

Hi,

I'm experiencind following Oops while mounting jffs2 NAND partiton:

Checked all inodes but still 0x13364c bytes of unchecked space?

Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = c0004000
[00000000] *pgd=00000000
Internal error: Oops: 817 [#1]
Modules linked in: nsswitch-04 nsquade1-03 nsblank-02 nsblank-01 nscpu-00 nsdetect nssysutil mmc_block mmc_core
CPU: 0
PC is at jffs2_garbage_collect_pass+0x1e8/0x53c
LR is at 0x1
pc : [<c00d0e88>]    lr : [<00000001>]    Not tainted

sp : c3e25ef0 ip : 60000093 fp : c3e25f58
r10: c3e380e8 r9 : 00000000 r8 : c3e38000
r7 : 00000000 r6 : c3e38000 r5 : c3e3802c r4 : c3ec98a8 r3 : c01d7940 r2 : 00000000 r1 : 00000000 r0 : 00000046
Flags: nZCv IRQs on FIQs on Mode SVC_32 Segment user
Control: 317F Table: 1342C000 DAC: 00000015 Process jffs2_gcd_mtd5 (pid: 138, stack limit = 0xc3e24194)

Stack: (0xc3e25ef0 to 0xc3e26000)

5ee0: c3e25f0c c3e25f00 c00428d0 c004281c
5f00: c3e25f38 c3e25f10 c0042f7c c00428c4 20000013 c3db8040 c3db8234 c3e25f5c 5f20: 00000000 00000000 c3e25f44 c3e25f48 20000013 c3e24000 c3e38000 00000000 5f40: 00000000 00000000 00000000 c3e25ff4 c3e25f5c c00d3d80 c00d0cb0 00000001 5f60: 00000000 00000080 00000000 00000000 00000000 c34e1f4c c34e1f40 c003aefc 5f80: c003aa70 40172c60 c3e25fa4 c3e25f98 c003af2c c003ae90 00000000 00000000 5fa0: 00000000 c3e25fb0 c001dd84 c0034758 00000000 c3e38000 c00d3c9c c003aa60 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5fe0: 00000000 00000000 00000000 c3e25ff8 c003aa60 c00d3cac 00000000 00000000

Backtrace: 
[<c00d0ca0>] (jffs2_garbage_collect_pass+0x0/0x53c) from [<c00d3d80>] (jffs2_garbage_collect_thread+0xe4/0x120)
[<c00d3c9c>] (jffs2_garbage_collect_thread+0x0/0x120) from [<c003aa60>] (do_exit+0x0/0x3d0)
 r6 = 00000000  r5 = 00000000  r4 = 00000000 
Code: e59f0348 ebfd9ea4 eaffffec ebfd9ea2 (e5877000) 
 <0>Kernel panic - not syncing: Fatal exception

Kernel version is 2.6.15 with CVS jffs2 from 10. Mar 2006 and omap2 patch applied (http://www.muru.com/linux/omap/patches/patch-2.6.15-omap2.bz2).

You can find jffs2 image and kernel objdump here:
ftp://ftp.linux-mips.org/pub/linux/mips/people/ladis/jffs2_oops/

I'm ready to do any testing you ask for.

Thanks a lot,
ladis

Well, this is harmless (caused by BUG() defined as (*(int *)0 = 0))

With more debugs enabled it ends like this:
Skipping ino #228 already checked
Skipping ino #229 already checked
Skipping ino #270 already checked
Skipping check of ino #273 with nlink zero
[lots of skipping check of ino #xxx with nlink zero] Skipping check of ino #800 with nlink zero
Skipping check of ino #801 with nlink zero
Skipping ino #802 already checked
Skipping check of ino #803 with nlink zero
Checked all inodes but still 0x13364c bytes of unchecked space?

Where checked_ino is 804 and highest_ino 803. ino 802 is already checked because userspace requested read meanwhile.

Ideas?

Thanks,
ladis

jffs2_scan_inode_node adds also inodes with zero nlink to unchecked_space, but jffs2_garbage_collect_pass skips them. That probably leads to above problem. After applying following patch BUG() is no longer triggered.

Index: fs/jffs2/gc.c
===================================================================
RCS file: /home/cvs/mtd/fs/jffs2/gc.c,v
retrieving revision 1.159

diff -u -r1.159 gc.c
@@ -211,12 +211,6 @@
 			continue;
 		}
 
-		if (!ic->nlink) {
-			D1(printk(KERN_DEBUG "Skipping check of ino #%d with nlink zero\n",
-				  ic->ino));
-			spin_unlock(&c->inocache_lock);
-			continue;
-		}
 		switch(ic->state) {
 		case INO_STATE_CHECKEDABSENT:
 		case INO_STATE_PRESENT:

I'm not sure what is correct solution. With this change there are lots of JFFS2 warning: (138) jffs2_get_inode_nodes: Eep. No valid nodes for ino #465. JFFS2 warning: (138) jffs2_do_read_inode_internal: no data nodes found for ino #465 Returned error for crccheck of ino #465. Expect badness...
warnings. Jffs2 summary feature is disabled. I'd be very gratefull for ideas.

ladis

(Resend in hope it will not be eaten by spam filter)

jffs2_scan_inode_node adds also inodes with zero nlink to unchecked_space, but jffs2_garbage_collect_pass skips them. That probably leads to above problem. After applying following patch BUG() is no longer triggered.

Index: fs/jffs2/gc.c
===================================================================
RCS file: /home/cvs/mtd/fs/jffs2/gc.c,v
retrieving revision 1.159

diff -u -r1.159 gc.c
@@ -211,12 +211,6 @@
 			continue;
 		}
 
-		if (!ic->nlink) {
-			D1(printk(KERN_DEBUG "Skipping check of ino #%d with nlink zero\n",
-				  ic->ino));
-			spin_unlock(&c->inocache_lock);
-			continue;
-		}
 		switch(ic->state) {
 		case INO_STATE_CHECKEDABSENT:
 		case INO_STATE_PRESENT:

I'm not sure what is correct solution. With this change there are lots of JFFS2 warning: (138) jffs2_get_inode_nodes: Eep. No valid nodes for ino #465. JFFS2 warning: (138) jffs2_do_read_inode_internal: no data nodes found for ino #465 Returned error for crccheck of ino #465. Expect badness...
warnings. Jffs2 summary feature is disabled. I'd be very gratefull for ideas.

ladis

<snip>

Actually, there is a jffs2_build_filesystem() function which walks all inodes with nlink == 0 and pretends to dispense with them. I believe that function is the right place to fix. I ganced at it, and it appeared to be thet it calls jffs2_mark_node_obsolete() for all nodes of this inode. jffs2_mark_node_obsolete() is a huge and fearsom monster-function which may do something wrong.

Best Regards,
Artem B. Bityutskiy,
St.-Petersburg, Russia.

Problem appeared to be GC thread waiting for inode being read. On next run this inode was incorrectly skipped before jffs2_do_read_inode_internal marked it obsolete. Bug was fixed by decrementing c->checked_ino before jffs2_garbage_collect_pass returns (for INO_STATE_READING) so on next GC run it is checked again.

Many thanks to David Woodhouse and Artem B. Bityuckiy for help with debugging.

Best regards,
ladis