Disappearing directories

Hello,

I think I might have stumbled across a bug in jffs2, that caused our JFFS2 NAND flash partitions to lose subdirectories now and then.

The embedded system in question was running on linux 2.6.38.2, but using the nandsim module, I was able to recreate the bug on workstations running linux 3.0.0 as well as linux 3.2.0.

The attached script should demonstrate the bug, however since the ways of the garbage collector are not exactly deterministic, it might not work every time. When the bug hits, the following lines can be found in dmesg:

Here is, what I think happens:

"mkdir tmpdir" creates inode 2 and links to it from inode 1 using the name tmpdir. This entry shares an eraseblock with the DELETE[12] files. "mkdir tmpdir/subdir" creates inode 6 and links to it from inode 2 using the name tmpdir. This entry shares an eraseblock with the KEEP[12] files. "mv tmpdir/subdir ." and "rmdir tmpdir" create three consecutive log entries:
- create new link to inode 6 from inode 1 using the name subdir
- delete entry named subdir from inode 2
- delete entry named tmpdir from inode 1
These these three steps share an eraseblock with DELETE[34]

Now after "rm DELETE*", the creation of inode 6 and the link from inode 2 to it share an eraseblock with lots of valid data, while all the other entries sit in very dirty eraseblocks. These eraseblocks are very likely to be garbage collected during the following "cp ../rnd DELETE" sequence.

If that happens, inode 2 as well as the link to it vanish. The deletion of the link from inode 2 to inode 6 also disappears. However the creation of a link from inode 2 to inode 6 still remains in flash, since it is "protected" by the heap of valid data.

This does not cause any trouble as long as the filesystem stays mounted. However if you unmount the partition at this point and then remount it, jffs2 can't get around the fact, that there are now two links to inode 6, even if one is from a directory, that itself doesn't exist anymore.

Best Regards,

i. A. Christoph Mammitzsch

Symeo GmbH
Professor-Messerschmitt-Str. 3
85579 Neubiberg / München
phone: +49 (89) 6607796-301

Symeo GmbH; Managing Directors: Dirk Brunnengräber, Christoph Rommel;
Registered Office: Munich; Commercial Register: Munich, HRB 157340

#!/bin/sh

# load needed modules
modprobe nandsim first_id_byte=0x2c second_id_byte=0xa1 third_id_byte=0x80 fourth_id_byte=0x15 parts=128
modprobe mtdblock

# generate some arbitrary data
dd if=/dev/urandom of=rnd bs=2048 count=64

# clean and mount simulated NAND partition
flash_erase /dev/mtd0 0 0
mkdir -p mnt_jffs2
mount -t jffs2 /dev/mtdblock0 mnt_jffs2

# set up a critical node structure
cd mnt_jffs2
mkdir tmpdir
cp ../rnd DELETE1
cp ../rnd DELETE2
cp ../rnd KEEP1
mkdir tmpdir/subdir
cp ../rnd KEEP2
cp ../rnd DELETE3
mv tmpdir/subdir .
rmdir tmpdir
cp ../rnd DELETE4
rm DELETE*
# and excercise the garbage collector
cp ../rnd DELETE1
cp ../rnd DELETE2
cp ../rnd DELETE3
cp ../rnd DELETE4
cp ../rnd DELETE5
cp ../rnd DELETE6
cp ../rnd DELETE7
cp ../rnd DELETE8
cp ../rnd DELETE9
cp ../rnd DELETE1
cp ../rnd DELETE2
cp ../rnd DELETE3
cp ../rnd DELETE4
cp ../rnd DELETE5
cp ../rnd DELETE6
cp ../rnd DELETE7
cp ../rnd DELETE8
cp ../rnd DELETE9
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
rm DELETE*
cd ..

# unmount and remount partition. Notice how ls barfs after remount, but not before.
ls -l mnt_jffs2
umount mnt_jffs2
mount -t jffs2 /dev/mtdblock0 mnt_jffs2
ls -l mnt_jffs2

# unmount again for next run
umount mnt_jffs2

Am 04.07.2014 19:00, schrieb Christoph Mammitzsch:

Update: I compiled the newest linux kernel (commit 812c5fa82bae9f377f49000d7636c19a8b61735c) and ran the test script on that kernel. The bug still showed 9 out of 10 times.

Regards,

Christoph Mammitzsch

Hello,

I am a little astonished at the amount of feedback I got for my bugreport. Apparently jffs2 losing entire directories is not as big a deal as I thought. Does that mean that JFFS2 is dead and I should use another flash filesystem instead, or did I simply post on the wrong mailing list?

Regards,

Christoph Mammitzsch

Is there any reason why you are not using UBIFS?

Thanks,
//richard

Am 03.08.2014 01:13, schrieb Richard Weinberger:

We introduced our first linux based product 2006, so UBIFS was not an option. Our second linux based product was designed in 2011. Back then UBIFS was still relatively new, and it seemed beneficial to keep differences between our two products minimal, so we kept using JFFS2.

Switching to UBIFS now means we would have to update and test quite a lot of scripts. And providing an update for existing stations would be even more problematic. Therefore I was hoping for a bugfix, so we could just deliver a new kernel image and be done with it.

Regards,

Christoph Mammitzsch

Am 05.08.2014 11:22, schrieb Christoph Mammitzsch:

I can understand your decisions.
TBH jffs2 is more of less abandoned, don't expect anyone to fix issues fast.

Is the issue you see a regression?
If you can find the first bad commit it might give someone a hint... git bisect will help you.

Thanks,
//richard