Fail to mount NILFS.

Hi,

I get error when mount nilfs2 partition.

mount.nilfs2 /dev/cciss/c0d1p1 /home/
mount.nilfs2: WARNING! - The NILFS on-disk format may change at any time. mount.nilfs2: WARNING! - Do not place critical data on a NILFS filesystem. mount.nilfs2: Error while mounting /dev/cciss/c0d1p1 on /home/: Invalid argument

dmesg output:
segctord starting. Construction interval = 5 seconds, CP frequency < 30 seconds
NILFS: corrupt root inode.

Used kernel: 2.6.30.4
nilfs-utils-2.0.12-3.i386

Is it possible to fix the partition?
Is it some tools exists, like fsck?

Thank you.
Yuri.

Hi,

Sorry for the inconvenience.
I'm afraid to say fsck has not been developed yet.

Are there any warnings or error messages of nilfs in syslog before the problem ?

I'll attach a patch to allow nilfs to mount plain checkpoints.

If you made the kernel from source, try the patch as follows:

 $ cd linux-2.6.30.4
 $ patch -p1 < path-to/0001-nilfs2-remove-snapshot-check-on-mount.patch
 $ make modules
 $ sudo rmmod nilfs2
 $ sudo insmod fs/nilfs2/nilfs2.ko
 $ sudo mount -t nilfs2 -r -o cp=xxx /dev/yyy /mnt

If you succeed to mount a valid past checkpoint, you will be able to do backup with rsync or so from the checkpoint.

Regards,
Ryusuke Konishi

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

Hi,

Thank you for patch.
I have try to mount as you propose, but receive following errors:
/sbin/mount.nilfs2 -r -o cp=30778 /dev/cciss/c0d1p1 /mnt/

NILFS: latest checkpoint=30778
NILFS: The specified checkpoint is not a snapshot (checkpoint number=30778).
NILFS: corrupt root inode.

If I try to mount older snapshot /sbin/mount.nilfs2 -r -o cp=27252 /dev/cciss/c0d1p1 /mnt/

then

NILFS: latest checkpoint=30778
NILFS: corrupt root inode.

If try to convert 30778 to snapshot I get
chcp: /dev/cciss/c0d1p1: cannot open NILFS

Thank you.

Yuri.

Hi,

Ignore this message. The patch actually changed nilfs to accept all checkpoints, but I left this error message.

This _is_ the problem. This shows the inode of root directory is broken or empty. I think you cannot mount generations which you see this message.

The first message is just information I added in the patch. But, uum, the root inode of checkpoint 27252 seems to be broken, too.

Can you go back to much older checkpoints ?

Yeah, chcp is not possible if there is no rw-mount. You don't have to do chcp because my workaround patch makes existent checkpoints mountable.

BTW, your nilfs-utils seems a bit old. I believe there was no problem causing filesystem corruption in nilfs-utils-2.0.12. But, I recommend you to use nilfs-utils-2.0.14 if available.

Regards,
Ryusuke Konishi

Hi,

I try to use older snapshot, that was successfully used before:

Aug 16 07:10:32 filer filer: Running: /sbin/mount.nilfs2 -r -o cp=5063 
/dev/cciss/c0d1p1 /snaps/2009-08-16_07_10_07
Aug 16 07:10:32 filer filer: Status: [OK]  Ouput:

But currently I get the same error.

The stored data is not critical for me so, as I have full backups, but I prefer to check possibility of the fs recovery, if possible.

I have update utils to 2.0.14 and kernel to 2.6.30.5 + your patch. Is it possible to add some printk to get additional information?

Thank you.
Yuri.

Hmm, this seems to be caused by corruption of DAT file. Maybe garbage collection failed to update the DAT file due to some bug.

Thanks. I will send you an additional patch for analysis later.

Thanks,
Ryusuke Konishi

Yuri, I will attach a patch to log the latest segment.

If you apply this, you will see messages like:

segctord starting. Construction interval = 5 seconds, CP frequency < 30 seconds
NILFS: looking segment (seg_start=1382400, seg_end=1384447, segnum=675, seg_seq=675)
NILFS: found super root: segnum=675, seq=675, pseg_start=1382558, pseg_offset=174
NILFS: latest checkpoint=26461

Then, you can dump summary of the segment with dumpseg command.

 # dumpseg 675
           ~~~ specify the segnum reported in the log.
 # dumpseg 674
 # dumpseg 673
 ..

Could you send me the dump log around the segment?

If you find an entry like:

finfo
ino = 6, cno = xxxxx, nblocks = x, ndatblk = x
vblocknr = yyyyyy, blkoff = 2, blocknr = xxxxxx
~~~~~~~ ~~~~~~~~~~
ino = 6 indicates The root inode is included in this block.
inode metadata file

This writeback is suspicious.

Thanks,
Ryusuke Konishi

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

Hi,

Attached file with dumpseg output.
It's look, that root inode in segnum=10938.

Thank you.
Yuri.

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

The segment in the attached log did not include the root inode. ( The entry with ino = 6, did not include the block with blkoff = 2 )

What will happen if you run the attached script?

 $ chmod +x find-root.sh
 $ sudo ./find-root.sh <device> 10938 | tee -a segment-with-root-inode.log

Thanks,
Ryusuke Konishi

#!/bin/bash

cmd=basename $0

function die() { echo "$1" 1>&2; exit 1; }

test -n "$1" -a -n "$2" -a -b "$1" || die "usage: $cmd <device> <segnum>"

dev=$1

for (( i=$2; i >= 0; i-- )); do
if dumpseg $dev $i | sed -ne '/ino = 6/,/finfo/{p}' | grep -q "blkoff = 2,"; then
echo "$cmd: root inode found in segnum=$i"
dumpseg $dev $i
exit 0
fi
done

die "$cmd: root inode not found"

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

Done.

Log file attached.

Thanks.
Yuri.

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

Ah, sorry. Pattern matching of the script was incomplete.

Could you retry with the revised script?

If the new script find the blocks storing the root inode, it will also dump them to files like:

ifile-2-seg649-1329203
ifile-2-seg649-1329259
ifile-2-seg649-1329315
ifile-2-seg649-1329372
ifile-2-seg649-1329428
ifile-2-seg649-1329455

Then, we can see if the root inode is actually broken or not.

Thanks,
Ryuske Konishi

#!/bin/bash

cmd=basename $0
blocksize=4k

function die() { echo "$1" 1>&2; exit 1; }

test -n "$1" -a -n "$2" -a -b "$1" || die "usage: $cmd <device> <segnum>"

dev=$1

for (( i=$2; i >= 0; i-- )); do
hit=dumpseg $dev $i | sed -ne '/ino = 6,/,/finfo/{p}' | sed -ne "/blkoff = 2,/{s/^.* blocknr = \([0-9]\+\).*$/\1/p}" if [ -n "$hit" ]; then
echo "$cmd: root inode found in segnum=$i"
dumpseg $dev $i

for b in $hit; do
echo "$cmd: dump root inode block: blocknr=$b"
dd if=$dev of=ifile-2-seg${i}-${b} bs=$blocksize skip=$b count=1
done
exit 0
fi
done

die "$cmd: root inode not found"

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

Hi,

There is output from script.
sudo ./find_root.sh /dev/cciss/c0d1p1 10938 | tee -a segment-with-root- inode.log
find_root.sh: root inode not found

Thanks.
Yuri.

Hi,

Thank you for following this analysis.

It reached the beginning of the partition.

What will happen if you start the script from the last segment number?

The last segment number is calculated from the block count which appears in /proc/partitions:

 $ cat /proc/partitions
major minor  #blocks  name

   8       16   71687325 sdb
   8       17   35840983 sdb1
   8       18   35841015 sdb2

 $ expr 35840983 / 4 / 2048 - 1
 4374
 $ sudo ./find-root.sh /dev/sdb1 4374

And, the attached patch will output some information about how disk block numbers are converted with DAT file.

I think this will give another hint on which metadata broke in nilfs.

Could you apply it and try mount again?

Thanks in advance,
Ryusuke Konishi

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users