nilfs2 fails to mount after io/error
Subsystems (2)
Hi,
I just had an usb-harddrive (with nilfs2 and ext3 mounted) throw some IO-Errors (Most likely due to a crappy usb controller) and now nilfs2 refuses to mount the partition:
NILFS warning: Sequence number mismatch
NILFS: error searching super root.
(ext3 still works fine)
Any idea?
Regards,
Adrian
Signature · 3 lines
RFC 1925:
(11) Every old idea will be proposed again with a different name and
a different presentation, regardless of whether it works.A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 7 lines · Adrian Ulrich
I just had an usb-harddrive (with nilfs2 and ext3 mounted) throw some IO-Errors (Most likely due to a crappy usb controller) and now nilfs2 refuses to mount the partition:
NILFS warning: Sequence number mismatch
NILFS: error searching super root.(ext3 still works fine)
Well you could try to mount on an older snapshot or create a snapshot first?? i dont have a running nilfs2 around here so i can't try out the advise :)
With regards,
Reinoud
A message that is not in the archive (sent off-list or not yet received): 20081127171006.GA3654-5cYspOl2ggRz6xQTk39kMVfVdRo2wo/[email protected]
Quoted 3 lines
Well you could try to mount on an older snapshot or create a snapshot first?? i dont have a running nilfs2 around here so i can't try out the advise :)
I do not have any Snapshots (only CPs) and chcp requires a mounted filesystem :-/
Hi,
Quoted 11 lines · Adrian Ulrich
Hi,
I just had an usb-harddrive (with nilfs2 and ext3 mounted) throw some IO-Errors (Most likely due to a crappy usb controller) and now nilfs2 refuses to mount the partition:
NILFS warning: Sequence number mismatch
NILFS: error searching super root.(ext3 still works fine)
Any idea?
Hmm, this error seems to be caused by some kind of bug in NILFS2 write routines. I'll verify their error handlings.
In the meantime, please try the contained patch as follows:
$ tar jxf nilfs-2.0.5.tar.bz2 $ cd nilfs-2.0.5 $ patch -p1 < this-mail.patch $ make $ sudo make install
It disables the check of sequence numbers and would allow you to mount the partition.
With regards,
Ryusuke Konishi
--
fs/recovery.c +2 -0
@@ -240,10 +240,12 @@ load_segment_summary(struct nilfs_sb_info *sbi, sector_t pseg_start, goto failed; }store_segsum_info (ssi, sum, sbi->s_super->s_blocksize); +#if 0 if (seg_seq != ssi->seg_seq) { ret =NILFS_SEG_FAIL_SEQ ; goto failed; } +#endif if (full_check) { offset =sizeof (sum->ss_datasum); check_bytes =
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 1 line
In the meantime, please try the contained patch as follows:
mount.nilfs2 now 'hangs' while mounting the filesystem:
- CPU-Usage ~ 10%
- 100% io-wait
Do you think that mount.nilfs2 will ever finish? (It's running since ~15 minutes)
Regards,
Adrian
A message that is not in the archive (sent off-list or not yet received): 20081128153732.be4f5cf8.adrian-4ZM2p5qjiQGewZBzVTKGGg@public.gmane.org
Hi,
Quoted 11 lines · Adrian Ulrich
Hi,
Quoted 1 line
In the meantime, please try the contained patch as follows:
mount.nilfs2 now 'hangs' while mounting the filesystem:
- CPU-Usage ~ 10%
- 100% io-waitDo you think that mount.nilfs2 will ever finish? (It's running since ~15 minutes)
Oh, that seems unusual. Please stop applying the patch.
The check of sequence number might be really correct.
In that case, recovery logic is suspicious. Or there was actually a critical violation of write order on the device.
For nilfs2, if a superblock was written ahead of the segment pointed by the superblock, this can happen.
Anyhow, I'll review the source code some more.
With regards,
Ryusuke Konishi
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 5 lines
For nilfs2, if a superblock was written ahead of the segment pointed by the superblock, this can happen.
Anyhow, I'll review the source code some more.
Ok. Let me know if you need something from me.
(metadata dump, etc..)
Regards,
Adrian
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Adrian Ulrich
Hi,
Quoted 5 lines
For nilfs2, if a superblock was written ahead of the segment pointed by the superblock, this can happen.
Anyhow, I'll review the source code some more.
Ok. Let me know if you need something from me.
(metadata dump, etc..)
Thanks,
Could you get debug log with a DEBUG option?
By commenting out the following line in nilfs2/fs/Makefile, you can make the nilfs2 module for debugging purpose.
# CONFIG_NILFS_DEBUG=y
Compilation and installation are as usual.
Then write the following option to a procfs file 'nilfs2/debug_option' before mounting the partition.
# modprobe nilfs2 # echo "-vvv recovery" > /proc/fs/nilfs2/debug_option # mount -t nilfs2 -r /dev/xxx /test ...
This will log trace messages of recovery function.
Thanks in advance,
Ryusuke Konishi
A message that is not in the archive (sent off-list or not yet received): [email protected]
Debug output:
NILFS nilfs_fill_super: start(silent=0)
NILFS: INFO: recovery required for readonly filesystem.
NILFS: write access will be enabled during recovery.
NILFS(recovery) nilfs_search_super_root: looking segment (seg_start=174272512, seg_end=174274559, segnum=85094, seg_seq=181656) NILFS(recovery) load_segment_summary: checking segment (pseg_start=174272512, full_check=1) NILFS(recovery) load_segment_summary: done (ret=4)
NILFS(recovery) nilfs_search_super_root: strayed: scan_newer=0, ret=4
NILFS warning: Sequence number mismatch
NILFS: error searching super root.
NILFS nilfs_fill_super: aborted
NILFS put_nilfs: the_nilfs on bdev sdb5 was freed
Do you also need the output with your first patch applied?
Regards,
Adrian
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 13 lines · Adrian Ulrich
Debug output:
NILFS
nilfs_fill_super: start(silent=0)
NILFS: INFO: recovery required for readonly filesystem.
NILFS: write access will be enabled during recovery.
NILFS(recovery)nilfs_search_super_root: looking segment (seg_start=174272512, seg_end=174274559, segnum=85094, seg_seq=181656) NILFS(recovery)load_segment_summary: checking segment (pseg_start=174272512, full_check=1) NILFS(recovery)load_segment_summary: done (ret=4)
NILFS(recovery)nilfs_search_super_root: strayed: scan_newer=0, ret=4
NILFS warning: Sequence number mismatch
NILFS: error searching super root.
NILFSnilfs_fill_super: aborted
NILFSput_nilfs:the_nilfson bdev sdb5 was freed
According to this log, there is actually a sequence number mismatch between superblock and the segment pointed by the superblock. The recovery function seems to work normally.
Quoted 2 lines
Do you also need the output with your first patch applied?
Instead of the previous patch, could you try the following patch?
It forces to mount a filesystem from the segment pointed by the superblock if there is no other failure.
If it succeeds, copy filesystem contents (just to be safe) and unmount it. Then reinstall the original module.
If successful, it may be able to re-mount as if nothing had happened.
Regards,
Ryusuke Konishi
--
fs/recovery.c +4 -0
@@ -240,10 +240,12 @@ load_segment_summary(struct nilfs_sb_info *sbi, sector_t pseg_start, goto failed; }store_segsum_info (ssi, sum, sbi->s_super->s_blocksize); +#if 0 if (seg_seq != ssi->seg_seq) { ret =NILFS_SEG_FAIL_SEQ ; goto failed; } +#endif if (full_check) { offset =sizeof (sum->ss_datasum); check_bytes =
@@ -960,6 +962,8 @@ int nilfs_search_super_root(struct the_nilfs *nilfs, struct nilfs_sb_info *sbi, nilfs->ns_cno = cno;/* nilfs->ns_cno = ri->ri_cno + 1 */ nilfs->ns_ctime = ssi.ctime; nilfs->ns_nextnum = nextnum; + ri->ri_need_recovery =NILFS_RECOVERY_SR_UPDATED ; + goto super_root_found;if (scan_newer) ri->ri_need_recovery =NILFS_RECOVERY_SR_UPDATED ;
A message that is not in the archive (sent off-list or not yet received): [email protected]
Still fails to mount :
# insmod fs/nilfs2.ko # echo "-vvv recovery" > /proc/fs/nilfs2/debug_option # mount -t nilfs2 -r /dev/wdigital5 /scratch/
Dec 3 06:47:50 echelon kernel: NILFS nilfs_fill_super: start(silent=0) Dec 3 06:47:50 echelon kernel: NILFS(recovery) nilfs_search_super_root: looking segment (seg_start=174272512, seg_end=174274559, segnum=85094, seg_seq=181656) Dec 3 06:47:50 echelon kernel: NILFS(recovery) load_segment_summary: checking segment (pseg_start=174272512, full_check=1) Dec 3 06:47:51 echelon kernel: NILFS(recovery) load_segment_summary: done (ret=0) Dec 3 06:47:51 echelon kernel: NILFS(recovery) nilfs_search_super_root: found super root: segnum=85094, seq=181656, pseg_start=174272512, pseg_offset=295 Dec 3 06:47:51 echelon kernel: NILFS nilfs_attach_checkpoint: failed to attach ifile (checkpoint number=93986, err=-22) Dec 3 06:47:51 echelon kernel: NILFS nilfs_fill_super: aborted Dec 3 06:47:51 echelon kernel: NILFS put_nilfs: the_nilfs on bdev sdb5 was freed
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · Adrian Ulrich
Still fails to mount :
# insmod fs/nilfs2.ko # echo "-vvv recovery" > /proc/fs/nilfs2/debug_option # mount -t nilfs2 -r /dev/wdigital5 /scratch/
Quoted 8 lines
Dec 3 06:47:50 echelon kernel: NILFS nilfs_fill_super: start(silent=0) Dec 3 06:47:50 echelon kernel: NILFS(recovery) nilfs_search_super_root: looking segment (seg_start=174272512, seg_end=174274559, segnum=85094, seg_seq=181656) Dec 3 06:47:50 echelon kernel: NILFS(recovery) load_segment_summary: checking segment (pseg_start=174272512, full_check=1) Dec 3 06:47:51 echelon kernel: NILFS(recovery) load_segment_summary: done (ret=0) Dec 3 06:47:51 echelon kernel: NILFS(recovery) nilfs_search_super_root: found super root: segnum=85094, seq=181656, pseg_start=174272512, pseg_offset=295 Dec 3 06:47:51 echelon kernel: NILFS nilfs_attach_checkpoint: failed to attach ifile (checkpoint number=93986, err=-22) Dec 3 06:47:51 echelon kernel: NILFS nilfs_fill_super: aborted Dec 3 06:47:51 echelon kernel: NILFS put_nilfs: the_nilfs on bdev sdb5 was freed
The segment pointed by the superblock was likely not the latest one.
Could you get a summary of segments as follows, and send it to me off-line?
bash# for (( i = 0; i <= 85096; i++ )); do dumpseg /dev/wdigital5 $i | grep -e creation -e segment -e "ino = 3,"; done > segdump.txt bash# bzip2 segdump.txt
This will generate the summary as follows, and it helps to identify the the last segment actually written.
segment: segnum = 0
partial segment
creation time = 2008-10-10 15:30:36
ino = 3, cno = 1, nblocks = 3, ndatblk = 3
partial segment
creation time = 2008-10-10 15:30:48
ino = 3, cno = 0, nblocks = 39, ndatblk = 38
partial segment
creation time = 2008-10-10 15:30:51
segment: segnum = 1
partial segment
creation time = 2008-10-10 15:30:52
segment: segnum = 2
partial segment
creation time = 2008-10-10 15:30:52
partial segment
creation time = 2008-10-10 15:30:54...
If you can find the latest segment from 'creation time' fields, then get a full dump, too.
# dumpseg /dev/wdigital5 [the number of the newest segment] > latest-seg.txt
Regards,
Ryusuke Konishi
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Adrian,
Thanks for dump logs.
segment: segnum = 85092
partial segment
creation time = 2008-11-26 17:41:05
partial segment
creation time = 2008-11-26 17:41:06
segment: segnum = 85093
partial segment -> pseg 174270464
creation time = 2008-11-26 17:41:06
ino = 3, cno = 0, nblocks = 86, ndatblk = 80
partial segment -> pseg 174271740
creation time = 2008-11-26 17:41:11
ino = 3, cno = 0, nblocks = 8, ndatblk = 3
partial segment -> pseg 174271758
creation time = 2008-11-26 17:42:47
ino = 3, cno = 0, nblocks = 735, ndatblk = 722
segment: segnum = 85094
partial segment -> pseg 174272512 <-- pointed by superblock
creation time = 2008-10-28 20:27:55
ino = 3, cno = 0, nblocks = 61, ndatblk = 51
partial segment
creation time = 2008-10-28 20:29:14
segment: segnum = 85095
partial segment
creation time = 2008-10-28 20:29:15According to your logs, the segment pointed by the superblock was not written as suspected.
This never occurs if the block device is supporting 'write barrier' feature properly.
Could you tell me the kernel version that you are using?
I feel that an additional disk flush operation may be needed for older kernels or such devices. (Of course, first priority should be on making fsck.)
Regards,
Ryusuke Konishi
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 3 lines
This never occurs if the block device is supporting 'write barrier' feature properly. Could you tell me the kernel version that you are using?
adrian@echelon:~$ uname -r 2.6.27.4
root@echelon:~# udevinfo -a -p /sys/block/sdb/|grep model
ATTRS{model}=="10EAVS External "
root@echelon:~# lsusb |grep Wes
Bus 001 Device 003: ID 1058:1001 Western Digital Technologies, Inc. External Hard DiskQuoted 1 line
(Of course, first priority should be on making fsck.)
Well: Is there any way to recover my data 'soon' or should i re-format the partition?
Regards,
Adrian
A message that is not in the archive (sent off-list or not yet received): 20081204092947.c29d230d.adrian-4ZM2p5qjiQGewZBzVTKGGg@public.gmane.org
Quoted 8 lines · Adrian Ulrich
Hi,
Quoted 3 lines
This never occurs if the block device is supporting 'write barrier' feature properly. Could you tell me the kernel version that you are using?
adrian@echelon:~$ uname -r 2.6.27.4
Uum, this kernel should support barriers also for USB devices.
Quoted 4 lines
root@echelon:~# udevinfo -a -p /sys/block/sdb/|grep model
ATTRS{model}=="10EAVS External "
root@echelon:~# lsusb |grep Wes
Bus 001 Device 003: ID 1058:1001 Western Digital Technologies, Inc. External Hard DiskDoes the device cache operate in writeback or writethough? ( FUA or not ? ) These parameters decide how barriers are implemented in scsi layer. You can show these by
# cat /sys/class/scsi_disk/x:x:x:x/{FUA,cache_type}where x:x:x:x is the address displayed by lsscsi command.
Quoted 4 lines
Quoted 1 line
(Of course, first priority should be on making fsck.)
Well: Is there any way to recover my data 'soon' or should i re-format the partition?
Wait for a moment. I'm now writing an impromptu tool to correct this kind of problem.
Regards,
Ryusuke Konishi
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 1 line
[...] where x:x:x:x is the address displayed by lsscsi command.
bash-3.1# lsscsi | grep 10EAVS
[4:0:0:0] disk WD 10EAVS External 1.05 /dev/wdigital
bash-3.1# cat /sys/class/scsi_disk/4\:0\:0\:0/cache_type write through
bash-3.1# cat /sys/class/scsi_disk/4\:0\:0\:0/FUA 0
Quoted 2 lines
Wait for a moment. I'm now writing an impromptu tool to correct this kind of problem.
Oh, great. I'll be glad to test it ;-)
Regards,
Adrian
A message that is not in the archive (sent off-list or not yet received): 20081204140431.ec72f1ec.adrian-4ZM2p5qjiQGewZBzVTKGGg@public.gmane.org
Hi,
Quoted 4 lines · Adrian Ulrich
Quoted 2 lines
Wait for a moment. I'm now writing an impromptu tool to correct this kind of problem.
Oh, great. I'll be glad to test it ;-)
Sorry to keep you waiting. ;-)
I will attach a patch against nilfs2-utils-2.0.6, which adds the tool 'fsck0.nilfs2'.
# fsck0.nilfs2 <device name>
will, hopefully, correct your partition.
(I'm not sure because it's not yet tested enough)
Regards,
Ryusuke Konishi
---
From ce7bac99115bdf1fdadc002ba2b14bcb1ae4c19c Mon Sep 17 00:00:00 2001
From: Ryusuke Konishi <[email protected]>
Date: Fri, 5 Dec 2008 00:53:21 +0900
Subject: [PATCH] nilfs2-utils: add test tool to correct log pointer in super block
--- sbin/mkfs/Makefile.am | 6 +- sbin/mkfs/fsck0.nilfs2.c | 958 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 963 insertions(+), 1 deletions(-) create mode 100644 sbin/mkfs/fsck0.nilfs2.c
sbin/mkfs/Makefile.am +5 -1
@@ -4,13 +4,17 @@ ## ## Written by Koji Sato <[email protected]>. -sbin_PROGRAMS = mkfs.nilfs2 +sbin_PROGRAMS = mkfs.nilfs2 fsck0.nilfs2 mkfs_nilfs2_SOURCES = mkfs.c bitops.c ../../lib/crc32.c mkfs.h mkfs_nilfs2_CFLAGS = -Wall mkfs_nilfs2_CPPFLAGS = -I$(top_srcdir)/include mkfs_nilfs2_LDADD = -luuid +fsck0_nilfs2_SOURCES = fsck0.nilfs2.c ../../lib/crc32.c mkfs.h +fsck0_nilfs2_CFLAGS = -Wall +fsck0_nilfs2_CPPFLAGS = -I$(top_srcdir)/include + install-exec-hook: list='$(sbin_PROGRAMS)'; \ for p in $$list; do \
sbin/mkfs/fsck0.nilfs2.c new file +958 -0
958 lines not on this page · raw message
Signature · 1 line
1.5.6.5
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi folks, hi Ryusuke,
Quoted 10 lines · Ryusuke Konishi
On Thu, 4 Dec 2008 14:04:31 +0100, Adrian Ulrich wrote:
Sorry to keep you waiting. ;-)I will attach a patch against nilfs2-utils-2.0.6, which adds the tool 'fsck0.nilfs2'.
# fsck0.nilfs2 <device name>will, hopefully, correct your partition.
(I'm not sure because it's not yet tested enough)
looks nice Ryusuke. Could the problem be averted though by *only* writing the superblock on dismount or when the origional block would be overwritten. Seperated by a disc flush of course :)
With regards,
Reinoud
A message that is not in the archive (sent off-list or not yet received): 20081206195308.GA11115-5cYspOl2ggRz6xQTk39kMVfVdRo2wo/[email protected]
Quoted 9 lines · Reinoud Zandijk
Hi folks, hi Ryusuke,
Quoted 6 lines · Ryusuke Konishi
On Thu, 4 Dec 2008 14:04:31 +0100, Adrian Ulrich wrote:
Sorry to keep you waiting. ;-)I will attach a patch against nilfs2-utils-2.0.6, which adds the tool 'fsck0.nilfs2'.
Quoted 8 lines
Quoted 4 lines
# fsck0.nilfs2 <device name>will, hopefully, correct your partition.
(I'm not sure because it's not yet tested enough)looks nice Ryusuke. Could the problem be averted though by *only* writing the superblock on dismount or when the origional block would be overwritten. Seperated by a disc flush of course :)
Well, unless NILFS itself had destroyed data by bugs, the filesystem should have a very good chance to be salvaged by only modifying the superblock like this tool does.
The GC may have reused data blocks needed by the re-selected old checkpoint, but it is likely very few since there is time lag from when blocks are marked unused to when they are actually overwritten under the current GC policy (i.e. FIFO order poclicy).
Is this answering your question?
Regards,
Ryusuke Konishi
Hi Ryusuke,
Quoted 2 lines
will, hopefully, correct your partition.
(I'm not sure because it's not yet tested enough)
Thanks a lot for the patch: Running fsck0 fixed the problem and i was able to mount the filesystem.
So far everything looks fine (data is stll there and i can write to the filesystem),
but: I got this in my syslog:
segctord starting. Construction interval = 5 seconds, CP frequency < 30 seconds NILFS warning: mounting fs with errors
Should i worry about this?
Regards,
Adrian
A message that is not in the archive (sent off-list or not yet received): 20081207132335.868f8479.adrian-4ZM2p5qjiQGewZBzVTKGGg@public.gmane.org
Hi Adrian,
Quoted 9 lines · Adrian Ulrich
Hi Ryusuke,
Quoted 2 lines
will, hopefully, correct your partition.
(I'm not sure because it's not yet tested enough)Thanks a lot for the patch: Running fsck0 fixed the problem and i was able to mount the filesystem.
So far everything looks fine (data is stll there and i can write to the filesystem),
That's the good news!
Quoted 7 lines
but: I got this in my syslog:
segctord starting. Construction interval = 5 seconds, CP frequency < 30 seconds NILFS warning: mounting fs with errors
Should i worry about this?
This message means that the filesystem has detected some sort of inconsistency, but you don't have to worry in your case; the error flag can be marked imprecisely when a filesystem has detected I/O errors even if no inconsistency occurs.
This flag is not cleared by fsck0 because I haven't yet implemented the feature which fully verifies coherency of a filesystem. ;)
Well, if you can't ignore the message, I can add a dummy force check option that does nothing but drop the flag.
Regards,
Ryusuke Konishi
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 2 lines
Well, if you can't ignore the message, I can add a dummy force check option that does nothing but drop the flag.
I can life with it.. Just wanted to ask if it's 'normal' :-)
Regards,
Adrian
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Adrian Ulrich
Hi,
Quoted 2 lines
Well, if you can't ignore the message, I can add a dummy force check option that does nothing but drop the flag.
I can life with it.. Just wanted to ask if it's 'normal' :-)
Regards,
Adrian
All right, I will support the option if someone wants it or after I will have implemented meaningful checks.
Thanks,
Ryusuke
Quoted 8 lines · Ryusuke Konishi
This never occurs if the block device is supporting 'write barrier' feature properly.
Could you tell me the kernel version that you are using?
I feel that an additional disk flush operation may be needed for older kernels or such devices. (Of course, first priority should be on making fsck.)
It's worth remembering that (unless something has changed recently) filesystems on a logical volume using LVM won't have barriers either.
cheers,
Chris
Signature · 4 lines
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC This email may come with a PGP signature as a file. Do not panic. For more info see: http://en.wikipedia.org/wiki/OpenPGP
_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 11 lines · Chris Samuel
Quoted 8 lines · Ryusuke Konishi
This never occurs if the block device is supporting 'write barrier' feature properly.
Could you tell me the kernel version that you are using?
I feel that an additional disk flush operation may be needed for older kernels or such devices. (Of course, first priority should be on making fsck.)
Quoted 5 lines
It's worth remembering that (unless something has changed recently) filesystems on a logical volume using LVM won't have barriers either.
cheers,
Chris
That's right! It should be counted.
Thanks,
Ryusuke Konishi