Skip to contentSkip to conversation

Kernel Bug after corrupted filesystem

DISCUSSION [email protected] 3 participants 10 messages Dec 30, 2008 – Jan 3, 2009 7 messages not in the archive NILFS2 filesystem

lore
Share & tools
lore
#0 lore raw

Hi,
I am experimenting with nilfs2 for a while now, because I like the Idea of this filesystem. The filesystem I have problems with now is a more complex problem. I used just for testing purpose a file on an ext3 filesystem as a cryptoloop device and I used nilfs2 on the loopback device. It worked for about half a year without a problem. But I have no Idea what was actualy going wrong (ext3 problem, cryptoloop problem, nilfs problem?) and I don't know if a filesystem under normal conditations get corrupted like this (actualy no real data loss either, because almost all files existed elswhere)

But when I now try to mount the nilfs2 filesystem it does not work. No result from the mount command, but the attached in dmsg. The system is still useable, but I am not able to shut it down (have to hold power button 3 seconds). And I am not able to unload the nilfs module either (is in use)

Hope this helps to further improve nilfs2.

robinx99

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

Hi,

Quoted 6 lines · On Tue, 30 Dec 2008 14:19:53 +0100, [email protected] wrote:

Hi,
I am experimenting with nilfs2 for a while now, because I like the Idea of this filesystem. The filesystem I have problems with now is a more complex problem. I used just for testing purpose a file on an ext3 filesystem as a cryptoloop device and I used nilfs2 on the loopback device. It worked for about half a year without a

Quoted 8 lines

problem. But I have no Idea what was actualy going wrong (ext3 problem, cryptoloop problem, nilfs problem?) and I don't know if a filesystem under normal conditations get corrupted like this (actualy no real data loss either, because almost all files existed elswhere)
But when I now try to mount the nilfs2 filesystem it does not work. No result from the mount command, but the attached in dmsg.

Thank you for reporting the problem.
Again, it seems that a btree corruption happened.
Sorry, but it's hard to track the cause directly from the stack dump.

According to the log, the write barrier is disabled (it may be a restriction of cryptoloop). If you met an unexpected system down before mounting nilfs, please let me know.

Quoted 4 lines

The system is still useable, but I am not able to shut it down (have to hold power button 3 seconds). And I am not able to unload the nilfs module either (is in use) Hope this helps to further improve nilfs2. robinx99

I've modified btree lookup routines to avoid system freeze and allow clean termination; I pushed it to our git repository.

Regards,
Ryusuke

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 3 lines · Ryusuke Konishi

According to the log, the write barrier is disabled (it may be a restriction of cryptoloop). If you met an unexpected system down before mounting nilfs, please let me know.

Uh-oh, it's my misconception. forget it.

Ryusuke

#5 lore raw
In reply to a message not in the archive

-------- Original-Nachricht --------

Quoted 9 lines

Datum: Wed, 31 Dec 2008 11:53:54 +0900 (JST)
Von: Ryusuke Konishi <[email protected]>
An: [email protected], [email protected]
Betreff: Re: [NILFS users] Kernel Bug after corrupted filesystem

Thank you for reporting the problem.
Again, it seems that a btree corruption happened.
Sorry, but it's hard to track the cause directly from the stack dump.

Is there anything else I could provide? That might help to track down problems

Quoted 4 lines

I've modified btree lookup routines to avoid system freeze and allow clean termination; I pushed it to our git repository.

That looks much better the module is now unloadable after failing to mount. It now gives an error message when trying to mount
mount.nilfs2: Error while mounting /dev/loop1 on /mnt/test: Input/output error

and now there are also some other things in dmsg I will atach this to the mail in hope that it helps

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw

Hi,

Quoted 5 lines · Dennis Peiler
Quoted 3 lines

Again, it seems that a btree corruption happened.
Sorry, but it's hard to track the cause directly from the stack dump.

Is there anything else I could provide? That might help to track down problems

Thanks.
Well, could you try to get some information with a debug build module?

First, pull the latest code from our git repository.

The debug build module can be built by commenting out the following line of the fs/Makefile.

#CONFIG_NILFS_DEBUG=y

Then, set debug levels before mounting as follows:

  # modprobe nilfs2
  # echo "-vvv fs -vvv recovery -vvv mdt -vvv btnode -vv dat" > /proc/fs/nilfs2/debug_option
  # mount -t nilfs2 /dev/xxx /test

This will log some more information in syslog including block addresses.
What I'd like to see next is content of key disk blocks, but first we need to know rough situation.

Regards,
Ryusuke

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 18 lines

Thanks.
Well, could you try to get some information with a debug build module?

First, pull the latest code from our git repository.

The debug build module can be built by commenting out the following line of the fs/Makefile.

#CONFIG_NILFS_DEBUG=y

Then, set debug levels before mounting as follows:

  # modprobe nilfs2
  # echo "-vvv fs -vvv recovery -vvv mdt -vvv btnode -vv dat" >
/proc/fs/nilfs2/debug_option
  # mount -t nilfs2 /dev/xxx /test
Quoted 5 lines

This will log some more information in syslog including block addresses.
What I'd like to see next is content of key disk blocks, but first we need to know rough situation.

Hi and happy new year,
ok have done that and atached the log hope this information is usefull and I hope this helps to further improve nilfs

Robinx99

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

#10 lore raw
In reply to a message not in the archive
Quoted 18 lines

Thanks.
Well, could you try to get some information with a debug build module?

First, pull the latest code from our git repository.

The debug build module can be built by commenting out the following line of the fs/Makefile.

#CONFIG_NILFS_DEBUG=y

Then, set debug levels before mounting as follows:

  # modprobe nilfs2
  # echo "-vvv fs -vvv recovery -vvv mdt -vvv btnode -vv dat" >
/proc/fs/nilfs2/debug_option
  # mount -t nilfs2 /dev/xxx /test
Quoted 5 lines

This will log some more information in syslog including block addresses.
What I'd like to see next is content of key disk blocks, but first we need to know rough situation.

Hi and happy new year,
ok have done that and atached the log hope this information is usefull and I hope this helps to further improve nilfs

Robinx99

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw

Happy new year,

Quoted 4 lines · On Thu, 01 Jan 2009 10:31:21 +0100, [email protected] wrote:

Hi and happy new year,
ok have done that and atached the log hope this information is usefull and I hope this helps to further improve nilfs

Robinx99

Thanks for your help.
I have another request:

 # dumpseg /dev/loop1 12

- This will dump the summary of the segment in question

 # dd if=/dev/loop1 bs=4k skip=24576 count=2 | hd

- This will dump the (broken) root block of the file system

According to your log, the btree root of DAT file (i.e. a table file to translate disk addresses) seems inconsistent. It might not be updated properly when GC moved its blocks. I suspect that it was caused by some sort of GC problem.

And, your nilfs partition seems to have shut down uncleanly before this mount error repeatedly happened. Do you remember what the first trouble was like?

Regards,
Ryusuke

A message that is not in the archive (sent off-list or not yet received): [email protected]

#14 lore raw

Oops didn't send the mail to the mailinglist, so I will resend. Hi

Quoted 5 lines

Thanks for your help.
I have another request:

 # dumpseg /dev/loop1 12
root@robin-laptop:/tmp# dumpseg /dev/loop1 12
segment: segnum = 12
  partial segment
    creation time = 2008-12-26 11:45:06
    nfinfo = 0
Quoted 7 lines

- This will dump the summary of the segment in question

 # dd if=/dev/loop1 bs=4k skip=24576 count=2 | hd

- This will dump the (broken) root block of the file system

Ok I atached it to the email bzip2 compressed

Quoted 9 lines

According to your log, the btree root of DAT file (i.e. a table file to translate disk addresses) seems inconsistent. It might not be updated properly when GC moved its blocks. I suspect that it was caused by some sort of GC problem.

And, your nilfs partition seems to have shut down uncleanly before this mount error repeatedly happened. Do you remember what the first trouble was like?

Actualy I have no real idea. Didn't start the computer for some days over christmas, befor christmas I could use the filesystem withoul any trouble, and on december 26 I was unable to mount it.
I actualy quite often moved files of the nilfs filesystem to an USB stick. Using comands like
"mv /mnt/nilfs/directory/*.avi /mnt/usb && poweroff". So there were some shutdowns I didn't watch, and I was thinking that the normal shutdown scripts should unmount every mounted filesystem. Well the computer was definitv off, but I have no idea if it could be possible that something importend was stoped to soon when it came to the point "sending all procecess the term signal"

Quoted 3 lines

Regards,
Ryusuke

robinx99

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

A message that is not in the archive (sent off-list or not yet received): [email protected]

#16 lore raw

Hi,

Quoted 6 lines · On Sat, 03 Jan 2009 08:54:38 +0100, [email protected] wrote:

Oops didn't send the mail to the mailinglist, so I will resend. Hi

Quoted 4 lines

Thanks for your help.
I have another request:

Quoted 7 lines
Quoted 1 line
 # dumpseg /dev/loop1 12
root@robin-laptop:/tmp# dumpseg /dev/loop1 12
segment: segnum = 12
  partial segment
    creation time = 2008-12-26 11:45:06
    nfinfo = 0

Uh, that was not good.

Quoted 7 lines
Quoted 6 lines

- This will dump the summary of the segment in question

 # dd if=/dev/loop1 bs=4k skip=24576 count=2 | hd

- This will dump the (broken) root block of the file system

Ok I atached it to the email bzip2 compressed

This was informative.

The last segment was that written for GC and I actually confirmed that the inconsistency of DAT root. It must be a bug of one of GC, btree, or the DAT file.

Quoted 21 lines
Quoted 9 lines

According to your log, the btree root of DAT file (i.e. a table file to translate disk addresses) seems inconsistent. It might not be updated properly when GC moved its blocks. I suspect that it was caused by some sort of GC problem.

And, your nilfs partition seems to have shut down uncleanly before this mount error repeatedly happened. Do you remember what the first trouble was like?

Actualy I have no real idea. Didn't start the computer for some days over christmas, befor christmas I could use the filesystem withoul any trouble, and on december 26 I was unable to mount it. I actualy quite often moved files of the nilfs filesystem to an USB stick. Using comands like "mv /mnt/nilfs/directory/*.avi /mnt/usb && poweroff". So there were some shutdowns I didn't watch, and I was thinking that the normal shutdown scripts should unmount every mounted filesystem. Well the computer was definitv off, but I have no idea if it could be possible that something importend was stoped to soon when it came to the point "sending all procecess the term signal"

Quoted 1 line

robinx99

Thank you once again.

There is no doubt that this corruption was caused by a bug of NILFS because NILFS verifies validity of every segment with checksums.

If NILFS doesn't have any bugs, and write barrier is assured, the file system must be recovered propery even if you have reset the PC during GC (unless hardware failures occur, of course).

So we have to review code to find out the cause, and I believe your information will help this.

Thanks,
Ryusuke