nilfs_cleanerd cannot open nilfs on device that exists

DISCUSSION Piotr Szymaniak 2 participants 7 messages Dec 12, 2012 – Jan 3, 2013 3 messages not in the archive NILFS2 filesystem

lore

Share & tools

lore

Piotr Szymaniak

author

2012-12-12 14:51 UTC #0 lore raw

Hello,

I got a power failure today and after power restore there's a weird issue with nilfs_cleanerd:

wloczykij ~ # ll /dev/sda3
brw-rw---- 1 root disk 8, 3 Dec 12 08:35 /dev/sda3

wloczykij ~ # mount | grep nilfs
/dev/root on / type nilfs2 (rw,relatime) /dev/mapper/backup on /mnt/home_backup type nilfs2 (rw,gcpid=2995) /dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=5333)

But when I try to run the cleaner:

Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: start
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot open nilfs on
/dev/sda3: No such file or directory
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot create cleanerd
on /dev/sda3: No such file or directory
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: shutdown

Never happend before. Shutdown should be nice (got UPS monitored by apcupsd, so I hope it worked - was in the middle of the night, so can't really tell if was superclean shutdown).

Piotr Szymaniak.

Signature · 2 lines

Smiejmy sie! Kto wie czy swiat potrwa jeszcze trzy tygodnie?
  -- Beaumarchais Pierre Augustin

Piotr Szymaniak

author

2012-12-12 14:54 UTC #1 lore raw

Quoted 21 lines · Piotr Szymaniak

Hello,
I got a power failure today and after power restore there's a weird issue with nilfs_cleanerd:
wloczykij ~ # ll /dev/sda3
brw-rw---- 1 root disk 8, 3 Dec 12 08:35 /dev/sda3
wloczykij ~ # mount | grep nilfs
/dev/root on / type nilfs2 (rw,relatime) /dev/mapper/backup on /mnt/home_backup type nilfs2 (rw,gcpid=2995) /dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=5333)
But when I try to run the cleaner:
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: start
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot open nilfs on
/dev/sda3: No such file or directory
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot create cleanerd
on /dev/sda3: No such file or directory
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: shutdown

Quoted 3 lines

Never happend before. Shutdown should be nice (got UPS monitored by apcupsd, so I hope it worked - was in the middle of the night, so can't really tell if was superclean shutdown).

I should also notice, that other cleaner of different drive is working without issues:
wloczykij ~ # ps aux | grep nilfs
root 2995 0.0 0.0 2348 808 ? Ss 12:19 0:00
/sbin/nilfs_cleanerd -n /dev/mapper/backup /mnt/backup

Piotr Szymaniak.

Signature · 4 lines

Przypomnialem sobie, jak powiedzial, ze po raz pierwszy w zyciu boi sie
piekla,  poniewaz  szykujemy sie do zniszczenia daru niebios.  Poczulem
nagle, ze chce mi sie rzygac.
  -- Stephen King, "The Green Mile"

Piotr Szymaniak

author

2012-12-12 16:10 UTC #2 lore raw

Quoted 26 lines · Piotr Szymaniak

Quoted 21 lines · Piotr Szymaniak
Hello,
I got a power failure today and after power restore there's a weird issue with nilfs_cleanerd:
wloczykij ~ # ll /dev/sda3
brw-rw---- 1 root disk 8, 3 Dec 12 08:35 /dev/sda3
wloczykij ~ # mount | grep nilfs
/dev/root on / type nilfs2 (rw,relatime) /dev/mapper/backup on /mnt/home_backup type nilfs2 (rw,gcpid=2995) /dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=5333)
But when I try to run the cleaner:
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: start
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot open nilfs on
/dev/sda3: No such file or directory
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot create cleanerd
on /dev/sda3: No such file or directory
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: shutdown
Quoted 3 lines
Never happend before. Shutdown should be nice (got UPS monitored by apcupsd, so I hope it worked - was in the middle of the night, so can't really tell if was superclean shutdown).

Quoted 5 lines

I should also notice, that other cleaner of different drive is working without issues:
wloczykij ~ # ps aux | grep nilfs
root 2995 0.0 0.0 2348 808 ? Ss 12:19 0:00
/sbin/nilfs_cleanerd -n /dev/mapper/backup /mnt/backup

Sorry for answering my own msg again.

Attached strace output.

Piotr Szymaniak.

Signature · 3 lines

Gdzie  w  tym  logika? Po co w ogole sie rodzic,  jesli jedyne,  co cie
czeka u kresu tego wszystkiego, to smierc?
  -- Graham Masterton, "The Burning"

execve("/sbin/nilfs_cleanerd", ["/sbin/nilfs_cleanerd", "-c", "/etc/nilfs_cleanerd.conf", "/dev/sda3"], [/* 28 vars */]) = 0
brk(0) = 0x865c000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7727000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=23848, ...}) = 0
mmap2(NULL, 23848, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7721000
close(3) = 0
open("/lib/libnilfs.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\240\24\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=18744, ...}) = 0
mmap2(NULL, 21660, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb771b000
mmap2(0xb771f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3) = 0xb771f000
close(3) = 0
open("/lib/libnilfsgc.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220\20\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=13652, ...}) = 0
mmap2(NULL, 16568, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7716000
mmap2(0xb7719000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0xb7719000
close(3) = 0
open("/lib/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\36\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=30552, ...}) = 0
mmap2(NULL, 33364, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb770d000
mmap2(0xb7714000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6) = 0xb7714000
close(3) = 0
open("/lib/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\21\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=13808, ...}) = 0
mmap2(NULL, 16588, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7708000
mmap2(0xb770b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0xb770b000
close(3) = 0
open("/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\314\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1709408, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7707000
mmap2(NULL, 1719044, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7563000

mmap2(0xb7701000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19e) = 0xb7701000
mmap2(0xb7704000, 11012, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7704000

close(3) = 0
open("/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260k\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=124345, ...}) = 0
mmap2(NULL, 107016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7548000

mmap2(0xb755f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0xb755f000
mmap2(0xb7561000, 4616, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7561000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7547000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7547940, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb7701000, 8192, PROT_READ)   = 0
mprotect(0xb755f000, 4096, PROT_READ)   = 0
mprotect(0xb770b000, 4096, PROT_READ)   = 0
mprotect(0xb7714000, 4096, PROT_READ)   = 0
mprotect(0xb771f000, 4096, PROT_READ)   = 0
mprotect(0xb7719000, 4096, PROT_READ)   = 0
mprotect(0x804e000, 4096, PROT_READ)    = 0
mprotect(0xb7749000, 4096, PROT_READ)   = 0
munmap(0xb7721000, 23848)               = 0
set_tid_address(0xb75479a8)             = 9341
set_robust_list(0xb75479b0, 12)         = 0

futex(0xbff37f24, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0xbff37f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, b7547940) = -1 EAGAIN (Resource temporarily unavailable) rt_sigaction(SIGRTMIN, {0xb754e540, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0xb754e5c0, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
uname({sys="Linux", node="wloczykij", ...}) = 0
readlink("/dev", 0xbff35baf, 4096) = -1 EINVAL (Invalid argument)
readlink("/dev/sda3", 0xbff35baf, 4096) = -1 EINVAL (Invalid argument)
brk(0) = 0x865c000
brk(0x867d000) = 0x867d000
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb75479a8) = 9342
exit_group(0) = ?
+++ exited with 0 +++

Vyacheslav Dubeyko

2012-12-12 20:43 UTC #3 lore raw

Quoted 10 lines · Piotr Szymaniak

Quoted 9 lines · Piotr Szymaniak
Quoted 8 lines · Piotr Szymaniak
Hello,
I got a power failure today and after power restore there's a weird issue with nilfs_cleanerd:
wloczykij ~ # ll /dev/sda3
brw-rw---- 1 root disk 8, 3 Dec 12 08:35 /dev/sda3

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
wloczykij ~ # mount | grep nilfs
/dev/root on / type nilfs2 (rw,relatime) /dev/mapper/backup on /mnt/home_backup type nilfs2 (rw,gcpid=2995) /dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=5333)
But when I try to run the cleaner:

Sorry, I slightly confused. As I understand, mount output shows that you have mounted /dev/sda3 with started nilfs_cleanerd (gcpid=5333). Why do you try to start nilfs_cleanerd by hands? Or I misunderstand something?

As I can see from strace, there is such error message:

futex(0xbff37f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, b7547940) = -1 EAGAIN (Resource temporarily unavailable)

I think that it is the reason of failing nilfs_cleanerd start.

Maybe, do you have working or failed nilfs_cleanerd for /dev/sda3 in background? Do you have any NILFS2-related messages in system log?

With the best regards,
Vyacheslav Dubeyko.

Quoted 17 lines

Quoted 11 lines
Quoted 10 lines
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: start
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot open nilfs on
/dev/sda3: No such file or directory
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: cannot create cleanerd
on /dev/sda3: No such file or directory
Dec 12 15:46:17 wloczykij nilfs_cleanerd[5333]: shutdown
Never happend before. Shutdown should be nice (got UPS monitored by apcupsd, so I hope it worked - was in the middle of the night, so can't really tell if was superclean shutdown).
Quoted 5 lines
I should also notice, that other cleaner of different drive is working without issues:
wloczykij ~ # ps aux | grep nilfs
root 2995 0.0 0.0 2348 808 ? Ss 12:19 0:00
/sbin/nilfs_cleanerd -n /dev/mapper/backup /mnt/backup

Quoted 11 lines

Sorry for answering my own msg again.
Attached strace output.
Piotr Szymaniak.
Signature · 4 lines
Gdzie  w  tym  logika? Po co w ogole sie rodzic,  jesli jedyne,  co cie
czeka u kresu tego wszystkiego, to smierc?
 -- Graham Masterton, "The Burning"
<strace-nilfs_cleanerd.dontexist.out>

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 7A92569A-D56B-45B0-9305-EBC2EDF90913-yeENwD64cLxBDgjK7y7TUQ@public.gmane.org

Piotr Szymaniak

author

2012-12-12 21:20 UTC #5 lore raw

Quoted 13 lines · Vyacheslav Dubeyko

Quoted 10 lines · Piotr Szymaniak
Quoted 9 lines · Piotr Szymaniak
Quoted 8 lines · Piotr Szymaniak
Hello,
I got a power failure today and after power restore there's a weird issue with nilfs_cleanerd:
wloczykij ~ # ll /dev/sda3
brw-rw---- 1 root disk 8, 3 Dec 12 08:35 /dev/sda3

Quoted 8 lines

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
wloczykij ~ # mount | grep nilfs
/dev/root on / type nilfs2 (rw,relatime) /dev/mapper/backup on /mnt/home_backup type nilfs2 (rw,gcpid=2995) /dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=5333)
But when I try to run the cleaner:
Sorry, I slightly confused. As I understand, mount output shows that you have mounted /dev/sda3 with started nilfs_cleanerd (gcpid=5333). Why do you try to start nilfs_cleanerd by hands? Or I misunderstand something?

It's not running. I got similar message on boot:

Dec 12 08:35:23 wloczykij nilfs_cleanerd[1689]: start
Dec 12 08:35:23 wloczykij nilfs_cleanerd[1689]: cannot open nilfs on
/dev/sda3: No such file or directory
Dec 12 08:35:23 wloczykij nilfs_cleanerd[1689]: cannot create cleanerd
on /dev/sda3: No such file or directory
Dec 12 08:35:23 wloczykij nilfs_cleanerd[1689]: shutdown

Quoted 7 lines

As I can see from strace, there is such error message:
futex(0xbff37f24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, b7547940) = -1 EAGAIN (Resource temporarily unavailable)
I think that it is the reason of failing nilfs_cleanerd start.
Maybe, do you have working or failed nilfs_cleanerd for /dev/sda3 in background? Do you have any NILFS2-related messages in system log?

ps ax shows only one nilfs_cleanerd process (at different drive).

Piotr Szymaniak.

Signature · 3 lines

 Podejrzane!  No,  yhm!  Takem  sobie  pomyślał!  Jezus  jakby  się  o tym
zwiedział, toby wtrząchnął puszkę fasoli!
  -- Stephen King, "The Dark Half"

A message that is not in the archive (sent off-list or not yet received): CAB7xdinBgjcLuKL6+Ydtz1mADDGv1kbw+wZ1Y=hOO1VF++iS1A@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): CAB7xdinBgjcLuKL6+Ydtz1mADDGv1kbw+wZ1Y=hOO1VF++iS1A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Vyacheslav Dubeyko

2013-01-03 10:44 UTC #8 lore raw

Hi,

Quoted 4 lines · sheng qiu

Hi all,
i am new on nilfs2. i want to collect the information about the segment cleaning overhead. for example, how many valid data copied during segment cleaning process. Can anyone give me some hint where i can add these stats codes within the nilfs2 source code? or which source files are related with segment cleaning?

The segments' cleaning code is distributed between nilfs-cleanerd daemon and NILFS2 driver itself. The nilfs-cleanerd daemon is a part of nilfs-utils. It collects information about dirty segments and initiates cleaning activity by means of ioctl on the driver side. So, you can see nilfs-cleanerd source code and linux/fs/nilfs2/ioctl.c source code as starting points. It is used in cleaning activity, as minimum, nilfs_ioctl_get_bdescs() and nilfs_ioctl_clean_segments(). I think that, maybe, it will be more easy and useful to use user-space nilfs-cleanerd side. Moreover, maybe, it make sense to use blktrace utility for your research as additional tool.

What information do you want to get for your research? How do you plan to analyze and interpret segments cleaning overhead?

But, currently, I think that such research can be a very difficult task because of unpredictability of cleaning overhead nature. It depends from filesystem aging state and history of previous filesystem operations very much, from my point of view. What the purpose of your research activity? Could you share results of your research in the direction of segments cleaning overhead?

With the best regards,
Vyacheslav Dubeyko.

Quoted 12 lines

I will be very appreciated for your help.
i tried to send email through the [email protected], however it fails to deliver, i have to write my email by replying this. hope you won't mind.
Thanks,
Sheng
Signature · 5 lines
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker    
email: [email protected]
College Station, TX 77843-3259

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Piotr Szymaniak

author

2012-12-12 22:28 UTC #9 lore raw

In reply to Piotr Szymaniak

Quoted 1 line · Piotr Szymaniak

Hello,

Sorry, it looks like I need more memory [1]. The issue is _again_ related to missing /dev/root that should point to my /dev/sda3 [2]. I was sure it was fixed meanwhile (had ~200 days uptime, so I just forgot about the issue…), but it looks like it isn't (at least in Gentoo Linux [3]).

Sorry for wasting your time. New udev, blargh. /-:

[1] http://article.gmane.org/gmane.comp.file-systems.nilfs.user/2437
[2] http://article.gmane.org/gmane.comp.file-systems.nilfs.user/2438
[3] https://bugs.gentoo.org/438380

Piotr Szymaniak.

Signature · 3 lines

 - You know nothing about woman!
 - Of course not, I'm a man, so how could I know?
  -- Berserk