mountd prevents spindown of non-exported disk
Subsystems (2)
Hi all,
I have two disks in my server, one of them (hda) being used for backups solely. To reduce noise level and power consumption, I have been trying to keep it running in standby mode (as opposed to active) most of the time.
Although there should be nothing accessing the disk except my custom backup cron job initiating at 5am daily, something was constantly bringing it back into active state after a rough 20-25 minutes. With the help of blktrace, I monitored every single I/O access to the disk and found a single process only that would cause the wake-up:
$ sudo blkparse -i hda.blktrace.0 Input file hda.blktrace.0 added [...] 3,0 0 6 88.950000000 6806 Q R 447 + 8 [rpc.mountd] [...]
So for some reason, rpc.mountd issues this disk request in regular intervals although nothing on the disk is being NFS-exported according to /etc/exports.
mountd in debug mode did not yield anything further, so I did another run with mountd hooked up to strace, and found out that the requests happen when I NFS-mount my other drive on the server (sda), or if it's already mounted, access files on sda (read: not hda). I assume the spinup event can be pinpointed down to calls to stat64 or open on hda:
open("/dev/mapper/backup-backup1", O_RDONLY) = 12
I should mention at this point that both disks have their partitions set up facilitating LVM2, and additionally use encryption layers through dm-crypt/LUKS on top of that. However, both sda and hda are treated in completely separate physical volumes, respectively, not interweaved in any way. So any access to sda should not touch hda filesystem-wise.
I'd glad if anyone could help me out by explaining why NFS operations influence disks that are not participating in a particular operation. Preferably with a solution how I can have both NFS running on sda and keep hda asleep unintermittedly.
By the way, I'm using NFSv3.
Cheers,
--Timo
Quoted 20 lines · On Wednesday February 20, [email protected]…
Hi all,
I have two disks in my server, one of them (hda) being used for backups solely. To reduce noise level and power consumption, I have been trying to keep it running in standby mode (as opposed to active) most of the time.
Although there should be nothing accessing the disk except my custom backup cron job initiating at 5am daily, something was constantly bringing it back into active state after a rough 20-25 minutes. With the help of blktrace, I monitored every single I/O access to the disk and found a single process only that would cause the wake-up:
$ sudo blkparse -i hda.blktrace.0 Input file hda.blktrace.0 added [...] 3,0 0 6 88.950000000 6806 Q R 447 + 8 [rpc.mountd] [...]
Quoted 3 lines
So for some reason, rpc.mountd issues this disk request in regular intervals although nothing on the disk is being NFS-exported according to
/etc/exports.
This is doubtlessly something related to libblkid.
We use libblkid to get a unique UID for each filesystem and use that as an identified in the filehandle.
We only ever ask it for the UID of specific devices that have been exported. However it is quite possible that it touches other devices as well...
I'm using libblkid in a way that it wasn't originally designed to be used. It was (as far as I can tell) designed to find a device given a UUID or similar. In that case you would expect it to touch every device.
I'm doing the reverse and it probably isn't doing things the ideal way.
I guess someone should figure out how libblkid works and make it work better for us......
You could always build your own nfs-utils and configure with --without-uuid.
Not an ideal solution...
NeilBrown
Hi Neil,
first of all, thanks for replying.
Quoted 26 lines · Neil Brown
Quoted 10 lines · On Wednesday February 20, [email protected]…
Although there should be nothing accessing the disk except my custom backup cron job initiating at 5am daily, something was constantly bringing it back into active state after a rough 20-25 minutes. With the help of blktrace, I monitored every single I/O access to the disk and found a single process only that would cause the wake-up:
So for some reason, rpc.mountd issues this disk request in regular intervals although nothing on the disk is being NFS-exported according to
/etc/exports.This is doubtlessly something related to libblkid.
We use libblkid to get a unique UID for each filesystem and use that as an identified in the filehandle.
We only ever ask it for the UID of specific devices that have been exported. However it is quite possible that it touches other devices as well...
I'm using libblkid in a way that it wasn't originally designed to be used. It was (as far as I can tell) designed to find a device given a UUID or similar. In that case you would expect it to touch every device.
Quoted 4 lines
You could always build your own nfs-utils and configure with --without-uuid.
Not an ideal solution...
What would be the downside of that solution? I suppose NFS would keep on working even if I disabled that switch, but you gave the impression there was some (major) drawback.
Cheers,
--Timo
Quoted 10 lines · On Thursday February 21, [email protected]…
Quoted 5 lines
You could always build your own nfs-utils and configure with --without-uuid.
Not an ideal solution...
What would be the downside of that solution? I suppose NFS would keep on working even if I disabled that switch, but you gave the impression there was some (major) drawback.
The fact that you have to compile your own tools rather than just use what the distro provides is always a bit of a bore.
Also, if you change the server to stop using UUIDs, then any clients which have the filesystem mounted will need to remount as the old mount will become stale.
So it isn't a major problem, but nor is it an ideal solution.
NeilBrown
Quoted 10 lines · Neil Brown
Quoted 8 lines · On Thursday February 21, [email protected]…
Quoted 4 lines
You could always build your own nfs-utils and configure with --without-uuid.
Not an ideal solution...
What would be the downside of that solution? I suppose NFS would keep on working even if I disabled that switch, but you gave the impression there was some (major) drawback.
Quoted 8 lines
The fact that you have to compile your own tools rather than just use what the distro provides is always a bit of a bore.
Also, if you change the server to stop using UUIDs, then any clients which have the filesystem mounted will need to remount as the old mount will become stale.
So it isn't a major problem, but nor is it an ideal solution.
Is there a proper place to file a bug report for this issue? I'm lacking the expertise to know whose responsibility it is, but I remember you mentioned libblkid before.
--Timo
Quoted 25 lines · Neil Brown
Quoted 14 lines · On Wednesday February 20, [email protected]…
Although there should be nothing accessing the disk except my custom backup cron job initiating at 5am daily, something was constantly bringing it back into active state after a rough 20-25 minutes. With the help of blktrace, I monitored every single I/O access to the disk and found a single process only that would cause the wake-up:
$ sudo blkparse -i hda.blktrace.0 Input file hda.blktrace.0 added [...] 3,0 0 6 88.950000000 6806 Q R 447 + 8 [rpc.mountd] [...]Quoted 3 lines
So for some reason, rpc.mountd issues this disk request in regular intervals although nothing on the disk is being NFS-exported according to
/etc/exports.This is doubtlessly something related to libblkid.
We only ever ask it for the UID of specific devices that have been exported. However it is quite possible that it touches other devices as well...
Quoted 7 lines
I'm using libblkid in a way that it wasn't originally designed to be used. It was (as far as I can tell) designed to find a device given a UUID or similar. In that case you would expect it to touch every device.
You could always build your own nfs-utils and configure with --without-uuid.
I did so and re-compiled the nfs-utils package, resulting in new packages of nfs-common and nfs-kernel-server under Ubuntu 7.10.
However, another re-run of blktrace yielded that the problem did not go away. There are still lots of disk accesses from mountd, and the strace output continuously contains open calls to the unmounted backup filesystem.
Is there anything else I can do in attempt to resolve this issue? I suppose pasting the complete output of strace might help, but I wouldn't do so without explicit confirmation of yours as the log-file is 3.2 MB large.
--Timo
Quoted 42 lines · Timo Reimann
Quoted 25 lines · Neil Brown
Quoted 14 lines · On Wednesday February 20, [email protected]…
Although there should be nothing accessing the disk except my custom backup cron job initiating at 5am daily, something was constantly bringing it back into active state after a rough 20-25 minutes. With the help of blktrace, I monitored every single I/O access to the disk and found a single process only that would cause the wake-up:
$ sudo blkparse -i hda.blktrace.0 Input file hda.blktrace.0 added [...] 3,0 0 6 88.950000000 6806 Q R 447 + 8 [rpc.mountd] [...]Quoted 3 lines
So for some reason, rpc.mountd issues this disk request in regular intervals although nothing on the disk is being NFS-exported according to
/etc/exports.This is doubtlessly something related to libblkid.
We only ever ask it for the UID of specific devices that have been exported. However it is quite possible that it touches other devices as well...
Quoted 7 lines
I'm using libblkid in a way that it wasn't originally designed to be used. It was (as far as I can tell) designed to find a device given a UUID or similar. In that case you would expect it to touch every device.
You could always build your own nfs-utils and configure with --without-uuid.
I did so and re-compiled the nfs-utils package, resulting in new packages of nfs-common and nfs-kernel-server under Ubuntu 7.10.
However, another re-run of blktrace yielded that the problem did not go away. There are still lots of disk accesses from mountd, and the strace output continuously contains open calls to the unmounted backup filesystem.
Quoted 4 lines
Is there anything else I can do in attempt to resolve this issue? I suppose pasting the complete output of strace might help, but I wouldn't do so without explicit confirmation of yours as the log-file is 3.2 MB large.
Probably the most likely to read such an strace are me, Neil, and Steve (<[email protected]>), so you might just send it privately to the three of us.
--b.
Quoted 4 lines · Timo Reimann
Is there anything else I can do in attempt to resolve this issue? I suppose pasting the complete output of strace might help, but I wouldn't do so without explicit confirmation of yours as the log-file is 3.2 MB large.
Maybe this is too simplistic, but have you tried exportfs -f? Old entry in the kernel table?
Whats in /var/lib/nfs/xtab?
~Ben
Quoted 8 lines · Benjamin Coddington
Quoted 4 lines · Timo Reimann
Is there anything else I can do in attempt to resolve this issue? I suppose pasting the complete output of strace might help, but I wouldn't do so without explicit confirmation of yours as the log-file is 3.2 MB large.
Maybe this is too simplistic, but have you tried exportfs -f? Old entry in the kernel table?
I had to reboot the machine a couple of times, so I suppose that is similar to calling exportfs -f.
Just to make sure, I tried the command too, but to no avail.
Quoted 1 line
Whats in
/var/lib/nfs/xtab?
It's empty. Is that reason to worry?
--Timo