Problem with nfs server reboot causing exports to jumble
Subsystems (2)
We're using the 2.6.18 x86_64 kernel (RH SL 5.1) with a NFS v3 server and NFS v3 clients using the automounter with options (rw,noatime,hard,intr,rsize=32768,wsize=32768,proto=tcp).
Our NFS server had several exports
/export/a/{files-a}
/export/b/{files-b}these were mounted on various clients.
We then added a new mount and export without rebooting:
/export/a/{files-a}
/export/b/{files-b}
/export/c/{files-c}All were visible on the clients, and everything was fine:
/nfs/a/{files-a}
/nfs/b/{files-b}
/nfs/c/{files-c}Our NFS server then crashed, and rebooted. After reboot the clients then had:
/nfs/a/{files-c}
/nfs/b/{files-a}
/nfs/c/{files-b}Note the files correspond to the wrong exports. Remounting on every client fixed the problem - once we realized what was going on! (files-a/b/c were all similar so it wasn't immediately aparent)
My speculation is the reboot changed some enumeration causing a mismatch between the clients and servers.
This is a new one by me - is this a known issue, and if so is a patch available?
Other than patching, are there any precautions we can take to avoid this in the future? (Other than not crashing the server, which we'd be happy to not to do. :)
Thanks
A message that is not in the archive (sent off-list or not yet received): 20100225192911.09CC318839E-12vJDDTx9R9AfugRpC6u6w@public.gmane.org
Wilson,
Quoted 39 lines · Wilson Snyder
We're using the 2.6.18
x86_64kernel (RH SL 5.1) with a NFS v3 server and NFS v3 clients using the automounter with options (rw,noatime,hard,intr,rsize=32768,wsize=32768,proto=tcp).Our NFS server had several exports
/export/a/{files-a} /export/b/{files-b}these were mounted on various clients.
We then added a new mount and export without rebooting:
/export/a/{files-a} /export/b/{files-b} /export/c/{files-c}All were visible on the clients, and everything was fine:
/nfs/a/{files-a} /nfs/b/{files-b} /nfs/c/{files-c}Our NFS server then crashed, and rebooted. After reboot the clients then had:
/nfs/a/{files-c} /nfs/b/{files-a} /nfs/c/{files-b}Note the files correspond to the wrong exports. Remounting on every client fixed the problem - once we realized what was going on! (files-a/b/c were all similar so it wasn't immediately aparent)
My speculation is the reboot changed some enumeration causing a mismatch between the clients and servers.
Quoted 6 lines
This is a new one by me - is this a known issue, and if so is a patch available?
Other than patching, are there any precautions we can take to avoid this in the future? (Other than not crashing the server, which we'd be happy to not to do. :)
Use the mp and fsid export options documented in the exports manpage.
-Ben
Quoted 43 lines
Wilson,
Quoted 39 lines · Wilson Snyder
We're using the 2.6.18
x86_64kernel (RH SL 5.1) with a NFS v3 server and NFS v3 clients using the automounter with options (rw,noatime,hard,intr,rsize=32768,wsize=32768,proto=tcp).Our NFS server had several exports
/export/a/{files-a} /export/b/{files-b}these were mounted on various clients.
We then added a new mount and export without rebooting:
/export/a/{files-a} /export/b/{files-b} /export/c/{files-c}All were visible on the clients, and everything was fine:
/nfs/a/{files-a} /nfs/b/{files-b} /nfs/c/{files-c}Our NFS server then crashed, and rebooted. After reboot the clients then had:
/nfs/a/{files-c} /nfs/b/{files-a} /nfs/c/{files-b}Note the files correspond to the wrong exports. Remounting on every client fixed the problem - once we realized what was going on! (files-a/b/c were all similar so it wasn't immediately aparent)
My speculation is the reboot changed some enumeration causing a mismatch between the clients and servers.
Quoted 8 lines
Quoted 6 lines
This is a new one by me - is this a known issue, and if so is a patch available?
Other than patching, are there any precautions we can take to avoid this in the future? (Other than not crashing the server, which we'd be happy to not to do. :)
Use the mp and fsid export options documented in the exports manpage.
Reading between the lines, are you suggesting the mounts weren't ready when the exports started (or from the pre-reboot's mount), causing the fsid's to be identical? I could see that happening.
I presume then that it's enough to just use mp - assuming each mountpoint has a unique UUID? Wouldn't adding fsid now (with the server and clients all up) require me to remount every client?
Finally, is there a way to examine the present (automatically assigned) fsid's so I can confirm this was the problem?
Thanks much
A message that is not in the archive (sent off-list or not yet received): 20100225203336.9F69118839E-12vJDDTx9R9AfugRpC6u6w@public.gmane.org
Hey Wilson,
Quoted 4 lines · Wilson Snyder
Quoted 1 line
Use the mp and fsid export options documented in the exports manpage.
Reading between the lines, are you suggesting the mounts weren't ready when the exports started
I'm suggesting that it's a good idea to use the mp export option so that you don't export your root filesystem in the off chance that the filesystems you want to export aren't mounted yet.
Quoted 2 lines
I presume then that it's enough to just use mp - assuming each mountpoint has a unique UUID?
Look into how the fsid that identifies each filesystem to the nfs client is chosen by the nfs server. Is it possible that it changed across the reboot? I don't know offhand. If it did change across a reboot, using the fsid export option to explicitly set the fsid for each export might prevent this happening again.
Quoted 2 lines
Wouldn't adding fsid now (with the server and clients all up) require me to remount every client?
Yep, I think so. Maybe you want to schedule some downtime.
Quoted 3 lines
Finally, is there a way to examine the present (automatically assigned) fsid's so I can confirm this was the problem?
Maybe you can try ethereal or something like that.
-Ben
Quoted 24 lines · On Thu, 25 Feb 2010 14:29:10 -0500 (EST)…
We're using the 2.6.18
x86_64kernel (RH SL 5.1) with a NFS v3 server and NFS v3 clients using the automounter with options (rw,noatime,hard,intr,rsize=32768,wsize=32768,proto=tcp).Our NFS server had several exports
/export/a/{files-a} /export/b/{files-b}these were mounted on various clients.
We then added a new mount and export without rebooting:
/export/a/{files-a} /export/b/{files-b} /export/c/{files-c}All were visible on the clients, and everything was fine:
/nfs/a/{files-a} /nfs/b/{files-b} /nfs/c/{files-c}
Quoted 6 lines
Our NFS server then crashed, and rebooted. After reboot the clients then had:
/nfs/a/{files-c} /nfs/b/{files-a} /nfs/c/{files-b}
There is lots of useful information you haven't given us, such as a copy of /etc/exports, and /etc/fstab, and information about what hardware the files are on, but at a guess:
files-a and files-b were on /dev/sdc and /dev/sdd
you plugged in a new hard drive to store files-c. It was called /dev/sde.
But after a reboot the devices were discovered in a different order and the one containing files-c is now called /dev/sdc, files-a are on /dev/sdd and files-b are on /dev/sde.
As nfs in 2.6.18 uses the device id to identify a filesystem, the name of the device that was mounted would be more important than the name of the place where it was mounted on the server.
In more recent kernels and nfs-utils, the uuid of the filessytem is used in preference to the device id, so this problem is avoided.
On your system you can add "fsid=1", "fsid=2" etc (avoid fsid=0) to help disambiguate.
When you do this, the nfs server will still honour the old device-id-base file handles, but will only give out the fsid= filehandles for new mounts. So it will not break your current clients, but they will need to remount before they can benefit from the increased stabilitiy of fsid= exports.
NeilBrown
Quoted 21 lines
Note the files correspond to the wrong exports. Remounting on every client fixed the problem - once we realized what was going on! (files-a/b/c were all similar so it wasn't immediately aparent)
My speculation is the reboot changed some enumeration causing a mismatch between the clients and servers.
This is a new one by me - is this a known issue, and if so is a patch available?
Other than patching, are there any precautions we can take to avoid this in the future? (Other than not crashing the server, which we'd be happy to not to do. :)
Thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html