XFS, NFS and inode64 on 2.6.27
Dear all,
after searching for a long time I guess I found that inode64 on XFS with kernel 2.6.27 (shipped with openSUSE 11.1) is incompatible with NFS. Sorry if this was already know, it wasn't for me and finding the problem was complicated.
This is my nfs4 root (fsid=0):
# ls -li insgesamt 0 4666 drwxr-xr-x 2 root root 48 15. Nov 19:54 daten 4668 drwxr-xr-x 2 root root 48 15. Nov 19:54 download2 4667 drwxr-xr-x 2 root root 48 15. Nov 19:54 dvd-images 4661 drwxr-xr-x 2 root root 48 19. Nov 06:51 mailsrv-backup 4669 drwxr-xr-x 2 root root 48 15. Nov 19:54 q 4670 drwxr-xr-x 2 root root 48 15. Nov 19:54 z
Then I mount those subdirs (mount --bind ....) to fill it with the dirs I want to share:
# ls -li
insgesamt 20
256 drwxr-xr-x 14 root root 4096 19. Nov 03:01 daten
6454034730 drwxrwx--- 3 zmi users 61 26. Mär 2009 download2
4070671 drwxr-xr-x 2 root root 76 13. Nov 12:17 dvd-images
6500330752 drwxr-xr-x 5 root root 4096 8. Nov 04:31 mailsrv-backup
8591091465 drwxr-xr-x 17 root root 4096 12. Nov 02:01 q
2147483904 drwxrwx--- 31 zmi bh 4096 7. Nov 09:58 zThe shares "daten" and "dvd-images" can be mounted from other servers. I simply went to the original dirs of this mount-bind, and created several new dirs:
mkdir 1 2 3 4 5 6 7 8 9
and one of them had an inode < 2G, so I moved the contents there and renamed the dirs, remounted the --bind mounts and now have this:
# ls -li
insgesamt 20
256 drwxr-xr-x 14 root root 4096 19. Nov 07:22 daten
184637244 drwxr-xr-x 3 root root 61 19. Nov 07:30 download2
4070671 drwxr-xr-x 2 root root 76 13. Nov 12:17 dvd-images
960533297 drwxr-xr-x 5 root root 4096 19. Nov 07:27 mailsrv-backup
184190222 drwxr-xr-x 16 root root 4096 19. Nov 07:41 q
184637237 drwxr-xr-x 30 root root 4096 19. Nov 07:42 zAnd everything mounts fine now.
(I still have one server that can mount, but not write, to such a share, but that may be another problem)
Just to let others know that can be a problem.
mfg zmi
Signature · 5 lines
// Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0660 / 415 65 31 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · Michael Monnerie
after searching for a long time I guess I found that inode64 on XFS with kernel 2.6.27 (shipped with openSUSE 11.1) is incompatible with NFS. Sorry if this was already know, it wasn't for me and finding the problem was complicated.
Interesting. I've never tried this, but I've read things that imply it should work. Just out of curiosity, is your NFS client also running a 64-bit kernel?
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · On Freitag, 20. November 2009 Nathaniel W. Turner wrote:
Just out of curiosity, is your NFS client also running a 64-bit
kernel?
Yep. I've used 64bit since years, as AMD has that since a long time in all their CPUs (there have just been some Semprons 32bit only). And now since every PC has >3GB RAM it's nearly a must.
mfg zmi
Signature · 5 lines
// Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0660 / 415 65 31 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Michael Monnerie
The shares "daten" and "dvd-images" can be mounted from other servers. I simply went to the original dirs of this mount-bind, and created several new dirs:
mkdir 1 2 3 4 5 6 7 8 9
and one of them had an inode < 2G, so I moved the contents there and renamed the dirs, remounted the --bind mounts and now have this:
So your NFS exports are not the roots of their respsective filesystems? This means NFSD uses non-standard filesystem IDs in the filehandles which have to encode the inode number of the export root. Your best option is to simplify switch to exporting a whole filesystem, alternatively you can try making sure NFSD uses the 16byte wide UUID style export. Note thast either way will only work with a 64bit kernel as the fs has no say in encoding the filesystem part of the handle and ino_t is always 32bit on 32bit platforms. This will also affect any other filesystem with 64bit inode numbers.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · On Samstag, 21. November 2009 Christoph Hellwig wrote:
So your NFS exports are not the roots of their respsective
filesystems?
That's the way it works with NFSv4. You create an NFS root dir with a special entry in /etc/exports:
/nfsserver 10.0.0.0/8(fsid=0,rw,insecure,no_subtree_check,crossmnt)
and if you want other parts exported, create a dir there and mount it:
mkdir /nfsserver/data
mount --bind /mydata /nfsserver/data
This just takes the inodes from there and creates something like a view into this dir. From the client, you
mount -t nfs4 server:/data /serverdata
and you get the /nfsserver/data mounted there.
Quoted 4 lines
This means NFSD uses non-standard filesystem IDs in the filehandles which have to encode the inode number of the export root. Your best option is to simplify switch to exporting a whole filesystem,
That's how it worked until NFSv3. I tried that also, didn't work either. The whole thing worked before I reinstalled it (this server now runs virtualized within XENserver), and maybe that made some subdirs have inodes > 32bit.
Quoted 2 lines
alternatively you can try making sure NFSD uses the 16byte wide UUID style export.
How'd I do that?
Quoted 4 lines
Note thast either way will only work
with a 64bit kernel as the fs has no say in encoding the filesystem part of the handle andino_tis always 32bit on 32bit platforms. This will also affect any other filesystem with 64bit inode numbers.
Not my problem, everything 64bit here. I use the kernel-nfsd, and that's why I was wondering to have this problem.
mfg zmi
Signature · 5 lines
// Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0660 / 415 65 31 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Michael Monnerie
Quoted 3 lines
filehandles which have to encode the inode number of the export root. Your best option is to simplify switch to exporting a whole filesystem,
That's how it worked until NFSv3. I tried that also, didn't work either. The whole thing worked before I reinstalled it (this server now runs virtualized within XENserver), and maybe that made some subdirs have inodes > 32bit.
Quoted 4 lines
Quoted 2 lines
alternatively you can try making sure NFSD uses the 16byte wide UUID style export.
How'd I do that?
I'm not sure, haven't deal with nfs utils a lot recently. I'd suggest you brings this up on [email protected].
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs