Skip to contentSkip to conversation

XFS, NFS and inode64 on 2.6.27

DISCUSSION Michael Monnerie 3 participants 4 messages Nov 23, 2009 – Nov 25, 2009 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

(After sending this to the XFS list, I was suggested by Christoph Hellwig to ask on this list. I'm not subscribed here, so please CC me)

Dear all,

after searching for a long time I guess I found that inode64 on XFS with kernel 2.6.27 (shipped with openSUSE 11.1) is incompatible with NFS. Sorry if this was already know, it wasn't for me and finding the problem was complicated.

This is my nfs4 root (fsid=0):

# ls -li
insgesamt 0
4666 drwxr-xr-x 2 root root 48 15. Nov 19:54 daten
4668 drwxr-xr-x 2 root root 48 15. Nov 19:54 download2
4667 drwxr-xr-x 2 root root 48 15. Nov 19:54 dvd-images
4661 drwxr-xr-x 2 root root 48 19. Nov 06:51 mailsrv-backup
4669 drwxr-xr-x 2 root root 48 15. Nov 19:54 q
4670 drwxr-xr-x 2 root root 48 15. Nov 19:54 z

Then I mount those subdirs (mount --bind ....) to fill it with the dirs I want to share:

# ls -li
insgesamt 20
        256 drwxr-xr-x 14 root root  4096 19. Nov 03:01 daten
 6454034730 drwxrwx---  3 zmi  users   61 26. Mär 2009  download2
    4070671 drwxr-xr-x  2 root root    76 13. Nov 12:17 dvd-images
 6500330752 drwxr-xr-x  5 root root  4096  8. Nov 04:31 mailsrv-backup
 8591091465 drwxr-xr-x 17 root root  4096 12. Nov 02:01 q
 2147483904 drwxrwx--- 31 zmi  bh    4096  7. Nov 09:58 z

The shares "daten" and "dvd-images" can be mounted from other servers. I simply went to the original dirs of this mount-bind, and created several new dirs:
mkdir 1 2 3 4 5 6 7 8 9
and one of them had an inode < 2G, so I moved the contents there and renamed the dirs, remounted the --bind mounts and now have this:

# ls -li
insgesamt 20
      256 drwxr-xr-x 14 root root 4096 19. Nov 07:22 daten
184637244 drwxr-xr-x  3 root root   61 19. Nov 07:30 download2
  4070671 drwxr-xr-x  2 root root   76 13. Nov 12:17 dvd-images
960533297 drwxr-xr-x  5 root root 4096 19. Nov 07:27 mailsrv-backup
184190222 drwxr-xr-x 16 root root 4096 19. Nov 07:41 q
184637237 drwxr-xr-x 30 root root 4096 19. Nov 07:42 z

And everything mounts fine now.

(I still have one server that can mount, but not write, to such a share, but that may be another problem)

Just to let others know that can be a problem.

mfg zmi

Signature · 5 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4
#2 lore raw
In reply to a message not in the archive
Quoted 1 line · Michael Monnerie

after searching for a long time I guess I found that inode64 on XFS w=

ith=20

Quoted 2 lines

kernel 2.6.27 (shipped with openSUSE 11.1) is incompatible with NFS.=20 Sorry if this was already know, it wasn't for me and finding the prob=

lem=20

Quoted 4 lines

was complicated.
=20
This is my nfs4 root (fsid=3D0):
=20

Quoted 10 lines
# ls -li
insgesamt 0
4666 drwxr-xr-x 2 root root 48 15. Nov 19:54 daten
4668 drwxr-xr-x 2 root root 48 15. Nov 19:54 download2
4667 drwxr-xr-x 2 root root 48 15. Nov 19:54 dvd-images
4661 drwxr-xr-x 2 root root 48 19. Nov 06:51 mailsrv-backup
4669 drwxr-xr-x 2 root root 48 15. Nov 19:54 q
4670 drwxr-xr-x 2 root root 48 15. Nov 19:54 z
=20
Then I mount those subdirs (mount --bind ....) to fill it with the di=

rs=20

Quoted 6 lines

I want to share:
=20

# ls -li
insgesamt 20
        256 drwxr-xr-x 14 root root  4096 19. Nov 03:01 daten
 6454034730 drwxrwx---  3 zmi  users   61 26. M=C3=A4r 2009  download=

2

Quoted 2 lines
    4070671 drwxr-xr-x  2 root root    76 13. Nov 12:17 dvd-images

6500330752 drwxr-xr-x 5 root root 4096 8. Nov 04:31 mailsrv-backu=

p

Quoted 4 lines

8591091465 drwxr-xr-x 17 root root 4096 12. Nov 02:01 q
2147483904 drwxrwx--- 31 zmi bh 4096 7. Nov 09:58 z
=20
The shares "daten" and "dvd-images" can be mounted from other servers=

=2E

But not the other four directories? What error do you get when you try= ?
What client are you using, and are you mounting with NFSv2, NFSv3, or NFSv4? Could we see a network trace of a failed mount? (So, run

tcpdump -s0 -wtmp.pcap

then attempt to mount one of the directories with a too-big inode number, then kill tcpdump and send us tmp.pcap.)

--b.

#3 lore raw
Quoted 5 lines · J. Bruce Fields

But not the other four directories? What error do you get when you try? What client are you using, and are you mounting with NFSv2, NFSv3, or NFSv4? Could we see a network trace of a failed mount? (So, run

From information Michael posted on the xfs list he is not exporting the

root of filesystems, which means we use a file system handle format that needs to encode the inode. And all but the uuid16 format can only encode 32 bit inode numbers. I very strongly suspect that this is the underlying issue here, but I'm currenly not uptodate on the filesystem handle selection in nfsutils.

#4 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · J. Bruce Fields

But not the other four directories? What error do you get when you try? What client are you using, and are you mounting with NFSv2, NFSv3, or NFSv4? Could we see a network trace of a failed mount? (So, run

From information Michael posted on the xfs list he is not exporting the

root of filesystems, which means we use a file system handle format that needs to encode the inode. And all but the uuid16 format can only encode 32 bit inode numbers. I very strongly suspect that this is the underlying issue here, but I'm currenly not uptodate on the filesystem handle selection in nfsutils.

If using an nfs-utils recent enough to pass down a uuid (can check this by looking at /proc/net/rpc/nfsd.export/filehandle after trying to access those directories, and seeing whether there's a uuid= option set), then I'd expect it to be using FSID_UUID16_INUM with v3 or v4 (v2 doesn't have large enough filehandles).

--b.