Skip to contentSkip to conversation

block dev minor > 255 and exporting fs

DISCUSSION Ed L Cashin 2 participants 3 messages Oct 5, 2005 – Oct 6, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi. I've noticed that an NFS mount times out when I export a filesystem residing on a block device with a "large" minor number, i.e. beyond the old limit of 255 from when there were only eight bits for the minor number of devices.

If I use a block device with a lower minor number, things work as expected, and if I "wrap" a high-numbered device in a trivial md set, using /dev/md0 with its minor number of zero, things work as expected.

Without initial success I've looked at the kernel sources to see where the nfs server might be using only eight of the twenty bits 2.6 uses for minor numbers. Does anyone know where that might be occuring?

The nfs server in my tests is a debian testing machine running 2.6.12-1-amd64-generic, and the client is a debian stable system running a custom 2.6.13-rc6 kernel, but I've seen this problem on other systems a while ago. At that time I found out that 255 was the magic minor number after which problems started occuring, if I recall correctly. If you don't have block devices with high minor numbers to test with, you can replicate this problem using the vblade:

http://sourceforge.net/projects/aoetools/

... and the aoe driver in any 2.6 kernel from 2.6.11. Anyway, here are the details for interested parties. The nfs server is "makki" and the client is "kokone".

  makki:/home/ecashin# modprobe aoe
  makki:/home/ecashin# ls -l /dev/etherd/e2.1
  brw-rw----  1 root disk 152, 336 2005-10-05 08:24 /dev/etherd/e2.1
  makki:/home/ecashin# mount /dev/etherd/e2.1 /mnt/aoe/e2.1 
  makki:/home/ecashin# grep aoe /etc/exports
  /mnt/aoe/e2.1 *.coraid.com(rw,sync)
  makki:/home/ecashin# 

On the client, mount times out.

root@kokone root# mount -t nfs makki:/mnt/aoe/e2.1 /mnt/makki
mount: makki:/mnt/aoe/e2.1: can't read superblock root@kokone root# tail /var/log/everything
...

  Oct  5 12:27:16 kokone kernel: nfs: server makki not responding, timed out
  Oct  5 12:27:37 kokone last message repeated 2 times
  root@kokone root# 

I can use a trivial one-device linear software RAID on the nfs server so that nfs doesn't see the high minor device number. This is just using a low-minor-number md device as a wrapper for the high-minor-number aoe device.

  makki:/home/ecashin# /etc/init.d/nfs-kernel-server stop && /etc/init.d/nfs-common stop
  Stopping NFS kernel daemon: mountd nfsd.
  Unexporting directories for NFS kernel daemon...done.
  Stopping NFS common utilities: statd.
  makki:/home/ecashin# umount /mnt/aoe/e2.1
  makki:/home/ecashin# ls -l /dev/md0
  brw-rw----  1 root disk 9, 0 2005-10-05 08:40 /dev/md0
  makki:/home/ecashin# mdadm -B --auto=md --force -l linear -n 1 /dev/md0 /dev/etherd/e2.1
  mdadm: array /dev/md0 built and started.
  makki:/home/ecashin# mount /dev/md0 /mnt/aoe/e2.1
  makki:/home/ecashin# ls /mnt/aoe/e2.1
  screen
  makki:/home/ecashin# /etc/init.d/nfs-common start && /etc/init.d/nfs-kernel-server start
  Starting NFS common utilities: statd.
  Exporting directories for NFS kernel daemon...done.
  Starting NFS kernel daemon: nfsd mountd.
  makki:/home/ecashin# 

Then on the client, all goes well:

root@kokone root# mount -t nfs makki:/mnt/aoe/e2.1 /mnt/makki root@kokone root# ls /mnt/makki
screen
root@kokone root# umount /mnt/makki

So I have a nice workaround, but I would rather not need it. Things go well *without* the md wrapper if the aoe device has a minor number below 256. What part of the nfs server doesn't use all twenty bits that 2.6 uses for the device minor number? I remember guessing that it was a handle or tag used in the protocol, but that was a long time ago.

  makki:/home/ecashin# /etc/init.d/nfs-kernel-server stop && /etc/init.d/nfs-common stop
  Stopping NFS kernel daemon: mountd nfsd.
  Unexporting directories for NFS kernel daemon...done.
  Stopping NFS common utilities: statd.
  makki:/home/ecashin# umount /mnt/aoe/e2.1 
  makki:/home/ecashin# mdadm -S /dev/md0
  makki:/home/ecashin# sync
  makki:/home/ecashin# ls -l /dev/etherd/e0.0
  brw-rw----  1 root disk 152, 0 2005-10-05 08:49 /dev/etherd/e0.0
  makki:/home/ecashin# mount /dev/etherd/e0.0 /mnt/aoe/e2.1
  makki:/home/ecashin# /etc/init.d/nfs-common start && /etc/init.d/nfs-kernel-server start
  Starting NFS common utilities: statd.
  Exporting directories for NFS kernel daemon...done.
  Starting NFS kernel daemon: nfsd mountd.
  makki:/home/ecashin# 

root@kokone root# mount -t nfs makki:/mnt/aoe/e2.1 /mnt/makki root@kokone root# ls /mnt/makki
screen
root@kokone root#

Signature · 11 lines
  Ed L Cashin <[email protected]>



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#1 lore raw
Quoted 4 lines · On Wednesday October 5, [email protected] wrote:

Hi. I've noticed that an NFS mount times out when I export a filesystem residing on a block device with a "large" minor number, i.e. beyond the old limit of 255 from when there were only eight bits for the minor number of devices.

Hmmm. it definitely shouldn't do that, but I'm not surprised. knfsd has some fairly old-fashioned ideas about how device number work. I'll put it on my todo list to look at, but in the mean time there is another work around you could use that is slightly less awkward than making an 'md' device.

If you would 'fsid=23', or some other number, as an option in /etc/exports, then it will use that number rather than the device number to identify the filesystem, and so it should work better.

NeilBrown

-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#2 lore raw
Quoted 9 lines · Neil Brown
Quoted 4 lines · On Wednesday October 5, [email protected] wrote:

Hi. I've noticed that an NFS mount times out when I export a filesystem residing on a block device with a "large" minor number, i.e. beyond the old limit of 255 from when there were only eight bits for the minor number of devices.

Hmmm. it definitely shouldn't do that, but I'm not surprised. knfsd has some fairly old-fashioned ideas about how device number work. I'll put it on my todo list to look at,

Excellent! I'd like to hear about what you find.

...

Quoted 7 lines

but in the mean time there is
another work around you could use that is slightly less awkward than making an 'md' device.

If you would 'fsid=23', or some other number, as an option in /etc/exports, then it will use that number rather than the device number to identify the filesystem, and so it should work better.

Well I'll be! That works great and is indeed much more convenient than using md. Thanks for the tip---I hadn't noticed fsid in the exports manpage before.

Signature · 11 lines
  Ed L Cashin <[email protected]>



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs