Skip to contentSkip to conversation

rpc.mountd reads /etc/mtab 17028 times, 100% CPU.

DISCUSSION Sander Smeenk 2 participants 3 messages Mar 25, 2015 – Mar 27, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I'm running a server that has 1500+ mounted local filesystems. All of these local filesystems are exported through NFS by this server.

When an NFS-client tries to access one of these exported filesystems, for example doing a simple 'ls' in a mounted NFS, rpc.mountd on the server takes 100% CPU and starts to spin over reading /etc/mtab which is linked to /proc/mounts and totals to about 200KB in size. The client stalls all the while rpc.mountd is busy reading /etc/mtab for every filesystem mounted, and then some more.

I made a strace of rpc.mountd during such a run to gather some stats:

# cut -d'(' -f1 straceout.25103 | sort | uniq -c | sort -n
      3 fadvise64
      6 ioctl
     11 lseek
     33 readlink
     43 select
     53 write
    809 statfs
    814 lstat
   1282 stat
  17043 mmap
  17043 munmap
  17088 fstat
  17895 open
  21728 openat
  37295 close
  40352 getdents
 458616 newfstatat
3763233 read
# grep -c "open.*mtab" straceout.25103 
17028

This entire process of reading /etc/mtab 17028 times takes a lot of time during which the client stalls, but in the end 'it just works fine'. It just takes ages when you try to tab-complete on a client.

What would be needed to debug and optimise this?
Could someone point me to the code that is involved in doing this? I'm not a good C-coder but perhaps i can debug more...

Any insights appreciated!

With regards,
-Sander.

Signature · 2 lines
| Happiness isn't enough for me. I demand EUPHORIA!
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7  FBD6 F3A9 9442 20CC 6CD2
#1 lore raw
Quoted 10 lines · Sander Smeenk

I'm running a server that has 1500+ mounted local filesystems. All of these local filesystems are exported through NFS by this server.

When an NFS-client tries to access one of these exported filesystems, for example doing a simple 'ls' in a mounted NFS, rpc.mountd on the server takes 100% CPU and starts to spin over reading /etc/mtab which is linked to /proc/mounts and totals to about 200KB in size. The client stalls all the while rpc.mountd is busy reading /etc/mtab for every filesystem mounted, and then some more.

Quoted 3 lines

This entire process of reading /etc/mtab 17028 times takes a lot of time during which the client stalls, but in the end 'it just works fine'. It just takes ages when you try to tab-complete on a client.

Taking a quick look... it appears the only way rpc.mountd will read /etc/mtab is if the 'crossmnt' export flag is set. Is this the case?

Quoted 2 lines

What would be needed to debug and optimise this?

Maybe reorder you exports so the 'hot' file systems are on the top of the list...

Quoted 1 line

Could someone point me to the code that is involved in doing this?

utils/mountd/cache.c:nfsd_fh():path = next_mnt(&mnt, exp->m_export.e_path);

steved.

#2 lore raw
Quoted 5 lines · Steve Dickson
Quoted 3 lines

This entire process of reading /etc/mtab 17028 times takes a lot of time during which the client stalls, but in the end 'it just works fine'. It just takes ages when you try to tab-complete on a client.

Taking a quick look... it appears the only way rpc.mountd will read /etc/mtab is if the 'crossmnt' export flag is set. Is this the case?

Yes.

IIRC this was necessary for this situation:
Server has filesytems mounted on /mnt/pt1 and /mnt/pt1/foo. Both /mnt/pt1 and /mnt/pt1/foo are exported with 'crossmnt'. Clients that mount just /mnt/pt1 can also reach /mnt/pt1/foo.

Quoted 2 lines
Quoted 1 line

Could someone point me to the code that is involved in doing this?

utils/mountd/cache.c:nfsd_fh():path = next_mnt(&mnt, exp->m_export.e_path);

Thanks. I'll dig around some more and see what i can find.

-Sndr.

Signature · 2 lines
| What are the little plastic endings on your shoelaces called?
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7  FBD6 F3A9 9442 20CC 6CD2