Reading libs fails through NFS
Hi all...
Anybody has any idea about why this fails:
fd = open("/lib/libnss_files.so.2", O_RDONLY);
res = read(fd,buf,512);/lib is NFS mounted:
192.168.0.1:/lib on /lib type nfs (ro,noatime,nfsvers=3,nolock,addr=192.168.0.1)
and the read fails.
The original code does a getprotobyname("tcp") (netpipe), that fails when it tries to read the same lib.
The node boots via PXE, with a version of libnss_files.so.2 on the /lib present in the initrd, which is replaced by the mounted one.
TIA
Signature · 4 lines
J.A. Magallon <jamagallon()able!es> \ Software is like sex: werewolf!able!es \ It's better when it's free Mandrake Linux release 10.0 (Cooker) for i586 Linux 2.4.23-rc1-jam2 (gcc 3.3.1 (Mandrake Linux 9.2 3.3.1-4mdk))
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == J A Magallon <J.A.> writes:
> Hi all... Anybody has any idea about why this fails:
> fd = open("/lib/libnss_files.so.2", O_RDONLY); res =
> read(fd,buf,512);No. Nobody else will be able to tell you either until you tell us what setup you are using.
Cheers,
Trond
Quoted 10 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == J A Magallon <J.A.> writes:
> Hi all... Anybody has any idea about why this fails: > fd = open("/lib/libnss_files.so.2", O_RDONLY); res = > read(fd,buf,512);No. Nobody else will be able to tell you either until you tell us what setup you are using.
I run a small bproc cluster. Nodes are diskless, boot with a custom initrd, and execute a simple linuxrc (comments and echo's stripped):
PATH=/bin:/sbin:/usr/bin:/usr/sbin
mount -n -o remount,rw /
mount -t proc none /proc
mount -t devpts -omode=0620 none /dev/pts
mount -t tmpfs none /dev/shm
mount -t tmpfs none /tmp
ifconfig lo up 127.0.0.1
modprobe eth0
ifconfig eth0 up
dhcpcd -H -D -R -N eth0
portmap
mount /lib
mount /bin
mount /sbin
mount /usr
mount /opt
mount /home
mount /work/shared
modprobe eth1
ifconfig eth1 up
dhcpcd -R -N eth1
ntpdate -v 192.168.0.1
modprobe bproc
bpslave -v -d -r 192.168.1.1
fstab for nodes (in /etc in initrd) is:
rootfs / rootfs defaults 0 0
none /proc proc defaults 0 0
none /dev/pts devpts mode=0620 0 0
none /dev/shm tmpfs defaults 0 0
none /tmp tmpfs defaults 0 0
192.168.0.1:/lib /lib nfs nfsvers=3,ro,noac,suid 192.168.0.1:/bin /bin nfs nfsvers=3,ro,noac,suid 192.168.0.1:/sbin /sbin nfs nfsvers=3,ro,noac,suid 192.168.0.1:/usr /usr nfs nfsvers=3,ro,noac,suid 192.168.0.1:/opt /opt nfs nfsvers=3,ro,noac,suid 192.168.0.1:/home /home nfs nfsvers=3,rw 192.168.0.1:/work /work/shared nfs nfsvers=3,rw
For example, /opt is just the mount point in initrd, so it is empty. It is ro, just soft to use:
annwn:/opt> pwd
/opt
annwn:/opt> ls
aleph/ coin/ intel/ mpich/
annwn:/opt> bpsh 0 pwd
/opt
annwn:/opt> bpsh 0 ls
ls: reading directory .: Invalid argument annwn:~> bpsh 0 ls /opt/*
ls: reading directory /opt/aleph: Invalid argument /opt/aleph:
ls: reading directory /opt/coin: Invalid argument
/opt/coin:
ls: reading directory /opt/intel: Invalid argument
/opt/intel:
ls:/opt/mpich:
reading directory /opt/mpich: Invalid argument
annwn:~> bpsh 0 strace ls /opt
...stat64("/opt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 open("/opt", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3 fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 getdents64(3, 0x8060360, 8192) = -1 EINVAL (Invalid argument) close(3) = 0
...
annwn:/opt> bpsh 0 strace ls
...open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3 fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 getdents64(3, 0x8060350, 8192) = -1 EINVAL (Invalid argument) close(3) = 0
...
It looks like readdir fails (getdents).
Uh ?
I use a custom kernel, still have to try with plain -rc3 + bproc. But do you have any ideas about what is going/what am I doing wrong ? This all worked some time ago with my hacked -jam kernels.
TIA for you attention.
Signature · 4 lines
J.A. Magallon <jamagallon()able!es> \ Software is like sex: werewolf!able!es \ It's better when it's free Mandrake Linux release 10.0 (Cooker) for i586 Linux 2.4.23-rc3-jam1 (gcc 3.3.1 (Mandrake Linux 9.2 3.3.1-4mdk))
Quoted 4 lines · J.A. Magallon
fd = open("/lib/libnss_files.so.2", O_RDONLY);The node boots via PXE, with a version of libnss_files.so.2 on the /lib present in the initrd, which is replaced by the mounted one.
Just a shot in the dark, but could there be a bug in the NFS code where it's getting upset that there's cached pages of the file in memory, but the file that the cached page is from isn't from the file that NFS can see? (note that this is possibly *different* than a stale NFS handle when a file is unlinked and then recreated - here chasing the origin of the page doesn't point at the NFS mount, but at the initrd mount.
Yes, when you mount over a directory, the previous contents are supposed to become invisible. I wonder if there's a bug with that if a file is read and pages cached 'sufficiently early' in the boot process (i.e. before the real root gets mounted over the initrd root..)