the infamous" device is busy" bug

NFS Maurice Volaski 3 participants 4 messages Jun 13, 2008 – Jun 16, 2008 2 messages not in the archive

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Maurice Volaski

author

2008-06-13 23:22 UTC #1 lore raw

I'm running 2.6.25-gentoo-r1 on an AMD 64 box along with nfs-utils-1.1.2-r1 and I have /var/lib/nfs symlinked to a separate filesystem, /database/nfs

Now the system is running drbd and heartbeat and /database is on drbd, so that if the server fails, the secondary server will be able to immediately take over nfs using the identical data on its /database/nfs.

If, however, I want to manually stop heartbeat on the primary server, I cannot because /database will not umount, complaining "the device is busy". I've tried to manually by shutting down all things nfs, but something is still holding it, and nfsd in the kernel is the culprit. I tested that by loading the module and from that point forward, trying to unmount /database just results in the "device is busy" complaint. That means the only way I can shutdown heartbeat is to physically shut down the server.

Signature · 16 lines

Maurice Volaski, mvolaski-Q33DkHiyi7z2fBVCVOL8/[email protected]
Computing Support, Rose F. Kennedy Center
Albert Einstein College of Medicine of Yeshiva University

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

NeilBrown

2008-06-14 00:59 UTC #2 lore raw

Quoted 9 lines · Maurice Volaski

I'm running 2.6.25-gentoo-r1 on an AMD 64 box along with nfs-utils-1.1.2-r1 and I have /var/lib/nfs symlinked to a separate filesystem, /database/nfs
Now the system is running drbd and heartbeat and /database is on drbd, so that if the server fails, the secondary server will be able to immediately take over nfs using the identical data on its /database/nfs.

Quoted 8 lines

If, however, I want to manually stop heartbeat on the primary server, I cannot because /database will not umount, complaining "the device is busy". I've tried to manually by shutting down all things nfs, but something is still holding it, and nfsd in the kernel is the culprit. I tested that by loading the module and from that point forward, trying to unmount /database just results in the "device is busy" complaint. That means the only way I can shutdown heartbeat is to physically shut down the server.

If nfsd is keeping the filesystem active, then
exportfs -f
should release it, though if there are active clients, one could grab it again very quickly.
Alternately
rpc.nfsd 0
exportfs -f
umount
rpc.nfsd 16

NeilBrown

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

A message that is not in the archive (sent off-list or not yet received): 53236.192.168.1.70.1213405151.squirrel-eq65iwfR9nKIECXXMXunQA@public.gmane.org

Maurice Volaski

author

2008-06-14 05:20 UTC #4 lore raw

Quoted 7 lines

should release it, though if there are active clients, one could grab it again very quickly.
Alternately
rpc.nfsd 0
exportfs -f
umount
rpc.nfsd 16

The rpc.nfsd 0 alone stopped this complaint and allowed the umount to succeed. Thanks. (Do I really want the last line "rpc.nfsd 16"? After all, I'm trying to turn nfs off.)

It sounds as if there is a bug here, but it's Gentoo nfs init script that's at fault because presumably it should be executing "rpc.nfsd "0 and "exportfs -f" in response to "/etc/init.d/nfs stop".

Signature · 16 lines

Maurice Volaski, mvolaski-Q33DkHiyi7z2fBVCVOL8/[email protected]
Computing Support, Rose F. Kennedy Center
Albert Einstein College of Medicine of Yeshiva University

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

J. Bruce Fields

2008-06-16 18:42 UTC #5 lore raw

Quoted 8 lines · Maurice Volaski

Quoted 7 lines
should release it, though if there are active clients, one could grab it again very quickly.
Alternately
rpc.nfsd 0
exportfs -f
umount
rpc.nfsd 16

Quoted 5 lines

The rpc.nfsd 0 alone stopped this complaint and allowed the umount to succeed. Thanks. (Do I really want the last line "rpc.nfsd 16"? After all, I'm trying to turn nfs off.)
It sounds as if there is a bug here,

It may be the nfsv4 reboot recovey code holding a reference to /var/lib/nfs/v4recovery/.

Quoted 3 lines

but it's Gentoo nfs init script
that's at fault because presumably it should be executing "rpc.nfsd "0 and "exportfs -f" in response to "/etc/init.d/nfs stop".

I'd think it should be, yes.

--b.