the remount problem [2.4.0] kind of solved [patch]
Hello,
the following patch against 2.4.0 will allow the kernel to write a message to the kernel log in case files are open for write or delete on a partition which should be remounted.
I run my System with Read-Only /usr File System and this works fairly well. I have a script to remount the different Filesystems if I want to upgrade them (using the cool apt-get from Debian).
Sometimes after upgrades I noticed, that I am unable to remount the /usr File system read only. Since I was unable to detect the Reason for it (no file according to lsof or fuser was open for writing) i decided to patch the kernel to debug the problem.
The Solution: some files are deleted but open. Most of the time this is due to an upgrade of a shared lib without restarting the related daemon. I am not sure why I dont see the open but deleted files in lsof (according to its man page "lsof -aL1 /usr" should work, but anyway. Here is my patch, in case you experience the same problems. Not sure if it is ok to include it that way into mainstream kernel since it might produce quite a few lines of logs and i am not sure if prinkt is save inside the file list lock anyway.
So my question: which user mode program can be used to detect those "open but deleted" mmaped files?
With my patch i get the inode an can grep in maps...
calista:/usr/src/linux# grep 145429 /proc/*/maps /proc/354/maps:40156000-4016d000 r-xp 00000000 08:05 145429 /usr/lib/jabber/jsm/jsm.so (deleted) /proc/354/maps:4016d000-4016e000 rw-p 00016000 08:05 145429 /usr/lib/jabber/jsm/jsm.so (deleted) /proc/366/maps:40156000-4016d000 r-xp 00000000 08:05 145429 /usr/lib/jabber/jsm/jsm.so (deleted) /proc/366/maps:4016d000-4016e000 rw-p 00016000 08:05 145429 /usr/lib/jabber/jsm/jsm.so (deleted)
Perhaps it is enough to run "grep '(deleted)' /proc/*/maps" in cron.daily?
Well, for Debian, it would be nice if we can make sure that on shared lib upgrade at least a list of programs which needs to be restarted is mailed to root. We do restartes on libc upgrade, but I guess it is not possible to restart for other shared libs (automatically).
Greetings
Bernd
usr/src/linux/fs/file_table.c Sun Jan 21 12:40:07 2001 renamed from usr/src/linux/fs/file_table.corg Sun Jan 21 12:29:04 2001 +6 -3
@@ -182,12 +182,15 @@ inode = file->f_dentry->d_inode; /* File with pending delete? */ - if (inode->i_nlink == 0) + if (inode->i_nlink == 0) { + printk("ro-remount failed: pending delete on %s, inode no %lu\n",kdevname(inode->i_dev),inode->i_ino); goto too_bad; - + } /* Writable file? */ - if (S_ISREG(inode->i_mode) && (file->f_mode & FMODE_WRITE)) + if (S_ISREG(inode->i_mode) && (file->f_mode & FMODE_WRITE)) { + printk("ro-remount failed: file open for write on %s, inode no %lu\n",kdevname(inode->i_dev),inode->i_ino); goto too_bad; + } } file_list_unlock(); return 1; /* Tis' cool bro. */
Greetings
Bernd
Signature · 8 lines
(OO) -- [email protected] -- ( .. ) ecki@{inka.de,linux.de,debian.org} http://home.pages.de/~eckes/ o--o *plush* 2048/93600EFD eckes@irc +497257930613 BE5-RIPE (O____O) When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl! - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Bernd Eckenfels <[email protected]> writes:
> Hello, the following patch against 2.4.0 will allow the kernel > to write a message to the kernel log in case files are open for > write or delete on a partition which should be remounted.
> I run my System with Read-Only /usr File System and this works > fairly well. I have a script to remount the different > Filesystems if I want to upgrade them (using the cool apt-get > from Debian).
> Sometimes after upgrades I noticed, that I am unable to remount > the /usr File system read only. Since I was unable to detect > the Reason for it (no file according to lsof or fuser was open > for writing) i decided to patch the kernel to debug the > problem.
> The Solution: some files are deleted but open. Most of the time > this is due to an upgrade of a shared lib without restarting > the related daemon. I am not sure why I dont see the open but > deleted files in lsof (according to its man page "lsof -aL1 > /usr" should work, but anyway. Here is my patch, in case you > experience the same problems. Not sure if it is ok to include > it that way into mainstream kernel since it might produce quite > a few lines of logs and i am not sure if prinkt is save inside > the file list lock anyway.
Make it optional. Add an entry in proc that turns the feature on and of and an config option disabling the code alltogether. Then send the patch in and see what they say.
> So my question: which user mode program can be used to detect > those "open but deleted" mmaped files?
> With my patch i get the inode an can grep in maps...
> calista:/usr/src/linux# grep 145429 /proc/*/maps > /proc/354/maps:40156000-4016d000 r-xp 00000000 08:05 145429 > /usr/lib/jabber/jsm/jsm.so (deleted)
> /proc/354/maps:4016d000-4016e000 rw-p 00016000 08:05 145429 > /usr/lib/jabber/jsm/jsm.so (deleted)
> /proc/366/maps:40156000-4016d000 r-xp 00000000 08:05 145429 > /usr/lib/jabber/jsm/jsm.so (deleted)
> /proc/366/maps:4016d000-4016e000 rw-p 00016000 08:05 145429 > /usr/lib/jabber/jsm/jsm.so (deleted)
Why in hell are library open for write? But it doesn't seem to be only libraries:
% cat /proc/self/maps
08048000-0804b000 r-xp 00000000 03:01 22109 /bin/cat 0804b000-0804c000 rw-p 00002000 03:01 22109 /bin/cat 40000000-40016000 r-xp 00000000 03:01 18160 /lib/ld-2.2.1.so 40016000-40017000 rw-p 00015000 03:01 18160 /lib/ld-2.2.1.so
40017000-40018000 rw-p 00000000 00:00 0
4001f000-40120000 r-xp 00000000 03:01 18171 /lib/libc-2.2.1.so
40120000-40127000 rw-p 00100000 03:01 18171 /lib/libc-2.2.1.so
40127000-4012b000 rw-p 00000000 00:00 0
bfffe000-c0000000 rwxp fffff000 00:00 0
> Perhaps it is enough to run "grep '(deleted)' /proc/*/maps" in > cron.daily?
> Well, for Debian, it would be nice if we can make sure that on > shared lib upgrade at least a list of programs which needs to > be restarted is mailed to root. We do restartes on libc > upgrade, but I guess it is not possible to restart for other > shared libs (automatically).
At boottime the filesystems are readonly, so any deamon that gets startet can only have read and exec permissions on files. Deleting a library and replacing it with a new one can't change that.
I think the problem comes from daemons that actually get restarted, maybe before the library is updated (so they will load the old/deleted one).
MfG
Goswin
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Hello,
for Short: I had a mail exchange with Vic Abell, the lsof Author, and in the next Version of lsof the open shared libs will be detected. So my Kernel Patch is no longer needed:
# ~root/rw
# rm /usr/lib/jabber/jsm/libjsm.so # ~root/ro mount: /usr busy # lsof_4.55A.linux/lsof -a +L1 /usr COMMAND PID USER FD TYPE DEVICE SIZE NLINK NODE NAME jabberd 9657 daemon mem DEL 8,5 0 145429 /usr/lib/jabber/jsm/jsm.so jabberd 9658 daemon mem DEL 8,5 0 145429 /usr/lib/jabber/jsm/jsm.so #
Still we need to address the problem of upgraded libs (at least in a Great Distribution like Debian is :)
Quoted 2 lines
Why in hell are library open for write? But it doesn't seem to be only libraries:
They are not open for write. They are open for mmaped read. The Problem with this is, that as long as the files are open, the filesystem cannot remove them from disk. This means, that as long as you have files open, even for read, which are deleted, a remount ro will fail.
The new lsof will find those mmaped files, so you can simply restart the associated binary.
Quoted 3 lines
At boottime the filesystems are readonly, so any deamon that gets startet can only have read and exec permissions on files. Deleting a library and replacing it with a new one can't change that.
Deleteing a lib which is open (which will happen if u upgrade a lib*.deb) will leave a open-but-delted file on the system, which in turn will disable you to remount the file system read only. In my case i switch from read-only /usr to read-write user, use apt-get -ufm upgrade and then want to switch back to read-only /usr. But exactly the later is not possible. It is also wasting shared memory since freshly started programs will use the new lib, the old ones will use the old, unlinked libs.
Quoted 3 lines
I think the problem comes from daemons that actually get restarted, maybe before the library is updated (so they will load the old/deleted one).
Yes, all daemons will get started with the old libs, since a upgrade always happens after system start :) But not only daemons. Think of Shells, getty, login, ...
Greetings
Bernd
Signature · 8 lines
(OO) -- [email protected] -- ( .. ) ecki@{inka.de,linux.de,debian.org} http://home.pages.de/~eckes/ o--o *plush* 2048/93600EFD eckes@irc +497257930613 BE5-RIPE (O____O) When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl! - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Bernd Eckenfels <[email protected]> writes:
Why in hell are library open for write? But it doesn't seem to be only libraries:
> They are not open for write. They are open for mmaped read. The > Problem with this is, that as long as the files are open, the > filesystem cannot remove them from disk. This means, that as > long as you have files open, even for read, which are deleted, > a remount ro will fail.
> The new lsof will find those mmaped files, so you can simply > restart the associated binary.
Ah, I see. Forgot about that.
Maybe the kernel coud swap in the deleted libraries and keep it in memory or real swap from then on instead of blocking the fs.
> Yes, all daemons will get started with the old libs, since a > upgrade always happens after system start :) But not only > daemons. Think of Shells, getty, login, ...
So, when you update glibc, / can't be remounted ro any more? Ugly.
MfG
Goswin
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 2 lines · Previously Goswin Brederlow
Maybe the kernel coud swap in the deleted libraries and keep it in memory or real swap from then on instead of blocking the fs.
No, you have no idea how large the file might grow and you need to keep that data somewhere.
Wichert.
Signature · 8 lines
_________________________________________________________________ / Nothing is fool-proof to a sufficiently talented fool \ | [email protected] http://www.liacs.nl/~wichert/ | | 1024D/2FA3BC2D 576E 100B 518D 2F16 36B0 2805 3CB8 9250 2FA3 BC2D | - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Wichert Akkerman
Quoted 2 lines · Previously Goswin Brederlow
Maybe the kernel coud swap in the deleted libraries and keep it in memory or real swap from then on instead of blocking the fs.
No, you have no idea how large the file might grow and you need to keep that data somewhere.
Grow? It makes little difference, since you can run out of filesystem space too.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/