Skip to contentSkip to conversation

Busy on BLKFLSBUF w/initrd

DISCUSSION Maciek Nowacki 5 participants 17 messages May 23, 2001 – May 25, 2001

lore
Share & tools
lore
#0 lore raw

Hi,

I have continuing problems with getting the initrd ramdisk out of memory once bootup is complete.

This is with recent -ac kernels which have the fix-up posted a few months ago applied.

The sequence is roughly:

- boot via pxelinux, loads up bzImage <1MB and root.romfs.gz ~7MB, expands to about 30.
- mount -n -t tmpfs none tmp
- cp root -> /tmp
- pivot_root -> /tmp
- umount old_root/dev (devfs kernel)
- umount old_root

Then another pivot_root a bit later on before /sbin/init is invoked. The point is to move the initrd to a tmpfs which will reclaim memory in the event that the filesystem cannot be unmounted, due to having mountpoints for loopback host filesystems in this case.

It seems to work fine, and the initrd _does_ unmount with no problems (unmounting old_root/dev does complain about mtab, but works, ls old_root/dev shows empty). However, blockdev --flushbufs /dev/rd/0 fails with busy on BLKFLSBUF once the system is running.

About the only odd thing besides the message about mtab from umount is the kernel notice right before entering /bin/sh on the romfs initrd - it prints out two messages about mounting the ramdisk:

May 22 09:14:30 wintermute kernel: Linux version 2.4.4-ac12 (maciek@wintermute) (gcc version 2.95.4 20010506 (Debian prerelease)) #1 Mon May 21 20:08:36 MDT 2001
[...]
May 22 09:14:31 wintermute kernel: RAMDISK: romfs filesystem found at block 0
May 22 09:14:31 wintermute kernel: RAMDISK: Loading 28216 blocks [1 disk] into ram disk... done.
May 22 09:14:31 wintermute kernel: Freeing initrd memory: 28216k freed
May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
May 22 09:14:31 wintermute kernel: change_root: old root has d_count=7
May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
May 22 09:14:31 wintermute kernel: Trying to unmount old root ... okay

Perhaps they're bumping up the reference count so that it is impossible to free the ramdisk later?

Thanks very much for any help!

Maciek

#1 lore raw
Quoted 9 lines · Maciek Nowacki
May 22 09:14:31 wintermute kernel: RAMDISK: romfs filesystem found at block 0
May 22 09:14:31 wintermute kernel: RAMDISK: Loading 28216 blocks [1 disk] into ram disk... done.
May 22 09:14:31 wintermute kernel: Freeing initrd memory: 28216k freed
May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
May 22 09:14:31 wintermute kernel: change_root: old root has d_count=7
May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
May 22 09:14:31 wintermute kernel: Trying to unmount old root ... okay

At that point /dev/ram0 _is_ killed.

Quoted 2 lines

Perhaps they're bumping up the reference count so that it is impossible to free the ramdisk later?

If you want to keep it until later (i.e. want to destiry it by hands) mkdir /initrd on your final root and old one will be remounted there. Again, "Trying to unmount old root ... okay" means that it already got an equivalent of BKLFLSBUF

I wonder why does it get -EBUSY if you repeat it... Uh-oh...

Folks, who the hell is responsible for rd_inodes[] idiocy?

#2 lore raw
Quoted 1 line · Alexander Viro

Folks, who the hell is responsible for rd_inodes[] idiocy?

That would have been me. It was simple and needed at the time.. feel free to rip it up :)

-Mike

#3 lore raw
Quoted 6 lines · Mike Galbraith
Quoted 1 line · Alexander Viro

Folks, who the hell is responsible for rd_inodes[] idiocy?

That would have been me. It was simple and needed at the time.. feel free to rip it up :)

Mike, I see what you are using it for, but you do realize that it means that creating /tmp/ram0 and opening it once will make /tmp impossible to unmount?

#4 lore raw
Quoted 4 lines · Alexander Viro
Quoted 2 lines · Mike Galbraith

On Wed, 23 May 2001, Alexander Viro wrote:

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Folks, who the hell is responsible for rd_inodes[] idiocy?

That would have been me. It was simple and needed at the time.. feel free to rip it up :)

Mike, I see what you are using it for, but you do realize that it means that creating /tmp/ram0 and opening it once will make /tmp impossible to unmount?

I don't _think_ that was the case at the time I did it. I tested the idio^Wbandaid before submission.. mighta fscked up though :)

-Mike

#5 lore raw
Quoted 11 lines · Mike Galbraith
Quoted 9 lines · Alexander Viro
Quoted 6 lines · Mike Galbraith
Quoted 1 line · Alexander Viro

Folks, who the hell is responsible for rd_inodes[] idiocy?

That would have been me. It was simple and needed at the time.. feel free to rip it up :)

Quoted 6 lines
Quoted 3 lines

Mike, I see what you are using it for, but you do realize that it means that creating /tmp/ram0 and opening it once will make /tmp impossible to unmount?

I don't _think_ that was the case at the time I did it. I tested the idio^Wbandaid before submission.. mighta fscked up though :)

Erm... You pin the inode down. That makes filesystem busy by any definition I can think of...

#6 lore raw
Quoted 2 lines · Alexander Viro

Erm... You pin the inode down. That makes filesystem busy by any definition I can think of...

That's just because the 2.4.4 code doesn't release it with a blkdev_put. The fix is in the ac patches.

Signature · 4 lines
Debian GNU/Linux 2.2 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#7 lore raw
In reply to Alexander Viro
Quoted 17 lines · Alexander Viro
Quoted 15 lines · Mike Galbraith
Quoted 12 lines · Alexander Viro
Quoted 6 lines · Mike Galbraith
Quoted 1 line · Alexander Viro

Folks, who the hell is responsible for rd_inodes[] idiocy?

That would have been me. It was simple and needed at the time.. feel free to rip it up :)

Mike, I see what you are using it for, but you do realize that it means that creating /tmp/ram0 and opening it once will make /tmp impossible to unmount?

Quoted 5 lines
Quoted 2 lines

I don't _think_ that was the case at the time I did it. I tested the idio^Wbandaid before submission.. mighta fscked up though :)

Erm... You pin the inode down. That makes filesystem busy by any definition I can think of...

Yes. I pulled the pins when I was done with them though. (at least I think I did.. been a long time)

-Mike

#8 lore raw
Quoted 5 lines · Mike Galbraith
Quoted 2 lines

Erm... You pin the inode down. That makes filesystem busy by any definition I can think of...

Yes. I pulled the pins when I was done with them though. (at least I think I did.. been a long time)

Only when you do rd_cleanup().

#9 lore raw
In reply to Maciek Nowacki
Quoted 14 lines · Alexander Viro
Quoted 9 lines · Maciek Nowacki
May 22 09:14:31 wintermute kernel: RAMDISK: romfs filesystem found at block 0
May 22 09:14:31 wintermute kernel: RAMDISK: Loading 28216 blocks [1 disk] into ram disk... done.
May 22 09:14:31 wintermute kernel: Freeing initrd memory: 28216k freed
May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
May 22 09:14:31 wintermute kernel: VFS: Mounted root (romfs filesystem) readonly.
May 22 09:14:31 wintermute kernel: change_root: old root has d_count=7
May 22 09:14:31 wintermute kernel: Mounted devfs on /dev
May 22 09:14:31 wintermute kernel: Trying to unmount old root ... okay
Quoted 1 line

At that point /dev/ram0 _is_ killed.

Really? Because I can still mount it later on without any trouble at all.. after init has started. Hmm, I will try later after running the system for a while to see if the data is still comprehensible.

Quoted 7 lines
Quoted 2 lines

Perhaps they're bumping up the reference count so that it is impossible to free the ramdisk later?

If you want to keep it until later (i.e. want to destiry it by hands) mkdir /initrd on your final root and old one will be remounted there. Again, "Trying to unmount old root ... okay" means that it already got an equivalent of BKLFLSBUF

Ah, okay.. I assumed this behavior had been removed. I will try this as well.

Maciek

#10 lore raw
Quoted 6 lines · Maciek Nowacki
Quoted 4 lines

If you want to keep it until later (i.e. want to destiry it by hands) mkdir /initrd on your final root and old one will be remounted there. Again, "Trying to unmount old root ... okay" means that it already got an equivalent of BKLFLSBUF

Ah, okay.. I assumed this behavior had been removed. I will try this as well.

change_root() in 2.4.4 gives you explicit destroy_buffers(). In 2.4.5-pre5 it simply does BLKFLSBUF - calls ioctl_by_bdev(). And BLKFLSBUF boils down to destroy_buffers().

I would really like to hear details re survival of the initrd contents. I've looked at the way rd.c "protects" the data and it seems to be b0rken - playing games with igrab() is not a good idea for driver...

#11 lore raw
In reply to Maciek Nowacki
Quoted 11 lines · Alexander Viro
Quoted 6 lines · Maciek Nowacki
Quoted 4 lines

If you want to keep it until later (i.e. want to destiry it by hands) mkdir /initrd on your final root and old one will be remounted there. Again, "Trying to unmount old root ... okay" means that it already got an equivalent of BKLFLSBUF

Ah, okay.. I assumed this behavior had been removed. I will try this as well.

Quoted 7 lines

change_root() in 2.4.4 gives you explicit destroy_buffers(). In 2.4.5-pre5 it simply does BLKFLSBUF - calls ioctl_by_bdev(). And BLKFLSBUF boils down to destroy_buffers().

I would really like to hear details re survival of the initrd contents. I've looked at the way rd.c "protects" the data and it seems to be b0rken - playing games with igrab() is not a good idea for driver...

I wrote out the contents of /dev/rd/0 a few times and diff'ed with the uncompressed image of the initrd on the server. No difference each time. The same after digging into swap, turning off swap, running blockdev --flushbufs several times (always with BLKFLSBUF: Device or resource busy).

The next test will be to create an initrd that has the 'initrd' directory..

(this is all with 2.4.4-ac13 +crypto-2.4.3.1 +alsa-cvs)

Maciek

#12 lore raw
Quoted 7 lines · Maciek Nowacki

On Wed, May 23, 2001 at 06:21:23PM -0400, Alexander Viro wrote:
I wrote out the contents of /dev/rd/0 a few times and diff'ed with the uncompressed image of the initrd on the server. No difference each time. The same after digging into swap, turning off swap, running blockdev --flushbufs several times (always with BLKFLSBUF: Device or resource busy).

The next test will be to create an initrd that has the 'initrd' directory..

Not initrd with /initrd in it, final root with /initrd, so that change_root() had a place to remount the thing on.

#13 lore raw
In reply to Maciek Nowacki
Quoted 10 lines · Alexander Viro
Quoted 6 lines · Maciek Nowacki

On Wed, May 23, 2001 at 06:21:23PM -0400, Alexander Viro wrote:
I wrote out the contents of /dev/rd/0 a few times and diff'ed with the uncompressed image of the initrd on the server. No difference each time. The same after digging into swap, turning off swap, running blockdev --flushbufs several times (always with BLKFLSBUF: Device or resource busy).

Quoted 4 lines
Quoted 1 line

The next test will be to create an initrd that has the 'initrd' directory..

Not initrd with /initrd in it, final root with /initrd, so that change_root() had a place to remount the thing on.

Yeah, but.. I get the message about change_root() before I have a chance to do anything (there is no /linuxrc on my initrd). I could try with a linuxrc, but I wanted to avoid that since since it seemed to be becoming obsolete.

Maciek

#14 lore raw
In reply to Maciek Nowacki

Problem seems to be solved. Here is what I did, for anyone who is interested in using a loopback file on a local disk as root:

- create a romfs initrd with /linuxrc -> /bin/sh and an empty /initrd dir

- boot, with root=/dev/rd/1 init=/bin/sh

- when /linuxrc comes up, cat /dev/rd/0 > /dev/rd/1

- CTRL-D, exit, etc. to invoke change_root -> /dev/rd/1

- umount /initrd/dev and /initrd

- load modules needed for boot.. ext2, ide, loopback, crypto, and sticky order-dependent things (in my case, uhci MUST precede eepro100)

- mount -n -t tmpfs none /tmp

- copy needed things to /tmp like /bin, /sbin, /lib (not modules, unless memory isn't a problem)

- mount -n -t devfs none /tmp/dev

- cd /tmp ; pivot_root . old_root ; chroot bin/sh ...

- now we are chrooted into bin/sh on a tmpfs root

- umount old_root/dev ; umount old_root

- mount real filesystem - mount host on /mnt, loopback on /real_root

- finally chroot into real_root/sbin/init

- once system is running, clear out unneeded files in tmpfs, unmount the devfs - memory usage goes to almost nothing (tmpfs is still needed since the host for the loopback lives there)

- blockdev --flushbufs /dev/rd/*

(note, I have glossed over some details. /proc is needed in several steps, and must be unmounted and remounted and so on..)

After running for a while and filling up memory, buffers has gone down far below the size of the ramdisk, and trying to mount /dev/rd/0 and /dev/rd/1 shows that they are no longer valid filesystems.

This method depends on the change_root() mechanism which I had assumed is becoming obsolete. It works, and there is no need to mess with /proc/sys/kernel/real_root_dev if the root is specified on the command line. Trying to use only pivot_root did not work as /dev/rd/0 could never be flushed (see previous messages in this thread).

Thanks very much to everyone who helped!

Maciek

#15 lore raw
Quoted 2 lines · Maciek Nowacki

Problem seems to be solved. Here is what I did, for anyone who is interested in using a loopback file on a local disk as root:

[snip recipe]

Quoted 5 lines

This method depends on the change_root() mechanism which I had assumed is becoming obsolete. It works, and there is no need to mess with /proc/sys/kernel/real_root_dev if the root is specified on the command line. Trying to use only pivot_root did not work as /dev/rd/0 could never be flushed (see previous messages in this thread).

I was having similiar problems a few months back. I was also trying to pivot_root out of an initial ramdisk and then unmount it. I got it working, but kept forgetting to post one of the fixes that I found necessary to make it work when using auto-mounted devfs.

Here it is (exported from BitKeeper, hence the a and b):

init/main.c +3 -0
diff -Nru a/init/main.c b/init/main.c
@@ -273,6 +273,9 @@
 #ifdef CONFIG_NFTL
 	{ "nftla", 0x5d00 },
 #endif
+#ifdef CONFIG_DEVFS_MOUNT
+	{ "rd/",     0x0100 },
+#endif
 	{ NULL, 0 }
 };
 

What this does is make /dev/rd/* parseable by name_to_kdev_t, with the result that ROOT_DEV gets set correctly in root_dev_setup and the initrd logic at the bottom of do_basic_setup doesn't automatically trigger the change_root. The same effect can be achieved by setting real_root_dev in proc in your ramdisk, but since that's not required for non-devfs ramdisks, it takes some digging for the uninitiated to figure out why things don't work.

With this fix and the fix for the missing bldev_put in drivers/block/rd.c (which has been in Alan's tree for ages), the pivot_root instructions given in the file Documentation/initrd.txt have been working fine for me.

Scott

Signature · 5 lines
=============================================================================
Scott Murray                                        email: [email protected]
http://www.spiteful.org (coming soon)                 ICQ: 10602428
-----------------------------------------------------------------------------
     "Good, bad ... I'm the guy with the gun." - Ash, "Army of Darkness"
#16 lore raw
In reply to Maciek Nowacki
Quoted 8 lines · Scott Murray
Quoted 5 lines · Maciek Nowacki

This method depends on the change_root() mechanism which I had assumed is becoming obsolete. It works, and there is no need to mess with /proc/sys/kernel/real_root_dev if the root is specified on the command line. Trying to use only pivot_root did not work as /dev/rd/0 could never be flushed (see previous messages in this thread).

Quoted 6 lines

I was having similiar problems a few months back. I was also trying to pivot_root out of an initial ramdisk and then unmount it. I got it working, but kept forgetting to post one of the fixes that I found necessary to make it work when using auto-mounted devfs.

[patch to init/main.c]

Cool, that did it. change_root doesn't occur anymore and BLKFLSBUF clears things out of memory without any muckage. I will add this one to my patch directory. Thanks!

Maciek