Warning from free_init_pages with large initrd
Hi,
While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel throws the following warning:
[ 0.955216] Unpacking initramfs... [ 5.391977] ------------[ cut here ]------------ [ 5.391986] WARNING: CPU: 0 PID: 1 at arch/x86/mm/init.c:671 free_init_pages+0x94/0xa0 [ 5.391988] Modules linked in: [ 5.391992] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.8.0-0.rc7.git2.1.fc26.x86_64 #1 [ 5.391993] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014 [ 5.391995] 0000000000000286 00000000c1677634 ffff9d943abffd58 ffffffffb6467073 [ 5.391999] 0000000000000000 0000000000000000 ffff9d943abffd98 ffffffffb60ae7ab [ 5.392001] 0000029fb71cab9d ffff9d9300000000 ffff9d9380000000 ffff9d9329861000 [ 5.392003] Call Trace: [ 5.392008] [<ffffffffb6467073>] dump_stack+0x86/0xc3 [ 5.392020] [<ffffffffb60ae7ab>] __warn+0xcb/0xf0 [ 5.392022] [<ffffffffb60ae8dd>] warn_slowpath_null+0x1d/0x20 [ 5.392024] [<ffffffffb606b094>] free_init_pages+0x94/0xa0 [ 5.392027] [<ffffffffb71cabe3>] ? unpack_to_rootfs+0x29d/0x29d [ 5.392039] [<ffffffffb71c88cd>] ? set_debug_rodata+0x12/0x12 [ 5.392042] [<ffffffffb71e6aee>] free_initrd_mem+0x21/0x23 [ 5.392044] [<ffffffffb71cacc8>] populate_rootfs+0xe5/0x105 [ 5.392046] [<ffffffffb6002190>] do_one_initcall+0x50/0x180 [ 5.392048] [<ffffffffb71c88cd>] ? set_debug_rodata+0x12/0x12 [ 5.392049] [<ffffffffb71c923b>] kernel_init_freeable+0x200/0x29c [ 5.392053] [<ffffffffb68e6b4e>] kernel_init+0xe/0x100 [ 5.392055] [<ffffffffb68f7b2f>] ret_from_fork+0x1f/0x40 [ 5.392056] [<ffffffffb68e6b40>] ? rest_init+0x140/0x140 [ 5.392062] ---[ end trace 0fbf5e487fcf7e56 ]--- [ 6.572994] Freeing initrd memory: 1416828K (ffff9d9329861000 - ffff9d9380000000)
The initrd is big because it holds all the files in the ISO (see
https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how it was made). The issue was originally filed on
https://bugzilla.redhat.com/show_bug.cgi?id=1274966 but was usually against older kernels...
Signature · 1 line
Sitsofe | http://sucs.org/~sits/
Quoted 5 lines · Sitsofe Wheeler
Hi,
While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel throws the following warning:
[...]
Quoted 7 lines
The initrd is big because it holds all the files in the ISO (see
https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how it was made). The issue was originally filed on
https://bugzilla.redhat.com/show_bug.cgi?id=1274966 but was usually against older kernels...
How big is big?
Quoted 7 lines · H. Peter Anvin
Quoted 4 lines · Sitsofe Wheeler
While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel throws the following warning:
[...]
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
The initrd is big because it holds all the files in the ISO (see
https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how it was made). The issue was originally filed on
https://bugzilla.redhat.com/show_bug.cgi?id=1274966 but was usually against older kernels...How big is big?
The one that matches that backtrace is 1385 Mbytes and was made from a Fedora rawhide ISO. Here are some sizes in megabytes from different versions of Fedora:
960 20/initrd0.img
1416 21/initrd0.img
1379 22/initrd0.img
1430 23/initrd0.img
1492 24/initrd.img
Signature · 1 line
Sitsofe | http://sucs.org/~sits/
Quoted 17 lines · Sitsofe Wheeler
Quoted 9 lines · H. Peter Anvin
Quoted 4 lines · Sitsofe Wheeler
While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel throws the following warning:
[...]
Quoted 3 lines
The initrd is big because it holds all the files in the ISO (see
https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how itwas
Quoted 5 lines
Quoted 3 lines
made). The issue was originally filed on
https://bugzilla.redhat.com/show_bug.cgi?id=1274966 but was usually against older kernels...How big is big?
Quoted 8 lines
The one that matches that backtrace is 1385 Mbytes and was made from a Fedora rawhide ISO. Here are some sizes in megabytes from different versions of Fedora:
960 20/initrd0.img
1416 21/initrd0.img
1379 22/initrd0.img
1430 23/initrd0.img
1492 24/initrd.img
And they all fail?
Also, is this BIOS or EFI?
When loading under BIOS there is currently a 2 GB limit with most loaders; it would have been 4 GB except that Grub had a bug (sigh.)
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse brevity and formatting.
Quoted 17 lines · Sitsofe Wheeler
Quoted 9 lines · H. Peter Anvin
Quoted 4 lines · Sitsofe Wheeler
While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel throws the following warning:
[...]
Quoted 3 lines
The initrd is big because it holds all the files in the ISO (see
https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how itwas
Quoted 5 lines
Quoted 3 lines
made). The issue was originally filed on
https://bugzilla.redhat.com/show_bug.cgi?id=1274966 but was usually against older kernels...How big is big?
Quoted 8 lines
The one that matches that backtrace is 1385 Mbytes and was made from a Fedora rawhide ISO. Here are some sizes in megabytes from different versions of Fedora:
960 20/initrd0.img
1416 21/initrd0.img
1379 22/initrd0.img
1430 23/initrd0.img
1492 24/initrd.img
Also: compressed or uncompressed (if it makes a difference?)
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse brevity and formatting.
Quoted 25 lines · H. Peter Anvin
Quoted 15 lines · Sitsofe Wheeler
Quoted 9 lines · H. Peter Anvin
Quoted 4 lines · Sitsofe Wheeler
While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel throws the following warning:
[...]
Quoted 3 lines
The initrd is big because it holds all the files in the ISO (see
https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how itwas
Quoted 3 lines
Quoted 1 line
made).
How big is big?
Quoted 8 lines
The one that matches that backtrace is 1385 Mbytes and was made from a Fedora rawhide ISO. Here are some sizes in megabytes from different versions of Fedora:
960 20/initrd0.img
1416 21/initrd0.img
1379 22/initrd0.img
1430 23/initrd0.img
1492 24/initrd.img
Quoted 2 lines
And they all fail?
From at least 23 onwards they all continue to boot but give the
previously mentioned warning in dmesg output.
Quoted 1 line
Also, is this BIOS or EFI?
Booting from BIOS, not EFI.
Quoted 1 line
Also: compressed or uncompressed (if it makes a difference?)
file reports the following
20 initrd starts with a gzip header
21-23 initrds start with a cpio header
24 and rawhide initrd starts with an xz header
Looking at livecd-iso-to-pxeboot it seems that the CD's contents is turned into a gzip'd cpio and then concatenated on the end of the usual initrd...
Signature · 1 line
Sitsofe | http://sucs.org/~sits/
Quoted 35 lines · Sitsofe Wheeler
Quoted 25 lines · H. Peter Anvin
Quoted 15 lines · Sitsofe Wheeler
Quoted 9 lines · H. Peter Anvin
Quoted 4 lines · Sitsofe Wheeler
While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel throws the following warning:
[...]
Quoted 3 lines
The initrd is big because it holds all the files in the ISO (see
https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how itwas
Quoted 3 lines
Quoted 1 line
made).
How big is big?
Quoted 8 lines
The one that matches that backtrace is 1385 Mbytes and was made from a Fedora rawhide ISO. Here are some sizes in megabytes from different versions of Fedora:
960 20/initrd0.img
1416 21/initrd0.img
1379 22/initrd0.img
1430 23/initrd0.img
1492 24/initrd.imgQuoted 1 line
And they all fail?
From at least 23 onwards they all continue to boot but give the previously mentioned warning in dmesg output.
Quoted 1 line
Also, is this BIOS or EFI?
Booting from BIOS, not EFI.
Quoted 1 line
Also: compressed or uncompressed (if it makes a difference?)
Quoted 9 lines
file reports the following
20 initrd starts with a gzip header
21-23 initrds start with a cpio header
24 and rawhide initrd starts with an xz headerLooking at livecd-iso-to-pxeboot it seems that the CD's contents is turned into a gzip'd cpio and then concatenated on the end of the usual initrd...
but the warning in free_init_pages() is about alignment, not size... Maybe the concatenation is bad?
Signature · 1 line
~Randy
(See http://www.gossamer-threads.com/lists/linux/kernel/2534175 for the history of this thread )
Quoted 3 lines · Randy Dunlap
but the warning in
free_init_pages()is about alignment, not size... Maybe the concatenation is bad?
What would l have to pull apart to be able to tell?
Having said that I've just noticed that newer versions of the script concatenate an ISO to the initrd (rather than another cpio) -
https://github.com/rhinstaller/livecd-tools/commit/8067be50907da9461e442c11a664c89e066ccac6#diff-88c69e43bb69726c532af3a136cc50e8
. Unfortunately after rebuilding the initrd with the livecd-iso-to-pxeboot version from Fedora 24 the warning persists. CC'ing a few of the tool's authors on this email.
Signature · 1 line
Sitsofe | http://sucs.org/~sits/
Quoted 8 lines · Sitsofe Wheeler
(See http://www.gossamer-threads.com/lists/linux/kernel/2534175 for the history of this thread )
Quoted 3 lines · Randy Dunlap
but the warning in
free_init_pages()is about alignment, not size... Maybe the concatenation is bad?
Quoted 8 lines
What would l have to pull apart to be able to tell?
Having said that I've just noticed that newer versions of the script concatenate an ISO to the initrd (rather than another cpio) -
https://github.com/rhinstaller/livecd-tools/commit/8067be50907da9461e442c11a664c89e066ccac6#diff-88c69e43bb69726c532af3a136cc50e8
. Unfortunately after rebuilding the initrd with the livecd-iso-to-pxeboot version from Fedora 24 the warning persists. CC'ing a few of the tool's authors on this email.
I would just comment that I wrote livecd-iso-to-pxeboot for a specific purpose -- virt-p2v -- where the size of the ISO rarely exceeded around 100-200 MB, which is large for an initramfs, but about an order of magnitude smaller than what you are trying.
Rich.
Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Quoted 9 lines · Sitsofe Wheeler
(See http://www.gossamer-threads.com/lists/linux/kernel/2534175 for the history of this thread )
Quoted 3 lines · Randy Dunlap
but the warning in
free_init_pages()is about alignment, not size... Maybe the concatenation is bad?
Quoted 8 lines
What would l have to pull apart to be able to tell?
Having said that I've just noticed that newer versions of the script concatenate an ISO to the initrd (rather than another cpio) -
https://github.com/rhinstaller/livecd-tools/commit/8067be50907da9461e442c11a664c89e066ccac6#diff-88c69e43bb69726c532af3a136cc50e8
. Unfortunately after rebuilding the initrd with the livecd-iso-to-pxeboot version from Fedora 24 the warning persists. CC'ing a few of the tool's authors on this email.
This really doesn't seem like a good approach, and certainly may cause memory exhaustion at boot time. Still would be good to track down the problem.
Overall the idea of downloading the whole shebang in the initial TFTP transaction ID a really bad idea. TFTP is slow and unreliable, and the PXE ROMs are often complete crap in terms of quality. It would be way better to bring up a small initramfs and then fetch the rest using http or a similar protocol from the very high quality Linux TCP stack. As far as I know, Fedora has supported this mode for *ages*.
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse brevity and formatting.
Quoted 19 lines · H. Peter Anvin
Quoted 9 lines · Sitsofe Wheeler
(See http://www.gossamer-threads.com/lists/linux/kernel/2534175 for the history of this thread )
Quoted 3 lines · Randy Dunlap
but the warning in
free_init_pages()is about alignment, not size... Maybe the concatenation is bad?Quoted 8 lines
What would l have to pull apart to be able to tell?
Having said that I've just noticed that newer versions of the script concatenate an ISO to the initrd (rather than another cpio) -
https://github.com/rhinstaller/livecd-tools/commit/8067be50907da9461e442c11a664c89e066ccac6#diff-88c69e43bb69726c532af3a136cc50e8
. Unfortunately after rebuilding the initrd with the livecd-iso-to-pxeboot version from Fedora 24 the warning persists. CC'ing a few of the tool's authors on this email.
Quoted 3 lines
This really doesn't seem like a good approach, and certainly may cause memory exhaustion at boot time. Still would be good to track down the problem.
Overall the idea of downloading the whole shebang in the initial TFTP transaction ID a really bad idea. TFTP is slow and unreliable, and the PXE ROMs are often complete crap in terms of quality. It would be way better to bring up a small initramfs and then fetch the rest using http or a similar protocol from the very high quality Linux TCP stack. As far as I know, Fedora has supported this mode for *ages*.
I wouldn't mind tracking down the problem too but I admit I don't know where to start on this alignment issue.
As for the process, it can be excruciating slow and can cause memory exhaustion but the initrd and vmlinuz are actually downloaded over HTTP courtesy of iPXE (so grabbing 1GByte happens in under a minute over HTTP) and these days machines have many gigabytes of memory so it actually works quite well once booted...
I was going to say that Fedora doesn't allow PXE booting of its live images (I know it always supported something for server installs) but I've just come across this
https://bugzilla.redhat.com/show_bug.cgi?id=1317780 so perhaps there's a another route to take in addition to this next time round...
Signature · 1 line
Sitsofe | http://sucs.org/~sits/
Quoted 5 lines · Sitsofe Wheeler
Quoted 4 lines · H. Peter Anvin
Quoted 3 lines · Sitsofe Wheeler
(See http://www.gossamer-threads.com/lists/linux/kernel/2534175 for the history of this thread )
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines · Randy Dunlap
but the warning in
free_init_pages()is about alignment, not size... Maybe the concatenation is bad?What would l have to pull apart to be able to tell?
This turned out to be down to a bug in iPXE initrd alignment which is fixed in https://git.ipxe.org/ipxe.git/commit/aa11f5deda29f91f5b8592965ed00f4a65002eff
and impacted any initrd.
Signature · 1 line
Sitsofe | http://sucs.org/~sits/