Skip to contentSkip to conversation

ppc64le vmlinuz is huge when building with BTF

DISCUSSION Dominique Martinet 6 participants 11 messages Jun 15, 2023 – Jun 16, 2023 BPF

lore
Share & tools
lore
#0 lore raw

Hi,

coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563

alice noticed the kernel packages got quite bigger, in particular for ppc64le I've confirmed that the vmlinuz file size jump when building with BTF:
currently released package with BTF:
https://dl-cdn.alpinelinux.org/alpine/edge/main/ppc64le/linux-lts-6.1.33-r0.apk
272M boot/vmlinuz-lts

test build without BTF:
https://gitlab.alpinelinux.org/martinetd/aports/-/jobs/1049335
44M boot/vmlinuz-lts

Is that a known issue?
We'll probably just turn off BTF for the ppc64le build for now, but it might be worth checking.

While I have your attention, even the x86_64 package grew much bigger than I thought it would, the installed modules directory go from 90MB to 108MB gzipped); it's a 18% increase (including kernel: 103->122MB) which is more than what I'd expect out of BTF.
Most users don't care about BTF so it'd be great if they could be built and installed separately (debug package all over again..) or limiting the growth a bit more if possible.
I haven't tried yet but at this point ikheaders is probably worth considering instead..
Perhaps we're missing some stripping option or something?

Thanks!

Signature · 1 line
Dominique Martinet | Asmadeus
#1 lore raw
Quoted 3 lines · Dominique Martinet

Hi,

coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563

hi,
it's probably burried somewhere in that discussion, but do you have kernel version (or commit) where that increase happened?

also link for used config would be great

thanks,
jirka

Quoted 34 lines

alice noticed the kernel packages got quite bigger, in particular for ppc64le I've confirmed that the vmlinuz file size jump when building with BTF:
currently released package with BTF:
https://dl-cdn.alpinelinux.org/alpine/edge/main/ppc64le/linux-lts-6.1.33-r0.apk
272M boot/vmlinuz-lts

test build without BTF:
https://gitlab.alpinelinux.org/martinetd/aports/-/jobs/1049335
44M boot/vmlinuz-lts

Is that a known issue?
We'll probably just turn off BTF for the ppc64le build for now, but it might be worth checking.

While I have your attention, even the x86_64 package grew much bigger than I thought it would, the installed modules directory go from 90MB to 108MB gzipped); it's a 18% increase (including kernel: 103->122MB) which is more than what I'd expect out of BTF.
Most users don't care about BTF so it'd be great if they could be built and installed separately (debug package all over again..) or limiting the growth a bit more if possible.
I haven't tried yet but at this point ikheaders is probably worth considering instead..
Perhaps we're missing some stripping option or something?

Thanks!

Signature · 1 line
Dominique Martinet | Asmadeus
#2 lore raw

Jiri Olsa wrote on Thu, Jun 15, 2023 at 10:39:18AM +0200:

Quoted 4 lines
Quoted 1 line

coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563

it's probably burried somewhere in that discussion, but do you have kernel version (or commit) where that increase happened?

Unfortunately not -- we've just tried on the 6.1.33 that's the current alpine lts kernel, but I cannot say since when this started because we've just enabled BTF recently.

Alpine also doesn't seem to keep old versions of apk files on its mirrors so while it probably has been happening since it got enabled I don't know how to check, and the commit enabling BTF has been done without a MR so there's no test log that'd allow seeing the package size either :/

Quoted 1 line

also link for used config would be great

alpine has two configs which both exhibit the issue (raw file link on commit before removing BTF):
https://gitlab.alpinelinux.org/alpine/aports/-/raw/749ee7117e1437b7ab3ef2590f7f2e3558fda3ef/main/linux-lts/virt.ppc64le.config
Size difference for linux-virt: 219 MiB -> 47 MiB
https://gitlab.alpinelinux.org/alpine/aports/-/raw/749ee7117e1437b7ab3ef2590f7f2e3558fda3ef/main/linux-lts/lts.ppc64le.config
Size difference for linux-lts: 306 MiB -> 74 MiB

We just disabled BTF for this arch for now, I honestly won't have time to dig further short term as I'm not that involved in this issue and day job is busy right now.

Thanks,

Signature · 1 line
Dominique
#3 lore raw
Quoted 16 lines · Dominique Martinet

Jiri Olsa wrote on Thu, Jun 15, 2023 at 10:39:18AM +0200:

Quoted 4 lines
Quoted 1 line

coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563

it's probably burried somewhere in that discussion, but do you have kernel version (or commit) where that increase happened?

Unfortunately not -- we've just tried on the 6.1.33 that's the current alpine lts kernel, but I cannot say since when this started because we've just enabled BTF recently.

Alpine also doesn't seem to keep old versions of apk files on its mirrors so while it probably has been happening since it got enabled I don't know how to check, and the commit enabling BTF has been done without a MR so there's no test log that'd allow seeing the package size either :/

Quoted 9 lines
Quoted 1 line

also link for used config would be great

alpine has two configs which both exhibit the issue (raw file link on commit before removing BTF):
https://gitlab.alpinelinux.org/alpine/aports/-/raw/749ee7117e1437b7ab3ef2590f7f2e3558fda3ef/main/linux-lts/virt.ppc64le.config
Size difference for linux-virt: 219 MiB -> 47 MiB
https://gitlab.alpinelinux.org/alpine/aports/-/raw/749ee7117e1437b7ab3ef2590f7f2e3558fda3ef/main/linux-lts/lts.ppc64le.config
Size difference for linux-lts: 306 MiB -> 74 MiB

I took a look (gunzip'ed and tar-extracted the apk to get the vmlinuz-lts file) and the BTF looks reasonable size-wise:

$ objdump -h vmlinuz-lts
...

13 .BTF 003d64cc c000000001400844 0000000001400844 01410844
2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA
14 .BTF_ids 00000248 c0000000017d6d10 00000000017d6d10 017e6d10
2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA

So BTF is ~4MB if I'm reading the above right.

However the problem I suspect is this:

51 .debug_info 0a488b55 0000000000000000 0000000000000000 026f8d20
2**0
CONTENTS, READONLY, DEBUGGING
52 .debug_abbrev 004d66f0 0000000000000000 0000000000000000 0cb81875 2**0
CONTENTS, READONLY, DEBUGGING
53 .debug_line 019192f6 0000000000000000 0000000000000000 0d057f65
2**0
CONTENTS, READONLY, DEBUGGING
54 .debug_frame 002d4fe0 0000000000000000 0000000000000000 0e971260 2**3
CONTENTS, READONLY, DEBUGGING
55 .debug_str 00390822 0000000000000000 0000000000000000 0ec46240
2**0
CONTENTS, READONLY, DEBUGGING
56 .debug_line_str 00000445 0000000000000000 0000000000000000
0efd6a62 2**0
CONTENTS, READONLY, DEBUGGING
57 .debug_loclists 0176a32c 0000000000000000 0000000000000000
0efd6ea7 2**0
CONTENTS, READONLY, DEBUGGING
58 .debug_rnglists 00382d7a 0000000000000000 0000000000000000
107411d3 2**0
CONTENTS, READONLY, DEBUGGING

The debug info hasn't been stripped, so I suspect the packaging spec file or equivalent - in perhaps trying to preserve the .BTF section - is preserving debug info too. DWARF needs to be there at BTF generation time in vmlinux but is usually stripped for non-debug packages.

FYI we're aiming to make BTF module-loadable via CONFIG_DEBUG_INFO_BTF=m in the future, I'm hoping to get an RFC patch out for that soon once other BTF-related issues are sorted. Hope this helps

Alan

#4 lore raw

Alan Maguire wrote on Thu, Jun 15, 2023 at 03:31:49PM +0100:

Quoted 7 lines

However the problem I suspect is this:

51 .debug_info 0a488b55 0000000000000000 0000000000000000 026f8d20
2**0
CONTENTS, READONLY, DEBUGGING
[...]

Quoted 5 lines

The debug info hasn't been stripped, so I suspect the packaging spec file or equivalent - in perhaps trying to preserve the .BTF section - is preserving debug info too. DWARF needs to be there at BTF generation time in vmlinux but is usually stripped for non-debug packages.

Thanks Alan and Eduard!
I guess I should have checked that first, it helps.

We're not stripping anything in vmlinuz for other archs -- the linker script already should be including only the bare minimum to decompress itself (+compressed useful bits), so I guess it's a Kbuild issue for the arch.
We can add a strip but I unfortunately have no way of testing ppc build, I'll ask around the build linux-kbuild and linuxppc-dev lists if that's expected; it shouldn't be that bad now that's figured out.

Quoted 3 lines

FYI we're aiming to make BTF module-loadable via CONFIG_DEBUG_INFO_BTF=m in the future, I'm hoping to get an RFC patch out for that soon once other BTF-related issues are sorted. Hope this helps

Oh, that's interesting -- I assume that'll only change the 'built-in' BTF info? Or will that also split BTF info in other modules as e.g. modfoo_btf.ko?
For x86_64 the size increase of vmlinuz itself is rather acceptable (<2MB), but the sheer amount of modules (the -lts package has over 3k modules...) means that even a small size increase for each module ends up taking proportionally a high amount of space (+20MB from 90MB), so being able to package separately would be appreciated (alpine likes splitting optional features in subpackages)
Packaging-wise I'm not sure it'd make sense to keep the overhead in other modules and just split the 'main' btf infos.

Otoh even if it doesn't help with packaging, having a smaller vmlinuz means faster boot and lower kernel memory footprint (I think *2?) for people who don't need it, so I think it's a good idea and we'll probably enable it once it becomes available. Thanks for the heads up!

Signature · 1 line
Dominique Martinet | Asmadeus
#5 lore raw
Quoted 19 lines · Dominique Martinet

Alan Maguire wrote on Thu, Jun 15, 2023 at 03:31:49PM +0100:

Quoted 7 lines

However the problem I suspect is this:

51 .debug_info 0a488b55 0000000000000000 0000000000000000 026f8d20
2**0
CONTENTS, READONLY, DEBUGGING
[...]

Quoted 5 lines

The debug info hasn't been stripped, so I suspect the packaging spec file or equivalent - in perhaps trying to preserve the .BTF section - is preserving debug info too. DWARF needs to be there at BTF generation time in vmlinux but is usually stripped for non-debug packages.

Thanks Alan and Eduard!
I guess I should have checked that first, it helps.

Quoted 13 lines

We're not stripping anything in vmlinuz for other archs -- the linker script already should be including only the bare minimum to decompress itself (+compressed useful bits), so I guess it's a Kbuild issue for the arch.
We can add a strip but I unfortunately have no way of testing ppc build, I'll ask around the build linux-kbuild and linuxppc-dev lists if that's expected; it shouldn't be that bad now that's figured out.

Quoted 3 lines

FYI we're aiming to make BTF module-loadable via CONFIG_DEBUG_INFO_BTF=m in the future, I'm hoping to get an RFC patch out for that soon once other BTF-related issues are sorted. Hope this helps

Besides this, in the past it was requested a few times to make CONFIG_DEBUG_INFO_BTF=y not imply CONFIG_DEBUG_INFO_DWARF. That is, make it possible to get BTF without .dwarf* sections being left in the vmlinux image.

Of course, implementation-wise, DWARF would still have to be generated at runtime, but would be automatically stripped out after BTF generation, unless CONFIG_DEBUG_INFO_DWARF* is set.

I think it would help in this case as well. So if anyone has free cycles, it would be awesome to make this possible.

Quoted 20 lines

Oh, that's interesting -- I assume that'll only change the 'built-in' BTF info? Or will that also split BTF info in other modules as e.g. modfoo_btf.ko?
For x86_64 the size increase of vmlinuz itself is rather acceptable (<2MB), but the sheer amount of modules (the -lts package has over 3k modules...) means that even a small size increase for each module ends up taking proportionally a high amount of space (+20MB from 90MB), so being able to package separately would be appreciated (alpine likes splitting optional features in subpackages)
Packaging-wise I'm not sure it'd make sense to keep the overhead in other modules and just split the 'main' btf infos.

Otoh even if it doesn't help with packaging, having a smaller vmlinuz means faster boot and lower kernel memory footprint (I think *2?) for people who don't need it, so I think it's a good idea and we'll probably enable it once it becomes available. Thanks for the heads up!

--
Dominique Martinet | Asmadeus

#6 lore raw
In reply to Dominique Martinet

[Cc linuxppc-dev]

Quoted 15 lines · Dominique Martinet

Alan Maguire wrote on Thu, Jun 15, 2023 at 03:31:49PM +0100:

Quoted 7 lines

However the problem I suspect is this:

51 .debug_info 0a488b55 0000000000000000 0000000000000000 026f8d20
2**0
CONTENTS, READONLY, DEBUGGING
[...]

Quoted 5 lines

The debug info hasn't been stripped, so I suspect the packaging spec file or equivalent - in perhaps trying to preserve the .BTF section - is preserving debug info too. DWARF needs to be there at BTF generation time in vmlinux but is usually stripped for non-debug packages.

Quoted 7 lines

Thanks Alan and Eduard!
I guess I should have checked that first, it helps.

We're not stripping anything in vmlinuz for other archs -- the linker script already should be including only the bare minimum to decompress itself (+compressed useful bits), so I guess it's a Kbuild issue for the arch.

For a related discussion, see:
http://lore.kernel.org/CAK18DXZKs2PNmLndeGYqkPxmrrBR=6ca3bhyYCj=GhyA7dHfAQ@mail.gmail.com

Quoted 3 lines

We can add a strip but I unfortunately have no way of testing ppc build, I'll ask around the build linux-kbuild and linuxppc-dev lists if that's expected; it shouldn't be that bad now that's figured out.

Stripping vmlinux would indeed be the way to go. As mentioned in the above link, fedora also packages a strip'ed vmlinux for ppc64le:
https://src.fedoraproject.org/rpms/kernel/blob/4af17bffde7a1eca9ab164e5de0e391c277998a4/f/kernel.spec#_1797

- Naveen

#7 lore raw

Naveen N Rao wrote on Fri, Jun 16, 2023 at 04:28:53PM +0530:

Quoted 7 lines
Quoted 4 lines

We're not stripping anything in vmlinuz for other archs -- the linker script already should be including only the bare minimum to decompress itself (+compressed useful bits), so I guess it's a Kbuild issue for the arch.

For a related discussion, see:
http://lore.kernel.org/CAK18DXZKs2PNmLndeGYqkPxmrrBR=6ca3bhyYCj=GhyA7dHfAQ@mail.gmail.com

Thanks, I didn't know that ppc64le boots straight into vmlinux, as 'make install' somehow installs something called 'vmlinuz-lts' (-lts coming out of localversion afaiu, but vmlinuz would come from the build scripts) ; this is somewhat confusing as vmlinuz on other archs is a compressed/pre-processed binary so I'd expect it to at least be stripped...

Quoted 7 lines
Quoted 3 lines

We can add a strip but I unfortunately have no way of testing ppc build, I'll ask around the build linux-kbuild and linuxppc-dev lists if that's expected; it shouldn't be that bad now that's figured out.

Stripping vmlinux would indeed be the way to go. As mentioned in the above link, fedora also packages a strip'ed vmlinux for ppc64le:
https://src.fedoraproject.org/rpms/kernel/blob/4af17bffde7a1eca9ab164e5de0e391c277998a4/f/kernel.spec#_1797

It feels somewhat wrong to add a strip just for ppc64le after make install, but I guess we probably ought to do the same...
I don't have any hardware to test booting the result though, I'll submit an update and ask for someone to test when it's done.
(bit busy but that doesn't take long, will do that tomorrow morning before I forget)

Thanks!

Signature · 1 line
Dominique Martinet | Asmadeus
#8 lore raw
Quoted 9 lines · Dominique Martinet

Naveen N Rao wrote on Fri, Jun 16, 2023 at 04:28:53PM +0530:

Quoted 7 lines
Quoted 4 lines

We're not stripping anything in vmlinuz for other archs -- the linker script already should be including only the bare minimum to decompress itself (+compressed useful bits), so I guess it's a Kbuild issue for the arch.

For a related discussion, see:
http://lore.kernel.org/CAK18DXZKs2PNmLndeGYqkPxmrrBR=6ca3bhyYCj=GhyA7dHfAQ@mail.gmail.com

Quoted 6 lines

Thanks, I didn't know that ppc64le boots straight into vmlinux, as 'make install' somehow installs something called 'vmlinuz-lts' (-lts coming out of localversion afaiu, but vmlinuz would come from the build scripts) ; this is somewhat confusing as vmlinuz on other archs is a compressed/pre-processed binary so I'd expect it to at least be stripped...

As far as I can tell, kernel's install script doesn't give out that name, so 'vmlinuz' is likely coming from the distro's /[s]bin/installkernel script. It probably needs an override to retain 'vmlinux'.

Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

We can add a strip but I unfortunately have no way of testing ppc build, I'll ask around the build linux-kbuild and linuxppc-dev lists if that's expected; it shouldn't be that bad now that's figured out.

Stripping vmlinux would indeed be the way to go. As mentioned in the above link, fedora also packages a strip'ed vmlinux for ppc64le:
https://src.fedoraproject.org/rpms/kernel/blob/4af17bffde7a1eca9ab164e5de0e391c277998a4/f/kernel.spec#_1797

Quoted 6 lines

It feels somewhat wrong to add a strip just for ppc64le after make install, but I guess we probably ought to do the same...
I don't have any hardware to test booting the result though, I'll submit an update and ask for someone to test when it's done.
(bit busy but that doesn't take long, will do that tomorrow morning before I forget)

Thanks! You're right that it's likely just powerpc that is different here. It sure would be nice if we can iron out issues with our zImage.

- Naveen

maintainer
#9 lore raw
In reply to Dominique Martinet
Quoted 10 lines · Dominique Martinet

Hi,

coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563

alice noticed the kernel packages got quite bigger, in particular for ppc64le I've confirmed that the vmlinuz file size jump when building with BTF:
currently released package with BTF:
https://dl-cdn.alpinelinux.org/alpine/edge/main/ppc64le/linux-lts-6.1.33-r0.apk
272M boot/vmlinuz-lts

Hi Dominique,

I've just checked the linked apk and it looks like DWARF sections are not stripped and take most of the space, e.g.:

$ llvm-objdump --headers boot/vmlinuz-lts | grep \.debug \
  | gawk '{ sum += strtonum("0x"$3); } END { print sum; }' | numfmt --to iec
228M

Compare this to BTF sections size:

$ llvm-objdump --headers boot/vmlinuz-lts | grep BTF \
  | gawk '{ sum += strtonum("0x"$3); } END { print sum; }' | numfmt --to iec
3,9M
Quoted 24 lines

test build without BTF:
https://gitlab.alpinelinux.org/martinetd/aports/-/jobs/1049335
44M boot/vmlinuz-lts

Is that a known issue?
We'll probably just turn off BTF for the ppc64le build for now, but it might be worth checking.

While I have your attention, even the x86_64 package grew much bigger than I thought it would, the installed modules directory go from 90MB to 108MB gzipped); it's a 18% increase (including kernel: 103->122MB) which is more than what I'd expect out of BTF.
Most users don't care about BTF so it'd be great if they could be built and installed separately (debug package all over again..) or limiting the growth a bit more if possible.
I haven't tried yet but at this point ikheaders is probably worth considering instead..
Perhaps we're missing some stripping option or something?

Thanks!

maintainer
#10 lore raw
Quoted 21 lines · Eduard Zingerman
Quoted 10 lines · Dominique Martinet

Hi,

coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563

alice noticed the kernel packages got quite bigger, in particular for ppc64le I've confirmed that the vmlinuz file size jump when building with BTF:
currently released package with BTF:
https://dl-cdn.alpinelinux.org/alpine/edge/main/ppc64le/linux-lts-6.1.33-r0.apk
272M boot/vmlinuz-lts

Hi Dominique,

I've just checked the linked apk and it looks like DWARF sections are not stripped and take most of the space, e.g.:

$ llvm-objdump --headers boot/vmlinuz-lts | grep \.debug \
  | gawk '{ sum += strtonum("0x"$3); } END { print sum; }' | numfmt --to iec
228M
Quoted 5 lines

Compare this to BTF sections size:

$ llvm-objdump --headers boot/vmlinuz-lts | grep BTF \
  | gawk '{ sum += strtonum("0x"$3); } END { print sum; }' | numfmt --to iec
3,9M

Heh, Alan answered while I was looking into it, sorry for the spam.

Quoted 26 lines
Quoted 25 lines

test build without BTF:
https://gitlab.alpinelinux.org/martinetd/aports/-/jobs/1049335
44M boot/vmlinuz-lts

Is that a known issue?
We'll probably just turn off BTF for the ppc64le build for now, but it might be worth checking.

While I have your attention, even the x86_64 package grew much bigger than I thought it would, the installed modules directory go from 90MB to 108MB gzipped); it's a 18% increase (including kernel: 103->122MB) which is more than what I'd expect out of BTF.
Most users don't care about BTF so it'd be great if they could be built and installed separately (debug package all over again..) or limiting the growth a bit more if possible.
I haven't tried yet but at this point ikheaders is probably worth considering instead..
Perhaps we're missing some stripping option or something?

Thanks!