ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes
Joe Lawrence
maintainerStarting with 5.14 kernels, I can reliably reproduce a crash [1] on ppc64le when loading livepatches containing late klp-relocations [2]. These are relocations, specific to livepatching, that are resolved not when a livepatch module is loaded, but only when a livepatch-target module is loaded.
There was previously related work by Josh and Peter [3] to simplify a lot of x86 and s390x code (at the time, the only two arches to HAVE_LIVEPATCH and STRICT_MODULE_RWX) as part of disallowing writable executable mappings.
Now that Power has STRICT_MODULE_RWX, I think we will need to consider this architecture as well.
The crash was originally spotted by the external kpatch-build tool [4] when building its integration tests on rhel-9-beta. It can also be reproduced by the endless-WIP klp-convert patchset [5], which brings klp-relocation creation from kpatch-build to the upstream build.
I further verified:
- turning STRICT_MODULE_RWX off resulted in no crash
- alternatively, reverting the following commits resulted in no crash:
d556e1be3332 ("livepatch: Remove module_disable_ro() usage")
0d9fbf78fefb ("module: Remove module_disable_ro()")I haven't started looking at a fix yet, but in the case of the x86 code update, its apply_relocate_add() implementation was modified to use a common text_poke() function to allowed us to drop module_{en,dis}ble_ro() games by the livepatching code.
I can take a closer look this week, but thought I'd send out a report in case this may be a known todo for STRICT_MODULE_RWX on Power.
-- Joe
[1] crashing kernel log
[ 84.837986] ===== TEST: klp-convert symbols ===== [ 84.858937] % modprobe test_klp_convert_mod [ 84.879040] % modprobe test_klp_convert1 [ 84.908056] BUG: Unable to handle kernel data access on write at 0xc0080000018402f0 [ 84.908067] Faulting instruction address: 0xc000000000056b58 [ 84.908072] Oops: Kernel access of bad area, sig: 11 [#1] [ 84.908077] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries [ 84.908082] Modules linked in: test_klp_convert1(K+) test_klp_convert_mod bonding tls rfkill pseries_rng drm fuse drm_panel_orientation_quirks xfs libcrc32c sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp vmx_crypto dm_mirror dm_region_hash dm_log dm_mod [last unloaded: test_klp_atomic_replace] [ 84.908114] CPU: 1 PID: 4205 Comm: modprobe Kdump: loaded Tainted: G K 5.14.0+ #2 [ 84.908121] NIP: c000000000056b58 LR: c000000000056b1c CTR: 0000000000000009 [ 84.908127] REGS: c00000005dce3480 TRAP: 0300 Tainted: G K (5.14.0+) [ 84.908132] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24224484 XER: 00000000 [ 84.908147] CFAR: c000000000056a68 DAR: c0080000018402f0 DSISR: 0a000000 IRQMASK: 0
GPR00: c000000000056b1c c00000005dce3720 c000000002a2af00 0000000000000000
GPR04: c0080000018402f0 396b00003d620000 e98b0020f8410018 00000000ffffffff
GPR08: 4e8004207d8903a6 0000000080000000 c0080000018382f0 000000000000000d
GPR12: 0000000000004000 c000000007fcf480 c00000004d7e2000 c0080000018706d8
GPR16: c008000001850228 c00000004d7e2c00 00000000ffffffff c0000000010d6248
GPR20: c00000000298c1c8 c008000001860380 c0080000018706f0 aaaaaaaaaaaaaaab
GPR24: c00000004d7e2b40 c008000001870000 c00800000184005c 000000000000008c
GPR28: c008000001860380 c008000000770008 c00000004d7e2000 c0080000018402f0
[ 84.908209] NIP [c000000000056b58] create_stub+0x78/0x240 [ 84.908217] LR [c000000000056b1c] create_stub+0x3c/0x240 [ 84.908223] Call Trace: [ 84.908225] [c00000005dce3720] [c00000004d7e2b40] 0xc00000004d7e2b40 (unreliable) [ 84.908232] [c00000005dce37a0] [c000000000056e0c] stub_for_addr+0xec/0x120 [ 84.908240] [c00000005dce37d0] [c000000000057f14] apply_relocate_add+0x814/0x9a0 [ 84.908247] [c00000005dce38d0] [c00000000021ca38] klp_apply_section_relocs+0x208/0x2d0 [ 84.908255] [c00000005dce39c0] [c00000000021cb90] klp_init_object_loaded+0x90/0x1d0 [ 84.908262] [c00000005dce3a50] [c00000000021d2dc] klp_enable_patch+0x32c/0x540 [ 84.908269] [c00000005dce3b10] [c008000001840030] test_klp_convert_init+0x28/0x48 [test_klp_convert1] [ 84.908277] [c00000005dce3b30] [c000000000012230] do_one_initcall+0x60/0x2c0 [ 84.908284] [c00000005dce3c00] [c00000000026012c] do_init_module+0x7c/0x3b0 [ 84.908290] [c00000005dce3c90] [c000000000262b74] __do_sys_finit_module+0xd4/0x160 [ 84.908296] [c00000005dce3db0] [c000000000030664] system_call_exception+0x144/0x280 [ 84.908303] [c00000005dce3e10] [c00000000000bff0] system_call_vectored_common+0xf0/0x280 [ 84.908310] --- interrupt: 3000 at 0x7fffa06d6b9c [ 84.908315] NIP: 00007fffa06d6b9c LR: 0000000000000000 CTR: 0000000000000000 [ 84.908320] REGS: c00000005dce3e80 TRAP: 3000 Tainted: G K (5.14.0+) [ 84.908325] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28224244 XER: 00000000 [ 84.908340] IRQMASK: 0
GPR00: 0000000000000161 00007fffc4f74ad0 00007fffa07d7100 0000000000000005
GPR04: 000000012a926ca0 0000000000000000 0000000000000005 0000000000000000
GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffa0f9c380 0000000000000020 0000000000000000
GPR16: 00000100010a1de0 0000000000000000 000000012a927d50 00000100010a02f8
GPR20: 0000000000000001 0000000000000908 00000100010a2020 00000100010a19b0
GPR24: 0000000000000000 0000000000000000 00000100010a2040 00000100010a03f0
GPR28: 00000100010a1e00 000000012a926ca0 0000000000040000 00000100010a19b0
[ 84.908399] NIP [00007fffa06d6b9c] 0x7fffa06d6b9c [ 84.908403] LR [0000000000000000] 0x0 [ 84.908406] --- interrupt: 3000 [ 84.908410] Instruction dump: [ 84.908413] 3d02ffb2 395f8000 3d208000 3ce0ffff 38c68d70 39088d84 79290020 60e7ffff [ 84.908423] e8a60014 e8c80008 e9080010 78e70020 <f8bf0000> f8df0008 f91f0010 811c0224 [ 84.908435] ---[ end trace 961b4b817da4a53b ]---
[2] https://www.kernel.org/doc/html/latest/livepatch/module-elf-format.html
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://github.com/dynup/kpatch/issues/1228
[5] https://github.com/joe-lawrence/linux/tree/klp-convert-v5-expanded-v5.14-rebase1
Quoted 5 lines · Joe Lawrence
Starting with 5.14 kernels, I can reliably reproduce a crash [1] on ppc64le when loading livepatches containing late klp-relocations [2]. These are relocations, specific to livepatching, that are resolved not when a livepatch module is loaded, but only when a livepatch-target module is loaded.
Hey Joe, thanks for the report.
Quoted 4 lines
I haven't started looking at a fix yet, but in the case of the x86 code update, its
apply_relocate_add()implementation was modified to use a commontext_poke()function to allowed us to drop module_{en,dis}ble_ro() games by the livepatching code.
It should be a similar fix for Power, our patch_instruction() uses a text poke area but apply_relocate_add() doesn't use it and does its own raw patching instead.
Quoted 2 lines
I can take a closer look this week, but thought I'd send out a report in case this may be a known todo for
STRICT_MODULE_RWXon Power.
I'm looking into this now, will update when there's progress. I personally wasn't aware but Jordan flagged this as an issue back in August [0]. Are the selftests in the klp-convert tree sufficient for testing? I'm not especially familiar with livepatching & haven't used the userspace tools.
- Russell
[0] https://github.com/linuxppc/issues/issues/375
Quoted 2 lines
-- Joe
Joe Lawrence
maintainerQuoted 6 lines · Russell Currey
I'm looking into this now, will update when there's progress. I personally wasn't aware but Jordan flagged this as an issue back in August [0]. Are the selftests in the klp-convert tree sufficient for testing? I'm not especially familiar with livepatching & haven't used the userspace tools.
Hi Russell, thanks for taking a look.
Testing with that klp-convert tree is probably the quickest and easiest way to verify the late relocations.
I'm happy to setup and test additional tools (ie, kpatch-build) with any potential changes as I know they take longer to config and run.
Thanks,
Signature · 1 line
Joe
Le 01/11/2021 à 14:48, Joe Lawrence a écrit :
Quoted 8 lines · Russell Currey
Quoted 6 lines
I'm looking into this now, will update when there's progress. I personally wasn't aware but Jordan flagged this as an issue back in August [0]. Are the selftests in the klp-convert tree sufficient for testing? I'm not especially familiar with livepatching & haven't used the userspace tools.
Quoted 8 lines
Hi Russell, thanks for taking a look.
Testing with that klp-convert tree is probably the quickest and easiest way to verify the late relocations.
I'm happy to setup and test additional tools (ie, kpatch-build) with any potential changes as I know they take longer to config and run.
Hello Joe,
I'm implementing LIVEPATCH on PPC32 and I wanted to test with STRICT_MODULE_RWX enabled so I took your branch as suggested, but I'm getting the following errors on build. What shall I do ?
CALL scripts/checksyscalls.sh CALL scripts/atomic/check-atomics.sh CHK include/generated/compile.h KLP lib/livepatch/test_klp_convert1.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyond nr_entries
klp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert1.ko] Error 255
KLP lib/livepatch/test_klp_convert2.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyond nr_entries
klp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert2.ko] Error 255
KLP lib/livepatch/test_klp_convert_sections.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyond nr_entries
klp-convert: Unable to load user-provided sympos make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert_sections.ko] Error 255 make[2]: Target '__modfinal' not remade because of errors. make[1]: *** [scripts/Makefile.modpost:145: __modpost] Error 2
make: *** [Makefile:1770: modules] Error 2
Christophe
Joe Lawrence
maintainerQuoted 28 lines · Christophe Leroy
Hello Joe,
I'm implementing LIVEPATCH on PPC32 and I wanted to test with
STRICT_MODULE_RWXenabled so I took your branch as suggested, but I'm getting the following errors on build. What shall I do ?CALL scripts/checksyscalls.sh CALL scripts/atomic/check-atomics.sh CHK include/generated/compile.h KLP lib/livepatch/test_klp_convert1.koklp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyond
nr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert1.ko] Error 255
KLPlib/livepatch/test_klp_convert2.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert2.ko] Error 255
KLPlib/livepatch/test_klp_convert_sections.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entries
Quoted 7 lines
klp-convert: Unable to load user-provided sympos make[2]: *** [
scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert_sections.ko] Error 255 make[2]: Target '__modfinal' not remade because of errors. make[1]: *** [scripts/Makefile.modpost:145:__modpost] Error 2
make: *** [Makefile:1770: modules] Error 2
Hi Christophe,
Interesting failure mode. That's klp-convert complaining that it found more relocations in a .klp.module_relocs.<objname> section than expected, i.e. nr_entries = sec->size / sizeof(struct klp_module_reloc).
A few possibilities: the ELF sec->size was incorrectly set/read by build/libelf (I doubt that). Or maybe the layout/size of struct klp_module_reloc is not consistent between kernel and userspace (I'm more suspicious of this).
Can you post a copy of the build's symbols.klp and lib/livepatch/test_klp_convert1.tmp.ko somewhere? I should be able to start debug with those files.
Thanks,
Signature · 1 line
Joe
Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
Quoted 43 lines · Christophe Leroy
Quoted 28 lines
Hello Joe,
I'm implementing LIVEPATCH on PPC32 and I wanted to test with
STRICT_MODULE_RWXenabled so I took your branch as suggested, but I'm getting the following errors on build. What shall I do ?CALL scripts/checksyscalls.sh CALL scripts/atomic/check-atomics.sh CHK include/generated/compile.h KLP lib/livepatch/test_klp_convert1.koklp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyond
nr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert1.ko] Error 255
KLPlib/livepatch/test_klp_convert2.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert2.ko] Error 255
KLPlib/livepatch/test_klp_convert_sections.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesQuoted 7 lines
klp-convert: Unable to load user-provided sympos make[2]: *** [
scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert_sections.ko] Error 255 make[2]: Target '__modfinal' not remade because of errors. make[1]: *** [scripts/Makefile.modpost:145:__modpost] Error 2
make: *** [Makefile:1770: modules] Error 2Hi Christophe,
Interesting failure mode. That's klp-convert complaining that it found more relocations in a .klp.module_relocs.<objname> section than expected, i.e.
nr_entries=sec->size/sizeof(struct klp_module_reloc).
Quoted 9 lines
A few possibilities: the ELF
sec->sizewas incorrectly set/read by build/libelf (I doubt that). Or maybe the layout/size of structklp_module_relocis not consistent between kernel and userspace (I'm more suspicious of this).Can you post a copy of the build's symbols.klp and
lib/livepatch/test_klp_convert1.tmp.kosomewhere? I should be able to start debug with those files.
I sent you both files off list.
It looks like klp-convert doesn't use the correct size. It finds a struct of size 12 hence 3 entries for a section of size 40.
On PPC32 the struct has size 8 (void * is 4 and int is 4).
But I'm cross-building from x86_64 where the struct is 8 + 4 = 12.
Can it be the reason ?
Christophe
Joe Lawrence
maintainerQuoted 58 lines · Christophe Leroy
Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
Quoted 43 lines · Christophe Leroy
Quoted 28 lines
Hello Joe,
I'm implementing LIVEPATCH on PPC32 and I wanted to test with
STRICT_MODULE_RWXenabled so I took your branch as suggested, but I'm getting the following errors on build. What shall I do ?CALL scripts/checksyscalls.sh CALL scripts/atomic/check-atomics.sh CHK include/generated/compile.h KLP lib/livepatch/test_klp_convert1.koklp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyond
nr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert1.ko] Error 255
KLPlib/livepatch/test_klp_convert2.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert2.ko] Error 255
KLPlib/livepatch/test_klp_convert_sections.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesQuoted 7 lines
klp-convert: Unable to load user-provided sympos make[2]: *** [
scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert_sections.ko] Error 255 make[2]: Target '__modfinal' not remade because of errors. make[1]: *** [scripts/Makefile.modpost:145:__modpost] Error 2
make: *** [Makefile:1770: modules] Error 2Hi Christophe,
Interesting failure mode. That's klp-convert complaining that it found more relocations in a .klp.module_relocs.<objname> section than expected, i.e.
nr_entries=sec->size/sizeof(struct klp_module_reloc).Quoted 9 lines
A few possibilities: the ELF
sec->sizewas incorrectly set/read by build/libelf (I doubt that). Or maybe the layout/size of structklp_module_relocis not consistent between kernel and userspace (I'm more suspicious of this).Can you post a copy of the build's symbols.klp and
lib/livepatch/test_klp_convert1.tmp.kosomewhere? I should be able to start debug with those files.I sent you both files off list.
Quoted 9 lines
It looks like klp-convert doesn't use the correct size. It finds a struct of size 12 hence 3 entries for a section of size 40.
On PPC32 the struct has size 8 (void * is 4 and int is 4).
But I'm cross-building from
x86_64where the struct is 8 + 4 = 12.Can it be the reason ?
I'm pretty sure that is it. I haven't had much runtime with klp-convert and cross-building (I've only found one big/little endian bug with x86_64->s390x) and was going to ask you how you were testing :)
Do you know if there are other kernel build tools that deal with similar situations? This seems like a tricky job for the userspace build tool to determine non-native target struct layout.
In the meantime, hacking in:
struct klp_module_reloc {
- void *sym;
+ uint32_t sym;
unsigned int sympos; } __packed;
gets me generating an output .ko file, but the readelf output doesn't look right.
I'll add this to the patchset TODO list, but may not get to it for a while -- is there any chance the above hack works or could you test a local non-cross build?
Thanks,
Signature · 1 line
Joe
Le 13/12/2021 à 18:26, Joe Lawrence a écrit :
Quoted 85 lines · Christophe Leroy
Quoted 58 lines
Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
Quoted 43 lines · Christophe Leroy
Quoted 28 lines
Hello Joe,
I'm implementing LIVEPATCH on PPC32 and I wanted to test with
STRICT_MODULE_RWXenabled so I took your branch as suggested, but I'm getting the following errors on build. What shall I do ?CALL scripts/checksyscalls.sh CALL scripts/atomic/check-atomics.sh CHK include/generated/compile.h KLP lib/livepatch/test_klp_convert1.koklp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyond
nr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert1.ko] Error 255
KLPlib/livepatch/test_klp_convert2.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert2.ko] Error 255
KLPlib/livepatch/test_klp_convert_sections.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesQuoted 7 lines
klp-convert: Unable to load user-provided sympos make[2]: *** [
scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert_sections.ko] Error 255 make[2]: Target '__modfinal' not remade because of errors. make[1]: *** [scripts/Makefile.modpost:145:__modpost] Error 2
make: *** [Makefile:1770: modules] Error 2Hi Christophe,
Interesting failure mode. That's klp-convert complaining that it found more relocations in a .klp.module_relocs.<objname> section than expected, i.e.
nr_entries=sec->size/sizeof(struct klp_module_reloc).Quoted 9 lines
A few possibilities: the ELF
sec->sizewas incorrectly set/read by build/libelf (I doubt that). Or maybe the layout/size of structklp_module_relocis not consistent between kernel and userspace (I'm more suspicious of this).Can you post a copy of the build's symbols.klp and
lib/livepatch/test_klp_convert1.tmp.kosomewhere? I should be able to start debug with those files.I sent you both files off list.
Quoted 9 lines
It looks like klp-convert doesn't use the correct size. It finds a struct of size 12 hence 3 entries for a section of size 40.
On PPC32 the struct has size 8 (void * is 4 and int is 4).
But I'm cross-building from
x86_64where the struct is 8 + 4 = 12.Can it be the reason ?
I'm pretty sure that is it. I haven't had much runtime with klp-convert and cross-building (I've only found one big/little endian bug with
x86_64->s390x) and was going to ask you how you were testing :)Do you know if there are other kernel build tools that deal with similar situations? This seems like a tricky job for the userspace build tool to determine non-native target struct layout.
In the meantime, hacking in:
struct klp_module_reloc {(quoted diff) +1
-1- void *sym; + uint32_t sym; unsigned int sympos; } __packed;
Quoted 7 lines
gets me generating an output .ko file, but the readelf output doesn't look right.
I'll add this to the patchset TODO list, but may not get to it for a while -- is there any chance the above hack works or could you test a local non-cross build?
No I have no way to do a non-cross build. My target is an embedded board with slow CPU and little memory.
I tested with your hack, I get:
root@vgoip:~# insmod /lib/modules/test_klp_convert1.ko insmod: can't insert '/lib/modules/test_klp_convert1.ko': unknown symbol in module, or unknown parameter root@vgoip:~# insmod /lib/modules/test_klp_livepatch.ko insmod: can't insert '/lib/modules/test_klp_livepatch.ko': unknown symbol in module, or unknown parameter
I agree with you readelf shows something went wrong with relocations.
Christophe
Joe Lawrence
maintainerQuoted 109 lines · Christophe Leroy
Le 13/12/2021 à 18:26, Joe Lawrence a écrit :
Quoted 85 lines · Christophe Leroy
Quoted 58 lines
Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
Quoted 43 lines · Christophe Leroy
Quoted 28 lines
Hello Joe,
I'm implementing LIVEPATCH on PPC32 and I wanted to test with
STRICT_MODULE_RWXenabled so I took your branch as suggested, but I'm getting the following errors on build. What shall I do ?CALL scripts/checksyscalls.sh CALL scripts/atomic/check-atomics.sh CHK include/generated/compile.h KLP lib/livepatch/test_klp_convert1.koklp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyond
nr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert1.ko] Error 255
KLPlib/livepatch/test_klp_convert2.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert2.ko] Error 255
KLPlib/livepatch/test_klp_convert_sections.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesQuoted 7 lines
klp-convert: Unable to load user-provided sympos make[2]: *** [
scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert_sections.ko] Error 255 make[2]: Target '__modfinal' not remade because of errors. make[1]: *** [scripts/Makefile.modpost:145:__modpost] Error 2
make: *** [Makefile:1770: modules] Error 2Hi Christophe,
Interesting failure mode. That's klp-convert complaining that it found more relocations in a .klp.module_relocs.<objname> section than expected, i.e.
nr_entries=sec->size/sizeof(struct klp_module_reloc).Quoted 9 lines
A few possibilities: the ELF
sec->sizewas incorrectly set/read by build/libelf (I doubt that). Or maybe the layout/size of structklp_module_relocis not consistent between kernel and userspace (I'm more suspicious of this).Can you post a copy of the build's symbols.klp and
lib/livepatch/test_klp_convert1.tmp.kosomewhere? I should be able to start debug with those files.I sent you both files off list.
Quoted 9 lines
It looks like klp-convert doesn't use the correct size. It finds a struct of size 12 hence 3 entries for a section of size 40.
On PPC32 the struct has size 8 (void * is 4 and int is 4).
But I'm cross-building from
x86_64where the struct is 8 + 4 = 12.Can it be the reason ?
I'm pretty sure that is it. I haven't had much runtime with klp-convert and cross-building (I've only found one big/little endian bug with
x86_64->s390x) and was going to ask you how you were testing :)Do you know if there are other kernel build tools that deal with similar situations? This seems like a tricky job for the userspace build tool to determine non-native target struct layout.
In the meantime, hacking in:
struct klp_module_reloc {(quoted diff) +1
-1- void *sym; + uint32_t sym; unsigned int sympos; } __packed;Quoted 7 lines
gets me generating an output .ko file, but the readelf output doesn't look right.
I'll add this to the patchset TODO list, but may not get to it for a while -- is there any chance the above hack works or could you test a local non-cross build?
No I have no way to do a non-cross build. My target is an embedded board with slow CPU and little memory.
I tested with your hack, I get:
root@vgoip:~# insmod /lib/modules/test_klp_convert1.ko insmod: can't insert '/lib/modules/test_klp_convert1.ko': unknown symbol in module, or unknown parameter root@vgoip:~# insmod /lib/modules/test_klp_livepatch.ko insmod: can't insert '/lib/modules/test_klp_livepatch.ko': unknown symbol in module, or unknown parameter
Quoted 2 lines
I agree with you readelf shows something went wrong with relocations.
Thanks for trying that. Can you point me to the cross-compiler suite that you are using for build and readelf? Kernel .config would be handy too and I can try to reproduce locally for debugging.
Thanks,
Signature · 1 line
Joe
Le 14/12/2021 à 14:00, Joe Lawrence a écrit :
Quoted 110 lines · Christophe Leroy
Quoted 109 lines
Le 13/12/2021 à 18:26, Joe Lawrence a écrit :
Quoted 85 lines · Christophe Leroy
Quoted 58 lines
Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
Quoted 43 lines · Christophe Leroy
Quoted 28 lines
Hello Joe,
I'm implementing LIVEPATCH on PPC32 and I wanted to test with
STRICT_MODULE_RWXenabled so I took your branch as suggested, but I'm getting the following errors on build. What shall I do ?CALL scripts/checksyscalls.sh CALL scripts/atomic/check-atomics.sh CHK include/generated/compile.h KLP lib/livepatch/test_klp_convert1.koklp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyond
nr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert1.ko] Error 255
KLPlib/livepatch/test_klp_convert2.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesklp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert2.ko] Error 255
KLPlib/livepatch/test_klp_convert_sections.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length beyondnr_entriesQuoted 7 lines
klp-convert: Unable to load user-provided sympos make[2]: *** [
scripts/Makefile.modfinal:79:lib/livepatch/test_klp_convert_sections.ko] Error 255 make[2]: Target '__modfinal' not remade because of errors. make[1]: *** [scripts/Makefile.modpost:145:__modpost] Error 2
make: *** [Makefile:1770: modules] Error 2Hi Christophe,
Interesting failure mode. That's klp-convert complaining that it found more relocations in a .klp.module_relocs.<objname> section than expected, i.e.
nr_entries=sec->size/sizeof(struct klp_module_reloc).Quoted 9 lines
A few possibilities: the ELF
sec->sizewas incorrectly set/read by build/libelf (I doubt that). Or maybe the layout/size of structklp_module_relocis not consistent between kernel and userspace (I'm more suspicious of this).Can you post a copy of the build's symbols.klp and
lib/livepatch/test_klp_convert1.tmp.kosomewhere? I should be able to start debug with those files.I sent you both files off list.
Quoted 9 lines
It looks like klp-convert doesn't use the correct size. It finds a struct of size 12 hence 3 entries for a section of size 40.
On PPC32 the struct has size 8 (void * is 4 and int is 4).
But I'm cross-building from
x86_64where the struct is 8 + 4 = 12.Can it be the reason ?
I'm pretty sure that is it. I haven't had much runtime with klp-convert and cross-building (I've only found one big/little endian bug with
x86_64->s390x) and was going to ask you how you were testing :)Do you know if there are other kernel build tools that deal with similar situations? This seems like a tricky job for the userspace build tool to determine non-native target struct layout.
In the meantime, hacking in:
struct klp_module_reloc {(quoted diff) +1
-1- void *sym; + uint32_t sym; unsigned int sympos; } __packed;Quoted 7 lines
gets me generating an output .ko file, but the readelf output doesn't look right.
I'll add this to the patchset TODO list, but may not get to it for a while -- is there any chance the above hack works or could you test a local non-cross build?
No I have no way to do a non-cross build. My target is an embedded board with slow CPU and little memory.
I tested with your hack, I get:
root@vgoip:~# insmod /lib/modules/test_klp_convert1.ko insmod: can't insert '/lib/modules/test_klp_convert1.ko': unknown symbol in module, or unknown parameter root@vgoip:~# insmod /lib/modules/test_klp_livepatch.ko insmod: can't insert '/lib/modules/test_klp_livepatch.ko': unknown symbol in module, or unknown parameter
Quoted 7 lines
Quoted 2 lines
I agree with you readelf shows something went wrong with relocations.
Thanks for trying that. Can you point me to the cross-compiler suite that you are using for build and readelf? Kernel .config would be handy too and I can try to reproduce locally for debugging.
Livepatch on PPC32 is WIP, branch 'livepatch' on
https://github.com/chleroy/linux.git.
You can use branch 'joe' on https://github.com/chleroy/linux.git which is a merge of your branch klp-convert-v5-expanded-v5.16-rebase1 with my livepatch branch.
defconfig is attached
Thanks for your help
Christophe
# CONFIG_SWAP is not setCONFIG_SYSVIPC=yCONFIG_NO_HZ=yCONFIG_HIGH_RES_TIMERS=yCONFIG_BPF_JIT=yCONFIG_VIRT_CPU_ACCOUNTING_NATIVE=yCONFIG_LOG_BUF_SHIFT=14CONFIG_EXPERT=y
# CONFIG_ELF_CORE is not set
# CONFIG_BASE_FULL is not set
# CONFIG_FUTEX is not setCONFIG_KALLSYMS_ALL=yCONFIG_PERF_EVENTS=y
# CONFIG_VM_EVENT_COUNTERS is not set
CONFIG_PPC_8xx=y
CONFIG_8xx_GPIO=yCONFIG_SMC_UCODE_PATCH=yCONFIG_PIN_TLB=yCONFIG_GEN_RTC=yCONFIG_HZ_100=yCONFIG_MATH_EMULATION=yCONFIG_PPC_16K_PAGES=yCONFIG_ADVANCED_OPTIONS=yCONFIG_LIVEPATCH=y
# CONFIG_SECCOMP is not setCONFIG_MODULES=yCONFIG_PARTITION_ADVANCED=yCONFIG_NET=yCONFIG_PACKET=yCONFIG_UNIX=yCONFIG_INET=yCONFIG_IP_MULTICAST=yCONFIG_IP_PNP=yCONFIG_SYN_COOKIES=y
# CONFIG_FW_LOADER is not setCONFIG_MTD=yCONFIG_MTD_BLOCK=yCONFIG_MTD_JEDECPROBE=yCONFIG_MTD_CFI_ADV_OPTIONS=yCONFIG_MTD_CFI_GEOMETRY=y
# CONFIG_MTD_MAP_BANK_WIDTH_1 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_2 is not set
# CONFIG_MTD_CFI_I1 is not set
# CONFIG_MTD_CFI_I2 is not setCONFIG_MTD_CFI_I4=yCONFIG_MTD_CFI_AMDSTD=yCONFIG_MTD_PHYSMAP=yCONFIG_MTD_PHYSMAP_OF=y
# CONFIG_BLK_DEV is not setCONFIG_NETDEVICES=yCONFIG_FS_ENET=y
# CONFIG_FS_ENET_HAS_SCC is not setCONFIG_DAVICOM_PHY=y
# CONFIG_INPUT is not set
# CONFIG_SERIO is not set
# CONFIG_VT is not set
# CONFIG_LEGACY_PTYS is not setCONFIG_SERIAL_CPM=yCONFIG_SERIAL_CPM_CONSOLE=yCONFIG_SPI=yCONFIG_SPI_FSL_SPI=y
# CONFIG_HWMON is not setCONFIG_WATCHDOG=y
CONFIG_8xxx_WDT=y
# CONFIG_USB_SUPPORT is not set
# CONFIG_DNOTIFY is not setCONFIG_TMPFS=yCONFIG_CRAMFS=yCONFIG_NFS_FS=yCONFIG_ROOT_NFS=yCONFIG_CRYPTO=yCONFIG_CRYPTO_DEV_TALITOS=yCONFIG_CRC32_SLICEBY4=yCONFIG_DYNAMIC_DEBUG=yCONFIG_DEBUG_INFO=yCONFIG_MAGIC_SYSRQ=yCONFIG_DEBUG_FS=yCONFIG_DEBUG_VM_PGTABLE=yCONFIG_DETECT_HUNG_TASK=yCONFIG_FUNCTION_TRACER=yCONFIG_FTRACE_STARTUP_TEST=y
# CONFIG_EVENT_TRACE_STARTUP_TEST is not setCONFIG_BDI_SWITCH=yCONFIG_PPC_EARLY_DEBUG=yCONFIG_TEST_LIVEPATCH=m
Hi Russell,
Quoted 27 lines · Russell Currey
Quoted 7 lines · Joe Lawrence
Starting with 5.14 kernels, I can reliably reproduce a crash [1] on ppc64le when loading livepatches containing late klp-relocations [2].
These are relocations, specific to livepatching, that are resolved not
when a livepatch module is loaded, but only when a livepatch-target module is loaded.Hey Joe, thanks for the report.
Quoted 6 lines
I haven't started looking at a fix yet, but in the case of the x86 code
update, itsapply_relocate_add()implementation was modified to use a
commontext_poke()function to allowed us to drop module_{en,dis}ble_ro() games by the livepatching code.It should be a similar fix for Power, our
patch_instruction()uses a text poke area butapply_relocate_add()doesn't use it and does its own
raw patching instead.Quoted 3 lines
I can take a closer look this week, but thought I'd send out a report
in case this may be a known todo forSTRICT_MODULE_RWXon Power.
Quoted 7 lines
I'm looking into this now, will update when there's progress. I personally wasn't aware but Jordan flagged this as an issue back in August [0]. Are the selftests in the klp-convert tree sufficient for testing? I'm not especially familiar with livepatching & haven't used
the userspace tools.
You can test this by livepatching any module since this only occurs when writing relocations for modules since the vmlinux relocations are written earlier before the module text is mapped read-only.
- Suraj
Quoted 8 lines
- Russell
[0] https://github.com/linuxppc/issues/issues/375
Quoted 2 lines
-- Joe