Skip to contentSkip to conversation

Linux 4.9.93

DISCUSSION Jean-Baptiste Theou 7 participants 13 messages Apr 9, 2018

lore
Share & tools
lore
#0 lore raw

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

Wondering if there is one of the test suite used on the review patchset that covers the CFI usecase.

Best regards,

[    0.249191] CPU features: detected feature: GIC system register CPU interface
[    0.256391] CPU features: detected feature: Privileged Access Never
[    0.262719] CPU features: detected feature: User Access Override
[    0.268791] CPU features: detected feature: 32-bit EL0 Support
[    0.274683] CPU features: detected feature: Kernel page table isolation (KPTI)
[    0.282166] CFI failure:
[    0.282169] CFI failure:
[    0.282172] CFI failure:
[    0.282173] CFI failure:
[    0.282175] CFI failure:
[    0.282176] CFI failure:
[    0.282177] CFI failure:
[    0.282178] CFI failure:
[    0.282188] ------------[ cut here ]------------
[    0.282189] ------------[ cut here ]------------
[    0.282190] ------------[ cut here ]------------
[    0.282191] ------------[ cut here ]------------
[    0.282193] ------------[ cut here ]------------
[    0.282196] kernel BUG at kernel/cfi.c:32!
[    0.282198] ------------[ cut here ]------------
[    0.282201] kernel BUG at kernel/cfi.c:32!
[    0.282202] ------------[ cut here ]------------
[    0.282204] kernel BUG at kernel/cfi.c:32!
[    0.282207] kernel BUG at kernel/cfi.c:32!
[    0.282209] kernel BUG at kernel/cfi.c:32!
[    0.282211] kernel BUG at kernel/cfi.c:32!
[    0.282214] kernel BUG at kernel/cfi.c:32!
[    0.282215] ------------[ cut here ]------------
[    0.282216] kernel BUG at kernel/cfi.c:32!
[    0.282218] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[    0.282224] Modules linked in:
[    0.282230] CPU: 2 PID: 25 Comm: migration/2 Not tainted 4.9.93-perf+ #39
[    0.282232] Hardware name: <REMOVED>
[    0.282235] task: fffffffbb3b36580 task.stack: fffffffbb30cc000
[    0.282250] PC is at __cfi_check_fail+0x14/0x1c
[    0.282253] LR is at __cfi_check_fail+0x14/0x1c
[    0.282255] pc : [<ffffff93b3f03d90>] lr : [<ffffff93b3f03d90>] pstate: 60c00085
[    0.282256] sp : fffffffbb30cfc30
[    0.282259] x29: fffffffbb30cfc30 x28: ffffff93b6415000 
[    0.282261] x27: 00000013b65c1000 x26: ffffff93b5ce6000 
[    0.282264] x25: ffffff93b5ce6000 x24: ffffff93b6419000 
[    0.282266] x23: ffffff93b65c1000 x22: ffffff93b65c4000 
[    0.282268] x21: 9d12f8172cb2f296 x20: 000000008180e3e0 
[    0.282271] x19: 0000000000000000 x18: 000000000000002c 
[    0.282274] x17: 00000000000fd054 x16: 0000000000000000 
[    0.282276] x15: ffffff93b65ec000 x14: 000000000000000c 
[    0.282279] x13: 0000000000000004 x12: 0000000000000000 
[    0.282281] x11: 0000000000000000 x10: 0000000001440144 
[    0.282283] x9 : 260822e8751d5000 x8 : 260822e8751d5000 
[    0.282286] x7 : 0000000000000000 x6 : fffffffbbac75b60 
[    0.282288] x5 : 0000000000000000 x4 : 0000000000000000 
[    0.282290] x3 : 000000003a657275 x2 : 0000000000000000 
[    0.282292] x1 : 0000000000000000 x0 : 000000000000000c 
[    0.282294] 
[    0.282294] PC: 0xffffff93b3f03d50:
[    0.282308] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282315] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282322] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282329] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282330] 
[    0.282330] LR: 0xffffff93b3f03d50:
[    0.282336] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282343] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282350] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282357] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282358] 
[    0.282358] SP: 0xfffffffbb30cfbf0:
[    0.282365] fbf0  b3f03d90 ffffff93 b30cfc30 fffffffb b3f03d90 ffffff93 60c00085 00000000
[    0.282372] fc10  b6415000 ffffff93 b642fa00 ffffff93 ffffffff ffffffff b3f03d90 ffffff93
[    0.282378] fc30  b30cfc70 fffffffb b3d458c0 ffffff93 00000080 00000000 00000001 00000000
[    0.282385] fc50  b65c4000 ffffff93 b64420f0 ffffff93 8180e3e0 00000000 00000002 00000000
[    0.282387] Process migration/2 (pid: 25, stack limit = 0xfffffffbb30cc000)
[    0.282389] Call trace:
[    0.282391] Exception stack(0xfffffffbb30cfb00 to 0xfffffffbb30cfc30)
[    0.282395] fb00: 000000000000000c 0000000000000000 0000000000000000 000000003a657275
[    0.282397] fb20: 0000000000000000 0000000000000000 fffffffbbac75b60 0000000000000000
[    0.282400] fb40: 260822e8751d5000 260822e8751d5000 0000000001440144 0000000000000000
[    0.282403] fb60: 0000000000000000 0000000000000004 000000000000000c ffffff93b65ec000
[    0.282405] fb80: 0000000000000000 00000000000fd054 000000000000002c 0000000000000000
[    0.282408] fba0: 000000008180e3e0 9d12f8172cb2f296 ffffff93b65c4000 ffffff93b65c1000
[    0.282411] fbc0: ffffff93b6419000 ffffff93b5ce6000 ffffff93b5ce6000 00000013b65c1000
[    0.282413] fbe0: ffffff93b6415000 fffffffbb30cfc30 ffffff93b3f03d90 fffffffbb30cfc30
[    0.282416] fc00: ffffff93b3f03d90 0000000060c00085 ffffff93b6415000 ffffff93b642fa00
[    0.282418] fc20: ffffffffffffffff ffffff93b3f03d90
[    0.282421] [<ffffff93b3f03d90>] __cfi_check_fail+0x14/0x1c
[    0.282430] [<ffffff93b3d458c0>] name_to_dev_t+0x0/0x47c
[    0.282436] [<ffffff93b3d51b80>] kpti_install_ng_mappings+0x178/0x2e0
[    0.282443] [<ffffff93b3eae950>] multi_cpu_stop+0x114/0x170
[    0.282445] [<ffffff93b3eaf08c>] cpu_stopper_thread+0x128/0x2e8
[    0.282452] [<ffffff93b3db5504>] smpboot_thread_fn+0x230/0x558
[    0.282455] [<ffffff93b3dae848>] kthread+0x21c/0x238
[    0.282459] [<ffffff93b3c838f0>] ret_from_fork+0x10/0x20
[    0.282464] Code: 910003fd d000d100 913ee400 94533cc7 (d4210000) 
#1 lore raw
Quoted 4 lines · Jean-Baptiste Theou

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

Is this a "clean" 4.9.93 tree or a "4.9.93 merged into android-common-4.9?

Quoted 89 lines

Wondering if there is one of the test suite used on the review patchset that covers the CFI usecase.

Best regards,

[    0.249191] CPU features: detected feature: GIC system register CPU interface
[    0.256391] CPU features: detected feature: Privileged Access Never
[    0.262719] CPU features: detected feature: User Access Override
[    0.268791] CPU features: detected feature: 32-bit EL0 Support
[    0.274683] CPU features: detected feature: Kernel page table isolation (KPTI)
[    0.282166] CFI failure:
[    0.282169] CFI failure:
[    0.282172] CFI failure:
[    0.282173] CFI failure:
[    0.282175] CFI failure:
[    0.282176] CFI failure:
[    0.282177] CFI failure:
[    0.282178] CFI failure:
[    0.282188] ------------[ cut here ]------------
[    0.282189] ------------[ cut here ]------------
[    0.282190] ------------[ cut here ]------------
[    0.282191] ------------[ cut here ]------------
[    0.282193] ------------[ cut here ]------------
[    0.282196] kernel BUG at kernel/cfi.c:32!
[    0.282198] ------------[ cut here ]------------
[    0.282201] kernel BUG at kernel/cfi.c:32!
[    0.282202] ------------[ cut here ]------------
[    0.282204] kernel BUG at kernel/cfi.c:32!
[    0.282207] kernel BUG at kernel/cfi.c:32!
[    0.282209] kernel BUG at kernel/cfi.c:32!
[    0.282211] kernel BUG at kernel/cfi.c:32!
[    0.282214] kernel BUG at kernel/cfi.c:32!
[    0.282215] ------------[ cut here ]------------
[    0.282216] kernel BUG at kernel/cfi.c:32!
[    0.282218] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[    0.282224] Modules linked in:
[    0.282230] CPU: 2 PID: 25 Comm: migration/2 Not tainted 4.9.93-perf+ #39
[    0.282232] Hardware name: <REMOVED>
[    0.282235] task: fffffffbb3b36580 task.stack: fffffffbb30cc000
[    0.282250] PC is at __cfi_check_fail+0x14/0x1c
[    0.282253] LR is at __cfi_check_fail+0x14/0x1c
[    0.282255] pc : [<ffffff93b3f03d90>] lr : [<ffffff93b3f03d90>] pstate: 60c00085
[    0.282256] sp : fffffffbb30cfc30
[    0.282259] x29: fffffffbb30cfc30 x28: ffffff93b6415000 
[    0.282261] x27: 00000013b65c1000 x26: ffffff93b5ce6000 
[    0.282264] x25: ffffff93b5ce6000 x24: ffffff93b6419000 
[    0.282266] x23: ffffff93b65c1000 x22: ffffff93b65c4000 
[    0.282268] x21: 9d12f8172cb2f296 x20: 000000008180e3e0 
[    0.282271] x19: 0000000000000000 x18: 000000000000002c 
[    0.282274] x17: 00000000000fd054 x16: 0000000000000000 
[    0.282276] x15: ffffff93b65ec000 x14: 000000000000000c 
[    0.282279] x13: 0000000000000004 x12: 0000000000000000 
[    0.282281] x11: 0000000000000000 x10: 0000000001440144 
[    0.282283] x9 : 260822e8751d5000 x8 : 260822e8751d5000 
[    0.282286] x7 : 0000000000000000 x6 : fffffffbbac75b60 
[    0.282288] x5 : 0000000000000000 x4 : 0000000000000000 
[    0.282290] x3 : 000000003a657275 x2 : 0000000000000000 
[    0.282292] x1 : 0000000000000000 x0 : 000000000000000c 
[    0.282294] 
[    0.282294] PC: 0xffffff93b3f03d50:
[    0.282308] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282315] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282322] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282329] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282330] 
[    0.282330] LR: 0xffffff93b3f03d50:
[    0.282336] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282343] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282350] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282357] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282358] 
[    0.282358] SP: 0xfffffffbb30cfbf0:
[    0.282365] fbf0  b3f03d90 ffffff93 b30cfc30 fffffffb b3f03d90 ffffff93 60c00085 00000000
[    0.282372] fc10  b6415000 ffffff93 b642fa00 ffffff93 ffffffff ffffffff b3f03d90 ffffff93
[    0.282378] fc30  b30cfc70 fffffffb b3d458c0 ffffff93 00000080 00000000 00000001 00000000
[    0.282385] fc50  b65c4000 ffffff93 b64420f0 ffffff93 8180e3e0 00000000 00000002 00000000
[    0.282387] Process migration/2 (pid: 25, stack limit = 0xfffffffbb30cc000)
[    0.282389] Call trace:
[    0.282391] Exception stack(0xfffffffbb30cfb00 to 0xfffffffbb30cfc30)
[    0.282395] fb00: 000000000000000c 0000000000000000 0000000000000000 000000003a657275
[    0.282397] fb20: 0000000000000000 0000000000000000 fffffffbbac75b60 0000000000000000
[    0.282400] fb40: 260822e8751d5000 260822e8751d5000 0000000001440144 0000000000000000
[    0.282403] fb60: 0000000000000000 0000000000000004 000000000000000c ffffff93b65ec000
[    0.282405] fb80: 0000000000000000 00000000000fd054 000000000000002c 0000000000000000
[    0.282408] fba0: 000000008180e3e0 9d12f8172cb2f296 ffffff93b65c4000 ffffff93b65c1000
[    0.282411] fbc0: ffffff93b6419000 ffffff93b5ce6000 ffffff93b5ce6000 00000013b65c1000
[    0.282413] fbe0: ffffff93b6415000 fffffffbb30cfc30 ffffff93b3f03d90 fffffffbb30cfc30
[    0.282416] fc00: ffffff93b3f03d90 0000000060c00085 ffffff93b6415000 ffffff93b642fa00
[    0.282418] fc20: ffffffffffffffff ffffff93b3f03d90
[    0.282421] [<ffffff93b3f03d90>] __cfi_check_fail+0x14/0x1c
Quoted 8 lines
[    0.282430] [<ffffff93b3d458c0>] name_to_dev_t+0x0/0x47c
[    0.282436] [<ffffff93b3d51b80>] kpti_install_ng_mappings+0x178/0x2e0
[    0.282443] [<ffffff93b3eae950>] multi_cpu_stop+0x114/0x170
[    0.282445] [<ffffff93b3eaf08c>] cpu_stopper_thread+0x128/0x2e8
[    0.282452] [<ffffff93b3db5504>] smpboot_thread_fn+0x230/0x558
[    0.282455] [<ffffff93b3dae848>] kthread+0x21c/0x238
[    0.282459] [<ffffff93b3c838f0>] ret_from_fork+0x10/0x20
[    0.282464] Code: 910003fd d000d100 913ee400 94533cc7 (d4210000) 

Oh fun :(

Dragging in a bunch more people to the cc: and to: lines to have them look at this...

thanks,

greg k-h

#2 lore raw
Quoted 98 lines · Greg KH
Quoted 4 lines · Jean-Baptiste Theou

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

Is this a "clean" 4.9.93 tree or a "4.9.93 merged into android-common-4.9?

Quoted 89 lines

Wondering if there is one of the test suite used on the review patchset that covers the CFI usecase.

Best regards,

[    0.249191] CPU features: detected feature: GIC system register CPU interface
[    0.256391] CPU features: detected feature: Privileged Access Never
[    0.262719] CPU features: detected feature: User Access Override
[    0.268791] CPU features: detected feature: 32-bit EL0 Support
[    0.274683] CPU features: detected feature: Kernel page table isolation (KPTI)
[    0.282166] CFI failure:
[    0.282169] CFI failure:
[    0.282172] CFI failure:
[    0.282173] CFI failure:
[    0.282175] CFI failure:
[    0.282176] CFI failure:
[    0.282177] CFI failure:
[    0.282178] CFI failure:
[    0.282188] ------------[ cut here ]------------
[    0.282189] ------------[ cut here ]------------
[    0.282190] ------------[ cut here ]------------
[    0.282191] ------------[ cut here ]------------
[    0.282193] ------------[ cut here ]------------
[    0.282196] kernel BUG at kernel/cfi.c:32!
[    0.282198] ------------[ cut here ]------------
[    0.282201] kernel BUG at kernel/cfi.c:32!
[    0.282202] ------------[ cut here ]------------
[    0.282204] kernel BUG at kernel/cfi.c:32!
[    0.282207] kernel BUG at kernel/cfi.c:32!
[    0.282209] kernel BUG at kernel/cfi.c:32!
[    0.282211] kernel BUG at kernel/cfi.c:32!
[    0.282214] kernel BUG at kernel/cfi.c:32!
[    0.282215] ------------[ cut here ]------------
[    0.282216] kernel BUG at kernel/cfi.c:32!
[    0.282218] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[    0.282224] Modules linked in:
[    0.282230] CPU: 2 PID: 25 Comm: migration/2 Not tainted 4.9.93-perf+ #39
[    0.282232] Hardware name: <REMOVED>
[    0.282235] task: fffffffbb3b36580 task.stack: fffffffbb30cc000
[    0.282250] PC is at __cfi_check_fail+0x14/0x1c
[    0.282253] LR is at __cfi_check_fail+0x14/0x1c
[    0.282255] pc : [<ffffff93b3f03d90>] lr : [<ffffff93b3f03d90>] pstate: 60c00085
[    0.282256] sp : fffffffbb30cfc30
[    0.282259] x29: fffffffbb30cfc30 x28: ffffff93b6415000 
[    0.282261] x27: 00000013b65c1000 x26: ffffff93b5ce6000 
[    0.282264] x25: ffffff93b5ce6000 x24: ffffff93b6419000 
[    0.282266] x23: ffffff93b65c1000 x22: ffffff93b65c4000 
[    0.282268] x21: 9d12f8172cb2f296 x20: 000000008180e3e0 
[    0.282271] x19: 0000000000000000 x18: 000000000000002c 
[    0.282274] x17: 00000000000fd054 x16: 0000000000000000 
[    0.282276] x15: ffffff93b65ec000 x14: 000000000000000c 
[    0.282279] x13: 0000000000000004 x12: 0000000000000000 
[    0.282281] x11: 0000000000000000 x10: 0000000001440144 
[    0.282283] x9 : 260822e8751d5000 x8 : 260822e8751d5000 
[    0.282286] x7 : 0000000000000000 x6 : fffffffbbac75b60 
[    0.282288] x5 : 0000000000000000 x4 : 0000000000000000 
[    0.282290] x3 : 000000003a657275 x2 : 0000000000000000 
[    0.282292] x1 : 0000000000000000 x0 : 000000000000000c 
[    0.282294] 
[    0.282294] PC: 0xffffff93b3f03d50:
[    0.282308] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282315] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282322] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282329] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282330] 
[    0.282330] LR: 0xffffff93b3f03d50:
[    0.282336] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282343] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282350] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282357] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282358] 
[    0.282358] SP: 0xfffffffbb30cfbf0:
[    0.282365] fbf0  b3f03d90 ffffff93 b30cfc30 fffffffb b3f03d90 ffffff93 60c00085 00000000
[    0.282372] fc10  b6415000 ffffff93 b642fa00 ffffff93 ffffffff ffffffff b3f03d90 ffffff93
[    0.282378] fc30  b30cfc70 fffffffb b3d458c0 ffffff93 00000080 00000000 00000001 00000000
[    0.282385] fc50  b65c4000 ffffff93 b64420f0 ffffff93 8180e3e0 00000000 00000002 00000000
[    0.282387] Process migration/2 (pid: 25, stack limit = 0xfffffffbb30cc000)
[    0.282389] Call trace:
[    0.282391] Exception stack(0xfffffffbb30cfb00 to 0xfffffffbb30cfc30)
[    0.282395] fb00: 000000000000000c 0000000000000000 0000000000000000 000000003a657275
[    0.282397] fb20: 0000000000000000 0000000000000000 fffffffbbac75b60 0000000000000000
[    0.282400] fb40: 260822e8751d5000 260822e8751d5000 0000000001440144 0000000000000000
[    0.282403] fb60: 0000000000000000 0000000000000004 000000000000000c ffffff93b65ec000
[    0.282405] fb80: 0000000000000000 00000000000fd054 000000000000002c 0000000000000000
[    0.282408] fba0: 000000008180e3e0 9d12f8172cb2f296 ffffff93b65c4000 ffffff93b65c1000
[    0.282411] fbc0: ffffff93b6419000 ffffff93b5ce6000 ffffff93b5ce6000 00000013b65c1000
[    0.282413] fbe0: ffffff93b6415000 fffffffbb30cfc30 ffffff93b3f03d90 fffffffbb30cfc30
[    0.282416] fc00: ffffff93b3f03d90 0000000060c00085 ffffff93b6415000 ffffff93b642fa00
[    0.282418] fc20: ffffffffffffffff ffffff93b3f03d90
[    0.282421] [<ffffff93b3f03d90>] __cfi_check_fail+0x14/0x1c

Where is this coming from? Out of tree patches?

Quoted 9 lines
Quoted 8 lines
[    0.282430] [<ffffff93b3d458c0>] name_to_dev_t+0x0/0x47c
[    0.282436] [<ffffff93b3d51b80>] kpti_install_ng_mappings+0x178/0x2e0
[    0.282443] [<ffffff93b3eae950>] multi_cpu_stop+0x114/0x170
[    0.282445] [<ffffff93b3eaf08c>] cpu_stopper_thread+0x128/0x2e8
[    0.282452] [<ffffff93b3db5504>] smpboot_thread_fn+0x230/0x558
[    0.282455] [<ffffff93b3dae848>] kthread+0x21c/0x238
[    0.282459] [<ffffff93b3c838f0>] ret_from_fork+0x10/0x20
[    0.282464] Code: 910003fd d000d100 913ee400 94533cc7 (d4210000) 
Quoted 4 lines

Oh fun :(

Dragging in a bunch more people to the cc: and to: lines to have them look at this...

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#3 lore raw
In reply to Greg KH
Quoted 5 lines · Greg KH
Quoted 4 lines · Jean-Baptiste Theou

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

How does one 'build a kernel with CFI' for arm64?

Quoted 110 lines

Is this a "clean" 4.9.93 tree or a "4.9.93 merged into android-common-4.9?

Quoted 89 lines

Wondering if there is one of the test suite used on the review patchset that covers the CFI usecase.

Best regards,

[    0.249191] CPU features: detected feature: GIC system register CPU interface
[    0.256391] CPU features: detected feature: Privileged Access Never
[    0.262719] CPU features: detected feature: User Access Override
[    0.268791] CPU features: detected feature: 32-bit EL0 Support
[    0.274683] CPU features: detected feature: Kernel page table isolation (KPTI)
[    0.282166] CFI failure:
[    0.282169] CFI failure:
[    0.282172] CFI failure:
[    0.282173] CFI failure:
[    0.282175] CFI failure:
[    0.282176] CFI failure:
[    0.282177] CFI failure:
[    0.282178] CFI failure:
[    0.282188] ------------[ cut here ]------------
[    0.282189] ------------[ cut here ]------------
[    0.282190] ------------[ cut here ]------------
[    0.282191] ------------[ cut here ]------------
[    0.282193] ------------[ cut here ]------------
[    0.282196] kernel BUG at kernel/cfi.c:32!
[    0.282198] ------------[ cut here ]------------
[    0.282201] kernel BUG at kernel/cfi.c:32!
[    0.282202] ------------[ cut here ]------------
[    0.282204] kernel BUG at kernel/cfi.c:32!
[    0.282207] kernel BUG at kernel/cfi.c:32!
[    0.282209] kernel BUG at kernel/cfi.c:32!
[    0.282211] kernel BUG at kernel/cfi.c:32!
[    0.282214] kernel BUG at kernel/cfi.c:32!
[    0.282215] ------------[ cut here ]------------
[    0.282216] kernel BUG at kernel/cfi.c:32!
[    0.282218] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[    0.282224] Modules linked in:
[    0.282230] CPU: 2 PID: 25 Comm: migration/2 Not tainted 4.9.93-perf+ #39
[    0.282232] Hardware name: <REMOVED>
[    0.282235] task: fffffffbb3b36580 task.stack: fffffffbb30cc000
[    0.282250] PC is at __cfi_check_fail+0x14/0x1c
[    0.282253] LR is at __cfi_check_fail+0x14/0x1c
[    0.282255] pc : [<ffffff93b3f03d90>] lr : [<ffffff93b3f03d90>] pstate: 60c00085
[    0.282256] sp : fffffffbb30cfc30
[    0.282259] x29: fffffffbb30cfc30 x28: ffffff93b6415000
[    0.282261] x27: 00000013b65c1000 x26: ffffff93b5ce6000
[    0.282264] x25: ffffff93b5ce6000 x24: ffffff93b6419000
[    0.282266] x23: ffffff93b65c1000 x22: ffffff93b65c4000
[    0.282268] x21: 9d12f8172cb2f296 x20: 000000008180e3e0
[    0.282271] x19: 0000000000000000 x18: 000000000000002c
[    0.282274] x17: 00000000000fd054 x16: 0000000000000000
[    0.282276] x15: ffffff93b65ec000 x14: 000000000000000c
[    0.282279] x13: 0000000000000004 x12: 0000000000000000
[    0.282281] x11: 0000000000000000 x10: 0000000001440144
[    0.282283] x9 : 260822e8751d5000 x8 : 260822e8751d5000
[    0.282286] x7 : 0000000000000000 x6 : fffffffbbac75b60
[    0.282288] x5 : 0000000000000000 x4 : 0000000000000000
[    0.282290] x3 : 000000003a657275 x2 : 0000000000000000
[    0.282292] x1 : 0000000000000000 x0 : 000000000000000c
[    0.282294]
[    0.282294] PC: 0xffffff93b3f03d50:
[    0.282308] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282315] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282322] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282329] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282330]
[    0.282330] LR: 0xffffff93b3f03d50:
[    0.282336] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282343] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282350] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282357] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282358]
[    0.282358] SP: 0xfffffffbb30cfbf0:
[    0.282365] fbf0  b3f03d90 ffffff93 b30cfc30 fffffffb b3f03d90 ffffff93 60c00085 00000000
[    0.282372] fc10  b6415000 ffffff93 b642fa00 ffffff93 ffffffff ffffffff b3f03d90 ffffff93
[    0.282378] fc30  b30cfc70 fffffffb b3d458c0 ffffff93 00000080 00000000 00000001 00000000
[    0.282385] fc50  b65c4000 ffffff93 b64420f0 ffffff93 8180e3e0 00000000 00000002 00000000
[    0.282387] Process migration/2 (pid: 25, stack limit = 0xfffffffbb30cc000)
[    0.282389] Call trace:
[    0.282391] Exception stack(0xfffffffbb30cfb00 to 0xfffffffbb30cfc30)
[    0.282395] fb00: 000000000000000c 0000000000000000 0000000000000000 000000003a657275
[    0.282397] fb20: 0000000000000000 0000000000000000 fffffffbbac75b60 0000000000000000
[    0.282400] fb40: 260822e8751d5000 260822e8751d5000 0000000001440144 0000000000000000
[    0.282403] fb60: 0000000000000000 0000000000000004 000000000000000c ffffff93b65ec000
[    0.282405] fb80: 0000000000000000 00000000000fd054 000000000000002c 0000000000000000
[    0.282408] fba0: 000000008180e3e0 9d12f8172cb2f296 ffffff93b65c4000 ffffff93b65c1000
[    0.282411] fbc0: ffffff93b6419000 ffffff93b5ce6000 ffffff93b5ce6000 00000013b65c1000
[    0.282413] fbe0: ffffff93b6415000 fffffffbb30cfc30 ffffff93b3f03d90 fffffffbb30cfc30
[    0.282416] fc00: ffffff93b3f03d90 0000000060c00085 ffffff93b6415000 ffffff93b642fa00
[    0.282418] fc20: ffffffffffffffff ffffff93b3f03d90
[    0.282421] [<ffffff93b3f03d90>] __cfi_check_fail+0x14/0x1c
Quoted 8 lines
[    0.282430] [<ffffff93b3d458c0>] name_to_dev_t+0x0/0x47c
[    0.282436] [<ffffff93b3d51b80>] kpti_install_ng_mappings+0x178/0x2e0
[    0.282443] [<ffffff93b3eae950>] multi_cpu_stop+0x114/0x170
[    0.282445] [<ffffff93b3eaf08c>] cpu_stopper_thread+0x128/0x2e8
[    0.282452] [<ffffff93b3db5504>] smpboot_thread_fn+0x230/0x558
[    0.282455] [<ffffff93b3dae848>] kthread+0x21c/0x238
[    0.282459] [<ffffff93b3c838f0>] ret_from_fork+0x10/0x20
[    0.282464] Code: 910003fd d000d100 913ee400 94533cc7 (d4210000)

Oh fun :(

Dragging in a bunch more people to the cc: and to: lines to have them look at this...

thanks,

greg k-h

#4 lore raw
Quoted 8 lines · On Mon, 9 Apr 2018 11:49:37 +0200 Ard Biesheuvel <[email protected]>…
Quoted 5 lines · Greg KH
Quoted 4 lines · Jean-Baptiste Theou

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

How does one 'build a kernel with CFI' for arm64?

From Google work on Android-4.9

https://android.googlesource.com/kernel/common/+/00a195e7c0752ff5d65c9caadfbcc226270ca232

I am not sure what is the plan on their side to upstream (Greg?), but definitely useful to isolate actual issues.

Quoted 4 lines
Quoted 4 lines

Is this a "clean" 4.9.93 tree or a "4.9.93 merged into android-common-4.9?

It's a "clean 4.9.93" + whatever is needed for Clang/CFI support

My take is that CFI doesn't like

* void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper)

and

remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings);

Maybe just flag this function to not use CFI? I remember that Sami Tolvanen did similar changes.

I know it's a bit out of context since CFI support for ARM64 is not upstream yet, but unfortunate that an stable patchset trigger such failures.

Thanks a lot

Best regards

Quoted 107 lines
Quoted 107 lines
Quoted 90 lines

Wondering if there is one of the test suite used on the review patchset that covers the CFI usecase.

Best regards,

[    0.249191] CPU features: detected feature: GIC system register CPU interface
[    0.256391] CPU features: detected feature: Privileged Access Never
[    0.262719] CPU features: detected feature: User Access Override
[    0.268791] CPU features: detected feature: 32-bit EL0 Support
[    0.274683] CPU features: detected feature: Kernel page table isolation (KPTI)
[    0.282166] CFI failure:
[    0.282169] CFI failure:
[    0.282172] CFI failure:
[    0.282173] CFI failure:
[    0.282175] CFI failure:
[    0.282176] CFI failure:
[    0.282177] CFI failure:
[    0.282178] CFI failure:
[    0.282188] ------------[ cut here ]------------
[    0.282189] ------------[ cut here ]------------
[    0.282190] ------------[ cut here ]------------
[    0.282191] ------------[ cut here ]------------
[    0.282193] ------------[ cut here ]------------
[    0.282196] kernel BUG at kernel/cfi.c:32!
[    0.282198] ------------[ cut here ]------------
[    0.282201] kernel BUG at kernel/cfi.c:32!
[    0.282202] ------------[ cut here ]------------
[    0.282204] kernel BUG at kernel/cfi.c:32!
[    0.282207] kernel BUG at kernel/cfi.c:32!
[    0.282209] kernel BUG at kernel/cfi.c:32!
[    0.282211] kernel BUG at kernel/cfi.c:32!
[    0.282214] kernel BUG at kernel/cfi.c:32!
[    0.282215] ------------[ cut here ]------------
[    0.282216] kernel BUG at kernel/cfi.c:32!
[    0.282218] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[    0.282224] Modules linked in:
[    0.282230] CPU: 2 PID: 25 Comm: migration/2 Not tainted 4.9.93-perf+ #39
[    0.282232] Hardware name: <REMOVED>
[    0.282235] task: fffffffbb3b36580 task.stack: fffffffbb30cc000
[    0.282250] PC is at __cfi_check_fail+0x14/0x1c
[    0.282253] LR is at __cfi_check_fail+0x14/0x1c
[    0.282255] pc : [<ffffff93b3f03d90>] lr : [<ffffff93b3f03d90>] pstate: 60c00085
[    0.282256] sp : fffffffbb30cfc30
[    0.282259] x29: fffffffbb30cfc30 x28: ffffff93b6415000
[    0.282261] x27: 00000013b65c1000 x26: ffffff93b5ce6000
[    0.282264] x25: ffffff93b5ce6000 x24: ffffff93b6419000
[    0.282266] x23: ffffff93b65c1000 x22: ffffff93b65c4000
[    0.282268] x21: 9d12f8172cb2f296 x20: 000000008180e3e0
[    0.282271] x19: 0000000000000000 x18: 000000000000002c
[    0.282274] x17: 00000000000fd054 x16: 0000000000000000
[    0.282276] x15: ffffff93b65ec000 x14: 000000000000000c
[    0.282279] x13: 0000000000000004 x12: 0000000000000000
[    0.282281] x11: 0000000000000000 x10: 0000000001440144
[    0.282283] x9 : 260822e8751d5000 x8 : 260822e8751d5000
[    0.282286] x7 : 0000000000000000 x6 : fffffffbbac75b60
[    0.282288] x5 : 0000000000000000 x4 : 0000000000000000
[    0.282290] x3 : 000000003a657275 x2 : 0000000000000000
[    0.282292] x1 : 0000000000000000 x0 : 000000000000000c
[    0.282294]
[    0.282294] PC: 0xffffff93b3f03d50:
[    0.282308] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282315] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282322] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282329] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282330]
[    0.282330] LR: 0xffffff93b3f03d50:
[    0.282336] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282343] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282350] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282357] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282358]
[    0.282358] SP: 0xfffffffbb30cfbf0:
[    0.282365] fbf0  b3f03d90 ffffff93 b30cfc30 fffffffb b3f03d90 ffffff93 60c00085 00000000
[    0.282372] fc10  b6415000 ffffff93 b642fa00 ffffff93 ffffffff ffffffff b3f03d90 ffffff93
[    0.282378] fc30  b30cfc70 fffffffb b3d458c0 ffffff93 00000080 00000000 00000001 00000000
[    0.282385] fc50  b65c4000 ffffff93 b64420f0 ffffff93 8180e3e0 00000000 00000002 00000000
[    0.282387] Process migration/2 (pid: 25, stack limit = 0xfffffffbb30cc000)
[    0.282389] Call trace:
[    0.282391] Exception stack(0xfffffffbb30cfb00 to 0xfffffffbb30cfc30)
[    0.282395] fb00: 000000000000000c 0000000000000000 0000000000000000 000000003a657275
[    0.282397] fb20: 0000000000000000 0000000000000000 fffffffbbac75b60 0000000000000000
[    0.282400] fb40: 260822e8751d5000 260822e8751d5000 0000000001440144 0000000000000000
[    0.282403] fb60: 0000000000000000 0000000000000004 000000000000000c ffffff93b65ec000
[    0.282405] fb80: 0000000000000000 00000000000fd054 000000000000002c 0000000000000000
[    0.282408] fba0: 000000008180e3e0 9d12f8172cb2f296 ffffff93b65c4000 ffffff93b65c1000
[    0.282411] fbc0: ffffff93b6419000 ffffff93b5ce6000 ffffff93b5ce6000 00000013b65c1000
[    0.282413] fbe0: ffffff93b6415000 fffffffbb30cfc30 ffffff93b3f03d90 fffffffbb30cfc30
[    0.282416] fc00: ffffff93b3f03d90 0000000060c00085 ffffff93b6415000 ffffff93b642fa00
[    0.282418] fc20: ffffffffffffffff ffffff93b3f03d90
[    0.282421] [<ffffff93b3f03d90>] __cfi_check_fail+0x14/0x1c
Quoted 8 lines
[    0.282430] [<ffffff93b3d458c0>] name_to_dev_t+0x0/0x47c
[    0.282436] [<ffffff93b3d51b80>] kpti_install_ng_mappings+0x178/0x2e0
[    0.282443] [<ffffff93b3eae950>] multi_cpu_stop+0x114/0x170
[    0.282445] [<ffffff93b3eaf08c>] cpu_stopper_thread+0x128/0x2e8
[    0.282452] [<ffffff93b3db5504>] smpboot_thread_fn+0x230/0x558
[    0.282455] [<ffffff93b3dae848>] kthread+0x21c/0x238
[    0.282459] [<ffffff93b3c838f0>] ret_from_fork+0x10/0x20
[    0.282464] Code: 910003fd d000d100 913ee400 94533cc7 (d4210000)  

Oh fun :(

Dragging in a bunch more people to the cc: and to: lines to have them look at this...

thanks,

greg k-h

#5 lore raw
Quoted 28 lines · Jean-Baptiste Theou
Quoted 8 lines · On Mon, 9 Apr 2018 11:49:37 +0200 Ard Biesheuvel <[email protected]>…
Quoted 5 lines · Greg KH
Quoted 4 lines · Jean-Baptiste Theou

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

How does one 'build a kernel with CFI' for arm64?

From Google work on Android-4.9

https://android.googlesource.com/kernel/common/+/00a195e7c0752ff5d65c9caadfbcc226270ca232

I am not sure what is the plan on their side to upstream (Greg?), but definitely useful to isolate actual issues.

Quoted 2 lines
Quoted 2 lines

Is this a "clean" 4.9.93 tree or a "4.9.93 merged into android-common-4.9?

It's a "clean 4.9.93" + whatever is needed for Clang/CFI support

My take is that CFI doesn't like

* void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper)

Quoted 8 lines

and

remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings);

Maybe just flag this function to not use CFI? I remember that Sami Tolvanen did similar changes.

From a quick scan, it looks like CFI uses shadow memory for function

prologues. Since we're taking the PA of a function pointer, presumably this no longer maps to valid shadow.

I'd expect the same to apply to uses of cpu_replace_ttbr1(), but it looks like the only user of that is marked as __init, and that patch adds __nocfi to __init functions.

So you probably need to mark kpti_install_ng_mappings() as __nocfi.

Quoted 2 lines

I know it's a bit out of context since CFI support for ARM64 is not upstream yet, but unfortunate that an stable patchset trigger such failures.

This is simply the nature of out-of-tree code.

In future, it would be very helpful if you could provide context for out-of-tree patches in the initial report.

Thanks,
Mark.

#6 lore raw
Quoted 46 lines · On Mon, 9 Apr 2018 11:07:41 +0100 Mark Rutland <[email protected]> wrote:
Quoted 26 lines · Jean-Baptiste Theou
Quoted 8 lines · On Mon, 9 Apr 2018 11:49:37 +0200 Ard Biesheuvel <[email protected]>…
Quoted 5 lines · Greg KH
Quoted 4 lines · Jean-Baptiste Theou

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

How does one 'build a kernel with CFI' for arm64?

From Google work on Android-4.9

https://android.googlesource.com/kernel/common/+/00a195e7c0752ff5d65c9caadfbcc226270ca232

I am not sure what is the plan on their side to upstream (Greg?), but definitely useful to isolate actual issues.

Quoted 2 lines
Quoted 2 lines

Is this a "clean" 4.9.93 tree or a "4.9.93 merged into android-common-4.9?

It's a "clean 4.9.93" + whatever is needed for Clang/CFI support

My take is that CFI doesn't like

Quoted 8 lines

* void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper)

and

remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings);

Maybe just flag this function to not use CFI? I remember that Sami Tolvanen did similar changes.

From a quick scan, it looks like CFI uses shadow memory for function prologues. Since we're taking the PA of a function pointer, presumably this no longer maps to valid shadow.

I'd expect the same to apply to uses of cpu_replace_ttbr1(), but it looks like the only user of that is marked as __init, and that patch adds __nocfi to __init functions.

So you probably need to mark kpti_install_ng_mappings() as __nocfi.

Quoted 8 lines
Quoted 2 lines

I know it's a bit out of context since CFI support for ARM64 is not upstream yet, but unfortunate that an stable patchset trigger such failures.

This is simply the nature of out-of-tree code.

In future, it would be very helpful if you could provide context for out-of-tree patches in the initial report.

I can pass the initial CFI failures by tagging the function with __nocfi, but still face issues down the road. That said, it's out of tree, so my problem.

Will investigate.

Thanks a lot for the quick support.

Best regards

Quoted 2 lines

Thanks,
Mark.

#7 lore raw
In reply to Jean-Baptiste Theou
Quoted 36 lines · Jean-Baptiste Theou
Quoted 8 lines · On Mon, 9 Apr 2018 11:49:37 +0200 Ard Biesheuvel <[email protected]>…
Quoted 6 lines · Greg KH
Quoted 5 lines · Jean-Baptiste Theou

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

How does one 'build a kernel with CFI' for arm64?

From Google work on Android-4.9

https://android.googlesource.com/kernel/common/+/00a195e7c0752ff5d65c9caadfbcc226270ca232

I am not sure what is the plan on their side to upstream (Greg?), but definitely useful to isolate actual issues.

Quoted 4 lines
Quoted 4 lines

Is this a "clean" 4.9.93 tree or a "4.9.93 merged into android-common-4.9?

It's a "clean 4.9.93" + whatever is needed for Clang/CFI support

My take is that CFI doesn't like

* void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper)

and

Quoted 8 lines

remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings);

Maybe just flag this function to not use CFI? I remember that Sami Tolvanen did similar changes.

I know it's a bit out of context since CFI support for ARM64 is not upstream yet, but unfortunate that an stable patchset trigger such failures.

I am sorry but if you are implying that we should have tested these patches against the out of tree CFI code, I have to disappoint you: that is simply not upstream’s job, and if the Google engineers merged this into their v4.9 tree without proper testing, may I suggest that you report it to them instead?

OTOH, if that is not what you are implying, please ignore the rant :-)

Quoted 111 lines

Thanks a lot

Best regards

Quoted 107 lines
Quoted 107 lines
Quoted 90 lines

Wondering if there is one of the test suite used on the review patchset that covers the CFI usecase.

Best regards,

[    0.249191] CPU features: detected feature: GIC system register CPU interface
[    0.256391] CPU features: detected feature: Privileged Access Never
[    0.262719] CPU features: detected feature: User Access Override
[    0.268791] CPU features: detected feature: 32-bit EL0 Support
[    0.274683] CPU features: detected feature: Kernel page table isolation (KPTI)
[    0.282166] CFI failure:
[    0.282169] CFI failure:
[    0.282172] CFI failure:
[    0.282173] CFI failure:
[    0.282175] CFI failure:
[    0.282176] CFI failure:
[    0.282177] CFI failure:
[    0.282178] CFI failure:
[    0.282188] ------------[ cut here ]------------
[    0.282189] ------------[ cut here ]------------
[    0.282190] ------------[ cut here ]------------
[    0.282191] ------------[ cut here ]------------
[    0.282193] ------------[ cut here ]------------
[    0.282196] kernel BUG at kernel/cfi.c:32!
[    0.282198] ------------[ cut here ]------------
[    0.282201] kernel BUG at kernel/cfi.c:32!
[    0.282202] ------------[ cut here ]------------
[    0.282204] kernel BUG at kernel/cfi.c:32!
[    0.282207] kernel BUG at kernel/cfi.c:32!
[    0.282209] kernel BUG at kernel/cfi.c:32!
[    0.282211] kernel BUG at kernel/cfi.c:32!
[    0.282214] kernel BUG at kernel/cfi.c:32!
[    0.282215] ------------[ cut here ]------------
[    0.282216] kernel BUG at kernel/cfi.c:32!
[    0.282218] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[    0.282224] Modules linked in:
[    0.282230] CPU: 2 PID: 25 Comm: migration/2 Not tainted 4.9.93-perf+ #39
[    0.282232] Hardware name: <REMOVED>
[    0.282235] task: fffffffbb3b36580 task.stack: fffffffbb30cc000
[    0.282250] PC is at __cfi_check_fail+0x14/0x1c
[    0.282253] LR is at __cfi_check_fail+0x14/0x1c
[    0.282255] pc : [<ffffff93b3f03d90>] lr : [<ffffff93b3f03d90>] pstate: 60c00085
[    0.282256] sp : fffffffbb30cfc30
[    0.282259] x29: fffffffbb30cfc30 x28: ffffff93b6415000
[    0.282261] x27: 00000013b65c1000 x26: ffffff93b5ce6000
[    0.282264] x25: ffffff93b5ce6000 x24: ffffff93b6419000
[    0.282266] x23: ffffff93b65c1000 x22: ffffff93b65c4000
[    0.282268] x21: 9d12f8172cb2f296 x20: 000000008180e3e0
[    0.282271] x19: 0000000000000000 x18: 000000000000002c
[    0.282274] x17: 00000000000fd054 x16: 0000000000000000
[    0.282276] x15: ffffff93b65ec000 x14: 000000000000000c
[    0.282279] x13: 0000000000000004 x12: 0000000000000000
[    0.282281] x11: 0000000000000000 x10: 0000000001440144
[    0.282283] x9 : 260822e8751d5000 x8 : 260822e8751d5000
[    0.282286] x7 : 0000000000000000 x6 : fffffffbbac75b60
[    0.282288] x5 : 0000000000000000 x4 : 0000000000000000
[    0.282290] x3 : 000000003a657275 x2 : 0000000000000000
[    0.282292] x1 : 0000000000000000 x0 : 000000000000000c
[    0.282294]
[    0.282294] PC: 0xffffff93b3f03d50:
[    0.282308] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282315] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282322] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282329] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282330]
[    0.282330] LR: 0xffffff93b3f03d50:
[    0.282336] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282343] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282350] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282357] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282358]
[    0.282358] SP: 0xfffffffbb30cfbf0:
[    0.282365] fbf0  b3f03d90 ffffff93 b30cfc30 fffffffb b3f03d90 ffffff93 60c00085 00000000
[    0.282372] fc10  b6415000 ffffff93 b642fa00 ffffff93 ffffffff ffffffff b3f03d90 ffffff93
[    0.282378] fc30  b30cfc70 fffffffb b3d458c0 ffffff93 00000080 00000000 00000001 00000000
[    0.282385] fc50  b65c4000 ffffff93 b64420f0 ffffff93 8180e3e0 00000000 00000002 00000000
[    0.282387] Process migration/2 (pid: 25, stack limit = 0xfffffffbb30cc000)
[    0.282389] Call trace:
[    0.282391] Exception stack(0xfffffffbb30cfb00 to 0xfffffffbb30cfc30)
[    0.282395] fb00: 000000000000000c 0000000000000000 0000000000000000 000000003a657275
[    0.282397] fb20: 0000000000000000 0000000000000000 fffffffbbac75b60 0000000000000000
[    0.282400] fb40: 260822e8751d5000 260822e8751d5000 0000000001440144 0000000000000000
[    0.282403] fb60: 0000000000000000 0000000000000004 000000000000000c ffffff93b65ec000
[    0.282405] fb80: 0000000000000000 00000000000fd054 000000000000002c 0000000000000000
[    0.282408] fba0: 000000008180e3e0 9d12f8172cb2f296 ffffff93b65c4000 ffffff93b65c1000
[    0.282411] fbc0: ffffff93b6419000 ffffff93b5ce6000 ffffff93b5ce6000 00000013b65c1000
[    0.282413] fbe0: ffffff93b6415000 fffffffbb30cfc30 ffffff93b3f03d90 fffffffbb30cfc30
[    0.282416] fc00: ffffff93b3f03d90 0000000060c00085 ffffff93b6415000 ffffff93b642fa00
[    0.282418] fc20: ffffffffffffffff ffffff93b3f03d90
[    0.282421] [<ffffff93b3f03d90>] __cfi_check_fail+0x14/0x1c
Quoted 8 lines
[    0.282430] [<ffffff93b3d458c0>] name_to_dev_t+0x0/0x47c
[    0.282436] [<ffffff93b3d51b80>] kpti_install_ng_mappings+0x178/0x2e0
[    0.282443] [<ffffff93b3eae950>] multi_cpu_stop+0x114/0x170
[    0.282445] [<ffffff93b3eaf08c>] cpu_stopper_thread+0x128/0x2e8
[    0.282452] [<ffffff93b3db5504>] smpboot_thread_fn+0x230/0x558
[    0.282455] [<ffffff93b3dae848>] kthread+0x21c/0x238
[    0.282459] [<ffffff93b3c838f0>] ret_from_fork+0x10/0x20
[    0.282464] Code: 910003fd d000d100 913ee400 94533cc7 (d4210000)  

Oh fun :(

Dragging in a bunch more people to the cc: and to: lines to have them look at this...

thanks,

greg k-h

#8 lore raw
Quoted 39 lines · On Mon, 9 Apr 2018 12:25:07 +0200 Ard Biesheuvel <[email protected]>…
Quoted 39 lines · Jean-Baptiste Theou
Quoted 8 lines · On Mon, 9 Apr 2018 11:49:37 +0200 Ard Biesheuvel <[email protected]>…
Quoted 6 lines · Greg KH
Quoted 5 lines · Jean-Baptiste Theou

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

How does one 'build a kernel with CFI' for arm64?

From Google work on Android-4.9

https://android.googlesource.com/kernel/common/+/00a195e7c0752ff5d65c9caadfbcc226270ca232

I am not sure what is the plan on their side to upstream (Greg?), but definitely useful to isolate actual issues.

Quoted 4 lines
Quoted 4 lines

Is this a "clean" 4.9.93 tree or a "4.9.93 merged into android-common-4.9?

It's a "clean 4.9.93" + whatever is needed for Clang/CFI support

My take is that CFI doesn't like

* void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper)

and

remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings);

Maybe just flag this function to not use CFI? I remember that Sami Tolvanen did similar changes.

Quoted 8 lines
Quoted 3 lines

I know it's a bit out of context since CFI support for ARM64 is not upstream yet, but unfortunate that an stable patchset trigger such failures.

I am sorry but if you are implying that we should have tested these patches against the out of tree CFI code, I have to disappoint you: that is simply not upstream’s job, and if the Google engineers merged this into their v4.9 tree without proper testing, may I suggest that you report it to them instead?

OTOH, if that is not what you are implying, please ignore the rant :-)

To be perfectly honest, I forgot that CFI wasn't an upstream feature. Indeed, I don't expect upstream test farm to run out of tree. The real answer here is to upstream CFI ;-)

Thanks a lot

Best regards

Quoted 112 lines
Quoted 112 lines

Thanks a lot

Best regards

Quoted 107 lines
Quoted 107 lines
Quoted 90 lines

Wondering if there is one of the test suite used on the review patchset that covers the CFI usecase.

Best regards,

[    0.249191] CPU features: detected feature: GIC system register CPU interface
[    0.256391] CPU features: detected feature: Privileged Access Never
[    0.262719] CPU features: detected feature: User Access Override
[    0.268791] CPU features: detected feature: 32-bit EL0 Support
[    0.274683] CPU features: detected feature: Kernel page table isolation (KPTI)
[    0.282166] CFI failure:
[    0.282169] CFI failure:
[    0.282172] CFI failure:
[    0.282173] CFI failure:
[    0.282175] CFI failure:
[    0.282176] CFI failure:
[    0.282177] CFI failure:
[    0.282178] CFI failure:
[    0.282188] ------------[ cut here ]------------
[    0.282189] ------------[ cut here ]------------
[    0.282190] ------------[ cut here ]------------
[    0.282191] ------------[ cut here ]------------
[    0.282193] ------------[ cut here ]------------
[    0.282196] kernel BUG at kernel/cfi.c:32!
[    0.282198] ------------[ cut here ]------------
[    0.282201] kernel BUG at kernel/cfi.c:32!
[    0.282202] ------------[ cut here ]------------
[    0.282204] kernel BUG at kernel/cfi.c:32!
[    0.282207] kernel BUG at kernel/cfi.c:32!
[    0.282209] kernel BUG at kernel/cfi.c:32!
[    0.282211] kernel BUG at kernel/cfi.c:32!
[    0.282214] kernel BUG at kernel/cfi.c:32!
[    0.282215] ------------[ cut here ]------------
[    0.282216] kernel BUG at kernel/cfi.c:32!
[    0.282218] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[    0.282224] Modules linked in:
[    0.282230] CPU: 2 PID: 25 Comm: migration/2 Not tainted 4.9.93-perf+ #39
[    0.282232] Hardware name: <REMOVED>
[    0.282235] task: fffffffbb3b36580 task.stack: fffffffbb30cc000
[    0.282250] PC is at __cfi_check_fail+0x14/0x1c
[    0.282253] LR is at __cfi_check_fail+0x14/0x1c
[    0.282255] pc : [<ffffff93b3f03d90>] lr : [<ffffff93b3f03d90>] pstate: 60c00085
[    0.282256] sp : fffffffbb30cfc30
[    0.282259] x29: fffffffbb30cfc30 x28: ffffff93b6415000
[    0.282261] x27: 00000013b65c1000 x26: ffffff93b5ce6000
[    0.282264] x25: ffffff93b5ce6000 x24: ffffff93b6419000
[    0.282266] x23: ffffff93b65c1000 x22: ffffff93b65c4000
[    0.282268] x21: 9d12f8172cb2f296 x20: 000000008180e3e0
[    0.282271] x19: 0000000000000000 x18: 000000000000002c
[    0.282274] x17: 00000000000fd054 x16: 0000000000000000
[    0.282276] x15: ffffff93b65ec000 x14: 000000000000000c
[    0.282279] x13: 0000000000000004 x12: 0000000000000000
[    0.282281] x11: 0000000000000000 x10: 0000000001440144
[    0.282283] x9 : 260822e8751d5000 x8 : 260822e8751d5000
[    0.282286] x7 : 0000000000000000 x6 : fffffffbbac75b60
[    0.282288] x5 : 0000000000000000 x4 : 0000000000000000
[    0.282290] x3 : 000000003a657275 x2 : 0000000000000000
[    0.282292] x1 : 0000000000000000 x0 : 000000000000000c
[    0.282294]
[    0.282294] PC: 0xffffff93b3f03d50:
[    0.282308] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282315] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282322] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282329] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282330]
[    0.282330] LR: 0xffffff93b3f03d50:
[    0.282336] 3d50  b9001ac8 f94002c8 370ffec8 17ffffbe d4210000 14000000 aa1603e0 f90007e8
[    0.282343] 3d70  94536017 f94007e8 17ffffe2 a9bf7bfd 910003fd d000d100 913ee400 94533cc7
[    0.282350] 3d90  d4210000 14000000 b0013788 2a1f03e0 f901c51f d65f03c0 f940406b 2a0203e8
[    0.282357] 3db0  2a0103e9 aa0003ea b400008b f9000145 f94000cb b40001ab a9bf7bfd 910003fd
[    0.282358]
[    0.282358] SP: 0xfffffffbb30cfbf0:
[    0.282365] fbf0  b3f03d90 ffffff93 b30cfc30 fffffffb b3f03d90 ffffff93 60c00085 00000000
[    0.282372] fc10  b6415000 ffffff93 b642fa00 ffffff93 ffffffff ffffffff b3f03d90 ffffff93
[    0.282378] fc30  b30cfc70 fffffffb b3d458c0 ffffff93 00000080 00000000 00000001 00000000
[    0.282385] fc50  b65c4000 ffffff93 b64420f0 ffffff93 8180e3e0 00000000 00000002 00000000
[    0.282387] Process migration/2 (pid: 25, stack limit = 0xfffffffbb30cc000)
[    0.282389] Call trace:
[    0.282391] Exception stack(0xfffffffbb30cfb00 to 0xfffffffbb30cfc30)
[    0.282395] fb00: 000000000000000c 0000000000000000 0000000000000000 000000003a657275
[    0.282397] fb20: 0000000000000000 0000000000000000 fffffffbbac75b60 0000000000000000
[    0.282400] fb40: 260822e8751d5000 260822e8751d5000 0000000001440144 0000000000000000
[    0.282403] fb60: 0000000000000000 0000000000000004 000000000000000c ffffff93b65ec000
[    0.282405] fb80: 0000000000000000 00000000000fd054 000000000000002c 0000000000000000
[    0.282408] fba0: 000000008180e3e0 9d12f8172cb2f296 ffffff93b65c4000 ffffff93b65c1000
[    0.282411] fbc0: ffffff93b6419000 ffffff93b5ce6000 ffffff93b5ce6000 00000013b65c1000
[    0.282413] fbe0: ffffff93b6415000 fffffffbb30cfc30 ffffff93b3f03d90 fffffffbb30cfc30
[    0.282416] fc00: ffffff93b3f03d90 0000000060c00085 ffffff93b6415000 ffffff93b642fa00
[    0.282418] fc20: ffffffffffffffff ffffff93b3f03d90
[    0.282421] [<ffffff93b3f03d90>] __cfi_check_fail+0x14/0x1c
Quoted 8 lines
[    0.282430] [<ffffff93b3d458c0>] name_to_dev_t+0x0/0x47c
[    0.282436] [<ffffff93b3d51b80>] kpti_install_ng_mappings+0x178/0x2e0
[    0.282443] [<ffffff93b3eae950>] multi_cpu_stop+0x114/0x170
[    0.282445] [<ffffff93b3eaf08c>] cpu_stopper_thread+0x128/0x2e8
[    0.282452] [<ffffff93b3db5504>] smpboot_thread_fn+0x230/0x558
[    0.282455] [<ffffff93b3dae848>] kthread+0x21c/0x238
[    0.282459] [<ffffff93b3c838f0>] ret_from_fork+0x10/0x20
[    0.282464] Code: 910003fd d000d100 913ee400 94533cc7 (d4210000)    

Oh fun :(

Dragging in a bunch more people to the cc: and to: lines to have them look at this...

thanks,

greg k-h

#9 lore raw
In reply to Jean-Baptiste Theou
Quoted 10 lines · Jean-Baptiste Theou
Quoted 7 lines · On Mon, 9 Apr 2018 11:49:37 +0200 Ard Biesheuvel <[email protected]>…
Quoted 5 lines · Greg KH
Quoted 4 lines · Jean-Baptiste Theou

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

Quoted 8 lines
Quoted 1 line

How does one 'build a kernel with CFI' for arm64?

From Google work on Android-4.9

https://android.googlesource.com/kernel/common/+/00a195e7c0752ff5d65c9caadfbcc226270ca232

I am not sure what is the plan on their side to upstream (Greg?), but definitely useful to isolate actual issues.

Ah, yeah, if you are taking anything from the Android tree, all bets are off :)

Greg Hackmann can probably answer the questions about CFI and these patches, as he's working on merging this stable release into the android-4.9 branch this week. I'll let him deal with this...

thanks,

greg k-h

#10 lore raw
Quoted 23 lines · Greg KH
Quoted 10 lines · Jean-Baptiste Theou
Quoted 7 lines · On Mon, 9 Apr 2018 11:49:37 +0200 Ard Biesheuvel <[email protected]>…
Quoted 5 lines · Greg KH
Quoted 4 lines · Jean-Baptiste Theou

Hi,

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

Quoted 8 lines
Quoted 1 line

How does one 'build a kernel with CFI' for arm64?

From Google work on Android-4.9

https://android.googlesource.com/kernel/common/+/00a195e7c0752ff5d65c9caadfbcc226270ca232

I am not sure what is the plan on their side to upstream (Greg?), but definitely useful to isolate actual issues.

Ah, yeah, if you are taking anything from the Android tree, all bets are off :)

Quoted 8 lines

Greg Hackmann can probably answer the questions about CFI and these patches, as he's working on merging this stable release into the android-4.9 branch this week. I'll let him deal with this...

thanks,

greg k-h

I've added Sami Tolvanen, who's handling CFI in the kernel.

Sami, what are the plans for upstreaming this work?

#11 lore raw
Quoted 1 line · Greg Hackmann

Sami, what are the plans for upstreaming this work?

CFI is a clang-specific feature that depends on LTO. Based on the earlier LTO discussion, we decided to collect some more evidence that clang's LTO doesn't actually break anything in the kernel before sending out the next revision of these patches. I will look into upstreaming this code again once we have sufficient test coverage, hopefully by the end of the year.

Sami

#12 lore raw
In reply to Jean-Baptiste Theou
Quoted 1 line · Jean-Baptiste Theou

Hi,

Hi,

Quoted 5 lines

After this patchset, a kernel built with CFI fails. Disabling UNMAP_KERNEL_AT_EL0 fix the issue obviously.

Wondering if there is one of the test suite used on the review patchset that covers the CFI usecase.

IIUC, you're saying that after my v4.9.y meltdown backport, your own backport of an out-of-tree patchset does not function.

There's no support for CFI in upsteam (at least as of v4.16), nor in v4.9.93. I have no idea as to how KPTI interacts with an out-of-tree patchset, so could you elaborate on the issue?

I've not seen any CFI patches on the linux-arm-kernel mailing list, so it's not clear to me how this interacts with KPTI.

Which patches do you have applied that implement CFI? Which compiler are you using?

[...]

Quoted 1 line
[    0.282196] kernel BUG at kernel/cfi.c:32!

This file doesn't exit in v4.9.93...

Quoted 1 line
[    0.282230] CPU: 2 PID: 25 Comm: migration/2 Not tainted 4.9.93-perf+ #39

... and here I see you have local changes applied.

It would be best if you tried to get CFI supported upstream before trying to backport it. That would be best discussed on the linux-arm-kernel mailing list.

Thanks,
Mark.