Skip to contentSkip to conversation

-next20181010 regression: thinkpad x60 (32 bit) dies during boot.

DISCUSSION Pavel Machek 4 participants 11 messages Oct 10, 2018 – Oct 12, 2018

lore
Share & tools
lore
#0 lore raw

Hi!

I updated to todays next... and boot crashes with

..

Call Trace:

kick_ilb
trigger_load_balance
? active_load..
scheduler_tick
update_process_times
tick_nohz_handler

-next20181005 worked ok.

Tell me if x86-32 works for you...

Best regards,
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#1 lore raw

Hi!

Quoted 4 lines

I updated to todays next... and boot crashes with

..

Call Trace:
Quoted 8 lines

kick_ilb
trigger_load_balance
? active_load..
scheduler_tick
update_process_times
tick_nohz_handler

-next20181005 worked ok.

Backtrace indicates problem with nohz, so I added idle=poll.

Now I have

Run /sbin/init as init process

Kernel panic - not syncing: stack-protector: Kernel stack is corrupted

in: pgd_alloc...
...
dump_stack
....
pgd_alloc
mm_init.isra.
mm_alloc
__do_execve_file
do_execve
...

Good ideas welcome :-).

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#2 lore raw

-next20181010,1011 regression: thinkpad x60 (32 bit) dies during boot.

Quoted 14 lines · Pavel Machek

Hi!

Quoted 12 lines

I updated to todays next... and boot crashes with

..

Call Trace:

kick_ilb
trigger_load_balance
? active_load..
scheduler_tick
update_process_times
tick_nohz_handler

-next20181005 worked ok.

Problem is still there in today's next.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#3 lore raw

Re: -next20181010,1011 regression: thinkpad x60 (32 bit) dies during boot.

Quoted 14 lines · Pavel Machek
Quoted 13 lines · Pavel Machek

Hi!

Quoted 11 lines

I updated to todays next... and boot crashes with

..

Call Trace:

kick_ilb
trigger_load_balance
? active_load..
scheduler_tick
update_process_times
tick_nohz_handler

Quoted 3 lines
Quoted 1 line
Quoted 1 line

-next20181005 worked ok.

Problem is still there in today's next.

So what came in between -next20181005 and the first bad one? kernel/sched/* being the first place to look at.

Thanks,

tglx

#4 lore raw

Re: -next20181010,1011 regression: thinkpad x60 (32 bit) dies during boot.

Hi!

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

I updated to todays next... and boot crashes with

..

Call Trace:

kick_ilb
trigger_load_balance
? active_load..
scheduler_tick
update_process_times
tick_nohz_handler

Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

-next20181005 worked ok.

Problem is still there in today's next.

So what came in between -next20181005 and the first bad one? kernel/sched/* being the first place to look at.

kernel/sched does not seem to contain anything too scary.

I know that -next20181005 works ok, and I know -next20181010 is bad. Is there easy way to bisect using that information? I can do bisect between -next and mainline, but that's a lot of patches and thus not much fun :-(.

In the meantime, I reproduced the failure with T40p. Is there someone with working x86-32 in -next?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#5 lore raw

Re: -next20181010,1011 regression: thinkpad x60 (32 bit) dies during boot.

Quoted 22 lines · Pavel Machek

Hi!

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

I updated to todays next... and boot crashes with

..

Call Trace:

kick_ilb
trigger_load_balance
? active_load..
scheduler_tick
update_process_times
tick_nohz_handler

Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

-next20181005 worked ok.

Problem is still there in today's next.

So what came in between -next20181005 and the first bad one? kernel/sched/* being the first place to look at.

kernel/sched does not seem to contain anything too scary.

Quoted 7 lines

I know that -next20181005 works ok, and I know -next20181010 is bad. Is there easy way to bisect using that information? I can do bisect between -next and mainline, but that's a lot of patches and thus not much fun :-(.

In the meantime, I reproduced the failure with T40p. Is there someone with working x86-32 in -next?

Does latest -tip fail too? If yes then I suspect bisection would be needed.

Thanks,

Ingo

#6 lore raw

Re: -next20181010,1011 regression: thinkpad x60 (32 bit) dies during boot.

Hi!

Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 1 line

Problem is still there in today's next.

So what came in between -next20181005 and the first bad one? kernel/sched/* being the first place to look at.

kernel/sched does not seem to contain anything too scary.

I know that -next20181005 works ok, and I know -next20181010 is bad. Is there easy way to bisect using that information? I can do bisect between -next and mainline, but that's a lot of patches and thus not much fun :-(.

Quoted 4 lines
Quoted 2 lines

In the meantime, I reproduced the failure with T40p. Is there someone with working x86-32 in -next?

Does latest -tip fail too? If yes then I suspect bisection would be needed.

I already started bisect on -next (T40p is my test machine, so bisect is not that bad).

The log so far is:

Pavel

# bad: [771b65e89c8a51d611b8049718693a4202e4f732] Add linux-next specific files for 20181011 # good: [7876320f88802b22d4e2daf7eb027dd14175a0f8] Linux 4.19-rc4
git bisect start '771b65e89c8a51d611b8049718693a4202e4f732' '7876320f88802b22d4e2daf7eb027dd14175a0f8' # good: [43faff25da004eabce691268da34065b3690f5ca] Merge remote-tracking branch 'net-next/master' git bisect good 43faff25da004eabce691268da34065b3690f5ca
# good: [3e2beb7db82a880319aa2f0dcafa820f3f5206d3] Merge remote-tracking branch 'spi/for-next' git bisect good 3e2beb7db82a880319aa2f0dcafa820f3f5206d3
# bad: [74411e5fd30ae540491c4d6142af6ee6b2b22f09] Merge remote-tracking branch 'char-misc/char-misc-next' git bisect bad 74411e5fd30ae540491c4d6142af6ee6b2b22f09
# bad: [c810d907775aa2aa753e836a122613fd2416b14d] Merge remote-tracking branch 'kvm/linux-next' git bisect bad c810d907775aa2aa753e836a122613fd2416b14d
# good: [fac07d2ba7b2764e3002ff9bc7861742a84a2ef6] Merge branch 'perf/core'
git bisect good fac07d2ba7b2764e3002ff9bc7861742a84a2ef6
# bad: [d74865bd3996c7a6f3e8ce6e626c1fe474e39494] Merge branch 'x86/mm'
git bisect bad d74865bd3996c7a6f3e8ce6e626c1fe474e39494
# good: [dcd2d0cece1608b2be9184786c900807ec947076] Merge branch 'x86/asm'
git bisect good dcd2d0cece1608b2be9184786c900807ec947076

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#7 lore raw

Avoid VLA in pgd_alloc kills boot on 32-bit machines was Re: -next20181010,1011 regression: thinkpad x60 (32 bit) dies during boot.

In reply to Ingo Molnar

Hi!

Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

So what came in between -next20181005 and the first bad one? kernel/sched/* being the first place to look at.

kernel/sched does not seem to contain anything too scary.

I know that -next20181005 works ok, and I know -next20181010 is bad. Is there easy way to bisect using that information? I can do bisect between -next and mainline, but that's a lot of patches and thus not much fun :-(.

Quoted 4 lines
Quoted 2 lines

In the meantime, I reproduced the failure with T40p. Is there someone with working x86-32 in -next?

Does latest -tip fail too? If yes then I suspect bisection would be needed.

And the winner is...

[1be3f247c2882a82279cbcf43717581ea943b692] x86/mm: Avoid VLA in pgd_alloc()

"Kernel stack is corrupted in: pgd_alloc" panic kind of suggests this is right commit.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#8 lore raw

Re: Avoid VLA in pgd_alloc kills boot on 32-bit machines was Re: -next20181010,1011 regression: thinkpad x60 (32 bit) dies during boot.

Quoted 4 lines · Pavel Machek

And the winner is...

[1be3f247c2882a82279cbcf43717581ea943b692] x86/mm: Avoid VLA in pgd_alloc()

That should be fixed now:

https://git.kernel.org/tip/184d47f0fd365108bd06ab26cdb3450b716269fd

Signature · 4 lines
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.
#9 lore raw

Re: Avoid VLA in pgd_alloc kills boot on 32-bit machines was Re: -next20181010,1011 regression: thinkpad x60 (32 bit) dies during boot.

Quoted 9 lines · Borislav Petkov
Quoted 4 lines · Pavel Machek

And the winner is...

[1be3f247c2882a82279cbcf43717581ea943b692] x86/mm: Avoid VLA in pgd_alloc()

That should be fixed now:

https://git.kernel.org/tip/184d47f0fd365108bd06ab26cdb3450b716269fd

Aha, ok.

-next20181012 indeed works.

Best regards,
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#10 lore raw

Re: Avoid VLA in pgd_alloc kills boot on 32-bit machines was Re: -next20181010,1011 regression: thinkpad x60 (32 bit) dies during boot.

In reply to Pavel Machek
Quoted 17 lines · Pavel Machek

Hi!

Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

So what came in between -next20181005 and the first bad one? kernel/sched/* being the first place to look at.

kernel/sched does not seem to contain anything too scary.

I know that -next20181005 works ok, and I know -next20181010 is bad. Is there easy way to bisect using that information? I can do bisect between -next and mainline, but that's a lot of patches and thus not much fun :-(.

Quoted 4 lines
Quoted 2 lines

In the meantime, I reproduced the failure with T40p. Is there someone with working x86-32 in -next?

Does latest -tip fail too? If yes then I suspect bisection would be needed.

Quoted 7 lines

And the winner is...

[1be3f247c2882a82279cbcf43717581ea943b692] x86/mm: Avoid VLA in pgd_alloc()

"Kernel stack is corrupted in: pgd_alloc" panic kind of suggests this is right commit.

git bisect log, for the reference... and ~ my config.

Pavel

# bad: [771b65e89c8a51d611b8049718693a4202e4f732] Add linux-next specific files for 20181011
# good: [7876320f88802b22d4e2daf7eb027dd14175a0f8] Linux 4.19-rc4
git bisect start '771b65e89c8a51d611b8049718693a4202e4f732' '7876320f88802b22d4e2daf7eb027dd14175a0f8'
# good: [43faff25da004eabce691268da34065b3690f5ca] Merge remote-tracking branch 'net-next/master'
git bisect good 43faff25da004eabce691268da34065b3690f5ca
# good: [3e2beb7db82a880319aa2f0dcafa820f3f5206d3] Merge remote-tracking branch 'spi/for-next'
git bisect good 3e2beb7db82a880319aa2f0dcafa820f3f5206d3
# bad: [74411e5fd30ae540491c4d6142af6ee6b2b22f09] Merge remote-tracking branch 'char-misc/char-misc-next' git bisect bad 74411e5fd30ae540491c4d6142af6ee6b2b22f09
# bad: [c810d907775aa2aa753e836a122613fd2416b14d] Merge remote-tracking branch 'kvm/linux-next'
git bisect bad c810d907775aa2aa753e836a122613fd2416b14d
# good: [fac07d2ba7b2764e3002ff9bc7861742a84a2ef6] Merge branch 'perf/core'
git bisect good fac07d2ba7b2764e3002ff9bc7861742a84a2ef6
# bad: [d74865bd3996c7a6f3e8ce6e626c1fe474e39494] Merge branch 'x86/mm'
git bisect bad d74865bd3996c7a6f3e8ce6e626c1fe474e39494
# good: [dcd2d0cece1608b2be9184786c900807ec947076] Merge branch 'x86/asm'
git bisect good dcd2d0cece1608b2be9184786c900807ec947076
# bad: [ae9260d80e517c8702b91b8e00d117e1e2834c33] Merge branch 'x86/cache'
git bisect bad ae9260d80e517c8702b91b8e00d117e1e2834c33
# good: [d5a581d84ae6b8a4a740464b80d8d9cf1e7947b2] x86/cpufeature: Macrofy inline assembly code to work around GCC inlining bugs git bisect good d5a581d84ae6b8a4a740464b80d8d9cf1e7947b2
# good: [245e5707dd7df01428459d97a9121f14a57dac6b] Merge branch 'x86/build'
git bisect good 245e5707dd7df01428459d97a9121f14a57dac6b
# bad: [7d27cb68cc307ee103e116d357e9baca35151c55] Merge branch 'x86/urgent' into x86/cache
git bisect bad 7d27cb68cc307ee103e116d357e9baca35151c55
# good: [2cc81c6992248ea37d0241bc325977bab310bc3b] x86/intel_rdt: Show missing resctrl mount options
git bisect good 2cc81c6992248ea37d0241bc325977bab310bc3b
# bad: [e8bd1803aec89dfce5758d88022963fe3248bc4c] x86/intel_rdt: Fix out-of-bounds memory access in CBM tests git bisect bad e8bd1803aec89dfce5758d88022963fe3248bc4c

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html