Skip to contentSkip to conversation

sched: division by zero in find_busiest_group()

Regression Hedi Berriche 2 participants 3 messages Dec 9, 2013 – Dec 18, 2013 Scheduler

lore
Share & tools
lore
#0 lore raw

Folks,

The following panic occurs *early* at boot time on high *enough* CPU count machines:

divide error: 0000 [#1] SMP

Modules linked in:
CPU: 22 PID: 1146 Comm: kworker/22:0 Not tainted 3.13.0-rc2-00122-gdea4f48 #8
Hardware name: Intel Corp. Stoutland Platform, BIOS 2.20 UEFI2.10 PI1.0 X64 2013-09-20
task: ffff8827d49f31c0 ti: ffff8827d4a18000 task.ti: ffff8827d4a18000
RIP: 0010:[<ffffffff810a345b>]  [<ffffffff810a345b>] find_busiest_group+0x26b/0x890
RSP: 0000:ffff8827d4a19b68  EFLAGS: 00010006
RAX: 0000000000007fff RBX: 0000000000008000 RCX: 0000000000000200
RDX: 0000000000000000 RSI: 0000000000008000 RDI: 0000000000000020
RBP: ffff8827d4a19cc0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff8827d4a19d28 R14: ffff8827d4a19b98 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8827dfd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000000b8 CR3: 00000000018da000 CR4: 00000000000007e0
Stack:
ffff8827d4b35800 0000000000000000 0000000000014600 0000000000014600
0000000000000000 ffff8827d4b35818 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000008000 0000000000000000
Call Trace:
[<ffffffff810a3be6>] load_balance+0x166/0x7f0
[<ffffffff810a477e>] idle_balance+0x10e/0x1b0
[<ffffffff815d83d3>] __schedule+0x723/0x780
[<ffffffff815d8459>] schedule+0x29/0x70
[<ffffffff810818b9>] worker_thread+0x1c9/0x400
[<ffffffff810816f0>] ? rescuer_thread+0x3e0/0x3e0
[<ffffffff81088562>] kthread+0xd2/0xf0
[<ffffffff81088490>] ? kthread_create_on_node+0x180/0x180
[<ffffffff815e437c>] ret_from_fork+0x7c/0xb0
[<ffffffff81088490>] ? kthread_create_on_node+0x180/0x180

Bisection points to

9abf24d sched: Check sched_domain before computing group power

but without it (as clearly indicated in the changelog) the kernel panics thus:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
IP: [<ffffffff810a3542>] update_group_power+0xa2/0x250
PGD 0 
Oops: 0000 [#1] SMP 
Modules linked in: 
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.12.0-00122-gdea4f48 #10 
Hardware name: Intel Corp. Stoutland Platform, BIOS 2.20 UEFI2.10 PI1.0 X64 2013-09-20
task: ffff881054528000 ti: ffff881054530000 task.ti: ffff881054530000
RIP: 0010:[<ffffffff810a3542>]  [<ffffffff810a3542>] update_group_power+0xa2/0x250
RSP: 0000:ffff881054531d48  EFLAGS: 00010287
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 00000000000000f0 RSI: 0000000000000100 RDI: 00000000000000c0
RBP: ffff881054531d70 R08: ffff89e7d4ae6018 R09: 0000000000000004
R10: ffff89e7d4ae6818 R11: ffffffff81098d5d R12: 0000000000000000
R13: 00000000000146c0 R14: ffff89e7d4ae6000 R15: ffff89e7d4ae6018
FS:  0000000000000000(0000) GS:ffff88105fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000010 CR3: 00000000018c2000 CR4: 00000000000007f0
Stack:
ffff89e7d4ae6000 ffff89e7d4ac0000 00000000000000f0 00000000000000f0
ffff898fd3a68c00 ffff881054531e30 ffffffff8109933f 0000000000000100
00000000000000ff 0000000000010448 00000000000000ff 0000010000000100
Call Trace:
[<ffffffff8109933f>] build_sched_domains+0xbff/0xc80
[<ffffffff81a3c89e>] sched_init_smp+0x3ad/0x469
[<ffffffff81a1c00c>] kernel_init_freeable+0xfa/0x207
[<ffffffff815b3ea0>] ? rest_init+0x80/0x80
[<ffffffff815b3eae>] kernel_init+0xe/0x120
[<ffffffff815d547c>] ret_from_fork+0x7c/0xb0
[<ffffffff815b3ea0>] ? rest_init+0x80/0x80

and this is because of:

863bffc sched/fair: Fix group power_orig computation

IOW, 9abf24d can't be blamed for it all, and this is not a case of a straightforward revert of a single commit.

Back to the division by zero itself, it's taking place in the inlined sg_capacity():

find_busiest_group
update_sd_lb_stats
update_sg_lb_stats
sg_capacity

5492 static inline int sg_capacity(struct lb_env *env, struct sched_group *group) 5493 {
5494 unsigned int capacity, smt, cpus;
5495 unsigned int power, power_orig;
5496

5497         power = group->sgp->power;
5498         power_orig = group->sgp->power_orig;
5499         cpus = group->group_weight;

5500
5501 /* smt := ceil(cpus / power), assumes: 1 < smt_power < 2 */
5502 smt = DIV_ROUND_UP(SCHED_POWER_SCALE * cpus, power_orig); <-- HERE

so we're arriving here with group->sgp->power_orig == 0.

Cheers,
Hedi.

P.S.

The following *works* around the panic:

kernel/sched/core.c +3 -0
index e85cda2..48c8d0b 100644
@@ -5735,6 +5735,9 @@ static int __sdt_alloc(const struct cpumask *cpu_map)
 			if (!sgp)
 				return -ENOMEM;
 
+			/* WAR: avoid a divison by zero in sg_capacity() */
+			sgp->power_orig = 1;
+
 			*per_cpu_ptr(sdd->sgp, j) = sgp;
 		}
 	}

and I wonder whether the following --on its own-- would make sense as a fix:

kernel/sched/fair.c +1 -1
index fd773ad..57578b3 100644
@@ -5495,7 +5495,7 @@ static inline int sg_capacity(struct lb_env *env, struct sched_group *group)
 	unsigned int power, power_orig;
 
 	power = group->sgp->power;
-	power_orig = group->sgp->power_orig;
+	power_orig = max_t(unsigned, group->sgp->power_orig, 1);
 	cpus = group->group_weight;
 
 	/* smt := ceil(cpus / power), assumes: 1 < smt_power < 2 */
Signature · 2 lines
Be careful of reading health books, you might die of a misprint.
	-- Mark Twain
#1 lore raw

On Mon, Dec 09, 2013 at 18:10 Hedi Berriche wrote:

| Folks,
| 
| The following panic occurs *early* at boot time on high *enough* CPU count
| machines:
| 
| divide error: 0000 [#1] SMP 
| Modules linked in:
| CPU: 22 PID: 1146 Comm: kworker/22:0 Not tainted 3.13.0-rc2-00122-gdea4f48 #8
| Hardware name: Intel Corp. Stoutland Platform, BIOS 2.20 UEFI2.10 PI1.0 X64 2013-09-20
| task: ffff8827d49f31c0 ti: ffff8827d4a18000 task.ti: ffff8827d4a18000
| RIP: 0010:[<ffffffff810a345b>]  [<ffffffff810a345b>] find_busiest_group+0x26b/0x890
| RSP: 0000:ffff8827d4a19b68  EFLAGS: 00010006
| RAX: 0000000000007fff RBX: 0000000000008000 RCX: 0000000000000200
| RDX: 0000000000000000 RSI: 0000000000008000 RDI: 0000000000000020
| RBP: ffff8827d4a19cc0 R08: 0000000000000000 R09: 0000000000000000
| R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
| R13: ffff8827d4a19d28 R14: ffff8827d4a19b98 R15: 0000000000000000
| FS:  0000000000000000(0000) GS:ffff8827dfd80000(0000) knlGS:0000000000000000
| CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
| CR2: 00000000000000b8 CR3: 00000000018da000 CR4: 00000000000007e0
| Stack:
| ffff8827d4b35800 0000000000000000 0000000000014600 0000000000014600
| 0000000000000000 ffff8827d4b35818 0000000000000000 0000000000000000
| 0000000000000000 0000000000000000 0000000000008000 0000000000000000
| Call Trace:
| [<ffffffff810a3be6>] load_balance+0x166/0x7f0
| [<ffffffff810a477e>] idle_balance+0x10e/0x1b0
| [<ffffffff815d83d3>] __schedule+0x723/0x780
| [<ffffffff815d8459>] schedule+0x29/0x70
| [<ffffffff810818b9>] worker_thread+0x1c9/0x400
| [<ffffffff810816f0>] ? rescuer_thread+0x3e0/0x3e0
| [<ffffffff81088562>] kthread+0xd2/0xf0
| [<ffffffff81088490>] ? kthread_create_on_node+0x180/0x180
| [<ffffffff815e437c>] ret_from_fork+0x7c/0xb0
| [<ffffffff81088490>] ? kthread_create_on_node+0x180/0x180

Hmm...had time to dig into this a bit deeper and looking at build_overlap_sched_groups(), specifically this bit of code:

kernel/sched/core.c:

5066 static int
5067 build_overlap_sched_groups(struct sched_domain *sd, int cpu) 5068 {
...
5109 /*

5110                  * Initialize sgp->power such that even if we mess up the
5111                  * domains and no possible iteration will get us here, we won't
5112                  * die on a /0 trap.
5113                  */

5114 sg->sgp->power = SCHED_POWER_SCALE * cpumask_weight(sg_span);

I'm wondering whether the same precaution should be used when it comes to sg->sgp->power_orig.

Cheers,
Hedi.

maintainer
#2 lore raw
Quoted 47 lines · Hedi Berriche

On Mon, Dec 09, 2013 at 18:10 Hedi Berriche wrote:

| Folks,
| 
| The following panic occurs *early* at boot time on high *enough* CPU count
| machines:
| 
| divide error: 0000 [#1] SMP 
| Modules linked in:
| CPU: 22 PID: 1146 Comm: kworker/22:0 Not tainted 3.13.0-rc2-00122-gdea4f48 #8
| Hardware name: Intel Corp. Stoutland Platform, BIOS 2.20 UEFI2.10 PI1.0 X64 2013-09-20
| task: ffff8827d49f31c0 ti: ffff8827d4a18000 task.ti: ffff8827d4a18000
| RIP: 0010:[<ffffffff810a345b>]  [<ffffffff810a345b>] find_busiest_group+0x26b/0x890
| RSP: 0000:ffff8827d4a19b68  EFLAGS: 00010006
| RAX: 0000000000007fff RBX: 0000000000008000 RCX: 0000000000000200
| RDX: 0000000000000000 RSI: 0000000000008000 RDI: 0000000000000020
| RBP: ffff8827d4a19cc0 R08: 0000000000000000 R09: 0000000000000000
| R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
| R13: ffff8827d4a19d28 R14: ffff8827d4a19b98 R15: 0000000000000000
| FS:  0000000000000000(0000) GS:ffff8827dfd80000(0000) knlGS:0000000000000000
| CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
| CR2: 00000000000000b8 CR3: 00000000018da000 CR4: 00000000000007e0
| Stack:
| ffff8827d4b35800 0000000000000000 0000000000014600 0000000000014600
| 0000000000000000 ffff8827d4b35818 0000000000000000 0000000000000000
| 0000000000000000 0000000000000000 0000000000008000 0000000000000000
| Call Trace:
| [<ffffffff810a3be6>] load_balance+0x166/0x7f0
| [<ffffffff810a477e>] idle_balance+0x10e/0x1b0
| [<ffffffff815d83d3>] __schedule+0x723/0x780
| [<ffffffff815d8459>] schedule+0x29/0x70
| [<ffffffff810818b9>] worker_thread+0x1c9/0x400
| [<ffffffff810816f0>] ? rescuer_thread+0x3e0/0x3e0
| [<ffffffff81088562>] kthread+0xd2/0xf0
| [<ffffffff81088490>] ? kthread_create_on_node+0x180/0x180
| [<ffffffff815e437c>] ret_from_fork+0x7c/0xb0
| [<ffffffff81088490>] ? kthread_create_on_node+0x180/0x180

Hmm...had time to dig into this a bit deeper and looking at build_overlap_sched_groups(), specifically this bit of code:

kernel/sched/core.c:

5066 static int
5067 build_overlap_sched_groups(struct sched_domain *sd, int cpu) 5068 {
...
5109 /*

Quoted 7 lines
5110                  * Initialize sgp->power such that even if we mess up the
5111                  * domains and no possible iteration will get us here, we won't
5112                  * die on a /0 trap.
5113                  */

5114 sg->sgp->power = SCHED_POWER_SCALE * cpumask_weight(sg_span);

I'm wondering whether the same precaution should be used when it comes to sg->sgp->power_orig.

http://marc.info/?l=linux-kernel&m=138684195315258