Skip to contentSkip to conversation

Heads-up: two regressions in v4.11-rc series

DISCUSSION Jesper Dangaard Brouer 4 participants 5 messages Apr 20, 2017 – Apr 21, 2017

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi Linus,

Just wanted to give a heads-up on two regressions in 4.11-rc series.

(1) page allocator optimization revert

Mel Gorman and I have been playing with optimizing the page allocator, but Tariq spotted that we caused a regression for (NIC) drivers that refill DMA RX rings in softirq context.

The end result was a revert, and this is waiting in AKPMs quilt queue:
http://ozlabs.org/~akpm/mmots/broken-out/revert-mm-page_alloc-only-use-per-cpu-allocator-for-irq-safe-requests.patch

(2) Busy softirq can cause userspace not to be scheduled

I bisected the problem to a499a5a14dbd ("sched/cputime: Increment kcpustat directly on irqtime account"). See email thread with
Subject: Bisected softirq accounting issue in v4.11-rc1~170^2~28
http://lkml.kernel.org/r/[email protected]

I don't know the scheduler code well enough to fix this, and will have to rely others to figure out this scheduler regression.

To make it clear: I'm only seeing this scheduler regression when a remote host is sending many many network packets, towards the kernel which keeps NAPI/softirq busy all the time. A possible hint: tool "top" only shows this in "si" column, while on v4.10 "top" also blames "ksoftirqd/N", plus "ps" reported cputime (0:00) seems wrong for ksoftirqd.

Signature · 10 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw
Quoted 21 lines · Jesper Dangaard Brouer

Hi Linus,

Just wanted to give a heads-up on two regressions in 4.11-rc series.

(1) page allocator optimization revert

Mel Gorman and I have been playing with optimizing the page allocator, but Tariq spotted that we caused a regression for (NIC) drivers that refill DMA RX rings in softirq context.

The end result was a revert, and this is waiting in AKPMs quilt queue:
http://ozlabs.org/~akpm/mmots/broken-out/revert-mm-page_alloc-only-use-per-cpu-allocator-for-irq-safe-requests.patch

(2) Busy softirq can cause userspace not to be scheduled

I bisected the problem to a499a5a14dbd ("sched/cputime: Increment kcpustat directly on irqtime account"). See email thread with
Subject: Bisected softirq accounting issue in v4.11-rc1~170^2~28
http://lkml.kernel.org/r/[email protected]

Quoted 8 lines

I don't know the scheduler code well enough to fix this, and will have to rely others to figure out this scheduler regression.

To make it clear: I'm only seeing this scheduler regression when a remote host is sending many many network packets, towards the kernel which keeps NAPI/softirq busy all the time. A possible hint: tool "top" only shows this in "si" column, while on v4.10 "top" also blames "ksoftirqd/N", plus "ps" reported cputime (0:00) seems wrong for ksoftirqd.

(I'm currently working on reproducing that one.)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
In reply to Jesper Dangaard Brouer
Quoted 4 lines · Jesper Dangaard Brouer

Hi Linus,

Just wanted to give a heads-up on two regressions in 4.11-rc series.

Quoted 9 lines

(1) page allocator optimization revert

Mel Gorman and I have been playing with optimizing the page allocator, but Tariq spotted that we caused a regression for (NIC) drivers that refill DMA RX rings in softirq context.

The end result was a revert, and this is waiting in AKPMs quilt queue:
http://ozlabs.org/~akpm/mmots/broken-out/revert-mm-page_alloc-only-use-per-cpu-allocator-for-irq-safe-requests.patch

This was flagged to Andrew that it should go in for either 4.11 or if there were concerns about how close to the release we are then put it in for 4.11-stable. At worst, I can do a resubmit to -stable myself after it gets merged in the next window if it falls between the cracks.

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#3 lore raw
Quoted 8 lines · Mel Gorman
Quoted 3 lines

The end result was a revert, and this is waiting in AKPMs quilt queue:
http://ozlabs.org/~akpm/mmots/broken-out/revert-mm-page_alloc-only-use-per-cpu-allocator-for-irq-safe-requests.patch

This was flagged to Andrew that it should go in for either 4.11 or if there were concerns about how close to the release we are then put it in for 4.11-stable. At worst, I can do a resubmit to -stable myself after it gets merged in the next window if it falls between the cracks.

This got merged (commit d34b0733b452: "Revert "mm, page_alloc: only use per-cpu allocator for irq-safe requests"").

The other issue (caused by commit a499a5a14dbd: "sched/cputime:
Increment kcpustat directly on irqtime account") is still open.

Frederic? Revert? But I guess it's something we can delay for backporting, it's presumably not possible to hit maliciously except on some fast local network attacker just causing an effective DoS.

Linus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw
Quoted 13 lines · Linus Torvalds
Quoted 8 lines · Mel Gorman
Quoted 3 lines

The end result was a revert, and this is waiting in AKPMs quilt queue:
http://ozlabs.org/~akpm/mmots/broken-out/revert-mm-page_alloc-only-use-per-cpu-allocator-for-irq-safe-requests.patch

This was flagged to Andrew that it should go in for either 4.11 or if there were concerns about how close to the release we are then put it in for 4.11-stable. At worst, I can do a resubmit to -stable myself after it gets merged in the next window if it falls between the cracks.

This got merged (commit d34b0733b452: "Revert "mm, page_alloc: only use per-cpu allocator for irq-safe requests"").

Quoted 6 lines

The other issue (caused by commit a499a5a14dbd: "sched/cputime:
Increment kcpustat directly on irqtime account") is still open.

Frederic? Revert? But I guess it's something we can delay for backporting, it's presumably not possible to hit maliciously except on some fast local network attacker just causing an effective DoS.

I can't tell about the security impact. But indeed I think we should rather delay for backporting if we can't manage to fix it in the upcoming days. Especially as you can't revert this patch alone, it's part of a whole series of ~ 30 commits that removed cputime_t and it's in the middle of the series, so those that come after depend on it and those that come before just don't make sense alone.

But I'll fix this ASAP.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>