Skip to contentSkip to conversation

New: os unresponsive during buffered I/O

Bugme-new Andrew Morton 4 participants 7 messages Jul 2, 2010 – Jul 20, 2010 1 message not in the archive Memory management

lore
Share & tools
lore

Andrew Morton

author maintainer
#1 lore raw
In reply to a message not in the archive
Quoted 18 lines · On Thu, 1 Jul 2010 11:57:37 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=16321

Summary: os unresponsive during buffered I/O
Product: IO/Storage
Version: 2.5
Kernel Version: 2.6.34
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Block Layer
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

Quoted 9 lines

I have been running these tests on my laptop running the 2.6.34 Debian kernel. When doing buffered I/O, the OS completely stalls to any interactivity. I cannot switch console tabs in my Desktop Environment and the mouse pointer does not move.

Eventually, I/O completes and every thing resumes to normal. There is no OOM seen during the I/O operation.
If doing direct I/O, interactivity does not get penalized.

1...

2...

3...

FUCK!!!

We've been trying to fix this stuff for ten years. Apparently, without success. Do we suck, or what?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
Quoted 33 lines · Andrew Morton
Quoted 18 lines · On Thu, 1 Jul 2010 11:57:37 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=16321

Summary: os unresponsive during buffered I/O
Product: IO/Storage
Version: 2.5
Kernel Version: 2.6.34
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Block Layer
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

Quoted 9 lines

I have been running these tests on my laptop running the 2.6.34 Debian kernel. When doing buffered I/O, the OS completely stalls to any interactivity. I cannot switch console tabs in my Desktop Environment and the mouse pointer does not move.

Eventually, I/O completes and every thing resumes to normal. There is no OOM seen during the I/O operation.
If doing direct I/O, interactivity does not get penalized.

1...

Quoted 8 lines

2...

3...

FUCK!!!

We've been trying to fix this stuff for ten years. Apparently, without success. Do we suck, or what?

We suck. This is assigned to the block layer, but it must be something a lot more fundemental than this. Either the vm is shitting itself, or something else is sucking up the juice completely. Or the CPU scheduler is going to pieces, I dunno. The report says that he cannot even move the mouse or switch console tabs, clearly we are not (only) dealing with an issue at the IO side.

What else is interesting is that only a few people seem to see this. What is different about their setup or their hardware?! I cannot reproduce reports like this, and if it happened for everybody for a simple 4 process random write like this, then everything would grind to a halt.

I propose we fly Andrew out to investigate in person :-)

Signature · 7 lines
Jens Axboe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author maintainer
#3 lore raw
Quoted 50 lines · On Sat, 03 Jul 2010 13:56:13 +0200 Jens Axboe <[email protected]> wrote:
Quoted 33 lines · Andrew Morton
Quoted 18 lines · On Thu, 1 Jul 2010 11:57:37 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=16321

Summary: os unresponsive during buffered I/O
Product: IO/Storage
Version: 2.5
Kernel Version: 2.6.34
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Block Layer
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

Quoted 9 lines

I have been running these tests on my laptop running the 2.6.34 Debian kernel. When doing buffered I/O, the OS completely stalls to any interactivity. I cannot switch console tabs in my Desktop Environment and the mouse pointer does not move.

Eventually, I/O completes and every thing resumes to normal. There is no OOM seen during the I/O operation.
If doing direct I/O, interactivity does not get penalized.

1...

Quoted 8 lines

2...

3...

FUCK!!!

We've been trying to fix this stuff for ten years. Apparently, without success. Do we suck, or what?

We suck. This is assigned to the block layer, but it must be something a lot more fundemental than this. Either the vm is shitting itself, or something else is sucking up the juice completely. Or the CPU scheduler is going to pieces, I dunno. The report says that he cannot even move the mouse or switch console tabs, clearly we are not (only) dealing with an issue at the IO side.

Quoted 7 lines

What else is interesting is that only a few people seem to see this. What is different about their setup or their hardware?! I cannot reproduce reports like this, and if it happened for everybody for a simple 4 process random write like this, then everything would grind to a halt.

I propose we fly Andrew out to investigate in person :-)

Hey, I suck more than anyone!

My laptop goes absolutely utterly mouse-wont-move comatose for tens of minutes when it fetchmails 100 emails and 100 spamassassins go berzerk. It could be either a CPU scheduler thing, or an IO thing, or an evil combination of both.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw
Quoted 6 lines · Andrew Morton

My laptop goes absolutely utterly mouse-wont-move comatose for tens of minutes when it fetchmails 100 emails and 100 spamassassins go berzerk. It could be either a CPU scheduler thing, or an IO thing, or an evil combination of both.

Try putting your spamassasins in a cgroup and see.

Signature · 8 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author maintainer
#5 lore raw
In reply to Andrew Morton
Quoted 1 line · On Fri, 2 Jul 2010 16:05:01 -0700 Andrew Morton <[email protected]>…
Quoted 1 line

https://bugzilla.kernel.org/show_bug.cgi?id=16321

The reporter has updated the report, says "This bug is consistently reproducible.". But no signs of interest from kernel developers yet.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
Quoted 2 lines · Andrew Morton

The reporter has updated the report, says "This bug is consistently reproducible.". But no signs of interest from kernel developers yet.

If only the reported can reproduce it that doesn't really help anyone.

And neither does hiding bugreports in bugzilla help when this could be discussed much more usefully on the list.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author maintainer
#7 lore raw
Quoted 5 lines · On Tue, 20 Jul 2010 11:08:55 -0400 Christoph Hellwig <[email protected]> wrote:
Quoted 2 lines · Andrew Morton

The reporter has updated the report, says "This bug is consistently reproducible.". But no signs of interest from kernel developers yet.

If only the reported can reproduce it that doesn't really help anyone.

Garbage. We have a reporter who is prepared to work with us. Has anyone tried talking him through using the relevant tracepoints? Sending patches to add new ones if needed?

Quoted 2 lines

And neither does hiding bugreports in bugzilla help when this could be discussed much more usefully on the list.

The guy's email address is right there. And there are over forty more people at https://bugzilla.kernel.org/show_bug.cgi?id=7372 seeing similar (or other!) problems.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>