Powertop shows events/0 waking at high rate due to ptys
Hi,
I noticed on one of my systems that powertop was showing events/0 waking at a high rate (~2000 times/sec, HZ=1000). After a lot of tracing I found it was due to writes pending to ptys which hadn't been read.
Specifically, in flush_to_ldisc(), these lines seem to be causing the repeated wakeups:
if (!tty->receive_room) {
schedule_delayed_work(&tty->buf.work, 1);
break;
}After reading the pending writes the system settles down and wakeups drops to a normal low level.
The program provoking this was Xen's xenconsoled, which manages running domains' consoles; the unread output was various domains console output. This doesn't seem to be related to Xen at all: Ian Jackson put together a small standalone program (attached) which reproduces the problem; he reports that this wakeup behaviour is a regression compared to a 2.6.16 Debian kernel.
Compile ptypolling with: gcc -o ptypolling ptypolling.c -g -lutil
After running ptypolling, powertop shows events/0 waking at a high rate. ptypolling itself is blocked in a select() and is idle.
Thanks,
J
/* demonstrate polling wakeup */
#include <unistd.h>
#include <pty.h>
#include <assert.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
static int master, slave;
int main(void) {
int r;
r = openpty(&master,&slave,0,0,0);
fprintf(stdout,"openpty=%d master=%d slave=%d\n", r, master, slave);
assert(!r);
r = fcntl(master, F_GETFL); assert(r>=0);
r = fcntl(master, F_SETFL, r | O_NONBLOCK);
char buf[400];
int i;
for (i=0; i<sizeof(buf)-2; i+=2)
sprintf(buf+i, "%02x", i);
buf[sizeof(buf)-2]= '\r';
buf[sizeof(buf)-1]= '\n';
fd_set writefds;
for (;;) {
FD_ZERO(&writefds);
FD_SET(master, &writefds);
fprintf(stdout,"select\n");
r= select(master+1, 0,&writefds,0, 0);
if (r==-1) {
fprintf(stdout,"select-fail %s\n", strerror(errno));
continue;
}
assert(r==1);
assert(FD_ISSET(master, &writefds)); fprintf(stdout,"write\n");
r= write(master,buf,sizeof(buf));
if (r==-1) {
fprintf(stdout,"write-fail %s\n", strerror(errno));
continue;
}
assert(r>=0 && r<=sizeof(buf));
fprintf(stdout,"wrote %d\n",r); }
}
Quoted 5 lines · On Mon, 27 Sep 2010 10:37:36 -0700 Jeremy Fitzhardinge <[email protected]> wrote:
Hi,
I noticed on one of my systems that powertop was showing events/0 waking at a high rate (~2000 times/sec, HZ=1000). After a lot of tracing I found it was due to writes pending to ptys which hadn't been read.
Yes ptys will trigger this case while its there with a normal tty but very hard to hit.
Quoted 2 lines
problem; he reports that this wakeup behaviour is a regression compared to a 2.6.16 Debian kernel.
Be aware that you'll get misleading results to an extent with old kernels. The old pty code is differently buggy and doesn't use the code path in question at all.
With a non pty and care you can duplicate the problem on old kernels.
Really the line discipline should wake the work queue when it sets tty->receive_room non-zero, but while only n_tty currently uses that facility the existing code doesn't do it in any kind of race-free manner and sometimes is only saved by the polling picking it up.
It's all really just a symptom of the fact that input and output buffers shouldn't be attached to the tty in the first place but to a struct representing the physical port. Fix that and the race conditions in serial output go away, as do the potential crashes and this wakeup stuff as well as a ton of locking in the irq/tx/rx paths. In several cases it also saves you an entire copy.
Unfortunately while I got the tty port structures into lots of places needed the job never gone done.
Alan
Quoted 12 lines · Alan Cox
Really the line discipline should wake the work queue when it sets
tty->receive_roomnon-zero, but while onlyn_ttycurrently uses that facility the existing code doesn't do it in any kind of race-free manner and sometimes is only saved by the polling picking it up.It's all really just a symptom of the fact that input and output buffers shouldn't be attached to the tty in the first place but to a struct representing the physical port. Fix that and the race conditions in serial output go away, as do the potential crashes and this wakeup stuff as well as a ton of locking in the irq/tx/rx paths. In several cases it also saves you an entire copy.
Quoted 2 lines
Unfortunately while I got the tty port structures into lots of places needed the job never gone done.
OK, so it sounds like there's a basic design problem here which will need some non-trivial work to fix. In the meantime we'll need to look at doing something to work around the issue, since it ends up consuming a non-trivial amount of CPU in events/0. I guess reducing HZ would be the first, simplest thing to do, but changing xenconsoled to avoid writing to readerless ptys might not be too hard.
Thanks,
J
Quoted 16 lines · Jeremy Fitzhardinge
Quoted 12 lines · Alan Cox
Really the line discipline should wake the work queue when it sets
tty->receive_roomnon-zero, but while onlyn_ttycurrently uses that facility the existing code doesn't do it in any kind of race-free manner and sometimes is only saved by the polling picking it up.It's all really just a symptom of the fact that input and output buffers shouldn't be attached to the tty in the first place but to a struct representing the physical port. Fix that and the race conditions in serial output go away, as do the potential crashes and this wakeup stuff as well as a ton of locking in the irq/tx/rx paths. In several cases it also saves you an entire copy.
Quoted 2 lines
Unfortunately while I got the tty port structures into lots of places needed the job never gone done.
Quoted 6 lines
OK, so it sounds like there's a basic design problem here which will need some non-trivial work to fix. In the meantime we'll need to look at doing something to work around the issue, since it ends up consuming a non-trivial amount of CPU in events/0. I guess reducing HZ would be the first, simplest thing to do, but changing xenconsoled to avoid writing to readerless ptys might not be too hard.
Replace "1" in schedule_delayed_work with (HZ/100) should make it better, and looks like good thing while better fix is prepared?
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html