Regression due to 7ff9554 "printk: convert byte-buffer to variable-length record buffer"
For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:
7ff9554 printk: convert byte-buffer to variable-length record buffer
In order to build, I also had to revert:
c4e00da driver-core: extend dev_printk() to pass structured data
Note: I'm running on an ARM system using a serial console, with earlyprintk enabled.
Quoted 11 lines · Stephen Warren
For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:
7ff9554 printk: convert byte-buffer to variable-length record buffer
In order to build, I also had to revert:
c4e00da driver-core: extend
dev_printk()to pass structured dataNote: I'm running on an ARM system using a serial console, with earlyprintk enabled.
This issue still occurs in next-20120510.
I've tracked it down to the assignment of msg->ts_nsec near the end of log_store(). If I comment this out, everything works. The problem is the assignment, not the call to local_clock():
fails:
msg->ts_nsec = local_clock(); fails: msg->ts_nsec = 0;//local_clock(); works: //msg->ts_nsec = local_clock();
I guess that msg isn't pointing at what it's supposed to. I'll keep trying to track this down, but if you could think about why this might happen (or even revert the changes until this is isolated), it'd be great.
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren
For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:
7ff9554 printk: convert byte-buffer to variable-length record buffer
In order to build, I also had to revert:
c4e00da driver-core: extend
dev_printk()to pass structured dataNote: I'm running on an ARM system using a serial console, with earlyprintk enabled.
This issue still occurs in next-20120510.
I've tracked it down to the assignment of
msg->ts_nsecnear the end oflog_store(). If I comment this out, everything works. The problem is the assignment, not the call tolocal_clock():
Quoted 6 lines
fails:
msg->ts_nsec =local_clock();
fails:
msg->ts_nsec = 0;//local_clock();
works:
//msg->ts_nsec =local_clock();
Weird.
What happens if you change it to:
cpu_clock(logbuf_cpu);
?
If it works, the timestamps look ok?
Kay
Quoted 27 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren
For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:
7ff9554 printk: convert byte-buffer to variable-length record buffer
In order to build, I also had to revert:
c4e00da driver-core: extend
dev_printk()to pass structured dataNote: I'm running on an ARM system using a serial console, with earlyprintk enabled.
This issue still occurs in next-20120510.
I've tracked it down to the assignment of
msg->ts_nsecnear the end oflog_store(). If I comment this out, everything works. The problem is the assignment, not the call tolocal_clock():Quoted 6 lines
fails:
msg->ts_nsec = local_clock(); fails: msg->ts_nsec = 0;//local_clock(); works: //msg->ts_nsec = local_clock();
Quoted 7 lines
Weird.
What happens if you change it to:
cpu_clock(logbuf_cpu);
?If it works, the timestamps look ok?
I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...
Quoted 30 lines · Stephen Warren
Quoted 29 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren
For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:
7ff9554 printk: convert byte-buffer to variable-length record buffer
In order to build, I also had to revert:
c4e00da driver-core: extend
dev_printk()to pass structured dataNote: I'm running on an ARM system using a serial console, with earlyprintk enabled.
This issue still occurs in next-20120510.
I've tracked it down to the assignment of
msg->ts_nsecnear the end oflog_store(). If I comment this out, everything works. The problem is the assignment, not the call tolocal_clock():Quoted 6 lines
fails:
msg->ts_nsec = local_clock(); fails: msg->ts_nsec = 0;//local_clock(); works: //msg->ts_nsec = local_clock();Weird.
Quoted 8 lines
Quoted 5 lines
What happens if you change it to:
cpu_clock(logbuf_cpu);
?If it works, the timestamps look ok?
I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...
Calling cpu_clock() instead of local_clock() fails in the same way.
Quoted 35 lines · Stephen Warren
Quoted 34 lines · Stephen Warren
Quoted 33 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren
For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:
7ff9554 printk: convert byte-buffer to variable-length record buffer
In order to build, I also had to revert:
c4e00da driver-core: extend
dev_printk()to pass structured dataNote: I'm running on an ARM system using a serial console, with earlyprintk enabled.
This issue still occurs in next-20120510.
I've tracked it down to the assignment of
msg->ts_nsecnear the end oflog_store(). If I comment this out, everything works. The problem is the assignment, not the call tolocal_clock():Quoted 6 lines
fails:
msg->ts_nsec =local_clock();
fails:
msg->ts_nsec = 0;//local_clock();
works:
//msg->ts_nsec =local_clock();Weird.
What happens if you change it to:
cpu_clock(logbuf_cpu);
?
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
If it works, the timestamps look ok?
I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...
Calling
cpu_clock()instead oflocal_clock()fails in the same way.
Ok, didn't really see the assign to 0 you tried, sorry. :)
And 'dmesg' works when you run the box with the line commented out?
And 'cat /dev/kmsg'?
Kay
Quoted 41 lines · Kay Sievers
Quoted 40 lines · Stephen Warren
Quoted 38 lines · Stephen Warren
Quoted 27 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren
For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:
7ff9554 printk: convert byte-buffer to variable-length record buffer
In order to build, I also had to revert:
c4e00da driver-core: extend
dev_printk()to pass structured dataNote: I'm running on an ARM system using a serial console, with earlyprintk enabled.
This issue still occurs in next-20120510.
I've tracked it down to the assignment of
msg->ts_nsecnear the end oflog_store(). If I comment this out, everything works. The problem is the assignment, not the call tolocal_clock():Quoted 6 lines
fails:
msg->ts_nsec = local_clock(); fails: msg->ts_nsec = 0;//local_clock(); works: //msg->ts_nsec = local_clock();Quoted 7 lines
Weird.
What happens if you change it to:
cpu_clock(logbuf_cpu);
?If it works, the timestamps look ok?
I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...
Quoted 7 lines
Quoted 1 line
Calling
cpu_clock()instead oflocal_clock()fails in the same way.Ok, didn't really see the assign to 0 you tried, sorry. :)
And 'dmesg' works when you run the box with the line commented out?
And 'cat
/dev/kmsg'?
Yes, both work and produce reasonable output. "cat /dev/kmsg" does hang at the end of the log until I CTRL-C it - is that expected?
Quoted 44 lines · Stephen Warren
Quoted 43 lines · Kay Sievers
Quoted 41 lines · Stephen Warren
Quoted 30 lines · Stephen Warren
Quoted 29 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren
For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:
7ff9554 printk: convert byte-buffer to variable-length record buffer
In order to build, I also had to revert:
c4e00da driver-core: extend
dev_printk()to pass structured dataNote: I'm running on an ARM system using a serial console, with earlyprintk enabled.
This issue still occurs in next-20120510.
I've tracked it down to the assignment of
msg->ts_nsecnear the end oflog_store(). If I comment this out, everything works. The problem is the assignment, not the call tolocal_clock():Quoted 6 lines
fails:
msg->ts_nsec =local_clock();
fails:
msg->ts_nsec = 0;//local_clock();
works:
//msg->ts_nsec =local_clock();Weird.
Quoted 8 lines
Quoted 5 lines
What happens if you change it to:
cpu_clock(logbuf_cpu);
?If it works, the timestamps look ok?
I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...
Calling
cpu_clock()instead oflocal_clock()fails in the same way.
Quoted 7 lines
Quoted 5 lines
Ok, didn't really see the assign to 0 you tried, sorry. :)
And 'dmesg' works when you run the box with the line commented out?
And 'cat
/dev/kmsg'?Yes, both work and produce reasonable output.
Really weird. I have zero ideas at what this could be. The messages in /dev/kmsg have seqnums, you see the 0 as the first?
5,0,0;Linux version 3.4.0-rc6+ ...
6,1,0;Command line: root=/dev/sda1 ...
Quoted 2 lines
"cat
/dev/kmsg" does hang
at the end of the log until I CTRL-C it - is that expected?
Yeah, 'cat' is a blocking read(), tools would use poll().
Kay
Quoted 56 lines · Kay Sievers
Quoted 44 lines · Stephen Warren
Quoted 43 lines · Kay Sievers
Quoted 41 lines · Stephen Warren
Quoted 30 lines · Stephen Warren
Quoted 29 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren
For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:
7ff9554 printk: convert byte-buffer to variable-length record buffer
In order to build, I also had to revert:
c4e00da driver-core: extend
dev_printk()to pass structured dataNote: I'm running on an ARM system using a serial console, with earlyprintk enabled.
This issue still occurs in next-20120510.
I've tracked it down to the assignment of
msg->ts_nsecnear the end oflog_store(). If I comment this out, everything works. The problem is the assignment, not the call tolocal_clock():Quoted 6 lines
fails:
msg->ts_nsec = local_clock(); fails: msg->ts_nsec = 0;//local_clock(); works: //msg->ts_nsec = local_clock();Weird.
Quoted 8 lines
Quoted 5 lines
What happens if you change it to:
cpu_clock(logbuf_cpu);
?If it works, the timestamps look ok?
I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...
Calling
cpu_clock()instead oflocal_clock()fails in the same way.Quoted 7 lines
Quoted 5 lines
Ok, didn't really see the assign to 0 you tried, sorry. :)
And 'dmesg' works when you run the box with the line commented out?
And 'cat
/dev/kmsg'?Yes, both work and produce reasonable output.
Really weird. I have zero ideas at what this could be. The messages in
/dev/kmsghave seqnums, you see the 0 as the first?
Quoted 7 lines
5,0,0;Linux version 3.4.0-rc6+ ...
6,1,0;Command line: root=/dev/sda1 ...Quoted 2 lines
"cat
/dev/kmsg" does hang
at the end of the log until I CTRL-C it - is that expected?Yeah, 'cat' is a blocking
read(), tools would usepoll().
Found the bug. __log_buf isn't correctly aligned, so the very first message gets the struct log written to an unaligned address. Subsequent entries would be correctly aligned in log_store(), except the kernel has already crashed by then.
The ts_nsec field is a 64-bit write, which triggers the alignment requirement. I'm not quite sure why the previous 16-bit writes for the various len fields didn't experience an issue, since __log_buf ended up at an odd address for me. Perhaps ARM doesn't have an alignment requirement for 16-bit writes, but does for 32-bit and greater.
I'll post a patch in a minute.