Skip to contentSkip to conversation

Regression due to 7ff9554 "printk: convert byte-buffer to variable-length record buffer"

DISCUSSION Stephen Warren 2 participants 9 messages May 9, 2012 – May 10, 2012 Linux-next tree

lore
Share & tools
lore
#0 lore raw

For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:

7ff9554 printk: convert byte-buffer to variable-length record buffer

In order to build, I also had to revert:

c4e00da driver-core: extend dev_printk() to pass structured data

Note: I'm running on an ARM system using a serial console, with earlyprintk enabled.

#1 lore raw
Quoted 11 lines · Stephen Warren

For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:

7ff9554 printk: convert byte-buffer to variable-length record buffer

In order to build, I also had to revert:

c4e00da driver-core: extend dev_printk() to pass structured data

Note: I'm running on an ARM system using a serial console, with earlyprintk enabled.

This issue still occurs in next-20120510.

I've tracked it down to the assignment of msg->ts_nsec near the end of log_store(). If I comment this out, everything works. The problem is the assignment, not the call to local_clock():

fails:

	msg->ts_nsec = local_clock();
fails:
	msg->ts_nsec = 0;//local_clock();
works:
	//msg->ts_nsec = local_clock();

I guess that msg isn't pointing at what it's supposed to. I'll keep trying to track this down, but if you could think about why this might happen (or even revert the changes until this is isolated), it'd be great.

#2 lore raw
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren

For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:

7ff9554 printk: convert byte-buffer to variable-length record buffer

In order to build, I also had to revert:

c4e00da driver-core: extend dev_printk() to pass structured data

Note: I'm running on an ARM system using a serial console, with earlyprintk enabled.

This issue still occurs in next-20120510.

I've tracked it down to the assignment of msg->ts_nsec near the end of log_store(). If I comment this out, everything works. The problem is the assignment, not the call to local_clock():

Quoted 6 lines

fails:
       msg->ts_nsec = local_clock();
fails:
       msg->ts_nsec = 0;//local_clock();
works:
       //msg->ts_nsec = local_clock();

Weird.

What happens if you change it to:
cpu_clock(logbuf_cpu);
?

If it works, the timestamps look ok?

Kay

#3 lore raw
Quoted 27 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren

For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:

7ff9554 printk: convert byte-buffer to variable-length record buffer

In order to build, I also had to revert:

c4e00da driver-core: extend dev_printk() to pass structured data

Note: I'm running on an ARM system using a serial console, with earlyprintk enabled.

This issue still occurs in next-20120510.

I've tracked it down to the assignment of msg->ts_nsec near the end of log_store(). If I comment this out, everything works. The problem is the assignment, not the call to local_clock():

Quoted 6 lines

fails:

       msg->ts_nsec = local_clock();
fails:
       msg->ts_nsec = 0;//local_clock();
works:
       //msg->ts_nsec = local_clock();
Quoted 7 lines

Weird.

What happens if you change it to:
cpu_clock(logbuf_cpu);
?

If it works, the timestamps look ok?

I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...

#4 lore raw
Quoted 30 lines · Stephen Warren
Quoted 29 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren

For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:

7ff9554 printk: convert byte-buffer to variable-length record buffer

In order to build, I also had to revert:

c4e00da driver-core: extend dev_printk() to pass structured data

Note: I'm running on an ARM system using a serial console, with earlyprintk enabled.

This issue still occurs in next-20120510.

I've tracked it down to the assignment of msg->ts_nsec near the end of log_store(). If I comment this out, everything works. The problem is the assignment, not the call to local_clock():

Quoted 6 lines

fails:

       msg->ts_nsec = local_clock();
fails:
       msg->ts_nsec = 0;//local_clock();
works:
       //msg->ts_nsec = local_clock();

Weird.

Quoted 8 lines
Quoted 5 lines

What happens if you change it to:
cpu_clock(logbuf_cpu);
?

If it works, the timestamps look ok?

I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...

Calling cpu_clock() instead of local_clock() fails in the same way.

#5 lore raw
Quoted 35 lines · Stephen Warren
Quoted 34 lines · Stephen Warren
Quoted 33 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren

For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:

7ff9554 printk: convert byte-buffer to variable-length record buffer

In order to build, I also had to revert:

c4e00da driver-core: extend dev_printk() to pass structured data

Note: I'm running on an ARM system using a serial console, with earlyprintk enabled.

This issue still occurs in next-20120510.

I've tracked it down to the assignment of msg->ts_nsec near the end of log_store(). If I comment this out, everything works. The problem is the assignment, not the call to local_clock():

Quoted 6 lines

fails:
       msg->ts_nsec = local_clock();
fails:
       msg->ts_nsec = 0;//local_clock();
works:
       //msg->ts_nsec = local_clock();

Weird.

What happens if you change it to:
  cpu_clock(logbuf_cpu);
?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

If it works, the timestamps look ok?

I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...

Calling cpu_clock() instead of local_clock() fails in the same way.

Ok, didn't really see the assign to 0 you tried, sorry. :)

And 'dmesg' works when you run the box with the line commented out?

And 'cat /dev/kmsg'?

Kay

#6 lore raw
Quoted 41 lines · Kay Sievers
Quoted 40 lines · Stephen Warren
Quoted 38 lines · Stephen Warren
Quoted 27 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren

For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:

7ff9554 printk: convert byte-buffer to variable-length record buffer

In order to build, I also had to revert:

c4e00da driver-core: extend dev_printk() to pass structured data

Note: I'm running on an ARM system using a serial console, with earlyprintk enabled.

This issue still occurs in next-20120510.

I've tracked it down to the assignment of msg->ts_nsec near the end of log_store(). If I comment this out, everything works. The problem is the assignment, not the call to local_clock():

Quoted 6 lines

fails:

       msg->ts_nsec = local_clock();
fails:
       msg->ts_nsec = 0;//local_clock();
works:
       //msg->ts_nsec = local_clock();
Quoted 7 lines

Weird.

What happens if you change it to:
cpu_clock(logbuf_cpu);
?

If it works, the timestamps look ok?

I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...

Quoted 7 lines
Quoted 1 line

Calling cpu_clock() instead of local_clock() fails in the same way.

Ok, didn't really see the assign to 0 you tried, sorry. :)

And 'dmesg' works when you run the box with the line commented out?

And 'cat /dev/kmsg'?

Yes, both work and produce reasonable output. "cat /dev/kmsg" does hang at the end of the log until I CTRL-C it - is that expected?

#7 lore raw
Quoted 44 lines · Stephen Warren
Quoted 43 lines · Kay Sievers
Quoted 41 lines · Stephen Warren
Quoted 30 lines · Stephen Warren
Quoted 29 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren

For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:

7ff9554 printk: convert byte-buffer to variable-length record buffer

In order to build, I also had to revert:

c4e00da driver-core: extend dev_printk() to pass structured data

Note: I'm running on an ARM system using a serial console, with earlyprintk enabled.

This issue still occurs in next-20120510.

I've tracked it down to the assignment of msg->ts_nsec near the end of log_store(). If I comment this out, everything works. The problem is the assignment, not the call to local_clock():

Quoted 6 lines

fails:
       msg->ts_nsec = local_clock();
fails:
       msg->ts_nsec = 0;//local_clock();
works:
       //msg->ts_nsec = local_clock();

Weird.

Quoted 8 lines
Quoted 5 lines

What happens if you change it to:
  cpu_clock(logbuf_cpu);
?

If it works, the timestamps look ok?

I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...

Calling cpu_clock() instead of local_clock() fails in the same way.

Quoted 7 lines
Quoted 5 lines

Ok, didn't really see the assign to 0 you tried, sorry. :)

And 'dmesg' works when you run the box with the line commented out?

And 'cat /dev/kmsg'?

Yes, both work and produce reasonable output.

Really weird. I have zero ideas at what this could be. The messages in /dev/kmsg have seqnums, you see the 0 as the first?

5,0,0;Linux version 3.4.0-rc6+ ...
6,1,0;Command line: root=/dev/sda1 ...

Quoted 2 lines

"cat /dev/kmsg" does hang
at the end of the log until I CTRL-C it - is that expected?

Yeah, 'cat' is a blocking read(), tools would use poll().

Kay

#8 lore raw
Quoted 56 lines · Kay Sievers
Quoted 44 lines · Stephen Warren
Quoted 43 lines · Kay Sievers
Quoted 41 lines · Stephen Warren
Quoted 30 lines · Stephen Warren
Quoted 29 lines · Kay Sievers
Quoted 19 lines · Stephen Warren
Quoted 11 lines · Stephen Warren

For me, next-20120508 prints nothing when booted, and I think also hangs. To solve this, I reverted:

7ff9554 printk: convert byte-buffer to variable-length record buffer

In order to build, I also had to revert:

c4e00da driver-core: extend dev_printk() to pass structured data

Note: I'm running on an ARM system using a serial console, with earlyprintk enabled.

This issue still occurs in next-20120510.

I've tracked it down to the assignment of msg->ts_nsec near the end of log_store(). If I comment this out, everything works. The problem is the assignment, not the call to local_clock():

Quoted 6 lines

fails:

       msg->ts_nsec = local_clock();
fails:
       msg->ts_nsec = 0;//local_clock();
works:
       //msg->ts_nsec = local_clock();

Weird.

Quoted 8 lines
Quoted 5 lines

What happens if you change it to:
cpu_clock(logbuf_cpu);
?

If it works, the timestamps look ok?

I doubt that would work - after all, assigning 0 fails, but not performing the assignment at all works. But, I'll go try it...

Calling cpu_clock() instead of local_clock() fails in the same way.

Quoted 7 lines
Quoted 5 lines

Ok, didn't really see the assign to 0 you tried, sorry. :)

And 'dmesg' works when you run the box with the line commented out?

And 'cat /dev/kmsg'?

Yes, both work and produce reasonable output.

Really weird. I have zero ideas at what this could be. The messages in /dev/kmsg have seqnums, you see the 0 as the first?

Quoted 7 lines

5,0,0;Linux version 3.4.0-rc6+ ...
6,1,0;Command line: root=/dev/sda1 ...

Quoted 2 lines

"cat /dev/kmsg" does hang
at the end of the log until I CTRL-C it - is that expected?

Yeah, 'cat' is a blocking read(), tools would use poll().

Found the bug. __log_buf isn't correctly aligned, so the very first message gets the struct log written to an unaligned address. Subsequent entries would be correctly aligned in log_store(), except the kernel has already crashed by then.

The ts_nsec field is a 64-bit write, which triggers the alignment requirement. I'm not quite sure why the previous 16-bit writes for the various len fields didn't experience an issue, since __log_buf ended up at an odd address for me. Perhaps ARM doesn't have an alignment requirement for 16-bit writes, but does for 32-bit and greater.

I'll post a patch in a minute.