Skip to contentSkip to conversation

printk: Do not lose last line in kmsg buffer dump

PATCH In review Vincent Whitchurch 4 participants 8 messages Jul 11, 2019 – Jul 16, 2019 Printk

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 printk: Do not lose last line in kmsg buffer dump

kmsg_dump_get_buffer() is supposed to select all the youngest log messages which fit into the provided buffer. It determines the correct start index by using msg_print_text() with a NULL buffer to calculate the size of each entry. However, when performing the actual writes, msg_print_text() only writes the entry to the buffer if the written len is lesser than the size of the buffer. So if the lengths of the selected youngest log messages happen to precisely fill up the provided buffer, the last log message is not included.

We don't want to modify msg_print_text() to fill up the buffer and start returning a length which is equal to the size of the buffer, since callers of its other users, such as kmsg_dump_get_line(), depend upon the current behaviour.

Instead, fix kmsg_dump_get_buffer() to compensate for this.

For example, with the following two final prints:

[    6.427502] AAAAAAAAAAAAA
[    6.427769] BBBBBBBB12345

A dump of a 64-byte buffer filled by kmsg_dump_get_buffer(), before this patch:

 00000000: 3c 30 3e 5b 20 20 20 20 36 2e 35 32 32 31 39 37  <0>[    6.522197
 00000010: 5d 20 41 41 41 41 41 41 41 41 41 41 41 41 41 0a  ] AAAAAAAAAAAAA.
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

After this patch:

 00000000: 3c 30 3e 5b 20 20 20 20 36 2e 34 35 36 36 37 38  <0>[    6.456678
 00000010: 5d 20 42 42 42 42 42 42 42 42 31 32 33 34 35 0a  ] BBBBBBBB12345.
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

---
v2: Move fix to kmsg_dump_get_buffer()

 kernel/printk/printk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/printk/printk.c +1 -1
index 1888f6a3b694..424abf802f02 100644
@@ -3274,7 +3274,7 @@ bool kmsg_dump_get_buffer(struct kmsg_dumper *dumper, bool syslog,
 	/* move first record forward until length fits into the buffer */
 	seq = dumper->cur_seq;
 	idx = dumper->cur_idx;
-	while (l > size && seq < dumper->next_seq) {
+	while (l >= size && seq < dumper->next_seq) {
 		struct printk_log *msg = log_from_idx(idx);
 
 		l -= msg_print_text(msg, true, time, NULL, 0);
Signature · 1 line
2.20.0
maintainer
#1 lore raw
Quoted 48 lines · Vincent Whitchurch

kmsg_dump_get_buffer() is supposed to select all the youngest log messages which fit into the provided buffer. It determines the correct start index by using msg_print_text() with a NULL buffer to calculate the size of each entry. However, when performing the actual writes, msg_print_text() only writes the entry to the buffer if the written len is lesser than the size of the buffer. So if the lengths of the selected youngest log messages happen to precisely fill up the provided buffer, the last log message is not included.

We don't want to modify msg_print_text() to fill up the buffer and start returning a length which is equal to the size of the buffer, since callers of its other users, such as kmsg_dump_get_line(), depend upon the current behaviour.

Instead, fix kmsg_dump_get_buffer() to compensate for this.

For example, with the following two final prints:

[    6.427502] AAAAAAAAAAAAA
[    6.427769] BBBBBBBB12345

A dump of a 64-byte buffer filled by kmsg_dump_get_buffer(), before this patch:

 00000000: 3c 30 3e 5b 20 20 20 20 36 2e 35 32 32 31 39 37  <0>[    6.522197
 00000010: 5d 20 41 41 41 41 41 41 41 41 41 41 41 41 41 0a  ] AAAAAAAAAAAAA.
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

After this patch:

 00000000: 3c 30 3e 5b 20 20 20 20 36 2e 34 35 36 36 37 38  <0>[    6.456678
 00000010: 5d 20 42 42 42 42 42 42 42 42 31 32 33 34 35 0a  ] BBBBBBBB12345.
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

---
v2: Move fix to kmsg_dump_get_buffer()

 kernel/printk/printk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/printk/printk.c +0 -0
index 1888f6a3b694..424abf802f02 100644

kernel/printk/printk.c bool kmsg_dump_get_buffer(struct kmsg_dumper *dumper, bool syslog, · line 3277

Quoted 5 lines
kernel/printk/printk.c +1 -1
@@ -3274,7 +3274,7 @@ bool kmsg_dump_get_buffer(struct kmsg_dumper *dumper, bool syslog,
 	/* move first record forward until length fits into the buffer */
 	seq = dumper->cur_seq;
 	idx = dumper->cur_idx;
-	while (l > size && seq < dumper->next_seq) {
+	while (l >= size && seq < dumper->next_seq) {

This cycle searches how many messages would fit into the buffer.

The patch looks like a hack using a hole that the next cycle does not longer check the number of really stored characters.

What would happen when msg_print_text() starts adding the trailing '\0' as suggested by
https://lkml.kernel.org/r/[email protected]

I would much more appreciate if we make the code more secure instead of stretching its weakness to the limits.

BTW: What is the motivation for this fix? Is a bug report or just some research of possible buffer overflows?

The commit message pretends that the problem is bigger than it really is. It is about one byte and not one line.

Best Regards,
Petr

#2 lore raw
Quoted 6 lines · Petr Mladek

The patch looks like a hack using a hole that the next cycle does not longer check the number of really stored characters.

What would happen when msg_print_text() starts adding the trailing '\0' as suggested by
https://lkml.kernel.org/r/[email protected]

I did have a look at that possibility, but I didn't see how that could work without potentially affecting userspace users of the syslog ABI. AFAICS the suggested change in msg_print_text() can be done in one of three ways:

(1) msg_print_text() adds the '\0' and includes this length both when it estimates the size (NULL buffer) and when it actually prints:

If we do this:
- kmsg_dump_get_line_nolock() would have to subtract 1 from the len since its callers expected that len is always smaller than the size of the buffer.
- The buffers given to use via the syslog interface will now include a '\0', potentially affecting userspace applications which use this ABI.

(2) msg_print_text() adds the '\0', and includes this in the length only when estimating the size, and not when it actually prints.

If we do this:
- SYSLOG_ACTION_SIZE_UNREAD tries uses the size estimate to give userspace a count of how many characters are present in the buffer, and now this count will start differing from the actual count that can be read, potentially affecting userspace applications.

(3) msg_print_text() adds the '\0', and does not include this length in the result at all.

If we do this:
- The original kmsg dump issue is not solved, since the last line is still lost.

Quoted 2 lines

BTW: What is the motivation for this fix? Is a bug report or just some research of possible buffer overflows?

The fix is not attempting to fix a buffer overflow, theoretical or otherwise.

It's a fix for a bug in functionality which has been observed on our
systems: We use pstore to save the kernel log when the kernel crashes, and sometimes the log in the pstore misses the last line, and since the last line usual says why we're panicing so it's rather important not to miss.

Quoted 2 lines

The commit message pretends that the problem is bigger than it really is. It is about one byte and not one line.

I'm not quite sure I follow. The current code does fail to include the *entire* last line.

The memcpy on line #1294 is never executed for the last line because we stop the loop because of the check on line #1289:

  1270  static size_t msg_print_text(const struct printk_log *msg, bool syslog, char *buf, size_t size)
  1271  {
  1272          const char *text = log_text(msg);
  1273          size_t text_size = msg->text_len;
  1274          size_t len = 0;
  1275  
  1276          do {
  1277                  const char *next = memchr(text, '\n', text_size);
  1278                  size_t text_len;
  1279  
  1280                  if (next) {
  1281                          text_len = next - text;
  1282                          next++;
  1283                          text_size -= next - text;
  1284                  } else {
  1285                          text_len = text_size;
  1286                  }
  1287  
  1288                  if (buf) {
  1289                          if (print_prefix(msg, syslog, NULL) +
  1290                              text_len + 1 > size - len)
  1291                                  break;
  1292  
  1293                          len += print_prefix(msg, syslog, buf + len);
  1294                          memcpy(buf + len, text, text_len);
  1295                          len += text_len;
  1296                          buf[len++] = '\n';
  1297                  } else {
  1298                          /* SYSLOG_ACTION_* buffer size only calculation */
  1299                          len += print_prefix(msg, syslog, NULL);
  1300                          len += text_len;
  1301                          len++;
  1302                  }
maintainer Reviewed-by on v2 1/1
#3 lore raw
In reply to Vincent Whitchurch
Quoted 29 lines · Vincent Whitchurch

kmsg_dump_get_buffer() is supposed to select all the youngest log messages which fit into the provided buffer. It determines the correct start index by using msg_print_text() with a NULL buffer to calculate the size of each entry. However, when performing the actual writes, msg_print_text() only writes the entry to the buffer if the written len is lesser than the size of the buffer. So if the lengths of the selected youngest log messages happen to precisely fill up the provided buffer, the last log message is not included.

We don't want to modify msg_print_text() to fill up the buffer and start returning a length which is equal to the size of the buffer, since callers of its other users, such as kmsg_dump_get_line(), depend upon the current behaviour.

Instead, fix kmsg_dump_get_buffer() to compensate for this.

For example, with the following two final prints:

[    6.427502] AAAAAAAAAAAAA
[    6.427769] BBBBBBBB12345

A dump of a 64-byte buffer filled by kmsg_dump_get_buffer(), before this patch:

 00000000: 3c 30 3e 5b 20 20 20 20 36 2e 35 32 32 31 39 37  <0>[    6.522197
 00000010: 5d 20 41 41 41 41 41 41 41 41 41 41 41 41 41 0a  ] AAAAAAAAAAAAA.
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Quoted 8 lines

After this patch:

 00000000: 3c 30 3e 5b 20 20 20 20 36 2e 34 35 36 36 37 38  <0>[    6.456678
 00000010: 5d 20 42 42 42 42 42 42 42 42 31 32 33 34 35 0a  ] BBBBBBBB12345.
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

I think that I need vacation. I have got lost in all the checks and got it wrongly in the morning.

This patch fixes the calculation of messages that might fit into the buffer. It makes sure that the function that writes the messages will really allow to write them.

It seems to be the correct fix.

Best Regards,
Petr

Reviewed-by
#4 lore raw
Quoted 25 lines · Petr Mladek
Quoted 13 lines

For example, with the following two final prints:

[    6.427502] AAAAAAAAAAAAA
[    6.427769] BBBBBBBB12345

A dump of a 64-byte buffer filled by kmsg_dump_get_buffer(), before this patch:

 00000000: 3c 30 3e 5b 20 20 20 20 36 2e 35 32 32 31 39 37  <0>[    6.522197
 00000010: 5d 20 41 41 41 41 41 41 41 41 41 41 41 41 41 0a  ] AAAAAAAAAAAAA.
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Quoted 8 lines

After this patch:

 00000000: 3c 30 3e 5b 20 20 20 20 36 2e 34 35 36 36 37 38  <0>[    6.456678
 00000010: 5d 20 42 42 42 42 42 42 42 42 31 32 33 34 35 0a  ] BBBBBBBB12345.
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

I think that I need vacation. I have got lost in all the checks and got it wrongly in the morning.

Quoted 7 lines

This patch fixes the calculation of messages that might fit into the buffer. It makes sure that the function that writes the messages will really allow to write them.

It seems to be the correct fix.

Looks correct to me as well.

	-ss
maintainer
#5 lore raw
Quoted 30 lines · Sergey Senozhatsky
Quoted 29 lines · Petr Mladek
Quoted 13 lines

For example, with the following two final prints:

[    6.427502] AAAAAAAAAAAAA
[    6.427769] BBBBBBBB12345

A dump of a 64-byte buffer filled by kmsg_dump_get_buffer(), before this patch:

 00000000: 3c 30 3e 5b 20 20 20 20 36 2e 35 32 32 31 39 37  <0>[    6.522197
 00000010: 5d 20 41 41 41 41 41 41 41 41 41 41 41 41 41 0a  ] AAAAAAAAAAAAA.
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Quoted 8 lines

After this patch:

 00000000: 3c 30 3e 5b 20 20 20 20 36 2e 34 35 36 36 37 38  <0>[    6.456678
 00000010: 5d 20 42 42 42 42 42 42 42 42 31 32 33 34 35 0a  ] BBBBBBBB12345.
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

I think that I need vacation. I have got lost in all the checks and got it wrongly in the morning.

This patch fixes the calculation of messages that might fit into the buffer. It makes sure that the function that writes the messages will really allow to write them.

Quoted 7 lines
Quoted 3 lines

It seems to be the correct fix.

Looks correct to me as well.

The patch has been committed into printk.git, branch for-5.3-fixes.

I am still a bit undecided whether to send pull request the following week or wait for 5.4. On one hand, it is very old bug (since 3.5). On the other hand, I think that it was not reported/fixed earlier only because it was hard to notice. And loosing the very last message is quite pity.

Anyway, I added stable tag.

Best Regards,
Petr

#6 lore raw
Quoted 11 lines · Petr Mladek
Quoted 3 lines

Looks correct to me as well.

The patch has been committed into printk.git, branch for-5.3-fixes.

I am still a bit undecided whether to send pull request the following week or wait for 5.4. On one hand, it is very old bug (since 3.5). On the other hand, I think that it was not reported/fixed earlier only because it was hard to notice. And loosing the very last message is quite pity.

My call would be - let's wait till next merge window.

	-ss
maintainer
#7 lore raw
Quoted 7 lines · Sergey Senozhatsky
Quoted 6 lines · Petr Mladek
Quoted 3 lines

Looks correct to me as well.

The patch has been committed into printk.git, branch for-5.3-fixes.

Quoted 7 lines
Quoted 5 lines

I am still a bit undecided whether to send pull request the following week or wait for 5.4. On one hand, it is very old bug (since 3.5). On the other hand, I think that it was not reported/fixed earlier only because it was hard to notice. And loosing the very last message is quite pity.

My call would be - let's wait till next merge window.

Thanks for your opinion.

I'll leave it for 5.4 unless there is another urgent fix that would trigger earlier pull request.

Best Regards,
Petr