hexdump: fix for non-aligned buffers

PATCH RESEND Superseded Horacio Mijail Antón Quiles 4 participants 7 messages Jul 8, 2015 – Jul 9, 2015 Library code

Share & tools

Horacio Mijail Antón Quiles

author

2015-07-08 23:44 UTC #0 lore raw

PATCH RESEND v4 1/1 hexdump: fix for non-aligned buffers

An hexdump with a buf not aligned to the groupsize causes non-naturally-aligned memory accesses. This was causing a kernel panic on the processor BlackFin BF527, when such an unaligned buffer was fed by the function ubifs_scanned_corruption in fs/ubifs/scan.c .

To fix this, if the buffer is not aligned to groupsize in a platform which does not define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS, then change the groupsize to 1, so alignment is no longer a problem.
This behavior is coherent with the way the function currently deals with inappropriate parameter combinations, which is to fall back to safe "defaults", even if that means changing the output format and the implicit access patterns that could have been expected.

Signed-off-by Horacio Mijail Antón Quiles <[email protected]>

---
Resent on 8 Jul because of no answers.

Resent on 29 Jun because of no answers.

Changes in v4:
- Added space before "*/" (per Joe Perches' indication)

Changes in v3:
- Removed trailing whitespace (per Joe Perches' indication)
- Changed mail headers to avoid encoding issues (per Joe Perches' hint)

Changes in v2:
- Changed from ad-hoc calculation to IS_ALIGNED() for readability (per Joe Perches' indication)
- Made the explanation clearer (aligned vs naturally aligned)

---
 lib/hexdump.c | 5 +++++
 1 file changed, 5 insertions(+)

lib/hexdump.c +5 -0

index 7ea0969..0601a35 100644
@@ -124,6 +124,11 @@ int hex_dump_to_buffer(const void *buf, size_t len, int rowsize, int groupsize,
 	if ((len % groupsize) != 0)	/* no mixed size output */
 		groupsize = 1;
 
+	/* fall back to 1-byte groups if buf is not aligned to groupsize */
+	if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
+	    !IS_ALIGNED((uintptr_t)buf, groupsize))
+		groupsize = 1;
+
 	ngroups = len / groupsize;
 	ascii_column = rowsize * 2 + rowsize / groupsize + 1;

Signature · 1 line

2.1.4

Joe Perches

2015-07-08 23:49 UTC #1 lore raw

Quoted 4 lines · Horacio Mijail Antón Quiles

An hexdump with a buf not aligned to the groupsize causes non-naturally-aligned memory accesses. This was causing a kernel panic on the processor BlackFin BF527, when such an unaligned buffer was fed by the function ubifs_scanned_corruption in fs/ubifs/scan.c .

Seems sensible enough to me.

lib/hexdump.c

Quoted 2 lines

---
lib/hexdump.c +0 -0

[]

Quoted 12 lines

lib/hexdump.c +5 -0

@@ -124,6 +124,11 @@ int hex_dump_to_buffer(const void *buf, size_t len, int rowsize, int groupsize,
 	if ((len % groupsize) != 0)	/* no mixed size output */
 		groupsize = 1;
 
+	/* fall back to 1-byte groups if buf is not aligned to groupsize */
+	if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
+	    !IS_ALIGNED((uintptr_t)buf, groupsize))
+		groupsize = 1;
+
 	ngroups = len / groupsize;
 	ascii_column = rowsize * 2 + rowsize / groupsize + 1;

Andrew Morton

maintainer

2015-07-09 00:03 UTC #2 lore raw

In reply to Horacio Mijail Antón Quiles

Quoted 5 lines · On Thu, 9 Jul 2015 01:44:18 +0200 Horacio Mijail Ant__n Quiles…

An hexdump with a buf not aligned to the groupsize causes non-naturally-aligned memory accesses. This was causing a kernel panic on the processor BlackFin BF527, when such an unaligned buffer was fed by the function ubifs_scanned_corruption in fs/ubifs/scan.c .

Quoted 7 lines

To fix this, if the buffer is not aligned to groupsize in a platform which does not define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS, then change the groupsize to 1, so alignment is no longer a problem.
This behavior is coherent with the way the function currently deals with inappropriate parameter combinations, which is to fall back to safe "defaults", even if that means changing the output format and the implicit access patterns that could have been expected.

CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS seems inappropriate for this. Having this unset means "can do unaligned accesses, but they're inefficient". It doesn't mean "unaligned accesses go oops".

But I can't the appropriate config knob. There's a CONFIG_CPU_HAS_NO_UNALIGNED, but that's an m68k-private thing.

Quoted 3 lines

Resent on 8 Jul because of no answers.
Resent on 29 Jun because of no answers.

During the merge window. So I have everything sitting there in my patches-to-process pile. The backlog is excessive this time (700+ emails) so I'm thinking I'll change things so I'll henceforth be processing patches-for-the-next-cycle during this-cycle, while keeping patches-for-next-cycle out of linux-next.

lib/hexdump.c int hex_dump_to_buffer(const void *buf, size_t len, int rowsize, int groupsize, · line 127

Quoted 7 lines

lib/hexdump.c +1 -0

@@ -124,6 +124,11 @@ int hex_dump_to_buffer(const void *buf, size_t len, int rowsize, int groupsize,
 	if ((len % groupsize) != 0)	/* no mixed size output */
 		groupsize = 1;
 
+	/* fall back to 1-byte groups if buf is not aligned to groupsize */

That isn't a great comment. It tells us what the code does (which is quite obvious just from reading it) but it doesn't tell us *why* it does it. Something like "if buf is not aligned to groupsize then fall back to 1-byte groups to avoid unaligned memory accesses on architectures which do not support them"?

But as mentioned above, that's different from "architectures which do not support them efficently"! Maybe we need a new config variable.

Or maybe blackfin should be handling the unaligned access trap and transparently handling it, like sparc?

Quoted 7 lines

lib/hexdump.c +4 -0

+	if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
+	    !IS_ALIGNED((uintptr_t)buf, groupsize))
+		groupsize = 1;
+
 
...

H. Mijail

author

2015-07-09 01:36 UTC #3 lore raw

Quoted 18 lines · Andrew Morton

Quoted 5 lines · On Thu, 9 Jul 2015 01:44:18 +0200 Horacio Mijail Ant__n Quiles…
An hexdump with a buf not aligned to the groupsize causes non-naturally-aligned memory accesses. This was causing a kernel panic on the processor BlackFin BF527, when such an unaligned buffer was fed by the function ubifs_scanned_corruption in fs/ubifs/scan.c .
Quoted 7 lines
To fix this, if the buffer is not aligned to groupsize in a platform which does not define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS, then change the groupsize to 1, so alignment is no longer a problem.
This behavior is coherent with the way the function currently deals with inappropriate parameter combinations, which is to fall back to safe "defaults", even if that means changing the output format and the implicit access patterns that could have been expected.

Quoted 6 lines

CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS seems inappropriate for this. Having this unset means "can do unaligned accesses, but they're inefficient". It doesn't mean "unaligned accesses go oops".
But I can't the appropriate config knob. There's a CONFIG_CPU_HAS_NO_UNALIGNED, but that’s an m68k-private thing.

I’m only a newbie, but I will argue that the lesser evil is checking CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS - until a new configure variable is defined.

In Documentation/unaligned-memory-access.txt, an undefined CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is taken as if to mean “the hardware isn’t able to access memory on arbitrary boundaries”.

And I certainly can’t see any more appropriate CONFIG_* variable.

The other alternative in Documentation/unaligned-memory-access.txt is the macro get_unaligned() from asm/unaligned.h. However, using get_unaligned() would mean a much more intrusive patch, since each case of the groupsize would be changed, and anyway we would still need to check CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS to avoid penalising everyone.

Lastly, as noted on the patch’s description, hex_dump_to_buffer() as it is has no qualms about sanitizing the received parameter values down to conservative values, and so any current callers are already used to having the expected output and the implicit memory access patterns changed. So I’d say that changing the groupsize is not only harmless, but expected.

Quoted 10 lines

Quoted 4 lines
Resent on 8 Jul because of no answers.
Resent on 29 Jun because of no answers.
During the merge window. So I have everything sitting there in my patches-to-process pile. The backlog is excessive this time (700+ emails) so I'm thinking I'll change things so I'll henceforth be processing patches-for-the-next-cycle during this-cycle, while keeping patches-for-next-cycle out of linux-next.

No problem for me - if I should squelch the next version of this patch for some time, please let me know.

Quoted 7 lines

lib/hexdump.c +0 -0

@@ -124,6 +124,11 @@ int hex_dump_to_buffer(const void *buf, size_t len, int rowsize, int groupsize,
 	if ((len % groupsize) != 0)	/* no mixed size output */
 		groupsize = 1;

Quoted 7 lines

lib/hexdump.c int hex_dump_to_buffer(const void *buf, size_t len, int rowsize, int groupsize, · line 127
Quoted 1 line
lib/hexdump.c +1 -0
+	/* fall back to 1-byte groups if buf is not aligned to groupsize */
That isn't a great comment. It tells us what the code does (which is quite obvious just from reading it) but it doesn't tell us *why* it does it. Something like "if buf is not aligned to groupsize then fall back to 1-byte groups to avoid unaligned memory accesses on architectures which do not support them”?

Thanks, note taken.

Quoted 7 lines

But as mentioned above, that's different from "architectures which do not support them efficently"! Maybe we need a new config variable.
Or maybe blackfin should be handling the unaligned access trap and transparently handling it, like sparc?

I’ll wait for anyone else to weight in…

Quoted 7 lines

lib/hexdump.c +4 -0

+	if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
+	    !IS_ALIGNED((uintptr_t)buf, groupsize))
+		groupsize = 1;
+
 
...

Andrew Morton

maintainer

2015-07-09 02:06 UTC #4 lore raw

Quoted 25 lines · On Thu, 9 Jul 2015 03:36:02 +0200 "H. Mijail" <[email protected]> wrote:

Quoted 18 lines · Andrew Morton
Quoted 5 lines · On Thu, 9 Jul 2015 01:44:18 +0200 Horacio Mijail Ant__n Quiles…
An hexdump with a buf not aligned to the groupsize causes non-naturally-aligned memory accesses. This was causing a kernel panic on the processor BlackFin BF527, when such an unaligned buffer was fed by the function ubifs_scanned_corruption in fs/ubifs/scan.c .
Quoted 7 lines
To fix this, if the buffer is not aligned to groupsize in a platform which does not define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS, then change the groupsize to 1, so alignment is no longer a problem.
This behavior is coherent with the way the function currently deals with inappropriate parameter combinations, which is to fall back to safe "defaults", even if that means changing the output format and the implicit access patterns that could have been expected.
Quoted 6 lines
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS seems inappropriate for this. Having this unset means "can do unaligned accesses, but they're inefficient". It doesn't mean "unaligned accesses go oops".
But I can't the appropriate config knob. There's a CONFIG_CPU_HAS_NO_UNALIGNED, but that's an m68k-private thing.

Quoted 7 lines

I'm only a newbie, but I will argue that the lesser evil is checking CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS - until a new configure variable is defined.
In Documentation/unaligned-memory-access.txt, an undefined CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is taken as if to mean 'the hardware isn't able to access memory on arbitrary boundaries'.

hm, yes, OK, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is a poor name.

Quoted 5 lines

The other alternative in Documentation/unaligned-memory-access.txt is the macro get_unaligned() from asm/unaligned.h. However, using get_unaligned() would mean a much more intrusive patch, since each case of the groupsize would be changed, and anyway we would still need to check CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS to avoid penalising everyone.

Actually, I think using get_unaligned() would be a better solution. For architectures which have CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y, get_unaligned() should be fast - just one instruction.

This way we avoid having different-appearing output on different architectures.

Quoted 4 lines

Quoted 4 lines
Quoted 3 lines
Resent on 8 Jul because of no answers.
Resent on 29 Jun because of no answers.

Quoted 8 lines

Quoted 5 lines
During the merge window. So I have everything sitting there in my patches-to-process pile. The backlog is excessive this time (700+ emails) so I'm thinking I'll change things so I'll henceforth be processing patches-for-the-next-cycle during this-cycle, while keeping patches-for-next-cycle out of linux-next.
No problem for me - if I should squelch the next version of this patch for some time, please let me know.

The merge window has ended ;)

Quoted 9 lines

Quoted 7 lines
But as mentioned above, that's different from "architectures which do not support them efficently"! Maybe we need a new config variable.
Or maybe blackfin should be handling the unaligned access trap and transparently handling it, like sparc?
I'll wait for anyone else to weight in'

Possibly blackfin *could* emulate unaligned accesses. But according to the documentation, hex_dump_to_buffer() needs to be altered anyway because we cannot rely on the architecture handling such accesses.

Geert Uytterhoeven

2015-07-09 07:16 UTC #5 lore raw

Quoted 6 lines · Andrew Morton

Quoted 5 lines
The other alternative in Documentation/unaligned-memory-access.txt is the macro get_unaligned() from asm/unaligned.h. However, using get_unaligned() would mean a much more intrusive patch, since each case of the groupsize would be changed, and anyway we would still need to check CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS to avoid penalising everyone.

Quoted 6 lines

Actually, I think using get_unaligned() would be a better solution. For architectures which have CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y, get_unaligned() should be fast - just one instruction.
This way we avoid having different-appearing output on different architectures.

Definitely.

A less optimal get_unaligned() will just be noise in the snprintf() processing time.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

H. Mijail

author

2015-07-09 19:08 UTC #6 lore raw

Quoted 10 lines · Geert Uytterhoeven

Quoted 6 lines · Andrew Morton
Quoted 5 lines
The other alternative in Documentation/unaligned-memory-access.txt is the macro get_unaligned() from asm/unaligned.h. However, using get_unaligned() would mean a much more intrusive patch, since each case of the groupsize would be changed, and anyway we would still need to check CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS to avoid penalising everyone.

Quoted 3 lines

Quoted 3 lines
Actually, I think using get_unaligned() would be a better solution. For architectures which have CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y, get_unaligned() should be fast - just one instruction.

D’oh! You’re right, of course.

Quoted 9 lines

Quoted 3 lines
This way we avoid having different-appearing output on different architectures.
Definitely.
A less optimal get_unaligned() will just be noise in the snprintf() processing time.

OK, so thanks Andrew and Geert for your comments. I’ll reimplement using get_unaligned.