Fix: bitfield: left shift undefined behavior
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 34 lines · Mathieu Desnoyers
bitfield.huses the left shift operator with a left operand which may be negative. The C99 standard states that shifting a negative value is undefined.When building with -Wshift-negative-value, we get this gcc warning:
In file included from
/home/smarchi/src/babeltrace/include/babeltrace/ctfser-internal.h:44:0,
from/home/smarchi/src/babeltrace/ctfser/ctfser.c:42:/home/smarchi/src/babeltrace/include/babeltrace/ctfser-internal.h: In function ‘bt_ctfser_write_unsigned_int’:/home/smarchi/src/babeltrace/include/babeltrace/bitfield-internal.h:116:24: error: left shift of negative value [-Werror=shift-negative-value]mask = ~((~(type) 0) << (__start % ts)); \ ^
/home/smarchi/src/babeltrace/include/babeltrace/bitfield-internal.h:222:2: note: in expansion of macro ‘_bt_bitfield_write_le’
_bt_bitfield_write_le(ptr, type, _start, _length, _v)
^~~~~~~~~~~~~~~~~~~~~/home/smarchi/src/babeltrace/include/babeltrace/ctfser-internal.h:418:3: note: in expansion of macro ‘bt_bitfield_write_le’
bt_bitfield_write_le(mmap_align_addr(ctfser->base_mma) +
^~~~~~~~~~~~~~~~~~~~This boils down to the fact that the expression ~((uint8_t)0) has type "signed int", which is used as an operand of the left shift. This is due to the integer promotion rules of C99 (6.3.3.1):
If an int can represent all values of the original type, the value is converted to an int; otherwise, it is converted to an unsigned int. These are called the integer promotions. All other types are unchanged by the integer promotions.
We also need to cast the result explicitly into the left hand side type to deal with:
warning: large integer implicitly truncated to unsigned type [-Woverflow]
Signed-off-by Mathieu Desnoyers <[email protected]>
Quoted 8 lines
---
Changes since v1:
- Generate compile-time error if the type argument passed to_bt_unsigned_cast()is larger thansizeof(uint64_t), this allows removing_bt_check_max_64bit,
- Introduce_br_fill_mask, which replaces_bt_bitwise_not,
- Clarify_bt_unsigned_castcomment,
- Expand explanation of the issue within the patch commit message.
I didn't spot any mistake by inspecting the code, but I get some errors running tests/lib/test_bitfield, so I guess there are some :). Does it pass on your side?
Quoted 34 lines
--- include/babeltrace/bitfield-internal.h | 77 ++++++++++++++++------------------ 1 file changed, 37 insertions(+), 40 deletions(-)include/babeltrace/bitfield-internal.h +10
-10@@ -55,21 +55,26 @@#define _bt_is_signed_type (type) ((type) -1 < (type) 0)/* -* NOTE: The cast to (uint64_t) below ensures that we're not casting a -* negative value, which is undefined in C. However, this limits the -* maximum type size of `type` and `v` to 64-bit. The -* _bt_check_max_64bit() is used to check that the users of this header -* do not use types with a size greater than 64-bit. +* Cast value `v` to an unsigned integer type of the size of type `type`. +* +* The unsigned cast ensures that we're not shifting a negative value, +* which is undefined in C. However, this limits the maximum type size +* of `type` to 64-bit. Generate a compile-time error if the size of +* `type` is larger than 64-bit. */ #define _bt_unsigned_cast (type, v) \ -({ \ - (sizeof(v) < sizeof(type)) ? \ - ((type) (v)) & ((type) (~(~(uint64_t) 0 << (sizeof(v) * CHAR_BIT)))) : \ - (type) (v); \ -}) + (sizeof(type) == sizeof(uint8_t) ? (uint8_t) (v) : \ + sizeof(type) == sizeof(uint16_t) ? (uint16_t) (v) : \ + sizeof(type) == sizeof(uint32_t) ? (uint32_t) (v) : \ + sizeof(type) == sizeof(uint64_t) ? (uint64_t) (v) : \
include/babeltrace/bitfield-internal.h line 77
Quoted 10 lines
include/babeltrace/bitfield-internal.h +7
-2+ sizeof(struct { int f:(sizeof(type) > sizeof(uint64_t) ? -1 : 1); })) -#define _bt_check_max_64bit (type) \ - char _max_64bit_assertion[sizeof(type) <= sizeof(uint64_t) ? 1 : -1] __attribute__((unused)) +/* +* _bt_fill_mask evaluates to an unsigned integer with the size of +* "type" with all bits set. It is meant to be used as a left operand to +* the shift-left operator to create bit masks. +*/ +#define _bt_fill_mask (type) _bt_unsigned_cast(type, ~(type) 0)
Thanks for the comment, it makes what this macro does very clear.
include/babeltrace/bitfield-internal.h do { \ · line 116
Quoted 9 lines
include/babeltrace/bitfield-internal.h +1
-1/* * bt_bitfield_write - write integer to a bitfield in native endianness @@ -108,15 +113,15 @@ do { \ \/* Trim v high bits */ \if (__length <sizeof (__v) *CHAR_BIT ) \ - __v &= ~((~(typeof (__v ))0 ) << __length); \ + __v &= (typeof (__v )) ~(_bt_fill_mask (typeof (__v)) << __length); \
Not a blocker, but this operation is done enough times that it might be worth giving it a name and make a macro for it. It could help make this big macro more readable, and we could test it on its own if needed.
/* Generate a mask of type type with the length least significant bits set. */
#define _bt_make_mask(type, length) \ ((type) ~(_bt_fill_mask(type) << length))
Simon
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 49 lines · ----- On May 8, 2019, at 11:49 AM, Simon Marchi [email protected] wrote:
Quoted 39 lines · Mathieu Desnoyers
bitfield.huses the left shift operator with a left operand which may be negative. The C99 standard states that shifting a negative value is undefined.When building with -Wshift-negative-value, we get this gcc warning:
In file included from
/home/smarchi/src/babeltrace/include/babeltrace/ctfser-internal.h:44:0,
from/home/smarchi/src/babeltrace/ctfser/ctfser.c:42:/home/smarchi/src/babeltrace/include/babeltrace/ctfser-internal.h: In function ‘bt_ctfser_write_unsigned_int’:/home/smarchi/src/babeltrace/include/babeltrace/bitfield-internal.h:116:24:
error: left shift of negative value [-Werror=shift-negative-value]mask = ~((~(type) 0) << (__start % ts)); \ ^
/home/smarchi/src/babeltrace/include/babeltrace/bitfield-internal.h:222:2: note:
in expansion of macro ‘_bt_bitfield_write_le’
_bt_bitfield_write_le(ptr, type, _start, _length, _v)^~~~~~~~~~~~~~~~~~~~~/home/smarchi/src/babeltrace/include/babeltrace/ctfser-internal.h:418:3: note:
in expansion of macro ‘bt_bitfield_write_le’
bt_bitfield_write_le(mmap_align_addr(ctfser->base_mma) + ^~~~~~~~~~~~~~~~~~~~This boils down to the fact that the expression ~((uint8_t)0) has type "signed int", which is used as an operand of the left shift. This is due to the integer promotion rules of C99 (6.3.3.1):
If an int can represent all values of the original type, the value is converted to an int; otherwise, it is converted to an unsigned int. These are called the integer promotions. All other types are unchanged by the integer promotions.
We also need to cast the result explicitly into the left hand side type to deal with:
warning: large integer implicitly truncated to unsigned type [-Woverflow]
Signed-off-by Mathieu Desnoyers <[email protected]>Quoted 8 lines
---
Changes since v1:
- Generate compile-time error if the type argument passed to_bt_unsigned_cast()is larger thansizeof(uint64_t), this allows removing_bt_check_max_64bit,
- Introduce_br_fill_mask, which replaces_bt_bitwise_not,
- Clarify_bt_unsigned_castcomment,
- Expand explanation of the issue within the patch commit message.
Quoted 3 lines
I didn't spot any mistake by inspecting the code, but I get some errors running tests/lib/test_bitfield, so I guess there are some :). Does it pass on your side?
What compiler do you use, and which compilation flags ? (it works here)
[...]
Quoted 14 lines
Review do { \ · line 116
Quoted 9 lines
/*
*bt_bitfield_write- write integer to a bitfield in native endianness(quoted diff) +1
-1@@ -108,15 +113,15 @@ do { \ \ /* Trim v high bits */ \ if (__length < sizeof(__v) * CHAR_BIT) \ - __v &= ~((~(typeof(__v)) 0) << __length); \ + __v &= (typeof(__v)) ~(_bt_fill_mask(typeof(__v)) << __length); \Not a blocker, but this operation is done enough times that it might be worth giving it a name and make a macro for it. It could help make this big macro more readable, and we could test it on its own if needed.
Quoted 5 lines
/* Generate a mask of type
typewith thelengthleast significant bits set. */#define _bt_make_mask(type, length) \ ((type) ~(_bt_fill_mask(type) << length))
Good point! Will do. Missing () around "length" though.
Thanks,
Mathieu
Quoted 2 lines
Simon
Signature · 7 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Mathieu Desnoyers
What compiler do you use, and which compilation flags ? (it works here)
"gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0", which is the system compiler on Ubuntu 18.04.
I just built with
./configure 'CFLAGS=-g3 -O0 -fsanitize=address -Wall'
and got some failures, the first one being
not ok 8 - Writing and reading back 0xC07EBC7, signed
# Failed test (test_bitfield.c:run_test_signed() at line 224)
# Failed reading value written "signed char"-wise, with start=0 and length=29. Read FFFFFFFFFFFFFFC7
# 0xFFFFFFC7 0xFFFFFFEB 0x7 0xC 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0
The test passes without the patch. If you can't reproduce it easily, I can dig on my side.
Quoted 14 lines
Quoted 14 lines
Review do { \ · line 116
Quoted 9 lines
/*
*bt_bitfield_write- write integer to a bitfield in native endianness(quoted diff) +1
-1@@ -108,15 +113,15 @@ do { \ \ /* Trim v high bits */ \ if (__length < sizeof(__v) * CHAR_BIT) \ - __v &= ~((~(typeof(__v)) 0) << __length); \ + __v &= (typeof(__v)) ~(_bt_fill_mask(typeof(__v)) << __length); \Not a blocker, but this operation is done enough times that it might be worth giving it a name and make a macro for it. It could help make this big macro more readable, and we could test it on its own if needed.
Quoted 7 lines
Quoted 5 lines
/* Generate a mask of type
typewith thelengthleast significant bits set. */#define _bt_make_mask(type, length) \ ((type) ~(_bt_fill_mask(type) << length))Good point! Will do. Missing () around "length" though.
Right, and I should have mentioned I didn't actually test that macro, so handle with care!
Simon
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 25 lines · ----- On May 8, 2019, at 12:08 PM, Simon Marchi [email protected] wrote:
Quoted 2 lines · Mathieu Desnoyers
What compiler do you use, and which compilation flags ? (it works here)
"gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0", which is the system compiler on Ubuntu 18.04.
I just built with
./configure 'CFLAGS=-g3 -O0 -fsanitize=address -Wall'
and got some failures, the first one being
not ok 8 - Writing and reading back
0xC07EBC7, signed
# Failed test (test_bitfield.c:run_test_signed() at line 224)
# Failed reading value written "signed char"-wise, with start=0 and length=29. Read FFFFFFFFFFFFFFC7
#0xFFFFFFC70xFFFFFFEB0x70xC0x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x0
Quoted 2 lines
The test passes without the patch. If you can't reproduce it easily, I can dig on my side.
it fails here too... the executable test-bitfield was renamed to test_bitfield and I had an old tree with the prior executable. Will investigate.
Thanks,
Mathieu
Quoted 27 lines
Quoted 15 lines
Quoted 15 lines
Review do { \ · line 116
Quoted 10 lines
/*
*bt_bitfield_write- write integer to a bitfield in native endianness(quoted diff) +1
-1@@ -108,15 +113,15 @@ do { \ \ /* Trim v high bits */ \ if (__length < sizeof(__v) * CHAR_BIT) \ - __v &= ~((~(typeof(__v)) 0) << __length); \ + __v &= (typeof(__v)) ~(_bt_fill_mask(typeof(__v)) << __length); \Not a blocker, but this operation is done enough times that it might be worth giving it a name and make a macro for it. It could help make this big macro more readable, and we could test it on its own if needed.
Quoted 7 lines
Quoted 5 lines
/* Generate a mask of type
typewith thelengthleast significant bits set. */#define _bt_make_mask(type, length) \ ((type) ~(_bt_fill_mask(type) << length))Good point! Will do. Missing () around "length" though.
Right, and I should have mentioned I didn't actually test that macro, so handle with care!
Simon
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com