Skip to contentSkip to conversation

fix libbpf hashmap with size_t shorter than long long

PATCH In review Jakub Bogusz 2 participants 6 messages Jun 21, 2020 – Jun 27, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fix libbpf hashmap with size_t shorter than long long

Hello,

I noticed that _bpftool crashes when building kernel tools (5.7.x) for 32-bit targets because in libbpf hashmap implementation hash_bits() function returning numbers exceeding hashmap buckets capacity.

Attached patch fixes this problem.

Regards,

Signature · 1 line
Jakub Bogusz    http://qboosh.pl/

Ensure that hash_bits returns value fits in given bits (for bits > 0):
multiplier is long long (which is the same or wider than size_t), so shift bits must be based on long long size, not __WORDSIZE.

tools/lib/bpf/hashmap.h renamed from tools/lib/bpf/hashmap.h.orig +1 -6
@@ -10,17 +10,12 @@
 
 #include <stdbool.h>
 #include <stddef.h>
-#ifdef __GLIBC__
-#include <bits/wordsize.h>
-#else
-#include <bits/reg.h>
-#endif
 #include "libbpf_internal.h"
 
 static inline size_t hash_bits(size_t h, int bits)
 {
 	/* shuffle bits and return requested number of upper bits */
-	return (h * 11400714819323198485llu) >> (__WORDSIZE - bits);
+	return (h * 11400714819323198485llu) >> (__SIZEOF_LONG_LONG__ * 8 - bits);
 }
 
 typedef size_t (*hashmap_hash_fn)(const void *key, void *ctx);
#1 lore raw
Quoted 9 lines · Jakub Bogusz

Hello,

I noticed that _bpftool crashes when building kernel tools (5.7.x) for 32-bit targets because in libbpf hashmap implementation hash_bits() function returning numbers exceeding hashmap buckets capacity.

Attached patch fixes this problem.

Thanks! But this was already fixed by Arnaldo Carvalho de Melo <[email protected]> in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available").

Quoted 5 lines

Regards,

--
Jakub Bogusz http://qboosh.pl/

#2 lore raw
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Jakub Bogusz

Hello,

I noticed that _bpftool crashes when building kernel tools (5.7.x) for 32-bit targets because in libbpf hashmap implementation hash_bits() function returning numbers exceeding hashmap buckets capacity.

Quoted 5 lines
Quoted 2 lines

Attached patch fixes this problem.

Thanks! But this was already fixed by Arnaldo Carvalho de Melo <[email protected]> in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available").

No, it's not:
This change worked around __WORDSIZE not always being available.

But the issue on (I)LP32 platforms is that 64-bit value is shifted by (32-bits) instead of (64-bits).

(__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm). I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and LP64 architectures.

Should I provide an updated patch to apply on top of acme change?

Regards,

Signature · 1 line
Jakub Bogusz    http://qboosh.pl/
#3 lore raw
Quoted 19 lines · Jakub Bogusz
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Jakub Bogusz

Hello,

I noticed that _bpftool crashes when building kernel tools (5.7.x) for 32-bit targets because in libbpf hashmap implementation hash_bits() function returning numbers exceeding hashmap buckets capacity.

Quoted 5 lines
Quoted 2 lines

Attached patch fixes this problem.

Thanks! But this was already fixed by Arnaldo Carvalho de Melo <[email protected]> in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available").

No, it's not:
This change worked around __WORDSIZE not always being available.

Quoted 7 lines

But the issue on (I)LP32 platforms is that 64-bit value is shifted by (32-bits) instead of (64-bits).

(__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm). I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and LP64 architectures.

Ah, I see. I actually mentioned __SIZEOF_ constants on the original fix patch. But I think in this case it has to use __SIZEOF_SIZE_T, which on 32-bit should be 4, right?

Quoted 1 line

Should I provide an updated patch to apply on top of acme change?

Yes, that would be good. But I think there is no need to penalize 32-bit arches with use of 64-bit long longs, and instead it's better to use #ifdef for 32-bit case vs 64-bit case. The multiplication constant will change, of course, should be 2654435769. I'd appreciate it if you can do the patch, thanks!

Quoted 6 lines

Regards,

--
Jakub Bogusz http://qboosh.pl/

#4 lore raw
Quoted 23 lines · Andrii Nakryiko
Quoted 22 lines · Jakub Bogusz
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Jakub Bogusz

Hello,

I noticed that _bpftool crashes when building kernel tools (5.7.x) for 32-bit targets because in libbpf hashmap implementation hash_bits() function returning numbers exceeding hashmap buckets capacity.

Quoted 5 lines
Quoted 2 lines

Attached patch fixes this problem.

Thanks! But this was already fixed by Arnaldo Carvalho de Melo <[email protected]> in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available").

No, it's not:
This change worked around __WORDSIZE not always being available.

But the issue on (I)LP32 platforms is that 64-bit value is shifted by (32-bits) instead of (64-bits).

Quoted 8 lines
Quoted 4 lines

(__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm). I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and LP64 architectures.

Ah, I see. I actually mentioned __SIZEOF_ constants on the original fix patch. But I think in this case it has to use __SIZEOF_SIZE_T, which on 32-bit should be 4, right?

After changing constant to 32-bit, yes (to be precise, it should use maximum of __SIZEOF_SIZE_T__ and __SIZEOF_LONG__ if constant is specified with UL suffix; there is no constant suffix available for size_t).

Quoted 7 lines
Quoted 1 line

Should I provide an updated patch to apply on top of acme change?

Yes, that would be good. But I think there is no need to penalize 32-bit arches with use of 64-bit long longs, and instead it's better to use #ifdef for 32-bit case vs 64-bit case. The multiplication constant will change, of course, should be 2654435769. I'd appreciate it if you can do the patch, thanks!

OK, so now the patch provides two variants:
- "long long" case for LP64 architectures
- "long" case for (I)LP32 architectures
(selected basing of __SIZEOF_ constants)
matter)

Regards,

Signature · 1 line
Jakub Bogusz    http://qboosh.pl/

Fix libbpf hashmap on (I)LP32 architectures

On ILP32, 64-bit result was shifted by value calculated for 32-bit long type and returned value was much outside hashmap capacity.
As advised by Andrii Nakryiko, this patch uses different hashing variant for architectures with size_t shorter than long long.

tools/lib/bpf/hashmap.h renamed from tools/lib/bpf/hashmap.h.orig +8 -4
@@ -11,14 +11,18 @@
 #include <stdbool.h>
 #include <stddef.h>
 #include <limits.h>
-#ifndef __WORDSIZE
-#define __WORDSIZE (__SIZEOF_LONG__ * 8)
-#endif
 
 static inline size_t hash_bits(size_t h, int bits)
 {
 	/* shuffle bits and return requested number of upper bits */
-	return (h * 11400714819323198485llu) >> (__WORDSIZE - bits);
+#if (__SIZEOF_SIZE_T__ == __SIZEOF_LONG_LONG__)
+	/* LP64 case */
+	return (h * 11400714819323198485llu) >> (__SIZEOF_LONG_LONG__ * 8 - bits);
+#elif (__SIZEOF_SIZE_T__ <= __SIZEOF_LONG__)
+	return (h * 2654435769lu) >> (__SIZEOF_LONG__ * 8 - bits);
+#else
+#	error "Unsupported size_t size"
+#endif
 }
 
 typedef size_t (*hashmap_hash_fn)(const void *key, void *ctx);
#5 lore raw
Quoted 46 lines · Jakub Bogusz
Quoted 23 lines · Andrii Nakryiko
Quoted 22 lines · Jakub Bogusz
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Jakub Bogusz

Hello,

I noticed that _bpftool crashes when building kernel tools (5.7.x) for 32-bit targets because in libbpf hashmap implementation hash_bits() function returning numbers exceeding hashmap buckets capacity.

Quoted 5 lines
Quoted 2 lines

Attached patch fixes this problem.

Thanks! But this was already fixed by Arnaldo Carvalho de Melo <[email protected]> in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available").

No, it's not:
This change worked around __WORDSIZE not always being available.

But the issue on (I)LP32 platforms is that 64-bit value is shifted by (32-bits) instead of (64-bits).

Quoted 8 lines
Quoted 4 lines

(__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm). I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and LP64 architectures.

Ah, I see. I actually mentioned __SIZEOF_ constants on the original fix patch. But I think in this case it has to use __SIZEOF_SIZE_T, which on 32-bit should be 4, right?

After changing constant to 32-bit, yes (to be precise, it should use maximum of __SIZEOF_SIZE_T__ and __SIZEOF_LONG__ if constant is specified with UL suffix; there is no constant suffix available for size_t).

Quoted 7 lines
Quoted 1 line

Should I provide an updated patch to apply on top of acme change?

Yes, that would be good. But I think there is no need to penalize 32-bit arches with use of 64-bit long longs, and instead it's better to use #ifdef for 32-bit case vs 64-bit case. The multiplication constant will change, of course, should be 2654435769. I'd appreciate it if you can do the patch, thanks!

Quoted 7 lines

OK, so now the patch provides two variants:
- "long long" case for LP64 architectures
- "long" case for (I)LP32 architectures
(selected basing of __SIZEOF_ constants)
matter)

Change looks good, thanks! But it would be more convenient for everyone if you submitted it not as an attachment, but as a proper patch email message.

Quoted 4 lines

Regards,

--
Jakub Bogusz http://qboosh.pl/