rte_memcpy() moves data incorrectly on Ubuntu 18.04 on Intel Skylake
https://bugs.dpdk.org/show_bug.cgi?id=97
Bug ID: 97
Summary: rte_memcpy() moves data incorrectly on Ubuntu 18.04 on
Intel Skylake
Product: DPDK
Version: 18.08
Hardware: x86
OS: Linux
Status: CONFIRMED
Severity: critical
Priority: Normal
Component: core
Assignee: [email protected]
Reporter: [email protected]Target Milestone: ---
Reported by:
https://mails.dpdk.org/archives/dev/2018-September/111522.html
We've recently encountered a weird issue with Ubuntu 18.04 on the Skylake server. I can always reproduce this crash and I could narrowed it down. I guess it could be a GCC issue.
[1] How to reproduce
- ConnectX-4Lx/ConnectX-5 with mlx5 PMD in DPDK 18.02/18.05/18.08
- Ubuntu 18.04 on Intel Skylake server
- gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0
- Testpmd crashes when it starts to forward traffic. Easy to reproduce.
- Only happens on the Skylake server.
[2] Failure point
The attached patch gives an insight of why it crashes. The following is the result of the patch and the GDB commands.
In summary, rte_memcpy() doesn't work as expected. In __mempool_generic_put(), there's rte_memcpy() to move the array of objects to the lcore cache. If I run memcmp() right after rte_memcpy(dst, src, n), data in dst differs from data in src. And it looks like some of data got shifted by a few bytes as you can see below.
[GDB command]
$dst = 0x7ffff4e09ea8
$src = 0x7fffce3fb970
$n = 256
x/32gx 0x7ffff4e09ea8
x/32gx 0x7fffce3fb970
testpmd: /home/mlnxtest/dpdk/build/include/rte_mempool.h:1140:__mempool_generic_put: Assertion `0' failed.
Thread 4 "lcore-slave-1" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffce3ff700 (LWP 69913)]
(gdb) x/32gx 0x7ffff4e09ea8
0x7ffff4e09ea8: 0x00007fffaac38ec0 0x00007fffaac38500
0x7ffff4e09eb8: 0x00007fffaac37b40 0x00007fffaac37180
0x7ffff4e09ec8: 0x850000007fffaac3 0x7b4000007fffaac3
0x7ffff4e09ed8: 0x00007fffaac35440 0x00007fffaac34a80
0x7ffff4e09ee8: 0xaac3850000007fff 0xaac37b4000007fff
0x7ffff4e09ef8: 0x00007fffaac32d40 0x00007fffaac32380
0x7ffff4e09f08: 0x7fffaac385000000 0x7fffaac37b400000
0x7ffff4e09f18: 0x00007fffaac30640 0x00007fffaac2fc80
0x7ffff4e09f28: 0x00007fffaac2f2c0 0x00007fffaac2e900
0x7ffff4e09f38: 0x00007fffaac2df40 0x00007fffaac2d580
0x7ffff4e09f48: 0x00007fffaac2cbc0 0x00007fffaac2c200
0x7ffff4e09f58: 0x00007fffaac2b840 0x00007fffaac2ae80
0x7ffff4e09f68: 0x00007fffaac2a4c0 0x00007fffaac29b00
0x7ffff4e09f78: 0x00007fffaac29140 0x00007fffaac28780
0x7ffff4e09f88: 0x00007fffaac27dc0 0x00007fffaac27400
0x7ffff4e09f98: 0x00007fffaac26a40 0x00007fffaac26080
(gdb) x/32gx 0x7fffce3fb970
0x7fffce3fb970: 0x00007fffaac38ec0 0x00007fffaac38500
0x7fffce3fb980: 0x00007fffaac37b40 0x00007fffaac37180
0x7fffce3fb990: 0x00007fffaac367c0 0x00007fffaac35e00
0x7fffce3fb9a0: 0x00007fffaac35440 0x00007fffaac34a80
0x7fffce3fb9b0: 0x00007fffaac340c0 0x00007fffaac33700
0x7fffce3fb9c0: 0x00007fffaac32d40 0x00007fffaac32380
0x7fffce3fb9d0: 0x00007fffaac319c0 0x00007fffaac31000
0x7fffce3fb9e0: 0x00007fffaac30640 0x00007fffaac2fc80
0x7fffce3fb9f0: 0x00007fffaac2f2c0 0x00007fffaac2e900
0x7fffce3fba00: 0x00007fffaac2df40 0x00007fffaac2d580
0x7fffce3fba10: 0x00007fffaac2cbc0 0x00007fffaac2c200
0x7fffce3fba20: 0x00007fffaac2b840 0x00007fffaac2ae80
0x7fffce3fba30: 0x00007fffaac2a4c0 0x00007fffaac29b00
0x7fffce3fba40: 0x00007fffaac29140 0x00007fffaac28780
0x7fffce3fba50: 0x00007fffaac27dc0 0x00007fffaac27400
0x7fffce3fba60: 0x00007fffaac26a40 0x00007fffaac26080AFAIK, AVX512F support is disabled by default in DPDK as it is still experimental (CONFIG_RTE_ENABLE_AVX512=n). But with gcc optimization, AVX2 version of rte_memcpy() seems to be optimized with 512b instructions. If I disable it by adding EXTRA_CFLAGS="-mno-avx512f", then it works fine and doesn't
crash.
Do you have any idea regarding this issue or are you already aware of it?
Thanks,
Yongseok
$ git diff
config/common_base +3 -3
@@ -275,8 +275,8 @@ CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=8 # # Compile burst-oriented Mellanox ConnectX-4 & ConnectX-5 (MLX5) PMD # -CONFIG_RTE_LIBRTE_MLX5_PMD=n -CONFIG_RTE_LIBRTE_MLX5_DEBUG=n +CONFIG_RTE_LIBRTE_MLX5_PMD=y +CONFIG_RTE_LIBRTE_MLX5_DEBUG=y CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS=n CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE=8
@@ -597,7 +597,7 @@ CONFIG_RTE_RING_USE_C11_MEM_MODEL=n # CONFIG_RTE_LIBRTE_MEMPOOL=y CONFIG_RTE_MEMPOOL_CACHE_MAX_SIZE=512 -CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG=n +CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG=y # # Compile Mempool drivers
(quoted diff) +0 -0
b/lib/librte_mempool/rte_mempool.h
index 8b1b7f7ed..9f48028d9 100644
lib/librte_mempool/rte_mempool.h +1 -0
@@ -39,6 +39,7 @@#include <errno.h> #include <inttypes.h> #include <sys/queue.h> +#include <assert.h> #include <rte_config.h> #include <rte_spinlock.h>
@@ -1123,6 +1124,22 @@ __mempool_generic_put(struct rte_mempool *mp, void *
const *obj_table,
/* Add elements back into the cache */
rte_memcpy(&cache_objs[0], obj_table, sizeof(void *) * n);+ if(memcmp(&cache_objs[0], obj_table, sizeof(void *) * n)) {
+ printf("[GDB command] \n"
+ "$dst = %p\n"
+ "$src = %p\n"
+ "$n = %ld\n"
+ "x/%ldgx %p\n"
+ "x/%ldgx %p\n",
+ (void *)&cache_objs[0],
+ (const void *)obj_table,
+ sizeof(void *) * n,
+ sizeof(void *) * n / 8, (void *)&cache_objs[0],
+ sizeof(void *) * n / 8, (const void *)obj_table
+ );
+ assert(0);
+ }
+
cache->len += n;
if (cache->len >= cache->flushthresh) {Signature · 2 lines
You are receiving this mail because: You are the assignee for the bug.
https://bugs.dpdk.org/show_bug.cgi?id=97
Thomas Monjalon ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|CONFIRMED |RESOLVED
Resolution|--- |FIXEDMehmet gelisin ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #59 from Thomas Monjalon ([email protected]) --- AVX512 is disabled in DPDK if an affected version of binutils is used. Bug was fixed in 17.11, 18.11 and upper.
--- Comment #60 from Thomas Monjalon ([email protected]) --- AVX512 is disabled in DPDK if an affected version of binutils is used. Bug was fixed in 17.11, 18.11 and upper.
--- Comment #61 from Mehmet gelisin ([email protected]) --- Description:
The vhost crypto library code contains a post message handler
(vhost_crypto_msg_post_handler) which calls vhost_crypto_create_sess() which in turn calls transform_cipher_param() depending on the operation type. It is transform_cipher_param() https://komiya-dental.com/ that handles the payload data. The
payload contains a cipher key length and a static VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH (64) byte key buffer. When
http://www.iu-bloomington.com/transform_cipher_param() handles the payload data it does not check to see if the buffer length doesn't exceed VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH. This missing check can cause
https://www.webb-dev.co.uk/
out of bound reads which could trigger a crash or a potential information leak. Also, the vhost crypto library code contains a post message handler (vhost_crypto_msg_post_handler) which calls
https://waytowhatsnext.com/vhost_crypto_create_sess() which in turn calls transform_chain_param() depending on the operation type. It is transform_chain_param() that handles the payload data. The payload contains a cipher key length and a static VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH (64) byte key buffer, it
http://www.acpirateradio.co.uk/
also contains a digest length and a static authentication key buffer (size: VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH(512)) and authentication key buffer length. None of these length values are validated. Which can lead to reading out of bound. http://www.logoarts.co.uk/
Description:
The vhost crypto library code contains a post message handler
(vhost_crypto_msg_post_handler) which calls vhost_crypto_create_sess() which in turn calls transform_cipher_param() depending on the operation
http://www.slipstone.co.uk/
type. It is transform_cipher_param() that handles the payload data. The payload contains a cipher key length and a static VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH (64) byte key buffer. When transform_cipher_param() handles the payload data it does not check to see if the buffer length doesn't exceed VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH. This missing check can cause out of bound reads which could trigger a crash or a potential
http://embermanchester.uk/
information leak. Also, the vhost crypto library code contains a post message handler (vhost_crypto_msg_post_handler) which calls vhost_crypto_create_sess() which in turn calls transform_chain_param() depending on the operation type. It is transform_chain_param() that
http://connstr.net/
handles the payload data. The payload contains a cipher key length and a static VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH (64) byte key buffer, it also contains a digest length and a static authentication key buffer (size: VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH(512)) and authentication key buffer length. None of these length values are validated. Which can lead to reading out of bound.
Description: http://joerg.li/
The vhost crypto library code contains a post message handler
(vhost_crypto_msg_post_handler) which calls vhost_crypto_create_sess() which in turn calls transform_cipher_param() depending on the operation type. It is transform_cipher_param() that handles the payload data. The payload contains a cipher key length and a static http://www.jopspeech.com/VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH (64) byte key buffer. When transform_cipher_param() handles the payload data it does not check to see if the buffer length doesn't exceed VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH. This missing check can cause out of bound reads which could trigger a crash or a potential
http://www.wearelondonmade.com/
information leak. Also, the vhost crypto library code contains a post message handler (vhost_crypto_msg_post_handler) which calls vhost_crypto_create_sess() which in turn calls transform_chain_param() depending on the operation type. It is transform_chain_param() that
http://www.compilatori.com/
handles the payload data. The payload contains a cipher key length and a static VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH (64) byte key buffer, it also contains a digest length and a static authentication key buffer (size: VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH(512)) and authentication
http://www-look-4.com/
key buffer length. None of these length values are validated. Which can lead to reading out of bound.
Signature · 2 lines
You are receiving this mail because: You are the assignee for the bug.