spdk_zmalloc error

Hi,
I meet a weird problem. The error below only happens in one machine of my cluster, but in other nodes, the program run successfully without bug. I try to build a memory pool, which consists of 256 buffers, each is 32M.
Here is my code snippet. For a memory pool, it will execute many times.
----------------------------------------------
#define SPDK_PAGE_SIZE (4096)
#define SPDK_BUFFER_SIZE (32ull<<20)
buffer_ = (char*)spdk_zmalloc(SPDK_BUFFER_SIZE,

                                      SPDK_PAGE_SIZE, NULL, 
                                      SPDK_ENV_SOCKET_ID_ANY,
                                      SPDK_MALLOC_DMA);

if(buffer_ == nullptr){
fprintf(stderr, "%s:%d: SPDK allocate memory failed\n", __FILE__, __LINE__); }
---------------------------------------

I found that in some buffer allocations, the variable buffer_ after the spdk_zmalloc is still a nullptr. It makes the program throw error output.
I start up spdk enviroment with hugemem=16384, the total memory pool size 8GB.
SPDK vesion is v20.01

Could someone give me some comments? thanks.

If everything on all machines are the same (OS/SW versions, HW) then it seems like a reasonable step might be replacing the memory in the one system that it's failing in?

Thx
Paul

-----Original Message-----
From: cfrancisy(a)gmail.com [mailto:cfrancisy(a)gmail.com]
Sent: Tuesday, April 14, 2020 1:04 AM
To: spdk(a)lists.01.org
Subject: [SPDK] spdk_zmalloc error

Hi,
I meet a weird problem. The error below only happens in one machine of my cluster, but in other nodes, the program run successfully without bug. I try to build a memory pool, which consists of 256 buffers, each is 32M.
Here is my code snippet. For a memory pool, it will execute many times.
----------------------------------------------
#define SPDK_PAGE_SIZE (4096)
#define SPDK_BUFFER_SIZE (32ull<<20)
buffer_ = (char*)spdk_zmalloc(SPDK_BUFFER_SIZE,

                                      SPDK_PAGE_SIZE, NULL, 
                                      SPDK_ENV_SOCKET_ID_ANY,
                                      SPDK_MALLOC_DMA); if(buffer_ == nullptr){ fprintf(stderr, "%s:%d: SPDK allocate memory failed\n", __FILE__, __LINE__); }

---------------------------------------

I found that in some buffer allocations, the variable buffer_ after the spdk_zmalloc is still a nullptr. It makes the program throw error output.
I start up spdk enviroment with hugemem=16384, the total memory pool size 8GB.
SPDK vesion is v20.01

Could someone give me some comments? thanks.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

I'm not sure that any memory is failing. I think the problem more likely is that DPDK's memory is getting fragmented in such a way that it cannot satisfy the 32MB allocations.

A couple of questions:

1) are you doing a lot of other spdk malloc operations between these 32MB allocations?
2) have you considered creating a buffer pool instead of doing malloc/free each time?

Typically in SPDK we try to do #2 whenever possible. Then once the memory is allocated, you do not need to worry about future allocation failures.

-Jim

On 4/14/20, 7:35 AM, "Luse, Paul E" <paul.e.luse(a)intel.com> wrote:

If everything on all machines are the same (OS/SW versions, HW) then it seems like a reasonable step might be replacing the memory in the one system that it's failing in?

Thx
Paul

-----Original Message-----
From: cfrancisy(a)gmail.com [mailto:cfrancisy(a)gmail.com]
Sent: Tuesday, April 14, 2020 1:04 AM
To: spdk(a)lists.01.org
Subject: [SPDK] spdk_zmalloc error

Hi,
I meet a weird problem. The error below only happens in one machine of my cluster, but in other nodes, the program run successfully without bug. I try to build a memory pool, which consists of 256 buffers, each is 32M.
Here is my code snippet. For a memory pool, it will execute many times.
----------------------------------------------
#define SPDK_PAGE_SIZE (4096)
#define SPDK_BUFFER_SIZE (32ull<<20)
buffer_ = (char*)spdk_zmalloc(SPDK_BUFFER_SIZE,
SPDK_PAGE_SIZE, NULL,
SPDK_ENV_SOCKET_ID_ANY,
SPDK_MALLOC_DMA); if(buffer_ == nullptr){ fprintf(stderr, "%s:%d: SPDK allocate memory failed\n", __FILE__, __LINE__); }
---------------------------------------

I found that in some buffer allocations, the variable buffer_ after the spdk_zmalloc is still a nullptr. It makes the program throw error output.
I start up spdk enviroment with hugemem=16384, the total memory pool size 8GB.
SPDK vesion is v20.01

Could someone give me some comments? thanks.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org