Content-Type: multipart/mixed; boundary="===============4579870173389617436==" MIME-Version: 1.0 From: lokesharo at gmail.com Subject: [SPDK] Query on scaling SPDK threads Date: Sun, 19 Sep 2021 08:26:34 +0000 Message-ID: <20210919082634.2824.3981@ml01.vlan13.01.org> List-ID: To: spdk@lists.01.org --===============4579870173389617436== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hello In the SPDK source code, the message pool is of size 262143 ( taken from _t= hread_lib_init() ) and the per thread cache size is SPDK_MSG_MEMPOOL_CACHE_= SIZE (1024). What this means is the first 255 threads created will have the= cache size as 1024 and after that the threads will be created but the cach= e will be NULL and the threads have to refer the global pool to get the msg= e object to send the messages. In my application, we create around 300 threads during init. So, 256th thre= ad onward, global pool is referenced for the msge object. = Now, when the program is run, some of the threads that got 1024 entries in = the cache while creation (i.e. threads between 0 - 255) remain unused. What= this means is that some of the entries from the global pool remain unused = always and the threads (from 256 - 300) are not able to get the free entrie= s from the global pool and this leads to spdk_thread_send_msg() failing for= the threads in range of 256 - 300. I have the following doubts here: 1. How do we scale the number of threads in the SPDK environment with the h= ardcoded value of cache size and pool size. 2. Why is the pool size set to 262143 and cache size to 1024. Is there some= logical explanation behind the numbers? 3. If the user changes the above two numbers, are there any issues with it,= be it performance or memory or any thing else? 4. Is there any maximum limit that the SPDK suggests for the scaling of thr= eads? If yes, what is the scaling model that SPDK suggests the user should = implement? Thanks Lokesh --===============4579870173389617436==--