thread->msg_cache is not safe when multiple threads send messages to same spdk_thread
sunshihao at huawei.com
hello,maintainer:
when i use spdk api spdk_thread_send_msg to send io read/write msg,i found a bug:
multiple threads send msg to one spdk_thread,they may get msg form local_thread->msg_cache,there is not a lock or cas to ensure the safety of the linked list。
msg = NULL;
if (local_thread != NULL) {
if (local_thread->msg_cache_count > 0) {
msg = SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */
assert(msg != NULL);
SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
local_thread->msg_cache_count--;
}
}i don't konw whether is my wrong way to use spdk_thread_send_msg or the list is not safe,so please give me some advice。 Thank you!
Harris
Hi,
local_thread gets its value from _get_thread(). _get_thread() returns the thread-local spdk_thread structure for the calling thread. So if multiple threads call spdk_thread_send_msg() in parallel, they will each use their own thread-local spdk_thread structure and no lock or synchronization is needed.
-Jim
On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com> wrote:
hello,maintainer:
when i use spdk api spdk_thread_send_msg to send io read/write msg,i found a bug:
multiple threads send msg to one spdk_thread,they may get msg form local_thread->msg_cache,there is not a lock or cas to
ensure the safety of the linked list。
msg = NULL;
if (local_thread != NULL) {
if (local_thread->msg_cache_count > 0) {
msg = SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */
assert(msg != NULL);
SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
local_thread->msg_cache_count--;
}
}
i don't konw whether is my wrong way to use spdk_thread_send_msg or the list is not safe,so please give me some advice。
Thank you!
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.orgSunshihao (Euler OS)
Dear maintainer Jim, thank you for your reply.
After reading your email, I know I have used the wrong way to submiting io.
When I create a thread to submit io, I don't create spdk_thread structure for the thread, this structure only be created in spdk_reactors_start api for every lcore; Must I create spdk_thread structure for every submit thread?
Please give me some advice, thank you!
-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com]
发送时间: 2020年9月16日 3:32
收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread
Hi,
local_thread gets its value from _get_thread(). _get_thread() returns the thread-local spdk_thread structure for the calling thread. So if multiple threads call spdk_thread_send_msg() in parallel, they will each use their own thread-local spdk_thread structure and no lock or synchronization is needed.
-Jim
On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com> wrote:
hello,maintainer:
when i use spdk api spdk_thread_send_msg to send io read/write msg,i found a bug:
multiple threads send msg to one spdk_thread,they may get msg form local_thread->msg_cache,there is not a lock or cas to
ensure the safety of the linked list。
msg = NULL;
if (local_thread != NULL) {
if (local_thread->msg_cache_count > 0) {
msg = SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */
assert(msg != NULL);
SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
local_thread->msg_cache_count--;
}
}
i don't konw whether is my wrong way to use spdk_thread_send_msg or the list is not safe,so please give me some advice。
Thank you!
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
Harris
Hi,
Can you describe more about your application and when and where it submits IO? Applications using the SPDK application framework (i.e. spdk_app_start()) typically do not need to create their own spdk_threads - they submit IO from the spdk_threads created by the reactors.
Thanks,
Jim
On 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" <sunshihao(a)huawei.com> wrote:
Dear maintainer Jim, thank you for your reply.
After reading your email, I know I have used the wrong way to submiting io.
When I create a thread to submit io, I don't create spdk_thread structure for the thread, this structure only be created in spdk_reactors_start api for every lcore;
Must I create spdk_thread structure for every submit thread?
Please give me some advice, thank you!
-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com]
发送时间: 2020年9月16日 3:32
收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread
Hi,
local_thread gets its value from _get_thread(). _get_thread() returns the thread-local spdk_thread structure for the calling thread. So if multiple threads call spdk_thread_send_msg() in parallel, they will each use their own thread-local spdk_thread structure and no lock or synchronization is needed.
-Jim
On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com> wrote:
hello,maintainer:
when i use spdk api spdk_thread_send_msg to send io read/write msg,i found a bug:
multiple threads send msg to one spdk_thread,they may get msg form local_thread->msg_cache,there is not a lock or cas to
ensure the safety of the linked list。
msg = NULL;
if (local_thread != NULL) {
if (local_thread->msg_cache_count > 0) {
msg = SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */
assert(msg != NULL);
SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
local_thread->msg_cache_count--;
}
}
i don't konw whether is my wrong way to use spdk_thread_send_msg or the list is not safe,so please give me some advice。
Thank you!
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.orgSunshihao (Euler OS)
Dear maintainer Jim,
We have modified spdk for some requirement, our io workflow is :
Frist we do init wok:
Spdk Init : spdk_env_init-> spdk_reactors_init-> spdk_thread_create(g_masterThread, &tmp_cpumask) -> spdk_reactors_start(we have modify code to make the g_masterThread never reactor run, just return to do io submit job, so we execte every msg send to g_masterThread)
And then, construct io resource:
Io resource construct: get a spdk_thread point thread_1-> spdk_bdev_open_ext(open by name)-> spdk_bdev_get_io_channel(send msg to thread_n to get io channel)
After finish that, submit io to thread_1
Submit io: spdk_thread_send_msg(thread_n, LaunchIo, io); LaunchIo will call bdev submit io function given by spdk; Io is a struct have all args needed by bdev submit fuction.
We do init work one time and do io resource construct and submit work many times for io read/write.
I have read the bdevperf code, and at the frist time call spdk_thread_send_msg in spdk_app_start, the local thread is not a spdk_thread, we get msg form pool; after that, in the reactor_run to call spdk_thread_send_msg, it is spdk_thread, so msg get from thread->msg_cache.
In our submit work, we only get the spdk_thread thread_1 to send msg; I set the local thread to thread_1 and create two or more thread to do submit work; so, it cause the problem when get msg.
Now, I call spdk_set_thread(NULL) befor spdk_thread_send_msg(thread_n, LaunchIo, io); the problem has been solved.
Above is what i used spdk to submit io, maybe I used wrong way somewhere, please give me some advice.
Thank you!
-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com]
发送时间: 2020年9月17日 1:33
收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
主题: [SPDK] Re: 答复: Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread
Hi,
Can you describe more about your application and when and where it submits IO? Applications using the SPDK application framework (i.e. spdk_app_start()) typically do not need to create their own spdk_threads - they submit IO from the spdk_threads created by the reactors.
Thanks,
Jim
On 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" <sunshihao(a)huawei.com> wrote:
Dear maintainer Jim, thank you for your reply.
After reading your email, I know I have used the wrong way to submiting io.
When I create a thread to submit io, I don't create spdk_thread structure for the thread, this structure only be created in spdk_reactors_start api for every lcore;
Must I create spdk_thread structure for every submit thread?
Please give me some advice, thank you!
-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com]
发送时间: 2020年9月16日 3:32
收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread
Hi,
local_thread gets its value from _get_thread(). _get_thread() returns the thread-local spdk_thread structure for the calling thread. So if multiple threads call spdk_thread_send_msg() in parallel, they will each use their own thread-local spdk_thread structure and no lock or synchronization is needed.
-Jim
On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com> wrote:
hello,maintainer:
when i use spdk api spdk_thread_send_msg to send io read/write msg,i found a bug:
multiple threads send msg to one spdk_thread,they may get msg form local_thread->msg_cache,there is not a lock or cas to
ensure the safety of the linked list。
msg = NULL;
if (local_thread != NULL) {
if (local_thread->msg_cache_count > 0) {
msg = SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */
assert(msg != NULL);
SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
local_thread->msg_cache_count--;
}
}
i don't konw whether is my wrong way to use spdk_thread_send_msg or the list is not safe,so please give me some advice。
Thank you!
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
松本周平 <shuheimatsumoto at gmail.com>
Hi,
Basically all runtime operations should be executed via spdk_thread_poll()
During runtime, all one-time operations are requested by spdk_thread_send_msg() and executed via spdk_thread_poll(). All periodic operations are executed as pollers. Each poller is registered via spdk_poller_register(), and spdk_poller_register() is called in a message handler, and the message is requested by spdk_thread_send_msg().
You read that at the first time call spdk_thread_send_msg in spdk_app_start, the local thread is not a spdk thread.
This means the first message call is to execute the start function of the first SPDK thread.
We don't specify any start function or main function when creating a SPDK thread.
After the first SPDK thread processes the first sent message, it creates another SPDK thread or a poller.
Hence the first SPDK thread can be executed via message or poller in spdk_thread_poll().
If your g_master_thread does not use a reactor, you have to call spdk_set_thread() by yourself before calling spdk_thread_send_msg().
Please take a look at example/nvmf/nvmf.c
Thanks,
Shuhei
Quoted 137 lines · Sunshihao (Euler OS)
Dear maintainer Jim,
We have modified spdk for some requirement, our io workflow is :
Frist we do init wok:
Spdk Init : spdk_env_init-> spdk_reactors_init->spdk_thread_create(g_masterThread, &tmp_cpumask)-> spdk_reactors_start(we have modify code to make theg_masterThreadnever reactor run, just return to do io submit job, so we execte every msg send tog_masterThread)And then, construct io resource:
Io resource construct: get aspdk_threadpoint thread_1->spdk_bdev_open_ext(open by name)->spdk_bdev_get_io_channel(send msg to thread_n to get io channel)After finish that, submit io to
thread_1
Submit io:spdk_thread_send_msg(thread_n, LaunchIo, io); LaunchIo will call bdev submit io function given by spdk; Io is a struct have all args needed by bdev submit fuction.We do init work one time and do io resource construct and submit work many times for io read/write.
I have read the bdevperf code, and at the frist time call
spdk_thread_send_msginspdk_app_start, the local thread is not aspdk_thread, we get msg form pool; after that, in thereactor_runto callspdk_thread_send_msg, it isspdk_thread, so msg get fromthread->msg_cache.In our submit work, we only get the
spdk_threadthread_1to send msg; I set the local thread tothread_1and create two or more thread to do submit work; so, it cause the problem when get msg.Now, I call
spdk_set_thread(NULL)beforspdk_thread_send_msg(thread_n, LaunchIo, io); the problem has been solved.Above is what i used spdk to submit io, maybe I used wrong way somewhere, please give me some advice.
Thank you!
-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com]
发送时间: 2020年9月17日 1:33
收件人: Storage Performance Development Kit <spdk(a)lists.01.org> 主题: [SPDK] Re: 答复: Re:thread->msg_cacheis not safe when multiple threads send messages to samespdk_threadHi,
Can you describe more about your application and when and where it submits IO? Applications using the SPDK application framework (i.e.
spdk_app_start()) typically do not need to create their ownspdk_threads- they submit IO from thespdk_threadscreated by the reactors.Thanks,
Jim
On 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" <sunshihao(a)huawei.com> wrote:
Dear maintainer Jim, thank you for your reply.
After reading your email, I know I have used the wrong way to
submiting io.When I create a thread to submit io, I don't create
spdk_thread
structure for the thread, this structure only be created inspdk_reactors_startapi for every lcore;
Must I createspdk_threadstructure for every submit thread?Please give me some advice, thank you!
-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com] 发送时间: 2020年9月16日 3:32
收件人: Storage Performance Development Kit <spdk(a)lists.01.org> 主题: [SPDK] Re:thread->msg_cacheis not safe when multiple threads
send messages to samespdk_threadHi,
local_threadgets its value from_get_thread()._get_thread()returns
the thread-localspdk_threadstructure for the calling thread. So if multiple threads callspdk_thread_send_msg()in parallel, they will each use their own thread-localspdk_threadstructure and no lock or synchronization is needed.-Jim On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com>wrote:
hello,maintainer:
when i use spdk apispdk_thread_send_msgto send io read/write
msg,i found a bug:
multiple threads send msg to one spdk_thread,they may get msg form
local_thread->msg_cache,there is not a lock or cas toensure the safety of the linked list。 msg = NULL; if (local_thread != NULL) { if (local_thread->msg_cache_count > 0) { msg = SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */ assert(msg != NULL);SLIST_REMOVE_HEAD(&local_thread->msg_cache, link); local_thread->msg_cache_count--; } } i don't konw whether is my wrong way to use spdk_thread_send_msgor the list is not safe,so please give me some advice。
Thank you!
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org