Skip to contentSkip to conversation

thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

SPDK sunshihao at huawei.com 2 participants 6 messages Sep 15, 2020 – Sep 17, 2020

lore
Share & tools
lore

sunshihao at huawei.com

#0 lore raw

hello,maintainer:
when i use spdk api spdk_thread_send_msg to send io read/write msg,i found a bug:
multiple threads send msg to one spdk_thread,they may get msg form local_thread->msg_cache,there is not a lock or cas to ensure the safety of the linked list。

         msg = NULL;

	if (local_thread != NULL) {
		if (local_thread->msg_cache_count > 0) {
				msg = SLIST_FIRST(&local_thread->msg_cache);    /* here maybe not thread safe */
				assert(msg != NULL);
				SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
				local_thread->msg_cache_count--;
		}
	}

i don't konw whether is my wrong way to use spdk_thread_send_msg or the list is not safe,so please give me some advice。 Thank you!

#1 lore raw

Hi,

local_thread gets its value from _get_thread(). _get_thread() returns the thread-local spdk_thread structure for the calling thread. So if multiple threads call spdk_thread_send_msg() in parallel, they will each use their own thread-local spdk_thread structure and no lock or synchronization is needed.

-Jim

On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com> wrote:

    hello,maintainer:
    when i use spdk api spdk_thread_send_msg to send io read/write msg,i found a bug:
    multiple threads send msg to one spdk_thread,they may get msg form local_thread->msg_cache,there is not a lock or cas to 
    ensure the safety of the linked list。 

             msg = NULL;

    	if (local_thread != NULL) {
    		if (local_thread->msg_cache_count > 0) {
    				msg = SLIST_FIRST(&local_thread->msg_cache);    /* here maybe not thread safe */
    				assert(msg != NULL);
    				SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
    				local_thread->msg_cache_count--;
    		}
    	}

    i don't konw whether is my wrong way to use spdk_thread_send_msg  or the list is not safe,so please give me some advice。
    Thank you!
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
#2 lore raw

Dear maintainer Jim, thank you for your reply.

After reading your email, I know I have used the wrong way to submiting io.

When I create a thread to submit io, I don't create spdk_thread structure for the thread, this structure only be created in spdk_reactors_start api for every lcore; Must I create spdk_thread structure for every submit thread?

Please give me some advice, thank you!

-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com]
发送时间: 2020年9月16日 3:32
收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

Hi,

local_thread gets its value from _get_thread(). _get_thread() returns the thread-local spdk_thread structure for the calling thread. So if multiple threads call spdk_thread_send_msg() in parallel, they will each use their own thread-local spdk_thread structure and no lock or synchronization is needed.

-Jim

On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com> wrote:

    hello,maintainer:
    when i use spdk api spdk_thread_send_msg to send io read/write msg,i found a bug:
    multiple threads send msg to one spdk_thread,they may get msg form local_thread->msg_cache,there is not a lock or cas to 
    ensure the safety of the linked list。 

             msg = NULL;

    	if (local_thread != NULL) {
    		if (local_thread->msg_cache_count > 0) {
    				msg = SLIST_FIRST(&local_thread->msg_cache);    /* here maybe not thread safe */
    				assert(msg != NULL);
    				SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
    				local_thread->msg_cache_count--;
    		}
    	}

    i don't konw whether is my wrong way to use spdk_thread_send_msg  or the list is not safe,so please give me some advice。
    Thank you!
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

#3 lore raw

Hi,

Can you describe more about your application and when and where it submits IO? Applications using the SPDK application framework (i.e. spdk_app_start()) typically do not need to create their own spdk_threads - they submit IO from the spdk_threads created by the reactors.

Thanks,

Jim

On 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" <sunshihao(a)huawei.com> wrote:

    Dear maintainer Jim, thank you for your reply.

    After reading your email, I know I have used the wrong way to submiting io.

    When I create a thread to submit io, I don't create spdk_thread structure for the thread, this structure only be created in spdk_reactors_start api for every lcore;
    Must I create spdk_thread structure for every submit thread?


    Please give me some advice, thank you!


    -----邮件原件-----
    发件人: Harris, James R [mailto:james.r.harris(a)intel.com] 
    发送时间: 2020年9月16日 3:32
    收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
    主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

    Hi,

    local_thread gets its value from _get_thread().  _get_thread() returns the thread-local spdk_thread structure for the calling thread. So if multiple threads call spdk_thread_send_msg() in parallel, they will each use their own thread-local spdk_thread structure and no lock or synchronization is needed.

    -Jim

    On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com> wrote:

        hello,maintainer:
        when i use spdk api spdk_thread_send_msg to send io read/write msg,i found a bug:
        multiple threads send msg to one spdk_thread,they may get msg form local_thread->msg_cache,there is not a lock or cas to 
        ensure the safety of the linked list。 

                 msg = NULL;

        	if (local_thread != NULL) {
        		if (local_thread->msg_cache_count > 0) {
        				msg = SLIST_FIRST(&local_thread->msg_cache);    /* here maybe not thread safe */
        				assert(msg != NULL);
        				SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
        				local_thread->msg_cache_count--;
        		}
        	}

        i don't konw whether is my wrong way to use spdk_thread_send_msg  or the list is not safe,so please give me some advice。
        Thank you!
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org

    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
#4 lore raw

Dear maintainer Jim,

We have modified spdk for some requirement, our io workflow is :

Frist we do init wok:
Spdk Init : spdk_env_init-> spdk_reactors_init-> spdk_thread_create(g_masterThread, &tmp_cpumask) -> spdk_reactors_start(we have modify code to make the g_masterThread never reactor run, just return to do io submit job, so we execte every msg send to g_masterThread)

And then, construct io resource:
Io resource construct: get a spdk_thread point thread_1-> spdk_bdev_open_ext(open by name)-> spdk_bdev_get_io_channel(send msg to thread_n to get io channel)

After finish that, submit io to thread_1
Submit io: spdk_thread_send_msg(thread_n, LaunchIo, io); LaunchIo will call bdev submit io function given by spdk; Io is a struct have all args needed by bdev submit fuction.

We do init work one time and do io resource construct and submit work many times for io read/write.

I have read the bdevperf code, and at the frist time call spdk_thread_send_msg in spdk_app_start, the local thread is not a spdk_thread, we get msg form pool; after that, in the reactor_run to call spdk_thread_send_msg, it is spdk_thread, so msg get from thread->msg_cache.

In our submit work, we only get the spdk_thread thread_1 to send msg; I set the local thread to thread_1 and create two or more thread to do submit work; so, it cause the problem when get msg.

Now, I call spdk_set_thread(NULL) befor spdk_thread_send_msg(thread_n, LaunchIo, io); the problem has been solved.

Above is what i used spdk to submit io, maybe I used wrong way somewhere, please give me some advice.

Thank you!

-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com]
发送时间: 2020年9月17日 1:33
收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
主题: [SPDK] Re: 答复: Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

Hi,

Can you describe more about your application and when and where it submits IO? Applications using the SPDK application framework (i.e. spdk_app_start()) typically do not need to create their own spdk_threads - they submit IO from the spdk_threads created by the reactors.

Thanks,

Jim

On 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" <sunshihao(a)huawei.com> wrote:

    Dear maintainer Jim, thank you for your reply.

    After reading your email, I know I have used the wrong way to submiting io.

    When I create a thread to submit io, I don't create spdk_thread structure for the thread, this structure only be created in spdk_reactors_start api for every lcore;
    Must I create spdk_thread structure for every submit thread?


    Please give me some advice, thank you!


    -----邮件原件-----
    发件人: Harris, James R [mailto:james.r.harris(a)intel.com] 
    发送时间: 2020年9月16日 3:32
    收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
    主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

    Hi,

    local_thread gets its value from _get_thread().  _get_thread() returns the thread-local spdk_thread structure for the calling thread. So if multiple threads call spdk_thread_send_msg() in parallel, they will each use their own thread-local spdk_thread structure and no lock or synchronization is needed.

    -Jim

    On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com> wrote:

        hello,maintainer:
        when i use spdk api spdk_thread_send_msg to send io read/write msg,i found a bug:
        multiple threads send msg to one spdk_thread,they may get msg form local_thread->msg_cache,there is not a lock or cas to 
        ensure the safety of the linked list。 

                 msg = NULL;

        	if (local_thread != NULL) {
        		if (local_thread->msg_cache_count > 0) {
        				msg = SLIST_FIRST(&local_thread->msg_cache);    /* here maybe not thread safe */
        				assert(msg != NULL);
        				SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
        				local_thread->msg_cache_count--;
        		}
        	}

        i don't konw whether is my wrong way to use spdk_thread_send_msg  or the list is not safe,so please give me some advice。
        Thank you!
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org

    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

#5 lore raw

Hi,

Basically all runtime operations should be executed via spdk_thread_poll()

During runtime, all one-time operations are requested by spdk_thread_send_msg() and executed via spdk_thread_poll(). All periodic operations are executed as pollers. Each poller is registered via spdk_poller_register(), and spdk_poller_register() is called in a message handler, and the message is requested by spdk_thread_send_msg().

You read that at the first time call spdk_thread_send_msg in spdk_app_start, the local thread is not a spdk thread.

This means the first message call is to execute the start function of the first SPDK thread.
We don't specify any start function or main function when creating a SPDK thread.
After the first SPDK thread processes the first sent message, it creates another SPDK thread or a poller.
Hence the first SPDK thread can be executed via message or poller in spdk_thread_poll().

If your g_master_thread does not use a reactor, you have to call spdk_set_thread() by yourself before calling spdk_thread_send_msg().

Please take a look at example/nvmf/nvmf.c

Thanks,
Shuhei

Quoted 137 lines · Sunshihao (Euler OS)

Dear maintainer Jim,

We have modified spdk for some requirement, our io workflow is :

Frist we do init wok:
Spdk Init : spdk_env_init-> spdk_reactors_init-> spdk_thread_create(g_masterThread, &tmp_cpumask) -> spdk_reactors_start(we have modify code to make the g_masterThread never reactor run, just return to do io submit job, so we execte every msg send to g_masterThread)

And then, construct io resource:
Io resource construct: get a spdk_thread point thread_1-> spdk_bdev_open_ext(open by name)-> spdk_bdev_get_io_channel(send msg to thread_n to get io channel)

After finish that, submit io to thread_1
Submit io: spdk_thread_send_msg(thread_n, LaunchIo, io); LaunchIo will call bdev submit io function given by spdk; Io is a struct have all args needed by bdev submit fuction.

We do init work one time and do io resource construct and submit work many times for io read/write.

I have read the bdevperf code, and at the frist time call spdk_thread_send_msg in spdk_app_start, the local thread is not a spdk_thread, we get msg form pool; after that, in the reactor_run to call spdk_thread_send_msg, it is spdk_thread, so msg get from thread->msg_cache.

In our submit work, we only get the spdk_thread thread_1 to send msg; I set the local thread to thread_1 and create two or more thread to do submit work; so, it cause the problem when get msg.

Now, I call spdk_set_thread(NULL) befor spdk_thread_send_msg(thread_n, LaunchIo, io); the problem has been solved.

Above is what i used spdk to submit io, maybe I used wrong way somewhere, please give me some advice.

Thank you!

-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com]
发送时间: 2020年9月17日 1:33
收件人: Storage Performance Development Kit <spdk(a)lists.01.org> 主题: [SPDK] Re: 答复: Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

Hi,

Can you describe more about your application and when and where it submits IO? Applications using the SPDK application framework (i.e. spdk_app_start()) typically do not need to create their own spdk_threads - they submit IO from the spdk_threads created by the reactors.

Thanks,

Jim

On 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" <sunshihao(a)huawei.com> wrote:

Dear maintainer Jim, thank you for your reply.

After reading your email, I know I have used the wrong way to
submiting io.

When I create a thread to submit io, I don't create spdk_thread
structure for the thread, this structure only be created in spdk_reactors_start api for every lcore;
Must I create spdk_thread structure for every submit thread?

Please give me some advice, thank you!

-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com] 发送时间: 2020年9月16日 3:32
收件人: Storage Performance Development Kit <spdk(a)lists.01.org> 主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads
send messages to same spdk_thread

Hi,

local_thread gets its value from _get_thread(). _get_thread() returns
the thread-local spdk_thread structure for the calling thread. So if multiple threads call spdk_thread_send_msg() in parallel, they will each use their own thread-local spdk_thread structure and no lock or synchronization is needed.

    -Jim

    On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com>

wrote:

hello,maintainer:
when i use spdk api spdk_thread_send_msg to send io read/write
msg,i found a bug:
multiple threads send msg to one spdk_thread,they may get msg form
local_thread->msg_cache,there is not a lock or cas to

        ensure the safety of the linked list。

                 msg = NULL;

                if (local_thread != NULL) {
                        if (local_thread->msg_cache_count > 0) {
                                        msg =
SLIST_FIRST(&local_thread->msg_cache);    /* here maybe not thread safe */
                                        assert(msg != NULL);
SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
                                        local_thread->msg_cache_count--;
                        }
                }

        i don't konw whether is my wrong way to use spdk_thread_send_msg

or the list is not safe,so please give me some advice。
Thank you!
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org