the qemu-nbd process automatically exit with the commit 43347d56c 'livepatch: send a fake signal to all blocking tasks'

DISCUSSION [email protected] 3 participants 7 messages Apr 14, 2021 – Apr 15, 2021 Live patching

Share & tools

[email protected]

author

2021-04-14 03:55 UTC #0 lore raw

I found the qemu-nbd process(started with qemu-nbd -t -c /dev/nbd0 nbd.qcow2) will automatically exit when I patched for functions of the nbd with livepatch.

The nbd relative source:
static int nbd_start_device_ioctl(struct nbd_device *nbd, struct block_device *bdev)

{                                                                               
        struct nbd_config *config = nbd->config;                                
        int ret;                                                                
                                                                                
        ret = nbd_start_device(nbd);                                            
        if (ret)                                                                
                return ret;                                                     
                                                                                
        if (max_part)                                                           
                bdev->bd_invalidated = 1;                                       
        mutex_unlock(&nbd->config_lock);                                        
        ret = wait_event_interruptible(config->recv_wq,                         
                                         atomic_read(&config->recv_threads) == 0);
        if (ret)                                                                
                sock_shutdown(nbd);                                             
        flush_workqueue(nbd->recv_workq);                                       
                                                                                
        mutex_lock(&nbd->config_lock);                                          
        nbd_bdev_reset(bdev);                                                   
        /* user requested, ignore socket errors */                              
        if (test_bit(NBD_RT_DISCONNECT_REQUESTED, &config->runtime_flags))      
                ret = 0;                                                        
        if (test_bit(NBD_RT_TIMEDOUT, &config->runtime_flags))                  
                ret = -ETIMEDOUT;                                               
        return ret;                                                             
}

When the nbd waits for atomic_read(&config->recv_threads) == 0, the klp will send a fake signal to it then the qemu-nbd process exits. And the signal of sysfs to control this action was removed in the commit 10b3d52790e 'livepatch: Remove signal sysfs attribute'. Are there other ways to control this action? How?

Thanks very much.

Miroslav Benes

maintainer

2021-04-14 11:27 UTC #1 lore raw

Hi,

Quoted 24 lines · On Wed, 14 Apr 2021, [email protected] wrote:

I found the qemu-nbd process(started with qemu-nbd -t -c /dev/nbd0 nbd.qcow2) will automatically exit when I patched for functions of the nbd with livepatch.

The nbd relative source:
static int nbd_start_device_ioctl(struct nbd_device *nbd, struct block_device *bdev)

{                                                                               
        struct nbd_config *config = nbd->config;                                
        int ret;                                                                
                                                                                
        ret = nbd_start_device(nbd);                                            
        if (ret)                                                                
                return ret;                                                     
                                                                                
        if (max_part)                                                           
                bdev->bd_invalidated = 1;                                       
        mutex_unlock(&nbd->config_lock);                                        
        ret = wait_event_interruptible(config->recv_wq,                         
                                         atomic_read(&config->recv_threads) == 0);
        if (ret)                                                                
                sock_shutdown(nbd);                                             
        flush_workqueue(nbd->recv_workq);                                       
                                                                                
        mutex_lock(&nbd->config_lock);

Quoted 8 lines

        nbd_bdev_reset(bdev);                                                   
        /* user requested, ignore socket errors */                              
        if (test_bit(NBD_RT_DISCONNECT_REQUESTED, &config->runtime_flags))      
                ret = 0;                                                        
        if (test_bit(NBD_RT_TIMEDOUT, &config->runtime_flags))                  
                ret = -ETIMEDOUT;                                               
        return ret;                                                             
}

So my understanding is that ndb spawns a number (config->recv_threads) of workqueue jobs and then waits for them to finish. It waits interruptedly. Now, any signal would make wait_event_interruptible() to return -ERESTARTSYS. Livepatch fake signal is no exception there. The error is then propagated back to the userspace. Unless a user requested a disconnection or there is timeout set. How does the userspace then reacts to it? Is _interruptible there because the userspace sends a signal in case of NBD_RT_DISCONNECT_REQUESTED set? How does the userspace handles ordinary signals? This all sounds a bit strange, but I may be missing something easily.

Quoted 5 lines

When the nbd waits for atomic_read(&config->recv_threads) == 0, the klp will send a fake signal to it then the qemu-nbd process exits. And the signal of sysfs to control this action was removed in the commit 10b3d52790e 'livepatch: Remove signal sysfs attribute'. Are there other ways to control this action? How?

No, there is no way currently. We send a fake signal automatically.

Regards
Miroslav

[email protected]

author

2021-04-14 14:54 UTC #2 lore raw

Quoted 37 lines · On Wed, 14 Apr 2021 13:27:43 +0200 (CEST) Miroslav Benes <[email protected]> wrote:

Hi,
Quoted 26 lines · On Wed, 14 Apr 2021, [email protected] wrote:
I found the qemu-nbd process(started with qemu-nbd -t -c /dev/nbd0 nbd.qcow2) will automatically exit when I patched for functions of the nbd with livepatch.
The nbd relative source:
static int nbd_start_device_ioctl(struct nbd_device *nbd, struct block_device *bdev)
{ struct nbd_config *config =
nbd->config; int
ret;
ret =
nbd_start_device(nbd); if
(ret) return
ret;
if
(max_part) bdev->bd_invalidated =
1;
mutex_unlock(&nbd->config_lock); ret = wait_event_interruptible(config->recv_wq, atomic_read(&config->recv_threads) == 0); if
(ret)
sock_shutdown(nbd);
flush_workqueue(nbd->recv_workq); 
        mutex_lock(&nbd->config_lock);                                          
        nbd_bdev_reset(bdev);                                                   
        /* user requested, ignore socket errors
Quoted 6 lines
*/ if (test_bit(NBD_RT_DISCONNECT_REQUESTED,
&config->runtime_flags)) ret =
0; if (test_bit(NBD_RT_TIMEDOUT,
&config->runtime_flags)) ret =
-ETIMEDOUT; return
ret; }

Quoted 12 lines

So my understanding is that ndb spawns a number (config->recv_threads) of workqueue jobs and then waits for them to finish. It waits interruptedly. Now, any signal would make wait_event_interruptible() to return -ERESTARTSYS. Livepatch fake signal is no exception there. The error is then propagated back to the userspace. Unless a user requested a disconnection or there is timeout set. How does the userspace then reacts to it? Is _interruptible there because the userspace sends a signal in case of NBD_RT_DISCONNECT_REQUESTED set? How does the userspace handles ordinary signals? This all sounds a bit strange, but I may be missing something easily.

Sorry, now I also don't know how the qemu-nbd handles these signals. I need to see its source.

Thank you very much.

Quoted 10 lines

Quoted 5 lines
When the nbd waits for atomic_read(&config->recv_threads) == 0, the klp will send a fake signal to it then the qemu-nbd process exits. And the signal of sysfs to control this action was removed in the commit 10b3d52790e 'livepatch: Remove signal sysfs attribute'. Are there other ways to control this action? How?
No, there is no way currently. We send a fake signal automatically.
Regards
Miroslav

[email protected]

author

2021-04-14 15:21 UTC #3 lore raw

In reply to Miroslav Benes

Quoted 55 lines · On Wed, 14 Apr 2021 13:27:43 +0200 (CEST) Miroslav Benes <[email protected]> wrote:

Hi,
Quoted 26 lines · On Wed, 14 Apr 2021, [email protected] wrote:
I found the qemu-nbd process(started with qemu-nbd -t -c /dev/nbd0 nbd.qcow2) will automatically exit when I patched for functions of the nbd with livepatch.
The nbd relative source:
static int nbd_start_device_ioctl(struct nbd_device *nbd, struct block_device *bdev)
{ struct nbd_config *config =
nbd->config; int
ret;
ret =
nbd_start_device(nbd); if
(ret) return
ret;
if
(max_part) bdev->bd_invalidated =
1;
mutex_unlock(&nbd->config_lock); ret = wait_event_interruptible(config->recv_wq, atomic_read(&config->recv_threads) == 0); if
(ret)
sock_shutdown(nbd);
flush_workqueue(nbd->recv_workq); 
        mutex_lock(&nbd->config_lock);                                          
        nbd_bdev_reset(bdev);                                                   
        /* user requested, ignore socket errors
Quoted 6 lines
*/ if (test_bit(NBD_RT_DISCONNECT_REQUESTED,
&config->runtime_flags)) ret =
0; if (test_bit(NBD_RT_TIMEDOUT,
&config->runtime_flags)) ret =
-ETIMEDOUT; return
ret; }
So my understanding is that ndb spawns a number (config->recv_threads) of workqueue jobs and then waits for them to finish. It waits interruptedly. Now, any signal would make wait_event_interruptible() to return -ERESTARTSYS. Livepatch fake signal is no exception there. The error is then propagated back to the userspace. Unless a user requested a disconnection or there is timeout set. How does the userspace then reacts to it? Is _interruptible there because the userspace sends a signal in case of NBD_RT_DISCONNECT_REQUESTED set? How does the userspace handles ordinary signals? This all sounds a bit strange, but I may be missing something easily.
Quoted 5 lines
When the nbd waits for atomic_read(&config->recv_threads) == 0, the klp will send a fake signal to it then the qemu-nbd process exits. And the signal of sysfs to control this action was removed in the commit 10b3d52790e 'livepatch: Remove signal sysfs attribute'. Are there other ways to control this action? How?

Quoted 4 lines

No, there is no way currently. We send a fake signal automatically.
Regards
Miroslav

It occurs IO error of the nbd device when I use livepatch of the nbd, and I guess that any livepatch on other kernel source maybe cause the IO error. Well, now I decide to workaround for this problem by adding a livepatch for the klp to disable a automatic fake signal.

Regards.

Josef Bacik

2021-04-14 17:21 UTC #4 lore raw

Quoted 58 lines · On 4/14/21 11:21 AM, [email protected] wrote:

Quoted 55 lines · On Wed, 14 Apr 2021 13:27:43 +0200 (CEST) Miroslav Benes <[email protected]> wrote:
Hi,
Quoted 26 lines · On Wed, 14 Apr 2021, [email protected] wrote:
I found the qemu-nbd process(started with qemu-nbd -t -c /dev/nbd0 nbd.qcow2) will automatically exit when I patched for functions of the nbd with livepatch.
The nbd relative source:
static int nbd_start_device_ioctl(struct nbd_device *nbd, struct block_device *bdev)
{ struct nbd_config *config =
nbd->config; int
ret;
ret =
nbd_start_device(nbd); if
(ret) return
ret;
if
(max_part) bdev->bd_invalidated =
1;
mutex_unlock(&nbd->config_lock); ret = wait_event_interruptible(config->recv_wq, atomic_read(&config->recv_threads) == 0); if
(ret)
sock_shutdown(nbd);
flush_workqueue(nbd->recv_workq);
         mutex_lock(&nbd->config_lock);
         nbd_bdev_reset(bdev);
         /* user requested, ignore socket errors
Quoted 6 lines
*/ if (test_bit(NBD_RT_DISCONNECT_REQUESTED,
&config->runtime_flags)) ret =
0; if (test_bit(NBD_RT_TIMEDOUT,
&config->runtime_flags)) ret =
-ETIMEDOUT; return
ret; }
So my understanding is that ndb spawns a number (config->recv_threads) of workqueue jobs and then waits for them to finish. It waits interruptedly. Now, any signal would make wait_event_interruptible() to return -ERESTARTSYS. Livepatch fake signal is no exception there. The error is then propagated back to the userspace. Unless a user requested a disconnection or there is timeout set. How does the userspace then reacts to it? Is _interruptible there because the userspace sends a signal in case of NBD_RT_DISCONNECT_REQUESTED set? How does the userspace handles ordinary signals? This all sounds a bit strange, but I may be missing something easily.
Quoted 5 lines
When the nbd waits for atomic_read(&config->recv_threads) == 0, the klp will send a fake signal to it then the qemu-nbd process exits. And the signal of sysfs to control this action was removed in the commit 10b3d52790e 'livepatch: Remove signal sysfs attribute'. Are there other ways to control this action? How?

Quoted 9 lines

Quoted 4 lines
No, there is no way currently. We send a fake signal automatically.
Regards
Miroslav
It occurs IO error of the nbd device when I use livepatch of the nbd, and I guess that any livepatch on other kernel source maybe cause the IO error. Well, now I decide to workaround for this problem by adding a livepatch for the klp to disable a automatic fake signal.

Would wait_event_killable() fix this problem? I'm not sure any client implementations depend on being able to send other signals to the client process, so it should be safe from that standpoint. Not sure if the livepatch thing would still get an error at that point tho. Thanks,

Josef

[email protected]

author

2021-04-15 06:27 UTC #5 lore raw

Quoted 69 lines · On Wed, 14 Apr 2021 13:21:37 -0400 Josef Bacik <[email protected]> wrote:

Quoted 60 lines · On 4/14/21 11:21 AM, [email protected] wrote:
Quoted 57 lines · On Wed, 14 Apr 2021 13:27:43 +0200 (CEST) Miroslav Benes <[email protected]> wrote:
Hi,
Quoted 26 lines · On Wed, 14 Apr 2021, [email protected] wrote:
I found the qemu-nbd process(started with qemu-nbd -t -c /dev/nbd0 nbd.qcow2) will automatically exit when I patched for functions of the nbd with livepatch.
The nbd relative source:
static int nbd_start_device_ioctl(struct nbd_device *nbd, struct block_device *bdev)
{ struct nbd_config *config =
nbd->config; int
ret;
ret =
nbd_start_device(nbd); if
(ret) return
ret;
if
(max_part) bdev->bd_invalidated =
1;
mutex_unlock(&nbd->config_lock); ret = wait_event_interruptible(config->recv_wq, atomic_read(&config->recv_threads) == 0); if
(ret)
sock_shutdown(nbd);
flush_workqueue(nbd->recv_workq);
         mutex_lock(&nbd->config_lock);
         nbd_bdev_reset(bdev);
         /* user requested, ignore socket errors
Quoted 6 lines
*/ if (test_bit(NBD_RT_DISCONNECT_REQUESTED,
&config->runtime_flags)) ret =
0; if (test_bit(NBD_RT_TIMEDOUT,
&config->runtime_flags)) ret =
-ETIMEDOUT; return
ret; }
So my understanding is that ndb spawns a number (config->recv_threads) of workqueue jobs and then waits for them to finish. It waits interruptedly. Now, any signal would make wait_event_interruptible() to return -ERESTARTSYS. Livepatch fake signal is no exception there. The error is then propagated back to the userspace. Unless a user requested a disconnection or there is timeout set. How does the userspace then reacts to it? Is _interruptible there because the userspace sends a signal in case of NBD_RT_DISCONNECT_REQUESTED set? How does the userspace handles ordinary signals? This all sounds a bit strange, but I may be missing something easily.
Quoted 5 lines
When the nbd waits for atomic_read(&config->recv_threads) == 0, the klp will send a fake signal to it then the qemu-nbd process exits. And the signal of sysfs to control this action was removed in the commit 10b3d52790e 'livepatch: Remove signal sysfs attribute'. Are there other ways to control this action? How?
No, there is no way currently. We send a fake signal automatically.
Quoted 7 lines
Quoted 2 lines
Regards
Miroslav
It occurs IO error of the nbd device when I use livepatch of the nbd, and I guess that any livepatch on other kernel source maybe cause the IO error. Well, now I decide to workaround for this problem by adding a livepatch for the klp to disable a automatic fake signal.

Quoted 6 lines

Would wait_event_killable() fix this problem? I'm not sure any client implementations depend on being able to send other signals to the client process, so it should be safe from that standpoint. Not sure if the livepatch thing would still get an error at that point tho. Thanks,
Josef

Yes, I tested that wait_event_killable() can fix this problem.

Thanks.

Miroslav Benes

maintainer

2021-04-15 08:37 UTC #6 lore raw

In reply to Josef Bacik

Quoted 69 lines · Josef Bacik

Quoted 58 lines · On 4/14/21 11:21 AM, [email protected] wrote:
Quoted 55 lines · On Wed, 14 Apr 2021 13:27:43 +0200 (CEST) Miroslav Benes <[email protected]> wrote:
Hi,
Quoted 26 lines · On Wed, 14 Apr 2021, [email protected] wrote:
I found the qemu-nbd process(started with qemu-nbd -t -c /dev/nbd0 nbd.qcow2) will automatically exit when I patched for functions of the nbd with livepatch.
The nbd relative source:
static int nbd_start_device_ioctl(struct nbd_device *nbd, struct block_device *bdev)
{ struct nbd_config *config =
nbd->config; int
ret;
ret =
nbd_start_device(nbd); if
(ret) return
ret;
if
(max_part) bdev->bd_invalidated =
1;
mutex_unlock(&nbd->config_lock); ret = wait_event_interruptible(config->recv_wq, atomic_read(&config->recv_threads) == 0); if
(ret)
sock_shutdown(nbd);
flush_workqueue(nbd->recv_workq);
         mutex_lock(&nbd->config_lock);
         nbd_bdev_reset(bdev);
         /* user requested, ignore socket errors
Quoted 6 lines
*/ if (test_bit(NBD_RT_DISCONNECT_REQUESTED,
&config->runtime_flags)) ret =
0; if (test_bit(NBD_RT_TIMEDOUT,
&config->runtime_flags)) ret =
-ETIMEDOUT; return
ret; }
So my understanding is that ndb spawns a number (config->recv_threads) of workqueue jobs and then waits for them to finish. It waits interruptedly. Now, any signal would make wait_event_interruptible() to return -ERESTARTSYS. Livepatch fake signal is no exception there. The error is then propagated back to the userspace. Unless a user requested a disconnection or there is timeout set. How does the userspace then reacts to it? Is _interruptible there because the userspace sends a signal in case of NBD_RT_DISCONNECT_REQUESTED set? How does the userspace handles ordinary signals? This all sounds a bit strange, but I may be missing something easily.
Quoted 5 lines
When the nbd waits for atomic_read(&config->recv_threads) == 0, the klp will send a fake signal to it then the qemu-nbd process exits. And the signal of sysfs to control this action was removed in the commit 10b3d52790e 'livepatch: Remove signal sysfs attribute'. Are there other ways to control this action? How?
Quoted 9 lines
Quoted 4 lines
No, there is no way currently. We send a fake signal automatically.
Regards
Miroslav
It occurs IO error of the nbd device when I use livepatch of the nbd, and I guess that any livepatch on other kernel source maybe cause the IO error. Well, now I decide to workaround for this problem by adding a livepatch for the klp to disable a automatic fake signal.

Quoted 4 lines

Would wait_event_killable() fix this problem? I'm not sure any client implementations depend on being able to send other signals to the client process, so it should be safe from that standpoint. Not sure if the livepatch thing would still get an error at that point tho. Thanks,

wait_event_killable() means that you would sleep uninterruptedly (still reacting to fatal signals), so the fake signal from livepatch would not be sent at all. set_notify_signal() handles TASK_INTERRUPTIBLE tasks. No disruption for the userspace and it would fix this problem.

There is a catch on the livepatch side of things. If there is a live patch for nbd_start_device_ioctl(), the transition process would get stuck until the task leaves the function (all workqueue jobs are processed). I gather it is unlikely to be it indefinite, so we can live with that, I think.

Miroslav