xen-blkfront: Don't send closing notification to backend in blkfront_closing()

PATCH Joe Jin 2 participants 7 messages Jul 8, 2011 – Jul 12, 2011

Subsystems (3)

Block layer Xen hypervisor interface Xen block

lore

Share & tools

lore

Joe Jin

author

2011-07-08 07:14 UTC #0 lore raw

When we do block attach detach test with below steps, umount hang and the guest unable to shutdown:

1. start guest with the latest kernel.
2. attach new disk by xm-attach in Dom0
3. mount new disk in guest
4. detach the disk by xm-detach in dom0
5. umount the partition/disk in guest, command hung. exactly at here, any IO request to the partition/disk will hang.

Checking the code we found when xm-detach command set backend state to Closing, will trigger blkback_changed() -> blkfront_closing() call. At the moment, the disk still opened by guest, so frontend will refuse the request, but in the blkfront_closing(), it send a notification to backend said that the frontend state switched to Closing, when backend got the event, it will disconnect from real device, at here any IO request will be stuck, even tried to release the disk by umount.

Per our test, below patch fix this issue.

Signed-off-by Joe Jin <[email protected]>

Signed-off-by Annie Li <[email protected]>

---
 xen-blkfront.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

drivers/block/xen-blkfront.c +1 -1

index b536a9c..f6d8ac2 100644
@@ -1088,7 +1088,7 @@ blkfront_closing(struct blkfront_info *info)
 	if (bdev->bd_openers) {
 		xenbus_dev_error(xbdev, -EBUSY,
 				 "Device in use; refusing to close");
-		xenbus_switch_state(xbdev, XenbusStateClosing);
+		xbdev->state = XenbusStateClosing;
 	} else {
 		xlvbd_release_gendisk(info);
 		xenbus_frontend_closed(xbdev);

Konrad Rzeszutek Wilk

2011-07-08 16:04 UTC #1 lore raw

Quoted 7 lines · Joe Jin

When we do block attach detach test with below steps, umount hang and the guest unable to shutdown:
1. start guest with the latest kernel.
2. attach new disk by xm-attach in Dom0
3. mount new disk in guest
4. detach the disk by xm-detach in dom0

I think you mean xm block-detach and xm-attach?

I tried with and without your patch and in both cases I get this in my guest:

sh-4.1# mount /dev/xvda /test

[  385.949749] EXT3-fs: barriers not enabled
[  385.960173] kjournald starting.  Commit interval 5 seconds
[  385.960418] EXT3-fs (xvda): using internal journal
[  385.960427] EXT3-fs (xvda): mounted filesystem with writeback data mode

sh-4.1# [ 411.176887] vbd vbd-51712: 16 Device in use; refusing to close

The commands on the other side (Dom0) were:

[root@tst009 ~]# xm block-list 6
Vdev BE handle state evt-ch ring-ref BE-path
51712 0 0 4 12 770 /local/domain/0/backend/vbd/6/51712
[root@tst009 ~]# xm block-detach 6 51712
Error: Device 51712 (vbd) could not be disconnected.
Usage: xm block-detach <Domain> <DevId> [-f|--force]

Destroy a domain's virtual block device.
[root@tst009 ~]# xm block-detach 6 51712 -f

Quoted 2 lines

5. umount the partition/disk in guest, command hung. exactly at here, any IO request to the partition/disk will hang.

I get that with the patch and without it:

sh-4.1#
sh-4.1# [ 519.814048] block xvda: device/vbd/51712 was hot-unplugged, 1 stale handles

sh-4.1# df -h

Filesystem            Size  Used Avail Use% Mounted on
none                  490M  120K  490M   1% /dev
none                  490M  131M  359M  27% /lib/modules/3.0.0-rc6-00052-g3edce4b-dirty
shm                    10M     0   10M   0% /dev/shm
var_tmp                10M     0   10M   0% /var/tmp
/dev/xvda              20G  173M   19G   1% /test

sh-4.1# umount /test

Any ideas?

Quoted 35 lines

Checking the code we found when xm-detach command set backend state to Closing, will trigger blkback_changed() -> blkfront_closing() call. At the moment, the disk still opened by guest, so frontend will refuse the request, but in the blkfront_closing(), it send a notification to backend said that the frontend state switched to Closing, when backend got the event, it will disconnect from real device, at here any IO request will be stuck, even tried to release the disk by umount.
Per our test, below patch fix this issue.
Signed-off-by Joe Jin <[email protected]>
Signed-off-by Annie Li <[email protected]>
---
 xen-blkfront.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/xen-blkfront.c +1 -2
index b536a9c..f6d8ac2 100644
@@ -1088,7 +1088,7 @@ blkfront_closing(struct blkfront_info *info)
 	if (bdev->bd_openers) {
 		xenbus_dev_error(xbdev, -EBUSY,
 				 "Device in use; refusing to close");
-		xenbus_switch_state(xbdev, XenbusStateClosing);
+		xbdev->state = XenbusStateClosing;
 	} else {
 		xlvbd_release_gendisk(info);
 		xenbus_frontend_closed(xbdev);
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Joe Jin

author

2011-07-09 00:26 UTC #2 lore raw

Konrad,

Thanks for the reply, see comments in lines.

Quoted 10 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Joe Jin
When we do block attach detach test with below steps, umount hang and the guest unable to shutdown:
1. start guest with the latest kernel.
2. attach new disk by xm-attach in Dom0
3. mount new disk in guest
4. detach the disk by xm-detach in dom0
I think you mean xm block-detach and xm-attach?

You are right and sorry for confusing.

Quoted 11 lines

I tried with and without your patch and in both cases I get this in my guest:
sh-4.1# mount /dev/xvda /test
[  385.949749] EXT3-fs: barriers not enabled
[  385.960173] kjournald starting.  Commit interval 5 seconds
[  385.960418] EXT3-fs (xvda): using internal journal
[  385.960427] EXT3-fs (xvda): mounted filesystem with writeback data mode
sh-4.1# [ 411.176887] vbd vbd-51712: 16 Device in use; refusing to close

Quoted 9 lines

The commands on the other side (Dom0) were:
[root@tst009 ~]# xm block-list 6
Vdev BE handle state evt-ch ring-ref BE-path
51712 0 0 4 12 770 /local/domain/0/backend/vbd/6/51712
[root@tst009 ~]# xm block-detach 6 51712
Error: Device 51712 (vbd) could not be disconnected.
Usage: xm block-detach <Domain> <DevId> [-f|--force]

The error caused by xm block-detach timeout to waiting the dev's state switch to Closed.

Quoted 3 lines

Destroy a domain's virtual block device.
[root@tst009 ~]# xm block-detach 6 51712 -f

With "--force", it always success but frontend did not disconnected if device opened by someone.

Quoted 16 lines

Quoted 3 lines
5. umount the partition/disk in guest, command hung. exactly at here, any IO request to the partition/disk will hang.
I get that with the patch and without it:
sh-4.1#
sh-4.1# [ 519.814048] block xvda: device/vbd/51712 was hot-unplugged, 1 stale handles
sh-4.1# df -h
Filesystem            Size  Used Avail Use% Mounted on
none                  490M  120K  490M   1% /dev
none                  490M  131M  359M  27% /lib/modules/3.0.0-rc6-00052-g3edce4b-dirty
shm                    10M     0   10M   0% /dev/shm
var_tmp                10M     0   10M   0% /var/tmp
/dev/xvda              20G  173M   19G   1% /test

Quoted 3 lines

sh-4.1# umount /test
Any ideas?

This caused by backend kthread stopped, any IO request to the real device will hang, that is the patch intend to resolving.

Thanks,
Joe

Quoted 30 lines

Quoted 30 lines
Checking the code we found when xm-detach command set backend state to Closing, will trigger blkback_changed() -> blkfront_closing() call. At the moment, the disk still opened by guest, so frontend will refuse the request, but in the blkfront_closing(), it send a notification to backend said that the frontend state switched to Closing, when backend got the event, it will disconnect from real device, at here any IO request will be stuck, even tried to release the disk by umount.
Per our test, below patch fix this issue.
Signed-off-by Joe Jin <[email protected]>
Signed-off-by Annie Li <[email protected]>
---
 xen-blkfront.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/xen-blkfront.c +1 -1
index b536a9c..f6d8ac2 100644
@@ -1088,7 +1088,7 @@ blkfront_closing(struct blkfront_info *info)
 	if (bdev->bd_openers) {
 		xenbus_dev_error(xbdev, -EBUSY,
 				 "Device in use; refusing to close");
-		xenbus_switch_state(xbdev, XenbusStateClosing);
+		xbdev->state = XenbusStateClosing;
 	} else {
 		xlvbd_release_gendisk(info);
 		xenbus_frontend_closed(xbdev);

Konrad Rzeszutek Wilk

2011-07-09 13:11 UTC #3 lore raw

Quoted 17 lines

Quoted 17 lines
Quoted 2 lines
5. umount the partition/disk in guest, command hung. exactly at here, any IO request to the partition/disk will hang.
I get that with the patch and without it:
sh-4.1#
sh-4.1# [ 519.814048] block xvda: device/vbd/51712 was hot-unplugged, 1 stale handles
sh-4.1# df -h
Filesystem            Size  Used Avail Use% Mounted on
none                  490M  120K  490M   1% /dev
none                  490M  131M  359M  27% /lib/modules/3.0.0-rc6-00052-g3edce4b-dirty
shm                    10M     0   10M   0% /dev/shm
var_tmp                10M     0   10M   0% /var/tmp
/dev/xvda              20G  173M   19G   1% /test
sh-4.1# umount /test

Quoted 4 lines

Quoted 1 line
Any ideas?
This caused by backend kthread stopped, any IO request to the real device will hang, that is the patch intend to resolving.

I get this hang with the patch (and without). Is there a corresponding patch to the backend? (The backend is the same exact version as DomU - 3.0-rc6 + #stable/for-jens).

Joe Jin

author

2011-07-11 07:53 UTC #4 lore raw

Quoted 17 lines · Konrad Rzeszutek Wilk

Quoted 17 lines
Quoted 17 lines
Quoted 2 lines
5. umount the partition/disk in guest, command hung. exactly at here, any IO request to the partition/disk will hang.
I get that with the patch and without it:
sh-4.1#
sh-4.1# [ 519.814048] block xvda: device/vbd/51712 was hot-unplugged, 1 stale handles
sh-4.1# df -h
Filesystem            Size  Used Avail Use% Mounted on
none                  490M  120K  490M   1% /dev
none                  490M  131M  359M  27% /lib/modules/3.0.0-rc6-00052-g3edce4b-dirty
shm                    10M     0   10M   0% /dev/shm
var_tmp                10M     0   10M   0% /var/tmp
/dev/xvda              20G  173M   19G   1% /test
sh-4.1# umount /test

Quoted 7 lines

Quoted 4 lines
Quoted 1 line
Any ideas?
This caused by backend kthread stopped, any IO request to the real device will hang, that is the patch intend to resolving.
I get this hang with the patch (and without). Is there a corresponding patch to the backend? (The backend is the same exact version as DomU - 3.0-rc6 + #stable/for-jens).

My guest using linux-3.0 the latest kernel, backend based 2.6.32 kernel. Without the patch, xm block-detach timeout and umount hang, with the patch xm block-detach timeout but umount works fine.

Thanks,
Joe

Joe Jin

author

2011-07-12 03:31 UTC #5 lore raw

In reply to Konrad Rzeszutek Wilk

Quoted 17 lines · Konrad Rzeszutek Wilk

Quoted 17 lines
Quoted 17 lines
Quoted 2 lines
5. umount the partition/disk in guest, command hung. exactly at here, any IO request to the partition/disk will hang.
I get that with the patch and without it:
sh-4.1#
sh-4.1# [ 519.814048] block xvda: device/vbd/51712 was hot-unplugged, 1 stale handles
sh-4.1# df -h
Filesystem            Size  Used Avail Use% Mounted on
none                  490M  120K  490M   1% /dev
none                  490M  131M  359M  27% /lib/modules/3.0.0-rc6-00052-g3edce4b-dirty
shm                    10M     0   10M   0% /dev/shm
var_tmp                10M     0   10M   0% /var/tmp
/dev/xvda              20G  173M   19G   1% /test
sh-4.1# umount /test

Quoted 7 lines

Quoted 4 lines
Quoted 1 line
Any ideas?
This caused by backend kthread stopped, any IO request to the real device will hang, that is the patch intend to resolving.
I get this hang with the patch (and without). Is there a corresponding patch to the backend? (The backend is the same exact version as DomU - 3.0-rc6 + #stable/for-jens).

I tried xen and linux the latest code,

# xm dmesg | grep "Xen version"
(XEN) Xen version 4.2-unstable ([email protected]) () Tue Jul 12 06:51:41 CST 2011

Dom0:

# uname -r
3.0.0-rc6-XYZxen

Guest

# uname -r
3.0.0-rc6-XYZxen

without the patch, after detach the device by xm block-detach timeout also umount hang in guest. with my patch, xm block-detach timeout in Dom0, umount in guest worked!

Thanks,
Joe

Konrad Rzeszutek Wilk

Reviewed-by

2011-07-12 15:04 UTC #6 lore raw

Quoted 2 lines

without the patch, after detach the device by xm block-detach timeout also umount hang in guest. with my patch, xm block-detach timeout in Dom0, umount in guest worked!

Ah, I failed to spot you did the unmount after the timeout. I tried to do the umount after forcing the block-detach. [and testing confirms your patch working properly]

Looks good to me - with the only exception that the description needs to have 'xm block-attach' and 'xm block-detach' - please change that and resubmit and attach

Reviewed-by Konrad Rzeszutek Wilk <[email protected]>

CC [email protected]

and make sure to have Jens in the 'To:' header.
Thanks!

Jens,

Are you OK picking the amended patch (shortly to be posted) for 3.0-rc7?