Skip to contentSkip to conversation

fix stubdom memory corruption

PATCH In review Stefano Stabellini 3 participants 13 messages Apr 14, 2009 – Apr 15, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fix stubdom memory corruption

Hi all,
this patch fixes a memory corruption in blkfront that happens every time we pass a sector aligned buffer (instead of a page aligned buffer) to blkfront_aio.
To trigger the COW we have to write at least a byte to each page of the buffer, but we must be careful not to overwrite useful content.

extras/mini-os/blkfront.c +7 -5
diff -r dbc4014882d0 extras/mini-os/blkfront.c
@@ -317,19 +317,21 @@
     req->sector_number = aiocbp->aio_offset / dev->info.sector_size;
 
     for (j = 0; j < n; j++) {
+        req->seg[j].first_sect = 0;
+        req->seg[j].last_sect = PAGE_SIZE / dev->info.sector_size - 1;
+    }
+    req->seg[0].first_sect = ((uintptr_t)aiocbp->aio_buf & ~PAGE_MASK) / dev->info.sector_size;
+    req->seg[n-1].last_sect = (((uintptr_t)aiocbp->aio_buf + aiocbp->aio_nbytes - 1) & ~PAGE_MASK) / dev->info.sector_size;
+    for (j = 0; j < n; j++) {
 	uintptr_t data = start + j * PAGE_SIZE;
         if (!write) {
             /* Trigger CoW if needed */
-            *(char*)data = 0;
+            *(char*)(data + (req->seg[j].first_sect << 9)) = 0;
             barrier();
         }
 	aiocbp->gref[j] = req->seg[j].gref =
             gnttab_grant_access(dev->dom, virtual_to_mfn(data), write);
-	req->seg[j].first_sect = 0;
-	req->seg[j].last_sect = PAGE_SIZE / dev->info.sector_size - 1;
     }
-    req->seg[0].first_sect = ((uintptr_t)aiocbp->aio_buf & ~PAGE_MASK) / dev->info.sector_size;
-    req->seg[n-1].last_sect = (((uintptr_t)aiocbp->aio_buf + aiocbp->aio_nbytes - 1) & ~PAGE_MASK) / dev->info.sector_size;
 
     dev->ring.req_prod_pvt = i + 1;
#1 lore raw

This one and vl.c patch have been applied.
stubdom has been rebuilt and reinstalled.
No improvement.
Name                                        ID   Mem VCPUs      State   Time(s) Domain-0                                     0  7019     2     r-----    633.9 RHELhvm                                     1  1024     1     ------      0.0

root@ServerXen331:/etc/xen# netstat -a|grep 590
tcp        0      0 *:5901                  *:*                     LISTEN     

Boris

--- On Tue, 4/14/09, Stefano Stabellini <[email protected]> wrote:
From: Stefano Stabellini <[email protected]>
Subject: [Xen-devel] [PATCH] fix stubdom memory corruption
To: "xen-devel" <[email protected]>
Date: Tuesday, April 14, 2009, 5:27 AM

Hi all,
this patch fixes a memory corruption in blkfront that happens every time we pass a sector aligned buffer (instead of a page aligned buffer) to blkfront_aio.
To trigger the COW we have to write at least a byte to each page of the buffer, but we must be careful not to overwrite useful content.

extras/mini-os/blkfront.c +4 -0
diff -r dbc4014882d0 extras/mini-os/blkfront.c
@@ -317,19 +317,21 @@
     req->sector_number = aiocbp->aio_offset / dev->info.sector_size;
 
     for (j = 0; j < n; j++) {
+        req->seg[j].first_sect = 0;
+        req->seg[j].last_sect = PAGE_SIZE / dev->info.sector_size - 1;
+    }
+    req->seg[0].first_sect = ((uintptr_t)aiocbp->aio_buf &

~PAGE_MASK) / dev->info.sector_size;
+ req->seg[n-1].last_sect = (((uintptr_t)aiocbp->aio_buf +
aiocbp->aio_nbytes - 1) & ~PAGE_MASK) / dev->info.sector_size;
+ for (j = 0; j < n; j++) {

 	uintptr_t data = start + j * PAGE_SIZE;
         if (!write) {
             /* Trigger CoW if needed */
-            *(char*)data = 0;
+            *(char*)(data + (req->seg[j].first_sect << 9)) = 0;
             barrier();
         }
 	aiocbp->gref[j] = req->seg[j].gref =
             gnttab_grant_access(dev->dom, virtual_to_mfn(data), write);
-	req->seg[j].first_sect = 0;
-	req->seg[j].last_sect = PAGE_SIZE / dev->info.sector_size - 1;
     }

- req->seg[0].first_sect = ((uintptr_t)aiocbp->aio_buf &
~PAGE_MASK) / dev->info.sector_size;
- req->seg[n-1].last_sect = (((uintptr_t)aiocbp->aio_buf +
aiocbp->aio_nbytes - 1) & ~PAGE_MASK) / dev->info.sector_size;

     dev->ring.req_prod_pvt = i + 1;

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#2 lore raw
Quoted 3 lines · Boris Derzhavets

This one and vl.c patch have been applied.
stubdom has been rebuilt and reinstalled.
No improvement.

Are you sure you correctly build the new code and running fs-backend? I am only asking this because it is really easy to have this kind of problems with stubdoms.

If your build and test environments are correct, are there any errors in the logfiles?

#3 lore raw

After patching i did:-

# cd /usr/src/xen-unstable.hg/stubdom
# make
# make install

I haven't rebuilt tools.
Started fs-backend in separate session as usual and tried to start stub domain

Boris

--- On Tue, 4/14/09, Stefano Stabellini <[email protected]> wrote:
From: Stefano Stabellini <[email protected]>
Subject: Re: [Xen-devel] [PATCH] fix stubdom memory corruption
To: "[email protected]" <[email protected]>
Cc: "xen-devel" <[email protected]>
Date: Tuesday, April 14, 2009, 8:55 AM

Quoted 3 lines · Boris Derzhavets

This one and vl.c patch have been applied.
stubdom has been rebuilt and reinstalled.
No improvement.

Are you sure you correctly build the new code and running fs-backend? I am only asking this because it is really easy to have this kind of problems with stubdoms.

If your build and test environments are correct, are there any errors in the logfiles?

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#4 lore raw
root@ServerXen331:/etc/xen# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  7019     2     r-----    702.0
RHELhvm                                   7  1024     1     ------      0.0

************************
Three logs generated
************************

root@ServerXen331:/var/log/xen# cat qemu-dm-RHELhvm-dm.log
domid: 9
Warning: vlan 0 is not connected to host network
Watching /local/domain/0/device-model/9/logdirty/next-active
Watching /local/domain/0/device-model/9/command
char device redirected to /dev/pts/3
/usr/src/xen-unstable.hg/tools/ioemu-dir/hw/xen_blktap.c:628: Init blktap pipes
xs_read(): vncpasswd get error. /vm/3f8762a5-b806-dca4-6442-0dfeb7da0917/vncpasswd.
medium change watch on `hdc' (index: 0): /dev/loop0
root@ServerXen331:/var/log/xen# cat qemu-dm-RHELhvm-dm.log.1
domid: 8
Warning: vlan 0 is not connected to host network
Watching /local/domain/0/device-model/8/logdirty/next-active
Watching /local/domain/0/device-model/8/command
char device redirected to /dev/pts/3
/usr/src/xen-unstable.hg/tools/ioemu-dir/hw/xen_blktap.c:628: Init blktap pipes
xs_read(): vncpasswd get error. /vm/3f8762a5-b806-dca4-6442-0dfeb7da0917/vncpasswd.
medium change watch on `hdc' (index: 0): /dev/loop0
root@ServerXen331:/var/log/xen# cat  qemu-dm-RHELhvm.log
Using config file "/etc/xen/RHELhvm-dm".
Started domain RHELhvm-dm (id=8)
Using config file "/etc/xen/RHELhvm-dm".
Unexpected error: <type 'exceptions.OSError'>

Please report to [email protected]
Traceback (most recent call last):
  File "/usr/sbin/xm", line 7, in <module>
    main.main(sys.argv)
  File "usr/lib/python2.5/site-packages/xen/xm/main.py", line 2980, in main _, rc = _run_cmd(cmd, cmd_name, args)
  File "usr/lib/python2.5/site-packages/xen/xm/main.py", line 3004, in _run_cmd return True, cmd(args)
  File "<string>", line 1, in <lambda>
  File "usr/lib/python2.5/site-packages/xen/xm/main.py", line 1360, in xm_importcommand cmd.main([command] + args)
  File "usr/lib/python2.5/site-packages/xen/xm/create.py", line 1370, in main do_console(sxp.child_value(config, 'name', -1))
  File "usr/lib/python2.5/site-packages/xen/xm/create.py", line 1397, in do_console (p, rv) = os.waitpid(cpid, os.WNOHANG)
OSError: [Errno 10] No child processes

Boris

--- On Tue, 4/14/09, Boris Derzhavets <[email protected]> wrote:
From: Boris Derzhavets <[email protected]>
Subject: Re: [Xen-devel] [PATCH] fix stubdom memory corruption
To: "Stefano Stabellini" <[email protected]>
Cc: "xen-devel" <[email protected]>
Date: Tuesday, April 14, 2009, 9:01 AM

After patching i did:-

# cd /usr/src/xen-unstable.hg/stubdom
# make
# make install

I haven't rebuilt tools.
Started fs-backend in separate session as usual and tried to start stub domain

Boris

--- On Tue, 4/14/09, Stefano Stabellini <[email protected]> wrote:
From: Stefano Stabellini <[email protected]>
Subject: Re: [Xen-devel] [PATCH] fix stubdom memory corruption
To: "[email protected]" <[email protected]>
Cc: "xen-devel" <[email protected]>
Date: Tuesday, April 14, 2009, 8:55 AM

Quoted 3 lines · Boris Derzhavets

This one and vl.c patch have been applied.
stubdom has been rebuilt and reinstalled.

No improvement.

Are you sure you correctly build the new code and running fs-backend? I am only asking this because it is really easy to have this kind of problems with stubdoms.

If your build and test environments are correct, are there any errors in the logfiles?

      _______________________________________________

Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#5 lore raw
Quoted 53 lines · Boris Derzhavets
root@ServerXen331:/etc/xen# xm list
Name                                        ID   Mem VCPUs      State  
Time(s)
Domain-0                                   0  7019     2     r-----    702.0
RHELhvm                                   7  1024     1     ------      0.0

************************
Three logs generated
************************

root@ServerXen331:/var/log/xen# cat qemu-dm-RHELhvm-dm.log
domid: 9
Warning: vlan 0 is not connected to host network
Watching /local/domain/0/device-model/9/logdirty/next-active
Watching /local/domain/0/device-model/9/command
char device redirected to /dev/pts/3
/usr/src/xen-unstable.hg/tools/ioemu-dir/hw/xen_blktap.c:628: Init
blktap pipes
xs_read(): vncpasswd get error.
/vm/3f8762a5-b806-dca4-6442-0dfeb7da0917/vncpasswd.
medium change watch on `hdc' (index: 0): /dev/loop0
root@ServerXen331:/var/log/xen# cat qemu-dm-RHELhvm-dm.log.1
domid: 8
Warning: vlan 0 is not connected to host network
Watching /local/domain/0/device-model/8/logdirty/next-active
Watching /local/domain/0/device-model/8/command
char device redirected to /dev/pts/3
/usr/src/xen-unstable.hg/tools/ioemu-dir/hw/xen_blktap.c:628: Init
blktap pipes
xs_read(): vncpasswd get error.
/vm/3f8762a5-b806-dca4-6442-0dfeb7da0917/vncpasswd.
medium change watch on `hdc' (index: 0): /dev/loop0
root@ServerXen331:/var/log/xen# cat  qemu-dm-RHELhvm.log
Using config file "/etc/xen/RHELhvm-dm".
Started domain RHELhvm-dm (id=8)
Using config file "/etc/xen/RHELhvm-dm".
Unexpected error: <type 'exceptions.OSError'>

Please report to [email protected]
Traceback (most recent call last):
File "/usr/sbin/xm", line 7, in <module>
main.main(sys.argv)
File "usr/lib/python2.5/site-packages/xen/xm/main.py", line 2980, in main

    _, rc = _run_cmd(cmd, cmd_name, args)

File "usr/lib/python2.5/site-packages/xen/xm/main.py", line 3004, in
_run_cmd
return True, cmd(args)
File "<string>", line 1, in <lambda>
File "usr/lib/python2.5/site-packages/xen/xm/main.py", line 1360, in
xm_importcommand
cmd.main([command] + args)
File "usr/lib/python2.5/site-packages/xen/xm/create.py", line 1370, in
main

Quoted 5 lines
    do_console(sxp.child_value(config, 'name', -1))

File "usr/lib/python2.5/site-packages/xen/xm/create.py", line 1397, in
do_console
(p, rv) = os.waitpid(cpid, os.WNOHANG)
OSError: [Errno 10] No child processes

If this is the only error you get, it seems to be unrelated to stubdoms. Could you please try to start a stubdom without autoconnect (no -c)?

#6 lore raw
Quoted 1 line

if this is the only error you get, it seems to be unrelated to stubdoms.

Yes

Quoted 1 line

Could you please try to start a stubdom without autoconnect (no -c)?

Sorry , what means "-c" ?
After fs-backend start i run in different terminal session :-

xm create RHELhvm-stubdom
vncviewer localhost:0

Maybe i follow outdated instructions.
Files RHELhvm-stubdom, RHELhvm-dm are located under /etc/xen

[root@ServerXen xen]# cat RHELhvm-stubdom
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
memory =1024
name = "RHELhvm"
vif = [ 'type=ioemu, bridge=eth0' ]
device_model = "/usr/lib/xen/bin/stubdom-dm"
boot="d"
sdl=0
opengl=0
vnc=0
stdvga=0

[root@ServerXen xen]# cat RHELhvm-dm
kernel = "/usr/lib/xen/boot/ioemu-stubdom.gz"
vif = [ ' ', 'bridge=eth0']

# vfb = [ 'type=sdl' ]
vfb = [ 'type=vnc' ]
disk = ['phy:/dev/loop0,hdc:cdrom,r','phy:/dev/sdb8,hda,w']

Boris.

--- On Wed, 4/15/09, Stefano Stabellini <[email protected]> wrote:
From: Stefano Stabellini <[email protected]>
Subject: Re: [Xen-devel] [PATCH] fix stubdom memory corruption
To: "[email protected]" <[email protected]>
Cc: "xen-devel" <[email protected]>
Date: Wednesday, April 15, 2009, 9:37 AM

Quoted 4 lines · Boris Derzhavets
root@ServerXen331:/etc/xen# xm list
Name                                        ID   Mem VCPUs      State  
Time(s)
Domain-0                                   0  7019     2     r-----   

702.0

Quoted 1 line

RHELhvm 7 1024 1 ------

0.0

Quoted 33 lines

************************
Three logs generated
************************

root@ServerXen331:/var/log/xen# cat qemu-dm-RHELhvm-dm.log
domid: 9
Warning: vlan 0 is not connected to host network
Watching /local/domain/0/device-model/9/logdirty/next-active
Watching /local/domain/0/device-model/9/command
char device redirected to /dev/pts/3
/usr/src/xen-unstable.hg/tools/ioemu-dir/hw/xen_blktap.c:628: Init
blktap pipes
xs_read(): vncpasswd get error.
/vm/3f8762a5-b806-dca4-6442-0dfeb7da0917/vncpasswd.
medium change watch on `hdc' (index: 0): /dev/loop0
root@ServerXen331:/var/log/xen# cat qemu-dm-RHELhvm-dm.log.1
domid: 8
Warning: vlan 0 is not connected to host network
Watching /local/domain/0/device-model/8/logdirty/next-active
Watching /local/domain/0/device-model/8/command
char device redirected to /dev/pts/3
/usr/src/xen-unstable.hg/tools/ioemu-dir/hw/xen_blktap.c:628: Init
blktap pipes
xs_read(): vncpasswd get error.
/vm/3f8762a5-b806-dca4-6442-0dfeb7da0917/vncpasswd.
medium change watch on `hdc' (index: 0): /dev/loop0
root@ServerXen331:/var/log/xen# cat  qemu-dm-RHELhvm.log
Using config file "/etc/xen/RHELhvm-dm".
Started domain RHELhvm-dm (id=8)
Using config file "/etc/xen/RHELhvm-dm".
Unexpected error: <type 'exceptions.OSError'>
Quoted 5 lines

Please report to [email protected] Traceback (most recent call last):
File "/usr/sbin/xm", line 7, in <module>
main.main(sys.argv)
File "usr/lib/python2.5/site-packages/xen/xm/main.py", line

2980, in main

Quoted 2 lines
    _, rc = _run_cmd(cmd, cmd_name, args)

File "usr/lib/python2.5/site-packages/xen/xm/main.py", line

3004, in

Quoted 4 lines

_run_cmd
return True, cmd(args)
File "<string>", line 1, in <lambda>
File "usr/lib/python2.5/site-packages/xen/xm/main.py", line

1360, in

Quoted 3 lines

xm_importcommand
cmd.main([command] + args)
File "usr/lib/python2.5/site-packages/xen/xm/create.py", line

1370, in

Quoted 3 lines

main

    do_console(sxp.child_value(config, 'name', -1))

File "usr/lib/python2.5/site-packages/xen/xm/create.py", line

1397, in

Quoted 3 lines

do_console
(p, rv) = os.waitpid(cpid, os.WNOHANG)
OSError: [Errno 10] No child processes

If this is the only error you get, it seems to be unrelated to stubdoms. Could you please try to start a stubdom without autoconnect (no -c)?

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#7 lore raw
In reply to Stefano Stabellini

Xen unstable 3.4-rc2-pre had been installed earlier

Boris.

--- On Tue, 4/14/09, Stefano Stabellini <[email protected]> wrote:
From: Stefano Stabellini <[email protected]>
Subject: Re: [Xen-devel] [PATCH] fix stubdom memory corruption
To: "[email protected]" <[email protected]>
Cc: "xen-devel" <[email protected]>
Date: Tuesday, April 14, 2009, 8:55 AM

Quoted 3 lines · Boris Derzhavets

This one and vl.c patch have been applied.
stubdom has been rebuilt and reinstalled.
No improvement.

Are you sure you correctly build the new code and running fs-backend? I am only asking this because it is really easy to have this kind of problems with stubdoms.

If your build and test environments are correct, are there any errors in the logfiles?

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#8 lore raw
Quoted 2 lines · Boris Derzhavets

Xen unstable 3.4-rc2-pre had been installed earlier

I think it may be worth to do a clean build and try again, because I can confirm that with my previous patches stubdoms work for me on xen and qemu tip.

#9 lore raw

I believe you worked with 2.6.18.8-xen kernel in Dom0.

Thanks.
Boris.

--- On Tue, 4/14/09, Stefano Stabellini <[email protected]> wrote:
From: Stefano Stabellini <[email protected]>
Subject: Re: [Xen-devel] [PATCH] fix stubdom memory corruption
To: "[email protected]" <[email protected]>
Cc: "xen-devel" <[email protected]>
Date: Tuesday, April 14, 2009, 11:33 AM

Quoted 2 lines · Boris Derzhavets

Xen unstable 3.4-rc2-pre had been installed earlier

I think it may be worth to do a clean build and try again, because I can confirm that with my previous patches stubdoms work for me on xen and qemu tip.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#10 lore raw
Quoted 2 lines · Boris Derzhavets

I believe you worked with 2.6.18.8-xen kernel in Dom0.

Yes, I'll try with a different kernel.

#11 lore raw

I had rebuilt Xen 3.4-rc2-pre from scratch.

Building Xen after fresh clone xen-unstable.hg :-

# make xen
# make tools
# make uninstall
# make install-xen
# make install-tools
# cd stubdom
# make
# make install

then tried Suse's 2.6.27.5 and 2.6.30-rc1-tip  under Xen 3.4-rc2-pre. Same error in logs:- /usr/src/xen-unstable.hg/tools/ioemu-dir/hw/xen_blktap.c:628: Init blktap pipes I guess pvops kernel is just missing blktap support
I need to check "make menuconfig" for Suse's one.

fs-backend is absolutly irresponsive with pvops kernel and seems to work as usual with 2.6.27.5
Thanks.
Boris.

--- On Wed, 4/15/09, Stefano Stabellini <[email protected]> wrote:
From: Stefano Stabellini <[email protected]>
Subject: Re: [Xen-devel] [PATCH] fix stubdom memory corruption
To: "[email protected]" <[email protected]>
Cc: "xen-devel" <[email protected]>
Date: Wednesday, April 15, 2009, 5:14 AM

Quoted 2 lines · Boris Derzhavets

I believe you worked with 2.6.18.8-xen kernel in Dom0.

Yes, I'll try with a different kernel.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#12 lore raw
Quoted 13 lines · Boris Derzhavets

I had rebuilt Xen 3.4-rc2-pre from scratch.

Building Xen after fresh clone xen-unstable.hg :-

# make xen
# make tools
# make uninstall
# make install-xen
# make install-tools
# cd stubdom
# make
# make install
Quoted 5 lines

then tried Suse's 2.6.27.5 and 2.6.30-rc1-tip  under Xen 3.4-rc2-pre. Same error in logs:- /usr/src/xen-unstable.hg/tools/ioemu-dir/hw/xen_blktap.c:628: Init blktap pipes I guess pvops kernel is just missing blktap support

Yep, pv_ops dom0 kernel is still missing blktap support.. Citrix blktap2 will be merged later to pv_ops dom0 kernel, iirc.

Quoted 3 lines

fs-backend is absolutly irresponsive with pvops kernel and seems to work as usual with 2.6.27.5

I've been using phy: LVM-volumes successfully with pv_ops dom0 kernel.

-- Pasi