Skip to contentSkip to conversation

Extend ioreq-server to support page write protection

PATCH Superseded Wei Ye 6 participants 46 messages Jul 28, 2014 – Aug 14, 2014

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Wei Ye

author
#0 lore raw

cover Extend ioreq-server to support page write protection

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to https://01.org/xen/blogs/srclarkx/2013/graphics-virtualization-
xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

Wei Ye (2):
x86: add p2m_ram_wp
ioreq-server: Support scatter page forwarding

 tools/libxc/xc_domain.c          |   32 ++++++
 tools/libxc/xenctrl.h            |   18 ++++
 xen/arch/x86/hvm/hvm.c           |  214 +++++++++++++++++++++++++++++++++++++-
 xen/arch/x86/mm/p2m-ept.c        |    1 +
 xen/include/asm-x86/hvm/domain.h |   14 +++
 xen/include/asm-x86/p2m.h        |    8 +-
 xen/include/public/hvm/hvm_op.h  |   12 +++
 7 files changed, 296 insertions(+), 3 deletions(-)
Signature · 1 line
1.7.9.5
#1 lore raw
Quoted 16 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology,
please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualization-
xengt) driver reside in Dom0 as a virtual graphics device model also need to trap
and emulate the guest's write operation to some specific memory pages, like the
memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write
operation. Page of this new p2m type is read only and for write, the request will
go to device model via ioreq-server.

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Jan

author
#2 lore raw
Quoted 25 lines

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Monday, July 28, 2014 4:25 PM
To: Ye, Wei
Cc: [email protected]; [email protected]; [email protected]; [email protected]; xen- [email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualization-
xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

Quoted 2 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d. An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 2 lines

Jan

#3 lore raw
Quoted 26 lines
Quoted 26 lines
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Monday, July 28, 2014 4:25 PM
To: Ye, Wei
Cc: [email protected]; [email protected]; [email protected]; [email protected]; xen- [email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualization-
xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

Quoted 7 lines
Quoted 2 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d.
An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

Jan

#4 lore raw
Quoted 37 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 26 lines
Quoted 26 lines
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Monday, July 28, 2014 4:25 PM
To: Ye, Wei
Cc: [email protected]; [email protected]; [email protected]; [email protected]; xen- [email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualization-
xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

Quoted 7 lines
Quoted 2 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d.
An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 8 lines

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Comments?

Thanks
Kevin

#5 lore raw
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 23:34, <[email protected]> wrote:

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Depends on whether you mean mmio_dm or mmio_direct here.

Jan

author
#6 lore raw
Quoted 17 lines

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 5, 2014 2:35 PM
To: Tian, Kevin
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ye, Wei; Lv, Zhiyuan; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 23:34, <[email protected]> wrote:

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 3 lines

Depends on whether you mean mmio_dm or mmio_direct here.

Jan

It's p2m_mmio_dm.

#7 lore raw
Quoted 18 lines
Quoted 18 lines
Quoted 1 line · On 05.08.14 at 08:46, <[email protected]> wrote:

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 5, 2014 2:35 PM
To: Tian, Kevin
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ye, Wei; Lv, Zhiyuan; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 23:34, <[email protected]> wrote:

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 3 lines
Quoted 1 line

Depends on whether you mean mmio_dm or mmio_direct here.

It's p2m_mmio_dm.

In which case - how would this help? You'd still not be able to deal with device side accesses. (And as a side note - a hypercall to convert between p2m_ram_rw and p2m_mmio_dm already exists:
HVMOP_set_mem_type.)

Jan

#8 lore raw
In reply to Tian, Kevin

Tian, Kevin wrote on 2014-08-05:

Quoted 49 lines
Quoted 32 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 17 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualization-
xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

Quoted 7 lines
Quoted 2 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d. An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 7 lines

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

Quoted 7 lines

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

Quoted 8 lines

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Best regards,
Yang

#9 lore raw
Quoted 57 lines
Quoted 1 line
Quoted 1 line · On 05.08.14 at 09:35, <[email protected]> wrote:

Tian, Kevin wrote on 2014-08-05:

Quoted 55 lines
Quoted 32 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 17 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualization-
xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

Quoted 7 lines
Quoted 2 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d. An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 7 lines

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 4 lines
Quoted 1 line

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

That would cause faults on reads as well, making it necessary to emulate them.

Jan

author
#10 lore raw
Quoted 69 lines

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 5, 2014 3:52 PM
To: Tian, Kevin; Ye, Wei; Zhang, Yang Z
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Lv, Zhiyuan; [email protected]; [email protected]; [email protected]
Subject: RE: [Xen-devel] [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 57 lines
Quoted 1 line
Quoted 1 line · On 05.08.14 at 09:35, <[email protected]> wrote:

Tian, Kevin wrote on 2014-08-05:

Quoted 55 lines
Quoted 32 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 17 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualization-
xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

Quoted 7 lines
Quoted 2 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d. An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 7 lines

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

That would cause faults on reads as well, making it necessary to emulate them.

In our usage case, the read access is much less than write (zero read in Linux guest and 1/4 for read access compared to write in windows guest).

Quoted 2 lines

Jan

#11 lore raw
In reply to Jan Beulich
Quoted 60 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 05, 2014 12:52 AM

Quoted 57 lines
Quoted 1 line
Quoted 1 line · On 05.08.14 at 09:35, <[email protected]> wrote:

Tian, Kevin wrote on 2014-08-05:

Quoted 55 lines
Quoted 32 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 17 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualization-
xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

Quoted 7 lines
Quoted 2 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d. An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 7 lines

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

That would cause faults on reads as well, making it necessary to emulate them.

Then it's same effect of p2m_mmio_dm.

Thanks
Kevin

#12 lore raw

Tian, Kevin wrote on 2014-08-05:

Quoted 62 lines
Quoted 62 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 05, 2014 12:52 AM

Quoted 59 lines
Quoted 58 lines
Quoted 34 lines

On 05.08.14 at 09:35, <[email protected]> wrote: Tian, Kevin wrote on 2014-08-05: From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 22 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 18 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to
multiple device models according to the io range. XenGT
(Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualizatio
n-
xengt) driver reside in Dom0 as a virtual graphics device
model also need to trap and emulate the guest's write
operation to some specific memory pages, like the memory
pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write
operation. Page of this new p2m type is read only and for
write, the request will go to device model via ioreq-server.

Quoted 7 lines
Quoted 2 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d. An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 7 lines

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Comments?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

That would cause faults on reads as well, making it necessary to emulate them.

Then it's same effect of p2m_mmio_dm.

p2m_mmio_dm will break VT-d but EPT misconfiguration doesn't.

Quoted 3 lines

Thanks
Kevin

Best regards,
Yang

#13 lore raw
Quoted 71 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:11 PM

Tian, Kevin wrote on 2014-08-05:

Quoted 16 lines
Quoted 16 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 05, 2014 12:52 AM

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

On 05.08.14 at 09:35, <[email protected]> wrote: Tian, Kevin wrote on 2014-08-05: From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 5 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 1 line
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualizatio
n-
xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics

translation table).

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d. An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side

ones not?

Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 11 lines
Quoted 1 line

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

Quoted 7 lines

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

That would cause faults on reads as well, making it necessary to emulate them.

Then it's same effect of p2m_mmio_dm.

p2m_mmio_dm will break VT-d but EPT misconfiguration doesn't.

I don't understand. Treating it as emulated io just means no valid p2m entry in EPT/VT-d. Note XenGT itself doesn't require VT-d, while you can still assign other devices this way.

Thanks
Kevin

#14 lore raw

Tian, Kevin wrote on 2014-08-06:

Quoted 83 lines
Quoted 70 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:11 PM

Tian, Kevin wrote on 2014-08-05:

Quoted 66 lines
Quoted 66 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 05, 2014 12:52 AM

Quoted 62 lines
Quoted 52 lines
Quoted 34 lines

On 05.08.14 at 09:35, <[email protected]> wrote: Tian, Kevin wrote on 2014-08-05: From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 17 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualizat io n- xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

Quoted 7 lines
Quoted 2 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d. An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 8 lines

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

Quoted 7 lines

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

That would cause faults on reads as well, making it necessary to emulate them.

Then it's same effect of p2m_mmio_dm.

p2m_mmio_dm will break VT-d but EPT misconfiguration doesn't.

I don't understand. Treating it as emulated io just means no valid p2m entry in EPT/VT-d. Note XenGT itself doesn't require VT-d, while you can still assign other devices this way.

Quoted 5 lines

From guest's view, it doesn't know the page is not valid in EPT/VT-d table. So if guest CPU touch this page, then hypervisor will intercept the access from EPT violation and handle it. But if guest uses that page as DMA buffer (a malicious guest may do it), then VT-d fault happens. Since DMA is not restartable, guest will never receive the right data.

Thanks
Kevin

Best regards,
Yang

#15 lore raw
Quoted 88 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:40 PM

Tian, Kevin wrote on 2014-08-06:

Quoted 33 lines
Quoted 33 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:11 PM

Tian, Kevin wrote on 2014-08-05:

Quoted 29 lines
Quoted 29 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 05, 2014 12:52 AM

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

On 05.08.14 at 09:35, <[email protected]> wrote: Tian, Kevin wrote on 2014-08-05: From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 17 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualizat io n- xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write operation to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the page write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and

vt-d.

Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 27 lines
Quoted 27 lines
Quoted 11 lines
Quoted 2 lines

An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 4 lines
Quoted 1 line

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

That would cause faults on reads as well, making it necessary to emulate them.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Then it's same effect of p2m_mmio_dm.

p2m_mmio_dm will break VT-d but EPT misconfiguration doesn't.

I don't understand. Treating it as emulated io just means no valid p2m entry in EPT/VT-d. Note XenGT itself doesn't require VT-d, while you can still assign other devices this way.

Quoted 6 lines

From guest's view, it doesn't know the page is not valid in EPT/VT-d table. So if guest CPU touch this page, then hypervisor will intercept the access from EPT violation and handle it. But if guest uses that page as DMA buffer (a malicious guest may do it), then VT-d fault happens. Since DMA is not restartable, guest will never receive the right data.

as you said, it's malicious guest, that's why we need zap the VT-d entries to avoid mallicous DMA to GPU page tables. In sane case the page is allocated by gfx driver so other device drivers with VT-d assigned devices shouldn't use it as DMA target. There is no 'right data' to be ensured in such case. :-)

Thanks
Kevin

#16 lore raw
In reply to Zhang, Yang Z
Quoted 104 lines

From: Tian, Kevin
Sent: Tuesday, August 05, 2014 7:49 PM

Quoted 31 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:40 PM

Tian, Kevin wrote on 2014-08-06:

Quoted 27 lines
Quoted 27 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:11 PM

Tian, Kevin wrote on 2014-08-05:

Quoted 23 lines
Quoted 23 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 05, 2014 12:52 AM

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

On 05.08.14 at 09:35, <[email protected]> wrote: Tian, Kevin wrote on 2014-08-05: From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 11 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 7 lines
Quoted 1 line

On 28.07.14 at 19:55, <[email protected]> wrote:

ioreq-server is proposed to forward PIO and MMIO request to multiple device models according to the io range. XenGT (Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtualizat io n- xengt) driver reside in Dom0 as a virtual graphics device model also need to trap and emulate the guest's write

operation

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

to some specific memory pages, like the memory pages used by guest graphics driver as PPGTT(per-process graphics translation table). We add an new p2m type "p2m_ram_wp" to trap the

page

Quoted 47 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

write operation. Page of this new p2m type is read only and for write, the request will go to device model via ioreq-server.

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and

vt-d.

Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 11 lines
Quoted 2 lines

An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

That would cause faults on reads as well, making it necessary to emulate them.

Then it's same effect of p2m_mmio_dm.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

p2m_mmio_dm will break VT-d but EPT misconfiguration doesn't.

I don't understand. Treating it as emulated io just means no valid p2m entry in EPT/VT-d. Note XenGT itself doesn't require VT-d, while you can still assign other devices this way.

From guest's view, it doesn't know the page is not valid in EPT/VT-d table. So

if

Quoted 1 line

guest CPU touch this page, then hypervisor will intercept the access from

EPT

Quoted 1 line

violation and handle it. But if guest uses that page as DMA buffer (a

malicious

Quoted 1 line

guest may do it), then VT-d fault happens. Since DMA is not restartable,

guest

Quoted 2 lines

will never receive the right data.

Quoted 8 lines

as you said, it's malicious guest, that's why we need zap the VT-d entries to avoid
mallicous DMA to GPU page tables. In sane case the page is allocated by gfx driver
so other device drivers with VT-d assigned devices shouldn't use it as DMA target.
There is no 'right data' to be ensured in such case. :-)

Think about a malicious guest may DMA to any holes...

Thanks
Kevin

#17 lore raw

Tian, Kevin wrote on 2014-08-06:

Quoted 40 lines
Quoted 40 lines

From: Tian, Kevin
Sent: Tuesday, August 05, 2014 7:49 PM

Quoted 37 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:40 PM

Tian, Kevin wrote on 2014-08-06:

Quoted 33 lines
Quoted 33 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:11 PM

Tian, Kevin wrote on 2014-08-05:

Quoted 29 lines
Quoted 29 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 05, 2014 12:52 AM

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines · On 05.08.14 at 09:35, <[email protected]> wrote:

Tian, Kevin wrote on 2014-08-05: From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 17 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 13 lines

On 28.07.14 at 19:55, <[email protected]> wrote:
ioreq-server is proposed to forward PIO and MMIO request to
multiple device models according to the io range. XenGT
(Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtual iz
at io n- xengt) driver reside in Dom0 as a virtual graphics
device model also need to trap and emulate the guest's write
operation to some specific memory pages, like the memory
pages used by guest graphics driver as PPGTT(per-process
graphics translation table). We add an new p2m type
"p2m_ram_wp" to trap the page write operation. Page of this
new p2m type is read only and for write, the request will go
to device model via

Quoted 61 lines

ioreq-server.

Quoted 60 lines
Quoted 51 lines
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 41 lines
Quoted 33 lines
Quoted 33 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

So how is this write-protection supposed to work on the IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d. An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 8 lines

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 4 lines
Quoted 1 line

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

That would cause faults on reads as well, making it necessary to emulate them.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Then it's same effect of p2m_mmio_dm.

p2m_mmio_dm will break VT-d but EPT misconfiguration doesn't.

I don't understand. Treating it as emulated io just means no valid p2m entry in EPT/VT-d. Note XenGT itself doesn't require VT-d, while you can still assign other devices this way.

Quoted 7 lines

From guest's view, it doesn't know the page is not valid in EPT/VT-d table. So if guest CPU touch this page, then hypervisor will intercept the access from EPT violation and handle it. But if guest uses that page as DMA buffer (a malicious guest may do it), then VT-d fault happens. Since DMA is not restartable, guest will never receive the right data.

as you said, it's malicious guest, that's why we need zap the VT-d

Malicious guest is just an example. We can assume a normal guest never does it.

Quoted 3 lines
Quoted 3 lines

entries to avoid mallicous DMA to GPU page tables. In sane case the page is allocated by gfx driver so other device drivers with VT-d assigned devices shouldn't use it as DMA target.

What about a guest really does it?

Quoted 4 lines
Quoted 2 lines

There is no 'right data' to be ensured in such case. :-)

Think about a malicious guest may DMA to any holes...

Other hole is not a RAM, so a VT-d fault is acceptable.

Quoted 3 lines

Thanks
Kevin

Best regards,
Yang

#18 lore raw
Quoted 101 lines · Yang Z

Tian, Kevin wrote on 2014-08-06:

Quoted 100 lines
Quoted 40 lines

From: Tian, Kevin
Sent: Tuesday, August 05, 2014 7:49 PM

Quoted 37 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:40 PM

Tian, Kevin wrote on 2014-08-06:

Quoted 33 lines
Quoted 33 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:11 PM

Tian, Kevin wrote on 2014-08-05:

Quoted 29 lines
Quoted 29 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 05, 2014 12:52 AM

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines · On 05.08.14 at 09:35, <[email protected]> wrote:

Tian, Kevin wrote on 2014-08-05: From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 21 lines
Quoted 1 line

On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:

Quoted 17 lines

[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to
support page write protection

On 28.07.14 at 19:55, <[email protected]> wrote:
ioreq-server is proposed to forward PIO and MMIO request to
multiple device models according to the io range. XenGT
(Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtual iz
at io n- xengt) driver reside in Dom0 as a virtual graphics
device model also need to trap and emulate the guest's write
operation to some specific memory pages, like the memory
pages used by guest graphics driver as PPGTT(per-process
graphics translation table). We add an new p2m type
"p2m_ram_wp" to trap the page write operation. Page of this
new p2m type is read only and for write, the request will go
to device model via

ioreq-server.

Quoted 59 lines
Quoted 58 lines
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 41 lines
Quoted 33 lines
Quoted 33 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

So how is this write-protection supposed to work on the
IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not supposed to work when sharing page tables between EPT and vt-d. An explicit command line "iommu=no-sharept" should be setted for avoiding undesirable iommu fault.

Quoted 8 lines

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond that. Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view between CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 4 lines
Quoted 1 line

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

That would cause faults on reads as well, making it necessary to emulate them.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Then it's same effect of p2m_mmio_dm.

p2m_mmio_dm will break VT-d but EPT misconfiguration doesn't.

I don't understand. Treating it as emulated io just means no valid p2m entry in EPT/VT-d. Note XenGT itself doesn't require VT-d, while you can still assign other devices this way.

From guest's view, it doesn't know the page is not valid in EPT/VT-d table. So if guest CPU touch this page, then hypervisor will intercept the access from EPT violation and handle it. But if guest uses that page as DMA buffer (a malicious guest may do it), then VT-d fault happens. Since DMA is not restartable, guest will never receive the right data.

Quoted 3 lines
Quoted 1 line
Quoted 1 line

as you said, it's malicious guest, that's why we need zap the VT-d

Malicious guest is just an example. We can assume a normal guest never does it.

What!? No you can't. That is thundering towards an XSA! The normal guest can become compromised.

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

entries to avoid mallicous DMA to GPU page tables. In sane case the page is allocated by gfx driver so other device drivers with VT-d assigned devices shouldn't use it as DMA target.

What about a guest really does it?

Oh, here you say it can do it. A bit confusing.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

There is no 'right data' to be ensured in such case. :-)

Think about a malicious guest may DMA to any holes...

Other hole is not a RAM, so a VT-d fault is acceptable.

I feel that we talking about the same issue that had been raised before about EPT and VT-d sharing a page and having issues with DMAing in RAM regions (like into the framebuffer).

And my understanding is that Intel is going to fix it, except that it is on the 'low-priority'. It sounds to me that it should be raised to a higher priority and be taken care of.

Quoted 15 lines
Quoted 4 lines

Thanks
Kevin

Best regards,
Yang

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#19 lore raw
Quoted 110 lines

From: Konrad Rzeszutek Wilk [mailto:[email protected]]
Sent: Wednesday, August 06, 2014 8:01 AM

Quoted 30 lines · Yang Z

Tian, Kevin wrote on 2014-08-06:

Quoted 29 lines
Quoted 29 lines

From: Tian, Kevin
Sent: Tuesday, August 05, 2014 7:49 PM

Quoted 26 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:40 PM

Tian, Kevin wrote on 2014-08-06:

Quoted 22 lines
Quoted 22 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:11 PM

Tian, Kevin wrote on 2014-08-05:

Quoted 18 lines
Quoted 18 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 05, 2014 12:52 AM

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines · On 05.08.14 at 09:35, <[email protected]> wrote:

Tian, Kevin wrote on 2014-08-05: From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

Quoted 10 lines

On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:
[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to
support page write protection

On 28.07.14 at 19:55, <[email protected]> wrote:
ioreq-server is proposed to forward PIO and MMIO request

to

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

multiple device models according to the io range. XenGT
(Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtual iz
at io n- xengt) driver reside in Dom0 as a virtual graphics
device model also need to trap and emulate the guest's

write

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

operation to some specific memory pages, like the memory
pages used by guest graphics driver as PPGTT(per-process
graphics translation table). We add an new p2m type
"p2m_ram_wp" to trap the page write operation. Page of

this

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

new p2m type is read only and for write, the request will go
to device model via

ioreq-server.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

So how is this write-protection supposed to work on the
IOMMU side when sharing page tables?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Thanks for pointing out this question. Write-protection is not
supposed to work when sharing page tables between EPT and
vt-d. An explicit command line "iommu=no-sharept" should be
setted for avoiding undesirable iommu fault.

Requiring the unconditional use of a specific command line option is certainly fine for experimental code, but not beyond

that.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side writes from the guest. Why would device side writes be okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view

between

Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 13 lines
Quoted 13 lines

CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 4 lines
Quoted 1 line

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

That would cause faults on reads as well, making it necessary to emulate them.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Then it's same effect of p2m_mmio_dm.

p2m_mmio_dm will break VT-d but EPT misconfiguration doesn't.

I don't understand. Treating it as emulated io just means no valid p2m entry in EPT/VT-d. Note XenGT itself doesn't require VT-d, while you can still assign other devices this way.

From guest's view, it doesn't know the page is not valid in EPT/VT-d table. So if guest CPU touch this page, then hypervisor will intercept the access from EPT violation and handle it. But if guest uses that page as DMA buffer (a malicious guest may do it), then VT-d fault happens. Since DMA is not restartable, guest will never receive the right data.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

as you said, it's malicious guest, that's why we need zap the VT-d

Malicious guest is just an example. We can assume a normal guest never

does it.

What!? No you can't. That is thundering towards an XSA! The normal guest can become compromised.

It would be a severe security hole to write-protect only the CPU path while leaving DMA writes valid. When we say a page is write-protected, it must be consistently enforced in both CPU/DMA paths. Now there is no other way to achieve it lacking of VT-d page fault, except treating it fully as a emulated MMIO range (means removing mapping in both EPT/VT-d page table), or excluding VT-d like this patch originally does.

Quoted 17 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

entries to avoid mallicous DMA to GPU page tables. In sane case the page is allocated by gfx driver so other device drivers with VT-d assigned devices shouldn't use it as DMA target.

What about a guest really does it?

Oh, here you say it can do it. A bit confusing.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

There is no 'right data' to be ensured in such case. :-)

Think about a malicious guest may DMA to any holes...

Other hole is not a RAM, so a VT-d fault is acceptable.

Quoted 8 lines

I feel that we talking about the same issue that had been raised before about EPT and VT-d sharing a page and having issues with DMAing in RAM regions (like into the framebuffer).

And my understanding is that Intel is going to fix it, except that it is on the 'low-priority'. It sounds to me that it should be raised to a higher priority and be taken care of.

I'm not in original discussion loop, but my feeling is w/o VT-d fault support anything related to write-protection (e.g. logdirty) will remains a gap here (i.e. the feature has to be exclusive to VT-d usage)

Thanks
Kevin

#20 lore raw
Quoted 9 lines
Quoted 8 lines
Quoted 1 line · On 06.08.14 at 18:08, <[email protected]> wrote:

From: Konrad Rzeszutek Wilk [mailto:[email protected]]

Quoted 1 line · Yang Z

Malicious guest is just an example. We can assume a normal guest never

does it.

What!? No you can't. That is thundering towards an XSA! The normal guest can become compromised.

Quoted 6 lines

It would be a severe security hole to write-protect only the CPU path while leaving DMA writes valid. When we say a page is write-protected, it must be consistently enforced in both CPU/DMA paths. Now there is no other way to achieve it lacking of VT-d page fault, except treating it fully as a emulated MMIO range (means removing mapping in both EPT/VT-d page table), or excluding VT-d like this patch originally does.

A fully emulated range means more than just removing the mapping
though: No passed through device must access the memory range, since you can't emulate such accesses. While this is fine for existing uses (LAPIC, HPET, ...), I don't think it is for things like a frame buffer. If otoh talk is exclusively about GPU page tables, then this ought to be fine, but would - to me - not fit with other aspects discussed in the context of these patches (like the supposedly large number of pages needing to be tracked for a guest).

Jan

#21 lore raw
Quoted 19 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Wednesday, August 06, 2014 11:45 PM

Quoted 9 lines
Quoted 8 lines
Quoted 1 line · On 06.08.14 at 18:08, <[email protected]> wrote:

From: Konrad Rzeszutek Wilk [mailto:[email protected]]

Quoted 1 line · Yang Z

Malicious guest is just an example. We can assume a normal guest never

does it.

What!? No you can't. That is thundering towards an XSA! The normal guest can become compromised.

Quoted 6 lines

It would be a severe security hole to write-protect only the CPU path while leaving DMA writes valid. When we say a page is write-protected, it must be consistently enforced in both CPU/DMA paths. Now there is no other way to achieve it lacking of VT-d page fault, except treating it fully as a emulated MMIO range (means removing mapping in both EPT/VT-d page table), or excluding VT-d like this patch originally does.

Quoted 5 lines

A fully emulated range means more than just removing the mapping
though: No passed through device must access the memory range, since you can't emulate such accesses. While this is fine for existing uses (LAPIC, HPET, ...), I don't think it is for things like a frame buffer. If otoh talk is exclusively about GPU page tables, then this

I don't understand here. why would a pass-through device wants to access frame buffer (I think you meant 'virtual' frame buffer)? Note XenGT itself doesn't use VT-d, and we didn't see such usage of having another device DMA to GPU resource. Then it's just same as any existing emulated MMIO ranges e.g. on a virtual NIC, where no pass-through device would access them.

Quoted 4 lines

ought to be fine, but would - to me - not fit with other aspects discussed in the context of these patches (like the supposedly large number of pages needing to be tracked for a guest).

So what's your suggestion here?

Thanks
Kevin

#22 lore raw
Quoted 17 lines
Quoted 11 lines
Quoted 1 line · On 07.08.14 at 18:28, <[email protected]> wrote:

From: Jan Beulich [mailto:[email protected]]
Sent: Wednesday, August 06, 2014 11:45 PM

Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 06.08.14 at 18:08, <[email protected]> wrote:

It would be a severe security hole to write-protect only the CPU path while leaving DMA writes valid. When we say a page is write-protected, it must be consistently enforced in both CPU/DMA paths. Now there is no other way to achieve it lacking of VT-d page fault, except treating it fully as a emulated MMIO range (means removing mapping in both EPT/VT-d page table), or excluding VT-d like this patch originally does.

Quoted 5 lines

A fully emulated range means more than just removing the mapping
though: No passed through device must access the memory range, since you can't emulate such accesses. While this is fine for existing uses (LAPIC, HPET, ...), I don't think it is for things like a frame buffer. If otoh talk is exclusively about GPU page tables, then this

Quoted 6 lines

I don't understand here. why would a pass-through device wants to access frame buffer (I think you meant 'virtual' frame buffer)? Note XenGT itself doesn't use VT-d, and we didn't see such usage of having another device DMA to GPU resource. Then it's just same as any existing emulated MMIO ranges e.g. on a virtual NIC, where no pass-through device would access them.

This is the same as the discussion about VT-d code not supporting large pages right now: An OS _might_ do clever things (merely dumping the frame buffer verbatim to disk or network would be one of the more trivial examples). Current OSes (or more precisely their drivers) may not do this, but I can only repeat my fundamental position here: We should not make assumptions about HVM guest behavior except in cases where we want to optimize certain things.

Quoted 5 lines
Quoted 3 lines

ought to be fine, but would - to me - not fit with other aspects discussed in the context of these patches (like the supposedly large number of pages needing to be tracked for a guest).

So what's your suggestion here?

I can't answer that without you clarifying whether talk earlier on was about the frame buffer or page tables.

Jan

#23 lore raw
Quoted 31 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Thursday, August 07, 2014 11:33 PM

Quoted 5 lines
Quoted 5 lines
Quoted 1 line · On 07.08.14 at 18:28, <[email protected]> wrote:

From: Jan Beulich [mailto:[email protected]]
Sent: Wednesday, August 06, 2014 11:45 PM

Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 06.08.14 at 18:08, <[email protected]> wrote:

It would be a severe security hole to write-protect only the CPU path

while

Quoted 1 line
Quoted 1 line
Quoted 1 line

leaving DMA writes valid. When we say a page is write-protected, it must

be

Quoted 1 line
Quoted 1 line
Quoted 1 line

consistently enforced in both CPU/DMA paths. Now there is no other way

to

Quoted 1 line
Quoted 1 line
Quoted 1 line

achieve it lacking of VT-d page fault, except treating it fully as a

emulated

Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

MMIO range (means removing mapping in both EPT/VT-d page table), or excluding VT-d like this patch originally does.

A fully emulated range means more than just removing the mapping
though: No passed through device must access the memory range, since you can't emulate such accesses. While this is fine for existing uses (LAPIC, HPET, ...), I don't think it is for things like a frame buffer. If otoh talk is exclusively about GPU page tables, then this

Quoted 6 lines

I don't understand here. why would a pass-through device wants to access frame buffer (I think you meant 'virtual' frame buffer)? Note XenGT itself doesn't use VT-d, and we didn't see such usage of having another device DMA to GPU resource. Then it's just same as any existing emulated MMIO ranges e.g. on a virtual NIC, where no pass-through device would access them.

Quoted 7 lines

This is the same as the discussion about VT-d code not supporting large pages right now: An OS _might_ do clever things (merely dumping the frame buffer verbatim to disk or network would be one of the more trivial examples). Current OSes (or more precisely their drivers) may not do this, but I can only repeat my fundamental position here: We should not make assumptions about HVM guest behavior except in cases where we want to optimize certain things.

with a complete HW support yes we should make assumption for guest behavior (e.g. with VT-x we can support any type of OS), but when HW support in VT-d is not fully ready, then making some assumption makes sense especially when no real usage is observed in this case.

Quoted 10 lines
Quoted 6 lines
Quoted 4 lines

ought to be fine, but would - to me - not fit with other aspects discussed in the context of these patches (like the supposedly large number of pages needing to be tracked for a guest).

So what's your suggestion here?

I can't answer that without you clarifying whether talk earlier on was about the frame buffer or page tables.

In our case it's only GPU page table. XenGT assigns frame buffer to the VM so there's no concern in that path.

Thanks
Kevin

#24 lore raw
In reply to Jan Beulich
Quoted 45 lines

From: Tian, Kevin
Sent: Friday, August 08, 2014 9:02 AM

Quoted 10 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Thursday, August 07, 2014 11:33 PM

Quoted 5 lines
Quoted 5 lines
Quoted 1 line · On 07.08.14 at 18:28, <[email protected]> wrote:

From: Jan Beulich [mailto:[email protected]]
Sent: Wednesday, August 06, 2014 11:45 PM

Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 06.08.14 at 18:08, <[email protected]> wrote:

It would be a severe security hole to write-protect only the CPU path

while

Quoted 1 line
Quoted 1 line
Quoted 1 line

leaving DMA writes valid. When we say a page is write-protected, it

must

Quoted 2 lines

be

Quoted 1 line
Quoted 1 line
Quoted 1 line

consistently enforced in both CPU/DMA paths. Now there is no other

way

Quoted 4 lines

to

Quoted 1 line
Quoted 1 line
Quoted 1 line

achieve it lacking of VT-d page fault, except treating it fully as a

emulated

Quoted 1 line
Quoted 1 line
Quoted 1 line

MMIO range (means removing mapping in both EPT/VT-d page table),

or

Quoted 15 lines
Quoted 8 lines
Quoted 7 lines
Quoted 1 line

excluding VT-d like this patch originally does.

A fully emulated range means more than just removing the mapping
though: No passed through device must access the memory range, since you can't emulate such accesses. While this is fine for existing uses (LAPIC, HPET, ...), I don't think it is for things like a frame buffer. If otoh talk is exclusively about GPU page tables, then this

Quoted 6 lines

I don't understand here. why would a pass-through device wants to access frame buffer (I think you meant 'virtual' frame buffer)? Note XenGT itself doesn't use VT-d, and we didn't see such usage of having another device DMA to GPU resource. Then it's just same as any existing emulated MMIO ranges e.g. on a virtual NIC, where no pass-through device would access them.

Quoted 7 lines

This is the same as the discussion about VT-d code not supporting large pages right now: An OS _might_ do clever things (merely dumping the frame buffer verbatim to disk or network would be one of the more trivial examples). Current OSes (or more precisely their drivers) may not do this, but I can only repeat my fundamental position here: We should not make assumptions about HVM guest behavior except in cases where we want to optimize certain things.

Quoted 1 line

with a complete HW support yes we should make assumption for guest

should -> shouldn't

Quoted 4 lines

behavior (e.g. with VT-x we can support any type of OS), but when HW support in VT-d is not fully ready, then making some assumption makes sense especially when no real usage is observed in this case.

author
#25 lore raw

Some summary of the discussion:

Without vt-d page fault support, a general write-protection is not possible. It's impossible to protect the write of both CPU and DMA path. But note that XenGT itself doesn't use VT-d, and there is no another usage of having another device DMA to GPU resource. Or even other device use the protect page for its DMA buffer, like a malicious guest, there is no security hole since it can only access it's our memory range. So, just keep protecting the CPU write path may be feasible for this moment.

Another way is treat the page as MMIO ranges(means removing mapping in both EPD/VT-D page table). In this way, both read/write access should be forwarded and emulated. For XenGT usage case, we observed that for the Linux guest, there is no read access to the ppgtt page tables. And for the windows guest, the read access is just about 20% of write. The read overhead is relatively small, so treat the pages as MMIO range may be possible.

Any comments?

Regards
Wei

Quoted 70 lines

-----Original Message-----
From: Tian, Kevin
Sent: Saturday, August 9, 2014 12:04 AM
To: Jan Beulich
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Dugger, Donald D; Ye, Wei; Zhang, Yang Z; Lv, Zhiyuan; [email protected]; Konrad Rzeszutek Wilk; [email protected]; [email protected]
Subject: RE: [Xen-devel] [PATCH v1 0/2] Extend ioreq-server to support page write protection

Quoted 51 lines

From: Tian, Kevin
Sent: Friday, August 08, 2014 9:02 AM

Quoted 12 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Thursday, August 07, 2014 11:33 PM

Quoted 6 lines
Quoted 6 lines
Quoted 1 line · On 07.08.14 at 18:28, <[email protected]> wrote:

From: Jan Beulich [mailto:[email protected]]
Sent: Wednesday, August 06, 2014 11:45 PM

Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 06.08.14 at 18:08, <[email protected]> wrote:

It would be a severe security hole to write-protect only the CPU path

while

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

leaving DMA writes valid. When we say a page is
write-protected, it

must

Quoted 3 lines

be

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

consistently enforced in both CPU/DMA paths. Now there is no other

way

Quoted 6 lines

to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

achieve it lacking of VT-d page fault, except treating it fully as a

emulated

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

MMIO range (means removing mapping in both EPT/VT-d page table),

or

Quoted 16 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line

excluding VT-d like this patch originally does.

A fully emulated range means more than just removing the mapping
though: No passed through device must access the memory range, since you can't emulate such accesses. While this is fine for existing uses (LAPIC, HPET, ...), I don't think it is for things like a frame buffer. If otoh talk is exclusively about GPU page tables, then this

Quoted 6 lines

I don't understand here. why would a pass-through device wants to access frame buffer (I think you meant 'virtual' frame buffer)? Note XenGT itself doesn't use VT-d, and we didn't see such usage of having another device DMA to GPU resource. Then it's just same as any existing emulated MMIO ranges e.g. on a virtual NIC, where no pass-through device would access them.

Quoted 7 lines

This is the same as the discussion about VT-d code not supporting large pages right now: An OS _might_ do clever things (merely dumping the frame buffer verbatim to disk or network would be one of the more trivial examples). Current OSes (or more precisely their drivers) may not do this, but I can only repeat my fundamental position here: We should not make assumptions about HVM guest behavior except in cases where we want to optimize certain things.

Quoted 1 line

with a complete HW support yes we should make assumption for guest

should -> shouldn't

Quoted 4 lines

behavior (e.g. with VT-x we can support any type of OS), but when HW support in VT-d is not fully ready, then making some assumption makes sense especially when no real usage is observed in this case.

#26 lore raw
Quoted 6 lines · Wei

Some summary of the discussion:

Without vt-d page fault support, a general write-protection is not possible. It's impossible to protect the write of both CPU and DMA path. But note that XenGT itself doesn't use VT-d, and there is no another usage of having another device DMA to GPU resource.

None that you're aware of. But there might be some out-of-tree users?

Quoted 4 lines

Or even
other device use the protect page for its DMA buffer, like a malicious guest, there is no security hole since it can only access it's our memory range.

Not true at all! If the guest has a read-only mapping it must be read-only for a reason. For example, it might have a read-only grant mapping of another VM's memory. Or an out-of-VM security tool might be enforcing W^X permissions.

Quoted 1 line

So, just keep protecting the CPU write path may be feasible for this moment.

I don't think so.

Quoted 7 lines

Another way is treat the page as MMIO ranges(means removing mapping in both EPD/VT-D page table). In this way, both read/write access should be forwarded and emulated. For XenGT usage case, we observed that for the Linux guest, there is no read access to the ppgtt page tables. And for the windows guest, the read access is just about 20% of write. The read overhead is relatively small, so treat the pages as MMIO range may be possible.

That might work for XenGT but it doesn't address the general problem at all. AFAICT the only _safe_ way to make a page read-only to a guest that can issue DMA is to remove it from the IOMMU tables - and that means having separate EPT and vtD tables, until hardware support for restartable IOMMU faults comes along.

Cheers,

Tim.

#27 lore raw
Quoted 4 lines

From: Tim Deegan [mailto:[email protected]]
Sent: Wednesday, August 13, 2014 1:39 AM

At 23:15 +0000 on 12 Aug (1407881757), Ye, Wei wrote:

Quoted 8 lines
Quoted 6 lines

Some summary of the discussion:

Without vt-d page fault support, a general write-protection is not possible. It's impossible to protect the write of both CPU and DMA path. But note that XenGT itself doesn't use VT-d, and there is no another usage of having another device DMA to GPU resource.

None that you're aware of. But there might be some out-of-tree users?

If there is, then it has to be in the unsupported feature list. I think it's fair enough since we all know w/o complete VT-d page fault support there's no solution for such usage. Pretty much like old days big real mode tricks before hardware support is ready.

Quoted 10 lines
Quoted 5 lines

Or even
other device use the protect page for its DMA buffer, like a malicious guest, there is no security hole since it can only access it's our memory range.

Not true at all! If the guest has a read-only mapping it must be read-only for a reason. For example, it might have a read-only grant mapping of another VM's memory. Or an out-of-VM security tool might be enforcing W^X permissions.

sorry original statement is inaccurate. there is no cross-VM security hole since EPT/VT-d are fully under VMM's control, but yes out-of-VM security tool might be broken...

Quoted 5 lines
Quoted 2 lines

So, just keep protecting the CPU write path may be feasible for this

moment.

I don't think so.

Could we introduce a cpu-write-protection attribute only for functional usages which don't care about DMA path like in XenGT? Then out-of-VM security tool relying on complete write-protection in both CPU/DMA paths still need to wait full VT-d page fault. I know it's not clean, but want to hear your comments anything we can do here in current stage. :-)

Quoted 9 lines
Quoted 8 lines

Another way is treat the page as MMIO ranges(means removing mapping in both EPD/VT-D page table). In this way, both read/write access should be forwarded and emulated. For XenGT usage case, we observed that for the Linux guest, there is no read access to the ppgtt page tables. And for the windows guest, the read access is just about 20% of write. The read overhead is relatively small, so treat the pages as MMIO range may be possible.

Quoted 5 lines

That might work for XenGT but it doesn't address the general problem at all. AFAICT the only _safe_ way to make a page read-only to a guest that can issue DMA is to remove it from the IOMMU tables - and that means having separate EPT and vtD tables, until hardware support for restartable IOMMU faults comes along.

Aren't we talking about a similar option? ours is more conservative, not only removing from IOMMU table, but also removing from EPT so both r/w are trapped which serves the read-only requirements with performance penalty. along this way we didn't aim to provide a general write-protection mechanism, but just to extend ioreq server to support scattered mmio ranges. If adding a new p2m type is deemed dirty, we may go to this option since it can meet XenGT's requirement, and doesn't force other usages to go same route.

Actually separate or shared EPT/VT-d tables doesn't mean anything here. Removing the page from IOMMU tables still breaks guest's attempt. You protected it from write, but didn't emulate the DMA so it's still broken.

Thanks
Kevin

#28 lore raw
Quoted 11 lines · Kevin
Quoted 11 lines

From: Tim Deegan [mailto:[email protected]]
Sent: Wednesday, August 13, 2014 1:39 AM

Quoted 6 lines · Wei

Some summary of the discussion:

Without vt-d page fault support, a general write-protection is not possible. It's impossible to protect the write of both CPU and DMA path. But note that XenGT itself doesn't use VT-d, and there is no another usage of having another device DMA to GPU resource.

Quoted 6 lines
Quoted 1 line

None that you're aware of. But there might be some out-of-tree users?

If there is, then it has to be in the unsupported feature list. I think it's fair enough since we all know w/o complete VT-d page fault support there's no solution for such usage. Pretty much like old days big real mode tricks before hardware support is ready.

I think there's confusion here about what we're taling about. Yes, until we have h/w support for restarting faulting DMA the IOMMU feature set will be incomplete. The important thing is that it should fail safe -- i.e. if a guest has a read-only mapping of a page it is better to disallow DMA altogether than to allow read-write DMA.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Or even
other device use the protect page for its DMA buffer, like a malicious guest, there is no security hole since it can only access it's our memory range.

Quoted 8 lines
Quoted 4 lines

Not true at all! If the guest has a read-only mapping it must be read-only for a reason. For example, it might have a read-only grant mapping of another VM's memory. Or an out-of-VM security tool might be enforcing W^X permissions.

sorry original statement is inaccurate. there is no cross-VM security hole since EPT/VT-d are fully under VMM's control, but yes out-of-VM security tool might be broken...

As I just said: "it might have a read-only grant mapping of another VM's memory". We can't let it DMA into that mapping.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

So, just keep protecting the CPU write path may be feasible for this

moment.

I don't think so.

Could we introduce a cpu-write-protection attribute only for functional usages which don't care about DMA path like in XenGT?

Possibly. Sorry for going over things that have probably already been covered but AIUI you are saying that XenGT needs to allow full DMA access to this memory area but only read-only CPU access. If that's what's needed then I think a new type with these semantics is the best way of doing it (and requiring no shared IOMMU/HAP tables). But:
- it's pointless if the guest can make the device issue DMA for it, (with a blitting engine or similar) since it can just use that DMA to make the writes it wants; and
- for the same reason, you will need to give the GPU its own set of IOMMU tables, separate from the ones used for other devices that the guest controls. Otherwise the guest can use its NIC to DMA into the read-only area.

Quoted 14 lines
Quoted 8 lines
Quoted 7 lines

Another way is treat the page as MMIO ranges(means removing mapping in both EPD/VT-D page table). In this way, both read/write access should be forwarded and emulated. For XenGT usage case, we observed that for the Linux guest, there is no read access to the ppgtt page tables. And for the windows guest, the read access is just about 20% of write. The read overhead is relatively small, so treat the pages as MMIO range may be possible.

Quoted 5 lines

That might work for XenGT but it doesn't address the general problem at all. AFAICT the only _safe_ way to make a page read-only to a guest that can issue DMA is to remove it from the IOMMU tables - and that means having separate EPT and vtD tables, until hardware support for restartable IOMMU faults comes along.

Quoted 3 lines

Aren't we talking about a similar option? ours is more conservative, not only removing from IOMMU table, but also removing from EPT so both r/w are trapped which serves the read-only requirements with performance penalty.

Yes they are similar. I would say mine is more conservative in a different way - not using the emulator unneccesarily. :)

Cheers,

Tim.

#29 lore raw
Quoted 22 lines

From: Tim Deegan [mailto:[email protected]]
Sent: Thursday, August 14, 2014 1:08 AM

Quoted 11 lines · Kevin
Quoted 11 lines

From: Tim Deegan [mailto:[email protected]]
Sent: Wednesday, August 13, 2014 1:39 AM

Quoted 6 lines · Wei

Some summary of the discussion:

Without vt-d page fault support, a general write-protection is not possible. It's impossible to protect the write of both CPU and DMA path. But note that XenGT itself doesn't use VT-d, and there is no another usage of having another device DMA to GPU resource.

Quoted 6 lines
Quoted 1 line

None that you're aware of. But there might be some out-of-tree users?

If there is, then it has to be in the unsupported feature list. I think it's fair enough since we all know w/o complete VT-d page fault support there's no solution for such usage. Pretty much like old days big real mode tricks before hardware support is ready.

Quoted 5 lines

I think there's confusion here about what we're taling about. Yes, until we have h/w support for restarting faulting DMA the IOMMU feature set will be incomplete. The important thing is that it should fail safe -- i.e. if a guest has a read-only mapping of a page it is better to disallow DMA altogether than to allow read-write DMA.

Yes, there did be some confusions here. :-)

I think we are on the same page that, complete 'read-only' or complete 'write-protection' is impossible w/o restarting faulting DMA, because there is no way to enforce the access permission in the IOMMU side.

Then whatever 'read-only' or 'write-protection" discussion now is a partial solution where only CPU path is protected, with the assumption that no valid usage of using DMA to access the concerned memory area. And then I agree disallowing DMA becomes a fail-safe means to capture violating usages.

Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Or even
other device use the protect page for its DMA buffer, like a malicious guest, there is no security hole since it can only access it's our memory range.

Quoted 8 lines
Quoted 4 lines

Not true at all! If the guest has a read-only mapping it must be read-only for a reason. For example, it might have a read-only grant mapping of another VM's memory. Or an out-of-VM security tool might be enforcing W^X permissions.

sorry original statement is inaccurate. there is no cross-VM security hole since EPT/VT-d are fully under VMM's control, but yes out-of-VM security tool might be broken...

As I just said: "it might have a read-only grant mapping of another VM's memory". We can't let it DMA into that mapping.

Quoted 5 lines
Quoted 4 lines
Quoted 1 line

So, just keep protecting the CPU write path may be feasible for this

moment.

I don't think so.

Quoted 8 lines
Quoted 2 lines

Could we introduce a cpu-write-protection attribute only for functional usages which don't care about DMA path like in XenGT?

Possibly. Sorry for going over things that have probably already been covered but AIUI you are saying that XenGT needs to allow full DMA access to this memory area but only read-only CPU access. If that's what's needed then I think a new type with these semantics is the best way of doing it (and requiring no shared IOMMU/HAP tables). But:

It's about GPU page table write-protection, so XenGT needs to trap writes in both CPU/DMA paths:

* CPU writes are captured by read-only permission in EPT
* GPU writes are captured by scanning GPU commands before they're submitted for execution, as part of the 'mediation' concept in XenGT mediated pass-through architecture.

Note XenGT doesn't use IOMMU because w/o SR-IOV a single device entry can't be used to server multiple VMs.

So XenGT can achieve true 'write-protection' w/o IOMMU restarting fault support. However it's very GPU specific. As a general implementation, I think a new type with below semantics should be enough:

- only CPU access to the protected area, so it's removed from IOMMU page table
- require IOMMU/HAP seperate

How would you call such a new type then? just p2m_write_protection or p2m_cpu_write_protection? (still need to differentiate write-protection from read-only, since the latter implies no writes)

Quoted 3 lines

- it's pointless if the guest can make the device issue DMA for it, (with a blitting engine or similar) since it can just use that DMA to make the writes it wants; and

Based on above explanation it's not a problem

Quoted 4 lines

- for the same reason, you will need to give the GPU its own set of IOMMU tables, separate from the ones used for other devices that the guest controls. Otherwise the guest can use its NIC to DMA into the read-only area.

XenGT doesn't use IOMMU. So it's enough as long as we zap the range from all IOMMU page tables

in the future when we have SR-IOV GPU support, there's no need to shadow GPU page table then, and thus no such a requirement of write-protection.

Quoted 16 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Another way is treat the page as MMIO ranges(means removing

mapping

Quoted 13 lines
Quoted 12 lines
Quoted 6 lines

in both EPD/VT-D page table). In this way, both read/write access should be forwarded and emulated. For XenGT usage case, we observed that for the Linux guest, there is no read access to the ppgtt page tables. And for the windows guest, the read access is just about 20% of write. The read overhead is relatively small, so treat the pages as MMIO range may be possible.

That might work for XenGT but it doesn't address the general problem at all. AFAICT the only _safe_ way to make a page read-only to a guest that can issue DMA is to remove it from the IOMMU tables - and that means having separate EPT and vtD tables, until hardware support for restartable IOMMU faults comes along.

Quoted 7 lines
Quoted 3 lines

Aren't we talking about a similar option? ours is more conservative, not only removing from IOMMU table, but also removing from EPT so both r/w are trapped which serves the read-only requirements with performance penalty.

Yes they are similar. I would say mine is more conservative in a different way - not using the emulator unneccesarily. :)

yeah from different view. Our original thought is if new p2m type is tricky, then treating it as mmio could be an easier thing to extend. Now since you also agree adding a new type is reasonable, then we can continue this direction (actually original patch is along this road, but didn't zap IOMMU pages. :-))

Thanks
Kevin

#30 lore raw
Quoted 13 lines · Kevin
Quoted 9 lines

From: Tim Deegan [mailto:[email protected]]

Quoted 2 lines

Could we introduce a cpu-write-protection attribute only for functional usages which don't care about DMA path like in XenGT?

Possibly. Sorry for going over things that have probably already been covered but AIUI you are saying that XenGT needs to allow full DMA access to this memory area but only read-only CPU access. If that's what's needed then I think a new type with these semantics is the best way of doing it (and requiring no shared IOMMU/HAP tables). But:

It's about GPU page table write-protection, so XenGT needs to trap writes in both CPU/DMA paths:

Quoted 7 lines

* CPU writes are captured by read-only permission in EPT
* GPU writes are captured by scanning GPU commands before they're submitted for execution, as part of the 'mediation' concept in XenGT mediated pass-through architecture.

Note XenGT doesn't use IOMMU because w/o SR-IOV a single device entry can't be used to server multiple VMs.

Oh, I see. So you don't actually want the GPUPTs mapped for DMA at all. That's much easier.

Quoted 7 lines

So XenGT can achieve true 'write-protection' w/o IOMMU restarting fault support. However it's very GPU specific. As a general implementation, I think a new type with below semantics should be enough:

- only CPU access to the protected area, so it's removed from IOMMU page table
- require IOMMU/HAP seperate

I think we might want to make those tables separate in general. IIRC we were discussing that in another thread already. But that aside, I think a plain read-only mapping will DTRT for you -- it allows read-only DMA from the GPUPTs, but that's no worse that the read-only CPU mapping.

After that, the only odd behaviour is that you're emulating writes to read-only memory. And that's where the new type is helpful: to say that we should be emulating writes rather than dropping them.

Quoted 3 lines

How would you call such a new type then? just p2m_write_protection or p2m_cpu_write_protection? (still need to differentiate write-protection from read-only, since the latter implies no writes)

p2m_mmio_write_dm?

Now that (I hope) I understand better what you need, I can see how just using plain mmio_dm and emulating reads would be a plausible solution too. So I think I'd be happy with either that or a new type.

Cheers,

Tim.

#31 lore raw
Quoted 32 lines

From: Tim Deegan [mailto:[email protected]]
Sent: Thursday, August 14, 2014 1:25 PM

Quoted 13 lines · Kevin
Quoted 9 lines

From: Tim Deegan [mailto:[email protected]]

Quoted 2 lines

Could we introduce a cpu-write-protection attribute only for functional usages which don't care about DMA path like in XenGT?

Possibly. Sorry for going over things that have probably already been covered but AIUI you are saying that XenGT needs to allow full DMA access to this memory area but only read-only CPU access. If that's what's needed then I think a new type with these semantics is the best way of doing it (and requiring no shared IOMMU/HAP tables). But:

It's about GPU page table write-protection, so XenGT needs to trap writes in both CPU/DMA paths:

Quoted 7 lines

* CPU writes are captured by read-only permission in EPT
* GPU writes are captured by scanning GPU commands before they're submitted for execution, as part of the 'mediation' concept in XenGT mediated pass-through architecture.

Note XenGT doesn't use IOMMU because w/o SR-IOV a single device entry can't be used to server multiple VMs.

Oh, I see. So you don't actually want the GPUPTs mapped for DMA at all. That's much easier.

Quoted 4 lines

So XenGT can achieve true 'write-protection' w/o IOMMU restarting fault support. However it's very GPU specific. As a general implementation, I think a new type with below semantics should be enough:

Quoted 8 lines
Quoted 3 lines

- only CPU access to the protected area, so it's removed from IOMMU page table
- require IOMMU/HAP seperate

I think we might want to make those tables separate in general. IIRC we were discussing that in another thread already. But that aside, I think a plain read-only mapping will DTRT for you -- it allows read-only DMA from the GPUPTs, but that's no worse that the read-only CPU mapping.

for 'plain read-only mapping' you mean existing p2m_ram_ro?

btw just curious how we handle p2m_ram_ro today. Does it play any trick to fail-safe in IOMMU side, e.g. zap the mapping or not? Otherwise when it allows read-only DMA there's no way to prevent DMA writes either. :-)

Quoted 4 lines

After that, the only odd behaviour is that you're emulating writes to read-only memory. And that's where the new type is helpful: to say that we should be emulating writes rather than dropping them.

so p2m_mmio_write_dm would go together with existing p2m_ram_ro in most condition checks, with only difference on final policy of emulation vs. drop.

Quoted 11 lines
Quoted 4 lines

How would you call such a new type then? just p2m_write_protection or p2m_cpu_write_protection? (still need to differentiate write-protection from read-only, since the latter implies no writes)

p2m_mmio_write_dm?

Now that (I hope) I understand better what you need, I can see how just using plain mmio_dm and emulating reads would be a plausible solution too. So I think I'd be happy with either that or a new type.

yes either way works. for now let's try the new p2m type since it is more intuitive regarding to 'write-protection'. :-)

Thanks
Kevin

#32 lore raw
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 15.08.14 at 00:53, <[email protected]> wrote:

btw just curious how we handle p2m_ram_ro today. Does it play any trick to fail-safe in IOMMU side, e.g. zap the mapping or not? Otherwise when it allows read-only DMA there's no way to prevent DMA writes either. :-)

Why would that be? A page marked r/o in the IOMMU page tables ought to very well be protected against DMA writes. It's just that these will cause faults, which - if happening too frequently - will result in the device getting turned off, quite likely (implicitly) killing the guest.

Jan

#33 lore raw
Quoted 8 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Thursday, August 14, 2014 4:12 PM

Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 15.08.14 at 00:53, <[email protected]> wrote:

btw just curious how we handle p2m_ram_ro today. Does it play any trick to fail-safe in IOMMU side, e.g. zap the mapping or not? Otherwise when it allows read-only DMA there's no way to prevent DMA writes either. :-)

Quoted 6 lines

Why would that be? A page marked r/o in the IOMMU page tables ought to very well be protected against DMA writes. It's just that these will cause faults, which - if happening too frequently - will result in the device getting turned off, quite likely (implicitly) killing the guest.

yes, as protection it's enough, and it's only a problem for emulation. sorry for messed msg after a long discussion thread. :/

Thanks
Kevin

#34 lore raw
In reply to Konrad Rzeszutek Wilk
Quoted 126 lines

From: Tian, Kevin
Sent: Wednesday, August 06, 2014 9:09 AM

Quoted 34 lines

From: Konrad Rzeszutek Wilk [mailto:[email protected]]
Sent: Wednesday, August 06, 2014 8:01 AM

Quoted 30 lines · Yang Z

Tian, Kevin wrote on 2014-08-06:

Quoted 29 lines
Quoted 29 lines

From: Tian, Kevin
Sent: Tuesday, August 05, 2014 7:49 PM

Quoted 26 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:40 PM

Tian, Kevin wrote on 2014-08-06:

Quoted 22 lines
Quoted 22 lines

From: Zhang, Yang Z
Sent: Tuesday, August 05, 2014 7:11 PM

Tian, Kevin wrote on 2014-08-05:

Quoted 18 lines
Quoted 18 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Tuesday, August 05, 2014 12:52 AM

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines · On 05.08.14 at 09:35, <[email protected]> wrote:

Tian, Kevin wrote on 2014-08-05: From: Jan Beulich
[mailto:[email protected]]
Sent: Monday, August 04, 2014 12:35 AM

On 04.08.14 at 07:05, <[email protected]> wrote:

Jan Beulich wrote on 2014-07-28:
[email protected]; [email protected]
Subject: Re: [PATCH v1 0/2] Extend ioreq-server to
support page write protection

On 28.07.14 at 19:55, <[email protected]> wrote:
ioreq-server is proposed to forward PIO and MMIO

request

Quoted 8 lines

to

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

multiple device models according to the io range. XenGT
(Intel Graphics Virtualization technology, please refer to
https://01.org/xen/blogs/srclarkx/2013/graphics-virtual iz
at io n- xengt) driver reside in Dom0 as a virtual graphics
device model also need to trap and emulate the guest's

write

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

operation to some specific memory pages, like the

memory

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

pages used by guest graphics driver as PPGTT(per-process
graphics translation table). We add an new p2m type
"p2m_ram_wp" to trap the page write operation. Page of

this

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

new p2m type is read only and for write, the request will

go

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

to device model via

ioreq-server.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

So how is this write-protection supposed to work on the
IOMMU side when sharing page tables?

Thanks for pointing out this question. Write-protection is not
supposed to work when sharing page tables between EPT

and

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

vt-d. An explicit command line "iommu=no-sharept" should

be

Quoted 54 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

setted for avoiding undesirable iommu fault.

Requiring the unconditional use of a specific command line
option is certainly fine for experimental code, but not beyond

that.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

Behavior should be correct by default in production code.

But what's worse here: The option _allows_ device side
writes from the guest. Why would device side writes be
okay, but CPU side ones not?

right, whether ept is shared or not doesn't address the concern here. In both cases we need maintain the same p2m view

between

Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 13 lines
Quoted 13 lines

CPU and device side, otherwise it's broken...

One option is to treat wp similar to logdirty, i.e. exclusive to VT-d device assignment, until in the future VT-d supports page fault. We can provide a boot option to override the default policy if user thinks OK.

2nd option, like Wei mentioned in another mail, is to treat such write protected PPGTT page tables as MMIO. new hypercall is required to change the p2m type between p2m_io and p2m_ram, based on allocation/free of guest page table. This way may impact performance on read though.

Quoted 4 lines
Quoted 1 line

Comments?

Another solution is using the EPT misconfiguration mechanism like what Xen does for MTRR emulation currently.

That would cause faults on reads as well, making it necessary to emulate them.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Then it's same effect of p2m_mmio_dm.

p2m_mmio_dm will break VT-d but EPT misconfiguration doesn't.

I don't understand. Treating it as emulated io just means no valid p2m entry in EPT/VT-d. Note XenGT itself doesn't require VT-d, while you can still assign other devices this way.

From guest's view, it doesn't know the page is not valid in EPT/VT-d table. So if guest CPU touch this page, then hypervisor will intercept the access from EPT violation and handle it. But if guest uses that page as DMA buffer (a malicious guest may do it), then VT-d fault happens. Since DMA is not restartable, guest will never receive the right data.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

as you said, it's malicious guest, that's why we need zap the VT-d

Malicious guest is just an example. We can assume a normal guest never

does it.

What!? No you can't. That is thundering towards an XSA! The normal guest can become compromised.

Quoted 7 lines

It would be a severe security hole to write-protect only the CPU path while leaving DMA writes valid. When we say a page is write-protected, it must be consistently enforced in both CPU/DMA paths. Now there is no other way to achieve it lacking of VT-d page fault, except treating it fully as a emulated MMIO range (means removing mapping in both EPT/VT-d page table), or excluding VT-d like this patch originally does.

Two corrections:

- it's still a functional problem instead of security problem. with EPT/VT-d a malicious VM anyway can only access its own memory.

- A general write-protection mechanism is not possible until VT-d page fault is there. The options just talk about XenGT specific requirement, which only cares about CPU modification to the GPU page table.

Thanks
Kevin

author
#35 lore raw

PATCH 1/2 x86: add p2m_ram_wp

In reply to Wei Ye

Add a new p2m type p2m_ram_wp. Page of p2m_ram_wp is read only, and write will go to the device model for emulation.

---
 xen/arch/x86/hvm/hvm.c    |    8 +++++++-
 xen/arch/x86/mm/p2m-ept.c |    1 +
 xen/include/asm-x86/p2m.h |    8 +++++++-
 3 files changed, 15 insertions(+), 2 deletions(-)
xen/arch/x86/hvm/hvm.c +7 -1
index 17ff011..0f20b62 100644
@@ -2738,7 +2738,8 @@ int hvm_hap_nested_page_fault(paddr_t gpa,
      * If this GFN is emulated MMIO or marked as read-only, pass the fault
      * to the mmio handler.
      */
-    if ( (p2mt == p2m_mmio_dm) || 
+    if ( (p2mt == p2m_mmio_dm) ||
+         (p2mt == p2m_ram_wp) || 
          (access_w && (p2mt == p2m_ram_ro)) )
     {
         put_gfn(p2m->domain, gfn);
@@ -3829,6 +3830,11 @@ static enum hvm_copy_result __hvm_copy(
             put_page(page);
             return HVMCOPY_unhandleable;
         }
+        if ( p2m_is_wp_ram(p2mt) )
+        {
+            put_page(page);
+            return HVMCOPY_bad_gfn_to_mfn;
+        }
 
         p = (char *)__map_domain_page(page) + (addr & ~PAGE_MASK);
 
xen/arch/x86/mm/p2m-ept.c +1 -0
index 15c6e83..1b4a83e 100644
@@ -125,6 +125,7 @@ static void ept_p2m_type_to_flags(ept_entry_t *entry, p2m_type_t type, p2m_acces
                                                     entry->mfn);
             break;
         case p2m_ram_logdirty:
+        case p2m_ram_wp:
         case p2m_ram_ro:
         case p2m_ram_shared:
             entry->r = entry->x = 1;
xen/include/asm-x86/p2m.h +7 -1
index 0ddbadb..bbec847 100644
@@ -71,6 +71,7 @@ typedef enum {
     p2m_ram_shared = 12,          /* Shared or sharable memory */
     p2m_ram_broken = 13,          /* Broken page, access cause domain crash */
     p2m_map_foreign  = 14,        /* ram pages from foreign domain */
+    p2m_ram_wp = 15,              /* Read-only; write go to the device model */
 } p2m_type_t;
 
 /*
@@ -116,7 +117,8 @@ typedef unsigned int p2m_query_t;
                        | p2m_to_mask(p2m_ram_paging_out)      \
                        | p2m_to_mask(p2m_ram_paged)           \
                        | p2m_to_mask(p2m_ram_paging_in)       \
-                       | p2m_to_mask(p2m_ram_shared))
+                       | p2m_to_mask(p2m_ram_shared)          \
+                       | p2m_to_mask(p2m_ram_wp))
 
 /* Types that represent a physmap hole that is ok to replace with a shared
  * entry */
@@ -167,6 +169,9 @@ typedef unsigned int p2m_query_t;
  * and must not be touched. */
 #define P2M_BROKEN_TYPES (p2m_to_mask(p2m_ram_broken))
 
+/* Write protection types */
+#define P2M_WP_TYPES (p2m_to_mask(p2m_ram_wp))
+
 /* Useful predicates */
 #define p2m_is_ram(_t) (p2m_to_mask(_t) & P2M_RAM_TYPES)
 #define p2m_is_hole(_t) (p2m_to_mask(_t) & P2M_HOLE_TYPES)
@@ -191,6 +196,7 @@ typedef unsigned int p2m_query_t;
 #define p2m_is_any_ram(_t)  (p2m_to_mask(_t) &                   \
                              (P2M_RAM_TYPES | P2M_GRANT_TYPES |  \
                               p2m_to_mask(p2m_map_foreign)))
+#define p2m_is_wp_ram(_t)   (p2m_to_mask(_t) & P2M_WP_TYPES)
 
 /* Per-p2m-table state */
 struct p2m_domain {
Signature · 1 line
1.7.9.5
#36 lore raw

Re: [PATCH v1 1/2] x86: add p2m_ram_wp

Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:
 xen/arch/x86/hvm/hvm.c    |    8 +++++++-
 xen/arch/x86/mm/p2m-ept.c |    1 +
 xen/include/asm-x86/p2m.h |    8 +++++++-
 3 files changed, 15 insertions(+), 2 deletions(-)

This can't be complete: At the very least p2m-pt.c also needs a change similar to the one to p2m-ept.c.

xen/arch/x86/hvm/hvm.c int hvm_hap_nested_page_fault(paddr_t gpa, · line 2743

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +2 -1
@@ -2738,7 +2738,8 @@ int hvm_hap_nested_page_fault(paddr_t gpa,
      * If this GFN is emulated MMIO or marked as read-only, pass the fault
      * to the mmio handler.
      */
-    if ( (p2mt == p2m_mmio_dm) || 
+    if ( (p2mt == p2m_mmio_dm) ||
+         (p2mt == p2m_ram_wp) || 
          (access_w && (p2mt == p2m_ram_ro)) )

Comparing your change with the surrounding existing code, you would pass even reads happening to fault (perhaps for another reason) to the DM, other than done for p2m_ram_ro. I don't think that's correct.

xen/arch/x86/hvm/hvm.c static enum hvm_copy_result __hvm_copy( · line 3837

Quoted 9 lines
xen/arch/x86/hvm/hvm.c +5 -0
@@ -3829,6 +3830,11 @@ static enum hvm_copy_result __hvm_copy(
             put_page(page);
             return HVMCOPY_unhandleable;
         }
+        if ( p2m_is_wp_ram(p2mt) )
+        {
+            put_page(page);
+            return HVMCOPY_bad_gfn_to_mfn;
+        }

And again this change can't be complete: __hvm_clear() would also need a similar change.

Quoted 11 lines
xen/arch/x86/hvm/hvm.c +3 -0
@@ -167,6 +169,9 @@ typedef unsigned int p2m_query_t;
  * and must not be touched. */
 #define P2M_BROKEN_TYPES (p2m_to_mask(p2m_ram_broken))
 
+/* Write protection types */
+#define P2M_WP_TYPES (p2m_to_mask(p2m_ram_wp))
+
 /* Useful predicates */
 #define p2m_is_ram(_t) (p2m_to_mask(_t) & P2M_RAM_TYPES)
 #define p2m_is_hole(_t) (p2m_to_mask(_t) & P2M_HOLE_TYPES)

xen/arch/x86/hvm/hvm.c typedef unsigned int p2m_query_t; · line 199

Quoted 4 lines
xen/arch/x86/hvm/hvm.c +1 -0
@@ -191,6 +196,7 @@ typedef unsigned int p2m_query_t;
 #define p2m_is_any_ram(_t)  (p2m_to_mask(_t) &                   \
                              (P2M_RAM_TYPES | P2M_GRANT_TYPES |  \
                               p2m_to_mask(p2m_map_foreign)))
+#define p2m_is_wp_ram(_t)   (p2m_to_mask(_t) & P2M_WP_TYPES)

To me such single-type classes don't seem very useful. Agreed there are a number of pre-existing ones, but for classes where future extensions are rather hard to imagine I wouldn't needlessly add classes right away. But Tim (whom you failed to Cc anyway) will have the final say here.

Jan

author
#37 lore raw

Re: [PATCH v1 1/2] x86: add p2m_ram_wp

Quoted 11 lines

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Monday, July 28, 2014 4:32 PM
To: Ye, Wei
Cc: [email protected]; [email protected]; [email protected]; [email protected]; xen- [email protected]; [email protected]; Tim Deegan
Subject: Re: [PATCH v1 1/2] x86: add p2m_ram_wp

Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:
 xen/arch/x86/hvm/hvm.c    |    8 +++++++-
 xen/arch/x86/mm/p2m-ept.c |    1 +
 xen/include/asm-x86/p2m.h |    8 +++++++-
 3 files changed, 15 insertions(+), 2 deletions(-)

This can't be complete: At the very least p2m-pt.c also needs a change similar to the one to p2m-ept.c.

Yes, got it.

Quoted 17 lines

xen/arch/x86/hvm/hvm.c int hvm_hap_nested_page_fault(paddr_t gpa, · line 2743

Quoted 11 lines
xen/arch/x86/hvm/hvm.c +2 -1
@@ -2738,7 +2738,8 @@ int hvm_hap_nested_page_fault(paddr_t gpa,
      * If this GFN is emulated MMIO or marked as read-only, pass the fault
      * to the mmio handler.
      */
-    if ( (p2mt == p2m_mmio_dm) ||
+    if ( (p2mt == p2m_mmio_dm) ||
+         (p2mt == p2m_ram_wp) ||
          (access_w && (p2mt == p2m_ram_ro)) )

Comparing your change with the surrounding existing code, you would pass even reads happening to fault (perhaps for another reason) to the DM, other than done for p2m_ram_ro. I don't think that's correct.

Yes, should also access_w like the p2m_ram_ro.

Quoted 4 lines
Quoted 4 lines
xen/arch/x86/hvm/hvm.c +0 -0
@@ -3829,6 +3830,11 @@ static enum hvm_copy_result __hvm_copy(
             put_page(page);
             return HVMCOPY_unhandleable;
         }
Quoted 9 lines

xen/arch/x86/hvm/hvm.c static enum hvm_copy_result __hvm_copy( · line 3837

Quoted 5 lines
xen/arch/x86/hvm/hvm.c +5 -0
+        if ( p2m_is_wp_ram(p2mt) )
+        {
+            put_page(page);
+            return HVMCOPY_bad_gfn_to_mfn;
+        }

And again this change can't be complete: __hvm_clear() would also need a similar change.

Got it and thanks.

Quoted 13 lines
Quoted 13 lines
xen/arch/x86/hvm/hvm.c +3 -0
@@ -167,6 +169,9 @@ typedef unsigned int p2m_query_t;
  * and must not be touched. */
 #define P2M_BROKEN_TYPES (p2m_to_mask(p2m_ram_broken))
 
+/* Write protection types */
+#define P2M_WP_TYPES (p2m_to_mask(p2m_ram_wp))
+
 /* Useful predicates */
 #define p2m_is_ram(_t) (p2m_to_mask(_t) & P2M_RAM_TYPES)
 #define p2m_is_hole(_t) (p2m_to_mask(_t) & P2M_HOLE_TYPES)
@@ -191,6 +196,7 @@ typedef unsigned int p2m_query_t;
 #define p2m_is_any_ram(_t)  (p2m_to_mask(_t) &                   \
                              (P2M_RAM_TYPES | P2M_GRANT_TYPES |  \
Quoted 9 lines

xen/arch/x86/hvm/hvm.c typedef unsigned int p2m_query_t; · line 199

Quoted 2 lines
xen/arch/x86/hvm/hvm.c +1 -0
                               p2m_to_mask(p2m_map_foreign)))
+#define p2m_is_wp_ram(_t)   (p2m_to_mask(_t) & P2M_WP_TYPES)

To me such single-type classes don't seem very useful. Agreed there are a number of pre-existing ones, but for classes where future extensions are rather hard to imagine I wouldn't needlessly add classes right away. But Tim (whom you failed to Cc anyway) will have the final say here.

How about using the existing p2m_mmio_dm, in that way, both read & write access will go to device model for emulation.

Quoted 1 line

Jan

#38 lore raw

Re: [PATCH v1 1/2] x86: add p2m_ram_wp

Quoted 25 lines
Quoted 16 lines
Quoted 1 line · On 04.08.14 at 07:10, <[email protected]> wrote:

From: Jan Beulich [mailto:[email protected]]

Quoted 14 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:
xen/include/asm-x86/p2m.h +3 -0
@@ -167,6 +169,9 @@ typedef unsigned int p2m_query_t;
  * and must not be touched. */
 #define P2M_BROKEN_TYPES (p2m_to_mask(p2m_ram_broken))
 
+/* Write protection types */
+#define P2M_WP_TYPES (p2m_to_mask(p2m_ram_wp))
+
 /* Useful predicates */
 #define p2m_is_ram(_t) (p2m_to_mask(_t) & P2M_RAM_TYPES)
 #define p2m_is_hole(_t) (p2m_to_mask(_t) & P2M_HOLE_TYPES)
@@ -191,6 +196,7 @@ typedef unsigned int p2m_query_t;
 #define p2m_is_any_ram(_t)  (p2m_to_mask(_t) &                   \
                              (P2M_RAM_TYPES | P2M_GRANT_TYPES |  \
Quoted 8 lines

xen/include/asm-x86/p2m.h typedef unsigned int p2m_query_t; · line 199

Quoted 2 lines
xen/include/asm-x86/p2m.h +1 -0
                               p2m_to_mask(p2m_map_foreign)))
+#define p2m_is_wp_ram(_t)   (p2m_to_mask(_t) & P2M_WP_TYPES)

To me such single-type classes don't seem very useful. Agreed there are a number of pre-existing ones, but for classes where future extensions are rather hard to imagine I wouldn't needlessly add classes right away. But Tim (whom you failed to Cc anyway) will have the final say here.

Quoted 2 lines

How about using the existing p2m_mmio_dm, in that way, both read & write access will go to device model for emulation.

Perhaps you didn't read what I wrote the way I intended it: I'm not opposed to the new P2M type. What I dislike is the single type class that you introduce. To check for that in the code, you could as well use direct comparisons with p2m_ram_wp.

Jan

author
#39 lore raw

Re: [PATCH v1 1/2] x86: add p2m_ram_wp

Quoted 38 lines

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 4, 2014 3:38 PM
To: Ye, Wei
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Tian, Kevin; Lv, Zhiyuan; [email protected]; [email protected]; Tim Deegan
Subject: RE: [PATCH v1 1/2] x86: add p2m_ram_wp

Quoted 7 lines
Quoted 7 lines
Quoted 1 line · On 04.08.14 at 07:10, <[email protected]> wrote:

From: Jan Beulich [mailto:[email protected]]

Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:
xen/include/asm-x86/p2m.h +0 -0
@@ -167,6 +169,9 @@ typedef unsigned int p2m_query_t;
  * and must not be touched. */
 #define P2M_BROKEN_TYPES (p2m_to_mask(p2m_ram_broken))
 
Quoted 5 lines
Quoted 5 lines

xen/include/asm-x86/p2m.h typedef unsigned int p2m_query_t; · line 176

Quoted 5 lines
xen/include/asm-x86/p2m.h +3 -0
+/* Write protection types */
+#define P2M_WP_TYPES (p2m_to_mask(p2m_ram_wp))
+
 /* Useful predicates */
 #define p2m_is_ram(_t) (p2m_to_mask(_t) & P2M_RAM_TYPES)

#define

Quoted 1 line
Quoted 1 line
Quoted 1 line

p2m_is_hole(_t) (p2m_to_mask(_t) & P2M_HOLE_TYPES) @@ -191,6

+196,7

Quoted 12 lines
Quoted 11 lines
Quoted 5 lines

@@ typedef unsigned int p2m_query_t;

 #define p2m_is_any_ram(_t)  (p2m_to_mask(_t) &                   \
                              (P2M_RAM_TYPES | P2M_GRANT_TYPES |  \
                               p2m_to_mask(p2m_map_foreign)))

+#define p2m_is_wp_ram(_t) (p2m_to_mask(_t) & P2M_WP_TYPES)

To me such single-type classes don't seem very useful. Agreed there are a number of pre-existing ones, but for classes where future extensions are rather hard to imagine I wouldn't needlessly add classes right away. But Tim (whom you failed to Cc anyway) will have the final say here.

Quoted 7 lines
Quoted 2 lines

How about using the existing p2m_mmio_dm, in that way, both read & write access will go to device model for emulation.

Perhaps you didn't read what I wrote the way I intended it: I'm not opposed to the new P2M type. What I dislike is the single type class that you introduce. To check for that in the code, you could as well use direct comparisons with p2m_ram_wp.

Sorry for my misunderstanding. Agree that this single class may be unnecessary. So I'll remove this class type and compare it directly with p2m_ram_wp. But another option maybe we can remove this new introduced p2m type, since it doesn't work well when page table sharing with IOMMU case. And just using the existing p2m type "p2m_mmio_dm".

Quoted 2 lines

Jan

author
#40 lore raw

PATCH 2/2 ioreq-server: Support scatter page forwarding

In reply to Wei Ye

Extend the interface to support add/remove scatter page list to be forwarded by a dedicated ioreq-server instance. Check and select a right ioreq-server instance for forwarding the wirte operation for a write protected page.

---
 tools/libxc/xc_domain.c          |   32 ++++++
 tools/libxc/xenctrl.h            |   18 ++++
 xen/arch/x86/hvm/hvm.c           |  206 +++++++++++++++++++++++++++++++++++++-
 xen/include/asm-x86/hvm/domain.h |   14 +++
 xen/include/public/hvm/hvm_op.h  |   12 +++
 5 files changed, 281 insertions(+), 1 deletion(-)
tools/libxc/xc_domain.c +32 -0
index 37ed141..9afc843 100644
@@ -1485,6 +1485,38 @@ int xc_hvm_unmap_pcidev_from_ioreq_server(xc_interface *xch, domid_t domid,
     return rc;
 }
 
+int xc_hvm_map_pages_to_ioreq_server(xc_interface *xch, domid_t domid,
+                                     ioservid_t id, uint16_t set,
+                                     uint16_t nr_pages, unsigned long *gpfn)
+{
+    DECLARE_HYPERCALL;
+    DECLARE_HYPERCALL_BUFFER(xen_hvm_map_pages_to_ioreq_server_t, arg);
+    int pg, rc = -1;
+
+    if ( arg == NULL )
+        return -1;
+    
+    if (nr_pages > MAX_MAP_BATCH_PAGES)
+        goto out;
+
+    hypercall.op     = __HYPERVISOR_hvm_op;
+    hypercall.arg[0] = HVMOP_map_pages_to_ioreq_server;
+    hypercall.arg[1] = HYPERCALL_BUFFER_AS_ARG(arg);
+
+    arg->domid = domid;
+    arg->id = id;
+    arg->set = set;
+    arg->nr_pages = nr_pages;
+    for ( pg = 0; pg < nr_pages; pg++ )
+        arg->page_list[pg] = gpfn[pg];
+
+    rc = do_xen_hypercall(xch, &hypercall);
+
+out:
+    xc_hypercall_buffer_free(xch, arg);
+    return rc;
+}
+
 int xc_hvm_destroy_ioreq_server(xc_interface *xch,
                                 domid_t domid,
                                 ioservid_t id)
tools/libxc/xenctrl.h +18 -0
index b55d857..e5e37b0 100644
@@ -1943,6 +1943,24 @@ int xc_hvm_unmap_pcidev_from_ioreq_server(xc_interface *xch,
                                           uint8_t function);
 
 /**
+ * This function registers/deregisters a set of pages to be write protected.
+ *
+ * @parm xch a handle to an open hypervisor interface.
+ * @parm domid the domain id to be serviced
+ * @parm id the IOREQ Server id.
+ * @parm set whether registers or deregisters: 1 for register, 0 for deregister
+ * @parm nr_pages the number of pages
+ * @parm pgfn the guest page frame number list
+ * @return 0 on success, -1 on failure.
+ */
+int xc_hvm_map_pages_to_ioreq_server(xc_interface *xch,
+                                          domid_t domid,
+                                          ioservid_t id,
+                                          uint16_t set,
+                                          uint16_t nr_pages,
+                                          unsigned long *gpfn);
+
+/**
  * This function destroys an IOREQ Server.
  *
  * @parm xch a handle to an open hypervisor interface.
xen/arch/x86/hvm/hvm.c +205 -1

255 lines not on this page · raw message

xen/include/asm-x86/hvm/domain.h +14 -0
index 291a2e0..47172ca 100644
@@ -48,6 +48,16 @@ struct hvm_ioreq_vcpu {
     evtchn_port_t    ioreq_evtchn;
 };
 
+struct pglist_hash_table {
+    struct pglist_hash_table *next;
+    unsigned long gpfn;
+};
+#define PGLIST_HASH_SIZE_SHIFT    8
+#define PGLIST_HASH_SIZE    (1 << PGLIST_HASH_SIZE_SHIFT)
+#define pglist_hash(x)      ((x) % PGLIST_HASH_SIZE)
+#define PGLIST_INVALID_GPFN       0
+#define PGLIST_HASH_ENTRY_SIZE    sizeof(struct pglist_hash_table)
+
 #define NR_IO_RANGE_TYPES (HVMOP_IO_RANGE_PCI + 1)
 #define MAX_NR_IO_RANGES  256
 
@@ -70,6 +80,10 @@ struct hvm_ioreq_server {
     evtchn_port_t          bufioreq_evtchn;
     struct rangeset        *range[NR_IO_RANGE_TYPES];
     bool_t                 enabled;
+
+    /* scatter page list need write protection */
+    struct pglist_hash_table   *pglist_hash_base;
+    spinlock_t             pglist_hash_lock;
 };
 
 struct hvm_domain {
xen/include/public/hvm/hvm_op.h +12 -0
index eeb0a60..53a966b 100644
@@ -367,6 +367,18 @@ struct xen_hvm_set_ioreq_server_state {
 typedef struct xen_hvm_set_ioreq_server_state xen_hvm_set_ioreq_server_state_t;
 DEFINE_XEN_GUEST_HANDLE(xen_hvm_set_ioreq_server_state_t);
 
+#define HVMOP_map_pages_to_ioreq_server 23
+#define MAX_MAP_BATCH_PAGES             128
+struct xen_hvm_map_pages_to_ioreq_server {
+    domid_t domid;   /* IN - domain to be serviced */
+    ioservid_t id;   /* IN - server id */
+    uint16_t set;    /* IN - 1: map pages, 0: unmap pages */
+    uint16_t nr_pages;  /* IN - page number */
+    unsigned long page_list[MAX_MAP_BATCH_PAGES]; /* IN - gpfn list */
+};
+typedef struct xen_hvm_map_pages_to_ioreq_server xen_hvm_map_pages_to_ioreq_server_t;
+DEFINE_XEN_GUEST_HANDLE(xen_hvm_map_pages_to_ioreq_server_t);
+
 #endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
 
 #endif /* __XEN_PUBLIC_HVM_HVM_OP_H__ */
Signature · 1 line
1.7.9.5
#41 lore raw

Re: [PATCH v1 2/2] ioreq-server: Support scatter page forwarding

xen/arch/x86/hvm/hvm.c static void hvm_ioreq_server_remove_all_vcpus(struct hvm_ioreq_server *s) · line 750

Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:
xen/arch/x86/hvm/hvm.c +4 -0
+static struct pglist_hash_table *hvm_ioreq_server_lookup_pglist_hash_table(
+                          struct pglist_hash_table *pglist_ht, unsigned long gpfn)
+{
+    int index = pglist_hash(gpfn);

Here and elsewhere: "unsigned int" for the hash index.

xen/arch/x86/hvm/hvm.c static void hvm_ioreq_server_remove_all_vcpus(struct hvm_ioreq_server *s) · line 765

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +6 -0
+static void hvm_ioreq_server_free_hash_chain(struct pglist_hash_table *chain)
+{
+    struct pglist_hash_table *p = chain;
+    struct pglist_hash_table *n;
+
+    while (p)

Coding style.

Quoted 4 lines
xen/arch/x86/hvm/hvm.c +4 -0
+static int hvm_ioreq_server_pglist_hash_add(struct pglist_hash_table *pglist_ht,
+                                            unsigned long gpfn)
+{
+    int index = pglist_hash(gpfn);

xen/arch/x86/hvm/hvm.c static void hvm_ioreq_server_remove_all_vcpus(struct hvm_ioreq_server *s) · line 795

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +10 -0
+    struct pglist_hash_table *ne;
+
+    if ( hvm_ioreq_server_lookup_pglist_hash_table(pglist_ht, gpfn) != NULL )
+        return -EINVAL;
+
+    if ( pglist_ht[index].gpfn == PGLIST_INVALID_GPFN )
+        pglist_ht[index].gpfn = gpfn;
+    else
+    {
+        ne = xmalloc_bytes(sizeof(pglist_ht[0]));

xmalloc() please.

Quoted 8 lines
xen/arch/x86/hvm/hvm.c +8 -0
+static int hvm_ioreq_server_pglist_hash_rem(struct pglist_hash_table *pglist_ht,
+                                            unsigned long gpfn)
+{
+    int index = pglist_hash(gpfn);
+    struct pglist_hash_table *next, *prev;
+
+    if ( pglist_ht[index].gpfn == gpfn )
+        pglist_ht[index].gpfn = PGLIST_INVALID_GPFN;

xen/arch/x86/hvm/hvm.c static void hvm_ioreq_server_remove_all_vcpus(struct hvm_ioreq_server *s) · line 823

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +10 -0
+    else
+    {
+        prev = &pglist_ht[index];
+        while ( 1 )
+        {
+            next = prev->next;
+            if ( !next )
+            {
+                printk("hvm_ioreq_server_pglist_hash_rem hash_table %p remove"
+                       " %lx not found\n", pglist_ht, gpfn);

The value of this log message is questionable anyway, but there's no way for this to be acceptable without a suitably low guest level being set.

xen/arch/x86/hvm/hvm.c static int hvm_ioreq_server_init(struct · line 1047

Quoted 9 lines
xen/arch/x86/hvm/hvm.c +3 -0
@@ -948,7 +1040,14 @@ static int hvm_ioreq_server_init(struct 
 hvm_ioreq_server *s, struct domain *d,
     spin_lock_init(&s->lock);
     INIT_LIST_HEAD(&s->ioreq_vcpu_list);
     spin_lock_init(&s->bufioreq_lock);
+    spin_lock_init(&s->pglist_hash_lock);
 
+    rc = -ENOMEM;
+    s->pglist_hash_base = xmalloc_bytes(PGLIST_HASH_ENTRY_SIZE * PGLIST_HASH_SIZE);

xmalloc_array() please.

xen/arch/x86/hvm/hvm.c static int hvm_ioreq_server_init(struct hvm_ioreq_server *s, struct domain *d, · line 1049

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +3 -0
+    if ( s->pglist_hash_base == NULL )
+        goto fail1;
+    memset(s->pglist_hash_base, 0, PGLIST_HASH_ENTRY_SIZE * PGLIST_HASH_SIZE);

And together with this, xzalloc_array().

Quoted 22 lines
xen/arch/x86/hvm/hvm.c +22 -0
+static int hvm_map_pages_to_ioreq_server(struct domain *d, ioservid_t id,
+                                        uint16_t set, uint16_t nr_pages,
+                                        unsigned long *gpfn)
+{
+    struct hvm_ioreq_server *s;
+    int rc;
+    unsigned int i;
+    p2m_type_t ot, nt;
+
+    if ( set )
+    {
+        ot = p2m_ram_rw;
+        nt = p2m_ram_wp;
+    }
+    else
+    {
+        ot = p2m_ram_wp;
+        nt = p2m_ram_rw;
+    }
+
+    spin_lock(&d->arch.hvm_domain.ioreq_server.lock);
+

xen/arch/x86/hvm/hvm.c static int hvm_set_ioreq_server_state(struct domain *d, ioservid_t id, · line 1415

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +10 -0
+    rc = -ENOENT;
+    list_for_each_entry ( s,
+                          &d->arch.hvm_domain.ioreq_server.list,
+                          list_entry )
+    {
+        if ( s == d->arch.hvm_domain.default_ioreq_server )
+            continue;
+
+        if ( s->id == id )
+        {

Consistency please: Either both if()-s use "continue" (and perhaps get combined into one), or (less desirable imo) both get combined to use a {} body.

xen/arch/x86/hvm/hvm.c static int hvm_set_ioreq_server_state(struct domain *d, ioservid_t id, · line 1420

Quoted 5 lines
xen/arch/x86/hvm/hvm.c +5 -0
+            spin_lock(&s->pglist_hash_lock);
+
+            for ( i = 0; i < nr_pages; i++ )
+            {
+                p2m_change_type_one(d, gpfn[i], ot, nt);

Error handling?

xen/arch/x86/hvm/hvm.c static int hvm_set_ioreq_server_state(struct domain *d, ioservid_t id, · line 1431

Quoted 11 lines
xen/arch/x86/hvm/hvm.c +11 -0
+                if ( set )
+                    rc = hvm_ioreq_server_pglist_hash_add(s->pglist_hash_base, gpfn[i]);
+                else
+                    rc = hvm_ioreq_server_pglist_hash_rem(s->pglist_hash_base, gpfn[i]);
+
+                 if ( rc != 0 )
+                     break;
+            }
+
+            spin_unlock(&s->pglist_hash_lock);
+            flush_tlb_mask(d->domain_dirty_cpumask);

Why?

xen/arch/x86/hvm/hvm.c static struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d, · line 2457

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +2 -0
@@ -2294,6 +2453,8 @@ static struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d,
     uint32_t cf8;
     uint8_t type;
     uint64_t addr;
+    unsigned long gpfn;
+    struct pglist_hash_table *he;

These ought to be moved down into a more narrow scope.

xen/arch/x86/hvm/hvm.c static struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d, · line 2530

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +6 -0
@@ -2361,6 +2522,15 @@ static struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d,
             if ( rangeset_contains_range(r, addr, end) )
                 return s;
 
+            gpfn = addr >> PAGE_SHIFT;
+            spin_lock(&s->pglist_hash_lock);
+            he = hvm_ioreq_server_lookup_pglist_hash_table(s->pglist_hash_base, gpfn);
+            spin_unlock(&s->pglist_hash_lock);
+
+            if ( he ) {

Coding style again.

xen/arch/x86/hvm/hvm.c static struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d, · line 2533

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +3 -0
+                return s;
+            }
+

And now the most fundamental question: Why is the existing (range set based) mechanism not enough? Is that because we limit the number of entries permitted on the range set? If so, the obvious question is - why is there no limit being enforced for your new type of pages?

Quoted 5 lines
xen/include/asm-x86/hvm/domain.h +0 -0
@@ -48,6 +48,16 @@ struct hvm_ioreq_vcpu {
     evtchn_port_t    ioreq_evtchn;
 };

xen/include/asm-x86/hvm/domain.h struct hvm_ioreq_vcpu { · line 59

Quoted 10 lines
xen/include/asm-x86/hvm/domain.h +9 -0
 
+struct pglist_hash_table {
+    struct pglist_hash_table *next;
+    unsigned long gpfn;
+};
+#define PGLIST_HASH_SIZE_SHIFT    8
+#define PGLIST_HASH_SIZE    (1 << PGLIST_HASH_SIZE_SHIFT)
+#define pglist_hash(x)      ((x) % PGLIST_HASH_SIZE)
+#define PGLIST_INVALID_GPFN       0
+#define PGLIST_HASH_ENTRY_SIZE    sizeof(struct pglist_hash_table)

None of this has any point in being in a header: There's only a single source file using these, so they should be private to that one.

xen/include/public/hvm/hvm_op.h struct xen_hvm_set_ioreq_server_state { · line 371

Quoted 8 lines
xen/include/public/hvm/hvm_op.h +2 -0
@@ -367,6 +367,18 @@ struct xen_hvm_set_ioreq_server_state {
 typedef struct xen_hvm_set_ioreq_server_state xen_hvm_set_ioreq_server_state_t;
 DEFINE_XEN_GUEST_HANDLE(xen_hvm_set_ioreq_server_state_t);
 
+#define HVMOP_map_pages_to_ioreq_server 23
+#define MAX_MAP_BATCH_PAGES             128

Too generic a name.

xen/include/public/hvm/hvm_op.h struct xen_hvm_set_ioreq_server_state { · line 376

Quoted 5 lines
xen/include/public/hvm/hvm_op.h +5 -0
+struct xen_hvm_map_pages_to_ioreq_server {
+    domid_t domid;   /* IN - domain to be serviced */
+    ioservid_t id;   /* IN - server id */
+    uint16_t set;    /* IN - 1: map pages, 0: unmap pages */
+    uint16_t nr_pages;  /* IN - page number */

"page count" or "number of pages".

xen/include/public/hvm/hvm_op.h struct xen_hvm_set_ioreq_server_state { · line 377

Quoted 1 line
xen/include/public/hvm/hvm_op.h +1 -0
+    unsigned long page_list[MAX_MAP_BATCH_PAGES]; /* IN - gpfn list */

No "unsigned long" in public interfaces. And no variable width types in interfaces not being subject to compat mode translation.

Jan

author
#42 lore raw

Re: [PATCH v1 2/2] ioreq-server: Support scatter page forwarding

Quoted 20 lines

-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Monday, July 28, 2014 4:58 PM
To: Ye, Wei
Cc: [email protected]; [email protected]; [email protected]; [email protected]; xen- [email protected]; [email protected]
Subject: Re: [PATCH v1 2/2] ioreq-server: Support scatter page forwarding

Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

+static struct pglist_hash_table

*hvm_ioreq_server_lookup_pglist_hash_table(

Review

Quoted 3 lines
(quoted diff) +3 -0
+                          struct pglist_hash_table *pglist_ht,
+unsigned long gpfn) {
+    int index = pglist_hash(gpfn);

Here and elsewhere: "unsigned int" for the hash index.

Quoted 8 lines

xen/arch/x86/hvm/hvm.c static void hvm_ioreq_server_remove_all_vcpus(struct hvm_ioreq_server *s) · line 765

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +6 -0
+static void hvm_ioreq_server_free_hash_chain(struct pglist_hash_table
+*chain) {
+    struct pglist_hash_table *p = chain;
+    struct pglist_hash_table *n;
+
+    while (p)

Coding style.

Will improve, thanks.

Quoted 17 lines
Quoted 2 lines

+static int hvm_ioreq_server_pglist_hash_add(struct pglist_hash_table

*pglist_ht,

xen/arch/x86/hvm/hvm.c static void hvm_ioreq_server_remove_all_vcpus(struct hvm_ioreq_server *s)

Quoted 5 lines
xen/arch/x86/hvm/hvm.c +5 -0
+                                            unsigned long gpfn) {
+    int index = pglist_hash(gpfn);
+    struct pglist_hash_table *ne;
+
+    if ( hvm_ioreq_server_lookup_pglist_hash_table(pglist_ht, gpfn) !=

NULL )

xen/arch/x86/hvm/hvm.c static void hvm_ioreq_server_remove_all_vcpus(struct hvm_ioreq_server *s) · line 795

Quoted 7 lines
xen/arch/x86/hvm/hvm.c +7 -0
+        return -EINVAL;
+
+    if ( pglist_ht[index].gpfn == PGLIST_INVALID_GPFN )
+        pglist_ht[index].gpfn = gpfn;
+    else
+    {
+        ne = xmalloc_bytes(sizeof(pglist_ht[0]));
Quoted 1 line

xmalloc() please.

Ok.

Quoted 18 lines
Quoted 2 lines

+static int hvm_ioreq_server_pglist_hash_rem(struct pglist_hash_table

*pglist_ht,

Quoted 5 lines
xen/arch/x86/hvm/hvm.c +5 -0
+                                            unsigned long gpfn) {
+    int index = pglist_hash(gpfn);
+    struct pglist_hash_table *next, *prev;
+
+    if ( pglist_ht[index].gpfn == gpfn )

xen/arch/x86/hvm/hvm.c static void hvm_ioreq_server_remove_all_vcpus(struct hvm_ioreq_server *s)

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +10 -0
+        pglist_ht[index].gpfn = PGLIST_INVALID_GPFN;
+    else
+    {
+        prev = &pglist_ht[index];
+        while ( 1 )
+        {
+            next = prev->next;
+            if ( !next )
+            {
+                printk("hvm_ioreq_server_pglist_hash_rem hash_table %p
Quoted 5 lines

remove"

Quoted 1 line

+ " %lx not found\n", pglist_ht, gpfn);

The value of this log message is questionable anyway, but there's no way for this to be acceptable without a suitably low guest level being set.

You mean should also print out the VM id in this log?

Quoted 14 lines

xen/arch/x86/hvm/hvm.c static int hvm_ioreq_server_init(struct · line 1048

Quoted 11 lines
xen/arch/x86/hvm/hvm.c +4 -0
@@ -948,7 +1040,14 @@ static int hvm_ioreq_server_init(struct
 hvm_ioreq_server *s, struct domain *d,
     spin_lock_init(&s->lock);
     INIT_LIST_HEAD(&s->ioreq_vcpu_list);
     spin_lock_init(&s->bufioreq_lock);
+    spin_lock_init(&s->pglist_hash_lock);
 
+    rc = -ENOMEM;
+    s->pglist_hash_base = xmalloc_bytes(PGLIST_HASH_ENTRY_SIZE *
+ PGLIST_HASH_SIZE);

xmalloc_array() please

Quoted 7 lines

xen/arch/x86/hvm/hvm.c static int hvm_ioreq_server_init(struct hvm_ioreq_server *s, struct domain *d,

Quoted 4 lines
xen/arch/x86/hvm/hvm.c +4 -0
+    if ( s->pglist_hash_base == NULL )
+        goto fail1;
+    memset(s->pglist_hash_base, 0, PGLIST_HASH_ENTRY_SIZE *
+ PGLIST_HASH_SIZE);

And together with this, xzalloc_array().

Ok.

Quoted 28 lines

Review static int hvm_ioreq_server_init(struct · line 1053

Quoted 1 line
(quoted diff) +1 -0
+static int hvm_map_pages_to_ioreq_server(struct domain *d, ioservid_t

id,

Quoted 26 lines
xen/arch/x86/hvm/hvm.c +26 -0
+                                        uint16_t set, uint16_t nr_pages,
+                                        unsigned long *gpfn) {
+    struct hvm_ioreq_server *s;
+    int rc;
+    unsigned int i;
+    p2m_type_t ot, nt;
+
+    if ( set )
+    {
+        ot = p2m_ram_rw;
+        nt = p2m_ram_wp;
+    }
+    else
+    {
+        ot = p2m_ram_wp;
+        nt = p2m_ram_rw;
+    }
+
+    spin_lock(&d->arch.hvm_domain.ioreq_server.lock);
+
+    rc = -ENOENT;
+    list_for_each_entry ( s,
+                          &d->arch.hvm_domain.ioreq_server.list,
+                          list_entry )
+    {
+        if ( s == d->arch.hvm_domain.default_ioreq_server )
Quoted 8 lines

xen/arch/x86/hvm/hvm.c static int hvm_set_ioreq_server_state(struct domain *d, ioservid_t id, · line 1415

Quoted 4 lines
xen/arch/x86/hvm/hvm.c +4 -0
+            continue;
+
+        if ( s->id == id )
+        {

Consistency please: Either both if()-s use "continue" (and perhaps get combined into one), or (less desirable imo) both get combined to use a {} body.

Ok.

Quoted 8 lines

xen/arch/x86/hvm/hvm.c static int hvm_set_ioreq_server_state(struct domain *d, ioservid_t id, · line 1420

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +5 -0
+            spin_lock(&s->pglist_hash_lock);
+
+            for ( i = 0; i < nr_pages; i++ )
+            {
+                p2m_change_type_one(d, gpfn[i], ot, nt);

Error handling?

Forget error handling, thanks for your reminding.

Quoted 4 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+                if ( set )
+                    rc = hvm_ioreq_server_pglist_hash_add(s->pglist_hash_base,

gpfn[i]);

Quoted 12 lines

xen/arch/x86/hvm/hvm.c static int hvm_set_ioreq_server_state(struct domain *d, ioservid_t id, · line 1431

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +10 -0
+                else
+                    rc =
+ hvm_ioreq_server_pglist_hash_rem(s->pglist_hash_base, gpfn[i]);
+
+                 if ( rc != 0 )
+                     break;
+            }
+
+            spin_unlock(&s->pglist_hash_lock);
+            flush_tlb_mask(d->domain_dirty_cpumask);

Why?

Will remove it.

Quoted 24 lines

Review static struct hvm_ioreq_server

Quoted 2 lines
(quoted diff) +0 -0
@@ -2294,6 +2453,8 @@ static struct hvm_ioreq_server

*hvm_select_ioreq_server(struct domain *d,

xen/arch/x86/hvm/hvm.c static struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d, · line 2457

Quoted 5 lines
xen/arch/x86/hvm/hvm.c +2 -0
     uint32_t cf8;
     uint8_t type;
     uint64_t addr;
+    unsigned long gpfn;
+    struct pglist_hash_table *he;

These ought to be moved down into a more narrow scope.

Review static struct hvm_ioreq_server

Quoted 1 line
(quoted diff) +0 -0
@@ -2361,6 +2522,15 @@ static struct hvm_ioreq_server

*hvm_select_ioreq_server(struct domain *d,

xen/arch/x86/hvm/hvm.c static struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d, · line 2530

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +6 -0
             if ( rangeset_contains_range(r, addr, end) )
                 return s;
 
+            gpfn = addr >> PAGE_SHIFT;
+            spin_lock(&s->pglist_hash_lock);
+            he = hvm_ioreq_server_lookup_pglist_hash_table(s-
 pglist_hash_base, gpfn);
+            spin_unlock(&s->pglist_hash_lock);
+
+            if ( he ) {
Quoted 6 lines

Coding style again.

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +3 -0
+                return s;
+            }
+

Ok.

Quoted 5 lines

And now the most fundamental question: Why is the existing (range set based) mechanism not enough? Is that because we limit the number of entries permitted on the range set? If so, the obvious question is - why is there no limit being enforced for your new type of pages?

In our usage, we should write protect all the pages allocated by GFX driver using as Per-process graphics translation table, This kind of pages is much more than MAX_NR_IO_RANGES(256). So we need a scatter page list for containing all the pages.

Quoted 6 lines
Quoted 6 lines
xen/include/asm-x86/hvm/domain.h +0 -0
@@ -48,6 +48,16 @@ struct hvm_ioreq_vcpu {
     evtchn_port_t    ioreq_evtchn;
 };
 
Quoted 12 lines

xen/include/asm-x86/hvm/domain.h struct hvm_ioreq_vcpu { · line 59

Quoted 9 lines
xen/include/asm-x86/hvm/domain.h +9 -0
+struct pglist_hash_table {
+    struct pglist_hash_table *next;
+    unsigned long gpfn;
+};
+#define PGLIST_HASH_SIZE_SHIFT    8
+#define PGLIST_HASH_SIZE    (1 << PGLIST_HASH_SIZE_SHIFT)
+#define pglist_hash(x)      ((x) % PGLIST_HASH_SIZE)
+#define PGLIST_INVALID_GPFN       0
+#define PGLIST_HASH_ENTRY_SIZE    sizeof(struct pglist_hash_table)

None of this has any point in being in a header: There's only a single source file using these, so they should be private to that one.

Will move to hvm.c

Quoted 5 lines

xen/include/public/hvm/hvm_op.h struct xen_hvm_set_ioreq_server_state

Quoted 4 lines
xen/include/public/hvm/hvm_op.h +0 -0
@@ -367,6 +367,18 @@ struct xen_hvm_set_ioreq_server_state

{ typedef

Quoted 8 lines

xen/include/public/hvm/hvm_op.h struct xen_hvm_set_ioreq_server_state { · line 371

Quoted 6 lines

struct xen_hvm_set_ioreq_server_state xen_hvm_set_ioreq_server_state_t; DEFINE_XEN_GUEST_HANDLE(xen_hvm_set_ioreq_server_state_t);

xen/include/public/hvm/hvm_op.h +2 -0
+#define HVMOP_map_pages_to_ioreq_server 23
+#define MAX_MAP_BATCH_PAGES             128

Too generic a name.

How about MAX_IOREQSVR_MAP_BATCH_PAGES ?

Quoted 8 lines

xen/include/public/hvm/hvm_op.h struct xen_hvm_set_ioreq_server_state { · line 376

Quoted 6 lines
xen/include/public/hvm/hvm_op.h +5 -0
+struct xen_hvm_map_pages_to_ioreq_server {
+    domid_t domid;   /* IN - domain to be serviced */
+    ioservid_t id;   /* IN - server id */
+    uint16_t set;    /* IN - 1: map pages, 0: unmap pages */
+    uint16_t nr_pages;  /* IN - page number */

"page count" or "number of pages".

Should be page count.

Quoted 7 lines
Quoted 3 lines

+ unsigned long page_list[MAX_MAP_BATCH_PAGES]; /* IN - gpfn list
+ */

No "unsigned long" in public interfaces. And no variable width types in interfaces not being subject to compat mode translation.

I'll change to use uint64_t.

Quoted 1 line

Jan

#43 lore raw

Re: [PATCH v1 2/2] ioreq-server: Support scatter page forwarding

Quoted 20 lines
Quoted 20 lines
Quoted 1 line · On 04.08.14 at 07:41, <[email protected]> wrote:

From: Jan Beulich [mailto:[email protected]]

Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

+static int hvm_ioreq_server_pglist_hash_rem(struct pglist_hash_table

*pglist_ht,

Quoted 15 lines
xen/arch/x86/hvm/hvm.c +15 -0
+                                            unsigned long gpfn) {
+    int index = pglist_hash(gpfn);
+    struct pglist_hash_table *next, *prev;
+
+    if ( pglist_ht[index].gpfn == gpfn )
+        pglist_ht[index].gpfn = PGLIST_INVALID_GPFN;
+    else
+    {
+        prev = &pglist_ht[index];
+        while ( 1 )
+        {
+            next = prev->next;
+            if ( !next )
+            {
+                printk("hvm_ioreq_server_pglist_hash_rem hash_table %p
Quoted 7 lines
Quoted 5 lines

remove"

Quoted 1 line

+ " %lx not found\n", pglist_ht, gpfn);

The value of this log message is questionable anyway, but there's no way for this to be acceptable without a suitably low guest level being set.

You mean should also print out the VM id in this log?

That's a secondary thing: Of course you need to ask yourself what use you could make of the message if you see it in some log, including when there was more than one VM running. But my point was that this lacks a suitable XENLOG_G_* prefix, thus representing a security issue (guest being able to flood the hypervisor log).

Quoted 10 lines
Quoted 5 lines

And now the most fundamental question: Why is the existing (range set based) mechanism not enough? Is that because we limit the number of entries permitted on the range set? If so, the obvious question is - why is there no limit being enforced for your new type of pages?

In our usage, we should write protect all the pages allocated by GFX driver using as Per-process graphics translation table,
This kind of pages is much more than MAX_NR_IO_RANGES(256). So we need a scatter page list for containing all the pages.

So am I reading this right that you could do with the existing mechanism if the limit was higher? Together with you not answering the question about the lack of an enforced limit in your new mechanism, this clearly tells me that you should use the existing one, raising the limit and - if necessary - dealing with the (security) fallout.

Quoted 8 lines
Quoted 8 lines

xen/include/public/hvm/hvm_op.h struct xen_hvm_set_ioreq_server_state

Quoted 3 lines
xen/include/public/hvm/hvm_op.h +0 -0
@@ -367,6 +367,18 @@ struct xen_hvm_set_ioreq_server_state

{ typedef

Quoted 4 lines

struct xen_hvm_set_ioreq_server_state xen_hvm_set_ioreq_server_state_t; DEFINE_XEN_GUEST_HANDLE(xen_hvm_set_ioreq_server_state_t);

Quoted 6 lines
Quoted 4 lines

xen/include/public/hvm/hvm_op.h struct xen_hvm_set_ioreq_server_state { · line 371

Quoted 2 lines
xen/include/public/hvm/hvm_op.h +2 -0
+#define HVMOP_map_pages_to_ioreq_server 23
+#define MAX_MAP_BATCH_PAGES             128

Too generic a name.

How about MAX_IOREQSVR_MAP_BATCH_PAGES ?

Butter, but still insufficient. XEN_IOREQSRV_MAX_MAP_BATCH_PAGES would go in the direction of being acceptable in a public header. But with the comments earlier on I suppose this will go away anyway.

Jan

#44 lore raw

Re: [PATCH v1 2/2] ioreq-server: Support scatter page forwarding

Quoted 49 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Monday, August 04, 2014 12:47 AM

Quoted 6 lines
Quoted 6 lines
Quoted 1 line · On 04.08.14 at 07:41, <[email protected]> wrote:

From: Jan Beulich [mailto:[email protected]]

Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 28.07.14 at 19:55, <[email protected]> wrote:

+static int hvm_ioreq_server_pglist_hash_rem(struct pglist_hash_table

*pglist_ht,

Quoted 1 line

+ unsigned long

gpfn) {

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
xen/arch/x86/hvm/hvm.c +6 -0
+    int index = pglist_hash(gpfn);
+    struct pglist_hash_table *next, *prev;
+
+    if ( pglist_ht[index].gpfn == gpfn )
+        pglist_ht[index].gpfn = PGLIST_INVALID_GPFN;
+    else
Quoted 8 lines
Quoted 8 lines

xen/arch/x86/hvm/hvm.c static void hvm_ioreq_server_remove_all_vcpus(struct hvm_ioreq_server *s)

Quoted 8 lines
xen/arch/x86/hvm/hvm.c +8 -0
+    {
+        prev = &pglist_ht[index];
+        while ( 1 )
+        {
+            next = prev->next;
+            if ( !next )
+            {
+                printk("hvm_ioreq_server_pglist_hash_rem

hash_table %p

Quoted 4 lines
Quoted 4 lines

remove"

Quoted 1 line

+ " %lx not found\n", pglist_ht, gpfn);

The value of this log message is questionable anyway, but there's no way

for

Quoted 3 lines
Quoted 1 line

this to be acceptable without a suitably low guest level being set.

You mean should also print out the VM id in this log?

That's a secondary thing: Of course you need to ask yourself what use you could make of the message if you see it in some log, including when there was more than one VM running. But my point was that this lacks a suitable XENLOG_G_* prefix, thus representing a security issue (guest being able to flood the hypervisor log).

Quoted 9 lines
Quoted 5 lines

And now the most fundamental question: Why is the existing (range set based) mechanism not enough? Is that because we limit the number of entries permitted on the range set? If so, the obvious question is - why is there no limit being enforced for your new type of pages?

In our usage, we should write protect all the pages allocated by GFX driver using as Per-process graphics translation table,
This kind of pages is much more than MAX_NR_IO_RANGES(256). So we

Quoted 9 lines

need a

Quoted 1 line

scatter page list for containing all the pages.

So am I reading this right that you could do with the existing mechanism if the limit was higher? Together with you not answering the question about the lack of an enforced limit in your new mechanism, this clearly tells me that you should use the existing one, raising the limit and - if necessary - dealing with the (security) fallout.

essentially those pages are not contiguous and dynamic, and thus a fixed range based structure even with a higher limitation doesn't fit here. Just think about a structure similar to what we have done to maintain shadow pfn is required here. A hash list is more flexible to serve such purpose.

Thanks
Kevin

#45 lore raw

Re: [PATCH v1 2/2] ioreq-server: Support scatter page forwarding

Quoted 9 lines
Quoted 8 lines
Quoted 1 line · On 04.08.14 at 23:39, <[email protected]> wrote:

From: Jan Beulich [mailto:[email protected]]
So am I reading this right that you could do with the existing mechanism if the limit was higher? Together with you not answering the question about the lack of an enforced limit in your new mechanism, this clearly tells me that you should use the existing one, raising the limit and - if necessary - dealing with the (security) fallout.

Quoted 4 lines

essentially those pages are not contiguous and dynamic, and thus a fixed range based structure even with a higher limitation doesn't fit here. Just think about a structure similar to what we have done to maintain shadow pfn is required here. A hash list is more flexible to serve such purpose.

I don't follow - range sets aren't a "fixed range based structure" to me. All we impose is a limit on the number of ranges you may have in place to avoid unbounded amounts of memory being tied up by a domain, or unbounded amounts of time are spent in accessing that tree. Obviously the same would need to be done for any new mechanism, so a new mechanism is only first choice if it provides other benefits over the original one.

Jan