Skip to contentSkip to conversation

testing: Build WHPX enabled binaries

PATCH Applied Philippe Mathieu-Daudé 8 participants 23 messages Sep 20, 2019 – Aug 19, 2020

lore
Share & tools
lore
#0 lore raw

v2 cover testing: Build WHPX enabled binaries

Add a job to cross-build QEMU with WHPX enabled.

Since the WHPX is currently broken, include the patch required to have successful Shippable build.

I previously included the WHPX headers shared by the Android project, and Daniel asked me to check the EULA. While trying to manually install the Windows SDK, I noticed the installer fetches archives directly, kindly asking where they are stored via the /fwlink API. Do the same, fetch the required archives and extract them. No need to accept EULA...

Docker build the image first, then build QEMU in a instance of this image. The image is internal to Shippable, the instances are not reachable and are thrown once the build is finished. What we collect from Shippable is the console output of QEMU build process, and if the build process succeed or failed. So far we do not redistribute the image or built binaries.

Philippe Mathieu-Daudé (3):
target/i386: Fix broken build with WHPX enabled
tests/docker: Add fedora-win10sdk-cross image
.shippable.yml: Build WHPX enabled binaries

 .shippable.yml                                |  2 ++
 target/i386/whpx-all.c                        |  1 +
 tests/docker/Makefile.include                 |  2 ++
 .../dockerfiles/fedora-win10sdk-cross.docker  | 23 ++++++++++++++++
 tests/docker/dockerfiles/win10sdk-dl.sh       | 27 +++++++++++++++++++
 5 files changed, 55 insertions(+)
 create mode 100644 tests/docker/dockerfiles/fedora-win10sdk-cross.docker
 create mode 100755 tests/docker/dockerfiles/win10sdk-dl.sh
Signature · 1 line
2.20.1
#3 lore raw

PATCH v2 2/3 tests/docker: Add fedora-win10sdk-cross image

To build WHPX (Windows Hypervisor) binaries, we need the WHPX headers provided by the Windows SDK.

Add a script that fetches the required MSI/CAB files from the latest SDK (currently 10.0.18362.1).

Headers are accessible under /opt/win10sdk/include.

Set the QEMU_CONFIGURE_OPTS environment variable accordingly, enabling HAX and WHPX. Due to CPP warnings related to Microsoft specific #pragmas, we also need to use the '--disable-werror' configure flag.

---
 tests/docker/Makefile.include                 |  2 ++
 .../dockerfiles/fedora-win10sdk-cross.docker  | 23 ++++++++++++++++
 tests/docker/dockerfiles/win10sdk-dl.sh       | 27 +++++++++++++++++++
 3 files changed, 52 insertions(+)
 create mode 100644 tests/docker/dockerfiles/fedora-win10sdk-cross.docker
 create mode 100755 tests/docker/dockerfiles/win10sdk-dl.sh
tests/docker/Makefile.include +2 -0
index 50a400b573..aa2e4e56dd 100644
@@ -125,6 +125,8 @@ docker-image-debian-ppc64-cross: docker-image-debian10
 docker-image-debian-riscv64-cross: docker-image-debian10
 docker-image-debian-sh4-cross: docker-image-debian10
 docker-image-debian-sparc64-cross: docker-image-debian10
+docker-image-fedora-win10sdk-cross: docker-image-fedora
+docker-image-fedora-win10sdk-cross: EXTRA_FILES:=$(DOCKER_FILES_DIR)/win10sdk-dl.sh
 
 docker-image-travis: NOUSER=1
 
tests/docker/dockerfiles/fedora-win10sdk-cross.docker new file +23 -0
new file mode 100644
index 0000000000..55ca933d40
@@ -0,0 +1,23 @@
+#
+# Docker MinGW64 cross-compiler target with WHPX header installed
+#
+# This docker target builds on the Fedora 30 base image.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+FROM qemu:fedora
+
+RUN dnf install -y \
+        cabextract \
+        msitools \
+        wget
+
+# Install WHPX headers from Windows Software Development Kit:
+# https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk
+ADD win10sdk-dl.sh /usr/local/bin/win10sdk-dl.sh
+RUN /usr/local/bin/win10sdk-dl.sh
+
+ENV QEMU_CONFIGURE_OPTS ${QEMU_CONFIGURE_OPTS} \
+    --cross-prefix=x86_64-w64-mingw32- \
+    --extra-cflags=-I/opt/win10sdk/include --disable-werror \
+    --enable-hax --enable-whpx
tests/docker/dockerfiles/win10sdk-dl.sh new file +27 -0
new file mode 100755
index 0000000000..1c35c2a252
@@ -0,0 +1,27 @@
+#!/bin/bash
+#
+# Install WHPX headers from Windows Software Development Kit
+# https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+WINDIR=/opt/win10sdk
+mkdir -p ${WINDIR}
+pushd ${WINDIR}
+# Get the bundle base for Windows SDK v10.0.18362.1
+BASE_URL=$(curl --silent --include 'http://go.microsoft.com/fwlink/?prd=11966&pver=1.0&plcid=0x409&clcid=0x409&ar=Windows10&sar=SDK&o1=10.0.18362.1' | sed -nE 's_Location: (.*)/\r_\1_p')/Installers
+# Fetch the MSI containing the headers
+wget --no-verbose ${BASE_URL}/'Windows SDK Desktop Headers x86-x86_en-us.msi'
+while true; do
+    # Fetch all cabinets required by this MSI
+    CAB_NAME=$(msiextract Windows\ SDK\ Desktop\ Headers\ x86-x86_en-us.msi 3>&1 2>&3 3>&-| sed -nE "s_.*Error opening file $PWD/(.*): No such file or directory_\1_p")
+    test -z "${CAB_NAME}" && break
+    wget --no-verbose ${BASE_URL}/${CAB_NAME}
+done
+rm *.{cab,msi}
+mkdir /opt/win10sdk/include
+# Only keep the WHPX headers
+for inc in "${WINDIR}/Program Files/Windows Kits/10/Include/10.0.18362.0/um"/WinHv*; do
+    ln -s "${inc}" /opt/win10sdk/include
+done
+popd > /dev/null
Signature · 1 line
2.20.1
#4 lore raw

PATCH v2 3/3 .shippable.yml: Build WHPX enabled binaries

In reply to Philippe Mathieu-Daudé
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)
.shippable.yml +2 -0
index bbc6f88510..01b33bd034 100644
@@ -25,6 +25,8 @@ env:
       TARGET_LIST=mips64el-softmmu,mips64el-linux-user
     - IMAGE=debian-ppc64el-cross
       TARGET_LIST=ppc64-softmmu,ppc64-linux-user,ppc64abi32-linux-user
+    - IMAGE=fedora-win10sdk-cross
+      TARGET_LIST=x86_64-softmmu,i386-softmmu
 build:
   pre_ci:
     # usually host ARCH is set by configure
Signature · 1 line
2.20.1
#1 lore raw

PATCH v2 1/3 target/i386: Fix broken build with WHPX enabled

In reply to Philippe Mathieu-Daudé

The WHPX build is broken since commit 12e9493df92 which removed the "hw/boards.h" where MachineState is declared:

  $ ./configure \
     --enable-hax --enable-whpx
  $ make x86_64-softmmu/all
  [...]
    CC      x86_64-softmmu/target/i386/whpx-all.o
  target/i386/whpx-all.c: In function 'whpx_accel_init':
  target/i386/whpx-all.c:1378:25: error: dereferencing pointer to
  incomplete type 'MachineState' {aka 'struct MachineState'}
       whpx->mem_quota = ms->ram_size;
                           ^~
  make[1]: *** [rules.mak:69: target/i386/whpx-all.o] Error 1
    CC      x86_64-softmmu/trace/generated-helpers.o
  make[1]: Target 'all' not remade because of errors.
  make: *** [Makefile:471: x86_64-softmmu/all] Error 2

Restore this header, partially reverting commit 12e9493df92.

---
 target/i386/whpx-all.c | 1 +
 1 file changed, 1 insertion(+)
target/i386/whpx-all.c +1 -0
index 0c15241ae4..def0c28480 100644
@@ -18,6 +18,7 @@
 #include "sysemu/cpus.h"
 #include "sysemu/runstate.h"
 #include "qemu/main-loop.h"
+#include "hw/boards.h"
 #include "qemu/error-report.h"
 #include "qapi/error.h"
 #include "migration/blocker.h"
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH v2 1/3] target/i386: Fix broken build with WHPX enabled

Quoted 35 lines · Philippe Mathieu-Daudé

The WHPX build is broken since commit 12e9493df92 which removed the "hw/boards.h" where MachineState is declared:

  $ ./configure \
     --enable-hax --enable-whpx
  $ make x86_64-softmmu/all
  [...]
    CC      x86_64-softmmu/target/i386/whpx-all.o
  target/i386/whpx-all.c: In function 'whpx_accel_init':
  target/i386/whpx-all.c:1378:25: error: dereferencing pointer to
  incomplete type 'MachineState' {aka 'struct MachineState'}
       whpx->mem_quota = ms->ram_size;
                           ^~
  make[1]: *** [rules.mak:69: target/i386/whpx-all.o] Error 1
    CC      x86_64-softmmu/trace/generated-helpers.o
  make[1]: Target 'all' not remade because of errors.
  make: *** [Makefile:471: x86_64-softmmu/all] Error 2

Restore this header, partially reverting commit 12e9493df92.

---
 target/i386/whpx-all.c | 1 +
 1 file changed, 1 insertion(+)
target/i386/whpx-all.c +0 -0
index 0c15241ae4..def0c28480 100644

target/i386/whpx-all.c line 24

Quoted 7 lines
target/i386/whpx-all.c +1 -0
@@ -18,6 +18,7 @@
 #include "sysemu/cpus.h"
 #include "sysemu/runstate.h"
 #include "qemu/main-loop.h"
+#include "hw/boards.h"
 #include "qemu/error-report.h"
 #include "qapi/error.h"
 #include "migration/blocker.h"

Queued, thanks.

Paolo

#5 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 19 lines · Philippe Mathieu-Daudé

Add a job to cross-build QEMU with WHPX enabled.

Since the WHPX is currently broken, include the patch required to have successful Shippable build.

I previously included the WHPX headers shared by the Android project, and Daniel asked me to check the EULA. While trying to manually install the Windows SDK, I noticed the installer fetches archives directly, kindly asking where they are stored via the /fwlink API. Do the same, fetch the required archives and extract them. No need to accept EULA...

Docker build the image first, then build QEMU in a instance of this image. The image is internal to Shippable, the instances are not reachable and are thrown once the build is finished. What we collect from Shippable is the console output of QEMU build process, and if the build process succeed or failed. So far we do not redistribute the image or built binaries.

Quoted 4 lines

Philippe Mathieu-Daudé (3):
target/i386: Fix broken build with WHPX enabled
tests/docker: Add fedora-win10sdk-cross image
.shippable.yml: Build WHPX enabled binaries

FWIW here is the result of this series:
https://app.shippable.com/github/philmd/qemu/runs/516/11/console
Duration 17 minutes (1076 seconds)

4m49s building the qemu:fedora-win10sdk-cross docker image, 11m10s building WHPX QEMU.

#6 lore raw

Hey Phil,

I have contacted our legal department for guidance on this specific use case and will update you when I hear back. Thank you for your patience.

Justin Terry

Quoted 47 lines

-----Original Message-----
From: Philippe Mathieu-Daudé <[email protected]>
Sent: Friday, September 20, 2019 8:18 AM
To: [email protected]; Justin Terry (VM) <[email protected]>
Cc: Daniel P . Berrangé <[email protected]>; Fam Zheng <[email protected]>; Thomas Huth <[email protected]>; Paolo Bonzini <[email protected]>; Alex Bennée <[email protected]>; Richard Henderson <[email protected]>; Eduardo Habkost <[email protected]>; Stefan Weil <[email protected]>
Subject: Re: [PATCH v2 0/3] testing: Build WHPX enabled binaries

Quoted 19 lines · Philippe Mathieu-Daudé

Add a job to cross-build QEMU with WHPX enabled.

Since the WHPX is currently broken, include the patch required to have successful Shippable build.

I previously included the WHPX headers shared by the Android project, and Daniel asked me to check the EULA. While trying to manually install the Windows SDK, I noticed the installer fetches archives directly, kindly asking where they are stored via the /fwlink API. Do the same, fetch the required archives and extract them. No need to accept EULA...

Docker build the image first, then build QEMU in a instance of this image. The image is internal to Shippable, the instances are not reachable and are thrown once the build is finished. What we collect from Shippable is the console output of QEMU build process, and if the build process succeed or failed. So far we do not redistribute the image or built binaries.

Quoted 4 lines

Philippe Mathieu-Daudé (3):
target/i386: Fix broken build with WHPX enabled
tests/docker: Add fedora-win10sdk-cross image
.shippable.yml: Build WHPX enabled binaries

FWIW here is the result of this series:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapp.
shippable.com%2Fgithub%2Fphilmd%2Fqemu%2Fruns%2F516%2F11%2Fcon sole&amp;data=02%7C01%7Cjuterry%40microsoft.com%7C733a566f3233427 8ae6f08d73dddb39f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6 37045894733463150&amp;sdata=55URgDII5r74QMUpLOD%2FWT5%2B5jbzyv nfCSdv%2FNaWDAw%3D&amp;reserved=0
Duration 17 minutes (1076 seconds)

4m49s building the qemu:fedora-win10sdk-cross docker image, 11m10s building WHPX QEMU.

#22 lore raw
In reply to Philippe Mathieu-Daudé
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series seems to have some coding style problems. See output below for more information:

Message-id: [email protected]
Subject: [PATCH v2 0/3] testing: Build WHPX enabled binaries
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384 Switched to a new branch 'test'
87eaddb .shippable.yml: Build WHPX enabled binaries e5d4b91 tests/docker: Add fedora-win10sdk-cross image 52ca140 target/i386: Fix broken build with WHPX enabled

=== OUTPUT BEGIN ===
1/3 Checking commit 52ca14097b01 (target/i386: Fix broken build with WHPX enabled) 2/3 Checking commit e5d4b91b58ca (tests/docker: Add fedora-win10sdk-cross image)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #38:
new file mode 100644

ERROR: line over 90 characters
#83: FILE: tests/docker/dockerfiles/win10sdk-dl.sh:12:
+BASE_URL=$(curl --silent --include 'http://go.microsoft.com/fwlink/?prd=11966&pver=1.0&plcid=0x409&clcid=0x409&ar=Windows10&sar=SDK&o1=10.0.18362.1' | sed -nE 's_Location: (.*)/\r_\1_p')/Installers

ERROR: line over 90 characters
#88: FILE: tests/docker/dockerfiles/win10sdk-dl.sh:17:
+ CAB_NAME=$(msiextract Windows\ SDK\ Desktop\ Headers\ x86-x86_en-us.msi 3>&1 2>&3 3>&-| sed -nE "s_.*Error opening file $PWD/(.*): No such file or directory_\1_p")

WARNING: line over 80 characters
#95: FILE: tests/docker/dockerfiles/win10sdk-dl.sh:24:
+for inc in "${WINDIR}/Program Files/Windows Kits/10/Include/10.0.18362.0/um"/WinHv*; do

total: 2 errors, 2 warnings, 58 lines checked

Patch 2/3 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

3/3 Checking commit 87eaddbdd398 (.shippable.yml: Build WHPX enabled binaries) === OUTPUT END ===

Test command exited with code: 1

The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#7 lore raw
In reply to Justin Terry (VM)

+launchpad ticket

Quoted 53 lines · Justin Terry (VM)

Hey Phil,

I have contacted our legal department for guidance on this specific use case and will update you when I hear back. Thank you for your patience.

Justin Terry

Quoted 47 lines

-----Original Message-----
From: Philippe Mathieu-Daudé <[email protected]>
Sent: Friday, September 20, 2019 8:18 AM
To: [email protected]; Justin Terry (VM) <[email protected]>
Cc: Daniel P . Berrangé <[email protected]>; Fam Zheng <[email protected]>; Thomas Huth <[email protected]>; Paolo Bonzini <[email protected]>; Alex Bennée <[email protected]>; Richard Henderson <[email protected]>; Eduardo Habkost <[email protected]>; Stefan Weil <[email protected]>
Subject: Re: [PATCH v2 0/3] testing: Build WHPX enabled binaries

Quoted 19 lines · Philippe Mathieu-Daudé

Add a job to cross-build QEMU with WHPX enabled.

Since the WHPX is currently broken, include the patch required to have successful Shippable build.

I previously included the WHPX headers shared by the Android project, and Daniel asked me to check the EULA. While trying to manually install the Windows SDK, I noticed the installer fetches archives directly, kindly asking where they are stored via the /fwlink API. Do the same, fetch the required archives and extract them. No need to accept EULA...

Docker build the image first, then build QEMU in a instance of this image. The image is internal to Shippable, the instances are not reachable and are thrown once the build is finished. What we collect from Shippable is the console output of QEMU build process, and if the build process succeed or failed. So far we do not redistribute the image or built binaries.

Quoted 4 lines

Philippe Mathieu-Daudé (3):
target/i386: Fix broken build with WHPX enabled tests/docker: Add fedora-win10sdk-cross image .shippable.yml: Build WHPX enabled binaries

FWIW here is the result of this series:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapp.
shippable.com%2Fgithub%2Fphilmd%2Fqemu%2Fruns%2F516%2F11%2Fcon sole&amp;data=02%7C01%7Cjuterry%40microsoft.com%7C733a566f3233427 8ae6f08d73dddb39f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6 37045894733463150&amp;sdata=55URgDII5r74QMUpLOD%2FWT5%2B5jbzyv nfCSdv%2FNaWDAw%3D&amp;reserved=0
Duration 17 minutes (1076 seconds)

4m49s building the qemu:fedora-win10sdk-cross docker image, 11m10s building WHPX QEMU.

#8 lore raw

Hi Justin, Sunil,

Quoted 8 lines · Philippe Mathieu-Daudé

+launchpad ticket

Quoted 5 lines · Justin Terry (VM)

Hey Phil,

I have contacted our legal department for guidance on this specific use case and will update you when I hear back. Thank you for your patience.

I recently understood legal changes can be very complex, thus it is implicit it can take years before getting updates.

Since the project is still actively developed, maybe you could provide a Azure CI job to build a WHPX binary. We don't need to have access to the binary, just to the exit status (success/fail) and build logs.

Do you think it is doable?

Thanks,

Phil.

Quoted 51 lines
Quoted 50 lines

Justin Terry

Quoted 47 lines

-----Original Message-----
From: Philippe Mathieu-Daudé <[email protected]>
Sent: Friday, September 20, 2019 8:18 AM
To: [email protected]; Justin Terry (VM) <[email protected]>
Cc: Daniel P . Berrangé <[email protected]>; Fam Zheng <[email protected]>; Thomas Huth <[email protected]>; Paolo Bonzini <[email protected]>; Alex Bennée <[email protected]>; Richard Henderson <[email protected]>; Eduardo Habkost <[email protected]>; Stefan Weil <[email protected]>
Subject: Re: [PATCH v2 0/3] testing: Build WHPX enabled binaries

Quoted 19 lines · Philippe Mathieu-Daudé

Add a job to cross-build QEMU with WHPX enabled.

Since the WHPX is currently broken, include the patch required to have successful Shippable build.

I previously included the WHPX headers shared by the Android project, and Daniel asked me to check the EULA. While trying to manually install the Windows SDK, I noticed the installer fetches archives directly, kindly asking where they are stored via the /fwlink API. Do the same, fetch the required archives and extract them. No need to accept EULA...

Docker build the image first, then build QEMU in a instance of this image. The image is internal to Shippable, the instances are not reachable and are thrown once the build is finished. What we collect from Shippable is the console output of QEMU build process, and if the build process succeed or failed. So far we do not redistribute the image or built binaries.

Quoted 4 lines

Philippe Mathieu-Daudé (3):
   target/i386: Fix broken build with WHPX enabled tests/docker: Add fedora-win10sdk-cross image .shippable.yml: Build WHPX enabled binaries

FWIW here is the result of this series:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapp.
shippable.com%2Fgithub%2Fphilmd%2Fqemu%2Fruns%2F516%2F11%2Fcon sole&amp;data=02%7C01%7Cjuterry%40microsoft.com%7C733a566f3233427 8ae6f08d73dddb39f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6 37045894733463150&amp;sdata=55URgDII5r74QMUpLOD%2FWT5%2B5jbzyv nfCSdv%2FNaWDAw%3D&amp;reserved=0
Duration 17 minutes (1076 seconds)

4m49s building the qemu:fedora-win10sdk-cross docker image, 11m10s building WHPX QEMU.

#9 lore raw
Quoted 1 line

Hi Justin, Sunil,

Justin has moved to a different team is no longer working with WHPX. Moving him to bcc.

Quoted 18 lines · Philippe Mathieu-Daudé
Quoted 8 lines

+launchpad ticket

Quoted 5 lines · Justin Terry (VM)

Hey Phil,

I have contacted our legal department for guidance on this specific use case and will update you when I hear back. Thank you for your patience.

I recently understood legal changes can be very complex, thus it is implicit it can take years before getting updates.

Since the project is still actively developed, maybe you could provide a Azure CI job to build a WHPX binary. We don't need to have access to the binary, just to the exit status (success/fail) and build logs.

Quoted 6 lines

Do you think it is doable?

Thanks,

Phil.

The ask generally sounds reasonable. But, can you help me understand the full scope of the ask. Few questions:
1. Stefan has a CI pipeline to build WHPX. What's the benefit of having another CI job, that doesn't export the binary, but, just the status?
2. Which branch is the CI pipeline expected to build?
3. Is the expectation also that it will build WHPX patches that are submitted to the WHPX branch?

#10 lore raw

Hi Sunil,

Quoted 4 lines · Sunil Muthuswamy
Quoted 1 line

Hi Justin, Sunil,

Justin has moved to a different team is no longer working with WHPX. Moving him to bcc.

OK. Does that mean you are the new responsible of updating the ticket regarding the WHPX headers and their license?

Quoted 21 lines
Quoted 21 lines · Philippe Mathieu-Daudé
Quoted 8 lines

+launchpad ticket

Quoted 5 lines · Justin Terry (VM)

Hey Phil,

I have contacted our legal department for guidance on this specific use case and will update you when I hear back. Thank you for your patience.

I recently understood legal changes can be very complex, thus it is implicit it can take years before getting updates.

Since the project is still actively developed, maybe you could provide a Azure CI job to build a WHPX binary. We don't need to have access to the binary, just to the exit status (success/fail) and build logs.

Do you think it is doable?

Quoted 7 lines
Quoted 4 lines

Thanks,

Phil.

The ask generally sounds reasonable. But, can you help me understand the full scope of the ask. Few questions:
1. Stefan has a CI pipeline to build WHPX.

Great! I didn't know Stefan already did it :)
Can you share the URL please, so we can integrate it with mainstream CI?

Quoted 2 lines

What's the benefit of having another CI
job, that doesn't export the binary, but, just the status?

As usual, we do not want to circumvent the license. IANAL but IIUC we can not force a CI job to accept the EULA when installing it, even to test it. So the best we can do is check if the build succeeded (exit status).

Quoted 1 line

2. Which branch is the CI pipeline expected to build?

'master', to be sure no regressions are introduced.

Quoted 2 lines

3. Is the expectation also that it will build WHPX patches that are submitted to the WHPX branch?

You describe a "downstream CI" testing, which is out of scope of the community public CI.

Regards,

Phil.

#11 lore raw

Am 03.08.20 um 12:51 schrieb Philippe Mathieu-Daudé:

Quoted 8 lines

Hi Sunil,

Quoted 3 lines · Sunil Muthuswamy

The ask generally sounds reasonable. But, can you help me understand the full scope of the ask. Few questions:
1. Stefan has a CI pipeline to build WHPX.

Great! I didn't know Stefan already did it :)
Can you share the URL please, so we can integrate it with mainstream CI?

I am sorry, but I don't have such a CI pipeline.

Stefan

#12 lore raw

Am 03.08.20 um 13:28 schrieb Stefan Weil:

Quoted 8 lines

Am 03.08.20 um 12:51 schrieb Philippe Mathieu-Daudé:

Quoted 6 lines

Hi Sunil,

Quoted 3 lines · Sunil Muthuswamy

The ask generally sounds reasonable. But, can you help me understand the full scope of the ask. Few questions:
1. Stefan has a CI pipeline to build WHPX.

Quoted 6 lines
Quoted 2 lines

Great! I didn't know Stefan already did it :)
Can you share the URL please, so we can integrate it with mainstream CI?

I am sorry, but I don't have such a CI pipeline.

Stefan

We can add a CI pipeline on Microsoft infrastructure by using a GitHub action.

Here is an example: https://github.com/stweil/qemu/actions.

I just sent a patch which adds the GitHub action for
https://github.com/qemu/qemu.

Peter, maybe you can add that patch even for 5.1.

Kind regards,

Stefan

#13 lore raw
Quoted 13 lines · Stefan Weil

Am 03.08.20 um 13:28 schrieb Stefan Weil:

Quoted 11 lines

Am 03.08.20 um 12:51 schrieb Philippe Mathieu-Daudé:

Quoted 8 lines

Hi Sunil,

Quoted 3 lines · Sunil Muthuswamy

The ask generally sounds reasonable. But, can you help me understand the full scope of the ask. Few questions:
1. Stefan has a CI pipeline to build WHPX.

Great! I didn't know Stefan already did it :)
Can you share the URL please, so we can integrate it with mainstream CI?

Quoted 7 lines
Quoted 3 lines

I am sorry, but I don't have such a CI pipeline.

Stefan

We can add a CI pipeline on Microsoft infrastructure by using a GitHub action.

Sorry for being ignorant, but how does that solve the legal questions just because it is running on GitHub instead of a different CI?

Thomas

#14 lore raw

Am 04.08.20 um 08:43 schrieb Thomas Huth:

Quoted 8 lines · Stefan Weil
Quoted 2 lines

We can add a CI pipeline on Microsoft infrastructure by using a GitHub action.

Sorry for being ignorant, but how does that solve the legal questions just because it is running on GitHub instead of a different CI?

Thomas

Sorry, I though that would be clear by looking at the included shell script.

The build does not use the Microsoft SDK. It gets the required header files from Mingw-w64. They added them in git master.

See
https://github.com/stweil/qemu/blob/master/.github/workflows/build.sh#L50
for code details.

It's still shameful that MS is forcing developers to waste time rewriting API headers, just because the MS legal departments are not able to understand the needs of Open Source development.

Stefan

#15 lore raw
Quoted 8 lines · Stefan Weil

Am 04.08.20 um 08:43 schrieb Thomas Huth:

Quoted 6 lines · Stefan Weil
Quoted 2 lines

We can add a CI pipeline on Microsoft infrastructure by using a GitHub action.

Sorry for being ignorant, but how does that solve the legal questions just because it is running on GitHub instead of a different CI?

Quoted 7 lines
Quoted 2 lines

Thomas

Sorry, I though that would be clear by looking at the included shell script.

The build does not use the Microsoft SDK. It gets the required header files from Mingw-w64. They added them in git master.

Oh, so we can do that with GitLab too now, we don't need to rely on the GitHub 'Actions' CI in particular, right?

Quoted 8 lines

See
https://github.com/stweil/qemu/blob/master/.github/workflows/build.sh#L50
for code details.

It's still shameful that MS is forcing developers to waste time rewriting API headers, just because the MS legal departments are not able to understand the needs of Open Source development.

There has be a big switch from Microsoft toward Open Source, I attended some of there talk at the Open Source Summit in 2018. Maybe we simply haven't contacted the right persons to make the changes...?

Quoted 5 lines

Stefan

#16 lore raw

Am 04.08.20 um 09:23 schrieb Philippe Mathieu-Daudé:

Quoted 9 lines · Stefan Weil
Quoted 8 lines

Am 04.08.20 um 08:43 schrieb Thomas Huth:

Quoted 6 lines · Stefan Weil
Quoted 2 lines

We can add a CI pipeline on Microsoft infrastructure by using a GitHub action.

Sorry for being ignorant, but how does that solve the legal questions just because it is running on GitHub instead of a different CI?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Thomas

Sorry, I though that would be clear by looking at the included shell script.

The build does not use the Microsoft SDK. It gets the required header files from Mingw-w64. They added them in git master.

Oh, so we can do that with GitLab too now, we don't need to rely on the GitHub 'Actions' CI in particular, right?

That's right. The build script was written for Ubuntu, so depending on the distribution used for GitLab CI it will need some modifications. If GitLab already has a recent Mingw-w64, it might be sufficient to fix the case of the header file names. Mingw-w64 uses winhvplatform.h while QEMU expects WinHvPlatform.h and so on. I used symbolic links to add the camel case filenames.

Quoted 10 lines
Quoted 7 lines

See
https://github.com/stweil/qemu/blob/master/.github/workflows/build.sh#L50
for code details.

It's still shameful that MS is forcing developers to waste time rewriting API headers, just because the MS legal departments are not able to understand the needs of Open Source development.

There has be a big switch from Microsoft toward Open Source, I attended some of there talk at the Open Source Summit in 2018. Maybe we simply haven't contacted the right persons to make the changes...?

Maybe, but it is difficult to find the right person in a large company like MS, and legal departments are often somehow special.

And yes, they learned that Open Source can help them for their business, too.

Stefan

#17 lore raw
Quoted 36 lines · Stefan Weil

Am 04.08.20 um 09:23 schrieb Philippe Mathieu-Daudé:

Quoted 9 lines · Stefan Weil
Quoted 8 lines

Am 04.08.20 um 08:43 schrieb Thomas Huth:

Quoted 6 lines · Stefan Weil
Quoted 2 lines

We can add a CI pipeline on Microsoft infrastructure by using a GitHub action.

Sorry for being ignorant, but how does that solve the legal questions just because it is running on GitHub instead of a different CI?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Thomas

Sorry, I though that would be clear by looking at the included shell script.

The build does not use the Microsoft SDK. It gets the required header files from Mingw-w64. They added them in git master.

Oh, so we can do that with GitLab too now, we don't need to rely on the GitHub 'Actions' CI in particular, right?

That's right. The build script was written for Ubuntu, so depending on the distribution used for GitLab CI it will need some modifications. If GitLab already has a recent Mingw-w64, it might be sufficient to fix the case of the header file names. Mingw-w64 uses winhvplatform.h while QEMU expects WinHvPlatform.h and so on. I used symbolic links to add the camel case filenames.

Quoted 7 lines
Quoted 7 lines

See
https://github.com/stweil/qemu/blob/master/.github/workflows/build.sh#L50
for code details.

It's still shameful that MS is forcing developers to waste time rewriting API headers, just because the MS legal departments are not able to understand the needs of Open Source development.

Quoted 7 lines
Quoted 3 lines

There has be a big switch from Microsoft toward Open Source, I attended some of there talk at the Open Source Summit in 2018. Maybe we simply haven't contacted the right persons to make the changes...?

Maybe, but it is difficult to find the right person in a large company like MS, and legal departments are often somehow special.

Sunil seems quite active with the WHPX development, and the section is listed as "Supported [my Microsoft]" in MAINTAINERS. I'm confident we have someone else able to help use finding the right contacts in the company :)

Quoted 8 lines

And yes, they learned that Open Source can help them for their business, too.

Stefan

#18 lore raw
In reply to Stefan Weil
Quoted 17 lines · Stefan Weil

Am 04.08.20 um 09:23 schrieb Philippe Mathieu-Daudé:

Quoted 15 lines · Stefan Weil
Quoted 14 lines

Am 04.08.20 um 08:43 schrieb Thomas Huth:

Quoted 8 lines · Stefan Weil
Quoted 2 lines

We can add a CI pipeline on Microsoft infrastructure by using a GitHub action.

Sorry for being ignorant, but how does that solve the legal questions just because it is running on GitHub instead of a different CI?

Thomas

Sorry, I though that would be clear by looking at the included shell script.

The build does not use the Microsoft SDK. It gets the required header files from Mingw-w64. They added them in git master.

Great, thanks for the clarification!

Quoted 9 lines
Quoted 2 lines

Oh, so we can do that with GitLab too now, we don't need to rely on the GitHub 'Actions' CI in particular, right?

That's right. The build script was written for Ubuntu, so depending on the distribution used for GitLab CI it will need some modifications. If GitLab already has a recent Mingw-w64, it might be sufficient to fix the case of the header file names. Mingw-w64 uses winhvplatform.h while QEMU expects WinHvPlatform.h and so on. I used symbolic links to add the camel case filenames.

I'm currently working on a patch series for our gitlab-CI that uses our containers to all possible kinds of cross-compiler builds (basically the ones that we are doing on shippable.com so far), including the 32-bit and 64-bit MinGW cross-compilation jobs. I can have a look whether I can integrate these headers there!

Thanks,
Thomas

#19 lore raw
Quoted 31 lines · Thomas Huth
Quoted 17 lines · Stefan Weil

Am 04.08.20 um 09:23 schrieb Philippe Mathieu-Daudé:

Quoted 15 lines · Stefan Weil
Quoted 14 lines

Am 04.08.20 um 08:43 schrieb Thomas Huth:

Quoted 8 lines · Stefan Weil
Quoted 2 lines

We can add a CI pipeline on Microsoft infrastructure by using a GitHub action.

Sorry for being ignorant, but how does that solve the legal questions just because it is running on GitHub instead of a different CI?

Thomas

Sorry, I though that would be clear by looking at the included shell script.

The build does not use the Microsoft SDK. It gets the required header files from Mingw-w64. They added them in git master.

Great, thanks for the clarification!

Quoted 9 lines
Quoted 2 lines

Oh, so we can do that with GitLab too now, we don't need to rely on the GitHub 'Actions' CI in particular, right?

That's right. The build script was written for Ubuntu, so depending on the distribution used for GitLab CI it will need some modifications. If GitLab already has a recent Mingw-w64, it might be sufficient to fix the case of the header file names. Mingw-w64 uses winhvplatform.h while QEMU expects WinHvPlatform.h and so on. I used symbolic links to add the camel case filenames.

Quoted 5 lines

I'm currently working on a patch series for our gitlab-CI that uses our containers to all possible kinds of cross-compiler builds (basically the ones that we are doing on shippable.com so far), including the 32-bit and 64-bit MinGW cross-compilation jobs. I can have a look whether I can integrate these headers there!

Fedora rawhide carries mingw64 v7.0.0, which was released in Nov 2019

The WHPX headers were added to mingw64 git a week later, so they're not available in any distro yet.

The mingw64 release schedule looks "sporadic" so maybe we can just request a new release to make WPHX stuff available. It'll thus be available for our CI in rawhide/sid shortly thereafter, which will be the best solution to let us do this in GitLab.

We certainly don't want to add yet another separate CI system just for WHPX.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#20 lore raw
In reply to Stefan Weil
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

It's still shameful that MS is forcing developers to waste time rewriting API headers, just because the MS legal departments are not able to understand the needs of Open Source development.

There has be a big switch from Microsoft toward Open Source, I attended some of there talk at the Open Source Summit in 2018. Maybe we simply haven't contacted the right persons to make the changes...?

Quoted 7 lines

Maybe, but it is difficult to find the right person in a large company like MS, and legal departments are often somehow special.

And yes, they learned that Open Source can help them for their business, too.

Stefan

Mike Battista is the program manager owner of the SDK license and should be able to take/respond to any feedback about the SDK licensing for open source projects (I have added him here). He has also been added to previous threads about the licensing and is also included in this conversation:
https://bugs.launchpad.net/qemu/+bug/1879672

- Sunil

#21 lore raw
Quoted 14 lines · Sunil Muthuswamy
Quoted 14 lines
Quoted 6 lines
Quoted 3 lines

It's still shameful that MS is forcing developers to waste time rewriting API headers, just because the MS legal departments are not able to understand the needs of Open Source development.

There has be a big switch from Microsoft toward Open Source, I attended some of there talk at the Open Source Summit in 2018. Maybe we simply haven't contacted the right persons to make the changes...?

Maybe, but it is difficult to find the right person in a large company like MS, and legal departments are often somehow special.

And yes, they learned that Open Source can help them for their business, too.

Quoted 7 lines
Quoted 1 line

Stefan

Mike Battista is the program manager owner of the SDK license and should be able to take/respond to any feedback about the SDK licensing for open source projects (I have added him here). He has also been added to previous threads about the licensing and is also included in this conversation:
https://bugs.launchpad.net/qemu/+bug/1879672

Hi Mike, thanks for helping us with this issue!

And thanks a lot Sunil to bring Mike here :)

Quoted 4 lines

- Sunil