Skip to contentSkip to conversation

gitlab: speed up msys windows jobs with GCC

PATCH In review Daniel P. Berrangé 3 participants 24 messages Aug 1, 2023 – Aug 3, 2023

lore
Share & tools
lore
#0 lore raw

cover gitlab: speed up msys windows jobs with GCC

This is an alternative and/or complementary to Thomas' proposal to use CLang with msys:

https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html

First of all, the current msys installer we're using is over 12 months out of date. Thus after running the install, pacman then replaces most of what we've just installed with new downloaded content. Using the most update installer cuts 3+1/2 minutes off the msys install time - 7 minutes becomes 3+1/2.

Secondly, QEMU defaults to compiling with -O2 and this is more computationally expensive for GCC. Switching to -O0 drops the build time from 60 minutes down to 45 minutes.

Thirdly, including debug symbols also has an overhead, and turning that off reduces time still further down to 38 minutes.

IOW, between all three changes, we can cut approx 25-26 minutes off the job execution time, bringing it nicely within the job timeout.

The actually phase of installing the mingw deps still accounts for about 10 minutes and has not been optimized.

Possibly the same trick of -O0 and skipping -g would also help the clang alternative Thomas' proposed. If so, that could be enough to let us enable more features / targets during the msys build.

Daniel P. Berrangé (8):
gitlab: remove duplication between msys jobs
gitlab: print timestamps during windows msys jobs
gitlab: always use updated msys installer
gitlab: drop $CI_PROJECT_DIR from cache path
gitlab: always populate cache for windows msys jobs
configure: support passthrough of -Dxxx args to meson
gitlab: disable optimization and debug symbols in msys build
gitlab: disable FF_SCRIPT_SECTIONS on msys jobs

 .gitlab-ci.d/windows.yml | 174 +++++++++++++++++++--------------------
 configure                |   5 ++
 2 files changed, 92 insertions(+), 87 deletions(-)
Signature · 1 line
2.41.0
#1 lore raw

PATCH 1/8 gitlab: remove duplication between msys jobs

Although they share a common parent, the two msys jobs still have massive duplication in their script definitions that can easily be collapsed.

---
 .gitlab-ci.d/windows.yml | 132 +++++++++++++++------------------------
 1 file changed, 49 insertions(+), 83 deletions(-)
.gitlab-ci.d/windows.yml +49 -83
index f889a468b5..f086540e40 100644
@@ -35,97 +35,63 @@
   - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu'  # Core update
   - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu'  # Normal update
   - taskkill /F /FI "MODULES eq msys-2.0.dll"
-
-msys2-64bit:
-  extends: .shared_msys2_builder
   script:
   - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
       bison diffutils flex
       git grep make sed
-      mingw-w64-x86_64-capstone
-      mingw-w64-x86_64-curl
-      mingw-w64-x86_64-cyrus-sasl
-      mingw-w64-x86_64-dtc
-      mingw-w64-x86_64-gcc
-      mingw-w64-x86_64-glib2
-      mingw-w64-x86_64-gnutls
-      mingw-w64-x86_64-gtk3
-      mingw-w64-x86_64-libgcrypt
-      mingw-w64-x86_64-libjpeg-turbo
-      mingw-w64-x86_64-libnfs
-      mingw-w64-x86_64-libpng
-      mingw-w64-x86_64-libssh
-      mingw-w64-x86_64-libtasn1
-      mingw-w64-x86_64-libusb
-      mingw-w64-x86_64-lzo2
-      mingw-w64-x86_64-nettle
-      mingw-w64-x86_64-ninja
-      mingw-w64-x86_64-pixman
-      mingw-w64-x86_64-pkgconf
-      mingw-w64-x86_64-python
-      mingw-w64-x86_64-SDL2
-      mingw-w64-x86_64-SDL2_image
-      mingw-w64-x86_64-snappy
-      mingw-w64-x86_64-spice
-      mingw-w64-x86_64-usbredir
-      mingw-w64-x86_64-zstd "
+      $MINGW_TARGET-capstone
+      $MINGW_TARGET-curl
+      $MINGW_TARGET-cyrus-sasl
+      $MINGW_TARGET-dtc
+      $MINGW_TARGET-gcc
+      $MINGW_TARGET-glib2
+      $MINGW_TARGET-gnutls
+      $MINGW_TARGET-gtk3
+      $MINGW_TARGET-libgcrypt
+      $MINGW_TARGET-libjpeg-turbo
+      $MINGW_TARGET-libnfs
+      $MINGW_TARGET-libpng
+      $MINGW_TARGET-libssh
+      $MINGW_TARGET-libtasn1
+      $MINGW_TARGET-libusb
+      $MINGW_TARGET-lzo2
+      $MINGW_TARGET-nettle
+      $MINGW_TARGET-ninja
+      $MINGW_TARGET-pixman
+      $MINGW_TARGET-pkgconf
+      $MINGW_TARGET-python
+      $MINGW_TARGET-SDL2
+      $MINGW_TARGET-SDL2_image
+      $MINGW_TARGET-snappy
+      $MINGW_TARGET-spice
+      $MINGW_TARGET-usbredir
+      $MINGW_TARGET-zstd "
   - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
-  - $env:MSYSTEM = 'MINGW64'     # Start a 64-bit MinGW environment
   - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
   - mkdir build
   - cd build
-  # Note: do not remove "--without-default-devices"!
-  # commit 9f8e6cad65a6 ("gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices"
-  # changed to compile QEMU with the --without-default-devices switch
-  # for the msys2 64-bit job, due to the build could not complete within
-  # the project timeout.
-  - ..\msys64\usr\bin\bash -lc '../configure --target-list=x86_64-softmmu
-      --without-default-devices --enable-fdt=system'
-  - ..\msys64\usr\bin\bash -lc 'make'
-  # qTests don't run successfully with "--without-default-devices",
-  # so let's exclude the qtests from CI for now.
-  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS=\"--no-suite qtest\" || { cat meson-logs/testlog.txt; exit 1; } ;'
+  - ..\msys64\usr\bin\bash -lc "../configure --enable-fdt=system $CONFIGURE_ARGS"
+  - ..\msys64\usr\bin\bash -lc "make"
+  - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
+
+msys2-64bit:
+  extends: .shared_msys2_builder
+  variables:
+    MINGW_TARGET: mingw-w64-x86_64
+    MSYSTEM: MINGW64
+    # do not remove "--without-default-devices"!
+    # commit 9f8e6cad65a6 ("gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices"
+    # changed to compile QEMU with the --without-default-devices switch
+    # for the msys2 64-bit job, due to the build could not complete within
+    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices
+    # qTests don't run successfully with "--without-default-devices",
+    # so let's exclude the qtests from CI for now.
+    TEST_ARGS: --no-suite qtest
 
 msys2-32bit:
   extends: .shared_msys2_builder
-  script:
-  - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
-      bison diffutils flex
-      git grep make sed
-      mingw-w64-i686-capstone
-      mingw-w64-i686-curl
-      mingw-w64-i686-cyrus-sasl
-      mingw-w64-i686-dtc
-      mingw-w64-i686-gcc
-      mingw-w64-i686-glib2
-      mingw-w64-i686-gnutls
-      mingw-w64-i686-gtk3
-      mingw-w64-i686-libgcrypt
-      mingw-w64-i686-libjpeg-turbo
-      mingw-w64-i686-libnfs
-      mingw-w64-i686-libpng
-      mingw-w64-i686-libssh
-      mingw-w64-i686-libtasn1
-      mingw-w64-i686-libusb
-      mingw-w64-i686-lzo2
-      mingw-w64-i686-nettle
-      mingw-w64-i686-ninja
-      mingw-w64-i686-pixman
-      mingw-w64-i686-pkgconf
-      mingw-w64-i686-python
-      mingw-w64-i686-SDL2
-      mingw-w64-i686-SDL2_image
-      mingw-w64-i686-snappy
-      mingw-w64-i686-spice
-      mingw-w64-i686-usbredir
-      mingw-w64-i686-zstd "
-  - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
-  - $env:MSYSTEM = 'MINGW32'     # Start a 32-bit MinGW environment
-  - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
-  - mkdir build
-  - cd build
-  - ..\msys64\usr\bin\bash -lc '../configure --target-list=ppc64-softmmu
-                                --enable-fdt=system'
-  - ..\msys64\usr\bin\bash -lc 'make'
-  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS=\"--no-suite qtest\" ||
-                                { cat meson-logs/testlog.txt; exit 1; }'
+  variables:
+    MINGW_TARGET: mingw-w64-i686
+    MSYSTEM: MINGW32
+    CONFIGURE_ARGS:  --target-list=ppc64-softmmu
+    TEST_ARGS: --no-suite qtest
Signature · 1 line
2.41.0
Reviewed-by on v1 1/8
#2 lore raw

Re: [PATCH 1/8] gitlab: remove duplication between msys jobs

Quoted 8 lines · Daniel P. Berrangé

Although they share a common parent, the two msys jobs still have massive duplication in their script definitions that can easily be collapsed.

---
  .gitlab-ci.d/windows.yml | 132 +++++++++++++++------------------------
  1 file changed, 49 insertions(+), 83 deletions(-)

As I discovered in the recent days, this is also quite helpful in case we ever want to switch to Clang, since we need to change the prefix of the packages there. And it's still easy to have distinct packages with some few lines of codes changes, as I tried it out in my Clang patch. So this patch now sounds fine to me:

#3 lore raw

PATCH 2/8 gitlab: print timestamps during windows msys jobs

In reply to Daniel P. Berrangé

It is hard to get visibility into where time is consumed in our Windows msys jobs. Adding a few log console messages with the timestamp will aid in our debugging.

---
 .gitlab-ci.d/windows.yml | 5 +++++
 1 file changed, 5 insertions(+)
.gitlab-ci.d/windows.yml +5 -0
index f086540e40..831b080d12 100644
@@ -19,6 +19,7 @@
     reports:
       junit: "build/meson-logs/testlog.junit.xml"
   before_script:
+  - Write-Output "Acquiring msys2.exe installer at $(Get-Date -Format u)"
   - If ( !(Test-Path -Path msys64\var\cache ) ) {
       mkdir msys64\var\cache
     }
@@ -27,6 +28,7 @@
       "https://github.com/msys2/msys2-installer/releases/download/2022-06-03/msys2-base-x86_64-20220603.sfx.exe"
       -outfile "msys64\var\cache\msys2.exe"
     }
+  - Write-Output "Invoking msys2.exe installer at $(Get-Date -Format u)"
   - msys64\var\cache\msys2.exe -y
   - ((Get-Content -path .\msys64\etc\\post-install\\07-pacman-key.post -Raw)
       -replace '--refresh-keys', '--version') |
@@ -36,6 +38,7 @@
   - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu'  # Normal update
   - taskkill /F /FI "MODULES eq msys-2.0.dll"
   script:
+  - Write-Output "Installing mingw packages at $(Get-Date -Format u)"
   - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
       bison diffutils flex
       git grep make sed
@@ -66,6 +69,7 @@
       $MINGW_TARGET-spice
       $MINGW_TARGET-usbredir
       $MINGW_TARGET-zstd "
+  - Write-Output "Running build at $(Get-Date -Format u)"
   - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
   - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
   - mkdir build
@@ -73,6 +77,7 @@
   - ..\msys64\usr\bin\bash -lc "../configure --enable-fdt=system $CONFIGURE_ARGS"
   - ..\msys64\usr\bin\bash -lc "make"
   - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
+  - Write-Output "Finished build at $(Get-Date -Format u)"
 
 msys2-64bit:
   extends: .shared_msys2_builder
Signature · 1 line
2.41.0
Reviewed-by on v1 2/8
#4 lore raw

Re: [PATCH 2/8] gitlab: print timestamps during windows msys jobs

Quoted 46 lines · Daniel P. Berrangé

It is hard to get visibility into where time is consumed in our Windows msys jobs. Adding a few log console messages with the timestamp will aid in our debugging.

---
  .gitlab-ci.d/windows.yml | 5 +++++
  1 file changed, 5 insertions(+)
.gitlab-ci.d/windows.yml +4 -0
index f086540e40..831b080d12 100644
@@ -19,6 +19,7 @@
      reports:
        junit: "build/meson-logs/testlog.junit.xml"
    before_script:
+  - Write-Output "Acquiring msys2.exe installer at $(Get-Date -Format u)"
    - If ( !(Test-Path -Path msys64\var\cache ) ) {
        mkdir msys64\var\cache
      }
@@ -27,6 +28,7 @@
        "https://github.com/msys2/msys2-installer/releases/download/2022-06-03/msys2-base-x86_64-20220603.sfx.exe"
        -outfile "msys64\var\cache\msys2.exe"
      }
+  - Write-Output "Invoking msys2.exe installer at $(Get-Date -Format u)"
    - msys64\var\cache\msys2.exe -y
    - ((Get-Content -path .\msys64\etc\\post-install\\07-pacman-key.post -Raw)
        -replace '--refresh-keys', '--version') |
@@ -36,6 +38,7 @@
    - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu'  # Normal update
    - taskkill /F /FI "MODULES eq msys-2.0.dll"
    script:
+  - Write-Output "Installing mingw packages at $(Get-Date -Format u)"
    - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
        bison diffutils flex
        git grep make sed
@@ -66,6 +69,7 @@
        $MINGW_TARGET-spice
        $MINGW_TARGET-usbredir
        $MINGW_TARGET-zstd "
+  - Write-Output "Running build at $(Get-Date -Format u)"
    - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
    - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
    - mkdir build

.gitlab-ci.d/windows.yml line 83

Quoted 7 lines
.gitlab-ci.d/windows.yml +1 -0
@@ -73,6 +77,7 @@
    - ..\msys64\usr\bin\bash -lc "../configure --enable-fdt=system $CONFIGURE_ARGS"
    - ..\msys64\usr\bin\bash -lc "make"
    - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
+  - Write-Output "Finished build at $(Get-Date -Format u)"
 
  msys2-64bit:
    extends: .shared_msys2_builder
#5 lore raw

PATCH 3/8 gitlab: always use updated msys installer

In reply to Daniel P. Berrangé

We current reference an msys installer binary from mid-2022, which means after installation, it immediately has to re-download a bunch of newer content. This wastes precious CI time.

The msys project publishes an installer binary with a fixed URL that always references the latest content. We cache the downloads in gitlab though and so once downloaded we would never re-fetch the installer leading back to the same problem.

To deal with this we also fetch the pgp signature for the installer on every run, and compare that to the previously cached signature. If the signature changes, we re-download the full installer.

This ensures we always have the latest installer for msys, while also maximising use of the gitlab cache.

---
 .gitlab-ci.d/windows.yml | 30 +++++++++++++++++++++++++++---
 1 file changed, 27 insertions(+), 3 deletions(-)
.gitlab-ci.d/windows.yml +27 -3
index 831b080d12..0bc04ad068 100644
@@ -23,10 +23,34 @@
   - If ( !(Test-Path -Path msys64\var\cache ) ) {
       mkdir msys64\var\cache
     }
-  - If ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
+  - Invoke-WebRequest
+    "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe.sig"
+    -outfile "msys2.exe.sig"
+  - if ( Test-Path -Path msys64\var\cache\msys2.exe.sig ) {
+      Write-Output "Cached installer sig" ;
+      if ( ((Get-FileHash msys2.exe.sig).Hash -ne (Get-FileHash msys64\var\cache\msys2.exe.sig).Hash) ) {
+        Write-Output "Mis-matched installer sig, new installer download required" ;
+        Remove-Item -Path msys64\var\cache\msys2.exe.sig ;
+        if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
+          Remove-Item -Path msys64\var\cache\msys2.exe
+        }
+      } else {
+        Write-Output "Matched installer sig, cached installer still valid"
+      }
+    } else {
+      Write-Output "No cached installer sig, new installer download required" ;
+      if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
+        Remove-Item -Path msys64\var\cache\msys2.exe
+      }
+    }
+  - if ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
+      Write-Output "Fetching latest installer" ;
       Invoke-WebRequest
-      "https://github.com/msys2/msys2-installer/releases/download/2022-06-03/msys2-base-x86_64-20220603.sfx.exe"
-      -outfile "msys64\var\cache\msys2.exe"
+      "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe"
+      -outfile "msys64\var\cache\msys2.exe" ;
+      Copy-Item -Path msys2.exe.sig -Destination msys64\var\cache\msys2.exe.sig
+    } else {
+      Write-Output "Using cached installer"
     }
   - Write-Output "Invoking msys2.exe installer at $(Get-Date -Format u)"
   - msys64\var\cache\msys2.exe -y
Signature · 1 line
2.41.0
Reviewed-by on v1 3/8
#6 lore raw

Re: [PATCH 3/8] gitlab: always use updated msys installer

Quoted 53 lines · Daniel P. Berrangé

We current reference an msys installer binary from mid-2022, which means after installation, it immediately has to re-download a bunch of newer content. This wastes precious CI time.

The msys project publishes an installer binary with a fixed URL that always references the latest content. We cache the downloads in gitlab though and so once downloaded we would never re-fetch the installer leading back to the same problem.

To deal with this we also fetch the pgp signature for the installer on every run, and compare that to the previously cached signature. If the signature changes, we re-download the full installer.

This ensures we always have the latest installer for msys, while also maximising use of the gitlab cache.

---
  .gitlab-ci.d/windows.yml | 30 +++++++++++++++++++++++++++---
  1 file changed, 27 insertions(+), 3 deletions(-)
.gitlab-ci.d/windows.yml +22 -1
index 831b080d12..0bc04ad068 100644
@@ -23,10 +23,34 @@
    - If ( !(Test-Path -Path msys64\var\cache ) ) {
        mkdir msys64\var\cache
      }
-  - If ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
+  - Invoke-WebRequest
+    "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe.sig"
+    -outfile "msys2.exe.sig"
+  - if ( Test-Path -Path msys64\var\cache\msys2.exe.sig ) {
+      Write-Output "Cached installer sig" ;
+      if ( ((Get-FileHash msys2.exe.sig).Hash -ne (Get-FileHash msys64\var\cache\msys2.exe.sig).Hash) ) {
+        Write-Output "Mis-matched installer sig, new installer download required" ;
+        Remove-Item -Path msys64\var\cache\msys2.exe.sig ;
+        if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
+          Remove-Item -Path msys64\var\cache\msys2.exe
+        }
+      } else {
+        Write-Output "Matched installer sig, cached installer still valid"
+      }
+    } else {
+      Write-Output "No cached installer sig, new installer download required" ;
+      if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
+        Remove-Item -Path msys64\var\cache\msys2.exe
+      }
+    }
+  - if ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
+      Write-Output "Fetching latest installer" ;
        Invoke-WebRequest

.gitlab-ci.d/windows.yml line 56

Quoted 10 lines
.gitlab-ci.d/windows.yml +5 -2
-      "https://github.com/msys2/msys2-installer/releases/download/2022-06-03/msys2-base-x86_64-20220603.sfx.exe"
-      -outfile "msys64\var\cache\msys2.exe"
+      "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe"
+      -outfile "msys64\var\cache\msys2.exe" ;
+      Copy-Item -Path msys2.exe.sig -Destination msys64\var\cache\msys2.exe.sig
+    } else {
+      Write-Output "Using cached installer"
      }
    - Write-Output "Invoking msys2.exe installer at $(Get-Date -Format u)"
    - msys64\var\cache\msys2.exe -y

... I think the original idea was to use a "tagged" version to avoid that we have some kind of "rolling release" here, but since the latest content is fetched anyway during the following update, that idea was likely not working as expected...

#7 lore raw

Re: [PATCH 3/8] gitlab: always use updated msys installer

Quoted 20 lines · Thomas Huth
Quoted 9 lines · Daniel P. Berrangé

We current reference an msys installer binary from mid-2022, which means after installation, it immediately has to re-download a bunch of newer content. This wastes precious CI time.

The msys project publishes an installer binary with a fixed URL that always references the latest content. We cache the downloads in gitlab though and so once downloaded we would never re-fetch the installer leading back to the same problem.

Quoted 8 lines

To deal with this we also fetch the pgp signature for the installer on every run, and compare that to the previously cached signature. If the signature changes, we re-download the full installer.

This ensures we always have the latest installer for msys, while also maximising use of the gitlab cache.

Quoted 6 lines

... I think the original idea was to use a "tagged" version to avoid that we have some kind of "rolling release" here, but since the latest content is fetched anyway during the following update, that idea was likely not working as expected...

I think using the "latest" installer is also more in keeping with what we do for other distros, where we'll always pick up the latest content when the containers get rebuilt, or macOS/FreeBSD where we pick the latest from Ports/HomeBrew.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#8 lore raw

PATCH 4/8 gitlab: drop $CI_PROJECT_DIR from cache path

In reply to Daniel P. Berrangé

The gitlab cache is limited to only handle content within the $CI_PROJECT_DIR hierarchy, and as such relative paths are always implicitly relative to $CI_PROJECT_DIR.

---
 .gitlab-ci.d/windows.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.d/windows.yml +1 -1
index 0bc04ad068..6454880cb7 100644
@@ -7,7 +7,7 @@
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
-      - ${CI_PROJECT_DIR}/msys64/var/cache
+      - msys64/var/cache
   needs: []
   stage: build
   timeout: 80m
Signature · 1 line
2.41.0
Reviewed-by on v1 4/8
#9 lore raw

Re: [PATCH 4/8] gitlab: drop $CI_PROJECT_DIR from cache path

Quoted 14 lines · Daniel P. Berrangé

The gitlab cache is limited to only handle content within the $CI_PROJECT_DIR hierarchy, and as such relative paths are always implicitly relative to $CI_PROJECT_DIR.

---
  .gitlab-ci.d/windows.yml | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.d/windows.yml +0 -0
index 0bc04ad068..6454880cb7 100644

.gitlab-ci.d/windows.yml line 13

Quoted 8 lines
.gitlab-ci.d/windows.yml +1 -1
@@ -7,7 +7,7 @@
    cache:
      key: "${CI_JOB_NAME}-cache"
      paths:
-      - ${CI_PROJECT_DIR}/msys64/var/cache
+      - msys64/var/cache
    needs: []
    stage: build
    timeout: 80m
#10 lore raw

PATCH 5/8 gitlab: always populate cache for windows msys jobs

In reply to Daniel P. Berrangé

The cache is used to hold the msys installer. Even if the build phase fails, we should still populate the cache as the installer will be valid for next time.

---
 .gitlab-ci.d/windows.yml | 1 +
 1 file changed, 1 insertion(+)
.gitlab-ci.d/windows.yml +1 -0
index 6454880cb7..34109a80f2 100644
@@ -8,6 +8,7 @@
     key: "${CI_JOB_NAME}-cache"
     paths:
       - msys64/var/cache
+    when: always
   needs: []
   stage: build
   timeout: 80m
Signature · 1 line
2.41.0
Reviewed-by on v1 5/8
#11 lore raw

Re: [PATCH 5/8] gitlab: always populate cache for windows msys jobs

Quoted 14 lines · Daniel P. Berrangé

The cache is used to hold the msys installer. Even if the build phase fails, we should still populate the cache as the installer will be valid for next time.

---
  .gitlab-ci.d/windows.yml | 1 +
  1 file changed, 1 insertion(+)
.gitlab-ci.d/windows.yml +0 -0
index 6454880cb7..34109a80f2 100644

.gitlab-ci.d/windows.yml line 14

Quoted 7 lines
.gitlab-ci.d/windows.yml +1 -0
@@ -8,6 +8,7 @@
      key: "${CI_JOB_NAME}-cache"
      paths:
        - msys64/var/cache
+    when: always
    needs: []
    stage: build
    timeout: 80m
#12 lore raw

PATCH 6/8 configure: support passthrough of -Dxxx args to meson

In reply to Daniel P. Berrangé

This can be useful for setting some meson global options, such as the optimization level or debug state.xs

---
 configure | 5 +++++
 1 file changed, 5 insertions(+)
configure +5 -0
index 26ec5e4f54..9fe3718b77 100755
@@ -757,6 +757,9 @@ for opt do
   # everything else has the same name in configure and meson
   --*) meson_option_parse "$opt" "$optarg"
   ;;
+  # Pass through -Dxxxx options to meson
+  -D*) meson_options="$meson_options $opt"
+  ;;
   esac
 done
 
@@ -887,6 +890,8 @@ cat << EOF
   pie             Position Independent Executables
   debug-tcg       TCG debugging (default is disabled)
 
+  -Dmesonoptname=val      passthrough option to meson unmodified
+
 NOTE: The object files are built at the place where configure is launched
 EOF
 exit 0
Signature · 1 line
2.41.0
#13 lore raw

Re: [PATCH 6/8] configure: support passthrough of -Dxxx args to meson

Quoted 23 lines · Daniel P. Berrangé

This can be useful for setting some meson global options, such as the optimization level or debug state.xs

---
  configure | 5 +++++
  1 file changed, 5 insertions(+)
configure +3 -0
index 26ec5e4f54..9fe3718b77 100755
@@ -757,6 +757,9 @@ for opt do
    # everything else has the same name in configure and meson
    --*) meson_option_parse "$opt" "$optarg"
    ;;
+  # Pass through -Dxxxx options to meson
+  -D*) meson_options="$meson_options $opt"
+  ;;
    esac
  done
 

configure cat << EOF · line 893

Quoted 4 lines
configure +1 -0
@@ -887,6 +890,8 @@ cat << EOF
    pie             Position Independent Executables
    debug-tcg       TCG debugging (default is disabled)
 
+  -Dmesonoptname=val      passthrough option to meson unmodified

I'd rather place that earlier in the help text, above the "meson_options_help" line in the configure script, next to the other --option=something lines.

Thomas

Quoted 3 lines
configure +0 -0
  NOTE: The object files are built at the place where configure is launched
  EOF
  exit 0
#14 lore raw

Re: [PATCH 6/8] configure: support passthrough of -Dxxx args to meson

Quoted 22 lines · Thomas Huth
Quoted 21 lines · Daniel P. Berrangé

This can be useful for setting some meson global options, such as the optimization level or debug state.xs

---
  configure | 5 +++++
  1 file changed, 5 insertions(+)
configure +3 -0
index 26ec5e4f54..9fe3718b77 100755
@@ -757,6 +757,9 @@ for opt do
    # everything else has the same name in configure and meson
    --*) meson_option_parse "$opt" "$optarg"
    ;;
+  # Pass through -Dxxxx options to meson
+  -D*) meson_options="$meson_options $opt"
+  ;;
    esac
  done
Quoted 8 lines

configure cat << EOF · line 892

Quoted 4 lines
configure +1 -0
@@ -887,6 +890,8 @@ cat << EOF
    pie             Position Independent Executables
    debug-tcg       TCG debugging (default is disabled)
+  -Dmesonoptname=val      passthrough option to meson unmodified

I'd rather place that earlier in the help text, above the "meson_options_help" line in the configure script, next to the other --option=something lines.

Sure I've no preference. I was expecting Paolo to tell me to put it somewhere else anyway :-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#15 lore raw

PATCH 7/8 gitlab: disable optimization and debug symbols in msys build

In reply to Daniel P. Berrangé

Building at -O2, adds 33% to the build time, over -O2. IOW a build that takes 45 minutes at -O0, takes 60 minutes at -O2. Turning off debug symbols drops it further, down to 38 minutes.

IOW, a "-O2 -g" build is 58% slower than a "-O0" build on msys in the gitlab CI windows shared runners.

---
 .gitlab-ci.d/windows.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
.gitlab-ci.d/windows.yml +2 -2
index 34109a80f2..552e3b751d 100644
@@ -113,7 +113,7 @@ msys2-64bit:
     # commit 9f8e6cad65a6 ("gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices"
     # changed to compile QEMU with the --without-default-devices switch
     # for the msys2 64-bit job, due to the build could not complete within
-    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices
+    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices -Ddebug=false -Doptimization=0
     # qTests don't run successfully with "--without-default-devices",
     # so let's exclude the qtests from CI for now.
     TEST_ARGS: --no-suite qtest
@@ -123,5 +123,5 @@ msys2-32bit:
   variables:
     MINGW_TARGET: mingw-w64-i686
     MSYSTEM: MINGW32
-    CONFIGURE_ARGS:  --target-list=ppc64-softmmu
+    CONFIGURE_ARGS:  --target-list=ppc64-softmmu -Ddebug=false -Doptimization=0
     TEST_ARGS: --no-suite qtest
Signature · 1 line
2.41.0
#16 lore raw

Re: [PATCH 7/8] gitlab: disable optimization and debug symbols in msys build

Quoted 26 lines · Daniel P. Berrangé

Building at -O2, adds 33% to the build time, over -O2. IOW a build that takes 45 minutes at -O0, takes 60 minutes at -O2. Turning off debug symbols drops it further, down to 38 minutes.

IOW, a "-O2 -g" build is 58% slower than a "-O0" build on msys in the gitlab CI windows shared runners.

---
  .gitlab-ci.d/windows.yml | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
.gitlab-ci.d/windows.yml +1 -1
index 34109a80f2..552e3b751d 100644
@@ -113,7 +113,7 @@ msys2-64bit:
      # commit 9f8e6cad65a6 ("gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices"
      # changed to compile QEMU with the --without-default-devices switch
      # for the msys2 64-bit job, due to the build could not complete within
-    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices
+    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices -Ddebug=false -Doptimization=0
      # qTests don't run successfully with "--without-default-devices",
      # so let's exclude the qtests from CI for now.
      TEST_ARGS: --no-suite qtest

.gitlab-ci.d/windows.yml msys2-32bit: · line 127

Quoted 6 lines
.gitlab-ci.d/windows.yml +1 -1
@@ -123,5 +123,5 @@ msys2-32bit:
    variables:
      MINGW_TARGET: mingw-w64-i686
      MSYSTEM: MINGW32
-    CONFIGURE_ARGS:  --target-list=ppc64-softmmu
+    CONFIGURE_ARGS:  --target-list=ppc64-softmmu -Ddebug=false -Doptimization=0
      TEST_ARGS: --no-suite qtest

This is IMHO a very good idea! But I think for now it's enough if you only change the 64-bit, isn't it?

Thomas

#17 lore raw

Re: [PATCH 7/8] gitlab: disable optimization and debug symbols in msys build

Quoted 28 lines · Thomas Huth
Quoted 27 lines · Daniel P. Berrangé

Building at -O2, adds 33% to the build time, over -O2. IOW a build that takes 45 minutes at -O0, takes 60 minutes at -O2. Turning off debug symbols drops it further, down to 38 minutes.

IOW, a "-O2 -g" build is 58% slower than a "-O0" build on msys in the gitlab CI windows shared runners.

---
  .gitlab-ci.d/windows.yml | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
.gitlab-ci.d/windows.yml +1 -1
index 34109a80f2..552e3b751d 100644
@@ -113,7 +113,7 @@ msys2-64bit:
      # commit 9f8e6cad65a6 ("gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices"
      # changed to compile QEMU with the --without-default-devices switch
      # for the msys2 64-bit job, due to the build could not complete within
-    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices
+    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices -Ddebug=false -Doptimization=0
      # qTests don't run successfully with "--without-default-devices",
      # so let's exclude the qtests from CI for now.
      TEST_ARGS: --no-suite qtest
@@ -123,5 +123,5 @@ msys2-32bit:
    variables:
Quoted 8 lines

.gitlab-ci.d/windows.yml msys2-32bit: · line 127

Quoted 5 lines
.gitlab-ci.d/windows.yml +1 -1
      MINGW_TARGET: mingw-w64-i686
      MSYSTEM: MINGW32
-    CONFIGURE_ARGS:  --target-list=ppc64-softmmu
+    CONFIGURE_ARGS:  --target-list=ppc64-softmmu -Ddebug=false -Doptimization=0
      TEST_ARGS: --no-suite qtest

This is IMHO a very good idea! But I think for now it's enough if you only change the 64-bit, isn't it?

My thought was if we do it for 32-bit too, we can enable some more targets to get a more comprehensive build.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#18 lore raw

PATCH 8/8 gitlab: disable FF_SCRIPT_SECTIONS on msys jobs

In reply to Daniel P. Berrangé

The FF_SCRIPT_SECTIONS=1 variable should ordinarily cause output from each line of the job script to be presented in a collapsible section with execution time listed.

While it works on Linux shared runners, when used with Windows runners with PowerShell, this option does not create any sections, and actually causes echo'ing of commands to be disabled, making it even worse to debug the jobs.

---
 .gitlab-ci.d/windows.yml | 4 ++++
 1 file changed, 4 insertions(+)
.gitlab-ci.d/windows.yml +4 -0
index 552e3b751d..cd7622a761 100644
@@ -12,6 +12,10 @@
   needs: []
   stage: build
   timeout: 80m
+  variables:
+    # This feature doesn't (currently) work with PowerShell, it stops
+    # the echo'ing of commands being run and doesn't show any timing
+    FF_SCRIPT_SECTIONS: 0
   artifacts:
     name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
     expire_in: 7 days
Signature · 1 line
2.41.0
Acked-by on v1 8/8
#19 lore raw

Re: [PATCH 8/8] gitlab: disable FF_SCRIPT_SECTIONS on msys jobs

Quoted 19 lines · Daniel P. Berrangé

The FF_SCRIPT_SECTIONS=1 variable should ordinarily cause output from each line of the job script to be presented in a collapsible section with execution time listed.

While it works on Linux shared runners, when used with Windows runners with PowerShell, this option does not create any sections, and actually causes echo'ing of commands to be disabled, making it even worse to debug the jobs.

---
  .gitlab-ci.d/windows.yml | 4 ++++
  1 file changed, 4 insertions(+)
.gitlab-ci.d/windows.yml +0 -0
index 552e3b751d..cd7622a761 100644

.gitlab-ci.d/windows.yml line 21

Quoted 10 lines
.gitlab-ci.d/windows.yml +4 -0
@@ -12,6 +12,10 @@
    needs: []
    stage: build
    timeout: 80m
+  variables:
+    # This feature doesn't (currently) work with PowerShell, it stops
+    # the echo'ing of commands being run and doesn't show any timing
+    FF_SCRIPT_SECTIONS: 0
    artifacts:
      name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
      expire_in: 7 days
#20 lore raw
In reply to Daniel P. Berrangé
Quoted 11 lines · Daniel P. Berrangé

This is an alternative and/or complementary to Thomas' proposal to use CLang with msys:

https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html

First of all, the current msys installer we're using is over 12 months out of date. Thus after running the install, pacman then replaces most of what we've just installed with new downloaded content. Using the most update installer cuts 3+1/2 minutes off the msys install time - 7 minutes becomes 3+1/2.

Quoted 3 lines

Secondly, QEMU defaults to compiling with -O2 and this is more computationally expensive for GCC. Switching to -O0 drops the build time from 60 minutes down to 45 minutes.

From the fine manual[*]: "The effectiveness of some warnings depends on optimizations also being enabled. For example '-Wsuggest-final-types' is more effective with link-time optimization and some instances of other warnings may not be issued at all unless optimization is enabled. While optimization in general improves the efficacy of control and data flow sensitive warnings, in some cases it may also cause false positives." Do we care?

[...]

[*] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Warning-Options.html

#21 lore raw
Quoted 17 lines · Markus Armbruster
Quoted 11 lines · Daniel P. Berrangé

This is an alternative and/or complementary to Thomas' proposal to use CLang with msys:

https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html

First of all, the current msys installer we're using is over 12 months out of date. Thus after running the install, pacman then replaces most of what we've just installed with new downloaded content. Using the most update installer cuts 3+1/2 minutes off the msys install time - 7 minutes becomes 3+1/2.

Quoted 3 lines

Secondly, QEMU defaults to compiling with -O2 and this is more computationally expensive for GCC. Switching to -O0 drops the build time from 60 minutes down to 45 minutes.

Quoted 7 lines

From the fine manual[*]: "The effectiveness of some warnings depends on optimizations also being enabled. For example '-Wsuggest-final-types' is more effective with link-time optimization and some instances of other warnings may not be issued at all unless optimization is enabled. While optimization in general improves the efficacy of control and data flow sensitive warnings, in some cases it may also cause false positives." Do we care?

In general, yes, we do care.

In this specific case though, we're battling to figure out the lesser of multiple evils.

Right now we configure with:

  --target-list=x86_64-softmmu --without-default-devices 

and so with optimization enabled, we'll get good warning coverage of a small amount of code, except we don't because people started ignoring the msys jobs as they timeout too frequently.

If we can use Thomas' clang switch or my -O0 patches, we can get within the timeout, so people can trust the job once again. If we can do both ideas and cut the time even more, then we can enable more features (perhaps drop --without-default-devices).

So the warnings might not be quite as good, but we'll have the warnings across a larger amount of code.

Alot of the warnings from the Linux/macOS builds will also apply in the Windows builds. I think on balance I'd probably prefer us to build a larger amount of code for Windows. This is in context of free shared runners at least.

As a more drastic option, we might need to consider using the Azure credits for Windows runners too. If we could have bigger VMs for Windows CI, we can build more and have better warnings at the same time.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#22 lore raw
Quoted 26 lines · Daniel P. Berrangé
Quoted 17 lines · Markus Armbruster
Quoted 11 lines · Daniel P. Berrangé

This is an alternative and/or complementary to Thomas' proposal to use CLang with msys:

   https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html

First of all, the current msys installer we're using is over 12 months out of date. Thus after running the install, pacman then replaces most of what we've just installed with new downloaded content. Using the most update installer cuts 3+1/2 minutes off the msys install time - 7 minutes becomes 3+1/2.

Quoted 3 lines

Secondly, QEMU defaults to compiling with -O2 and this is more computationally expensive for GCC. Switching to -O0 drops the build time from 60 minutes down to 45 minutes.

Quoted 7 lines

From the fine manual[*]: "The effectiveness of some warnings depends on
optimizations also being enabled. For example '-Wsuggest-final-types' is more effective with link-time optimization and some instances of other warnings may not be issued at all unless optimization is enabled. While optimization in general improves the efficacy of control and data flow sensitive warnings, in some cases it may also cause false positives." Do we care?

Quoted 4 lines

In general, yes, we do care.

In this specific case though, we're battling to figure out the lesser of multiple evils.

I agree. Additionally, we also test compiling for Windows with the MinGW cross compiler suite in a Fedora container, and we still use the default optimization there, so we should have that covered.

Thomas

#23 lore raw
In reply to Daniel P. Berrangé
Quoted 22 lines · Daniel P. Berrangé

This is an alternative and/or complementary to Thomas' proposal to use CLang with msys:

   https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html

First of all, the current msys installer we're using is over 12 months out of date. Thus after running the install, pacman then replaces most of what we've just installed with new downloaded content. Using the most update installer cuts 3+1/2 minutes off the msys install time - 7 minutes becomes 3+1/2.

Secondly, QEMU defaults to compiling with -O2 and this is more computationally expensive for GCC. Switching to -O0 drops the build time from 60 minutes down to 45 minutes.

Thirdly, including debug symbols also has an overhead, and turning that off reduces time still further down to 38 minutes.

IOW, between all three changes, we can cut approx 25-26 minutes off the job execution time, bringing it nicely within the job timeout.

Quoted 7 lines

The actually phase of installing the mingw deps still accounts for about 10 minutes and has not been optimized.

Possibly the same trick of -O0 and skipping -g would also help the clang alternative Thomas' proposed. If so, that could be enough to let us enable more features / targets during the msys build.

I really like the idea! And I guess my idea with Clang needs some more work 'til it is acceptable, so let's go with your idea for now to fix the timeout problem in the CI ... we can still optimize later with Clang in case we found a good solution for that ms_struct problem...

Thomas